Skip to contentSkip to conversation

TTM shrinker helpers and xe buffer object shrinker

PATCH Superseded Thomas Hellström 6 participants 41 messages Jul 3, 2024 – Aug 16, 2024

Subsystems (5)

v6 of 17 Superseded Latest is v17 →

lore
Share & tools
lore

Thomas Hellström

author maintainer
#0 lore raw

v6 cover TTM shrinker helpers and xe buffer object shrinker

This series implements TTM shrinker / eviction helpers and an xe bo shrinker. It builds on two previous series, *and obsoletes these*. First

https://www.mail-archive.com/[email protected]/msg484425.html

Second the previous TTM shrinker series

https://lore.kernel.org/linux-mm/[email protected]/T/

Where the comment about layering
https://lore.kernel.org/linux-mm/[email protected]/T/#ma918844aa8a6efe8768fdcda0c6590d5c93850c9

now addressed, and this version also implements shmem objects for backup rather than direct swap-cache insertions, which was used in the previuos series. It turns out that with per-page backup / shrinking, shmem objects appears to work just as well as direct swap-cache insertions with the added benefit that was introduced in the previous TTM shrinker series to avoid running out of swap entries isn't really needed.

Patch 1-4 implements restartable LRU list iteration.

Patch 5 implements a LRU walker + resv locking helper

Patch 6 moves TTM swapping over to the walker.

Patch 7 moves TTM eviction over to the walker.

Patch 8 could in theory be skipped but introduces a possibility to easily add or test multiple backup backends, like the direct swap-cache insertion or even files into fast dedicated nvme storage for for example.

Patch 9 introduces helpers in the ttm_pool code for page-by-page shrinking and recovery. It avoids having to temporarily allocate a huge amount of memory to be able to shrink a buffer object. It also introduces the possibility to immediately write-back pages if needed, since that tends to be a bit delayed when left to kswapd.

Patch 10 Adds a simple error injection to the above code to help increase test coverage.

Patch 11 Implements an xe bo shrinker and a common helper in TTM for shrinking.

Patch 12-21 are really a separate POC series, for introducing drm_exec locking in TTM. The patch touches both drm_exec and dma-buf and is for now marked as an RFC:

Patch 12 Increases (removes) the XE_PL_TT watermark.

v2:
- Squash obsolete revision history in the patch commit messages.
- Fix a couple of review comments by Christian
- Don't store the mem_type in the TTM managers but in the resource cursor.
- Rename introduced TTM *back_up* function names to *backup*
- Add ttm pool recovery fault injection.
- Shrinker xe kunit test
- Various bugfixes

v3:
- Address some review comments from Matthew Brost and Christian König.
- Use the restartable LRU walk for TTM swapping and eviction.
- Provide a POC drm_exec locking implementation for exhaustive eviction. (Christian König).

v4:
- Remove the RFC exhaustive eviction part. While the path to exhaustive eviction is pretty clear and demonstrated in v3, there is still some drm_exec work that needs to be agreed and implemented.
- Add shrinker power management. On some hw we need to wake when shrinking.
- Fix the lru walker helper for -EALREADY errors.
- Add drm/xe: Increase the XE_PL_TT watermark.

v5:
- Update also TTM kunit tests
- Handle ghost- and zombie objects in the shrinker.
- A couple of compile- and UAF fixes reported by Kernel Build Robot and Dan Carpenter.

v6:
- Address review comments from Matthew Brost as detailed in patches 4/12, 5/12, 6/12, 7/12, 8/12.

Cc Somalapuram Amaranath <[email protected]>
Cc Christian König <[email protected]>
Cc Matthew Brost <[email protected]>

Thomas Hellström (12):
drm/ttm: Allow TTM LRU list nodes of different types drm/ttm: Slightly clean up LRU list iteration
drm/ttm: Use LRU hitches
drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist
moves
drm/ttm: Provide a generic LRU walker helper
drm/ttm: Use the LRU walker helper for swapping
drm/ttm: Use the LRU walker for eviction
drm/ttm: Add a virtual base class for graphics memory backup drm/ttm/pool: Provide a helper to shrink pages
drm/ttm: Use fault-injection to test error paths
drm/ttm, drm/xe: Add a shrinker for xe bos
drm/xe: Increase the XE_PL_TT watermark

 drivers/gpu/drm/Kconfig                       |  10 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |   4 +
 drivers/gpu/drm/ttm/Makefile                  |   2 +-
 drivers/gpu/drm/ttm/tests/ttm_bo_test.c       |   6 +-
 drivers/gpu/drm/ttm/tests/ttm_resource_test.c |   2 +-
 drivers/gpu/drm/ttm/ttm_backup_shmem.c        | 139 ++++++
 drivers/gpu/drm/ttm/ttm_bo.c                  | 458 ++++++++----------
 drivers/gpu/drm/ttm/ttm_bo_util.c             | 223 +++++++++
 drivers/gpu/drm/ttm/ttm_device.c              |  29 +-
 drivers/gpu/drm/ttm/ttm_pool.c                | 412 +++++++++++++++-
 drivers/gpu/drm/ttm/ttm_resource.c            | 268 ++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c                  |  37 ++
 drivers/gpu/drm/xe/Makefile                   |   1 +
 drivers/gpu/drm/xe/tests/xe_bo.c              | 118 +++++
 drivers/gpu/drm/xe/tests/xe_bo_test.c         |   1 +
 drivers/gpu/drm/xe/tests/xe_bo_test.h         |   1 +
 drivers/gpu/drm/xe/xe_bo.c                    | 155 +++++-
 drivers/gpu/drm/xe/xe_bo.h                    |  26 +
 drivers/gpu/drm/xe/xe_device.c                |   8 +
 drivers/gpu/drm/xe/xe_device_types.h          |   2 +
 drivers/gpu/drm/xe/xe_shrinker.c              | 287 +++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h              |  18 +
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c           |   3 +-
 drivers/gpu/drm/xe/xe_vm.c                    |   4 +
 include/drm/ttm/ttm_backup.h                  | 137 ++++++
 include/drm/ttm/ttm_bo.h                      |  51 +-
 include/drm/ttm/ttm_pool.h                    |   5 +
 include/drm/ttm/ttm_resource.h                |  99 +++-
 include/drm/ttm/ttm_tt.h                      |  20 +
 29 files changed, 2149 insertions(+), 377 deletions(-)
 create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
 create mode 100644 include/drm/ttm/ttm_backup.h
Signature · 1 line
2.44.0
author maintainer
#1 lore raw

PATCH v6 1/12 drm/ttm: Allow TTM LRU list nodes of different types

To be able to handle list unlocking while traversing the LRU list, we want the iterators not only to point to the next position of the list traversal, but to insert themselves as list nodes at that point to work around the fact that the next node might otherwise disappear from the list while the iterator is pointing to it.

These list nodes need to be easily distinguishable from other list nodes so that others traversing the list can skip over them.

So declare a struct ttm_lru_item, with a struct list_head member and a type enum. This will slightly increase the size of a struct ttm_resource.

Changes in previous series:
- Update enum ttm_lru_item_type documentation.
v3:
- Introduce ttm_lru_first_res_or_null()
(Christian König, Thomas Hellström)
v5:
- Update also the TTM test code (Xe CI).

---
 drivers/gpu/drm/ttm/tests/ttm_bo_test.c       |  6 +-
 drivers/gpu/drm/ttm/tests/ttm_resource_test.c |  2 +-
 drivers/gpu/drm/ttm/ttm_device.c              |  4 +-
 drivers/gpu/drm/ttm/ttm_resource.c            | 89 +++++++++++++++----
 include/drm/ttm/ttm_resource.h                | 54 ++++++++++-
 5 files changed, 129 insertions(+), 26 deletions(-)
drivers/gpu/drm/ttm/tests/ttm_bo_test.c +3 -3
index d1b32303d051..f0a7eb62116c 100644
@@ -271,7 +271,7 @@ static void ttm_bo_unreserve_basic(struct kunit *test)
 
 	man = ttm_manager_type(priv->ttm_dev, mem_type);
 	KUNIT_ASSERT_EQ(test,
-			list_is_last(&res1->lru, &man->lru[bo->priority]), 1);
+			list_is_last(&res1->lru.link, &man->lru[bo->priority]), 1);
 
 	ttm_resource_free(bo, &res2);
 	ttm_resource_free(bo, &res1);
@@ -308,11 +308,11 @@ static void ttm_bo_unreserve_pinned(struct kunit *test)
 	err = ttm_resource_alloc(bo, place, &res2);
 	KUNIT_ASSERT_EQ(test, err, 0);
 	KUNIT_ASSERT_EQ(test,
-			list_is_last(&res2->lru, &priv->ttm_dev->pinned), 1);
+			list_is_last(&res2->lru.link, &priv->ttm_dev->pinned), 1);
 
 	ttm_bo_unreserve(bo);
 	KUNIT_ASSERT_EQ(test,
-			list_is_last(&res1->lru, &priv->ttm_dev->pinned), 1);
+			list_is_last(&res1->lru.link, &priv->ttm_dev->pinned), 1);
 
 	ttm_resource_free(bo, &res1);
 	ttm_resource_free(bo, &res2);
drivers/gpu/drm/ttm/tests/ttm_resource_test.c +1 -1
index 9c2f13e53162..22260e7aea58 100644
@@ -198,7 +198,7 @@ static void ttm_resource_fini_basic(struct kunit *test)
 	ttm_resource_init(bo, place, res);
 	ttm_resource_fini(man, res);
 
-	KUNIT_ASSERT_TRUE(test, list_empty(&res->lru));
+	KUNIT_ASSERT_TRUE(test, list_empty(&res->lru.link));
 	KUNIT_ASSERT_EQ(test, man->usage, 0);
 }
 
drivers/gpu/drm/ttm/ttm_device.c +2 -2
index 434cf0258000..09411978a13a 100644
@@ -274,14 +274,14 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,
 	struct ttm_resource *res;
 
 	spin_lock(&bdev->lru_lock);
-	while ((res = list_first_entry_or_null(list, typeof(*res), lru))) {
+	while ((res = ttm_lru_first_res_or_null(list))) {
 		struct ttm_buffer_object *bo = res->bo;
 
 		/* Take ref against racing releases once lru_lock is unlocked */
 		if (!ttm_bo_get_unless_zero(bo))
 			continue;
 
-		list_del_init(&res->lru);
+		list_del_init(&bo->resource->lru.link);
 		spin_unlock(&bdev->lru_lock);
 
 		if (bo->ttm)
drivers/gpu/drm/ttm/ttm_resource.c +70 -19

162 lines not on this page · raw message

include/drm/ttm/ttm_resource.h +53 -1

74 lines not on this page · raw message

Signature · 1 line
2.44.0
author maintainer
#2 lore raw

PATCH v6 2/12 drm/ttm: Slightly clean up LRU list iteration

In reply to Thomas Hellström

To make the transition to using lru hitches easier, simplify the ttm_resource_manager_next() interface to only take the cursor and reuse ttm_resource_manager_next() functionality from ttm_resource_manager_first().

---
 drivers/gpu/drm/ttm/ttm_resource.c | 48 +++++++++++++-----------------
 include/drm/ttm/ttm_resource.h     | 10 ++++---
 2 files changed, 27 insertions(+), 31 deletions(-)
drivers/gpu/drm/ttm/ttm_resource.c +21 -27
index db9a7a3717c4..8bfbddddc0e8 100644
@@ -496,50 +496,44 @@ struct ttm_resource *
 ttm_resource_manager_first(struct ttm_resource_manager *man,
 			   struct ttm_resource_cursor *cursor)
 {
-	struct ttm_lru_item *lru;
-
 	lockdep_assert_held(&man->bdev->lru_lock);
 
-	for (cursor->priority = 0; cursor->priority < TTM_MAX_BO_PRIORITY;
-	     ++cursor->priority)
-		list_for_each_entry(lru, &man->lru[cursor->priority], link) {
-			if (ttm_lru_item_is_res(lru))
-				return ttm_lru_item_to_res(lru);
-		}
-
-	return NULL;
+	cursor->priority = 0;
+	cursor->man = man;
+	cursor->cur = &man->lru[cursor->priority];
+	return ttm_resource_manager_next(cursor);
 }
 
 /**
  * ttm_resource_manager_next
  *
- * @man: resource manager to iterate over
  * @cursor: cursor to record the position
- * @res: the current resource pointer
  *
- * Returns the next resource from the resource manager.
+ * Return: the next resource from the resource manager.
  */
 struct ttm_resource *
-ttm_resource_manager_next(struct ttm_resource_manager *man,
-			  struct ttm_resource_cursor *cursor,
-			  struct ttm_resource *res)
+ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
 {
-	struct ttm_lru_item *lru = &res->lru;
+	struct ttm_resource_manager *man = cursor->man;
+	struct ttm_lru_item *lru;
 
 	lockdep_assert_held(&man->bdev->lru_lock);
 
-	list_for_each_entry_continue(lru, &man->lru[cursor->priority], link) {
-		if (ttm_lru_item_is_res(lru))
-			return ttm_lru_item_to_res(lru);
-	}
-
-	for (++cursor->priority; cursor->priority < TTM_MAX_BO_PRIORITY;
-	     ++cursor->priority)
-		list_for_each_entry(lru, &man->lru[cursor->priority], link) {
-			if (ttm_lru_item_is_res(lru))
-				ttm_lru_item_to_res(lru);
+	for (;;) {
+		lru = list_entry(cursor->cur, typeof(*lru), link);
+		list_for_each_entry_continue(lru, &man->lru[cursor->priority], link) {
+			if (ttm_lru_item_is_res(lru)) {
+				cursor->cur = &lru->link;
+				return ttm_lru_item_to_res(lru);
+			}
 		}
 
+		if (++cursor->priority >= TTM_MAX_BO_PRIORITY)
+			break;
+
+		cursor->cur = &man->lru[cursor->priority];
+	}
+
 	return NULL;
 }
 
include/drm/ttm/ttm_resource.h +6 -4
index 1511d91e290d..7d81fd5b5b83 100644
@@ -272,11 +272,15 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
 /**
  * struct ttm_resource_cursor
  *
+ * @man: The resource manager currently being iterated over.
+ * @cur: The list head the cursor currently points to.
  * @priority: the current priority
  *
  * Cursor to iterate over the resources in a manager.
  */
 struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct list_head *cur;
 	unsigned int priority;
 };
 
@@ -438,9 +442,7 @@ struct ttm_resource *
 ttm_resource_manager_first(struct ttm_resource_manager *man,
 			   struct ttm_resource_cursor *cursor);
 struct ttm_resource *
-ttm_resource_manager_next(struct ttm_resource_manager *man,
-			  struct ttm_resource_cursor *cursor,
-			  struct ttm_resource *res);
+ttm_resource_manager_next(struct ttm_resource_cursor *cursor);
 
 struct ttm_resource *
 ttm_lru_first_res_or_null(struct list_head *head);
@@ -455,7 +457,7 @@ ttm_lru_first_res_or_null(struct list_head *head);
  */
 #define ttm_resource_manager_for_each_res(man, cursor, res)		\
 	for (res = ttm_resource_manager_first(man, cursor); res;	\
-	     res = ttm_resource_manager_next(man, cursor, res))
+	     res = ttm_resource_manager_next(cursor))
 
 struct ttm_kmap_iter *
 ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
Signature · 1 line
2.44.0
author maintainer
#3 lore raw

PATCH v6 3/12 drm/ttm: Use LRU hitches

In reply to Thomas Hellström

Have iterators insert themselves into the list they are iterating over using hitch list nodes. Since only the iterator owner can remove these list nodes from the list, it's safe to unlock the list and when continuing, use them as a starting point. Due to the way LRU bumping works in TTM, newly added items will not be missed, and bumped items will be iterated over a second time before reaching the end of the list.

The exception is list with bulk move sublists. When bumping a sublist, a hitch that is part of that sublist will also be moved and we might miss items if restarting from it. This will be addressed in a later patch.

Changes in previous series:
- Updated ttm_resource_cursor_fini() documentation. v2:
- Don't reorder ttm_resource_manager_first() and _next(). (Christian König).
- Use list_add instead of list_move
(Christian König)
v3:
- Split into two patches, one cleanup, one new functionality (Christian König)
- use ttm_resource_cursor_fini_locked() instead of open-coding (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo.c       |  1 +
 drivers/gpu/drm/ttm/ttm_device.c   |  9 +++--
 drivers/gpu/drm/ttm/ttm_resource.c | 56 +++++++++++++++++++++++++-----
 include/drm/ttm/ttm_resource.h     |  9 +++--
 4 files changed, 62 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 6396dece0db1..43eda720657f 100644
@@ -621,6 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
 		if (locked)
 			dma_resv_unlock(res->bo->base.resv);
 	}
+	ttm_resource_cursor_fini_locked(&cursor);
 
 	if (!bo) {
 		if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
drivers/gpu/drm/ttm/ttm_device.c +7 -2
index 09411978a13a..f9e9b1ec8c8a 100644
@@ -170,12 +170,17 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
 			num_pages = PFN_UP(bo->base.size);
 			ret = ttm_bo_swapout(bo, ctx, gfp_flags);
 			/* ttm_bo_swapout has dropped the lru_lock */
-			if (!ret)
+			if (!ret) {
+				ttm_resource_cursor_fini(&cursor);
 				return num_pages;
-			if (ret != -EBUSY)
+			}
+			if (ret != -EBUSY) {
+				ttm_resource_cursor_fini(&cursor);
 				return ret;
+			}
 		}
 	}
+	ttm_resource_cursor_fini_locked(&cursor);
 	spin_unlock(&bdev->lru_lock);
 	return 0;
 }
drivers/gpu/drm/ttm/ttm_resource.c +47 -9

97 lines not on this page · raw message

include/drm/ttm/ttm_resource.h +7 -2
index 7d81fd5b5b83..8fac781f641e 100644
@@ -273,17 +273,22 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
  * struct ttm_resource_cursor
  *
  * @man: The resource manager currently being iterated over.
- * @cur: The list head the cursor currently points to.
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
  * @priority: the current priority
  *
  * Cursor to iterate over the resources in a manager.
  */
 struct ttm_resource_cursor {
 	struct ttm_resource_manager *man;
-	struct list_head *cur;
+	struct ttm_lru_item hitch;
 	unsigned int priority;
 };
 
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+
 /**
  * struct ttm_lru_bulk_move_pos
  *
Signature · 1 line
2.44.0
maintainer Reviewed-by on v6 3/12
#4 lore raw

Re: [PATCH v6 03/12] drm/ttm: Use LRU hitches

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 26 lines

Have iterators insert themselves into the list they are iterating over using hitch list nodes. Since only the iterator owner can remove these list nodes from the list, it's safe to unlock the list and when continuing, use them as a starting point. Due to the way LRU bumping works in TTM, newly added items will not be missed, and bumped items will be iterated over a second time before reaching the end of the list.

The exception is list with bulk move sublists. When bumping a sublist, a hitch that is part of that sublist will also be moved and we might miss items if restarting from it. This will be addressed in a later patch.

Changes in previous series:
- Updated ttm_resource_cursor_fini() documentation. v2:
- Don't reorder ttm_resource_manager_first() and _next(). (Christian König).
- Use list_add instead of list_move
(Christian König)
v3:
- Split into two patches, one cleanup, one new functionality (Christian König)
- use ttm_resource_cursor_fini_locked() instead of open-coding (Matthew Brost)

Quoted 6 lines
Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
Quoted 178 lines
---
  drivers/gpu/drm/ttm/ttm_bo.c       |  1 +
  drivers/gpu/drm/ttm/ttm_device.c   |  9 +++--
  drivers/gpu/drm/ttm/ttm_resource.c | 56 +++++++++++++++++++++++++-----
  include/drm/ttm/ttm_resource.h     |  9 +++--
  4 files changed, 62 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 6396dece0db1..43eda720657f 100644
@@ -621,6 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
  		if (locked)
  			dma_resv_unlock(res->bo->base.resv);
  	}
+	ttm_resource_cursor_fini_locked(&cursor);
 
  	if (!bo) {
  		if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
drivers/gpu/drm/ttm/ttm_device.c +7 -2
index 09411978a13a..f9e9b1ec8c8a 100644
@@ -170,12 +170,17 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
  			num_pages = PFN_UP(bo->base.size);
  			ret = ttm_bo_swapout(bo, ctx, gfp_flags);
  			/* ttm_bo_swapout has dropped the lru_lock */
-			if (!ret)
+			if (!ret) {
+				ttm_resource_cursor_fini(&cursor);
  				return num_pages;
-			if (ret != -EBUSY)
+			}
+			if (ret != -EBUSY) {
+				ttm_resource_cursor_fini(&cursor);
  				return ret;
+			}
  		}
  	}
+	ttm_resource_cursor_fini_locked(&cursor);
  	spin_unlock(&bdev->lru_lock);
  	return 0;
  }
drivers/gpu/drm/ttm/ttm_resource.c +47 -9
index 8bfbddddc0e8..9c8b6499edfb 100644
@@ -33,6 +33,37 @@
 
  #include <drm/drm_util.h>
 
+/**
+ * ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage
+ * @cursor: The struct ttm_resource_cursor to finalize.
+ *
+ * The function pulls the LRU list cursor off any lists it was previusly
+ * attached to. Needs to be called with the LRU lock held. The function
+ * can be called multiple times after eachother.
+ */
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	list_del_init(&cursor->hitch.link);
+}
+
+/**
+ * ttm_resource_cursor_fini() - Finalize the LRU list cursor usage
+ * @cursor: The struct ttm_resource_cursor to finalize.
+ *
+ * The function pulls the LRU list cursor off any lists it was previusly
+ * attached to. Needs to be called without the LRU list lock held. The
+ * function can be called multiple times after eachother.
+ */
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor)
+{
+	spinlock_t *lru_lock = &cursor->man->bdev->lru_lock;
+
+	spin_lock(lru_lock);
+	ttm_resource_cursor_fini_locked(cursor);
+	spin_unlock(lru_lock);
+}
+
  /**
   * ttm_lru_bulk_move_init - initialize a bulk move structure
   * @bulk: the structure to init
@@ -485,12 +516,15 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
  EXPORT_SYMBOL(ttm_resource_manager_debug);
 
  /**
- * ttm_resource_manager_first
- *
+ * ttm_resource_manager_first() - Start iterating over the resources
+ * of a resource manager
   * @man: resource manager to iterate over
   * @cursor: cursor to record the position
   *
- * Returns the first resource from the resource manager.
+ * Initializes the cursor and starts iterating. When done iterating,
+ * the caller must explicitly call ttm_resource_cursor_fini().
+ *
+ * Return: The first resource from the resource manager.
   */
  struct ttm_resource *
  ttm_resource_manager_first(struct ttm_resource_manager *man,
@@ -500,13 +534,15 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
 
  	cursor->priority = 0;
  	cursor->man = man;
-	cursor->cur = &man->lru[cursor->priority];
+	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	list_add(&cursor->hitch.link, &man->lru[cursor->priority]);
+
  	return ttm_resource_manager_next(cursor);
  }
 
  /**
- * ttm_resource_manager_next
- *
+ * ttm_resource_manager_next() - Continue iterating over the resource manager
+ * resources
   * @cursor: cursor to record the position
   *
   * Return: the next resource from the resource manager.
@@ -520,10 +556,10 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
  	lockdep_assert_held(&man->bdev->lru_lock);
 
  	for (;;) {
-		lru = list_entry(cursor->cur, typeof(*lru), link);
+		lru = &cursor->hitch;
  		list_for_each_entry_continue(lru, &man->lru[cursor->priority], link) {
  			if (ttm_lru_item_is_res(lru)) {
-				cursor->cur = &lru->link;
+				list_move(&cursor->hitch.link, &lru->link);
  				return ttm_lru_item_to_res(lru);
  			}
  		}
@@ -531,9 +567,11 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
  		if (++cursor->priority >= TTM_MAX_BO_PRIORITY)
  			break;
 
-		cursor->cur = &man->lru[cursor->priority];
+		list_move(&cursor->hitch.link, &man->lru[cursor->priority]);
  	}
 
+	ttm_resource_cursor_fini_locked(cursor);
+
  	return NULL;
  }
 
include/drm/ttm/ttm_resource.h +7 -2
index 7d81fd5b5b83..8fac781f641e 100644
@@ -273,17 +273,22 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
   * struct ttm_resource_cursor
   *
   * @man: The resource manager currently being iterated over.
- * @cur: The list head the cursor currently points to.
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
   * @priority: the current priority
   *
   * Cursor to iterate over the resources in a manager.
   */
  struct ttm_resource_cursor {
  	struct ttm_resource_manager *man;
-	struct list_head *cur;
+	struct ttm_lru_item hitch;
  	unsigned int priority;
  };
 
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+
  /**
   * struct ttm_lru_bulk_move_pos
   *
author maintainer
#5 lore raw

PATCH v6 4/12 drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

In reply to Thomas Hellström

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++
 drivers/gpu/drm/ttm/ttm_resource.c     | 92 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.c             |  4 ++
 include/drm/ttm/ttm_resource.h         | 56 ++++++++++------
 4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (r)
 		return r;
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
 	vm->is_compute_context = false;
 
 	vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 error_free_delayed:
 	dma_fence_put(vm->last_tlb_flush);
 	dma_fence_put(vm->last_unlocked);
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
 	amdgpu_vm_fini_entities(vm);
 
 	return r;
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 		}
 	}
 
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
 }
 
 /**
drivers/gpu/drm/ttm/ttm_resource.c +92 -0

143 lines not on this page · raw message

drivers/gpu/drm/xe/xe_vm.c +4 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 
 	INIT_WORK(&vm->destroy_work, vm_destroy_work_func);
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
 	INIT_LIST_HEAD(&vm->preempt.exec_queues);
 	vm->preempt.min_run_period_ms = 10;	/* FIXME: Wire up to uAPI */
 
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 	mutex_destroy(&vm->snap_mutex);
 	for_each_tile(tile, xe, id)
 		xe_range_fence_tree_fini(&vm->rftree[id]);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
 	kfree(vm);
 	if (flags & XE_VM_FLAG_LR_MODE)
 		xe_pm_runtime_put(xe);
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct work_struct *w)
 		XE_WARN_ON(vm->pt_root[id]);
 
 	trace_xe_vm_free(vm);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
 	kfree(vm);
 }
 
include/drm/ttm/ttm_resource.h +35 -21

83 lines not on this page · raw message

Signature · 1 line
2.44.0
maintainer Reviewed-by on v6 4/12
#6 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

Quoted 27 lines · Thomas Hellström

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Quoted 3 lines
Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
Quoted 316 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++
 drivers/gpu/drm/ttm/ttm_resource.c     | 92 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.c             |  4 ++
 include/drm/ttm/ttm_resource.h         | 56 ++++++++++------
 4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (r)
 		return r;
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
 	vm->is_compute_context = false;
 
 	vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 error_free_delayed:
 	dma_fence_put(vm->last_tlb_flush);
 	dma_fence_put(vm->last_unlocked);
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
 	amdgpu_vm_fini_entities(vm);
 
 	return r;
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 		}
 	}
 
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
 }
 
 /**
drivers/gpu/drm/ttm/ttm_resource.c +92 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@
 
 #include <drm/drm_util.h>
 
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct ttm_lru_bulk_move *bulk,
+					       struct ttm_resource_cursor *cursor)
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last->lru.link);
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct ttm_lru_bulk_move *bulk)
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list, bulk_link)
+		ttm_resource_cursor_move_bulk_tail(bulk, cursor);
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct ttm_lru_bulk_move *bulk)
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list, bulk_link)
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+
 /**
  * ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage
  * @cursor: The struct ttm_resource_cursor to finalize.
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor)
 {
 	lockdep_assert_held(&cursor->man->bdev->lru_lock);
 	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
 }
 
 /**
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor)
 void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
 {
 	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
 }
 EXPORT_SYMBOL(ttm_lru_bulk_move_init);
 
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+
 /**
  * ttm_lru_bulk_move_tail - bulk move range of resources to the LRU tail.
  *
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
 {
 	unsigned i, j;
 
+	ttm_bulk_move_adjust_cursors(bulk);
 	for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
 		for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {
 			struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j];
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
 }
 EXPORT_SYMBOL(ttm_resource_manager_debug);
 
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor *cursor,
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next = ttm_lru_item_to_res(next_lru);
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link, &bulk->cursor_list);
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+
 /**
  * ttm_resource_manager_first() - Start iterating over the resources
  * of a resource manager
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
 	cursor->priority = 0;
 	cursor->man = man;
 	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);
 	list_add(&cursor->hitch.link, &man->lru[cursor->priority]);
 
 	return ttm_resource_manager_next(cursor);
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
 		lru = &cursor->hitch;
 		list_for_each_entry_continue(lru, &man->lru[cursor->priority], link) {
 			if (ttm_lru_item_is_res(lru)) {
+				ttm_resource_cursor_check_bulk(cursor, lru);
 				list_move(&cursor->hitch.link, &lru->link);
 				return ttm_lru_item_to_res(lru);
 			}
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
 			break;
 
 		list_move(&cursor->hitch.link, &man->lru[cursor->priority]);
+		ttm_resource_cursor_clear_bulk(cursor);
 	}
 
 	ttm_resource_cursor_fini_locked(cursor);
drivers/gpu/drm/xe/xe_vm.c +4 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 
 	INIT_WORK(&vm->destroy_work, vm_destroy_work_func);
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
 	INIT_LIST_HEAD(&vm->preempt.exec_queues);
 	vm->preempt.min_run_period_ms = 10;	/* FIXME: Wire up to uAPI */
 
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 	mutex_destroy(&vm->snap_mutex);
 	for_each_tile(tile, xe, id)
 		xe_range_fence_tree_fini(&vm->rftree[id]);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
 	kfree(vm);
 	if (flags & XE_VM_FLAG_LR_MODE)
 		xe_pm_runtime_put(xe);
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct work_struct *w)
 		XE_WARN_ON(vm->pt_root[id]);
 
 	trace_xe_vm_free(vm);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
 	kfree(vm);
 }
 
include/drm/ttm/ttm_resource.h +35 -21
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
 	return container_of(item, struct ttm_resource, lru);
 }
 
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
-
 /**
  * struct ttm_lru_bulk_move_pos
  *
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {
 
 /**
  * struct ttm_lru_bulk_move
- *
  * @pos: first/last lru entry for resources in the each domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock.
  *
  * Container for the current bulk move state. Should be used with
  * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
  */
 struct ttm_lru_bulk_move {
 	struct ttm_lru_bulk_move_pos pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];
+	struct list_head cursor_list;
 };
 
+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange @hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+
 /**
  * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
  * struct sg_table backed struct ttm_resource.
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct ttm_resource_manager *man)
 
 void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk);
 void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);
 
 void ttm_resource_add_bulk_move(struct ttm_resource *res,
 				struct ttm_buffer_object *bo);
Signature · 1 line
2.44.0
maintainer
#7 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

In reply to Thomas Hellström

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 11 lines

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

This is the only patch in the series which I see critical.

I think the final goal when using drm_exec in TTMs eviction path is to keep all evicted (or evicting) BOs locked until we have enough space.

This means that for bulk move sections on the LRU we would lock the first BO and would only drop that lock again if we have gone over the full bulk move section and know that all BOs are not valuable for eviction.

Because of this the issue of having to consider hitches move with a bulk move section on the LRU doesn't even occur because for that a concurrent process would need to grab the common lock of the BOs in the bulk move section.

Regards,
Christian.

Quoted 332 lines

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++
  drivers/gpu/drm/ttm/ttm_resource.c     | 92 ++++++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_vm.c             |  4 ++
  include/drm/ttm/ttm_resource.h         | 56 ++++++++++------
  4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  	if (r)
  		return r;
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
  	vm->is_compute_context = false;
 
  	vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  error_free_delayed:
  	dma_fence_put(vm->last_tlb_flush);
  	dma_fence_put(vm->last_unlocked);
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
  	amdgpu_vm_fini_entities(vm);
 
  	return r;
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
  		}
  	}
 
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
  }
 
  /**
drivers/gpu/drm/ttm/ttm_resource.c +92 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@
 
  #include <drm/drm_util.h>
 
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct ttm_lru_bulk_move *bulk,
+					       struct ttm_resource_cursor *cursor)
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last->lru.link);
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct ttm_lru_bulk_move *bulk)
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list, bulk_link)
+		ttm_resource_cursor_move_bulk_tail(bulk, cursor);
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct ttm_lru_bulk_move *bulk)
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list, bulk_link)
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+
  /**
   * ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage
   * @cursor: The struct ttm_resource_cursor to finalize.
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor)
  {
  	lockdep_assert_held(&cursor->man->bdev->lru_lock);
  	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
  }
 
  /**
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor)
  void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
  {
  	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
  }
  EXPORT_SYMBOL(ttm_lru_bulk_move_init);
 
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+
  /**
   * ttm_lru_bulk_move_tail - bulk move range of resources to the LRU tail.
   *
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)
  {
  	unsigned i, j;
 
+	ttm_bulk_move_adjust_cursors(bulk);
  	for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
  		for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {
  			struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j];
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
  }
  EXPORT_SYMBOL(ttm_resource_manager_debug);
 
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor *cursor,
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next = ttm_lru_item_to_res(next_lru);
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link, &bulk->cursor_list);
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+
  /**
   * ttm_resource_manager_first() - Start iterating over the resources
   * of a resource manager
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,
  	cursor->priority = 0;
  	cursor->man = man;
  	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);
  	list_add(&cursor->hitch.link, &man->lru[cursor->priority]);
 
  	return ttm_resource_manager_next(cursor);
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
  		lru = &cursor->hitch;
  		list_for_each_entry_continue(lru, &man->lru[cursor->priority], link) {
  			if (ttm_lru_item_is_res(lru)) {
+				ttm_resource_cursor_check_bulk(cursor, lru);
  				list_move(&cursor->hitch.link, &lru->link);
  				return ttm_lru_item_to_res(lru);
  			}
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)
  			break;
 
  		list_move(&cursor->hitch.link, &man->lru[cursor->priority]);
+		ttm_resource_cursor_clear_bulk(cursor);
  	}
 
  	ttm_resource_cursor_fini_locked(cursor);
drivers/gpu/drm/xe/xe_vm.c +4 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 
  	INIT_WORK(&vm->destroy_work, vm_destroy_work_func);
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
  	INIT_LIST_HEAD(&vm->preempt.exec_queues);
  	vm->preempt.min_run_period_ms = 10;	/* FIXME: Wire up to uAPI */
 
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
  	mutex_destroy(&vm->snap_mutex);
  	for_each_tile(tile, xe, id)
  		xe_range_fence_tree_fini(&vm->rftree[id]);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
  	kfree(vm);
  	if (flags & XE_VM_FLAG_LR_MODE)
  		xe_pm_runtime_put(xe);
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct work_struct *w)
  		XE_WARN_ON(vm->pt_root[id]);
 
  	trace_xe_vm_free(vm);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
  	kfree(vm);
  }
 
include/drm/ttm/ttm_resource.h +35 -21
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
  	return container_of(item, struct ttm_resource, lru);
  }
 
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
-
  /**
   * struct ttm_lru_bulk_move_pos
   *
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {
 
  /**
   * struct ttm_lru_bulk_move
- *
   * @pos: first/last lru entry for resources in the each domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock.
   *
   * Container for the current bulk move state. Should be used with
   * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
   */
  struct ttm_lru_bulk_move {
  	struct ttm_lru_bulk_move_pos pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];
+	struct list_head cursor_list;
  };
 
+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange @hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor *cursor);
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+
  /**
   * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
   * struct sg_table backed struct ttm_resource.
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct ttm_resource_manager *man)
 
  void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk);
  void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);
 
  void ttm_resource_add_bulk_move(struct ttm_resource *res,
  				struct ttm_buffer_object *bo);
author maintainer
#8 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

Hi, Christian,

Quoted 24 lines · Christian König

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 11 lines

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

This is the only patch in the series which I see critical.

I think the final goal when using drm_exec in TTMs eviction path is to
keep all evicted (or evicting) BOs locked until we have enough space.

This means that for bulk move sections on the LRU we would lock the first BO and would only drop that lock again if we have gone over the full bulk move section and know that all BOs are not valuable for eviction.

Quoted 7 lines

Because of this the issue of having to consider hitches move with a bulk
move section on the LRU doesn't even occur because for that a concurrent
process would need to grab the common lock of the BOs in the bulk move
section.

While I agree that this is something we should strive towards, following the previous discussion I already reworked this patch completely to remove the dual hitches and make it less fragile. After that you mentioned you were ok with the high level approach for these first four patches here:

https://lists.freedesktop.org/archives/dri-devel/2024-April/450288.html

So is that not any longer the case?

To recap, the concerns I'm seeing with the "kept common lock" approach are

a) Since when we release the LRU lock and the common bulk bo lock is not yet locked, a LRU bump may happen and the hitch will go with it. So to avoid that we need to place the hitch *before* then considered resource in the LRU list rather than *after*. Now on the next iteration we need to come up with some way to choose what's really the next resource? If the next resource pointer is the same we already considered, should we assume it might have been freed and re-alloced with the same virtual address?

b) It will be up to the user of the lru traversal to actually guarantee that locks are held across a bulk part, to make the resource traversal reasonably self-contained. In this case the LRU walker, because there's where the bo locking happens.
This means that any other code that aims to walk the LRUs for various reasons, and doesn't provide any held lock guarantees, may be subject to unexpected results if someone bumped the LRU.
So we would basically tailor the resource iteration here for a single use-case and not make it robust for various use-cases.

So my suggestion is we keep this until we've come up with a bullet- proof way to sort out a) and b) above and then we can rip it out.

/Thomas

Quoted 384 lines

Regards,
Christian.

Quoted 378 lines

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++ drivers/gpu/drm/ttm/ttm_resource.c     | 92
++++++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_vm.c             |  4 ++ include/drm/ttm/ttm_resource.h         | 56 ++++++++++------

  4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
   	if (r)
   		return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+

   vm->is_compute_context = false;

   vm->use_cpu_for_update = !!(adev-

Quoted 1 line

vm_manager.vm_update_mode &

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,

  error_free_delayed:
   dma_fence_put(vm->last_tlb_flush);
   dma_fence_put(vm->last_unlocked);

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
Quoted 1 line

lru_bulk_move);

   amdgpu_vm_fini_entities(vm);

   return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device
 *adev, struct amdgpu_vm *vm)
   		}
   	}

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
Quoted 1 line

lru_bulk_move);

  }

  /**

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@


  #include <drm/drm_util.h>

drivers/gpu/drm/ttm/ttm_resource.c +12 -0
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct

ttm_lru_bulk_move *bulk,

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+					       struct

ttm_resource_cursor *cursor)

drivers/gpu/drm/ttm/ttm_resource.c +13 -0
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last-
Quoted 1 line

lru.link);

drivers/gpu/drm/ttm/ttm_resource.c +5 -0
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct ttm_lru_bulk_move

*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +5 -0
+		ttm_resource_cursor_move_bulk_tail(bulk, cursor);
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct ttm_lru_bulk_move

*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+

  /**
   * ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage
   * @cursor: The struct ttm_resource_cursor to finalize.

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct
 ttm_resource_cursor *cursor)
   {
   	lockdep_assert_held(&cursor->man->bdev->lru_lock);
   	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
   }


  /**

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct
 ttm_resource_cursor *cursor)
   void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
   {
   	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
   }
   EXPORT_SYMBOL(ttm_lru_bulk_move_init);

drivers/gpu/drm/ttm/ttm_resource.c +17 -0
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+

  /**
   * ttm_lru_bulk_move_tail - bulk move range of resources to the LRU tail.
   *

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct
 ttm_lru_bulk_move *bulk)
   {
   	unsigned i, j;

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+	ttm_bulk_move_adjust_cursors(bulk);

   for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
   for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) { struct ttm_lru_bulk_move_pos *pos = &bulk-

Quoted 1 line

pos[i][j];

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct
 ttm_resource_manager *man,
   }
   EXPORT_SYMBOL(ttm_resource_manager_debug);

drivers/gpu/drm/ttm/ttm_resource.c +14 -0
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor *cursor,
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next = ttm_lru_item_to_res(next_lru);
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link, &bulk-
Quoted 1 line

cursor_list);

drivers/gpu/drm/ttm/ttm_resource.c +8 -0
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+

  /**
   * ttm_resource_manager_first() - Start iterating over the resources
   * of a resource manager

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct
 ttm_resource_manager *man,
   	cursor->priority = 0;
   	cursor->man = man;
   	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);

   list_add(&cursor->hitch.link, &man->lru[cursor-

Quoted 1 line

priority]);


   return ttm_resource_manager_next(cursor);

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
   		lru = &cursor->hitch;

   list_for_each_entry_continue(lru, &man-

Quoted 1 line

lru[cursor->priority], link) {

   if (ttm_lru_item_is_res(lru)) {

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+				ttm_resource_cursor_check_bulk(cur

sor, lru);
   list_move(&cursor->hitch.link, &lru->link);
   return ttm_lru_item_to_res(lru); }

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
   			break;


   list_move(&cursor->hitch.link, &man->lru[cursor-

Quoted 1 line

priority]);

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+		ttm_resource_cursor_clear_bulk(cursor);

   }

   ttm_resource_cursor_fini_locked(cursor);

drivers/gpu/drm/xe/xe_vm.c +0 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct xe_device
 *xe, u32 flags)


   INIT_WORK(&vm->destroy_work, vm_destroy_work_func);

drivers/gpu/drm/xe/xe_vm.c +2 -0
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+

   INIT_LIST_HEAD(&vm->preempt.exec_queues);
   vm->preempt.min_run_period_ms = 10; /* FIXME: Wire up
to uAPI */

drivers/gpu/drm/xe/xe_vm.c +1 -0
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct xe_device
 *xe, u32 flags)
   	mutex_destroy(&vm->snap_mutex);
   	for_each_tile(tile, xe, id)
   		xe_range_fence_tree_fini(&vm->rftree[id]);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
   	kfree(vm);
   	if (flags & XE_VM_FLAG_LR_MODE)

   xe_pm_runtime_put(xe);

drivers/gpu/drm/xe/xe_vm.c +0 -0
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct
 work_struct *w)
   		XE_WARN_ON(vm->pt_root[id]);


   trace_xe_vm_free(vm);

drivers/gpu/drm/xe/xe_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);

   kfree(vm);
  }

include/drm/ttm/ttm_resource.h +0 -0
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
   	return container_of(item, struct ttm_resource, lru);
   }

include/drm/ttm/ttm_resource.h +0 -17
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor

*cursor);

include/drm/ttm/ttm_resource.h +0 -3
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
-

  /**
   * struct ttm_lru_bulk_move_pos
   *

include/drm/ttm/ttm_resource.h +0 -0
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {


  /**
   * struct ttm_lru_bulk_move
- *
   * @pos: first/last lru entry for resources in the each domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock. *
   * Container for the current bulk move state. Should be used with
   * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().

include/drm/ttm/ttm_resource.h +0 -0
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
    */
   struct ttm_lru_bulk_move {

   struct ttm_lru_bulk_move_pos
pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];

include/drm/ttm/ttm_resource.h +1 -0
+	struct list_head cursor_list;

  };

+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange @hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */

include/drm/ttm/ttm_resource.h +10 -0
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor

*cursor);

include/drm/ttm/ttm_resource.h +3 -0
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+

  /**
   * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
   * struct sg_table backed struct ttm_resource.

include/drm/ttm/ttm_resource.h +0 -0
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct
 ttm_resource_manager *man)


  void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk); void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);

include/drm/ttm/ttm_resource.h +2 -0
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);


  void ttm_resource_add_bulk_move(struct ttm_resource *res, struct ttm_buffer_object *bo);

maintainer
#9 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

Hey Thomas,

Am 04.07.24 um 14:41 schrieb Thomas Hellström:

Quoted 33 lines

Hi, Christian,

Quoted 23 lines · Christian König

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 11 lines

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

This is the only patch in the series which I see critical.

I think the final goal when using drm_exec in TTMs eviction path is to
keep all evicted (or evicting) BOs locked until we have enough space.

This means that for bulk move sections on the LRU we would lock the first BO and would only drop that lock again if we have gone over the full bulk move section and know that all BOs are not valuable for eviction.

Quoted 7 lines

Because of this the issue of having to consider hitches move with a bulk
move section on the LRU doesn't even occur because for that a concurrent
process would need to grab the common lock of the BOs in the bulk move
section.

Quoted 3 lines

While I agree that this is something we should strive towards, following the previous discussion I already reworked this patch completely to remove the dual hitches and make it less fragile.

Yeah seen that and it indeed makes it much easier to understand what's going on.

Quoted 6 lines

After that you mentioned you were ok with the high level approach for these first four patches here:

https://lists.freedesktop.org/archives/dri-devel/2024-April/450288.html

So is that not any longer the case?

I'm ok with having it as intermediate step, but for that it's a bit much of an hammer.

On the other hand having clean ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() calls is probably something we should keep around anyway.

Quoted 11 lines

To recap, the concerns I'm seeing with the "kept common lock" approach are

a) Since when we release the LRU lock and the common bulk bo lock is not yet locked, a LRU bump may happen and the hitch will go with it. So to avoid that we need to place the hitch *before* then considered resource in the LRU list rather than *after*. Now on the next iteration we need to come up with some way to choose what's really the next resource? If the next resource pointer is the same we already considered, should we assume it might have been freed and re-alloced with the same virtual address?

My idea is for the general flow is this:

1. Grab the lru lock
2. Grab a reference to the BO after the hitch, eventually trylock the BO or just continue with a prelocked one
3. If locking wasn't successfully
    4. Drop the lru lock
    5. Block on the BO lock
    6. Check that this resource/BO is still the one the cursor points to, if not drop the lock and restart from #1
    7. Grab the lru lock
8. Advance the cursor.
9. Drop the lru lock.
10. Try to evict or swap the BO
11. Repeat if still not able to allocate memory.

The BO could be prelocked if it's part of the currently processed bulk or previously contended and prelocked by drm_exec.

And instead of checking if the resource is in the right domain we check if the resource/BO is still the one where the cursor points to.

This way we don't care if the resource was reallocated and by coincident ended up right after the cursor hitch again. As long as we still point to the BO we just locked everything is fine.

Quoted 9 lines

b) It will be up to the user of the lru traversal to actually guarantee that locks are held across a bulk part, to make the resource traversal reasonably self-contained. In this case the LRU walker, because there's where the bo locking happens.
This means that any other code that aims to walk the LRUs for various reasons, and doesn't provide any held lock guarantees, may be subject to unexpected results if someone bumped the LRU.
So we would basically tailor the resource iteration here for a single use-case and not make it robust for various use-cases.

Yeah, that's also going into a direction I was questioning. Do we have use cases for the resource iterator were we don't lock the BO?

If not why don't we integrate all this into the first_resource() and next_resource() functions instead? Obviously with some helpers in the BO code.

Quoted 2 lines

So my suggestion is we keep this until we've come up with a bullet- proof way to sort out a) and b) above and then we can rip it out.

Yeah if we can't make progress otherwise that works for me as well.

Regards,
Christian.

Quoted 395 lines

/Thomas

Quoted 381 lines

Regards,
Christian.

Quoted 377 lines

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++
   drivers/gpu/drm/ttm/ttm_resource.c     | 92
++++++++++++++++++++++++++
   drivers/gpu/drm/xe/xe_vm.c             |  4 ++
   include/drm/ttm/ttm_resource.h         | 56 ++++++++++------
   4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
   	if (r)
   		return r;
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
   	vm->is_compute_context = false;
 

   vm->use_cpu_for_update = !!(adev-

Quoted 1 line

vm_manager.vm_update_mode &

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
   error_free_delayed:
   	dma_fence_put(vm->last_tlb_flush);
   	dma_fence_put(vm->last_unlocked);
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
 > lru_bulk_move);
   	amdgpu_vm_fini_entities(vm);

   return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device
 *adev, struct amdgpu_vm *vm)
   		}
   	}
 
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
 > lru_bulk_move);
   }

  /**

drivers/gpu/drm/ttm/ttm_resource.c +31 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@
 
   #include <drm/drm_util.h>
 
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct
 ttm_lru_bulk_move *bulk,
+					       struct
 ttm_resource_cursor *cursor)
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last-
 > lru.link);
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct ttm_lru_bulk_move

*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +5 -0
+		ttm_resource_cursor_move_bulk_tail(bulk, cursor);
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct ttm_lru_bulk_move

*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk->cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+
   /**
    * ttm_resource_cursor_fini_locked() - Finalize the LRU list

cursor usage
   * @cursor: The struct ttm_resource_cursor to finalize.

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct
 ttm_resource_cursor *cursor)
   {
   	lockdep_assert_held(&cursor->man->bdev->lru_lock);
   	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
   }
 

  /**

drivers/gpu/drm/ttm/ttm_resource.c +18 -0
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct
 ttm_resource_cursor *cursor)
   void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
   {
   	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
   }
   EXPORT_SYMBOL(ttm_lru_bulk_move_init);
 
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+
   /**
    * ttm_lru_bulk_move_tail - bulk move range of resources to the

LRU tail.
   *

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct
 ttm_lru_bulk_move *bulk)
   {
   	unsigned i, j;
 
+	ttm_bulk_move_adjust_cursors(bulk);
   	for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
   		for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {

   struct ttm_lru_bulk_move_pos *pos = &bulk-

Quoted 1 line

pos[i][j];

drivers/gpu/drm/ttm/ttm_resource.c +22 -0
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct
 ttm_resource_manager *man,
   }
   EXPORT_SYMBOL(ttm_resource_manager_debug);
 
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor *cursor,
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next = ttm_lru_item_to_res(next_lru);
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link, &bulk-
 > cursor_list);
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+
   /**

   * ttm_resource_manager_first() - Start iterating over the
resources
   * of a resource manager

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct
 ttm_resource_manager *man,
   	cursor->priority = 0;
   	cursor->man = man;
   	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);
   	list_add(&cursor->hitch.link, &man->lru[cursor-
 > priority]);

   return ttm_resource_manager_next(cursor);

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
   		lru = &cursor->hitch;
   		list_for_each_entry_continue(lru, &man-
 > lru[cursor->priority], link) {
   			if (ttm_lru_item_is_res(lru)) {
+				ttm_resource_cursor_check_bulk(cur
 sor, lru);

   list_move(&cursor->hitch.link,
&lru->link);
   return ttm_lru_item_to_res(lru); }

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
   			break;
 
   		list_move(&cursor->hitch.link, &man->lru[cursor-
 > priority]);
+		ttm_resource_cursor_clear_bulk(cursor);
   	}

   ttm_resource_cursor_fini_locked(cursor);

drivers/gpu/drm/xe/xe_vm.c +2 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct xe_device
 *xe, u32 flags)
 
   	INIT_WORK(&vm->destroy_work, vm_destroy_work_func);
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
   	INIT_LIST_HEAD(&vm->preempt.exec_queues);
   	vm->preempt.min_run_period_ms = 10;	/* FIXME: Wire up

to uAPI */

drivers/gpu/drm/xe/xe_vm.c +1 -0
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct xe_device
 *xe, u32 flags)
   	mutex_destroy(&vm->snap_mutex);
   	for_each_tile(tile, xe, id)
   		xe_range_fence_tree_fini(&vm->rftree[id]);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
   	kfree(vm);
   	if (flags & XE_VM_FLAG_LR_MODE)

   xe_pm_runtime_put(xe);

drivers/gpu/drm/xe/xe_vm.c +1 -0
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct
 work_struct *w)
   		XE_WARN_ON(vm->pt_root[id]);
 
   	trace_xe_vm_free(vm);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
   	kfree(vm);
   }
include/drm/ttm/ttm_resource.h +0 -20
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)
   	return container_of(item, struct ttm_resource, lru);
   }
 
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor
 *cursor);
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
-
   /**
    * struct ttm_lru_bulk_move_pos

   *

include/drm/ttm/ttm_resource.h +0 -1
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {
 
   /**
    * struct ttm_lru_bulk_move
- *
    * @pos: first/last lru entry for resources in the each

domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock. *
   * Container for the current bulk move state. Should be used with
   * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().

include/drm/ttm/ttm_resource.h +9 -0
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
    */
   struct ttm_lru_bulk_move {
   	struct ttm_lru_bulk_move_pos
 pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];
+	struct list_head cursor_list;
   };
 
+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange

@hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */

include/drm/ttm/ttm_resource.h +10 -0
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct ttm_resource_cursor

*cursor);

include/drm/ttm/ttm_resource.h +3 -0
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor *cursor);
+
   /**
    * struct ttm_kmap_iter_iomap - Specialization for a struct

io_mapping +
   * struct sg_table backed struct ttm_resource.

include/drm/ttm/ttm_resource.h +2 -0
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct
 ttm_resource_manager *man)
 
   void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk);
   void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);
 
   void ttm_resource_add_bulk_move(struct ttm_resource *res,

   struct ttm_buffer_object *bo);

author maintainer
#10 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

Quoted 135 lines · Christian König

Hey Thomas,

Am 04.07.24 um 14:41 schrieb Thomas Hellström:

Quoted 38 lines

Hi, Christian,

Quoted 27 lines · Christian König

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 11 lines

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

This is the only patch in the series which I see critical.

I think the final goal when using drm_exec in TTMs eviction path is
to
keep all evicted (or evicting) BOs locked until we have enough space.

This means that for bulk move sections on the LRU we would lock the
first BO and would only drop that lock again if we have gone over the
full bulk move section and know that all BOs are not valuable for eviction.

Quoted 8 lines

Because of this the issue of having to consider hitches move with a
bulk
move section on the LRU doesn't even occur because for that a concurrent
process would need to grab the common lock of the BOs in the bulk move
section.

Quoted 3 lines

While I agree that this is something we should strive towards, following the previous discussion I already reworked this patch completely to remove the dual hitches and make it less fragile.

Yeah seen that and it indeed makes it much easier to understand what's
going on.

Quoted 7 lines

After that you mentioned you were ok with the high level approach for
these first four patches here:

https://lists.freedesktop.org/archives/dri-devel/2024-April/450288.html

So is that not any longer the case?

I'm ok with having it as intermediate step, but for that it's a bit much
of an hammer.

On the other hand having clean ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() calls is probably something we should keep around anyway.

Quoted 4 lines

To recap, the concerns I'm seeing with the "kept common lock" approach
are

Quoted 12 lines

a) Since when we release the LRU lock and the common bulk bo lock is
not yet locked, a LRU bump may happen and the hitch will go with it. So
to avoid that we need to place the hitch *before* then considered resource in the LRU list rather than *after*. Now on the next iteration
we need to come up with some way to choose what's really the next resource? If the next resource pointer is the same we already considered, should we assume it might have been freed and re- alloced
with the same virtual address?

My idea is for the general flow is this:

1. Grab the lru lock
2. Grab a reference to the BO after the hitch, eventually trylock the BO
or just continue with a prelocked one
3. If locking wasn't successfully
     4. Drop the lru lock
     5. Block on the BO lock
     6. Check that this resource/BO is still the one the cursor points
to, if not drop the lock and restart from #1
     7. Grab the lru lock
8. Advance the cursor.
9. Drop the lru lock.
10. Try to evict or swap the BO
11. Repeat if still not able to allocate memory.

The BO could be prelocked if it's part of the currently processed bulk
or previously contended and prelocked by drm_exec.

And instead of checking if the resource is in the right domain we check
if the resource/BO is still the one where the cursor points to.

This way we don't care if the resource was reallocated and by coincident
ended up right after the cursor hitch again. As long as we still point
to the BO we just locked everything is fine.

Quoted 7 lines

b) It will be up to the user of the lru traversal to actually guarantee
that locks are held across a bulk part, to make the resource traversal
reasonably self-contained. In this case the LRU walker, because there's
where the bo locking happens.

Quoted 8 lines

This means that any other code that aims to walk the LRUs for various
reasons, and doesn't provide any held lock guarantees, may be subject
to unexpected results if someone bumped the LRU.
So we would basically tailor the resource iteration here for a single
use-case and not make it robust for various use-cases.

Yeah, that's also going into a direction I was questioning. Do we have
use cases for the resource iterator were we don't lock the BO?

Quoted 4 lines

If not why don't we integrate all this into the first_resource() and next_resource() functions instead? Obviously with some helpers in the BO
code.

That'd be if we moved this out to a drm-level layer like the work Oak started for cross-component eviction targeting SVM.

I guess it's also my desire for keeping components separated as much as possible, but I'm aware others may feel differently about that.

Quoted 5 lines
Quoted 3 lines

So my suggestion is we keep this until we've come up with a bullet- proof way to sort out a) and b) above and then we can rip it out.

Yeah if we can't make progress otherwise that works for me as well.

Then I'd say let's go for this and revisit.

So what are the ARs here?
Making sure we have a clean init and fini is something I've thought of as well.

Related to that, what's your opinion on using DEFINE_CLASS() and scoped_guard() in TTM for automatic cleanup of the iterator when leaving the loop scope?

https://elixir.bootlin.com/linux/v6.10-rc6/source/include/linux/cleanup.h#L168

Thanks,
Thomas

Quoted 430 lines

Regards,
Christian.

Quoted 425 lines

/Thomas

Quoted 411 lines

Regards,
Christian.

Quoted 407 lines

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc: Christian König <[email protected]>
Cc: Somalapuram Amaranath <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++ drivers/gpu/drm/ttm/ttm_resource.c     | 92
++++++++++++++++++++++++++
   drivers/gpu/drm/xe/xe_vm.c             |  4 ++ include/drm/ttm/ttm_resource.h         | 56 ++++++++++------

   4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
    	if (r)
    		return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+

    vm->is_compute_context = false;

    vm->use_cpu_for_update = !!(adev-

Quoted 1 line

vm_manager.vm_update_mode &

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,

   error_free_delayed:
    dma_fence_put(vm->last_tlb_flush);
    dma_fence_put(vm->last_unlocked);

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
Quoted 1 line

lru_bulk_move);

    amdgpu_vm_fini_entities(vm);

    return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0 -0
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device
 *adev, struct amdgpu_vm *vm)
    		}
    	}

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
Quoted 1 line

lru_bulk_move);

   }

   /**

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@


   #include <drm/drm_util.h>

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor

*cursor)

drivers/gpu/drm/ttm/ttm_resource.c +9 -0
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct

ttm_lru_bulk_move *bulk,

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+					       struct

ttm_resource_cursor *cursor)

drivers/gpu/drm/ttm/ttm_resource.c +13 -0
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last-
Quoted 1 line

lru.link);

drivers/gpu/drm/ttm/ttm_resource.c +5 -0
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct

ttm_lru_bulk_move
*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk-
Quoted 1 line

cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+		ttm_resource_cursor_move_bulk_tail(bulk,

cursor);

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct

ttm_lru_bulk_move
*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk-
Quoted 1 line

cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+

   /**
    * ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage
    * @cursor: The struct ttm_resource_cursor to finalize.

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct
 ttm_resource_cursor *cursor)
    {
    	lockdep_assert_held(&cursor->man->bdev->lru_lock);
    	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
    }


   /**

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct
 ttm_resource_cursor *cursor)
    void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
    {
    	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
    }
    EXPORT_SYMBOL(ttm_lru_bulk_move_init);

drivers/gpu/drm/ttm/ttm_resource.c +17 -0
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+

   /**
    * ttm_lru_bulk_move_tail - bulk move range of resources to the
LRU tail.
    *

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct
 ttm_lru_bulk_move *bulk)
    {
    	unsigned i, j;

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+	ttm_bulk_move_adjust_cursors(bulk);

    for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) { for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) { struct ttm_lru_bulk_move_pos *pos = &bulk-

Quoted 1 line

pos[i][j];

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct
 ttm_resource_manager *man,
    }
    EXPORT_SYMBOL(ttm_resource_manager_debug);

drivers/gpu/drm/ttm/ttm_resource.c +2 -0
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor

*cursor,

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next =

ttm_lru_item_to_res(next_lru);

drivers/gpu/drm/ttm/ttm_resource.c +9 -0
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link,

&bulk-

Quoted 1 line

cursor_list);

drivers/gpu/drm/ttm/ttm_resource.c +8 -0
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+

   /**
    * ttm_resource_manager_first() - Start iterating over the resources
    * of a resource manager

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct
 ttm_resource_manager *man,
    	cursor->priority = 0;
    	cursor->man = man;
    	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);

    list_add(&cursor->hitch.link, &man->lru[cursor-

Quoted 1 line

priority]);


    return ttm_resource_manager_next(cursor);

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
    		lru = &cursor->hitch;

    list_for_each_entry_continue(lru, &man-

Quoted 1 line

lru[cursor->priority], link) {

    if (ttm_lru_item_is_res(lru)) {

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+				ttm_resource_cursor_check_bulk

(cur
sor, lru);
    list_move(&cursor->hitch.link, &lru->link);
    return ttm_lru_item_to_res(lru);
    }

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
    			break;


    list_move(&cursor->hitch.link, &man-

Quoted 2 lines

lru[cursor-
priority]);

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+		ttm_resource_cursor_clear_bulk(cursor);

    }

    ttm_resource_cursor_fini_locked(cursor);

drivers/gpu/drm/xe/xe_vm.c +0 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct

xe_device
*xe, u32 flags)

    INIT_WORK(&vm->destroy_work, vm_destroy_work_func);

drivers/gpu/drm/xe/xe_vm.c +2 -0
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+

    INIT_LIST_HEAD(&vm->preempt.exec_queues);
    vm->preempt.min_run_period_ms = 10; /* FIXME: Wire
up
to uAPI */

drivers/gpu/drm/xe/xe_vm.c +0 -0
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct

xe_device
*xe, u32 flags)
    mutex_destroy(&vm->snap_mutex); for_each_tile(tile, xe, id) xe_range_fence_tree_fini(&vm->rftree[id]);

drivers/gpu/drm/xe/xe_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);

    kfree(vm);
    if (flags & XE_VM_FLAG_LR_MODE)
    xe_pm_runtime_put(xe);

drivers/gpu/drm/xe/xe_vm.c +0 -0
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct
 work_struct *w)
    		XE_WARN_ON(vm->pt_root[id]);


    trace_xe_vm_free(vm);

drivers/gpu/drm/xe/xe_vm.c +1 -0
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);

    kfree(vm);
   }

include/drm/ttm/ttm_resource.h +0 -0
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item
 *item)
    	return container_of(item, struct ttm_resource, lru);
    }

include/drm/ttm/ttm_resource.h +0 -17
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct

ttm_resource_cursor
*cursor);

include/drm/ttm/ttm_resource.h +0 -2
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor

*cursor);
-
   /**
    * struct ttm_lru_bulk_move_pos
    *

include/drm/ttm/ttm_resource.h +0 -0
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {


   /**
    * struct ttm_lru_bulk_move
- *
    * @pos: first/last lru entry for resources in the each domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock.
    *
    * Container for the current bulk move state. Should be used with
    * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().

include/drm/ttm/ttm_resource.h +0 -0
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
     */
    struct ttm_lru_bulk_move {

    struct ttm_lru_bulk_move_pos
pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];

include/drm/ttm/ttm_resource.h +1 -0
+	struct list_head cursor_list;

   };

+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange @hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */

include/drm/ttm/ttm_resource.h +10 -0
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct

ttm_resource_cursor
*cursor);

include/drm/ttm/ttm_resource.h +2 -0
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor

*cursor);
+
   /**
    * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
    * struct sg_table backed struct ttm_resource.

include/drm/ttm/ttm_resource.h +0 -0
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct
 ttm_resource_manager *man)


   void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk); void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);

include/drm/ttm/ttm_resource.h +2 -0
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);


   void ttm_resource_add_bulk_move(struct ttm_resource *res, struct ttm_buffer_object *bo);

maintainer
#11 lore raw

Re: [PATCH v6 04/12] drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves

Am 04.07.24 um 15:53 schrieb Thomas Hellström:

Quoted 137 lines · Christian König
Quoted 126 lines

Hey Thomas,

Am 04.07.24 um 14:41 schrieb Thomas Hellström:

Quoted 38 lines

Hi, Christian,

Quoted 27 lines · Christian König

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 11 lines

To address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.

Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.

This is the only patch in the series which I see critical.

I think the final goal when using drm_exec in TTMs eviction path is
to
keep all evicted (or evicting) BOs locked until we have enough space.

This means that for bulk move sections on the LRU we would lock the
first BO and would only drop that lock again if we have gone over the
full bulk move section and know that all BOs are not valuable for eviction.

Quoted 8 lines

Because of this the issue of having to consider hitches move with a
bulk
move section on the LRU doesn't even occur because for that a concurrent
process would need to grab the common lock of the BOs in the bulk move
section.

Quoted 3 lines

While I agree that this is something we should strive towards, following the previous discussion I already reworked this patch completely to remove the dual hitches and make it less fragile.

Yeah seen that and it indeed makes it much easier to understand what's
going on.

Quoted 7 lines

After that you mentioned you were ok with the high level approach for
these first four patches here:

https://lists.freedesktop.org/archives/dri-devel/2024-April/450288.html

So is that not any longer the case?

I'm ok with having it as intermediate step, but for that it's a bit much
of an hammer.

On the other hand having clean ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() calls is probably something we should keep around anyway.

Quoted 4 lines

To recap, the concerns I'm seeing with the "kept common lock" approach
are

Quoted 12 lines

a) Since when we release the LRU lock and the common bulk bo lock is
not yet locked, a LRU bump may happen and the hitch will go with it. So
to avoid that we need to place the hitch *before* then considered resource in the LRU list rather than *after*. Now on the next iteration
we need to come up with some way to choose what's really the next resource? If the next resource pointer is the same we already considered, should we assume it might have been freed and re- alloced
with the same virtual address?

My idea is for the general flow is this:

1. Grab the lru lock
2. Grab a reference to the BO after the hitch, eventually trylock the BO
or just continue with a prelocked one
3. If locking wasn't successfully
     4. Drop the lru lock
     5. Block on the BO lock
     6. Check that this resource/BO is still the one the cursor
points
to, if not drop the lock and restart from #1
     7. Grab the lru lock
8. Advance the cursor.
9. Drop the lru lock.
10. Try to evict or swap the BO
11. Repeat if still not able to allocate memory.

The BO could be prelocked if it's part of the currently processed bulk
or previously contended and prelocked by drm_exec.

And instead of checking if the resource is in the right domain we check
if the resource/BO is still the one where the cursor points to.

This way we don't care if the resource was reallocated and by coincident
ended up right after the cursor hitch again. As long as we still point
to the BO we just locked everything is fine.

Quoted 7 lines

b) It will be up to the user of the lru traversal to actually guarantee
that locks are held across a bulk part, to make the resource traversal
reasonably self-contained. In this case the LRU walker, because there's
where the bo locking happens.

Quoted 8 lines

This means that any other code that aims to walk the LRUs for various
reasons, and doesn't provide any held lock guarantees, may be subject
to unexpected results if someone bumped the LRU.
So we would basically tailor the resource iteration here for a single
use-case and not make it robust for various use-cases.

Quoted 7 lines

Yeah, that's also going into a direction I was questioning. Do we have
use cases for the resource iterator were we don't lock the BO? If not why don't we integrate all this into the first_resource() and next_resource() functions instead? Obviously with some helpers in the BO
code.

That'd be if we moved this out to a drm-level layer like the work Oak started for cross-component eviction targeting SVM.

Quoted 7 lines

I guess it's also my desire for keeping components separated as much as possible, but I'm aware others may feel differently about that.

Quoted 3 lines
Quoted 2 lines

So my suggestion is we keep this until we've come up with a bullet- proof way to sort out a) and b) above and then we can rip it out.

Yeah if we can't make progress otherwise that works for me as well.

Then I'd say let's go for this and revisit.

Ok in that case feel free to add my Acked-by to this patch.

Quoted 8 lines

So what are the ARs here?
Making sure we have a clean init and fini is something I've thought of as well.

Related to that, what's your opinion on using DEFINE_CLASS() and scoped_guard() in TTM for automatic cleanup of the iterator when leaving the loop scope?

Good question, I was wondering about that as well.

On the one hand it makes error handling easier on the other have I worked with scope based error handling in both C and C++ before and there are always down sides as well.

Not sure if that's really a good idea or not.

Regards,
Christian.

Quoted 433 lines

https://elixir.bootlin.com/linux/v6.10-rc6/source/include/linux/cleanup.h#L168

Thanks,
Thomas

Quoted 427 lines

Regards,
Christian.

Quoted 424 lines

/Thomas

Quoted 411 lines

Regards,
Christian.

Quoted 407 lines

Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2. v6:
- Add some lockdep asserts (Matthew Brost)
- Avoid NULL pointer dereference (Matthew Brost)
- No need to check bo->resource before dereferencing bo->bulk_move (Matthew Brost)

Cc: Christian König <[email protected]>
Cc: Somalapuram Amaranath <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>

---
    drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |  4 ++
    drivers/gpu/drm/ttm/ttm_resource.c     | 92
++++++++++++++++++++++++++
    drivers/gpu/drm/xe/xe_vm.c             |  4 ++
    include/drm/ttm/ttm_resource.h         | 56 ++++++++++------
    4 files changed, 135 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -0
index 3abfa66d72a2..97743993d711 100644
@@ -2420,6 +2420,8 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
    	if (r)
    		return r;
 
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
    	vm->is_compute_context = false;
 

    vm->use_cpu_for_update = !!(adev-

Quoted 1 line

vm_manager.vm_update_mode &

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
@@ -2484,6 +2486,7 @@ int amdgpu_vm_init(struct amdgpu_device
 *adev, struct amdgpu_vm *vm,
    error_free_delayed:
    	dma_fence_put(vm->last_tlb_flush);
    	dma_fence_put(vm->last_unlocked);
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
 > lru_bulk_move);
    	amdgpu_vm_fini_entities(vm);

    return r;

drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -0
@@ -2640,6 +2643,7 @@ void amdgpu_vm_fini(struct amdgpu_device
 *adev, struct amdgpu_vm *vm)
    		}
    	}
 
+	ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm-
 > lru_bulk_move);
    }

   /**

drivers/gpu/drm/ttm/ttm_resource.c +26 -0
index 9c8b6499edfb..b6a2daac5518 100644
@@ -33,6 +33,53 @@
 
    #include <drm/drm_util.h>
 
+/* Detach the cursor from the bulk move list*/
+static void
+ttm_resource_cursor_clear_bulk(struct ttm_resource_cursor
 *cursor)
+{
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	cursor->bulk = NULL;
+	list_del_init(&cursor->bulk_link);
+}
+
+/* Move the cursor to the end of the bulk move list it's in */
+static void ttm_resource_cursor_move_bulk_tail(struct
 ttm_lru_bulk_move *bulk,
+					       struct
 ttm_resource_cursor *cursor)
+{
+	struct ttm_lru_bulk_move_pos *pos;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+
+	if (WARN_ON_ONCE(bulk != cursor->bulk)) {
+		list_del_init(&cursor->bulk_link);
+		return;
+	}
+
+	pos = &bulk->pos[cursor->mem_type][cursor->priority];
+	if (pos->last)
+		list_move(&cursor->hitch.link, &pos->last-
Quoted 1 line

lru.link);

drivers/gpu/drm/ttm/ttm_resource.c +5 -0
+	ttm_resource_cursor_clear_bulk(cursor);
+}
+
+/* Move all cursors attached to a bulk move to its end */
+static void ttm_bulk_move_adjust_cursors(struct

ttm_lru_bulk_move
*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk-
Quoted 1 line

cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+		ttm_resource_cursor_move_bulk_tail(bulk,

cursor);

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+}
+
+/* Remove a cursor from an empty bulk move list */
+static void ttm_bulk_move_drop_cursors(struct

ttm_lru_bulk_move
*bulk)

drivers/gpu/drm/ttm/ttm_resource.c +4 -0
+{
+	struct ttm_resource_cursor *cursor, *next;
+
+	list_for_each_entry_safe(cursor, next, &bulk-
Quoted 1 line

cursor_list,

bulk_link)

drivers/gpu/drm/ttm/ttm_resource.c +3 -0
+		ttm_resource_cursor_clear_bulk(cursor);
+}
+
    /**
     * ttm_resource_cursor_fini_locked() - Finalize the LRU list

cursor usage
    * @cursor: The struct ttm_resource_cursor to finalize.

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -45,6 +92,7 @@ void ttm_resource_cursor_fini_locked(struct
 ttm_resource_cursor *cursor)
    {
    	lockdep_assert_held(&cursor->man->bdev->lru_lock);
    	list_del_init(&cursor->hitch.link);
+	ttm_resource_cursor_clear_bulk(cursor);
    }
 

   /**

drivers/gpu/drm/ttm/ttm_resource.c +18 -0
@@ -73,9 +121,27 @@ void ttm_resource_cursor_fini(struct
 ttm_resource_cursor *cursor)
    void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk)
    {
    	memset(bulk, 0, sizeof(*bulk));
+	INIT_LIST_HEAD(&bulk->cursor_list);
    }
    EXPORT_SYMBOL(ttm_lru_bulk_move_init);
 
+/**
+ * ttm_lru_bulk_move_fini - finalize a bulk move structure
+ * @bdev: The struct ttm_device
+ * @bulk: the structure to finalize
+ *
+ * Sanity checks that bulk moves don't have any
+ * resources left and hence no cursors attached.
+ */
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk)
+{
+	spin_lock(&bdev->lru_lock);
+	ttm_bulk_move_drop_cursors(bulk);
+	spin_unlock(&bdev->lru_lock);
+}
+EXPORT_SYMBOL(ttm_lru_bulk_move_fini);
+
    /**
     * ttm_lru_bulk_move_tail - bulk move range of resources to

the
LRU tail.
    *

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -88,6 +154,7 @@ void ttm_lru_bulk_move_tail(struct
 ttm_lru_bulk_move *bulk)
    {
    	unsigned i, j;
 
+	ttm_bulk_move_adjust_cursors(bulk);
    	for (i = 0; i < TTM_NUM_MEM_TYPES; ++i) {
    		for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) {

    struct ttm_lru_bulk_move_pos *pos =
&bulk-

Quoted 1 line

pos[i][j];

drivers/gpu/drm/ttm/ttm_resource.c +14 -0
@@ -515,6 +582,28 @@ void ttm_resource_manager_debug(struct
 ttm_resource_manager *man,
    }
    EXPORT_SYMBOL(ttm_resource_manager_debug);
 
+static void
+ttm_resource_cursor_check_bulk(struct ttm_resource_cursor
 *cursor,
+			       struct ttm_lru_item *next_lru)
+{
+	struct ttm_resource *next =
 ttm_lru_item_to_res(next_lru);
+	struct ttm_lru_bulk_move *bulk = NULL;
+	struct ttm_buffer_object *bo = next->bo;
+
+	lockdep_assert_held(&cursor->man->bdev->lru_lock);
+	bulk = bo->bulk_move;
+
+	if (cursor->bulk != bulk) {
+		if (bulk) {
+			list_move_tail(&cursor->bulk_link,

&bulk-

Quoted 1 line

cursor_list);

drivers/gpu/drm/ttm/ttm_resource.c +8 -0
+			cursor->mem_type = next->mem_type;
+		} else {
+			list_del_init(&cursor->bulk_link);
+		}
+		cursor->bulk = bulk;
+	}
+}
+
    /**
     * ttm_resource_manager_first() - Start iterating over the

resources
    * of a resource manager

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -535,6 +624,7 @@ ttm_resource_manager_first(struct
 ttm_resource_manager *man,
    	cursor->priority = 0;
    	cursor->man = man;
    	ttm_lru_item_init(&cursor->hitch, TTM_LRU_HITCH);
+	INIT_LIST_HEAD(&cursor->bulk_link);
    	list_add(&cursor->hitch.link, &man->lru[cursor-
 > priority]);

    return ttm_resource_manager_next(cursor);

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
@@ -559,6 +649,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
    		lru = &cursor->hitch;
    		list_for_each_entry_continue(lru, &man-
 > lru[cursor->priority], link) {
    			if (ttm_lru_item_is_res(lru)) {
+				ttm_resource_cursor_check_bulk

(cur
sor, lru);
    list_move(&cursor->hitch.link,
&lru->link);
    return
ttm_lru_item_to_res(lru);
    }

drivers/gpu/drm/ttm/ttm_resource.c +0 -0
@@ -568,6 +659,7 @@ ttm_resource_manager_next(struct
 ttm_resource_cursor *cursor)
    			break;
 
    		list_move(&cursor->hitch.link, &man-
Quoted 2 lines

lru[cursor-
priority]);

drivers/gpu/drm/ttm/ttm_resource.c +1 -0
+		ttm_resource_cursor_clear_bulk(cursor);
    	}
 
    	ttm_resource_cursor_fini_locked(cursor);
drivers/gpu/drm/xe/xe_vm.c +0 -0
index 5b166fa03684..0c7e327bc9a2 100644
@@ -1335,6 +1335,8 @@ struct xe_vm *xe_vm_create(struct

xe_device
*xe, u32 flags)

    INIT_WORK(&vm->destroy_work, vm_destroy_work_func);

drivers/gpu/drm/xe/xe_vm.c +2 -0
+	ttm_lru_bulk_move_init(&vm->lru_bulk_move);
+
    	INIT_LIST_HEAD(&vm->preempt.exec_queues);
    	vm->preempt.min_run_period_ms = 10;	/* FIXME: Wire

up
to uAPI */

drivers/gpu/drm/xe/xe_vm.c +0 -0
@@ -1458,6 +1460,7 @@ struct xe_vm *xe_vm_create(struct

xe_device
*xe, u32 flags)
    mutex_destroy(&vm->snap_mutex); for_each_tile(tile, xe, id) xe_range_fence_tree_fini(&vm->rftree[id]);

drivers/gpu/drm/xe/xe_vm.c +2 -0
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
    	kfree(vm);
    	if (flags & XE_VM_FLAG_LR_MODE)
    		xe_pm_runtime_put(xe);
@@ -1601,6 +1604,7 @@ static void vm_destroy_work_func(struct
 work_struct *w)
    		XE_WARN_ON(vm->pt_root[id]);
 
    	trace_xe_vm_free(vm);
+	ttm_lru_bulk_move_fini(&xe->ttm, &vm->lru_bulk_move);
    	kfree(vm);
    }
include/drm/ttm/ttm_resource.h +0 -17
index 8fac781f641e..571abb4861a6 100644
@@ -269,26 +269,6 @@ ttm_lru_item_to_res(struct ttm_lru_item
 *item)
    	return container_of(item, struct ttm_resource, lru);
    }
 
-/**
- * struct ttm_resource_cursor
- *
- * @man: The resource manager currently being iterated over.
- * @hitch: A hitch list node inserted before the next resource
- * to iterate over.
- * @priority: the current priority
- *
- * Cursor to iterate over the resources in a manager.
- */
-struct ttm_resource_cursor {
-	struct ttm_resource_manager *man;
-	struct ttm_lru_item hitch;
-	unsigned int priority;
-};
-
-void ttm_resource_cursor_fini_locked(struct

ttm_resource_cursor
*cursor);

include/drm/ttm/ttm_resource.h +0 -2
-
-void ttm_resource_cursor_fini(struct ttm_resource_cursor

*cursor);

include/drm/ttm/ttm_resource.h +0 -2
-
    /**
     * struct ttm_lru_bulk_move_pos
     *
@@ -304,8 +284,9 @@ struct ttm_lru_bulk_move_pos {
 
    /**
     * struct ttm_lru_bulk_move
- *
     * @pos: first/last lru entry for resources in the each

domain/priority
+ * @cursor_list: The list of cursors currently traversing any of
+ * the sublists of @pos. Protected by the ttm device's lru_lock.
    *
    * Container for the current bulk move state. Should be used
with
    * ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move().

include/drm/ttm/ttm_resource.h +7 -0
@@ -315,8 +296,39 @@ struct ttm_lru_bulk_move_pos {
     */
    struct ttm_lru_bulk_move {
    	struct ttm_lru_bulk_move_pos
 pos[TTM_NUM_MEM_TYPES][TTM_MAX_BO_PRIORITY];
+	struct list_head cursor_list;
    };
 
+/**
+ * struct ttm_resource_cursor
+ * @man: The resource manager currently being iterated over
+ * @hitch: A hitch list node inserted before the next resource
+ * to iterate over.
+ * @bulk_link: A list link for the list of cursors traversing

the
+ * bulk sublist of @bulk. Protected by the ttm device's lru_lock.
+ * @bulk: Pointer to struct ttm_lru_bulk_move whose subrange @hitch is
+ * inserted to. NULL if none. Never dereference this pointer since
+ * the struct ttm_lru_bulk_move object pointed to might have been
+ * freed. The pointer is only for comparison.
+ * @mem_type: The memory type of the LRU list being traversed.
+ * This field is valid iff @bulk != NULL.
+ * @priority: the current priority
+ *
+ * Cursor to iterate over the resources in a manager.
+ */

include/drm/ttm/ttm_resource.h +10 -0
+struct ttm_resource_cursor {
+	struct ttm_resource_manager *man;
+	struct ttm_lru_item hitch;
+	struct list_head bulk_link;
+	struct ttm_lru_bulk_move *bulk;
+	unsigned int mem_type;
+	unsigned int priority;
+};
+
+void ttm_resource_cursor_fini_locked(struct

ttm_resource_cursor
*cursor);

include/drm/ttm/ttm_resource.h +2 -0
+
+void ttm_resource_cursor_fini(struct ttm_resource_cursor

*cursor);
+
   /**
    * struct ttm_kmap_iter_iomap - Specialization for a struct
io_mapping +
    * struct sg_table backed struct ttm_resource.

include/drm/ttm/ttm_resource.h +2 -0
@@ -405,6 +417,8 @@ ttm_resource_manager_cleanup(struct
 ttm_resource_manager *man)
 
    void ttm_lru_bulk_move_init(struct ttm_lru_bulk_move *bulk);
    void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk);
+void ttm_lru_bulk_move_fini(struct ttm_device *bdev,
+			    struct ttm_lru_bulk_move *bulk);
 
    void ttm_resource_add_bulk_move(struct ttm_resource *res,

    struct ttm_buffer_object *bo);

author maintainer
#12 lore raw

PATCH v6 5/12 drm/ttm: Provide a generic LRU walker helper

In reply to Thomas Hellström

Provide a generic LRU walker in TTM, in the spirit of drm_gem_lru_scan() but building on the restartable TTM LRU functionality.

The LRU walker optionally supports locking objects as part of a ww mutex locking transaction, to mimic to some extent the current functionality in ttm. However any -EDEADLK return is converted to -ENOSPC and then to -ENOMEM before reaching the driver, so that the driver will need to backoff and possibly retry without being able to keep the ticket.

v3:
- Move the helper to core ttm.
- Remove the drm_exec usage from it for now, it will be reintroduced later in the series.
v4:
- Handle the -EALREADY case if ticketlocking.
v6:
- Some cleanup and added code comments (Matthew Brost)
- Clarified the ticketlock in the commit message (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo_util.c | 156 ++++++++++++++++++++++++++++++
 include/drm/ttm/ttm_bo.h          |  35 +++++++
 2 files changed, 191 insertions(+)
drivers/gpu/drm/ttm/ttm_bo_util.c +156 -0
index 0b3f4267130c..c4f678f30fc2 100644
@@ -768,3 +768,159 @@ int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
 	ttm_tt_destroy(bo->bdev, ttm);
 	return ret;
 }
+
+static bool ttm_lru_walk_trylock(struct ttm_lru_walk *walk,
+				 struct ttm_buffer_object *bo,
+				 bool *needs_unlock)
+{
+	struct ttm_operation_ctx *ctx = walk->ctx;
+
+	*needs_unlock = false;
+
+	if (dma_resv_trylock(bo->base.resv)) {
+		*needs_unlock = true;
+		return true;
+	}
+
+	if (bo->base.resv == ctx->resv && ctx->allow_res_evict) {
+		dma_resv_assert_held(bo->base.resv);
+		return true;
+	}
+
+	return false;
+}
+
+static int ttm_lru_walk_ticketlock(struct ttm_lru_walk *walk,
+				   struct ttm_buffer_object *bo,
+				   bool *needs_unlock)
+{
+	struct dma_resv *resv = bo->base.resv;
+	int ret;
+
+	if (walk->ctx->interruptible)
+		ret = dma_resv_lock_interruptible(resv, walk->ticket);
+	else
+		ret = dma_resv_lock(resv, walk->ticket);
+
+	if (!ret) {
+		*needs_unlock = true;
+		/*
+		 * Only a single ticketlock per loop. Ticketlocks are prone
+		 * to return -EDEADLK causing the eviction to fail, so
+		 * after waiting for the ticketlock, revert back to
+		 * trylocking for this walk.
+		 */
+		walk->ticket = NULL;
+	} else if (ret == -EDEADLK) {
+		/* Caller needs to exit the ww transaction. */
+		ret = -ENOSPC;
+	}
+
+	return ret;
+}
+
+static void ttm_lru_walk_unlock(struct ttm_buffer_object *bo, bool locked)
+{
+	if (locked)
+		dma_resv_unlock(bo->base.resv);
+}
+
+/**
+ * ttm_lru_walk_for_evict() - Perform a LRU list walk, with actions taken on
+ * valid items.
+ * @walk: describe the walks and actions taken
+ * @bdev: The TTM device.
+ * @man: The struct ttm_resource manager whose LRU lists we're walking.
+ * @target: The end condition for the walk.
+ *
+ * The LRU lists of @man are walk, and for each struct ttm_resource encountered,
+ * the corresponding ttm_buffer_object is locked and taken a reference on, and
+ * the LRU lock is dropped. the LRU lock may be dropped before locking and, in
+ * that case, it's verified that the item actually remains on the LRU list after
+ * the lock, and that the buffer object didn't switch resource in between.
+ *
+ * With a locked object, the actions indicated by @walk->process_bo are
+ * performed, and after that, the bo is unlocked, the refcount dropped and the
+ * next struct ttm_resource is processed. Here, the walker relies on
+ * TTM's restartable LRU list implementation.
+ *
+ * Typically @walk->process_bo() would return the number of pages evicted,
+ * swapped or shrunken, so that when the total exceeds @target, or when the
+ * LRU list has been walked in full, iteration is terminated. It's also terminated
+ * on error. Note that the definition of @target is done by the caller, it
+ * could have a different meaning than the number of pages.
+ *
+ * Note that the way dma_resv individualization is done, locking needs to be done
+ * either with the LRU lock held (trylocking only) or with a reference on the
+ * object.
+ *
+ * Return: The progress made towards target or negative error code on error.
+ */
+long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
+			    struct ttm_resource_manager *man, long target)
+{
+	struct ttm_resource_cursor cursor;
+	struct ttm_resource *res;
+	long progress = 0;
+	long lret;
+
+	spin_lock(&bdev->lru_lock);
+	ttm_resource_manager_for_each_res(man, &cursor, res) {
+		struct ttm_buffer_object *bo = res->bo;
+		bool bo_needs_unlock = false;
+		bool bo_locked = false;
+		int mem_type;
+
+		if (!bo || bo->resource != res)
+			continue;
+
+		/*
+		 * Attempt a trylock before taking a reference on the bo,
+		 * since if we do it the other way around, and the trylock fails,
+		 * we need to drop the lru lock to put the bo.
+		 */
+
+		if (ttm_lru_walk_trylock(walk, bo, &bo_needs_unlock))
+			bo_locked = true;
+		else if (!walk->ticket || walk->ctx->no_wait_gpu ||
+			 walk->trylock_only)
+			continue;
+
+		if (!ttm_bo_get_unless_zero(bo)) {
+			ttm_lru_walk_unlock(bo, bo_needs_unlock);
+			continue;
+		}
+
+		mem_type = res->mem_type;
+		spin_unlock(&bdev->lru_lock);
+
+		lret = 0;
+		if (!bo_locked)
+			lret = ttm_lru_walk_ticketlock(walk, bo, &bo_needs_unlock);
+
+		/*
+		 * Note that in between the release of the lru lock and the
+		 * ticketlock, the bo may have switched resource,
+		 * and also memory type, since the resource may have been
+		 * freed and allocated again with a different memory type.
+		 * In that case, just skip it.
+		 */
+		if (!lret && bo->resource == res && res->mem_type == mem_type)
+			lret = walk->ops->process_bo(walk, bo);
+
+		ttm_lru_walk_unlock(bo, bo_needs_unlock);
+		ttm_bo_put(bo);
+		if (lret == -EBUSY || lret == -EALREADY)
+			lret = 0;
+		progress = (lret < 0) ? lret : progress + lret;
+
+		cond_resched();
+		spin_lock(&bdev->lru_lock);
+		if (progress < 0 || progress >= target)
+			break;
+	}
+	ttm_resource_cursor_fini_locked(&cursor);
+	spin_unlock(&bdev->lru_lock);
+
+	return progress;
+}
include/drm/ttm/ttm_bo.h +35 -0
index ef0f52f56ebc..10bff3aecd5c 100644
@@ -194,6 +194,41 @@ struct ttm_operation_ctx {
 	uint64_t bytes_moved;
 };
 
+struct ttm_lru_walk;
+
+/** struct ttm_lru_walk_ops - Operations for a LRU walk. */
+struct ttm_lru_walk_ops {
+	/**
+	 * process_bo - Process this bo.
+	 * @walk: struct ttm_lru_walk describing the walk.
+	 * @bo: A locked and referenced buffer object.
+	 *
+	 * Return: Negative error code on error, User-defined positive value
+	 * (typically, but not always number of processed pages) on success.
+	 * On success, the returned values are summed by the walk and the
+	 * walk exits when its garget is met.
+	 * 0 also indicates success, -EBUSY means this bo was skipped.
+	 */
+	long (*process_bo)(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo);
+};
+
+/**
+ * struct ttm_lru_walk - Structure describing a LRU walk.
+ */
+struct ttm_lru_walk {
+	/** @ops: Pointer to the ops structure. */
+	const struct ttm_lru_walk_ops *ops;
+	/** @ctx: Pointer to the struct ttm_operation_ctx. */
+	struct ttm_operation_ctx *ctx;
+	/** @ticket: The struct ww_acquire_ctx if any. */
+	struct ww_acquire_ctx *ticket;
+	/** @tryock_only: Only use trylock for locking. */
+	bool trylock_only;
+};
+
+long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
+			    struct ttm_resource_manager *man, long target);
+
 /**
  * ttm_bo_get - reference a struct ttm_buffer_object
  *
Signature · 1 line
2.44.0
author maintainer
#13 lore raw

PATCH v6 6/12 drm/ttm: Use the LRU walker helper for swapping

In reply to Thomas Hellström

Rework the TTM swapping to use the LRU walker helper. This helps fixing up the ttm_bo_swapout() interface to be consistent about not requiring any locking.

For now mimic the current behaviour of using trylock only. We could be using ticket-locks here but defer that until it's deemed necessary. The TTM swapout functionality is a bit weird anyway since it alternates between memory types without exhausting TTM_PL_SYSTEM first.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

v6:
- Improve on error code translation in the swapout callback (Matthew Brost).

---
 drivers/gpu/drm/ttm/ttm_bo.c     | 111 ++++++++++++++++++++-----------
 drivers/gpu/drm/ttm/ttm_device.c |  30 ++-------
 include/drm/ttm/ttm_bo.h         |   5 +-
 3 files changed, 82 insertions(+), 64 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +74 -37
index 43eda720657f..1053cdca131e 100644
@@ -1118,11 +1118,23 @@ int ttm_bo_wait_ctx(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx)
 }
 EXPORT_SYMBOL(ttm_bo_wait_ctx);
 
-int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
-		   gfp_t gfp_flags)
+/**
+ * struct ttm_bo_swapout_walk - Parameters for the swapout walk
+ */
+struct ttm_bo_swapout_walk {
+	/** @walk: The walk base parameters. */
+	struct ttm_lru_walk walk;
+	/** @gfp_flags: The gfp flags to use for ttm_tt_swapout() */
+	gfp_t gfp_flags;
+};
+
+static long
+ttm_bo_swapout_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
 {
-	struct ttm_place place;
-	bool locked;
+	struct ttm_place place = {.mem_type = bo->resource->mem_type};
+	struct ttm_bo_swapout_walk *swapout_walk =
+		container_of(walk, typeof(*swapout_walk), walk);
+	struct ttm_operation_ctx *ctx = walk->ctx;
 	long ret;
 
 	/*
@@ -1131,28 +1143,29 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 	 * The driver may use the fact that we're moving from SYSTEM
 	 * as an indication that we're about to swap out.
 	 */
-	memset(&place, 0, sizeof(place));
-	place.mem_type = bo->resource->mem_type;
-	if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
-		return -EBUSY;
+	if (!bo->bdev->funcs->eviction_valuable(bo, &place)) {
+		ret = -EBUSY;
+		goto out;
+	}
 
 	if (!bo->ttm || !ttm_tt_is_populated(bo->ttm) ||
 	    bo->ttm->page_flags & TTM_TT_FLAG_EXTERNAL ||
-	    bo->ttm->page_flags & TTM_TT_FLAG_SWAPPED ||
-	    !ttm_bo_get_unless_zero(bo)) {
-		if (locked)
-			dma_resv_unlock(bo->base.resv);
-		return -EBUSY;
+	    bo->ttm->page_flags & TTM_TT_FLAG_SWAPPED) {
+		ret = -EBUSY;
+		goto out;
 	}
 
 	if (bo->deleted) {
-		ret = ttm_bo_cleanup_refs(bo, false, false, locked);
-		ttm_bo_put(bo);
-		return ret == -EBUSY ? -ENOSPC : ret;
-	}
+		pgoff_t num_pages = bo->ttm->num_pages;
 
-	/* TODO: Cleanup the locking */
-	spin_unlock(&bo->bdev->lru_lock);
+		ret = ttm_bo_wait_ctx(bo, ctx);
+		if (ret)
+			goto out;
+
+		ttm_bo_cleanup_memtype_use(bo);
+		ret = num_pages;
+		goto out;
+	}
 
 	/*
 	 * Move to system cached
@@ -1164,12 +1177,13 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 		memset(&hop, 0, sizeof(hop));
 		place.mem_type = TTM_PL_SYSTEM;
 		ret = ttm_resource_alloc(bo, &place, &evict_mem);
-		if (unlikely(ret))
+		if (ret)
 			goto out;
 
 		ret = ttm_bo_handle_move_mem(bo, evict_mem, true, ctx, &hop);
-		if (unlikely(ret != 0)) {
-			WARN(ret == -EMULTIHOP, "Unexpected multihop in swaput - likely driver bug.\n");
+		if (ret) {
+			WARN(ret == -EMULTIHOP,
+			     "Unexpected multihop in swapout - likely driver bug.\n");
 			ttm_resource_free(bo, &evict_mem);
 			goto out;
 		}
@@ -1179,30 +1193,53 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 	 * Make sure BO is idle.
 	 */
 	ret = ttm_bo_wait_ctx(bo, ctx);
-	if (unlikely(ret != 0))
+	if (ret)
 		goto out;
 
 	ttm_bo_unmap_virtual(bo);
-
-	/*
-	 * Swap out. Buffer will be swapped in again as soon as
-	 * anyone tries to access a ttm page.
-	 */
 	if (bo->bdev->funcs->swap_notify)
 		bo->bdev->funcs->swap_notify(bo);
 
 	if (ttm_tt_is_populated(bo->ttm))
-		ret = ttm_tt_swapout(bo->bdev, bo->ttm, gfp_flags);
+		ret = ttm_tt_swapout(bo->bdev, bo->ttm, swapout_walk->gfp_flags);
 out:
+	/* Consider -ENOMEM and -ENOSPC non-fatal. */
+	if (ret == -ENOMEM || ret == -ENOSPC)
+		ret = -EBUSY;
 
-	/*
-	 * Unreserve without putting on LRU to avoid swapping out an
-	 * already swapped buffer.
-	 */
-	if (locked)
-		dma_resv_unlock(bo->base.resv);
-	ttm_bo_put(bo);
-	return ret == -EBUSY ? -ENOSPC : ret;
+	return ret;
+}
+
+const struct ttm_lru_walk_ops ttm_swap_ops = {
+	.process_bo = ttm_bo_swapout_cb,
+};
+
+/**
+ * ttm_bo_swapout() - Swap out buffer objects on the LRU list to shmem.
+ * @bdev: The ttm device.
+ * @ctx: The ttm_operation_ctx governing the swapout operation.
+ * @man: The resource manager whose resources / buffer objects are
+ * goint to be swapped out.
+ * @gfp_flags: The gfp flags used for shmem page allocations.
+ * @target: The desired number of pages to swap out.
+ *
+ * Return: The number of pages actually swapped out, or negative error code
+ * on error.
+ */
+long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
+		    struct ttm_resource_manager *man, gfp_t gfp_flags,
+		    pgoff_t target)
+{
+	struct ttm_bo_swapout_walk swapout_walk = {
+		.walk = {
+			.ops = &ttm_swap_ops,
+			.ctx = ctx,
+			.trylock_only = true,
+		},
+		.gfp_flags = gfp_flags,
+	};
+
+	return ttm_lru_walk_for_evict(&swapout_walk.walk, bdev, man, target);
 }
 
 void ttm_bo_tt_destroy(struct ttm_buffer_object *bo)
drivers/gpu/drm/ttm/ttm_device.c +5 -25
index f9e9b1ec8c8a..ee575d8a54c0 100644
@@ -148,40 +148,20 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags)
 int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
 		       gfp_t gfp_flags)
 {
-	struct ttm_resource_cursor cursor;
 	struct ttm_resource_manager *man;
-	struct ttm_resource *res;
 	unsigned i;
-	int ret;
+	long lret;
 
-	spin_lock(&bdev->lru_lock);
 	for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
 		man = ttm_manager_type(bdev, i);
 		if (!man || !man->use_tt)
 			continue;
 
-		ttm_resource_manager_for_each_res(man, &cursor, res) {
-			struct ttm_buffer_object *bo = res->bo;
-			uint32_t num_pages;
-
-			if (!bo || bo->resource != res)
-				continue;
-
-			num_pages = PFN_UP(bo->base.size);
-			ret = ttm_bo_swapout(bo, ctx, gfp_flags);
-			/* ttm_bo_swapout has dropped the lru_lock */
-			if (!ret) {
-				ttm_resource_cursor_fini(&cursor);
-				return num_pages;
-			}
-			if (ret != -EBUSY) {
-				ttm_resource_cursor_fini(&cursor);
-				return ret;
-			}
-		}
+		lret = ttm_bo_swapout(bdev, ctx, man, gfp_flags, 1);
+		/* Can be both positive (num_pages) and negative (error) */
+		if (lret)
+			return lret;
 	}
-	ttm_resource_cursor_fini_locked(&cursor);
-	spin_unlock(&bdev->lru_lock);
 	return 0;
 }
 EXPORT_SYMBOL(ttm_device_swapout);
include/drm/ttm/ttm_bo.h +3 -2
index 10bff3aecd5c..de97ea9fa75f 100644
@@ -417,8 +417,9 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map);
 int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map);
 void ttm_bo_vunmap(struct ttm_buffer_object *bo, struct iosys_map *map);
 int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo);
-int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
-		   gfp_t gfp_flags);
+long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
+		    struct ttm_resource_manager *man, gfp_t gfp_flags,
+		    pgoff_t target);
 void ttm_bo_pin(struct ttm_buffer_object *bo);
 void ttm_bo_unpin(struct ttm_buffer_object *bo);
 int ttm_mem_evict_first(struct ttm_device *bdev,
Signature · 1 line
2.44.0
maintainer Reviewed-by on v6 6/12
#14 lore raw

Re: [PATCH v6 06/12] drm/ttm: Use the LRU walker helper for swapping

Quoted 11 lines · Thomas Hellström

Rework the TTM swapping to use the LRU walker helper. This helps fixing up the ttm_bo_swapout() interface to be consistent about not requiring any locking.

For now mimic the current behaviour of using trylock only. We could be using ticket-locks here but defer that until it's deemed necessary. The TTM swapout functionality is a bit weird anyway since it alternates between memory types without exhausting TTM_PL_SYSTEM first.

Quoted 3 lines
Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
Quoted 245 lines

v6:
- Improve on error code translation in the swapout callback (Matthew Brost).

---
 drivers/gpu/drm/ttm/ttm_bo.c     | 111 ++++++++++++++++++++-----------
 drivers/gpu/drm/ttm/ttm_device.c |  30 ++-------
 include/drm/ttm/ttm_bo.h         |   5 +-
 3 files changed, 82 insertions(+), 64 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +74 -37
index 43eda720657f..1053cdca131e 100644
@@ -1118,11 +1118,23 @@ int ttm_bo_wait_ctx(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx)
 }
 EXPORT_SYMBOL(ttm_bo_wait_ctx);
 
-int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
-		   gfp_t gfp_flags)
+/**
+ * struct ttm_bo_swapout_walk - Parameters for the swapout walk
+ */
+struct ttm_bo_swapout_walk {
+	/** @walk: The walk base parameters. */
+	struct ttm_lru_walk walk;
+	/** @gfp_flags: The gfp flags to use for ttm_tt_swapout() */
+	gfp_t gfp_flags;
+};
+
+static long
+ttm_bo_swapout_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
 {
-	struct ttm_place place;
-	bool locked;
+	struct ttm_place place = {.mem_type = bo->resource->mem_type};
+	struct ttm_bo_swapout_walk *swapout_walk =
+		container_of(walk, typeof(*swapout_walk), walk);
+	struct ttm_operation_ctx *ctx = walk->ctx;
 	long ret;
 
 	/*
@@ -1131,28 +1143,29 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 	 * The driver may use the fact that we're moving from SYSTEM
 	 * as an indication that we're about to swap out.
 	 */
-	memset(&place, 0, sizeof(place));
-	place.mem_type = bo->resource->mem_type;
-	if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
-		return -EBUSY;
+	if (!bo->bdev->funcs->eviction_valuable(bo, &place)) {
+		ret = -EBUSY;
+		goto out;
+	}
 
 	if (!bo->ttm || !ttm_tt_is_populated(bo->ttm) ||
 	    bo->ttm->page_flags & TTM_TT_FLAG_EXTERNAL ||
-	    bo->ttm->page_flags & TTM_TT_FLAG_SWAPPED ||
-	    !ttm_bo_get_unless_zero(bo)) {
-		if (locked)
-			dma_resv_unlock(bo->base.resv);
-		return -EBUSY;
+	    bo->ttm->page_flags & TTM_TT_FLAG_SWAPPED) {
+		ret = -EBUSY;
+		goto out;
 	}
 
 	if (bo->deleted) {
-		ret = ttm_bo_cleanup_refs(bo, false, false, locked);
-		ttm_bo_put(bo);
-		return ret == -EBUSY ? -ENOSPC : ret;
-	}
+		pgoff_t num_pages = bo->ttm->num_pages;
 
-	/* TODO: Cleanup the locking */
-	spin_unlock(&bo->bdev->lru_lock);
+		ret = ttm_bo_wait_ctx(bo, ctx);
+		if (ret)
+			goto out;
+
+		ttm_bo_cleanup_memtype_use(bo);
+		ret = num_pages;
+		goto out;
+	}
 
 	/*
 	 * Move to system cached
@@ -1164,12 +1177,13 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 		memset(&hop, 0, sizeof(hop));
 		place.mem_type = TTM_PL_SYSTEM;
 		ret = ttm_resource_alloc(bo, &place, &evict_mem);
-		if (unlikely(ret))
+		if (ret)
 			goto out;
 
 		ret = ttm_bo_handle_move_mem(bo, evict_mem, true, ctx, &hop);
-		if (unlikely(ret != 0)) {
-			WARN(ret == -EMULTIHOP, "Unexpected multihop in swaput - likely driver bug.\n");
+		if (ret) {
+			WARN(ret == -EMULTIHOP,
+			     "Unexpected multihop in swapout - likely driver bug.\n");
 			ttm_resource_free(bo, &evict_mem);
 			goto out;
 		}
@@ -1179,30 +1193,53 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
 	 * Make sure BO is idle.
 	 */
 	ret = ttm_bo_wait_ctx(bo, ctx);
-	if (unlikely(ret != 0))
+	if (ret)
 		goto out;
 
 	ttm_bo_unmap_virtual(bo);
-
-	/*
-	 * Swap out. Buffer will be swapped in again as soon as
-	 * anyone tries to access a ttm page.
-	 */
 	if (bo->bdev->funcs->swap_notify)
 		bo->bdev->funcs->swap_notify(bo);
 
 	if (ttm_tt_is_populated(bo->ttm))
-		ret = ttm_tt_swapout(bo->bdev, bo->ttm, gfp_flags);
+		ret = ttm_tt_swapout(bo->bdev, bo->ttm, swapout_walk->gfp_flags);
 out:
+	/* Consider -ENOMEM and -ENOSPC non-fatal. */
+	if (ret == -ENOMEM || ret == -ENOSPC)
+		ret = -EBUSY;
 
-	/*
-	 * Unreserve without putting on LRU to avoid swapping out an
-	 * already swapped buffer.
-	 */
-	if (locked)
-		dma_resv_unlock(bo->base.resv);
-	ttm_bo_put(bo);
-	return ret == -EBUSY ? -ENOSPC : ret;
+	return ret;
+}
+
+const struct ttm_lru_walk_ops ttm_swap_ops = {
+	.process_bo = ttm_bo_swapout_cb,
+};
+
+/**
+ * ttm_bo_swapout() - Swap out buffer objects on the LRU list to shmem.
+ * @bdev: The ttm device.
+ * @ctx: The ttm_operation_ctx governing the swapout operation.
+ * @man: The resource manager whose resources / buffer objects are
+ * goint to be swapped out.
+ * @gfp_flags: The gfp flags used for shmem page allocations.
+ * @target: The desired number of pages to swap out.
+ *
+ * Return: The number of pages actually swapped out, or negative error code
+ * on error.
+ */
+long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
+		    struct ttm_resource_manager *man, gfp_t gfp_flags,
+		    pgoff_t target)
+{
+	struct ttm_bo_swapout_walk swapout_walk = {
+		.walk = {
+			.ops = &ttm_swap_ops,
+			.ctx = ctx,
+			.trylock_only = true,
+		},
+		.gfp_flags = gfp_flags,
+	};
+
+	return ttm_lru_walk_for_evict(&swapout_walk.walk, bdev, man, target);
 }
 
 void ttm_bo_tt_destroy(struct ttm_buffer_object *bo)
drivers/gpu/drm/ttm/ttm_device.c +5 -25
index f9e9b1ec8c8a..ee575d8a54c0 100644
@@ -148,40 +148,20 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags)
 int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
 		       gfp_t gfp_flags)
 {
-	struct ttm_resource_cursor cursor;
 	struct ttm_resource_manager *man;
-	struct ttm_resource *res;
 	unsigned i;
-	int ret;
+	long lret;
 
-	spin_lock(&bdev->lru_lock);
 	for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) {
 		man = ttm_manager_type(bdev, i);
 		if (!man || !man->use_tt)
 			continue;
 
-		ttm_resource_manager_for_each_res(man, &cursor, res) {
-			struct ttm_buffer_object *bo = res->bo;
-			uint32_t num_pages;
-
-			if (!bo || bo->resource != res)
-				continue;
-
-			num_pages = PFN_UP(bo->base.size);
-			ret = ttm_bo_swapout(bo, ctx, gfp_flags);
-			/* ttm_bo_swapout has dropped the lru_lock */
-			if (!ret) {
-				ttm_resource_cursor_fini(&cursor);
-				return num_pages;
-			}
-			if (ret != -EBUSY) {
-				ttm_resource_cursor_fini(&cursor);
-				return ret;
-			}
-		}
+		lret = ttm_bo_swapout(bdev, ctx, man, gfp_flags, 1);
+		/* Can be both positive (num_pages) and negative (error) */
+		if (lret)
+			return lret;
 	}
-	ttm_resource_cursor_fini_locked(&cursor);
-	spin_unlock(&bdev->lru_lock);
 	return 0;
 }
 EXPORT_SYMBOL(ttm_device_swapout);
include/drm/ttm/ttm_bo.h +3 -2
index 10bff3aecd5c..de97ea9fa75f 100644
@@ -417,8 +417,9 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map);
 int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map);
 void ttm_bo_vunmap(struct ttm_buffer_object *bo, struct iosys_map *map);
 int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo);
-int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
-		   gfp_t gfp_flags);
+long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
+		    struct ttm_resource_manager *man, gfp_t gfp_flags,
+		    pgoff_t target);
 void ttm_bo_pin(struct ttm_buffer_object *bo);
 void ttm_bo_unpin(struct ttm_buffer_object *bo);
 int ttm_mem_evict_first(struct ttm_device *bdev,
Signature · 1 line
2.44.0
author maintainer
#15 lore raw

PATCH v6 7/12 drm/ttm: Use the LRU walker for eviction

In reply to Thomas Hellström

Use the LRU walker for eviction. This helps
removing a lot of code with weird locking
semantics.

The functionality is slightly changed so that when trylocked buffer objects are exhausted, we continue to interleave walks with ticket-locks while there is still progress made. The list walks are not restarted in-between evictions.

Also provide a separate ttm_bo_evict_first() function for its single user. The context of that user allows sleeping dma_resv locks.

v6:
- Various cleanups suggested by Matthew Brost.
- Fix error return code of ttm_bo_evict_first(). (Matthew Brost)
- Fix an error check that was inverted. (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo.c       | 346 ++++++++++++-----------------
 drivers/gpu/drm/ttm/ttm_resource.c |  21 +-
 include/drm/ttm/ttm_bo.h           |   8 +-
 3 files changed, 144 insertions(+), 231 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +137 -209
index 1053cdca131e..603b9353f436 100644
@@ -224,80 +224,6 @@ static void ttm_bo_flush_all_fences(struct ttm_buffer_object *bo)
 	dma_resv_iter_end(&cursor);
 }
 
-/**
- * ttm_bo_cleanup_refs
- * If bo idle, remove from lru lists, and unref.
- * If not idle, block if possible.
- *
- * Must be called with lru_lock and reservation held, this function
- * will drop the lru lock and optionally the reservation lock before returning.
- *
- * @bo:                    The buffer object to clean-up
- * @interruptible:         Any sleeps should occur interruptibly.
- * @no_wait_gpu:           Never wait for gpu. Return -EBUSY instead.
- * @unlock_resv:           Unlock the reservation lock as well.
- */
-
-static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
-			       bool interruptible, bool no_wait_gpu,
-			       bool unlock_resv)
-{
-	struct dma_resv *resv = &bo->base._resv;
-	int ret;
-
-	if (dma_resv_test_signaled(resv, DMA_RESV_USAGE_BOOKKEEP))
-		ret = 0;
-	else
-		ret = -EBUSY;
-
-	if (ret && !no_wait_gpu) {
-		long lret;
-
-		if (unlock_resv)
-			dma_resv_unlock(bo->base.resv);
-		spin_unlock(&bo->bdev->lru_lock);
-
-		lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP,
-					     interruptible,
-					     30 * HZ);
-
-		if (lret < 0)
-			return lret;
-		else if (lret == 0)
-			return -EBUSY;
-
-		spin_lock(&bo->bdev->lru_lock);
-		if (unlock_resv && !dma_resv_trylock(bo->base.resv)) {
-			/*
-			 * We raced, and lost, someone else holds the reservation now,
-			 * and is probably busy in ttm_bo_cleanup_memtype_use.
-			 *
-			 * Even if it's not the case, because we finished waiting any
-			 * delayed destruction would succeed, so just return success
-			 * here.
-			 */
-			spin_unlock(&bo->bdev->lru_lock);
-			return 0;
-		}
-		ret = 0;
-	}
-
-	if (ret) {
-		if (unlock_resv)
-			dma_resv_unlock(bo->base.resv);
-		spin_unlock(&bo->bdev->lru_lock);
-		return ret;
-	}
-
-	spin_unlock(&bo->bdev->lru_lock);
-	ttm_bo_cleanup_memtype_use(bo);
-
-	if (unlock_resv)
-		dma_resv_unlock(bo->base.resv);
-
-	return 0;
-}
-
 /*
  * Block for the dma_resv object to become idle, lock the buffer and clean up
  * the resource and tt object.
@@ -505,151 +431,153 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
 }
 EXPORT_SYMBOL(ttm_bo_eviction_valuable);
 
-/*
- * Check the target bo is allowable to be evicted or swapout, including cases:
- *
- * a. if share same reservation object with ctx->resv, have assumption
- * reservation objects should already be locked, so not lock again and
- * return true directly when either the opreation allow_reserved_eviction
- * or the target bo already is in delayed free list;
+/**
+ * ttm_bo_evict_first() - Evict the first bo on the manager's LRU list.
+ * @bdev: The ttm device.
+ * @man: The manager whose bo to evict.
+ * @ctx: The TTM operation ctx governing the eviction.
  *
- * b. Otherwise, trylock it.
+ * Return: 0 if successful or the resource disappeared. Negative error code on error.
  */
-static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
-					   struct ttm_operation_ctx *ctx,
-					   const struct ttm_place *place,
-					   bool *locked, bool *busy)
+int ttm_bo_evict_first(struct ttm_device *bdev, struct ttm_resource_manager *man,
+		       struct ttm_operation_ctx *ctx)
 {
-	bool ret = false;
+	struct ttm_resource_cursor cursor;
+	struct ttm_buffer_object *bo;
+	struct ttm_resource *res;
+	unsigned int mem_type;
+	int ret = 0;
 
-	if (bo->pin_count) {
-		*locked = false;
-		if (busy)
-			*busy = false;
-		return false;
+	spin_lock(&bdev->lru_lock);
+	res = ttm_resource_manager_first(man, &cursor);
+	if (!res) {
+		ret = -ENOENT;
+		goto out_no_ref;
 	}
+	bo = res->bo;
+	if (!ttm_bo_get_unless_zero(bo))
+		goto out_no_ref;
+	mem_type = res->mem_type;
+	spin_unlock(&bdev->lru_lock);
+	ret = ttm_bo_reserve(bo, ctx->interruptible, ctx->no_wait_gpu, NULL);
+	if (ret)
+		goto out_no_lock;
+	if (bo->resource != res || res->mem_type != mem_type)
+		goto out_bo_moved;
 
-	if (bo->base.resv == ctx->resv) {
-		dma_resv_assert_held(bo->base.resv);
-		if (ctx->allow_res_evict)
-			ret = true;
-		*locked = false;
-		if (busy)
-			*busy = false;
+	if (bo->deleted) {
+		ret = ttm_bo_wait_ctx(bo, ctx);
+		if (!ret)
+			ttm_bo_cleanup_memtype_use(bo);
 	} else {
-		ret = dma_resv_trylock(bo->base.resv);
-		*locked = ret;
-		if (busy)
-			*busy = !ret;
-	}
-
-	if (ret && place && (bo->resource->mem_type != place->mem_type ||
-		!bo->bdev->funcs->eviction_valuable(bo, place))) {
-		ret = false;
-		if (*locked) {
-			dma_resv_unlock(bo->base.resv);
-			*locked = false;
-		}
+		ret = ttm_bo_evict(bo, ctx);
 	}
+out_bo_moved:
+	dma_resv_unlock(bo->base.resv);
+out_no_lock:
+	ttm_bo_put(bo);
+	ttm_resource_cursor_fini(&cursor);
+	return ret;
 
+out_no_ref:
+	ttm_resource_cursor_fini_locked(&cursor);
+	spin_unlock(&bdev->lru_lock);
 	return ret;
 }
 
 /**
- * ttm_mem_evict_wait_busy - wait for a busy BO to become available
- *
- * @busy_bo: BO which couldn't be locked with trylock
- * @ctx: operation context
- * @ticket: acquire ticket
- *
- * Try to lock a busy buffer object to avoid failing eviction.
+ * struct ttm_bo_evict_walk - Parameters for the evict walk.
  */
-static int ttm_mem_evict_wait_busy(struct ttm_buffer_object *busy_bo,
-				   struct ttm_operation_ctx *ctx,
-				   struct ww_acquire_ctx *ticket)
-{
-	int r;
-
-	if (!busy_bo || !ticket)
-		return -EBUSY;
-
-	if (ctx->interruptible)
-		r = dma_resv_lock_interruptible(busy_bo->base.resv,
-							  ticket);
-	else
-		r = dma_resv_lock(busy_bo->base.resv, ticket);
-
-	/*
-	 * TODO: It would be better to keep the BO locked until allocation is at
-	 * least tried one more time, but that would mean a much larger rework
-	 * of TTM.
-	 */
-	if (!r)
-		dma_resv_unlock(busy_bo->base.resv);
-
-	return r == -EDEADLK ? -EBUSY : r;
-}
+struct ttm_bo_evict_walk {
+	/** @walk: The walk base parameters. */
+	struct ttm_lru_walk walk;
+	/** @place: The place passed to the resource allocation. */
+	const struct ttm_place *place;
+	/** @evictor: The buffer object we're trying to make room for. */
+	struct ttm_buffer_object *evictor;
+	/** @res: The allocated resource if any. */
+	struct ttm_resource **res;
+	/** @evicted: Number of successful evictions. */
+	unsigned long evicted;
+};
 
-int ttm_mem_evict_first(struct ttm_device *bdev,
-			struct ttm_resource_manager *man,
-			const struct ttm_place *place,
-			struct ttm_operation_ctx *ctx,
-			struct ww_acquire_ctx *ticket)
+static long ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
 {
-	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
-	struct ttm_resource_cursor cursor;
-	struct ttm_resource *res;
-	bool locked = false;
-	int ret;
+	struct ttm_bo_evict_walk *evict_walk =
+		container_of(walk, typeof(*evict_walk), walk);
+	long lret;
 
-	spin_lock(&bdev->lru_lock);
-	ttm_resource_manager_for_each_res(man, &cursor, res) {
-		bool busy;
-
-		if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place,
-						    &locked, &busy)) {
-			if (busy && !busy_bo && ticket !=
-			    dma_resv_locking_ctx(res->bo->base.resv))
-				busy_bo = res->bo;
-			continue;
-		}
+	if (!bo->bdev->funcs->eviction_valuable(bo, evict_walk->place))
+		return 0;
 
-		if (ttm_bo_get_unless_zero(res->bo)) {
-			bo = res->bo;
-			break;
-		}
-		if (locked)
-			dma_resv_unlock(res->bo->base.resv);
+	if (bo->deleted) {
+		lret = ttm_bo_wait_ctx(bo, walk->ctx);
+		if (!lret)
+			ttm_bo_cleanup_memtype_use(bo);
+	} else {
+		lret = ttm_bo_evict(bo, walk->ctx);
 	}
-	ttm_resource_cursor_fini_locked(&cursor);
 
-	if (!bo) {
-		if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
-			busy_bo = NULL;
-		spin_unlock(&bdev->lru_lock);
-		ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
-		if (busy_bo)
-			ttm_bo_put(busy_bo);
-		return ret;
-	}
+	if (lret)
+		goto out;
 
-	if (bo->deleted) {
-		ret = ttm_bo_cleanup_refs(bo, ctx->interruptible,
-					  ctx->no_wait_gpu, locked);
-		ttm_bo_put(bo);
-		return ret;
-	}
+	evict_walk->evicted++;
+	if (evict_walk->res)
+		lret = ttm_resource_alloc(evict_walk->evictor, evict_walk->place,
+					  evict_walk->res);
+	if (lret == 0)
+		return 1;
+out:
+	/* Errors that should terminate the walk. */
+	if (lret == -ENOSPC)
+		return -EBUSY;
 
-	spin_unlock(&bdev->lru_lock);
+	return lret;
+}
 
-	ret = ttm_bo_evict(bo, ctx);
-	if (locked)
-		ttm_bo_unreserve(bo);
-	else
-		ttm_bo_move_to_lru_tail_unlocked(bo);
+static const struct ttm_lru_walk_ops ttm_evict_walk_ops = {
+	.process_bo = ttm_bo_evict_cb,
+};
 
-	ttm_bo_put(bo);
-	return ret;
+static int ttm_bo_evict_alloc(struct ttm_device *bdev,
+			      struct ttm_resource_manager *man,
+			      const struct ttm_place *place,
+			      struct ttm_buffer_object *evictor,
+			      struct ttm_operation_ctx *ctx,
+			      struct ww_acquire_ctx *ticket,
+			      struct ttm_resource **res)
+{
+	struct ttm_bo_evict_walk evict_walk = {
+		.walk = {
+			.ops = &ttm_evict_walk_ops,
+			.ctx = ctx,
+			.ticket = ticket,
+		},
+		.place = place,
+		.evictor = evictor,
+		.res = res,
+	};
+	long lret;
+
+	evict_walk.walk.trylock_only = true;
+	lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
+	if (lret || !ticket)
+		goto out;
+
+	/* If ticket-locking, repeat while making progress. */
+	evict_walk.walk.trylock_only = false;
+	do {
+		/* The walk may clear the evict_walk.walk.ticket field */
+		evict_walk.walk.ticket = ticket;
+		evict_walk.evicted = 0;
+		lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
+	} while (!lret && evict_walk.evicted);
+out:
+	if (lret < 0)
+		return lret;
+	if (lret == 0)
+		return -EBUSY;
+	return 0;
 }
 
 /**
@@ -760,6 +688,7 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo,
 	for (i = 0; i < placement->num_placement; ++i) {
 		const struct ttm_place *place = &placement->placement[i];
 		struct ttm_resource_manager *man;
+		bool may_evict;
 
 		man = ttm_manager_type(bdev, place->mem_type);
 		if (!man || !ttm_resource_manager_used(man))
@@ -769,22 +698,21 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo,
 				    TTM_PL_FLAG_FALLBACK))
 			continue;
 
-		do {
-			ret = ttm_resource_alloc(bo, place, res);
-			if (unlikely(ret && ret != -ENOSPC))
+		may_evict = (force_space && place->mem_type != TTM_PL_SYSTEM);
+		ret = ttm_resource_alloc(bo, place, res);
+		if (ret) {
+			if (ret != -ENOSPC)
 				return ret;
-			if (likely(!ret) || !force_space)
-				break;
-
-			ret = ttm_mem_evict_first(bdev, man, place, ctx,
-						  ticket);
-			if (unlikely(ret == -EBUSY))
-				break;
-			if (unlikely(ret))
+			if (!may_evict)
+				continue;
+
+			ret = ttm_bo_evict_alloc(bdev, man, place, bo, ctx,
+						 ticket, res);
+			if (ret == -EBUSY)
+				continue;
+			if (ret)
 				return ret;
-		} while (1);
-		if (ret)
-			continue;
+		}
 
 		ret = ttm_bo_add_move_fence(bo, man, ctx->no_wait_gpu);
 		if (unlikely(ret)) {
drivers/gpu/drm/ttm/ttm_resource.c +4 -17
index b6a2daac5518..9dc727d416cc 100644
@@ -512,24 +512,11 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
 	};
 	struct dma_fence *fence;
 	int ret;
-	unsigned i;
-
-	/*
-	 * Can't use standard list traversal since we're unlocking.
-	 */
 
-	spin_lock(&bdev->lru_lock);
-	for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
-		while (!list_empty(&man->lru[i])) {
-			spin_unlock(&bdev->lru_lock);
-			ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
-						  NULL);
-			if (ret)
-				return ret;
-			spin_lock(&bdev->lru_lock);
-		}
-	}
-	spin_unlock(&bdev->lru_lock);
+	do {
+		ret = ttm_bo_evict_first(bdev, man, &ctx);
+		cond_resched();
+	} while (!ret);
 
 	spin_lock(&man->move_lock);
 	fence = dma_fence_get(man->move);
include/drm/ttm/ttm_bo.h +3 -5
index de97ea9fa75f..e577528f5dfc 100644
@@ -422,11 +422,9 @@ long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
 		    pgoff_t target);
 void ttm_bo_pin(struct ttm_buffer_object *bo);
 void ttm_bo_unpin(struct ttm_buffer_object *bo);
-int ttm_mem_evict_first(struct ttm_device *bdev,
-			struct ttm_resource_manager *man,
-			const struct ttm_place *place,
-			struct ttm_operation_ctx *ctx,
-			struct ww_acquire_ctx *ticket);
+int ttm_bo_evict_first(struct ttm_device *bdev,
+		       struct ttm_resource_manager *man,
+		       struct ttm_operation_ctx *ctx);
 vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo,
 			     struct vm_fault *vmf);
 vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
Signature · 1 line
2.44.0
maintainer Reviewed-by on v6 7/12
#16 lore raw

Re: [PATCH v6 07/12] drm/ttm: Use the LRU walker for eviction

Quoted 14 lines · Thomas Hellström

Use the LRU walker for eviction. This helps
removing a lot of code with weird locking
semantics.

The functionality is slightly changed so that when trylocked buffer objects are exhausted, we continue to interleave walks with ticket-locks while there is still progress made. The list walks are not restarted in-between evictions.

Also provide a separate ttm_bo_evict_first() function for its single user. The context of that user allows sleeping dma_resv locks.

Quoted 8 lines

v6:
- Various cleanups suggested by Matthew Brost.
- Fix error return code of ttm_bo_evict_first(). (Matthew Brost)
- Fix an error check that was inverted. (Matthew Brost)

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
Quoted 467 lines
---
 drivers/gpu/drm/ttm/ttm_bo.c       | 346 ++++++++++++-----------------
 drivers/gpu/drm/ttm/ttm_resource.c |  21 +-
 include/drm/ttm/ttm_bo.h           |   8 +-
 3 files changed, 144 insertions(+), 231 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +137 -209
index 1053cdca131e..603b9353f436 100644
@@ -224,80 +224,6 @@ static void ttm_bo_flush_all_fences(struct ttm_buffer_object *bo)
 	dma_resv_iter_end(&cursor);
 }
 
-/**
- * ttm_bo_cleanup_refs
- * If bo idle, remove from lru lists, and unref.
- * If not idle, block if possible.
- *
- * Must be called with lru_lock and reservation held, this function
- * will drop the lru lock and optionally the reservation lock before returning.
- *
- * @bo:                    The buffer object to clean-up
- * @interruptible:         Any sleeps should occur interruptibly.
- * @no_wait_gpu:           Never wait for gpu. Return -EBUSY instead.
- * @unlock_resv:           Unlock the reservation lock as well.
- */
-
-static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
-			       bool interruptible, bool no_wait_gpu,
-			       bool unlock_resv)
-{
-	struct dma_resv *resv = &bo->base._resv;
-	int ret;
-
-	if (dma_resv_test_signaled(resv, DMA_RESV_USAGE_BOOKKEEP))
-		ret = 0;
-	else
-		ret = -EBUSY;
-
-	if (ret && !no_wait_gpu) {
-		long lret;
-
-		if (unlock_resv)
-			dma_resv_unlock(bo->base.resv);
-		spin_unlock(&bo->bdev->lru_lock);
-
-		lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP,
-					     interruptible,
-					     30 * HZ);
-
-		if (lret < 0)
-			return lret;
-		else if (lret == 0)
-			return -EBUSY;
-
-		spin_lock(&bo->bdev->lru_lock);
-		if (unlock_resv && !dma_resv_trylock(bo->base.resv)) {
-			/*
-			 * We raced, and lost, someone else holds the reservation now,
-			 * and is probably busy in ttm_bo_cleanup_memtype_use.
-			 *
-			 * Even if it's not the case, because we finished waiting any
-			 * delayed destruction would succeed, so just return success
-			 * here.
-			 */
-			spin_unlock(&bo->bdev->lru_lock);
-			return 0;
-		}
-		ret = 0;
-	}
-
-	if (ret) {
-		if (unlock_resv)
-			dma_resv_unlock(bo->base.resv);
-		spin_unlock(&bo->bdev->lru_lock);
-		return ret;
-	}
-
-	spin_unlock(&bo->bdev->lru_lock);
-	ttm_bo_cleanup_memtype_use(bo);
-
-	if (unlock_resv)
-		dma_resv_unlock(bo->base.resv);
-
-	return 0;
-}
-
 /*
  * Block for the dma_resv object to become idle, lock the buffer and clean up
  * the resource and tt object.
@@ -505,151 +431,153 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
 }
 EXPORT_SYMBOL(ttm_bo_eviction_valuable);
 
-/*
- * Check the target bo is allowable to be evicted or swapout, including cases:
- *
- * a. if share same reservation object with ctx->resv, have assumption
- * reservation objects should already be locked, so not lock again and
- * return true directly when either the opreation allow_reserved_eviction
- * or the target bo already is in delayed free list;
+/**
+ * ttm_bo_evict_first() - Evict the first bo on the manager's LRU list.
+ * @bdev: The ttm device.
+ * @man: The manager whose bo to evict.
+ * @ctx: The TTM operation ctx governing the eviction.
  *
- * b. Otherwise, trylock it.
+ * Return: 0 if successful or the resource disappeared. Negative error code on error.
  */
-static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
-					   struct ttm_operation_ctx *ctx,
-					   const struct ttm_place *place,
-					   bool *locked, bool *busy)
+int ttm_bo_evict_first(struct ttm_device *bdev, struct ttm_resource_manager *man,
+		       struct ttm_operation_ctx *ctx)
 {
-	bool ret = false;
+	struct ttm_resource_cursor cursor;
+	struct ttm_buffer_object *bo;
+	struct ttm_resource *res;
+	unsigned int mem_type;
+	int ret = 0;
 
-	if (bo->pin_count) {
-		*locked = false;
-		if (busy)
-			*busy = false;
-		return false;
+	spin_lock(&bdev->lru_lock);
+	res = ttm_resource_manager_first(man, &cursor);
+	if (!res) {
+		ret = -ENOENT;
+		goto out_no_ref;
 	}
+	bo = res->bo;
+	if (!ttm_bo_get_unless_zero(bo))
+		goto out_no_ref;
+	mem_type = res->mem_type;
+	spin_unlock(&bdev->lru_lock);
+	ret = ttm_bo_reserve(bo, ctx->interruptible, ctx->no_wait_gpu, NULL);
+	if (ret)
+		goto out_no_lock;
+	if (bo->resource != res || res->mem_type != mem_type)
+		goto out_bo_moved;
 
-	if (bo->base.resv == ctx->resv) {
-		dma_resv_assert_held(bo->base.resv);
-		if (ctx->allow_res_evict)
-			ret = true;
-		*locked = false;
-		if (busy)
-			*busy = false;
+	if (bo->deleted) {
+		ret = ttm_bo_wait_ctx(bo, ctx);
+		if (!ret)
+			ttm_bo_cleanup_memtype_use(bo);
 	} else {
-		ret = dma_resv_trylock(bo->base.resv);
-		*locked = ret;
-		if (busy)
-			*busy = !ret;
-	}
-
-	if (ret && place && (bo->resource->mem_type != place->mem_type ||
-		!bo->bdev->funcs->eviction_valuable(bo, place))) {
-		ret = false;
-		if (*locked) {
-			dma_resv_unlock(bo->base.resv);
-			*locked = false;
-		}
+		ret = ttm_bo_evict(bo, ctx);
 	}
+out_bo_moved:
+	dma_resv_unlock(bo->base.resv);
+out_no_lock:
+	ttm_bo_put(bo);
+	ttm_resource_cursor_fini(&cursor);
+	return ret;
 
+out_no_ref:
+	ttm_resource_cursor_fini_locked(&cursor);
+	spin_unlock(&bdev->lru_lock);
 	return ret;
 }
 
 /**
- * ttm_mem_evict_wait_busy - wait for a busy BO to become available
- *
- * @busy_bo: BO which couldn't be locked with trylock
- * @ctx: operation context
- * @ticket: acquire ticket
- *
- * Try to lock a busy buffer object to avoid failing eviction.
+ * struct ttm_bo_evict_walk - Parameters for the evict walk.
  */
-static int ttm_mem_evict_wait_busy(struct ttm_buffer_object *busy_bo,
-				   struct ttm_operation_ctx *ctx,
-				   struct ww_acquire_ctx *ticket)
-{
-	int r;
-
-	if (!busy_bo || !ticket)
-		return -EBUSY;
-
-	if (ctx->interruptible)
-		r = dma_resv_lock_interruptible(busy_bo->base.resv,
-							  ticket);
-	else
-		r = dma_resv_lock(busy_bo->base.resv, ticket);
-
-	/*
-	 * TODO: It would be better to keep the BO locked until allocation is at
-	 * least tried one more time, but that would mean a much larger rework
-	 * of TTM.
-	 */
-	if (!r)
-		dma_resv_unlock(busy_bo->base.resv);
-
-	return r == -EDEADLK ? -EBUSY : r;
-}
+struct ttm_bo_evict_walk {
+	/** @walk: The walk base parameters. */
+	struct ttm_lru_walk walk;
+	/** @place: The place passed to the resource allocation. */
+	const struct ttm_place *place;
+	/** @evictor: The buffer object we're trying to make room for. */
+	struct ttm_buffer_object *evictor;
+	/** @res: The allocated resource if any. */
+	struct ttm_resource **res;
+	/** @evicted: Number of successful evictions. */
+	unsigned long evicted;
+};
 
-int ttm_mem_evict_first(struct ttm_device *bdev,
-			struct ttm_resource_manager *man,
-			const struct ttm_place *place,
-			struct ttm_operation_ctx *ctx,
-			struct ww_acquire_ctx *ticket)
+static long ttm_bo_evict_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
 {
-	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
-	struct ttm_resource_cursor cursor;
-	struct ttm_resource *res;
-	bool locked = false;
-	int ret;
+	struct ttm_bo_evict_walk *evict_walk =
+		container_of(walk, typeof(*evict_walk), walk);
+	long lret;
 
-	spin_lock(&bdev->lru_lock);
-	ttm_resource_manager_for_each_res(man, &cursor, res) {
-		bool busy;
-
-		if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place,
-						    &locked, &busy)) {
-			if (busy && !busy_bo && ticket !=
-			    dma_resv_locking_ctx(res->bo->base.resv))
-				busy_bo = res->bo;
-			continue;
-		}
+	if (!bo->bdev->funcs->eviction_valuable(bo, evict_walk->place))
+		return 0;
 
-		if (ttm_bo_get_unless_zero(res->bo)) {
-			bo = res->bo;
-			break;
-		}
-		if (locked)
-			dma_resv_unlock(res->bo->base.resv);
+	if (bo->deleted) {
+		lret = ttm_bo_wait_ctx(bo, walk->ctx);
+		if (!lret)
+			ttm_bo_cleanup_memtype_use(bo);
+	} else {
+		lret = ttm_bo_evict(bo, walk->ctx);
 	}
-	ttm_resource_cursor_fini_locked(&cursor);
 
-	if (!bo) {
-		if (busy_bo && !ttm_bo_get_unless_zero(busy_bo))
-			busy_bo = NULL;
-		spin_unlock(&bdev->lru_lock);
-		ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket);
-		if (busy_bo)
-			ttm_bo_put(busy_bo);
-		return ret;
-	}
+	if (lret)
+		goto out;
 
-	if (bo->deleted) {
-		ret = ttm_bo_cleanup_refs(bo, ctx->interruptible,
-					  ctx->no_wait_gpu, locked);
-		ttm_bo_put(bo);
-		return ret;
-	}
+	evict_walk->evicted++;
+	if (evict_walk->res)
+		lret = ttm_resource_alloc(evict_walk->evictor, evict_walk->place,
+					  evict_walk->res);
+	if (lret == 0)
+		return 1;
+out:
+	/* Errors that should terminate the walk. */
+	if (lret == -ENOSPC)
+		return -EBUSY;
 
-	spin_unlock(&bdev->lru_lock);
+	return lret;
+}
 
-	ret = ttm_bo_evict(bo, ctx);
-	if (locked)
-		ttm_bo_unreserve(bo);
-	else
-		ttm_bo_move_to_lru_tail_unlocked(bo);
+static const struct ttm_lru_walk_ops ttm_evict_walk_ops = {
+	.process_bo = ttm_bo_evict_cb,
+};
 
-	ttm_bo_put(bo);
-	return ret;
+static int ttm_bo_evict_alloc(struct ttm_device *bdev,
+			      struct ttm_resource_manager *man,
+			      const struct ttm_place *place,
+			      struct ttm_buffer_object *evictor,
+			      struct ttm_operation_ctx *ctx,
+			      struct ww_acquire_ctx *ticket,
+			      struct ttm_resource **res)
+{
+	struct ttm_bo_evict_walk evict_walk = {
+		.walk = {
+			.ops = &ttm_evict_walk_ops,
+			.ctx = ctx,
+			.ticket = ticket,
+		},
+		.place = place,
+		.evictor = evictor,
+		.res = res,
+	};
+	long lret;
+
+	evict_walk.walk.trylock_only = true;
+	lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
+	if (lret || !ticket)
+		goto out;
+
+	/* If ticket-locking, repeat while making progress. */
+	evict_walk.walk.trylock_only = false;
+	do {
+		/* The walk may clear the evict_walk.walk.ticket field */
+		evict_walk.walk.ticket = ticket;
+		evict_walk.evicted = 0;
+		lret = ttm_lru_walk_for_evict(&evict_walk.walk, bdev, man, 1);
+	} while (!lret && evict_walk.evicted);
+out:
+	if (lret < 0)
+		return lret;
+	if (lret == 0)
+		return -EBUSY;
+	return 0;
 }
 
 /**
@@ -760,6 +688,7 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo,
 	for (i = 0; i < placement->num_placement; ++i) {
 		const struct ttm_place *place = &placement->placement[i];
 		struct ttm_resource_manager *man;
+		bool may_evict;
 
 		man = ttm_manager_type(bdev, place->mem_type);
 		if (!man || !ttm_resource_manager_used(man))
@@ -769,22 +698,21 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo,
 				    TTM_PL_FLAG_FALLBACK))
 			continue;
 
-		do {
-			ret = ttm_resource_alloc(bo, place, res);
-			if (unlikely(ret && ret != -ENOSPC))
+		may_evict = (force_space && place->mem_type != TTM_PL_SYSTEM);
+		ret = ttm_resource_alloc(bo, place, res);
+		if (ret) {
+			if (ret != -ENOSPC)
 				return ret;
-			if (likely(!ret) || !force_space)
-				break;
-
-			ret = ttm_mem_evict_first(bdev, man, place, ctx,
-						  ticket);
-			if (unlikely(ret == -EBUSY))
-				break;
-			if (unlikely(ret))
+			if (!may_evict)
+				continue;
+
+			ret = ttm_bo_evict_alloc(bdev, man, place, bo, ctx,
+						 ticket, res);
+			if (ret == -EBUSY)
+				continue;
+			if (ret)
 				return ret;
-		} while (1);
-		if (ret)
-			continue;
+		}
 
 		ret = ttm_bo_add_move_fence(bo, man, ctx->no_wait_gpu);
 		if (unlikely(ret)) {
drivers/gpu/drm/ttm/ttm_resource.c +4 -17
index b6a2daac5518..9dc727d416cc 100644
@@ -512,24 +512,11 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
 	};
 	struct dma_fence *fence;
 	int ret;
-	unsigned i;
-
-	/*
-	 * Can't use standard list traversal since we're unlocking.
-	 */
 
-	spin_lock(&bdev->lru_lock);
-	for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) {
-		while (!list_empty(&man->lru[i])) {
-			spin_unlock(&bdev->lru_lock);
-			ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
-						  NULL);
-			if (ret)
-				return ret;
-			spin_lock(&bdev->lru_lock);
-		}
-	}
-	spin_unlock(&bdev->lru_lock);
+	do {
+		ret = ttm_bo_evict_first(bdev, man, &ctx);
+		cond_resched();
+	} while (!ret);
 
 	spin_lock(&man->move_lock);
 	fence = dma_fence_get(man->move);
include/drm/ttm/ttm_bo.h +3 -5
index de97ea9fa75f..e577528f5dfc 100644
@@ -422,11 +422,9 @@ long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,
 		    pgoff_t target);
 void ttm_bo_pin(struct ttm_buffer_object *bo);
 void ttm_bo_unpin(struct ttm_buffer_object *bo);
-int ttm_mem_evict_first(struct ttm_device *bdev,
-			struct ttm_resource_manager *man,
-			const struct ttm_place *place,
-			struct ttm_operation_ctx *ctx,
-			struct ww_acquire_ctx *ticket);
+int ttm_bo_evict_first(struct ttm_device *bdev,
+		       struct ttm_resource_manager *man,
+		       struct ttm_operation_ctx *ctx);
 vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo,
 			     struct vm_fault *vmf);
 vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
Signature · 1 line
2.44.0
author maintainer
#17 lore raw

PATCH v6 8/12 drm/ttm: Add a virtual base class for graphics memory backup

In reply to Thomas Hellström

Initially intended for experimenting with different backup solutions (shmem vs direct swap cache insertion), abstract the backup destination using a virtual base class.

Also provide a sample implementation for shmem.

While when settling on a preferred backup solution, one could perhaps skip the abstraction, this functionality may actually come in handy for configurable dedicated graphics memory backup to fast nvme files or similar, whithout affecting swap-space. Could indeed be useful for VRAM backup on S4 and other cases.

v5:
- Fix a UAF. (kernel test robot, Dan Carptenter) v6:
- Rename ttm_backup_shmem_copy_page() function argument (Matthew Brost)
- Add some missing documentation

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/Makefile           |   2 +-
 drivers/gpu/drm/ttm/ttm_backup_shmem.c | 139 +++++++++++++++++++++++++
 include/drm/ttm/ttm_backup.h           | 137 ++++++++++++++++++++++++
 3 files changed, 277 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c
 create mode 100644 include/drm/ttm/ttm_backup.h
drivers/gpu/drm/ttm/Makefile +1 -1
index dad298127226..5e980dd90e41 100644
@@ -4,7 +4,7 @@
 
 ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \
 	ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \
-	ttm_device.o ttm_sys_manager.o
+	ttm_device.o ttm_sys_manager.o ttm_backup_shmem.o
 ttm-$(CONFIG_AGP) += ttm_agp_backend.o
 
 obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_backup_shmem.c new file +139 -0

139 lines not on this page · raw message

include/drm/ttm/ttm_backup.h new file +137 -0

137 lines not on this page · raw message

Signature · 1 line
2.44.0
maintainer Reviewed-by on v6 8/12
#18 lore raw

Re: [PATCH v6 08/12] drm/ttm: Add a virtual base class for graphics memory backup

Quoted 20 lines · Thomas Hellström

Initially intended for experimenting with different backup solutions (shmem vs direct swap cache insertion), abstract the backup destination using a virtual base class.

Also provide a sample implementation for shmem.

While when settling on a preferred backup solution, one could perhaps skip the abstraction, this functionality may actually come in handy for configurable dedicated graphics memory backup to fast nvme files or similar, whithout affecting swap-space. Could indeed be useful for VRAM backup on S4 and other cases.

v5:
- Fix a UAF. (kernel test robot, Dan Carptenter) v6:
- Rename ttm_backup_shmem_copy_page() function argument (Matthew Brost)
- Add some missing documentation

Quoted 3 lines
Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
Quoted 314 lines
---
 drivers/gpu/drm/ttm/Makefile           |   2 +-
 drivers/gpu/drm/ttm/ttm_backup_shmem.c | 139 +++++++++++++++++++++++++
 include/drm/ttm/ttm_backup.h           | 137 ++++++++++++++++++++++++
 3 files changed, 277 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c
 create mode 100644 include/drm/ttm/ttm_backup.h
drivers/gpu/drm/ttm/Makefile +1 -1
index dad298127226..5e980dd90e41 100644
@@ -4,7 +4,7 @@
 
 ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \
 	ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \
-	ttm_device.o ttm_sys_manager.o
+	ttm_device.o ttm_sys_manager.o ttm_backup_shmem.o
 ttm-$(CONFIG_AGP) += ttm_agp_backend.o
 
 obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_backup_shmem.c new file +139 -0
new file mode 100644
index 000000000000..3d23a34d9f34
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_backup.h>
+#include <linux/page-flags.h>
+
+/**
+ * struct ttm_backup_shmem - A shmem based ttm_backup subclass.
+ * @backup: The base struct ttm_backup
+ * @filp: The associated shmem object
+ */
+struct ttm_backup_shmem {
+	struct ttm_backup backup;
+	struct file *filp;
+};
+
+static struct ttm_backup_shmem *to_backup_shmem(struct ttm_backup *backup)
+{
+	return container_of(backup, struct ttm_backup_shmem, backup);
+}
+
+static void ttm_backup_shmem_drop(struct ttm_backup *backup, unsigned long handle)
+{
+	handle -= 1;
+	shmem_truncate_range(file_inode(to_backup_shmem(backup)->filp), handle,
+			     handle + 1);
+}
+
+static int ttm_backup_shmem_copy_page(struct ttm_backup *backup, struct page *dst,
+				      unsigned long handle, bool intr)
+{
+	struct file *filp = to_backup_shmem(backup)->filp;
+	struct address_space *mapping = filp->f_mapping;
+	struct folio *from_folio;
+
+	handle -= 1;
+	from_folio = shmem_read_folio(mapping, handle);
+	if (IS_ERR(from_folio))
+		return PTR_ERR(from_folio);
+
+	/* Note: Use drm_memcpy_from_wc? */
+	copy_highpage(dst, folio_file_page(from_folio, handle));
+	folio_put(from_folio);
+
+	return 0;
+}
+
+static unsigned long
+ttm_backup_shmem_backup_page(struct ttm_backup *backup, struct page *page,
+			     bool writeback, pgoff_t i, gfp_t page_gfp,
+			     gfp_t alloc_gfp)
+{
+	struct file *filp = to_backup_shmem(backup)->filp;
+	struct address_space *mapping = filp->f_mapping;
+	unsigned long handle = 0;
+	struct folio *to_folio;
+	int ret;
+
+	to_folio = shmem_read_folio_gfp(mapping, i, alloc_gfp);
+	if (IS_ERR(to_folio))
+		return handle;
+
+	folio_mark_accessed(to_folio);
+	folio_lock(to_folio);
+	folio_mark_dirty(to_folio);
+	copy_highpage(folio_file_page(to_folio, i), page);
+	handle = i + 1;
+
+	if (writeback && !folio_mapped(to_folio) && folio_clear_dirty_for_io(to_folio)) {
+		struct writeback_control wbc = {
+			.sync_mode = WB_SYNC_NONE,
+			.nr_to_write = SWAP_CLUSTER_MAX,
+			.range_start = 0,
+			.range_end = LLONG_MAX,
+			.for_reclaim = 1,
+		};
+		folio_set_reclaim(to_folio);
+		ret = mapping->a_ops->writepage(folio_page(to_folio, 0), &wbc);
+		if (!folio_test_writeback(to_folio))
+			folio_clear_reclaim(to_folio);
+		/* If writepage succeeds, it unlocks the folio */
+		if (ret)
+			folio_unlock(to_folio);
+	} else {
+		folio_unlock(to_folio);
+	}
+
+	folio_put(to_folio);
+
+	return handle;
+}
+
+static void ttm_backup_shmem_fini(struct ttm_backup *backup)
+{
+	struct ttm_backup_shmem *sbackup = to_backup_shmem(backup);
+
+	fput(sbackup->filp);
+	kfree(sbackup);
+}
+
+static const struct ttm_backup_ops ttm_backup_shmem_ops = {
+	.drop = ttm_backup_shmem_drop,
+	.copy_backed_up_page = ttm_backup_shmem_copy_page,
+	.backup_page = ttm_backup_shmem_backup_page,
+	.fini = ttm_backup_shmem_fini,
+};
+
+/**
+ * ttm_backup_shmem_create() - Create a shmem-based struct backup.
+ * @size: The maximum size (in bytes) to back up.
+ *
+ * Create a backup utilizing shmem objects.
+ *
+ * Return: A pointer to a struct ttm_backup on success,
+ * an error pointer on error.
+ */
+struct ttm_backup *ttm_backup_shmem_create(loff_t size)
+{
+	struct ttm_backup_shmem *sbackup =
+		kzalloc(sizeof(*sbackup), GFP_KERNEL | __GFP_ACCOUNT);
+	struct file *filp;
+
+	if (!sbackup)
+		return ERR_PTR(-ENOMEM);
+
+	filp = shmem_file_setup("ttm shmem backup", size, 0);
+	if (IS_ERR(filp)) {
+		kfree(sbackup);
+		return ERR_CAST(filp);
+	}
+
+	sbackup->filp = filp;
+	sbackup->backup.ops = &ttm_backup_shmem_ops;
+
+	return &sbackup->backup;
+}
+EXPORT_SYMBOL_GPL(ttm_backup_shmem_create);
include/drm/ttm/ttm_backup.h new file +137 -0
new file mode 100644
index 000000000000..5f8c7d3069ef
@@ -0,0 +1,137 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _TTM_BACKUP_H_
+#define _TTM_BACKUP_H_
+
+#include <linux/mm_types.h>
+#include <linux/shmem_fs.h>
+
+struct ttm_backup;
+
+/**
+ * ttm_backup_handle_to_page_ptr() - Convert handle to struct page pointer
+ * @handle: The handle to convert.
+ *
+ * Converts an opaque handle received from the
+ * struct ttm_backoup_ops::backup_page() function to an (invalid)
+ * struct page pointer suitable for a struct page array.
+ *
+ * Return: An (invalid) struct page pointer.
+ */
+static inline struct page *
+ttm_backup_handle_to_page_ptr(unsigned long handle)
+{
+	return (struct page *)(handle << 1 | 1);
+}
+
+/**
+ * ttm_backup_page_ptr_is_handle() - Whether a struct page pointer is a handle
+ * @page: The struct page pointer to check.
+ *
+ * Return: true if the struct page pointer is a handld returned from
+ * ttm_backup_handle_to_page_ptr(). False otherwise.
+ */
+static inline bool ttm_backup_page_ptr_is_handle(const struct page *page)
+{
+	return (unsigned long)page & 1;
+}
+
+/**
+ * ttm_backup_page_ptr_to_handle() - Convert a struct page pointer to a handle
+ * @page: The struct page pointer to convert
+ *
+ * Return: The handle that was previously used in
+ * ttm_backup_handle_to_page_ptr() to obtain a struct page pointer, suitable
+ * for use as argument in the struct ttm_backup_ops drop() or
+ * copy_backed_up_page() functions.
+ */
+static inline unsigned long
+ttm_backup_page_ptr_to_handle(const struct page *page)
+{
+	WARN_ON(!ttm_backup_page_ptr_is_handle(page));
+	return (unsigned long)page >> 1;
+}
+
+/** struct ttm_backup_ops - A struct ttm_backup backend operations */
+struct ttm_backup_ops {
+	/**
+	 * drop - release memory associated with a handle
+	 * @backup: The struct backup pointer used to obtain the handle
+	 * @handle: The handle obtained from the @backup_page function.
+	 */
+	void (*drop)(struct ttm_backup *backup, unsigned long handle);
+
+	/**
+	 * copy_backed_up_page - Copy the contents of a previously backed
+	 * up page
+	 * @backup: The struct backup pointer used to back up the page.
+	 * @dst: The struct page to copy into.
+	 * @handle: The handle returned when the page was backed up.
+	 * @intr: Try to perform waits interruptable or at least killable.
+	 *
+	 * Return: 0 on success, Negative error code on failure, notably
+	 * -EINTR if @intr was set to true and a signal is pending.
+	 */
+	int (*copy_backed_up_page)(struct ttm_backup *backup, struct page *dst,
+				   unsigned long handle, bool intr);
+
+	/**
+	 * backup_page - Backup a page
+	 * @backup: The struct backup pointer to use.
+	 * @page: The page to back up.
+	 * @writeback: Whether to perform immediate writeback of the page.
+	 * This may have performance implications.
+	 * @i: A unique integer for each page and each struct backup.
+	 * This is a hint allowing the backup backend to avoid managing
+	 * its address space separately.
+	 * @page_gfp: The gfp value used when the page was allocated.
+	 * This is used for accounting purposes.
+	 * @alloc_gfp: The gpf to be used when the backend needs to allocaete
+	 * memory.
+	 *
+	 * Return: A handle on success. 0 on failure.
+	 * (This is following the swp_entry_t convention).
+	 *
+	 * Note: This function could be extended to back up a folio and
+	 * backends would then split the folio internally if needed.
+	 * Drawback is that the caller would then have to keep track of
+	 * the folio size- and usage.
+	 */
+	unsigned long (*backup_page)(struct ttm_backup *backup, struct page *page,
+				     bool writeback, pgoff_t i, gfp_t page_gfp,
+				     gfp_t alloc_gfp);
+	/**
+	 * fini - Free the struct backup resources after last use.
+	 * @backup: Pointer to the struct backup whose resources to free.
+	 *
+	 * After a call to @fini, it's illegal to use the @backup pointer.
+	 */
+	void (*fini)(struct ttm_backup *backup);
+};
+
+/**
+ * struct ttm_backup - Abstract a backup backend.
+ * @ops: The operations as described above.
+ *
+ * The struct ttm_backup is intended to be subclassed by the
+ * backend implementation.
+ */
+struct ttm_backup {
+	const struct ttm_backup_ops *ops;
+};
+
+/**
+ * ttm_backup_shmem_create() - Create a shmem-based struct backup.
+ * @size: The maximum size (in bytes) to back up.
+ *
+ * Create a backup utilizing shmem objects.
+ *
+ * Return: A pointer to a struct ttm_backup on success,
+ * an error pointer on error.
+ */
+struct ttm_backup *ttm_backup_shmem_create(loff_t size);
+
+#endif
Signature · 1 line
2.44.0
maintainer
#19 lore raw

Re: [PATCH v6 08/12] drm/ttm: Add a virtual base class for graphics memory backup

In reply to Thomas Hellström

Am 03.07.24 um 17:38 schrieb Thomas Hellström:

Quoted 5 lines

Initially intended for experimenting with different backup solutions (shmem vs direct swap cache insertion), abstract the backup destination using a virtual base class.

Also provide a sample implementation for shmem.

Let's postpone this and all following patches and merge the LRU changes first.

Christian.

Quoted 329 lines

While when settling on a preferred backup solution, one could perhaps skip the abstraction, this functionality may actually come in handy for configurable dedicated graphics memory backup to fast nvme files or similar, whithout affecting swap-space. Could indeed be useful for VRAM backup on S4 and other cases.

v5:
- Fix a UAF. (kernel test robot, Dan Carptenter) v6:
- Rename ttm_backup_shmem_copy_page() function argument (Matthew Brost)
- Add some missing documentation

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
  drivers/gpu/drm/ttm/Makefile           |   2 +-
  drivers/gpu/drm/ttm/ttm_backup_shmem.c | 139 +++++++++++++++++++++++++
  include/drm/ttm/ttm_backup.h           | 137 ++++++++++++++++++++++++
  3 files changed, 277 insertions(+), 1 deletion(-)

create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c create mode 100644 include/drm/ttm/ttm_backup.h

drivers/gpu/drm/ttm/Makefile +1 -1
index dad298127226..5e980dd90e41 100644
@@ -4,7 +4,7 @@
 
  ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \
  	ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \
-	ttm_device.o ttm_sys_manager.o
+	ttm_device.o ttm_sys_manager.o ttm_backup_shmem.o
  ttm-$(CONFIG_AGP) += ttm_agp_backend.o
 
  obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_backup_shmem.c new file +139 -0
new file mode 100644
index 000000000000..3d23a34d9f34
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_backup.h>
+#include <linux/page-flags.h>
+
+/**
+ * struct ttm_backup_shmem - A shmem based ttm_backup subclass.
+ * @backup: The base struct ttm_backup
+ * @filp: The associated shmem object
+ */
+struct ttm_backup_shmem {
+	struct ttm_backup backup;
+	struct file *filp;
+};
+
+static struct ttm_backup_shmem *to_backup_shmem(struct ttm_backup *backup)
+{
+	return container_of(backup, struct ttm_backup_shmem, backup);
+}
+
+static void ttm_backup_shmem_drop(struct ttm_backup *backup, unsigned long handle)
+{
+	handle -= 1;
+	shmem_truncate_range(file_inode(to_backup_shmem(backup)->filp), handle,
+			     handle + 1);
+}
+
+static int ttm_backup_shmem_copy_page(struct ttm_backup *backup, struct page *dst,
+				      unsigned long handle, bool intr)
+{
+	struct file *filp = to_backup_shmem(backup)->filp;
+	struct address_space *mapping = filp->f_mapping;
+	struct folio *from_folio;
+
+	handle -= 1;
+	from_folio = shmem_read_folio(mapping, handle);
+	if (IS_ERR(from_folio))
+		return PTR_ERR(from_folio);
+
+	/* Note: Use drm_memcpy_from_wc? */
+	copy_highpage(dst, folio_file_page(from_folio, handle));
+	folio_put(from_folio);
+
+	return 0;
+}
+
+static unsigned long
+ttm_backup_shmem_backup_page(struct ttm_backup *backup, struct page *page,
+			     bool writeback, pgoff_t i, gfp_t page_gfp,
+			     gfp_t alloc_gfp)
+{
+	struct file *filp = to_backup_shmem(backup)->filp;
+	struct address_space *mapping = filp->f_mapping;
+	unsigned long handle = 0;
+	struct folio *to_folio;
+	int ret;
+
+	to_folio = shmem_read_folio_gfp(mapping, i, alloc_gfp);
+	if (IS_ERR(to_folio))
+		return handle;
+
+	folio_mark_accessed(to_folio);
+	folio_lock(to_folio);
+	folio_mark_dirty(to_folio);
+	copy_highpage(folio_file_page(to_folio, i), page);
+	handle = i + 1;
+
+	if (writeback && !folio_mapped(to_folio) && folio_clear_dirty_for_io(to_folio)) {
+		struct writeback_control wbc = {
+			.sync_mode = WB_SYNC_NONE,
+			.nr_to_write = SWAP_CLUSTER_MAX,
+			.range_start = 0,
+			.range_end = LLONG_MAX,
+			.for_reclaim = 1,
+		};
+		folio_set_reclaim(to_folio);
+		ret = mapping->a_ops->writepage(folio_page(to_folio, 0), &wbc);
+		if (!folio_test_writeback(to_folio))
+			folio_clear_reclaim(to_folio);
+		/* If writepage succeeds, it unlocks the folio */
+		if (ret)
+			folio_unlock(to_folio);
+	} else {
+		folio_unlock(to_folio);
+	}
+
+	folio_put(to_folio);
+
+	return handle;
+}
+
+static void ttm_backup_shmem_fini(struct ttm_backup *backup)
+{
+	struct ttm_backup_shmem *sbackup = to_backup_shmem(backup);
+
+	fput(sbackup->filp);
+	kfree(sbackup);
+}
+
+static const struct ttm_backup_ops ttm_backup_shmem_ops = {
+	.drop = ttm_backup_shmem_drop,
+	.copy_backed_up_page = ttm_backup_shmem_copy_page,
+	.backup_page = ttm_backup_shmem_backup_page,
+	.fini = ttm_backup_shmem_fini,
+};
+
+/**
+ * ttm_backup_shmem_create() - Create a shmem-based struct backup.
+ * @size: The maximum size (in bytes) to back up.
+ *
+ * Create a backup utilizing shmem objects.
+ *
+ * Return: A pointer to a struct ttm_backup on success,
+ * an error pointer on error.
+ */
+struct ttm_backup *ttm_backup_shmem_create(loff_t size)
+{
+	struct ttm_backup_shmem *sbackup =
+		kzalloc(sizeof(*sbackup), GFP_KERNEL | __GFP_ACCOUNT);
+	struct file *filp;
+
+	if (!sbackup)
+		return ERR_PTR(-ENOMEM);
+
+	filp = shmem_file_setup("ttm shmem backup", size, 0);
+	if (IS_ERR(filp)) {
+		kfree(sbackup);
+		return ERR_CAST(filp);
+	}
+
+	sbackup->filp = filp;
+	sbackup->backup.ops = &ttm_backup_shmem_ops;
+
+	return &sbackup->backup;
+}
+EXPORT_SYMBOL_GPL(ttm_backup_shmem_create);
include/drm/ttm/ttm_backup.h new file +137 -0
new file mode 100644
index 000000000000..5f8c7d3069ef
@@ -0,0 +1,137 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _TTM_BACKUP_H_
+#define _TTM_BACKUP_H_
+
+#include <linux/mm_types.h>
+#include <linux/shmem_fs.h>
+
+struct ttm_backup;
+
+/**
+ * ttm_backup_handle_to_page_ptr() - Convert handle to struct page pointer
+ * @handle: The handle to convert.
+ *
+ * Converts an opaque handle received from the
+ * struct ttm_backoup_ops::backup_page() function to an (invalid)
+ * struct page pointer suitable for a struct page array.
+ *
+ * Return: An (invalid) struct page pointer.
+ */
+static inline struct page *
+ttm_backup_handle_to_page_ptr(unsigned long handle)
+{
+	return (struct page *)(handle << 1 | 1);
+}
+
+/**
+ * ttm_backup_page_ptr_is_handle() - Whether a struct page pointer is a handle
+ * @page: The struct page pointer to check.
+ *
+ * Return: true if the struct page pointer is a handld returned from
+ * ttm_backup_handle_to_page_ptr(). False otherwise.
+ */
+static inline bool ttm_backup_page_ptr_is_handle(const struct page *page)
+{
+	return (unsigned long)page & 1;
+}
+
+/**
+ * ttm_backup_page_ptr_to_handle() - Convert a struct page pointer to a handle
+ * @page: The struct page pointer to convert
+ *
+ * Return: The handle that was previously used in
+ * ttm_backup_handle_to_page_ptr() to obtain a struct page pointer, suitable
+ * for use as argument in the struct ttm_backup_ops drop() or
+ * copy_backed_up_page() functions.
+ */
+static inline unsigned long
+ttm_backup_page_ptr_to_handle(const struct page *page)
+{
+	WARN_ON(!ttm_backup_page_ptr_is_handle(page));
+	return (unsigned long)page >> 1;
+}
+
+/** struct ttm_backup_ops - A struct ttm_backup backend operations */
+struct ttm_backup_ops {
+	/**
+	 * drop - release memory associated with a handle
+	 * @backup: The struct backup pointer used to obtain the handle
+	 * @handle: The handle obtained from the @backup_page function.
+	 */
+	void (*drop)(struct ttm_backup *backup, unsigned long handle);
+
+	/**
+	 * copy_backed_up_page - Copy the contents of a previously backed
+	 * up page
+	 * @backup: The struct backup pointer used to back up the page.
+	 * @dst: The struct page to copy into.
+	 * @handle: The handle returned when the page was backed up.
+	 * @intr: Try to perform waits interruptable or at least killable.
+	 *
+	 * Return: 0 on success, Negative error code on failure, notably
+	 * -EINTR if @intr was set to true and a signal is pending.
+	 */
+	int (*copy_backed_up_page)(struct ttm_backup *backup, struct page *dst,
+				   unsigned long handle, bool intr);
+
+	/**
+	 * backup_page - Backup a page
+	 * @backup: The struct backup pointer to use.
+	 * @page: The page to back up.
+	 * @writeback: Whether to perform immediate writeback of the page.
+	 * This may have performance implications.
+	 * @i: A unique integer for each page and each struct backup.
+	 * This is a hint allowing the backup backend to avoid managing
+	 * its address space separately.
+	 * @page_gfp: The gfp value used when the page was allocated.
+	 * This is used for accounting purposes.
+	 * @alloc_gfp: The gpf to be used when the backend needs to allocaete
+	 * memory.
+	 *
+	 * Return: A handle on success. 0 on failure.
+	 * (This is following the swp_entry_t convention).
+	 *
+	 * Note: This function could be extended to back up a folio and
+	 * backends would then split the folio internally if needed.
+	 * Drawback is that the caller would then have to keep track of
+	 * the folio size- and usage.
+	 */
+	unsigned long (*backup_page)(struct ttm_backup *backup, struct page *page,
+				     bool writeback, pgoff_t i, gfp_t page_gfp,
+				     gfp_t alloc_gfp);
+	/**
+	 * fini - Free the struct backup resources after last use.
+	 * @backup: Pointer to the struct backup whose resources to free.
+	 *
+	 * After a call to @fini, it's illegal to use the @backup pointer.
+	 */
+	void (*fini)(struct ttm_backup *backup);
+};
+
+/**
+ * struct ttm_backup - Abstract a backup backend.
+ * @ops: The operations as described above.
+ *
+ * The struct ttm_backup is intended to be subclassed by the
+ * backend implementation.
+ */
+struct ttm_backup {
+	const struct ttm_backup_ops *ops;
+};
+
+/**
+ * ttm_backup_shmem_create() - Create a shmem-based struct backup.
+ * @size: The maximum size (in bytes) to back up.
+ *
+ * Create a backup utilizing shmem objects.
+ *
+ * Return: A pointer to a struct ttm_backup on success,
+ * an error pointer on error.
+ */
+struct ttm_backup *ttm_backup_shmem_create(loff_t size);
+
+#endif
author maintainer
#20 lore raw

PATCH v6 9/12 drm/ttm/pool: Provide a helper to shrink pages

In reply to Thomas Hellström

Provide a helper to shrink ttm_tt page-vectors on a per-page basis. A ttm_backup backend could then in theory get away with allocating a single temporary page for each struct ttm_tt.

This is accomplished by splitting larger pages before trying to back them up.

In the future we could allow ttm_backup to handle backing up large pages as well, but currently there's no benefit in doing that, since the shmem backup backend would have to split those anyway to avoid allocating too much temporary memory, and if the backend instead inserts pages into the swap-cache, those are split on reclaim by the core.

Due to potential backup- and recover errors, allow partially swapped out struct ttm_tt's, although mark them as swapped out stopping them from being swapped out a second time. More details in the ttm_pool.c DOC section.

v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 397 +++++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c   |  37 +++
 include/drm/ttm/ttm_pool.h     |   5 +
 include/drm/ttm/ttm_tt.h       |  20 ++
 4 files changed, 446 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +384 -13
index 6e1fd6985ffc..38e50cf81b0a 100644
@@ -41,6 +41,7 @@
 #include <asm/set_memory.h>
 #endif
 
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <drm/ttm/ttm_bo.h>
@@ -58,6 +59,32 @@ struct ttm_pool_dma {
 	unsigned long vaddr;
 };
 
+/**
+ * struct ttm_pool_tt_restore - State representing restore from backup
+ * @alloced_pages: Total number of already allocated pages for the ttm_tt.
+ * @restored_pages: Number of (sub) pages restored from swap for this
+ *		     chunk of 1 << @order pages.
+ * @first_page: The ttm page ptr representing for @old_pages[0].
+ * @caching_divide: Page pointer where subsequent pages are cached.
+ * @old_pages: Backup copy of page pointers that were replaced by the new
+ *	       page allocation.
+ * @pool: The pool used for page allocation while restoring.
+ * @order: The order of the last page allocated while restoring.
+ *
+ * Recovery from backup might fail when we've recovered less than the
+ * full ttm_tt. In order not to loose any data (yet), keep information
+ * around that allows us to restart a failed ttm backup recovery.
+ */
+struct ttm_pool_tt_restore {
+	pgoff_t alloced_pages;
+	pgoff_t restored_pages;
+	struct page **first_page;
+	struct page **caching_divide;
+	struct ttm_pool *pool;
+	unsigned int order;
+	struct page *old_pages[];
+};
+
 static unsigned long page_pool_size;
 
 MODULE_PARM_DESC(page_pool_size, "Number of pages in the WC/UC/DMA pool");
@@ -354,11 +381,102 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 	return p->private;
 }
 
+/*
+ * To be able to insert single pages into backup directly,
+ * we need to split multi-order page allocations and make them look
+ * like single-page allocations.
+ */
+static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct page *p)
+{
+	unsigned int order = ttm_pool_page_order(pool, p);
+	pgoff_t nr;
+
+	if (!order)
+		return;
+
+	split_page(p, order);
+	nr = 1UL << order;
+	while (nr--)
+		(p++)->private = 0;
+}
+
+/**
+ * DOC: Partial backup and restoration of a struct ttm_tt.
+ *
+ * Swapout using ttm_backup::ops::backup_page() and swapin using
+ * ttm_backup::ops::copy_backed_up_page() may fail.
+ * The former most likely due to lack of swap-space or memory, the latter due
+ * to lack of memory or because of signal interruption during waits.
+ *
+ * Backupfailure is easily handled by using a ttm_tt pages vector that holds
+ * both swap entries and page pointers. This has to be taken into account when
+ * restoring such a ttm_tt from backup, and when freeing it while backed up.
+ * When restoring, for simplicity, new pages are actually allocated from the
+ * pool and the contents of any old pages are copied in and then the old pages
+ * are released.
+ *
+ * For restoration failures, the struct ttm_pool_tt_restore holds sufficient state
+ * to be able to resume an interrupted restore, and that structure is freed once
+ * the restoration is complete. If the struct ttm_tt is destroyed while there
+ * is a valid struct ttm_pool_tt_restore attached, that is also properly taken
+ * care of.
+ */
+
+static bool ttm_pool_restore_valid(const struct ttm_pool_tt_restore *restore)
+{
+	return restore && restore->restored_pages < (1 << restore->order);
+}
+
+static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
+			       struct ttm_backup *backup,
+			       struct ttm_operation_ctx *ctx)
+{
+	unsigned int i, nr = 1 << restore->order;
+	int ret = 0;
+
+	if (!ttm_pool_restore_valid(restore))
+		return 0;
+
+	for (i = restore->restored_pages; i < nr; ++i) {
+		struct page *p = restore->old_pages[i];
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
+
+			if (handle == 0)
+				continue;
+
+			ret = backup->ops->copy_backed_up_page
+				(backup, restore->first_page[i],
+				 handle, ctx->interruptible);
+			if (ret)
+				break;
+
+			backup->ops->drop(backup, handle);
+		} else if (p) {
+			/*
+			 * We could probably avoid splitting the old page
+			 * using clever logic, but ATM we don't care.
+			 */
+			ttm_pool_split_for_swap(restore->pool, p);
+			copy_highpage(restore->first_page[i], p);
+			__free_pages(p, 0);
+		}
+
+		restore->restored_pages++;
+		restore->old_pages[i] = NULL;
+		cond_resched();
+	}
+
+	return ret;
+}
+
 /* Called when we got a page, either from a pool or newly allocated */
 static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 				   struct page *p, dma_addr_t **dma_addr,
 				   unsigned long *num_pages,
-				   struct page ***pages)
+				   struct page ***pages,
+				   struct ttm_pool_tt_restore *restore)
 {
 	unsigned int i;
 	int r;
@@ -369,6 +487,16 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 			return r;
 	}
 
+	if (restore) {
+		memcpy(restore->old_pages, *pages,
+		       (1 << order) * sizeof(*restore->old_pages));
+		memset(*pages, 0, (1 << order) * sizeof(**pages));
+		restore->order = order;
+		restore->restored_pages = 0;
+		restore->first_page = *pages;
+		restore->alloced_pages += 1UL << order;
+	}
+
 	*num_pages -= 1 << order;
 	for (i = 1 << order; i; --i, ++(*pages), ++p)
 		**pages = p;
@@ -394,22 +522,39 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt,
 				pgoff_t start_page, pgoff_t end_page)
 {
 	struct page **pages = &tt->pages[start_page];
+	struct ttm_backup *backup = tt->backup;
 	unsigned int order;
 	pgoff_t i, nr;
 
 	for (i = start_page; i < end_page; i += nr, pages += nr) {
 		struct ttm_pool_type *pt = NULL;
+		struct page *p = *pages;
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
+
+			nr = 1;
+			if (handle != 0)
+				backup->ops->drop(backup, handle);
+			continue;
+		}
+
+		if (pool) {
+			order = ttm_pool_page_order(pool, p);
+			nr = (1UL << order);
+			if (tt->dma_address)
+				ttm_pool_unmap(pool, tt->dma_address[i], nr);
 
-		order = ttm_pool_page_order(pool, *pages);
-		nr = (1UL << order);
-		if (tt->dma_address)
-			ttm_pool_unmap(pool, tt->dma_address[i], nr);
+			pt = ttm_pool_select_type(pool, caching, order);
+		} else {
+			order = p->private;
+			nr = (1UL << order);
+		}
 
-		pt = ttm_pool_select_type(pool, caching, order);
 		if (pt)
-			ttm_pool_type_give(pt, *pages);
+			ttm_pool_type_give(pt, p);
 		else
-			ttm_pool_free_page(pool, caching, order, *pages);
+			ttm_pool_free_page(pool, caching, order, p);
 	}
 }
 
@@ -453,9 +598,37 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	for (order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages));
-	     num_pages;
-	     order = min_t(unsigned int, order, __fls(num_pages))) {
+	order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages));
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {
+		if (!tt->restore) {
+			gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
+
+			if (ctx->gfp_retry_mayfail)
+				gfp |= __GFP_RETRY_MAYFAIL;
+
+			tt->restore =
+				kvzalloc(struct_size(tt->restore, old_pages,
+						     (size_t)1 << order), gfp);
+			/* RFC: Possibly loop on -ENOMEM and reduce order. */
+			if (!tt->restore)
+				return -ENOMEM;
+		} else if (ttm_pool_restore_valid(tt->restore)) {
+			struct ttm_pool_tt_restore *restore = tt->restore;
+
+			num_pages -= restore->alloced_pages;
+			order = min_t(unsigned int, order, __fls(num_pages));
+			pages += restore->alloced_pages;
+			r = ttm_pool_restore_tt(restore, tt->backup, ctx);
+			if (r)
+				return r;
+			caching = restore->caching_divide;
+		}
+
+		tt->restore->pool = pool;
+	}
+
+	for (; num_pages; order = min_t(unsigned int, order, __fls(num_pages))) {
 		struct ttm_pool_type *pt;
 
 		page_caching = tt->caching;
@@ -472,11 +645,19 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				r = ttm_pool_page_allocated(pool, order, p,
 							    &dma_addr,
 							    &num_pages,
-							    &pages);
+							    &pages,
+							    tt->restore);
 				if (r)
 					goto error_free_page;
 
 				caching = pages;
+				if (ttm_pool_restore_valid(tt->restore)) {
+					r = ttm_pool_restore_tt(tt->restore, tt->backup,
+								ctx);
+					if (r)
+						goto error_free_all;
+				}
+
 				if (num_pages < (1 << order))
 					break;
 
@@ -496,9 +677,17 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				caching = pages;
 			}
 			r = ttm_pool_page_allocated(pool, order, p, &dma_addr,
-						    &num_pages, &pages);
+						    &num_pages, &pages,
+						    tt->restore);
 			if (r)
 				goto error_free_page;
+
+			if (ttm_pool_restore_valid(tt->restore)) {
+				r = ttm_pool_restore_tt(tt->restore, tt->backup, ctx);
+				if (r)
+					goto error_free_all;
+			}
+
 			if (PageHighMem(p))
 				caching = pages;
 		}
@@ -517,12 +706,26 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (r)
 		goto error_free_all;
 
+	if (tt->restore) {
+		kvfree(tt->restore);
+		tt->restore = NULL;
+	}
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP)
+		tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+
 	return 0;
 
 error_free_page:
 	ttm_pool_free_page(pool, page_caching, order, p);
 
 error_free_all:
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {
+		tt->restore->caching_divide = caching;
+		return r;
+	}
+
 	num_pages = tt->num_pages - num_pages;
 	caching_divide = caching - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
@@ -549,6 +752,174 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)
 }
 EXPORT_SYMBOL(ttm_pool_free);
 
+/**
+ * ttm_pool_release_backed_up() - Release content of a swapped-out struct ttm_tt
+ * @tt: The struct ttm_tt.
+ *
+ * Release handles with associated content or any remaining pages of
+ * a backed-up struct ttm_tt.
+ */
+void ttm_pool_release_backed_up(struct ttm_tt *tt)
+{
+	struct ttm_backup *backup = tt->backup;
+	struct ttm_pool_tt_restore *restore;
+	pgoff_t i, start_page = 0;
+	unsigned long handle;
+
+	if (!(tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return;
+
+	restore = tt->restore;
+
+	if (ttm_pool_restore_valid(restore)) {
+		pgoff_t nr = 1UL << restore->order;
+
+		for (i = restore->restored_pages; i < nr; ++i) {
+			struct page *p = restore->old_pages[i];
+
+			if (ttm_backup_page_ptr_is_handle(p)) {
+				handle = ttm_backup_page_ptr_to_handle(p);
+				if (handle == 0)
+					continue;
+
+				backup->ops->drop(backup, handle);
+			} else if (p) {
+				ttm_pool_split_for_swap(restore->pool, p);
+				__free_pages(p, 0);
+			}
+		}
+	}
+
+	if (restore) {
+		pgoff_t mid = restore->caching_divide - tt->pages;
+
+		start_page = restore->alloced_pages;
+		/* Pages that might be dma-mapped and non-cached */
+		ttm_pool_free_range(restore->pool, tt, tt->caching,
+				    0, mid);
+		/* Pages that might be dma-mapped but cached */
+		ttm_pool_free_range(restore->pool, tt, ttm_cached,
+				    mid, restore->alloced_pages);
+	}
+
+	/* Shrunken pages. Cached and not dma-mapped. */
+	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt->num_pages);
+
+	if (restore) {
+		kvfree(restore);
+		tt->restore = NULL;
+	}
+
+	tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP | TTM_TT_FLAG_SWAPPED);
+}
+
+/**
+ * ttm_pool_backup_tt() - Back up or purge a struct ttm_tt
+ * @pool: The pool used when allocating the struct ttm_tt.
+ * @ttm: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system.
+ * @writeback: If !@purge, Try to write out directly to the
+ * underlying persistent media.
+ *
+ * Back up or purge a struct ttm_tt. If @purge is true, then
+ * all pages will be freed directly to the system rather than to the pool
+ * they were allocated from, making the function behave similarly to
+ * ttm_pool_free(). If @purge is false the pages will be backed up instead,
+ * exchanged for handles.
+ * A subsequent call to ttm_pool_alloc() will then read back the content and
+ * a subsequent call to ttm_pool_release_shrunken() will drop it.
+ * If backup of a page fails for whatever reason, @ttm will still be
+ * partially backed up, retaining those pages for which backup fails.
+ *
+ * Return: Number of pages actually backed up or freed, or negative
+ * error code on error.
+ */
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge,
+			bool writeback)
+{
+	struct ttm_backup *backup = ttm->backup;
+	struct page *page;
+	unsigned long handle;
+	gfp_t alloc_gfp;
+	gfp_t gfp;
+	int ret = 0;
+	pgoff_t shrunken = 0;
+	pgoff_t i, num_pages;
+
+	if ((!get_nr_swap_pages() && !purge) ||
+	    pool->use_dma_alloc ||
+	    (ttm->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return -EBUSY;
+
+#ifdef CONFIG_X86
+	/* Anything returned to the system needs to be cached. */
+	if (ttm->caching != ttm_cached)
+		set_pages_array_wb(ttm->pages, ttm->num_pages);
+#endif
+
+	if (ttm->dma_address || purge) {
+		for (i = 0; i < ttm->num_pages; i += num_pages) {
+			unsigned int order;
+
+			page = ttm->pages[i];
+			if (unlikely(!page)) {
+				num_pages = 1;
+				continue;
+			}
+
+			order = ttm_pool_page_order(pool, page);
+			num_pages = 1UL << order;
+			if (ttm->dma_address)
+				ttm_pool_unmap(pool, ttm->dma_address[i],
+					       num_pages);
+			if (purge) {
+				shrunken += num_pages;
+				page->private = 0;
+				__free_pages(page, order);
+				memset(ttm->pages + i, 0,
+				       num_pages * sizeof(*ttm->pages));
+			}
+		}
+	}
+
+	if (purge)
+		return shrunken;
+
+	if (pool->use_dma32)
+		gfp = GFP_DMA32;
+	else
+		gfp = GFP_HIGHUSER;
+
+	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
+
+	for (i = 0; i < ttm->num_pages; ++i) {
+		page = ttm->pages[i];
+		if (unlikely(!page))
+			continue;
+
+		ttm_pool_split_for_swap(pool, page);
+
+		handle = backup->ops->backup_page(backup, page, writeback, i,
+						  gfp, alloc_gfp);
+		if (handle) {
+			ttm->pages[i] = ttm_backup_handle_to_page_ptr(handle);
+			put_page(page);
+			shrunken++;
+		} else {
+			/* We allow partially shrunken tts */
+			ret = -ENOMEM;
+			break;
+		}
+		cond_resched();
+	}
+
+	if (shrunken)
+		ttm->page_flags |= (TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+
+	return shrunken ? shrunken : ret;
+}
+
 /**
  * ttm_pool_init - Initialize a pool
  *
drivers/gpu/drm/ttm/ttm_tt.c +37 -0
index 4b51b9023126..98ce25197b38 100644
@@ -40,6 +40,7 @@
 #include <drm/drm_cache.h>
 #include <drm/drm_device.h>
 #include <drm/drm_util.h>
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_bo.h>
 #include <drm/ttm/ttm_tt.h>
 
@@ -158,6 +159,7 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm,
 	ttm->swap_storage = NULL;
 	ttm->sg = bo->sg;
 	ttm->caching = caching;
+	ttm->restore = NULL;
 }
 
 int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
@@ -182,6 +184,12 @@ void ttm_tt_fini(struct ttm_tt *ttm)
 		fput(ttm->swap_storage);
 	ttm->swap_storage = NULL;
 
+	ttm_pool_release_backed_up(ttm);
+	if (ttm->backup) {
+		ttm->backup->ops->fini(ttm->backup);
+		ttm->backup = NULL;
+	}
+
 	if (ttm->pages)
 		kvfree(ttm->pages);
 	else
@@ -253,6 +261,35 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
 
+/**
+ * ttm_tt_backup() - Helper to back up a struct ttm_tt.
+ * @bdev: The TTM device.
+ * @tt: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system,
+ * bypassing any pooling.
+ * @writeback: If !@purge, try to write out directly to the
+ * underlying persistent media.
+ *
+ * Helper for a TTM driver to use from the bo_shrink() method to shrink
+ * a struct ttm_tt, after it has done the necessary unbinding. This function
+ * will update the page accounting and call ttm_pool_shrink_tt to free pages
+ * or move them to the swap cache.
+ *
+ * Return: Number of pages freed or swapped out, or negative error code on
+ * error.
+ */
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt, bool purge,
+		   bool writeback)
+{
+	long ret = ttm_pool_backup_tt(&bdev->pool, tt, purge, writeback);
+
+	if (ret > 0)
+		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
+
+	return ret;
+}
+EXPORT_SYMBOL(ttm_tt_backup);
+
 /**
  * ttm_tt_swapout - swap out tt object
  *
include/drm/ttm/ttm_pool.h +5 -0
index 160d954a261e..4e4db369952b 100644
@@ -89,6 +89,11 @@ void ttm_pool_fini(struct ttm_pool *pool);
 
 int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);
 
+void ttm_pool_release_backed_up(struct ttm_tt *tt);
+
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm,
+			bool purge, bool writeback);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
include/drm/ttm/ttm_tt.h +20 -0
index 2b9d856ff388..6b990f1e7dd0 100644
@@ -32,11 +32,13 @@
 #include <drm/ttm/ttm_caching.h>
 #include <drm/ttm/ttm_kmap_iter.h>
 
+struct ttm_backup;
 struct ttm_device;
 struct ttm_tt;
 struct ttm_resource;
 struct ttm_buffer_object;
 struct ttm_operation_ctx;
+struct ttm_pool_tt_restore;
 
 /**
  * struct ttm_tt - This is a structure holding the pages, caching- and aperture
@@ -85,6 +87,9 @@ struct ttm_tt {
 	 * fault handling abuses the DMA api a bit and dma_map_attrs can't be
 	 * used to assure pgprot always matches.
 	 *
+	 * TTM_TT_FLAG_PRIV_BACKED_UP: TTM internal only. This is set if the
+	 * struct ttm_tt has been (possibly partially) backed up.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -96,6 +101,7 @@ struct ttm_tt {
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 
 #define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
+#define TTM_TT_FLAG_PRIV_BACKED_UP	BIT(6)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -105,11 +111,21 @@ struct ttm_tt {
 	dma_addr_t *dma_address;
 	/** @swap_storage: Pointer to shmem struct file for swap storage. */
 	struct file *swap_storage;
+	/**
+	 * @backup: Pointer to backup struct for backed up tts.
+	 * RFC: Could possibly be unified with @swap_storage.
+	 */
+	struct ttm_backup *backup;
 	/**
 	 * @caching: The current caching state of the pages, see enum
 	 * ttm_caching.
 	 */
 	enum ttm_caching caching;
+	/**
+	 * @restore: Partial restoration from backup state.
+	 * RFC: Incorporate in struct ttm_backup?
+	 */
+	struct ttm_pool_tt_restore *restore;
 };
 
 /**
@@ -230,6 +246,10 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 					    struct ttm_tt *tt);
 unsigned long ttm_tt_pages_limit(void);
+
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt, bool purge,
+		   bool writeback);
+
 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>
 
Signature · 1 line
2.44.0
maintainer
#21 lore raw

Re: [PATCH v6 09/12] drm/ttm/pool: Provide a helper to shrink pages

Quoted 259 lines · Thomas Hellström

Provide a helper to shrink ttm_tt page-vectors on a per-page basis. A ttm_backup backend could then in theory get away with allocating a single temporary page for each struct ttm_tt.

This is accomplished by splitting larger pages before trying to back them up.

In the future we could allow ttm_backup to handle backing up large pages as well, but currently there's no benefit in doing that, since the shmem backup backend would have to split those anyway to avoid allocating too much temporary memory, and if the backend instead inserts pages into the swap-cache, those are split on reclaim by the core.

Due to potential backup- and recover errors, allow partially swapped out struct ttm_tt's, although mark them as swapped out stopping them from being swapped out a second time. More details in the ttm_pool.c DOC section.

v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 397 +++++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c   |  37 +++
 include/drm/ttm/ttm_pool.h     |   5 +
 include/drm/ttm/ttm_tt.h       |  20 ++
 4 files changed, 446 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +156 -11
index 6e1fd6985ffc..38e50cf81b0a 100644
@@ -41,6 +41,7 @@
 #include <asm/set_memory.h>
 #endif
 
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <drm/ttm/ttm_bo.h>
@@ -58,6 +59,32 @@ struct ttm_pool_dma {
 	unsigned long vaddr;
 };
 
+/**
+ * struct ttm_pool_tt_restore - State representing restore from backup
+ * @alloced_pages: Total number of already allocated pages for the ttm_tt.
+ * @restored_pages: Number of (sub) pages restored from swap for this
+ *		     chunk of 1 << @order pages.
+ * @first_page: The ttm page ptr representing for @old_pages[0].
+ * @caching_divide: Page pointer where subsequent pages are cached.
+ * @old_pages: Backup copy of page pointers that were replaced by the new
+ *	       page allocation.
+ * @pool: The pool used for page allocation while restoring.
+ * @order: The order of the last page allocated while restoring.
+ *
+ * Recovery from backup might fail when we've recovered less than the
+ * full ttm_tt. In order not to loose any data (yet), keep information
+ * around that allows us to restart a failed ttm backup recovery.
+ */
+struct ttm_pool_tt_restore {
+	pgoff_t alloced_pages;
+	pgoff_t restored_pages;
+	struct page **first_page;
+	struct page **caching_divide;
+	struct ttm_pool *pool;
+	unsigned int order;
+	struct page *old_pages[];
+};
+
 static unsigned long page_pool_size;
 
 MODULE_PARM_DESC(page_pool_size, "Number of pages in the WC/UC/DMA pool");
@@ -354,11 +381,102 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 	return p->private;
 }
 
+/*
+ * To be able to insert single pages into backup directly,
+ * we need to split multi-order page allocations and make them look
+ * like single-page allocations.
+ */
+static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct page *p)
+{
+	unsigned int order = ttm_pool_page_order(pool, p);
+	pgoff_t nr;
+
+	if (!order)
+		return;
+
+	split_page(p, order);
+	nr = 1UL << order;
+	while (nr--)
+		(p++)->private = 0;
+}
+
+/**
+ * DOC: Partial backup and restoration of a struct ttm_tt.
+ *
+ * Swapout using ttm_backup::ops::backup_page() and swapin using
+ * ttm_backup::ops::copy_backed_up_page() may fail.
+ * The former most likely due to lack of swap-space or memory, the latter due
+ * to lack of memory or because of signal interruption during waits.
+ *
+ * Backupfailure is easily handled by using a ttm_tt pages vector that holds
+ * both swap entries and page pointers. This has to be taken into account when
+ * restoring such a ttm_tt from backup, and when freeing it while backed up.
+ * When restoring, for simplicity, new pages are actually allocated from the
+ * pool and the contents of any old pages are copied in and then the old pages
+ * are released.
+ *
+ * For restoration failures, the struct ttm_pool_tt_restore holds sufficient state
+ * to be able to resume an interrupted restore, and that structure is freed once
+ * the restoration is complete. If the struct ttm_tt is destroyed while there
+ * is a valid struct ttm_pool_tt_restore attached, that is also properly taken
+ * care of.
+ */
+
+static bool ttm_pool_restore_valid(const struct ttm_pool_tt_restore *restore)
+{
+	return restore && restore->restored_pages < (1 << restore->order);
+}
+
+static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
+			       struct ttm_backup *backup,
+			       struct ttm_operation_ctx *ctx)
+{
+	unsigned int i, nr = 1 << restore->order;
+	int ret = 0;
+
+	if (!ttm_pool_restore_valid(restore))
+		return 0;
+
+	for (i = restore->restored_pages; i < nr; ++i) {
+		struct page *p = restore->old_pages[i];
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
+
+			if (handle == 0)
+				continue;
+
+			ret = backup->ops->copy_backed_up_page
+				(backup, restore->first_page[i],
+				 handle, ctx->interruptible);
+			if (ret)
+				break;
+
+			backup->ops->drop(backup, handle);
+		} else if (p) {
+			/*
+			 * We could probably avoid splitting the old page
+			 * using clever logic, but ATM we don't care.
+			 */
+			ttm_pool_split_for_swap(restore->pool, p);
+			copy_highpage(restore->first_page[i], p);
+			__free_pages(p, 0);
+		}
+
+		restore->restored_pages++;
+		restore->old_pages[i] = NULL;
+		cond_resched();
+	}
+
+	return ret;
+}
+
 /* Called when we got a page, either from a pool or newly allocated */
 static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 				   struct page *p, dma_addr_t **dma_addr,
 				   unsigned long *num_pages,
-				   struct page ***pages)
+				   struct page ***pages,
+				   struct ttm_pool_tt_restore *restore)
 {
 	unsigned int i;
 	int r;
@@ -369,6 +487,16 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
 			return r;
 	}
 
+	if (restore) {
+		memcpy(restore->old_pages, *pages,
+		       (1 << order) * sizeof(*restore->old_pages));
+		memset(*pages, 0, (1 << order) * sizeof(**pages));
+		restore->order = order;
+		restore->restored_pages = 0;
+		restore->first_page = *pages;
+		restore->alloced_pages += 1UL << order;
+	}
+
 	*num_pages -= 1 << order;
 	for (i = 1 << order; i; --i, ++(*pages), ++p)
 		**pages = p;
@@ -394,22 +522,39 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt,
 				pgoff_t start_page, pgoff_t end_page)
 {
 	struct page **pages = &tt->pages[start_page];
+	struct ttm_backup *backup = tt->backup;
 	unsigned int order;
 	pgoff_t i, nr;
 
 	for (i = start_page; i < end_page; i += nr, pages += nr) {
 		struct ttm_pool_type *pt = NULL;
+		struct page *p = *pages;
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle = ttm_backup_page_ptr_to_handle(p);
+
+			nr = 1;
+			if (handle != 0)
+				backup->ops->drop(backup, handle);
+			continue;
+		}
+
+		if (pool) {
+			order = ttm_pool_page_order(pool, p);
+			nr = (1UL << order);
+			if (tt->dma_address)
+				ttm_pool_unmap(pool, tt->dma_address[i], nr);
 
-		order = ttm_pool_page_order(pool, *pages);
-		nr = (1UL << order);
-		if (tt->dma_address)
-			ttm_pool_unmap(pool, tt->dma_address[i], nr);
+			pt = ttm_pool_select_type(pool, caching, order);
+		} else {
+			order = p->private;
+			nr = (1UL << order);
+		}
 
-		pt = ttm_pool_select_type(pool, caching, order);
 		if (pt)
-			ttm_pool_type_give(pt, *pages);
+			ttm_pool_type_give(pt, p);
 		else
-			ttm_pool_free_page(pool, caching, order, *pages);
+			ttm_pool_free_page(pool, caching, order, p);
 	}
 }
 
@@ -453,9 +598,37 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	for (order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages));
-	     num_pages;
-	     order = min_t(unsigned int, order, __fls(num_pages))) {
+	order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages));
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 613

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+		if (!tt->restore) {
+			gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
+
+			if (ctx->gfp_retry_mayfail)
+				gfp |= __GFP_RETRY_MAYFAIL;
+
+			tt->restore =
+				kvzalloc(struct_size(tt->restore, old_pages,
+						     (size_t)1 << order), gfp);
+			/* RFC: Possibly loop on -ENOMEM and reduce order. */

I'd say this is fine as is. If we can't allocate memory from an array of pages here we likely pretty much screwed, right? e.g. We likely don't have a chance of actually allocating new pages for the backing store anyways. Also wouldn't the restart be broken if we can't fully track the state of the restore?

Quoted 213 lines
drivers/gpu/drm/ttm/ttm_pool.c +171 -2
+			if (!tt->restore)
+				return -ENOMEM;
+		} else if (ttm_pool_restore_valid(tt->restore)) {
+			struct ttm_pool_tt_restore *restore = tt->restore;
+
+			num_pages -= restore->alloced_pages;
+			order = min_t(unsigned int, order, __fls(num_pages));
+			pages += restore->alloced_pages;
+			r = ttm_pool_restore_tt(restore, tt->backup, ctx);
+			if (r)
+				return r;
+			caching = restore->caching_divide;
+		}
+
+		tt->restore->pool = pool;
+	}
+
+	for (; num_pages; order = min_t(unsigned int, order, __fls(num_pages))) {
 		struct ttm_pool_type *pt;
 
 		page_caching = tt->caching;
@@ -472,11 +645,19 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				r = ttm_pool_page_allocated(pool, order, p,
 							    &dma_addr,
 							    &num_pages,
-							    &pages);
+							    &pages,
+							    tt->restore);
 				if (r)
 					goto error_free_page;
 
 				caching = pages;
+				if (ttm_pool_restore_valid(tt->restore)) {
+					r = ttm_pool_restore_tt(tt->restore, tt->backup,
+								ctx);
+					if (r)
+						goto error_free_all;
+				}
+
 				if (num_pages < (1 << order))
 					break;
 
@@ -496,9 +677,17 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				caching = pages;
 			}
 			r = ttm_pool_page_allocated(pool, order, p, &dma_addr,
-						    &num_pages, &pages);
+						    &num_pages, &pages,
+						    tt->restore);
 			if (r)
 				goto error_free_page;
+
+			if (ttm_pool_restore_valid(tt->restore)) {
+				r = ttm_pool_restore_tt(tt->restore, tt->backup, ctx);
+				if (r)
+					goto error_free_all;
+			}
+
 			if (PageHighMem(p))
 				caching = pages;
 		}
@@ -517,12 +706,26 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (r)
 		goto error_free_all;
 
+	if (tt->restore) {
+		kvfree(tt->restore);
+		tt->restore = NULL;
+	}
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP)
+		tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+
 	return 0;
 
 error_free_page:
 	ttm_pool_free_page(pool, page_caching, order, p);
 
 error_free_all:
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {
+		tt->restore->caching_divide = caching;
+		return r;
+	}
+
 	num_pages = tt->num_pages - num_pages;
 	caching_divide = caching - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
@@ -549,6 +752,174 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)
 }
 EXPORT_SYMBOL(ttm_pool_free);
 
+/**
+ * ttm_pool_release_backed_up() - Release content of a swapped-out struct ttm_tt
+ * @tt: The struct ttm_tt.
+ *
+ * Release handles with associated content or any remaining pages of
+ * a backed-up struct ttm_tt.
+ */
+void ttm_pool_release_backed_up(struct ttm_tt *tt)
+{
+	struct ttm_backup *backup = tt->backup;
+	struct ttm_pool_tt_restore *restore;
+	pgoff_t i, start_page = 0;
+	unsigned long handle;
+
+	if (!(tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return;
+
+	restore = tt->restore;
+
+	if (ttm_pool_restore_valid(restore)) {
+		pgoff_t nr = 1UL << restore->order;
+
+		for (i = restore->restored_pages; i < nr; ++i) {
+			struct page *p = restore->old_pages[i];
+
+			if (ttm_backup_page_ptr_is_handle(p)) {
+				handle = ttm_backup_page_ptr_to_handle(p);
+				if (handle == 0)
+					continue;
+
+				backup->ops->drop(backup, handle);
+			} else if (p) {
+				ttm_pool_split_for_swap(restore->pool, p);
+				__free_pages(p, 0);
+			}
+		}
+	}
+
+	if (restore) {
+		pgoff_t mid = restore->caching_divide - tt->pages;
+
+		start_page = restore->alloced_pages;
+		/* Pages that might be dma-mapped and non-cached */
+		ttm_pool_free_range(restore->pool, tt, tt->caching,
+				    0, mid);
+		/* Pages that might be dma-mapped but cached */
+		ttm_pool_free_range(restore->pool, tt, ttm_cached,
+				    mid, restore->alloced_pages);
+	}
+
+	/* Shrunken pages. Cached and not dma-mapped. */
+	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt->num_pages);
+
+	if (restore) {
+		kvfree(restore);
+		tt->restore = NULL;
+	}
+
+	tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP | TTM_TT_FLAG_SWAPPED);
+}
+
+/**
+ * ttm_pool_backup_tt() - Back up or purge a struct ttm_tt
+ * @pool: The pool used when allocating the struct ttm_tt.
+ * @ttm: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system.
+ * @writeback: If !@purge, Try to write out directly to the
+ * underlying persistent media.
+ *
+ * Back up or purge a struct ttm_tt. If @purge is true, then
+ * all pages will be freed directly to the system rather than to the pool
+ * they were allocated from, making the function behave similarly to
+ * ttm_pool_free(). If @purge is false the pages will be backed up instead,
+ * exchanged for handles.
+ * A subsequent call to ttm_pool_alloc() will then read back the content and
+ * a subsequent call to ttm_pool_release_shrunken() will drop it.
+ * If backup of a page fails for whatever reason, @ttm will still be
+ * partially backed up, retaining those pages for which backup fails.
+ *
+ * Return: Number of pages actually backed up or freed, or negative
+ * error code on error.
+ */
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge,
+			bool writeback)
+{
+	struct ttm_backup *backup = ttm->backup;
+	struct page *page;
+	unsigned long handle;
+	gfp_t alloc_gfp;
+	gfp_t gfp;
+	int ret = 0;
+	pgoff_t shrunken = 0;
+	pgoff_t i, num_pages;
+
+	if ((!get_nr_swap_pages() && !purge) ||
+	    pool->use_dma_alloc ||
+	    (ttm->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return -EBUSY;
+
+#ifdef CONFIG_X86
+	/* Anything returned to the system needs to be cached. */
+	if (ttm->caching != ttm_cached)
+		set_pages_array_wb(ttm->pages, ttm->num_pages);
+#endif
+
+	if (ttm->dma_address || purge) {
+		for (i = 0; i < ttm->num_pages; i += num_pages) {
+			unsigned int order;
+
+			page = ttm->pages[i];
+			if (unlikely(!page)) {
+				num_pages = 1;
+				continue;
+			}
+
+			order = ttm_pool_page_order(pool, page);
+			num_pages = 1UL << order;
+			if (ttm->dma_address)
+				ttm_pool_unmap(pool, ttm->dma_address[i],
+					       num_pages);
+			if (purge) {

drivers/gpu/drm/ttm/ttm_pool.c void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt) · line 885

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+				shrunken += num_pages;
+				page->private = 0;
+				__free_pages(page, order);
+				memset(ttm->pages + i, 0,
+				       num_pages * sizeof(*ttm->pages));
+			}
+		}
+	}
+
+	if (purge)

if (purge || !backup)?

Quoted 53 lines
drivers/gpu/drm/ttm/ttm_pool.c +37 -0
+		return shrunken;
+
+	if (pool->use_dma32)
+		gfp = GFP_DMA32;
+	else
+		gfp = GFP_HIGHUSER;
+
+	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
+
+	for (i = 0; i < ttm->num_pages; ++i) {
+		page = ttm->pages[i];
+		if (unlikely(!page))
+			continue;
+
+		ttm_pool_split_for_swap(pool, page);
+
+		handle = backup->ops->backup_page(backup, page, writeback, i,
+						  gfp, alloc_gfp);
+		if (handle) {
+			ttm->pages[i] = ttm_backup_handle_to_page_ptr(handle);
+			put_page(page);
+			shrunken++;
+		} else {
+			/* We allow partially shrunken tts */
+			ret = -ENOMEM;
+			break;
+		}
+		cond_resched();
+	}
+
+	if (shrunken)
+		ttm->page_flags |= (TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+
+	return shrunken ? shrunken : ret;
+}
+
 /**
  * ttm_pool_init - Initialize a pool
  *
drivers/gpu/drm/ttm/ttm_tt.c +1 -0
index 4b51b9023126..98ce25197b38 100644
@@ -40,6 +40,7 @@
 #include <drm/drm_cache.h>
 #include <drm/drm_device.h>
 #include <drm/drm_util.h>
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_bo.h>
 #include <drm/ttm/ttm_tt.h>
 

drivers/gpu/drm/ttm/ttm_tt.c static void ttm_tt_init_fields(struct ttm_tt *ttm, · line 162

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +1 -0
@@ -158,6 +159,7 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm,
 	ttm->swap_storage = NULL;
 	ttm->sg = bo->sg;
 	ttm->caching = caching;
+	ttm->restore = NULL;

Set backup to NULL? Seems problematic if not set to NULL and driver doesn't choose to set the backup.

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_fini(struct ttm_tt *ttm) · line 188

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_tt.c +2 -0
 }
 
 int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
@@ -182,6 +184,12 @@ void ttm_tt_fini(struct ttm_tt *ttm)
 		fput(ttm->swap_storage);
 	ttm->swap_storage = NULL;
 
+	ttm_pool_release_backed_up(ttm);
+	if (ttm->backup) {

In patch 12 you don't set this to NULL on error. You will have to set it to NULL their or change this too:

if (ttm->backup && !IS_ERR(ttm->backup))

Quoted 96 lines
drivers/gpu/drm/ttm/ttm_tt.c +33 -0
+		ttm->backup->ops->fini(ttm->backup);
+		ttm->backup = NULL;
+	}
+
 	if (ttm->pages)
 		kvfree(ttm->pages);
 	else
@@ -253,6 +261,35 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);
 
+/**
+ * ttm_tt_backup() - Helper to back up a struct ttm_tt.
+ * @bdev: The TTM device.
+ * @tt: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system,
+ * bypassing any pooling.
+ * @writeback: If !@purge, try to write out directly to the
+ * underlying persistent media.
+ *
+ * Helper for a TTM driver to use from the bo_shrink() method to shrink
+ * a struct ttm_tt, after it has done the necessary unbinding. This function
+ * will update the page accounting and call ttm_pool_shrink_tt to free pages
+ * or move them to the swap cache.
+ *
+ * Return: Number of pages freed or swapped out, or negative error code on
+ * error.
+ */
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt, bool purge,
+		   bool writeback)
+{
+	long ret = ttm_pool_backup_tt(&bdev->pool, tt, purge, writeback);
+
+	if (ret > 0)
+		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
+
+	return ret;
+}
+EXPORT_SYMBOL(ttm_tt_backup);
+
 /**
  * ttm_tt_swapout - swap out tt object
  *
include/drm/ttm/ttm_pool.h +5 -0
index 160d954a261e..4e4db369952b 100644
@@ -89,6 +89,11 @@ void ttm_pool_fini(struct ttm_pool *pool);
 
 int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);
 
+void ttm_pool_release_backed_up(struct ttm_tt *tt);
+
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm,
+			bool purge, bool writeback);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
include/drm/ttm/ttm_tt.h +6 -0
index 2b9d856ff388..6b990f1e7dd0 100644
@@ -32,11 +32,13 @@
 #include <drm/ttm/ttm_caching.h>
 #include <drm/ttm/ttm_kmap_iter.h>
 
+struct ttm_backup;
 struct ttm_device;
 struct ttm_tt;
 struct ttm_resource;
 struct ttm_buffer_object;
 struct ttm_operation_ctx;
+struct ttm_pool_tt_restore;
 
 /**
  * struct ttm_tt - This is a structure holding the pages, caching- and aperture
@@ -85,6 +87,9 @@ struct ttm_tt {
 	 * fault handling abuses the DMA api a bit and dma_map_attrs can't be
 	 * used to assure pgprot always matches.
 	 *
+	 * TTM_TT_FLAG_PRIV_BACKED_UP: TTM internal only. This is set if the
+	 * struct ttm_tt has been (possibly partially) backed up.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -96,6 +101,7 @@ struct ttm_tt {
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 
 #define TTM_TT_FLAG_PRIV_POPULATED	BIT(5)
+#define TTM_TT_FLAG_PRIV_BACKED_UP	BIT(6)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;

include/drm/ttm/ttm_tt.h struct ttm_tt { · line 116

Quoted 6 lines
include/drm/ttm/ttm_tt.h +3 -0
@@ -105,11 +111,21 @@ struct ttm_tt {
 	dma_addr_t *dma_address;
 	/** @swap_storage: Pointer to shmem struct file for swap storage. */
 	struct file *swap_storage;
+	/**
+	 * @backup: Pointer to backup struct for backed up tts.
+	 * RFC: Could possibly be unified with @swap_storage.

I think longterm unifying with swap_storage is problably a good idea. Kinda goofy having two backup mechanisms.

In the meantime, can you add a comment that is this is a driver owned field? This confused me until I looked at the last patch in this series where this field was being setup.

include/drm/ttm/ttm_tt.h struct ttm_tt { · line 126

Quoted 10 lines
include/drm/ttm/ttm_tt.h +5 -0
+	 */
+	struct ttm_backup *backup;
 	/**
 	 * @caching: The current caching state of the pages, see enum
 	 * ttm_caching.
 	 */
 	enum ttm_caching caching;
+	/**
+	 * @restore: Partial restoration from backup state.
+	 * RFC: Incorporate in struct ttm_backup?

I think having a standalone restore field makes sense. Also probably mention is this a TTM private field and drivers shouldn't touch this.

Matt

Quoted 19 lines
include/drm/ttm/ttm_tt.h +6 -0
+	 */
+	struct ttm_pool_tt_restore *restore;
 };
 
 /**
@@ -230,6 +246,10 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 					    struct ttm_tt *tt);
 unsigned long ttm_tt_pages_limit(void);
+
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt, bool purge,
+		   bool writeback);
+
 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>
 
Signature · 1 line
2.44.0
author maintainer
#22 lore raw

Re: [PATCH v6 09/12] drm/ttm/pool: Provide a helper to shrink pages

Quoted 806 lines · Matthew Brost
Quoted 322 lines · Thomas Hellström

Provide a helper to shrink ttm_tt page-vectors on a per-page basis. A ttm_backup backend could then in theory get away with allocating a single temporary page for each struct ttm_tt.

This is accomplished by splitting larger pages before trying to back them up.

In the future we could allow ttm_backup to handle backing up large pages as well, but currently there's no benefit in doing that, since the shmem backup backend would have to split those anyway to avoid allocating too much temporary memory, and if the backend instead inserts pages into the swap-cache, those are split on reclaim by the core.

Due to potential backup- and recover errors, allow partially swapped
out struct ttm_tt's, although mark them as swapped out stopping them
from being swapped out a second time. More details in the ttm_pool.c
DOC section.

v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
 drivers/gpu/drm/ttm/ttm_pool.c | 397
+++++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c   |  37 +++
 include/drm/ttm/ttm_pool.h     |   5 +
 include/drm/ttm/ttm_tt.h       |  20 ++

 4 files changed, 446 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 6e1fd6985ffc..38e50cf81b0a 100644
@@ -41,6 +41,7 @@
  #include <asm/set_memory.h>
  #endif


+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <drm/ttm/ttm_bo.h>

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -58,6 +59,32 @@ struct ttm_pool_dma {
  	unsigned long vaddr;
  };


+/**
+ * struct ttm_pool_tt_restore - State representing restore from backup
+ * @alloced_pages: Total number of already allocated pages for the ttm_tt.
+ * @restored_pages: Number of (sub) pages restored from swap for this
+ *      chunk of 1 << @order pages.
+ * @first_page: The ttm page ptr representing for @old_pages[0].
+ * @caching_divide: Page pointer where subsequent pages are cached.
+ * @old_pages: Backup copy of page pointers that were replaced by the new
+ *        page allocation.
+ * @pool: The pool used for page allocation while restoring.
+ * @order: The order of the last page allocated while restoring.
+ *
+ * Recovery from backup might fail when we've recovered less than the
+ * full ttm_tt. In order not to loose any data (yet), keep information
+ * around that allows us to restart a failed ttm backup recovery.
+ */

drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+struct ttm_pool_tt_restore {
+	pgoff_t alloced_pages;
+	pgoff_t restored_pages;
+	struct page **first_page;
+	struct page **caching_divide;
+	struct ttm_pool *pool;
+	unsigned int order;
+	struct page *old_pages[];
+};
+

 static unsigned long page_pool_size;

 MODULE_PARM_DESC(page_pool_size, "Number of pages in the WC/UC/DMA pool");

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -354,11 +381,102 @@ static unsigned int
 ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
  	return p->private;
  }


+/*
+ * To be able to insert single pages into backup directly,
+ * we need to split multi-order page allocations and make them look
+ * like single-page allocations.
+ */
+static void ttm_pool_split_for_swap(struct ttm_pool *pool, struct page *p)

drivers/gpu/drm/ttm/ttm_pool.c +19 -0
+{
+	unsigned int order = ttm_pool_page_order(pool, p);
+	pgoff_t nr;
+
+	if (!order)
+		return;
+
+	split_page(p, order);
+	nr = 1UL << order;
+	while (nr--)
+		(p++)->private = 0;
+}
+
+/**
+ * DOC: Partial backup and restoration of a struct ttm_tt.
+ *
+ * Swapout using ttm_backup::ops::backup_page() and swapin using
+ * ttm_backup::ops::copy_backed_up_page() may fail.
+ * The former most likely due to lack of swap-space or memory, the

latter due
+ * to lack of memory or because of signal interruption during waits.
+ *
+ * Backupfailure is easily handled by using a ttm_tt pages vector that holds
+ * both swap entries and page pointers. This has to be taken into account when
+ * restoring such a ttm_tt from backup, and when freeing it while backed up.
+ * When restoring, for simplicity, new pages are actually allocated from the
+ * pool and the contents of any old pages are copied in and then the old pages
+ * are released.
+ *
+ * For restoration failures, the struct ttm_pool_tt_restore holds sufficient state
+ * to be able to resume an interrupted restore, and that structure is freed once
+ * the restoration is complete. If the struct ttm_tt is destroyed while there
+ * is a valid struct ttm_pool_tt_restore attached, that is also properly taken
+ * care of.
+ */

drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+
+static bool ttm_pool_restore_valid(const struct

ttm_pool_tt_restore *restore)

drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+{
+	return restore && restore->restored_pages < (1 << restore-
Quoted 1 line

order);

drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+}
+
+static int ttm_pool_restore_tt(struct ttm_pool_tt_restore

*restore,

drivers/gpu/drm/ttm/ttm_pool.c +14 -0
+			       struct ttm_backup *backup,
+			       struct ttm_operation_ctx *ctx)
+{
+	unsigned int i, nr = 1 << restore->order;
+	int ret = 0;
+
+	if (!ttm_pool_restore_valid(restore))
+		return 0;
+
+	for (i = restore->restored_pages; i < nr; ++i) {
+		struct page *p = restore->old_pages[i];
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle =

ttm_backup_page_ptr_to_handle(p);

drivers/gpu/drm/ttm/ttm_pool.c +14 -0
+
+			if (handle == 0)
+				continue;
+
+			ret = backup->ops->copy_backed_up_page
+				(backup, restore->first_page[i],
+				 handle, ctx->interruptible);
+			if (ret)
+				break;
+
+			backup->ops->drop(backup, handle);
+		} else if (p) {
+			/*
+			 * We could probably avoid splitting the

old page

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			 * using clever logic, but ATM we don't

care.

drivers/gpu/drm/ttm/ttm_pool.c +14 -0
+			 */
+			ttm_pool_split_for_swap(restore->pool, p);
+			copy_highpage(restore->first_page[i], p);
+			__free_pages(p, 0);
+		}
+
+		restore->restored_pages++;
+		restore->old_pages[i] = NULL;
+		cond_resched();
+	}
+
+	return ret;
+}
+

 /* Called when we got a page, either from a pool or newly allocated */
 static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,
     struct page *p, dma_addr_t **dma_addr,
     unsigned long *num_pages,

drivers/gpu/drm/ttm/ttm_pool.c +2 -1
-				   struct page ***pages)
+				   struct page ***pages,
+				   struct ttm_pool_tt_restore

*restore)
 {
  unsigned int i;
  int r;

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -369,6 +487,16 @@ static int ttm_pool_page_allocated(struct
 ttm_pool *pool, unsigned int order,
  			return r;
  	}

drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+	if (restore) {
+		memcpy(restore->old_pages, *pages,
+		       (1 << order) * sizeof(*restore-
Quoted 1 line

old_pages));

drivers/gpu/drm/ttm/ttm_pool.c +7 -0
+		memset(*pages, 0, (1 << order) * sizeof(**pages));
+		restore->order = order;
+		restore->restored_pages = 0;
+		restore->first_page = *pages;
+		restore->alloced_pages += 1UL << order;
+	}
+

  *num_pages -= 1 << order;
  for (i = 1 << order; i; --i, ++(*pages), ++p) **pages = p;

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -394,22 +522,39 @@ static void ttm_pool_free_range(struct
 ttm_pool *pool, struct ttm_tt *tt,

  pgoff_t start_page, pgoff_t end_page)
 {
  struct page **pages = &tt->pages[start_page];

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+	struct ttm_backup *backup = tt->backup;

  unsigned int order;
  pgoff_t i, nr;

  for (i = start_page; i < end_page; i += nr, pages += nr) { struct ttm_pool_type *pt = NULL;

drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+		struct page *p = *pages;
+
+		if (ttm_backup_page_ptr_is_handle(p)) {
+			unsigned long handle =

ttm_backup_page_ptr_to_handle(p);

drivers/gpu/drm/ttm/ttm_pool.c +12 -0
+
+			nr = 1;
+			if (handle != 0)
+				backup->ops->drop(backup, handle);
+			continue;
+		}
+
+		if (pool) {
+			order = ttm_pool_page_order(pool, p);
+			nr = (1UL << order);
+			if (tt->dma_address)
+				ttm_pool_unmap(pool, tt-
Quoted 1 line

dma_address[i], nr);

drivers/gpu/drm/ttm/ttm_pool.c +0 -4
-		order = ttm_pool_page_order(pool, *pages);
-		nr = (1UL << order);
-		if (tt->dma_address)
-			ttm_pool_unmap(pool, tt->dma_address[i],

nr);

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			pt = ttm_pool_select_type(pool, caching,

order);

drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+		} else {
+			order = p->private;
+			nr = (1UL << order);
+		}

drivers/gpu/drm/ttm/ttm_pool.c +0 -1
-		pt = ttm_pool_select_type(pool, caching, order);

  if (pt)

drivers/gpu/drm/ttm/ttm_pool.c +1 -1
-			ttm_pool_type_give(pt, *pages);
+			ttm_pool_type_give(pt, p);

  else

drivers/gpu/drm/ttm/ttm_pool.c +0 -1
-			ttm_pool_free_page(pool, caching, order,

*pages);

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			ttm_pool_free_page(pool, caching, order,

p);
  }
 }

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -453,9 +598,37 @@ int ttm_pool_alloc(struct ttm_pool *pool,
 struct ttm_tt *tt,

  else
  gfp_flags |= GFP_HIGHUSER;

drivers/gpu/drm/ttm/ttm_pool.c +0 -1
-	for (order = min_t(unsigned int, MAX_PAGE_ORDER,

__fls(num_pages));

drivers/gpu/drm/ttm/ttm_pool.c +0 -2
-	     num_pages;
-	     order = min_t(unsigned int, order, __fls(num_pages)))

{

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+	order = min_t(unsigned int, MAX_PAGE_ORDER,

__fls(num_pages));

drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {
+		if (!tt->restore) {
+			gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
+
+			if (ctx->gfp_retry_mayfail)
+				gfp |= __GFP_RETRY_MAYFAIL;
+
+			tt->restore =
+				kvzalloc(struct_size(tt->restore,

old_pages,

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+						     (size_t)1 <<

order), gfp);

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			/* RFC: Possibly loop on -ENOMEM and
Quoted 1 line

reduce order. */

I'd say this is fine as is. If we can't allocate memory from an array of
pages here we likely pretty much screwed, right? e.g. We likely don't have a chance of actually allocating new pages for the backing store anyways. Also wouldn't the restart be broken if we can't fully track the
state of the restore?

Quoted 259 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+			if (!tt->restore)
+				return -ENOMEM;
+		} else if (ttm_pool_restore_valid(tt->restore)) {
+			struct ttm_pool_tt_restore *restore = tt-
Quoted 1 line

restore;

drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+
+			num_pages -= restore->alloced_pages;
+			order = min_t(unsigned int, order,

__fls(num_pages));

drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+			pages += restore->alloced_pages;
+			r = ttm_pool_restore_tt(restore, tt-
Quoted 1 line

backup, ctx);

drivers/gpu/drm/ttm/ttm_pool.c +9 -0
+			if (r)
+				return r;
+			caching = restore->caching_divide;
+		}
+
+		tt->restore->pool = pool;
+	}
+
+	for (; num_pages; order = min_t(unsigned int, order,

__fls(num_pages))) {
  struct ttm_pool_type *pt;

  page_caching = tt->caching;

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -472,11 +645,19 @@ int ttm_pool_alloc(struct ttm_pool *pool,
 struct ttm_tt *tt,
  				r = ttm_pool_page_allocated(pool,
 order, p,


&dma_addr,

&num_pages,

drivers/gpu/drm/ttm/ttm_pool.c +0 -1
-							   

&pages);

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+							   

&pages,

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+							    tt-
Quoted 1 line

restore);

  if (r)
  goto error_free_page;
  caching = pages;

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+				if (ttm_pool_restore_valid(tt-
Quoted 1 line

restore)) {

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+					r =

ttm_pool_restore_tt(tt->restore, tt->backup,

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+								ct

x);

drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+					if (r)
+						goto

error_free_all;

drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+				}
+

  if (num_pages < (1 << order)) break;

drivers/gpu/drm/ttm/ttm_pool.c +2 -1
@@ -496,9 +677,17 @@ int ttm_pool_alloc(struct ttm_pool *pool,
 struct ttm_tt *tt,
  				caching = pages;
  			}
  			r = ttm_pool_page_allocated(pool, order,
 p, &dma_addr,
-						    &num_pages,
 &pages);
+						    &num_pages,
 &pages,
+						    tt->restore);
  			if (r)

  goto error_free_page;

drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+
+			if (ttm_pool_restore_valid(tt->restore)) {
+				r = ttm_pool_restore_tt(tt-
Quoted 1 line

restore, tt->backup, ctx);

drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+				if (r)
+					goto error_free_all;
+			}
+

  if (PageHighMem(p))
  caching = pages;
  }

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -517,12 +706,26 @@ int ttm_pool_alloc(struct ttm_pool *pool,
 struct ttm_tt *tt,
  	if (r)
  		goto error_free_all;

drivers/gpu/drm/ttm/ttm_pool.c +9 -0
+	if (tt->restore) {
+		kvfree(tt->restore);
+		tt->restore = NULL;
+	}
+
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP)
+		tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+

  return 0;

 error_free_page:
  ttm_pool_free_page(pool, page_caching, order, p);
 error_free_all:

drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+	if (tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP) {
+		tt->restore->caching_divide = caching;
+		return r;
+	}
+

  num_pages = tt->num_pages - num_pages; caching_divide = caching - tt->pages; ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -549,6 +752,174 @@ void ttm_pool_free(struct ttm_pool *pool,
 struct ttm_tt *tt)
  }
  EXPORT_SYMBOL(ttm_pool_free);


+/**
+ * ttm_pool_release_backed_up() - Release content of a swapped-out struct ttm_tt
+ * @tt: The struct ttm_tt.
+ *
+ * Release handles with associated content or any remaining pages of
+ * a backed-up struct ttm_tt.
+ */

drivers/gpu/drm/ttm/ttm_pool.c +20 -0
+void ttm_pool_release_backed_up(struct ttm_tt *tt)
+{
+	struct ttm_backup *backup = tt->backup;
+	struct ttm_pool_tt_restore *restore;
+	pgoff_t i, start_page = 0;
+	unsigned long handle;
+
+	if (!(tt->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return;
+
+	restore = tt->restore;
+
+	if (ttm_pool_restore_valid(restore)) {
+		pgoff_t nr = 1UL << restore->order;
+
+		for (i = restore->restored_pages; i < nr; ++i) {
+			struct page *p = restore->old_pages[i];
+
+			if (ttm_backup_page_ptr_is_handle(p)) {
+				handle =

ttm_backup_page_ptr_to_handle(p);

drivers/gpu/drm/ttm/ttm_pool.c +6 -0
+				if (handle == 0)
+					continue;
+
+				backup->ops->drop(backup, handle);
+			} else if (p) {
+				ttm_pool_split_for_swap(restore-
Quoted 1 line

pool, p);

drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+				__free_pages(p, 0);
+			}
+		}
+	}
+
+	if (restore) {
+		pgoff_t mid = restore->caching_divide - tt->pages;
+
+		start_page = restore->alloced_pages;
+		/* Pages that might be dma-mapped and non-cached

*/

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+		ttm_pool_free_range(restore->pool, tt, tt-
Quoted 1 line

caching,

drivers/gpu/drm/ttm/ttm_pool.c +8 -0
+				    0, mid);
+		/* Pages that might be dma-mapped but cached */
+		ttm_pool_free_range(restore->pool, tt, ttm_cached,
+				    mid, restore->alloced_pages);
+	}
+
+	/* Shrunken pages. Cached and not dma-mapped. */
+	ttm_pool_free_range(NULL, tt, ttm_cached, start_page, tt-
Quoted 1 line

num_pages);

drivers/gpu/drm/ttm/ttm_pool.c +7 -0
+
+	if (restore) {
+		kvfree(restore);
+		tt->restore = NULL;
+	}
+
+	tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP |

TTM_TT_FLAG_SWAPPED);

drivers/gpu/drm/ttm/ttm_pool.c +12 -0
+}
+
+/**
+ * ttm_pool_backup_tt() - Back up or purge a struct ttm_tt
+ * @pool: The pool used when allocating the struct ttm_tt.
+ * @ttm: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system.
+ * @writeback: If !@purge, Try to write out directly to the
+ * underlying persistent media.
+ *
+ * Back up or purge a struct ttm_tt. If @purge is true, then
+ * all pages will be freed directly to the system rather than to

the pool
+ * they were allocated from, making the function behave similarly to
+ * ttm_pool_free(). If @purge is false the pages will be backed up instead,
+ * exchanged for handles.
+ * A subsequent call to ttm_pool_alloc() will then read back the content and
+ * a subsequent call to ttm_pool_release_shrunken() will drop it.
+ * If backup of a page fails for whatever reason, @ttm will still be
+ * partially backed up, retaining those pages for which backup fails.
+ *
+ * Return: Number of pages actually backed up or freed, or negative
+ * error code on error.
+ */
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge,

drivers/gpu/drm/ttm/ttm_pool.c +36 -0
+			bool writeback)
+{
+	struct ttm_backup *backup = ttm->backup;
+	struct page *page;
+	unsigned long handle;
+	gfp_t alloc_gfp;
+	gfp_t gfp;
+	int ret = 0;
+	pgoff_t shrunken = 0;
+	pgoff_t i, num_pages;
+
+	if ((!get_nr_swap_pages() && !purge) ||
+	    pool->use_dma_alloc ||
+	    (ttm->page_flags & TTM_TT_FLAG_PRIV_BACKED_UP))
+		return -EBUSY;
+
+#ifdef CONFIG_X86
+	/* Anything returned to the system needs to be cached. */
+	if (ttm->caching != ttm_cached)
+		set_pages_array_wb(ttm->pages, ttm->num_pages);
+#endif
+
+	if (ttm->dma_address || purge) {
+		for (i = 0; i < ttm->num_pages; i += num_pages) {
+			unsigned int order;
+
+			page = ttm->pages[i];
+			if (unlikely(!page)) {
+				num_pages = 1;
+				continue;
+			}
+
+			order = ttm_pool_page_order(pool, page);
+			num_pages = 1UL << order;
+			if (ttm->dma_address)
+				ttm_pool_unmap(pool, ttm-
Quoted 1 line

dma_address[i],

drivers/gpu/drm/ttm/ttm_pool.c +7 -0
+					       num_pages);
+			if (purge) {
+				shrunken += num_pages;
+				page->private = 0;
+				__free_pages(page, order);
+				memset(ttm->pages + i, 0,
+				       num_pages * sizeof(*ttm-
Quoted 1 line

pages));

drivers/gpu/drm/ttm/ttm_pool.c void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt) · line 885

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+			}
+		}
+	}
+
+	if (purge)

if (purge || !backup)?

Quoted 57 lines
drivers/gpu/drm/ttm/ttm_pool.c +8 -0
+		return shrunken;
+
+	if (pool->use_dma32)
+		gfp = GFP_DMA32;
+	else
+		gfp = GFP_HIGHUSER;
+
+	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN |

__GFP_RETRY_MAYFAIL;

drivers/gpu/drm/ttm/ttm_pool.c +9 -0
+
+	for (i = 0; i < ttm->num_pages; ++i) {
+		page = ttm->pages[i];
+		if (unlikely(!page))
+			continue;
+
+		ttm_pool_split_for_swap(pool, page);
+
+		handle = backup->ops->backup_page(backup, page,

writeback, i,

drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+						  gfp, alloc_gfp);
+		if (handle) {
+			ttm->pages[i] =

ttm_backup_handle_to_page_ptr(handle);

drivers/gpu/drm/ttm/ttm_pool.c +17 -0
+			put_page(page);
+			shrunken++;
+		} else {
+			/* We allow partially shrunken tts */
+			ret = -ENOMEM;
+			break;
+		}
+		cond_resched();
+	}
+
+	if (shrunken)
+		ttm->page_flags |= (TTM_TT_FLAG_PRIV_BACKED_UP |
+				    TTM_TT_FLAG_SWAPPED);
+
+	return shrunken ? shrunken : ret;
+}
+

 /**
  * ttm_pool_init - Initialize a pool
  *

drivers/gpu/drm/ttm/ttm_tt.c +1 -0
index 4b51b9023126..98ce25197b38 100644
@@ -40,6 +40,7 @@
  #include <drm/drm_cache.h>
  #include <drm/drm_device.h>
  #include <drm/drm_util.h>
+#include <drm/ttm/ttm_backup.h>
  #include <drm/ttm/ttm_bo.h>
  #include <drm/ttm/ttm_tt.h>

drivers/gpu/drm/ttm/ttm_tt.c static void ttm_tt_init_fields(struct ttm_tt · line 163

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_tt.c +1 -0
@@ -158,6 +159,7 @@ static void ttm_tt_init_fields(struct ttm_tt
 *ttm,
  	ttm->swap_storage = NULL;
  	ttm->sg = bo->sg;
  	ttm->caching = caching;
+	ttm->restore = NULL;

Set backup to NULL? Seems problematic if not set to NULL and driver doesn't choose to set the backup.

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_fini(struct ttm_tt *ttm) · line 188

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_tt.c +0 -0
  }


 int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,

drivers/gpu/drm/ttm/ttm_tt.c +0 -0
@@ -182,6 +184,12 @@ void ttm_tt_fini(struct ttm_tt *ttm)
  		fput(ttm->swap_storage);
  	ttm->swap_storage = NULL;

drivers/gpu/drm/ttm/ttm_tt.c +2 -0
+	ttm_pool_release_backed_up(ttm);
+	if (ttm->backup) {

In patch 12 you don't set this to NULL on error. You will have to set it
to NULL their or change this too:

if (ttm->backup && !IS_ERR(ttm->backup))

Quoted 112 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+		ttm->backup->ops->fini(ttm->backup);
+		ttm->backup = NULL;
+	}
+

  if (ttm->pages)
  kvfree(ttm->pages);
  else

drivers/gpu/drm/ttm/ttm_tt.c +0 -0
@@ -253,6 +261,35 @@ int ttm_tt_swapin(struct ttm_tt *ttm)
  }
  EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapin);


+/**
+ * ttm_tt_backup() - Helper to back up a struct ttm_tt.
+ * @bdev: The TTM device.
+ * @tt: The struct ttm_tt.
+ * @purge: Don't back up but release pages directly to system,
+ * bypassing any pooling.
+ * @writeback: If !@purge, try to write out directly to the
+ * underlying persistent media.
+ *
+ * Helper for a TTM driver to use from the bo_shrink() method to shrink
+ * a struct ttm_tt, after it has done the necessary unbinding. This function
+ * will update the page accounting and call ttm_pool_shrink_tt to free pages
+ * or move them to the swap cache.
+ *
+ * Return: Number of pages freed or swapped out, or negative error code on
+ * error.
+ */
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt, bool purge,

drivers/gpu/drm/ttm/ttm_tt.c +3 -0
+		   bool writeback)
+{
+	long ret = ttm_pool_backup_tt(&bdev->pool, tt, purge,

writeback);

drivers/gpu/drm/ttm/ttm_tt.c +8 -0
+
+	if (ret > 0)
+		tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED;
+
+	return ret;
+}
+EXPORT_SYMBOL(ttm_tt_backup);
+

 /**
  * ttm_tt_swapout - swap out tt object
  *

include/drm/ttm/ttm_pool.h +0 -0
index 160d954a261e..4e4db369952b 100644
@@ -89,6 +89,11 @@ void ttm_pool_fini(struct ttm_pool *pool);


 int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m);

include/drm/ttm/ttm_pool.h +5 -0
+void ttm_pool_release_backed_up(struct ttm_tt *tt);
+
+long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm,
+			bool purge, bool writeback);
+

 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);

include/drm/ttm/ttm_tt.h +0 -0
index 2b9d856ff388..6b990f1e7dd0 100644
@@ -32,11 +32,13 @@
  #include <drm/ttm/ttm_caching.h>
  #include <drm/ttm/ttm_kmap_iter.h>


+struct ttm_backup;
 struct ttm_device;
 struct ttm_tt;
 struct ttm_resource;
 struct ttm_buffer_object;
 struct ttm_operation_ctx;
+struct ttm_pool_tt_restore;

 /**
  * struct ttm_tt - This is a structure holding the pages, caching- and aperture

include/drm/ttm/ttm_tt.h +0 -0
@@ -85,6 +87,9 @@ struct ttm_tt {
  	 * fault handling abuses the DMA api a bit and

dma_map_attrs can't be
  * used to assure pgprot always matches.
  *

include/drm/ttm/ttm_tt.h +1 -0
+	 * TTM_TT_FLAG_PRIV_BACKED_UP: TTM internal only. This is

set if the

include/drm/ttm/ttm_tt.h +2 -0
+	 * struct ttm_tt has been (possibly partially) backed up.
+	 *

  * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
  * set by TTM after ttm_tt_populate() has successfully returned, and is
  * then unset when TTM calls ttm_tt_unpopulate().

include/drm/ttm/ttm_tt.h +0 -0
@@ -96,6 +101,7 @@ struct ttm_tt {
  #define TTM_TT_FLAG_DECRYPTED		BIT(4)


 #define TTM_TT_FLAG_PRIV_POPULATED BIT(5)
+#define TTM_TT_FLAG_PRIV_BACKED_UP BIT(6)
  uint32_t page_flags;
  /** @num_pages: Number of pages in the page array. */ uint32_t num_pages;

include/drm/ttm/ttm_tt.h +0 -0
@@ -105,11 +111,21 @@ struct ttm_tt {
  	dma_addr_t *dma_address;
  	/** @swap_storage: Pointer to shmem struct file for swap

storage. */
  struct file *swap_storage;

include/drm/ttm/ttm_tt.h struct ttm_tt { · line 116

Quoted 3 lines
include/drm/ttm/ttm_tt.h +3 -0
+	/**
+	 * @backup: Pointer to backup struct for backed up tts.
+	 * RFC: Could possibly be unified with @swap_storage.

I think longterm unifying with swap_storage is problably a good idea. Kinda goofy having two backup mechanisms.

In the meantime, can you add a comment that is this is a driver owned field? This confused me until I looked at the last patch in this series
where this field was being setup.

Quoted 14 lines

include/drm/ttm/ttm_tt.h struct ttm_tt { · line 126

Quoted 11 lines
include/drm/ttm/ttm_tt.h +2 -0
+	 */
+	struct ttm_backup *backup;

  /**
  * @caching: The current caching state of the pages, see enum
  * ttm_caching.
  */
  enum ttm_caching caching;

include/drm/ttm/ttm_tt.h +3 -0
+	/**
+	 * @restore: Partial restoration from backup state.
+	 * RFC: Incorporate in struct ttm_backup?

I think having a standalone restore field makes sense. Also probably mention is this a TTM private field and drivers shouldn't touch this.

Thanks for reviewing, Matt. I'll incorporate all your suggestions, but assert that ttm->backup is valid in ttm_tt.c

Thanks,
Thomas

Quoted 25 lines

Matt

Quoted 22 lines
include/drm/ttm/ttm_tt.h +2 -0
+	 */
+	struct ttm_pool_tt_restore *restore;

 };

 /**

include/drm/ttm/ttm_tt.h +0 -0
@@ -230,6 +246,10 @@ void ttm_tt_mgr_init(unsigned long num_pages,
 unsigned long num_dma32_pages);

 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
      struct ttm_tt *tt); unsigned long ttm_tt_pages_limit(void);

include/drm/ttm/ttm_tt.h +2 -0
+
+long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,

bool purge,

include/drm/ttm/ttm_tt.h +2 -0
+		   bool writeback);
+

 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>

Signature · 1 line
2.44.0
author maintainer
#23 lore raw

PATCH v6 10/12 drm/ttm: Use fault-injection to test error paths

In reply to Thomas Hellström

Use fault-injection to test partial TTM swapout and interrupted swapin. Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that the swapin and release_shrunken functionality.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/Kconfig        | 10 ++++++++++
 drivers/gpu/drm/ttm/ttm_pool.c | 17 ++++++++++++++++-
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/Kconfig +10 -0
index fd0749c0c630..9f27271bfab8 100644
@@ -272,6 +272,16 @@ config DRM_GPUVM
 	  GPU-VM representation providing helpers to manage a GPUs virtual
 	  address space
 
+config DRM_TTM_BACKUP_FAULT_INJECT
+	bool "Enable fault injection during TTM backup"
+	depends on DRM_TTM
+	default n
+	help
+	  Inject recoverable failures during TTM backup and recovery of
+	  backed-up objects. For DRM driver developers only.
+
+	  If in doubt, choose N.
+
 config DRM_BUDDY
 	tristate
 	depends on DRM
drivers/gpu/drm/ttm/ttm_pool.c +16 -1
index 38e50cf81b0a..d32a1f2e5e50 100644
@@ -431,6 +431,7 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
 			       struct ttm_backup *backup,
 			       struct ttm_operation_ctx *ctx)
 {
+	static unsigned long __maybe_unused swappedin;
 	unsigned int i, nr = 1 << restore->order;
 	int ret = 0;
 
@@ -446,6 +447,13 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
 			if (handle == 0)
 				continue;
 
+			if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT) &&
+			    ctx->interruptible &&
+			    ++swappedin % 100 == 0) {
+				ret = -EINTR;
+				break;
+			}
+
 			ret = backup->ops->copy_backed_up_page
 				(backup, restore->first_page[i],
 				 handle, ctx->interruptible);
@@ -892,7 +900,14 @@ long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge,
 
 	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
 
-	for (i = 0; i < ttm->num_pages; ++i) {
+	num_pages = ttm->num_pages;
+
+	/* Pretend doing fault injection by shrinking only half of the pages. */
+
+	if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT))
+		num_pages = DIV_ROUND_UP(num_pages, 2);
+
+	for (i = 0; i < num_pages; ++i) {
 		page = ttm->pages[i];
 		if (unlikely(!page))
 			continue;
Signature · 1 line
2.44.0
maintainer
#24 lore raw

Re: [PATCH v6 10/12] drm/ttm: Use fault-injection to test error paths

Quoted 49 lines · Thomas Hellström

Use fault-injection to test partial TTM swapout and interrupted swapin. Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that the swapin and release_shrunken functionality.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/Kconfig        | 10 ++++++++++
 drivers/gpu/drm/ttm/ttm_pool.c | 17 ++++++++++++++++-
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/Kconfig +10 -0
index fd0749c0c630..9f27271bfab8 100644
@@ -272,6 +272,16 @@ config DRM_GPUVM
 	  GPU-VM representation providing helpers to manage a GPUs virtual
 	  address space
 
+config DRM_TTM_BACKUP_FAULT_INJECT
+	bool "Enable fault injection during TTM backup"
+	depends on DRM_TTM
+	default n
+	help
+	  Inject recoverable failures during TTM backup and recovery of
+	  backed-up objects. For DRM driver developers only.
+
+	  If in doubt, choose N.
+
 config DRM_BUDDY
 	tristate
 	depends on DRM
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
index 38e50cf81b0a..d32a1f2e5e50 100644
@@ -431,6 +431,7 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
 			       struct ttm_backup *backup,
 			       struct ttm_operation_ctx *ctx)
 {
+	static unsigned long __maybe_unused swappedin;
 	unsigned int i, nr = 1 << restore->order;
 	int ret = 0;
 

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore, · line 455

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +6 -0
@@ -446,6 +447,13 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,
 			if (handle == 0)
 				continue;
 
+			if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT) &&
+			    ctx->interruptible &&
+			    ++swappedin % 100 == 0) {
+				ret = -EINTR;
+				break;
+			}

So here this -EINTR would be kicked to the user IOCTL which triggered the BO validate and retry? The restore then should be able to successfully pick up where it left off?

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+
 			ret = backup->ops->copy_backed_up_page
 				(backup, restore->first_page[i],
 				 handle, ctx->interruptible);

drivers/gpu/drm/ttm/ttm_pool.c long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge, · line 908

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +6 -1
@@ -892,7 +900,14 @@ long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge,
 
 	alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
 
-	for (i = 0; i < ttm->num_pages; ++i) {
+	num_pages = ttm->num_pages;
+
+	/* Pretend doing fault injection by shrinking only half of the pages. */
+
+	if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT))
+		num_pages = DIV_ROUND_UP(num_pages, 2);

So what happens here? Half the pages swapped out, then upon restore half swapped back in? The shrinker continues to walk until enough pages swapped out?

Matt

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+
+	for (i = 0; i < num_pages; ++i) {
 		page = ttm->pages[i];
 		if (unlikely(!page))
 			continue;
Signature · 1 line
2.44.0
author maintainer
#25 lore raw

Re: [PATCH v6 10/12] drm/ttm: Use fault-injection to test error paths

Quoted 63 lines · Matthew Brost
Quoted 62 lines · Thomas Hellström

Use fault-injection to test partial TTM swapout and interrupted swapin.
Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that
the swapin and release_shrunken functionality.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
 drivers/gpu/drm/Kconfig        | 10 ++++++++++ drivers/gpu/drm/ttm/ttm_pool.c | 17 ++++++++++++++++-

 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/Kconfig +0 -0
index fd0749c0c630..9f27271bfab8 100644
@@ -272,6 +272,16 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs virtual
    address space

drivers/gpu/drm/Kconfig +6 -0
+config DRM_TTM_BACKUP_FAULT_INJECT
+	bool "Enable fault injection during TTM backup"
+	depends on DRM_TTM
+	default n
+	help
+	  Inject recoverable failures during TTM backup and

recovery of

drivers/gpu/drm/Kconfig +4 -0
+	  backed-up objects. For DRM driver developers only.
+
+	  If in doubt, choose N.
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
index 38e50cf81b0a..d32a1f2e5e50 100644
@@ -431,6 +431,7 @@ static int ttm_pool_restore_tt(struct
 ttm_pool_tt_restore *restore,
  			       struct ttm_backup *backup,
  			       struct ttm_operation_ctx *ctx)
  {
+	static unsigned long __maybe_unused swappedin;
  	unsigned int i, nr = 1 << restore->order;
  	int ret = 0;

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -446,6 +447,13 @@ static int ttm_pool_restore_tt(struct
 ttm_pool_tt_restore *restore,
  			if (handle == 0)
  				continue;

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			if

(IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT) &&

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			    ctx->interruptible &&
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore, · line 455

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+			    ++swappedin % 100 == 0) {
+				ret = -EINTR;
+				break;
+			}

So here this -EINTR would be kicked to the user IOCTL which triggered the BO validate and retry? The restore then should be able to successfully pick up where it left off?

Yes, that's the point. For the direct swap-cache backend I initially used (before concluding that the shmem one actually seemed to work fine), we had an interruptible wait here.

Supporting interrupts is generally a good thing but for the pool code, this makes the code considerably more complicated. However, this is a good way to ensure drivers actually support -EINTR for the call chain. If not, adding interrupt capability "later" will most likely be a PITA.

Quoted 16 lines
Quoted 16 lines

+
  ret = backup->ops->copy_backed_up_page (backup, restore->first_page[i], handle, ctx->interruptible);

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -892,7 +900,14 @@ long ttm_pool_backup_tt(struct ttm_pool *pool,
 struct ttm_tt *ttm, bool purge,


  alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;

drivers/gpu/drm/ttm/ttm_pool.c +3 -1
-	for (i = 0; i < ttm->num_pages; ++i) {
+	num_pages = ttm->num_pages;
+
+	/* Pretend doing fault injection by shrinking only half of

the pages. */

Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge, · line 908

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+
+	if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT))
+		num_pages = DIV_ROUND_UP(num_pages, 2);

So what happens here? Half the pages swapped out, then upon restore half
swapped back in? The shrinker continues to walk until enough pages swapped out?

Yes, exactly. Ideally we'd want some intermediate state here so that a partially swapped out bo is still eligible for further shrinking.

/Thomas

Quoted 11 lines

Matt

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+
+	for (i = 0; i < num_pages; ++i) {

  page = ttm->pages[i];
  if (unlikely(!page))
  continue;

Signature · 1 line
2.44.0
maintainer Reviewed-by
#26 lore raw

Re: [PATCH v6 10/12] drm/ttm: Use fault-injection to test error paths

Quoted 107 lines · Thomas Hellström
Quoted 63 lines · Matthew Brost
Quoted 62 lines · Thomas Hellström

Use fault-injection to test partial TTM swapout and interrupted swapin.
Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that
the swapin and release_shrunken functionality.

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
 drivers/gpu/drm/Kconfig        | 10 ++++++++++ drivers/gpu/drm/ttm/ttm_pool.c | 17 ++++++++++++++++-

 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/Kconfig +0 -0
index fd0749c0c630..9f27271bfab8 100644
@@ -272,6 +272,16 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs virtual
    address space

drivers/gpu/drm/Kconfig +6 -0
+config DRM_TTM_BACKUP_FAULT_INJECT
+	bool "Enable fault injection during TTM backup"
+	depends on DRM_TTM
+	default n
+	help
+	  Inject recoverable failures during TTM backup and

recovery of

drivers/gpu/drm/Kconfig +4 -0
+	  backed-up objects. For DRM driver developers only.
+
+	  If in doubt, choose N.
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
index 38e50cf81b0a..d32a1f2e5e50 100644
@@ -431,6 +431,7 @@ static int ttm_pool_restore_tt(struct
 ttm_pool_tt_restore *restore,
  			       struct ttm_backup *backup,
  			       struct ttm_operation_ctx *ctx)
  {
+	static unsigned long __maybe_unused swappedin;
  	unsigned int i, nr = 1 << restore->order;
  	int ret = 0;

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -446,6 +447,13 @@ static int ttm_pool_restore_tt(struct
 ttm_pool_tt_restore *restore,
  			if (handle == 0)
  				continue;

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			if

(IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT) &&

drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+			    ctx->interruptible &&
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore, · line 455

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+			    ++swappedin % 100 == 0) {
+				ret = -EINTR;
+				break;
+			}

So here this -EINTR would be kicked to the user IOCTL which triggered the BO validate and retry? The restore then should be able to successfully pick up where it left off?

Yes, that's the point. For the direct swap-cache backend I initially used (before concluding that the shmem one actually seemed to work fine), we had an interruptible wait here.

Supporting interrupts is generally a good thing but for the pool code, this makes the code considerably more complicated. However, this is a good way to ensure drivers actually support -EINTR for the call chain. If not, adding interrupt capability "later" will most likely be a PITA.

Quoted 16 lines
Quoted 16 lines

+
  ret = backup->ops->copy_backed_up_page (backup, restore->first_page[i], handle, ctx->interruptible);

drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -892,7 +900,14 @@ long ttm_pool_backup_tt(struct ttm_pool *pool,
 struct ttm_tt *ttm, bool purge,


  alloc_gfp = GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;

drivers/gpu/drm/ttm/ttm_pool.c +3 -1
-	for (i = 0; i < ttm->num_pages; ++i) {
+	num_pages = ttm->num_pages;
+
+	/* Pretend doing fault injection by shrinking only half of

the pages. */

Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge, · line 908

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +3 -0
+
+	if (IS_ENABLED(CONFIG_DRM_TTM_BACKUP_FAULT_INJECT))
+		num_pages = DIV_ROUND_UP(num_pages, 2);

So what happens here? Half the pages swapped out, then upon restore half
swapped back in? The shrinker continues to walk until enough pages swapped out?

Quoted 3 lines

Yes, exactly. Ideally we'd want some intermediate state here so that a partially swapped out bo is still eligible for further shrinking.

Cool, glad my understanding is correct. Having error injection is always a good idea to ensure error paths / corner cases work rather than finding they blow up much later when these cases somehow get triggered.

With that:

Quoted 15 lines

/Thomas

Quoted 11 lines

Matt

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+
+	for (i = 0; i < num_pages; ++i) {

  page = ttm->pages[i];
  if (unlikely(!page))
  continue;

Signature · 1 line
2.44.0
author maintainer
#27 lore raw

PATCH v6 11/12 drm/ttm, drm/xe: Add a shrinker for xe bos

In reply to Thomas Hellström

Rather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo_util.c     |  67 ++++++
 drivers/gpu/drm/xe/Makefile           |   1 +
 drivers/gpu/drm/xe/tests/xe_bo.c      | 118 +++++++++++
 drivers/gpu/drm/xe/tests/xe_bo_test.c |   1 +
 drivers/gpu/drm/xe/tests/xe_bo_test.h |   1 +
 drivers/gpu/drm/xe/xe_bo.c            | 155 ++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h            |  26 +++
 drivers/gpu/drm/xe/xe_device.c        |   8 +
 drivers/gpu/drm/xe/xe_device_types.h  |   2 +
 drivers/gpu/drm/xe/xe_shrinker.c      | 287 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h      |  18 ++
 include/drm/ttm/ttm_bo.h              |   3 +
 12 files changed, 671 insertions(+), 16 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
drivers/gpu/drm/ttm/ttm_bo_util.c +67 -0
index c4f678f30fc2..563e96a4cf06 100644
@@ -924,3 +924,70 @@ long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
 
 	return progress;
 }
+EXPORT_SYMBOL(ttm_lru_walk_for_evict);
+
+/**
+ * ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer object.
+ * @walk: The struct xe_ttm_lru_walk that describes the walk.
+ * @bo: The buffer object.
+ * @purge: Whether to attempt to purge the bo content since it's no
+ * longer needed.
+ * @writeback: If !@purge, attempt to write out to persistent storage.
+ *
+ * The function uses the ttm_tt_back_up functionality to back up or
+ * purge a struct ttm_tt. If the bo is not in system, it's first
+ * moved there.
+ *
+ * Return: The number of pages shrunken or purged, or
+ * negative error code on failure.
+ */
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		       bool purge, bool writeback)
+{
+	static const struct ttm_place sys_placement_flags = {
+		.fpfn = 0,
+		.lpfn = 0,
+		.mem_type = TTM_PL_SYSTEM,
+		.flags = 0,
+	};
+	static struct ttm_placement sys_placement = {
+		.num_placement = 1,
+		.placement = &sys_placement_flags,
+	};
+	struct ttm_operation_ctx *ctx = walk->ctx;
+	struct ttm_tt *tt = bo->ttm;
+	long lret;
+
+	dma_resv_assert_held(bo->base.resv);
+
+	if (!tt || !ttm_tt_is_populated(tt))
+		return 0;
+
+	if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+		int ret = ttm_bo_validate(bo, &sys_placement, ctx);
+
+		if (ret) {
+			if (ret == -EINTR || ret == -EDEADLK ||
+			    ret == -ERESTARTSYS)
+				return ret;
+			return 0;
+		}
+	}
+
+	lret = ttm_bo_wait_ctx(bo, ctx);
+	if (lret < 0) {
+		if (lret == -ERESTARTSYS)
+			return lret;
+		return 0;
+	}
+
+	if (bo->deleted)
+		lret = ttm_tt_backup(bo->bdev, tt, true, writeback);
+	else
+		lret = ttm_tt_backup(bo->bdev, tt, purge, writeback);
+	if (lret < 0 && lret != -EINTR)
+		return 0;
+
+	return lret;
+}
+EXPORT_SYMBOL(ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
index b1e03bfe4a68..1eba51bdd172 100644
@@ -112,6 +112,7 @@ xe-y += xe_bb.o \
 	xe_ring_ops.o \
 	xe_sa.o \
 	xe_sched_job.o \
+	xe_shrinker.o \
 	xe_step.o \
 	xe_sync.o \
 	xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +118 -0

127 lines not on this page · raw message

drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
index a324cde77db8..317fa923e287 100644
@@ -10,6 +10,7 @@
 static struct kunit_case xe_bo_tests[] = {
 	KUNIT_CASE(xe_ccs_migrate_kunit),
 	KUNIT_CASE(xe_bo_evict_kunit),
+	KUNIT_CASE_SLOW(xe_bo_shrink_kunit),
 	{}
 };
 
drivers/gpu/drm/xe/tests/xe_bo_test.h +1 -0
index 0113ab45066a..7f44d14a45c5 100644
@@ -10,5 +10,6 @@ struct kunit;
 
 void xe_ccs_migrate_kunit(struct kunit *test);
 void xe_bo_evict_kunit(struct kunit *test);
+void xe_bo_shrink_kunit(struct kunit *test);
 
 #endif
drivers/gpu/drm/xe/xe_bo.c +139 -16

282 lines not on this page · raw message

drivers/gpu/drm/xe/xe_bo.h +26 -0
index 6de894c728f5..8463e3f3f6f1 100644
@@ -63,6 +63,7 @@
 #define XE_BO_PROPS_INVALID	(-1)
 
 struct sg_table;
+struct xe_ttm_lru_walk;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -126,6 +127,28 @@ static inline struct xe_bo *xe_bo_get(struct xe_bo *bo)
 	return bo;
 }
 
+/*
+ * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
+ * xe bo
+ * @bo: The bo for which we want to obtain a refcount.
+ *
+ * There is a short window between where the bo's GEM object refcount reaches
+ * zero and where we put the final ttm_bo reference. Code in the eviction- and
+ * shrinking path should therefore attempt to grab a gem object reference before
+ * trying to use members outside of the base class ttm object. This function is
+ * intended for that purpose. On successful return, this function must be paired
+ * with an xe_bo_put().
+ *
+ * Return: @bo on success, NULL on failure.
+ */
+static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)
+{
+	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
+		return NULL;
+
+	return bo;
+}
+
 static inline void xe_bo_put(struct xe_bo *bo)
 {
 	if (bo)
@@ -315,6 +338,9 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)
 
 #define i915_gem_object_flush_if_display(obj)		((void)(obj))
 
+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		  bool purge, bool writeback);
+
 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given
drivers/gpu/drm/xe/xe_device.c +8 -0
index cfda7cb5df2c..58fecc4b0a18 100644
@@ -47,6 +47,7 @@
 #include "xe_perf.h"
 #include "xe_pm.h"
 #include "xe_query.h"
+#include "xe_shrinker.h"
 #include "xe_sriov.h"
 #include "xe_tile.h"
 #include "xe_ttm_stolen_mgr.h"
@@ -241,6 +242,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
 	if (xe->unordered_wq)
 		destroy_workqueue(xe->unordered_wq);
 
+	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
+		xe_shrinker_destroy(xe->mem.shrinker);
+
 	ttm_device_fini(&xe->ttm);
 }
 
@@ -270,6 +274,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 	if (err)
 		goto err;
 
+	xe->mem.shrinker = xe_shrinker_create(xe);
+	if (IS_ERR(xe->mem.shrinker))
+		return ERR_CAST(xe->mem.shrinker);
+
 	xe->info.devid = pdev->device;
 	xe->info.revid = pdev->revision;
 	xe->info.force_execlist = xe_modparam.force_execlist;
drivers/gpu/drm/xe/xe_device_types.h +2 -0
index c37be471d11c..3d5440aba52e 100644
@@ -325,6 +325,8 @@ struct xe_device {
 		struct xe_mem_region vram;
 		/** @mem.sys_mgr: system TTM manager */
 		struct ttm_resource_manager sys_mgr;
+		/** @mem.sys_mgr: system memory shrinker. */
+		struct xe_shrinker *shrinker;
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_shrinker.c new file +287 -0

287 lines not on this page · raw message

drivers/gpu/drm/xe/xe_shrinker.h new file +18 -0
new file mode 100644
index 000000000000..28a038f4fcbf
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_SHRINKER_H_
+#define _XE_SHRINKER_H_
+
+struct xe_shrinker;
+struct xe_device;
+
+void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable);
+
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
+
+void xe_shrinker_destroy(struct xe_shrinker *shrinker);
+
+#endif
include/drm/ttm/ttm_bo.h +3 -0
index e577528f5dfc..c7e81ae025d9 100644
@@ -229,6 +229,9 @@ struct ttm_lru_walk {
 long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
 			    struct ttm_resource_manager *man, long target);
 
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		       bool purge, bool writeback);
+
 /**
  * ttm_bo_get - reference a struct ttm_buffer_object
  *
Signature · 1 line
2.44.0
maintainer
#28 lore raw

Re: [PATCH v6 11/12] drm/ttm, drm/xe: Add a shrinker for xe bos

Quoted 88 lines · Thomas Hellström

Rather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
 drivers/gpu/drm/ttm/ttm_bo_util.c     |  67 ++++++
 drivers/gpu/drm/xe/Makefile           |   1 +
 drivers/gpu/drm/xe/tests/xe_bo.c      | 118 +++++++++++
 drivers/gpu/drm/xe/tests/xe_bo_test.c |   1 +
 drivers/gpu/drm/xe/tests/xe_bo_test.h |   1 +
 drivers/gpu/drm/xe/xe_bo.c            | 155 ++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h            |  26 +++
 drivers/gpu/drm/xe/xe_device.c        |   8 +
 drivers/gpu/drm/xe/xe_device_types.h  |   2 +
 drivers/gpu/drm/xe/xe_shrinker.c      | 287 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h      |  18 ++
 include/drm/ttm/ttm_bo.h              |   3 +
 12 files changed, 671 insertions(+), 16 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c
 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
drivers/gpu/drm/ttm/ttm_bo_util.c +36 -0
index c4f678f30fc2..563e96a4cf06 100644
@@ -924,3 +924,70 @@ long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
 
 	return progress;
 }
+EXPORT_SYMBOL(ttm_lru_walk_for_evict);
+
+/**
+ * ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer object.
+ * @walk: The struct xe_ttm_lru_walk that describes the walk.
+ * @bo: The buffer object.
+ * @purge: Whether to attempt to purge the bo content since it's no
+ * longer needed.
+ * @writeback: If !@purge, attempt to write out to persistent storage.
+ *
+ * The function uses the ttm_tt_back_up functionality to back up or
+ * purge a struct ttm_tt. If the bo is not in system, it's first
+ * moved there.
+ *
+ * Return: The number of pages shrunken or purged, or
+ * negative error code on failure.
+ */
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		       bool purge, bool writeback)
+{
+	static const struct ttm_place sys_placement_flags = {
+		.fpfn = 0,
+		.lpfn = 0,
+		.mem_type = TTM_PL_SYSTEM,
+		.flags = 0,
+	};
+	static struct ttm_placement sys_placement = {
+		.num_placement = 1,
+		.placement = &sys_placement_flags,
+	};
+	struct ttm_operation_ctx *ctx = walk->ctx;
+	struct ttm_tt *tt = bo->ttm;
+	long lret;
+
+	dma_resv_assert_held(bo->base.resv);
+

drivers/gpu/drm/ttm/ttm_bo_util.c long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev, · line 972

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +10 -0
+	if (!tt || !ttm_tt_is_populated(tt))
+		return 0;
+
+	if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+		int ret = ttm_bo_validate(bo, &sys_placement, ctx);
+
+		if (ret) {
+			if (ret == -EINTR || ret == -EDEADLK ||
+			    ret == -ERESTARTSYS)
+				return ret;

Can you explain the various error code returns / supression in this function?

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
+			return 0;
+		}
+	}
+
+	lret = ttm_bo_wait_ctx(bo, ctx);

drivers/gpu/drm/ttm/ttm_bo_util.c long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev, · line 987

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +10 -0
+	if (lret < 0) {
+		if (lret == -ERESTARTSYS)
+			return lret;
+		return 0;
+	}
+
+	if (bo->deleted)
+		lret = ttm_tt_backup(bo->bdev, tt, true, writeback);
+	else
+		lret = ttm_tt_backup(bo->bdev, tt, purge, writeback);

Hmm, missed this in my previous review. It is frowned upon having multiple bools as arguments. Could this be reworked with flags? Same goes for all functions in the series with multiple bool arguments.

Quoted 197 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +6 -0
+	if (lret < 0 && lret != -EINTR)
+		return 0;
+
+	return lret;
+}
+EXPORT_SYMBOL(ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
index b1e03bfe4a68..1eba51bdd172 100644
@@ -112,6 +112,7 @@ xe-y += xe_bb.o \
 	xe_ring_ops.o \
 	xe_sa.o \
 	xe_sched_job.o \
+	xe_shrinker.o \
 	xe_step.o \
 	xe_sync.o \
 	xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +118 -0
index 9f3c02826464..49617f16dc76 100644
@@ -6,6 +6,8 @@
 #include <kunit/test.h>
 #include <kunit/visibility.h>
 
+#include <uapi/linux/sysinfo.h>
+
 #include "tests/xe_bo_test.h"
 #include "tests/xe_pci_test.h"
 #include "tests/xe_test.h"
@@ -350,3 +352,119 @@ void xe_bo_evict_kunit(struct kunit *test)
 	xe_call_for_each_device(evict_test_run_device);
 }
 EXPORT_SYMBOL_IF_KUNIT(xe_bo_evict_kunit);
+
+struct xe_bo_link {
+	struct list_head link;
+	struct xe_bo *bo;
+};
+
+#define XE_BO_SHRINK_SIZE ((unsigned long)SZ_64M)
+
+/*
+ * Try to create system bos corresponding to twice the amount
+ * of available system memory to test shrinker functionality.
+ * If no swap space is available to accommodate the
+ * memory overcommit, mark bos purgeable.
+ */
+static int shrink_test_run_device(struct xe_device *xe)
+{
+	struct kunit *test = xe_cur_kunit();
+	LIST_HEAD(bos);
+	struct xe_bo_link *link, *next;
+	struct sysinfo si;
+	size_t total, alloced;
+	unsigned int interrupted = 0, successful = 0;
+
+	si_meminfo(&si);
+	total = si.freeram * si.mem_unit;
+
+	kunit_info(test, "Free ram is %lu bytes. Will allocate twice of that.\n",
+		   (unsigned long) total);
+
+	total <<= 1;
+	for (alloced = 0; alloced < total ; alloced += XE_BO_SHRINK_SIZE) {
+		struct xe_bo *bo;
+		unsigned int mem_type;
+
+		link = kzalloc(sizeof(*link), GFP_KERNEL);
+		if (!link) {
+			KUNIT_FAIL(test, "Unexpeced link allocation failure\n");
+			break;
+		}
+
+		INIT_LIST_HEAD(&link->link);
+
+		/* We can create bos using WC caching here. But it is slower. */
+		bo = xe_bo_create_user(xe, NULL, NULL, XE_BO_SHRINK_SIZE,
+				       DRM_XE_GEM_CPU_CACHING_WB,
+				       ttm_bo_type_device,
+				       XE_BO_FLAG_SYSTEM);
+		if (IS_ERR(bo)) {
+			if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
+			    bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
+				KUNIT_FAIL(test, "Error creating bo: %pe\n", bo);
+			kfree(link);
+			break;
+		}
+		link->bo = bo;
+		list_add_tail(&link->link, &bos);
+		xe_bo_lock(bo, false);
+
+		/*
+		 * If we're low on swap entries, we can't shrink unless the bo
+		 * is marked purgeable.
+		 */
+		if (get_nr_swap_pages() < (XE_BO_SHRINK_SIZE >> PAGE_SHIFT) * 128) {
+			struct xe_ttm_tt *xe_tt =
+				container_of(bo->ttm.ttm, typeof(*xe_tt), ttm);
+			long num_pages = xe_tt->ttm.num_pages;
+
+			xe_tt->purgeable = true;
+			xe_shrinker_mod_pages(xe->mem.shrinker, -num_pages,
+					      num_pages);
+		}
+
+		mem_type = bo->ttm.resource->mem_type;
+		xe_bo_unlock(bo);
+		if (mem_type != XE_PL_TT)
+			KUNIT_FAIL(test, "Bo in incorrect memory type: %u\n",
+				   bo->ttm.resource->mem_type);
+		cond_resched();
+		if (signal_pending(current))
+			break;
+	}
+
+	/* Read back and destroy bos */
+	list_for_each_entry_safe_reverse(link, next, &bos, link) {
+		static struct ttm_operation_ctx ctx = {.interruptible = true};
+		struct xe_bo *bo = link->bo;
+		int ret;
+
+		if (!signal_pending(current)) {
+			xe_bo_lock(bo, NULL);
+			ret = ttm_bo_validate(&bo->ttm, &tt_placement, &ctx);
+			xe_bo_unlock(bo);
+			if (ret && ret != -EINTR)
+				KUNIT_FAIL(test, "Validation failed: %pe\n",
+					   ERR_PTR(ret));
+			else if (ret)
+				interrupted++;
+			else
+				successful++;
+		}
+		xe_bo_put(link->bo);
+		list_del(&link->link);
+		kfree(link);
+		cond_resched();
+	}
+	kunit_info(test, "Readbacks interrupted: %u successful: %u\n",
+		   interrupted, successful);
+
+	return 0;
+}
+
+void xe_bo_shrink_kunit(struct kunit *test)
+{
+	xe_call_for_each_device(shrink_test_run_device);
+}
+EXPORT_SYMBOL_IF_KUNIT(xe_bo_shrink_kunit);
drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
index a324cde77db8..317fa923e287 100644
@@ -10,6 +10,7 @@
 static struct kunit_case xe_bo_tests[] = {
 	KUNIT_CASE(xe_ccs_migrate_kunit),
 	KUNIT_CASE(xe_bo_evict_kunit),
+	KUNIT_CASE_SLOW(xe_bo_shrink_kunit),
 	{}
 };
 
drivers/gpu/drm/xe/tests/xe_bo_test.h +1 -0
index 0113ab45066a..7f44d14a45c5 100644
@@ -10,5 +10,6 @@ struct kunit;
 
 void xe_ccs_migrate_kunit(struct kunit *test);
 void xe_bo_evict_kunit(struct kunit *test);
+void xe_bo_shrink_kunit(struct kunit *test);
 
 #endif
drivers/gpu/drm/xe/xe_bo.c +2 -0
index 65c696966e96..6ab63d1642ae 100644
@@ -10,6 +10,7 @@
 #include <drm/drm_drv.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/drm_managed.h>
+#include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
 #include <drm/ttm/ttm_tt.h>
@@ -25,6 +26,7 @@
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
 #include "xe_res_cursor.h"
+#include "xe_shrinker.h"
 #include "xe_trace_bo.h"
 #include "xe_ttm_stolen_mgr.h"
 #include "xe_vm.h"
@@ -278,11 +280,15 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo,
 	}
 }

drivers/gpu/drm/xe/xe_bo.c static void xe_evict_flags(struct ttm_buffer_object *tbo, · line 290

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4 -1
 
+/* struct xe_ttm_tt - Subclassed ttm_tt for xe */
 struct xe_ttm_tt {
 	struct ttm_tt ttm;
-	struct device *dev;
+	/** @xe - The xe device */
+	struct xe_device *xe;
 	struct sg_table sgt;
 	struct sg_table *sg;
+	/** @purgeable - Whether the bo is purgeable (WONTNEED) */

So we need to add WONTNEED to our uAPI, right?

Quoted 40 lines
drivers/gpu/drm/xe/xe_bo.c +6 -4
+	bool purgeable;
 };
 
 static int xe_tt_map_sg(struct ttm_tt *tt)
@@ -291,7 +297,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
 	unsigned long num_pages = tt->num_pages;
 	int ret;
 
-	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
+	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+		   !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE));
 
 	if (xe_tt->sg)
 		return 0;
@@ -299,13 +306,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
 	ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt->pages,
 						num_pages, 0,
 						(u64)num_pages << PAGE_SHIFT,
-						xe_sg_segment_size(xe_tt->dev),
+						xe_sg_segment_size(xe_tt->xe->drm.dev),
 						GFP_KERNEL);
 	if (ret)
 		return ret;
 
 	xe_tt->sg = &xe_tt->sgt;
-	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg, DMA_BIDIRECTIONAL,
+	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg, DMA_BIDIRECTIONAL,
 			      DMA_ATTR_SKIP_CPU_SYNC);
 	if (ret) {
 		sg_free_table(xe_tt->sg);
@@ -321,7 +328,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
-		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
+		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
 		xe_tt->sg = NULL;

drivers/gpu/drm/xe/xe_bo.c struct sg_table *xe_bo_sg(struct xe_bo *bo) · line 351

Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
@@ -336,21 +343,41 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
 	return xe_tt->sg;
 }
 
+/*
+ * Account ttm pages against the device shrinker's shrinkable and
+ * purgeable counts.
+ */
+static void xe_ttm_tt_account(struct ttm_tt *tt, bool add)
+{

Again I think bools are frowned upon as arguments. Maybe just two functions - add / sub?

Quoted 45 lines
drivers/gpu/drm/xe/xe_bo.c +22 -5
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	long num_pages = tt->num_pages;
+
+	if (!add)
+		num_pages = -num_pages;
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, num_pages);
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, num_pages, 0);
+}
+
 static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 				       u32 page_flags)
 {
 	struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
 	struct xe_device *xe = xe_bo_device(bo);
-	struct xe_ttm_tt *tt;
+	struct xe_ttm_tt *xe_tt;
+	struct ttm_tt *tt;
 	unsigned long extra_pages;
 	enum ttm_caching caching;
 	int err;
 
-	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
-	if (!tt)
+	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
+	if (!xe_tt)
 		return NULL;
 
-	tt->dev = xe->drm.dev;
+	tt = &xe_tt->ttm;
+	xe_tt->xe = xe;
 
 	extra_pages = 0;
 	if (xe_bo_needs_ccs_pages(bo))
@@ -387,42 +414,128 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 		caching = ttm_uncached;
 	}
 
-	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching, extra_pages);
+	if (ttm_bo->type != ttm_bo_type_sg)
+		page_flags |= TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE;
+
+	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching, extra_pages);

drivers/gpu/drm/xe/xe_bo.c static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, · line 428

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4 -2
 	if (err) {
-		kfree(tt);
+		kfree(xe_tt);
 		return NULL;
 	}
 
-	return &tt->ttm;
+	tt->backup = ttm_backup_shmem_create(tt->num_pages << PAGE_SHIFT);
+	if (IS_ERR(tt->backup)) {
+		ttm_tt_fini(tt);

Mentioned this the previous review I think you need set tt->backup to NULL here or update ttm_tt_fini to understand IS_ERR(tt->backup).

Also maybe dump question but could we just have a global backup for all BOs? Would that be better than each BO creating its own backup?

Quoted 87 lines
drivers/gpu/drm/xe/xe_bo.c +58 -4
+		kfree(xe_tt);
+		return NULL;
+	}
+
+	return tt;
 }
 
 static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 			      struct ttm_operation_ctx *ctx)
 {
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 	int err;
 
 	/*
 	 * dma-bufs are not populated with pages, and the dma-
 	 * addresses are set up when moved to XE_PL_TT.
 	 */
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return 0;
 
 	err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	if (err)
 		return err;
 
-	return err;
+	xe_tt->purgeable = false;
+	xe_ttm_tt_account(tt, true);
+
+	return 0;
 }
 
 static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return;
 
 	xe_tt_unmap_sg(tt);
 
-	return ttm_pool_free(&ttm_dev->pool, tt);
+	ttm_pool_free(&ttm_dev->pool, tt);
+	xe_ttm_tt_account(tt, false);
+}
+
+/**
+ * xe_bo_shrink() - Try to shrink an xe bo.
+ * @walk:  - The walk parameters
+ * @bo: The TTM buffer object
+ * @purge: Only consider purgeable bos.
+ * @writeback: Try to write back to persistent storage.
+ *
+ * Try to shrink- or purge a bo, and if it succeeds, unmap dma.
+ * Note that we need to be able to handle also non xe bos
+ * (ghost bos), but only if the struct ttm_tt is embedded in
+ * a struct xe_ttm_tt.
+ *
+ * Return: The number of pages shrunken or purged, or negative error
+ * code on failure.
+ */
+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		  bool purge, bool writeback)
+{
+	struct ttm_tt *tt = bo->ttm;
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	struct ttm_place place = {.mem_type = bo->resource->mem_type};
+	struct xe_bo *xe_bo = ttm_to_xe_bo(bo);
+	struct xe_device *xe = xe_tt->xe;
+	bool needs_rpm;
+	long lret = 0L;
+
+	if (!tt || !ttm_tt_is_populated(tt) ||
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE) ||
+	    (purge && !xe_tt->purgeable))
+		return 0L;
+
+	if (!ttm_bo_eviction_valuable(bo, &place))
+		return 0L;
+
+	/* Beware of zombies (GEM object refcount == 0) and ghosts. */
+	if (!xe_bo_is_xe_bo(bo) || !xe_bo_get_unless_zero(xe_bo)) {
+		struct ttm_placement null_placement = { .num_placement = 0 };
+
+		lret = ttm_bo_wait_ctx(bo, walk->ctx);
+		if (lret)

drivers/gpu/drm/xe/xe_bo.c static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, · line 521

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+			return lret;
+
+		/* Purge the bo content! */
+		ttm_bo_validate(bo, &null_placement, walk->ctx);
+		return tt->num_pages;
+	}
+
+	/* System CCS needs gpu copy when moving PL_TT -> PL_SYSTEM */
+	needs_rpm = (!IS_DGFX(xe) && bo->resource->mem_type != XE_PL_SYSTEM &&
+		     xe_bo && xe_bo_needs_ccs_pages(xe_bo) && !xe_tt->purgeable);

Is xe_bo check really needed here?

Quoted 21 lines
drivers/gpu/drm/xe/xe_bo.c +17 -0
+	if (needs_rpm && !xe_pm_runtime_get_if_active(xe))
+		goto out_unref;
+
+	lret = ttm_bo_try_shrink(walk, bo, xe_tt->purgeable, writeback);
+	if (needs_rpm)
+		xe_pm_runtime_put(xe);
+
+	if (lret > 0) {
+		xe_assert(xe, !ttm_tt_is_populated(tt));
+
+		xe_ttm_tt_account(tt, false);
+	}
+
+out_unref:
+	xe_bo_put(xe_bo);
+
+	return lret;
 }
 
 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)

drivers/gpu/drm/xe/xe_bo.c struct xe_bo *___xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo, · line 1354

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -1238,6 +1351,7 @@ struct xe_bo *___xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo,
 	struct ttm_operation_ctx ctx = {
 		.interruptible = true,
 		.no_wait_gpu = false,
+		.gfp_retry_mayfail = true,

Can you explain why you are setting this?

Quoted 22 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
 	};
 	struct ttm_placement *placement;
 	uint32_t alignment;
@@ -1681,6 +1795,8 @@ int xe_bo_pin_external(struct xe_bo *bo)
 	}
 
 	ttm_bo_pin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -1739,6 +1855,8 @@ int xe_bo_pin(struct xe_bo *bo)
 	}
 
 	ttm_bo_pin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c void xe_bo_unpin_external(struct xe_bo *bo) · line 1896

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +3 -0
@@ -1773,6 +1891,9 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	spin_unlock(&xe->pinned.lock);
 
 	ttm_bo_unpin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);
+

Nit: Extra newline.

Quoted 347 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -1801,6 +1922,8 @@ void xe_bo_unpin(struct xe_bo *bo)
 	}
 
 	ttm_bo_unpin(&bo->ttm);
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);
 }
 
 /**
drivers/gpu/drm/xe/xe_bo.h +26 -0
index 6de894c728f5..8463e3f3f6f1 100644
@@ -63,6 +63,7 @@
 #define XE_BO_PROPS_INVALID	(-1)
 
 struct sg_table;
+struct xe_ttm_lru_walk;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -126,6 +127,28 @@ static inline struct xe_bo *xe_bo_get(struct xe_bo *bo)
 	return bo;
 }
 
+/*
+ * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
+ * xe bo
+ * @bo: The bo for which we want to obtain a refcount.
+ *
+ * There is a short window between where the bo's GEM object refcount reaches
+ * zero and where we put the final ttm_bo reference. Code in the eviction- and
+ * shrinking path should therefore attempt to grab a gem object reference before
+ * trying to use members outside of the base class ttm object. This function is
+ * intended for that purpose. On successful return, this function must be paired
+ * with an xe_bo_put().
+ *
+ * Return: @bo on success, NULL on failure.
+ */
+static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)
+{
+	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
+		return NULL;
+
+	return bo;
+}
+
 static inline void xe_bo_put(struct xe_bo *bo)
 {
 	if (bo)
@@ -315,6 +338,9 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)
 
 #define i915_gem_object_flush_if_display(obj)		((void)(obj))
 
+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		  bool purge, bool writeback);
+
 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given
drivers/gpu/drm/xe/xe_device.c +8 -0
index cfda7cb5df2c..58fecc4b0a18 100644
@@ -47,6 +47,7 @@
 #include "xe_perf.h"
 #include "xe_pm.h"
 #include "xe_query.h"
+#include "xe_shrinker.h"
 #include "xe_sriov.h"
 #include "xe_tile.h"
 #include "xe_ttm_stolen_mgr.h"
@@ -241,6 +242,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
 	if (xe->unordered_wq)
 		destroy_workqueue(xe->unordered_wq);
 
+	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
+		xe_shrinker_destroy(xe->mem.shrinker);
+
 	ttm_device_fini(&xe->ttm);
 }
 
@@ -270,6 +274,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 	if (err)
 		goto err;
 
+	xe->mem.shrinker = xe_shrinker_create(xe);
+	if (IS_ERR(xe->mem.shrinker))
+		return ERR_CAST(xe->mem.shrinker);
+
 	xe->info.devid = pdev->device;
 	xe->info.revid = pdev->revision;
 	xe->info.force_execlist = xe_modparam.force_execlist;
drivers/gpu/drm/xe/xe_device_types.h +2 -0
index c37be471d11c..3d5440aba52e 100644
@@ -325,6 +325,8 @@ struct xe_device {
 		struct xe_mem_region vram;
 		/** @mem.sys_mgr: system TTM manager */
 		struct ttm_resource_manager sys_mgr;
+		/** @mem.sys_mgr: system memory shrinker. */
+		struct xe_shrinker *shrinker;
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_shrinker.c new file +232 -0
new file mode 100644
index 000000000000..3f9554bdc06b
@@ -0,0 +1,287 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <linux/shrinker.h>
+#include <linux/swap.h>
+
+#include <drm/ttm/ttm_bo.h>
+#include <drm/ttm/ttm_tt.h>
+
+#include "xe_bo.h"
+#include "xe_pm.h"
+#include "xe_shrinker.h"
+
+/**
+ * struct xe_shrinker - per-device shrinker
+ * @xe: Back pointer to the device.
+ * @lock: Lock protecting accounting.
+ * @shrinkable_pages: Number of pages that are currently shrinkable.
+ * @purgeable_pages: Number of pages that are currently purgeable.
+ * @shrink: Pointer to the mm shrinker.
+ * @pm_worker: Worker to wake up the device if required.
+ */
+struct xe_shrinker {
+	struct xe_device *xe;
+	rwlock_t lock;
+	long shrinkable_pages;
+	long purgeable_pages;
+	struct shrinker *shrink;
+	struct work_struct pm_worker;
+};
+
+/**
+ * struct xe_shrink_lru_walk - lru_walk subclass for shrinker
+ * @walk: The embedded base class.
+ * @xe: Pointer to the xe device.
+ * @purge: Purgeable only request from the srinker.
+ * @writeback: Try to write back to persistent storage.
+ */
+struct xe_shrink_lru_walk {
+	struct ttm_lru_walk walk;
+	struct xe_device *xe;
+	bool purge;
+	bool writeback;
+};
+
+static struct xe_shrinker *to_xe_shrinker(struct shrinker *shrink)
+{
+	return shrink->private_data;
+}
+
+static struct xe_shrink_lru_walk *
+to_xe_shrink_lru_walk(struct ttm_lru_walk *walk)
+{
+	return container_of(walk, struct xe_shrink_lru_walk, walk);
+}
+
+/**
+ * xe_shrinker_mod_pages() - Modify shrinker page accounting
+ * @shrinker: Pointer to the struct xe_shrinker.
+ * @shrinkable: Shrinkable pages delta. May be negative.
+ * @purgeable: Purgeable page delta. May be negative.
+ *
+ * Modifies the shrinkable and purgeable pages accounting.
+ */
+void
+xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable)
+{
+	write_lock(&shrinker->lock);
+	shrinker->shrinkable_pages += shrinkable;
+	shrinker->purgeable_pages += purgeable;
+	write_unlock(&shrinker->lock);
+}
+
+static long xe_shrinker_process_bo(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
+{
+	struct xe_shrink_lru_walk *shrink_walk = to_xe_shrink_lru_walk(walk);
+
+	return xe_bo_shrink(walk, bo, shrink_walk->purge, shrink_walk->writeback);
+}
+
+static long xe_shrinker_walk(struct xe_shrink_lru_walk *shrink_walk, long target)
+{
+	struct xe_device *xe = shrink_walk->xe;
+	struct ttm_resource_manager *man;
+	unsigned int mem_type;
+	long sofar = 0;
+	long lret;
+
+	for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT; ++mem_type) {
+		man = ttm_manager_type(&xe->ttm, mem_type);
+		if (!man || !man->use_tt)
+			continue;
+
+		lret = ttm_lru_walk_for_evict(&shrink_walk->walk, &xe->ttm, man, target);
+		if (lret < 0)
+			return lret;
+
+		sofar += lret;
+		if (sofar >= target)
+			break;
+	}
+
+	return sofar;
+}
+
+static unsigned long
+xe_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	unsigned long num_pages;
+
+	num_pages = get_nr_swap_pages();
+	read_lock(&shrinker->lock);
+	num_pages = min_t(unsigned long, num_pages, shrinker->shrinkable_pages);
+	num_pages += shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	return num_pages ? num_pages : SHRINK_EMPTY;
+}
+
+static const struct ttm_lru_walk_ops xe_shrink_ops = {
+	.process_bo = xe_shrinker_process_bo,
+};
+
+/*
+ * Check if we need runtime pm, and if so try to grab a reference if
+ * already active. If grabbing a reference fails, queue a worker that
+ * does it for us outside of reclaim, but don't wait for it to complete.
+ * If bo shrinking needs an rpm reference and we don't have it (yet),
+ * that bo will be skipped anyway.
+ */
+static bool xe_shrinker_runtime_pm_get(struct xe_shrinker *shrinker, bool force,
+				       unsigned long nr_to_scan)
+{
+	struct xe_device *xe = shrinker->xe;
+
+	if (IS_DGFX(xe) || !xe_device_has_flat_ccs(xe) ||
+	    !get_nr_swap_pages())
+		return false;
+
+	if (!force) {
+		read_lock(&shrinker->lock);
+		force = (nr_to_scan > shrinker->purgeable_pages);
+		read_unlock(&shrinker->lock);
+		if (!force)
+			return false;
+	}
+
+	if (!xe_pm_runtime_get_if_active(xe)) {
+		queue_work(xe->unordered_wq, &shrinker->pm_worker);
+		return false;
+	}
+
+	return true;
+}
+
+static void xe_shrinker_runtime_pm_put(struct xe_shrinker *shrinker, bool runtime_pm)
+{
+	if (runtime_pm)
+		xe_pm_runtime_put(shrinker->xe);
+}
+
+static unsigned long xe_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	bool is_kswapd = current_is_kswapd();
+	struct ttm_operation_ctx ctx = {
+		.interruptible = false,
+		.no_wait_gpu = !is_kswapd,
+	};
+	unsigned long nr_to_scan, freed = 0;
+	struct xe_shrink_lru_walk shrink_walk = {
+		.walk = {
+			.ops = &xe_shrink_ops,
+			.ctx = &ctx,
+			.trylock_only = true,
+		},
+		.xe = shrinker->xe,
+		.purge = true,
+		.writeback = is_kswapd,
+	};
+	bool runtime_pm;
+	bool purgeable;
+	long ret;
+
+	sc->nr_scanned = 0;
+	nr_to_scan = sc->nr_to_scan;
+
+	read_lock(&shrinker->lock);
+	purgeable = !!shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	/* Might need runtime PM. Try to wake early if it looks like it. */
+	runtime_pm = xe_shrinker_runtime_pm_get(shrinker, false, nr_to_scan);
+
+	while (purgeable && freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+	if (freed < nr_to_scan)
+		nr_to_scan -= freed;
+	else
+		nr_to_scan = 0;
+	if (!nr_to_scan)
+		goto out;
+
+	/* If we didn't wake before, try to do it now if needed. */
+	if (!runtime_pm)
+		runtime_pm = xe_shrinker_runtime_pm_get(shrinker, true, 0);
+
+	shrink_walk.purge = false;
+	nr_to_scan = sc->nr_to_scan;
+	while (freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+
+out:
+	xe_shrinker_runtime_pm_put(shrinker, runtime_pm);
+	return freed ? freed : SHRINK_STOP;

drivers/gpu/drm/xe/xe_shrinker.c line 242

Quoted 10 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +10 -0
+}
+
+/* Wake up the device for shrinking. */
+static void xe_shrinker_pm(struct work_struct *work)
+{
+	struct xe_shrinker *shrinker =
+		container_of(work, typeof(*shrinker), pm_worker);
+
+	xe_pm_runtime_get(shrinker->xe);
+	xe_pm_runtime_put(shrinker->xe);

So I don't really get this. How does this help the shrinker get a PM ref? The small window between xe_pm_runtime_get / put the shrinker grabs one via xe_pm_runtime_get_if_active? Seems fragile.

Matt

Quoted 86 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +45 -0
+}
+
+/**
+ * xe_shrinker_create() - Create an xe per-device shrinker
+ * @xe: Pointer to the xe device.
+ *
+ * Returns: A pointer to the created shrinker on success,
+ * Negative error code on failure.
+ */
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe)
+{
+	struct xe_shrinker *shrinker = kzalloc(sizeof(*shrinker), GFP_KERNEL);
+
+	if (!shrinker)
+		return ERR_PTR(-ENOMEM);
+
+	shrinker->shrink = shrinker_alloc(0, "xe system shrinker");
+	if (!shrinker->shrink) {
+		kfree(shrinker);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm);
+	shrinker->xe = xe;
+	rwlock_init(&shrinker->lock);
+	shrinker->shrink->count_objects = xe_shrinker_count;
+	shrinker->shrink->scan_objects = xe_shrinker_scan;
+	shrinker->shrink->private_data = shrinker;
+	shrinker_register(shrinker->shrink);
+
+	return shrinker;
+}
+
+/**
+ * xe_shrinker_destroy() - Destroy an xe per-device shrinker
+ * @shrinker: Pointer to the shrinker to destroy.
+ */
+void xe_shrinker_destroy(struct xe_shrinker *shrinker)
+{
+	xe_assert(shrinker->xe, !shrinker->shrinkable_pages);
+	xe_assert(shrinker->xe, !shrinker->purgeable_pages);
+	shrinker_free(shrinker->shrink);
+	flush_work(&shrinker->pm_worker);
+	kfree(shrinker);
+}
drivers/gpu/drm/xe/xe_shrinker.h new file +18 -0
new file mode 100644
index 000000000000..28a038f4fcbf
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_SHRINKER_H_
+#define _XE_SHRINKER_H_
+
+struct xe_shrinker;
+struct xe_device;
+
+void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long shrinkable, long purgeable);
+
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
+
+void xe_shrinker_destroy(struct xe_shrinker *shrinker);
+
+#endif
include/drm/ttm/ttm_bo.h +3 -0
index e577528f5dfc..c7e81ae025d9 100644
@@ -229,6 +229,9 @@ struct ttm_lru_walk {
 long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
 			    struct ttm_resource_manager *man, long target);
 
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		       bool purge, bool writeback);
+
 /**
  * ttm_bo_get - reference a struct ttm_buffer_object
  *
Signature · 1 line
2.44.0
author maintainer
#29 lore raw

Re: [PATCH v6 11/12] drm/ttm, drm/xe: Add a shrinker for xe bos

Quoted 103 lines · Matthew Brost
Quoted 102 lines · Thomas Hellström

Rather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
 drivers/gpu/drm/ttm/ttm_bo_util.c     |  67 ++++++ drivers/gpu/drm/xe/Makefile           |   1 + drivers/gpu/drm/xe/tests/xe_bo.c      | 118 +++++++++++ drivers/gpu/drm/xe/tests/xe_bo_test.c |   1 + drivers/gpu/drm/xe/tests/xe_bo_test.h |   1 + drivers/gpu/drm/xe/xe_bo.c            | 155 ++++++++++++-- drivers/gpu/drm/xe/xe_bo.h            |  26 +++ drivers/gpu/drm/xe/xe_device.c        |   8 + drivers/gpu/drm/xe/xe_device_types.h  |   2 + drivers/gpu/drm/xe/xe_shrinker.c      | 287
++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h      |  18 ++ include/drm/ttm/ttm_bo.h              |   3 +

 12 files changed, 671 insertions(+), 16 deletions(-)

 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h

drivers/gpu/drm/ttm/ttm_bo_util.c +0 -0
index c4f678f30fc2..563e96a4cf06 100644
@@ -924,3 +924,70 @@ long ttm_lru_walk_for_evict(struct
 ttm_lru_walk *walk, struct ttm_device *bdev,


  return progress;
 }

drivers/gpu/drm/ttm/ttm_bo_util.c +4 -0
+EXPORT_SYMBOL(ttm_lru_walk_for_evict);
+
+/**
+ * ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer

object.
+ * @walk: The struct xe_ttm_lru_walk that describes the walk.
+ * @bo: The buffer object.
+ * @purge: Whether to attempt to purge the bo content since it's no
+ * longer needed.
+ * @writeback: If !@purge, attempt to write out to persistent storage.
+ *
+ * The function uses the ttm_tt_back_up functionality to back up or
+ * purge a struct ttm_tt. If the bo is not in system, it's first
+ * moved there.
+ *
+ * Return: The number of pages shrunken or purged, or
+ * negative error code on failure.
+ */
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/ttm/ttm_bo_util.c +23 -0
+		       bool purge, bool writeback)
+{
+	static const struct ttm_place sys_placement_flags = {
+		.fpfn = 0,
+		.lpfn = 0,
+		.mem_type = TTM_PL_SYSTEM,
+		.flags = 0,
+	};
+	static struct ttm_placement sys_placement = {
+		.num_placement = 1,
+		.placement = &sys_placement_flags,
+	};
+	struct ttm_operation_ctx *ctx = walk->ctx;
+	struct ttm_tt *tt = bo->ttm;
+	long lret;
+
+	dma_resv_assert_held(bo->base.resv);
+
+	if (!tt || !ttm_tt_is_populated(tt))
+		return 0;
+
+	if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+		int ret = ttm_bo_validate(bo, &sys_placement,

ctx);

Quoted 8 lines

drivers/gpu/drm/ttm/ttm_bo_util.c long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev, · line 972

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
+
+		if (ret) {
+			if (ret == -EINTR || ret == -EDEADLK ||
+			    ret == -ERESTARTSYS)
+				return ret;

Can you explain the various error code returns / supression in this function?

Want me to add a comment in the code or inline here? Anyway, the error codes are codes for which the caller wants to restart. (Signal delivery or deadlock). For other errors just move on to the next bo on the LRU list.

Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +13 -0
+			return 0;
+		}
+	}
+
+	lret = ttm_bo_wait_ctx(bo, ctx);
+	if (lret < 0) {
+		if (lret == -ERESTARTSYS)
+			return lret;
+		return 0;
+	}
+
+	if (bo->deleted)
+		lret = ttm_tt_backup(bo->bdev, tt, true,
Quoted 8 lines
Quoted 4 lines

writeback);

drivers/gpu/drm/ttm/ttm_bo_util.c +2 -0
+	else
+		lret = ttm_tt_backup(bo->bdev, tt, purge,

writeback);

Hmm, missed this in my previous review. It is frowned upon having multiple bools as arguments. Could this be reworked with flags? Same goes for all functions in the series with multiple bool arguments.

I agree. Ill see if I can make this look betteer.

Quoted 225 lines
Quoted 225 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +6 -0
+	if (lret < 0 && lret != -EINTR)
+		return 0;
+
+	return lret;
+}
+EXPORT_SYMBOL(ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
index b1e03bfe4a68..1eba51bdd172 100644
@@ -112,6 +112,7 @@ xe-y += xe_bb.o \
  	xe_ring_ops.o \
  	xe_sa.o \
  	xe_sched_job.o \
+	xe_shrinker.o \
  	xe_step.o \
  	xe_sync.o \
  	xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +0 -0
index 9f3c02826464..49617f16dc76 100644
@@ -6,6 +6,8 @@
  #include <kunit/test.h>
  #include <kunit/visibility.h>

drivers/gpu/drm/xe/tests/xe_bo.c +2 -0
+#include <uapi/linux/sysinfo.h>
+

 #include "tests/xe_bo_test.h"
 #include "tests/xe_pci_test.h"
 #include "tests/xe_test.h"

drivers/gpu/drm/xe/tests/xe_bo.c +27 -0
@@ -350,3 +352,119 @@ void xe_bo_evict_kunit(struct kunit *test)
  	xe_call_for_each_device(evict_test_run_device);
  }
  EXPORT_SYMBOL_IF_KUNIT(xe_bo_evict_kunit);
+
+struct xe_bo_link {
+	struct list_head link;
+	struct xe_bo *bo;
+};
+
+#define XE_BO_SHRINK_SIZE ((unsigned long)SZ_64M)
+
+/*
+ * Try to create system bos corresponding to twice the amount
+ * of available system memory to test shrinker functionality.
+ * If no swap space is available to accommodate the
+ * memory overcommit, mark bos purgeable.
+ */
+static int shrink_test_run_device(struct xe_device *xe)
+{
+	struct kunit *test = xe_cur_kunit();
+	LIST_HEAD(bos);
+	struct xe_bo_link *link, *next;
+	struct sysinfo si;
+	size_t total, alloced;
+	unsigned int interrupted = 0, successful = 0;
+
+	si_meminfo(&si);
+	total = si.freeram * si.mem_unit;
+
+	kunit_info(test, "Free ram is %lu bytes. Will allocate

twice of that.\n",

drivers/gpu/drm/xe/tests/xe_bo.c +4 -0
+		   (unsigned long) total);
+
+	total <<= 1;
+	for (alloced = 0; alloced < total ; alloced +=

XE_BO_SHRINK_SIZE) {

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+		struct xe_bo *bo;
+		unsigned int mem_type;
+
+		link = kzalloc(sizeof(*link), GFP_KERNEL);
+		if (!link) {
+			KUNIT_FAIL(test, "Unexpeced link

allocation failure\n");

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+			break;
+		}
+
+		INIT_LIST_HEAD(&link->link);
+
+		/* We can create bos using WC caching here. But it

is slower. */

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+		bo = xe_bo_create_user(xe, NULL, NULL,

XE_BO_SHRINK_SIZE,

drivers/gpu/drm/xe/tests/xe_bo.c +5 -0
+				       DRM_XE_GEM_CPU_CACHING_WB,
+				       ttm_bo_type_device,
+				       XE_BO_FLAG_SYSTEM);
+		if (IS_ERR(bo)) {
+			if (bo != ERR_PTR(-ENOMEM) && bo !=

ERR_PTR(-ENOSPC) &&

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+			    bo != ERR_PTR(-EINTR) && bo !=

ERR_PTR(-ERESTARTSYS))

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+				KUNIT_FAIL(test, "Error creating

bo: %pe\n", bo);

drivers/gpu/drm/xe/tests/xe_bo.c +9 -0
+			kfree(link);
+			break;
+		}
+		link->bo = bo;
+		list_add_tail(&link->link, &bos);
+		xe_bo_lock(bo, false);
+
+		/*
+		 * If we're low on swap entries, we can't shrink

unless the bo

drivers/gpu/drm/xe/tests/xe_bo.c +3 -0
+		 * is marked purgeable.
+		 */
+		if (get_nr_swap_pages() < (XE_BO_SHRINK_SIZE >>

PAGE_SHIFT) * 128) {

drivers/gpu/drm/xe/tests/xe_bo.c +2 -0
+			struct xe_ttm_tt *xe_tt =
+				container_of(bo->ttm.ttm,

typeof(*xe_tt), ttm);

drivers/gpu/drm/xe/tests/xe_bo.c +4 -0
+			long num_pages = xe_tt->ttm.num_pages;
+
+			xe_tt->purgeable = true;
+			xe_shrinker_mod_pages(xe->mem.shrinker, -

num_pages,

drivers/gpu/drm/xe/tests/xe_bo.c +7 -0
+					      num_pages);
+		}
+
+		mem_type = bo->ttm.resource->mem_type;
+		xe_bo_unlock(bo);
+		if (mem_type != XE_PL_TT)
+			KUNIT_FAIL(test, "Bo in incorrect memory

type: %u\n",

drivers/gpu/drm/xe/tests/xe_bo.c +9 -0
+				   bo->ttm.resource->mem_type);
+		cond_resched();
+		if (signal_pending(current))
+			break;
+	}
+
+	/* Read back and destroy bos */
+	list_for_each_entry_safe_reverse(link, next, &bos, link) {
+		static struct ttm_operation_ctx ctx =

{.interruptible = true};

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+		struct xe_bo *bo = link->bo;
+		int ret;
+
+		if (!signal_pending(current)) {
+			xe_bo_lock(bo, NULL);
+			ret = ttm_bo_validate(&bo->ttm,

&tt_placement, &ctx);

drivers/gpu/drm/xe/tests/xe_bo.c +3 -0
+			xe_bo_unlock(bo);
+			if (ret && ret != -EINTR)
+				KUNIT_FAIL(test, "Validation

failed: %pe\n",

drivers/gpu/drm/xe/tests/xe_bo.c +12 -0
+					   ERR_PTR(ret));
+			else if (ret)
+				interrupted++;
+			else
+				successful++;
+		}
+		xe_bo_put(link->bo);
+		list_del(&link->link);
+		kfree(link);
+		cond_resched();
+	}
+	kunit_info(test, "Readbacks interrupted: %u successful:

%u\n",

drivers/gpu/drm/xe/tests/xe_bo.c +10 -0
+		   interrupted, successful);
+
+	return 0;
+}
+
+void xe_bo_shrink_kunit(struct kunit *test)
+{
+	xe_call_for_each_device(shrink_test_run_device);
+}
+EXPORT_SYMBOL_IF_KUNIT(xe_bo_shrink_kunit);
drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
index a324cde77db8..317fa923e287 100644
@@ -10,6 +10,7 @@
  static struct kunit_case xe_bo_tests[] = {
  	KUNIT_CASE(xe_ccs_migrate_kunit),
  	KUNIT_CASE(xe_bo_evict_kunit),
+	KUNIT_CASE_SLOW(xe_bo_shrink_kunit),
  	{}
  };

drivers/gpu/drm/xe/tests/xe_bo_test.h +0 -0
index 0113ab45066a..7f44d14a45c5 100644
@@ -10,5 +10,6 @@ struct kunit;


 void xe_ccs_migrate_kunit(struct kunit *test);
 void xe_bo_evict_kunit(struct kunit *test);
+void xe_bo_shrink_kunit(struct kunit *test);

 #endif

drivers/gpu/drm/xe/xe_bo.c +2 -0
index 65c696966e96..6ab63d1642ae 100644
@@ -10,6 +10,7 @@
  #include <drm/drm_drv.h>
  #include <drm/drm_gem_ttm_helper.h>
  #include <drm/drm_managed.h>
+#include <drm/ttm/ttm_backup.h>
  #include <drm/ttm/ttm_device.h>
  #include <drm/ttm/ttm_placement.h>
  #include <drm/ttm/ttm_tt.h>
@@ -25,6 +26,7 @@
  #include "xe_pm.h"
  #include "xe_preempt_fence.h"
  #include "xe_res_cursor.h"
+#include "xe_shrinker.h"
  #include "xe_trace_bo.h"
  #include "xe_ttm_stolen_mgr.h"
  #include "xe_vm.h"
@@ -278,11 +280,15 @@ static void xe_evict_flags(struct
 ttm_buffer_object *tbo,
  	}
  }


+/* struct xe_ttm_tt - Subclassed ttm_tt for xe */ struct xe_ttm_tt {
  struct ttm_tt ttm;

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c static void xe_evict_flags(struct ttm_buffer_object *tbo, · line 290

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +2 -1
-	struct device *dev;
+	/** @xe - The xe device */
+	struct xe_device *xe;

  struct sg_table sgt;
  struct sg_table *sg;

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	/** @purgeable - Whether the bo is purgeable (WONTNEED) */

So we need to add WONTNEED to our uAPI, right?

Not strictly, but we have work ongoing to implement this. Bos that UMD pool are typically WONTNEED and unless we mark them as such, shrinking of these will not work unless we have swap-space available, and then only with a completely unnecessary copy...

Quoted 53 lines
Quoted 53 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	bool purgeable;

 };

 static int xe_tt_map_sg(struct ttm_tt *tt)

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -291,7 +297,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
  	unsigned long num_pages = tt->num_pages;
  	int ret;

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
+	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+		   !(tt->page_flags &

TTM_TT_FLAG_EXTERNAL_MAPPABLE));

  if (xe_tt->sg)
  return 0;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -299,13 +306,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)

  ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt-

Quoted 1 line

pages,

  num_pages, 0, (u64)num_pages << PAGE_SHIFT,
-

						xe_sg_segment_size(xe_tt->dev),
drivers/gpu/drm/xe/xe_bo.c +1 -0
+						xe_sg_segment_size

(xe_tt->xe->drm.dev),
  GFP_KERNEL); if (ret)
  return ret;

  xe_tt->sg = &xe_tt->sgt;

drivers/gpu/drm/xe/xe_bo.c +0 -1
-	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg,

DMA_BIDIRECTIONAL,

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,

DMA_BIDIRECTIONAL,
        DMA_ATTR_SKIP_CPU_SYNC); if (ret) {
  sg_free_table(xe_tt->sg);

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -321,7 +328,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)

  struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);

  if (xe_tt->sg) {

drivers/gpu/drm/xe/xe_bo.c +1 -1
-		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
+		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,

    DMA_BIDIRECTIONAL, 0); sg_free_table(xe_tt->sg); xe_tt->sg = NULL;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -336,21 +343,41 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
  	return xe_tt->sg;
  }


+/*

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c struct sg_table *xe_bo_sg(struct xe_bo *bo) · line 351

Quoted 5 lines

+ * Account ttm pages against the device shrinker's shrinkable and
+ * purgeable counts.
+ */

drivers/gpu/drm/xe/xe_bo.c +2 -0
+static void xe_ttm_tt_account(struct ttm_tt *tt, bool add)
+{

Again I think bools are frowned upon as arguments. Maybe just two functions - add / sub?

OK,

Quoted 69 lines
Quoted 63 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);

drivers/gpu/drm/xe/xe_bo.c +7 -0
+	long num_pages = tt->num_pages;
+
+	if (!add)
+		num_pages = -num_pages;
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0,

num_pages);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker,

num_pages, 0);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+}
+

 static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
         u32 page_flags) {
  struct xe_bo *bo = ttm_to_xe_bo(ttm_bo); struct xe_device *xe = xe_bo_device(bo);

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	struct xe_ttm_tt *tt;
+	struct xe_ttm_tt *xe_tt;
+	struct ttm_tt *tt;

  unsigned long extra_pages;
  enum ttm_caching caching;
  int err;

drivers/gpu/drm/xe/xe_bo.c +2 -2
-	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
-	if (!tt)
+	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
+	if (!xe_tt)

  return NULL;

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	tt->dev = xe->drm.dev;
+	tt = &xe_tt->ttm;
+	xe_tt->xe = xe;


  extra_pages = 0;
  if (xe_bo_needs_ccs_pages(bo))

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -387,42 +414,128 @@ static struct ttm_tt
 *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
  		caching = ttm_uncached;
  	}

drivers/gpu/drm/xe/xe_bo.c +0 -1
-	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching,

extra_pages);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (ttm_bo->type != ttm_bo_type_sg)
+		page_flags |= TTM_TT_FLAG_EXTERNAL |

TTM_TT_FLAG_EXTERNAL_MAPPABLE;

drivers/gpu/drm/xe/xe_bo.c +2 -0
+
+	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching,

extra_pages);
  if (err) {

drivers/gpu/drm/xe/xe_bo.c +1 -1
-		kfree(tt);
+		kfree(xe_tt);

  return NULL;
  }

drivers/gpu/drm/xe/xe_bo.c +1 -1
-	return &tt->ttm;
+	tt->backup = ttm_backup_shmem_create(tt->num_pages <<

PAGE_SHIFT);

drivers/gpu/drm/xe/xe_bo.c static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, · line 428

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (IS_ERR(tt->backup)) {
+		ttm_tt_fini(tt);

Mentioned this the previous review I think you need set tt->backup to NULL here or update ttm_tt_fini to understand IS_ERR(tt->backup).

Quoted 3 lines

Also maybe dump question but could we just have a global backup for all
BOs? Would that be better than each BO creating its own backup?

I initially made the code like that, when we had a global backend directly into the swap cache. But I figure shmem wants one file per bo, otherwise we'd need one gigantic shmem object... Not sure if that works...

Quoted 105 lines
Quoted 105 lines
drivers/gpu/drm/xe/xe_bo.c +5 -0
+		kfree(xe_tt);
+		return NULL;
+	}
+
+	return tt;

 }

 static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
        struct ttm_operation_ctx *ctx) {

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);
  int err;

  /*
  * dma-bufs are not populated with pages, and the dma-
  * addresses are set up when moved to XE_PL_TT. */

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))

  return 0;

  err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx); if (err)
  return err;

drivers/gpu/drm/xe/xe_bo.c +4 -1
-	return err;
+	xe_tt->purgeable = false;
+	xe_ttm_tt_account(tt, true);
+
+	return 0;

 }

 static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))

  return;

  xe_tt_unmap_sg(tt);

drivers/gpu/drm/xe/xe_bo.c +17 -1
-	return ttm_pool_free(&ttm_dev->pool, tt);
+	ttm_pool_free(&ttm_dev->pool, tt);
+	xe_ttm_tt_account(tt, false);
+}
+
+/**
+ * xe_bo_shrink() - Try to shrink an xe bo.
+ * @walk:  - The walk parameters
+ * @bo: The TTM buffer object
+ * @purge: Only consider purgeable bos.
+ * @writeback: Try to write back to persistent storage.
+ *
+ * Try to shrink- or purge a bo, and if it succeeds, unmap dma.
+ * Note that we need to be able to handle also non xe bos
+ * (ghost bos), but only if the struct ttm_tt is embedded in
+ * a struct xe_ttm_tt.
+ *
+ * Return: The number of pages shrunken or purged, or negative

error
+ * code on failure.
+ */
+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/xe/xe_bo.c +4 -0
+		  bool purge, bool writeback)
+{
+	struct ttm_tt *tt = bo->ttm;
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct ttm_place place = {.mem_type = bo->resource-
Quoted 1 line

mem_type};

drivers/gpu/drm/xe/xe_bo.c +14 -0
+	struct xe_bo *xe_bo = ttm_to_xe_bo(bo);
+	struct xe_device *xe = xe_tt->xe;
+	bool needs_rpm;
+	long lret = 0L;
+
+	if (!tt || !ttm_tt_is_populated(tt) ||
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE) ||
+	    (purge && !xe_tt->purgeable))
+		return 0L;
+
+	if (!ttm_bo_eviction_valuable(bo, &place))
+		return 0L;
+
+	/* Beware of zombies (GEM object refcount == 0) and

ghosts. */

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	if (!xe_bo_is_xe_bo(bo) || !xe_bo_get_unless_zero(xe_bo))

{

drivers/gpu/drm/xe/xe_bo.c +1 -0
+		struct ttm_placement null_placement = {

.num_placement = 0 };

drivers/gpu/drm/xe/xe_bo.c +10 -0
+
+		lret = ttm_bo_wait_ctx(bo, walk->ctx);
+		if (lret)
+			return lret;
+
+		/* Purge the bo content! */
+		ttm_bo_validate(bo, &null_placement, walk->ctx);
+		return tt->num_pages;
+	}
+
Quoted 8 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	/* System CCS needs gpu copy when moving PL_TT ->

PL_SYSTEM */

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	needs_rpm = (!IS_DGFX(xe) && bo->resource->mem_type !=

XE_PL_SYSTEM &&

drivers/gpu/drm/xe/xe_bo.c +1 -0
+		     xe_bo && xe_bo_needs_ccs_pages(xe_bo) &&

!xe_tt->purgeable);

Is xe_bo check really needed here?

Yes, I think otherwise xe_bo_needs_ccs_pages will be called with NULL for ghost objects that aren't xe_bo.

Quoted 23 lines
Quoted 23 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+	if (needs_rpm && !xe_pm_runtime_get_if_active(xe))
+		goto out_unref;
+
+	lret = ttm_bo_try_shrink(walk, bo, xe_tt->purgeable,

writeback);

drivers/gpu/drm/xe/xe_bo.c +13 -0
+	if (needs_rpm)
+		xe_pm_runtime_put(xe);
+
+	if (lret > 0) {
+		xe_assert(xe, !ttm_tt_is_populated(tt));
+
+		xe_ttm_tt_account(tt, false);
+	}
+
+out_unref:
+	xe_bo_put(xe_bo);
+
+	return lret;

 }

 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c struct xe_bo *___xe_bo_create_locked(struct · line 1355

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -1238,6 +1351,7 @@ struct xe_bo *___xe_bo_create_locked(struct
 xe_device *xe, struct xe_bo *bo,
  	struct ttm_operation_ctx ctx = {
  		.interruptible = true,
  		.no_wait_gpu = false,
+		.gfp_retry_mayfail = true,

Can you explain why you are setting this?

Hm. We might want this in a separate patch. Without this, bo memory allocation will typically never fail, but instead start the OOM killer. I don't think that's the behaviour we want, but yeah deserves its own patch and review.

Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
  	};
  	struct ttm_placement *placement;

  uint32_t alignment;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1681,6 +1795,8 @@ int xe_bo_pin_external(struct xe_bo *bo)
  	}


  ttm_bo_pin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1739,6 +1855,8 @@ int xe_bo_pin(struct xe_bo *bo)
  	}


  ttm_bo_pin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1773,6 +1891,9 @@ void xe_bo_unpin_external(struct xe_bo *bo)
Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c void xe_bo_unpin_external(struct xe_bo *bo) · line 1896

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
  	spin_unlock(&xe->pinned.lock);


  ttm_bo_unpin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +3 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);
+

Nit: Extra newline.

Will fix.

Quoted 397 lines
Quoted 397 lines


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1801,6 +1922,8 @@ void xe_bo_unpin(struct xe_bo *bo)
  	}


  ttm_bo_unpin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);

 }

 /**

drivers/gpu/drm/xe/xe_bo.h +0 -0
index 6de894c728f5..8463e3f3f6f1 100644
@@ -63,6 +63,7 @@
  #define XE_BO_PROPS_INVALID	(-1)


 struct sg_table;
+struct xe_ttm_lru_walk;

 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);

drivers/gpu/drm/xe/xe_bo.h +0 -0
@@ -126,6 +127,28 @@ static inline struct xe_bo *xe_bo_get(struct
 xe_bo *bo)
  	return bo;
  }


+/*
+ * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
+ * xe bo
+ * @bo: The bo for which we want to obtain a refcount.
+ *
+ * There is a short window between where the bo's GEM object refcount reaches
+ * zero and where we put the final ttm_bo reference. Code in the eviction- and
+ * shrinking path should therefore attempt to grab a gem object reference before
+ * trying to use members outside of the base class ttm object. This function is
+ * intended for that purpose. On successful return, this function must be paired
+ * with an xe_bo_put().
+ *
+ * Return: @bo on success, NULL on failure.
+ */
+static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)

drivers/gpu/drm/xe/xe_bo.h +7 -0
+{
+	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
+		return NULL;
+
+	return bo;
+}
+

 static inline void xe_bo_put(struct xe_bo *bo)
 {
  if (bo)

drivers/gpu/drm/xe/xe_bo.h +0 -0
@@ -315,6 +338,9 @@ static inline unsigned int
 xe_sg_segment_size(struct device *dev)


 #define
i915_gem_object_flush_if_display(obj) ((void)(obj))

+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/xe/xe_bo.h +2 -0
+		  bool purge, bool writeback);
+

 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given

drivers/gpu/drm/xe/xe_device.c +1 -0
index cfda7cb5df2c..58fecc4b0a18 100644
@@ -47,6 +47,7 @@
  #include "xe_perf.h"
  #include "xe_pm.h"
  #include "xe_query.h"
+#include "xe_shrinker.h"
  #include "xe_sriov.h"
  #include "xe_tile.h"
  #include "xe_ttm_stolen_mgr.h"
@@ -241,6 +242,9 @@ static void xe_device_destroy(struct drm_device
 *dev, void *dummy)
  	if (xe->unordered_wq)
  		destroy_workqueue(xe->unordered_wq);

drivers/gpu/drm/xe/xe_device.c +3 -0
+	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
+		xe_shrinker_destroy(xe->mem.shrinker);
+

  ttm_device_fini(&xe->ttm);
 }

drivers/gpu/drm/xe/xe_device.c +0 -0
@@ -270,6 +274,10 @@ struct xe_device *xe_device_create(struct
 pci_dev *pdev,
  	if (err)
  		goto err;

drivers/gpu/drm/xe/xe_device.c +4 -0
+	xe->mem.shrinker = xe_shrinker_create(xe);
+	if (IS_ERR(xe->mem.shrinker))
+		return ERR_CAST(xe->mem.shrinker);
+

  xe->info.devid = pdev->device; xe->info.revid = pdev->revision; xe->info.force_execlist = xe_modparam.force_execlist;

drivers/gpu/drm/xe/xe_device_types.h +2 -0
index c37be471d11c..3d5440aba52e 100644
@@ -325,6 +325,8 @@ struct xe_device {
  		struct xe_mem_region vram;
  		/** @mem.sys_mgr: system TTM manager */
  		struct ttm_resource_manager sys_mgr;
+		/** @mem.sys_mgr: system memory shrinker. */
+		struct xe_shrinker *shrinker;
  	} mem;


  /** @sriov: device level virtualization data */

drivers/gpu/drm/xe/xe_shrinker.c new file +20 -0
new file mode 100644
index 000000000000..3f9554bdc06b
@@ -0,0 +1,287 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <linux/shrinker.h>
+#include <linux/swap.h>
+
+#include <drm/ttm/ttm_bo.h>
+#include <drm/ttm/ttm_tt.h>
+
+#include "xe_bo.h"
+#include "xe_pm.h"
+#include "xe_shrinker.h"
+
+/**
+ * struct xe_shrinker - per-device shrinker
+ * @xe: Back pointer to the device.
+ * @lock: Lock protecting accounting.
+ * @shrinkable_pages: Number of pages that are currently

shrinkable.
+ * @purgeable_pages: Number of pages that are currently purgeable.
+ * @shrink: Pointer to the mm shrinker.
+ * @pm_worker: Worker to wake up the device if required.
+ */

drivers/gpu/drm/xe/xe_shrinker.c new file +32 -0
+struct xe_shrinker {
+	struct xe_device *xe;
+	rwlock_t lock;
+	long shrinkable_pages;
+	long purgeable_pages;
+	struct shrinker *shrink;
+	struct work_struct pm_worker;
+};
+
+/**
+ * struct xe_shrink_lru_walk - lru_walk subclass for shrinker
+ * @walk: The embedded base class.
+ * @xe: Pointer to the xe device.
+ * @purge: Purgeable only request from the srinker.
+ * @writeback: Try to write back to persistent storage.
+ */
+struct xe_shrink_lru_walk {
+	struct ttm_lru_walk walk;
+	struct xe_device *xe;
+	bool purge;
+	bool writeback;
+};
+
+static struct xe_shrinker *to_xe_shrinker(struct shrinker *shrink)
+{
+	return shrink->private_data;
+}
+
+static struct xe_shrink_lru_walk *
+to_xe_shrink_lru_walk(struct ttm_lru_walk *walk)
+{
+	return container_of(walk, struct xe_shrink_lru_walk,

walk);

drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+}
+
+/**
+ * xe_shrinker_mod_pages() - Modify shrinker page accounting
+ * @shrinker: Pointer to the struct xe_shrinker.
+ * @shrinkable: Shrinkable pages delta. May be negative.
+ * @purgeable: Purgeable page delta. May be negative.
+ *
+ * Modifies the shrinkable and purgeable pages accounting.
+ */
+void
+xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long

shrinkable, long purgeable)

drivers/gpu/drm/xe/xe_shrinker.c new file +8 -0
+{
+	write_lock(&shrinker->lock);
+	shrinker->shrinkable_pages += shrinkable;
+	shrinker->purgeable_pages += purgeable;
+	write_unlock(&shrinker->lock);
+}
+
+static long xe_shrinker_process_bo(struct ttm_lru_walk *walk,

struct ttm_buffer_object *bo)

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+{
+	struct xe_shrink_lru_walk *shrink_walk =

to_xe_shrink_lru_walk(walk);

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+
+	return xe_bo_shrink(walk, bo, shrink_walk->purge,

shrink_walk->writeback);

drivers/gpu/drm/xe/xe_shrinker.c new file +3 -0
+}
+
+static long xe_shrinker_walk(struct xe_shrink_lru_walk

*shrink_walk, long target)

drivers/gpu/drm/xe/xe_shrinker.c new file +14 -0
+{
+	struct xe_device *xe = shrink_walk->xe;
+	struct ttm_resource_manager *man;
+	unsigned int mem_type;
+	long sofar = 0;
+	long lret;
+
+	for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT;
++mem_type) {
+		man = ttm_manager_type(&xe->ttm, mem_type);
+		if (!man || !man->use_tt)
+			continue;
+
+		lret = ttm_lru_walk_for_evict(&shrink_walk->walk,

&xe->ttm, man, target);

drivers/gpu/drm/xe/xe_shrinker.c new file +13 -0
+		if (lret < 0)
+			return lret;
+
+		sofar += lret;
+		if (sofar >= target)
+			break;
+	}
+
+	return sofar;
+}
+
+static unsigned long
+xe_shrinker_count(struct shrinker *shrink, struct shrink_control

*sc)

drivers/gpu/drm/xe/xe_shrinker.c new file +7 -0
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	unsigned long num_pages;
+
+	num_pages = get_nr_swap_pages();
+	read_lock(&shrinker->lock);
+	num_pages = min_t(unsigned long, num_pages, shrinker-
Quoted 1 line

shrinkable_pages);

drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+	num_pages += shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	return num_pages ? num_pages : SHRINK_EMPTY;
+}
+
+static const struct ttm_lru_walk_ops xe_shrink_ops = {
+	.process_bo = xe_shrinker_process_bo,
+};
+
+/*
+ * Check if we need runtime pm, and if so try to grab a reference

if
+ * already active. If grabbing a reference fails, queue a worker that
+ * does it for us outside of reclaim, but don't wait for it to complete.
+ * If bo shrinking needs an rpm reference and we don't have it (yet),
+ * that bo will be skipped anyway.
+ */
+static bool xe_shrinker_runtime_pm_get(struct xe_shrinker *shrinker, bool force,

drivers/gpu/drm/xe/xe_shrinker.c new file +18 -0
+				       unsigned long nr_to_scan)
+{
+	struct xe_device *xe = shrinker->xe;
+
+	if (IS_DGFX(xe) || !xe_device_has_flat_ccs(xe) ||
+	    !get_nr_swap_pages())
+		return false;
+
+	if (!force) {
+		read_lock(&shrinker->lock);
+		force = (nr_to_scan > shrinker->purgeable_pages);
+		read_unlock(&shrinker->lock);
+		if (!force)
+			return false;
+	}
+
+	if (!xe_pm_runtime_get_if_active(xe)) {
+		queue_work(xe->unordered_wq, &shrinker-
Quoted 1 line

pm_worker);

drivers/gpu/drm/xe/xe_shrinker.c new file +7 -0
+		return false;
+	}
+
+	return true;
+}
+
+static void xe_shrinker_runtime_pm_put(struct xe_shrinker

*shrinker, bool runtime_pm)

drivers/gpu/drm/xe/xe_shrinker.c new file +6 -0
+{
+	if (runtime_pm)
+		xe_pm_runtime_put(shrinker->xe);
+}
+
+static unsigned long xe_shrinker_scan(struct shrinker *shrink,

struct shrink_control *sc)

drivers/gpu/drm/xe/xe_shrinker.c new file +30 -0
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	bool is_kswapd = current_is_kswapd();
+	struct ttm_operation_ctx ctx = {
+		.interruptible = false,
+		.no_wait_gpu = !is_kswapd,
+	};
+	unsigned long nr_to_scan, freed = 0;
+	struct xe_shrink_lru_walk shrink_walk = {
+		.walk = {
+			.ops = &xe_shrink_ops,
+			.ctx = &ctx,
+			.trylock_only = true,
+		},
+		.xe = shrinker->xe,
+		.purge = true,
+		.writeback = is_kswapd,
+	};
+	bool runtime_pm;
+	bool purgeable;
+	long ret;
+
+	sc->nr_scanned = 0;
+	nr_to_scan = sc->nr_to_scan;
+
+	read_lock(&shrinker->lock);
+	purgeable = !!shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	/* Might need runtime PM. Try to wake early if it looks

like it. */

drivers/gpu/drm/xe/xe_shrinker.c new file +1 -0
+	runtime_pm = xe_shrinker_runtime_pm_get(shrinker, false,

nr_to_scan);

drivers/gpu/drm/xe/xe_shrinker.c new file +18 -0
+
+	while (purgeable && freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+	if (freed < nr_to_scan)
+		nr_to_scan -= freed;
+	else
+		nr_to_scan = 0;
+	if (!nr_to_scan)
+		goto out;
+
+	/* If we didn't wake before, try to do it now if needed.

*/

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+	if (!runtime_pm)
+		runtime_pm = xe_shrinker_runtime_pm_get(shrinker,

true, 0);

drivers/gpu/drm/xe/xe_shrinker.c new file +23 -0
+
+	shrink_walk.purge = false;
+	nr_to_scan = sc->nr_to_scan;
+	while (freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+
+out:
+	xe_shrinker_runtime_pm_put(shrinker, runtime_pm);
+	return freed ? freed : SHRINK_STOP;
+}
+
+/* Wake up the device for shrinking. */
+static void xe_shrinker_pm(struct work_struct *work)
+{
+	struct xe_shrinker *shrinker =
+		container_of(work, typeof(*shrinker), pm_worker);
Quoted 8 lines

drivers/gpu/drm/xe/xe_shrinker.c line 242

Quoted 3 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +3 -0
+
+	xe_pm_runtime_get(shrinker->xe);
+	xe_pm_runtime_put(shrinker->xe);

So I don't really get this. How does this help the shrinker get a PM ref? The small window between xe_pm_runtime_get / put the shrinker grabs
one via xe_pm_runtime_get_if_active? Seems fragile.

The pm has a delay after the last put (Typically 1s?), and if the shrinker fails obtaining a reference it will be retried. Typically it will then succeed and when it does start a new delay.

This doesn't strictly guarantee shrinking progress, but there's no way we can synchronously delay shrinking until we have the pm reference.

What we could perhaps do is to report back that we actually made progress but postpone the actual shrinking to the worker with writeback set to on; hoping that the shrinking core doesn't give up before it sees the released memory...

But I'd prefer to leave that out until we see that that's actually a real problem.

Best would be if we could have access the LNL CCS memory using the CPU...

/Thomas

Quoted 96 lines

Matt

Quoted 93 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+}
+
+/**
+ * xe_shrinker_create() - Create an xe per-device shrinker
+ * @xe: Pointer to the xe device.
+ *
+ * Returns: A pointer to the created shrinker on success,
+ * Negative error code on failure.
+ */
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe)
+{
+	struct xe_shrinker *shrinker = kzalloc(sizeof(*shrinker),

GFP_KERNEL);

drivers/gpu/drm/xe/xe_shrinker.c new file +5 -0
+
+	if (!shrinker)
+		return ERR_PTR(-ENOMEM);
+
+	shrinker->shrink = shrinker_alloc(0, "xe system

shrinker");

drivers/gpu/drm/xe/xe_shrinker.c new file +28 -0
+	if (!shrinker->shrink) {
+		kfree(shrinker);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm);
+	shrinker->xe = xe;
+	rwlock_init(&shrinker->lock);
+	shrinker->shrink->count_objects = xe_shrinker_count;
+	shrinker->shrink->scan_objects = xe_shrinker_scan;
+	shrinker->shrink->private_data = shrinker;
+	shrinker_register(shrinker->shrink);
+
+	return shrinker;
+}
+
+/**
+ * xe_shrinker_destroy() - Destroy an xe per-device shrinker
+ * @shrinker: Pointer to the shrinker to destroy.
+ */
+void xe_shrinker_destroy(struct xe_shrinker *shrinker)
+{
+	xe_assert(shrinker->xe, !shrinker->shrinkable_pages);
+	xe_assert(shrinker->xe, !shrinker->purgeable_pages);
+	shrinker_free(shrinker->shrink);
+	flush_work(&shrinker->pm_worker);
+	kfree(shrinker);
+}
drivers/gpu/drm/xe/xe_shrinker.h new file +12 -0
new file mode 100644
index 000000000000..28a038f4fcbf
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_SHRINKER_H_
+#define _XE_SHRINKER_H_
+
+struct xe_shrinker;
+struct xe_device;
+
+void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long

shrinkable, long purgeable);

drivers/gpu/drm/xe/xe_shrinker.h new file +6 -0
+
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
+
+void xe_shrinker_destroy(struct xe_shrinker *shrinker);
+
+#endif
include/drm/ttm/ttm_bo.h +0 -0
index e577528f5dfc..c7e81ae025d9 100644
@@ -229,6 +229,9 @@ struct ttm_lru_walk {

 long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
      struct ttm_resource_manager *man, long target);

+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

include/drm/ttm/ttm_bo.h +2 -0
+		       bool purge, bool writeback);
+

 /**
  * ttm_bo_get - reference a struct ttm_buffer_object *

Signature · 1 line
2.44.0
maintainer
#30 lore raw

Re: [PATCH v6 11/12] drm/ttm, drm/xe: Add a shrinker for xe bos

Quoted 113 lines · Thomas Hellström
Quoted 103 lines · Matthew Brost
Quoted 102 lines · Thomas Hellström

Rather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>

---
 drivers/gpu/drm/ttm/ttm_bo_util.c     |  67 ++++++ drivers/gpu/drm/xe/Makefile           |   1 + drivers/gpu/drm/xe/tests/xe_bo.c      | 118 +++++++++++ drivers/gpu/drm/xe/tests/xe_bo_test.c |   1 + drivers/gpu/drm/xe/tests/xe_bo_test.h |   1 + drivers/gpu/drm/xe/xe_bo.c            | 155 ++++++++++++-- drivers/gpu/drm/xe/xe_bo.h            |  26 +++ drivers/gpu/drm/xe/xe_device.c        |   8 + drivers/gpu/drm/xe/xe_device_types.h  |   2 + drivers/gpu/drm/xe/xe_shrinker.c      | 287
++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_shrinker.h      |  18 ++ include/drm/ttm/ttm_bo.h              |   3 +

 12 files changed, 671 insertions(+), 16 deletions(-)

 create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h

drivers/gpu/drm/ttm/ttm_bo_util.c +0 -0
index c4f678f30fc2..563e96a4cf06 100644
@@ -924,3 +924,70 @@ long ttm_lru_walk_for_evict(struct
 ttm_lru_walk *walk, struct ttm_device *bdev,


  return progress;
 }

drivers/gpu/drm/ttm/ttm_bo_util.c +4 -0
+EXPORT_SYMBOL(ttm_lru_walk_for_evict);
+
+/**
+ * ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer

object.
+ * @walk: The struct xe_ttm_lru_walk that describes the walk.
+ * @bo: The buffer object.
+ * @purge: Whether to attempt to purge the bo content since it's no
+ * longer needed.
+ * @writeback: If !@purge, attempt to write out to persistent storage.
+ *
+ * The function uses the ttm_tt_back_up functionality to back up or
+ * purge a struct ttm_tt. If the bo is not in system, it's first
+ * moved there.
+ *
+ * Return: The number of pages shrunken or purged, or
+ * negative error code on failure.
+ */
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/ttm/ttm_bo_util.c +23 -0
+		       bool purge, bool writeback)
+{
+	static const struct ttm_place sys_placement_flags = {
+		.fpfn = 0,
+		.lpfn = 0,
+		.mem_type = TTM_PL_SYSTEM,
+		.flags = 0,
+	};
+	static struct ttm_placement sys_placement = {
+		.num_placement = 1,
+		.placement = &sys_placement_flags,
+	};
+	struct ttm_operation_ctx *ctx = walk->ctx;
+	struct ttm_tt *tt = bo->ttm;
+	long lret;
+
+	dma_resv_assert_held(bo->base.resv);
+
+	if (!tt || !ttm_tt_is_populated(tt))
+		return 0;
+
+	if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+		int ret = ttm_bo_validate(bo, &sys_placement,

ctx);

Quoted 8 lines

drivers/gpu/drm/ttm/ttm_bo_util.c long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev, · line 972

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
+
+		if (ret) {
+			if (ret == -EINTR || ret == -EDEADLK ||
+			    ret == -ERESTARTSYS)
+				return ret;

Can you explain the various error code returns / supression in this function?

Quoted 5 lines

Want me to add a comment in the code or inline here? Anyway, the error codes are codes for which the caller wants to restart. (Signal delivery or deadlock). For other errors just move on to the next bo on the LRU list.

Yes, that would help make this clear and help ensure correctness.

Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +13 -0
+			return 0;
+		}
+	}
+
+	lret = ttm_bo_wait_ctx(bo, ctx);
+	if (lret < 0) {
+		if (lret == -ERESTARTSYS)
+			return lret;
+		return 0;
+	}
+
+	if (bo->deleted)
+		lret = ttm_tt_backup(bo->bdev, tt, true,
Quoted 8 lines
Quoted 4 lines

writeback);

drivers/gpu/drm/ttm/ttm_bo_util.c +2 -0
+	else
+		lret = ttm_tt_backup(bo->bdev, tt, purge,

writeback);

Hmm, missed this in my previous review. It is frowned upon having multiple bools as arguments. Could this be reworked with flags? Same goes for all functions in the series with multiple bool arguments.

Quoted 2 lines

I agree. Ill see if I can make this look betteer.

+1

Quoted 234 lines
Quoted 225 lines
Quoted 225 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +6 -0
+	if (lret < 0 && lret != -EINTR)
+		return 0;
+
+	return lret;
+}
+EXPORT_SYMBOL(ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
index b1e03bfe4a68..1eba51bdd172 100644
@@ -112,6 +112,7 @@ xe-y += xe_bb.o \
  	xe_ring_ops.o \
  	xe_sa.o \
  	xe_sched_job.o \
+	xe_shrinker.o \
  	xe_step.o \
  	xe_sync.o \
  	xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +0 -0
index 9f3c02826464..49617f16dc76 100644
@@ -6,6 +6,8 @@
  #include <kunit/test.h>
  #include <kunit/visibility.h>

drivers/gpu/drm/xe/tests/xe_bo.c +2 -0
+#include <uapi/linux/sysinfo.h>
+

 #include "tests/xe_bo_test.h"
 #include "tests/xe_pci_test.h"
 #include "tests/xe_test.h"

drivers/gpu/drm/xe/tests/xe_bo.c +27 -0
@@ -350,3 +352,119 @@ void xe_bo_evict_kunit(struct kunit *test)
  	xe_call_for_each_device(evict_test_run_device);
  }
  EXPORT_SYMBOL_IF_KUNIT(xe_bo_evict_kunit);
+
+struct xe_bo_link {
+	struct list_head link;
+	struct xe_bo *bo;
+};
+
+#define XE_BO_SHRINK_SIZE ((unsigned long)SZ_64M)
+
+/*
+ * Try to create system bos corresponding to twice the amount
+ * of available system memory to test shrinker functionality.
+ * If no swap space is available to accommodate the
+ * memory overcommit, mark bos purgeable.
+ */
+static int shrink_test_run_device(struct xe_device *xe)
+{
+	struct kunit *test = xe_cur_kunit();
+	LIST_HEAD(bos);
+	struct xe_bo_link *link, *next;
+	struct sysinfo si;
+	size_t total, alloced;
+	unsigned int interrupted = 0, successful = 0;
+
+	si_meminfo(&si);
+	total = si.freeram * si.mem_unit;
+
+	kunit_info(test, "Free ram is %lu bytes. Will allocate

twice of that.\n",

drivers/gpu/drm/xe/tests/xe_bo.c +4 -0
+		   (unsigned long) total);
+
+	total <<= 1;
+	for (alloced = 0; alloced < total ; alloced +=

XE_BO_SHRINK_SIZE) {

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+		struct xe_bo *bo;
+		unsigned int mem_type;
+
+		link = kzalloc(sizeof(*link), GFP_KERNEL);
+		if (!link) {
+			KUNIT_FAIL(test, "Unexpeced link

allocation failure\n");

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+			break;
+		}
+
+		INIT_LIST_HEAD(&link->link);
+
+		/* We can create bos using WC caching here. But it

is slower. */

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+		bo = xe_bo_create_user(xe, NULL, NULL,

XE_BO_SHRINK_SIZE,

drivers/gpu/drm/xe/tests/xe_bo.c +5 -0
+				       DRM_XE_GEM_CPU_CACHING_WB,
+				       ttm_bo_type_device,
+				       XE_BO_FLAG_SYSTEM);
+		if (IS_ERR(bo)) {
+			if (bo != ERR_PTR(-ENOMEM) && bo !=

ERR_PTR(-ENOSPC) &&

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+			    bo != ERR_PTR(-EINTR) && bo !=

ERR_PTR(-ERESTARTSYS))

drivers/gpu/drm/xe/tests/xe_bo.c +1 -0
+				KUNIT_FAIL(test, "Error creating

bo: %pe\n", bo);

drivers/gpu/drm/xe/tests/xe_bo.c +9 -0
+			kfree(link);
+			break;
+		}
+		link->bo = bo;
+		list_add_tail(&link->link, &bos);
+		xe_bo_lock(bo, false);
+
+		/*
+		 * If we're low on swap entries, we can't shrink

unless the bo

drivers/gpu/drm/xe/tests/xe_bo.c +3 -0
+		 * is marked purgeable.
+		 */
+		if (get_nr_swap_pages() < (XE_BO_SHRINK_SIZE >>

PAGE_SHIFT) * 128) {

drivers/gpu/drm/xe/tests/xe_bo.c +2 -0
+			struct xe_ttm_tt *xe_tt =
+				container_of(bo->ttm.ttm,

typeof(*xe_tt), ttm);

drivers/gpu/drm/xe/tests/xe_bo.c +4 -0
+			long num_pages = xe_tt->ttm.num_pages;
+
+			xe_tt->purgeable = true;
+			xe_shrinker_mod_pages(xe->mem.shrinker, -

num_pages,

drivers/gpu/drm/xe/tests/xe_bo.c +7 -0
+					      num_pages);
+		}
+
+		mem_type = bo->ttm.resource->mem_type;
+		xe_bo_unlock(bo);
+		if (mem_type != XE_PL_TT)
+			KUNIT_FAIL(test, "Bo in incorrect memory

type: %u\n",

drivers/gpu/drm/xe/tests/xe_bo.c +9 -0
+				   bo->ttm.resource->mem_type);
+		cond_resched();
+		if (signal_pending(current))
+			break;
+	}
+
+	/* Read back and destroy bos */
+	list_for_each_entry_safe_reverse(link, next, &bos, link) {
+		static struct ttm_operation_ctx ctx =

{.interruptible = true};

drivers/gpu/drm/xe/tests/xe_bo.c +6 -0
+		struct xe_bo *bo = link->bo;
+		int ret;
+
+		if (!signal_pending(current)) {
+			xe_bo_lock(bo, NULL);
+			ret = ttm_bo_validate(&bo->ttm,

&tt_placement, &ctx);

drivers/gpu/drm/xe/tests/xe_bo.c +3 -0
+			xe_bo_unlock(bo);
+			if (ret && ret != -EINTR)
+				KUNIT_FAIL(test, "Validation

failed: %pe\n",

drivers/gpu/drm/xe/tests/xe_bo.c +12 -0
+					   ERR_PTR(ret));
+			else if (ret)
+				interrupted++;
+			else
+				successful++;
+		}
+		xe_bo_put(link->bo);
+		list_del(&link->link);
+		kfree(link);
+		cond_resched();
+	}
+	kunit_info(test, "Readbacks interrupted: %u successful:

%u\n",

drivers/gpu/drm/xe/tests/xe_bo.c +10 -0
+		   interrupted, successful);
+
+	return 0;
+}
+
+void xe_bo_shrink_kunit(struct kunit *test)
+{
+	xe_call_for_each_device(shrink_test_run_device);
+}
+EXPORT_SYMBOL_IF_KUNIT(xe_bo_shrink_kunit);
drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
index a324cde77db8..317fa923e287 100644
@@ -10,6 +10,7 @@
  static struct kunit_case xe_bo_tests[] = {
  	KUNIT_CASE(xe_ccs_migrate_kunit),
  	KUNIT_CASE(xe_bo_evict_kunit),
+	KUNIT_CASE_SLOW(xe_bo_shrink_kunit),
  	{}
  };

drivers/gpu/drm/xe/tests/xe_bo_test.h +0 -0
index 0113ab45066a..7f44d14a45c5 100644
@@ -10,5 +10,6 @@ struct kunit;


 void xe_ccs_migrate_kunit(struct kunit *test);
 void xe_bo_evict_kunit(struct kunit *test);
+void xe_bo_shrink_kunit(struct kunit *test);

 #endif

drivers/gpu/drm/xe/xe_bo.c +2 -0
index 65c696966e96..6ab63d1642ae 100644
@@ -10,6 +10,7 @@
  #include <drm/drm_drv.h>
  #include <drm/drm_gem_ttm_helper.h>
  #include <drm/drm_managed.h>
+#include <drm/ttm/ttm_backup.h>
  #include <drm/ttm/ttm_device.h>
  #include <drm/ttm/ttm_placement.h>
  #include <drm/ttm/ttm_tt.h>
@@ -25,6 +26,7 @@
  #include "xe_pm.h"
  #include "xe_preempt_fence.h"
  #include "xe_res_cursor.h"
+#include "xe_shrinker.h"
  #include "xe_trace_bo.h"
  #include "xe_ttm_stolen_mgr.h"
  #include "xe_vm.h"
@@ -278,11 +280,15 @@ static void xe_evict_flags(struct
 ttm_buffer_object *tbo,
  	}
  }


+/* struct xe_ttm_tt - Subclassed ttm_tt for xe */ struct xe_ttm_tt {
  struct ttm_tt ttm;

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c static void xe_evict_flags(struct ttm_buffer_object *tbo, · line 290

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +2 -1
-	struct device *dev;
+	/** @xe - The xe device */
+	struct xe_device *xe;

  struct sg_table sgt;
  struct sg_table *sg;

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	/** @purgeable - Whether the bo is purgeable (WONTNEED) */

So we need to add WONTNEED to our uAPI, right?

Quoted 5 lines

Not strictly, but we have work ongoing to implement this. Bos that UMD pool are typically WONTNEED and unless we mark them as such, shrinking of these will not work unless we have swap-space available, and then only with a completely unnecessary copy...

Makes sense.

Quoted 133 lines
Quoted 53 lines
Quoted 53 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	bool purgeable;

 };

 static int xe_tt_map_sg(struct ttm_tt *tt)

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -291,7 +297,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
  	unsigned long num_pages = tt->num_pages;
  	int ret;

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
+	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+		   !(tt->page_flags &

TTM_TT_FLAG_EXTERNAL_MAPPABLE));

  if (xe_tt->sg)
  return 0;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -299,13 +306,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)

  ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt-

Quoted 1 line

pages,

  num_pages, 0, (u64)num_pages << PAGE_SHIFT,
-

						xe_sg_segment_size(xe_tt->dev),
drivers/gpu/drm/xe/xe_bo.c +1 -0
+						xe_sg_segment_size

(xe_tt->xe->drm.dev),
  GFP_KERNEL); if (ret)
  return ret;

  xe_tt->sg = &xe_tt->sgt;

drivers/gpu/drm/xe/xe_bo.c +0 -1
-	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg,

DMA_BIDIRECTIONAL,

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,

DMA_BIDIRECTIONAL,
        DMA_ATTR_SKIP_CPU_SYNC); if (ret) {
  sg_free_table(xe_tt->sg);

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -321,7 +328,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)

  struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);

  if (xe_tt->sg) {

drivers/gpu/drm/xe/xe_bo.c +1 -1
-		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
+		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,

    DMA_BIDIRECTIONAL, 0); sg_free_table(xe_tt->sg); xe_tt->sg = NULL;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -336,21 +343,41 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
  	return xe_tt->sg;
  }


+/*

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c struct sg_table *xe_bo_sg(struct xe_bo *bo) · line 351

Quoted 5 lines

+ * Account ttm pages against the device shrinker's shrinkable and
+ * purgeable counts.
+ */

drivers/gpu/drm/xe/xe_bo.c +2 -0
+static void xe_ttm_tt_account(struct ttm_tt *tt, bool add)
+{

Again I think bools are frowned upon as arguments. Maybe just two functions - add / sub?

OK,

Quoted 69 lines
Quoted 63 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);

drivers/gpu/drm/xe/xe_bo.c +7 -0
+	long num_pages = tt->num_pages;
+
+	if (!add)
+		num_pages = -num_pages;
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0,

num_pages);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker,

num_pages, 0);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+}
+

 static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
         u32 page_flags) {
  struct xe_bo *bo = ttm_to_xe_bo(ttm_bo); struct xe_device *xe = xe_bo_device(bo);

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	struct xe_ttm_tt *tt;
+	struct xe_ttm_tt *xe_tt;
+	struct ttm_tt *tt;

  unsigned long extra_pages;
  enum ttm_caching caching;
  int err;

drivers/gpu/drm/xe/xe_bo.c +2 -2
-	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
-	if (!tt)
+	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
+	if (!xe_tt)

  return NULL;

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	tt->dev = xe->drm.dev;
+	tt = &xe_tt->ttm;
+	xe_tt->xe = xe;


  extra_pages = 0;
  if (xe_bo_needs_ccs_pages(bo))

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -387,42 +414,128 @@ static struct ttm_tt
 *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
  		caching = ttm_uncached;
  	}

drivers/gpu/drm/xe/xe_bo.c +0 -1
-	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching,

extra_pages);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (ttm_bo->type != ttm_bo_type_sg)
+		page_flags |= TTM_TT_FLAG_EXTERNAL |

TTM_TT_FLAG_EXTERNAL_MAPPABLE;

drivers/gpu/drm/xe/xe_bo.c +2 -0
+
+	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching,

extra_pages);
  if (err) {

drivers/gpu/drm/xe/xe_bo.c +1 -1
-		kfree(tt);
+		kfree(xe_tt);

  return NULL;
  }

drivers/gpu/drm/xe/xe_bo.c +1 -1
-	return &tt->ttm;
+	tt->backup = ttm_backup_shmem_create(tt->num_pages <<

PAGE_SHIFT);

drivers/gpu/drm/xe/xe_bo.c static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, · line 428

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (IS_ERR(tt->backup)) {
+		ttm_tt_fini(tt);

Mentioned this the previous review I think you need set tt->backup to NULL here or update ttm_tt_fini to understand IS_ERR(tt->backup).

Quoted 9 lines
Quoted 3 lines

Also maybe dump question but could we just have a global backup for all
BOs? Would that be better than each BO creating its own backup?

I initially made the code like that, when we had a global backend directly into the swap cache. But I figure shmem wants one file per bo, otherwise we'd need one gigantic shmem object... Not sure if that works...

If one backup per shmem sense, then let's do that.

I do think we should remove the restriction of only supporting 1 backup per BO so if a driver wants to use a global backup it can.

e.g. Don't fini / free the backup in ttm_tt_fini rather have the driver completely own the backup field. Kinda goofy for the driver to setup the field but TTM to fini / free it.

Quoted 114 lines
Quoted 105 lines
Quoted 105 lines
drivers/gpu/drm/xe/xe_bo.c +5 -0
+		kfree(xe_tt);
+		return NULL;
+	}
+
+	return tt;

 }

 static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
        struct ttm_operation_ctx *ctx) {

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);
  int err;

  /*
  * dma-bufs are not populated with pages, and the dma-
  * addresses are set up when moved to XE_PL_TT. */

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))

  return 0;

  err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx); if (err)
  return err;

drivers/gpu/drm/xe/xe_bo.c +4 -1
-	return err;
+	xe_tt->purgeable = false;
+	xe_ttm_tt_account(tt, true);
+
+	return 0;

 }

 static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {

drivers/gpu/drm/xe/xe_bo.c +2 -1
-	if (tt->page_flags & TTM_TT_FLAG_EXTERNAL)
+	if ((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))

  return;

  xe_tt_unmap_sg(tt);

drivers/gpu/drm/xe/xe_bo.c +17 -1
-	return ttm_pool_free(&ttm_dev->pool, tt);
+	ttm_pool_free(&ttm_dev->pool, tt);
+	xe_ttm_tt_account(tt, false);
+}
+
+/**
+ * xe_bo_shrink() - Try to shrink an xe bo.
+ * @walk:  - The walk parameters
+ * @bo: The TTM buffer object
+ * @purge: Only consider purgeable bos.
+ * @writeback: Try to write back to persistent storage.
+ *
+ * Try to shrink- or purge a bo, and if it succeeds, unmap dma.
+ * Note that we need to be able to handle also non xe bos
+ * (ghost bos), but only if the struct ttm_tt is embedded in
+ * a struct xe_ttm_tt.
+ *
+ * Return: The number of pages shrunken or purged, or negative

error
+ * code on failure.
+ */
+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/xe/xe_bo.c +4 -0
+		  bool purge, bool writeback)
+{
+	struct ttm_tt *tt = bo->ttm;
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct

xe_ttm_tt, ttm);

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	struct ttm_place place = {.mem_type = bo->resource-
Quoted 1 line

mem_type};

drivers/gpu/drm/xe/xe_bo.c +14 -0
+	struct xe_bo *xe_bo = ttm_to_xe_bo(bo);
+	struct xe_device *xe = xe_tt->xe;
+	bool needs_rpm;
+	long lret = 0L;
+
+	if (!tt || !ttm_tt_is_populated(tt) ||
+	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE) ||
+	    (purge && !xe_tt->purgeable))
+		return 0L;
+
+	if (!ttm_bo_eviction_valuable(bo, &place))
+		return 0L;
+
+	/* Beware of zombies (GEM object refcount == 0) and

ghosts. */

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	if (!xe_bo_is_xe_bo(bo) || !xe_bo_get_unless_zero(xe_bo))

{

drivers/gpu/drm/xe/xe_bo.c +1 -0
+		struct ttm_placement null_placement = {

.num_placement = 0 };

drivers/gpu/drm/xe/xe_bo.c +10 -0
+
+		lret = ttm_bo_wait_ctx(bo, walk->ctx);
+		if (lret)
+			return lret;
+
+		/* Purge the bo content! */
+		ttm_bo_validate(bo, &null_placement, walk->ctx);
+		return tt->num_pages;
+	}
+
Quoted 8 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	/* System CCS needs gpu copy when moving PL_TT ->

PL_SYSTEM */

drivers/gpu/drm/xe/xe_bo.c +1 -0
+	needs_rpm = (!IS_DGFX(xe) && bo->resource->mem_type !=

XE_PL_SYSTEM &&

drivers/gpu/drm/xe/xe_bo.c +1 -0
+		     xe_bo && xe_bo_needs_ccs_pages(xe_bo) &&

!xe_tt->purgeable);

Is xe_bo check really needed here?

Quoted 3 lines

Yes, I think otherwise xe_bo_needs_ccs_pages will be called with NULL for ghost objects that aren't xe_bo.

Got it.

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+	if (needs_rpm && !xe_pm_runtime_get_if_active(xe))
+		goto out_unref;
+
+	lret = ttm_bo_try_shrink(walk, bo, xe_tt->purgeable,

writeback);

drivers/gpu/drm/xe/xe_bo.c +13 -0
+	if (needs_rpm)
+		xe_pm_runtime_put(xe);
+
+	if (lret > 0) {
+		xe_assert(xe, !ttm_tt_is_populated(tt));
+
+		xe_ttm_tt_account(tt, false);
+	}
+
+out_unref:
+	xe_bo_put(xe_bo);
+
+	return lret;

 }

 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)

Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c struct xe_bo *___xe_bo_create_locked(struct · line 1355

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -1238,6 +1351,7 @@ struct xe_bo *___xe_bo_create_locked(struct
 xe_device *xe, struct xe_bo *bo,
  	struct ttm_operation_ctx ctx = {
  		.interruptible = true,
  		.no_wait_gpu = false,
+		.gfp_retry_mayfail = true,

Can you explain why you are setting this?

Quoted 5 lines

Hm. We might want this in a separate patch. Without this, bo memory allocation will typically never fail, but instead start the OOM killer. I don't think that's the behaviour we want, but yeah deserves its own patch and review.

Sounds good.

Quoted 440 lines
Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
  	};
  	struct ttm_placement *placement;

  uint32_t alignment;

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1681,6 +1795,8 @@ int xe_bo_pin_external(struct xe_bo *bo)
  	}


  ttm_bo_pin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1739,6 +1855,8 @@ int xe_bo_pin(struct xe_bo *bo)
  	}


  ttm_bo_pin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, false);


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1773,6 +1891,9 @@ void xe_bo_unpin_external(struct xe_bo *bo)
Quoted 8 lines

drivers/gpu/drm/xe/xe_bo.c void xe_bo_unpin_external(struct xe_bo *bo) · line 1896

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
  	spin_unlock(&xe->pinned.lock);


  ttm_bo_unpin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +3 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);
+

Nit: Extra newline.

Will fix.

Quoted 397 lines
Quoted 397 lines


  /*
  * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1801,6 +1922,8 @@ void xe_bo_unpin(struct xe_bo *bo)
  	}


  ttm_bo_unpin(&bo->ttm);

drivers/gpu/drm/xe/xe_bo.c +2 -0
+	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
+		xe_ttm_tt_account(bo->ttm.ttm, true);

 }

 /**

drivers/gpu/drm/xe/xe_bo.h +0 -0
index 6de894c728f5..8463e3f3f6f1 100644
@@ -63,6 +63,7 @@
  #define XE_BO_PROPS_INVALID	(-1)


 struct sg_table;
+struct xe_ttm_lru_walk;

 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);

drivers/gpu/drm/xe/xe_bo.h +0 -0
@@ -126,6 +127,28 @@ static inline struct xe_bo *xe_bo_get(struct
 xe_bo *bo)
  	return bo;
  }


+/*
+ * xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an
+ * xe bo
+ * @bo: The bo for which we want to obtain a refcount.
+ *
+ * There is a short window between where the bo's GEM object refcount reaches
+ * zero and where we put the final ttm_bo reference. Code in the eviction- and
+ * shrinking path should therefore attempt to grab a gem object reference before
+ * trying to use members outside of the base class ttm object. This function is
+ * intended for that purpose. On successful return, this function must be paired
+ * with an xe_bo_put().
+ *
+ * Return: @bo on success, NULL on failure.
+ */
+static inline __must_check struct xe_bo *xe_bo_get_unless_zero(struct xe_bo *bo)

drivers/gpu/drm/xe/xe_bo.h +7 -0
+{
+	if (!bo || !kref_get_unless_zero(&bo->ttm.base.refcount))
+		return NULL;
+
+	return bo;
+}
+

 static inline void xe_bo_put(struct xe_bo *bo)
 {
  if (bo)

drivers/gpu/drm/xe/xe_bo.h +0 -0
@@ -315,6 +338,9 @@ static inline unsigned int
 xe_sg_segment_size(struct device *dev)


 #define
i915_gem_object_flush_if_display(obj) ((void)(obj))

+long xe_bo_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

drivers/gpu/drm/xe/xe_bo.h +2 -0
+		  bool purge, bool writeback);
+

 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given

drivers/gpu/drm/xe/xe_device.c +1 -0
index cfda7cb5df2c..58fecc4b0a18 100644
@@ -47,6 +47,7 @@
  #include "xe_perf.h"
  #include "xe_pm.h"
  #include "xe_query.h"
+#include "xe_shrinker.h"
  #include "xe_sriov.h"
  #include "xe_tile.h"
  #include "xe_ttm_stolen_mgr.h"
@@ -241,6 +242,9 @@ static void xe_device_destroy(struct drm_device
 *dev, void *dummy)
  	if (xe->unordered_wq)
  		destroy_workqueue(xe->unordered_wq);

drivers/gpu/drm/xe/xe_device.c +3 -0
+	if (!IS_ERR_OR_NULL(xe->mem.shrinker))
+		xe_shrinker_destroy(xe->mem.shrinker);
+

  ttm_device_fini(&xe->ttm);
 }

drivers/gpu/drm/xe/xe_device.c +0 -0
@@ -270,6 +274,10 @@ struct xe_device *xe_device_create(struct
 pci_dev *pdev,
  	if (err)
  		goto err;

drivers/gpu/drm/xe/xe_device.c +4 -0
+	xe->mem.shrinker = xe_shrinker_create(xe);
+	if (IS_ERR(xe->mem.shrinker))
+		return ERR_CAST(xe->mem.shrinker);
+

  xe->info.devid = pdev->device; xe->info.revid = pdev->revision; xe->info.force_execlist = xe_modparam.force_execlist;

drivers/gpu/drm/xe/xe_device_types.h +2 -0
index c37be471d11c..3d5440aba52e 100644
@@ -325,6 +325,8 @@ struct xe_device {
  		struct xe_mem_region vram;
  		/** @mem.sys_mgr: system TTM manager */
  		struct ttm_resource_manager sys_mgr;
+		/** @mem.sys_mgr: system memory shrinker. */
+		struct xe_shrinker *shrinker;
  	} mem;


  /** @sriov: device level virtualization data */

drivers/gpu/drm/xe/xe_shrinker.c new file +20 -0
new file mode 100644
index 000000000000..3f9554bdc06b
@@ -0,0 +1,287 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include <linux/shrinker.h>
+#include <linux/swap.h>
+
+#include <drm/ttm/ttm_bo.h>
+#include <drm/ttm/ttm_tt.h>
+
+#include "xe_bo.h"
+#include "xe_pm.h"
+#include "xe_shrinker.h"
+
+/**
+ * struct xe_shrinker - per-device shrinker
+ * @xe: Back pointer to the device.
+ * @lock: Lock protecting accounting.
+ * @shrinkable_pages: Number of pages that are currently

shrinkable.
+ * @purgeable_pages: Number of pages that are currently purgeable.
+ * @shrink: Pointer to the mm shrinker.
+ * @pm_worker: Worker to wake up the device if required.
+ */

drivers/gpu/drm/xe/xe_shrinker.c new file +32 -0
+struct xe_shrinker {
+	struct xe_device *xe;
+	rwlock_t lock;
+	long shrinkable_pages;
+	long purgeable_pages;
+	struct shrinker *shrink;
+	struct work_struct pm_worker;
+};
+
+/**
+ * struct xe_shrink_lru_walk - lru_walk subclass for shrinker
+ * @walk: The embedded base class.
+ * @xe: Pointer to the xe device.
+ * @purge: Purgeable only request from the srinker.
+ * @writeback: Try to write back to persistent storage.
+ */
+struct xe_shrink_lru_walk {
+	struct ttm_lru_walk walk;
+	struct xe_device *xe;
+	bool purge;
+	bool writeback;
+};
+
+static struct xe_shrinker *to_xe_shrinker(struct shrinker *shrink)
+{
+	return shrink->private_data;
+}
+
+static struct xe_shrink_lru_walk *
+to_xe_shrink_lru_walk(struct ttm_lru_walk *walk)
+{
+	return container_of(walk, struct xe_shrink_lru_walk,

walk);

drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+}
+
+/**
+ * xe_shrinker_mod_pages() - Modify shrinker page accounting
+ * @shrinker: Pointer to the struct xe_shrinker.
+ * @shrinkable: Shrinkable pages delta. May be negative.
+ * @purgeable: Purgeable page delta. May be negative.
+ *
+ * Modifies the shrinkable and purgeable pages accounting.
+ */
+void
+xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long

shrinkable, long purgeable)

drivers/gpu/drm/xe/xe_shrinker.c new file +8 -0
+{
+	write_lock(&shrinker->lock);
+	shrinker->shrinkable_pages += shrinkable;
+	shrinker->purgeable_pages += purgeable;
+	write_unlock(&shrinker->lock);
+}
+
+static long xe_shrinker_process_bo(struct ttm_lru_walk *walk,

struct ttm_buffer_object *bo)

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+{
+	struct xe_shrink_lru_walk *shrink_walk =

to_xe_shrink_lru_walk(walk);

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+
+	return xe_bo_shrink(walk, bo, shrink_walk->purge,

shrink_walk->writeback);

drivers/gpu/drm/xe/xe_shrinker.c new file +3 -0
+}
+
+static long xe_shrinker_walk(struct xe_shrink_lru_walk

*shrink_walk, long target)

drivers/gpu/drm/xe/xe_shrinker.c new file +14 -0
+{
+	struct xe_device *xe = shrink_walk->xe;
+	struct ttm_resource_manager *man;
+	unsigned int mem_type;
+	long sofar = 0;
+	long lret;
+
+	for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT;
++mem_type) {
+		man = ttm_manager_type(&xe->ttm, mem_type);
+		if (!man || !man->use_tt)
+			continue;
+
+		lret = ttm_lru_walk_for_evict(&shrink_walk->walk,

&xe->ttm, man, target);

drivers/gpu/drm/xe/xe_shrinker.c new file +13 -0
+		if (lret < 0)
+			return lret;
+
+		sofar += lret;
+		if (sofar >= target)
+			break;
+	}
+
+	return sofar;
+}
+
+static unsigned long
+xe_shrinker_count(struct shrinker *shrink, struct shrink_control

*sc)

drivers/gpu/drm/xe/xe_shrinker.c new file +7 -0
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	unsigned long num_pages;
+
+	num_pages = get_nr_swap_pages();
+	read_lock(&shrinker->lock);
+	num_pages = min_t(unsigned long, num_pages, shrinker-
Quoted 1 line

shrinkable_pages);

drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+	num_pages += shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	return num_pages ? num_pages : SHRINK_EMPTY;
+}
+
+static const struct ttm_lru_walk_ops xe_shrink_ops = {
+	.process_bo = xe_shrinker_process_bo,
+};
+
+/*
+ * Check if we need runtime pm, and if so try to grab a reference

if
+ * already active. If grabbing a reference fails, queue a worker that
+ * does it for us outside of reclaim, but don't wait for it to complete.
+ * If bo shrinking needs an rpm reference and we don't have it (yet),
+ * that bo will be skipped anyway.
+ */
+static bool xe_shrinker_runtime_pm_get(struct xe_shrinker *shrinker, bool force,

drivers/gpu/drm/xe/xe_shrinker.c new file +18 -0
+				       unsigned long nr_to_scan)
+{
+	struct xe_device *xe = shrinker->xe;
+
+	if (IS_DGFX(xe) || !xe_device_has_flat_ccs(xe) ||
+	    !get_nr_swap_pages())
+		return false;
+
+	if (!force) {
+		read_lock(&shrinker->lock);
+		force = (nr_to_scan > shrinker->purgeable_pages);
+		read_unlock(&shrinker->lock);
+		if (!force)
+			return false;
+	}
+
+	if (!xe_pm_runtime_get_if_active(xe)) {
+		queue_work(xe->unordered_wq, &shrinker-
Quoted 1 line

pm_worker);

drivers/gpu/drm/xe/xe_shrinker.c new file +7 -0
+		return false;
+	}
+
+	return true;
+}
+
+static void xe_shrinker_runtime_pm_put(struct xe_shrinker

*shrinker, bool runtime_pm)

drivers/gpu/drm/xe/xe_shrinker.c new file +6 -0
+{
+	if (runtime_pm)
+		xe_pm_runtime_put(shrinker->xe);
+}
+
+static unsigned long xe_shrinker_scan(struct shrinker *shrink,

struct shrink_control *sc)

drivers/gpu/drm/xe/xe_shrinker.c new file +30 -0
+{
+	struct xe_shrinker *shrinker = to_xe_shrinker(shrink);
+	bool is_kswapd = current_is_kswapd();
+	struct ttm_operation_ctx ctx = {
+		.interruptible = false,
+		.no_wait_gpu = !is_kswapd,
+	};
+	unsigned long nr_to_scan, freed = 0;
+	struct xe_shrink_lru_walk shrink_walk = {
+		.walk = {
+			.ops = &xe_shrink_ops,
+			.ctx = &ctx,
+			.trylock_only = true,
+		},
+		.xe = shrinker->xe,
+		.purge = true,
+		.writeback = is_kswapd,
+	};
+	bool runtime_pm;
+	bool purgeable;
+	long ret;
+
+	sc->nr_scanned = 0;
+	nr_to_scan = sc->nr_to_scan;
+
+	read_lock(&shrinker->lock);
+	purgeable = !!shrinker->purgeable_pages;
+	read_unlock(&shrinker->lock);
+
+	/* Might need runtime PM. Try to wake early if it looks

like it. */

drivers/gpu/drm/xe/xe_shrinker.c new file +1 -0
+	runtime_pm = xe_shrinker_runtime_pm_get(shrinker, false,

nr_to_scan);

drivers/gpu/drm/xe/xe_shrinker.c new file +18 -0
+
+	while (purgeable && freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+	if (freed < nr_to_scan)
+		nr_to_scan -= freed;
+	else
+		nr_to_scan = 0;
+	if (!nr_to_scan)
+		goto out;
+
+	/* If we didn't wake before, try to do it now if needed.

*/

drivers/gpu/drm/xe/xe_shrinker.c new file +2 -0
+	if (!runtime_pm)
+		runtime_pm = xe_shrinker_runtime_pm_get(shrinker,

true, 0);

drivers/gpu/drm/xe/xe_shrinker.c new file +23 -0
+
+	shrink_walk.purge = false;
+	nr_to_scan = sc->nr_to_scan;
+	while (freed < nr_to_scan) {
+		ret = xe_shrinker_walk(&shrink_walk, nr_to_scan);
+		if (ret <= 0)
+			break;
+
+		freed += ret;
+	}
+
+	sc->nr_scanned = freed;
+
+out:
+	xe_shrinker_runtime_pm_put(shrinker, runtime_pm);
+	return freed ? freed : SHRINK_STOP;
+}
+
+/* Wake up the device for shrinking. */
+static void xe_shrinker_pm(struct work_struct *work)
+{
+	struct xe_shrinker *shrinker =
+		container_of(work, typeof(*shrinker), pm_worker);
Quoted 8 lines

drivers/gpu/drm/xe/xe_shrinker.c line 242

Quoted 3 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +3 -0
+
+	xe_pm_runtime_get(shrinker->xe);
+	xe_pm_runtime_put(shrinker->xe);

So I don't really get this. How does this help the shrinker get a PM ref? The small window between xe_pm_runtime_get / put the shrinker grabs
one via xe_pm_runtime_get_if_active? Seems fragile.

Quoted 4 lines

The pm has a delay after the last put (Typically 1s?), and if the shrinker fails obtaining a reference it will be retried. Typically it will then succeed and when it does start a new delay.

Ah, forgot that their is a delay on the final put. This make sense now. Can't really think of a better solution that this at the moment. Maybe add a comment indicating this?

Matt

Quoted 114 lines

This doesn't strictly guarantee shrinking progress, but there's no way we can synchronously delay shrinking until we have the pm reference.

What we could perhaps do is to report back that we actually made progress but postpone the actual shrinking to the worker with writeback set to on; hoping that the shrinking core doesn't give up before it sees the released memory...

But I'd prefer to leave that out until we see that that's actually a real problem.

Best would be if we could have access the LNL CCS memory using the CPU...

/Thomas

Quoted 96 lines

Matt

Quoted 93 lines
drivers/gpu/drm/xe/xe_shrinker.c new file +12 -0
+}
+
+/**
+ * xe_shrinker_create() - Create an xe per-device shrinker
+ * @xe: Pointer to the xe device.
+ *
+ * Returns: A pointer to the created shrinker on success,
+ * Negative error code on failure.
+ */
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe)
+{
+	struct xe_shrinker *shrinker = kzalloc(sizeof(*shrinker),

GFP_KERNEL);

drivers/gpu/drm/xe/xe_shrinker.c new file +5 -0
+
+	if (!shrinker)
+		return ERR_PTR(-ENOMEM);
+
+	shrinker->shrink = shrinker_alloc(0, "xe system

shrinker");

drivers/gpu/drm/xe/xe_shrinker.c new file +28 -0
+	if (!shrinker->shrink) {
+		kfree(shrinker);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm);
+	shrinker->xe = xe;
+	rwlock_init(&shrinker->lock);
+	shrinker->shrink->count_objects = xe_shrinker_count;
+	shrinker->shrink->scan_objects = xe_shrinker_scan;
+	shrinker->shrink->private_data = shrinker;
+	shrinker_register(shrinker->shrink);
+
+	return shrinker;
+}
+
+/**
+ * xe_shrinker_destroy() - Destroy an xe per-device shrinker
+ * @shrinker: Pointer to the shrinker to destroy.
+ */
+void xe_shrinker_destroy(struct xe_shrinker *shrinker)
+{
+	xe_assert(shrinker->xe, !shrinker->shrinkable_pages);
+	xe_assert(shrinker->xe, !shrinker->purgeable_pages);
+	shrinker_free(shrinker->shrink);
+	flush_work(&shrinker->pm_worker);
+	kfree(shrinker);
+}
drivers/gpu/drm/xe/xe_shrinker.h new file +12 -0
new file mode 100644
index 000000000000..28a038f4fcbf
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_SHRINKER_H_
+#define _XE_SHRINKER_H_
+
+struct xe_shrinker;
+struct xe_device;
+
+void xe_shrinker_mod_pages(struct xe_shrinker *shrinker, long

shrinkable, long purgeable);

drivers/gpu/drm/xe/xe_shrinker.h new file +6 -0
+
+struct xe_shrinker *xe_shrinker_create(struct xe_device *xe);
+
+void xe_shrinker_destroy(struct xe_shrinker *shrinker);
+
+#endif
include/drm/ttm/ttm_bo.h +0 -0
index e577528f5dfc..c7e81ae025d9 100644
@@ -229,6 +229,9 @@ struct ttm_lru_walk {

 long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
      struct ttm_resource_manager *man, long target);

+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,

include/drm/ttm/ttm_bo.h +2 -0
+		       bool purge, bool writeback);
+

 /**
  * ttm_bo_get - reference a struct ttm_buffer_object *

Signature · 1 line
2.44.0
reviewer
#31 lore raw

Re: [PATCH v6 11/12] drm/ttm, drm/xe: Add a shrinker for xe bos

In reply to Thomas Hellström

Hi,

Quoted 412 lines · Thomas Hellström

Rather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.

Leverage the newly added TTM per-page shrinking and shmem backup support.

Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.

v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot).

Cc Christian König <[email protected]>
Cc Somalapuram Amaranath <[email protected]>
Cc Matthew Brost <[email protected]>
---
  drivers/gpu/drm/ttm/ttm_bo_util.c     |  67 ++++++
  drivers/gpu/drm/xe/Makefile           |   1 +
  drivers/gpu/drm/xe/tests/xe_bo.c      | 118 +++++++++++
  drivers/gpu/drm/xe/tests/xe_bo_test.c |   1 +
  drivers/gpu/drm/xe/tests/xe_bo_test.h |   1 +
  drivers/gpu/drm/xe/xe_bo.c            | 155 ++++++++++++--
  drivers/gpu/drm/xe/xe_bo.h            |  26 +++
  drivers/gpu/drm/xe/xe_device.c        |   8 +
  drivers/gpu/drm/xe/xe_device_types.h  |   2 +
  drivers/gpu/drm/xe/xe_shrinker.c      | 287 ++++++++++++++++++++++++++
  drivers/gpu/drm/xe/xe_shrinker.h      |  18 ++
  include/drm/ttm/ttm_bo.h              |   3 +
  12 files changed, 671 insertions(+), 16 deletions(-)

create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h

drivers/gpu/drm/ttm/ttm_bo_util.c +67 -0
index c4f678f30fc2..563e96a4cf06 100644
@@ -924,3 +924,70 @@ long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,
 
  	return progress;
  }
+EXPORT_SYMBOL(ttm_lru_walk_for_evict);
+
+/**
+ * ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer object.
+ * @walk: The struct xe_ttm_lru_walk that describes the walk.
+ * @bo: The buffer object.
+ * @purge: Whether to attempt to purge the bo content since it's no
+ * longer needed.
+ * @writeback: If !@purge, attempt to write out to persistent storage.
+ *
+ * The function uses the ttm_tt_back_up functionality to back up or
+ * purge a struct ttm_tt. If the bo is not in system, it's first
+ * moved there.
+ *
+ * Return: The number of pages shrunken or purged, or
+ * negative error code on failure.
+ */
+long ttm_bo_try_shrink(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo,
+		       bool purge, bool writeback)
+{
+	static const struct ttm_place sys_placement_flags = {
+		.fpfn = 0,
+		.lpfn = 0,
+		.mem_type = TTM_PL_SYSTEM,
+		.flags = 0,
+	};
+	static struct ttm_placement sys_placement = {
+		.num_placement = 1,
+		.placement = &sys_placement_flags,
+	};
+	struct ttm_operation_ctx *ctx = walk->ctx;
+	struct ttm_tt *tt = bo->ttm;
+	long lret;
+
+	dma_resv_assert_held(bo->base.resv);
+
+	if (!tt || !ttm_tt_is_populated(tt))
+		return 0;
+
+	if (bo->resource->mem_type != TTM_PL_SYSTEM) {
+		int ret = ttm_bo_validate(bo, &sys_placement, ctx);
+
+		if (ret) {
+			if (ret == -EINTR || ret == -EDEADLK ||
+			    ret == -ERESTARTSYS)
+				return ret;
+			return 0;
+		}
+	}
+
+	lret = ttm_bo_wait_ctx(bo, ctx);
+	if (lret < 0) {
+		if (lret == -ERESTARTSYS)
+			return lret;
+		return 0;
+	}
+
+	if (bo->deleted)
+		lret = ttm_tt_backup(bo->bdev, tt, true, writeback);
+	else
+		lret = ttm_tt_backup(bo->bdev, tt, purge, writeback);
+	if (lret < 0 && lret != -EINTR)
+		return 0;
+
+	return lret;
+}
+EXPORT_SYMBOL(ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
index b1e03bfe4a68..1eba51bdd172 100644
@@ -112,6 +112,7 @@ xe-y += xe_bb.o \
  	xe_ring_ops.o \
  	xe_sa.o \
  	xe_sched_job.o \
+	xe_shrinker.o \
  	xe_step.o \
  	xe_sync.o \
  	xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +118 -0
index 9f3c02826464..49617f16dc76 100644
@@ -6,6 +6,8 @@
  #include <kunit/test.h>
  #include <kunit/visibility.h>
 
+#include <uapi/linux/sysinfo.h>
+
  #include "tests/xe_bo_test.h"
  #include "tests/xe_pci_test.h"
  #include "tests/xe_test.h"
@@ -350,3 +352,119 @@ void xe_bo_evict_kunit(struct kunit *test)
  	xe_call_for_each_device(evict_test_run_device);
  }
  EXPORT_SYMBOL_IF_KUNIT(xe_bo_evict_kunit);
+
+struct xe_bo_link {
+	struct list_head link;
+	struct xe_bo *bo;
+};
+
+#define XE_BO_SHRINK_SIZE ((unsigned long)SZ_64M)
+
+/*
+ * Try to create system bos corresponding to twice the amount
+ * of available system memory to test shrinker functionality.
+ * If no swap space is available to accommodate the
+ * memory overcommit, mark bos purgeable.
+ */
+static int shrink_test_run_device(struct xe_device *xe)
+{
+	struct kunit *test = xe_cur_kunit();
+	LIST_HEAD(bos);
+	struct xe_bo_link *link, *next;
+	struct sysinfo si;
+	size_t total, alloced;
+	unsigned int interrupted = 0, successful = 0;
+
+	si_meminfo(&si);
+	total = si.freeram * si.mem_unit;
+
+	kunit_info(test, "Free ram is %lu bytes. Will allocate twice of that.\n",
+		   (unsigned long) total);
+
+	total <<= 1;
+	for (alloced = 0; alloced < total ; alloced += XE_BO_SHRINK_SIZE) {
+		struct xe_bo *bo;
+		unsigned int mem_type;
+
+		link = kzalloc(sizeof(*link), GFP_KERNEL);
+		if (!link) {
+			KUNIT_FAIL(test, "Unexpeced link allocation failure\n");
+			break;
+		}
+
+		INIT_LIST_HEAD(&link->link);
+
+		/* We can create bos using WC caching here. But it is slower. */
+		bo = xe_bo_create_user(xe, NULL, NULL, XE_BO_SHRINK_SIZE,
+				       DRM_XE_GEM_CPU_CACHING_WB,
+				       ttm_bo_type_device,
+				       XE_BO_FLAG_SYSTEM);
+		if (IS_ERR(bo)) {
+			if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
+			    bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
+				KUNIT_FAIL(test, "Error creating bo: %pe\n", bo);
+			kfree(link);
+			break;
+		}
+		link->bo = bo;
+		list_add_tail(&link->link, &bos);
+		xe_bo_lock(bo, false);
+
+		/*
+		 * If we're low on swap entries, we can't shrink unless the bo
+		 * is marked purgeable.
+		 */
+		if (get_nr_swap_pages() < (XE_BO_SHRINK_SIZE >> PAGE_SHIFT) * 128) {
+			struct xe_ttm_tt *xe_tt =
+				container_of(bo->ttm.ttm, typeof(*xe_tt), ttm);
+			long num_pages = xe_tt->ttm.num_pages;
+
+			xe_tt->purgeable = true;
+			xe_shrinker_mod_pages(xe->mem.shrinker, -num_pages,
+					      num_pages);
+		}
+
+		mem_type = bo->ttm.resource->mem_type;
+		xe_bo_unlock(bo);
+		if (mem_type != XE_PL_TT)
+			KUNIT_FAIL(test, "Bo in incorrect memory type: %u\n",
+				   bo->ttm.resource->mem_type);
+		cond_resched();
+		if (signal_pending(current))
+			break;
+	}
+
+	/* Read back and destroy bos */
+	list_for_each_entry_safe_reverse(link, next, &bos, link) {
+		static struct ttm_operation_ctx ctx = {.interruptible = true};
+		struct xe_bo *bo = link->bo;
+		int ret;
+
+		if (!signal_pending(current)) {
+			xe_bo_lock(bo, NULL);
+			ret = ttm_bo_validate(&bo->ttm, &tt_placement, &ctx);
+			xe_bo_unlock(bo);
+			if (ret && ret != -EINTR)
+				KUNIT_FAIL(test, "Validation failed: %pe\n",
+					   ERR_PTR(ret));
+			else if (ret)
+				interrupted++;
+			else
+				successful++;
+		}
+		xe_bo_put(link->bo);
+		list_del(&link->link);
+		kfree(link);
+		cond_resched();
+	}
+	kunit_info(test, "Readbacks interrupted: %u successful: %u\n",
+		   interrupted, successful);
+
+	return 0;
+}
+
+void xe_bo_shrink_kunit(struct kunit *test)
+{
+	xe_call_for_each_device(shrink_test_run_device);
+}
+EXPORT_SYMBOL_IF_KUNIT(xe_bo_shrink_kunit);
drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
index a324cde77db8..317fa923e287 100644
@@ -10,6 +10,7 @@
  static struct kunit_case xe_bo_tests[] = {
  	KUNIT_CASE(xe_ccs_migrate_kunit),
  	KUNIT_CASE(xe_bo_evict_kunit),
+	KUNIT_CASE_SLOW(xe_bo_shrink_kunit),
  	{}
  };
 
drivers/gpu/drm/xe/tests/xe_bo_test.h +1 -0
index 0113ab45066a..7f44d14a45c5 100644
@@ -10,5 +10,6 @@ struct kunit;
 
  void xe_ccs_migrate_kunit(struct kunit *test);
  void xe_bo_evict_kunit(struct kunit *test);
+void xe_bo_shrink_kunit(struct kunit *test);
 
  #endif
drivers/gpu/drm/xe/xe_bo.c +38 -10
index 65c696966e96..6ab63d1642ae 100644
@@ -10,6 +10,7 @@
  #include <drm/drm_drv.h>
  #include <drm/drm_gem_ttm_helper.h>
  #include <drm/drm_managed.h>
+#include <drm/ttm/ttm_backup.h>
  #include <drm/ttm/ttm_device.h>
  #include <drm/ttm/ttm_placement.h>
  #include <drm/ttm/ttm_tt.h>
@@ -25,6 +26,7 @@
  #include "xe_pm.h"
  #include "xe_preempt_fence.h"
  #include "xe_res_cursor.h"
+#include "xe_shrinker.h"
  #include "xe_trace_bo.h"
  #include "xe_ttm_stolen_mgr.h"
  #include "xe_vm.h"
@@ -278,11 +280,15 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo,
  	}
  }
 
+/* struct xe_ttm_tt - Subclassed ttm_tt for xe */
  struct xe_ttm_tt {
  	struct ttm_tt ttm;
-	struct device *dev;
+	/** @xe - The xe device */
+	struct xe_device *xe;
  	struct sg_table sgt;
  	struct sg_table *sg;
+	/** @purgeable - Whether the bo is purgeable (WONTNEED) */
+	bool purgeable;
  };
 
  static int xe_tt_map_sg(struct ttm_tt *tt)
@@ -291,7 +297,8 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
  	unsigned long num_pages = tt->num_pages;
  	int ret;
 
-	XE_WARN_ON(tt->page_flags & TTM_TT_FLAG_EXTERNAL);
+	XE_WARN_ON((tt->page_flags & TTM_TT_FLAG_EXTERNAL) &&
+		   !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE));
 
  	if (xe_tt->sg)
  		return 0;
@@ -299,13 +306,13 @@ static int xe_tt_map_sg(struct ttm_tt *tt)
  	ret = sg_alloc_table_from_pages_segment(&xe_tt->sgt, tt->pages,
  						num_pages, 0,
  						(u64)num_pages << PAGE_SHIFT,
-						xe_sg_segment_size(xe_tt->dev),
+						xe_sg_segment_size(xe_tt->xe->drm.dev),
  						GFP_KERNEL);
  	if (ret)
  		return ret;
 
  	xe_tt->sg = &xe_tt->sgt;
-	ret = dma_map_sgtable(xe_tt->dev, xe_tt->sg, DMA_BIDIRECTIONAL,
+	ret = dma_map_sgtable(xe_tt->xe->drm.dev, xe_tt->sg, DMA_BIDIRECTIONAL,
  			      DMA_ATTR_SKIP_CPU_SYNC);
  	if (ret) {
  		sg_free_table(xe_tt->sg);
@@ -321,7 +328,7 @@ static void xe_tt_unmap_sg(struct ttm_tt *tt)
  	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
  	if (xe_tt->sg) {
-		dma_unmap_sgtable(xe_tt->dev, xe_tt->sg,
+		dma_unmap_sgtable(xe_tt->xe->drm.dev, xe_tt->sg,
  				  DMA_BIDIRECTIONAL, 0);
  		sg_free_table(xe_tt->sg);
  		xe_tt->sg = NULL;
@@ -336,21 +343,41 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
  	return xe_tt->sg;
  }
 
+/*
+ * Account ttm pages against the device shrinker's shrinkable and
+ * purgeable counts.
+ */
+static void xe_ttm_tt_account(struct ttm_tt *tt, bool add)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	long num_pages = tt->num_pages;
+
+	if (!add)
+		num_pages = -num_pages;
+
+	if (xe_tt->purgeable)
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, 0, num_pages);
+	else
+		xe_shrinker_mod_pages(xe_tt->xe->mem.shrinker, num_pages, 0);
+}
+
  static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
  				       u32 page_flags)
  {
  	struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
  	struct xe_device *xe = xe_bo_device(bo);
-	struct xe_ttm_tt *tt;
+	struct xe_ttm_tt *xe_tt;
+	struct ttm_tt *tt;
  	unsigned long extra_pages;
  	enum ttm_caching caching;
  	int err;
 
-	tt = kzalloc(sizeof(*tt), GFP_KERNEL);
-	if (!tt)
+	xe_tt = kzalloc(sizeof(*xe_tt), GFP_KERNEL);
+	if (!xe_tt)
  		return NULL;
 
-	tt->dev = xe->drm.dev;
+	tt = &xe_tt->ttm;
+	xe_tt->xe = xe;
 
  	extra_pages = 0;
  	if (xe_bo_needs_ccs_pages(bo))
@@ -387,42 +414,128 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
  		caching = ttm_uncached;
  	}
 
-	err = ttm_tt_init(&tt->ttm, &bo->ttm, page_flags, caching, extra_pages);
+	if (ttm_bo->type != ttm_bo_type_sg)
+		page_flags |= TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE;

drivers/gpu/drm/xe/xe_bo.c static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, · line 426

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +4 -2
+
+	err = ttm_tt_init(tt, &bo->ttm, page_flags, caching, extra_pages);
  	if (err) {
-		kfree(tt);
+		kfree(xe_tt);
  		return NULL;
  	}
 
-	return &tt->ttm;
+	tt->backup = ttm_backup_shmem_create(tt->num_pages << PAGE_SHIFT);

I guess we should make this (loff_t)tt->num_pages << PAGE_SHIFT or similar?

author maintainer
#32 lore raw

PATCH v6 12/12 drm/xe: Increase the XE_PL_TT watermark

In reply to Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -2
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
 	u64 gtt_size;
 
 	si_meminfo(&si);
+	/* Potentially restrict amount of TT memory here. */
 	gtt_size = (u64)si.totalram * si.mem_unit;
-	/* TTM limits allocation of all TTM devices by 50% of system memory */
-	gtt_size /= 2;
 
 	man->use_tt = true;
 	man->func = &xe_ttm_sys_mgr_func;
Signature · 1 line
2.44.0
#33 lore raw

Re: [PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

Quoted 7 lines · Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

Quoted 8 lines

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

Sorry if I missed some patch changing it but I did not found in this series anything changing the 50% limit in ttm_global_init(). When I debugged some Vulkan tests allocate a lot of memory, the reason that KMD was not allocating memory wash this ttm_global limit that is shared with all devices using TTM.

Quoted 21 lines
---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -2
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
 	u64 gtt_size;
 
 	si_meminfo(&si);
+	/* Potentially restrict amount of TT memory here. */
 	gtt_size = (u64)si.totalram * si.mem_unit;
-	/* TTM limits allocation of all TTM devices by 50% of system memory */
-	gtt_size /= 2;
 
 	man->use_tt = true;
 	man->func = &xe_ttm_sys_mgr_func;
maintainer
#34 lore raw

Re: [PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

Quoted 13 lines · Jose
Quoted 12 lines · Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Quoted 8 lines
Quoted 3 lines

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

Sorry if I missed some patch changing it but I did not found in this series anything changing the 50% limit in ttm_global_init(). When I debugged some Vulkan tests allocate a lot of memory, the reason that KMD was not allocating memory wash this ttm_global limit that is shared with all devices using TTM.

I'm reviewing this series and starting make sense of all this.

Thomas please correct me if I'm wrong here...

The limit set in ttm_global_init is the watermark for the TTM pool where if exceeded upon freeing a BO's pages the pages are actually freed rather than just returning to the TTM pool cache. The global watermark is reason why in issue #2438 it observed a bunch of memory is still consumed when nothing is running or any BOs exist - pages are being cached in the TTM pool. The global watermark doesn't actually limit the amount system memory TTM can allocate. A shrinker also exists which can free cached pages in the TTM pool if memory pressure exists or 'echo 3 > /proc/sys/vm/drop_caches' is done.

The watermark changed in this patch, is the actual limit for the number of pages we can allocate for BOs. With a shrinker hooked into BOs, we now can freely allocate all of the system pages for BOs and if memory pressure exists idle BOs pages are swapped to shmem via the shrinker and restored upon next GPU use.

Matt

Quoted 22 lines
Quoted 21 lines
---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -2
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
 	u64 gtt_size;
 
 	si_meminfo(&si);
+	/* Potentially restrict amount of TT memory here. */
 	gtt_size = (u64)si.totalram * si.mem_unit;
-	/* TTM limits allocation of all TTM devices by 50% of system memory */
-	gtt_size /= 2;
 
 	man->use_tt = true;
 	man->func = &xe_ttm_sys_mgr_func;
author maintainer
#35 lore raw

Re: [PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

Quoted 30 lines · Matthew Brost
Quoted 17 lines · Jose
Quoted 7 lines · Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

Quoted 8 lines

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

Quoted 7 lines

Sorry if I missed some patch changing it but I did not found in this series anything changing the 50% limit in ttm_global_init(). When I debugged some Vulkan tests allocate a lot of memory, the reason that KMD was not allocating memory wash this ttm_global limit that is shared
with all devices using TTM.

I'm reviewing this series and starting make sense of all this.

Thomas please correct me if I'm wrong here...

Quoted 8 lines

The limit set in ttm_global_init is the watermark for the TTM pool where
if exceeded upon freeing a BO's pages the pages are actually freed rather than just returning to the TTM pool cache. The global watermark
is reason why in issue #2438 it observed a bunch of memory is still consumed when nothing is running or any BOs exist - pages are being cached in the TTM pool.

This is correct.

Quoted 6 lines

The global watermark doesn't actually limit the
amount system memory TTM can allocate. A shrinker also exists which can
free cached pages in the TTM pool if memory pressure exists or 'echo 3 >
/proc/sys/vm/drop_caches' is done.

Yes, this is also true except the global watermark should be called the pool watermark.

There is another global watermark that, if the amount of pages used for graphics (PL_SYSTEM or PL_TT) exceeds 50% of system, bo swapping starts. That means bos are pulled of the various LRU lists in a device round-robin fashion and moved to shmem objects, in the anticipation that theses shmem objects can then be paged out to disc by the core. However, this is done even if there is no disc swap-space attached. Also if this shmem swapping fails, nothing happens but the allocation is free to proceed anyway.

This is what I typically refer to as the global watermark. It used to be implemented by an opportunistic swapper process (similar to kswapd) and a direct swapper (similar to direct reclaim) and the 50% limit was configurable, but much of that functionality was ripped out. I didn't follow the discussion preceding that change, though.

Quoted 4 lines

The watermark changed in this patch, is the actual limit for the number
of pages we can allocate for BOs.

What is changed in this patch is actually the amount to memory we can have in the TT placement and therefore also bound to the device. If this limit is exceeded, eviction from TT to SYSTEM will take place in addition to the above global swapping. This limit is per device. So now we can in theory have all of system bound to a device.

Quoted 5 lines

With a shrinker hooked into BOs, we
now can freely allocate all of the system pages for BOs and if memory pressure exists idle BOs pages are swapped to shmem via the shrinker and
restored upon next GPU use.

Exactly.

Quoted 2 lines

Matt

/Thomas

Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

Signed-off-by: Thomas Hellström
<[email protected]>
---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--

 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -0
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
  	u64 gtt_size;


  si_meminfo(&si);

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -0
+	/* Potentially restrict amount of TT memory here. */

  gtt_size = (u64)si.totalram * si.mem_unit;

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -1
-	/* TTM limits allocation of all TTM devices by 50% of

system memory */

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -1
-	gtt_size /= 2;


  man->use_tt = true;
  man->func = &xe_ttm_sys_mgr_func;

maintainer Reviewed-by on v6 12/12
#36 lore raw

Re: [PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

In reply to Thomas Hellström
Quoted 12 lines · Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Quoted 5 lines

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

Quoted 22 lines
---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -2
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
 	u64 gtt_size;
 
 	si_meminfo(&si);
+	/* Potentially restrict amount of TT memory here. */
 	gtt_size = (u64)si.totalram * si.mem_unit;
-	/* TTM limits allocation of all TTM devices by 50% of system memory */
-	gtt_size /= 2;
 
 	man->use_tt = true;
 	man->func = &xe_ttm_sys_mgr_func;
Signature · 1 line
2.44.0
author maintainer
#37 lore raw

Re: [PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

Quoted 13 lines · Matthew Brost
Quoted 12 lines · Thomas Hellström

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Quoted 7 lines
Quoted 5 lines

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

Thanks for reviewing!
/Thomas

Quoted 25 lines
Quoted 25 lines

---
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--

 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -0
index 9844a8edbfe1..d38b91872da3 100644
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
  	u64 gtt_size;


  si_meminfo(&si);

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -0
+	/* Potentially restrict amount of TT memory here. */

  gtt_size = (u64)si.totalram * si.mem_unit;

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -1
-	/* TTM limits allocation of all TTM devices by 50% of

system memory */

drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +0 -1
-	gtt_size /= 2;


  man->use_tt = true;
  man->func = &xe_ttm_sys_mgr_func;

Signature · 1 line
2.44.0
#38 lore raw

✓ CI.Patch_applied: success for TTM shrinker helpers and xe buffer object shrinker (rev6)

In reply to Thomas Hellström
== Series Details ==

== Series Details ==

Series: TTM shrinker helpers and xe buffer object shrinker (rev6)
URL : https://patchwork.freedesktop.org/series/131815/
State : success

== Summary ==

=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: f9bae9be5922 drm-tip: 2024y-07m-03d-15h-56m-08s UTC integration manifest === git am output follows ===
Applying: drm/ttm: Allow TTM LRU list nodes of different types
Applying: drm/ttm: Slightly clean up LRU list iteration
Applying: drm/ttm: Use LRU hitches
Applying: drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves
Applying: drm/ttm: Provide a generic LRU walker helper
Applying: drm/ttm: Use the LRU walker helper for swapping
Applying: drm/ttm: Use the LRU walker for eviction
Applying: drm/ttm: Add a virtual base class for graphics memory backup
Applying: drm/ttm/pool: Provide a helper to shrink pages
Applying: drm/ttm: Use fault-injection to test error paths
Applying: drm/ttm, drm/xe: Add a shrinker for xe bos
Applying: drm/xe: Increase the XE_PL_TT watermark

#39 lore raw

✗ CI.checkpatch: warning for TTM shrinker helpers and xe buffer object shrinker (rev6)

In reply to Thomas Hellström
== Series Details ==

== Series Details ==

Series: TTM shrinker helpers and xe buffer object shrinker (rev6)
URL : https://patchwork.freedesktop.org/series/131815/
State : warning

== Summary ==

+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 51ce9f6cd981d42d7467409d7dbc559a450abc1e
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 81a86c968e0e541d7e4fdf5c04b58ad8e35ecb8f

Author: Thomas Hellström <[email protected]>
Date:   Wed Jul 3 17:38:13 2024 +0200

drm/xe: Increase the XE_PL_TT watermark

The XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_SYSTEM memory.

The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.

Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.

+ /mt/dim checkpatch f9bae9be5922368e36324928b898ff4de5280e30 drm-intel b606d4e5c496 drm/ttm: Allow TTM LRU list nodes of different types
52d1cc0f4ed2 drm/ttm: Slightly clean up LRU list iteration
2b9cc3b8ef0c drm/ttm: Use LRU hitches
05f6c9f0cb17 drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves 4bd764e605c9 drm/ttm: Provide a generic LRU walker helper
b2d00bad54e5 drm/ttm: Use the LRU walker helper for swapping
fcbe98e2c9db drm/ttm: Use the LRU walker for eviction
5f3d74c7637c drm/ttm: Add a virtual base class for graphics memory backup Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:49: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #49:
new file mode 100644

total: 0 errors, 1 warnings, 0 checks, 284 lines checked
e909931efcdc drm/ttm/pool: Provide a helper to shrink pages
3c62b059423f drm/ttm: Use fault-injection to test error paths
-:28: WARNING:CONFIG_DESCRIPTION: please write a help paragraph that fully describes the config symbol #28: FILE: drivers/gpu/drm/Kconfig:275:
+config DRM_TTM_BACKUP_FAULT_INJECT
+ bool "Enable fault injection during TTM backup"
+ depends on DRM_TTM
+ default n
+ help
+ Inject recoverable failures during TTM backup and recovery of
+ backed-up objects. For DRM driver developers only.
+
+ If in doubt, choose N.
+

total: 0 errors, 1 warnings, 0 checks, 51 lines checked 6a5aceb5f7a8 drm/ttm, drm/xe: Add a shrinker for xe bos Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply' Traceback (most recent call last):
File "scripts/spdxcheck.py", line 6, in <module>
from ply import lex, yacc
ModuleNotFoundError: No module named 'ply' -:168: CHECK:SPACING: No space is necessary after a cast #168: FILE: drivers/gpu/drm/xe/tests/xe_bo.c:382:
+ (unsigned long) total);

-:677: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating? #677:
new file mode 100644

total: 0 errors, 1 warnings, 1 checks, 891 lines checked 81a86c968e0e drm/xe: Increase the XE_PL_TT watermark

#40 lore raw

✗ CI.KUnit: failure for TTM shrinker helpers and xe buffer object shrinker (rev6)

In reply to Thomas Hellström
== Series Details ==

== Series Details ==

Series: TTM shrinker helpers and xe buffer object shrinker (rev6)
URL : https://patchwork.freedesktop.org/series/131815/
State : failure

== Summary ==

+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig [17:35:41] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[17:35:46] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
../lib/iomap.c:156:5: warning: no previous prototype for ‘ioread64_lo_hi’ [-Wmissing-prototypes]
  156 | u64 ioread64_lo_hi(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~
../lib/iomap.c:163:5: warning: no previous prototype for ‘ioread64_hi_lo’ [-Wmissing-prototypes]
  163 | u64 ioread64_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~
../lib/iomap.c:170:5: warning: no previous prototype for ‘ioread64be_lo_hi’ [-Wmissing-prototypes]
  170 | u64 ioread64be_lo_hi(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:178:5: warning: no previous prototype for ‘ioread64be_hi_lo’ [-Wmissing-prototypes]
  178 | u64 ioread64be_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:264:6: warning: no previous prototype for ‘iowrite64_lo_hi’ [-Wmissing-prototypes]
  264 | void iowrite64_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:272:6: warning: no previous prototype for ‘iowrite64_hi_lo’ [-Wmissing-prototypes]
  272 | void iowrite64_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:280:6: warning: no previous prototype for ‘iowrite64be_lo_hi’ [-Wmissing-prototypes]
  280 | void iowrite64be_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~
../lib/iomap.c:288:6: warning: no previous prototype for ‘iowrite64be_hi_lo’ [-Wmissing-prototypes]
  288 | void iowrite64be_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~

[17:36:11] Starting KUnit Kernel (1/1)...
[17:36:11] ============================================================ Running tests with:

$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[17:36:11] =================== guc_dbm (7 subtests) ===================
[17:36:11] [PASSED] test_empty
[17:36:11] [PASSED] test_default
[17:36:11] ======================== test_size  ========================
[17:36:11] [PASSED] 4
[17:36:11] [PASSED] 8
[17:36:11] [PASSED] 32
[17:36:11] [PASSED] 256
[17:36:11] ==================== [PASSED] test_size ====================
[17:36:11] ======================= test_reuse  ========================
[17:36:11] [PASSED] 4
[17:36:11] [PASSED] 8
[17:36:11] [PASSED] 32
[17:36:11] [PASSED] 256
[17:36:11] =================== [PASSED] test_reuse ====================
[17:36:11] =================== test_range_overlap  ====================
[17:36:11] [PASSED] 4
[17:36:11] [PASSED] 8
[17:36:11] [PASSED] 32
[17:36:11] [PASSED] 256
[17:36:11] =============== [PASSED] test_range_overlap ================
[17:36:11] =================== test_range_compact  ====================
[17:36:11] [PASSED] 4
[17:36:11] [PASSED] 8
[17:36:11] [PASSED] 32
[17:36:11] [PASSED] 256
[17:36:11] =============== [PASSED] test_range_compact ================
[17:36:11] ==================== test_range_spare  =====================
[17:36:11] [PASSED] 4
[17:36:11] [PASSED] 8
[17:36:11] [PASSED] 32
[17:36:11] [PASSED] 256
[17:36:11] ================ [PASSED] test_range_spare =================
[17:36:11] ===================== [PASSED] guc_dbm =====================
[17:36:11] =================== guc_idm (6 subtests) ===================
[17:36:11] [PASSED] bad_init
[17:36:11] [PASSED] no_init
[17:36:11] [PASSED] init_fini
[17:36:11] [PASSED] check_used
[17:36:11] [PASSED] check_quota
[17:36:11] [PASSED] check_all
[17:36:11] ===================== [PASSED] guc_idm =====================
[17:36:11] ================== no_relay (3 subtests) ===================
[17:36:11] [PASSED] xe_drops_guc2pf_if_not_ready
[17:36:11] [PASSED] xe_drops_guc2vf_if_not_ready
[17:36:11] [PASSED] xe_rejects_send_if_not_ready
[17:36:11] ==================== [PASSED] no_relay =====================
[17:36:11] ================== pf_relay (14 subtests) ==================
[17:36:11] [PASSED] pf_rejects_guc2pf_too_short
[17:36:11] [PASSED] pf_rejects_guc2pf_too_long
[17:36:11] [PASSED] pf_rejects_guc2pf_no_payload
[17:36:11] [PASSED] pf_fails_no_payload
[17:36:11] [PASSED] pf_fails_bad_origin
[17:36:11] [PASSED] pf_fails_bad_type
[17:36:11] [PASSED] pf_txn_reports_error
[17:36:11] [PASSED] pf_txn_sends_pf2guc
[17:36:11] [PASSED] pf_sends_pf2guc
[17:36:11] [SKIPPED] pf_loopback_nop
[17:36:11] [SKIPPED] pf_loopback_echo
[17:36:11] [SKIPPED] pf_loopback_fail
[17:36:11] [SKIPPED] pf_loopback_busy
[17:36:11] [SKIPPED] pf_loopback_retry
[17:36:11] ==================== [PASSED] pf_relay =====================
[17:36:11] ================== vf_relay (3 subtests) ===================
[17:36:11] [PASSED] vf_rejects_guc2vf_too_short
[17:36:11] [PASSED] vf_rejects_guc2vf_too_long
[17:36:11] [PASSED] vf_rejects_guc2vf_no_payload
[17:36:11] ==================== [PASSED] vf_relay =====================
[17:36:11] ================= pf_service (11 subtests) =================
[17:36:11] [PASSED] pf_negotiate_any
[17:36:11] [PASSED] pf_negotiate_base_match
[17:36:11] [PASSED] pf_negotiate_base_newer
[17:36:11] [PASSED] pf_negotiate_base_next
[17:36:11] [SKIPPED] pf_negotiate_base_older
[17:36:11] [PASSED] pf_negotiate_base_prev
[17:36:11] [PASSED] pf_negotiate_latest_match
[17:36:11] [PASSED] pf_negotiate_latest_newer
[17:36:11] [PASSED] pf_negotiate_latest_next
[17:36:11] [SKIPPED] pf_negotiate_latest_older
[17:36:11] [SKIPPED] pf_negotiate_latest_prev
[17:36:11] =================== [PASSED] pf_service ====================
[17:36:11] ===================== lmtt (1 subtest) =====================
[17:36:11] ======================== test_ops  =========================
[17:36:11] [PASSED] 2-level
[17:36:11] [PASSED] multi-level
[17:36:11] ==================== [PASSED] test_ops =====================
[17:36:11] ====================== [PASSED] lmtt =======================
[17:36:11] ==================== xe_bo (3 subtests) ====================
[17:36:11] [SKIPPED] xe_ccs_migrate_kunit
[17:36:11] [SKIPPED] xe_bo_evict_kunit
[17:36:11] [SKIPPED] xe_bo_shrink_kunit
[17:36:11] ===================== [SKIPPED] xe_bo ======================
[17:36:11] ================== xe_dma_buf (1 subtest) ==================
[17:36:11] [SKIPPED] xe_dma_buf_kunit
[17:36:11] =================== [SKIPPED] xe_dma_buf ===================
[17:36:11] ================== xe_migrate (1 subtest) ==================
[17:36:11] [SKIPPED] xe_migrate_sanity_kunit
[17:36:11] =================== [SKIPPED] xe_migrate ===================
[17:36:11] =================== xe_mocs (2 subtests) ===================
[17:36:11] [SKIPPED] xe_live_mocs_kernel_kunit
[17:36:11] [SKIPPED] xe_live_mocs_reset_kunit
[17:36:11] ==================== [SKIPPED] xe_mocs =====================
[17:36:11] ==================== args (11 subtests) ====================
[17:36:11] [PASSED] count_args_test
[17:36:11] [PASSED] call_args_example
[17:36:11] [PASSED] call_args_test
[17:36:11] [PASSED] drop_first_arg_example
[17:36:11] [PASSED] drop_first_arg_test
[17:36:11] [PASSED] first_arg_example
[17:36:11] [PASSED] first_arg_test
[17:36:11] [PASSED] last_arg_example
[17:36:11] [PASSED] last_arg_test
[17:36:11] [PASSED] pick_arg_example
[17:36:11] [PASSED] sep_comma_example
[17:36:11] ====================== [PASSED] args =======================
[17:36:11] =================== xe_pci (2 subtests) ====================
[17:36:11] [PASSED] xe_gmdid_graphics_ip
[17:36:11] [PASSED] xe_gmdid_media_ip
[17:36:11] ===================== [PASSED] xe_pci ======================
[17:36:11] ==================== xe_rtp (1 subtest) ====================
[17:36:11] ================== xe_rtp_process_tests  ===================
[17:36:11] [PASSED] coalesce-same-reg
[17:36:11] [PASSED] no-match-no-add
[17:36:11] [PASSED] match-or
[17:36:11] [PASSED] match-or-xfail
[17:36:11] [PASSED] no-match-no-add-multiple-rules
[17:36:11] [PASSED] two-regs-two-entries
[17:36:11] [PASSED] clr-one-set-other
[17:36:11] [PASSED] set-field
[17:36:11] [PASSED] conflict-duplicate
[17:36:11] [PASSED] conflict-not-disjoint
[17:36:11] [PASSED] conflict-reg-type
stty: 'standard input': Inappropriate ioctl for device
[17:36:11] ============== [PASSED] xe_rtp_process_tests ===============
[17:36:11] ===================== [PASSED] xe_rtp ======================
[17:36:11] ==================== xe_wa (1 subtest) =====================
[17:36:11] ======================== xe_wa_gt  =========================
[17:36:11] [PASSED] TIGERLAKE (B0)
[17:36:11] [PASSED] DG1 (A0)
[17:36:11] [PASSED] DG1 (B0)
[17:36:11] [PASSED] ALDERLAKE_S (A0)
[17:36:11] [PASSED] ALDERLAKE_S (B0)
[17:36:11] [PASSED] ALDERLAKE_S (C0)
[17:36:11] [PASSED] ALDERLAKE_S (D0)
[17:36:11] [PASSED] ALDERLAKE_P (A0)
[17:36:11] [PASSED] ALDERLAKE_P (B0)
[17:36:11] [PASSED] ALDERLAKE_P (C0)
[17:36:11] [PASSED] ALDERLAKE_S_RPLS (D0)
[17:36:11] [PASSED] ALDERLAKE_P_RPLU (E0)
[17:36:11] [PASSED] DG2_G10 (C0)
[17:36:11] [PASSED] DG2_G11 (B1)
[17:36:11] [PASSED] DG2_G12 (A1)
[17:36:11] [PASSED] METEORLAKE (g:A0, m:A0)
[17:36:11] [PASSED] METEORLAKE (g:A0, m:A0)
[17:36:11] [PASSED] METEORLAKE (g:A0, m:A0)
[17:36:11] [PASSED] LUNARLAKE (g:A0, m:A0)
[17:36:11] [PASSED] LUNARLAKE (g:B0, m:A0)
[17:36:11] ==================== [PASSED] xe_wa_gt =====================
[17:36:11] ====================== [PASSED] xe_wa ======================
[17:36:11] ============================================================
[17:36:11] Testing complete. Ran 112 tests: passed: 97, skipped: 15
[17:36:11] Elapsed time: 29.833s total, 4.225s configuring, 25.382s building, 0.182s running

+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/tests/.kunitconfig [17:36:11] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[17:36:13] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
../lib/iomap.c:156:5: warning: no previous prototype for ‘ioread64_lo_hi’ [-Wmissing-prototypes]
  156 | u64 ioread64_lo_hi(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~
../lib/iomap.c:163:5: warning: no previous prototype for ‘ioread64_hi_lo’ [-Wmissing-prototypes]
  163 | u64 ioread64_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~
../lib/iomap.c:170:5: warning: no previous prototype for ‘ioread64be_lo_hi’ [-Wmissing-prototypes]
  170 | u64 ioread64be_lo_hi(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:178:5: warning: no previous prototype for ‘ioread64be_hi_lo’ [-Wmissing-prototypes]
  178 | u64 ioread64be_hi_lo(const void __iomem *addr)
      |     ^~~~~~~~~~~~~~~~
../lib/iomap.c:264:6: warning: no previous prototype for ‘iowrite64_lo_hi’ [-Wmissing-prototypes]
  264 | void iowrite64_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:272:6: warning: no previous prototype for ‘iowrite64_hi_lo’ [-Wmissing-prototypes]
  272 | void iowrite64_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~
../lib/iomap.c:280:6: warning: no previous prototype for ‘iowrite64be_lo_hi’ [-Wmissing-prototypes]
  280 | void iowrite64be_lo_hi(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~
../lib/iomap.c:288:6: warning: no previous prototype for ‘iowrite64be_hi_lo’ [-Wmissing-prototypes]
  288 | void iowrite64be_hi_lo(u64 val, void __iomem *addr)
      |      ^~~~~~~~~~~~~~~~~

[17:36:34] Starting KUnit Kernel (1/1)...
[17:36:34] ============================================================ Running tests with:

$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[17:36:34] ============ drm_test_pick_cmdline (2 subtests) ============
[17:36:34] [PASSED] drm_test_pick_cmdline_res_1920_1080_60
[17:36:34] =============== drm_test_pick_cmdline_named  ===============
[17:36:34] [PASSED] NTSC
[17:36:34] [PASSED] NTSC-J
[17:36:34] [PASSED] PAL
[17:36:34] [PASSED] PAL-M
[17:36:34] =========== [PASSED] drm_test_pick_cmdline_named ===========
[17:36:34] ============== [PASSED] drm_test_pick_cmdline ==============
[17:36:34] ================== drm_buddy (7 subtests) ==================
[17:36:34] [PASSED] drm_test_buddy_alloc_limit
[17:36:34] [PASSED] drm_test_buddy_alloc_optimistic
[17:36:34] [PASSED] drm_test_buddy_alloc_pessimistic
[17:36:34] [PASSED] drm_test_buddy_alloc_pathological
[17:36:34] [PASSED] drm_test_buddy_alloc_contiguous
[17:36:34] [PASSED] drm_test_buddy_alloc_clear
[17:36:34] [PASSED] drm_test_buddy_alloc_range_bias
[17:36:34] ==================== [PASSED] drm_buddy ====================
[17:36:34] ============= drm_cmdline_parser (40 subtests) =============
[17:36:34] [PASSED] drm_test_cmdline_force_d_only
[17:36:34] [PASSED] drm_test_cmdline_force_D_only_dvi
[17:36:34] [PASSED] drm_test_cmdline_force_D_only_hdmi
[17:36:34] [PASSED] drm_test_cmdline_force_D_only_not_digital
[17:36:34] [PASSED] drm_test_cmdline_force_e_only
[17:36:34] [PASSED] drm_test_cmdline_res
[17:36:34] [PASSED] drm_test_cmdline_res_vesa
[17:36:34] [PASSED] drm_test_cmdline_res_vesa_rblank
[17:36:34] [PASSED] drm_test_cmdline_res_rblank
[17:36:34] [PASSED] drm_test_cmdline_res_bpp
[17:36:34] [PASSED] drm_test_cmdline_res_refresh
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_margins
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_force_off
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_analog
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_digital
[17:36:34] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced_margins_force_on
[17:36:34] [PASSED] drm_test_cmdline_res_margins_force_on
[17:36:34] [PASSED] drm_test_cmdline_res_vesa_margins
[17:36:34] [PASSED] drm_test_cmdline_name
[17:36:34] [PASSED] drm_test_cmdline_name_bpp
[17:36:34] [PASSED] drm_test_cmdline_name_option
[17:36:34] [PASSED] drm_test_cmdline_name_bpp_option
[17:36:34] [PASSED] drm_test_cmdline_rotate_0
[17:36:34] [PASSED] drm_test_cmdline_rotate_90
[17:36:34] [PASSED] drm_test_cmdline_rotate_180
[17:36:34] [PASSED] drm_test_cmdline_rotate_270
[17:36:34] [PASSED] drm_test_cmdline_hmirror
[17:36:34] [PASSED] drm_test_cmdline_vmirror
[17:36:34] [PASSED] drm_test_cmdline_margin_options
[17:36:34] [PASSED] drm_test_cmdline_multiple_options
[17:36:34] [PASSED] drm_test_cmdline_bpp_extra_and_option
[17:36:34] [PASSED] drm_test_cmdline_extra_and_option
[17:36:34] [PASSED] drm_test_cmdline_freestanding_options
[17:36:34] [PASSED] drm_test_cmdline_freestanding_force_e_and_options
[17:36:34] [PASSED] drm_test_cmdline_panel_orientation
[17:36:34] ================ drm_test_cmdline_invalid  =================
[17:36:34] [PASSED] margin_only
[17:36:34] [PASSED] interlace_only
[17:36:34] [PASSED] res_missing_x
[17:36:34] [PASSED] res_missing_y
[17:36:34] [PASSED] res_bad_y
[17:36:34] [PASSED] res_missing_y_bpp
[17:36:34] [PASSED] res_bad_bpp
[17:36:34] [PASSED] res_bad_refresh
[17:36:34] [PASSED] res_bpp_refresh_force_on_off
[17:36:34] [PASSED] res_invalid_mode
[17:36:34] [PASSED] res_bpp_wrong_place_mode
[17:36:34] [PASSED] name_bpp_refresh
[17:36:34] [PASSED] name_refresh
[17:36:34] [PASSED] name_refresh_wrong_mode
[17:36:34] [PASSED] name_refresh_invalid_mode
[17:36:34] [PASSED] rotate_multiple
[17:36:34] [PASSED] rotate_invalid_val
[17:36:34] [PASSED] rotate_truncated
[17:36:34] [PASSED] invalid_option
[17:36:34] [PASSED] invalid_tv_option
[17:36:34] [PASSED] truncated_tv_option
[17:36:34] ============ [PASSED] drm_test_cmdline_invalid =============
[17:36:34] =============== drm_test_cmdline_tv_options  ===============
[17:36:34] [PASSED] NTSC
[17:36:34] [PASSED] NTSC_443
[17:36:34] [PASSED] NTSC_J
[17:36:34] [PASSED] PAL
[17:36:34] [PASSED] PAL_M
[17:36:34] [PASSED] PAL_N
[17:36:34] [PASSED] SECAM
[17:36:34] [PASSED] MONO_525
[17:36:34] [PASSED] MONO_625
[17:36:34] =========== [PASSED] drm_test_cmdline_tv_options ===========
[17:36:34] =============== [PASSED] drm_cmdline_parser ================
[17:36:34] ========== drmm_connector_hdmi_init (19 subtests) ==========
[17:36:34] [PASSED] drm_test_connector_hdmi_init_valid
[17:36:34] [PASSED] drm_test_connector_hdmi_init_bpc_8
[17:36:34] [PASSED] drm_test_connector_hdmi_init_bpc_10
[17:36:34] [PASSED] drm_test_connector_hdmi_init_bpc_12
[17:36:34] [PASSED] drm_test_connector_hdmi_init_bpc_invalid
[17:36:34] [PASSED] drm_test_connector_hdmi_init_bpc_null
[17:36:34] [PASSED] drm_test_connector_hdmi_init_formats_empty
[17:36:34] [PASSED] drm_test_connector_hdmi_init_formats_no_rgb
[17:36:34] [PASSED] drm_test_connector_hdmi_init_null_ddc
[17:36:34] [PASSED] drm_test_connector_hdmi_init_null_product
[17:36:34] [PASSED] drm_test_connector_hdmi_init_null_vendor
[17:36:34] [PASSED] drm_test_connector_hdmi_init_product_length_exact
[17:36:34] [PASSED] drm_test_connector_hdmi_init_product_length_too_long
[17:36:34] [PASSED] drm_test_connector_hdmi_init_product_valid
[17:36:34] [PASSED] drm_test_connector_hdmi_init_vendor_length_exact
[17:36:34] [PASSED] drm_test_connector_hdmi_init_vendor_length_too_long
[17:36:34] [PASSED] drm_test_connector_hdmi_init_vendor_valid
[17:36:34] ========= drm_test_connector_hdmi_init_type_valid  =========
[17:36:34] [PASSED] HDMI-A
[17:36:34] [PASSED] HDMI-B
[17:36:34] ===== [PASSED] drm_test_connector_hdmi_init_type_valid =====
[17:36:34] ======== drm_test_connector_hdmi_init_type_invalid  ========
[17:36:34] [PASSED] Unknown
[17:36:34] [PASSED] VGA
[17:36:34] [PASSED] DVI-I
[17:36:34] [PASSED] DVI-D
[17:36:34] [PASSED] DVI-A
[17:36:34] [PASSED] Composite
[17:36:34] [PASSED] SVIDEO
[17:36:34] [PASSED] LVDS
[17:36:34] [PASSED] Component
[17:36:34] [PASSED] DIN
[17:36:34] [PASSED] DP
[17:36:34] [PASSED] TV
[17:36:34] [PASSED] eDP
[17:36:34] [PASSED] Virtual
[17:36:34] [PASSED] DSI
[17:36:34] [PASSED] DPI
[17:36:34] [PASSED] Writeback
[17:36:34] [PASSED] SPI
[17:36:34] [PASSED] USB
[17:36:34] ==== [PASSED] drm_test_connector_hdmi_init_type_invalid ====
[17:36:34] ============ [PASSED] drmm_connector_hdmi_init =============
[17:36:34] ============= drmm_connector_init (3 subtests) =============
[17:36:34] [PASSED] drm_test_drmm_connector_init
[17:36:34] [PASSED] drm_test_drmm_connector_init_null_ddc
[17:36:34] ========= drm_test_drmm_connector_init_type_valid  =========
[17:36:34] [PASSED] Unknown
[17:36:34] [PASSED] VGA
[17:36:34] [PASSED] DVI-I
[17:36:34] [PASSED] DVI-D
[17:36:34] [PASSED] DVI-A
[17:36:34] [PASSED] Composite
[17:36:34] [PASSED] SVIDEO
[17:36:34] [PASSED] LVDS
[17:36:34] [PASSED] Component
[17:36:34] [PASSED] DIN
[17:36:34] [PASSED] DP
[17:36:34] [PASSED] HDMI-A
[17:36:34] [PASSED] HDMI-B
[17:36:34] [PASSED] TV
[17:36:34] [PASSED] eDP
[17:36:34] [PASSED] Virtual
[17:36:34] [PASSED] DSI
[17:36:34] [PASSED] DPI
[17:36:34] [PASSED] Writeback
[17:36:34] [PASSED] SPI
[17:36:34] [PASSED] USB
[17:36:34] ===== [PASSED] drm_test_drmm_connector_init_type_valid =====
[17:36:34] =============== [PASSED] drmm_connector_init ===============
[17:36:34] = drm_connector_attach_broadcast_rgb_property (2 subtests) =
[17:36:34] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property
[17:36:34] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property_hdmi_connector
[17:36:34] === [PASSED] drm_connector_attach_broadcast_rgb_property ===
[17:36:34] ========== drm_get_tv_mode_from_name (2 subtests) ==========
[17:36:34] ========== drm_test_get_tv_mode_from_name_valid  ===========
[17:36:34] [PASSED] NTSC
[17:36:34] [PASSED] NTSC-443
[17:36:34] [PASSED] NTSC-J
[17:36:34] [PASSED] PAL
[17:36:34] [PASSED] PAL-M
[17:36:34] [PASSED] PAL-N
[17:36:34] [PASSED] SECAM
[17:36:34] [PASSED] Mono
[17:36:34] ====== [PASSED] drm_test_get_tv_mode_from_name_valid =======
[17:36:34] [PASSED] drm_test_get_tv_mode_from_name_truncated
[17:36:34] ============ [PASSED] drm_get_tv_mode_from_name ============
[17:36:34] = drm_test_connector_hdmi_compute_mode_clock (12 subtests) =
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc_vic_1
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc_vic_1
[17:36:34] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_double
[17:36:34] = drm_test_connector_hdmi_compute_mode_clock_yuv420_valid  =
[17:36:34] [PASSED] VIC 96
[17:36:34] [PASSED] VIC 97
[17:36:34] [PASSED] VIC 101
[17:36:34] [PASSED] VIC 102
[17:36:34] [PASSED] VIC 106
[17:36:34] [PASSED] VIC 107
[17:36:34] === [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_valid ===
[17:36:34] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_10_bpc
[17:36:34] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_12_bpc
[17:36:34] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_8_bpc
[17:36:34] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_10_bpc
[17:36:34] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_12_bpc
[17:36:34] === [PASSED] drm_test_connector_hdmi_compute_mode_clock ====
[17:36:34] == drm_hdmi_connector_get_broadcast_rgb_name (2 subtests) ==
[17:36:34] === drm_test_drm_hdmi_connector_get_broadcast_rgb_name  ====
[17:36:34] [PASSED] Automatic
[17:36:34] [PASSED] Full
[17:36:34] [PASSED] Limited 16:235
[17:36:34] === [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name ===
[17:36:34] [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name_invalid
[17:36:34] ==== [PASSED] drm_hdmi_connector_get_broadcast_rgb_name ====
[17:36:34] == drm_hdmi_connector_get_output_format_name (2 subtests) ==
[17:36:34] === drm_test_drm_hdmi_connector_get_output_format_name  ====
[17:36:34] [PASSED] RGB
[17:36:34] [PASSED] YUV 4:2:0
[17:36:34] [PASSED] YUV 4:2:2
[17:36:34] [PASSED] YUV 4:4:4
[17:36:34] === [PASSED] drm_test_drm_hdmi_connector_get_output_format_name ===
[17:36:34] [PASSED] drm_test_drm_hdmi_connector_get_output_format_name_invalid
[17:36:34] ==== [PASSED] drm_hdmi_connector_get_output_format_name ====
[17:36:34] ============= drm_damage_helper (21 subtests) ==============
[17:36:34] [PASSED] drm_test_damage_iter_no_damage
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_fractional_src
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_src_moved
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_fractional_src_moved
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_not_visible
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_no_crtc
[17:36:34] [PASSED] drm_test_damage_iter_no_damage_no_fb
[17:36:34] [PASSED] drm_test_damage_iter_simple_damage
[17:36:34] [PASSED] drm_test_damage_iter_single_damage
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_intersect_src
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_outside_src
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_fractional_src
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_intersect_fractional_src
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_outside_fractional_src
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_src_moved
[17:36:34] [PASSED] drm_test_damage_iter_single_damage_fractional_src_moved
[17:36:34] [PASSED] drm_test_damage_iter_damage
[17:36:34] [PASSED] drm_test_damage_iter_damage_one_intersect
[17:36:34] [PASSED] drm_test_damage_iter_damage_one_outside
[17:36:34] [PASSED] drm_test_damage_iter_damage_src_moved
[17:36:34] [PASSED] drm_test_damage_iter_damage_not_visible
[17:36:34] ================ [PASSED] drm_damage_helper ================
[17:36:34] ============== drm_dp_mst_helper (3 subtests) ==============
[17:36:34] ============== drm_test_dp_mst_calc_pbn_mode  ==============
[17:36:34] [PASSED] Clock 154000 BPP 30 DSC disabled
[17:36:34] [PASSED] Clock 234000 BPP 30 DSC disabled
[17:36:34] [PASSED] Clock 297000 BPP 24 DSC disabled
[17:36:34] [PASSED] Clock 332880 BPP 24 DSC enabled
[17:36:34] [PASSED] Clock 324540 BPP 24 DSC enabled
[17:36:34] ========== [PASSED] drm_test_dp_mst_calc_pbn_mode ==========
[17:36:34] ============== drm_test_dp_mst_calc_pbn_div  ===============
[17:36:34] [PASSED] Link rate 2000000 lane count 4
[17:36:34] [PASSED] Link rate 2000000 lane count 2
[17:36:34] [PASSED] Link rate 2000000 lane count 1
[17:36:34] [PASSED] Link rate 1350000 lane count 4
[17:36:34] [PASSED] Link rate 1350000 lane count 2
[17:36:34] [PASSED] Link rate 1350000 lane count 1
[17:36:34] [PASSED] Link rate 1000000 lane count 4
[17:36:34] [PASSED] Link rate 1000000 lane count 2
[17:36:34] [PASSED] Link rate 1000000 lane count 1
[17:36:34] [PASSED] Link rate 810000 lane count 4
[17:36:34] [PASSED] Link rate 810000 lane count 2
[17:36:34] [PASSED] Link rate 810000 lane count 1
[17:36:34] [PASSED] Link rate 540000 lane count 4
[17:36:34] [PASSED] Link rate 540000 lane count 2
[17:36:34] [PASSED] Link rate 540000 lane count 1
[17:36:34] [PASSED] Link rate 270000 lane count 4
[17:36:34] [PASSED] Link rate 270000 lane count 2
[17:36:34] [PASSED] Link rate 270000 lane count 1
[17:36:34] [PASSED] Link rate 162000 lane count 4
[17:36:34] [PASSED] Link rate 162000 lane count 2
[17:36:34] [PASSED] Link rate 162000 lane count 1
[17:36:34] ========== [PASSED] drm_test_dp_mst_calc_pbn_div ===========
[17:36:34] ========= drm_test_dp_mst_sideband_msg_req_decode  =========
[17:36:34] [PASSED] DP_ENUM_PATH_RESOURCES with port number
[17:36:34] [PASSED] DP_POWER_UP_PHY with port number
[17:36:34] [PASSED] DP_POWER_DOWN_PHY with port number
[17:36:34] [PASSED] DP_ALLOCATE_PAYLOAD with SDP stream sinks
[17:36:34] [PASSED] DP_ALLOCATE_PAYLOAD with port number
[17:36:34] [PASSED] DP_ALLOCATE_PAYLOAD with VCPI
[17:36:34] [PASSED] DP_ALLOCATE_PAYLOAD with PBN
[17:36:34] [PASSED] DP_QUERY_PAYLOAD with port number
[17:36:34] [PASSED] DP_QUERY_PAYLOAD with VCPI
[17:36:34] [PASSED] DP_REMOTE_DPCD_READ with port number
[17:36:34] [PASSED] DP_REMOTE_DPCD_READ with DPCD address
[17:36:34] [PASSED] DP_REMOTE_DPCD_READ with max number of bytes
[17:36:34] [PASSED] DP_REMOTE_DPCD_WRITE with port number
[17:36:34] [PASSED] DP_REMOTE_DPCD_WRITE with DPCD address
[17:36:34] [PASSED] DP_REMOTE_DPCD_WRITE with data array
[17:36:34] [PASSED] DP_REMOTE_I2C_READ with port number
[17:36:34] [PASSED] DP_REMOTE_I2C_READ with I2C device ID
[17:36:34] [PASSED] DP_REMOTE_I2C_READ with transactions array
[17:36:34] [PASSED] DP_REMOTE_I2C_WRITE with port number
[17:36:34] [PASSED] DP_REMOTE_I2C_WRITE with I2C device ID
[17:36:34] [PASSED] DP_REMOTE_I2C_WRITE with data array
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream ID
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with client ID
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream event
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with valid stream event
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream behavior
[17:36:34] [PASSED] DP_QUERY_STREAM_ENC_STATUS with a valid stream behavior
[17:36:34] ===== [PASSED] drm_test_dp_mst_sideband_msg_req_decode =====
[17:36:34] ================ [PASSED] drm_dp_mst_helper ================
[17:36:34] ================== drm_exec (7 subtests) ===================
[17:36:34] [PASSED] sanitycheck
[17:36:34] [PASSED] test_lock
[17:36:34] [PASSED] test_lock_unlock
[17:36:34] [PASSED] test_duplicates
[17:36:34] [PASSED] test_prepare
[17:36:34] [PASSED] test_prepare_array
[17:36:34] [PASSED] test_multiple_loops
[17:36:34] ==================== [PASSED] drm_exec =====================
[17:36:34] =========== drm_format_helper_test (17 subtests) ===========
[17:36:34] ============== drm_test_fb_xrgb8888_to_gray8  ==============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ========== [PASSED] drm_test_fb_xrgb8888_to_gray8 ==========
[17:36:34] ============= drm_test_fb_xrgb8888_to_rgb332  ==============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb332 ==========
[17:36:34] ============= drm_test_fb_xrgb8888_to_rgb565  ==============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb565 ==========
[17:36:34] ============ drm_test_fb_xrgb8888_to_xrgb1555  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_xrgb1555 =========
[17:36:34] ============ drm_test_fb_xrgb8888_to_argb1555  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_argb1555 =========
[17:36:34] ============ drm_test_fb_xrgb8888_to_rgba5551  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_rgba5551 =========
[17:36:34] ============= drm_test_fb_xrgb8888_to_rgb888  ==============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb888 ==========
[17:36:34] ============ drm_test_fb_xrgb8888_to_argb8888  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_argb8888 =========
[17:36:34] =========== drm_test_fb_xrgb8888_to_xrgb2101010  ===========
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======= [PASSED] drm_test_fb_xrgb8888_to_xrgb2101010 =======
[17:36:34] =========== drm_test_fb_xrgb8888_to_argb2101010  ===========
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======= [PASSED] drm_test_fb_xrgb8888_to_argb2101010 =======
[17:36:34] ============== drm_test_fb_xrgb8888_to_mono  ===============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ========== [PASSED] drm_test_fb_xrgb8888_to_mono ===========
[17:36:34] ==================== drm_test_fb_swab  =====================
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ================ [PASSED] drm_test_fb_swab =================
[17:36:34] ============ drm_test_fb_xrgb8888_to_xbgr8888  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_xbgr8888 =========
[17:36:34] ============ drm_test_fb_xrgb8888_to_abgr8888  =============
[17:36:34] [PASSED] single_pixel_source_buffer
[17:36:34] [PASSED] single_pixel_clip_rectangle
[17:36:34] [PASSED] well_known_colors
[17:36:34] [PASSED] destination_pitch
[17:36:34] ======== [PASSED] drm_test_fb_xrgb8888_to_abgr8888 =========
[17:36:34] ================= drm_test_fb_clip_offset  =================
[17:36:34] [PASSED] pass through
[17:36:34] [PASSED] horizontal offset
[17:36:34] [PASSED] vertical offset
[17:36:34] [PASSED] horizontal and vertical offset
[17:36:34] [PASSED] horizontal offset (custom pitch)
[17:36:34] [PASSED] vertical offset (custom pitch)
[17:36:34] [PASSED] horizontal and vertical offset (custom pitch)
[17:36:34] ============= [PASSED] drm_test_fb_clip_offset =============
[17:36:34] ============== drm_test_fb_build_fourcc_list  ==============
[17:36:34] [PASSED] no native formats
[17:36:34] [PASSED] XRGB8888 as native format
[17:36:34] [PASSED] remove duplicates
[17:36:34] [PASSED] convert alpha formats
[17:36:34] [PASSED] random formats
[17:36:34] ========== [PASSED] drm_test_fb_build_fourcc_list ==========
[17:36:34] =================== drm_test_fb_memcpy  ====================
[17:36:34] [PASSED] single_pixel_source_buffer: XR24 little-endian (0x34325258)
[17:36:34] [PASSED] single_pixel_source_buffer: XRA8 little-endian (0x38415258)
[17:36:34] [PASSED] single_pixel_source_buffer: YU24 little-endian (0x34325559)
[17:36:34] [PASSED] single_pixel_clip_rectangle: XB24 little-endian (0x34324258)
[17:36:34] [PASSED] single_pixel_clip_rectangle: XRA8 little-endian (0x38415258)
[17:36:34] [PASSED] single_pixel_clip_rectangle: YU24 little-endian (0x34325559)
[17:36:34] [PASSED] well_known_colors: XB24 little-endian (0x34324258)
[17:36:34] [PASSED] well_known_colors: XRA8 little-endian (0x38415258)
[17:36:34] [PASSED] well_known_colors: YU24 little-endian (0x34325559)
[17:36:34] [PASSED] destination_pitch: XB24 little-endian (0x34324258)
[17:36:34] [PASSED] destination_pitch: XRA8 little-endian (0x38415258)
[17:36:34] [PASSED] destination_pitch: YU24 little-endian (0x34325559)
[17:36:34] =============== [PASSED] drm_test_fb_memcpy ================
[17:36:34] ============= [PASSED] drm_format_helper_test ==============
[17:36:34] ================= drm_format (18 subtests) =================
[17:36:34] [PASSED] drm_test_format_block_width_invalid
[17:36:34] [PASSED] drm_test_format_block_width_one_plane
[17:36:34] [PASSED] drm_test_format_block_width_two_plane
[17:36:34] [PASSED] drm_test_format_block_width_three_plane
[17:36:34] [PASSED] drm_test_format_block_width_tiled
[17:36:34] [PASSED] drm_test_format_block_height_invalid
[17:36:34] [PASSED] drm_test_format_block_height_one_plane
[17:36:34] [PASSED] drm_test_format_block_height_two_plane
[17:36:34] [PASSED] drm_test_format_block_height_three_plane
[17:36:34] [PASSED] drm_test_format_block_height_tiled
[17:36:34] [PASSED] drm_test_format_min_pitch_invalid
[17:36:34] [PASSED] drm_test_format_min_pitch_one_plane_8bpp
[17:36:34] [PASSED] drm_test_format_min_pitch_one_plane_16bpp
[17:36:34] [PASSED] drm_test_format_min_pitch_one_plane_24bpp
[17:36:34] [PASSED] drm_test_format_min_pitch_one_plane_32bpp
[17:36:34] [PASSED] drm_test_format_min_pitch_two_plane
[17:36:34] [PASSED] drm_test_format_min_pitch_three_plane_8bpp
[17:36:34] [PASSED] drm_test_format_min_pitch_tiled
[17:36:34] =================== [PASSED] drm_format ====================
[17:36:34] =============== drm_framebuffer (1 subtest) ================
[17:36:34] =============== drm_test_framebuffer_create  ===============
[17:36:34] [PASSED] ABGR8888 normal sizes
[17:36:34] [PASSED] ABGR8888 max sizes
[17:36:34] [PASSED] ABGR8888 pitch greater than min required
[17:36:34] [PASSED] ABGR8888 pitch less than min required
[17:36:34] [PASSED] ABGR8888 Invalid width
[17:36:34] [PASSED] ABGR8888 Invalid buffer handle
[17:36:34] [PASSED] No pixel format
[17:36:34] [PASSED] ABGR8888 Width 0
[17:36:34] [PASSED] ABGR8888 Height 0
[17:36:34] [PASSED] ABGR8888 Out of bound height * pitch combination
[17:36:34] [PASSED] ABGR8888 Large buffer offset
[17:36:34] [PASSED] ABGR8888 Set DRM_MODE_FB_MODIFIERS without modifiers
[17:36:34] [PASSED] ABGR8888 Valid buffer modifier
[17:36:34] [PASSED] ABGR8888 Invalid buffer modifier(DRM_FORMAT_MOD_SAMSUNG_64_32_TILE)
[17:36:34] [PASSED] ABGR8888 Extra pitches without DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] ABGR8888 Extra pitches with DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] NV12 Normal sizes
[17:36:34] [PASSED] NV12 Max sizes
[17:36:34] [PASSED] NV12 Invalid pitch
[17:36:34] [PASSED] NV12 Invalid modifier/missing DRM_MODE_FB_MODIFIERS flag
[17:36:34] [PASSED] NV12 different  modifier per-plane
[17:36:34] [PASSED] NV12 with DRM_FORMAT_MOD_SAMSUNG_64_32_TILE
[17:36:34] [PASSED] NV12 Valid modifiers without DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] NV12 Modifier for inexistent plane
[17:36:34] [PASSED] NV12 Handle for inexistent plane
[17:36:34] [PASSED] NV12 Handle for inexistent plane without DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] YVU420 DRM_MODE_FB_MODIFIERS set without modifier
[17:36:34] [PASSED] YVU420 Normal sizes
[17:36:34] [PASSED] YVU420 Max sizes
[17:36:34] [PASSED] YVU420 Invalid pitch
[17:36:34] [PASSED] YVU420 Different pitches
[17:36:34] [PASSED] YVU420 Different buffer offsets/pitches
[17:36:34] [PASSED] YVU420 Modifier set just for plane 0, without DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] YVU420 Modifier set just for planes 0, 1, without DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] YVU420 Modifier set just for plane 0, 1, with DRM_MODE_FB_MODIFIERS
[17:36:34] [PASSED] YVU420 Valid modifier
[17:36:34] [PASSED] YVU420 Different modifiers per plane
[17:36:34] [PASSED] YVU420 Modifier for inexistent plane
[17:36:34] [PASSED] X0L2 Normal sizes
[17:36:34] [PASSED] X0L2 Max sizes
[17:36:34] [PASSED] X0L2 Invalid pitch
[17:36:34] [PASSED] X0L2 Pitch greater than minimum required
[17:36:34] [PASSED] X0L2 Handle for inexistent plane
[17:36:34] [PASSED] X0L2 Offset for inexistent plane, without DRM_MODE_FB_MODIFIERS set
[17:36:34] [PASSED] X0L2 Modifier without DRM_MODE_FB_MODIFIERS set
[17:36:34] [PASSED] X0L2 Valid modifier
[17:36:34] [PASSED] X0L2 Modifier for inexistent plane
[17:36:34] =========== [PASSED] drm_test_framebuffer_create ===========
[17:36:34] ================= [PASSED] drm_framebuffer =================
[17:36:34] ================ drm_gem_shmem (8 subtests) ================
[17:36:34] [PASSED] drm_gem_shmem_test_obj_create
[17:36:34] [PASSED] drm_gem_shmem_test_obj_create_private
[17:36:34] [PASSED] drm_gem_shmem_test_pin_pages
[17:36:34] [PASSED] drm_gem_shmem_test_vmap
[17:36:34] [PASSED] drm_gem_shmem_test_get_pages_sgt
[17:36:34] [PASSED] drm_gem_shmem_test_get_sg_table
[17:36:34] [PASSED] drm_gem_shmem_test_madvise
[17:36:34] [PASSED] drm_gem_shmem_test_purge
[17:36:34] ================== [PASSED] drm_gem_shmem ==================
[17:36:34] === drm_atomic_helper_connector_hdmi_check (22 subtests) ===
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode_vic_1
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode_vic_1
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode_vic_1
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_changed
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_not_changed
[17:36:34] [PASSED] drm_test_check_hdmi_funcs_reject_rate
[17:36:34] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback
[17:36:34] [PASSED] drm_test_check_max_tmds_rate_format_fallback
[17:36:34] [PASSED] drm_test_check_output_bpc_crtc_mode_changed
[17:36:34] [PASSED] drm_test_check_output_bpc_crtc_mode_not_changed
[17:36:34] [PASSED] drm_test_check_output_bpc_dvi
[17:36:34] [PASSED] drm_test_check_output_bpc_format_vic_1
[17:36:34] [PASSED] drm_test_check_output_bpc_format_display_8bpc_only
[17:36:34] [PASSED] drm_test_check_output_bpc_format_display_rgb_only
[17:36:34] [PASSED] drm_test_check_output_bpc_format_driver_8bpc_only
[17:36:34] [PASSED] drm_test_check_output_bpc_format_driver_rgb_only
[17:36:34] [PASSED] drm_test_check_tmds_char_rate_rgb_8bpc
[17:36:34] [PASSED] drm_test_check_tmds_char_rate_rgb_10bpc
[17:36:34] [PASSED] drm_test_check_tmds_char_rate_rgb_12bpc
[17:36:34] ===== [PASSED] drm_atomic_helper_connector_hdmi_check ======
[17:36:34] === drm_atomic_helper_connector_hdmi_reset (6 subtests) ====
[17:36:34] [PASSED] drm_test_check_broadcast_rgb_value
[17:36:34] [PASSED] drm_test_check_bpc_8_value
[17:36:34] [PASSED] drm_test_check_bpc_10_value
[17:36:34] [PASSED] drm_test_check_bpc_12_value
[17:36:34] [PASSED] drm_test_check_format_value
[17:36:34] [PASSED] drm_test_check_tmds_char_value
[17:36:34] ===== [PASSED] drm_atomic_helper_connector_hdmi_reset ======
[17:36:34] ================= drm_managed (2 subtests) =================
[17:36:34] [PASSED] drm_test_managed_release_action
[17:36:34] [PASSED] drm_test_managed_run_action
[17:36:34] =================== [PASSED] drm_managed ===================
[17:36:34] =================== drm_mm (6 subtests) ====================
[17:36:34] [PASSED] drm_test_mm_init
[17:36:34] [PASSED] drm_test_mm_debug
[17:36:34] [PASSED] drm_test_mm_align32
[17:36:34] [PASSED] drm_test_mm_align64
[17:36:34] [PASSED] drm_test_mm_lowest
[17:36:34] [PASSED] drm_test_mm_highest
[17:36:34] ===================== [PASSED] drm_mm ======================
[17:36:34] ============= drm_modes_analog_tv (5 subtests) =============
[17:36:34] [PASSED] drm_test_modes_analog_tv_mono_576i
[17:36:34] [PASSED] drm_test_modes_analog_tv_ntsc_480i
[17:36:34] [PASSED] drm_test_modes_analog_tv_ntsc_480i_inlined
[17:36:34] [PASSED] drm_test_modes_analog_tv_pal_576i
[17:36:34] [PASSED] drm_test_modes_analog_tv_pal_576i_inlined
[17:36:34] =============== [PASSED] drm_modes_analog_tv ===============
[17:36:34] ============== drm_plane_helper (2 subtests) ===============
[17:36:34] =============== drm_test_check_plane_state  ================
[17:36:34] [PASSED] clipping_simple
[17:36:34] [PASSED] clipping_rotate_reflect
[17:36:34] [PASSED] positioning_simple
[17:36:34] [PASSED] upscaling
[17:36:34] [PASSED] downscaling
[17:36:34] [PASSED] rounding1
[17:36:34] [PASSED] rounding2
[17:36:34] [PASSED] rounding3
[17:36:34] [PASSED] rounding4
[17:36:34] =========== [PASSED] drm_test_check_plane_state ============
[17:36:34] =========== drm_test_check_invalid_plane_state  ============
[17:36:34] [PASSED] positioning_invalid
[17:36:34] [PASSED] upscaling_invalid
stty: 'standard input': Inappropriate ioctl for device
[17:36:34] [PASSED] downscaling_invalid
[17:36:34] ======= [PASSED] drm_test_check_invalid_plane_state ========
[17:36:34] ================ [PASSED] drm_plane_helper =================
[17:36:34] ====== drm_connector_helper_tv_get_modes (1 subtest) =======
[17:36:34] ====== drm_test_connector_helper_tv_get_modes_check  =======
[17:36:34] [PASSED] None
[17:36:34] [PASSED] PAL
[17:36:34] [PASSED] NTSC
[17:36:34] [PASSED] Both, NTSC Default
[17:36:34] [PASSED] Both, PAL Default
[17:36:34] [PASSED] Both, NTSC Default, with PAL on command-line
[17:36:34] [PASSED] Both, PAL Default, with NTSC on command-line
[17:36:34] == [PASSED] drm_test_connector_helper_tv_get_modes_check ===
[17:36:34] ======== [PASSED] drm_connector_helper_tv_get_modes ========
[17:36:34] ================== drm_rect (9 subtests) ===================
[17:36:34] [PASSED] drm_test_rect_clip_scaled_div_by_zero
[17:36:34] [PASSED] drm_test_rect_clip_scaled_not_clipped
[17:36:34] [PASSED] drm_test_rect_clip_scaled_clipped
[17:36:34] [PASSED] drm_test_rect_clip_scaled_signed_vs_unsigned
[17:36:34] ================= drm_test_rect_intersect  =================
[17:36:34] [PASSED] top-left x bottom-right: 2x2+1+1 x 2x2+0+0
[17:36:34] [PASSED] top-right x bottom-left: 2x2+0+0 x 2x2+1-1
[17:36:34] [PASSED] bottom-left x top-right: 2x2+1-1 x 2x2+0+0
[17:36:34] [PASSED] bottom-right x top-left: 2x2+0+0 x 2x2+1+1
[17:36:34] [PASSED] right x left: 2x1+0+0 x 3x1+1+0
[17:36:34] [PASSED] left x right: 3x1+1+0 x 2x1+0+0
[17:36:34] [PASSED] up x bottom: 1x2+0+0 x 1x3+0-1
[17:36:34] [PASSED] bottom x up: 1x3+0-1 x 1x2+0+0
[17:36:34] [PASSED] touching corner: 1x1+0+0 x 2x2+1+1
[17:36:34] [PASSED] touching side: 1x1+0+0 x 1x1+1+0
[17:36:34] [PASSED] equal rects: 2x2+0+0 x 2x2+0+0
[17:36:34] [PASSED] inside another: 2x2+0+0 x 1x1+1+1
[17:36:34] [PASSED] far away: 1x1+0+0 x 1x1+3+6
[17:36:34] [PASSED] points intersecting: 0x0+5+10 x 0x0+5+10
[17:36:34] [PASSED] points not intersecting: 0x0+0+0 x 0x0+5+10
[17:36:34] ============= [PASSED] drm_test_rect_intersect =============
[17:36:34] ================ drm_test_rect_calc_hscale  ================
[17:36:34] [PASSED] normal use
[17:36:34] [PASSED] out of max range
[17:36:34] [PASSED] out of min range
[17:36:34] [PASSED] zero dst
[17:36:34] [PASSED] negative src
[17:36:34] [PASSED] negative dst
[17:36:34] ============ [PASSED] drm_test_rect_calc_hscale ============
[17:36:34] ================ drm_test_rect_calc_vscale  ================
[17:36:34] [PASSED] normal use
[17:36:34] [PASSED] out of max range
[17:36:34] [PASSED] out of min range
[17:36:34] [PASSED] zero dst
[17:36:34] [PASSED] negative src
[17:36:34] [PASSED] negative dst
[17:36:34] ============ [PASSED] drm_test_rect_calc_vscale ============
[17:36:34] ================== drm_test_rect_rotate  ===================
[17:36:34] [PASSED] reflect-x
[17:36:34] [PASSED] reflect-y
[17:36:34] [PASSED] rotate-0
[17:36:34] [PASSED] rotate-90
[17:36:34] [PASSED] rotate-180
[17:36:34] [PASSED] rotate-270
[17:36:34] ============== [PASSED] drm_test_rect_rotate ===============
[17:36:34] ================ drm_test_rect_rotate_inv  =================
[17:36:34] [PASSED] reflect-x
[17:36:34] [PASSED] reflect-y
[17:36:34] [PASSED] rotate-0
[17:36:34] [PASSED] rotate-90
[17:36:34] [PASSED] rotate-180
[17:36:34] [PASSED] rotate-270
[17:36:34] ============ [PASSED] drm_test_rect_rotate_inv =============
[17:36:34] ==================== [PASSED] drm_rect =====================
[17:36:34] ============================================================
[17:36:34] Testing complete. Ran 515 tests: passed: 515
[17:36:34] Elapsed time: 23.139s total, 1.749s configuring, 21.220s building, 0.157s running

+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/ttm/tests/.kunitconfig [17:36:35] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[17:36:36] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
[17:36:45] Starting KUnit Kernel (1/1)...
[17:36:45] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[17:36:45] ================= ttm_device (5 subtests) ==================
[17:36:45] [PASSED] ttm_device_init_basic
[17:36:45] [PASSED] ttm_device_init_multiple
[17:36:45] [PASSED] ttm_device_fini_basic
[17:36:45] [PASSED] ttm_device_init_no_vma_man
[17:36:45] ================== ttm_device_init_pools  ==================
[17:36:45] [PASSED] No DMA allocations, no DMA32 required
[17:36:45] [PASSED] DMA allocations, DMA32 required
[17:36:45] [PASSED] No DMA allocations, DMA32 required
[17:36:45] [PASSED] DMA allocations, no DMA32 required
[17:36:45] ============== [PASSED] ttm_device_init_pools ==============
[17:36:45] =================== [PASSED] ttm_device ====================
[17:36:45] ================== ttm_pool (8 subtests) ===================
[17:36:45] ================== ttm_pool_alloc_basic  ===================
[17:36:45] [PASSED] One page
[17:36:45] [PASSED] More than one page
[17:36:45] [PASSED] Above the allocation limit
[17:36:45] [PASSED] One page, with coherent DMA mappings enabled
[17:36:45] [PASSED] Above the allocation limit, with coherent DMA mappings enabled
[17:36:45] ============== [PASSED] ttm_pool_alloc_basic ===============
[17:36:45] ============== ttm_pool_alloc_basic_dma_addr  ==============
[17:36:45] [PASSED] One page
[17:36:45] [PASSED] More than one page
[17:36:45] [PASSED] Above the allocation limit
[17:36:45] [PASSED] One page, with coherent DMA mappings enabled
[17:36:45] [PASSED] Above the allocation limit, with coherent DMA mappings enabled
[17:36:45] ========== [PASSED] ttm_pool_alloc_basic_dma_addr ==========
[17:36:45] [PASSED] ttm_pool_alloc_order_caching_match
[17:36:45] [PASSED] ttm_pool_alloc_caching_mismatch
[17:36:45] [PASSED] ttm_pool_alloc_order_mismatch
[17:36:45] [PASSED] ttm_pool_free_dma_alloc
[17:36:45] [PASSED] ttm_pool_free_no_dma_alloc
[17:36:45] [PASSED] ttm_pool_fini_basic
[17:36:45] ==================== [PASSED] ttm_pool =====================
[17:36:45] ================ ttm_resource (8 subtests) =================
[17:36:45] ================= ttm_resource_init_basic  =================
[17:36:45] [PASSED] Init resource in TTM_PL_SYSTEM
[17:36:45] [PASSED] Init resource in TTM_PL_VRAM
[17:36:45] [PASSED] Init resource in a private placement
[17:36:45] [PASSED] Init resource in TTM_PL_SYSTEM, set placement flags
[17:36:45] ============= [PASSED] ttm_resource_init_basic =============
[17:36:45] [PASSED] ttm_resource_init_pinned
[17:36:45] [PASSED] ttm_resource_fini_basic
[17:36:45] [PASSED] ttm_resource_manager_init_basic
[17:36:45] [PASSED] ttm_resource_manager_usage_basic
[17:36:45] [PASSED] ttm_resource_manager_set_used_basic
[17:36:45] [PASSED] ttm_sys_man_alloc_basic
[17:36:45] [PASSED] ttm_sys_man_free_basic
[17:36:45] ================== [PASSED] ttm_resource ===================
[17:36:45] =================== ttm_tt (15 subtests) ===================
[17:36:45] ==================== ttm_tt_init_basic  ====================
[17:36:45] [PASSED] Page-aligned size
[17:36:45] [PASSED] Extra pages requested
[17:36:45] ================ [PASSED] ttm_tt_init_basic ================
[17:36:45] [PASSED] ttm_tt_init_misaligned
[17:36:45] [PASSED] ttm_tt_fini_basic
[17:36:45] [PASSED] ttm_tt_fini_sg
[17:36:45] [PASSED] ttm_tt_fini_shmem
[17:36:45] [PASSED] ttm_tt_create_basic
[17:36:45] [PASSED] ttm_tt_create_invalid_bo_type
[17:36:45] [PASSED] ttm_tt_create_ttm_exists
[17:36:45] [PASSED] ttm_tt_create_failed
[17:36:45] [PASSED] ttm_tt_destroy_basic
[17:36:45] [PASSED] ttm_tt_populate_null_ttm
[17:36:45] [PASSED] ttm_tt_populate_populated_ttm
[17:36:45] [PASSED] ttm_tt_unpopulate_basic
[17:36:45] [PASSED] ttm_tt_unpopulate_empty_ttm
[17:36:45] [PASSED] ttm_tt_swapin_basic
[17:36:45] ===================== [PASSED] ttm_tt ======================
[17:36:45] =================== ttm_bo (14 subtests) ===================
[17:36:45] =========== ttm_bo_reserve_optimistic_no_ticket  ===========
[17:36:45] [PASSED] Cannot be interrupted and sleeps
[17:36:45] [PASSED] Cannot be interrupted, locks straight away
[17:36:45] [PASSED] Can be interrupted, sleeps
[17:36:45] ======= [PASSED] ttm_bo_reserve_optimistic_no_ticket =======
[17:36:45] [PASSED] ttm_bo_reserve_locked_no_sleep
[17:36:45] [PASSED] ttm_bo_reserve_no_wait_ticket
[17:36:45] [PASSED] ttm_bo_reserve_double_resv
[17:36:45] [PASSED] ttm_bo_reserve_interrupted
[17:36:45] [PASSED] ttm_bo_reserve_deadlock
[17:36:45] [PASSED] ttm_bo_unreserve_basic
[17:36:45] [PASSED] ttm_bo_unreserve_pinned
[17:36:45] [PASSED] ttm_bo_unreserve_bulk
[17:36:45] [PASSED] ttm_bo_put_basic
[17:36:45] [PASSED] ttm_bo_put_shared_resv
[17:36:45] [PASSED] ttm_bo_pin_basic
[17:36:45] [PASSED] ttm_bo_pin_unpin_resource
[17:36:45] [PASSED] ttm_bo_multiple_pin_one_unpin
[17:36:45] ===================== [PASSED] ttm_bo ======================
[17:36:45] ============== ttm_bo_validate (22 subtests) ===============
[17:36:45] ============== ttm_bo_init_reserved_sys_man  ===============
[17:36:45] [PASSED] Buffer object for userspace
[17:36:45] [PASSED] Kernel buffer object
[17:36:45] [PASSED] Shared buffer object
[17:36:45] ========== [PASSED] ttm_bo_init_reserved_sys_man ===========
[17:36:45] ============== ttm_bo_init_reserved_mock_man  ==============
[17:36:45] [PASSED] Buffer object for userspace
[17:36:45] [PASSED] Kernel buffer object
[17:36:45] [PASSED] Shared buffer object
[17:36:45] ========== [PASSED] ttm_bo_init_reserved_mock_man ==========
[17:36:45] [PASSED] ttm_bo_init_reserved_resv
[17:36:45] ================== ttm_bo_validate_basic  ==================
[17:36:45] [PASSED] Buffer object for userspace
[17:36:45] [PASSED] Kernel buffer object
[17:36:45] [PASSED] Shared buffer object
[17:36:45] ============== [PASSED] ttm_bo_validate_basic ==============
[17:36:45] [PASSED] ttm_bo_validate_invalid_placement
[17:36:45] ============= ttm_bo_validate_same_placement  ==============
[17:36:45] [PASSED] System manager
[17:36:45] [PASSED] VRAM manager
[17:36:45] ========= [PASSED] ttm_bo_validate_same_placement ==========
[17:36:45] [PASSED] ttm_bo_validate_failed_alloc
[17:36:45] [PASSED] ttm_bo_validate_pinned
[17:36:45] [PASSED] ttm_bo_validate_busy_placement
[17:36:45] ================ ttm_bo_validate_multihop  =================
[17:36:45] [PASSED] Buffer object for userspace
[17:36:45] [PASSED] Kernel buffer object
[17:36:45] [PASSED] Shared buffer object
[17:36:45] ============ [PASSED] ttm_bo_validate_multihop =============
[17:36:45] ========== ttm_bo_validate_no_placement_signaled  ==========
[17:36:45] [PASSED] Buffer object in system domain, no page vector
[17:36:45] [PASSED] Buffer object in system domain with an existing page vector
[17:36:45] ====== [PASSED] ttm_bo_validate_no_placement_signaled ======
[17:36:45] ======== ttm_bo_validate_no_placement_not_signaled  ========
[17:36:45] [PASSED] Buffer object for userspace
[17:36:45] [PASSED] Kernel buffer object
[17:36:45] [PASSED] Shared buffer object
[17:36:45] ==== [PASSED] ttm_bo_validate_no_placement_not_signaled ====
[17:36:45] [PASSED] ttm_bo_validate_move_fence_signaled
[17:36:45] ========= ttm_bo_validate_move_fence_not_signaled  =========
[17:36:45] [PASSED] Waits for GPU
[17:36:45] [PASSED] Tries to lock straight away
[17:36:46] ===== [PASSED] ttm_bo_validate_move_fence_not_signaled =====
[17:36:46] [PASSED] ttm_bo_validate_swapout
[17:36:46] [PASSED] ttm_bo_validate_happy_evict
[17:36:46] # ttm_bo_validate_all_pinned_evict: EXPECTATION FAILED at drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c:902
[17:36:46] Expected err == -12, but
[17:36:46]     err == 0 (0x0)
[17:36:46] [FAILED] ttm_bo_validate_all_pinned_evict
[17:36:46] # ttm_bo_validate_allowed_only_evict: EXPECTATION FAILED at drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c:964
stty: 'standard input': Inappropriate ioctl for device
[17:36:46] Expected bo_pinned->resource->mem_type == mem_type, but
[17:36:46]     bo_pinned->resource->mem_type == 0 (0x0)
[17:36:46]     mem_type == 2 (0x2)
[17:36:46] # ttm_bo_validate_allowed_only_evict: EXPECTATION FAILED at drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c:965
[17:36:46] Expected bo_evictable->resource->mem_type == mem_type_evict, but
[17:36:46]     bo_evictable->resource->mem_type == 2 (0x2)
[17:36:46]     mem_type_evict == 0 (0x0)
[17:36:46] [FAILED] ttm_bo_validate_allowed_only_evict
[17:36:46] [PASSED] ttm_bo_validate_deleted_evict
[17:36:46] [PASSED] ttm_bo_validate_busy_domain_evict
[17:36:46] [PASSED] ttm_bo_validate_evict_gutting
[17:36:46] [PASSED] ttm_bo_validate_recrusive_evict
[17:36:46]     # module: ttm_bo_validate_test
[17:36:46] # ttm_bo_validate: pass:20 fail:2 skip:0 total:22
[17:36:46] # Totals: pass:33 fail:2 skip:0 total:35
[17:36:46] ================= [FAILED] ttm_bo_validate =================
[17:36:46] ============================================================
[17:36:46] Testing complete. Ran 102 tests: passed: 100, failed: 2
[17:36:46] Failures: ttm_bo_validate.ttm_bo_validate_all_pinned_evict, ttm_bo_validate.ttm_bo_validate_allowed_only_evict
[17:36:46] Elapsed time: 11.253s total, 1.736s configuring, 8.845s building, 0.581s running

+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel