TTM shrinker helpers and xe buffer object shrinker
Subsystems (4)
Thomas Hellström
maintainerv17 cover TTM shrinker helpers and xe buffer object shrinker
https://lore.kernel.org/linux-mm/[email protected]/T/
Where the comment about layering
https://lore.kernel.org/linux-mm/[email protected]/T/#ma918844aa8a6efe8768fdcda0c6590d5c93850c9
now addressed, and this version also implements shmem objects for backup rather than direct swap-cache insertions, which was used in the previuos series. It turns out that with per-page backup / shrinking, shmem objects appears to work just as well as direct swap-cache insertions with the added benefit that was introduced in the previous TTM shrinker series to avoid running out of swap entries isn't really needed.
The series earlier consisted of a LRU traversal part and the current part. The LRU traversal part is merged, but is still mentioned in the history below.
Patch 1 introduces a backup implemententaion.
Patch 2 introduces functionality in the ttm_pool code for page-by-page shrinking and recovery. It avoids having to temporarily allocate a huge amount of memory to be able to shrink a buffer object. It also introduces the possibility to immediately write-back pages if needed.
Patch 3 Adds a simple error injection to the above code to help increase test coverage.
Patch 4 Implements a macro for LRU iteration.
Patch 5 Introduces driver-facing helpers for shrinking.
Patch 6 Implements the xe bo shrinker.
Patch 7 Increases (removes) the XE_PL_TT watermark.
v2:
- Squash obsolete revision history in the patch commit messages.
- Fix a couple of review comments by Christian
- Don't store the mem_type in the TTM managers but in the resource cursor.
- Rename introduced TTM *back_up* function names to *backup*
- Add ttm pool recovery fault injection.
- Shrinker xe kunit test
- Various bugfixes
v3:
- Address some review comments from Matthew Brost and Christian König.
- Use the restartable LRU walk for TTM swapping and eviction.
- Provide a POC drm_exec locking implementation for exhaustive eviction. (Christian König).
v4:
- Remove the RFC exhaustive eviction part. While the path to exhaustive eviction is pretty clear and demonstrated in v3, there is still some drm_exec work that needs to be agreed and implemented.
- Add shrinker power management. On some hw we need to wake when shrinking.
- Fix the lru walker helper for -EALREADY errors.
- Add drm/xe: Increase the XE_PL_TT watermark.
v5:
- Update also TTM kunit tests
- Handle ghost- and zombie objects in the shrinker.
- A couple of compile- and UAF fixes reported by Kernel Build Robot and Dan Carpenter.
v6:
- Address review comments from Matthew Brost on the restartable LRU traversal path.
v7:
- Split out TTM restartable LRU traversal path and merge that.
- Adapt the review comments on that series.
v8:
- Address review comments from Matthew Brost as detailed in the respective patches.
v9:
- Rebase and fix compilation errors
v10:
- Use a LRU iteration macro rather than a function with a callback.
- Rebasing and cleanups
- Address some additional review comments from Matt Brost.
- Drop the shrinker selftest. It was already merged as a swapout self-test.
v11:
- Move more core interaction to additional TTM helpers.
- Don't back up without __GFP_FS, and don't start writeback without __GFP_IO.
- Rebase.
v12:
- Fix an indentation flaw.
- Rebase
v13:
- Remove the backup base-class, and use direct calls for ttm_backup (Christian König).
- Rebase on the ttm_backup changes.
- Move shrunken bos from the LRU list to the unevictable list.
- Provide an accessor function with sanity checks to set the
- ttm_tt::backup field.
- Update documentation.
v14:
- Update documentation of ttm_backup_bytes_avail().
- Work around converting between struct file * and struct ttm-backup *.
- Don't set up backup for imported buffers.
v15:
- ttm_backup return value change. (Christian König)
- ttm_pool restore/alloc implementation rework (Christian König)
- Documentation update.
- Remove cond_resched (Christian König)
v16:
- Two patches merged. Rebase.
v17:
- Add R-Bs and Acks. Rebase.
Thomas Hellström (7):
drm/ttm: Provide a shmem backup implementation drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages drm/ttm: Use fault-injection to test error paths drm/ttm: Add a macro to perform LRU iteration drm/ttm: Add helpers for shrinking
drm/xe: Add a shrinker for xe bos
drm/xe: Increase the XE_PL_TT watermark
drivers/gpu/drm/ttm/Makefile | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 207 ++++++++++ drivers/gpu/drm/ttm/ttm_bo_util.c | 247 +++++++++++- drivers/gpu/drm/ttm/ttm_pool.c | 577 ++++++++++++++++++++++++--- drivers/gpu/drm/ttm/ttm_tt.c | 83 ++++ drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/tests/xe_bo.c | 6 +- drivers/gpu/drm/xe/xe_bo.c | 202 +++++++++- drivers/gpu/drm/xe/xe_bo.h | 36 ++ drivers/gpu/drm/xe/xe_device.c | 8 + drivers/gpu/drm/xe/xe_device_types.h | 2 + drivers/gpu/drm/xe/xe_shrinker.c | 258 ++++++++++++ drivers/gpu/drm/xe/xe_shrinker.h | 18 + drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +- include/drm/ttm/ttm_backup.h | 74 ++++ include/drm/ttm/ttm_bo.h | 93 +++++ include/drm/ttm/ttm_pool.h | 8 + include/drm/ttm/ttm_tt.h | 69 +++- 18 files changed, 1814 insertions(+), 80 deletions(-) create mode 100644 drivers/gpu/drm/ttm/ttm_backup.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h create mode 100644 include/drm/ttm/ttm_backup.h
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 1/7 drm/ttm: Provide a shmem backup implementation
Provide a standalone shmem backup implementation. Given the ttm_backup interface, this could later on be extended to providing other backup implementation than shmem, with one use-case being GPU swapout to a user-provided fd.
v5:
- Fix a UAF. (kernel test robot, Dan Carptenter)
v6:
- Rename ttm_backup_shmem_copy_page() function argument (Matthew Brost)
- Add some missing documentation
v8:
- Use folio_file_page to get to the page we want to writeback instead of using the first page of the folio.
v13:
- Remove the base class abstraction (Christian König)
- Include ttm_backup_bytes_avail().
v14:
- Fix kerneldoc for ttm_backup_bytes_avail() (0-day)
- Work around casting of __randomize_layout struct pointer (0-day) v15:
- Return negative error code from ttm_backup_backup_page() (Christian König)
- Doc fixes. (Christian König).
--- drivers/gpu/drm/ttm/Makefile | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 207 +++++++++++++++++++++++++++++++ include/drm/ttm/ttm_backup.h | 74 +++++++++++ 3 files changed, 282 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/ttm/ttm_backup.c create mode 100644 include/drm/ttm/ttm_backup.h
drivers/gpu/drm/ttm/Makefile +1 -1
@@ -4,7 +4,7 @@ ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \ ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \ - ttm_device.o ttm_sys_manager.o + ttm_device.o ttm_sys_manager.o ttm_backup.o ttm-$(CONFIG_AGP) += ttm_agp_backend.o obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_backup.c new file +207 -0
207 lines not on this page · raw message
include/drm/ttm/ttm_backup.h new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 2/7 drm/ttm/pool, drm/ttm/tt: Provide a helper to shrink pages
In reply to Thomas HellströmProvide a helper to shrink ttm_tt page-vectors on a per-page basis. A ttm_backup backend could then in theory get away with allocating a single temporary page for each struct ttm_tt.
This is accomplished by splitting larger pages before trying to back them up.
In the future we could allow ttm_backup to handle backing up large pages as well, but currently there's no benefit in doing that, since the shmem backup backend would have to split those anyway to avoid allocating too much temporary memory, and if the backend instead inserts pages into the swap-cache, those are split on reclaim by the core.
Due to potential backup- and recover errors, allow partially swapped out struct ttm_tt's, although mark them as swapped out stopping them from being swapped out a second time. More details in the ttm_pool.c DOC section.
v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface. v8:
- Use a struct for flags for readability (Matt Brost)
- Address misc other review comments (Matt Brost) v9:
- Update the kerneldoc for the ttm_tt::backup field. v10:
- Rebase.
v13:
- Rebase on ttm_backup interface change. Update kerneldoc.
- Rebase and adjust ttm_tt_is_swapped().
v15:
- Rebase on ttm_backup return value change.
- Rebase on previous restructuring of ttm_pool_alloc()
- Rework the ttm_pool backup interface (Christian König)
- Remove cond_resched() (Christian König)
- Get rid of the need to allocate an intermediate page array when restoring a multi-order page (Christian König)
- Update documentation.
--- drivers/gpu/drm/ttm/ttm_pool.c | 554 +++++++++++++++++++++++++++++---- drivers/gpu/drm/ttm/ttm_tt.c | 54 ++++ include/drm/ttm/ttm_pool.h | 8 + include/drm/ttm/ttm_tt.h | 67 +++- 4 files changed, 629 insertions(+), 54 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +502 -52
@@ -41,6 +41,7 @@#include <asm/set_memory.h> #endif +#include <drm/ttm/ttm_backup.h> #include <drm/ttm/ttm_pool.h> #include <drm/ttm/ttm_tt.h> #include <drm/ttm/ttm_bo.h>
@@ -75,6 +76,35 @@ struct ttm_pool_alloc_state {enum ttm_caching tt_caching; }; +/** +* struct ttm_pool_tt_restore - State representing restore from backup +* @pool: The pool used for page allocation while restoring. +* @snapshot_alloc: A snapshot of the most recent struct ttm_pool_alloc_state. +* @alloced_page: Pointer to the page most recently allocated from a pool or system. +* @first_dma: The dma address corresponding to @alloced_page if dma_mapping +* is requested. +* @alloced_pages: The number of allocated pages present in the struct ttm_tt +* page vector from this restore session. +* @restored_pages: The number of 4K pages restored for @alloced_page (which +* is typically a multi-order page). +* @page_caching: The struct ttm_tt requested caching +* @order: The order of @alloced_page. +* +* Recovery from backup might fail when we've recovered less than the +* full ttm_tt. In order not to loose any data (yet), keep information +* around that allows us to restart a failed ttm backup recovery. +*/ +struct ttm_pool_tt_restore { +struct ttm_pool *pool; +struct ttm_pool_alloc_state snapshot_alloc; +struct page *alloced_page; +dma_addr_t first_dma; +pgoff_t alloced_pages; +pgoff_t restored_pages; +enum ttm_caching page_caching; +unsigned int order; +}; +static unsigned long page_pool_size;MODULE_PARM_DESC (page_pool_size,"Number of pages in the WC/UC/DMA pool" );
@@ -199,12 +229,11 @@ static int ttm_pool_apply_caching(struct ttm_pool_alloc_state *alloc)return 0 ; } -/* Map pages of 1 << order size and fill the DMA address array */ +/* DMA Map pages of 1 << order size and return the resulting dma_address. */ static int ttm_pool_map (struct ttm_pool *pool,unsigned int order, -struct page *p,dma_addr_t **dma_addr) +struct page *p,dma_addr_t *dma_addr) {dma_addr_t addr; -unsigned int i;if (pool->use_dma_alloc) {struct ttm_pool_dma *dma = (void *)p->private;
@@ -218,10 +247,7 @@ static int ttm_pool_map(struct ttm_pool *pool, unsigned int order,return -EFAULT ; } -for (i =1 << order; i ; --i) { - *(*dma_addr)++ = addr; - addr +=PAGE_SIZE ; - } + *dma_addr = addr;return 0 ; }
@@ -371,6 +397,190 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)return p->private; } +/* +* Split larger pages so that we can free each PAGE_SIZE page as soon +* as it has been backed up, in order to avoid memory pressure during +* reclaim. +*/ +static void ttm_pool_split_for_swap (struct ttm_pool *pool,struct page *p) +{ +unsigned int order =ttm_pool_page_order (pool, p); +pgoff_t nr; + +if (!order) +return ; + +split_page (p, order); + nr =1UL << order; +while (nr--) + (p++)->private =0 ; +} + +/** +* DOC: Partial backup and restoration of a struct ttm_tt. +* +* Swapout using ttm_backup_backup_page() and swapin using +* ttm_backup_copy_page() may fail. +* The former most likely due to lack of swap-space or memory, the latter due +* to lack of memory or because of signal interruption during waits. +* +* Backup failure is easily handled by using a ttm_tt pages vector that holds +* both backup handles and page pointers. This has to be taken into account when +* restoring such a ttm_tt from backup, and when freeing it while backed up. +* When restoring, for simplicity, new pages are actually allocated from the +* pool and the contents of any old pages are copied in and then the old pages +* are released. +* +* For restoration failures, the struct ttm_pool_tt_restore holds sufficient state +* to be able to resume an interrupted restore, and that structure is freed once +* the restoration is complete. If the struct ttm_tt is destroyed while there +* is a valid struct ttm_pool_tt_restore attached, that is also properly taken +* care of. +*/ + +/* Is restore ongoing for the currently allocated page? */ +static bool ttm_pool_restore_valid (const struct ttm_pool_tt_restore *restore) +{ +return restore && restore->restored_pages < (1 << restore->order); +} + +/* DMA unmap and free a multi-order page, either to the relevant pool or to system. */ +static pgoff_t ttm_pool_unmap_and_free (struct ttm_pool *pool,struct page *page, +const dma_addr_t *dma_addr,enum ttm_caching caching) +{ +struct ttm_pool_type *pt =NULL ; +unsigned int order; +pgoff_t nr; + +if (pool) { + order =ttm_pool_page_order (pool, page); + nr = (1UL << order); +if (dma_addr) +ttm_pool_unmap (pool, *dma_addr, nr); + + pt =ttm_pool_select_type (pool, caching, order); + }else { + order = page->private; + nr = (1UL << order); + } + +if (pt) +ttm_pool_type_give (pt, page); +else +ttm_pool_free_page (pool, caching, order, page); + +return nr; +} + +/* Populate the page-array using the most recent allocated multi-order page. */ +static void ttm_pool_allocated_page_commit (struct page *allocated, +dma_addr_t first_dma, +struct ttm_pool_alloc_state *alloc, +pgoff_t nr) +{ +pgoff_t i; + +for (i =0 ; i < nr; ++i) + *alloc->pages++ = allocated++; + + alloc->remaining_pages -= nr; + +if (!alloc->dma_addr) +return ; + +for (i =0 ; i < nr; ++i) { + *alloc->dma_addr++ = first_dma; + first_dma +=PAGE_SIZE ; + } +} + +/* +* When restoring, restore backed-up content to the newly allocated page and +* if successful, populate the page-table and dma-address arrays. +*/ +static int ttm_pool_restore_commit (struct ttm_pool_tt_restore *restore, +struct ttm_backup *backup, +const struct ttm_operation_ctx *ctx, +struct ttm_pool_alloc_state *alloc) + +{ +pgoff_t i, nr =1UL << restore->order; +struct page **first_page = alloc->pages; +struct page *p; +int ret =0 ; + +for (i = restore->restored_pages; i < nr; ++i) { + p = first_page[i]; +if (ttm_backup_page_ptr_is_handle (p)) { +unsigned long handle =ttm_backup_page_ptr_to_handle (p); + +if (handle ==0 ) { + restore->restored_pages++; +continue ; + } + + ret =ttm_backup_copy_page (backup, restore->alloced_page + i, + handle, ctx->interruptible); +if (ret) +break ; + +ttm_backup_drop (backup, handle); + }else if (p) { +/* +* We could probably avoid splitting the old page +* using clever logic, but ATM we don't care, as +* we prioritize releasing memory ASAP. Note that +* here, the old retained page is always write-back +* cached. +*/ +ttm_pool_split_for_swap (restore->pool, p); +copy_highpage (restore->alloced_page + i, p); +__free_pages (p,0 ); + } + + restore->restored_pages++; + first_page[i] =ttm_backup_handle_to_page_ptr (0 ); + } + +if (ret) { +if (!restore->restored_pages) { +dma_addr_t *dma_addr = alloc->dma_addr ? &restore->first_dma :NULL ; + +ttm_pool_unmap_and_free (restore->pool, restore->alloced_page, + dma_addr, restore->page_caching); + restore->restored_pages = nr; + } +return ret; + } + +ttm_pool_allocated_page_commit (restore->alloced_page, restore->first_dma, + alloc, nr); +if (restore->page_caching == alloc->tt_caching ||PageHighMem (restore->alloced_page)) + alloc->caching_divide = alloc->pages; + restore->snapshot_alloc = *alloc; + restore->alloced_pages += nr; + +return 0 ; +} + +/* If restoring, save information needed for ttm_pool_restore_commit(). */ +static void +ttm_pool_page_allocated_restore (struct ttm_pool *pool,unsigned int order, +struct page *p, +enum ttm_caching page_caching, +dma_addr_t first_dma, +struct ttm_pool_tt_restore *restore, +const struct ttm_pool_alloc_state *alloc) +{ + restore->pool = pool; + restore->order = order; + restore->restored_pages =0 ; + restore->page_caching = page_caching; + restore->first_dma = first_dma; + restore->alloced_page = p; + restore->snapshot_alloc = *alloc; +} +/* * Called when we got a page, either from a pool or newly allocated. * if needed, dma map the page and populate the dma address array.
@@ -380,10 +590,11 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)*/ static int ttm_pool_page_allocated (struct ttm_pool *pool,unsigned int order,struct page *p,enum ttm_caching page_caching, -struct ttm_pool_alloc_state *alloc) +struct ttm_pool_alloc_state *alloc, +struct ttm_pool_tt_restore *restore) { -pgoff_t i, nr =1UL << order;bool caching_consistent; +dma_addr_t first_dma;int r =0 ; caching_consistent = (page_caching == alloc->tt_caching) ||PageHighMem (p);
@@ -395,17 +606,20 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order, }if (alloc->dma_addr) { - r =ttm_pool_map (pool, order, p, &alloc->dma_addr); + r =ttm_pool_map (pool, order, p, &first_dma);if (r)return r; } - alloc->remaining_pages -= nr; -for (i =0 ; i < nr; ++i) - *alloc->pages++ = p++; +if (restore) { +ttm_pool_page_allocated_restore (pool, order, p, page_caching, + first_dma, restore, alloc); + }else { +ttm_pool_allocated_page_commit (p, first_dma, alloc,1UL << order); -if (caching_consistent) - alloc->caching_divide = alloc->pages; +if (caching_consistent) + alloc->caching_divide = alloc->pages; + }return 0 ; }
@@ -428,22 +642,24 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt, pgoff_t start_page, pgoff_t end_page) {struct page **pages = &tt->pages[start_page]; -unsigned int order; +struct ttm_backup *backup = tt->backup;pgoff_t i, nr;for (i = start_page; i < end_page; i += nr, pages += nr) { -struct ttm_pool_type *pt =NULL ; +struct page *p = *pages; - order =ttm_pool_page_order (pool, *pages); - nr = (1UL << order); -if (tt->dma_address) -ttm_pool_unmap (pool, tt->dma_address[i], nr); + nr =1 ; +if (ttm_backup_page_ptr_is_handle (p)) { +unsigned long handle =ttm_backup_page_ptr_to_handle (p); - pt =ttm_pool_select_type (pool, caching, order); -if (pt) -ttm_pool_type_give (pt, *pages); -else -ttm_pool_free_page (pool, caching, order, *pages); +if (handle !=0 ) +ttm_backup_drop (backup, handle); + }else if (p) { +dma_addr_t *dma_addr = tt->dma_address ? + tt->dma_address + i :NULL ; + + nr =ttm_pool_unmap_and_free (pool, p, dma_addr, caching); + } } }
@@ -467,22 +683,11 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest, return min_t(unsigned int, highest, __fls(alloc->remaining_pages)); } -/** -* ttm_pool_alloc - Fill a ttm_tt object -* -* @pool: ttm_pool to use -* @tt: ttm_tt object to fill -* @ctx: operation context -* -* Fill the ttm_tt object with pages and also make sure to DMA map them when -* necessary. -* -* Returns: 0 on successe, negative error code otherwise. -*/ -int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, -struct ttm_operation_ctx *ctx) +static int __ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +const struct ttm_operation_ctx *ctx, +struct ttm_pool_alloc_state *alloc, +struct ttm_pool_tt_restore *restore) { -struct ttm_pool_alloc_state alloc;enum ttm_caching page_caching;gfp_t gfp_flags =GFP_USER ;pgoff_t caching_divide;
@@ -491,10 +696,8 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,struct page *p;int r; -ttm_pool_alloc_state_init (tt, &alloc); - -WARN_ON (!alloc.remaining_pages ||ttm_tt_is_populated (tt)); -WARN_ON (alloc.dma_addr && !pool->dev); +WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt)); +WARN_ON (alloc->dma_addr && !pool->dev);if (tt->page_flags &TTM_TT_FLAG_ZERO_ALLOC ) gfp_flags |= __GFP_ZERO;
@@ -509,9 +712,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, page_caching = tt->caching; allow_pools = true; -for (order =ttm_pool_alloc_find_order (MAX_PAGE_ORDER , &alloc); - alloc.remaining_pages; - order =ttm_pool_alloc_find_order (order, &alloc)) { +for (order =ttm_pool_alloc_find_order (MAX_PAGE_ORDER , alloc); + alloc->remaining_pages; + order =ttm_pool_alloc_find_order (order, alloc)) {struct ttm_pool_type *pt;/* First, try to allocate a page from a pool if one exists. */
@@ -541,30 +744,120 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, r = -ENOMEM ; goto error_free_all; } - r =ttm_pool_page_allocated (pool, order, p, page_caching, &alloc); + r =ttm_pool_page_allocated (pool, order, p, page_caching, alloc, + restore);if (r) goto error_free_page; + +if (ttm_pool_restore_valid (restore)) { + r =ttm_pool_restore_commit (restore, tt->backup, ctx, alloc); +if (r) + goto error_free_all; + } } - r =ttm_pool_apply_caching (&alloc); + r =ttm_pool_apply_caching (alloc);if (r) goto error_free_all; +kfree (tt->restore); + tt->restore =NULL ; +return 0 ; error_free_page:ttm_pool_free_page (pool, page_caching, order, p); error_free_all: - caching_divide = alloc.caching_divide - tt->pages; +if (tt->restore) +return r; + + caching_divide = alloc->caching_divide - tt->pages;ttm_pool_free_range (pool, tt, tt->caching,0 , caching_divide);ttm_pool_free_range (pool, tt, ttm_cached, caching_divide, - tt->num_pages - alloc.remaining_pages); + tt->num_pages - alloc->remaining_pages);return r; } + +/** +* ttm_pool_alloc - Fill a ttm_tt object +* +* @pool: ttm_pool to use +* @tt: ttm_tt object to fill +* @ctx: operation context +* +* Fill the ttm_tt object with pages and also make sure to DMA map them when +* necessary. +* +* Returns: 0 on successe, negative error code otherwise. +*/ +int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +struct ttm_operation_ctx *ctx) +{ +struct ttm_pool_alloc_state alloc; + +if (WARN_ON (ttm_tt_is_backed_up (tt))) +return -EINVAL ; + +ttm_pool_alloc_state_init (tt, &alloc); + +return __ttm_pool_alloc (pool, tt, ctx, &alloc,NULL ); +}EXPORT_SYMBOL (ttm_pool_alloc); +/** +* ttm_pool_restore_and_alloc - Fill a ttm_tt, restoring previously backed-up +* content. +* +* @pool: ttm_pool to use +* @tt: ttm_tt object to fill +* @ctx: operation context +* +* Fill the ttm_tt object with pages and also make sure to DMA map them when +* necessary. Read in backed-up content. +* +* Returns: 0 on successe, negative error code otherwise. +*/ +int ttm_pool_restore_and_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +const struct ttm_operation_ctx *ctx) +{ +struct ttm_pool_alloc_state alloc; + +if (WARN_ON (!ttm_tt_is_backed_up (tt))) +return -EINVAL ; + +if (!tt->restore) { +gfp_t gfp =GFP_KERNEL | __GFP_NOWARN; + +ttm_pool_alloc_state_init (tt, &alloc); +if (ctx->gfp_retry_mayfail) + gfp |= __GFP_RETRY_MAYFAIL; + + tt->restore =kzalloc (sizeof (*tt->restore), gfp); +if (!tt->restore) +return -ENOMEM ; + + tt->restore->snapshot_alloc = alloc; + tt->restore->pool = pool; + tt->restore->restored_pages =1 ; + }else { +struct ttm_pool_tt_restore *restore = tt->restore; +int ret; + + alloc = restore->snapshot_alloc; +if (ttm_pool_restore_valid (tt->restore)) { + ret =ttm_pool_restore_commit (restore, tt->backup, ctx, &alloc); +if (ret) +return ret; + } +if (!alloc.remaining_pages) +return 0 ; + } + +return __ttm_pool_alloc (pool, tt, ctx, &alloc, tt->restore); +} +/** * ttm_pool_free - Free the backing pages from a ttm_tt object *
@@ -582,6 +875,163 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt) }EXPORT_SYMBOL (ttm_pool_free); +/** +* ttm_pool_drop_backed_up() - Release content of a swapped-out struct ttm_tt +* @tt: The struct ttm_tt. +* +* Release handles with associated content or any remaining pages of +* a backed-up struct ttm_tt. +*/ +void ttm_pool_drop_backed_up (struct ttm_tt *tt) +{ +struct ttm_pool_tt_restore *restore; +pgoff_t start_page =0 ; + +WARN_ON (!ttm_tt_is_backed_up (tt)); + + restore = tt->restore; + +/* +* Unmap and free any uncommitted restore page. +* any tt page-array backup entries already read back has +* been cleared already +*/ +if (ttm_pool_restore_valid (restore)) { +dma_addr_t *dma_addr = tt->dma_address ? &restore->first_dma :NULL ; + +ttm_pool_unmap_and_free (restore->pool, restore->alloced_page, + dma_addr, restore->page_caching); + restore->restored_pages =1UL << restore->order; + } + +/* +* If a restore is ongoing, part of the tt pages may have a +* caching different than writeback. +*/ +if (restore) { +pgoff_t mid = restore->snapshot_alloc.caching_divide - tt->pages; + + start_page = restore->alloced_pages; +WARN_ON (mid > start_page); +/* Pages that might be dma-mapped and non-cached */ +ttm_pool_free_range (restore->pool, tt, tt->caching, +0 , mid); +/* Pages that might be dma-mapped but cached */ +ttm_pool_free_range (restore->pool, tt, ttm_cached, + mid, restore->alloced_pages); +kfree (restore); + tt->restore =NULL ; + } + +ttm_pool_free_range (NULL , tt, ttm_cached, start_page, tt->num_pages); +} + +/** +* ttm_pool_backup() - Back up or purge a struct ttm_tt +* @pool: The pool used when allocating the struct ttm_tt. +* @tt: The struct ttm_tt. +* @flags: Flags to govern the backup behaviour. +* +* Back up or purge a struct ttm_tt. If @purge is true, then +* all pages will be freed directly to the system rather than to the pool +* they were allocated from, making the function behave similarly to +* ttm_pool_free(). If @purge is false the pages will be backed up instead, +* exchanged for handles. +* A subsequent call to ttm_pool_restore_and_alloc() will then read back the content and +* a subsequent call to ttm_pool_drop_backed_up() will drop it. +* If backup of a page fails for whatever reason, @ttm will still be +* partially backed up, retaining those pages for which backup fails. +* In that case, this function can be retried, possibly after freeing up +* memory resources. +* +* Return: Number of pages actually backed up or freed, or negative +* error code on error. +*/ +long ttm_pool_backup (struct ttm_pool *pool,struct ttm_tt *tt, +const struct ttm_backup_flags *flags) +{ +struct ttm_backup *backup = tt->backup; +struct page *page; +unsigned long handle; +gfp_t alloc_gfp; +gfp_t gfp; +int ret =0 ; +pgoff_t shrunken =0 ; +pgoff_t i, num_pages; + +if (WARN_ON (ttm_tt_is_backed_up (tt))) +return -EINVAL ; + +if ((!ttm_backup_bytes_avail () && !flags->purge) || + pool->use_dma_alloc ||ttm_tt_is_backed_up (tt)) +return -EBUSY ; + +#ifdef CONFIG_X86 +/* Anything returned to the system needs to be cached. */ +if (tt->caching != ttm_cached) +set_pages_array_wb (tt->pages, tt->num_pages); +#endif + +if (tt->dma_address || flags->purge) { +for (i =0 ; i < tt->num_pages; i += num_pages) { +unsigned int order; + + page = tt->pages[i]; +if (unlikely (!page)) { + num_pages =1 ; +continue ; + } + + order =ttm_pool_page_order (pool, page); + num_pages =1UL << order; +if (tt->dma_address) +ttm_pool_unmap (pool, tt->dma_address[i], + num_pages); +if (flags->purge) { + shrunken += num_pages; + page->private =0 ; +__free_pages (page, order); +memset (tt->pages + i,0 , + num_pages *sizeof (*tt->pages)); + } + } + } + +if (flags->purge) +return shrunken; + +if (pool->use_dma32) + gfp =GFP_DMA32 ; +else + gfp =GFP_HIGHUSER ; + + alloc_gfp =GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL; + +for (i =0 ; i < tt->num_pages; ++i) { +s64 shandle; + + page = tt->pages[i]; +if (unlikely (!page)) +continue ; + +ttm_pool_split_for_swap (pool, page); + + shandle =ttm_backup_backup_page (backup, page, flags->writeback, i, + gfp, alloc_gfp); +if (shandle <0 ) { +/* We allow partially shrunken tts */ + ret = shandle; +break ; + } + handle = shandle; + tt->pages[i] =ttm_backup_handle_to_page_ptr (handle); +put_page (page); + shrunken++; + } + +return shrunken ? shrunken : ret; +} +/** * ttm_pool_init - Initialize a pool *
drivers/gpu/drm/ttm/ttm_tt.c +54 -0
84 lines not on this page · raw message
include/drm/ttm/ttm_pool.h +8 -0
@@ -33,6 +33,7 @@struct device ;struct seq_file ; +struct ttm_backup_flags ;struct ttm_operation_ctx ;struct ttm_pool ;struct ttm_tt ;
@@ -89,6 +90,13 @@ void ttm_pool_fini(struct ttm_pool *pool);int ttm_pool_debugfs (struct ttm_pool *pool,struct seq_file *m); +void ttm_pool_drop_backed_up (struct ttm_tt *tt); + +long ttm_pool_backup (struct ttm_pool *pool,struct ttm_tt *ttm, +const struct ttm_backup_flags *flags); +int ttm_pool_restore_and_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +const struct ttm_operation_ctx *ctx); +int ttm_pool_mgr_init (unsigned long num_pages);void ttm_pool_mgr_fini (void );
include/drm/ttm/ttm_tt.h +65 -2
119 lines not on this page · raw message
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 3/7 drm/ttm: Use fault-injection to test error paths
In reply to Thomas HellströmUse fault-injection to test partial TTM swapout and interrupted swapin. Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that the swapin and release_shrunken functionality.
v8:
- Use the core fault-injection system.
v9:
- Fix compliation failure for !CONFIG_FAULT_INJECTION
--- drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -1
@@ -48,6 +48,13 @@#include "ttm_module.h" +#ifdef CONFIG_FAULT_INJECTION +#include <linux/fault-inject.h> +static DECLARE_FAULT_ATTR (backup_fault_inject); +#else +#define should_fail (...) false +#endif +/** * struct ttm_pool_dma - Helper object for coherent DMA mappings *
@@ -514,6 +521,12 @@ static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,if (ttm_backup_page_ptr_is_handle (p)) {unsigned long handle =ttm_backup_page_ptr_to_handle (p); +if (IS_ENABLED (CONFIG_FAULT_INJECTION ) && ctx->interruptible && +should_fail (&backup_fault_inject,1 )) { + ret = -EINTR ; +break ; + } +if (handle ==0 ) { restore->restored_pages++;continue ;
@@ -1007,7 +1020,13 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt, alloc_gfp =GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL; -for (i =0 ; i < tt->num_pages; ++i) { + num_pages = tt->num_pages; + +/* Pretend doing fault injection by shrinking only half of the pages. */ +if (IS_ENABLED (CONFIG_FAULT_INJECTION ) &&should_fail (&backup_fault_inject,1 )) + num_pages =DIV_ROUND_UP (num_pages,2 ); + +for (i =0 ; i < num_pages; ++i) {s64 shandle; page = tt->pages[i];
@@ -1293,6 +1312,10 @@ int ttm_pool_mgr_init(unsigned long num_pages) &ttm_pool_debugfs_globals_fops);debugfs_create_file ("page_pool_shrink" ,0400 , ttm_debugfs_root,NULL , &ttm_pool_debugfs_shrink_fops); +#ifdef CONFIG_FAULT_INJECTION +fault_create_debugfs_attr ("backup_fault_inject" , ttm_debugfs_root, + &backup_fault_inject); +#endif #endif mm_shrinker =shrinker_alloc (0 ,"drm-ttm_pool" );
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 4/7 drm/ttm: Add a macro to perform LRU iteration
In reply to Thomas HellströmFollowing the design direction communicated here:
https://lore.kernel.org/linux-mm/[email protected]/T/#ma918844aa8a6efe8768fdcda0c6590d5c93850c9
Export a LRU walker for driver shrinker use. The walker initially supports only trylocking, since that's the method used by shrinkes. The walker makes use of scoped_guard() to allow exiting from the LRU walk loop without performing any explicit unlocking or cleanup.
v8:
- Split out from another patch.
- Use a struct for bool arguments to increase readability (Matt Brost).
- Unmap user-space cpu-mappings before shrinking pages.
- Explain non-fatal error codes (Matt Brost)
v10:
- Instead of using the existing helper, Wrap the interface inside out and provide a loop to de-midlayer things the LRU iteration (Christian König).
- Removing the R-B by Matt Brost since the patch was significantly changed.
v11:
- Split the patch up to include just the LRU walk helper.
v12:
- Indent after scoped_guard() (Matt Brost)
v15:
- Adapt to new definition of scoped_guard()
--- drivers/gpu/drm/ttm/ttm_bo_util.c | 140 +++++++++++++++++++++++++++++- include/drm/ttm/ttm_bo.h | 72 +++++++++++++++ 2 files changed, 208 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_bo_util.c +136 -4
@@ -769,12 +769,10 @@ int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)return ret; } -static bool ttm_lru_walk_trylock (struct ttm_lru_walk *walk, +static bool ttm_lru_walk_trylock (struct ttm_operation_ctx *ctx,struct ttm_buffer_object *bo,bool *needs_unlock) { -struct ttm_operation_ctx *ctx = walk->ctx; - *needs_unlock = false;if (dma_resv_trylock (bo->base.resv)) {
@@ -877,7 +875,7 @@ s64 ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,* since if we do it the other way around, and the trylock fails, * we need to drop the lru lock to put the bo. */ -if (ttm_lru_walk_trylock (walk, bo, &bo_needs_unlock)) +if (ttm_lru_walk_trylock (walk->ctx, bo, &bo_needs_unlock)) bo_locked = true;else if (!walk->ticket || walk->ctx->no_wait_gpu || walk->trylock_only)
@@ -920,3 +918,137 @@ s64 ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,return progress; } +EXPORT_SYMBOL (ttm_lru_walk_for_evict); + +static void ttm_bo_lru_cursor_cleanup_bo (struct ttm_bo_lru_cursor *curs) +{ +struct ttm_buffer_object *bo = curs->bo; + +if (bo) { +if (curs->needs_unlock) +dma_resv_unlock (bo->base.resv); +ttm_bo_put (bo); + curs->bo =NULL ; + } +} + +/** +* ttm_bo_lru_cursor_fini() - Stop using a struct ttm_bo_lru_cursor +* and clean up any iteration it was used for. +* @curs: The cursor. +*/ +void ttm_bo_lru_cursor_fini (struct ttm_bo_lru_cursor *curs) +{ +spinlock_t *lru_lock = &curs->res_curs.man->bdev->lru_lock; + +ttm_bo_lru_cursor_cleanup_bo (curs); +spin_lock (lru_lock); +ttm_resource_cursor_fini (&curs->res_curs); +spin_unlock (lru_lock); +} +EXPORT_SYMBOL (ttm_bo_lru_cursor_fini); + +/** +* ttm_bo_lru_cursor_init() - Initialize a struct ttm_bo_lru_cursor +* @curs: The ttm_bo_lru_cursor to initialize. +* @man: The ttm resource_manager whose LRU lists to iterate over. +* @ctx: The ttm_operation_ctx to govern the locking. +* +* Initialize a struct ttm_bo_lru_cursor. Currently only trylocking +* or prelocked buffer objects are available as detailed by +* @ctx::resv and @ctx::allow_res_evict. Ticketlocking is not +* supported. +* +* Return: Pointer to @curs. The function does not fail. +*/ +struct ttm_bo_lru_cursor * +ttm_bo_lru_cursor_init (struct ttm_bo_lru_cursor *curs, +struct ttm_resource_manager *man, +struct ttm_operation_ctx *ctx) +{ +memset (curs,0 ,sizeof (*curs)); +ttm_resource_cursor_init (&curs->res_curs, man); + curs->ctx = ctx; + +return curs; +} +EXPORT_SYMBOL (ttm_bo_lru_cursor_init); + +static struct ttm_buffer_object * +ttm_bo_from_res_reserved (struct ttm_resource *res,struct ttm_bo_lru_cursor *curs) +{ +struct ttm_buffer_object *bo = res->bo; + +if (!ttm_lru_walk_trylock (curs->ctx, bo, &curs->needs_unlock)) +return NULL ; + +if (!ttm_bo_get_unless_zero (bo)) { +if (curs->needs_unlock) +dma_resv_unlock (bo->base.resv); +return NULL ; + } + + curs->bo = bo; +return bo; +} + +/** +* ttm_bo_lru_cursor_next() - Continue iterating a manager's LRU lists +* to find and lock buffer object. +* @curs: The cursor initialized using ttm_bo_lru_cursor_init() and +* ttm_bo_lru_cursor_first(). +* +* Return: A pointer to a locked and reference-counted buffer object, +* or NULL if none could be found and looping should be terminated. +*/ +struct ttm_buffer_object *ttm_bo_lru_cursor_next (struct ttm_bo_lru_cursor *curs) +{ +spinlock_t *lru_lock = &curs->res_curs.man->bdev->lru_lock; +struct ttm_resource *res =NULL ; +struct ttm_buffer_object *bo; + +ttm_bo_lru_cursor_cleanup_bo (curs); + +spin_lock (lru_lock); +for (;;) { + res =ttm_resource_manager_next (&curs->res_curs); +if (!res) +break ; + + bo =ttm_bo_from_res_reserved (res, curs); +if (bo) +break ; + } + +spin_unlock (lru_lock); +return res ? bo :NULL ; +} +EXPORT_SYMBOL (ttm_bo_lru_cursor_next); + +/** +* ttm_bo_lru_cursor_first() - Start iterating a manager's LRU lists +* to find and lock buffer object. +* @curs: The cursor initialized using ttm_bo_lru_cursor_init(). +* +* Return: A pointer to a locked and reference-counted buffer object, +* or NULL if none could be found and looping should be terminated. +*/ +struct ttm_buffer_object *ttm_bo_lru_cursor_first (struct ttm_bo_lru_cursor *curs) +{ +spinlock_t *lru_lock = &curs->res_curs.man->bdev->lru_lock; +struct ttm_buffer_object *bo; +struct ttm_resource *res; + +spin_lock (lru_lock); + res =ttm_resource_manager_first (&curs->res_curs); +if (!res) { +spin_unlock (lru_lock); +return NULL ; + } + + bo =ttm_bo_from_res_reserved (res, curs); +spin_unlock (lru_lock); + +return bo ? bo :ttm_bo_lru_cursor_next (curs); +} +EXPORT_SYMBOL (ttm_bo_lru_cursor_first);
include/drm/ttm/ttm_bo.h +72 -0
76 lines not on this page · raw message
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 5/7 drm/ttm: Add helpers for shrinking
In reply to Thomas HellströmAdd a number of helpers for shrinking that access core TTM and core MM functionality in a way that make them unsuitable for driver open-coding.
v11:
- New patch (split off from previous) and additional helpers. v13:
- Adapt to ttm_backup interface change.
- Take resource off LRU when backed up.
Christian König <[email protected]>
--- drivers/gpu/drm/ttm/ttm_bo_util.c | 107 +++++++++++++++++++++++++++++- drivers/gpu/drm/ttm/ttm_tt.c | 29 ++++++++ include/drm/ttm/ttm_bo.h | 21 ++++++ include/drm/ttm/ttm_tt.h | 2 + 4 files changed, 158 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_bo_util.c +106 -1
@@ -28,7 +28,7 @@/* * Authors: Thomas Hellstrom <thellstrom-at-vmware-dot-com> */ - +#include <linux/swap.h> #include <linux/vmalloc.h> #include <drm/ttm/ttm_bo.h>
@@ -1052,3 +1052,108 @@ struct ttm_buffer_object *ttm_bo_lru_cursor_first(struct ttm_bo_lru_cursor *cursreturn bo ? bo :ttm_bo_lru_cursor_next (curs); }EXPORT_SYMBOL (ttm_bo_lru_cursor_first); + +/** +* ttm_bo_shrink() - Helper to shrink a ttm buffer object. +* @ctx: The struct ttm_operation_ctx used for the shrinking operation. +* @bo: The buffer object. +* @flags: Flags governing the shrinking behaviour. +* +* The function uses the ttm_tt_back_up functionality to back up or +* purge a struct ttm_tt. If the bo is not in system, it's first +* moved there. +* +* Return: The number of pages shrunken or purged, or +* negative error code on failure. +*/ +long ttm_bo_shrink (struct ttm_operation_ctx *ctx,struct ttm_buffer_object *bo, +const struct ttm_bo_shrink_flags flags) +{ +static const struct ttm_place sys_placement_flags = { + .fpfn =0 , + .lpfn =0 , + .mem_type =TTM_PL_SYSTEM , + .flags =0 , + }; +static struct ttm_placement sys_placement = { + .num_placement =1 , + .placement = &sys_placement_flags, + }; +struct ttm_tt *tt = bo->ttm; +long lret; + +dma_resv_assert_held (bo->base.resv); + +if (flags.allow_move && bo->resource->mem_type !=TTM_PL_SYSTEM ) { +int ret =ttm_bo_validate (bo, &sys_placement, ctx); + +/* Consider -ENOMEM and -ENOSPC non-fatal. */ +if (ret) { +if (ret == -ENOMEM || ret == -ENOSPC ) + ret = -EBUSY ; +return ret; + } + } + +ttm_bo_unmap_virtual (bo); + lret =ttm_bo_wait_ctx (bo, ctx); +if (lret <0 ) +return lret; + +if (bo->bulk_move) { +spin_lock (&bo->bdev->lru_lock); +ttm_resource_del_bulk_move (bo->resource, bo); +spin_unlock (&bo->bdev->lru_lock); + } + + lret =ttm_tt_backup (bo->bdev, tt, (struct ttm_backup_flags ) + {.purge = flags.purge, + .writeback = flags.writeback}); + +if (lret <=0 && bo->bulk_move) { +spin_lock (&bo->bdev->lru_lock); +ttm_resource_add_bulk_move (bo->resource, bo); +spin_unlock (&bo->bdev->lru_lock); + } + +if (lret <0 && lret != -EINTR ) +return -EBUSY ; + +return lret; +} +EXPORT_SYMBOL (ttm_bo_shrink); + +/** +* ttm_bo_shrink_suitable() - Whether a bo is suitable for shinking +* @ctx: The struct ttm_operation_ctx governing the shrinking. +* @bo: The candidate for shrinking. +* +* Check whether the object, given the information available to TTM, +* is suitable for shinking, This function can and should be used +* before attempting to shrink an object. +* +* Return: true if suitable. false if not. +*/ +bool ttm_bo_shrink_suitable (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx) +{ +return bo->ttm &&ttm_tt_is_populated (bo->ttm) && !bo->pin_count && + (!ctx->no_wait_gpu || +dma_resv_test_signaled (bo->base.resv,DMA_RESV_USAGE_BOOKKEEP )); +} +EXPORT_SYMBOL (ttm_bo_shrink_suitable); + +/** +* ttm_bo_shrink_avoid_wait() - Whether to avoid waiting for GPU +* during shrinking +* +* In some situations, like direct reclaim, waiting (in particular gpu waiting) +* should be avoided since it may stall a system that could otherwise make progress +* shrinking something else less time consuming. +* +* Return: true if gpu waiting should be avoided, false if not. +*/ +bool ttm_bo_shrink_avoid_wait (void ) +{ +return !current_is_kswapd (); +} +EXPORT_SYMBOL (ttm_bo_shrink_avoid_wait);
drivers/gpu/drm/ttm/ttm_tt.c +29 -0
@@ -531,3 +531,32 @@ unsigned long ttm_tt_pages_limit(void)return ttm_pages_limit; }EXPORT_SYMBOL (ttm_tt_pages_limit); + +/** +* ttm_tt_setup_backup() - Allocate and assign a backup structure for a ttm_tt +* @tt: The ttm_tt for wich to allocate and assign a backup structure. +* +* Assign a backup structure to be used for tt backup. This should +* typically be done at bo creation, to avoid allocations at shrinking +* time. +* +* Return: 0 on success, negative error code on failure. +*/ +int ttm_tt_setup_backup (struct ttm_tt *tt) +{ +struct ttm_backup *backup = +ttm_backup_shmem_create (((loff_t )tt->num_pages) <<PAGE_SHIFT ); + +if (WARN_ON_ONCE (!(tt->page_flags &TTM_TT_FLAG_EXTERNAL_MAPPABLE ))) +return -EINVAL ; + +if (IS_ERR (backup)) +return PTR_ERR (backup); + +if (tt->backup) +ttm_backup_fini (tt->backup); + + tt->backup = backup; +return 0 ; +} +EXPORT_SYMBOL (ttm_tt_setup_backup);
include/drm/ttm/ttm_bo.h +21 -0
@@ -225,6 +225,27 @@ struct ttm_lru_walk {s64 ttm_lru_walk_for_evict (struct ttm_lru_walk *walk,struct ttm_device *bdev,struct ttm_resource_manager *man,s64 target); +/** +* struct ttm_bo_shrink_flags - flags to govern the bo shrinking behaviour +* @purge: Purge the content rather than backing it up. +* @writeback: Attempt to immediately write content to swap space. +* @allow_move: Allow moving to system before shrinking. This is typically +* not desired for zombie- or ghost objects (with zombie object meaning +* objects with a zero gem object refcount) +*/ +struct ttm_bo_shrink_flags { +u32 purge :1 ; +u32 writeback :1 ; +u32 allow_move :1 ; +}; + +long ttm_bo_shrink (struct ttm_operation_ctx *ctx,struct ttm_buffer_object *bo, +const struct ttm_bo_shrink_flags flags); + +bool ttm_bo_shrink_suitable (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx); + +bool ttm_bo_shrink_avoid_wait (void ); +/** * ttm_bo_get - reference a struct ttm_buffer_object *
include/drm/ttm/ttm_tt.h +2 -0
@@ -298,6 +298,8 @@ long ttm_tt_backup(struct ttm_device *bdev, struct ttm_tt *tt,int ttm_tt_restore (struct ttm_device *bdev,struct ttm_tt *tt,const struct ttm_operation_ctx *ctx); +int ttm_tt_setup_backup (struct ttm_tt *tt); +#if IS_ENABLED (CONFIG_AGP )#include <linux/agp_backend.h>
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 6/7 drm/xe: Add a shrinker for xe bos
In reply to Thomas HellströmRather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.
Leverage the newly added TTM per-page shrinking and shmem backup support.
Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.
v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
v4:
- Add runtime pm for the shrinker. Shrinking may require an active device for CCS metadata copying.
v5:
- Separately purge ghost- and zombie objects in the shrinker.
- Fix a format specifier - type inconsistency. (Kernel test robot). v7:
- s/long/s64/ (Christian König)
- s/sofar/progress/ (Matt Brost)
v8:
- Rebase on Xe KUNIT update.
- Add content verifying to the shrinker kunit test.
- Split out TTM changes to a separate patch.
- Get rid of multiple bool arguments for clarity (Matt Brost)
- Avoid an error pointer dereference (Matt Brost)
- Avoid an integer overflow (Matt Auld)
- Address misc review comments by Matt Brost.
v9:
- Fix a compliation error.
- Rebase.
v10:
- Update to new LRU walk interface.
- Rework ghost-, zombie and purged object shrinking.
- Rebase.
v11:
- Use additional TTM helpers.
- Honor __GFP_FS and __GFP_IO
- Rebase.
v13:
- Use ttm_tt_setup_backup().
v14:
- Don't set up backup on imported bos.
v15:
- Rebase on backup interface changes.
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/tests/xe_bo.c | 6 +- drivers/gpu/drm/xe/xe_bo.c | 202 +++++++++++++++++++-- drivers/gpu/drm/xe/xe_bo.h | 36 ++++ drivers/gpu/drm/xe/xe_device.c | 8 + drivers/gpu/drm/xe/xe_device_types.h | 2 + drivers/gpu/drm/xe/xe_shrinker.c | 258 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_shrinker.h | 18 ++ 8 files changed, 513 insertions(+), 18 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
drivers/gpu/drm/xe/Makefile +1 -0
@@ -100,6 +100,7 @@ xe-y += xe_bb.o \ xe_rtp.o \ xe_sa.o \ xe_sched_job.o \ + xe_shrinker.o \ xe_step.o \ xe_survivability_mode.o \ xe_sync.o \
drivers/gpu/drm/xe/tests/xe_bo.c +5 -1
@@ -514,8 +514,13 @@ static int shrink_test_run_device(struct xe_device *xe)* other way around, they may not be subject to swapping... */ if (alloced < purgeable) { +xe_ttm_tt_account_subtract (&xe_tt->ttm); xe_tt->purgeable = true; +xe_ttm_tt_account_add (&xe_tt->ttm); bo->ttm.priority =0 ; +spin_lock (&bo->ttm.bdev->lru_lock); +ttm_bo_move_to_lru_tail (&bo->ttm); +spin_unlock (&bo->ttm.bdev->lru_lock); }else {int ret =shrink_test_fill_random (bo, &prng, link);
@@ -570,7 +575,6 @@ static int shrink_test_run_device(struct xe_device *xe)if (ret == -EINTR ) intr = true; }while (ret == -EINTR && !signal_pending (current)); -if (!ret && !purgeable) failed =shrink_test_verify (test, bo, count, &prng, link);
drivers/gpu/drm/xe/xe_bo.c +185 -17
326 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.h +36 -0
@@ -148,6 +148,28 @@ static inline struct xe_bo *xe_bo_get(struct xe_bo *bo)void xe_bo_put (struct xe_bo *bo); +/* +* xe_bo_get_unless_zero() - Conditionally obtain a GEM object refcount on an +* xe bo +* @bo: The bo for which we want to obtain a refcount. +* +* There is a short window between where the bo's GEM object refcount reaches +* zero and where we put the final ttm_bo reference. Code in the eviction- and +* shrinking path should therefore attempt to grab a gem object reference before +* trying to use members outside of the base class ttm object. This function is +* intended for that purpose. On successful return, this function must be paired +* with an xe_bo_put(). +* +* Return: @bo on success, NULL on failure. +*/ +static inline __must_check struct xe_bo *xe_bo_get_unless_zero (struct xe_bo *bo) +{ +if (!bo || !kref_get_unless_zero (&bo->ttm.base.refcount)) +return NULL ; + +return bo; +} +static inline void __xe_bo_unset_bulk_move (struct xe_bo *bo) {if (bo)
@@ -349,6 +371,20 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)return round_down (max /2 ,PAGE_SIZE ); } +/** +* struct xe_bo_shrink_flags - flags governing the shrink behaviour. +* @purge: Only purging allowed. Don't shrink if bo not purgeable. +* @writeback: Attempt to immediately move content to swap. +*/ +struct xe_bo_shrink_flags { +u32 purge :1 ; +u32 writeback :1 ; +}; + +long xe_bo_shrink (struct ttm_operation_ctx *ctx,struct ttm_buffer_object *bo, +const struct xe_bo_shrink_flags flags, +unsigned long *scanned); +#if IS_ENABLED (CONFIG_DRM_XE_KUNIT_TEST )/** * xe_bo_is_mem_type - Whether the bo currently resides in the given
drivers/gpu/drm/xe/xe_device.c +8 -0
@@ -52,6 +52,7 @@#include "xe_pmu.h" #include "xe_pxp.h" #include "xe_query.h" +#include "xe_shrinker.h" #include "xe_sriov.h" #include "xe_tile.h" #include "xe_ttm_stolen_mgr.h"
@@ -396,6 +397,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)if (xe->unordered_wq)destroy_workqueue (xe->unordered_wq); +if (!IS_ERR_OR_NULL (xe->mem.shrinker)) +xe_shrinker_destroy (xe->mem.shrinker); +if (xe->destroy_wq)destroy_workqueue (xe->destroy_wq);
@@ -428,6 +432,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,if (err) goto err; + xe->mem.shrinker =xe_shrinker_create (xe); +if (IS_ERR (xe->mem.shrinker)) +return ERR_CAST (xe->mem.shrinker); + xe->info.devid = pdev->device; xe->info.revid = pdev->revision; xe->info.force_execlist = xe_modparam.force_execlist;
drivers/gpu/drm/xe/xe_device_types.h +2 -0
@@ -372,6 +372,8 @@ struct xe_device {struct xe_vram_region vram;/** @mem.sys_mgr: system TTM manager */ struct ttm_resource_manager sys_mgr; +/** @mem.sys_mgr: system memory shrinker. */ +struct xe_shrinker *shrinker; } mem;/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_shrinker.c new file +258 -0
258 lines not on this page · raw message
drivers/gpu/drm/xe/xe_shrinker.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef _XE_SHRINKER_H_ +#define _XE_SHRINKER_H_ + +struct xe_shrinker ; +struct xe_device ; + +void xe_shrinker_mod_pages (struct xe_shrinker *shrinker,long shrinkable,long purgeable); + +struct xe_shrinker *xe_shrinker_create (struct xe_device *xe); + +void xe_shrinker_destroy (struct xe_shrinker *shrinker); + +#endif
Signature · 1 line
2.48.1
Thomas Hellström
maintainerPATCH v17 7/7 drm/xe: Increase the XE_PL_TT watermark
In reply to Thomas HellströmThe XE_PL_TT watermark was set to 50% of system memory. The idea behind that was unclear since the net effect is that TT memory will be evicted to TTM_PL_SYSTEM memory if that watermark is exceeded, requiring PPGTT rebinds and dma remapping. But there is no similar watermark for TTM_PL_1SYSTEM memory.
The TTM functionality that tries to swap out system memory to shmem objects if a 50% limit of total system memory is reached is orthogonal to this, and with the shrinker added, it's no longer in effect.
Replace the 50% TTM_PL_TT limit with a 100% limit, in effect allowing all graphics memory to be bound to the device unless it has been swapped out by the shrinker.
--- drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -2
@@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)u64 gtt_size;si_meminfo (&si); +/* Potentially restrict amount of TT memory here. */ gtt_size = (u64 )si.totalram * si.mem_unit; -/* TTM limits allocation of all TTM devices by 50% of system memory */ - gtt_size /=2 ; man->use_tt = true; man->func = &xe_ttm_sys_mgr_func;
Signature · 1 line
2.48.1