TTM shrinker helpers and xe buffer object shrinker
Subsystems (14)
Thomas Hellström
maintainerv3 cover TTM shrinker helpers and xe buffer object shrinker
This series implements TTM shrinker / eviction helpers and an xe bo shrinker. It builds on two previous series, *and obsoletes these*. First
https://www.mail-archive.com/[email protected]/msg484425.html
Second the previous TTM shrinker series
https://lore.kernel.org/linux-mm/[email protected]/T/
Where the comment about layering
https://lore.kernel.org/linux-mm/[email protected]/T/#ma918844aa8a6efe8768fdcda0c6590d5c93850c9
now addressed, and this version also implements shmem objects for backup rather than direct swap-cache insertions, which was used in the previuos series. It turns out that with per-page backup / shrinking, shmem objects appears to work just as well as direct swap-cache insertions with the added benefit that was introduced in the previous TTM shrinker series to avoid running out of swap entries isn't really needed.
Patch 1-4 implements restartable LRU list iteration.
Patch 5 implements a LRU walker + resv locking helper
Patch 6 moves TTM swapping over to the walker.
Patch 7 moves TTM eviction over to the walker.
Patch 8 could in theory be skipped but introduces a possibility to easily add or test multiple backup backends, like the direct swap-cache insertion or even files into fast dedicated nvme storage for for example.
Patch 9 introduces helpers in the ttm_pool code for page-by-page shrinking and recovery. It avoids having to temporarily allocate a huge amount of memory to be able to shrink a buffer object. It also introduces the possibility to immediately write-back pages if needed, since that tends to be a bit delayed when left to kswapd.
Patch 10 Adds a simple error injection to the above code to help increase test coverage.
Patch 11 Implements an xe bo shrinker and a common helper in TTM for shrinking.
Patch 12-21 are really a separate POC series, for introducing drm_exec locking in TTM. The patch touches both drm_exec and dma-buf and is for now marked as an RFC:
Patch 12 Introduces dma_resv_trylock_ctx.
Patches 13-14 deal with introducing drm_exec_trylock.
Patch 15 adds a snapshot capability to drm_exec.
Patch 16 adds an evict mode locking capability to drm_exec
Patch 17 converts the LRU + locking walker to drm_exec.
Patch 18 converts TTM vm to use drm_exec.
Patch 19 converts the xe fault handler to drm_exec.
Patch 20 converts bo initialization locking to drm_exec
Patch 21 introduces drm_exec locking around some of the bo validation callsites in drm_exec.
v2:
- Squash obsolete revision history in the patch commit messages.
- Fix a couple of review comments by Christian
- Don't store the mem_type in the TTM managers but in the resource cursor.
- Rename introduced TTM *back_up* function names to *backup*
- Add ttm pool recovery fault injection.
- Shrinker xe kunit test
- Various bugfixes
v3:
- Address some review comments from Matthew Brost and Christian König.
- Use the restartable LRU walk for TTM swapping and eviction.
- Provide a POC drm_exec locking implementation for exhaustive eviction. (Christian König).
Thomas Hellström (21):
drm/ttm: Allow TTM LRU list nodes of different types drm/ttm: Slightly clean up LRU list iteration
drm/ttm: Use LRU hitches
drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist
moves
drm/ttm: Provide a generic LRU walker helper
drm/ttm: Use the LRU walker helper for swapping
drm/ttm: Use the LRU walker for eviction
drm/ttm: Add a virtual base class for graphics memory backup drm/ttm/pool: Provide a helper to shrink pages
drm/ttm: Use fault-injection to test error paths
drm/ttm, drm/xe: Add a shrinker for xe bos dma-buf/dma-resv: Introduce dma_resv_trylock_ctx() drm/exec: Rework contended locking
drm/exec: Introduce a drm_exec_trylock_obj() function drm/exec: Add a snapshot capability
drm/exec: Introduce an evict mode
drm/ttm: Support drm_exec locking for eviction and swapping drm/ttm: Convert ttm vm to using drm_exec
drm/xe: Use drm_exec for fault locking
drm/ttm: Use drm_exec_trylock for bo initialization drm/xe: Initial support for drm exec locking during validate
drivers/gpu/drm/Kconfig | 10 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 12 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 +- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 209 ++++++- drivers/gpu/drm/drm_gpuvm.c | 8 +- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_gem.c | 4 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/radeon/radeon_gem.c | 4 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +- drivers/gpu/drm/ttm/Makefile | 2 +- drivers/gpu/drm/ttm/ttm_backup_shmem.c | 137 +++++ drivers/gpu/drm/ttm/ttm_bo.c | 516 +++++++++--------- drivers/gpu/drm/ttm/ttm_bo_util.c | 241 ++++++++ drivers/gpu/drm/ttm/ttm_bo_vm.c | 101 ++-- drivers/gpu/drm/ttm/ttm_device.c | 29 +- drivers/gpu/drm/ttm/ttm_pool.c | 412 +++++++++++++- drivers/gpu/drm/ttm/ttm_resource.c | 264 +++++++-- drivers/gpu/drm/ttm/ttm_tt.c | 37 ++ drivers/gpu/drm/vmwgfx/vmwgfx_page_dirty.c | 6 +- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/display/xe_fb_pin.c | 2 +- drivers/gpu/drm/xe/tests/xe_bo.c | 124 ++++- drivers/gpu/drm/xe/tests/xe_bo_test.c | 1 + drivers/gpu/drm/xe/tests/xe_bo_test.h | 1 + drivers/gpu/drm/xe/tests/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_bo.c | 173 ++++-- drivers/gpu/drm/xe/xe_bo.h | 8 +- drivers/gpu/drm/xe/xe_device.c | 8 + drivers/gpu/drm/xe/xe_device_types.h | 2 + drivers/gpu/drm/xe/xe_dma_buf.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 2 +- drivers/gpu/drm/xe/xe_gt_pagefault.c | 6 +- drivers/gpu/drm/xe/xe_shrinker.c | 224 ++++++++ drivers/gpu/drm/xe/xe_shrinker.h | 18 + drivers/gpu/drm/xe/xe_vm.c | 18 +- include/drm/drm_exec.h | 62 ++- include/drm/ttm/ttm_backup.h | 136 +++++ include/drm/ttm/ttm_bo.h | 59 +- include/drm/ttm/ttm_pool.h | 5 + include/drm/ttm/ttm_resource.h | 99 +++- include/drm/ttm/ttm_tt.h | 20 + include/linux/dma-resv.h | 23 +- 53 files changed, 2535 insertions(+), 531 deletions(-) create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h create mode 100644 include/drm/ttm/ttm_backup.h
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 1/21 drm/ttm: Allow TTM LRU list nodes of different types
To be able to handle list unlocking while traversing the LRU list, we want the iterators not only to point to the next position of the list traversal, but to insert themselves as list nodes at that point to work around the fact that the next node might otherwise disappear from the list while the iterator is pointing to it.
These list nodes need to be easily distinguishable from other list nodes so that others traversing the list can skip over them.
So declare a struct ttm_lru_item, with a struct list_head member and a type enum. This will slightly increase the size of a struct ttm_resource.
Changes in previous series:
- Update enum ttm_lru_item_type documentation.
v3:
- Introduce ttm_lru_first_res_or_null()
(Christian König, Thomas Hellström)
--- drivers/gpu/drm/ttm/ttm_device.c | 4 +- drivers/gpu/drm/ttm/ttm_resource.c | 89 +++++++++++++++++++++++------- include/drm/ttm/ttm_resource.h | 54 +++++++++++++++++- 3 files changed, 125 insertions(+), 22 deletions(-)
drivers/gpu/drm/ttm/ttm_device.c +2 -2
@@ -274,14 +274,14 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,struct ttm_resource *res;spin_lock (&bdev->lru_lock); -while ((res =list_first_entry_or_null (list,typeof (*res), lru))) { +while ((res =ttm_lru_first_res_or_null (list))) {struct ttm_buffer_object *bo = res->bo;/* Take ref against racing releases once lru_lock is unlocked */ if (!ttm_bo_get_unless_zero (bo))continue ; -list_del_init (&res->lru); +list_del_init (&bo->resource->lru.link);spin_unlock (&bdev->lru_lock);if (bo->ttm)
drivers/gpu/drm/ttm/ttm_resource.c +70 -19
162 lines not on this page · raw message
include/drm/ttm/ttm_resource.h +53 -1
74 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH v3 01/21] drm/ttm: Allow TTM LRU list nodes of different types
Quoted 21 lines · Thomas Hellström
To be able to handle list unlocking while traversing the LRU list, we want the iterators not only to point to the next position of the list traversal, but to insert themselves as list nodes at that point to work around the fact that the next node might otherwise disappear from the list while the iterator is pointing to it.
These list nodes need to be easily distinguishable from other list nodes so that others traversing the list can skip over them.
So declare a struct
ttm_lru_item, with a structlist_headmember and a type enum. This will slightly increase the size of a structttm_resource.Changes in previous series:
- Update enumttm_lru_item_typedocumentation.
v3:
- Introducettm_lru_first_res_or_null()
(Christian König, Thomas Hellström)
Quoted 3 lines
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 288 lines
Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- drivers/gpu/drm/ttm/ttm_device.c | 4 +- drivers/gpu/drm/ttm/ttm_resource.c | 89 +++++++++++++++++++++++------- include/drm/ttm/ttm_resource.h | 54 +++++++++++++++++- 3 files changed, 125 insertions(+), 22 deletions(-)drivers/gpu/drm/ttm/ttm_device.c +2
-2@@ -274,14 +274,14 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,struct ttm_resource *res;spin_lock (&bdev->lru_lock); -while ((res =list_first_entry_or_null (list,typeof (*res), lru))) { +while ((res =ttm_lru_first_res_or_null (list))) {struct ttm_buffer_object *bo = res->bo;/* Take ref against racing releases once lru_lock is unlocked */ if (!ttm_bo_get_unless_zero (bo))continue ; -list_del_init (&res->lru); +list_del_init (&bo->resource->lru.link);spin_unlock (&bdev->lru_lock);if (bo->ttm)drivers/gpu/drm/ttm/ttm_resource.c +70
-19@@ -70,8 +70,8 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)dma_resv_assert_held (pos->last->bo->base.resv); man =ttm_manager_type (pos->first->bo->bdev, i); -list_bulk_move_tail (&man->lru[j], &pos->first->lru, - &pos->last->lru); +list_bulk_move_tail (&man->lru[j], &pos->first->lru.link, + &pos->last->lru.link); } } }@@ -84,14 +84,38 @@ ttm_lru_bulk_move_pos(struct ttm_lru_bulk_move *bulk, struct ttm_resource *res)return &bulk->pos[res->mem_type][res->bo->priority]; } +/* Return the previous resource on the list (skip over non-resource list items) */ +static struct ttm_resource *ttm_lru_prev_res (struct ttm_resource *cur) +{ +struct ttm_lru_item *lru = &cur->lru; + +do { + lru =list_prev_entry (lru, link); + }while (!ttm_lru_item_is_res (lru)); + +return ttm_lru_item_to_res (lru); +} + +/* Return the next resource on the list (skip over non-resource list items) */ +static struct ttm_resource *ttm_lru_next_res (struct ttm_resource *cur) +{ +struct ttm_lru_item *lru = &cur->lru; + +do { + lru =list_next_entry (lru, link); + }while (!ttm_lru_item_is_res (lru)); + +return ttm_lru_item_to_res (lru); +} +/* Move the resource to the tail of the bulk move range */ static void ttm_lru_bulk_move_pos_tail (struct ttm_lru_bulk_move_pos *pos,struct ttm_resource *res) {if (pos->last != res) {if (pos->first == res) - pos->first =list_next_entry (res, lru); -list_move (&res->lru, &pos->last->lru); + pos->first =ttm_lru_next_res (res); +list_move (&res->lru.link, &pos->last->lru.link); pos->last = res; } }@@ -122,11 +146,11 @@ static void ttm_lru_bulk_move_del(struct ttm_lru_bulk_move *bulk, pos->first =NULL ; pos->last =NULL ; }else if (pos->first == res) { - pos->first =list_next_entry (res, lru); + pos->first =ttm_lru_next_res (res); }else if (pos->last == res) { - pos->last =list_prev_entry (res, lru); + pos->last =ttm_lru_prev_res (res); }else { -list_move (&res->lru, &pos->last->lru); +list_move (&res->lru.link, &pos->last->lru.link); } }@@ -155,7 +179,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)lockdep_assert_held (&bo->bdev->lru_lock);if (bo->pin_count) { -list_move_tail (&res->lru, &bdev->pinned); +list_move_tail (&res->lru.link, &bdev->pinned); }else if (bo->bulk_move) { struct ttm_lru_bulk_move_pos *pos =@@ -166,7 +190,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)struct ttm_resource_manager *man; man =ttm_manager_type (bdev, res->mem_type); -list_move_tail (&res->lru, &man->lru[bo->priority]); +list_move_tail (&res->lru.link, &man->lru[bo->priority]); } }@@ -197,9 +221,9 @@ void ttm_resource_init(struct ttm_buffer_object *bo, man =ttm_manager_type (bo->bdev, place->mem_type);spin_lock (&bo->bdev->lru_lock);if (bo->pin_count) -list_add_tail (&res->lru, &bo->bdev->pinned); +list_add_tail (&res->lru.link, &bo->bdev->pinned);else -list_add_tail (&res->lru, &man->lru[bo->priority]); +list_add_tail (&res->lru.link, &man->lru[bo->priority]); man->usage += res->size;spin_unlock (&bo->bdev->lru_lock); }@@ -221,7 +245,7 @@ void ttm_resource_fini(struct ttm_resource_manager *man,struct ttm_device *bdev = man->bdev;spin_lock (&bdev->lru_lock); -list_del_init (&res->lru); +list_del_init (&res->lru.link); man->usage -= res->size;spin_unlock (&bdev->lru_lock); }@@ -472,14 +496,16 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor) { - struct ttm_resource *res; + struct ttm_lru_item *lru;lockdep_assert_held (&man->bdev->lru_lock);for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority) -list_for_each_entry (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }return NULL ; }@@ -498,15 +524,40 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,struct ttm_resource_cursor *cursor, struct ttm_resource *res) { + struct ttm_lru_item *lru = &res->lru; +lockdep_assert_held (&man->bdev->lru_lock); -list_for_each_entry_continue (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority) -list_for_each_entry (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +ttm_lru_item_to_res (lru); + } + +return NULL ; +} + +/** +* ttm_lru_first_res_or_null() - Return the first resource on an lru list +* @head: The list head of the lru list. +* +* Return: Pointer to the first resource on the lru list or NULL if +* there is none. +*/ +struct ttm_resource *ttm_lru_first_res_or_null (struct list_head *head) +{ +struct ttm_lru_item *lru; + +list_for_each_entry (lru, head, link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }return NULL ; }include/drm/ttm/ttm_resource.h +53
-1@@ -49,6 +49,43 @@ struct io_mapping;struct sg_table ;struct scatterlist ; +/** +* enum ttm_lru_item_type - enumerate ttm_lru_item subclasses +*/ +enum ttm_lru_item_type { +/** @TTM_LRU_RESOURCE: The resource subclass */ +TTM_LRU_RESOURCE , +/** @TTM_LRU_HITCH: The iterator hitch subclass */ +TTM_LRU_HITCH +}; + +/** +* struct ttm_lru_item - The TTM lru list node base class +* @link: The list link +* @type: The subclass type +*/ +struct ttm_lru_item { +struct list_head link; +enum ttm_lru_item_type type; +}; + +/** +* ttm_lru_item_init() - initialize a struct ttm_lru_item +* @item: The item to initialize +* @type: The subclass type +*/ +static inline void ttm_lru_item_init (struct ttm_lru_item *item, +enum ttm_lru_item_type type) +{ + item->type = type; +INIT_LIST_HEAD (&item->link); +} + +static inline bool ttm_lru_item_is_res (const struct ttm_lru_item *item) +{ +return item->type ==TTM_LRU_RESOURCE ; +} +struct ttm_resource_manager_func {/** * struct ttm_resource_manager_func member alloc @@ -217,9 +254,21 @@ struct ttm_resource {/** * @lru: Least recently used list, see &ttm_resource_manager.lru */ -struct list_head lru; +struct ttm_lru_item lru; }; +/** +* ttm_lru_item_to_res() - Downcast a struct ttm_lru_item to a struct ttm_resource +* @item: The struct ttm_lru_item to downcast +* +* Return: Pointer to the embedding struct ttm_resource +*/ +static inline struct ttm_resource * +ttm_lru_item_to_res (struct ttm_lru_item *item) +{ + return container_of(item, struct ttm_resource, lru); +} +/** * struct ttm_resource_cursor * @@ -393,6 +442,9 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,struct ttm_resource_cursor *cursor, struct ttm_resource *res); +struct ttm_resource * +ttm_lru_first_res_or_null (struct list_head *head); +/** * ttm_resource_manager_for_each_res - iterate over all resources * @man: the resource manager Signature · 1 line
2.44.0
Re: [PATCH v3 01/21] drm/ttm: Allow TTM LRU list nodes of different types
In reply to Thomas HellströmAm 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 21 lines
To be able to handle list unlocking while traversing the LRU list, we want the iterators not only to point to the next position of the list traversal, but to insert themselves as list nodes at that point to work around the fact that the next node might otherwise disappear from the list while the iterator is pointing to it.
These list nodes need to be easily distinguishable from other list nodes so that others traversing the list can skip over them.
So declare a struct
ttm_lru_item, with a structlist_headmember and a type enum. This will slightly increase the size of a structttm_resource.Changes in previous series:
- Update enumttm_lru_item_typedocumentation.
v3:
- Introducettm_lru_first_res_or_null()
(Christian König, Thomas Hellström)
Quoted 5 lines
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>
Quoted 283 lines
--- drivers/gpu/drm/ttm/ttm_device.c | 4 +- drivers/gpu/drm/ttm/ttm_resource.c | 89 +++++++++++++++++++++++------- include/drm/ttm/ttm_resource.h | 54 +++++++++++++++++- 3 files changed, 125 insertions(+), 22 deletions(-)drivers/gpu/drm/ttm/ttm_device.c +2
-2@@ -274,14 +274,14 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,struct ttm_resource *res;spin_lock (&bdev->lru_lock); -while ((res =list_first_entry_or_null (list,typeof (*res), lru))) { +while ((res =ttm_lru_first_res_or_null (list))) {struct ttm_buffer_object *bo = res->bo;/* Take ref against racing releases once lru_lock is unlocked */ if (!ttm_bo_get_unless_zero (bo))continue ; -list_del_init (&res->lru); +list_del_init (&bo->resource->lru.link);spin_unlock (&bdev->lru_lock);if (bo->ttm)drivers/gpu/drm/ttm/ttm_resource.c +70
-19@@ -70,8 +70,8 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)dma_resv_assert_held (pos->last->bo->base.resv); man =ttm_manager_type (pos->first->bo->bdev, i); -list_bulk_move_tail (&man->lru[j], &pos->first->lru, - &pos->last->lru); +list_bulk_move_tail (&man->lru[j], &pos->first->lru.link, + &pos->last->lru.link); } } }@@ -84,14 +84,38 @@ ttm_lru_bulk_move_pos(struct ttm_lru_bulk_move *bulk, struct ttm_resource *res)return &bulk->pos[res->mem_type][res->bo->priority]; } +/* Return the previous resource on the list (skip over non-resource list items) */ +static struct ttm_resource *ttm_lru_prev_res (struct ttm_resource *cur) +{ +struct ttm_lru_item *lru = &cur->lru; + +do { + lru =list_prev_entry (lru, link); + }while (!ttm_lru_item_is_res (lru)); + +return ttm_lru_item_to_res (lru); +} + +/* Return the next resource on the list (skip over non-resource list items) */ +static struct ttm_resource *ttm_lru_next_res (struct ttm_resource *cur) +{ +struct ttm_lru_item *lru = &cur->lru; + +do { + lru =list_next_entry (lru, link); + }while (!ttm_lru_item_is_res (lru)); + +return ttm_lru_item_to_res (lru); +} +/* Move the resource to the tail of the bulk move range */ static void ttm_lru_bulk_move_pos_tail (struct ttm_lru_bulk_move_pos *pos,struct ttm_resource *res) {if (pos->last != res) {if (pos->first == res) - pos->first =list_next_entry (res, lru); -list_move (&res->lru, &pos->last->lru); + pos->first =ttm_lru_next_res (res); +list_move (&res->lru.link, &pos->last->lru.link); pos->last = res; } }@@ -122,11 +146,11 @@ static void ttm_lru_bulk_move_del(struct ttm_lru_bulk_move *bulk, pos->first =NULL ; pos->last =NULL ; }else if (pos->first == res) { - pos->first =list_next_entry (res, lru); + pos->first =ttm_lru_next_res (res); }else if (pos->last == res) { - pos->last =list_prev_entry (res, lru); + pos->last =ttm_lru_prev_res (res); }else { -list_move (&res->lru, &pos->last->lru); +list_move (&res->lru.link, &pos->last->lru.link); } }@@ -155,7 +179,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)lockdep_assert_held (&bo->bdev->lru_lock);if (bo->pin_count) { -list_move_tail (&res->lru, &bdev->pinned); +list_move_tail (&res->lru.link, &bdev->pinned); }else if (bo->bulk_move) { struct ttm_lru_bulk_move_pos *pos =@@ -166,7 +190,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)struct ttm_resource_manager *man; man =ttm_manager_type (bdev, res->mem_type); -list_move_tail (&res->lru, &man->lru[bo->priority]); +list_move_tail (&res->lru.link, &man->lru[bo->priority]); } }@@ -197,9 +221,9 @@ void ttm_resource_init(struct ttm_buffer_object *bo, man =ttm_manager_type (bo->bdev, place->mem_type);spin_lock (&bo->bdev->lru_lock);if (bo->pin_count) -list_add_tail (&res->lru, &bo->bdev->pinned); +list_add_tail (&res->lru.link, &bo->bdev->pinned);else -list_add_tail (&res->lru, &man->lru[bo->priority]); +list_add_tail (&res->lru.link, &man->lru[bo->priority]); man->usage += res->size;spin_unlock (&bo->bdev->lru_lock); }@@ -221,7 +245,7 @@ void ttm_resource_fini(struct ttm_resource_manager *man,struct ttm_device *bdev = man->bdev;spin_lock (&bdev->lru_lock); -list_del_init (&res->lru); +list_del_init (&res->lru.link); man->usage -= res->size;spin_unlock (&bdev->lru_lock); }@@ -472,14 +496,16 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor) { - struct ttm_resource *res; + struct ttm_lru_item *lru;lockdep_assert_held (&man->bdev->lru_lock);for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority) -list_for_each_entry (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }return NULL ; }@@ -498,15 +524,40 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,struct ttm_resource_cursor *cursor, struct ttm_resource *res) { + struct ttm_lru_item *lru = &res->lru; +lockdep_assert_held (&man->bdev->lru_lock); -list_for_each_entry_continue (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority) -list_for_each_entry (res, &man->lru[cursor->priority], lru) -return res; +list_for_each_entry (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) +ttm_lru_item_to_res (lru); + } + +return NULL ; +} + +/** +* ttm_lru_first_res_or_null() - Return the first resource on an lru list +* @head: The list head of the lru list. +* +* Return: Pointer to the first resource on the lru list or NULL if +* there is none. +*/ +struct ttm_resource *ttm_lru_first_res_or_null (struct list_head *head) +{ +struct ttm_lru_item *lru; + +list_for_each_entry (lru, head, link) { +if (ttm_lru_item_is_res (lru)) +return ttm_lru_item_to_res (lru); + }return NULL ; }include/drm/ttm/ttm_resource.h +53
-1@@ -49,6 +49,43 @@ struct io_mapping;struct sg_table ;struct scatterlist ; +/** +* enum ttm_lru_item_type - enumerate ttm_lru_item subclasses +*/ +enum ttm_lru_item_type { +/** @TTM_LRU_RESOURCE: The resource subclass */ +TTM_LRU_RESOURCE , +/** @TTM_LRU_HITCH: The iterator hitch subclass */ +TTM_LRU_HITCH +}; + +/** +* struct ttm_lru_item - The TTM lru list node base class +* @link: The list link +* @type: The subclass type +*/ +struct ttm_lru_item { +struct list_head link; +enum ttm_lru_item_type type; +}; + +/** +* ttm_lru_item_init() - initialize a struct ttm_lru_item +* @item: The item to initialize +* @type: The subclass type +*/ +static inline void ttm_lru_item_init (struct ttm_lru_item *item, +enum ttm_lru_item_type type) +{ + item->type = type; +INIT_LIST_HEAD (&item->link); +} + +static inline bool ttm_lru_item_is_res (const struct ttm_lru_item *item) +{ +return item->type ==TTM_LRU_RESOURCE ; +} +struct ttm_resource_manager_func {/** * struct ttm_resource_manager_func member alloc @@ -217,9 +254,21 @@ struct ttm_resource {/** * @lru: Least recently used list, see &ttm_resource_manager.lru */ -struct list_head lru; +struct ttm_lru_item lru; }; +/** +* ttm_lru_item_to_res() - Downcast a struct ttm_lru_item to a struct ttm_resource +* @item: The struct ttm_lru_item to downcast +* +* Return: Pointer to the embedding struct ttm_resource +*/ +static inline struct ttm_resource * +ttm_lru_item_to_res (struct ttm_lru_item *item) +{ + return container_of(item, struct ttm_resource, lru); +} +/** * struct ttm_resource_cursor * @@ -393,6 +442,9 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,struct ttm_resource_cursor *cursor, struct ttm_resource *res); +struct ttm_resource * +ttm_lru_first_res_or_null (struct list_head *head); +/** * ttm_resource_manager_for_each_res - iterate over all resources * @man: the resource manager
Thomas Hellström
maintainerPATCH v3 2/21 drm/ttm: Slightly clean up LRU list iteration
In reply to Thomas HellströmTo make the transition to using lru hitches easier, simplify the ttm_resource_manager_next() interface to only take the cursor and reuse ttm_resource_manager_next() functionality from ttm_resource_manager_first().
--- drivers/gpu/drm/ttm/ttm_resource.c | 48 +++++++++++++----------------- include/drm/ttm/ttm_resource.h | 10 ++++--- 2 files changed, 27 insertions(+), 31 deletions(-)
drivers/gpu/drm/ttm/ttm_resource.c +21 -27
@@ -496,50 +496,44 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor) { - struct ttm_lru_item *lru; - lockdep_assert_held(&man->bdev->lru_lock); -for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -return NULL ; + cursor->priority =0 ; + cursor->man = man; + cursor->cur = &man->lru[cursor->priority]; +return ttm_resource_manager_next (cursor); }/** * ttm_resource_manager_next * -* @man: resource manager to iterate over * @cursor: cursor to record the position -* @res: the current resource pointer * -* Returns the next resource from the resource manager. +* Return: the next resource from the resource manager. */ struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res) +ttm_resource_manager_next (struct ttm_resource_cursor *cursor) { -struct ttm_lru_item *lru = &res->lru; +struct ttm_resource_manager *man = cursor->man; +struct ttm_lru_item *lru;lockdep_assert_held (&man->bdev->lru_lock); -list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -ttm_lru_item_to_res (lru); +for (;;) { + lru =list_entry (cursor->cur,typeof (*lru), link); +list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) { + cursor->cur = &lru->link; +return ttm_lru_item_to_res (lru); + } } +if (++cursor->priority >=TTM_MAX_BO_PRIORITY ) +break ; + + cursor->cur = &man->lru[cursor->priority]; + } +return NULL ; }
include/drm/ttm/ttm_resource.h +6 -4
@@ -272,11 +272,15 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)/** * struct ttm_resource_cursor * +* @man: The resource manager currently being iterated over. +* @cur: The list head the cursor currently points to. * @priority: the current priority * * Cursor to iterate over the resources in a manager. */ struct ttm_resource_cursor { +struct ttm_resource_manager *man; +struct list_head *cur;unsigned int priority; };
@@ -438,9 +442,7 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor);struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res); +ttm_resource_manager_next (struct ttm_resource_cursor *cursor);struct ttm_resource *ttm_lru_first_res_or_null (struct list_head *head);
@@ -455,7 +457,7 @@ ttm_lru_first_res_or_null(struct list_head *head);*/ #define ttm_resource_manager_for_each_res (man, cursor, res) \ for (res = ttm_resource_manager_first(man, cursor); res; \ - res = ttm_resource_manager_next(man, cursor, res)) + res = ttm_resource_manager_next(cursor))struct ttm_kmap_iter * ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
Signature · 1 line
2.44.0
Re: [PATCH v3 02/21] drm/ttm: Slightly clean up LRU list iteration
Quoted 8 lines · Thomas Hellström
To make the transition to using lru hitches easier, simplify the
ttm_resource_manager_next()interface to only take the cursor and reusettm_resource_manager_next()functionality fromttm_resource_manager_first().Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 126 lines
Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- drivers/gpu/drm/ttm/ttm_resource.c | 48 +++++++++++++----------------- include/drm/ttm/ttm_resource.h | 10 ++++--- 2 files changed, 27 insertions(+), 31 deletions(-)drivers/gpu/drm/ttm/ttm_resource.c +21
-27@@ -496,50 +496,44 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor) { - struct ttm_lru_item *lru; - lockdep_assert_held(&man->bdev->lru_lock); -for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -return NULL ; + cursor->priority =0 ; + cursor->man = man; + cursor->cur = &man->lru[cursor->priority]; +return ttm_resource_manager_next (cursor); }/** * ttm_resource_manager_next * -* @man: resource manager to iterate over * @cursor: cursor to record the position -* @res: the current resource pointer * -* Returns the next resource from the resource manager. +* Return: the next resource from the resource manager. */ struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res) +ttm_resource_manager_next (struct ttm_resource_cursor *cursor) { -struct ttm_lru_item *lru = &res->lru; +struct ttm_resource_manager *man = cursor->man; +struct ttm_lru_item *lru;lockdep_assert_held (&man->bdev->lru_lock); -list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -ttm_lru_item_to_res (lru); +for (;;) { + lru =list_entry (cursor->cur,typeof (*lru), link); +list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) { + cursor->cur = &lru->link; +return ttm_lru_item_to_res (lru); + } } +if (++cursor->priority >=TTM_MAX_BO_PRIORITY ) +break ; + + cursor->cur = &man->lru[cursor->priority]; + } +return NULL ; }include/drm/ttm/ttm_resource.h +6
-4@@ -272,11 +272,15 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)/** * struct ttm_resource_cursor * +* @man: The resource manager currently being iterated over. +* @cur: The list head the cursor currently points to. * @priority: the current priority * * Cursor to iterate over the resources in a manager. */ struct ttm_resource_cursor { +struct ttm_resource_manager *man; +struct list_head *cur;unsigned int priority; };@@ -438,9 +442,7 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor);struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res); +ttm_resource_manager_next (struct ttm_resource_cursor *cursor);struct ttm_resource *ttm_lru_first_res_or_null (struct list_head *head);@@ -455,7 +457,7 @@ ttm_lru_first_res_or_null(struct list_head *head);*/ #define ttm_resource_manager_for_each_res (man, cursor, res) \ for (res = ttm_resource_manager_first(man, cursor); res; \ - res = ttm_resource_manager_next(man, cursor, res)) + res = ttm_resource_manager_next(cursor))struct ttm_kmap_iter * ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,Signature · 1 line
2.44.0
Re: [PATCH v3 02/21] drm/ttm: Slightly clean up LRU list iteration
In reply to Thomas HellströmAm 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 10 lines
To make the transition to using lru hitches easier, simplify the
ttm_resource_manager_next()interface to only take the cursor and reusettm_resource_manager_next()functionality fromttm_resource_manager_first().Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>
Quoted 121 lines
--- drivers/gpu/drm/ttm/ttm_resource.c | 48 +++++++++++++----------------- include/drm/ttm/ttm_resource.h | 10 ++++--- 2 files changed, 27 insertions(+), 31 deletions(-)drivers/gpu/drm/ttm/ttm_resource.c +21
-27@@ -496,50 +496,44 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor) { - struct ttm_lru_item *lru; - lockdep_assert_held(&man->bdev->lru_lock); -for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -return NULL ; + cursor->priority =0 ; + cursor->man = man; + cursor->cur = &man->lru[cursor->priority]; +return ttm_resource_manager_next (cursor); }/** * ttm_resource_manager_next * -* @man: resource manager to iterate over * @cursor: cursor to record the position -* @res: the current resource pointer * -* Returns the next resource from the resource manager. +* Return: the next resource from the resource manager. */ struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res) +ttm_resource_manager_next (struct ttm_resource_cursor *cursor) { -struct ttm_lru_item *lru = &res->lru; +struct ttm_resource_manager *man = cursor->man; +struct ttm_lru_item *lru;lockdep_assert_held (&man->bdev->lru_lock); -list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -return ttm_lru_item_to_res (lru); - } - -for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; - ++cursor->priority) -list_for_each_entry (lru, &man->lru[cursor->priority], link) { -if (ttm_lru_item_is_res (lru)) -ttm_lru_item_to_res (lru); +for (;;) { + lru =list_entry (cursor->cur,typeof (*lru), link); +list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) { +if (ttm_lru_item_is_res (lru)) { + cursor->cur = &lru->link; +return ttm_lru_item_to_res (lru); + } } +if (++cursor->priority >=TTM_MAX_BO_PRIORITY ) +break ; + + cursor->cur = &man->lru[cursor->priority]; + } +return NULL ; }include/drm/ttm/ttm_resource.h +6
-4@@ -272,11 +272,15 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)/** * struct ttm_resource_cursor * +* @man: The resource manager currently being iterated over. +* @cur: The list head the cursor currently points to. * @priority: the current priority * * Cursor to iterate over the resources in a manager. */ struct ttm_resource_cursor { +struct ttm_resource_manager *man; +struct list_head *cur;unsigned int priority; };@@ -438,9 +442,7 @@ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man, struct ttm_resource_cursor *cursor);struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res); +ttm_resource_manager_next (struct ttm_resource_cursor *cursor);struct ttm_resource *ttm_lru_first_res_or_null (struct list_head *head);@@ -455,7 +457,7 @@ ttm_lru_first_res_or_null(struct list_head *head);*/ #define ttm_resource_manager_for_each_res (man, cursor, res) \ for (res = ttm_resource_manager_first(man, cursor); res; \ - res = ttm_resource_manager_next(man, cursor, res)) + res = ttm_resource_manager_next(cursor))struct ttm_kmap_iter * ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
Thomas Hellström
maintainerPATCH v3 3/21 drm/ttm: Use LRU hitches
In reply to Thomas HellströmHave iterators insert themselves into the list they are iterating over using hitch list nodes. Since only the iterator owner can remove these list nodes from the list, it's safe to unlock the list and when continuing, use them as a starting point. Due to the way LRU bumping works in TTM, newly added items will not be missed, and bumped items will be iterated over a second time before reaching the end of the list.
The exception is list with bulk move sublists. When bumping a sublist, a hitch that is part of that sublist will also be moved and we might miss items if restarting from it. This will be addressed in a later patch.
Changes in previous series:
- Updated ttm_resource_cursor_fini() documentation. v2:
- Don't reorder ttm_resource_manager_first() and _next(). (Christian König).
- Use list_add instead of list_move
(Christian König)
v3:
- Split into two patches, one cleanup, one new functionality (Christian König)
- use ttm_resource_cursor_fini_locked() instead of open-coding (Matthew Brost)
--- drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_device.c | 9 +++-- drivers/gpu/drm/ttm/ttm_resource.c | 56 +++++++++++++++++++++++++----- include/drm/ttm/ttm_resource.h | 9 +++-- 4 files changed, 62 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
@@ -621,6 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,if (locked)dma_resv_unlock (res->bo->base.resv); } +ttm_resource_cursor_fini_locked (&cursor);if (!bo) {if (busy_bo && !ttm_bo_get_unless_zero (busy_bo))
drivers/gpu/drm/ttm/ttm_device.c +7 -2
@@ -170,12 +170,17 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, num_pages =PFN_UP (bo->base.size); ret =ttm_bo_swapout (bo, ctx, gfp_flags);/* ttm_bo_swapout has dropped the lru_lock */ -if (!ret) +if (!ret) { +ttm_resource_cursor_fini (&cursor);return num_pages; -if (ret != -EBUSY ) + } +if (ret != -EBUSY ) { +ttm_resource_cursor_fini (&cursor);return ret; + } } } +ttm_resource_cursor_fini_locked (&cursor);spin_unlock (&bdev->lru_lock);return 0 ; }
drivers/gpu/drm/ttm/ttm_resource.c +47 -9
97 lines not on this page · raw message
include/drm/ttm/ttm_resource.h +7 -2
@@ -273,17 +273,22 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)* struct ttm_resource_cursor * * @man: The resource manager currently being iterated over. -* @cur: The list head the cursor currently points to. +* @hitch: A hitch list node inserted before the next resource +* to iterate over. * @priority: the current priority * * Cursor to iterate over the resources in a manager. */ struct ttm_resource_cursor {struct ttm_resource_manager *man; -struct list_head *cur; +struct ttm_lru_item hitch;unsigned int priority; }; +void ttm_resource_cursor_fini_locked (struct ttm_resource_cursor *cursor); + +void ttm_resource_cursor_fini (struct ttm_resource_cursor *cursor); +/** * struct ttm_lru_bulk_move_pos *
Signature · 1 line
2.44.0
Re: [PATCH v3 03/21] drm/ttm: Use LRU hitches
Quoted 26 lines · Thomas Hellström
Have iterators insert themselves into the list they are iterating over using hitch list nodes. Since only the iterator owner can remove these list nodes from the list, it's safe to unlock the list and when continuing, use them as a starting point. Due to the way LRU bumping works in TTM, newly added items will not be missed, and bumped items will be iterated over a second time before reaching the end of the list.
The exception is list with bulk move sublists. When bumping a sublist, a hitch that is part of that sublist will also be moved and we might miss items if restarting from it. This will be addressed in a later patch.
Changes in previous series:
- Updatedttm_resource_cursor_fini()documentation. v2:
- Don't reorderttm_resource_manager_first()and_next(). (Christian König).
- Uselist_addinstead oflist_move
(Christian König)
v3:
- Split into two patches, one cleanup, one new functionality (Christian König)
- usettm_resource_cursor_fini_locked()instead of open-coding (Matthew Brost)
Quoted 3 lines
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 183 lines
Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- drivers/gpu/drm/ttm/ttm_bo.c | 1 + drivers/gpu/drm/ttm/ttm_device.c | 9 +++-- drivers/gpu/drm/ttm/ttm_resource.c | 56 +++++++++++++++++++++++++----- include/drm/ttm/ttm_resource.h | 9 +++-- 4 files changed, 62 insertions(+), 13 deletions(-)drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -621,6 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,if (locked)dma_resv_unlock (res->bo->base.resv); } +ttm_resource_cursor_fini_locked (&cursor);if (!bo) {if (busy_bo && !ttm_bo_get_unless_zero (busy_bo))drivers/gpu/drm/ttm/ttm_device.c +7
-2@@ -170,12 +170,17 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, num_pages =PFN_UP (bo->base.size); ret =ttm_bo_swapout (bo, ctx, gfp_flags);/* ttm_bo_swapout has dropped the lru_lock */ -if (!ret) +if (!ret) { +ttm_resource_cursor_fini (&cursor);return num_pages; -if (ret != -EBUSY ) + } +if (ret != -EBUSY ) { +ttm_resource_cursor_fini (&cursor);return ret; + } } } +ttm_resource_cursor_fini_locked (&cursor);spin_unlock (&bdev->lru_lock);return 0 ; }drivers/gpu/drm/ttm/ttm_resource.c +47
-9@@ -33,6 +33,37 @@#include <drm/drm_util.h> +/** +* ttm_resource_cursor_fini_locked() - Finalize the LRU list cursor usage +* @cursor: The struct ttm_resource_cursor to finalize. +* +* The function pulls the LRU list cursor off any lists it was previusly +* attached to. Needs to be called with the LRU lock held. The function +* can be called multiple times after eachother. +*/ +void ttm_resource_cursor_fini_locked (struct ttm_resource_cursor *cursor) +{ +lockdep_assert_held (&cursor->man->bdev->lru_lock); +list_del_init (&cursor->hitch.link); +} + +/** +* ttm_resource_cursor_fini() - Finalize the LRU list cursor usage +* @cursor: The struct ttm_resource_cursor to finalize. +* +* The function pulls the LRU list cursor off any lists it was previusly +* attached to. Needs to be called without the LRU list lock held. The +* function can be called multiple times after eachother. +*/ +void ttm_resource_cursor_fini (struct ttm_resource_cursor *cursor) +{ +spinlock_t *lru_lock = &cursor->man->bdev->lru_lock; + +spin_lock (lru_lock); +ttm_resource_cursor_fini_locked (cursor); +spin_unlock (lru_lock); +} +/** * ttm_lru_bulk_move_init - initialize a bulk move structure * @bulk: the structure to init @@ -485,12 +516,15 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,EXPORT_SYMBOL (ttm_resource_manager_debug);/** -* ttm_resource_manager_first -* +* ttm_resource_manager_first() - Start iterating over the resources +* of a resource manager * @man: resource manager to iterate over * @cursor: cursor to record the position * -* Returns the first resource from the resource manager. +* Initializes the cursor and starts iterating. When done iterating, +* the caller must explicitly call ttm_resource_cursor_fini(). +* +* Return: The first resource from the resource manager. */ struct ttm_resource * ttm_resource_manager_first(struct ttm_resource_manager *man,@@ -500,13 +534,15 @@ ttm_resource_manager_first(struct ttm_resource_manager *man, cursor->priority =0 ; cursor->man = man; - cursor->cur = &man->lru[cursor->priority]; +ttm_lru_item_init (&cursor->hitch,TTM_LRU_HITCH ); +list_add (&cursor->hitch.link, &man->lru[cursor->priority]); +return ttm_resource_manager_next (cursor); }/** -* ttm_resource_manager_next -* +* ttm_resource_manager_next() - Continue iterating over the resource manager +* resources * @cursor: cursor to record the position * * Return: the next resource from the resource manager. @@ -520,10 +556,10 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)lockdep_assert_held (&man->bdev->lru_lock);for (;;) { - lru =list_entry (cursor->cur,typeof (*lru), link); + lru = &cursor->hitch;list_for_each_entry_continue (lru, &man->lru[cursor->priority], link) {if (ttm_lru_item_is_res (lru)) { - cursor->cur = &lru->link; +list_move (&cursor->hitch.link, &lru->link);return ttm_lru_item_to_res (lru); } }@@ -531,9 +567,11 @@ ttm_resource_manager_next(struct ttm_resource_cursor *cursor)if (++cursor->priority >=TTM_MAX_BO_PRIORITY )break ; - cursor->cur = &man->lru[cursor->priority]; +list_move (&cursor->hitch.link, &man->lru[cursor->priority]); } +ttm_resource_cursor_fini_locked (cursor); +return NULL ; }include/drm/ttm/ttm_resource.h +7
-2@@ -273,17 +273,22 @@ ttm_lru_item_to_res(struct ttm_lru_item *item)* struct ttm_resource_cursor * * @man: The resource manager currently being iterated over. -* @cur: The list head the cursor currently points to. +* @hitch: A hitch list node inserted before the next resource +* to iterate over. * @priority: the current priority * * Cursor to iterate over the resources in a manager. */ struct ttm_resource_cursor {struct ttm_resource_manager *man; -struct list_head *cur; +struct ttm_lru_item hitch;unsigned int priority; }; +void ttm_resource_cursor_fini_locked (struct ttm_resource_cursor *cursor); + +void ttm_resource_cursor_fini (struct ttm_resource_cursor *cursor); +/** * struct ttm_lru_bulk_move_pos * Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 4/21 drm/ttm, drm/amdgpu, drm/xe: Consider hitch moves within bulk sublist moves
In reply to Thomas HellströmTo address the problem with hitches moving when bulk move sublists are lru-bumped, register the list cursors with the ttm_lru_bulk_move structure when traversing its list, and when lru-bumping the list, move the cursor hitch to the tail. This also means it's mandatory for drivers to call ttm_lru_bulk_move_init() and ttm_lru_bulk_move_fini() when initializing and finalizing the bulk move structure, so add those calls to the amdgpu- and xe driver.
Compared to v1 this is slightly more code but less fragile and hopefully easier to understand.
Changes in previous series:
- Completely rework the functionality
- Avoid a NULL pointer dereference assigning manager->mem_type
- Remove some leftover code causing build problems v2:
- For hitch bulk tail moves, store the mem_type in the cursor instead of with the manager.
v3:
- Remove leftover mem_type member from change in v2.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 ++ drivers/gpu/drm/ttm/ttm_resource.c | 89 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.c | 4 ++ include/drm/ttm/ttm_resource.h | 56 ++++++++++------ 4 files changed, 132 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +4 -0
@@ -2422,6 +2422,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,if (r)return r; +ttm_lru_bulk_move_init (&vm->lru_bulk_move); + vm->is_compute_context = false; vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
@@ -2486,6 +2488,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, error_free_delayed:dma_fence_put (vm->last_tlb_flush);dma_fence_put (vm->last_unlocked); +ttm_lru_bulk_move_fini (&adev->mman.bdev, &vm->lru_bulk_move);amdgpu_vm_fini_entities (vm);return r;
@@ -2642,6 +2645,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) } } +ttm_lru_bulk_move_fini (&adev->mman.bdev, &vm->lru_bulk_move); }/**
drivers/gpu/drm/ttm/ttm_resource.c +89 -0
140 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.c +4 -0
@@ -1339,6 +1339,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)INIT_WORK (&vm->destroy_work, vm_destroy_work_func); +ttm_lru_bulk_move_init (&vm->lru_bulk_move); +INIT_LIST_HEAD (&vm->preempt.exec_queues); vm->preempt.min_run_period_ms =10 ;/* FIXME: Wire up to uAPI */
@@ -1456,6 +1458,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)mutex_destroy (&vm->snap_mutex);for_each_tile (tile, xe, id)xe_range_fence_tree_fini (&vm->rftree[id]); +ttm_lru_bulk_move_fini (&xe->ttm, &vm->lru_bulk_move);kfree (vm);if (!(flags &XE_VM_FLAG_MIGRATION ))xe_pm_runtime_put (xe);
@@ -1599,6 +1602,7 @@ static void vm_destroy_work_func(struct work_struct *w)XE_WARN_ON (vm->pt_root[id]);trace_xe_vm_free (vm); +ttm_lru_bulk_move_fini (&xe->ttm, &vm->lru_bulk_move);kfree (vm); }
include/drm/ttm/ttm_resource.h +35 -21
83 lines not on this page · raw message
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 5/21 drm/ttm: Provide a generic LRU walker helper
In reply to Thomas HellströmProvide a generic LRU walker in TTM, in the spirit of drm_gem_lru_scan() but building on the restartable TTM LRU functionality.
The LRU walker optionally supports locking objects as part of a ww mutex locking transaction, to mimic to some extent the current functionality in ttm. However any -EDEADLK return is converted to -ENOMEM, so that the driver will need to back off and possibly retry without being able to keep the ticket.
v3:
- Move the helper to core ttm.
- Remove the drm_exec usage from it for now, it will be reintroduced later in the series.
--- drivers/gpu/drm/ttm/ttm_bo_util.c | 145 ++++++++++++++++++++++++++++++ include/drm/ttm/ttm_bo.h | 32 +++++++ 2 files changed, 177 insertions(+)
drivers/gpu/drm/ttm/ttm_bo_util.c +145 -0
@@ -768,3 +768,148 @@ int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)ttm_tt_destroy (bo->bdev, ttm);return ret; } + +static bool ttm_lru_walk_trylock (struct ttm_lru_walk *walk, +struct ttm_buffer_object *bo, +bool *needs_unlock) +{ +struct ttm_operation_ctx *ctx = walk->ctx; + + *needs_unlock = false; + +if (dma_resv_trylock (bo->base.resv)) { + *needs_unlock = true; +return true; + } + +if (bo->base.resv == ctx->resv && ctx->allow_res_evict) { +dma_resv_assert_held (bo->base.resv); +return true; + } + +return false; +} + +static int ttm_lru_walk_ticketlock (struct ttm_lru_walk *walk, +struct ttm_buffer_object *bo, +bool *needs_unlock) +{ +struct dma_resv *resv = bo->base.resv; +int ret; + +if (walk->ctx->interruptible) + ret =dma_resv_lock_interruptible (resv, walk->ticket); +else + ret =dma_resv_lock (resv, walk->ticket); + +if (ret == -EDEADLK ) + ret = -ENOSPC ; + +if (!ret) { + *needs_unlock = true; +/* Only a single ticketlock per loop */ + walk->ticket =NULL ; + } + +return ret; +} + +static void ttm_lru_walk_unlock (struct ttm_buffer_object *bo,bool locked) +{ +if (locked) +dma_resv_unlock (bo->base.resv); +} + +/** +* ttm_lru_walk_for_evict() - Perform a LRU list walk, with actions taken on +* valid items. +* @walk: describe the walks and actions taken +* @bdev: The TTM device. +* @man: The struct ttm_resource manager whose LRU lists we're walking. +* @target: The end condition for the walk. +* +* The LRU lists of @man are walk, and for each struct ttm_resource encountered, +* the corresponding ttm_buffer_object is locked and taken a reference on, and +* the LRU lock is dropped. the LRU lock may be dropped before locking and, in +* that case, it's verified that the item actually remains on the LRU list after +* the lock, and that the buffer object didn't switch resource in between. +* +* With a locked object, the actions indicated by @walk->process_bo are +* performed, and after that, the bo is unlocked, the refcount dropped and the +* next struct ttm_resource is processed. Here, the walker relies on +* TTM's restartable LRU list implementation. +* +* Typically @walk->process_bo() would return the number of pages evicted, +* swapped or shrunken, so that when the total exceeds @target, or when the +* LRU list has been walked in full, iteration is terminated. It's also terminated +* on error. Note that the definition of @target is done by the caller, it +* could have a different meaning than the number of pages. +* +* Note that the way dma_resv individualization is done, locking needs to be done +* either with the LRU lock held (trylocking only) or with a reference on the +* object. +* +* Return: The progress made towards target or negative error code on error. +*/ +long ttm_lru_walk_for_evict (struct ttm_lru_walk *walk,struct ttm_device *bdev, +struct ttm_resource_manager *man,long target) +{ +struct ttm_resource_cursor cursor; +struct ttm_resource *res; +long sofar =0 ; +long lret; + +spin_lock (&bdev->lru_lock); +ttm_resource_manager_for_each_res (man, &cursor, res) { +struct ttm_buffer_object *bo = res->bo; +bool bo_needs_unlock = false; +bool bo_locked = false; +int mem_type; + +if (!bo || bo->resource != res) +continue ; + +if (ttm_lru_walk_trylock (walk, bo, &bo_needs_unlock)) + bo_locked = true; +else if ((!walk->ticket) || walk->ctx->no_wait_gpu || + walk->trylock_only) +continue ; + +if (!ttm_bo_get_unless_zero (bo)) { +ttm_lru_walk_unlock (bo, bo_needs_unlock); +continue ; + } + + mem_type = res->mem_type; +spin_unlock (&bdev->lru_lock); + + lret =0 ; +if (!bo_locked && walk->ticket) + lret =ttm_lru_walk_ticketlock (walk, bo, &bo_needs_unlock); + +/* +* Note that in between the release of the lru lock and the +* ticketlock, the bo may have switched resource, +* and also memory type, since the resource may have been +* freed and allocated again with a different memory type. +* In that case, just skip it. +*/ +if (!lret && bo->resource == res && res->mem_type == mem_type) + lret = walk->ops->process_bo (walk, bo); + +ttm_lru_walk_unlock (bo, bo_needs_unlock); +ttm_bo_put (bo); +if (lret == -EBUSY ) + lret =0 ; + sofar = (lret <0 ) ? lret : sofar + lret; +if (sofar <0 || sofar >= target) + goto out; + +cond_resched (); +spin_lock (&bdev->lru_lock); + } +spin_unlock (&bdev->lru_lock); +out: +ttm_resource_cursor_fini (&cursor); +return sofar; +}
include/drm/ttm/ttm_bo.h +32 -0
@@ -190,6 +190,38 @@ struct ttm_operation_ctx {uint64_t bytes_moved; }; +struct ttm_lru_walk ; + +/** struct ttm_lru_walk_ops - Operations for a LRU walk. */ +struct ttm_lru_walk_ops { +/** +* process_bo - Process this bo. +* @walk: struct ttm_lru_walk describing the walk. +* @bo: A locked and referenced buffer object. +* +* Return: Negative error code on error, Number of processed pages on +* success. 0 also indicates success. +*/ +long (*process_bo)(struct ttm_lru_walk *walk,struct ttm_buffer_object *bo); +}; + +/** +* struct ttm_lru_walk - Structure describing a LRU walk. +*/ +struct ttm_lru_walk { +/** @ops: Pointer to the ops structure. */ +const struct ttm_lru_walk_ops *ops; +/** @ctx: Pointer to the struct ttm_operation_ctx. */ +struct ttm_operation_ctx *ctx; +/** @ticket: The struct ww_acquire_ctx if any. */ +struct ww_acquire_ctx *ticket; +/** @tryock_only: Only use trylock for locking. */ +bool trylock_only; +}; + +long ttm_lru_walk_for_evict (struct ttm_lru_walk *walk,struct ttm_device *bdev, +struct ttm_resource_manager *man,long target); +/** * ttm_bo_get - reference a struct ttm_buffer_object *
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 6/21 drm/ttm: Use the LRU walker helper for swapping
In reply to Thomas HellströmRework the TTM swapping to use the LRU walker helper. This helps fixing up the ttm_bo_swapout() interface to be consistent about not requiring any locking.
For now mimic the current behaviour of using trylock only. We could be using ticket-locks here but defer that until it's deemed necessary. The TTM swapout functionality is a bit weird anyway since it alternates between memory types without exhausting TTM_PL_SYSTEM first.
--- drivers/gpu/drm/ttm/ttm_bo.c | 112 +++++++++++++++++++++---------- drivers/gpu/drm/ttm/ttm_device.c | 30 ++------- include/drm/ttm/ttm_bo.h | 5 +- 3 files changed, 83 insertions(+), 64 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +75 -37
@@ -1118,11 +1118,23 @@ int ttm_bo_wait_ctx(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx) }EXPORT_SYMBOL (ttm_bo_wait_ctx); -int ttm_bo_swapout (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx, -gfp_t gfp_flags) +/** +* struct ttm_bo_swapout_walk - Parameters for the swapout walk +*/ +struct ttm_bo_swapout_walk { +/** @walk: The walk base parameters. */ +struct ttm_lru_walk walk; +/** @gfp_flags: The gfp flags to use for ttm_tt_swapout() */ +gfp_t gfp_flags; +}; + +static long +ttm_bo_swapout_cb (struct ttm_lru_walk *walk,struct ttm_buffer_object *bo) { -struct ttm_place place; -bool locked; +struct ttm_place place = {.mem_type = bo->resource->mem_type}; +struct ttm_bo_swapout_walk *swapout_walk = +container_of (walk,typeof (*swapout_walk), walk); +struct ttm_operation_ctx *ctx = walk->ctx;long ret;/*
@@ -1131,28 +1143,29 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,* The driver may use the fact that we're moving from SYSTEM * as an indication that we're about to swap out. */ -memset (&place,0 ,sizeof (place)); - place.mem_type = bo->resource->mem_type; -if (!ttm_bo_evict_swapout_allowable (bo, ctx, &place, &locked,NULL )) -return -EBUSY ; +if (!bo->bdev->funcs->eviction_valuable (bo, &place)) { + ret = -EBUSY ; + goto out; + }if (!bo->ttm || !ttm_tt_is_populated (bo->ttm) || bo->ttm->page_flags &TTM_TT_FLAG_EXTERNAL || - bo->ttm->page_flags &TTM_TT_FLAG_SWAPPED || - !ttm_bo_get_unless_zero (bo)) { -if (locked) -dma_resv_unlock (bo->base.resv); -return -EBUSY ; + bo->ttm->page_flags &TTM_TT_FLAG_SWAPPED ) { + ret = -EBUSY ; + goto out; }if (bo->deleted) { - ret =ttm_bo_cleanup_refs (bo, false, false, locked); -ttm_bo_put (bo); -return ret == -EBUSY ? -ENOSPC : ret; - } +pgoff_t num_pages = bo->ttm->num_pages; -/* TODO: Cleanup the locking */ -spin_unlock (&bo->bdev->lru_lock); + ret =ttm_bo_wait_ctx (bo, ctx); +if (ret) + goto out; + +ttm_bo_cleanup_memtype_use (bo); + ret = num_pages; + goto out; + }/* * Move to system cached
@@ -1164,12 +1177,13 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,memset (&hop,0 ,sizeof (hop)); place.mem_type =TTM_PL_SYSTEM ; ret =ttm_resource_alloc (bo, &place, &evict_mem); -if (unlikely (ret)) +if (ret) goto out; ret =ttm_bo_handle_move_mem (bo, evict_mem, true, ctx, &hop); -if (unlikely (ret !=0 )) { -WARN (ret == -EMULTIHOP ,"Unexpected multihop in swaput - likely driver bug.\n" ); +if (ret) { +WARN (ret == -EMULTIHOP , +"Unexpected multihop in swapout - likely driver bug.\n" );ttm_resource_free (bo, &evict_mem); goto out; }
@@ -1179,30 +1193,54 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,* Make sure BO is idle. */ ret =ttm_bo_wait_ctx (bo, ctx); -if (unlikely (ret !=0 )) +if (ret) goto out;ttm_bo_unmap_virtual (bo); - -/* -* Swap out. Buffer will be swapped in again as soon as -* anyone tries to access a ttm page. -*/ if (bo->bdev->funcs->swap_notify) bo->bdev->funcs->swap_notify (bo);if (ttm_tt_is_populated (bo->ttm)) - ret =ttm_tt_swapout (bo->bdev, bo->ttm, gfp_flags); + ret =ttm_tt_swapout (bo->bdev, bo->ttm, swapout_walk->gfp_flags); out: +/* Consider some error codes fatal. Others may continue the walk. */ +if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS || + ret == -EAGAIN || ret >0 ) +return ret; -/* -* Unreserve without putting on LRU to avoid swapping out an -* already swapped buffer. -*/ -if (locked) -dma_resv_unlock (bo->base.resv); -ttm_bo_put (bo); -return ret == -EBUSY ? -ENOSPC : ret; +return 0 ; +} + +const struct ttm_lru_walk_ops ttm_swap_ops = { + .process_bo = ttm_bo_swapout_cb, +}; + +/** +* ttm_bo_swapout() - Swap out buffer objects on the LRU list to shmem. +* @bdev: The ttm device. +* @ctx: The ttm_operation_ctx governing the swapout operation. +* @man: The resource manager whose resources / buffer objects are +* goint to be swapped out. +* @gfp_flags: The gfp flags used for shmem page allocations. +* @target: The desired number of pages to swap out. +* +* Return: The number of pages actually swapped out, or negative error code +* on error. +*/ +long ttm_bo_swapout (struct ttm_device *bdev,struct ttm_operation_ctx *ctx, +struct ttm_resource_manager *man,gfp_t gfp_flags, +pgoff_t target) +{ +struct ttm_bo_swapout_walk swapout_walk = { + .walk = { + .ops = &ttm_swap_ops, + .ctx = ctx, + .trylock_only = true, + }, + .gfp_flags = gfp_flags, + }; + +return ttm_lru_walk_for_evict (&swapout_walk.walk, bdev, man, target); }void ttm_bo_tt_destroy (struct ttm_buffer_object *bo)
drivers/gpu/drm/ttm/ttm_device.c +5 -25
@@ -148,40 +148,20 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags)int ttm_device_swapout (struct ttm_device *bdev,struct ttm_operation_ctx *ctx,gfp_t gfp_flags) { -struct ttm_resource_cursor cursor;struct ttm_resource_manager *man; -struct ttm_resource *res;unsigned i; -int ret; +long lret; -spin_lock (&bdev->lru_lock);for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)continue ; -ttm_resource_manager_for_each_res (man, &cursor, res) { -struct ttm_buffer_object *bo = res->bo; -uint32_t num_pages; - -if (!bo || bo->resource != res) -continue ; - - num_pages =PFN_UP (bo->base.size); - ret =ttm_bo_swapout (bo, ctx, gfp_flags); -/* ttm_bo_swapout has dropped the lru_lock */ -if (!ret) { -ttm_resource_cursor_fini (&cursor); -return num_pages; - } -if (ret != -EBUSY ) { -ttm_resource_cursor_fini (&cursor); -return ret; - } - } + lret =ttm_bo_swapout (bdev, ctx, man, gfp_flags,1 ); +/* Can be both positive (num_pages) and negative (error) */ +if (lret) +return lret; } -ttm_resource_cursor_fini_locked (&cursor); -spin_unlock (&bdev->lru_lock);return 0 ; }EXPORT_SYMBOL (ttm_device_swapout);
include/drm/ttm/ttm_bo.h +3 -2
@@ -410,8 +410,9 @@ void ttm_bo_kunmap(struct ttm_bo_kmap_obj *map);int ttm_bo_vmap (struct ttm_buffer_object *bo,struct iosys_map *map);void ttm_bo_vunmap (struct ttm_buffer_object *bo,struct iosys_map *map);int ttm_bo_mmap_obj (struct vm_area_struct *vma,struct ttm_buffer_object *bo); -int ttm_bo_swapout (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx, -gfp_t gfp_flags); +long ttm_bo_swapout (struct ttm_device *bdev,struct ttm_operation_ctx *ctx, +struct ttm_resource_manager *man,gfp_t gfp_flags, +pgoff_t target);void ttm_bo_pin (struct ttm_buffer_object *bo);void ttm_bo_unpin (struct ttm_buffer_object *bo); int ttm_mem_evict_first(struct ttm_device *bdev,
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 7/21 drm/ttm: Use the LRU walker for eviction
In reply to Thomas HellströmUse the LRU walker for eviction. This helps
removing a lot of code with weird locking
semantics.
The functionality is slightly changed so that when trylocked buffer objects are exhausted, we continue to interleave walks with ticket-locks while there is still progress made. The list walks are not restarted in-between evictions.
Also provide a separate ttm_bo_evict_first() function for its single user. The context of that user allows sleeping dma_resv locks.
--- drivers/gpu/drm/ttm/ttm_bo.c | 350 ++++++++++++----------------- drivers/gpu/drm/ttm/ttm_resource.c | 20 +- include/drm/ttm/ttm_bo.h | 8 +- 3 files changed, 145 insertions(+), 233 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +139 -211
@@ -224,80 +224,6 @@ static void ttm_bo_flush_all_fences(struct ttm_buffer_object *bo)dma_resv_iter_end (&cursor); } -/** -* ttm_bo_cleanup_refs -* If bo idle, remove from lru lists, and unref. -* If not idle, block if possible. -* -* Must be called with lru_lock and reservation held, this function -* will drop the lru lock and optionally the reservation lock before returning. -* -* @bo: The buffer object to clean-up -* @interruptible: Any sleeps should occur interruptibly. -* @no_wait_gpu: Never wait for gpu. Return -EBUSY instead. -* @unlock_resv: Unlock the reservation lock as well. -*/ - -static int ttm_bo_cleanup_refs (struct ttm_buffer_object *bo, -bool interruptible,bool no_wait_gpu, -bool unlock_resv) -{ -struct dma_resv *resv = &bo->base._resv; -int ret; - -if (dma_resv_test_signaled (resv,DMA_RESV_USAGE_BOOKKEEP )) - ret =0 ; -else - ret = -EBUSY ; - -if (ret && !no_wait_gpu) { -long lret; - -if (unlock_resv) -dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); - - lret =dma_resv_wait_timeout (resv,DMA_RESV_USAGE_BOOKKEEP , - interruptible, -30 *HZ ); - -if (lret <0 ) -return lret; -else if (lret ==0 ) -return -EBUSY ; - -spin_lock (&bo->bdev->lru_lock); -if (unlock_resv && !dma_resv_trylock (bo->base.resv)) { -/* -* We raced, and lost, someone else holds the reservation now, -* and is probably busy in ttm_bo_cleanup_memtype_use. -* -* Even if it's not the case, because we finished waiting any -* delayed destruction would succeed, so just return success -* here. -*/ -spin_unlock (&bo->bdev->lru_lock); -return 0 ; - } - ret =0 ; - } - -if (ret) { -if (unlock_resv) -dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); -return ret; - } - -spin_unlock (&bo->bdev->lru_lock); -ttm_bo_cleanup_memtype_use (bo); - -if (unlock_resv) -dma_resv_unlock (bo->base.resv); - -return 0 ; -} -/* * Block for the dma_resv object to become idle, lock the buffer and clean up * the resource and tt object.
@@ -505,151 +431,154 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo, }EXPORT_SYMBOL (ttm_bo_eviction_valuable); -/* -* Check the target bo is allowable to be evicted or swapout, including cases: -* -* a. if share same reservation object with ctx->resv, have assumption -* reservation objects should already be locked, so not lock again and -* return true directly when either the opreation allow_reserved_eviction -* or the target bo already is in delayed free list; +/** +* ttm_bo_evict_first() - Evict the first bo on the manager's LRU list. +* @bdev: The ttm device. +* @man: The manager whose bo to evict. +* @ctx: The TTM operation ctx governing the eviction. * -* b. Otherwise, trylock it. +* Return: 0 if successful or the resource disappeared. Negative error code on error. */ -static bool ttm_bo_evict_swapout_allowable (struct ttm_buffer_object *bo, -struct ttm_operation_ctx *ctx, -const struct ttm_place *place, -bool *locked,bool *busy) +int ttm_bo_evict_first (struct ttm_device *bdev,struct ttm_resource_manager *man, +struct ttm_operation_ctx *ctx) { -bool ret = false; +struct ttm_resource_cursor cursor; +struct ttm_buffer_object *bo; +struct ttm_resource *res; +unsigned int mem_type; +int ret =0 ; -if (bo->pin_count) { - *locked = false; -if (busy) - *busy = false; -return false; +spin_lock (&bdev->lru_lock); + res =ttm_resource_manager_first (man, &cursor); +if (!res) { + ret = -ENOENT ; + goto out_no_ref; } + bo = res->bo; +if (!ttm_bo_get_unless_zero (bo)) + goto out_no_ref; + mem_type = res->mem_type; +spin_unlock (&bdev->lru_lock); + ret =ttm_bo_reserve (bo, ctx->interruptible, ctx->no_wait_gpu,NULL ); +if (ret) + goto out_no_lock; +if (bo->resource != res || res->mem_type != mem_type) + goto out_bad_res; -if (bo->base.resv == ctx->resv) { -dma_resv_assert_held (bo->base.resv); -if (ctx->allow_res_evict) - ret = true; - *locked = false; -if (busy) - *busy = false; +if (bo->deleted) { + ret =ttm_bo_wait_ctx (bo, ctx); +if (ret) +ttm_bo_cleanup_memtype_use (bo); }else { - ret =dma_resv_trylock (bo->base.resv); - *locked = ret; -if (busy) - *busy = !ret; - } - -if (ret && place && (bo->resource->mem_type != place->mem_type || - !bo->bdev->funcs->eviction_valuable (bo, place))) { - ret = false; -if (*locked) { -dma_resv_unlock (bo->base.resv); - *locked = false; - } + ret =ttm_bo_evict (bo, ctx); } - +out_bad_res: +dma_resv_unlock (bo->base.resv); +out_no_lock: +ttm_bo_put (bo); +ttm_resource_cursor_fini (&cursor);return ret; + +out_no_ref: +ttm_resource_cursor_fini_locked (&cursor); +spin_unlock (&bdev->lru_lock); +return -ENOENT ; }/** -* ttm_mem_evict_wait_busy - wait for a busy BO to become available -* -* @busy_bo: BO which couldn't be locked with trylock -* @ctx: operation context -* @ticket: acquire ticket -* -* Try to lock a busy buffer object to avoid failing eviction. +* struct ttm_bo_evict_walk - Parameters for the evict walk. */ -static int ttm_mem_evict_wait_busy (struct ttm_buffer_object *busy_bo, -struct ttm_operation_ctx *ctx, -struct ww_acquire_ctx *ticket) -{ -int r; - -if (!busy_bo || !ticket) -return -EBUSY ; - -if (ctx->interruptible) - r =dma_resv_lock_interruptible (busy_bo->base.resv, - ticket); -else - r =dma_resv_lock (busy_bo->base.resv, ticket); - -/* -* TODO: It would be better to keep the BO locked until allocation is at -* least tried one more time, but that would mean a much larger rework -* of TTM. -*/ -if (!r) -dma_resv_unlock (busy_bo->base.resv); - -return r == -EDEADLK ? -EBUSY : r; -} +struct ttm_bo_evict_walk { +/** @walk: The walk base parameters. */ +struct ttm_lru_walk walk; +/** @place: The place passed to the resource allocation. */ +const struct ttm_place *place; +/** @evictor: The buffer object we're trying to make room for. */ +struct ttm_buffer_object *evictor; +/** @res: The allocated resource if any. */ +struct ttm_resource **res; +/** @evicted: The number of evicted pages. */ +unsigned long evicted; +}; -int ttm_mem_evict_first (struct ttm_device *bdev, -struct ttm_resource_manager *man, -const struct ttm_place *place, -struct ttm_operation_ctx *ctx, -struct ww_acquire_ctx *ticket) +static long ttm_bo_evict_cb (struct ttm_lru_walk *walk,struct ttm_buffer_object *bo) { -struct ttm_buffer_object *bo =NULL , *busy_bo =NULL ; -struct ttm_resource_cursor cursor; -struct ttm_resource *res; -bool locked = false; -int ret; +struct ttm_bo_evict_walk *evict_walk = +container_of (walk,typeof (*evict_walk), walk); +long lret; -spin_lock (&bdev->lru_lock); -ttm_resource_manager_for_each_res (man, &cursor, res) { -bool busy; - -if (!ttm_bo_evict_swapout_allowable (res->bo, ctx, place, - &locked, &busy)) { -if (busy && !busy_bo && ticket != -dma_resv_locking_ctx (res->bo->base.resv)) - busy_bo = res->bo; -continue ; - } +if (!bo->bdev->funcs->eviction_valuable (bo, evict_walk->place)) +return 0 ; -if (ttm_bo_get_unless_zero (res->bo)) { - bo = res->bo; -break ; - } -if (locked) -dma_resv_unlock (res->bo->base.resv); +if (bo->deleted) { + lret =ttm_bo_wait_ctx (bo, walk->ctx); +if (!lret) +ttm_bo_cleanup_memtype_use (bo); + }else { + lret =ttm_bo_evict (bo, walk->ctx); } -ttm_resource_cursor_fini_locked (&cursor); -if (!bo) { -if (busy_bo && !ttm_bo_get_unless_zero (busy_bo)) - busy_bo =NULL ; -spin_unlock (&bdev->lru_lock); - ret =ttm_mem_evict_wait_busy (busy_bo, ctx, ticket); -if (busy_bo) -ttm_bo_put (busy_bo); -return ret; - } +if (lret) + goto out; -if (bo->deleted) { - ret =ttm_bo_cleanup_refs (bo, ctx->interruptible, - ctx->no_wait_gpu, locked); -ttm_bo_put (bo); -return ret; - } + evict_walk->evicted++; +if (evict_walk->res) + lret =ttm_resource_alloc (evict_walk->evictor, evict_walk->place, + evict_walk->res); +if (lret ==0 ) +return 1 ; +out: +/* Errors that should terminate the walk. */ +if (lret == -ENOMEM || lret == -EINTR || lret == -ERESTARTSYS || + lret == -EAGAIN ) +return lret; -spin_unlock (&bdev->lru_lock); +return 0 ; +} - ret =ttm_bo_evict (bo, ctx); -if (locked) -ttm_bo_unreserve (bo); -else -ttm_bo_move_to_lru_tail_unlocked (bo); +static const struct ttm_lru_walk_ops ttm_evict_walk_ops = { + .process_bo = ttm_bo_evict_cb, +}; -ttm_bo_put (bo); -return ret; +static int ttm_bo_evict_alloc (struct ttm_device *bdev, +struct ttm_resource_manager *man, +const struct ttm_place *place, +struct ttm_buffer_object *evictor, +struct ttm_operation_ctx *ctx, +struct ww_acquire_ctx *ticket, +struct ttm_resource **res) +{ +struct ttm_bo_evict_walk evict_walk = { + .walk = { + .ops = &ttm_evict_walk_ops, + .ctx = ctx, + .ticket = ticket, + }, + .place = place, + .evictor = evictor, + .res = res, + }; +long lret; + + evict_walk.walk.trylock_only = true; + lret =ttm_lru_walk_for_evict (&evict_walk.walk, bdev, man,1 ); +if (lret || !ticket) + goto out; + +/* If ticket-locking, repeat while making progress. */ + evict_walk.walk.trylock_only = false; +do { +/* The walk may clear the evict_walk.walk.ticket field */ + evict_walk.walk.ticket = ticket; + evict_walk.evicted =0 ; + lret =ttm_lru_walk_for_evict (&evict_walk.walk, bdev, man,1 ); + }while (!lret && evict_walk.evicted); +out: +if (lret <0 ) +return lret; +if (lret ==0 ) +return -EBUSY ; +return 0 ; }/**
@@ -760,6 +689,7 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo,for (i =0 ; i < placement->num_placement; ++i) {const struct ttm_place *place = &placement->placement[i];struct ttm_resource_manager *man; +bool may_evict; man =ttm_manager_type (bdev, place->mem_type);if (!man || !ttm_resource_manager_used (man))
@@ -769,22 +699,21 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo, TTM_PL_FLAG_FALLBACK))continue ; -do { - ret =ttm_resource_alloc (bo, place, res); -if (unlikely (ret && ret != -ENOSPC )) + may_evict = (force_space && place->mem_type !=TTM_PL_SYSTEM ); + ret =ttm_resource_alloc (bo, place, res); +if (ret) { +if (ret != -ENOSPC )return ret; -if (likely (!ret) || !force_space) -break ; - - ret =ttm_mem_evict_first (bdev, man, place, ctx, - ticket); -if (unlikely (ret == -EBUSY )) -break ; -if (unlikely (ret)) +if (!may_evict) +continue ; + + ret =ttm_bo_evict_alloc (bdev, man, place, bo, ctx, + ticket, res); +if (ret == -EBUSY ) +continue ; +if (ret)return ret; - }while (1 ); -if (ret) -continue ; + } ret =ttm_bo_add_move_fence (bo, man, ctx->no_wait_gpu);if (unlikely (ret)) {
@@ -796,7 +725,6 @@ static int ttm_bo_alloc_resource(struct ttm_buffer_object *bo, }return 0 ; } -return -ENOSPC ; }
drivers/gpu/drm/ttm/ttm_resource.c +3 -17
@@ -508,24 +508,10 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, };struct dma_fence *fence;int ret; -unsigned i; - -/* -* Can't use standard list traversal since we're unlocking. -*/ -spin_lock (&bdev->lru_lock); -for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) { -while (!list_empty (&man->lru[i])) { -spin_unlock (&bdev->lru_lock); - ret =ttm_mem_evict_first (bdev, man,NULL , &ctx, -NULL ); -if (ret) -return ret; -spin_lock (&bdev->lru_lock); - } - } -spin_unlock (&bdev->lru_lock); +do { + ret =ttm_bo_evict_first (bdev, man, &ctx); + }while (!ret);spin_lock (&man->move_lock); fence =dma_fence_get (man->move);
include/drm/ttm/ttm_bo.h +3 -5
@@ -415,11 +415,9 @@ long ttm_bo_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, pgoff_t target);void ttm_bo_pin (struct ttm_buffer_object *bo);void ttm_bo_unpin (struct ttm_buffer_object *bo); -int ttm_mem_evict_first (struct ttm_device *bdev, -struct ttm_resource_manager *man, -const struct ttm_place *place, -struct ttm_operation_ctx *ctx, -struct ww_acquire_ctx *ticket); +int ttm_bo_evict_first (struct ttm_device *bdev, +struct ttm_resource_manager *man, +struct ttm_operation_ctx *ctx);vm_fault_t ttm_bo_vm_reserve (struct ttm_buffer_object *bo,struct vm_fault *vmf); vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 8/21 drm/ttm: Add a virtual base class for graphics memory backup
In reply to Thomas HellströmInitially intended for experimenting with different backup solutions (shmem vs direct swap cache insertion), abstract the backup destination using a virtual base class.
Also provide a sample implementation for shmem.
While when settling on a preferred backup solution, one could perhaps skip the abstraction, this functionality may actually come in handy for configurable dedicated graphics memory backup to fast nvme files or similar, whithout affecting swap-space. Could indeed be useful for VRAM backup on S4 and other cases.
--- drivers/gpu/drm/ttm/Makefile | 2 +- drivers/gpu/drm/ttm/ttm_backup_shmem.c | 137 +++++++++++++++++++++++++ include/drm/ttm/ttm_backup.h | 136 ++++++++++++++++++++++++ 3 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/ttm/ttm_backup_shmem.c create mode 100644 include/drm/ttm/ttm_backup.h
drivers/gpu/drm/ttm/Makefile +1 -1
@@ -4,7 +4,7 @@ ttm-y := ttm_tt.o ttm_bo.o ttm_bo_util.o ttm_bo_vm.o ttm_module.o \ ttm_execbuf_util.o ttm_range_manager.o ttm_resource.o ttm_pool.o \ - ttm_device.o ttm_sys_manager.o + ttm_device.o ttm_sys_manager.o ttm_backup_shmem.o ttm-$(CONFIG_AGP) += ttm_agp_backend.o obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_backup_shmem.c new file +137 -0
137 lines not on this page · raw message
include/drm/ttm/ttm_backup.h new file +136 -0
136 lines not on this page · raw message
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 9/21 drm/ttm/pool: Provide a helper to shrink pages
In reply to Thomas HellströmProvide a helper to shrink ttm_tt page-vectors on a per-page basis. A ttm_backup backend could then in theory get away with allocating a single temporary page for each struct ttm_tt.
This is accomplished by splitting larger pages before trying to back them up.
In the future we could allow ttm_backup to handle backing up large pages as well, but currently there's no benefit in doing that, since the shmem backup backend would have to split those anyway to avoid allocating too much temporary memory, and if the backend instead inserts pages into the swap-cache, those are split on reclaim by the core.
Due to potential backup- and recover errors, allow partially swapped out struct ttm_tt's, although mark them as swapped out stopping them from being swapped out a second time. More details in the ttm_pool.c DOC section.
v2:
- A couple of cleanups and error fixes in ttm_pool_back_up_tt.
- s/back_up/backup/
- Add a writeback parameter to the exported interface.
--- drivers/gpu/drm/ttm/ttm_pool.c | 397 +++++++++++++++++++++++++++++++-- drivers/gpu/drm/ttm/ttm_tt.c | 37 +++ include/drm/ttm/ttm_pool.h | 5 + include/drm/ttm/ttm_tt.h | 20 ++ 4 files changed, 446 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +384 -13
@@ -41,6 +41,7 @@#include <asm/set_memory.h> #endif +#include <drm/ttm/ttm_backup.h> #include <drm/ttm/ttm_pool.h> #include <drm/ttm/ttm_tt.h> #include <drm/ttm/ttm_bo.h>
@@ -58,6 +59,32 @@ struct ttm_pool_dma {unsigned long vaddr; }; +/** +* struct ttm_pool_tt_restore - State representing restore from backup +* @alloced_pages: Total number of already allocated pages for the ttm_tt. +* @restored_pages: Number of (sub) pages restored from swap for this +* chunk of 1 << @order pages. +* @first_page: The ttm page ptr representing for @old_pages[0]. +* @caching_divide: Page pointer where subsequent pages are cached. +* @old_pages: Backup copy of page pointers that were replaced by the new +* page allocation. +* @pool: The pool used for page allocation while restoring. +* @order: The order of the last page allocated while restoring. +* +* Recovery from backup might fail when we've recovered less than the +* full ttm_tt. In order not to loose any data (yet), keep information +* around that allows us to restart a failed ttm backup recovery. +*/ +struct ttm_pool_tt_restore { +pgoff_t alloced_pages; +pgoff_t restored_pages; +struct page **first_page; +struct page **caching_divide; +struct ttm_pool *pool; +unsigned int order; +struct page *old_pages[]; +}; +static unsigned long page_pool_size;MODULE_PARM_DESC (page_pool_size,"Number of pages in the WC/UC/DMA pool" );
@@ -354,11 +381,102 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)return p->private; } +/* +* To be able to insert single pages into backup directly, +* we need to split multi-order page allocations and make them look +* like single-page allocations. +*/ +static void ttm_pool_split_for_swap (struct ttm_pool *pool,struct page *p) +{ +unsigned int order =ttm_pool_page_order (pool, p); +pgoff_t nr; + +if (!order) +return ; + +split_page (p, order); + nr =1UL << order; +while (nr--) + (p++)->private =0 ; +} + +/** +* DOC: Partial backup and restoration of a struct ttm_tt. +* +* Swapout using ttm_backup::ops::backup_page() and swapin using +* ttm_backup::ops::copy_backed_up_page() may fail. +* The former most likely due to lack of swap-space or memory, the latter due +* to lack of memory or because of signal interruption during waits. +* +* Backupfailure is easily handled by using a ttm_tt pages vector that holds +* both swap entries and page pointers. This has to be taken into account when +* restoring such a ttm_tt from backup, and when freeing it while backed up. +* When restoring, for simplicity, new pages are actually allocated from the +* pool and the contents of any old pages are copied in and then the old pages +* are released. +* +* For restoration failures, the struct ttm_pool_tt_restore holds sufficient state +* to be able to resume an interrupted restore, and that structure is freed once +* the restoration is complete. If the struct ttm_tt is destroyed while there +* is a valid struct ttm_pool_tt_restore attached, that is also properly taken +* care of. +*/ + +static bool ttm_pool_restore_valid (const struct ttm_pool_tt_restore *restore) +{ +return restore && restore->restored_pages < (1 << restore->order); +} + +static int ttm_pool_restore_tt (struct ttm_pool_tt_restore *restore, +struct ttm_backup *backup, +struct ttm_operation_ctx *ctx) +{ +unsigned int i, nr =1 << restore->order; +int ret =0 ; + +if (!ttm_pool_restore_valid (restore)) +return 0 ; + +for (i = restore->restored_pages; i < nr; ++i) { +struct page *p = restore->old_pages[i]; + +if (ttm_backup_page_ptr_is_handle (p)) { +unsigned long handle =ttm_backup_page_ptr_to_handle (p); + +if (handle ==0 ) +continue ; + + ret = backup->ops->copy_backed_up_page + (backup, restore->first_page[i], + handle, ctx->interruptible); +if (ret) +break ; + + backup->ops->drop (backup, handle); + }else if (p) { +/* +* We could probably avoid splitting the old page +* using clever logic, but ATM we don't care. +*/ +ttm_pool_split_for_swap (restore->pool, p); +copy_highpage (restore->first_page[i], p); +__free_pages (p,0 ); + } + + restore->restored_pages++; + restore->old_pages[i] =NULL ; +cond_resched (); + } + +return ret; +} +/* Called when we got a page, either from a pool or newly allocated */ static int ttm_pool_page_allocated (struct ttm_pool *pool,unsigned int order,struct page *p,dma_addr_t **dma_addr,unsigned long *num_pages, -struct page ***pages) +struct page ***pages, +struct ttm_pool_tt_restore *restore) {unsigned int i;int r;
@@ -369,6 +487,16 @@ static int ttm_pool_page_allocated(struct ttm_pool *pool, unsigned int order,return r; } +if (restore) { +memcpy (restore->old_pages, *pages, + (1 << order) *sizeof (*restore->old_pages)); +memset (*pages,0 , (1 << order) *sizeof (**pages)); + restore->order = order; + restore->restored_pages =0 ; + restore->first_page = *pages; + restore->alloced_pages +=1UL << order; + } + *num_pages -=1 << order;for (i =1 << order; i; --i, ++(*pages), ++p) **pages = p;
@@ -394,22 +522,39 @@ static void ttm_pool_free_range(struct ttm_pool *pool, struct ttm_tt *tt, pgoff_t start_page, pgoff_t end_page) {struct page **pages = &tt->pages[start_page]; +struct ttm_backup *backup = tt->backup;unsigned int order;pgoff_t i, nr;for (i = start_page; i < end_page; i += nr, pages += nr) {struct ttm_pool_type *pt =NULL ; +struct page *p = *pages; + +if (ttm_backup_page_ptr_is_handle (p)) { +unsigned long handle =ttm_backup_page_ptr_to_handle (p); + + nr =1 ; +if (handle !=0 ) + backup->ops->drop (backup, handle); +continue ; + } + +if (pool) { + order =ttm_pool_page_order (pool, p); + nr = (1UL << order); +if (tt->dma_address) +ttm_pool_unmap (pool, tt->dma_address[i], nr); - order =ttm_pool_page_order (pool, *pages); - nr = (1UL << order); -if (tt->dma_address) -ttm_pool_unmap (pool, tt->dma_address[i], nr); + pt =ttm_pool_select_type (pool, caching, order); + }else { + order = p->private; + nr = (1UL << order); + } - pt =ttm_pool_select_type (pool, caching, order);if (pt) -ttm_pool_type_give (pt, *pages); +ttm_pool_type_give (pt, p);else -ttm_pool_free_page (pool, caching, order, *pages); +ttm_pool_free_page (pool, caching, order, p); } }
@@ -453,9 +598,37 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,else gfp_flags |=GFP_HIGHUSER ; - for (order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages)); - num_pages; - order = min_t(unsigned int, order, __fls(num_pages))) { + order = min_t(unsigned int, MAX_PAGE_ORDER, __fls(num_pages)); + +if (tt->page_flags &TTM_TT_FLAG_PRIV_BACKED_UP ) { +if (!tt->restore) { +gfp_t gfp =GFP_KERNEL | __GFP_NOWARN; + +if (ctx->gfp_retry_mayfail) + gfp |= __GFP_RETRY_MAYFAIL; + + tt->restore = +kvzalloc (struct_size (tt->restore, old_pages, + (size_t )1 << order), gfp); +/* RFC: Possibly loop on -ENOMEM and reduce order. */ +if (!tt->restore) +return -ENOMEM ; + }else if (ttm_pool_restore_valid (tt->restore)) { +struct ttm_pool_tt_restore *restore = tt->restore; + + num_pages -= restore->alloced_pages; + order = min_t(unsigned int, order, __fls(num_pages)); + pages += restore->alloced_pages; + r =ttm_pool_restore_tt (restore, tt->backup, ctx); +if (r) +return r; + caching = restore->caching_divide; + } + + tt->restore->pool = pool; + } + + for (; num_pages; order = min_t(unsigned int, order, __fls(num_pages))) {struct ttm_pool_type *pt; page_caching = tt->caching;
@@ -472,11 +645,19 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, r =ttm_pool_page_allocated (pool, order, p, &dma_addr, &num_pages, - &pages); + &pages, + tt->restore);if (r) goto error_free_page; caching = pages; +if (ttm_pool_restore_valid (tt->restore)) { + r =ttm_pool_restore_tt (tt->restore, tt->backup, + ctx); +if (r) + goto error_free_all; + } +if (num_pages < (1 << order))break ;
@@ -496,9 +677,17 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, caching = pages; } r =ttm_pool_page_allocated (pool, order, p, &dma_addr, - &num_pages, &pages); + &num_pages, &pages, + tt->restore);if (r) goto error_free_page; + +if (ttm_pool_restore_valid (tt->restore)) { + r =ttm_pool_restore_tt (tt->restore, tt->backup, ctx); +if (r) + goto error_free_all; + } +if (PageHighMem (p)) caching = pages; }
@@ -517,12 +706,26 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,if (r) goto error_free_all; +if (tt->restore) { +kvfree (tt->restore); + tt->restore =NULL ; + } + +if (tt->page_flags &TTM_TT_FLAG_PRIV_BACKED_UP ) + tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP | +TTM_TT_FLAG_SWAPPED ); +return 0 ; error_free_page:ttm_pool_free_page (pool, page_caching, order, p); error_free_all: +if (tt->page_flags &TTM_TT_FLAG_PRIV_BACKED_UP ) { + tt->restore->caching_divide = caching; +return r; + } + num_pages = tt->num_pages - num_pages; caching_divide = caching - tt->pages;ttm_pool_free_range (pool, tt, tt->caching,0 , caching_divide);
@@ -549,6 +752,174 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt) }EXPORT_SYMBOL (ttm_pool_free); +/** +* ttm_pool_release_backed_up() - Release content of a swapped-out struct ttm_tt +* @tt: The struct ttm_tt. +* +* Release handles with associated content or any remaining pages of +* a backed-up struct ttm_tt. +*/ +void ttm_pool_release_backed_up (struct ttm_tt *tt) +{ +struct ttm_backup *backup = tt->backup; +struct ttm_pool_tt_restore *restore; +pgoff_t i, start_page =0 ; +unsigned long handle; + +if (!(tt->page_flags &TTM_TT_FLAG_PRIV_BACKED_UP )) +return ; + + restore = tt->restore; + +if (ttm_pool_restore_valid (restore)) { +pgoff_t nr =1UL << restore->order; + +for (i = restore->restored_pages; i < nr; ++i) { +struct page *p = restore->old_pages[i]; + +if (ttm_backup_page_ptr_is_handle (p)) { + handle =ttm_backup_page_ptr_to_handle (p); +if (handle ==0 ) +continue ; + + backup->ops->drop (backup, handle); + }else if (p) { +ttm_pool_split_for_swap (restore->pool, p); +__free_pages (p,0 ); + } + } + } + +if (restore) { +pgoff_t mid = restore->caching_divide - tt->pages; + + start_page = restore->alloced_pages; +/* Pages that might be dma-mapped and non-cached */ +ttm_pool_free_range (restore->pool, tt, tt->caching, +0 , mid); +/* Pages that might be dma-mapped but cached */ +ttm_pool_free_range (restore->pool, tt, ttm_cached, + mid, restore->alloced_pages); + } + +/* Shrunken pages. Cached and not dma-mapped. */ +ttm_pool_free_range (NULL , tt, ttm_cached, start_page, tt->num_pages); + +if (restore) { +kvfree (restore); + tt->restore =NULL ; + } + + tt->page_flags &= ~(TTM_TT_FLAG_PRIV_BACKED_UP |TTM_TT_FLAG_SWAPPED ); +} + +/** +* ttm_pool_backup_tt() - Back up or purge a struct ttm_tt +* @pool: The pool used when allocating the struct ttm_tt. +* @ttm: The struct ttm_tt. +* @purge: Don't back up but release pages directly to system. +* @writeback: If !@purge, Try to write out directly to the +* underlying persistent media. +* +* Back up or purge a struct ttm_tt. If @purge is true, then +* all pages will be freed directly to the system rather than to the pool +* they were allocated from, making the function behave similarly to +* ttm_pool_free(). If @purge is false the pages will be backed up instead, +* exchanged for handles. +* A subsequent call to ttm_pool_alloc() will then read back the content and +* a subsequent call to ttm_pool_release_shrunken() will drop it. +* If backup of a page fails for whatever reason, @ttm will still be +* partially backed up, retaining those pages for which backup fails. +* +* Return: Number of pages actually backed up or freed, or negative +* error code on error. +*/ +long ttm_pool_backup_tt (struct ttm_pool *pool,struct ttm_tt *ttm,bool purge, +bool writeback) +{ +struct ttm_backup *backup = ttm->backup; +struct page *page; +unsigned long handle; +gfp_t alloc_gfp; +gfp_t gfp; +int ret =0 ; +pgoff_t shrunken =0 ; +pgoff_t i, num_pages; + +if ((!get_nr_swap_pages () && !purge) || + pool->use_dma_alloc || + (ttm->page_flags &TTM_TT_FLAG_PRIV_BACKED_UP )) +return -EBUSY ; + +#ifdef CONFIG_X86 +/* Anything returned to the system needs to be cached. */ +if (ttm->caching != ttm_cached) +set_pages_array_wb (ttm->pages, ttm->num_pages); +#endif + +if (ttm->dma_address || purge) { +for (i =0 ; i < ttm->num_pages; i += num_pages) { +unsigned int order; + + page = ttm->pages[i]; +if (unlikely (!page)) { + num_pages =1 ; +continue ; + } + + order =ttm_pool_page_order (pool, page); + num_pages =1UL << order; +if (ttm->dma_address) +ttm_pool_unmap (pool, ttm->dma_address[i], + num_pages); +if (purge) { + shrunken += num_pages; + page->private =0 ; +__free_pages (page, order); +memset (ttm->pages + i,0 , + num_pages *sizeof (*ttm->pages)); + } + } + } + +if (purge) +return shrunken; + +if (pool->use_dma32) + gfp =GFP_DMA32 ; +else + gfp =GFP_HIGHUSER ; + + alloc_gfp =GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL; + +for (i =0 ; i < ttm->num_pages; ++i) { + page = ttm->pages[i]; +if (unlikely (!page)) +continue ; + +ttm_pool_split_for_swap (pool, page); + + handle = backup->ops->backup_page (backup, page, writeback, i, + gfp, alloc_gfp); +if (handle) { + ttm->pages[i] =ttm_backup_handle_to_page_ptr (handle); +put_page (page); + shrunken++; + }else { +/* We allow partially shrunken tts */ + ret = -ENOMEM ; +break ; + } +cond_resched (); + } + +if (shrunken) + ttm->page_flags |= (TTM_TT_FLAG_PRIV_BACKED_UP | +TTM_TT_FLAG_SWAPPED ); + +return shrunken ? shrunken : ret; +} +/** * ttm_pool_init - Initialize a pool *
drivers/gpu/drm/ttm/ttm_tt.c +37 -0
@@ -40,6 +40,7 @@#include <drm/drm_cache.h> #include <drm/drm_device.h> #include <drm/drm_util.h> +#include <drm/ttm/ttm_backup.h> #include <drm/ttm/ttm_bo.h> #include <drm/ttm/ttm_tt.h>
@@ -158,6 +159,7 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm, ttm->swap_storage =NULL ; ttm->sg = bo->sg; ttm->caching = caching; + ttm->restore =NULL ; } int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
@@ -182,6 +184,12 @@ void ttm_tt_fini(struct ttm_tt *ttm)fput (ttm->swap_storage); ttm->swap_storage =NULL ; +ttm_pool_release_backed_up (ttm); +if (ttm->backup) { + ttm->backup->ops->fini (ttm->backup); + ttm->backup =NULL ; + } +if (ttm->pages)kvfree (ttm->pages);else
@@ -252,6 +260,35 @@ int ttm_tt_swapin(struct ttm_tt *ttm)return ret; } +/** +* ttm_tt_backup() - Helper to back up a struct ttm_tt. +* @bdev: The TTM device. +* @tt: The struct ttm_tt. +* @purge: Don't back up but release pages directly to system, +* bypassing any pooling. +* @writeback: If !@purge, try to write out directly to the +* underlying persistent media. +* +* Helper for a TTM driver to use from the bo_shrink() method to shrink +* a struct ttm_tt, after it has done the necessary unbinding. This function +* will update the page accounting and call ttm_pool_shrink_tt to free pages +* or move them to the swap cache. +* +* Return: Number of pages freed or swapped out, or negative error code on +* error. +*/ +long ttm_tt_backup (struct ttm_device *bdev,struct ttm_tt *tt,bool purge, +bool writeback) +{ +long ret =ttm_pool_backup_tt (&bdev->pool, tt, purge, writeback); + +if (ret >0 ) + tt->page_flags &= ~TTM_TT_FLAG_PRIV_POPULATED ; + +return ret; +} +EXPORT_SYMBOL (ttm_tt_backup); +/** * ttm_tt_swapout - swap out tt object *
include/drm/ttm/ttm_pool.h +5 -0
@@ -89,6 +89,11 @@ void ttm_pool_fini(struct ttm_pool *pool);int ttm_pool_debugfs (struct ttm_pool *pool,struct seq_file *m); +void ttm_pool_release_backed_up (struct ttm_tt *tt); + +long ttm_pool_backup_tt (struct ttm_pool *pool,struct ttm_tt *ttm, +bool purge,bool writeback); +int ttm_pool_mgr_init (unsigned long num_pages);void ttm_pool_mgr_fini (void );
include/drm/ttm/ttm_tt.h +20 -0
@@ -32,11 +32,13 @@#include <drm/ttm/ttm_caching.h> #include <drm/ttm/ttm_kmap_iter.h> +struct ttm_backup ;struct ttm_device ;struct ttm_tt ;struct ttm_resource ;struct ttm_buffer_object ;struct ttm_operation_ctx ; +struct ttm_pool_tt_restore ;/** * struct ttm_tt - This is a structure holding the pages, caching- and aperture
@@ -85,6 +87,9 @@ struct ttm_tt {* fault handling abuses the DMA api a bit and dma_map_attrs can't be * used to assure pgprot always matches. * +* TTM_TT_FLAG_PRIV_BACKED_UP: TTM internal only. This is set if the +* struct ttm_tt has been (possibly partially) backed up. +* * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is * set by TTM after ttm_tt_populate() has successfully returned, and is * then unset when TTM calls ttm_tt_unpopulate().
@@ -96,6 +101,7 @@ struct ttm_tt {#define TTM_TT_FLAG_DECRYPTED BIT(4)#define TTM_TT_FLAG_PRIV_POPULATED BIT(5) +#define TTM_TT_FLAG_PRIV_BACKED_UP BIT(6)uint32_t page_flags;/** @num_pages: Number of pages in the page array. */ uint32_t num_pages;
@@ -105,11 +111,21 @@ struct ttm_tt {dma_addr_t *dma_address;/** @swap_storage: Pointer to shmem struct file for swap storage. */ struct file *swap_storage; +/** +* @backup: Pointer to backup struct for backed up tts. +* RFC: Could possibly be unified with @swap_storage. +*/ +struct ttm_backup *backup;/** * @caching: The current caching state of the pages, see enum * ttm_caching. */ enum ttm_caching caching; +/** +* @restore: Partial restoration from backup state. +* RFC: Incorporate in struct ttm_backup? +*/ +struct ttm_pool_tt_restore *restore; };/**
@@ -230,6 +246,10 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);struct ttm_kmap_iter *ttm_kmap_iter_tt_init (struct ttm_kmap_iter_tt *iter_tt,struct ttm_tt *tt);unsigned long ttm_tt_pages_limit (void ); + +long ttm_tt_backup (struct ttm_device *bdev,struct ttm_tt *tt,bool purge, +bool writeback); +#if IS_ENABLED (CONFIG_AGP )#include <linux/agp_backend.h>
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 10/21 drm/ttm: Use fault-injection to test error paths
In reply to Thomas HellströmUse fault-injection to test partial TTM swapout and interrupted swapin. Return -EINTR for swapin to test the callers ability to handle and restart the swapin, and on swapout perform a partial swapout to test that the swapin and release_shrunken functionality.
--- drivers/gpu/drm/Kconfig | 10 ++++++++++ drivers/gpu/drm/ttm/ttm_pool.c | 17 ++++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/Kconfig +10 -0
@@ -258,6 +258,16 @@ config DRM_GPUVM GPU-VM representation providing helpers to manage a GPUs virtual address space +config DRM_TTM_BACKUP_FAULT_INJECT + bool "Enable fault injection during TTM backup" + depends on DRM_TTM + default n + help + Inject recoverable failures during TTM backup and recovery of + backed-up objects. For DRM driver developers only. + + If in doubt, choose N. + config DRM_BUDDY tristate depends on DRM
drivers/gpu/drm/ttm/ttm_pool.c +16 -1
@@ -431,6 +431,7 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore, struct ttm_backup *backup, struct ttm_operation_ctx *ctx) { +static unsigned long __maybe_unused swappedin;unsigned int i, nr =1 << restore->order;int ret =0 ;
@@ -446,6 +447,13 @@ static int ttm_pool_restore_tt(struct ttm_pool_tt_restore *restore,if (handle ==0 )continue ; +if (IS_ENABLED (CONFIG_DRM_TTM_BACKUP_FAULT_INJECT ) && + ctx->interruptible && + ++swappedin %100 ==0 ) { + ret = -EINTR ; +break ; + } + ret = backup->ops->copy_backed_up_page (backup, restore->first_page[i], handle, ctx->interruptible);
@@ -892,7 +900,14 @@ long ttm_pool_backup_tt(struct ttm_pool *pool, struct ttm_tt *ttm, bool purge, alloc_gfp =GFP_KERNEL | __GFP_HIGH | __GFP_NOWARN | __GFP_RETRY_MAYFAIL; -for (i =0 ; i < ttm->num_pages; ++i) { + num_pages = ttm->num_pages; + +/* Pretend doing fault injection by shrinking only half of the pages. */ + +if (IS_ENABLED (CONFIG_DRM_TTM_BACKUP_FAULT_INJECT )) + num_pages =DIV_ROUND_UP (num_pages,2 ); + +for (i =0 ; i < num_pages; ++i) { page = ttm->pages[i];if (unlikely (!page))continue ;
Signature · 1 line
2.44.0
Thomas Hellström
maintainerPATCH v3 11/21 drm/ttm, drm/xe: Add a shrinker for xe bos
In reply to Thomas HellströmRather than relying on the TTM watermark accounting add a shrinker for xe_bos in TT or system memory.
Leverage the newly added TTM per-page shrinking and shmem backup support.
Although xe doesn't fully support WONTNEED (purgeable) bos yet, introduce and add shrinker support for purgeable ttm_tts.
v2:
- Cleanups bugfixes and a KUNIT shrinker test.
- Add writeback support, and activate if kswapd.
v3:
- Move the try_shrink() helper to core TTM.
- Minor cleanups.
--- drivers/gpu/drm/ttm/ttm_bo_util.c | 67 ++++++++ drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/tests/xe_bo.c | 118 ++++++++++++++ drivers/gpu/drm/xe/tests/xe_bo_test.c | 1 + drivers/gpu/drm/xe/tests/xe_bo_test.h | 1 + drivers/gpu/drm/xe/xe_bo.c | 128 +++++++++++++-- drivers/gpu/drm/xe/xe_bo.h | 4 + drivers/gpu/drm/xe/xe_device.c | 8 + drivers/gpu/drm/xe/xe_device_types.h | 2 + drivers/gpu/drm/xe/xe_shrinker.c | 224 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_shrinker.h | 18 +++ include/drm/ttm/ttm_bo.h | 3 + 12 files changed, 559 insertions(+), 16 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_shrinker.c create mode 100644 drivers/gpu/drm/xe/xe_shrinker.h
drivers/gpu/drm/ttm/ttm_bo_util.c +67 -0
@@ -913,3 +913,70 @@ long ttm_lru_walk_for_evict(struct ttm_lru_walk *walk, struct ttm_device *bdev,ttm_resource_cursor_fini (&cursor);return sofar; } +EXPORT_SYMBOL (ttm_lru_walk_for_evict); + +/** +* ttm_bo_try_shrink - LRU walk helper to shrink a ttm buffer object. +* @walk: The struct xe_ttm_lru_walk that describes the walk. +* @bo: The buffer object. +* @purge: Whether to attempt to purge the bo content since it's no +* longer needed. +* @writeback: If !@purge, attempt to write out to persistent storage. +* +* The function uses the ttm_tt_back_up functionality to back up or +* purge a struct ttm_tt. If the bo is not in system, it's first +* moved there. +* +* Return: The number of pages shrunken or purged, or +* negative error code on failure. +*/ +long ttm_bo_try_shrink (struct ttm_lru_walk *walk,struct ttm_buffer_object *bo, +bool purge,bool writeback) +{ +static const struct ttm_place sys_placement_flags = { + .fpfn =0 , + .lpfn =0 , + .mem_type =TTM_PL_SYSTEM , + .flags =0 , + }; +static struct ttm_placement sys_placement = { + .num_placement =1 , + .placement = &sys_placement_flags, + }; +struct ttm_operation_ctx *ctx = walk->ctx; +struct ttm_tt *tt = bo->ttm; +long lret; + +dma_resv_assert_held (bo->base.resv); + +if (!tt || !ttm_tt_is_populated (tt)) +return 0 ; + +if (bo->resource->mem_type !=TTM_PL_SYSTEM ) { +int ret =ttm_bo_validate (bo, &sys_placement, ctx); + +if (ret) { +if (ret == -EINTR || ret == -EDEADLK || + ret == -ERESTARTSYS ) +return ret; +return 0 ; + } + } + + lret =ttm_bo_wait_ctx (bo, ctx); +if (lret <0 ) { +if (lret == -ERESTARTSYS ) +return lret; +return 0 ; + } + +if (bo->deleted) + lret =ttm_tt_backup (bo->bdev, tt, true, writeback); +else + lret =ttm_tt_backup (bo->bdev, tt, purge, writeback); +if (lret <0 && lret != -EINTR ) +return 0 ; + +return lret; +} +EXPORT_SYMBOL (ttm_bo_try_shrink);
drivers/gpu/drm/xe/Makefile +1 -0
@@ -130,6 +130,7 @@ xe-y += xe_bb.o \ xe_ring_ops.o \ xe_sa.o \ xe_sched_job.o \ + xe_shrinker.o \ xe_step.o \ xe_sync.o \ xe_tile.o \
drivers/gpu/drm/xe/tests/xe_bo.c +118 -0
127 lines not on this page · raw message
drivers/gpu/drm/xe/tests/xe_bo_test.c +1 -0
@@ -10,6 +10,7 @@static struct kunit_case xe_bo_tests[] = {KUNIT_CASE (xe_ccs_migrate_kunit),KUNIT_CASE (xe_bo_evict_kunit), +KUNIT_CASE_SLOW (xe_bo_shrink_kunit), {} };
drivers/gpu/drm/xe/tests/xe_bo_test.h +1 -0
@@ -10,5 +10,6 @@ struct kunit;void xe_ccs_migrate_kunit (struct kunit *test);void xe_bo_evict_kunit (struct kunit *test); +void xe_bo_shrink_kunit (struct kunit *test);#endif
drivers/gpu/drm/xe/xe_bo.c +112 -16
255 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.h +4 -0
@@ -63,6 +63,7 @@#define XE_BO_PROPS_INVALID (-1)struct sg_table ; +struct xe_ttm_lru_walk ;struct xe_bo *xe_bo_alloc (void );void xe_bo_free (struct xe_bo *bo);
@@ -315,6 +316,9 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)#define i915_gem_object_flush_if_display (obj) ((void)(obj)) +long xe_bo_shrink (struct ttm_lru_walk *walk,struct ttm_buffer_object *bo, +bool purge,bool writeback); +#if IS_ENABLED (CONFIG_DRM_XE_KUNIT_TEST )/** * xe_bo_is_mem_type - Whether the bo currently resides in the given
drivers/gpu/drm/xe/xe_device.c +8 -0
@@ -42,6 +42,7 @@#include "xe_pcode.h" #include "xe_pm.h" #include "xe_query.h" +#include "xe_shrinker.h" #include "xe_sriov.h" #include "xe_tile.h" #include "xe_ttm_stolen_mgr.h"
@@ -239,6 +240,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)if (xe->unordered_wq)destroy_workqueue (xe->unordered_wq); +if (!IS_ERR_OR_NULL (xe->mem.shrinker)) +xe_shrinker_destroy (xe->mem.shrinker); +ttm_device_fini (&xe->ttm); }
@@ -268,6 +272,10 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,if (err) goto err; + xe->mem.shrinker =xe_shrinker_create (xe); +if (IS_ERR (xe->mem.shrinker)) +return ERR_CAST (xe->mem.shrinker); + xe->info.devid = pdev->device; xe->info.revid = pdev->revision; xe->info.force_execlist = xe_modparam.force_execlist;
drivers/gpu/drm/xe/xe_device_types.h +2 -0
@@ -319,6 +319,8 @@ struct xe_device {struct xe_mem_region vram;/** @mem.sys_mgr: system TTM manager */ struct ttm_resource_manager sys_mgr; +/** @mem.sys_mgr: system memory shrinker. */ +struct xe_shrinker *shrinker; } mem;/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_shrinker.c new file +224 -0
224 lines not on this page · raw message
drivers/gpu/drm/xe/xe_shrinker.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef _XE_SHRINKER_H_ +#define _XE_SHRINKER_H_ + +struct xe_shrinker ; +struct xe_device ; + +void xe_shrinker_mod_pages (struct xe_shrinker *shrinker,long shrinkable,long purgeable); + +struct xe_shrinker *xe_shrinker_create (struct xe_device *xe); + +void xe_shrinker_destroy (struct xe_shrinker *shrinker); + +#endif
include/drm/ttm/ttm_bo.h +3 -0
@@ -222,6 +222,9 @@ struct ttm_lru_walk {long ttm_lru_walk_for_evict (struct ttm_lru_walk *walk,struct ttm_device *bdev,struct ttm_resource_manager *man,long target); +long ttm_bo_try_shrink (struct ttm_lru_walk *walk,struct ttm_buffer_object *bo, +bool purge,bool writeback); +/** * ttm_bo_get - reference a struct ttm_buffer_object *
Signature · 1 line
2.44.0
Thomas Hellström
maintainerRFC PATCH v3 12/21 dma-buf/dma-resv: Introduce dma_resv_trylock_ctx()
In reply to Thomas HellströmFor the drm_exec_trylock() functionality, there is a need to be able to trylock a dma-resv object as part of a drm_exec transaction. Therefore expose a variant of dma_resv_trylock that also takes a struct ww_acquire_ctx parameter.
--- include/linux/dma-resv.h | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
include/linux/dma-resv.h +22 -1
@@ -405,6 +405,27 @@ static inline int dma_resv_lock_slow_interruptible(struct dma_resv *obj,return ww_mutex_lock_slow_interruptible (&obj->lock, ctx); } +/** +* dma_resv_trylock_ctx - trylock the reservation object +* @obj: the reservation object +* @ctx: The ww acquire context or NULL. +* +* Tries to lock the reservation object for exclusive access and modification. +* Note, that the lock is only against other writers, readers will run +* concurrently with a writer under RCU. The seqlock is used to notify readers +* if they overlap with a writer. The context parameter ensures that other +* ww transactions can perform deadlock backoff if necessary, and that +* subsequent attempts to dma_resv_lock() @obj for @ctx will return +* -EALREADY. +* +* Return: true if the lock was acquired, false otherwise. +*/ +static inline bool __must_check +dma_resv_trylock_ctx (struct dma_resv *obj,struct ww_acquire_ctx *ctx) +{ +return ww_mutex_trylock (&obj->lock, ctx); +} +/** * dma_resv_trylock - trylock the reservation object * @obj: the reservation object
@@ -421,7 +442,7 @@ static inline int dma_resv_lock_slow_interruptible(struct dma_resv *obj,*/ static inline bool __must_check dma_resv_trylock (struct dma_resv *obj) { -return ww_mutex_trylock (&obj->lock,NULL ); +return dma_resv_trylock_ctx (obj,NULL ); }/**
Signature · 1 line
2.44.0
Thomas Hellström
maintainerRFC PATCH v3 13/21 drm/exec: Rework contended locking
In reply to Thomas HellströmIf contention and backoff occurs during a drm_exec ww transaction, the contended lock is not locked again until the next orinary attempt to lock a dma_resv lock. However, with the introduction of drm_exec_trylock(), that doesn't work, since the locking of the contended lock needs to be a sleeping lock. Neither can we ignore locking the contended lock during a trylock since that would violate at least the ww_mutex annotations.
So resolve this by actually locking the contended lock during drm_exec_retry_on_contention(). However, this introduces a new point of failure since locking the contended lock may return -EINTR.
Hence drm_exec_retry_on_contention() must take an error parameter and also return a value indicating success.
--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35 ++++++++++++++----- drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++--- 17 files changed, 92 insertions(+), 62 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +8 -8
@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct kgd_mem *mem,drm_exec_init (&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&ctx->exec) { ret =amdgpu_vm_lock_pd (vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; ret =drm_exec_prepare_obj (&ctx->exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; }
@@ -1199,14 +1199,14 @@ static int reserve_bo_and_cond_vms(struct kgd_mem *mem, ret =amdgpu_vm_lock_pd (entry->bo_va->base.vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; ++ctx->n_vms; } ret =drm_exec_prepare_obj (&ctx->exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; }
@@ -2619,7 +2619,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info)list_for_each_entry (peer_vm, &process_info->vm_list_head, vm_list_node) { ret =amdgpu_vm_lock_pd (peer_vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) goto unreserve_out; }
@@ -2631,7 +2631,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) gobj = &mem->bo->tbo.base; ret =drm_exec_prepare_obj (&exec, gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) goto unreserve_out; }
@@ -2875,7 +2875,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *list_for_each_entry (peer_vm, &process_info->vm_list_head, vm_list_node) { ret =amdgpu_vm_lock_pd (peer_vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) {pr_err ("Locking VM PD failed, ret: %d\n" , ret); goto ttm_reserve_fail;
@@ -2891,7 +2891,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu * gobj = &mem->bo->tbo.base; ret =drm_exec_prepare_obj (&exec, gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) {pr_err ("drm_exec_prepare_obj failed, ret: %d\n" , ret); goto ttm_reserve_fail;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +3 -3
@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, drm_exec_until_all_locked(&p->exec) { r =amdgpu_vm_lock_pd (&fpriv->vm, &p->exec,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages;
@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,/* One fence for TTM and one for each CS job */ r =drm_exec_prepare_obj (&p->exec, &e->bo->tbo.base,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages;
@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,if (p->uf_bo) { r =drm_exec_prepare_obj (&p->exec, &p->uf_bo->tbo.base,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages; }
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +2 -2
@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) {DRM_ERROR ("failed to reserve CSA,PD BOs: err=%d\n" , r); goto error;
@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) {DRM_ERROR ("failed to reserve CSA,PD BOs: err=%d\n" , r); goto error;
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +4 -4
@@ -239,12 +239,12 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj,drm_exec_init (&exec,DRM_EXEC_IGNORE_DUPLICATES ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_prepare_obj (&exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }
@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, drm_exec_until_all_locked(&exec) {if (gobj) { r =drm_exec_lock_obj (&exec, gobj); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; } r =amdgpu_vm_lock_pd (&fpriv->vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +4 -4
@@ -1221,12 +1221,12 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &ctx_data->meta_data_obj->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; }
@@ -1292,12 +1292,12 @@ int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &ctx_data->meta_data_obj->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +2 -2
@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }
@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct amdgpu_device *adev, struct amdgpu_fpriv *fpriv) r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +4 -4
@@ -89,12 +89,12 @@ static int map_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; }
@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1 -1
@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm =drm_priv_to_vm (pdd->drm_priv); r =amdgpu_vm_lock_pd (vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + r =drm_exec_retry_on_contention (&ctx->exec, r);if (unlikely (r)) {pr_debug ("failed %d to reserve bo\n" , r); goto unreserve_out;
drivers/gpu/drm/drm_exec.c +26 -9
@@ -28,12 +28,12 @@* drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); -* drm_exec_retry_on_contention(&exec); +* ret = drm_exec_retry_on_contention(&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); -* drm_exec_retry_on_contention(&exec); +* ret = drm_exec_retry_on_contention(&exec, ret); * if (ret) * goto error; * }
@@ -48,7 +48,8 @@*/ /* Dummy value used to initially enter the retry loop */ -#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)/* Unlock all objects and drop references */ static void drm_exec_unlock_all (struct drm_exec *exec)
@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec)return true; } -drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true; }EXPORT_SYMBOL (drm_exec_cleanup);
@@ -194,6 +194,27 @@ static int drm_exec_lock_contended(struct drm_exec *exec)return ret; } +/** +* drm_exec_handle_contended() - Perform cleanup before a ww transaction restart +* @exec: Pointer to the drm_exec object. +* +* Unlocks all held resvs and re-locks the contended object. +* +* Return: 0 on success, negative error code on failure. +*/ +int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/** * drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state
@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) {int ret; - ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (exec->prelocked == obj) {drm_gem_object_put (exec->prelocked); exec->prelocked =NULL ;
drivers/gpu/drm/drm_gpuvm.c +4 -4
@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct drm_gpuvm_exec *vm_exec)drm_exec_until_all_locked (exec) { ret =drm_gpuvm_prepare_vm (gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; ret =drm_gpuvm_prepare_objects (gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err;if (vm_exec->extra.fn) { ret = vm_exec->extra.fn (vm_exec); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; }
@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec,drm_exec_until_all_locked (exec) { ret =drm_gpuvm_prepare_range (gpuvm, exec, addr, range, vm_exec->num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; }
drivers/gpu/drm/imagination/pvr_job.c +1 -1
@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data,drm_exec_until_all_locked (exec) {int err =jobs_lock_all_objs (exec, job_data, job_count); -drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention (exec, err);if (err)return err; }
drivers/gpu/drm/msm/msm_gem_submit.c +1 -1
@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit)for (unsigned i =0 ; i < submit->nr_bos; i++) {struct drm_gem_object *obj = submit->bos[i].obj; ret =drm_exec_prepare_obj (&submit->exec, obj,1 ); -drm_exec_retry_on_contention (&submit->exec); + ret =drm_exec_retry_on_contention (&submit->exec, ret);if (ret) goto error; }
drivers/gpu/drm/nouveau/nouveau_uvmm.c +1 -1
@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job,drm_exec_init (exec, vme->flags,0 );drm_exec_until_all_locked (exec) { ret =bind_lock_validate (job, exec, vme->num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) { op =list_last_op (&bind_job->ops); goto unwind;
drivers/gpu/drm/tests/drm_exec_test.c +6 -6
@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;
@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;drm_exec_unlock_obj (&exec, &gobj); ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;
@@ -110,13 +110,13 @@ static void test_duplicates(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_IGNORE_DUPLICATES ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ; ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;
@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_prepare_obj (&exec, &gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;
drivers/gpu/drm/xe/xe_gt_pagefault.c +2 -2
@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { ret =xe_pf_begin (&exec, vma, atomic, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (ret) goto unlock_dma_resv;
@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { ret =xe_pf_begin (&exec, vma, true, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (ret)break ; }
drivers/gpu/drm/xe/xe_vm.c +5 -5
@@ -501,7 +501,7 @@ static void preempt_rebind_work_func(struct work_struct *w)bool done = false; err =xe_preempt_work_begin (&exec, vm, &done); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (err || done) {drm_exec_fini (&exec);if (err &&xe_vm_validate_should_retry (&exec, err, &end))
@@ -1052,7 +1052,7 @@ static void xe_vma_destroy_unlocked(struct xe_vma *vma)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { err =xe_vm_lock_vma (&exec, vma); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (XE_WARN_ON (err))break ; }
@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, err =0 ;if (!bo->vm) { err =drm_exec_lock_obj (&exec,xe_vm_obj (vm)); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); }if (!err) { err =drm_exec_lock_obj (&exec, &bo->ttm.base); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); }if (err) {drm_exec_fini (&exec);
@@ -2884,7 +2884,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { err =vm_bind_ioctl_ops_lock_and_prep (&exec, vm, vops); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (err) goto unlock;
include/drm/drm_exec.h +18 -5
@@ -51,6 +51,8 @@ struct drm_exec {struct drm_gem_object *prelocked; }; +int drm_exec_handle_contended (struct drm_exec *exec); +/** * drm_exec_obj() - Return the object for a give drm_exec index * @exec: Pointer to the drm_exec context
@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \/** * drm_exec_retry_on_contention - restart the loop to grap all locks * @exec: drm_exec object +* @_ret: The current error status * * Control flow helper to continue when a contention was detected and we need to * clean up and re-start the loop to prepare all GEM objects. +* +* Return: If no loop restart occurred: The error status. */ -#define drm_exec_retry_on_contention (exec) \ - do { \ - if (unlikely(drm_exec_is_contended(exec))) \ - goto *__drm_exec_retry_ptr; \ - } while (0) +#define drm_exec_retry_on_contention (exec, _ret) \ + ({ \ + struct drm_exec *__exec = (exec); \ + int __ret = (_ret); \ + \ + if (unlikely(drm_exec_is_contended(__exec))) { \ + WARN_ON(__ret != -EDEADLK); \ + __ret = drm_exec_handle_contended(__exec); \ + if (!__ret) \ + goto *__drm_exec_retry_ptr; \ + } \ + __ret; \ + })/** * drm_exec_is_contended - check for contention
Signature · 1 line
2.44.0
Christian König
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 8 lines
If contention and backoff occurs during a
drm_execww transaction, the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction ofdrm_exec_trylock(), that doesn't work, since the locking of the contended lock needs to be a sleeping lock. Neither can we ignore locking the contended lock during a trylock since that would violate at least theww_mutexannotations.
Quoted 6 lines
So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new point of failure since locking the contended lock may return-EINTR.Hence
drm_exec_retry_on_contention()must take an error parameter and also return a value indicating success.
After thinking more about that I have to pretty clearly NAK this.
It's an intentional design decision to guarantee that at the start of the loop no object is locked.
This is because Sima and I wanted to integrate userptr handling into drm_exec as well in the long term.
I think we should just document that drm_exec_trylock() can't be used to lock the first BO in the loop and explicitly WARN if that's the case.
Regards,
Christian.
Quoted 608 lines
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35 ++++++++++++++----- drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++--- 17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +8
-8@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct kgd_mem *mem,drm_exec_init (&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&ctx->exec) { ret =amdgpu_vm_lock_pd (vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; ret =drm_exec_prepare_obj (&ctx->exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; }@@ -1199,14 +1199,14 @@ static int reserve_bo_and_cond_vms(struct kgd_mem *mem, ret =amdgpu_vm_lock_pd (entry->bo_va->base.vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; ++ctx->n_vms; } ret =drm_exec_prepare_obj (&ctx->exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret);if (unlikely (ret)) goto error; }@@ -2619,7 +2619,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info)list_for_each_entry (peer_vm, &process_info->vm_list_head, vm_list_node) { ret =amdgpu_vm_lock_pd (peer_vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) goto unreserve_out; }@@ -2631,7 +2631,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) gobj = &mem->bo->tbo.base; ret =drm_exec_prepare_obj (&exec, gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) goto unreserve_out; }@@ -2875,7 +2875,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *list_for_each_entry (peer_vm, &process_info->vm_list_head, vm_list_node) { ret =amdgpu_vm_lock_pd (peer_vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) {pr_err ("Locking VM PD failed, ret: %d\n" , ret); goto ttm_reserve_fail;@@ -2891,7 +2891,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu * gobj = &mem->bo->tbo.base; ret =drm_exec_prepare_obj (&exec, gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely (ret)) {pr_err ("drm_exec_prepare_obj failed, ret: %d\n" , ret); goto ttm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +3
-3@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, drm_exec_until_all_locked(&p->exec) { r =amdgpu_vm_lock_pd (&fpriv->vm, &p->exec,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages;@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,/* One fence for TTM and one for each CS job */ r =drm_exec_prepare_obj (&p->exec, &e->bo->tbo.base,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages;@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,if (p->uf_bo) { r =drm_exec_prepare_obj (&p->exec, &p->uf_bo->tbo.base,1 + p->gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely (r)) goto out_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +2
-2@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) {DRM_ERROR ("failed to reserve CSA,PD BOs: err=%d\n" , r); goto error;@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) {DRM_ERROR ("failed to reserve CSA,PD BOs: err=%d\n" , r); goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +4
-4@@ -239,12 +239,12 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj,drm_exec_init (&exec,DRM_EXEC_IGNORE_DUPLICATES ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_prepare_obj (&exec, &bo->tbo.base,1 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, drm_exec_until_all_locked(&exec) {if (gobj) { r =drm_exec_lock_obj (&exec, gobj); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; } r =amdgpu_vm_lock_pd (&fpriv->vm, &exec,2 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +4
-4@@ -1221,12 +1221,12 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &ctx_data->meta_data_obj->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; }@@ -1292,12 +1292,12 @@ int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &ctx_data->meta_data_obj->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +2
-2@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct amdgpu_device *adev, struct amdgpu_fpriv *fpriv) r =amdgpu_vm_lock_pd (vm, &exec,0 );if (likely (!r)) r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error; }drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +4
-4@@ -89,12 +89,12 @@ static int map_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto error_fini_exec; }@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { r =drm_exec_lock_obj (&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; r =amdgpu_vm_lock_pd (vm, &exec,0 ); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely (r)) goto out_unlock; }drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-1@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm =drm_priv_to_vm (pdd->drm_priv); r =amdgpu_vm_lock_pd (vm, &ctx->exec,2 ); -drm_exec_retry_on_contention (&ctx->exec); + r =drm_exec_retry_on_contention (&ctx->exec, r);if (unlikely (r)) {pr_debug ("failed %d to reserve bo\n" , r); goto unreserve_out;drivers/gpu/drm/drm_exec.c +26
-9@@ -28,12 +28,12 @@* drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); -* drm_exec_retry_on_contention(&exec); +* ret = drm_exec_retry_on_contention(&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); -* drm_exec_retry_on_contention(&exec); +* ret = drm_exec_retry_on_contention(&exec, ret); * if (ret) * goto error; * } @@ -48,7 +48,8 @@*/ /* Dummy value used to initially enter the retry loop */ -#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)/* Unlock all objects and drop references */ static void drm_exec_unlock_all (struct drm_exec *exec)@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec)return true; } -drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true; }EXPORT_SYMBOL (drm_exec_cleanup);@@ -194,6 +194,27 @@ static int drm_exec_lock_contended(struct drm_exec *exec)return ret; } +/** +* drm_exec_handle_contended() - Perform cleanup before a ww transaction restart +* @exec: Pointer to the drm_exec object. +* +* Unlocks all held resvs and re-locks the contended object. +* +* Return: 0 on success, negative error code on failure. +*/ +int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/** * drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state @@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) {int ret; - ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (exec->prelocked == obj) {drm_gem_object_put (exec->prelocked); exec->prelocked =NULL ;drivers/gpu/drm/drm_gpuvm.c +4
-4@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct drm_gpuvm_exec *vm_exec)drm_exec_until_all_locked (exec) { ret =drm_gpuvm_prepare_vm (gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; ret =drm_gpuvm_prepare_objects (gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err;if (vm_exec->extra.fn) { ret = vm_exec->extra.fn (vm_exec); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; }@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec,drm_exec_until_all_locked (exec) { ret =drm_gpuvm_prepare_range (gpuvm, exec, addr, range, vm_exec->num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) goto err; }drivers/gpu/drm/imagination/pvr_job.c +1
-1@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data,drm_exec_until_all_locked (exec) {int err =jobs_lock_all_objs (exec, job_data, job_count); -drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention (exec, err);if (err)return err; }drivers/gpu/drm/msm/msm_gem_submit.c +1
-1@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit)for (unsigned i =0 ; i < submit->nr_bos; i++) {struct drm_gem_object *obj = submit->bos[i].obj; ret =drm_exec_prepare_obj (&submit->exec, obj,1 ); -drm_exec_retry_on_contention (&submit->exec); + ret =drm_exec_retry_on_contention (&submit->exec, ret);if (ret) goto error; }drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job,drm_exec_init (exec, vme->flags,0 );drm_exec_until_all_locked (exec) { ret =bind_lock_validate (job, exec, vme->num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) { op =list_last_op (&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +6
-6@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;drm_exec_unlock_obj (&exec, &gobj); ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;@@ -110,13 +110,13 @@ static void test_duplicates(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_IGNORE_DUPLICATES ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ; ret =drm_exec_lock_obj (&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test)drm_exec_init (&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret =drm_exec_prepare_obj (&exec, &gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);KUNIT_EXPECT_EQ (test, ret,0 );if (ret)break ;drivers/gpu/drm/xe/xe_gt_pagefault.c +2
-2@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { ret =xe_pf_begin (&exec, vma, atomic, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (ret) goto unlock_dma_resv;@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { ret =xe_pf_begin (&exec, vma, true, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (ret)break ; }drivers/gpu/drm/xe/xe_vm.c +5
-5@@ -501,7 +501,7 @@ static void preempt_rebind_work_func(struct work_struct *w)bool done = false; err =xe_preempt_work_begin (&exec, vm, &done); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (err || done) {drm_exec_fini (&exec);if (err &&xe_vm_validate_should_retry (&exec, err, &end))@@ -1052,7 +1052,7 @@ static void xe_vma_destroy_unlocked(struct xe_vma *vma)drm_exec_init (&exec,0 ,0 ); drm_exec_until_all_locked(&exec) { err =xe_vm_lock_vma (&exec, vma); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (XE_WARN_ON (err))break ; }@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, err =0 ;if (!bo->vm) { err =drm_exec_lock_obj (&exec,xe_vm_obj (vm)); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); }if (!err) { err =drm_exec_lock_obj (&exec, &bo->ttm.base); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); }if (err) {drm_exec_fini (&exec);@@ -2884,7 +2884,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { err =vm_bind_ioctl_ops_lock_and_prep (&exec, vm, vops); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (err) goto unlock;include/drm/drm_exec.h +18
-5@@ -51,6 +51,8 @@ struct drm_exec {struct drm_gem_object *prelocked; }; +int drm_exec_handle_contended (struct drm_exec *exec); +/** * drm_exec_obj() - Return the object for a give drm_exec index * @exec: Pointer to the drm_exec context @@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \/** * drm_exec_retry_on_contention - restart the loop to grap all locks * @exec: drm_exec object +* @_ret: The current error status * * Control flow helper to continue when a contention was detected and we need to * clean up and re-start the loop to prepare all GEM objects. +* +* Return: If no loop restart occurred: The error status. */ -#define drm_exec_retry_on_contention (exec) \ - do { \ - if (unlikely(drm_exec_is_contended(exec))) \ - goto *__drm_exec_retry_ptr; \ - } while (0) +#define drm_exec_retry_on_contention (exec, _ret) \ + ({ \ + struct drm_exec *__exec = (exec); \ + int __ret = (_ret); \ + \ + if (unlikely(drm_exec_is_contended(__exec))) { \ + WARN_ON(__ret != -EDEADLK); \ + __ret = drm_exec_handle_contended(__exec); \ + if (!__ret) \ + goto *__drm_exec_retry_ptr; \ + } \ + __ret; \ + })/** * drm_exec_is_contended - check for contention
Thomas Hellström
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Quoted 15 lines · Christian König
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 14 lines
If contention and backoff occurs during a
drm_execww transaction, the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction ofdrm_exec_trylock(), that doesn't work, since the locking of the contended lock needs to be a sleeping lock. Neither can we ignore locking the contended lock during a trylock since that would violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new point
of failure since locking the contended lock may return-EINTR.
Quoted 6 lines
Quoted 3 lines
Hence
drm_exec_retry_on_contention()must take an error parameter and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
I thought we were beyond upfront NAKing in the first reply :/
Quoted 5 lines
It's an intentional design decision to guarantee that at the start of the loop no object is locked.
This is because Sima and I wanted to integrate userptr handling into
drm_execas well in the long term.
First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a gpuvm helper (without using GEM objects) as we've discussed previously. Anyway if still there would be helpers in drm_exec for some other generic userptr solution, those need to be done before the ww_acquire_ctx_init(). The contended locking here is done after, so I can't really see how these would clash.
Still, If we need to come up with another solution, I think it's fair we clearly sort out why.
Quoted 3 lines
I think we should just document that
drm_exec_trylock()can't be used to
lock the first BO in the loop and explicitly WARN if that's the case.
Unfortunately that's not sufficient for the general use-case. If we want to keep the ttm_bo_vm approach of dropping the mmap lock when there is contention on the bo resv, we need to be able to trylock on first lock. Also bo creation is using trylock but might be able to use a sleeping lock there. But if that sleeping lock triggers an -EDEADLK (DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an object that never was fully created as a contending object.
So the only really working alternative solution I can see is that drm_exec_trylock simply fails if there is a contended lock and we'd need to live with the weird bo creation situation described above.
/Thomas
Quoted 740 lines
Regards,
Christian.Quoted 735 lines
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++--drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++---drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35
++++++++++++++-----
drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++---17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct kgd_mem *mem, drm_exec_init(&ctx->exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&ctx->exec) { ret = amdgpu_vm_lock_pd(vm, &ctx->exec, 2); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret); if (unlikely(ret)) goto error;
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static int reserve_bo_and_cond_vms(struct kgd_mem *mem,
ret = amdgpu_vm_lock_pd(entry->bo_va-Quoted 1 line
base.vm,
&ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error; ++ctx->n_vms;
}
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info)list_for_each_entry(peer_vm, &process_info-
Quoted 1 line
vm_list_head,
vm_list_node) { ret =amdgpu_vm_lock_pd(peer_vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info)
gobj =&mem->bo->tbo.base; ret =drm_exec_prepare_obj(&exec, gobj, 1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret =amdgpu_vm_lock_pd(peer_vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) { pr_err("Locking VM PD failed, ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
gobj =&mem->bo->tbo.base; ret =drm_exec_prepare_obj(&exec, gobj, 1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) {pr_err("drm_exec_prepare_obj failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
drm_exec_until_all_locked(&p->exec){
r = amdgpu_vm_lock_pd(&fpriv->vm,&p->exec, 1 + p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, /* One fence for TTM and one for each CSjob */
r = drm_exec_prepare_obj(&p->exec, &e->bo-Quoted 1 line
tbo.base,
1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->uf_bo) {r = drm_exec_prepare_obj(&p->exec, &p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) {
DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) {
DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1@@ -239,12 +239,12 @@ static void amdgpu_gem_object_close(structdrm_gem_object *obj, drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_prepare_obj(&exec, &bo->tbo.base, 1); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, drm_exec_until_all_locked(&exec) { if (gobj) { r = drm_exec_lock_obj(&exec, gobj); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error; }
r =amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec,&ctx_data->meta_data_obj-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec,&ctx_data->meta_data_obj-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev,struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct amdgpu_device *adev, struct amdgpu_fpriv *fpriv) r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1@@ -89,12 +89,12 @@ static int map_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm = drm_priv_to_vm(pdd->drm_priv);
r =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + r = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec, r);
if (unlikely(r)) {
pr_debug("failed %d to reserve bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +2
-2@@ -28,12 +28,12 @@ * drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * }@@ -48,7 +48,8 @@*/
/* Dummy value used to initially enter the retry loop */drivers/gpu/drm/drm_exec.c +2
-1-#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)
/* Unlock all objects and drop references */ static voiddrm_exec_unlock_all(struct drm_exec *exec)drivers/gpu/drm/drm_exec.c +0
-0@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; }drivers/gpu/drm/drm_exec.c +1
-2-drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true;
}
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static int drm_exec_lock_contended(struct drm_exec *exec) return ret; }
+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/**
*drm_exec_lock_obj- lock a GEM object for use
* @exec: thedrm_execobject with the statedrivers/gpu/drm/drm_exec.c +0
-0@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) { int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (
exec->prelocked== obj) {drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct drm_gpuvm_exec *vm_exec)
drm_exec_until_all_locked(exec){
ret =drm_gpuvm_prepare_vm(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
ret =drm_gpuvm_prepare_objects(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
if (vm_exec->extra.fn) {
ret =vm_exec->extra.fn(vm_exec);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range,vm_exec-
Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data, drm_exec_until_all_locked(exec) { int err = jobs_lock_all_objs(exec, job_data, job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention (exec, err);if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) {struct
drm_gem_object*obj = submit-Quoted 1 line
bos[i].obj;
ret =
drm_exec_prepare_obj(&submit->exec, obj, 1);drivers/gpu/drm/msm/msm_gem_submit.c +0
-1- drm_exec_retry_on_contention(&submit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +0
-0@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) {ret = bind_lock_validate(job, exec, vme-
Quoted 1 line
num_fences);drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) {
op =list_last_op(&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +2
-2@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -110,13 +110,13 @@ static void test_duplicates(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj, 1); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { ret = xe_pf_begin(&exec, vma, atomic, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); if (ret) goto unlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { ret = xe_pf_begin(&exec, vma, true, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); if (ret) break;}
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static void preempt_rebind_work_func(struct work_struct *w) bool done = false;
err =xe_preempt_work_begin(&exec, vm, &done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err);if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -1052,7 +1052,7 @@ static void xe_vma_destroy_unlocked(struct xe_vma *vma) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { err = xe_vm_lock_vma(&exec, vma); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); if (XE_WARN_ON(err)) break;}
drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, err = 0; if (!bo->vm) { err = drm_exec_lock_obj(&exec,xe_vm_obj (vm)); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); } if (!err) { err = drm_exec_lock_obj(&exec,
&bo->ttm.base);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -2884,7 +2884,7 @@ static int vm_bind_ioctl_ops_execute(structxe_vm *vm, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { err = vm_bind_ioctl_ops_lock_and_prep(&exec, vm, vops); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); if (err)goto unlock;
include/drm/drm_exec.h +0
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; };include/drm/drm_exec.h +2
-0+int drm_exec_handle_contended (struct drm_exec *exec); +/**
*drm_exec_obj()- Return the object for a givedrm_execindex
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap all locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was detected and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ *
+ * Return: If no loop restart occurred: The error status. */include/drm/drm_exec.h +1
-5-#define drm_exec_retry_on_contention (exec) \ - do { \ - if (unlikely(drm_exec_is_contended(exec))) \ - goto *__drm_exec_retry_ptr; \ - } while (0) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+if (unlikely (drm_exec_is_contended (__exec))){ \
include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ })
/**
*drm_exec_is_contended- check for contention
Christian König
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 16 lines · Christian König
Quoted 15 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 14 lines
If contention and backoff occurs during a
drm_execww transaction, the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction ofdrm_exec_trylock(), that doesn't work, since the locking of the contended lock needs to be a sleeping lock. Neither can we ignore locking the contended lock during a trylock since that would violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new point
of failure since locking the contended lock may return-EINTR.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Hence
drm_exec_retry_on_contention()must take an error parameter and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach before.
I was a bit annoyed seeing that again. But could as well be that my response never got out or that I'm mixing things up.
Quoted 8 lines
Quoted 5 lines
It's an intentional design decision to guarantee that at the start of the loop no object is locked.
This is because Sima and I wanted to integrate userptr handling into
drm_execas well in the long term.First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a gpuvm helper (without using GEM objects) as we've discussed previously.
We might be talking past each other. That sounds like SVM, e.g. on demand paging.
What I mean is pre-faulting during command submission like radeon, amdgpu and i915 do for the userptr handling.
For that you need to re-start the whole handling similar to how you need to re-start for the mutex locking when you detect that the page array is stale, the difference is that you are not allowed to hold any resv locks while pre-faulting.
That's why it is a requirement that the drm_exec loop starts without any locks held.
Quoted 4 lines
Anyway if still there would be helpers in
drm_execfor some other generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after, so I can't really see how these would clash.
Yes, that indeed was a problem. The ww_acquire_ctx_init() was intentionally moved into drm_exec_cleanup() to partially prevent that issue.
I haven't fully figured out how to do handle everything exactly, but at least in principle it can be made work. With this change here it becomes impossible.
Quoted 10 lines
Still, If we need to come up with another solution, I think it's fair we clearly sort out why.
Quoted 3 lines
I think we should just document that
drm_exec_trylock()can't be used to
lock the first BO in the loop and explicitly WARN if that's the case.Unfortunately that's not sufficient for the general use-case. If we want to keep the
ttm_bo_vmapproach of dropping the mmap lock when there is contention on the bo resv, we need to be able to trylock on first lock.
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the mmap() and dma_resv() locking order long ago. See dma_resv_lockdep() which is enforcing that.
Quoted 4 lines
Also bo creation is using trylock but might be able to use
a sleeping lock there. But if that sleeping lock triggers an-EDEADLK(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an object that never was fully created as a contending object.
I wanted to eliminate that as well by not validating the BO during initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked() {
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}
if (r)
ttm_bo_put(bo);
return r;
I have that on a branch here somewhere prepared, but never got the time to clean it up.
Regards,
Christian.
Quoted 744 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and we'd need to live with the weird bo creation situation described above./Thomas
Quoted 737 lines
Regards,
Christian.Quoted 734 lines
Cc Christian König<[email protected]>Cc Somalapuram Amaranath<[email protected]>Cc Matthew Brost<[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström<[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35 ++++++++++++++----- drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++--- 17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2
-2@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct kgd_mem *mem, drm_exec_init(&ctx->exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&ctx->exec) { ret = amdgpu_vm_lock_pd(vm, &ctx->exec, 2); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx->exec, ret); if (unlikely(ret)) goto error; ret = drm_exec_prepare_obj(&ctx->exec, &bo- > tbo.base, 1); -drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2
-2@@ -1199,14 +1199,14 @@ static int reserve_bo_and_cond_vms(structkgd_mem *mem, ret = amdgpu_vm_lock_pd(entry->bo_va- > base.vm, &ctx->exec, 2); -drm_exec_retry_on_contention (&ctx->exec); + ret =drm_exec_retry_on_contention (&ctx- > exec, ret); if (unlikely(ret)) goto error; ++ctx->n_vms; } ret = drm_exec_prepare_obj(&ctx->exec, &bo- > tbo.base, 1); -drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1@@ -2619,7 +2619,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) list_for_each_entry(peer_vm, &process_info- > vm_list_head, vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec, 2); - drm_exec_retry_on_contention(&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1@@ -2631,7 +2631,7 @@ static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) gobj = &mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);if (unlikely(ret)) goto
unreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) { pr_err("Locking VM PD failed, ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*gobj =
&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (unlikely(ret)) { pr_err("drm_exec_prepare_obj
failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(structamdgpu_cs_parser *p, drm_exec_until_all_locked(&p->exec) { r = amdgpu_vm_lock_pd(&fpriv->vm, &p->exec, 1 + p- > gang_size); -drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r); if (unlikely(r))goto
out_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, /* One fence for TTM and one for each CSjob */
r = drm_exec_prepare_obj(&p->exec, &e->bo-Quoted 1 line
tbo.base,
1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->uf_bo) { r = drm_exec_prepare_obj(&p->exec, &p- > uf_bo->tbo.base, 1 + p- > gang_size); -drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) {DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) {DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +2
-2@@ -239,12 +239,12 @@ static void amdgpu_gem_object_close(structdrm_gem_object *obj, drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_prepare_obj(&exec, &bo->tbo.base, 1); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;}
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +2
-2@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, drm_exec_until_all_locked(&exec) { if (gobj) { r = drm_exec_lock_obj(&exec, gobj); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error; } r = amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r))goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +2
-2@@ -1221,12 +1221,12 @@ int amdgpu_mes_ctx_map_meta_data(structamdgpu_device *adev, drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &ctx_data->meta_data_obj- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r))goto
error_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +2
-2@@ -1292,12 +1292,12 @@ int amdgpu_mes_ctx_unmap_meta_data(structamdgpu_device *adev, drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &ctx_data->meta_data_obj- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r))goto
out_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev,struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r))goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct amdgpu_device *adev, struct amdgpu_fpriv *fpriv) r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r))goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +2
-2@@ -89,12 +89,12 @@ static int map_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec;}
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +2
-2@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;}
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-1@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm = drm_priv_to_vm(pdd->drm_priv); r = amdgpu_vm_lock_pd(vm, &ctx->exec, 2); -drm_exec_retry_on_contention (&ctx->exec); + r =drm_exec_retry_on_contention (&ctx- > exec, r); if (unlikely(r)) {pr_debug("failed %d to reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +7
-5@@ -28,12 +28,12 @@ * drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * }@@ -48,7 +48,8 @@*/ /* Dummy value used to initially enter the retry loop */ -#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK) /* Unlock all objects and drop references */ static void drm_exec_unlock_all(struct drm_exec *exec)@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; } -drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ; return true; } EXPORT_SYMBOL(drm_exec_cleanup);@@ -194,6 +194,27 @@ static int drm_exec_lock_contended(struct drm_exec *exec) return ret; } +/** +* drm_exec_handle_contended() - Perform cleanup before a ww transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-4+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); + /*** drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state @@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) { int ret; - ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; - if (exec->prelocked == obj) { drm_gem_object_put(exec->prelocked);
exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +3
-3@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct drm_gpuvm_exec *vm_exec) drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_vm(gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret); if (ret) goto err; ret = drm_gpuvm_prepare_objects(gpuvm, exec, num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret); if (ret) goto err; if (vm_exec->extra.fn) { ret = vm_exec->extra.fn(vm_exec); -drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +1
-1@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(structdrm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range, vm_exec- > num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +1
-1@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data, drm_exec_until_all_locked(exec) { int err = jobs_lock_all_objs(exec, job_data, job_count); -drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention (exec, err); if (err)return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-1@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) { struct drm_gem_object *obj = submit- > bos[i].obj; ret = drm_exec_prepare_obj(&submit->exec, obj,1 ); - drm_exec_retry_on_contention(&submit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) { ret = bind_lock_validate(job, exec, vme- > num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret); if (ret) {op =
list_last_op(&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +3
-3@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break; drm_exec_unlock_obj(&exec, &gobj); ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret)break;
drivers/gpu/drm/tests/drm_exec_test.c +2
-2@@ -110,13 +110,13 @@ static void test_duplicates(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break; ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret)break;
drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj, 1); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;drivers/gpu/drm/xe/xe_gt_pagefault.c +2
-2@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { ret = xe_pf_begin(&exec, vma, atomic, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); if (ret) goto unlock_dma_resv;@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { ret = xe_pf_begin(&exec, vma, true, tile->id); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); if (ret) break;}
drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -501,7 +501,7 @@ static void preempt_rebind_work_func(struct work_struct *w) bool done = false; err = xe_preempt_work_begin(&exec, vm, &done); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); if (err || done) { drm_exec_fini(&exec);if (err &&
xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -1052,7 +1052,7 @@ static void xe_vma_destroy_unlocked(struct xe_vma *vma) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { err = xe_vm_lock_vma(&exec, vma); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); if (XE_WARN_ON(err)) break;}
drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, err = 0; if (!bo->vm) { err = drm_exec_lock_obj(&exec,xe_vm_obj (vm)); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); } if (!err) { err = drm_exec_lock_obj(&exec,
&bo->ttm.base);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -2884,7 +2884,7 @@ static int vm_bind_ioctl_ops_execute(structxe_vm *vm, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { err = vm_bind_ioctl_ops_lock_and_prep(&exec, vm, vops); -drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention (&exec, err); if (err)goto unlock;
include/drm/drm_exec.h +2
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; }; +int drm_exec_handle_contended (struct drm_exec *exec); + /*** drm_exec_obj() - Return the object for a give drm_exec index * @exec: Pointer to the drm_exec context @@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap all locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was detected
and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ *
+ * Return: If no loop restart occurred: The error status. */include/drm/drm_exec.h +1
-5-#define drm_exec_retry_on_contention (exec) \ - do { \ - if (unlikely(drm_exec_is_contended(exec))) \ - goto *__drm_exec_retry_ptr; \ - } while (0) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+if (unlikely (drm_exec_is_contended (__exec))){ \
include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ }) /*** drm_exec_is_contended - check for contention
Thomas Hellström
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Quoted 26 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 25 lines · Christian König
Quoted 24 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 17 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction ofdrm_exec_trylock(), that doesn't work, since the locking of the contended lock needs to be a sleeping lock. Neither can we ignore
locking the contended lock during a trylock since that would violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new point
of failure since locking the contended lock may return-EINTR.Quoted 4 lines
Hence
drm_exec_retry_on_contention()must take an error parameter
and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
Quoted 8 lines
Quoted 1 line
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach
before.I was a bit annoyed seeing that again. But could as well be that my response never got out or that I'm mixing things up.
I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
It's an intentional design decision to guarantee that at the start of
the loop no object is locked.Quoted 7 lines
Quoted 3 lines
This is because Sima and I wanted to integrate userptr handling into
drm_execas well in the long term.First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a gpuvm
helper (without using GEM objects) as we've discussed previously.
Quoted 5 lines
We might be talking past each other. That sounds like SVM, e.g. on demand paging.
What I mean is pre-faulting during command submission like radeon, amdgpu and i915 do for the userptr handling.
Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in drm_gpuvm for this sort of stuff. And I think that's the concluding argument by Sima as well.
In any case, If the planned drm_exec development is to be a full execbuf helper, I think we need a capable sub-helper for ONLY the ww transaction locking as well, with support for the various locking primitives. In particular if we're going to be able to port i915 ww transaction locking over. There are more uses of the ww locking transacions than execbuf.
Quoted 9 lines
For that you need to re-start the whole handling similar to how you need
to re-start for the mutex locking when you detect that the page array is
stale, the difference is that you are not allowed to hold any resv locks
while pre-faulting.
Quoted 3 lines
That's why it is a requirement that the
drm_execloop starts without any
locks held.
But wouldn't you need an outer (userptr) loop and an inner (ww_transaction) loop for this? Why would we want to re-validate userptrs on -EDEADLKS?
Quoted 32 lines
Quoted 6 lines
Anyway if still there would be helpers in
drm_execfor some other generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after, so I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was intentionally moved intodrm_exec_cleanup()to partially prevent that issue.I haven't fully figured out how to do handle everything exactly, but at
least in principle it can be made work. With this change here it becomes
impossible.Quoted 9 lines
Still, If we need to come up with another solution, I think it's fair
we clearly sort out why.Quoted 5 lines
I think we should just document that
drm_exec_trylock()can't be used
to
lock the first BO in the loop and explicitly WARN if that's the case.Quoted 5 lines
Unfortunately that's not sufficient for the general use-case. If we want to keep the
ttm_bo_vmapproach of dropping the mmap lock when there is contention on the bo resv, we need to be able to trylock on
first lock.
Quoted 5 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order long
ago. Seedma_resv_lockdep()which is enforcing that.
I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In particular not while waiting for mutexes that may be blocked by gpu activity.
Quoted 22 lines
Quoted 7 lines
Also bo creation is using trylock but might be able to use a sleeping lock there. But if that sleeping lock triggers an - EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}
Quoted 8 lines
if (r)
ttm_bo_put(bo);return r;
I have that on a branch here somewhere prepared, but never got the time
to clean it up.
Still, bo creation and validation may be part of a ww transaction as well, like page-table bos (Although those are pre-locked so perhaps not a good example). But in the general case, I'm not sure this is sufficient for all use-cases.
/Thomas
Quoted 830 lines
Regards,
Christian.Quoted 826 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and we'd need to live with the weird bo creation situation described above./Thomas
Quoted 819 lines
Regards,
Christian.Quoted 816 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>
---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++--drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++---drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35
++++++++++++++-----
drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23
+++++++++---17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0);
drm_exec_until_all_locked(&ctx->exec){
ret =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,
ret = amdgpu_vm_lock_pd(entry->bo_va-Quoted 1 line
base.vm,
&ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error; ++ctx->n_vms;
}
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx->exec,ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD failed, ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) {pr_err("drm_exec_prepare_obj failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
drm_exec_until_all_locked(&p->exec){
r = amdgpu_vm_lock_pd(&fpriv->vm,&p->exec, 1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r =drm_exec_retry_on_contention (&p->exec, r);if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, /* One fence for TTM and one for eachCS
job */
r = drm_exec_prepare_obj(&p->exec, &e-Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->uf_bo) {r = drm_exec_prepare_obj(&p->exec, &p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
r =drm_exec_prepare_obj(&exec, &bo->tbo.base, 1);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, drm_exec_until_all_locked(&exec) { if (gobj) { r = drm_exec_lock_obj(&exec, gobj); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error; }
r =amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec,&ctx_data-
Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
r =drm_exec_lock_obj(&exec, &bo->tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm = drm_priv_to_vm(pdd->drm_priv);
r =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec, r);
if (unlikely(r)) {
pr_debug("failed %d to reserve bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +2
-2@@ -28,12 +28,12 @@ * drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret)* goto error;
* }drivers/gpu/drm/drm_exec.c +0
-0@@ -48,7 +48,8 @@*/
/* Dummy value used to initially enter the retry loop */drivers/gpu/drm/drm_exec.c +2
-1-#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)
/* Unlock all objects and drop references */ static voiddrm_exec_unlock_all(struct drm_exec *exec)drivers/gpu/drm/drm_exec.c +0
-0@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; }drivers/gpu/drm/drm_exec.c +1
-2-drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true;
}
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static int drm_exec_lock_contended(struct drm_exec *exec) return ret; }
+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/**
*drm_exec_lock_obj- lock a GEM object for use
* @exec: thedrm_execobject with the statedrivers/gpu/drm/drm_exec.c +0
-0@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) { int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (
exec->prelocked== obj) {drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret =drm_gpuvm_prepare_vm(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
ret =drm_gpuvm_prepare_objects(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
if (vm_exec->extra.fn) {
ret =vm_exec->extra.fn(vm_exec);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range,vm_exec-
Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data, drm_exec_until_all_locked(exec) { int err = jobs_lock_all_objs(exec, job_data, job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention (exec, err);if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) {struct
drm_gem_object*obj = submit-Quoted 1 line
bos[i].obj;
ret = drm_exec_prepare_obj(&submit-
Quoted 1 line
exec,
obj, 1);
drivers/gpu/drm/msm/msm_gem_submit.c +0
-1- drm_exec_retry_on_contention(&submit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +0
-0@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) {ret = bind_lock_validate(job, exec, vme-
Quoted 1 line
num_fences);drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) {
op =list_last_op(&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret)break;
drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -110,13 +110,13 @@ static void test_duplicates(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, 0); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj, 1); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0);if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) {ret = xe_pf_begin(&exec, vma, atomic, tile-
Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt,struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
ret =xe_pf_begin(&exec, vma, true, tile->id);drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static void preempt_rebind_work_func(struct work_struct *w) bool done = false;
err =xe_preempt_work_begin(&exec, vm, &done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err =drm_exec_lock_obj(&exec, xe_vm_obj(vm));
-
drm_exec_retry_on_contention(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err =drm_exec_lock_obj(&exec, &bo->ttm.base);
-
drm_exec_retry_on_contention(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
err = vm_bind_ioctl_ops_lock_and_prep(&exec, vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err)
goto unlock;include/drm/drm_exec.h +0
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; };include/drm/drm_exec.h +2
-0+int drm_exec_handle_contended (struct drm_exec *exec); +/**
*drm_exec_obj()- Return the object for a givedrm_execindex
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was detected
and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ *
+ * Return: If no loop restart occurred: The error status. */
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \
include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+if (unlikely (drm_exec_is_contended (__exec))){ \
include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ })
/**
*drm_exec_is_contended- check for contention
Thomas Hellström
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Hi, Christian.
I'd appreciate if you could respond to the below, since it is a bit hard to try to design around a problem I don't believe exists, and come up with a good solution for that.
In short.
1) To prefault userptr we have to exit the ww transaction anyway.
2) Any contended lock held at loop start is completely encapsulated in the ww transaction and can and will be unlocked when exiting it, so this patch doesn't introduce any additional problems for userptr handling AFAICT.
3) The need for a fully capable ww transaction helper moving forward. If we need a tool that also does userptr locking, then I think we need to separate that from the ww transaction tool and only pass the latter around to TTM.
Thanks,
Thomas
Quoted 1,065 lines · Thomas Hellström
Quoted 31 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 30 lines · Christian König
Quoted 29 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 21 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction
ofdrm_exec_trylock(), that doesn't work, since the locking of the
contended lock needs to be a sleeping lock. Neither can we ignore
locking the contended lock during a trylock since that would violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new
point
of failure since locking the contended lock may return - EINTR.Quoted 4 lines
Hence
drm_exec_retry_on_contention()must take an error parameter
and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
Quoted 8 lines
Quoted 1 line
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach
before.I was a bit annoyed seeing that again. But could as well be that my response never got out or that I'm mixing things up.
I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
It's an intentional design decision to guarantee that at the start of
the loop no object is locked.Quoted 7 lines
Quoted 3 lines
This is because Sima and I wanted to integrate userptr handling into
drm_execas well in the long term.First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a gpuvm
helper (without using GEM objects) as we've discussed previously.Quoted 5 lines
We might be talking past each other. That sounds like SVM, e.g. on demand paging.
What I mean is pre-faulting during command submission like radeon, amdgpu and i915 do for the userptr handling.
Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in
drm_gpuvmfor this sort of stuff. And I think that's the concluding argument by Sima as well.In any case, If the planned
drm_execdevelopment is to be a full execbuf helper, I think we need a capable sub-helper for ONLY the ww transaction locking as well, with support for the various locking primitives. In particular if we're going to be able to port i915 ww transaction locking over. There are more uses of the ww locking transacions than execbuf.Quoted 10 lines
For that you need to re-start the whole handling similar to how you need
to re-start for the mutex locking when you detect that the page array
is
stale, the difference is that you are not allowed to hold any resv locks
while pre-faulting.Quoted 4 lines
That's why it is a requirement that the
drm_execloop starts without
any
locks held.But wouldn't you need an outer (userptr) loop and an inner (
ww_transaction) loop for this? Why would we want to re-validate userptrs on-EDEADLKS?Quoted 38 lines
Quoted 7 lines
Anyway if still there would be helpers in
drm_execfor some other generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after, so
I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was intentionally moved intodrm_exec_cleanup()to partially prevent that
issue.I haven't fully figured out how to do handle everything exactly, but
at
least in principle it can be made work. With this change here it becomes
impossible.Quoted 10 lines
Still, If we need to come up with another solution, I think it's fair
we clearly sort out why.Quoted 6 lines
I think we should just document that
drm_exec_trylock()can't be
used
to
lock the first BO in the loop and explicitly WARN if that's the case.Quoted 7 lines
Unfortunately that's not sufficient for the general use-case. If we
want to keep thettm_bo_vmapproach of dropping the mmap lock when
there is contention on the bo resv, we need to be able to trylock on
first lock.Quoted 5 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order long
ago. Seedma_resv_lockdep()which is enforcing that.I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In particular not while waiting for mutexes that may be blocked by gpu activity.
Quoted 22 lines
Quoted 7 lines
Also bo creation is using trylock but might be able to use a sleeping lock there. But if that sleeping lock triggers an - EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}Quoted 8 lines
if (r)
ttm_bo_put(bo);return r;
I have that on a branch here somewhere prepared, but never got the time
to clean it up.Still, bo creation and validation may be part of a ww transaction as well, like page-table bos (Although those are pre-locked so perhaps not
a good example). But in the general case, I'm not sure this is sufficient for all use-cases./Thomas
Quoted 870 lines
Regards,
Christian.Quoted 866 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and we'd
need to live with the weird bo creation situation described above./Thomas
Quoted 857 lines
Regards,
Christian.Quoted 854 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>
---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++--- --
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++--drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++---drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35
++++++++++++++-----
drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23
+++++++++---17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&ctx->exec){ ret =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,
ret = amdgpu_vm_lock_pd(entry-Quoted 2 lines
bo_va-
base.vm,
&ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error; ++ctx->n_vms;
}
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD failed,
ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec, gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) {pr_err("drm_exec_prepare_obj failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
drm_exec_until_all_locked(&p->exec){
r = amdgpu_vm_lock_pd(&fpriv->vm,&p->exec, 1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, /* One fence for TTM and one foreach
CS
job */
r = drm_exec_prepare_obj(&p->exec, &e-Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->uf_bo) { r = drm_exec_prepare_obj(&p->exec,&p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
r = drm_exec_prepare_obj(&exec, &bo-Quoted 1 line
tbo.base,
1);
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct
drm_device
*dev, void *data,
drm_exec_until_all_locked(&exec){
if (gobj) {
r =drm_exec_lock_obj(&exec, gobj);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}
r =amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ intamdgpu_mes_ctx_map_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
r =drm_exec_lock_obj(&exec, &bo->tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm = drm_priv_to_vm(pdd->drm_priv);
r = amdgpu_vm_lock_pd(vm, &ctx-Quoted 1 line
exec,
2);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, r);
if (unlikely(r)) { pr_debug("failed %d to reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +2
-2@@ -28,12 +28,12 @@ * drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret)* goto error;
* }drivers/gpu/drm/drm_exec.c +0
-0@@ -48,7 +48,8 @@*/
/* Dummy value used to initially enter the retry loop */drivers/gpu/drm/drm_exec.c +2
-1-#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)
/* Unlock all objects and drop references */ static voiddrm_exec_unlock_all(struct drm_exec *exec)drivers/gpu/drm/drm_exec.c +0
-0@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; }drivers/gpu/drm/drm_exec.c +1
-2-drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true;
}
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static intdrm_exec_lock_contended(struct
drm_exec*exec)
return ret;
}
+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/**
*drm_exec_lock_obj- lock a GEM object for use
* @exec: thedrm_execobject with the statedrivers/gpu/drm/drm_exec.c +0
-0@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) { int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (
exec->prelocked== obj) {drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret =drm_gpuvm_prepare_vm(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
ret =drm_gpuvm_prepare_objects(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
if (vm_exec->extra.fn) {
ret =vm_exec->extra.fn(vm_exec);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range,vm_exec-
Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct
drm_exec
*exec, structpvr_job_data*job_data,drm_exec_until_all_locked(exec){
int err =jobs_lock_all_objs(exec, job_data, job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err = drm_exec_retry_on_contention(exec,err);
if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) {struct
drm_gem_object*obj = submit-Quoted 1 line
bos[i].obj;
ret = drm_exec_prepare_obj(&submit-
Quoted 1 line
exec,
obj, 1);
-
drm_exec_retry_on_contention(&submit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +0
-0@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) {ret = bind_lock_validate(job, exec, vme-
Quoted 1 line
num_fences);drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret) {
op =list_last_op(&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0);if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -110,13 +110,13 @@ static void test_duplicates(structkunit
*test)
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0); if (ret)
break;
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj, 1); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) {ret = xe_pf_begin(&exec, vma, atomic, tile-
Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt,struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
ret = xe_pf_begin(&exec, vma, true, tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static voidpreempt_rebind_work_func(struct
work_struct*w)
bool done = false;
err =xe_preempt_work_begin(&exec, vm, &done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err = drm_exec_lock_obj(&exec,xe_vm_obj(vm));
-
drm_exec_retry_on_contention
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err = drm_exec_lock_obj(&exec,&bo->ttm.base);
-
drm_exec_retry_on_contention
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
err = vm_bind_ioctl_ops_lock_and_prep(&exec, vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err)
goto unlock;include/drm/drm_exec.h +0
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; };include/drm/drm_exec.h +2
-0+int drm_exec_handle_contended (struct drm_exec *exec); +/**
*drm_exec_obj()- Return the object for a givedrm_execindex
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was detected
and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ *
+ * Return: If no loop restart occurred: The error status. */
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \
include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+if (unlikely (drm_exec_is_contended (__exec))){ \
include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ })
/**
*drm_exec_is_contended- check for contention
Christian König
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Hi Thomas,
Am 28.05.24 um 08:36 schrieb Thomas Hellström:
Quoted 8 lines
Hi, Christian.
I'd appreciate if you could respond to the below, since it is a bit hard to try to design around a problem I don't believe exists, and come up with a good solution for that.
In short.
1) To prefault userptr we have to exit the ww transaction anyway.
Yeah and I would rather like to have that handling in drm_exec at some time.
Basically a GEM object with outdated struct pages backing it can be handled in the same loop at the ww transaction.
Quoted 4 lines
2) Any contended lock held at loop start is completely encapsulated in the ww transaction and can and will be unlocked when exiting it, so this patch doesn't introduce any additional problems for userptr handling AFAICT.
The drm_exec object was intentionally design to not have anything locked at the beginning of the loop. See the discussion I had with Sima around that when pushing the drm_exec object upstream.
I would really like to stick with that design and honestly don't see the reason to change that. Contenting on a trylock seem to be much more questionable.
Quoted 4 lines
3) The need for a fully capable ww transaction helper moving forward. If we need a tool that also does userptr locking, then I think we need to separate that from the ww transaction tool and only pass the latter around to TTM.
drm_exec is *not* meant to be a ww_transaction helper.
The functionality here is to support drivers in their CS interface and that includes userptr handling as well as a couple of other things.
Regards,
Christian.
Quoted 1,051 lines
Thanks,
ThomasQuoted 1,046 lines · Thomas Hellström
Quoted 31 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 30 lines · Christian König
Quoted 29 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 21 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary attempt to lock adma_resvlock. However, with the introduction
ofdrm_exec_trylock(), that doesn't work, since the locking of the
contended lock needs to be a sleeping lock. Neither can we ignore
locking the contended lock during a trylock since that would violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a new
point
of failure since locking the contended lock may return - EINTR.Quoted 4 lines
Hence
drm_exec_retry_on_contention()must take an error parameter
and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
Quoted 7 lines
Quoted 1 line
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach
before.I was a bit annoyed seeing that again. But could as well be that my response never got out or that I'm mixing things up.
I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
It's an intentional design decision to guarantee that at the start of
the loop no object is locked.This is because Sima and I wanted to integrate userptr handling into
drm_execas well in the long term.First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a gpuvm
helper (without using GEM objects) as we've discussed previously.Quoted 5 lines
We might be talking past each other. That sounds like SVM, e.g. on demand paging.
What I mean is pre-faulting during command submission like radeon, amdgpu and i915 do for the userptr handling.
Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in
drm_gpuvmfor this sort of stuff. And I think that's the concluding argument by Sima as well.In any case, If the planned
drm_execdevelopment is to be a full execbuf helper, I think we need a capable sub-helper for ONLY the ww transaction locking as well, with support for the various locking primitives. In particular if we're going to be able to port i915 ww transaction locking over. There are more uses of the ww locking transacions than execbuf.Quoted 9 lines
For that you need to re-start the whole handling similar to how you need
to re-start for the mutex locking when you detect that the page array
is
stale, the difference is that you are not allowed to hold any resv locks
while pre-faulting.Quoted 4 lines
That's why it is a requirement that the
drm_execloop starts without
any
locks held.But wouldn't you need an outer (userptr) loop and an inner (
ww_transaction) loop for this? Why would we want to re-validate userptrs on-EDEADLKS?Quoted 35 lines
Quoted 6 lines
Anyway if still there would be helpers in
drm_execfor some other generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after, so
I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was intentionally moved intodrm_exec_cleanup()to partially prevent that
issue.I haven't fully figured out how to do handle everything exactly, but
at
least in principle it can be made work. With this change here it becomes
impossible.Quoted 10 lines
Still, If we need to come up with another solution, I think it's fair
we clearly sort out why.Quoted 6 lines
I think we should just document that
drm_exec_trylock()can't be
used
to
lock the first BO in the loop and explicitly WARN if that's the case.Quoted 7 lines
Unfortunately that's not sufficient for the general use-case. If we
want to keep thettm_bo_vmapproach of dropping the mmap lock when
there is contention on the bo resv, we need to be able to trylock on
first lock.Quoted 5 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order long
ago. Seedma_resv_lockdep()which is enforcing that.I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In particular not while waiting for mutexes that may be blocked by gpu activity.
Quoted 18 lines
Quoted 6 lines
Also bo creation is using trylock but might be able to use
a sleeping lock there. But if that sleeping lock triggers an - EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}Quoted 8 lines
if (r)
ttm_bo_put(bo);return r;
I have that on a branch here somewhere prepared, but never got the time
to clean it up.Still, bo creation and validation may be part of a ww transaction as well, like page-table bos (Although those are pre-locked so perhaps not
a good example). But in the general case, I'm not sure this is sufficient for all use-cases./Thomas
Quoted 868 lines
Regards,
Christian.Quoted 865 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and we'd
need to live with the weird bo creation situation described above./Thomas
Quoted 857 lines
Regards,
Christian.Quoted 854 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++--- -- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35 ++++++++++++++----- drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-- drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++--- 17 files changed, 92 insertions(+), 62 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1152,12 +1152,12 @@ static int reserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&ctx->exec){ ret =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;ret = drm_exec_prepare_obj(&ctx->exec, &bo-
Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,ret = amdgpu_vm_lock_pd(entry-
Quoted 2 lines
bo_va-
base.vm,
&ctx->exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error;
++ctx->n_vms;
}ret = drm_exec_prepare_obj(&ctx->exec, &bo-
Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&ctx->exec); + ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static intvalidate_invalid_user_pages(struct
amdkfd_process_info*process_info)gobj =
&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*
list_for_each_entry(peer_vm, &process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD
failed,
ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*gobj =
&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("drm_exec_prepare_obj
failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, drm_exec_until_all_locked(&p->exec) { r = amdgpu_vm_lock_pd(&fpriv->vm, &p->exec,1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p->exec,r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, /* One fence for TTM and one foreach
CS
job */
r = drm_exec_prepare_obj(&p->exec,
&e-Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->uf_bo) { r = drm_exec_prepare_obj(&p->exec,&p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages; }drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PDBOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) {DRM_ERROR("failed to reserve CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
r = drm_exec_prepare_obj(&exec, &bo-Quoted 1 line
tbo.base,
1);
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +2
-2-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; }@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct
drm_device
*dev, void *data,
drm_exec_until_all_locked(&exec){
if (gobj) {
r =drm_exec_lock_obj(&exec, gobj);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}r =
amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error; }drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ intamdgpu_mes_ctx_map_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +2
-2-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; }@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +2
-2-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; }drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r);if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error; }drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
r =drm_exec_lock_obj(&exec, &bo->tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +4
-4-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto error_fini_exec; }@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo->tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock;}
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-1@@ -1499,7 +1499,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) vm = drm_priv_to_vm(pdd->drm_priv); r = amdgpu_vm_lock_pd(vm, &ctx- > exec,2 ); - drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, r);
if (unlikely(r)) { pr_debug("failed %d to
reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +2
-2@@ -28,12 +28,12 @@ * drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret)* goto error;
* }drivers/gpu/drm/drm_exec.c +3
-3@@ -48,7 +48,8 @@*/ /* Dummy value used to initially enter the retry loop */ -#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK) /* Unlock all objects and drop references */ static void drm_exec_unlock_all(struct drm_exec *exec)@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; } -drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ; return true; }
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static intdrm_exec_lock_contended(struct
drm_exec*exec)
return ret;
}+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-4+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); + /*** drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state @@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) { int ret; - ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; - if (exec->prelocked == obj) {
drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret = drm_gpuvm_prepare_vm(gpuvm, exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;ret = drm_gpuvm_prepare_objects(gpuvm, exec,
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;if (
vm_exec->extra.fn) {
ret =vm_exec->extra.fn(vm_exec);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +1
-1@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(structdrm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range, vm_exec- > num_fences); -drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct
drm_exec
*exec, structpvr_job_data*job_data,
drm_exec_until_all_locked(exec){
int err = jobs_lock_all_objs(exec,job_data,job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err = drm_exec_retry_on_contention(exec,err);
if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) { struct drm_gem_object *obj = submit- > bos[i].obj; ret = drm_exec_prepare_obj(&submit-Quoted 1 line
exec,
obj, 1);
-
drm_exec_retry_on_contention(&submit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) { ret = bind_lock_validate(job, exec, vme- > num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) {
op =list_last_op(&bind_job->ops); goto unwind;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0);if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +2
-2@@ -83,14 +83,14 @@ static void test_lock_unlock(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret); KUNIT_EXPECT_EQ(test, ret, 0); if (ret) break; drm_exec_unlock_obj(&exec, &gobj); ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -110,13 +110,13 @@ static void test_duplicates(structkunit
*test)
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;ret =
drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj, 1); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention (&exec, ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1@@ -200,7 +200,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { ret = xe_pf_begin(&exec, vma, atomic, tile- > id); -drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt,struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){
ret = xe_pf_begin(&exec, vma, true, tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret = drm_exec_retry_on_contention(&exec,ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static voidpreempt_rebind_work_func(struct
work_struct*w)
bool done = false;err = xe_preempt_work_begin(&exec, vm,
&done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err =
drm_exec_lock_obj(&exec,xe_vm_obj(vm));
-
drm_exec_retry_on_contention
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err =
drm_exec_lock_obj(&exec,&bo->ttm.base);
-
drm_exec_retry_on_contention
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){
err = vm_bind_ioctl_ops_lock_and_prep(&exec,
vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err = drm_exec_retry_on_contention(&exec,err);
if (err)
goto unlock;include/drm/drm_exec.h +2
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; }; +int drm_exec_handle_contended (struct drm_exec *exec); + /*** drm_exec_obj() - Return the object for a give drm_exec index
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was
detected
and we need to
* clean up and re-start the loop to prepare all GEM
objects.
+ *
+ * Return: If no loop restart occurred: The error status. */
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \
include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+if (unlikely (drm_exec_is_contended (__exec))){ \
include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ }) /*** drm_exec_is_contended - check for contention
Thomas Hellström
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Quoted 10 lines · Christian König
Hi Thomas,
Am 28.05.24 um 08:36 schrieb Thomas Hellström:
Quoted 7 lines
Hi, Christian.
I'd appreciate if you could respond to the below, since it is a bit hard to try to design around a problem I don't believe exists, and come
up with a good solution for that.
Quoted 8 lines
Quoted 2 lines
In short.
1) To prefault userptr we have to exit the ww transaction anyway.Yeah and I would rather like to have that handling in
drm_execat some time.Basically a GEM object with outdated struct pages backing it can be handled in the same loop at the ww transaction.
OK.
Quoted 13 lines
Quoted 6 lines
2) Any contended lock held at loop start is completely encapsulated in
the ww transaction and can and will be unlocked when exiting it, so this patch doesn't introduce any additional problems for userptr handling AFAICT.The
drm_execobject was intentionally design to not have anything locked
at the beginning of the loop. See the discussion I had with Sima around
that when pushing thedrm_execobject upstream.
Quoted 4 lines
I would really like to stick with that design and honestly don't see the
reason to change that. Contenting on a trylock seem to be much more questionable.
The change here is to make sure we *don't* have contention in a trylock, which is otherwise inherent in the current drm_exec design.
What I'm trying to say here is that we end up with the contended lock grabbed at loop start you already conceptually have a conflicting lock held (the we_class::acquire_key). Both these can be resolved.
Quoted 9 lines
Quoted 8 lines
3) The need for a fully capable ww transaction helper moving forward.
If we need a tool that also does userptr locking, then I think we need
to separate that from the ww transaction tool and only pass the latter
around to TTM.
Quoted 5 lines
drm_execis *not* meant to be aww_transactionhelper.The functionality here is to support drivers in their CS interface and
that includes userptr handling as well as a couple of other things.
Then if so, I don't think drm_exec is the correct functionality to pass to TTM to resolve the eviction issues, but rather a ww transaction helper that can be used standalone *and* by drm_exec. Now the functionality would be more or less what drm exec is today, but slightly augmented.
But then IMHO instead of changing name and more or less replicating what drm_exec is today wouldn't it be a better idea to subclass drm_exec into a full-fledged CS helper at the time when that functionality is indeed added?
/Thomas
Quoted 1,219 lines
Regards,
Christian.Quoted 1,214 lines
Thanks,
ThomasQuoted 1,209 lines · Thomas Hellström
Quoted 37 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 36 lines · Christian König
Quoted 35 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 27 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary
attempt to lock adma_resvlock. However, with the introduction
ofdrm_exec_trylock(), that doesn't work, since the locking of
the
contended lock needs to be a sleeping lock. Neither can we
ignore
locking the contended lock during a trylock since that would
violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a
new
point
of failure since locking the contended lock may return - EINTR.Quoted 4 lines
Hence
drm_exec_retry_on_contention()must take an error parameter
and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
Quoted 8 lines
Quoted 1 line
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach
before.I was a bit annoyed seeing that again. But could as well be that my
response never got out or that I'm mixing things up.I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
It's an intentional design decision to guarantee that at the
start of
the loop no object is locked.This is because Sima and I wanted to integrate userptr handling
intodrm_execas well in the long term.Quoted 6 lines
First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a
gpuvm
helper (without using GEM objects) as we've discussed previously.Quoted 7 lines
We might be talking past each other. That sounds like SVM, e.g. on
demand paging.What I mean is pre-faulting during command submission like radeon,
amdgpu and i915 do for the userptr handling.Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in
drm_gpuvmfor this sort of stuff.
And I think that's the concluding argument by Sima as well.In any case, If the planned
drm_execdevelopment is to be a full execbuf helper, I think we need a capable sub-helper for ONLY the ww
transaction locking as well, with support for the various locking primitives. In particular if we're going to be able to port i915 ww
transaction locking over. There are more uses of the ww locking transacions than execbuf.Quoted 11 lines
For that you need to re-start the whole handling similar to how you
need
to re-start for the mutex locking when you detect that the page array
is
stale, the difference is that you are not allowed to hold any resv
locks
while pre-faulting.Quoted 4 lines
That's why it is a requirement that the
drm_execloop starts without
any
locks held.But wouldn't you need an outer (userptr) loop and an inner (
ww_transaction) loop for this? Why would we want to re-validate userptrs on-EDEADLKS?Quoted 45 lines
Quoted 8 lines
Anyway if still there would be helpers in
drm_execfor some other
generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after,
so
I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was intentionally moved intodrm_exec_cleanup()to partially prevent
that
issue.I haven't fully figured out how to do handle everything exactly,
but
at
least in principle it can be made work. With this change here it
becomes
impossible.Quoted 13 lines
Still, If we need to come up with another solution, I think it's
fair
we clearly sort out why.Quoted 8 lines
I think we should just document that
drm_exec_trylock()can't
be
used
to
lock the first BO in the loop and explicitly WARN if that's the
case.Quoted 9 lines
Unfortunately that's not sufficient for the general use-case. If
we
want to keep thettm_bo_vmapproach of dropping the mmap lock when
there is contention on the bo resv, we need to be able to trylock
on
first lock.Quoted 7 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order
long
ago. Seedma_resv_lockdep()which is enforcing that.I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In particular not while waiting for mutexes that may be blocked by gpu
activity.Quoted 23 lines
Quoted 7 lines
Also bo creation is using trylock but might be able to use a sleeping lock there. But if that sleeping lock triggers an -
EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during
initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}if (r)
ttm_bo_put(bo);Quoted 6 lines
return r;
I have that on a branch here somewhere prepared, but never got the
time
to clean it up.Still, bo creation and validation may be part of a ww transaction as
well, like page-table bos (Although those are pre-locked so perhaps
not
a good example). But in the general case, I'm not sure this is sufficient for all use-cases./Thomas
Quoted 995 lines
Regards,
Christian.Quoted 992 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and
we'd
need to live with the weird bo creation situation described above./Thomas
Quoted 982 lines
Regards,
Christian.Quoted 979 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>
---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16
++++---
--
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6
++--
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +- -
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8
++---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8
++---
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +- -
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8
++---
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35
++++++++++++++-----
drivers/gpu/drm/drm_gpuvm.c | 8
++---
drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12
+++----
drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +- -
drivers/gpu/drm/xe/xe_vm.c | 10
+++---
include/drm/drm_exec.h | 23
+++++++++---17 files changed, 92 insertions(+), 62 deletions(-)diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cindexe4d4e55c08ad..4a08a692aa1f100644drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1152,12 +1152,12 @@ static intreserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&ctx->exec){ ret =amdgpu_vm_lock_pd(vm, &ctx->exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,
ret = amdgpu_vm_lock_pd(entry-Quoted 2 lines
bo_va-
base.vm,&ctx-
Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret)) goto error; ++ctx->n_vms;
}
ret = drm_exec_prepare_obj(&ctx->exec, &bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static intvalidate_invalid_user_pages(struct
amdkfd_process_info*process_info)
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm, &exec,
2);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD failed,
ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("drm_exec_prepare_obj
failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
drm_exec_until_all_locked(&p->exec){ r = amdgpu_vm_lock_pd(&fpriv->vm, &p-Quoted 1 line
exec,
1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
/* One fence for TTM and one for each
CS
job */
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&e-
Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
if (p->uf_bo) {
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;
}drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec,&bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);
drm_exec_until_all_locked(&exec){
r = drm_exec_prepare_obj(&exec, &bo-Quoted 1 line
tbo.base,
1);
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct
drm_device
*dev, void *data,drm_exec_until_all_locked(&exec){ if (gobj) {
r =drm_exec_lock_obj(&exec, gobj);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-0+ r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}
r =amdgpu_vm_lock_pd(&fpriv->vm, &exec, 2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ intamdgpu_mes_ctx_map_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct
amdgpu_device
*adev,
structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) {r = drm_exec_lock_obj(&exec, &bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static intsvm_range_reserve_bos(struct
svm_validate_context*ctx, bool intr)
vm = drm_priv_to_vm(pdd-Quoted 1 line
drm_priv);
r = amdgpu_vm_lock_pd(vm, &ctx-Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, r);
if (unlikely(r)) { pr_debug("failed %d to reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +1
-1@@ -28,12 +28,12 @@ * drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB,1);
- *drm_exec_retry_on_contention(&exec);
+ * ret =
drm_exec_retry_on_contention(&exec,
ret);
* if (ret)
* goto error;
* }drivers/gpu/drm/drm_exec.c +0
-0@@ -48,7 +48,8 @@*/
/* Dummy value used to initially enter the retry loop */drivers/gpu/drm/drm_exec.c +2
-1-#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)
/* Unlock all objects and drop references */ static voiddrm_exec_unlock_all(struct drm_exec *exec)drivers/gpu/drm/drm_exec.c +0
-0@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; }drivers/gpu/drm/drm_exec.c +1
-2-drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true;
}
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static intdrm_exec_lock_contended(struct
drm_exec*exec)
return ret;
}
+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww
transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/**
*drm_exec_lock_obj- lock a GEM object for use
* @exec: thedrm_execobject with the statedrivers/gpu/drm/drm_exec.c +0
-0@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct
drm_exec
*exec,
structdrm_gem_object*obj)
{
int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (
exec->prelocked== obj) {drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret =drm_gpuvm_prepare_vm(gpuvm, exec, num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
ret = drm_gpuvm_prepare_objects(gpuvm, exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
if (vm_exec->extra.fn) {
ret = vm_exec-Quoted 1 line
extra.fn(vm_exec);-
drm_exec_retry_on_contention(exec);drivers/gpu/drm/drm_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range,vm_exec-
Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct
drm_exec
*exec, structpvr_job_data*job_data,drm_exec_until_all_locked(exec){ int err = jobs_lock_all_objs(exec,job_data,job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err = drm_exec_retry_on_contention(exec,err);
if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) {struct
drm_gem_object*obj = submit-Quoted 1 line
bos[i].obj;
ret =
drm_exec_prepare_obj(&submit-Quoted 1 line
exec,
obj, 1);
-
drm_exec_retry_on_contention(&su
bmit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +0
-0@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) {ret = bind_lock_validate(job, exec, vme-
Quoted 1 line
num_fences);drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret) {
op = list_last_op(&bind_job-Quoted 1 line
ops);
goto unwind;
drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -83,14 +83,14 @@ static void test_lock_unlock(structkunit
*test)
drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&exec){ ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0); if (ret)
break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -110,13 +110,13 @@ static void test_duplicates(structkunit
*test)
drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);
drm_exec_until_all_locked(&exec){
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0); if (ret)
break;
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -200,7 +200,7 @@ static int handle_pagefault(struct
xe_gt
*gt,
struct pagefault *pf)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){ ret = xe_pf_begin(&exec, vma, atomic, tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt,struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){ ret = xe_pf_begin(&exec, vma, true, tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static voidpreempt_rebind_work_func(struct
work_struct*w)
bool done = false;
err =xe_preempt_work_begin(&exec, vm, &done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err = drm_exec_lock_obj(&exec,xe_vm_obj(vm));
-
drm_exec_retry_on_conten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err = drm_exec_lock_obj(&exec,&bo->ttm.base);
-
drm_exec_retry_on_conten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){ err = vm_bind_ioctl_ops_lock_and_prep(&exec,
vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err)
goto unlock;include/drm/drm_exec.h +0
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; };include/drm/drm_exec.h +2
-0+int drm_exec_handle_contended (struct drm_exec *exec); +/**
*drm_exec_obj()- Return the object for a givedrm_exec
index
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap
all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention was
detected
and we need to
* clean up and re-start the loop to prepare all GEM objects.
+ *
+ * Return: If no loop restart occurred: The error status.
*/
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \
include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+ if(unlikely(drm_exec_is_contended(__exec)))
{ \include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ })
/**
*drm_exec_is_contended- check for contention
Christian König
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Am 28.05.24 um 10:07 schrieb Thomas Hellström:
Quoted 12 lines · Christian König
Quoted 11 lines
Quoted 5 lines
2) Any contended lock held at loop start is completely encapsulated in
the ww transaction and can and will be unlocked when exiting it, so this patch doesn't introduce any additional problems for userptr handling AFAICT.The
drm_execobject was intentionally design to not have anything locked
at the beginning of the loop. See the discussion I had with Sima around
that when pushing thedrm_execobject upstream.
Quoted 6 lines
Quoted 4 lines
I would really like to stick with that design and honestly don't see the
reason to change that. Contenting on a trylock seem to be much more questionable.The change here is to make sure we *don't* have contention in a trylock, which is otherwise inherent in the current
drm_execdesign.
My sentence was probably a bit misleading. What I wanted to say is that trylock as first thing in the loop sounds really odd to me.
See the intention of a trylock is to acquire something optional. What we do for the freshly allocated BO and the 'don't try to block with the mmap lock held' case is actually kind something different.
A clean approach would be to to have the BO initialization and backing store allocation steps separated. In this case you don't even need to use trylock here.
And for the VM fault locking case the clean approach would be to tell the drm_exec object of the vm_fault parameters so that this helper can do the drop all locks, drop the mmap lock, acquire the blocking lock and return -EAGAIN.
This has the huge benefit that we not only stop blocking for the faulted BO, but eventually all others which might need to move so that the faulted BO is CPU accessible. I think that this is actually the more problematic case.
Quoted 3 lines
What I'm trying to say here is that we end up with the contended lock grabbed at loop start you already conceptually have a conflicting lock held (the
we_class::acquire_key). Both these can be resolved.
Yeah, I'm perfectly aware of that. But this is just a shortcoming of lockdep and not a real problem.
During the drm_exec code review we already moved the ww_acquire_init() into the cleanup function so that it's only called at the start of the loop. Background is that we ran into lockdep warnings with that otherwise.
But functionally it would still work if we do this in drm_exec_ini().
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
3) The need for a fully capable ww transaction helper moving forward.
If we need a tool that also does userptr locking, then I think we need
to separate that from the ww transaction tool and only pass the latter
around to TTM.Quoted 5 lines
drm_execis *not* meant to be aww_transactionhelper.The functionality here is to support drivers in their CS interface and
that includes userptr handling as well as a couple of other things.Then if so, I don't think
drm_execis the correct functionality to pass to TTM to resolve the eviction issues, but rather a ww transaction helper that can be used standalone *and* bydrm_exec. Now the functionality would be more or less what drm exec is today, but slightly augmented.
Quoted 4 lines
But then IMHO instead of changing name and more or less replicating what
drm_execis today wouldn't it be a better idea to subclassdrm_execinto a full-fledged CS helper at the time when that functionality is indeed added?
Mhm, interesting idea. But it still means that the base object needs to be designed in a way to not prevent the implementation of the subclass.
Christian.
Quoted 1,219 lines
/Thomas
Quoted 1,216 lines
Regards,
Christian.Quoted 1,213 lines
Thanks,
ThomasQuoted 1,209 lines · Thomas Hellström
Quoted 37 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 36 lines · Christian König
Quoted 35 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 27 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary
attempt to lock adma_resvlock. However, with the introduction
ofdrm_exec_trylock(), that doesn't work, since the locking of
the
contended lock needs to be a sleeping lock. Neither can we
ignore
locking the contended lock during a trylock since that would
violate
at least theww_mutexannotations.So resolve this by actually locking the contended lock during
drm_exec_retry_on_contention(). However, this introduces a
new
point
of failure since locking the contended lock may return - EINTR.Quoted 4 lines
Hence
drm_exec_retry_on_contention()must take an error parameter
and
also return a value indicating success.After thinking more about that I have to pretty clearly NAK this.
Quoted 8 lines
Quoted 1 line
I thought we were beyond upfront NAKing in the first reply :/
Well my memory could fail me, but I mentioned concerns on this approach
before.I was a bit annoyed seeing that again. But could as well be that my
response never got out or that I'm mixing things up.I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
It's an intentional design decision to guarantee that at the
start of
the loop no object is locked.This is because Sima and I wanted to integrate userptr handling
intodrm_execas well in the long term.Quoted 6 lines
First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up as a
gpuvm
helper (without using GEM objects) as we've discussed previously.Quoted 7 lines
We might be talking past each other. That sounds like SVM, e.g. on
demand paging.What I mean is pre-faulting during command submission like radeon,
amdgpu and i915 do for the userptr handling.Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in
drm_gpuvmfor this sort of stuff.
And I think that's the concluding argument by Sima as well.In any case, If the planned
drm_execdevelopment is to be a full execbuf helper, I think we need a capable sub-helper for ONLY the ww
transaction locking as well, with support for the various locking primitives. In particular if we're going to be able to port i915 ww
transaction locking over. There are more uses of the ww locking transacions than execbuf.Quoted 11 lines
For that you need to re-start the whole handling similar to how you
need
to re-start for the mutex locking when you detect that the page array
is
stale, the difference is that you are not allowed to hold any resv
locks
while pre-faulting.Quoted 4 lines
That's why it is a requirement that the
drm_execloop starts without
any
locks held.But wouldn't you need an outer (userptr) loop and an inner (
ww_transaction) loop for this? Why would we want to re-validate userptrs on-EDEADLKS?Quoted 45 lines
Quoted 8 lines
Anyway if still there would be helpers in
drm_execfor some other
generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after,
so
I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was intentionally moved intodrm_exec_cleanup()to partially prevent
that
issue.I haven't fully figured out how to do handle everything exactly,
but
at
least in principle it can be made work. With this change here it
becomes
impossible.Quoted 13 lines
Still, If we need to come up with another solution, I think it's
fair
we clearly sort out why.Quoted 8 lines
I think we should just document that
drm_exec_trylock()can't
be
used
to
lock the first BO in the loop and explicitly WARN if that's the
case.Quoted 9 lines
Unfortunately that's not sufficient for the general use-case. If
we
want to keep thettm_bo_vmapproach of dropping the mmap lock when
there is contention on the bo resv, we need to be able to trylock
on
first lock.Quoted 7 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order
long
ago. Seedma_resv_lockdep()which is enforcing that.I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In particular not while waiting for mutexes that may be blocked by gpu
activity.Quoted 23 lines
Quoted 7 lines
Also bo creation is using trylock but might be able to use
a sleeping lock there. But if that sleeping lock triggers an -
EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during
initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}if (r)
ttm_bo_put(bo);Quoted 6 lines
return r;
I have that on a branch here somewhere prepared, but never got the
time
to clean it up.Still, bo creation and validation may be part of a ww transaction as
well, like page-table bos (Although those are pre-locked so perhaps
not
a good example). But in the general case, I'm not sure this is sufficient for all use-cases./Thomas
Quoted 995 lines
Regards,
Christian.Quoted 992 lines
So the only really working alternative solution I can see is that
drm_exec_trylocksimply fails if there is a contended lock and
we'd
need to live with the weird bo creation situation described above./Thomas
Quoted 982 lines
Regards,
Christian.Quoted 979 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>--- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16 ++++--- -- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-(quoted diff) +8
-3- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +- - drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8 ++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- drivers/gpu/drm/drm_exec.c | 35 ++++++++++++++----- drivers/gpu/drm/drm_gpuvm.c | 8 ++--- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/msm/msm_gem_submit.c | 2 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +- drivers/gpu/drm/tests/drm_exec_test.c | 12 +++---- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +- - drivers/gpu/drm/xe/xe_vm.c | 10 +++--- include/drm/drm_exec.h | 23 +++++++++--- 17 files changed, 92 insertions(+), 62 deletions(-)diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cindexe4d4e55c08ad..4a08a692aa1f100644drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1152,12 +1152,12 @@ static intreserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&ctx->exec){ ret = amdgpu_vm_lock_pd(vm,&ctx->exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;ret = drm_exec_prepare_obj(&ctx->exec,
&bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,ret = amdgpu_vm_lock_pd(entry-
Quoted 2 lines
bo_va-
base.vm,&ctx-
Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error;
++ctx->n_vms;
}ret = drm_exec_prepare_obj(&ctx->exec,
&bo-Quoted 1 line
tbo.base, 1);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret = drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2619,7 +2619,7 @@ static intvalidate_invalid_user_pages(struct
amdkfd_process_info*process_info)
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2631,7 +2631,7 @@ static intvalidate_invalid_user_pages(struct
amdkfd_process_info*process_info)gobj =
&mem->bo->tbo.base; ret =
drm_exec_prepare_obj(&exec,
gobj,
1);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2875,7 +2875,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD
failed,
ret:
%d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2891,7 +2891,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*gobj =
&mem->bo->tbo.base; ret =
drm_exec_prepare_obj(&exec,
gobj,
1);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("drm_exec_prepare_obj
failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
drm_exec_until_all_locked(&p->exec){ r = amdgpu_vm_lock_pd(&fpriv->vm, &p-Quoted 1 line
exec,
1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-1-drm_exec_retry_on_contention (&p->exec); + r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
/* One fence for TTM and one for
each
CS
job */
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&e-
Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
if (p->uf_bo) {
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;
}drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec,
&bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r)) { DRM_ERROR("failed to reserve
CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm, r = amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r)) r = drm_exec_lock_obj(&exec,&bo-
Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r)) { DRM_ERROR("failed to reserve
CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec,DRM_EXEC_IGNORE_DUPLICATES,
0);
drm_exec_until_all_locked(&exec){
r = drm_exec_prepare_obj(&exec, &bo-Quoted 1 line
tbo.base,
1);
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;r =
amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct
drm_device
*dev, void *data,
drm_exec_until_all_locked(&exec){ if (gobj) {
r = drm_exec_lock_obj(&exec,
gobj);
-drm_exec_retry_on_contention(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-0+ r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}r = amdgpu_vm_lock_pd(&fpriv->vm, &exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ intamdgpu_mes_ctx_map_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;r =
amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;r =
amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct
amdgpu_device
*adev,
structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec,
&bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec,
&bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;r =
amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +2
-2@@ -152,12 +152,12 @@ static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm, drm_exec_init(&exec, 0, 0); drm_exec_until_all_locked(&exec) { r = drm_exec_lock_obj(&exec, &bo- > tbo.base); -drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention (&exec, r); if (unlikely(r)) goto out_unlock; r = amdgpu_vm_lock_pd(vm, &exec, 0); -drm_exec_retry_on_contention (&exec); + r = drm_exec_retry_on_contention(&exec,r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static intsvm_range_reserve_bos(struct
svm_validate_context*ctx, bool intr)
vm = drm_priv_to_vm(pdd-Quoted 1 line
drm_priv);r = amdgpu_vm_lock_pd(vm, &ctx-
Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, r);
if (unlikely(r)) { pr_debug("failed %d to
reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +1
-1@@ -28,12 +28,12 @@ * drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA, 1); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; * * ret = drm_exec_prepare_obj(&exec, boB,1);
- *drm_exec_retry_on_contention(&exec);
+ * ret =
drm_exec_retry_on_contention(&exec,
ret);
* if (ret)
* goto error;
* }drivers/gpu/drm/drm_exec.c +2
-1@@ -48,7 +48,8 @@*/ /* Dummy value used to initially enter the retry loop*/ -#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK) /* Unlock all objects and drop references */static void drm_exec_unlock_all(struct
drm_exec
*exec)drivers/gpu/drm/drm_exec.c +1
-2@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct drm_exec *exec) return true; } -drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ; return true; }
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static intdrm_exec_lock_contended(struct
drm_exec*exec)
return ret;
}+/**
+ *drm_exec_handle_contended()- Perform cleanup before a ww
transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); + /*** drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state @@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct
drm_exec
*exec,
structdrm_gem_object*obj)
{
int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; - if (exec->prelocked == obj) { drm_gem_object_put(exec->prelocked); exec->prelocked = NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret = drm_gpuvm_prepare_vm(gpuvm, exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;ret = drm_gpuvm_prepare_objects(gpuvm,
exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;if (
vm_exec->extra.fn) {
ret = vm_exec-Quoted 1 line
extra.fn(vm_exec);-
drm_exec_retry_on_contention(exec);drivers/gpu/drm/drm_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@ drm_gpuvm_exec_lock_range(struct drm_gpuvm_exec *vm_exec, drm_exec_until_all_locked(exec) { ret = drm_gpuvm_prepare_range(gpuvm, exec, addr, range, vm_exec-Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret = drm_exec_retry_on_contention(exec,ret);
if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct
drm_exec
*exec, structpvr_job_data*job_data,
drm_exec_until_all_locked(exec){ int err = jobs_lock_all_objs(exec,job_data,job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err = drm_exec_retry_on_contention(exec,err);
if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static int submit_lock_objects(struct msm_gem_submit *submit) for (unsigned i = 0; i < submit->nr_bos; i++) { struct drm_gem_object *obj =submit-
Quoted 1 line
bos[i].obj;
ret =
drm_exec_prepare_obj(&submit-Quoted 1 line
exec,
obj, 1);
-
drm_exec_retry_on_contention(&su
bmit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1@@ -1350,7 +1350,7 @@ nouveau_uvmm_bind_job_submit(structnouveau_job *job, drm_exec_init(exec, vme->flags, 0); drm_exec_until_all_locked(exec) { ret = bind_lock_validate(job, exec, vme- > num_fences); -drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention (exec, ret);if (ret) {
op = list_last_op(&bind_job-Quoted 1 line
ops);
goto unwind;
drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -83,14 +83,14 @@ static void test_lock_unlock(structkunit
*test)
drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&exec){ ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -110,13 +110,13 @@ static void test_duplicates(structkunit
*test)
drm_exec_init(&exec,DRM_EXEC_IGNORE_DUPLICATES,
0);
drm_exec_until_all_locked(&exec){
ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;ret =
drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -137,7 +137,7 @@ static void test_prepare(struct kunit *test) drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_prepare_obj(&exec, &gobj,1 ); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -200,7 +200,7 @@ static int handle_pagefault(struct
xe_gt
*gt,
struct pagefault *pf)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){ ret = xe_pf_begin(&exec, vma, atomic,
tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct xe_gt *gt,struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){ ret = xe_pf_begin(&exec, vma, true,
tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static voidpreempt_rebind_work_func(struct
work_struct*w)
bool done = false;err = xe_preempt_work_begin(&exec, vm,
&done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err =
drm_exec_lock_obj(&exec,xe_vm_obj(vm));
-
drm_exec_retry_on_conten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err =
drm_exec_lock_obj(&exec,&bo->ttm.base);
-
drm_exec_retry_on_conten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES, 0);drm_exec_until_all_locked(&exec){ err =
vm_bind_ioctl_ops_lock_and_prep(&exec,
vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err)
goto unlock;include/drm/drm_exec.h +2
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; }; +int drm_exec_handle_contended (struct drm_exec *exec); + /*** drm_exec_obj() - Return the object for a give
drm_exec
index
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_, __LINE__): \ /*** drm_exec_retry_on_contention - restart the loop to grap
all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention
was
detected
and we need to
* clean up and re-start the loop to prepare all GEM
objects.
+ *
+ * Return: If no loop restart occurred: The error status.
*/
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \
include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+ if(unlikely(drm_exec_is_contended(__exec)))
{ \include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ }) /*** drm_exec_is_contended - check for contention
Thomas Hellström
maintainerRe: [RFC PATCH v3 13/21] drm/exec: Rework contended locking
Quoted 47 lines · Christian König
Am 28.05.24 um 10:07 schrieb Thomas Hellström:
Quoted 21 lines · Christian König
Quoted 14 lines
Quoted 8 lines
2) Any contended lock held at loop start is completely encapsulated
in
the ww transaction and can and will be unlocked when exiting it, so
this patch doesn't introduce any additional problems for userptr
handling AFAICT.The
drm_execobject was intentionally design to not have anything locked
at the beginning of the loop. See the discussion I had with Sima around
that when pushing thedrm_execobject upstream.Quoted 6 lines
I would really like to stick with that design and honestly don't see
the
reason to change that. Contenting on a trylock seem to be much more
questionable.Quoted 3 lines
The change here is to make sure we *don't* have contention in a trylock, which is otherwise inherent in the current
drm_execdesign.My sentence was probably a bit misleading. What I wanted to say is that
trylock as first thing in the loop sounds really odd to me.See the intention of a trylock is to acquire something optional. What we
do for the freshly allocated BO and the 'don't try to block with the mmap lock held' case is actually kind something different.A clean approach would be to to have the BO initialization and backing
store allocation steps separated. In this case you don't even need to use trylock here.And for the VM fault locking case the clean approach would be to tell the
drm_execobject of thevm_faultparameters so that this helper can
do the drop all locks, drop the mmap lock, acquire the blocking lock and
return-EAGAIN.
Quoted 5 lines
This has the huge benefit that we not only stop blocking for the faulted
BO, but eventually all others which might need to move so that the faulted BO is CPU accessible. I think that this is actually the more problematic case.
Yes, this fault handler approach sounds ok to me. Do you mean we make drm_exec aware of both the mmap lock and the desire not to block while it is held? I figure that could come in handy for SVM gpu pagefaults as well.
For the bo initalization, the suggested solution there deviates a lot from the non-exec case, where a never-failing trylock on creation is sometimes necessary. This will probably be an unwanted obstacle for drm_exec conversion.
What do you think about the following for bo init? This also basically means the user of drm_exec can also *choose* to avoid trylock problems already at the start of the drm_exec loop.
https://patchwork.freedesktop.org/patch/595863/?series=133643&rev=7
and
https://patchwork.freedesktop.org/patch/595866/?series=133643&rev=7
Then if we also do the fault handler mode we could drop this patch assuming that drivers have all tools to handle existing cases of ww_mutex_trylock().
Quoted 46 lines
Quoted 6 lines
What I'm trying to say here is that we end up with the contended lock
grabbed at loop start you already conceptually have a conflicting lock
held (thewe_class::acquire_key). Both these can be resolved.Yeah, I'm perfectly aware of that. But this is just a shortcoming of lockdep and not a real problem.
During the
drm_execcode review we already moved theww_acquire_init()
into the cleanup function so that it's only called at the start of the
loop. Background is that we ran into lockdep warnings with that otherwise.But functionally it would still work if we do this in
drm_exec_ini().Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
3) The need for a fully capable ww transaction helper moving forward.
If we need a tool that also does userptr locking, then I think we
need
to separate that from the ww transaction tool and only pass the latter
around to TTM.Quoted 7 lines
drm_execis *not* meant to be aww_transactionhelper.The functionality here is to support drivers in their CS interface
and
that includes userptr handling as well as a couple of other things.Then if so, I don't think
drm_execis the correct functionality to pass
to TTM to resolve the eviction issues, but rather a ww transaction helper that can be used standalone *and* bydrm_exec. Now the functionality would be more or less what drm exec is today, but slightly augmented.Quoted 4 lines
But then IMHO instead of changing name and more or less replicating what
drm_execis today wouldn't it be a better idea to subclassdrm_execinto a full-fledged CS helper at the time when that functionality is indeed added?
Quoted 4 lines
Mhm, interesting idea. But it still means that the base object needs to
be designed in a way to not prevent the implementation of the subclass.
Sure, although I am still of the opinion that the patch at hand does not prevent that, but agreed violates a design principle.
I think we need to be careful to avoid a situaion where it is possible to open-code something that is desriable / needed by drivers, and that is not possible withing drm_exec, but still the driver needs to use drm_exec since it's required for other tools.
/Thomas
Quoted 1,387 lines
Christian.
Quoted 1,384 lines
/Thomas
Quoted 1,381 lines
Regards,
Christian.Quoted 1,378 lines
Thanks,
ThomasQuoted 1,374 lines · Thomas Hellström
Quoted 47 lines · Christian König
Am 22.05.24 um 16:32 schrieb Thomas Hellström:
Quoted 40 lines · Christian König
Quoted 39 lines
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 38 lines
If contention and backoff occurs during a
drm_execww transaction,
the contended lock is not locked again until the next orinary
attempt to lock adma_resvlock. However, with the introduction
ofdrm_exec_trylock(), that doesn't work, since the locking
of
the
contended lock needs to be a sleeping lock. Neither can
we
ignore
locking the contended lock during a trylock since that
would
violate
at least theww_mutexannotations.So resolve this by actually locking the contended
lock
duringdrm_exec_retry_on_contention(). However, this
introduces
a
new
point
of failure since locking the contended lock may
return -
EINTR.Hence
drm_exec_retry_on_contention()must take an
error
parameter
and
also return a value indicating success.Quoted 6 lines
Quoted 4 lines
After thinking more about that I have to pretty clearly NAK
this.I thought we were beyond upfront NAKing in the first reply :/
Quoted 8 lines
Well my memory could fail me, but I mentioned concerns on this
approach
before.I was a bit annoyed seeing that again. But could as well be that my
response never got out or that I'm mixing things up.I haven't seen it at least. Last discussion on this I saw was here. I didn't see a follow-up on that.
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
It's an intentional design decision to guarantee that at
the
start of
the loop no object is locked.This is because Sima and I wanted to integrate userptr handling
intodrm_execas well in the long term.Quoted 7 lines
First I agree the interface looks worse with this patch. But I thought generic userptr handling were going to end up
as a
gpuvm
helper (without using GEM objects) as we've discussed previously.Quoted 8 lines
We might be talking past each other. That sounds like SVM, e.g.
on
demand paging.What I mean is pre-faulting during command submission like radeon,
amdgpu and i915 do for the userptr handling.Yes, then we're talking about the same thing.
We discussed in this thread here, started by Dave.
I still think the right place is in
drm_gpuvmfor this sort of
stuff.
And I think that's the concluding argument by Sima as well.In any case, If the planned
drm_execdevelopment is to be a full
execbuf helper, I think we need a capable sub-helper for ONLY the
ww
transaction locking as well, with support for the various locking
primitives. In particular if we're going to be able to port i915
ww
transaction locking over. There are more uses of the ww locking
transacions than execbuf.Quoted 14 lines
For that you need to re-start the whole handling similar to how
you
need
to re-start for the mutex locking when you detect that the page
array
is
stale, the difference is that you are not allowed to hold any
resv
locks
while pre-faulting.Quoted 5 lines
That's why it is a requirement that the
drm_execloop starts
without
any
locks held.But wouldn't you need an outer (userptr) loop and an inner (
ww_transaction) loop for this? Why would we want to re- validate
userptrs on-EDEADLKS?Quoted 53 lines
Quoted 10 lines
Anyway if still there would be helpers in
drm_execfor some
other
generic userptr solution, those need to be done before theww_acquire_ctx_init(). The contended locking here is done after,
so
I
can't really see how these would clash.Yes, that indeed was a problem. The
ww_acquire_ctx_init()was
intentionally moved intodrm_exec_cleanup()to partially prevent
that
issue.I haven't fully figured out how to do handle everything exactly,
but
at
least in principle it can be made work. With this change here
it
becomes
impossible.Quoted 15 lines
Still, If we need to come up with another solution, I think
it's
fair
we clearly sort out why.Quoted 9 lines
I think we should just document that
drm_exec_trylock()can't
be
used
to
lock the first BO in the loop and explicitly WARN if that's
the
case.Quoted 11 lines
Unfortunately that's not sufficient for the general use- case.
If
we
want to keep thettm_bo_vmapproach of dropping the mmap lock
when
there is contention on the bo resv, we need to be able to trylock
on
first lock.Quoted 7 lines
Mhm, why exactly do we still have that dance in the first place?
I mean we have sorted out the
mmap()anddma_resv()locking order
long
ago. Seedma_resv_lockdep()which is enforcing that.I explained that in my reply here:
https://lore.kernel.org/dri-devel/[email protected]/
We shouldn't be holding the mmap lock when waiting for stuff. In
particular not while waiting for mutexes that may be blocked by
gpu
activity.Quoted 26 lines
Quoted 10 lines
Also bo creation is using trylock but might be able to use
a sleeping lock there. But if that sleeping lock triggers an
-
EDEADLK
(DEBUG_WW_MUTEX_SLOWPATH) we have the weird situation of referencing an
object that never was fully created as a contending object.I wanted to eliminate that as well by not validating the BO during
initialization any more.
So bo creation would then be:
ttm_bo_init(bo)
drm_exec_while_not_all_locked(){
drm_exec_prepare_object(bo, 1);
ttm_bo_validate(bo);
}if (r)
ttm_bo_put(bo);Quoted 7 lines
return r;
I have that on a branch here somewhere prepared, but never got
the
time
to clean it up.Still, bo creation and validation may be part of a ww transaction
as
well, like page-table bos (Although those are pre-locked so perhaps
not
a good example). But in the general case, I'm not sure this is
sufficient for all use-cases./Thomas
Quoted 1,120 lines
Regards,
Christian.Quoted 1,117 lines
So the only really working alternative solution I can see is
thatdrm_exec_trylocksimply fails if there is a contended lock
and
we'd
need to live with the weird bo creation situation described
above./Thomas
Quoted 1,104 lines
Regards,
Christian.Quoted 1,101 lines
Cc: Christian König<[email protected]>
Cc: Somalapuram Amaranath<[email protected]>
Cc: Matthew Brost<[email protected]> Cc:<[email protected]>
Signed-off-by: Thomas Hellström<[email protected]>
---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16
++++---
--
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 6
++--
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-
-
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 8
++---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 8
++---
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 4 +-
- drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c | 8
++---
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +-
drivers/gpu/drm/drm_exec.c | 35
++++++++++++++-----
drivers/gpu/drm/drm_gpuvm.c | 8
++---
drivers/gpu/drm/imagination/pvr_job.c | 2 +-
drivers/gpu/drm/msm/msm_gem_submit.c | 2 +-
drivers/gpu/drm/nouveau/nouveau_uvmm.c | 2 +-
drivers/gpu/drm/tests/drm_exec_test.c | 12
+++----
drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 +-
-
drivers/gpu/drm/xe/xe_vm.c | 10
+++---
include/drm/drm_exec.h | 23
+++++++++---17 files changed, 92 insertions(+), 62deletions(-)
diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.cindexe4d4e55c08ad..4a08a692aa1f100644
---a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c(quoted diff) +0
-0@@ -1152,12 +1152,12 @@ static intreserve_bo_and_vm(struct
kgd_mem
*mem,
drm_exec_init(&ctx->exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&ctx->exec){
ret = amdgpu_vm_lock_pd(vm, &ctx-Quoted 1 line
exec,
2);
(quoted diff) +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
ret = drm_exec_prepare_obj(&ctx-Quoted 1 line
exec,
&bo-
Quoted 1 line
tbo.base, 1);
(quoted diff) +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}(quoted diff) +0
-0@@ -1199,14 +1199,14 @@ static intreserve_bo_and_cond_vms(struct
kgd_mem*mem,
ret =
amdgpu_vm_lock_pd(entry-Quoted 2 lines
bo_va-
base.vm,&ctx -
Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention
(&ctx-Quoted 1 line
exec);
(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, ret);
if (unlikely(ret))
goto error;
++ctx->n_vms;
}
ret = drm_exec_prepare_obj(&ctx-Quoted 1 line
exec,
&bo-
Quoted 1 line
tbo.base, 1);
(quoted diff) +0
-1- drm_exec_retry_on_contention(&ctx-Quoted 1 line
exec);
(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec,
ret);
if (unlikely(ret))
goto error;
}(quoted diff) +0
-0@@ -2619,7 +2619,7 @@ static intvalidate_invalid_user_pages(struct
amdkfd_process_info*process_info)
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);
-
drm_exec_retry_on_contention
(&exec);(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }(quoted diff) +0
-0@@ -2631,7 +2631,7 @@ static int
validate_invalid_user_pages(struct amdkfd_process_info *process_info)
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);
-
drm_exec_retry_on_contention
(&exec);(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) gotounreserve_out; }(quoted diff) +0
-0@@ -2875,7 +2875,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*
list_for_each_entry(peer_vm,
&process_info-Quoted 1 line
vm_list_head,
vm_list_node) { ret = amdgpu_vm_lock_pd(peer_vm,
&exec,
2);
-
drm_exec_retry_on_contention
(&exec);(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("Locking VM PD
failed,
ret:
%d\n", ret);
gotottm_reserve_fail;(quoted diff) +0
-0@@ -2891,7 +2891,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info,struct
dma_fence__rcu*
gobj =&mem->bo->tbo.base; ret = drm_exec_prepare_obj(&exec,
gobj,
1);
-
drm_exec_retry_on_contention
(&exec);(quoted diff) +1
-0+ ret =drm_exec_retry_on_contention(&exec,
ret);
if (unlikely(ret)) { pr_err("drm_exec_pre pare_obj
failed, ret: %d\n", ret);
gotottm_reserve_fail;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -897,7 +897,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
drm_exec_until_all_locked(&p->exec){
r = amdgpu_vm_lock_pd(&fpriv->vm,
&p-Quoted 1 line
exec,
1
+ p-Quoted 1 line
gang_size);drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-1- drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r = drm_exec_retry_on_contention(&p-Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -905,7 +905,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
/* One fence for TTM and one for
each
CS
job */
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&e-
Quoted 2 lines
bo-
tbo.base,1 + p-
Quoted 1 line
gang_size);-
drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -915,7 +915,7 @@ static intamdgpu_cs_parser_bos(struct
amdgpu_cs_parser*p,
if (p->uf_bo) {
r = drm_exec_prepare_obj(&p-Quoted 1 line
exec,
&p-
Quoted 1 line
uf_bo->tbo.base,1 + p-
Quoted 1 line
gang_size);-
drm_exec_retry_on_contention(&p-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+ r =drm_exec_retry_on_contention(&p-
Quoted 1 line
exec,
r);
if (unlikely(r))
gotoout_free_user_pages;
}drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -74,7 +74,7 @@ int amdgpu_map_static_csa(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +0
-0@@ -114,7 +114,7 @@ intamdgpu_unmap_static_csa(struct
amdgpu_device*adev, structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r)) { DRM_ERROR("failed to reserve CSA,PD
BOs:
err=%d\n", r);
goto error;drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -239,12 +239,12 @@ static voidamdgpu_gem_object_close(struct
drm_gem_object*obj,
drm_exec_init(&exec,DRM_EXEC_IGNORE_DUPLICATES,
0);
drm_exec_until_all_locked(&exec){
r = drm_exec_prepare_obj(&exec, &bo-Quoted 1 line
tbo.base,
1);
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0
-0@@ -776,13 +776,13 @@ int amdgpu_gem_va_ioctl(struct
drm_device
*dev, void *data,drm_exec_until_all_locked(&exec){ if (gobj) {
r =drm_exec_lock_obj(&exec, gobj);
-
drm_exec_retry_on_contention
(&exec);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-0+ r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}
r = amdgpu_vm_lock_pd(&fpriv->vm,
&exec,
2);drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1221,12 +1221,12 @@ intamdgpu_mes_ctx_map_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +0
-0@@ -1292,12 +1292,12 @@ intamdgpu_mes_ctx_unmap_meta_data(struct
amdgpu_device*adev,
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec,
&ctx_data-Quoted 2 lines
meta_data_obj-
tbo.base);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
}
diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
indexe22cb2b5cd92..72b8213e352c100644drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -77,7 +77,7 @@ int amdgpu_seq64_map(struct
amdgpu_device
*adev,
structamdgpu_vm*vm,
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +0
-0@@ -138,7 +138,7 @@ void amdgpu_seq64_unmap(struct
amdgpu_device
*adev, structamdgpu_fpriv*fpriv)
r =amdgpu_vm_lock_pd(vm, &exec, 0); if (likely(!r))
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
goto error;
}
diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c
indexe01c1c8e64c4..63392ce43945100644drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -89,12 +89,12 @@ static int map_ring_data(struct
amdgpu_device
*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoerror_fini_exec;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoerror_fini_exec;
}drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +0
-0@@ -152,12 +152,12 @@ static intunmap_ring_data(struct
amdgpu_device*adev, structamdgpu_vm*vm,
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
r = drm_exec_lock_obj(&exec, &bo-Quoted 1 line
tbo.base);
drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
r =amdgpu_vm_lock_pd(vm, &exec, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c +1
-1-drm_exec_retry_on_contention (&exec); + r =drm_exec_retry_on_contention(&exec,
r);
if (unlikely(r))
gotoout_unlock;
}drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0
-0@@ -1499,7 +1499,7 @@ static intsvm_range_reserve_bos(struct
svm_validate_context*ctx, bool intr)
vm = drm_priv_to_vm(pdd-Quoted 1 line
drm_priv);
r = amdgpu_vm_lock_pd(vm,
&ctx-Quoted 1 line
exec,
2);
-
drm_exec_retry_on_contention
(&ctx-Quoted 1 line
exec);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1
-0+ r =drm_exec_retry_on_contention(&ctx-
Quoted 1 line
exec, r);
if (unlikely(r)) { pr_debug("failed %d to
reserve
bo\n", r);
gotounreserve_out;drivers/gpu/drm/drm_exec.c +1
-1@@ -28,12 +28,12 @@ * drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ); * drm_exec_until_all_locked(&exec) { * ret = drm_exec_prepare_obj(&exec, boA,1 ); - *drm_exec_retry_on_contention (&exec); + * ret =drm_exec_retry_on_contention (&exec, ret); * if (ret) * goto error; ** ret = drm_exec_prepare_obj(&exec,
boB,
1);
- *drm_exec_retry_on_contention(&exec);
+ * ret =
drm_exec_retry_on_contention(&exec,
ret);
* if (ret)
* goto error;
* }drivers/gpu/drm/drm_exec.c +0
-0@@ -48,7 +48,8 @@*/
/* Dummy value used to initially enter the retry loop
*/drivers/gpu/drm/drm_exec.c +2
-1-#define DRM_EXEC_DUMMY ((void *)~0) +#define DRM_EXEC_DUMMY ERR_PTR(-ESTALE) +#define DRM_EXEC_CONTENDED ERR_PTR(-EDEADLK)
/* Unlock all objects and drop references */ static voiddrm_exec_unlock_all(struct drm_exec *exec)drivers/gpu/drm/drm_exec.c +0
-0@@ -131,8 +132,7 @@ bool drm_exec_cleanup(struct
drm_exec
*exec)
return true;
}drivers/gpu/drm/drm_exec.c +1
-2-drm_exec_unlock_all (exec); - exec->num_objects =0 ; + exec->contended =NULL ;return true;
}
EXPORT_SYMBOL(drm_exec_cleanup);drivers/gpu/drm/drm_exec.c +0
-0@@ -194,6 +194,27 @@ static intdrm_exec_lock_contended(struct
drm_exec*exec)
return ret;
}
+/**
+ *drm_exec_handle_contended()- Perform cleanup before
a ww
transaction restart
+ * @exec: Pointer to thedrm_execobject.
+ *
+ * Unlocks all held resvs and re-locks the contended object.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_exec.c +13
-0+int drm_exec_handle_contended (struct drm_exec *exec) +{ +int ret; + +drm_exec_unlock_all (exec); + exec->num_objects =0 ; + ret =drm_exec_lock_contended (exec); + exec->contended =DRM_EXEC_CONTENDED ; + +return ret; +} +EXPORT_SYMBOL (drm_exec_handle_contended); +/**
*drm_exec_lock_obj- lock a GEM object for use
* @exec: thedrm_execobject with the statedrivers/gpu/drm/drm_exec.c +0
-0@@ -209,10 +230,6 @@ int drm_exec_lock_obj(struct
drm_exec
*exec,
structdrm_gem_object*obj)
{
int ret;drivers/gpu/drm/drm_exec.c +0
-4- ret =drm_exec_lock_contended (exec); -if (unlikely (ret)) -return ret; -if (
exec->prelocked== obj) {drm_gem_object_put(exec->prelocked);exec->prelocked= NULL;drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1254,18 +1254,18 @@ drm_gpuvm_exec_lock(struct
drm_gpuvm_exec*vm_exec)
drm_exec_until_all_locked(exec){
ret = drm_gpuvm_prepare_vm(gpuvm,
exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
ret =
drm_gpuvm_prepare_objects(gpuvm,
exec,num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
if (vm_exec->extra.fn) {
ret = vm_exec-Quoted 1 line
extra.fn(vm_exec);-
drm_exec_retry_on_contention
(exec);drivers/gpu/drm/drm_gpuvm.c +1
-0+ ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/drm_gpuvm.c +0
-0@@ -1346,7 +1346,7 @@drm_gpuvm_exec_lock_range(struct
drm_gpuvm_exec*vm_exec,drm_exec_until_all_locked(exec){ ret = drm_gpuvm_prepare_range(gpuvm, exec,
addr,
range,
vm_exec-Quoted 1 line
num_fences);drivers/gpu/drm/drm_gpuvm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret)
goto err;
}drivers/gpu/drm/imagination/pvr_job.c +0
-0@@ -574,7 +574,7 @@ prepare_job_resvs_for_each(struct
drm_exec
*exec, structpvr_job_data*job_data,drm_exec_until_all_locked(exec){ int err = jobs_lock_all_objs(exec,job_data,job_count);drivers/gpu/drm/imagination/pvr_job.c +1
-1-drm_exec_retry_on_contention (exec); + err =drm_exec_retry_on_contention(exec,
err);
if (err)
return err;
}drivers/gpu/drm/msm/msm_gem_submit.c +0
-0@@ -259,7 +259,7 @@ static intsubmit_lock_objects(struct
msm_gem_submit*submit)
for (unsigned i = 0; i < submit-Quoted 1 line
nr_bos;i++)
{
structdrm_gem_object*obj = submit-Quoted 1 line
bos[i].obj;
ret =
drm_exec_prepare_obj(&submit-Quoted 1 line
exec,
obj, 1);
-
drm_exec_retry_on_contention
(&su
bmit-Quoted 1 line
exec);
drivers/gpu/drm/msm/msm_gem_submit.c +1
-0+ ret =
drm_exec_retry_on_contention(&submit->exec, ret);
if (ret)
goto error;
}drivers/gpu/drm/nouveau/nouveau_uvmm.c +0
-0@@ -1350,7 +1350,7 @@nouveau_uvmm_bind_job_submit(struct
nouveau_job*job,
drm_exec_init(exec, vme->flags, 0);drm_exec_until_all_locked(exec){ ret = bind_lock_validate(job, exec, vme-Quoted 1 line
num_fences);drivers/gpu/drm/nouveau/nouveau_uvmm.c +1
-1-drm_exec_retry_on_contention (exec); + ret =drm_exec_retry_on_contention(exec,
ret);
if (ret) {
op = list_last_op(&bind_job-Quoted 1 line
ops);
goto unwind;
drivers/gpu/drm/tests/drm_exec_test.c +1
-1@@ -63,7 +63,7 @@ static void test_lock(struct kunit *test) drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT ,0 ); drm_exec_until_all_locked(&exec) { ret = drm_exec_lock_obj(&exec, &gobj); -drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -83,14 +83,14 @@ static voidtest_lock_unlock(struct
kunit
*test)
drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&exec){
ret = drm_exec_lock_obj(&exec,
&gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0); if (ret)
break;
drm_exec_unlock_obj(&exec, &gobj); ret =drm_exec_lock_obj(&exec, &gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -110,13 +110,13 @@ static voidtest_duplicates(struct
kunit
*test)
drm_exec_init(&exec,DRM_EXEC_IGNORE_DUPLICATES,
0);
drm_exec_until_all_locked(&exec){
ret = drm_exec_lock_obj(&exec,
&gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;
ret = drm_exec_lock_obj(&exec,
&gobj);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/tests/drm_exec_test.c +0
-0@@ -137,7 +137,7 @@ static void test_prepare(structkunit
*test)
drm_exec_init(&exec,DRM_EXEC_INTERRUPTIBLE_WAIT,
0);
drm_exec_until_all_locked(&exec){
ret = drm_exec_prepare_obj(&exec,
&gobj,
1);drivers/gpu/drm/tests/drm_exec_test.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
KUNIT_EXPECT_EQ(test, ret, 0);
if (ret)
break;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -200,7 +200,7 @@ static inthandle_pagefault(struct
xe_gt
*gt,
struct pagefault *pf)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
ret = xe_pf_begin(&exec, vma,
atomic,
tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
gotounlock_dma_resv;drivers/gpu/drm/xe/xe_gt_pagefault.c +0
-0@@ -543,7 +543,7 @@ static int handle_acc(struct
xe_gt
*gt,
struct
acc *acc)
drm_exec_init(&exec, 0, 0);drm_exec_until_all_locked(&exec){ ret = xe_pf_begin(&exec, vma, true, tile-Quoted 1 line
id);
drivers/gpu/drm/xe/xe_gt_pagefault.c +1
-1-drm_exec_retry_on_contention (&exec); + ret =drm_exec_retry_on_contention(&exec,
ret);
if (ret)
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -501,7 +501,7 @@ static void
preempt_rebind_work_func(struct work_struct *w)
bool done = false;
err = xe_preempt_work_begin(&exec, vm,
&done);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err || done) {
drm_exec_fini(&exec);
if (err &&xe_vm_validate_should_retry(&exec, err, &end))drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -1052,7 +1052,7 @@ static voidxe_vma_destroy_unlocked(struct
xe_vma*vma)
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec){
err =xe_vm_lock_vma(&exec, vma);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (XE_WARN_ON(err))
break;
}drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2148,11 +2148,11 @@ static struct xe_vma *new_vma(struct
xe_vm
*vm, structdrm_gpuva_op_map*op,
err = 0;
if (!bo->vm) {
err = drm_exec_lock_obj(&exec,xe_vm_obj(vm));
-
drm_exec_retry_on_co
nten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (!err) {
err = drm_exec_lock_obj(&exec,&bo->ttm.base);
-
drm_exec_retry_on_co
nten
tion
(&
exec);drivers/gpu/drm/xe/xe_vm.c +1
-0+ err =
drm_exec_retry_on_contention(&exec, err);
}
if (err) {drm_exec_fini(&exec);drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2884,7 +2884,7 @@ static intvm_bind_ioctl_ops_execute(struct
xe_vm*vm,
DRM_EXEC_IGNORE_DUPLICATES,
0);
drm_exec_until_all_locked(&exec){
err =
vm_bind_ioctl_ops_lock_and_prep(&exec,
vm,
vops);drivers/gpu/drm/xe/xe_vm.c +1
-1-drm_exec_retry_on_contention (&exec); + err =drm_exec_retry_on_contention(&exec,
err);
if (err)
goto unlock;include/drm/drm_exec.h +0
-0@@ -51,6 +51,8 @@ struct drm_exec { struct drm_gem_object *prelocked; };
+intdrm_exec_handle_contended(struct drm_exec *exec);
+
/**
*drm_exec_obj()- Return the object for a givedrm_exec
index
* @exec: Pointer to thedrm_execcontextinclude/drm/drm_exec.h +0
-0@@ -113,15 +115,26 @@ __PASTE(__drm_exec_,
__LINE__):\/**
*drm_exec_retry_on_contention- restart the loop to
grap
all
locks
* @exec:drm_execobject
+ * @_ret: The current error status
*
* Control flow helper to continue when a contention
was
detected
and we need to
* clean up and re-start the loop to prepare all GEM
objects.
+ *
+ * Return: If no loop restart occurred: The error status.
*/
-#definedrm_exec_retry_on_contention(exec)\include/drm/drm_exec.h +0
-1- do{ \include/drm/drm_exec.h +0
-1- if(unlikely(drm_exec_is_contended(exec))) \
include/drm/drm_exec.h +0
-1- goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-1- }while (0 ) +#define drm_exec_retry_on_contention(exec,
_ret) \include/drm/drm_exec.h +1
-0+ ({\include/drm/drm_exec.h +1
-0+ struct drm_exec *__exec =(exec); \
include/drm/drm_exec.h +1
-0+ int __ret =(
_ret); \include/drm/drm_exec.h +1
-0+\include/drm/drm_exec.h +1
-0+ if(unlikely(drm_exec_is_contended(__exec)))
{ \include/drm/drm_exec.h +1
-0+ WARN_ON(__ret != -EDEADLK); \
include/drm/drm_exec.h +1
-0+ __ret =
drm_exec_handle_contended(__exec); \include/drm/drm_exec.h +1
-0+ if(!__ret) \
include/drm/drm_exec.h +1
-0+ goto
*__drm_exec_retry_ptr; \include/drm/drm_exec.h +1
-0+ }\include/drm/drm_exec.h +1
-0+ __ret;\include/drm/drm_exec.h +1
-0+ })
/**
*drm_exec_is_contended- check for contention
Thomas Hellström
maintainerRFC PATCH v3 14/21 drm/exec: Introduce a drm_exec_trylock_obj() function
In reply to Thomas HellströmTTM needs to trylock dma_resv objects in a couple of places. However this functionality is missing in drm_exec. Introduce it.
Note that in addition to the -EBUSY error returned on failure to take the lock, the operation may return -ENOMEM if there was a failure to allocate memory for the drm_exec held lock array.
This failure mode could be avoided if the drm_exec structure instead maintained a linked list of locked objects, similar to i915.
--- drivers/gpu/drm/drm_exec.c | 50 +++++++++++++++++++++++++++++++++++--- include/drm/drm_exec.h | 1 + 2 files changed, 47 insertions(+), 4 deletions(-)
drivers/gpu/drm/drm_exec.c +46 -4
@@ -139,14 +139,17 @@ EXPORT_SYMBOL(drm_exec_cleanup);/* Track the locked object in the array */ static int drm_exec_obj_locked (struct drm_exec *exec, -struct drm_gem_object *obj) +struct drm_gem_object *obj, +gfp_t gfp) { +might_alloc (gfp); +if (unlikely (exec->num_objects == exec->max_objects)) {size_t size = exec->max_objects *sizeof (void *);void *tmp; tmp =kvrealloc (exec->objects, size, size +PAGE_SIZE , -GFP_KERNEL ); + gfp);if (!tmp)return -ENOMEM ;
@@ -179,7 +182,7 @@ static int drm_exec_lock_contended(struct drm_exec *exec)dma_resv_lock_slow (obj->resv, &exec->ticket); } - ret =drm_exec_obj_locked (exec, obj); + ret =drm_exec_obj_locked (exec, obj,GFP_KERNEL );if (unlikely (ret)) goto error_unlock;
@@ -215,6 +218,45 @@ int drm_exec_handle_contended(struct drm_exec *exec) }EXPORT_SYMBOL (drm_exec_handle_contended); +/** +* drm_exec_trylock_obj - trylock a GEM object for use +* @exec: the drm_exec object with the state. +* @obj: the GEM object to lock. +* +* Trylock a GEM object for use and grab a reference to it. +* +* Returns: -EALREADY when object is already locked (can be suppressed by +* setting the DRM_EXEC_IGNORE_DUPLICATES flag), -ENOMEM when memory +* allocation failed, and zero for success. If the object was already +* locked, -EBUSY will be returned. +*/ +int drm_exec_trylock_obj (struct drm_exec *exec,struct drm_gem_object *obj) +{ +int ret; + +might_alloc (GFP_ATOMIC ); + +if (exec->prelocked == obj) { +drm_gem_object_put (exec->prelocked); + exec->prelocked =NULL ; +return 0 ; + } + +if (!dma_resv_trylock_ctx (obj->resv, &exec->ticket)) { +if (dma_resv_locking_ctx (obj->resv) == &exec->ticket) +return (exec->flags &DRM_EXEC_IGNORE_DUPLICATES ) ?0 : -EALREADY ; +else +return -EBUSY ; + } + + ret =drm_exec_obj_locked (exec, obj,GFP_ATOMIC | __GFP_NOWARN); +if (ret) +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL (drm_exec_trylock_obj); +/** * drm_exec_lock_obj - lock a GEM object for use * @exec: the drm_exec object with the state
@@ -254,7 +296,7 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj)if (unlikely (ret))return ret; - ret =drm_exec_obj_locked (exec, obj); + ret =drm_exec_obj_locked (exec, obj,GFP_KERNEL );if (ret) goto error_unlock;
include/drm/drm_exec.h +1 -0
@@ -152,6 +152,7 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr);void drm_exec_fini (struct drm_exec *exec);bool drm_exec_cleanup (struct drm_exec *exec);int drm_exec_lock_obj (struct drm_exec *exec,struct drm_gem_object *obj); +int drm_exec_trylock_obj (struct drm_exec *exec,struct drm_gem_object *obj);void drm_exec_unlock_obj (struct drm_exec *exec,struct drm_gem_object *obj);int drm_exec_prepare_obj (struct drm_exec *exec,struct drm_gem_object *obj,unsigned int num_fences);
Signature · 1 line
2.44.0
Thomas Hellström
maintainerRFC PATCH v3 15/21 drm/exec: Add a snapshot capability
In reply to Thomas HellströmWhen validating a buffer object for submission, we might need to lock a number of object for eviction to make room for the validation.
This makes it pretty likely that validation will eventually succeed, since eventually the validating process will hold most dma_resv locks of the buffer objects residing in the memory type being validated for.
However, once validation of a single object has succeeded it might not be beneficial to hold on to those locks anymore, and the validator would want to drop the locks of all objects taken during validation.
Introduce a drm_exec snapshot functionality that can be used to record the locks held at a certain time, and a restore functionality that restores the drm_exec state to the snapshot by dropping all locks.
Snapshots can be nested if needed.
--- drivers/gpu/drm/drm_exec.c | 55 +++++++++++++++++++++++++++++++++++++- include/drm/drm_exec.h | 23 +++++++++++++++- 2 files changed, 76 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_exec.c +54 -1
@@ -57,6 +57,7 @@ static void drm_exec_unlock_all(struct drm_exec *exec)struct drm_gem_object *obj;unsigned long index; +WARN_ON (exec->snap);drm_exec_for_each_locked_object_reverse (exec, index, obj) {dma_resv_unlock (obj->resv);drm_gem_object_put (obj);
@@ -90,6 +91,7 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->num_objects =0 ; exec->contended =DRM_EXEC_DUMMY ; exec->prelocked =NULL ; + exec->snap =NULL ; }EXPORT_SYMBOL (drm_exec_init);
@@ -301,7 +303,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) goto error_unlock;return 0 ; - error_unlock:dma_resv_unlock (obj->resv);return ret;
@@ -395,5 +396,57 @@ int drm_exec_prepare_array(struct drm_exec *exec, }EXPORT_SYMBOL (drm_exec_prepare_array); +/** +* drm_exec_restore() - Restore the drm_exec state to the point of a snapshot. +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Restores the drm_exec object by means of unlocking and dropping references +* to objects locked after the snapshot. +*/ +void drm_exec_restore (struct drm_exec *exec,struct drm_exec_snapshot *snap) +{ +struct drm_gem_object *obj; +unsigned int index; + + exec->snap = snap->saved_snap; + +drm_exec_for_each_locked_object_reverse (exec, index, obj) { +if (index +1 == snap->num_locked) +break ; + +dma_resv_unlock (obj->resv); +drm_gem_object_put (obj); + exec->objects[index] =NULL ; + } + + exec->num_objects = snap->num_locked; + +if (!exec->prelocked) + exec->prelocked = snap->prelocked; +else +drm_gem_object_put (snap->prelocked); +} +EXPORT_SYMBOL (drm_exec_restore); + +/** +* drm_exec_snapshot() - Take a snapshot of the drm_exec state +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Records the @exec state in @snap. The @snap object is typically allocated +* in the stack of the caller. +*/ +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap) +{ + snap->num_locked = exec->num_objects; + snap->prelocked = exec->prelocked; +if (snap->prelocked) +drm_gem_object_get (snap->prelocked); + snap->saved_snap = exec->snap; + exec->snap = snap; +} +EXPORT_SYMBOL (drm_exec_snapshot); +MODULE_DESCRIPTION ("DRM execution context" );MODULE_LICENSE ("Dual MIT/GPL" );
include/drm/drm_exec.h +22 -1
@@ -19,7 +19,6 @@ struct drm_exec {* @flags: Flags to control locking behavior */ u32 flags; -/** * @ticket: WW ticket used for acquiring locks */
@@ -49,6 +48,25 @@ struct drm_exec {* @prelocked: already locked GEM object due to contention */ struct drm_gem_object *prelocked; + +/** +* @snap: Pointer to the last snapshot taken or NULL if none. +*/ +struct drm_exec_snapshot *snap; +}; + +/** +* struct drm_exec_snapshot - drm_exec snapshot information +*/ +struct drm_exec_snapshot { +/** @saved_snap: Pointer to the previous snapshot or NULL. */ +struct drm_exec_snapshot *saved_snap; + +/** @prelocked: Refcounted pointer to the prelocked object at snapshot time. */ +struct drm_gem_object *prelocked; + +/** @num_locked: Number of locked objects at snapshot time. */ +unsigned long num_locked; };int drm_exec_handle_contended (struct drm_exec *exec);
@@ -160,5 +178,8 @@ int drm_exec_prepare_array(struct drm_exec *exec,struct drm_gem_object **objects, unsigned int num_objects, unsigned int num_fences); +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap); +void drm_exec_restore (struct drm_exec *exec,struct drm_exec_snapshot *snap); +#endif
Signature · 1 line
2.44.0
Christian König
maintainerRe: [RFC PATCH v3 15/21] drm/exec: Add a snapshot capability
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 10 lines
When validating a buffer object for submission, we might need to lock a number of object for eviction to make room for the validation.
This makes it pretty likely that validation will eventually succeed, since eventually the validating process will hold most
dma_resvlocks of the buffer objects residing in the memory type being validated for.However, once validation of a single object has succeeded it might not be beneficial to hold on to those locks anymore, and the validator would want to drop the locks of all objects taken during validation.
Exactly avoiding that was one of the goals of developing the drm_exec object.
When objects are unlocked after evicting them it just gives concurrent operations an opportunity to lock them and re-validate them into the contended domain.
So why should that approach here be beneficial at all?
Regards,
Christian.
Quoted 151 lines
Introduce a
drm_execsnapshot functionality that can be used to record the locks held at a certain time, and a restore functionality that restores thedrm_execstate to the snapshot by dropping all locks.Snapshots can be nested if needed.
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- drivers/gpu/drm/drm_exec.c | 55 +++++++++++++++++++++++++++++++++++++- include/drm/drm_exec.h | 23 +++++++++++++++- 2 files changed, 76 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_exec.c +54
-1@@ -57,6 +57,7 @@ static void drm_exec_unlock_all(struct drm_exec *exec)struct drm_gem_object *obj;unsigned long index; +WARN_ON (exec->snap);drm_exec_for_each_locked_object_reverse (exec, index, obj) {dma_resv_unlock (obj->resv);drm_gem_object_put (obj);@@ -90,6 +91,7 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->num_objects =0 ; exec->contended =DRM_EXEC_DUMMY ; exec->prelocked =NULL ; + exec->snap =NULL ; }EXPORT_SYMBOL (drm_exec_init);@@ -301,7 +303,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) goto error_unlock;return 0 ; - error_unlock:dma_resv_unlock (obj->resv);return ret;@@ -395,5 +396,57 @@ int drm_exec_prepare_array(struct drm_exec *exec, }EXPORT_SYMBOL (drm_exec_prepare_array); +/** +* drm_exec_restore() - Restore the drm_exec state to the point of a snapshot. +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Restores the drm_exec object by means of unlocking and dropping references +* to objects locked after the snapshot. +*/ +void drm_exec_restore (struct drm_exec *exec,struct drm_exec_snapshot *snap) +{ +struct drm_gem_object *obj; +unsigned int index; + + exec->snap = snap->saved_snap; + +drm_exec_for_each_locked_object_reverse (exec, index, obj) { +if (index +1 == snap->num_locked) +break ; + +dma_resv_unlock (obj->resv); +drm_gem_object_put (obj); + exec->objects[index] =NULL ; + } + + exec->num_objects = snap->num_locked; + +if (!exec->prelocked) + exec->prelocked = snap->prelocked; +else +drm_gem_object_put (snap->prelocked); +} +EXPORT_SYMBOL (drm_exec_restore); + +/** +* drm_exec_snapshot() - Take a snapshot of the drm_exec state +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Records the @exec state in @snap. The @snap object is typically allocated +* in the stack of the caller. +*/ +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap) +{ + snap->num_locked = exec->num_objects; + snap->prelocked = exec->prelocked; +if (snap->prelocked) +drm_gem_object_get (snap->prelocked); + snap->saved_snap = exec->snap; + exec->snap = snap; +} +EXPORT_SYMBOL (drm_exec_snapshot); +MODULE_DESCRIPTION ("DRM execution context" );MODULE_LICENSE ("Dual MIT/GPL" );include/drm/drm_exec.h +22
-1@@ -19,7 +19,6 @@ struct drm_exec {* @flags: Flags to control locking behavior */ u32 flags; -/** * @ticket: WW ticket used for acquiring locks */ @@ -49,6 +48,25 @@ struct drm_exec {* @prelocked: already locked GEM object due to contention */ struct drm_gem_object *prelocked; + +/** +* @snap: Pointer to the last snapshot taken or NULL if none. +*/ +struct drm_exec_snapshot *snap; +}; + +/** +* struct drm_exec_snapshot - drm_exec snapshot information +*/ +struct drm_exec_snapshot { +/** @saved_snap: Pointer to the previous snapshot or NULL. */ +struct drm_exec_snapshot *saved_snap; + +/** @prelocked: Refcounted pointer to the prelocked object at snapshot time. */ +struct drm_gem_object *prelocked; + +/** @num_locked: Number of locked objects at snapshot time. */ +unsigned long num_locked; };int drm_exec_handle_contended (struct drm_exec *exec);@@ -160,5 +178,8 @@ int drm_exec_prepare_array(struct drm_exec *exec,struct drm_gem_object **objects, unsigned int num_objects, unsigned int num_fences); +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap); +void drm_exec_restore (struct drm_exec *exec,struct drm_exec_snapshot *snap); +#endif
Thomas Hellström
maintainerRe: [RFC PATCH v3 15/21] drm/exec: Add a snapshot capability
Quoted 21 lines · Christian König
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 11 lines
When validating a buffer object for submission, we might need to lock
a number of object for eviction to make room for the validation.This makes it pretty likely that validation will eventually succeed,
since eventually the validating process will hold mostdma_resvlocks
of the buffer objects residing in the memory type being validated for.Quoted 5 lines
However, once validation of a single object has succeeded it might not
be beneficial to hold on to those locks anymore, and the validator would want to drop the locks of all objects taken during validation.Exactly avoiding that was one of the goals of developing the
drm_execobject.
Quoted 6 lines
When objects are unlocked after evicting them it just gives concurrent
operations an opportunity to lock them and re-validate them into the contended domain.So why should that approach here be beneficial at all?
It's a matter of being nice to the rest of the system while *still guaranteeing progress*. For each object we're trying to validate, we keep on evicting other objects until we make progress even if we lock all the objects in the domain.
If we were unlocking after each eviction, we can't really guarantee progress.
OTOH, a concurrent locker of the object may well be one with higher priority (lower ticket number) just wanting to perform a pagefault
So it's a tradeoff between locking just locking other processes out to allow us to make one step of progress and to in addition hit them with the big sledgehammer.
/Thomas
Quoted 177 lines
Regards,
Christian.Quoted 172 lines
Introduce a
drm_execsnapshot functionality that can be used to record the locks held at a certain time, and a restore functionality
that restores thedrm_execstate to the snapshot by dropping all locks.Snapshots can be nested if needed.
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/drm_exec.c| 55
+++++++++++++++++++++++++++++++++++++-
include/drm/drm_exec.h | 23 +++++++++++++++-2 files changed, 76 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_exec.c +0
-0@@ -57,6 +57,7 @@ static void drm_exec_unlock_all(struct drm_exec *exec) struct drm_gem_object *obj; unsigned long index;drivers/gpu/drm/drm_exec.c +1
-0+WARN_ON (exec->snap);
drm_exec_for_each_locked_object_reverse(exec, index, obj){
dma_resv_unlock(obj->resv);drm_gem_object_put(obj);drivers/gpu/drm/drm_exec.c +1
-0@@ -90,6 +91,7 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->num_objects = 0; exec->contended = DRM_EXEC_DUMMY; exec->prelocked = NULL; + exec->snap =NULL ; } EXPORT_SYMBOL(drm_exec_init);drivers/gpu/drm/drm_exec.c +0
-0@@ -301,7 +303,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) goto error_unlock;
return 0;
-
error_unlock:
dma_resv_unlock(obj->resv);
return ret;drivers/gpu/drm/drm_exec.c +0
-0@@ -395,5 +396,57 @@ int drm_exec_prepare_array(struct drm_exec *exec, } EXPORT_SYMBOL(drm_exec_prepare_array);
+/**
+ *drm_exec_restore()- Restore thedrm_execstate to the point of a snapshot.
+ * @exec: Thedrm_execobject with the state.
+ * @snap: The snapshot state.
+ *
+ * Restores thedrm_execobject by means of unlocking and dropping references
+ * to objects locked after the snapshot.
+ */
+voiddrm_exec_restore(struct drm_exec *exec, struct drm_exec_snapshot *snap)drivers/gpu/drm/drm_exec.c +7
-0+{ +struct drm_gem_object *obj; +unsigned int index; + + exec->snap = snap->saved_snap; + +drm_exec_for_each_locked_object_reverse (exec, index, obj){
drivers/gpu/drm/drm_exec.c +23
-0+if (index +1 == snap->num_locked) +break ; + +dma_resv_unlock (obj->resv); +drm_gem_object_put (obj); + exec->objects[index] =NULL ; + } + + exec->num_objects = snap->num_locked; + +if (!exec->prelocked) + exec->prelocked = snap->prelocked; +else +drm_gem_object_put (snap->prelocked); +} +EXPORT_SYMBOL (drm_exec_restore); + +/** +* drm_exec_snapshot() - Take a snapshot of the drm_exec state +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Records the @exec state in @snap. The @snap object is typically allocated
+ * in the stack of the caller.
+ */
+voiddrm_exec_snapshot(struct drm_exec *exec, struct drm_exec_snapshot *snap)drivers/gpu/drm/drm_exec.c +10
-0+{ + snap->num_locked = exec->num_objects; + snap->prelocked = exec->prelocked; +if (snap->prelocked) +drm_gem_object_get (snap->prelocked); + snap->saved_snap = exec->snap; + exec->snap = snap; +} +EXPORT_SYMBOL (drm_exec_snapshot); +
MODULE_DESCRIPTION("DRM execution context");
MODULE_LICENSE("Dual MIT/GPL");include/drm/drm_exec.h +3
-1@@ -19,7 +19,6 @@ struct drm_exec {* @flags: Flags to control locking behavior */ u32 flags; - /*** @ticket: WW ticket used for acquiring locks */ @@ -49,6 +48,25 @@ struct drm_exec {* @prelocked: already locked GEM object due to contention */ struct drm_gem_object *prelocked; + +/** +* @snap: Pointer to the last snapshot taken or NULL if none.
include/drm/drm_exec.h +9
-0+*/ +struct drm_exec_snapshot *snap; +}; + +/** +* struct drm_exec_snapshot - drm_exec snapshot information +*/ +struct drm_exec_snapshot { +/** @saved_snap: Pointer to the previous snapshot or NULL. */
include/drm/drm_exec.h +3
-0+struct drm_exec_snapshot *saved_snap; + +/** @prelocked: Refcounted pointer to the prelocked object at snapshot time. */
include/drm/drm_exec.h +3
-0+struct drm_gem_object *prelocked; + +/** @num_locked: Number of locked objects at snapshot time. */
include/drm/drm_exec.h +1
-0+unsigned long num_locked;};
intdrm_exec_handle_contended(struct drm_exec *exec);include/drm/drm_exec.h +2
-0@@ -160,5 +178,8 @@ int drm_exec_prepare_array(struct drm_exec *exec, struct drm_gem_object **objects, unsigned int num_objects, unsigned int num_fences); +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap); +void drm_exec_restore(struct drm_exec *exec, struct
drm_exec_snapshot*snap);
+
#endif
Thomas Hellström
maintainerRe: [RFC PATCH v3 15/21] drm/exec: Add a snapshot capability
Quoted 41 lines · Thomas Hellström
Quoted 24 lines · Christian König
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 11 lines
When validating a buffer object for submission, we might need to lock
a number of object for eviction to make room for the validation.This makes it pretty likely that validation will eventually succeed,
since eventually the validating process will hold mostdma_resvlocks
of the buffer objects residing in the memory type being validated for.Quoted 7 lines
However, once validation of a single object has succeeded it might
not
be beneficial to hold on to those locks anymore, and the validator
would want to drop the locks of all objects taken during validation.Exactly avoiding that was one of the goals of developing the
drm_exec
object.Quoted 7 lines
When objects are unlocked after evicting them it just gives concurrent
operations an opportunity to lock them and re-validate them into the
contended domain.So why should that approach here be beneficial at all?
It's a matter of being nice to the rest of the system while *still guaranteeing progress*. For each object we're trying to validate, we keep on evicting other objects until we make progress even if we lock all the objects in the domain.
If we were unlocking after each eviction, we can't really guarantee progress.
Quoted 8 lines
OTOH, a concurrent locker of the object may well be one with higher priority (lower ticket number) just wanting to perform a pagefault
So it's a tradeoff between locking just locking other processes out to
allow us to make one step of progress and to in addition hit them with
the big sledgehammer.
I thought I'd also mention that the ideal solution here I think would be to have an rw_mutex per manager. Ordinary allocations take it in read mode, evictions take it in write mode. Now the bad thing is it sits in between ww_mutexes so it would have to be a ww_rw_mutex which would probably be too nasty to implement.
/Thomas
Quoted 190 lines
/Thomas
Quoted 186 lines
Regards,
Christian.Quoted 181 lines
Introduce a
drm_execsnapshot functionality that can be used to record the locks held at a certain time, and a restore functionality
that restores thedrm_execstate to the snapshot by dropping all locks.Snapshots can be nested if needed.
Cc: Christian König <[email protected]>
Cc: Somalapuram Amaranath <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: <[email protected]>
Signed-off-by: Thomas Hellström <[email protected]>
---
drivers/gpu/drm/drm_exec.c| 55
+++++++++++++++++++++++++++++++++++++-
include/drm/drm_exec.h | 23 +++++++++++++++-2 files changed, 76 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_exec.c +0
-0@@ -57,6 +57,7 @@ static void drm_exec_unlock_all(struct drm_exec *exec) struct drm_gem_object *obj; unsigned long index;drivers/gpu/drm/drm_exec.c +1
-0+WARN_ON (exec->snap);
drm_exec_for_each_locked_object_reverse(exec, index, obj)
{
dma_resv_unlock(obj->resv);drm_gem_object_put(obj);drivers/gpu/drm/drm_exec.c +1
-0@@ -90,6 +91,7 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->num_objects = 0; exec->contended = DRM_EXEC_DUMMY; exec->prelocked = NULL; + exec->snap =NULL ; } EXPORT_SYMBOL(drm_exec_init);drivers/gpu/drm/drm_exec.c +0
-0@@ -301,7 +303,6 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) goto error_unlock;
return 0;
-
error_unlock:
dma_resv_unlock(obj->resv);
return ret;drivers/gpu/drm/drm_exec.c +0
-0@@ -395,5 +396,57 @@ int drm_exec_prepare_array(struct drm_exec *exec, } EXPORT_SYMBOL(drm_exec_prepare_array);
+/**
+ *drm_exec_restore()- Restore thedrm_execstate to the point of
a snapshot.
+ * @exec: Thedrm_execobject with the state.
+ * @snap: The snapshot state.
+ *
+ * Restores thedrm_execobject by means of unlocking and dropping
references
+ * to objects locked after the snapshot.
+ */
+voiddrm_exec_restore(struct drm_exec *exec, struct drm_exec_snapshot *snap)drivers/gpu/drm/drm_exec.c +7
-0+{ +struct drm_gem_object *obj; +unsigned int index; + + exec->snap = snap->saved_snap; + + drm_exec_for_each_locked_object_reverse(exec, index,obj)
{drivers/gpu/drm/drm_exec.c +23
-0+if (index +1 == snap->num_locked) +break ; + +dma_resv_unlock (obj->resv); +drm_gem_object_put (obj); + exec->objects[index] =NULL ; + } + + exec->num_objects = snap->num_locked; + +if (!exec->prelocked) + exec->prelocked = snap->prelocked; +else +drm_gem_object_put (snap->prelocked); +} +EXPORT_SYMBOL (drm_exec_restore); + +/** +* drm_exec_snapshot() - Take a snapshot of the drm_exec state +* @exec: The drm_exec object with the state. +* @snap: The snapshot state. +* +* Records the @exec state in @snap. The @snap object is typically
allocated
+ * in the stack of the caller.
+ */
+voiddrm_exec_snapshot(struct drm_exec *exec, struct drm_exec_snapshot *snap)drivers/gpu/drm/drm_exec.c +10
-0+{ + snap->num_locked = exec->num_objects; + snap->prelocked = exec->prelocked; +if (snap->prelocked) +drm_gem_object_get (snap->prelocked); + snap->saved_snap = exec->snap; + exec->snap = snap; +} +EXPORT_SYMBOL (drm_exec_snapshot); +
MODULE_DESCRIPTION("DRM execution context");
MODULE_LICENSE("Dual MIT/GPL");include/drm/drm_exec.h +0
-1@@ -19,7 +19,6 @@ struct drm_exec {* @flags: Flags to control locking behavior */ u32 flags; - /*** @ticket: WW ticket used for acquiring locks */ @@ -49,6 +48,25 @@ struct drm_exec { * @prelocked: already locked GEM object due tocontention
*/
structdrm_gem_object*prelocked;include/drm/drm_exec.h +3
-0+ +/** +* @snap: Pointer to the last snapshot taken or NULL if none.
include/drm/drm_exec.h +9
-0+*/ +struct drm_exec_snapshot *snap; +}; + +/** +* struct drm_exec_snapshot - drm_exec snapshot information +*/ +struct drm_exec_snapshot { +/** @saved_snap: Pointer to the previous snapshot or NULL.
*/include/drm/drm_exec.h +3
-0+struct drm_exec_snapshot *saved_snap; + +/** @prelocked: Refcounted pointer to the prelocked object
at snapshot time. */include/drm/drm_exec.h +3
-0+struct drm_gem_object *prelocked; + +/** @num_locked: Number of locked objects at snapshot time. */
include/drm/drm_exec.h +1
-0+unsigned long num_locked;};
intdrm_exec_handle_contended(struct drm_exec *exec);include/drm/drm_exec.h +2
-0@@ -160,5 +178,8 @@ int drm_exec_prepare_array(struct drm_exec *exec, struct drm_gem_object **objects, unsigned int num_objects, unsigned int num_fences); +void drm_exec_snapshot (struct drm_exec *exec,struct drm_exec_snapshot *snap); +void drm_exec_restore(struct drm_exec *exec, struct
drm_exec_snapshot*snap);
+
#endif
Thomas Hellström
maintainerRFC PATCH v3 16/21 drm/exec: Introduce an evict mode
In reply to Thomas HellströmLocking for eviction is in some way different from locking for submission:
1) We can't lock objects that are already locked for submission, hence DRM_EXEC_IGNORE_DUPLICATES must be unset.
2) We must be able to re-lock objects locked for eviction, either for submission or for yet another eviction, in particular objects sharing a single resv must be considered.
3) There is no point to keep a contending object after the transaction restart. We don't know whether we actually want to use it again.
So introduce a drm_exec evict mode, and for now instead of explicitly setting it using a function call or implement separate locking functions that use evict mode, assume evict mode if there is a snapshot registered. This can easily be changed later.
To keep track of resvs locked for eviction, use a pointer set implemented by an xarray. This is probably not the most efficient data structure but used as an easy-to-implement first approach. If the set is empty (evict mode never used), the performance- and memory usage impact will be very small.
TODO: Probably want to implement the set using an open addressing hash table.
--- drivers/gpu/drm/drm_exec.c | 77 ++++++++++++++++++++++++++++++++++---- include/drm/drm_exec.h | 15 ++++++++ 2 files changed, 85 insertions(+), 7 deletions(-)
drivers/gpu/drm/drm_exec.c +70 -7
@@ -65,6 +65,10 @@ static void drm_exec_unlock_all(struct drm_exec *exec)drm_gem_object_put (exec->prelocked); exec->prelocked =NULL ; + +/* garbage collect */ +xa_destroy (&exec->resv_set); +xa_init (&exec->resv_set); }/**
@@ -92,6 +96,8 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->contended =DRM_EXEC_DUMMY ; exec->prelocked =NULL ; exec->snap =NULL ; + exec->drop_contended = false; +xa_init (&exec->resv_set); }EXPORT_SYMBOL (drm_exec_init);
@@ -110,6 +116,7 @@ void drm_exec_fini(struct drm_exec *exec)drm_gem_object_put (exec->contended);ww_acquire_fini (&exec->ticket); } +xa_destroy (&exec->resv_set); }EXPORT_SYMBOL (drm_exec_fini);
@@ -139,6 +146,30 @@ bool drm_exec_cleanup(struct drm_exec *exec) }EXPORT_SYMBOL (drm_exec_cleanup); +static unsigned long drm_exec_resv_to_key (const struct dma_resv *resv) +{ + return (unsigned long)resv / __alignof__(typeof(*resv)); +} + +static void +drm_exec_resv_set_erase (struct drm_exec *exec,unsigned long key) +{ +if (xa_load (&exec->resv_set, key)) +xa_erase (&exec->resv_set, key); +} + +static bool drm_exec_in_evict_mode (struct drm_exec *exec) +{ +return !!exec->snap; +} + +static void drm_exec_set_evict_mode (struct drm_exec *exec, +struct drm_exec_snapshot *snap) +{ + exec->snap = snap; + exec->flags &= ~DRM_EXEC_IGNORE_DUPLICATES ; +} +/* Track the locked object in the array */ static int drm_exec_obj_locked(struct drm_exec *exec, struct drm_gem_object *obj,
@@ -161,6 +192,14 @@ static int drm_exec_obj_locked(struct drm_exec *exec,drm_gem_object_get (obj); exec->objects[exec->num_objects++] = obj; +/* +* Errors here are not fatal, It means the object we locked +* for eviction can't be locked again. If that is problematic +* we may need to reconsider this. +*/ +if (drm_exec_in_evict_mode (exec)) + (void )xa_store (&exec->resv_set,drm_exec_resv_to_key (obj->resv), + obj->resv, gfp | __GFP_NOWARN);return 0 ; }
@@ -184,6 +223,9 @@ static int drm_exec_lock_contended(struct drm_exec *exec)dma_resv_lock_slow (obj->resv, &exec->ticket); } +if (exec->drop_contended) + goto error_unlock; + ret =drm_exec_obj_locked (exec, obj,GFP_KERNEL );if (unlikely (ret)) goto error_unlock;
@@ -245,10 +287,19 @@ int drm_exec_trylock_obj(struct drm_exec *exec, struct drm_gem_object *obj) }if (!dma_resv_trylock_ctx (obj->resv, &exec->ticket)) { -if (dma_resv_locking_ctx (obj->resv) == &exec->ticket) -return (exec->flags &DRM_EXEC_IGNORE_DUPLICATES ) ?0 : -EALREADY ; -else +if (dma_resv_locking_ctx (obj->resv) == &exec->ticket) { +unsigned long key =drm_exec_resv_to_key (obj->resv); + +if (exec->flags &DRM_EXEC_IGNORE_DUPLICATES || +xa_load (&exec->resv_set, key)) { +if (!drm_exec_in_evict_mode (exec)) +drm_exec_resv_set_erase (exec, key); +return 0 ; + } +return -EALREADY ; + }else {return -EBUSY ; + } } ret =drm_exec_obj_locked (exec, obj,GFP_ATOMIC | __GFP_NOWARN);
@@ -288,12 +339,20 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj)if (unlikely (ret == -EDEADLK )) {drm_gem_object_get (obj); exec->contended = obj; + exec->drop_contended =drm_exec_in_evict_mode (exec);return -EDEADLK ; } -if (unlikely (ret == -EALREADY ) && - exec->flags &DRM_EXEC_IGNORE_DUPLICATES ) -return 0 ; +if (unlikely (ret == -EALREADY )) { +unsigned long key =drm_exec_resv_to_key (obj->resv); + +if (exec->flags &DRM_EXEC_IGNORE_DUPLICATES || +xa_load (&exec->resv_set, key)) { +if (!drm_exec_in_evict_mode (exec)) +drm_exec_resv_set_erase (exec, key); +return 0 ; + } + }if (unlikely (ret))return ret;
@@ -324,6 +383,7 @@ void drm_exec_unlock_obj(struct drm_exec *exec, struct drm_gem_object *obj)for (i = exec->num_objects; i--;) {if (exec->objects[i] == obj) { +drm_exec_resv_set_erase (exec,drm_exec_resv_to_key (obj->resv));dma_resv_unlock (obj->resv);for (++i; i < exec->num_objects; ++i) exec->objects[i -1 ] = exec->objects[i];
@@ -415,12 +475,14 @@ void drm_exec_restore(struct drm_exec *exec, struct drm_exec_snapshot *snap)if (index +1 == snap->num_locked)break ; +xa_erase (&exec->resv_set,drm_exec_resv_to_key (obj->resv));dma_resv_unlock (obj->resv);drm_gem_object_put (obj); exec->objects[index] =NULL ; } exec->num_objects = snap->num_locked; + exec->flags = snap->flags;if (!exec->prelocked) exec->prelocked = snap->prelocked;
@@ -443,8 +505,9 @@ void drm_exec_snapshot(struct drm_exec *exec, struct drm_exec_snapshot *snap) snap->prelocked = exec->prelocked;if (snap->prelocked)drm_gem_object_get (snap->prelocked); + snap->flags = exec->flags; snap->saved_snap = exec->snap; - exec->snap = snap; +drm_exec_set_evict_mode (exec, snap); }EXPORT_SYMBOL (drm_exec_snapshot);
include/drm/drm_exec.h +15 -0
@@ -5,6 +5,7 @@#include <linux/compiler.h> #include <linux/ww_mutex.h> +#include <linux/xarray.h> #define DRM_EXEC_INTERRUPTIBLE_WAIT BIT(0)#define DRM_EXEC_IGNORE_DUPLICATES BIT(1)
@@ -53,6 +54,17 @@ struct drm_exec {* @snap: Pointer to the last snapshot taken or NULL if none. */ struct drm_exec_snapshot *snap; + +/** +* @resv_set: Set of pointers to locked objects in evict mode. +*/ +struct xarray resv_set; + +/** +* @drop_contended: Drop the contended object after WW transaction +* relaxation. +*/ +bool drop_contended; };/**
@@ -67,6 +79,9 @@ struct drm_exec_snapshot {/** @num_locked: Number of locked objects at snapshot time. */ unsigned long num_locked; + +/** @flags: The drm_exec flags at snapshot time. */ +u32 flags; };int drm_exec_handle_contended (struct drm_exec *exec);
Signature · 1 line
2.44.0
Christian König
maintainerRe: [RFC PATCH v3 16/21] drm/exec: Introduce an evict mode
Am 21.05.24 um 09:16 schrieb Thomas Hellström:
Quoted 8 lines
Locking for eviction is in some way different from locking for submission:
1) We can't lock objects that are already locked for submission, hence
DRM_EXEC_IGNORE_DUPLICATESmust be unset.
2) We must be able to re-lock objects locked for eviction, either for submission or for yet another eviction, in particular objects sharing a single resv must be considered.
Yeah, I was already thinking about that as well.
My idea so far was to have a separate function for locking eviction BOs. This function would then use trylock or blocking depending on some setting.
Quoted 3 lines
3) There is no point to keep a contending object after the transaction restart. We don't know whether we actually want to use it again.
Well that isn't true as far as I know.
If we don't use trylock we still need to lock the object after rollback to make sure that we waited for it to become available.
Regards,
Christian.
Quoted 233 lines
So introduce a
drm_execevict mode, and for now instead of explicitly setting it using a function call or implement separate locking functions that use evict mode, assume evict mode if there is a snapshot registered. This can easily be changed later.To keep track of resvs locked for eviction, use a pointer set implemented by an xarray. This is probably not the most efficient data structure but used as an easy-to-implement first approach. If the set is empty (evict mode never used), the performance- and memory usage impact will be very small.
TODO: Probably want to implement the set using an open addressing hash table.
Cc Christian König <[email protected]>Cc Somalapuram Amaranath <[email protected]>Cc Matthew Brost <[email protected]>Cc <[email protected]>Signed-off-by Thomas Hellström <[email protected]>--- drivers/gpu/drm/drm_exec.c | 77 ++++++++++++++++++++++++++++++++++---- include/drm/drm_exec.h | 15 ++++++++ 2 files changed, 85 insertions(+), 7 deletions(-)drivers/gpu/drm/drm_exec.c +70
-7@@ -65,6 +65,10 @@ static void drm_exec_unlock_all(struct drm_exec *exec)drm_gem_object_put (exec->prelocked); exec->prelocked =NULL ; + +/* garbage collect */ +xa_destroy (&exec->resv_set); +xa_init (&exec->resv_set); }/** @@ -92,6 +96,8 @@ void drm_exec_init(struct drm_exec *exec, u32 flags, unsigned nr) exec->contended =DRM_EXEC_DUMMY ; exec->prelocked =NULL ; exec->snap =NULL ; + exec->drop_contended = false; +xa_init (&exec->resv_set); }EXPORT_SYMBOL (drm_exec_init);@@ -110,6 +116,7 @@ void drm_exec_fini(struct drm_exec *exec)drm_gem_object_put (exec->contended);ww_acquire_fini (&exec->ticket); } +xa_destroy (&exec->resv_set); }EXPORT_SYMBOL (drm_exec_fini);@@ -139,6 +146,30 @@ bool drm_exec_cleanup(struct drm_exec *exec) }EXPORT_SYMBOL (drm_exec_cleanup); +static unsigned long drm_exec_resv_to_key (const struct dma_resv *resv) +{ + return (unsigned long)resv / __alignof__(typeof(*resv)); +} + +static void +drm_exec_resv_set_erase (struct drm_exec *exec,unsigned long key) +{ +if (xa_load (&exec->resv_set, key)) +xa_erase (&exec->resv_set, key); +} + +static bool drm_exec_in_evict_mode (struct drm_exec *exec) +{ +return !!exec->snap; +} + +static void drm_exec_set_evict_mode (struct drm_exec *exec, +struct drm_exec_snapshot *snap) +{ + exec->snap = snap; + exec->flags &= ~DRM_EXEC_IGNORE_DUPLICATES ; +} +/* Track the locked object in the array */ static int drm_exec_obj_locked(struct drm_exec *exec, struct drm_gem_object *obj,@@ -161,6 +192,14 @@ static int drm_exec_obj_locked(struct drm_exec *exec,drm_gem_object_get (obj); exec->objects[exec->num_objects++] = obj; +/* +* Errors here are not fatal, It means the object we locked +* for eviction can't be locked again. If that is problematic +* we may need to reconsider this. +*/ +if (drm_exec_in_evict_mode (exec)) + (void )xa_store (&exec->resv_set,drm_exec_resv_to_key (obj->resv), + obj->resv, gfp | __GFP_NOWARN);return 0 ; }@@ -184,6 +223,9 @@ static int drm_exec_lock_contended(struct drm_exec *exec)dma_resv_lock_slow (obj->resv, &exec->ticket); } +if (exec->drop_contended) + goto error_unlock; + ret =drm_exec_obj_locked (exec, obj,GFP_KERNEL );if (unlikely (ret)) goto error_unlock;@@ -245,10 +287,19 @@ int drm_exec_trylock_obj(struct drm_exec *exec, struct drm_gem_object *obj) }if (!dma_resv_trylock_ctx (obj->resv, &exec->ticket)) { -if (dma_resv_locking_ctx (obj->resv) == &exec->ticket) -return (exec->flags &DRM_EXEC_IGNORE_DUPLICATES ) ?0 : -EALREADY ; -else +if (dma_resv_locking_ctx (obj->resv) == &exec->ticket) { +unsigned long key =drm_exec_resv_to_key (obj->resv); + +if (exec->flags &DRM_EXEC_IGNORE_DUPLICATES || +xa_load (&exec->resv_set, key)) { +if (!drm_exec_in_evict_mode (exec)) +drm_exec_resv_set_erase (exec, key); +return 0 ; + } +return -EALREADY ; + }else {return -EBUSY ; + } } ret =drm_exec_obj_locked (exec, obj,GFP_ATOMIC | __GFP_NOWARN);@@ -288,12 +339,20 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj)if (unlikely (ret == -EDEADLK )) {drm_gem_object_get (obj); exec->contended = obj; + exec->drop_contended =drm_exec_in_evict_mode (exec);return -EDEADLK ; } -if (unlikely (ret == -EALREADY ) && - exec->flags &DRM_EXEC_IGNORE_DUPLICATES ) -return 0 ; +if (unlikely (ret == -EALREADY )) { +unsigned long key =drm_exec_resv_to_key (obj->resv); + +if (exec->flags &DRM_EXEC_IGNORE_DUPLICATES || +xa_load (&exec->resv_set, key)) { +if (!drm_exec_in_evict_mode (exec)) +drm_exec_resv_set_erase (exec, key); +return 0 ; + } + }if (unlikely (ret))return ret;@@ -324,6 +383,7 @@ void drm_exec_unlock_obj(struct drm_exec *exec, struct drm_gem_object *obj)for (i = exec->num_objects; i--;) {if (exec->objects[i] == obj) { +drm_exec_resv_set_erase (exec,drm_exec_resv_to_key (obj->resv));dma_resv_unlock (obj->resv);for (++i; i < exec->num_objects; ++i) exec->objects[i -1 ] = exec->objects[i];@@ -415,12 +475,14 @@ void drm_exec_restore(struct drm_exec *exec, struct drm_exec_snapshot *snap)if (index +1 == snap->num_locked)break ; +xa_erase (&exec->resv_set,drm_exec_resv_to_key (obj->resv));dma_resv_unlock (obj->resv);drm_gem_object_put (obj); exec->objects[index] =NULL ; } exec->num_objects = snap->num_locked; + exec->flags = snap->flags;if (!exec->prelocked) exec->prelocked = snap->prelocked;@@ -443,8 +505,9 @@ void drm_exec_snapshot(struct drm_exec *exec, struct drm_exec_snapshot *snap) snap->prelocked = exec->prelocked;if (snap->prelocked)drm_gem_object_get (snap->prelocked); + snap->flags = exec->flags; snap->saved_snap = exec->snap; - exec->snap = snap; +drm_exec_set_evict_mode (exec, snap); }EXPORT_SYMBOL (drm_exec_snapshot);include/drm/drm_exec.h +15
-0@@ -5,6 +5,7 @@#include <linux/compiler.h> #include <linux/ww_mutex.h> +#include <linux/xarray.h> #define DRM_EXEC_INTERRUPTIBLE_WAIT BIT(0)#define DRM_EXEC_IGNORE_DUPLICATES BIT(1)@@ -53,6 +54,17 @@ struct drm_exec {* @snap: Pointer to the last snapshot taken or NULL if none. */ struct drm_exec_snapshot *snap; + +/** +* @resv_set: Set of pointers to locked objects in evict mode. +*/ +struct xarray resv_set; + +/** +* @drop_contended: Drop the contended object after WW transaction +* relaxation. +*/ +bool drop_contended; };/** @@ -67,6 +79,9 @@ struct drm_exec_snapshot {/** @num_locked: Number of locked objects at snapshot time. */ unsigned long num_locked; + +/** @flags: The drm_exec flags at snapshot time. */ +u32 flags; };int drm_exec_handle_contended (struct drm_exec *exec);