Introduce drm evictable lru
Subsystems (10)
cover Introduce drm evictable lru
We plan to implement xe driver's shared virtual memory manager (aka SVM) without buffer object concept. This means we won't build our shared virtual memory manager upon TTM infrastructure like amdgpu does.
Even though this approach is more efficient, it does create a problem for memory eviction when there is memory pressure: memory allocated by SVM and memory allocated by TTM should be able to mutually evict from each other. TTM's resource manager maintains a LRU list for each memory type and this list is used to pick up the memory eviction victim. Since we don't use TTM for SVM implementation, SVM allocated memory can't be added to TTM resource manager's LRU list. Thus SVM allocated memory and TTM allocated memory are not mutually evictable.
See more discussion on this topic here:
https://www.spinics.net/lists/dri-devel/msg410740.html
This series solve this problem by creating a shared LRU list b/t SVM and TTM, or any other resource manager.
The basic idea is, abstract a drm_lru_entity structure which is supposed to be embedded in ttm_resource structure, or any other resource manager. The resource LRU list is a list of drm_lru_entity. drm_lru_entity has eviction function pointers which can be used to call back drivers' specific eviction function to evict a memory resource.
Introduce global drm_lru_manager to struct drm_device to manage LRU lists. Each memory type or memory region can have a LRU list. TTM resource manager's LRU list functions including bulk move functions are moved to drm lru manager. drm lru manager provides a evict_first function to evict the first memory resource from LRU list. This function can be called from TTM, SVM or any other resource manager, so all the memory allocated in the drm sub-system can be mutually evicted.
The lru_lock is also moved from struct ttm_device to struct drm_device.
Opens:
1) memory accounting: currently the ttm resource manager's memory accounting functions is kept at ttm resource manager. Since memory accounting should be cross TTM and SVM, it should be ideally in the drm lru manager layer. This will be polished in the future.
2) eviction callback function interface: The current eviction function interface is designed to meet TTM memory eviction requirements. When SVM is in the picture, this interface need to be futher tunned to meet SVM requirement also.
This series is not tested and it is only compiled for xe driver. Some minor changes are needed for other driver such as amdgpu, nouveau etc. I intended to send this out as a request for comment series to get some early feedback, to see whether this is the right direction to go. I will futher polish this series after a direction is agreed.
Oak Zeng (11):
drm/ttm: re-parameter ttm_device_init
drm: move lru_lock from ttm_device to drm_device
drm: introduce drm evictable LRU
drm: Add evict function pointer to drm lru entity
drm: Replace ttm macros with drm macros
drm/ttm: Set lru manager to ttm resource manager drm/ttm: re-parameterize a few ttm functions
drm: Initialize drm lru manager
drm/ttm: Use drm LRU manager iterator
drm/ttm: Implement ttm memory evict functions drm/ttm: Write ttm functions using drm lru manager functions
drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 + .../gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 10 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/drm_evictable_lru.c | 266 ++++++++++++++++++ drivers/gpu/drm/drm_gem_vram_helper.c | 10 +- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 6 +- drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 + drivers/gpu/drm/i915/intel_region_ttm.c | 4 +- drivers/gpu/drm/i915/selftests/mock_region.c | 2 +- drivers/gpu/drm/loongson/lsdc_ttm.c | 10 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 22 +- drivers/gpu/drm/qxl/qxl_ttm.c | 6 +- drivers/gpu/drm/radeon/radeon_ttm.c | 10 +- drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 247 ++++++++++++---- drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 55 ++-- drivers/gpu/drm/ttm/ttm_module.h | 3 +- drivers/gpu/drm/ttm/ttm_range_manager.c | 14 +- drivers/gpu/drm/ttm/ttm_resource.c | 242 +++------------- drivers/gpu/drm/ttm/ttm_sys_manager.c | 8 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 +- .../gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 + drivers/gpu/drm/xe/xe_bo.c | 48 ++-- drivers/gpu/drm/xe/xe_bo.h | 5 +- drivers/gpu/drm/xe/xe_device.c | 2 +- drivers/gpu/drm/xe/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 6 +- drivers/gpu/drm/xe/xe_migrate.c | 6 +- drivers/gpu/drm/xe/xe_res_cursor.h | 10 +- drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 8 +- drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 18 +- drivers/gpu/drm/xe/xe_vm.c | 6 +- drivers/gpu/drm/xe/xe_vm_types.h | 2 +- include/drm/drm_device.h | 12 + include/drm/drm_evictable_lru.h | 260 +++++++++++++++++ include/drm/ttm/ttm_bo.h | 10 +- include/drm/ttm/ttm_device.h | 13 +- include/drm/ttm/ttm_range_manager.h | 17 +- include/drm/ttm/ttm_resource.h | 117 +++----- 49 files changed, 1042 insertions(+), 501 deletions(-) create mode 100644 drivers/gpu/drm/drm_evictable_lru.c create mode 100644 include/drm/drm_evictable_lru.h
Signature · 1 line
2.26.3
RFC 1/11 drm/ttm: re-parameter ttm_device_init
Change the 3rd parameter of ttm_device_init from struct device * to struct drm_device *. This is a prepare work for moving lru_lock from ttm_device to drm_device.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +- drivers/gpu/drm/drm_gem_vram_helper.c | 2 +- drivers/gpu/drm/i915/intel_region_ttm.c | 2 +- drivers/gpu/drm/loongson/lsdc_ttm.c | 2 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 2 +- drivers/gpu/drm/radeon/radeon_ttm.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 7 ++++--- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 2 +- drivers/gpu/drm/xe/xe_device.c | 2 +- include/drm/ttm/ttm_device.h | 3 ++- 11 files changed, 15 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -1
@@ -1817,7 +1817,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)mutex_init (&adev->mman.gtt_window_lock);/* No others user of address space so set it to 0 */ - r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev->dev, + r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev_to_drm(adev), adev_to_drm(adev)->anon_inode->i_mapping, adev_to_drm(adev)->vma_offset_manager, adev->need_swiotlb,
drivers/gpu/drm/drm_gem_vram_helper.c +1 -1
@@ -1002,7 +1002,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev, vmm->vram_base = vram_base; vmm->vram_size = vram_size; - ret =ttm_device_init (&vmm->bdev, &bo_driver, dev->dev, + ret =ttm_device_init (&vmm->bdev, &bo_driver, dev, dev->anon_inode->i_mapping, dev->vma_offset_manager, false, true);
drivers/gpu/drm/i915/intel_region_ttm.c +1 -1
@@ -33,7 +33,7 @@ int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)struct drm_device *drm = &dev_priv->drm;return ttm_device_init (&dev_priv->bdev,i915_ttm_driver (), - drm->dev, drm->anon_inode->i_mapping, + drm, drm->anon_inode->i_mapping, drm->vma_offset_manager, false, false); }
drivers/gpu/drm/loongson/lsdc_ttm.c +1 -1
@@ -548,7 +548,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev)unsigned long num_gtt_pages;int ret; - ret =ttm_device_init (&ldev->bdev, &lsdc_bo_driver, ddev->dev, + ret =ttm_device_init (&ldev->bdev, &lsdc_bo_driver, ddev, ddev->anon_inode->i_mapping, ddev->vma_offset_manager, false, true);if (ret)
drivers/gpu/drm/nouveau/nouveau_ttm.c +1 -1
@@ -299,7 +299,7 @@ nouveau_ttm_init(struct nouveau_drm *drm) drm->agp.cma = pci->agp.cma; } - ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, drm->dev->dev, + ret = ttm_device_init(&drm->ttm.bdev, &nouveau_bo_driver, dev, dev->anon_inode->i_mapping, dev->vma_offset_manager, drm_need_swiotlb(drm->client.mmu.dmabits),
drivers/gpu/drm/radeon/radeon_ttm.c +1 -1
@@ -688,7 +688,7 @@ int radeon_ttm_init(struct radeon_device *rdev)int r;/* No others user of address space so set it to 0 */ - r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->dev, + r = ttm_device_init(&rdev->mman.bdev, &radeon_bo_driver, rdev->ddev, rdev->ddev->anon_inode->i_mapping, rdev->ddev->vma_offset_manager, rdev->need_swiotlb,
drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c +1 -1
@@ -16,7 +16,7 @@ int ttm_device_kunit_init(struct ttm_test_devices *priv,struct drm_device *drm = priv->drm;int err; - err =ttm_device_init (ttm, &ttm_dev_funcs, drm->dev, + err =ttm_device_init (ttm, &ttm_dev_funcs, drm, drm->anon_inode->i_mapping, drm->vma_offset_manager, use_dma_alloc, use_dma32);
drivers/gpu/drm/ttm/ttm_device.c +4 -3
@@ -179,7 +179,7 @@ EXPORT_SYMBOL(ttm_device_swapout);* * @bdev: A pointer to a struct ttm_device to initialize. * @funcs: Function table for the device. -* @dev: The core kernel device pointer for DMA mappings and allocations. +* @drm: drm_device pointer * @mapping: The address space to use for this bo. * @vma_manager: A pointer to a vma manager. * @use_dma_alloc: If coherent DMA allocation API should be used.
@@ -190,7 +190,7 @@ EXPORT_SYMBOL(ttm_device_swapout);* !0: Failure. */ int ttm_device_init (struct ttm_device *bdev,const struct ttm_device_funcs *funcs, -struct device *dev,struct address_space *mapping, +struct drm_device *drm,struct address_space *mapping,struct drm_vma_offset_manager *vma_manager,bool use_dma_alloc,bool use_dma32) {
@@ -213,7 +213,8 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func bdev->funcs = funcs;ttm_sys_man_init (bdev); -ttm_pool_init (&bdev->pool, dev,NUMA_NO_NODE , use_dma_alloc, use_dma32); +ttm_pool_init (&bdev->pool, drm?drm->dev:NULL ,NUMA_NO_NODE , + use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager;spin_lock_init (&bdev->lru_lock);
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +1 -1
@@ -1047,7 +1047,7 @@ static int vmw_driver_load(struct vmw_private *dev_priv, u32 pci_id) } ret = ttm_device_init(&dev_priv->bdev, &vmw_bo_driver, - dev_priv->drm.dev, + &dev_priv->drm, dev_priv->drm.anon_inode->i_mapping, dev_priv->drm.vma_offset_manager, dev_priv->map_mode == vmw_dma_alloc_coherent,
drivers/gpu/drm/xe/xe_device.c +1 -1
@@ -191,7 +191,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,if (IS_ERR (xe))return xe; - err =ttm_device_init (&xe->ttm, &xe_ttm_funcs, xe->drm.dev, + err =ttm_device_init (&xe->ttm, &xe_ttm_funcs, &xe->drm, xe->drm.anon_inode->i_mapping, xe->drm.vma_offset_manager, false, false);if (WARN_ON (err))
include/drm/ttm/ttm_device.h +2 -1
@@ -29,6 +29,7 @@#include <linux/workqueue.h> #include <drm/ttm/ttm_resource.h> #include <drm/ttm/ttm_pool.h> +#include <drm/drm_device.h> struct ttm_device ;struct ttm_placement ;
@@ -288,7 +289,7 @@ static inline void ttm_set_driver_manager(struct ttm_device *bdev, int type, }int ttm_device_init (struct ttm_device *bdev,const struct ttm_device_funcs *funcs, -struct device *dev,struct address_space *mapping, +struct drm_device *drm,struct address_space *mapping,struct drm_vma_offset_manager *vma_manager,bool use_dma_alloc,bool use_dma32);void ttm_device_fini (struct ttm_device *bdev);
Signature · 1 line
2.26.3
RFC 2/11 drm: move lru_lock from ttm_device to drm_device
In reply to Oak ZengIn the coming patches, we will share the lru list b/t ttm bo based memory allocator and hmm/svm based memory allocator. Thus lru_lock (which is used mainly to protect the lru list) is moved from struct ttm_device to struct drm_device, so this lock can be shared b/t those two memory allocators.
To minimize code change, struct ttm_device still hold a weak reference of lru_lock, so ttm layer can still reference to this lock easily.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +- drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++---------- drivers/gpu/drm/ttm/ttm_device.c | 18 ++++----- drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++---------- drivers/gpu/drm/xe/xe_bo.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 4 +- drivers/gpu/drm/xe/xe_vm.c | 4 +- include/drm/drm_device.h | 5 +++ include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_device.h | 4 +- 13 files changed, 72 insertions(+), 66 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -2
@@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec,void amdgpu_vm_move_to_lru_tail (struct amdgpu_device *adev,struct amdgpu_vm *vm) { -spin_lock (&adev->mman.bdev.lru_lock); +spin_lock (adev->mman.bdev.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&adev->mman.bdev.lru_lock); +spin_unlock (adev->mman.bdev.lru_lock); }/* Create scheduler entities for page table updates */
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +2 -2
@@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct ttm_resource_manager *man) vis_usage =amdgpu_vram_mgr_vis_size (adev, block);atomic64_add (vis_usage, &mgr->vis_usage); -spin_lock (&man->bdev->lru_lock); +spin_lock (man->bdev->lru_lock); man->usage += rsv->size; -spin_unlock (&man->bdev->lru_lock); +spin_unlock (man->bdev->lru_lock);list_move (&rsv->blocks, &mgr->reserved_pages); } }
drivers/gpu/drm/drm_drv.c +1 -0
@@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,INIT_LIST_HEAD (&dev->managed.resources);spin_lock_init (&dev->managed.lock); +spin_lock_init (&dev->lru_lock);/* no per-device feature limits by default */ dev->driver_features = ~0u ;
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +2 -2
@@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)/* * Put on the correct LRU list depending on the MADV status */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (shrinkable) {/* Try to keep shmem_tt from being considered for shrinking. */ bo->priority =TTM_MAX_BO_PRIORITY -1 ;
@@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj) }ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }/*
drivers/gpu/drm/ttm/ttm_bo.c +20 -20
@@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct ttm_buffer_object *bo,* @bo: The buffer object. * * Move this BO to the tail of all lru lists used to lookup and reserve an -* object. This function must be called with struct ttm_global::lru_lock +* object. This function must be called with struct drm_device::lru_lock * held, and is used to make a BO less likely to be considered for eviction. */ void ttm_bo_move_to_lru_tail (struct ttm_buffer_object *bo)
@@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,if (bo->bulk_move == bulk)return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); bo->bulk_move = bulk;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_set_bulk_move);
@@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo)* reference it any more. The only tricky case is the trylock on * the resv object while holding the lru_lock. */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); bo->base.resv = &bo->base._resv; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }return r;
@@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP, interruptible,
@@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,else if (lret ==0 )return -EBUSY ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (unlock_resv && !dma_resv_trylock (bo->base.resv)) {/* * We raced, and lost, someone else holds the reservation now,
@@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,* delayed destruction would succeed, so just return success * here. */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return 0 ; } ret =0 ;
@@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,if (ret) {if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return ret; } -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);ttm_bo_cleanup_memtype_use (bo);if (unlock_resv)
@@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)ttm_bo_flush_all_fences (bo); bo->deleted = true; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);/* * Make pinned bos immediately available to
@@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref) }kref_init (&bo->kref); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);INIT_WORK (&bo->delayed_delete, ttm_bo_delayed_delete);queue_work (bdev->wq, &bo->delayed_delete);
@@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,bool locked = false;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);ttm_resource_manager_for_each_res (man, &cursor, res) {bool busy;
@@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,if (!bo) {if (busy_bo && !ttm_bo_get_unless_zero (busy_bo)) busy_bo =NULL ; -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_mem_evict_wait_busy (busy_bo, ctx, ticket);if (busy_bo)ttm_bo_put (busy_bo);
@@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,return ret; } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_bo_evict (bo, ctx);if (locked)
@@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo) {dma_resv_assert_held (bo->base.resv);WARN_ON_ONCE (!kref_read (&bo->kref)); -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); ++bo->pin_count; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_pin);
@@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)if (WARN_ON_ONCE (!bo->pin_count))return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); --bo->pin_count;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_unpin);
@@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, }/* TODO: Cleanup the locking */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);/* * Move to system cached
drivers/gpu/drm/ttm/ttm_device.c +9 -9
@@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,unsigned i;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)
@@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,return ret; } } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);return 0 ; }EXPORT_SYMBOL (ttm_device_swapout);
@@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager; -spin_lock_init (&bdev->lru_lock); + bdev->lru_lock = &drm->lru_lock;INIT_LIST_HEAD (&bdev->pinned); bdev->dev_mapping = mapping;mutex_lock (&ttm_global_mutex);
@@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)drain_workqueue (bdev->wq);destroy_workqueue (bdev->wq); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i)if (list_empty (&man->lru[0 ]))pr_debug ("Swap list %d was clean\n" , i); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);ttm_pool_fini (&bdev->pool);ttm_global_release ();
@@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, {struct ttm_resource *res; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);while ((res =list_first_entry_or_null (list,typeof (*res), lru))) {struct ttm_buffer_object *bo = res->bo;
@@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,continue ;list_del_init (&res->lru); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);if (bo->ttm)ttm_tt_unpopulate (bo->bdev, bo->ttm);ttm_bo_put (bo); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock); } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); }void ttm_device_clear_dma_mappings (struct ttm_device *bdev)
drivers/gpu/drm/ttm/ttm_resource.c +21 -21
133 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.c +2 -2
@@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct ttm_buffer_object *ttm_bo)* the ttm_bo refcount is zero at this point. So trylocking *should* * always succeed here, as long as we hold the lru lock. */ -spin_lock (&ttm_bo->bdev->lru_lock); +spin_lock (ttm_bo->bdev->lru_lock); locked =dma_resv_trylock (ttm_bo->base.resv); -spin_unlock (&ttm_bo->bdev->lru_lock); +spin_unlock (ttm_bo->bdev->lru_lock);XE_WARN_ON (!locked);return locked;
drivers/gpu/drm/xe/xe_exec.c +2 -2
@@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)xe_vm_reactivate_rebind (vm);if (!err && !xe_vm_no_dma_fences (vm)) { -spin_lock (&xe->ttm.lru_lock); +spin_lock (xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&xe->ttm.lru_lock); +spin_unlock (xe->ttm.lru_lock); } err_repin:
drivers/gpu/drm/xe/xe_vm.c +2 -2
@@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct *w)#undef retry_required -spin_lock (&vm->xe->ttm.lru_lock); +spin_lock (vm->xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&vm->xe->ttm.lru_lock); +spin_unlock (vm->xe->ttm.lru_lock);/* Point of no return. */ arm_preempt_fences (vm, &preempt_fences);
include/drm/drm_device.h +5 -0
@@ -326,6 +326,11 @@ struct drm_device {*/ struct list_head debugfs_list; +/** +* @lru_lock: Protection for the per manager LRU and destroy lists. +*/ +spinlock_t lru_lock; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )
include/drm/ttm/ttm_bo.h +2 -2
@@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo);static inline void ttm_bo_move_to_lru_tail_unlocked (struct ttm_buffer_object *bo) { -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); } static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,
include/drm/ttm/ttm_device.h +2 -2
@@ -248,9 +248,9 @@ struct ttm_device {struct ttm_pool pool;/** -* @lru_lock: Protection for the per manager LRU and ddestroy lists. +* @lru_lock: Weak reference to drm_device::lru_lock. */ -spinlock_t lru_lock; +spinlock_t *lru_lock;/** * @pinned: Buffer objects which are pinned and so not on any LRU list.
Signature · 1 line
2.26.3
Christian König
maintainerRe: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
Am 02.11.23 um 05:32 schrieb Oak Zeng:
Quoted 10 lines
In the coming patches, we will share the lru list b/t ttm bo based memory allocator and hmm/svm based memory allocator. Thus
lru_lock(which is used mainly to protect the lru list) is moved from structttm_deviceto structdrm_device, so this lock can be shared b/t those two memory allocators.To minimize code change, struct
ttm_devicestill hold a weak reference oflru_lock, so ttm layer can still reference to this lock easily.
I would rather like to see drm_device to become the base class of ttm_device.
Similar to how drm_gem_object is the base class of ttm_buffer_object.
That is probably a bit more work, but would also eliminate some of the duplicate house keeping we currently have (e.g. bdev pointer in ttm_buffer_object etc...).
Moving then stuff from the ttm_device into the drm_device becomes trivial.
Regards,
Christian.
Quoted 566 lines
Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +- drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++---------- drivers/gpu/drm/ttm/ttm_device.c | 18 ++++----- drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++---------- drivers/gpu/drm/xe/xe_bo.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 4 +- drivers/gpu/drm/xe/xe_vm.c | 4 +- include/drm/drm_device.h | 5 +++ include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_device.h | 4 +- 13 files changed, 72 insertions(+), 66 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2
-2@@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec,void amdgpu_vm_move_to_lru_tail (struct amdgpu_device *adev,struct amdgpu_vm *vm) { -spin_lock (&adev->mman.bdev.lru_lock); +spin_lock (adev->mman.bdev.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&adev->mman.bdev.lru_lock); +spin_unlock (adev->mman.bdev.lru_lock); }/* Create scheduler entities for page table updates */ drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +2
-2@@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct ttm_resource_manager *man) vis_usage =amdgpu_vram_mgr_vis_size (adev, block);atomic64_add (vis_usage, &mgr->vis_usage); -spin_lock (&man->bdev->lru_lock); +spin_lock (man->bdev->lru_lock); man->usage += rsv->size; -spin_unlock (&man->bdev->lru_lock); +spin_unlock (man->bdev->lru_lock);list_move (&rsv->blocks, &mgr->reserved_pages); } }drivers/gpu/drm/drm_drv.c +1
-0@@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,INIT_LIST_HEAD (&dev->managed.resources);spin_lock_init (&dev->managed.lock); +spin_lock_init (&dev->lru_lock);/* no per-device feature limits by default */ dev->driver_features = ~0u ;drivers/gpu/drm/i915/gem/i915_gem_ttm.c +2
-2@@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)/* * Put on the correct LRU list depending on the MADV status */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (shrinkable) {/* Try to keep shmem_tt from being considered for shrinking. */ bo->priority =TTM_MAX_BO_PRIORITY -1 ;@@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj) }ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }/* drivers/gpu/drm/ttm/ttm_bo.c +20
-20@@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct ttm_buffer_object *bo,* @bo: The buffer object. * * Move this BO to the tail of all lru lists used to lookup and reserve an -* object. This function must be called with struct ttm_global::lru_lock +* object. This function must be called with struct drm_device::lru_lock * held, and is used to make a BO less likely to be considered for eviction. */ void ttm_bo_move_to_lru_tail (struct ttm_buffer_object *bo)@@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object *bo,if (bo->bulk_move == bulk)return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); bo->bulk_move = bulk;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_set_bulk_move);@@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo)* reference it any more. The only tricky case is the trylock on * the resv object while holding the lru_lock. */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); bo->base.resv = &bo->base._resv; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }return r;@@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP, interruptible,@@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,else if (lret ==0 )return -EBUSY ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (unlock_resv && !dma_resv_trylock (bo->base.resv)) {/* * We raced, and lost, someone else holds the reservation now, @@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,* delayed destruction would succeed, so just return success * here. */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return 0 ; } ret =0 ;@@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,if (ret) {if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return ret; } -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);ttm_bo_cleanup_memtype_use (bo);if (unlock_resv)@@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)ttm_bo_flush_all_fences (bo); bo->deleted = true; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);/* * Make pinned bos immediately available to @@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref) }kref_init (&bo->kref); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);INIT_WORK (&bo->delayed_delete, ttm_bo_delayed_delete);queue_work (bdev->wq, &bo->delayed_delete);@@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,bool locked = false;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);ttm_resource_manager_for_each_res (man, &cursor, res) {bool busy;@@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,if (!bo) {if (busy_bo && !ttm_bo_get_unless_zero (busy_bo)) busy_bo =NULL ; -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_mem_evict_wait_busy (busy_bo, ctx, ticket);if (busy_bo)ttm_bo_put (busy_bo);@@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,return ret; } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_bo_evict (bo, ctx);if (locked)@@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo) {dma_resv_assert_held (bo->base.resv);WARN_ON_ONCE (!kref_read (&bo->kref)); -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); ++bo->pin_count; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_pin);@@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)if (WARN_ON_ONCE (!bo->pin_count))return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); --bo->pin_count;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_unpin);@@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, }/* TODO: Cleanup the locking */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);/* * Move to system cached drivers/gpu/drm/ttm/ttm_device.c +9
-9@@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,unsigned i;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)@@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,return ret; } } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);return 0 ; }EXPORT_SYMBOL (ttm_device_swapout);@@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager; -spin_lock_init (&bdev->lru_lock); + bdev->lru_lock = &drm->lru_lock;INIT_LIST_HEAD (&bdev->pinned); bdev->dev_mapping = mapping;mutex_lock (&ttm_global_mutex);@@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)drain_workqueue (bdev->wq);destroy_workqueue (bdev->wq); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i)if (list_empty (&man->lru[0 ]))pr_debug ("Swap list %d was clean\n" , i); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);ttm_pool_fini (&bdev->pool);ttm_global_release ();@@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, {struct ttm_resource *res; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);while ((res =list_first_entry_or_null (list,typeof (*res), lru))) {struct ttm_buffer_object *bo = res->bo;@@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev,continue ;list_del_init (&res->lru); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);if (bo->ttm)ttm_tt_unpopulate (bo->bdev, bo->ttm);ttm_bo_put (bo); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock); } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); }void ttm_device_clear_dma_mappings (struct ttm_device *bdev)drivers/gpu/drm/ttm/ttm_resource.c +21
-21@@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init);* @bulk: bulk move structure * * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that -* resource order never changes. Should be called with &ttm_device.lru_lock held. +* resource order never changes. Should be called with &drm_device.lru_lock held. */ void ttm_lru_bulk_move_tail (struct ttm_lru_bulk_move *bulk) {@@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)if (!pos->first)continue ; -lockdep_assert_held (&pos->first->bo->bdev->lru_lock); +lockdep_assert_held (pos->first->bo->bdev->lru_lock);dma_resv_assert_held (pos->first->bo->base.resv);dma_resv_assert_held (pos->last->bo->base.resv);@@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource *res)struct ttm_buffer_object *bo = res->bo;struct ttm_device *bdev = bo->bdev; -lockdep_assert_held (&bo->bdev->lru_lock); +lockdep_assert_held (bo->bdev->lru_lock);if (bo->pin_count) {list_move_tail (&res->lru, &bdev->pinned);@@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo, res->bo = bo; man =ttm_manager_type (bo->bdev, place->mem_type); -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->pin_count)list_add_tail (&res->lru, &bo->bdev->pinned);else list_add_tail (&res->lru, &man->lru[bo->priority]); man->usage += res->size; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_resource_init);@@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager *man, {struct ttm_device *bdev = man->bdev; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);list_del_init (&res->lru); man->usage -= res->size; -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); }EXPORT_SYMBOL (ttm_resource_fini);@@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo,if (ret)return ret; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_resource_add_bulk_move (*res_ptr, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return 0 ; }@@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo, struct ttm_resource **res)if (!*res)return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_resource_del_bulk_move (*res, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); man =ttm_manager_type (bo->bdev, (*res)->mem_type); man->func->free (man, *res); *res =NULL ;@@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res,void ttm_resource_set_bo (struct ttm_resource *res,struct ttm_buffer_object *bo) { -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); res->bo = bo; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }/** @@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,* Can't use standard list traversal since we're unlocking. */ -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) {while (!list_empty (&man->lru[i])) { -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_mem_evict_first (bdev, man,NULL , &ctx,NULL );if (ret)return ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock); } } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);spin_lock (&man->move_lock); fence =dma_fence_get (man->move);@@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man) {uint64_t usage; -spin_lock (&man->bdev->lru_lock); +spin_lock (man->bdev->lru_lock); usage = man->usage; -spin_unlock (&man->bdev->lru_lock); +spin_unlock (man->bdev->lru_lock);return usage; }EXPORT_SYMBOL (ttm_resource_manager_usage);@@ -502,7 +502,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man, {struct ttm_resource *res; -lockdep_assert_held (&man->bdev->lru_lock); +lockdep_assert_held (man->bdev->lru_lock);for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority)@@ -526,7 +526,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,struct ttm_resource_cursor *cursor, struct ttm_resource *res) { - lockdep_assert_held(&man->bdev->lru_lock); + lockdep_assert_held(man->bdev->lru_lock);list_for_each_entry_continue (res, &man->lru[cursor->priority], lru)return res;drivers/gpu/drm/xe/xe_bo.c +2
-2@@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct ttm_buffer_object *ttm_bo)* the ttm_bo refcount is zero at this point. So trylocking *should* * always succeed here, as long as we hold the lru lock. */ -spin_lock (&ttm_bo->bdev->lru_lock); +spin_lock (ttm_bo->bdev->lru_lock); locked =dma_resv_trylock (ttm_bo->base.resv); -spin_unlock (&ttm_bo->bdev->lru_lock); +spin_unlock (ttm_bo->bdev->lru_lock);XE_WARN_ON (!locked);return locked;drivers/gpu/drm/xe/xe_exec.c +2
-2@@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)xe_vm_reactivate_rebind (vm);if (!err && !xe_vm_no_dma_fences (vm)) { -spin_lock (&xe->ttm.lru_lock); +spin_lock (xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&xe->ttm.lru_lock); +spin_unlock (xe->ttm.lru_lock); } err_repin:drivers/gpu/drm/xe/xe_vm.c +2
-2@@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct *w)#undef retry_required -spin_lock (&vm->xe->ttm.lru_lock); +spin_lock (vm->xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&vm->xe->ttm.lru_lock); +spin_unlock (vm->xe->ttm.lru_lock);/* Point of no return. */ arm_preempt_fences (vm, &preempt_fences);include/drm/drm_device.h +5
-0@@ -326,6 +326,11 @@ struct drm_device {*/ struct list_head debugfs_list; +/** +* @lru_lock: Protection for the per manager LRU and destroy lists. +*/ +spinlock_t lru_lock; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )include/drm/ttm/ttm_bo.h +2
-2@@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo);static inline void ttm_bo_move_to_lru_tail_unlocked (struct ttm_buffer_object *bo) { -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); } static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,include/drm/ttm/ttm_device.h +2
-2@@ -248,9 +248,9 @@ struct ttm_device {struct ttm_pool pool;/** -* @lru_lock: Protection for the per manager LRU and ddestroy lists. +* @lru_lock: Weak reference to drm_device::lru_lock. */ -spinlock_t lru_lock; +spinlock_t *lru_lock;/** * @pinned: Buffer objects which are pinned and so not on any LRU list.
RE: [RFC 02/11] drm: move lru_lock from ttm_device to drm_device
Quoted 19 lines
-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, November 2, 2023 8:53 AM
To: Zeng, Oak <[email protected]>; [email protected]; intel- [email protected]
Cc: [email protected]; [email protected]; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [RFC 02/11] drm: movelru_lockfromttm_devicetodrm_deviceAm 02.11.23 um 05:32 schrieb Oak Zeng:
Quoted 7 lines
In the coming patches, we will share the lru list b/t ttm bo based memory allocator and hmm/svm based memory allocator. Thus
lru_lock(which is used mainly to protect the lru list) is moved from structttm_deviceto structdrm_device, so this lock can be shared b/t those two memory allocators.
Quoted 6 lines
Quoted 3 lines
To minimize code change, struct
ttm_devicestill hold a weak reference oflru_lock, so ttm layer can still reference to this lock easily.I would rather like to see
drm_deviceto become the base class ofttm_device.
Yah...so drm_dev is the base of ttm_device, and ttm_device is the base of amdgpu_device or xe_device...
Quoted 2 lines
Similar to how
drm_gem_objectis the base class ofttm_buffer_object.
And ttm_buffer_object is base of amdgpu_bo
Pretty uniformed structure 😊
Quoted 4 lines
That is probably a bit more work, but would also eliminate some of the duplicate house keeping we currently have (e.g. bdev pointer in
ttm_buffer_objectetc...).
Right, if we do that, we can cast a ttm buffer object to amdgpu_bo, then get amdgpu_device of this amdgpu_bo, then get the ttm_device. Need to write a function to get this. Yes, this way we only need to keep a amdgpu_device pointer in amdgpu_bo. Cleaner than keep bdev pointer in tbo.
Quoted 2 lines
Moving then stuff from the
ttm_deviceinto thedrm_devicebecomes trivial.
Agree.
Oak
Quoted 604 lines
Regards,
Christian.Quoted 18 lines
Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 4 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 4 +- drivers/gpu/drm/ttm/ttm_bo.c | 40 +++++++++---------- drivers/gpu/drm/ttm/ttm_device.c | 18 ++++----- drivers/gpu/drm/ttm/ttm_resource.c | 42 ++++++++++---------- drivers/gpu/drm/xe/xe_bo.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 4 +- drivers/gpu/drm/xe/xe_vm.c | 4 +- include/drm/drm_device.h | 5 +++ include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_device.h | 4 +- 13 files changed, 72 insertions(+), 66 deletions(-)Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.cdrivers/gpu/drm/amd/amdgpu/amdgpu_vm.c int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct
Quoted 4 lines
index
f5daadcec865..747bcad86d5d100644drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +0
-0@@ -368,9 +368,9 @@ int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct
drm_exec*exec,Review
Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2
-2void amdgpu_vm_move_to_lru_tail (struct amdgpu_device *adev,struct amdgpu_vm *vm) { -spin_lock (&adev->mman.bdev.lru_lock); +spin_lock (adev->mman.bdev.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&adev->mman.bdev.lru_lock); +spin_unlock (adev->mman.bdev.lru_lock); }/* Create scheduler entities for page table updates */ (quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.cdrivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c static void amdgpu_vram_mgr_do_reserve(struct
Quoted 4 lines
index
c7085a747b03..b83e1741905e100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0@@ -290,9 +290,9 @@ static void amdgpu_vram_mgr_do_reserve(struct
ttm_resource_manager*man)Review
Quoted 24 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +2
-2vis_usage =amdgpu_vram_mgr_vis_size (adev, block);atomic64_add (vis_usage, &mgr->vis_usage); -spin_lock (&man->bdev->lru_lock); +spin_lock (man->bdev->lru_lock); man->usage += rsv->size; -spin_unlock (&man->bdev->lru_lock); +spin_unlock (man->bdev->lru_lock);list_move (&rsv->blocks, &mgr->reserved_pages); } }drivers/gpu/drm/drm_drv.c +1
-0@@ -623,6 +623,7 @@ static int drm_dev_init(struct drm_device *dev,INIT_LIST_HEAD (&dev->managed.resources);spin_lock_init (&dev->managed.lock); +spin_lock_init (&dev->lru_lock);/* no per-device feature limits by default */ dev->driver_features = ~0u ;(quoted diff) +0
-0
b/drivers/gpu/drm/i915/gem/i915_gem_ttm.cdrivers/gpu/drm/i915/gem/i915_gem_ttm.c void i915_ttm_adjust_lru(struct drm_i915_gem_object
Quoted 4 lines
index
9227f8146a58..c46f54f83f54100644drivers/gpu/drm/i915/gem/i915_gem_ttm.c +0
-0@@ -984,7 +984,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object*obj)
drivers/gpu/drm/i915/gem/i915_gem_ttm.c void i915_ttm_adjust_lru(struct drm_i915_gem_object
Quoted 9 lines
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1
-1/* * Put on the correct LRU list depending on the MADV status */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (shrinkable) {/* Try to keep shmem_tt from being considered for shrinking. */ bo->priority =TTM_MAX_BO_PRIORITY -1 ;@@ -1013,7 +1013,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object*obj)
drivers/gpu/drm/ttm/ttm_bo.c static void ttm_bo_mem_space_debug(struct
Quoted 13 lines
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1
-1}ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }/* drivers/gpu/drm/ttm/ttm_bo.c +0
-0@@ -68,7 +68,7 @@ static void ttm_bo_mem_space_debug(struct
ttm_buffer_object*bo,drivers/gpu/drm/ttm/ttm_bo.c void ttm_bo_set_bulk_move(struct ttm_buffer_object
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1* @bo: The buffer object. * * Move this BO to the tail of all lru lists used to lookup and reserve an -* object. This function must be called with struct ttm_global::lru_lock +* object. This function must be called with struct drm_device::lru_lock * held, and is used to make a BO less likely to be considered for eviction. */ void ttm_bo_move_to_lru_tail (struct ttm_buffer_object *bo)@@ -102,13 +102,13 @@ void ttm_bo_set_bulk_move(struct ttm_buffer_object*bo,
drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_individualize_resv(struct
Quoted 16 lines
drivers/gpu/drm/ttm/ttm_bo.c +2
-2if (bo->bulk_move == bulk)return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); bo->bulk_move = bulk;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_set_bulk_move);@@ -202,9 +202,9 @@ static int ttm_bo_individualize_resv(struct
ttm_buffer_object*bo)drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_cleanup_refs(struct ttm_buffer_object
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_bo.c +2
-2* reference it any more. The only tricky case is the trylock on * the resv object while holding the lru_lock. */ -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); bo->base.resv = &bo->base._resv; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }return r;@@ -255,7 +255,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object*bo,
drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_cleanup_refs(struct ttm_buffer_object
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); lret = dma_resv_wait_timeout(resv, DMA_RESV_USAGE_BOOKKEEP, interruptible,@@ -266,7 +266,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object*bo,
drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
Quoted 8 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1else if (lret ==0 )return -EBUSY ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (unlock_resv && !dma_resv_trylock (bo->base.resv)) {/* * We raced, and lost, someone else holds the reservation now,
drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_cleanup_refs(struct ttm_buffer_object
Quoted 1 line
drivers/gpu/drm/ttm/ttm_bo.c +0
-0@@ -276,7 +276,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object*bo,
drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_cleanup_refs(struct ttm_buffer_object
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1* delayed destruction would succeed, so just return success * here. */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return 0 ; } ret =0 ;@@ -285,11 +285,11 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object*bo,
drivers/gpu/drm/ttm/ttm_bo.c int ttm_bo_swapout(struct ttm_buffer_object *bo,
Quoted 87 lines
drivers/gpu/drm/ttm/ttm_bo.c +11
-11if (ret) {if (unlock_resv)dma_resv_unlock (bo->base.resv); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return ret; } -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);ttm_bo_cleanup_memtype_use (bo);if (unlock_resv)@@ -351,7 +351,7 @@ static void ttm_bo_release(struct kref *kref)ttm_bo_flush_all_fences (bo); bo->deleted = true; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);/* * Make pinned bos immediately available to @@ -367,7 +367,7 @@ static void ttm_bo_release(struct kref *kref) }kref_init (&bo->kref); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);INIT_WORK (&bo->delayed_delete, ttm_bo_delayed_delete);queue_work (bdev->wq, &bo->delayed_delete);@@ -598,7 +598,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,bool locked = false;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);ttm_resource_manager_for_each_res (man, &cursor, res) {bool busy;@@ -621,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,if (!bo) {if (busy_bo && !ttm_bo_get_unless_zero (busy_bo)) busy_bo =NULL ; -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_mem_evict_wait_busy (busy_bo, ctx, ticket);if (busy_bo)ttm_bo_put (busy_bo);@@ -635,7 +635,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev,return ret; } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_bo_evict (bo, ctx);if (locked)@@ -658,11 +658,11 @@ void ttm_bo_pin(struct ttm_buffer_object *bo) {dma_resv_assert_held (bo->base.resv);WARN_ON_ONCE (!kref_read (&bo->kref)); -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->resource)ttm_resource_del_bulk_move (bo->resource, bo); ++bo->pin_count; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_pin);@@ -679,11 +679,11 @@ void ttm_bo_unpin(struct ttm_buffer_object *bo)if (WARN_ON_ONCE (!bo->pin_count))return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); --bo->pin_count;if (bo->resource)ttm_resource_add_bulk_move (bo->resource, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_bo_unpin);@@ -1156,7 +1156,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo,struct
ttm_operation_ctx*ctx,Review
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1}/* TODO: Cleanup the locking */ -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);/* * Move to system cached (quoted diff) +0
-0
b/drivers/gpu/drm/ttm/ttm_device.cdrivers/gpu/drm/ttm/ttm_device.c int ttm_device_swapout(struct ttm_device *bdev, struct
Quoted 4 lines
index
12014788b595..d18eca86ebd6100644drivers/gpu/drm/ttm/ttm_device.c +0
-0@@ -147,7 +147,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct
ttm_operation_ctx*ctx,drivers/gpu/drm/ttm/ttm_device.c int ttm_device_swapout(struct ttm_device *bdev, struct
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_device.c +1
-1unsigned i;int ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)@@ -169,7 +169,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct
ttm_operation_ctx*ctx,drivers/gpu/drm/ttm/ttm_device.c int ttm_device_init(struct ttm_device *bdev, const struct
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_device.c +1
-1return ret; } } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);return 0 ; }EXPORT_SYMBOL (ttm_device_swapout);@@ -217,7 +217,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct
ttm_device_funcs*funcdrivers/gpu/drm/ttm/ttm_device.c static void ttm_device_clear_lru_dma_mappings(struct
Quoted 23 lines
drivers/gpu/drm/ttm/ttm_device.c +3
-3use_dma_alloc, use_dma32); bdev->vma_manager = vma_manager; -spin_lock_init (&bdev->lru_lock); + bdev->lru_lock = &drm->lru_lock;INIT_LIST_HEAD (&bdev->pinned); bdev->dev_mapping = mapping;mutex_lock (&ttm_global_mutex);@@ -244,11 +244,11 @@ void ttm_device_fini(struct ttm_device *bdev)drain_workqueue (bdev->wq);destroy_workqueue (bdev->wq); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i)if (list_empty (&man->lru[0 ]))pr_debug ("Swap list %d was clean\n" , i); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);ttm_pool_fini (&bdev->pool);ttm_global_release ();@@ -260,7 +260,7 @@ static void ttm_device_clear_lru_dma_mappings(struct
ttm_device*bdev,drivers/gpu/drm/ttm/ttm_device.c static void ttm_device_clear_lru_dma_mappings(struct
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_device.c +1
-1{struct ttm_resource *res; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);while ((res =list_first_entry_or_null (list,typeof (*res), lru))) {struct ttm_buffer_object *bo = res->bo;@@ -269,15 +269,15 @@ static void ttm_device_clear_lru_dma_mappings(struct
ttm_device*bdev,Review
Quoted 19 lines
drivers/gpu/drm/ttm/ttm_device.c +3
-3continue ;list_del_init (&res->lru); -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);if (bo->ttm)ttm_tt_unpopulate (bo->bdev, bo->ttm);ttm_bo_put (bo); -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock); } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); }void ttm_device_clear_dma_mappings (struct ttm_device *bdev)(quoted diff) +0
-0
b/drivers/gpu/drm/ttm/ttm_resource.cdrivers/gpu/drm/ttm/ttm_resource.c void ttm_resource_move_to_lru_tail(struct ttm_resource
Quoted 22 lines
index
46ff9c75bb12..6ada77f51fba100644drivers/gpu/drm/ttm/ttm_resource.c +2
-2@@ -48,7 +48,7 @@ EXPORT_SYMBOL(ttm_lru_bulk_move_init);* @bulk: bulk move structure * * Bulk move BOs to the LRU tail, only valid to use when driver makes sure that -* resource order never changes. Should be called with &ttm_device.lru_lock held. +* resource order never changes. Should be called with &drm_device.lru_lock held. */ void ttm_lru_bulk_move_tail (struct ttm_lru_bulk_move *bulk) {@@ -62,7 +62,7 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk)if (!pos->first)continue ; -lockdep_assert_held (&pos->first->bo->bdev->lru_lock); +lockdep_assert_held (pos->first->bo->bdev->lru_lock);dma_resv_assert_held (pos->first->bo->base.resv);dma_resv_assert_held (pos->last->bo->base.resv);@@ -148,7 +148,7 @@ void ttm_resource_move_to_lru_tail(struct ttm_resource*res)
drivers/gpu/drm/ttm/ttm_resource.c void ttm_resource_fini(struct ttm_resource_manager
Quoted 25 lines
drivers/gpu/drm/ttm/ttm_resource.c +3
-3struct ttm_buffer_object *bo = res->bo;struct ttm_device *bdev = bo->bdev; -lockdep_assert_held (&bo->bdev->lru_lock); +lockdep_assert_held (bo->bdev->lru_lock);if (bo->pin_count) {list_move_tail (&res->lru, &bdev->pinned);@@ -191,13 +191,13 @@ void ttm_resource_init(struct ttm_buffer_object *bo, res->bo = bo; man =ttm_manager_type (bo->bdev, place->mem_type); -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);if (bo->pin_count)list_add_tail (&res->lru, &bo->bdev->pinned);else list_add_tail (&res->lru, &man->lru[bo->priority]); man->usage += res->size; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }EXPORT_SYMBOL (ttm_resource_init);@@ -216,10 +216,10 @@ void ttm_resource_fini(struct ttm_resource_manager*man,
drivers/gpu/drm/ttm/ttm_resource.c void ttm_resource_free(struct ttm_buffer_object *bo,
Quoted 25 lines
drivers/gpu/drm/ttm/ttm_resource.c +4
-4{struct ttm_device *bdev = man->bdev; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);list_del_init (&res->lru); man->usage -= res->size; -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); }EXPORT_SYMBOL (ttm_resource_fini);@@ -235,9 +235,9 @@ int ttm_resource_alloc(struct ttm_buffer_object *bo,if (ret)return ret; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_resource_add_bulk_move (*res_ptr, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock);return 0 ; }@@ -248,9 +248,9 @@ void ttm_resource_free(struct ttm_buffer_object *bo,struct
ttm_resource**res)drivers/gpu/drm/ttm/ttm_resource.c int ttm_resource_manager_evict_all(struct ttm_device
Quoted 24 lines
drivers/gpu/drm/ttm/ttm_resource.c +4
-4if (!*res)return ; -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_resource_del_bulk_move (*res, bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); man =ttm_manager_type (bo->bdev, (*res)->mem_type); man->func->free (man, *res); *res =NULL ;@@ -368,9 +368,9 @@ bool ttm_resource_compat(struct ttm_resource *res,void ttm_resource_set_bo (struct ttm_resource *res,struct ttm_buffer_object *bo) { -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock); res->bo = bo; -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); }/** @@ -424,18 +424,18 @@ int ttm_resource_manager_evict_all(struct ttm_device*bdev,
drivers/gpu/drm/ttm/ttm_resource.c uint64_t ttm_resource_manager_usage(struct
Quoted 23 lines
drivers/gpu/drm/ttm/ttm_resource.c +4
-4* Can't use standard list traversal since we're unlocking. */ -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock);for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) {while (!list_empty (&man->lru[i])) { -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock); ret =ttm_mem_evict_first (bdev, man,NULL , &ctx,NULL );if (ret)return ret; -spin_lock (&bdev->lru_lock); +spin_lock (bdev->lru_lock); } } -spin_unlock (&bdev->lru_lock); +spin_unlock (bdev->lru_lock);spin_lock (&man->move_lock); fence =dma_fence_get (man->move);@@ -463,9 +463,9 @@ uint64_t ttm_resource_manager_usage(struct
ttm_resource_manager*man)drivers/gpu/drm/ttm/ttm_resource.c ttm_resource_manager_first(struct
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_resource.c +2
-2{uint64_t usage; -spin_lock (&man->bdev->lru_lock); +spin_lock (man->bdev->lru_lock); usage = man->usage; -spin_unlock (&man->bdev->lru_lock); +spin_unlock (man->bdev->lru_lock);return usage; }EXPORT_SYMBOL (ttm_resource_manager_usage);@@ -502,7 +502,7 @@ ttm_resource_manager_first(struct
ttm_resource_manager*man,drivers/gpu/drm/ttm/ttm_resource.c ttm_resource_manager_next(struct
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_resource.c +1
-1{struct ttm_resource *res; -lockdep_assert_held (&man->bdev->lru_lock); +lockdep_assert_held (man->bdev->lru_lock);for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; ++cursor->priority)@@ -526,7 +526,7 @@ ttm_resource_manager_next(struct
ttm_resource_manager*man,drivers/gpu/drm/xe/xe_bo.c static bool xe_ttm_bo_lock_in_destructor(struct
Quoted 13 lines
drivers/gpu/drm/ttm/ttm_resource.c +1
-1struct ttm_resource_cursor *cursor, struct ttm_resource *res) { - lockdep_assert_held(&man->bdev->lru_lock); + lockdep_assert_held(man->bdev->lru_lock);list_for_each_entry_continue (res, &man->lru[cursor->priority], lru)return res;drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -946,9 +946,9 @@ static bool xe_ttm_bo_lock_in_destructor(struct
ttm_buffer_object*ttm_bo)drivers/gpu/drm/xe/xe_exec.c int xe_exec_ioctl(struct drm_device *dev, void *data,
Quoted 16 lines
drivers/gpu/drm/xe/xe_bo.c +2
-2* the ttm_bo refcount is zero at this point. So trylocking *should* * always succeed here, as long as we hold the lru lock. */ -spin_lock (&ttm_bo->bdev->lru_lock); +spin_lock (ttm_bo->bdev->lru_lock); locked =dma_resv_trylock (ttm_bo->base.resv); -spin_unlock (&ttm_bo->bdev->lru_lock); +spin_unlock (ttm_bo->bdev->lru_lock);XE_WARN_ON (!locked);return locked;drivers/gpu/drm/xe/xe_exec.c +0
-0@@ -370,9 +370,9 @@ int xe_exec_ioctl(struct drm_device *dev, void *data,struct
drm_file*file)drivers/gpu/drm/xe/xe_vm.c static void preempt_rebind_work_func(struct work_struct
Quoted 16 lines
drivers/gpu/drm/xe/xe_exec.c +2
-2xe_vm_reactivate_rebind (vm);if (!err && !xe_vm_no_dma_fences (vm)) { -spin_lock (&xe->ttm.lru_lock); +spin_lock (xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&xe->ttm.lru_lock); +spin_unlock (xe->ttm.lru_lock); } err_repin:drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -651,9 +651,9 @@ static void preempt_rebind_work_func(struct work_struct*w)
include/drm/ttm/ttm_bo.h void ttm_bo_move_to_lru_tail(struct ttm_buffer_object
Quoted 32 lines
drivers/gpu/drm/xe/xe_vm.c +2
-2#undef retry_required -spin_lock (&vm->xe->ttm.lru_lock); +spin_lock (vm->xe->ttm.lru_lock);ttm_lru_bulk_move_tail (&vm->lru_bulk_move); -spin_unlock (&vm->xe->ttm.lru_lock); +spin_unlock (vm->xe->ttm.lru_lock);/* Point of no return. */ arm_preempt_fences (vm, &preempt_fences);include/drm/drm_device.h +5
-0@@ -326,6 +326,11 @@ struct drm_device {*/ struct list_head debugfs_list; +/** +* @lru_lock: Protection for the per manager LRU and destroy lists. +*/ +spinlock_t lru_lock; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )include/drm/ttm/ttm_bo.h +0
-0@@ -290,9 +290,9 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object*bo);
Quoted 27 lines
include/drm/ttm/ttm_bo.h +2
-2static inline void ttm_bo_move_to_lru_tail_unlocked (struct ttm_buffer_object *bo) { -spin_lock (&bo->bdev->lru_lock); +spin_lock (bo->bdev->lru_lock);ttm_bo_move_to_lru_tail (bo); -spin_unlock (&bo->bdev->lru_lock); +spin_unlock (bo->bdev->lru_lock); } static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo,include/drm/ttm/ttm_device.h +2
-2@@ -248,9 +248,9 @@ struct ttm_device {struct ttm_pool pool;/** -* @lru_lock: Protection for the per manager LRU and ddestroy lists. +* @lru_lock: Weak reference to drm_device::lru_lock. */ -spinlock_t lru_lock; +spinlock_t *lru_lock;/** * @pinned: Buffer objects which are pinned and so not on any LRU list.
RFC 3/11 drm: introduce drm evictable LRU
In reply to Oak Zengdrm LRU manager is introuced for resource eviction purpose. It maintains a LRU list per resource type. It provides functions to add or remove resource to or from the list. It also provides function to retrieve the first entity on the LRU list.
drm LRU manager also provides functions for bulk moving resources on the LRU lists.
drm LRU manager also does very basic memory accounting function, i.e., LRU manager keeps a size of this resource type and a usage member for how much of resource has been added to this LRU manager's LRU list. TTM resource manager memory accounting functoins such as struct ttm_resource_manager::size and struct ttm_resource_manger::usage are still kept. In the future, when SVM codes are in the picture, those memory accounting functions need some rework to consider the memory used by both TTM and SVM.
For one device, a global drm LRU manager per resource type should be created/initialized at device initialization time. Drm LRU manager instances are embedded in struct drm_device.
It is pretty much moving some of the ttm resource manager functions to the drm layer. The reason of this code refactory is, we want to create a single LRU list for memory allocated from BO(buffer object) based driver and hmm/svm(shared virtual memory) based driver, thus BO driver and svm driver can evict memory from each other.
Previously the LRU list in TTM resource manager (lru field in struct ttm_reource_manager) is coupled with ttm_buffer_object concept, i.e., each ttm resource is backed by a ttm_buffer_object and the LRU list is essentially a list of ttm_buffer_object. Due to this behavior, the TTM resource manager can't be used by hmm/svm driver as we don't plan to have the BO concept for the hmm/svm implemenation. So we decouple the evictable LRU list from the BO concept in this series.
The design goal of drm lru manager is to make it as lean as possible. So each lru entity only has a list node member used to link this entity to the evictable LRU list, and the basic resource size/type/priority of this entity. It doesn't have any driver specify information. A lru entity also has a function pointer of evict function. This is used to implement ttm or svm specific eviction function. A lru entity is supposed to be embedded in a driver specific structure such as struct ttm_resource, see the usage in the next patch of this series.
The ttm resource manager, and some of the ttm_bo functions such as ttm_mem_evict_first will be rewriten using the new drm lru manager library, see the next patch in this series.
The future hmm/svm implemenation will call lru manager function to add hmm/svm allocations to the shared evictable lru list.
Lock design: previously ttm_resource LRU list is protected by a device global ttm_device::lru_lock (bdev->lru_lock in codes). This lock also protects ttm_buffer_object::pin_count, ttm_resource_manager::usage, ttm_resource::bo, ttm_device::pinned list etc. With this refactory, lru_lock is moved out of ttm_device and is added to struct drm_deive, so it can be shared b/t ttm code and svm code.
--- drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++ include/drm/drm_device.h | 7 + include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++ 4 files changed, 428 insertions(+) create mode 100644 drivers/gpu/drm/drm_evictable_lru.c create mode 100644 include/drm/drm_evictable_lru.h
drivers/gpu/drm/Makefile +1 -0
@@ -46,6 +46,7 @@ drm-y := \ drm_vblank_work.o \ drm_vma_manager.o \ drm_gpuva_mgr.o \ + drm_evictable_lru.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \
drivers/gpu/drm/drm_evictable_lru.c new file +232 -0
232 lines not on this page · raw message
include/drm/drm_device.h +7 -0
@@ -8,6 +8,7 @@#include <drm/drm_legacy.h> #include <drm/drm_mode_config.h> +#include <drm/drm_evictable_lru.h> struct drm_driver ;struct drm_minor ;
@@ -331,6 +332,12 @@ struct drm_device {*/ spinlock_t lru_lock; +/** +* @lru_mgr: Device global lru managers per memory type or memory +* region. Each lru manager manages a lru list of this memory type. +*/ +struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES ]; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )
include/drm/drm_evictable_lru.h new file +188 -0
188 lines not on this page · raw message
Signature · 1 line
2.26.3
Christian König
maintainerRe: [RFC 03/11] drm: introduce drm evictable LRU
Am 02.11.23 um 05:32 schrieb Oak Zeng:
Quoted 2 lines
drm LRU manager is introuced for resource eviction purpose. It maintains a LRU list per resource type.
Shouldn't we first add the possible resource types in a separate patch?
Quoted 3 lines
It provides functions to add or remove
resource to or from the list. It also provides function to retrieve the first entity on the LRU list.
+ functions to iterate over them.
Quoted 4 lines
drm LRU manager also provides functions for bulk moving resources on the LRU lists.
Quoted 8 lines
drm LRU manager also does very basic memory accounting function, i.e., LRU manager keeps a size of this resource type and a usage member for how much of resource has been added to this LRU manager's LRU list. TTM resource manager memory accounting functoins such as struct
ttm_resource_manager::sizeand structttm_resource_manger::usageare still kept. In the future, when SVM codes are in the picture, those memory accounting functions need some rework to consider the memory used by both TTM and SVM.
Please keep in mind that this structure needs to extremely small to be usable for SVM. E.g. struct page size small :)
At least HMM based implementations ideally wants to have one for each page or something like that.
Quoted 10 lines
For one device, a global drm LRU manager per resource type should be created/initialized at device initialization time. Drm LRU manager instances are embedded in struct
drm_device.It is pretty much moving some of the ttm resource manager functions to the drm layer. The reason of this code refactory is, we want to create a single LRU list for memory allocated from BO(buffer object) based driver and hmm/svm(shared virtual memory) based driver, thus BO driver and svm driver can evict memory from each other.
Quoted 4 lines
Previously the LRU list in TTM resource manager (lru field in struct
ttm_reource_manager) is coupled withttm_buffer_objectconcept, i.e., each ttm resource is backed by attm_buffer_objectand the LRU list is essentially a list ofttm_buffer_object.
Actually it's the other way around. The resource provides the backing of the BO.
And when a BO moves around it can temporary be that multiple resource point to the same BO.
I also want to have a more advanced iterator at some point where we grab the BO lock for keeping a reference into the LRU list. Not sure how to do this if we don't have the BO here any more.
Need to think about that further,
Christian.
Quoted 507 lines
Due to this behavior, the
TTM resource manager can't be used by hmm/svm driver as we don't plan to have the BO concept for the hmm/svm implemenation. So we decouple the evictable LRU list from the BO concept in this series.The design goal of drm lru manager is to make it as lean as possible. So each lru entity only has a list node member used to link this entity to the evictable LRU list, and the basic resource size/type/priority of this entity. It doesn't have any driver specify information. A lru entity also has a function pointer of evict function. This is used to implement ttm or svm specific eviction function. A lru entity is supposed to be embedded in a driver specific structure such as struct
ttm_resource, see the usage in the next patch of this series.The ttm resource manager, and some of the
ttm_bofunctions such asttm_mem_evict_firstwill be rewriten using the new drm lru manager library, see the next patch in this series.The future hmm/svm implemenation will call lru manager function to add hmm/svm allocations to the shared evictable lru list.
Lock design: previously
ttm_resourceLRU list is protected by a device globalttm_device::lru_lock(bdev->lru_lockin codes). This lock also protectsttm_buffer_object::pin_count,ttm_resource_manager::usage,ttm_resource::bo,ttm_device::pinnedlist etc. With this refactory,lru_lockis moved out ofttm_deviceand is added to structdrm_deive, so it can be shared b/t ttm code and svm code.Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++ include/drm/drm_device.h | 7 + include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++ 4 files changed, 428 insertions(+)create mode 100644
drivers/gpu/drm/drm_evictable_lru.ccreate mode 100644include/drm/drm_evictable_lru.hdrivers/gpu/drm/Makefile +1
-0@@ -46,6 +46,7 @@ drm-y := \ drm_vblank_work.o \ drm_vma_manager.o \ drm_gpuva_mgr.o \ + drm_evictable_lru.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \drivers/gpu/drm/drm_evictable_lru.c new file +232
-0@@ -0,0 +1,232 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/lockdep.h> +#include <linux/container_of.h> +#include <drm/drm_evictable_lru.h> +#include <drm/drm_device.h> + +static inline struct drm_lru_mgr *entity_to_mgr (struct drm_lru_entity *entity) +{ +struct drm_lru_mgr *mgr; + + mgr = &entity->drm->lru_mgr[entity->mem_type]; +BUG_ON (!mgr->used); + +return mgr; +} + +void drm_lru_entity_init (struct drm_lru_entity *entity,struct drm_device *drm, +uint32_t mem_type,uint64_t size,uint32_t priority) +{ + entity->drm = drm; + entity->mem_type = mem_type; + entity->size = size; + entity->priority = priority; +INIT_LIST_HEAD (&entity->lru); +} + +/** +* drm_lru_mgr_init +* +* @mgr: drm lru manager to init +* @size: size of the resource managed by this manager +* @lock: pointer of the global lru_lock +* +* Initialize a drm lru manager +*/ +void drm_lru_mgr_init (struct drm_lru_mgr *mgr,uint64_t size,spinlock_t *lock) +{ +unsigned j; + + mgr->used = true; + mgr->size = size; + mgr->usage =0 ; + mgr->lru_lock = lock; + +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; j++) +INIT_LIST_HEAD (&mgr->lru[j]); +} + +void drm_lru_bulk_move_init (struct drm_lru_bulk_move *bulk_move) +{ +memset (bulk_move,0 ,sizeof (*bulk_move)); +} + +/** +* drm_lru_first +* +* @mgr: drm lru manager to iterate over +* @cursor: cursor of the current position +* +* Returns the first entity in drm lru manager +*/ +struct drm_lru_entity * +drm_lru_first (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor) +{ +struct drm_lru_entity *entity; + +lockdep_assert_held (mgr->lru_lock); + +for (cursor->priority =0 ; cursor->priority <DRM_MAX_LRU_PRIORITY ; ++cursor->priority) +list_for_each_entry (entity, &mgr->lru[cursor->priority], lru) +return entity; + +return NULL ; +} + +/** +* drm_lru_next +* +* @mgr: drm lru manager to iterate over +* @cursor: cursor of the current position +* @entity: the current lru entity pointer +* +* Returns the next entity from drm lru manager +*/ +struct drm_lru_entity * +drm_lru_next (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor, +struct drm_lru_entity *entity) +{ +lockdep_assert_held (mgr->lru_lock); + +list_for_each_entry_continue (entity, &mgr->lru[cursor->priority], lru) +return entity; + +for (++cursor->priority; cursor->priority <DRM_MAX_LRU_PRIORITY ; ++cursor->priority) +list_for_each_entry (entity, &mgr->lru[cursor->priority], lru) +return entity; + +return NULL ; +} + +/** +* drm_lru_move_to_tail +* +* @entity: the lru entity to move to lru tail +* +* Move a lru entity to lru tail +*/ +void drm_lru_move_to_tail (struct drm_lru_entity * entity) +{ +struct list_head *lru; +struct drm_lru_mgr *mgr; + + mgr =entity_to_mgr (entity); +lockdep_assert_held (mgr->lru_lock); + lru = &mgr->lru[entity->priority]; +list_move_tail (&entity->lru, lru); +} + +/** +* drm_lru_bulk_move_range_tail +* +* @range: bulk move range +* @entity: lru_entity to move +* +* Move a lru_entity to the tail of a bulk move range +*/ +void drm_lru_bulk_move_range_tail (struct drm_lru_bulk_move_range *range, +struct drm_lru_entity *entity) +{ +if (entity == range->last) +return ; + +if (entity == range->first) + range->first = container_of(entity->lru.next, struct drm_lru_entity, lru); + +if (range->last) +list_move (&entity->lru, &range->last->lru); + + range->last = entity; +} +EXPORT_SYMBOL (drm_lru_bulk_move_range_tail); + +/** +* drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail. +* +* @bulk: bulk_move structure +* +* Bulk move entities to the LRU tail, only valid to use when driver makes sure that +* resource order never changes. +*/ +void drm_lru_bulk_move_tail (struct drm_lru_bulk_move *bulk) +{ + +unsigned i, j; + +for (i =0 ; i <DRM_NUM_MEM_TYPES ; ++i) { +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; ++j) { +struct drm_lru_bulk_move_range *range = &bulk->range[i][j]; +struct drm_lru_mgr *mgr; + +if (!range->first) +continue ; + + mgr =entity_to_mgr (range->first); +lockdep_assert_held (mgr->lru_lock); +list_bulk_move_tail (&mgr->lru[range->first->priority], &range->first->lru, + &range->last->lru); + } + } +} +EXPORT_SYMBOL (drm_lru_bulk_move_tail); + +/** +* drm_lru_add_bulk_move +* +* @entity: the lru entity to add to the bulk move range +* @bulk_move: the bulk move ranges to add the entity +* +* Add a lru entity to the tail of a bulk move range +*/ +void drm_lru_add_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move) +{ +struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + +if (!range->first) { + range->first = entity; + range->last = entity; +return ; + } + +drm_lru_bulk_move_range_tail (range, entity); +} + +EXPORT_SYMBOL (drm_lru_add_bulk_move); +/** +* drm_lru_del_bulk_move +* +* @entity: the lru entity to move from the bulk move range +* @bulk_move: the bulk move ranges to move the entity out of +* +* Move a lru entity out of bulk move range. This doesn't +* delete entity from lru manager's lru list. +*/ +void drm_lru_del_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move) +{ +struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + +if (unlikely (WARN_ON (!range->first || !range->last) || + (range->first == entity && range->last == entity))) { + range->first =NULL ; + range->last =NULL ; + }else if (range->first == entity) { + range->first =container_of (entity->lru.next, + struct drm_lru_entity, lru); + }else if (range->last == entity) { + range->last =container_of (entity->lru.prev, + struct drm_lru_entity, lru); + }else { +list_move (&entity->lru, &range->last->lru); + } +} +EXPORT_SYMBOL (drm_lru_del_bulk_move);include/drm/drm_device.h +7
-0@@ -8,6 +8,7 @@#include <drm/drm_legacy.h> #include <drm/drm_mode_config.h> +#include <drm/drm_evictable_lru.h> struct drm_driver ;struct drm_minor ;@@ -331,6 +332,12 @@ struct drm_device {*/ spinlock_t lru_lock; +/** +* @lru_mgr: Device global lru managers per memory type or memory +* region. Each lru manager manages a lru list of this memory type. +*/ +struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES ]; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )include/drm/drm_evictable_lru.h new file +188
-0@@ -0,0 +1,188 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#ifndef _DRM_EVICTABLE_LRU_H_ +#define _DRM_EVICTABLE_LRU_H_ + +#include <linux/list.h> +#include <linux/spinlock_types.h> +#include <linux/spinlock.h> + +struct drm_device ; + +#define DRM_MAX_LRU_PRIORITY 4 +#define DRM_NUM_MEM_TYPES 8 + +/** +* struct drm_lru_entity +* +* @drm: drm device that this entity belongs to +* @mem_type: The memory type that this entity belongs to +* @size: resource size of this entity +* @priority: The priority of this entity +* @lru: least recent used list node, see &drm_lru_mgr.lru +* +* This structure represents an entity in drm_lru_mgr's +* list. This structure is supposed to be embedded in +* user's data structure. +*/ +struct drm_lru_entity { +struct drm_device *drm; +uint32_t mem_type; +uint64_t size; +uint32_t priority; +struct list_head lru; +}; + +/** +* struct drm_lru_mgr +* +* @used: whether this lru manager is used or not +* @size: size of the resource +* @usage: how much resource has been used +* @lru_lock: a weak reference to the global lru_lock +* @lru: least recent used list, per priority +* +* This structure maintains all the buffer allocations +* in a least recent used list, so a victim for eviction +* can be easily found. +*/ +struct drm_lru_mgr { +bool used; +uint64_t size; +uint64_t usage; +spinlock_t *lru_lock; +struct list_head lru[DRM_MAX_LRU_PRIORITY ]; +}; + +/** +* struct drm_lru_cursor +* +* @priority: the current priority +* +* Cursor to iterate over all entities in lru manager. +*/ +struct drm_lru_cursor { +unsigned priority; +}; + +/** +* struct drm_lru_bulk_move_range +* +* @first: the first entity in the range +* @last: the last entity in the range +* +* Range of entities on a lru list. +*/ +struct drm_lru_bulk_move_range +{ +struct drm_lru_entity *first; +struct drm_lru_entity *last; +}; + +/** +* struct drm_lru_bulk_move +* +* @range: An array of bulk move range, each corelates to the drm_lru_mgr's +* lru list of the same memory type and same priority. +* +* A collection of bulk_move range which can be used to move drm_lru_entity +* on the lru list in a bulk way. It should be initialized through +* drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call +* drm_lru_add_bulk_move/drm_lru_del_bulk_move. +*/ +struct drm_lru_bulk_move { +struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES ][DRM_MAX_LRU_PRIORITY ]; +}; + + + +/** +* drm_lru_add_entity +* +* @entity: the lru entity to add +* @mgr: the drm lru manager +* @priority: specify which priority list to add +* +* Add an entity to lru list +*/ +static inline void drm_lru_add_entity (struct drm_lru_entity *entity, +struct drm_lru_mgr *mgr,unsigned priority) +{ +lockdep_assert_held (mgr->lru_lock); +list_add_tail (&entity->lru, &mgr->lru[priority]); + mgr->usage += entity->size; +} + +/** +* drm_lru_remove_entity +* +* @entity: the lru entity to remove +* @mgr: the drm lru manager +* +* Remove an entity from lru list +*/ +static inline void drm_lru_remove_entity (struct drm_lru_entity *entity, +struct drm_lru_mgr *mgr) +{ +lockdep_assert_held (mgr->lru_lock); +list_del_init (&entity->lru); + mgr->usage -= entity->size; +} + +/** +* drm_lru_mgr_fini +* +* @mgr: the drm lru manager +* +* de-initialize a lru manager +*/ +static inline void drm_lru_mgr_fini (struct drm_lru_mgr *mgr) +{ + mgr->used = false; +} + +void drm_lru_entity_init (struct drm_lru_entity *entity,struct drm_device *drm, +uint32_t mem_type,uint64_t size,uint32_t priority); + +struct drm_lru_entity * +drm_lru_first (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor); + +struct drm_lru_entity * +drm_lru_next (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor, +struct drm_lru_entity *entity); + +void drm_lru_mgr_init (struct drm_lru_mgr *mgr,uint64_t size, +spinlock_t *lru_lock); + +void drm_lru_move_to_tail (struct drm_lru_entity * entity); + +void drm_lru_bulk_move_init (struct drm_lru_bulk_move *bulk_move); + + +void drm_lru_bulk_move_tail (struct drm_lru_bulk_move *bulk); + +void drm_lru_bulk_move_range_tail (struct drm_lru_bulk_move_range *range, +struct drm_lru_entity *entity); + +void drm_lru_add_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move); + +void drm_lru_del_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move); +/** +* drm_lru_for_each_entity +* +* @mgr: the drm lru manager +* @cursor: cursor for the current position +* @entity: the current drm_lru_entity +* +* Iterate over all entities in drm lru manager +*/ +#define drm_lru_for_each_entity (mgr, cursor, entity) \ + for (entity = drm_lru_first(mgr, cursor); entity; \ + entity = drm_lru_next(mgr, cursor, entity)) + +#endif
RE: [RFC 03/11] drm: introduce drm evictable LRU
Quoted 11 lines
-----Original Message-----
From: Christian König <[email protected]>
Sent: Thursday, November 2, 2023 9:24 AM
To: Zeng, Oak <[email protected]>; [email protected]; intel- [email protected]
Cc: [email protected]; [email protected]; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [RFC 03/11] drm: introduce drm evictable LRU
Quoted 5 lines
Am 02.11.23 um 05:32 schrieb Oak Zeng:
Quoted 2 lines
drm LRU manager is introuced for resource eviction purpose. It maintains a LRU list per resource type.
Shouldn't we first add the possible resource types in a separate patch?
Resource type in my description message is not a good name. for resource type we have:
System memory
Gpu stolen memory
Normal gpu vram
Some device such as Intel's pvc has sub-device concept (it is called tile in xe driver). For such device, we create multiple vram type ttm resource manager and multiple lru manager, one for each sub-device...So currently we only defined a DRM_NUM_MEM_TYPES in lru manager, but not defining each resource (memory) type.
Quoted 6 lines
Quoted 4 lines
It provides functions to add or remove
resource to or from the list. It also provides function to retrieve the first entity on the LRU list.+ functions to iterate over them.
Yes basic iterate functions are implemented in this patch. Will add it to description message.
Quoted 14 lines
Quoted 5 lines
drm LRU manager also provides functions for bulk moving resources on the LRU lists.
Quoted 8 lines
drm LRU manager also does very basic memory accounting function, i.e., LRU manager keeps a size of this resource type and a usage member for how much of resource has been added to this LRU manager's LRU list. TTM resource manager memory accounting functoins such as struct
ttm_resource_manager::sizeand structttm_resource_manger::usageare still kept. In the future, when SVM codes are in the picture, those memory accounting functions need some rework to consider the memory used by both TTM and SVM.
Quoted 5 lines
Please keep in mind that this structure needs to extremely small to be usable for SVM. E.g. struct page size small :)
At least HMM based implementations ideally wants to have one for each page or something like that.
Very good point. List node and eviction function pointer are necessary for drm_lru_entity. I will look whether we can remove other members. At least we can remove the drm_device pointer if we make drm_device base class of ttm_device as you suggested in previous patch comment.
And mem_type and priority can use bitfield, so a dword is enough.
Quoted 11 lines
Quoted 11 lines
For one device, a global drm LRU manager per resource type should be created/initialized at device initialization time. Drm LRU manager instances are embedded in struct
drm_device.It is pretty much moving some of the ttm resource manager functions to the drm layer. The reason of this code refactory is, we want to create a single LRU list for memory allocated from BO(buffer object) based driver and hmm/svm(shared virtual memory) based driver, thus BO driver and svm driver can evict memory from each other.
Quoted 7 lines
Quoted 4 lines
Previously the LRU list in TTM resource manager (lru field in struct
ttm_reource_manager) is coupled withttm_buffer_objectconcept, i.e., each ttm resource is backed by attm_buffer_objectand the LRU list is essentially a list ofttm_buffer_object.Actually it's the other way around. The resource provides the backing of the BO.
You are right. Will fix this description.
Quoted 9 lines
And when a BO moves around it can temporary be that multiple resource point to the same BO.
I also want to have a more advanced iterator at some point where we grab the BO lock for keeping a reference into the LRU list. Not sure how to do this if we don't have the BO here any more.
Need to think about that further,
Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class of ttm_resource or any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
Oak
Quoted 518 lines
Christian.
Review
Quoted 51 lines
Due to this behavior, the
TTM resource manager can't be used by hmm/svm driver as we don't plan to have the BO concept for the hmm/svm implemenation. So we decouple the evictable LRU list from the BO concept in this series.The design goal of drm lru manager is to make it as lean as possible. So each lru entity only has a list node member used to link this entity to the evictable LRU list, and the basic resource size/type/priority of this entity. It doesn't have any driver specify information. A lru entity also has a function pointer of evict function. This is used to implement ttm or svm specific eviction function. A lru entity is supposed to be embedded in a driver specific structure such as struct
ttm_resource, see the usage in the next patch of this series.The ttm resource manager, and some of the
ttm_bofunctions such asttm_mem_evict_firstwill be rewriten using the new drm lru manager library, see the next patch in this series.The future hmm/svm implemenation will call lru manager function to add hmm/svm allocations to the shared evictable lru list.
Lock design: previously
ttm_resourceLRU list is protected by a device globalttm_device::lru_lock(bdev->lru_lockin codes). This lock also protectsttm_buffer_object::pin_count,ttm_resource_manager::usage,ttm_resource::bo,ttm_device::pinnedlist etc. With this refactory,lru_lockis moved out ofttm_deviceand is added to structdrm_deive, so it can be shared b/t ttm code and svm code.Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_evictable_lru.c | 232 ++++++++++++++++++++++++++++ include/drm/drm_device.h | 7 + include/drm/drm_evictable_lru.h | 188 ++++++++++++++++++++++ 4 files changed, 428 insertions(+)create mode 100644
drivers/gpu/drm/drm_evictable_lru.ccreate mode 100644include/drm/drm_evictable_lru.hdrivers/gpu/drm/Makefile +1
-0@@ -46,6 +46,7 @@ drm-y := \ drm_vblank_work.o \ drm_vma_manager.o \ drm_gpuva_mgr.o \ + drm_evictable_lru.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \(quoted diff) +0
-0
b/drivers/gpu/drm/drm_evictable_lru.cQuoted 68 lines
new file mode 100644
index 000000000000..2ba9105cca03drivers/gpu/drm/drm_evictable_lru.c new file +63
-0@@ -0,0 +1,232 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/lockdep.h> +#include <linux/container_of.h> +#include <drm/drm_evictable_lru.h> +#include <drm/drm_device.h> + +static inline struct drm_lru_mgr *entity_to_mgr (struct drm_lru_entity *entity) +{ +struct drm_lru_mgr *mgr; + + mgr = &entity->drm->lru_mgr[entity->mem_type]; +BUG_ON (!mgr->used); + +return mgr; +} + +void drm_lru_entity_init (struct drm_lru_entity *entity,struct drm_device *drm, +uint32_t mem_type,uint64_t size,uint32_t priority) +{ + entity->drm = drm; + entity->mem_type = mem_type; + entity->size = size; + entity->priority = priority; +INIT_LIST_HEAD (&entity->lru); +} + +/** +* drm_lru_mgr_init +* +* @mgr: drm lru manager to init +* @size: size of the resource managed by this manager +* @lock: pointer of the global lru_lock +* +* Initialize a drm lru manager +*/ +void drm_lru_mgr_init (struct drm_lru_mgr *mgr,uint64_t size,spinlock_t *lock) +{ +unsigned j; + + mgr->used = true; + mgr->size = size; + mgr->usage =0 ; + mgr->lru_lock = lock; + +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; j++) +INIT_LIST_HEAD (&mgr->lru[j]); +} + +void drm_lru_bulk_move_init (struct drm_lru_bulk_move *bulk_move) +{ +memset (bulk_move,0 ,sizeof (*bulk_move)); +} + +/** +* drm_lru_first +* +* @mgr: drm lru manager to iterate over +* @cursor: cursor of the current position +* drivers/gpu/drm/drm_evictable_lru.c line 73
Quoted 10 lines
drivers/gpu/drm/drm_evictable_lru.c new file +10
-0+* Returns the first entity in drm lru manager +*/ +struct drm_lru_entity * +drm_lru_first (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor) +{ +struct drm_lru_entity *entity; + +lockdep_assert_held (mgr->lru_lock); + + for(cursor->priority = 0; cursor->priority < DRM_MAX_LRU_PRIORITY;++cursor->priority)
Quoted 15 lines
drivers/gpu/drm/drm_evictable_lru.c new file +15
-0+list_for_each_entry (entity, &mgr->lru[cursor->priority], lru) +return entity; + +return NULL ; +} + +/** +* drm_lru_next +* +* @mgr: drm lru manager to iterate over +* @cursor: cursor of the current position +* @entity: the current lru entity pointer +* +* Returns the next entity from drm lru manager +*/ drivers/gpu/drm/drm_evictable_lru.c
Quoted 10 lines
drivers/gpu/drm/drm_evictable_lru.c new file +10
-0+struct drm_lru_entity * +drm_lru_next (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor, +struct drm_lru_entity *entity) +{ +lockdep_assert_held (mgr->lru_lock); + +list_for_each_entry_continue (entity, &mgr->lru[cursor->priority], lru) +return entity; + + for(++cursor->priority; cursor->priority < DRM_MAX_LRU_PRIORITY;++cursor->priority)
Quoted 24 lines
drivers/gpu/drm/drm_evictable_lru.c new file +24
-0+list_for_each_entry (entity, &mgr->lru[cursor->priority], lru) +return entity; + +return NULL ; +} + +/** +* drm_lru_move_to_tail +* +* @entity: the lru entity to move to lru tail +* +* Move a lru entity to lru tail +*/ +void drm_lru_move_to_tail (struct drm_lru_entity * entity) +{ +struct list_head *lru; +struct drm_lru_mgr *mgr; + + mgr =entity_to_mgr (entity); +lockdep_assert_held (mgr->lru_lock); + lru = &mgr->lru[entity->priority]; +list_move_tail (&entity->lru, lru); +} +drivers/gpu/drm/drm_evictable_lru.c
Quoted 10 lines
drivers/gpu/drm/drm_evictable_lru.c new file +10
-0+/** +* drm_lru_bulk_move_range_tail +* +* @range: bulk move range +* @entity: lru_entity to move +* +* Move a lru_entity to the tail of a bulk move range +*/ +void drm_lru_bulk_move_range_tail(struct drm_lru_bulk_move_range *range, + struct
drm_lru_entity*entity)drivers/gpu/drm/drm_evictable_lru.c
Quoted 6 lines
drivers/gpu/drm/drm_evictable_lru.c new file +6
-0+{ +if (entity == range->last) +return ; + +if (entity == range->first) + range->first = container_of(entity->lru.next, struct drm_lru_entity,lru);
Quoted 25 lines
drivers/gpu/drm/drm_evictable_lru.c new file +24
-0+ +if (range->last) +list_move (&entity->lru, &range->last->lru); + + range->last = entity; +} +EXPORT_SYMBOL (drm_lru_bulk_move_range_tail); + +/** +* drm_lru_bulk_move_tail - bulk move range of entities to the LRU tail. +* +* @bulk: bulk_move structure +* +* Bulk move entities to the LRU tail, only valid to use when driver makes sure that +* resource order never changes. +*/ +void drm_lru_bulk_move_tail (struct drm_lru_bulk_move *bulk) +{ + +unsigned i, j; + +for (i =0 ; i <DRM_NUM_MEM_TYPES ; ++i) { +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; ++j) { + struct drm_lru_bulk_move_range *range = &bulk-range[i][j];
drivers/gpu/drm/drm_evictable_lru.c
Quoted 8 lines
drivers/gpu/drm/drm_evictable_lru.c new file +8
-0+struct drm_lru_mgr *mgr; + +if (!range->first) +continue ; + + mgr =entity_to_mgr (range->first); +lockdep_assert_held (mgr->lru_lock); + list_bulk_move_tail(&mgr->lru[range->first->priority],
&range->first->lru,Quoted 6 lines
drivers/gpu/drm/drm_evictable_lru.c new file +6
-0+ &range->last->lru); + } + } +} +EXPORT_SYMBOL (drm_lru_bulk_move_tail); +drivers/gpu/drm/drm_evictable_lru.c
Quoted 10 lines
drivers/gpu/drm/drm_evictable_lru.c new file +10
-0+/** +* drm_lru_add_bulk_move +* +* @entity: the lru entity to add to the bulk move range +* @bulk_move: the bulk move ranges to add the entity +* +* Add a lru entity to the tail of a bulk move range +*/ +void drm_lru_add_bulk_move (struct drm_lru_entity *entity, + struct drm_lru_bulk_move
*bulk_move)Quoted 164 lines
drivers/gpu/drm/drm_evictable_lru.c new file +46
-0+{ +struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + +if (!range->first) { + range->first = entity; + range->last = entity; +return ; + } + +drm_lru_bulk_move_range_tail (range, entity); +} + +EXPORT_SYMBOL (drm_lru_add_bulk_move); +/** +* drm_lru_del_bulk_move +* +* @entity: the lru entity to move from the bulk move range +* @bulk_move: the bulk move ranges to move the entity out of +* +* Move a lru entity out of bulk move range. This doesn't +* delete entity from lru manager's lru list. +*/ +void drm_lru_del_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move) +{ +struct drm_lru_bulk_move_range *range; + + range = &bulk_move->range[entity->mem_type][entity->priority]; + +if (unlikely (WARN_ON (!range->first || !range->last) || + (range->first == entity && range->last == entity))) { + range->first =NULL ; + range->last =NULL ; + }else if (range->first == entity) { + range->first =container_of (entity->lru.next, + struct drm_lru_entity, lru); + }else if (range->last == entity) { + range->last =container_of (entity->lru.prev, + struct drm_lru_entity, lru); + }else { +list_move (&entity->lru, &range->last->lru); + } +} +EXPORT_SYMBOL (drm_lru_del_bulk_move);include/drm/drm_device.h +7
-0@@ -8,6 +8,7 @@#include <drm/drm_legacy.h> #include <drm/drm_mode_config.h> +#include <drm/drm_evictable_lru.h> struct drm_driver ;struct drm_minor ;@@ -331,6 +332,12 @@ struct drm_device {*/ spinlock_t lru_lock; +/** +* @lru_mgr: Device global lru managers per memory type or memory +* region. Each lru manager manages a lru list of this memory type. +*/ +struct drm_lru_mgr lru_mgr[DRM_NUM_MEM_TYPES ]; +/* Everything below here is for legacy driver, never use! */ /* private: */ #if IS_ENABLED (CONFIG_DRM_LEGACY )include/drm/drm_evictable_lru.h new file +87
-0@@ -0,0 +1,188 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#ifndef _DRM_EVICTABLE_LRU_H_ +#define _DRM_EVICTABLE_LRU_H_ + +#include <linux/list.h> +#include <linux/spinlock_types.h> +#include <linux/spinlock.h> + +struct drm_device ; + +#define DRM_MAX_LRU_PRIORITY 4 +#define DRM_NUM_MEM_TYPES 8 + +/** +* struct drm_lru_entity +* +* @drm: drm device that this entity belongs to +* @mem_type: The memory type that this entity belongs to +* @size: resource size of this entity +* @priority: The priority of this entity +* @lru: least recent used list node, see &drm_lru_mgr.lru +* +* This structure represents an entity in drm_lru_mgr's +* list. This structure is supposed to be embedded in +* user's data structure. +*/ +struct drm_lru_entity { +struct drm_device *drm; +uint32_t mem_type; +uint64_t size; +uint32_t priority; +struct list_head lru; +}; + +/** +* struct drm_lru_mgr +* +* @used: whether this lru manager is used or not +* @size: size of the resource +* @usage: how much resource has been used +* @lru_lock: a weak reference to the global lru_lock +* @lru: least recent used list, per priority +* +* This structure maintains all the buffer allocations +* in a least recent used list, so a victim for eviction +* can be easily found. +*/ +struct drm_lru_mgr { +bool used; +uint64_t size; +uint64_t usage; +spinlock_t *lru_lock; +struct list_head lru[DRM_MAX_LRU_PRIORITY ]; +}; + +/** +* struct drm_lru_cursor +* +* @priority: the current priority +* +* Cursor to iterate over all entities in lru manager. +*/ +struct drm_lru_cursor { +unsigned priority; +}; + +/** +* struct drm_lru_bulk_move_range +* +* @first: the first entity in the range +* @last: the last entity in the range +* +* Range of entities on a lru list. +*/ +struct drm_lru_bulk_move_range +{ +struct drm_lru_entity *first; +struct drm_lru_entity *last; +}; + +/** +* struct drm_lru_bulk_move +* include/drm/drm_evictable_lru.h line 97
Quoted 10 lines
include/drm/drm_evictable_lru.h new file +10
-0+* @range: An array of bulk move range, each corelates to the drm_lru_mgr's +* lru list of the same memory type and same priority. +* +* A collection of bulk_move range which can be used to move drm_lru_entity +* on the lru list in a bulk way. It should be initialized through +* drm_lru_bulk_move_init. Add/delete a drm_lru_entity to bulk move should call +* drm_lru_add_bulk_move/drm_lru_del_bulk_move. +*/ +struct drm_lru_bulk_move { +struct drm_lru_bulk_move_range range[DRM_NUM_MEM_TYPES][DRM_MAX_LRU_PRIORITY];
Quoted 91 lines
include/drm/drm_evictable_lru.h new file +91
-0+}; + + + +/** +* drm_lru_add_entity +* +* @entity: the lru entity to add +* @mgr: the drm lru manager +* @priority: specify which priority list to add +* +* Add an entity to lru list +*/ +static inline void drm_lru_add_entity (struct drm_lru_entity *entity, +struct drm_lru_mgr *mgr,unsigned priority) +{ +lockdep_assert_held (mgr->lru_lock); +list_add_tail (&entity->lru, &mgr->lru[priority]); + mgr->usage += entity->size; +} + +/** +* drm_lru_remove_entity +* +* @entity: the lru entity to remove +* @mgr: the drm lru manager +* +* Remove an entity from lru list +*/ +static inline void drm_lru_remove_entity (struct drm_lru_entity *entity, +struct drm_lru_mgr *mgr) +{ +lockdep_assert_held (mgr->lru_lock); +list_del_init (&entity->lru); + mgr->usage -= entity->size; +} + +/** +* drm_lru_mgr_fini +* +* @mgr: the drm lru manager +* +* de-initialize a lru manager +*/ +static inline void drm_lru_mgr_fini (struct drm_lru_mgr *mgr) +{ + mgr->used = false; +} + +void drm_lru_entity_init (struct drm_lru_entity *entity,struct drm_device *drm, +uint32_t mem_type,uint64_t size,uint32_t priority); + +struct drm_lru_entity * +drm_lru_first (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor); + +struct drm_lru_entity * +drm_lru_next (struct drm_lru_mgr *mgr,struct drm_lru_cursor *cursor, +struct drm_lru_entity *entity); + +void drm_lru_mgr_init (struct drm_lru_mgr *mgr,uint64_t size, +spinlock_t *lru_lock); + +void drm_lru_move_to_tail (struct drm_lru_entity * entity); + +void drm_lru_bulk_move_init (struct drm_lru_bulk_move *bulk_move); + + +void drm_lru_bulk_move_tail (struct drm_lru_bulk_move *bulk); + +void drm_lru_bulk_move_range_tail (struct drm_lru_bulk_move_range *range, +struct drm_lru_entity *entity); + +void drm_lru_add_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move); + +void drm_lru_del_bulk_move (struct drm_lru_entity *entity, +struct drm_lru_bulk_move *bulk_move); +/** +* drm_lru_for_each_entity +* +* @mgr: the drm lru manager +* @cursor: cursor for the current position +* @entity: the current drm_lru_entity +* +* Iterate over all entities in drm lru manager +*/ +#define drm_lru_for_each_entity (mgr, cursor, entity) \ + for (entity = drm_lru_first(mgr, cursor); entity; \ + entity = drm_lru_next(mgr, cursor, entity)) + +#endif
Christian König
maintainerRe: [RFC 03/11] drm: introduce drm evictable LRU
Am 03.11.23 um 05:04 schrieb Zeng, Oak:[SNIP]
Quoted 6 lines
I also want to have a more advanced iterator at some point where we grab the BO lock for keeping a reference into the LRU list. Not sure how to do this if we don't have the BO here any more.
Need to think about that further,
Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class ofttm_resourceor any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
This is just a crazy idea I had because TTM tends to perform bad on certain tasks.
When we start to evict something we use a callback which indicates if an eviction is valuable or not. So it can happen that we have to skip quite a bunch of BOs on the LRU until we found one which is worth evicting.
Not it can be that the first eviction doesn't make enough room to fulfill the allocation requirement, in this case we currently start over at the beginning searching for some BO to evict.
I want to avoid this by being able to have cursors into the LRU, e.g. the next BO which can't move until we have evicted the current one.
BTW: How do you handle eviction here? I mean we can't call the evict callback with the spinlock held easily?
Christian.
Quoted 3 lines
Oak
RE: [RFC 03/11] drm: introduce drm evictable LRU
From: Christian König <[email protected]>
Sent: Friday, November 3, 2023 5:36 AM
To: Zeng, Oak <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [RFC 03/11] drm: introduce drm evictable LRU
Am 03.11.23 um 05:04 schrieb Zeng, Oak:[SNIP]
I also want to have a more advanced iterator at some point where we grab
the BO lock for keeping a reference into the LRU list. Not sure how to
do this if we don't have the BO here any more.
Need to think about that further,
Don't quite get the what you want to do with the advanced iterator. But with this work, the lru entity is a base class of ttm_resource or any other resource struct in hmm/svm. Lru is decoupled from bo concept - this is why this lru can be shared with svm code which is bo-less.
This is just a crazy idea I had because TTM tends to perform bad on certain tasks.
When we start to evict something we use a callback which indicates if an eviction is valuable or not. So it can happen that we have to skip quite a bunch of BOs on the LRU until we found one which is worth evicting.
Not it can be that the first eviction doesn't make enough room to fulfill the allocation requirement, in this case we currently start over at the beginning searching for some BO to evict.
I want to avoid this by being able to have cursors into the LRU, e.g. the next BO which can't move until we have evicted the current one.
Got you now. I didn’t know this problem so I didn’t try to fix this efficiency problem in this series. Theoretically I think we can fix this issue this way: change ttm_mem_evict_first to ttm_mem_evict_first_n and add a parameter to this function to specify how much room we want to yield; then we evict the first n objects to make enough room before return, or fail if we can’t make enough room. This scheme would need the caller of ttm_mem_evict_first to tell how much room he need – I think reasonable.
BTW: How do you handle eviction here? I mean we can't call the evict callback with the spinlock held easily?
I was actually struggling when I refactored ttm_mem_evict_first function. I moved this function to lru manager and abstracted 3 callback functions (evict_allowable/valuable, evict_entity, evict_busy_entity) – those need to relook when hmm/svm codes come in picture. I tried not to change any logic of this function – I know people worked on this function in the past 15 years so better to be very careful.
So in my current implementation, spinlock is held calling the evict_entity callback. Spinlock is unlocked before calling ttm_bo_evict in the evict_entity callback and re-held if we need to move entity in lru list. See details in patch 4 and patch 10. So it keeps exactly the original call sequence but does look awkward.
But I think you are right. We can release the spinlock in the drm_lru_evict_first function before calling evict callback.
Oak
Christian.
Oak
RFC 4/11 drm: Add evict function pointer to drm lru entity
In reply to Oak ZengDrm lru manager provides generic functions to manage lru list, and function to evict a lru entity. But how to evict an entity is implemented in an entity's sub-class. This patch introduces a few function pointers to drm lru entity for this purpose. Those functions are abstracted from the current ttm resource eviction process. They need to be tunned in the future when svm code comes into the picture.
Also implemented a drm_lru_evict_first function to evict the first lru entity from lru manager. Both ttm and svm codes are supposed to call this function to evict the first resource from lru list. This way ttm and svm codes can mutually evict resources from each other.
--- drivers/gpu/drm/drm_evictable_lru.c | 36 +++++++++++++- include/drm/drm_evictable_lru.h | 74 ++++++++++++++++++++++++++++- 2 files changed, 108 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_evictable_lru.c +35 -1
@@ -19,13 +19,15 @@ static inline struct drm_lru_mgr *entity_to_mgr(struct drm_lru_entity *entity) }void drm_lru_entity_init (struct drm_lru_entity *entity,struct drm_device *drm, -uint32_t mem_type,uint64_t size,uint32_t priority) +uint32_t mem_type,uint64_t size,uint32_t priority, +struct drm_lru_evict_func *evict_func) { entity->drm = drm; entity->mem_type = mem_type; entity->size = size; entity->priority = priority;INIT_LIST_HEAD (&entity->lru); + entity->evict_func = evict_func; }/**
@@ -230,3 +232,35 @@ void drm_lru_del_bulk_move(struct drm_lru_entity *entity, } }EXPORT_SYMBOL (drm_lru_del_bulk_move); + +int drm_lru_evict_first (struct drm_lru_mgr *mgr, +const struct drm_lru_evict_ctx *evict_ctx) +{ +struct drm_lru_entity *entity, *busy_entity =NULL ; +struct drm_lru_cursor cursor; +bool locked = false, busy = false, found = false; + +spin_lock (mgr->lru_lock); + +/* First find a victim to evict*/ +drm_lru_for_each_entity (mgr, &cursor, entity) { +if (!entity->evict_func->evict_allowable (entity, + evict_ctx, &busy, &locked)) { +if (!busy_entity && busy) + busy_entity = entity; +continue ; + } + found = true; +break ; + } + +/* We didn't find a victim, but we found a busy entity, i.e., +* other clients hold a reservation lock of this entity. Try +* to wait and evict this busy entity. +*/ +if (!found && busy_entity) +return busy_entity->evict_func->evict_busy_entity (busy_entity, evict_ctx); + +/* If here, we found a victim to evict*/ +return entity->evict_func->evict_entity (entity, evict_ctx, locked); +}
include/drm/drm_evictable_lru.h +73 -1
111 lines not on this page · raw message
Signature · 1 line
2.26.3
RFC 5/11 drm: Replace ttm macros with drm macros
In reply to Oak ZengTTM_MAX_BO_PRIORITY and TTM_NUM_MEM_TYPES are move from ttm to drm, so:
s/TTM_MAX_BO_PRIORITY/DRM_MAX_LRU_PRIORITY s/TTM_NUM_MEM_TYPES/DRM_NUM_MEM_TYPES
--- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 2 +- drivers/gpu/drm/i915/intel_region_ttm.c | 2 +- drivers/gpu/drm/i915/selftests/mock_region.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 8 ++++---- drivers/gpu/drm/ttm/ttm_resource.c | 12 ++++++------ drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +- drivers/gpu/drm/xe/xe_bo.h | 2 +- include/drm/ttm/ttm_device.h | 6 +++--- include/drm/ttm/ttm_range_manager.h | 4 ++-- include/drm/ttm/ttm_resource.h | 10 ++++------ 11 files changed, 25 insertions(+), 27 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1 -1
@@ -987,7 +987,7 @@ void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)spin_lock (bo->bdev->lru_lock);if (shrinkable) {/* Try to keep shmem_tt from being considered for shrinking. */ - bo->priority =TTM_MAX_BO_PRIORITY -1 ; + bo->priority =DRM_MAX_LRU_PRIORITY -1 ; }else if (obj->mm.madv !=I915_MADV_WILLNEED ) { bo->priority =I915_TTM_PRIO_PURGE ; }else if (!i915_gem_object_has_pages (obj)) {
drivers/gpu/drm/i915/intel_region_ttm.c +1 -1
@@ -63,7 +63,7 @@ int intel_region_to_ttm_type(const struct intel_memory_region *mem)return TTM_PL_SYSTEM ; type = mem->instance +TTM_PL_PRIV ; -GEM_BUG_ON (type >=TTM_NUM_MEM_TYPES ); +GEM_BUG_ON (type >=DRM_NUM_MEM_TYPES );return type; }
drivers/gpu/drm/i915/selftests/mock_region.c +1 -1
@@ -111,7 +111,7 @@ mock_region_create(struct drm_i915_private *i915, resource_size_t io_size) {int instance =ida_alloc_max (&i915->selftest.mock_region_instances, -TTM_NUM_MEM_TYPES -TTM_PL_PRIV -1 , +DRM_NUM_MEM_TYPES -TTM_PL_PRIV -1 ,GFP_KERNEL );if (instance <0 )
drivers/gpu/drm/ttm/ttm_device.c +4 -4
@@ -148,7 +148,7 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,int ret;spin_lock (bdev->lru_lock); -for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { +for (i =TTM_PL_SYSTEM ; i <DRM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)continue ;
@@ -245,7 +245,7 @@ void ttm_device_fini(struct ttm_device *bdev)destroy_workqueue (bdev->wq);spin_lock (bdev->lru_lock); -for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) +for (i =0 ; i <DRM_MAX_LRU_PRIORITY ; ++i)if (list_empty (&man->lru[0 ]))pr_debug ("Swap list %d was clean\n" , i);spin_unlock (bdev->lru_lock);
@@ -287,12 +287,12 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev)ttm_device_clear_lru_dma_mappings (bdev, &bdev->pinned); -for (i =TTM_PL_SYSTEM ; i <TTM_NUM_MEM_TYPES ; ++i) { +for (i =TTM_PL_SYSTEM ; i <DRM_NUM_MEM_TYPES ; ++i) { man =ttm_manager_type (bdev, i);if (!man || !man->use_tt)continue ; -for (j =0 ; j <TTM_MAX_BO_PRIORITY ; ++j) +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; ++j)ttm_device_clear_lru_dma_mappings (bdev, &man->lru[j]); } }
drivers/gpu/drm/ttm/ttm_resource.c +6 -6
@@ -54,8 +54,8 @@ void ttm_lru_bulk_move_tail(struct ttm_lru_bulk_move *bulk) {unsigned i, j; -for (i =0 ; i <TTM_NUM_MEM_TYPES ; ++i) { -for (j =0 ; j <TTM_MAX_BO_PRIORITY ; ++j) { +for (i =0 ; i <DRM_NUM_MEM_TYPES ; ++i) { +for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; ++j) {struct ttm_lru_bulk_move_pos *pos = &bulk->pos[i][j];struct ttm_resource_manager *man;
@@ -393,7 +393,7 @@ void ttm_resource_manager_init(struct ttm_resource_manager *man, man->size = size; man->usage =0 ; -for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) +for (i =0 ; i <DRM_MAX_LRU_PRIORITY ; ++i)INIT_LIST_HEAD (&man->lru[i]); man->move =NULL ; }
@@ -425,7 +425,7 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,*/ spin_lock (bdev->lru_lock); -for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) { +for (i =0 ; i <DRM_MAX_LRU_PRIORITY ; ++i) {while (!list_empty (&man->lru[i])) {spin_unlock (bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx,
@@ -504,7 +504,7 @@ ttm_resource_manager_first(struct ttm_resource_manager *man,lockdep_assert_held (man->bdev->lru_lock); -for (cursor->priority =0 ; cursor->priority <TTM_MAX_BO_PRIORITY ; +for (cursor->priority =0 ; cursor->priority <DRM_MAX_LRU_PRIORITY ; ++cursor->priority)list_for_each_entry (res, &man->lru[cursor->priority], lru)return res;
@@ -531,7 +531,7 @@ ttm_resource_manager_next(struct ttm_resource_manager *man,list_for_each_entry_continue (res, &man->lru[cursor->priority], lru)return res; -for (++cursor->priority; cursor->priority <TTM_MAX_BO_PRIORITY ; +for (++cursor->priority; cursor->priority <DRM_MAX_LRU_PRIORITY ; ++cursor->priority)list_for_each_entry (res, &man->lru[cursor->priority], lru)return res;
drivers/gpu/drm/vmwgfx/vmwgfx_bo.c +1 -1
@@ -383,7 +383,7 @@ static int vmw_bo_init(struct vmw_private *dev_priv,memset (vmw_bo,0 ,sizeof (*vmw_bo)); -BUILD_BUG_ON (TTM_MAX_BO_PRIORITY <=3 ); +BUILD_BUG_ON (DRM_MAX_LRU_PRIORITY <=3 ); vmw_bo->tbo.priority =3 ; vmw_bo->res_tree =RB_ROOT ;
drivers/gpu/drm/vmwgfx/vmwgfx_bo.h +1 -1
@@ -82,7 +82,7 @@ struct vmw_bo {struct ttm_bo_kmap_obj map;struct rb_root res_tree; -u32 res_prios[TTM_MAX_BO_PRIORITY ]; +u32 res_prios[DRM_MAX_LRU_PRIORITY ];atomic_t cpu_writers;/* Not ref-counted. Protected by binding_mutex */
drivers/gpu/drm/xe/xe_bo.h +1 -1
@@ -72,7 +72,7 @@#define XE_PL_TT TTM_PL_TT#define XE_PL_VRAM0 TTM_PL_VRAM#define XE_PL_VRAM1 (XE_PL_VRAM0 + 1) -#define XE_PL_STOLEN (TTM_NUM_MEM_TYPES - 1) +#define XE_PL_STOLEN (DRM_NUM_MEM_TYPES - 1)#define XE_BO_PROPS_INVALID (-1)
include/drm/ttm/ttm_device.h +3 -3
@@ -235,7 +235,7 @@ struct ttm_device {/** * @man_drv: An array of resource_managers, one per resource type. */ -struct ttm_resource_manager *man_drv[TTM_NUM_MEM_TYPES ]; +struct ttm_resource_manager *man_drv[DRM_NUM_MEM_TYPES ];/** * @vma_manager: Address space manager for finding BOs to mmap.
@@ -277,14 +277,14 @@ static inline struct ttm_resource_manager * ttm_manager_type(struct ttm_device *bdev, int mem_type) {BUILD_BUG_ON (__builtin_constant_p (mem_type) - && mem_type >= TTM_NUM_MEM_TYPES); + && mem_type >= DRM_NUM_MEM_TYPES);return bdev->man_drv[mem_type]; }static inline void ttm_set_driver_manager (struct ttm_device *bdev,int type,struct ttm_resource_manager *manager) { -BUILD_BUG_ON (__builtin_constant_p (type) && type >=TTM_NUM_MEM_TYPES ); +BUILD_BUG_ON (__builtin_constant_p (type) && type >=DRM_NUM_MEM_TYPES ); bdev->man_drv[type] = manager; }
include/drm/ttm/ttm_range_manager.h +2 -2
@@ -43,14 +43,14 @@ static __always_inline int ttm_range_man_init(struct ttm_device *bdev, unsigned int type, bool use_tt, unsigned long p_size) { - BUILD_BUG_ON(__builtin_constant_p(type) && type >= TTM_NUM_MEM_TYPES); + BUILD_BUG_ON(__builtin_constant_p(type) && type >= DRM_NUM_MEM_TYPES);return ttm_range_man_init_nocheck (bdev, type, use_tt, p_size); } static __always_inline int ttm_range_man_fini(struct ttm_device *bdev,unsigned int type) { -BUILD_BUG_ON (__builtin_constant_p (type) && type >=TTM_NUM_MEM_TYPES ); +BUILD_BUG_ON (__builtin_constant_p (type) && type >=DRM_NUM_MEM_TYPES );return ttm_range_man_fini_nocheck (bdev, type); }#endif
include/drm/ttm/ttm_resource.h +4 -6
@@ -34,9 +34,7 @@#include <drm/drm_print.h> #include <drm/ttm/ttm_caching.h> #include <drm/ttm/ttm_kmap_iter.h> - -#define TTM_MAX_BO_PRIORITY 4U -#define TTM_NUM_MEM_TYPES 8 +#include <drm/drm_evictable_lru.h> struct ttm_device ;struct ttm_resource_manager ;
@@ -167,7 +165,7 @@ struct ttm_resource_manager {/* * Protected by the bdev->lru_lock. */ -struct list_head lru[TTM_MAX_BO_PRIORITY ]; +struct list_head lru[DRM_MAX_LRU_PRIORITY ];/** * @usage: How much of the resources are used, protected by the
@@ -253,7 +251,7 @@ struct ttm_lru_bulk_move_pos {* ttm_lru_bulk_move_init() and ttm_bo_set_bulk_move(). */ struct ttm_lru_bulk_move { -struct ttm_lru_bulk_move_pos pos[TTM_NUM_MEM_TYPES ][TTM_MAX_BO_PRIORITY ]; +struct ttm_lru_bulk_move_pos pos[DRM_NUM_MEM_TYPES ][DRM_MAX_LRU_PRIORITY ]; };/**
@@ -309,7 +307,7 @@ ttm_resource_manager_set_used(struct ttm_resource_manager *man, bool used) {int i; -for (i =0 ; i <TTM_MAX_BO_PRIORITY ; i++) +for (i =0 ; i <DRM_MAX_LRU_PRIORITY ; i++)WARN_ON (!list_empty (&man->lru[i])); man->use_type = used; }
Signature · 1 line
2.26.3
RFC 6/11 drm/ttm: Set lru manager to ttm resource manager
In reply to Oak ZengAdd a weak reference of lru manager to ttm resource manager, and add a function to set lru manager for ttm resource manager.
--- include/drm/ttm/ttm_resource.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
include/drm/ttm/ttm_resource.h +19 -0
@@ -46,6 +46,7 @@ struct iosys_map;struct io_mapping ;struct sg_table ;struct scatterlist ; +struct drm_lru_mgr ;struct ttm_resource_manager_func {/**
@@ -172,6 +173,12 @@ struct ttm_resource_manager {* bdev->lru_lock. */ uint64_t usage; + +/** +* @lru_mgr: weak reference of the lru manager that manages lru +* list for this ttm resource manager. +*/ +struct drm_lru_mgr *lru_mgr; };/**
@@ -326,6 +333,18 @@ static inline bool ttm_resource_manager_used(struct ttm_resource_manager *man)return man->use_type; } +/** +* ttm_resource_manager_set_lru_mgr +* +* @man: ttm resource manager +* @mgr: pointing to lru manager +*/ +static inline void +ttm_resource_manager_set_lru_mgr (struct ttm_resource_manager *man,struct drm_lru_mgr *mgr) +{ + man->lru_mgr = mgr; +} +/** * ttm_resource_manager_cleanup *
Signature · 1 line
2.26.3
RFC 7/11 drm/ttm: re-parameterize a few ttm functions
In reply to Oak ZengAdd a struct drm_device *drm parameter to function ttm_range_man_init, ttm_range_man_fini, ttm_sys_man_init, and ttm_sys_man_free. This drm parameter will be used in the coming patches to retrieve and initialize drm lru manager.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 ++++---- drivers/gpu/drm/drm_gem_vram_helper.c | 8 ++++---- drivers/gpu/drm/loongson/lsdc_ttm.c | 8 ++++---- drivers/gpu/drm/nouveau/nouveau_ttm.c | 8 ++++---- drivers/gpu/drm/qxl/qxl_ttm.c | 6 +++--- drivers/gpu/drm/radeon/radeon_ttm.c | 8 ++++---- drivers/gpu/drm/ttm/ttm_device.c | 2 +- drivers/gpu/drm/ttm/ttm_module.h | 3 ++- drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_sys_manager.c | 6 +++++- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 4 ++-- include/drm/ttm/ttm_range_manager.h | 13 +++++++------ 12 files changed, 44 insertions(+), 36 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4 -4
@@ -75,7 +75,7 @@ static int amdgpu_ttm_init_on_chip(struct amdgpu_device *adev, unsigned int type, uint64_t size_in_page) { -return ttm_range_man_init (&adev->mman.bdev, type, +return ttm_range_man_init (adev_to_drm (adev), &adev->mman.bdev, type, false, size_in_page); }
@@ -2026,9 +2026,9 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev)amdgpu_vram_mgr_fini (adev);amdgpu_gtt_mgr_fini (adev);amdgpu_preempt_mgr_fini (adev); -ttm_range_man_fini (&adev->mman.bdev,AMDGPU_PL_GDS ); -ttm_range_man_fini (&adev->mman.bdev,AMDGPU_PL_GWS ); -ttm_range_man_fini (&adev->mman.bdev,AMDGPU_PL_OA ); +ttm_range_man_fini (adev_to_drm (adev), &adev->mman.bdev,AMDGPU_PL_GDS ); +ttm_range_man_fini (adev_to_drm (adev), &adev->mman.bdev,AMDGPU_PL_GWS ); +ttm_range_man_fini (adev_to_drm (adev), &adev->mman.bdev,AMDGPU_PL_OA );ttm_device_fini (&adev->mman.bdev); adev->mman.initialized = false;DRM_INFO ("amdgpu: ttm finalized\n" );
drivers/gpu/drm/drm_gem_vram_helper.c +4 -4
@@ -1009,7 +1009,7 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev,if (ret)return ret; - ret =ttm_range_man_init (&vmm->bdev,TTM_PL_VRAM , + ret =ttm_range_man_init (dev, &vmm->bdev,TTM_PL_VRAM , false, vram_size >>PAGE_SHIFT );if (ret)return ret;
@@ -1017,9 +1017,9 @@ static int drm_vram_mm_init(struct drm_vram_mm *vmm, struct drm_device *dev,return 0 ; } -static void drm_vram_mm_cleanup (struct drm_vram_mm *vmm) +static void drm_vram_mm_cleanup (struct drm_device *drm,struct drm_vram_mm *vmm) { -ttm_range_man_fini (&vmm->bdev,TTM_PL_VRAM ); +ttm_range_man_fini (drm, &vmm->bdev,TTM_PL_VRAM );ttm_device_fini (&vmm->bdev); }
@@ -1056,7 +1056,7 @@ static void drm_vram_helper_release_mm(struct drm_device *dev)if (!dev->vram_mm)return ; -drm_vram_mm_cleanup (dev->vram_mm); +drm_vram_mm_cleanup (dev, dev->vram_mm);kfree (dev->vram_mm); dev->vram_mm =NULL ; }
drivers/gpu/drm/loongson/lsdc_ttm.c +4 -4
@@ -533,8 +533,8 @@ static void lsdc_ttm_fini(struct drm_device *ddev, void *data) {struct lsdc_device *ldev = (struct lsdc_device *)data; -ttm_range_man_fini (&ldev->bdev,TTM_PL_VRAM ); -ttm_range_man_fini (&ldev->bdev,TTM_PL_TT ); +ttm_range_man_fini (ddev, &ldev->bdev,TTM_PL_VRAM ); +ttm_range_man_fini (ddev, &ldev->bdev,TTM_PL_TT );ttm_device_fini (&ldev->bdev);
@@ -556,7 +556,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev) num_vram_pages = ldev->vram_size >>PAGE_SHIFT ; - ret =ttm_range_man_init (&ldev->bdev,TTM_PL_VRAM , false, num_vram_pages); + ret =ttm_range_man_init (&ldev->base, &ldev->bdev,TTM_PL_VRAM , false, num_vram_pages);if (unlikely (ret))return ret;
@@ -567,7 +567,7 @@ int lsdc_ttm_init(struct lsdc_device *ldev) num_gtt_pages = ldev->gtt_size >>PAGE_SHIFT ; - ret =ttm_range_man_init (&ldev->bdev,TTM_PL_TT , true, num_gtt_pages); + ret =ttm_range_man_init (&ldev->base, &ldev->bdev,TTM_PL_TT , true, num_gtt_pages);if (unlikely (ret))return ret;
drivers/gpu/drm/nouveau/nouveau_ttm.c +4 -4
@@ -194,7 +194,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm)ttm_resource_manager_set_used (man, true);return 0 ; }else { -return ttm_range_man_init (&drm->ttm.bdev,TTM_PL_VRAM , false, +return ttm_range_man_init (drm->dev, &drm->ttm.bdev,TTM_PL_VRAM , false, drm->gem.vram_available >>PAGE_SHIFT ); } }
@@ -211,7 +211,7 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm)ttm_set_driver_manager (&drm->ttm.bdev,TTM_PL_VRAM ,NULL );kfree (man); }else -ttm_range_man_fini (&drm->ttm.bdev,TTM_PL_VRAM ); +ttm_range_man_fini (drm->dev, &drm->ttm.bdev,TTM_PL_VRAM ); } static int
@@ -226,7 +226,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)else if (!drm->agp.bridge) func = &nv04_gart_manager;else -return ttm_range_man_init (&drm->ttm.bdev,TTM_PL_TT , true, +return ttm_range_man_init (drm->dev, &drm->ttm.bdev,TTM_PL_TT , true, size_pages); man =kzalloc (sizeof (*man),GFP_KERNEL );
@@ -248,7 +248,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm)if (drm->client.device.info.family <NV_DEVICE_INFO_V0_TESLA && drm->agp.bridge) -ttm_range_man_fini (&drm->ttm.bdev,TTM_PL_TT ); +ttm_range_man_fini (drm->dev, &drm->ttm.bdev,TTM_PL_TT );else {ttm_resource_manager_set_used (man, false);ttm_resource_manager_evict_all (&drm->ttm.bdev, man);
drivers/gpu/drm/qxl/qxl_ttm.c +3 -3
@@ -186,7 +186,7 @@ static int qxl_ttm_init_mem_type(struct qxl_device *qdev, unsigned int type, uint64_t size) { -return ttm_range_man_init (&qdev->mman.bdev, type, false, size); +return ttm_range_man_init (&qdev->ddev, &qdev->mman.bdev, type, false, size); }int qxl_ttm_init (struct qxl_device *qdev)
@@ -227,8 +227,8 @@ int qxl_ttm_init(struct qxl_device *qdev)void qxl_ttm_fini (struct qxl_device *qdev) { -ttm_range_man_fini (&qdev->mman.bdev,TTM_PL_VRAM ); -ttm_range_man_fini (&qdev->mman.bdev,TTM_PL_PRIV ); +ttm_range_man_fini (&qdev->ddev, &qdev->mman.bdev,TTM_PL_VRAM ); +ttm_range_man_fini (&qdev->ddev, &qdev->mman.bdev,TTM_PL_PRIV );ttm_device_fini (&qdev->mman.bdev);DRM_INFO ("qxl: ttm finalized\n" ); }
drivers/gpu/drm/radeon/radeon_ttm.c +4 -4
@@ -68,13 +68,13 @@ struct radeon_device *radeon_get_rdev(struct ttm_device *bdev)static int radeon_ttm_init_vram (struct radeon_device *rdev) { -return ttm_range_man_init (&rdev->mman.bdev,TTM_PL_VRAM , +return ttm_range_man_init (rdev->ddev, &rdev->mman.bdev,TTM_PL_VRAM , false, rdev->mc.real_vram_size >>PAGE_SHIFT ); }static int radeon_ttm_init_gtt (struct radeon_device *rdev) { -return ttm_range_man_init (&rdev->mman.bdev,TTM_PL_TT , +return ttm_range_man_init (rdev->ddev, &rdev->mman.bdev,TTM_PL_TT , true, rdev->mc.gtt_size >>PAGE_SHIFT ); }
@@ -753,8 +753,8 @@ void radeon_ttm_fini(struct radeon_device *rdev) }radeon_bo_unref (&rdev->stolen_vga_memory); } -ttm_range_man_fini (&rdev->mman.bdev,TTM_PL_VRAM ); -ttm_range_man_fini (&rdev->mman.bdev,TTM_PL_TT ); +ttm_range_man_fini (rdev->ddev, &rdev->mman.bdev,TTM_PL_VRAM ); +ttm_range_man_fini (rdev->ddev, &rdev->mman.bdev,TTM_PL_TT );ttm_device_fini (&rdev->mman.bdev);radeon_gart_fini (rdev); rdev->mman.initialized = false;
drivers/gpu/drm/ttm/ttm_device.c +1 -1
@@ -212,7 +212,7 @@ int ttm_device_init(struct ttm_device *bdev, const struct ttm_device_funcs *func bdev->funcs = funcs; -ttm_sys_man_init (bdev); +ttm_sys_man_init (drm, bdev);ttm_pool_init (&bdev->pool, drm?drm->dev:NULL ,NUMA_NO_NODE , use_dma_alloc, use_dma32);
drivers/gpu/drm/ttm/ttm_module.h +2 -1
@@ -35,9 +35,10 @@struct dentry ;struct ttm_device ; +struct drm_device ;extern struct dentry *ttm_debugfs_root; -void ttm_sys_man_init (struct ttm_device *bdev); +void ttm_sys_man_init (struct drm_device *drm,struct ttm_device *bdev);#endif /* _TTM_MODULE_H_ */
drivers/gpu/drm/ttm/ttm_range_manager.c +4 -2
@@ -166,6 +166,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = {* ttm_range_man_init_nocheck - Initialise a generic range manager for the * selected memory type. * +* @drm: drm device * @bdev: ttm device * @type: memory manager type * @use_tt: if the memory manager uses tt
@@ -175,7 +176,7 @@ static const struct ttm_resource_manager_func ttm_range_manager_func = {* * Return: %0 on success or a negative error code on failure */ -int ttm_range_man_init_nocheck (struct ttm_device *bdev, +int ttm_range_man_init_nocheck (struct drm_device *drm,struct ttm_device *bdev,unsigned type,bool use_tt,unsigned long p_size) {
@@ -206,12 +207,13 @@ EXPORT_SYMBOL(ttm_range_man_init_nocheck);* ttm_range_man_fini_nocheck - Remove the generic range manager from a slot * and tear it down. * +* @drm: drm device * @bdev: ttm device * @type: memory manager type * * Return: %0 on success or a negative error code on failure */ -int ttm_range_man_fini_nocheck (struct ttm_device *bdev, +int ttm_range_man_fini_nocheck (struct drm_device *drm,struct ttm_device *bdev,unsigned type) {struct ttm_resource_manager *man =ttm_manager_type (bdev, type);
drivers/gpu/drm/ttm/ttm_sys_manager.c +5 -1
@@ -20,6 +20,10 @@ static int ttm_sys_man_alloc(struct ttm_resource_manager *man,return 0 ; } +/* FIXME: Need to call drm_lru_mgr_fini. This requires a +* struct drm_device *drm parameter which need the change +* the definition of struct ttm_resource_manager_func. A +* very intrusive change. Leave it for now */ static void ttm_sys_man_free (struct ttm_resource_manager *man,struct ttm_resource *res) {
@@ -32,7 +36,7 @@ static const struct ttm_resource_manager_func ttm_sys_manager_func = { .free = ttm_sys_man_free, }; -void ttm_sys_man_init (struct ttm_device *bdev) +void ttm_sys_man_init (struct drm_device *drm,struct ttm_device *bdev) {struct ttm_resource_manager *man = &bdev->sysman;
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +2 -2
@@ -708,7 +708,7 @@ static int vmw_dma_masks(struct vmw_private *dev_priv)static int vmw_vram_manager_init (struct vmw_private *dev_priv) {int ret; - ret =ttm_range_man_init (&dev_priv->bdev,TTM_PL_VRAM , false, + ret =ttm_range_man_init (&dev_priv->drm, &dev_priv->bdev,TTM_PL_VRAM , false, dev_priv->vram_size >>PAGE_SHIFT );ttm_resource_manager_set_used (ttm_manager_type (&dev_priv->bdev,TTM_PL_VRAM ), false);return ret;
@@ -716,7 +716,7 @@ static int vmw_vram_manager_init(struct vmw_private *dev_priv)static void vmw_vram_manager_fini (struct vmw_private *dev_priv) { -ttm_range_man_fini (&dev_priv->bdev,TTM_PL_VRAM ); +ttm_range_man_fini (&dev_priv->drm, &dev_priv->bdev,TTM_PL_VRAM ); } static int vmw_setup_pci_resources(struct vmw_private *dev,
include/drm/ttm/ttm_range_manager.h +7 -6
@@ -34,23 +34,24 @@ to_ttm_range_mgr_node(struct ttm_resource *res) return container_of(res, struct ttm_range_mgr_node, base); } -int ttm_range_man_init_nocheck (struct ttm_device *bdev, +int ttm_range_man_init_nocheck (struct drm_device *drm,struct ttm_device *bdev,unsigned type,bool use_tt,unsigned long p_size); -int ttm_range_man_fini_nocheck (struct ttm_device *bdev, +int ttm_range_man_fini_nocheck (struct drm_device *drm,struct ttm_device *bdev,unsigned type); -static __always_inline int ttm_range_man_init(struct ttm_device *bdev, +static __always_inline int ttm_range_man_init(struct drm_device *drm, +struct ttm_device *bdev,unsigned int type,bool use_tt,unsigned long p_size) {BUILD_BUG_ON (__builtin_constant_p (type) && type >=DRM_NUM_MEM_TYPES ); -return ttm_range_man_init_nocheck (bdev, type, use_tt, p_size); +return ttm_range_man_init_nocheck (drm, bdev, type, use_tt, p_size); } -static __always_inline int ttm_range_man_fini(struct ttm_device *bdev, +static __always_inline int ttm_range_man_fini(struct drm_device *drm, struct ttm_device *bdev,unsigned int type) {BUILD_BUG_ON (__builtin_constant_p (type) && type >=DRM_NUM_MEM_TYPES ); -return ttm_range_man_fini_nocheck (bdev, type); +return ttm_range_man_fini_nocheck (drm, bdev, type); }#endif
Signature · 1 line
2.26.3
RFC 8/11 drm: Initialize drm lru manager
In reply to Oak ZengInitialize lru_mgr for each memory type or memory region. Also set ttm_resource_manager's weak reference to drm lru manager.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 6 ++++++ drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 ++++++++++ drivers/gpu/drm/nouveau/nouveau_ttm.c | 12 ++++++++++++ drivers/gpu/drm/ttm/ttm_range_manager.c | 6 ++++++ drivers/gpu/drm/ttm/ttm_sys_manager.c | 2 ++ drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 ++++++ drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 6 ++++++ drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 6 ++++++ 10 files changed, 66 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c +6 -0
@@ -278,6 +278,7 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size) {struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr;struct ttm_resource_manager *man = &mgr->manager; +struct drm_device *drm =adev_to_drm (adev);uint64_t start, size; man->use_tt = true;
@@ -292,6 +293,9 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size)ttm_set_driver_manager (&adev->mman.bdev,TTM_PL_TT , &mgr->manager);ttm_resource_manager_set_used (man, true); + +drm_lru_mgr_init (&drm->lru_mgr[TTM_PL_TT ], gtt_size, &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[TTM_PL_TT ]);return 0 ; }
@@ -307,6 +311,7 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev) {struct amdgpu_gtt_mgr *mgr = &adev->mman.gtt_mgr;struct ttm_resource_manager *man = &mgr->manager; +struct drm_device *drm =adev_to_drm (adev);int ret;ttm_resource_manager_set_used (man, false);
@@ -321,4 +326,5 @@ void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev)ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&adev->mman.bdev,TTM_PL_TT ,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[TTM_PL_TT ]); }
drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c +6 -0
@@ -100,6 +100,7 @@ static const struct ttm_resource_manager_func amdgpu_preempt_mgr_func = {int amdgpu_preempt_mgr_init (struct amdgpu_device *adev) {struct ttm_resource_manager *man = &adev->mman.preempt_mgr; +struct drm_device *drm =adev_to_drm (adev);int ret; man->use_tt = true;
@@ -115,6 +116,9 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev)ttm_set_driver_manager (&adev->mman.bdev,AMDGPU_PL_PREEMPT , man);ttm_resource_manager_set_used (man, true); + +drm_lru_mgr_init (&drm->lru_mgr[AMDGPU_PL_PREEMPT ], (1 <<30 ), &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[VMW_PL_SYSTEM ]);return 0 ; }
@@ -129,6 +133,7 @@ int amdgpu_preempt_mgr_init(struct amdgpu_device *adev)void amdgpu_preempt_mgr_fini (struct amdgpu_device *adev) {struct ttm_resource_manager *man = &adev->mman.preempt_mgr; +struct drm_device *drm =adev_to_drm (adev);int ret;ttm_resource_manager_set_used (man, false);
@@ -141,4 +146,5 @@ void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev)ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&adev->mman.bdev,AMDGPU_PL_PREEMPT ,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[AMDGPU_PL_PREEMPT ]); }
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +6 -0
@@ -884,6 +884,7 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev) {struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr;struct ttm_resource_manager *man = &mgr->manager; +struct drm_device *drm =adev_to_drm (adev);int err; ttm_resource_manager_init(man, &adev->mman.bdev,
@@ -907,6 +908,9 @@ int amdgpu_vram_mgr_init(struct amdgpu_device *adev)ttm_set_driver_manager (&adev->mman.bdev,TTM_PL_VRAM , &mgr->manager);ttm_resource_manager_set_used (man, true); +drm_lru_mgr_init (&drm->lru_mgr[TTM_PL_VRAM ], adev->gmc.real_vram_size, + &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[TTM_PL_VRAM ]);return 0 ; }
@@ -922,6 +926,7 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev) {struct amdgpu_vram_mgr *mgr = &adev->mman.vram_mgr;struct ttm_resource_manager *man = &mgr->manager; +struct drm_device *drm =adev_to_drm (adev);int ret;struct amdgpu_vram_reservation *rsv, *temp;
@@ -945,4 +950,5 @@ void amdgpu_vram_mgr_fini(struct amdgpu_device *adev)ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&adev->mman.bdev,TTM_PL_VRAM ,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[TTM_PL_VRAM ]); }
drivers/gpu/drm/i915/i915_ttm_buddy_manager.c +10 -0
@@ -304,6 +304,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev, {struct ttm_resource_manager *man;struct i915_ttm_buddy_manager *bman; +struct drm_i915_private *i915 =container_of (bdev, + struct drm_i915_private, bdev); +struct drm_device *drm = &i915->drm;int err; bman =kzalloc (sizeof (*bman),GFP_KERNEL );
@@ -329,6 +332,9 @@ int i915_ttm_buddy_man_init(struct ttm_device *bdev,ttm_resource_manager_set_used (man, true);ttm_set_driver_manager (bdev, type, man); +drm_lru_mgr_init (&drm->lru_mgr[type], bman->mm.size >>PAGE_SHIFT , &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[type]); +return 0 ; err_free_bman:
@@ -350,6 +356,9 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type) {struct ttm_resource_manager *man =ttm_manager_type (bdev, type);struct i915_ttm_buddy_manager *bman =to_buddy_manager (man); +struct drm_i915_private *i915 =container_of (bdev, + struct drm_i915_private, bdev); +struct drm_device *drm = &i915->drm;struct drm_buddy *mm = &bman->mm;int ret;
@@ -369,6 +378,7 @@ int i915_ttm_buddy_man_fini(struct ttm_device *bdev, unsigned int type)mutex_unlock (&bman->lock);ttm_resource_manager_cleanup (man); +drm_lru_mgr_fini (&drm->lru_mgr[type]);kfree (bman);return 0 ;
drivers/gpu/drm/nouveau/nouveau_ttm.c +12 -0
@@ -182,6 +182,7 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm) {if (drm->client.device.info.family >=NV_DEVICE_INFO_V0_TESLA ) {struct ttm_resource_manager *man =kzalloc (sizeof (*man),GFP_KERNEL ); +struct drm_device *drm_dev = drm->dev;if (!man)return -ENOMEM ;
@@ -192,6 +193,9 @@ nouveau_ttm_init_vram(struct nouveau_drm *drm) drm->gem.vram_available >> PAGE_SHIFT);ttm_set_driver_manager (&drm->ttm.bdev,TTM_PL_VRAM , man);ttm_resource_manager_set_used (man, true); +drm_lru_mgr_init (&drm_dev->lru_mgr[TTM_PL_VRAM ], + drm->gem.vram_available >>PAGE_SHIFT , &drm_dev->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm_dev->lru_mgr[TTM_PL_VRAM ]);return 0 ; }else { return ttm_range_man_init(drm->dev, &drm->ttm.bdev, TTM_PL_VRAM, false,
@@ -205,10 +209,13 @@ nouveau_ttm_fini_vram(struct nouveau_drm *drm)struct ttm_resource_manager *man =ttm_manager_type (&drm->ttm.bdev,TTM_PL_VRAM );if (drm->client.device.info.family >=NV_DEVICE_INFO_V0_TESLA ) { +struct drm_device *drm_dev = drm->dev; +ttm_resource_manager_set_used (man, false);ttm_resource_manager_evict_all (&drm->ttm.bdev, man);ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&drm->ttm.bdev,TTM_PL_VRAM ,NULL ); +drm_lru_mgr_fini (&drm_dev->lru_mgr[TTM_PL_VRAM ]);kfree (man); }else ttm_range_man_fini (drm->dev, &drm->ttm.bdev,TTM_PL_VRAM );
@@ -220,6 +227,7 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)struct ttm_resource_manager *man;unsigned long size_pages = drm->gem.gart_available >>PAGE_SHIFT ;const struct ttm_resource_manager_func *func =NULL ; +struct drm_device *drm_dev = drm->dev;if (drm->client.device.info.family >=NV_DEVICE_INFO_V0_TESLA ) func = &nouveau_gart_manager;
@@ -238,6 +246,8 @@ nouveau_ttm_init_gtt(struct nouveau_drm *drm)ttm_resource_manager_init (man, &drm->ttm.bdev, size_pages);ttm_set_driver_manager (&drm->ttm.bdev,TTM_PL_TT , man);ttm_resource_manager_set_used (man, true); +drm_lru_mgr_init (&drm_dev->lru_mgr[TTM_PL_TT ], size_pages, &drm_dev->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm_dev->lru_mgr[TTM_PL_TT ]);return 0 ; }
@@ -245,6 +255,7 @@ static void nouveau_ttm_fini_gtt(struct nouveau_drm *drm) {struct ttm_resource_manager *man =ttm_manager_type (&drm->ttm.bdev,TTM_PL_TT ); +struct drm_device *drm_dev = drm->dev;if (drm->client.device.info.family <NV_DEVICE_INFO_V0_TESLA && drm->agp.bridge)
@@ -254,6 +265,7 @@ nouveau_ttm_fini_gtt(struct nouveau_drm *drm)ttm_resource_manager_evict_all (&drm->ttm.bdev, man);ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&drm->ttm.bdev,TTM_PL_TT ,NULL ); +drm_lru_mgr_fini (&drm_dev->lru_mgr[TTM_PL_TT ]);kfree (man); } }
drivers/gpu/drm/ttm/ttm_range_manager.c +6 -0
@@ -36,6 +36,7 @@#include <drm/drm_mm.h> #include <linux/slab.h> #include <linux/spinlock.h> +#include <drm/drm_evictable_lru.h> /* * Currently we use a spinlock for the lock, but a mutex *may* be
@@ -199,6 +200,10 @@ int ttm_range_man_init_nocheck(struct drm_device *drm, struct ttm_device *bdev,ttm_set_driver_manager (bdev, type, &rman->manager);ttm_resource_manager_set_used (man, true); + +drm_lru_mgr_init (&drm->lru_mgr[type], p_size, &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[type]); +return 0 ; }EXPORT_SYMBOL (ttm_range_man_init_nocheck);
@@ -236,6 +241,7 @@ int ttm_range_man_fini_nocheck(struct drm_device *drm, struct ttm_device *bdev,ttm_resource_manager_cleanup (man);ttm_set_driver_manager (bdev, type,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[type]);kfree (rman);return 0 ; }
drivers/gpu/drm/ttm/ttm_sys_manager.c +2 -0
@@ -50,4 +50,6 @@ void ttm_sys_man_init(struct drm_device *drm, struct ttm_device *bdev)ttm_resource_manager_init (man, bdev,0 );ttm_set_driver_manager (bdev,TTM_PL_SYSTEM , man);ttm_resource_manager_set_used (man, true); +drm_lru_mgr_init (&drm->lru_mgr[TTM_PL_SYSTEM ],0 , &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[TTM_PL_SYSTEM ]); }
drivers/gpu/drm/vmwgfx/vmwgfx_system_manager.c +6 -0
@@ -62,6 +62,7 @@ int vmw_sys_man_init(struct vmw_private *dev_priv)struct ttm_device *bdev = &dev_priv->bdev;struct ttm_resource_manager *man =kzalloc (sizeof (*man),GFP_KERNEL ); +struct drm_device *drm = &dev_priv->drm;if (!man)return -ENOMEM ;
@@ -72,6 +73,9 @@ int vmw_sys_man_init(struct vmw_private *dev_priv)ttm_resource_manager_init (man, bdev,0 );ttm_set_driver_manager (bdev,VMW_PL_SYSTEM , man);ttm_resource_manager_set_used (man, true); + +drm_lru_mgr_init (&drm->lru_mgr[WMW_PL_SYSTEM ],0 , &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[VMW_PL_SYSTEM ]);return 0 ; }
@@ -79,6 +83,7 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv) {struct ttm_resource_manager *man =ttm_manager_type (&dev_priv->bdev,VMW_PL_SYSTEM ); +struct drm_device *drm = &dev_priv->drm;ttm_resource_manager_evict_all (&dev_priv->bdev, man);
@@ -86,5 +91,6 @@ void vmw_sys_man_fini(struct vmw_private *dev_priv)ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&dev_priv->bdev,VMW_PL_SYSTEM ,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[VMW_PL_SYSTEM ]);kfree (man); }
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +6 -0
@@ -96,11 +96,13 @@ static void ttm_sys_mgr_fini(struct drm_device *drm, void *arg)ttm_resource_manager_cleanup (man);ttm_set_driver_manager (&xe->ttm,XE_PL_TT ,NULL ); +drm_lru_mgr_fini (&drm->lru_mgr[XE_PL_TT ]); }int xe_ttm_sys_mgr_init (struct xe_device *xe) {struct ttm_resource_manager *man = &xe->mem.sys_mgr; +struct drm_device *drm = &xe->drm;struct sysinfo si;u64 gtt_size;
@@ -114,5 +116,9 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)ttm_resource_manager_init (man, &xe->ttm, gtt_size >>PAGE_SHIFT );ttm_set_driver_manager (&xe->ttm,XE_PL_TT , man);ttm_resource_manager_set_used (man, true); + +drm_lru_mgr_init (&drm->lru_mgr[XE_PL_TT ], gtt_size >>PAGE_SHIFT , &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[XE_PL_TT ]); +return drmm_add_action_or_reset (&xe->drm, ttm_sys_mgr_fini, xe); }
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c +6 -0
@@ -328,6 +328,8 @@ static void ttm_vram_mgr_fini(struct drm_device *dev, void *arg)ttm_resource_manager_cleanup (&mgr->manager);ttm_set_driver_manager (&xe->ttm, mgr->mem_type,NULL ); + +drm_lru_mgr_fini (&dev->lru_mgr[mgr->mem_type]); } int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
@@ -335,6 +337,7 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr, u64 default_page_size) {struct ttm_resource_manager *man = &mgr->manager; +struct drm_device *drm = &xe->drm;int err; man->func = &xe_ttm_vram_mgr_func;
@@ -350,6 +353,9 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,ttm_set_driver_manager (&xe->ttm, mem_type, &mgr->manager);ttm_resource_manager_set_used (&mgr->manager, true); +drm_lru_mgr_init (&drm->lru_mgr[mem_type], size, &drm->lru_lock); +ttm_resource_manager_set_lru_mgr (man, &drm->lru_mgr[mem_type]); +return drmm_add_action_or_reset (&xe->drm, ttm_vram_mgr_fini, mgr); }
Signature · 1 line
2.26.3
RFC 9/11 drm/ttm: Use drm LRU manager iterator
In reply to Oak ZengSince TTM resource LRU list is moved to drm LRU manager layer, use drm lru manager iterator instead of TTM resource manager iterator. TTM resource manager iterator is deleted. No function change.
--- drivers/gpu/drm/ttm/ttm_bo.c | 7 ++-- drivers/gpu/drm/ttm/ttm_device.c | 10 ++++-- drivers/gpu/drm/ttm/ttm_resource.c | 51 ------------------------------ include/drm/ttm/ttm_resource.h | 33 ++----------------- 4 files changed, 14 insertions(+), 87 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +5 -2
@@ -43,6 +43,7 @@#include <linux/module.h> #include <linux/atomic.h> #include <linux/dma-resv.h> +#include <drm/drm_evictable_lru.h> #include "ttm_module.h"
@@ -593,15 +594,17 @@ int ttm_mem_evict_first(struct ttm_device *bdev, struct ww_acquire_ctx *ticket) { struct ttm_buffer_object *bo = NULL, *busy_bo = NULL; - struct ttm_resource_cursor cursor; + struct drm_lru_cursor cursor; struct ttm_resource *res; + struct drm_lru_entity *entity; bool locked = false; int ret; spin_lock(bdev->lru_lock); - ttm_resource_manager_for_each_res(man, &cursor, res) { + drm_lru_for_each_entity(man->lru_mgr, &cursor, entity) { bool busy; + res = container_of(entity, struct ttm_resource, lru_entity); if (!ttm_bo_evict_swapout_allowable(res->bo, ctx, place, &locked, &busy)) { if (busy && !busy_bo && ticket !=
drivers/gpu/drm/ttm/ttm_device.c +7 -3
@@ -33,6 +33,7 @@#include <drm/ttm/ttm_device.h> #include <drm/ttm/ttm_tt.h> #include <drm/ttm/ttm_placement.h> +#include <drm/drm_evictable_lru.h> #include "ttm_module.h"
@@ -141,7 +142,8 @@ int ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags)int ttm_device_swapout (struct ttm_device *bdev,struct ttm_operation_ctx *ctx,gfp_t gfp_flags) { -struct ttm_resource_cursor cursor; +struct drm_lru_cursor cursor; +struct drm_lru_entity *entity;struct ttm_resource_manager *man;struct ttm_resource *res;unsigned i;
@@ -153,10 +155,12 @@ int ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx,if (!man || !man->use_tt)continue ; -ttm_resource_manager_for_each_res (man, &cursor, res) { -struct ttm_buffer_object *bo = res->bo; +drm_lru_for_each_entity (man->lru_mgr, &cursor, entity) { +struct ttm_buffer_object *bo;uint32_t num_pages; + res = container_of(entity, struct ttm_resource, lru_entity); + bo = res->bo;if (!bo || bo->resource != res)continue ;
drivers/gpu/drm/ttm/ttm_resource.c +0 -51
57 lines not on this page · raw message
include/drm/ttm/ttm_resource.h +2 -31
@@ -207,6 +207,7 @@ struct ttm_bus_placement {* @placement: Placement flags. * @bus: Placement on io bus accessible to the CPU * @bo: weak reference to the BO, protected by ttm_device::lru_lock +* @lru_entity: lru entity for this ttm resource. * * Structure indicating the placement and space resources used by a * buffer object.
@@ -223,17 +224,7 @@ struct ttm_resource {* @lru: Least recently used list, see &ttm_resource_manager.lru */ struct list_head lru; -}; - -/** -* struct ttm_resource_cursor -* -* @priority: the current priority -* -* Cursor to iterate over the resources in a manager. -*/ -struct ttm_resource_cursor { -unsigned int priority; +struct drm_lru_entity lru_entity; };/**
@@ -402,26 +393,6 @@ uint64_t ttm_resource_manager_usage(struct ttm_resource_manager *man);void ttm_resource_manager_debug (struct ttm_resource_manager *man,struct drm_printer *p); -struct ttm_resource * -ttm_resource_manager_first (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor); -struct ttm_resource * -ttm_resource_manager_next (struct ttm_resource_manager *man, -struct ttm_resource_cursor *cursor, -struct ttm_resource *res); - -/** -* ttm_resource_manager_for_each_res - iterate over all resources -* @man: the resource manager -* @cursor: struct ttm_resource_cursor for the current position -* @res: the current resource -* -* Iterate over all the evictable resources in a resource manager. -*/ -#define ttm_resource_manager_for_each_res (man, cursor, res) \ - for (res = ttm_resource_manager_first(man, cursor); res; \ - res = ttm_resource_manager_next(man, cursor, res)) -struct ttm_kmap_iter * ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io, struct io_mapping *iomap,
Signature · 1 line
2.26.3
RFC 10/11 drm/ttm: Implement ttm memory evict functions
In reply to Oak ZengImplement ttm_mem_evict_valuable, ttm_mem_evict_entity and ttm_mem_evict_busy_entity. Those are callback functions from drm lru manager. Register those functions during drm lru entity initialization. Those 3 functions are splitted from original ttm_mem_evict_first function.
Reimplemented ttm_mem_evict_first function using drm_lru_evict_first function. For now, drm_lru_evict_first just calls back to above 3 functions which are splitted from ttm_mem_evict_first function, so there is no function change. In the future, when SVM code is added, drm_lru_evict_first function can also calls into SVM resource eviction functions, thus TTM and SVM can mutually evict resources from each other.
--- drivers/gpu/drm/ttm/ttm_bo.c | 192 ++++++++++++++++++++++++++++------- include/drm/ttm/ttm_bo.h | 2 + 2 files changed, 158 insertions(+), 36 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +156 -36
@@ -587,50 +587,148 @@ static int ttm_mem_evict_wait_busy(struct ttm_buffer_object *busy_bo,return r == -EDEADLK ? -EBUSY : r; } -int ttm_mem_evict_first (struct ttm_device *bdev, -struct ttm_resource_manager *man, -const struct ttm_place *place, -struct ttm_operation_ctx *ctx, -struct ww_acquire_ctx *ticket) +struct ttm_mem_evict_ctx { +const struct ttm_place *place; +struct ttm_operation_ctx *ctx; +struct ww_acquire_ctx *ticket; +}; + +/** +* ttm_mem_evict_allowable +* +* @lru_entity: The struct ttm_resource::lru_entity when this resource is +* added to drm lru list. +* @place: The preferred ttm placement where we want to evict memory for +* more memory space. If the current ttm_resource doesn't match the preferred +* placement, then there is no need to evict the current resource. +* @ctx: ttm operation context +* @ticket: dma reservation's context used to lock resource +* @busy: used to return whether the current resource is busy (i.e., locked +* by other clients) +* @locked: used to return whether this resource is locked during this check, +* i.e., successfully trylocked bo's dma reservation object +* +* Check whether we are allowed to evict a memory resource. Return true if we +* are allowed to evict resource; otherwise false. +* +* When this function returns true, a resource reference counter (bo's reference) +* is hold. This reference counter need to be released after evict operation later +* on. +* +* This function should be called with lru_lock hold. +*/ +bool ttm_mem_evict_allowable (struct drm_lru_entity *lru_entity, +const struct drm_lru_evict_ctx *lru_evict_ctx, +bool *busy,bool *locked) { -struct ttm_buffer_object *bo =NULL , *busy_bo =NULL ; -struct drm_lru_cursor cursor;struct ttm_resource *res; -struct drm_lru_entity *entity; -bool locked = false; -int ret; +struct ttm_buffer_object *bo =NULL ; +struct ttm_device *bdev; +const struct ttm_place *place; +struct ttm_operation_ctx *ctx; +struct ww_acquire_ctx *ticket; +struct ttm_mem_evict_ctx *evict_ctx; -spin_lock (bdev->lru_lock); -drm_lru_for_each_entity (man->lru_mgr, &cursor, entity) { -bool busy; + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + place = evict_ctx->place; + ctx = evict_ctx->ctx; + ticket = evict_ctx->ticket; - res = container_of(entity, struct ttm_resource, lru_entity); -if (!ttm_bo_evict_swapout_allowable (res->bo, ctx, place, - &locked, &busy)) { -if (busy && !busy_bo && ticket != -dma_resv_locking_ctx (res->bo->base.resv)) - busy_bo = res->bo; -continue ; - } + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; -if (ttm_bo_get_unless_zero (res->bo)) { - bo = res->bo; -break ; - } -if (locked) -dma_resv_unlock (res->bo->base.resv); - } +if (!ttm_bo_evict_swapout_allowable (bo, ctx, place, locked, busy)) { +if (busy && ticket !=dma_resv_locking_ctx (bo->base.resv)) + *busy = true; -if (!bo) { -if (busy_bo && !ttm_bo_get_unless_zero (busy_bo)) - busy_bo =NULL ; -spin_unlock (bdev->lru_lock); - ret =ttm_mem_evict_wait_busy (busy_bo, ctx, ticket); -if (busy_bo) -ttm_bo_put (busy_bo); -return ret; +return false; } +if (ttm_bo_get_unless_zero (bo)) +return true; + +if (locked) +dma_resv_unlock (bo->base.resv); + +return false; +} + +/** +* ttm_mem_evict_busy_entity +* +* @lru_entity: The struct ttm_resource::lru_entity when this resource is +* added to drm lru list. +* @ctx: ttm operation context +* @ticket: dma reservation's context used to lock resource +* +* Evict a busy memory resource. +* This function should be called with lru_lock hold. +*/ +int ttm_mem_evict_busy_entity (struct drm_lru_entity *lru_entity, +const struct drm_lru_evict_ctx *lru_evict_ctx) +{ +struct ttm_resource *res; +struct ttm_buffer_object *bo =NULL ; +struct ttm_device *bdev; +int ret; +struct ttm_operation_ctx *ctx; +struct ww_acquire_ctx *ticket; +struct ttm_mem_evict_ctx *evict_ctx; + + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + ctx = evict_ctx->ctx; + ticket = evict_ctx->ticket; + + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; + +if (bo && !ttm_bo_get_unless_zero (bo)) + bo =NULL ; +spin_unlock (bdev->lru_lock); + ret =ttm_mem_evict_wait_busy (bo, ctx, ticket); +/* FIXME: this is code copied originally from ttm_mem_evict_first. +* 1) Shouldn't we ttm_bo_evict this bo also? Otherwise how can we +* make any memory space? +* 2) We also need to check whether this busy entity is in the same +* ttm_place as specified in lru_evict_ctx::place; if not, there is +* no help to wait this busy entity. +*/ +if (bo) +ttm_bo_put (bo); + +return ret; +} + +/** +* @lru_entity: The struct ttm_resource::lru_entity when this resource is +* added to drm lru list. +* @ctx: ttm operation context +* @locked: whether this resource is dma-reserved (if reserved, we need to +* unreserve it in this function) +* +* Evict a memory resource corresponding to a lru_entity. This should be +* called holding lru_lock +* +*/ +int ttm_mem_evict_entity (struct drm_lru_entity *lru_entity, +const struct drm_lru_evict_ctx *lru_evict_ctx,bool locked) +{ +struct ttm_resource *res; +struct ttm_buffer_object *bo =NULL ; +struct ttm_device *bdev; +int ret; +struct ttm_operation_ctx *ctx; +struct ttm_mem_evict_ctx *evict_ctx; + + evict_ctx = (struct ttm_mem_evict_ctx *)lru_evict_ctx; + ctx = evict_ctx->ctx; + + res = container_of(lru_entity, struct ttm_resource, lru_entity); + bo = res->bo; + bdev = bo->bdev; +if (bo->deleted) { ret =ttm_bo_cleanup_refs (bo, ctx->interruptible, ctx->no_wait_gpu, locked);
@@ -650,6 +748,28 @@ int ttm_mem_evict_first(struct ttm_device *bdev,return ret; } +struct drm_lru_evict_func ttm_evict_func = { + .evict_allowable = ttm_mem_evict_allowable, + .evict_busy_entity = ttm_mem_evict_busy_entity, + .evict_entity = ttm_mem_evict_entity +}; +EXPORT_SYMBOL (ttm_evict_func); + +int ttm_mem_evict_first (struct ttm_device *bdev, +struct ttm_resource_manager *man, +const struct ttm_place *place, +struct ttm_operation_ctx *ctx, +struct ww_acquire_ctx *ticket) +{ +struct drm_lru_evict_ctx evict_ctx = { + .data1 = place, + .data2 = ctx, + .data3 = ticket + }; + +return drm_lru_evict_first (man->lru_mgr, &evict_ctx); +} +/** * ttm_bo_pin - Pin the buffer object. * @bo: The buffer object to pin
include/drm/ttm/ttm_bo.h +2 -0
@@ -50,6 +50,7 @@ struct ttm_place;struct ttm_resource ;struct ttm_resource_manager ;struct ttm_tt ; +struct drm_lru_evict_func ;/** * enum ttm_bo_type
@@ -424,4 +425,5 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp);void ttm_bo_tt_destroy (struct ttm_buffer_object *bo); +extern struct drm_lru_evict_func ttm_evict_func;#endif
Signature · 1 line
2.26.3
RFC 11/11 drm/ttm: Write ttm functions using drm lru manager functions
In reply to Oak ZengReplace struct ttm_resource::lru with drm lru entity. Replace struct ttm_resource_manager::lru[] with drm lru manager. Remove ttm_lru_bulk_move functions and definitions as those are moved to drm lru manager.
Some of ttm resource, ttm bo and ttm device functions are re-written using drm lru manager functions.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 20 +-- drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +-- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 10 +- drivers/gpu/drm/ttm/ttm_range_manager.c | 2 +- drivers/gpu/drm/ttm/ttm_resource.c | 155 ++++---------------- drivers/gpu/drm/xe/xe_bo.c | 44 +++--- drivers/gpu/drm/xe/xe_bo.h | 3 +- drivers/gpu/drm/xe/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 6 +- drivers/gpu/drm/xe/xe_res_cursor.h | 10 +- drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 2 +- drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 12 +- drivers/gpu/drm/xe/xe_vm.c | 2 +- drivers/gpu/drm/xe/xe_vm_types.h | 2 +- include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_resource.h | 59 ++------ 21 files changed, 112 insertions(+), 253 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -1
@@ -369,7 +369,7 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev, struct amdgpu_vm *vm) {spin_lock (adev->mman.bdev.lru_lock); -ttm_lru_bulk_move_tail (&vm->lru_bulk_move); +drm_lru_bulk_move_tail (&vm->lru_bulk_move);spin_unlock (adev->mman.bdev.lru_lock); }
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +1 -1
@@ -337,7 +337,7 @@ struct amdgpu_vm {struct amdgpu_task_info task_info;/* Store positions of group of BOs */ -struct ttm_lru_bulk_move lru_bulk_move; +struct drm_lru_bulk_move lru_bulk_move;/* Flag to indicate if VM is used for compute */ bool is_compute_context;
drivers/gpu/drm/ttm/tests/ttm_device_test.c +1 -1
@@ -90,7 +90,7 @@ static void ttm_device_fini_basic(struct kunit *test)ttm_device_fini (ttm_dev);KUNIT_ASSERT_FALSE (test, man->use_type); -KUNIT_ASSERT_TRUE (test,list_empty (&man->lru[0 ])); +KUNIT_ASSERT_TRUE (test,list_empty (&man->lru_mgr->lru[0 ]));KUNIT_ASSERT_NULL (test, ttm_dev->man_drv[TTM_PL_SYSTEM ]); }
drivers/gpu/drm/ttm/ttm_bo.c +10 -10
@@ -92,11 +92,11 @@ EXPORT_SYMBOL(ttm_bo_move_to_lru_tail);* resulting in much less overhead of maintaining the LRU. * The only requirement is that the resources stay together on the LRU and are * never separated. This is enforces by setting the bulk_move structure on a BO. -* ttm_lru_bulk_move_tail() should be used to move all resources to the tail of +* drm_lru_bulk_move_tail() should be used to move all resources to the tail of * their LRU list. */ void ttm_bo_set_bulk_move (struct ttm_buffer_object *bo, -struct ttm_lru_bulk_move *bulk) +struct drm_lru_bulk_move *bulk) {dma_resv_assert_held (bo->base.resv);
@@ -122,8 +122,8 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,bool old_use_tt, new_use_tt;int ret; - old_use_tt = !bo->resource ||ttm_manager_type (bdev, bo->resource->mem_type)->use_tt; - new_use_tt =ttm_manager_type (bdev, mem->mem_type)->use_tt; + old_use_tt = !bo->resource ||ttm_manager_type (bdev, bo->resource->lru_entity.mem_type)->use_tt; + new_use_tt =ttm_manager_type (bdev, mem->lru_entity.mem_type)->use_tt;ttm_bo_unmap_virtual (bo);
@@ -139,7 +139,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,if (ret) goto out_err; -if (mem->mem_type !=TTM_PL_SYSTEM ) { +if (mem->lru_entity.mem_type !=TTM_PL_SYSTEM ) { ret =ttm_tt_populate (bo->bdev, bo->ttm, ctx);if (ret) goto out_err;
@@ -492,7 +492,7 @@ bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,struct ttm_device *bdev = bo->bdev;dma_resv_assert_held (bo->base.resv); -if (bo->resource->mem_type ==TTM_PL_SYSTEM ) +if (bo->resource->lru_entity.mem_type ==TTM_PL_SYSTEM )return true;/* Don't evict this BO if it's outside of the
@@ -540,7 +540,7 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo, *busy = !ret; } -if (ret && place && (bo->resource->mem_type != place->mem_type || +if (ret && place && (bo->resource->lru_entity.mem_type != place->mem_type || !bo->bdev->funcs->eviction_valuable (bo, place))) { ret = false;if (*locked) {
@@ -1039,7 +1039,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,/* * We might need to add a TTM. */ -if (!bo->resource || bo->resource->mem_type ==TTM_PL_SYSTEM ) { +if (!bo->resource || bo->resource->lru_entity.mem_type ==TTM_PL_SYSTEM ) { ret =ttm_tt_create (bo, true);if (ret)return ret;
@@ -1259,7 +1259,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,* as an indication that we're about to swap out. */ memset (&place,0 ,sizeof (place)); - place.mem_type = bo->resource->mem_type; + place.mem_type = bo->resource->lru_entity.mem_type;if (!ttm_bo_evict_swapout_allowable (bo, ctx, &place, &locked,NULL ))return -EBUSY ;
@@ -1284,7 +1284,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,/* * Move to system cached */ -if (bo->resource->mem_type !=TTM_PL_SYSTEM ) { +if (bo->resource->lru_entity.mem_type !=TTM_PL_SYSTEM ) {struct ttm_resource *evict_mem;struct ttm_place hop;
drivers/gpu/drm/ttm/ttm_bo_util.c +10 -10
@@ -145,7 +145,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, {struct ttm_device *bdev = bo->bdev;struct ttm_resource_manager *dst_man = -ttm_manager_type (bo->bdev, dst_mem->mem_type); +ttm_manager_type (bo->bdev, dst_mem->lru_entity.mem_type);struct ttm_tt *ttm = bo->ttm;struct ttm_resource *src_mem = bo->resource;struct ttm_resource_manager *src_man;
@@ -160,7 +160,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,if (WARN_ON (!src_mem))return -EINVAL ; - src_man =ttm_manager_type (bdev, src_mem->mem_type); + src_man =ttm_manager_type (bdev, src_mem->lru_entity.mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { ret =ttm_tt_populate (bdev, ttm, ctx);
@@ -184,7 +184,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, clear = src_iter->ops->maps_tt && (!ttm || !ttm_tt_is_populated (ttm));if (!(clear && ttm && !(ttm->page_flags &TTM_TT_FLAG_ZERO_ALLOC ))) -ttm_move_memcpy (clear,PFN_UP (dst_mem->size), dst_iter, src_iter); +ttm_move_memcpy (clear,PFN_UP (dst_mem->lru_entity.size), dst_iter, src_iter);if (!src_iter->ops->maps_tt)ttm_kmap_iter_linear_io_fini (&_src_iter.io, bdev, src_mem);
@@ -293,7 +293,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,struct ttm_resource_manager *man;enum ttm_caching caching; - man =ttm_manager_type (bo->bdev, res->mem_type); + man =ttm_manager_type (bo->bdev, res->lru_entity.mem_type); caching = man->use_tt ? bo->ttm->caching : res->bus.caching;return ttm_prot_from_caching (caching, tmp);
@@ -393,9 +393,9 @@ int ttm_bo_kmap(struct ttm_buffer_object *bo, map->virtual =NULL ; map->bo = bo; -if (num_pages >PFN_UP (bo->resource->size)) +if (num_pages >PFN_UP (bo->resource->lru_entity.size))return -EINVAL ; -if ((start_page + num_pages) >PFN_UP (bo->resource->size)) +if ((start_page + num_pages) >PFN_UP (bo->resource->lru_entity.size))return -EINVAL ; ret =ttm_mem_io_reserve (bo->bdev, bo->resource);
@@ -607,7 +607,7 @@ static void ttm_bo_move_pipeline_evict(struct ttm_buffer_object *bo,struct ttm_device *bdev = bo->bdev;struct ttm_resource_manager *from; - from =ttm_manager_type (bdev, bo->resource->mem_type); + from =ttm_manager_type (bdev, bo->resource->lru_entity.mem_type);/** * BO doesn't have a TTM we need to bind/unbind. Just remember
@@ -646,8 +646,8 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo, struct ttm_resource *new_mem) {struct ttm_device *bdev = bo->bdev; -struct ttm_resource_manager *from =ttm_manager_type (bdev, bo->resource->mem_type); -struct ttm_resource_manager *man =ttm_manager_type (bdev, new_mem->mem_type); +struct ttm_resource_manager *from =ttm_manager_type (bdev, bo->resource->lru_entity.mem_type); +struct ttm_resource_manager *man =ttm_manager_type (bdev, new_mem->lru_entity.mem_type);int ret =0 ;dma_resv_add_fence (bo->base.resv, fence,DMA_RESV_USAGE_KERNEL );
@@ -680,7 +680,7 @@ void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo, struct ttm_resource *new_mem) {struct ttm_device *bdev = bo->bdev; -struct ttm_resource_manager *man =ttm_manager_type (bdev, new_mem->mem_type); +struct ttm_resource_manager *man =ttm_manager_type (bdev, new_mem->lru_entity.mem_type);int ret; ret =ttm_bo_wait_free_node (bo, man->use_tt);
drivers/gpu/drm/ttm/ttm_bo_vm.c +1 -1
@@ -421,7 +421,7 @@ int ttm_bo_vm_access(struct vm_area_struct *vma, unsigned long addr,if (ret)return ret; -switch (bo->resource->mem_type) { +switch (bo->resource->lru_entity.mem_type) {case TTM_PL_SYSTEM : fallthrough;case TTM_PL_TT :
drivers/gpu/drm/ttm/ttm_device.c +6 -4
@@ -250,7 +250,7 @@ void ttm_device_fini(struct ttm_device *bdev)spin_lock (bdev->lru_lock);for (i =0 ; i <DRM_MAX_LRU_PRIORITY ; ++i) -if (list_empty (&man->lru[0 ])) +if (list_empty (&man->lru_mgr->lru[0 ]))pr_debug ("Swap list %d was clean\n" , i);spin_unlock (bdev->lru_lock);
@@ -263,16 +263,18 @@ static void ttm_device_clear_lru_dma_mappings(struct ttm_device *bdev, struct list_head *list) {struct ttm_resource *res; +struct drm_lru_entity *entity;spin_lock (bdev->lru_lock); -while ((res =list_first_entry_or_null (list,typeof (*res), lru))) { +while ((entity =list_first_entry_or_null (list,typeof (*entity), lru))){ + res = container_of(entity, struct ttm_resource, lru_entity);struct ttm_buffer_object *bo = res->bo;/* Take ref against racing releases once lru_lock is unlocked */ if (!ttm_bo_get_unless_zero (bo))continue ; -list_del_init (&res->lru); +list_del_init (&entity->lru);spin_unlock (bdev->lru_lock);if (bo->ttm)
@@ -297,7 +299,7 @@ void ttm_device_clear_dma_mappings(struct ttm_device *bdev)continue ;for (j =0 ; j <DRM_MAX_LRU_PRIORITY ; ++j) -ttm_device_clear_lru_dma_mappings (bdev, &man->lru[j]); +ttm_device_clear_lru_dma_mappings (bdev, &man->lru_mgr->lru[j]); } }EXPORT_SYMBOL (ttm_device_clear_dma_mappings);
drivers/gpu/drm/ttm/ttm_range_manager.c +1 -1
@@ -84,7 +84,7 @@ static int ttm_range_man_alloc(struct ttm_resource_manager *man,spin_lock (&rman->lock); ret =drm_mm_insert_node_in_range (mm, &node->mm_nodes[0 ], -PFN_UP (node->base.size), +PFN_UP (node->base.lru_entity.size), bo->page_alignment,0 , place->fpfn, lpfn, mode);spin_unlock (&rman->lock);
drivers/gpu/drm/ttm/ttm_resource.c +25 -130
257 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.c +22 -22
161 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.h +2 -1
@@ -9,6 +9,7 @@#include "xe_bo_types.h" #include "xe_macros.h" #include "xe_vm_types.h" +#include <drm/drm_evictable_lru.h> #define XE_DEFAULT_GTT_SIZE_MB 3072ULL/* 3GB by default */
@@ -83,7 +84,7 @@ void xe_bo_free(struct xe_bo *bo);struct xe_bo *__xe_bo_create_locked (struct xe_device *xe,struct xe_bo *bo,struct xe_tile *tile,struct dma_resv *resv, -struct ttm_lru_bulk_move *bulk,size_t size, +struct drm_lru_bulk_move *bulk,size_t size,enum ttm_bo_type type,u32 flags);struct xe_bo * xe_bo_create_locked_range(struct xe_device *xe,
drivers/gpu/drm/xe/xe_dma_buf.c +2 -2
@@ -82,7 +82,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,if (!xe_bo_is_pinned (bo)) {if (!attach->peer2peer || - bo->ttm.resource->mem_type ==XE_PL_SYSTEM ) { + bo->ttm.resource->lru_entity.mem_type ==XE_PL_SYSTEM ) {if (xe_bo_can_migrate (bo,XE_PL_TT )) r =xe_bo_migrate (bo,XE_PL_TT );else
@@ -92,7 +92,7 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,return ERR_PTR (r); } -switch (bo->ttm.resource->mem_type) { +switch (bo->ttm.resource->lru_entity.mem_type) {case XE_PL_TT : sgt = drm_prime_pages_to_sg(obj->dev, bo->ttm.ttm->pages,
drivers/gpu/drm/xe/xe_exec.c +1 -1
@@ -371,7 +371,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)if (!err && !xe_vm_no_dma_fences (vm)) {spin_lock (xe->ttm.lru_lock); -ttm_lru_bulk_move_tail (&vm->lru_bulk_move); +drm_lru_bulk_move_tail (&vm->lru_bulk_move);spin_unlock (xe->ttm.lru_lock); }
drivers/gpu/drm/xe/xe_migrate.c +3 -3
@@ -643,8 +643,8 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,u64 src_L0, dst_L0;int pass =0 ;int err; -bool src_is_vram =mem_type_is_vram (src->mem_type); -bool dst_is_vram =mem_type_is_vram (dst->mem_type); +bool src_is_vram =mem_type_is_vram (src->lru_entity.mem_type); +bool dst_is_vram =mem_type_is_vram (dst->lru_entity.mem_type);bool copy_ccs =xe_device_has_flat_ccs (xe) &&xe_bo_needs_ccs_pages (src_bo) &&xe_bo_needs_ccs_pages (dst_bo);bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
@@ -895,7 +895,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,struct xe_bo *bo, struct ttm_resource *dst) { - bool clear_vram = mem_type_is_vram(dst->mem_type); + bool clear_vram = mem_type_is_vram(dst->lru_entity.mem_type);struct xe_gt *gt = m->tile->primary_gt;struct xe_device *xe =gt_to_xe (gt);struct dma_fence *fence =NULL ;
drivers/gpu/drm/xe/xe_res_cursor.h +5 -5
@@ -53,8 +53,8 @@ static struct drm_buddy *xe_res_get_buddy(struct ttm_resource *res)struct xe_device *xe =ttm_to_xe_device (res->bo->bdev);struct ttm_resource_manager *mgr; -if (res->mem_type !=XE_PL_STOLEN ) -return &xe->tiles[res->mem_type -XE_PL_VRAM0 ].mem.vram_mgr->mm; +if (res->lru_entity.mem_type !=XE_PL_STOLEN ) +return &xe->tiles[res->lru_entity.mem_type -XE_PL_VRAM0 ].mem.vram_mgr->mm; mgr =ttm_manager_type (&xe->ttm,XE_PL_STOLEN );
@@ -79,9 +79,9 @@ static inline void xe_res_first(struct ttm_resource *res,if (!res) goto fallback; -XE_WARN_ON (start + size > res->size); +XE_WARN_ON (start + size > res->lru_entity.size); - cur->mem_type = res->mem_type; + cur->mem_type = res->lru_entity.mem_type;switch (cur->mem_type) {case XE_PL_STOLEN :
@@ -128,7 +128,7 @@ static inline void xe_res_first(struct ttm_resource *res, cur->remaining = size; cur->node =NULL ; cur->mem_type =XE_PL_TT ; -XE_WARN_ON (res && start + size > res->size); +XE_WARN_ON (res && start + size > res->lru_entity.size); }static inline void __xe_res_sg_next (struct xe_res_cursor *cur)
drivers/gpu/drm/xe/xe_ttm_sys_mgr.c +1 -1
@@ -48,7 +48,7 @@ static int xe_ttm_sys_mgr_new(struct ttm_resource_manager *man, } node->base.mm_nodes[0 ].start =0 ; - node->base.mm_nodes[0 ].size =PFN_UP (node->base.base.size); + node->base.mm_nodes[0 ].size =PFN_UP (node->base.base.lru_entity.size); node->base.base.start =XE_BO_INVALID_OFFSET ; *res = &node->base.base;
drivers/gpu/drm/xe/xe_ttm_vram_mgr.c +6 -6
@@ -83,11 +83,11 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,if (place->fpfn || lpfn != man->size >>PAGE_SHIFT ) vres->flags |=DRM_BUDDY_RANGE_ALLOCATION ; -if (WARN_ON (!vres->base.size)) { +if (WARN_ON (!vres->base.lru_entity.size)) { err = -EINVAL ; goto error_fini; } - size = vres->base.size; + size = vres->base.lru_entity.size; min_page_size = mgr->default_page_size;if (tbo->page_alignment)
@@ -150,8 +150,8 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man, }while (remaining_size);if (place->flags &TTM_PL_FLAG_CONTIGUOUS ) { -if (!drm_buddy_block_trim (mm, vres->base.size, &vres->blocks)) - size = vres->base.size; +if (!drm_buddy_block_trim (mm, vres->base.lru_entity.size, &vres->blocks)) + size = vres->base.lru_entity.size; }if (lpfn <= mgr->visible_size >>PAGE_SHIFT ) {
@@ -378,14 +378,14 @@ int xe_ttm_vram_mgr_alloc_sgt(struct xe_device *xe,enum dma_data_direction dir, struct sg_table **sgt) { - struct xe_tile *tile = &xe->tiles[res->mem_type - XE_PL_VRAM0]; + struct xe_tile *tile = &xe->tiles[res->lru_entity.mem_type - XE_PL_VRAM0];struct xe_ttm_vram_mgr_resource *vres =to_xe_ttm_vram_mgr_resource (res);struct xe_res_cursor cursor;struct scatterlist *sg;int num_entries =0 ;int i, r; -if (vres->used_visible_size < res->size) +if (vres->used_visible_size < res->lru_entity.size)return -EOPNOTSUPP ; *sgt =kmalloc (sizeof (**sgt),GFP_KERNEL );
drivers/gpu/drm/xe/xe_vm.c +1 -1
@@ -652,7 +652,7 @@ static void preempt_rebind_work_func(struct work_struct *w)#undef retry_requiredspin_lock (vm->xe->ttm.lru_lock); -ttm_lru_bulk_move_tail (&vm->lru_bulk_move); +drm_lru_bulk_move_tail (&vm->lru_bulk_move);spin_unlock (vm->xe->ttm.lru_lock);/* Point of no return. */
drivers/gpu/drm/xe/xe_vm_types.h +1 -1
@@ -149,7 +149,7 @@ struct xe_vm {struct dma_resv resv;/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ -struct ttm_lru_bulk_move lru_bulk_move; +struct drm_lru_bulk_move lru_bulk_move;u64 size;
include/drm/ttm/ttm_bo.h +2 -2
@@ -118,7 +118,7 @@ struct ttm_buffer_object {struct ttm_resource *resource;struct ttm_tt *ttm;bool deleted; -struct ttm_lru_bulk_move *bulk_move; +struct drm_lru_bulk_move *bulk_move;unsigned priority;unsigned pin_count;
@@ -355,7 +355,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx);void ttm_bo_put (struct ttm_buffer_object *bo);void ttm_bo_set_bulk_move (struct ttm_buffer_object *bo, -struct ttm_lru_bulk_move *bulk); +struct drm_lru_bulk_move *bulk);bool ttm_bo_eviction_valuable (struct ttm_buffer_object *bo,const struct ttm_place *place); int ttm_bo_init_reserved(struct ttm_device *bdev, struct ttm_buffer_object *bo,
include/drm/ttm/ttm_resource.h +10 -49
115 lines not on this page · raw message
Signature · 1 line
2.26.3
Patchwork
CI[Intel-xe] ✗ CI.Patch_applied: failure for Introduce drm evictable lru
In reply to Oak Zeng== Series Details ==
== Series Details ==
Series: Introduce drm evictable lru
URL : https://patchwork.freedesktop.org/series/125879/
State : failure
== Summary ==
=== Applying kernel patches on branch 'drm-xe-next' with base: === Base commit: ff616f1d0 drm/xe/xe2: Program correct MOCS registers === git am output follows ===
error: patch failed: drivers/gpu/drm/xe/xe_bo.c:946
error: drivers/gpu/drm/xe/xe_bo.c: patch does not apply
error: patch failed: include/drm/drm_device.h:326
error: include/drm/drm_device.h: patch does not apply
hint: Use 'git am --show-current-patch' to see the failed patch
Applying: drm/ttm: re-parameter ttm_device_init
Applying: drm: move lru_lock from ttm_device to drm_device Patch failed at 0002 drm: move lru_lock from ttm_device to drm_device When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Thomas Hellström
maintainerHi Oak, Christian
Quoted 125 lines · Oak Zeng
We plan to implement xe driver's shared virtual memory manager (aka SVM) without buffer object concept. This means we won't build our shared virtual memory manager upon TTM infrastructure like amdgpu does.
Even though this approach is more efficient, it does create a problem for memory eviction when there is memory pressure: memory allocated by SVM and memory allocated by TTM should be able to mutually evict from each other. TTM's resource manager maintains a LRU list for each memory type and this list is used to pick up the memory eviction victim. Since we don't use TTM for SVM implementation, SVM allocated memory can't be added to TTM resource manager's LRU list. Thus SVM allocated memory and TTM allocated memory are not mutually evictable.
See more discussion on this topic here:
https://www.spinics.net/lists/dri-devel/msg410740.htmlThis series solve this problem by creating a shared LRU list b/t SVM and TTM, or any other resource manager.
The basic idea is, abstract a
drm_lru_entitystructure which is supposed to be embedded inttm_resourcestructure, or any other resource manager. The resource LRU list is a list ofdrm_lru_entity.drm_lru_entityhas eviction function pointers which can be used to call back drivers' specific eviction function to evict a memory resource.Introduce global
drm_lru_managerto structdrm_deviceto manage LRU lists. Each memory type or memory region can have a LRU list. TTM resource manager's LRU list functions including bulk move functions are moved to drm lru manager. drm lru manager provides aevict_firstfunction to evict the first memory resource from LRU list. This function can be called from TTM, SVM or any other resource manager, so all the memory allocated in the drm sub-system can be mutually evicted.The
lru_lockis also moved from structttm_deviceto structdrm_device.Opens:
1) memory accounting: currently the ttm resource manager's memory accounting functions is kept at ttm resource manager. Since memory accounting should be cross TTM and SVM, it should be ideally in the drm lru manager layer. This will be polished in the future.2) eviction callback function interface: The current eviction function interface is designed to meet TTM memory eviction requirements. When SVM is in the picture, this interface need to be futher tunned to meet SVM requirement also.
This series is not tested and it is only compiled for xe driver. Some minor changes are needed for other driver such as amdgpu, nouveau etc. I intended to send this out as a request for comment series to get some early feedback, to see whether this is the right direction to go. I will futher polish this series after a direction is agreed.
Oak Zeng (11):
drm/ttm: re-parameterttm_device_init
drm: movelru_lockfromttm_devicetodrm_device
drm: introduce drm evictable LRU
drm: Add evict function pointer to drm lru entity
drm: Replace ttm macros with drm macros
drm/ttm: Set lru manager to ttm resource manager drm/ttm: re-parameterize a few ttm functions
drm: Initialize drm lru manager
drm/ttm: Use drm LRU manager iterator
drm/ttm: Implement ttm memory evict functions drm/ttm: Write ttm functions using drm lru manager functionsdrivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 + .../gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c | 6 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 10 +- drivers/gpu/drm/drm_drv.c | 1 + drivers/gpu/drm/drm_evictable_lru.c | 266 ++++++++++++++++++ drivers/gpu/drm/drm_gem_vram_helper.c | 10 +- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 6 +- drivers/gpu/drm/i915/i915_ttm_buddy_manager.c | 10 + drivers/gpu/drm/i915/intel_region_ttm.c | 4 +- drivers/gpu/drm/i915/selftests/mock_region.c | 2 +- drivers/gpu/drm/loongson/lsdc_ttm.c | 10 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 22 +- drivers/gpu/drm/qxl/qxl_ttm.c | 6 +- drivers/gpu/drm/radeon/radeon_ttm.c | 10 +- drivers/gpu/drm/ttm/tests/ttm_device_test.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_kunit_helpers.c | 2 +- drivers/gpu/drm/ttm/ttm_bo.c | 247 ++++++++++++---- drivers/gpu/drm/ttm/ttm_bo_util.c | 20 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/ttm/ttm_device.c | 55 ++-- drivers/gpu/drm/ttm/ttm_module.h | 3 +- drivers/gpu/drm/ttm/ttm_range_manager.c | 14 +- drivers/gpu/drm/ttm/ttm_resource.c | 242 +++------------- drivers/gpu/drm/ttm/ttm_sys_manager.c | 8 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_bo.h | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 +- .../gpu/drm/vmwgfx/vmwgfx_system_manager.c | 6 + drivers/gpu/drm/xe/xe_bo.c | 48 ++-- drivers/gpu/drm/xe/xe_bo.h | 5 +- drivers/gpu/drm/xe/xe_device.c | 2 +- drivers/gpu/drm/xe/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/xe_exec.c | 6 +- drivers/gpu/drm/xe/xe_migrate.c | 6 +- drivers/gpu/drm/xe/xe_res_cursor.h | 10 +- drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 8 +- drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 18 +- drivers/gpu/drm/xe/xe_vm.c | 6 +- drivers/gpu/drm/xe/xe_vm_types.h | 2 +- include/drm/drm_device.h | 12 + include/drm/drm_evictable_lru.h | 260 +++++++++++++++++ include/drm/ttm/ttm_bo.h | 10 +- include/drm/ttm/ttm_device.h | 13 +- include/drm/ttm/ttm_range_manager.h | 17 +- include/drm/ttm/ttm_resource.h | 117 +++----- 49 files changed, 1042 insertions(+), 501 deletions(-)
Quoted 3 lines
create mode 100644
drivers/gpu/drm/drm_evictable_lru.ccreate mode 100644include/drm/drm_evictable_lru.h
Some additional comments after looking through this patchset and the comments.
1) General: IMO a good start.
2) Where should this reside? I'm not a big fan of *prescribing* that a component should be part of a specific device structure. IMO that leads to dumping the whole drm namespace into this component rather than to keep it isolated with as few dependencies as possible. I would make the part that should reside in the drm device a "struct drm_lru_device", and the resource base class a "struct drm_lru_resource". Whether the drm device then wants to embed the struct drm_lru_device (nice if you need this component) or have a pointer to it (nice if you don't need this component) and whether the ttm_device should subclass the drm device becomes pretty orthogonal to the actual implementation, and we'd avoid hard to follow code cross-references all over the place. The drm_gpuvm IMO did a good job with this, but I know there are other opinions on this, and I don't want it to become a blocker.
3) Christian's comment about cursors into the LRU for LRU traversal restart
This is quickly becoming a must for the Xe driver, and a very-nice to have for a working TTM shrinker, but I think we should start to bring what we have in TTM currently over and do the cursors as a separate task. I have it pretty much on top of my priority list currently. Both downstream i915 and drm_gpuvm have a way to handle this nicely by permutating the LRU list just before the lock needs to be released, so I was thinking something similar. The complication would be the bulk move tracking.
4) The struct drm_lru_evict_ctx - This one needs to be common across all users since it's set up by the evictor and used by the evictee ops.
/Thomas