drm/ttm/memcg/lru: enable memcg tracking for ttm, xe and amdgpu driver (part 2) (v2).
Subsystems (14)
Dave Airlie
maintainerThis is just a repost with a number of sashiko identified problems that I fixed.
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting this landed for xe, we can drop the amdgpu changes for now if they can't get across the line.
I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align with the two global vmstats. It adds an accounting flag to TTM alloc/populate, and enables memcg tracking and shrinker support in TTM.
Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail to fully understand beyond it doesn't solve all the problems we ever have had with cgroups and drm, so we shouldn't even bother, and maybe we could do it at the object level, and integrated with dmem, and android cross process accounting, but I still feel this is a good baseline.
I think this is the right layer to hook this into TTM, where we allocate memory and I think accounting for this memory in a proper way should be done.
Intel folks (Thomas/Maarten) please review and express concerns as well.
Regards,
Dave.
Dave Airlie
maintainerPATCH 1/10 memcg: add support for GPU page counters. (v5)
From: Dave Airlie <[email protected]>
This introduces 2 new statistics and 3 new memcontrol APIs for dealing with GPU system memory allocations.
The stats corresponds to the same stats in the global vmstat, for number of active GPU pages, and number of pages in pools that can be reclaimed.
The first API charges a order of pages to a objcg, and sets the objcg on the pages like kmem does, and updates the active/reclaim statistic.
The second API uncharges a page from the obj cgroup it is currently charged to.
The third API allows moving a page to/from reclaim and between obj cgroups. When pages are added to the pool lru, this just updates accounting. When pages are being removed from a pool lru, they can be taken from the parent objcg so this allows them to be uncharged from there and transferred to a new child objcg.
---
v2: use memcg_node_stat_items
v3: fix null ptr dereference in uncharge
v4: AI review: fix parameter names, fix problem with reclaim moving doing wrong thing
v5: fix the build with CONFIG_MEMCG=n
--- Documentation/admin-guide/cgroup-v2.rst | 6 ++ include/linux/memcontrol.h | 34 ++++++++ mm/memcontrol.c | 104 ++++++++++++++++++++++++ 3 files changed, 144 insertions(+)
Documentation/admin-guide/cgroup-v2.rst +6 -0
@@ -1573,6 +1573,12 @@ The following nested keys are defined. vmalloc (npn) Amount of memory used for vmap backed memory. + gpu_active (npn) + Amount of system memory used for GPU devices. + + gpu_reclaim (npn) + Amount of system memory cached for GPU devices. + shmem Amount of cached filesystem data that is swap-backed, such as tmpfs, shm segments, shared anonymous mmap()s
include/linux/memcontrol.h +34 -0
@@ -1583,6 +1583,7 @@ static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb)#endif /* CONFIG_CGROUP_WRITEBACK */ struct sock ; +#ifdef CONFIG_MEMCG extern struct static_key_false memcg_sockets_enabled_key;#define mem_cgroup_sockets_enabled static_branch_unlikely(&memcg_sockets_enabled_key)
@@ -1629,6 +1630,17 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg) }#endif +bool mem_cgroup_charge_gpu_page (struct obj_cgroup *objcg,struct page *page, +unsigned int order, +gfp_t gfp_mask,bool reclaim); +void mem_cgroup_uncharge_gpu_page (struct page *page, +unsigned int order, +bool reclaim); +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *objcg, +struct page *page, +unsigned int order, +bool to_reclaim); +int alloc_shrinker_info (struct mem_cgroup *memcg);void free_shrinker_info (struct mem_cgroup *memcg);void set_shrinker_bit (struct mem_cgroup *memcg,int nid,int shrinker_id);
@@ -1665,6 +1677,28 @@ static inline void mem_cgroup_sk_uncharge(const struct sock *sk, { } +static inline bool mem_cgroup_charge_gpu_page (struct obj_cgroup *objcg, +struct page *page, +unsigned int order, +gfp_t gfp_mask,bool reclaim) +{ +return true; +} + +static inline void mem_cgroup_uncharge_gpu_page (struct page *page, +unsigned int order, +bool reclaim) +{ +} + +static inline bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *objcg, +struct page *page, +unsigned int order, +bool to_reclaim) +{ +return true; +} +static inline void set_shrinker_bit (struct mem_cgroup *memcg,int nid,int shrinker_id) {
mm/memcontrol.c +104 -0
122 lines not on this page · raw message
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 2/10 ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This flag does nothing yet, but this just changes the APIs to accept it in the future across all users.
This flag will eventually be filled out with when to account a tt populate to a memcg.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 ++- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 5 +++-- drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c | 4 ++-- drivers/gpu/drm/loongson/lsdc_ttm.c | 3 ++- drivers/gpu/drm/nouveau/nouveau_bo.c | 6 ++++-- drivers/gpu/drm/radeon/radeon_ttm.c | 3 ++- drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_pool_test.c | 16 ++++++++-------- drivers/gpu/drm/ttm/tests/ttm_tt_test.c | 12 ++++++------ drivers/gpu/drm/ttm/ttm_bo.c | 7 ++++--- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +++- drivers/gpu/drm/ttm/ttm_pool.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_tt.c | 8 +++++--- drivers/gpu/drm/vmwgfx/vmwgfx_blit.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c | 7 ++++--- drivers/gpu/drm/xe/xe_bo.c | 5 +++-- include/drm/ttm/ttm_bo.h | 1 + include/drm/ttm/ttm_device.h | 1 + include/drm/ttm/ttm_pool.h | 1 + include/drm/ttm/ttm_tt.h | 1 + 22 files changed, 63 insertions(+), 44 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -1
@@ -1220,6 +1220,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,*/ static int amdgpu_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct amdgpu_device *adev =amdgpu_ttm_adev (bdev);
@@ -1243,7 +1244,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev, pool = &adev->mman.ttm_pools[gtt->pool_id]; else pool = &adev->mman.bdev.pool; - ret =ttm_pool_alloc (pool, ttm, ctx); + ret =ttm_pool_alloc (pool, ttm, memcg_account, ctx);if (ret)return ret;
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +3 -2
@@ -317,6 +317,7 @@ static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,static int i915_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct i915_ttm_tt *i915_tt =container_of (ttm,typeof (*i915_tt), ttm);
@@ -324,7 +325,7 @@ static int i915_ttm_tt_populate(struct ttm_device *bdev,if (i915_tt->is_shmem)return i915_ttm_tt_shmem_populate (bdev, ttm, ctx); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); }static void i915_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)
@@ -815,7 +816,7 @@ static int __i915_ttm_get_pages(struct drm_i915_gem_object *obj, }if (bo->ttm && !ttm_tt_is_populated (bo->ttm)) { - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +1 -1
@@ -624,7 +624,7 @@ int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,/* Populate ttm with pages if needed. Typically system memory. */ if (ttm && (dst_man->use_tt || (ttm->page_flags &TTM_TT_FLAG_SWAPPED ))) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }
drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +2 -2
@@ -91,7 +91,7 @@ static int i915_ttm_backup(struct i915_gem_apply_to_region *apply, goto out_no_lock; backup_bo =i915_gem_to_ttm (backup); - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (err) goto out_no_populate;
@@ -190,7 +190,7 @@ static int i915_ttm_restore(struct i915_gem_apply_to_region *apply,if (!backup_bo->resource) err =ttm_bo_validate (backup_bo,i915_ttm_sys_placement (), &ctx);if (!err) - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (!err) { err =i915_gem_obj_copy_ttm (obj, backup, pm_apply->allow_gpu, false);
drivers/gpu/drm/loongson/lsdc_ttm.c +2 -1
@@ -111,6 +111,7 @@ lsdc_ttm_tt_create(struct ttm_buffer_object *tbo, uint32_t page_flags)static int lsdc_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {bool slave = !!(ttm->page_flags &TTM_TT_FLAG_EXTERNAL );
@@ -123,7 +124,7 @@ static int lsdc_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void lsdc_ttm_tt_unpopulate(struct ttm_device *bdev,
drivers/gpu/drm/nouveau/nouveau_bo.c +4 -2
@@ -1417,7 +1417,9 @@ vm_fault_t nouveau_ttm_fault_reserve_notify(struct ttm_buffer_object *bo)static int nouveau_ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {struct ttm_tt *ttm_dma = (void *)ttm;struct nouveau_drm *drm;
@@ -1434,7 +1436,7 @@ nouveau_ttm_tt_populate(struct ttm_device *bdev, drm =nouveau_bdev (bdev); -return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, memcg_account, ctx); } static void
drivers/gpu/drm/radeon/radeon_ttm.c +2 -1
@@ -526,6 +526,7 @@ static struct radeon_ttm_tt *radeon_ttm_tt_to_gtt(struct radeon_device *rdev,static int radeon_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct radeon_device *rdev =radeon_get_rdev (bdev);
@@ -547,7 +548,7 @@ static int radeon_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, memcg_account, ctx); }static void radeon_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)
drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +1 -1
@@ -538,7 +538,7 @@ static void ttm_bo_validate_no_placement_signaled(struct kunit *test)if (params->with_ttm) { old_tt = priv->ttm_dev->funcs->ttm_tt_create (bo,0 ); -ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, &ctx); +ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, false, &ctx); bo->ttm = old_tt; }
drivers/gpu/drm/ttm/tests/ttm_pool_test.c +8 -8
@@ -89,7 +89,7 @@ static struct ttm_pool *ttm_pool_pre_populated(struct kunit *test,ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -157,7 +157,7 @@ static void ttm_pool_alloc_basic(struct kunit *test)KUNIT_ASSERT_EQ (test, pool->nid,NUMA_NO_NODE );KUNIT_ASSERT_EQ (test, pool->alloc_flags, params->alloc_flags); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);
@@ -220,7 +220,7 @@ static void ttm_pool_alloc_basic_dma_addr(struct kunit *test)ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);
@@ -253,7 +253,7 @@ static void ttm_pool_alloc_order_caching_match(struct kunit *test) tt =ttm_tt_kunit_init (test,0 , caching, size);KUNIT_ASSERT_NOT_NULL (test, tt); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));
@@ -285,7 +285,7 @@ static void ttm_pool_alloc_caching_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -319,7 +319,7 @@ static void ttm_pool_alloc_order_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -349,7 +349,7 @@ static void ttm_pool_free_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx); pt = &pool->caching[caching].orders[order];KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));
@@ -379,7 +379,7 @@ static void ttm_pool_free_no_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,0 ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx);ttm_pool_free (pool, tt);ttm_tt_fini (tt);
drivers/gpu/drm/ttm/tests/ttm_tt_test.c +6 -6
@@ -262,7 +262,7 @@ static void ttm_tt_populate_null_ttm(struct kunit *test)struct ttm_operation_ctx ctx = { };int err; - err =ttm_tt_populate (devs->ttm_dev,NULL , &ctx); + err =ttm_tt_populate (devs->ttm_dev,NULL , false, &ctx);KUNIT_ASSERT_EQ (test, err, -EINVAL ); }
@@ -283,11 +283,11 @@ static void ttm_tt_populate_populated_ttm(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); populated_page = *tt->pages; - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_PTR_EQ (test, populated_page, *tt->pages); }
@@ -307,7 +307,7 @@ static void ttm_tt_unpopulate_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));
@@ -351,7 +351,7 @@ static void ttm_tt_swapin_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));
@@ -361,7 +361,7 @@ static void ttm_tt_swapin_basic(struct kunit *test)KUNIT_ASSERT_TRUE (test, tt->page_flags &TTM_TT_FLAG_SWAPPED );/* Swapout depopulates TT, allocate pages and then swap them in */ - err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, &ctx); + err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); err =ttm_tt_swapin (tt);
drivers/gpu/drm/ttm/ttm_bo.c +4 -3
@@ -145,7 +145,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, goto out_err;if (mem->mem_type !=TTM_PL_SYSTEM ) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret) goto out_err; }
@@ -1255,6 +1255,7 @@ void ttm_bo_tt_destroy(struct ttm_buffer_object *bo)* is set to true. */ int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_device *bdev = bo->bdev;
@@ -1268,7 +1269,7 @@ int ttm_bo_populate(struct ttm_buffer_object *bo,return 0 ; swapped =ttm_tt_is_swapped (tt); - ret =ttm_tt_populate (bdev, tt, ctx); + ret =ttm_tt_populate (bdev, tt, memcg_account, ctx);if (ret)return ret;
@@ -1293,7 +1294,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);ttm_bo_unreserve (bo);return ret; }
drivers/gpu/drm/ttm/ttm_bo_util.c +3 -3
@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man =ttm_manager_type (bdev, src_mem->mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }
@@ -352,7 +352,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,BUG_ON (!ttm); - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
@@ -533,7 +533,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map)pgprot_t prot;void *vaddr; - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
drivers/gpu/drm/ttm/ttm_bo_vm.c +3 -1
@@ -224,7 +224,9 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, }; ttm = bo->ttm; - err =ttm_bo_populate (bo, &ctx); + err =ttm_bo_populate (bo, + false, + &ctx);if (err) {if (err == -EINTR || err == -ERESTARTSYS || err == -EAGAIN )
drivers/gpu/drm/ttm/ttm_pool.c +4 -2
@@ -761,6 +761,7 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest, }static int __ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,const struct ttm_operation_ctx *ctx,struct ttm_pool_alloc_state *alloc,struct ttm_pool_tt_restore *restore)
@@ -871,6 +872,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,* Returns: 0 on successe, negative error code otherwise. */ int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_pool_alloc_state alloc;
@@ -880,7 +882,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,ttm_pool_alloc_state_init (tt, &alloc); -return __ttm_pool_alloc (pool, tt, ctx, &alloc,NULL ); +return __ttm_pool_alloc (pool, tt, memcg_account, ctx, &alloc,NULL ); }EXPORT_SYMBOL (ttm_pool_alloc);
@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,return 0 ; } -return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); +return __ttm_pool_alloc (pool, tt, false, ctx, &alloc, restore); }/**
drivers/gpu/drm/ttm/ttm_tt.c +5 -3
@@ -368,7 +368,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,EXPORT_SYMBOL_FOR_TESTS_ONLY (ttm_tt_swapout);int ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {int ret;
@@ -397,9 +399,9 @@ int ttm_tt_populate(struct ttm_device *bdev, }if (bdev->funcs->ttm_tt_populate) - ret = bdev->funcs->ttm_tt_populate (bdev, ttm, ctx); + ret = bdev->funcs->ttm_tt_populate (bdev, ttm, memcg_account, ctx);else - ret =ttm_pool_alloc (&bdev->pool, ttm, ctx); + ret =ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);if (ret) goto error;
drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +2 -2
@@ -569,13 +569,13 @@ int vmw_bo_cpu_blit(struct vmw_bo *vmw_dst,dma_resv_assert_held (src->base.resv);if (!ttm_tt_is_populated (dst->ttm)) { - ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, &ctx); + ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, false, &ctx);if (ret)return ret; }if (!ttm_tt_is_populated (src->ttm)) { - ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, &ctx); + ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, false, &ctx);if (ret)return ret; }
drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +4 -3
@@ -360,7 +360,8 @@ static void vmw_ttm_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)static int vmw_ttm_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm,bool memcg_account, +struct ttm_operation_ctx *ctx) {bool external = (ttm->page_flags &TTM_TT_FLAG_EXTERNAL ) !=0 ;
@@ -372,7 +373,7 @@ static int vmw_ttm_populate(struct ttm_device *bdev, ttm->dma_address, ttm->num_pages); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void vmw_ttm_unpopulate(struct ttm_device *bdev,
@@ -580,7 +581,7 @@ int vmw_bo_create_and_populate(struct vmw_private *dev_priv,if (unlikely (ret !=0 ))return ret; - ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, &ctx); + ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, false, &ctx);if (likely (ret ==0 )) {struct vmw_ttm_tt *vmw_tt = container_of(vbo->tbo.ttm, struct vmw_ttm_tt, dma_ttm);
drivers/gpu/drm/xe/xe_bo.c +3 -2
@@ -536,6 +536,7 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, }static int xe_ttm_tt_populate (struct ttm_device *ttm_dev,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) { struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err =ttm_tt_restore (ttm_dev, tt, ctx); }else {ttm_tt_clear_backed_up (tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)return err;
@@ -1926,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (ttm_manager_type (tbo->bdev, tbo->resource->mem_type)->use_tt) { err =xe_bo_wait_usage_kernel (bo, ctx);if (!err) - err =ttm_bo_populate (&bo->ttm, ctx); + err =ttm_bo_populate (&bo->ttm, false, ctx); }else if (should_migrate_to_smem (bo)) {xe_assert (xe_bo_device (bo), bo->flags &XE_BO_FLAG_SYSTEM ); err =xe_bo_migrate (bo,XE_PL_TT , ctx, exec);
include/drm/ttm/ttm_bo.h +1 -0
@@ -475,6 +475,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp);void ttm_bo_tt_destroy (struct ttm_buffer_object *bo);int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx);int ttm_bo_setup_export (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx);
include/drm/ttm/ttm_device.h +1 -0
@@ -85,6 +85,7 @@ struct ttm_device_funcs {*/ int (*ttm_tt_populate)(struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/**
include/drm/ttm/ttm_pool.h +1 -0
@@ -78,6 +78,7 @@ struct ttm_pool { };int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx);void ttm_pool_free (struct ttm_pool *pool,struct ttm_tt *tt);
include/drm/ttm/ttm_tt.h +1 -0
@@ -250,6 +250,7 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,* Calls the driver method to allocate pages for a ttm */ int ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/**
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 3/10 ttm/pool: initialise the shrinker earlier (v2)
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
Later memcg enablement needs the shrinker initialised before the list lru, Just move it for now, but also handle the list being uninitialised.
---
v2: sashiko identified a problem with the list handling.
--- drivers/gpu/drm/ttm/ttm_pool.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +17 -13
@@ -440,10 +440,14 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)down_read (&pool_shrink_rwsem);spin_lock (&shrinker_lock); - pt =list_first_entry (&shrinker_list,typeof (*pt), shrinker_list); -list_move_tail (&pt->shrinker_list, &shrinker_list); + pt =list_first_entry_or_null (&shrinker_list,typeof (*pt), shrinker_list); +if (pt) +list_move_tail (&pt->shrinker_list, &shrinker_list);spin_unlock (&shrinker_lock); +if (!pt) +return 0 ; + num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); num_pages *=1 << pt->order;
@@ -1402,6 +1406,17 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); + mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); +if (!mm_shrinker) +return -ENOMEM ; + + mm_shrinker->count_objects = ttm_pool_shrinker_count; + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; + mm_shrinker->batch =TTM_SHRINKER_BATCH ; + mm_shrinker->seeks =1 ; + +shrinker_register (mm_shrinker); +for (i =0 ; i <NR_PAGE_ORDERS ; ++i) {ttm_pool_type_init (&global_write_combined[i],NULL , ttm_write_combined, i);
@@ -1424,17 +1439,6 @@ int ttm_pool_mgr_init(unsigned long num_pages)#endif #endif - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); -if (!mm_shrinker) -return -ENOMEM ; - - mm_shrinker->count_objects = ttm_pool_shrinker_count; - mm_shrinker->scan_objects = ttm_pool_shrinker_scan; - mm_shrinker->batch =TTM_SHRINKER_BATCH ; - mm_shrinker->seeks =1 ; - -shrinker_register (mm_shrinker); -return 0 ; }
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 4/10 ttm: add objcg pointer to bo and tt (v3)
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This just adds the obj cgroup pointer to the bo and tt structs, and sets it between them.
---
v2: add the put and a setter helper
v3: handle transfer objects reference counts
--- drivers/gpu/drm/ttm/ttm_bo.c | 2 ++ drivers/gpu/drm/ttm/ttm_bo_util.c | 2 ++ drivers/gpu/drm/ttm/ttm_tt.c | 1 + include/drm/ttm/ttm_bo.h | 20 ++++++++++++++++++++ include/drm/ttm/ttm_tt.h | 2 ++ 5 files changed, 27 insertions(+)
drivers/gpu/drm/ttm/ttm_bo.c +2 -0
@@ -48,6 +48,7 @@#include <linux/atomic.h> #include <linux/cgroup_dmem.h> #include <linux/dma-resv.h> +#include <linux/memcontrol.h> #include "ttm_module.h" #include "ttm_bo_internal.h"
@@ -315,6 +316,7 @@ static void ttm_bo_release(struct kref *kref)dma_resv_unlock (bo->base.resv); } +obj_cgroup_put (bo->objcg);atomic_dec (&ttm_glob.bo_count); bo->destroy (bo); }
drivers/gpu/drm/ttm/ttm_bo_util.c +2 -0
@@ -238,6 +238,8 @@ static int ttm_buffer_object_transfer(struct ttm_buffer_object *bo,return -ENOMEM ; fbo->base = *bo; +if (bo->objcg) +obj_cgroup_get (bo->objcg);/** * Fix up members that we shouldn't copy directly:
drivers/gpu/drm/ttm/ttm_tt.c +1 -0
@@ -164,6 +164,7 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm, ttm->caching = caching; ttm->restore =NULL ; ttm->backup =NULL ; + ttm->objcg = bo->objcg; } int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
include/drm/ttm/ttm_bo.h +20 -0
@@ -135,6 +135,12 @@ struct ttm_buffer_object {* reservation lock. */ struct sg_table *sg; + +/** +* @objcg: object cgroup to charge this to if it ends up using system memory. +* NULL means don't charge. +*/ +struct obj_cgroup *objcg; };#define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -344,6 +350,20 @@ ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo)spin_unlock (&bo->bdev->lru_lock); } +/** +* ttm_bo_set_cgroup - assign a cgroup to a buffer object. +* @bo: The bo to set the cgroup for +* @objcg: the cgroup to set. +* +* This transfers the cgroup reference to the bo. From this +* point on the cgroup reference is owned by the ttm bo. +*/ +static inline void ttm_bo_set_cgroup (struct ttm_buffer_object *bo, +struct obj_cgroup *objcg) +{ + bo->objcg = objcg; +} +static inline void ttm_bo_assign_mem (struct ttm_buffer_object *bo,struct ttm_resource *new_mem) {
include/drm/ttm/ttm_tt.h +2 -0
@@ -126,6 +126,8 @@ struct ttm_tt {enum ttm_caching caching;/** @restore: Partial restoration from backup state. TTM private */ struct ttm_pool_tt_restore *restore; +/** @objcg: Object cgroup for this TT allocation */ +struct obj_cgroup *objcg; };/**
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 5/10 ttm/pool: enable memcg tracking and shrinker. (v3)
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This enables all the backend code to use the list lru in memcg mode, and set the shrinker to be memcg aware.
It adds the loop case for when pooled pages end up being reparented to a higher memcg group, that newer memcg can search for them there and take them back.
---
v2: just use the proper stats.
v3: fix objcg check to return void
--- drivers/gpu/drm/ttm/ttm_pool.c | 124 +++++++++++++++++++++++++++------ mm/list_lru.c | 1 + 2 files changed, 102 insertions(+), 23 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +101 -23
@@ -144,7 +144,9 @@ static int ttm_pool_nid(struct ttm_pool *pool) }/* Allocate pages of size 1 << order with the given gfp_flags */ -static struct page *ttm_pool_alloc_page (struct ttm_pool *pool,gfp_t gfp_flags, +static struct page *ttm_pool_alloc_page (struct ttm_pool *pool, +struct obj_cgroup *objcg, +gfp_t gfp_flags,unsigned int order) {const unsigned int beneficial_order =ttm_pool_beneficial_order (pool);
@@ -172,7 +174,10 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, p =alloc_pages_node (pool->nid, gfp_flags, order);if (p) { p->private = order; -mod_lruvec_page_state (p,NR_GPU_ACTIVE ,1 << order); +if (!mem_cgroup_charge_gpu_page (objcg, p, order, gfp_flags, false)) { +__free_pages (p, order); +return NULL ; + } }return p; }
@@ -209,8 +214,7 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,static void __free_pages_gpu_account (struct page *p,unsigned int order,bool reclaim) { -mod_lruvec_page_state (p, reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , - -(1 << order)); +mem_cgroup_uncharge_gpu_page (p, order, reclaim);__free_pages (p, order); }
@@ -317,12 +321,11 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,NULL ); +list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p));rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , -num_pages); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , num_pages); +mem_cgroup_move_gpu_page_reclaim (NULL , p, pt->order, true); } static enum lru_status take_one_from_lru(struct list_head *item,
@@ -337,20 +340,56 @@ static enum lru_status take_one_from_lru(struct list_head *item,return LRU_REMOVED ; } -/* Take pages from a specific pool_type, return NULL when nothing available */ -static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid) +static int pool_lru_get_page (struct ttm_pool_type *pt,int nid, +struct page **page_out, +struct obj_cgroup *objcg, +struct mem_cgroup *memcg) {int ret;struct page *p =NULL ;unsigned long nr_to_walk =1 ; +unsigned int num_pages =1 << pt->order; - ret =list_lru_walk_node (&pt->pages, nid, take_one_from_lru, (void *)&p, &nr_to_walk); + ret =list_lru_walk_one (&pt->pages, nid, memcg, take_one_from_lru, (void *)&p, &nr_to_walk);if (ret ==1 && p) { -atomic_long_sub (1 << pt->order, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , (1 << pt->order)); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , -(1 << pt->order)); +atomic_long_sub (num_pages, &allocated_pages[nid]); + +if (!mem_cgroup_move_gpu_page_reclaim (objcg, p, pt->order, false)) { +__free_pages (p, pt->order); + p =NULL ; + } } -return p; + *page_out = p; +return ret; +} + +/* Take pages from a specific pool_type, return NULL when nothing available */ +static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid, +struct obj_cgroup *orig_objcg) +{ +struct page *page_out =NULL ; +int ret; +struct mem_cgroup *orig_memcg = orig_objcg ?get_mem_cgroup_from_objcg (orig_objcg) :NULL ; +struct mem_cgroup *memcg = orig_memcg; + +/* +* Attempt to get a page from the current memcg, but if it hasn't got any in it's level, +* go up to the parent and check there. This helps the scenario where multiple apps get +* started into their own cgroup from a common parent and want to reuse the pools. +*/ +while (!page_out) { + ret =pool_lru_get_page (pt, nid, &page_out, orig_objcg, memcg); +if (ret ==1 ) +break ; +if (!memcg) +break ; + memcg =parent_mem_cgroup (memcg); +if (!memcg) +break ; + } + +mem_cgroup_put (orig_memcg); +return page_out; }/* Initialize and add a pool type to the global shrinker list */
@@ -360,7 +399,7 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, pt->pool = pool; pt->caching = caching; pt->order = order; -list_lru_init (&pt->pages); +list_lru_init_memcg (&pt->pages, mm_shrinker);spin_lock (&shrinker_lock);list_add_tail (&pt->shrinker_list, &shrinker_list);
@@ -403,6 +442,31 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt)ttm_pool_dispose_list (pt, &dispose); } +/* +* This function doesn't currently check dma32, because no driver using this +* support dma32. This should be added and debugged when that changes. +*/ +static void ttm_pool_check_objcg (struct obj_cgroup *objcg) +{ +#ifdef CONFIG_MEMCG +int r =0 ; +struct mem_cgroup *memcg; +if (!objcg) +return ; + + memcg =get_mem_cgroup_from_objcg (objcg); +for (unsigned i =0 ; i <NR_PAGE_ORDERS ; i++) { + r =memcg_list_lru_alloc (memcg, &global_write_combined[i].pages,GFP_KERNEL ); +if (r) +break ; + r =memcg_list_lru_alloc (memcg, &global_uncached[i].pages,GFP_KERNEL ); +if (r) +break ; + } +mem_cgroup_put (memcg); +#endif +} +/* Return the pool_type to use for the given caching and order */ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, enum ttm_caching caching,
@@ -432,7 +496,9 @@ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, }/* Free pages using the per-node shrinker list */ -static unsigned int ttm_pool_shrink (int nid,unsigned long num_to_free) +static unsigned int ttm_pool_shrink (int nid, +struct mem_cgroup *memcg, +unsigned long num_to_free) {LIST_HEAD (dispose);struct ttm_pool_type *pt;
@@ -448,7 +514,11 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)if (!pt)return 0 ; - num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); +if (!memcg) { + num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); + }else { + num_pages =list_lru_walk_one (&pt->pages, nid, memcg, pool_move_to_dispose_list, &dispose, &num_to_free); + } num_pages *=1 << pt->order;ttm_pool_dispose_list (pt, &dispose);
@@ -777,6 +847,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; +struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);
@@ -794,6 +865,9 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, page_caching = tt->caching; allow_pools = true; + +ttm_pool_check_objcg (objcg); +for (order =ttm_pool_alloc_find_order (MAX_PAGE_ORDER , alloc); alloc->remaining_pages; order =ttm_pool_alloc_find_order (order, alloc)) {
@@ -803,7 +877,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, p =NULL ; pt =ttm_pool_select_type (pool, page_caching, order);if (pt && allow_pools) - p =ttm_pool_type_take (pt,ttm_pool_nid (pool)); + p =ttm_pool_type_take (pt,ttm_pool_nid (pool), objcg);/* * If that fails or previously failed, allocate from system.
@@ -814,7 +888,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,if (!p) { page_caching = ttm_cached; allow_pools = false; - p =ttm_pool_alloc_page (pool, gfp_flags, order); + p =ttm_pool_alloc_page (pool, objcg, gfp_flags, order); }/* If that fails, lower the order if possible and retry. */ if (!p) {
@@ -960,7 +1034,7 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)while (atomic_long_read (&allocated_pages[nid]) > pool_node_limit[nid]) {unsigned long diff =atomic_long_read (&allocated_pages[nid]) - pool_node_limit[nid]; -ttm_pool_shrink (nid, diff); +ttm_pool_shrink (nid,NULL , diff); } }EXPORT_SYMBOL (ttm_pool_free);
@@ -1214,10 +1288,14 @@ static unsigned long ttm_pool_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) {unsigned long num_freed =0 ; +int num_pools; +spin_lock (&shrinker_lock); + num_pools =list_count_nodes (&shrinker_list); +spin_unlock (&shrinker_lock);do - num_freed +=ttm_pool_shrink (sc->nid, sc->nr_to_scan); -while (num_freed < sc->nr_to_scan && + num_freed +=ttm_pool_shrink (sc->nid, sc->memcg, sc->nr_to_scan); +while (num_pools-- >=0 && num_freed < sc->nr_to_scan &&atomic_long_read (&allocated_pages[sc->nid])); sc->nr_scanned = num_freed;
@@ -1406,7 +1484,7 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); + mm_shrinker =shrinker_alloc (SHRINKER_MEMCG_AWARE |SHRINKER_NUMA_AWARE ,"drm-ttm_pool" );if (!mm_shrinker)return -ENOMEM ;
mm/list_lru.c +1 -0
@@ -627,6 +627,7 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,return 0 ;return __memcg_list_lru_alloc (memcg, lru, gfp); } +EXPORT_SYMBOL_GPL (memcg_list_lru_alloc);int folio_memcg_list_lru_alloc (struct folio *folio,struct list_lru *lru,gfp_t gfp)
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 6/10 ttm: hook up memcg placement flags.
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds a placement flag that requests that any bo with this placement flag set gets accounted for memcg if it's a system memory allocation.
--- drivers/gpu/drm/ttm/ttm_bo.c | 4 ++-- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/xe/xe_bo.c | 2 +- include/drm/ttm/ttm_placement.h | 3 +++ 5 files changed, 10 insertions(+), 7 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +2 -2
@@ -146,7 +146,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, goto out_err;if (mem->mem_type !=TTM_PL_SYSTEM ) { - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , ctx);if (ret) goto out_err; }
@@ -1296,7 +1296,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, bo->resource->placement &TTM_PL_FLAG_MEMCG , ctx);ttm_bo_unreserve (bo);return ret; }
drivers/gpu/drm/ttm/ttm_bo_util.c +3 -3
@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man =ttm_manager_type (bdev, src_mem->mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, dst_mem->placement &TTM_PL_FLAG_MEMCG , ctx);if (ret)return ret; }
@@ -354,7 +354,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,BUG_ON (!ttm); - ret =ttm_bo_populate (bo, false, &ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , &ctx);if (ret)return ret;
@@ -535,7 +535,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map)pgprot_t prot;void *vaddr; - ret =ttm_bo_populate (bo, false, &ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , &ctx);if (ret)return ret;
drivers/gpu/drm/ttm/ttm_bo_vm.c +1 -1
@@ -225,7 +225,7 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, ttm = bo->ttm; err =ttm_bo_populate (bo, - false, + bo->resource->placement &TTM_PL_FLAG_MEMCG , &ctx);if (err) { if (err == -EINTR || err == -ERESTARTSYS ||
drivers/gpu/drm/xe/xe_bo.c +1 -1
@@ -1927,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (ttm_manager_type (tbo->bdev, tbo->resource->mem_type)->use_tt) { err =xe_bo_wait_usage_kernel (bo, ctx);if (!err) - err =ttm_bo_populate (&bo->ttm, false, ctx); + err =ttm_bo_populate (&bo->ttm, tbo->resource->placement &TTM_PL_FLAG_MEMCG , ctx); }else if (should_migrate_to_smem (bo)) {xe_assert (xe_bo_device (bo), bo->flags &XE_BO_FLAG_SYSTEM ); err =xe_bo_migrate (bo,XE_PL_TT , ctx, exec);
include/drm/ttm/ttm_placement.h +3 -0
@@ -70,6 +70,9 @@/* Placement is only used during eviction */ #define TTM_PL_FLAG_FALLBACK (1 << 4) +/* Placement should account mem cgroup */ +#define TTM_PL_FLAG_MEMCG (1 << 5) +/** * struct ttm_place *
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 7/10 memcontrol: allow objcg api when memcg is config off.
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
amdgpu wants to use the objcg api and not have to enable ifdef around it, so just add a dummy function for the config off path.
--- include/linux/memcontrol.h | 5 +++++ 1 file changed, 5 insertions(+)
include/linux/memcontrol.h +5 -0
@@ -1827,6 +1827,11 @@ static inline void __memcg_kmem_uncharge_page(struct page *page, int order) { } +static inline struct obj_cgroup *get_obj_cgroup_from_current (void ) +{ +return NULL ; +} +static inline struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {return NULL ;
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 8/10 amdgpu: add support for memory cgroups
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds support for adding a obj cgroup to a buffer object, and passing in the placement flags to make sure it's accounted properly.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 13 +++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 +++ mm/memcontrol.c | 1 + 5 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1 -0
@@ -188,6 +188,7 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, bp.domain = initial_domain; bp.bo_ptr_size =sizeof (struct amdgpu_bo ); bp.xcp_id_plus1 = xcp_id_plus1; + bp.objcg =get_obj_cgroup_from_current (); r =amdgpu_bo_create_user (adev, &bp, &ubo);if (r)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +9 -4
@@ -159,7 +159,7 @@ void amdgpu_bo_placement_from_domain(struct amdgpu_bo *abo, u32 domain) places[c].mem_type = abo->flags &AMDGPU_GEM_CREATE_PREEMPTIBLE ?AMDGPU_PL_PREEMPT :TTM_PL_TT ; - places[c].flags =0 ; + places[c].flags =TTM_PL_FLAG_MEMCG ;/* * When GTT is just an alternative to VRAM make sure that we * only use it as fallback and still try to fill up VRAM first.
@@ -174,7 +174,7 @@ void amdgpu_bo_placement_from_domain(struct amdgpu_bo *abo, u32 domain) places[c].fpfn =0 ; places[c].lpfn =0 ; places[c].mem_type =TTM_PL_SYSTEM ; - places[c].flags =0 ; + places[c].flags =TTM_PL_FLAG_MEMCG ; c++; }
@@ -654,16 +654,21 @@ int amdgpu_bo_create(struct amdgpu_device *adev, size =ALIGN (size,PAGE_SIZE ); } -if (!amdgpu_bo_validate_size (adev, size, bp->domain)) +if (!amdgpu_bo_validate_size (adev, size, bp->domain)) { +obj_cgroup_put (bp->objcg);return -ENOMEM ; + }BUG_ON (bp->bo_ptr_size <sizeof (struct amdgpu_bo )); *bo_ptr =NULL ; bo =kvzalloc (bp->bo_ptr_size,GFP_KERNEL ); -if (bo ==NULL ) +if (bo ==NULL ) { +obj_cgroup_put (bp->objcg);return -ENOMEM ; + }drm_gem_private_object_init (adev_to_drm (adev), &bo->tbo.base, size); +ttm_bo_set_cgroup (&bo->tbo, bp->objcg);/* hand the reference to the ttm bo */ bo->tbo.base.funcs = &amdgpu_gem_object_funcs; bo->vm_bo =NULL ; bo->preferred_domains = bp->preferred_domain ? bp->preferred_domain :
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +1 -0
@@ -55,6 +55,7 @@ struct amdgpu_bo_param {enum ttm_bo_type type;bool no_wait_gpu;struct dma_resv *resv; +struct obj_cgroup *objcg;void (*destroy)(struct ttm_buffer_object *bo);/* xcp partition number plus 1, 0 means any partition */ int8_t xcp_id_plus1;
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -0
@@ -155,11 +155,14 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT |AMDGPU_GEM_DOMAIN_CPU ); } +for (int i =0 ; i < abo->placement.num_placement; i++) + abo->placements[i].flags &= ~TTM_PL_FLAG_MEMCG ;break ;case TTM_PL_TT :case AMDGPU_PL_PREEMPT :default :amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + abo->placements[0 ].flags &= ~TTM_PL_FLAG_MEMCG ;break ; } *placement = abo->placement;
mm/memcontrol.c +1 -0
@@ -3032,6 +3032,7 @@ __always_inline struct obj_cgroup *current_obj_cgroup(void)return rcu_dereference_check (root_mem_cgroup->nodeinfo[nid]->objcg,1 ); } +EXPORT_SYMBOL_GPL (current_obj_cgroup);struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 9/10 ttm: add support for a module option to disable memcg integration
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds a kconfig and a module option to turn off ttm memcg integration completely.
When this is used, no object will ever end up using memcg aware paths.
There is an existing workload that cgroup support might regress, the systems are setup to allocate 1GB of uncached pages at system startup to prime the pool, then any further users will take them from the pool. The current cgroup code might handle that, but it also may regress, so add an option to ttm to avoid using memcg for the pool pages.
--- drivers/gpu/drm/Kconfig | 7 +++++++ drivers/gpu/drm/ttm/ttm_pool.c | 24 +++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-)
drivers/gpu/drm/Kconfig +7 -0
@@ -257,6 +257,13 @@ config DRM_TTM_HELPER help Helpers for ttm-based gem objects +config DRM_TTM_MEMCG + bool "Enable TTM mem cgroup by default" + depends on DRM_TTM + depends on MEMCG + help + Enable the memcg integration by default + config DRM_GEM_DMA_HELPER tristate depends on DRM
drivers/gpu/drm/ttm/ttm_pool.c +21 -3
@@ -119,6 +119,24 @@ static unsigned long page_pool_size;MODULE_PARM_DESC (page_pool_size,"Number of pages in the WC/UC/DMA pool per NUMA node" );module_param (page_pool_size, ulong,0644 ); +/* +* Don't use the memcg aware lru for pooled pages. +* +* There are use-cases where for example one application in a cgroup will preallocate 1GB +* of uncached pages, and immediately release them into the pool, for other consumers +* to use. This use-case could be handled with a proper cgroup hierarchy, but to allow +* that use case to continue to operate as-is, add a module option. +* +* This still stores the pages in the list_lru, it just doesn't use the memcg when +* adding/removing them. +*/ +#define DEFAULT_TTM_MEMCG IS_ENABLED(CONFIG_DRM_TTM_MEMCG) +static bool ttm_memcg =DEFAULT_TTM_MEMCG ; + +MODULE_PARM_DESC(ttm_memcg, "Allow using cgroups with TTM " + "[default=" __stringify(DEFAULT_TTM_MEMCG) "])"); +module_param (ttm_memcg, bool,0444 ); +static unsigned long pool_node_limit[MAX_NUMNODES ];static atomic_long_t allocated_pages[MAX_NUMNODES ];
@@ -321,7 +339,7 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p)); +list_lru_add (&pt->pages, &p->lru, nid, ttm_memcg ?page_memcg_check (p) :NULL );rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]);
@@ -370,7 +388,7 @@ static struct page *ttm_pool_type_take(struct ttm_pool_type *pt, int nid,struct page *page_out =NULL ;int ret;struct mem_cgroup *orig_memcg = orig_objcg ?get_mem_cgroup_from_objcg (orig_objcg) :NULL ; -struct mem_cgroup *memcg = orig_memcg; +struct mem_cgroup *memcg = ttm_memcg ? orig_memcg :NULL ;/* * Attempt to get a page from the current memcg, but if it hasn't got any in it's level,
@@ -847,7 +865,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; -struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ; +struct obj_cgroup *objcg = (ttm_memcg && memcg_account) ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 10/10 xe: create a flag to enable memcg accounting for XE as well.
In reply to Dave AirlieFrom: Maarten Lankhorst <[email protected]>
This adds support for memcg accounting to ttm object used by xe driver.
--- drivers/gpu/drm/xe/xe_bo.c | 10 +++++++--- drivers/gpu/drm/xe/xe_bo.h | 1 + drivers/gpu/drm/xe/xe_lrc.c | 3 ++- drivers/gpu/drm/xe/xe_oa.c | 3 ++- drivers/gpu/drm/xe/xe_pt.c | 3 ++- 5 files changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +7 -3
@@ -56,6 +56,7 @@ static const struct ttm_place sys_placement_flags = { .flags = 0, }; +/* TTM_PL_FLAG_MEMCG is not set, those placements are used for eviction */ static struct ttm_placement sys_placement = { .num_placement =1 , .placement = &sys_placement_flags,
@@ -181,8 +182,8 @@ static void try_add_system(struct xe_device *xe, struct xe_bo *bo, bo->placements[*c] = (struct ttm_place ) { .mem_type =XE_PL_TT , - .flags = (bo_flags &XE_BO_FLAG_VRAM_MASK ) ? -TTM_PL_FLAG_FALLBACK :0 , + .flags =TTM_PL_FLAG_MEMCG | ((bo_flags &XE_BO_FLAG_VRAM_MASK ) ? +TTM_PL_FLAG_FALLBACK :0 ), }; *c +=1 ; }
@@ -2384,6 +2385,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, placement = (type == ttm_bo_type_sg || bo->flags &XE_BO_FLAG_DEFER_BACKING ) ? &sys_placement : &bo->placement; + +if (bo->flags &XE_BO_FLAG_ACCOUNTED ) +ttm_bo_set_cgroup (&bo->ttm,get_obj_cgroup_from_current ()); err =ttm_bo_init_reserved (&xe->ttm, &bo->ttm, type, placement, alignment, &ctx,NULL , resv, xe_ttm_bo_destroy);
@@ -3361,7 +3365,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, args->size & ~PAGE_MASK ))return -EINVAL ; - bo_flags =0 ; + bo_flags =XE_BO_FLAG_ACCOUNTED ;if (args->flags &DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING ) bo_flags |=XE_BO_FLAG_DEFER_BACKING ;
drivers/gpu/drm/xe/xe_bo.h +1 -0
@@ -52,6 +52,7 @@#define XE_BO_FLAG_CPU_ADDR_MIRROR BIT(24)#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26) +#define XE_BO_FLAG_ACCOUNTED BIT(27)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)
drivers/gpu/drm/xe/xe_lrc.c +2 -1
@@ -1661,7 +1661,8 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct xe_vXE_BO_FLAG_GGTT_INVALIDATE ;if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM | +XE_BO_FLAG_ACCOUNTED ; bo = xe_bo_create_pin_map_novm(xe, tile, bo_size, ttm_bo_type_kernel,
drivers/gpu/drm/xe/xe_oa.c +2 -1
@@ -910,7 +910,8 @@ static int xe_oa_alloc_oa_buffer(struct xe_oa_stream *stream, size_t size) bo =xe_bo_create_pin_map_novm (stream->oa->xe, stream->gt->tile, size, ttm_bo_type_kernel, - vram |XE_BO_FLAG_GGTT , false); + vram |XE_BO_FLAG_GGTT |XE_BO_FLAG_ACCOUNTED , + false);if (IS_ERR (bo))return PTR_ERR (bo);
drivers/gpu/drm/xe/xe_pt.c +2 -1
@@ -123,7 +123,8 @@ struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_tile *tile,XE_BO_FLAG_IGNORE_MIN_PAGE_SIZE |XE_BO_FLAG_NO_RESV_EVICT |XE_BO_FLAG_PAGETABLE ;if (vm->xef)/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM | +XE_BO_FLAG_ACCOUNTED ; pt->level = level;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 03/10] ttm/pool: initialise the shrinker earlier (v2)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Deadlock due to leaked `pool_shrink_rwsem` read lock on early return in `ttm_pool_shrink()`.…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Deadlock due to leaked pool_shrink_rwsem read lock on early return in ttm_pool_shrink().
- [High] Fatal NULL pointer dereference caused by bypassed global pool initialization upon shrinker_alloc failure. --
commit a5f142f7cded6383539224542edc3c44efc5294c
Author: Dave Airlie <[email protected]>
ttm/pool: initialise the shrinker earlier (v2)
This commit moves the shrinker initialization earlier in the module load sequence to accommodate future memcg enablement. It also adds a check in the shrinker callback to handle uninitialized lists.
Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -440,10 +440,14 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)down_read (&pool_shrink_rwsem);spin_lock (&shrinker_lock);
drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free) · line 450
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-2- pt =list_first_entry (&shrinker_list,typeof (*pt), shrinker_list); -list_move_tail (&pt->shrinker_list, &shrinker_list); + pt =list_first_entry_or_null (&shrinker_list,typeof (*pt), shrinker_list); +if (pt) +list_move_tail (&pt->shrinker_list, &shrinker_list);spin_unlock (&shrinker_lock); +if (!pt) +return 0 ; +
[Severity: High]
Does this early return leak the pool_shrink_rwsem read lock?
We acquire the lock just a few lines above via down_read(&pool_shrink_rwsem). If the shrinker list is empty and pt is NULL, returning here skips the up_read() call at the end of the function, which could lead to a deadlock when another writer or reader attempts to acquire it.
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1412
Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +4
-0@@ -1402,6 +1406,17 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); + mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); +if (!mm_shrinker) +return -ENOMEM ; +
[Severity: High]
If shrinker_alloc() fails and returns early here, does it leave the global pool structures completely uninitialized?
Because the caller ttm_global_init() ignores the return value of ttm_pool_mgr_init(), driver initialization will continue. The newly moved return bypasses the initialization loop for structures like global_write_combined.
When the driver later attempts to allocate from these pools, or when ttm_pool_mgr_fini() calls list_del() on the zero-filled uninitialized structures, can this cause a NULL pointer dereference?
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +7
-0+ mm_shrinker->count_objects = ttm_pool_shrinker_count; + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; + mm_shrinker->batch =TTM_SHRINKER_BATCH ; + mm_shrinker->seeks =1 ; + +shrinker_register (mm_shrinker); +for (i =0 ; i <NR_PAGE_ORDERS ; ++i) {ttm_pool_type_init (&global_write_combined[i],NULL , ttm_write_combined, i);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH 04/10] ttm: add objcg pointer to bo and tt (v3)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The obj_cgroup reference acquired in ttm_buffer_object_transfer() is leaked on the error path. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The obj_cgroup reference acquired in ttm_buffer_object_transfer() is leaked on the error path. --
commit 1cccf014e6a8377ef4e98ad7bba76a7229069c1c
Author: Dave Airlie <[email protected]>
Subject: ttm: add objcg pointer to bo and tt (v3)
This commit adds the obj cgroup pointer to the bo and tt structs, and sets it between them.
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +2
-0@@ -238,6 +238,8 @@ static int ttm_buffer_object_transfer(struct ttm_buffer_object *bo,return -ENOMEM ; fbo->base = *bo; +if (bo->objcg) +obj_cgroup_get (bo->objcg);/** * Fix up members that we shouldn't copy directly:
[ ... ]
drivers/gpu/drm/ttm/ttm_bo_util.c
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo_util.c +0
-0ret =dma_resv_trylock (&fbo->base.base._resv);WARN_ON (!ret); ret =dma_resv_reserve_fences (&fbo->base.base._resv,TTM_NUM_MOVE_FENCES );if (ret) {dma_resv_unlock (&fbo->base.base._resv);kfree (fbo);return ret; }
[Severity: High]
Does this error path leak the objcg reference?
We acquire a new reference for the transfer buffer object with obj_cgroup_get() earlier in ttm_buffer_object_transfer(). If dma_resv_reserve_fences() fails, this error path frees the memory for fbo and returns, but it doesn't seem to drop the newly acquired objcg reference.
Should there be a call to obj_cgroup_put() before returning here to ensure the memory cgroup doesn't stay pinned indefinitely?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH 02/10] ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The `memcg_account` flag is not plumbed through `ttm_tt_restore` and `ttm_pool_restore_and_alloc`…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The memcg_account flag is not plumbed through ttm_tt_restore and ttm_pool_restore_and_alloc, bypassing memory cgroup accounting for restored TTs. --
commit b7727d0f7c7e75f86466225e149812b98610b27c
Author: Dave Airlie <[email protected]>
ttm: add a memcg accounting flag to the alloc/populate APIs
This flag does nothing yet, but this just changes the APIs to accept it in the future across all users.
[ ... ]
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 941
Quoted 7 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,return 0 ; } -return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); +return __ttm_pool_alloc (pool, tt, false, ctx, &alloc, restore); }
[Severity: High]
Should ttm_pool_restore_and_alloc() and ttm_tt_restore() also be updated to accept the memcg_account flag?
By hardcoding false here, pages allocated to restore a backed-up TT will bypass memory cgroup accounting. This could allow userspace to circumvent memory limits by triggering swapout/backup and subsequently forcing a restore.
[ ... ]
Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 559
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +1
-1@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err =ttm_tt_restore (ttm_dev, tt, ctx); }else {ttm_tt_clear_backed_up (tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)
[Severity: High]
Because ttm_tt_restore() wasn't updated to take the memcg_account parameter, the flag passed into xe_ttm_tt_populate() is dropped when a backed-up buffer is restored. Does this mean restored TTs in the xe driver will fail to be tracked by their memory cgroup?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH 06/10] ttm: hook up memcg placement flags.
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Dereferencing `bo->resource` in `ttm_bo_setup_export()` without checking for NULL causes a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Dereferencing bo->resource in ttm_bo_setup_export() without checking for NULL causes a kernel panic if the buffer object has been gutted. --
commit 55aef0c5658ae86fbf245de25768c668ff9edb05
Author: Dave Airlie <[email protected]>
ttm: hook up memcg placement flags.
This commit adds a placement flag to request that a buffer object with this flag is accounted for memcg if it's a system memory allocation.
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_bo.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_bo.c int ttm_bo_setup_export(struct ttm_buffer_object *bo, · line 1299
Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1@@ -1296,7 +1296,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, bo->resource->placement &TTM_PL_FLAG_MEMCG , ctx);
[Severity: Critical]
Does this code unconditionally dereference bo->resource?
If a buffer object has been gutted (evicted without a fallback placement), bo->resource will be NULL.
Could a user-space application trigger a kernel panic by exporting such a gutted buffer via the DRM_IOCTL_PRIME_HANDLE_TO_FD IOCTL, which ultimately calls ttm_bo_setup_export() on the DMABUF export path?
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_bo.c +0
-0ttm_bo_unreserve (bo);return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH 09/10] ttm: add support for a module option to disable memcg integration
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] The memcg-aware shrinker count callback returns the global node pool size instead of the memcg-specific pool size, severely distorting proportional reclaim.
- [High] DMA-allocated pages permanently corrupt NR_GPU_ACTIVE and NR_GPU_RECLAIM vmstat counters when added to and freed from the TTM pool. --
commit 6e92a3554c189b1a664eb6626ab6f741e3bdc804
Author: Dave Airlie <[email protected]>
ttm: add support for a module option to disable memcg integration
This adds a kconfig and a module option to turn off ttm memcg integration completely. When this is used, no object will ever end up using memcg aware paths.
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p) · line 345
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -321,7 +339,7 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p)); +list_lru_add (&pt->pages, &p->lru, nid, ttm_memcg ?page_memcg_check (p) :NULL );rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]);
[Severity: High]
This is a pre-existing issue, but does this path corrupt the NR_GPU_ACTIVE and NR_GPU_RECLAIM vmstat counters for DMA-allocated pages?
In ttm_pool_alloc_page(), pages allocated via the DMA API bypass the mem_cgroup_charge_gpu_page() call, so the NR_GPU_ACTIVE counter is never incremented.
However, when these DMA pages are returned to the pool here in ttm_pool_type_give(), the code unconditionally updates the stats by calling mem_cgroup_move_gpu_page_reclaim(NULL, p, pt->order, true). Because the page has no tracked memcg, this subtracts from the global node's NR_GPU_ACTIVE (causing an underflow) and adds to NR_GPU_RECLAIM.
Furthermore, when the DMA page is ultimately freed from the pool:
drivers/gpu/drm/ttm/ttm_pool.c:ttm_pool_free_page() {
if (!pool || !ttm_pool_uses_dma_alloc(pool)) {
__free_pages_gpu_account(p, order, reclaim);
return;
}
...
dma_free_attrs(...);
}It skips __free_pages_gpu_account() and is freed directly. Thus, NR_GPU_RECLAIM is never decremented, causing a permanent and unbounded leak of the global reclaim counter on the node.
[ ... ]
[Severity: High]
This is a pre-existing issue, but does the memcg-aware shrinker count callback severely distort proportional reclaim by returning the global pool size?
When the VM reclaims memory under a specific memory cgroup, it calls ttm_pool_shrinker_count() with sc->memcg set:
drivers/gpu/drm/ttm/ttm_pool.c:ttm_pool_shrinker_count() {
unsigned long num_pages = atomic_long_read(&allocated_pages[sc->nid]);
return num_pages ? num_pages : SHRINK_EMPTY;
}This unconditionally returns the global number of allocated pages on the node instead of the number of pages actually in the specified memcg's LRU list.
Could this cause the VM to vastly overestimate the amount of reclaimable memory for any memcg under pressure (potentially reporting gigabytes of global pool pages as reclaimable by a small cgroup)?
It seems this would skew the proportional reclaim heuristics, causing the VM to apply excessive pressure to this shrinker while under-pressuring other memory caches.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
[email protected]
AI reviewRe: [PATCH 08/10] amdgpu: add support for memory cgroups
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Evicted VRAM BOs bypass memory cgroup (memcg) accounting because TTM_PL_FLAG_MEMCG is explicitly…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Evicted VRAM BOs bypass memory cgroup (memcg) accounting because TTM_PL_FLAG_MEMCG is explicitly cleared during eviction. --
commit 05e0eb5ded826da3ea51ec14b35acb22f46c4aaa
Author: Dave Airlie <[email protected]>
amdgpu: add support for memory cgroups
This adds support for adding a obj cgroup to a buffer object, and passing in the placement flags to make sure it's accounted properly.
[ ... ]
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 159
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-0@@ -155,11 +155,14 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT |AMDGPU_GEM_DOMAIN_CPU ); } +for (int i =0 ; i < abo->placement.num_placement; i++) + abo->placements[i].flags &= ~TTM_PL_FLAG_MEMCG ;
[Severity: High]
Does clearing TTM_PL_FLAG_MEMCG here allow an application to bypass memcg limits?
If a container continually allocates VRAM BOs, older BOs will be evicted to system memory. Because the TTM_PL_FLAG_MEMCG flag is explicitly removed from the placement flags in amdgpu_evict_flags(), it appears that the system memory pages allocated during this eviction will not be charged to the application's memcg, potentially allowing unbounded host memory consumption.
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 165
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0break ;case TTM_PL_TT :case AMDGPU_PL_PREEMPT :default :amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + abo->placements[0 ].flags &= ~TTM_PL_FLAG_MEMCG ;
[Severity: High]
Does this similarly drop memcg accounting when evicting other memory types to the CPU domain?
Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0
-0break ; } *placement = abo->placement;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
[email protected]
AI reviewRe: [PATCH 01/10] memcg: add support for GPU page counters. (v5)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [High] Faking `MEMCG_DATA_KMEM` without charging the `MEMCG_KMEM` stat causes state corruption and…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] Faking MEMCG_DATA_KMEM without charging the MEMCG_KMEM stat causes state corruption and potential counter underflow.
- [High] mem_cgroup_move_gpu_page_reclaim ignores the to_reclaim argument during cross-cgroup moves, corrupting the active/reclaim stats.
- [High] The logic condition in mem_cgroup_move_gpu_page_reclaim skips charge transfers for uncharged pages, bypassing memory limits.
- [High] Charging a GPU page to the root memcg triggers a VM_WARN_ON_ONCE and permanently leaks memory usage counters. --
commit 1761466dad501528141242e7d0630e4e61d30d71
Author: Dave Airlie <[email protected]> Date: Mon Jul 6 05:22:30 2026 +0000
memcg: add support for GPU page counters. (v5)
This introduces 2 new statistics and 3 new memcontrol APIs for dealing with GPU system memory allocations.
The stats corresponds to the same stats in the global vmstat, for number of active GPU pages, and number of pages in pools that can be reclaimed.
Quoted 4 lines
mm/memcontrol.c +0
-0
[ ... ]
Quoted 16 lines
mm/memcontrol.c +13
-0@@ -5508,6 +5512,106 @@ void mem_cgroup_flush_workqueue(void) } +/** +* mem_cgroup_charge_gpu_page - charge a page to GPU memory tracking +* @objcg: objcg to charge, NULL charges root memcg +* @page: page to charge +* @order: page allocation order +* @gfp_mask: gfp mode +* @reclaim: charge the reclaim counter instead of the active one. +* +* Charge the order sized @page to the objcg. Returns %true if the charge fit within +* @objcg's configured limit, %false if it doesn't. +*/ +bool mem_cgroup_charge_gpu_page (struct obj_cgroup *objcg,struct page *page, +unsigned int order,gfp_t gfp_mask,bool reclaim)
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5536
Quoted 10 lines
mm/memcontrol.c +10
-0+{ +unsigned int nr_pages =1 << order; +struct mem_cgroup *memcg =NULL ; +struct lruvec *lruvec; +int ret; + +if (objcg) { + memcg =get_mem_cgroup_from_objcg (objcg); + + ret =try_charge_memcg (memcg, gfp_mask, nr_pages);
[Severity: High]
Does this trigger a kernel warning and counter leak for root processes?
When a root process allocates GPU memory, current_obj_cgroup() can return a valid obj_cgroup for the root memcg. Calling try_charge_memcg() on the root memcg succeeds and subsequently calls refill_stock(). Inside refill_stock(), this triggers VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg)).
Additionally, mem_cgroup_uncharge_gpu_page() explicitly skips refill_stock() if it is the root memcg. Since the drain mechanism is used to uncharge the counters in this API, does this permanently leak the root memcg's page counters?
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5544
Quoted 7 lines
mm/memcontrol.c +7
-0+if (ret) { +mem_cgroup_put (memcg); +return false; + } + +obj_cgroup_get (objcg); +page_set_objcg (page, objcg);
[Severity: High]
Will this cause MEMCG_KMEM counter underflow and state corruption?
page_set_objcg() overwrites page->memcg_data with objcg | MEMCG_DATA_KMEM. However, unlike standard kmem paths using obj_cgroup_charge_pages(), this function does not charge the MEMCG_KMEM counter.
If a driver leaks the page (failing to uncharge it), the normal page allocator will eventually process it. free_pages_prepare() sees PageMemcgKmem(page) is true and calls __memcg_kmem_uncharge_page(), which decrements MEMCG_KMEM. Since it was never incremented, this causes a permanent underflow.
Also, if this exported API is mistakenly used on pages that already have tracking (like shmem), does this silently corrupt their memcg_data invariants?
Quoted 9 lines
mm/memcontrol.c +9
-0+ } + + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); +mod_lruvec_state (lruvec, reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , nr_pages); + +mem_cgroup_put (memcg); +return true; +} +EXPORT_SYMBOL_GPL (mem_cgroup_charge_gpu_page);
[ ... ]
Quoted 4 lines
mm/memcontrol.c +4
-0+/** +* mem_cgroup_move_gpu_reclaim - move pages from gpu to gpu reclaim and back +* @new_objcg: objcg to move page to, NULL if just stats update. +* @nr_pages: number of pages to move
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5595
Quoted 10 lines
mm/memcontrol.c +10
-0+* @to_reclaim: true moves pages into reclaim, false moves them back +*/ +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *new_objcg, +struct page *page, +unsigned int order, +bool to_reclaim) +{ +struct obj_cgroup *objcg =page_objcg (page); + +if (!objcg || !new_objcg || objcg == new_objcg) {
[Severity: High]
Does this skip charge transfers for uncharged pages, allowing memory limit bypass?
If a page was allocated without memcg accounting (e.g., by a system process or during early boot), its objcg is NULL.
When a container (with a valid new_objcg) allocates this page from the global GPU pool, the !objcg condition evaluates to true. The function enters this block, toggles stats locally, and returns true, skipping the actual charge transfer to new_objcg.
Does this allow a container to use the GPU page without it being charged against its memcg memory limit?
Quoted 5 lines
mm/memcontrol.c +5
-0+struct mem_cgroup *memcg = objcg ?get_mem_cgroup_from_objcg (objcg) :NULL ; +struct lruvec *lruvec; +unsigned long flags; +int nr_pages =1 << order; +
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5610
Quoted 10 lines
mm/memcontrol.c +10
-0+ lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); +local_irq_save (flags); +mod_lruvec_state (lruvec, to_reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , nr_pages); +mod_lruvec_state (lruvec, to_reclaim ?NR_GPU_ACTIVE :NR_GPU_RECLAIM , -nr_pages); +local_irq_restore (flags); +mem_cgroup_put (memcg); +return true; + }else { +mem_cgroup_uncharge_gpu_page (page, order, true); +return mem_cgroup_charge_gpu_page (new_objcg, page, order,0 , false);
[Severity: High]
Does this ignore the to_reclaim argument during cross-cgroup moves?
This unconditionally calls mem_cgroup_uncharge_gpu_page() with reclaim=true and mem_cgroup_charge_gpu_page() with reclaim=false. This hardcodes the transition as moving from the RECLAIM state to the ACTIVE state.
If a caller attempts to move a page between obj_cgroups while keeping its state as ACTIVE, wouldn't this erroneously uncharge the old cgroup's RECLAIM state (causing an underflow) and charge the new cgroup's ACTIVE state?
Quoted 3 lines
mm/memcontrol.c +3
-0+ } +} +EXPORT_SYMBOL_GPL (mem_cgroup_move_gpu_page_reclaim);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH 05/10] ttm/pool: enable memcg tracking and shrinker. (v3)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider: - [Critical] Direct map caching attributes (WC/UC) are not restored to Write-Back (WB) when a page is…
Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [Critical] Direct map caching attributes (WC/UC) are not restored to Write-Back (WB) when a page is freed due to a memcg charging failure.
- [High] Missing list_lru_destroy() causes global list corruption and Use-After-Free when TTM pools are destroyed.
- [High] ttm_pool_check_objcg fails to allocate memcg LRU entries for device-specific pools, breaking memcg-aware shrinking.
- [High] ttm_pool_shrinker_count returns the global pool count instead of the memcg-specific count, breaking memcg reclaim proportionality.
- [High] A race condition in ttm_pool_type_give() allows the shrinker to free a page before local metadata updates finish.
- [High] Uncharged pages stolen from the root pool by accounted BOs are not charged to the requesting memcg.
- [Medium] Ignoring memcg_list_lru_alloc() failure causes a VM_WARN_ON when freeing pages to the pool.
- [Medium] ttm_pool_check_objcg hardcodes GFP_KERNEL, ignoring the caller's allocation context flags.
- [Medium] Missing up_read(&pool_shrink_rwsem) when shrinker_list is empty in ttm_pool_shrink.
- [Low] Off-by-two loop bounds error in ttm_pool_shrinker_scan() causes redundant scanning iterations.
--
commit 3d1a95525ccb3182a802e2681e222aef27971cef
Author: Dave Airlie <[email protected]>
Subject: ttm/pool: enable memcg tracking and shrinker. (v3)
This patch introduces memory cgroup (memcg) tracking and memcg-aware shrinking for the TTM page pool. It updates the LRU lists to operate in memcg mode and allows pooled pages reparented to a higher memcg to be reused.
[ ... ]
Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p) · line 328
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_pool.c +2
-3@@ -317,12 +321,11 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,NULL ); +list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p));rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , -num_pages); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , num_pages); +mem_cgroup_move_gpu_page_reclaim (NULL , p, pt->order, true);
[Severity: High]
Can a background shrinker isolate and free this page before local metadata updates finish? Calling list_lru_add() exposes the page to the global LRU. If a shrinker frees it concurrently, the subsequent call to mem_cgroup_move_gpu_page_reclaim() will dereference page_objcg(p) on a freed page, causing a use-after-free regression.
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0} static enum lru_status take_one_from_lru(struct list_head *item,
[ ... ]
Quoted 13 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-1@@ -337,20 +340,56 @@ static enum lru_status take_one_from_lru(struct list_head *item, +static int pool_lru_get_page (struct ttm_pool_type *pt,int nid, +struct page **page_out, +struct obj_cgroup *objcg, +struct mem_cgroup *memcg) {int ret;struct page *p =NULL ;unsigned long nr_to_walk =1 ; +unsigned int num_pages =1 << pt->order; - ret =list_lru_walk_node (&pt->pages, nid, take_one_from_lru, (void *)&p, &nr_to_walk); + ret =list_lru_walk_one (&pt->pages, nid, memcg, take_one_from_lru, (void *)&p, &nr_to_walk);
drivers/gpu/drm/ttm/ttm_pool.c static enum lru_status take_one_from_lru(struct list_head *item, · line 357
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-3if (ret ==1 && p) { -atomic_long_sub (1 << pt->order, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , (1 << pt->order)); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , -(1 << pt->order)); +atomic_long_sub (num_pages, &allocated_pages[nid]); + +if (!mem_cgroup_move_gpu_page_reclaim (objcg, p, pt->order, false)) { +__free_pages (p, pt->order); + p =NULL ; + }
[Severity: Critical]
When mem_cgroup_move_gpu_page_reclaim() fails, calling __free_pages() directly bypasses ttm_pool_free_page(). Does this skip restoring x86 direct map cache attributes to Write-Back via set_pages_wb? Returning WC or UC pages to the kernel allocator could cause Machine Check Exceptions or data corruption. Additionally, if the page was allocated via the DMA API, this would leak the DMA mapping by skipping dma_free_attrs().
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +8
-1} -return p; + *page_out = p; +return ret; +} + +/* Take pages from a specific pool_type, return NULL when nothing available */ +static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid, +struct obj_cgroup *orig_objcg) +{
[ ... ]
Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5
-0+/* +* Attempt to get a page from the current memcg, but if it hasn't got any in it's level, +* go up to the parent and check there. This helps the scenario where multiple apps get +* started into their own cgroup from a common parent and want to reuse the pools. +*/
drivers/gpu/drm/ttm/ttm_pool.c static enum lru_status take_one_from_lru(struct list_head *item, · line 389
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10
-0+while (!page_out) { + ret =pool_lru_get_page (pt, nid, &page_out, orig_objcg, memcg); +if (ret ==1 ) +break ; +if (!memcg) +break ; + memcg =parent_mem_cgroup (memcg); +if (!memcg) +break ; + }
[Severity: High]
When taking an uncharged page from the root pool, does this loop pass it downstream to be incorrectly handled as uncharged? In mem_cgroup_move_gpu_page_reclaim(), an early exit for missing objcg skips charging the new_objcg. This would allow an accounted buffer object to steal uncharged pages from the root pool without being charged for them, bypassing container memory limits.
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, · line 402
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_pool.c +4
-1+ +mem_cgroup_put (orig_memcg); +return page_out; }/* Initialize and add a pool type to the global shrinker list */ @@ -360,7 +399,7 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, pt->pool = pool; pt->caching = caching; pt->order = order; -list_lru_init (&pt->pages); +list_lru_init_memcg (&pt->pages, mm_shrinker);
[Severity: High]
Since list_lru_init_memcg() registers the LRU lists to the global memcg_list_lrus list, do we need a corresponding list_lru_destroy() in ttm_pool_type_fini()? Without it, unloading a TTM device and freeing the pool memory could leave the LRU list linked in the global list, leading to a use-after-free regression during subsequent memcg operations.
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0spin_lock (&shrinker_lock);list_add_tail (&pt->shrinker_list, &shrinker_list);
[ ... ]
Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5
-0+/* +* This function doesn't currently check dma32, because no driver using this +* support dma32. This should be added and debugged when that changes. +*/ +static void ttm_pool_check_objcg (struct obj_cgroup *objcg)
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_fini(struct ttm_pool_type *pt) · line 459
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10
-0+{ +#ifdef CONFIG_MEMCG +int r =0 ; +struct mem_cgroup *memcg; +if (!objcg) +return ; + + memcg =get_mem_cgroup_from_objcg (objcg); +for (unsigned i =0 ; i <NR_PAGE_ORDERS ; i++) { + r =memcg_list_lru_alloc (memcg, &global_write_combined[i].pages,GFP_KERNEL );
[Severity: Medium]
Could using GFP_KERNEL here unconditionally be unsafe? The caller __ttm_pool_alloc() determines safe allocation flags (such as handling gfp_retry_mayfail), but they are ignored here. This might trigger the OOM killer unexpectedly or cause deadlocks if called from contexts where FS/IO operations are unsafe.
drivers/gpu/drm/ttm/ttm_pool.c
Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +2
-0+if (r) +break ;
[Severity: Medium]
Does ignoring the memcg_list_lru_alloc() failure here lead to a VM_WARN_ON later? If the allocation fails due to memory pressure and we break early, freeing pages to this LRU might cause list_lru_add() to fail finding the memcg-specific list. This would fall back to the parent memcg and potentially break accurate tracking.
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_fini(struct ttm_pool_type *pt) · line 465
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +4
-0+ r =memcg_list_lru_alloc (memcg, &global_uncached[i].pages,GFP_KERNEL ); +if (r) +break ; + }
[Severity: High]
Is it intentional to only pre-allocate global_write_combined and global_uncached here? The global_dma32_* pools and any device-specific caching pools appear to be missing. Pages added to these uninitialized pools might hit a VM_WARN_ON and fall back to the parent memcg, completely breaking memcg-aware shrinking for them.
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +3
-0+mem_cgroup_put (memcg); +#endif +}
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free) · line 515
Quoted 11 lines
/* Free pages using the per-node shrinker list */
drivers/gpu/drm/ttm/ttm_pool.c +3
-1-static unsigned int ttm_pool_shrink (int nid,unsigned long num_to_free) +static unsigned int ttm_pool_shrink (int nid, +struct mem_cgroup *memcg, +unsigned long num_to_free) {LIST_HEAD (dispose);struct ttm_pool_type *pt;@@ -448,7 +514,11 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)if (!pt)return 0 ;
[Severity: Medium]
If shrinker_list is empty and we return 0 early, does this leak the pool_shrink_rwsem read lock? A missing up_read() here could eventually lead to deadlocks when a writer tries to acquire the lock.
Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +5
-1- num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); +if (!memcg) { + num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); + }else { + num_pages =list_lru_walk_one (&pt->pages, nid, memcg, pool_move_to_dispose_list, &dispose, &num_to_free); + } num_pages *=1 << pt->order;
[ ... ]
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -1214,10 +1288,14 @@ static unsigned long ttm_pool_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) {unsigned long num_freed =0 ;
drivers/gpu/drm/ttm/ttm_pool.c static unsigned long ttm_pool_shrinker_scan(struct shrinker *shrink, · line 1298
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-2+int num_pools; +spin_lock (&shrinker_lock); + num_pools =list_count_nodes (&shrinker_list); +spin_unlock (&shrinker_lock); do - num_freed += ttm_pool_shrink(sc->nid, sc->nr_to_scan); - while (num_freed < sc->nr_to_scan && + num_freed += ttm_pool_shrink(sc->nid, sc->memcg, sc->nr_to_scan); + while (num_pools-- >= 0 && num_freed < sc->nr_to_scan &&
[Severity: Low]
Does the num_pools-- >= 0 condition cause redundant scanning iterations? If num_pools is 1, the loop executes for 1, 0, and -1, resulting in 3 iterations instead of 1. This would force redundant scanning of the same pools in the LRU and waste CPU cycles.
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1487
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1atomic_long_read(&allocated_pages[sc->nid])); sc->nr_scanned = num_freed;@@ -1406,7 +1484,7 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); + mm_shrinker =shrinker_alloc (SHRINKER_MEMCG_AWARE |SHRINKER_NUMA_AWARE ,"drm-ttm_pool" );
[Severity: High]
With SHRINKER_MEMCG_AWARE enabled, the shrinker is expected to report the number of objects owned by sc->memcg. However, ttm_pool_shrinker_count() unconditionally returns the total number of pages in the global pool (allocated_pages[sc->nid]). Could this falsely inflate the target memcg's apparent footprint, tricking the VM into applying massive, disproportionate reclaim pressure against it?
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0if (!mm_shrinker)return -ENOMEM ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH 10/10] xe: create a flag to enable memcg accounting for XE as well.
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Dumb buffer allocations bypass memory cgroup accounting. - [High] Internal userspace-triggered…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Dumb buffer allocations bypass memory cgroup accounting.
- [High] Internal userspace-triggered buffer allocations (seqno_bo and cgp_bo) bypass memory cgroup accounting.
- [High] Page tables dynamically allocated in asynchronous contexts bypass memory cgroup accounting. --
commit d0632a50e763992c9a7f84d8a817feb26c078e3a
Author: Maarten Lankhorst <[email protected]> Date: Mon Jul 6 05:22:39 2026 +0000
xe: create a flag to enable memcg accounting for XE as well.
This adds support for memcg accounting to ttm object used by xe driver.
Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_bo.c struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, · line 2390
Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +3
-0@@ -2384,6 +2385,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, placement = (type == ttm_bo_type_sg || bo->flags &XE_BO_FLAG_DEFER_BACKING ) ? &sys_placement : &bo->placement; + +if (bo->flags &XE_BO_FLAG_ACCOUNTED ) +ttm_bo_set_cgroup (&bo->ttm,get_obj_cgroup_from_current ());
[Severity: High]
Will get_obj_cgroup_from_current() correctly account memory when page tables are allocated dynamically?
Page tables are sometimes created asynchronously in worker threads, such as during SVM page fault handling or preempt rebinds via xe_pt_create(). In these cases, "current" refers to the kworker thread rather than the user process.
Since set_active_memcg() isn't used to restore the original user's memcg context in the worker, will get_obj_cgroup_from_current() evaluate against the worker's cgroup and bypass the user's memcg limits?
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0err =ttm_bo_init_reserved (&xe->ttm, &bo->ttm, type, placement, alignment, &ctx,NULL , resv, xe_ttm_bo_destroy);
[ ... ]
drivers/gpu/drm/xe/xe_bo.c int xe_gem_create_ioctl(struct drm_device *dev, void *data, · line 3368
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +1
-1@@ -3361,7 +3365,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, args->size & ~PAGE_MASK ))return -EINVAL ; - bo_flags =0 ; + bo_flags =XE_BO_FLAG_ACCOUNTED ;
[Severity: High]
Does xe_bo_dumb_create() also need to set this flag?
When userspace calls DRM_IOCTL_MODE_CREATE_DUMB, it is handled by xe_bo_dumb_create(), which calls xe_bo_create_user():
bo = xe_bo_create_user(xe, NULL, args->size, DRM_XE_GEM_CPU_CACHING_WC, XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) | XE_BO_FLAG_FORCE_WC | XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL);
This appears to bypass the newly added memcg accounting because XE_BO_FLAG_ACCOUNTED is not passed in the bitmask here.
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0if (args->flags &DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING ) bo_flags |=XE_BO_FLAG_DEFER_BACKING ;
[ ... ]
Quoted 5 lines
drivers/gpu/drm/xe/xe_lrc.c +0
-0
drivers/gpu/drm/xe/xe_lrc.c static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct xe_v · line 1668
Quoted 9 lines
drivers/gpu/drm/xe/xe_lrc.c +2
-1@@ -1661,7 +1661,8 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct xe_vXE_BO_FLAG_GGTT_INVALIDATE ;if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM | +XE_BO_FLAG_ACCOUNTED ; bo = xe_bo_create_pin_map_novm(xe, tile, bo_size, ttm_bo_type_kernel,
[Severity: High]
Should we also add XE_BO_FLAG_ACCOUNTED to the auxiliary buffers created for userspace queues?
Just below this allocation in xe_lrc_init(), seqno_bo is allocated without the flag:
bo = xe_bo_create_pin_map_novm(xe, tile, PAGE_SIZE, ttm_bo_type_kernel, XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_INVALIDATE | XE_BO_FLAG_SYSTEM, false);
Similarly, in xe_exec_queue_group_init(), cgp_bo is also allocated for userspace queues without XE_BO_FLAG_ACCOUNTED:
bo = xe_bo_create_pin_map_novm(xe, tile, SZ_4K, ttm_bo_type_kernel, XE_BO_FLAG_VRAM_IF_DGFX(tile) | XE_BO_FLAG_PINNED_LATE_RESTORE | XE_BO_FLAG_FORCE_USER_VRAM | XE_BO_FLAG_GGTT_INVALIDATE | XE_BO_FLAG_GGTT, false);
If these are triggered by DRM_IOCTL_XE_EXEC_QUEUE_CREATE, could an application spawn many execution queues and consume unaccounted memory?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Dave Airlie
maintainerQuoted 8 lines · Dave Airlie
This is just a repost with a number of sashiko identified problems that I fixed.
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting this landed for xe, we can drop the amdgpu changes for now if they can't get across the line.
I've put the latest code at
https://github.com/airlied/linux/tree/ttm-memcg-objcg
I've been fixing more sashiko found issues in there before I repost in a few days.
I've reordered things a little in the branch but mostly the same code.
Dave.
Quoted 23 lines
I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align with the two global vmstats. It adds an accounting flag to TTM alloc/populate, and enables memcg tracking and shrinker support in TTM.
Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail to fully understand beyond it doesn't solve all the problems we ever have had with cgroups and drm, so we shouldn't even bother, and maybe we could do it at the object level, and integrated with dmem, and android cross process accounting, but I still feel this is a good baseline.
I think this is the right layer to hook this into TTM, where we allocate memory and I think accounting for this memory in a proper way should be done.
Intel folks (Thomas/Maarten) please review and express concerns as well.
Regards,
Dave.
Thomas Hellström
maintainerQuoted 17 lines · Dave Airlie
Quoted 4 lines · Dave Airlie
This is just a repost with a number of sashiko identified problems that I fixed.
Quoted 8 lines
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting
this landed for xe, we can drop the amdgpu changes for now if they can't get
across the line.I've put the latest code at
https://github.com/airlied/linux/tree/ttm-memcg-objcg
Quoted 8 lines
I've been fixing more sashiko found issues in there before I repost in
a few days.I've reordered things a little in the branch but mostly the same code.
Dave.
I have a couple of additional patches to this series to ensure shmem memory allocated as part of shrinking or swapout ends up accounted to the same cgroup that allocated the TTM memory. It looks like recursive per-cgroup shrinking is not an issue, since somebody thought about that already.
Quoted 22 lines
Quoted 22 lines
I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align
with the two global vmstats. It adds an accounting flag to TTM alloc/populate,
and enables memcg tracking and shrinker support in TTM.Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail
to fully understand beyond it doesn't solve all the problems we ever have had
with cgroups and drm, so we shouldn't even bother, and maybe we could do it at
the object level, and integrated with dmem, and android cross process accounting,
but I still feel this is a good baseline.
Quoted 7 lines
Quoted 7 lines
I think this is the right layer to hook this into TTM, where we allocate memory
and I think accounting for this memory in a proper way should be done.Intel folks (Thomas/Maarten) please review and express concerns as well.
The initial use-case for us is being able to pin rather substantial amounts of system memory due to potential upcoming hardware features. While RDMA is using RLIMIT_MEMLOCK for this, that is per process and an attacker could easily launch a number of processes and pin all of system memory, so the thing optionally protecting from the DoS is the memcg limit. So regardless of whether we also account against RLIMIT_MEMLOCK or not, we need an additional cgroups limit.
Moving forward, there also a need of limiting the amount of allocated graphics memory only, due to the way certain apps behave when probing for available memory. :/.
I have unfortunately missed large parts of the memcg vs dmemcg discussion, but I asked Claude to fish out as much of the essentials as possible from the mailing lists so I can read up.
Thanks,
Thomas
Quoted 4 lines
Quoted 4 lines
Regards,
Dave.
Matthew Brost
maintainerRe: [PATCH 02/10] ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieQuoted 39 lines · Dave Airlie
From: Dave Airlie <[email protected]>
This flag does nothing yet, but this just changes the APIs to accept it in the future across all users.
This flag will eventually be filled out with when to account a tt populate to a memcg.
Signed-off-by Dave Airlie <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 ++- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 5 +++-- drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c | 4 ++-- drivers/gpu/drm/loongson/lsdc_ttm.c | 3 ++- drivers/gpu/drm/nouveau/nouveau_bo.c | 6 ++++-- drivers/gpu/drm/radeon/radeon_ttm.c | 3 ++- drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_pool_test.c | 16 ++++++++-------- drivers/gpu/drm/ttm/tests/ttm_tt_test.c | 12 ++++++------ drivers/gpu/drm/ttm/ttm_bo.c | 7 ++++--- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +++- drivers/gpu/drm/ttm/ttm_pool.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_tt.c | 8 +++++--- drivers/gpu/drm/vmwgfx/vmwgfx_blit.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c | 7 ++++--- drivers/gpu/drm/xe/xe_bo.c | 5 +++-- include/drm/ttm/ttm_bo.h | 1 + include/drm/ttm/ttm_device.h | 1 + include/drm/ttm/ttm_pool.h | 1 + include/drm/ttm/ttm_tt.h | 1 + 22 files changed, 63 insertions(+), 44 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo, · line 1224
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0@@ -1220,6 +1220,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,*/ static int amdgpu_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx)
Would it be better to add a field to the ttm_operation_ctx for memcg_account? Might be a little goofy if TTM internals toggle a field in ttm_operation_ctx but IMO better than adding bool argument everywhere.
Matt
Quoted 568 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-1{struct amdgpu_device *adev =amdgpu_ttm_adev (bdev);@@ -1243,7 +1244,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev, pool = &adev->mman.ttm_pools[gtt->pool_id]; else pool = &adev->mman.bdev.pool; - ret =ttm_pool_alloc (pool, ttm, ctx); + ret =ttm_pool_alloc (pool, ttm, memcg_account, ctx);if (ret)return ret;drivers/gpu/drm/i915/gem/i915_gem_ttm.c +3
-2@@ -317,6 +317,7 @@ static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,static int i915_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct i915_ttm_tt *i915_tt =container_of (ttm,typeof (*i915_tt), ttm);@@ -324,7 +325,7 @@ static int i915_ttm_tt_populate(struct ttm_device *bdev,if (i915_tt->is_shmem)return i915_ttm_tt_shmem_populate (bdev, ttm, ctx); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); }static void i915_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)@@ -815,7 +816,7 @@ static int __i915_ttm_get_pages(struct drm_i915_gem_object *obj, }if (bo->ttm && !ttm_tt_is_populated (bo->ttm)) { - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +1
-1@@ -624,7 +624,7 @@ int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,/* Populate ttm with pages if needed. Typically system memory. */ if (ttm && (dst_man->use_tt || (ttm->page_flags &TTM_TT_FLAG_SWAPPED ))) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +2
-2@@ -91,7 +91,7 @@ static int i915_ttm_backup(struct i915_gem_apply_to_region *apply, goto out_no_lock; backup_bo =i915_gem_to_ttm (backup); - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (err) goto out_no_populate;@@ -190,7 +190,7 @@ static int i915_ttm_restore(struct i915_gem_apply_to_region *apply,if (!backup_bo->resource) err =ttm_bo_validate (backup_bo,i915_ttm_sys_placement (), &ctx);if (!err) - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (!err) { err =i915_gem_obj_copy_ttm (obj, backup, pm_apply->allow_gpu, false);drivers/gpu/drm/loongson/lsdc_ttm.c +2
-1@@ -111,6 +111,7 @@ lsdc_ttm_tt_create(struct ttm_buffer_object *tbo, uint32_t page_flags)static int lsdc_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {bool slave = !!(ttm->page_flags &TTM_TT_FLAG_EXTERNAL );@@ -123,7 +124,7 @@ static int lsdc_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void lsdc_ttm_tt_unpopulate(struct ttm_device *bdev,drivers/gpu/drm/nouveau/nouveau_bo.c +4
-2@@ -1417,7 +1417,9 @@ vm_fault_t nouveau_ttm_fault_reserve_notify(struct ttm_buffer_object *bo)static int nouveau_ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {struct ttm_tt *ttm_dma = (void *)ttm;struct nouveau_drm *drm;@@ -1434,7 +1436,7 @@ nouveau_ttm_tt_populate(struct ttm_device *bdev, drm =nouveau_bdev (bdev); -return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, memcg_account, ctx); } static voiddrivers/gpu/drm/radeon/radeon_ttm.c +2
-1@@ -526,6 +526,7 @@ static struct radeon_ttm_tt *radeon_ttm_tt_to_gtt(struct radeon_device *rdev,static int radeon_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct radeon_device *rdev =radeon_get_rdev (bdev);@@ -547,7 +548,7 @@ static int radeon_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, memcg_account, ctx); }static void radeon_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +1
-1@@ -538,7 +538,7 @@ static void ttm_bo_validate_no_placement_signaled(struct kunit *test)if (params->with_ttm) { old_tt = priv->ttm_dev->funcs->ttm_tt_create (bo,0 ); -ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, &ctx); +ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, false, &ctx); bo->ttm = old_tt; }drivers/gpu/drm/ttm/tests/ttm_pool_test.c +8
-8@@ -89,7 +89,7 @@ static struct ttm_pool *ttm_pool_pre_populated(struct kunit *test,ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);@@ -157,7 +157,7 @@ static void ttm_pool_alloc_basic(struct kunit *test)KUNIT_ASSERT_EQ (test, pool->nid,NUMA_NO_NODE );KUNIT_ASSERT_EQ (test, pool->alloc_flags, params->alloc_flags); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);@@ -220,7 +220,7 @@ static void ttm_pool_alloc_basic_dma_addr(struct kunit *test)ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);@@ -253,7 +253,7 @@ static void ttm_pool_alloc_order_caching_match(struct kunit *test) tt =ttm_tt_kunit_init (test,0 , caching, size);KUNIT_ASSERT_NOT_NULL (test, tt); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));@@ -285,7 +285,7 @@ static void ttm_pool_alloc_caching_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);@@ -319,7 +319,7 @@ static void ttm_pool_alloc_order_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);@@ -349,7 +349,7 @@ static void ttm_pool_free_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx); pt = &pool->caching[caching].orders[order];KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));@@ -379,7 +379,7 @@ static void ttm_pool_free_no_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,0 ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx);ttm_pool_free (pool, tt);ttm_tt_fini (tt);drivers/gpu/drm/ttm/tests/ttm_tt_test.c +6
-6@@ -262,7 +262,7 @@ static void ttm_tt_populate_null_ttm(struct kunit *test)struct ttm_operation_ctx ctx = { };int err; - err =ttm_tt_populate (devs->ttm_dev,NULL , &ctx); + err =ttm_tt_populate (devs->ttm_dev,NULL , false, &ctx);KUNIT_ASSERT_EQ (test, err, -EINVAL ); }@@ -283,11 +283,11 @@ static void ttm_tt_populate_populated_ttm(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); populated_page = *tt->pages; - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_PTR_EQ (test, populated_page, *tt->pages); }@@ -307,7 +307,7 @@ static void ttm_tt_unpopulate_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));@@ -351,7 +351,7 @@ static void ttm_tt_swapin_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));@@ -361,7 +361,7 @@ static void ttm_tt_swapin_basic(struct kunit *test)KUNIT_ASSERT_TRUE (test, tt->page_flags &TTM_TT_FLAG_SWAPPED );/* Swapout depopulates TT, allocate pages and then swap them in */ - err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, &ctx); + err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); err =ttm_tt_swapin (tt);drivers/gpu/drm/ttm/ttm_bo.c +4
-3@@ -145,7 +145,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, goto out_err;if (mem->mem_type !=TTM_PL_SYSTEM ) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret) goto out_err; }@@ -1255,6 +1255,7 @@ void ttm_bo_tt_destroy(struct ttm_buffer_object *bo)* is set to true. */ int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_device *bdev = bo->bdev;@@ -1268,7 +1269,7 @@ int ttm_bo_populate(struct ttm_buffer_object *bo,return 0 ; swapped =ttm_tt_is_swapped (tt); - ret =ttm_tt_populate (bdev, tt, ctx); + ret =ttm_tt_populate (bdev, tt, memcg_account, ctx);if (ret)return ret;@@ -1293,7 +1294,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);ttm_bo_unreserve (bo);return ret; }drivers/gpu/drm/ttm/ttm_bo_util.c +3
-3@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man =ttm_manager_type (bdev, src_mem->mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }@@ -352,7 +352,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,BUG_ON (!ttm); - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;@@ -533,7 +533,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map)pgprot_t prot;void *vaddr; - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;drivers/gpu/drm/ttm/ttm_bo_vm.c +3
-1@@ -224,7 +224,9 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, }; ttm = bo->ttm; - err =ttm_bo_populate (bo, &ctx); + err =ttm_bo_populate (bo, + false, + &ctx);if (err) {if (err == -EINTR || err == -ERESTARTSYS || err == -EAGAIN )drivers/gpu/drm/ttm/ttm_pool.c +4
-2@@ -761,6 +761,7 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest, }static int __ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,const struct ttm_operation_ctx *ctx,struct ttm_pool_alloc_state *alloc,struct ttm_pool_tt_restore *restore)@@ -871,6 +872,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,* Returns: 0 on successe, negative error code otherwise. */ int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_pool_alloc_state alloc;@@ -880,7 +882,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,ttm_pool_alloc_state_init (tt, &alloc); -return __ttm_pool_alloc (pool, tt, ctx, &alloc,NULL ); +return __ttm_pool_alloc (pool, tt, memcg_account, ctx, &alloc,NULL ); }EXPORT_SYMBOL (ttm_pool_alloc);@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,return 0 ; } -return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); +return __ttm_pool_alloc (pool, tt, false, ctx, &alloc, restore); }/** drivers/gpu/drm/ttm/ttm_tt.c +5
-3@@ -368,7 +368,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,EXPORT_SYMBOL_FOR_TESTS_ONLY (ttm_tt_swapout);int ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {int ret;@@ -397,9 +399,9 @@ int ttm_tt_populate(struct ttm_device *bdev, }if (bdev->funcs->ttm_tt_populate) - ret = bdev->funcs->ttm_tt_populate (bdev, ttm, ctx); + ret = bdev->funcs->ttm_tt_populate (bdev, ttm, memcg_account, ctx);else - ret =ttm_pool_alloc (&bdev->pool, ttm, ctx); + ret =ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);if (ret) goto error;drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +2
-2@@ -569,13 +569,13 @@ int vmw_bo_cpu_blit(struct vmw_bo *vmw_dst,dma_resv_assert_held (src->base.resv);if (!ttm_tt_is_populated (dst->ttm)) { - ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, &ctx); + ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, false, &ctx);if (ret)return ret; }if (!ttm_tt_is_populated (src->ttm)) { - ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, &ctx); + ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, false, &ctx);if (ret)return ret; }drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +4
-3@@ -360,7 +360,8 @@ static void vmw_ttm_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)static int vmw_ttm_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm,bool memcg_account, +struct ttm_operation_ctx *ctx) {bool external = (ttm->page_flags &TTM_TT_FLAG_EXTERNAL ) !=0 ;@@ -372,7 +373,7 @@ static int vmw_ttm_populate(struct ttm_device *bdev, ttm->dma_address, ttm->num_pages); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void vmw_ttm_unpopulate(struct ttm_device *bdev,@@ -580,7 +581,7 @@ int vmw_bo_create_and_populate(struct vmw_private *dev_priv,if (unlikely (ret !=0 ))return ret; - ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, &ctx); + ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, false, &ctx);if (likely (ret ==0 )) {struct vmw_ttm_tt *vmw_tt = container_of(vbo->tbo.ttm, struct vmw_ttm_tt, dma_ttm);drivers/gpu/drm/xe/xe_bo.c +3
-2@@ -536,6 +536,7 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, }static int xe_ttm_tt_populate (struct ttm_device *ttm_dev,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) { struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err =ttm_tt_restore (ttm_dev, tt, ctx); }else {ttm_tt_clear_backed_up (tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)return err;@@ -1926,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (ttm_manager_type (tbo->bdev, tbo->resource->mem_type)->use_tt) { err =xe_bo_wait_usage_kernel (bo, ctx);if (!err) - err =ttm_bo_populate (&bo->ttm, ctx); + err =ttm_bo_populate (&bo->ttm, false, ctx); }else if (should_migrate_to_smem (bo)) {xe_assert (xe_bo_device (bo), bo->flags &XE_BO_FLAG_SYSTEM ); err =xe_bo_migrate (bo,XE_PL_TT , ctx, exec);include/drm/ttm/ttm_bo.h +1
-0@@ -475,6 +475,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp);void ttm_bo_tt_destroy (struct ttm_buffer_object *bo);int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx);int ttm_bo_setup_export (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx);include/drm/ttm/ttm_device.h +1
-0@@ -85,6 +85,7 @@ struct ttm_device_funcs {*/ int (*ttm_tt_populate)(struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/** include/drm/ttm/ttm_pool.h +1
-0@@ -78,6 +78,7 @@ struct ttm_pool { };int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx);void ttm_pool_free (struct ttm_pool *pool,struct ttm_tt *tt);include/drm/ttm/ttm_tt.h +1
-0@@ -250,6 +250,7 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,* Calls the driver method to allocate pages for a ttm */ int ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/** Signature · 1 line
2.54.0
Thomas Hellström
maintainerHi, Dave,
Quoted 32 lines · Dave Airlie
This is just a repost with a number of sashiko identified problems that I fixed.
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting
this landed for xe, we can drop the amdgpu changes for now if they can't get
across the line.I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align
with the two global vmstats. It adds an accounting flag to TTM alloc/populate,
and enables memcg tracking and shrinker support in TTM.Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail
to fully understand beyond it doesn't solve all the problems we ever have had
with cgroups and drm, so we shouldn't even bother, and maybe we could do it at
the object level, and integrated with dmem, and android cross process accounting,
but I still feel this is a good baseline.
Quoted 7 lines
I think this is the right layer to hook this into TTM, where we allocate memory
and I think accounting for this memory in a proper way should be done.Intel folks (Thomas/Maarten) please review and express concerns as well.
Some questions about the design that might belong in the cover-letter.
- First, Since from my understanding gpu shmem allocations are charged against memcg, (i915 igfx and to some extent dgfx), I think this makes sense, although it would be good to have an outlook:
1) What about system memory allocations targeting suspend / hibernate evictions, will they be charged against the root cgroup?
2) Plan for limits on total gpu pages / pinned gpu pages? Is dmemcg a fit for such limits (coexisting with memcg) or should such limits be implemented in memcg?
3) IIRC I noticed a patch for dma-heaps where dmemcg was suggested for android, since it separates GPU memory and system memory. Do we plan simething similar here, based on config options?
Thanks,
Thomas
Quoted 3 lines
Regards,
Dave.
Thomas Hellström
maintainerRe: [PATCH 01/10] memcg: add support for GPU page counters. (v5)
In reply to Dave AirlieQuoted 38 lines · Dave Airlie
From: Dave Airlie <[email protected]>
This introduces 2 new statistics and 3 new memcontrol APIs for dealing
with GPU system memory allocations.The stats corresponds to the same stats in the global vmstat, for number of active GPU pages, and number of pages in pools that can be reclaimed.
The first API charges a order of pages to a objcg, and sets the objcg on the pages like kmem does, and updates the active/reclaim statistic.
The second API uncharges a page from the obj cgroup it is currently charged
to.The third API allows moving a page to/from reclaim and between obj cgroups.
When pages are added to the pool lru, this just updates accounting. When pages are being removed from a pool lru, they can be taken from the parent objcg so this allows them to be uncharged from there and transferred
to a new child objcg.Signed-off-by Dave Airlie <[email protected]>---
v2: usememcg_node_stat_items
v3: fix null ptr dereference in uncharge
v4: AI review: fix parameter names, fix problem with reclaim moving doing wrong thing
v5: fix the build withCONFIG_MEMCG=n
---
Documentation/admin-guide/cgroup-v2.rst | 6 ++ include/linux/memcontrol.h | 34 ++++++++ mm/memcontrol.c | 104
++++++++++++++++++++++++
Quoted 1 line
3 files changed, 144 insertions(+)
Looks like there are a couple of sashiko issues remaining for this patch:
https://sashiko.dev/#/message/20260706052330.1110909-2-airlied%40gmail.com
Quoted 28 lines
Documentation/admin-guide/cgroup-v2.rst +0
-0@@ -1573,6 +1573,12 @@ The following nested keys are defined. vmalloc (npn)Amount of memory used for vmap backed memory.
Documentation/admin-guide/cgroup-v2.rst +6
-0+ gpu_active (npn) + Amount of system memory used for GPU devices. + + gpu_reclaim (npn) + Amount of system memory cached for GPU devices. +shmem
Amount of cached filesystem data that is swap- backed,
such as tmpfs, shm segments, shared anonymous mmap()sinclude/linux/memcontrol.h +0
-0@@ -1583,6 +1583,7 @@ static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb) #endif /* CONFIG_CGROUP_WRITEBACK */
Quoted 3 lines
struct sock;
+
Unrelated
Quoted 94 lines
#ifdef
CONFIG_MEMCG
extern structstatic_key_falsememcg_sockets_enabled_key; #definemem_cgroup_sockets_enabledstatic_branch_unlikely(&memcg_sockets_enabled_key)include/linux/memcontrol.h +0
-0@@ -1629,6 +1630,17 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg) } #endif
+bool mem_cgroup_charge_gpu_page(structobj_cgroup*objcg, struct page *page,include/linux/memcontrol.h +10
-0+ unsigned int order, + gfp_t gfp_mask, bool reclaim); +void mem_cgroup_uncharge_gpu_page (struct page *page, + unsigned int order, + bool reclaim); +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *objcg, + struct page *page, + unsigned int order, + bool to_reclaim); +int
alloc_shrinker_info(struct mem_cgroup *memcg); voidfree_shrinker_info(struct mem_cgroup *memcg); voidset_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);include/linux/memcontrol.h +0
-0@@ -1665,6 +1677,28 @@ static inline void mem_cgroup_sk_uncharge(const struct sock *sk, { }
+static inline bool mem_cgroup_charge_gpu_page(structobj_cgroup*objcg,include/linux/memcontrol.h +3
-0+ struct page *page, + unsigned int order, + gfp_t gfp_mask, boolreclaim)
include/linux/memcontrol.h +11
-0+{ +return true; +} + +static inline void mem_cgroup_uncharge_gpu_page (struct page *page, +unsigned int order, +bool reclaim) +{ +} + +static inline bool mem_cgroup_move_gpu_page_reclaim(struct
obj_cgroup*objcg,include/linux/memcontrol.h +1
-0+ struct page*page,
include/linux/memcontrol.h +1
-0+ unsigned intorder,
include/linux/memcontrol.h +5
-0+ bool to_reclaim) +{ +return true; +} +static inline void
set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id){mm/memcontrol.c +2
-0@@ -423,6 +423,8 @@ static const unsigned int memcg_node_stat_items[] = { #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif +NR_GPU_ACTIVE , +NR_GPU_RECLAIM , };
static const unsigned int memcg_stat_items[] = {mm/memcontrol.c +0
-0@@ -1553,6 +1555,8 @@ static const struct memory_stat memory_stats[] = { { "percpu", MEMCG_PERCPU_B }, { "sock", MEMCG_SOCK }, {"vmalloc",
NR_VMALLOC},mm/memcontrol.c +1
-0+ {"
gpu_active",NR_GPU_ACTIVE},mm/memcontrol.c +1
-0+ {"
gpu_reclaim",NR_GPU_RECLAIM},
{
"shmem",NR_SHMEM},
#ifdefCONFIG_ZSWAP
{
"zswap",MEMCG_ZSWAP_B},mm/memcontrol.c +0
-0@@ -5508,6 +5512,106 @@ void mem_cgroup_flush_workqueue(void) flush_workqueue(memcg_wq); }
Quoted 3 lines
+/**
+ *mem_cgroup_charge_gpu_page- charge a page to GPU memory tracking
Nit: Patch guidelines call for ()
Quoted 5 lines
+ * @objcg: objcg to charge, NULL charges root memcg
+ * @page: page to charge
+ * @order: page allocation order
+ * @gfp_mask: gfp mode
+ * @reclaim: charge the reclaim counter instead of the active one.
Return:
Quoted 69 lines
+ *
+ * Charge the order sized @page to the objcg. Returns %true if the charge fit within
+ * @objcg's configured limit, %false if it doesn't.
+ */
+bool mem_cgroup_charge_gpu_page(structobj_cgroup*objcg, struct page *page,mm/memcontrol.c +1
-0+ unsigned int order, gfp_t gfp_mask,bool reclaim)
mm/memcontrol.c +21
-0+{ +unsigned int nr_pages =1 << order; +struct mem_cgroup *memcg =NULL ; +struct lruvec *lruvec; +int ret; + +if (objcg) { + memcg =get_mem_cgroup_from_objcg (objcg); + + ret =try_charge_memcg (memcg, gfp_mask, nr_pages); +if (ret) { +mem_cgroup_put (memcg); +return false; + } + +obj_cgroup_get (objcg); +page_set_objcg (page, objcg); + } + + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); + mod_lruvec_state(lruvec, reclaim ? NR_GPU_RECLAIM :
NR_GPU_ACTIVE,nr_pages);mm/memcontrol.c +8
-0+ +mem_cgroup_put (memcg); +return true; +} +EXPORT_SYMBOL_GPL (mem_cgroup_charge_gpu_page); + +/** +* mem_cgroup_uncharge_gpu_page - uncharge a page from GPU memory tracking
+ * @page: page to uncharge
+ * @order: order of the page allocation
+ * @reclaim: uncharge the reclaim counter instead of the active.
+ */mm/memcontrol.c +12
-0+void mem_cgroup_uncharge_gpu_page (struct page *page, + unsigned int order, bool reclaim) +{ +struct obj_cgroup *objcg =page_objcg (page); +struct mem_cgroup *memcg; +struct lruvec *lruvec; +int nr_pages =1 << order; + + memcg = objcg ?get_mem_cgroup_from_objcg (objcg) :NULL ; + + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); + mod_lruvec_state(lruvec, reclaim ? NR_GPU_RECLAIM :
NR_GPU_ACTIVE, -nr_pages);mm/memcontrol.c +11
-0+ +if (memcg && !mem_cgroup_is_root (memcg)) +refill_stock (memcg, nr_pages); + page->memcg_data =0 ; +obj_cgroup_put (objcg); +mem_cgroup_put (memcg); +} +EXPORT_SYMBOL_GPL (mem_cgroup_uncharge_gpu_page); + +/** +* mem_cgroup_move_gpu_reclaim - move pages from gpu to gpu reclaim
Quoted 4 lines
and back
+ * @new_objcg: objcg to move page to, NULL if just stats update.
+ * @nr_pages: number of pages to move
+ * @to_reclaim: true moves pages into reclaim, false moves them back
Return:
Quoted 35 lines
+ */
mm/memcontrol.c +9
-0+bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *new_objcg, + struct page *page, + unsigned int order, + bool to_reclaim) +{ +struct obj_cgroup *objcg =page_objcg (page); + +if (!objcg || !new_objcg || objcg == new_objcg) { + struct mem_cgroup *memcg = objcg ?
get_mem_cgroup_from_objcg(objcg): NULL;mm/memcontrol.c +7
-0+struct lruvec *lruvec; +unsigned long flags; +int nr_pages =1 << order; + + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); +local_irq_save (flags); + mod_lruvec_state(lruvec, to_reclaim ? NR_GPU_RECLAIM:
NR_GPU_ACTIVE,nr_pages);mm/memcontrol.c +1
-0+ mod_lruvec_state(lruvec, to_reclaim ? NR_GPU_ACTIVE:
NR_GPU_RECLAIM, -nr_pages);mm/memcontrol.c +6
-0+local_irq_restore (flags); +mem_cgroup_put (memcg); +return true; + }else { +mem_cgroup_uncharge_gpu_page (page, order, true); + return mem_cgroup_charge_gpu_page(new_objcg, page,order, 0, false);
mm/memcontrol.c +4
-0+ } +} +EXPORT_SYMBOL_GPL (mem_cgroup_move_gpu_page_reclaim); +static int
__initcgroup_memory(char *s)
{
char *token;
Thomas Hellström
maintainerRe: [PATCH 02/10] ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieQuoted 7 lines · Dave Airlie
From: Dave Airlie <[email protected]>
This flag does nothing yet, but this just changes the APIs to accept it in the future across all users.
This flag will eventually be filled out with when to account a tt populate to a memcg.
I agree with Matt Brost's comment about the ttm_operation_ctx.
There are also some sashiko comments that appear valid:
https://sashiko.dev/#/message/20260706053512.D207E1F000E9%40smtp.kernel.org
Thanks,
Thomas
Quoted 490 lines
Signed-off-by Dave Airlie <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 ++- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 5 +++-- drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c | 4 ++-- drivers/gpu/drm/loongson/lsdc_ttm.c | 3 ++- drivers/gpu/drm/nouveau/nouveau_bo.c | 6 ++++-- drivers/gpu/drm/radeon/radeon_ttm.c | 3 ++-drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c| 2 +- drivers/gpu/drm/ttm/tests/ttm_pool_test.c | 16 ++++++++------ --
drivers/gpu/drm/ttm/tests/ttm_tt_test.c | 12 ++++++------ drivers/gpu/drm/ttm/ttm_bo.c | 7 ++++--- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +++- drivers/gpu/drm/ttm/ttm_pool.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_tt.c | 8 +++++--- drivers/gpu/drm/vmwgfx/vmwgfx_blit.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c | 7 ++++--- drivers/gpu/drm/xe/xe_bo.c | 5 +++-- include/drm/ttm/ttm_bo.h | 1 + include/drm/ttm/ttm_device.h | 1 + include/drm/ttm/ttm_pool.h | 1 + include/drm/ttm/ttm_tt.h | 1 +22 files changed, 63 insertions(+), 44 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0@@ -1220,6 +1220,7 @@ static struct ttm_tt*amdgpu_ttm_tt_create(struct ttm_buffer_object *bo, */ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, + bool memcg_account, struct ttm_operation_ctx *ctx) {struct
amdgpu_device*adev =amdgpu_ttm_adev(bdev);drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0
-0@@ -1243,7 +1244,7 @@ static int amdgpu_ttm_tt_populate(structttm_device *bdev, pool = &adev->mman.ttm_pools[gtt->pool_id];else
pool =&adev->mman.bdev.pool;drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-1- ret =ttm_pool_alloc (pool, ttm, ctx); + ret =ttm_pool_alloc (pool, ttm, memcg_account, ctx);if (ret)
return ret;drivers/gpu/drm/i915/gem/i915_gem_ttm.c +0
-0@@ -317,6 +317,7 @@ static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,
static int i915_ttm_tt_populate(structttm_device*bdev, structttm_tt*ttm,drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1
-0+ bool memcg_account,struct
ttm_operation_ctx*ctx) {
structi915_ttm_tt*i915_tt=container_of(ttm, typeof(*i915_tt), ttm);drivers/gpu/drm/i915/gem/i915_gem_ttm.c +0
-0@@ -324,7 +325,7 @@ static int i915_ttm_tt_populate(struct ttm_device *bdev, if (i915_tt->is_shmem) return i915_ttm_tt_shmem_populate(bdev, ttm, ctx);drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1
-1-return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);}
static voidi915_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)drivers/gpu/drm/i915/gem/i915_gem_ttm.c +0
-0@@ -815,7 +816,7 @@ static int __i915_ttm_get_pages(struct drm_i915_gem_object *obj, }
if (bo->ttm&& !ttm_tt_is_populated(bo->ttm)) {drivers/gpu/drm/i915/gem/i915_gem_ttm.c +1
-1- ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)
return ret;drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +0
-0@@ -624,7 +624,7 @@ int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
/* Populate ttm with pages if needed. Typically system memory. */
if (ttm && (dst_man->use_tt|| (ttm->page_flags&TTM_TT_FLAG_SWAPPED))) {drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +1
-1- ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)
return ret;
}drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +0
-0@@ -91,7 +91,7 @@ static int i915_ttm_backup(structi915_gem_apply_to_region *apply, goto out_no_lock;
backup_bo=i915_gem_to_ttm(backup);drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +1
-1- err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (err)
gotoout_no_populate;drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +1
-1@@ -190,7 +190,7 @@ static int i915_ttm_restore(struct i915_gem_apply_to_region *apply, if (!backup_bo->resource) err = ttm_bo_validate(backup_bo,i915_ttm_sys_placement (), &ctx); if (!err) - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx); if (!err) {err = i915_gem_obj_copy_ttm(obj, backup, pm_apply-
Quoted 1 line
allow_gpu,false);
drivers/gpu/drm/loongson/lsdc_ttm.c +0
-0@@ -111,6 +111,7 @@ lsdc_ttm_tt_create(struct ttm_buffer_object *tbo, uint32_t page_flags)
static int lsdc_ttm_tt_populate(structttm_device*bdev, structttm_tt*ttm,drivers/gpu/drm/loongson/lsdc_ttm.c +1
-0+ bool memcg_account,struct
ttm_operation_ctx*ctx) {
bool slave = !!(ttm->page_flags &TTM_TT_FLAG_EXTERNAL);drivers/gpu/drm/loongson/lsdc_ttm.c +0
-0@@ -123,7 +124,7 @@ static int lsdc_ttm_tt_populate(struct ttm_device *bdev, return 0; }drivers/gpu/drm/loongson/lsdc_ttm.c +1
-1-return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);}
static void lsdc_ttm_tt_unpopulate(structttm_device*bdev,drivers/gpu/drm/nouveau/nouveau_bo.c +0
-0@@ -1417,7 +1417,9 @@ vm_fault_t nouveau_ttm_fault_reserve_notify(struct ttm_buffer_object *bo)
static int
nouveau_ttm_tt_populate(structttm_device*bdev,drivers/gpu/drm/nouveau/nouveau_bo.c +0
-1- struct ttm_tt *ttm, struct ttm_operation_ctx*ctx)
drivers/gpu/drm/nouveau/nouveau_bo.c +3
-0+struct ttm_tt *ttm, + bool memcg_account, + struct ttm_operation_ctx *ctx){
structttm_tt*ttm_dma= (void *)ttm;
structnouveau_drm*drm;drivers/gpu/drm/nouveau/nouveau_bo.c +0
-0@@ -1434,7 +1436,7 @@ nouveau_ttm_tt_populate(struct ttm_device *bdev,
drm =nouveau_bdev(bdev);drivers/gpu/drm/nouveau/nouveau_bo.c +1
-1-return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, ctx); + return ttm_pool_alloc(&drm->ttm.bdev.pool, ttm,
memcg_account, ctx);
}
static voiddrivers/gpu/drm/radeon/radeon_ttm.c +0
-0@@ -526,6 +526,7 @@ static struct radeon_ttm_tt *radeon_ttm_tt_to_gtt(struct radeon_device *rdev,
static int radeon_ttm_tt_populate(structttm_device*bdev, structttm_tt*ttm,drivers/gpu/drm/radeon/radeon_ttm.c +1
-0+ bool memcg_account,struct
ttm_operation_ctx*ctx) {
structradeon_device*rdev =radeon_get_rdev(bdev);drivers/gpu/drm/radeon/radeon_ttm.c +0
-0@@ -547,7 +548,7 @@ static int radeon_ttm_tt_populate(struct ttm_device *bdev, return 0; }drivers/gpu/drm/radeon/radeon_ttm.c +1
-1-return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, ctx); + return ttm_pool_alloc(&rdev->mman.bdev.pool, ttm,
memcg_account, ctx);
}
static voidradeon_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +0
-0@@ -538,7 +538,7 @@ static void ttm_bo_validate_no_placement_signaled(struct kunit *test)
if (params->with_ttm) {
old_tt=priv->ttm_dev->funcs->ttm_tt_create(bo, 0);drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +1
-1-ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, &ctx); + ttm_pool_alloc(&priv->ttm_dev->pool, old_tt, false,&ctx);
bo->ttm=old_tt;
}drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -89,7 +89,7 @@ static struct ttm_pool *ttm_pool_pre_populated(struct kunit *test,
ttm_pool_init(pool, devs->dev, NUMA_NO_NODE, TTM_ALLOCATION_POOL_USE_DMA_ALLOC);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);
ttm_pool_free(pool, tt);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -157,7 +157,7 @@ static void ttm_pool_alloc_basic(struct kunit *test) KUNIT_ASSERT_EQ(test, pool->nid, NUMA_NO_NODE);KUNIT_ASSERT_EQ(test,
pool->alloc_flags, params-Quoted 1 line
alloc_flags);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);KUNIT_ASSERT_EQ(test, tt->num_pages, expected_num_pages);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -220,7 +220,7 @@ static void ttm_pool_alloc_basic_dma_addr(struct kunit *test)
ttm_pool_init(pool, devs->dev, NUMA_NO_NODE, TTM_ALLOCATION_POOL_USE_DMA_ALLOC);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);KUNIT_ASSERT_EQ(test, tt->num_pages, expected_num_pages);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -253,7 +253,7 @@ static void ttm_pool_alloc_order_caching_match(struct kunit *test) tt = ttm_tt_kunit_init(test, 0, caching, size); KUNIT_ASSERT_NOT_NULL(test, tt);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);
KUNIT_ASSERT_TRUE(test, !list_lru_count(&pt->pages));drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -285,7 +285,7 @@ static void ttm_pool_alloc_caching_mismatch(struct kunit *test) KUNIT_ASSERT_FALSE(test, !list_lru_count(&pt_pool->pages)); KUNIT_ASSERT_TRUE(test, !list_lru_count(&pt_tt->pages));drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);
ttm_pool_free(pool, tt);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -319,7 +319,7 @@ static void ttm_pool_alloc_order_mismatch(struct kunit *test) KUNIT_ASSERT_FALSE(test, !list_lru_count(&pt_pool->pages)); KUNIT_ASSERT_TRUE(test, !list_lru_count(&pt_tt->pages));drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1- err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);
KUNIT_ASSERT_EQ(test, err, 0);
ttm_pool_free(pool, tt);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -349,7 +349,7 @@ static void ttm_pool_free_dma_alloc(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, pool);
ttm_pool_init(pool, devs->dev, NUMA_NO_NODE, TTM_ALLOCATION_POOL_USE_DMA_ALLOC);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1-ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx);
pt = &pool->caching[caching].orders[order];KUNIT_ASSERT_TRUE(test, !list_lru_count(&pt->pages));drivers/gpu/drm/ttm/tests/ttm_pool_test.c +0
-0@@ -379,7 +379,7 @@ static void ttm_pool_free_no_dma_alloc(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, pool);
ttm_pool_init(pool, devs->dev, NUMA_NO_NODE, 0);drivers/gpu/drm/ttm/tests/ttm_pool_test.c +1
-1-ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx);
ttm_pool_free(pool, tt);
ttm_tt_fini(tt);drivers/gpu/drm/ttm/tests/ttm_tt_test.c +0
-0@@ -262,7 +262,7 @@ static void ttm_tt_populate_null_ttm(struct kunit *test) struct ttm_operation_ctx ctx = { }; int err;drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_tt_populate (devs->ttm_dev,NULL , &ctx); + err =ttm_tt_populate (devs->ttm_dev,NULL , false, &ctx);
KUNIT_ASSERT_EQ(test, err, -EINVAL);
}drivers/gpu/drm/ttm/tests/ttm_tt_test.c +0
-0@@ -283,11 +283,11 @@ static void ttm_tt_populate_populated_ttm(struct kunit *test) err = ttm_tt_init(tt, bo, 0, ttm_cached, 0); KUNIT_ASSERT_EQ(test, err, 0);drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);
KUNIT_ASSERT_EQ(test, err, 0);
populated_page=*tt->pages;drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);
KUNIT_ASSERT_PTR_EQ(test, populated_page, *tt->pages); }drivers/gpu/drm/ttm/tests/ttm_tt_test.c +0
-0@@ -307,7 +307,7 @@ static void ttm_tt_unpopulate_basic(struct kunit *test) err = ttm_tt_init(tt, bo, 0, ttm_cached, 0); KUNIT_ASSERT_EQ(test, err, 0);drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);
KUNIT_ASSERT_EQ(test, err, 0);KUNIT_ASSERT_TRUE(test, ttm_tt_is_populated(tt));drivers/gpu/drm/ttm/tests/ttm_tt_test.c +0
-0@@ -351,7 +351,7 @@ static void ttm_tt_swapin_basic(struct kunit *test) err = ttm_tt_init(tt, bo, 0, ttm_cached, 0); KUNIT_ASSERT_EQ(test, err, 0);drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);
KUNIT_ASSERT_EQ(test, err, 0);KUNIT_ASSERT_TRUE(test, ttm_tt_is_populated(tt));drivers/gpu/drm/ttm/tests/ttm_tt_test.c +0
-0@@ -361,7 +361,7 @@ static void ttm_tt_swapin_basic(struct kunit *test)
KUNIT_ASSERT_TRUE(test, tt->page_flags & TTM_TT_FLAG_SWAPPED);
/* Swapout depopulates TT, allocate pages and then swap them in */drivers/gpu/drm/ttm/tests/ttm_tt_test.c +1
-1- err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, &ctx); + err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, false, &ctx);
KUNIT_ASSERT_EQ(test, err, 0);
err =ttm_tt_swapin(tt);drivers/gpu/drm/ttm/ttm_bo.c +0
-0@@ -145,7 +145,7 @@ static int ttm_bo_handle_move_mem(structttm_buffer_object *bo, goto out_err;
if (mem->mem_type!=TTM_PL_SYSTEM) {drivers/gpu/drm/ttm/ttm_bo.c +1
-1- ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)
gotoout_err;
}drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -1255,6 +1255,7 @@ void ttm_bo_tt_destroy(struct ttm_buffer_object*bo) * is set to true. */ int ttm_bo_populate(struct ttm_buffer_object *bo, + bool memcg_account, struct ttm_operation_ctx *ctx) {struct
ttm_device*bdev =bo->bdev;drivers/gpu/drm/ttm/ttm_bo.c +0
-0@@ -1268,7 +1269,7 @@ int ttm_bo_populate(struct ttm_buffer_object *bo, return 0;
swapped =ttm_tt_is_swapped(tt);drivers/gpu/drm/ttm/ttm_bo.c +1
-1- ret =ttm_tt_populate (bdev, tt, ctx); + ret =ttm_tt_populate (bdev, tt, memcg_account, ctx);if (ret)
return ret;drivers/gpu/drm/ttm/ttm_bo.c +0
-0@@ -1293,7 +1294,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo, if (ret != 0) return ret;drivers/gpu/drm/ttm/ttm_bo.c +1
-1- ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);
ttm_bo_unreserve(bo);
return ret;
}drivers/gpu/drm/ttm/ttm_bo_util.c +0
-0@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man = ttm_manager_type(bdev, src_mem->mem_type);if (ttm && ((
ttm->page_flags&TTM_TT_FLAG_SWAPPED) ||dst_man->use_tt)) {drivers/gpu/drm/ttm/ttm_bo_util.c +1
-1- ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)
return ret;
}drivers/gpu/drm/ttm/ttm_bo_util.c +0
-0@@ -352,7 +352,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,
BUG_ON(!ttm);drivers/gpu/drm/ttm/ttm_bo_util.c +1
-1- ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)
return ret;drivers/gpu/drm/ttm/ttm_bo_util.c +0
-0@@ -533,7 +533,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map) pgprot_t prot; void *vaddr;drivers/gpu/drm/ttm/ttm_bo_util.c +1
-1- ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)
return ret;drivers/gpu/drm/ttm/ttm_bo_vm.c +0
-0@@ -224,7 +224,9 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, };
ttm =bo->ttm;drivers/gpu/drm/ttm/ttm_bo_vm.c +3
-1- err =ttm_bo_populate (bo, &ctx); + err =ttm_bo_populate (bo, + false, + &ctx);if (err) {
if (err ==-EINTR|| err ==-ERESTARTSYS|| err ==-EAGAIN)drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -761,6 +761,7 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest, }
static int __ttm_pool_alloc(structttm_pool*pool, structttm_tt*tt,drivers/gpu/drm/ttm/ttm_pool.c +1
-0+ bool memcg_account,const struct
ttm_operation_ctx*ctx, structttm_pool_alloc_state*alloc, structttm_pool_tt_restore*restore)
drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool · line 876
Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-0@@ -871,6 +872,7 @@ static int __ttm_pool_alloc(struct ttm_pool*pool, struct ttm_tt *tt, * Returns: 0 on successe, negative error code otherwise. */ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, + bool memcg_account,
Regardless whether memcg_account is passed as a separate parameter or part of @ctx, needs kerneldoc update.
Thanks,
Thomas
Quoted 213 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0struct ttm_operation_ctx *ctx) {struct
ttm_pool_alloc_statealloc;drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -880,7 +882,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
ttm_pool_alloc_state_init(tt, &alloc);drivers/gpu/drm/ttm/ttm_pool.c +1
-1-return __ttm_pool_alloc (pool, tt, ctx, &alloc,NULL ); + return __ttm_pool_alloc(pool, tt, memcg_account, ctx,&alloc, NULL);
}
EXPORT_SYMBOL(ttm_pool_alloc);drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt, return 0; }drivers/gpu/drm/ttm/ttm_pool.c +1
-1-return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); + return __ttm_pool_alloc(pool, tt, false, ctx, &alloc,restore);
}
/**drivers/gpu/drm/ttm/ttm_tt.c +0
-0@@ -368,7 +368,9 @@ int ttm_tt_swapout(struct ttm_device *bdev,struct ttm_tt *ttm, EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_swapout);
int ttm_tt_populate(structttm_device*bdev,drivers/gpu/drm/ttm/ttm_tt.c +0
-1- struct ttm_tt *ttm, struct ttm_operation_ctx*ctx)
drivers/gpu/drm/ttm/ttm_tt.c +3
-0+ struct ttm_tt *ttm, + bool memcg_account, + struct ttm_operation_ctx *ctx){
int ret;drivers/gpu/drm/ttm/ttm_tt.c +0
-0@@ -397,9 +399,9 @@ int ttm_tt_populate(struct ttm_device *bdev, }
if (bdev->funcs->ttm_tt_populate)drivers/gpu/drm/ttm/ttm_tt.c +1
-1- ret = bdev->funcs->ttm_tt_populate (bdev, ttm, ctx); + ret = bdev->funcs->ttm_tt_populate(bdev, ttm,
memcg_account, ctx);
elsedrivers/gpu/drm/ttm/ttm_tt.c +1
-1- ret =ttm_pool_alloc (&bdev->pool, ttm, ctx); + ret = ttm_pool_alloc(&bdev->pool, ttm,
memcg_account, ctx);
if (ret)
goto error;drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +0
-0@@ -569,13 +569,13 @@ int vmw_bo_cpu_blit(struct vmw_bo *vmw_dst, dma_resv_assert_held(src->base.resv);
if (!ttm_tt_is_populated(dst->ttm)) {drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +0
-1- ret = dst->bdev->funcs->ttm_tt_populate(dst->bdev,
dst->ttm, &ctx);drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +1
-0+ ret = dst->bdev->funcs->ttm_tt_populate(dst->bdev,
dst->ttm, false, &ctx);
if (ret)
return ret;
}
if (!ttm_tt_is_populated(src->ttm)) {drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +0
-1- ret = src->bdev->funcs->ttm_tt_populate(src->bdev,
src->ttm, &ctx);drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +1
-0+ ret = src->bdev->funcs->ttm_tt_populate(src->bdev,
src->ttm, false, &ctx);
if (ret)
return ret;
}drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +0
-0@@ -360,7 +360,8 @@ static void vmw_ttm_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
static int vmw_ttm_populate(structttm_device*bdev,drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +0
-1- struct ttm_tt *ttm, struct
ttm_operation_ctx*ctx)drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +2
-0+ struct ttm_tt *ttm, bool memcg_account, + struct ttm_operation_ctx *ctx){
bool external = (ttm->page_flags&TTM_TT_FLAG_EXTERNAL) != 0;drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +0
-0@@ -372,7 +373,7 @@ static int vmw_ttm_populate(struct ttm_device *bdev,ttm-
Quoted 1 line
dma_address,ttm-
Quoted 1 line
num_pages);drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +1
-1-return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);}
static void vmw_ttm_unpopulate(structttm_device*bdev,drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +0
-0@@ -580,7 +581,7 @@ int vmw_bo_create_and_populate(struct vmw_private *dev_priv, if (unlikely(ret != 0)) return ret;drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +1
-1- ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, &ctx); + ret = vmw_ttm_populate(vbo->tbo.bdev, vbo->tbo.ttm, false,&ctx);
if (likely(ret == 0)) {
structvmw_ttm_tt*vmw_tt=container_of(vbo->tbo.ttm, struct vmw_ttm_tt, dma_ttm);drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -536,6 +536,7 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, }
static int xe_ttm_tt_populate(structttm_device*ttm_dev, structttm_tt*tt,drivers/gpu/drm/xe/xe_bo.c +1
-0+ bool memcg_account,struct
ttm_operation_ctx*ctx) {
structxe_ttm_tt*xe_tt=container_of(tt, struct xe_ttm_tt, ttm);drivers/gpu/drm/xe/xe_bo.c +1
-1@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err = ttm_tt_restore(ttm_dev, tt, ctx); } else { ttm_tt_clear_backed_up(tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)
return err;drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -1926,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (
ttm_manager_type(tbo->bdev, tbo->resource->mem_type)-Quoted 1 line
use_tt) {err =
xe_bo_wait_usage_kernel(bo, ctx); if (!err)drivers/gpu/drm/xe/xe_bo.c +1
-1- err =ttm_bo_populate (&bo->ttm, ctx); + err =ttm_bo_populate (&bo->ttm, false, ctx);} else if (
should_migrate_to_smem(bo)) {xe_assert(xe_bo_device(bo), bo->flags & XE_BO_FLAG_SYSTEM);
err =xe_bo_migrate(bo, XE_PL_TT, ctx, exec);include/drm/ttm/ttm_bo.h +1
-0@@ -475,6 +475,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp); void ttm_bo_tt_destroy(struct ttm_buffer_object *bo); int ttm_bo_populate(struct ttm_buffer_object *bo, + bool memcg_account, struct ttm_operation_ctx *ctx); int ttm_bo_setup_export(struct ttm_buffer_object *bo,struct
ttm_operation_ctx*ctx);include/drm/ttm/ttm_device.h +1
-0@@ -85,6 +85,7 @@ struct ttm_device_funcs {*/ int (*ttm_tt_populate)(struct ttm_device *bdev, struct ttm_tt *ttm, + bool memcg_account, struct ttm_operation_ctx *ctx);
/**include/drm/ttm/ttm_pool.h +0
-0@@ -78,6 +78,7 @@ struct ttm_pool { };
int ttm_pool_alloc(structttm_pool*pool, structttm_tt*tt,include/drm/ttm/ttm_pool.h +1
-0+ bool memcg_account,struct
ttm_operation_ctx*ctx); voidttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);include/drm/ttm/ttm_tt.h +1
-0@@ -250,6 +250,7 @@ int ttm_tt_swapout(struct ttm_device *bdev,struct ttm_tt *ttm, * Calls the driver method to allocate pages for a ttm */ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, + bool memcg_account, struct ttm_operation_ctx *ctx);
/**
Thomas Hellström
maintainerRe: [PATCH 03/10] ttm/pool: initialise the shrinker earlier (v2)
In reply to Dave AirlieQuoted 11 lines · Dave Airlie
From: Dave Airlie <[email protected]>
Later memcg enablement needs the shrinker initialised before the list lru,
Just move it for now, but also handle the list being uninitialised.Signed-off-by Dave Airlie <[email protected]>---
v2: sashiko identified a problem with the list handling.
---
Couple of seemingly valid Sashiko issues:
https://sashiko.dev/#/message/20260706053333.AEABE1F000E9%40smtp.kernel.org
Thanks,
Thomas
Quoted 67 lines
drivers/gpu/drm/ttm/ttm_pool.c | 30 +++++++++++++++++-------------
1 file changed, 17 insertions(+), 13 deletions(-)drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -440,10 +440,14 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
down_read(&pool_shrink_rwsem);
spin_lock(&shrinker_lock);drivers/gpu/drm/ttm/ttm_pool.c +0
-1- pt = list_first_entry(&shrinker_list, typeof(*pt),
shrinker_list);drivers/gpu/drm/ttm/ttm_pool.c +1
-1-list_move_tail (&pt->shrinker_list, &shrinker_list); + pt = list_first_entry_or_null(&shrinker_list, typeof(*pt),
shrinker_list);drivers/gpu/drm/ttm/ttm_pool.c +2
-0+if (pt) +list_move_tail (&pt->shrinker_list, &shrinker_list);
spin_unlock(&shrinker_lock);drivers/gpu/drm/ttm/ttm_pool.c +3
-0+if (!pt) +return 0 ; +
num_pages=list_lru_walk_node(&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free);num_pages*= 1 <<pt->order;drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -1402,6 +1406,17 @@ int ttm_pool_mgr_init(unsigned long num_pages) spin_lock_init(&shrinker_lock); INIT_LIST_HEAD(&shrinker_list);drivers/gpu/drm/ttm/ttm_pool.c +1
-0+ mm_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "drm-
ttm_pool");drivers/gpu/drm/ttm/ttm_pool.c +10
-0+if (!mm_shrinker) +return -ENOMEM ; + + mm_shrinker->count_objects = ttm_pool_shrinker_count; + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; + mm_shrinker->batch =TTM_SHRINKER_BATCH ; + mm_shrinker->seeks =1 ; + +shrinker_register (mm_shrinker); +for (i = 0; i <
NR_PAGE_ORDERS; ++i) {ttm_pool_type_init(&global_write_combined[i], NULL, ttm_write_combined, i);drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -1424,17 +1439,6 @@ int ttm_pool_mgr_init(unsigned long num_pages) #endif #endifdrivers/gpu/drm/ttm/ttm_pool.c +0
-1- mm_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "drm-
ttm_pool");drivers/gpu/drm/ttm/ttm_pool.c +0
-10-if (!mm_shrinker) -return -ENOMEM ; - - mm_shrinker->count_objects = ttm_pool_shrinker_count; - mm_shrinker->scan_objects = ttm_pool_shrinker_scan; - mm_shrinker->batch =TTM_SHRINKER_BATCH ; - mm_shrinker->seeks =1 ; - -shrinker_register (mm_shrinker); -return 0;
}
Dave Airlie
maintainerQuoted 44 lines · Thomas Hellström
Hi, Dave,
Quoted 32 lines · Dave Airlie
This is just a repost with a number of sashiko identified problems that I fixed.
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting
this landed for xe, we can drop the amdgpu changes for now if they can't get
across the line.I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align
with the two global vmstats. It adds an accounting flag to TTM alloc/populate,
and enables memcg tracking and shrinker support in TTM.Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail
to fully understand beyond it doesn't solve all the problems we ever have had
with cgroups and drm, so we shouldn't even bother, and maybe we could do it at
the object level, and integrated with dmem, and android cross process accounting,
but I still feel this is a good baseline.Quoted 7 lines
I think this is the right layer to hook this into TTM, where we allocate memory
and I think accounting for this memory in a proper way should be done.Intel folks (Thomas/Maarten) please review and express concerns as well.
Quoted 8 lines
Some questions about the design that might belong in the cover-letter.
- First, Since from my understanding gpu shmem allocations are charged against memcg, (i915 igfx and to some extent dgfx), I think this makes sense, although it would be good to have an outlook:
1) What about system memory allocations targeting suspend / hibernate evictions, will they be charged against the root cgroup?
This was a problem in my first memory tracking patchset, I didn't want to track these allocations against the stats, but Matt Brost fixed some problems with this idea. I think maybe charging against the root cgroup is the proper option here.
Quoted 4 lines
2) Plan for limits on total gpu pages / pinned gpu pages? Is dmemcg a fit for such limits (coexisting with memcg) or should such limits be implemented in memcg?
I was just letting memcg control this, at least account first, then work out how to limit within how things work now, I'm not sure how dmemcg fits in here, I need to also look into the other discussions and try and workout where dmemcg could be used.
Quoted 4 lines
3) IIRC I noticed a patch for dma-heaps where dmemcg was suggested for android, since it separates GPU memory and system memory. Do we plan simething similar here, based on config options?
I think that is Christian's concern, that we shouldn't do this at all here, and we should do it with dmemcg instead, but it seemed a bit handwavy and I'm not sure how to turn it into a concrete way forward.
Dave.
Re: [PATCH 05/10] ttm/pool: enable memcg tracking and shrinker. (v3)
In reply to Dave AirlieQuoted 276 lines · Dave Airlie
From: Dave Airlie <[email protected]>
This enables all the backend code to use the list lru in memcg mode, and set the shrinker to be memcg aware.
It adds the loop case for when pooled pages end up being reparented to a higher memcg group, that newer memcg can search for them there and take them back.
Signed-off-by Dave Airlie <[email protected]>---
v2: just use the proper stats.
v3: fix objcg check to return void--- drivers/gpu/drm/ttm/ttm_pool.c | 124 +++++++++++++++++++++++++++------ mm/list_lru.c | 1 + 2 files changed, 102 insertions(+), 23 deletions(-)drivers/gpu/drm/ttm/ttm_pool.c +101
-23@@ -144,7 +144,9 @@ static int ttm_pool_nid(struct ttm_pool *pool) }/* Allocate pages of size 1 << order with the given gfp_flags */ -static struct page *ttm_pool_alloc_page (struct ttm_pool *pool,gfp_t gfp_flags, +static struct page *ttm_pool_alloc_page (struct ttm_pool *pool, +struct obj_cgroup *objcg, +gfp_t gfp_flags,unsigned int order) {const unsigned int beneficial_order =ttm_pool_beneficial_order (pool);@@ -172,7 +174,10 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, p =alloc_pages_node (pool->nid, gfp_flags, order);if (p) { p->private = order; -mod_lruvec_page_state (p,NR_GPU_ACTIVE ,1 << order); +if (!mem_cgroup_charge_gpu_page (objcg, p, order, gfp_flags, false)) { +__free_pages (p, order); +return NULL ; + } }return p; }@@ -209,8 +214,7 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,static void __free_pages_gpu_account (struct page *p,unsigned int order,bool reclaim) { -mod_lruvec_page_state (p, reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , - -(1 << order)); +mem_cgroup_uncharge_gpu_page (p, order, reclaim);__free_pages (p, order); }@@ -317,12 +321,11 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,NULL ); +list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p));rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , -num_pages); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , num_pages); +mem_cgroup_move_gpu_page_reclaim (NULL , p, pt->order, true); } static enum lru_status take_one_from_lru(struct list_head *item,@@ -337,20 +340,56 @@ static enum lru_status take_one_from_lru(struct list_head *item,return LRU_REMOVED ; } -/* Take pages from a specific pool_type, return NULL when nothing available */ -static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid) +static int pool_lru_get_page (struct ttm_pool_type *pt,int nid, +struct page **page_out, +struct obj_cgroup *objcg, +struct mem_cgroup *memcg) {int ret;struct page *p =NULL ;unsigned long nr_to_walk =1 ; +unsigned int num_pages =1 << pt->order; - ret =list_lru_walk_node (&pt->pages, nid, take_one_from_lru, (void *)&p, &nr_to_walk); + ret =list_lru_walk_one (&pt->pages, nid, memcg, take_one_from_lru, (void *)&p, &nr_to_walk);if (ret ==1 && p) { -atomic_long_sub (1 << pt->order, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , (1 << pt->order)); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , -(1 << pt->order)); +atomic_long_sub (num_pages, &allocated_pages[nid]); + +if (!mem_cgroup_move_gpu_page_reclaim (objcg, p, pt->order, false)) { +__free_pages (p, pt->order); + p =NULL ; + } } -return p; + *page_out = p; +return ret; +} + +/* Take pages from a specific pool_type, return NULL when nothing available */ +static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid, +struct obj_cgroup *orig_objcg) +{ +struct page *page_out =NULL ; +int ret; +struct mem_cgroup *orig_memcg = orig_objcg ?get_mem_cgroup_from_objcg (orig_objcg) :NULL ; +struct mem_cgroup *memcg = orig_memcg; + +/* +* Attempt to get a page from the current memcg, but if it hasn't got any in it's level, +* go up to the parent and check there. This helps the scenario where multiple apps get +* started into their own cgroup from a common parent and want to reuse the pools. +*/ +while (!page_out) { + ret =pool_lru_get_page (pt, nid, &page_out, orig_objcg, memcg); +if (ret ==1 ) +break ; +if (!memcg) +break ; + memcg =parent_mem_cgroup (memcg); +if (!memcg) +break ; + } + +mem_cgroup_put (orig_memcg); +return page_out; }/* Initialize and add a pool type to the global shrinker list */ @@ -360,7 +399,7 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, pt->pool = pool; pt->caching = caching; pt->order = order; -list_lru_init (&pt->pages); +list_lru_init_memcg (&pt->pages, mm_shrinker);spin_lock (&shrinker_lock);list_add_tail (&pt->shrinker_list, &shrinker_list);@@ -403,6 +442,31 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt)ttm_pool_dispose_list (pt, &dispose); } +/* +* This function doesn't currently check dma32, because no driver using this +* support dma32. This should be added and debugged when that changes. +*/ +static void ttm_pool_check_objcg (struct obj_cgroup *objcg) +{ +#ifdef CONFIG_MEMCG +int r =0 ; +struct mem_cgroup *memcg; +if (!objcg) +return ; + + memcg =get_mem_cgroup_from_objcg (objcg); +for (unsigned i =0 ; i <NR_PAGE_ORDERS ; i++) { + r =memcg_list_lru_alloc (memcg, &global_write_combined[i].pages,GFP_KERNEL ); +if (r) +break ; + r =memcg_list_lru_alloc (memcg, &global_uncached[i].pages,GFP_KERNEL ); +if (r) +break ; + } +mem_cgroup_put (memcg); +#endif +} +/* Return the pool_type to use for the given caching and order */ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, enum ttm_caching caching,@@ -432,7 +496,9 @@ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, }/* Free pages using the per-node shrinker list */ -static unsigned int ttm_pool_shrink (int nid,unsigned long num_to_free) +static unsigned int ttm_pool_shrink (int nid, +struct mem_cgroup *memcg, +unsigned long num_to_free) {LIST_HEAD (dispose);struct ttm_pool_type *pt;@@ -448,7 +514,11 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)if (!pt)return 0 ; - num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); +if (!memcg) { + num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); + }else { + num_pages =list_lru_walk_one (&pt->pages, nid, memcg, pool_move_to_dispose_list, &dispose, &num_to_free); + } num_pages *=1 << pt->order;ttm_pool_dispose_list (pt, &dispose);@@ -777,6 +847,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; +struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);@@ -794,6 +865,9 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, page_caching = tt->caching; allow_pools = true; + +ttm_pool_check_objcg (objcg); +for (order =ttm_pool_alloc_find_order (MAX_PAGE_ORDER , alloc); alloc->remaining_pages; order =ttm_pool_alloc_find_order (order, alloc)) {@@ -803,7 +877,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, p =NULL ; pt =ttm_pool_select_type (pool, page_caching, order);if (pt && allow_pools) - p =ttm_pool_type_take (pt,ttm_pool_nid (pool)); + p =ttm_pool_type_take (pt,ttm_pool_nid (pool), objcg);/* * If that fails or previously failed, allocate from system. @@ -814,7 +888,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,if (!p) { page_caching = ttm_cached; allow_pools = false; - p =ttm_pool_alloc_page (pool, gfp_flags, order); + p =ttm_pool_alloc_page (pool, objcg, gfp_flags, order); }/* If that fails, lower the order if possible and retry. */ if (!p) {@@ -960,7 +1034,7 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)while (atomic_long_read (&allocated_pages[nid]) > pool_node_limit[nid]) {unsigned long diff =atomic_long_read (&allocated_pages[nid]) - pool_node_limit[nid]; -ttm_pool_shrink (nid, diff); +ttm_pool_shrink (nid,NULL , diff); } }EXPORT_SYMBOL (ttm_pool_free);@@ -1214,10 +1288,14 @@ static unsigned long ttm_pool_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) {unsigned long num_freed =0 ; +int num_pools; +spin_lock (&shrinker_lock); + num_pools =list_count_nodes (&shrinker_list); +spin_unlock (&shrinker_lock);do - num_freed +=ttm_pool_shrink (sc->nid, sc->nr_to_scan); -while (num_freed < sc->nr_to_scan && + num_freed +=ttm_pool_shrink (sc->nid, sc->memcg, sc->nr_to_scan); +while (num_pools-- >=0 && num_freed < sc->nr_to_scan &&atomic_long_read (&allocated_pages[sc->nid])); sc->nr_scanned = num_freed;@@ -1406,7 +1484,7 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); + mm_shrinker =shrinker_alloc (SHRINKER_MEMCG_AWARE |SHRINKER_NUMA_AWARE ,"drm-ttm_pool" );if (!mm_shrinker)return -ENOMEM ;mm/list_lru.c +0
-0
mm/list_lru.c int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, · line 633
Quoted 7 lines
mm/list_lru.c +1
-0@@ -627,6 +627,7 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,return 0 ;return __memcg_list_lru_alloc (memcg, lru, gfp); } +EXPORT_SYMBOL_GPL (memcg_list_lru_alloc);int folio_memcg_list_lru_alloc (struct folio *folio,struct list_lru *lru,gfp_t gfp)
Hi Dave, all,
Thanks for the series. The user facing memcg counters, the objcg plumbing and the memcg aware charging at BO create and destroy look correct to me. While going through the series, I got curious about one specific choice made for keeping pool pages charged to their giving cgroup and partitioning the pool's list_lru per nid and memcg as a consequence.
The layering concern:
The kernel's memory hierarchy has a consistent convention: the lower layers of the allocator stack are memcg agnostic shared infrastructure, and memcg accounting sits above them tracking who currently owns live memory. For example:
- The buddy allocator has no per cgroup partitioning of freelists. Cgroups don't have different views of buddy serveing everyone identically.
- The slab layer's freelists are memcg agnostic. The freed object sits uncharged in the slab cache and the next kmalloc + __GFP_ACCOUNT from any cgroup gets charges at that boundary.
The TTM pool is logically another layer of the same kind of shared reuse infrastructure. It exists specifically to amortize the cost of memory allocations, the same way slab exists to amortize object shape init across kmalloc and kfree. kmem_cache is analogous to ttm_pool_type, each having its own freelist. By the layering convention, the pool should be memcg agnostic in storage where any cgroup can put a page in, any cgroup can take a page out, and charging happens at the boundary where the page becomes a live BO owned by a cgroup.
Patch 05 diverges from this: mem_cgroup_move_gpu_page_reclaim(NULL, p, pt->order, true) in ttm_pool_type_give keeps pool pages charged as NR_GPU_RECLAIM to the giving cgroup, and list_lru_init_memcg partitions the list_lru per nid and memcg. That's the same shape as partitioning buddy's freelists or slab's per CPU caches per cgroup, which we wouldn't do.
Consequences of the divergence:
The consequences that follow from this specific choice and that wouldn't apply if the pool followed the buddy/slab convention:
1. Pool storage is now partitioned M ways for M memcgs.
2. ttm_pool_type_take needs a parent walk (while loop climbing parent_mem_cgroup) to enable any cross cgroup reuse instead of serving from common pool which opposite to the intention of having global resource.
3. Long lived sibling cgroups can't share cached pool pages because their common ancestor holds no pages of its own, so the parent walk terminates empty. For example, A memcg can hold 500 MB of cached pool pages for a week while idle. B ramps up beside it, walks up looking for cached pages, finds every ancestor's sublist empty, and ends up paying cost for every page it needs, even though physically identical pages are available in common pool.
4. MM driven reclaim invokes the shrinker M memcg x N numa nodes times via shrink_slab_memcg's per memcg bitmap walk.
5. ttm_pool_shrink has two branches: memcg=NULL uses list_lru_walk_node (one call, all sublists). A specific memcg uses list_lru_walk_one (one sublist per call). TTM's own
pool cap enforcement in ttm_pool_free takes the flat branch whereas ttm_pool_shrinker_scan always passes sc->memcg and takes the per-cgroup branch (M x N calls per reclaim cycle). If pool pages followed the buddyslab convention, the shrinker wouldn't need to be memcg aware and both callers would use the flat walk.
Questions:
1. What's the motivation for keeping pool pages charged rather than following the buddy and slab convention? If it's guarding against a cgroup launders BOs through the pool to stay under memory.max pattern, note that try_charge_memcg fires on every take ( from pool or buddy ) under either accounting model, so a cgroup cannot exceed memory.max by cycling through the pool in the first place. The pool_node_limit merely caps the total system wide pool memory to prevent absolute pool growth. If it's memory.current stability, that's not preserved for slab either. kfree drops the originating memcg's memory.current immediately.
2. If keeping the pool charged is the right choice, is the plan to accept the layering deviation and document it, or is there a longer term direction where the pool moves back to memcg agnostic storage?
I do not intend to block it. The patch does what it says and I'm keen to see GPU memcg accounting land. Just want the rationale for architectural choice in case I have missed something or if design needs reconsideration.
Thanks,
Pranjal Arya
Qualcomm