drm/ttm/memcg/lru: enable memcg tracking for ttm, xe and amdgpu driver (part 2).
Subsystems (14)
Dave Airlie
maintainerI committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting this landed for xe, we can drop the amdgpu changes for now if they can't get across the line.
I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align with the two global vmstats. It adds an accounting flag to TTM alloc/populate, and enables memcg tracking and shrinker support in TTM.
Then it adds amdgpu and xe support.
I think for this to land, Christian holds the main objection which I still fail to fully understand beyond it doesn't solve all the problems we ever have had with cgroups and drm, so we shouldn't even bother, and maybe we could do it at the object level, and integrated with dmem, and android cross process accounting, but I still feel this is a good baseline.
I think this is the right layer to hook this into TTM, where we allocate memory and I think accounting for this memory in a proper way should be done.
Intel folks (Thomas/Maarten) please review and express concerns as well.
Regards,
Dave.
Dave Airlie
maintainerPATCH 1/10 memcg: add support for GPU page counters. (v4)
From: Dave Airlie <[email protected]>
This introduces 2 new statistics and 3 new memcontrol APIs for dealing with GPU system memory allocations.
The stats corresponds to the same stats in the global vmstat, for number of active GPU pages, and number of pages in pools that can be reclaimed.
The first API charges a order of pages to a objcg, and sets the objcg on the pages like kmem does, and updates the active/reclaim statistic.
The second API uncharges a page from the obj cgroup it is currently charged to.
The third API allows moving a page to/from reclaim and between obj cgroups. When pages are added to the pool lru, this just updates accounting. When pages are being removed from a pool lru, they can be taken from the parent objcg so this allows them to be uncharged from there and transferred to a new child objcg.
---
v2: use memcg_node_stat_items
v3: fix null ptr dereference in uncharge
v4: AI review: fix parameter names, fix problem with reclaim moving doing wrong thing
--- Documentation/admin-guide/cgroup-v2.rst | 6 ++ include/linux/memcontrol.h | 11 +++ mm/memcontrol.c | 104 ++++++++++++++++++++++++ 3 files changed, 121 insertions(+)
Documentation/admin-guide/cgroup-v2.rst +6 -0
@@ -1573,6 +1573,12 @@ The following nested keys are defined. vmalloc (npn) Amount of memory used for vmap backed memory. + gpu_active (npn) + Amount of system memory used for GPU devices. + + gpu_reclaim (npn) + Amount of system memory cached for GPU devices. + shmem Amount of cached filesystem data that is swap-backed, such as tmpfs, shm segments, shared anonymous mmap()s
include/linux/memcontrol.h +11 -0
@@ -1583,6 +1583,17 @@ static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb)#endif /* CONFIG_CGROUP_WRITEBACK */ struct sock ; +bool mem_cgroup_charge_gpu_page (struct obj_cgroup *objcg,struct page *page, +unsigned int order, +gfp_t gfp_mask,bool reclaim); +void mem_cgroup_uncharge_gpu_page (struct page *page, +unsigned int order, +bool reclaim); +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *objcg, +struct page *page, +unsigned int order, +bool to_reclaim); +#ifdef CONFIG_MEMCG extern struct static_key_false memcg_sockets_enabled_key;#define mem_cgroup_sockets_enabled static_branch_unlikely(&memcg_sockets_enabled_key)
mm/memcontrol.c +104 -0
122 lines not on this page · raw message
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 2/10 ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This flag does nothing yet, but this just changes the APIs to accept it in the future across all users.
This flag will eventually be filled out with when to account a tt populate to a memcg.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 ++- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 5 +++-- drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c | 4 ++-- drivers/gpu/drm/loongson/lsdc_ttm.c | 3 ++- drivers/gpu/drm/nouveau/nouveau_bo.c | 6 ++++-- drivers/gpu/drm/radeon/radeon_ttm.c | 3 ++- drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c | 2 +- drivers/gpu/drm/ttm/tests/ttm_pool_test.c | 16 ++++++++-------- drivers/gpu/drm/ttm/tests/ttm_tt_test.c | 12 ++++++------ drivers/gpu/drm/ttm/ttm_bo.c | 7 ++++--- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +++- drivers/gpu/drm/ttm/ttm_pool.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_tt.c | 8 +++++--- drivers/gpu/drm/vmwgfx/vmwgfx_blit.c | 4 ++-- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c | 7 ++++--- drivers/gpu/drm/xe/xe_bo.c | 5 +++-- include/drm/ttm/ttm_bo.h | 1 + include/drm/ttm/ttm_device.h | 1 + include/drm/ttm/ttm_pool.h | 1 + include/drm/ttm/ttm_tt.h | 1 + 22 files changed, 63 insertions(+), 44 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -1
@@ -1220,6 +1220,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,*/ static int amdgpu_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct amdgpu_device *adev =amdgpu_ttm_adev (bdev);
@@ -1243,7 +1244,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev, pool = &adev->mman.ttm_pools[gtt->pool_id]; else pool = &adev->mman.bdev.pool; - ret =ttm_pool_alloc (pool, ttm, ctx); + ret =ttm_pool_alloc (pool, ttm, memcg_account, ctx);if (ret)return ret;
drivers/gpu/drm/i915/gem/i915_gem_ttm.c +3 -2
@@ -317,6 +317,7 @@ static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,static int i915_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct i915_ttm_tt *i915_tt =container_of (ttm,typeof (*i915_tt), ttm);
@@ -324,7 +325,7 @@ static int i915_ttm_tt_populate(struct ttm_device *bdev,if (i915_tt->is_shmem)return i915_ttm_tt_shmem_populate (bdev, ttm, ctx); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); }static void i915_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)
@@ -815,7 +816,7 @@ static int __i915_ttm_get_pages(struct drm_i915_gem_object *obj, }if (bo->ttm && !ttm_tt_is_populated (bo->ttm)) { - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +1 -1
@@ -624,7 +624,7 @@ int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,/* Populate ttm with pages if needed. Typically system memory. */ if (ttm && (dst_man->use_tt || (ttm->page_flags &TTM_TT_FLAG_SWAPPED ))) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }
drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.c +2 -2
@@ -91,7 +91,7 @@ static int i915_ttm_backup(struct i915_gem_apply_to_region *apply, goto out_no_lock; backup_bo =i915_gem_to_ttm (backup); - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (err) goto out_no_populate;
@@ -190,7 +190,7 @@ static int i915_ttm_restore(struct i915_gem_apply_to_region *apply,if (!backup_bo->resource) err =ttm_bo_validate (backup_bo,i915_ttm_sys_placement (), &ctx);if (!err) - err =ttm_bo_populate (backup_bo, &ctx); + err =ttm_bo_populate (backup_bo, false, &ctx);if (!err) { err =i915_gem_obj_copy_ttm (obj, backup, pm_apply->allow_gpu, false);
drivers/gpu/drm/loongson/lsdc_ttm.c +2 -1
@@ -111,6 +111,7 @@ lsdc_ttm_tt_create(struct ttm_buffer_object *tbo, uint32_t page_flags)static int lsdc_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {bool slave = !!(ttm->page_flags &TTM_TT_FLAG_EXTERNAL );
@@ -123,7 +124,7 @@ static int lsdc_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void lsdc_ttm_tt_unpopulate(struct ttm_device *bdev,
drivers/gpu/drm/nouveau/nouveau_bo.c +4 -2
@@ -1417,7 +1417,9 @@ vm_fault_t nouveau_ttm_fault_reserve_notify(struct ttm_buffer_object *bo)static int nouveau_ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {struct ttm_tt *ttm_dma = (void *)ttm;struct nouveau_drm *drm;
@@ -1434,7 +1436,7 @@ nouveau_ttm_tt_populate(struct ttm_device *bdev, drm =nouveau_bdev (bdev); -return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&drm->ttm.bdev.pool, ttm, memcg_account, ctx); } static void
drivers/gpu/drm/radeon/radeon_ttm.c +2 -1
@@ -526,6 +526,7 @@ static struct radeon_ttm_tt *radeon_ttm_tt_to_gtt(struct radeon_device *rdev,static int radeon_ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct radeon_device *rdev =radeon_get_rdev (bdev);
@@ -547,7 +548,7 @@ static int radeon_ttm_tt_populate(struct ttm_device *bdev,return 0 ; } -return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, ctx); +return ttm_pool_alloc (&rdev->mman.bdev.pool, ttm, memcg_account, ctx); }static void radeon_ttm_tt_unpopulate (struct ttm_device *bdev,struct ttm_tt *ttm)
drivers/gpu/drm/ttm/tests/ttm_bo_validate_test.c +1 -1
@@ -538,7 +538,7 @@ static void ttm_bo_validate_no_placement_signaled(struct kunit *test)if (params->with_ttm) { old_tt = priv->ttm_dev->funcs->ttm_tt_create (bo,0 ); -ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, &ctx); +ttm_pool_alloc (&priv->ttm_dev->pool, old_tt, false, &ctx); bo->ttm = old_tt; }
drivers/gpu/drm/ttm/tests/ttm_pool_test.c +8 -8
@@ -89,7 +89,7 @@ static struct ttm_pool *ttm_pool_pre_populated(struct kunit *test,ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -157,7 +157,7 @@ static void ttm_pool_alloc_basic(struct kunit *test)KUNIT_ASSERT_EQ (test, pool->nid,NUMA_NO_NODE );KUNIT_ASSERT_EQ (test, pool->alloc_flags, params->alloc_flags); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);
@@ -220,7 +220,7 @@ static void ttm_pool_alloc_basic_dma_addr(struct kunit *test)ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_EQ (test, tt->num_pages, expected_num_pages);
@@ -253,7 +253,7 @@ static void ttm_pool_alloc_order_caching_match(struct kunit *test) tt =ttm_tt_kunit_init (test,0 , caching, size);KUNIT_ASSERT_NOT_NULL (test, tt); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));
@@ -285,7 +285,7 @@ static void ttm_pool_alloc_caching_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -319,7 +319,7 @@ static void ttm_pool_alloc_order_mismatch(struct kunit *test)KUNIT_ASSERT_FALSE (test, !list_lru_count (&pt_pool->pages));KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt_tt->pages)); - err =ttm_pool_alloc (pool, tt, &simple_ctx); + err =ttm_pool_alloc (pool, tt, false, &simple_ctx);KUNIT_ASSERT_EQ (test, err,0 );ttm_pool_free (pool, tt);
@@ -349,7 +349,7 @@ static void ttm_pool_free_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,TTM_ALLOCATION_POOL_USE_DMA_ALLOC ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx); pt = &pool->caching[caching].orders[order];KUNIT_ASSERT_TRUE (test, !list_lru_count (&pt->pages));
@@ -379,7 +379,7 @@ static void ttm_pool_free_no_dma_alloc(struct kunit *test)KUNIT_ASSERT_NOT_NULL (test, pool);ttm_pool_init (pool, devs->dev,NUMA_NO_NODE ,0 ); -ttm_pool_alloc (pool, tt, &simple_ctx); +ttm_pool_alloc (pool, tt, false, &simple_ctx);ttm_pool_free (pool, tt);ttm_tt_fini (tt);
drivers/gpu/drm/ttm/tests/ttm_tt_test.c +6 -6
@@ -262,7 +262,7 @@ static void ttm_tt_populate_null_ttm(struct kunit *test)struct ttm_operation_ctx ctx = { };int err; - err =ttm_tt_populate (devs->ttm_dev,NULL , &ctx); + err =ttm_tt_populate (devs->ttm_dev,NULL , false, &ctx);KUNIT_ASSERT_EQ (test, err, -EINVAL ); }
@@ -283,11 +283,11 @@ static void ttm_tt_populate_populated_ttm(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); populated_page = *tt->pages; - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_PTR_EQ (test, populated_page, *tt->pages); }
@@ -307,7 +307,7 @@ static void ttm_tt_unpopulate_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));
@@ -351,7 +351,7 @@ static void ttm_tt_swapin_basic(struct kunit *test) err =ttm_tt_init (tt, bo,0 , ttm_cached,0 );KUNIT_ASSERT_EQ (test, err,0 ); - err =ttm_tt_populate (devs->ttm_dev, tt, &ctx); + err =ttm_tt_populate (devs->ttm_dev, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 );KUNIT_ASSERT_TRUE (test,ttm_tt_is_populated (tt));
@@ -361,7 +361,7 @@ static void ttm_tt_swapin_basic(struct kunit *test)KUNIT_ASSERT_TRUE (test, tt->page_flags &TTM_TT_FLAG_SWAPPED );/* Swapout depopulates TT, allocate pages and then swap them in */ - err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, &ctx); + err =ttm_pool_alloc (&devs->ttm_dev->pool, tt, false, &ctx);KUNIT_ASSERT_EQ (test, err,0 ); err =ttm_tt_swapin (tt);
drivers/gpu/drm/ttm/ttm_bo.c +4 -3
@@ -145,7 +145,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, goto out_err;if (mem->mem_type !=TTM_PL_SYSTEM ) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret) goto out_err; }
@@ -1255,6 +1255,7 @@ void ttm_bo_tt_destroy(struct ttm_buffer_object *bo)* is set to true. */ int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_device *bdev = bo->bdev;
@@ -1268,7 +1269,7 @@ int ttm_bo_populate(struct ttm_buffer_object *bo,return 0 ; swapped =ttm_tt_is_swapped (tt); - ret =ttm_tt_populate (bdev, tt, ctx); + ret =ttm_tt_populate (bdev, tt, memcg_account, ctx);if (ret)return ret;
@@ -1293,7 +1294,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);ttm_bo_unreserve (bo);return ret; }
drivers/gpu/drm/ttm/ttm_bo_util.c +3 -3
@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man =ttm_manager_type (bdev, src_mem->mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { - ret =ttm_bo_populate (bo, ctx); + ret =ttm_bo_populate (bo, false, ctx);if (ret)return ret; }
@@ -352,7 +352,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,BUG_ON (!ttm); - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
@@ -533,7 +533,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map)pgprot_t prot;void *vaddr; - ret =ttm_bo_populate (bo, &ctx); + ret =ttm_bo_populate (bo, false, &ctx);if (ret)return ret;
drivers/gpu/drm/ttm/ttm_bo_vm.c +3 -1
@@ -224,7 +224,9 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, }; ttm = bo->ttm; - err =ttm_bo_populate (bo, &ctx); + err =ttm_bo_populate (bo, + false, + &ctx);if (err) {if (err == -EINTR || err == -ERESTARTSYS || err == -EAGAIN )
drivers/gpu/drm/ttm/ttm_pool.c +4 -2
@@ -761,6 +761,7 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest, }static int __ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,const struct ttm_operation_ctx *ctx,struct ttm_pool_alloc_state *alloc,struct ttm_pool_tt_restore *restore)
@@ -871,6 +872,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,* Returns: 0 on successe, negative error code otherwise. */ int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) {struct ttm_pool_alloc_state alloc;
@@ -880,7 +882,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,ttm_pool_alloc_state_init (tt, &alloc); -return __ttm_pool_alloc (pool, tt, ctx, &alloc,NULL ); +return __ttm_pool_alloc (pool, tt, memcg_account, ctx, &alloc,NULL ); }EXPORT_SYMBOL (ttm_pool_alloc);
@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,return 0 ; } -return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); +return __ttm_pool_alloc (pool, tt, false, ctx, &alloc, restore); }/**
drivers/gpu/drm/ttm/ttm_tt.c +5 -3
@@ -368,7 +368,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,EXPORT_SYMBOL_FOR_TESTS_ONLY (ttm_tt_swapout);int ttm_tt_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm, +bool memcg_account, +struct ttm_operation_ctx *ctx) {int ret;
@@ -397,9 +399,9 @@ int ttm_tt_populate(struct ttm_device *bdev, }if (bdev->funcs->ttm_tt_populate) - ret = bdev->funcs->ttm_tt_populate (bdev, ttm, ctx); + ret = bdev->funcs->ttm_tt_populate (bdev, ttm, memcg_account, ctx);else - ret =ttm_pool_alloc (&bdev->pool, ttm, ctx); + ret =ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx);if (ret) goto error;
drivers/gpu/drm/vmwgfx/vmwgfx_blit.c +2 -2
@@ -569,13 +569,13 @@ int vmw_bo_cpu_blit(struct vmw_bo *vmw_dst,dma_resv_assert_held (src->base.resv);if (!ttm_tt_is_populated (dst->ttm)) { - ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, &ctx); + ret = dst->bdev->funcs->ttm_tt_populate (dst->bdev, dst->ttm, false, &ctx);if (ret)return ret; }if (!ttm_tt_is_populated (src->ttm)) { - ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, &ctx); + ret = src->bdev->funcs->ttm_tt_populate (src->bdev, src->ttm, false, &ctx);if (ret)return ret; }
drivers/gpu/drm/vmwgfx/vmwgfx_ttm_buffer.c +4 -3
@@ -360,7 +360,8 @@ static void vmw_ttm_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)static int vmw_ttm_populate (struct ttm_device *bdev, -struct ttm_tt *ttm,struct ttm_operation_ctx *ctx) +struct ttm_tt *ttm,bool memcg_account, +struct ttm_operation_ctx *ctx) {bool external = (ttm->page_flags &TTM_TT_FLAG_EXTERNAL ) !=0 ;
@@ -372,7 +373,7 @@ static int vmw_ttm_populate(struct ttm_device *bdev, ttm->dma_address, ttm->num_pages); -return ttm_pool_alloc (&bdev->pool, ttm, ctx); +return ttm_pool_alloc (&bdev->pool, ttm, memcg_account, ctx); } static void vmw_ttm_unpopulate(struct ttm_device *bdev,
@@ -580,7 +581,7 @@ int vmw_bo_create_and_populate(struct vmw_private *dev_priv,if (unlikely (ret !=0 ))return ret; - ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, &ctx); + ret =vmw_ttm_populate (vbo->tbo.bdev, vbo->tbo.ttm, false, &ctx);if (likely (ret ==0 )) {struct vmw_ttm_tt *vmw_tt = container_of(vbo->tbo.ttm, struct vmw_ttm_tt, dma_ttm);
drivers/gpu/drm/xe/xe_bo.c +3 -2
@@ -536,6 +536,7 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, }static int xe_ttm_tt_populate (struct ttm_device *ttm_dev,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) { struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err =ttm_tt_restore (ttm_dev, tt, ctx); }else {ttm_tt_clear_backed_up (tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)return err;
@@ -1926,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (ttm_manager_type (tbo->bdev, tbo->resource->mem_type)->use_tt) { err =xe_bo_wait_usage_kernel (bo, ctx);if (!err) - err =ttm_bo_populate (&bo->ttm, ctx); + err =ttm_bo_populate (&bo->ttm, false, ctx); }else if (should_migrate_to_smem (bo)) {xe_assert (xe_bo_device (bo), bo->flags &XE_BO_FLAG_SYSTEM ); err =xe_bo_migrate (bo,XE_PL_TT , ctx, exec);
include/drm/ttm/ttm_bo.h +1 -0
@@ -475,6 +475,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res, pgprot_t tmp);void ttm_bo_tt_destroy (struct ttm_buffer_object *bo);int ttm_bo_populate (struct ttm_buffer_object *bo, +bool memcg_account,struct ttm_operation_ctx *ctx);int ttm_bo_setup_export (struct ttm_buffer_object *bo,struct ttm_operation_ctx *ctx);
include/drm/ttm/ttm_device.h +1 -0
@@ -85,6 +85,7 @@ struct ttm_device_funcs {*/ int (*ttm_tt_populate)(struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/**
include/drm/ttm/ttm_pool.h +1 -0
@@ -78,6 +78,7 @@ struct ttm_pool { };int ttm_pool_alloc (struct ttm_pool *pool,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx);void ttm_pool_free (struct ttm_pool *pool,struct ttm_tt *tt);
include/drm/ttm/ttm_tt.h +1 -0
@@ -250,6 +250,7 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,* Calls the driver method to allocate pages for a ttm */ int ttm_tt_populate (struct ttm_device *bdev,struct ttm_tt *ttm, +bool memcg_account,struct ttm_operation_ctx *ctx);/**
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 3/10 ttm/pool: initialise the shrinker earlier
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
Later memcg enablement needs the shrinker initialised before the list lru, Just move it for now.
--- drivers/gpu/drm/ttm/ttm_pool.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +11 -11
@@ -1402,6 +1402,17 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); + mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); +if (!mm_shrinker) +return -ENOMEM ; + + mm_shrinker->count_objects = ttm_pool_shrinker_count; + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; + mm_shrinker->batch =TTM_SHRINKER_BATCH ; + mm_shrinker->seeks =1 ; + +shrinker_register (mm_shrinker); +for (i =0 ; i <NR_PAGE_ORDERS ; ++i) {ttm_pool_type_init (&global_write_combined[i],NULL , ttm_write_combined, i);
@@ -1424,17 +1435,6 @@ int ttm_pool_mgr_init(unsigned long num_pages)#endif #endif - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); -if (!mm_shrinker) -return -ENOMEM ; - - mm_shrinker->count_objects = ttm_pool_shrinker_count; - mm_shrinker->scan_objects = ttm_pool_shrinker_scan; - mm_shrinker->batch =TTM_SHRINKER_BATCH ; - mm_shrinker->seeks =1 ; - -shrinker_register (mm_shrinker); -return 0 ; }
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 4/10 ttm: add objcg pointer to bo and tt (v2)
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This just adds the obj cgroup pointer to the bo and tt structs, and sets it between them.
---
v2: add the put and a setter helper
--- drivers/gpu/drm/ttm/ttm_bo.c | 2 ++ drivers/gpu/drm/ttm/ttm_tt.c | 1 + include/drm/ttm/ttm_bo.h | 20 ++++++++++++++++++++ include/drm/ttm/ttm_tt.h | 2 ++ 4 files changed, 25 insertions(+)
drivers/gpu/drm/ttm/ttm_bo.c +2 -0
@@ -48,6 +48,7 @@#include <linux/atomic.h> #include <linux/cgroup_dmem.h> #include <linux/dma-resv.h> +#include <linux/memcontrol.h> #include "ttm_module.h" #include "ttm_bo_internal.h"
@@ -315,6 +316,7 @@ static void ttm_bo_release(struct kref *kref)dma_resv_unlock (bo->base.resv); } +obj_cgroup_put (bo->objcg);atomic_dec (&ttm_glob.bo_count); bo->destroy (bo); }
drivers/gpu/drm/ttm/ttm_tt.c +1 -0
@@ -164,6 +164,7 @@ static void ttm_tt_init_fields(struct ttm_tt *ttm, ttm->caching = caching; ttm->restore =NULL ; ttm->backup =NULL ; + ttm->objcg = bo->objcg; } int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo,
include/drm/ttm/ttm_bo.h +20 -0
@@ -135,6 +135,12 @@ struct ttm_buffer_object {* reservation lock. */ struct sg_table *sg; + +/** +* @objcg: object cgroup to charge this to if it ends up using system memory. +* NULL means don't charge. +*/ +struct obj_cgroup *objcg; };#define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -344,6 +350,20 @@ ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo)spin_unlock (&bo->bdev->lru_lock); } +/** +* ttm_bo_set_cgroup - assign a cgroup to a buffer object. +* @bo: The bo to set the cgroup for +* @objcg: the cgroup to set. +* +* This transfers the cgroup reference to the bo. From this +* point on the cgroup reference is owned by the ttm bo. +*/ +static inline void ttm_bo_set_cgroup (struct ttm_buffer_object *bo, +struct obj_cgroup *objcg) +{ + bo->objcg = objcg; +} +static inline void ttm_bo_assign_mem (struct ttm_buffer_object *bo,struct ttm_resource *new_mem) {
include/drm/ttm/ttm_tt.h +2 -0
@@ -126,6 +126,8 @@ struct ttm_tt {enum ttm_caching caching;/** @restore: Partial restoration from backup state. TTM private */ struct ttm_pool_tt_restore *restore; +/** @objcg: Object cgroup for this TT allocation */ +struct obj_cgroup *objcg; };/**
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 5/10 ttm/pool: enable memcg tracking and shrinker. (v3)
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This enables all the backend code to use the list lru in memcg mode, and set the shrinker to be memcg aware.
It adds the loop case for when pooled pages end up being reparented to a higher memcg group, that newer memcg can search for them there and take them back.
---
v2: just use the proper stats.
v3: fix objcg check to return void
--- drivers/gpu/drm/ttm/ttm_pool.c | 126 +++++++++++++++++++++++++++------ mm/list_lru.c | 1 + 2 files changed, 104 insertions(+), 23 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +103 -23
@@ -144,7 +144,9 @@ static int ttm_pool_nid(struct ttm_pool *pool) }/* Allocate pages of size 1 << order with the given gfp_flags */ -static struct page *ttm_pool_alloc_page (struct ttm_pool *pool,gfp_t gfp_flags, +static struct page *ttm_pool_alloc_page (struct ttm_pool *pool, +struct obj_cgroup *objcg, +gfp_t gfp_flags,unsigned int order) {const unsigned int beneficial_order =ttm_pool_beneficial_order (pool);
@@ -172,7 +174,10 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, p =alloc_pages_node (pool->nid, gfp_flags, order);if (p) { p->private = order; -mod_lruvec_page_state (p,NR_GPU_ACTIVE ,1 << order); +if (!mem_cgroup_charge_gpu_page (objcg, p, order, gfp_flags, false)) { +__free_pages (p, order); +return NULL ; + } }return p; }
@@ -209,8 +214,7 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,static void __free_pages_gpu_account (struct page *p,unsigned int order,bool reclaim) { -mod_lruvec_page_state (p, reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , - -(1 << order)); +mem_cgroup_uncharge_gpu_page (p, order, reclaim);__free_pages (p, order); }
@@ -317,12 +321,11 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,NULL ); +list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p));rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , -num_pages); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , num_pages); +mem_cgroup_move_gpu_page_reclaim (NULL , p, pt->order, true); } static enum lru_status take_one_from_lru(struct list_head *item,
@@ -337,20 +340,56 @@ static enum lru_status take_one_from_lru(struct list_head *item,return LRU_REMOVED ; } -/* Take pages from a specific pool_type, return NULL when nothing available */ -static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid) +static int pool_lru_get_page (struct ttm_pool_type *pt,int nid, +struct page **page_out, +struct obj_cgroup *objcg, +struct mem_cgroup *memcg) {int ret;struct page *p =NULL ;unsigned long nr_to_walk =1 ; +unsigned int num_pages =1 << pt->order; - ret =list_lru_walk_node (&pt->pages, nid, take_one_from_lru, (void *)&p, &nr_to_walk); + ret =list_lru_walk_one (&pt->pages, nid, memcg, take_one_from_lru, (void *)&p, &nr_to_walk);if (ret ==1 && p) { -atomic_long_sub (1 << pt->order, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , (1 << pt->order)); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , -(1 << pt->order)); +atomic_long_sub (num_pages, &allocated_pages[nid]); + +if (!mem_cgroup_move_gpu_page_reclaim (objcg, p, pt->order, false)) { +__free_pages (p, pt->order); + p =NULL ; + } } -return p; + *page_out = p; +return ret; +} + +/* Take pages from a specific pool_type, return NULL when nothing available */ +static struct page *ttm_pool_type_take (struct ttm_pool_type *pt,int nid, +struct obj_cgroup *orig_objcg) +{ +struct page *page_out =NULL ; +int ret; +struct mem_cgroup *orig_memcg = orig_objcg ?get_mem_cgroup_from_objcg (orig_objcg) :NULL ; +struct mem_cgroup *memcg = orig_memcg; + +/* +* Attempt to get a page from the current memcg, but if it hasn't got any in it's level, +* go up to the parent and check there. This helps the scenario where multiple apps get +* started into their own cgroup from a common parent and want to reuse the pools. +*/ +while (!page_out) { + ret =pool_lru_get_page (pt, nid, &page_out, orig_objcg, memcg); +if (ret ==1 ) +break ; +if (!memcg) +break ; + memcg =parent_mem_cgroup (memcg); +if (!memcg) +break ; + } + +mem_cgroup_put (orig_memcg); +return page_out; }/* Initialize and add a pool type to the global shrinker list */
@@ -360,7 +399,7 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, pt->pool = pool; pt->caching = caching; pt->order = order; -list_lru_init (&pt->pages); +list_lru_init_memcg (&pt->pages, mm_shrinker);spin_lock (&shrinker_lock);list_add_tail (&pt->shrinker_list, &shrinker_list);
@@ -403,6 +442,31 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt)ttm_pool_dispose_list (pt, &dispose); } +/* +* This function doesn't currently check dma32, because no driver using this +* support dma32. This should be added and debugged when that changes. +*/ +static void ttm_pool_check_objcg (struct obj_cgroup *objcg) +{ +#ifdef CONFIG_MEMCG +int r =0 ; +struct mem_cgroup *memcg; +if (!objcg) +return ; + + memcg =get_mem_cgroup_from_objcg (objcg); +for (unsigned i =0 ; i <NR_PAGE_ORDERS ; i++) { + r =memcg_list_lru_alloc (memcg, &global_write_combined[i].pages,GFP_KERNEL ); +if (r) +break ; + r =memcg_list_lru_alloc (memcg, &global_uncached[i].pages,GFP_KERNEL ); +if (r) +break ; + } +mem_cgroup_put (memcg); +#endif +} +/* Return the pool_type to use for the given caching and order */ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, enum ttm_caching caching,
@@ -432,7 +496,9 @@ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, }/* Free pages using the per-node shrinker list */ -static unsigned int ttm_pool_shrink (int nid,unsigned long num_to_free) +static unsigned int ttm_pool_shrink (int nid, +struct mem_cgroup *memcg, +unsigned long num_to_free) {LIST_HEAD (dispose);struct ttm_pool_type *pt;
@@ -444,7 +510,11 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)list_move_tail (&pt->shrinker_list, &shrinker_list);spin_unlock (&shrinker_lock); - num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); +if (!memcg) { + num_pages =list_lru_walk_node (&pt->pages, nid, pool_move_to_dispose_list, &dispose, &num_to_free); + }else { + num_pages =list_lru_walk_one (&pt->pages, nid, memcg, pool_move_to_dispose_list, &dispose, &num_to_free); + } num_pages *=1 << pt->order;ttm_pool_dispose_list (pt, &dispose);
@@ -612,6 +682,7 @@ static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,*/ ttm_pool_split_for_swap (restore->pool, p);copy_highpage (restore->alloced_page + i, p); + p->memcg_data =0 ;__free_pages_gpu_account (p,0 , false); }
@@ -773,6 +844,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; +struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);
@@ -790,6 +862,9 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, page_caching = tt->caching; allow_pools = true; + +ttm_pool_check_objcg (objcg); +for (order =ttm_pool_alloc_find_order (MAX_PAGE_ORDER , alloc); alloc->remaining_pages; order =ttm_pool_alloc_find_order (order, alloc)) {
@@ -799,7 +874,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, p =NULL ; pt =ttm_pool_select_type (pool, page_caching, order);if (pt && allow_pools) - p =ttm_pool_type_take (pt,ttm_pool_nid (pool)); + p =ttm_pool_type_take (pt,ttm_pool_nid (pool), objcg);/* * If that fails or previously failed, allocate from system.
@@ -810,7 +885,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,if (!p) { page_caching = ttm_cached; allow_pools = false; - p =ttm_pool_alloc_page (pool, gfp_flags, order); + p =ttm_pool_alloc_page (pool, objcg, gfp_flags, order); }/* If that fails, lower the order if possible and retry. */ if (!p) {
@@ -956,7 +1031,7 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt)while (atomic_long_read (&allocated_pages[nid]) > pool_node_limit[nid]) {unsigned long diff =atomic_long_read (&allocated_pages[nid]) - pool_node_limit[nid]; -ttm_pool_shrink (nid, diff); +ttm_pool_shrink (nid,NULL , diff); } }EXPORT_SYMBOL (ttm_pool_free);
@@ -1076,6 +1151,7 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *tt,if (flags->purge) { shrunken += num_pages; page->private =0 ; + page->memcg_data =0 ;__free_pages_gpu_account (page, order, false);memset (tt->pages + i,0 , num_pages *sizeof (*tt->pages));
@@ -1210,10 +1286,14 @@ static unsigned long ttm_pool_shrinker_scan(struct shrinker *shrink, struct shrink_control *sc) {unsigned long num_freed =0 ; +int num_pools; +spin_lock (&shrinker_lock); + num_pools =list_count_nodes (&shrinker_list); +spin_unlock (&shrinker_lock);do - num_freed +=ttm_pool_shrink (sc->nid, sc->nr_to_scan); -while (num_freed < sc->nr_to_scan && + num_freed +=ttm_pool_shrink (sc->nid, sc->memcg, sc->nr_to_scan); +while (num_pools-- >=0 && num_freed < sc->nr_to_scan &&atomic_long_read (&allocated_pages[sc->nid])); sc->nr_scanned = num_freed;
@@ -1402,7 +1482,7 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); + mm_shrinker =shrinker_alloc (SHRINKER_MEMCG_AWARE |SHRINKER_NUMA_AWARE ,"drm-ttm_pool" );if (!mm_shrinker)return -ENOMEM ;
mm/list_lru.c +1 -0
@@ -627,6 +627,7 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,return 0 ;return __memcg_list_lru_alloc (memcg, lru, gfp); } +EXPORT_SYMBOL_GPL (memcg_list_lru_alloc);int folio_memcg_list_lru_alloc (struct folio *folio,struct list_lru *lru,gfp_t gfp)
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 6/10 ttm: hook up memcg placement flags.
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds a placement flag that requests that any bo with this placement flag set gets accounted for memcg if it's a system memory allocation.
--- drivers/gpu/drm/ttm/ttm_bo.c | 4 ++-- drivers/gpu/drm/ttm/ttm_bo_util.c | 6 +++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/xe/xe_bo.c | 2 +- include/drm/ttm/ttm_placement.h | 3 +++ 5 files changed, 10 insertions(+), 7 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +2 -2
@@ -146,7 +146,7 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, goto out_err;if (mem->mem_type !=TTM_PL_SYSTEM ) { - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , ctx);if (ret) goto out_err; }
@@ -1296,7 +1296,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, bo->resource->placement &TTM_PL_FLAG_MEMCG , ctx);ttm_bo_unreserve (bo);return ret; }
drivers/gpu/drm/ttm/ttm_bo_util.c +3 -3
@@ -167,7 +167,7 @@ int ttm_bo_move_memcpy(struct ttm_buffer_object *bo, src_man =ttm_manager_type (bdev, src_mem->mem_type);if (ttm && ((ttm->page_flags &TTM_TT_FLAG_SWAPPED ) || dst_man->use_tt)) { - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, dst_mem->placement &TTM_PL_FLAG_MEMCG , ctx);if (ret)return ret; }
@@ -352,7 +352,7 @@ static int ttm_bo_kmap_ttm(struct ttm_buffer_object *bo,BUG_ON (!ttm); - ret =ttm_bo_populate (bo, false, &ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , &ctx);if (ret)return ret;
@@ -533,7 +533,7 @@ int ttm_bo_vmap(struct ttm_buffer_object *bo, struct iosys_map *map)pgprot_t prot;void *vaddr; - ret =ttm_bo_populate (bo, false, &ctx); + ret =ttm_bo_populate (bo, mem->placement &TTM_PL_FLAG_MEMCG , &ctx);if (ret)return ret;
drivers/gpu/drm/ttm/ttm_bo_vm.c +1 -1
@@ -225,7 +225,7 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, ttm = bo->ttm; err =ttm_bo_populate (bo, - false, + bo->resource->placement &TTM_PL_FLAG_MEMCG , &ctx);if (err) { if (err == -EINTR || err == -ERESTARTSYS ||
drivers/gpu/drm/xe/xe_bo.c +1 -1
@@ -1927,7 +1927,7 @@ static int xe_bo_fault_migrate(struct xe_bo *bo, struct ttm_operation_ctx *ctx,if (ttm_manager_type (tbo->bdev, tbo->resource->mem_type)->use_tt) { err =xe_bo_wait_usage_kernel (bo, ctx);if (!err) - err =ttm_bo_populate (&bo->ttm, false, ctx); + err =ttm_bo_populate (&bo->ttm, tbo->resource->placement &TTM_PL_FLAG_MEMCG , ctx); }else if (should_migrate_to_smem (bo)) {xe_assert (xe_bo_device (bo), bo->flags &XE_BO_FLAG_SYSTEM ); err =xe_bo_migrate (bo,XE_PL_TT , ctx, exec);
include/drm/ttm/ttm_placement.h +3 -0
@@ -70,6 +70,9 @@/* Placement is only used during eviction */ #define TTM_PL_FLAG_FALLBACK (1 << 4) +/* Placement should account mem cgroup */ +#define TTM_PL_FLAG_MEMCG (1 << 5) +/** * struct ttm_place *
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 7/10 memcontrol: allow objcg api when memcg is config off.
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
amdgpu wants to use the objcg api and not have to enable ifdef around it, so just add a dummy function for the config off path.
--- include/linux/memcontrol.h | 5 +++++ 1 file changed, 5 insertions(+)
include/linux/memcontrol.h +5 -0
@@ -1804,6 +1804,11 @@ static inline void __memcg_kmem_uncharge_page(struct page *page, int order) { } +static inline struct obj_cgroup *get_obj_cgroup_from_current (void ) +{ +return NULL ; +} +static inline struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {return NULL ;
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 8/10 amdgpu: add support for memory cgroups
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds support for adding a obj cgroup to a buffer object, and passing in the placement flags to make sure it's accounted properly.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 13 +++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 3 +++ mm/memcontrol.c | 1 + 5 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +1 -0
@@ -188,6 +188,7 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, bp.domain = initial_domain; bp.bo_ptr_size =sizeof (struct amdgpu_bo ); bp.xcp_id_plus1 = xcp_id_plus1; + bp.objcg =get_obj_cgroup_from_current (); r =amdgpu_bo_create_user (adev, &bp, &ubo);if (r)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +9 -4
@@ -159,7 +159,7 @@ void amdgpu_bo_placement_from_domain(struct amdgpu_bo *abo, u32 domain) places[c].mem_type = abo->flags &AMDGPU_GEM_CREATE_PREEMPTIBLE ?AMDGPU_PL_PREEMPT :TTM_PL_TT ; - places[c].flags =0 ; + places[c].flags =TTM_PL_FLAG_MEMCG ;/* * When GTT is just an alternative to VRAM make sure that we * only use it as fallback and still try to fill up VRAM first.
@@ -174,7 +174,7 @@ void amdgpu_bo_placement_from_domain(struct amdgpu_bo *abo, u32 domain) places[c].fpfn =0 ; places[c].lpfn =0 ; places[c].mem_type =TTM_PL_SYSTEM ; - places[c].flags =0 ; + places[c].flags =TTM_PL_FLAG_MEMCG ; c++; }
@@ -654,16 +654,21 @@ int amdgpu_bo_create(struct amdgpu_device *adev, size =ALIGN (size,PAGE_SIZE ); } -if (!amdgpu_bo_validate_size (adev, size, bp->domain)) +if (!amdgpu_bo_validate_size (adev, size, bp->domain)) { +obj_cgroup_put (bp->objcg);return -ENOMEM ; + }BUG_ON (bp->bo_ptr_size <sizeof (struct amdgpu_bo )); *bo_ptr =NULL ; bo =kvzalloc (bp->bo_ptr_size,GFP_KERNEL ); -if (bo ==NULL ) +if (bo ==NULL ) { +obj_cgroup_put (bp->objcg);return -ENOMEM ; + }drm_gem_private_object_init (adev_to_drm (adev), &bo->tbo.base, size); +ttm_bo_set_cgroup (&bo->tbo, bp->objcg);/* hand the reference to the ttm bo */ bo->tbo.base.funcs = &amdgpu_gem_object_funcs; bo->vm_bo =NULL ; bo->preferred_domains = bp->preferred_domain ? bp->preferred_domain :
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +1 -0
@@ -55,6 +55,7 @@ struct amdgpu_bo_param {enum ttm_bo_type type;bool no_wait_gpu;struct dma_resv *resv; +struct obj_cgroup *objcg;void (*destroy)(struct ttm_buffer_object *bo);/* xcp partition number plus 1, 0 means any partition */ int8_t xcp_id_plus1;
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -0
@@ -155,11 +155,14 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT |AMDGPU_GEM_DOMAIN_CPU ); } +for (int i =0 ; i < abo->placement.num_placement; i++) + abo->placements[i].flags &= ~TTM_PL_FLAG_MEMCG ;break ;case TTM_PL_TT :case AMDGPU_PL_PREEMPT :default :amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + abo->placements[0 ].flags &= ~TTM_PL_FLAG_MEMCG ;break ; } *placement = abo->placement;
mm/memcontrol.c +1 -0
@@ -3032,6 +3032,7 @@ __always_inline struct obj_cgroup *current_obj_cgroup(void)return rcu_dereference_check (root_mem_cgroup->nodeinfo[nid]->objcg,1 ); } +EXPORT_SYMBOL_GPL (current_obj_cgroup);struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 9/10 ttm: add support for a module option to disable memcg integration
In reply to Dave AirlieFrom: Dave Airlie <[email protected]>
This adds a kconfig and a module option to turn off ttm memcg integration completely.
When this is used, no object will ever end up using memcg aware paths.
There is an existing workload that cgroup support might regress, the systems are setup to allocate 1GB of uncached pages at system startup to prime the pool, then any further users will take them from the pool. The current cgroup code might handle that, but it also may regress, so add an option to ttm to avoid using memcg for the pool pages.
--- drivers/gpu/drm/Kconfig | 7 +++++++ drivers/gpu/drm/ttm/ttm_pool.c | 24 +++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-)
drivers/gpu/drm/Kconfig +7 -0
@@ -257,6 +257,13 @@ config DRM_TTM_HELPER help Helpers for ttm-based gem objects +config DRM_TTM_MEMCG + bool "Enable TTM mem cgroup by default" + depends on DRM_TTM + depends on MEMCG + help + Enable the memcg integration by default + config DRM_GEM_DMA_HELPER tristate depends on DRM
drivers/gpu/drm/ttm/ttm_pool.c +21 -3
@@ -119,6 +119,24 @@ static unsigned long page_pool_size;MODULE_PARM_DESC (page_pool_size,"Number of pages in the WC/UC/DMA pool per NUMA node" );module_param (page_pool_size, ulong,0644 ); +/* +* Don't use the memcg aware lru for pooled pages. +* +* There are use-cases where for example one application in a cgroup will preallocate 1GB +* of uncached pages, and immediately release them into the pool, for other consumers +* to use. This use-case could be handled with a proper cgroup hierarchy, but to allow +* that use case to continue to operate as-is, add a module option. +* +* This still stores the pages in the list_lru, it just doesn't use the memcg when +* adding/removing them. +*/ +#define DEFAULT_TTM_MEMCG IS_ENABLED(CONFIG_DRM_TTM_MEMCG) +static bool ttm_memcg =DEFAULT_TTM_MEMCG ; + +MODULE_PARM_DESC(ttm_memcg, "Allow using cgroups with TTM " + "[default=" __stringify(DEFAULT_TTM_MEMCG) "])"); +module_param (ttm_memcg, bool,0444 ); +static unsigned long pool_node_limit[MAX_NUMNODES ];static atomic_long_t allocated_pages[MAX_NUMNODES ];
@@ -321,7 +339,7 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p)); +list_lru_add (&pt->pages, &p->lru, nid, ttm_memcg ?page_memcg_check (p) :NULL );rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]);
@@ -370,7 +388,7 @@ static struct page *ttm_pool_type_take(struct ttm_pool_type *pt, int nid,struct page *page_out =NULL ;int ret;struct mem_cgroup *orig_memcg = orig_objcg ?get_mem_cgroup_from_objcg (orig_objcg) :NULL ; -struct mem_cgroup *memcg = orig_memcg; +struct mem_cgroup *memcg = ttm_memcg ? orig_memcg :NULL ;/* * Attempt to get a page from the current memcg, but if it hasn't got any in it's level,
@@ -844,7 +862,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; -struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ; +struct obj_cgroup *objcg = (ttm_memcg && memcg_account) ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);
Signature · 1 line
2.54.0
Dave Airlie
maintainerPATCH 10/10 xe: create a flag to enable memcg accounting for XE as well.
In reply to Dave AirlieFrom: Maarten Lankhorst <[email protected]>
This adds support for memcg accounting to ttm object used by xe driver.
--- drivers/gpu/drm/xe/xe_bo.c | 10 +++++++--- drivers/gpu/drm/xe/xe_bo.h | 1 + drivers/gpu/drm/xe/xe_lrc.c | 3 ++- drivers/gpu/drm/xe/xe_oa.c | 3 ++- drivers/gpu/drm/xe/xe_pt.c | 3 ++- 5 files changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +7 -3
@@ -56,6 +56,7 @@ static const struct ttm_place sys_placement_flags = { .flags = 0, }; +/* TTM_PL_FLAG_MEMCG is not set, those placements are used for eviction */ static struct ttm_placement sys_placement = { .num_placement =1 , .placement = &sys_placement_flags,
@@ -181,8 +182,8 @@ static void try_add_system(struct xe_device *xe, struct xe_bo *bo, bo->placements[*c] = (struct ttm_place ) { .mem_type =XE_PL_TT , - .flags = (bo_flags &XE_BO_FLAG_VRAM_MASK ) ? -TTM_PL_FLAG_FALLBACK :0 , + .flags =TTM_PL_FLAG_MEMCG | ((bo_flags &XE_BO_FLAG_VRAM_MASK ) ? +TTM_PL_FLAG_FALLBACK :0 ), }; *c +=1 ; }
@@ -2384,6 +2385,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, placement = (type == ttm_bo_type_sg || bo->flags &XE_BO_FLAG_DEFER_BACKING ) ? &sys_placement : &bo->placement; + +if (bo->flags &XE_BO_FLAG_ACCOUNTED ) +ttm_bo_set_cgroup (&bo->ttm,get_obj_cgroup_from_current ()); err =ttm_bo_init_reserved (&xe->ttm, &bo->ttm, type, placement, alignment, &ctx,NULL , resv, xe_ttm_bo_destroy);
@@ -3361,7 +3365,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, args->size & ~PAGE_MASK ))return -EINVAL ; - bo_flags =0 ; + bo_flags =XE_BO_FLAG_ACCOUNTED ;if (args->flags &DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING ) bo_flags |=XE_BO_FLAG_DEFER_BACKING ;
drivers/gpu/drm/xe/xe_bo.h +1 -0
@@ -52,6 +52,7 @@#define XE_BO_FLAG_CPU_ADDR_MIRROR BIT(24)#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26) +#define XE_BO_FLAG_ACCOUNTED BIT(27)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)
drivers/gpu/drm/xe/xe_lrc.c +2 -1
@@ -1661,7 +1661,8 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct xe_vXE_BO_FLAG_GGTT_INVALIDATE ;if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM | +XE_BO_FLAG_ACCOUNTED ; bo = xe_bo_create_pin_map_novm(xe, tile, bo_size, ttm_bo_type_kernel,
drivers/gpu/drm/xe/xe_oa.c +2 -1
@@ -910,7 +910,8 @@ static int xe_oa_alloc_oa_buffer(struct xe_oa_stream *stream, size_t size) bo =xe_bo_create_pin_map_novm (stream->oa->xe, stream->gt->tile, size, ttm_bo_type_kernel, - vram |XE_BO_FLAG_GGTT , false); + vram |XE_BO_FLAG_GGTT |XE_BO_FLAG_ACCOUNTED , + false);if (IS_ERR (bo))return PTR_ERR (bo);
drivers/gpu/drm/xe/xe_pt.c +2 -1
@@ -123,7 +123,8 @@ struct xe_pt *xe_pt_create(struct xe_vm *vm, struct xe_tile *tile,XE_BO_FLAG_IGNORE_MIN_PAGE_SIZE |XE_BO_FLAG_NO_RESV_EVICT |XE_BO_FLAG_PAGETABLE ;if (vm->xef)/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM | +XE_BO_FLAG_ACCOUNTED ; pt->level = level;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 04/10] ttm: add objcg pointer to bo and tt (v2)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Refcount underflow and use-after-free of obj_cgroup due to shallow copy in ghost objects. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Refcount underflow and use-after-free of obj_cgroup due to shallow copy in ghost objects. --
commit 0958e13ef29f184401c5e0dda949311750032e9b
Author: Dave Airlie <[email protected]>
ttm: add objcg pointer to bo and tt (v2)
This commit adds the obj cgroup pointer to the bo and tt structs, and sets it between them.
drivers/gpu/drm/ttm/ttm_bo.c static void ttm_bo_release(struct kref *kref) · line 319
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-0@@ -315,6 +316,7 @@ static void ttm_bo_release(struct kref *kref)dma_resv_unlock (bo->base.resv); } +obj_cgroup_put (bo->objcg);
[Severity: High]
Does this introduce a double-put and refcount underflow for the objcg during buffer migrations?
Looking at ttm_buffer_object_transfer() in drivers/gpu/drm/ttm/ttm_bo_util.c, it performs a shallow copy to create a ghost object:
drivers/gpu/drm/ttm/ttm_bo_util.c:ttm_buffer_object_transfer() {
...
fbo = kmalloc(sizeof(*fbo), GFP_KERNEL);
...
fbo->base = *bo;
...
}Because fbo->base = *bo directly copies the bo->objcg pointer without acquiring an additional reference via obj_cgroup_get(), both the original buffer object and the new ghost object will hold the exact same objcg pointer.
When these objects are eventually destroyed, both will execute ttm_bo_release() and call obj_cgroup_put(bo->objcg) on the same cgroup, resulting in an unbalanced put and a potential use-after-free.
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_bo.c +0
-0atomic_dec (&ttm_glob.bo_count); bo->destroy (bo); }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH 07/10] memcontrol: allow objcg api when memcg is config off.
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Missing `!CONFIG_MEMCG` stubs for GPU memcg accounting functions (`mem_cgroup_charge_gpu_page`…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Missing !CONFIG_MEMCG stubs for GPU memcg accounting functions (mem_cgroup_charge_gpu_page, etc.) causes a linker error. --
commit ba45048ba307805ed185b05a25fe4419acb46277
Author: Dave Airlie <[email protected]>
memcontrol: allow objcg api when memcg is config off.
This commit adds a dummy function for get_obj_cgroup_from_current() when CONFIG_MEMCG is disabled. This allows drivers like amdgpu to use the objcg API without needing ifdef blocks around it.
include/linux/memcontrol.h static inline void __memcg_kmem_uncharge_page(struct page *page, int order) · line 1811
Quoted 13 lines
include/linux/memcontrol.h +5
-0@@ -1804,6 +1804,11 @@ static inline void __memcg_kmem_uncharge_page(struct page *page, int order) { } +static inline struct obj_cgroup *get_obj_cgroup_from_current (void ) +{ +return NULL ; +} +
[Severity: Low]
Are there other objcg API functions that also need dummy stubs to prevent linker errors when CONFIG_MEMCG is disabled?
For example, mem_cgroup_charge_gpu_page() is declared unconditionally in include/linux/memcontrol.h:
bool mem_cgroup_charge_gpu_page(struct obj_cgroup *objcg, struct page *page,
unsigned int order,
gfp_t gfp_mask, bool reclaim);
This function is used directly without an ifdef block in the TTM driver:
drivers/gpu/drm/ttm/ttm_pool.c:ttm_pool_alloc_page() {
...
if (!mem_cgroup_charge_gpu_page(objcg, p, order, gfp_flags, false)) {
...
}Does building a kernel with CONFIG_MEMCG=n and CONFIG_DRM_TTM=y result in undefined reference linker errors for mem_cgroup_charge_gpu_page() and other related GPU memcg accounting functions?
Quoted 3 lines
include/linux/memcontrol.h +0
-0static inline struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {return NULL ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
[email protected]
AI reviewRe: [PATCH 03/10] ttm/pool: initialise the shrinker earlier
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [Critical] Premature shrinker registration leads to kernel crash on empty list traversal
- [High] Unhandled memory allocation failure in ttm_pool_mgr_init() leaves global memory pools uninitialized
Pre-existing issues:
- [Critical] Concurrent shrinker invocation during module teardown causes invalid pointer dereference
- [High] Missing error handling for list_lru_init_memcg() leading to NULL pointer dereference --
commit 979d1132acdad03dc6f5c6c5ed80def9a1603b9f
Author: Dave Airlie <[email protected]>
ttm/pool: initialise the shrinker earlier
Later memcg enablement needs the shrinker initialised before the list lru, Just move it for now.
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1408
Quoted 11 lines
drivers/gpu/drm/ttm/ttm_pool.c +4
-0@@ -1402,6 +1402,17 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); + mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); +if (!mm_shrinker) +return -ENOMEM ; +
[Severity: High]
If shrinker_alloc() fails here and returns -ENOMEM, ttm_pool_mgr_init() exits early before initializing global_write_combined, global_uncached, and other pools.
Since the caller ttm_global_init() ignores the return value of ttm_pool_mgr_init(), will this allow TTM initialization to continue with uninitialized global pools, resulting in a NULL pointer dereference later when list_lru_walk_one() accesses the zero-initialized list_lru structures?
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1415
Quoted 7 lines
drivers/gpu/drm/ttm/ttm_pool.c +7
-0+ mm_shrinker->count_objects = ttm_pool_shrinker_count; + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; + mm_shrinker->batch =TTM_SHRINKER_BATCH ; + mm_shrinker->seeks =1 ; + +shrinker_register (mm_shrinker); +
[Severity: Critical]
Does registering the shrinker at this point expose it before the shrinker_list is populated by ttm_pool_type_init() in the loop below?
If the shrinker is invoked immediately, for example via memory pressure or a debugfs write, ttm_pool_shrink() will perform list_first_entry() on an empty shrinker_list:
ttm_pool_shrinker_scan()->ttm_pool_shrink() {
spin_lock(&shrinker_lock);
pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list);
list_move_tail(&pt->shrinker_list, &shrinker_list);
spin_unlock(&shrinker_lock);
}Could this result in a kernel panic by returning a garbage pointer from the empty list and passing it to list_lru_walk_node()?
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1418
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0for (i =0 ; i <NR_PAGE_ORDERS ; ++i) {ttm_pool_type_init (&global_write_combined[i],NULL , ttm_write_combined, i);
[Severity: High]
This is a pre-existing issue, but is it safe to ignore the return value of list_lru_init_memcg() called internally by ttm_pool_type_init()?
ttm_pool_type_init() {
...
list_lru_init_memcg(&pt->pages, mm_shrinker);
}If list_lru_init_memcg() fails due to system memory pressure and returns an error code like -ENOMEM, pt->pages.node remains NULL while the pool type is still added to the active lists. Could subsequent operations like list_lru_add() dereference this NULL pointer?
[ ... ]
[Severity: Critical]
This is a pre-existing issue, but looking at the teardown path in ttm_pool_mgr_fini(), the global pools are destroyed and removed from shrinker_list before the shrinker is unregistered:
ttm_pool_mgr_fini() {
for (i = 0; i < NR_PAGE_ORDERS; ++i) {
ttm_pool_type_fini(&global_write_combined[i]);
...
}
shrinker_free(mm_shrinker);
}If the shrinker is invoked concurrently during this window, it will again execute list_first_entry() on an empty shrinker_list in ttm_pool_shrink().
Unlike ttm_pool_fini(), ttm_pool_mgr_fini() does not call ttm_pool_synchronize_shrinkers() before destroying the pools. Does this leave a window open for an invalid pointer dereference during module teardown?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH 02/10] ttm: add a memcg accounting flag to the alloc/populate APIs
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The newly added `memcg_account` flag is not propagated to the TT restore path, causing GPU memory…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The newly added memcg_account flag is not propagated to the TT restore path, causing GPU memory to escape memory cgroup limits after being swapped out and restored. --
commit 915e66ed2150a0401b7163c79a1f518da0005591
Author: Dave Airlie <[email protected]>
ttm: add a memcg accounting flag to the alloc/populate APIs
This commit adds a memcg_account flag to the TTM populate and allocate APIs to prepare for future memcg accounting. It updates driver callbacks to accept the new parameter.
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 940
Quoted 6 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -935,7 +937,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,return 0 ; } -return __ttm_pool_alloc (pool, tt, ctx, &alloc, restore); +return __ttm_pool_alloc (pool, tt, false, ctx, &alloc, restore);
[Severity: High]
Does this hardcoded false cause memory cgroup limits to be bypassed for restored pages?
When a TT object is swapped out (backed up), its pages are freed and uncharged from the memcg. When swapped back in, the pages are re-allocated, but because ttm_pool_restore_and_alloc() hardcodes false for the memcg_account parameter, they are not re-charged.
This causes a bypass of memory cgroup limits for restored pages, which could allow an application to allocate GPU memory, trigger swap, and bypass the restrictions upon restore.
Quoted 7 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0}/** drivers/gpu/drm/xe/xe_bo.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 554
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0@@ -536,6 +536,7 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo, }static int xe_ttm_tt_populate (struct ttm_device *ttm_dev,struct ttm_tt *tt, +bool memcg_account,struct ttm_operation_ctx *ctx) { struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);@@ -553,7 +554,7 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, err =ttm_tt_restore (ttm_dev, tt, ctx);
[Severity: High]
Will we need to update ttm_tt_restore() to accept the memcg_account parameter as well?
Since ttm_tt_restore() doesn't accept the new memcg_account parameter, drivers like xe here receive the flag but cannot pass it down when restoring a backed-up TT object, losing the accounting information.
Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +1
-1}else {ttm_tt_clear_backed_up (tt); - err =ttm_pool_alloc (&ttm_dev->pool, tt, ctx); + err =ttm_pool_alloc (&ttm_dev->pool, tt, memcg_account, ctx); }if (err)return err;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH 05/10] ttm/pool: enable memcg tracking and shrinker. (v3)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider: - [Critical] Missing `list_lru_destroy` in `ttm_pool_type_fini` causes an exploitable Use-After-Free during…
Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
- [Critical] Missing list_lru_destroy in ttm_pool_type_fini causes an exploitable Use-After-Free during cgroup reparenting.
- [High] Missing memcg_list_lru_alloc for DMA32 pools and ignored allocation failures cause continuous VM_WARN_ON spam and broken isolation.
- [High] Clearing page->memcg_data before uncharging causes permanent memcg charge leaks.
- [High] pool_lru_get_page frees write-combined or uncached pages directly to the buddy allocator without restoring them to write-back, causing cache aliasing.
- [High] ttm_pool_check_objcg hardcodes GFP_KERNEL, overriding caller constraints like __GFP_RETRY_MAYFAIL.
- [High] Unaccounted allocations from the pool permanently charge the old memcg for active memory, breaking isolation.
- [Medium] ttm_pool_shrinker_count returns the global pool size rather than the memcg-specific pool size, breaking memcg-aware reclaim heuristics. --
commit 71c02307644b082161406f13e214b752caedbe33
Author: Dave Airlie <[email protected]>
Subject: ttm/pool: enable memcg tracking and shrinker. (v3)
This commit enables memcg tracking for the TTM pool backend and makes the shrinker memcg aware. It also introduces logic to allow newer memcgs to reparent and reuse pooled pages from parent memcgs.
[ ... ]
Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
Quoted 1 line
drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -337,20 +340,56 @@ static enum lru_status take_one_from_lru(struct list_head *item,
[ ... ]
Quoted 12 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-1+static int pool_lru_get_page (struct ttm_pool_type *pt,int nid, +struct page **page_out, +struct obj_cgroup *objcg, +struct mem_cgroup *memcg) {int ret;struct page *p =NULL ;unsigned long nr_to_walk =1 ; +unsigned int num_pages =1 << pt->order; - ret =list_lru_walk_node (&pt->pages, nid, take_one_from_lru, (void *)&p, &nr_to_walk); + ret =list_lru_walk_one (&pt->pages, nid, memcg, take_one_from_lru, (void *)&p, &nr_to_walk);
drivers/gpu/drm/ttm/ttm_pool.c static enum lru_status take_one_from_lru(struct list_head *item, · line 360
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +6
-3if (ret ==1 && p) { -atomic_long_sub (1 << pt->order, &allocated_pages[nid]); -mod_lruvec_page_state (p,NR_GPU_ACTIVE , (1 << pt->order)); -mod_lruvec_page_state (p,NR_GPU_RECLAIM , -(1 << pt->order)); +atomic_long_sub (num_pages, &allocated_pages[nid]); + +if (!mem_cgroup_move_gpu_page_reclaim (objcg, p, pt->order, false)) { +__free_pages (p, pt->order); + p =NULL ; + }
[Severity: High]
Does this direct call to __free_pages() bypass the required caching restorations?
In pool_lru_get_page(), when mem_cgroup_move_gpu_page_reclaim() fails, the page is freed directly to the buddy allocator. If the page came from a write-combined or uncached pool, this skips the set_pages_wb() call normally performed by ttm_pool_free_page().
Could this lead to cache aliasing issues and machine check exceptions on architectures like x86 since the direct map caching attributes are not restored to write-back?
[Severity: High]
Will unaccounted allocations permanently charge the previous memcg owner?
When a process not tracked by a memcg (or the root memcg) allocates memory from the pool, pool_lru_get_page() passes a NULL objcg to mem_cgroup_move_gpu_page_reclaim().
Looking at mm/memcontrol.c:mem_cgroup_move_gpu_page_reclaim(), if the new objcg is NULL, it evaluates the condition (!new_objcg) as true and moves the charge from NR_GPU_RECLAIM to NR_GPU_ACTIVE on the old memcg instead of uncharging it.
Doesn't this leave the old memcg permanently charged for pages it no longer owns while the root process uses them, breaking memcg isolation?
Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +3
-1} -return p; + *page_out = p; +return ret; +}
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_fini(struct ttm_pool_type *pt) · line 443
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0@@ -403,6 +442,31 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt)ttm_pool_dispose_list (pt, &dispose); }
[Severity: Critical]
Is list_lru_destroy() required here to prevent a use-after-free during cgroup reparenting?
When a DRM driver unbinds or the TTM module is unloaded, ttm_pool_type_fini() cleans up the pool type but does not call list_lru_destroy(). The list_lru structures remain registered in the global memcg_list_lrus list.
If a memory cgroup is subsequently deleted, memcg_reparent_list_lrus() iterates over this global list and accesses the freed memory. Can we add a call to list_lru_destroy(&pt->pages) to safely unregister it?
Quoted 11 lines
drivers/gpu/drm/ttm/ttm_pool.c +11
-0+/* +* This function doesn't currently check dma32, because no driver using this +* support dma32. This should be added and debugged when that changes. +*/ +static void ttm_pool_check_objcg (struct obj_cgroup *objcg) +{ +#ifdef CONFIG_MEMCG +int r =0 ; +struct mem_cgroup *memcg; +if (!objcg) +return ;
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_fini(struct ttm_pool_type *pt) · line 465
Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10
-0+ + memcg =get_mem_cgroup_from_objcg (objcg); +for (unsigned i =0 ; i <NR_PAGE_ORDERS ; i++) { + r =memcg_list_lru_alloc (memcg, &global_write_combined[i].pages,GFP_KERNEL ); +if (r) +break ; + r =memcg_list_lru_alloc (memcg, &global_uncached[i].pages,GFP_KERNEL ); +if (r) +break ; + }
[Severity: High]
Can skipping the global_dma32_* pools here cause kernel warning spam and broken tracking?
The loop in ttm_pool_check_objcg() explicitly allocates list_lrus for global_write_combined and global_uncached pools, but skips the DMA32 variants.
If a driver using DMA32 pools frees a page to the pool, list_lru_add() will attempt to add it to the memcg's list. Because the list wasn't allocated, lock_list_lru_of_memcg() fails to find the list, triggers a VM_WARN_ON(!css_is_dying), and escalates to the parent memcg. Could this be triggered trivially by userspace interacting with DMA32 DRM drivers inside a memory cgroup?
[Severity: High]
Is it safe to use a hardcoded GFP_KERNEL allocation flag here?
The __ttm_pool_alloc() caller might operate under strict constraints, passing a ttm_operation_ctx that sets gfp_retry_mayfail to avoid deadlocks with the GPU shrinker.
Because ttm_pool_check_objcg() uses GFP_KERNEL directly instead of propagating the caller's allocation flags, it drops constraints like __GFP_RETRY_MAYFAIL and __GFP_NOWARN. Might this allow the allocator to block indefinitely or invoke the OOM killer while critical DRM locks are held?
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +3
-0+mem_cgroup_put (memcg); +#endif +}
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore, · line 687
Quoted 7 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-0@@ -612,6 +682,7 @@ static int ttm_pool_restore_commit(struct ttm_pool_tt_restore *restore,*/ ttm_pool_split_for_swap (restore->pool, p);copy_highpage (restore->alloced_page + i, p); + p->memcg_data =0 ;__free_pages_gpu_account (p,0 , false); }
[Severity: High]
Does clearing p->memcg_data before uncharging cause the memcg charge to leak?
By explicitly setting p->memcg_data = 0 here, the subsequent call to __free_pages_gpu_account() -> mem_cgroup_uncharge_gpu_page() cannot identify which memcg to uncharge, since page_objcg(page) will return NULL.
Wouldn't this make the uncharge a no-op, permanently leaking the memcg GPU charge counters and the obj_cgroup reference?
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_mgr_init(unsigned long num_pages) · line 1485
Quoted 6 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -1402,7 +1482,7 @@ int ttm_pool_mgr_init(unsigned long num_pages)spin_lock_init (&shrinker_lock);INIT_LIST_HEAD (&shrinker_list); - mm_shrinker =shrinker_alloc (SHRINKER_NUMA_AWARE ,"drm-ttm_pool" ); + mm_shrinker =shrinker_alloc (SHRINKER_MEMCG_AWARE |SHRINKER_NUMA_AWARE ,"drm-ttm_pool" );
[Severity: Medium]
How does the shrinker count callback handle specific memory cgroups now that it is memcg aware?
This registers the shrinker with SHRINKER_MEMCG_AWARE. The VM subsystem will now call the count_objects callback (ttm_pool_shrinker_count) for specific memory cgroups under pressure, passing sc->memcg.
However, ttm_pool_shrinker_count() ignores sc->memcg and always returns the global pool size via atomic_long_read(&allocated_pages[sc->nid]). Could this cause inefficient and disproportionate reclaim pressure, as a memcg with zero TTM pages is falsely reported as holding the global amount?
Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0if (!mm_shrinker)return -ENOMEM ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH 01/10] memcg: add support for GPU page counters. (v4)
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [High] mem_cgroup_move_gpu_page_reclaim silently fails to move pages to or from the root cgroup…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] mem_cgroup_move_gpu_page_reclaim silently fails to move pages to or from the root cgroup, bypassing memory limits and leaking charges.
- [High] Tagging GPU pages with MEMCG_DATA_KMEM without corresponding kmem counter charges causes global kmem underflows on standard page frees.
- [Medium] Cross-cgroup page moves in mem_cgroup_move_gpu_page_reclaim completely ignore the to_reclaim parameter and corrupt stats.
- [Low] Missing static inline stubs for GPU memory tracking APIs when CONFIG_MEMCG is disabled.
--
commit 497e4e8b1bbad8552dbc2f2daeb121cd40f196af
Author: Dave Airlie <[email protected]>
memcg: add support for GPU page counters. (v4)
This introduces new statistics and memcontrol APIs for dealing with GPU system memory allocations. It allows charging, uncharging, and moving pages between object cgroups and reclaim states.
Quoted 9 lines
include/linux/memcontrol.h +1
-0@@ -1583,6 +1583,17 @@ static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb)#endif /* CONFIG_CGROUP_WRITEBACK */ struct sock ; +bool mem_cgroup_charge_gpu_page(struct obj_cgroup *objcg, struct page *page,
include/linux/memcontrol.h static inline void mem_cgroup_flush_foreign(struct bdi_writeback *wb) · line 1596
Quoted 10 lines
include/linux/memcontrol.h +10
-0+unsigned int order, + gfp_t gfp_mask, bool reclaim); +void mem_cgroup_uncharge_gpu_page (struct page *page, +unsigned int order, +bool reclaim); +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *objcg, +struct page *page, +unsigned int order, +bool to_reclaim); +
[Severity: Low]
Are there missing static inline stubs for these APIs when CONFIG_MEMCG is disabled? Will this cause undefined reference link errors for drivers using these functions on non-memcg builds, since they are declared globally but only implemented in mm/memcontrol.c?
Quoted 3 lines
include/linux/memcontrol.h +0
-0#ifdef CONFIG_MEMCG extern struct static_key_false memcg_sockets_enabled_key;#define mem_cgroup_sockets_enabled static_branch_unlikely(&memcg_sockets_enabled_key)
[ ... ]
Quoted 4 lines
mm/memcontrol.c +0
-0
[ ... ]
Quoted 7 lines
mm/memcontrol.c +3
-0@@ -5508,6 +5512,106 @@ void mem_cgroup_flush_workqueue(void)flush_workqueue (memcg_wq); } +/** +* mem_cgroup_charge_gpu_page - charge a page to GPU memory tracking +* @objcg: objcg to charge, NULL charges root memcg
[ ... ]
Quoted 9 lines
mm/memcontrol.c +9
-0+bool mem_cgroup_charge_gpu_page (struct obj_cgroup *objcg,struct page *page, +unsigned int order,gfp_t gfp_mask,bool reclaim) +{ +unsigned int nr_pages =1 << order; +struct mem_cgroup *memcg =NULL ; +struct lruvec *lruvec; +int ret; + +if (objcg) {
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5544
Quoted 10 lines
mm/memcontrol.c +10
-0+ memcg =get_mem_cgroup_from_objcg (objcg); + + ret =try_charge_memcg (memcg, gfp_mask, nr_pages); +if (ret) { +mem_cgroup_put (memcg); +return false; + } + +obj_cgroup_get (objcg); +page_set_objcg (page, objcg);
[Severity: High]
Since page_set_objcg() embeds the MEMCG_DATA_KMEM flag into the page, if this GPU page is later freed directly by the kernel rather than through mem_cgroup_uncharge_gpu_page(), will the VM route the uncharge to __memcg_kmem_uncharge_page()?
Could this improperly decrement the global kmem counters (which were never incremented during the GPU page charging) and bypass the decrement of the NR_GPU_ACTIVE or NR_GPU_RECLAIM stats?
Quoted 9 lines
mm/memcontrol.c +9
-0+ } + + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); +mod_lruvec_state (lruvec, reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , nr_pages); + +mem_cgroup_put (memcg); +return true; +} +EXPORT_SYMBOL_GPL (mem_cgroup_charge_gpu_page);
[ ... ]
Quoted 4 lines
mm/memcontrol.c +4
-0+/** +* mem_cgroup_move_gpu_reclaim - move pages from gpu to gpu reclaim and back +* @new_objcg: objcg to move page to, NULL if just stats update. +* @nr_pages: number of pages to move
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5595
Quoted 10 lines
mm/memcontrol.c +10
-0+* @to_reclaim: true moves pages into reclaim, false moves them back +*/ +bool mem_cgroup_move_gpu_page_reclaim (struct obj_cgroup *new_objcg, +struct page *page, +unsigned int order, +bool to_reclaim) +{ +struct obj_cgroup *objcg =page_objcg (page); + +if (!objcg || !new_objcg || objcg == new_objcg) {
[Severity: High]
Does this conditional unintentionally merge cross-cgroup moves involving the root cgroup (where objcg or new_objcg is NULL) into the fast-path for in-place state transitions?
If a page is moved to or from the root cgroup, it looks like this will silently fail to transfer ownership to the new child cgroup, bypassing memory limits or leaving charges stuck in the original cgroup indefinitely.
Quoted 5 lines
mm/memcontrol.c +5
-0+struct mem_cgroup *memcg = objcg ?get_mem_cgroup_from_objcg (objcg) :NULL ; +struct lruvec *lruvec; +unsigned long flags; +int nr_pages =1 << order; +
mm/memcontrol.c void mem_cgroup_flush_workqueue(void) · line 5610
Quoted 10 lines
mm/memcontrol.c +10
-0+ lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); +local_irq_save (flags); +mod_lruvec_state (lruvec, to_reclaim ?NR_GPU_RECLAIM :NR_GPU_ACTIVE , nr_pages); +mod_lruvec_state (lruvec, to_reclaim ?NR_GPU_ACTIVE :NR_GPU_RECLAIM , -nr_pages); +local_irq_restore (flags); +mem_cgroup_put (memcg); +return true; + }else { +mem_cgroup_uncharge_gpu_page (page, order, true); +return mem_cgroup_charge_gpu_page (new_objcg, page, order,0 , false);
[Severity: Medium]
Does this block ignore the to_reclaim boolean parameter by hardcoding true for the uncharge and false for the charge?
It looks like this will corrupt the stats by always decrementing the RECLAIM counter of the old cgroup and always incrementing the ACTIVE counter of the new cgroup, regardless of the caller's requested state.
Quoted 3 lines
mm/memcontrol.c +3
-0+ } +} +EXPORT_SYMBOL_GPL (mem_cgroup_move_gpu_page_reclaim);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH 06/10] ttm: hook up memcg placement flags.
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Direct dereference of `bo->resource` without a NULL check in `ttm_bo_setup_export` causes a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Direct dereference of bo->resource without a NULL check in ttm_bo_setup_export causes a kernel panic when exporting pipeline-gutted BOs. --
commit dc7f000aeb70e2be50c58ca89d0162684ebb0574
Author: Dave Airlie <[email protected]>
ttm: hook up memcg placement flags.
This commit adds a placement flag to request memory cgroup accounting for system memory allocations. It updates various callers of ttm_bo_populate() to pass this flag by checking the placement flags of the buffer object's resource.
Quoted 4 lines
drivers/gpu/drm/ttm/ttm_bo.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_bo.c int ttm_bo_setup_export(struct ttm_buffer_object *bo, · line 1299
Quoted 6 lines
drivers/gpu/drm/ttm/ttm_bo.c +1
-1@@ -1296,7 +1296,7 @@ int ttm_bo_setup_export(struct ttm_buffer_object *bo,if (ret !=0 )return ret; - ret =ttm_bo_populate (bo, false, ctx); + ret =ttm_bo_populate (bo, bo->resource->placement &TTM_PL_FLAG_MEMCG , ctx);
[Severity: Critical]
Can this dereference a NULL pointer if the buffer object has been pipeline-gutted?
Under memory pressure, TTM eviction logic may pipeline-gut the BO via ttm_bo_pipeline_gutting(), which frees the resource and sets bo->resource to NULL.
If userspace then invokes the DMA-BUF export ioctl on this gutted BO, the driver's export callback (like amdgpu_gem_prime_export) can call ttm_bo_setup_export() without re-validating the BO. This results in an unconditional dereference of the NULL pointer when evaluating bo->resource->placement.
Would it be safer to check if bo->resource is valid before checking its placement flags?
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_bo.c +0
-0ttm_bo_unreserve (bo);return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH 09/10] ttm: add support for a module option to disable memcg integration
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [Medium] The Kconfig option DRM_TTM_MEMCG lacks a default y directive, which silently disables TTM memcg accounting by default, contradicting the intended behavior described in the commit message.
Pre-existing issues:
- [High] Explicitly zeroing page->memcg_data prior to uncharging skips the memcg uncharge process, causing persistent obj_cgroup reference leaks and memcg counter leaks.
- [High] DMA-allocated pool pages bypass accounting but are unconditionally moved between active/reclaim states, permanently underflowing NR_GPU_ACTIVE and leaking NR_GPU_RECLAIM stats. --
commit 29deabc39c8a2c600d3de5d3662fd9003ff7aa7a
Author: Dave Airlie <[email protected]>
ttm: add support for a module option to disable memcg integration
This adds a kconfig and a module option to turn off ttm memcg integration completely.
drivers/gpu/drm/Kconfig config DRM_TTM_HELPER · line 263
Quoted 11 lines
drivers/gpu/drm/Kconfig +4
-0@@ -257,6 +257,13 @@ config DRM_TTM_HELPER help Helpers for ttm-based gem objects +config DRM_TTM_MEMCG + bool "Enable TTM mem cgroup by default" + depends on DRM_TTM + depends on MEMCG
[Severity: Medium]
Does this Kconfig option need a "default y" directive? The help text and commit message state that the feature is enabled by default, but without it, the configuration will default to disabled in standard builds.
Quoted 3 lines
drivers/gpu/drm/Kconfig +3
-0+ help + Enable the memcg integration by default +
[ ... ]
Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0
-0
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p) · line 345
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -321,7 +339,7 @@ static void ttm_pool_type_give(struct ttm_pool_type *pt, struct page *p)INIT_LIST_HEAD (&p->lru);rcu_read_lock (); -list_lru_add (&pt->pages, &p->lru, nid,page_memcg_check (p)); +list_lru_add (&pt->pages, &p->lru, nid, ttm_memcg ?page_memcg_check (p) :NULL );rcu_read_unlock ();atomic_long_add (num_pages, &allocated_pages[nid]);
[Severity: High]
This isn't a bug introduced by this patch, but does the call to mem_cgroup_move_gpu_page_reclaim() that immediately follows this code corrupt the memcg stats for DMA pages?
When ttm_pool_uses_dma_alloc() is true, ttm_pool_alloc_page() bypasses mem_cgroup_charge_gpu_page(), meaning NR_GPU_ACTIVE is never incremented.
However, when the page is added to the pool here in ttm_pool_type_give(), it unconditionally calls mem_cgroup_move_gpu_page_reclaim(..., true), which subtracts from NR_GPU_ACTIVE and adds to NR_GPU_RECLAIM.
Finally, when the shrinker frees the page, ttm_pool_free_page() bypasses __free_pages_gpu_account(), so the RECLAIM stat is never decremented. Does this permanently underflow NR_GPU_ACTIVE and leak NR_GPU_RECLAIM?
[ ... ]
drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 868
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +1
-1@@ -844,7 +862,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,bool allow_pools;struct page *p;int r; -struct obj_cgroup *objcg = memcg_account ? tt->objcg :NULL ; +struct obj_cgroup *objcg = (ttm_memcg && memcg_account) ? tt->objcg :NULL ;WARN_ON (!alloc->remaining_pages ||ttm_tt_is_populated (tt));WARN_ON (alloc->dma_addr && !pool->dev);
[Severity: High]
This is a pre-existing issue, but does explicitly zeroing page->memcg_data prior to uncharging cause an obj_cgroup leak?
During backup restoration in ttm_pool_restore_commit():
copy_highpage(restore->alloced_page + i, p); p->memcg_data = 0; __free_pages_gpu_account(p, 0, false);
And similarly during purging in ttm_pool_shrink():
page->private = 0; page->memcg_data = 0; __free_pages_gpu_account(page, order, false);
Since __free_pages_gpu_account() relies on mem_cgroup_uncharge_gpu_page() which requires memcg_data to locate the correct obj_cgroup, doesn't clearing it beforehand cause the uncharge to silently fail, leaking the stats and the reference?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
[email protected]
AI reviewRe: [PATCH 08/10] amdgpu: add support for memory cgroups
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Evicted GPU buffers bypass memory cgroup limits because the memcg accounting flag is cleared…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Evicted GPU buffers bypass memory cgroup limits because the memcg accounting flag is cleared during VRAM eviction and pages are never recharged upon re-validation.
- [Low] Missing dummy implementations for GPU page memory control functions when CONFIG_MEMCG is disabled causes a linker error.
--
commit 14a7ef9775310750fa8f3933b124ce94b537e5c7
Author: Dave Airlie <[email protected]>
amdgpu: add support for memory cgroups
This adds support for adding a obj cgroup to a buffer object, and passing in the placement flags to make sure it's accounted properly.
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 159
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-0@@ -155,11 +155,14 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT |AMDGPU_GEM_DOMAIN_CPU ); } +for (int i =0 ; i < abo->placement.num_placement; i++) + abo->placements[i].flags &= ~TTM_PL_FLAG_MEMCG ;
[Severity: High]
Does clearing TTM_PL_FLAG_MEMCG during eviction allow memory cgroup limits to be bypassed?
When a user forces eviction (for example, by allocating more VRAM), the flag is cleared in amdgpu_evict_flags() and TTM moves the buffer to system memory. The pages are then allocated via ttm_pool_alloc() with memcg_account set to false, leaving them uncharged.
If the user later re-validates the buffer into GTT, ttm_bo_validate() calls ttm_resource_compatible(). This function only checks the memory type and contiguous flags, ignoring TTM_PL_FLAG_MEMCG.
This seems to allow the buffer to remain backed by uncharged system pages, potentially letting a containerized workload exhaust host system memory.
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 165
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1
-0break ;case TTM_PL_TT :case AMDGPU_PL_PREEMPT :default :amdgpu_bo_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + abo->placements[0 ].flags &= ~TTM_PL_FLAG_MEMCG ;
[Severity: High]
Does this also allow bypassing memory cgroup limits for TTM_PL_TT evictions to the CPU domain for the same reason?
[ ... ]
mm/memcontrol.c __always_inline struct obj_cgroup *current_obj_cgroup(void) · line 3038
Quoted 12 lines
mm/memcontrol.c +1
-0@@ -3032,6 +3032,7 @@ __always_inline struct obj_cgroup *current_obj_cgroup(void)return rcu_dereference_check (root_mem_cgroup->nodeinfo[nid]->objcg,1 ); } +EXPORT_SYMBOL_GPL (current_obj_cgroup);struct obj_cgroup *get_obj_cgroup_from_folio (struct folio *folio) {
[Severity: Low]
Will this series cause a linker error when CONFIG_DRM_TTM=y and CONFIG_MEMCG=n?
It looks like mem_cgroup_charge_gpu_page(), mem_cgroup_uncharge_gpu_page(), and mem_cgroup_move_gpu_page_reclaim() are declared unconditionally in include/linux/memcontrol.h.
However, their definitions in mm/memcontrol.c are skipped when CONFIG_MEMCG is disabled.
Since ttm_pool_alloc_page() in drivers/gpu/drm/ttm/ttm_pool.c calls these without ifdef guards, this seems to lead to undefined references during linking.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
[email protected]
AI reviewRe: [PATCH 10/10] xe: create a flag to enable memcg accounting for XE as well.
In reply to Dave AirlieThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Missing memcg accounting for userspace dumb buffers created via xe_bo_dumb_create. - [High] Page…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing memcg accounting for userspace dumb buffers created via xe_bo_dumb_create.
- [High] Page tables created asynchronously via workqueues are charged to the kworker's root memcg instead of the user's memcg.
- [High] BOs created with deferred backing bypass memcg accounting upon CPU page fault.
--
commit ebf9090c8281d875d2f980bb34d324c88a8b277f
Author: Maarten Lankhorst <[email protected]>
xe: create a flag to enable memcg accounting for XE as well.
This adds support for memcg accounting to ttm object used by xe driver.
drivers/gpu/drm/xe/xe_bo.c static const struct ttm_place sys_placement_flags = { · line 60
Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0@@ -56,6 +56,7 @@ static const struct ttm_place sys_placement_flags = { .flags = 0, }; +/* TTM_PL_FLAG_MEMCG is not set, those placements are used for eviction */ static struct ttm_placement sys_placement = {
[Severity: High]
If userspace creates a buffer object with the deferred backing flag, mmaps it, and accesses it via the CPU before the GPU validates it, xe_bo_init_locked() assigns sys_placement as the initial placement. Since sys_placement lacks TTM_PL_FLAG_MEMCG, will a CPU page fault (which calls xe_bo_fault_migrate() and ttm_bo_populate()) allocate memory without charging the memcg? Does this allow userspace to bypass memcg limits for deferred backing buffers?
Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0.num_placement = 1, .placement = &sys_placement_flags,};
[ ... ]
drivers/gpu/drm/xe/xe_bo.c struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, · line 2390
Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +3
-0@@ -2384,6 +2385,9 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, placement = (type == ttm_bo_type_sg || bo->flags &XE_BO_FLAG_DEFER_BACKING ) ? &sys_placement : &bo->placement; + +if (bo->flags &XE_BO_FLAG_ACCOUNTED ) +ttm_bo_set_cgroup (&bo->ttm,get_obj_cgroup_from_current ());
[Severity: High]
When page tables are created asynchronously, such as from GPU page fault handlers or rebinding operations executing in workqueues via xe_pt_create(), get_obj_cgroup_from_current() will resolve to the kworker's context. Does this cause these allocations to be incorrectly charged to the root memcg instead of the user's memcg?
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0err = ttm_bo_init_reserved(&xe->ttm, &bo->ttm, type, placement, alignment,
[ ... ]
drivers/gpu/drm/xe/xe_bo.c int xe_gem_create_ioctl(struct drm_device *dev, void *data, · line 3370
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +1
-1@@ -3361,7 +3365,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,if (XE_IOCTL_DBG (xe, args->size & ~PAGE_MASK ))return -EINVAL ; - bo_flags =0 ; + bo_flags =XE_BO_FLAG_ACCOUNTED ;if (args->flags &DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING ) bo_flags |=XE_BO_FLAG_DEFER_BACKING ;
[Severity: High]
This correctly adds the accounting flag for standard GEM object creation, but are dumb buffers created via xe_bo_dumb_create() missing this flag?
Dumb buffers are also allocated on behalf of userspace via DRM_IOCTL_MODE_CREATE_DUMB:
xe_bo_dumb_create() {
...
bo = xe_bo_create_user(xe, NULL, args->size,
DRM_XE_GEM_CPU_CACHING_WC,
XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
XE_BO_FLAG_FORCE_WC |
XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL);
}Do these also need XE_BO_FLAG_ACCOUNTED to prevent a malicious or buggy userspace application from bypassing memcg limits to exhaust host memory?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Christian König
maintainerQuoted 14 lines · Dave Airlie
I committed the vmstat counters and list lru changes, and they are now in tree.
This is the remainder of this series. Intel have expressed interest in getting this landed for xe, we can drop the amdgpu changes for now if they can't get across the line.
I've dropped all previous acks/reviews.
This series adds the memcg counters for GPU active and GPU reclaim to align with the two global vmstats. It adds an accounting flag to TTM alloc/populate, and enables memcg tracking and shrinker support in TTM.
Then it adds amdgpu and xe support.
Quoted 8 lines
I think for this to land, Christian holds the main objection which I still fail to fully understand beyond it doesn't solve all the problems we ever have had with cgroups and drm, so we shouldn't even bother, and maybe we could do it at the object level, and integrated with dmem, and android cross process accounting, but I still feel this is a good baseline.
I think this is the right layer to hook this into TTM, where we allocate memory and I think accounting for this memory in a proper way should be done.
I think we pretty much agreed now to do this through dmem?
At least dmem got the ability to optionally account allocations towards memcg as well and as far as I see there is no real benefit any more to mess with this in the ttm_pool.
In general the pool seems to be the completely wrong layer to handle that for TTM. Background is that you can't reclaim individual pages anyway, you always need to reclaim full buffers and that is only possible on higher layers.
Regards,
Christian.
Quoted 6 lines
Intel folks (Thomas/Maarten) please review and express concerns as well.
Regards,
Dave.