mm/slab, alloc_tag: reduce obj_ext memory waste
Subsystems (8)
Vlastimil Babka (SUSE)
maintainerv3 cover mm/slab, alloc_tag: reduce obj_ext memory waste
The recent fixes for objext array handling inspired me to look into this finally. It's been bothering me that the memory usage of struct slabobj_ext depend only on config options and not whether the fields are actually used. So with both CONFIG_MEMCG=y and CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref field. And thus:
1) Having memory allocation profiling config-enabled but not boot-enabled means wasted memory on unused codetag_refs. This makes it less suitable for a general distro config and the page allocator side doesn't suffer from this, only slab and percpu.
2) Complementary, with memory allocation profiling enabled, there are caches/slabs that don't need the objcg field, so memory is wasted on those.
This series should solve the point 1) fully for slab, pcpuobj_ext handling can be perhaps improved similarly, haven't looked into that.
For 2) it avoids allocating objcg fields for KMALLOC_NORMAL and KMALLOC_NO_OBJ_EXT caches where we know they are not necessary because kmalloc() with __GFP_ACCOUNT will pick a KMALLOC_CGROUP type (except with SLUB_TINY).
The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT and then we know objcg fields are always needed. But also they can be created without SLAB_ACCOUNT and then some allocations have __GFP_ACCOUNT and some not and we don't know that in advance.
This series introduces a SLAB_MAY_ACCOUNT flag that's currently internal only and is applied to all caches (unless kmem accounting is disabled) except KMALLOC_NORMAL (unless that aliases KMALLOC_RECLAIM) and KMALLOC_NO_OBJ_EXT.
As a followup we can make SLAB_MAY_ACCOUNT explicit and add it to to caches where we know __GFP_ACCOUNT is used. Then we could only honour __GFP_ACCOUNT for those, while warning for an unexpected usage elsewhere.
Based on slab/for-next-fixes
Git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/objext_split
Moderately tested, nothing seems to crash.
To check for regressions, I forward-ported a microbenchmark hacked into slub_kunit that was used to evaluate sheaves. You can find it here along with the series:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=objext_split-v3-bench
Tried 3 scenarios, MEMCG and KFENCE were always enabled:
- CONFIG_MEM_ALLOC_PROFILING=n
- CONFIG_MEM_ALLOC_PROFILING=y but _ENABLED_BY_DEFAULT=n
- same but booted with sysctl.vm.mem_profiling=1
The results are quite noisy, but no regression was apparent, except perhaps few percents for the last case. I don't expect it will be visible in any real workloads.
---
Changes in v3:
- Review tags from Harry, Hao, Suren - thanks!
- Patch "skip handle_failed_objexts_alloc() with profiling disabled"
removed, the check is done as part of "reduce slabobj_ext memory with allocation profiling disabled" using slab_obj_ext_has_codetag()
(Harry).
- Move kfence checks earlier in mark_obj_codetag_empty() in Patch 1
(Harry).
- Remove SLAB_MAY_ACCOUNT also for KMALLOC_NO_OBJ_EXT caches and more
precise description in Patch 11 (Hao)
- Add a new cleanup patch 13 that removes also memcg accounting from
kfence allocations, as suggested by Harry.
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Apply Suren's R-b:, thanks!
- Expanded explanation about no longer accounting KFENCE objects (Suren)
- Also skip them in in mark_obj_codetag_empty() (sashiko)
- Update slab_obj_ext() comments (sashiko)
- Separate slab_obj_ext_objcg() and slab_obj_ext_set_objcg() (Suren)
- Add SLAB_MAY_ACCOUNT internal flag instead of relying on is_kmalloc_normal(); also handle mem_cgroup_kmem_disabled()
- Also fix the SLUB_TINY kmalloc aliasing handling, per Harry.
- Link to v1: https://patch.msgid.link/[email protected]
---
Vlastimil Babka (SUSE) (13):
mm/slab: skip kfence objects in allocation profiling mm/slab: remove objs_per_slab()
mm: move struct slabobj_ext to mm/slab.h
mm/slab: make slab_obj_ext() determine object index mm/slab: abstract slabobj_ext.objcg access
mm/slab: abstract slabobj_ext.ref access
mm/slab: replace slab.stride with obj_exts_in_object mm/slab: change struct slabobj_ext to a union
mm/slab: introduce slab_obj_ext_has_codetag()
mm/slab: reduce slabobj_ext memory with allocation profiling disabled mm/slab: add cache_ and slab_needs_objcg() helpers mm/slab: stop allocating objcg pointers when unnecessary mm/slab, kfence, memcg: completely remove obj_ext for kfence objects
Documentation/mm/allocation-profiling.rst | 7 ++ include/linux/memcontrol.h | 13 -- include/linux/slab.h | 3 + mm/kfence/core.c | 12 -- mm/kfence/kfence.h | 3 - mm/kfence/kfence_test.c | 2 +- mm/memcontrol.c | 40 ++++--- mm/slab.h | 189 ++++++++++++++++++++++++------ mm/slab_common.c | 26 +++- mm/slub.c | 184 ++++++++++++++++++----------- 10 files changed, 325 insertions(+), 154 deletions(-)
---
base-commit: 5ff172f6c94d282d83cb88bdfec5f647ad9c6105
change-id: 20260714-b4-objext_split-da82426257d5
Vlastimil Babka (SUSE)
maintainerPATCH v3 1/13 mm/slab: skip kfence objects in allocation profiling
struct kfence_metadata only contains struct slabobj_ext with CONFIG_MEMCG, which is then used for the "fake" slab's obj_exts field. If CONFIG_MEMCG is enabled, the struct can also end up used for memory allocation profiling. If CONFIG_MEMCG is disabled but profiling is enabled, it will end up allocating its obj_exts via prepare_slab_obj_exts_hook() and assigning them to the fake struct slab. These will probably then never be freed.
So things sorta work, but not always in the intended and optimal way. The upcoming changes to slabobj_ext layout would additionally need a proper refactoring to keep working.
However, there's little benefit in accounting KFENCE objects. KFENCE allocations are rare and there can be only CONFIG_KFENCE_NUM_OBJECTS (default to 255) outstanding ones at any time. For any callsite prominent enough in the memory allocation profiling stats, allocations served from KFENCE will be lost in the noise.
Thus let's not complicate things and simply stop accounting KFENCE objects in allocation profiling and skip them in the related slab hooks.
We also need to skip kfence objects in mark_obj_codetag_empty() in case a sheaf is allocated from kfence, per earlier sashiko review.
--- Documentation/mm/allocation-profiling.rst | 7 +++++++ mm/slub.c | 9 +++++++++ 2 files changed, 16 insertions(+)
Documentation/mm/allocation-profiling.rst +7 -0
@@ -112,3 +112,10 @@ break it out by rhashtable type. - Then, use the following form for your allocations: alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) + +Notes +===== + +- When a slab object is allocated from KFENCE, its accounting is skipped. + KFENCE allocations are rare and limited to a small number, so this omission + is negligible.
mm/slub.c +9 -0
@@ -2067,6 +2067,9 @@ static inline void mark_obj_codetag_empty(const void *obj)struct slab *obj_slab;unsigned long slab_exts; +if (is_kfence_address (obj)) +return ; + obj_slab =virt_to_slab (obj); slab_exts =slab_obj_exts (obj_slab);if (slab_exts) {
@@ -2352,6 +2355,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,if (alloc_flags &SLAB_ALLOC_NO_RECURSE )return ; +if (is_kfence_address (object)) +return ; + slab =virt_to_slab (object); obj_exts =prepare_slab_obj_exts_hook (s, slab, flags, alloc_flags, object);/*
@@ -2399,6 +2405,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **pfor (i =0 ; i < objects; i++) {unsigned int off =obj_to_index (s, slab, p[i]); +if (is_kfence_address (p[i])) +continue ; +alloc_tag_sub (&slab_obj_ext (slab, obj_exts, off)->ref, s->size); }put_slab_obj_exts (obj_exts);
Signature · 1 line
2.55.0
Re: [PATCH v3 01/13] mm/slab: skip kfence objects in allocation profiling
Quoted 35 lines · Vlastimil Babka (SUSE)
struct
kfence_metadataonly contains structslabobj_extwithCONFIG_MEMCG, which is then used for the "fake" slab'sobj_extsfield. IfCONFIG_MEMCGis enabled, the struct can also end up used for memory allocation profiling. IfCONFIG_MEMCGis disabled but profiling is enabled, it will end up allocating itsobj_extsviaprepare_slab_obj_exts_hook()and assigning them to the fake struct slab. These will probably then never be freed.So things sorta work, but not always in the intended and optimal way. The upcoming changes to
slabobj_extlayout would additionally need a proper refactoring to keep working.However, there's little benefit in accounting KFENCE objects. KFENCE allocations are rare and there can be only
CONFIG_KFENCE_NUM_OBJECTS(default to 255) outstanding ones at any time. For any callsite prominent enough in the memory allocation profiling stats, allocations served from KFENCE will be lost in the noise.Thus let's not complicate things and simply stop accounting KFENCE objects in allocation profiling and skip them in the related slab hooks.
We also need to skip kfence objects in
mark_obj_codetag_empty()in case a sheaf is allocated from kfence, per earlier sashiko review.Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- Documentation/mm/allocation-profiling.rst | 7 +++++++ mm/slub.c | 9 +++++++++ 2 files changed, 16 insertions(+)Documentation/mm/allocation-profiling.rst +0
-0
Documentation/mm/allocation-profiling.rst break it out by rhashtable type. · line 119
Quoted 8 lines
Documentation/mm/allocation-profiling.rst +5
-0@@ -112,3 +112,10 @@ break it out by rhashtable type. - Then, use the following form for your allocations: alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) + +Notes +===== + +- When a slab object is allocated from KFENCE, its accounting is skipped.
should this be 'allocation profiling is skipped' instead?
Otherwise, looks good to me.
Quoted 40 lines
Documentation/mm/allocation-profiling.rst +2
-0+ KFENCE allocations are rare and limited to a small number, so this omission + is negligible.mm/slub.c +9
-0@@ -2067,6 +2067,9 @@ static inline void mark_obj_codetag_empty(const void *obj)struct slab *obj_slab;unsigned long slab_exts; +if (is_kfence_address (obj)) +return ; + obj_slab =virt_to_slab (obj); slab_exts =slab_obj_exts (obj_slab);if (slab_exts) {@@ -2352,6 +2355,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,if (alloc_flags &SLAB_ALLOC_NO_RECURSE )return ; +if (is_kfence_address (object)) +return ; + slab =virt_to_slab (object); obj_exts =prepare_slab_obj_exts_hook (s, slab, flags, alloc_flags, object);/* @@ -2399,6 +2405,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **pfor (i =0 ; i < objects; i++) {unsigned int off =obj_to_index (s, slab, p[i]); +if (is_kfence_address (p[i])) +continue ; +alloc_tag_sub (&slab_obj_ext (slab, obj_exts, off)->ref, s->size); }put_slab_obj_exts (obj_exts);Signature · 2 lines
-- 2.55.0
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 01/13] mm/slab: skip kfence objects in allocation profiling
Quoted 37 lines · Hao Li
Quoted 36 lines · Vlastimil Babka (SUSE)
struct
kfence_metadataonly contains structslabobj_extwithCONFIG_MEMCG, which is then used for the "fake" slab'sobj_extsfield. IfCONFIG_MEMCGis enabled, the struct can also end up used for memory allocation profiling. IfCONFIG_MEMCGis disabled but profiling is enabled, it will end up allocating itsobj_extsviaprepare_slab_obj_exts_hook()and assigning them to the fake struct slab. These will probably then never be freed.So things sorta work, but not always in the intended and optimal way. The upcoming changes to
slabobj_extlayout would additionally need a proper refactoring to keep working.However, there's little benefit in accounting KFENCE objects. KFENCE allocations are rare and there can be only
CONFIG_KFENCE_NUM_OBJECTS(default to 255) outstanding ones at any time. For any callsite prominent enough in the memory allocation profiling stats, allocations served from KFENCE will be lost in the noise.Thus let's not complicate things and simply stop accounting KFENCE objects in allocation profiling and skip them in the related slab hooks.
We also need to skip kfence objects in
mark_obj_codetag_empty()in case a sheaf is allocated from kfence, per earlier sashiko review.Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- Documentation/mm/allocation-profiling.rst | 7 +++++++ mm/slub.c | 9 +++++++++ 2 files changed, 16 insertions(+)Documentation/mm/allocation-profiling.rst +0
-0@@ -112,3 +112,10 @@ break it out by rhashtable type.
Quoted 9 lines
Documentation/mm/allocation-profiling.rst break it out by rhashtable type. · line 119
Quoted 7 lines
Documentation/mm/allocation-profiling.rst +5
-0- Then, use the following form for your allocations: alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) + +Notes +===== + +- When a slab object is allocated from KFENCE, its accounting is skipped.should this be 'allocation profiling is skipped' instead?
The document uses the word "accounting" when it comes to individual allocations, so this fits.
Quoted 2 lines
Otherwise, looks good to me.
Reviewed-by Hao Li <[email protected]>
Thanks!
Vlastimil Babka (SUSE)
maintainerPATCH v3 2/13 mm/slab: remove objs_per_slab()
In reply to Vlastimil Babka (SUSE)The function has an unused kmem_cache argument and almost nothing uses it anyway; doing slab->objects is simpler. Remove it with the last two users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: ‘typeof’ applied to a bit-field" but we don't need to keep a wrapper just for that.
--- mm/kfence/kfence_test.c | 2 +- mm/slab.h | 6 ------ mm/slub.c | 3 +-- 3 files changed, 2 insertions(+), 9 deletions(-)
mm/kfence/kfence_test.c +1 -1
@@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat* memcg accounting works correctly. */ KUNIT_EXPECT_EQ (test,obj_to_index (s, slab, alloc),0U ); -KUNIT_EXPECT_EQ (test,objs_per_slab (s, slab),1 ); +KUNIT_EXPECT_EQ (test, ((unsigned int )slab->objects),1 );if (policy ==ALLOCATE_ANY )return alloc;
mm/slab.h +0 -6
@@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,return __obj_to_index (cache,slab_address (slab), obj); } -static inline int objs_per_slab (const struct kmem_cache *cache, -const struct slab *slab) -{ -return slab->objects; -} -/* * State of the slab allocator. *
mm/slub.c +1 -2
@@ -2130,7 +2130,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags) {const bool allow_spin =alloc_flags_allow_spinning (alloc_flags); -unsigned int objects =objs_per_slab (s, slab);bool new_slab = alloc_flags &SLAB_ALLOC_NEW_SLAB ;unsigned long new_exts;unsigned long old_exts;
@@ -2186,7 +2185,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,#endif retry: old_exts =READ_ONCE (slab->obj_exts); -handle_failed_objexts_alloc (old_exts, vec, objects); +handle_failed_objexts_alloc (old_exts, vec, slab->objects);if (new_slab) {/*
Signature · 1 line
2.55.0
Hao Li
reviewerRe: [PATCH v3 02/13] mm/slab: remove objs_per_slab()
Quoted 9 lines · Vlastimil Babka (SUSE)
The function has an unused
kmem_cacheargument and almost nothing uses it anyway; doingslab->objectsis simpler. Remove it with the last two users.KUNIT_EXPECT_EQ()needs a cast to avoid "error: ‘typeof’ applied to a bit-field" but we don't need to keep a wrapper just for that.Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Harry Yoo (Oracle) <[email protected]>Reviewed-by Hao Li <[email protected]>Reviewed-by Suren Baghdasayan <[email protected]>
Oops, the r-b tag got duplicated :)
Quoted 62 lines
Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/kfence/kfence_test.c | 2 +- mm/slab.h | 6 ------ mm/slub.c | 3 +-- 3 files changed, 2 insertions(+), 9 deletions(-)mm/kfence/kfence_test.c +1
-1@@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat* memcg accounting works correctly. */ KUNIT_EXPECT_EQ (test,obj_to_index (s, slab, alloc),0U ); -KUNIT_EXPECT_EQ (test,objs_per_slab (s, slab),1 ); +KUNIT_EXPECT_EQ (test, ((unsigned int )slab->objects),1 );if (policy ==ALLOCATE_ANY )return alloc;mm/slab.h +0
-6@@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,return __obj_to_index (cache,slab_address (slab), obj); } -static inline int objs_per_slab (const struct kmem_cache *cache, -const struct slab *slab) -{ -return slab->objects; -} -/* * State of the slab allocator. * mm/slub.c +1
-2@@ -2130,7 +2130,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags) {const bool allow_spin =alloc_flags_allow_spinning (alloc_flags); -unsigned int objects =objs_per_slab (s, slab);bool new_slab = alloc_flags &SLAB_ALLOC_NEW_SLAB ;unsigned long new_exts;unsigned long old_exts;@@ -2186,7 +2185,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,#endif retry: old_exts =READ_ONCE (slab->obj_exts); -handle_failed_objexts_alloc (old_exts, vec, objects); +handle_failed_objexts_alloc (old_exts, vec, slab->objects);if (new_slab) {/* Signature · 2 lines
-- 2.55.0
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerPATCH v3 3/13 mm: move struct slabobj_ext to mm/slab.h
In reply to Vlastimil Babka (SUSE)Users of include/linux/memcontrol.h don't need to see this internal structure. Further changes to the struct will reduce recompiling.
--- include/linux/memcontrol.h | 13 ------------- mm/slab.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 13 deletions(-)
include/linux/memcontrol.h +0 -13
@@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) { }static inline int mem_cgroup_init (void ) {return 0 ; }#endif /* CONFIG_MEMCG */ -/* -* Extended information for slab objects stored as an array in page->memcg_data -* if MEMCG_DATA_OBJEXTS is set. -*/ -struct slabobj_ext { -#ifdef CONFIG_MEMCG -struct obj_cgroup *objcg; -#endif -#ifdef CONFIG_MEM_ALLOC_PROFILING -union codetag_ref ref; -#endif -}__aligned(8) ; -static inline struct lruvec *parent_lruvec (struct lruvec *lruvec) {struct mem_cgroup *memcg;
mm/slab.h +13 -0
@@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void)return false; } +/* +* Extended information for slab objects stored as an array in page->memcg_data +* if MEMCG_DATA_OBJEXTS is set. +*/ +struct slabobj_ext { +#ifdef CONFIG_MEMCG +struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +union codetag_ref ref; +#endif +}__aligned(8) ; +#ifdef CONFIG_SLAB_OBJ_EXT /*
Signature · 1 line
2.55.0
Vlastimil Babka (SUSE)
maintainerPATCH v3 4/13 mm/slab: make slab_obj_ext() determine object index
In reply to Vlastimil Babka (SUSE)All callers perform the same obj_to_index() calculation to pass the index. Simplify by passing object pointer instead and determining the index by slab_obj_ext().
--- mm/memcontrol.c | 12 +++--------- mm/slab.h | 19 +++++++++++-------- mm/slub.c | 22 +++++++--------------- 3 files changed, 21 insertions(+), 32 deletions(-)
mm/memcontrol.c +3 -9
@@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)*/ unsigned long obj_exts;struct slabobj_ext *obj_ext; -unsigned int off; obj_exts =slab_obj_exts (slab);if (!obj_exts)return NULL ;get_slab_obj_exts (obj_exts); - off =obj_to_index (slab->slab_cache, slab, p); - obj_ext =slab_obj_ext (slab, obj_exts, off); + obj_ext =slab_obj_ext (slab->slab_cache, slab, obj_exts, p);if (obj_ext->objcg) {struct obj_cgroup *objcg = obj_ext->objcg;
@@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,size_t obj_size =obj_full_size (s);struct obj_cgroup *objcg;struct slab *slab; -unsigned long off;size_t i;/*
@@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, obj_exts =slab_obj_exts (slab);get_slab_obj_exts (obj_exts); - off =obj_to_index (s, slab, p[i]); - obj_ext =slab_obj_ext (slab, obj_exts, off); + obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]);obj_cgroup_get (objcg); obj_ext->objcg = objcg;put_slab_obj_exts (obj_exts);
@@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,struct obj_cgroup *objcg;struct slabobj_ext *obj_ext;struct obj_stock_pcp *stock; -unsigned int off; - off =obj_to_index (s, slab, p[i]); - obj_ext =slab_obj_ext (slab, obj_exts, off); + obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]); objcg = obj_ext->objcg;if (!objcg)continue ;
mm/slab.h +11 -8
@@ -579,7 +579,7 @@ struct slabobj_ext {* obj_exts = slab_obj_exts(slab); * if (obj_exts) { * get_slab_obj_exts(obj_exts); -* obj_ext = slab_obj_ext(slab, obj_exts, obj_to_index(s, slab, obj)); +* obj_ext = slab_obj_ext(s, slab, obj_exts, obj); * // do something with obj_ext * put_slab_obj_exts(obj_exts); * }
@@ -639,21 +639,24 @@ static inline unsigned int slab_get_stride(struct slab *slab)/* * slab_obj_ext - get the pointer to the slab object extension metadata * associated with an object in a slab. +* @s: cache that the slab belongs to * @slab: a pointer to the slab struct * @obj_exts: a pointer to the object extension vector -* @index: an index of the object +* @obj: a pointer to the object * * Returns a pointer to the object extension associated with the object. * Must be called within a section covered by get/put_slab_obj_exts(). */ -static inline struct slabobj_ext *slab_obj_ext (struct slab *slab, -unsigned long obj_exts, -unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext (struct kmem_cache *s,struct slab *slab,unsigned long obj_exts, +const void *obj) {struct slabobj_ext *obj_ext; +unsigned int index;VM_WARN_ON_ONCE (obj_exts !=slab_obj_exts (slab)); + index =obj_to_index (s, slab, obj); obj_ext = (struct slabobj_ext *)(obj_exts +slab_get_stride (slab) * index);return kasan_reset_tag (obj_ext);
@@ -669,9 +672,9 @@ static inline unsigned long slab_obj_exts(struct slab *slab)return 0 ; } -static inline struct slabobj_ext *slab_obj_ext (struct slab *slab, -unsigned long obj_exts, -unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext (struct kmem_cache *s,struct slab *slab,unsigned long obj_exts, +const void *obj) {return NULL ; }
mm/slub.c +7 -15
@@ -2073,11 +2073,10 @@ static inline void mark_obj_codetag_empty(const void *obj) obj_slab =virt_to_slab (obj); slab_exts =slab_obj_exts (obj_slab);if (slab_exts) { +struct slabobj_ext *ext; +get_slab_obj_exts (slab_exts); -unsigned int offs =obj_to_index (obj_slab->slab_cache, - obj_slab, obj); -struct slabobj_ext *ext =slab_obj_ext (obj_slab, - slab_exts, offs); + ext =slab_obj_ext (obj_slab->slab_cache, obj_slab, slab_exts, obj);if (unlikely (is_codetag_empty (&ext->ref))) {put_slab_obj_exts (slab_exts);
@@ -2365,10 +2364,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,* check should be added before alloc_tag_add(). */ if (obj_exts) { -unsigned int obj_idx =obj_to_index (s, slab, object); -get_slab_obj_exts (obj_exts); - obj_ext =slab_obj_ext (slab, obj_exts, obj_idx); + obj_ext =slab_obj_ext (s, slab, obj_exts, object);alloc_tag_add (&obj_ext->ref, current->alloc_tag, s->size);put_slab_obj_exts (obj_exts); }else {
@@ -2389,7 +2386,6 @@ static noinline void __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - int i; unsigned long obj_exts;/* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */
@@ -2401,13 +2397,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **preturn ;get_slab_obj_exts (obj_exts); -for (i =0 ; i < objects; i++) { -unsigned int off =obj_to_index (s, slab, p[i]); - +for (int i =0 ; i < objects; i++) {if (is_kfence_address (p[i]))continue ; -alloc_tag_sub (&slab_obj_ext (slab, obj_exts, off)->ref, s->size); +alloc_tag_sub (&slab_obj_ext (s, slab, obj_exts, p[i])->ref, s->size); }put_slab_obj_exts (obj_exts); }
@@ -2492,7 +2486,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)struct kmem_cache *s;struct page *page;struct slab *slab; -unsigned long off; page =virt_to_page (p);if (PageLargeKmalloc (page)) {
@@ -2532,8 +2525,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) obj_exts =slab_obj_exts (slab);if (obj_exts) {get_slab_obj_exts (obj_exts); - off =obj_to_index (s, slab, p); - obj_ext =slab_obj_ext (slab, obj_exts, off); + obj_ext =slab_obj_ext (s, slab, obj_exts, p);if (unlikely (obj_ext->objcg)) {put_slab_obj_exts (obj_exts);return true;
Signature · 1 line
2.55.0
Vlastimil Babka (SUSE)
maintainerPATCH v3 5/13 mm/slab: abstract slabobj_ext.objcg access
In reply to Vlastimil Babka (SUSE)In preparation for changes to the structure, abstract getting and setting the objcg field with slab_obj_ext_objcg() and slab_obj_ext_set_objcg().
Rename the field to _objcg to make an unexpected direct access a compile error.
No functional change intended.
--- mm/kfence/core.c | 2 +- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 15 ++++++++++++++- mm/slub.c | 2 +- 4 files changed, 24 insertions(+), 9 deletions(-)
mm/kfence/core.c +1 -1
@@ -1249,7 +1249,7 @@ void __kfence_free(void *addr)struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr);#ifdef CONFIG_MEMCG -KFENCE_WARN_ON (meta->obj_exts.objcg); +KFENCE_WARN_ON (slab_obj_ext_objcg (&meta->obj_exts));#endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing
mm/memcontrol.c +8 -6
@@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)*/ unsigned long obj_exts;struct slabobj_ext *obj_ext; +struct obj_cgroup *objcg; obj_exts =slab_obj_exts (slab);if (!obj_exts)
@@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (slab->slab_cache, slab, obj_exts, p); -if (obj_ext->objcg) { -struct obj_cgroup *objcg = obj_ext->objcg; - + objcg =slab_obj_ext_objcg (obj_ext); +if (objcg) {put_slab_obj_exts (obj_exts);return obj_cgroup_memcg (objcg); }
@@ -3614,8 +3614,10 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, obj_exts =slab_obj_exts (slab);get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]); +obj_cgroup_get (objcg); - obj_ext->objcg = objcg; +slab_obj_ext_set_objcg (obj_ext, objcg); +put_slab_obj_exts (obj_exts); }
@@ -3633,11 +3635,11 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,struct obj_stock_pcp *stock; obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]); - objcg = obj_ext->objcg; + objcg =slab_obj_ext_objcg (obj_ext);if (!objcg)continue ; - obj_ext->objcg =NULL ; +slab_obj_ext_set_objcg (obj_ext,NULL ); stock =trylock_stock ();__refill_obj_stock (objcg, stock, obj_size, true);
mm/slab.h +14 -1
@@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)*/ struct slabobj_ext {#ifdef CONFIG_MEMCG -struct obj_cgroup *objcg; +struct obj_cgroup *_objcg;#endif #ifdef CONFIG_MEM_ALLOC_PROFILING union codetag_ref ref;
@@ -662,6 +662,19 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,return kasan_reset_tag (obj_ext); } +#ifdef CONFIG_MEMCG +static inline struct obj_cgroup *slab_obj_ext_objcg (struct slabobj_ext *obj_ext) +{ +return obj_ext->_objcg; +} + +static inline void slab_obj_ext_set_objcg (struct slabobj_ext *obj_ext, +struct obj_cgroup *objcg) +{ + obj_ext->_objcg = objcg; +} +#endif +int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,unsigned int alloc_flags);
mm/slub.c +1 -1
@@ -2526,7 +2526,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)if (obj_exts) {get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (s, slab, obj_exts, p); -if (unlikely (obj_ext->objcg)) { +if (unlikely (slab_obj_ext_objcg (obj_ext))) {put_slab_obj_exts (obj_exts);return true; }
Signature · 1 line
2.55.0
Re: [PATCH v3 05/13] mm/slab: abstract slabobj_ext.objcg access
Quoted 6 lines · Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract getting and setting the objcg field with
slab_obj_ext_objcg()andslab_obj_ext_set_objcg().
Rename the field to_objcgto make an unexpected direct access a compile error.
Quoted 6 lines
No functional change intended.
Reviewed-by Hao Li <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Vlastimil Babka (SUSE)
maintainerPATCH v3 6/13 mm/slab: abstract slabobj_ext.ref access
In reply to Vlastimil Babka (SUSE)In preparation for changes to the structure, abstract access to the ref field with a slab_obj_ext_codetag_ref() function. Rename the field to _ctref to make an unexpected direct access a compile error.
No functional change intended.
--- mm/slab.h | 10 +++++++++- mm/slub.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 15 deletions(-)
mm/slab.h +9 -1
@@ -558,7 +558,7 @@ struct slabobj_ext {struct obj_cgroup *_objcg;#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -union codetag_ref ref; +union codetag_ref _ctref;#endif }__aligned(8) ;
@@ -675,6 +675,14 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext, }#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline union codetag_ref * +slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) +{ +return &obj_ext->_ctref; +} +#endif +int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,unsigned int alloc_flags);
mm/slub.c +28 -14
99 lines not on this page · raw message
Signature · 1 line
2.55.0
Re: [PATCH v3 06/13] mm/slab: abstract slabobj_ext.ref access
Quoted 10 lines · Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract access to the ref field with a
slab_obj_ext_codetag_ref()function. Rename the field to_ctrefto make an unexpected direct access a compile error.No functional change intended.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
With a suggestion: it would be nice to have debug warnings that trigger when slab_obj_ext_codetag_ref() is compiled but not supposed to be called.
e.g.) for kfence objects, or later in the series when we reduce slabobj_ext memory (!slab_obj_ext_has_codetag())
Similarly, it would be nice (later in the series) to have a warning in slab_obj_ext_objcg() when slab_needs_objcg() returns false.
Quoted 3 lines
mm/slab.h | 10 +++++++++- mm/slub.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 15 deletions(-)
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 06/13] mm/slab: abstract slabobj_ext.ref access
Quoted 19 lines · Harry Yoo
Quoted 10 lines · Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract access to the ref field with a
slab_obj_ext_codetag_ref()function. Rename the field to_ctrefto make an unexpected direct access a compile error.No functional change intended.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Reviewed-by Harry Yoo (Oracle) <[email protected]>With a suggestion: it would be nice to have debug warnings that trigger when
slab_obj_ext_codetag_ref()is compiled but not supposed to be called.
Quoted 5 lines
e.g.) for kfence objects, or later in the series when we reduce
slabobj_extmemory (!slab_obj_ext_has_codetag())Similarly, it would be nice (later in the series) to have a warning in
slab_obj_ext_objcg()whenslab_needs_objcg()returns false.
OK good idea but we can't do it right now as the helpers are introduced later.
So for 9/13 mm/slab: introduce slab_obj_ext_has_codetag():
mm/slab.h +2 -0
@@ -726,6 +726,8 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { +VM_WARN_ON_ONCE (!slab_obj_ext_has_codetag ()); +if (IS_ENABLED (CONFIG_MEMCG )) obj_ext +=1 ;
... and for objcg we would need to start passing slab pointer to slab_obj_ext_objcg() and slab_obj_ext_set_objcg(). Hmm...
Quoted 3 lines
Quoted 3 lines
mm/slab.h | 10 +++++++++- mm/slub.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 15 deletions(-)
Harry Yoo
maintainerRe: [PATCH v3 06/13] mm/slab: abstract slabobj_ext.ref access
Quoted 44 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · Harry Yoo
Quoted 10 lines · Vlastimil Babka (SUSE)
In preparation for changes to the structure, abstract access to the ref field with a
slab_obj_ext_codetag_ref()function. Rename the field to_ctrefto make an unexpected direct access a compile error.No functional change intended.
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Reviewed-by Harry Yoo (Oracle) <[email protected]>With a suggestion: it would be nice to have debug warnings that trigger when
slab_obj_ext_codetag_ref()is compiled but not supposed to be called.Quoted 5 lines
e.g.) for kfence objects, or later in the series when we reduce
slabobj_extmemory (!slab_obj_ext_has_codetag())Similarly, it would be nice (later in the series) to have a warning in
slab_obj_ext_objcg()whenslab_needs_objcg()returns false.OK good idea but we can't do it right now as the helpers are introduced later.
So for 9/13 mm/slab: introduce
slab_obj_ext_has_codetag():mm/slab.h +2
-0@@ -726,6 +726,8 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { +VM_WARN_ON_ONCE (!slab_obj_ext_has_codetag ()); +if (IS_ENABLED (CONFIG_MEMCG )) obj_ext +=1 ;
Quoted 2 lines
... and for objcg we would need to start passing slab pointer to
slab_obj_ext_objcg()andslab_obj_ext_set_objcg(). Hmm...
But slab_obj_ext_codetag_ref() already takes the parameter to calculate offset. I think it's worth as it'll be optimized away on non-debug kerenels and there is no way to detect type confusion due to subtle errors in the objext layout.
Vlastimil Babka (SUSE)
maintainer Reviewed-by Hao Li Reviewed-by Suren Baghdasaryan Reviewed-by Harry Yoo (Oracle)Re: [PATCH v3 06/13] mm/slab: abstract slabobj_ext.ref access
Quoted 13 lines · Harry Yoo
Quoted 13 lines
mm/slab.h +2
-0@@ -726,6 +726,8 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { +VM_WARN_ON_ONCE (!slab_obj_ext_has_codetag ()); +if (IS_ENABLED (CONFIG_MEMCG )) obj_ext +=1 ;
Quoted 7 lines
Quoted 2 lines
... and for objcg we would need to start passing slab pointer to
slab_obj_ext_objcg()andslab_obj_ext_set_objcg(). Hmm...But
slab_obj_ext_codetag_ref()already takes the parameter to calculate offset. I think it's worth as it'll be optimized away on non-debug kerenels and there is no way to detect type confusion due to subtle errors in the objext layout.
OK, so updated to this. VM_WARN_ON_ONCE() added later.
From db9c3501c0f9747fda8fa92d4f640b6fac8050e0 Mon Sep 17 00:00:00 2001
From: "Vlastimil Babka (SUSE)" <[email protected]>
Date: Mon, 27 Jul 2026 14:53:59 +0200
Subject: [PATCH] mm/slab: abstract slabobj_ext.objcg access
In preparation for changes to the structure, abstract getting and setting the objcg field with slab_obj_ext_objcg() and slab_obj_ext_set_objcg().
Rename the field to _objcg to make an unexpected direct access a compile error.
The helpers take a slab pointer, which is currently unused, but will be used by a debug check later.
Since there is no slab pointer easily available in __kfence_free(), just drop the debug check there. The whole memcg_kmem accounting in kfence is to be removed later anyway.
Otherwise, no functional change intended.
--- mm/kfence/core.c | 3 --- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 17 ++++++++++++++++- mm/slub.c | 2 +- 4 files changed, 25 insertions(+), 11 deletions(-)
mm/kfence/core.c +0 -3
@@ -1248,9 +1248,6 @@ void __kfence_free(void *addr) {struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); -#ifdef CONFIG_MEMCG -KFENCE_WARN_ON (meta->obj_exts.objcg); -#endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing * the object, as the object page may be recycled for other-typed
mm/memcontrol.c +8 -6
@@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)*/ unsigned long obj_exts;struct slabobj_ext *obj_ext; +struct obj_cgroup *objcg; obj_exts =slab_obj_exts (slab);if (!obj_exts)
@@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (slab->slab_cache, slab, obj_exts, p); -if (obj_ext->objcg) { -struct obj_cgroup *objcg = obj_ext->objcg; - + objcg =slab_obj_ext_objcg (slab, obj_ext); +if (objcg) {put_slab_obj_exts (obj_exts);return obj_cgroup_memcg (objcg); }
@@ -3614,8 +3614,10 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, obj_exts =slab_obj_exts (slab);get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]); +obj_cgroup_get (objcg); - obj_ext->objcg = objcg; +slab_obj_ext_set_objcg (slab, obj_ext, objcg); +put_slab_obj_exts (obj_exts); }
@@ -3633,11 +3635,11 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,struct obj_stock_pcp *stock; obj_ext =slab_obj_ext (s, slab, obj_exts, p[i]); - objcg = obj_ext->objcg; + objcg =slab_obj_ext_objcg (slab, obj_ext);if (!objcg)continue ; - obj_ext->objcg =NULL ; +slab_obj_ext_set_objcg (slab, obj_ext,NULL ); stock =trylock_stock ();__refill_obj_stock (objcg, stock, obj_size, true);
mm/slab.h +16 -1
@@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void)*/ struct slabobj_ext {#ifdef CONFIG_MEMCG -struct obj_cgroup *objcg; +struct obj_cgroup *_objcg;#endif #ifdef CONFIG_MEM_ALLOC_PROFILING union codetag_ref ref;
@@ -662,6 +662,21 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,return kasan_reset_tag (obj_ext); } +#ifdef CONFIG_MEMCG +static inline struct obj_cgroup * +slab_obj_ext_objcg (struct slab *slab,struct slabobj_ext *obj_ext) +{ +return obj_ext->_objcg; +} + +static inline void +slab_obj_ext_set_objcg (struct slab *slab,struct slabobj_ext *obj_ext, +struct obj_cgroup *objcg) +{ + obj_ext->_objcg = objcg; +} +#endif +int alloc_slab_obj_exts (struct slab *slab,struct kmem_cache *s,gfp_t gfp,unsigned int alloc_flags);
mm/slub.c +1 -1
@@ -2526,7 +2526,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)if (obj_exts) {get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (s, slab, obj_exts, p); -if (unlikely (obj_ext->objcg)) { +if (unlikely (slab_obj_ext_objcg (slab, obj_ext))) {put_slab_obj_exts (obj_exts);return true; }
Signature · 1 line
2.55.0
Harry Yoo
maintainerRe: [PATCH v3 06/13] mm/slab: abstract slabobj_ext.ref access
Quoted 20 lines · Vlastimil Babka (SUSE)
OK, so updated to this.
VM_WARN_ON_ONCE()added later.From
db9c3501c0f9747fda8fa92d4f640b6fac8050e0Mon Sep 17 00:00:00 2001
From: "Vlastimil Babka (SUSE)" <[email protected]>
Date: Mon, 27 Jul 2026 14:53:59 +0200
Subject: [PATCH] mm/slab: abstract slabobj_ext.objcg accessIn preparation for changes to the structure, abstract getting and setting the objcg field with
slab_obj_ext_objcg()andslab_obj_ext_set_objcg().
Rename the field to_objcgto make an unexpected direct access a compile error.The helpers take a slab pointer, which is currently unused, but will be used by a debug check later.
Since there is no slab pointer easily available in__kfence_free(), just drop the debug check there. The wholememcg_kmemaccounting in kfence is to be removed later anyway.
Quoted 8 lines
Otherwise, no functional change intended.
Reviewed-by Hao Li <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Harry Yoo (Oracle) <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Thanks, still looks good to me!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerPATCH v3 7/13 mm/slab: replace slab.stride with obj_exts_in_object
In reply to Vlastimil Babka (SUSE)The stride field is used to convert object index to an slabobj_ext so both compact arrays (kmalloc() or in-slab-leftover) and spread in-object-padding obj_ext layouts are supported.
In practice thus the stride is always sizeof(slabobj_ext) or s->size.
This simplifies the calculations, but with the upcoming slabobj_ext handling changes, it will be easier to stop storing the stride and instead just have a flag whether obj_ext is in the object padding. obj_exts_in_object() can then rely on this flag and slab_obj_ext() can use that to determine the stride.
No functional change intended. Performance impact TBD, hopefully in the noise.
--- mm/slab.h | 44 +++++++++++++++++++++++++------------------- mm/slub.c | 42 ++++++++++++++++-------------------------- 2 files changed, 41 insertions(+), 45 deletions(-)
mm/slab.h +25 -19
@@ -81,10 +81,10 @@ struct freelist_counters {#ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field -* to save memory. In case ->stride field is not available, +* to save memory. If these free bits are not available, * such optimizations are disabled. */ -unsigned int stride; + unsigned obj_exts_in_object:1;#endif }; };
@@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long obj_exts) }#ifdef CONFIG_64BIT -static inline void slab_set_stride (struct slab *slab,unsigned int stride) +static inline bool obj_exts_in_object (struct slab *slab) { - slab->stride = stride; -} -static inline unsigned int slab_get_stride (struct slab *slab) -{ -return slab->stride; +/* +* Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to +* check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but +* allocations within_slab_leftover are preferred. And those may be +* possible or not depending on the particular slab's size. +*/ +return slab->obj_exts_in_object; }#else -static inline void slab_set_stride (struct slab *slab,unsigned int stride) +static inline bool obj_exts_in_object (struct slab *slab) { -VM_WARN_ON_ONCE (stride !=sizeof (struct slabobj_ext )); -} -static inline unsigned int slab_get_stride (struct slab *slab) -{ -return sizeof (struct slabobj_ext ); +return false; }#endif
@@ -657,8 +655,15 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,VM_WARN_ON_ONCE (obj_exts !=slab_obj_exts (slab)); index =obj_to_index (s, slab, obj); - obj_ext = (struct slabobj_ext *)(obj_exts + -slab_get_stride (slab) * index); + +if (!obj_exts_in_object (slab)) { + obj_ext = ((struct slabobj_ext *)obj_exts) + index; + }else { +unsigned int stride = s->size; + + obj_ext = (struct slabobj_ext *)(obj_exts + index * stride); + } +return kasan_reset_tag (obj_ext); }
@@ -700,9 +705,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts, return NULL; } -static inline void slab_set_stride (struct slab *slab,unsigned int stride) { } -static inline unsigned int slab_get_stride (struct slab *slab) {return 0 ; } - +static inline bool obj_exts_in_object (struct slab *slab) +{ +return false; +}#endif /* CONFIG_SLAB_OBJ_EXT */
mm/slub.c +16 -26
103 lines not on this page · raw message
Signature · 1 line
2.55.0
Re: [PATCH v3 07/13] mm/slab: replace slab.stride with obj_exts_in_object
Quoted 6 lines · Vlastimil Babka (SUSE)
The stride field is used to convert object index to an
slabobj_extso both compact arrays (kmalloc()or in-slab-leftover) and spread in-object-paddingobj_extlayouts are supported.In practice thus the stride is always
sizeof(slabobj_ext)ors->size.
Quoted 8 lines
This simplifies the calculations, but with the upcoming
slabobj_exthandling changes, it will be easier to stop storing the stride and instead just have a flag whetherobj_extis in the object padding.obj_exts_in_object()can then rely on this flag andslab_obj_ext()can use that to determine the stride.No functional change intended. Performance impact TBD, hopefully in the noise.
Performance impact is not TBD anymore?
Quoted 4 lines
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 07/13] mm/slab: replace slab.stride with obj_exts_in_object
Quoted 13 lines · Harry Yoo
Quoted 12 lines · Vlastimil Babka (SUSE)
The stride field is used to convert object index to an
slabobj_extso both compact arrays (kmalloc()or in-slab-leftover) and spread in-object-paddingobj_extlayouts are supported.In practice thus the stride is always
sizeof(slabobj_ext)ors->size.This simplifies the calculations, but with the upcoming
slabobj_exthandling changes, it will be easier to stop storing the stride and instead just have a flag whetherobj_extis in the object padding.obj_exts_in_object()can then rely on this flag andslab_obj_ext()can use that to determine the stride.
Quoted 4 lines
Quoted 2 lines
No functional change intended. Performance impact TBD, hopefully in the noise.
Performance impact is not TBD anymore?
Ah right, thanks. Since I didn't measure this commit separately, will remove the note completely and leave it to cover letter -> merge commit only.
Quoted 7 lines
Quoted 4 lines
Reviewed-by Suren Baghdasaryan <[email protected]>Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Reviewed-by Harry Yoo (Oracle) <[email protected]>
Thanks!
Vlastimil Babka (SUSE)
maintainerPATCH v3 8/13 mm/slab: change struct slabobj_ext to a union
In reply to Vlastimil Babka (SUSE)Currently, struct slabobj_ext can hold both objcg pointer and codetag_ref (when both are compile-enabled) and there is an array of as many slabobj_ext instances as there are objects in a slab.
This makes the layout fixed so even if codetag_ref is unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currently kmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled.
To make this more flexible, change the layout so that struct slabobj_ext becomes a union of objcg pointer and codetag_ref (to ensure uniform size; in practice both are the same size anyway). The slabobj_ext array then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent.
cache_obj_ext_size() returns the effective size of (0-2) struct slabobj_ext's for a cache, slab_obj_ext_size() for a slab. Currently both return a constant value derived from the config options, but will be made dynamic later. Replace all sizeof(slabobj_ext) usage with these.
No functional change intended, the layout is still effectively static.
--- mm/slab.h | 49 +++++++++++++++++++++++++++++++++++++++---------- mm/slub.c | 19 +++++++++++-------- 2 files changed, 50 insertions(+), 18 deletions(-)
mm/slab.h +39 -10
@@ -550,18 +550,42 @@ static inline bool need_kmalloc_no_objext(void) }/* -* Extended information for slab objects stored as an array in page->memcg_data -* if MEMCG_DATA_OBJEXTS is set. +* Extended information for slab objects stored as a pointer to an array in +* slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +/* +* All elements of the union should be pointer-sized to avoid memory +* waste +*/ +union {#ifdef CONFIG_MEMCG -struct obj_cgroup *_objcg; +struct obj_cgroup *_objcg;#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -union codetag_ref _ctref; +union codetag_ref _ctref;#endif + }; }__aligned(8) ; +static inline size_t cache_obj_ext_size (struct kmem_cache *s) +{ +size_t sz =0 ; + +if (IS_ENABLED (CONFIG_MEMCG )) + sz +=1 ; + +if (IS_ENABLED (CONFIG_MEM_ALLOC_PROFILING )) + sz +=1 ; + +return sizeof (struct slabobj_ext ) * sz; +} + +static inline size_t slab_obj_ext_size (struct slab *slab) +{ +return cache_obj_ext_size (slab->slab_cache); +} +#ifdef CONFIG_SLAB_OBJ_EXT /*
@@ -651,18 +675,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts, {struct slabobj_ext *obj_ext;unsigned int index; +unsigned int stride;VM_WARN_ON_ONCE (obj_exts !=slab_obj_exts (slab)); index =obj_to_index (s, slab, obj); -if (!obj_exts_in_object (slab)) { - obj_ext = ((struct slabobj_ext *)obj_exts) + index; - }else { -unsigned int stride = s->size; +if (!obj_exts_in_object (slab)) + stride =slab_obj_ext_size (slab); +else + stride = s->size; - obj_ext = (struct slabobj_ext *)(obj_exts + index * stride); - } + obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);return kasan_reset_tag (obj_ext); }
@@ -670,12 +694,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,#ifdef CONFIG_MEMCG static inline struct obj_cgroup *slab_obj_ext_objcg (struct slabobj_ext *obj_ext) { +/* if objcg exists, it comes first, so we don't need to do anything */ return obj_ext->_objcg; }static inline void slab_obj_ext_set_objcg (struct slabobj_ext *obj_ext,struct obj_cgroup *objcg) { +/* if objcg exists, it comes first, so we don't need to do anything */ obj_ext->_objcg = objcg; }#endif
@@ -684,6 +710,9 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { +if (IS_ENABLED (CONFIG_MEMCG )) + obj_ext +=1 ; +return &obj_ext->_ctref; }#endif
mm/slub.c +11 -8
@@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache *s)static inline unsigned int obj_exts_size_in_slab (struct slab *slab) { -return sizeof (struct slabobj_ext ) * slab->objects; +return slab_obj_ext_size (slab) * slab->objects; } static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,
@@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p) off +=kasan_metadata_size (s, false);if (obj_exts_in_object (slab)) - off +=sizeof (struct slabobj_ext ); + off +=slab_obj_ext_size (slab);if (off !=size_from_object (s))/* Beginning of the filler is the free pointer */
@@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p) off +=kasan_metadata_size (s, false);if (obj_exts_in_object (slab)) - off +=sizeof (struct slabobj_ext ); + off +=slab_obj_ext_size (slab);if (size_from_object (s) == off)return 1 ;
@@ -2092,6 +2092,8 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)static inline void handle_failed_objexts_alloc (struct slab *slab,unsigned long obj_exts,struct slabobj_ext *vec) { +unsigned int stride; +/* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this
@@ -2100,11 +2102,13 @@ static inline void handle_failed_objexts_alloc(struct slab *slab,if (obj_exts !=OBJEXTS_ALLOC_FAIL )return ; + stride =slab_obj_ext_size (slab) /sizeof (*vec); +for (unsigned int i =0 ; i < slab->objects; i++) {union codetag_ref *ref =slab_obj_ext_codetag_ref (slab, vec);set_codetag_empty (ref); - vec++; + vec += stride; } }
@@ -2130,7 +2134,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,unsigned long new_exts;unsigned long old_exts;struct slabobj_ext *vec; -size_t sz =sizeof (struct slabobj_ext ) * slab->objects; +size_t sz =slab_obj_ext_size (slab) * slab->objects; gfp &= ~OBJCGS_CLEAR_MASK ;/*
@@ -2273,8 +2277,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)get_slab_obj_exts (obj_exts);for_each_object (addr, s,slab_address (slab), slab->objects) -memset (kasan_reset_tag (addr) + offset,0 , -sizeof (struct slabobj_ext )); +memset (kasan_reset_tag (addr) + offset,0 ,slab_obj_ext_size (slab));put_slab_obj_exts (obj_exts);#ifdef CONFIG_MEMCG
@@ -7933,7 +7936,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) aligned_size =ALIGN (size, s->align);#if defined(CONFIG_SLAB_OBJ_EXT ) && defined(CONFIG_64BIT )if (slab_args_unmergeable (args, s->flags) && - (aligned_size - size >=sizeof (struct slabobj_ext ))) + (aligned_size - size >=cache_obj_ext_size (s))) s->flags |=SLAB_OBJ_EXT_IN_OBJ ;#endif size = aligned_size;
Signature · 1 line
2.55.0
Re: [PATCH v3 08/13] mm/slab: change struct slabobj_ext to a union
Quoted 22 lines · Vlastimil Babka (SUSE)
Currently, struct
slabobj_extcan hold both objcg pointer andcodetag_ref(when both are compile-enabled) and there is an array of as manyslabobj_extinstances as there are objects in a slab.This makes the layout fixed so even if
codetag_refis unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currentlykmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled.To make this more flexible, change the layout so that struct
slabobj_extbecomes a union of objcg pointer andcodetag_ref(to ensure uniform size; in practice both are the same size anyway). Theslabobj_extarray then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent.
cache_obj_ext_size()returns the effective size of (0-2) struct slabobj_ext's for a cache,slab_obj_ext_size()for a slab. Currently both return a constant value derived from the config options, but will be made dynamic later. Replace allsizeof(slabobj_ext)usage with these.
Quoted 5 lines
No functional change intended, the layout is still effectively static.
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Looks good to me,
Re: [PATCH v3 08/13] mm/slab: change struct slabobj_ext to a union
In reply to Vlastimil Babka (SUSE)Quoted 22 lines · Vlastimil Babka (SUSE)
Currently, struct
slabobj_extcan hold both objcg pointer andcodetag_ref(when both are compile-enabled) and there is an array of as manyslabobj_extinstances as there are objects in a slab.This makes the layout fixed so even if
codetag_refis unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currentlykmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled.To make this more flexible, change the layout so that struct
slabobj_extbecomes a union of objcg pointer andcodetag_ref(to ensure uniform size; in practice both are the same size anyway). Theslabobj_extarray then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent.
cache_obj_ext_size()returns the effective size of (0-2) struct slabobj_ext's for a cache,slab_obj_ext_size()for a slab. Currently both return a constant value derived from the config options, but will be made dynamic later. Replace allsizeof(slabobj_ext)usage with these.
Quoted 5 lines
No functional change intended, the layout is still effectively static.
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
LGTM.
Signature · 2 lines
Thanks, Hao
Hao Li
reviewerRe: [PATCH v3 08/13] mm/slab: change struct slabobj_ext to a union
In reply to Vlastimil Babka (SUSE)Quoted 207 lines · Vlastimil Babka (SUSE)
Currently, struct
slabobj_extcan hold both objcg pointer andcodetag_ref(when both are compile-enabled) and there is an array of as manyslabobj_extinstances as there are objects in a slab.This makes the layout fixed so even if
codetag_refis unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currentlykmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled.To make this more flexible, change the layout so that struct
slabobj_extbecomes a union of objcg pointer andcodetag_ref(to ensure uniform size; in practice both are the same size anyway). Theslabobj_extarray then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent.
cache_obj_ext_size()returns the effective size of (0-2) struct slabobj_ext's for a cache,slab_obj_ext_size()for a slab. Currently both return a constant value derived from the config options, but will be made dynamic later. Replace allsizeof(slabobj_ext)usage with these.No functional change intended, the layout is still effectively static.
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/slab.h | 49 +++++++++++++++++++++++++++++++++++++++---------- mm/slub.c | 19 +++++++++++-------- 2 files changed, 50 insertions(+), 18 deletions(-)mm/slab.h +39
-10@@ -550,18 +550,42 @@ static inline bool need_kmalloc_no_objext(void) }/* -* Extended information for slab objects stored as an array in page->memcg_data -* if MEMCG_DATA_OBJEXTS is set. +* Extended information for slab objects stored as a pointer to an array in +* slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +/* +* All elements of the union should be pointer-sized to avoid memory +* waste +*/ +union {#ifdef CONFIG_MEMCG -struct obj_cgroup *_objcg; +struct obj_cgroup *_objcg;#endif #ifdef CONFIG_MEM_ALLOC_PROFILING -union codetag_ref _ctref; +union codetag_ref _ctref;#endif + }; }__aligned(8) ; +static inline size_t cache_obj_ext_size (struct kmem_cache *s) +{ +size_t sz =0 ; + +if (IS_ENABLED (CONFIG_MEMCG )) + sz +=1 ; + +if (IS_ENABLED (CONFIG_MEM_ALLOC_PROFILING )) + sz +=1 ; + +return sizeof (struct slabobj_ext ) * sz; +} + +static inline size_t slab_obj_ext_size (struct slab *slab) +{ +return cache_obj_ext_size (slab->slab_cache); +} +#ifdef CONFIG_SLAB_OBJ_EXT /* @@ -651,18 +675,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts, {struct slabobj_ext *obj_ext;unsigned int index; +unsigned int stride;VM_WARN_ON_ONCE (obj_exts !=slab_obj_exts (slab)); index =obj_to_index (s, slab, obj); -if (!obj_exts_in_object (slab)) { - obj_ext = ((struct slabobj_ext *)obj_exts) + index; - }else { -unsigned int stride = s->size; +if (!obj_exts_in_object (slab)) + stride =slab_obj_ext_size (slab); +else + stride = s->size; - obj_ext = (struct slabobj_ext *)(obj_exts + index * stride); - } + obj_ext = (struct slabobj_ext *)(obj_exts + index * stride);return kasan_reset_tag (obj_ext); }@@ -670,12 +694,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,#ifdef CONFIG_MEMCG static inline struct obj_cgroup *slab_obj_ext_objcg (struct slabobj_ext *obj_ext) { +/* if objcg exists, it comes first, so we don't need to do anything */ return obj_ext->_objcg; }static inline void slab_obj_ext_set_objcg (struct slabobj_ext *obj_ext,struct obj_cgroup *objcg) { +/* if objcg exists, it comes first, so we don't need to do anything */ obj_ext->_objcg = objcg; }#endif @@ -684,6 +710,9 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { +if (IS_ENABLED (CONFIG_MEMCG )) + obj_ext +=1 ; +return &obj_ext->_ctref; }#endif mm/slub.c +10
-7@@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache *s)static inline unsigned int obj_exts_size_in_slab (struct slab *slab) { -return sizeof (struct slabobj_ext ) * slab->objects; +return slab_obj_ext_size (slab) * slab->objects; } static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,@@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p) off +=kasan_metadata_size (s, false);if (obj_exts_in_object (slab)) - off +=sizeof (struct slabobj_ext ); + off +=slab_obj_ext_size (slab);if (off !=size_from_object (s))/* Beginning of the filler is the free pointer */ @@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p) off +=kasan_metadata_size (s, false);if (obj_exts_in_object (slab)) - off +=sizeof (struct slabobj_ext ); + off +=slab_obj_ext_size (slab);if (size_from_object (s) == off)return 1 ;@@ -2092,6 +2092,8 @@ static inline bool mark_failed_objexts_alloc(struct slab *slab)static inline void handle_failed_objexts_alloc (struct slab *slab,unsigned long obj_exts,struct slabobj_ext *vec) { +unsigned int stride; +/* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this @@ -2100,11 +2102,13 @@ static inline void handle_failed_objexts_alloc(struct slab *slab,if (obj_exts !=OBJEXTS_ALLOC_FAIL )return ; + stride =slab_obj_ext_size (slab) /sizeof (*vec); +for (unsigned int i =0 ; i < slab->objects; i++) {union codetag_ref *ref =slab_obj_ext_codetag_ref (slab, vec);set_codetag_empty (ref); - vec++; + vec += stride; } }@@ -2130,7 +2134,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,unsigned long new_exts;unsigned long old_exts;struct slabobj_ext *vec; -size_t sz =sizeof (struct slabobj_ext ) * slab->objects; +size_t sz =slab_obj_ext_size (slab) * slab->objects; gfp &= ~OBJCGS_CLEAR_MASK ;/* @@ -2273,8 +2277,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)get_slab_obj_exts (obj_exts);for_each_object (addr, s,slab_address (slab), slab->objects) -memset (kasan_reset_tag (addr) + offset,0 , -sizeof (struct slabobj_ext )); +memset (kasan_reset_tag (addr) + offset,0 ,slab_obj_ext_size (slab));put_slab_obj_exts (obj_exts);#ifdef CONFIG_MEMCG
mm/slub.c static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) · line 7939
Quoted 5 lines
mm/slub.c +1
-1@@ -7933,7 +7936,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) aligned_size =ALIGN (size, s->align);#if defined(CONFIG_SLAB_OBJ_EXT ) && defined(CONFIG_64BIT )if (slab_args_unmergeable (args, s->flags) && - (aligned_size - size >=sizeof (struct slabobj_ext ))) + (aligned_size - size >=cache_obj_ext_size (s)))
Nit: cache_obj_ext_size() could return 0, and the checking will always be true. But I think this is harmless.
Quoted 7 lines
mm/slub.c +0
-0s->flags |=SLAB_OBJ_EXT_IN_OBJ ;#endif size = aligned_size;Signature · 2 lines
-- 2.55.0
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 08/13] mm/slab: change struct slabobj_ext to a union
Quoted 10 lines · Hao Li
mm/slub.c static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) · line 7939
Quoted 6 lines · Vlastimil Babka (SUSE)
mm/slub.c +1
-1@@ -7933,7 +7936,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) aligned_size =ALIGN (size, s->align);#if defined(CONFIG_SLAB_OBJ_EXT ) && defined(CONFIG_64BIT )if (slab_args_unmergeable (args, s->flags) && - (aligned_size - size >=sizeof (struct slabobj_ext ))) + (aligned_size - size >=cache_obj_ext_size (s)))Nit:
cache_obj_ext_size()could return 0, and the checking will always be true. But I think this is harmless.
Yeah should be harmless as then no objexts are used for the cache anyway. Thanks!
Quoted 8 lines
Quoted 7 lines
mm/slub.c +0
-0s->flags |=SLAB_OBJ_EXT_IN_OBJ ;#endif size = aligned_size;Signature · 2 lines
-- 2.55.0
Vlastimil Babka (SUSE)
maintainerPATCH v3 9/13 mm/slab: introduce slab_obj_ext_has_codetag()
In reply to Vlastimil Babka (SUSE)mem_alloc_profiling_enabled() allows evaluating (with a static key) if memory profiling is currently enabled. mem_profiling_support is a variable where false means it's not possible to enable it anymore, because the system was booted with "never" or it was later shut down. This is possible to query by mem_alloc_profiling_permanently_disabled().
To make slabobj_ext array size handling dynamic, we need a snapshot of mem_alloc_profiling_permanently_disabled() early in boot, so that's not affected by a later shutdown. We also need it to be static key based for performance. Neither mem_alloc_profiling_enabled() nor mem_alloc_profiling_permanently_disabled() satisfy this.
Therefore introduce slab_obj_ext_has_codetag() with an underlying static key for that use case. Its state is made to reflect the result of mem_alloc_profiling_permanently_disabled() during kmem_cache_init(), which does happen after setup_early_mem_profiling().
--- mm/slab.h | 16 ++++++++++++++++ mm/slub.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
mm/slab.h +16 -0
@@ -568,6 +568,22 @@ struct slabobj_ext { }; }__aligned(8) ; +#ifdef CONFIG_MEM_ALLOC_PROFILING +DECLARE_STATIC_KEY_MAYBE (CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT , + slab_obj_ext_has_codetag_key); + +static inline bool slab_obj_ext_has_codetag (void ) +{ +return static_branch_maybe (CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT , + &slab_obj_ext_has_codetag_key); +} +#else +static inline bool slab_obj_ext_has_codetag (void ) +{ +return false; +} +#endif +static inline size_t cache_obj_ext_size (struct kmem_cache *s) {size_t sz =0 ;
mm/slub.c +30 -0
@@ -213,6 +213,11 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled);static DEFINE_STATIC_KEY_FALSE (strict_numa);#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +DEFINE_STATIC_KEY_MAYBE (CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT , + slab_obj_ext_has_codetag_key); +#endif +/* Structure holding extra parameters for slab allocations */ struct slab_alloc_context {unsigned long caller_addr;
@@ -2420,6 +2425,25 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, __alloc_tagging_slab_free_hook(s, slab, p, objects); } +/* +* Make sure the static key used by slab_obj_ext_has_codetag() reflects the +* value of !mem_alloc_profiling_permanently_disabled() +* +* Any later mem alloc profiling shutdown won't be reflected in the static key +* because obj_exts with codetags might already exist. +*/ +static void __init slab_obj_ext_has_codetag_init (void ) +{ +bool need_codetag = !mem_alloc_profiling_permanently_disabled (); + +if (need_codetag !=static_key_enabled (&slab_obj_ext_has_codetag_key)) { +if (need_codetag) +static_branch_enable (&slab_obj_ext_has_codetag_key); +else +static_branch_disable (&slab_obj_ext_has_codetag_key); + } +} +#else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void
@@ -2434,6 +2458,10 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, { } +static inline void slab_obj_ext_has_codetag_init (void ) +{ +} +#endif /* CONFIG_MEM_ALLOC_PROFILING */
@@ -8546,6 +8574,8 @@ void __init kmem_cache_init(void) boot_kmem_cache_node;int node; +slab_obj_ext_has_codetag_init (); +if (debug_guardpage_minorder ()) slub_max_order =0 ;
Signature · 1 line
2.55.0
Re: [PATCH v3 09/13] mm/slab: introduce slab_obj_ext_has_codetag()
Quoted 12 lines · Vlastimil Babka (SUSE)
mem_alloc_profiling_enabled()allows evaluating (with a static key) if memory profiling is currently enabled.mem_profiling_supportis a variable where false means it's not possible to enable it anymore, because the system was booted with "never" or it was later shut down. This is possible to query bymem_alloc_profiling_permanently_disabled().To make
slabobj_extarray size handling dynamic, we need a snapshot ofmem_alloc_profiling_permanently_disabled()early in boot, so that's not affected by a later shutdown. We also need it to be static key based for performance. Neithermem_alloc_profiling_enabled()normem_alloc_profiling_permanently_disabled()satisfy this.
Quoted 7 lines
Therefore introduce
slab_obj_ext_has_codetag()with an underlying static key for that use case. Its state is made to reflect the result ofmem_alloc_profiling_permanently_disabled()duringkmem_cache_init(), which does happen aftersetup_early_mem_profiling().Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>
LGTM.
Signature · 2 lines
Thanks, Hao
Re: [PATCH v3 09/13] mm/slab: introduce slab_obj_ext_has_codetag()
In reply to Vlastimil Babka (SUSE)Quoted 12 lines · Vlastimil Babka (SUSE)
mem_alloc_profiling_enabled()allows evaluating (with a static key) if memory profiling is currently enabled.mem_profiling_supportis a variable where false means it's not possible to enable it anymore, because the system was booted with "never" or it was later shut down. This is possible to query bymem_alloc_profiling_permanently_disabled().To make
slabobj_extarray size handling dynamic, we need a snapshot ofmem_alloc_profiling_permanently_disabled()early in boot, so that's not affected by a later shutdown. We also need it to be static key based for performance. Neithermem_alloc_profiling_enabled()normem_alloc_profiling_permanently_disabled()satisfy this.
Quoted 8 lines
Therefore introduce
slab_obj_ext_has_codetag()with an underlying static key for that use case. Its state is made to reflect the result ofmem_alloc_profiling_permanently_disabled()duringkmem_cache_init(), which does happen aftersetup_early_mem_profiling().Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerPATCH v3 10/13 mm/slab: reduce slabobj_ext memory with allocation profiling disabled
In reply to Vlastimil Babka (SUSE)When memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (thus wasting) memory for the codetag_ref parts of slabobj_ext metadata.
Do this by using the new slab_obj_ext_has_codetag() helper in cache_obj_ext_size().
Additionally add a slab_obj_ext_has_codetag() check in handle_failed_objexts_alloc(). The function might get called with memory allocation profiling disabled, when the obj_ext array is allocated for objcg pointers only. Setting codetag refs as empty is unnecessary in that case, and with them not allocated anymore would now result in memory corruption.
--- mm/slab.h | 2 +- mm/slub.c | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-)
mm/slab.h +1 -1
@@ -591,7 +591,7 @@ static inline size_t cache_obj_ext_size(struct kmem_cache *s)if (IS_ENABLED (CONFIG_MEMCG )) sz +=1 ; -if (IS_ENABLED (CONFIG_MEM_ALLOC_PROFILING )) +if (slab_obj_ext_has_codetag ()) sz +=1 ;return sizeof (struct slabobj_ext ) * sz;
mm/slub.c +3 -0
@@ -2099,6 +2099,9 @@ static inline void handle_failed_objexts_alloc(struct slab *slab, {unsigned int stride; +if (!slab_obj_ext_has_codetag ()) +return ; +/* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this
Signature · 1 line
2.55.0
Re: [PATCH v3 10/13] mm/slab: reduce slabobj_ext memory with allocation profiling disabled
Per sashiko [1], a fixed up version below. Couldn't trigger the issue although seems to me it's real. Perhaps can happen only with SLUB_TINY otherwise the sheaf can't have obj_exts allocated from objcg charge path.
[1] https://sashiko.dev/#/patchset/20260727-b4-objext_split-v3-0-c29ef0f1f257%40kernel.org?part=10
----8<----
From 905c6500e597a8ecf123ad167698f0a0c1a764bd Mon Sep 17 00:00:00 2001
From: "Vlastimil Babka (SUSE)" <[email protected]>
Date: Wed, 8 Jul 2026 17:04:36 +0200
Subject: [PATCH] mm/slab: reduce slabobj_ext memory with allocation profiling
disabled
When memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (thus wasting) memory for the codetag_ref parts of slabobj_ext metadata.
Do this by using the new slab_obj_ext_has_codetag() helper in cache_obj_ext_size().
Additionally add slab_obj_ext_has_codetag() checks in mark_obj_codetag_empty() and handle_failed_objexts_alloc(). The functions might get called with memory allocation profiling disabled, when the obj_ext array is allocated for objcg pointers only. Setting codetag refs as empty is unnecessary in that case, and with them not allocated anymore would now result in memory corruption.
--- mm/slab.h | 2 +- mm/slub.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
mm/slab.h +1 -1
@@ -591,7 +591,7 @@ static inline size_t cache_obj_ext_size(struct kmem_cache *s)if (IS_ENABLED (CONFIG_MEMCG )) sz +=1 ; -if (IS_ENABLED (CONFIG_MEM_ALLOC_PROFILING )) +if (slab_obj_ext_has_codetag ()) sz +=1 ;return sizeof (struct slabobj_ext ) * sz;
mm/slub.c +6 -0
@@ -2064,6 +2064,9 @@ static inline void mark_obj_codetag_empty(const void *obj)struct slab *obj_slab;unsigned long slab_exts; +if (!slab_obj_ext_has_codetag ()) +return ; +if (is_kfence_address (obj))return ;
@@ -2099,6 +2102,9 @@ static inline void handle_failed_objexts_alloc(struct slab *slab, {unsigned int stride; +if (!slab_obj_ext_has_codetag ()) +return ; +/* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this
Signature · 1 line
2.55.0
Re: [PATCH v3 10/13] mm/slab: reduce slabobj_ext memory with allocation profiling disabled
Quoted 27 lines · Vlastimil Babka (SUSE)
Per sashiko [1], a fixed up version below. Couldn't trigger the issue although seems to me it's real. Perhaps can happen only with
SLUB_TINYotherwise the sheaf can't haveobj_extsallocated from objcg charge path.[1] https://sashiko.dev/#/patchset/20260727-b4-objext_split-v3-0-c29ef0f1f257%40kernel.org?part=10
----8<----
From905c6500e597a8ecf123ad167698f0a0c1a764bdMon Sep 17 00:00:00 2001
From: "Vlastimil Babka (SUSE)" <[email protected]>
Date: Wed, 8 Jul 2026 17:04:36 +0200
Subject: [PATCH] mm/slab: reduceslabobj_extmemory with allocation profiling
disabledWhen memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (thus wasting) memory for the
codetag_refparts ofslabobj_extmetadata.Do this by using the new
slab_obj_ext_has_codetag()helper incache_obj_ext_size().Additionally add
slab_obj_ext_has_codetag()checks inmark_obj_codetag_empty()andhandle_failed_objexts_alloc(). The functions might get called with memory allocation profiling disabled, when theobj_extarray is allocated for objcg pointers only. Setting codetag refs as empty is unnecessary in that case, and with them not allocated anymore would now result in memory corruption.
Quoted 2 lines
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>
LGTM.
Signature · 2 lines
Thanks, Hao
Re: [PATCH v3 10/13] mm/slab: reduce slabobj_ext memory with allocation profiling disabled
In reply to Vlastimil Babka (SUSE)Quoted 27 lines · Vlastimil Babka (SUSE)
Per sashiko [1], a fixed up version below. Couldn't trigger the issue although seems to me it's real. Perhaps can happen only with
SLUB_TINYotherwise the sheaf can't haveobj_extsallocated from objcg charge path.[1] https://sashiko.dev/#/patchset/20260727-b4-objext_split-v3-0-c29ef0f1f257%40kernel.org?part=10
----8<----
From905c6500e597a8ecf123ad167698f0a0c1a764bdMon Sep 17 00:00:00 2001
From: "Vlastimil Babka (SUSE)" <[email protected]>
Date: Wed, 8 Jul 2026 17:04:36 +0200
Subject: [PATCH] mm/slab: reduceslabobj_extmemory with allocation profiling
disabledWhen memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (thus wasting) memory for the
codetag_refparts ofslabobj_extmetadata.Do this by using the new
slab_obj_ext_has_codetag()helper incache_obj_ext_size().Additionally add
slab_obj_ext_has_codetag()checks inmark_obj_codetag_empty()andhandle_failed_objexts_alloc(). The functions might get called with memory allocation profiling disabled, when theobj_extarray is allocated for objcg pointers only. Setting codetag refs as empty is unnecessary in that case, and with them not allocated anymore would now result in memory corruption.
Quoted 3 lines
Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerPATCH v3 11/13 mm/slab: add cache_ and slab_needs_objcg() helpers
In reply to Vlastimil Babka (SUSE)Slabs of some caches never need the objcg part of struct slabobj_ext. Introduce helpers to query this for a cache or a slab.
Introduce SLAB_MAY_ACCOUNT flag that is currently only internal and all caches have it set except:
- KMALLOC_NORMAL caches, as long as KMALLOC_RECLAIM caches are separate
- KMALLOC_NO_OBJ_EXT caches, if they exist
For named caches we currently can't derive SLAB_MAY_ACCOUNT from SLAB_ACCOUNT because some caches might be created without SLAB_ACCOUNT and then used both with and without __GFP_ACCOUNT concurrently, allocating obj_ext arrays on demand. So just add the SLAB_MAY_ACCOUNT to all kmem caches, unless kmem accounting is disabled.
This can be improved later by finding out all caches used with __GFP_ACCOUNT, creating them with the SLAB_MAY_ACCOUNT flag explicitly, and then ignoring __GFP_ACCOUNT for all other caches (possibly with a warning).
To make the evaluation of slab_needs_objcg() faster in the allocation and free fast paths, add a obj_exts_needs_objcg flag into slab itself. This optimization is only available on 64bit architectures where free bits are available for the flag.
--- include/linux/slab.h | 3 +++ mm/kfence/core.c | 3 +++ mm/slab.h | 31 +++++++++++++++++++++++++++++-- mm/slab_common.c | 26 +++++++++++++++++++++----- mm/slub.c | 6 +++++- 5 files changed, 61 insertions(+), 8 deletions(-)
include/linux/slab.h +3 -0
@@ -45,6 +45,7 @@ enum _slab_flag_bits {#endif #ifdef CONFIG_MEMCG _SLAB_ACCOUNT, + _SLAB_MAY_ACCOUNT,#endif #ifdef CONFIG_KASAN_GENERIC _SLAB_KASAN,
@@ -204,8 +205,10 @@ enum _slab_flag_bits {*/ #ifdef CONFIG_MEMCG # define SLAB_ACCOUNT __SLAB_FLAG_BIT(_SLAB_ACCOUNT) +# define SLAB_MAY_ACCOUNT __SLAB_FLAG_BIT(_SLAB_MAY_ACCOUNT)#else # define SLAB_ACCOUNT __SLAB_FLAG_UNUSED +# define SLAB_MAY_ACCOUNT __SLAB_FLAG_UNUSED#endif #ifdef CONFIG_KASAN_GENERIC
mm/kfence/core.c +3 -0
@@ -640,6 +640,9 @@ static unsigned long kfence_init_pool(void)struct slab *slab =page_slab (page); slab->obj_exts = (unsigned long )&kfence_metadata_init[i /2 -1 ].obj_exts |MEMCG_DATA_OBJEXTS ; +#ifdef CONFIG_64BIT + slab->obj_exts_needs_objcg =1 ; +#endif #endif }
mm/slab.h +29 -2
@@ -81,10 +81,11 @@ struct freelist_counters {#ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field -* to save memory. If these free bits are not available, -* such optimizations are disabled. +* to save memory or CPU. If these free bits are not +* available, such optimizations are disabled. */ unsigned obj_exts_in_object:1; + unsigned obj_exts_needs_objcg:1;#endif }; };
@@ -584,6 +585,32 @@ static inline bool slab_obj_ext_has_codetag(void) }#endif +#ifdef CONFIG_MEMCG +static inline bool cache_needs_objcg (struct kmem_cache *cache) +{ +return (cache->flags &SLAB_MAY_ACCOUNT ); +} + +static inline bool slab_needs_objcg (struct slab *slab) +{ +#ifdef CONFIG_64BIT +return slab->obj_exts_needs_objcg; +#else +return cache_needs_objcg (slab->slab_cache); +#endif +} +#else +static inline bool cache_needs_objcg (struct kmem_cache *cache) +{ +return false; +} + +static inline bool slab_needs_objcg (struct slab *slab) +{ +return false; +} +#endif +static inline size_t cache_obj_ext_size (struct kmem_cache *s) {size_t sz =0 ;
mm/slab_common.c +21 -5
@@ -52,7 +52,7 @@ struct kmem_cache *kmem_cache; SLAB_OBJ_EXT_IN_OBJ)#define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \ - SLAB_CACHE_DMA32 | SLAB_ACCOUNT) + SLAB_CACHE_DMA32 | SLAB_ACCOUNT | SLAB_MAY_ACCOUNT)/* * Merge control. If this is set then no merging of slab caches will occur.
@@ -359,6 +359,13 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, goto out_unlock; } +/* +* For now we assume any cache can be used with __GFP_ACCOUNT and thus +* may need to store objcg pointers for objects +*/ +if (!mem_cgroup_kmem_disabled ()) + flags |=SLAB_MAY_ACCOUNT ; +/* Fail closed on bad usersize of useroffset values. */ if (!IS_ENABLED(CONFIG_HARDENED_USERCOPY) || WARN_ON(!args->usersize && args->useroffset) ||
@@ -984,11 +991,20 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)#endif /* -* If CONFIG_MEMCG is enabled, disable cache merging for -* KMALLOC_NORMAL caches. +* If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not +* aliased with any other type, make sure it's never merged with any other +* cache. +* +* In other cases the kmalloc cache may end up being used for a +* __GFP_ACCOUNT allocation so mark it as such. The exception is a +* KMALLOC_NO_OBJ_EXT cache. */ -if (IS_ENABLED (CONFIG_MEMCG ) && (type ==KMALLOC_NORMAL )) - flags |=SLAB_NO_MERGE ; +if (!mem_cgroup_kmem_disabled ()) { +if (type ==KMALLOC_NORMAL &&KMALLOC_RECLAIM !=KMALLOC_NORMAL ) + flags |=SLAB_NO_MERGE ; +else if (!(flags &SLAB_NO_OBJ_EXT )) + flags |=SLAB_MAY_ACCOUNT ; + }if (minalign >ARCH_KMALLOC_MINALIGN ) { aligned_size =ALIGN (aligned_size, minalign);
mm/slub.c +5 -1
@@ -2555,7 +2555,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags)* of slab_obj_exts being allocated from the same slab and thus the slab * becoming effectively unfreeable. */ -if (is_kmalloc_normal (s)) +if (!cache_needs_objcg (s))return true;/* Ignore already charged objects. */
@@ -3433,6 +3433,10 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, slab->objects =oo_objects (oo); +#ifdef CONFIG_64BIT +if (cache_needs_objcg (s)) + slab->obj_exts_needs_objcg =1 ; +#endif slab->slab_cache = s;kasan_poison_slab (slab);
Signature · 1 line
2.55.0
Re: [PATCH v3 11/13] mm/slab: add cache_ and slab_needs_objcg() helpers
Quoted 20 lines · Vlastimil Babka (SUSE)
Slabs of some caches never need the objcg part of struct
slabobj_ext. Introduce helpers to query this for a cache or a slab.Introduce
SLAB_MAY_ACCOUNTflag that is currently only internal and all caches have it set except:-
KMALLOC_NORMALcaches, as long asKMALLOC_RECLAIMcaches are separate
-KMALLOC_NO_OBJ_EXTcaches, if they existFor named caches we currently can't derive
SLAB_MAY_ACCOUNTfromSLAB_ACCOUNTbecause some caches might be created withoutSLAB_ACCOUNTand then used both with and without__GFP_ACCOUNTconcurrently, allocatingobj_extarrays on demand. So just add theSLAB_MAY_ACCOUNTto all kmem caches, unless kmem accounting is disabled.This can be improved later by finding out all caches used with
__GFP_ACCOUNT, creating them with theSLAB_MAY_ACCOUNTflag explicitly, and then ignoring__GFP_ACCOUNTfor all other caches (possibly with a warning).
Quoted 8 lines
To make the evaluation of
slab_needs_objcg()faster in the allocation and free fast paths, add aobj_exts_needs_objcgflag into slab itself. This optimization is only available on 64bit architectures where free bits are available for the flag.Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerPATCH v3 12/13 mm/slab: stop allocating objcg pointers when unnecessary
In reply to Vlastimil Babka (SUSE)Start using the slab_needs_objcg() helper to calculate slabobj_ext size. Caches that we know to never need objcg pointers (currently KMALLOC_NORMAL caches) will thus stop wasting memory on them when memory allocation profiling is enabled.
For things to work properly, we need to also add slab_needs_objcg() checks to mem_cgroup_from_obj_slab() and memcg_slab_free_hook(), because when obj_exts array exists for a slab only due to mem_alloc profiling, we would otherwise attempt to access a non-existing objcg pointer in that slab.
The function __memcg_slab_post_alloc_hook() should not be possible to call for a slab where slab_needs_objcg() is false, but add a DEBUG_VM check there to prevent breaking this assumption accidentally.
--- mm/memcontrol.c | 6 ++++++ mm/slab.h | 14 +++++++++++--- mm/slub.c | 3 +++ 3 files changed, 20 insertions(+), 3 deletions(-)
mm/memcontrol.c +6 -0
@@ -2871,6 +2871,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)if (!obj_exts)return NULL ; +if (!slab_needs_objcg (slab)) +return NULL ; +get_slab_obj_exts (obj_exts); obj_ext =slab_obj_ext (slab->slab_cache, slab, obj_exts, p); objcg =slab_obj_ext_objcg (obj_ext);
@@ -3580,6 +3583,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, slab =virt_to_slab (p[i]); +if (IS_ENABLED (CONFIG_DEBUG_VM ) &&WARN_ON_ONCE (!slab_needs_objcg (slab))) +continue ; +if (!slab_obj_exts (slab) &&alloc_slab_obj_exts (slab, s, flags, slab_alloc_flags)) {continue ;
mm/slab.h +11 -3
@@ -615,7 +615,7 @@ static inline size_t cache_obj_ext_size(struct kmem_cache *s) {size_t sz =0 ; -if (IS_ENABLED (CONFIG_MEMCG )) +if (cache_needs_objcg (s)) sz +=1 ;if (slab_obj_ext_has_codetag ())
@@ -626,7 +626,15 @@ static inline size_t cache_obj_ext_size(struct kmem_cache *s)static inline size_t slab_obj_ext_size (struct slab *slab) { -return cache_obj_ext_size (slab->slab_cache); +size_t sz =0 ; + +if (slab_needs_objcg (slab)) + sz +=1 ; + +if (slab_obj_ext_has_codetag ()) + sz +=1 ; + +return sizeof (struct slabobj_ext ) * sz; }#ifdef CONFIG_SLAB_OBJ_EXT
@@ -753,7 +761,7 @@ static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext,static inline union codetag_ref *slab_obj_ext_codetag_ref (struct slab *slab,struct slabobj_ext *obj_ext) { -if (IS_ENABLED (CONFIG_MEMCG )) +if (slab_needs_objcg (slab)) obj_ext +=1 ;return &obj_ext->_ctref;
mm/slub.c +3 -0
@@ -2510,6 +2510,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p,if (likely (!obj_exts))return ; +if (!slab_needs_objcg (slab)) +return ; +get_slab_obj_exts (obj_exts);__memcg_slab_free_hook (s, slab, p, objects, obj_exts);put_slab_obj_exts (obj_exts);
Signature · 1 line
2.55.0
Re: [PATCH v3 12/13] mm/slab: stop allocating objcg pointers when unnecessary
Quoted 11 lines · Vlastimil Babka (SUSE)
Start using the
slab_needs_objcg()helper to calculateslabobj_extsize. Caches that we know to never need objcg pointers (currentlyKMALLOC_NORMALcaches) will thus stop wasting memory on them when memory allocation profiling is enabled.For things to work properly, we need to also add
slab_needs_objcg()checks tomem_cgroup_from_obj_slab()andmemcg_slab_free_hook(), because whenobj_extsarray exists for a slab only due tomem_allocprofiling, we would otherwise attempt to access a non-existing objcg pointer in that slab.
Quoted 3 lines
The function
__memcg_slab_post_alloc_hook()should not be possible to call for a slab whereslab_needs_objcg()is false, but add aDEBUG_VMcheck there to prevent breaking this assumption accidentally.
Perhaps this part is not necessary as slab_obj_ext_set_objcg() now has a debug warning?
Quoted 4 lines
Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Otherwise LGTM so please feel free to add:
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerRe: [PATCH v3 12/13] mm/slab: stop allocating objcg pointers when unnecessary
Quoted 12 lines · Harry Yoo
Quoted 11 lines · Vlastimil Babka (SUSE)
Start using the
slab_needs_objcg()helper to calculateslabobj_extsize. Caches that we know to never need objcg pointers (currentlyKMALLOC_NORMALcaches) will thus stop wasting memory on them when memory allocation profiling is enabled.For things to work properly, we need to also add
slab_needs_objcg()checks tomem_cgroup_from_obj_slab()andmemcg_slab_free_hook(), because whenobj_extsarray exists for a slab only due tomem_allocprofiling, we would otherwise attempt to access a non-existing objcg pointer in that slab.
Quoted 6 lines
Quoted 3 lines
The function
__memcg_slab_post_alloc_hook()should not be possible to call for a slab whereslab_needs_objcg()is false, but add aDEBUG_VMcheck there to prevent breaking this assumption accidentally.Perhaps this part is not necessary as
slab_obj_ext_set_objcg()now has a debug warning?
Indeed. Removed. Also moved the those debug warnings from 11/13 to here to avoid bisectability issues, as you pointed out offlist.
Quoted 7 lines
Quoted 4 lines
Reviewed-by Hao Li <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>---
Otherwise LGTM so please feel free to add:
Reviewed-by Harry Yoo <[email protected]>
Thanks!
Vlastimil Babka (SUSE)
maintainerPATCH v3 13/13 mm/slab, kfence, memcg: completely remove obj_ext for kfence objects
In reply to Vlastimil Babka (SUSE)We have already disabled memory allocation profiling for objects allocated for KFENCE to avoid complexity. KFENCE allocations are rare and there can be only CONFIG_KFENCE_NUM_OBJECTS (default to 255) outstanding ones at any time, so they are among noise in the profiling stats.
For the same reasons, we can stop memcg_kmem accounting of kfence objects as their memory usage will be negligible wrt any practical memcg limits.
This allows us simplifying the code and getting rid of is_kfence_address() checks in various places, including slab_obj_ext()'s usage of obj_to_index(). Instead we rely on the fact that slab_obj_exts() will now always return 0 for a kfence object's fake slab, which makes those places unreachable.
All we need to do to keep this assumption valid is not to allocate obj_exts for kfence objects, so the checks need to guard alloc_slab_obj_exts() where necessary.
--- mm/kfence/core.c | 15 --------------- mm/kfence/kfence.h | 3 --- mm/memcontrol.c | 12 +++++++----- mm/slab.h | 6 +++++- mm/slub.c | 31 ++++++++++++++++--------------- 5 files changed, 28 insertions(+), 39 deletions(-)
mm/kfence/core.c +0 -15
@@ -636,14 +636,6 @@ static unsigned long kfence_init_pool(void) page =pfn_to_page (start_pfn + i);__SetPageSlab (page); -#ifdef CONFIG_MEMCG -struct slab *slab =page_slab (page); - slab->obj_exts = (unsigned long )&kfence_metadata_init[i /2 -1 ].obj_exts | -MEMCG_DATA_OBJEXTS ; -#ifdef CONFIG_64BIT - slab->obj_exts_needs_objcg =1 ; -#endif -#endif }/*
@@ -707,10 +699,6 @@ static unsigned long kfence_init_pool(void)continue ; page =pfn_to_page (start_pfn + i); -#ifdef CONFIG_MEMCG -struct slab *slab =page_slab (page); - slab->obj_exts =0 ; -#endif __ClearPageSlab (page); }
@@ -1251,9 +1239,6 @@ void __kfence_free(void *addr) {struct kfence_metadata *meta =addr_to_metadata ((unsigned long )addr); -#ifdef CONFIG_MEMCG -KFENCE_WARN_ON (slab_obj_ext_objcg (&meta->obj_exts)); -#endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing * the object, as the object page may be recycled for other-typed
mm/kfence/kfence.h +0 -3
@@ -102,9 +102,6 @@ struct kfence_metadata { struct kfence_track free_track __guarded_by(&lock);/* For updating alloc_covered on frees. */ u32 alloc_stack_hash __guarded_by(&lock); -#ifdef CONFIG_MEMCG -struct slabobj_ext obj_exts; -#endif };#define KFENCE_METADATA_SIZE PAGE_ALIGN(sizeof(struct kfence_metadata) * \
mm/memcontrol.c +7 -5
@@ -3583,13 +3583,15 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, slab =virt_to_slab (p[i]); -if (IS_ENABLED (CONFIG_DEBUG_VM ) &&WARN_ON_ONCE (!slab_needs_objcg (slab))) -continue ; +if (!slab_obj_exts (slab)) { +if (is_kfence_address (p[i])) +continue ; +if (alloc_slab_obj_exts (slab, s, flags, slab_alloc_flags)) +continue ; + } -if (!slab_obj_exts (slab) && -alloc_slab_obj_exts (slab, s, flags, slab_alloc_flags)) { +if (IS_ENABLED (CONFIG_DEBUG_VM ) &&WARN_ON_ONCE (!slab_needs_objcg (slab)))continue ; - }/* * if we fail and size is 1, memcg_alloc_abort_single() will
mm/slab.h +5 -1
@@ -730,7 +730,11 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_exts,VM_WARN_ON_ONCE (obj_exts !=slab_obj_exts (slab)); - index =obj_to_index (s, slab, obj); +/* +* KFENCE objects have NULL obj_exts and thus can't reach this +* and we don't need obj_to_index() +*/ + index =__obj_to_index (s,slab_address (slab), obj);if (!obj_exts_in_object (slab)) stride =slab_obj_ext_size (slab);
mm/slub.c +16 -15
@@ -2064,9 +2064,6 @@ static inline void mark_obj_codetag_empty(const void *obj)struct slab *obj_slab;unsigned long slab_exts; -if (is_kfence_address (obj)) -return ; - obj_slab =virt_to_slab (obj); slab_exts =slab_obj_exts (obj_slab);if (slab_exts) {
@@ -2329,11 +2326,15 @@ static inline unsigned long prepare_slab_obj_exts_hook(struct kmem_cache *s, struct slab *slab, gfp_t flags, unsigned int alloc_flags, void *p) { - if (!slab_obj_exts(slab) && - alloc_slab_obj_exts(slab, s, flags, alloc_flags)) { - pr_warn_once("%s, %s: Failed to create slab extension vector!\n", - __func__, s->name); -return 0 ; + if (!slab_obj_exts(slab)) { + if (is_kfence_address(p)) + return 0; + + if (alloc_slab_obj_exts(slab, s, flags, alloc_flags)) { + pr_warn_once("%s, %s: Failed to create slab extension vector!\n", + __func__, s->name); +return 0 ; + } }return slab_obj_exts (slab);
@@ -2358,9 +2359,6 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,if (alloc_flags &SLAB_ALLOC_NO_RECURSE )return ; -if (is_kfence_address (object)) -return ; - slab =virt_to_slab (object); obj_exts =prepare_slab_obj_exts_hook (s, slab, flags, alloc_flags, object);/*
@@ -2380,7 +2378,13 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags,put_slab_obj_exts (obj_exts); }else { -alloc_tag_set_inaccurate (current->alloc_tag); +/* +* KFENCE allocations are rare and the amount of outstanding +* ones is limited to a small number so it's not worth setting +* tags as inaccurate because of them. +*/ +if (!is_kfence_address (object)) +alloc_tag_set_inaccurate (current->alloc_tag); } }
@@ -2411,9 +2415,6 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **pfor (int i =0 ; i < objects; i++) {struct slabobj_ext *ext; -if (is_kfence_address (p[i])) -continue ; - ext =slab_obj_ext (s, slab, obj_exts, p[i]);alloc_tag_sub (slab_obj_ext_codetag_ref (slab, ext), s->size); }
Signature · 1 line
2.55.0
Re: [PATCH v3 13/13] mm/slab, kfence, memcg: completely remove obj_ext for kfence objects
Quoted 16 lines · Vlastimil Babka (SUSE)
We have already disabled memory allocation profiling for objects allocated for KFENCE to avoid complexity. KFENCE allocations are rare and there can be only
CONFIG_KFENCE_NUM_OBJECTS(default to 255) outstanding ones at any time, so they are among noise in the profiling stats.For the same reasons, we can stop
memcg_kmemaccounting of kfence objects as their memory usage will be negligible wrt any practical memcg limits.This allows us simplifying the code and getting rid of
is_kfence_address()checks in various places, including slab_obj_ext()'s usage ofobj_to_index(). Instead we rely on the fact thatslab_obj_exts()will now always return 0 for a kfence object's fake slab, which makes those places unreachable.
Quoted 6 lines
All we need to do to keep this assumption valid is not to allocate
obj_extsfor kfence objects, so the checks need to guardalloc_slab_obj_exts()where necessary.Suggested-by Harry Yoo <[email protected]>Signed-off-by Vlastimil Babka (SUSE) <[email protected]>
LGTM.
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerQuoted 27 lines · Vlastimil Babka (SUSE)
---
Vlastimil Babka (SUSE) (13):
mm/slab: skip kfence objects in allocation profiling mm/slab: removeobjs_per_slab()
mm: move structslabobj_exttomm/slab.h
mm/slab: makeslab_obj_ext()determine object index mm/slab: abstract slabobj_ext.objcg access
mm/slab: abstract slabobj_ext.ref access
mm/slab: replace slab.stride withobj_exts_in_objectmm/slab: change structslabobj_extto a union
mm/slab: introduceslab_obj_ext_has_codetag()
mm/slab: reduceslabobj_extmemory with allocation profiling disabled mm/slab: addcache_andslab_needs_objcg()helpers mm/slab: stop allocating objcg pointers when unnecessary mm/slab, kfence, memcg: completely removeobj_extfor kfence objectsDocumentation/mm/allocation-profiling.rst | 7 ++ include/linux/memcontrol.h | 13 -- include/linux/slab.h | 3 + mm/kfence/core.c | 12 -- mm/kfence/kfence.h | 3 - mm/kfence/kfence_test.c | 2 +- mm/memcontrol.c | 40 ++++--- mm/slab.h | 189 ++++++++++++++++++++++++------ mm/slab_common.c | 26 +++- mm/slub.c | 184 ++++++++++++++++++----------- 10 files changed, 325 insertions(+), 154 deletions(-)
Quoted 3 lines
---
base-commit:5ff172f6c94d282d83cb88bdfec5f647ad9c6105
change-id: 20260714-b4-objext_split-da82426257d5
Applied (with fixed up 10/13 per sashiko, and 09/13 with a debug check per Harry). To slab.git slab/for-7.3/objext_split and merged to slab/for-next.