mm/slab, alloc_tag: reduce obj_ext memory waste
Get series with b4
b4 am [email protected] change-id 20260714-b4-objext_split-da82426257d5Versions newest first
- v3 Latest13 patches 25 repliesIn review
- v213 patches 32 repliesSuperseded
- v1 RFC12 patches 30 repliesSuperseded
Changes in v3 from the cover letter
- Review tags from Harry, Hao, Suren - thanks!
- Patch "skip handle_failed_objexts_alloc() with profiling disabled"
removed, the check is done as part of "reduce slabobj_ext memory with allocation profiling disabled" using slab_obj_ext_has_codetag()
(Harry).
- Move kfence checks earlier in mark_obj_codetag_empty() in Patch 1
(Harry).
- Remove SLAB_MAY_ACCOUNT also for KMALLOC_NO_OBJ_EXT caches and more
precise description in Patch 11 (Hao)
- Add a new cleanup patch 13 that removes also memcg accounting from
kfence allocations, as suggested by Harry.
- Link to v2: https://patch.msgid.link/[email protected]
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/13 | mm/slab: skip kfence objects in allocation profilingDocumentation/ mm/ | +16 | 2 replies | |
| 2/13 | mm/slab: remove objs_per_slab()mm/ mm/ mm/ | +2 | 1 reply | |
| 3/13 | mm: move struct slabobj_ext to mm/slab.hinclude/ mm/ | +13 | 0 replies | |
| 4/13 | mm/slab: make slab_obj_ext() determine object indexmm/ mm/ mm/ | +21 | 0 replies | |
| 5/13 | mm/slab: abstract slabobj_ext.objcg accessmm/ mm/ mm/ +1 more | +24 | 1 reply | |
| 6/13 | mm/slab: abstract slabobj_ext.ref accessmm/ mm/ | +37 | 5 replies | |
| 7/13 | mm/slab: replace slab.stride with obj_exts_in_objectmm/ mm/ | +41 | 2 replies | |
| 8/13 | mm/slab: change struct slabobj_ext to a unionmm/ mm/ | +50 | 4 replies | |
| 9/13 | mm/slab: introduce slab_obj_ext_has_codetag()mm/ mm/ | +46 | 2 replies | |
| 10/13 | mm/slab: reduce slabobj_ext memory with allocation profiling disabledmm/ mm/ | +4 | 3 replies | |
| 11/13 | mm/slab: add cache_ and slab_needs_objcg() helpersinclude/ mm/ mm/ +2 more | +61 | 1 reply | |
| 12/13 | mm/slab: stop allocating objcg pointers when unnecessarymm/ mm/ mm/ | +20 | 2 replies | |
| 13/13 | mm/slab, kfence, memcg: completely remove obj_ext for kfence objectsmm/ mm/ mm/ +2 more | +28 | 1 reply |
Reviews trailers per patch
All 13 patches have a Reviewed-by or Acked-by.
| Patch | Suren Baghdasaryan | Harry Yoo (Oracle) | Hao Li |
|---|---|---|---|
| 1/13 | R-b | ||
| 2/13 | R-b R-b | R-b | R-b |
| 3/13 | R-b | R-b | R-b |
| 4/13 | R-b | R-b | R-b |
| 5/13 | R-b | R-b | R-b |
| 6/13 | R-b R-b | R-b R-b | R-b R-b |
| 7/13 | R-b | R-b | R-b |
| 8/13 | R-b | R-b | R-b |
| 9/13 | R-b | R-b | R-b |
| 10/13 | R-b R-b | R-b | R-b |
| 11/13 | R-b | R-b | |
| 12/13 | R-b | R-b | |
| 13/13 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH v3 00/13] mm/slab, alloc_tag: reduce obj_ext memory waste
The recent fixes for objext array handling inspired me to look into this finally. It's been bothering me that the memory usage of struct slabobj_ext depend only on config options and not whether the fields are actually used. So with both CONFIG_MEMCG=y and CONFIG_MEM_ALLOC_PROFILING=y there is always objcg field and codetag_ref field. And thus:
1) Having memory allocation profiling config-enabled but not boot-enabled means wasted memory on unused codetag_refs. This makes it less suitable for a general distro config and the page allocator side doesn't suffer from this, only slab and percpu.
2) Complementary, with memory allocation profiling enabled, there are caches/slabs that don't need the objcg field, so memory is wasted on those.
This series should solve the point 1) fully for slab, pcpuobj_ext handling can be perhaps improved similarly, haven't looked into that.
For 2) it avoids allocating objcg fields for KMALLOC_NORMAL and KMALLOC_NO_OBJ_EXT caches where we know they are not necessary because kmalloc() with __GFP_ACCOUNT will pick a KMALLOC_CGROUP type (except with SLUB_TINY).
The named kmem_caches are tricky. They can be created with SLAB_ACCOUNT and then we know objcg fields are always needed. But also they can be created without SLAB_ACCOUNT and then some allocations have __GFP_ACCOUNT and some not and we don't know that in advance.
This series introduces a SLAB_MAY_ACCOUNT flag that's currently internal only and is applied to all caches (unless kmem accounting is disabled) except KMALLOC_NORMAL (unless that aliases KMALLOC_RECLAIM) and KMALLOC_NO_OBJ_EXT.
As a followup we can make SLAB_MAY_ACCOUNT explicit and add it to to caches where we know __GFP_ACCOUNT is used. Then we could only honour __GFP_ACCOUNT for those, while warning for an unexpected usage elsewhere.
Based on slab/for-next-fixes
Git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/objext_split
Moderately tested, nothing seems to crash.
To check for regressions, I forward-ported a microbenchmark hacked into slub_kunit that was used to evaluate sheaves. You can find it here along with the series:
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=objext_split-v3-bench
Tried 3 scenarios, MEMCG and KFENCE were always enabled:
- CONFIG_MEM_ALLOC_PROFILING=n
- CONFIG_MEM_ALLOC_PROFILING=y but _ENABLED_BY_DEFAULT=n
- same but booted with sysctl.vm.mem_profiling=1
The results are quite noisy, but no regression was apparent, except perhaps few percents for the last case. I don't expect it will be visible in any real workloads.
---
Changes in v3:
- Review tags from Harry, Hao, Suren - thanks!
- Patch "skip handle_failed_objexts_alloc() with profiling disabled"
removed, the check is done as part of "reduce slabobj_ext memory with allocation profiling disabled" using slab_obj_ext_has_codetag()
(Harry).
- Move kfence checks earlier in mark_obj_codetag_empty() in Patch 1
(Harry).
- Remove SLAB_MAY_ACCOUNT also for KMALLOC_NO_OBJ_EXT caches and more
precise description in Patch 11 (Hao)
- Add a new cleanup patch 13 that removes also memcg accounting from
kfence allocations, as suggested by Harry.
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Apply Suren's R-b:, thanks!
- Expanded explanation about no longer accounting KFENCE objects (Suren)
- Also skip them in in mark_obj_codetag_empty() (sashiko)
- Update slab_obj_ext() comments (sashiko)
- Separate slab_obj_ext_objcg() and slab_obj_ext_set_objcg() (Suren)
- Add SLAB_MAY_ACCOUNT internal flag instead of relying on is_kmalloc_normal(); also handle mem_cgroup_kmem_disabled()
- Also fix the SLUB_TINY kmalloc aliasing handling, per Harry.
- Link to v1: https://patch.msgid.link/[email protected]
---
Vlastimil Babka (SUSE) (13):
mm/slab: skip kfence objects in allocation profiling mm/slab: remove objs_per_slab()
mm: move struct slabobj_ext to mm/slab.h
mm/slab: make slab_obj_ext() determine object index mm/slab: abstract slabobj_ext.objcg access
mm/slab: abstract slabobj_ext.ref access
mm/slab: replace slab.stride with obj_exts_in_object mm/slab: change struct slabobj_ext to a union
mm/slab: introduce slab_obj_ext_has_codetag()
mm/slab: reduce slabobj_ext memory with allocation profiling disabled mm/slab: add cache_ and slab_needs_objcg() helpers mm/slab: stop allocating objcg pointers when unnecessary mm/slab, kfence, memcg: completely remove obj_ext for kfence objects
Documentation/mm/allocation-profiling.rst | 7 ++ include/linux/memcontrol.h | 13 -- include/linux/slab.h | 3 + mm/kfence/core.c | 12 -- mm/kfence/kfence.h | 3 - mm/kfence/kfence_test.c | 2 +- mm/memcontrol.c | 40 ++++--- mm/slab.h | 189 ++++++++++++++++++++++++------ mm/slab_common.c | 26 +++- mm/slub.c | 184 ++++++++++++++++++----------- 10 files changed, 325 insertions(+), 154 deletions(-)
---
base-commit: 5ff172f6c94d282d83cb88bdfec5f647ad9c6105
change-id: 20260714-b4-objext_split-da82426257d5