mm/memcontrol: batch memcg charging in __memcg_slab_post_alloc_hook
Subsystems (3)
PATCH v2 1/1 mm/memcontrol: batch memcg charging in __memcg_slab_post_alloc_hook
From: teawater <[email protected]>
When kmem_cache_alloc_bulk() allocates multiple objects, the post-alloc hook __memcg_slab_post_alloc_hook() previously charged memcg one object at a time, even though consecutive objects may reside on slabs backed by the same pgdat node.
Batch the memcg charging by scanning ahead from the current position to find a contiguous run of objects whose slabs share the same pgdat, then issue a single __obj_cgroup_charge() / __consume_obj_stock() call for the entire run. The per-object obj_ext assignment loop is preserved as-is since it cannot be further collapsed.
This implements the TODO comment left in commit bc730030f956 ("memcg:
combine slab obj stock charging and accounting").
The existing error-recovery contract is unchanged: if size == 1 then memcg_alloc_abort_single() will free the sole object, and for larger bulk allocations kmem_cache_free_bulk() will uncharge any objects that were already charged before the failure.
Benchmark using kmem_cache_alloc_bulk() with SLAB_ACCOUNT (iters=100000):
bulk=32 before: 215 ns/object after: 174 ns/object (-19%) bulk=1 before: 344 ns/object after: 335 ns/object ( ~)
No measurable regression for bulk=1, as expected.
---
Changelog:
v2:
According to the comments in [1], add code to handle the integer overflow issue.
[1] https://sashiko.dev/#/patchset/20260316084839.1342163-1-hui.zhu%40linux.dev
mm/memcontrol.c | 77 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 58 insertions(+), 19 deletions(-)
mm/memcontrol.c +58 -19
@@ -3448,51 +3448,90 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,return false; } -for (i =0 ; i < size; i++) { +for (i =0 ; i < size; ) {unsigned long obj_exts;struct slabobj_ext *obj_ext;struct obj_stock_pcp *stock; +struct pglist_data *pgdat; +int batch_bytes; +size_t run_len =0 ; +size_t j; +size_t max_size; +bool skip_next = false; slab =virt_to_slab (p[i]);if (!slab_obj_exts (slab) &&alloc_slab_obj_exts (slab, s, flags, false)) { + i++;continue ; } + pgdat =slab_pgdat (slab); + run_len =1 ; + +/* +* The value of batch_bytes must not exceed +* (INT_MAX - PAGE_SIZE) to prevent integer overflow in +* the final accumulation performed by __account_obj_stock(). +*/ + max_size =min ((size_t )((INT_MAX -PAGE_SIZE ) / obj_size), + size); + +for (j = i +1 ; j < max_size; j++) { +struct slab *slab_j =virt_to_slab (p[j]); + +if (slab_pgdat (slab_j) != pgdat) +break ; + +if (!slab_obj_exts (slab_j) && +alloc_slab_obj_exts (slab_j, s, flags, false)) { + skip_next = true; +break ; + } + + run_len++; + } +/* -* if we fail and size is 1, memcg_alloc_abort_single() will +* If we fail and size is 1, memcg_alloc_abort_single() will * just free the object, which is ok as we have not assigned -* objcg to its obj_ext yet -* -* for larger sizes, kmem_cache_free_bulk() will uncharge -* any objects that were already charged and obj_ext assigned +* objcg to its obj_ext yet. * -* TODO: we could batch this until slab_pgdat(slab) changes -* between iterations, with a more complicated undo +* For larger sizes, kmem_cache_free_bulk() will uncharge +* any objects that were already charged and obj_ext assigned. */ + batch_bytes = obj_size * run_len; stock =trylock_stock (); -if (!stock || !__consume_obj_stock (objcg, stock, obj_size)) { +if (!stock || !__consume_obj_stock (objcg, stock, batch_bytes)) {size_t remainder;unlock_stock (stock); -if (__obj_cgroup_charge (objcg, flags, obj_size, &remainder)) +if (__obj_cgroup_charge (objcg, flags, batch_bytes, &remainder))return false; stock =trylock_stock ();if (remainder)__refill_obj_stock (objcg, stock, remainder, false); } -__account_obj_stock (objcg, stock, obj_size, -slab_pgdat (slab),cache_vmstat_idx (s)); +__account_obj_stock (objcg, stock, batch_bytes, + pgdat,cache_vmstat_idx (s));unlock_stock (stock); - obj_exts =slab_obj_exts (slab); -get_slab_obj_exts (obj_exts); - off =obj_to_index (s, slab, p[i]); - obj_ext =slab_obj_ext (slab, obj_exts, off); -obj_cgroup_get (objcg); - obj_ext->objcg = objcg; -put_slab_obj_exts (obj_exts); +for (j =0 ; j < run_len; j++) { + slab =virt_to_slab (p[i + j]); + obj_exts =slab_obj_exts (slab); +get_slab_obj_exts (obj_exts); + off =obj_to_index (s, slab, p[i + j]); + obj_ext =slab_obj_ext (slab, obj_exts, off); +obj_cgroup_get (objcg); + obj_ext->objcg = objcg; +put_slab_obj_exts (obj_exts); + } + +if (skip_next) + i = i + run_len +1 ; +else + i += run_len; }return true;
Signature · 1 line
2.43.0
Andrew Morton
maintainerQuoted 1 line · On Fri, 20 Mar 2026 10:07:45 +0800 Hui Zhu <[email protected]> wrote:
From: teawater <[email protected]>
We do prefer real names in Linux commits, please. Can I rewrite this patch as From: Hui Zhu <[email protected]>?
Quoted 19 lines
When
kmem_cache_alloc_bulk()allocates multiple objects, the post-alloc hook__memcg_slab_post_alloc_hook()previously charged memcg one object at a time, even though consecutive objects may reside on slabs backed by the same pgdat node.Batch the memcg charging by scanning ahead from the current position to find a contiguous run of objects whose slabs share the same pgdat, then issue a single
__obj_cgroup_charge()/__consume_obj_stock()call for the entire run. The per-objectobj_extassignment loop is preserved as-is since it cannot be further collapsed.This implements the TODO comment left in commit
bc730030f956("memcg:
combine slab obj stock charging and accounting").The existing error-recovery contract is unchanged: if size == 1 then
memcg_alloc_abort_single()will free the sole object, and for larger bulk allocationskmem_cache_free_bulk()will uncharge any objects that were already charged before the failure.
Quoted 7 lines
Benchmark using
kmem_cache_alloc_bulk()withSLAB_ACCOUNT(iters=100000):bulk=32 before: 215 ns/object after: 174 ns/object (-19%) bulk=1 before: 344 ns/object after: 335 ns/object ( ~)No measurable regression for bulk=1, as expected.
Could memcg maintainers please review this?
Thanks.
Andrew Morton
maintainerQuoted 19 lines · On Fri, 20 Mar 2026 10:07:45 +0800 Hui Zhu <[email protected]> wrote:
When
kmem_cache_alloc_bulk()allocates multiple objects, the post-alloc hook__memcg_slab_post_alloc_hook()previously charged memcg one object at a time, even though consecutive objects may reside on slabs backed by the same pgdat node.Batch the memcg charging by scanning ahead from the current position to find a contiguous run of objects whose slabs share the same pgdat, then issue a single
__obj_cgroup_charge()/__consume_obj_stock()call for the entire run. The per-objectobj_extassignment loop is preserved as-is since it cannot be further collapsed.This implements the TODO comment left in commit
bc730030f956("memcg:
combine slab obj stock charging and accounting").The existing error-recovery contract is unchanged: if size == 1 then
memcg_alloc_abort_single()will free the sole object, and for larger bulk allocationskmem_cache_free_bulk()will uncharge any objects that were already charged before the failure.
Quoted 7 lines
Benchmark using
kmem_cache_alloc_bulk()withSLAB_ACCOUNT(iters=100000):bulk=32 before: 215 ns/object after: 174 ns/object (-19%) bulk=1 before: 344 ns/object after: 335 ns/object ( ~)No measurable regression for bulk=1, as expected.
I noticed that the AI review of your v1 patch reported a few potential issues:
https://sashiko.dev/#/patchset/[email protected]
Can you please take a look, see if any of this is valid for v2?
Unfortunately the bot wasn't able to check v2 because it couldn't get the patch to apply. I've checked that this patch does apply cleanly to current mm-stable, which is on the bot's try-to-apply list. So if you wish to get checking of the latest patch, please send us a v3 and that will trigger a retry.
Quoted 23 lines · On Fri, 20 Mar 2026 10:07:45 +0800 Hui Zhu <[email protected]> wrote:
Quoted 21 lines
When
kmem_cache_alloc_bulk()allocates multiple objects, the post-alloc
hook__memcg_slab_post_alloc_hook()previously charged memcg one object at a time, even though consecutive objects may reside on slabs backed by the same pgdat node.Batch the memcg charging by scanning ahead from the current position to find a contiguous run of objects whose slabs share the same pgdat, then issue a single
__obj_cgroup_charge()/__consume_obj_stock()call for the entire run. The per-objectobj_extassignment loop is preserved as-is since it cannot be further collapsed.This implements the TODO comment left in commit
bc730030f956("memcg:
combine slab obj stock charging and accounting").The existing error-recovery contract is unchanged: if size == 1 then
memcg_alloc_abort_single()will free the sole object, and for larger bulk allocationskmem_cache_free_bulk()will uncharge any objects that were already charged before the failure.
Quoted 8 lines
Quoted 8 lines
Benchmark using
kmem_cache_alloc_bulk()withSLAB_ACCOUNT(iters=100000):bulk=32 before: 215 ns/object after: 174 ns/object (-19%) bulk=1 before: 344 ns/object after: 335 ns/object ( ~)
No measurable regression for bulk=1, as expected.
Hi Andrew,
Quoted 11 lines
I noticed that the AI review of your v1 patch reported a few potential issues:
https://sashiko.dev/#/patchset/[email protected]Can you please take a look, see if any of this is valid for v2?
Unfortunately the bot wasn't able to check v2 because it couldn't get the patch to apply. I've checked that this patch does apply cleanly to current mm-stable, which is on the bot's try-to-apply list. So if you wish to get checking of the latest patch, please send us a v3 and that will trigger a retry.
I will send a v3 to make sure the patch is OK.
Best,
Hui