memcg: cache obj_stock by memcg, not by objcg pointer

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node, but the per-CPU obj_stock_pcp still keys cached_objcg by pointer. Cross-NUMA workloads now see a drain on every refill and a miss on every consume that targets a sibling per-node objcg of the same memcg, producing the 67.7% stress-ng switch-mq regression reported by LKP.

stock->nr_bytes are fungible across per-node objcgs of one memcg. Treat the cache as keyed by memcg in __consume_obj_stock() and __refill_obj_stock() so siblings share the reserve. Compare via READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so the rcu_read_lock contract on obj_cgroup_memcg() does not apply.

Sharing the reserve without re-caching means bytes funded by one per-node objcg's slow path can be consumed/freed under a different sibling, leaving sub-page residue on whichever sibling was cached at drain time. The pre-existing obj_cgroup_release() path would WARN and silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter. Forward the residue into a per-node objcg of the same (post-reparent) memcg at release time instead, so it can be reconciled later via a refill atomic_xchg or another release; the chain terminates at root_mem_cgroup, whose page_counter has no enforced limit.

Please note that this is temporary fix and will be reverted when per-node kmem accounting is introduced.

Update the stale invariant comment on __account_obj_stock().

Qi Zheng built a specialized reproducer [1] for the corner case and confirmed the fix.

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/oe-lkp/[email protected] open here

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Link https://lore.kernel.org/[email protected]/ [1] open here

Signed-off-by Shakeel Butt <[email protected]>

Debugged-by Qi Zheng <[email protected]>

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Race condition during memcg offlining causes a permanent page counter leak when forwarding sub_bytes.
- [High] Unbounded accumulation of sub_bytes in a non-allocating parent memcg causes a permanent page counter leak and integer overflow. --

commit e90ccbfce3bb9538bf1cdf185a1c10a7269e0939
Author: Shakeel Butt <[email protected]>

memcg: cache obj_stock by memcg, not by objcg pointer

This commit changes obj_stock_pcp caching from pointer-based to memcg-based so sibling per-node objcgs share the reserve. It also forwards sub-page residue into a per-node objcg of the same memcg during release so it can be reconciled later instead of being lost.

Link https://lore.kernel.org/[email protected]/ open here

[1]

Quoted 6 lines

mm/memcontrol.c +0 -0

 index d978e18b9b2d0..a547ec7c42d16 100644
 --- a/mm/memcontrol.c
 +++ b/mm/memcontrol.c

[ ... ]

Quoted 25 lines

mm/memcontrol.c +0 -0

@@ -160,23 +170,33 @@ static void obj_cgroup_release(struct percpu_ref *ref)
 */
 nr_bytes = atomic_read(&objcg->nr_charged_bytes);
 - WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));
 nr_pages = nr_bytes >> PAGE_SHIFT;
 + sub_bytes = nr_bytes & (PAGE_SIZE - 1);
 
 - if (nr_pages) {
 + if (nr_pages || sub_bytes) {
 struct mem_cgroup *memcg;
 
 - memcg = get_mem_cgroup_from_objcg(objcg);
 - mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
 - memcg1_account_kmem(memcg, -nr_pages);
 - if (!mem_cgroup_is_root(memcg))
 - memcg_uncharge(memcg, nr_pages);
 - mem_cgroup_put(memcg);
 + rcu_read_lock();
 + memcg = obj_cgroup_memcg(objcg);
 +
 + if (nr_pages) {
 + mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
 + memcg1_account_kmem(memcg, -nr_pages);

mm/memcontrol.c static void obj_cgroup_release(struct percpu_ref *ref)

Quoted 10 lines

mm/memcontrol.c +0 -0

 + if (!mem_cgroup_is_root(memcg))
 + memcg_uncharge(memcg, nr_pages);
 + }
 +
 + if (sub_bytes && !mem_cgroup_is_root(memcg)) {
 + struct obj_cgroup *fwd;
 +
 + fwd = rcu_dereference(
 + memcg->nodeinfo[numa_node_id()]->objcg);
 + if (fwd)

Could there be a race condition during memcg offlining here that leads to a page counter leak?

If this executes concurrently with the offlining of the memcg, memcg_reparent_objcgs() might have already replaced the nodeinfo objcg pointer with NULL. In this race window, fwd evaluates to NULL, and the sub_bytes would be dropped.

Since they are dropped without being uncharged or forwarded, it seems the page counter would permanently leak these charges.

Quoted 4 lines

+ atomic_add(sub_bytes, &fwd->nr_charged_bytes);

Does this unbounded addition create a risk of integer overflow and page counter leaks if the parent memcg never allocates?

If the parent memcg is used strictly as a hierarchical container (like system.slice) and performs no kernel allocations, its objcg is never cached in any CPU's obj_stock_pcp.

Because __refill_obj_stock() relies on the cache to periodically drain nr_charged_bytes, it would never be invoked for the parent. Could child residues continuously accumulate in the parent's nr_charged_bytes without ever being uncharged, eventually overflowing the 32-bit atomic_t?

Quoted 5 lines

+ }
+ rcu_read_unlock();
}

Signature · 2 lines

Sashiko AI review ·
https://sashiko.dev/#/patchset/[email protected]?part=1

Quoted 33 lines · Shakeel Butt

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node, but the per-CPU obj_stock_pcp still keys cached_objcg by pointer. Cross-NUMA workloads now see a drain on every refill and a miss on every consume that targets a sibling per-node objcg of the same memcg, producing the 67.7% stress-ng switch-mq regression reported by LKP.

stock->nr_bytes are fungible across per-node objcgs of one memcg. Treat the cache as keyed by memcg in __consume_obj_stock() and __refill_obj_stock() so siblings share the reserve. Compare via READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so the rcu_read_lock contract on obj_cgroup_memcg() does not apply.

Sharing the reserve without re-caching means bytes funded by one per-node objcg's slow path can be consumed/freed under a different sibling, leaving sub-page residue on whichever sibling was cached at drain time. The pre-existing obj_cgroup_release() path would WARN and silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter. Forward the residue into a per-node objcg of the same (post-reparent) memcg at release time instead, so it can be reconciled later via a refill atomic_xchg or another release; the chain terminates at root_mem_cgroup, whose page_counter has no enforced limit.

Please note that this is temporary fix and will be reverted when per-node kmem accounting is introduced.

Update the stale invariant comment on __account_obj_stock().

Qi Zheng built a specialized reproducer [1] for the corner case and confirmed the fix.

Quoted 6 lines

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/oe-lkp/[email protected] open here

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Link https://lore.kernel.org/[email protected]/ [1] open here

Signed-off-by Shakeel Butt <[email protected]>

Debugged-by Qi Zheng <[email protected]>

Sashiko's review below and my response.

Quoted 68 lines

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Race condition during memcg offlining causes a permanent page counter leak when forwarding sub_bytes.
- [High] Unbounded accumulation of sub_bytes in a non-allocating parent memcg causes a permanent page counter leak and integer overflow. --

commit e90ccbfce3bb9538bf1cdf185a1c10a7269e0939
Author: Shakeel Butt <[email protected]>

memcg: cache obj_stock by memcg, not by objcg pointer

This commit changes obj_stock_pcp caching from pointer-based to memcg-based so sibling per-node objcgs share the reserve. It also forwards sub-page residue into a per-node objcg of the same memcg during release so it can be reconciled later instead of being lost.

Link https://lore.kernel.org/[email protected]/ open here

[1]

Quoted 6 lines

mm/memcontrol.c +0 -0

  index d978e18b9b2d0..a547ec7c42d16 100644
  --- a/mm/memcontrol.c
  +++ b/mm/memcontrol.c

[ ... ]

Quoted 25 lines

mm/memcontrol.c +0 -0

@@ -160,23 +170,33 @@ static void obj_cgroup_release(struct percpu_ref *ref)
  */
  nr_bytes = atomic_read(&objcg->nr_charged_bytes);
  - WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));
  nr_pages = nr_bytes >> PAGE_SHIFT;
  + sub_bytes = nr_bytes & (PAGE_SIZE - 1);
 
  - if (nr_pages) {
  + if (nr_pages || sub_bytes) {
  struct mem_cgroup *memcg;
 
  - memcg = get_mem_cgroup_from_objcg(objcg);
  - mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
  - memcg1_account_kmem(memcg, -nr_pages);
  - if (!mem_cgroup_is_root(memcg))
  - memcg_uncharge(memcg, nr_pages);
  - mem_cgroup_put(memcg);
  + rcu_read_lock();
  + memcg = obj_cgroup_memcg(objcg);
  +
  + if (nr_pages) {
  + mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
  + memcg1_account_kmem(memcg, -nr_pages);

mm/memcontrol.c static void obj_cgroup_release(struct percpu_ref *ref)

Quoted 10 lines

mm/memcontrol.c +0 -0

  + if (!mem_cgroup_is_root(memcg))
  + memcg_uncharge(memcg, nr_pages);
  + }
  +
  + if (sub_bytes && !mem_cgroup_is_root(memcg)) {
  + struct obj_cgroup *fwd;
  +
  + fwd = rcu_dereference(
  + memcg->nodeinfo[numa_node_id()]->objcg);
  + if (fwd)

Could there be a race condition during memcg offlining here that leads to a page counter leak?

Quoted 7 lines

If this executes concurrently with the offlining of the memcg, memcg_reparent_objcgs() might have already replaced the nodeinfo objcg pointer with NULL. In this race window, fwd evaluates to NULL, and the sub_bytes would be dropped.

Since they are dropped without being uncharged or forwarded, it seems the page counter would permanently leak these charges.

This is not much a concern as it is bounded (i.e. nr_nodes * PAGE_SIZE - 1) and previously for the warning case, we were already dropping the sub-page charges.

Quoted 7 lines

Quoted 4 lines

+ atomic_add(sub_bytes, &fwd->nr_charged_bytes);

Does this unbounded addition create a risk of integer overflow and page counter leaks if the parent memcg never allocates?

Quoted 9 lines

If the parent memcg is used strictly as a hierarchical container (like system.slice) and performs no kernel allocations, its objcg is never cached in any CPU's obj_stock_pcp.

Because __refill_obj_stock() relies on the cache to periodically drain nr_charged_bytes, it would never be invoked for the parent. Could child residues continuously accumulate in the parent's nr_charged_bytes without ever being uncharged, eventually overflowing the 32-bit atomic_t?

This can be a concern for a long running cgroup. However fixing this would add complexity not worth it. This is a temporary fix and will be reverted in newer kernels.

Quoted 10 lines

Quoted 6 lines

+ }
+ rcu_read_unlock();
}

Signature · 2 lines

Sashiko AI review ·
https://sashiko.dev/#/patchset/[email protected]?part=1

Quoted 25 lines · Shakeel Butt

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node, but the per-CPU obj_stock_pcp still keys cached_objcg by pointer. Cross-NUMA workloads now see a drain on every refill and a miss on every consume that targets a sibling per-node objcg of the same memcg, producing the 67.7% stress-ng switch-mq regression reported by LKP.

stock->nr_bytes are fungible across per-node objcgs of one memcg. Treat the cache as keyed by memcg in __consume_obj_stock() and __refill_obj_stock() so siblings share the reserve. Compare via READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so the rcu_read_lock contract on obj_cgroup_memcg() does not apply.

Sharing the reserve without re-caching means bytes funded by one per-node objcg's slow path can be consumed/freed under a different sibling, leaving sub-page residue on whichever sibling was cached at drain time. The pre-existing obj_cgroup_release() path would WARN and silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter. Forward the residue into a per-node objcg of the same (post-reparent) memcg at release time instead, so it can be reconciled later via a refill atomic_xchg or another release; the chain terminates at root_mem_cgroup, whose page_counter has no enforced limit.

Quoted 2 lines

Please note that this is temporary fix and will be reverted when per-node kmem accounting is introduced.

Quoted 26 lines · Shakeel Butt

Quoted 25 lines · Shakeel Butt

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node, but the per-CPU obj_stock_pcp still keys cached_objcg by pointer. Cross-NUMA workloads now see a drain on every refill and a miss on every consume that targets a sibling per-node objcg of the same memcg, producing the 67.7% stress-ng switch-mq regression reported by LKP.

stock->nr_bytes are fungible across per-node objcgs of one memcg. Treat the cache as keyed by memcg in __consume_obj_stock() and __refill_obj_stock() so siblings share the reserve. Compare via READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so the rcu_read_lock contract on obj_cgroup_memcg() does not apply.

Sharing the reserve without re-caching means bytes funded by one per-node objcg's slow path can be consumed/freed under a different sibling, leaving sub-page residue on whichever sibling was cached at drain time. The pre-existing obj_cgroup_release() path would WARN and silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter. Forward the residue into a per-node objcg of the same (post-reparent) memcg at release time instead, so it can be reconciled later via a refill atomic_xchg or another release; the chain terminates at root_mem_cgroup, whose page_counter has no enforced limit.

Quoted 2 lines

Quoted 2 lines

Please note that this is temporary fix and will be reverted when per-node kmem accounting is introduced.

... because once per-node kmem accounting is introduced, "stock->nr_bytes are fungible across per-node objcgs of one memcg" no longer holds?

And the follow-up plain is to revert this and address it with a multi-objcg percpu stock [1], similar to a multi-memcg percpu charge cache we have now, right? (regardless of per-node kmem accounting's progress)

If this temporary fix imposes other potential correctness issues, would it make sense to land [1] in mainline before the next LTS release and skip this temporary fix?

[1] https://lore.kernel.org/oe-lkp/[email protected]

Quoted 29 lines · Harry Yoo

Quoted 26 lines · Shakeel Butt

Quoted 25 lines · Shakeel Butt

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node, but the per-CPU obj_stock_pcp still keys cached_objcg by pointer. Cross-NUMA workloads now see a drain on every refill and a miss on every consume that targets a sibling per-node objcg of the same memcg, producing the 67.7% stress-ng switch-mq regression reported by LKP.

stock->nr_bytes are fungible across per-node objcgs of one memcg. Treat the cache as keyed by memcg in __consume_obj_stock() and __refill_obj_stock() so siblings share the reserve. Compare via READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so the rcu_read_lock contract on obj_cgroup_memcg() does not apply.

Sharing the reserve without re-caching means bytes funded by one per-node objcg's slow path can be consumed/freed under a different sibling, leaving sub-page residue on whichever sibling was cached at drain time. The pre-existing obj_cgroup_release() path would WARN and silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter. Forward the residue into a per-node objcg of the same (post-reparent) memcg at release time instead, so it can be reconciled later via a refill atomic_xchg or another release; the chain terminates at root_mem_cgroup, whose page_counter has no enforced limit.

Quoted 6 lines

Quoted 2 lines

Quoted 2 lines

Please note that this is temporary fix and will be reverted when per-node kmem accounting is introduced.

... because once per-node kmem accounting is introduced, "stock->nr_bytes are fungible across per-node objcgs of one memcg" no longer holds?

Yes

Quoted 1 line

Quoted 4 lines

And the follow-up plain is to revert this and address it with a multi-objcg percpu stock [1], similar to a multi-memcg percpu charge cache we have now, right? (regardless of per-node kmem accounting's progress)

Yes

Quoted 6 lines

If this temporary fix imposes other potential correctness issues, would it make sense to land [1] in mainline before the next LTS release and skip this temporary fix?

[1] https://lore.kernel.org/oe-lkp/[email protected]

The full clean solution might take one more cycle and I think we can not just ignore 67% regression on 7.1.

Quoted 3 lines

The full clean solution might take one more cycle and I think we can not just ignore 67% regression on 7.1.

That is valid point, unfortunately.

One more thing I have to ask... for v7.1, wouldn't it be a safer option to revert the per-node object change and re-introduce it once we have a cleaner solution?

This change was introduced in v5, but the implementation before v4 had been exposed in -next for a while, and I think we don't have enough justification to keep the per-node objcgs change, at least for v7.1, given that we have an unexpected last-minute regression and
correctness concerns (albeit slight).

Andrew, please don't send this patch to Linus until we decide on the option.

Quoted 1 line · On Tue, 19 May 2026 13:11:13 -0700 Shakeel Butt <[email protected]> wrote:

Andrew, please don't send this patch to Linus until we decide on the option.

No probs, I added a note-to-self.

Quoted 5 lines · Andrew Morton

Quoted 1 line · On Tue, 19 May 2026 13:11:13 -0700 Shakeel Butt <[email protected]> wrote:

Andrew, please don't send this patch to Linus until we decide on the option.

No probs, I added a note-to-self.

Here is my suggestion on how to proceed.

1. Let's drop this patch (memcg: cache obj_stock by memcg, not by objcg pointer) from mm-tree.

2. Let's not add anything to 7.1.

3. 7.2+ will have the multi-objcg series [1] and the patches will have fixes tag. If someone uses non-LTS 7.1, they can easily find those patches and can backport them.

Please let me know if there are any concerns.

[1] https://lore.kernel.org/[email protected]/

Quoted 2 lines · Harry Yoo

[...]

Quoted 4 lines

Quoted 3 lines

The full clean solution might take one more cycle and I think we can not just ignore 67% regression on 7.1.

Quoted 5 lines

That is valid point, unfortunately.

One more thing I have to ask... for v7.1, wouldn't it be a safer option to revert the per-node object change and re-introduce it once we have a cleaner solution?

The issue with that revert is that we reintroduce all node lru locking in the objcg reparenting path.

Quoted 5 lines

This change was introduced in v5, but the implementation before v4 had been exposed in -next for a while, and I think we don't have enough justification to keep the per-node objcgs change, at least for v7.1, given that we have an unexpected last-minute regression and
correctness concerns (albeit slight).

I am waiting for Oliver to test the multi-objcg patch I sent. If that also resolves the regression then we have one more option i.e. backport that to 7.1 to fix the regression.

So to summarize, for future kernels we will be having multi-objcg in some form. For 7.1, we have to decide between (1) do nothing (2) this patch or (3) backport the multi-objcg path to 7.1.