memcg: shrink obj_stock_pcp and cache multiple objcgs

The struct obj_stock_pcp stores a pointer to pglist_data for the slab stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The pointer is not strictly required: NODE_DATA() can recover it from the node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE as the "no stats cached" sentinel.

At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Acked-by Muchun Song <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

---

Changes since v1:
- Added tags in the commit message

  mm/memcontrol.c | 26 +++++++++++++++++++-------
  1 file changed, 19 insertions(+), 7 deletions(-)

Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int' which is 4 bytes on 64-bit machines. Switch the field to uint16_t to shrink the per-CPU cache.

The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1 == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds U16_MAX. The accumulator also needs to stay within uint16_t between page-aligned flushes on 64KiB pages where PAGE_SIZE itself is U16_MAX + 1.

Accumulate the new total in an 'unsigned int' local, then:

1. Flush whenever the accumulator would hit U16_MAX. Together with the existing allow_uncharge flush at PAGE_SIZE, this keeps the uint16_t safe on PAGE_SIZE <= 64KiB.

2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and powerpc 44x), push any sub-page remainder above U16_MAX into objcg->nr_charged_bytes via atomic_add before storing back, so the store cannot silently truncate. The PAGE_SHIFT > 16 guard folds the branch out at compile time on smaller page sizes.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

---

Changes since v1:
- Collected tags
- Rearrange fields of obj_stock_pcp (David Laight)
- Fix comparison operator (Harry)

  mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
  1 file changed, 27 insertions(+), 6 deletions(-)

mm/memcontrol.c +3 -2

index d7c162946719..e4f00a8159d5 100644
@@ -2019,8 +2019,8 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
 
  struct obj_stock_pcp {
  	local_trylock_t lock;
-	unsigned int nr_bytes;
  	struct obj_cgroup *cached_objcg;
+	uint16_t nr_bytes;
  	int16_t node_id;
  	int nr_slab_reclaimable_b;
  	int nr_slab_unreclaimable_b;

@@ -3331,6 +3331,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
  			       bool allow_uncharge)
  {
  	unsigned int nr_pages = 0;
+	unsigned int stock_nr_bytes;
 
  	if (!stock) {
  		nr_pages = nr_bytes >> PAGE_SHIFT;

@@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
  		goto out;
  	}
 
+	stock_nr_bytes = stock->nr_bytes;
  	if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
  		drain_obj_stock(stock);
  		obj_cgroup_get(objcg);
-		stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)

mm/memcontrol.c +20 -3

+	if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
+	    stock_nr_bytes > U16_MAX) {
+		nr_pages = stock_nr_bytes >> PAGE_SHIFT;
+		stock_nr_bytes &= (PAGE_SIZE - 1);
+
+		/*
+		 * On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on
+		 * hexagon and powerpc 44x), the sub-page remainder can
+		 * still exceed U16_MAX. Push the excess back to
+		 * objcg->nr_charged_bytes so the store into uint16_t
+		 * cannot silently truncate; folded out at compile time
+		 * on smaller page sizes.
+		 */
+		if (PAGE_SHIFT > 16 && stock_nr_bytes > U16_MAX) {
+			unsigned int kept = stock_nr_bytes & U16_MAX;
 
-	if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
-		nr_pages = stock->nr_bytes >> PAGE_SHIFT;
-		stock->nr_bytes &= (PAGE_SIZE - 1);
+			atomic_add(stock_nr_bytes - kept,
+				   &objcg->nr_charged_bytes);
+			stock_nr_bytes = kept;
+		}
  	}
+	stock->nr_bytes = stock_nr_bytes;
 
  out:
  	if (nr_pages)

Quoted 66 lines · Shakeel Butt

Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int' which is 4 bytes on 64-bit machines. Switch the field to uint16_t to shrink the per-CPU cache.

The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1 == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds U16_MAX. The accumulator also needs to stay within uint16_t between page-aligned flushes on 64KiB pages where PAGE_SIZE itself is U16_MAX + 1.

Accumulate the new total in an 'unsigned int' local, then:

1. Flush whenever the accumulator would hit U16_MAX. Together with the existing allow_uncharge flush at PAGE_SIZE, this keeps the uint16_t safe on PAGE_SIZE <= 64KiB.

2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and powerpc 44x), push any sub-page remainder above U16_MAX into objcg->nr_charged_bytes via atomic_add before storing back, so the store cannot silently truncate. The PAGE_SHIFT > 16 guard folds the branch out at compile time on smaller page sizes.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

---

Changes since v1:
- Collected tags
- Rearrange fields of obj_stock_pcp (David Laight)
- Fix comparison operator (Harry)

  mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
  1 file changed, 27 insertions(+), 6 deletions(-)

mm/memcontrol.c +3 -1

index d7c162946719..e4f00a8159d5 100644
@@ -2019,8 +2019,8 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
  struct obj_stock_pcp {
  	local_trylock_t lock;
-	unsigned int nr_bytes;
  	struct obj_cgroup *cached_objcg;
+	uint16_t nr_bytes;
  	int16_t node_id;
  	int nr_slab_reclaimable_b;
  	int nr_slab_unreclaimable_b;

@@ -3331,6 +3331,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
  			       bool allow_uncharge)
  {
  	unsigned int nr_pages = 0;
+	unsigned int stock_nr_bytes;
  	if (!stock) {
  		nr_pages = nr_bytes >> PAGE_SHIFT;

@@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
  		goto out;
  	}
+	stock_nr_bytes = stock->nr_bytes;
  	if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
  		drain_obj_stock(stock);
  		obj_cgroup_get(objcg);

mm/memcontrol.c static void __refill_obj_stock(struct obj_cgroup *objcg, · line 3353

Quoted 10 lines

mm/memcontrol.c +4 -2

-		stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
+		stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
  				? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
  		WRITE_ONCE(stock->cached_objcg, objcg);
  		allow_uncharge = true;	/* Allow uncharge when objcg changes */
  	}
-	stock->nr_bytes += nr_bytes;
+	stock_nr_bytes += nr_bytes;
+
+	/* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */

                                                           ^

should also be changed to: don't refill > U16_MAX ?

Otherwise:

Acked-by Qi Zheng <[email protected]>

Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int' which is 4 bytes on 64-bit machines. Switch the field to uint16_t to shrink the per-CPU cache.

The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1 == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds U16_MAX. The accumulator also needs to stay within uint16_t between page-aligned flushes on 64KiB pages where PAGE_SIZE itself is U16_MAX + 1.

Accumulate the new total in an 'unsigned int' local, then:

1. Flush whenever the accumulator would hit U16_MAX. Together with the existing allow_uncharge flush at PAGE_SIZE, this keeps the uint16_t safe on PAGE_SIZE <= 64KiB.

2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and powerpc 44x), push any sub-page remainder above U16_MAX into objcg->nr_charged_bytes via atomic_add before storing back, so the store cannot silently truncate. The PAGE_SHIFT > 16 guard folds the branch out at compile time on smaller page sizes.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

Currently struct obj_stock_pcp stores cached slab stats in 'int' which is 4 bytes per counter on 64-bit machines. Switch them to int16_t to shrink the cached metadata.

The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a sufficiently long run of accumulations would overflow the cache. Add an explicit S16_MAX guard before each add: when the next add would push abs(*bytes) past S16_MAX, fold the cached value into @nr and flush directly via mod_objcg_mlstate() before the accumulation.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

---

Changes since v2:
- Collected tags

  mm/memcontrol.c | 16 +++++++++++++---
  1 file changed, 13 insertions(+), 3 deletions(-)

Currently struct obj_stock_pcp stores cached slab stats in 'int' which is 4 bytes per counter on 64-bit machines. Switch them to int16_t to shrink the cached metadata.

The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a sufficiently long run of accumulations would overflow the cache. Add an explicit S16_MAX guard before each add: when the next add would push abs(*bytes) past S16_MAX, fold the cached value into @nr and flush directly via mod_objcg_mlstate() before the accumulation.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

Currently struct obj_stock_pcp stores cached slab stats in 'int' which is 4 bytes per counter on 64-bit machines. Switch them to int16_t to shrink the cached metadata.

The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a sufficiently long run of accumulations would overflow the cache. Add an explicit S16_MAX guard before each add: when the next add would push abs(*bytes) past S16_MAX, fold the cached value into @nr and flush directly via mod_objcg_mlstate() before the accumulation.

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Tested-by kernel test robot <[email protected]>

Signed-off-by Shakeel Butt <[email protected]>

Reviewed-by Harry Yoo (Oracle) <[email protected]>

---

Changes since v2:
- Collected tags

 mm/memcontrol.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

mm/memcontrol.c +3 -3

index e4f00a8159d5..78c02451312b 100644
@@ -2022,8 +2022,8 @@ struct obj_stock_pcp {
 	struct obj_cgroup *cached_objcg;
 	uint16_t nr_bytes;
 	int16_t node_id;
-	int nr_slab_reclaimable_b;
-	int nr_slab_unreclaimable_b;
+	int16_t nr_slab_reclaimable_b;
+	int16_t nr_slab_unreclaimable_b;
 
 	struct work_struct work;
 	unsigned long flags;

@@ -3158,7 +3158,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
 				struct obj_stock_pcp *stock, int nr,
 				struct pglist_data *pgdat, enum node_stat_item idx)
 {
-	int *bytes;
+	int16_t *bytes;
 
 	/*
 	 * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make

@@ -3195,6 +3195,16 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
 
 	bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
 					       : &stock->nr_slab_unreclaimable_b;

mm/memcontrol.c +0 -0

 	/*
 	 * Even for large object >= PAGE_SIZE, the vmstat data will still be
 	 * cached locally at least once before pushing it out.

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node so that reparenting LRU folios can take per-node lru locks. As a side effect, the per-CPU obj_stock_pcp -- which caches exactly one cached_objcg -- thrashes on workloads where threads of the same memcg run on different NUMA nodes. The kernel test robot reported a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.

Mirror the multi-slot pattern already used by memcg_stock_pcp: turn nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all slots on consume/refill/account, prefer empty slots when inserting, and evict a random slot only when full. With multiple slots a CPU can hold the per-node objcg variants of one memcg plus a few siblings without ever forcing a drain.

A single int8_t index records which slot the cached slab stats belong to; the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK = 5 the layout (verified with pahole) is:

offset 0 : lock(1) + index(1) + node_id(2) + slab stats(4) = 8B

 offset 8  : nr_bytes[5]                                     = 10B
 offset 18 : padding                                         = 6B
 offset 24 : cached[5]                                       = 40B

offset 64 : (line 2) work_struct + flags (cold)

so consume_obj_stock, refill_obj_stock and the slab account path each touch exactly one 64-byte cache line on non-debug 64-bit builds.

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/oe-lkp/[email protected] open here

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Signed-off-by Shakeel Butt <[email protected]>

Tested-by kernel test robot <[email protected]>

---

Changes since v1:
- Use round robin for drain

mm/memcontrol.c | 188 ++++++++++++++++++++++++++++++++++--------------

1 file changed, 136 insertions(+), 52 deletions(-)

mm/memcontrol.c +0 -0

index 78c02451312b..ba17633b0bd0 100644
@@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
 	* However, it can be PAGE_SIZE or (x * PAGE_SIZE).

mm/memcontrol.c +0 -0

 	*          92 bytes are added to objcg->nr_charged_bytes

Quoted 51 lines · Shakeel Butt

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node so that reparenting LRU folios can take per-node lru locks. As a side effect, the per-CPU obj_stock_pcp -- which caches exactly one cached_objcg -- thrashes on workloads where threads of the same memcg run on different NUMA nodes. The kernel test robot reported a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.

Mirror the multi-slot pattern already used by memcg_stock_pcp: turn nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all slots on consume/refill/account, prefer empty slots when inserting, and evict a random slot only when full. With multiple slots a CPU can hold the per-node objcg variants of one memcg plus a few siblings without ever forcing a drain.

A single int8_t index records which slot the cached slab stats belong to; the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK = 5 the layout (verified with pahole) is:

offset 0 : lock(1) + index(1) + node_id(2) + slab stats(4) = 8B

 offset 8  : nr_bytes[5]                                     = 10B
 offset 18 : padding                                         = 6B
 offset 24 : cached[5]                                       = 40B

offset 64 : (line 2) work_struct + flags (cold)

so consume_obj_stock, refill_obj_stock and the slab account path each touch exactly one 64-byte cache line on non-debug 64-bit builds.

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/oe-lkp/[email protected] open here

Fixes 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")

Signed-off-by Shakeel Butt <[email protected]>

Tested-by kernel test robot <[email protected]>

---

Changes since v1:
- Use round robin for drain

mm/memcontrol.c | 188 ++++++++++++++++++++++++++++++++++--------------

1 file changed, 136 insertions(+), 52 deletions(-)

mm/memcontrol.c +0 -0

index 78c02451312b..ba17633b0bd0 100644
@@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
 	* However, it can be PAGE_SIZE or (x * PAGE_SIZE).

mm/memcontrol.c static void obj_cgroup_release(struct percpu_ref *ref) · line 159

Quoted 10 lines

mm/memcontrol.c +1 -1

 	*
 	* The following sequence can lead to it:
- 	* 1) CPU0: objcg == stock->cached_objcg
+ 	* 1) CPU0: objcg cached in one of stock->cached[i]
 	* 2) CPU1: we do a small allocation (e.g. 92 bytes),
 	*          PAGE_SIZE bytes are charged
 	* 3) CPU1: a process from another memcg is allocating something,
 	*          the stock if flushed,
 	*          objcg->nr_charged_bytes = PAGE_SIZE - 92
 	* 5) CPU0: we do release this object,

           ^
           4

Since you're already modifying the comments in this section, would you mind fixing the numbering as well? I noticed that the sequence was wrong a while back :)

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node so that reparenting LRU folios can take per-node lru locks. As a side effect, the per-CPU obj_stock_pcp -- which caches a single cached_objcg pointer -- thrashes on workloads where threads of the same memcg run on different NUMA nodes. The kernel test robot reported a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.

Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg pointer") landed as a temporary fix by treating sibling per-node objcgs as equivalent for the cache lookup, intended to be reverted once per-node kmem accounting is introduced. This series takes a more general approach: cache multiple objcgs per CPU using the multi-slot pattern memcg_stock_pcp already uses, so the per-node objcg variants of one memcg can all coexist in the stock without ever forcing a drain. The temporary fix can then be reverted.

To avoid increasing the per-CPU cache footprint, the first three patches shrink the existing single-slot obj_stock_pcp fields. The final patch converts cached_objcg and nr_bytes into NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire consume/refill/account hot path fits within a single 64-byte cache line on non-debug 64-bit builds (verified with pahole).

Quoted 17 lines · On Thu, 21 May 2026 18:19:04 -0700 Shakeel Butt <[email protected]> wrote:

Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") split a memcg's single obj_cgroup into one per NUMA node so that reparenting LRU folios can take per-node lru locks. As a side effect, the per-CPU obj_stock_pcp -- which caches a single cached_objcg pointer -- thrashes on workloads where threads of the same memcg run on different NUMA nodes. The kernel test robot reported a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.

Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg pointer") landed as a temporary fix by treating sibling per-node objcgs as equivalent for the cache lookup, intended to be reverted once per-node kmem accounting is introduced. This series takes a more general approach: cache multiple objcgs per CPU using the multi-slot pattern memcg_stock_pcp already uses, so the per-node objcg variants of one memcg can all coexist in the stock without ever forcing a drain. The temporary fix can then be reverted.

Quoted 6 lines

To avoid increasing the per-CPU cache footprint, the first three patches shrink the existing single-slot obj_stock_pcp fields. The final patch converts cached_objcg and nr_bytes into NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire consume/refill/account hot path fits within a single 64-byte cache line on non-debug 64-bit builds (verified with pahole).

Thanks, I added this to mm.git's mm-new branch, along with a couple of possible todo notes from the review.

Sashiko asked a thing:

	https://sashiko.dev/#/patchset/[email protected]

Did you already see this? The footers there indicate that an email was sent out but I don't know if it works?