Skip to contentSkip to conversation

mm/memcontrol: avoid false sharing between vmstats and events

PATCH Applied Usama Arif 4 participants 4 messages Aug 17 – Aug 18

Subsystems (3)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH 1/1 mm/memcontrol: avoid false sharing between vmstats and events

Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline.

The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu.

Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration.

The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).

Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")
---
 include/linux/memcontrol.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/memcontrol.h +7 -2
index e78bc98ab229b..215e2e87f42b2 100644
@@ -268,10 +268,15 @@ struct mem_cgroup {
 #endif
 	int kmemcg_id;
 
-	struct memcg_vmstats_percpu __percpu *vmstats_percpu;
-
 #ifdef CONFIG_CGROUP_WRITEBACK
 	struct list_head cgwb_list;
+#endif
+
+	/* Keep the hot per-CPU stats pointer away from memory event counters. */
+	struct memcg_vmstats_percpu __percpu *vmstats_percpu
+		____cacheline_aligned_in_smp;
+
+#ifdef CONFIG_CGROUP_WRITEBACK
 	struct wb_domain cgwb_domain;
 	struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
 #endif
Signature · 1 line
2.53.0-Meta
maintainer
#1 lore raw
Quoted 15 lines · On Mon, 17 Aug 2026 03:38:35 -0700 Usama Arif <[email protected]> wrote:

Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline.

The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu.

Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration.

Quoted 4 lines

The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).

Cool, thanks.

Quoted 1 line
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")

That's in mm-stable so I'll squeeze this fix into next week's 7.3-rc1 merge pile.

maintainer Tested-by kernel test robot
#2 lore raw
Quoted 22 lines · Andrew Morton
Quoted 15 lines · On Mon, 17 Aug 2026 03:38:35 -0700 Usama Arif <[email protected]> wrote:

Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline.

The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu.

Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration.

Quoted 4 lines

The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).

Quoted 6 lines

Cool, thanks.

Quoted 1 line
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")

That's in mm-stable so I'll squeeze this fix into next week's 7.3-rc1 merge pile.

Thanks Andrew. We should also add the following tag due to http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation

Quoted 2 lines
Acked-by on v1 1/1
#3 lore raw
In reply to Usama Arif
Quoted 20 lines · Usama Arif

Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline.

The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu.

Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration.

The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).

Quoted 5 lines
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")

Thanks!

Quoted 28 lines
---
 include/linux/memcontrol.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/memcontrol.h +7 -2
index e78bc98ab229b..215e2e87f42b2 100644
@@ -268,10 +268,15 @@ struct mem_cgroup {
 #endif
 	int kmemcg_id;
 
-	struct memcg_vmstats_percpu __percpu *vmstats_percpu;
-
 #ifdef CONFIG_CGROUP_WRITEBACK
 	struct list_head cgwb_list;
+#endif
+
+	/* Keep the hot per-CPU stats pointer away from memory event counters. */
+	struct memcg_vmstats_percpu __percpu *vmstats_percpu
+		____cacheline_aligned_in_smp;
+
+#ifdef CONFIG_CGROUP_WRITEBACK
 	struct wb_domain cgwb_domain;
 	struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
 #endif
Signature · 1 line
2.53.0-Meta
Signature · 2 lines
Michal Hocko
SUSE Labs