mm/memcontrol: avoid false sharing between vmstats and events
Subsystems (3)
PATCH 1/1 mm/memcontrol: avoid false sharing between vmstats and events
Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline.
The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu.
Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration.
The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).
--- include/linux/memcontrol.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/memcontrol.h +7 -2
@@ -268,10 +268,15 @@ struct mem_cgroup {#endif int kmemcg_id; -struct memcg_vmstats_percpu __percpu *vmstats_percpu; -#ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list; +#endif + +/* Keep the hot per-CPU stats pointer away from memory event counters. */ +struct memcg_vmstats_percpu __percpu *vmstats_percpu +____cacheline_aligned_in_smp ; + +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain cgwb_domain;struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT ];#endif
Signature · 1 line
2.53.0-Meta
Andrew Morton
maintainerQuoted 15 lines · On Mon, 17 Aug 2026 03:38:35 -0700 Usama Arif <[email protected]> wrote:
Moving v1 userspace eventfd handling into
memcontrol-v1.cshrank struct vmpressure from 112 to 24 bytes whenCONFIG_MEMCG_V1is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hotvmstats_percpupointer onto the same cacheline.The stress-ng mremap stressor exercises
MADV_PAGEOUTwith swap disabled, generating about 20 millionMEMCG_SWAP_FAILupdates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths loadvmstats_percpu.Move
cgwb_listinto the existing alignment gap and cacheline-alignvmstats_percpu. This separates the pointer from the event counters without increasing the size of structmem_cgroupin the tested configuration.
Quoted 4 lines
The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).
Cool, thanks.
Quoted 1 line
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")
That's in mm-stable so I'll squeeze this fix into next week's 7.3-rc1 merge pile.
Quoted 22 lines · Andrew Morton
Quoted 15 lines · On Mon, 17 Aug 2026 03:38:35 -0700 Usama Arif <[email protected]> wrote:
Moving v1 userspace eventfd handling into
memcontrol-v1.cshrank struct vmpressure from 112 to 24 bytes whenCONFIG_MEMCG_V1is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hotvmstats_percpupointer onto the same cacheline.The stress-ng mremap stressor exercises
MADV_PAGEOUTwith swap disabled, generating about 20 millionMEMCG_SWAP_FAILupdates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths loadvmstats_percpu.Move
cgwb_listinto the existing alignment gap and cacheline-alignvmstats_percpu. This separates the pointer from the event counters without increasing the size of structmem_cgroupin the tested configuration.Quoted 4 lines
The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).
Quoted 6 lines
Cool, thanks.
Quoted 1 line
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")That's in mm-stable so I'll squeeze this fix into next week's 7.3-rc1 merge pile.
Thanks Andrew. We should also add the following tag due to http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation
Quoted 2 lines
Quoted 20 lines · Usama Arif
Moving v1 userspace eventfd handling into
memcontrol-v1.cshrank struct vmpressure from 112 to 24 bytes whenCONFIG_MEMCG_V1is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hotvmstats_percpupointer onto the same cacheline.The stress-ng mremap stressor exercises
MADV_PAGEOUTwith swap disabled, generating about 20 millionMEMCG_SWAP_FAILupdates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths loadvmstats_percpu.Move
cgwb_listinto the existing alignment gap and cacheline-alignvmstats_percpu. This separates the pointer from the event counters without increasing the size of structmem_cgroupin the tested configuration.The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%).
Quoted 5 lines
Fixes ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c")Reported-by kernel test robot <[email protected]>Signed-off-by Usama Arif <[email protected]>Acked-by Shakeel Butt <[email protected]>
Thanks!
Quoted 28 lines
--- include/linux/memcontrol.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)include/linux/memcontrol.h +7
-2@@ -268,10 +268,15 @@ struct mem_cgroup {#endif int kmemcg_id; -struct memcg_vmstats_percpu __percpu *vmstats_percpu; -#ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list; +#endif + +/* Keep the hot per-CPU stats pointer away from memory event counters. */ +struct memcg_vmstats_percpu __percpu *vmstats_percpu +____cacheline_aligned_in_smp ; + +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain cgwb_domain;struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT ];#endif Signature · 1 line
2.53.0-Meta
Signature · 2 lines
Michal Hocko SUSE Labs