mm/memcontrol: fix flushing per-cpu counters in memcg_hotplug_cpu_dead
Subsystems (2)
PATCH 1/2 mm/memcontrol: fix flushing per-cpu counters in memcg_hotplug_cpu_dead
Use correct memcg pointer.
--- mm/memcontrol.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/memcontrol.c +2 -2
@@ -2259,7 +2259,7 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) x =this_cpu_xchg (memcg->vmstats_percpu->stat[i],0 );if (x)for (mi = memcg; mi; mi =parent_mem_cgroup (mi)) -atomic_long_add (x, &memcg->vmstats[i]); +atomic_long_add (x, &mi->vmstats[i]);if (i >=NR_VM_NODE_STAT_ITEMS )continue ;
@@ -2282,7 +2282,7 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) x =this_cpu_xchg (memcg->vmstats_percpu->events[i],0 );if (x)for (mi = memcg; mi; mi =parent_mem_cgroup (mi)) -atomic_long_add (x, &memcg->vmevents[i]); +atomic_long_add (x, &mi->vmevents[i]); } }
PATCH 2/2 mm/memcontrol: split local and nested atomic vmstats/vmevents counters
This is alternative solution for problem addressed in commit 815744d75152 ("mm: memcontrol: don't batch updates of local VM stats and events").
Instead of adding second set of percpu counters which wastes memory and slows down showing statistics in cgroup-v1 this patch use two arrays of atomic counters: local and nested statistics.
Then update has the same amount of atomic operations: local update and one nested for each parent cgroup. Readers of hierarchical statistics have to sum two atomics which isn't a big deal.
All updates are still batched using one set of percpu counters.
--- include/linux/memcontrol.h | 19 +++++++---------- mm/memcontrol.c | 48 +++++++++++++++++++------------------------- 2 files changed, 29 insertions(+), 38 deletions(-)
include/linux/memcontrol.h +8 -11
@@ -269,16 +269,16 @@ struct mem_cgroup {atomic_t moving_account;struct task_struct *move_lock_task; -/* Legacy local VM stats and events */ -struct memcg_vmstats_percpu __percpu *vmstats_local; -/* Subtree VM stats and events (batched updates) */ struct memcg_vmstats_percpu __percpu *vmstats_percpu;MEMCG_PADDING (_pad2_); -atomic_long_t vmstats[MEMCG_NR_STAT ]; -atomic_long_t vmevents[NR_VM_EVENT_ITEMS ]; +atomic_long_t vmstats_local[MEMCG_NR_STAT ]; +atomic_long_t vmstats_nested[MEMCG_NR_STAT ]; + +atomic_long_t vmevents_local[NR_VM_EVENT_ITEMS ]; +atomic_long_t vmevents_nested[NR_VM_EVENT_ITEMS ];/* memory.events */ atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS ];
@@ -557,7 +557,8 @@ void unlock_page_memcg(struct page *page);*/ static inline unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx) { -long x =atomic_long_read (&memcg->vmstats[idx]); +long x =atomic_long_read (&memcg->vmstats_local[idx]) + +atomic_long_read (&memcg->vmstats_nested[idx]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;
@@ -572,11 +573,7 @@ static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx)static inline unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx) { -long x =0 ; -int cpu; - -for_each_possible_cpu (cpu) - x +=per_cpu (memcg->vmstats_local->stat[idx], cpu); +long x =atomic_long_read (&memcg->vmstats_local[idx]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;
mm/memcontrol.c +21 -27
108 lines not on this page · raw message
Johannes Weiner
maintainerRe: [PATCH 2/2] mm/memcontrol: split local and nested atomic vmstats/vmevents counters
Quoted 7 lines · Konstantin Khlebnikov
This is alternative solution for problem addressed in commit
815744d75152("mm: memcontrol: don't batch updates of local VM stats and events").Instead of adding second set of percpu counters which wastes memory and slows down showing statistics in cgroup-v1 this patch use two arrays of atomic counters: local and nested statistics.
Quoted 7 lines
Then update has the same amount of atomic operations: local update and one nested for each parent cgroup. Readers of hierarchical statistics have to sum two atomics which isn't a big deal.
All updates are still batched using one set of percpu counters.
Signed-off-by Konstantin Khlebnikov <[email protected]>
Yeah that looks better. Note that it was never about the atomics, though, but rather the number of cachelines dirtied. Your patch should solve this problem as well, but it might be a good idea to run will-it-scale on it to make sure the struct layout is still fine.
Re: [PATCH 2/2] mm/memcontrol: split local and nested atomic vmstats/vmevents counters
Quoted 12 lines · Johannes Weiner
Quoted 11 lines · Konstantin Khlebnikov
This is alternative solution for problem addressed in commit
815744d75152("mm: memcontrol: don't batch updates of local VM stats and events").Instead of adding second set of percpu counters which wastes memory and slows down showing statistics in cgroup-v1 this patch use two arrays of atomic counters: local and nested statistics.
Then update has the same amount of atomic operations: local update and one nested for each parent cgroup. Readers of hierarchical statistics have to sum two atomics which isn't a big deal.
Quoted 9 lines
Quoted 3 lines
All updates are still batched using one set of percpu counters.
Signed-off-by Konstantin Khlebnikov <[email protected]>Yeah that looks better. Note that it was never about the atomics, though, but rather the number of cachelines dirtied. Your patch should solve this problem as well, but it might be a good idea to run will-it-scale on it to make sure the struct layout is still fine.
Looks like this patch shows 2% regression for 24 core 2 numa node machine I have. Compete remove of these counters gives 2% boost. Also I cannot reproduce regression fixed by commit 815744d75152 - revert have no effect.
So, feel free to ignore second patch. I'll play with this a little more.
Maybe atomic per-numa counters could give nice balance between scalability add overhead. Ideally this memory could be mapped in per-cpu manner to give atomic access via fs/gs.
Quoted 4 lines · Konstantin Khlebnikov
Use correct memcg pointer.
Fixes 42a300353577 ("mm: memcontrol: fix recursive statistics correctness & scalabilty")Signed-off-by Konstantin Khlebnikov <[email protected]>
Oops, nice catch.