memcg: optimize memcg_rstat_updated
Subsystems (3)
Shakeel Butt
maintainerPATCH v2 1/1 memcg: optimize memcg_rstat_updated
Currently the kernel maintains the stats updates per-memcg which is needed to implement stats flushing threshold. On the update side, the update is added to the per-cpu per-memcg update of the given memcg and all of its ancestors. However when the given memcg has passed the flushing threshold, all of its ancestors should have passed the threshold as well. There is no need to traverse up the memcg tree to maintain the stats updates.
Perf profile collected from our fleet shows that memcg_rstat_updated is one of the most expensive memcg function i.e. a lot of cumulative CPU is being spent on it. So, even small micro optimizations matter a lot. This patch is microbenchmarked with multiple instances of netperf on a single machine with locally running netserver and we see couple of percentage of improvement.
---
Changes since v1:
- Fix the condition (Longman)
- Ran netperf
mm/memcontrol.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
mm/memcontrol.c +9 -7
@@ -585,18 +585,20 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val)cgroup_rstat_updated (memcg->css.cgroup, cpu); statc =this_cpu_ptr (memcg->vmstats_percpu);for (; statc; statc = statc->parent) { +/* +* If @memcg is already flushable then all its ancestors are +* flushable as well and also there is no need to increase +* stats_updates. +*/ +if (memcg_vmstats_needs_flush (statc->vmstats)) +break ; + stats_updates =READ_ONCE (statc->stats_updates) +abs (val);WRITE_ONCE (statc->stats_updates, stats_updates);if (stats_updates <MEMCG_CHARGE_BATCH )continue ; -/* -* If @memcg is already flush-able, increasing stats_updates is -* redundant. Avoid the overhead of the atomic update. -*/ -if (!memcg_vmstats_needs_flush (statc->vmstats)) -atomic64_add (stats_updates, - &statc->vmstats->stats_updates); +atomic64_add (stats_updates, &statc->vmstats->stats_updates);WRITE_ONCE (statc->stats_updates,0 ); } }
Signature · 1 line
2.47.1
Quoted 8 lines · Shakeel Butt
Currently the kernel maintains the stats updates per-memcg which is needed to implement stats flushing threshold. On the update side, the update is added to the per-cpu per-memcg update of the given memcg and all of its ancestors. However when the given memcg has passed the flushing threshold, all of its ancestors should have passed the threshold as well. There is no need to traverse up the memcg tree to maintain the stats updates.
Quoted 8 lines
Perf profile collected from our fleet shows that
memcg_rstat_updatedis one of the most expensive memcg function i.e. a lot of cumulative CPU is being spent on it. So, even small micro optimizations matter a lot. This patch is microbenchmarked with multiple instances of netperf on a single machine with locally running netserver and we see couple of percentage of improvement.Signed-off-by Shakeel Butt <[email protected]>
Thanks!
Quoted 7 lines · Shakeel Butt
Currently the kernel maintains the stats updates per-memcg which is needed to implement stats flushing threshold. On the update side, the update is added to the per-cpu per-memcg update of the given memcg and all of its ancestors. However when the given memcg has passed the flushing threshold, all of its ancestors should have passed the threshold as well. There is no need to traverse up the memcg tree to maintain the stats updates.
Nice. We also skip updating the per-CPU stats updates counter, which is unnecessary in this case as well.
Quoted 9 lines
Perf profile collected from our fleet shows that
memcg_rstat_updatedis one of the most expensive memcg function i.e. a lot of cumulative CPU is being spent on it. So, even small micro optimizations matter a lot. This patch is microbenchmarked with multiple instances of netperf on a single machine with locally running netserver and we see couple of percentage of improvement.Signed-off-by Shakeel Butt <[email protected]>