mm: workingset: make shadow node shrinker memcg aware
Subsystems (6)
v2 cover mm: workingset: make shadow node shrinker memcg aware
Hi,
Workingset code was recently made memcg aware, but shadow node shrinker is still global. As a result, one small cgroup can consume all memory available for shadow nodes, possibly hurting other cgroups by reclaiming their shadow nodes, even though reclaim distances stored in its shadow nodes have no effect. To avoid this, we need to make shadow node shrinker memcg aware.
The actual work is done in patch 6 of the series. Patches 1 and 2 prepare memcg/shrinker infrastructure for the change. Patch 3 is just a collateral cleanup. Patch 4 makes radix_tree_node accounted, which is necessary for making shadow node shrinker memcg aware. Patch 5 reduces shadow nodes overhead in case workload mostly uses anonymous pages.
Changes in v2:
- Use (FILE_ACTIVE+FILE_INACTIVE)/2 instead of FILE_ACTIVE for maximal refault distance (Johannes).
Thanks,
Vladimir Davydov (6):
mm: memcontrol: enable kmem accounting for all cgroups in the legacy
hierarchy
mm: vmscan: pass root_mem_cgroup instead of NULL to memcg aware
shrinker
mm: memcontrol: zap memcg_kmem_online helper
radix-tree: account radix_tree_node to memory cgroup
mm: workingset: size shadow nodes lru basing on file cache size
mm: workingset: make shadow node shrinker memcg aware
include/linux/memcontrol.h | 20 +++++++++---------- lib/radix-tree.c | 16 +++++++++++++--- mm/memcontrol.c | 48 ++++++++-------------------------------------- mm/slab_common.c | 2 +- mm/vmscan.c | 15 ++++++++++----- mm/workingset.c | 10 ++++++++-- 6 files changed, 50 insertions(+), 61 deletions(-)
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/6 mm: memcontrol: enable kmem accounting for all cgroups in the legacy hierarchy
Currently, in the legacy hierarchy kmem accounting is off for all cgroups by default and must be enabled explicitly by writing something to memory.kmem.limit_in_bytes. Since we don't support reclaim on hitting kmem limit, nor do we have any plans to implement it, this is likely to be -1, just to enable kmem accounting and limit kernel memory consumption by the memory.limit_in_bytes along with user memory.
This user API was introduced when the implementation of kmem accounting lacked slab shrinker support and hence was useless in practice. Things have changed since then - slab shrinkers were made memcg aware, the accounting overhead seems to be negligible, and a failure to charge a kmem allocation should not have critical consequences, because we only account those kernel objects that should be safe to fail. That's why kmem accounting is enabled by default for all cgroups in the default hierarchy, which will eventually replace the legacy one.
The ability to enable kmem accounting for some cgroups while keeping it disabled for others is getting difficult to maintain. E.g. to make shadow node shrinker memcg aware (see mm/workingset.c), we need to know the relationship between the number of shadow nodes allocated for a cgroup and the size of its lru list. If kmem accounting is enabled for all cgroups there is no problem, but what should we do if kmem accounting is enabled only for half of cgroups? We've no other choice but use global lru stats while scanning root cgroup's shadow nodes, but that would be wrong if kmem accounting was enabled for all cgroups (which is the case if the unified hierarchy is used), in which case we should use lru stats of the root cgroup's lruvec.
That being said, let's enable kmem accounting for all memory cgroups by default. If one finds it unstable or too costly, it can always be disabled system-wide by passing cgroup.memory=nokmem to the kernel at boot time.
--- mm/memcontrol.c | 41 +++++------------------------------------ 1 file changed, 5 insertions(+), 36 deletions(-)
mm/memcontrol.c +5 -36
@@ -2824,6 +2824,9 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) {int memcg_id; +if (cgroup_memory_nokmem) +return 0 ; +BUG_ON (memcg->kmemcg_id >=0 );BUG_ON (memcg->kmem_state);
@@ -2844,24 +2847,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg)return 0 ; } -static int memcg_propagate_kmem (struct mem_cgroup *parent, -struct mem_cgroup *memcg) -{ -int ret =0 ; - -mutex_lock (&memcg_limit_mutex); -/* -* If the parent cgroup is not kmem-online now, it cannot be -* onlined after this point, because it has at least one child -* already. -*/ -if (memcg_kmem_online (parent) || - (cgroup_subsys_on_dfl (memory_cgrp_subsys) && !cgroup_memory_nokmem)) - ret =memcg_online_kmem (memcg); -mutex_unlock (&memcg_limit_mutex); -return ret; -} -static void memcg_offline_kmem (struct mem_cgroup *memcg) {struct cgroup_subsys_state *css;
@@ -2920,10 +2905,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) } }#else -static int memcg_propagate_kmem (struct mem_cgroup *parent,struct mem_cgroup *memcg) -{ -return 0 ; -}static int memcg_online_kmem (struct mem_cgroup *memcg) {return 0 ;
@@ -2939,22 +2920,10 @@ static void memcg_free_kmem(struct mem_cgroup *memcg)static int memcg_update_kmem_limit (struct mem_cgroup *memcg,unsigned long limit) { -int ret =0 ; +int ret;mutex_lock (&memcg_limit_mutex); -/* Top-level cgroup doesn't propagate from root */ -if (!memcg_kmem_online (memcg)) { -if (cgroup_is_populated (memcg->css.cgroup) || - (memcg->use_hierarchy &&memcg_has_children (memcg))) - ret = -EBUSY ; -if (ret) - goto out; - ret =memcg_online_kmem (memcg); -if (ret) - goto out; - } ret =page_counter_limit (&memcg->kmem, limit); -out:mutex_unlock (&memcg_limit_mutex);return ret; }
@@ -4205,7 +4174,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &memcg->css; } - error =memcg_propagate_kmem (parent, memcg); + error =memcg_online_kmem (memcg);if (error) goto fail;
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/6 mm: vmscan: pass root_mem_cgroup instead of NULL to memcg aware shrinker
In reply to Vladimir DavydovIt's just convenient to implement a memcg aware shrinker when you know that shrink_control->memcg != NULL unless memcg_kmem_enabled() returns false.
--- mm/vmscan.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
mm/vmscan.c +10 -5
@@ -382,9 +382,8 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl,* * @memcg specifies the memory cgroup to target. If it is not NULL, * only shrinkers with SHRINKER_MEMCG_AWARE set will be called to scan -* objects from the memory cgroup specified. Otherwise all shrinkers -* are called, and memcg aware shrinkers are supposed to scan the -* global list then. +* objects from the memory cgroup specified. Otherwise, only unaware +* shrinkers are called. * * @nr_scanned and @nr_eligible form a ratio that indicate how much of * the available objects should be scanned. Page reclaim for example
@@ -404,7 +403,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid,struct shrinker *shrinker;unsigned long freed =0 ; -if (memcg && !memcg_kmem_online (memcg)) +if (memcg && (!memcg_kmem_enabled () || !mem_cgroup_online (memcg)))return 0 ;if (nr_scanned ==0 )
@@ -428,7 +427,13 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, .memcg = memcg, }; -if (memcg && !(shrinker->flags &SHRINKER_MEMCG_AWARE )) +/* +* If kernel memory accounting is disabled, we ignore +* SHRINKER_MEMCG_AWARE flag and call all shrinkers +* passing NULL for memcg. +*/ +if (memcg_kmem_enabled () && + !!memcg != !!(shrinker->flags &SHRINKER_MEMCG_AWARE ))continue ;if (!(shrinker->flags &SHRINKER_NUMA_AWARE ))
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 3/6 mm: memcontrol: zap memcg_kmem_online helper
In reply to Vladimir DavydovAs kmem accounting is now either enabled for all cgroups or disabled system-wide, there's no point in having memcg_kmem_online() helper - instead one can use memcg_kmem_enabled() and mem_cgroup_online(), as shrink_slab() now does.
There are only two places left where this helper is used - __memcg_kmem_charge() and memcg_create_kmem_cache(). The former can only be called if memcg_kmem_enabled() returned true. Since the cgroup it operates on is online, mem_cgroup_is_root() check will be enough.
memcg_create_kmem_cache() can't use mem_cgroup_online() helper instead of memcg_kmem_online(), because it relies on the fact that in memcg_offline_kmem() memcg->kmem_state is changed before memcg_deactivate_kmem_caches() is called, but there we can just open-code the check.
--- include/linux/memcontrol.h | 10 ---------- mm/memcontrol.c | 2 +- mm/slab_common.c | 2 +- 3 files changed, 2 insertions(+), 12 deletions(-)
include/linux/memcontrol.h +0 -10
@@ -795,11 +795,6 @@ static inline bool memcg_kmem_enabled(void)return static_branch_unlikely (&memcg_kmem_enabled_key); } -static inline bool memcg_kmem_online (struct mem_cgroup *memcg) -{ -return memcg->kmem_state ==KMEM_ONLINE ; -} -/* * In general, we'll do everything in our power to not incur in any overhead * for non-memcg users for the kmem functions. Not even a function call, if we
@@ -909,11 +904,6 @@ static inline bool memcg_kmem_enabled(void)return false; } -static inline bool memcg_kmem_online (struct mem_cgroup *memcg) -{ -return false; -} -static inline int memcg_kmem_charge (struct page *page,gfp_t gfp,int order) {return 0 ;
mm/memcontrol.c +1 -1
@@ -2346,7 +2346,7 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)int ret =0 ; memcg =get_mem_cgroup_from_mm (current->mm); -if (memcg_kmem_online (memcg)) +if (!mem_cgroup_is_root (memcg)) ret =__memcg_kmem_charge_memcg (page, gfp, order, memcg);css_put (&memcg->css);return ret;
mm/slab_common.c +1 -1
@@ -510,7 +510,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending. */ -if (!memcg_kmem_online (memcg)) +if (memcg->kmem_state !=KMEM_ONLINE ) goto out_unlock; idx =memcg_cache_id (memcg);
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 4/6 radix-tree: account radix_tree_node to memory cgroup
In reply to Vladimir DavydovAllocation of radix_tree_node objects can be easily triggered from userspace, so we should account them to memory cgroup. Besides, we need them accounted for making shadow node shrinker per memcg (see mm/workingset.c).
A tricky thing about accounting radix_tree_node objects is that they are mostly allocated through radix_tree_preload(), so we can't just set SLAB_ACCOUNT for radix_tree_node_cachep - that would likely result in a lot of unrelated cgroups using objects from each other's caches.
One way to overcome this would be making radix tree preloads per memcg, but that would probably look cumbersome and overcomplicated.
Instead, we make radix_tree_node_alloc() first try to allocate from the cache with __GFP_ACCOUNT, no matter if the caller has preloaded or not, and only if it fails fall back on using per cpu preloads. This should make most allocations accounted.
--- lib/radix-tree.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
lib/radix-tree.c +13 -3
@@ -227,6 +227,15 @@ radix_tree_node_alloc(struct radix_tree_root *root)struct radix_tree_preload *rtp;/* +* Even if the caller has preloaded, try to allocate from the +* cache first for the new node to get accounted. +*/ + ret =kmem_cache_alloc (radix_tree_node_cachep, + gfp_mask | __GFP_ACCOUNT | __GFP_NOWARN); +if (ret) + goto out; + +/* * Provided the caller has preloaded here, we will always * succeed in getting a node here (and never reach * kmem_cache_alloc)
@@ -243,10 +252,11 @@ radix_tree_node_alloc(struct radix_tree_root *root)* for debugging. */ kmemleak_update_trace (ret); + goto out; } -if (ret ==NULL ) - ret =kmem_cache_alloc (radix_tree_node_cachep, gfp_mask); - + ret =kmem_cache_alloc (radix_tree_node_cachep, + gfp_mask | __GFP_ACCOUNT); +out:BUG_ON (radix_tree_is_indirect_ptr (ret));return ret; }
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 5/6 mm: workingset: size shadow nodes lru basing on file cache size
In reply to Vladimir DavydovA page is activated on refault if the refault distance stored in the corresponding shadow entry is less than the number of active file pages. Since active file pages can't occupy more than half memory, we assume that the maximal effective refault distance can't be greater than half the number of present pages and size the shadow nodes lru list appropriately. Generally speaking, this assumption is correct, but it can result in wasting a considerable chunk of memory on stale shadow nodes in case the portion of file pages is small, e.g. if a workload mostly uses anonymous memory.
To sort this out, we need to compute the size of shadow nodes lru basing not on the maximal possible, but the current size of file cache. We could take the size of active file lru for the maximal refault distance, but active lru is pretty unstable - it can shrink dramatically at runtime possibly disrupting workingset detection logic.
Instead we assume that the maximal refault distance equals half the total number of file cache pages. This will protect us against active file lru size fluctuations while still being correct, because size of active lru is normally maintained lower than size of inactive lru.
--- mm/workingset.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/workingset.c +3 -1
@@ -349,7 +349,9 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, shadow_nodes =list_lru_shrink_count (&workingset_shadow_nodes, sc);local_irq_enable (); - pages =node_present_pages (sc->nid); + pages =node_page_state (sc->nid,NR_ACTIVE_FILE ) + +node_page_state (sc->nid,NR_INACTIVE_FILE ); +/* * Active cache pages are limited to 50% of memory, and shadow * entries that represent a refault distance bigger than that
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2 5/6] mm: workingset: size shadow nodes lru basing on file cache size
Quoted 16 lines · Vladimir Davydov
A page is activated on refault if the refault distance stored in the corresponding shadow entry is less than the number of active file pages. Since active file pages can't occupy more than half memory, we assume that the maximal effective refault distance can't be greater than half the number of present pages and size the shadow nodes lru list appropriately. Generally speaking, this assumption is correct, but it can result in wasting a considerable chunk of memory on stale shadow nodes in case the portion of file pages is small, e.g. if a workload mostly uses anonymous memory.
To sort this out, we need to compute the size of shadow nodes lru basing not on the maximal possible, but the current size of file cache. We could take the size of active file lru for the maximal refault distance, but active lru is pretty unstable - it can shrink dramatically at runtime possibly disrupting workingset detection logic.
Quoted 6 lines
Instead we assume that the maximal refault distance equals half the total number of file cache pages. This will protect us against active file lru size fluctuations while still being correct, because size of active lru is normally maintained lower than size of inactive lru.
Signed-off-by Vladimir Davydov <[email protected]>
Begrudgingly, because I don't think it matters that much and I like the dumber version. But it's a reasonable change nonetheless.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 6/6 mm: workingset: make shadow node shrinker memcg aware
In reply to Vladimir DavydovWorkingset code was recently made memcg aware, but shadow node shrinker is still global. As a result, one small cgroup can consume all memory available for shadow nodes, possibly hurting other cgroups by reclaiming their shadow nodes, even though reclaim distances stored in its shadow nodes have no effect. To avoid this, we need to make shadow node shrinker memcg aware.
--- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 5 ++--- mm/workingset.c | 10 +++++++--- 3 files changed, 19 insertions(+), 6 deletions(-)
include/linux/memcontrol.h +10 -0
@@ -403,6 +403,9 @@ int mem_cgroup_select_victim_node(struct mem_cgroup *memcg);void mem_cgroup_update_lru_size (struct lruvec *lruvec,enum lru_list lru,int nr_pages); +unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg, +int nid,unsigned int lru_mask); +static inline unsigned long mem_cgroup_get_lru_size (struct lruvec *lruvec,enum lru_list lru) {
@@ -661,6 +664,13 @@ mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru, { } +static inline unsigned long +mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg, +int nid,unsigned int lru_mask) +{ +return 0 ; +} +static inline void mem_cgroup_print_oom_info (struct mem_cgroup *memcg,struct task_struct *p) {
mm/memcontrol.c +2 -3
@@ -638,9 +638,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,__this_cpu_add (memcg->stat->nr_page_events, nr_pages); } -static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg, -int nid, -unsigned int lru_mask) +unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg, +int nid,unsigned int lru_mask) {unsigned long nr =0 ;int zid;
mm/workingset.c +7 -3
@@ -349,8 +349,12 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, shadow_nodes =list_lru_shrink_count (&workingset_shadow_nodes, sc);local_irq_enable (); - pages =node_page_state (sc->nid,NR_ACTIVE_FILE ) + -node_page_state (sc->nid,NR_INACTIVE_FILE ); +if (memcg_kmem_enabled ()) + pages =mem_cgroup_node_nr_lru_pages (sc->memcg, sc->nid, +LRU_ALL_FILE ); +else + pages =node_page_state (sc->nid,NR_ACTIVE_FILE ) + +node_page_state (sc->nid,NR_INACTIVE_FILE );/* * Active cache pages are limited to 50% of memory, and shadow
@@ -460,7 +464,7 @@ static struct shrinker workingset_shadow_shrinker = { .count_objects = count_shadow_nodes, .scan_objects = scan_shadow_nodes, .seeks = DEFAULT_SEEKS, - .flags = SHRINKER_NUMA_AWARE, + .flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE, };/*
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Vladimir Davydov
Workingset code was recently made memcg aware, but shadow node shrinker is still global. As a result, one small cgroup can consume all memory available for shadow nodes, possibly hurting other cgroups by reclaiming their shadow nodes, even though reclaim distances stored in its shadow nodes have no effect. To avoid this, we need to make shadow node shrinker memcg aware.
Signed-off-by Vladimir Davydov <[email protected]>
w00t!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>