mm: reparent slab memory on cgroup removal
Subsystems (4)
v6 cover mm: reparent slab memory on cgroup removal
# Why do we need this?
We've noticed that the number of dying cgroups is steadily growing on most of our hosts in production. The following investigation revealed an issue in userspace memory reclaim code [1], accounting of kernel stacks [2], and also the mainreason: slab objects.
The underlying problem is quite simple: any page charged
to a cgroup holds a reference to it, so the cgroup can't be reclaimed unless all charged pages are gone. If a slab object is actively used by other cgroups, it won't be reclaimed, and will prevent the origin cgroup from being reclaimed.
Slab objects, and first of all vfs cache, is shared between cgroups, which are using the same underlying fs, and what's even more important, it's shared between multiple generations of the same workload. So if something is running periodically every time in a new cgroup (like how systemd works), we do accumulate multiple dying cgroups.
Strictly speaking pagecache isn't different here, but there is a key difference:
we disable protection and apply some extra pressure on LRUs of dying cgroups, and these LRUs contain all charged pages.
My experiments show that with the disabled kernel memory accounting the number of dying cgroups stabilizes at a relatively small number (~100, depends on memory pressure and cgroup creation rate), and with kernel memory accounting it grows pretty steadily up to several thousands.
Memory cgroups are quite complex and big objects (mostly due to percpu stats), so it leads to noticeable memory losses. Memory occupied by dying cgroups is measured in hundreds of megabytes. I've even seen a host with more than 100Gb of memory wasted for dying cgroups. It leads to a degradation of performance with the uptime, and generally limits the usage of cgroups.
My previous attempt [3] to fix the problem by applying extra pressure on slab shrinker lists caused a regressions with xfs and ext4, and has been reverted [4]. The following attempts to find the right balance [5, 6] were not successful.
So instead of trying to find a maybe non-existing balance, let's do reparent the accounted slabs to the parent cgroup on cgroup removal.
# Implementation approach
There is however a significant problem with reparenting of slab memory:
there is no list of charged pages. Some of them are in shrinker lists, but not all. Introducing of a new list is really not an option.
But fortunately there is a way forward: every slab page has a stable pointer to the corresponding kmem_cache. So the idea is to reparent kmem_caches instead of slab pages.
It's actually simpler and cheaper, but requires some underlying changes:
1) Make kmem_caches to hold a single reference to the memory cgroup, instead of a separate reference per every slab page.
2) Stop setting page->mem_cgroup pointer for memcg slab pages and use page->kmem_cache->memcg indirection instead. It's used only on slab page release, so it shouldn't be a big issue.
3) Introduce a refcounter for non-root slab caches. It's required to be able to destroy kmem_caches when they become empty and release the associated memory cgroup.
There is a bonus: currently we do release empty kmem_caches on cgroup removal, however all other are waiting for the releasing of the memory cgroup. These refactorings allow kmem_caches to be released as soon as they become inactive and free.
Some additional implementation details are provided in corresponding commit messages.
# Results
Below is the average number of dying cgroups on two groups of our production hosts. They do run some sort of web frontend workload, the memory pressure is moderate. As we can see, with the kernel memory reparenting the number stabilizes in 60s range; however with the original version it grows almost linearly and doesn't show any signs of plateauing. The difference in slab and percpu usage between patched and unpatched versions also grows linearly. In 7 days it exceeded 200Mb.
day 0 1 2 3 4 5 6 7
original 56 362 628 752 1070 1250 1490 1560
patched 23 46 51 55 60 57 67 69
mem diff(Mb) 22 74 123 152 164 182 214 241
# History
v6:
1) split biggest patches into parts to make the review easier
2) changed synchronization around the dying flag
3) sysfs entry removal on deactivation is back
4) got rid of redundant rcu wait on kmem_cache release
5) fixed getting memcg pointer in mem_cgroup_from_kmem()
5) fixed missed smp_rmb()
6) removed redundant CONFIG_SLOB
7) some renames and cosmetic fixes
v5:
1) fixed a compilation warning around missing kmemcg_queue_cache_shutdown()
2) s/rcu_read_lock()/rcu_read_unlock() in memcg_kmem_get_cache()
v4:
1) removed excessive memcg != parent check in memcg_deactivate_kmem_caches()
2) fixed rcu_read_lock() usage in memcg_charge_slab()
3) fixed synchronization around dying flag in kmemcg_queue_cache_shutdown()
4) refreshed test results data
5) reworked PageTail() checks in memcg_from_slab_page()
6) added some comments in multiple places
v3:
1) reworked memcg kmem_cache search on allocation path
2) fixed /proc/kpagecgroup interface
v2:
1) switched to percpu kmem_cache refcounter
2) a reference to kmem_cache is held during the allocation
3) slabs stats are fixed for !MEMCG case (and the refactoring is separated into a standalone patch)
4) kmem_cache reparenting is performed from deactivatation context
v1:
https://lkml.org/lkml/2019/4/17/1095
# Links
[1]: commit 68600f623d69 ("mm: don't miss the last page because of round-off error")
[2]: commit 9b6f7e163cd0 ("mm: rework memcg kernel stack accounting") [3]: commit 172b06c32b94 ("mm: slowly shrink slabs with a relatively small number of objects")
[4]: commit a9a238e83fbb ("Revert "mm: slowly shrink slabs with a relatively small number of objects")
[5]: https://lkml.org/lkml/2019/1/28/1865
[6]: https://marc.info/?l=linux-mm&m=155064763626437&w=2
Roman Gushchin (10):
mm: add missing smp read barrier on getting memcg kmem_cache pointer
mm: postpone kmem_cache memcg pointer initialization to
memcg_link_cache()
mm: rename slab delayed deactivation functions and fields
mm: generalize postponed non-root kmem_cache deactivation
mm: introduce __memcg_kmem_uncharge_memcg()
mm: unify SLAB and SLUB page accounting
mm: synchronize access to kmem_cache dying flag using a spinlock
mm: rework non-root kmem_cache lifecycle management
mm: stop setting page->mem_cgroup pointer for slab pages
mm: reparent slab memory on cgroup removal
include/linux/memcontrol.h | 10 +++ include/linux/slab.h | 13 +-- mm/list_lru.c | 11 ++- mm/memcontrol.c | 102 +++++++++++++++------- mm/slab.c | 25 ++---- mm/slab.h | 140 +++++++++++++++++++++--------- mm/slab_common.c | 170 ++++++++++++++++++++++--------------- mm/slub.c | 24 +----- 8 files changed, 312 insertions(+), 183 deletions(-)
Signature · 1 line
2.20.1
PATCH v6 1/10 mm: add missing smp read barrier on getting memcg kmem_cache pointer
Johannes noticed that reading the memcg kmem_cache pointer in cache_from_memcg_idx() is performed using READ_ONCE() macro, which doesn't implement a SMP barrier, which is required by the logic.
Add a proper smp_rmb() to be paired with smp_wmb() in memcg_create_kmem_cache().
The same applies to memcg_create_kmem_cache() itself, which reads the same value without barriers and READ_ONCE().
--- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)
mm/slab.h +1 -0
@@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx)* memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ +smp_rmb (); cachep =READ_ONCE (arr->entries[idx]);rcu_read_unlock ();
mm/slab_common.c +2 -1
@@ -652,7 +652,8 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* allocation (see memcg_kmem_get_cache()), several threads can try to * create the same cache, but only one of them may succeed. */ -if (arr->entries[idx]) +smp_rmb (); +if (READ_ONCE (arr->entries[idx])) goto out_unlock;cgroup_name (css->cgroup, memcg_name_buf,sizeof (memcg_name_buf));
Signature · 1 line
2.20.1
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
Quoted 6 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.
Quoted 8 lines
Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>
This seems like independent to the series. Shouldn't this be Cc'ed stable?
Quoted 34 lines
--- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)mm/slab.h +1
-0@@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx)* memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ +smp_rmb (); cachep =READ_ONCE (arr->entries[idx]);rcu_read_unlock ();mm/slab_common.c +2
-2@@ -652,7 +652,8 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* allocation (see memcg_kmem_get_cache()), several threads can try to * create the same cache, but only one of them may succeed. */ -if (arr->entries[idx]) +smp_rmb (); +if (READ_ONCE (arr->entries[idx])) goto out_unlock;cgroup_name (css->cgroup, memcg_name_buf,sizeof (memcg_name_buf)); --2.20.1
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
Quoted 13 lines · Shakeel Butt
Quoted 12 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().
Quoted 6 lines
Quoted 2 lines
Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>Reviewed-by Shakeel Butt <[email protected]>This seems like independent to the series. Shouldn't this be Cc'ed stable?
It is independent, but let's keep it here to avoid merge conflicts.
It has been so for a long time, and nobody complained, so I'm not sure if we really need a stable backport. Do you have a different opinion?
Thank you!
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
Quoted 18 lines · Roman Gushchin
Quoted 16 lines · Shakeel Butt
Quoted 14 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>
Quoted 9 lines
Quoted 3 lines
Reviewed-by Shakeel Butt <[email protected]>This seems like independent to the series. Shouldn't this be Cc'ed stable?
It is independent, but let's keep it here to avoid merge conflicts.
It has been so for a long time, and nobody complained, so I'm not sure if we really need a stable backport. Do you have a different opinion?
Nah, it's fine as it is.
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
In reply to Roman GushchinQuoted 5 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.
Quoted 8 lines
Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
In reply to Roman GushchinQuoted 23 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>--- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)mm/slab.h +0
-0
mm/slab.h cache_from_memcg_idx(struct kmem_cache *s, int idx) · line 264
Quoted 5 lines
mm/slab.h +1
-0@@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx)* memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ +smp_rmb (); cachep =READ_ONCE (arr->entries[idx]);
Hmm, we used to have lockless_dereference() here, but it was replaced with READ_ONCE some time ago. The commit message claims that READ_ONCE has an implicit read barrier in it.
commit 506458efaf153c1ea480591c5602a5a3ba5a3b76
Author: Will Deacon <[email protected]> Date: Tue Oct 24 11:22:48 2017 +0100
locking/barriers: Convert users of lockless_dereference() to READ_ONCE()
READ_ONCE() now has an implicit smp_read_barrier_depends() call, so it can be used instead of lockless_dereference() without any change in semantics.
commit 76ebbe78f7390aee075a7f3768af197ded1bdfbb
Author: Will Deacon <[email protected]> Date: Tue Oct 24 11:22:47 2017 +0100
locking/barriers: Add implicit smp_read_barrier_depends() to READ_ONCE()
In preparation for the removal of lockless_dereference(), which is the same as READ_ONCE() on all architectures other than Alpha, add an implicit smp_read_barrier_depends() to READ_ONCE() so that it can be used to head dependency chains on all architectures.
Johannes Weiner
maintainerRe: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
Quoted 19 lines · Vladimir Davydov
Quoted 18 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>--- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)
Quoted 14 lines
mm/slab.h cache_from_memcg_idx(struct kmem_cache *s, int idx) · line 264
Quoted 10 lines
mm/slab.h +1
-0@@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx)* memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ +smp_rmb (); cachep =READ_ONCE (arr->entries[idx]);Hmm, we used to have
lockless_dereference()here, but it was replaced withREAD_ONCEsome time ago. The commit message claims thatREAD_ONCEhas an implicit read barrier in it.
Thanks for catching this Vladimir. I wasn't aware of this change to the memory model. Indeed, we don't need to change anything here.
Re: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer
Quoted 35 lines · Johannes Weiner
Quoted 19 lines · Vladimir Davydov
Quoted 18 lines · Roman Gushchin
Johannes noticed that reading the memcg
kmem_cachepointer incache_from_memcg_idx()is performed usingREAD_ONCE()macro, which doesn't implement a SMP barrier, which is required by the logic.Add a proper
smp_rmb()to be paired withsmp_wmb()inmemcg_create_kmem_cache().The same applies to
memcg_create_kmem_cache()itself, which reads the same value without barriers andREAD_ONCE().Suggested-by Johannes Weiner <[email protected]>Signed-off-by Roman Gushchin <[email protected]>--- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)Quoted 14 lines
mm/slab.h cache_from_memcg_idx(struct kmem_cache *s, int idx) · line 264
Quoted 10 lines
mm/slab.h +1
-0@@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx)* memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ +smp_rmb (); cachep =READ_ONCE (arr->entries[idx]);Hmm, we used to have
lockless_dereference()here, but it was replaced withREAD_ONCEsome time ago. The commit message claims thatREAD_ONCEhas an implicit read barrier in it.
Quoted 2 lines
Thanks for catching this Vladimir. I wasn't aware of this change to the memory model. Indeed, we don't need to change anything here.
Cool, I'm dropping this patch.
Thanks!
PATCH v6 2/10 mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()
In reply to Roman GushchinInitialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params().
Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting.
For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe.
--- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-)
mm/slab.c +1 -1
@@ -1239,7 +1239,7 @@ void __init kmem_cache_init(void) nr_node_ids *sizeof (struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0);list_add (&kmem_cache->list, &slab_caches); -memcg_link_cache (kmem_cache); +memcg_link_cache (kmem_cache,NULL ); slab_state =PARTIAL ;/*
mm/slab.h +3 -2
@@ -290,7 +290,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, }extern void slab_init_memcg_params (struct kmem_cache *); -extern void memcg_link_cache (struct kmem_cache *s); +extern void memcg_link_cache (struct kmem_cache *s,struct mem_cgroup *memcg);extern void slab_deactivate_memcg_cache_rcu_sched (struct kmem_cache *s,void (*deact_fn)(struct kmem_cache *));
@@ -345,7 +345,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache (struct kmem_cache *s) +static inline void memcg_link_cache (struct kmem_cache *s, +struct mem_cgroup *memcg) { }
mm/slab_common.c +7 -7
@@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) }static int init_memcg_params (struct kmem_cache *s, -struct mem_cgroup *memcg,struct kmem_cache *root_cache) +struct kmem_cache *root_cache) {struct memcg_cache_array *arr;if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg;INIT_LIST_HEAD (&s->memcg_params.children_node);INIT_LIST_HEAD (&s->memcg_params.kmem_caches_node);return 0 ;
@@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs)return ret; } -void memcg_link_cache (struct kmem_cache *s) +void memcg_link_cache (struct kmem_cache *s,struct mem_cgroup *memcg) {if (is_root_cache (s)) {list_add (&s->root_caches_node, &slab_root_caches); }else { + s->memcg_params.memcg = memcg;list_add (&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node,
@@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) }#else static inline int init_memcg_params (struct kmem_cache *s, -struct mem_cgroup *memcg,struct kmem_cache *root_cache) +struct kmem_cache *root_cache) {return 0 ; }
@@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err =init_memcg_params (s, memcg, root_cache); + err =init_memcg_params (s, root_cache);if (err) goto out_free_cache;
@@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount =1 ;list_add (&s->list, &slab_caches); -memcg_link_cache (s); +memcg_link_cache (s, memcg); out:if (err)return ERR_PTR (err);
@@ -998,7 +998,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name,create_boot_cache (s, name, size, flags, useroffset, usersize);list_add (&s->list, &slab_caches); -memcg_link_cache (s); +memcg_link_cache (s,NULL ); s->refcount =1 ;return s; }
mm/slub.c +1 -1
@@ -4213,7 +4213,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) }slab_init_memcg_params (s);list_add (&s->list, &slab_caches); -memcg_link_cache (s); +memcg_link_cache (s,NULL );return s; }
Signature · 1 line
2.20.1
PATCH v6 3/10 mm: rename slab delayed deactivation functions and fields
In reply to Roman GushchinThe delayed work/rcu deactivation infrastructure of non-root kmem_caches can be also used for asynchronous release of these objects. Let's get rid of the word "deactivation" in corresponding names to make the code look better after generalization.
It's easier to make the renaming first, so that the generalized code will look consistent from scratch.
Let's rename struct memcg_cache_params fields:
deact_fn -> work_fn
deact_rcu_head -> rcu_head
deact_work -> work
And RCU/delayed work callbacks in slab common code:
kmemcg_deactivate_rcufn -> kmemcg_rcufn kmemcg_deactivate_workfn -> kmemcg_workfn
This patch contains no functional changes, only renamings.
--- include/linux/slab.h | 6 +++--- mm/slab.h | 2 +- mm/slab_common.c | 30 +++++++++++++++--------------- 3 files changed, 19 insertions(+), 19 deletions(-)
include/linux/slab.h +3 -3
@@ -642,10 +642,10 @@ struct memcg_cache_params {struct list_head children_node;struct list_head kmem_caches_node; -void (*deact_fn)(struct kmem_cache *); +void (*work_fn)(struct kmem_cache *);union { -struct rcu_head deact_rcu_head; -struct work_struct deact_work; +struct rcu_head rcu_head; +struct work_struct work; }; }; };
mm/slab.h +1 -1
@@ -292,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,extern void slab_init_memcg_params (struct kmem_cache *);extern void memcg_link_cache (struct kmem_cache *s,struct mem_cgroup *memcg);extern void slab_deactivate_memcg_cache_rcu_sched (struct kmem_cache *s, -void (*deact_fn)(struct kmem_cache *)); +void (*work_fn)(struct kmem_cache *));#else /* CONFIG_MEMCG_KMEM */
mm/slab_common.c +15 -15
@@ -692,17 +692,17 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,put_online_cpus (); } -static void kmemcg_deactivate_workfn (struct work_struct *work) +static void kmemcg_workfn (struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work);get_online_cpus ();get_online_mems ();mutex_lock (&slab_mutex); - s->memcg_params.deact_fn (s); + s->memcg_params.work_fn (s);mutex_unlock (&slab_mutex);
@@ -713,36 +713,36 @@ static void kmemcg_deactivate_workfn(struct work_struct *work)css_put (&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn (struct rcu_head *head) +static void kmemcg_rcufn (struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head);/* -* We need to grab blocking locks. Bounce to ->deact_work. The +* We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be * initialized eariler. */ -INIT_WORK (&s->memcg_params.deact_work, kmemcg_deactivate_workfn); -queue_work (memcg_kmem_cache_wq, &s->memcg_params.deact_work); +INIT_WORK (&s->memcg_params.work, kmemcg_workfn); +queue_work (memcg_kmem_cache_wq, &s->memcg_params.work); }/** * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a * sched RCU grace period * @s: target kmem_cache -* @deact_fn: deactivation function to call +* @work_fn: deactivation function to call * -* Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex +* Schedule @work_fn to be invoked with online cpus, mems and slab_mutex * held after a sched RCU grace period. The slab is guaranteed to stay -* alive until @deact_fn is finished. This is to be used from +* alive until @work_fn is finished. This is to be used from * __kmemcg_cache_deactivate(). */ void slab_deactivate_memcg_cache_rcu_sched (struct kmem_cache *s, -void (*deact_fn)(struct kmem_cache *)) +void (*work_fn)(struct kmem_cache *)) {if (WARN_ON_ONCE (is_root_cache (s)) || -WARN_ON_ONCE (s->memcg_params.deact_fn)) +WARN_ON_ONCE (s->memcg_params.work_fn))return ;if (s->memcg_params.root_cache->memcg_params.dying)
@@ -751,8 +751,8 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,/* pin memcg so that @s doesn't get destroyed in the middle */ css_get (&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; -call_rcu (&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + s->memcg_params.work_fn = work_fn; +call_rcu (&s->memcg_params.rcu_head, kmemcg_rcufn); }void memcg_deactivate_kmem_caches (struct mem_cgroup *memcg)
Signature · 1 line
2.20.1
Re: [PATCH v6 03/10] mm: rename slab delayed deactivation functions and fields
Quoted 13 lines · Roman Gushchin
The delayed work/rcu deactivation infrastructure of non-root
kmem_cachescan be also used for asynchronous release of these objects. Let's get rid of the word "deactivation" in corresponding names to make the code look better after generalization.It's easier to make the renaming first, so that the generalized code will look consistent from scratch.
Let's rename struct
memcg_cache_paramsfields:
deact_fn->work_fn
deact_rcu_head->rcu_head
deact_work-> work
Quoted 7 lines
And RCU/delayed work callbacks in slab common code:
kmemcg_deactivate_rcufn->kmemcg_rcufnkmemcg_deactivate_workfn->kmemcg_workfnThis patch contains no functional changes, only renamings.
Signed-off-by Roman Gushchin <[email protected]>
PATCH v6 4/10 mm: generalize postponed non-root kmem_cache deactivation
In reply to Roman GushchinCurrently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches release, but requires generalization for SLAB case.
Introduce kmemcg_cache_deactivate() function, which calls allocator-specific __kmem_cache_deactivate() and schedules execution of __kmem_cache_deactivate_after_rcu() with all necessary locks in a worker context after an rcu grace period.
Here is the new calling scheme:
kmemcg_cache_deactivate()
__kmemcg_cache_deactivate() SLAB/SLUB-specific
kmemcg_rcufn() rcu
kmemcg_workfn() work
__kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specificinstead of:
__kmemcg_cache_deactivate() SLAB/SLUB-specific
slab_deactivate_memcg_cache_rcu_sched() SLUB-only
kmemcg_rcufn() rcu
kmemcg_workfn() work
kmemcg_cache_deact_after_rcu() SLUB-onlyFor consistency, all allocator-specific functions start with "__".
--- mm/slab.c | 4 ++++ mm/slab.h | 3 +-- mm/slab_common.c | 27 ++++++++------------------- mm/slub.c | 8 +------- 4 files changed, 14 insertions(+), 28 deletions(-)
mm/slab.c +4 -0
@@ -2252,6 +2252,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) {__kmem_cache_shrink (cachep); } + +void __kmemcg_cache_deactivate_after_rcu (struct kmem_cache *s) +{ +}#endif int __kmem_cache_shutdown (struct kmem_cache *cachep)
mm/slab.h +1 -2
@@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *);void __kmem_cache_release (struct kmem_cache *);int __kmem_cache_shrink (struct kmem_cache *);void __kmemcg_cache_deactivate (struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu (struct kmem_cache *s);void slab_kmem_cache_release (struct kmem_cache *);struct seq_file ;
@@ -291,8 +292,6 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,extern void slab_init_memcg_params (struct kmem_cache *);extern void memcg_link_cache (struct kmem_cache *s,struct mem_cgroup *memcg); -extern void slab_deactivate_memcg_cache_rcu_sched (struct kmem_cache *s, -void (*work_fn)(struct kmem_cache *));#else /* CONFIG_MEMCG_KMEM */
mm/slab_common.c +8 -19
@@ -709,7 +709,7 @@ static void kmemcg_workfn(struct work_struct *work)put_online_mems ();put_online_cpus (); -/* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ +/* done, put the ref from kmemcg_cache_deactivate() */ css_put (&s->memcg_params.memcg->css); }
@@ -727,31 +727,21 @@ static void kmemcg_rcufn(struct rcu_head *head)queue_work (memcg_kmem_cache_wq, &s->memcg_params.work); } -/** -* slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a -* sched RCU grace period -* @s: target kmem_cache -* @work_fn: deactivation function to call -* -* Schedule @work_fn to be invoked with online cpus, mems and slab_mutex -* held after a sched RCU grace period. The slab is guaranteed to stay -* alive until @work_fn is finished. This is to be used from -* __kmemcg_cache_deactivate(). -*/ -void slab_deactivate_memcg_cache_rcu_sched (struct kmem_cache *s, -void (*work_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate (struct kmem_cache *s) {if (WARN_ON_ONCE (is_root_cache (s)) ||WARN_ON_ONCE (s->memcg_params.work_fn))return ; +__kmemcg_cache_deactivate (s); +if (s->memcg_params.root_cache->memcg_params.dying)return ;/* pin memcg so that @s doesn't get destroyed in the middle */ css_get (&s->memcg_params.memcg->css); - s->memcg_params.work_fn = work_fn; + s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;call_rcu (&s->memcg_params.rcu_head, kmemcg_rcufn); }
@@ -774,7 +764,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)if (!c)continue ; -__kmemcg_cache_deactivate (c); +kmemcg_cache_deactivate (c); arr->entries[idx] =NULL ; }mutex_unlock (&slab_mutex);
@@ -867,11 +857,10 @@ static void flush_memcg_workqueue(struct kmem_cache *s)mutex_unlock (&slab_mutex);/* -* SLUB deactivates the kmem_caches through call_rcu. Make +* SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ -if (IS_ENABLED (CONFIG_SLUB )) -rcu_barrier (); +rcu_barrier ();/* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB
mm/slub.c +1 -7
@@ -4022,7 +4022,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) }#ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu (struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu (struct kmem_cache *s) {/* * Called with all the locks held after a sched RCU grace period.
@@ -4048,12 +4048,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s)*/ slub_set_cpu_partial (s,0 ); s->min_partial = 0; - -/* -* s->cpu_partial is checked locklessly (see put_cpu_partial), so -* we have to make sure the change is visible before shrinking. -*/ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); }#endif /* CONFIG_MEMCG */
Signature · 1 line
2.20.1
Re: [PATCH v6 04/10] mm: generalize postponed non-root kmem_cache deactivation
Quoted 24 lines · Roman Gushchin
Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root
kmem_cache. This mechanism can be reused forkmem_cachesrelease, but requires generalization for SLAB case.Introduce
kmemcg_cache_deactivate()function, which calls allocator-specific__kmem_cache_deactivate()and schedules execution of__kmem_cache_deactivate_after_rcu()with all necessary locks in a worker context after an rcu grace period.Here is the new calling scheme:
kmemcg_cache_deactivate()__kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_rcufn() rcu kmemcg_workfn() work __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specificinstead of:
__kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_rcufn() rcu kmemcg_workfn() work kmemcg_cache_deact_after_rcu() SLUB-only
Quoted 3 lines
For consistency, all allocator-specific functions start with "__".
Signed-off-by Roman Gushchin <[email protected]>
Much easier to review after extracting renaming, thanks.
PATCH v6 5/10 mm: introduce __memcg_kmem_uncharge_memcg()
In reply to Roman GushchinLet's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work.
This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_uncharge_memcg() if memcg_kmem_enabled() check is passed.
--- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-)
include/linux/memcontrol.h +10 -0
@@ -1304,6 +1304,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order);void __memcg_kmem_uncharge (struct page *page,int order);int __memcg_kmem_charge_memcg (struct page *page,gfp_t gfp,int order,struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg (struct mem_cgroup *memcg, +unsigned int nr_pages);extern struct static_key_false memcg_kmem_enabled_key;extern struct workqueue_struct *memcg_kmem_cache_wq;
@@ -1345,6 +1347,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp,return __memcg_kmem_charge_memcg (page, gfp, order, memcg);return 0 ; } + +static inline void memcg_kmem_uncharge_memcg (struct page *page,int order, +struct mem_cgroup *memcg) +{ +if (memcg_kmem_enabled ()) +__memcg_kmem_uncharge_memcg (memcg,1 << order); +} +/* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function
mm/memcontrol.c +17 -8
@@ -2731,6 +2731,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)css_put (&memcg->css);return ret; } + +/** +* __memcg_kmem_uncharge_memcg: uncharge a kmem page +* @memcg: memcg to uncharge +* @nr_pages: number of pages to uncharge +*/ +void __memcg_kmem_uncharge_memcg (struct mem_cgroup *memcg, +unsigned int nr_pages) +{ +if (!cgroup_subsys_on_dfl (memory_cgrp_subsys)) +page_counter_uncharge (&memcg->kmem, nr_pages); + +page_counter_uncharge (&memcg->memory, nr_pages); +if (do_memsw_account ()) +page_counter_uncharge (&memcg->memsw, nr_pages); +}/** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge
@@ -2745,14 +2761,7 @@ void __memcg_kmem_uncharge(struct page *page, int order)return ;VM_BUG_ON_PAGE (mem_cgroup_is_root (memcg), page); - -if (!cgroup_subsys_on_dfl (memory_cgrp_subsys)) -page_counter_uncharge (&memcg->kmem, nr_pages); - -page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); - +__memcg_kmem_uncharge_memcg (memcg, nr_pages); page->mem_cgroup =NULL ;/* slab pages do not have PageKmemcg flag set */
Signature · 1 line
2.20.1
Re: [PATCH v6 05/10] mm: introduce __memcg_kmem_uncharge_memcg()
Quoted 11 lines · Roman Gushchin
Let's separate the page counter modification code out of
__memcg_kmem_uncharge()in a way similar to what__memcg_kmem_charge()and__memcg_kmem_charge_memcg()work.This will allow to reuse this code later using a new
memcg_kmem_uncharge_memcg()wrapper, which calls__memcg_kmem_uncharge_memcg()ifmemcg_kmem_enabled()check is passed.Signed-off-by Roman Gushchin <[email protected]>Reviewed-by Shakeel Butt <[email protected]>
PATCH v6 6/10 mm: unify SLAB and SLUB page accounting
In reply to Roman GushchinCurrently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls.
--- mm/slab.c | 19 +++---------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 30 insertions(+), 28 deletions(-)
mm/slab.c +3 -16
@@ -1360,7 +1360,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) {struct page *page; -int nr_pages; flags |= cachep->allocflags;
@@ -1370,17 +1369,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,return NULL ; } -if (memcg_charge_slab (page, flags, cachep->gfporder, cachep)) { +if (charge_slab_page (page, flags, cachep->gfporder, cachep)) {__free_pages (page, cachep->gfporder);return NULL ; } - nr_pages = (1 << cachep->gfporder); -if (cachep->flags &SLAB_RECLAIM_ACCOUNT ) -mod_lruvec_page_state (page,NR_SLAB_RECLAIMABLE , nr_pages); -else -mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE , nr_pages); -__SetPageSlab (page);/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (page))
@@ -1395,12 +1388,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,static void kmem_freepages (struct kmem_cache *cachep,struct page *page) {int order = cachep->gfporder; -unsigned long nr_freed = (1 << order); - -if (cachep->flags &SLAB_RECLAIM_ACCOUNT ) -mod_lruvec_page_state (page,NR_SLAB_RECLAIMABLE , -nr_freed); -else -mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE , -nr_freed);BUG_ON (!PageSlab (page));__ClearPageSlabPfmemalloc (page);
@@ -1409,8 +1396,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping =NULL ;if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; -memcg_uncharge_slab (page, order, cachep); + current->reclaim_state->reclaimed_slab +=1 << order; +uncharge_slab_page (page, order, cachep);__free_pages (page, order); }
mm/slab.h +25 -0
@@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer,void __kmem_cache_free_bulk (struct kmem_cache *,size_t ,void **);int __kmem_cache_alloc_bulk (struct kmem_cache *,gfp_t ,size_t ,void **); +static inline int cache_vmstat_idx (struct kmem_cache *s) +{ +return (s->flags &SLAB_RECLAIM_ACCOUNT ) ? +NR_SLAB_RECLAIMABLE :NR_SLAB_UNRECLAIMABLE ; +} +#ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */
@@ -362,6 +368,25 @@ static inline struct kmem_cache *virt_to_cache(const void *obj)return page->slab_cache; } +static __always_inline int charge_slab_page(struct page *page, +gfp_t gfp,int order, +struct kmem_cache *s) +{ +int ret =memcg_charge_slab (page, gfp, order, s); + +if (!ret) +mod_lruvec_page_state (page,cache_vmstat_idx (s),1 << order); + +return ret; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, +struct kmem_cache *s) +{ +mod_lruvec_page_state (page,cache_vmstat_idx (s), -(1 << order)); +memcg_uncharge_slab (page, order, s); +} +static inline struct kmem_cache *cache_from_obj (struct kmem_cache *s,void *x) {struct kmem_cache *cachep;
mm/slub.c +2 -12
@@ -1488,7 +1488,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,else page =__alloc_pages_node (node, flags, order); -if (page &&memcg_charge_slab (page, flags, order, s)) { +if (page &&charge_slab_page (page, flags, order, s)) {__free_pages (page, order); page =NULL ; }
@@ -1681,11 +1681,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)if (!page)return NULL ; -mod_lruvec_page_state (page, - (s->flags &SLAB_RECLAIM_ACCOUNT ) ? -NR_SLAB_RECLAIMABLE :NR_SLAB_UNRECLAIMABLE , -1 <<oo_order (oo)); -inc_slabs_node (s,page_to_nid (page), page->objects);return page;
@@ -1719,18 +1714,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page)check_object (s, page, p,SLUB_RED_INACTIVE ); } -mod_lruvec_page_state (page, - (s->flags &SLAB_RECLAIM_ACCOUNT ) ? -NR_SLAB_RECLAIMABLE :NR_SLAB_UNRECLAIMABLE , - -pages); -__ClearPageSlabPfmemalloc (page);__ClearPageSlab (page); page->mapping =NULL ;if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; -memcg_uncharge_slab (page, order, s); +uncharge_slab_page (page, order, s);__free_pages (page, order); }
Signature · 1 line
2.20.1
PATCH v6 7/10 mm: synchronize access to kmem_cache dying flag using a spinlock
In reply to Roman GushchinCurrently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of kmem_caches is synchronized using the slab_mutex.
It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of kmem_caches.
So let's switch over to the irq-save flavor of the spinlock-based synchronization.
--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
mm/slab_common.c +15 -4
@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {
@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();
@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending. */ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; + +spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock; idx =memcg_cache_id (memcg);
@@ -735,14 +743,17 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)__kmemcg_cache_deactivate (s); +spin_lock_irq (&memcg_kmem_wq_lock);if (s->memcg_params.root_cache->memcg_params.dying) -return ; + goto unlock;/* pin memcg so that @s doesn't get destroyed in the middle */ css_get (&s->memcg_params.memcg->css); s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;call_rcu (&s->memcg_params.rcu_head, kmemcg_rcufn); +unlock: +spin_unlock_irq (&memcg_kmem_wq_lock); }void memcg_deactivate_kmem_caches (struct mem_cgroup *memcg)
@@ -852,9 +863,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s)static void flush_memcg_workqueue (struct kmem_cache *s) { -mutex_lock (&slab_mutex); +spin_lock_irq (&memcg_kmem_wq_lock); s->memcg_params.dying = true; -mutex_unlock (&slab_mutex); +spin_unlock_irq (&memcg_kmem_wq_lock);/* * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make
Signature · 1 line
2.20.1
Johannes Weiner
maintainerRe: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
Quoted 39 lines · Roman Gushchin
Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of
kmem_cachesis synchronized using theslab_mutex.It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of
kmem_caches.So let's switch over to the irq-save flavor of the spinlock-based synchronization.
Signed-off-by Roman Gushchin <[email protected]>--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)mm/slab_common.c +2
-0@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending.
mm/slab_common.c void memcg_create_kmem_cache(struct mem_cgroup *memcg, · line 652
Quoted 10 lines
mm/slab_common.c +7
-1*/ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; + +spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock;
What does this lock protect? The dying flag could get set right after the unlock.
Re: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
Quoted 45 lines · Johannes Weiner
Quoted 44 lines · Roman Gushchin
Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of
kmem_cachesis synchronized using theslab_mutex.It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of
kmem_caches.So let's switch over to the irq-save flavor of the spinlock-based synchronization.
Signed-off-by Roman Gushchin <[email protected]>--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)mm/slab_common.c +5
-1@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending. */ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; +
Quoted 9 lines
mm/slab_common.c void memcg_create_kmem_cache(struct mem_cgroup *memcg, · line 652
Quoted 5 lines
mm/slab_common.c +4
-0+spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock;What does this lock protect? The dying flag could get set right after the unlock.
Hi Johannes!
Here is my logic:
1) flush_memcg_workqueue() must guarantee that no new memcg kmem_caches will be created, and there are no works queued, which will touch the root kmem_cache, so it can be released
2) so it sets the dying flag, waits for an rcu grace period and flushes the workqueue (that means for all in-flight works)
3) dying flag in checked in kmemcg_cache_shutdown() and kmemcg_cache_deactivate(), so that if it set, no new works/rcu tasks will be queued. corresponding queue_work()/call_rcu() are all under memcg_kmem_wq_lock lock.
4) memcg_schedule_kmem_cache_create() doesn't check the dying flag (probably to avoid taking locks on a hot path), but it does memcg_create_kmem_cache(), which is part of the scheduled work. And it does it at the very beginning, so even if new kmem_caches are scheduled to be created, the root kmem_cache won't be touched.
Previously the flag was checked under slab_mutex, but now we set it under memcg_kmem_wq_lock lock. So I'm not sure we can read it without taking this lock.
If the flag will be set after unlock, it's fine. It means that the work has already been scheduled, and flush_workqueue() in flush_memcg_workqueue() will wait for it. The only problem is if we don't see the flag after flush_workqueue() is called, but I don't see how it's possible.
Does it makes sense? I'm sure there are ways to make it more obvious. Please, let me know if you've any ideas.
Thank you!
Re: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
Quoted 79 lines · Roman Gushchin
Quoted 45 lines · Johannes Weiner
Quoted 44 lines · Roman Gushchin
Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of
kmem_cachesis synchronized using theslab_mutex.It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of
kmem_caches.So let's switch over to the irq-save flavor of the spinlock-based synchronization.
Signed-off-by Roman Gushchin <[email protected]>--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)mm/slab_common.c +5
-1@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending. */ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; +Quoted 9 lines
mm/slab_common.c void memcg_create_kmem_cache(struct mem_cgroup *memcg, · line 652
Quoted 5 lines
mm/slab_common.c +4
-0+spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock;What does this lock protect? The dying flag could get set right after the unlock.
Hi Johannes!
Here is my logic:
1)
flush_memcg_workqueue()must guarantee that no new memcgkmem_cacheswill be created, and there are no works queued, which will touch the rootkmem_cache, so it can be released
2) so it sets the dying flag, waits for an rcu grace period and flushes the workqueue (that means for all in-flight works)
3) dying flag in checked inkmemcg_cache_shutdown()andkmemcg_cache_deactivate(), so that if it set, no new works/rcu tasks will be queued. corresponding queue_work()/call_rcu() are all undermemcg_kmem_wq_locklock.
4)memcg_schedule_kmem_cache_create()doesn't check the dying flag (probably to avoid taking locks on a hot path), but it doesmemcg_create_kmem_cache(), which is part of the scheduled work. And it does it at the very beginning, so even if newkmem_cachesare scheduled to be created, the rootkmem_cachewon't be touched.Previously the flag was checked under
slab_mutex, but now we set it undermemcg_kmem_wq_locklock. So I'm not sure we can read it without taking this lock.
Quoted 8 lines
If the flag will be set after unlock, it's fine. It means that the work has already been scheduled, and
flush_workqueue()inflush_memcg_workqueue()will wait for it. The only problem is if we don't see the flag afterflush_workqueue()is called, but I don't see how it's possible.Does it makes sense? I'm sure there are ways to make it more obvious. Please, let me know if you've any ideas.
Hm, after some thoughts, I've found that the problem is that we check the dying flag of the root cache. But it's the same in the existing code.
So currently (without my patches):
1) we do set the dying flag under slab_mutex
2) waiting for the workqueue to flush
3) grabbing the slab_mutex and going to release the root kmem_cache
a concurrent memcg_kmem_cache_create_func() can be scheduled after 2), grab the slab_mutex after 3) and check the kmem_cache->memcg_params.dying flag of already released kmem_cache.
The reason why it's not a real problem is that it's expected from a user that kmem_cache will not be used for new allocations after calling kmem_cache_destroy(). It means no new memcg kmem_cache creation will be scheduled, and we can avoid checking the dying flag at all.
Does this makes sense?
Thanks!
Re: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
In reply to Roman GushchinQuoted 39 lines · Roman Gushchin
Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of
kmem_cachesis synchronized using theslab_mutex.It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of
kmem_caches.So let's switch over to the irq-save flavor of the spinlock-based synchronization.
Signed-off-by Roman Gushchin <[email protected]>--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)mm/slab_common.c +2
-0@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending.
mm/slab_common.c void memcg_create_kmem_cache(struct mem_cgroup *memcg, · line 652
Quoted 10 lines
mm/slab_common.c +7
-1*/ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; + +spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock;
I do understand why we need to sync setting dying flag for a kmem cache about to be destroyed in flush_memcg_workqueue vs checking the flag inkmemcg_cache_deactivate: this is needed so that we don't schedule a new deactivation work after we flush RCU/workqueue. However, I don't think it's necessary to check the dying flag here, in memcg_create_kmem_cache:
we can't schedule a new cache creation work after kmem_cache_destroy has started, because one mustn't allocate from a dead kmem cache; since we flush the queue before getting to actual destruction, no cache creation work can be pending. Yeah, it might happen that a cache creation work starts execution while flush_memcg_workqueue is in progress, but I don't see any point in optimizing this case - after all, cache destruction is a very cold path. Since checking the flag in memcg_create_kmem_cache raises question, I suggest to simply drop this check.
Anyway, it would be nice to see some comment in the code explaining why we check dying flag under a spin lock in kmemcg_cache_deactivate.
Quoted 33 lines
mm/slab_common.c +6
-3idx =memcg_cache_id (memcg);@@ -735,14 +743,17 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)__kmemcg_cache_deactivate (s); +spin_lock_irq (&memcg_kmem_wq_lock);if (s->memcg_params.root_cache->memcg_params.dying) -return ; + goto unlock;/* pin memcg so that @s doesn't get destroyed in the middle */ css_get (&s->memcg_params.memcg->css); s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;call_rcu (&s->memcg_params.rcu_head, kmemcg_rcufn); +unlock: +spin_unlock_irq (&memcg_kmem_wq_lock); }void memcg_deactivate_kmem_caches (struct mem_cgroup *memcg)@@ -852,9 +863,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s)static void flush_memcg_workqueue (struct kmem_cache *s) { -mutex_lock (&slab_mutex); +spin_lock_irq (&memcg_kmem_wq_lock); s->memcg_params.dying = true; -mutex_unlock (&slab_mutex); +spin_unlock_irq (&memcg_kmem_wq_lock);/* * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make
Re: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
Quoted 56 lines · Vladimir Davydov
Quoted 39 lines · Roman Gushchin
Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of
kmem_cachesis synchronized using theslab_mutex.It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of
kmem_caches.So let's switch over to the irq-save flavor of the spinlock-based synchronization.
Signed-off-by Roman Gushchin <[email protected]>--- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)mm/slab_common.c +2
-0@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,#ifdef CONFIG_MEMCG_KMEM LIST_HEAD (slab_root_caches); +static DEFINE_SPINLOCK (memcg_kmem_wq_lock);void slab_init_memcg_params (struct kmem_cache *s) {@@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,struct memcg_cache_array *arr;struct kmem_cache *s =NULL ;char *cache_name; +bool dying;int idx;get_online_cpus ();@@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,* The memory cgroup could have been offlined while the cache * creation work was pending. mm/slab_common.c void memcg_create_kmem_cache(struct mem_cgroup *memcg, · line 652
Quoted 10 lines
mm/slab_common.c +7
-1*/ -if (memcg->kmem_state !=KMEM_ONLINE || root_cache->memcg_params.dying) +if (memcg->kmem_state !=KMEM_ONLINE ) + goto out_unlock; + +spin_lock_irq (&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; +spin_unlock_irq (&memcg_kmem_wq_lock); +if (dying) goto out_unlock;I do understand why we need to sync setting dying flag for a kmem cache about to be destroyed in
flush_memcg_workqueuevs checking the flag inkmemcg_cache_deactivate: this is needed so that we don't schedule a new deactivation work after we flush RCU/workqueue. However, I don't think it's necessary to check the dying flag here, inmemcg_create_kmem_cache:
Quoted 8 lines
we can't schedule a new cache creation work after
kmem_cache_destroyhas started, because one mustn't allocate from a dead kmem cache; since we flush the queue before getting to actual destruction, no cache creation work can be pending. Yeah, it might happen that a cache creation work starts execution whileflush_memcg_workqueueis in progress, but I don't see any point in optimizing this case - after all, cache destruction is a very cold path. Since checking the flag inmemcg_create_kmem_cacheraises question, I suggest to simply drop this check.
Yeah, I came to the same conclusion (in a thread with Johannes), that this check is not required. I'll drop it in a separate patch.
Quoted 3 lines
Anyway, it would be nice to see some comment in the code explaining why we check dying flag under a spin lock in
kmemcg_cache_deactivate.
Sure, will add some.
Btw, thank you very much for reviewing the series!
PATCH v6 8/10 mm: rework non-root kmem_cache lifecycle management
In reply to Roman GushchinCurrently each charged slab page holds a reference to the cgroup to which it's charged. Kmem_caches are held by the memcg and are released all together with the memory cgroup. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is very far from optimal.
Let's rework it in a way that allows releasing individual kmem_caches as soon as the cgroup is offline, the kmem_cache is empty and there are no pending allocations.
To make it possible, let's introduce a new percpu refcounter for non-root kmem caches. The counter is initialized to the percpu mode, and is switched to the atomic mode during kmem_cache deactivation. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete.
To shutdown non-active empty kmem_caches, let's reuse the work queue, previously used for the kmem_cache deactivation. Once the reference counter reaches 0, let's schedule an asynchronous kmem_cache release.
* I used the following simple approach to test the performance (stolen from another patchset by T. Harding):
time find / -name fname-no-exist
echo 2 > /proc/sys/vm/drop_caches
repeat 10 times
Results:
orig patched
real 0m1.455s real 0m1.355s
user 0m0.206s user 0m0.219s
sys 0m0.855s sys 0m0.807s
real 0m1.487s real 0m1.699s
user 0m0.221s user 0m0.256s
sys 0m0.806s sys 0m0.948s
real 0m1.515s real 0m1.505s
user 0m0.183s user 0m0.215s
sys 0m0.876s sys 0m0.858s
real 0m1.291s real 0m1.380s
user 0m0.193s user 0m0.198s
sys 0m0.843s sys 0m0.786s
real 0m1.364s real 0m1.374s
user 0m0.180s user 0m0.182s
sys 0m0.868s sys 0m0.806s
real 0m1.352s real 0m1.312s
user 0m0.201s user 0m0.212s
sys 0m0.820s sys 0m0.761s
real 0m1.302s real 0m1.349s
user 0m0.205s user 0m0.203s
sys 0m0.803s sys 0m0.792s
real 0m1.334s real 0m1.301s
user 0m0.194s user 0m0.201s
sys 0m0.806s sys 0m0.779s
real 0m1.426s real 0m1.434s
user 0m0.216s user 0m0.181s
sys 0m0.824s sys 0m0.864s
real 0m1.350s real 0m1.295s
user 0m0.200s user 0m0.190s
sys 0m0.842s sys 0m0.811s
So it looks like the difference is not noticeable in this test.
--- include/linux/slab.h | 3 +- mm/memcontrol.c | 51 +++++++++++++++++++++------- mm/slab.h | 45 +++++++------------------ mm/slab_common.c | 79 ++++++++++++++++++++++++++------------------ 4 files changed, 100 insertions(+), 78 deletions(-)
include/linux/slab.h +2 -1
@@ -16,6 +16,7 @@#include <linux/overflow.h> #include <linux/types.h> #include <linux/workqueue.h> +#include <linux/percpu-refcount.h> /*
@@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *);void memcg_create_kmem_cache (struct mem_cgroup *,struct kmem_cache *);void memcg_deactivate_kmem_caches (struct mem_cgroup *); -void memcg_destroy_kmem_caches (struct mem_cgroup *);/* * Please use this macro to create slab caches. Simply specify the
@@ -641,6 +641,7 @@ struct memcg_cache_params {struct mem_cgroup *memcg;struct list_head children_node;struct list_head kmem_caches_node; +struct percpu_ref refcnt;void (*work_fn)(struct kmem_cache *);union {
mm/memcontrol.c +39 -12
95 lines not on this page · raw message
mm/slab.h +12 -33
70 lines not on this page · raw message
mm/slab_common.c +47 -32
146 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v6 08/10] mm: rework non-root kmem_cache lifecycle management
Quoted 68 lines · Roman Gushchin
Currently each charged slab page holds a reference to the cgroup to which it's charged.
Kmem_cachesare held by the memcg and are released all together with the memory cgroup. It means that none ofkmem_cachesare released unless at least one reference to the memcg exists, which is very far from optimal.Let's rework it in a way that allows releasing individual
kmem_cachesas soon as the cgroup is offline, thekmem_cacheis empty and there are no pending allocations.To make it possible, let's introduce a new percpu refcounter for non-root kmem caches. The counter is initialized to the percpu mode, and is switched to the atomic mode during
kmem_cachedeactivation. The counter is bumped for every charged page and also for every running allocation. So thekmem_cachecan't be released unless all allocations complete.To shutdown non-active empty
kmem_caches, let's reuse the work queue, previously used for thekmem_cachedeactivation. Once the reference counter reaches 0, let's schedule an asynchronouskmem_cacherelease.* I used the following simple approach to test the performance (stolen from another patchset by T. Harding):
time find / -name fname-no-exist
echo 2 >/proc/sys/vm/drop_caches
repeat 10 timesResults:
orig patched
real 0m1.455s real 0m1.355s
user 0m0.206s user 0m0.219s
sys 0m0.855s sys 0m0.807sreal 0m1.487s real 0m1.699s
user 0m0.221s user 0m0.256s
sys 0m0.806s sys 0m0.948sreal 0m1.515s real 0m1.505s
user 0m0.183s user 0m0.215s
sys 0m0.876s sys 0m0.858sreal 0m1.291s real 0m1.380s
user 0m0.193s user 0m0.198s
sys 0m0.843s sys 0m0.786sreal 0m1.364s real 0m1.374s
user 0m0.180s user 0m0.182s
sys 0m0.868s sys 0m0.806sreal 0m1.352s real 0m1.312s
user 0m0.201s user 0m0.212s
sys 0m0.820s sys 0m0.761sreal 0m1.302s real 0m1.349s
user 0m0.205s user 0m0.203s
sys 0m0.803s sys 0m0.792sreal 0m1.334s real 0m1.301s
user 0m0.194s user 0m0.201s
sys 0m0.806s sys 0m0.779sreal 0m1.426s real 0m1.434s
user 0m0.216s user 0m0.181s
sys 0m0.824s sys 0m0.864s
Quoted 7 lines
real 0m1.350s real 0m1.295s
user 0m0.200s user 0m0.190s
sys 0m0.842s sys 0m0.811sSo it looks like the difference is not noticeable in this test.
Signed-off-by Roman Gushchin <[email protected]>
PATCH v6 9/10 mm: stop setting page->mem_cgroup pointer for slab pages
In reply to Roman GushchinEvery slab page charged to a non-root memory cgroup has a pointer to the memory cgroup and holds a reference to it, which protects a non-empty memory cgroup from being released. At the same time the page has a pointer to the corresponding kmem_cache, and also hold a reference to the kmem_cache. And kmem_cache by itself holds a reference to the cgroup.
So there is clearly some redundancy, which allows to stop setting the page->mem_cgroup pointer and rely on getting memcg pointer indirectly via kmem_cache. Further it will allow to change this pointer easier, without a need to go over all charged pages.
So let's stop setting page->mem_cgroup pointer for slab pages, and stop using the css refcounter directly for protecting the memory cgroup from going away. Instead rely on kmem_cache as an intermediate object.
Make sure that vmstats and shrinker lists are working as previously, as well as /proc/kpagecgroup interface.
--- mm/list_lru.c | 3 +- mm/memcontrol.c | 12 ++++---- mm/slab.h | 74 ++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 70 insertions(+), 19 deletions(-)
mm/list_lru.c +2 -1
@@ -12,6 +12,7 @@#include <linux/slab.h> #include <linux/mutex.h> #include <linux/memcontrol.h> +#include "slab.h" #ifdef CONFIG_MEMCG_KMEM static LIST_HEAD (list_lrus);
@@ -63,7 +64,7 @@ static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr)if (!memcg_kmem_enabled ())return NULL ; page =virt_to_head_page (ptr); -return page->mem_cgroup; +return memcg_from_slab_page (page); } static inline struct list_lru_one *
mm/memcontrol.c +7 -5
@@ -485,7 +485,10 @@ ino_t page_cgroup_ino(struct page *page)unsigned long ino =0 ;rcu_read_lock (); - memcg =READ_ONCE (page->mem_cgroup); +if (PageHead (page) &&PageSlab (page)) + memcg =memcg_from_slab_page (page); +else + memcg =READ_ONCE (page->mem_cgroup);while (memcg && !(memcg->css.flags &CSS_ONLINE )) memcg =parent_mem_cgroup (memcg);if (memcg)
@@ -2727,9 +2730,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order,cancel_charge (memcg, nr_pages);return -ENOMEM ; } - - page->mem_cgroup = memcg; -return 0 ; }
@@ -2752,8 +2752,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg =get_mem_cgroup_from_current ();if (!mem_cgroup_is_root (memcg)) { ret =__memcg_kmem_charge_memcg (page, gfp, order, memcg); -if (!ret) +if (!ret) { + page->mem_cgroup = memcg;__SetPageKmemcg (page); + } }css_put (&memcg->css);return ret;
mm/slab.h +61 -13
115 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v6 09/10] mm: stop setting page->mem_cgroup pointer for slab pages
Quoted 17 lines · Roman Gushchin
Every slab page charged to a non-root memory cgroup has a pointer to the memory cgroup and holds a reference to it, which protects a non-empty memory cgroup from being released. At the same time the page has a pointer to the corresponding
kmem_cache, and also hold a reference to thekmem_cache. Andkmem_cacheby itself holds a reference to the cgroup.So there is clearly some redundancy, which allows to stop setting the
page->mem_cgrouppointer and rely on getting memcg pointer indirectly viakmem_cache. Further it will allow to change this pointer easier, without a need to go over all charged pages.So let's stop setting
page->mem_cgrouppointer for slab pages, and stop using the css refcounter directly for protecting the memory cgroup from going away. Instead rely onkmem_cacheas an intermediate object.
Quoted 4 lines
Make sure that vmstats and shrinker lists are working as previously, as well as
/proc/kpagecgroupinterface.Signed-off-by Roman Gushchin <[email protected]>
PATCH v6 10/10 mm: reparent slab memory on cgroup removal
In reply to Roman GushchinLet's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application).
So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as a part of the deactivation process.
Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer and drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Quite simple.
Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock() or with slab_mutex held.
We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere.
Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything.
--- include/linux/slab.h | 4 ++-- mm/list_lru.c | 8 +++++++- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 23 +++++++++++++++++------ mm/slab_common.c | 22 +++++++++++++++++++--- 5 files changed, 53 insertions(+), 18 deletions(-)
include/linux/slab.h +2 -2
@@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *);int kmem_cache_shrink (struct kmem_cache *);void memcg_create_kmem_cache (struct mem_cgroup *,struct kmem_cache *); -void memcg_deactivate_kmem_caches (struct mem_cgroup *); +void memcg_deactivate_kmem_caches (struct mem_cgroup *,struct mem_cgroup *);/* * Please use this macro to create slab caches. Simply specify the
@@ -638,7 +638,7 @@ struct memcg_cache_params {bool dying; };struct { -struct mem_cgroup *memcg; +struct mem_cgroup __rcu *memcg;struct list_head children_node;struct list_head kmem_caches_node;struct percpu_ref refcnt;
mm/list_lru.c +7 -1
@@ -77,11 +77,15 @@ list_lru_from_kmem(struct list_lru_node *nlru, void *ptr,if (!nlru->memcg_lrus) goto out; +rcu_read_lock (); memcg =mem_cgroup_from_kmem (ptr); -if (!memcg) +if (!memcg) { +rcu_read_unlock (); goto out; + } l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); +rcu_read_unlock (); out:if (memcg_ptr) *memcg_ptr = memcg;
@@ -131,12 +135,14 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item)spin_lock (&nlru->lock);if (list_empty (item)) { +rcu_read_lock (); l =list_lru_from_kmem (nlru, item, &memcg);list_add_tail (item, &l->list);/* Set shrinker bit if the first element was added */ if (!l->nr_items++)memcg_set_shrinker_bit (memcg, nid,lru_shrinker_id (lru)); +rcu_read_unlock (); nlru->nr_items++;spin_unlock (&nlru->lock);return true;
mm/memcontrol.c +8 -6
@@ -3209,15 +3209,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)*/ memcg->kmem_state =KMEM_ALLOCATED ; -memcg_deactivate_kmem_caches (memcg); - - kmemcg_id = memcg->kmemcg_id; -BUG_ON (kmemcg_id <0 ); - parent =parent_mem_cgroup (memcg);if (!parent) parent = root_mem_cgroup; +memcg_deactivate_kmem_caches (memcg, parent); + + kmemcg_id = memcg->kmemcg_id; +BUG_ON (kmemcg_id <0 ); +/* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus
@@ -3250,7 +3250,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg)if (memcg->kmem_state ==KMEM_ALLOCATED ) {WARN_ON (!list_empty (&memcg->kmem_caches));static_branch_dec (&memcg_kmem_enabled_key); -WARN_ON (page_counter_read (&memcg->kmem)); } }#else
@@ -4675,6 +4674,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)/* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM +INIT_LIST_HEAD (&memcg->kmem_caches); +#endif root_mem_cgroup = memcg;return &memcg->css; }
mm/slab.h +17 -6
@@ -268,7 +268,7 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) s =READ_ONCE (page->slab_cache);if (s && !is_root_cache (s)) -return s->memcg_params.memcg; +return rcu_dereference (s->memcg_params.memcg);return NULL ; }
@@ -285,10 +285,18 @@ static __always_inline int memcg_charge_slab(struct page *page,struct lruvec *lruvec;int ret; - memcg = s->memcg_params.memcg; +rcu_read_lock (); + memcg =rcu_dereference (s->memcg_params.memcg); +while (memcg && !css_tryget_online (&memcg->css)) + memcg =parent_mem_cgroup (memcg); +rcu_read_unlock (); + +if (unlikely (!memcg)) +return true; + ret =memcg_kmem_charge_memcg (page, gfp, order, memcg);if (ret) -return ret; + goto out; lruvec =mem_cgroup_lruvec (page_pgdat (page), memcg);mod_lruvec_state (lruvec,cache_vmstat_idx (s),1 << order);
@@ -296,8 +304,9 @@ static __always_inline int memcg_charge_slab(struct page *page,/* transer try_charge() page references to kmem_cache */ percpu_ref_get_many (&s->memcg_params.refcnt,1 << order);css_put_many (&memcg->css,1 << order); - -return 0 ; +out: +css_put (&memcg->css); +return ret; }/*
@@ -310,10 +319,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,struct mem_cgroup *memcg;struct lruvec *lruvec; - memcg = s->memcg_params.memcg; +rcu_read_lock (); + memcg =rcu_dereference (s->memcg_params.memcg); lruvec =mem_cgroup_lruvec (page_pgdat (page), memcg);mod_lruvec_state (lruvec,cache_vmstat_idx (s), -(1 << order));memcg_kmem_uncharge_memcg (page, order, memcg); +rcu_read_unlock ();percpu_ref_put_many (&s->memcg_params.refcnt,1 << order); }
mm/slab_common.c +19 -3
@@ -237,7 +237,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg)list_add (&s->root_caches_node, &slab_root_caches); }else {css_get (&memcg->css); - s->memcg_params.memcg = memcg; +rcu_assign_pointer (s->memcg_params.memcg, memcg);list_add (&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node,
@@ -252,7 +252,9 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else {list_del (&s->memcg_params.children_node);list_del (&s->memcg_params.kmem_caches_node); -css_put (&s->memcg_params.memcg->css); +mem_cgroup_put (rcu_dereference_protected (s->memcg_params.memcg, +lockdep_is_held (&slab_mutex))); +rcu_assign_pointer (s->memcg_params.memcg,NULL ); } }#else
@@ -793,11 +795,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)spin_unlock_irq (&memcg_kmem_wq_lock); } -void memcg_deactivate_kmem_caches (struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches (struct mem_cgroup *memcg, +struct mem_cgroup *parent) {int idx;struct memcg_cache_array *arr;struct kmem_cache *s, *c; +unsigned int nr_reparented; idx =memcg_cache_id (memcg);
@@ -815,6 +819,18 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)kmemcg_cache_deactivate (c); arr->entries[idx] =NULL ; } + nr_reparented =0 ; +list_for_each_entry (s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { +rcu_assign_pointer (s->memcg_params.memcg, parent); +css_put (&memcg->css); + nr_reparented++; + } +if (nr_reparented) { +list_splice_init (&memcg->kmem_caches, + &parent->kmem_caches); +css_get_many (&parent->css, nr_reparented); + }mutex_unlock (&slab_mutex);put_online_mems ();
Signature · 1 line
2.20.1
Quoted 84 lines · Roman Gushchin
Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application).
So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of
kmem_caches. Reparenting is performed as a part of the deactivation process.Since the parent cgroup is already charged, everything we need to do is to splice the list of
kmem_cachesto the parent'skmem_cacheslist, swap the memcg pointer and drop the css refcounter for eachkmem_cacheand adjust the parent's css refcounter. Quite simple.Please, note that
kmem_cache->memcg_params.memcgisn't a stable pointer anymore. It's safe to read it underrcu_read_lock()or withslab_mutexheld.We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere.
Local slab stats (
NR_SLAB_RECLAIMABLEandNR_SLAB_UNRECLAIMABLE) aren't used anywhere exceptcount_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything.Signed-off-by Roman Gushchin <[email protected]>--- include/linux/slab.h | 4 ++-- mm/list_lru.c | 8 +++++++- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 23 +++++++++++++++++------ mm/slab_common.c | 22 +++++++++++++++++++--- 5 files changed, 53 insertions(+), 18 deletions(-)include/linux/slab.h +2
-2@@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *);int kmem_cache_shrink (struct kmem_cache *);void memcg_create_kmem_cache (struct mem_cgroup *,struct kmem_cache *); -void memcg_deactivate_kmem_caches (struct mem_cgroup *); +void memcg_deactivate_kmem_caches (struct mem_cgroup *,struct mem_cgroup *);/* * Please use this macro to create slab caches. Simply specify the @@ -638,7 +638,7 @@ struct memcg_cache_params {bool dying; };struct { -struct mem_cgroup *memcg; +struct mem_cgroup __rcu *memcg;struct list_head children_node;struct list_head kmem_caches_node;struct percpu_ref refcnt;mm/list_lru.c +5
-1@@ -77,11 +77,15 @@ list_lru_from_kmem(struct list_lru_node *nlru, void *ptr,if (!nlru->memcg_lrus) goto out; +rcu_read_lock (); memcg =mem_cgroup_from_kmem (ptr); -if (!memcg) +if (!memcg) { +rcu_read_unlock (); goto out; + } l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); +rcu_read_unlock (); out:if (memcg_ptr) *memcg_ptr = memcg;@@ -131,12 +135,14 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item)
mm/list_lru.c bool list_lru_add(struct list_lru *lru, struct list_head *item) · line 145
Quoted 10 lines
mm/list_lru.c +2
-0spin_lock (&nlru->lock);if (list_empty (item)) { +rcu_read_lock (); l =list_lru_from_kmem (nlru, item, &memcg);list_add_tail (item, &l->list);/* Set shrinker bit if the first element was added */ if (!l->nr_items++)memcg_set_shrinker_bit (memcg, nid,lru_shrinker_id (lru)); +rcu_read_unlock ();
AFAICS we don't need rcu_read_lock here, because holding nlru->lock guarantees that the cgroup doesn't get freed. If that's correct, I think we better remove __rcu mark and use READ_ONCE for accessing memcg_params.memcg, thus making it the caller's responsibility to ensure the cgroup lifetime.
Quoted 51 lines
mm/list_lru.c +0
-0nlru->nr_items++;spin_unlock (&nlru->lock);return true;mm/memcontrol.c +8
-6@@ -3209,15 +3209,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)*/ memcg->kmem_state =KMEM_ALLOCATED ; -memcg_deactivate_kmem_caches (memcg); - - kmemcg_id = memcg->kmemcg_id; -BUG_ON (kmemcg_id <0 ); - parent =parent_mem_cgroup (memcg);if (!parent) parent = root_mem_cgroup; +memcg_deactivate_kmem_caches (memcg, parent); + + kmemcg_id = memcg->kmemcg_id; +BUG_ON (kmemcg_id <0 ); +/* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3250,7 +3250,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg)if (memcg->kmem_state ==KMEM_ALLOCATED ) {WARN_ON (!list_empty (&memcg->kmem_caches));static_branch_dec (&memcg_kmem_enabled_key); -WARN_ON (page_counter_read (&memcg->kmem)); } }#else @@ -4675,6 +4674,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)/* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM +INIT_LIST_HEAD (&memcg->kmem_caches); +#endif root_mem_cgroup = memcg;return &memcg->css; }mm/slab.h +0
-0
mm/slab.h static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) · line 271
Quoted 5 lines
mm/slab.h +1
-1@@ -268,7 +268,7 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) s =READ_ONCE (page->slab_cache);if (s && !is_root_cache (s)) -return s->memcg_params.memcg; +return rcu_dereference (s->memcg_params.memcg);
I guess it's worth updating the comment with a few words re cgroup lifetime.
Andrew Morton
maintainerQuoted 2 lines · On Tue, 4 Jun 2019 19:44:44 -0700 Roman Gushchin <[email protected]> wrote:
So instead of trying to find a maybe non-existing balance, let's do reparent the accounted slabs to the parent cgroup on cgroup removal.
s/slabs/slab caches/. Take more care with the terminology, please...
Quoted 4 lines
There is a bonus: currently we do release empty
kmem_cacheson cgroup removal, however all other are waiting for the releasing of the memory cgroup. These refactorings allowkmem_cachesto be released as soon as they become inactive and free.
Unclear.
s/All other/releasing of all non-empty slab caches depends upon the releasing/
I think?
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 4 Jun 2019 19:44:44 -0700 Roman Gushchin <[email protected]> wrote:
So instead of trying to find a maybe non-existing balance, let's do reparent the accounted slabs to the parent cgroup on cgroup removal.
s/slabs/slab caches/. Take more care with the terminology, please...
Slabs are effectively reparented too (what's most important, their references), but I agree, "slab caches" suits better here.
Quoted 6 lines
Quoted 5 lines
There is a bonus: currently we do release empty
kmem_cacheson cgroup removal, however all other are waiting for the releasing of the memory cgroup. These refactorings allowkmem_cachesto be released as soon as they become inactive and free.
Quoted 6 lines
Unclear.
s/All other/releasing of all non-empty slab caches depends upon the releasing/
I think?
How about this?
There is a bonus: currently we release all memcg kmem_caches all together with the memory cgroup itself. This patchset allows individual kmem_caches to be released as soon as they become inactive and free.
--
Sorry, my bad, I was focused on patches, and didn't give enough attention to the cover letter. I hope to get some feedback from Vladimir, and then post a next version with these issues fixed.
Thank you for looking into it!
Roman