mm: shrinker: fix shrinker_info teardown race with expansion
Subsystems (3)
Qi Zheng
reviewerPATCH 1/1 mm: shrinker: fix shrinker_info teardown race with expansion
From: Qi Zheng <[email protected]>
The expand_shrinker_info() iterates all visible memcgs under shrinker_mutex, including memcgs that have not finished ->css_online() yet.
Once pn->shrinker_info has been published, teardown must stay serialized with expand_shrinker_info() until that memcg is either fully online or no longer visible to iteration. Today alloc_shrinker_info() breaks that rule by dropping shrinker_mutex before freeing a partially initialized shrinker_info array, which may cause the following race:
CPU0 CPU1
==== ====
css_create
--> list_add_tail_rcu(&css->sibling, &parent_css->children);
online_css
--> mem_cgroup_css_online
--> alloc_shrinker_info
--> alloc node0 info
rcu_assign_pointer(C->node0->shrinker_info, old0)
alloc node1 info -> FAIL -> goto err
mutex_unlock(shrinker_mutex)
shrinker_alloc()
--> shrinker_memcg_alloc
--> mutex_lock(shrinker_mutex)
expand_shrinker_info
--> mem_cgroup_iter see the memcg
expand_one_shrinker_info
--> old0 = C->node0->shrinker_info
memcpy(new->unit, old0->unit, ...);
free_shrinker_info
--> kvfree(old0);
/* double free !! */
kvfree_rcu(old0, rcu);The same problem exists later in mem_cgroup_css_online(). If alloc_shrinker_info() succeeds but a subsequent objcg allocation fails, the free_objcg -> free_shrinker_info() unwind path tears down the already published pn->shrinker_info arrays without shrinker_mutex. The expand_one_shrinker_info() can race with that teardown in the same way, leading to use-after-free or double-free of the old shrinker_info.
Fix this by serializing shrinker_info teardown with shrinker_mutex, and by keeping alloc_shrinker_info() error cleanup inside the locked section.
--- mm/shrinker.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
mm/shrinker.c +11 -2
@@ -59,12 +59,14 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new,return 0 ; } -void free_shrinker_info (struct mem_cgroup *memcg) +static void __free_shrinker_info (struct mem_cgroup *memcg) {struct mem_cgroup_per_node *pn;struct shrinker_info *info;int nid; +lockdep_assert_held (&shrinker_mutex); +for_each_node (nid) { pn = memcg->nodeinfo[nid]; info =rcu_dereference_protected (pn->shrinker_info, true);
@@ -74,6 +76,13 @@ void free_shrinker_info(struct mem_cgroup *memcg) } } +void free_shrinker_info (struct mem_cgroup *memcg) +{ +mutex_lock (&shrinker_mutex); +__free_shrinker_info (memcg); +mutex_unlock (&shrinker_mutex); +} +int alloc_shrinker_info (struct mem_cgroup *memcg) {int nid, ret =0 ;
@@ -98,8 +107,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg)return ret; err: +__free_shrinker_info (memcg);mutex_unlock (&shrinker_mutex); -free_shrinker_info (memcg);return -ENOMEM ; }
Signature · 1 line
2.54.0
Quoted 51 lines · Qi Zheng
From: Qi Zheng <[email protected]>
The
expand_shrinker_info()iterates all visible memcgs undershrinker_mutex, including memcgs that have not finished ->css_online() yet.Once
pn->shrinker_infohas been published, teardown must stay serialized withexpand_shrinker_info()until that memcg is either fully online or no longer visible to iteration. Todayalloc_shrinker_info()breaks that rule by droppingshrinker_mutexbefore freeing a partially initializedshrinker_infoarray, which may cause the following race:CPU0 CPU1
==== ====
css_create--> list_add_tail_rcu(&css->sibling, &parent_css->children); online_css --> mem_cgroup_css_online --> alloc_shrinker_info --> alloc node0 info rcu_assign_pointer(C->node0->shrinker_info, old0) alloc node1 info -> FAIL -> goto err mutex_unlock(shrinker_mutex) shrinker_alloc() --> shrinker_memcg_alloc --> mutex_lock(shrinker_mutex) expand_shrinker_info --> mem_cgroup_iter see the memcg expand_one_shrinker_info --> old0 = C->node0->shrinker_info memcpy(new->unit, old0->unit, ...); free_shrinker_info --> kvfree(old0); /* double free !! */ kvfree_rcu(old0, rcu);The same problem exists later in
mem_cgroup_css_online(). Ifalloc_shrinker_info()succeeds but a subsequent objcg allocation fails, thefree_objcg->free_shrinker_info()unwind path tears down the already publishedpn->shrinker_infoarrays withoutshrinker_mutex. Theexpand_one_shrinker_info()can race with that teardown in the same way, leading to use-after-free or double-free of the oldshrinker_info.
Quoted 6 lines
Fix this by serializing
shrinker_infoteardown withshrinker_mutex, and by keepingalloc_shrinker_info()error cleanup inside the locked section.Fixes 307bececcd12 ("mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}")Signed-off-by Qi Zheng <[email protected]>
Thanks.