Skip to contentSkip to conversation

mm: shrinker: fix shrinker_info teardown race with expansion

PATCH Applied Qi Zheng 2 participants 2 messages Jun 17

Subsystems (3)
lore
Share & tools
lore

Qi Zheng

author reviewer
#0 lore raw

PATCH 1/1 mm: shrinker: fix shrinker_info teardown race with expansion

From: Qi Zheng <[email protected]>

The expand_shrinker_info() iterates all visible memcgs under shrinker_mutex, including memcgs that have not finished ->css_online() yet.

Once pn->shrinker_info has been published, teardown must stay serialized with expand_shrinker_info() until that memcg is either fully online or no longer visible to iteration. Today alloc_shrinker_info() breaks that rule by dropping shrinker_mutex before freeing a partially initialized shrinker_info array, which may cause the following race:

CPU0 CPU1
==== ====

css_create

--> list_add_tail_rcu(&css->sibling, &parent_css->children);
    online_css
    --> mem_cgroup_css_online
        --> alloc_shrinker_info
            --> alloc node0 info
                rcu_assign_pointer(C->node0->shrinker_info, old0)
                alloc node1 info -> FAIL -> goto err
                mutex_unlock(shrinker_mutex)

                       shrinker_alloc()
                       --> shrinker_memcg_alloc
                           --> mutex_lock(shrinker_mutex)
                               expand_shrinker_info
                               --> mem_cgroup_iter see the memcg
                                   expand_one_shrinker_info
                                   --> old0 = C->node0->shrinker_info
                                       memcpy(new->unit, old0->unit, ...);

                free_shrinker_info
                --> kvfree(old0);

                                       /* double free !! */
                                       kvfree_rcu(old0, rcu);

The same problem exists later in mem_cgroup_css_online(). If alloc_shrinker_info() succeeds but a subsequent objcg allocation fails, the free_objcg -> free_shrinker_info() unwind path tears down the already published pn->shrinker_info arrays without shrinker_mutex. The expand_one_shrinker_info() can race with that teardown in the same way, leading to use-after-free or double-free of the old shrinker_info.

Fix this by serializing shrinker_info teardown with shrinker_mutex, and by keeping alloc_shrinker_info() error cleanup inside the locked section.

Fixes 307bececcd12 ("mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}")
---
 mm/shrinker.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
mm/shrinker.c +11 -2
index 7082d01c8c9d..a70aab124a0e 100644
@@ -59,12 +59,14 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new,
 	return 0;
 }
 
-void free_shrinker_info(struct mem_cgroup *memcg)
+static void __free_shrinker_info(struct mem_cgroup *memcg)
 {
 	struct mem_cgroup_per_node *pn;
 	struct shrinker_info *info;
 	int nid;
 
+	lockdep_assert_held(&shrinker_mutex);
+
 	for_each_node(nid) {
 		pn = memcg->nodeinfo[nid];
 		info = rcu_dereference_protected(pn->shrinker_info, true);
@@ -74,6 +76,13 @@ void free_shrinker_info(struct mem_cgroup *memcg)
 	}
 }
 
+void free_shrinker_info(struct mem_cgroup *memcg)
+{
+	mutex_lock(&shrinker_mutex);
+	__free_shrinker_info(memcg);
+	mutex_unlock(&shrinker_mutex);
+}
+
 int alloc_shrinker_info(struct mem_cgroup *memcg)
 {
 	int nid, ret = 0;
@@ -98,8 +107,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg)
 	return ret;
 
 err:
+	__free_shrinker_info(memcg);
 	mutex_unlock(&shrinker_mutex);
-	free_shrinker_info(memcg);
 	return -ENOMEM;
 }
 
Signature · 1 line
2.54.0
reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 51 lines · Qi Zheng

From: Qi Zheng <[email protected]>

The expand_shrinker_info() iterates all visible memcgs under shrinker_mutex, including memcgs that have not finished ->css_online() yet.

Once pn->shrinker_info has been published, teardown must stay serialized with expand_shrinker_info() until that memcg is either fully online or no longer visible to iteration. Today alloc_shrinker_info() breaks that rule by dropping shrinker_mutex before freeing a partially initialized shrinker_info array, which may cause the following race:

CPU0 CPU1
==== ====

css_create

--> list_add_tail_rcu(&css->sibling, &parent_css->children);
   online_css
   --> mem_cgroup_css_online
       --> alloc_shrinker_info
           --> alloc node0 info
               rcu_assign_pointer(C->node0->shrinker_info, old0)
               alloc node1 info -> FAIL -> goto err
               mutex_unlock(shrinker_mutex)

                      shrinker_alloc()
                      --> shrinker_memcg_alloc
                          --> mutex_lock(shrinker_mutex)
                              expand_shrinker_info
                              --> mem_cgroup_iter see the memcg
                                  expand_one_shrinker_info
                                  --> old0 = C->node0->shrinker_info
                                      memcpy(new->unit, old0->unit, ...);

               free_shrinker_info
               --> kvfree(old0);

                                      /* double free !! */
                                      kvfree_rcu(old0, rcu);

The same problem exists later in mem_cgroup_css_online(). If alloc_shrinker_info() succeeds but a subsequent objcg allocation fails, the free_objcg -> free_shrinker_info() unwind path tears down the already published pn->shrinker_info arrays without shrinker_mutex. The expand_one_shrinker_info() can race with that teardown in the same way, leading to use-after-free or double-free of the old shrinker_info.

Quoted 6 lines

Fix this by serializing shrinker_info teardown with shrinker_mutex, and by keeping alloc_shrinker_info() error cleanup inside the locked section.

Fixes 307bececcd12 ("mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}")

Thanks.