Skip to contentSkip to conversation

mm/list_lru: Fix possible race in memcg_reparent_list_lru_node()

PATCH Superseded Waiman Long 2 participants 3 messages Mar 28, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/list_lru: Fix possible race in memcg_reparent_list_lru_node()

Muchun Song found out there could be a race between list_lru_add() and memcg_reparent_list_lru_node() causing the later function to miss reparenting of a lru entry as shown below:

CPU0: CPU1:
list_lru_add()

     spin_lock(&nlru->lock)
     l = list_lru_from_kmem(memcg)
                                     memcg_reparent_objcgs(memcg)
                                     memcg_reparent_list_lrus(memcg)
                                         memcg_reparent_list_lru()
                                             memcg_reparent_list_lru_node()
                                                 if (!READ_ONCE(nlru->nr_items))
                                                     // Miss reparenting
                                                     return
     // Assume 0->1
     l->nr_items++
     // Assume 0->1
     nlru->nr_items++

Though it is not likely that a list_lru_node that has 0 item suddenly has a newly added lru entry at the end of its life. The race is still theoretically possible.

Adding a spin_is_locked() check will likely be enough for x86, but it is less certain for other arches with a more relaxed memory semantics like arcm64 and ppc. To avoid race, this patch moves the nr_items check to within the lock critical section.

Fixes 405cc51fc104 ("mm/list_lru: optimize memcg_reparent_list_lru_node()")
---
 mm/list_lru.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
mm/list_lru.c +6 -6
index c669d87001a6..8aec8ebd5995 100644
@@ -394,18 +394,18 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid,
 	int dst_idx = dst_memcg->kmemcg_id;
 	struct list_lru_one *src, *dst;
 
-	/*
-	 * If there is no lru entry in this nlru, we can skip it immediately.
-	 */
-	if (!READ_ONCE(nlru->nr_items))
-		return;
-
 	/*
 	 * Since list_lru_{add,del} may be called under an IRQ-safe lock,
 	 * we have to use IRQ-safe primitives here to avoid deadlock.
 	 */
 	spin_lock_irq(&nlru->lock);
 
+	/*
+	 * If there is no lru entry in this nlru, we can skip it immediately.
+	 */
+	if (!nlru->nr_items)
+		goto out;
+
 	src = list_lru_from_memcg_idx(lru, nid, src_idx);
 	if (!src)
 		goto out;
Signature · 1 line
2.27.0
#1 lore raw
Quoted 24 lines · Waiman Long

Muchun Song found out there could be a race between list_lru_add() and memcg_reparent_list_lru_node() causing the later function to miss reparenting of a lru entry as shown below:

CPU0: CPU1:
list_lru_add()

      spin_lock(&nlru->lock)
      l = list_lru_from_kmem(memcg)
                                      memcg_reparent_objcgs(memcg)
                                      memcg_reparent_list_lrus(memcg)
                                          memcg_reparent_list_lru()
                                              memcg_reparent_list_lru_node()
                                                  if (!READ_ONCE(nlru->nr_items))
                                                      // Miss reparenting
                                                      return
      // Assume 0->1
      l->nr_items++
      // Assume 0->1
      nlru->nr_items++

Though it is not likely that a list_lru_node that has 0 item suddenly has a newly added lru entry at the end of its life. The race is still theoretically possible.

Quoted 6 lines

Adding a spin_is_locked() check will likely be enough for x86, but it is less certain for other arches with a more relaxed memory semantics like arcm64 and ppc. To avoid race, this patch moves the nr_items check to within the lock critical section.

Fixes 405cc51fc104 ("mm/list_lru: optimize memcg_reparent_list_lru_node()")

Sorry, I should have added

Quoted 24 lines
---
  mm/list_lru.c | 12 ++++++------
  1 file changed, 6 insertions(+), 6 deletions(-)
mm/list_lru.c +0 -6
index c669d87001a6..8aec8ebd5995 100644
@@ -394,18 +394,18 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid,
  	int dst_idx = dst_memcg->kmemcg_id;
  	struct list_lru_one *src, *dst;
 
-	/*
-	 * If there is no lru entry in this nlru, we can skip it immediately.
-	 */
-	if (!READ_ONCE(nlru->nr_items))
-		return;
-
  	/*
  	 * Since list_lru_{add,del} may be called under an IRQ-safe lock,
  	 * we have to use IRQ-safe primitives here to avoid deadlock.
  	 */
  	spin_lock_irq(&nlru->lock);

mm/list_lru.c static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, · line 411

Quoted 10 lines
mm/list_lru.c +6 -0
 
+	/*
+	 * If there is no lru entry in this nlru, we can skip it immediately.
+	 */
+	if (!nlru->nr_items)
+		goto out;
+
  	src = list_lru_from_memcg_idx(lru, nid, src_idx);
  	if (!src)
  		goto out;

Cheers,
Longman

#2 lore raw
In reply to Waiman Long
Quoted 25 lines · Waiman Long

Muchun Song found out there could be a race between list_lru_add() and memcg_reparent_list_lru_node() causing the later function to miss reparenting of a lru entry as shown below:

CPU0: CPU1:
list_lru_add()

     spin_lock(&nlru->lock)
     l = list_lru_from_kmem(memcg)
                                     memcg_reparent_objcgs(memcg)
                                     memcg_reparent_list_lrus(memcg)
                                         memcg_reparent_list_lru()
                                             memcg_reparent_list_lru_node()
                                                 if (!READ_ONCE(nlru->nr_items))
                                                     // Miss reparenting
                                                     return
     // Assume 0->1
     l->nr_items++
     // Assume 0->1
     nlru->nr_items++

Though it is not likely that a list_lru_node that has 0 item suddenly has a newly added lru entry at the end of its life. The race is still theoretically possible.

Quoted 7 lines

Adding a spin_is_locked() check will likely be enough for x86, but it is less certain for other arches with a more relaxed memory semantics like arcm64 and ppc. To avoid race, this patch moves the nr_items check to within the lock critical section.

Fixes 405cc51fc104 ("mm/list_lru: optimize memcg_reparent_list_lru_node()")

How about the following patch? It is low overhead on x86_64. Even on relaxed memory mode, I think it is also lower overhead since it avoid a store operation to nlru->lock.

We do not need to insert a smp_wmb() into the list_lru_add() since spin_lock() always implies at least a load acquiring semantics.

Thanks.

mm/list_lru.c +0 -0
index c669d87001a6..0e58374b629b 100644
@@ -397,8 +397,11 @@ static void memcg_reparent_list_lru_node(struct

list_lru *lru, int nid,

        /*
         * If there is no lru entry in this nlru, we can skip it immediately.
         */

- if (!READ_ONCE(nlru->nr_items))
- return;
+ if (!READ_ONCE(nlru->nr_items)) {
+ smp_rmb();
+ if (!spin_is_locked(&nlru->lock))
+ return;
+ }

/*
* Since list_lru_{add,del} may be called under an IRQ-safe lock,