Skip to contentSkip to conversation

mm/list_lru: drain before clearing xarray entry on reparent

PATCH Applied Shakeel Butt 3 participants 3 messages Jun 1 – Jun 2

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm/list_lru: drain before clearing xarray entry on reparent

memcg_reparent_list_lrus() clears the dying memcg's xarray entry with xas_store(&xas, NULL) before reparenting its per-node lists into the parent. This opens a window where a concurrent list_lru_del() arriving for the dying memcg sees xa_load() == NULL, walks to the parent in lock_list_lru_of_memcg(), takes the parent's per-node lock, and calls list_del_init() on an item still physically linked on the dying memcg's list.

If another in-flight thread holds the dying memcg's per-node lock at the same moment (another list_lru_del, or a list_lru_walk_one running an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.

Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent list_lru op that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or sees LONG_MIN and walks to the parent, where the items now live.

Fixes fb56fdf8b9a2 ("mm/list_lru: split the lock to per-cgroup scope")

---
Changes since v1:
- Use xa_erase_irq() instead of xa_erase() (Sashiko & Claude).
- Added comment on CSS_DYING check in memcg_list_lru_alloc avoiding a new mlru allocation.

 mm/list_lru.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
mm/list_lru.c +12 -9
index dd29bcf8eb5f..d454bce9a78e 100644
@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
 	mutex_lock(&list_lrus_mutex);
 	list_for_each_entry(lru, &memcg_list_lrus, list) {
 		struct list_lru_memcg *mlru;
-		XA_STATE(xas, &lru->xa, memcg->kmemcg_id);
 
 		/*
-		 * Lock the Xarray to ensure no on going list_lru_memcg
-		 * allocation and further allocation will see css_is_dying().
+		 * css_is_dying() check in memcg_list_lru_alloc() avoids
+		 * allocating a new mlru since CSS_DYING is already set for this
+		 * memcg a rcu grace period ago.
 		 */
-		xas_lock_irq(&xas);
-		mlru = xas_store(&xas, NULL);
-		xas_unlock_irq(&xas);
+		mlru = xa_load(&lru->xa, memcg->kmemcg_id);
 		if (!mlru)
 			continue;
 
 		/*
-		 * With Xarray value set to NULL, holding the lru lock below
-		 * prevents list_lru_{add,del,isolate} from touching the lru,
-		 * safe to reparent.
+		 * Reparent each per-node list and mark the child dead
+		 * (LONG_MIN) before clearing xarray entry otherwise a
+		 * concurrent list_lru_del() may corrupt the list if it arrives
+		 * after xarray clear but before reparenting as
+		 * lock_list_lru_of_memcg will acquire parent's lock while the
+		 * item is still on child's list.
 		 */
 		for_each_node(i)
 			memcg_reparent_list_lru_one(lru, i, &mlru->node[i], parent);
 
+		xa_erase_irq(&lru->xa, memcg->kmemcg_id);
+
 		/*
 		 * Here all list_lrus corresponding to the cgroup are guaranteed
 		 * to remain empty, we can safely free this lru, any further
Signature · 1 line
2.53.0-Meta
Reviewed-by on v2 1/1
#1 lore raw
Quoted 78 lines · Shakeel Butt

memcg_reparent_list_lrus() clears the dying memcg's xarray entry with xas_store(&xas, NULL) before reparenting its per-node lists into the parent. This opens a window where a concurrent list_lru_del() arriving for the dying memcg sees xa_load() == NULL, walks to the parent in lock_list_lru_of_memcg(), takes the parent's per-node lock, and calls list_del_init() on an item still physically linked on the dying memcg's list.

If another in-flight thread holds the dying memcg's per-node lock at the same moment (another list_lru_del, or a list_lru_walk_one running an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.

Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent list_lru op that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or sees LONG_MIN and walks to the parent, where the items now live.

Fixes fb56fdf8b9a2 ("mm/list_lru: split the lock to per-cgroup scope")

---
Changes since v1:
- Use xa_erase_irq() instead of xa_erase() (Sashiko & Claude).
- Added comment on CSS_DYING check in memcg_list_lru_alloc avoiding a new mlru allocation.

 mm/list_lru.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
mm/list_lru.c +12 -10
index dd29bcf8eb5f..d454bce9a78e 100644
@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
        mutex_lock(&list_lrus_mutex);
        list_for_each_entry(lru, &memcg_list_lrus, list) {
                struct list_lru_memcg *mlru;
-               XA_STATE(xas, &lru->xa, memcg->kmemcg_id);
 
                /*
-                * Lock the Xarray to ensure no on going list_lru_memcg
-                * allocation and further allocation will see css_is_dying().
+                * css_is_dying() check in memcg_list_lru_alloc() avoids
+                * allocating a new mlru since CSS_DYING is already set for this
+                * memcg a rcu grace period ago.
                 */
-               xas_lock_irq(&xas);
-               mlru = xas_store(&xas, NULL);
-               xas_unlock_irq(&xas);
+               mlru = xa_load(&lru->xa, memcg->kmemcg_id);
                if (!mlru)
                        continue;
 
                /*
-                * With Xarray value set to NULL, holding the lru lock below
-                * prevents list_lru_{add,del,isolate} from touching the lru,
-                * safe to reparent.
+                * Reparent each per-node list and mark the child dead
+                * (LONG_MIN) before clearing xarray entry otherwise a
+                * concurrent list_lru_del() may corrupt the list if it arrives
+                * after xarray clear but before reparenting as
+                * lock_list_lru_of_memcg will acquire parent's lock while the
+                * item is still on child's list.
                 */
                for_each_node(i)
                        memcg_reparent_list_lru_one(lru, i, &mlru->node[i], parent);
 
+               xa_erase_irq(&lru->xa, memcg->kmemcg_id);
+
                /*
                 * Here all list_lrus corresponding to the cgroup are guaranteed
                 * to remain empty, we can safely free this lru, any further
--
Quoted 3 lines

2.53.0-Meta

Nice catch! Thanks a lot!

reviewer Acked-by on v2 1/1
#2 lore raw
In reply to Shakeel Butt
Quoted 43 lines · Shakeel Butt

memcg_reparent_list_lrus() clears the dying memcg's xarray entry with xas_store(&xas, NULL) before reparenting its per-node lists into the parent. This opens a window where a concurrent list_lru_del() arriving for the dying memcg sees xa_load() == NULL, walks to the parent in lock_list_lru_of_memcg(), takes the parent's per-node lock, and calls list_del_init() on an item still physically linked on the dying memcg's list.

If another in-flight thread holds the dying memcg's per-node lock at the same moment (another list_lru_del, or a list_lru_walk_one running an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.

Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent list_lru op that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or sees LONG_MIN and walks to the parent, where the items now live.

Fixes fb56fdf8b9a2 ("mm/list_lru: split the lock to per-cgroup scope")

---
Changes since v1:
- Use xa_erase_irq() instead of xa_erase() (Sashiko & Claude).
- Added comment on CSS_DYING check in memcg_list_lru_alloc avoiding a new mlru allocation.

mm/list_lru.c | 21 ++++++++++++---------

1 file changed, 12 insertions(+), 9 deletions(-)
mm/list_lru.c +0 -0
index dd29bcf8eb5f..d454bce9a78e 100644
@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
 	mutex_lock(&list_lrus_mutex);

mm/list_lru.c void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren · line 480

Quoted 10 lines
mm/list_lru.c +3 -3
 	list_for_each_entry(lru, &memcg_list_lrus, list) {
 		struct list_lru_memcg *mlru;
- 		XA_STATE(xas, &lru->xa, memcg->kmemcg_id);
 
 		/*
-		 * Lock the Xarray to ensure no on going list_lru_memcg
-		 * allocation and further allocation will see css_is_dying().
+		 * css_is_dying() check in memcg_list_lru_alloc() avoids
+		 * allocating a new mlru since CSS_DYING is already set for this
+		 * memcg a rcu grace period ago.

I see. xas_lock_irqsave() in memcg_list_lru_alloc() functions as an RCU read lock.

Thanks.

Quoted 30 lines
mm/list_lru.c +9 -6
 		 */
- 		xas_lock_irq(&xas);
- 		mlru = xas_store(&xas, NULL);
- 		xas_unlock_irq(&xas);
+ 		mlru = xa_load(&lru->xa, memcg->kmemcg_id);
 		if (!mlru)
 			continue;
 
 		/*
-		 * With Xarray value set to NULL, holding the lru lock below
-		 * prevents list_lru_{add,del,isolate} from touching the lru,
-		 * safe to reparent.
+		 * Reparent each per-node list and mark the child dead
+		 * (LONG_MIN) before clearing xarray entry otherwise a
+		 * concurrent list_lru_del() may corrupt the list if it arrives
+		 * after xarray clear but before reparenting as
+		 * lock_list_lru_of_memcg will acquire parent's lock while the
+		 * item is still on child's list.
 */
 		for_each_node(i)
 			memcg_reparent_list_lru_one(lru, i, &mlru->node[i], parent);
 
+ 		xa_erase_irq(&lru->xa, memcg->kmemcg_id);
+
 		/*
 		 * Here all list_lrus corresponding to the cgroup are guaranteed
 		 * to remain empty, we can safely free this lru, any further
Signature · 1 line
2.53.0-Meta