mm/list_lru: drain before clearing xarray entry on reparent
Subsystems (4)
PATCH v2 1/1 mm/list_lru: drain before clearing xarray entry on reparent
memcg_reparent_list_lrus() clears the dying memcg's xarray entry with xas_store(&xas, NULL) before reparenting its per-node lists into the parent. This opens a window where a concurrent list_lru_del() arriving for the dying memcg sees xa_load() == NULL, walks to the parent in lock_list_lru_of_memcg(), takes the parent's per-node lock, and calls list_del_init() on an item still physically linked on the dying memcg's list.
If another in-flight thread holds the dying memcg's per-node lock at the same moment (another list_lru_del, or a list_lru_walk_one running an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.
Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent list_lru op that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or sees LONG_MIN and walks to the parent, where the items now live.
---
Changes since v1:
- Use xa_erase_irq() instead of xa_erase() (Sashiko & Claude).
- Added comment on CSS_DYING check in memcg_list_lru_alloc avoiding a new mlru allocation.
mm/list_lru.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
mm/list_lru.c +12 -9
@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parenmutex_lock (&list_lrus_mutex);list_for_each_entry (lru, &memcg_list_lrus, list) {struct list_lru_memcg *mlru; -XA_STATE (xas, &lru->xa, memcg->kmemcg_id);/* -* Lock the Xarray to ensure no on going list_lru_memcg -* allocation and further allocation will see css_is_dying(). +* css_is_dying() check in memcg_list_lru_alloc() avoids +* allocating a new mlru since CSS_DYING is already set for this +* memcg a rcu grace period ago. */ -xas_lock_irq (&xas); - mlru =xas_store (&xas,NULL ); -xas_unlock_irq (&xas); + mlru =xa_load (&lru->xa, memcg->kmemcg_id);if (!mlru)continue ;/* -* With Xarray value set to NULL, holding the lru lock below -* prevents list_lru_{add,del,isolate} from touching the lru, -* safe to reparent. +* Reparent each per-node list and mark the child dead +* (LONG_MIN) before clearing xarray entry otherwise a +* concurrent list_lru_del() may corrupt the list if it arrives +* after xarray clear but before reparenting as +* lock_list_lru_of_memcg will acquire parent's lock while the +* item is still on child's list. */ for_each_node (i)memcg_reparent_list_lru_one (lru, i, &mlru->node[i], parent); +xa_erase_irq (&lru->xa, memcg->kmemcg_id); +/* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further
Signature · 1 line
2.53.0-Meta
Quoted 78 lines · Shakeel Butt
memcg_reparent_list_lrus()clears the dying memcg's xarray entry withxas_store(&xas, NULL)before reparenting its per-node lists into the parent. This opens a window where a concurrentlist_lru_del()arriving for the dying memcg seesxa_load()== NULL, walks to the parent inlock_list_lru_of_memcg(), takes the parent's per-node lock, and callslist_del_init()on an item still physically linked on the dying memcg's list.If another in-flight thread holds the dying memcg's per-node lock at the same moment (another
list_lru_del, or alist_lru_walk_onerunning an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent
list_lruop that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or seesLONG_MINand walks to the parent, where the items now live.Fixes fb56fdf8b9a2 ("mm/list_lru: split the lock to per-cgroup scope")Signed-off-by Shakeel Butt <[email protected]>Reported-by Chris Mason <[email protected]>---
Changes since v1:
- Usexa_erase_irq()instead ofxa_erase()(Sashiko & Claude).
- Added comment onCSS_DYINGcheck inmemcg_list_lru_allocavoiding a new mlru allocation.mm/list_lru.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)mm/list_lru.c +12
-10@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parenmutex_lock (&list_lrus_mutex);list_for_each_entry (lru, &memcg_list_lrus, list) {struct list_lru_memcg *mlru; -XA_STATE (xas, &lru->xa, memcg->kmemcg_id);/* -* Lock the Xarray to ensure no on going list_lru_memcg -* allocation and further allocation will see css_is_dying(). +* css_is_dying() check in memcg_list_lru_alloc() avoids +* allocating a new mlru since CSS_DYING is already set for this +* memcg a rcu grace period ago. */ -xas_lock_irq (&xas); - mlru =xas_store (&xas,NULL ); -xas_unlock_irq (&xas); + mlru =xa_load (&lru->xa, memcg->kmemcg_id);if (!mlru)continue ;/* -* With Xarray value set to NULL, holding the lru lock below -* prevents list_lru_{add,del,isolate} from touching the lru, -* safe to reparent. +* Reparent each per-node list and mark the child dead +* (LONG_MIN) before clearing xarray entry otherwise a +* concurrent list_lru_del() may corrupt the list if it arrives +* after xarray clear but before reparenting as +* lock_list_lru_of_memcg will acquire parent's lock while the +* item is still on child's list. */ for_each_node (i)memcg_reparent_list_lru_one (lru, i, &mlru->node[i], parent); +xa_erase_irq (&lru->xa, memcg->kmemcg_id); +/* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further --
Quoted 3 lines
2.53.0-Meta
Nice catch! Thanks a lot!
Quoted 43 lines · Shakeel Butt
memcg_reparent_list_lrus()clears the dying memcg's xarray entry withxas_store(&xas, NULL)before reparenting its per-node lists into the parent. This opens a window where a concurrentlist_lru_del()arriving for the dying memcg seesxa_load()== NULL, walks to the parent inlock_list_lru_of_memcg(), takes the parent's per-node lock, and callslist_del_init()on an item still physically linked on the dying memcg's list.If another in-flight thread holds the dying memcg's per-node lock at the same moment (another
list_lru_del, or alist_lru_walk_onerunning an isolate callback), both threads modify ->next/->prev pointers on the same physical list under different locks. Adjacent items can corrupt each other's links.Fix it by reversing the order: reparent each per-node list and mark the child's list lru dead and then clear the xarray entry. Any concurrent
list_lruop that finds the still-set xarray entry either takes the dying memcg's per-node lock (synchronizing with the drain) or seesLONG_MINand walks to the parent, where the items now live.Fixes fb56fdf8b9a2 ("mm/list_lru: split the lock to per-cgroup scope")Signed-off-by Shakeel Butt <[email protected]>Reported-by Chris Mason <[email protected]>---
Changes since v1:
- Usexa_erase_irq()instead ofxa_erase()(Sashiko & Claude).
- Added comment onCSS_DYINGcheck inmemcg_list_lru_allocavoiding a new mlru allocation.
mm/list_lru.c| 21 ++++++++++++---------1 file changed, 12 insertions(+), 9 deletions(-)mm/list_lru.c +0
-0@@ -473,26 +473,29 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parenmutex_lock (&list_lrus_mutex);
mm/list_lru.c void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren · line 480
Quoted 10 lines
mm/list_lru.c +3
-3list_for_each_entry (lru, &memcg_list_lrus, list) {struct list_lru_memcg *mlru; -XA_STATE (xas, &lru->xa, memcg->kmemcg_id);/* -* Lock the Xarray to ensure no on going list_lru_memcg -* allocation and further allocation will see css_is_dying(). +* css_is_dying() check in memcg_list_lru_alloc() avoids +* allocating a new mlru since CSS_DYING is already set for this +* memcg a rcu grace period ago.
I see. xas_lock_irqsave() in memcg_list_lru_alloc() functions as an RCU read lock.
Thanks.
Quoted 30 lines
mm/list_lru.c +9
-6*/ -xas_lock_irq (&xas); - mlru =xas_store (&xas,NULL ); -xas_unlock_irq (&xas); + mlru =xa_load (&lru->xa, memcg->kmemcg_id);if (!mlru)continue ;/* -* With Xarray value set to NULL, holding the lru lock below -* prevents list_lru_{add,del,isolate} from touching the lru, -* safe to reparent. +* Reparent each per-node list and mark the child dead +* (LONG_MIN) before clearing xarray entry otherwise a +* concurrent list_lru_del() may corrupt the list if it arrives +* after xarray clear but before reparenting as +* lock_list_lru_of_memcg will acquire parent's lock while the +* item is still on child's list. */ for_each_node (i)memcg_reparent_list_lru_one (lru, i, &mlru->node[i], parent); +xa_erase_irq (&lru->xa, memcg->kmemcg_id); +/* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further Signature · 1 line
2.53.0-Meta