mm: list_lru: fix NULL deref when a folio moves to the root memcg
Subsystems (2)
PATCH 1/1 mm: list_lru: fix NULL deref when a folio moves to the root memcg
folio_memcg_list_lru_alloc() makes sure that a folio's memcg has its own lists in a list_lru, such as the THP deferred split queue. It looks up the folio's memcg twice: a fast path under RCU checks if the lists exist, and a slow path takes a reference and allocates them. The fast path returns early for the root memcg, which uses the global lists. The slow path does not check for the root.
The memcg can change between the two lookups: when a cgroup is removed, its folios are moved to the parent memcg. If the folio is moved up to the root in between, the slow path gets the root, and __memcg_list_lru_alloc() dereferences the root's parent, which is NULL.
Users see a NULL pointer dereference oops while a THP is allocated, such as in a page fault, in khugepaged or at swap-in. tmpfs huge pages are affected too, and for them the oops can also happen in reclaim, for example in kswapd. The task that hits the oops is killed, or the machine panics if panic_on_oops is set.
The race is narrow: the folio has to be moved up to the root right between the two lookups. For example, its cgroup is directly below the root and is removed, or has its memory controller disabled, at that moment.
Fix it by skipping the root memcg in the slow path too. The root needs no lists of its own.
--- mm/list_lru.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/list_lru.c +8 -2
@@ -645,9 +645,15 @@ int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,if (likely (res))return 0 ; -/* Allocation may block, pin the memcg */ +/* +* Allocation may block, so pin the memcg. If the folio's cgroup +* was removed after the check above, the folio has moved up to a +* parent memcg, which can be the root. The root uses the global +* lists and has no parent, so skip it. +*/ memcg =get_mem_cgroup_from_folio (folio); - res =__memcg_list_lru_alloc (memcg, lru, gfp); +if (!mem_cgroup_is_root (memcg)) + res =__memcg_list_lru_alloc (memcg, lru, gfp);mem_cgroup_put (memcg);return res; }
Signature · 1 line
2.53.0-Meta
Muchun Song
reviewerQuoted 46 lines · Shakeel Butt
folio_memcg_list_lru_alloc()makes sure that a folio's memcg has its own lists in alist_lru, such as the THP deferred split queue. It looks up the folio's memcg twice: a fast path under RCU checks if the lists exist, and a slow path takes a reference and allocates them. The fast path returns early for the root memcg, which uses the global lists. The slow path does not check for the root.The memcg can change between the two lookups: when a cgroup is removed, its folios are moved to the parent memcg. If the folio is moved up to the root in between, the slow path gets the root, and
__memcg_list_lru_alloc()dereferences the root's parent, which is NULL.Users see a NULL pointer dereference oops while a THP is allocated, such as in a page fault, in khugepaged or at swap-in. tmpfs huge pages are affected too, and for them the oops can also happen in reclaim, for example in kswapd. The task that hits the oops is killed, or the machine panics if
panic_on_oopsis set.The race is narrow: the folio has to be moved up to the root right between the two lookups. For example, its cgroup is directly below the root and is removed, or has its memory controller disabled, at that moment.
Fix it by skipping the root memcg in the slow path too. The root needs no lists of its own.
Fixes ae64f07a6a40 ("mm: list_lru: introduce folio_memcg_list_lru_alloc()")Reported-by Chris Mason <[email protected]>Signed-off-by Shakeel Butt <[email protected]>---
mm/list_lru.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)mm/list_lru.c +0
-0@@ -645,9 +645,15 @@ int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,if (likely (res))return 0 ;
mm/list_lru.c int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, · line 655
Quoted 10 lines
mm/list_lru.c +7
-2-/* Allocation may block, pin the memcg */ +/* +* Allocation may block, so pin the memcg. If the folio's cgroup +* was removed after the check above, the folio has moved up to a +* parent memcg, which can be the root. The root uses the global +* lists and has no parent, so skip it. +*/ memcg =get_mem_cgroup_from_folio (folio); - res =__memcg_list_lru_alloc (memcg, lru, gfp); +if (!mem_cgroup_is_root (memcg))
How about using memcg_list_lru_allocated() to check this, it's consistent with the above check?
Thanks,
Muchun
Quoted 7 lines
mm/list_lru.c +1
-0+ res =__memcg_list_lru_alloc (memcg, lru, gfp);mem_cgroup_put (memcg);return res; }Signature · 1 line
2.53.0-Meta
Quoted 46 lines · Muchun Song
Quoted 46 lines · Shakeel Butt
folio_memcg_list_lru_alloc()makes sure that a folio's memcg has its own lists in alist_lru, such as the THP deferred split queue. It looks up the folio's memcg twice: a fast path under RCU checks if the lists exist, and a slow path takes a reference and allocates them. The fast path returns early for the root memcg, which uses the global lists. The slow path does not check for the root.The memcg can change between the two lookups: when a cgroup is removed, its folios are moved to the parent memcg. If the folio is moved up to the root in between, the slow path gets the root, and
__memcg_list_lru_alloc()dereferences the root's parent, which is NULL.Users see a NULL pointer dereference oops while a THP is allocated, such as in a page fault, in khugepaged or at swap-in. tmpfs huge pages are affected too, and for them the oops can also happen in reclaim, for example in kswapd. The task that hits the oops is killed, or the machine panics if
panic_on_oopsis set.The race is narrow: the folio has to be moved up to the root right between the two lookups. For example, its cgroup is directly below the root and is removed, or has its memory controller disabled, at that moment.
Fix it by skipping the root memcg in the slow path too. The root needs no lists of its own.
Fixes ae64f07a6a40 ("mm: list_lru: introduce folio_memcg_list_lru_alloc()")Reported-by Chris Mason <[email protected]>Signed-off-by Shakeel Butt <[email protected]>---
mm/list_lru.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)mm/list_lru.c +0
-0@@ -645,9 +645,15 @@ int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,if (likely (res))return 0 ;
Quoted 13 lines
mm/list_lru.c int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, · line 655
Quoted 10 lines
mm/list_lru.c +7
-2-/* Allocation may block, pin the memcg */ +/* +* Allocation may block, so pin the memcg. If the folio's cgroup +* was removed after the check above, the folio has moved up to a +* parent memcg, which can be the root. The root uses the global +* lists and has no parent, so skip it. +*/ memcg =get_mem_cgroup_from_folio (folio); - res =__memcg_list_lru_alloc (memcg, lru, gfp); +if (!mem_cgroup_is_root (memcg))How about using
memcg_list_lru_allocated()to check this, it's consistent with the above check?
Yes, that makes sense. Will send out v2.
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [High] mm/list_lru: infinite loop in…
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [High] mm/list_lru: infinite loop in __memcg_list_lru_alloc() on xas_nomem() failure [1]
--
commit 17fdd494fc11d51fee607883de9cf2593b4f8f83
Author: Shakeel Butt <[email protected]>
mm: list_lru: fix NULL deref when a folio moves to the root memcg
This commit skips calling __memcg_list_lru_alloc() in the slow path of folio_memcg_list_lru_alloc() if the folio's memcg has been reparented to the root memcg.
Quoted 12 lines
mm/list_lru.c +3
-1@@ -645,9 +645,15 @@ int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,if (likely (res))return 0 ; -/* Allocation may block, pin the memcg */ +/* +* Allocation may block, so pin the memcg. If the folio's cgroup +* was removed after the check above, the folio has moved up to a
mm/list_lru.c int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, · line 659
Quoted 10 lines
mm/list_lru.c +5
-1+* parent memcg, which can be the root. The root uses the global +* lists and has no parent, so skip it. +*/ memcg =get_mem_cgroup_from_folio (folio); - res =__memcg_list_lru_alloc (memcg, lru, gfp); +if (!mem_cgroup_is_root (memcg)) + res =__memcg_list_lru_alloc (memcg, lru, gfp);mem_cgroup_put (memcg);return res; }
[Severity: High]
This problem wasn't introduced by this patch, but can __memcg_list_lru_alloc() enter an infinite loop [1] when xas_nomem() fails while populating an ancestor cgroup?
mm/list_lru.c:__memcg_list_lru_alloc() {
...
do {
pos = memcg;
parent = parent_mem_cgroup(pos);
while (!memcg_list_lru_allocated(parent, lru)) {
pos = parent;
parent = parent_mem_cgroup(pos);
}
...
xas_set(&xas, pos->kmemcg_id);
do {
xas_lock_irqsave(&xas, flags);
if (!xas_load(&xas) && !css_is_dying(&pos->css)) {
xas_store(&xas, mlru);
if (!xas_error(&xas))
mlru = NULL;
}
xas_unlock_irqrestore(&xas, flags);
} while (xas_nomem(&xas, gfp));
} while (pos != memcg && !css_is_dying(&pos->css));
...
return xas_error(&xas);
}When storing mlru for an ancestor (pos != memcg) needs a new xa_node, xas_store() sets xas to -ENOMEM and leaves pos unpopulated in lru->xa. If xas_nomem() then fails to allocate a node (for example under memory pressure, or when gfp &= GFP_RECLAIM_MASK strips GFP_NOWAIT), it returns false with xas still in the error state.
Because the outer loop does not check !xas_error(&xas), it repeats for the same unpopulated ancestor pos, where xas_set() resets xas.xa_node to XAS_RESTART and clears the error, looping forever and causing a soft or hard lockup.
[1] https://sashiko.dev/#/bug/linux-21c1dba9-8d6f-4368-b38b-9a57cacb056f
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1