mm: list_lru: don't use a new memcg's lists for an offline memcg
Subsystems (2)
PATCH 1/1 mm: list_lru: don't use a new memcg's lists for an offline memcg
A memcg's list_lru lists are found by its kmemcg_id, which is its private ID. Offlining a memcg moves its lists to the parent and frees the ID right away, with no RCU grace period. The next new memcg can get the same ID and allocate its own lists at that index.
A caller that holds only the RCU read lock, such as __folio_unqueue_deferred_split(), can still hold the offline memcg. If it is preempted until the ID is reused, it then locks the new memcg's list and unlinks a folio that now sits on the parent's list, without holding the parent's lock. A concurrent change to the parent's list can then leave a pointer to a freed folio on it.
Users see list corruption reports, crashes or silent memory corruption when memory cgroups are removed and created while THPs are freed, on kernels with preemptible RCU, which is the default on x86 and arm64. They can also see a warning from memcg_reparent_list_lru_one() when the new memcg is removed, because the wrong unlink made its count negative.
Fix it by recording the owning memcg in each list_lru_memcg, and by treating the index as empty when its lists belong to another memcg. The lookup then moves up to the parent, whose list holds the folio.
--- include/linux/list_lru.h | 2 ++ mm/list_lru.c | 18 +++++++++++------- 2 files changed, 13 insertions(+), 7 deletions(-)
include/linux/list_lru.h +2 -0
@@ -38,6 +38,8 @@ struct list_lru_one {struct list_lru_memcg {struct rcu_head rcu; +/* owning memcg, only compared and never dereferenced */ +const void *memcg;/* array of per cgroup per node lists, indexed by node id */ struct list_lru_one node[]; };
mm/list_lru.c +11 -7
@@ -72,12 +72,15 @@ static int lru_shrinker_id(struct list_lru *lru) }static inline struct list_lru_one * -list_lru_from_memcg_idx (struct list_lru *lru,int nid,int idx) +list_lru_from_memcg (struct list_lru *lru,int nid,struct mem_cgroup *memcg) { +int idx =memcg_kmem_id (memcg); +if (list_lru_memcg_aware (lru) && idx >=0 ) {struct list_lru_memcg *mlru =xa_load (&lru->xa, idx); -return mlru ? &mlru->node[nid] :NULL ; +/* Lists at a reused index belong to another memcg. */ +return mlru && mlru->memcg == memcg ? &mlru->node[nid] :NULL ; }return &lru->node[nid].lru; }
@@ -91,7 +94,7 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid,rcu_read_lock (); again: - l =list_lru_from_memcg_idx (lru, nid,memcg_kmem_id (*memcg)); + l =list_lru_from_memcg (lru, nid, *memcg);if (likely (l)) {lock_list_lru (l, irq, irq_flags);if (likely (READ_ONCE (l->nr_items) !=LONG_MIN )) {
@@ -132,7 +135,7 @@ static inline bool list_lru_memcg_aware(struct list_lru *lru) }static inline struct list_lru_one * -list_lru_from_memcg_idx (struct list_lru *lru,int nid,int idx) +list_lru_from_memcg (struct list_lru *lru,int nid,struct mem_cgroup *memcg) {return &lru->node[nid].lru; }
@@ -313,7 +316,7 @@ unsigned long list_lru_count_one(struct list_lru *lru,long count;rcu_read_lock (); - l =list_lru_from_memcg_idx (lru, nid,memcg_kmem_id (memcg)); + l =list_lru_from_memcg (lru, nid, memcg); count = l ?READ_ONCE (l->nr_items) :0 ;rcu_read_unlock ();
@@ -502,11 +505,10 @@ static void memcg_reparent_list_lru_one(struct list_lru *lru, int nid,struct list_lru_one *src, struct mem_cgroup *dst_memcg) { - int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *dst;spin_lock_irq (&src->lock); - dst =list_lru_from_memcg_idx (lru, nid, dst_idx); + dst =list_lru_from_memcg (lru, nid, dst_memcg);spin_lock_nested (&dst->lock,SINGLE_DEPTH_NESTING );list_splice_init (&src->list, &dst->list);
@@ -602,6 +604,8 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,if (!mlru)return -ENOMEM ; } +/* Set before the lists are published, and never changed after. */ + mlru->memcg = pos;xas_set (&xas, pos->kmemcg_id); do { xas_lock_irqsave(&xas, flags);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm: list_lru: missing mlru->memcg check in memcg_list_lru_allocated() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm: list_lru: missing mlru->memcg check in memcg_list_lru_allocated()
--
commit d213b82b7b4ca4978c858df10a2a8cdea14626e4
Author: Shakeel Butt <[email protected]>
mm: list_lru: don't use a new memcg's lists for an offline memcg
This commit records the owning memcg in struct list_lru_memcg and checks it in list_lru_from_memcg() so that an offline memcg whose kmemcg_id was reused does not use the new memcg's lists.
Quoted 22 lines
mm/list_lru.c +5
-2@@ -72,12 +72,15 @@ static int lru_shrinker_id(struct list_lru *lru) }static inline struct list_lru_one * -list_lru_from_memcg_idx (struct list_lru *lru,int nid,int idx) +list_lru_from_memcg (struct list_lru *lru,int nid,struct mem_cgroup *memcg) { +int idx =memcg_kmem_id (memcg); +if (list_lru_memcg_aware (lru) && idx >=0 ) {struct list_lru_memcg *mlru =xa_load (&lru->xa, idx); -return mlru ? &mlru->node[nid] :NULL ; +/* Lists at a reused index belong to another memcg. */ +return mlru && mlru->memcg == memcg ? &mlru->node[nid] :NULL ; }return &lru->node[nid].lru; }
[ ... ]
mm/list_lru.c static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, · line 611
Quoted 9 lines
mm/list_lru.c +2
-0@@ -602,6 +604,8 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,if (!mlru)return -ENOMEM ; } +/* Set before the lists are published, and never changed after. */ + mlru->memcg = pos;xas_set (&xas, pos->kmemcg_id); do { xas_lock_irqsave(&xas, flags);
[Severity: High]
Does memcg_list_lru_allocated() also need to check mlru->memcg == memcg?
mm/list_lru.c:memcg_list_lru_allocated() {
int idx = memcg->kmemcg_id;
return idx < 0 || xa_load(&lru->xa, idx);
}Consider a hierarchy root -> P -> C where P is online and neither P nor C has allocated a list_lru_memcg in lru->xa yet:
CPU0 CPU1folio_memcg_list_lru_alloc(folio, lru)
rcu_read_lock()
memcg = folio_memcg(folio) /* C */
mem_cgroup_css_offline(C)
memcg_reparent_objcgs(C) mem_cgroup_private_id_kill(C)
/* M2 gets C's kmemcg_id and
* allocates lru->xa[idx] */
memcg_list_lru_allocated(C, lru)
xa_load(&lru->xa, idx) /* M2's mlru */
/* returns true; P is not allocated */ rcu_read_unlock()
deferred_split_folio(folio)
folio_lru_add(folio)
lock_list_lru_of_memcg(lru, nid, &P)
list_lru_from_memcg(lru, nid, P) /* NULL */ VM_WARN_ON(!css_is_dying(&P->css))
/* walks up to root and queues folio on root's list */
Once P later allocates its own list_lru_memcg, __folio_unqueue_deferred_split() locks P's list_lru_one while unlinking the folio from root's list without holding root's lock, corrupting the list. A matching RCU read lock around xa_load() and the mlru->memcg == memcg check in memcg_list_lru_allocated() would avoid this.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 40 lines · On Fri, Nov 13, 2026 at 12:15:11AM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm:list_lru: missingmlru->memcgcheck inmemcg_list_lru_allocated()
--commit
d213b82b7b4ca4978c858df10a2a8cdea14626e4
Author: Shakeel Butt <[email protected]>mm:
list_lru: don't use a new memcg's lists for an offline memcgThis commit records the owning memcg in struct
list_lru_memcgand checks it inlist_lru_from_memcg()so that an offline memcg whosekmemcg_idwas reused does not use the new memcg's lists.Quoted 22 lines
mm/list_lru.c +5
-2@@ -72,12 +72,15 @@ static int lru_shrinker_id(struct list_lru *lru) }static inline struct list_lru_one * -list_lru_from_memcg_idx (struct list_lru *lru,int nid,int idx) +list_lru_from_memcg (struct list_lru *lru,int nid,struct mem_cgroup *memcg) { +int idx =memcg_kmem_id (memcg); +if (list_lru_memcg_aware (lru) && idx >=0 ) {struct list_lru_memcg *mlru =xa_load (&lru->xa, idx); -return mlru ? &mlru->node[nid] :NULL ; +/* Lists at a reused index belong to another memcg. */ +return mlru && mlru->memcg == memcg ? &mlru->node[nid] :NULL ; }return &lru->node[nid].lru; }[ ... ]
Quoted 12 lines
mm/list_lru.c static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, · line 611
Quoted 9 lines
mm/list_lru.c +2
-0@@ -602,6 +604,8 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,if (!mlru)return -ENOMEM ; } +/* Set before the lists are published, and never changed after. */ + mlru->memcg = pos;xas_set (&xas, pos->kmemcg_id); do { xas_lock_irqsave(&xas, flags);[Severity: High]
Doesmemcg_list_lru_allocated()also need to checkmlru->memcg== memcg?
Yes this is correct and Andrew already have a fix for this.
Quoted 5 lines · On Thu, 12 Nov 2026 18:37:10 -0800 Shakeel Butt <[email protected]> wrote:
Quoted 2 lines
[Severity: High]
Doesmemcg_list_lru_allocated()also need to checkmlru->memcg== memcg?Yes this is correct and Andrew already have a fix for this.
From: Andrew Morton <[email protected]>
Subject: mm/list_lru: verify memcg owner before skipping allocation
Date: Thu, 12 Nov 2026 16:08:59 -0800
list_lru_from_memcg() rejects an entry at a reused kmemcg_id when it belongs to another memcg. memcg_list_lru_allocated() still treats any entry at that index as a list for the requested memcg. Its callers can therefore skip the allocation of that memcg's ancestor lists.
With child C and parent P's lists both unallocated, a reader can retain C while it is offlined. If an unrelated new memcg reuses C's ID and allocates its own list, the reader mistakes that entry for C's and skips allocation of P's list. Once the folio's objcg points to P, deferred_split_folio() can add it to the root list. If P later gets a list, unqueue locks P's list while unlinking the folio from root's list, which can corrupt that list under concurrent operations.
Require the entry to belong to the requested memcg before returning true. Hold RCU over the XArray load and owner comparison because this helper is also called outside the folio fast path's RCU section.
---
mm/list_lru.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
mm/list_lru.c renamed from mm/list_lru.c~mm-list_lru-dont-use-a-new-memcgs-lists-for-an-offline-memcg-fix +11 -1
@@ -568,9 +568,19 @@ void memcg_reparent_list_lrus(struct memstatic inline bool memcg_list_lru_allocated (struct mem_cgroup *memcg,struct list_lru *lru) { +struct list_lru_memcg *mlru; +bool allocated;int idx = memcg->kmemcg_id; -return idx <0 ||xa_load (&lru->xa, idx); +if (idx <0 ) +return true; + +rcu_read_lock (); + mlru =xa_load (&lru->xa, idx); + allocated = mlru && mlru->memcg == memcg; +rcu_read_unlock (); + +return allocated; } static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
_