mm: mglru: splice all generations when reparenting a memcg

Noah reported list_del corruption on single-node hosts running 7.2.11, which already has commit bf4ade7dbd76 ("memcg: keep folio's objcg same as its node"). The bad pointer is an MGLRU list head in a freed mem_cgroup_per_node:

slab mem_cgroup_per_node start ff31bfa36a3c9e00 pointer offset 528 size 1920
list_del corruption. next->prev should be ffacf881ccc70e08, but was ff31bfa36a3ca010. (next=ff31bfa36a3ca010)

When a memcg goes offline, __lru_gen_reparent_memcg() moves its MGLRU lists to the parent. It moves only the generations (gens) from min_seq to max_seq, and it puts each list at the same index in the parent.

A folio's gen in folio->flags can be ahead of the list it is on. In the parent, the same index can stand for an older gen, so the folio's gen can end up older than its list. Once the parent's min_seq passes that gen, sort_folio() puts the folio on the list for max_seq + 1, where it stays until the next aging.

If the parent goes offline before that aging, the folio is not moved to the grandparent. It stays on the parent's list after the parent is freed, and the next list_del() on it touches freed memory. This needs two cgroups on one branch to go offline one after the other, such as a container and then its pod.

The parent always has MAX_NR_GENS gens at this point, so every index is valid in it. Move all MAX_NR_GENS lists, not only the gens from min_seq to max_seq.

A reproducer that drives these steps through the lru_gen debugfs file gets a KASAN use-after-free on the freed mem_cgroup_per_node without this patch, and no report with it.

Fixes f304652609ea ("mm: vmscan: prepare for reparenting MGLRU folios")

Reported-by Noah Feldt <[email protected]>

Closes https://lore.kernel.org/all/[email protected]/ open here

Signed-off-by Shakeel Butt <[email protected]>

Cc <[email protected]>

---
 mm/vmscan.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

mm/vmscan.c +9 -4

index 6a1af9b92349..902616a3b755 100644
@@ -4751,19 +4751,25 @@ void max_lru_gen_memcg(struct mem_cgroup *memcg, int nid)
  * The same generation has different meanings in the parent and child memcg,
  * so this compromise method will cause the LRU inversion problem. But as the
  * system runs, this problem will be fixed automatically.
+ *
+ * Move all MAX_NR_GENS lists, not only the gens from min_seq to max_seq.
+ * An earlier reparenting can leave a folio whose gen is older than the list
+ * it is on. Once min_seq passes that gen, sort_folio() puts the folio on the
+ * list for max_seq + 1, where it stays until the next aging. If that list
+ * were skipped, the folio would stay behind on a memcg that is about to be
+ * freed. The parent has MAX_NR_GENS gens here, so every index is valid in it.
  */
 static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec *parent_lruvec,
                                     int zone, int type)
 {
        struct lru_gen_folio *child_lrugen, *parent_lrugen;
        enum lru_list lru = type * LRU_INACTIVE_FILE;
-       int i;
+       int gen;
 
        child_lrugen = &child_lruvec->lrugen;
        parent_lrugen = &parent_lruvec->lrugen;
 
-       for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
-               int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
+       for (gen = 0; gen < MAX_NR_GENS; gen++) {
                long nr_pages = child_lrugen->nr_pages[gen][type][zone];
                int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
                int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
--

2.51.0

Quoted 33 lines · Shakeel Butt

Noah reported list_del corruption on single-node hosts running 7.2.11, which already has commit bf4ade7dbd76 ("memcg: keep folio's objcg same as its node"). The bad pointer is an MGLRU list head in a freed mem_cgroup_per_node:

slab mem_cgroup_per_node start ff31bfa36a3c9e00 pointer offset 528 size 1920
list_del corruption. next->prev should be ffacf881ccc70e08, but was ff31bfa36a3ca010. (next=ff31bfa36a3ca010)

When a memcg goes offline, __lru_gen_reparent_memcg() moves its MGLRU lists to the parent. It moves only the generations (gens) from min_seq to max_seq, and it puts each list at the same index in the parent.

A folio's gen in folio->flags can be ahead of the list it is on. In the parent, the same index can stand for an older gen, so the folio's gen can end up older than its list. Once the parent's min_seq passes that gen, sort_folio() puts the folio on the list for max_seq + 1, where it stays until the next aging.

If the parent goes offline before that aging, the folio is not moved to the grandparent. It stays on the parent's list after the parent is freed, and the next list_del() on it touches freed memory. This needs two cgroups on one branch to go offline one after the other, such as a container and then its pod.

The parent always has MAX_NR_GENS gens at this point, so every index is valid in it. Move all MAX_NR_GENS lists, not only the gens from min_seq to max_seq.

A reproducer that drives these steps through the lru_gen debugfs file gets a KASAN use-after-free on the freed mem_cgroup_per_node without this patch, and no report with it.

Quoted 5 lines

Fixes f304652609ea ("mm: vmscan: prepare for reparenting MGLRU folios")

Reported-by Noah Feldt <[email protected]>

Closes https://lore.kernel.org/all/[email protected]/ open here

Signed-off-by Shakeel Butt <[email protected]>

Cc <[email protected]>

Thanks very much, Shakeel! I saw Kairui has exactly the same code here:

[PATCH v4 14/17] mm/mglru: reparent folios from all generations
https://lore.kernel.org/linux-mm/[email protected]/

somehow, Shakeel's patch makes us need the change earlier even before mglru-fg lands?