Skip to contentSkip to conversation

Eliminate Dying Memory Cgroup

PATCH Superseded Qi Zheng 11 participants 107 messages Jan 14 – Feb 2

Subsystems (17)
Memory management VFS MM core Tracing +13 more subsystems

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Qi Zheng

author reviewer
#0 lore raw

v3 cover Eliminate Dying Memory Cgroup

From: Qi Zheng <[email protected]>

Changes in v3:
- modify the commit message in [PATCH v2 04/28], [PATCH v2 06/28], [PATCH v2 13/28], [PATCH v2 24/28] and [PATCH v2 27/28] (suggested by David Hildenbrand, Chen Ridong and Johannes Weiner)
- change code style in [PATCH v3 8/30], [PATCH v3 15/30] and [PATCH v3 27/30] (suggested by Johannes Weiner and Shakeel Butt)
- use get_mem_cgroup_from_folio() + mem_cgroup_put() to replace holding rcu lock in [PATCH v3 14/30] and [PATCH v3 19/30]
(pointed by Johannes Weiner)
- add a comment to folio_split_queue_lock() in [PATCH v3 17/30] (suggested by Shakeel Butt)
- modify the comment above folio_lruvec() in [PATCH v3 24/30] (suggested by Johannes Weiner)
- fix rcu lock issue in lru_note_cost_refault()
(pointed by Shakeel Butt)
- add [PATCH v3 28/30] to fix non-hierarchical memcg1_stats issues (pointed by Yosry Ahmed)
- fix lru_zone_size issue in [PATCH v2 24/28] and [PATCH v2 25/28]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20260114

Changes in v2:
- add [PATCH v2 04/28] and remove local_irq_disable() in evict_folios() (pointed by Harry Yoo)
- recheck objcg in [PATCH v2 07/28] (pointed by Harry Yoo)
- modify the commit message in [PATCH v2 12/28] and [PATCH v2 21/28] (pointed by Harry Yoo)
- use rcu lock to protect mm_state in [PATCH v2 14/28] (pointed by Harry Yoo)
- fix bad unlock balance warning in [PATCH v2 23/28]
- change nr_pages type to long in [PATCH v2 25/28] (pointed by Harry Yoo)
- incease mm_state->seq during reparenting to make mm walker work properly in [PATCH v2 25/28] (pointed by Harry Yoo)
- add [PATCH v2 18/28] to fix WARNING in folio_memcg() (pointed by Harry Yoo)
- collect Reviewed-bys
- rebase onto the next-20251216

Changes in v1:
- drop [PATCH RFC 02/28]
- drop THP split queue related part, which has been merged as a separate patchset[2]
- prevent memory cgroup release in folio_split_queue_lock{_irqsave}() in [PATCH v1 16/26]
- Separate the reparenting function of traditional LRU folios to [PATCH v1 22/26]
- adapted to the MGLRU scenarios in [PATCH v1 23/26]
- refactor memcg_reparent_objcgs() in [PATCH v1 24/26]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20251028

Hi all,

Introduction
============

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup. A consensus has already been reached regarding this approach recently [1].

Background
==========

The issue of a dying memory cgroup refers to a situation where a memory cgroup is no longer being used by users, but memory (the metadata associated with memory cgroups) remains allocated to it. This situation may potentially result in memory leaks or inefficiencies in memory reclamation and has persisted as an issue for several years. Any memory allocation that endures longer than the lifespan (from the users' perspective) of a memory cgroup can lead to the issue of dying memory cgroup. We have exerted greater efforts to tackle this problem by introducing the infrastructure of object cgroup [2].

Presently, numerous types of objects (slab objects, non-slab kernel allocations, per-CPU objects) are charged to the object cgroup without holding a reference to the original memory cgroup. The final allocations for LRU pages (anonymous pages and file pages) are charged at allocation time and continues to hold a reference to the original memory cgroup until reclaimed.

File pages are more complex than anonymous pages as they can be shared among different memory cgroups and may persist beyond the lifespan of the memory cgroup. The long-term pinning of file pages to memory cgroups is a widespread issue that causes recurring problems in practical scenarios [3]. File pages remain unreclaimed for extended periods. Additionally, they are accessed by successive instances (second, third, fourth, etc.) of the same job, which is restarted into a new cgroup each time. As a result, unreclaimable dying memory cgroups accumulate, leading to memory wastage and significantly reducing the efficiency of page reclamation.

Fundamentals
============

A folio will no longer pin its corresponding memory cgroup. It is necessary to ensure that the memory cgroup or the lruvec associated with the memory cgroup is not released when a user obtains a pointer to the memory cgroup or lruvec returned by folio_memcg() or folio_lruvec(). Users are required to hold the RCU read lock or acquire a reference to the memory cgroup associated with the folio to prevent its release if they are not concerned about the binding stability between the folio and its corresponding memory cgroup. However, some users of folio_lruvec() (i.e., the lruvec lock) desire a stable binding between the folio and its corresponding memory cgroup. An approach is needed to ensure the stability of the binding while the lruvec lock is held, and to detect the situation of holding the incorrect lruvec lock when there is a race condition during memory cgroup reparenting. The following four steps are taken to achieve these goals.

1. The first step to be taken is to identify all users of both functions (folio_memcg() and folio_lruvec()) who are not concerned about binding stability and implement appropriate measures (such as holding a RCU read lock or temporarily obtaining a reference to the memory cgroup for a brief period) to prevent the release of the memory cgroup.

2. Secondly, the following refactoring of folio_lruvec_lock() demonstrates how to ensure the binding stability from the user's perspective of folio_lruvec().

   struct lruvec *folio_lruvec_lock(struct folio *folio)
   {
           struct lruvec *lruvec;
           rcu_read_lock();
   retry:
           lruvec = folio_lruvec(folio);
           spin_lock(&lruvec->lru_lock);
           if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
                   spin_unlock(&lruvec->lru_lock);
                   goto retry;
           }

           return lruvec;

}

From the perspective of memory cgroup removal, the entire reparenting process (altering the binding relationship between folio and its memory cgroup and moving the LRU lists to its parental memory cgroup) should be carried out under both the lruvec lock of the memory cgroup being removed and the lruvec lock of its parent.

3. Finally, transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup.

Effect
======

Finally, it can be observed that the quantity of dying memory cgroups will not experience a significant increase if the following test script is executed to reproduce the issue.

```bash
#!/bin/bash

# Create a temporary file 'temp' filled with zero bytes
dd if=/dev/zero of=temp bs=4096 count=1

# Display memory-cgroup info from /proc/cgroups
cat /proc/cgroups | grep memory

for i in {0..2000}
do
    mkdir /sys/fs/cgroup/memory/test$i
    echo $$ > /sys/fs/cgroup/memory/test$i/cgroup.procs

    # Append 'temp' file content to 'log'
    cat temp >> log

    echo $$ > /sys/fs/cgroup/memory/cgroup.procs

    # Potentially create a dying memory cgroup
    rmdir /sys/fs/cgroup/memory/test$i
done

# Display memory-cgroup info after test
cat /proc/cgroups | grep memory

rm -f temp log
```

Comments and suggestions are welcome!

Thanks,
Qi

[1].https://lore.kernel.org/linux-mm/Z6OkXXYDorPrBvEQ@hm-sls2/ [2].https://lwn.net/Articles/895431/ [3].https://github.com/systemd/systemd/pull/36827

Muchun Song (22):
mm: memcontrol: remove dead code of checking parent memory cgroup
mm: workingset: use folio_lruvec() in workingset_refault()
mm: rename unlock_page_lruvec_irq and its variants
mm: vmscan: refactor move_folios_to_lru()
mm: memcontrol: allocate object cgroup for non-kmem case
mm: memcontrol: return root object cgroup for root memory cgroup
mm: memcontrol: prevent memory cgroup release in
get_mem_cgroup_from_folio()
buffer: prevent memory cgroup release in folio_alloc_buffers()
writeback: prevent memory cgroup release in writeback module
mm: memcontrol: prevent memory cgroup release in
count_memcg_folio_events()
mm: page_io: prevent memory cgroup release in page_io module
mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
mm: mglru: prevent memory cgroup release in mglru
mm: memcontrol: prevent memory cgroup release in
mem_cgroup_swap_full()
mm: workingset: prevent memory cgroup release in lru_gen_eviction()
mm: workingset: prevent lruvec release in workingset_refault()
mm: zswap: prevent lruvec release in zswap_folio_swapin()
mm: swap: prevent lruvec release in lru_gen_clear_refs()
mm: workingset: prevent lruvec release in workingset_activation()
mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
mm: memcontrol: eliminate the problem of dying memory cgroup for LRU
folios
mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers

Qi Zheng (8):
mm: vmscan: prepare for the refactoring the move_folios_to_lru()
mm: thp: prevent memory cgroup release in
folio_split_queue_lock{_irqsave}()
mm: zswap: prevent memory cgroup release in zswap_compress()
mm: do not open-code lruvec lock
mm: vmscan: prepare for reparenting traditional LRU folios
mm: vmscan: prepare for reparenting MGLRU folios
mm: memcontrol: refactor memcg_reparent_objcgs()
mm: memcontrol: prepare for reparenting state_local

 fs/buffer.c                      |   4 +-
 fs/fs-writeback.c                |  22 +-
 include/linux/memcontrol.h       | 187 ++++++------
 include/linux/mm_inline.h        |   6 +
 include/linux/mmzone.h           |  20 ++
 include/linux/swap.h             |  20 ++
 include/trace/events/writeback.h |   3 +
 mm/compaction.c                  |  43 ++-
 mm/huge_memory.c                 |  22 +-
 mm/memcontrol-v1.c               |  31 +-
 mm/memcontrol-v1.h               |   3 +
 mm/memcontrol.c                  | 472 ++++++++++++++++++++-----------
 mm/migrate.c                     |   2 +
 mm/mlock.c                       |   2 +-
 mm/page_io.c                     |   8 +-
 mm/percpu.c                      |   2 +-
 mm/shrinker.c                    |   6 +-
 mm/swap.c                        |  61 +++-
 mm/vmscan.c                      | 280 +++++++++++++-----
 mm/workingset.c                  |  25 +-
 mm/zswap.c                       |   5 +
 21 files changed, 835 insertions(+), 389 deletions(-)
Signature · 1 line
2.20.1
author reviewer
#1 lore raw

PATCH v3 1/30 mm: memcontrol: remove dead code of checking parent memory cgroup

From: Muchun Song <[email protected]>

Since the no-hierarchy mode has been deprecated after the commit:

commit bef8620cd8e0 ("mm: memcg: deprecate the non-hierarchical mode").

As a result, parent_mem_cgroup() will not return NULL except when passing the root memcg, and the root memcg cannot be offline. Hence, it's safe to remove the check on the returned value of parent_mem_cgroup(). Remove the corresponding dead code.

---
 mm/memcontrol.c | 5 -----
 mm/shrinker.c   | 6 +-----
 2 files changed, 1 insertion(+), 10 deletions(-)
mm/memcontrol.c +0 -5
index f9455b4790ba5..c68af9fea0abd 100644
@@ -3352,9 +3352,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 		return;
 
 	parent = parent_mem_cgroup(memcg);
-	if (!parent)
-		parent = root_mem_cgroup;
-
 	memcg_reparent_list_lrus(memcg, parent);
 
 	/*
@@ -3645,8 +3642,6 @@ struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg)
 			break;
 		}
 		memcg = parent_mem_cgroup(memcg);
-		if (!memcg)
-			memcg = root_mem_cgroup;
 	}
 	return memcg;
 }
mm/shrinker.c +1 -5
index 4a93fd433689a..e8e092a2f7f41 100644
@@ -286,14 +286,10 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg)
 {
 	int nid, index, offset;
 	long nr;
-	struct mem_cgroup *parent;
+	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
 	struct shrinker_info *child_info, *parent_info;
 	struct shrinker_info_unit *child_unit, *parent_unit;
 
-	parent = parent_mem_cgroup(memcg);
-	if (!parent)
-		parent = root_mem_cgroup;
-
 	/* Prevent from concurrent shrinker_info expand */
 	mutex_lock(&shrinker_mutex);
 	for_each_node(nid) {
Signature · 1 line
2.20.1
author reviewer
#2 lore raw

PATCH v3 2/30 mm: workingset: use folio_lruvec() in workingset_refault()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

Use folio_lruvec() to simplify the code.

---
 mm/workingset.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
mm/workingset.c +1 -6
index 13422d3047158..ed12744d93a29 100644
@@ -534,8 +534,6 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
 void workingset_refault(struct folio *folio, void *shadow)
 {
 	bool file = folio_is_file_lru(folio);
-	struct pglist_data *pgdat;
-	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
 	bool workingset;
 	long nr;
@@ -557,10 +555,7 @@ void workingset_refault(struct folio *folio, void *shadow)
 	 * locked to guarantee folio_memcg() stability throughout.
 	 */
 	nr = folio_nr_pages(folio);
-	memcg = folio_memcg(folio);
-	pgdat = folio_pgdat(folio);
-	lruvec = mem_cgroup_lruvec(memcg, pgdat);
-
+	lruvec = folio_lruvec(folio);
 	mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
 
 	if (!workingset_test_recent(shadow, file, &workingset, true))
Signature · 1 line
2.20.1
author reviewer
#3 lore raw

PATCH v3 3/30 mm: rename unlock_page_lruvec_irq and its variants

In reply to Qi Zheng

From: Muchun Song <[email protected]>

It is inappropriate to use folio_lruvec_lock() variants in conjunction with unlock_page_lruvec() variants, as this involves the inconsistent operation of locking a folio while unlocking a page. To rectify this, the functions unlock_page_lruvec{_irq, _irqrestore} are renamed to lruvec_unlock{_irq,_irqrestore}.

---
 include/linux/memcontrol.h | 10 +++++-----
 mm/compaction.c            | 14 +++++++-------
 mm/huge_memory.c           |  2 +-
 mm/mlock.c                 |  2 +-
 mm/swap.c                  | 12 ++++++------
 mm/vmscan.c                |  4 ++--
 6 files changed, 22 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +5 -5
index af352cabedbae..6a44e79a8bd23 100644
@@ -1480,17 +1480,17 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
 	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
-static inline void unlock_page_lruvec(struct lruvec *lruvec)
+static inline void lruvec_unlock(struct lruvec *lruvec)
 {
 	spin_unlock(&lruvec->lru_lock);
 }
 
-static inline void unlock_page_lruvec_irq(struct lruvec *lruvec)
+static inline void lruvec_unlock_irq(struct lruvec *lruvec)
 {
 	spin_unlock_irq(&lruvec->lru_lock);
 }
 
-static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,
+static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
 		unsigned long flags)
 {
 	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
@@ -1512,7 +1512,7 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio,
 		if (folio_matches_lruvec(folio, locked_lruvec))
 			return locked_lruvec;
 
-		unlock_page_lruvec_irq(locked_lruvec);
+		lruvec_unlock_irq(locked_lruvec);
 	}
 
 	return folio_lruvec_lock_irq(folio);
@@ -1526,7 +1526,7 @@ static inline void folio_lruvec_relock_irqsave(struct folio *folio,
 		if (folio_matches_lruvec(folio, *lruvecp))
 			return;
 
-		unlock_page_lruvec_irqrestore(*lruvecp, *flags);
+		lruvec_unlock_irqrestore(*lruvecp, *flags);
 	}
 
 	*lruvecp = folio_lruvec_lock_irqsave(folio, flags);
mm/compaction.c +7 -7
index 1e8f8eca318c6..c3e338aaa0ffb 100644
@@ -913,7 +913,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		 */
 		if (!(low_pfn % COMPACT_CLUSTER_MAX)) {
 			if (locked) {
-				unlock_page_lruvec_irqrestore(locked, flags);
+				lruvec_unlock_irqrestore(locked, flags);
 				locked = NULL;
 			}
 
@@ -964,7 +964,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			}
 			/* for alloc_contig case */
 			if (locked) {
-				unlock_page_lruvec_irqrestore(locked, flags);
+				lruvec_unlock_irqrestore(locked, flags);
 				locked = NULL;
 			}
 
@@ -1053,7 +1053,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			if (unlikely(page_has_movable_ops(page)) &&
 			    !PageMovableOpsIsolated(page)) {
 				if (locked) {
-					unlock_page_lruvec_irqrestore(locked, flags);
+					lruvec_unlock_irqrestore(locked, flags);
 					locked = NULL;
 				}
 
@@ -1158,7 +1158,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		/* If we already hold the lock, we can skip some rechecking */
 		if (lruvec != locked) {
 			if (locked)
-				unlock_page_lruvec_irqrestore(locked, flags);
+				lruvec_unlock_irqrestore(locked, flags);
 
 			compact_lock_irqsave(&lruvec->lru_lock, &flags, cc);
 			locked = lruvec;
@@ -1226,7 +1226,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 isolate_fail_put:
 		/* Avoid potential deadlock in freeing page under lru_lock */
 		if (locked) {
-			unlock_page_lruvec_irqrestore(locked, flags);
+			lruvec_unlock_irqrestore(locked, flags);
 			locked = NULL;
 		}
 		folio_put(folio);
@@ -1242,7 +1242,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		 */
 		if (nr_isolated) {
 			if (locked) {
-				unlock_page_lruvec_irqrestore(locked, flags);
+				lruvec_unlock_irqrestore(locked, flags);
 				locked = NULL;
 			}
 			putback_movable_pages(&cc->migratepages);
@@ -1274,7 +1274,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 
 isolate_abort:
 	if (locked)
-		unlock_page_lruvec_irqrestore(locked, flags);
+		lruvec_unlock_irqrestore(locked, flags);
 	if (folio) {
 		folio_set_lru(folio);
 		folio_put(folio);
mm/huge_memory.c +1 -1
index a6d37902b73d2..99515838faa51 100644
@@ -3899,7 +3899,7 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
 		folio_ref_unfreeze(folio, folio_cache_ref_count(folio) + 1);
 
 		if (do_lru)
-			unlock_page_lruvec(lruvec);
+			lruvec_unlock(lruvec);
 
 		if (ci)
 			swap_cluster_unlock(ci);
mm/mlock.c +1 -1
index 2f699c3497a57..66740e16679c3 100644
@@ -205,7 +205,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch)
 	}
 
 	if (lruvec)
-		unlock_page_lruvec_irq(lruvec);
+		lruvec_unlock_irq(lruvec);
 	folios_put(fbatch);
 }
 
mm/swap.c +6 -6
index bb19ccbece464..245ba159e01d7 100644
@@ -91,7 +91,7 @@ static void page_cache_release(struct folio *folio)
 
 	__page_cache_release(folio, &lruvec, &flags);
 	if (lruvec)
-		unlock_page_lruvec_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore(lruvec, flags);
 }
 
 void __folio_put(struct folio *folio)
@@ -175,7 +175,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 	}
 
 	if (lruvec)
-		unlock_page_lruvec_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore(lruvec, flags);
 	folios_put(fbatch);
 }
 
@@ -349,7 +349,7 @@ void folio_activate(struct folio *folio)
 
 	lruvec = folio_lruvec_lock_irq(folio);
 	lru_activate(lruvec, folio);
-	unlock_page_lruvec_irq(lruvec);
+	lruvec_unlock_irq(lruvec);
 	folio_set_lru(folio);
 }
 #endif
@@ -963,7 +963,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 
 		if (folio_is_zone_device(folio)) {
 			if (lruvec) {
-				unlock_page_lruvec_irqrestore(lruvec, flags);
+				lruvec_unlock_irqrestore(lruvec, flags);
 				lruvec = NULL;
 			}
 			if (folio_ref_sub_and_test(folio, nr_refs))
@@ -977,7 +977,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 		/* hugetlb has its own memcg */
 		if (folio_test_hugetlb(folio)) {
 			if (lruvec) {
-				unlock_page_lruvec_irqrestore(lruvec, flags);
+				lruvec_unlock_irqrestore(lruvec, flags);
 				lruvec = NULL;
 			}
 			free_huge_folio(folio);
@@ -991,7 +991,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 		j++;
 	}
 	if (lruvec)
-		unlock_page_lruvec_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore(lruvec, flags);
 	if (!j) {
 		folio_batch_reinit(folios);
 		return;
mm/vmscan.c +2 -2
index a0e3f51de749e..1ede4f23b9a6f 100644
@@ -1853,7 +1853,7 @@ bool folio_isolate_lru(struct folio *folio)
 		folio_get(folio);
 		lruvec = folio_lruvec_lock_irq(folio);
 		lruvec_del_folio(lruvec, folio);
-		unlock_page_lruvec_irq(lruvec);
+		lruvec_unlock_irq(lruvec);
 		ret = true;
 	}
 
@@ -7871,7 +7871,7 @@ void check_move_unevictable_folios(struct folio_batch *fbatch)
 	if (lruvec) {
 		__count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued);
 		__count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
-		unlock_page_lruvec_irq(lruvec);
+		lruvec_unlock_irq(lruvec);
 	} else if (pgscanned) {
 		count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned);
 	}
Signature · 1 line
2.20.1
author reviewer
#4 lore raw

PATCH v3 4/30 mm: vmscan: prepare for the refactoring the move_folios_to_lru()

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

Once we refactor move_folios_to_lru(), its callers will no longer have to hold the lruvec lock; For shrink_inactive_list(), shrink_active_list() and evict_folios(), IRQ disabling is only needed for __count_vm_events() and __mod_node_page_state().

To avoid using local_irq_disable() on the PREEMPT_RT kernel, let's make all callers of move_folios_to_lru() use IRQ-safed count_vm_events() and mod_node_page_state().

---
 mm/vmscan.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
mm/vmscan.c +7 -7
index 1ede4f23b9a6f..5c59c275c4463 100644
@@ -2045,12 +2045,12 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 
 	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
 					stat.nr_demoted);
-	__mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
+	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 	item = PGSTEAL_KSWAPD + reclaimer_offset(sc);
 	if (!cgroup_reclaim(sc))
-		__count_vm_events(item, nr_reclaimed);
+		count_vm_events(item, nr_reclaimed);
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
-	__count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
+	count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
 
 	lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
 					nr_scanned - nr_reclaimed);
@@ -2195,10 +2195,10 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	nr_activate = move_folios_to_lru(lruvec, &l_active);
 	nr_deactivate = move_folios_to_lru(lruvec, &l_inactive);
 
-	__count_vm_events(PGDEACTIVATE, nr_deactivate);
+	count_vm_events(PGDEACTIVATE, nr_deactivate);
 	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
 
-	__mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
+	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 
 	lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
 	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
@@ -4788,9 +4788,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 
 	item = PGSTEAL_KSWAPD + reclaimer_offset(sc);
 	if (!cgroup_reclaim(sc))
-		__count_vm_events(item, reclaimed);
+		count_vm_events(item, reclaimed);
 	count_memcg_events(memcg, item, reclaimed);
-	__count_vm_events(PGSTEAL_ANON + type, reclaimed);
+	count_vm_events(PGSTEAL_ANON + type, reclaimed);
 
 	spin_unlock_irq(&lruvec->lru_lock);
 
Signature · 1 line
2.20.1
Reviewed-by on v3 4/30
#5 lore raw

Re: [PATCH v3 04/30] mm: vmscan: prepare for the refactoring the move_folios_to_lru()

Quoted 11 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Once we refactor move_folios_to_lru(), its callers will no longer have to hold the lruvec lock; For shrink_inactive_list(), shrink_active_list() and evict_folios(), IRQ disabling is only needed for __count_vm_events() and __mod_node_page_state().

To avoid using local_irq_disable() on the PREEMPT_RT kernel, let's make all callers of move_folios_to_lru() use IRQ-safed count_vm_events() and mod_node_page_state().

Quoted 5 lines

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
reviewer Acked-by on v3 4/30
#6 lore raw

Re: [PATCH v3 04/30] mm: vmscan: prepare for the refactoring the move_folios_to_lru()

In reply to Qi Zheng
Quoted 11 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Once we refactor move_folios_to_lru(), its callers will no longer have to hold the lruvec lock; For shrink_inactive_list(), shrink_active_list() and evict_folios(), IRQ disabling is only needed for __count_vm_events() and __mod_node_page_state().

Quoted 8 lines

To avoid using local_irq_disable() on the PREEMPT_RT kernel, let's make all callers of move_folios_to_lru() use IRQ-safed count_vm_events() and mod_node_page_state().

author reviewer
#7 lore raw

PATCH v3 5/30 mm: vmscan: refactor move_folios_to_lru()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In a subsequent patch, we'll reparent the LRU folios. The folios that are moved to the appropriate LRU list can undergo reparenting during the move_folios_to_lru() process. Hence, it's incorrect for the caller to hold a lruvec lock. Instead, we should utilize the more general interface of folio_lruvec_relock_irq() to obtain the correct lruvec lock.

This patch involves only code refactoring and doesn't introduce any functional changes.

---
 mm/vmscan.c | 46 +++++++++++++++++++++-------------------------
 1 file changed, 21 insertions(+), 25 deletions(-)
mm/vmscan.c +21 -25
index 5c59c275c4463..20cd54c5cbc79 100644
@@ -1907,24 +1907,27 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,
 /*
  * move_folios_to_lru() moves folios from private @list to appropriate LRU list.
  *
- * Returns the number of pages moved to the given lruvec.
+ * Returns the number of pages moved to the appropriate lruvec.
+ *
+ * Note: The caller must not hold any lruvec lock.
  */
-static unsigned int move_folios_to_lru(struct lruvec *lruvec,
-		struct list_head *list)
+static unsigned int move_folios_to_lru(struct list_head *list)
 {
 	int nr_pages, nr_moved = 0;
+	struct lruvec *lruvec = NULL;
 	struct folio_batch free_folios;
 
 	folio_batch_init(&free_folios);
 	while (!list_empty(list)) {
 		struct folio *folio = lru_to_folio(list);
 
+		lruvec = folio_lruvec_relock_irq(folio, lruvec);
 		VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 		list_del(&folio->lru);
 		if (unlikely(!folio_evictable(folio))) {
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
 			folio_putback_lru(folio);
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec = NULL;
 			continue;
 		}
 
@@ -1946,19 +1949,15 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,
 
 			folio_unqueue_deferred_split(folio);
 			if (folio_batch_add(&free_folios, folio) == 0) {
-				spin_unlock_irq(&lruvec->lru_lock);
+				lruvec_unlock_irq(lruvec);
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);
-				spin_lock_irq(&lruvec->lru_lock);
+				lruvec = NULL;
 			}
 
 			continue;
 		}
 
-		/*
-		 * All pages were isolated from the same lruvec (and isolation
-		 * inhibits memcg migration).
-		 */
 		VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio);
 		lruvec_add_folio(lruvec, folio);
 		nr_pages = folio_nr_pages(folio);
@@ -1967,11 +1966,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,
 			workingset_age_nonresident(lruvec, nr_pages);
 	}
 
+	if (lruvec)
+		lruvec_unlock_irq(lruvec);
+
 	if (free_folios.nr) {
-		spin_unlock_irq(&lruvec->lru_lock);
 		mem_cgroup_uncharge_folios(&free_folios);
 		free_unref_folios(&free_folios);
-		spin_lock_irq(&lruvec->lru_lock);
 	}
 
 	return nr_moved;
@@ -2040,8 +2040,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	nr_reclaimed = shrink_folio_list(&folio_list, pgdat, sc, &stat, false,
 					 lruvec_memcg(lruvec));
 
-	spin_lock_irq(&lruvec->lru_lock);
-	move_folios_to_lru(lruvec, &folio_list);
+	move_folios_to_lru(&folio_list);
 
 	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
 					stat.nr_demoted);
@@ -2052,6 +2051,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
 	count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
 
+	spin_lock_irq(&lruvec->lru_lock);
 	lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
 					nr_scanned - nr_reclaimed);
 
@@ -2190,16 +2190,14 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	/*
 	 * Move folios back to the lru list.
 	 */
-	spin_lock_irq(&lruvec->lru_lock);
-
-	nr_activate = move_folios_to_lru(lruvec, &l_active);
-	nr_deactivate = move_folios_to_lru(lruvec, &l_inactive);
+	nr_activate = move_folios_to_lru(&l_active);
+	nr_deactivate = move_folios_to_lru(&l_inactive);
 
 	count_vm_events(PGDEACTIVATE, nr_deactivate);
 	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
-
 	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 
+	spin_lock_irq(&lruvec->lru_lock);
 	lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
 	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
 			nr_deactivate, nr_rotated, sc->priority, file);
@@ -4773,14 +4771,14 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 			set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active));
 	}
 
-	spin_lock_irq(&lruvec->lru_lock);
-
-	move_folios_to_lru(lruvec, &list);
+	move_folios_to_lru(&list);
 
 	walk = current->reclaim_state->mm_walk;
 	if (walk && walk->batched) {
 		walk->lruvec = lruvec;
+		spin_lock_irq(&lruvec->lru_lock);
 		reset_batch_size(walk);
+		spin_unlock_irq(&lruvec->lru_lock);
 	}
 
 	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -4792,8 +4790,6 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	count_memcg_events(memcg, item, reclaimed);
 	count_vm_events(PGSTEAL_ANON + type, reclaimed);
 
-	spin_unlock_irq(&lruvec->lru_lock);
-
 	list_splice_init(&clean, &list);
 
 	if (!list_empty(&list)) {
Signature · 1 line
2.20.1
Reviewed-by on v3 5/30
#8 lore raw

Re: [PATCH v3 05/30] mm: vmscan: refactor move_folios_to_lru()

Quoted 8 lines · Qi Zheng

From: Muchun Song <[email protected]>

In a subsequent patch, we'll reparent the LRU folios. The folios that are moved to the appropriate LRU list can undergo reparenting during the move_folios_to_lru() process. Hence, it's incorrect for the caller to hold a lruvec lock. Instead, we should utilize the more general interface of folio_lruvec_relock_irq() to obtain the correct lruvec lock.

Quoted 8 lines

This patch involves only code refactoring and doesn't introduce any functional changes.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#9 lore raw

PATCH v3 6/30 mm: memcontrol: allocate object cgroup for non-kmem case

In reply to Qi Zheng

From: Muchun Song <[email protected]>

To allow LRU page reparenting, the objcg infrastructure is no longer solely applicable to the kmem case. In this patch, we extend the scope of the objcg infrastructure beyond the kmem case, enabling LRU folios to reuse it for folio charging purposes.

It should be noted that LRU folios are not accounted for at the root level, yet the folio->memcg_data points to the root_mem_cgroup. Hence, the folio->memcg_data of LRU folios always points to a valid pointer. However, the root_mem_cgroup does not possess an object cgroup. Therefore, we also allocate an object cgroup for the root_mem_cgroup.

---
 mm/memcontrol.c | 51 +++++++++++++++++++++++--------------------------
 1 file changed, 24 insertions(+), 27 deletions(-)
mm/memcontrol.c +24 -27
index c68af9fea0abd..a9057f24cc48f 100644
@@ -206,10 +206,10 @@ static struct obj_cgroup *obj_cgroup_alloc(void)
 	return objcg;
 }
 
-static void memcg_reparent_objcgs(struct mem_cgroup *memcg,
-				  struct mem_cgroup *parent)
+static void memcg_reparent_objcgs(struct mem_cgroup *memcg)
 {
 	struct obj_cgroup *objcg, *iter;
+	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
 
 	objcg = rcu_replace_pointer(memcg->objcg, NULL, true);
 
@@ -3315,30 +3315,17 @@ void folio_split_memcg_refs(struct folio *folio, unsigned old_order,
 	css_get_many(&__folio_memcg(folio)->css, new_refs);
 }
 
-static int memcg_online_kmem(struct mem_cgroup *memcg)
+static void memcg_online_kmem(struct mem_cgroup *memcg)
 {
-	struct obj_cgroup *objcg;
-
 	if (mem_cgroup_kmem_disabled())
-		return 0;
+		return;
 
 	if (unlikely(mem_cgroup_is_root(memcg)))
-		return 0;
-
-	objcg = obj_cgroup_alloc();
-	if (!objcg)
-		return -ENOMEM;
-
-	objcg->memcg = memcg;
-	rcu_assign_pointer(memcg->objcg, objcg);
-	obj_cgroup_get(objcg);
-	memcg->orig_objcg = objcg;
+		return;
 
 	static_branch_enable(&memcg_kmem_online_key);
 
 	memcg->kmemcg_id = memcg->id.id;
-
-	return 0;
 }
 
 static void memcg_offline_kmem(struct mem_cgroup *memcg)
@@ -3353,12 +3340,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 
 	parent = parent_mem_cgroup(memcg);
 	memcg_reparent_list_lrus(memcg, parent);
-
-	/*
-	 * Objcg's reparenting must be after list_lru's, make sure list_lru
-	 * helpers won't use parent's list_lru until child is drained.
-	 */
-	memcg_reparent_objcgs(memcg, parent);
 }
 
 #ifdef CONFIG_CGROUP_WRITEBACK
@@ -3871,9 +3852,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
 {
 	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
+	struct obj_cgroup *objcg;
 
-	if (memcg_online_kmem(memcg))
-		goto remove_id;
+	memcg_online_kmem(memcg);
 
 	/*
 	 * A memcg must be visible for expand_shrinker_info()
@@ -3883,6 +3864,15 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
 	if (alloc_shrinker_info(memcg))
 		goto offline_kmem;
 
+	objcg = obj_cgroup_alloc();
+	if (!objcg)
+		goto free_shrinker;
+
+	objcg->memcg = memcg;
+	rcu_assign_pointer(memcg->objcg, objcg);
+	obj_cgroup_get(objcg);
+	memcg->orig_objcg = objcg;
+
 	if (unlikely(mem_cgroup_is_root(memcg)) && !mem_cgroup_disabled())
 		queue_delayed_work(system_unbound_wq, &stats_flush_dwork,
 				   FLUSH_TIME);
@@ -3905,9 +3895,10 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
 	xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);
 
 	return 0;
+free_shrinker:
+	free_shrinker_info(memcg);
 offline_kmem:
 	memcg_offline_kmem(memcg);
-remove_id:
 	mem_cgroup_private_id_remove(memcg);
 	return -ENOMEM;
 }
@@ -3925,6 +3916,12 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
 
 	memcg_offline_kmem(memcg);
 	reparent_deferred_split_queue(memcg);
+	/*
+	 * The reparenting of objcg must be after the reparenting of the
+	 * list_lru and deferred_split_queue above, which ensures that they will
+	 * not mistakenly get the parent list_lru and deferred_split_queue.
+	 */
+	memcg_reparent_objcgs(memcg);
 	reparent_shrinker_deferred(memcg);
 	wb_memcg_offline(memcg);
 	lru_gen_offline_memcg(memcg);
Signature · 1 line
2.20.1
author reviewer
#10 lore raw

PATCH v3 7/30 mm: memcontrol: return root object cgroup for root memory cgroup

In reply to Qi Zheng

From: Muchun Song <[email protected]>

Memory cgroup functions such as get_mem_cgroup_from_folio() and get_mem_cgroup_from_mm() return a valid memory cgroup pointer, even for the root memory cgroup. In contrast, the situation for object cgroups has been different.

Previously, the root object cgroup couldn't be returned because it didn't exist. Now that a valid root object cgroup exists, for the sake of consistency, it's necessary to align the behavior of object-cgroup-related operations with that of memory cgroup APIs.

---
 include/linux/memcontrol.h | 26 +++++++++++++++++-----
 mm/memcontrol.c            | 45 ++++++++++++++++++++------------------
 mm/percpu.c                |  2 +-
 3 files changed, 45 insertions(+), 28 deletions(-)
include/linux/memcontrol.h +20 -6
index 6a44e79a8bd23..7b3d8f341ff10 100644
@@ -332,6 +332,7 @@ struct mem_cgroup {
 #define MEMCG_CHARGE_BATCH 64U
 
 extern struct mem_cgroup *root_mem_cgroup;
+extern struct obj_cgroup *root_obj_cgroup;
 
 enum page_memcg_data_flags {
 	/* page->memcg_data is a pointer to an slabobj_ext vector */
@@ -549,6 +550,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)
 	return (memcg == root_mem_cgroup);
 }
 
+static inline bool obj_cgroup_is_root(const struct obj_cgroup *objcg)
+{
+	return objcg == root_obj_cgroup;
+}
+
 static inline bool mem_cgroup_disabled(void)
 {
 	return !cgroup_subsys_enabled(memory_cgrp_subsys);
@@ -775,23 +781,26 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
 
 static inline bool obj_cgroup_tryget(struct obj_cgroup *objcg)
 {
+	if (obj_cgroup_is_root(objcg))
+		return true;
 	return percpu_ref_tryget(&objcg->refcnt);
 }
 
-static inline void obj_cgroup_get(struct obj_cgroup *objcg)
+static inline void obj_cgroup_get_many(struct obj_cgroup *objcg,
+				       unsigned long nr)
 {
-	percpu_ref_get(&objcg->refcnt);
+	if (!obj_cgroup_is_root(objcg))
+		percpu_ref_get_many(&objcg->refcnt, nr);
 }
 
-static inline void obj_cgroup_get_many(struct obj_cgroup *objcg,
-				       unsigned long nr)
+static inline void obj_cgroup_get(struct obj_cgroup *objcg)
 {
-	percpu_ref_get_many(&objcg->refcnt, nr);
+	obj_cgroup_get_many(objcg, 1);
 }
 
 static inline void obj_cgroup_put(struct obj_cgroup *objcg)
 {
-	if (objcg)
+	if (objcg && !obj_cgroup_is_root(objcg))
 		percpu_ref_put(&objcg->refcnt);
 }
 
@@ -1088,6 +1097,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)
 	return true;
 }
 
+static inline bool obj_cgroup_is_root(const struct obj_cgroup *objcg)
+{
+	return true;
+}
+
 static inline bool mem_cgroup_disabled(void)
 {
 	return true;
mm/memcontrol.c +24 -21

119 lines not on this page · raw message

mm/percpu.c +1 -1
index c9f8df6c34c30..1a9328f66a7f7 100644
@@ -1622,7 +1622,7 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp,
 		return true;
 
 	objcg = current_obj_cgroup();
-	if (!objcg)
+	if (!objcg || obj_cgroup_is_root(objcg))
 		return true;
 
 	if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size)))
Signature · 1 line
2.20.1
Reviewed-by on v3 7/30
#11 lore raw

Re: [PATCH v3 07/30] mm: memcontrol: return root object cgroup for root memory cgroup

Quoted 12 lines · Qi Zheng

From: Muchun Song <[email protected]>

Memory cgroup functions such as get_mem_cgroup_from_folio() and get_mem_cgroup_from_mm() return a valid memory cgroup pointer, even for the root memory cgroup. In contrast, the situation for object cgroups has been different.

Previously, the root object cgroup couldn't be returned because it didn't exist. Now that a valid root object cgroup exists, for the sake of consistency, it's necessary to align the behavior of object-cgroup-related operations with that of memory cgroup APIs.

Quoted 5 lines

---

Looks good to me, so:

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#12 lore raw

PATCH v3 8/30 mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/memcontrol.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memcontrol.c +7 -3
index 982c9f5cf72cb..0458fc2e810ff 100644
@@ -991,14 +991,18 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)
  */
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
 {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
 	if (mem_cgroup_disabled())
 		return NULL;
 
+	if (!folio_memcg_charged(folio))
+		return root_mem_cgroup;
+
 	rcu_read_lock();
-	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
-		memcg = root_mem_cgroup;
+	do {
+		memcg = folio_memcg(folio);
+	} while (unlikely(!css_tryget(&memcg->css)));
 	rcu_read_unlock();
 	return memcg;
 }
Signature · 1 line
2.20.1
maintainer Acked-by on v3 8/30
#13 lore raw

Re: [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()

Quoted 7 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

Quoted 5 lines

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().

This serves as a preparatory measure for the reparenting of the LRU pages.

This patch is adding a retry loop as css_tryget() on folio_memcg() can fail because after reparenting functionality the folio no longer holds down the memcg. Please clarify this reasoning in the commit.

With that:

maintainer
#14 lore raw

Re: [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()

In reply to Qi Zheng
Quoted 33 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/memcontrol.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memcontrol.c +1 -1
index 982c9f5cf72cb..0458fc2e810ff 100644
@@ -991,14 +991,18 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)
  */
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
 {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
 	if (mem_cgroup_disabled())
 		return NULL;

mm/memcontrol.c struct mem_cgroup *get_mem_cgroup_from_current(void) · line 1005

Quoted 10 lines
mm/memcontrol.c +6 -2
 
+	if (!folio_memcg_charged(folio))
+		return root_mem_cgroup;
+
 	rcu_read_lock();
-	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
-		memcg = root_mem_cgroup;
+	do {
+		memcg = folio_memcg(folio);
+	} while (unlikely(!css_tryget(&memcg->css)));

I went back to [1] where AI raised the following concern which I want to address:

Quoted 3 lines

If css_tryget() fails (e.g. refcount is 0), this loop spins indefinitely with the RCU read lock held. Is it guaranteed that folio_memcg() will return a different, alive memcg in subsequent iterations?

Will css_tryget() ever fail for the memcg returned by folio_memcg()? Let's suppose memcg of a given folio is being offlined. The objcg reparenting happens in memcg_reparent_objcgs() which is called in offline_css() chain and we know that the offline context holds a reference on the css being offlined (see css_killed_work_fn()).

Also let's suppose the offline process has the last reference on the memcg's css. Now we have following two scenarios:

Scenario 1:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_tryget(memcg)
css_put(css)

In the above case css_tryget() will not fail.

Scenario 2:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_put(css) // last reference
css_tryget(memcg)
// retry on failure

In the above case the context in get_mem_cgroup_from_folio() will retry and will get different memcg during reparenting happening before the last css_put(css).

So, I think we are good and AI is mistaken.

Folks, please check if I missed something.

Quoted 3 lines

If the folio is isolated (e.g. via migrate_misplaced_folio()), it might be missed by reparenting logic that iterates LRU lists.

LRU isolation will not impact reparenting logic, so we can discount this as well.

Quoted 5 lines

In that case, the
folio would continue pointing to the dying memcg, leading to a hard lockup.

Also, folio_memcg() calls __folio_memcg(), which reads folio->memcg_data without READ_ONCE().

Oh I think I know why AI is confused. It is because it is looking at folio->memcg i.e. state with this patch only and not the state after the series. In the current state the folio holds the reference on memcg, so css_tryget() will never fail.

Quoted 7 lines

Since this loop waits for memcg_data to be updated
by another CPU (reparenting), could the compiler hoist the load out of the loop, preventing the update from being seen?

Finally, the previous code fell back to root_mem_cgroup on failure. Is it safe to remove that fallback? If css_tryget() fails unexpectedly, hanging seems more severe than the previous behavior of warning and falling back.

[1] https://lore.kernel.org/all/[email protected]/

author reviewer
#15 lore raw

Re: [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()

Quoted 81 lines · Shakeel Butt
Quoted 33 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
  mm/memcontrol.c | 10 +++++++---
  1 file changed, 7 insertions(+), 3 deletions(-)
mm/memcontrol.c +1 -1
index 982c9f5cf72cb..0458fc2e810ff 100644
@@ -991,14 +991,18 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)
   */
  struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
  {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
  	if (mem_cgroup_disabled())
  		return NULL;

mm/memcontrol.c struct mem_cgroup *get_mem_cgroup_from_current(void) · line 1005

Quoted 10 lines
mm/memcontrol.c +6 -2
 
+	if (!folio_memcg_charged(folio))
+		return root_mem_cgroup;
+
  	rcu_read_lock();
-	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
-		memcg = root_mem_cgroup;
+	do {
+		memcg = folio_memcg(folio);
+	} while (unlikely(!css_tryget(&memcg->css)));

I went back to [1] where AI raised the following concern which I want to address:

Quoted 3 lines

If css_tryget() fails (e.g. refcount is 0), this loop spins indefinitely with the RCU read lock held. Is it guaranteed that folio_memcg() will return a different, alive memcg in subsequent iterations?

Will css_tryget() ever fail for the memcg returned by folio_memcg()? Let's suppose memcg of a given folio is being offlined. The objcg reparenting happens in memcg_reparent_objcgs() which is called in offline_css() chain and we know that the offline context holds a reference on the css being offlined (see css_killed_work_fn()).

Also let's suppose the offline process has the last reference on the memcg's css. Now we have following two scenarios:

Scenario 1:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_tryget(memcg)
css_put(css)

In the above case css_tryget() will not fail.

Scenario 2:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_put(css) // last reference
css_tryget(memcg)
// retry on failure

Quoted 7 lines

In the above case the context in get_mem_cgroup_from_folio() will retry and will get different memcg during reparenting happening before the last css_put(css).

So, I think we are good and AI is mistaken.

Folks, please check if I missed something.

LGTM, thank you for such a detailed analysis!

Quoted 30 lines
Quoted 4 lines

If the folio is isolated (e.g. via migrate_misplaced_folio()), it might be missed by reparenting logic that iterates LRU lists.

LRU isolation will not impact reparenting logic, so we can discount this as well.

Quoted 5 lines

In that case, the
folio would continue pointing to the dying memcg, leading to a hard lockup.

Also, folio_memcg() calls __folio_memcg(), which reads folio->memcg_data without READ_ONCE().

Oh I think I know why AI is confused. It is because it is looking at folio->memcg i.e. state with this patch only and not the state after the series. In the current state the folio holds the reference on memcg, so css_tryget() will never fail.

Quoted 7 lines

Since this loop waits for memcg_data to be updated
by another CPU (reparenting), could the compiler hoist the load out of the loop, preventing the update from being seen?

Finally, the previous code fell back to root_mem_cgroup on failure. Is it safe to remove that fallback? If css_tryget() fails unexpectedly, hanging seems more severe than the previous behavior of warning and falling back.

[1] https://lore.kernel.org/all/[email protected]/

#16 lore raw

Re: [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()

In reply to Shakeel Butt
Quoted 81 lines · Shakeel Butt
Quoted 33 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/memcontrol.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memcontrol.c +1 -1
index 982c9f5cf72cb..0458fc2e810ff 100644
@@ -991,14 +991,18 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)
  */
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
 {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
 	if (mem_cgroup_disabled())
 		return NULL;

mm/memcontrol.c struct mem_cgroup *get_mem_cgroup_from_current(void) · line 1005

Quoted 10 lines
mm/memcontrol.c +6 -2
 
+	if (!folio_memcg_charged(folio))
+		return root_mem_cgroup;
+
 	rcu_read_lock();
-	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
-		memcg = root_mem_cgroup;
+	do {
+		memcg = folio_memcg(folio);
+	} while (unlikely(!css_tryget(&memcg->css)));

I went back to [1] where AI raised the following concern which I want to address:

Quoted 3 lines

If css_tryget() fails (e.g. refcount is 0), this loop spins indefinitely with the RCU read lock held. Is it guaranteed that folio_memcg() will return a different, alive memcg in subsequent iterations?

Will css_tryget() ever fail for the memcg returned by folio_memcg()? Let's suppose memcg of a given folio is being offlined. The objcg reparenting happens in memcg_reparent_objcgs() which is called in offline_css() chain and we know that the offline context holds a reference on the css being offlined (see css_killed_work_fn()).

Also let's suppose the offline process has the last reference on the memcg's css. Now we have following two scenarios:

Scenario 1:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_tryget(memcg)
css_put(css)

In the above case css_tryget() will not fail.

Scenario 2:

get_mem_cgroup_from_folio() css_killed_work_fn()
memcg = folio_memcg(folio) offline_css(css)
memcg_reparent_objcgs()
css_put(css) // last reference
css_tryget(memcg)
// retry on failure

Quoted 7 lines

In the above case the context in get_mem_cgroup_from_folio() will retry and will get different memcg during reparenting happening before the last css_put(css).

So, I think we are good and AI is mistaken.

Folks, please check if I missed something.

LGTM and I think we're good.

Quoted 9 lines
Quoted 2 lines

If the folio is isolated (e.g. via migrate_misplaced_folio()), it might be missed by reparenting logic that iterates LRU lists.

LRU isolation will not impact reparenting logic, so we can discount this as well.

Quoted 3 lines

In that case, the
folio would continue pointing to the dying memcg, leading to a hard lockup.

Quoted 7 lines
Quoted 2 lines

Also, folio_memcg() calls __folio_memcg(), which reads folio->memcg_data without READ_ONCE().

Oh I think I know why AI is confused. It is because it is looking at folio->memcg i.e. state with this patch only and not the state after the series. In the current state the folio holds the reference on memcg, so css_tryget() will never fail.

Makes sense!

Quoted 9 lines
Quoted 7 lines

Since this loop waits for memcg_data to be updated
by another CPU (reparenting), could the compiler hoist the load out of the loop, preventing the update from being seen?

Finally, the previous code fell back to root_mem_cgroup on failure. Is it safe to remove that fallback? If css_tryget() fails unexpectedly, hanging seems more severe than the previous behavior of warning and falling back.

[1] https://lore.kernel.org/all/[email protected]/

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#17 lore raw

PATCH v3 9/30 buffer: prevent memory cgroup release in folio_alloc_buffers()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the function get_mem_cgroup_from_folio() is employed to safeguard against the release of the memory cgroup. This serves as a preparatory measure for the reparenting of the LRU pages.

---
 fs/buffer.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/buffer.c +2 -2
index b67791690ed33..d80b635cff162 100644
@@ -926,8 +926,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
 	long offset;
 	struct mem_cgroup *memcg, *old_memcg;
 
-	/* The folio lock pins the memcg */
-	memcg = folio_memcg(folio);
+	memcg = get_mem_cgroup_from_folio(folio);
 	old_memcg = set_active_memcg(memcg);
 
 	head = NULL;
@@ -948,6 +947,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
 	}
 out:
 	set_active_memcg(old_memcg);
+	mem_cgroup_put(memcg);
 	return head;
 /*
  * In case anything failed, we just free everything we got.
Signature · 1 line
2.20.1
author reviewer
#18 lore raw

PATCH v3 10/30 writeback: prevent memory cgroup release in writeback module

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the function get_mem_cgroup_css_from_folio() and the rcu read lock are employed to safeguard against the release of the memory cgroup.

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 fs/fs-writeback.c                | 22 +++++++++++-----------
 include/linux/memcontrol.h       |  9 +++++++--
 include/trace/events/writeback.h |  3 +++
 mm/memcontrol.c                  | 14 ++++++++------
 4 files changed, 29 insertions(+), 19 deletions(-)
fs/fs-writeback.c +11 -11
index d061666b733c7..d35606e3bc2e9 100644
@@ -279,15 +279,13 @@ void __inode_attach_wb(struct inode *inode, struct folio *folio)
 	if (inode_cgwb_enabled(inode)) {
 		struct cgroup_subsys_state *memcg_css;
 
-		if (folio) {
-			memcg_css = mem_cgroup_css_from_folio(folio);
-			wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC);
-		} else {
-			/* must pin memcg_css, see wb_get_create() */
+		/* must pin memcg_css, see wb_get_create() */
+		if (folio)
+			memcg_css = get_mem_cgroup_css_from_folio(folio);
+		else
 			memcg_css = task_get_css(current, memory_cgrp_id);
-			wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC);
-			css_put(memcg_css);
-		}
+		wb = wb_get_create(bdi, memcg_css, GFP_ATOMIC);
+		css_put(memcg_css);
 	}
 
 	if (!wb)
@@ -979,16 +977,16 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folio
 	if (!wbc->wb || wbc->no_cgroup_owner)
 		return;
 
-	css = mem_cgroup_css_from_folio(folio);
+	css = get_mem_cgroup_css_from_folio(folio);
 	/* dead cgroups shouldn't contribute to inode ownership arbitration */
 	if (!css_is_online(css))
-		return;
+		goto out;
 
 	id = css->id;
 
 	if (id == wbc->wb_id) {
 		wbc->wb_bytes += bytes;
-		return;
+		goto out;
 	}
 
 	if (id == wbc->wb_lcand_id)
@@ -1001,6 +999,8 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folio
 		wbc->wb_tcand_bytes += bytes;
 	else
 		wbc->wb_tcand_bytes -= min(bytes, wbc->wb_tcand_bytes);
+out:
+	css_put(css);
 }
 EXPORT_SYMBOL_GPL(wbc_account_cgroup_owner);
 
include/linux/memcontrol.h +7 -2
index 7b3d8f341ff10..6b987f7089ca4 100644
@@ -895,7 +895,7 @@ static inline bool mm_match_cgroup(struct mm_struct *mm,
 	return match;
 }
 
-struct cgroup_subsys_state *mem_cgroup_css_from_folio(struct folio *folio);
+struct cgroup_subsys_state *get_mem_cgroup_css_from_folio(struct folio *folio);
 ino_t page_cgroup_ino(struct page *page);
 
 static inline bool mem_cgroup_online(struct mem_cgroup *memcg)
@@ -1564,9 +1564,14 @@ static inline void mem_cgroup_track_foreign_dirty(struct folio *folio,
 	if (mem_cgroup_disabled())
 		return;
 
+	if (!folio_memcg_charged(folio))
+		return;
+
+	rcu_read_lock();
 	memcg = folio_memcg(folio);
-	if (unlikely(memcg && &memcg->css != wb->memcg_css))
+	if (unlikely(&memcg->css != wb->memcg_css))
 		mem_cgroup_track_foreign_dirty_slowpath(folio, wb);
+	rcu_read_unlock();
 }
 
 void mem_cgroup_flush_foreign(struct bdi_writeback *wb);
include/trace/events/writeback.h +3 -0
index 4d3d8c8f3a1bc..b849b8cc96b1e 100644
@@ -294,7 +294,10 @@ TRACE_EVENT(track_foreign_dirty,
 		__entry->ino		= inode ? inode->i_ino : 0;
 		__entry->memcg_id	= wb->memcg_css->id;
 		__entry->cgroup_ino	= __trace_wb_assign_cgroup(wb);
+
+		rcu_read_lock();
 		__entry->page_cgroup_ino = cgroup_ino(folio_memcg(folio)->css.cgroup);
+		rcu_read_unlock();
 	),
 
 	TP_printk("bdi %s[%llu]: ino=%lu memcg_id=%u cgroup_ino=%lu page_cgroup_ino=%lu",
mm/memcontrol.c +8 -6
index 0458fc2e810ff..b5d4139ea1aaa 100644
@@ -243,7 +243,7 @@ DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);
 EXPORT_SYMBOL(memcg_bpf_enabled_key);
 
 /**
- * mem_cgroup_css_from_folio - css of the memcg associated with a folio
+ * get_mem_cgroup_css_from_folio - acquire a css of the memcg associated with a folio
  * @folio: folio of interest
  *
  * If memcg is bound to the default hierarchy, css of the memcg associated
@@ -253,14 +253,16 @@ EXPORT_SYMBOL(memcg_bpf_enabled_key);
  * If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup
  * is returned.
  */
-struct cgroup_subsys_state *mem_cgroup_css_from_folio(struct folio *folio)
+struct cgroup_subsys_state *get_mem_cgroup_css_from_folio(struct folio *folio)
 {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
-	if (!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		memcg = root_mem_cgroup;
+	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return &root_mem_cgroup->css;
 
-	return &memcg->css;
+	memcg = get_mem_cgroup_from_folio(folio);
+
+	return memcg ? &memcg->css : &root_mem_cgroup->css;
 }
 
 /**
Signature · 1 line
2.20.1
author reviewer
#19 lore raw

PATCH v3 11/30 mm: memcontrol: prevent memory cgroup release in count_memcg_folio_events()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in count_memcg_folio_events().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 include/linux/memcontrol.h | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +8 -3
index 6b987f7089ca4..f1556759d0d3f 100644
@@ -976,10 +976,15 @@ void count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx,
 static inline void count_memcg_folio_events(struct folio *folio,
 		enum vm_event_item idx, unsigned long nr)
 {
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 
-	if (memcg)
-		count_memcg_events(memcg, idx, nr);
+	if (!folio_memcg_charged(folio))
+		return;
+
+	rcu_read_lock();
+	memcg = folio_memcg(folio);
+	count_memcg_events(memcg, idx, nr);
+	rcu_read_unlock();
 }
 
 static inline void count_memcg_events_mm(struct mm_struct *mm,
Signature · 1 line
2.20.1
author reviewer
#20 lore raw

PATCH v3 12/30 mm: page_io: prevent memory cgroup release in page_io module

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in swap_writeout() and bio_associate_blkg_from_page().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/page_io.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
mm/page_io.c +6 -2
index a2c034660c805..63b262f4c5a9b 100644
@@ -276,10 +276,14 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
 		count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
 		goto out_unlock;
 	}
+
+	rcu_read_lock();
 	if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
+		rcu_read_unlock();
 		folio_mark_dirty(folio);
 		return AOP_WRITEPAGE_ACTIVATE;
 	}
+	rcu_read_unlock();
 
 	__swap_writepage(folio, swap_plug);
 	return 0;
@@ -307,11 +311,11 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
 	struct cgroup_subsys_state *css;
 	struct mem_cgroup *memcg;
 
-	memcg = folio_memcg(folio);
-	if (!memcg)
+	if (!folio_memcg_charged(folio))
 		return;
 
 	rcu_read_lock();
+	memcg = folio_memcg(folio);
 	css = cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys);
 	bio_associate_blkg_from_css(bio, css);
 	rcu_read_unlock();
Signature · 1 line
2.20.1
author reviewer
#21 lore raw

PATCH v3 13/30 mm: migrate: prevent memory cgroup release in folio_migrate_mapping()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In __folio_migrate_mapping(), the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_migrate_mapping().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/migrate.c | 2 ++
 1 file changed, 2 insertions(+)
mm/migrate.c +2 -0
index 4688b9e38cd2f..1aec3f5509538 100644
@@ -671,6 +671,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 		struct lruvec *old_lruvec, *new_lruvec;
 		struct mem_cgroup *memcg;
 
+		rcu_read_lock();
 		memcg = folio_memcg(folio);
 		old_lruvec = mem_cgroup_lruvec(memcg, oldzone->zone_pgdat);
 		new_lruvec = mem_cgroup_lruvec(memcg, newzone->zone_pgdat);
@@ -698,6 +699,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 			mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr);
 			__mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr);
 		}
+		rcu_read_unlock();
 	}
 	local_irq_enable();
 
Signature · 1 line
2.20.1
author reviewer
#22 lore raw

PATCH v3 14/30 mm: mglru: prevent memory cgroup release in mglru

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mglru.

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/vmscan.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
mm/vmscan.c +16 -6
index 20cd54c5cbc79..f206d4dac9e77 100644
@@ -3481,8 +3481,10 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
 	if (folio_nid(folio) != pgdat->node_id)
 		return NULL;
 
+	rcu_read_lock();
 	if (folio_memcg(folio) != memcg)
-		return NULL;
+		folio = NULL;
+	rcu_read_unlock();
 
 	return folio;
 }
@@ -4239,12 +4241,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	unsigned long addr = pvmw->address;
 	struct vm_area_struct *vma = pvmw->vma;
 	struct folio *folio = pfn_folio(pvmw->pfn);
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 	struct pglist_data *pgdat = folio_pgdat(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
-	struct lru_gen_mm_state *mm_state = get_mm_state(lruvec);
-	DEFINE_MAX_SEQ(lruvec);
-	int gen = lru_gen_from_seq(max_seq);
+	struct lruvec *lruvec;
+	struct lru_gen_mm_state *mm_state;
+	unsigned long max_seq;
+	int gen;
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
@@ -4279,6 +4281,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 		}
 	}
 
+	memcg = get_mem_cgroup_from_folio(folio);
+	lruvec = mem_cgroup_lruvec(memcg, pgdat);
+	max_seq = READ_ONCE((lruvec)->lrugen.max_seq);
+	gen = lru_gen_from_seq(max_seq);
+	mm_state = get_mm_state(lruvec);
+
 	lazy_mmu_mode_enable();
 
 	pte -= (addr - start) / PAGE_SIZE;
@@ -4319,6 +4327,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	if (mm_state && suitable_to_scan(i, young))
 		update_bloom_filter(mm_state, max_seq, pvmw->pmd);
 
+	mem_cgroup_put(memcg);
+
 	return true;
 }
 
Signature · 1 line
2.20.1
maintainer Acked-by on v3 14/30
#23 lore raw

Re: [PATCH v3 14/30] mm: mglru: prevent memory cgroup release in mglru

Quoted 7 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

Quoted 8 lines

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mglru.

This serves as a preparatory measure for the reparenting of the LRU pages.

Reviewed-by on v3 14/30
#24 lore raw

Re: [PATCH v3 14/30] mm: mglru: prevent memory cgroup release in mglru

In reply to Qi Zheng
Quoted 10 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mglru.

Quoted 6 lines

This serves as a preparatory measure for the reparenting of the LRU pages.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#25 lore raw

PATCH v3 15/30 mm: memcontrol: prevent memory cgroup release in mem_cgroup_swap_full()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mem_cgroup_swap_full().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/memcontrol.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
mm/memcontrol.c +10 -8
index b5d4139ea1aaa..548e67dbf2386 100644
@@ -5283,27 +5283,29 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
 bool mem_cgroup_swap_full(struct folio *folio)
 {
 	struct mem_cgroup *memcg;
+	bool ret = false;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 
 	if (vm_swap_full())
 		return true;
-	if (do_memsw_account())
-		return false;
+	if (do_memsw_account() || !folio_memcg_charged(folio))
+		return ret;
 
+	rcu_read_lock();
 	memcg = folio_memcg(folio);
-	if (!memcg)
-		return false;
-
 	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
 		unsigned long usage = page_counter_read(&memcg->swap);
 
 		if (usage * 2 >= READ_ONCE(memcg->swap.high) ||
-		    usage * 2 >= READ_ONCE(memcg->swap.max))
-			return true;
+		    usage * 2 >= READ_ONCE(memcg->swap.max)) {
+			ret = true;
+			break;
+		}
 	}
+	rcu_read_unlock();
 
-	return false;
+	return ret;
 }
 
 static int __init setup_swap_account(char *s)
Signature · 1 line
2.20.1
author reviewer
#26 lore raw

PATCH v3 16/30 mm: workingset: prevent memory cgroup release in lru_gen_eviction()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in lru_gen_eviction().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/workingset.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
mm/workingset.c +7 -2
index ed12744d93a29..79b40f058cd48 100644
@@ -241,11 +241,14 @@ static void *lru_gen_eviction(struct folio *folio)
 	int refs = folio_lru_refs(folio);
 	bool workingset = folio_test_workingset(folio);
 	int tier = lru_tier_from_refs(refs, workingset);
-	struct mem_cgroup *memcg = folio_memcg(folio);
+	struct mem_cgroup *memcg;
 	struct pglist_data *pgdat = folio_pgdat(folio);
+	unsigned short memcg_id;
 
 	BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH > BITS_PER_LONG - EVICTION_SHIFT);
 
+	rcu_read_lock();
+	memcg = folio_memcg(folio);
 	lruvec = mem_cgroup_lruvec(memcg, pgdat);
 	lrugen = &lruvec->lrugen;
 	min_seq = READ_ONCE(lrugen->min_seq[type]);
@@ -253,8 +256,10 @@ static void *lru_gen_eviction(struct folio *folio)
 
 	hist = lru_hist_from_seq(min_seq);
 	atomic_long_add(delta, &lrugen->evicted[hist][type][tier]);
+	memcg_id = mem_cgroup_private_id(memcg);
+	rcu_read_unlock();
 
-	return pack_shadow(mem_cgroup_private_id(memcg), pgdat, token, workingset);
+	return pack_shadow(memcg_id, pgdat, token, workingset);
 }
 
 /*
Signature · 1 line
2.20.1
author reviewer
#27 lore raw

PATCH v3 17/30 mm: thp: prevent memory cgroup release in folio_split_queue_lock{_irqsave}()

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_split_queue_lock{_irqsave}().

---
 mm/huge_memory.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
mm/huge_memory.c +18 -2
index 99515838faa51..eb611239c1a74 100644
@@ -1154,13 +1154,29 @@ split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flags
 
 static struct deferred_split *folio_split_queue_lock(struct folio *folio)
 {
-	return split_queue_lock(folio_nid(folio), folio_memcg(folio));
+	struct deferred_split *queue;
+
+	rcu_read_lock();
+	queue = split_queue_lock(folio_nid(folio), folio_memcg(folio));
+	/*
+	 * The memcg destruction path is acquiring the split queue lock for
+	 * reparenting. Once you have it locked, it's safe to drop the rcu lock.
+	 */
+	rcu_read_unlock();
+
+	return queue;
 }
 
 static struct deferred_split *
 folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags)
 {
-	return split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
+	struct deferred_split *queue;
+
+	rcu_read_lock();
+	queue = split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
+	rcu_read_unlock();
+
+	return queue;
 }
 
 static inline void split_queue_unlock(struct deferred_split *queue)
Signature · 1 line
2.20.1
reviewer Acked-by on v3 17/30
#28 lore raw

Re: [PATCH v3 17/30] mm: thp: prevent memory cgroup release in folio_split_queue_lock{_irqsave}()

Quoted 11 lines · Qi Zheng

From: Qi Zheng <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

Quoted 8 lines

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_split_queue_lock{_irqsave}().

author reviewer
#29 lore raw

PATCH v3 18/30 mm: zswap: prevent memory cgroup release in zswap_compress()

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in zswap_compress().

---
 mm/zswap.c | 3 +++
 1 file changed, 3 insertions(+)
mm/zswap.c +3 -0
index a3811b05ab579..cd926e1c03c92 100644
@@ -892,11 +892,14 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,
 	 * to the active LRU list in the case.
 	 */
 	if (comp_ret || !dlen || dlen >= PAGE_SIZE) {
+		rcu_read_lock();
 		if (!mem_cgroup_zswap_writeback_enabled(
 					folio_memcg(page_folio(page)))) {
+			rcu_read_unlock();
 			comp_ret = comp_ret ? comp_ret : -EINVAL;
 			goto unlock;
 		}
+		rcu_read_unlock();
 		comp_ret = 0;
 		dlen = PAGE_SIZE;
 		dst = kmap_local_page(page);
Signature · 1 line
2.20.1
reviewer Acked-by on v3 18/30
#30 lore raw

Re: [PATCH v3 18/30] mm: zswap: prevent memory cgroup release in zswap_compress()

Quoted 11 lines · Qi Zheng

From: Qi Zheng <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

Quoted 6 lines

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in zswap_compress().

Reviewed-by on v3 18/30
#31 lore raw

Re: [PATCH v3 18/30] mm: zswap: prevent memory cgroup release in zswap_compress()

In reply to Qi Zheng
Quoted 7 lines · Qi Zheng

From: Qi Zheng <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.

Quoted 7 lines

In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in zswap_compress().

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#32 lore raw

PATCH v3 19/30 mm: workingset: prevent lruvec release in workingset_refault()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.

In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_refault().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/workingset.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
mm/workingset.c +6 -2
index 79b40f058cd48..5fbf316fb0e71 100644
@@ -539,6 +539,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
 void workingset_refault(struct folio *folio, void *shadow)
 {
 	bool file = folio_is_file_lru(folio);
+	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
 	bool workingset;
 	long nr;
@@ -560,11 +561,12 @@ void workingset_refault(struct folio *folio, void *shadow)
 	 * locked to guarantee folio_memcg() stability throughout.
 	 */
 	nr = folio_nr_pages(folio);
-	lruvec = folio_lruvec(folio);
+	memcg = get_mem_cgroup_from_folio(folio);
+	lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
 	mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
 
 	if (!workingset_test_recent(shadow, file, &workingset, true))
-		return;
+		goto out;
 
 	folio_set_active(folio);
 	workingset_age_nonresident(lruvec, nr);
@@ -580,6 +582,8 @@ void workingset_refault(struct folio *folio, void *shadow)
 		lru_note_cost_refault(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr);
 	}
+out:
+	mem_cgroup_put(memcg);
 }
 
 /**
Signature · 1 line
2.20.1
maintainer Acked-by on v3 19/30
#33 lore raw

Re: [PATCH v3 19/30] mm: workingset: prevent lruvec release in workingset_refault()

Quoted 10 lines · Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.

In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_refault().

Quoted 6 lines

This serves as a preparatory measure for the reparenting of the LRU pages.

author reviewer
#34 lore raw

PATCH v3 20/30 mm: zswap: prevent lruvec release in zswap_folio_swapin()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.

In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in zswap_folio_swapin().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/zswap.c | 2 ++
 1 file changed, 2 insertions(+)
mm/zswap.c +2 -0
index cd926e1c03c92..2f410507cbc8b 100644
@@ -663,8 +663,10 @@ void zswap_folio_swapin(struct folio *folio)
 	struct lruvec *lruvec;
 
 	if (folio) {
+		rcu_read_lock();
 		lruvec = folio_lruvec(folio);
 		atomic_long_inc(&lruvec->zswap_lruvec_state.nr_disk_swapins);
+		rcu_read_unlock();
 	}
 }
 
Signature · 1 line
2.20.1
author reviewer
#35 lore raw

PATCH v3 21/30 mm: swap: prevent lruvec release in lru_gen_clear_refs()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.

In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in lru_gen_clear_refs().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/swap.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
mm/swap.c +5 -3
index 245ba159e01d7..cb1148a92d8ec 100644
@@ -412,18 +412,20 @@ static void lru_gen_inc_refs(struct folio *folio)
 
 static bool lru_gen_clear_refs(struct folio *folio)
 {
-	struct lru_gen_folio *lrugen;
 	int gen = folio_lru_gen(folio);
 	int type = folio_is_file_lru(folio);
+	unsigned long seq;
 
 	if (gen < 0)
 		return true;
 
 	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
 
-	lrugen = &folio_lruvec(folio)->lrugen;
+	rcu_read_lock();
+	seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
+	rcu_read_unlock();
 	/* whether can do without shuffling under the LRU lock */
-	return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
+	return gen == lru_gen_from_seq(seq);
 }
 
 #else /* !CONFIG_LRU_GEN */
Signature · 1 line
2.20.1
author reviewer
#36 lore raw

PATCH v3 22/30 mm: workingset: prevent lruvec release in workingset_activation()

In reply to Qi Zheng

From: Muchun Song <[email protected]>

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.

In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_activation().

This serves as a preparatory measure for the reparenting of the LRU pages.

---
 mm/workingset.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/workingset.c +4 -1
index 5fbf316fb0e71..2be53098f6282 100644
@@ -596,8 +596,11 @@ void workingset_activation(struct folio *folio)
 	 * Filter non-memcg pages here, e.g. unmap can call
 	 * mark_page_accessed() on VDSO pages.
 	 */
-	if (mem_cgroup_disabled() || folio_memcg_charged(folio))
+	if (mem_cgroup_disabled() || folio_memcg_charged(folio)) {
+		rcu_read_lock();
 		workingset_age_nonresident(folio_lruvec(folio), folio_nr_pages(folio));
+		rcu_read_unlock();
+	}
 }
 
 /*
Signature · 1 line
2.20.1
author reviewer
#37 lore raw

PATCH v3 23/30 mm: do not open-code lruvec lock

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(), lruvec_lock_irq() and lruvec_lock_irqsave().

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

---
 include/linux/memcontrol.h |  5 +++++
 mm/vmscan.c                | 38 +++++++++++++++++++-------------------
 2 files changed, 24 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +5 -0
index f1556759d0d3f..4b6f20dc694ba 100644
@@ -1499,6 +1499,11 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
 	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
+static inline void lruvec_lock_irq(struct lruvec *lruvec)
+{
+	spin_lock_irq(&lruvec->lru_lock);
+}
+
 static inline void lruvec_unlock(struct lruvec *lruvec)
 {
 	spin_unlock(&lruvec->lru_lock);
mm/vmscan.c +19 -19
index f206d4dac9e77..c48ff6e05e004 100644
@@ -2020,7 +2020,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 
 	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &folio_list,
 				     &nr_scanned, sc, lru);
@@ -2032,7 +2032,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned);
 	__count_vm_events(PGSCAN_ANON + file, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	if (nr_taken == 0)
 		return 0;
@@ -2051,7 +2051,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
 	count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 	lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
 					nr_scanned - nr_reclaimed);
 
@@ -2130,7 +2130,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 
 	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &l_hold,
 				     &nr_scanned, sc, lru);
@@ -2141,7 +2141,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 		__count_vm_events(PGREFILL, nr_scanned);
 	count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	while (!list_empty(&l_hold)) {
 		struct folio *folio;
@@ -2197,7 +2197,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
 	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 	lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
 	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
 			nr_deactivate, nr_rotated, sc->priority, file);
@@ -3832,9 +3832,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 		}
 
 		if (walk->batched) {
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
 			reset_batch_size(walk);
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
 		}
 
 		cond_resched();
@@ -3993,7 +3993,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	if (seq < READ_ONCE(lrugen->max_seq))
 		return false;
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
 
@@ -4008,7 +4008,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 		if (inc_min_seq(lruvec, type, swappiness))
 			continue;
 
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
 		cond_resched();
 		goto restart;
 	}
@@ -4043,7 +4043,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	/* make sure preceding modifications appear */
 	smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
 unlock:
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	return success;
 }
@@ -4739,7 +4739,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &type, &list);
 
@@ -4748,7 +4748,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	if (evictable_min_seq(lrugen->min_seq, swappiness) + MIN_NR_GENS > lrugen->max_seq)
 		scanned = 0;
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	if (list_empty(&list))
 		return scanned;
@@ -4786,9 +4786,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	walk = current->reclaim_state->mm_walk;
 	if (walk && walk->batched) {
 		walk->lruvec = lruvec;
-		spin_lock_irq(&lruvec->lru_lock);
+		lruvec_lock_irq(lruvec);
 		reset_batch_size(walk);
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
 	}
 
 	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -5226,7 +5226,7 @@ static void lru_gen_change_state(bool enabled)
 		for_each_node(nid) {
 			struct lruvec *lruvec = get_lruvec(memcg, nid);
 
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
 
 			VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
 			VM_WARN_ON_ONCE(!state_is_valid(lruvec));
@@ -5234,12 +5234,12 @@ static void lru_gen_change_state(bool enabled)
 			lruvec->lrugen.enabled = enabled;
 
 			while (!(enabled ? fill_evictable(lruvec) : drain_evictable(lruvec))) {
-				spin_unlock_irq(&lruvec->lru_lock);
+				lruvec_unlock_irq(lruvec);
 				cond_resched();
-				spin_lock_irq(&lruvec->lru_lock);
+				lruvec_lock_irq(lruvec);
 			}
 
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
 		}
 
 		cond_resched();
Signature · 1 line
2.20.1
#38 lore raw

Re: [PATCH v3 23/30] mm: do not open-code lruvec lock

Quoted 4 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(),

                                  ~~ typo, 'no'?
Quoted 4 lines

lruvec_lock_irq() and lruvec_lock_irqsave().

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code

I didn't see lruvec_lock() is introduced in this patch, do I miss anthing?

Quoted 182 lines

places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

---
 include/linux/memcontrol.h |  5 +++++
 mm/vmscan.c                | 38 +++++++++++++++++++-------------------
 2 files changed, 24 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +5 -0
index f1556759d0d3f..4b6f20dc694ba 100644
@@ -1499,6 +1499,11 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
 	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
+static inline void lruvec_lock_irq(struct lruvec *lruvec)
+{
+	spin_lock_irq(&lruvec->lru_lock);
+}
+
 static inline void lruvec_unlock(struct lruvec *lruvec)
 {
 	spin_unlock(&lruvec->lru_lock);
mm/vmscan.c +19 -19
index f206d4dac9e77..c48ff6e05e004 100644
@@ -2020,7 +2020,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 
 	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &folio_list,
 				     &nr_scanned, sc, lru);
@@ -2032,7 +2032,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned);
 	__count_vm_events(PGSCAN_ANON + file, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	if (nr_taken == 0)
 		return 0;
@@ -2051,7 +2051,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
 	count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 	lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
 					nr_scanned - nr_reclaimed);
 
@@ -2130,7 +2130,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 
 	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &l_hold,
 				     &nr_scanned, sc, lru);
@@ -2141,7 +2141,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 		__count_vm_events(PGREFILL, nr_scanned);
 	count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	while (!list_empty(&l_hold)) {
 		struct folio *folio;
@@ -2197,7 +2197,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
 	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 	lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
 	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
 			nr_deactivate, nr_rotated, sc->priority, file);
@@ -3832,9 +3832,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
 		}
 
 		if (walk->batched) {
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
 			reset_batch_size(walk);
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
 		}
 
 		cond_resched();
@@ -3993,7 +3993,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	if (seq < READ_ONCE(lrugen->max_seq))
 		return false;
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
 
@@ -4008,7 +4008,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 		if (inc_min_seq(lruvec, type, swappiness))
 			continue;
 
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
 		cond_resched();
 		goto restart;
 	}
@@ -4043,7 +4043,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	/* make sure preceding modifications appear */
 	smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
 unlock:
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	return success;
 }
@@ -4739,7 +4739,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
 	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
 	scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &type, &list);
 
@@ -4748,7 +4748,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	if (evictable_min_seq(lrugen->min_seq, swappiness) + MIN_NR_GENS > lrugen->max_seq)
 		scanned = 0;
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
 	if (list_empty(&list))
 		return scanned;
@@ -4786,9 +4786,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	walk = current->reclaim_state->mm_walk;
 	if (walk && walk->batched) {
 		walk->lruvec = lruvec;
-		spin_lock_irq(&lruvec->lru_lock);
+		lruvec_lock_irq(lruvec);
 		reset_batch_size(walk);
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
 	}
 
 	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -5226,7 +5226,7 @@ static void lru_gen_change_state(bool enabled)
 		for_each_node(nid) {
 			struct lruvec *lruvec = get_lruvec(memcg, nid);
 
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
 
 			VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
 			VM_WARN_ON_ONCE(!state_is_valid(lruvec));
@@ -5234,12 +5234,12 @@ static void lru_gen_change_state(bool enabled)
 			lruvec->lrugen.enabled = enabled;
 
 			while (!(enabled ? fill_evictable(lruvec) : drain_evictable(lruvec))) {
-				spin_unlock_irq(&lruvec->lru_lock);
+				lruvec_unlock_irq(lruvec);
 				cond_resched();
-				spin_lock_irq(&lruvec->lru_lock);
+				lruvec_lock_irq(lruvec);
 			}
 
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
 		}
 
 		cond_resched();
Signature · 1 line
2.20.1
author reviewer
#39 lore raw

Re: [PATCH v3 23/30] mm: do not open-code lruvec lock

Quoted 5 lines · Baoquan He
Quoted 4 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(),

Quoted 8 lines
                                   ~~ typo, 'no'?
Quoted 4 lines

lruvec_lock_irq() and lruvec_lock_irqsave().

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code

I didn't see lruvec_lock() is introduced in this patch, do I miss anthing?

I forgot to update commit message when I deleted lruvec_lock(), will modify it in the next version.

Thanks!

Quoted 184 lines
Quoted 183 lines

places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

---
  include/linux/memcontrol.h |  5 +++++
  mm/vmscan.c                | 38 +++++++++++++++++++-------------------
  2 files changed, 24 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +5 -0
index f1556759d0d3f..4b6f20dc694ba 100644
@@ -1499,6 +1499,11 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
  	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
  }
 
+static inline void lruvec_lock_irq(struct lruvec *lruvec)
+{
+	spin_lock_irq(&lruvec->lru_lock);
+}
+
  static inline void lruvec_unlock(struct lruvec *lruvec)
  {
  	spin_unlock(&lruvec->lru_lock);
mm/vmscan.c +19 -19
index f206d4dac9e77..c48ff6e05e004 100644
@@ -2020,7 +2020,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 
  	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
  	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &folio_list,
  				     &nr_scanned, sc, lru);
@@ -2032,7 +2032,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
  	count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned);
  	__count_vm_events(PGSCAN_ANON + file, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
  	if (nr_taken == 0)
  		return 0;
@@ -2051,7 +2051,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
  	count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);
  	count_vm_events(PGSTEAL_ANON + file, nr_reclaimed);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
  	lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
  					nr_scanned - nr_reclaimed);
 
@@ -2130,7 +2130,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 
  	lru_add_drain();
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
  	nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &l_hold,
  				     &nr_scanned, sc, lru);
@@ -2141,7 +2141,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
  		__count_vm_events(PGREFILL, nr_scanned);
  	count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned);
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
  	while (!list_empty(&l_hold)) {
  		struct folio *folio;
@@ -2197,7 +2197,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
  	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
  	mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
  	lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
  	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
  			nr_deactivate, nr_rotated, sc->priority, file);
@@ -3832,9 +3832,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk)
  		}
 
  		if (walk->batched) {
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
  			reset_batch_size(walk);
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
  		}
 
  		cond_resched();
@@ -3993,7 +3993,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
  	if (seq < READ_ONCE(lrugen->max_seq))
  		return false;
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
  	VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
 
@@ -4008,7 +4008,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
  		if (inc_min_seq(lruvec, type, swappiness))
  			continue;
 
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
  		cond_resched();
  		goto restart;
  	}
@@ -4043,7 +4043,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
  	/* make sure preceding modifications appear */
  	smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
  unlock:
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
  	return success;
  }
@@ -4739,7 +4739,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
  	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
  	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
 
-	spin_lock_irq(&lruvec->lru_lock);
+	lruvec_lock_irq(lruvec);
 
  	scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &type, &list);
 
@@ -4748,7 +4748,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
  	if (evictable_min_seq(lrugen->min_seq, swappiness) + MIN_NR_GENS > lrugen->max_seq)
  		scanned = 0;
 
-	spin_unlock_irq(&lruvec->lru_lock);
+	lruvec_unlock_irq(lruvec);
 
  	if (list_empty(&list))
  		return scanned;
@@ -4786,9 +4786,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
  	walk = current->reclaim_state->mm_walk;
  	if (walk && walk->batched) {
  		walk->lruvec = lruvec;
-		spin_lock_irq(&lruvec->lru_lock);
+		lruvec_lock_irq(lruvec);
  		reset_batch_size(walk);
-		spin_unlock_irq(&lruvec->lru_lock);
+		lruvec_unlock_irq(lruvec);
  	}
 
  	mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -5226,7 +5226,7 @@ static void lru_gen_change_state(bool enabled)
  		for_each_node(nid) {
  			struct lruvec *lruvec = get_lruvec(memcg, nid);
 
-			spin_lock_irq(&lruvec->lru_lock);
+			lruvec_lock_irq(lruvec);
 
  			VM_WARN_ON_ONCE(!seq_is_valid(lruvec));
  			VM_WARN_ON_ONCE(!state_is_valid(lruvec));
@@ -5234,12 +5234,12 @@ static void lru_gen_change_state(bool enabled)
  			lruvec->lrugen.enabled = enabled;
 
  			while (!(enabled ? fill_evictable(lruvec) : drain_evictable(lruvec))) {
-				spin_unlock_irq(&lruvec->lru_lock);
+				lruvec_unlock_irq(lruvec);
  				cond_resched();
-				spin_lock_irq(&lruvec->lru_lock);
+				lruvec_lock_irq(lruvec);
  			}
 
-			spin_unlock_irq(&lruvec->lru_lock);
+			lruvec_unlock_irq(lruvec);
  		}
 
  		cond_resched();
Signature · 1 line
2.20.1
reviewer Acked-by on v3 23/30
#40 lore raw

Re: [PATCH v3 23/30] mm: do not open-code lruvec lock

In reply to Qi Zheng
Quoted 10 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(), lruvec_lock_irq() and lruvec_lock_irqsave().

Quoted 7 lines

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

maintainer Acked-by on v3 23/30
#41 lore raw

Re: [PATCH v3 23/30] mm: do not open-code lruvec lock

In reply to Qi Zheng
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(), lruvec_lock_irq() and lruvec_lock_irqsave().

Quoted 7 lines

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

Reviewed-by on v3 23/30
#42 lore raw

Re: [PATCH v3 23/30] mm: do not open-code lruvec lock

In reply to Qi Zheng
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but not the paired lruvec_lock(), lruvec_lock_irq() and lruvec_lock_irqsave().

Quoted 8 lines

There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock() and lruvec_lock_irq(), and change all open-code places to use these helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.

---

Assuming Baoquan's comment will be addressed,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#43 lore raw

PATCH v3 24/30 mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
    rcu_read_lock();
retry:
    lruvec = folio_lruvec(folio);
    /* There is a possibility of folio reparenting at this point. */
    spin_lock(&lruvec->lru_lock);
    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
        /*
         * The wrong lruvec lock was acquired, and a retry is required.
         * This is because the folio resides on the parent memcg lruvec
         * list.
         */
        spin_unlock(&lruvec->lru_lock);
        goto retry;
    }

    /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
    spin_lock(&lruvec->lru_lock);
    spin_lock(&lruvec_parent->lru_lock);
    /* Transfer folios from the lruvec list to the parent's. */
    spin_unlock(&lruvec_parent->lru_lock);
    spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
 include/linux/memcontrol.h | 45 +++++++++++++++++++----------
 include/linux/swap.h       |  1 +
 mm/compaction.c            | 29 +++++++++++++++----
 mm/memcontrol.c            | 59 +++++++++++++++++++++-----------------
 mm/swap.c                  |  4 +++
 5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +30 -15
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
  * folio_lruvec - return lruvec for isolating/putting an LRU folio
  * @folio: Pointer to the folio.
  *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
  */
 static inline struct lruvec *folio_lruvec(struct folio *folio)
 {
@@ -761,18 +769,15 @@ struct mem_cgroup *get_mem_cgroup_from_current(void);
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
 
 struct lruvec *folio_lruvec_lock(struct folio *folio);
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
 struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
 struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
 						unsigned long *flags);
-
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio);
-#else
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-#endif
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
 
 static inline
 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
@@ -1199,11 +1204,6 @@ static inline struct lruvec *folio_lruvec(struct folio *folio)
 	return &pgdat->__lruvec;
 }
 
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-
 static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
 {
 	return NULL;
@@ -1262,6 +1262,7 @@ static inline struct lruvec *folio_lruvec_lock(struct folio *folio)
 {
 	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
 	spin_lock(&pgdat->__lruvec.lru_lock);
 	return &pgdat->__lruvec;
 }
@@ -1270,6 +1271,7 @@ static inline struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
 {
 	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
 	spin_lock_irq(&pgdat->__lruvec.lru_lock);
 	return &pgdat->__lruvec;
 }
@@ -1279,6 +1281,7 @@ static inline struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
 {
 	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
 	spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
 	return &pgdat->__lruvec;
 }
@@ -1500,24 +1503,36 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
 }
 
 static inline void lruvec_lock_irq(struct lruvec *lruvec)
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
 {
+	rcu_read_lock();
 	spin_lock_irq(&lruvec->lru_lock);
 }
 
 static inline void lruvec_unlock(struct lruvec *lruvec)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
 {
 	spin_unlock(&lruvec->lru_lock);
+	rcu_read_unlock();
 }
 
 static inline void lruvec_unlock_irq(struct lruvec *lruvec)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
 {
 	spin_unlock_irq(&lruvec->lru_lock);
+	rcu_read_unlock();
 }
 
 static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
 		unsigned long flags)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
 {
 	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+	rcu_read_unlock();
 }
 
 /* Test requires a stable folio->memcg binding, see folio_memcg() */
include/linux/swap.h +1 -0
index 62fc7499b4089..e60f45b48e74d 100644
@@ -330,6 +330,7 @@ extern unsigned long totalreserve_pages;
 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
 		unsigned int nr_io, unsigned int nr_rotated)
 		__releases(lruvec->lru_lock);
+		__releases(rcu)
 void lru_note_cost_refault(struct folio *);
 void folio_add_lru(struct folio *);
 void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
mm/compaction.c +23 -6
index c3e338aaa0ffb..3648ce22c8072 100644
@@ -518,6 +518,24 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,
 	return true;
 }
 
+static struct lruvec *
+compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags,
+				  struct compact_control *cc)
+{
+	struct lruvec *lruvec;
+
+	rcu_read_lock();
+retry:
+	lruvec = folio_lruvec(folio);
+	compact_lock_irqsave(&lruvec->lru_lock, flags, cc);
+	if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+		spin_unlock_irqrestore(&lruvec->lru_lock, *flags);
+		goto retry;
+	}
+
+	return lruvec;
+}
+
 /*
  * Compaction requires the taking of some coarse locks that are potentially
  * very heavily contended. The lock should be periodically unlocked to avoid
@@ -839,7 +857,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 {
 	pg_data_t *pgdat = cc->zone->zone_pgdat;
 	unsigned long nr_scanned = 0, nr_isolated = 0;
-	struct lruvec *lruvec;
+	struct lruvec *lruvec = NULL;
 	unsigned long flags = 0;
 	struct lruvec *locked = NULL;
 	struct folio *folio = NULL;
@@ -1153,18 +1171,17 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		if (!folio_test_clear_lru(folio))
 			goto isolate_fail_put;
 
-		lruvec = folio_lruvec(folio);
+		if (locked)
+			lruvec = folio_lruvec(folio);
 
 		/* If we already hold the lock, we can skip some rechecking */
-		if (lruvec != locked) {
+		if (lruvec != locked || !locked) {
 			if (locked)
 				lruvec_unlock_irqrestore(locked, flags);
 
-			compact_lock_irqsave(&lruvec->lru_lock, &flags, cc);
+			lruvec = compact_folio_lruvec_lock_irqsave(folio, &flags, cc);
 			locked = lruvec;
 
-			lruvec_memcg_debug(lruvec, folio);
-
 			/*
 			 * Try get exclusive access under lock. If marked for
 			 * skip, the scan is aborted unless the current context
mm/memcontrol.c +33 -26

99 lines not on this page · raw message

mm/swap.c +4 -0
index cb1148a92d8ec..7e53479ca1732 100644
@@ -240,6 +240,7 @@ void folio_rotate_reclaimable(struct folio *folio)
 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
 		unsigned int nr_io, unsigned int nr_rotated)
 		__releases(lruvec->lru_lock)
+		__releases(rcu)
 {
 	unsigned long cost;
 
@@ -253,6 +254,7 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
 	cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
 	if (!cost) {
 		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();
 		return;
 	}
 
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
 		}
 
 		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();
 		lruvec = parent_lruvec(lruvec);
 		if (!lruvec)
 			break;
+		rcu_read_lock();
 		spin_lock_irq(&lruvec->lru_lock);
 	}
 }
Signature · 1 line
2.20.1
#44 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

kernel test robot noticed the following build errors:

Hi Qi,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20260114]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260114-193921
base:   next-20260114
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: powerpc64-randconfig-002-20260115 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

>> include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'
     772 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:773:13: error: use of undeclared identifier 'rcu'
     773 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:775:14: error: use of undeclared identifier 'lruvec'
     775 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:776:13: error: use of undeclared identifier 'rcu'
     776 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:779:14: error: use of undeclared identifier 'lruvec'
     779 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:780:13: error: use of undeclared identifier 'rcu'
     780 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
    1532 |         __releases(rcu)
         |                    ^~~
   In file included from crypto/ahash.c:26:
   In file included from include/net/netlink.h:6:
   In file included from include/linux/netlink.h:9:
   In file included from include/net/scm.h:13:
   In file included from include/net/compat.h:8:
   include/linux/compat.h:454:22: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
     454 |         case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
         |                             ^        ~
   arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
      18 |         unsigned long sig[_NSIG_WORDS];
         |         ^
   In file included from crypto/ahash.c:26:
   In file included from include/net/netlink.h:6:
   In file included from include/linux/netlink.h:9:
   In file included from include/net/scm.h:13:
   In file included from include/net/compat.h:8:
   include/linux/compat.h:454:10: warning: array index 7 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
     454 |         case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
         |                 ^     ~
   include/linux/compat.h:130:2: note: array 'sig' declared here
     130 |         compat_sigset_word      sig[_COMPAT_NSIG_WORDS];
         |         ^
   include/linux/compat.h:454:42: warning: array index 6 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
     454 |         case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
         |                                                 ^     ~
   include/linux/compat.h:130:2: note: array 'sig' declared here
     130 |         compat_sigset_word      sig[_COMPAT_NSIG_WORDS];
         |         ^
   include/linux/compat.h:454:53: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
     454 |         case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
         |                                                            ^        ~
   arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
      18 |         unsigned long sig[_NSIG_WORDS];
         |         ^
   In file included from crypto/ahash.c:26:
   In file included from include/net/netlink.h:6:
   In file included from include/linux/netlink.h:9:
   In file included from include/net/scm.h:13:
   In file included from include/net/compat.h:8:
   include/linux/compat.h:456:22: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
     456 |         case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
         |                             ^        ~
   arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
      18 |         unsigned long sig[_NSIG_WORDS];
         |         ^
   In file included from crypto/ahash.c:26:
   In file included from include/net/netlink.h:6:
   In file included from include/linux/netlink.h:9:
   In file included from include/net/scm.h:13:
   In file included from include/net/compat.h:8:
   include/linux/compat.h:456:10: warning: array index 5 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
     456 |         case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
         |                 ^     ~
   include/linux/compat.h:130:2: note: array 'sig' declared here
     130 |         compat_sigset_word      sig[_COMPAT_NSIG_WORDS];
         |         ^
   include/linux/compat.h:456:42: warning: array index 4 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
     456 |         case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
         |                                                 ^     ~
   include/linux/compat.h:130:2: note: array 'sig' declared here
     130 |         compat_sigset_word      sig[_COMPAT_NSIG_WORDS];
         |         ^
   include/linux/compat.h:456:53: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
     456 |         case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
         |                                                            ^        ~
   arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
      18 |         unsigned long sig[_NSIG_WORDS];
         |         ^

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:

vim +/lruvec +772 include/linux/memcontrol.h

770
771 struct lruvec *folio_lruvec_lock(struct folio *folio);

 > 772		__acquires(&lruvec->lru_lock)

773 __acquires(rcu)
774 struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
775 __acquires(&lruvec->lru_lock)
776 __acquires(rcu)
777 struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
778 unsigned long *flags);
779 __acquires(&lruvec->lru_lock)
780 __acquires(rcu)
781

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#45 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Hi,

The compilation problem I see in the report occurs under the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage

Could you please test whether the following fixes can resolve the compilation issue?

Thanks!

```
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e0b84b109b7ac..f28c5ef9bf604 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -742,15 +742,9 @@ struct mem_cgroup *get_mem_cgroup_from_current(void);
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);

 struct lruvec *folio_lruvec_lock(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
                                                unsigned long *flags);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)

 static inline
 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
@@ -1477,38 +1471,10 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
        return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }

-static inline void lruvec_lock_irq(struct lruvec *lruvec)
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
-{
-       rcu_read_lock();
-       spin_lock_irq(&lruvec->lru_lock);
-}
-
-static inline void lruvec_unlock(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irq(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irq(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
-               unsigned long flags)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
-       rcu_read_unlock();
-}
+void lruvec_lock_irq(struct lruvec *lruvec);
+void lruvec_unlock(struct lruvec *lruvec);
+void lruvec_unlock_irq(struct lruvec *lruvec);
+void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags);

 /* Test requires a stable folio->memcg binding, see folio_memcg() */
 static inline bool folio_matches_lruvec(struct folio *folio,
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4449d1f371a56..77a6a9fa11981 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -328,9 +328,7 @@ extern unsigned long totalreserve_pages;

 /* linux/mm/swap.c */
 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
-               unsigned int nr_io, unsigned int nr_rotated)
-               __releases(lruvec->lru_lock);
-               __releases(rcu)
+               unsigned int nr_io, unsigned int nr_rotated);
 void lru_note_cost_refault(struct folio *);
 void folio_add_lru(struct folio *);
 void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c303c483f55a4..8907c9ea33bbc 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1313,6 +1313,39 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
        }
 }

+void lruvec_lock_irq(struct lruvec *lruvec)
+       __acquires(&lruvec->lru_lock)
+       __acquires(rcu)
+{
+       rcu_read_lock();
+       spin_lock_irq(&lruvec->lru_lock);
+}
+
+void lruvec_unlock(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irq(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irq(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irqrestore(struct lruvec *lruvec,
+               unsigned long flags)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+       rcu_read_unlock();
+}
+
 /**
  * folio_lruvec_lock - Lock the lruvec for a folio.
  * @folio: Pointer to the folio.

```
Quoted 149 lines · kernel test robot

Hi Qi,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20260114]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260114-193921
base:   next-20260114
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: powerpc64-randconfig-002-20260115 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'

772 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:773:13: error: use of undeclared identifier 'rcu'
773 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:775:14: error: use of undeclared identifier 'lruvec'
775 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:776:13: error: use of undeclared identifier 'rcu'
776 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:779:14: error: use of undeclared identifier 'lruvec'
779 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:780:13: error: use of undeclared identifier 'rcu'
780 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
1507 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
1515 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
1523 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
1532 | __releases(rcu)
| ^~~
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:22: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:10: warning: array index 7 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:42: warning: array index 6 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:53: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:22: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:10: warning: array index 5 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:42: warning: array index 4 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:53: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:

vim +/lruvec +772 include/linux/memcontrol.h

    770	
    771	struct lruvec *folio_lruvec_lock(struct folio *folio);
Quoted 1 line

772 __acquires(&lruvec->lru_lock)

    773		__acquires(rcu)
    774	struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
    775		__acquires(&lruvec->lru_lock)
    776		__acquires(rcu)
    777	struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
    778							unsigned long *flags);
    779		__acquires(&lruvec->lru_lock)
    780		__acquires(rcu)
    781	
#46 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 10 lines · Qi Zheng

Hi,

The compilation problem I see in the report occurs under the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage

Could you please test whether the following fixes can resolve the compilation issue?

Hi Zheng, is it ok to give a try to use the reproduce steps in the original report [1] to verify the fix? As for now, we haven't support the auto verification for the on demand request yet.

[1] https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce

Quoted 281 lines

Thanks!

```

include/linux/memcontrol.h +4 -38
index e0b84b109b7ac..f28c5ef9bf604 100644
@@ -742,15 +742,9 @@ struct mem_cgroup *get_mem_cgroup_from_current(void);
 struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
 
 struct lruvec *folio_lruvec_lock(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
                                                unsigned long *flags);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 
 static inline
 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
@@ -1477,38 +1471,10 @@ static inline struct lruvec *parent_lruvec(struct
 lruvec *lruvec)
        return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
 }
 
-static inline void lruvec_lock_irq(struct lruvec *lruvec)
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
-{
-       rcu_read_lock();
-       spin_lock_irq(&lruvec->lru_lock);
-}
-
-static inline void lruvec_unlock(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irq(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irq(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
-               unsigned long flags)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
-       rcu_read_unlock();
-}
+void lruvec_lock_irq(struct lruvec *lruvec);
+void lruvec_unlock(struct lruvec *lruvec);
+void lruvec_unlock_irq(struct lruvec *lruvec);
+void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags);
 
 /* Test requires a stable folio->memcg binding, see folio_memcg() */

static inline bool folio_matches_lruvec(struct folio *folio,

include/linux/swap.h +1 -3
index 4449d1f371a56..77a6a9fa11981 100644
@@ -328,9 +328,7 @@ extern unsigned long totalreserve_pages;
 
 /* linux/mm/swap.c */
 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
-               unsigned int nr_io, unsigned int nr_rotated)
-               __releases(lruvec->lru_lock);
-               __releases(rcu)
+               unsigned int nr_io, unsigned int nr_rotated);
 void lru_note_cost_refault(struct folio *);
 void folio_add_lru(struct folio *);
 void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
mm/memcontrol.c +33 -0
index c303c483f55a4..8907c9ea33bbc 100644
@@ -1313,6 +1313,39 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
        }
 }
 
+void lruvec_lock_irq(struct lruvec *lruvec)
+       __acquires(&lruvec->lru_lock)
+       __acquires(rcu)
+{
+       rcu_read_lock();
+       spin_lock_irq(&lruvec->lru_lock);
+}
+
+void lruvec_unlock(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irq(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irq(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irqrestore(struct lruvec *lruvec,
+               unsigned long flags)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+       rcu_read_unlock();
+}
+
 /**
  * folio_lruvec_lock - Lock the lruvec for a folio.
  * @folio: Pointer to the folio.

```

Quoted 149 lines · kernel test robot

Hi Qi,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20260114]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260114-193921
base:   next-20260114
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: powerpc64-randconfig-002-20260115 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'

772 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:773:13: error: use of undeclared identifier 'rcu'
773 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:775:14: error: use of undeclared identifier 'lruvec'
775 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:776:13: error: use of undeclared identifier 'rcu'
776 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:779:14: error: use of undeclared identifier 'lruvec'
779 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:780:13: error: use of undeclared identifier 'rcu'
780 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
1507 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
1515 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
1523 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
1532 | __releases(rcu)
| ^~~
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:22: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:10: warning: array index 7 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:42: warning: array index 6 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:53: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:22: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:10: warning: array index 5 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:42: warning: array index 4 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:53: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:

vim +/lruvec +772 include/linux/memcontrol.h

    770	
    771	struct lruvec *folio_lruvec_lock(struct folio *folio);
Quoted 1 line

772 __acquires(&lruvec->lru_lock)

    773		__acquires(rcu)
    774	struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
    775		__acquires(&lruvec->lru_lock)
    776		__acquires(rcu)
    777	struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
    778							unsigned long *flags);
    779		__acquires(&lruvec->lru_lock)
    780		__acquires(rcu)
    781	
author reviewer
#47 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 9 lines · Philip Li
Quoted 8 lines · Qi Zheng

Hi,

The compilation problem I see in the report occurs under the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage

Quoted 6 lines
Quoted 2 lines

Could you please test whether the following fixes can resolve the compilation issue?

Hi Zheng, is it ok to give a try to use the reproduce steps in the original report [1] to verify the fix? As for now, we haven't support the auto verification for the on demand request yet.

OK, I found a Debian 12 machine and reproduced this compilation error with the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage (clang version == 22)

Thank you for reporting this issue!

Quoted 284 lines

[1] https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce

Quoted 281 lines

Thanks!

```

include/linux/memcontrol.h +4 -38
index e0b84b109b7ac..f28c5ef9bf604 100644
@@ -742,15 +742,9 @@ struct mem_cgroup *get_mem_cgroup_from_current(void);
  struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
 
  struct lruvec *folio_lruvec_lock(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
  struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
  struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
                                                 unsigned long *flags);
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
 
  static inline
  struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
@@ -1477,38 +1471,10 @@ static inline struct lruvec *parent_lruvec(struct
 lruvec *lruvec)
         return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));
  }
 
-static inline void lruvec_lock_irq(struct lruvec *lruvec)
-       __acquires(&lruvec->lru_lock)
-       __acquires(rcu)
-{
-       rcu_read_lock();
-       spin_lock_irq(&lruvec->lru_lock);
-}
-
-static inline void lruvec_unlock(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irq(struct lruvec *lruvec)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irq(&lruvec->lru_lock);
-       rcu_read_unlock();
-}
-
-static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
-               unsigned long flags)
-       __releases(&lruvec->lru_lock)
-       __releases(rcu)
-{
-       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
-       rcu_read_unlock();
-}
+void lruvec_lock_irq(struct lruvec *lruvec);
+void lruvec_unlock(struct lruvec *lruvec);
+void lruvec_unlock_irq(struct lruvec *lruvec);
+void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags);
 
  /* Test requires a stable folio->memcg binding, see folio_memcg() */

static inline bool folio_matches_lruvec(struct folio *folio,

include/linux/swap.h +1 -3
index 4449d1f371a56..77a6a9fa11981 100644
@@ -328,9 +328,7 @@ extern unsigned long totalreserve_pages;
 
  /* linux/mm/swap.c */
  void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
-               unsigned int nr_io, unsigned int nr_rotated)
-               __releases(lruvec->lru_lock);
-               __releases(rcu)
+               unsigned int nr_io, unsigned int nr_rotated);
  void lru_note_cost_refault(struct folio *);
  void folio_add_lru(struct folio *);
  void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
mm/memcontrol.c +33 -0
index c303c483f55a4..8907c9ea33bbc 100644
@@ -1313,6 +1313,39 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
         }
  }
 
+void lruvec_lock_irq(struct lruvec *lruvec)
+       __acquires(&lruvec->lru_lock)
+       __acquires(rcu)
+{
+       rcu_read_lock();
+       spin_lock_irq(&lruvec->lru_lock);
+}
+
+void lruvec_unlock(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irq(struct lruvec *lruvec)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irq(&lruvec->lru_lock);
+       rcu_read_unlock();
+}
+
+void lruvec_unlock_irqrestore(struct lruvec *lruvec,
+               unsigned long flags)
+       __releases(&lruvec->lru_lock)
+       __releases(rcu)
+{
+       spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+       rcu_read_unlock();
+}
+
  /**
   * folio_lruvec_lock - Lock the lruvec for a folio.
   * @folio: Pointer to the folio.

```

Quoted 149 lines · kernel test robot

Hi Qi,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20260114]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260114-193921
base:   next-20260114
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: powerpc64-randconfig-002-20260115 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'

772 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:773:13: error: use of undeclared identifier 'rcu'
773 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:775:14: error: use of undeclared identifier 'lruvec'
775 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:776:13: error: use of undeclared identifier 'rcu'
776 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:779:14: error: use of undeclared identifier 'lruvec'
779 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:780:13: error: use of undeclared identifier 'rcu'
780 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
1507 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
1515 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
1523 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
1532 | __releases(rcu)
| ^~~
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:22: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:454:10: warning: array index 7 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:42: warning: array index 6 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:454:53: warning: array index 3 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
454 | case 4: v.sig[7] = (set->sig[3] >> 32); v.sig[6] = set->sig[3];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:22: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:13:
In file included from include/net/compat.h:8:
include/linux/compat.h:456:10: warning: array index 5 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:42: warning: array index 4 is past the end of the array (that has type 'compat_sigset_word[2]' (aka 'unsigned int[2]')) [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
include/linux/compat.h:130:2: note: array 'sig' declared here
130 | compat_sigset_word sig[_COMPAT_NSIG_WORDS];
| ^
include/linux/compat.h:456:53: warning: array index 2 is past the end of the array (that has type 'const unsigned long[1]') [-Warray-bounds]
456 | case 3: v.sig[5] = (set->sig[2] >> 32); v.sig[4] = set->sig[2];
| ^ ~
arch/powerpc/include/uapi/asm/signal.h:18:2: note: array 'sig' declared here
18 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:

vim +/lruvec +772 include/linux/memcontrol.h

     770	
     771	struct lruvec *folio_lruvec_lock(struct folio *folio);
Quoted 1 line

772 __acquires(&lruvec->lru_lock)

     773		__acquires(rcu)
     774	struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
     775		__acquires(&lruvec->lru_lock)
     776		__acquires(rcu)
     777	struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
     778							unsigned long *flags);
     779		__acquires(&lruvec->lru_lock)
     780		__acquires(rcu)
     781	
#48 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng
kernel test robot noticed the following build errors:

Hi Qi,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20260114]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260114-193921
base:   next-20260114
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: mips-randconfig-r071-20260115 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
smatch version: v0.5.0-8985-g2614ff1a
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from kernel/sched/core.c:16:
In file included from include/linux/syscalls_api.h:1:
In file included from include/linux/syscalls.h:96:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:53:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

>> include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
    1532 |         __releases(rcu)
         |                    ^~~
   In file included from kernel/sched/core.c:83:
   In file included from arch/mips/include/asm/tlb.h:24:
   In file included from include/asm-generic/tlb.h:15:
>> include/linux/swap.h:333:14: error: use of undeclared identifier 'rcu'
     333 |                 __releases(rcu)
         |                            ^~~
   kernel/sched/core.c:7764:12: warning: array index -1 is before the beginning of the array [-Warray-bounds]
    7764 |                                        preempt_modes[preempt_dynamic_mode] : "undef",
         |                                        ^             ~~~~~~~~~~~~~~~~~~~~
   kernel/sched/core.c:7739:1: note: array 'preempt_modes' declared here
    7739 | const char *preempt_modes[] = {
         | ^

1 warning and 5 errors generated.
--
In file included from kernel/sched/fair.c:57:
In file included from kernel/sched/sched.h:47:

>> include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
    1532 |         __releases(rcu)
         |                    ^~~

4 errors generated.

vim +/rcu +1507 include/linux/memcontrol.h

1504
1505 static inline void lruvec_lock_irq(struct lruvec *lruvec)
1506 __acquires(&lruvec->lru_lock)

> 1507		__acquires(rcu)

1508 {
1509 rcu_read_lock();
1510 spin_lock_irq(&lruvec->lru_lock);
1511 }
1512

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#49 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng
kernel test robot noticed the following build warnings:

Hi Qi,

kernel test robot noticed the following build warnings:

[auto build test WARNING on next-20260115]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all trace/for-next linus/master dennis-percpu/for-next v6.19-rc5 v6.19-rc4 v6.19-rc3 v6.19-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Qi-Zheng/mm-memcontrol-remove-dead-code-of-checking-parent-memory-cgroup/20260115-224607
base:   next-20260115
patch link:    https://lore.kernel.org/r/0252f9acc29d4b1e9b8252dc003aff065c8ac1f6.1768389889.git.zhengqi.arch%40bytedance.com
patch subject: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
config: sparc64-allmodconfig (https://download.01.org/0day-ci/archive/20260116/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260116/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from kernel/sched/core.c:16:
In file included from include/linux/syscalls_api.h:1:
In file included from include/linux/syscalls.h:95:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:53:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:
include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'

     772 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:773:13: error: use of undeclared identifier 'rcu'
     773 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:775:14: error: use of undeclared identifier 'lruvec'
     775 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:776:13: error: use of undeclared identifier 'rcu'
     776 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:779:14: error: use of undeclared identifier 'lruvec'
     779 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:780:13: error: use of undeclared identifier 'rcu'
     780 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1507:13: error: use of undeclared identifier 'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared identifier 'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared identifier 'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared identifier 'rcu'
    1532 |         __releases(rcu)
         |                    ^~~
   In file included from kernel/sched/core.c:83:
   In file included from arch/sparc/include/asm/tlb.h:5:
   In file included from arch/sparc/include/asm/tlb_64.h:5:
   include/linux/swap.h:333:14: error: use of undeclared identifier 'rcu'
     333 |                 __releases(rcu)
         |                            ^~~
>> kernel/sched/core.c:7772:12: warning: array index -1 is before the beginning of the array [-Warray-bounds]
    7772 |                                        preempt_modes[preempt_dynamic_mode] : "undef",
         |                                        ^             ~~~~~~~~~~~~~~~~~~~~
   kernel/sched/core.c:7747:1: note: array 'preempt_modes' declared here
    7747 | const char *preempt_modes[] = {
         | ^

1 warning and 11 errors generated.

vim +7772 kernel/sched/core.c

8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7750  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7751  const char *preempt_model_str(void)
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7752  {
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7753  	bool brace = IS_ENABLED(CONFIG_PREEMPT_RT) &&
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7754  		(IS_ENABLED(CONFIG_PREEMPT_DYNAMIC) ||
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7755  		 IS_ENABLED(CONFIG_PREEMPT_LAZY));
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7756  	static char buf[128];
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7757  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7758  	if (IS_ENABLED(CONFIG_PREEMPT_BUILD)) {
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7759  		struct seq_buf s;
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7760  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7761  		seq_buf_init(&s, buf, sizeof(buf));
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7762  		seq_buf_puts(&s, "PREEMPT");
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7763  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7764  		if (IS_ENABLED(CONFIG_PREEMPT_RT))
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7765  			seq_buf_printf(&s, "%sRT%s",
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7766  				       brace ? "_{" : "_",
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7767  				       brace ? "," : "");
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7768  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7769  		if (IS_ENABLED(CONFIG_PREEMPT_DYNAMIC)) {
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7770  			seq_buf_printf(&s, "(%s)%s",
3ebb1b6522392f6 Thomas Weißschuh          2025-06-26  7771  				       preempt_dynamic_mode >= 0 ?
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14 @7772  				       preempt_modes[preempt_dynamic_mode] : "undef",
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7773  				       brace ? "}" : "");
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7774  			return seq_buf_str(&s);
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7775  		}
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7776  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7777  		if (IS_ENABLED(CONFIG_PREEMPT_LAZY)) {
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7778  			seq_buf_printf(&s, "LAZY%s",
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7779  				       brace ? "}" : "");
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7780  			return seq_buf_str(&s);
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7781  		}
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7782  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7783  		return seq_buf_str(&s);
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7784  	}
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7785  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7786  	if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY_BUILD))
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7787  		return "VOLUNTARY";
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7788  
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7789  	return "NONE";
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7790  }
8bdc5daaa01e305 Sebastian Andrzej Siewior 2025-03-14  7791  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reviewer
#50 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng
Quoted 141 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
     rcu_read_lock();
retry:
     lruvec = folio_lruvec(folio);
     /* There is a possibility of folio reparenting at this point. */
     spin_lock(&lruvec->lru_lock);
     if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
         /*
          * The wrong lruvec lock was acquired, and a retry is required.
          * This is because the folio resides on the parent memcg lruvec
          * list.
          */
         spin_unlock(&lruvec->lru_lock);
         goto retry;
     }

     /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
     spin_lock(&lruvec->lru_lock);
     spin_lock(&lruvec_parent->lru_lock);
     /* Transfer folios from the lruvec list to the parent's. */
     spin_unlock(&lruvec_parent->lru_lock);
     spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
  include/linux/memcontrol.h | 45 +++++++++++++++++++----------
  include/linux/swap.h       |  1 +
  mm/compaction.c            | 29 +++++++++++++++----
  mm/memcontrol.c            | 59 +++++++++++++++++++++-----------------
  mm/swap.c                  |  4 +++
  5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +18 -15
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
   * folio_lruvec - return lruvec for isolating/putting an LRU folio
   * @folio: Pointer to the folio.
   *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
   */
  static inline struct lruvec *folio_lruvec(struct folio *folio)
  {
@@ -761,18 +769,15 @@ struct mem_cgroup *get_mem_cgroup_from_current(void);
  struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
 
  struct lruvec *folio_lruvec_lock(struct folio *folio);
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
  struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
  struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
  						unsigned long *flags);
-
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio);
-#else
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-#endif
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
 
  static inline
  struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){
@@ -1199,11 +1204,6 @@ static inline struct lruvec *folio_lruvec(struct folio *folio)
  	return &pgdat->__lruvec;
  }
 
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-
  static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
  {
  	return NULL;
@@ -1262,6 +1262,7 @@ static inline struct lruvec *folio_lruvec_lock(struct folio *folio)
  {
  	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
  	spin_lock(&pgdat->__lruvec.lru_lock);
  	return &pgdat->__lruvec;
  }
@@ -1270,6 +1271,7 @@ static inline struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
  {
  	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
  	spin_lock_irq(&pgdat->__lruvec.lru_lock);
  	return &pgdat->__lruvec;
  }
@@ -1279,6 +1281,7 @@ static inline struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
  {
  	struct pglist_data *pgdat = folio_pgdat(folio);
 
+	rcu_read_lock();
  	spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
  	return &pgdat->__lruvec;
  }

include/linux/memcontrol.h static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) · line 1507

Quoted 5 lines
include/linux/memcontrol.h +2 -0
@@ -1500,24 +1503,36 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)
  }
 
  static inline void lruvec_lock_irq(struct lruvec *lruvec)
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)

It seems that functions marked as inline cannot be decorated with __acquires? We’ve had to move these little helpers into memcontrol.c and declare them as extern, but they’re so short that it hardly feels worth the trouble. My own inclination is to drop the __acquires annotations—mainly for performance reasons.

Quoted 36 lines
include/linux/memcontrol.h +10 -0
  {
+	rcu_read_lock();
  	spin_lock_irq(&lruvec->lru_lock);
  }
 
  static inline void lruvec_unlock(struct lruvec *lruvec)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
  {
  	spin_unlock(&lruvec->lru_lock);
+	rcu_read_unlock();
  }
 
  static inline void lruvec_unlock_irq(struct lruvec *lruvec)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
  {
  	spin_unlock_irq(&lruvec->lru_lock);
+	rcu_read_unlock();
  }
 
  static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec,
  		unsigned long flags)
+	__releases(&lruvec->lru_lock)
+	__releases(rcu)
  {
  	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+	rcu_read_unlock();
  }
 
  /* Test requires a stable folio->memcg binding, see folio_memcg() */
include/linux/swap.h +0 -0
index 62fc7499b4089..e60f45b48e74d 100644

include/linux/swap.h extern unsigned long totalreserve_pages; · line 333

Quoted 4 lines
include/linux/swap.h +1 -0
@@ -330,6 +330,7 @@ extern unsigned long totalreserve_pages;
  void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  		unsigned int nr_io, unsigned int nr_rotated)
  		__releases(lruvec->lru_lock);
+		__releases(rcu)

Missed a semicolon.

Quoted 184 lines
include/linux/swap.h +0 -0
  void lru_note_cost_refault(struct folio *);
  void folio_add_lru(struct folio *);
  void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
mm/compaction.c +23 -6
index c3e338aaa0ffb..3648ce22c8072 100644
@@ -518,6 +518,24 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,
  	return true;
  }
 
+static struct lruvec *
+compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags,
+				  struct compact_control *cc)
+{
+	struct lruvec *lruvec;
+
+	rcu_read_lock();
+retry:
+	lruvec = folio_lruvec(folio);
+	compact_lock_irqsave(&lruvec->lru_lock, flags, cc);
+	if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+		spin_unlock_irqrestore(&lruvec->lru_lock, *flags);
+		goto retry;
+	}
+
+	return lruvec;
+}
+
  /*
   * Compaction requires the taking of some coarse locks that are potentially
   * very heavily contended. The lock should be periodically unlocked to avoid
@@ -839,7 +857,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
  {
  	pg_data_t *pgdat = cc->zone->zone_pgdat;
  	unsigned long nr_scanned = 0, nr_isolated = 0;
-	struct lruvec *lruvec;
+	struct lruvec *lruvec = NULL;
  	unsigned long flags = 0;
  	struct lruvec *locked = NULL;
  	struct folio *folio = NULL;
@@ -1153,18 +1171,17 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
  		if (!folio_test_clear_lru(folio))
  			goto isolate_fail_put;
 
-		lruvec = folio_lruvec(folio);
+		if (locked)
+			lruvec = folio_lruvec(folio);
 
  		/* If we already hold the lock, we can skip some rechecking */
-		if (lruvec != locked) {
+		if (lruvec != locked || !locked) {
  			if (locked)
  				lruvec_unlock_irqrestore(locked, flags);
 
-			compact_lock_irqsave(&lruvec->lru_lock, &flags, cc);
+			lruvec = compact_folio_lruvec_lock_irqsave(folio, &flags, cc);
  			locked = lruvec;
 
-			lruvec_memcg_debug(lruvec, folio);
-
  			/*
  			 * Try get exclusive access under lock. If marked for
  			 * skip, the scan is aborted unless the current context
mm/memcontrol.c +33 -26
index 548e67dbf2386..a1573600d4188 100644
@@ -1201,23 +1201,6 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
  	}
  }
 
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-	struct mem_cgroup *memcg;
-
-	if (mem_cgroup_disabled())
-		return;
-
-	memcg = folio_memcg(folio);
-
-	if (!memcg)
-		VM_BUG_ON_FOLIO(!mem_cgroup_is_root(lruvec_memcg(lruvec)), folio);
-	else
-		VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != memcg, folio);
-}
-#endif
-
  /**
   * folio_lruvec_lock - Lock the lruvec for a folio.
   * @folio: Pointer to the folio.
@@ -1227,14 +1210,22 @@ void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
   * - folio_test_lru false
   * - folio frozen (refcount of 0)
   *
- * Return: The lruvec this folio is on with its lock held.
+ * Return: The lruvec this folio is on with its lock held and rcu read lock held.
   */
  struct lruvec *folio_lruvec_lock(struct folio *folio)
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
  {
-	struct lruvec *lruvec = folio_lruvec(folio);
+	struct lruvec *lruvec;
 
+	rcu_read_lock();
+retry:
+	lruvec = folio_lruvec(folio);
  	spin_lock(&lruvec->lru_lock);
-	lruvec_memcg_debug(lruvec, folio);
+	if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+		spin_unlock(&lruvec->lru_lock);
+		goto retry;
+	}
 
  	return lruvec;
  }
@@ -1249,14 +1240,22 @@ struct lruvec *folio_lruvec_lock(struct folio *folio)
   * - folio frozen (refcount of 0)
   *
   * Return: The lruvec this folio is on with its lock held and interrupts
- * disabled.
+ * disabled and rcu read lock held.
   */
  struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
  {
-	struct lruvec *lruvec = folio_lruvec(folio);
+	struct lruvec *lruvec;
 
+	rcu_read_lock();
+retry:
+	lruvec = folio_lruvec(folio);
  	spin_lock_irq(&lruvec->lru_lock);
-	lruvec_memcg_debug(lruvec, folio);
+	if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+		spin_unlock_irq(&lruvec->lru_lock);
+		goto retry;
+	}
 
  	return lruvec;
  }
@@ -1272,15 +1271,23 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
   * - folio frozen (refcount of 0)
   *
   * Return: The lruvec this folio is on with its lock held and interrupts
- * disabled.
+ * disabled and rcu read lock held.
   */
  struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
  		unsigned long *flags)
+	__acquires(&lruvec->lru_lock)
+	__acquires(rcu)
  {
-	struct lruvec *lruvec = folio_lruvec(folio);
+	struct lruvec *lruvec;
 
+	rcu_read_lock();
+retry:
+	lruvec = folio_lruvec(folio);
  	spin_lock_irqsave(&lruvec->lru_lock, *flags);
-	lruvec_memcg_debug(lruvec, folio);
+	if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+		spin_unlock_irqrestore(&lruvec->lru_lock, *flags);
+		goto retry;
+	}
 
  	return lruvec;
  }
mm/swap.c +1 -0
index cb1148a92d8ec..7e53479ca1732 100644
@@ -240,6 +240,7 @@ void folio_rotate_reclaimable(struct folio *folio)
  void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  		unsigned int nr_io, unsigned int nr_rotated)
  		__releases(lruvec->lru_lock)
+		__releases(rcu)
  {
  	unsigned long cost;
 

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 257

Quoted 4 lines
mm/swap.c +1 -0
@@ -253,6 +254,7 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  	cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
  	if (!cost) {
  		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();

Better to use lruvec_unlock_irq(lruvec)?

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 289

Quoted 8 lines
mm/swap.c +1 -0
  		return;
  	}
 
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  		}
 
  		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();

Ditto.

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 294

Quoted 5 lines
mm/swap.c +1 -0
  		lruvec = parent_lruvec(lruvec);
  		if (!lruvec)
  			break;
+		rcu_read_lock();
  		spin_lock_irq(&lruvec->lru_lock);

lruvec_lock_irq(lruvec)?

Thanks.

Quoted 2 lines
mm/swap.c +0 -0
  	}
  }
author reviewer
#51 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 151 lines · Muchun Song
Quoted 148 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
     rcu_read_lock();
retry:
     lruvec = folio_lruvec(folio);
     /* There is a possibility of folio reparenting at this point. */
     spin_lock(&lruvec->lru_lock);
     if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
         /*
          * The wrong lruvec lock was acquired, and a retry is required.
          * This is because the folio resides on the parent memcg lruvec
          * list.
          */
         spin_unlock(&lruvec->lru_lock);
         goto retry;
     }

     /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
     spin_lock(&lruvec->lru_lock);
     spin_lock(&lruvec_parent->lru_lock);
     /* Transfer folios from the lruvec list to the parent's. */
     spin_unlock(&lruvec_parent->lru_lock);
     spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
  include/linux/memcontrol.h | 45 +++++++++++++++++++---------- include/linux/swap.h       |  1 +
  mm/compaction.c            | 29 +++++++++++++++---- mm/memcontrol.c            | 59 +++++++++++++++++++++----------------- mm/swap.c                  |  4 +++

  5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +1 -1
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec 
 *mem_cgroup_lruvec(struct mem_cgroup *memcg,
    * folio_lruvec - return lruvec for isolating/putting an LRU folio
    * @folio: Pointer to the folio.
    *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the 

returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
   */
  static inline struct lruvec *folio_lruvec(struct folio *folio) {

include/linux/memcontrol.h +6 -9
@@ -761,18 +769,15 @@ struct mem_cgroup 
 *get_mem_cgroup_from_current(void);
   struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
   struct lruvec *folio_lruvec_lock(struct folio *folio);
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
                           unsigned long *flags);
-
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio);
-#else
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-#endif
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)

  static inline
  struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){

include/linux/memcontrol.h +3 -5
@@ -1199,11 +1204,6 @@ static inline struct lruvec 
 *folio_lruvec(struct folio *folio)
       return &pgdat->__lruvec;
   }
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-
   static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
   {
       return NULL;
@@ -1262,6 +1262,7 @@ static inline struct lruvec 
 *folio_lruvec_lock(struct folio *folio)
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock(&pgdat->__lruvec.lru_lock);
       return &pgdat->__lruvec;
   }
@@ -1270,6 +1271,7 @@ static inline struct lruvec 
 *folio_lruvec_lock_irq(struct folio *folio)
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock_irq(&pgdat->__lruvec.lru_lock);
       return &pgdat->__lruvec;
   }
@@ -1279,6 +1281,7 @@ static inline struct lruvec 
 *folio_lruvec_lock_irqsave(struct folio *folio,
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
       return &pgdat->__lruvec;
   }
@@ -1500,24 +1503,36 @@ static inline struct lruvec 
 *parent_lruvec(struct lruvec *lruvec)
Quoted 8 lines

include/linux/memcontrol.h static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) · line 1507

Quoted 4 lines
include/linux/memcontrol.h +2 -0
   }
   static inline void lruvec_lock_irq(struct lruvec *lruvec)
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)

It seems that functions marked as inline cannot be decorated with __acquires? We’ve had to move these little helpers into memcontrol.c and declare them as extern, but they’re so short that it hardly feels

Right, I received a compilation error reported LKP:

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier 'lruvec'

     772 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:773:13: error: use of undeclared 
identifier 'rcu'
     773 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:775:14: error: use of undeclared 
identifier 'lruvec'
     775 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:776:13: error: use of undeclared 
identifier 'rcu'
     776 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:779:14: error: use of undeclared 
identifier 'lruvec'
     779 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:780:13: error: use of undeclared 
identifier 'rcu'
     780 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1507:13: error: use of undeclared 
identifier 'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared 
identifier 'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared 
identifier 'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared 
identifier 'rcu'
    1532 |         __releases(rcu)

And I reproduced this error with the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage (clang version >= 22)

Quoted 2 lines

worth the trouble. My own inclination is to drop the __acquires annotations—mainly for performance reasons.

If no one else objects, I will drop __acquires/__releases in the next version.

Quoted 240 lines
Quoted 33 lines

  {
+    rcu_read_lock();
      spin_lock_irq(&lruvec->lru_lock);
  }
  static inline void lruvec_unlock(struct lruvec *lruvec)
+    __releases(&lruvec->lru_lock)
+    __releases(rcu)
  {
      spin_unlock(&lruvec->lru_lock);
+    rcu_read_unlock();
  }
  static inline void lruvec_unlock_irq(struct lruvec *lruvec) +    __releases(&lruvec->lru_lock)
+    __releases(rcu)
  {
      spin_unlock_irq(&lruvec->lru_lock);
+    rcu_read_unlock();
  }
  static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags)
+    __releases(&lruvec->lru_lock)
+    __releases(rcu)
  {
      spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+    rcu_read_unlock();
  }
  /* Test requires a stable folio->memcg binding, see folio_memcg() */

include/linux/swap.h +0 -0
index 62fc7499b4089..e60f45b48e74d 100644

include/linux/swap.h extern unsigned long totalreserve_pages; · line 333

Quoted 4 lines
include/linux/swap.h +1 -0
@@ -330,6 +330,7 @@ extern unsigned long totalreserve_pages;
   void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
           unsigned int nr_io, unsigned int nr_rotated)
           __releases(lruvec->lru_lock);
+        __releases(rcu)

Missed a semicolon.

Quoted 182 lines
include/linux/swap.h +0 -0
   void lru_note_cost_refault(struct folio *);
   void folio_add_lru(struct folio *);
   void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
mm/compaction.c +18 -0
index c3e338aaa0ffb..3648ce22c8072 100644
@@ -518,6 +518,24 @@ static bool compact_lock_irqsave(spinlock_t 
 *lock, unsigned long *flags,
       return true;
   }
+static struct lruvec *
+compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long 
 *flags,
+                  struct compact_control *cc)
+{
+    struct lruvec *lruvec;
+
+    rcu_read_lock();
+retry:
+    lruvec = folio_lruvec(folio);
+    compact_lock_irqsave(&lruvec->lru_lock, flags, cc);
+    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+        spin_unlock_irqrestore(&lruvec->lru_lock, *flags);
+        goto retry;
+    }
+
+    return lruvec;
+}
+
   /*
    * Compaction requires the taking of some coarse locks that are potentially

   * very heavily contended. The lock should be periodically unlocked to avoid

mm/compaction.c +1 -1
@@ -839,7 +857,7 @@ isolate_migratepages_block(struct compact_control 
 *cc, unsigned long low_pfn,
   {
       pg_data_t *pgdat = cc->zone->zone_pgdat;
       unsigned long nr_scanned = 0, nr_isolated = 0;
-    struct lruvec *lruvec;
+    struct lruvec *lruvec = NULL;
       unsigned long flags = 0;
       struct lruvec *locked = NULL;

      struct folio *folio = NULL;

mm/compaction.c +4 -5
@@ -1153,18 +1171,17 @@ isolate_migratepages_block(struct 
 compact_control *cc, unsigned long low_pfn,
           if (!folio_test_clear_lru(folio))
               goto isolate_fail_put;
-        lruvec = folio_lruvec(folio);
+        if (locked)
+            lruvec = folio_lruvec(folio);
           /* If we already hold the lock, we can skip some rechecking */
-        if (lruvec != locked) {
+        if (lruvec != locked || !locked) {
               if (locked)
                   lruvec_unlock_irqrestore(locked, flags);
-            compact_lock_irqsave(&lruvec->lru_lock, &flags, cc);
+            lruvec = compact_folio_lruvec_lock_irqsave(folio, &flags, 
 cc);
               locked = lruvec;
-            lruvec_memcg_debug(lruvec, folio);
-
               /*
                * Try get exclusive access under lock. If marked for
                * skip, the scan is aborted unless the current context
mm/memcontrol.c +0 -17
index 548e67dbf2386..a1573600d4188 100644
@@ -1201,23 +1201,6 @@ void mem_cgroup_scan_tasks(struct mem_cgroup 
 *memcg,
       }
   }
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-    struct mem_cgroup *memcg;
-
-    if (mem_cgroup_disabled())
-        return;
-
-    memcg = folio_memcg(folio);
-
-    if (!memcg)
-        VM_BUG_ON_FOLIO(!mem_cgroup_is_root(lruvec_memcg(lruvec)), 
 folio);
-    else
-        VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != memcg, folio);
-}
-#endif
-
   /**
    * folio_lruvec_lock - Lock the lruvec for a folio.

   * @folio: Pointer to the folio.

mm/memcontrol.c +1 -1
@@ -1227,14 +1210,22 @@ void lruvec_memcg_debug(struct lruvec *lruvec, 
 struct folio *folio)
    * - folio_test_lru false
    * - folio frozen (refcount of 0)
    *
- * Return: The lruvec this folio is on with its lock held.
+ * Return: The lruvec this folio is on with its lock held and rcu 

read lock held.
   */
  struct lruvec *folio_lruvec_lock(struct folio *folio) +    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
  {
-    struct lruvec *lruvec = folio_lruvec(folio);
+    struct lruvec *lruvec;
+    rcu_read_lock();
+retry:
+    lruvec = folio_lruvec(folio); spin_lock(&lruvec->lru_lock);
-    lruvec_memcg_debug(lruvec, folio);
+    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { +        spin_unlock(&lruvec->lru_lock);
+        goto retry;
+    }
      return lruvec;
  }

mm/memcontrol.c +22 -6
@@ -1249,14 +1240,22 @@ struct lruvec *folio_lruvec_lock(struct folio 
 *folio)
    * - folio frozen (refcount of 0)
    *
    * Return: The lruvec this folio is on with its lock held and interrupts
- * disabled.
+ * disabled and rcu read lock held.
    */
   struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   {
-    struct lruvec *lruvec = folio_lruvec(folio);
+    struct lruvec *lruvec;
+    rcu_read_lock();
+retry:
+    lruvec = folio_lruvec(folio);
       spin_lock_irq(&lruvec->lru_lock);
-    lruvec_memcg_debug(lruvec, folio);
+    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+        spin_unlock_irq(&lruvec->lru_lock);
+        goto retry;
+    }
       return lruvec;
   }
@@ -1272,15 +1271,23 @@ struct lruvec *folio_lruvec_lock_irq(struct 
 folio *folio)
    * - folio frozen (refcount of 0)
    *
    * Return: The lruvec this folio is on with its lock held and interrupts
- * disabled.
+ * disabled and rcu read lock held.
    */
   struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
           unsigned long *flags)
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   {
-    struct lruvec *lruvec = folio_lruvec(folio);
+    struct lruvec *lruvec;
+    rcu_read_lock();
+retry:
+    lruvec = folio_lruvec(folio);
       spin_lock_irqsave(&lruvec->lru_lock, *flags);
-    lruvec_memcg_debug(lruvec, folio);
+    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
+        spin_unlock_irqrestore(&lruvec->lru_lock, *flags);
+        goto retry;
+    }
       return lruvec;
   }
mm/swap.c +1 -0
index cb1148a92d8ec..7e53479ca1732 100644
@@ -240,6 +240,7 @@ void folio_rotate_reclaimable(struct folio *folio)
   void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
           unsigned int nr_io, unsigned int nr_rotated)
           __releases(lruvec->lru_lock)
+        __releases(rcu)
   {
       unsigned long cost;

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec · line 258

Quoted 5 lines
mm/swap.c +1 -0
@@ -253,6 +254,7 @@ void lru_note_cost_unlock_irq(struct lruvec 
 *lruvec, bool file,
       cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
       if (!cost) {
           spin_unlock_irq(&lruvec->lru_lock);
+        rcu_read_unlock();

Better to use lruvec_unlock_irq(lruvec)?

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec · line 289

Quoted 7 lines
mm/swap.c +1 -0
           return;
       }
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec 
 *lruvec, bool file,
           }
           spin_unlock_irq(&lruvec->lru_lock);
+        rcu_read_unlock();

Ditto.

Quoted 7 lines

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 294

Quoted 5 lines
mm/swap.c +1 -0
           lruvec = parent_lruvec(lruvec);
           if (!lruvec)
               break;
+        rcu_read_lock();
           spin_lock_irq(&lruvec->lru_lock);

lruvec_lock_irq(lruvec)?

OK, will do.

Thanks,
Qi

Quoted 6 lines

Thanks.

Quoted 2 lines
mm/swap.c +0 -0
       }
   }
maintainer
#52 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 217 lines · Qi Zheng
Quoted 153 lines · Muchun Song
Quoted 150 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
     rcu_read_lock();
retry:
     lruvec = folio_lruvec(folio);
     /* There is a possibility of folio reparenting at this point. */
     spin_lock(&lruvec->lru_lock);
     if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
         /*
          * The wrong lruvec lock was acquired, and a retry is required.
          * This is because the folio resides on the parent memcg lruvec
          * list.
          */
         spin_unlock(&lruvec->lru_lock);
         goto retry;
     }

     /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
     spin_lock(&lruvec->lru_lock);
     spin_lock(&lruvec_parent->lru_lock);
     /* Transfer folios from the lruvec list to the parent's. */
     spin_unlock(&lruvec_parent->lru_lock);
     spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
  include/linux/memcontrol.h | 45 +++++++++++++++++++---------- include/linux/swap.h       |  1 +
  mm/compaction.c            | 29 +++++++++++++++---- mm/memcontrol.c            | 59 +++++++++++++++++++++----------------- mm/swap.c                  |  4 +++

  5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +1 -1
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec
 *mem_cgroup_lruvec(struct mem_cgroup *memcg,
    * folio_lruvec - return lruvec for isolating/putting an LRU folio
    * @folio: Pointer to the folio.
    *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the

returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
   */
  static inline struct lruvec *folio_lruvec(struct folio *folio) {

include/linux/memcontrol.h +6 -9
@@ -761,18 +769,15 @@ struct mem_cgroup
 *get_mem_cgroup_from_current(void);
   struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio);
   struct lruvec *folio_lruvec_lock(struct folio *folio);
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)
   struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
                           unsigned long *flags);
-
-#ifdef CONFIG_DEBUG_VM
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio);
-#else
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-#endif
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)

  static inline
  struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){

include/linux/memcontrol.h +0 -5
@@ -1199,11 +1204,6 @@ static inline struct lruvec
 *folio_lruvec(struct folio *folio)
       return &pgdat->__lruvec;
   }
-static inline
-void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
-{
-}
-

  static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)
  {
      return NULL;

include/linux/memcontrol.h +3 -0
@@ -1262,6 +1262,7 @@ static inline struct lruvec
 *folio_lruvec_lock(struct folio *folio)
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock(&pgdat->__lruvec.lru_lock);
       return &pgdat->__lruvec;
   }
@@ -1270,6 +1271,7 @@ static inline struct lruvec
 *folio_lruvec_lock_irq(struct folio *folio)
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock_irq(&pgdat->__lruvec.lru_lock);
       return &pgdat->__lruvec;
   }
@@ -1279,6 +1281,7 @@ static inline struct lruvec
 *folio_lruvec_lock_irqsave(struct folio *folio,
   {
       struct pglist_data *pgdat = folio_pgdat(folio);
+    rcu_read_lock();
       spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);
       return &pgdat->__lruvec;
   }
@@ -1500,24 +1503,36 @@ static inline struct lruvec
 *parent_lruvec(struct lruvec *lruvec)
Quoted 8 lines

include/linux/memcontrol.h static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) · line 1507

Quoted 4 lines
include/linux/memcontrol.h +2 -0
   }
   static inline void lruvec_lock_irq(struct lruvec *lruvec)
+    __acquires(&lruvec->lru_lock)
+    __acquires(rcu)

It seems that functions marked as inline cannot be decorated with __acquires? We’ve had to move these little helpers into memcontrol.c and declare them as extern, but they’re so short that it hardly feels

Right, I received a compilation error reported LKP:

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier

'lruvec'

     772 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:773:13: error: use of undeclared identifier
'rcu'
     773 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:775:14: error: use of undeclared identifier
'lruvec'
     775 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:776:13: error: use of undeclared identifier
'rcu'
     776 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:779:14: error: use of undeclared identifier
'lruvec'
     779 |         __acquires(&lruvec->lru_lock)
         |                     ^~~~~~
   include/linux/memcontrol.h:780:13: error: use of undeclared identifier
'rcu'
     780 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1507:13: error: use of undeclared identifier
'rcu'
    1507 |         __acquires(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1515:13: error: use of undeclared identifier
'rcu'
    1515 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1523:13: error: use of undeclared identifier
'rcu'
    1523 |         __releases(rcu)
         |                    ^~~
   include/linux/memcontrol.h:1532:13: error: use of undeclared identifier
'rcu'
    1532 |         __releases(rcu)

And I reproduced this error with the following configuration:

Quoted 9 lines

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage (clang version >= 22)

Quoted 2 lines

worth the trouble. My own inclination is to drop the __acquires annotations—mainly for performance reasons.

If no one else objects, I will drop __acquires/__releases in the next version.

If you drop these annotations from header file and keep in the C file, do you still get the compilation error?

author reviewer
#53 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 9 lines · Shakeel Butt
Quoted 8 lines · Qi Zheng
Quoted 5 lines · Muchun Song
Quoted 2 lines · Qi Zheng

From: Muchun Song <[email protected]>

[...]

Quoted 63 lines
Quoted 63 lines
Quoted 4 lines

It seems that functions marked as inline cannot be decorated with __acquires? We’ve had to move these little helpers into memcontrol.c and declare them as extern, but they’re so short that it hardly feels

Right, I received a compilation error reported LKP:

All errors (new ones prefixed by >>):

In file included from crypto/ahash.c:26:
In file included from include/net/netlink.h:6:
In file included from include/linux/netlink.h:9:
In file included from include/net/scm.h:9:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:

Quoted 1 line
Quoted 1 line

include/linux/memcontrol.h:772:14: error: use of undeclared identifier

'lruvec'
772 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:773:13: error: use of undeclared identifier
'rcu'
773 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:775:14: error: use of undeclared identifier
'lruvec'
775 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:776:13: error: use of undeclared identifier
'rcu'
776 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:779:14: error: use of undeclared identifier
'lruvec'
779 | __acquires(&lruvec->lru_lock)
| ^~~~~~
include/linux/memcontrol.h:780:13: error: use of undeclared identifier
'rcu'
780 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1507:13: error: use of undeclared identifier
'rcu'
1507 | __acquires(rcu)
| ^~~
include/linux/memcontrol.h:1515:13: error: use of undeclared identifier
'rcu'
1515 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1523:13: error: use of undeclared identifier
'rcu'
1523 | __releases(rcu)
| ^~~
include/linux/memcontrol.h:1532:13: error: use of undeclared identifier
'rcu'

     1532 |         __releases(rcu)

And I reproduced this error with the following configuration:

1. enable CONFIG_WARN_CONTEXT_ANALYSIS_ALL
2. make CC=clang bzImage (clang version >= 22)

Quoted 2 lines

worth the trouble. My own inclination is to drop the __acquires annotations—mainly for performance reasons.

Quoted 6 lines
Quoted 3 lines

If no one else objects, I will drop __acquires/__releases in the next version.

If you drop these annotations from header file and keep in the C file, do you still get the compilation error?

I did test it this way, and it does fix the compilation error, but Muchun thinks these functions are very simple and there's no need to put them in a C file.

Quoted 1 line
maintainer
#54 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 2 lines · Qi Zheng

[...]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

If no one else objects, I will drop __acquires/__releases in the next version.

Quoted 7 lines
Quoted 2 lines

If you drop these annotations from header file and keep in the C file, do you still get the compilation error?

I did test it this way, and it does fix the compilation error, but Muchun thinks these functions are very simple and there's no need to put them in a C file.

I think we should prefer correctness at this stage. We can always change these functions later if we see performance impact of these. So, I would recommend to keep them in C file along with these annotations.

maintainer Acked-by on v3 24/30
#55 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng
Quoted 45 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
    rcu_read_lock();
retry:
    lruvec = folio_lruvec(folio);
    /* There is a possibility of folio reparenting at this point. */
    spin_lock(&lruvec->lru_lock);
    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
        /*
         * The wrong lruvec lock was acquired, and a retry is required.
         * This is because the folio resides on the parent memcg lruvec
         * list.
         */
        spin_unlock(&lruvec->lru_lock);
        goto retry;
    }

    /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
    spin_lock(&lruvec->lru_lock);
    spin_lock(&lruvec_parent->lru_lock);
    /* Transfer folios from the lruvec list to the parent's. */
    spin_unlock(&lruvec_parent->lru_lock);
    spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

Quoted 5 lines

This patch serves as a preparation for the reparenting of LRU folios.

#56 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

In reply to Qi Zheng
Quoted 88 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
    rcu_read_lock();
retry:
    lruvec = folio_lruvec(folio);
    /* There is a possibility of folio reparenting at this point. */
    spin_lock(&lruvec->lru_lock);
    if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
        /*
         * The wrong lruvec lock was acquired, and a retry is required.
         * This is because the folio resides on the parent memcg lruvec
         * list.
         */
        spin_unlock(&lruvec->lru_lock);
        goto retry;
    }

    /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
    spin_lock(&lruvec->lru_lock);
    spin_lock(&lruvec_parent->lru_lock);
    /* Transfer folios from the lruvec list to the parent's. */
    spin_unlock(&lruvec_parent->lru_lock);
    spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
 include/linux/memcontrol.h | 45 +++++++++++++++++++----------
 include/linux/swap.h       |  1 +
 mm/compaction.c            | 29 +++++++++++++++----
 mm/memcontrol.c            | 59 +++++++++++++++++++++-----------------
 mm/swap.c                  |  4 +++
 5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +9 -1
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
  * folio_lruvec - return lruvec for isolating/putting an LRU folio
  * @folio: Pointer to the folio.
  *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
  */
 static inline struct lruvec *folio_lruvec(struct folio *folio)
 {
mm/memcontrol.c +0 -0
index 548e67dbf2386..a1573600d4188 100644
mm/swap.c +0 -0
index cb1148a92d8ec..7e53479ca1732 100644

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 290

Quoted 5 lines
mm/swap.c +1 -0
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
 		}
 
 		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();
 		lruvec = parent_lruvec(lruvec);

It looks bit weird to call parent_lruvec(lruvec) outside RCU read lock because the reason why it holds RCU read lock is to prevent release of memory cgroup and its lruvec.

I guess this isn't broken (for now) because all callers of lru_note_cost_unlock_irq() are holding a reference to the memcg?

Quoted 6 lines
mm/swap.c +1 -0
 		if (!lruvec)
 			break;
+		rcu_read_lock();
 		spin_lock_irq(&lruvec->lru_lock);
 	}
 }
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#57 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 99 lines · Harry Yoo
Quoted 88 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
     rcu_read_lock();
retry:
     lruvec = folio_lruvec(folio);
     /* There is a possibility of folio reparenting at this point. */
     spin_lock(&lruvec->lru_lock);
     if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
         /*
          * The wrong lruvec lock was acquired, and a retry is required.
          * This is because the folio resides on the parent memcg lruvec
          * list.
          */
         spin_unlock(&lruvec->lru_lock);
         goto retry;
     }

     /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
     spin_lock(&lruvec->lru_lock);
     spin_lock(&lruvec_parent->lru_lock);
     /* Transfer folios from the lruvec list to the parent's. */
     spin_unlock(&lruvec_parent->lru_lock);
     spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
  include/linux/memcontrol.h | 45 +++++++++++++++++++----------
  include/linux/swap.h       |  1 +
  mm/compaction.c            | 29 +++++++++++++++----
  mm/memcontrol.c            | 59 +++++++++++++++++++++-----------------
  mm/swap.c                  |  4 +++
  5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +9 -1
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
   * folio_lruvec - return lruvec for isolating/putting an LRU folio
   * @folio: Pointer to the folio.
   *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
   */
  static inline struct lruvec *folio_lruvec(struct folio *folio)
  {
mm/memcontrol.c +0 -0
index 548e67dbf2386..a1573600d4188 100644
mm/swap.c +0 -0
index cb1148a92d8ec..7e53479ca1732 100644

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 290

Quoted 5 lines
mm/swap.c +1 -0
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  		}
 
  		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();
  		lruvec = parent_lruvec(lruvec);

It looks bit weird to call parent_lruvec(lruvec) outside RCU read lock because the reason why it holds RCU read lock is to prevent release of memory cgroup and its lruvec.

Quoted 2 lines

I guess this isn't broken (for now) because all callers of lru_note_cost_unlock_irq() are holding a reference to the memcg?

I checked all the callers again, and they do indeed hold the refcnt for the memcg, so it's safe for now. But it seems rather fragile, perhaps we should also include parent_lruvec() within the RCU lock.

Quoted 8 lines
Quoted 7 lines
mm/swap.c +1 -0
  		if (!lruvec)
  			break;
+		rcu_read_lock();
  		spin_lock_irq(&lruvec->lru_lock);
  	}
  }
#58 lore raw

Re: [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

Quoted 101 lines · Qi Zheng
Quoted 98 lines · Harry Yoo
Quoted 88 lines · Qi Zheng

From: Muchun Song <[email protected]>

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.

In the folio_lruvec_lock(folio) function:

```
     rcu_read_lock();
retry:
     lruvec = folio_lruvec(folio);
     /* There is a possibility of folio reparenting at this point. */
     spin_lock(&lruvec->lru_lock);
     if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
         /*
          * The wrong lruvec lock was acquired, and a retry is required.
          * This is because the folio resides on the parent memcg lruvec
          * list.
          */
         spin_unlock(&lruvec->lru_lock);
         goto retry;
     }

     /* Reaching here indicates that folio_memcg() is stable. */
```

In the memcg_reparent_objcgs(memcg) function:

```
     spin_lock(&lruvec->lru_lock);
     spin_lock(&lruvec_parent->lru_lock);
     /* Transfer folios from the lruvec list to the parent's. */
     spin_unlock(&lruvec_parent->lru_lock);
     spin_unlock(&lruvec->lru_lock);
```

After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.

Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.

This patch serves as a preparation for the reparenting of LRU folios.

---
  include/linux/memcontrol.h | 45 +++++++++++++++++++----------
  include/linux/swap.h       |  1 +
  mm/compaction.c            | 29 +++++++++++++++----
  mm/memcontrol.c            | 59 +++++++++++++++++++++-----------------
  mm/swap.c                  |  4 +++
  5 files changed, 91 insertions(+), 47 deletions(-)
include/linux/memcontrol.h +9 -1
index 4b6f20dc694ba..26c3c0e375f58 100644
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,
   * folio_lruvec - return lruvec for isolating/putting an LRU folio
   * @folio: Pointer to the folio.
   *
- * This function relies on folio->mem_cgroup being stable.
+ * Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec.
+ * Note that this alone will NOT guarantee the stability of the folio->lruvec
+ * association; the folio can be reparented to an ancestor if this races with
+ * cgroup deletion.
+ *
+ * Use folio_lruvec_lock() to ensure both lifetime and stability of the binding.
+ * Once a lruvec is locked, folio_lruvec() can be called on other folios, and
+ * their binding is stable if the returned lruvec matches the one the caller has
+ * locked. Useful for lock batching.
   */
  static inline struct lruvec *folio_lruvec(struct folio *folio)
  {
mm/memcontrol.c +0 -0
index 548e67dbf2386..a1573600d4188 100644
mm/swap.c +0 -0
index cb1148a92d8ec..7e53479ca1732 100644

mm/swap.c void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, · line 289

Quoted 4 lines
mm/swap.c +1 -0
@@ -284,9 +286,11 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
  		}
  		spin_unlock_irq(&lruvec->lru_lock);
+		rcu_read_unlock();
  		lruvec = parent_lruvec(lruvec);

It looks bit weird to call parent_lruvec(lruvec) outside RCU read lock because the reason why it holds RCU read lock is to prevent release of memory cgroup and its lruvec.

Quoted 5 lines
Quoted 2 lines

I guess this isn't broken (for now) because all callers of lru_note_cost_unlock_irq() are holding a reference to the memcg?

I checked all the callers again, and they do indeed hold the refcnt for the memcg, so it's safe for now.

Thanks for double checking!

Quoted 1 line

But it seems rather fragile,

Yeah, it's fragile and

Quoted 1 line

perhaps we should also include parent_lruvec() within the RCU lock.

that would be much better.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
mm/swap.c +1 -0
  		if (!lruvec)
  			break;
+		rcu_read_lock();
  		spin_lock_irq(&lruvec->lru_lock);
  	}
  }
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#59 lore raw

PATCH v3 25/30 mm: vmscan: prepare for reparenting traditional LRU folios

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. For traditional LRU list, each lruvec of every memcg comprises four LRU lists. Due to the symmetry of the LRU lists, it is feasible to transfer the LRU lists from a memcg to its parent memcg during the reparenting process.

This commit implements the specific function, which will be used during the reparenting process.

---
 include/linux/mmzone.h |  4 ++++
 include/linux/swap.h   | 19 +++++++++++++++++++
 mm/swap.c              | 37 +++++++++++++++++++++++++++++++++++++
 mm/vmscan.c            | 19 -------------------
 4 files changed, 60 insertions(+), 19 deletions(-)
include/linux/mmzone.h +4 -0
index 6a7db0fee54a3..1014b5a93c09c 100644
@@ -366,6 +366,10 @@ enum lruvec_flags {
 	LRUVEC_NODE_CONGESTED,
 };
 
+#ifdef CONFIG_MEMCG
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+#endif /* CONFIG_MEMCG */
+
 #endif /* !__GENERATING_BOUNDS_H */
 
 /*
include/linux/swap.h +19 -0
index e60f45b48e74d..4449d1f371a56 100644
@@ -636,5 +636,24 @@ static inline bool mem_cgroup_swap_full(struct folio *folio)
 }
 #endif
 
+/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to
+ * and including the specified highidx
+ * @zone: The current zone in the iterator
+ * @pgdat: The pgdat which node_zones are being iterated
+ * @idx: The index variable
+ * @highidx: The index of the highest zone to return
+ *
+ * This macro iterates through all managed zones up to and including the specified highidx.
+ * The zone iterator enters an invalid state after macro call and must be reinitialized
+ * before it can be used again.
+ */
+#define for_each_managed_zone_pgdat(zone, pgdat, idx, highidx)	\
+	for ((idx) = 0, (zone) = (pgdat)->node_zones;		\
+	    (idx) <= (highidx);					\
+	    (idx)++, (zone)++)					\
+		if (!managed_zone(zone))			\
+			continue;				\
+		else
+
 #endif /* __KERNEL__*/
 #endif /* _LINUX_SWAP_H */
mm/swap.c +37 -0
index 7e53479ca1732..cb40e80da53cd 100644
@@ -1090,6 +1090,43 @@ void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
 	fbatch->nr = j;
 }
 
+#ifdef CONFIG_MEMCG
+static void lruvec_reparent_lru(struct lruvec *child_lruvec,
+				struct lruvec *parent_lruvec,
+				enum lru_list lru, int nid)
+{
+	int zid;
+	struct zone *zone;
+
+	if (lru != LRU_UNEVICTABLE)
+		list_splice_tail_init(&child_lruvec->lists[lru], &parent_lruvec->lists[lru]);
+
+	for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+		unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
+
+		mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
+	}
+}
+
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int nid;
+
+	for_each_node(nid) {
+		enum lru_list lru;
+		struct lruvec *child_lruvec, *parent_lruvec;
+
+		child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
+		parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
+		parent_lruvec->anon_cost += child_lruvec->anon_cost;
+		parent_lruvec->file_cost += child_lruvec->file_cost;
+
+		for_each_lru(lru)
+			lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
+	}
+}
+#endif
+
 static const struct ctl_table swap_sysctl_table[] = {
 	{
 		.procname	= "page-cluster",
mm/vmscan.c +0 -19
index c48ff6e05e004..e738082874878 100644
@@ -270,25 +270,6 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg)
 }
 #endif
 
-/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to
- * and including the specified highidx
- * @zone: The current zone in the iterator
- * @pgdat: The pgdat which node_zones are being iterated
- * @idx: The index variable
- * @highidx: The index of the highest zone to return
- *
- * This macro iterates through all managed zones up to and including the specified highidx.
- * The zone iterator enters an invalid state after macro call and must be reinitialized
- * before it can be used again.
- */
-#define for_each_managed_zone_pgdat(zone, pgdat, idx, highidx)	\
-	for ((idx) = 0, (zone) = (pgdat)->node_zones;		\
-	    (idx) <= (highidx);					\
-	    (idx)++, (zone)++)					\
-		if (!managed_zone(zone))			\
-			continue;				\
-		else
-
 static void set_task_reclaim_state(struct task_struct *task,
 				   struct reclaim_state *rs)
 {
Signature · 1 line
2.20.1
reviewer Acked-by on v3 25/30
#60 lore raw

Re: [PATCH v3 25/30] mm: vmscan: prepare for reparenting traditional LRU folios

Quoted 8 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. For traditional LRU list, each lruvec of every memcg comprises four LRU lists. Due to the symmetry of the LRU lists, it is feasible to transfer the LRU lists from a memcg to its parent memcg during the reparenting process.

Quoted 6 lines

This commit implements the specific function, which will be used during the reparenting process.

With one comment below.

Quoted 12 lines
---
  include/linux/mmzone.h |  4 ++++
  include/linux/swap.h   | 19 +++++++++++++++++++
  mm/swap.c              | 37 +++++++++++++++++++++++++++++++++++++
  mm/vmscan.c            | 19 -------------------
  4 files changed, 60 insertions(+), 19 deletions(-)
include/linux/mmzone.h +0 -0
index 6a7db0fee54a3..1014b5a93c09c 100644

include/linux/mmzone.h enum lruvec_flags { · line 371

Quoted 6 lines
include/linux/mmzone.h +3 -0
@@ -366,6 +366,10 @@ enum lruvec_flags {
  	LRUVEC_NODE_CONGESTED,
  };
 
+#ifdef CONFIG_MEMCG
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+#endif /* CONFIG_MEMCG */

Can we move it to swap.h since it is declared in swap.c?

Quoted 111 lines
include/linux/mmzone.h +1 -0
+
  #endif /* !__GENERATING_BOUNDS_H */
 
  /*
include/linux/swap.h +19 -0
index e60f45b48e74d..4449d1f371a56 100644
@@ -636,5 +636,24 @@ static inline bool mem_cgroup_swap_full(struct folio *folio)
  }
  #endif
 
+/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to
+ * and including the specified highidx
+ * @zone: The current zone in the iterator
+ * @pgdat: The pgdat which node_zones are being iterated
+ * @idx: The index variable
+ * @highidx: The index of the highest zone to return
+ *
+ * This macro iterates through all managed zones up to and including the specified highidx.
+ * The zone iterator enters an invalid state after macro call and must be reinitialized
+ * before it can be used again.
+ */
+#define for_each_managed_zone_pgdat(zone, pgdat, idx, highidx)	\
+	for ((idx) = 0, (zone) = (pgdat)->node_zones;		\
+	    (idx) <= (highidx);					\
+	    (idx)++, (zone)++)					\
+		if (!managed_zone(zone))			\
+			continue;				\
+		else
+
  #endif /* __KERNEL__*/
  #endif /* _LINUX_SWAP_H */
mm/swap.c +37 -0
index 7e53479ca1732..cb40e80da53cd 100644
@@ -1090,6 +1090,43 @@ void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
  	fbatch->nr = j;
  }
 
+#ifdef CONFIG_MEMCG
+static void lruvec_reparent_lru(struct lruvec *child_lruvec,
+				struct lruvec *parent_lruvec,
+				enum lru_list lru, int nid)
+{
+	int zid;
+	struct zone *zone;
+
+	if (lru != LRU_UNEVICTABLE)
+		list_splice_tail_init(&child_lruvec->lists[lru], &parent_lruvec->lists[lru]);
+
+	for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+		unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
+
+		mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
+	}
+}
+
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int nid;
+
+	for_each_node(nid) {
+		enum lru_list lru;
+		struct lruvec *child_lruvec, *parent_lruvec;
+
+		child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
+		parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
+		parent_lruvec->anon_cost += child_lruvec->anon_cost;
+		parent_lruvec->file_cost += child_lruvec->file_cost;
+
+		for_each_lru(lru)
+			lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
+	}
+}
+#endif
+
  static const struct ctl_table swap_sysctl_table[] = {
  	{
  		.procname	= "page-cluster",
mm/vmscan.c +0 -19
index c48ff6e05e004..e738082874878 100644
@@ -270,25 +270,6 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg)
  }
  #endif
 
-/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to
- * and including the specified highidx
- * @zone: The current zone in the iterator
- * @pgdat: The pgdat which node_zones are being iterated
- * @idx: The index variable
- * @highidx: The index of the highest zone to return
- *
- * This macro iterates through all managed zones up to and including the specified highidx.
- * The zone iterator enters an invalid state after macro call and must be reinitialized
- * before it can be used again.
- */
-#define for_each_managed_zone_pgdat(zone, pgdat, idx, highidx)	\
-	for ((idx) = 0, (zone) = (pgdat)->node_zones;		\
-	    (idx) <= (highidx);					\
-	    (idx)++, (zone)++)					\
-		if (!managed_zone(zone))			\
-			continue;				\
-		else
-
  static void set_task_reclaim_state(struct task_struct *task,
  				   struct reclaim_state *rs)
  {
maintainer
#61 lore raw

Re: [PATCH v3 25/30] mm: vmscan: prepare for reparenting traditional LRU folios

In reply to Qi Zheng
Quoted 2 lines · Qi Zheng

From: Qi Zheng <[email protected]>

[...]

mm/swap.c void folio_batch_remove_exceptionals(struct folio_batch *fbatch) · line 1092

Quoted 6 lines
mm/swap.c +0 -0
@@ -1090,6 +1090,43 @@ void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
 	fbatch->nr = j;
 }
 

Why not define the following two functions in memcontrol.c?

Quoted 37 lines
mm/swap.c +37 -0
+#ifdef CONFIG_MEMCG
+static void lruvec_reparent_lru(struct lruvec *child_lruvec,
+				struct lruvec *parent_lruvec,
+				enum lru_list lru, int nid)
+{
+	int zid;
+	struct zone *zone;
+
+	if (lru != LRU_UNEVICTABLE)
+		list_splice_tail_init(&child_lruvec->lists[lru], &parent_lruvec->lists[lru]);
+
+	for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+		unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
+
+		mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
+	}
+}
+
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int nid;
+
+	for_each_node(nid) {
+		enum lru_list lru;
+		struct lruvec *child_lruvec, *parent_lruvec;
+
+		child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
+		parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
+		parent_lruvec->anon_cost += child_lruvec->anon_cost;
+		parent_lruvec->file_cost += child_lruvec->file_cost;
+
+		for_each_lru(lru)
+			lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
+	}
+}
+#endif
+
author reviewer
#62 lore raw

Re: [PATCH v3 25/30] mm: vmscan: prepare for reparenting traditional LRU folios

Quoted 4 lines · Shakeel Butt
Quoted 2 lines · Qi Zheng

From: Qi Zheng <[email protected]>

[...]

Quoted 8 lines

mm/swap.c void folio_batch_remove_exceptionals(struct folio_batch *fbatch) · line 1092

Quoted 6 lines
mm/swap.c +0 -0
@@ -1090,6 +1090,43 @@ void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
  	fbatch->nr = j;
  }
 

Why not define the following two functions in memcontrol.c?

Ah, Because Johannes previously suggested [1] putting it in swap.c:

```
Lastly, vmscan.c is the reclaim policy. Mechanical LRU shuffling like
this is better placed in mm/swap.c.
```

[1]. https://lore.kernel.org/all/[email protected]/

Quoted 40 lines
Quoted 38 lines
mm/swap.c +37 -0
+#ifdef CONFIG_MEMCG
+static void lruvec_reparent_lru(struct lruvec *child_lruvec,
+				struct lruvec *parent_lruvec,
+				enum lru_list lru, int nid)
+{
+	int zid;
+	struct zone *zone;
+
+	if (lru != LRU_UNEVICTABLE)
+		list_splice_tail_init(&child_lruvec->lists[lru], &parent_lruvec->lists[lru]);
+
+	for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+		unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
+
+		mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
+	}
+}
+
+void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int nid;
+
+	for_each_node(nid) {
+		enum lru_list lru;
+		struct lruvec *child_lruvec, *parent_lruvec;
+
+		child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
+		parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
+		parent_lruvec->anon_cost += child_lruvec->anon_cost;
+		parent_lruvec->file_cost += child_lruvec->file_cost;
+
+		for_each_lru(lru)
+			lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
+	}
+}
+#endif
+
author reviewer
#63 lore raw

PATCH v3 26/30 mm: vmscan: prepare for reparenting MGLRU folios

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

Similar to traditional LRU folios, in order to solve the dying memcg problem, we also need to reparenting MGLRU folios to the parent memcg when memcg offline.

However, there are the following challenges:

1. Each lruvec has between MIN_NR_GENS and MAX_NR_GENS generations, the number of generations of the parent and child memcg may be different, so we cannot simply transfer MGLRU folios in the child memcg to the parent memcg as we did for traditional LRU folios.
2. The generation information is stored in folio->flags, but we cannot traverse these folios while holding the lru lock, otherwise it may cause softlockup.
3. In walk_update_folio(), the gen of folio and corresponding lru size may be updated, but the folio is not immediately moved to the corresponding lru list. Therefore, there may be folios of different generations on an LRU list.
4. In lru_gen_del_folio(), the generation to which the folio belongs is found based on the generation information in folio->flags, and the corresponding LRU size will be updated. Therefore, we need to update the lru size correctly during reparenting, otherwise the lru size may be updated incorrectly in lru_gen_del_folio().

Finally, this patch chose a compromise method, which is to splice the lru list in the child memcg to the lru list of the same generation in the parent memcg during reparenting. And in order to ensure that the parent memcg has the same generation, we need to increase the generations in the parent memcg to the MAX_NR_GENS before reparenting.

Of course, the same generation has different meanings in the parent and child memcg, this will cause confusion in the hot and cold information of folios. But other than that, this method is simple enough, the lru size is correct, and there is no need to consider some concurrency issues (such as lru_gen_del_folio()).

To prepare for the above work, this commit implements the specific functions, which will be used during reparenting.

---
 include/linux/mmzone.h |  16 +++++
 mm/vmscan.c            | 144 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 160 insertions(+)
include/linux/mmzone.h +16 -0
index 1014b5a93c09c..a41f4f0ae5eb7 100644
@@ -628,6 +628,9 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg);
 void lru_gen_offline_memcg(struct mem_cgroup *memcg);
 void lru_gen_release_memcg(struct mem_cgroup *memcg);
 void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid);
+void max_lru_gen_memcg(struct mem_cgroup *memcg);
+bool recheck_lru_gen_max_memcg(struct mem_cgroup *memcg);
+void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 #else /* !CONFIG_LRU_GEN */
 
@@ -668,6 +671,19 @@ static inline void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid)
 {
 }
 
+static inline void max_lru_gen_memcg(struct mem_cgroup *memcg)
+{
+}
+
+static inline bool recheck_lru_gen_max_memcg(struct mem_cgroup *memcg)
+{
+	return true;
+}
+
+static inline void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
 #endif /* CONFIG_LRU_GEN */
 
 struct lruvec {
mm/vmscan.c +144 -0

150 lines not on this page · raw message

Signature · 1 line
2.20.1
author reviewer
#64 lore raw

PATCH v3 26/30 mm: mglru: do not call update_lru_size() during reparenting

From: Qi Zheng <[email protected]>

Only non-hierarchical lruvec_stats->state_local needs to be reparented, so handle it in reparent_state_local(), and remove the unreasonable update_lru_size() call in __lru_gen_reparent_memcg().

---
 mm/vmscan.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
mm/vmscan.c +9 -7
index 56714f3bc6f88..5e7a32e3cffbc 100644
@@ -4544,7 +4544,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
 				     int zone, int type)
 {
 	struct lru_gen_folio *child_lrugen, *parent_lrugen;
-	enum lru_list lru = type * LRU_INACTIVE_FILE;
 	int i;
 
 	child_lrugen = &child_lruvec->lrugen;
@@ -4553,7 +4552,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
 	for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
 		int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
 		long nr_pages = child_lrugen->nr_pages[gen][type][zone];
-		int dst_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
 
 		/* Assuming that child pages are colder than parent pages */
 		list_splice_init(&child_lrugen->folios[gen][type][zone],
@@ -4562,8 +4560,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
 		WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
 		WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
 			   parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
-
-		update_lru_size(parent_lruvec, lru + dst_lru_active, zone, nr_pages);
 	}
 }
 
@@ -4575,15 +4571,21 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 		struct lruvec *child_lruvec, *parent_lruvec;
 		int type, zid;
 		struct zone *zone;
+		enum lru_list lru;
 
 		child_lruvec = get_lruvec(memcg, nid);
 		parent_lruvec = get_lruvec(parent, nid);
 
-		for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+		for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1)
 			for (type = 0; type < ANON_AND_FILE; type++)
 				__lru_gen_reparent_memcg(child_lruvec, parent_lruvec, zid, type);
-			mem_cgroup_update_lru_size(parent_lruvec, LRU_UNEVICTABLE, zid,
-				mem_cgroup_get_zone_lru_size(child_lruvec, LRU_UNEVICTABLE, zid));
+
+		for_each_lru(lru) {
+			for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+				unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
+
+				mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
+			}
 		}
 	}
 }
Signature · 1 line
2.20.1
maintainer
#65 lore raw

Re: [PATCH v3 26/30 fix] mm: mglru: do not call update_lru_size() during reparenting

Quoted 6 lines · On Thu, 15 Jan 2026 18:44:44 +0800 Qi Zheng <[email protected]> wrote:

From: Qi Zheng <[email protected]>

Only non-hierarchical lruvec_stats->state_local needs to be reparented, so handle it in reparent_state_local(), and remove the unreasonable update_lru_size() call in __lru_gen_reparent_memcg().

Thanks, added as a squashable fix against "mm: vmscan: prepare for reparenting MGLRU folios".

#66 lore raw

Re: [PATCH v3 26/30 fix] mm: mglru: do not call update_lru_size() during reparenting

In reply to Qi Zheng
Quoted 5 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Only non-hierarchical lruvec_stats->state_local needs to be reparented, so handle it in reparent_state_local(), and remove the unreasonable update_lru_size() call in __lru_gen_reparent_memcg().

Hmm well, how are the hierarchical statistics consistent when pages are reparented from an "active" gen to an "inactive" gen, or the other way around?

They'll become inconsistent when those pages are reclaimed or moved between generations?

Signature · 2 lines
Cheers,
Harry / Hyeonggon
#67 lore raw

Re: [PATCH v3 26/30 fix] mm: mglru: do not call update_lru_size() during reparenting

Quoted 7 lines · Harry Yoo
Quoted 5 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Only non-hierarchical lruvec_stats->state_local needs to be reparented, so handle it in reparent_state_local(), and remove the unreasonable update_lru_size() call in __lru_gen_reparent_memcg().

Quoted 5 lines

Hmm well, how are the hierarchical statistics consistent when pages are reparented from an "active" gen to an "inactive" gen, or the other way around?

They'll become inconsistent when those pages are reclaimed or moved between generations?

FYI we've observed this while testing downstream implementation as it led to MemAvailable being unreasonably high due to inconsistent statistics.

The solution was, if lru_gen_is_active(child, gen) and lru_gen_is_active(parent, gen) do not match, # of pages being reparented must be subtracted from the child's statistics (and up to the root, as it's hierarchical), and added to the parent's statistics for the generation.

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#68 lore raw

Re: [PATCH v3 26/30 fix] mm: mglru: do not call update_lru_size() during reparenting

Quoted 10 lines · Harry Yoo
Quoted 9 lines · Harry Yoo
Quoted 5 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Only non-hierarchical lruvec_stats->state_local needs to be reparented, so handle it in reparent_state_local(), and remove the unreasonable update_lru_size() call in __lru_gen_reparent_memcg().

Hmm well, how are the hierarchical statistics consistent when pages are reparented from an "active" gen to an "inactive" gen, or the other way around?

Oh, I completely forgot about that. If update_lru_size() is not called during the rreparenting, this issue should be considered separately.

Quoted 8 lines
Quoted 3 lines

They'll become inconsistent when those pages are reclaimed or moved between generations?

FYI we've observed this while testing downstream implementation as it led to MemAvailable being unreasonably high due to inconsistent statistics.

Quoted 5 lines

The solution was, if lru_gen_is_active(child, gen) and lru_gen_is_active(parent, gen) do not match, # of pages being reparented must be subtracted from the child's statistics (and up to the root, as it's hierarchical), and added to the parent's statistics for the generation.

Make sense, will fix it in v4.

Thanks!

Quoted 1 line
maintainer
#69 lore raw

Re: [PATCH v3 26/30] mm: vmscan: prepare for reparenting MGLRU folios

In reply to Qi Zheng

Axel, Yuanchu & Wei, please help reviewing this patch.

Quoted 240 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Similar to traditional LRU folios, in order to solve the dying memcg problem, we also need to reparenting MGLRU folios to the parent memcg when memcg offline.

However, there are the following challenges:

1. Each lruvec has between MIN_NR_GENS and MAX_NR_GENS generations, the number of generations of the parent and child memcg may be different, so we cannot simply transfer MGLRU folios in the child memcg to the parent memcg as we did for traditional LRU folios.
2. The generation information is stored in folio->flags, but we cannot traverse these folios while holding the lru lock, otherwise it may cause softlockup.
3. In walk_update_folio(), the gen of folio and corresponding lru size may be updated, but the folio is not immediately moved to the corresponding lru list. Therefore, there may be folios of different generations on an LRU list.
4. In lru_gen_del_folio(), the generation to which the folio belongs is found based on the generation information in folio->flags, and the corresponding LRU size will be updated. Therefore, we need to update the lru size correctly during reparenting, otherwise the lru size may be updated incorrectly in lru_gen_del_folio().

Finally, this patch chose a compromise method, which is to splice the lru list in the child memcg to the lru list of the same generation in the parent memcg during reparenting. And in order to ensure that the parent memcg has the same generation, we need to increase the generations in the parent memcg to the MAX_NR_GENS before reparenting.

Of course, the same generation has different meanings in the parent and child memcg, this will cause confusion in the hot and cold information of folios. But other than that, this method is simple enough, the lru size is correct, and there is no need to consider some concurrency issues (such as lru_gen_del_folio()).

To prepare for the above work, this commit implements the specific functions, which will be used during reparenting.

---
 include/linux/mmzone.h |  16 +++++
 mm/vmscan.c            | 144 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 160 insertions(+)
include/linux/mmzone.h +16 -0
index 1014b5a93c09c..a41f4f0ae5eb7 100644
@@ -628,6 +628,9 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg);
 void lru_gen_offline_memcg(struct mem_cgroup *memcg);
 void lru_gen_release_memcg(struct mem_cgroup *memcg);
 void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid);
+void max_lru_gen_memcg(struct mem_cgroup *memcg);
+bool recheck_lru_gen_max_memcg(struct mem_cgroup *memcg);
+void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 #else /* !CONFIG_LRU_GEN */
 
@@ -668,6 +671,19 @@ static inline void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid)
 {
 }
 
+static inline void max_lru_gen_memcg(struct mem_cgroup *memcg)
+{
+}
+
+static inline bool recheck_lru_gen_max_memcg(struct mem_cgroup *memcg)
+{
+	return true;
+}
+
+static inline void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
 #endif /* CONFIG_LRU_GEN */
 
 struct lruvec {
mm/vmscan.c +144 -0
index e738082874878..6bc8047b7aec5 100644
@@ -4445,6 +4445,150 @@ void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid)
 		lru_gen_rotate_memcg(lruvec, MEMCG_LRU_HEAD);
 }
 
+bool recheck_lru_gen_max_memcg(struct mem_cgroup *memcg)
+{
+	int nid;
+
+	for_each_node(nid) {
+		struct lruvec *lruvec = get_lruvec(memcg, nid);
+		int type;
+
+		for (type = 0; type < ANON_AND_FILE; type++) {
+			if (get_nr_gens(lruvec, type) != MAX_NR_GENS)
+				return false;
+		}
+	}
+
+	return true;
+}
+
+static void try_to_inc_max_seq_nowalk(struct mem_cgroup *memcg,
+				      struct lruvec *lruvec)
+{
+	struct lru_gen_mm_list *mm_list = get_mm_list(memcg);
+	struct lru_gen_mm_state *mm_state = get_mm_state(lruvec);
+	int swappiness = mem_cgroup_swappiness(memcg);
+	DEFINE_MAX_SEQ(lruvec);
+	bool success = false;
+
+	/*
+	 * We are not iterating the mm_list here, updating mm_state->seq is just
+	 * to make mm walkers work properly.
+	 */
+	if (mm_state) {
+		spin_lock(&mm_list->lock);
+		VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq);
+		if (max_seq > mm_state->seq) {
+			WRITE_ONCE(mm_state->seq, mm_state->seq + 1);
+			success = true;
+		}
+		spin_unlock(&mm_list->lock);
+	} else {
+		success = true;
+	}
+
+	if (success)
+		inc_max_seq(lruvec, max_seq, swappiness);
+}
+
+/*
+ * We need to ensure that the folios of child memcg can be reparented to the
+ * same gen of the parent memcg, so the gens of the parent memcg needed be
+ * incremented to the MAX_NR_GENS before reparenting.
+ */
+void max_lru_gen_memcg(struct mem_cgroup *memcg)
+{
+	int nid;
+
+	for_each_node(nid) {
+		struct lruvec *lruvec = get_lruvec(memcg, nid);
+		int type;
+
+		for (type = 0; type < ANON_AND_FILE; type++) {
+			while (get_nr_gens(lruvec, type) < MAX_NR_GENS) {
+				try_to_inc_max_seq_nowalk(memcg, lruvec);
+				cond_resched();
+			}
+		}
+	}
+}
+
+/*
+ * Compared to traditional LRU, MGLRU faces the following challenges:
+ *
+ * 1. Each lruvec has between MIN_NR_GENS and MAX_NR_GENS generations, the
+ *    number of generations of the parent and child memcg may be different,
+ *    so we cannot simply transfer MGLRU folios in the child memcg to the
+ *    parent memcg as we did for traditional LRU folios.
+ * 2. The generation information is stored in folio->flags, but we cannot
+ *    traverse these folios while holding the lru lock, otherwise it may
+ *    cause softlockup.
+ * 3. In walk_update_folio(), the gen of folio and corresponding lru size
+ *    may be updated, but the folio is not immediately moved to the
+ *    corresponding lru list. Therefore, there may be folios of different
+ *    generations on an LRU list.
+ * 4. In lru_gen_del_folio(), the generation to which the folio belongs is
+ *    found based on the generation information in folio->flags, and the
+ *    corresponding LRU size will be updated. Therefore, we need to update
+ *    the lru size correctly during reparenting, otherwise the lru size may
+ *    be updated incorrectly in lru_gen_del_folio().
+ *
+ * Finally, we choose a compromise method, which is to splice the lru list in
+ * the child memcg to the lru list of the same generation in the parent memcg
+ * during reparenting.
+ *
+ * The same generation has different meanings in the parent and child memcg,
+ * so this compromise method will cause the LRU inversion problem. But as the
+ * system runs, this problem will be fixed automatically.
+ */
+static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec *parent_lruvec,
+				     int zone, int type)
+{
+	struct lru_gen_folio *child_lrugen, *parent_lrugen;
+	enum lru_list lru = type * LRU_INACTIVE_FILE;
+	int i;
+
+	child_lrugen = &child_lruvec->lrugen;
+	parent_lrugen = &parent_lruvec->lrugen;
+
+	for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
+		int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
+		long nr_pages = child_lrugen->nr_pages[gen][type][zone];
+		int dst_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
+
+		/* Assuming that child pages are colder than parent pages */
+		list_splice_init(&child_lrugen->folios[gen][type][zone],
+				 &parent_lrugen->folios[gen][type][zone]);
+
+		WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
+		WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
+			   parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+
+		update_lru_size(parent_lruvec, lru + dst_lru_active, zone, nr_pages);
+	}
+}
+
+void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int nid;
+
+	for_each_node(nid) {
+		struct lruvec *child_lruvec, *parent_lruvec;
+		int type, zid;
+		struct zone *zone;
+
+		child_lruvec = get_lruvec(memcg, nid);
+		parent_lruvec = get_lruvec(parent, nid);
+
+		for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
+			for (type = 0; type < ANON_AND_FILE; type++)
+				__lru_gen_reparent_memcg(child_lruvec, parent_lruvec, zid, type);
+			mem_cgroup_update_lru_size(parent_lruvec, LRU_UNEVICTABLE, zid,
+				mem_cgroup_get_zone_lru_size(child_lruvec, LRU_UNEVICTABLE, zid));
+		}
+	}
+}
+
 #endif /* CONFIG_MEMCG */
 
 /******************************************************************************
Signature · 1 line
2.20.1
maintainer
#70 lore raw

Re: [PATCH v3 26/30] mm: vmscan: prepare for reparenting MGLRU folios

In reply to Qi Zheng

I asked AI to review this patch and it find following potential issues:

Author: Qi Zheng <[email protected]>

mm: vmscan: prepare for reparenting MGLRU folios

This commit adds infrastructure for reparenting MGLRU folios from dying memcg to parent memcg, including functions to ensure parent has MAX_NR_GENS generations before reparenting.

Quoted 4 lines
mm/vmscan.c +0 -0
index f1a45c94c2fc..284292653d14 100644

[ ... ]

Quoted 7 lines
mm/vmscan.c +7 -0
+static void try_to_inc_max_seq_nowalk(struct mem_cgroup *memcg,
+				      struct lruvec *lruvec)
+{
+	struct lru_gen_mm_list *mm_list = get_mm_list(memcg);
+	struct lru_gen_mm_state *mm_state = get_mm_state(lruvec);
+	int swappiness = mem_cgroup_swappiness(memcg);
+	DEFINE_MAX_SEQ(lruvec);

mm/vmscan.c void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) · line 4481

Quoted 10 lines
mm/vmscan.c +10 -0
+	bool success = false;
+
+	/*
+	 * We are not iterating the mm_list here, updating mm_state->seq is just
+	 * to make mm walkers work properly.
+	 */
+	if (mm_state) {
+		spin_lock(&mm_list->lock);
+		VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq);
+		if (max_seq > mm_state->seq) {
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When mm_state->seq equals max_seq (which can happen when mm walkers have completed their iteration), this condition is false, success remains false, and inc_max_seq() is never called.

The caller max_lru_gen_memcg() loops with:

    while (get_nr_gens(lruvec, type) < MAX_NR_GENS) {
        try_to_inc_max_seq_nowalk(memcg, lruvec);
        cond_resched();
    }

If try_to_inc_max_seq_nowalk() fails to increment max_seq, get_nr_gens() returns the same value, and the while loop continues indefinitely.

Can this cause an infinite loop during memcg offline when mm_state->seq has caught up to max_seq?

Suggested fix: Remove the success variable and always call inc_max_seq(). The mm_state->seq update is only for coordination with mm walkers - it should not gate whether we increment the generation. inc_max_seq() has its own staleness check (seq < READ_ONCE(lrugen->max_seq)) so calling it unconditionally is safe:

    if (mm_state) {
        spin_lock(&mm_list->lock);
        VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq);
        if (max_seq > mm_state->seq)
            WRITE_ONCE(mm_state->seq, mm_state->seq + 1);
        spin_unlock(&mm_list->lock);
    }

    inc_max_seq(lruvec, max_seq, swappiness);
Quoted 11 lines
mm/vmscan.c +11 -0
+			WRITE_ONCE(mm_state->seq, mm_state->seq + 1);
+			success = true;
+		}
+		spin_unlock(&mm_list->lock);
+	} else {
+		success = true;
+	}
+
+	if (success)
+		inc_max_seq(lruvec, max_seq, swappiness);
+}

[ ... ]

Quoted 6 lines
mm/vmscan.c +6 -0
+static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec *parent_lruvec,
+				     int zone, int type)
+{
+	struct lru_gen_folio *child_lrugen, *parent_lrugen;
+	enum lru_list lru = type * LRU_INACTIVE_FILE;
+	int i;

[ ... ]

mm/vmscan.c void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) · line 4567

Quoted 5 lines
mm/vmscan.c +5 -0
+		WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
+		WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
+			   parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+
+		update_lru_size(parent_lruvec, lru + dst_lru_active, zone, nr_pages);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This update_lru_size() call updates both __update_lru_size() counters and mem_cgroup_update_lru_size(). However, lru_gen_reparent_memcg() also calls mem_cgroup_update_lru_size() for LRU_UNEVICTABLE. The combination appears to cause incorrect accounting.

Note: This issue was fixed in later commit ("mm: mglru: do not call update_lru_size() during reparenting").

Quoted 2 lines
mm/vmscan.c +2 -0
+	}
+}
author reviewer
#71 lore raw

Re: [PATCH v3 26/30] mm: vmscan: prepare for reparenting MGLRU folios

Quoted 1 line · Shakeel Butt

I asked AI to review this patch and it find following potential issues:

Thanks.

Quoted 37 lines

Author: Qi Zheng <[email protected]>

mm: vmscan: prepare for reparenting MGLRU folios

This commit adds infrastructure for reparenting MGLRU folios from dying memcg to parent memcg, including functions to ensure parent has MAX_NR_GENS generations before reparenting.

Quoted 4 lines
mm/vmscan.c +0 -0
index f1a45c94c2fc..284292653d14 100644

[ ... ]

Quoted 7 lines
mm/vmscan.c +7 -0
+static void try_to_inc_max_seq_nowalk(struct mem_cgroup *memcg,
+				      struct lruvec *lruvec)
+{
+	struct lru_gen_mm_list *mm_list = get_mm_list(memcg);
+	struct lru_gen_mm_state *mm_state = get_mm_state(lruvec);
+	int swappiness = mem_cgroup_swappiness(memcg);
+	DEFINE_MAX_SEQ(lruvec);

mm/vmscan.c void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) · line 4481

Quoted 10 lines
mm/vmscan.c +10 -0
+	bool success = false;
+
+	/*
+	 * We are not iterating the mm_list here, updating mm_state->seq is just
+	 * to make mm walkers work properly.
+	 */
+	if (mm_state) {
+		spin_lock(&mm_list->lock);
+		VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq);
+		if (max_seq > mm_state->seq) {
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 3 lines

When mm_state->seq equals max_seq (which can happen when mm walkers have completed their iteration), this condition is false, success remains false, and inc_max_seq() is never called.

I expect that once mm walkers complete their iterations, the number of generations will have increased, so it won't cause an infinite loop.

Quoted 60 lines

The caller max_lru_gen_memcg() loops with:

     while (get_nr_gens(lruvec, type) < MAX_NR_GENS) {
         try_to_inc_max_seq_nowalk(memcg, lruvec);
         cond_resched();
     }

If try_to_inc_max_seq_nowalk() fails to increment max_seq, get_nr_gens() returns the same value, and the while loop continues indefinitely.

Can this cause an infinite loop during memcg offline when mm_state->seq has caught up to max_seq?

Suggested fix: Remove the success variable and always call inc_max_seq(). The mm_state->seq update is only for coordination with mm walkers - it should not gate whether we increment the generation. inc_max_seq() has its own staleness check (seq < READ_ONCE(lrugen->max_seq)) so calling it unconditionally is safe:

     if (mm_state) {
         spin_lock(&mm_list->lock);
         VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq);
         if (max_seq > mm_state->seq)
             WRITE_ONCE(mm_state->seq, mm_state->seq + 1);
         spin_unlock(&mm_list->lock);
     }

     inc_max_seq(lruvec, max_seq, swappiness);
Quoted 11 lines
mm/vmscan.c +11 -0
+			WRITE_ONCE(mm_state->seq, mm_state->seq + 1);
+			success = true;
+		}
+		spin_unlock(&mm_list->lock);
+	} else {
+		success = true;
+	}
+
+	if (success)
+		inc_max_seq(lruvec, max_seq, swappiness);
+}

[ ... ]

Quoted 6 lines
mm/vmscan.c +6 -0
+static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec *parent_lruvec,
+				     int zone, int type)
+{
+	struct lru_gen_folio *child_lrugen, *parent_lrugen;
+	enum lru_list lru = type * LRU_INACTIVE_FILE;
+	int i;

[ ... ]

mm/vmscan.c void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) · line 4567

Quoted 5 lines
mm/vmscan.c +5 -0
+		WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
+		WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
+			   parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+
+		update_lru_size(parent_lruvec, lru + dst_lru_active, zone, nr_pages);
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 7 lines

This update_lru_size() call updates both __update_lru_size() counters and mem_cgroup_update_lru_size(). However, lru_gen_reparent_memcg() also calls mem_cgroup_update_lru_size() for LRU_UNEVICTABLE. The combination appears to cause incorrect accounting.

Note: This issue was fixed in later commit ("mm: mglru: do not call update_lru_size() during reparenting").

Right.

Quoted 4 lines
Quoted 3 lines
mm/vmscan.c +2 -0
+	}
+}
author reviewer
#72 lore raw

PATCH v3 27/30 mm: memcontrol: refactor memcg_reparent_objcgs()

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

Refactor the memcg_reparent_objcgs() to facilitate subsequent reparenting LRU folios here.

---
 mm/memcontrol.c | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)
mm/memcontrol.c +24 -5
index a1573600d4188..70583394f421f 100644
@@ -208,15 +208,12 @@ static struct obj_cgroup *obj_cgroup_alloc(void)
 	return objcg;
 }
 
-static void memcg_reparent_objcgs(struct mem_cgroup *memcg)
+static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memcg,
+							 struct mem_cgroup *parent)
 {
 	struct obj_cgroup *objcg, *iter;
-	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
 
 	objcg = rcu_replace_pointer(memcg->objcg, NULL, true);
-
-	spin_lock_irq(&objcg_lock);
-
 	/* 1) Ready to reparent active objcg. */
 	list_add(&objcg->list, &memcg->objcg_list);
 	/* 2) Reparent active objcg and already reparented objcgs to parent. */
@@ -225,7 +222,29 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg)
 	/* 3) Move already reparented objcgs to the parent's list */
 	list_splice(&memcg->objcg_list, &parent->objcg_list);
 
+	return objcg;
+}
+
+static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	spin_lock_irq(&objcg_lock);
+}
+
+static inline void reparent_unlocks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
 	spin_unlock_irq(&objcg_lock);
+}
+
+static void memcg_reparent_objcgs(struct mem_cgroup *memcg)
+{
+	struct obj_cgroup *objcg;
+	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
+
+	reparent_locks(memcg, parent);
+
+	objcg = __memcg_reparent_objcgs(memcg, parent);
+
+	reparent_unlocks(memcg, parent);
 
 	percpu_ref_kill(&objcg->refcnt);
 }
Signature · 1 line
2.20.1
Reviewed-by on v3 27/30
#74 lore raw

Re: [PATCH v3 27/30] mm: memcontrol: refactor memcg_reparent_objcgs()

In reply to Qi Zheng
Quoted 8 lines · Qi Zheng

From: Qi Zheng <[email protected]>

Refactor the memcg_reparent_objcgs() to facilitate subsequent reparenting LRU folios here.

---

Looks good to me,

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#76 lore raw

PATCH v3 28/30 mm: memcontrol: prepare for reparenting state_local

In reply to Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

---
 include/linux/memcontrol.h |  4 +++
 mm/memcontrol-v1.c         | 16 +++++++++++
 mm/memcontrol-v1.h         |  3 ++
 mm/memcontrol.c            | 56 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 79 insertions(+)
include/linux/memcontrol.h +4 -0
index 26c3c0e375f58..f84a23f13ffb4 100644
@@ -963,12 +963,16 @@ static inline void mod_memcg_page_state(struct page *page,
 
 unsigned long memcg_events(struct mem_cgroup *memcg, int event);
 unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx);
+void reparent_memcg_state_local(struct mem_cgroup *memcg,
+				struct mem_cgroup *parent, int idx);
 unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
 bool memcg_stat_item_valid(int idx);
 bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
+void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+				       struct mem_cgroup *parent, int idx);
 
 void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
 void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +16 -0
index f0ef650d2317b..800606135e7ba 100644
@@ -1898,6 +1898,22 @@ static const unsigned int memcg1_events[] = {
 	PGMAJFAULT,
 };
 
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
+}
+
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
+}
+
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 {
 	unsigned long memory, memsw;
mm/memcontrol-v1.h +3 -0
index eb3c3c1056574..45528195d3578 100644
@@ -41,6 +41,7 @@ static inline bool do_memsw_account(void)
 
 unsigned long memcg_events_local(struct mem_cgroup *memcg, int event);
 unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx);
+void mod_memcg_page_state_local(struct mem_cgroup *memcg, int idx, unsigned long val);
 unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item);
 bool memcg1_alloc_events(struct mem_cgroup *memcg);
 void memcg1_free_events(struct mem_cgroup *memcg);
@@ -73,6 +74,8 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
 			   unsigned long nr_memory, int nid);
 
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
mm/memcontrol.c +56 -0

74 lines not on this page · raw message

Signature · 1 line
2.20.1
author reviewer
#77 lore raw

PATCH v3 1/2 mm: memcontrol: fix lruvec_stats->state_local reparenting

From: Qi Zheng <[email protected]>

---
 include/linux/memcontrol.h |  2 --
 mm/memcontrol-v1.c         |  8 --------
 mm/memcontrol-v1.h         |  5 ++++-
 mm/memcontrol.c            | 19 ++++++++-----------
 4 files changed, 12 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +0 -2
index 1fe554eec1e25..e0b84b109b7ac 100644
@@ -944,8 +944,6 @@ bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx);
 
 void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
 void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +0 -8
index 03b924920d6a5..daf9bad8c45ea 100644
@@ -1909,14 +1909,6 @@ void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *pa
 		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
 }
 
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
-	int i;
-
-	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
-		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
-}
-
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 {
 	unsigned long memory, memsw;
mm/memcontrol-v1.h +4 -1
index 45528195d3578..5b1188f3d4173 100644
@@ -75,7 +75,6 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
 
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
 void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
@@ -116,6 +115,10 @@ static inline void memcg1_uncharge_batch(struct mem_cgroup *memcg,
 
 static inline void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) {}
 
+static inline void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
 static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {}
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; }
 static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
mm/memcontrol.c +8 -11
index 7333a37830051..b7b35143d4d2d 100644
@@ -225,13 +225,13 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
 	return objcg;
 }
 
-#ifdef CONFIG_MEMCG_V1
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx);
 static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
 
 static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {
-	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		return;
+	int i;
 
 	synchronize_rcu();
 
@@ -239,13 +239,10 @@ static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgr
 
 	/* The following counts are all non-hierarchical and need to be reparented. */
 	reparent_memcg1_state_local(memcg, parent);
-	reparent_memcg1_lruvec_state_local(memcg, parent);
-}
-#else
-static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
+
+	for (i = 0; i < NR_LRU_LISTS; i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, i);
 }
-#endif
 
 static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {
@@ -510,8 +507,8 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 	return x;
 }
 
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx)
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx)
 {
 	int i = memcg_stats_index(idx);
 	int nid;
Signature · 1 line
2.20.1
author reviewer
#78 lore raw

PATCH v3 2/2 mm: memcontrol: change state_locals to atomic_long_t type

From: Qi Zheng <[email protected]>

---
 mm/memcontrol.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)
mm/memcontrol.c +10 -11
index b7b35143d4d2d..c303c483f55a4 100644
@@ -456,7 +456,7 @@ struct lruvec_stats {
 	long state[NR_MEMCG_NODE_STAT_ITEMS];
 
 	/* Non-hierarchical (CPU aggregated) state */
-	long state_local[NR_MEMCG_NODE_STAT_ITEMS];
+	atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS];
 
 	/* Pending child counts during tree propagation */
 	long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
@@ -499,7 +499,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 		return 0;
 
 	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
-	x = READ_ONCE(pn->lruvec_stats->state_local[i]);
+	x = atomic_long_read(&(pn->lruvec_stats->state_local[i]));
 #ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
@@ -524,8 +524,7 @@ static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
 
 		parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec);
 
-		WRITE_ONCE(parent_pn->lruvec_stats->state_local[i],
-			   parent_pn->lruvec_stats->state_local[i] + value);
+		atomic_long_add(value, &(parent_pn->lruvec_stats->state_local[i]));
 	}
 }
 
@@ -620,8 +619,8 @@ struct memcg_vmstats {
 	unsigned long		events[NR_MEMCG_EVENTS];
 
 	/* Non-hierarchical (CPU aggregated) page state & events */
-	long			state_local[MEMCG_VMSTAT_SIZE];
-	unsigned long		events_local[NR_MEMCG_EVENTS];
+	atomic_long_t		state_local[MEMCG_VMSTAT_SIZE];
+	atomic_long_t		events_local[NR_MEMCG_EVENTS];
 
 	/* Pending child counts during tree propagation */
 	long			state_pending[MEMCG_VMSTAT_SIZE];
@@ -824,7 +823,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
 	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
 		return 0;
 
-	x = READ_ONCE(memcg->vmstats->state_local[i]);
+	x = atomic_long_read(&(memcg->vmstats->state_local[i]));
 #ifdef CONFIG_SMP
 	if (x < 0)
 		x = 0;
@@ -841,7 +840,7 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg,
 	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
 		return;
 
-	WRITE_ONCE(parent->vmstats->state_local[i], parent->vmstats->state_local[i] + value);
+	atomic_long_add(value, &(parent->vmstats->state_local[i]));
 }
 #endif
 
@@ -1001,7 +1000,7 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)
 	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event))
 		return 0;
 
-	return READ_ONCE(memcg->vmstats->events_local[i]);
+	return atomic_long_read(&(memcg->vmstats->events_local[i]));
 }
 #endif
 
@@ -4126,7 +4125,7 @@ struct aggregate_control {
 	/* pointer to the aggregated (CPU and subtree aggregated) counters */
 	long *aggregate;
 	/* pointer to the non-hierarchichal (CPU aggregated) counters */
-	long *local;
+	atomic_long_t *local;
 	/* pointer to the pending child counters during tree propagation */
 	long *pending;
 	/* pointer to the parent's pending counters, could be NULL */
@@ -4165,7 +4164,7 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac)
 
 		/* Aggregate counts on this level and propagate upwards */
 		if (delta_cpu)
-			ac->local[i] += delta_cpu;
+			atomic_long_add(delta_cpu, &(ac->local[i]));
 
 		if (delta) {
 			ac->aggregate[i] += delta;
Signature · 1 line
2.20.1
author reviewer
#80 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Thu, 15 Jan 2026 18:41:38 +0800 Qi Zheng <[email protected]> wrote:

From: Qi Zheng <[email protected]>

What was "fixed"?

[1/2] fixed the issue reported by https://lore.kernel.org/all/[email protected]/.

Previuosly, state_local was protected by ss_rstat_lock. Since [PATCH v3 28/30] added a concurrent path, [2/2] changed state_local to the atomic_long_t type for protection.

Quoted 3 lines

I queued this and [2/2] as squashable fixes against "mm: memcontrol:
prepare for reparenting state_local".

Thanks!

Quoted 1 line
maintainer
#81 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

In reply to Qi Zheng
Quoted 70 lines · Qi Zheng

From: Qi Zheng <[email protected]>

---
 include/linux/memcontrol.h |  2 --
 mm/memcontrol-v1.c         |  8 --------
 mm/memcontrol-v1.h         |  5 ++++-
 mm/memcontrol.c            | 19 ++++++++-----------
 4 files changed, 12 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +0 -2
index 1fe554eec1e25..e0b84b109b7ac 100644
@@ -944,8 +944,6 @@ bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx);
 
 void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
 void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +0 -8
index 03b924920d6a5..daf9bad8c45ea 100644
@@ -1909,14 +1909,6 @@ void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *pa
 		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
 }
 
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
-	int i;
-
-	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
-		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
-}
-
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 {
 	unsigned long memory, memsw;
mm/memcontrol-v1.h +4 -1
index 45528195d3578..5b1188f3d4173 100644
@@ -75,7 +75,6 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
 
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
 void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
@@ -116,6 +115,10 @@ static inline void memcg1_uncharge_batch(struct mem_cgroup *memcg,
 
 static inline void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) {}
 
+static inline void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
 static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {}
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; }
 static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
mm/memcontrol.c +0 -0
index 7333a37830051..b7b35143d4d2d 100644

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 233

Quoted 10 lines
mm/memcontrol.c +2 -1
@@ -225,13 +225,13 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
 	return objcg;
 }
 
-#ifdef CONFIG_MEMCG_V1
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx);
 static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
 
 static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {

No reparenting local stats for v2.

Quoted 37 lines
mm/memcontrol.c +6 -10
-	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		return;
+	int i;
 
 	synchronize_rcu();
 
@@ -239,13 +239,10 @@ static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgr
 
 	/* The following counts are all non-hierarchical and need to be reparented. */
 	reparent_memcg1_state_local(memcg, parent);
-	reparent_memcg1_lruvec_state_local(memcg, parent);
-}
-#else
-static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
+
+	for (i = 0; i < NR_LRU_LISTS; i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, i);
 }
-#endif
 
 static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {
@@ -510,8 +507,8 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 	return x;
 }
 
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx)
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx)
 {
 	int i = memcg_stats_index(idx);
 	int nid;
Signature · 1 line
2.20.1
author reviewer
#82 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 79 lines · Shakeel Butt
Quoted 78 lines · Qi Zheng

From: Qi Zheng <[email protected]>

---
  include/linux/memcontrol.h |  2 --
  mm/memcontrol-v1.c         |  8 --------
  mm/memcontrol-v1.h         |  5 ++++-
  mm/memcontrol.c            | 19 ++++++++-----------
  4 files changed, 12 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +0 -2
index 1fe554eec1e25..e0b84b109b7ac 100644
@@ -944,8 +944,6 @@ bool memcg_vm_event_item_valid(enum vm_event_item idx);
  unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
  unsigned long lruvec_page_state_local(struct lruvec *lruvec,
  				      enum node_stat_item idx);
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx);
 
  void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
  void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +0 -8
index 03b924920d6a5..daf9bad8c45ea 100644
@@ -1909,14 +1909,6 @@ void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *pa
  		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
  }
 
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
-	int i;
-
-	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
-		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
-}
-
  void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
  {
  	unsigned long memory, memsw;
mm/memcontrol-v1.h +4 -1
index 45528195d3578..5b1188f3d4173 100644
@@ -75,7 +75,6 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
 
  void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
  void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
  void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
  static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
@@ -116,6 +115,10 @@ static inline void memcg1_uncharge_batch(struct mem_cgroup *memcg,
 
  static inline void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) {}
 
+static inline void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
  static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {}
  static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; }
  static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
mm/memcontrol.c +2 -1
index 7333a37830051..b7b35143d4d2d 100644
@@ -225,13 +225,13 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
  	return objcg;
  }
 
-#ifdef CONFIG_MEMCG_V1
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx);
  static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
 
Quoted 4 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 233

Quoted 2 lines
mm/memcontrol.c +0 -0
  static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
  {

No reparenting local stats for v2.

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Quoted 38 lines
Quoted 38 lines
mm/memcontrol.c +6 -10
-	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		return;
+	int i;
 
  	synchronize_rcu();
 
@@ -239,13 +239,10 @@ static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgr
 
  	/* The following counts are all non-hierarchical and need to be reparented. */
  	reparent_memcg1_state_local(memcg, parent);
-	reparent_memcg1_lruvec_state_local(memcg, parent);
-}
-#else
-static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
+
+	for (i = 0; i < NR_LRU_LISTS; i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, i);
  }
-#endif
 
  static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
  {
@@ -510,8 +507,8 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
  	return x;
  }
 
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-				       struct mem_cgroup *parent, int idx)
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+					      struct mem_cgroup *parent, int idx)
  {
  	int i = memcg_stats_index(idx);
  	int nid;
Signature · 1 line
2.20.1
reviewer
#83 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 82 lines · Qi Zheng
Quoted 66 lines · Shakeel Butt
Quoted 65 lines · Qi Zheng

From: Qi Zheng <[email protected]>

---
 include/linux/memcontrol.h |  2 --
 mm/memcontrol-v1.c         |  8 --------
 mm/memcontrol-v1.h         |  5 ++++-
 mm/memcontrol.c            | 19 ++++++++-----------
 4 files changed, 12 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +0 -2
index 1fe554eec1e25..e0b84b109b7ac 100644
@@ -944,8 +944,6 @@ bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
        enum node_stat_item idx);
-void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-        struct mem_cgroup *parent, int idx);
   void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
 void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +0 -7
index 03b924920d6a5..daf9bad8c45ea 100644
@@ -1909,14 +1909,6 @@ void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *pa
  reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
 }
 -void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
- reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
-}
-
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 {
  unsigned long memory, memsw;
mm/memcontrol-v1.h +3 -1
index 45528195d3578..5b1188f3d4173 100644
@@ -75,7 +75,6 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
   void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
 void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
-void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
   void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
@@ -116,6 +115,10 @@ static inline void memcg1_uncharge_batch(struct mem_cgroup *memcg,
   static inline void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) {}
 +static inline void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+
 static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {}
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; }
 static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
mm/memcontrol.c +0 -0
index 7333a37830051..b7b35143d4d2d 100644
@@ -225,13 +225,13 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
  return objcg;
Quoted 8 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 233

Quoted 7 lines
mm/memcontrol.c +2 -0
 }
 -#ifdef CONFIG_MEMCG_V1
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+       struct mem_cgroup *parent, int idx);
 static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
   static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {

No reparenting local stats for v2.

Quoted 2 lines

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Here we might need to elaborate a bit. Specifically, in the function count_shadow_nodes, the use of lruvec_page_state_local to obtain LRU and SLAB pages seems to also require these logics to work correctly. For SLAB, it appears that the statistics here have already been problematic for a while since SLAB pages have been reparented, right?

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
mm/memcontrol.c +6 -9
- if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
- return;
+ int i;
    synchronize_rcu();
 @@ -239,13 +239,10 @@ static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgr
    /* The following counts are all non-hierarchical and need to be reparented. */
  reparent_memcg1_state_local(memcg, parent);
- reparent_memcg1_lruvec_state_local(memcg, parent);
-}
-#else
-static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
-{
+
+ for (i = 0; i < NR_LRU_LISTS; i++)
+ reparent_memcg_lruvec_state_local(memcg, parent, i);
 }
-#endif
   static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgroup *parent)
 {
@@ -510,8 +507,8 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
  return x;
 }
 -void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
-        struct mem_cgroup *parent, int idx)
+static void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+       struct mem_cgroup *parent, int idx)
 {
  int i = memcg_stats_index(idx);
  int nid;
Signature · 1 line
2.20.1
maintainer
#84 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 4 lines · Muchun Song
Quoted 4 lines
Quoted 3 lines

No reparenting local stats for v2.

Quoted 9 lines
Quoted 2 lines

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Here we might need to elaborate a bit. Specifically, in the function count_shadow_nodes, the use of lruvec_page_state_local to obtain LRU and SLAB pages seems to also require these logics to work correctly. For SLAB, it appears that the statistics here have already been problematic for a while since SLAB pages have been reparented, right?

Thanks a lot, now it is clear and yes it seems like SLAB is problematic but now I am wondering if it is really worth fixing. For LRU pages, how about using lruvec_lru_size() defined in vmscan.c. That would at least keep count_shadow_nodes() working irrespective of LRU reparenting.

author reviewer
#85 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 15 lines · Shakeel Butt
Quoted 4 lines · Muchun Song
Quoted 4 lines
Quoted 3 lines

No reparenting local stats for v2.

Quoted 9 lines
Quoted 2 lines

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Here we might need to elaborate a bit. Specifically, in the function count_shadow_nodes, the use of lruvec_page_state_local to obtain LRU and SLAB pages seems to also require these logics to work correctly. For SLAB, it appears that the statistics here have already been problematic for a while since SLAB pages have been reparented, right?

Quoted 4 lines

Thanks a lot, now it is clear and yes it seems like SLAB is problematic but now I am wondering if it is really worth fixing. For LRU pages, how about using lruvec_lru_size() defined in vmscan.c. That would at least keep count_shadow_nodes() working irrespective of LRU reparenting.

Do you mean calling lruvec_lru_size() in count_shadow_nodes()? But numa_stat interface also reads lruvec_stats->state and make it visible to the user.

Quoted 1 line
maintainer
#86 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 18 lines · Qi Zheng
Quoted 15 lines · Shakeel Butt
Quoted 13 lines · Muchun Song
Quoted 6 lines
Quoted 3 lines

No reparenting local stats for v2.

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Here we might need to elaborate a bit. Specifically, in the function count_shadow_nodes, the use of lruvec_page_state_local to obtain LRU and SLAB pages seems to also require these logics to work correctly. For SLAB, it appears that the statistics here have already been problematic for a while since SLAB pages have been reparented, right?

Quoted 6 lines
Quoted 4 lines

Thanks a lot, now it is clear and yes it seems like SLAB is problematic but now I am wondering if it is really worth fixing. For LRU pages, how about using lruvec_lru_size() defined in vmscan.c. That would at least keep count_shadow_nodes() working irrespective of LRU reparenting.

Do you mean calling lruvec_lru_size() in count_shadow_nodes()?

Yes but I am mainly brainstorming. We can keep the reparenting local stats for both v1 and v2 for now as it is not a performance critical path. I am more worried about the stats update path where upward traversal of memcg for CSS_DYING can be costly and I don't want that in v2.

Quoted 4 lines

But
numa_stat interface also reads lruvec_stats->state and make it visible to the user.

Not sure how this is relevant.

author reviewer
#87 lore raw

Re: [PATCH v3 28/30 fix 1/2] mm: memcontrol: fix lruvec_stats->state_local reparenting

Quoted 19 lines · Shakeel Butt
Quoted 18 lines · Qi Zheng
Quoted 15 lines · Shakeel Butt
Quoted 13 lines · Muchun Song
Quoted 6 lines
Quoted 3 lines

No reparenting local stats for v2.

It seems that lruvec_stats->state_local (non-hierarchical) needs to be relocated in both v1 and v2.

Here we might need to elaborate a bit. Specifically, in the function count_shadow_nodes, the use of lruvec_page_state_local to obtain LRU and SLAB pages seems to also require these logics to work correctly. For SLAB, it appears that the statistics here have already been problematic for a while since SLAB pages have been reparented, right?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Thanks a lot, now it is clear and yes it seems like SLAB is problematic but now I am wondering if it is really worth fixing. For LRU pages, how about using lruvec_lru_size() defined in vmscan.c. That would at least keep count_shadow_nodes() working irrespective of LRU reparenting.

Do you mean calling lruvec_lru_size() in count_shadow_nodes()?

Yes but I am mainly brainstorming. We can keep the reparenting local

OK, I will take a closer look.

Quoted 11 lines

stats for both v1 and v2 for now as it is not a performance critical path. I am more worried about the stats update path where upward traversal of memcg for CSS_DYING can be costly and I don't want that in v2.

Quoted 4 lines

But
numa_stat interface also reads lruvec_stats->state and make it visible to the user.

Not sure how this is relevant.

My mistake, please ignore it.

Thanks!

maintainer
#88 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

In reply to Qi Zheng
Quoted 11 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

Also move the upward traversal code in mod_memcg_state() and mod_memcg_lruvec_state() you added in later patch to this patch and make it under CONFIG_MEMCG_V1.

Something like:

#ifdef CONFIG_MEMCG_V1
	while (memcg_is_dying(memcg))
		memcg = parent_mem_cgroup(memcg);

#endif

Quoted 85 lines
---
 include/linux/memcontrol.h |  4 +++
 mm/memcontrol-v1.c         | 16 +++++++++++
 mm/memcontrol-v1.h         |  3 ++
 mm/memcontrol.c            | 56 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 79 insertions(+)
include/linux/memcontrol.h +4 -0
index 26c3c0e375f58..f84a23f13ffb4 100644
@@ -963,12 +963,16 @@ static inline void mod_memcg_page_state(struct page *page,
 
 unsigned long memcg_events(struct mem_cgroup *memcg, int event);
 unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx);
+void reparent_memcg_state_local(struct mem_cgroup *memcg,
+				struct mem_cgroup *parent, int idx);
 unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
 bool memcg_stat_item_valid(int idx);
 bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
+void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+				       struct mem_cgroup *parent, int idx);
 
 void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
 void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +16 -0
index f0ef650d2317b..800606135e7ba 100644
@@ -1898,6 +1898,22 @@ static const unsigned int memcg1_events[] = {
 	PGMAJFAULT,
 };
 
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
+}
+
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
+}
+
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 {
 	unsigned long memory, memsw;
mm/memcontrol-v1.h +3 -0
index eb3c3c1056574..45528195d3578 100644
@@ -41,6 +41,7 @@ static inline bool do_memsw_account(void)
 
 unsigned long memcg_events_local(struct mem_cgroup *memcg, int event);
 unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx);
+void mod_memcg_page_state_local(struct mem_cgroup *memcg, int idx, unsigned long val);
 unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item);
 bool memcg1_alloc_events(struct mem_cgroup *memcg);
 void memcg1_free_events(struct mem_cgroup *memcg);
@@ -73,6 +74,8 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
 			   unsigned long nr_memory, int nid);
 
 void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
 void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
 static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
mm/memcontrol.c +0 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
 	return objcg;
 }

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 236

Quoted 10 lines
mm/memcontrol.c +9 -0
 
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

kernel/cgroup/cgroup.c +4 -4
index e717208cfb18..549a8e026194 100644
@@ -6046,8 +6046,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode)
  */
 static void css_killed_work_fn(struct work_struct *work)
 {
-	struct cgroup_subsys_state *css =
-		container_of(work, struct cgroup_subsys_state, destroy_work);
+	struct cgroup_subsys_state *css = container_of(to_rcu_work(work),
+				 struct cgroup_subsys_state, destroy_rwork);
 
 	cgroup_lock();
 
@@ -6068,8 +6068,8 @@ static void css_killed_ref_fn(struct percpu_ref *ref)
 		container_of(ref, struct cgroup_subsys_state, refcnt);
 
 	if (atomic_dec_and_test(&css->online_cnt)) {
-		INIT_WORK(&css->destroy_work, css_killed_work_fn);
-		queue_work(cgroup_offline_wq, &css->destroy_work);
+		INIT_RCU_WORK(&css->destroy_rwork, css_killed_work_fn);
+		queue_rcu_work(cgroup_offline_wq, &css->destroy_rwork);
 	}
 }
 
Quoted 13 lines
mm/memcontrol.c +13 -0
+
+	__mem_cgroup_flush_stats(memcg, true);
+
+	/* The following counts are all non-hierarchical and need to be reparented. */
+	reparent_memcg1_state_local(memcg, parent);
+	reparent_memcg1_lruvec_state_local(memcg, parent);
+}
+#else
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+#endif
+
author reviewer
#89 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 7 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

Quoted 8 lines
Quoted 5 lines

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

OK, will do.

Quoted 11 lines

Also move the upward traversal code in mod_memcg_state() and mod_memcg_lruvec_state() you added in later patch to this patch and make it under CONFIG_MEMCG_V1.

Something like:

#ifdef CONFIG_MEMCG_V1
	while (memcg_is_dying(memcg))
		memcg = parent_mem_cgroup(memcg);

#endif

OK, will do.

Quoted 88 lines
Quoted 88 lines
---
  include/linux/memcontrol.h |  4 +++
  mm/memcontrol-v1.c         | 16 +++++++++++
  mm/memcontrol-v1.h         |  3 ++
  mm/memcontrol.c            | 56 ++++++++++++++++++++++++++++++++++++++
  4 files changed, 79 insertions(+)
include/linux/memcontrol.h +4 -0
index 26c3c0e375f58..f84a23f13ffb4 100644
@@ -963,12 +963,16 @@ static inline void mod_memcg_page_state(struct page *page,
 
  unsigned long memcg_events(struct mem_cgroup *memcg, int event);
  unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx);
+void reparent_memcg_state_local(struct mem_cgroup *memcg,
+				struct mem_cgroup *parent, int idx);
  unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
  bool memcg_stat_item_valid(int idx);
  bool memcg_vm_event_item_valid(enum vm_event_item idx);
  unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
  unsigned long lruvec_page_state_local(struct lruvec *lruvec,
  				      enum node_stat_item idx);
+void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+				       struct mem_cgroup *parent, int idx);
 
  void mem_cgroup_flush_stats(struct mem_cgroup *memcg);
  void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg);
mm/memcontrol-v1.c +16 -0
index f0ef650d2317b..800606135e7ba 100644
@@ -1898,6 +1898,22 @@ static const unsigned int memcg1_events[] = {
  	PGMAJFAULT,
  };
 
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_state_local(memcg, parent, memcg1_stats[i]);
+}
+
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++)
+		reparent_memcg_lruvec_state_local(memcg, parent, memcg1_stats[i]);
+}
+
  void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
  {
  	unsigned long memory, memsw;
mm/memcontrol-v1.h +3 -0
index eb3c3c1056574..45528195d3578 100644
@@ -41,6 +41,7 @@ static inline bool do_memsw_account(void)
 
  unsigned long memcg_events_local(struct mem_cgroup *memcg, int event);
  unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx);
+void mod_memcg_page_state_local(struct mem_cgroup *memcg, int idx, unsigned long val);
  unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item);
  bool memcg1_alloc_events(struct mem_cgroup *memcg);
  void memcg1_free_events(struct mem_cgroup *memcg);
@@ -73,6 +74,8 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
  			   unsigned long nr_memory, int nid);
 
  void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s);
+void reparent_memcg1_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+void reparent_memcg1_lruvec_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent);
 
  void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
  static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
mm/memcontrol.c +0 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
  	return objcg;
  }
 
Quoted 13 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 236

Quoted 9 lines
mm/memcontrol.c +9 -0
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

Quoted 3 lines

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

It does look much simpler, will do.

Thanks,
Qi

Quoted 40 lines
kernel/cgroup/cgroup.c +4 -4
index e717208cfb18..549a8e026194 100644
@@ -6046,8 +6046,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode)
   */
  static void css_killed_work_fn(struct work_struct *work)
  {
-	struct cgroup_subsys_state *css =
-		container_of(work, struct cgroup_subsys_state, destroy_work);
+	struct cgroup_subsys_state *css = container_of(to_rcu_work(work),
+				 struct cgroup_subsys_state, destroy_rwork);
 
  	cgroup_lock();
 
@@ -6068,8 +6068,8 @@ static void css_killed_ref_fn(struct percpu_ref *ref)
  		container_of(ref, struct cgroup_subsys_state, refcnt);
 
  	if (atomic_dec_and_test(&css->online_cnt)) {
-		INIT_WORK(&css->destroy_work, css_killed_work_fn);
-		queue_work(cgroup_offline_wq, &css->destroy_work);
+		INIT_RCU_WORK(&css->destroy_rwork, css_killed_work_fn);
+		queue_rcu_work(cgroup_offline_wq, &css->destroy_rwork);
  	}
  }
 
Quoted 13 lines
mm/memcontrol.c +13 -0
+
+	__mem_cgroup_flush_stats(memcg, true);
+
+	/* The following counts are all non-hierarchical and need to be reparented. */
+	reparent_memcg1_state_local(memcg, parent);
+	reparent_memcg1_lruvec_state_local(memcg, parent);
+}
+#else
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
+}
+#endif
+
#90 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 42 lines · Qi Zheng
Quoted 7 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

Quoted 8 lines
Quoted 5 lines

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

OK, will do.

Quoted 12 lines
Quoted 12 lines
mm/memcontrol.c +5 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
  	return objcg;
  }
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
Quoted 8 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 235

Quoted 4 lines
mm/memcontrol.c +4 -0
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Quoted 2 lines

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

What happens if someone grabbed a refcount and then release the rcu read lock before percpu refkill and then call mod_memcg[_lruvec]_state()?

In this case, can we end up reparenting in the middle of non-hierarchical stat update because they don't have RCU grace period?

Something like

T1 T2

- rcu_read_lock()
- get memcg refcnt
- rcu_read_unlock()

- call mod_memcg_state()
- CSS_IS_DYING is not set
- Set CSS_IS_DYING
- Trigger percpu refkill

- Trigger offline_css()
-> reparent non-hierarchical - update non-hierarchical stats
stats
- put memcg refcount

Quoted 7 lines
Quoted 2 lines

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

It does look much simpler, will do.

Thanks,
Qi

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#91 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 65 lines · Harry Yoo
Quoted 42 lines · Qi Zheng
Quoted 7 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

Quoted 8 lines
Quoted 5 lines

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

OK, will do.

Quoted 12 lines
Quoted 12 lines
mm/memcontrol.c +5 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
   	return objcg;
   }
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
Quoted 8 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 235

Quoted 4 lines
mm/memcontrol.c +4 -0
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Quoted 2 lines

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

What happens if someone grabbed a refcount and then release the rcu read lock before percpu refkill and then call mod_memcg[_lruvec]_state()?

In this case, can we end up reparenting in the middle of non-hierarchical stat update because they don't have RCU grace period?

Something like

T1 T2

- rcu_read_lock()
- get memcg refcnt
- rcu_read_unlock()

- call mod_memcg_state()
- CSS_IS_DYING is not set
- Set CSS_IS_DYING
- Trigger percpu refkill

Quoted 4 lines

- Trigger offline_css()
-> reparent non-hierarchical - update non-hierarchical stats
stats
- put memcg refcount

Good catch, I think you are right.

The rcu lock should be added to mod_memcg_state() and mod_memcg_lruvec_state().

I will update to v4 as soon as possible.

Thanks,
Qi

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

It does look much simpler, will do.

Thanks,
Qi

#92 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 73 lines · Qi Zheng
Quoted 65 lines · Harry Yoo
Quoted 42 lines · Qi Zheng
Quoted 7 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

Quoted 8 lines
Quoted 5 lines

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

OK, will do.

Quoted 12 lines
Quoted 12 lines
mm/memcontrol.c +5 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
   	return objcg;
   }
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
Quoted 8 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 235

Quoted 4 lines
mm/memcontrol.c +4 -0
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Quoted 2 lines

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

What happens if someone grabbed a refcount and then release the rcu read lock before percpu refkill and then call mod_memcg[_lruvec]_state()?

In this case, can we end up reparenting in the middle of non-hierarchical stat update because they don't have RCU grace period?

Something like

T1 T2

- rcu_read_lock()
- get memcg refcnt
- rcu_read_unlock()

- call mod_memcg_state()
- CSS_IS_DYING is not set
- Set CSS_IS_DYING
- Trigger percpu refkill

Quoted 4 lines

- Trigger offline_css()
-> reparent non-hierarchical - update non-hierarchical stats
stats
- put memcg refcount

Quoted 4 lines

Good catch, I think you are right.

The rcu lock should be added to mod_memcg_state() and mod_memcg_lruvec_state().

Thanks for confirming!

Because it's quite confusing, let me ask few more questions...

Q1. Yosry mentioned [1] [2] that stat updates should be done in the same RCU section that is used to grab a refcount of the cgroup.

But I don't think your work is relying on that. Instead, I guess, it's relying on the CSS_DYING check from reader side to determine whether it should update stats of the child or parent memcg, right?

-> That being said, when rcu_read_lock() is called _after_ stats are
reparented, the reader must observe that the CSS_DYING flag is set.

[1] https://lore.kernel.org/all/utl6esq7jz5e4f7kwgrpwdjc2rm3yi33ljb6xkm5nxzufa4o7s@hblq2piu3vnz
[2] https://lore.kernel.org/all/ebdhvcwygvnfejai5azhg3sjudsjorwmlcvmzadpkhexoeq3tb@5gj5y2exdhpn

Q2. When a reader checks CSS_DYING flag, how is the flag change guaranteed to be visible to the reader without any lock, memory barrier, or atomic ops involved?

As Shakeel mentioned elsewhere, I hope some explanations for correctness to be included in the commit message :)

Quoted 1 line

I will update to v4 as soon as possible.

Thanks a lot!

I'll wait for that and will review carefully to make sure it's correct ;)

Quoted 10 lines

Thanks,
Qi

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

It does look much simpler, will do.

Thanks,
Qi

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#93 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 81 lines · Harry Yoo
Quoted 73 lines · Qi Zheng
Quoted 65 lines · Harry Yoo
Quoted 42 lines · Qi Zheng
Quoted 7 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng

From: Qi Zheng <[email protected]>

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. The following counts are all non-hierarchical and need to be reparented to prevent the counts of parent memcg overflow.

Quoted 8 lines
Quoted 5 lines

1. memcg->vmstats->state_local[i]
2. pn->lruvec_stats->state_local[i]

This commit implements the specific function, which will be used during the reparenting process.

Please add more explanation which was discussed in the email chain at
https://lore.kernel.org/all/5dsb6q2r4xsi24kk5gcnckljuvgvvp6nwifwvc4wuho5hsifeg@5ukg2dq6ini5/

OK, will do.

Quoted 12 lines
Quoted 12 lines
mm/memcontrol.c +5 -0
index 70583394f421f..7aa32b97c9f17 100644
@@ -225,6 +225,28 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc
    	return objcg;
    }
+#ifdef CONFIG_MEMCG_V1
+static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force);
+
+static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)
+{
Quoted 8 lines

mm/memcontrol.c static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memc · line 235

Quoted 4 lines
mm/memcontrol.c +4 -0
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return;
+
+	synchronize_rcu();

Hmm synchrinuze_rcu() is a heavy hammer here. Also you would need rcu read lock in mod_memcg_state() & mod_memcg_lruvec_state() for this synchronize_rcu().

Quoted 2 lines

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

What happens if someone grabbed a refcount and then release the rcu read lock before percpu refkill and then call mod_memcg[_lruvec]_state()?

In this case, can we end up reparenting in the middle of non-hierarchical stat update because they don't have RCU grace period?

Something like

T1 T2

- rcu_read_lock()
- get memcg refcnt
- rcu_read_unlock()

- call mod_memcg_state()
- CSS_IS_DYING is not set
- Set CSS_IS_DYING
- Trigger percpu refkill

Quoted 4 lines

- Trigger offline_css()
-> reparent non-hierarchical - update non-hierarchical stats
stats
- put memcg refcount

Quoted 4 lines

Good catch, I think you are right.

The rcu lock should be added to mod_memcg_state() and mod_memcg_lruvec_state().

Thanks for confirming!

Quoted 7 lines

Because it's quite confusing, let me ask few more questions...

Q1. Yosry mentioned [1] [2] that stat updates should be done in the same RCU section that is used to grab a refcount of the cgroup.

But I don't think your work is relying on that. Instead, I guess, it's relying on the CSS_DYING check from reader side to determine whether it

Only relying the CSS_DYING check is insufficient. Otherwise, the following race may occur:

T1 T2

memcg_is_dying is false
Set CSS_IS_DYING
reparent non-hierarchical update non-hierarchical stats for child

So IIUC we should add rcu lock, then:

T1 T2

rcu_read_lock memcg_is_dying is false

Set CSS_IS_DYING
update non-hierarchical stats for child rcu_read_unlock

synchronize_rcu or rcu work
--> reparent non-hierarchical

Quoted 11 lines

should update stats of the child or parent memcg, right?

-> That being said, when rcu_read_lock() is called _after_ stats are
reparented, the reader must observe that the CSS_DYING flag is set.

[1] https://lore.kernel.org/all/utl6esq7jz5e4f7kwgrpwdjc2rm3yi33ljb6xkm5nxzufa4o7s@hblq2piu3vnz
[2] https://lore.kernel.org/all/ebdhvcwygvnfejai5azhg3sjudsjorwmlcvmzadpkhexoeq3tb@5gj5y2exdhpn

Q2. When a reader checks CSS_DYING flag, how is the flag change guaranteed to be visible to the reader without any lock, memory barrier, or atomic ops involved?

The main situation requiring CSS_DYING check is as follow:

T1 T2

Set CSS_IS_DYING

synchronize_rcu or rcu work
--> reparent non-hierarchical
rcu_read_lock() memcg_is_dying is true update non-hierarchical stats for parent

Referring to the "Memory-Barrier Guarantees" section in [1], synchronize_rcu() can guarantee that T2 can see CSS_IS_DYING. Right?

[1]. https://www.kernel.org/doc/Documentation/RCU/Design/Requirements/Requirements.rst

Thanks,
Qi

Quoted 21 lines

As Shakeel mentioned elsewhere, I hope some explanations for correctness to be included in the commit message :)

Quoted 1 line

I will update to v4 as soon as possible.

Thanks a lot!

I'll wait for that and will review carefully to make sure it's correct ;)

Quoted 10 lines

Thanks,
Qi

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hmm instead of synchronize_rcu() here, we can use queue_rcu_work() in css_killed_ref_fn(). It would be as simple as the following:

It does look much simpler, will do.

Thanks,
Qi

#94 lore raw

Re: [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local

Quoted 51 lines · Qi Zheng
Quoted 39 lines · Harry Yoo
Quoted 31 lines · Qi Zheng
Quoted 25 lines · Harry Yoo
Quoted 4 lines · Qi Zheng
Quoted 1 line · Shakeel Butt

On Wed, Jan 14, 2026 at 07:32:55PM +0800, Qi Zheng wrote:

Since these two functions require memcg or lruvec, they are already within the critical section of the RCU lock.

What happens if someone grabbed a refcount and then release the rcu read lock before percpu refkill and then call mod_memcg[_lruvec]_state()?

In this case, can we end up reparenting in the middle of non-hierarchical stat update because they don't have RCU grace period?

Something like

T1 T2

- rcu_read_lock()
- get memcg refcnt
- rcu_read_unlock()

- call mod_memcg_state()
- CSS_IS_DYING is not set
- Set CSS_IS_DYING
- Trigger percpu refkill

Quoted 4 lines

- Trigger offline_css()
-> reparent non-hierarchical - update non-hierarchical stats
stats
- put memcg refcount

Quoted 4 lines

Good catch, I think you are right.

The rcu lock should be added to mod_memcg_state() and mod_memcg_lruvec_state().

Thanks for confirming!

Quoted 7 lines

Because it's quite confusing, let me ask few more questions...

Q1. Yosry mentioned [1] [2] that stat updates should be done in the same RCU section that is used to grab a refcount of the cgroup.

But I don't think your work is relying on that. Instead, I guess, it's relying on the CSS_DYING check from reader side to determine whether it

Only relying the CSS_DYING check is insufficient. Otherwise, the following race may occur:

Quoted 7 lines

T1 T2

memcg_is_dying is false
Set CSS_IS_DYING
reparent non-hierarchical update non-hierarchical stats for child

So IIUC we should add rcu lock, then:

Right, RCU here is used to enforce ordering between reparenting non-hierarchical stats and stat updates for child.

Quoted 30 lines

T1 T2

rcu_read_lock memcg_is_dying is false

Set CSS_IS_DYING
update non-hierarchical stats for child rcu_read_unlock

synchronize_rcu or rcu work
--> reparent non-hierarchical

Quoted 11 lines

should update stats of the child or parent memcg, right?

-> That being said, when rcu_read_lock() is called _after_ stats are
reparented, the reader must observe that the CSS_DYING flag is set.

[1] https://lore.kernel.org/all/utl6esq7jz5e4f7kwgrpwdjc2rm3yi33ljb6xkm5nxzufa4o7s@hblq2piu3vnz
[2] https://lore.kernel.org/all/ebdhvcwygvnfejai5azhg3sjudsjorwmlcvmzadpkhexoeq3tb@5gj5y2exdhpn

Q2. When a reader checks CSS_DYING flag, how is the flag change guaranteed to be visible to the reader without any lock, memory barrier, or atomic ops involved?

The main situation requiring CSS_DYING check is as follow:

T1 T2

Set CSS_IS_DYING

Quoted 7 lines

synchronize_rcu or rcu work
--> reparent non-hierarchical
rcu_read_lock() memcg_is_dying is true update non-hierarchical stats for parent

Referring to the "Memory-Barrier Guarantees" section in [1],

Today I learned a little bit about RCU's requirements :)

Quoted 1 line

synchronize_rcu() can guarantee that T2 can see CSS_IS_DYING. Right?

Right. It is fair to assume that a read-side critical section either

1) ends before synchronize_rcu(), in this case, the reader might see CSS_DYING flag or not, but it doesn't matter because this reader ends before reparenting the stats.

2) starts after synchronize_rcu() starts, in this case, all stores before the grace period starts must be propagated to all CPUs before the critical section starts. Thus, such a reader is guaranteed to see the CSS_DYING flag set.

Linux Kernel Memory Model [2] explicitly explains 1) and 2):

Quoted 16 lines

RCU (Read-Copy-Update) is a powerful synchronization mechanism. It rests on two concepts: grace periods and read-side critical sections.

A grace period is the span of time occupied by a call to synchronize_rcu(). A read-side critical section (or just critical section, for short) is a region of code delimited by rcu_read_lock() at the start and rcu_read_unlock() at the end. Critical sections can be nested, although we won't make use of this fact.

As far as memory models are concerned, RCU's main feature is its Grace-Period Guarantee, which states that a critical section can never span a full grace period. In more detail, the Guarantee says:

For any critical section C and any grace period G, at least one of the following statements must hold:

Quoted 7 lines

(1) C ends before G does, and in addition, every store that
propagates to C's CPU before the end of C must propagate to every CPU before G ends.

(2) G starts before C does, and in addition, every store that
propagates to G's CPU before the start of G must propagate to every CPU before C starts.

^ this

[2] https://docs.kernel.org/dev-tools/lkmm/docs/explanation.html

Quoted 2 lines

[1]. https://www.kernel.org/doc/Documentation/RCU/Design/Requirements/Requirements.rst
Qi

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#95 lore raw

PATCH v3 29/30 mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios

In reply to Qi Zheng

From: Muchun Song <[email protected]>

Now that everything is set up, switch folio->memcg_data pointers to objcgs, update the accessors, and execute reparenting on cgroup death.

Finally, folio->memcg_data of LRU folios and kmem folios will always point to an object cgroup pointer. The folio->memcg_data of slab folios will point to an vector of object cgroups.

---
 include/linux/memcontrol.h |  77 +++++----------
 mm/memcontrol-v1.c         |  15 +--
 mm/memcontrol.c            | 195 +++++++++++++++++++++++--------------
 3 files changed, 156 insertions(+), 131 deletions(-)
include/linux/memcontrol.h +25 -52
index f84a23f13ffb4..1fe554eec1e25 100644
@@ -369,9 +369,6 @@ enum objext_flags {
 #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1)
 
 #ifdef CONFIG_MEMCG
-
-static inline bool folio_memcg_kmem(struct folio *folio);
-
 /*
  * After the initialization objcg->memcg is always pointing at
  * a valid memcg, but can be atomically swapped to the parent memcg.
@@ -385,43 +382,19 @@ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg)
 }
 
 /*
- * __folio_memcg - Get the memory cgroup associated with a non-kmem folio
- * @folio: Pointer to the folio.
- *
- * Returns a pointer to the memory cgroup associated with the folio,
- * or NULL. This function assumes that the folio is known to have a
- * proper memory cgroup pointer. It's not safe to call this function
- * against some type of folios, e.g. slab folios or ex-slab folios or
- * kmem folios.
- */
-static inline struct mem_cgroup *__folio_memcg(struct folio *folio)
-{
-	unsigned long memcg_data = folio->memcg_data;
-
-	VM_BUG_ON_FOLIO(folio_test_slab(folio), folio);
-	VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio);
-	VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio);
-
-	return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
-}
-
-/*
- * __folio_objcg - get the object cgroup associated with a kmem folio.
+ * folio_objcg - get the object cgroup associated with a folio.
  * @folio: Pointer to the folio.
  *
  * Returns a pointer to the object cgroup associated with the folio,
  * or NULL. This function assumes that the folio is known to have a
- * proper object cgroup pointer. It's not safe to call this function
- * against some type of folios, e.g. slab folios or ex-slab folios or
- * LRU folios.
+ * proper object cgroup pointer.
  */
-static inline struct obj_cgroup *__folio_objcg(struct folio *folio)
+static inline struct obj_cgroup *folio_objcg(struct folio *folio)
 {
 	unsigned long memcg_data = folio->memcg_data;
 
 	VM_BUG_ON_FOLIO(folio_test_slab(folio), folio);
 	VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio);
-	VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio);
 
 	return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
 }
@@ -435,21 +408,30 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)
  * proper memory cgroup pointer. It's not safe to call this function
  * against some type of folios, e.g. slab folios or ex-slab folios.
  *
- * For a non-kmem folio any of the following ensures folio and memcg binding
- * stability:
+ * For a folio any of the following ensures folio and objcg binding stability:
  *
  * - the folio lock
  * - LRU isolation
  * - exclusive reference
  *
- * For a kmem folio a caller should hold an rcu read lock to protect memcg
- * associated with a kmem folio from being released.
+ * Based on the stable binding of folio and objcg, for a folio any of the
+ * following ensures folio and memcg binding stability:
+ *
+ * - cgroup_mutex
+ * - the lruvec lock
+ *
+ * If the caller only want to ensure that the page counters of memcg are
+ * updated correctly, ensure that the binding stability of folio and objcg
+ * is sufficient.
+ *
+ * Note: The caller should hold an rcu read lock or cgroup_mutex to protect
+ * memcg associated with a folio from being released.
  */
 static inline struct mem_cgroup *folio_memcg(struct folio *folio)
 {
-	if (folio_memcg_kmem(folio))
-		return obj_cgroup_memcg(__folio_objcg(folio));
-	return __folio_memcg(folio);
+	struct obj_cgroup *objcg = folio_objcg(folio);
+
+	return objcg ? obj_cgroup_memcg(objcg) : NULL;
 }
 
 /*
@@ -473,15 +455,10 @@ static inline bool folio_memcg_charged(struct folio *folio)
  * has an associated memory cgroup pointer or an object cgroups vector or
  * an object cgroup.
  *
- * For a non-kmem folio any of the following ensures folio and memcg binding
- * stability:
+ * The page and objcg or memcg binding rules can refer to folio_memcg().
  *
- * - the folio lock
- * - LRU isolation
- * - exclusive reference
- *
- * For a kmem folio a caller should hold an rcu read lock to protect memcg
- * associated with a kmem folio from being released.
+ * A caller should hold an rcu read lock to protect memcg associated with a
+ * page from being released.
  */
 static inline struct mem_cgroup *folio_memcg_check(struct folio *folio)
 {
@@ -490,18 +467,14 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio)
 	 * for slabs, READ_ONCE() should be used here.
 	 */
 	unsigned long memcg_data = READ_ONCE(folio->memcg_data);
+	struct obj_cgroup *objcg;
 
 	if (memcg_data & MEMCG_DATA_OBJEXTS)
 		return NULL;
 
-	if (memcg_data & MEMCG_DATA_KMEM) {
-		struct obj_cgroup *objcg;
-
-		objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
-		return obj_cgroup_memcg(objcg);
-	}
+	objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
 
-	return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
+	return objcg ? obj_cgroup_memcg(objcg) : NULL;
 }
 
 static inline struct mem_cgroup *page_memcg_check(struct page *page)
mm/memcontrol-v1.c +9 -6
index 800606135e7ba..03b924920d6a5 100644
@@ -613,6 +613,7 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg)
 void memcg1_swapout(struct folio *folio, swp_entry_t entry)
 {
 	struct mem_cgroup *memcg, *swap_memcg;
+	struct obj_cgroup *objcg;
 	unsigned int nr_entries;
 	unsigned short oldid;
 
@@ -625,12 +626,13 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
 	if (!do_memsw_account())
 		return;
 
-	memcg = folio_memcg(folio);
-
-	VM_WARN_ON_ONCE_FOLIO(!memcg, folio);
-	if (!memcg)
+	objcg = folio_objcg(folio);
+	VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
+	if (!objcg)
 		return;
 
+	rcu_read_lock();
+	memcg = obj_cgroup_memcg(objcg);
 	/*
 	 * Check if this swap entry is already recorded. This can happen
 	 * when MADV_PAGEOUT is called multiple times on pages that remain
@@ -658,7 +660,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
 	folio_unqueue_deferred_split(folio);
 	folio->memcg_data = 0;
 
-	if (!mem_cgroup_is_root(memcg))
+	if (!obj_cgroup_is_root(objcg))
 		page_counter_uncharge(&memcg->memory, nr_entries);
 
 	if (memcg != swap_memcg) {
@@ -679,7 +681,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
 	preempt_enable_nested();
 	memcg1_check_events(memcg, folio_nid(folio));
 
-	css_put(&memcg->css);
+	rcu_read_unlock();
+	obj_cgroup_put(objcg);
 }
 
 /*
mm/memcontrol.c +122 -73

398 lines not on this page · raw message

Signature · 1 line
2.20.1
author reviewer
#96 lore raw

PATCH v3 30/30 mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers

In reply to Qi Zheng

From: Muchun Song <[email protected]>

We must ensure the folio is deleted from or added to the correct lruvec list. So, add VM_WARN_ON_ONCE_FOLIO() to catch invalid users. The VM_BUG_ON_PAGE() in move_pages_to_lru() can be removed as add_page_to_lru_list() will perform the necessary check.

---
 include/linux/mm_inline.h | 6 ++++++
 mm/vmscan.c               | 1 -
 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/mm_inline.h +6 -0
index fa2d6ba811b53..ad50688d89dba 100644
@@ -342,6 +342,8 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
 {
 	enum lru_list lru = folio_lru_list(folio);
 
+	VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio);
+
 	if (lru_gen_add_folio(lruvec, folio, false))
 		return;
 
@@ -356,6 +358,8 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio)
 {
 	enum lru_list lru = folio_lru_list(folio);
 
+	VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio);
+
 	if (lru_gen_add_folio(lruvec, folio, true))
 		return;
 
@@ -370,6 +374,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio)
 {
 	enum lru_list lru = folio_lru_list(folio);
 
+	VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio);
+
 	if (lru_gen_del_folio(lruvec, folio, false))
 		return;
 
mm/vmscan.c +0 -1
index 6bc8047b7aec5..56714f3bc6f88 100644
@@ -1939,7 +1939,6 @@ static unsigned int move_folios_to_lru(struct list_head *list)
 			continue;
 		}
 
-		VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio);
 		lruvec_add_folio(lruvec, folio);
 		nr_pages = folio_nr_pages(folio);
 		nr_moved += nr_pages;
Signature · 1 line
2.20.1
syzbot Tested-by [email protected] on the v3 series
#97 lore raw
In reply to Qi Zheng
syzbot ci has tested the following series

syzbot ci has tested the following series

[v3] Eliminate Dying Memory Cgroup
https://lore.kernel.org/all/[email protected]
* [PATCH v3 01/30] mm: memcontrol: remove dead code of checking parent memory cgroup
* [PATCH v3 02/30] mm: workingset: use folio_lruvec() in workingset_refault()
* [PATCH v3 03/30] mm: rename unlock_page_lruvec_irq and its variants
* [PATCH v3 04/30] mm: vmscan: prepare for the refactoring the move_folios_to_lru()
* [PATCH v3 05/30] mm: vmscan: refactor move_folios_to_lru()
* [PATCH v3 06/30] mm: memcontrol: allocate object cgroup for non-kmem case
* [PATCH v3 07/30] mm: memcontrol: return root object cgroup for root memory cgroup
* [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()
* [PATCH v3 09/30] buffer: prevent memory cgroup release in folio_alloc_buffers()
* [PATCH v3 10/30] writeback: prevent memory cgroup release in writeback module
* [PATCH v3 11/30] mm: memcontrol: prevent memory cgroup release in count_memcg_folio_events()
* [PATCH v3 12/30] mm: page_io: prevent memory cgroup release in page_io module
* [PATCH v3 13/30] mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
* [PATCH v3 14/30] mm: mglru: prevent memory cgroup release in mglru
* [PATCH v3 15/30] mm: memcontrol: prevent memory cgroup release in mem_cgroup_swap_full()
* [PATCH v3 16/30] mm: workingset: prevent memory cgroup release in lru_gen_eviction()
* [PATCH v3 17/30] mm: thp: prevent memory cgroup release in folio_split_queue_lock{_irqsave}()
* [PATCH v3 18/30] mm: zswap: prevent memory cgroup release in zswap_compress()
* [PATCH v3 19/30] mm: workingset: prevent lruvec release in workingset_refault()
* [PATCH v3 20/30] mm: zswap: prevent lruvec release in zswap_folio_swapin()
* [PATCH v3 21/30] mm: swap: prevent lruvec release in lru_gen_clear_refs()
* [PATCH v3 22/30] mm: workingset: prevent lruvec release in workingset_activation()
* [PATCH v3 23/30] mm: do not open-code lruvec lock
* [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
* [PATCH v3 25/30] mm: vmscan: prepare for reparenting traditional LRU folios
* [PATCH v3 26/30] mm: vmscan: prepare for reparenting MGLRU folios
* [PATCH v3 27/30] mm: memcontrol: refactor memcg_reparent_objcgs()
* [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local
* [PATCH v3 29/30] mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios
* [PATCH v3 30/30] mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers

and found the following issue:

UBSAN: array-index-out-of-bounds in reparent_memcg_lruvec_state_local

Full report is available here:
https://ci.syzbot.org/series/45c0b58d-255a-4579-9880-497bdbd4fb99

***

UBSAN: array-index-out-of-bounds in reparent_memcg_lruvec_state_local
tree:      linux-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next
base:      b775e489bec70895b7ef6b66927886bbac79598f
arch:      amd64
compiler:  Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
config:    https://ci.syzbot.org/builds/4d8819ab-0f94-42e8-bd70-87c7e83c37d2/config
syz repro: https://ci.syzbot.org/findings/7850f5dd-4ac7-4b74-85ff-a75ddddebbee/syz_repro
------------[ cut here ]------------
UBSAN: array-index-out-of-bounds in mm/memcontrol.c:530:3
index 33 is out of range for type 'long[33]'
CPU: 1 UID: 0 PID: 31 Comm: kworker/1:1 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: cgroup_offline css_killed_work_fn
Call Trace:
 <TASK>
 dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
 ubsan_epilogue+0xa/0x30 lib/ubsan.c:233
 __ubsan_handle_out_of_bounds+0xe8/0xf0 lib/ubsan.c:455
 reparent_memcg_lruvec_state_local+0x34f/0x460 mm/memcontrol.c:530
 reparent_memcg1_lruvec_state_local+0xa7/0xc0 mm/memcontrol-v1.c:1917
 reparent_state_local mm/memcontrol.c:242 [inline]
 memcg_reparent_objcgs mm/memcontrol.c:299 [inline]
 mem_cgroup_css_offline+0xc7c/0xc90 mm/memcontrol.c:4054
 offline_css kernel/cgroup/cgroup.c:5760 [inline]
 css_killed_work_fn+0x12f/0x570 kernel/cgroup/cgroup.c:6055
 process_one_work+0x949/0x15a0 kernel/workqueue.c:3279
 process_scheduled_works kernel/workqueue.c:3362 [inline]
 worker_thread+0x9af/0xee0 kernel/workqueue.c:3443
 kthread+0x388/0x470 kernel/kthread.c:467
 ret_from_fork+0x51b/0xa40 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
 </TASK>
---[ end trace ]---
Kernel panic - not syncing: UBSAN: panic_on_warn set ...
CPU: 1 UID: 0 PID: 31 Comm: kworker/1:1 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: cgroup_offline css_killed_work_fn
Call Trace:
 <TASK>
 vpanic+0x1e0/0x670 kernel/panic.c:490
 panic+0xc5/0xd0 kernel/panic.c:627
 check_panic_on_warn+0x89/0xb0 kernel/panic.c:377
 __ubsan_handle_out_of_bounds+0xe8/0xf0 lib/ubsan.c:455
 reparent_memcg_lruvec_state_local+0x34f/0x460 mm/memcontrol.c:530
 reparent_memcg1_lruvec_state_local+0xa7/0xc0 mm/memcontrol-v1.c:1917
 reparent_state_local mm/memcontrol.c:242 [inline]
 memcg_reparent_objcgs mm/memcontrol.c:299 [inline]
 mem_cgroup_css_offline+0xc7c/0xc90 mm/memcontrol.c:4054
 offline_css kernel/cgroup/cgroup.c:5760 [inline]
 css_killed_work_fn+0x12f/0x570 kernel/cgroup/cgroup.c:6055
 process_one_work+0x949/0x15a0 kernel/workqueue.c:3279
 process_scheduled_works kernel/workqueue.c:3362 [inline]
 worker_thread+0x9af/0xee0 kernel/workqueue.c:3443
 kthread+0x388/0x470 kernel/kthread.c:467
 ret_from_fork+0x51b/0xa40 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
 </TASK>

Kernel Offset: disabled
Rebooting in 86400 seconds..

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

author reviewer
#98 lore raw
Quoted 53 lines · syzbot ci

syzbot ci has tested the following series

[v3] Eliminate Dying Memory Cgroup
https://lore.kernel.org/all/[email protected]
* [PATCH v3 01/30] mm: memcontrol: remove dead code of checking parent memory cgroup
* [PATCH v3 02/30] mm: workingset: use folio_lruvec() in workingset_refault()
* [PATCH v3 03/30] mm: rename unlock_page_lruvec_irq and its variants
* [PATCH v3 04/30] mm: vmscan: prepare for the refactoring the move_folios_to_lru()
* [PATCH v3 05/30] mm: vmscan: refactor move_folios_to_lru()
* [PATCH v3 06/30] mm: memcontrol: allocate object cgroup for non-kmem case
* [PATCH v3 07/30] mm: memcontrol: return root object cgroup for root memory cgroup
* [PATCH v3 08/30] mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()
* [PATCH v3 09/30] buffer: prevent memory cgroup release in folio_alloc_buffers()
* [PATCH v3 10/30] writeback: prevent memory cgroup release in writeback module
* [PATCH v3 11/30] mm: memcontrol: prevent memory cgroup release in count_memcg_folio_events()
* [PATCH v3 12/30] mm: page_io: prevent memory cgroup release in page_io module
* [PATCH v3 13/30] mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
* [PATCH v3 14/30] mm: mglru: prevent memory cgroup release in mglru
* [PATCH v3 15/30] mm: memcontrol: prevent memory cgroup release in mem_cgroup_swap_full()
* [PATCH v3 16/30] mm: workingset: prevent memory cgroup release in lru_gen_eviction()
* [PATCH v3 17/30] mm: thp: prevent memory cgroup release in folio_split_queue_lock{_irqsave}()
* [PATCH v3 18/30] mm: zswap: prevent memory cgroup release in zswap_compress()
* [PATCH v3 19/30] mm: workingset: prevent lruvec release in workingset_refault()
* [PATCH v3 20/30] mm: zswap: prevent lruvec release in zswap_folio_swapin()
* [PATCH v3 21/30] mm: swap: prevent lruvec release in lru_gen_clear_refs()
* [PATCH v3 22/30] mm: workingset: prevent lruvec release in workingset_activation()
* [PATCH v3 23/30] mm: do not open-code lruvec lock
* [PATCH v3 24/30] mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
* [PATCH v3 25/30] mm: vmscan: prepare for reparenting traditional LRU folios
* [PATCH v3 26/30] mm: vmscan: prepare for reparenting MGLRU folios
* [PATCH v3 27/30] mm: memcontrol: refactor memcg_reparent_objcgs()
* [PATCH v3 28/30] mm: memcontrol: prepare for reparenting state_local
* [PATCH v3 29/30] mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios
* [PATCH v3 30/30] mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers

and found the following issue:

UBSAN: array-index-out-of-bounds in reparent_memcg_lruvec_state_local

Full report is available here:
https://ci.syzbot.org/series/45c0b58d-255a-4579-9880-497bdbd4fb99

***

UBSAN: array-index-out-of-bounds in reparent_memcg_lruvec_state_local
tree:      linux-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next
base:      b775e489bec70895b7ef6b66927886bbac79598f
arch:      amd64
compiler:  Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
config:    https://ci.syzbot.org/builds/4d8819ab-0f94-42e8-bd70-87c7e83c37d2/config
syz repro: https://ci.syzbot.org/findings/7850f5dd-4ac7-4b74-85ff-a75ddddebbee/syz_repro
Quoted 3 lines
------------[ cut here ]------------
UBSAN: array-index-out-of-bounds in mm/memcontrol.c:530:3

index 33 is out of range for type 'long[33]'

Oh, the size of lruvec_stats->state_local is NR_MEMCG_NODE_STAT_ITEMS, but memcg1_stats contains MEMCG_SWAP, which is outside the array range.

It seems that only the following items need to be reparented:

1). NR_LRU_LISTS
2). NR_SLAB_RECLAIMABLE_B + NR_SLAB_UNRECLAIMABLE_B

But for 2), since we reparented the slab page a long time ago, it seems there has always been a problem. So this patchset will only handle 1).

Quoted 60 lines
CPU: 1 UID: 0 PID: 31 Comm: kworker/1:1 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: cgroup_offline css_killed_work_fn
Call Trace:
  <TASK>
  dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
  ubsan_epilogue+0xa/0x30 lib/ubsan.c:233
  __ubsan_handle_out_of_bounds+0xe8/0xf0 lib/ubsan.c:455
  reparent_memcg_lruvec_state_local+0x34f/0x460 mm/memcontrol.c:530
  reparent_memcg1_lruvec_state_local+0xa7/0xc0 mm/memcontrol-v1.c:1917
  reparent_state_local mm/memcontrol.c:242 [inline]
  memcg_reparent_objcgs mm/memcontrol.c:299 [inline]
  mem_cgroup_css_offline+0xc7c/0xc90 mm/memcontrol.c:4054
  offline_css kernel/cgroup/cgroup.c:5760 [inline]
  css_killed_work_fn+0x12f/0x570 kernel/cgroup/cgroup.c:6055
  process_one_work+0x949/0x15a0 kernel/workqueue.c:3279
  process_scheduled_works kernel/workqueue.c:3362 [inline]
  worker_thread+0x9af/0xee0 kernel/workqueue.c:3443
  kthread+0x388/0x470 kernel/kthread.c:467
  ret_from_fork+0x51b/0xa40 arch/x86/kernel/process.c:158
  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
  </TASK>
---[ end trace ]---
Kernel panic - not syncing: UBSAN: panic_on_warn set ...
CPU: 1 UID: 0 PID: 31 Comm: kworker/1:1 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: cgroup_offline css_killed_work_fn
Call Trace:
  <TASK>
  vpanic+0x1e0/0x670 kernel/panic.c:490
  panic+0xc5/0xd0 kernel/panic.c:627
  check_panic_on_warn+0x89/0xb0 kernel/panic.c:377
  __ubsan_handle_out_of_bounds+0xe8/0xf0 lib/ubsan.c:455
  reparent_memcg_lruvec_state_local+0x34f/0x460 mm/memcontrol.c:530
  reparent_memcg1_lruvec_state_local+0xa7/0xc0 mm/memcontrol-v1.c:1917
  reparent_state_local mm/memcontrol.c:242 [inline]
  memcg_reparent_objcgs mm/memcontrol.c:299 [inline]
  mem_cgroup_css_offline+0xc7c/0xc90 mm/memcontrol.c:4054
  offline_css kernel/cgroup/cgroup.c:5760 [inline]
  css_killed_work_fn+0x12f/0x570 kernel/cgroup/cgroup.c:6055
  process_one_work+0x949/0x15a0 kernel/workqueue.c:3279
  process_scheduled_works kernel/workqueue.c:3362 [inline]
  worker_thread+0x9af/0xee0 kernel/workqueue.c:3443
  kthread+0x388/0x470 kernel/kthread.c:467
  ret_from_fork+0x51b/0xa40 arch/x86/kernel/process.c:158
  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
  </TASK>

Kernel Offset: disabled
Rebooting in 86400 seconds..

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

maintainer
#99 lore raw
In reply to Qi Zheng
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

I'll suppress the usual added-to-mm email spray.

author reviewer
#100 lore raw
Quoted 11 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

I'll suppress the usual added-to-mm email spray.

Hi Andrew,

The issue reported by syzbot needs to be addressed. If you want to test this patchset, would you like me to provide a fix patch, or would you prefer me to update to v4?

Thanks,
Qi

maintainer
#101 lore raw
Quoted 13 lines · On Thu, 15 Jan 2026 11:52:04 +0800 Qi Zheng <[email protected]> wrote:
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

Quoted 7 lines
Quoted 1 line

I'll suppress the usual added-to-mm email spray.

Hi Andrew,

The issue reported by syzbot needs to be addressed. If you want to test this patchset, would you like me to provide a fix patch, or would you prefer me to update to v4?

A fix would be preferred if that's reasonable - it's a lot of patches be resending!

author reviewer
#102 lore raw
Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Thu, 15 Jan 2026 11:52:04 +0800 Qi Zheng <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

I'll suppress the usual added-to-mm email spray.

Quoted 8 lines
Quoted 5 lines

Hi Andrew,

The issue reported by syzbot needs to be addressed. If you want to test this patchset, would you like me to provide a fix patch, or would you prefer me to update to v4?

A fix would be preferred if that's reasonable - it's a lot of patches be resending!

OK, I'll send the fix ASAP.

#103 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

I'll suppress the usual added-to-mm email spray.

Since this is so large and we are late on in the cycle can I in this case can I explicitly ask for at least 1 sub-M tag on each commit before queueing for Linus please?

We are seeing kernel bot reports here so let's obviously stabilise this for a while also.

Thanks, Lorenzo

maintainer
#104 lore raw
Quoted 11 lines · On Thu, 15 Jan 2026 12:40:12 +0000 Lorenzo Stoakes…
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

Quoted 5 lines
Quoted 1 line

I'll suppress the usual added-to-mm email spray.

Since this is so large and we are late on in the cycle can I in this case can I explicitly ask for at least 1 sub-M tag on each commit before queueing for Linus please?

Well, kinda.

fs/buffer.c
fs/fs-writeback.c
include/linux/memcontrol.h
include/linux/mm_inline.h
include/linux/mmzone.h
include/linux/swap.h
include/trace/events/writeback.h
mm/compaction.c
mm/huge_memory.c
mm/memcontrol.c
mm/memcontrol-v1.c
mm/memcontrol-v1.h
mm/migrate.c
mm/mlock.c
mm/page_io.c
mm/percpu.c
mm/shrinker.c
mm/swap.c
mm/vmscan.c
mm/workingset.c
mm/zswap.c

That's a lot of reviewers to round up! And there are far worse cases - MM patchsets are often splattered elsewhere. We can't have MM patchsets getting stalled because some video driver developer is on leave or got laid off. Not suggesting that you were really suggesting that!

As this is officially a memcg patch, I'd be looking to memcg maintainers for guidance while viewing acks from others as nice-to-have, rather than must-have.

Quoted 2 lines

We are seeing kernel bot reports here so let's obviously stabilise this for a while also.

Yeah, I'm not feeling optimistic about getting all this into the next merge window. But just one day in mm-new led to David's secret ci-bot discovering a missed rcu_unlock due to a cross-tree integration thing.

I'll keep the series around for at least a few days, see how things progress.

#105 lore raw
Quoted 43 lines · Andrew Morton
Quoted 11 lines · On Thu, 15 Jan 2026 12:40:12 +0000 Lorenzo Stoakes…
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

Quoted 5 lines
Quoted 1 line

I'll suppress the usual added-to-mm email spray.

Since this is so large and we are late on in the cycle can I in this case can I explicitly ask for at least 1 sub-M tag on each commit before queueing for Linus please?

Well, kinda.

fs/buffer.c
fs/fs-writeback.c
include/linux/memcontrol.h
include/linux/mm_inline.h
include/linux/mmzone.h
include/linux/swap.h
include/trace/events/writeback.h
mm/compaction.c
mm/huge_memory.c
mm/memcontrol.c
mm/memcontrol-v1.c
mm/memcontrol-v1.h
mm/migrate.c
mm/mlock.c
mm/page_io.c
mm/percpu.c
mm/shrinker.c
mm/swap.c
mm/vmscan.c
mm/workingset.c
mm/zswap.c

Quoted 5 lines

That's a lot of reviewers to round up! And there are far worse cases - MM patchsets are often splattered elsewhere. We can't have MM patchsets getting stalled because some video driver developer is on leave or got laid off. Not suggesting that you were really suggesting that!

Yeah, obviously judgment needs to be applied in these situations - an 'M' implies community trusts sensible decisions, so since this is really about the cgroup behaviour, I'd say simply requiring at least 1 M per-patch from any of:

M: Johannes Weiner <[email protected]>
M: Michal Hocko <[email protected]>
M: Roman Gushchin <[email protected]>
M: Shakeel Butt <[email protected]>

Suffices.

I am obviously not suggesting that we require sign off from _all_ sub-M's for _all_ affected files, and then some changes may be blurry.

For the most part I think it's usually _fairly_ obvious which part of MAINTAINERS applies, and in cases where it doesn't obviously people can be pinged for opinions.

Quoted 4 lines

As this is officially a memcg patch, I'd be looking to memcg maintainers for guidance while viewing acks from others as nice-to-have, rather than must-have.

Yeah agreed.

Quoted 7 lines
Quoted 3 lines

We are seeing kernel bot reports here so let's obviously stabilise this for a while also.

Yeah, I'm not feeling optimistic about getting all this into the next merge window. But just one day in mm-new led to David's secret ci-bot discovering a missed rcu_unlock due to a cross-tree integration thing.

Yeah and that's not a big deal, things can wait a little while esp. the bigger changes!

Stabilising it is more important :)

Quoted 4 lines

I'll keep the series around for at least a few days, see how things progress.

Sounds sensible!

Cheers, Lorenzo

reviewer
#106 lore raw
Quoted 55 lines · Lorenzo Stoakes
Quoted 43 lines · Andrew Morton
Quoted 11 lines · On Thu, 15 Jan 2026 12:40:12 +0000 Lorenzo Stoakes…
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Wed, 14 Jan 2026 19:26:43 +0800 Qi Zheng <[email protected]> wrote:

This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup.

Thanks. I'll add this to mm.git for testing. A patchset of this magnitude at -rc5 is a little ambitious, but Linus is giving us an rc8 so let's see.

Quoted 5 lines
Quoted 1 line

I'll suppress the usual added-to-mm email spray.

Since this is so large and we are late on in the cycle can I in this case can I explicitly ask for at least 1 sub-M tag on each commit before queueing for Linus please?

Well, kinda.

fs/buffer.c
fs/fs-writeback.c
include/linux/memcontrol.h
include/linux/mm_inline.h
include/linux/mmzone.h
include/linux/swap.h
include/trace/events/writeback.h
mm/compaction.c
mm/huge_memory.c
mm/memcontrol.c
mm/memcontrol-v1.c
mm/memcontrol-v1.h
mm/migrate.c
mm/mlock.c
mm/page_io.c
mm/percpu.c
mm/shrinker.c
mm/swap.c
mm/vmscan.c
mm/workingset.c
mm/zswap.c

Quoted 5 lines

That's a lot of reviewers to round up! And there are far worse cases - MM patchsets are often splattered elsewhere. We can't have MM patchsets getting stalled because some video driver developer is on leave or got laid off. Not suggesting that you were really suggesting that!

Yeah, obviously judgment needs to be applied in these situations - an 'M' implies community trusts sensible decisions, so since this is really about the cgroup behaviour, I'd say simply requiring at least 1 M per-patch from any of:

Quoted 6 lines

M: Johannes Weiner <[email protected]>
M: Michal Hocko <[email protected]>
M: Roman Gushchin <[email protected]>
M: Shakeel Butt <[email protected]>

Suffices.

I have seen a good deal of review feedback from Johannes, Roman and Shakeel (thx!). I have it on my todo list as well but the series is really large and it is not that easy to find time to do the proper review. Anyway, unlike before xmas when there was barely any review and I asked to slow down I feel much more confident just by seeing acks from others memcg maintainers.

That being said, if I fail to find proper time to review myself I am fully confident to rely on other memcg maintainers here. So this should not be blocked waiting for me.

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs