mm: workingset: account MGLRU pages in count_shadow_nodes()

From: Hui Zhu <[email protected]>

Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the number of lru pages") switched count_shadow_nodes() to lruvec_lru_size(). With CONFIG_MEMCG enabled, lruvec_lru_size() reads mz->lru_zone_size, which is only maintained through mem_cgroup_update_lru_size(). MGLRU never goes there: all of its accounting (lru_gen_update_size(), reset_batch_size(), inc_max_seq() and __lru_gen_reparent_memcg()) uses __update_lru_size(), which skips the memcg array, and the classic lruvec_add/del_folio() paths return early once lru_gen_add/del_folio() succeeded.

The four evictable LRU lists are therefore always seen as empty when MGLRU is on, and the shadow node budget (pages >> 3) is based on slab and unevictable pages only. For a memcg holding 1 GiB of page cache and 64 MiB of slab the budget drops from ~35k nodes to ~2k, so the workingset shadow shrinker reclaims eviction tokens almost as fast as they are created. Refaults then find live pages instead of shadow entries, lru_gen_refault() cannot restore the workingset state of hot pages, and thrashing protection is lost. This hits every memcg reclaim and, since the root memcg is iterated as well, global reclaim too.

Fix this by adding the per-generation counters of an enabled multi-gen LRU to the budget. lrugen->nr_pages[] is eventually consistent and may go transiently negative while batched updates are pending, so clamp the sum at zero. Keep lruvec_lru_size() as it still accounts for the unevictable pages.

---
 mm/workingset.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

index f351798e723a..da20034d28fc 100644
@@ -653,6 +653,37 @@ void workingset_update_node(struct xa_node *node)
 	}
 }
 
+#ifdef CONFIG_LRU_GEN
+/*
+ * The per-generation counters are eventually consistent and may go
+ * transiently negative while batched updates are pending, so clamp
+ * the sum at zero.
+ */
+static unsigned long count_lru_gen_pages(struct lruvec *lruvec)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	long nr_pages = 0;
+	int gen, type, zone;
+
+	if (!lrugen->enabled)
+		return 0;
+
+	for (gen = 0; gen < MAX_NR_GENS; gen++) {
+		for (type = 0; type < ANON_AND_FILE; type++) {
+			long *cnt = lrugen->nr_pages[gen][type];
+
+			for (zone = 0; zone < MAX_NR_ZONES; zone++)
+				nr_pages += READ_ONCE(cnt[zone]);
+		}
+	}
+
+	if (nr_pages <= 0)
+		return 0;
+
+	return nr_pages;
+}
+#endif
+
 static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 					struct shrink_control *sc)
 {

@@ -697,6 +728,15 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 		for (pages = 0, i = 0; i < NR_LRU_LISTS; i++)
 			pages += lruvec_lru_size(lruvec, i, MAX_NR_ZONES - 1);
 
+#ifdef CONFIG_LRU_GEN
+		/*
+		 * MGLRU accounts for the evictable pages in
+		 * lrugen->nr_pages[] instead of mz->lru_zone_size, which
+		 * lruvec_lru_size() reads, so the loop above misses them.
+		 */
+		pages += count_lru_gen_pages(lruvec);
+#endif
+
 		pages += lruvec_page_state_local(
 			lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT;
 		pages += lruvec_page_state_local(

2.53.0

Instead of adding more complexity and increasing divergence between classic LRU and MGLRU, let's fix lruvec_page_state_local for all. Usage of lruvec_lru_size instead lruvec_page_state_local was mainly due to concern on adding upward tree traversal on stats update side i.e. get_non_dying_memcg_[start|end].

However offlining should be rate. Let's just use get_non_dying_memcg_[start|end] everywhere and not just v1 and measure the impact.

Shakeel