Skip to contentSkip to conversation

mm/mglru: use folio_mark_accessed to replace folio_set_active

PATCH Applied Barry Song (Xiaomi) 1 participant 1 message May 26

Subsystems (4)
lore
Share & tools
lore

Barry Song (Xiaomi)

author reviewer
#0 lore raw

PATCH v3 1/1 mm/mglru: use folio_mark_accessed to replace folio_set_active

MGLRU gives high priority to folios mapped in page tables. As a result, folio_set_active() is invoked for all folios read during page faults. In practice, however, readahead can bring in many folios that are never accessed via page tables.

A previous attempt by Lei Liu proposed introducing a separate LRU for readahead[1] to make readahead pages easier to reclaim, but that approach is likely over-engineered.

Before commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folios with PG_active were always placed in the youngest generation, leading to over-protection and increased refaults. After that commit, PG_active folios are placed in the second youngest generation, which is still too optimistic given the presence of readahead. In contrast, the classic active/inactive scheme is more conservative.

This patch switches to using folio_mark_accessed() and
begins prefaulted file folios from the second oldest
generation instead of active generations.
We should also adjust the following accordingly:
- WORKINGSET_ACTIVATE: aligned with setting active for refaulted workingset folios;
- lru_gen_folio_seq(): place (pre)faulted file folios into the second oldest generation;
- promote second-scanned folios to workingset in folio_check_references(): we now have to depend on folio_lru_refs() > 1, since we previously relied on PG_referenced being set during the first scan, but PG_referenced is now set earlier.

On x86, running a kernel build inside a memcg with a 1GB memory limit using 20 threads.

w/o patch:
real 1m50.764s
user 25m32.305s
sys 4m0.012s
pswpin: 1333245
pswpout: 4366443
pgpgin: 6962592
pgpgout: 17780712
swpout_zero: 1019603
swpin_zero: 14764
refault_file: 287794
refault_anon: 1347963

w/ patch:
real 1m48.879s
user 25m29.224s
sys 3m37.421s
pswpin: 568480
pswpout: 2322657
pgpgin: 4073416
pgpgout: 9613408
swpout_zero: 593275
swpin_zero: 9118
refault_file: 262505
refault_anon: 577550

active/inactive LRU:

real 1m49.928s
user 25m28.196s
sys 3m40.740s
pswpin: 463452
pswpout: 2309119
pgpgin: 4438856
pgpgout: 9568628
swpout_zero: 743704
swpin_zero: 7244
refault_file: 562555
refault_anon: 470694

Lance and Xueyuan made a huge contribution to this patch through testing.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Cc Pedro Falcato <[email protected]>
Cc Kairui Song <[email protected]>
Cc Qi Zheng <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc wangzicheng <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Lei Liu <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Axel Rasmussen <[email protected]>
Cc Yuanchu Xie <[email protected]>
Cc Wei Xu <[email protected]>
Cc Will Deacon <[email protected]>
Cc Kalesh Singh <[email protected]>

---
-v3:
* Fix 2nd pte-scanned promotion in folio_check_references(), per Kairui;
* Restore anon folios behaviour in lru_gen_folio_seq(), per Kairui; -v2:
https://lore.kernel.org/linux-mm/[email protected]/
* Fix WORKINGSET_ACTIVATE - workingset will be set to active during refault;
* Avoid unconditional protecting anon folios in lru_gen_folio_seq();
* Also adjusted workingset set accordingly in folio_check_references(). -v1:
https://lore.kernel.org/linux-mm/[email protected]/
-rfc was:
[PATCH RFC] mm/mglru: lazily activate folios while folios are really mapped
https://lore.kernel.org/linux-mm/[email protected]/

 include/linux/mm_inline.h |  2 +-
 mm/swap.c                 | 16 +++++++++++++---
 mm/vmscan.c               |  6 +++++-
 mm/workingset.c           | 10 ++++++----
 4 files changed, 25 insertions(+), 9 deletions(-)
include/linux/mm_inline.h +1 -1
index a171070e15f0..a8430a7ae054 100644
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
 		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
 		gen = MIN_NR_GENS;
 	else
-		gen = MAX_NR_GENS - folio_test_workingset(folio);
+		gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
 
 	return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
 }
mm/swap.c +13 -3
index 5cc44f0de987..a44829dcde7a 100644
@@ -509,10 +509,20 @@ void folio_add_lru(struct folio *folio)
 			folio_test_unevictable(folio), folio);
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 
-	/* see the comment in lru_gen_folio_seq() */
+	/*
+	 * For refaulted workingset folios, set PG_active so they
+	 * can be added to active generations.
+	 * For prefaulted file folios, folio_mark_accessed() sets
+	 * PG_referenced so lru_gen_folio_seq() places them into
+	 * the second oldest generation.
+	 */
 	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
-	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
-		folio_set_active(folio);
+	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
+		if (folio_test_workingset(folio))
+			folio_set_active(folio);
+		else if (!folio_test_referenced(folio))
+			folio_mark_accessed(folio);
+	}
 
 	folio_batch_add_and_move(folio, lru_add);
 }
mm/vmscan.c +5 -1
index e452cb043d46..745a55a3f7de 100644
@@ -848,7 +848,11 @@ static bool lru_gen_set_refs(struct folio *folio)
 		return false;
 	}
 
-	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+	/* Promote on second access */
+	if (folio_lru_refs(folio) > 1)
+		set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+	else
+		folio_mark_accessed(folio);
 	return true;
 }
 #else
mm/workingset.c +6 -4
index 07e6836d0502..f351798e723a 100644
@@ -319,11 +319,13 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
 
-	/* see folio_add_lru() where folio_set_active() will be called */
-	if (lru_gen_in_fault())
-		mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
-
 	if (workingset) {
+		/*
+		 * see folio_add_lru(), where folio_set_active() is
+		 * called for workingset folios
+		 */
+		if (lru_gen_in_fault())
+			mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
 		folio_set_workingset(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
 	} else
Signature · 1 line
2.34.1