mm: mglru: fix swappiness behavior

Classic LRU respects swappiness, while MGLRU only weakly honors it when selecting the scan type. The fallback logic can easily override the bias established by get_type_to_scan(). As a result, swappiness has a much smaller overall impact when changing from 1 to 200, as shown by the data presented later in this cover letter.

In this patchset, patches 1-8 from Kairui introduce a model that uses refaults to determine nr_to_scan[] for both file and anon in MGLRU. For each reclaim operation, we calculate how many file and anon folios to scan, similar to CLRU, rather than relying on get_type_to_scan() and fallback mechanisms to decide what to reclaim, which have proven rather unreliable in honoring swappiness. This model also allows scanning and reclaiming from the two youngest generations, including the youngest generation itself. This can help resolve OOM situations[1] and reduce the need for aging.

As Kairui mentioned, the OOM reproducer was demonstrated previously [2], and that series was merged in v7 [3]. The problem was largely alleviated as described, but not completely fixed. Under higher memory pressure, the system could still hit OOM. Before this series, Kairui could still reproduce OOM after hundreds of iterations. With the fixing patch in this series applied, the problem no longer occurs.

Patch 9 from Barry fixes a cold/hot inversion exacerbated by patches 1–8 (which also exists in mainline) and improves reclaim efficiency.

Patches 10-15 from Gaofei and Barry optimize aging based on folio distribution imbalance and improve reclaim behavior at extreme swappiness values such as 1 and 200.

Patch 16 from Bo Zhang addresses the GFP_NOIO issue in MGLRU.

After the patchset, the results track CLRU across the full swappiness range (see table below), with more file scanning at low swappiness and more anon scanning at high swappiness.

pgsteal_file

swappiness   CLRU          MGLRU         MGLRU+patches
1            6,620,680     745,968       6,197,601
36           807,807       508,974       564,256
71           553,144       417,700       420,008
106          402,880       402,474       363,635
141          240,293       337,797       312,624
176          187,682       330,045       238,722
200          0             343,949       0

pgsteal_anon

swappiness   CLRU          MGLRU         MGLRU+patches
1            4,436,199     2,671,934     5,011,932
36           2,858,117     3,032,747     3,039,483
71           3,042,392     2,827,735     3,132,431
106          3,292,622     3,184,314     3,238,658
141          3,612,869     2,999,000     3,306,901
176          3,827,154     3,134,127     3,714,411
200          13,825,283    3,269,753     14,481,445

pgscan_file

swappiness   CLRU          MGLRU         MGLRU+patches
1            11,853,625    877,311       13,108,154
36           1,016,826     582,783       657,973
71           662,823       474,772       457,686
106          496,374       451,285       390,056
141          277,109       382,604       329,819
176          206,261       372,059       246,141
200          0             393,608       0

pgscan_anon

swappiness   CLRU          MGLRU         MGLRU+patches
1            20,804,386    5,847,747     6,936,021
36           17,830,330    6,740,075     5,975,744
71           18,357,293    6,126,194     6,797,243
106          19,938,920    6,431,355     7,602,990
141          19,609,579    5,997,762     8,540,965
176          21,612,749    6,071,032     10,532,510
200          44,776,474    6,544,125     30,296,434

refault_file

swappiness   CLRU          MGLRU         MGLRU+patches
1            6,523,070     667,302       6,118,731
36           722,110       431,585       486,604
71           477,062       340,241       342,746
106          327,610       324,747       286,465
141          161,827       265,959       233,860
176          115,551       258,515       171,174
200          0             268,049       0

refault_anon

swappiness   CLRU          MGLRU         MGLRU+patches
1            867,728       552,829       1,368,563
36           419,635       657,745       636,567
71           536,299       611,345       689,528
106          580,742       693,689       707,227
141          706,151       639,850       753,779
176          706,499       685,086       934,553
200          4,693,850     733,273       5,491,234

With a normal swappiness setting (without extreme bias), I see almost no noticeable performance difference between the two versions.

For example, with swappiness=60, the average results over 5 rounds are:

metric          W/o patch     W/ patch
real            1m49.273s     1m48.836s
user            25m33.186s    25m33.368s
sys             3m41.851s     3m38.388s

[1] https://github.com/ryncsn/emm-test-project/tree/master/file-anon-mix-pressure
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/

v9:
* reduce one iteration for lruvec_populated_min_seq();
* let lruvec_is_reclaimable() count all zones, for min_ttl;
* refine lru_gen_imbalanced(), AGING_PRIORITY macro, roundup/down swappiness; per Kairui;
* let evict_folio_lists() return "drained" per Kairui, thanks!
* one bounded retry for scanned=0 and reduce concurrent aging
for no-populatable(drained) aging;
v8 link:
https://lore.kernel.org/linux-mm/[email protected]/

v8:
* Fixed Gaofei Lyu's name;
* for debugfs proactive reclaim, use DEF_PRIORITY;
* refine no-populatable aging (limit frequency, allow an aging after eviction returns 0 scanned)
* refine "restrict eviction accounting to eligible zones" for potential premature OOM;
* Collect Baolin's reviewed-by tag, thanks!
v7 link:
https://lore.kernel.org/linux-mm/[email protected]/

v7:
* fix a build error of bisect and address some other sashiko minor comments;
* add no-populatable aging;
v6 link:
https://lore.kernel.org/linux-mm/[email protected]/

v6:
* rebase on top of mm-new(a lot of conflicts due to recent active mglru developments);
* only count eligible zones below sc->reclaim_idx for lruvec_evictable_size() and lru_gen_imbalanced();
* GFP_NOIO further fixing by Bo Zhang, exclude unreclaimable type in lruvec_evictable_size().
* Add Kunwu's reviewed-by, thanks!
* RFC v5 link:
https://lore.kernel.org/linux-mm/[email protected]/

RFC v5:
* Remove the aging optimizations, which have been merged - thanks for Baolin, Baoqun, Lian Wang, Kairui, Kunwu, Ridong reviewing on this;
* Add Kairui's model for determining nr_to_scan[] for both types.
* Fix cold/hot inversion for rejected folios - Barry.
* Improve and clean up imbalanced aging, and optimize extreme swappiness values of 1 and 200 - Gaofei, Barry.
* Address the GFP_NOIO issue in MGLRU - Bo Zhang.
* RFC v4 link:
https://lore.kernel.org/linux-mm/[email protected]/

RFC v4:
* Clean up isolate_folios() based on Baolin's feedback. Thanks!
* Further improve isolate_folios() and scan_folios(), including handling of exhausted reclaim types.
* Add fixes and performance improvements to inc_min_seq(), and reduce lock contention.
* Improve the handling of imbalanced aging and remaining generation-based aging.
* RFC v3 link:
https://lore.kernel.org/linux-mm/[email protected]/

RFC v3:
- Added Ridong's patch of improving isolate_folios() readability;
- for lru_gen_imbalanced(), apply some suggestions from Zicheng and Kairui, considering lruvec size and use macro;
- drop "mm: mglru: only fall back when reclaim is running at high priority"
- mitigate aging for balanced swappiess value(near middle 100) as swappiness bias is less a concern for them; but slightly increased aging overhead might not be good.
* RFC v2 link:
https://lore.kernel.org/linux-mm/[email protected]/

RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!

Barry Song (Xiaomi) (5):
mm/mglru: Avoid placing rejected folios below populated gen mm/mglru: Run aging when one type is drained
mm/mglru: Restrict eviction accounting to eligible zones mm/mglru: Reduce LRU lock contention by shrinking folio batches in
inc_min_seq
mm/mglru: Make swappiness=1 file reclaim more aggressive

Bo Zhang (1):
mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO
allocation

Gaofei Lyu (2):
mm/mglru: Run aging when pages are rare outside the youngest gen mm/mglru: Avoid prolonged anon reclaim with swappiness=200

Kairui Song (8):
mm/mglru: report evictable sizes per type
mm/mglru: extract lruvec_populated_min_seq() helper mm/mglru: reclaim from the oldest populated gen mm/mglru: lift the forced protection of reclaimer mm/mglru: skip zones above reclaim_idx instead of rotating them mm/mglru: return for unavailable anon reclaim earlier mm/mglru: balance anon and file scan budget by refault rate mm/mglru: lift youngest gen protection for higher priority

 include/linux/mm_inline.h |  22 +-
 mm/trace_vmscan.h         |   4 +-
 mm/vmscan.c               | 514 +++++++++++++++++++++++++-------------
 mm/vmscan.h               |   7 +-
 4 files changed, 364 insertions(+), 183 deletions(-)

2.39.3 (Apple Git-146)