mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
Subsystems (7)
cover mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:
- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.
This patch removes those producer-side acquisitions entirely. The producer-local inputs (PGROTATE_*, PGRECLAIM_PAGEOUT_*) become per-LRU vmstat counters; WORKINGSET_RESTORE_* already captures the refault input. prepare_scan_control() reads the raw cost signal lock-free from those vmstats and folds the delta into a per-lruvec accumulator. A dedicated per-lruvec cost_lock — not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru() — serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear.
Trade-offs:
- Signal freshness is slightly worse: cost reads see rstat- aggregated values that can lag until periodic / reader-triggered flushing. Decay timing is also coarser since multiple producer events may be batched into one read-side halving check. The cost signal is a heuristic feeding the anon-vs-file split, it's not a precise control loop — it's deliberately smoothed by the lrusize/4 halving. Producing/consuming it with a tiny lag should not be perceptible.
- Per-lruvec footprint grows by 2 unsigned longs + a spinlock, its a small cost.
== Numbers ==
Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem:
usemem -n 16 --prealloc --prefault --random $((256*1024*1024))
run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.
Workload rates are identical on both kernels (the bench drives the same memory pressure):
baseline patched delta
pgscan_direct / s 172,662 171,817 ~0%
pgsteal_direct / s 67,162 66,306 ~0%
workingset_refault_anon / s
40,696 39,830 ~0%perf lock contention (total wait per 30 s window):
Lock Name Before After % change
shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated)
(= lru_note_cost_unlock_irq)
workingset_refault+0x167 385.26 ms 0 -100% (eliminated)
(= lru_note_cost_refault)
shrink_node+0x4ad 689.43 ms 26.95 ms -96%
shrink_active_list 208.34 ms 15.97 ms -92%
lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53%Total LRU lock wait ~4.23 s ~1.66 s -61%
The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk.
The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec.
The numbers above can be reproduced using the script in [1].
== Alternatives considered ==
1. cost_lock for both producer and consumer (no vmstat indirection):
Keep the producer loop, just swap lru_lock for a new per-lruvec cost_lock. Decouples cost from LRU manipulation, but producers still synchronously contend on cost_lock, the parent_lruvec() walk is still required (O(memcg-depth) acquisitions per recording, now on cost_lock), and lru_reparent_memcg() still needs explicit cost-splice. We can do much better and this series removes the producer lock entirely and gets hierarchy propagation for "free" via rstat.
2. Attempt to switch to using MGLRU's scan model:
MGLRU has no anon_cost/file_cost at all. It replaces the cost heuristic with generation-based aging: per-LRU sequence numbers (min_seq/max_seq) age folios into generations, and the older-generation type is the one to scan. So lru_note_cost_unlock_irq() / lru_note_cost_refault() are simply not called when lru_gen_enabled() — by design it sidesteps every concern this patch addresses.
But MGLRU is not a substitute for fixing classic LRU:
- It relies on a lot of things including per-lruvec generation lists, bloom filters, mm_struct walk infrastructure, working-set protection tiers and a whole sysfs interface. Replacing classic LRU's cost recording with the MGLRU model would mean dragging in all of that.
- It changes scan-balance semantics, not just the locking, so it's a heuristic change we would need to evaluate separately. There are known regressions (database/anon-heavy workloads sensitive to swappiness, or file-cache-dominated workloads where MGLRU's bloom-filter protection differs from classic refault tracking).
This patch preserves classic-LRU semantics.
3. Atomic cost counter:
lrusize/4 halving has no clean atomic form, and the parent walk still has to run explicitly. Reusing vmstats gives per-CPU aggregation AND rstat hierarchy propagation for free.
4. Drop cost_lock from the existing patch and reuse lru_lock in the consumer (prepare_scan_control()):
Saves 1 lock space per lruvec but re-couples the cost path to LRU manipulation, though just from the consumer side this time. prepare_scan_control() runs at the start of every shrink_lruvec() cycle, so under sustained memory pressure it would take lru_lock on the hot path and block isolate_lru_folios() / move_folios_to_lru() / folio_add_lru() i.e. when reclaim is in flight. A dedicated cost_lock is never taken by anyone except the consumer cost calucation.
[1] https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884
Usama Arif (1):
mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance
cost
include/linux/mmzone.h | 11 +++++-- include/linux/swap.h | 3 -- mm/memcontrol-v1.c | 4 +-- mm/memcontrol.c | 4 +++ mm/mmzone.c | 1 + mm/swap.c | 69 ------------------------------------------ mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++------ mm/vmstat.c | 4 +++ mm/workingset.c | 5 --- 9 files changed, 74 insertions(+), 91 deletions(-)
Signature · 1 line
2.53.0-Meta
RFC 1/1 mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() also walks multiplies cost-update overhead by memcg hierarchy depth.
Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Add explicit node_stat counters for the producer-local inputs:
PGRECLAIM_PAGEOUT_{ANON,FILE} - reclaim-driven pageout submissions
(formerly stat.nr_pageout, weighted by SWAP_CLUSTER_MAX).
PGROTATE_{ANON,FILE} - reclaim-driven rotations, bumped from
both shrink_inactive_list (by nr_scanned - nr_reclaimed) and shrink_active_list (by nr_rotated), unweighted.
WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.
In prepare_scan_control() the raw cost signal is recomputed lock-free of lru_lock from monotonic counters:
now = (PGRECLAIM_PAGEOUT_X + WORKINGSET_RESTORE_X) * SWAP_CLUSTER_MAX
+ PGROTATE_X
The delta against a per-lruvec prev_cost[] snapshot is folded into cost_accum[]. The lrusize/4 halving threshold is preserved, but the decay check now happens at the read site instead of on every producer update.
A dedicated per-lruvec spinlock, cost_lock, serialises the prev_cost RMW, the accumulator update, and the halving check against concurrent reclaimers in the same memcg+node.
Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.
--- include/linux/mmzone.h | 11 +++++-- include/linux/swap.h | 3 -- mm/memcontrol-v1.c | 4 +-- mm/memcontrol.c | 4 +++ mm/mmzone.c | 1 + mm/swap.c | 69 ------------------------------------------ mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++------ mm/vmstat.c | 4 +++ mm/workingset.c | 5 --- 9 files changed, 74 insertions(+), 91 deletions(-)
include/linux/mmzone.h +9 -2
@@ -323,6 +323,10 @@ enum node_stat_item {PGSCAN_PROACTIVE ,PGSCAN_ANON ,PGSCAN_FILE , +PGRECLAIM_PAGEOUT_ANON , +PGRECLAIM_PAGEOUT_FILE , +PGROTATE_ANON , +PGROTATE_FILE ,PGREFILL , #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB,
@@ -763,9 +767,12 @@ struct lruvec {* These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. +* Updated and decayed at prepare_scan_control() time; cost_lock +* serialises that update. */ -unsigned long anon_cost; -unsigned long file_cost; +unsigned long prev_cost[ANON_AND_FILE ]; +unsigned long cost_accum[ANON_AND_FILE ]; +spinlock_t cost_lock;/* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age;/* Refaults at the time of last reclaim cycle */
include/linux/swap.h +0 -3
@@ -309,9 +309,6 @@ extern unsigned long totalreserve_pages;/* linux/mm/swap.c */ -void lru_note_cost_unlock_irq (struct lruvec *lruvec,bool file, -unsigned int nr_io,unsigned int nr_rotated); -void lru_note_cost_refault (struct folio *);void folio_add_lru (struct folio *);void folio_add_lru_vma (struct folio *,struct vm_area_struct *);void mark_page_accessed (struct page *);
mm/memcontrol-v1.c +2 -2
@@ -1988,8 +1988,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)for_each_online_pgdat (pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost_accum[WORKINGSET_ANON ]; + file_cost += mz->lruvec.cost_accum[WORKINGSET_FILE ]; }seq_buf_printf (s,"anon_cost %lu\n" , anon_cost);seq_buf_printf (s,"file_cost %lu\n" , file_cost);
mm/memcontrol.c +4 -0
@@ -419,6 +419,10 @@ static const unsigned int memcg_node_stat_items[] = {PGSCAN_PROACTIVE ,PGSCAN_ANON ,PGSCAN_FILE , +PGRECLAIM_PAGEOUT_ANON , +PGRECLAIM_PAGEOUT_FILE , +PGROTATE_ANON , +PGROTATE_FILE ,PGREFILL , #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB,
mm/mmzone.c +1 -0
@@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec)memset (lruvec,0 ,sizeof (struct lruvec ));spin_lock_init (&lruvec->lru_lock); +spin_lock_init (&lruvec->cost_lock);zswap_lruvec_state_init (lruvec);for_each_lru (lru)
mm/swap.c +0 -69
81 lines not on this page · raw message
mm/vmscan.c +54 -10
85 lines not on this page · raw message
mm/vmstat.c +4 -0
@@ -1289,6 +1289,10 @@ const char * const vmstat_text[] = { [I(PGSCAN_PROACTIVE)] = "pgscan_proactive", [I(PGSCAN_ANON)] = "pgscan_anon", [I(PGSCAN_FILE)] = "pgscan_file", + [I(PGRECLAIM_PAGEOUT_ANON)] = "pgreclaim_pageout_anon", + [I(PGRECLAIM_PAGEOUT_FILE)] = "pgreclaim_pageout_file", + [I(PGROTATE_ANON)] = "pgrotate_anon", + [I(PGROTATE_FILE)] = "pgrotate_file", [I (PGREFILL )] ="pgrefill" , #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] = "nr_hugetlb",
mm/workingset.c +0 -5
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow)/* Folio was active prior to eviction */ if (workingset) {folio_set_workingset (folio); -/* -* XXX: Move to folio_add_lru() when it supports new vs -* putback -*/ -lru_note_cost_refault (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file, nr); } out:
Signature · 1 line
2.53.0-Meta