mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
Subsystems (7)
v3 cover mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:
- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.
This series removes those producer-side acquisitions entirely. The rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters. NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout(); charge it through lruvec_stat_mod_folio() so it is available per lruvec and aggregated through the memcg hierarchy. Reclaim does not submit filesystem folios for writeback from this path, so pageout contributes only to anon cost. WORKINGSET_RESTORE_* already captures the refault input.
prepare_scan_control() reads the raw cost signals without lru_lock:
anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX
It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear.
Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.
Trade-offs:
- Cost reads see rstat-aggregated values that can lag until periodic / reader-triggered flushing.
- Per-lruvec footprint grows by 2 unsigned longs + a spinlock, which is a small cost.
- NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat, adding memcg stat accounting to the reclaim writeout path while preserving the existing node-level total.
== Numbers ==
Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem:
usemem -n 16 --prealloc --prefault --random $((256*1024*1024))
run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.
Workload rates are identical on both kernels (the bench drives the same memory pressure):
baseline patched delta
pgscan_direct / s 172,662 171,817 ~0%
pgsteal_direct / s 67,162 66,306 ~0%
workingset_refault_anon / s
40,696 39,830 ~0%perf lock contention (total wait per 30 s window):
Lock Name Before After % change
shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated)
(= lru_note_cost_unlock_irq)
workingset_refault+0x167 385.26 ms 0 -100% (eliminated)
(= lru_note_cost_refault)
shrink_node+0x4ad 689.43 ms 26.95 ms -96%
shrink_active_list 208.34 ms 15.97 ms -92%
lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53%Total LRU lock wait ~4.23 s ~1.66 s -61%
The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk.
The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec.
The numbers above can be reproduced using the script in [1].
== Series structure ==
1. mm/vmstat, mm/memcontrol: add _monotonic vmstat readers Non-clamping siblings of lruvec_page_state() / node_page_state() / global_node_page_state() for callers that snapshot a monotonically-incremented counter and subtract two samples. Unsigned modular subtraction on the raw wrapping value stays correct across a signed-long overflow that the existing clamp would otherwise turn into a huge spurious delta (a real hazard on 32-bit).
2. mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The optimization described above; samples PGROTATE_*, NR_VMSCAN_WRITE (for anon) and WORKINGSET_RESTORE_* via lruvec_page_state_monotonic() from patch 1. MGLRU also updates PGROTATE_* so the counters remain meaningful with either reclaim implementation, although MGLRU does not consume this cost signal.
== Alternatives considered ==
1. cost_lock for both producer and consumer (no vmstat indirection):
Keep the producer loop, just swap lru_lock for a new per-lruvec cost_lock. Decouples cost from LRU manipulation, but producers still synchronously contend on cost_lock, the parent_lruvec() walk is still required (O(memcg-depth) acquisitions per recording, now on cost_lock), and lru_reparent_memcg() still needs explicit cost-splice. We can do much better and this series removes the producer lock entirely and gets hierarchy propagation for "free" via rstat.
2. Attempt to switch to using MGLRU's scan model:
MGLRU has no anon_cost/file_cost at all. It replaces the cost heuristic with generation-based aging: per-LRU sequence numbers (min_seq/max_seq) age folios into generations, and the older-generation type is the one to scan. So lru_note_cost_unlock_irq() / lru_note_cost_refault() are simply not called when lru_gen_enabled() — by design it sidesteps every concern this patch addresses.
But MGLRU is not a substitute for fixing classic LRU:
- It relies on a lot of things including per-lruvec generation lists, bloom filters, mm_struct walk infrastructure, working-set protection tiers and a whole sysfs interface. Replacing classic LRU's cost recording with the MGLRU model would mean dragging in all of that.
- It changes scan-balance semantics, not just the locking, so it's a heuristic change we would need to evaluate separately. There are known regressions (database/anon-heavy workloads sensitive to swappiness, or file-cache-dominated workloads where MGLRU's bloom-filter protection differs from classic refault tracking).
This series keeps classic LRU's scan-balance model and fixes where its cost history is sampled and aged.
3. Atomic cost counter:
lrusize/4 halving has no clean atomic form, and the parent walk still has to run explicitly. Reusing vmstats gives per-CPU aggregation AND rstat hierarchy propagation for free.
4. Drop cost_lock from the existing patch and reuse lru_lock in the consumer (prepare_scan_control()):
Saves 1 lock space per lruvec but re-couples the cost path to LRU manipulation, though just from the consumer side this time. prepare_scan_control() runs at the start of every shrink_lruvec() cycle, so under sustained memory pressure it would take lru_lock on the hot path and block isolate_lru_folios() / move_folios_to_lru() / folio_add_lru() i.e. when reclaim is in flight. A dedicated cost_lock is never taken by anyone except the consumer cost calculation.
[1] https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Reuse NR_VMSCAN_WRITE for anon pageout cost instead of adding PGRECLAIM_PAGEOUT_{ANON,FILE}. Reclaim no longer writes filesystem folios from this path, so there is no file pageout cost. (Johannes)
- Charge NR_VMSCAN_WRITE through the lruvec/memcg stats and remove the now-unused reclaim_stat.nr_pageout plumbing.
- Account PGROTATE_{ANON,FILE} in the MGLRU eviction path so the new counters remain meaningful regardless of the active LRU implementation. (sashiko)
- MGLRU updates PGROTATE_{ANON,FILE} so the counters remain meaningful with it. (sashiko)
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Sample via the newly introduced lruvec_page_state_monotonic()
to fix a 32-bit delta underflow when the underlying signed
long wraps past LONG_MAX (Johannes and sashiko)
RFC -> v1: https://lore.kernel.org/all/[email protected]/
- Document in coverletter and commit message how the read-side vmstat accumulator improves cost-model aging across reclaim gaps (Johannes)
- Fully decay the cost_accum below lrusize / 4 using a while loop (sashiko)
Usama Arif (2):
mm/vmstat, mm/memcontrol: add _monotonic vmstat readers mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance
cost
include/linux/memcontrol.h | 8 ++++ include/linux/mmzone.h | 10 ++++- include/linux/swap.h | 3 -- include/linux/vmstat.h | 17 +++++++- mm/memcontrol-v1.c | 4 +- mm/memcontrol.c | 39 ++++++++++++++++++ mm/mmzone.c | 1 + mm/swap.c | 69 ------------------------------- mm/vmscan.c | 84 +++++++++++++++++++++++++++++++------- mm/vmstat.c | 13 ++++++ mm/workingset.c | 5 --- 11 files changed, 157 insertions(+), 96 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH v3 1/2 mm/vmstat, mm/memcontrol: add _monotonic vmstat readers
lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than
as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers.
It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator.
Add non-clamping siblings that return the underlying state value cast to unsigned long:
global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()
With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper.
The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero.
This is a prerequisite for the following patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds the unsigned modular delta into a per-lruvec cost_accum[].
--- include/linux/memcontrol.h | 8 ++++++++ include/linux/vmstat.h | 16 ++++++++++++++++ mm/memcontrol.c | 36 ++++++++++++++++++++++++++++++++++++ mm/vmstat.c | 11 +++++++++++ 4 files changed, 71 insertions(+)
include/linux/memcontrol.h +8 -0
@@ -931,6 +931,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);bool memcg_stat_item_valid (int idx);bool memcg_vm_event_item_valid (enum vm_event_item idx);unsigned long lruvec_page_state (struct lruvec *lruvec,enum node_stat_item idx); +unsigned long lruvec_page_state_monotonic (struct lruvec *lruvec, +enum node_stat_item idx);unsigned long lruvec_page_state_local (struct lruvec *lruvec,enum node_stat_item idx);
@@ -1378,6 +1380,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec,return node_page_state (lruvec_pgdat (lruvec), idx); } +static inline unsigned long lruvec_page_state_monotonic (struct lruvec *lruvec, +enum node_stat_item idx) +{ +return node_page_state_monotonic (lruvec_pgdat (lruvec), idx); +} +static inline unsigned long lruvec_page_state_local (struct lruvec *lruvec,enum node_stat_item idx) {
include/linux/vmstat.h +16 -0
@@ -194,6 +194,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item)return x; } +/* +* Non-clamping variant of global_node_page_state() intended for callers that +* snapshot a monotonically-incremented counter and subtract two samples. +* Returns the raw wrapping value so that unsigned modular subtraction stays +* correct across a signed-long overflow (a real hazard on 32-bit) that the +* clamp in global_node_page_state() would otherwise turn into a huge spurious +* delta. Do NOT use for non-monotonic page-count reads. +*/ +static inline unsigned long global_node_page_state_monotonic (enum node_stat_item item) +{ +return (unsigned long )atomic_long_read (&vm_node_stat[item]); +} +static inline unsigned long global_node_page_state (enum node_stat_item item) {VM_WARN_ON_ONCE (vmstat_item_in_bytes (item));
@@ -259,11 +272,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat, enum node_stat_item item);extern unsigned long node_page_state_pages (struct pglist_data *pgdat,enum node_stat_item item); +extern unsigned long node_page_state_monotonic (struct pglist_data *pgdat, +enum node_stat_item item);extern void fold_vm_numa_events (void );#else #define sum_zone_node_page_state (node, item) global_zone_page_state(item)#define node_page_state (node, item) global_node_page_state(item)#define node_page_state_pages (node, item) global_node_page_state_pages(item) +#define node_page_state_monotonic (node, item) global_node_page_state_monotonic(item)static inline void fold_vm_numa_events (void ) { }
mm/memcontrol.c +36 -0
@@ -502,6 +502,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)return x; } +/** +* lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling +* @lruvec: the LRU vector to read from +* @idx: the node_stat_item to read +* +* Returns the raw state[idx] value cast to unsigned long, skipping the +* clamp-negative-to-zero step in lruvec_page_state(). Intended for callers +* that snapshot a monotonically-incremented counter and subtract two +* samples: unsigned modular arithmetic then yields the correct delta across +* a signed-long wraparound (a real hazard on 32-bit) that the clamp would +* otherwise turn into a huge spurious delta. +* +* Do NOT use for non-monotonic page-count reads where a transient negative +* reading from per-CPU delta skew must present as zero. +* +* XXX: This helper (and its node/global peers) exists because we place +* monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*) +* into enum node_stat_item. +*/ +unsigned long lruvec_page_state_monotonic (struct lruvec *lruvec, +enum node_stat_item idx) +{ +struct mem_cgroup_per_node *pn; +int i; + +if (mem_cgroup_disabled ()) +return node_page_state_monotonic (lruvec_pgdat (lruvec), idx); + + i =memcg_stats_index (idx); +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return 0 ; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); +return (unsigned long )READ_ONCE (pn->lruvec_stats->state[i]); +} +unsigned long lruvec_page_state_local (struct lruvec *lruvec,enum node_stat_item idx) {
mm/vmstat.c +11 -0
@@ -1024,6 +1024,17 @@ unsigned long node_page_state(struct pglist_data *pgdat, return node_page_state_pages(pgdat, item); } + +/* +* Non-clamping variant of node_page_state() intended for callers that +* snapshot a monotonically-incremented counter and subtract two samples. +* See global_node_page_state_monotonic() for the rationale. +*/ +unsigned long node_page_state_monotonic (struct pglist_data *pgdat, +enum node_stat_item item) +{ +return (unsigned long )atomic_long_read (&pgdat->vm_stat[item]); +}#endif /*
Signature · 1 line
2.53.0-Meta
PATCH v3 2/2 mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
In reply to Usama ArifThe anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.
Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.
Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.
In prepare_scan_control() the raw cost signals are recomputed without lru_lock as:
anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX
The delta against a per-lruvec prev_cost[] snapshot is folded into cost_accum[]. Since one vmstat delta can cover many producer events between reclaim passes, halve cost_accum[] until the total is back within the lrusize/4 bound instead of halving only once.
Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.
A dedicated per-lruvec spinlock, cost_lock, serialises the prev_cost RMW, the accumulator update, and the halving check against concurrent reclaimers in the same memcg+node.
Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.
NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.
memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost_accum[] instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.
Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.
--- include/linux/mmzone.h | 10 ++++- include/linux/swap.h | 3 -- include/linux/vmstat.h | 1 - mm/memcontrol-v1.c | 4 +- mm/memcontrol.c | 5 ++- mm/mmzone.c | 1 + mm/swap.c | 69 ---------------------------------- mm/vmscan.c | 84 +++++++++++++++++++++++++++++++++++------- mm/vmstat.c | 2 + mm/workingset.c | 5 --- 10 files changed, 87 insertions(+), 97 deletions(-)
include/linux/mmzone.h +8 -2
@@ -323,6 +323,8 @@ enum node_stat_item {PGSCAN_PROACTIVE ,PGSCAN_ANON ,PGSCAN_FILE , +PGROTATE_ANON , +PGROTATE_FILE ,PGREFILL , #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB,
@@ -763,9 +765,13 @@ struct lruvec {* These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. +* Updated and decayed at prepare_scan_control() time; cost_lock +* serialises that update. */ -unsigned long anon_cost; -unsigned long file_cost; +unsigned long prev_cost[ANON_AND_FILE ]; +unsigned long cost_accum[ANON_AND_FILE ]; +/* Protects prev_cost and cost_accum. */ +spinlock_t cost_lock;/* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age;/* Refaults at the time of last reclaim cycle */
include/linux/swap.h +0 -3
@@ -309,9 +309,6 @@ extern unsigned long totalreserve_pages;/* linux/mm/swap.c */ -void lru_note_cost_unlock_irq (struct lruvec *lruvec,bool file, -unsigned int nr_io,unsigned int nr_rotated); -void lru_note_cost_refault (struct folio *);void folio_add_lru (struct folio *);void folio_add_lru_vma (struct folio *,struct vm_area_struct *);void mark_page_accessed (struct page *);
include/linux/vmstat.h +0 -1
@@ -20,7 +20,6 @@ struct reclaim_stat {unsigned nr_congested;unsigned nr_writeback;unsigned nr_immediate; -unsigned nr_pageout;unsigned nr_activate[ANON_AND_FILE ];unsigned nr_ref_keep;unsigned nr_unmap_fail;
mm/memcontrol-v1.c +2 -2
@@ -1988,8 +1988,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)for_each_online_pgdat (pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost_accum[WORKINGSET_ANON ]; + file_cost += mz->lruvec.cost_accum[WORKINGSET_FILE ]; }seq_buf_printf (s,"anon_cost %lu\n" , anon_cost);seq_buf_printf (s,"file_cost %lu\n" , file_cost);
mm/memcontrol.c +4 -1
@@ -393,6 +393,7 @@ static const unsigned int memcg_node_stat_items[] = {NR_SHMEM_THPS ,NR_FILE_THPS ,NR_ANON_THPS , +NR_VMSCAN_WRITE ,NR_VMALLOC ,NR_KERNEL_STACK_KB , NR_PAGETABLE,
@@ -419,6 +420,8 @@ static const unsigned int memcg_node_stat_items[] = {PGSCAN_PROACTIVE ,PGSCAN_ANON ,PGSCAN_FILE , +PGROTATE_ANON , +PGROTATE_FILE ,PGREFILL , #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB,
@@ -518,7 +521,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)* reading from per-CPU delta skew must present as zero. * * XXX: This helper (and its node/global peers) exists because we place -* monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*) +* monotonically-incremented event counters (NR_VMSCAN_WRITE and PGROTATE_*) * into enum node_stat_item. */ unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
mm/mmzone.c +1 -0
@@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec)memset (lruvec,0 ,sizeof (struct lruvec ));spin_lock_init (&lruvec->lru_lock); +spin_lock_init (&lruvec->cost_lock);zswap_lruvec_state_init (lruvec);for_each_lru (lru)
mm/swap.c +0 -69
81 lines not on this page · raw message
mm/vmscan.c +70 -14
141 lines not on this page · raw message
mm/vmstat.c +2 -0
@@ -1300,6 +1300,8 @@ const char * const vmstat_text[] = { [I(PGSCAN_PROACTIVE)] = "pgscan_proactive", [I(PGSCAN_ANON)] = "pgscan_anon", [I(PGSCAN_FILE)] = "pgscan_file", + [I(PGROTATE_ANON)] = "pgrotate_anon", + [I(PGROTATE_FILE)] = "pgrotate_file", [I (PGREFILL )] ="pgrefill" , #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] = "nr_hugetlb",
mm/workingset.c +0 -5
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow)/* Folio was active prior to eviction */ if (workingset) {folio_set_workingset (folio); -/* -* XXX: Move to folio_add_lru() when it supports new vs -* putback -*/ -lru_note_cost_refault (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file, nr); } out:
Signature · 1 line
2.53.0-Meta
Johannes Weiner
maintainerQuoted 18 lines · Usama Arif
mm/vmscan.c +13
-1@@ -2303,12 +2301,63 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc)mem_cgroup_flush_stats_ratelimited (sc->target_mem_cgroup);/* -* Determine the scan balance between anon and file LRUs. +* Determine the scan balance between anon and file LRUs from per-LRU +* vmstat counters. The raw cost per side is: +* +* PGROTATE - reclaim-driven rotations, bumped from both +* shrink_inactive_list and shrink_active_list +* (CPU work). +* NR_VMSCAN_WRITE - reclaim-driven anon pageout IO. +* WORKINGSET_RESTORE - refaults of previously-workingset pages. +* +* The two IO terms are weighted by SWAP_CLUSTER_MAX to reflect the +* higher cost of an IO over a rotation. +* +* Reads are lock-free per-cpu sum collations, rstat-aggregated up
mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2326
Quoted 10 lines
mm/vmscan.c +10
-0+* the memcg hierarchy by mem_cgroup_flush_stats_ratelimited() above. +* Use lruvec_page_state_monotonic() so the unsigned subtraction +* `now - prev_cost[f]` yields the correct delta across a signed-long +* wraparound of the underlying counter (a real hazard on 32-bit that +* the clamp in lruvec_page_state() would otherwise turn into a huge +* spurious delta). +* +* The delta against prev_cost is folded into cost_accum, which is +* halved on both sides until their sum is within lrusize/4. +* cost_lock serialises concurrent reclaimers in the same memcg+node.
IMO that's a lot of describing what the code does. Why not stick closer to the original comments?
Quoted 7 lines
mm/vmscan.c +2
-4*/ -spin_lock_irq (&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; -spin_unlock_irq (&target_lruvec->lru_lock); +spin_lock (&target_lruvec->cost_lock); +for (int f =0 ; f <=1 ; f++) {
mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2339
Quoted 10 lines
mm/vmscan.c +10
-0+unsigned long now, delta; + + now =lruvec_page_state_monotonic (target_lruvec,PGROTATE_ANON + f) + +lruvec_page_state_monotonic (target_lruvec, +WORKINGSET_RESTORE_BASE + f) * +SWAP_CLUSTER_MAX ; +if (f ==WORKINGSET_ANON ) + now +=lruvec_page_state_monotonic (target_lruvec, +NR_VMSCAN_WRITE ) * +SWAP_CLUSTER_MAX ;
It's hard to prove overflow behavior is correct. I would keep the delta extraction dead simple, then do the weight math on the delta.
Quoted 7 lines
mm/vmscan.c +7
-0+ delta = now - target_lruvec->prev_cost[f]; + target_lruvec->prev_cost[f] = now; + target_lruvec->cost_accum[f] += delta; + } +unsigned long lrusize = +lruvec_page_state (target_lruvec,NR_INACTIVE_ANON ) + + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +
mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2356
Quoted 10 lines
mm/vmscan.c +10
-0+lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ) + +lruvec_page_state (target_lruvec,NR_ACTIVE_FILE ); +unsigned long cost_limit = lrusize /4 ; + +while (target_lruvec->cost_accum[WORKINGSET_ANON ] > cost_limit || + target_lruvec->cost_accum[WORKINGSET_FILE ] > cost_limit || + target_lruvec->cost_accum[WORKINGSET_ANON ] + + target_lruvec->cost_accum[WORKINGSET_FILE ] > cost_limit) { + target_lruvec->cost_accum[WORKINGSET_ANON ] /=2 ; + target_lruvec->cost_accum[WORKINGSET_FILE ] /=2 ;
Why do you need to check them individually? Between reclaim cycles, there are no scans->rotations. And I don't see how you could get refault events several times the size of the LRU, let alone in excess of ULONG_MAX.
mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2360
Quoted 4 lines
mm/vmscan.c +4
-0+ } + sc->anon_cost = target_lruvec->cost_accum[WORKINGSET_ANON ]; + sc->file_cost = target_lruvec->cost_accum[WORKINGSET_FILE ]; +spin_unlock (&target_lruvec->cost_lock);
I realize these long descriptor names make it hard, but I think this can be cleaned up a bit and written in a more idiomatic way.
How about this (totally untested):
struct lru_cost {
unsigned long count;
unsigned long last_rotated;
unsigned long last_io;
};struct lruvec {
struct lru_cost cost[2];
}; struct lru_cost *anon_cost, *file_cost;
unsigned long lrusize;
/*
* Determine the scan balance between anon and file LRUs.
*
* The cost model is based on rotations, refaults and
* reclaim-driven writes (anon only) on each side.
*
* These event counters are monotonic, so each reclaim cycle
* the delta since the last scan is extracted and incorporated
* into a decaying average. This ensures currency, as workloads
* change over time, and avoids overflow in the calculations.
*/
spin_lock_irq(&target_lruvec->cost_lock);
for (file = 0; file <= 1; file++) {
struct lru_cost *cost = &target_lruvec->cost[file];
unsigned long rotated, io, nr_rotated, nr_io;
rotated = lruvec_page_state_monotonic(target_lruvec,
PGROTATE_ANON + file);
io = lruvec_page_state_monotonic(target_lruvec,
WORKINGSET_RESTORE_BASE + file);
if (!file)
io += lruvec_page_state_monotonic(target_lruvec,
NR_VMSCAN_WRITE);
nr_rotated = rotated - cost->last_rotated;
nr_io = io - cost->last_io;
/*
* Reflect the relative cost of incurring IO and spending CPU
* time on rotations. This doesn't attempt to make a precise
* comparison, it just says: if reloads are about comparable
* between the LRU lists, or rotations are overwhelmingly
* different between them, adjust scan balance for CPU work.
*/
cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated;
cost->last_rotated = rotated;
cost->last_io = io;
}
anon_cost = &target_lruvec->cost[WORKINGSET_ANON];
file_cost = &target_lruvec->cost[WORKINGSET_FILE];
lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) +
lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +
lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) +
lruvec_page_state(target_lruvec, NR_ACTIVE_FILE);
while (anon_cost->count + file_cost->count > lrusize / 4) {
anon_cost->count /= 2;
file_cost->count /= 2;
}
sc->anon_cost = anon_cost->count;
sc->file_cost = file_cost->count;
spin_unlock(&target_lruvec->cost_lock);Quoted 77 lines · On Fri, 17 Jul 2026 23:08:08 +0200 Johannes Weiner <[email protected]> wrote:
Quoted 18 lines · Usama Arif
mm/vmscan.c +13
-1@@ -2303,12 +2301,63 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc)mem_cgroup_flush_stats_ratelimited (sc->target_mem_cgroup);/* -* Determine the scan balance between anon and file LRUs. +* Determine the scan balance between anon and file LRUs from per-LRU +* vmstat counters. The raw cost per side is: +* +* PGROTATE - reclaim-driven rotations, bumped from both +* shrink_inactive_list and shrink_active_list +* (CPU work). +* NR_VMSCAN_WRITE - reclaim-driven anon pageout IO. +* WORKINGSET_RESTORE - refaults of previously-workingset pages. +* +* The two IO terms are weighted by SWAP_CLUSTER_MAX to reflect the +* higher cost of an IO over a rotation. +* +* Reads are lock-free per-cpu sum collations, rstat-aggregated up mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2326
Quoted 10 lines
mm/vmscan.c +10
-0+* the memcg hierarchy by mem_cgroup_flush_stats_ratelimited() above. +* Use lruvec_page_state_monotonic() so the unsigned subtraction +* `now - prev_cost[f]` yields the correct delta across a signed-long +* wraparound of the underlying counter (a real hazard on 32-bit that +* the clamp in lruvec_page_state() would otherwise turn into a huge +* spurious delta). +* +* The delta against prev_cost is folded into cost_accum, which is +* halved on both sides until their sum is within lrusize/4. +* cost_lock serialises concurrent reclaimers in the same memcg+node. IMO that's a lot of describing what the code does. Why not stick closer to the original comments?
Quoted 7 lines
mm/vmscan.c +2
-4*/ -spin_lock_irq (&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; -spin_unlock_irq (&target_lruvec->lru_lock); +spin_lock (&target_lruvec->cost_lock); +for (int f =0 ; f <=1 ; f++) {mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2339
Quoted 10 lines
mm/vmscan.c +10
-0+unsigned long now, delta; + + now =lruvec_page_state_monotonic (target_lruvec,PGROTATE_ANON + f) + +lruvec_page_state_monotonic (target_lruvec, +WORKINGSET_RESTORE_BASE + f) * +SWAP_CLUSTER_MAX ; +if (f ==WORKINGSET_ANON ) + now +=lruvec_page_state_monotonic (target_lruvec, +NR_VMSCAN_WRITE ) * +SWAP_CLUSTER_MAX ;It's hard to prove overflow behavior is correct. I would keep the delta extraction dead simple, then do the weight math on the delta.
Quoted 7 lines
mm/vmscan.c +7
-0+ delta = now - target_lruvec->prev_cost[f]; + target_lruvec->prev_cost[f] = now; + target_lruvec->cost_accum[f] += delta; + } +unsigned long lrusize = +lruvec_page_state (target_lruvec,NR_INACTIVE_ANON ) + + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2356
Quoted 10 lines
mm/vmscan.c +10
-0+lruvec_page_state (target_lruvec,NR_INACTIVE_FILE ) + +lruvec_page_state (target_lruvec,NR_ACTIVE_FILE ); +unsigned long cost_limit = lrusize /4 ; + +while (target_lruvec->cost_accum[WORKINGSET_ANON ] > cost_limit || + target_lruvec->cost_accum[WORKINGSET_FILE ] > cost_limit || + target_lruvec->cost_accum[WORKINGSET_ANON ] + + target_lruvec->cost_accum[WORKINGSET_FILE ] > cost_limit) { + target_lruvec->cost_accum[WORKINGSET_ANON ] /=2 ; + target_lruvec->cost_accum[WORKINGSET_FILE ] /=2 ;Why do you need to check them individually? Between reclaim cycles, there are no
scans->rotations. And I don't see how you could get refault events several times the size of the LRU, let alone in excess ofULONG_MAX.
Quoted 8 lines
mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2360
Quoted 4 lines
mm/vmscan.c +4
-0+ } + sc->anon_cost = target_lruvec->cost_accum[WORKINGSET_ANON ]; + sc->file_cost = target_lruvec->cost_accum[WORKINGSET_FILE ]; +spin_unlock (&target_lruvec->cost_lock);I realize these long descriptor names make it hard, but I think this can be cleaned up a bit and written in a more idiomatic way.
Thanks for the prototype and the above reivew comments!
I integrated all of them and sent them as v4.
And best part of it is, sashiko is happy as well now with v4 lol