Skip to contentSkip to conversation

mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

PATCH Superseded Usama Arif 2 participants 5 messages Jul 17 – Jul 20

Subsystems (7)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v3 cover mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:

- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.

This series removes those producer-side acquisitions entirely. The rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters. NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout(); charge it through lruvec_stat_mod_folio() so it is available per lruvec and aggregated through the memcg hierarchy. Reclaim does not submit filesystem folios for writeback from this path, so pageout contributes only to anon cost. WORKINGSET_RESTORE_* already captures the refault input.

prepare_scan_control() reads the raw cost signals without lru_lock:

anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX

It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

Trade-offs:
- Cost reads see rstat-aggregated values that can lag until periodic / reader-triggered flushing.
- Per-lruvec footprint grows by 2 unsigned longs + a spinlock, which is a small cost.
- NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat, adding memcg stat accounting to the reclaim writeout path while preserving the existing node-level total.

== Numbers ==

Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem:

  usemem -n 16 --prealloc --prefault --random $((256*1024*1024))

run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.

Workload rates are identical on both kernels (the bench drives the same memory pressure):

                          baseline    patched      delta
  pgscan_direct  / s      172,662     171,817      ~0%
  pgsteal_direct / s       67,162      66,306      ~0%
  workingset_refault_anon / s
                           40,696      39,830      ~0%

perf lock contention (total wait per 30 s window):

  Lock Name                Before      After     % change
  shrink_lruvec+0x770     722.84 ms    0         -100% (eliminated)
        (= lru_note_cost_unlock_irq)
  workingset_refault+0x167 385.26 ms   0         -100% (eliminated)
        (= lru_note_cost_refault)
  shrink_node+0x4ad       689.43 ms    26.95 ms  -96%
  shrink_active_list      208.34 ms    15.97 ms  -92%
  lru_add_drain_cpu+0x34    1.96 s    917.71 ms  -53%

Total LRU lock wait ~4.23 s ~1.66 s -61%

The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk.

The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec.

The numbers above can be reproduced using the script in [1].

== Series structure ==

1. mm/vmstat, mm/memcontrol: add _monotonic vmstat readers Non-clamping siblings of lruvec_page_state() / node_page_state() / global_node_page_state() for callers that snapshot a monotonically-incremented counter and subtract two samples. Unsigned modular subtraction on the raw wrapping value stays correct across a signed-long overflow that the existing clamp would otherwise turn into a huge spurious delta (a real hazard on 32-bit).

2. mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The optimization described above; samples PGROTATE_*, NR_VMSCAN_WRITE (for anon) and WORKINGSET_RESTORE_* via lruvec_page_state_monotonic() from patch 1. MGLRU also updates PGROTATE_* so the counters remain meaningful with either reclaim implementation, although MGLRU does not consume this cost signal.

== Alternatives considered ==

1. cost_lock for both producer and consumer (no vmstat indirection):
Keep the producer loop, just swap lru_lock for a new per-lruvec cost_lock. Decouples cost from LRU manipulation, but producers still synchronously contend on cost_lock, the parent_lruvec() walk is still required (O(memcg-depth) acquisitions per recording, now on cost_lock), and lru_reparent_memcg() still needs explicit cost-splice. We can do much better and this series removes the producer lock entirely and gets hierarchy propagation for "free" via rstat.

2. Attempt to switch to using MGLRU's scan model:
MGLRU has no anon_cost/file_cost at all. It replaces the cost heuristic with generation-based aging: per-LRU sequence numbers (min_seq/max_seq) age folios into generations, and the older-generation type is the one to scan. So lru_note_cost_unlock_irq() / lru_note_cost_refault() are simply not called when lru_gen_enabled() — by design it sidesteps every concern this patch addresses.
But MGLRU is not a substitute for fixing classic LRU:
- It relies on a lot of things including per-lruvec generation lists, bloom filters, mm_struct walk infrastructure, working-set protection tiers and a whole sysfs interface. Replacing classic LRU's cost recording with the MGLRU model would mean dragging in all of that.
- It changes scan-balance semantics, not just the locking, so it's a heuristic change we would need to evaluate separately. There are known regressions (database/anon-heavy workloads sensitive to swappiness, or file-cache-dominated workloads where MGLRU's bloom-filter protection differs from classic refault tracking).
This series keeps classic LRU's scan-balance model and fixes where its cost history is sampled and aged.

3. Atomic cost counter:
lrusize/4 halving has no clean atomic form, and the parent walk still has to run explicitly. Reusing vmstats gives per-CPU aggregation AND rstat hierarchy propagation for free.

4. Drop cost_lock from the existing patch and reuse lru_lock in the consumer (prepare_scan_control()):
Saves 1 lock space per lruvec but re-couples the cost path to LRU manipulation, though just from the consumer side this time. prepare_scan_control() runs at the start of every shrink_lruvec() cycle, so under sustained memory pressure it would take lru_lock on the hot path and block isolate_lru_folios() / move_folios_to_lru() / folio_add_lru() i.e. when reclaim is in flight. A dedicated cost_lock is never taken by anyone except the consumer cost calculation.

[1] https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884

v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Reuse NR_VMSCAN_WRITE for anon pageout cost instead of adding PGRECLAIM_PAGEOUT_{ANON,FILE}. Reclaim no longer writes filesystem folios from this path, so there is no file pageout cost. (Johannes)
- Charge NR_VMSCAN_WRITE through the lruvec/memcg stats and remove the now-unused reclaim_stat.nr_pageout plumbing.
- Account PGROTATE_{ANON,FILE} in the MGLRU eviction path so the new counters remain meaningful regardless of the active LRU implementation. (sashiko)
- MGLRU updates PGROTATE_{ANON,FILE} so the counters remain meaningful with it. (sashiko)

v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Sample via the newly introduced lruvec_page_state_monotonic()
to fix a 32-bit delta underflow when the underlying signed
long wraps past LONG_MAX (Johannes and sashiko)

RFC -> v1: https://lore.kernel.org/all/[email protected]/
- Document in coverletter and commit message how the read-side vmstat accumulator improves cost-model aging across reclaim gaps (Johannes)
- Fully decay the cost_accum below lrusize / 4 using a while loop (sashiko)

Usama Arif (2):
mm/vmstat, mm/memcontrol: add _monotonic vmstat readers mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance
cost

 include/linux/memcontrol.h |  8 ++++
 include/linux/mmzone.h     | 10 ++++-
 include/linux/swap.h       |  3 --
 include/linux/vmstat.h     | 17 +++++++-
 mm/memcontrol-v1.c         |  4 +-
 mm/memcontrol.c            | 39 ++++++++++++++++++
 mm/mmzone.c                |  1 +
 mm/swap.c                  | 69 -------------------------------
 mm/vmscan.c                | 84 +++++++++++++++++++++++++++++++-------
 mm/vmstat.c                | 13 ++++++
 mm/workingset.c            |  5 ---
 11 files changed, 157 insertions(+), 96 deletions(-)
Signature · 1 line
2.53.0-Meta
author
#1 lore raw

PATCH v3 1/2 mm/vmstat, mm/memcontrol: add _monotonic vmstat readers

lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than
as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers.

It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator.

Add non-clamping siblings that return the underlying state value cast to unsigned long:

global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()

With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper.

The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero.

This is a prerequisite for the following patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds the unsigned modular delta into a per-lruvec cost_accum[].

---
 include/linux/memcontrol.h |  8 ++++++++
 include/linux/vmstat.h     | 16 ++++++++++++++++
 mm/memcontrol.c            | 36 ++++++++++++++++++++++++++++++++++++
 mm/vmstat.c                | 11 +++++++++++
 4 files changed, 71 insertions(+)
include/linux/memcontrol.h +8 -0
index e1f46a0016fc..b40bc4f6fe4a 100644
@@ -931,6 +931,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
 bool memcg_stat_item_valid(int idx);
 bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+					  enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
 
@@ -1378,6 +1380,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec,
 	return node_page_state(lruvec_pgdat(lruvec), idx);
 }
 
+static inline unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+							enum node_stat_item idx)
+{
+	return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
+}
+
 static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 						    enum node_stat_item idx)
 {
include/linux/vmstat.h +16 -0
index 3c9c266cf782..fb8c76289e02 100644
@@ -194,6 +194,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item)
 	return x;
 }
 
+/*
+ * Non-clamping variant of global_node_page_state() intended for callers that
+ * snapshot a monotonically-incremented counter and subtract two samples.
+ * Returns the raw wrapping value so that unsigned modular subtraction stays
+ * correct across a signed-long overflow (a real hazard on 32-bit) that the
+ * clamp in global_node_page_state() would otherwise turn into a huge spurious
+ * delta. Do NOT use for non-monotonic page-count reads.
+ */
+static inline unsigned long global_node_page_state_monotonic(enum node_stat_item item)
+{
+	return (unsigned long)atomic_long_read(&vm_node_stat[item]);
+}
+
 static inline unsigned long global_node_page_state(enum node_stat_item item)
 {
 	VM_WARN_ON_ONCE(vmstat_item_in_bytes(item));
@@ -259,11 +272,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat,
 						enum node_stat_item item);
 extern unsigned long node_page_state_pages(struct pglist_data *pgdat,
 					   enum node_stat_item item);
+extern unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
+					       enum node_stat_item item);
 extern void fold_vm_numa_events(void);
 #else
 #define sum_zone_node_page_state(node, item) global_zone_page_state(item)
 #define node_page_state(node, item) global_node_page_state(item)
 #define node_page_state_pages(node, item) global_node_page_state_pages(item)
+#define node_page_state_monotonic(node, item) global_node_page_state_monotonic(item)
 static inline void fold_vm_numa_events(void)
 {
 }
mm/memcontrol.c +36 -0
index 56cd4af08232..b4a357c5f7e0 100644
@@ -502,6 +502,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)
 	return x;
 }
 
+/**
+ * lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling
+ * @lruvec: the LRU vector to read from
+ * @idx: the node_stat_item to read
+ *
+ * Returns the raw state[idx] value cast to unsigned long, skipping the
+ * clamp-negative-to-zero step in lruvec_page_state(). Intended for callers
+ * that snapshot a monotonically-incremented counter and subtract two
+ * samples: unsigned modular arithmetic then yields the correct delta across
+ * a signed-long wraparound (a real hazard on 32-bit) that the clamp would
+ * otherwise turn into a huge spurious delta.
+ *
+ * Do NOT use for non-monotonic page-count reads where a transient negative
+ * reading from per-CPU delta skew must present as zero.
+ *
+ * XXX: This helper (and its node/global peers) exists because we place
+ * monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*)
+ * into enum node_stat_item.
+ */
+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+					  enum node_stat_item idx)
+{
+	struct mem_cgroup_per_node *pn;
+	int i;
+
+	if (mem_cgroup_disabled())
+		return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
+
+	i = memcg_stats_index(idx);
+	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
+		return 0;
+
+	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
+	return (unsigned long)READ_ONCE(pn->lruvec_stats->state[i]);
+}
+
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx)
 {
mm/vmstat.c +11 -0
index f534972f517d..c4364f0eb08a 100644
@@ -1024,6 +1024,17 @@ unsigned long node_page_state(struct pglist_data *pgdat,
 
 	return node_page_state_pages(pgdat, item);
 }
+
+/*
+ * Non-clamping variant of node_page_state() intended for callers that
+ * snapshot a monotonically-incremented counter and subtract two samples.
+ * See global_node_page_state_monotonic() for the rationale.
+ */
+unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
+					enum node_stat_item item)
+{
+	return (unsigned long)atomic_long_read(&pgdat->vm_stat[item]);
+}
 #endif
 
 /*
Signature · 1 line
2.53.0-Meta
author
#2 lore raw

PATCH v3 2/2 mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

In reply to Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

In prepare_scan_control() the raw cost signals are recomputed without lru_lock as:

anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX

The delta against a per-lruvec prev_cost[] snapshot is folded into cost_accum[]. Since one vmstat delta can cover many producer events between reclaim passes, halve cost_accum[] until the total is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the prev_cost RMW, the accumulator update, and the halving check against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost_accum[] instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

---
 include/linux/mmzone.h | 10 ++++-
 include/linux/swap.h   |  3 --
 include/linux/vmstat.h |  1 -
 mm/memcontrol-v1.c     |  4 +-
 mm/memcontrol.c        |  5 ++-
 mm/mmzone.c            |  1 +
 mm/swap.c              | 69 ----------------------------------
 mm/vmscan.c            | 84 +++++++++++++++++++++++++++++++++++-------
 mm/vmstat.c            |  2 +
 mm/workingset.c        |  5 ---
 10 files changed, 87 insertions(+), 97 deletions(-)
include/linux/mmzone.h +8 -2
index ca2712187147..3ac467959e34 100644
@@ -323,6 +323,8 @@ enum node_stat_item {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
+	PGROTATE_ANON,
+	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
@@ -763,9 +765,13 @@ struct lruvec {
 	 * These track the cost of reclaiming one LRU - file or anon -
 	 * over the other. As the observed cost of reclaiming one LRU
 	 * increases, the reclaim scan balance tips toward the other.
+	 * Updated and decayed at prepare_scan_control() time; cost_lock
+	 * serialises that update.
 	 */
-	unsigned long			anon_cost;
-	unsigned long			file_cost;
+	unsigned long			prev_cost[ANON_AND_FILE];
+	unsigned long			cost_accum[ANON_AND_FILE];
+	/* Protects prev_cost and cost_accum. */
+	spinlock_t			cost_lock;
 	/* Non-resident age, driven by LRU movement */
 	atomic_long_t			nonresident_age;
 	/* Refaults at the time of last reclaim cycle */
include/linux/swap.h +0 -3
index 6d72778e6cc3..d35a4761ebd7 100644
@@ -309,9 +309,6 @@ extern unsigned long totalreserve_pages;
 
 
 /* linux/mm/swap.c */
-void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
-		unsigned int nr_io, unsigned int nr_rotated);
-void lru_note_cost_refault(struct folio *);
 void folio_add_lru(struct folio *);
 void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
include/linux/vmstat.h +0 -1
index fb8c76289e02..5b31d8e7ae40 100644
@@ -20,7 +20,6 @@ struct reclaim_stat {
 	unsigned nr_congested;
 	unsigned nr_writeback;
 	unsigned nr_immediate;
-	unsigned nr_pageout;
 	unsigned nr_activate[ANON_AND_FILE];
 	unsigned nr_ref_keep;
 	unsigned nr_unmap_fail;
mm/memcontrol-v1.c +2 -2
index 765069211567..c7a52bb68f4c 100644
@@ -1988,8 +1988,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 		for_each_online_pgdat(pgdat) {
 			mz = memcg->nodeinfo[pgdat->node_id];
 
-			anon_cost += mz->lruvec.anon_cost;
-			file_cost += mz->lruvec.file_cost;
+			anon_cost += mz->lruvec.cost_accum[WORKINGSET_ANON];
+			file_cost += mz->lruvec.cost_accum[WORKINGSET_FILE];
 		}
 		seq_buf_printf(s, "anon_cost %lu\n", anon_cost);
 		seq_buf_printf(s, "file_cost %lu\n", file_cost);
mm/memcontrol.c +4 -1
index b4a357c5f7e0..8693aad26ca2 100644
@@ -393,6 +393,7 @@ static const unsigned int memcg_node_stat_items[] = {
 	NR_SHMEM_THPS,
 	NR_FILE_THPS,
 	NR_ANON_THPS,
+	NR_VMSCAN_WRITE,
 	NR_VMALLOC,
 	NR_KERNEL_STACK_KB,
 	NR_PAGETABLE,
@@ -419,6 +420,8 @@ static const unsigned int memcg_node_stat_items[] = {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
+	PGROTATE_ANON,
+	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
@@ -518,7 +521,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)
  * reading from per-CPU delta skew must present as zero.
  *
  * XXX: This helper (and its node/global peers) exists because we place
- * monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*)
+ * monotonically-incremented event counters (NR_VMSCAN_WRITE and PGROTATE_*)
  * into enum node_stat_item.
  */
 unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
mm/mmzone.c +1 -0
index 0c8f181d9d50..17139db4d291 100644
@@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec)
 
 	memset(lruvec, 0, sizeof(struct lruvec));
 	spin_lock_init(&lruvec->lru_lock);
+	spin_lock_init(&lruvec->cost_lock);
 	zswap_lruvec_state_init(lruvec);
 
 	for_each_lru(lru)
mm/swap.c +0 -69

81 lines not on this page · raw message

mm/vmscan.c +70 -14

141 lines not on this page · raw message

mm/vmstat.c +2 -0
index c4364f0eb08a..87d4a6781367 100644
@@ -1300,6 +1300,8 @@ const char * const vmstat_text[] = {
 	[I(PGSCAN_PROACTIVE)]			= "pgscan_proactive",
 	[I(PGSCAN_ANON)]			= "pgscan_anon",
 	[I(PGSCAN_FILE)]			= "pgscan_file",
+	[I(PGROTATE_ANON)]			= "pgrotate_anon",
+	[I(PGROTATE_FILE)]			= "pgrotate_file",
 	[I(PGREFILL)]				= "pgrefill",
 #ifdef CONFIG_HUGETLB_PAGE
 	[I(NR_HUGETLB)]				= "nr_hugetlb",
mm/workingset.c +0 -5
index f351798e723a..7ac2b88c80ae 100644
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow)
 	/* Folio was active prior to eviction */
 	if (workingset) {
 		folio_set_workingset(folio);
-		/*
-		 * XXX: Move to folio_add_lru() when it supports new vs
-		 * putback
-		 */
-		lru_note_cost_refault(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr);
 	}
 out:
Signature · 1 line
2.53.0-Meta
maintainer
#3 lore raw
Quoted 18 lines · Usama Arif
mm/vmscan.c +13 -1
@@ -2303,12 +2301,63 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc)
 	mem_cgroup_flush_stats_ratelimited(sc->target_mem_cgroup);
 
 	/*
-	 * Determine the scan balance between anon and file LRUs.
+	 * Determine the scan balance between anon and file LRUs from per-LRU
+	 * vmstat counters. The raw cost per side is:
+	 *
+	 *	PGROTATE	   - reclaim-driven rotations, bumped from both
+	 *			     shrink_inactive_list and shrink_active_list
+	 *			     (CPU work).
+	 *	NR_VMSCAN_WRITE    - reclaim-driven anon pageout IO.
+	 *	WORKINGSET_RESTORE - refaults of previously-workingset pages.
+	 *
+	 * The two IO terms are weighted by SWAP_CLUSTER_MAX to reflect the
+	 * higher cost of an IO over a rotation.
+	 *
+	 * Reads are lock-free per-cpu sum collations, rstat-aggregated up

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2326

Quoted 10 lines
mm/vmscan.c +10 -0
+	 * the memcg hierarchy by mem_cgroup_flush_stats_ratelimited() above.
+	 * Use lruvec_page_state_monotonic() so the unsigned subtraction
+	 * `now - prev_cost[f]` yields the correct delta across a signed-long
+	 * wraparound of the underlying counter (a real hazard on 32-bit that
+	 * the clamp in lruvec_page_state() would otherwise turn into a huge
+	 * spurious delta).
+	 *
+	 * The delta against prev_cost is folded into cost_accum, which is
+	 * halved on both sides until their sum is within lrusize/4.
+	 * cost_lock serialises concurrent reclaimers in the same memcg+node.

IMO that's a lot of describing what the code does. Why not stick closer to the original comments?

Quoted 7 lines
mm/vmscan.c +2 -4
 	 */
-	spin_lock_irq(&target_lruvec->lru_lock);
-	sc->anon_cost = target_lruvec->anon_cost;
-	sc->file_cost = target_lruvec->file_cost;
-	spin_unlock_irq(&target_lruvec->lru_lock);
+	spin_lock(&target_lruvec->cost_lock);
+	for (int f = 0; f <= 1; f++) {

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2339

Quoted 10 lines
mm/vmscan.c +10 -0
+		unsigned long now, delta;
+
+		now = lruvec_page_state_monotonic(target_lruvec, PGROTATE_ANON + f) +
+		      lruvec_page_state_monotonic(target_lruvec,
+						  WORKINGSET_RESTORE_BASE + f) *
+				SWAP_CLUSTER_MAX;
+		if (f == WORKINGSET_ANON)
+			now += lruvec_page_state_monotonic(target_lruvec,
+							   NR_VMSCAN_WRITE) *
+				SWAP_CLUSTER_MAX;

It's hard to prove overflow behavior is correct. I would keep the delta extraction dead simple, then do the weight math on the delta.

Quoted 7 lines
mm/vmscan.c +7 -0
+		delta = now - target_lruvec->prev_cost[f];
+		target_lruvec->prev_cost[f] = now;
+		target_lruvec->cost_accum[f] += delta;
+	}
+	unsigned long lrusize =
+		lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) +
+		lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2356

Quoted 10 lines
mm/vmscan.c +10 -0
+		lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) +
+		lruvec_page_state(target_lruvec, NR_ACTIVE_FILE);
+	unsigned long cost_limit = lrusize / 4;
+
+	while (target_lruvec->cost_accum[WORKINGSET_ANON] > cost_limit ||
+	       target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit ||
+	       target_lruvec->cost_accum[WORKINGSET_ANON] +
+	       target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit) {
+		target_lruvec->cost_accum[WORKINGSET_ANON] /= 2;
+		target_lruvec->cost_accum[WORKINGSET_FILE] /= 2;

Why do you need to check them individually? Between reclaim cycles, there are no scans->rotations. And I don't see how you could get refault events several times the size of the LRU, let alone in excess of ULONG_MAX.

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2360

Quoted 4 lines
mm/vmscan.c +4 -0
+	}
+	sc->anon_cost = target_lruvec->cost_accum[WORKINGSET_ANON];
+	sc->file_cost = target_lruvec->cost_accum[WORKINGSET_FILE];
+	spin_unlock(&target_lruvec->cost_lock);

I realize these long descriptor names make it hard, but I think this can be cleaned up a bit and written in a more idiomatic way.

How about this (totally untested):

struct lru_cost {
	unsigned long count;
	unsigned long last_rotated;
	unsigned long last_io;
};
struct lruvec {
	struct lru_cost cost[2];
};
	struct lru_cost *anon_cost, *file_cost;
	unsigned long lrusize;
	
	/*
	 * Determine the scan balance between anon and file LRUs.
	 *
	 * The cost model is based on rotations, refaults and
	 * reclaim-driven writes (anon only) on each side.
	 *
	 * These event counters are monotonic, so each reclaim cycle
	 * the delta since the last scan is extracted and incorporated
	 * into a decaying average. This ensures currency, as workloads
	 * change over time, and avoids overflow in the calculations.
	 */
	spin_lock_irq(&target_lruvec->cost_lock);

	for (file = 0; file <= 1; file++) {
		struct lru_cost *cost = &target_lruvec->cost[file];
		unsigned long rotated, io, nr_rotated, nr_io;

		rotated = lruvec_page_state_monotonic(target_lruvec,
						PGROTATE_ANON + file);
		io = lruvec_page_state_monotonic(target_lruvec,
						WORKINGSET_RESTORE_BASE + file);
		if (!file)
			io += lruvec_page_state_monotonic(target_lruvec,
						NR_VMSCAN_WRITE);

		nr_rotated = rotated - cost->last_rotated;
		nr_io = io - cost->last_io;

		/*
		 * Reflect the relative cost of incurring IO and spending CPU
		 * time on rotations. This doesn't attempt to make a precise
		 * comparison, it just says: if reloads are about comparable
		 * between the LRU lists, or rotations are overwhelmingly
		 * different between them, adjust scan balance for CPU work.
		 */
		cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated;

		cost->last_rotated = rotated;
		cost->last_io = io;
	}

	anon_cost = &target_lruvec->cost[WORKINGSET_ANON];
	file_cost = &target_lruvec->cost[WORKINGSET_FILE];

	lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) +
		  lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +
		  lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) +
		  lruvec_page_state(target_lruvec, NR_ACTIVE_FILE);

	while (anon_cost->count + file_cost->count > lrusize / 4) {
		anon_cost->count /= 2;
		file_cost->count /= 2;
	}

	sc->anon_cost = anon_cost->count;
	sc->file_cost = file_cost->count;

	spin_unlock(&target_lruvec->cost_lock);
author
#4 lore raw
Quoted 77 lines · On Fri, 17 Jul 2026 23:08:08 +0200 Johannes Weiner <[email protected]> wrote:
Quoted 18 lines · Usama Arif
mm/vmscan.c +13 -1
@@ -2303,12 +2301,63 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc)
 	mem_cgroup_flush_stats_ratelimited(sc->target_mem_cgroup);
 
 	/*
-	 * Determine the scan balance between anon and file LRUs.
+	 * Determine the scan balance between anon and file LRUs from per-LRU
+	 * vmstat counters. The raw cost per side is:
+	 *
+	 *	PGROTATE	   - reclaim-driven rotations, bumped from both
+	 *			     shrink_inactive_list and shrink_active_list
+	 *			     (CPU work).
+	 *	NR_VMSCAN_WRITE    - reclaim-driven anon pageout IO.
+	 *	WORKINGSET_RESTORE - refaults of previously-workingset pages.
+	 *
+	 * The two IO terms are weighted by SWAP_CLUSTER_MAX to reflect the
+	 * higher cost of an IO over a rotation.
+	 *
+	 * Reads are lock-free per-cpu sum collations, rstat-aggregated up

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2326

Quoted 10 lines
mm/vmscan.c +10 -0
+	 * the memcg hierarchy by mem_cgroup_flush_stats_ratelimited() above.
+	 * Use lruvec_page_state_monotonic() so the unsigned subtraction
+	 * `now - prev_cost[f]` yields the correct delta across a signed-long
+	 * wraparound of the underlying counter (a real hazard on 32-bit that
+	 * the clamp in lruvec_page_state() would otherwise turn into a huge
+	 * spurious delta).
+	 *
+	 * The delta against prev_cost is folded into cost_accum, which is
+	 * halved on both sides until their sum is within lrusize/4.
+	 * cost_lock serialises concurrent reclaimers in the same memcg+node.

IMO that's a lot of describing what the code does. Why not stick closer to the original comments?

Quoted 7 lines
mm/vmscan.c +2 -4
 	 */
-	spin_lock_irq(&target_lruvec->lru_lock);
-	sc->anon_cost = target_lruvec->anon_cost;
-	sc->file_cost = target_lruvec->file_cost;
-	spin_unlock_irq(&target_lruvec->lru_lock);
+	spin_lock(&target_lruvec->cost_lock);
+	for (int f = 0; f <= 1; f++) {

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2339

Quoted 10 lines
mm/vmscan.c +10 -0
+		unsigned long now, delta;
+
+		now = lruvec_page_state_monotonic(target_lruvec, PGROTATE_ANON + f) +
+		      lruvec_page_state_monotonic(target_lruvec,
+						  WORKINGSET_RESTORE_BASE + f) *
+				SWAP_CLUSTER_MAX;
+		if (f == WORKINGSET_ANON)
+			now += lruvec_page_state_monotonic(target_lruvec,
+							   NR_VMSCAN_WRITE) *
+				SWAP_CLUSTER_MAX;

It's hard to prove overflow behavior is correct. I would keep the delta extraction dead simple, then do the weight math on the delta.

Quoted 7 lines
mm/vmscan.c +7 -0
+		delta = now - target_lruvec->prev_cost[f];
+		target_lruvec->prev_cost[f] = now;
+		target_lruvec->cost_accum[f] += delta;
+	}
+	unsigned long lrusize =
+		lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) +
+		lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2356

Quoted 10 lines
mm/vmscan.c +10 -0
+		lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) +
+		lruvec_page_state(target_lruvec, NR_ACTIVE_FILE);
+	unsigned long cost_limit = lrusize / 4;
+
+	while (target_lruvec->cost_accum[WORKINGSET_ANON] > cost_limit ||
+	       target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit ||
+	       target_lruvec->cost_accum[WORKINGSET_ANON] +
+	       target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit) {
+		target_lruvec->cost_accum[WORKINGSET_ANON] /= 2;
+		target_lruvec->cost_accum[WORKINGSET_FILE] /= 2;

Why do you need to check them individually? Between reclaim cycles, there are no scans->rotations. And I don't see how you could get refault events several times the size of the LRU, let alone in excess of ULONG_MAX.

Quoted 8 lines

mm/vmscan.c static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · line 2360

Quoted 4 lines
mm/vmscan.c +4 -0
+	}
+	sc->anon_cost = target_lruvec->cost_accum[WORKINGSET_ANON];
+	sc->file_cost = target_lruvec->cost_accum[WORKINGSET_FILE];
+	spin_unlock(&target_lruvec->cost_lock);

I realize these long descriptor names make it hard, but I think this can be cleaned up a bit and written in a more idiomatic way.

Thanks for the prototype and the above reivew comments!

I integrated all of them and sent them as v4.

And best part of it is, sashiko is happy as well now with v4 lol