Skip to contentSkip to conversation

mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

PATCH Superseded Usama Arif 4 participants 13 messages Jul 20 – Jul 27

Subsystems (8)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v4 cover mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:

- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.

This series removes those producer-side acquisitions entirely. The rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters. NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout(); charge it through lruvec_stat_mod_folio() so it is available per lruvec and aggregated through the memcg hierarchy. Reclaim does not submit filesystem folios for writeback from this path, so pageout contributes only to anon cost. WORKINGSET_RESTORE_* already captures the refault input.

prepare_scan_control() reads the raw cost signals without lru_lock:

anon = PGROTATE_ANON +
(NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX
file = PGROTATE_FILE +
WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX

It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

Trade-offs:
- Cost reads see rstat-aggregated values that can lag until periodic / reader-triggered flushing.
- Per-lruvec footprint grows by 4 unsigned longs + a spinlock (a struct lru_cost { count, last_rotated, last_io } per side), which is a small cost.
- NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat, adding memcg stat accounting to the reclaim writeout path while preserving the existing node-level total.

== Numbers ==

Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem:

  usemem -n 16 --prealloc --prefault --random $((256*1024*1024))

run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.

Workload rates are identical on both kernels (the bench drives the same memory pressure):

                          baseline    patched      delta
  pgscan_direct  / s      172,662     171,817      ~0%
  pgsteal_direct / s       67,162      66,306      ~0%
  workingset_refault_anon / s
                           40,696      39,830      ~0%

perf lock contention (total wait per 30 s window):

  Lock Name                Before      After     % change
  shrink_lruvec+0x770     722.84 ms    0         -100% (eliminated)
        (= lru_note_cost_unlock_irq)
  workingset_refault+0x167 385.26 ms   0         -100% (eliminated)
        (= lru_note_cost_refault)
  shrink_node+0x4ad       689.43 ms    26.95 ms  -96%
  shrink_active_list      208.34 ms    15.97 ms  -92%
  lru_add_drain_cpu+0x34    1.96 s    917.71 ms  -53%

Total LRU lock wait ~4.23 s ~1.66 s -61%

The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk.

The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec.

The numbers above can be reproduced using the script in [1].

== Series structure ==

1. mm/vmstat, mm/memcontrol: add _monotonic vmstat readers Non-clamping siblings of lruvec_page_state() / node_page_state() / global_node_page_state() for callers that snapshot a monotonically-incremented counter and subtract two samples. Unsigned modular subtraction on the raw wrapping value stays correct across a signed-long overflow that the existing clamp would otherwise turn into a huge spurious delta (a real hazard on 32-bit).

2. mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
The optimization described above; samples PGROTATE_*, NR_VMSCAN_WRITE (for anon) and WORKINGSET_RESTORE_* via lruvec_page_state_monotonic() from patch 1. MGLRU also updates PGROTATE_* so the counters remain meaningful with either reclaim implementation, although MGLRU does not consume this cost signal.

== Alternatives considered ==

1. cost_lock for both producer and consumer (no vmstat indirection):
Keep the producer loop, just swap lru_lock for a new per-lruvec cost_lock. Decouples cost from LRU manipulation, but producers still synchronously contend on cost_lock, the parent_lruvec() walk is still required (O(memcg-depth) acquisitions per recording, now on cost_lock), and lru_reparent_memcg() still needs explicit cost-splice. We can do much better and this series removes the producer lock entirely and gets hierarchy propagation for "free" via rstat.

2. Attempt to switch to using MGLRU's scan model:
MGLRU has no anon_cost/file_cost at all. It replaces the cost heuristic with generation-based aging: per-LRU sequence numbers (min_seq/max_seq) age folios into generations, and the older-generation type is the one to scan. So lru_note_cost_unlock_irq() / lru_note_cost_refault() are simply not called when lru_gen_enabled() — by design it sidesteps every concern this patch addresses.
But MGLRU is not a substitute for fixing classic LRU:
- It relies on a lot of things including per-lruvec generation lists, bloom filters, mm_struct walk infrastructure, working-set protection tiers and a whole sysfs interface. Replacing classic LRU's cost recording with the MGLRU model would mean dragging in all of that.
- It changes scan-balance semantics, not just the locking, so it's a heuristic change we would need to evaluate separately. There are known regressions (database/anon-heavy workloads sensitive to swappiness, or file-cache-dominated workloads where MGLRU's bloom-filter protection differs from classic refault tracking).
This series keeps classic LRU's scan-balance model and fixes where its cost history is sampled and aged.

3. Atomic cost counter:
lrusize/4 halving has no clean atomic form, and the parent walk still has to run explicitly. Reusing vmstats gives per-CPU aggregation AND rstat hierarchy propagation for free.

4. Drop cost_lock from the existing patch and reuse lru_lock in the consumer (prepare_scan_control()):
Saves 1 lock space per lruvec but re-couples the cost path to LRU manipulation, though just from the consumer side this time. prepare_scan_control() runs at the start of every shrink_lruvec() cycle, so under sustained memory pressure it would take lru_lock on the hot path and block isolate_lru_folios() / move_folios_to_lru() / folio_add_lru() i.e. when reclaim is in flight. A dedicated cost_lock is never taken by anyone except the consumer cost calculation.

[1] https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884

v3 -> v4: https://lore.kernel.org/all/[email protected]/
- Fold the two per-lruvec cost arrays (prev_cost[], cost_accum[]) into a struct lru_cost { count, last_rotated, last_io } cost[ANON_AND_FILE] in struct lruvec. (Johannes)
- Sample rotated and io separately in prepare_scan_control() and compute the raw per-counter deltas before applying the SWAP_CLUSTER_MAX weighting, rather than sampling a pre-weighted sum and taking a delta from it. Easier to reason about the overflow behaviour. (Johannes)
- Drop the per-side checks in the halving loop; anon > limit implies anon + file > limit for unsigned counters, so only the sum check
is needed. (Johannes)
- Trim the prepare_scan_control() comment back to something close to the original wording. (Johannes)

v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Reuse NR_VMSCAN_WRITE for anon pageout cost instead of adding PGRECLAIM_PAGEOUT_{ANON,FILE}. Reclaim no longer writes filesystem folios from this path, so there is no file pageout cost. (Johannes)
- Charge NR_VMSCAN_WRITE through the lruvec/memcg stats and remove the now-unused reclaim_stat.nr_pageout plumbing.
- Account PGROTATE_{ANON,FILE} in the MGLRU eviction path so the new counters remain meaningful regardless of the active LRU implementation. (sashiko)
- MGLRU updates PGROTATE_{ANON,FILE} so the counters remain meaningful with it. (sashiko)

v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Sample via the newly introduced lruvec_page_state_monotonic()
to fix a 32-bit delta underflow when the underlying signed
long wraps past LONG_MAX (Johannes and sashiko)

RFC -> v1: https://lore.kernel.org/all/[email protected]/
- Document in coverletter and commit message how the read-side vmstat accumulator improves cost-model aging across reclaim gaps (Johannes)
- Fully decay the cost_accum below lrusize / 4 using a while loop (sashiko)

Usama Arif (2):
mm/vmstat, mm/memcontrol: add _monotonic vmstat readers mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance
cost

 include/linux/memcontrol.h |  8 ++++
 include/linux/mmzone.h     | 15 +++++-
 include/linux/swap.h       |  3 --
 include/linux/vmstat.h     | 17 ++++++-
 mm/memcontrol-v1.c         |  4 +-
 mm/memcontrol.c            | 39 ++++++++++++++++
 mm/mmzone.c                |  1 +
 mm/swap.c                  | 69 ----------------------------
 mm/vmscan.c                | 93 ++++++++++++++++++++++++++++++++------
 mm/vmstat.c                | 13 ++++++
 mm/workingset.c            |  5 --
 11 files changed, 171 insertions(+), 96 deletions(-)
Signature · 1 line
2.53.0-Meta
author
#1 lore raw

PATCH v4 1/2 mm/vmstat, mm/memcontrol: add _monotonic vmstat readers

lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than
as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers.

It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator.

Add non-clamping siblings that return the underlying state value cast to unsigned long:

global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()

With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper.

The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero.

This is a prerequisite for the following patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds the unsigned modular delta into a per-lruvec cost_accum[].

---
 include/linux/memcontrol.h |  8 ++++++++
 include/linux/vmstat.h     | 16 ++++++++++++++++
 mm/memcontrol.c            | 36 ++++++++++++++++++++++++++++++++++++
 mm/vmstat.c                | 11 +++++++++++
 4 files changed, 71 insertions(+)
include/linux/memcontrol.h +8 -0
index e1f46a0016fc..b40bc4f6fe4a 100644
@@ -931,6 +931,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
 bool memcg_stat_item_valid(int idx);
 bool memcg_vm_event_item_valid(enum vm_event_item idx);
 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+					  enum node_stat_item idx);
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx);
 
@@ -1378,6 +1380,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec,
 	return node_page_state(lruvec_pgdat(lruvec), idx);
 }
 
+static inline unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+							enum node_stat_item idx)
+{
+	return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
+}
+
 static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 						    enum node_stat_item idx)
 {
include/linux/vmstat.h +16 -0
index 3c9c266cf782..fb8c76289e02 100644
@@ -194,6 +194,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item)
 	return x;
 }
 
+/*
+ * Non-clamping variant of global_node_page_state() intended for callers that
+ * snapshot a monotonically-incremented counter and subtract two samples.
+ * Returns the raw wrapping value so that unsigned modular subtraction stays
+ * correct across a signed-long overflow (a real hazard on 32-bit) that the
+ * clamp in global_node_page_state() would otherwise turn into a huge spurious
+ * delta. Do NOT use for non-monotonic page-count reads.
+ */
+static inline unsigned long global_node_page_state_monotonic(enum node_stat_item item)
+{
+	return (unsigned long)atomic_long_read(&vm_node_stat[item]);
+}
+
 static inline unsigned long global_node_page_state(enum node_stat_item item)
 {
 	VM_WARN_ON_ONCE(vmstat_item_in_bytes(item));
@@ -259,11 +272,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat,
 						enum node_stat_item item);
 extern unsigned long node_page_state_pages(struct pglist_data *pgdat,
 					   enum node_stat_item item);
+extern unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
+					       enum node_stat_item item);
 extern void fold_vm_numa_events(void);
 #else
 #define sum_zone_node_page_state(node, item) global_zone_page_state(item)
 #define node_page_state(node, item) global_node_page_state(item)
 #define node_page_state_pages(node, item) global_node_page_state_pages(item)
+#define node_page_state_monotonic(node, item) global_node_page_state_monotonic(item)
 static inline void fold_vm_numa_events(void)
 {
 }
mm/memcontrol.c +36 -0
index 56cd4af08232..b4a357c5f7e0 100644
@@ -502,6 +502,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)
 	return x;
 }
 
+/**
+ * lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling
+ * @lruvec: the LRU vector to read from
+ * @idx: the node_stat_item to read
+ *
+ * Returns the raw state[idx] value cast to unsigned long, skipping the
+ * clamp-negative-to-zero step in lruvec_page_state(). Intended for callers
+ * that snapshot a monotonically-incremented counter and subtract two
+ * samples: unsigned modular arithmetic then yields the correct delta across
+ * a signed-long wraparound (a real hazard on 32-bit) that the clamp would
+ * otherwise turn into a huge spurious delta.
+ *
+ * Do NOT use for non-monotonic page-count reads where a transient negative
+ * reading from per-CPU delta skew must present as zero.
+ *
+ * XXX: This helper (and its node/global peers) exists because we place
+ * monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*)
+ * into enum node_stat_item.
+ */
+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
+					  enum node_stat_item idx)
+{
+	struct mem_cgroup_per_node *pn;
+	int i;
+
+	if (mem_cgroup_disabled())
+		return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
+
+	i = memcg_stats_index(idx);
+	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
+		return 0;
+
+	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
+	return (unsigned long)READ_ONCE(pn->lruvec_stats->state[i]);
+}
+
 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
 				      enum node_stat_item idx)
 {
mm/vmstat.c +11 -0
index f534972f517d..c4364f0eb08a 100644
@@ -1024,6 +1024,17 @@ unsigned long node_page_state(struct pglist_data *pgdat,
 
 	return node_page_state_pages(pgdat, item);
 }
+
+/*
+ * Non-clamping variant of node_page_state() intended for callers that
+ * snapshot a monotonically-incremented counter and subtract two samples.
+ * See global_node_page_state_monotonic() for the rationale.
+ */
+unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
+					enum node_stat_item item)
+{
+	return (unsigned long)atomic_long_read(&pgdat->vm_stat[item]);
+}
 #endif
 
 /*
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v4 1/2
#2 lore raw

Re: [PATCH v4 1/2] mm/vmstat, mm/memcontrol: add _monotonic vmstat readers

Quoted 38 lines · Usama Arif

lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than
as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers.

It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator.

Add non-clamping siblings that return the underlying state value cast to unsigned long:

global_node_page_state_monotonic()
node_page_state_monotonic()
lruvec_page_state_monotonic()

With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper.

The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero.

This is a prerequisite for the following patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds the unsigned modular delta into a per-lruvec cost_accum[].

Quoted 2 lines
author
#3 lore raw

PATCH v4 2/2 mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

In reply to Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

---
 include/linux/mmzone.h | 15 ++++++-
 include/linux/swap.h   |  3 --
 include/linux/vmstat.h |  1 -
 mm/memcontrol-v1.c     |  4 +-
 mm/memcontrol.c        |  5 ++-
 mm/mmzone.c            |  1 +
 mm/swap.c              | 69 -------------------------------
 mm/vmscan.c            | 93 +++++++++++++++++++++++++++++++++++-------
 mm/vmstat.c            |  2 +
 mm/workingset.c        |  5 ---
 10 files changed, 101 insertions(+), 97 deletions(-)
include/linux/mmzone.h +13 -2
index ca2712187147..85303c5867c8 100644
@@ -323,6 +323,8 @@ enum node_stat_item {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
+	PGROTATE_ANON,
+	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
@@ -755,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent,
 
 #endif /* CONFIG_LRU_GEN */
 
+struct lru_cost {
+	unsigned long		count;
+	unsigned long		last_rotated;
+	unsigned long		last_io;
+};
+
 struct lruvec {
 	struct list_head		lists[NR_LRU_LISTS];
 	/* per lruvec lru_lock for memcg */
@@ -763,9 +771,12 @@ struct lruvec {
 	 * These track the cost of reclaiming one LRU - file or anon -
 	 * over the other. As the observed cost of reclaiming one LRU
 	 * increases, the reclaim scan balance tips toward the other.
+	 * Updated and decayed at prepare_scan_control() time; cost_lock
+	 * serialises that update.
 	 */
-	unsigned long			anon_cost;
-	unsigned long			file_cost;
+	struct lru_cost			cost[ANON_AND_FILE];
+	/* Protects cost[]. */
+	spinlock_t			cost_lock;
 	/* Non-resident age, driven by LRU movement */
 	atomic_long_t			nonresident_age;
 	/* Refaults at the time of last reclaim cycle */
include/linux/swap.h +0 -3
index 6d72778e6cc3..d35a4761ebd7 100644
@@ -309,9 +309,6 @@ extern unsigned long totalreserve_pages;
 
 
 /* linux/mm/swap.c */
-void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
-		unsigned int nr_io, unsigned int nr_rotated);
-void lru_note_cost_refault(struct folio *);
 void folio_add_lru(struct folio *);
 void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
include/linux/vmstat.h +0 -1
index fb8c76289e02..5b31d8e7ae40 100644
@@ -20,7 +20,6 @@ struct reclaim_stat {
 	unsigned nr_congested;
 	unsigned nr_writeback;
 	unsigned nr_immediate;
-	unsigned nr_pageout;
 	unsigned nr_activate[ANON_AND_FILE];
 	unsigned nr_ref_keep;
 	unsigned nr_unmap_fail;
mm/memcontrol-v1.c +2 -2
index 765069211567..091bc9ffee44 100644
@@ -1988,8 +1988,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
 		for_each_online_pgdat(pgdat) {
 			mz = memcg->nodeinfo[pgdat->node_id];
 
-			anon_cost += mz->lruvec.anon_cost;
-			file_cost += mz->lruvec.file_cost;
+			anon_cost += mz->lruvec.cost[WORKINGSET_ANON].count;
+			file_cost += mz->lruvec.cost[WORKINGSET_FILE].count;
 		}
 		seq_buf_printf(s, "anon_cost %lu\n", anon_cost);
 		seq_buf_printf(s, "file_cost %lu\n", file_cost);
mm/memcontrol.c +4 -1
index b4a357c5f7e0..8693aad26ca2 100644
@@ -393,6 +393,7 @@ static const unsigned int memcg_node_stat_items[] = {
 	NR_SHMEM_THPS,
 	NR_FILE_THPS,
 	NR_ANON_THPS,
+	NR_VMSCAN_WRITE,
 	NR_VMALLOC,
 	NR_KERNEL_STACK_KB,
 	NR_PAGETABLE,
@@ -419,6 +420,8 @@ static const unsigned int memcg_node_stat_items[] = {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
+	PGROTATE_ANON,
+	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
@@ -518,7 +521,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)
  * reading from per-CPU delta skew must present as zero.
  *
  * XXX: This helper (and its node/global peers) exists because we place
- * monotonically-incremented event counters (PGROTATE_*, PGRECLAIM_PAGEOUT_*)
+ * monotonically-incremented event counters (NR_VMSCAN_WRITE and PGROTATE_*)
  * into enum node_stat_item.
  */
 unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
mm/mmzone.c +1 -0
index 0c8f181d9d50..17139db4d291 100644
@@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec)
 
 	memset(lruvec, 0, sizeof(struct lruvec));
 	spin_lock_init(&lruvec->lru_lock);
+	spin_lock_init(&lruvec->cost_lock);
 	zswap_lruvec_state_init(lruvec);
 
 	for_each_lru(lru)
mm/swap.c +0 -69

81 lines not on this page · raw message

mm/vmscan.c +79 -14

157 lines not on this page · raw message

mm/vmstat.c +2 -0
index c4364f0eb08a..87d4a6781367 100644
@@ -1300,6 +1300,8 @@ const char * const vmstat_text[] = {
 	[I(PGSCAN_PROACTIVE)]			= "pgscan_proactive",
 	[I(PGSCAN_ANON)]			= "pgscan_anon",
 	[I(PGSCAN_FILE)]			= "pgscan_file",
+	[I(PGROTATE_ANON)]			= "pgrotate_anon",
+	[I(PGROTATE_FILE)]			= "pgrotate_file",
 	[I(PGREFILL)]				= "pgrefill",
 #ifdef CONFIG_HUGETLB_PAGE
 	[I(NR_HUGETLB)]				= "nr_hugetlb",
mm/workingset.c +0 -5
index f351798e723a..7ac2b88c80ae 100644
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow)
 	/* Folio was active prior to eviction */
 	if (workingset) {
 		folio_set_workingset(folio);
-		/*
-		 * XXX: Move to folio_add_lru() when it supports new vs
-		 * putback
-		 */
-		lru_note_cost_refault(folio);
 		mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr);
 	}
 out:
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v4 2/2
#4 lore raw
Quoted 18 lines · Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

Quoted 6 lines

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

vvv

Quoted 18 lines

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Quoted 14 lines

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

^^^

This is a quite a bit of explaining the code, which is kind of drowning out what's really going on. How about something like:

---

The balance formula for anon and file, respectively, is this:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run.

WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE already exist and have been accounted all along; PGROTATE_* are added for replacing the rotation event callbacks.

This is overall cheaper and has fewer lock aquisition sites.

---

Quoted 39 lines

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

Quoted 1 line

Otherwise, looks good to me.

maintainer
#5 lore raw
In reply to Usama Arif
Quoted 97 lines · Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

Quoted 1 line

Oh man too long commit message, please follow Johannes's suggestion.

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

BTW really awesome work and I really like how these patches have reduced the lru lock contention drastically.

author
#6 lore raw
Quoted 98 lines · Shakeel Butt
Quoted 97 lines · Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

Quoted 3 lines
Quoted 1 line

Oh man too long commit message, please follow Johannes's suggestion.

Will do!

Quoted 8 lines

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Quoted 7 lines

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

I do think the userspace counters have diagnostic value. They can be another metric to show how much anon/file reclaim work is spent on pages that are not immediately reclaimed, including pages retained on actie LRU. It could help explain if there is elevated reclaim CPU usage.

But tbf, I don't have a concrete usecase, and I wouldn't put up a patch to expose them to userspace if it werent for this patch itself.

I definitely dont want to create another hidden lruvec rstat infrastructure for this, it would be too much code churn. If the stats really shouldn't be exposed to userspace, would something like below be acceptable?

drivers/base/node.c +1 -1
index 3da91929ad4e..3321d4e62234 100644
@@ -612,7 +612,7 @@ static ssize_t node_read_vmstat(struct device *dev,
 				     sum_zone_numa_event_state(nid, i));
 
 #endif
-	for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) {
+	for (i = 0; i < NR_VM_NODE_STAT_ITEMS_PUBLIC; i++) {
 		unsigned long pages = node_page_state_pages(pgdat, i);
 
 		if (vmstat_item_print_in_thp(i))
include/linux/mmzone.h +4 -2
index 85303c5867c8..ada3b136a65d 100644
@@ -323,8 +323,6 @@ enum node_stat_item {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
-	PGROTATE_ANON,
-	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
@@ -333,6 +331,10 @@ enum node_stat_item {
 	NR_KERNEL_FILE_PAGES,
 	NR_GPU_ACTIVE,	/* Pages assigned to GPU objects */
 	NR_GPU_RECLAIM,	/* Pages in shrinkable GPU pools */
+	NR_VM_NODE_STAT_ITEMS_PUBLIC,
+	/* Internal-only node stats must remain at the end. */
+	PGROTATE_ANON = NR_VM_NODE_STAT_ITEMS_PUBLIC,
+	PGROTATE_FILE,
 	NR_VM_NODE_STAT_ITEMS
 };
 
mm/memcontrol.c +2 -2
index 8693aad26ca2..fd1960eed732 100644
@@ -420,12 +420,12 @@ static const unsigned int memcg_node_stat_items[] = {
 	PGSCAN_PROACTIVE,
 	PGSCAN_ANON,
 	PGSCAN_FILE,
-	PGROTATE_ANON,
-	PGROTATE_FILE,
 	PGREFILL,
 #ifdef CONFIG_HUGETLB_PAGE
 	NR_HUGETLB,
 #endif
+	PGROTATE_ANON,
+	PGROTATE_FILE,
 };
 
 static const unsigned int memcg_stat_items[] = {
mm/vmstat.c +9 -4
index 87d4a6781367..6b14136a6e51 100644
@@ -1300,8 +1300,6 @@ const char * const vmstat_text[] = {
 	[I(PGSCAN_PROACTIVE)]			= "pgscan_proactive",
 	[I(PGSCAN_ANON)]			= "pgscan_anon",
 	[I(PGSCAN_FILE)]			= "pgscan_file",
-	[I(PGROTATE_ANON)]			= "pgrotate_anon",
-	[I(PGROTATE_FILE)]			= "pgrotate_file",
 	[I(PGREFILL)]				= "pgrefill",
 #ifdef CONFIG_HUGETLB_PAGE
 	[I(NR_HUGETLB)]				= "nr_hugetlb",
@@ -1310,6 +1308,8 @@ const char * const vmstat_text[] = {
 	[I(NR_KERNEL_FILE_PAGES)]		= "nr_kernel_file_pages",
 	[I(NR_GPU_ACTIVE)]			= "nr_gpu_active",
 	[I(NR_GPU_RECLAIM)]			= "nr_gpu_reclaim",
+	[I(PGROTATE_ANON)]			= "pgrotate_anon",
+	[I(PGROTATE_FILE)]			= "pgrotate_file",
 #undef I
 
 	/* system-wide enum vm_stat_item counters */
@@ -1771,7 +1771,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,
 	seq_printf(m, "Node %d, zone %8s", pgdat->node_id, zone->name);
 	if (is_zone_first_populated(pgdat, zone)) {
 		seq_printf(m, "\n  per-node stats");
-		for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) {
+		for (i = 0; i < NR_VM_NODE_STAT_ITEMS_PUBLIC; i++) {
 			unsigned long pages = node_page_state_pages(pgdat, i);
 
 			if (vmstat_item_print_in_thp(i))
@@ -1948,6 +1948,11 @@ static int vmstat_show(struct seq_file *m, void *arg)
 {
 	unsigned long *l = arg;
 	unsigned long off = l - (unsigned long *)m->private;
+	unsigned long node_off = NR_VM_ZONE_STAT_ITEMS + NR_VM_NUMA_EVENT_ITEMS;
+
+	if (off >= node_off + NR_VM_NODE_STAT_ITEMS_PUBLIC &&
+	    off < node_off + NR_VM_NODE_STAT_ITEMS)
+		return SEQ_SKIP;
 
 	seq_puts(m, vmstat_text[off]);
 	seq_put_decimal_ull(m, " ", *l);
@@ -2025,7 +2030,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 				__func__, zone_stat_name(i), val);
 		}
 	}
-	for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) {
+	for (i = 0; i < NR_VM_NODE_STAT_ITEMS_PUBLIC; i++) {
 		/*
 		 * Skip checking stats known to go negative occasionally.
 		 */
Quoted 2 lines

BTW really awesome work and I really like how these patches have reduced the lru lock contention drastically.

Thank you!!

maintainer
#7 lore raw
Quoted 22 lines · Usama Arif
Quoted 9 lines

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Quoted 7 lines

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

I do think the userspace counters have diagnostic value. They can be another metric to show how much anon/file reclaim work is spent on pages that are not immediately reclaimed, including pages retained on actie LRU. It could help explain if there is elevated reclaim CPU usage.

Quoted 7 lines

But tbf, I don't have a concrete usecase, and I wouldn't put up a patch to expose them to userspace if it werent for this patch itself.

I definitely dont want to create another hidden lruvec rstat infrastructure for this, it would be too much code churn. If the stats really shouldn't be exposed to userspace, would something like below be acceptable?

Yes, the code seems reasonable to me.

maintainer
#8 lore raw
In reply to Usama Arif
Quoted 122 lines · Usama Arif
Quoted 98 lines · Shakeel Butt
Quoted 97 lines · Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

Quoted 3 lines
Quoted 1 line

Oh man too long commit message, please follow Johannes's suggestion.

Will do!

Quoted 8 lines

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Quoted 7 lines

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

Quoted 4 lines

I do think the userspace counters have diagnostic value. They can be another metric to show how much anon/file reclaim work is spent on pages that are not immediately reclaimed, including pages retained on actie LRU. It could help explain if there is elevated reclaim CPU usage.

Hm, this is an interesting discussion.

The existing PGROTATED is kind of useless. When it's bumped:

- truncation fails and moves folios to the LRU tail for faster reclaim
- writeback ends on folio that was tried to reclaim, moves to LRU tail for fast retry

Ok, who needs to know this? The new counters you're adding, on the other hand, they capture the information you frequently need for understanding reclaim behavior.

That said, looking closer at this one more time, you can already (almost) derive them fully from existing counters:

drotated = (dPGSCAN_ANON - dPGSTEAL_ANON) + dPGDEACTIVATE_ANON

The first two are already split by anon and file. The third you'd have to (1) split by type and (2) stop counting, or have a separate event, for MADV_COLD driven deactivations.

And (2) seems overdue anyway. There is no point in maintaining counts for what userspace madvises. And it adds noise when it gets conflated with a stat tracking kernel reclaim behavior.

maintainer
#9 lore raw
Quoted 135 lines · Johannes Weiner
Quoted 122 lines · Usama Arif
Quoted 98 lines · Shakeel Butt
Quoted 97 lines · Usama Arif

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:

- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two

    scalars into sc->{anon,file}_cost.

- lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.

This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.

Replace the producer-side accumulators with a read-side accumulator fed from per-LRU vmstat counters. The old producer formula was:

cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated

Reuse NR_VMSCAN_WRITE for reclaim-driven anon pageout submissions. It is already bumped by writeout() for the same successful outcome that fed reclaim_stat.nr_pageout. Reclaim does not submit filesystem folios from this path, so there is no file pageout term. Charge NR_VMSCAN_WRITE via lruvec_stat_mod_folio() and include it in memcg_node_stat_items so it can be sampled per lruvec and aggregated through the memcg hierarchy.

Add explicit PGROTATE_{ANON,FILE} node_stat counters for the remaining producer-local input. They are bumped from shrink_inactive_list() by nr_scanned - nr_reclaimed and from shrink_active_list() by nr_rotated. WORKINGSET_RESTORE_{ANON,FILE} already captures the refault IO that lru_note_cost_refault() used to bill.

Add a per-side struct lru_cost { count, last_rotated, last_io } to struct lruvec. In prepare_scan_control() the two monotonic inputs are sampled separately - rotated from PGROTATE_ANON/FILE, io from WORKINGSET_RESTORE_BASE + f plus (for anon) NR_VMSCAN_WRITE - and the raw per-side deltas are computed against cost->last_rotated and cost->last_io before the SWAP_CLUSTER_MAX IO weighting is applied. Extracting the deltas from the individual counters (rather than from a pre-weighted sum) keeps the unsigned modular subtraction bounded by the true per-counter growth, so a signed-long wraparound of any underlying vmstat still yields the correct delta on 32-bit. The weighted delta is folded into cost->count. Since one vmstat delta can cover many producer events between reclaim passes, halve cost->count on both sides until their sum is back within the lrusize/4 bound instead of halving only once.

Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.

A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.

Hierarchy aggregation is now implicit in the vmstat accounting. The producer-side parent_lruvec() walk and lru_reparent_memcg() cost splice existed only because anon_cost/file_cost were private lruvec fields. With the cost expressed as lruvec vmstats, rstat propagates the underlying counters through the memcg hierarchy and prepare_scan_control() consumes the same ratelimited rstat view as the surrounding reclaim heuristics.

NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.

memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.

Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation, and PGROTATE_{ANON,FILE} are bumped from evict_folios() so per-memcg observability of rotation-driven reclaim work stays consistent across both implementations.

Quoted 3 lines
Quoted 1 line

Oh man too long commit message, please follow Johannes's suggestion.

Will do!

Quoted 8 lines

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Quoted 7 lines

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

Quoted 4 lines

I do think the userspace counters have diagnostic value. They can be another metric to show how much anon/file reclaim work is spent on pages that are not immediately reclaimed, including pages retained on actie LRU. It could help explain if there is elevated reclaim CPU usage.

Hm, this is an interesting discussion.

The existing PGROTATED is kind of useless. When it's bumped:

- truncation fails and moves folios to the LRU tail for faster reclaim
- writeback ends on folio that was tried to reclaim, moves to LRU tail for fast retry

Quoted 8 lines

Ok, who needs to know this? The new counters you're adding, on the other hand, they capture the information you frequently need for understanding reclaim behavior.

That said, looking closer at this one more time, you can already (almost) derive them fully from existing counters:

drotated = (dPGSCAN_ANON - dPGSTEAL_ANON) + dPGDEACTIVATE_ANON

Talking with Usama offlist, there are two problems with this:

- PGDEACTIVATE is obviously wrong. We want the the pages we don't deactivate (hot file executables)

- PGSCAN includes things that didn't work out due to caller restrictions (reclaim_idx) and race conditions (tryget, !lru) that shouldn't contribute to the balance.

So let's go with PGROTATE as Usama proposed.

But I'm against hiding them from the vmstats.

I'd rather remove or rename the existing one if we're worried about confusion. We've historically applied a much lower bar to vmstat stability. It's geared towards understanding detailed MM behavior behavior, which undergoes constant changes.

maintainer
#10 lore raw
Quoted 3 lines · Johannes Weiner
Quoted 2 lines · Johannes Weiner
Quoted 1 line · Usama Arif

[...]

Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 9 lines

The patch looks good but I have one question. We already have concept of pgrotated through /proc/vmstat which represents number of pages we have moved to the tail of inactive LRU to make them next reclaim candidates. This patch is exposing pgrotate_[anon|file] to userspace. Here these metrics represents either the number of pages we scanned but didn't reclaim plus number of pages given one more trip in the active LRU.

Quoted 7 lines

Older and newer pgrotate metrics kind of represent something different. It may cause confusion. Now before suggesting to change the name, let me ask do we really need to expose these to the userspace? How exactly users can use these new metrics? I think you can use them here without exposing to userspace. So, does it makes sense to delay userspace exposure later when we have a more solid usecase for that unless you have it already.

I do think the userspace counters have diagnostic value. They can be another metric to show how much anon/file reclaim work is spent on pages that are not immediately reclaimed, including pages retained on actie LRU. It could help explain if there is elevated reclaim CPU usage.

Quoted 6 lines
Quoted 6 lines

Hm, this is an interesting discussion.

The existing PGROTATED is kind of useless. When it's bumped:

- truncation fails and moves folios to the LRU tail for faster reclaim
- writeback ends on folio that was tried to reclaim, moves to LRU tail for fast retry

Yes PGROTATED only tells number of pages moved to LRU tail and nothing about the why. I can see that from user's perspective the why part is more important.

Quoted 19 lines
Quoted 9 lines

Ok, who needs to know this? The new counters you're adding, on the other hand, they capture the information you frequently need for understanding reclaim behavior.

That said, looking closer at this one more time, you can already (almost) derive them fully from existing counters:

drotated = (dPGSCAN_ANON - dPGSTEAL_ANON) + dPGDEACTIVATE_ANON

Talking with Usama offlist, there are two problems with this:

- PGDEACTIVATE is obviously wrong. We want the the pages we don't deactivate (hot file executables)

- PGSCAN includes things that didn't work out due to caller restrictions (reclaim_idx) and race conditions (tryget, !lru) that shouldn't contribute to the balance.

Quoted 8 lines

So let's go with PGROTATE as Usama proposed.

But I'm against hiding them from the vmstats.

I'd rather remove or rename the existing one if we're worried about confusion. We've historically applied a much lower bar to vmstat stability. It's geared towards understanding detailed MM behavior behavior, which undergoes constant changes.

Let's keep the deprecation part orthogonal to this series. No objection from me if these new metrics help in understanding the MM reclaim behavior. The only request I have is to introduce these metrics as a separate independent patch. First introduce the metrics with the detail on how they can be useful and then later use them for the internal balancing algorithm.

maintainer
#11 lore raw
In reply to Usama Arif
Quoted 18 lines · On Mon, 20 Jul 2026 09:41:21 -0700 Usama Arif <[email protected]> wrote:

The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:

- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.

Quoted 1 line

This series removes those producer-side acquisitions entirely.

Thanks. We're still awaiting review of [2/2]. I'll get this under test in order to parallelize things.

Quoted 23 lines

Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.

Workload rates are identical on both kernels (the bench drives the same memory pressure):

                          baseline    patched      delta
  pgscan_direct  / s      172,662     171,817      ~0%
  pgsteal_direct / s       67,162      66,306      ~0%
  workingset_refault_anon / s
                           40,696      39,830      ~0%

perf lock contention (total wait per 30 s window):

  Lock Name                Before      After     % change
  shrink_lruvec+0x770     722.84 ms    0         -100% (eliminated)
        (= lru_note_cost_unlock_irq)
  workingset_refault+0x167 385.26 ms   0         -100% (eliminated)
        (= lru_note_cost_refault)
  shrink_node+0x4ad       689.43 ms    26.95 ms  -96%
  shrink_active_list      208.34 ms    15.97 ms  -92%
  lru_add_drain_cpu+0x34    1.96 s    917.71 ms  -53%
Quoted 1 line

Total LRU lock wait ~4.23 s ~1.66 s -61%

So it saves 2.5 seconds out of 30 seconds?

author
#12 lore raw
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Mon, 20 Jul 2026 09:41:21 -0700 Usama Arif <[email protected]> wrote:

The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically:

- shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq().
- shrink_active_list() does the same after rotation accounting.
- workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost.
- prepare_scan_control() snapshots anon_cost/file_cost under lru_lock.
- lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth.

Quoted 4 lines
Quoted 1 line

This series removes those producer-side acquisitions entirely.

Thanks. We're still awaiting review of [2/2]. I'll get this under test in order to parallelize things.

Thanks Andrew! Yes I think we have the reviews, but just 1 outstanding comment from Shakeel and need to shorten commit message as Johannes mentioned. Hopefully we will have a final version next week.

Quoted 24 lines
Quoted 24 lines

Measurement is a 30 s perf lock record -a window over otherwise-idle hardware.

Workload rates are identical on both kernels (the bench drives the same memory pressure):

                          baseline    patched      delta
  pgscan_direct  / s      172,662     171,817      ~0%
  pgsteal_direct / s       67,162      66,306      ~0%
  workingset_refault_anon / s
                           40,696      39,830      ~0%

perf lock contention (total wait per 30 s window):

  Lock Name                Before      After     % change
  shrink_lruvec+0x770     722.84 ms    0         -100% (eliminated)
        (= lru_note_cost_unlock_irq)
  workingset_refault+0x167 385.26 ms   0         -100% (eliminated)
        (= lru_note_cost_refault)
  shrink_node+0x4ad       689.43 ms    26.95 ms  -96%
  shrink_active_list      208.34 ms    15.97 ms  -92%
  lru_add_drain_cpu+0x34    1.96 s    917.71 ms  -53%
Quoted 3 lines
Quoted 1 line

Total LRU lock wait ~4.23 s ~1.66 s -61%

So it saves 2.5 seconds out of 30 seconds?

Yes 2.5 seconds total CPU time in the 30 second perf record.