mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch removed from -mm tree
The quilt patch titled
Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
has been removed from the -mm tree. Its filename was
mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: Usama Arif <[email protected]>
Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost
Date: Mon, 27 Jul 2026 09:23:25 -0700
The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are:
- shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink.
- shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink.
- workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault.
- prepare_scan_control() takes lru_lock just to snapshot the two
scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call.
This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth.
The balance formula for anon and file, respectively, is this:
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated
Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run.
Use PGROTATE_* from the preceding patch for the rotation input. WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can be sampled per lruvec and aggregated through the memcg hierarchy. This is overall cheaper and has fewer lock acquisition sites.
Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass.
A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node.
NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed.
memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged.
Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation. The preceding patch also bumps PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work is accounted consistently across both implementations.
---
include/linux/mmzone.h | 13 +++++- include/linux/vmstat.h | 1 mm/folio.c | 69 ---------------------------------- mm/internal.h | 3 - mm/memcontrol-v1.c | 4 - mm/memcontrol.c | 1 mm/mmzone.c | 1 mm/vmscan.c | 79 ++++++++++++++++++++++++++++++++------- mm/workingset.c | 5 -- 9 files changed, 81 insertions(+), 95 deletions(-)
include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +11 -2
@@ -757,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_c#endif /* CONFIG_LRU_GEN */ +struct lru_cost { +unsigned long count; +unsigned long last_rotated; +unsigned long last_io; +}; +struct lruvec {struct list_head lists[NR_LRU_LISTS ];/* per lruvec lru_lock for memcg */
@@ -765,9 +771,12 @@ struct lruvec {* These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. +* Updated and decayed at prepare_scan_control() time; cost_lock +* serialises that update. */ -unsigned long anon_cost; -unsigned long file_cost; +struct lru_cost cost[ANON_AND_FILE ]; +/* Protects cost[]. */ +spinlock_t cost_lock;/* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age;/* Refaults at the time of last reclaim cycle */
include/linux/vmstat.h renamed from include/linux/vmstat.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +0 -1
@@ -20,7 +20,6 @@ struct reclaim_stat {unsigned nr_congested;unsigned nr_writeback;unsigned nr_immediate; -unsigned nr_pageout;unsigned nr_activate[ANON_AND_FILE ];unsigned nr_ref_keep;unsigned nr_unmap_fail;
mm/folio.c renamed from mm/folio.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +0 -69
81 lines not on this page · raw message
mm/internal.h renamed from mm/internal.h~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +0 -3
@@ -41,9 +41,6 @@ void workingset_refault(struct folio *fovoid workingset_activation (struct folio *folio);/* mm/folio.c */ -void lru_note_cost_unlock_irq (struct lruvec *lruvec,bool file, -unsigned int nr_io,unsigned int nr_rotated); -void lru_note_cost_refault (struct folio *folio);void folio_add_lru_vma (struct folio *folio,struct vm_area_struct *vma);static inline bool folio_may_be_lru_cached (struct folio *folio)
mm/memcontrol.c renamed from mm/memcontrol.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +1 -0
@@ -399,6 +399,7 @@ static const unsigned int memcg_node_staNR_SHMEM_THPS ,NR_FILE_THPS ,NR_ANON_THPS , +NR_VMSCAN_WRITE ,NR_VMALLOC ,NR_KERNEL_STACK_KB , NR_PAGETABLE,
mm/memcontrol-v1.c renamed from mm/memcontrol-v1.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +2 -2
@@ -2287,8 +2287,8 @@ void memcg1_stat_format(struct mem_cgroufor_each_online_pgdat (pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost[WORKINGSET_ANON ].count; + file_cost += mz->lruvec.cost[WORKINGSET_FILE ].count; }seq_buf_printf (s,"anon_cost %lu\n" , anon_cost);seq_buf_printf (s,"file_cost %lu\n" , file_cost);
mm/mmzone.c renamed from mm/mmzone.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +1 -0
@@ -79,6 +79,7 @@ void lruvec_init(struct lruvec *lruvec)memset (lruvec,0 ,sizeof (struct lruvec ));spin_lock_init (&lruvec->lru_lock); +spin_lock_init (&lruvec->cost_lock);zswap_lruvec_state_init (lruvec);for_each_lru (lru)
mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +66 -13
117 lines not on this page · raw message
mm/workingset.c renamed from mm/workingset.c~mm-vmscan-reduce-lru_lock-contention-via-vmstat-derived-scan-balance-cost +0 -5
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *fo/* Folio was active prior to eviction */ if (workingset) {folio_set_workingset (folio); -/* -* XXX: Move to folio_add_lru() when it supports new vs -* putback -*/ -lru_note_cost_refault (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file, nr); } out:
_
Patches currently in -mm which might be from [email protected] are
mm-huge_memory-transfer-the-pmd-dirty-bit-to-the-folio-on-zap.patch