mm/mglru: dynamically protect readahead fault folios under refault pressure
Subsystems (4)
RFC PATCH 1/1 mm/mglru: dynamically protect readahead fault folios under refault pressure
Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.
The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive forever.
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit in folio_add_lru() for fault-path file folios
- keep the folio active while credit is available, and otherwise let the original behavior stand
- decay/reset the credit as generations advance and when an lruvec is initialized
This means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages around unnecessarily.
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):
- with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
The fix restores the readahead protection lost by the regression while preserving the original intent of the optimization: do not keep readahead pages around if the workload does not need them.
--- include/linux/mmzone.h | 2 ++ mm/swap.c | 82 ++++++++++++++++++++++++++++++++++++++---- mm/vmscan.c | 7 ++++ mm/workingset.c | 18 ++++++++++ 4 files changed, 102 insertions(+), 7 deletions(-)
include/linux/mmzone.h +2 -0
@@ -578,6 +578,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* credit: file refaults indicate fault-path file folios need protection */ +atomic_long_t ra_refaults;/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */
mm/swap.c +75 -7
99 lines not on this page · raw message
mm/vmscan.c +7 -0
@@ -3931,6 +3931,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappinessbool success;int prev, next;int type, zone; +long old, new;struct lru_gen_folio *lrugen = &lruvec->lrugen; restart:if (seq <READ_ONCE (lrugen->max_seq))
@@ -3983,6 +3984,11 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappinessreset_ctrl_pos (lruvec, type, false);WRITE_ONCE (lrugen->timestamps[next], jiffies); +/* decay readahead protection credit so stale signal doesn't persist */ + old =atomic_long_read (&lrugen->ra_refaults); +do { + new = (old *3 ) /4 ; + }while (!atomic_long_try_cmpxchg (&lrugen->ra_refaults, &old, new));/* make sure preceding modifications appear */ smp_store_release (&lrugen->max_seq, lrugen->max_seq +1 ); unlock:
@@ -5784,6 +5790,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) lrugen->max_seq =MIN_NR_GENS +1 ; lrugen->enabled =lru_gen_enabled (); +atomic_long_set (&lrugen->ra_refaults,0 );for (i =0 ; i <=MIN_NR_GENS +1 ; i++) lrugen->timestamps[i] = jiffies;
mm/workingset.c +18 -0
@@ -319,6 +319,24 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); +if (type ==LRU_GEN_FILE ) { +/* +* Cap credit at total file pages to avoid runaway while +* allowing sustained protection. +*/ +long cap =lruvec_page_state (lruvec,NR_LRU_BASE +LRU_INACTIVE_FILE ) + +lruvec_page_state (lruvec,NR_LRU_BASE +LRU_ACTIVE_FILE ); +long add = (long )delta *VM_READAHEAD_PAGES ; +long old, new; + + old =atomic_long_read (&lrugen->ra_refaults); +do { +if (old >= cap) +break ; + new =min (cap, old + add); + }while (!atomic_long_try_cmpxchg (&lrugen->ra_refaults, &old, new)); + } +if (workingset) {/* * see folio_add_lru(), where folio_set_active() is
Signature · 1 line
2.43.0
Quoted 37 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive forever.
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the original behavior stand
- decay/reset the credit as generations advance and when an lruvec is initializedThis means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages around unnecessarily.
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):
Quoted 7 lines
- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
Hello Ehab,
We ran into the same issue on our side too. I hesitated to report or fix that as I'm working on MGLRU-FG which fixed the problem on my side:
https://lore.kernel.org/linux-mm/[email protected]/
I especially mentioned it, see the parts after:
"recent change in lru_gen_folio_seq that bumps new folios with refs == 1"
Latest version still being tested which you can use directly:
https://github.com/ryncsn/linux/commits/b4/mglru-fg-v1.8/
Do you mind have a look of that as well? I think in the long term that is the right direction. With our test the regression is gone and performance is even better.
And is there any easy way to reproduce the specific case you are reporting?
Quoted 88 lines
The fix restores the readahead protection lost by the regression while preserving the original intent of the optimization: do not keep readahead pages around if the workload does not need them.
Fixes 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active")Signed-off-by Ehab Ababneh <[email protected]>--- include/linux/mmzone.h | 2 ++ mm/swap.c | 82 ++++++++++++++++++++++++++++++++++++++---- mm/vmscan.c | 7 ++++ mm/workingset.c | 18 ++++++++++ 4 files changed, 102 insertions(+), 7 deletions(-)include/linux/mmzone.h +2
-0@@ -578,6 +578,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* credit: file refaults indicate fault-path file folios need protection */ +atomic_long_t ra_refaults;/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */ mm/swap.c +54
-0@@ -70,6 +70,70 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock_irq = INIT_LOCAL_LOCK(lock_irq), }; +#ifdef CONFIG_LRU_GEN +/* Refill two default readahead windows to amortize shared-counter updates. */ +#define RA_REFAULT_LOCAL_BATCH (VM_READAHEAD_PAGES * 2) + +struct ra_credit_cache { +/* Batch shared credit per CPU to avoid a contended atomic RMW per folio. */ +/* only compared for identity, never dereferenced */ +struct lru_gen_folio *lrugen; +long credit; +}; + +static DEFINE_PER_CPU(struct ra_credit_cache, ra_credit_cache); + +/* +* Spend readahead protection credit from a per-CPU bucket, refilled in batches +* from the shared per-lruvec counter, so the fault path avoids an atomic RMW on +* a contended cacheline for every folio. +*/ +static bool lru_gen_take_ra_credit (struct folio *folio) +{ +struct lru_gen_folio *lrugen; +long nr_pages =folio_nr_pages (folio); +struct ra_credit_cache *cache; +bool taken = false; +long old, new; + +rcu_read_lock (); + lrugen = &folio_lruvec (folio)->lrugen; + cache =get_cpu_ptr (&ra_credit_cache); + +/* credit cached for a different lruvec is forfeited, bounded by the batch */ +if (cache->lrugen != lrugen) { + cache->lrugen = lrugen; + cache->credit =0 ; + } + +if (cache->credit < nr_pages) { + old =atomic_long_read (&lrugen->ra_refaults); +while (old >0 ) { + new = old -min_t (long, old,RA_REFAULT_LOCAL_BATCH ); +if (atomic_long_try_cmpxchg (&lrugen->ra_refaults, &old, new)) { + cache->credit += old - new; +break ; + } + } + } + +if (cache->credit >= nr_pages) { + cache->credit -= nr_pages; + taken = true; + } + +put_cpu_ptr (&ra_credit_cache); +rcu_read_unlock ();
mm/swap.c static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { · line 136
Quoted 10 lines
mm/swap.c +10
-0+ + return taken; +} +#else +static bool lru_gen_take_ra_credit (struct folio *folio) +{ +return false; +} +#endif /* CONFIG_LRU_GEN */ +
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect in_fault && folio_test_swapbacked folios with PG_active?
Barry Song (Xiaomi)
reviewerQuoted 47 lines · Kairui Song
Quoted 44 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive forever.
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the original behavior stand
- decay/reset the credit as generations advance and when an lruvec is initializedThis means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages around unnecessarily.
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?
Hi Ehab and Kairui,
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the kernel build.
Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
"I also retested the LevelDB benchmark from the cache_ext paper [5]. Interestingly, mainline MGLRU already beats CLRU on this one after a recent change in lru_gen_folio_seq that bumps new folios with refs == 1 to the second-oldest generation."
I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case actually different from Kairui's?
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
include/linux/mm_inline.h +3 -2
@@ -239,7 +239,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,* |<---------------------------- MAX_NR_GENS ---------------------------->| */ if (folio_test_active (folio)) - gen =MIN_NR_GENS -folio_test_workingset (folio); + gen =MIN_NR_GENS -folio_test_workingset (folio) + + (type ? !folio_test_workingset (folio) :0 );else if (reclaiming) gen =MAX_NR_GENS ; else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -247,7 +248,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, (folio_test_dirty(folio) || folio_test_writeback(folio)))) gen =MIN_NR_GENS ; else - gen =MAX_NR_GENS - (folio_test_workingset (folio) ||folio_test_referenced (folio)); + gen =MAX_NR_GENS -folio_test_workingset (folio);return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); }
mm/folio.c +3 -13
@@ -470,20 +470,10 @@ void folio_add_lru(struct folio *folio) folio_test_unevictable(folio), folio);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); -/* -* For refaulted workingset folios, set PG_active so they -* can be added to active generations. -* For prefaulted file folios, folio_mark_accessed() sets -* PG_referenced so lru_gen_folio_seq() places them into -* the second oldest generation. -*/ +/* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled () && !folio_test_unevictable (folio) && -lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) { -if (folio_test_workingset (folio)) -folio_set_active (folio); -else if (!folio_test_referenced (folio)) -folio_mark_accessed (folio); - } +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) +folio_set_active (folio);folio_batch_add_and_move (folio, lru_add); }
mm/vmscan.c +1 -5
@@ -851,11 +851,7 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags) return false; } -/* Promote on second access */ -if (folio_lru_refs (folio) >1 ) -set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset)); -else -folio_mark_accessed (folio); +set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset));return true; }#else
mm/workingset.c +4 -6
@@ -319,13 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); +/* see folio_add_lru() where folio_set_active() will be called */ +if (lru_gen_in_fault ()) +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); +if (workingset) { -/* -* see folio_add_lru(), where folio_set_active() is -* called for workingset folios -*/ -if (lru_gen_in_fault ()) -mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta);folio_set_workingset (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + type, delta); }else
Signature · 1 line
2.34.1
Hi Barry, Kairui,
Thanks for the feedback and suggestions.
Quoted 83 lines
-----Original Message-----
From: Barry Song (Xiaomi) <[email protected]>
Sent: Tuesday, September 1, 2026 3:05 PM
To: [email protected]; Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 13 lines · Kairui Song
Quoted 11 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Quoted 3 lines
Quoted 3 lines
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive
forever.
Quoted 17 lines
Quoted 17 lines
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
initializedQuoted 4 lines
Quoted 4 lines
This means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages
around unnecessarily.
Quoted 13 lines
Quoted 12 lines
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?Hi Ehab and Kairui,
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the kernel build. Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
"I also retested the LevelDB benchmark from thecache_extpaper [5]. Interestingly, mainline MGLRU already beats CLRU on this one after a recent change inlru_gen_folio_seqthat bumps new folios with refs == 1 to the second-oldest generation."
Quoted 6 lines
I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case actually different from Kairui's? (The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks,
Ehab
Quoted 101 lines
include/linux/mm_inline.h index renamed from include/linux/mm_inline.h +0
-0
621c8653d8f7..91396c796a34100644include/linux/mm_inline.h +2
-1@@ -239,7 +239,8 @@ static inline unsigned long lru_gen_folio_seq(conststruct lruvec *lruvec, * |<---------------------------- MAX_NR_GENS ---------------------------->| */ if (folio_test_active (folio)) - gen =MIN_NR_GENS -folio_test_workingset (folio); + gen =MIN_NR_GENS -folio_test_workingset (folio) + + (type ? !folio_test_workingset (folio) :0 );else if (reclaiming) gen =MAX_NR_GENS ;include/linux/mm_inline.h +1
-1else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||@@ -247,7 +248,7 @@ static inline unsigned long lru_gen_folio_seq(conststruct lruvec *lruvec, (folio_test_dirty(folio) || folio_test_writeback(folio)))) gen = MIN_NR_GENS; else - gen =MAX_NR_GENS - (folio_test_workingset (folio) || folio_test_referenced(folio)); + gen =MAX_NR_GENS -folio_test_workingset (folio);return max(READ_ONCE(lrugen->max_seq) - gen + 1,
READ_ONCE(lrugen->min_seq[type]));
}mm/folio.c +3
-13@@ -470,20 +470,10 @@ void folio_add_lru(struct folio *folio) folio_test_unevictable(folio), folio);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio); -/* -* For refaulted workingset folios, set PG_active so they -* can be added to active generations. -* For prefaulted file folios, folio_mark_accessed() sets -* PG_referenced so lru_gen_folio_seq() places them into -* the second oldest generation. -*/ +/* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled () && !folio_test_unevictable (folio) && -lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) { -if (folio_test_workingset (folio)) -folio_set_active (folio); -else if (!folio_test_referenced (folio)) -folio_mark_accessed (folio); - } +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) +folio_set_active (folio); folio_batch_add_and_move(folio, lru_add); } diff --git
a/mm/vmscan.cb/mm/vmscan.cindexf11491ee9ed5..aa500ae9371a100644mm/vmscan.c +1
-5@@ -851,11 +851,7 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)return false; } -/* Promote on second access */ -if (folio_lru_refs (folio) >1 ) -set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS , BIT(PG_workingset)); -else -folio_mark_accessed (folio); +set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset));return true;}
#elsemm/workingset.c index renamed from mm/workingset.c +0
-0
7ac2b88c80ae..a79b73ee9762100644mm/workingset.c +4
-6@@ -319,13 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); +/* see folio_add_lru() where folio_set_active() will be called */ +if (lru_gen_in_fault ()) +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); +if (workingset) { -/* -* see folio_add_lru(), where folio_set_active() is -* called for workingset folios -*/ -if (lru_gen_in_fault ()) -mod_lruvec_state (lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);mm/workingset.c +0
-0folio_set_workingset (folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE +type, delta);
} else
--
2.34.1
Barry Song
reviewerQuoted 95 lines · Ehab
Hi Barry, Kairui,
Thanks for the feedback and suggestions.
Quoted 83 lines
-----Original Message-----
From: Barry Song (Xiaomi) <[email protected]>
Sent: Tuesday, September 1, 2026 3:05 PM
To: [email protected]; Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 13 lines · Kairui Song
Quoted 11 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Quoted 3 lines
Quoted 3 lines
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive
forever.
Quoted 17 lines
Quoted 17 lines
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
initializedQuoted 4 lines
Quoted 4 lines
This means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages
around unnecessarily.
Quoted 13 lines
Quoted 12 lines
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?Hi Ehab and Kairui,
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the kernel build. Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
"I also retested the LevelDB benchmark from thecache_extpaper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a recent change inlru_gen_folio_seqthat bumps new folios with refs == 1 to the second-oldest generation."Quoted 6 lines
I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case actually different from Kairui's? (The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/[email protected]/
Best Regards
Barry
Quoted 127 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 2, 2026 2:58 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 85 lines · Ehab
Hi Barry, Kairui,
Thanks for the feedback and suggestions.
Quoted 80 lines
-----Original Message-----
From: Barry Song (Xiaomi) <[email protected]>
Sent: Tuesday, September 1, 2026 3:05 PM
To: [email protected]; Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 14 lines · Kairui Song
Quoted 12 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Quoted 4 lines
Quoted 4 lines
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive
forever.
Quoted 17 lines
Quoted 17 lines
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
initializedQuoted 5 lines
Quoted 5 lines
This means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages
around unnecessarily.
Quoted 13 lines
Quoted 12 lines
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?Hi Ehab and Kairui,
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the
kernel build.
Quoted 9 lines
Quoted 9 lines
Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
"I also retested the LevelDB benchmark from thecache_extpaper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a recent change inlru_gen_folio_seqthat bumps new folios with refs == 1 to the second-oldest generation."I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case
actually different from Kairui's?
Quoted 5 lines
Quoted 4 lines
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Quoted 5 lines
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/
Sure, I will do. I will report back when I have the results.
Thanks,
Ehab
Quoted 2 lines
Best Regards
Barry
Hi Kairui, Barry,
Apologies for the delay in getting back to you with these results — my test environment got corrupted and I had to spend some time recovering it before I could re-run the benchmark.
Quoted 114 lines
-----Original Message-----
From: Ababneh, Ehab
Sent: Wednesday, September 2, 2026 3:02 PM
To: 'Barry Song' <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 12 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 2, 2026 2:58 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 1 line
On Thu, Sep 3, 2026 at 5:52 AM Ababneh, Ehab
Quoted 19 lines
Quoted 18 lines · wrote:
Hi Barry, Kairui,
Thanks for the feedback and suggestions.
Quoted 13 lines
-----Original Message-----
From: Barry Song (Xiaomi) <[email protected]>
Sent: Tuesday, September 1, 2026 3:05 PM
To: [email protected]; Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureOn Wed, Sep 2, 2026 at 2:21 AM Kairui Song <[email protected]>
Quoted 14 lines · wrote:
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines · Ehab Ababneh
Quoted 12 lines
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from
it.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
The goal is to keep the optimization without keeping readahead pages alive
forever.
Quoted 4 lines
Quoted 4 lines
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually
helping.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
initializedThis means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
around unnecessarily.
Quoted 5 lines
Quoted 5 lines
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
replace
folio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?Hi Ehab and Kairui,
Quoted 33 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the
kernel build.
Quoted 1 line
Quoted 1 line
Also, for example, Kairui's cover letter mentioned the LevelDB
benchmark:
Quoted 25 lines
Quoted 9 lines
Quoted 9 lines
"I also retested the LevelDB benchmark from the
cache_extpaper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a recent change inlru_gen_folio_seqthat bumps new folios with refs == 1 to the second-oldest generation."I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case
actually different from Kairui's?
Quoted 15 lines
Quoted 5 lines
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further
gains.)I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks for the pointer — I gave your MGLRU-FG fix a try against the same Cassandra read benchmark I used for my patch (4 nodes, 720s, 100 readers).
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/s
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (6cbdd9726fb5, "mm/mglru: use folio_mark_accessed to replace folio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
Revert of 6cbdd9726fb5:
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
So MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms.
Happy to dig further if useful — let me know if you'd like more detail on the latency distribution or a longer run to confirm these numbers are stable.
Thanks,
Ehab
Quoted 14 lines
Quoted 6 lines
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/Sure, I will do. I will report back when I have the results.
Thanks,
EhabQuoted 2 lines
Best Regards
Barry
Barry Song
reviewerQuoted 6 lines · Ehab
Hi Kairui, Barry,
Apologies for the delay in getting back to you with these results — my test environment got corrupted and I had to spend some time recovering it before I could re-run the benchmark.
No worries, Ehab. And thanks very much for your testing!
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Quoted 5 lines
Thanks for the pointer — I gave your MGLRU-FG fix a try against the same Cassandra read benchmark I used for my patch (4 nodes, 720s, 100 readers).
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/s
I noticed this is even higher than your dynamic
readahead fix: "throughput ~51.9k-52.7k op/s"
Is this a run-to-run variation, or is the improvement stable across runs?
Quoted 11 lines
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (
6cbdd9726fb5, "mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/sRevert of
6cbdd9726fb5:p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
Quoted 7 lines
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/sSo MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms.
We also have this :
https://lore.kernel.org/linux-mm/[email protected]/
you once reported it as :
"throughput was approximately
49.4K to 50.3K ops/s, averaging about 50.0K ops/s, while the 99th-percentile latency remained around 7.0 to 7.1 ms"
Quoted 5 lines
Happy to dig further if useful — let me know if you'd like more detail on the latency distribution or a longer run to confirm these numbers are stable.
Thanks
Barry
Hi Barry,
Quoted 39 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 23, 2026 9:08 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 6 lines · Ehab
Hi Kairui, Barry,
Apologies for the delay in getting back to you with these results — my test environment got corrupted and I had to spend some time recovering it before I could re-run the benchmark.
No worries, Ehab. And thanks very much for your testing!
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks for the pointer — I gave your MGLRU-FG fix a try against the same Cassandra read benchmark I used for my patch (4 nodes, 720s, 100
readers).
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/sI noticed this is even higher than your dynamic readahead fix: "throughput ~51.9k-52.7k op/s"
Is this a run-to-run variation, or is the improvement stable across runs?
Yes, I can't explain the slight gain in throughput but loss in latency compared to my fix. I'll probably study this in more detail when I can dedicate some time to it.
Typically, I get a unique result immediately after a reboot, but subsequent runs are very consistent. The variation is usually around 0.1 ms gain/loss in latency and approximately 100-200 requests per second in throughput. I discard the results from the first run.
Quoted 19 lines
Quoted 11 lines
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (
6cbdd9726fb5, "mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/sRevert of
6cbdd9726fb5:p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/sQuoted 7 lines
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/sSo MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms.
Quoted 9 lines
We also have this :
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/you once reported it as :
"throughput was approximately
49.4K to 50.3K ops/s, averaging about 50.0K ops/s, while the 99th-percentile latency remained around 7.0 to 7.1 ms"
Yes, this is from before the data was corrupted. I get slightly different results with my fix and with the revert, but the overall pattern remains consistent.
Thanks,
Ehab
Quoted 8 lines
Quoted 5 lines
Happy to dig further if useful — let me know if you'd like more detail on the latency distribution or a longer run to confirm these numbers are stable.
Thanks
Barry
Quoted 49 lines · Ababneh, Ehab <[email protected]> 于 2026年9月25日周五 01:54写道:
Hi Barry,
Quoted 46 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 23, 2026 9:08 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 6 lines · Ehab
Hi Kairui, Barry,
Apologies for the delay in getting back to you with these results — my test environment got corrupted and I had to spend some time recovering it before I could re-run the benchmark.
No worries, Ehab. And thanks very much for your testing!
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks for the pointer — I gave your MGLRU-FG fix a try against the same Cassandra read benchmark I used for my patch (4 nodes, 720s, 100
readers).
Quoted 3 lines
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/sI noticed this is even higher than your dynamic readahead fix: "throughput ~51.9k-52.7k op/s"
Quoted 6 lines
Quoted 2 lines
Is this a run-to-run variation, or is the improvement stable across runs?
Yes, I can't explain the slight gain in throughput but loss in latency compared to my fix. I'll probably study this in more detail when I can dedicate some time to it.
Hi Ehab,
That's very interesting, I think it means we are getting a bit better at identifying the working set. But maybe a slightly high cost of aging? And P99 is usually very sensitive to noises, one long tailing aging event could make it look a lot worse.
Quoted 4 lines
Typically, I get a unique result immediately after a reboot, but subsequent runs are very consistent. The variation is usually around 0.1 ms gain/loss in latency and approximately 100-200 requests per second in throughput. I discard the results from the first run.
Any way I can reproduce this? E.g. what is the test bench, database size, benchmark, machine spec, etc?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (
6cbdd9726fb5, "mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/sRevert of
6cbdd9726fb5:p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/sSo MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms
Did you test the V2 or V1 of MGLRU-FG? V2 includes Barry's aging optimization; I'm not sure if this latency is caused by aging or maybe some other change.
And is there any swap device you used, or is the test using the same base commit? MM also changes significantly between different baselines.
Hi Kairui,
Quoted 109 lines
-----Original Message-----
From: Kairui Song <[email protected]>
Sent: Saturday, September 26, 2026 12:06 AM
To: Ababneh, Ehab <[email protected]>
Cc: Barry Song <[email protected]>; [email protected]; Axel Rasmussen <[email protected]>; Lance Yang <[email protected]>; LKML <[email protected]>; linux-mm <[email protected]>; Qi Zheng <[email protected]>; Shakeel Butt <[email protected]>; Wei Xu <[email protected]>; Yuanchu Xie <[email protected]>; Yu Zhao <[email protected]>
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureAbabneh, Ehab <[email protected]> 于 2026年9月25日周五
Quoted 9 lines · 01:54写道:
Hi Barry,
Quoted 6 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 23, 2026 9:08 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected];
[email protected]; [email protected];Quoted 15 lines
Quoted 15 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 6 lines · Ehab
Hi Kairui, Barry,
Apologies for the delay in getting back to you with these results — my test environment got corrupted and I had to spend some time recovering it before I could re-run the benchmark.
Quoted 8 lines
Quoted 8 lines
No worries, Ehab. And thanks very much for your testing!
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those
workloads.
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Thanks for the pointer — I gave your MGLRU-FG fix a try against the same Cassandra read benchmark I used for my patch (4 nodes, 720s, 100
readers).
Quoted 3 lines
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/sI noticed this is even higher than your dynamic readahead fix:
"throughput ~51.9k-52.7k op/s"Quoted 6 lines
Quoted 2 lines
Is this a run-to-run variation, or is the improvement stable across runs?
Yes, I can't explain the slight gain in throughput but loss in latency compared to my fix. I'll probably study this in more detail when I can dedicate some time to it.
Hi Ehab,
That's very interesting, I think it means we are getting a bit better at identifying the working set. But maybe a slightly high cost of aging? And P99 is usually very sensitive to noises, one long tailing aging event could make it look a lot worse.
Quoted 4 lines
Typically, I get a unique result immediately after a reboot, but subsequent runs are very consistent. The variation is usually around 0.1 ms gain/loss in latency and approximately 100-200 requests per second in throughput. I discard the results from the first run.
Any way I can reproduce this? E.g. what is the test bench, database size, benchmark, machine spec, etc?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (
6cbdd9726fb5, "mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/sRevert of
6cbdd9726fb5:p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/sQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/sSo MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms
Quoted 6 lines
Did you test the V2 or V1 of MGLRU-FG? V2 includes Barry's aging optimization; I'm not sure if this latency is caused by aging or maybe some other change.
And is there any swap device you used, or is the test using the same base commit? MM also changes significantly between different baselines.
My setup runs Cassandra 4.1.5 on Java 11. The host has two Intel Xeon Platinum 8468 sockets, 96 physical CPU cores (192 logical CPUs), two NUMA nodes, and 440 GiB of system-visible RAM. Cassandra instances use CPU affinity, with memory interleaved across NUMA nodes.
I run four Cassandra instances, each backed by a dedicated 3.5 TB NVMe drive. Each drive has a separate mount, and holds its instance’s data files, commit log, and saved caches.
The stress workload uses a pre-populated table. Its schema, partitioning, replication, compaction, and compression settings, along with the stress profile and concurrency, are recorded because they can affect performance.
I used more than one kernel version across my test runs. The most recent was Linux 7.3.0-rc5. Within each set of setups being compared, I kept the kernel version consistent.
For the MGLRU-FG tests, I used the ryncsn/b4/mglru-fg-v1.8 branch. The MGLRU-FG implementation is commit 6bcf4ae70b5e; the branch tip is a619ab0105d9.
Quoted 2 lines · Ehab
Hi Kairui,
...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Results with MGLRU-FG:
Op rate: 55.0k - 56.5k op/s
...
Quoted 40 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Latency 99th percentile: 6.8 - 6.9 ms
For reference, here's where the other variants landed on the same setup:
Regression (
6cbdd9726fb5, "mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/sRevert of
6cbdd9726fb5:p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/sQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
My dynamic readahead-credit fix:
p99 ~5.8 ms, throughput ~51.9k-52.7k op/sSo MGLRU-FG recovers most of the latency regression, but the revert and my fix still recover more of it: p99 with MGLRU-FG is noticeably higher than with the revert and my fix (6.8-6.9 ms vs ~5.5-5.8 ms), though it's still a big improvement over the regression's 9.2-9.5 ms
Quoted 6 lines
Did you test the V2 or V1 of MGLRU-FG? V2 includes Barry's aging optimization; I'm not sure if this latency is caused by aging or maybe some other change.
And is there any swap device you used, or is the test using the same base commit? MM also changes significantly between different baselines.
My setup runs Cassandra 4.1.5 on Java 11. The host has two Intel Xeon Platinum 8468 sockets, 96 physical CPU cores (192 logical CPUs), two NUMA nodes, and 440 GiB of system-visible RAM. Cassandra instances use CPU affinity, with memory interleaved across NUMA nodes.
I run four Cassandra instances, each backed by a dedicated 3.5 TB NVMe drive. Each drive has a separate mount, and holds its instance’s data files, commit log, and saved caches.
The stress workload uses a pre-populated table. Its schema, partitioning, replication, compaction, and compression settings, along with the stress profile and concurrency, are recorded because they can affect performance.
Quoted 7 lines
I used more than one kernel version across my test runs. The most recent was Linux 7.3.0-rc5. Within each set of setups being compared, I kept the kernel version consistent.
For the MGLRU-FG tests, I used the ryncsn/b4/mglru-fg-v1.8 branch. The MGLRU-FG implementation is commit
6bcf4ae70b5e; the branch tip isa619ab0105d9.
Thanks for the info! That v1.8 is missing recent upstream changes indeed, so aging could be a bit slower, causing the jitter. I also found a few other minor improvements doable, I'll also try to run some Cassandra benchmarks too. Higher throughput and worse P99 is an interesting effect, I think it means the LRU is doing better at protecting the working set and maybe spending a bit more effort on reclaim / aging, which could be optimized away. I will try to reproduce and optimize for that as well while keep the series updated.
Quoted 125 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Wednesday, September 2, 2026 2:58 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 85 lines · Ehab
Hi Barry, Kairui,
Thanks for the feedback and suggestions.
Quoted 80 lines
-----Original Message-----
From: Barry Song (Xiaomi) <[email protected]>
Sent: Tuesday, September 1, 2026 3:05 PM
To: [email protected]; Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureQuoted 14 lines · Kairui Song
Quoted 12 lines · Ehab Ababneh
Commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active") introduced a regression for workloads that rely on readahead to keep sequential file access efficient.The problem is that MGLRU can place fault-path file folios in older generations, so memory pressure can reclaim readahead folios before the workload touches them. In our Cassandra read benchmark, this raised p99 latency to about 9.2-9.5 ms and cut throughput to roughly 41.8k-43.6k op/s; the revert restored the workload to about 5.5-5.6 ms and 51.9k-53.1k op/s.
Quoted 4 lines
Quoted 4 lines
Readahead is important for sequential I/O and mmap scans, but it should not be retained when the workload does not benefit from it. The goal is to keep the optimization without keeping readahead pages alive
forever.
Quoted 17 lines
Quoted 17 lines
This patch provides a middle ground: keep the original behavior by default, but temporarily protect fault-path file folios when repeated file refaults show that readahead is actually helping.
The mechanism is dynamic and self-tuning:
- add a per-lruvec readahead/refault credit
- accumulate credit on file refaults in the MGLRU refault path
- consume credit infolio_add_lru()for fault-path file folios
- keep the folio active while credit is available, and otherwise let the
original behavior stand
- decay/reset the credit as generations advance and when an lruvec is
initializedQuoted 5 lines
Quoted 5 lines
This means we only protect fault-path file folios when refault pressure shows that the workload is actively benefiting from readahead. If the workload does not need that protection, the original optimization remains intact and we do not keep readahead pages
around unnecessarily.
Quoted 13 lines
Quoted 12 lines
Benchmark results for the Cassandra read workload
(4 nodes, 720s, 100 readers):- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s[...]
Quoted 4 lines
Just an idea. For an short term and easy fix, what if we simply revert than, then only protect
in_fault&&folio_test_swapbackedfolios withPG_active?Hi Ehab and Kairui,
Thanks very much for your reports and discussion.
I'm not quite sure we want to revert it entirely, as that would immediately regress the workloads improved by the commit, such as the
kernel build.
Quoted 9 lines
Quoted 9 lines
Also, for example, Kairui's cover letter mentioned the LevelDB benchmark:
"I also retested the LevelDB benchmark from thecache_extpaper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a recent change inlru_gen_folio_seqthat bumps new folios with refs == 1 to the second-oldest generation."I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case
actually different from Kairui's?
Quoted 5 lines
Quoted 4 lines
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Quoted 5 lines
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/
Hi Barry,
Thanks for sending the suggested fix. I tested it with the Cassandra benchmark, and the results look promising. Across the runs, throughput was approximately 49.4K to 50.3K ops/s, averaging about 50.0K ops/s, while the 99th-percentile latency remained around 7.0 to 7.1 ms. This represents an improvement over the baseline of approximately 43.98K ops/s and 9.025 ms p99 latency.
The underlying idea in both solutions is very similar: A credit-based system protects file folios when readahead proves useful. In my version, I further tuned credit accounting and used per-CPU batching to reduce atomic contention on multicore systems.
Thanks,
Ehab
Quoted 2 lines
Best Regards
Barry
Barry Song
reviewerOn Fri, Sep 4, 2026 at 3:46 AM Ababneh, Ehab <[email protected]> wrote:
[...]
Quoted 25 lines
Quoted 19 lines
Quoted 3 lines
Quoted 3 lines
I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case
actually different from Kairui's?
Quoted 5 lines
Quoted 4 lines
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further gains.)
Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Quoted 5 lines
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/
Quoted 8 lines
Hi Barry,
Thanks for sending the suggested fix. I tested it with the Cassandra benchmark, and the results look promising. Across the runs, throughput was approximately 49.4K to 50.3K ops/s, averaging about 50.0K ops/s, while the 99th-percentile latency remained around 7.0 to 7.1 ms. This represents an improvement over the baseline of approximately 43.98K ops/s and 9.025 ms p99 latency.
I am happy to see it is much better, but it still doesn't seem as good as it was at the very beginning?
- with commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/s
Quoted 5 lines
The underlying idea in both solutions is very similar: A credit-based system protects file folios when readahead proves useful. In my version, I further tuned credit accounting and used per-CPU batching to reduce atomic contention on multicore systems.
Your benchmark seems quite likely to access readahead folios. Let me dig into your patch a bit deeper and see if we can find a combination of the two approaches.
BTW, does Kairui's MGLRU-FG bring your benchmark back to the performance you had at the very beginning?
Best Regards
Barry
Quoted 51 lines
-----Original Message-----
From: Barry Song <[email protected]>
Sent: Thursday, September 3, 2026 2:38 PM
To: Ababneh, Ehab <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH] mm/mglru: dynamically protect readahead fault folios under refault pressureOn Fri, Sep 4, 2026 at 3:46 AM Ababneh, Ehab <[email protected]> wrote:
[...]Quoted 27 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
I guess we could instead try to mitigate the cases where some workloads are negatively affected while preserving the original intention. Does the fix below address both of your cases, or is Ehab's case
actually different from Kairui's?
Quoted 6 lines
Quoted 5 lines
(The kernel-build test on my machine looks quite positive. It not only preserves the original optimization, but even provides further
gains.)Quoted 9 lines
I agree that reverting the commit that caused the regression is not the optimal path. I expect there are many workloads and scenarios that benefit from the behavior introduced by that commit, so reverting it could unnecessarily regress those workloads.
I will run the Cassandra benchmark with Kairui's MGLRU-FG patches to see whether they address the issue I am seeing. I will send the results when they are ready.
Quoted 5 lines
Thanks very much, Ehab. Could you please also run my proposed quick fix:
https://lore.kernel.org/linux-mm/20260901220430.79810-1-
[email protected]/Quoted 7 lines
Hi Barry,
Thanks for sending the suggested fix. I tested it with the Cassandra benchmark, and the results look promising. Across the runs, throughput was approximately 49.4K to 50.3K ops/s, averaging about 50.0K ops/s, while the 99th-percentile latency remained around 7.0 to 7.1 ms. This represents an improvement over the baseline of approximately 43.98K
Quoted 6 lines
ops/s and 9.025 ms p99 latency.
Quoted 1 line
I am happy to see it is much better, but it still doesn't seem as good as it was at the very beginning?
Hi Barry, All,
It is not as performant as its best, but it recovers most of the regression. During the regression, latency had spiked to around 9 ms. After applying the patch, it has come back down to 5.8 ms. Overall, this is still better than where it was about two months ago, when I was typically seeing 6.2 to 6.3 ms.
Quoted 15 lines
- with commit
6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto replacefolio_set_active"):
p99 ~9.2-9.5 ms, throughput ~41.8k-43.6k op/s
- with revert of commit6cbdd9726fb5("mm/mglru: usefolio_mark_accessedto
replacefolio_set_active"):
p99 ~5.5-5.6 ms, throughput ~51.9k-53.1k op/s
- with this fix: p99 ~5.8 ms, throughput ~51.9k-52.7k op/sQuoted 5 lines
The underlying idea in both solutions is very similar: A credit-based system protects file folios when readahead proves useful. In my version, I further tuned credit accounting and used per-CPU batching to reduce atomic contention on multicore systems.
Quoted 3 lines
Your benchmark seems quite likely to access readahead folios.
Let me dig into your patch a bit deeper and see if we can find a combination of the two approaches.
Yes, it does seem to access readahead folios. I also think this benchmark exercises an access pattern that is generic enough that similar behavior could show up in a wide range of real-world applications, rather than being something specific to this particular workload.
Quoted 3 lines
BTW, does Kairui's MGLRU-FG bring your benchmark back to the performance you had at the very beginning?
Not yet, but I'll be trying Kairui's MGLRU-FG today or tomorrow and will report Back once I have some results.
Thanks,
Ehab
Quoted 2 lines
Best Regards
Barry