mm/mglru: use folio_mark_accessed to replace folio_set_active
Subsystems (4)
Barry Song (Xiaomi)
reviewerPATCH v2 1/1 mm/mglru: use folio_mark_accessed to replace folio_set_active
MGLRU gives high priority to folios mapped in page tables. As a result, folio_set_active() is invoked for all folios read during page faults. In practice, however, readahead can bring in many folios that are never accessed via page tables.
A previous attempt by Lei Liu proposed introducing a separate LRU for readahead[1] to make readahead pages easier to reclaim, but that approach is likely over-engineered.
Before commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folios with PG_active were always placed in the youngest generation, leading to over-protection and increased refaults. After that commit, PG_active folios are placed in the second youngest generation, which is still too optimistic given the presence of readahead. In contrast, the classic active/inactive scheme is more conservative.
This patch switches to using folio_mark_accessed(). If folio_check_references() later detects referenced PTEs, the folio will be promoted based on the reference flag set by folio_mark_accessed(). We should also adjust WORKINGSET_ACTIVATE and lru_gen_folio_seq(); for example, we should not unconditionally protect anon folios accordingly.
The following uses a simple model to demonstrate why the current code is not ideal. It runs fio-3.42 in a memcg, reading a file in a strided pattern—4KB every 64KB—to simulate prefaulted pages that may not be accessed.
#!/bin/bash
CG_NAME="mglru_verify_test"
CG_PATH="/sys/fs/cgroup/$CG_NAME"
MEM_LIMIT="400M"
HOT_SIZE="600M"
# 1. Environment Setup
sudo rmdir "$CG_PATH" 2>/dev/null
sudo mkdir -p "$CG_PATH"
sudo chown -R $USER:$USER "$CG_PATH"
echo "$MEM_LIMIT" > "$CG_PATH/memory.max"
# 2. Prepare Data Files
dd if=/dev/urandom of=hot_data.bin bs=1M count=600 conv=notrunc 2>/dev/null sync
echo 3 > /proc/sys/vm/drop_caches
# 3. Start Workload (Working Set)
(
echo $BASHPID > "$CG_PATH/cgroup.procs"
exec ./fio-3.42 --name=hot_ws --rw=read --bs=4K --size=$HOT_SIZE --runtime=600 \
--zonemode=strided --zonesize=4K --zonerange=64K \
--time_based --direct=0 --filename=hot_data.bin --ioengine=mmap \
--fadvise_hint=0 --group_reporting --numjobs=1 > fio.stats ) &
WORKLOAD_PID=$!
# 4. Waiting for hot data to warm up
sleep 30
BASE_FILE=$(grep "workingset_refault_file" "$CG_PATH/memory.stat" | awk '{print $2}')
# 5. Running workload for 60second
sleep 60
# 6. Report refault and IO bandwidth
FINAL_FILE=$(grep "workingset_refault_file" "$CG_PATH/memory.stat" | awk '{print $2}') FINAL_D_FILE=$((FINAL_FILE - BASE_FILE))
echo "File Refault Delta is $FINAL_D_FILE"
kill $WORKLOAD_PID 2>/dev/null
sleep 2
grep -E "READ|WRITE" fio.stats \
| awk '{for(i=1;i<=NF;i++){if($i ~ /^bw=/) bw=$i; if($i ~ /^io=/) io=$i} print $1, bw, io}' rm -f hot_data.bin fio.stats
Without the patch, we observed 12883855 file refaults and a very low bandwidth of 58.5 MiB/s, because prefaulted but unused pages occupy hot positions, continuously pushing out the real working set and causing incorrect reclaim. With the patch, we observed 0 refaults and bandwidth increased to 5078 MiB/s.
Running the same test on x86:
w/o patch:
File Refault Delta is 3240029
READ: bw=13.2MiB/s io=1183MiB
w/ patch:
File Refault Delta is 0
READ: bw=7708MiB/s io=676GiB
On x86, running a kernel build inside a memcg with a 1GB memory limit using 20 threads.
w/o patch:
real 1m50.764s
user 25m32.305s
sys 4m0.012s
pswpin: 1333245
pswpout: 4366443
pgpgin: 6962592
pgpgout: 17780712swpout_zero: 1019603swpin_zero: 14764refault_file: 287794refault_anon: 1347963
w/ patch:
real 1m48.915s
user 25m31.261s
sys 3m43.685s
pswpin: 915629
pswpout: 3207173
pgpgin: 5249268
pgpgout: 13154492swpout_zero: 816100swpin_zero: 15676refault_file: 257271refault_anon: 931259
active/inactive LRU:
real 1m49.928s
user 25m28.196s
sys 3m40.740s
pswpin: 463452
pswpout: 2309119
pgpgin: 4438856
pgpgout: 9568628swpout_zero: 743704swpin_zero: 7244refault_file: 562555refault_anon: 470694
Lance and Xueyuan made a huge contribution to this patch through testing.
[1] https://lore.kernel.org/linux-mm/[email protected]/
---
-v2:
* Fix WORKINGSET_ACTIVATE - workingset will be set to active during refault;
* Avoid unconditional protecting anon folios in lru_gen_folio_seq();
* Also adjusted workingset set accordingly in folio_check_references().
-v1:
https://lore.kernel.org/linux-mm/[email protected]/
-rfc was:
[PATCH RFC] mm/mglru: lazily activate folios while folios are really mapped
https://lore.kernel.org/linux-mm/[email protected]/
include/linux/mm_inline.h | 7 +++---- mm/swap.c | 8 ++++++-- mm/vmscan.c | 3 ++- mm/workingset.c | 8 ++++---- 4 files changed, 15 insertions(+), 11 deletions(-)
include/linux/mm_inline.h +3 -4
@@ -242,12 +242,11 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, gen =MIN_NR_GENS -folio_test_workingset (folio); else if (reclaiming) gen =MAX_NR_GENS ; - else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) || - (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio)))) + else if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio))) gen =MIN_NR_GENS ; else - gen =MAX_NR_GENS -folio_test_workingset (folio); + gen =MAX_NR_GENS -folio_test_workingset (folio) ||folio_test_referenced (folio);return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); }
mm/swap.c +6 -2
@@ -511,8 +511,12 @@ void folio_add_lru(struct folio *folio)/* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled () && !folio_test_unevictable (folio) && -lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) -folio_set_active (folio); +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) { +if (folio_test_workingset (folio)) +folio_set_active (folio); +else if (!folio_test_referenced (folio)) +folio_mark_accessed (folio); + }folio_batch_add_and_move (folio, lru_add); }
mm/vmscan.c +2 -1
@@ -848,7 +848,8 @@ static bool lru_gen_set_refs(struct folio *folio) return false; } -set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset)); +if (folio_test_active (folio)) +set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset));return true; }#else
mm/workingset.c +4 -4
@@ -319,11 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); -/* see folio_add_lru() where folio_set_active() will be called */ -if (lru_gen_in_fault ()) -mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); -if (workingset) { +if (lru_gen_in_fault ()) { +folio_set_active (folio); +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); + }folio_set_workingset (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + type, delta); }else
Signature · 1 line
2.34.1
Quoted 24 lines · Barry Song (Xiaomi)
MGLRU gives high priority to folios mapped in page tables. As a result,
folio_set_active()is invoked for all folios read during page faults. In practice, however, readahead can bring in many folios that are never accessed via page tables.A previous attempt by Lei Liu proposed introducing a separate LRU for readahead[1] to make readahead pages easier to reclaim, but that approach is likely over-engineered.
Before commit
4d5d14a01e2c("mm/mglru: rework workingset protection"), folios withPG_activewere always placed in the youngest generation, leading to over-protection and increased refaults. After that commit,PG_activefolios are placed in the second youngest generation, which is still too optimistic given the presence of readahead. In contrast, the classic active/inactive scheme is more conservative.This patch switches to using
folio_mark_accessed(). Iffolio_check_references()later detects referenced PTEs, the folio will be promoted based on the reference flag set byfolio_mark_accessed(). We should also adjustWORKINGSET_ACTIVATEandlru_gen_folio_seq(); for example, we should not unconditionally protect anon folios accordingly.
Quoted 4 lines
The following uses a simple model to demonstrate why the current code is not ideal. It runs fio-3.42 in a memcg, reading a file in a strided pattern—4KB every 64KB—to simulate prefaulted pages that may not be accessed.
Thanks so much for the update! This looks much better than V1 RFC.
Quoted 6 lines
Without the patch, we observed 12883855 file refaults and a very low bandwidth of 58.5 MiB/s, because prefaulted but unused pages occupy hot positions, continuously pushing out the real working set and causing incorrect reclaim. With the patch, we observed 0 refaults and bandwidth increased to 5078 MiB/s.
Quoted 8 lines
Running the same test on x86:
w/o patch:
File Refault Delta is 3240029
READ: bw=13.2MiB/s io=1183MiBw/ patch:
File Refault Delta is 0
READ: bw=7708MiB/s io=676GiB
Really nice result here.
include/linux/mm_inline.h static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · line 246
Quoted 13 lines
include/linux/mm_inline.h +2
-3@@ -242,12 +242,11 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, gen =MIN_NR_GENS -folio_test_workingset (folio); else if (reclaiming) gen =MAX_NR_GENS ; - else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) || - (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio)))) + else if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))
This change is more than just "use folio_mark_accessed to replace folio_set_active", you are changing the pattern for all anon folios here? But I think we are mostly working with file folios here especially in the first test you posted?
I do like your new if condition though, but maybe it's not the right time, or better add some words about it, how anon are handled now.
include/linux/mm_inline.h static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · line 249
Quoted 4 lines
include/linux/mm_inline.h +1
-1gen =MIN_NR_GENS ; else - gen =MAX_NR_GENS -folio_test_workingset (folio); + gen =MAX_NR_GENS -folio_test_workingset (folio) ||folio_test_referenced (folio);
Nice, this aligns exactly with what I want here :), see lru_gen_folio_seq:
https://lore.kernel.org/linux-mm/[email protected]/
But I think you got the precedence wrong? Did you mean:
(MAX_NR_GENS - folio_test_workingset(folio)) || folio_test_referenced(folio)
Or?MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio))
Quoted 8 lines
include/linux/mm_inline.h +0
-0return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); }mm/swap.c +0
-0
mm/swap.c void folio_add_lru(struct folio *folio) · line 516
Quoted 8 lines
mm/swap.c +3
-2@@ -511,8 +511,12 @@ void folio_add_lru(struct folio *folio)/* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled () && !folio_test_unevictable (folio) && -lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) -folio_set_active (folio); +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) { +if (folio_test_workingset (folio)) +folio_set_active (folio);
What if we also move this set_active to mm/workingset.c? I think that the only place where a folio with PG_workingset can get here?
mm/swap.c void folio_add_lru(struct folio *folio) · line 519
Quoted 3 lines
mm/swap.c +3
-0+else if (!folio_test_referenced (folio)) +folio_mark_accessed (folio); + }
Same here? Also this folio_mark_accessed only means folio_set_referenced here, which matches the folio_test_referenced check you just added in lru_gen_folio_seq right?
Quoted 8 lines
mm/swap.c +0
-0folio_batch_add_and_move (folio, lru_add); }mm/vmscan.c +0
-0
mm/vmscan.c static bool lru_gen_set_refs(struct folio *folio) · line 852
Quoted 6 lines
mm/vmscan.c +2
-1@@ -848,7 +848,8 @@ static bool lru_gen_set_refs(struct folio *folio)return false; } -set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset)); +if (folio_test_active (folio)) +set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset));
This looks a bit strange, PG_active is always unset for on LRU folios for MGLRU right?
There is even a comment for that:
"PG_active is always cleared while a folio is on one of lrugen->folios[]"
And lru_gen_set_refs is mostly for on-LRU folios or isolated ones, either would have this flag. So PG_workingset seems broken by this point, rmap walk no longer get PG_workingset set, which removes the "promote on 2nd access" mechanism entirely?
Quoted 10 lines
mm/vmscan.c +0
-0return true; }#else mm/workingset.c +0
-0@@ -319,11 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]);
mm/workingset.c static void lru_gen_refault(struct folio *folio, void *shadow) · line 326
Quoted 10 lines
mm/workingset.c +4
-4-/* see folio_add_lru() where folio_set_active() will be called */ -if (lru_gen_in_fault ()) -mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); -if (workingset) { +if (lru_gen_in_fault ()) { +folio_set_active (folio); +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); + }
Overall I agree with this directly that file folios shouldn't get over protected. It will be even better if we actually activate the folio using things like refault distance... Which can be done later.
BTW, this would break reactivation if we age periodically. That is less of a problem compared to overprotection, I think.
And do you think we can keep a folio_set_active if folio_test_anon == true in swap.c? Proactive activation relies on the assumption that applications never expect an jitter for ordinary memory access, which holds true for anon folios, but file folios don't follow that idea.
And I'm not trying to seize the topic but my previous experiment using refault distance-based reactivation resulted in a 500% performance gain for some databases with no other regressions observed:
https://lwn.net/Articles/945266/
So for this patch I think it's on the right track, we can definitely merge some follow up of this patch first, and then refine the pattern step by step toward refault distance, MGLRU-FG or both plus others. That will take a long time, so let's fix the low-hanging fruit first.
Really looking forward to a few more updates from this patch :) !
Barry Song
reviewerQuoted 69 lines · Kairui Song
Quoted 24 lines · Barry Song (Xiaomi)
MGLRU gives high priority to folios mapped in page tables. As a result,
folio_set_active()is invoked for all folios read during page faults. In practice, however, readahead can bring in many folios that are never accessed via page tables.A previous attempt by Lei Liu proposed introducing a separate LRU for readahead[1] to make readahead pages easier to reclaim, but that approach is likely over-engineered.
Before commit
4d5d14a01e2c("mm/mglru: rework workingset protection"), folios withPG_activewere always placed in the youngest generation, leading to over-protection and increased refaults. After that commit,PG_activefolios are placed in the second youngest generation, which is still too optimistic given the presence of readahead. In contrast, the classic active/inactive scheme is more conservative.This patch switches to using
folio_mark_accessed(). Iffolio_check_references()later detects referenced PTEs, the folio will be promoted based on the reference flag set byfolio_mark_accessed(). We should also adjustWORKINGSET_ACTIVATEandlru_gen_folio_seq(); for example, we should not unconditionally protect anon folios accordingly.Quoted 4 lines
The following uses a simple model to demonstrate why the current code is not ideal. It runs fio-3.42 in a memcg, reading a file in a strided pattern—4KB every 64KB—to simulate prefaulted pages that may not be accessed.
Thanks so much for the update! This looks much better than V1 RFC.
Quoted 6 lines
Without the patch, we observed 12883855 file refaults and a very low bandwidth of 58.5 MiB/s, because prefaulted but unused pages occupy hot positions, continuously pushing out the real working set and causing incorrect reclaim. With the patch, we observed 0 refaults and bandwidth increased to 5078 MiB/s.
Quoted 8 lines
Running the same test on x86:
w/o patch:
File Refault Delta is 3240029
READ: bw=13.2MiB/s io=1183MiBw/ patch:
File Refault Delta is 0
READ: bw=7708MiB/s io=676GiBReally nice result here.
include/linux/mm_inline.h static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · line 246
Quoted 13 lines
include/linux/mm_inline.h +2
-3@@ -242,12 +242,11 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, gen =MIN_NR_GENS -folio_test_workingset (folio); else if (reclaiming) gen =MAX_NR_GENS ; - else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) || - (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio)))) + else if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))This change is more than just "use
folio_mark_accessedto replacefolio_set_active", you are changing the pattern for all anon folios here? But I think we are mostly working with file folios here especially in the first test you posted?
Quoted 2 lines
I do like your new if condition though, but maybe it's not the right time, or better add some words about it, how anon are handled now.
Right. For anon folios, the generation is currently hardcoded to MIN_NR_GENS, so skipping folio_set_active will not help anon pages. It is hardcoded regardless of the current anon folio state. This change actually aligns anon behavior with file folios. Otherwise, skipping folio_set_active() would have no effect on anon folios.
Maybe I can change the subject to " mm/mglru: avoid over-promoting PF folios" and add some words in v3.
Quoted 6 lines
include/linux/mm_inline.h static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · line 249
Quoted 5 lines
include/linux/mm_inline.h +1
-1gen =MIN_NR_GENS ; else - gen =MAX_NR_GENS -folio_test_workingset (folio); + gen =MAX_NR_GENS -folio_test_workingset (folio) ||folio_test_referenced (folio);
Quoted 8 lines
Nice, this aligns exactly with what I want here :), see
lru_gen_folio_seq:
https://lore.kernel.org/linux-mm/[email protected]/But I think you got the precedence wrong? Did you mean:
(MAX_NR_GENS-folio_test_workingset(folio)) ||folio_test_referenced(folio)Or?
MAX_NR_GENS- (folio_test_workingset(folio)||folio_test_referenced(folio))
I mean this one. Sorry for the confusion. will fix it in v3.
Quoted 10 lines
Quoted 10 lines
include/linux/mm_inline.h +0
-0return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); }mm/swap.c +0
-0@@ -511,8 +511,12 @@ void folio_add_lru(struct folio *folio)
Quoted 10 lines
mm/swap.c void folio_add_lru(struct folio *folio) · line 516
Quoted 7 lines
mm/swap.c +3
-2/* see the comment in lru_gen_folio_seq() */ if (lru_gen_enabled () && !folio_test_unevictable (folio) && -lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) -folio_set_active (folio); +lru_gen_in_fault () && !(current->flags &PF_MEMALLOC )) { +if (folio_test_workingset (folio)) +folio_set_active (folio);What if we also move this
set_activetomm/workingset.c? I think that the only place where a folio withPG_workingsetcan get here?
I agree that folio_set_active() is duplicated between mm/workingset.c and here. Actually, I should have removed folio_set_active(folio) from mm/workingset.c to centralize the active/referenced state handling in folio_add_lru(). But somehow I forgot to do that before sending the patch.
Maybe we should just centralize it for now; that seems better. We can reconsider it later together with your larger follow-up work.
Quoted 8 lines
mm/swap.c void folio_add_lru(struct folio *folio) · line 519
Quoted 4 lines
mm/swap.c +3
-0+else if (!folio_test_referenced (folio)) +folio_mark_accessed (folio); + }Same here? Also this
folio_mark_accessedonly meansfolio_set_referencedhere, which matches thefolio_test_referencedcheck you just added inlru_gen_folio_seqright?
Right. With folio_test_referenced(), we place folios into the second oldest generation. Otherwise, putting them into the oldest generation seems too aggressive for scanning. But putting them into the active generation also seems too aggressive for promotion.
Quoted 19 lines
Quoted 9 lines
mm/swap.c +0
-0folio_batch_add_and_move (folio, lru_add); }mm/vmscan.c +0
-0mm/vmscan.c static bool lru_gen_set_refs(struct folio *folio) · line 852
Quoted 6 lines
mm/vmscan.c +2
-1@@ -848,7 +848,8 @@ static bool lru_gen_set_refs(struct folio *folio)return false; } -set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset)); +if (folio_test_active (folio)) +set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_workingset));This looks a bit strange,
PG_activeis always unset for on LRU folios for MGLRU right?
Quoted 2 lines
There is even a comment for that:
"PG_activeis always cleared while a folio is on one of lrugen->folios[]"
There still seem to be some cases where this flag remains set after I added the printk.
@@ -848,8 +848,10 @@ static bool lru_gen_set_refs(struct folio *folio)
return false;
}- if (folio_test_active(folio))
+ if (folio_test_active(folio)) {
+ pr_err("%s folio:%lx\n", __func__, folio);
set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS,
BIT(PG_workingset));
+ }
return true;}
[ 78.928932] vmscan: lru_gen_set_refs folio:fffff645c58d99c0 [ 81.198056] vmscan: lru_gen_set_refs folio:fffff645c791a200 [ 94.071659] vmscan: lru_gen_set_refs folio:fffff645c773c8c0 [ 94.116162] vmscan: lru_gen_set_refs folio:fffff645c4f02300 [ 110.215317] vmscan: lru_gen_set_refs folio:fffff645c7c8ea40 [ 118.679921] vmscan: lru_gen_set_refs folio:fffff645c5727840 [ 135.709167] vmscan: lru_gen_set_refs folio:fffff645c51d2ac0 [ 139.118206] vmscan: lru_gen_set_refs folio:fffff645c7749fc0
But you are generally right. Those active flags are probably set after folios are added to the LRU.
Quoted 5 lines
And
lru_gen_set_refsis mostly for on-LRU folios or isolated ones, either would have this flag. SoPG_workingsetseems broken by this point, rmap walk no longer getPG_workingsetset, which removes the "promote on 2nd access" mechanism entirely?
Yes. I was trying to preserve the “promote on second access” behavior, so I used the “active” flag. But it is somewhat misused.
Before this patch:
1st scan sets the referenced flag;
2nd scan promotes based on seeing the referenced flag set by the first access.
Now the flow is:
1st scan already observes the referenced flag;
2nd scan tries to rely on the active flag. However, as you pointed out, the active flag
is cleared after being set, so it can’t reliably serve this purpose.
Maybe I can add a ref and only set workingset when both PG_referenced and that ref are present.
@@ -848,8 +848,11 @@ static bool lru_gen_set_refs(struct folio *folio)
return false;
}- if (folio_test_active(folio))
+ /* Promote on second access */
+ if (folio_lru_refs(folio) > 1)
set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS,
BIT(PG_workingset));
+ else
+ folio_mark_accessed(folio);
return true;}
It’s a bit ugly, but I can’t find another flag for it. Do you have a better idea?
Quoted 26 lines
Quoted 11 lines
mm/vmscan.c +0
-0return true; }#else mm/workingset.c +0
-0@@ -319,11 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]);mm/workingset.c static void lru_gen_refault(struct folio *folio, void *shadow) · line 326
Quoted 10 lines
mm/workingset.c +4
-4-/* see folio_add_lru() where folio_set_active() will be called */ -if (lru_gen_in_fault ()) -mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); -if (workingset) { +if (lru_gen_in_fault ()) { +folio_set_active (folio); +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); + }Overall I agree with this directly that file folios shouldn't get over protected. It will be even better if we actually activate the folio using things like refault distance... Which can be done later.
Quoted 7 lines
BTW, this would break reactivation if we age periodically. That is less of a problem compared to overprotection, I think.
And do you think we can keep a
folio_set_activeiffolio_test_anon== true inswap.c? Proactive activation relies on the assumption that applications never expect an jitter for ordinary memory access, which holds true for anon folios, but file folios don't follow that idea.
I’m not entirely sure, but I see that removing folio_set_active() could also help reduce anon refaults. Also, having two separate branches for file and anon seems to make the code a bit ugly.
Quoted 5 lines
And I'm not trying to seize the topic but my previous experiment using refault distance-based reactivation resulted in a 500% performance gain for some databases with no other regressions observed:
https://lwn.net/Articles/945266/
Yep, that is really great!
I'd be happy to review and work on your refault distance-based reactivation when I have some time.
Quoted 7 lines
So for this patch I think it's on the right track, we can definitely merge some follow up of this patch first, and then refine the pattern step by step toward refault distance, MGLRU-FG or both plus others. That will take a long time, so let's fix the low-hanging fruit first.
Really looking forward to a few more updates from this patch :) !
Thanks very much for your review.
Best Regards
Barry
Barry Song
reviewerQuoted 9 lines
Quoted 5 lines
And do you think we can keep a
folio_set_activeiffolio_test_anon== true inswap.c? Proactive activation relies on the assumption that applications never expect an jitter for ordinary memory access, which holds true for anon folios, but file folios don't follow that idea.I’m not entirely sure, but I see that removing
folio_set_active()could also help reduce anon refaults. Also, having two separate branches for file and anon seems to make the code a bit ugly.
I did a quick test by restoring anon’s behavior as shown below:
include/linux/mm_inline.h +0 -0
@@ -242,8 +242,9 @@ static inline unsigned long
lru_gen_folio_seq(const struct lruvec *lruvec,
gen = MIN_NR_GENS - folio_test_workingset(folio);
else if (reclaiming)
gen = MAX_NR_GENS;- else if (folio_test_reclaim(folio) &&
- (folio_test_dirty(folio) || folio_test_writeback(folio)))
+ else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
+ (folio_test_reclaim(folio) &&
+ (folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
gen = MAX_NR_GENS - (folio_test_workingset(folio) ||folio_test_referenced(folio));
I can still see significant anon refault reduction, so I agree I can keep the existing behavior for anon. However, there is no need to set active, since (!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) is hardcoded there.
I guess it is probably because file behavior is improved, which then indirectly improves anon as well. So I feel there is no need to change the subject in v3.
Thanks
Barry