skip empty PUD subtrees during aging with a PUD-level Bloom filter
Subsystems (4)
Baoquan He
reviewerv3 cover skip empty PUD subtrees during aging with a PUD-level Bloom filter
Problem
=======
MGLRU's aging walks every PUD of every mm in the mm_list. For a PUD whose 512 PMDs all fail the existing PMD-level Bloom filter, the walker still descends into the PUD and iterates all 512 PMD entries. When an mm has no pages on a given node this is structural waste: lru_gen_use_mm() sets mm->lru_gen.bitmap to -1 (all nodes) on every context switch, so the aging walker of *every* node -- kswapd, direct reclaim, proactive aging -- walks the same mm.
On a 2-node KVM guest with the workload pinned to node 0, node 1's aging walks of those mms scan 1,363 leaf entries per walk while finding no folio that belongs to node 1's lruvec, and 94% of those walks are entirely empty.
Approach
========
Add a PUD-level Bloom filter, one level above the existing PMD filter. The walker now:
1. Tests the PUD filter before descending into a PUD subtree.
2. If the PUD had nothing young for this lruvec last generation, skips the whole 1 GiB region -- avoiding 512 PMD lookups.
3. walk_pmd_range() reports whether it found anything young; walk_pud_range() records that in the double-buffered PUD filter.
4. The rmap feedback path (lru_gen_look_around()) marks hot PUDs, so regions whose hotness is only observed by eviction are re-examined promptly.
5. force_scan walks bypass the PUD test, so manual aging and newly added mms always rescan and re-populate the filter.
The filter key is (mm, PUD-aligned address), as suggested by Kairui. It is a stable identity of the range on both sides of the interface which the aging walk computes from args->mm and addr, lru_gen_look_around() from vma->vm_mm and pvmw->address. So the rmap feedback path needs no page-table walk at all.
The double-buffered filter flips each generation and force_scan bypasses it, so a missed re-mark costs at most one generation, and reclaim is still safe for a wrongly skipped subtree because folio_referenced() consults the PTE young bits, not the filter.
Change log
==========
v2 -> v3:
- Use (mm, PUD-aligned address) as the PUD filter key instead of the PUD page-table pointer. This removes the pud_offset(p4d_offset(pgd_offset())) page-table walk that the v2 lru_gen_look_around() hunk had to do to obtain the key with withc Barry is very worried about adding cost to a hot paths. This is suggested by Kairui.
- Store both levels in one array, filters[BLOOM_FILTER_LEVELS] [NR_BLOOM_FILTERS], instead of separate pmd_filters/pud_filters (Kairui's idea).
- Count skipped subtrees (MM_PUD_EMPTY_SKIPPED, debugfs column "S" of TYFALWEES, and a tracepoint field) so the avoided cost is measurable.
Test
====
8 GiB KVM guest, 4 vCPU, two 4 GiB NUMA nodes, 7.4.0-rc1 with the series. "base" below is the same kernel with the filter disabled (patch 1 only):
same .config, same workload, medians of 3 runs.
1) Cross-node empty walks. 16 workers x 128 MiB pinned to node 0 (numactl --membind=0), so node 1 owns none of these pages. 100 aging passes, per pass, both nodes aged:
base filter change
node 1 leaf entries scanned 1,213 99 -92%
walks with leaves 17 1 -94%
PUDs skipped 0 52
node 0 leaf entries scanned 12,308 11,793 -4.2%
leaf entries assoc. 9,175 8,902 -3.0%
worker RSS delta 0% 0%
forced pass, node 1 leaf/walk
before the filtered passes 1,363.0 1,361.4
after them 1,363.0 1,341.3
A forced pass bypasses the filter, so it costs the same on both kernels and the same before and after 100 filtered passes: the filter is a cache, not a state. Node 0 keeps its associated leaves (-3.0% is run-to-run noise), so the coarser filter does not starve the node that owns them.
2) Correctness, 11 checks, all pass: a cold PUD is skipped; touching it does not revive it; force_scan re-marks it and the next pass walks it again; a munmap + mmap at the same address shows no stale suppression; a 1 GiB range migrated from node 0 to node 1 is re-scanned by node 1 after one forced pass; no premature reclaim in any of them; dmesg clean.
3) Reclaim pressure. Four 512 MiB workers in a memory.max=2G/swap.max=1G memcg, i.e. exactly at the limit, so reclaim is always active; each touches one 256 MiB half and flips it every second, so a half is cold for a few aging generations and then hot again. Aging runs on both nodes every 0.5 s, for 600 s; medians of 3 runs each:
oom_kill 0 vs 0
hot half resident, min 261,902 vs 261,887 (of 262,144 pages)
pgscan_anon 84.5M vs 110.6M 1.31x
pgsteal_anon 4.47M vs 3.55M 0.79x
pswpout 4.45M vs 3.54M 0.80x
workingset_refault_anon 4.47M vs 3.55M 0.79x
node 1 walks with leaves 4,568 vs 4 -99.9% No OOM, no residency loss, fewer evictions and refaults. The extra scanning is expected: pgscan_anon counts folios examined by the LRU scanner on both nodes, not page-table walks, and the baseline's own spread on these counters is 1.5x (pgscan_anon 73.1-112.4M between its three runs), so both directions are inside the noise. PUD subtrees skipped on node 0 are 0.002-0.015% of its associated leaves.
4) Low memory. Two checks for the phone case. First, when a 1 GiB PUD covers most of a process, can the filter still save anything? Second, if it cannot help, does it cost anything?
a) Aging cost on a single node. CPU time per pass, taken from the utime+stime of the task that writes to debugfs:
shape base filter change
32 GiB VA, 0.2% touched 2400 us 1750 us -27%
(32 PUDs per mm)
3 GiB VA, 2% touched 7650 us 7650 us 0%
(3 PUDs per mm)
64 MiB, fully touched 8250 us 7950 us 0%
The leaves scanned, the leaves associated and the number of walks with leaves are the same in all three rows. If a mm has only a few 1 GiB subtrees, the filter has nothing to save and it spends nothing. If a mm has more, it saves about 10.5 us of CPU per skipped subtree per pass, which is the cost of 512 PMD-level Bloom tests.
b) The pressure test of (3), repeated on a single-node 3.9 GiB guest. mem=6G cuts the 2x4 GiB memory layout so that one node has 3.9 GiB and the other has none. 2 runs each:
oom_kill 0 vs 0
hot half resident, min 262,121 vs 262,051 (of 262,144 pages)
pgscan_anon 99.6M vs 97.3M 0.98x
pgsteal_anon 3.50M vs 3.54M 1.01x
pswpout 3.49M vs 3.53M 1.01x
workingset_refault_anon 3.50M vs 3.54M 1.01x
Residency is the same, and there is no OOM. The patched kernel evicts and refaults 1% more, and scans 2% less. We think this is noise, for two reasons. Between two runs of the baseline kernel the same numbers move by 16%. And in the 2-node test of (3) the same ratio is 0.79x -- lower, not higher -- so the direction is not stable across tests.
Even if the 1% is real, it is what the design expects. A subtree is skipped only when it had no young page for this lruvec in the last generation. The pages in such a subtree are already cold, so it is normal that they are reclaimed first. On the only node the filter skips 16-18 subtrees per pass, which is 0.013% of its associated leaves.
Baoquan He (4):
mm/mglru: add MM_WALK_EMPTY stats and tracepoint mm/mglru: add PUD-level Bloom filter state and generic helpers mm/mglru: skip cold PUD subtrees during aging mm/mglru: count PUD subtrees skipped by the PUD-level filter
include/linux/mmzone.h | 15 ++++- mm/trace_vmscan.h | 30 +++++++++ mm/vmscan.c | 134 +++++++++++++++++++++++++++++++++-------- 3 files changed, 151 insertions(+), 28 deletions(-)
Signature · 1 line
2.52.0
Baoquan He
reviewerPATCH v3 1/4 mm/mglru: add MM_WALK_EMPTY stats and tracepoint
From: Baoquan He <[email protected]>
Add per-walk counters to measure empty aging walks which traverse an mm's page tables but find no folio for the current lruvec (node+memcg). These are common on multi-NUMA node systems because lru_gen_use_mm() marks an mm for all nodes at every context switch.
New counters (accumulated in mm_state->stats[]):
MM_LEAF_ASSOCIATED - leaf entries whose folio is in this lruvec MM_WALK_TOTAL - page-table walks completed MM_WALK_EMPTY - walks that found no folio in this lruvec MM_LEAF_EMPTY_WALKS - leaf entries scanned during empty walks
A new tracepoint, mm_vmscan_lru_gen_walk(), fires after each walk, and the debugfs lru_gen output ("TYFALWEE") exposes the new counters.
--- include/linux/mmzone.h | 4 ++++ mm/trace_vmscan.h | 28 ++++++++++++++++++++++++++++ mm/vmscan.c | 32 ++++++++++++++++++++++++++++---- 3 files changed, 60 insertions(+), 4 deletions(-)
include/linux/mmzone.h +4 -0
@@ -579,6 +579,10 @@ enum {MM_LEAF_YOUNG ,/* young leaf entries */ MM_NONLEAF_FOUND ,/* non-leaf entries found in Bloom filters */ MM_NONLEAF_ADDED ,/* non-leaf entries added to Bloom filters */ +MM_LEAF_ASSOCIATED ,/* leaf entries whose folio is in this lruvec (node+memcg) */ +MM_WALK_TOTAL ,/* page-table walks completed */ +MM_WALK_EMPTY ,/* walks that found no folio in this lruvec */ +MM_LEAF_EMPTY_WALKS ,/* leaf entries scanned during empty walks */ NR_MM_STATS };
mm/trace_vmscan.h +28 -0
@@ -763,6 +763,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, __entry->nid, __print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops)) ); +TRACE_EVENT (mm_vmscan_lru_gen_walk, + + TP_PROTO(int nid, unsigned long seq, int leaf_total, + int leaf_associated, bool empty), + +TP_ARGS (nid, seq, leaf_total, leaf_associated, empty), + +TP_STRUCT__entry ( + __field(int, nid) + __field(unsigned long, seq) + __field(int, leaf_total) + __field(int, leaf_associated) +__field (bool, empty) + ), + +TP_fast_assign ( + __entry->nid = nid; + __entry->seq = seq; + __entry->leaf_total = leaf_total; + __entry->leaf_associated = leaf_associated; + __entry->empty = empty; + ), + +TP_printk ("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d" , + __entry->nid, __entry->seq, __entry->leaf_total, + __entry->leaf_associated, __entry->empty) +); +#endif /* _TRACE_VMSCAN_H */ /* This part must be outside protection */
mm/vmscan.c +28 -4
@@ -3590,6 +3590,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ; + walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (folio_test_large (folio)) {const unsigned int max_nr = (end - addr) >>PAGE_SHIFT ;
@@ -3688,6 +3690,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_areaif (!folio) goto next; + walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (!pmdp_test_and_clear_young_notify (vma, addr, pmd + i)) goto next;
@@ -4151,8 +4155,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,do { success =iterate_mm_list (walk, &mm); -if (mm) +if (mm) { +bool empty = false; +walk_mm (mm, walk); +/* +* A walk that traversed page tables but found no folio +* belonging to this lruvec (node+memcg) is pure waste. +*/ +if (walk->mm_stats[MM_LEAF_TOTAL ]) { + walk->mm_stats[MM_WALK_TOTAL ]++; +if (walk->mm_stats[MM_LEAF_ASSOCIATED ] ==0 ) { + walk->mm_stats[MM_WALK_EMPTY ]++; + walk->mm_stats[MM_LEAF_EMPTY_WALKS ] += + walk->mm_stats[MM_LEAF_TOTAL ]; + empty = true; + } + } +trace_mm_vmscan_lru_gen_walk ( +lruvec_pgdat (lruvec)->node_id, walk->seq, + walk->mm_stats[MM_LEAF_TOTAL ], + walk->mm_stats[MM_LEAF_ASSOCIATED ], empty); + } }while (mm); done:if (success) {
@@ -5638,14 +5662,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,seq_puts (m," " );for (i =0 ; i <NR_MM_STATS ; i++) { -const char *s ="xxxx" ; +const char *s ="xxxxxxxx" ;unsigned long n =0 ;if (seq == max_seq &&NR_HIST_GENS ==1 ) { - s ="TYFA" ; + s ="TYFALWEE" ; n =READ_ONCE (mm_state->stats[hist][i]); }else if (seq != max_seq &&NR_HIST_GENS >1 ) { - s ="tyfa" ; + s ="tyfalwee" ; n =READ_ONCE (mm_state->stats[hist][i]); }
Signature · 1 line
2.52.0
Baoquan He
reviewerRe: [PATCH v3 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint
Quoted 14 lines · Baoquan He
From: Baoquan He <[email protected]>
Add per-walk counters to measure empty aging walks which traverse an mm's page tables but find no folio for the current lruvec (node+memcg). These are common on multi-NUMA node systems because
lru_gen_use_mm()marks an mm for all nodes at every context switch.New counters (accumulated in mm_state->stats[]):
MM_LEAF_ASSOCIATED - leaf entries whose folio is in this lruvec MM_WALK_TOTAL - page-table walks completed MM_WALK_EMPTY - walks that found no folio in this lruvec MM_LEAF_EMPTY_WALKS - leaf entries scanned during empty walks
Quoted 7 lines
A new tracepoint,
mm_vmscan_lru_gen_walk(), fires after each walk, and the debugfslru_genoutput ("TYFALWEE") exposes the new counters.Reviewed-by Barry Song <[email protected]>Signed-off-by Baoquan He <[email protected]>Signed-off-by Baoquan He <[email protected]>
~~~
Git format-patch automatically added my linux.dev mail address, I should have noticed this and remove it.
Quoted 133 lines
--- include/linux/mmzone.h | 4 ++++ mm/trace_vmscan.h | 28 ++++++++++++++++++++++++++++ mm/vmscan.c | 32 ++++++++++++++++++++++++++++---- 3 files changed, 60 insertions(+), 4 deletions(-)include/linux/mmzone.h +4
-0@@ -579,6 +579,10 @@ enum {MM_LEAF_YOUNG ,/* young leaf entries */ MM_NONLEAF_FOUND ,/* non-leaf entries found in Bloom filters */ MM_NONLEAF_ADDED ,/* non-leaf entries added to Bloom filters */ +MM_LEAF_ASSOCIATED ,/* leaf entries whose folio is in this lruvec (node+memcg) */ +MM_WALK_TOTAL ,/* page-table walks completed */ +MM_WALK_EMPTY ,/* walks that found no folio in this lruvec */ +MM_LEAF_EMPTY_WALKS ,/* leaf entries scanned during empty walks */ NR_MM_STATS };mm/trace_vmscan.h +28
-0@@ -763,6 +763,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, __entry->nid, __print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops)) ); +TRACE_EVENT (mm_vmscan_lru_gen_walk, + + TP_PROTO(int nid, unsigned long seq, int leaf_total, + int leaf_associated, bool empty), + +TP_ARGS (nid, seq, leaf_total, leaf_associated, empty), + +TP_STRUCT__entry ( + __field(int, nid) + __field(unsigned long, seq) + __field(int, leaf_total) + __field(int, leaf_associated) +__field (bool, empty) + ), + +TP_fast_assign ( + __entry->nid = nid; + __entry->seq = seq; + __entry->leaf_total = leaf_total; + __entry->leaf_associated = leaf_associated; + __entry->empty = empty; + ), + +TP_printk ("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d" , + __entry->nid, __entry->seq, __entry->leaf_total, + __entry->leaf_associated, __entry->empty) +); +#endif /* _TRACE_VMSCAN_H */ /* This part must be outside protection */ mm/vmscan.c +28
-4@@ -3590,6 +3590,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ; + walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (folio_test_large (folio)) {const unsigned int max_nr = (end - addr) >>PAGE_SHIFT ;@@ -3688,6 +3690,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_areaif (!folio) goto next; + walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (!pmdp_test_and_clear_young_notify (vma, addr, pmd + i)) goto next;@@ -4151,8 +4155,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,do { success =iterate_mm_list (walk, &mm); -if (mm) +if (mm) { +bool empty = false; +walk_mm (mm, walk); +/* +* A walk that traversed page tables but found no folio +* belonging to this lruvec (node+memcg) is pure waste. +*/ +if (walk->mm_stats[MM_LEAF_TOTAL ]) { + walk->mm_stats[MM_WALK_TOTAL ]++; +if (walk->mm_stats[MM_LEAF_ASSOCIATED ] ==0 ) { + walk->mm_stats[MM_WALK_EMPTY ]++; + walk->mm_stats[MM_LEAF_EMPTY_WALKS ] += + walk->mm_stats[MM_LEAF_TOTAL ]; + empty = true; + } + } +trace_mm_vmscan_lru_gen_walk ( +lruvec_pgdat (lruvec)->node_id, walk->seq, + walk->mm_stats[MM_LEAF_TOTAL ], + walk->mm_stats[MM_LEAF_ASSOCIATED ], empty); + } }while (mm); done:if (success) {@@ -5638,14 +5662,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,seq_puts (m," " );for (i =0 ; i <NR_MM_STATS ; i++) { -const char *s ="xxxx" ; +const char *s ="xxxxxxxx" ;unsigned long n =0 ;if (seq == max_seq &&NR_HIST_GENS ==1 ) { - s ="TYFA" ; + s ="TYFALWEE" ; n =READ_ONCE (mm_state->stats[hist][i]); }else if (seq != max_seq &&NR_HIST_GENS >1 ) { - s ="tyfa" ; + s ="tyfalwee" ; n =READ_ONCE (mm_state->stats[hist][i]); }Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v3 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint
In reply to Baoquan HeThank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] mm/mglru: get_pte_pfn() and get_pmd_pfn() skip non-young entries in walk stats - [Medium]…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm/mglru: get_pte_pfn() and get_pmd_pfn() skip non-young entries in walk stats
- [Medium] mm/mglru: walk_pte_range() undercounts MM_LEAF_ASSOCIATED on large folios
- [Low] mm/mglru: mm_vmscan_lru_gen_walk tracepoint defined outside CONFIG_LRU_GEN
--
commit b5309c24549f959bf1eb3e88053e43540b30ee90
Author: Baoquan He <[email protected]>
mm/mglru: add MM_WALK_EMPTY stats and tracepoint
This commit adds per-walk statistics and a tracepoint to track MGLRU aging page-table walks that find no folios belonging to the target lruvec.
mm/trace_vmscan.h TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, · line 769
Quoted 12 lines
mm/trace_vmscan.h +4
-0@@ -763,6 +763,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, __entry->nid, __print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops)) ); +TRACE_EVENT(mm_vmscan_lru_gen_walk, + + TP_PROTO(int nid, unsigned long seq, int leaf_total, + int leaf_associated, bool empty),
[Severity: Low]
Should mm_vmscan_lru_gen_walk be placed inside the #ifdef CONFIG_LRU_GEN block alongside mm_mglru_scan_folios and mm_mglru_aging? Its only caller in try_to_inc_max_seq() is built under CONFIG_LRU_GEN, so defining it here generates unused tracepoint code and tracefs entries when CONFIG_LRU_GEN is disabled.
[ ... ]
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3588
Quoted 12 lines
mm/vmscan.c +2
-0@@ -3582,6 +3582,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ; + walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (folio_test_large (folio)) {const unsigned int max_nr = (end - addr) >>PAGE_SHIFT ;
[Severity: Medium]
Does walk_pte_range() undercount MM_LEAF_ASSOCIATED when a large folio spans a batch of contiguous PTEs?
When folio_test_large(folio) is true, folio_pte_batch_flags() sets nr to the number of batched PTEs and adds nr - 1 to walk->mm_stats[MM_LEAF_TOTAL] and nr to walk->mm_stats[MM_LEAF_YOUNG], before the loop advances by nr entries:
mm/vmscan.c:walk_pte_range() {
...
if (folio_test_large(folio)) {
const unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
nr = folio_pte_batch_flags(folio, NULL, cur_pte, &ptent,
max_nr, FPB_MERGE_YOUNG_DIRTY);
total += nr - 1;
walk->mm_stats[MM_LEAF_TOTAL] += nr - 1;
}
...
young += nr;
walk->mm_stats[MM_LEAF_YOUNG] += nr;
}Because MM_LEAF_ASSOCIATED is only incremented by 1 before the batch size is computed, the remaining nr - 1 PTEs in the batch are not counted, and MM_LEAF_YOUNG can end up larger than MM_LEAF_ASSOCIATED.
[ ... ]
Quoted 18 lines
mm/vmscan.c +12
-1@@ -4158,8 +4162,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, do { success = iterate_mm_list(walk, &mm); - if (mm) + if (mm) { + bool empty = false; + walk_mm(mm, walk); + /* + * A walk that traversed page tables but found no folio + * belonging to this lruvec (node+memcg) is pure waste. + */ + if (walk->mm_stats[MM_LEAF_TOTAL]) { + walk->mm_stats[MM_WALK_TOTAL]++; + if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) { + walk->mm_stats[MM_WALK_EMPTY]++; + walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=
mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, · line 4187
Quoted 10 lines
mm/vmscan.c +9
-0+ walk->mm_stats[MM_LEAF_TOTAL ]; + empty = true; + } + } +trace_mm_vmscan_lru_gen_walk ( +lruvec_pgdat (lruvec)->node_id, walk->seq, + walk->mm_stats[MM_LEAF_TOTAL ], + walk->mm_stats[MM_LEAF_ASSOCIATED ], empty); + } }while (mm);
[Severity: Medium]
Does this count a walk as empty whenever an mm has folios in this lruvec whose PTEs and PMDs are all non-young?
In walk_pte_range() and walk_pmd_range_locked(), MM_LEAF_ASSOCIATED is incremented only after get_pte_pfn() or get_pmd_pfn() succeeds, but both helpers reject non-young entries up front when the mm has no MMU notifiers:
mm/vmscan.c:get_pte_pfn() {
...
if (!pte_young(pte) && !mm_has_notifiers(vma->vm_mm))
return -1;As a result, non-young leaf entries mapping folios in this lruvec are never counted in MM_LEAF_ASSOCIATED, and any walk over an mm whose mapped folios in this lruvec are all cold has MM_LEAF_ASSOCIATED == 0 and is recorded in MM_WALK_EMPTY, MM_LEAF_EMPTY_WALKS, and trace_mm_vmscan_lru_gen_walk() with empty=1.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Baoquan He
reviewerRe: [PATCH v3 1/4] mm/mglru: add MM_WALK_EMPTY stats and tracepoint
Quoted 8 lines · On 11/14/26 at 03:42am, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm/mglru:get_pte_pfn()andget_pmd_pfn()skip non-young entries in walk stats
- [Medium] mm/mglru:walk_pte_range()undercountsMM_LEAF_ASSOCIATEDon large folios
- [Low] mm/mglru:mm_vmscan_lru_gen_walktracepoint defined outsideCONFIG_LRU_GEN
Thanks for the review. All three points are correct. Please see the reply in inline paragraphs.
Quoted 21 lines
--
commit
b5309c24549f959bf1eb3e88053e43540b30ee90
Author: Baoquan He <[email protected]>mm/mglru: add
MM_WALK_EMPTYstats and tracepointThis commit adds per-walk statistics and a tracepoint to track MGLRU aging page-table walks that find no folios belonging to the target lruvec.
Quoted 11 lines
mm/trace_vmscan.h +3
-0@@ -763,6 +763,34 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, __entry->nid, __print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops)) ); +TRACE_EVENT(mm_vmscan_lru_gen_walk, + + TP_PROTO(int nid, unsigned long seq, int leaf_total,
Quoted 8 lines
mm/trace_vmscan.h TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, · line 769
Quoted 1 line
mm/trace_vmscan.h +1
-0+ int leaf_associated, bool empty),[Severity: Low]
Shouldmm_vmscan_lru_gen_walkbe placed inside the #ifdefCONFIG_LRU_GENblock alongsidemm_mglru_scan_foliosandmm_mglru_aging? Its only caller intry_to_inc_max_seq()is built underCONFIG_LRU_GEN, so defining it here generates unused tracepoint code and tracefs entries whenCONFIG_LRU_GENis disabled.
Will fix. mm_vmscan_lru_gen_walk will be put inside the same block as mm_mglru_scan_folios and mm_mglru_aging.
Quoted 11 lines
[ ... ]
Quoted 8 lines
mm/vmscan.c +0
-0@@ -3582,6 +3582,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,if (!folio)continue ;
Quoted 8 lines
mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3596
Quoted 4 lines
mm/vmscan.c +2
-0+ walk->mm_stats[MM_LEAF_ASSOCIATED ]++; +if (folio_test_large (folio)) {const unsigned int max_nr = (end - addr) >>PAGE_SHIFT ;[Severity: Medium]
Doeswalk_pte_range()undercountMM_LEAF_ASSOCIATEDwhen a large folio spans a batch of contiguous PTEs?
MM_LEAF_ASSOCIATED is not scaled by the batch size for large folios.
Will fix. For a batched large folio we now add nr - 1 as well, so MM_LEAF_YOUNG can no longer be larger than MM_LEAF_ASSOCIATED.
Quoted 68 lines
When
folio_test_large(folio)is true,folio_pte_batch_flags()sets nr to the number of batched PTEs and adds nr - 1 to walk->mm_stats[MM_LEAF_TOTAL] and nr to walk->mm_stats[MM_LEAF_YOUNG], before the loop advances by nr entries:mm/vmscan.c:walk_pte_range() { ... if (folio_test_large(folio)) { const unsigned int max_nr = (end - addr) >> PAGE_SHIFT; nr = folio_pte_batch_flags(folio, NULL, cur_pte, &ptent, max_nr, FPB_MERGE_YOUNG_DIRTY); total += nr - 1; walk->mm_stats[MM_LEAF_TOTAL] += nr - 1; } ... young += nr; walk->mm_stats[MM_LEAF_YOUNG] += nr; }Because
MM_LEAF_ASSOCIATEDis only incremented by 1 before the batch size is computed, the remaining nr - 1 PTEs in the batch are not counted, andMM_LEAF_YOUNGcan end up larger thanMM_LEAF_ASSOCIATED.[ ... ]
Quoted 18 lines
mm/vmscan.c +12
-1@@ -4158,8 +4162,28 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, do { success = iterate_mm_list(walk, &mm); - if (mm) + if (mm) { + bool empty = false; + walk_mm(mm, walk); + /* + * A walk that traversed page tables but found no folio + * belonging to this lruvec (node+memcg) is pure waste. + */ + if (walk->mm_stats[MM_LEAF_TOTAL]) { + walk->mm_stats[MM_WALK_TOTAL]++; + if (walk->mm_stats[MM_LEAF_ASSOCIATED] == 0) { + walk->mm_stats[MM_WALK_EMPTY]++; + walk->mm_stats[MM_LEAF_EMPTY_WALKS] +=mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, · line 4187
Quoted 10 lines
mm/vmscan.c +9
-0+ walk->mm_stats[MM_LEAF_TOTAL ]; + empty = true; + } + } +trace_mm_vmscan_lru_gen_walk ( +lruvec_pgdat (lruvec)->node_id, walk->seq, + walk->mm_stats[MM_LEAF_TOTAL ], + walk->mm_stats[MM_LEAF_ASSOCIATED ], empty); + } }while (mm);[Severity: Medium]
Does this count a walk as empty whenever an mm has folios in this lruvec whose PTEs and PMDs are all non-young?In
walk_pte_range()andwalk_pmd_range_locked(),MM_LEAF_ASSOCIATEDis incremented only afterget_pte_pfn()orget_pmd_pfn()succeeds, but both helpers reject non-young entries up front when the mm has no MMU notifiers:mm/vmscan.c:get_pte_pfn() { ... if (!pte_young(pte) && !mm_has_notifiers(vma->vm_mm)) return -1;
Quoted 5 lines
As a result, non-young leaf entries mapping folios in this lruvec are never counted in
MM_LEAF_ASSOCIATED, and any walk over an mm whose mapped folios in this lruvec are all cold hasMM_LEAF_ASSOCIATED== 0 and is recorded inMM_WALK_EMPTY,MM_LEAF_EMPTY_WALKS, andtrace_mm_vmscan_lru_gen_walk()with empty=1.
get_pte_pfn() / get_pmd_pfn() reject non-young entries before MM_LEAF_ASSOCIATED is counted, so a walk is reported as "empty" when the mm has folios in this lruvec but all of them are cold.
Yes, that is what the code does, and the name did not say it. The behaviour is on purpose. The PUD filter in patch 3 carries a subtree only when it had young entries for this lruvec, so "nothing young" is exactly what we want to measure. Counting the folios before the young test would add a folio lookup for every cold leaf, and that is the cost this series tries to remove.
So v4 will fix the naming and the comments instead: MM_LEAF_ASSOCIATED is "young leaf entries whose folio is in this lruvec", MM_WALK_EMPTY is "walks that found nothing young for this lruvec", and the comment in try_to_inc_max_seq() says the same.
Quoted 3 lines
Signature · 2 lines
-- Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Baoquan He
reviewerPATCH v3 2/4 mm/mglru: add PUD-level Bloom filter state and generic helpers
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Generalize the Bloom filter state and helpers so that another filter level can reuse the same hash, double-buffering and reset logic.
Replace the PMD-level filter array with a two-dimensional filters array indexed by level, and add BLOOM_FILTER_PUD and BLOOM_FILTER_PMD. The PUD-level filters operate at PUD granularity and generalize the suppression one level up: when all PMDs under a PUD fail the PMD-level filter, the whole PMD iteration can be skipped. Cross-node empty walks are the most visible case: a foreign mm's PUDs have no young entries for this lruvec, so the entire PMD iteration is pure waste.
The existing PMD-level code now goes through the generic test/update/reset_bloom_filter() helpers.
No behavior change yet; the PUD-level filters are populated and used by the next patch.
--- include/linux/mmzone.h | 10 ++++++-- mm/vmscan.c | 58 +++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 19 deletions(-)
include/linux/mmzone.h +8 -2
@@ -589,6 +589,12 @@ enum {/* double-buffering Bloom filters */ #define NR_BLOOM_FILTERS 2 +enum bloom_filter_level { +BLOOM_FILTER_PUD , +BLOOM_FILTER_PMD , +BLOOM_FILTER_LEVELS , +}; +struct lru_gen_mm_state {/* synced with max_seq after each iteration */ unsigned long seq;
@@ -596,8 +602,8 @@ struct lru_gen_mm_state {struct list_head *head;/* where the last iteration ended before */ struct list_head *tail; -/* Bloom filters flip after each iteration */ -unsigned long *filters[NR_BLOOM_FILTERS ]; +/* Bloom filters flip after each iteration, indexed by level */ +unsigned long *filters[BLOOM_FILTER_LEVELS ][NR_BLOOM_FILTERS ];/* the mm stats for debugging */ unsigned long stats[NR_HIST_GENS ][NR_MM_STATS ]; };
mm/vmscan.c +41 -17
138 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v3 2/4] mm/mglru: add PUD-level Bloom filter state and generic helpers
Quoted 21 lines · Baoquan He
From: Baoquan He <[email protected]>
Generalize the Bloom filter state and helpers so that another filter level can reuse the same hash, double-buffering and reset logic.
Replace the PMD-level filter array with a two-dimensional filters array indexed by level, and add
BLOOM_FILTER_PUDandBLOOM_FILTER_PMD. The PUD-level filters operate at PUD granularity and generalize the suppression one level up: when all PMDs under a PUD fail the PMD-level filter, the whole PMD iteration can be skipped. Cross-node empty walks are the most visible case: a foreign mm's PUDs have no young entries for this lruvec, so the entire PMD iteration is pure waste.The existing PMD-level code now goes through the generic test/update/reset_bloom_filter() helpers.
No behavior change yet; the PUD-level filters are populated and used by the next patch.
Signed-off-by Baoquan He <[email protected]>Signed-off-by Baoquan He <[email protected]>
...
mm/vmscan.c static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) · line 2800
Quoted 8 lines
mm/vmscan.c +0
-0@@ -2798,6 +2798,10 @@ static bool __maybe_unused seq_is_valid(struct lruvec *lruvec)* walk_pmd_range(); the eviction also report them when walking the rmap * in lru_gen_look_around(). *
Not a problem for this patch, but the comment above may want an update in the next commit. This describes things only at the PMD level, while the PUD level, the ~1/5 and ~1/2 FP figures still hold for (same m, k and n). But the PMD false positive only causes a wasted PTE walk if the PUD gate also passes, so the effective FP rate is weighted by the PUD part too. The PUD filter has its own FP, which is rarer but costs a whole 1 GiB subtree, so maybe the 4 KiB bitmap for PUD is not that obviously oversized (despite the PUD number are much lower than PMD we are using same sized bitmaps).
mm/vmscan.c static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) · line 2807
Quoted 7 lines
mm/vmscan.c +4
-0+* A second, coarser pair of filters sits one level up. The filters are +* indexed by level: BLOOM_FILTER_PMD for the PMD-level filters above, and +* BLOOM_FILTER_PUD for the PUD-level pair. +* * For future optimizations: * 1. It's not necessary to keep both filters all the time. The spare one can be * freed after the RCU grace period and reallocated if needed again.
Still not a problem but this optimization is more important now. We have a extra 8KB peak overhead per lruvec if I did the math right, not too bad as there won't be too many of them, even on large machines (memcg number is currently hard limited to 64k and they are lazily allocated). But worth mentioning and optimized later.
mm/vmscan.c static void get_item_key(void *item, int *key) · line 2830
Quoted 7 lines
mm/vmscan.c +1
-2@@ -2823,14 +2827,13 @@ static void get_item_key(void *item, int *key) key[1 ] = hash >>BLOOM_FILTER_SHIFT ; } -static bool test_bloom_filter (struct lru_gen_mm_state *mm_state,unsigned long seq, -void *item) +static bool __test_bloom_filter (unsigned long **filters,unsigned long seq,void *item)
There is only one caller of this (test_bloom_filter), do we really need the __ version?
Quoted 11 lines
mm/vmscan.c +1
-1{int key[2 ];unsigned long *filter;int gen =filter_gen_from_seq (seq); - filter =READ_ONCE (mm_state->filters[gen]); + filter =READ_ONCE (filters[gen]);if (!filter)return true;
mm/vmscan.c static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long s · line 2845
Quoted 6 lines
mm/vmscan.c +1
-2@@ -2839,14 +2842,13 @@ static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long sreturn test_bit (key[0 ], filter) &&test_bit (key[1 ], filter); } -static void update_bloom_filter (struct lru_gen_mm_state *mm_state,unsigned long seq, -void *item) +static void __update_bloom_filter (unsigned long **filters,unsigned long seq,void *item)
Also this one, only one caller, the __ intermediate layer seems unnecessary.
Quoted 62 lines
mm/vmscan.c +23
-5{int key[2 ];unsigned long *filter;int gen =filter_gen_from_seq (seq); - filter =READ_ONCE (mm_state->filters[gen]); + filter =READ_ONCE (filters[gen]);if (!filter)return ;@@ -2858,12 +2860,12 @@ static void update_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned longset_bit (key[1 ], filter); } -static void reset_bloom_filter (struct lru_gen_mm_state *mm_state,unsigned long seq) +static void __reset_bloom_filter (unsigned long **filters,unsigned long seq) {unsigned long *filter;int gen =filter_gen_from_seq (seq); - filter = mm_state->filters[gen]; + filter = filters[gen];if (filter) {bitmap_clear (filter,0 ,BIT (BLOOM_FILTER_SHIFT ));return ;@@ -2871,7 +2873,25 @@ static void reset_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long filter =bitmap_zalloc (BIT (BLOOM_FILTER_SHIFT ), __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); -WRITE_ONCE (mm_state->filters[gen], filter); +WRITE_ONCE (filters[gen], filter); +} + +static bool test_bloom_filter (struct lru_gen_mm_state *mm_state,int level, +unsigned long seq,void *item) +{ +return __test_bloom_filter (mm_state->filters[level], seq, item); +} + +static void update_bloom_filter (struct lru_gen_mm_state *mm_state,int level, +unsigned long seq,void *item) +{ +__update_bloom_filter (mm_state->filters[level], seq, item); +} + +static void reset_bloom_filter (struct lru_gen_mm_state *mm_state,int level, +unsigned long seq) +{ +__reset_bloom_filter (mm_state->filters[level], seq); }/****************************************************************************** @@ -3114,7 +3134,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **itespin_unlock (&mm_list->lock);if (mm && first) -reset_bloom_filter (mm_state, walk->seq +1 ); +reset_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 );if (*iter)mmdrop (*iter);
mm/vmscan.c static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, · line 3795
Quoted 6 lines
mm/vmscan.c +2
-1@@ -3771,7 +3791,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); } -if (!walk->force_scan && !test_bloom_filter (mm_state, walk->seq, pmd + i)) +if (!walk->force_scan && + !test_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq, pmd + i))
This looks nice.
Quoted 33 lines
mm/vmscan.c +4
-3continue ; walk->mm_stats[MM_NONLEAF_FOUND ]++;@@ -3782,7 +3803,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_ADDED ]++;/* carry over to the next generation */ -update_bloom_filter (mm_state, walk->seq +1 , pmd + i); +update_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 , pmd + i); }walk_pmd_range_locked (pud,-1 , vma, args, bitmap, &first);@@ -4417,8 +4438,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)/* feedback from rmap walkers to page table walkers */ if (mm_state &&suitable_to_scan (i, young)) -update_bloom_filter (mm_state,READ_ONCE (lruvec->lrugen.max_seq), - pvmw->pmd); +update_bloom_filter (mm_state,BLOOM_FILTER_PMD , +READ_ONCE (lruvec->lrugen.max_seq), pvmw->pmd);mem_cgroup_put (memcg);@@ -5981,6 +6002,7 @@ void lru_gen_init_memcg(struct mem_cgroup *memcg)void lru_gen_exit_memcg (struct mem_cgroup *memcg) {int i; +int level;int nid;struct lru_gen_mm_list *mm_list =get_mm_list (memcg);@@ -5998,9 +6020,11 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg)if (!mm_state)
mm/vmscan.c void lru_gen_exit_memcg(struct mem_cgroup *memcg) · line 6027
Quoted 10 lines
mm/vmscan.c +5
-3continue ; -for (i =0 ; i <NR_BLOOM_FILTERS ; i++) { -bitmap_free (mm_state->filters[i]); - mm_state->filters[i] =NULL ; +for (level =0 ; level <BLOOM_FILTER_LEVELS ; level++) { +for (i =0 ; i <NR_BLOOM_FILTERS ; i++) { +bitmap_free (mm_state->filters[level][i]); + mm_state->filters[level][i] =NULL ; + }
So just a few nit style and comment issue from me, rest looks good, thanks!
Baoquan He
reviewerPATCH v3 3/4 mm/mglru: skip cold PUD subtrees during aging
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
The aging walks into every PUD and runs the PMD-level Bloom filter on each PMD. Add a coarser PUD-level filter one level up:
- walk_pmd_range() reports whether the subtree held anything young for this lruvec. That is the same condition which already carries a PMD over into the next generation: a young PMD entry, a huge page owned by this lruvec, or PTEs with enough young ones to pass suitable_to_scan().
- walk_pud_range() records that in the PUD filter, and
- on subsequent generations, skips the whole PUD subtree when the filter says it had nothing young last generation.
Use (mm, PUD-aligned address) as the filter key rather than the PUD page table pointer. It is a stable identity for the range being walked, is available without walking the page tables in lru_gen_look_around(), and does not depend on the lifetime of a page table page.
The double-buffered filter flips with each new iteration, so newly hot or migrated-in pages are re-checked promptly rather than suppressed indefinitely. force_scan walks bypass the PUD test, so manual aging and newly added mms always rescan and re-populate the filter.
To keep hot regions marked, also report the covering PUD from the rmap feedback path (lru_gen_look_around()), so regions whose hotness is only observed by eviction will be re-scanned.
--- mm/vmscan.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
mm/vmscan.c +38 -7
@@ -3133,8 +3133,10 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **itespin_unlock (&mm_list->lock); -if (mm && first) +if (mm && first) {reset_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 ); +reset_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq +1 ); + }if (*iter)mmdrop (*iter);
@@ -3738,10 +3740,11 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area *first =-1 ; } -static void walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end, +static bool walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end,struct mm_walk *args) {int i; +bool young = false;pmd_t *pmd;unsigned long next;unsigned long addr;
@@ -3778,8 +3781,10 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_LEAF_TOTAL ]++; -if (pfn !=-1 ) +if (pfn !=-1 ) {walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; + }continue ; }
@@ -3789,6 +3794,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ;walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; } if (!walk->force_scan &&
@@ -3801,6 +3807,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ; walk->mm_stats[MM_NONLEAF_ADDED ]++; + young = true;/* carry over to the next generation */ update_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 , pmd + i);
@@ -3810,6 +3817,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,if (i <PTRS_PER_PMD &&get_next_vma (PUD_MASK ,PMD_SIZE , args, &start, &end)) goto restart; + +return young; } static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,
@@ -3820,6 +3829,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,unsigned long addr;unsigned long next;struct lru_gen_mm_walk *walk = args->private; +struct lru_gen_mm_state *mm_state =get_mm_state (walk->lruvec);VM_WARN_ON_ONCE (p4d_leaf (*p4d));
@@ -3827,13 +3837,29 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, restart:for (i =pud_index (start), addr = start; addr != end; i++, addr = next) {pud_t val =pudp_get (pud + i); +void *pud_key; next =pud_addr_end (addr, end);if (!pud_present (val) ||WARN_ON_ONCE (pud_leaf (val)))continue ; -walk_pmd_range (pud + i, addr, next, args); +/* +* Use (mm, PUD-aligned address) as the filter key instead of +* the page table pointer. It is stable across the aging walk +* and the rmap feedback path, and avoids walking the page +* tables in lru_gen_look_around(). +*/ + pud_key = (void *)((unsigned long )args->mm ^ (addr &PUD_MASK )); + +/* Skip a subtree that had nothing young for this lruvec last gen */ +if (!walk->force_scan && + !test_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq, pud_key)) +continue ; + +if (walk_pmd_range (pud + i, addr, next, args)) +update_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq +1 , + pud_key);if (need_resched () || walk->batched >=MAX_LRU_BATCH ) { end = (addr | ~PUD_MASK ) +1 ;
@@ -4437,9 +4463,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)lazy_mmu_mode_disable ();/* feedback from rmap walkers to page table walkers */ -if (mm_state &&suitable_to_scan (i, young)) -update_bloom_filter (mm_state,BLOOM_FILTER_PMD , -READ_ONCE (lruvec->lrugen.max_seq), pvmw->pmd); +if (mm_state &&suitable_to_scan (i, young)) { +unsigned long max_seq =READ_ONCE (lruvec->lrugen.max_seq); +void *pud_key = (void *)((unsigned long )vma->vm_mm ^ + (pvmw->address &PUD_MASK )); + +update_bloom_filter (mm_state,BLOOM_FILTER_PMD , max_seq, pvmw->pmd); +update_bloom_filter (mm_state,BLOOM_FILTER_PUD , max_seq, pud_key); + }mem_cgroup_put (memcg);
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v3 3/4] mm/mglru: skip cold PUD subtrees during aging
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] mm/mglru: walk_pmd_range() marks PUD young on non-leaf PMD accessed bit - [Medium] mm/mglru…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm/mglru: walk_pmd_range() marks PUD young on non-leaf PMD accessed bit
- [Medium] mm/mglru: walk_pmd_range() marks PUD young for THP before memcg and notifier checks
- [Medium] mm/mglru: PUDs with PMD-mapped THPs are suppressed indefinitely once skipped
--
commit 31572808d36982038b4b945e98794001c99a5d21
Author: Baoquan He <[email protected]>
mm/mglru: skip cold PUD subtrees during aging
This commit introduces a PUD-level Bloom filter keyed by (mm, PUD-aligned address) to skip PUD subtrees that had no young pages in the previous generation during MGLRU aging walks, and updates it from the rmap feedback path in lru_gen_look_around().
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 14 lines
mm/vmscan.c +2
-1@@ -3730,10 +3732,11 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area *first =-1 ; } -static void walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end, +static bool walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end,struct mm_walk *args) {int i; +bool young = false;pmd_t *pmd;unsigned long next;unsigned long addr;
mm/vmscan.c static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, · line 3781
Quoted 10 lines
mm/vmscan.c +3
-1@@ -3770,8 +3773,10 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_LEAF_TOTAL ]++; -if (pfn !=-1 ) +if (pfn !=-1 ) {walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; + }continue ; }
[Severity: Medium]
Does setting young = true as soon as get_pmd_pfn() != -1 mark the PUD young before checking memcg ownership or MMU notifiers?
get_pmd_pfn() only checks that the PFN is on pgdat and that pmd_young() or mm_has_notifiers() is true, while the memcg check in get_pfn_folio() and the notifier young check in pmdp_test_and_clear_young_notify() are deferred to walk_pmd_range_locked(). A PUD with PMD-mapped THPs belonging to another memcg on the same node, or cold THPs in an mm with MMU notifiers, will therefore be recorded in BLOOM_FILTER_PUD on every generation and never skipped.
Quoted 9 lines
mm/vmscan.c +1
-0@@ -3781,6 +3786,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ;walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; } if (!walk->force_scan &&
mm/vmscan.c static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, · line 3805
Quoted 7 lines
mm/vmscan.c +1
-0@@ -3793,6 +3799,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ; walk->mm_stats[MM_NONLEAF_ADDED ]++; + young = true;/* carry over to the next generation */ update_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 , pmd + i);
[Severity: Medium]
Should young = true be omitted from the non-leaf pmd_young(val) block?
When should_clear_pmd_young() is true, setting young = true there happens before checking BLOOM_FILTER_PMD or calling walk_pte_range(). Because CPU page table walks set the accessed bit on non-leaf PMDs regardless of which node or memcg owns the underlying PTEs, any accessed non-leaf PMD will cause walk_pmd_range() to return true and carry the PUD into BLOOM_FILTER_PUD even when every PMD in the subtree fails the PMD Bloom filter or has no folios in walk->lruvec. The young = true after walk_pte_range() already handles non-leaf PMDs that pass suitable_to_scan().
Quoted 35 lines
mm/vmscan.c +10
-1@@ -3802,6 +3809,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,if (i <PTRS_PER_PMD &&get_next_vma (PUD_MASK ,PMD_SIZE , args, &start, &end)) goto restart; + +return young; } static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,@@ -3812,6 +3821,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,unsigned long addr;unsigned long next;struct lru_gen_mm_walk *walk = args->private; +struct lru_gen_mm_state *mm_state =get_mm_state (walk->lruvec);VM_WARN_ON_ONCE (p4d_leaf (*p4d));@@ -3819,13 +3829,29 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, restart:for (i =pud_index (start), addr = start; addr != end; i++, addr = next) {pud_t val =pudp_get (pud + i); +void *pud_key; next =pud_addr_end (addr, end);if (!pud_present (val) ||WARN_ON_ONCE (pud_leaf (val)))continue ; -walk_pmd_range (pud + i, addr, next, args); +/* +* Use (mm, PUD-aligned address) as the filter key instead of +* the page table pointer. It is stable across the aging walk +* and the rmap feedback path, and avoids walking the page +* tables in lru_gen_look_around(). +*/
mm/vmscan.c static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, · line 3854
Quoted 10 lines
mm/vmscan.c +10
-0+ pud_key = (void *)((unsigned long )args->mm ^ (addr &PUD_MASK )); + +/* Skip a subtree that had nothing young for this lruvec last gen */ +if (!walk->force_scan && + !test_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq, pud_key)) +continue ; + +if (walk_pmd_range (pud + i, addr, next, args)) +update_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq +1 , + pud_key);
[Severity: Medium]
Once a PUD is absent from BLOOM_FILTER_PUD, can PMD-mapped THPs inside it ever be scanned by page-table aging again without force_scan?
Previously, BLOOM_FILTER_PMD was checked after pmd_trans_huge() in walk_pmd_range(), so PMD-mapped THPs were never gated by a Bloom filter. Gating walk_pmd_range() on BLOOM_FILTER_PUD skips the entire PUD when pud_key is not present, and iterate_mm_list() clears the next generation's filter via reset_bloom_filter(mm_state, BLOOM_FILTER_PUD, walk->seq + 1). The only rmap feedback that populates BLOOM_FILTER_PUD is lru_gen_look_around(), which folio_referenced_one() only calls when pvmw.pte is non-NULL:
mm/rmap.c:folio_referenced_one() {
...
if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte &&
pte_present(pteval)) {
if (lru_gen_look_around(&pvmw, nr))
referenced++;
} else if (pvmw.pte) {
...
} else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
if (pmdp_clear_flush_young_notify(vma, address,
pvmw.pmd))
referenced++;
...
}For PMD-mapped THPs (pvmw.pte == NULL), rmap eviction checks never update BLOOM_FILTER_PUD, so any PUD that goes cold for one generation or receives newly faulted or collapsed THPs after the initial force_scan will be skipped by aging indefinitely.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Baoquan He
reviewerRe: [PATCH v3 3/4] mm/mglru: skip cold PUD subtrees during aging
Quoted 8 lines · On 11/14/26 at 03:42am, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm/mglru:walk_pmd_range()marks PUD young on non-leaf PMD accessed bit
- [Medium] mm/mglru:walk_pmd_range()marks PUD young for THP before memcg and notifier checks
- [Medium] mm/mglru: PUDs with PMD-mapped THPs are suppressed indefinitely once skipped
Thanks for the review. All three points are real. The third one is the most important. Will Changes in v4. Please see the inline reply for each item.
Quoted 48 lines
--
commit
31572808d36982038b4b945e98794001c99a5d21
Author: Baoquan He <[email protected]>mm/mglru: skip cold PUD subtrees during aging
This commit introduces a PUD-level Bloom filter keyed by (mm, PUD-aligned address) to skip PUD subtrees that had no young pages in the previous generation during MGLRU aging walks, and updates it from the rmap feedback path in
lru_gen_look_around().Quoted 4 lines
mm/vmscan.c +0
-0[ ... ]
Quoted 14 lines
mm/vmscan.c +2
-1@@ -3730,10 +3732,11 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area *first =-1 ; } -static void walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end, +static bool walk_pmd_range (pud_t *pud,unsigned long start,unsigned long end,struct mm_walk *args) {int i; +bool young = false;pmd_t *pmd;unsigned long next;unsigned long addr;mm/vmscan.c static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, · line 3781
Quoted 10 lines
mm/vmscan.c +3
-1@@ -3770,8 +3773,10 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_LEAF_TOTAL ]++; -if (pfn !=-1 ) +if (pfn !=-1 ) {walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; + }continue ; }[Severity: Medium]
Does setting young = true as soon asget_pmd_pfn()!= -1 mark the PUD young before checking memcg ownership or MMU notifiers?
Quoted 7 lines
get_pmd_pfn()only checks that the PFN is on pgdat and thatpmd_young()ormm_has_notifiers()is true, while the memcg check inget_pfn_folio()and the notifier young check inpmdp_test_and_clear_young_notify()are deferred towalk_pmd_range_locked(). A PUD with PMD-mapped THPs belonging to another memcg on the same node, or cold THPs in an mm with MMU notifiers, will therefore be recorded inBLOOM_FILTER_PUDon every generation and never skipped.
Yes, that is too early. get_pmd_pfn() only checks that the PFN is on this node and that the accessed bit is set. The memcg check and the notifier young check happen later, inside walk_pmd_range_locked().
Will fix in v4 by making walk_pmd_range_locked() return true only when it found a young entry for this lruvec, and using that value here:
Quoted 21 lines
Quoted 10 lines
mm/vmscan.c +1
-0@@ -3781,6 +3786,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ;walk_pmd_range_locked (pud, addr, vma, args, bitmap, &first); + young = true; } if (!walk->force_scan &&mm/vmscan.c static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, · line 3805
Quoted 7 lines
mm/vmscan.c +1
-0@@ -3793,6 +3799,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,continue ; walk->mm_stats[MM_NONLEAF_ADDED ]++; + young = true;/* carry over to the next generation */ update_bloom_filter (mm_state,BLOOM_FILTER_PMD , walk->seq +1 , pmd + i);[Severity: Medium]
Should young = true be omitted from the non-leafpmd_young(val)block?
Quoted 8 lines
When
should_clear_pmd_young()is true, setting young = true there happens before checkingBLOOM_FILTER_PMDor callingwalk_pte_range(). Because CPU page table walks set the accessed bit on non-leaf PMDs regardless of which node or memcg owns the underlying PTEs, any accessed non-leaf PMD will causewalk_pmd_range()to return true and carry the PUD intoBLOOM_FILTER_PUDeven when every PMD in the subtree fails the PMD Bloom filter or has no folios inwalk->lruvec. The young = true afterwalk_pte_range()already handles non-leaf PMDs that passsuitable_to_scan().
Agreed, will remove. The accessed bit of a non-leaf PMD says nothing about which lruvec owns the pages, and the first walker of a generation clears it. A second walker (another node, or kswapd) then sees it cleared and skips the whole PUD, so the skip depends on walk order instead of on the lruvec.
In v4 a PUD is marked young only when walk_pte_range() reports young leaves for this lruvec, or when a young huge page of this lruvec is found. The batched flush below contributes too, because batched huge pages are processed there:
young |= walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first);
Quoted 42 lines
Quoted 42 lines
mm/vmscan.c +16
-1@@ -3802,6 +3809,8 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,if (i <PTRS_PER_PMD &&get_next_vma (PUD_MASK ,PMD_SIZE , args, &start, &end)) goto restart; + +return young; } static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,@@ -3812,6 +3821,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,unsigned long addr;unsigned long next;struct lru_gen_mm_walk *walk = args->private; +struct lru_gen_mm_state *mm_state =get_mm_state (walk->lruvec);VM_WARN_ON_ONCE (p4d_leaf (*p4d));@@ -3819,13 +3829,29 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, restart:for (i =pud_index (start), addr = start; addr != end; i++, addr = next) {pud_t val =pudp_get (pud + i); +void *pud_key; next =pud_addr_end (addr, end);if (!pud_present (val) ||WARN_ON_ONCE (pud_leaf (val)))continue ; -walk_pmd_range (pud + i, addr, next, args); +/* +* Use (mm, PUD-aligned address) as the filter key instead of +* the page table pointer. It is stable across the aging walk +* and the rmap feedback path, and avoids walking the page +* tables in lru_gen_look_around(). +*/ + pud_key = (void *)((unsigned long )args->mm ^ (addr &PUD_MASK )); + +/* Skip a subtree that had nothing young for this lruvec last gen */ +if (!walk->force_scan && + !test_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq, pud_key)) +continue ;
Quoted 8 lines
mm/vmscan.c static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, · line 3862
Quoted 4 lines
mm/vmscan.c +4
-0+ +if (walk_pmd_range (pud + i, addr, next, args)) +update_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq +1 , + pud_key);[Severity: Medium]
Once a PUD is absent fromBLOOM_FILTER_PUD, can PMD-mapped THPs inside it ever be scanned by page-table aging again withoutforce_scan?
Agreed, and this is the important one. For a PMD-mapped folio folio_referenced_one() takes the pmdp_clear_flush_young_notify() branch and never calls lru_gen_look_around(), so once a PUD is skipped nothing puts its key back, until a forced scan or a new mm.
v4 will add the missing feedback.
Quoted 32 lines
Previously,
BLOOM_FILTER_PMDwas checked afterpmd_trans_huge()inwalk_pmd_range(), so PMD-mapped THPs were never gated by a Bloom filter. Gatingwalk_pmd_range()onBLOOM_FILTER_PUDskips the entire PUD whenpud_keyis not present, anditerate_mm_list()clears the next generation's filter viareset_bloom_filter(mm_state, BLOOM_FILTER_PUD, walk->seq + 1). The only rmap feedback that populatesBLOOM_FILTER_PUDislru_gen_look_around(), whichfolio_referenced_one()only calls when pvmw.pte is non-NULL:mm/rmap.c:folio_referenced_one() { ... if (lru_gen_enabled() && !lru_gen_switching() && pvmw.pte && pte_present(pteval)) { if (lru_gen_look_around(&pvmw, nr)) referenced++; } else if (pvmw.pte) { ... } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_flush_young_notify(vma, address, pvmw.pmd)) referenced++; ... }For PMD-mapped THPs (pvmw.pte == NULL), rmap eviction checks never update
BLOOM_FILTER_PUD, so any PUD that goes cold for one generation or receives newly faulted or collapsed THPs after the initialforce_scanwill be skipped by aging indefinitely.Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Baoquan He
reviewerPATCH v3 4/4 mm/mglru: count PUD subtrees skipped by the PUD-level filter
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Add MM_PUD_EMPTY_SKIPPED, incremented in walk_pud_range() whenever a PUD subtree is skipped by the filter. Like other counters accumulated per walk, shown in debugfs lru_gen output (the "S" column of TYFALWEES) and via the mm_vmscan_lru_gen_walk tracepoint, so the avoided cost by the filter can be measured directly.
--- include/linux/mmzone.h | 1 + mm/trace_vmscan.h | 10 ++++++---- mm/vmscan.c | 13 ++++++++----- 3 files changed, 15 insertions(+), 9 deletions(-)
include/linux/mmzone.h +1 -0
@@ -583,6 +583,7 @@ enum {MM_WALK_TOTAL ,/* page-table walks completed */ MM_WALK_EMPTY ,/* walks that found no folio in this lruvec */ MM_LEAF_EMPTY_WALKS ,/* leaf entries scanned during empty walks */ +MM_PUD_EMPTY_SKIPPED ,/* PUD subtrees skipped by the PUD-level filter */ NR_MM_STATS };
mm/trace_vmscan.h +6 -4
@@ -766,9 +766,9 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, TRACE_EVENT(mm_vmscan_lru_gen_walk, TP_PROTO(int nid, unsigned long seq, int leaf_total, - int leaf_associated, bool empty), + int leaf_associated, bool empty, int pud_skipped), - TP_ARGS(nid, seq, leaf_total, leaf_associated, empty), + TP_ARGS(nid, seq, leaf_total, leaf_associated, empty, pud_skipped), TP_STRUCT__entry( __field(int, nid)
@@ -776,6 +776,7 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk, __field(int, leaf_total) __field(int, leaf_associated) __field(bool, empty) +__field (int, pud_skipped) ), TP_fast_assign(
@@ -784,11 +785,12 @@ TRACE_EVENT(mm_vmscan_lru_gen_walk, __entry->leaf_total = leaf_total; __entry->leaf_associated = leaf_associated; __entry->empty = empty; + __entry->pud_skipped = pud_skipped; ), -TP_printk ("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d" , +TP_printk ("nid=%d seq=%lu leaf_total=%d leaf_associated=%d empty=%d pud_skipped=%d" , __entry->nid, __entry->seq, __entry->leaf_total, - __entry->leaf_associated, __entry->empty) + __entry->leaf_associated, __entry->empty, __entry->pud_skipped) );#endif /* _TRACE_VMSCAN_H */
mm/vmscan.c +8 -5
@@ -3854,8 +3854,10 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end,/* Skip a subtree that had nothing young for this lruvec last gen */ if (!walk->force_scan && - !test_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq, pud_key)) + !test_bloom_filter (mm_state,BLOOM_FILTER_PUD , walk->seq, pud_key)) { + walk->mm_stats[MM_PUD_EMPTY_SKIPPED ]++;continue ; + }if (walk_pmd_range (pud + i, addr, next, args)) update_bloom_filter(mm_state, BLOOM_FILTER_PUD, walk->seq + 1,
@@ -4222,7 +4224,8 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq,trace_mm_vmscan_lru_gen_walk (lruvec_pgdat (lruvec)->node_id, walk->seq, walk->mm_stats[MM_LEAF_TOTAL ], - walk->mm_stats[MM_LEAF_ASSOCIATED ], empty); + walk->mm_stats[MM_LEAF_ASSOCIATED ], empty, + walk->mm_stats[MM_PUD_EMPTY_SKIPPED ]); } }while (mm); done:
@@ -5714,14 +5717,14 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,seq_puts (m," " );for (i =0 ; i <NR_MM_STATS ; i++) { -const char *s ="xxxxxxxx" ; +const char *s ="xxxxxxxxx" ;unsigned long n =0 ;if (seq == max_seq &&NR_HIST_GENS ==1 ) { - s ="TYFALWEE" ; + s ="TYFALWEES" ; n =READ_ONCE (mm_state->stats[hist][i]); }else if (seq != max_seq &&NR_HIST_GENS >1 ) { - s ="tyfalwee" ; + s ="tyfalwees" ; n =READ_ONCE (mm_state->stats[hist][i]); }
Signature · 1 line
2.52.0