RFC mm/mglru: skip empty PUD subtrees during aging with PUD-level Bloom filter
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest5 of 6 patches 7 repliesIn review
5 of 6 patches are in this archive. The cover letter announces 6.
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/6 | mm/mglru: add PUD-level Bloom filter stateinclude/ | +9 | 2 replies | |
| 2/6 | mm/mglru: refactor Bloom filter helpers for two filter levelsmm/ | +24 | 0 replies | |
| 3/6 | mm/mglru: skip empty PUD subtrees during agingmm/ | +47 | 2 replies | |
| 4/6 | mm/mglru: report hot PUDs from the rmap feedback pathmm/ | +7 | 2 replies | |
| 5/6 | mm/mglru: add MM_WALK_EMPTY stats and tracepoint for cross-node measurementinclude/ include/ mm/ | +60 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/6] mm/mglru: skip empty PUD subtrees during aging with PUD-level Bloom filter
Problem
=======
MGLRU's aging walks every present PUD of every mm in the mm_list. When an mm has no pages on a given NUMA node, all 512 PMDs in each PUD fail the existing PMD-level Bloom filter test, yet the walker still descends every PUD and iterates every PMD — pure waste.
These cross-node empty walks are structural: lru_gen_use_mm() sets mm->lru_gen.bitmap to -1 (all nodes) at each context switch, so every node's kswapd independently walks the same mm. On a 2-node KVM guest, ~80% of all aging walks are empty; the worst-case leaf-traversal waste with a fully resident remote mm is ~73%.
Approach
========
Add a PUD-level Bloom filter (one level above the existing PMD filter). The walker now:
1. Checks the PUD filter before descending into a PUD subtree.
2. If the filter says the PUD had no young entries last generation, skips the entire 1GB region — avoiding 512 PMD lookups.
3. walk_pmd_range() reports upward whether it found any young leaf entries; walk_pud_range() records that in the double-buffered PUD filter.
4. The rmap feedback path (lru_gen_look_around()) marks hot PUDs in the filter, so newly hot or migrated-in regions are re-examined promptly rather than suppressed indefinitely.
5. force_scan walks bypass the PUD test, so manual aging and newly added mm's always populate the filter.
The double-buffered filter flips each aging generation, and the eviction feedback keeps hot regions marked, so correctness holds naturally without tracking per-page residency or requiring changes to page fault / rmap hot paths.
The testing results are from a 2-node KVM guest. Real-hardware results on a ≥4-socket system, would be valuable. I will update the statistics here if have.
Baoquan He (6):
mm/mglru: add PUD-level Bloom filter state
mm/mglru: refactor Bloom filter helpers for two filter levels mm/mglru: skip empty PUD subtrees during aging mm/mglru: report hot PUDs from the rmap feedback path mm/mglru: add MM_WALK_EMPTY stats and tracepoint for cross-node
measurement
mm/mglru: count PUD subtrees skipped by the PUD-level filter
include/linux/mmzone.h | 15 +++- include/trace/events/vmscan.h | 30 ++++++++ mm/vmscan.c | 126 +++++++++++++++++++++++++++++----- 3 files changed, 153 insertions(+), 18 deletions(-)
--
2.54.0