Skip to content

RFC mm/mglru: skip empty PUD subtrees during aging with PUD-level Bloom filter

Baoquan He linux-mm 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest5 of 6 patches 7 repliesIn review

5 of 6 patches are in this archive. The cover letter announces 6.

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6mm/mglru: add PUD-level Bloom filter stateinclude/linux/mmzone.h+9 -12 replies
2/6mm/mglru: refactor Bloom filter helpers for two filter levelsmm/vmscan.c+24 -90 replies
3/6mm/mglru: skip empty PUD subtrees during agingmm/vmscan.c+47 -32 replies
4/6mm/mglru: report hot PUDs from the rmap feedback pathmm/vmscan.c+7 -12 replies
5/6mm/mglru: add MM_WALK_EMPTY stats and tracepoint for cross-node measurementinclude/linux/mmzone.h include/trace/events/vmscan.h mm/vmscan.c+60 -40 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/6] mm/mglru: skip empty PUD subtrees during aging with PUD-level Bloom filter

Problem
=======

MGLRU's aging walks every present PUD of every mm in the mm_list. When an mm has no pages on a given NUMA node, all 512 PMDs in each PUD fail the existing PMD-level Bloom filter test, yet the walker still descends every PUD and iterates every PMD — pure waste.

These cross-node empty walks are structural: lru_gen_use_mm() sets mm->lru_gen.bitmap to -1 (all nodes) at each context switch, so every node's kswapd independently walks the same mm. On a 2-node KVM guest, ~80% of all aging walks are empty; the worst-case leaf-traversal waste with a fully resident remote mm is ~73%.

Approach
========

Add a PUD-level Bloom filter (one level above the existing PMD filter). The walker now:

1. Checks the PUD filter before descending into a PUD subtree.
2. If the filter says the PUD had no young entries last generation, skips the entire 1GB region — avoiding 512 PMD lookups.
3. walk_pmd_range() reports upward whether it found any young leaf entries; walk_pud_range() records that in the double-buffered PUD filter.
4. The rmap feedback path (lru_gen_look_around()) marks hot PUDs in the filter, so newly hot or migrated-in regions are re-examined promptly rather than suppressed indefinitely.
5. force_scan walks bypass the PUD test, so manual aging and newly added mm's always populate the filter.

The double-buffered filter flips each aging generation, and the eviction feedback keeps hot regions marked, so correctness holds naturally without tracking per-page residency or requiring changes to page fault / rmap hot paths.

The testing results are from a 2-node KVM guest. Real-hardware results on a ≥4-socket system, would be valuable. I will update the statistics here if have.

Baoquan He (6):
mm/mglru: add PUD-level Bloom filter state
mm/mglru: refactor Bloom filter helpers for two filter levels mm/mglru: skip empty PUD subtrees during aging mm/mglru: report hot PUDs from the rmap feedback path mm/mglru: add MM_WALK_EMPTY stats and tracepoint for cross-node
measurement
mm/mglru: count PUD subtrees skipped by the PUD-level filter

 include/linux/mmzone.h        |  15 +++-
 include/trace/events/vmscan.h |  30 ++++++++
 mm/vmscan.c                   | 126 +++++++++++++++++++++++++++++-----
 3 files changed, 153 insertions(+), 18 deletions(-)

--
2.54.0