Skip to content

khugepaged: add mTHP collapse support

Nico Pache linux-doc linux-mm linux-trace-kernel lkml 19 versions – Applied

Status from this message: [merged mm-stable] documentation-mm-update-the-admin-guide-for-mthp-collapse.patch removed from -mm tree

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v19 Latest14 patches 34 repliesApplied 14/14
  2. v1814 patches 130 repliesSuperseded
  3. v1714 patches 29 repliesSuperseded
Earlier versions · 16
VersionPostedPatchesRepliesStatus
v1613 patches31 repliesSuperseded
v1513 patches41 repliesSuperseded
v1416 patches22 repliesSuperseded
v1316 patches25 repliesSuperseded
v1215 patches75 repliesSuperseded
v1115 patches63 repliesSuperseded
v1013 patches75 repliesSuperseded
v914 patches36 repliesSuperseded
v815 patches14 repliesSuperseded
v712 patches43 repliesSuperseded
v612 patches3 repliesSuperseded
v512 patches28 repliesSuperseded
v412 patches21 repliesSuperseded
v312 patches3 repliesSuperseded
v2 RFC9 patches45 repliesSuperseded
v1 RFC11 patches41 repliesSuperseded

Changes in v19 from the cover letter

- Rebased onto mm-unstable (base: 9282f9bdbdf2, where v18 was based)
- Added RBs/Acks
- [patch 04] Make max_ptes_none const; guard pr_warn_once with check so we only warn for non-zero intermediate values and use a single "return 0" path (David)
- [patch 06] Add comment explaining the mmu_notifier_range covers only the modified PTE range (David); change BUG_ON to VM_WARN_ON_ONCE (David); fix update_mmu_cache_range() arch safety issue: reinstall PMD via pmd_populate() before calling map_anon_folio_pte_nopf() and hold the PTE lock nested inside pmd_ptl during the operation (David, Lance); drop the now-unnecessary smp_wmb() as __folio_mark_uptodate() provides the required ordering (David, Lance)
- [patch 07] Clarify commit message: "mTHP to a folio of equal or smaller size, possibly resulting in a partially mapped source folio" (David)
- [patch 08] Add Lorenzo's RB and David's Ack; move TODO comment about shared pages to patch 4 (David)
- [patch 10] Rename collapse_allowable_orders() to collapse_possible_orders() and add collapse_possible() boolean wrapper for callers that only need a yes/no answer (David)
- [patch 11] Major rework: replace the DFS stack-based algorithm with a simpler linear forward-walking approach using offset + order (David); remove mthp_range struct, mthp_bitmap_stack[], stack push/pop functions, and MTHP_STACK_SIZE; add max_order_from_offset() helper using __ffs(offset) for natural alignment; rename mthp_bitmap to mthp_present_ptes (David); remove temporary mthp_bitmap_mask and use bitmap_weight_from() directly (David); fix result propagation so mthp_collapse() returns enum scan_result and properly propagates SCAN_ALLOC_HUGE_PAGE_FAIL and SCAN_PTE_MAPPED_HUGEPAGE to callers (Lance); fold in v18 fixup for potential use-after-free of vma in mthp_collapse() by passing NULL to collapse_max_ptes_none()
- [patch 12] Rework to match new linear algorithm; expand the set of failures that allow retrying at a lower order
- [patch 14] Re-add note about file/shmem still only collapsing to PMD-sized THPs (David)

Patches in v19 All 14 applied

#Subject and filesSizeRepliesApplied
1/14mm/khugepaged: generalize hugepage_vma_revalidate for mTHP supportmm/khugepaged.c+18 -80 repliesApplied
2/14mm/khugepaged: generalize alloc_charge_folio()Documentation/admin-guide/mm/transhuge.rst include/linux/huge_mm.h mm/huge_memory.c +1 more+27 -70 repliesApplied
3/14mm/khugepaged: rework max_ptes_* handling with helper functionsmm/khugepaged.c+84 -360 repliesApplied
4/14mm/khugepaged: generalize __collapse_huge_page_* for mTHP supportmm/khugepaged.c+93 -331 replyApplied
5/14mm/khugepaged: require collapse_huge_page to enter/exit with the lock droppedmm/khugepaged.c+8 -82 repliesApplied
6/14mm/khugepaged: generalize collapse_huge_page for mTHP collapsemm/khugepaged.c+67 -384 repliesApplied
7/14mm/khugepaged: skip collapsing mTHP to smaller ordersmm/khugepaged.c+8 -01 replyApplied
8/14mm/khugepaged: add per-order mTHP collapse failure statisticsDocumentation/admin-guide/mm/transhuge.rst include/linux/huge_mm.h mm/huge_memory.c +1 more+37 -21 replyApplied
9/14mm/khugepaged: improve tracepoints for mTHP ordersinclude/trace/events/huge_memory.h mm/khugepaged.c+27 -161 replyApplied
10/14mm/khugepaged: introduce collapse_possible_orders helper functionsmm/khugepaged.c+21 -32 repliesApplied
11/14mm/khugepaged: Introduce mTHP collapse supportmm/khugepaged.c+138 -813 repliesApplied
12/14mm/khugepaged: avoid unnecessary mTHP collapse attemptsmm/khugepaged.c+13 -03 repliesApplied
13/14mm/khugepaged: run khugepaged for all ordersmm/khugepaged.c+20 -160 repliesApplied
14/14Documentation: mm: update the admin guide for mTHP collapseDocumentation/admin-guide/mm/transhuge.rst+32 -173 repliesApplied

Reviews trailers per patch

All 14 patches have a Reviewed-by or Acked-by.

PatchWei YangLance YangBaolin WangLorenzo StoakesZi YanUsama ArifDavid Hildenbrand (Arm)Bagas Sanjaya
1/14R-b R-b R-b R-b R-b A-b A-b
2/14R-b R-b R-b R-b R-b A-b A-b
3/14R-b R-b R-b A-b A-b
4/14R-b R-b R-b A-b
5/14R-b R-b R-b A-b
6/14R-b R-b R-b A-b
7/14R-b R-b R-b A-b A-b
8/14R-b R-b A-b
9/14R-b R-b R-b A-b
10/14R-b R-b R-b A-b
11/14R-b A-b
12/14R-b R-b A-b A-b
13/14R-b R-b A-b A-b
14/14R-b R-b A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 8 reviewers.

Read the v19 thread

Cover letter [PATCH mm-unstable v19 00/14] khugepaged: add mTHP collapse support

The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.

To achieve this we generalize the khugepaged functions to no longer depend on PMD_ORDER. Then during the PMD scan, we use a bitmap to track individual pages that are occupied (!none/zero). After the PMD scan is done, we use the bitmap to find the optimal mTHP sizes for the PMD range. The restriction on max_ptes_none is removed during the scan, to make sure we account for the whole PMD range in the bitmap. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained.

We currently only support max_ptes_none values of 0 or HPAGE_PMD_NR - 1 (ie 511). If any other value is specified, the kernel will emit a warning and mTHP collapse will default to max_ptes_none=0. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse.
It is now also possible to collapse to mTHPs without requiring the PMD THP size to be enabled. These limitations are to prevent collapse "creep" behavior. This prevents constantly promoting mTHPs to the next available size, which would occur because a collapse introduces more non-zero pages that would satisfy the promotion condition on subsequent scans.

Patch 1-2: Generalize hugepage_vma_revalidate and alloc_charge_folio
for arbitrary orders.

Patch 3:     Rework max_ptes_* handling into helper functions
Patch 4:     Generalize __collapse_huge_page_* for mTHP support
Patch 5:     Require collapse_huge_page to enter/exit with the lock dropped
Patch 6:     Generalize collapse_huge_page for mTHP collapse
Patch 7:     Skip collapsing mTHP to smaller orders
Patch 8-9:   Add per-order mTHP statistics and tracepoints
Patch 10:    Introduce collapse_possible_orders helper functions
Patch 11-13: Introduce bitmap and mTHP collapse support, fully enabled
Patch 14:    Documentation

Testing:
- Built for x86_64, aarch64, ppc64le, and s390x
- ran all arches on test suites provided by the kernel-tests project
- internal testing suites: functional testing and performance testing
- selftests mm
- I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit)
The summary from my testings was that there was no significant regression noticed through this test. In some cases my changes had better collapse latencies, and was able to scan more pages in the same amount of time/work, but for the most part the results were consistent.
- redis testing. I did some testing with these changes along with my defer changes (see followup [2] post for more details). We've decided to get the mTHP changes merged first before attempting the defer series.
- some basic testing on 64k page size.
- lots of general use.

[1] - https://gitlab.com/npache/khugepaged_mthp_test
[2] - https://lore.kernel.org/lkml/[email protected]/

V19 Changes:
- Rebased onto mm-unstable (base: 9282f9bdbdf2, where v18 was based)
- Added RBs/Acks
- [patch 04] Make max_ptes_none const; guard pr_warn_once with check so we only warn for non-zero intermediate values and use a single "return 0" path (David)
- [patch 06] Add comment explaining the mmu_notifier_range covers only the modified PTE range (David); change BUG_ON to VM_WARN_ON_ONCE (David); fix update_mmu_cache_range() arch safety issue: reinstall PMD via pmd_populate() before calling map_anon_folio_pte_nopf() and hold the PTE lock nested inside pmd_ptl during the operation (David, Lance); drop the now-unnecessary smp_wmb() as __folio_mark_uptodate() provides the required ordering (David, Lance)
- [patch 07] Clarify commit message: "mTHP to a folio of equal or smaller size, possibly resulting in a partially mapped source folio" (David)
- [patch 08] Add Lorenzo's RB and David's Ack; move TODO comment about shared pages to patch 4 (David)
- [patch 10] Rename collapse_allowable_orders() to collapse_possible_orders() and add collapse_possible() boolean wrapper for callers that only need a yes/no answer (David)
- [patch 11] Major rework: replace the DFS stack-based algorithm with a simpler linear forward-walking approach using offset + order (David); remove mthp_range struct, mthp_bitmap_stack[], stack push/pop functions, and MTHP_STACK_SIZE; add max_order_from_offset() helper using __ffs(offset) for natural alignment; rename mthp_bitmap to mthp_present_ptes (David); remove temporary mthp_bitmap_mask and use bitmap_weight_from() directly (David); fix result propagation so mthp_collapse() returns enum scan_result and properly propagates SCAN_ALLOC_HUGE_PAGE_FAIL and SCAN_PTE_MAPPED_HUGEPAGE to callers (Lance); fold in v18 fixup for potential use-after-free of vma in mthp_collapse() by passing NULL to collapse_max_ptes_none()
- [patch 12] Rework to match new linear algorithm; expand the set of failures that allow retrying at a lower order
- [patch 14] Re-add note about file/shmem still only collapsing to PMD-sized THPs (David)

Baolin Wang (1):
mm/khugepaged: run khugepaged for all orders

Dev Jain (1):
mm/khugepaged: generalize alloc_charge_folio()

Nico Pache (12):
mm/khugepaged: generalize hugepage_vma_revalidate for mTHP support mm/khugepaged: rework max_ptes_* handling with helper functions mm/khugepaged: generalize __collapse_huge_page_* for mTHP support mm/khugepaged: require collapse_huge_page to enter/exit with the lock
dropped
mm/khugepaged: generalize collapse_huge_page for mTHP collapse mm/khugepaged: skip collapsing mTHP to smaller orders mm/khugepaged: add per-order mTHP collapse failure statistics mm/khugepaged: improve tracepoints for mTHP orders mm/khugepaged: introduce collapse_possible_orders helper functions mm/khugepaged: Introduce mTHP collapse support mm/khugepaged: avoid unnecessary mTHP collapse attempts
Documentation: mm: update the admin guide for mTHP collapse

 Documentation/admin-guide/mm/transhuge.rst |  71 ++-
 include/linux/huge_mm.h                    |   5 +
 include/trace/events/huge_memory.h         |  34 +-
 mm/huge_memory.c                           |  11 +
 mm/khugepaged.c                            | 616 ++++++++++++++++-----
 5 files changed, 569 insertions(+), 168 deletions(-)

base-commit: 9282f9bdbdf22a1914f0b085f2fe5e8141cbc3de

Signature · 1 line
2.54.0