mm: PMD-level swap entries for anonymous THPs
Subsystems (24)
Usama Arif
reviewerv8 cover mm: PMD-level swap entries for anonymous THPs
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into HPAGE_PMD_NR PTE-level swap entries via TTU_SPLIT_HUGE_PMD before unmap. This series introduces a PMD-level swap entry so the huge mapping can survive the swap round-trip and do_huge_pmd_swap_page() can restore the PMD mapping directly on swap-in, without waiting for khugepaged to collapse the range later.
The PMD swap entry is a compact page-table encoding for HPAGE_PMD_NR consecutive swap slots. swap_map accounting remains per-slot and is unchanged. Importantly, a PMD swap entry does not promise that the swap cache always contains one PMD-sized folio. While the cache is empty or contains one PMD-sized folio, PMD-level handling can proceed. Once the cache has split/per-slot state, users either inspect the individual slots directly (mincore) or split the PMD swap entry and retry through the PTE path (fault, swapoff, MADV_WILLNEED, UFFDIO_MOVE). MADV_FREE does not consult the cache at all: it frees a whole PMD swap entry in place and only splits when the advised range covers part of the PMD. Likewise, if any slot is still backed by zswap's per-page store, PMD-order swap-in consumers split and let the PTE path load the range page by page; an all-on-disk range can still be read back as one PMD-sized folio.
The series is ordered so every consumer can handle PMD swap entries before the swap-out producer starts installing them. The swap-out patch is the last functional change.
Performance:
Measured with vm-scalability's case-swap-w-seq benchmark [1]. Four pinned workers repeatedly write a 6 GiB anonymous working set on a 4 vCPU / 4 GiB guest, forcing the set out to swap and back in. Swap is an 8 GiB NOCOW raw virtio device (cache=none, aio=native), zswap is disabled, and THP enabled/defrag are both "always". The numbers below are medians of five interleaved runs per kernel after one warm-up run:
Metric Baseline Patched Change Aggregate throughput 584.3 MiB/s 2408.9 MiB/s +312.2% (4.12x) Elapsed time 85.43 s 20.72 s -75.7% Major faults 1,814,699 228,466 -87.4% Swap I/O rate 1.02 GiB/s 4.10 GiB/s +303.9%
This is a swap-intensive synthetic workload, so it mostly demonstrates the reduction in swap-fault and page-table overhead from preserving PMD mappings. I don't want to use sythetic workloads to show benefits of the series. IMHO, the main advantage comes from long-running workloads I expect the bigger win to come from fewer TLB misses, less khugepaged work, and less kernel churn from larger folios, although I have not found a benchmark that captures that well. PMD swap entries also move us closer to eliminating page-table deposits for anonymous THPs, which would provide memory savings.
Sashiko reviews on intermediate patches:
Because the swap-out producer is the last functional patch, the consumer code added by the patches before it is unreachable at the point it is introduced. Every previous sashiko review has reported that code as broken on the basis of a state it cannot yet be in; the series is ordered this way deliberately. See [2].
Notes on zswap:
Native PMD-order zswap load/store is intentionally left for a follow-up. Alexandre Ghiti is currently working on this.
This series can still preserve PMD swap entries while zswap is enabled:
zswap stores the THP as order-0 entries, and PMD-order swap-in consumers split any range that has zswap entries before reading it. If zswap has written the whole range back to disk, or the swap cache still contains one PMD-sized folio, PMD-level handling can proceed.
Testing:
The 17 pmd_swap selftests pass on x86_64 with zswap both disabled and enabled. PMD_SWAP_DEVICE was the sole active swap device, so the swapoff test ran in both configurations. Note that with zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so in the log; the zswap-disabled run is the one that proves PMD restoration.
hmm-tests was run with CONFIG_DEBUG_VM=y and panic_on_warn=1, both on this series and on the base commit: identical results either way (pass:35 fail:3 skip:40). The three failures are O_TMPFILE on the test VM's 9p /tmp, not kernel behaviour, and the THP paths (migrate_anon_huge_*, migrate_partial_unmap_fault, benchmark_thp_migration) pass.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
[2] https://lore.kernel.org/all/[email protected]/
v7 -> v8: https://lore.kernel.org/all/[email protected]/
- Cover letter: add the case-swap-w-seq performance results. (Andrew Morton)
- Patches 2-8: reword the opening paragraph as "Prepare for ..." instead of referring to a later patch. (David Hildenbrand)
- Patch 4 (powerpc): move the PMD exclusive helpers next to the soft-dirty PMD helpers and express them the same way, as pte_pmd()/pmd_pte() wrappers over the PTE helpers. (David Hildenbrand)
- Patch 6 (s390): drop the comment above the helpers, keep the explanation in the RSTE swap layout above __SWP_OFFSET_MASK_RSTE, and move the helpers into the existing CONFIG_ARCH_HAS_PMD_SOFTLEAVES block above pmd_swp_soft_dirty(). (David Hildenbrand)
- Patch 7 (x86): add static_assert(_PAGE_SWP_EXCLUSIVE != _PAGE_PSE) so a 32-bit build that ever selects ARCH_HAS_PMD_SOFTLEAVES fails to compile rather than producing pmd_present() swap entries. (Kiryl Shutsemau)
- Patch 10: rename the flag to to_migration_entries, use two-tab continuation indentation, reword the split comment, turn the comment on split_pmd_to_migration_entries() into kerneldoc, and fold the try_to_migrate_one() call onto one line. Also drop that helper's pmd_trans_huge() || pmd_is_valid_softleaf() test, which was both too loose and silently skipped the split, in favour of a VM_WARN_ON_ONCE() at the top of __split_huge_pmd_locked() asserting that to_migration_entries implies a present or device-private PMD. (David Hildenbrand)
- Patch 11: drop the VM_WARN_ON_ONCE()/force and its comment from the swap decode arm - 10/29 now asserts the to_migration_entries contract at the top of __split_huge_pmd_locked() instead - and build the replacement PTEs by advancing pte_next_swp_offset() rather than rebuilding each entry, which hoists the soft-dirty/uffd/exclusive tests out of the loop. (David Hildenbrand).
- Patch 12 (new): split the swap-side changes out of the fork patch, so the swap-entry range duplication gets its own patch for the swap maintainers. Keeps the single-slot names as inline wrappers so existing callers are untouched. (David Hildenbrand)
- Patch 13 (fork): report a failed swap dup as -EIO and let copy_pmd_range() own the GFP_KERNEL retry, as the PTE path does, dropping the open-coded retry loop; move the mm counter update into each entry-type arm. (David Hildenbrand)
- Patch 16 (smaps): smaps_account_swap() takes nr_pages rather than a byte size, and the local is called swapcount. (David Hildenbrand)
- No other functional change. All twelve sashiko findings on v7 were analysed and none are defects of this series.
v6 -> v7: https://lore.kernel.org/all/[email protected]/
- Rebase onto akpm/mm-new at baa8de2f3448 and adapt to the new get_swap_device() contract and linear_anon_page_index(). The series grows from 12 to 29 patches.
- Patch 1: code unchanged; reword the commit message and collect review tags.
- Patches 2-9: split the six architecture helpers from generic detection and add debug_vm_pgtable coverage. Use the s390 RSTE exclusive bit, make the x86 helper return bool, clear all PMD swap overlay bits before softleaf decoding, and add the architecture maintainers to Cc.
- Patches 10-11: add a preparatory no-functional-change cleanup of the migration-splitting API and keep the PMD swap split separate.
- Patch 12: reject a multi-slot duplication range that crosses a swap-cluster boundary.
- Patch 13: drop the zswap_load() change, now upstream as 1a904e0d3c43, and check for zswap after swap-cache insertion while all slots are pinned.
- Patch 14: scan every subpage for hardware poison, discard a failed or newly allocated unmapped PMD-sized folio before PTE fallback, and recheck the PMD before splitting it.
- Patches 15-23: split the non-present PMD walkers by subsystem. Add the guard-advice patch so MADV_GUARD_INSTALL/REMOVE leaves PMD swap entries whole; the other split patches preserve v6 behaviour.
- Patch 24: honour the current THP policy, discard a failed PMD-sized folio, and recheck the PMD before PTE fallback.
- Patch 25: do not split after cached-folio revalidation loses a race; retry so a restored present THP remains whole.
- Patches 26-27: separate the independent PTE-batching hardware-poison fix, scan subpages directly, and discard failed or never-mapped PMD-sized folios before PTE retry instead of making the whole range fail with SIGBUS.
- Patch 28: keep the normal swap-out path unchanged, but make failed producer preconditions warn and return -EBUSY rather than BUG or return -EINVAL.
- Patch 29: grow the selftests from 16 to 17, adding swapin_sync/cache-residency coverage and more robust feature, fallback, privilege, data-integrity, and swap-device-priority handling.
v5 -> v6: https://lore.kernel.org/all/[email protected]/
- Add patch 1 to rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio(). No functional change. (Dev Jain)
- Patch 2: warn when pmd_softleaf_to_folio() is given a non-PFN softleaf rather than silently returning NULL. (Dev Jain)
- Patch 4: bound the fork extend-table fallback to one retry, re-read the PMD under its lock, normalize unrecoverable copy_huge_pmd() errors to -ENOMEM so copy_pmd_range() cannot clear and leak the source swap PMD, and drop a redundant thp_migration_supported() gate.
- Patch 5: check multi-page swap-cache insertions for zswap-backed slots in __swap_cache_add_check() under the cluster lock, both before allocation and before insertion, and reject mixed zswap/disk state with -EBUSY. (Yosry Ahmed, Nhat Pham)
- Patch 6: on a failed non-uptodate PMD-order read, remove the large folio from swap cache before splitting so order-0
fallback retries individual slots rather than poisoning the whole
2 MiB range; retain hardware-poisoned folios for per-subpage handling.
- Patch 7: make HMM snapshot mode report a PMD swap entry as non-resident, matching PTE swap entries, rather than HMM_PFN_ERROR. Drop redundant thp_migration_supported() gates and simplify non-present PMD handling.
- Patch 8: factor PMD MADV_WILLNEED prefetch into swapin_pmd_swap_entry(), split and retry through PTEs after any PMD-order swapin failure, and replace the racy folio_test_locked() plus folio_lock() sequence with folio_trylock().
- Patch 9: guard PMD-swap UFFDIO_MOVE code with CONFIG_THP_SWAP, clarify RWP marker propagation, and reject a PMD swap entry at the destination with -EEXIST so UFFDIO_MOVE cannot loop forever on -EAGAIN.
- Patch 10: honor current THP/VMA policy before PMD-order swap-in, recheck that the PMD is still the original swap entry before splitting for PTE fallback, and provide the CONFIG_TRANSPARENT_HUGEPAGE wp_huge_pmd() declaration/stub needed by THP=n builds.
- Patch 11: make an invalid set_pmd_swap_entry() walk context warn and return -EINVAL instead of falsely reporting success and corrupting the MM_ANONPAGES/MM_SWAPENTS accounting, and add an exact PMD-size folio precondition check. (Luiz Capitulino)
- Patch 12: use /proc/swaps for prerequisite detection, check MADV_HUGEPAGE, and distinguish an environment that cannot allocate a PMD THP (SKIP) from a swap-out validation failure (FAIL). Add partial-mprotect and partial-munmap split coverage. Strengthen munmap/MADV_FREE VmSwap accounting, pagemap slot-offset checks, and mprotect/mremap swapped-state checks; force mremap to move, check munmap()'s return, and mark the UFFDIO_MOVE destination MADV_HUGEPAGE before asserting PMD restoration. Move common setup and cleanup into one fixture, merge the swapoff fixture, remove the redundant cycles test, and make the data pattern differ between base pages so the split tests can detect incorrect slot ordering. Order fork-COW so the parent writes while the child still holds the untouched shared swap entry. (Luiz Capitulino)
- Clarify commit messages throughout. Retain TTU_SPLIT_HUGE_PMD after prototyping its removal: removing it here requires an extra rmap walk and broadens the series beyond PMD swap entries. (Matthew Wilcox)
- Rebase onto akpm/mm-new from 15 August (4b65683fd25f).
v4 -> v5: https://lore.kernel.org/all/[email protected]/
- Commit message improvements for almost all patches (Yosry for zswap patch)
- Patch 1: make pmd_to_softleaf_folio() reject softleaf entries that do not encode a PFN, so a PMD swap offset is never interpreted as one. PMD swap entries remain valid softleaf entries for classification. (sashiko)
- Patch 2: use the existing pmd_swp_uffd() helper and force freeze=false for PMD swap entries, which have no struct page for the migration-entry freeze path. (sashiko)
- Patch 3: document that the caller's page-table or swap-cache reference pins every source slot while a partial PMD-sized duplication is rolled back. Keep the pre-existing PTE fork retry behavior outside this series. (sashiko)
- Patch 5: split to the PTE path rather than mapping a PMD-sized folio containing a hardware-poisoned subpage, and restore PAGE_NONE when swapoff restores a UFFD marker in an RWP VMA. (sashiko)
- Patch 6: account SwapPss for a PMD swap entry one slot at a time because the slots can have different swap reference counts.
- Patch 7: if PMD-order MADV_WILLNEED encounters newly populated per-page zswap state, revalidate and remove the failed clean PMD-sized cache folio before retrying through PTEs.
- Patch 8: mark a moved PMD swap entry for UFFD when the UFFDIO_MOVE destination VMA is RWP-registered. (sashiko)
- Patch 9: restore PAGE_NONE for UFFD RWP swap-in, preserve the original write-fault state through swap-slot release and COW handling, remove the unnecessary LRU drain, and prevent PTE batching from mapping a poisoned subpage. (sashiko)
- Patch 10: add and document thp_swpout_pmd, which counts PMD mappings replaced by PMD-level swap entries rather than swapped folios.
- Patch 11: register pmd_swap with the default mm selftest runner, preserve errno across UFFDIO_MOVE cleanup, check swapoff residency before the first memory access, add a parent-side write and verification to the fork+COW test, and add RWP regression coverage for swap-in, UFFDIO_MOVE, and swapoff. (sashiko)
- Keep do_huge_pmd_swap_page() in patch 9. Patches 6 and 7 only add consumers; patch 10 remains the first producer, so no PMD swap entry can reach those paths before the fault handler is present. (sashiko)
- Rebase onto latest akpm/mm-new from 22 July (5e0603ba185a)
v3 -> v4: https://lore.kernel.org/all/[email protected]/
- Patch 1: guard the new arch-specific pmd_swp_mkexclusive / pmd_swp_exclusive / pmd_swp_clear_exclusive helpers on arm64, loongarch, powerpc, riscv, s390, and x86 with CONFIG_ARCH_HAS_PMD_SOFTLEAVES, matching the pattern already
used for pmd_swp_soft_dirty. Also fixes the redefinition-vs- generic-fallback build errors kernel test robot reported on i386-allnoconfig-bpf and riscv-allnoconfig-bpf, and rewraps
the patch 1 commit message paragraphs to ~75 columns.
(sashiko, kernel test robot, Usama Arif)
- Patch 2: switch the trailing folio_remove_rmap_pmd() gate in __split_huge_pmd_locked() from *pmd to old_pmd, old_pmd retains
the original present-or-non-present classification for every
branch above. (sashiko)
- Patch 3: teach swap_retry_table_alloc() (and the underlying swap_extend_table_alloc()) to accept an nr parameter and scan
every slot in [ci_off, ci_off + nr) before committing an extend-table allocation. (sashiko)
- Patch 4: rename zswap_range_has_entry() to zswap_is_present() so
the same helper serves both single-slot (nr=1) and range queries, and switch the implementation from XA_STATE + xas_find() to xa_find(), which handles RCU locking and internal-retry markers itself. Rename the callers in patches 5, 7, 9. (Yosry)
- Patch 9: refuse to map a swap-cache folio in do_huge_pmd_swap_page() when folio_contain_hwpoisoned_page() reports a poisoned subpage; split the PMD swap entry so do_swap_page() can return VM_FAULT_HWPOISON per subpage instead of the PMD handler mapping
the corrupted memory as one THP. Mirrors the PageHWPoison check
the PTE swap-in path already performs. (sashiko)
- Patch 9: note explicitly in the commit message that PMD-order swap-in deliberately skips the order-0 readahead paths, order-0 readahead would populate per-page swap-cache state and force the
PMD swap entry to split before the fault could finish. (Kairui)
- Patch 10: move mm_prepare_for_swap_entries() into set_pmd_swap_entry() between folio_dup_swap() and set_pmd_at()
so this mm is on init_mm.mmlist before any swap PMD referencing slots with a non-zero swap_map becomes visible. Matches the PTE swap-out ordering. (sashiko)
- rebase onto latest akpm/mm-new (61cccb8363fcc282d4ae0555b8739dd227f5ad0b)
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Clarified the PMD swap entry rule: it is a compact encoding for HPAGE_PMD_NR swap slots, not a guarantee that swap cache always has one PMD-sized folio. (Lance Yang)
- Swapoff, fault, MADV_WILLNEED, and UFFDIO_MOVE now classify the whole PMD swap-cache range and split/retry through the PTE path for split/per-slot cache state. (Lance Yang)
- mincore handles PMD swap entries without assuming one lookup covers a split swap-cache range. (Lance Yang)
- UFFDIO_MOVE rechecks all HPAGE_PMD_NR slots before moving an empty PMD swap-cache range, avoiding stale rmap metadata for per-slot cached folios.
- Added a standalone zswap prerequisite patch from Alexandre that distinguishes all-on-disk large-folio ranges from ranges with per-page zswap entries.
- Replaced the global zswap-ever-enabled policy with per-range zswap
checks: PMD swap entries can still be installed while zswap is enabled, and PMD-order swap-in consumers split when the range has per-page zswap state.
- Added a mincore selftest and updated MADV_WILLNEED coverage so the test checks that the PMD swap entry remains in place until first touch. Total pmd_swap coverage is now 14 tests.
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Patch 1: convert two additional softleaf_to_pmd() callers that landed in mm-unstable since v1 (mm/debug_vm_pgtable.c, mm/migrate_device.c) (Dev)
- Patch 2: rename helper ensure_on_mmlist() to mm_prepare_for_swap_entries() to better describe its purpose
(David)
- Patch 3: drop VM_WARN_ON_ONCE(!pmd_is_migration_entry) as
Dev posted it as a separate patch.
- Patch 5 (new): move softleaf_to_folio() inside the device-private branch in migrate_vma_collect_pmd(); same class of fix as patch 4 but for the migrate-device PMD walker.
- Patch 6 (new): rename CONFIG_ARCH_ENABLE_THP_MIGRATION to CONFIG_ARCH_HAS_PMD_SOFTLEAVES so the gate that now drives swap-entry support too is named for what it actually controls
(PMD softleaf entries), not just migration. (Dev)
- Patch 7: add the missing pmd_swp_exclusive / mkexclusive / clear_exclusive helpers for powerpc.
- Patches 10 and 14: use upstream swapin_sync() (bundles swap_cache_alloc_folio + swap_read_folio + the -EEXIST race
retry) instead of the bespoke swapin_alloc_pmd_folio() helper
from v1; do_swap_page and shmem_swapin_folio use the same
helper (Kairui)
- Patch 10: construct a stack vm_fault for the swapoff swap-in so
the allocator can resolve a mempolicy, mirroring how the PTE
swapoff path (unuse_pte_range) already does it.
- Patch 11: extend coverage to check_pmd_state() in khugepaged so a swapped-out PMD-mapped THP is treated as SCAN_PMD_MAPPED (matches the existing migration-entry handling). Route PMD swap entries in the pmd_trans_huge_lock() branch of mincore_pte_range() through mincore_pmd_swap() so a swapped-out PMD-mapped THP isn't reported as resident.
- Patch 12 (new): handle PMD swap entries in MADV_WILLNEED via swapin_sync(BIT(HPAGE_PMD_ORDER)); a naive order-0 read-ahead
would force the subsequent fault to split.
- Patch 13: refuse UFFDIO_MOVE with -EBUSY if the swap-cache folio
was split between swap-out and the move, matching
move_pages_pte()'s rejection of large folios; otherwise only one
of the 512 anon-rmaps would be re-anchored to dst_vma.
- Patch 16: alloc_fill_swap_thp() now uses the existing mmap_pmd_aligned() helper so tests don't flake/skip based on VA placement; new MADV_WILLNEED test that watches the PMD-order
mTHP swpin counter; swapoff test restructured to use the kselftest_harness ASSERT cleanup blocks (no double swapoff, no verify-after-munmap).
- Collected Acks and Reviews
Usama Arif (30):
mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
arm64: mm: add PMD swap-exclusive helpers
loongarch: mm: add PMD swap-exclusive helpers
powerpc: mm: add PMD swap-exclusive helpers
riscv: mm: add PMD swap-exclusive helpers
s390: mm: add PMD swap-exclusive helpers
x86: mm: add PMD swap-exclusive helpers
mm: recognize PMD swap entries in the softleaf layer mm/debug_vm_pgtable: test PMD swap-exclusive helpers
mm: make PMD migration-entry splitting explicit
mm: split PMD swap entries into PTE swap entries
mm/swap: allow duplicating a range of swap entries
mm: handle PMD swap entries in fork path
mm: zswap: reject high-order swap cache allocations backed by zswap
mm: swap in PMD swap entries as whole THPs during swapoff fs/proc: account PMD swap entries in smaps
mm: handle soft-dirty and uffd-wp on PMD swap entries mm/hmm: fault PMD swap entries on demand
mm: free PMD swap entries in zap_huge_pmd()
mm/madvise: free PMD swap entries with MADV_FREE mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE mm/mincore: report PMD swap-cache residency
mm/khugepaged: treat PMD swap entries as mapped THPs
mm: handle PMD swap entries in MADV_WILLNEED
mm: handle PMD swap entries in UFFDIO_MOVE
mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
mm: handle PMD swap entry faults on swap-in
mm: install PMD swap entries on swap-out
selftests/mm: add PMD swap entry tests
Documentation/admin-guide/mm/transhuge.rst | 5 + arch/arm64/include/asm/pgtable.h | 6 + arch/loongarch/include/asm/pgtable.h | 19 + arch/powerpc/include/asm/book3s/64/pgtable.h | 6 + arch/riscv/include/asm/pgtable.h | 15 + arch/s390/include/asm/pgtable.h | 20 +- arch/x86/include/asm/pgtable.h | 20 + fs/proc/task_mmu.c | 45 +- include/linux/huge_mm.h | 40 +- include/linux/leafops.h | 44 +- include/linux/pgtable.h | 17 + include/linux/swap.h | 12 +- include/linux/vm_event_item.h | 1 + include/linux/zswap.h | 6 + mm/debug_vm_pgtable.c | 40 + mm/hmm.c | 11 +- mm/huge_memory.c | 714 +++++++++++-- mm/internal.h | 58 ++ mm/khugepaged.c | 6 + mm/madvise.c | 169 +++- mm/memory.c | 65 +- mm/migrate_device.c | 7 +- mm/mincore.c | 47 +- mm/mprotect.c | 2 +- mm/rmap.c | 27 +- mm/swap.h | 30 +- mm/swap_state.c | 83 +- mm/swapfile.c | 245 ++++- mm/userfaultfd.c | 14 + mm/vmscan.c | 9 +- mm/vmstat.c | 1 + mm/zswap.c | 12 +- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 +++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 38 files changed, 2630 insertions(+), 191 deletions(-) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 1/30 mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What it does is decode the softleaf entry stored in the PMD and return the folio that entry references - the direction softleaf_to_folio() already spells out.
No functional change intended.
--- include/linux/leafops.h | 4 ++-- mm/huge_memory.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
include/linux/leafops.h +2 -2
@@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd) }/** -* pmd_to_softleaf_folio() - Convert the PMD entry to a folio. +* pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * * The PMD entry is expected to be a valid PMD softleaf entry.
@@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)* Returns: the folio the softleaf entry references if this is a valid softleaf * entry, otherwise NULL. */ -static inline struct folio *pmd_to_softleaf_folio (pmd_t pmd) +static inline struct folio *pmd_softleaf_to_folio (pmd_t pmd) {const softleaf_t entry =softleaf_from_pmd (pmd);
mm/huge_memory.c +1 -1
@@ -2518,7 +2518,7 @@ static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,if (!thp_migration_supported ())WARN_ONCE (1 ,"Non present huge pmd without pmd migration enabled!" ); -return pmd_to_softleaf_folio (pmdval); +return pmd_softleaf_to_folio (pmdval); } static bool has_deposited_pgtable(struct vm_area_struct *vma, pmd_t pmdval,
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 2/30 arm64: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
arm64 encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse PTE_SWP_EXCLUSIVE.
--- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/include/asm/pgtable.h +6 -0
@@ -599,6 +599,12 @@ static inline int pmd_protnone(pmd_t pmd)#define pmd_swp_clear_uffd (pmd) \ pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive (pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif #define pmd_write (pmd) pte_write(pmd_pte(pmd))
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 3/30 loongarch: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
A LoongArch swap PMD is the swap PTE value plus _PAGE_HUGE, and _PAGE_SWP_EXCLUSIVE sits outside both the type and the offset field, so the PMD helpers can use the same bit.
--- arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/loongarch/include/asm/pgtable.h +19 -0
@@ -351,6 +351,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SWP_EXCLUSIVE; +return pmd; +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_SWP_EXCLUSIVE; +return pmd; +} +#endif +#define pte_none (pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_no_exec (pte) (pte_val(pte) & _PAGE_NO_EXEC)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 4/30 powerpc: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
book3s64 builds a swap PMD by running the PTE encoding over pmd_pte(), so the PMD helpers wrap the PTE ones and reuse _PAGE_SWP_EXCLUSIVE, exactly as the soft-dirty PMD helpers next to them do. It is also the only powerpc variant that selects ARCH_HAS_PMD_SOFTLEAVES, via PPC_THP.
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/powerpc/include/asm/book3s/64/pgtable.h +6 -0
@@ -1060,6 +1060,12 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#endif #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_clear_exclusive (pmd) pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif +#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE static inline int pmd_protnone (pmd_t pmd) {
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 5/30 riscv: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
riscv encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse _PAGE_SWP_EXCLUSIVE.
--- arch/riscv/include/asm/pgtable.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/riscv/include/asm/pgtable.h +15 -0
@@ -1219,6 +1219,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) }#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pte_swp_exclusive (pmd_pte (pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_mkexclusive (pmd_pte (pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_clear_exclusive (pmd_pte (pmd))); +} +#define __pmd_to_swp_entry (pmd) ((swp_entry_t) { pmd_val(pmd) })#define __swp_entry_to_pmd (swp) __pmd((swp).val)#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 6/30 s390: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap entry for the common code. Give it its own exclusive bit rather than borrowing the PTE-format macro, and document it in the RSTE swap layout, where bit 52 was listed as unused.
--- arch/s390/include/asm/pgtable.h | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
arch/s390/include/asm/pgtable.h +18 -2
@@ -333,6 +333,7 @@ void setup_protection_map(void);/* Common bits in region and segment table entries, for swap entries */ #define _RST_ENTRY_COMM 0x0010/* Common-Region/Segment, marks swap entry */ #define _RST_ENTRY_INVALID 0x0020/* invalid region/segment table entry */ +#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800/* SW exclusive swap bit, see mk_swap_rste() */ #define _CRST_ENTRIES 2048/* number of region/segment table entries */ #define _PAGE_ENTRIES 256/* number of page table entries */
@@ -893,6 +894,21 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd) }#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return set_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _RST_ENTRY_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return clear_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} +#define pmd_swp_soft_dirty (pmd) pmd_soft_dirty(pmd)#define pmd_swp_mksoft_dirty (pmd) pmd_mksoft_dirty(pmd)#define pmd_swp_clear_soft_dirty (pmd) pmd_clear_soft_dirty(pmd)
@@ -1900,15 +1916,15 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)* Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste * as invalid. * A swap entry is indicated by bit pattern (rste & 0x011) == 0x010 -* | offset |Xtype |11TT|S0| +* | offset |Etype |11TT|S0| * |0000000000111111111122222222223333333333444444444455|555555|5566|66| * |0123456789012345678901234567890123456789012345678901|234567|8901|23| * * Bits 0-51 store the offset. +* Bit 52 (E) is used to remember PG_anon_exclusive (_RST_ENTRY_SWP_EXCLUSIVE). * Bits 53-57 store the type. * Bit 62 (S) is used for softdirty tracking. * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT. -* Bit 52 (X) is unused. */ #define __SWP_OFFSET_MASK_RSTE ((1UL << 52) - 1)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 7/30 x86: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse _PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves for PG_anon_exclusive. 32-bit x86 aliases that bit to _PAGE_PSE and does not select ARCH_HAS_PMD_SOFTLEAVES; assert that, because an exclusive swap PMD would otherwise be pmd_present().
--- arch/x86/include/asm/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
arch/x86/include/asm/pgtable.h +20 -0
@@ -1525,6 +1525,26 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_EXCLUSIVE); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +/* 32-bit aliases this to _PAGE_PSE, making an exclusive swap PMD present. */ +static_assert (_PAGE_SWP_EXCLUSIVE != _PAGE_PSE); + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty (pte_t pte) {
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 8/30 mm: recognize PMD swap entries in the softleaf layer
In reply to Usama ArifReclaim splits a PMD-mapped anonymous THP into PTE-level swap entries before unmapping it, so an ordinary swap entry has never had to appear in a PMD. Prepare for installing one there instead: the softleaf layer is where every consumer decodes non-present PMDs.
Accept swap entries as valid PMD softleaves and add pmd_is_swap_entry(). A swap entry carries no PFN, so make pmd_softleaf_to_folio() warn and return NULL rather than interpret a swap offset as a page frame number.
Unlike migration and device-private entries, a PMD swap entry can also carry the swap-exclusive marker, which softleaf_from_pmd() has to strip before decoding. Strip all three overlays unconditionally while we are
here: each clear is a plain bit clear, so testing first only buys a branch.
--- include/linux/leafops.h | 40 ++++++++++++++++++++++++++++------------ include/linux/pgtable.h | 17 +++++++++++++++++ 2 files changed, 45 insertions(+), 12 deletions(-)
include/linux/leafops.h +28 -12
@@ -98,10 +98,9 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)if (pmd_present (pmd) ||pmd_none (pmd))return softleaf_mk_none (); -if (pmd_swp_soft_dirty (pmd)) - pmd =pmd_swp_clear_soft_dirty (pmd); -if (pmd_swp_uffd (pmd)) - pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_soft_dirty (pmd); + pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_exclusive (pmd); arch_entry =__pmd_to_swp_entry (pmd);/* Temporary until swp_entry_t eliminated. */
@@ -634,18 +633,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)*/ static inline bool softleaf_is_valid_pmd_entry (softleaf_t entry) { -/* Only device private, migration entries valid for PMD. */ return softleaf_is_device_private (entry) || -softleaf_is_migration (entry); +softleaf_is_migration (entry) || +softleaf_is_swap (entry); +} + +/** +* pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? +* @pmd: PMD entry. +* +* Returns: true if the PMD encodes a swap entry, otherwise false. +*/ +static inline bool pmd_is_swap_entry (pmd_t pmd) +{ +return softleaf_is_swap (softleaf_from_pmd (pmd)); }/** * pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry? * @pmd: PMD entry. * -* PMD leaf entries are valid only if they are device private or migration -* entries. This function asserts that a PMD leaf entry is valid in this -* respect. +* PMD leaf entries are valid only if they are device private, migration, +* or swap entries. This function asserts that a PMD leaf entry is valid +* in this respect. * * Returns: true if the PMD entry is a valid leaf entry, otherwise false. */
@@ -660,10 +670,12 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)* pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * -* The PMD entry is expected to be a valid PMD softleaf entry. +* The PMD entry is expected to be a valid PMD softleaf entry that references a +* PFN, that is a migration or device private entry. A PMD swap entry is a valid +* softleaf entry but encodes swap slots rather than a PFN, so it has no folio. * -* Returns: the folio the softleaf entry references if this is a valid softleaf -* entry, otherwise NULL. +* Returns: the folio the softleaf entry references, or NULL if the entry is not +* a valid PMD softleaf entry or does not reference a PFN. */ static inline struct folio *pmd_softleaf_to_folio (pmd_t pmd) {
@@ -673,6 +685,10 @@ static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)VM_WARN_ON_ONCE (true);return NULL ; } +if (!softleaf_has_pfn (entry)) { +VM_WARN_ON_ONCE (true); +return NULL ; + }return softleaf_to_folio (entry); }
include/linux/pgtable.h +17 -0
@@ -1917,6 +1917,23 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) }#endif +#ifndef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd; +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return false; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd; +} +#endif +#ifndef __HAVE_PFNMAP_TRACKING/* * Interfaces that can be used by architecture code to keep track of
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 9/30 mm/debug_vm_pgtable: test PMD swap-exclusive helpers
In reply to Usama ArifAn architecture that picked a PMD exclusive bit overlapping the swap type or offset field would otherwise only be caught by data corruption at runtime. Mirror pte_swap_exclusive_tests() at PMD level.
--- mm/debug_vm_pgtable.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
mm/debug_vm_pgtable.c +40 -0
@@ -802,6 +802,45 @@ static void __init pte_swap_exclusive_tests(struct pgtable_debug_args *args)WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static void __init pmd_swap_exclusive_tests (struct pgtable_debug_args *args) +{ +swp_entry_t entry; +softleaf_t softleaf; +pmd_t pmd; + +if (!has_transparent_hugepage ()) +return ; + +pr_debug ("Validating PMD swap exclusive\n" ); + entry = args->swp_entry; + + pmd =softleaf_to_pmd (entry); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); + + pmd =pmd_swp_mkexclusive (pmd); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (!pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (pmd_swp_soft_dirty (pmd)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); + + pmd =pmd_swp_clear_exclusive (pmd); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); +} +#else /* !CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ +static void __init pmd_swap_exclusive_tests (struct pgtable_debug_args *args) { } +#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ +static void __init pte_swap_tests (struct pgtable_debug_args *args) {swp_entry_t arch_entry;
@@ -1322,6 +1361,7 @@ static int __init debug_vm_pgtable(void)pmd_leaf_soft_dirty_tests (&args);pte_swap_exclusive_tests (&args); +pmd_swap_exclusive_tests (&args);pte_swap_tests (&args);pmd_softleaf_tests (&args);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 10/30 mm: make PMD migration-entry splitting explicit
In reply to Usama Arif__split_huge_pmd() and friends take a "freeze" boolean that every caller has to pass and almost every caller passes as false. The name says nothing about what it selects, and the one thing it does select - PTE migration entries instead of PTE mappings - is only ever wanted by the rmap migration path.
Rename it to to_migration_entries, keep it private to mm/huge_memory.c, and add split_pmd_to_migration_entries() for try_to_migrate_one(), the only caller that wants it.
migrate_vma_split_unmapped_folio() also passed freeze=true, but only ever runs on a PMD that is already a migration entry, which the generic helper expands into PTE migration entries either way. Its folio_get() only existed to balance the put_page() that freeze=true performs, so both go.
split_pmd_to_migration_entries() is only ever handed a present or device-private PMD, so assert that in __split_huge_pmd_locked() instead of silently skipping anything else.
Other than that assertion, no functional change intended.
--- include/linux/huge_mm.h | 21 ++++++------ mm/huge_memory.c | 74 ++++++++++++++++++++++++++--------------- mm/memory.c | 4 +-- mm/migrate_device.c | 7 +--- mm/mprotect.c | 2 +- mm/rmap.c | 8 ++--- 6 files changed, 65 insertions(+), 51 deletions(-)
include/linux/huge_mm.h +11 -10
@@ -430,7 +430,7 @@ int folio_memcg_alloc_deferred(struct folio *folio);void deferred_split_folio (struct folio *folio,bool partially_mapped);void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze); +unsigned long address);/** * pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf entry?
@@ -462,12 +462,10 @@ static inline bool pmd_is_huge(pmd_t pmd)do { \pmd_t *____pmd = (__pmd); \if (pmd_is_huge (*____pmd)) \ -__split_huge_pmd (__vma, __pmd, __address, \ - false); \ +__split_huge_pmd (__vma, __pmd, __address); \ } while (0) -void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address, -bool freeze); +void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address);void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud,unsigned long address);
@@ -590,7 +588,9 @@ static inline bool thp_migration_supported(void) }void split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address, -pmd_t *pmd,bool freeze); +pmd_t *pmd); +void split_pmd_to_migration_entries (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd);bool unmap_huge_pmd_locked (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp,struct folio *folio); void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,
@@ -690,12 +690,13 @@ static inline void deferred_split_folio(struct folio *folio, bool partially_mapp do { } while (0)static inline void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze) {} +unsigned long address) {}static inline void split_huge_pmd_address (struct vm_area_struct *vma, -unsigned long address,bool freeze) {} +unsigned long address) {}static inline void split_huge_pmd_locked (struct vm_area_struct *vma, -unsigned long address,pmd_t *pmd, -bool freeze) {} +unsigned long address,pmd_t *pmd) {} +static inline void split_pmd_to_migration_entries (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd) {} static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp,
mm/huge_memory.c +47 -27
171 lines not on this page · raw message
mm/memory.c +2 -2
@@ -2096,7 +2096,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, next =pmd_addr_end (addr, end);if (pmd_is_huge (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE ) -__split_huge_pmd (vma, pmd, addr, false); +__split_huge_pmd (vma, pmd, addr);else if (zap_huge_pmd (tlb, vma, pmd, addr)) { addr = next;continue ;
@@ -6382,7 +6382,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) split:/* COW or write-notify handled on pte level: split pmd. */ -__split_huge_pmd (vma, vmf->pmd, vmf->address, false); +__split_huge_pmd (vma, vmf->pmd, vmf->address);return VM_FAULT_FALLBACK ; }
mm/migrate_device.c +1 -6
@@ -918,12 +918,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; -/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
mm/mprotect.c +1 -1
@@ -530,7 +530,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb,if (pmd_is_huge (_pmd)) {if ((next - addr !=HPAGE_PMD_SIZE ) ||pgtable_split_needed (vma, cp_flags)) { -__split_huge_pmd (vma, pmd, addr, false); +__split_huge_pmd (vma, pmd, addr);/* * For file-backed, the pmd could have been * cleared; make sure pmd populated if
mm/rmap.c +3 -5
@@ -2289,8 +2289,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* We temporarily have to drop the PTL and * restart so we can process the PTE-mapped THP. */ -split_huge_pmd_locked (vma, pvmw.address, - pvmw.pmd, false); +split_huge_pmd_locked (vma, pvmw.address, pvmw.pmd); flags &= ~TTU_SPLIT_HUGE_PMD ;page_vma_mapped_walk_restart (&pvmw);continue ;
@@ -2515,13 +2514,12 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SPLIT_HUGE_PMD ) {/* -* split_huge_pmd_locked() might leave the +* split_pmd_to_migration_entries() might leave the * folio mapped through PTEs. Retry the walk * so we can detect this scenario and properly * abort the walk. */ -split_huge_pmd_locked (vma, pvmw.address, - pvmw.pmd, true); +split_pmd_to_migration_entries (vma, pvmw.address, pvmw.pmd); flags &= ~TTU_SPLIT_HUGE_PMD ;page_vma_mapped_walk_restart (&pvmw);continue ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 11/30 mm: split PMD swap entries into PTE swap entries
In reply to Usama ArifOnce a PMD can hold a swap entry, everything that splits a PMD - mprotect() or munmap() over part of the range, MADV_FREE, a pagewalk with no PMD handler - has to be able to split that entry too. A swap PMD already passes the pmd_is_valid_softleaf() gate, so without this it reaches __split_huge_pmd_locked() and falls through to the present-PMD path, which pmdp_invalidate()s it and calls pmd_page() on a non-present entry.
No reference counting is needed: a swap entry pins no folio, and swap_map is already one per slot, so the PTEs simply take over what the PMD held.
The migration-only entry point cannot reach the new branch:page_vma_mapped_walk() never hands back a swap PMD, and __split_huge_pmd_locked() already asserts that to_migration_entries implies a present or device-private PMD.
Test the pre-split old_pmd rather than re-reading *pmd in the trailing folio_remove_rmap_pmd() gate, so every entry-type test in the function interrogates the same snapshot. That part is cosmetic: pmdp_invalidate() leaves the PMD present as far as software is concerned.
--- mm/huge_memory.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
mm/huge_memory.c +21 -1
@@ -3306,6 +3306,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,folio_add_anon_rmap_ptes (folio, page,HPAGE_PMD_NR , vma, haddr, rmap_flags); } + }else if (pmd_is_swap_entry (*pmd)) { + old_pmd = *pmd; + soft_dirty =pmd_swp_soft_dirty (old_pmd); + uffd_wp =pmd_swp_uffd (old_pmd); + anon_exclusive =pmd_swp_exclusive (old_pmd); }else {/* * Up to this point the pmd is present and huge and userland has
@@ -3443,6 +3448,21 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,VM_WARN_ON (!pte_none (ptep_get (pte + i)));set_pte_at (mm, addr, pte + i, entry); } + }else if (pmd_is_swap_entry (old_pmd)) { +pte_t entry =softleaf_to_pte (softleaf_from_pmd (old_pmd)); + +if (soft_dirty) + entry =pte_swp_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_swp_mkuffd (entry); +if (anon_exclusive) + entry =pte_swp_mkexclusive (entry); + +for (i =0 , addr = haddr; i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE ) { +VM_WARN_ON (!pte_none (ptep_get (pte + i))); +set_pte_at (mm, addr, pte + i, entry); + entry =pte_next_swp_offset (entry); + } }else {pte_t entry;
@@ -3470,7 +3490,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, }pte_unmap (pte); -if (!pmd_is_migration_entry (*pmd)) +if (!pmd_is_migration_entry (old_pmd) && !pmd_is_swap_entry (old_pmd))folio_remove_rmap_pmd (folio, page, vma);if (to_migration_entries)put_page (page);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 12/30 mm/swap: allow duplicating a range of swap entries
In reply to Usama Arifswap_dup_entry_direct() duplicates one slot at a time. The PMD swap entry fork path needs HPAGE_PMD_NR of them, and doing that one slot at a time would take and drop the cluster lock HPAGE_PMD_NR times.
Give it an @nr argument and rename it swap_dup_entries_direct(), and do the same for swap_retry_table_alloc(), whose GFP_KERNEL retry has to cover the same range - the caller does not know which slot in it overflowed. Keep the old single-slot names as inline wrappers so existing callers are untouched.
Unlike the put side, @nr is handed straight to the per-cluster helper, so the range has to sit inside one cluster. That holds for the only caller passing nr > 1: a PMD swap entry only exists under CONFIG_THP_SWAP, where SWAPFILE_CLUSTER == HPAGE_PMD_NR, and a PMD-order folio's slots are only ever allocated at a cluster head (see alloc_swap_scan_cluster()), so the range is exactly one cluster. Reject a crossing range with -EINVAL so a future caller cannot walk off the end of the swap table.
No functional change intended.
--- include/linux/swap.h | 12 +++++++++- mm/swap.h | 13 +++++++++- mm/swapfile.c | 56 +++++++++++++++++++++++++++++++++----------- 3 files changed, 65 insertions(+), 16 deletions(-)
include/linux/swap.h +11 -1
@@ -414,9 +414,14 @@ sector_t swap_folio_sector(struct folio *folio);* All entries must be allocated by folio_alloc_swap(). And they must have * a swap count > 1. See comments of folio_*_swap helpers for more info. */ -int swap_dup_entry_direct (swp_entry_t entry); +int swap_dup_entries_direct (swp_entry_t entry,int nr);void swap_put_entries_direct (swp_entry_t entry,int nr); +static inline int swap_dup_entry_direct (swp_entry_t entry) +{ +return swap_dup_entries_direct (entry,1 ); +} +/* * folio_free_swap tries to free the swap entries pinned by a swap cache * folio, it has to be here to be called by other components.
@@ -458,6 +463,11 @@ static inline void free_swap_cache(struct folio *folio) { } +static inline int swap_dup_entries_direct (swp_entry_t ent,int nr) +{ +return 0 ; +} +static inline int swap_dup_entry_direct (swp_entry_t ent) {return 0 ;
mm/swap.h +12 -1
@@ -222,7 +222,12 @@ static inline void swap_cluster_unlock_irq(struct swap_cluster_info *ci)spin_unlock_irq (&ci->lock); } -extern int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp); +int swap_retry_table_alloc_nr (swp_entry_t entry,unsigned int nr,gfp_t gfp); + +static inline int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp) +{ +return swap_retry_table_alloc_nr (entry,1 , gfp); +}/* * Below are the core routines for doing swap for a folio.
@@ -428,6 +433,12 @@ static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)return 0 ; } +static inline int swap_retry_table_alloc_nr (swp_entry_t entry,unsigned int nr, +gfp_t gfp) +{ +return -EINVAL ; +} +static inline int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp) {return -EINVAL ;
mm/swapfile.c +42 -14
141 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 13/30 mm: handle PMD swap entries in fork path
In reply to Usama Arifcopy_huge_pmd() only knows about migration and device-private PMDs, so a PMD swap entry would fall through to the present-PMD path and fork() would duplicate it without taking a reference on the slots it points at.
Copy it the way copy_nonpresent_pte() copies a PTE swap entry: duplicate the swap references, clear the exclusive marker on the source, put the destination mm on mmlist, and account the child's slots to MM_SWAPENTS.
The GFP_ATOMIC extend-table allocation inside the dup can fail. Report that as -EIO and let copy_pmd_range() retry with GFP_KERNEL, as copy_nonpresent_pte() and copy_pte_range() already do for a PTE swap entry. copy_huge_pmd() hands the entry back so the caller knows which range to allocate for.
Only -ENOMEM is reported that way. The other failures mean the entry itself is bad, and swap_retry_table_alloc_nr() returns 0 for those, so collapsing them into -EIO as the PTE path does would spin in the caller's retry rather than failing the fork.
While here, move the mm counter update into each entry-type arm, as the PTE version does, so the swap arm can account MM_SWAPENTS instead of MM_ANONPAGES.
--- include/linux/huge_mm.h | 3 +- mm/huge_memory.c | 62 ++++++++++++++++++++++++++++++----------- mm/memory.c | 12 +++++++- 3 files changed, 58 insertions(+), 19 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -10,7 +10,8 @@vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf);int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm,pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, -struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma); +struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, +softleaf_t *entryp);bool huge_pmd_set_accessed (struct vm_fault *vmf); int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr,
mm/huge_memory.c +45 -17
100 lines not on this page · raw message
mm/memory.c +11 -1
@@ -1437,11 +1437,21 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, do { next = pmd_addr_end(addr, end); if (pmd_is_huge(*src_pmd)) { + softleaf_t entry = softleaf_mk_none(); int err; VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma); +again: err = copy_huge_pmd(dst_mm, src_mm, dst_pmd, src_pmd, - addr, dst_vma, src_vma); + addr, dst_vma, src_vma, &entry); + if (err == -EIO) { + VM_WARN_ON_ONCE(!entry.val); + if (swap_retry_table_alloc_nr(entry, + HPAGE_PMD_NR, + GFP_KERNEL) < 0) + return -ENOMEM; + goto again; + } if (err == -ENOMEM) return -ENOMEM; if (!err)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 14/30 mm: zswap: reject high-order swap cache allocations backed by zswap
In reply to Usama Arifzswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Reject that in __swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, which zswap_load() handles per page.
Export zswap_is_present() for the PMD-order swap-in users added later.
--- include/linux/zswap.h | 6 ++++++ mm/swap_state.c | 39 +++++++++++++++++++++++++-------------- mm/zswap.c | 12 +++++++++--- 3 files changed, 40 insertions(+), 17 deletions(-)
include/linux/zswap.h +6 -0
@@ -35,6 +35,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);void zswap_folio_swapin (struct folio *folio);bool zswap_is_enabled (void );bool zswap_never_enabled (void ); +bool zswap_is_present (swp_entry_t entry,unsigned int nr);#else struct zswap_lruvec_state {};
@@ -73,6 +74,11 @@ static inline bool zswap_never_enabled(void)return true; } +static inline bool zswap_is_present (swp_entry_t entry,unsigned int nr) +{ +return false; +} +#endif #endif /* _LINUX_ZSWAP_H */
mm/swap_state.c +25 -14
@@ -12,6 +12,7 @@#include <linux/kernel_stat.h> #include <linux/mempolicy.h> #include <linux/swap.h> +#include <linux/zswap.h> #include <linux/leafops.h> #include <linux/init.h> #include <linux/pagemap.h>
@@ -466,26 +467,27 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,__swap_cache_do_add_folio (ci, folio, entry);spin_unlock (&ci->lock); +/* +* Now that the folio is in the swap cache, zswap can no longer start +* storing or writing back any slot in the range, so this is a stable +* answer. Reject a high-order allocation over a range that already +* has per-page zswap entries. +*/ +if (order &&zswap_is_present (entry, nr_pages)) { + err = -EBUSY ; + goto delete_folio; + } +if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); + err = -ENOMEM ;count_mthp_stat (order,MTHP_STAT_SWPIN_FALLBACK_CHARGE ); -return ERR_PTR (-ENOMEM ); + goto delete_folio; }if (order >1 &&folio_memcg_alloc_deferred (folio)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); -return ERR_PTR (-ENOMEM ); + err = -ENOMEM ; + goto delete_folio; }/* memsw uncharges swap when folio is added to swap cache */
@@ -499,6 +501,15 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,/* Caller will initiate read into locked new_folio */ folio_add_lru (folio);return folio; + +delete_folio: +spin_lock (&ci->lock); +__swap_cache_do_del_folio (ci, folio, entry, shadow); +spin_unlock (&ci->lock); +folio_unlock (folio); +/* nr_pages refs from swap cache, 1 from allocation */ +folio_put_refs (folio, nr_pages +1 ); +return ERR_PTR (err); }/**
mm/zswap.c +9 -3
@@ -1587,11 +1587,14 @@ bool zswap_store(struct folio *folio)* @entry: base swap entry of the range * @nr: number of contiguous slots to check * -* Context: The caller must keep the range pinned, otherwise the answer can -* change under it. +* Context: Unless the caller keeps the range pinned, the answer is only a +* hint: zswap can store or write back a slot right after this returns. A +* caller that needs a stable answer must pin the range first, as +* __swap_cache_alloc() does by inserting the folio into the swap cache before +* asking. * Return: true if at least one slot in the range is in zswap. */ -static bool zswap_is_present (swp_entry_t entry,unsigned int nr) +bool zswap_is_present (swp_entry_t entry,unsigned int nr) {pgoff_t offset =swp_offset (entry);struct xarray *tree =swap_zswap_tree (entry);
@@ -1605,6 +1608,9 @@ static bool zswap_is_present(swp_entry_t entry, unsigned int nr)*/ BUILD_BUG_ON (SWAPFILE_CLUSTER >ZSWAP_ADDRESS_SPACE_PAGES ); +if (zswap_never_enabled ()) +return false; +return xa_find (tree, &index, offset + nr -1 ,XA_PRESENT ); }
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 15/30 mm: swap in PMD swap entries as whole THPs during swapoff
In reply to Usama Arifswapoff walks every mm and faults every slot of the device back in. unuse_pte_range() only understands PTEs, so a PMD swap entry would never be found and swapoff would never complete.
A PMD swap entry is a compact encoding for HPAGE_PMD_NR slots, not a promise that the swap cache holds one folio for them. Add swap_pmd_cache_lookup() to classify the covered range as empty, one PMD-sized folio, or already split, and unuse_pmd() to map the first two cases back in as one THP, preserving soft-dirty, exclusive and UFFD state.
Everything else falls back to PTEs: a split cache, per-page zswap state, a failed PMD-order allocation or read, or a poisoned subpage. Check PageHWPoison on every subpage rather than the folio-level flag, which memory_failure() only sets after taking the folio lock.
All the fallback reasons are observed without the PMD lock and possibly after sleeping, so they share one exit that re-checks the PMD is still the entry we were called for before splitting it. That exit also drops a folio that is not uptodate, or that has never been mapped, from the swap cache:
the PTE path cannot re-read the first, and would add a single-page rmap to the second.
--- mm/internal.h | 16 ++++ mm/swap.h | 17 +++++ mm/swap_state.c | 44 +++++++++++ mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 266 insertions(+)
mm/internal.h +16 -0
@@ -24,6 +24,22 @@struct folio_batch ; +/* +* Unlike folio_contain_hwpoisoned_page(), this does not rely on the folio-level +* PG_has_hwpoisoned, which memory_failure() only sets after taking the folio +* lock and so can lag a tail-page poison. +*/ +static inline bool folio_has_hwpoisoned_subpage (const struct folio *folio) +{ +long nr =folio_nr_pages (folio); +long i; + +for (i =0 ; i < nr; i++) +if (PageHWPoison (folio_page (folio, i))) +return true; +return false; +} +/* mm/workingset.c */ bool workingset_test_recent (void *shadow,bool file,bool *workingset,bool flush);
mm/swap.h +17 -0
@@ -316,6 +316,23 @@ static inline bool folio_matches_swap_entry(const struct folio *folio,bool swap_cache_has_folio (swp_entry_t entry);struct folio *swap_cache_get_folio (swp_entry_t entry);void *swap_cache_get_shadow (swp_entry_t entry); +enum swap_pmd_cache { +SWAP_PMD_CACHE_EMPTY , +SWAP_PMD_CACHE_HUGE , +SWAP_PMD_CACHE_SPLIT , +}; + +#ifdef CONFIG_THP_SWAP +enum swap_pmd_cache swap_pmd_cache_lookup (swp_entry_t entry, +struct folio **foliop); +#else +static inline enum swap_pmd_cache swap_pmd_cache_lookup (swp_entry_t entry, +struct folio **foliop) +{ + *foliop =NULL ; +return SWAP_PMD_CACHE_EMPTY ; +} +#endif void swap_cache_del_folio (struct folio *folio); struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask, unsigned long orders, struct vm_fault *vmf,
mm/swap_state.c +44 -0
50 lines not on this page · raw message
mm/swapfile.c +189 -0
209 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 16/30 fs/proc: account PMD swap entries in smaps
In reply to Usama Arifsmaps_pmd_entry() derives a page from a non-present PMD and gives up if it cannot. A PMD swap entry carries no PFN, so a swapped-out THP is missing from both Swap and SwapPss.
The slots of a PMD swap entry can have different swap reference counts, so SwapPss has to be computed one slot at a time rather than divided once. Factor that loop out of smaps_pte_entry() and call it from both levels, then return early: a swapped-out THP is not resident and must not be added to AnonHugePages or Rss.
While touching PMD softleaf accounting, match the PTE path and treat device-private PMD entries as fake-present. This lets smaps use mapcount-based PSS and private/shared accounting instead of treating them like migration entries.
--- fs/proc/task_mmu.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
fs/proc/task_mmu.c +28 -13
@@ -957,6 +957,25 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)#endif } +static void smaps_account_swap (struct mem_size_stats *mss, +softleaf_t entry,unsigned long nr_pages) +{ + mss->swap += nr_pages <<PAGE_SHIFT ; +do { +int swapcount =swp_swapcount (entry); + +if (swapcount >=2 ) { +u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; + +do_div (pss_delta, swapcount); + mss->swap_pss += pss_delta; + }else { + mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; + } + entry.val++; + }while (--nr_pages); +} +static void smaps_pte_entry (pte_t *pte,unsigned long addr,struct mm_walk *walk) {
@@ -978,18 +997,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,const softleaf_t entry =softleaf_from_pte (ptent);if (softleaf_is_swap (entry)) { -int mapcount; - - mss->swap +=PAGE_SIZE ; - mapcount =swp_swapcount (entry); -if (mapcount >=2 ) { -u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; - -do_div (pss_delta, mapcount); - mss->swap_pss += pss_delta; - }else { - mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; - } +smaps_account_swap (mss, entry,1 ); }else if (softleaf_has_pfn (entry)) {if (softleaf_is_device_private (entry)) present = true;
@@ -1022,8 +1030,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, }else if (unlikely (thp_migration_supported ())) {const softleaf_t entry =softleaf_from_pmd (*pmd); -if (softleaf_has_pfn (entry)) +if (softleaf_is_swap (entry)) { +smaps_account_swap (mss, entry,HPAGE_PMD_NR ); +return ; + } +if (softleaf_has_pfn (entry)) { +if (softleaf_is_device_private (entry)) + present = true; page =softleaf_to_page (entry); + } }if (IS_ERR_OR_NULL (page))return ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 17/30 mm: handle soft-dirty and uffd-wp on PMD swap entries
In reply to Usama ArifSoft-dirty and uffd-wp live in the PMD, not in the swap entry, so every path that manipulates them for a PMD migration entry has to do the same for a PMD swap entry. Otherwise clearing soft-dirty skips a swapped-out THP, UFFDIO_WRITEPROTECT does not arm it, and mremap does not mark the destination.
change_huge_pmd() needs nothing beyond dropping its thp_migration_supported() gate: a PMD swap entry encodes no write permission, so it falls into the default arm of change_non_present_huge_pmd() and picks up only the uffd-wp update.
--- fs/proc/task_mmu.c | 4 ++-- mm/huge_memory.c | 11 +++++++++-- 2 files changed, 11 insertions(+), 4 deletions(-)
fs/proc/task_mmu.c +2 -2
@@ -1655,7 +1655,7 @@ static void clear_soft_dirty_pmd(struct vm_area_struct *vma, pmd =pmd_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
@@ -2481,7 +2481,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma, old =pmdp_invalidate_ad (vma, addr, pmdp); pmd =pmd_mkuffd (old);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_mkuffd (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
mm/huge_memory.c +9 -2
@@ -2637,7 +2637,8 @@ static inline int pmd_move_must_withdraw(spinlock_t *new_pmd_ptl,static pmd_t move_soft_dirty_pmd (pmd_t pmd) {if (pgtable_supports_soft_dirty ()) { -if (unlikely (pmd_is_migration_entry (pmd))) +if (unlikely (pmd_is_migration_entry (pmd) || +pmd_is_swap_entry (pmd))) pmd =pmd_swp_mksoft_dirty (pmd);else if (pmd_present (pmd)) pmd =pmd_mksoft_dirty (pmd);
@@ -2729,6 +2730,12 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,pmd_t newpmd;VM_WARN_ON (!pmd_is_valid_softleaf (*pmd)); + +/* +* Note that a PMD swap entry falls into the default branch below: it +* does not encode write permission in the entry type, so only the +* uffd_wp flag update at the end applies to it. +*/ if (softleaf_is_migration_write (entry)) {const struct folio *folio =softleaf_to_folio (entry);
@@ -2789,7 +2796,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,if (!ptl)return 0 ; -if (thp_migration_supported () &&pmd_is_valid_softleaf (*pmd)) { +if (pmd_is_valid_softleaf (*pmd)) {change_non_present_huge_pmd (mm, addr, pmd, uffd_prot, uffd_prot_resolve); goto unlock;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 18/30 mm/hmm: fault PMD swap entries on demand
In reply to Usama Arifhmm_vma_handle_absent_pmd() reports anything non-present that is not device-private as HMM_PFN_ERROR, so a swapped-out THP would look like a permanent failure rather than memory that needs paging in.
Treat it like a PTE swap entry: record the fault when one was asked for, and otherwise report the range as non-resident with no flags, which is what hmm_vma_handle_pte() already does in snapshot mode.
--- mm/hmm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/hmm.c +10 -1
@@ -377,12 +377,21 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start, required_fault =hmm_range_need_fault (hmm_vma_walk, hmm_pfns, npages,0 );if (required_fault) { -if (softleaf_is_device_private (entry)) +if (softleaf_is_device_private (entry) || +softleaf_is_swap (entry))return hmm_record_fault (addr, end, required_fault, walk);else return -EFAULT ; } +/* +* A swapped-out THP is not resident. Report it as not-valid, +* matching what hmm_vma_handle_pte() does for a PTE swap entry when +* no fault was requested. +*/ +if (softleaf_is_swap (entry)) +return hmm_pfns_fill (start, end, range,0 ); + return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); }#else
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 19/30 mm: free PMD swap entries in zap_huge_pmd()
In reply to Usama Arifzap_huge_pmd() assumes any non-present PMD it clears references a folio, so a PMD swap entry would reach normal_or_softleaf_folio_pmd() and warn. Worse, nothing would release the slots it points at: unmapping a swapped-out THP would leak the whole PMD's worth of swap.
Handle it before the folio lookup, as zap_pte_range() already does for a PTE swap entry.
--- mm/huge_memory.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/huge_memory.c +10 -0
@@ -2605,6 +2605,16 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,arch_check_zapped_pmd (vma, orig_pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr); +if (pmd_is_swap_entry (orig_pmd)) { +softleaf_t entry =softleaf_from_pmd (orig_pmd); + +zap_deposited_table (mm, pmd); +spin_unlock (ptl); +swap_put_entries_direct (entry,HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , -HPAGE_PMD_NR ); +return true; + } + is_present =pmd_present (orig_pmd); folio =normal_or_softleaf_folio_pmd (vma, addr, orig_pmd, is_present); has_deposit =has_deposited_pgtable (vma, orig_pmd, folio);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 20/30 mm/madvise: free PMD swap entries with MADV_FREE
In reply to Usama Arifmadvise_free_pte_range() only calls madvise_free_huge_pmd() for a present THP, and its PTE loop cannot descend into a non-present PMD, so MADV_FREE over a swapped-out THP silently does nothing. Over the same range as PTE swap entries it frees every slot, which is what userspace gets today.
Free it at PMD granularity when the advice covers the whole PMD, and split and let the PTE path handle the subrange when it does not.
zap_deposited_table() moves above its new caller; it is otherwise unchanged.
--- mm/huge_memory.c | 34 +++++++++++++++++++++++++--------- mm/madvise.c | 2 +- 2 files changed, 26 insertions(+), 10 deletions(-)
mm/huge_memory.c +25 -9
@@ -2424,6 +2424,15 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)return 0 ; } +static inline void zap_deposited_table (struct mm_struct *mm,pmd_t *pmd) +{ +pgtable_t pgtable; + + pgtable =pgtable_trans_huge_withdraw (mm, pmd); +pte_free_defer (mm, pgtable); +mm_dec_nr_ptes (mm); +} +/* * Return true if we do MADV_FREE successfully on entire pmd page. * Otherwise, return false.
@@ -2448,6 +2457,22 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, goto out;if (unlikely (!pmd_present (orig_pmd))) { +if (pmd_is_swap_entry (orig_pmd)) { +softleaf_t entry =softleaf_from_pmd (orig_pmd); + +if (next - addr !=HPAGE_PMD_SIZE ) { +spin_unlock (ptl); +__split_huge_pmd (vma, pmd, addr); + goto out_unlocked; + } + +pmdp_huge_get_and_clear (mm, addr, pmd); +zap_deposited_table (mm, pmd); +spin_unlock (ptl); +swap_put_entries_direct (entry,HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , -HPAGE_PMD_NR ); +return true; + }VM_WARN_ON_ONCE (!pmd_is_migration_entry (orig_pmd) && !pmd_is_device_private_entry (orig_pmd)); goto out;
@@ -2502,15 +2527,6 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,return ret; } -static inline void zap_deposited_table (struct mm_struct *mm,pmd_t *pmd) -{ -pgtable_t pgtable; - - pgtable =pgtable_trans_huge_withdraw (mm, pmd); -pte_free_defer (mm, pgtable); -mm_dec_nr_ptes (mm); -} -static void zap_huge_pmd_folio (struct mm_struct *mm,struct vm_area_struct *vma,pmd_t pmdval,struct folio *folio,bool is_present) {
mm/madvise.c +1 -1
@@ -678,7 +678,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,int nr, max_nr; next =pmd_addr_end (addr, end); -if (pmd_trans_huge (*pmd)) +if (pmd_trans_huge (*pmd) ||pmd_is_swap_entry (*pmd))if (madvise_free_huge_pmd (tlb, vma, pmd, addr, next))return 0 ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 21/30 mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT
In reply to Usama ArifA swapped-out THP has no resident folio to deactivate or reclaim, which is also the conclusion the PTE path reaches when it walks the swap entries and skips every one.
madvise_cold_or_pageout_pte_range() gets there the long way round:pmd_trans_huge() is false for a PMD swap entry, so it falls out of the huge branch and tries to map a PTE table that a non-present PMD does not have.
The entry can also appear after that first, lockless test, so re-check under the PMD lock; otherwise a PMD that turned into a swap entry while pmd_trans_huge_lock() was being taken trips the VM_WARN_ON_ONCE() that expects a migration or device-private entry.
--- mm/madvise.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
mm/madvise.c +12 -0
@@ -385,6 +385,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, !can_do_file_pageout (vma);#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* +* Swapped-out THPs have no resident folio to deactivate or reclaim. +* Avoid descending into or splitting a PMD swap entry. +*/ +if (pmd_is_swap_entry (*pmd)) { + walk->action =ACTION_CONTINUE ; +return 0 ; + } +if (pmd_trans_huge (*pmd)) {pmd_t orig_pmd;unsigned long next =pmd_addr_end (addr, end);
@@ -395,6 +404,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ; orig_pmd = *pmd; +if (pmd_is_swap_entry (orig_pmd)) + goto huge_unlock; +if (unlikely (!pmd_present (orig_pmd))) {VM_WARN_ON_ONCE (!pmd_is_migration_entry (orig_pmd) && !pmd_is_device_private_entry (orig_pmd));
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 22/30 mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE
In reply to Usama ArifBoth callbacks test bare pmd_trans_huge(), which is false for a PMD swap entry, so they return without consuming the PMD and walk_pmd_range() splits it - only for the PTE callbacks to find the range populated and abort, or to find no guard markers and do nothing.
Both already short-circuit at PMD level for a present THP. Do the same for a swapped-out one: it is just as populated, and just as free of guard markers.
--- mm/madvise.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
mm/madvise.c +12 -4
@@ -1269,8 +1269,12 @@ static int guard_install_pmd_entry(pmd_t *pmd, unsigned long addr, {pmd_t pmdval =pmdp_get (pmd); -/* If huge return >0 so we abort the operation + zap. */ -return pmd_trans_huge (pmdval); +/* +* If huge return >0 so we abort the operation + zap. A PMD swap entry +* is a swapped-out THP: also populated, and splitting it here would +* only demote it before the zap. +*/ +return pmd_trans_huge (pmdval) ||pmd_is_swap_entry (pmdval); } static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
@@ -1409,8 +1413,12 @@ static int guard_remove_pmd_entry(pmd_t *pmd, unsigned long addr, {pmd_t pmdval =pmdp_get (pmd); -/* If huge, cannot have guard pages present, so no-op - skip. */ -if (pmd_trans_huge (pmdval)) +/* +* If huge, cannot have guard pages present, so no-op - skip. The same +* is true of a PMD swap entry, which must not be split just to discover +* there is nothing to remove. +*/ +if (pmd_trans_huge (pmdval) ||pmd_is_swap_entry (pmdval)) walk->action =ACTION_CONTINUE ;return 0 ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 23/30 mm/mincore: report PMD swap-cache residency
In reply to Usama Arifmincore_pte_range() reports every page under a huge PMD as resident. That is wrong for a swapped-out one: mincore() would tell userspace that memory it will have to fault in is already in core.
A PMD swap entry is only a compact encoding for HPAGE_PMD_NR slots, so answer from the swap cache instead. One PMD-sized folio covering the range makes every page resident exactly when that folio is uptodate; an empty cache makes none of them; and if the folio was split while the entry stayed in place, fall back to looking up each covered slot as mincore_swap() does for a PTE swap entry.
--- mm/mincore.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)
mm/mincore.c +46 -1
@@ -85,6 +85,48 @@ static unsigned char mincore_swap(swp_entry_t entry, bool shmem)return present; } +#ifdef CONFIG_THP_SWAP +static void mincore_pmd_swap (swp_entry_t entry,unsigned long addr, +unsigned long end,unsigned char *vec) +{ +unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long start = (addr - haddr) >>PAGE_SHIFT ; +unsigned long nr = (end - addr) >>PAGE_SHIFT ; +struct folio *folio; +enum swap_pmd_cache state; +int i; + + state =swap_pmd_cache_lookup (entry, &folio); +if (state ==SWAP_PMD_CACHE_HUGE ) { +memset (vec,folio_test_uptodate (folio), nr); +folio_put (folio); +return ; + } + +if (state ==SWAP_PMD_CACHE_EMPTY ) { +memset (vec,0 , nr); +return ; + } + +/* +* The PMD swap entry is only a compact encoding for consecutive swap +* slots. If the PMD-sized swapcache folio was split, report residency +* from the individual slots covered by this mincore() range. +*/ +for (i =0 ; i < nr; i++) + vec[i] =mincore_swap (swp_entry (swp_type (entry), +swp_offset (entry) + start + i), + false); +} +#else /* !CONFIG_THP_SWAP */ +static void mincore_pmd_swap (swp_entry_t entry,unsigned long addr, +unsigned long end,unsigned char *vec) +{ +/* Nothing produces PMD swap entries without CONFIG_THP_SWAP. */ +memset (vec,0 , (end - addr) >>PAGE_SHIFT ); +} +#endif /* CONFIG_THP_SWAP */ +/* * Later we can get more picky about what "in core" means precisely. * For now, simply check to see if the page is in the page cache,
@@ -171,7 +213,10 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_is_swap_entry (*pmd)) +mincore_pmd_swap (softleaf_from_pmd (*pmd), addr, end, vec); +else +memset (vec,1 , nr);spin_unlock (ptl); goto out; }
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 24/30 mm/khugepaged: treat PMD swap entries as mapped THPs
In reply to Usama Arifcheck_pmd_state() would report a PMD swap entry as SCAN_NO_PTE_TABLE - it is not present and, on x86, is also pmd_bad(). The range is already a THP; it just happens to be on disk.
Return SCAN_PMD_MAPPED next to the existing migration-entry case and before the pmd_present() and pmd_bad() tests.
This does change MADV_COLLAPSE. Today a swapped-out THP is HPAGE_PMD_NR PTE swap entries and MADV_COLLAPSE, which ignores khugepaged_max_ptes_swap, swaps them all in. With a PMD swap entry it reports SCAN_PMD_MAPPED, which madvise_collapse() counts as success, so the call returns 0 without making the range resident. It still faults back in as a whole THP on first touch; only the point at which the I/O happens moves.
--- mm/khugepaged.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/khugepaged.c +6 -0
@@ -1052,6 +1052,12 @@ static inline enum scan_result check_pmd_state(pmd_t *pmd)*/ if (pmd_is_migration_entry (pmde))return SCAN_PMD_MAPPED ; +/* +* A PMD-mapped THP that has been swapped out is still a THP from +* khugepaged's perspective; treat it like a present huge PMD. +*/ +if (pmd_is_swap_entry (pmde)) +return SCAN_PMD_MAPPED ;if (!pmd_present (pmde))return SCAN_NO_PTE_TABLE ;if (pmd_trans_huge (pmde))
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 25/30 mm: handle PMD swap entries in MADV_WILLNEED
In reply to Usama Arifswapin_walk_pmd_entry() walks PTEs and cannot descend into a non-present PMD, so MADV_WILLNEED is a no-op on a PMD swap entry. Reading the slots back one at a time would not be much better: order-0 readahead populates per-page swap-cache state, and the fault that follows would then have to split the entry the prefetch was supposed to help.
Prefetch the whole range as one PMD-order folio instead, leaving the entry in place so the eventual fault still takes do_huge_pmd_swap_page(). Check the THP policy first: the range was PMD-mapped when it was swapped out, but MADV_NOHUGEPAGE or the sysfs knob may have changed since, and reading at PMD order would then hand do_swap_page() a PMD-sized folio to map with PTEs.
A split cache, per-page zswap state, or a failed PMD-order read or allocation falls back to PTEs. Leaving the entry alone would prefetch nothing at all, while the PTE path can still read the slots at order 0. A folio that failed to read is dropped from the swap cache first, so the PTE retry re-reads each slot rather than every later fault returning SIGBUS.
Only split if the PMD still looks like the entry that was observed - the decision is taken after the PMD lock has been dropped, so a racing fault may have swapped the range back in as a THP. The test is lockless, so it narrows that window rather than closing it.
--- mm/madvise.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+)
mm/madvise.c +139 -0
@@ -33,6 +33,7 @@#include <linux/shmem_fs.h> #include <linux/mmu_notifier.h> #include <linux/swap_ops.h> +#include <linux/zswap.h> #include <asm/tlb.h>
@@ -194,6 +195,108 @@ static int madvise_update_vma(vm_flags_t new_flags, }#ifdef CONFIG_SWAP +/* +* Prefetch a whole PMD swap entry as one PMD-order folio. +* +* Called with the PMD lock held; always drops it. Returns true when the +* caller should ask the walker to retry so the PTE path can handle the +* covered slots individually. +*/ +static bool swapin_pmd_swap_entry (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry, +spinlock_t *ptl) +{ +struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; +enum swap_pmd_cache cache_state; +struct swap_info_struct *si; +struct folio *folio; +bool split = false; + +/* +* The range was PMD-mapped when it was swapped out, but the policy may +* have changed since: MADV_NOHUGEPAGE, or the sysfs knob. Reading it +* back at PMD order would then hand do_swap_page() a PMD-sized folio to +* map with PTEs, which is exactly what the policy forbids. Split now +* and let the PTE path prefetch at order 0 - the next fault would split +* the entry anyway, so nothing is lost that the VMA still permits. +*/ +if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_PAGEFAULT , +HPAGE_PMD_ORDER )) { +spin_unlock (ptl); +return true; + } + + cache_state =swap_pmd_cache_lookup (entry, &folio); +if (cache_state ==SWAP_PMD_CACHE_HUGE ) { +/* Already cached as one PMD-sized folio, nothing to do. */ +folio_put (folio); +spin_unlock (ptl); +return false; + } +if (cache_state ==SWAP_PMD_CACHE_SPLIT || +zswap_is_present (entry,HPAGE_PMD_NR )) { +spin_unlock (ptl); +return true; + } + +/* +* Pin the swap device under the PMD lock so the PMD-swap-entry +* observation keeps the entry valid for swapin_sync(). +*/ + si =get_swap_device (entry); +spin_unlock (ptl); +if (IS_ERR_OR_NULL (si)) +return false; + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE ,BIT (HPAGE_PMD_ORDER ), + &vmf,NULL ,0 ); + +/* +* Fall back to PTE-order swapin: a PMD-order failure does not mean +* that individual slots cannot be read. +*/ +if (IS_ERR_OR_NULL (folio)) { + split = true; + goto out; + } + +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { + split = true; + goto out_put; + } + +/* +* A trylock only succeeds once the read has completed, so this never +* blocks MADV_WILLNEED on in-flight I/O. A read that failed - a +* PMD-order zswap load that found per-page state, or an I/O error - +* leaves the folio clean and not uptodate. Drop it from the swap cache +* so the PTE retry reads each slot again; leaving it there would make +* the next fault return VM_FAULT_SIGBUS. Another thread may have +* removed it already, so revalidate the association first. +*/ +if (!folio_trylock (folio)) + goto out_put; + +if (!folio_test_uptodate (folio)) { +if (folio_matches_swap_entry (folio, entry)) +swap_cache_del_folio (folio); + split = true; + } +folio_unlock (folio); + +out_put: +folio_put (folio); +out: +/* Keep the device pinned until the last use of @entry. */ +put_swap_device (si); +return split; +} +static int swapin_walk_pmd_entry (pmd_t *pmd,unsigned long start,unsigned long end,struct mm_walk *walk) {
@@ -203,6 +306,41 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,spinlock_t *ptl;unsigned long addr; + ptl =pmd_trans_huge_lock (pmd, vma); +if (ptl) { +pmd_t pmdval = *pmd; + +if (pmd_is_swap_entry (pmdval)) { +/* swapin_pmd_swap_entry() always drops the PMD lock. */ +if (!swapin_pmd_swap_entry (vma, pmd, start, +softleaf_from_pmd (pmdval), + ptl)) + goto ret; +/* +* Only split if this still looks like the entry we +* observed. The fallback was decided after the PMD lock +* was dropped, so a racing fault may have swapped the +* range back in as a THP, and splitting that would +* demote a perfectly good huge mapping for an advisory +* hint. The test is lockless, so it narrows that window +* rather than closing it. +*/ +if (pmd_same (pmdval,pmdp_get_lockless (pmd))) { +__split_huge_pmd (vma, pmd, start); + walk->action =ACTION_AGAIN ; + goto ret; + } +/* +* Somebody else changed the PMD. Leave it alone and let +* the PTE loop below deal with whatever is there now; +* it simply finds no page table if the range came back +* as a THP. +*/ + }else { +spin_unlock (ptl); + } + } +for (addr = start; addr < end; addr +=PAGE_SIZE ) {pte_t pte;softleaf_t entry;
@@ -231,6 +369,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,if (ptep)pte_unmap_unlock (ptep, ptl);swap_read_submit (&ctx); +ret:cond_resched ();return 0 ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 26/30 mm: handle PMD swap entries in UFFDIO_MOVE
In reply to Usama Arifmove_pages_huge_pmd() returns -ENOENT for any PMD that is neither trans_huge nor a migration entry, so an aligned UFFDIO_MOVE over a swapped-out THP fails even though a PMD swap entry is a perfectly good mapping to move. Falling back to the PTE path is no help either: splitting yields PTE swap entries pointing at the same swap-cache folio, and move_pages_ptes() refuses any swap-cache folio that is still large.
move_swap_pmd() is modelled on move_swap_pte(): it moves the entry under both PMD locks, propagates soft-dirty, arms the UFFD marker for an RWP-registered destination, carries the deposited page table across, and requires pmd_swp_exclusive() for the same single-owner semantics.
The entry can only be moved whole while the covered swap cache is empty or holds one PMD-sized folio. A cached folio is locked and revalidated, then its anon rmap is re-anchored to the destination VMA; an empty cache is re-checked slot by slot under both PMD locks, because a per-slot folio that appeared meanwhile would need the PTE path to fix up its rmap metadata. A range that is already split is split and retried through PTEs. Revalidation failure just returns -EAGAIN: its usual cause is a racing fault that made src_pmd a healthy present THP, which must not be shattered.
Finally, reject a PMD swap entry at the *destination* with -EEXIST. It is not a hole, and unlike a migration entry it does not resolve on its own:pte_alloc() skips a !pmd_none PMD, pte_offset_map_rw_nolock() then fails, and the resulting -EAGAIN would be retried forever.
--- mm/huge_memory.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++- mm/userfaultfd.c | 14 +++++ 2 files changed, 171 insertions(+), 1 deletion(-)
mm/huge_memory.c +157 -1
@@ -2949,6 +2949,78 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,#endif #ifdef CONFIG_USERFAULTFD +#ifdef CONFIG_THP_SWAP +/* +* Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are +* acquired here; src_folio (if present) must already be locked. The deposited +* page table backing the source THP is moved across with the entry. +*/ +static int move_swap_pmd (struct mm_struct *mm,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +pmd_t *dst_pmd,pmd_t *src_pmd, +pmd_t orig_dst_pmd,pmd_t orig_src_pmd, +spinlock_t *dst_ptl,spinlock_t *src_ptl, +struct folio *src_folio,swp_entry_t entry) +{ +pgtable_t src_pgtable; +pmd_t moved_pmd; + +/* +* The folio may have been freed and reused for a different swap entry +* while it was unlocked. Re-verify the association. +*/ +if (src_folio &&unlikely (!folio_matches_swap_entry (src_folio, entry) || +folio_nr_pages (src_folio) !=HPAGE_PMD_NR )) +return -EAGAIN ; + +double_pt_lock (dst_ptl, src_ptl); + +if (!pmd_same (*src_pmd, orig_src_pmd) || + !pmd_same (*dst_pmd, orig_dst_pmd)) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + +/* +* If the folio is in the swap cache, re-anchor its anon rmap to the +* destination VMA so a future swap-in fault at dst_addr finds it. +* Otherwise, re-check the whole PMD swap range: a PMD swap entry is +* only a compact encoding for HPAGE_PMD_NR swap slots, and any per-slot +* cached folio would need the PTE move path to update its rmap +* metadata. +*/ +if (src_folio) { +folio_move_anon_rmap (src_folio, dst_vma); + src_folio->index =linear_anon_page_index (dst_vma, dst_addr); + }else { +unsigned int type =swp_type (entry); +pgoff_t offset =swp_offset (entry); +int i; + +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +if (swap_cache_has_folio (swp_entry (type, offset + i))) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + } + } + + moved_pmd =pmdp_huge_get_and_clear (mm, src_addr, src_pmd); +if (pgtable_supports_soft_dirty ()) + moved_pmd =pmd_swp_mksoft_dirty (moved_pmd); +/* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ +if (userfaultfd_rwp (dst_vma)) + moved_pmd =pmd_swp_mkuffd (moved_pmd); +set_pmd_at (mm, dst_addr, dst_pmd, moved_pmd); + + src_pgtable =pgtable_trans_huge_withdraw (mm, src_pmd); +pgtable_trans_huge_deposit (mm, dst_pmd, src_pgtable); + +double_pt_unlock (dst_ptl, src_ptl); +return 0 ; +} +#endif /* CONFIG_THP_SWAP */ +/* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by * the caller, but it must return after releasing the page_table_lock.
@@ -2983,11 +3055,95 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }if (!pmd_trans_huge (src_pmdval)) { -spin_unlock (src_ptl);if (pmd_is_migration_entry (src_pmdval)) { +spin_unlock (src_ptl);pmd_migration_entry_wait (mm, src_pmd);return -EAGAIN ; } +#ifdef CONFIG_THP_SWAP +if (pmd_is_swap_entry (src_pmdval)) { +swp_entry_t entry; +struct swap_info_struct *si; +enum swap_pmd_cache cache_state; + +/* +* UFFDIO_MOVE on anon mappings requires single-owner +* semantics; refuse to move a shared swap entry. +*/ +if (!pmd_swp_exclusive (src_pmdval)) { +spin_unlock (src_ptl); +return -EBUSY ; + } + + entry =softleaf_from_pmd (src_pmdval); +spin_unlock (src_ptl); + +/* +* Pin the swap device against a racing swapoff. NULL +* means swapoff is in progress, which resolves on its +* own, so ask the caller to retry. An error pointer +* means the entry names no swap device at all: that +* never resolves, so report it instead of spinning in +* the caller's -EAGAIN loop. +*/ + si =get_swap_device (entry); +if (!si) +return -EAGAIN ; +if (IS_ERR (si)) +return PTR_ERR (si); + + src_folio =NULL ; + cache_state =swap_pmd_cache_lookup (entry, &src_folio); +if (cache_state ==SWAP_PMD_CACHE_SPLIT ) { +put_swap_device (si); +__split_huge_pmd (src_vma, src_pmd, src_addr); +return -EAGAIN ; + } + +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , + mm, src_addr, + src_addr +HPAGE_PMD_SIZE ); +mmu_notifier_invalidate_range_start (&range); + +if (src_folio) { +folio_lock (src_folio); +/* +* Do not split on failure here. The usual cause +* is that a racing fault swapped the range back +* in and dropped the folio from the swap cache, +* so src_pmd is now a healthy present THP; +* splitting it would destroy the very mapping +* UFFDIO_MOVE is trying to move whole. The +* caller's -EAGAIN retry re-reads src_pmd and +* picks the right path, exactly as +* move_swap_pte() relies on for the PTE case. +*/ +if (!folio_matches_swap_entry (src_folio, entry) || +folio_nr_pages (src_folio) !=HPAGE_PMD_NR ) { +folio_unlock (src_folio); +folio_put (src_folio); +mmu_notifier_invalidate_range_end (&range); +put_swap_device (si); +return -EAGAIN ; + } + } + + dst_ptl =pmd_lockptr (mm, dst_pmd); + err =move_swap_pmd (mm, dst_vma, dst_addr, src_addr, + dst_pmd, src_pmd, dst_pmdval, + src_pmdval, dst_ptl, src_ptl, + src_folio, entry); + +mmu_notifier_invalidate_range_end (&range); +if (src_folio) { +folio_unlock (src_folio); +folio_put (src_folio); + } +put_swap_device (si); +return err; + } +#endif /* CONFIG_THP_SWAP */ +spin_unlock (src_ptl);return -ENOENT ; }
mm/userfaultfd.c +14 -0
@@ -2053,6 +2053,20 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,break ; } +/* +* A PMD swap entry at dst is a swapped-out THP, not a hole, +* and unlike a PMD migration entry it will not resolve on its +* own. Nothing below faults it back in: pte_alloc() skips a +* !pmd_none PMD, pte_offset_map_rw_nolock() then fails on the +* non-present PMD, and the -EAGAIN that produces would be +* retried forever by the loop below. Be strict, exactly as for +* a present THP. +*/ +if (unlikely (pmd_is_swap_entry (dst_pmdval))) { + err = -EEXIST ; +break ; + } + ptl =pmd_trans_huge_lock (src_pmd, src_vma);if (ptl) {/* Check if we can move the pmd without splitting it. */
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 27/30 mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
In reply to Usama Arifdo_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at the other subpages. A poisoned subpage that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.
Skip batching for such a folio and let each page take its own fault, where the existing check catches it. Test every subpage rather than the folio-level PG_has_hwpoisoned: memory_failure() sets PageHWPoison() on the subpage before it acquires the folio lock, which we hold here.
--- mm/memory.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/memory.c +8 -1
@@ -5030,7 +5030,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_idx =0 ; address = vmf->address; ptep = vmf->pte; -if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +/* +* Scan every subpage rather than testing the folio-level +* PG_has_hwpoisoned: memory_failure() sets PageHWPoison on the subpage +* before it takes the folio lock, and we hold that lock, so the +* folio-level flag can still be clear here. +*/ +if (folio_test_large (folio) &&folio_test_swapcache (folio) && + !folio_has_hwpoisoned_subpage (folio)) {int nr =folio_nr_pages (folio);unsigned long idx =folio_page_idx (folio, page);unsigned long folio_start = address - idx *PAGE_SIZE ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 28/30 mm: handle PMD swap entry faults on swap-in
In reply to Usama ArifNothing faults a PMD swap entry back in. __handle_mm_fault() recognises a non-present PMD as device-private or a migration entry and returns 0 for anything else, so a swapped-out THP would refault forever.
do_huge_pmd_swap_page() resolves the whole mapping in one go, mirroring do_swap_page() at PMD granularity, and restores soft-dirty, uffd-wp and write permission the same way. It deliberately skips the order-0 readahead
paths: the fault already asks for the whole range, and readahead would populate per-page swap-cache state and force the entry to split before the fault could finish.
A PMD swap entry only promises HPAGE_PMD_NR consecutive slots, not that the cache still holds one folio for them. When it does not - the VMA no longer permits PMD-order THPs, the cache has been split, a slot is in zswap, the allocation or read fails, the memcg charge fails, or the cache insertion loses a race - split the entry and return 0 so the retry lands in do_swap_page(). That keeps a transient PMD-order allocation failure from becoming VM_FAULT_OOM. Only split if the PMD is still the entry we were called for, since every reason to fall back was observed without the PMD lock.
Before falling back with a locked folio, drop it from the swap cache if it has never been mapped or is not uptodate: do_swap_page() would hand an unmapped PMD-sized folio to folio_add_new_anon_rmap() as a whole while installing one PTE, and a folio that failed to read would make every later fault return VM_FAULT_SIGBUS instead of re-reading the slots. An uptodate anon folio stays cached so a poisoned subpage stays visible. This mirrors unuse_pmd_entry().
Refuse to map a folio with a poisoned subpage so do_swap_page() can report VM_FAULT_HWPOISON for the offending page, and drop the exclusive marker when the folio is under writeback to an SWP_STABLE_WRITES backend such as zram, so a later write COWs rather than corrupting the writeback.
When the PMD ends up read-only but the fault was a write, call wp_huge_pmd() from the same handler so the COW does not need a second fault. Mask VM_FAULT_FALLBACK out of what it returns: splitting to PTE level is a normal outcome, but the bit is part of VM_FAULT_ERROR and arch fault handlers BUG() on it without an accompanying signal.
--- include/linux/huge_mm.h | 14 +++ mm/huge_memory.c | 257 ++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 42 +++++++ mm/memory.c | 40 +------ 4 files changed, 319 insertions(+), 34 deletions(-)
include/linux/huge_mm.h +14 -0
@@ -549,6 +549,15 @@ vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf);vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf); +#ifdef CONFIG_THP_SWAP +vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +#else +static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +extern struct folio *huge_zero_folio;extern unsigned long huge_zero_pfn;
@@ -747,6 +756,11 @@ static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf)return 0 ; } +static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{ +return 0 ; +} +static inline bool is_huge_zero_folio (const struct folio *folio) {return false;
mm/huge_memory.c +257 -0
269 lines not on this page · raw message
mm/internal.h +42 -0
48 lines not on this page · raw message
mm/memory.c +6 -34
@@ -4613,38 +4613,6 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)return 0 ; } -/* -* Check if we should call folio_free_swap to free the swap cache. -* folio_free_swap only frees the swap cache to release the slot if swap -* count is zero, so we don't need to check the swap count here. -*/ -static inline bool should_try_to_free_swap (struct swap_info_struct *si, -struct folio *folio, -struct vm_area_struct *vma, -bool exclusive, -unsigned int fault_flags) -{ -if (!folio_test_swapcache (folio)) -return false; -/* -* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap -* cache can help save some IO or memory overhead, but these devices -* are fast, and meanwhile, swap cache pinning the slot deferring the -* release of metadata or fragmentation is a more critical issue. -*/ -if (data_race (si->flags &SWP_SYNCHRONOUS_IO )) -return true; -if (mem_cgroup_swap_full (folio) || (vma->vm_flags &VM_LOCKED ) || -folio_test_mlocked (folio)) -return true; - -/* -* Free the swapcache only if we are the exclusive user and -* this is a write fault. -*/ -return (fault_flags &FAULT_FLAG_WRITE ) && exclusive; -} -static vm_fault_t pte_marker_clear (struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
@@ -6372,8 +6340,8 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)return VM_FAULT_FALLBACK ; } -/* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +vm_fault_t wp_huge_pmd (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;
@@ -6403,6 +6371,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) return VM_FAULT_FALLBACK; } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static vm_fault_t create_huge_pud (struct vm_fault *vmf) {
@@ -6666,6 +6635,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pmd_is_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd); +else if (IS_ENABLED (CONFIG_THP_SWAP ) && +pmd_is_swap_entry (vmf.orig_pmd)) +return do_huge_pmd_swap_page (&vmf);return 0 ; }if (pmd_trans_huge (vmf.orig_pmd)) {
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 29/30 mm: install PMD swap entries on swap-out
In reply to Usama ArifWhen reclaim swaps out a PMD-mapped anonymous THP it first splits the PMD via TTU_SPLIT_HUGE_PMD. The huge mapping is lost for the whole swap
round-trip: swapping the range back in takes HPAGE_PMD_NR faults and leaves as many small mappings, and the process waits for khugepaged to collapse them again.
The PMD does not have to be split. A contiguous run of slots was already secured when the folio was added to the swap cache - a non-contiguous allocation would have split the folio first - so the whole mapping can be replaced by one PMD-level swap entry encoding the first slot.
shrink_folio_list() therefore stops asking for TTU_SPLIT_HUGE_PMD for a PMD-mappable folio already in the swap cache, and try_to_unmap_one() grows a PMD branch. TTU_SPLIT_HUGE_PMD remains the fallback for everything else.
set_pmd_swap_entry() is deliberately close in shape to set_pmd_migration_entry(): invalidate the mapping while keeping the original for rollback, take a swap reference on every slot, transfer the exclusive state, propagate the dirty bit to the folio so writeback is not lost, add the mm to mmlist before the entry becomes visible, and carry over soft-dirty and uffd-wp. Any step that can fail restores the mapping first.
The entry encodes exactly what the PTE entries would, so swap_map accounting is unchanged: each slot carries a count of one, released individually on a later split or together on swap-in.
zswap needs no handling here. It stores the folio as order-0 entries, and the PMD-order swap-in users split and fall back to PTEs if any covered slot turns out to have a zswap entry.
thp_swpout_pmd counts PMD mappings replaced this way. Unlike thp_swpout it counts mappings rather than folios, so a fork-shared THP can increment it once per mapping.
--- Documentation/admin-guide/mm/transhuge.rst | 5 ++ include/linux/huge_mm.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 84 ++++++++++++++++++++++ mm/rmap.c | 19 +++++ mm/vmscan.c | 9 ++- mm/vmstat.c | 1 + 7 files changed, 120 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/transhuge.rst +5 -0
@@ -632,6 +632,11 @@ thp_swpout is incremented every time a huge page is swapout in one piece without splitting. +thp_swpout_pmd + is incremented every time a PMD mapping is replaced by a PMD-level + swap entry. A fork-shared THP can increment this counter once for each + PMD mapping that is swapped out. + thp_swpout_fallback is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space
include/linux/huge_mm.h +2 -0
@@ -551,6 +551,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);#ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio);#else static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) {
include/linux/vm_event_item.h +1 -0
@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_ZERO_PAGE_ALLOC_FAILED ,THP_SWPOUT ,THP_SWPOUT_FALLBACK , +THP_SWPOUT_PMD , #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE,
mm/huge_memory.c +84 -0
87 lines not on this page · raw message
mm/rmap.c +19 -0
@@ -2284,6 +2284,25 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_abort; } +#ifdef CONFIG_THP_SWAP +/* +* If the folio is in the swap cache and we're not +* asked to split, install a PMD-level swap entry. +*/ +if (!(flags &TTU_SPLIT_HUGE_PMD ) && +folio_test_anon (folio) && +folio_test_swapcache (folio)) { +if (set_pmd_swap_entry (&pvmw, folio)) + goto walk_abort; + +add_mm_counter (mm,MM_ANONPAGES , + -HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , +HPAGE_PMD_NR ); + goto walk_done; + } +#endif +if (flags &TTU_SPLIT_HUGE_PMD ) {/* * We temporarily have to drop the PTL and
mm/vmscan.c +8 -1
@@ -1408,7 +1408,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio); -if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the +* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio))) flags |=TTU_SPLIT_HUGE_PMD ;/* * Without TTU_SYNC, try_to_unmap will only begin to
mm/vmstat.c +1 -0
@@ -1435,6 +1435,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I (THP_SWPOUT_PMD )] ="thp_swpout_pmd" , #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v8 30/30 selftests/mm: add PMD swap entry tests
In reply to Usama ArifEach test gets a fresh PMD-mapped THP from fixture setup, fills it with a pattern that differs between base pages so a reordered split is detectable, swaps it out with MADV_PAGEOUT, and verifies thp_swpout_pmd increased before the test body runs.
The tests are basic, swapin_sync, fork, fork_cow, write, rwp_swapin, munmap, mprotect, split_mprotect, split_munmap, uffdio_move, mremap, pagemap, mincore, madvise_free, madvise_willneed and swapoff.
MADV_PAGEOUT leaves the folio in the swap cache on an asynchronous device, so a fault would only remap it and never reach swapin_sync(). Fixture setup therefore asks the current cgroup to reclaim afterwards, and uses mincore()
- which reports a PMD swap entry as resident exactly while the cache holds it - to tell whether that worked. Tests run either way; swapin_sync skips if the cache survived. The swapoff test runs only when PMD_SWAP_DEVICE is the sole active swap device, and restores it at its original priority.
Skip rather than fail where the kernel never promised anything. thp_swpout_pmd and thp_swpout_fallback are system-wide, so test the fallback counter first: if reclaim split any THP we cannot be sure it was not ours. That also covers a kernel without CONFIG_THP_SWAP. A kernel with no thp_swpout_pmd counter does not implement the feature at all.
With zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so.
Add a /proc/vmstat field reader to vm_util.c so other tests can use it too, and register pmd_swap with run_vmtests.sh and the default runner.
--- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 6 files changed, 1025 insertions(+) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c
tools/testing/selftests/mm/Makefile +2 -0
@@ -104,6 +104,7 @@ TEST_GEN_FILES += guard-regions TEST_GEN_FILES += merge TEST_GEN_FILES += rmap TEST_GEN_FILES += folio_split_race_test +TEST_GEN_FILES += pmd_swap TEST_GEN_FILES += soft-dirty ifeq ($(ARCH),x86_64)
@@ -162,6 +163,7 @@ TEST_PROGS += ksft_mremap.sh TEST_PROGS += ksft_pagemap.sh TEST_PROGS += ksft_pfnmap.sh TEST_PROGS += ksft_pkey.sh +TEST_PROGS += ksft_pmd_swap.sh TEST_PROGS += ksft_process_madv.sh TEST_PROGS += ksft_process_mrelease.sh TEST_PROGS += ksft_rmap.sh
tools/testing/selftests/mm/ksft_pmd_swap.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0 + +./run_vmtests.sh -t pmd_swap
tools/testing/selftests/mm/pmd_swap.c new file +989 -0
989 lines not on this page · raw message
tools/testing/selftests/mm/run_vmtests.sh +4 -0
@@ -69,6 +69,8 @@ separated by spaces: test pagemap_scan IOCTL - pfnmap tests for VM_PFNMAP handling +- pmd_swap + tests for PMD-level swap entries - process_madv test for process_madv - cow
@@ -414,6 +416,8 @@ CATEGORY="pagemap" run_test ./pagemap_ioctl CATEGORY="pfnmap" run_test ./pfnmap +CATEGORY="pmd_swap" run_test ./pmd_swap + # COW tests CATEGORY="cow" run_test ./cow
tools/testing/selftests/mm/vm_util.c +24 -0
@@ -180,6 +180,30 @@ uint64_t read_pmd_pagesize(void)return strtoul (buf,NULL ,10 ); } +long read_vmstat (const char *name) +{ +char line[MAX_LINE_LENGTH ]; +size_t name_len =strlen (name); +long val =-1 ; +FILE *f; + + f =fopen ("/proc/vmstat" ,"r" ); +if (!f) +return -1 ; +/* +* Match the whole field name: "thp_swpout" is a prefix of +* "thp_swpout_fallback" and "thp_swpout_pmd". +*/ +while (fgets (line,sizeof (line), f)) { +if (!strncmp (line, name, name_len) && line[name_len] ==' ' ) { + val =strtol (line + name_len +1 ,NULL ,10 ); +break ; + } + } +fclose (f); +return val; +} +unsigned long rss_anon (void ) {unsigned long rss_anon =0 ;
tools/testing/selftests/mm/vm_util.h +2 -0
@@ -89,6 +89,8 @@ unsigned long pagemap_get_pfn(int fd, char *start);void clear_softdirty (void );bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void ); +/* Returns the counter value, or -1 if /proc/vmstat has no such field. */ +long read_vmstat (const char *name);unsigned long rss_anon (void );bool check_huge_anon (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);bool check_huge_file (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.53.0-Meta
Huacai Chen
maintainerRe: [PATCH v8 03/30] loongarch: mm: add PMD swap-exclusive helpers
In reply to Usama ArifHi, Usama,
Quoted 31 lines · Usama Arif
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.A LoongArch swap PMD is the swap PTE value plus
_PAGE_HUGE, and_PAGE_SWP_EXCLUSIVEsits outside both the type and the offset field, so the PMD helpers can use the same bit.Cc Huacai Chen <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by David Hildenbrand (Arm) <[email protected]>--- arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)arch/loongarch/include/asm/pgtable.h +7
-0@@ -351,6 +351,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SWP_EXCLUSIVE; +return pmd; +} +
arch/loongarch/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 370
Quoted 10 lines
arch/loongarch/include/asm/pgtable.h +10
-0+static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_SWP_EXCLUSIVE; +return pmd; +}
I think the order should be pmd_swp_exclusive, pmd_swp_mkexclusive and pmd_swp_clear_exclusive. That looks more natural.
The riscv version is already like this, but others are not.
Huacai
Quoted 8 lines
arch/loongarch/include/asm/pgtable.h +2
-1+#endif +#define pte_none (pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_no_exec (pte) (pte_val(pte) & _PAGE_NO_EXEC) --2.53.0-Meta
David Hildenbrand (Arm)
maintainerQuoted 432 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into
HPAGE_PMD_NRPTE-level swap entries viaTTU_SPLIT_HUGE_PMDbefore unmap. This series introduces a PMD-level swap entry so the huge mapping can survive the swap round-trip anddo_huge_pmd_swap_page()can restore the PMD mapping directly on swap-in, without waiting for khugepaged to collapse the range later.The PMD swap entry is a compact page-table encoding for
HPAGE_PMD_NRconsecutive swap slots.swap_mapaccounting remains per-slot and is unchanged. Importantly, a PMD swap entry does not promise that the swap cache always contains one PMD-sized folio. While the cache is empty or contains one PMD-sized folio, PMD-level handling can proceed. Once the cache has split/per-slot state, users either inspect the individual slots directly (mincore) or split the PMD swap entry and retry through the PTE path (fault, swapoff,MADV_WILLNEED,UFFDIO_MOVE).MADV_FREEdoes not consult the cache at all: it frees a whole PMD swap entry in place and only splits when the advised range covers part of the PMD. Likewise, if any slot is still backed by zswap's per-page store, PMD-order swap-in consumers split and let the PTE path load the range page by page; an all-on-disk range can still be read back as one PMD-sized folio.The series is ordered so every consumer can handle PMD swap entries before the swap-out producer starts installing them. The swap-out patch is the last functional change.
Performance:
Measured with vm-scalability's case-swap-w-seq benchmark [1]. Four pinned workers repeatedly write a 6 GiB anonymous working set on a 4 vCPU / 4 GiB guest, forcing the set out to swap and back in. Swap is an 8 GiB NOCOW raw virtio device (cache=none, aio=native), zswap is disabled, and THP enabled/defrag are both "always". The numbers below are medians of five interleaved runs per kernel after one warm-up run:
Metric Baseline Patched Change Aggregate throughput 584.3 MiB/s 2408.9 MiB/s +312.2% (4.12x) Elapsed time 85.43 s 20.72 s -75.7% Major faults 1,814,699 228,466 -87.4% Swap I/O rate 1.02 GiB/s 4.10 GiB/s +303.9%This is a swap-intensive synthetic workload, so it mostly demonstrates the reduction in swap-fault and page-table overhead from preserving PMD mappings. I don't want to use sythetic workloads to show benefits of the series. IMHO, the main advantage comes from long-running workloads I expect the bigger win to come from fewer TLB misses, less khugepaged work, and less kernel churn from larger folios, although I have not found a benchmark that captures that well. PMD swap entries also move us closer to eliminating page-table deposits for anonymous THPs, which would provide memory savings.
Sashiko reviews on intermediate patches:
Because the swap-out producer is the last functional patch, the consumer code added by the patches before it is unreachable at the point it is introduced. Every previous sashiko review has reported that code as broken on the basis of a state it cannot yet be in; the series is ordered this way deliberately. See [2].
Notes on zswap:
Native PMD-order zswap load/store is intentionally left for a follow-up. Alexandre Ghiti is currently working on this.
This series can still preserve PMD swap entries while zswap is enabled:
zswap stores the THP as order-0 entries, and PMD-order swap-in consumers split any range that has zswap entries before reading it. If zswap has written the whole range back to disk, or the swap cache still contains one PMD-sized folio, PMD-level handling can proceed.Testing:
The 17
pmd_swapselftests pass onx86_64with zswap both disabled and enabled.PMD_SWAP_DEVICEwas the sole active swap device, so the swapoff test ran in both configurations. Note that with zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so in the log; the zswap-disabled run is the one that proves PMD restoration.hmm-tests was run with
CONFIG_DEBUG_VM=yand panic_on_warn=1, both on this series and on the base commit: identical results either way (pass:35 fail:3 skip:40). The three failures areO_TMPFILEon the test VM's 9p /tmp, not kernel behaviour, and the THP paths (migrate_anon_huge_*,migrate_partial_unmap_fault,benchmark_thp_migration) pass.[1] https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
[2] https://lore.kernel.org/all/[email protected]/v7 -> v8: https://lore.kernel.org/all/[email protected]/
- Cover letter: add the case-swap-w-seq performance results. (Andrew Morton)
- Patches 2-8: reword the opening paragraph as "Prepare for ..." instead of referring to a later patch. (David Hildenbrand)
- Patch 4 (powerpc): move the PMD exclusive helpers next to the soft-dirty PMD helpers and express them the same way, as pte_pmd()/pmd_pte() wrappers over the PTE helpers. (David Hildenbrand)
- Patch 6 (s390): drop the comment above the helpers, keep the explanation in the RSTE swap layout above__SWP_OFFSET_MASK_RSTE, and move the helpers into the existingCONFIG_ARCH_HAS_PMD_SOFTLEAVESblock abovepmd_swp_soft_dirty(). (David Hildenbrand)
- Patch 7 (x86): addstatic_assert(_PAGE_SWP_EXCLUSIVE != _PAGE_PSE)so a 32-bit build that ever selectsARCH_HAS_PMD_SOFTLEAVESfails to compile rather than producingpmd_present()swap entries. (Kiryl Shutsemau)
- Patch 10: rename the flag toto_migration_entries, use two-tab continuation indentation, reword the split comment, turn the comment onsplit_pmd_to_migration_entries()into kerneldoc, and fold thetry_to_migrate_one()call onto one line. Also drop that helper'spmd_trans_huge()||pmd_is_valid_softleaf()test, which was both too loose and silently skipped the split, in favour of aVM_WARN_ON_ONCE()at the top of__split_huge_pmd_locked()asserting thatto_migration_entriesimplies a present or device-private PMD. (David Hildenbrand)
- Patch 11: drop the VM_WARN_ON_ONCE()/force and its comment from the swap decode arm - 10/29 now asserts theto_migration_entriescontract at the top of__split_huge_pmd_locked()instead - and build the replacement PTEs by advancingpte_next_swp_offset()rather than rebuilding each entry, which hoists the soft-dirty/uffd/exclusive tests out of the loop. (David Hildenbrand).
- Patch 12 (new): split the swap-side changes out of the fork patch, so the swap-entry range duplication gets its own patch for the swap maintainers. Keeps the single-slot names as inline wrappers so existing callers are untouched. (David Hildenbrand)
- Patch 13 (fork): report a failed swap dup as-EIOand letcopy_pmd_range()own theGFP_KERNELretry, as the PTE path does, dropping the open-coded retry loop; move the mm counter update into each entry-type arm. (David Hildenbrand)
- Patch 16 (smaps):smaps_account_swap()takesnr_pagesrather than a byte size, and the local is called swapcount. (David Hildenbrand)
- No other functional change. All twelve sashiko findings on v7 were analysed and none are defects of this series.v6 -> v7: https://lore.kernel.org/all/[email protected]/
- Rebase onto akpm/mm-new atbaa8de2f3448and adapt to the newget_swap_device()contract andlinear_anon_page_index(). The series grows from 12 to 29 patches.
- Patch 1: code unchanged; reword the commit message and collect review tags.
- Patches 2-9: split the six architecture helpers from generic detection and adddebug_vm_pgtablecoverage. Use the s390 RSTE exclusive bit, make the x86 helper return bool, clear all PMD swap overlay bits before softleaf decoding, and add the architecture maintainers to Cc.
- Patches 10-11: add a preparatory no-functional-change cleanup of the migration-splitting API and keep the PMD swap split separate.
- Patch 12: reject a multi-slot duplication range that crosses a swap-cluster boundary.
- Patch 13: drop thezswap_load()change, now upstream as1a904e0d3c43, and check for zswap after swap-cache insertion while all slots are pinned.
- Patch 14: scan every subpage for hardware poison, discard a failed or newly allocated unmapped PMD-sized folio before PTE fallback, and recheck the PMD before splitting it.
- Patches 15-23: split the non-present PMD walkers by subsystem. Add the guard-advice patch so MADV_GUARD_INSTALL/REMOVE leaves PMD swap entries whole; the other split patches preserve v6 behaviour.
- Patch 24: honour the current THP policy, discard a failed PMD-sized folio, and recheck the PMD before PTE fallback.
- Patch 25: do not split after cached-folio revalidation loses a race; retry so a restored present THP remains whole.
- Patches 26-27: separate the independent PTE-batching hardware-poison fix, scan subpages directly, and discard failed or never-mapped PMD-sized folios before PTE retry instead of making the whole range fail with SIGBUS.
- Patch 28: keep the normal swap-out path unchanged, but make failed producer preconditions warn and return-EBUSYrather than BUG or return-EINVAL.
- Patch 29: grow the selftests from 16 to 17, adding swapin_sync/cache-residency coverage and more robust feature, fallback, privilege, data-integrity, and swap-device-priority handling.v5 -> v6: https://lore.kernel.org/all/[email protected]/
- Add patch 1 to renamepmd_to_softleaf_folio()topmd_softleaf_to_folio(). No functional change. (Dev Jain)
- Patch 2: warn whenpmd_softleaf_to_folio()is given a non-PFN softleaf rather than silently returning NULL. (Dev Jain)
- Patch 4: bound the fork extend-table fallback to one retry, re-read the PMD under its lock, normalize unrecoverablecopy_huge_pmd()errors to-ENOMEMsocopy_pmd_range()cannot clear and leak the source swap PMD, and drop a redundantthp_migration_supported()gate.
- Patch 5: check multi-page swap-cache insertions for zswap-backed slots in__swap_cache_add_check()under the cluster lock, both before allocation and before insertion, and reject mixed zswap/disk state with-EBUSY. (Yosry Ahmed, Nhat Pham)
- Patch 6: on a failed non-uptodate PMD-order read, remove the large folio from swap cache before splitting so order-0
fallback retries individual slots rather than poisoning the whole
2 MiB range; retain hardware-poisoned folios for per-subpage handling.
- Patch 7: make HMM snapshot mode report a PMD swap entry as non-resident, matching PTE swap entries, rather thanHMM_PFN_ERROR. Drop redundantthp_migration_supported()gates and simplify non-present PMD handling.
- Patch 8: factor PMDMADV_WILLNEEDprefetch intoswapin_pmd_swap_entry(), split and retry through PTEs after any PMD-order swapin failure, and replace the racyfolio_test_locked()plusfolio_lock()sequence withfolio_trylock().
- Patch 9: guard PMD-swapUFFDIO_MOVEcode withCONFIG_THP_SWAP, clarify RWP marker propagation, and reject a PMD swap entry at the destination with-EEXISTsoUFFDIO_MOVEcannot loop forever on-EAGAIN.
- Patch 10: honor current THP/VMA policy before PMD-order swap-in, recheck that the PMD is still the original swap entry before splitting for PTE fallback, and provide theCONFIG_TRANSPARENT_HUGEPAGEwp_huge_pmd()declaration/stub needed by THP=n builds.
- Patch 11: make an invalidset_pmd_swap_entry()walk context warn and return-EINVALinstead of falsely reporting success and corrupting the MM_ANONPAGES/MM_SWAPENTS accounting, and add an exact PMD-size folio precondition check. (Luiz Capitulino)
- Patch 12: use/proc/swapsfor prerequisite detection, checkMADV_HUGEPAGE, and distinguish an environment that cannot allocate a PMD THP (SKIP) from a swap-out validation failure (FAIL). Add partial-mprotect and partial-munmap split coverage. Strengthen munmap/MADV_FREE VmSwap accounting, pagemap slot-offset checks, and mprotect/mremap swapped-state checks; force mremap to move, check munmap()'s return, and mark theUFFDIO_MOVEdestinationMADV_HUGEPAGEbefore asserting PMD restoration. Move common setup and cleanup into one fixture, merge the swapoff fixture, remove the redundant cycles test, and make the data pattern differ between base pages so the split tests can detect incorrect slot ordering. Order fork-COW so the parent writes while the child still holds the untouched shared swap entry. (Luiz Capitulino)
- Clarify commit messages throughout. RetainTTU_SPLIT_HUGE_PMDafter prototyping its removal: removing it here requires an extra rmap walk and broadens the series beyond PMD swap entries. (Matthew Wilcox)
- Rebase onto akpm/mm-new from 15 August (4b65683fd25f).v4 -> v5: https://lore.kernel.org/all/[email protected]/
- Commit message improvements for almost all patches (Yosry for zswap patch)
- Patch 1: makepmd_to_softleaf_folio()reject softleaf entries that do not encode a PFN, so a PMD swap offset is never interpreted as one. PMD swap entries remain valid softleaf entries for classification. (sashiko)
- Patch 2: use the existingpmd_swp_uffd()helper and force freeze=false for PMD swap entries, which have no struct page for the migration-entry freeze path. (sashiko)
- Patch 3: document that the caller's page-table or swap-cache reference pins every source slot while a partial PMD-sized duplication is rolled back. Keep the pre-existing PTE fork retry behavior outside this series. (sashiko)
- Patch 5: split to the PTE path rather than mapping a PMD-sized folio containing a hardware-poisoned subpage, and restorePAGE_NONEwhen swapoff restores a UFFD marker in an RWP VMA. (sashiko)
- Patch 6: account SwapPss for a PMD swap entry one slot at a time because the slots can have different swap reference counts.
- Patch 7: if PMD-orderMADV_WILLNEEDencounters newly populated per-page zswap state, revalidate and remove the failed clean PMD-sized cache folio before retrying through PTEs.
- Patch 8: mark a moved PMD swap entry for UFFD when theUFFDIO_MOVEdestination VMA is RWP-registered. (sashiko)
- Patch 9: restorePAGE_NONEfor UFFD RWP swap-in, preserve the original write-fault state through swap-slot release and COW handling, remove the unnecessary LRU drain, and prevent PTE batching from mapping a poisoned subpage. (sashiko)
- Patch 10: add and documentthp_swpout_pmd, which counts PMD mappings replaced by PMD-level swap entries rather than swapped folios.
- Patch 11: registerpmd_swapwith the default mm selftest runner, preserve errno acrossUFFDIO_MOVEcleanup, check swapoff residency before the first memory access, add a parent-side write and verification to the fork+COW test, and add RWP regression coverage for swap-in,UFFDIO_MOVE, and swapoff. (sashiko)
- Keepdo_huge_pmd_swap_page()in patch 9. Patches 6 and 7 only add consumers; patch 10 remains the first producer, so no PMD swap entry can reach those paths before the fault handler is present. (sashiko)
- Rebase onto latest akpm/mm-new from 22 July (5e0603ba185a)v3 -> v4: https://lore.kernel.org/all/[email protected]/
- Patch 1: guard the new arch-specificpmd_swp_mkexclusive/pmd_swp_exclusive/pmd_swp_clear_exclusivehelpers on arm64, loongarch, powerpc, riscv, s390, and x86 withCONFIG_ARCH_HAS_PMD_SOFTLEAVES, matching the pattern already
used forpmd_swp_soft_dirty. Also fixes the redefinition-vs- generic-fallback build errors kernel test robot reported on i386-allnoconfig-bpf and riscv-allnoconfig-bpf, and rewraps
the patch 1 commit message paragraphs to ~75 columns.
(sashiko, kernel test robot, Usama Arif)
- Patch 2: switch the trailingfolio_remove_rmap_pmd()gate in__split_huge_pmd_locked()from *pmd toold_pmd,old_pmdretains
the original present-or-non-present classification for every
branch above. (sashiko)
- Patch 3: teachswap_retry_table_alloc()(and the underlyingswap_extend_table_alloc()) to accept an nr parameter and scan
every slot in [ci_off,ci_off+ nr) before committing an extend-table allocation. (sashiko)
- Patch 4: renamezswap_range_has_entry()tozswap_is_present()so
the same helper serves both single-slot (nr=1) and range queries, and switch the implementation fromXA_STATE+xas_find()toxa_find(), which handles RCU locking and internal-retry markers itself. Rename the callers in patches 5, 7, 9. (Yosry)
- Patch 9: refuse to map a swap-cache folio indo_huge_pmd_swap_page()whenfolio_contain_hwpoisoned_page()reports a poisoned subpage; split the PMD swap entry sodo_swap_page()can returnVM_FAULT_HWPOISONper subpage instead of the PMD handler mapping
the corrupted memory as one THP. Mirrors the PageHWPoison check
the PTE swap-in path already performs. (sashiko)
- Patch 9: note explicitly in the commit message that PMD-order swap-in deliberately skips the order-0 readahead paths, order-0 readahead would populate per-page swap-cache state and force the
PMD swap entry to split before the fault could finish. (Kairui)
- Patch 10: movemm_prepare_for_swap_entries()intoset_pmd_swap_entry()betweenfolio_dup_swap()andset_pmd_at()
so this mm is on init_mm.mmlist before any swap PMD referencing slots with a non-zeroswap_mapbecomes visible. Matches the PTE swap-out ordering. (sashiko)
- rebase onto latest akpm/mm-new (61cccb8363fcc282d4ae0555b8739dd227f5ad0b)v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Clarified the PMD swap entry rule: it is a compact encoding forHPAGE_PMD_NRswap slots, not a guarantee that swap cache always has one PMD-sized folio. (Lance Yang)
- Swapoff, fault,MADV_WILLNEED, andUFFDIO_MOVEnow classify the whole PMD swap-cache range and split/retry through the PTE path for split/per-slot cache state. (Lance Yang)
- mincore handles PMD swap entries without assuming one lookup covers a split swap-cache range. (Lance Yang)
-UFFDIO_MOVErechecks allHPAGE_PMD_NRslots before moving an empty PMD swap-cache range, avoiding stale rmap metadata for per-slot cached folios.
- Added a standalone zswap prerequisite patch from Alexandre that distinguishes all-on-disk large-folio ranges from ranges with per-page zswap entries.
- Replaced the global zswap-ever-enabled policy with per-range zswap
checks: PMD swap entries can still be installed while zswap is enabled, and PMD-order swap-in consumers split when the range has per-page zswap state.
- Added a mincore selftest and updatedMADV_WILLNEEDcoverage so the test checks that the PMD swap entry remains in place until first touch. Totalpmd_swapcoverage is now 14 tests.v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Patch 1: convert two additionalsoftleaf_to_pmd()callers that landed in mm-unstable since v1 (mm/debug_vm_pgtable.c,mm/migrate_device.c) (Dev)
- Patch 2: rename helperensure_on_mmlist()tomm_prepare_for_swap_entries()to better describe its purpose
(David)
- Patch 3: dropVM_WARN_ON_ONCE(!pmd_is_migration_entry)as
Dev posted it as a separate patch.
- Patch 5 (new): movesoftleaf_to_folio()inside the device-private branch inmigrate_vma_collect_pmd(); same class of fix as patch 4 but for the migrate-device PMD walker.
- Patch 6 (new): renameCONFIG_ARCH_ENABLE_THP_MIGRATIONtoCONFIG_ARCH_HAS_PMD_SOFTLEAVESso the gate that now drives swap-entry support too is named for what it actually controls
(PMD softleaf entries), not just migration. (Dev)
- Patch 7: add the missingpmd_swp_exclusive/ mkexclusive /clear_exclusivehelpers for powerpc.
- Patches 10 and 14: use upstreamswapin_sync()(bundlesswap_cache_alloc_folio+swap_read_folio+ the-EEXISTrace
retry) instead of the bespokeswapin_alloc_pmd_folio()helper
from v1;do_swap_pageandshmem_swapin_foliouse the same
helper (Kairui)
- Patch 10: construct a stackvm_faultfor the swapoff swap-in so
the allocator can resolve a mempolicy, mirroring how the PTE
swapoff path (unuse_pte_range) already does it.
- Patch 11: extend coverage tocheck_pmd_state()in khugepaged so a swapped-out PMD-mapped THP is treated asSCAN_PMD_MAPPED(matches the existing migration-entry handling). Route PMD swap entries in thepmd_trans_huge_lock()branch ofmincore_pte_range()throughmincore_pmd_swap()so a swapped-out PMD-mapped THP isn't reported as resident.
- Patch 12 (new): handle PMD swap entries inMADV_WILLNEEDvia swapin_sync(BIT(HPAGE_PMD_ORDER)); a naive order-0 read-ahead
would force the subsequent fault to split.
- Patch 13: refuseUFFDIO_MOVEwith-EBUSYif the swap-cache folio
was split between swap-out and the move, matching
move_pages_pte()'s rejection of large folios; otherwise only one
of the 512 anon-rmaps would be re-anchored todst_vma.
- Patch 16:alloc_fill_swap_thp()now uses the existingmmap_pmd_aligned()helper so tests don't flake/skip based on VA placement; newMADV_WILLNEEDtest that watches the PMD-order
mTHP swpin counter; swapoff test restructured to use thekselftest_harnessASSERT cleanup blocks (no double swapoff, no verify-after-munmap).
- Collected Acks and ReviewsUsama Arif (30):
mm: renamepmd_to_softleaf_folio()topmd_softleaf_to_folio()
arm64: mm: add PMD swap-exclusive helpers
loongarch: mm: add PMD swap-exclusive helpers
powerpc: mm: add PMD swap-exclusive helpers
riscv: mm: add PMD swap-exclusive helpers
s390: mm: add PMD swap-exclusive helpers
x86: mm: add PMD swap-exclusive helpers
mm: recognize PMD swap entries in the softleaf layermm/debug_vm_pgtable: test PMD swap-exclusive helpers
mm: make PMD migration-entry splitting explicit
mm: split PMD swap entries into PTE swap entries
mm/swap: allow duplicating a range of swap entries
mm: handle PMD swap entries in fork path
mm: zswap: reject high-order swap cache allocations backed by zswap
mm: swap in PMD swap entries as whole THPs during swapoff fs/proc: account PMD swap entries in smaps
mm: handle soft-dirty and uffd-wp on PMD swap entries mm/hmm: fault PMD swap entries on demand
mm: free PMD swap entries inzap_huge_pmd()
mm/madvise: free PMD swap entries withMADV_FREEmm/madvise: skip PMD swap entries forMADV_COLDandMADV_PAGEOUTmm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE mm/mincore: report PMD swap-cache residency
mm/khugepaged: treat PMD swap entries as mapped THPs
mm: handle PMD swap entries inMADV_WILLNEED
mm: handle PMD swap entries inUFFDIO_MOVE
mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
mm: handle PMD swap entry faults on swap-in
mm: install PMD swap entries on swap-out
selftests/mm: add PMD swap entry testsDocumentation/admin-guide/mm/transhuge.rst | 5 + arch/arm64/include/asm/pgtable.h | 6 + arch/loongarch/include/asm/pgtable.h | 19 + arch/powerpc/include/asm/book3s/64/pgtable.h | 6 + arch/riscv/include/asm/pgtable.h | 15 + arch/s390/include/asm/pgtable.h | 20 +- arch/x86/include/asm/pgtable.h | 20 + fs/proc/task_mmu.c | 45 +- include/linux/huge_mm.h | 40 +- include/linux/leafops.h | 44 +- include/linux/pgtable.h | 17 + include/linux/swap.h | 12 +- include/linux/vm_event_item.h | 1 + include/linux/zswap.h | 6 + mm/debug_vm_pgtable.c | 40 + mm/hmm.c | 11 +- mm/huge_memory.c | 714 +++++++++++-- mm/internal.h | 58 ++ mm/khugepaged.c | 6 + mm/madvise.c | 169 +++- mm/memory.c | 65 +- mm/migrate_device.c | 7 +- mm/mincore.c | 47 +- mm/mprotect.c | 2 +- mm/rmap.c | 27 +- mm/swap.h | 30 +- mm/swap_state.c | 83 +- mm/swapfile.c | 245 ++++- mm/userfaultfd.c | 14 + mm/vmscan.c | 9 +- mm/vmstat.c | 1 + mm/zswap.c | 12 +- tools/testing/selftests/mm/Makefile | 2 +
Quoted 9 lines
tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 +++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 38 files changed, 2630 insertions(+), 191 deletions(-) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c
I was hoping that we could get this into the next merge window, but as we are approach rc6 and I didn't even manage to review all patches (shame on me), I assume this would be a quite fit.
So I assume this series is one of the things that we'll try to get in shape over the next merge window to queue it early after rc1.
Signature · 3 lines
Cheers, David
Zi Yan
reviewerQuoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into
HPAGE_PMD_NRPTE-level swap entries viaTTU_SPLIT_HUGE_PMDbefore unmap. This series introduces a PMD-level swap entry so the huge mapping can survive the swap round-trip anddo_huge_pmd_swap_page()can restore the PMD mapping directly on swap-in, without waiting for khugepaged to collapse the range later.The PMD swap entry is a compact page-table encoding for
HPAGE_PMD_NRconsecutive swap slots.swap_mapaccounting remains per-slot and is unchanged. Importantly, a PMD swap entry does not promise that the swap cache always contains one PMD-sized folio. While the cache is empty or contains one PMD-sized folio, PMD-level handling can proceed. Once the cache has split/per-slot state, users either inspect the individual slots directly (mincore) or split the PMD swap entry and retry through the PTE path (fault, swapoff,MADV_WILLNEED,UFFDIO_MOVE).MADV_FREEdoes not consult the cache at all: it frees a whole PMD swap entry in place and only splits when the advised range covers part of the PMD. Likewise, if any slot is still backed by zswap's per-page store, PMD-order swap-in consumers split and let the PTE path load the range page by page; an all-on-disk range can still be read back as one PMD-sized folio.The series is ordered so every consumer can handle PMD swap entries before the swap-out producer starts installing them. The swap-out patch is the last functional change.
<snip>
Quoted 7 lines
I was hoping that we could get this into the next merge window, but as we are approach rc6 and I didn't even manage to review all patches (shame on me), I assume this would be a quite fit.
So I assume this series is one of the things that we'll try to get in shape over the next merge window to queue it early after rc1.
I am going to review this series, since I have done with Lorenzo’s VM_SPECIAL one.
Best Regards,
Yan, Zi
Lance Yang
reviewerHey Usama,
Staring at the swap-entry bits across architectures, and RISC-V looks a bit off in that regard ...
With Svrsw60t59b and CONFIG_MEM_SOFT_DIRTY=y, setting swap soft-dirty also sets _PAGE_EXEC. A migration PMD then looks present ... so the fault handler skips the migration wait :(
And yeah, this is a pre-existing bug. I've sent a separate fix for it.
https://lore.kernel.org/linux-mm/[email protected]/
BTW, could we also add more checks to pmd_leaf_soft_dirty_tests() so we catch this kind of issue early? Setting a swap flag shouldn't make a non-present PMD present, IIUC.
---8<---
mm/debug_vm_pgtable.c +3 -1
@@ -762,7 +762,9 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_is_huge (pmd));WARN_ON (!pmd_is_valid_softleaf (pmd)); -WARN_ON (!pmd_swp_soft_dirty (pmd_swp_mksoft_dirty (pmd))); + pmd =pmd_swp_mksoft_dirty (pmd); +WARN_ON (!pmd_swp_soft_dirty (pmd)); +WARN_ON (pmd_present (pmd)); WARN_ON(pmd_swp_soft_dirty(pmd_swp_clear_soft_dirty(pmd))); }#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
--
[...]
Cheers, Lance
Lance Yang
reviewerRe: [PATCH v8 26/30] mm: handle PMD swap entries in UFFDIO_MOVE
In reply to Usama ArifHey Usama,
On Fri, Oct 02, 2026 at 02:52:40AM -0700, Usama Arif wrote:
[...]
Quoted 7 lines
mm/huge_memory.c +7
-0+static int move_swap_pmd (struct mm_struct *mm,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +pmd_t *dst_pmd,pmd_t *src_pmd, +pmd_t orig_dst_pmd,pmd_t orig_src_pmd, +spinlock_t *dst_ptl,spinlock_t *src_ptl, +struct folio *src_folio,swp_entry_t entry) +{
[...]
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3010
Quoted 4 lines
mm/huge_memory.c +4
-0+ + moved_pmd =pmdp_huge_get_and_clear (mm, src_addr, src_pmd); +if (pgtable_supports_soft_dirty ()) + moved_pmd =pmd_swp_mksoft_dirty (moved_pmd);
Shouldn't we clear the source UFFD marker here?
moved_pmd = pmd_swp_clear_uffd(moved_pmd);
just like move_swap_pte(), no?
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3014
Quoted 4 lines
mm/huge_memory.c +4
-0+/* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ +if (userfaultfd_rwp (dst_vma)) + moved_pmd =pmd_swp_mkuffd (moved_pmd); +set_pmd_at (mm, dst_addr, dst_pmd, moved_pmd);
static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma,
unsigned long dst_addr, unsigned long src_addr, pte_t *dst_pte, pte_t *src_pte,
pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval, spinlock_t *dst_ptl, spinlock_t *src_ptl, struct folio *src_folio,
struct swap_info_struct *si, swp_entry_t entry)
{
...
orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte); ... orig_src_pte = pte_swp_clear_uffd(orig_src_pte); /* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ if (userfaultfd_rwp(dst_vma)) orig_src_pte = pte_swp_mkuffd(orig_src_pte); set_pte_at(mm, dst_addr, dst_pte, orig_src_pte);
...
}
Suppose the source is an exclusive swap PMD carrying a UFFD-WP marker, and the destination is an empty hole registered for synchronous UFFD-WP without RWP. We haven't write-protected the destination, but a successful whole-PMD MOVE carries the source's UFFD-WP marker into it.
On swap-in, that marker is restored in the present PMD, leaving it write-protected. The write fault then reaches handle_userfault(vmf, VM_UFFD_WP), even though we never write-protected the destination ...
That would be quite a surprise for userspace, and could leave the writer stuck if the unexpected WP event goes unhandled :)
vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
...
bool exclusive, stable_writes, rwp_restore = false;
bool write = vmf->flags & FAULT_FLAG_WRITE;
...
pmd = folio_mk_pmd(folio, vma->vm_page_prot);
...
if (pmd_swp_uffd(vmf->orig_pmd))
pmd = pmd_mkuffd(pmd);
if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) {
pmd = pmd_modify(pmd, PAGE_NONE);
rwp_restore = true;
}
...
if (exclusive) {
if (!rwp_restore && (vma->vm_flags & VM_WRITE) &&
!userfaultfd_huge_pmd_wp(vma, pmd) &&
!pmd_needs_soft_dirty_wp(vma, pmd)) {
pmd = pmd_mkwrite(pmd, vma);
if (write)
pmd = pmd_mkdirty(pmd);
}
rmap_flags |= RMAP_EXCLUSIVE;
}
...
vmf->orig_pmd = pmd;
...
if (write && !pmd_write(pmd) && !rwp_restore) {
vm_fault_t wp_ret = wp_huge_pmd(vmf);
...
}...
}
vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
...
if (vma_is_anonymous(vma)) {
if (likely(!unshare) &&
userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) {
if (userfaultfd_wp_async(vmf->vma))
goto split;
return handle_userfault(vmf, VM_UFFD_WP);
}
return do_huge_pmd_wp_page(vmf);
}...
}
So, clearing the source UFFD-WP marker before the destination RWP check would match the PTE helper and keep the RWP re-arm :)
Am I missing something?
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3023
Quoted 8 lines
mm/huge_memory.c +8
-0+ src_pgtable =pgtable_trans_huge_withdraw (mm, src_pmd); +pgtable_trans_huge_deposit (mm, dst_pmd, src_pgtable); + +double_pt_unlock (dst_ptl, src_ptl); + return 0; +} +#endif /* CONFIG_THP_SWAP */ +
[...]
Cheers, Lance
Lance Yang
reviewerRe: [PATCH v8 28/30] mm: handle PMD swap entry faults on swap-in
In reply to Usama ArifHey Usama,
Not an expert on swap, so the below may be naive ...
On Fri, Oct 02, 2026 at 02:52:42AM -0700, Usama Arif wrote:
[...]
Quoted 229 lines
(quoted diff) +229
-0+#ifdef CONFIG_THP_SWAP +/** + * do_huge_pmd_swap_page() - Handle a fault on a PMD-level swap entry. + * @vmf: Fault context. vmf->orig_pmd contains the swap PMD. + * + * A PMD swap entry is a compact encoding for HPAGE_PMD_NR consecutive swap + * slots. If the swap cache still has one PMD-sized folio covering the range, + * map it directly at PMD level. If the range has been split into per-page + * cache state, or zswap may have per-page state for it, split the PMD swap + * entry and retry at PTE granularity. + * + * Return: VM_FAULT_* flags. + */ +vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct folio *folio; + struct page *page; + struct swap_info_struct *si; + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + softleaf_t entry; + swp_entry_t swp_entry; + pmd_t pmd; + vm_fault_t ret = 0; + bool exclusive, stable_writes, rwp_restore = false; + bool write = vmf->flags & FAULT_FLAG_WRITE; + rmap_t rmap_flags = RMAP_NONE; + enum swap_pmd_cache cache_state; + + entry = softleaf_from_pmd(vmf->orig_pmd); + if (unlikely(!softleaf_is_swap(entry))) + return 0; + + if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT, + HPAGE_PMD_ORDER)) { + __split_huge_pmd(vma, vmf->pmd, haddr); + return 0; + } + + swp_entry = entry; + + /* Prevent swapoff from happening to us. */ + si = get_swap_device(swp_entry); + if (IS_ERR_OR_NULL(si)) { + if (IS_ERR(si)) + return VM_FAULT_SIGBUS; + return 0; + } + + cache_state = swap_pmd_cache_lookup(swp_entry, &folio); + if (cache_state == SWAP_PMD_CACHE_SPLIT) + goto split_fallback; + if (!folio) { + /* + * PMD swap entries encode ordinary per-page swap slots. If any + * slot is in zswap, split and let the PTE swap path load the + * range per page. Otherwise the range is all on disk and can be + * read back as one PMD-sized folio. + */ + if (zswap_is_present(swp_entry, HPAGE_PMD_NR)) + goto split_fallback; + + folio = swapin_sync(swp_entry, GFP_HIGHUSER_MOVABLE, + BIT(HPAGE_PMD_ORDER), vmf, NULL, 0); + if (IS_ERR_OR_NULL(folio)) + goto split_fallback; + + /* Had to read from swap area: Major fault */ + ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + } + + ret |= folio_lock_or_retry(folio, vmf); + if (ret & VM_FAULT_RETRY) + goto out_release; + + /* Verify the folio is still in swap cache and matches our entry */ + if (unlikely(!folio_matches_swap_entry(folio, swp_entry))) + goto out_page; + + /* + * Folio should be PMD-sized; if not (e.g. split in swap cache), + * split the PMD swap entry and retry at PTE level. + */ + if (folio_nr_pages(folio) != HPAGE_PMD_NR) + goto unlock_split_fallback; + + /* + * A read that failed - a PMD-order zswap load that found per-page + * state, or an I/O error - leaves the folio clean and not uptodate. + * Fall back so the PTE retry reads each slot again rather than + * returning SIGBUS for the whole range. + */ + if (unlikely(!folio_test_uptodate(folio))) + goto unlock_split_fallback; + + /* + * If any subpage is hardware-poisoned, split the PMD swap entry and + * let the PTE swap-in path handle each page individually so + * do_swap_page() can return VM_FAULT_HWPOISON for the poisoned + * subpage rather than mapping the corrupted memory as one THP. + */ + if (unlikely(folio_has_hwpoisoned_subpage(folio))) + goto unlock_split_fallback; + + page = folio_page(folio, 0); + arch_swap_restore(folio_swap(swp_entry, folio), folio); + + folio_throttle_swaprate(folio, GFP_KERNEL); + + /* Lock the PMD and verify it hasn't changed */ + vmf->ptl = pmd_lock(mm, vmf->pmd); + if (unlikely(!pmd_same(vmf->orig_pmd, pmdp_get(vmf->pmd)))) { + spin_unlock(vmf->ptl); + goto out_page; + } + + exclusive = pmd_swp_exclusive(vmf->orig_pmd); + + /* + * Some swap backends (e.g. zram) don't support concurrent page + * modifications while under writeback. If we map exclusive on such + * a backend while the folio is still under writeback, the writeback + * may see partial modifications and corrupt the swap slot. Drop the + * exclusive marker and only map R/O for that case; further GUP + * references can't appear once the page is fully unmapped, so this + * is safe. + */ + /* Lockless like do_swap_page(): SWP_STABLE_WRITES never changes. */ + stable_writes = data_race(si->flags & SWP_STABLE_WRITES); + if (exclusive && folio_test_writeback(folio) && stable_writes) + exclusive = false; + + /* + * Set up the PMD mapping. Similar to do_swap_page() but at PMD level. + */ + add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + + pmd = folio_mk_pmd(folio, vma->vm_page_prot); + pmd = pmd_mkyoung(pmd); + + if (pmd_swp_soft_dirty(vmf->orig_pmd)) + pmd = pmd_mksoft_dirty(pmd); + if (pmd_swp_uffd(vmf->orig_pmd)) + pmd = pmd_mkuffd(pmd); + if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) { + pmd = pmd_modify(pmd, PAGE_NONE); + rwp_restore = true; + } + + /* + * Check exclusivity to determine if we can map writable. + */ + if (exclusive) { + if (!rwp_restore && (vma->vm_flags & VM_WRITE) && + !userfaultfd_huge_pmd_wp(vma, pmd) && + !pmd_needs_soft_dirty_wp(vma, pmd)) { + pmd = pmd_mkwrite(pmd, vma); + if (write) + pmd = pmd_mkdirty(pmd); + } + rmap_flags |= RMAP_EXCLUSIVE; + } + + flush_icache_pages(vma, page, HPAGE_PMD_NR); + + if (!folio_test_anon(folio)) + folio_add_new_anon_rmap(folio, vma, haddr, rmap_flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, haddr, rmap_flags); + + folio_put_swap(folio, NULL); + + set_pmd_at(mm, haddr, vmf->pmd, pmd); + update_mmu_cache_pmd(vma, haddr, vmf->pmd); + + /* Update orig_pmd for any follow-up wp_huge_pmd() below. */ + vmf->orig_pmd = pmd; + + /* + * Conditionally try to free up the swap cache. Do it after mapping, + * so raced page faults will likely see the folio in swap cache and + * wait on the folio lock. + */ + if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags)) + folio_free_swap(folio); + + spin_unlock(vmf->ptl); + + folio_unlock(folio); + put_swap_device(si); + + /* + * If the write fault wasn't satisfied above (folio is shared without + * exclusivity), call wp_huge_pmd() to handle COW or + * userfaultfd-wp without forcing a second fault. + * + * wp_huge_pmd() may return VM_FAULT_FALLBACK if it had to split the + * PMD; that's a normal outcome, and the natural PTE-level refault will + * complete the COW. Mask it so callers (and the arch fault handler) + * don't see VM_FAULT_FALLBACK as a fatal VM_FAULT_ERROR. + */ + if (write && !pmd_write(pmd) && !rwp_restore) { + vm_fault_t wp_ret = wp_huge_pmd(vmf); + + wp_ret &= ~VM_FAULT_FALLBACK; + ret |= wp_ret; + if (ret & VM_FAULT_ERROR) + ret &= VM_FAULT_ERROR; + } + + return ret; + +out_page: + folio_unlock(folio); +out_release: + folio_put(folio); + put_swap_device(si); + return ret; + +unlock_split_fallback: + /* + * PTE fallback cannot add a single-page rmap to a PMD-sized folio that + * has never been mapped: do_swap_page() would hand the whole folio to + * folio_add_new_anon_rmap() while installing one PTE. Nor can it do + * anything useful with a folio that failed to read. Remove either from
Review
Quoted 10 lines
(quoted diff) +10
-0+ * the swap cache so each slot is read back into its own order-0 folio. + * An uptodate anon swap-cache folio can be mapped one PTE at a time and + * must stay cached, so that any poisoned subpage stays visible to + * do_swap_page(). This mirrors unuse_pmd_entry(). + */ + if (folio_matches_swap_entry(folio, swp_entry) && + (!folio_test_uptodate(folio) || !folio_test_anon(folio))) + swap_cache_del_folio(folio); + folio_unlock(folio); + folio_put(folio);
Requesting BIT(HPAGE_PMD_ORDER) doesn't prevent swapin_sync() from returning a cached order-0 folio ...
struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
{
...
do {
folio = swap_cache_get_folio(entry);
if (folio)
return folio;
folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
} while (PTR_ERR(folio) == -EEXIST);...
}
static int zswap_writeback_entry(struct zswap_entry *entry,
swp_entry_t swpentry)
{
...
folio = __swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
...
if (IS_ERR(folio))
return PTR_ERR(folio);
...
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
goto err;
}
xa_erase(tree, offset);
...
/* folio is up to date */
folio_mark_uptodate(folio);
folio_set_dropbehind(folio);
...
folio_put(folio);
/* start writeback */
__swap_writeout(&ctx, folio);
swap_write_submit(&ctx);
return 0;...
}
void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
...
folio_start_writeback(folio);
folio_unlock(folio);
swap_add_folio(ctx, folio, WRITE);
}Say we hit the following race:
1) swap_pmd_cache_lookup() finds the range empty, before zswap writeback inserts its order-0 folio.
2) zswap writeback puts that small folio in the swap cache at the first slot, then erases the last zswap entry in the range.
3) The fault thread checks zswap and gets false from zswap_is_present(). It then calls swapin_sync(), which returns that cached small folio.
4) After __swap_writeout() unlocks the folio, the fault thread can lock it while writeback is still pending. The folio isn't PMD-sized, so we reach that cleanup ...
zswap has already dropped its allocation reference, and writeback itself doesn't hold one. So, without any other references, we're left with the swap-cache reference and our fault reference.
The folio is still !anon, so swap_cache_del_folio() removes the cache reference, then folio_put() drops ours. That can drop the last reference while I/O is still using the folio ... :(
So ... could we just unlock and put it on the size mismatch, then jump to split_fallback? That would keep the folio in the swap cache for do_swap_page().
Something like:
---8<---
mm/huge_memory.c +8 -2
@@ -2507,9 +2507,14 @@ vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)/* * Folio should be PMD-sized; if not (e.g. split in swap cache), * split the PMD swap entry and retry at PTE level. +* Keep the folio cached: zswap writeback may still be in flight and +* relies on the swap-cache reference to keep it alive. */ -if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) - goto unlock_split_fallback; +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { +folio_unlock (folio); +folio_put (folio); + goto split_fallback; + }/* * A read that failed - a PMD-order zswap load that found per-page
@@ -2647,6 +2652,7 @@ vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) unlock_split_fallback:/* +* Only PMD-sized folios reach this cleanup. * PTE fallback cannot add a single-page rmap to a PMD-sized folio that * has never been mapped: do_swap_page() would hand the whole folio to * folio_add_new_anon_rmap() while installing one PTE. Nor can it do
--
Hopefully I didn't miss anything :D
Quoted 5 lines
(quoted diff) +5
-0+split_fallback: + /* + * Only split if the PMD is still the swap entry we were called for. + * All the reasons we get here (allocation failure, zswap state, a + * split or poisoned cached folio) were observed without the PMD lock,
Review
Quoted 10 lines
(quoted diff) +10
-0+ * so a racing thread may already have swapped the range back in as a + * THP -- splitting that would silently demote a perfectly good huge + * mapping. + */ + if (pmd_same(vmf->orig_pmd, pmdp_get_lockless(vmf->pmd))) + __split_huge_pmd(vma, vmf->pmd, haddr); + put_swap_device(si); + return 0; +} +#endif /* CONFIG_THP_SWAP */
[...]
Cheers, Lance
Usama Arif
reviewerQuoted 8 lines
Quoted 8 lines
I was hoping that we could get this into the next merge window, but as we are approach rc6 and I didn't even manage to review all patches (shame on me), I assume this would be a quite fit.
So I assume this series is one of the things that we'll try to get in shape over the next merge window to queue it early after rc1.
No worries! Really do appreciate all the reviews and the feedback! I will need to learn your magic of reviewing so many different ongoing series at the same time.
Quoted 2 lines
I am going to review this series, since I have done with Lorenzo’s
VM_SPECIALone.
Thanks Zi!
Quoted 4 lines
Best Regards,
Yan, Zi
Re: [PATCH v8 04/30] powerpc: mm: add PMD swap-exclusive helpers
In reply to Usama ArifDon't forget to CC <[email protected]> when addressing powerpc architecture.
Le 02/10/2026 à 11:52, Usama Arif a écrit :
Quoted 32 lines
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.book3s64 builds a swap PMD by running the PTE encoding over
pmd_pte(), so the PMD helpers wrap the PTE ones and reuse_PAGE_SWP_EXCLUSIVE, exactly as the soft-dirty PMD helpers next to them do. It is also the only powerpc variant that selectsARCH_HAS_PMD_SOFTLEAVES, viaPPC_THP.Cc Madhavan Srinivasan <[email protected]>Signed-off-by Usama Arif <[email protected]>--- arch/powerpc/include/asm/book3s/64/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/powerpc/include/asm/book3s/64/pgtable.h +6
-0@@ -1060,6 +1060,12 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#endif #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_clear_exclusive (pmd) pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif +#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE static inline int pmd_protnone (pmd_t pmd) {
Re: [PATCH v8 07/30] x86: mm: add PMD swap-exclusive helpers
In reply to Usama ArifQuoted 27 lines · Usama Arif
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse
_PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves forPG_anon_exclusive. 32-bit x86 aliases that bit to_PAGE_PSEand does not selectARCH_HAS_PMD_SOFTLEAVES; assert that, because an exclusive swap PMD would otherwise bepmd_present().Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>Reviewed-by David Hildenbrand (Arm) <[email protected]>--- arch/x86/include/asm/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)arch/x86/include/asm/pgtable.h +0
-0
arch/x86/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 1529
Quoted 5 lines
arch/x86/include/asm/pgtable.h +2
-0@@ -1525,6 +1525,26 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_EXCLUSIVE); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +/* 32-bit aliases this to _PAGE_PSE, making an exclusive swap PMD present. */
Is it better to say? Since the above comment sounds like 32-bit specific, but the actual constraint is about not being _PAGE_PSE.
_PAGE_SWP_EXCLUSIVE cannot be _PAGE_PSE, otherwise making an exclusive swap PMD present.
Quoted 11 lines
arch/x86/include/asm/pgtable.h +11
-0+static_assert (_PAGE_SWP_EXCLUSIVE != _PAGE_PSE); + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_EXCLUSIVE; +}
arch/x86/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 1550
Quoted 10 lines
arch/x86/include/asm/pgtable.h +7
-0+ +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty (pte_t pte) {
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Yosry Ahmed
maintainerRe: [PATCH v8 14/30] mm: zswap: reject high-order swap cache allocations backed by zswap
In reply to Usama ArifQuoted 11 lines · Usama Arif
zswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Reject that in
__swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, whichzswap_load()handles per page.
Quoted 5 lines
Export
zswap_is_present()for the PMD-order swap-in users added later.Co-developed-by Alexandre Ghiti <[email protected]>Signed-off-by Alexandre Ghiti <[email protected]>Signed-off-by Usama Arif <[email protected]>
Overall looks good, but I think it can be split into 3 patches (one of which doesn't have to be part of this series), see below.
Quoted 87 lines
--- include/linux/zswap.h | 6 ++++++ mm/swap_state.c | 39 +++++++++++++++++++++++++-------------- mm/zswap.c | 12 +++++++++--- 3 files changed, 40 insertions(+), 17 deletions(-)include/linux/zswap.h +6
-0@@ -35,6 +35,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);void zswap_folio_swapin (struct folio *folio);bool zswap_is_enabled (void );bool zswap_never_enabled (void ); +bool zswap_is_present (swp_entry_t entry,unsigned int nr);#else struct zswap_lruvec_state {};@@ -73,6 +74,11 @@ static inline bool zswap_never_enabled(void)return true; } +static inline bool zswap_is_present (swp_entry_t entry,unsigned int nr) +{ +return false; +} +#endif #endif /* _LINUX_ZSWAP_H */ mm/swap_state.c +16
-14@@ -12,6 +12,7 @@#include <linux/kernel_stat.h> #include <linux/mempolicy.h> #include <linux/swap.h> +#include <linux/zswap.h> #include <linux/leafops.h> #include <linux/init.h> #include <linux/pagemap.h> @@ -466,26 +467,27 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,__swap_cache_do_add_folio (ci, folio, entry);spin_unlock (&ci->lock); +/* +* Now that the folio is in the swap cache, zswap can no longer start +* storing or writing back any slot in the range, so this is a stable +* answer. Reject a high-order allocation over a range that already +* has per-page zswap entries. +*/ +if (order &&zswap_is_present (entry, nr_pages)) { + err = -EBUSY ; + goto delete_folio; + } +if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); + err = -ENOMEM ;count_mthp_stat (order,MTHP_STAT_SWPIN_FALLBACK_CHARGE ); -return ERR_PTR (-ENOMEM ); + goto delete_folio; }if (order >1 &&folio_memcg_alloc_deferred (folio)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); -return ERR_PTR (-ENOMEM ); + err = -ENOMEM ; + goto delete_folio; }/* memsw uncharges swap when folio is added to swap cache */ @@ -499,6 +501,15 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,/* Caller will initiate read into locked new_folio */ folio_add_lru (folio);
mm/swap_state.c static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, · line 512
Quoted 10 lines
mm/swap_state.c +9
-0return folio; + +delete_folio: +spin_lock (&ci->lock); +__swap_cache_do_del_folio (ci, folio, entry, shadow); +spin_unlock (&ci->lock); +folio_unlock (folio); +/* nr_pages refs from swap cache, 1 from allocation */ +folio_put_refs (folio, nr_pages +1 ); +return ERR_PTR (err);
Nit: You can pull the refactoring into a separate patch (as well as most zswap changes, see below), so that this patch only has the logical change to bail for higher orders if any entry is in zswap.
Quoted 8 lines
mm/swap_state.c +0
-0}/** mm/zswap.c +0
-0
mm/zswap.c bool zswap_store(struct folio *folio) · line 1594
Quoted 10 lines
mm/zswap.c +5
-2@@ -1587,11 +1587,14 @@ bool zswap_store(struct folio *folio)* @entry: base swap entry of the range * @nr: number of contiguous slots to check * -* Context: The caller must keep the range pinned, otherwise the answer can -* change under it. +* Context: Unless the caller keeps the range pinned, the answer is only a +* hint: zswap can store or write back a slot right after this returns. A +* caller that needs a stable answer must pin the range first, as +* __swap_cache_alloc() does by inserting the folio into the swap cache before +* asking.
I would rather avoid naming specific functions as it will go stale quickly, and instead list the ways of "pinning" a range.
Also, generally, the changes to mm/zswap.c seem orthogonal to this patch and can be a separate patch (which would be clearer if __swap_cache_alloc() isn't named here).
Quoted 20 lines
mm/zswap.c +4
-2* Return: true if at least one slot in the range is in zswap. */ -static bool zswap_is_present (swp_entry_t entry,unsigned int nr) +bool zswap_is_present (swp_entry_t entry,unsigned int nr) {pgoff_t offset =swp_offset (entry);struct xarray *tree =swap_zswap_tree (entry);@@ -1605,6 +1608,9 @@ static bool zswap_is_present(swp_entry_t entry, unsigned int nr)*/ BUILD_BUG_ON (SWAPFILE_CLUSTER >ZSWAP_ADDRESS_SPACE_PAGES ); +if (zswap_never_enabled ()) +return false; +return xa_find (tree, &index, offset + nr -1 ,XA_PRESENT ); } --2.53.0-Meta
Re: [PATCH v8 08/30] mm: recognize PMD swap entries in the softleaf layer
In reply to Usama ArifQuoted 27 lines · Usama Arif
Reclaim splits a PMD-mapped anonymous THP into PTE-level swap entries before unmapping it, so an ordinary swap entry has never had to appear in a PMD. Prepare for installing one there instead: the softleaf layer is where every consumer decodes non-present PMDs.
Accept swap entries as valid PMD softleaves and add
pmd_is_swap_entry(). A swap entry carries no PFN, so makepmd_softleaf_to_folio()warn and return NULL rather than interpret a swap offset as a page frame number.Unlike migration and device-private entries, a PMD swap entry can also carry the swap-exclusive marker, which
softleaf_from_pmd()has to strip before decoding. Strip all three overlays unconditionally while we are
here: each clear is a plain bit clear, so testing first only buys a branch.Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>--- include/linux/leafops.h | 40 ++++++++++++++++++++++++++++------------ include/linux/pgtable.h | 17 +++++++++++++++++ 2 files changed, 45 insertions(+), 12 deletions(-)include/linux/leafops.h +0
-0
include/linux/leafops.h static inline softleaf_t softleaf_from_pmd(pmd_t pmd) · line 103
Quoted 10 lines
include/linux/leafops.h +3
-4@@ -98,10 +98,9 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)if (pmd_present (pmd) ||pmd_none (pmd))return softleaf_mk_none (); -if (pmd_swp_soft_dirty (pmd)) - pmd =pmd_swp_clear_soft_dirty (pmd); -if (pmd_swp_uffd (pmd)) - pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_soft_dirty (pmd); + pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_exclusive (pmd);
I notice there is pte_swp_clear_flags() and wonder if we want pmd_swp_clear_flags() to match it.
include/linux/leafops.h static inline bool pmd_is_migration_entry(pmd_t pmd) · line 638
Quoted 12 lines
include/linux/leafops.h +2
-2arch_entry =__pmd_to_swp_entry (pmd);/* Temporary until swp_entry_t eliminated. */ @@ -634,18 +633,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)*/ static inline bool softleaf_is_valid_pmd_entry (softleaf_t entry) { -/* Only device private, migration entries valid for PMD. */ return softleaf_is_device_private (entry) || -softleaf_is_migration (entry); +softleaf_is_migration (entry) || +softleaf_is_swap (entry);
Using softleaf_is_swap() here makes softleaf_is_valid_pmd_entry() return false positive when pmd swap entry is not possible at the moment. Adding it only after pmd swap entry is possible can close the gap. But the whole series should be applied together, so it should be fine.
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v8 09/30] mm/debug_vm_pgtable: test PMD swap-exclusive helpers
In reply to Usama ArifQuoted 4 lines · Usama Arif
An architecture that picked a PMD exclusive bit overlapping the swap type or offset field would otherwise only be caught by data corruption at runtime. Mirror
pte_swap_exclusive_tests()at PMD level.
Quoted 8 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/debug_vm_pgtable.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v8 30/30] selftests/mm: add PMD swap entry tests
In reply to Usama ArifQuoted 23 lines · Usama Arif
Each test gets a fresh PMD-mapped THP from fixture setup, fills it with a pattern that differs between base pages so a reordered split is detectable, swaps it out with
MADV_PAGEOUT, and verifiesthp_swpout_pmdincreased before the test body runs.The tests are basic,
swapin_sync, fork,fork_cow, write,rwp_swapin, munmap, mprotect,split_mprotect,split_munmap,uffdio_move, mremap, pagemap, mincore,madvise_free,madvise_willneedand swapoff.
MADV_PAGEOUTleaves the folio in the swap cache on an asynchronous device, so a fault would only remap it and never reachswapin_sync(). Fixture setup therefore asks the current cgroup to reclaim afterwards, and usesmincore()
- which reports a PMD swap entry as resident exactly while the cache holds it - to tell whether that worked. Tests run either way;swapin_syncskips if the cache survived. The swapoff test runs only whenPMD_SWAP_DEVICEis the sole active swap device, and restores it at its original priority.Skip rather than fail where the kernel never promised anything.
thp_swpout_pmdandthp_swpout_fallbackare system-wide, so test the fallback counter first: if reclaim split any THP we cannot be sure it was not ours. That also covers a kernel withoutCONFIG_THP_SWAP. A kernel with nothp_swpout_pmdcounter does not implement the feature at all.
Quoted 8 lines
With zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so.
Add a
/proc/vmstatfield reader tovm_util.cso other tests can use it too, and registerpmd_swapwithrun_vmtests.shand the default runner.Assisted-by LLMSigned-off-by Usama Arif <[email protected]>
I confirmed this should be taking the swapin_sync() path on swap-in now with the addition of the swap cache eviction code. Also, I reviewed most of this and it looks good to me so:
Quoted 1,106 lines
--- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 6 files changed, 1025 insertions(+)create mode 100755
tools/testing/selftests/mm/ksft_pmd_swap.shcreate mode 100644tools/testing/selftests/mm/pmd_swap.ctools/testing/selftests/mm/Makefile +2
-0@@ -104,6 +104,7 @@ TEST_GEN_FILES += guard-regions TEST_GEN_FILES += merge TEST_GEN_FILES += rmap TEST_GEN_FILES += folio_split_race_test +TEST_GEN_FILES += pmd_swap TEST_GEN_FILES += soft-dirty ifeq ($(ARCH),x86_64)@@ -162,6 +163,7 @@ TEST_PROGS += ksft_mremap.sh TEST_PROGS += ksft_pagemap.sh TEST_PROGS += ksft_pfnmap.sh TEST_PROGS += ksft_pkey.sh +TEST_PROGS += ksft_pmd_swap.sh TEST_PROGS += ksft_process_madv.sh TEST_PROGS += ksft_process_mrelease.sh TEST_PROGS += ksft_rmap.shtools/testing/selftests/mm/ksft_pmd_swap.sh new file +4
-0@@ -0,0 +1,4 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0 + +./run_vmtests.sh -t pmd_swaptools/testing/selftests/mm/pmd_swap.c new file +989
-0@@ -0,0 +1,989 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Test PMD-level swap entries and their users. */ +#define _GNU_SOURCE +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <sys/mman.h> +#include <sys/wait.h> +#include <fcntl.h> +#include <errno.h> +#include <stdint.h> +#include <sys/random.h> +#include <sys/stat.h> +#include <sys/swap.h> +#include <sys/syscall.h> +#include <sys/ioctl.h> +#include <poll.h> +#include <pthread.h> +#include <linux/userfaultfd.h> +#include <time.h> + +#include "kselftest_harness.h" +#include "vm_util.h" + +#define ZSWAP_ENABLED_PATH "/sys/module/zswap/parameters/enabled" + +/* pagemap: bits 0-54 hold the PFN, or type|offset for a swap entry. */ +#define PM_PFRAME_MASK ((1ULL << 55) - 1) +/* Must match MAX_SWAPFILES_SHIFT in include/linux/swap.h. */ +#define MAX_SWAPFILES_SHIFT 5 + +static bool check_swapped (int pagemap_fd,char *addr,unsigned long size) +{ +unsigned long off; + +for (off =0 ; off < size; off +=getpagesize ()) +if (!pagemap_is_swapped (pagemap_fd, addr + off)) +return false; +return true; +} + +static bool zswap_enabled (void ) +{ +char enabled =0 ; +FILE *f; + + f =fopen (ZSWAP_ENABLED_PATH ,"r" ); +if (!f) +return false; + +if (fscanf (f," %c" , &enabled) !=1 ) + enabled =0 ; +fclose (f); + +return enabled =='Y' || enabled =='y' || enabled =='1' ; +} + +static bool swap_available (unsigned long required_bytes) +{ +unsigned long required_kb = (required_bytes +1023 ) /1024 ; +unsigned long size_kb, used_kb; +char line[256 ]; +bool ret = false; +FILE *f; + + f =fopen ("/proc/swaps" ,"r" ); +if (!f) +return false; + +/* Skip the header. */ +if (!fgets (line,sizeof (line), f)) + goto out; + +while (fgets (line,sizeof (line), f)) { +if (sscanf (line,"%*s %*s %lu %lu" , &size_kb, &used_kb) ==2 && + size_kb >= used_kb && size_kb - used_kb >= required_kb) { + ret = true; +break ; + } + } + +out: +fclose (f); +return ret; +} + +static bool same_swap_device (const struct stat *a,const struct stat *b) +{ +if (S_ISBLK (a->st_mode) &&S_ISBLK (b->st_mode)) +return a->st_rdev == b->st_rdev; +return a->st_dev == b->st_dev && a->st_ino == b->st_ino; +} + +/* +* Returns true if @swap_dev is the one and only active swap device, and stores +* its /proc/swaps priority in *prio so the caller can put it back the way it +* found it. +*/ +static bool swap_device_is_only_active (const char *swap_dev,int *prio) +{ +struct stat expected, active; +char path[256 ], line[512 ]; +unsigned int nr_active =0 ; +bool matches = false; +FILE *f; + + *prio =-1 ; + +if (stat (swap_dev, &expected)) +return false; + + f =fopen ("/proc/swaps" ,"r" ); +if (!f) +return false; +if (!fgets (line,sizeof (line), f)) + goto out; + +while (fgets (line,sizeof (line), f)) { +int line_prio; + +if (sscanf (line,"%255s %*s %*s %*s %d" , path, &line_prio) !=2 ) +continue ; + nr_active++; +if (nr_active >1 ) + goto out; + matches = !stat (path, &active) && +same_swap_device (&expected, &active); +if (matches) + *prio = line_prio; + } + +out: +fclose (f); +return nr_active ==1 && matches; +} + +/* Re-enable a device swapoff()ed by this test, at its original priority. */ +static int swapon_restore (const char *swap_dev,int prio) +{ +int flags =0 ; + +if (prio >=0 ) + flags =SWAP_FLAG_PREFER | + ((prio <<SWAP_FLAG_PRIO_SHIFT ) &SWAP_FLAG_PRIO_MASK ); +return swapon (swap_dev, flags); +} + +/* Locate this task's cgroup-v2 directory. An empty relative path is the root. */ +static bool cgroup2_self_dir (char *buf,size_t len) +{ +char mnt[PATH_MAX ] ="" , type[64 ], rel[PATH_MAX ] ="" ; +char line[2 *PATH_MAX ]; +size_t rel_len; +FILE *f; + + f =fopen ("/proc/self/mounts" ,"r" ); +if (!f) +return false; +while (fgets (line,sizeof (line), f)) { +if (sscanf (line,"%*s %4095s %63s" , mnt, type) ==2 && + !strcmp (type,"cgroup2" )) + goto found_mnt; + } + mnt[0 ] ='\0' ; +found_mnt: +fclose (f); +if (!mnt[0 ]) +return false; + + f =fopen ("/proc/self/cgroup" ,"r" ); +if (!f) +return false; +while (fgets (line,sizeof (line), f)) { +if (strncmp (line,"0::" ,3 )) +continue ; + rel_len =strcspn (line +3 ,"\n" ); +if (rel_len && rel_len <sizeof (rel)) { +memcpy (rel, line +3 , rel_len); + rel[rel_len] ='\0' ; + } +break ; + } +fclose (f); +if (!rel[0 ]) +return false; +if (!strcmp (rel,"/" )) + rel[0 ] ='\0' ; +return snprintf (buf, len,"%s%s/memory.reclaim" , mnt, rel) < (int )len; +} + +static bool cgroup_reclaim (unsigned long bytes) +{ +char path[PATH_MAX ], val[32 ]; +ssize_t written; +int fd, len, err; + +if (!cgroup2_self_dir (path,sizeof (path))) +return false; + + fd =open (path,O_WRONLY ); +if (fd <0 ) +return false; + len =snprintf (val,sizeof (val),"%lu" , bytes); + written =write (fd, val, len); + err = errno; +close (fd); +/* -EAGAIN means it reclaimed something but fell short, which is fine. */ +return written == len || err ==EAGAIN ; +} + +/* +* mincore() reports 1 over a PMD swap entry iff the swap cache still holds it. +* This samples the first slot only: it is a hint used to decide whether the +* swap cache was evicted, not an assertion about the whole range. +*/ +static bool swapcache_resident (char *mem) +{ +unsigned char vec[1 ]; + +if (mincore (mem,getpagesize (), vec)) +return true; +return vec[0 ] &1 ; +} + +/* +* MADV_PAGEOUT leaves the folio in the swap cache on asynchronous swap +* devices, so faulting would just remap the cached folio and never reach +* swapin_sync(). Push it out so the PMD-order swap-in path is exercised. +*/ +static bool drop_swapcache (char *mem,unsigned long pmd_size) +{ +unsigned long want = pmd_size; +int i; + +for (i =0 ; i <3 &&swapcache_resident (mem); i++, want *=4 ) +if (!cgroup_reclaim (want)) +break ; +return !swapcache_resident (mem); +} + +static unsigned int random_seed (void ) +{ +unsigned int seed; + +if (getrandom (&seed,sizeof (seed),0 ) !=sizeof (seed)) + seed = (unsigned int )time (NULL ); +return seed; +} + +static unsigned long test_page_size (void ) +{ +static unsigned long page_size; + +if (!page_size) + page_size =getpagesize (); +return page_size; +} + +/* +* Two base pages of the same PMD must never hold identical bytes, or the split +* tests cannot tell that the slots came back in the wrong order. A single byte +* cannot encode a page index on its own - HPAGE_PMD_NR is 8192 on arm64 with +* 64K pages - so spell the index out in the first two bytes of every page. +*/ +static unsigned char pattern_byte (unsigned int seed,unsigned long off) +{ +unsigned long page_size =test_page_size (); +unsigned long idx = off & (page_size -1 ); + +if (idx <2 ) +return (unsigned char )(seed + ((off / page_size) >> (idx *8 ))); + +return (unsigned char )(seed + off + (off >>8 ) + (off >>16 )); +} + +static void fill_pattern (char *buf,unsigned long size,unsigned int seed) +{ +unsigned long i; + +for (i =0 ; i < size; i++) + buf[i] = (char )pattern_byte (seed, i); +} + +static bool verify_pattern_range (char *buf,unsigned long size, +unsigned int seed,unsigned long offset) +{ +unsigned long i; + +for (i =0 ; i < size; i++) +if ((unsigned char )buf[i] !=pattern_byte (seed, offset + i)) +return false; +return true; +} + +static bool verify_pattern (char *buf,unsigned long size,unsigned int seed) +{ +return verify_pattern_range (buf, size, seed,0 ); +} + +static bool verify_zero (char *buf,unsigned long size) +{ +unsigned long i; + +for (i =0 ; i < size; i++) +if (buf[i]) +return false; +return true; +} + +/* +* mmap an anonymous PMD-aligned region of pmd_size bytes. Over-allocates +* by one PMD and trims the unaligned head/tail so the returned address is +* PMD-aligned (required for whole-PMD UFFDIO_MOVE). +*/ +static char *mmap_pmd_aligned (unsigned long pmd_size) +{ +unsigned long pad = pmd_size; +char *raw, *aligned; + + raw =mmap (NULL , pmd_size + pad,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (raw ==MAP_FAILED ) +return MAP_FAILED ; + + aligned = (char *)(((uintptr_t )raw + pmd_size -1 ) & ~(pmd_size -1 )); +if (aligned != raw) +munmap (raw, aligned - raw); +if (aligned + pmd_size != raw + pmd_size + pad) +munmap (aligned + pmd_size, + (raw + pmd_size + pad) - (aligned + pmd_size)); +return aligned; +} + +/* Per-process swapped size in bytes, from /proc/self/status VmSwap. */ +static unsigned long read_vmswap (void ) +{ +char line[256 ]; +unsigned long kb =0 ; +FILE *f; + + f =fopen ("/proc/self/status" ,"r" ); +if (!f) +return 0 ; +while (fgets (line,sizeof (line), f)) { +if (!strncmp (line,"VmSwap:" ,7 )) { + kb =strtoul (line +7 ,NULL ,10 ); +break ; + } + } +fclose (f); +return kb *1024 ; +} + +/* +* Swap the PMD range out. Returns true if a PMD swap entry was installed. +* On failure *swap_failed is set unless this environment simply cannot swap +* at PMD granularity, in which case the caller should skip rather than fail. +*/ +static bool swap_out_pmd (char *mem,unsigned long pmd_size,int pagemap_fd, +bool *swap_failed) +{ +long pmd_before =read_vmstat ("thp_swpout_pmd" ); +long fallback_before =read_vmstat ("thp_swpout_fallback" ); +long pmd_after, fallback_after; +bool swapped; + +/* +* A kernel without PMD swap entry support has no thp_swpout_pmd at +* all. Skip rather than report a failure the kernel never promised. +*/ +if (pmd_before <0 || fallback_before <0 ) { +ksft_print_msg ("no thp_swpout_pmd counter; PMD swap entries unsupported\n" ); +return false; + } + +if (madvise (mem, pmd_size,MADV_PAGEOUT )) { +ksft_print_msg ("MADV_PAGEOUT failed: %s\n" ,strerror (errno)); + *swap_failed = true; +return false; + } + + swapped =check_swapped (pagemap_fd, mem, pmd_size); + pmd_after =read_vmstat ("thp_swpout_pmd" ); + fallback_after =read_vmstat ("thp_swpout_fallback" ); +ksft_print_msg ("thp_swpout_pmd: %ld -> %ld, fallback: %ld -> %ld\n" , + pmd_before, pmd_after, fallback_before, fallback_after); + +if (!swapped) { +ksft_print_msg ("MADV_PAGEOUT did not swap the whole PMD range\n" ); + *swap_failed = true; +return false; + } +/* +* Both counters are system-wide, so another task swapping a THP during +* the window above can move either of them. Test the fallback counter +* first: if reclaim split *any* THP we cannot be sure ours was not the +* one, and skipping is the safe direction. That also covers a kernel +* built without CONFIG_THP_SWAP, where folio_alloc_swap() returns +* -E2BIG for every PMD-order folio. +*/ +if (fallback_after > fallback_before) { +ksft_print_msg ("PMD swap unavailable; reclaim used PTE fallback\n" ); +return false; + } +if (pmd_after > pmd_before) +return true; + + *swap_failed = true; +return false; +} + +static char *alloc_fill_swap_thp (unsigned long pmd_size,int pagemap_fd, +unsigned int seed,bool *swap_failed) +{ +char *mem; + + *swap_failed = false; + + mem =mmap_pmd_aligned (pmd_size); +if (mem ==MAP_FAILED ) +return MAP_FAILED ; + +if (madvise (mem, pmd_size,MADV_HUGEPAGE )) { +ksft_print_msg ("MADV_HUGEPAGE failed: %s\n" ,strerror (errno)); +munmap (mem, pmd_size); +return MAP_FAILED ; + } +fill_pattern (mem, pmd_size, seed); + +if (!check_huge_anon (mem, pmd_size,1 , pmd_size)) { +munmap (mem, pmd_size); +return MAP_FAILED ; + } +if (!swap_out_pmd (mem, pmd_size, pagemap_fd, swap_failed)) { +munmap (mem, pmd_size); +return MAP_FAILED ; + } + +return mem; +} + +struct rwp_access_args { +unsigned char *addr; +unsigned char expected; +bool write; +bool ok; +}; + +static void *rwp_access_thread (void *data) +{ +struct rwp_access_args *args = data; + +if (args->write) + *args->addr = args->expected; + args->ok = *args->addr == args->expected; +return NULL ; +} + +static int register_rwp (char *addr,unsigned long size,bool protect) +{ +struct uffdio_register reg = {}; +struct uffdio_rwprotect rwp = {}; +struct uffdio_api api = {}; +int uffd; + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK ); +if (uffd <0 ) +return -1 ; + + api.api =UFFD_API ; + api.features =UFFD_FEATURE_RWP ; +if (ioctl (uffd,UFFDIO_API , &api) || + !(api.features &UFFD_FEATURE_RWP )) + goto error; + + reg.range.start = (unsigned long )addr; + reg.range.len = size; + reg.mode =UFFDIO_REGISTER_MODE_RWP ; +if (ioctl (uffd,UFFDIO_REGISTER , ®)) + goto error; + +if (!protect) +return uffd; + + rwp.range.start = (unsigned long )addr; + rwp.range.len = size; + rwp.mode =UFFDIO_RWPROTECT_MODE_RWP ; +if (!ioctl (uffd,UFFDIO_RWPROTECT , &rwp)) +return uffd; + +error: +close (uffd); +return -1 ; +} + +static bool expect_rwp_fault (int uffd,char *addr,unsigned long size, +unsigned char expected,bool write) +{ +struct rwp_access_args args = { + .addr = (unsigned char *)addr, + .expected = expected, + .write = write, + }; +struct uffdio_rwprotect rwp = { + .range = { + .start = (unsigned long )addr, + .len = size, + }, + }; +struct pollfd pollfd = { + .fd = uffd, + .events =POLLIN , + }; +struct uffd_msg msg = {}; +pthread_t thread; +bool saw_rwp = false; +int ret; + +if (pthread_create (&thread,NULL , rwp_access_thread, &args)) +return false; + + ret =poll (&pollfd,1 ,5000 ); +if (ret ==1 && (pollfd.revents &POLLIN ) && +read (uffd, &msg,sizeof (msg)) == (ssize_t )sizeof (msg)) { + saw_rwp = msg.event ==UFFD_EVENT_PAGEFAULT && + (msg.arg.pagefault.flags &UFFD_PAGEFAULT_FLAG_RWP ); + } + +/* Resolve the access even on failure so the worker cannot remain blocked. */ +ioctl (uffd,UFFDIO_RWPROTECT , &rwp); +if (pthread_join (thread,NULL )) +return false; +return saw_rwp && args.ok; +} + +FIXTURE (pmd_swap) +{ +unsigned long pmd_size; +unsigned long mem_len; +int pagemap_fd; +int uffd; +unsigned int seed; +bool zswap_enabled; +bool swap_disabled; +int swap_prio; +bool swapcache_dropped; +const char *swap_dev; +char *mem; +char *aux; +}; + +FIXTURE_SETUP (pmd_swap) +{ +bool swap_failed; + + self->pagemap_fd =-1 ; + self->uffd =-1 ; + self->mem =MAP_FAILED ; + self->aux =MAP_FAILED ; + self->mem_len =0 ; + self->swap_disabled = false; + self->swap_prio =-1 ; + self->swapcache_dropped = false; + self->swap_dev =NULL ; + + self->pmd_size =read_pmd_pagesize (); +if (!self->pmd_size) +SKIP (return,"Cannot determine PMD size\n" ); + + self->pagemap_fd =open ("/proc/self/pagemap" ,O_RDONLY ); +if (self->pagemap_fd <0 ) +SKIP (return,"Cannot open /proc/self/pagemap\n" ); + +if (!swap_available (self->pmd_size)) +SKIP (return,"No active swap device has enough free space\n" ); + + self->seed =random_seed (); + self->zswap_enabled =zswap_enabled (); + self->mem =alloc_fill_swap_thp (self->pmd_size, self->pagemap_fd, + self->seed, &swap_failed); +if (self->mem ==MAP_FAILED ) { +ASSERT_FALSE (swap_failed); +SKIP (return,"Could not create swapped THP\n" ); + } + self->mem_len = self->pmd_size; + self->swapcache_dropped =drop_swapcache (self->mem, self->pmd_size); +ksft_print_msg ("swap cache %s the swapped-out THP\n" , + self->swapcache_dropped ?"no longer holds" + :"still holds" ); +} + +FIXTURE_TEARDOWN (pmd_swap) +{ +int swap_err =0 ; +int swap_ret =0 ; + +if (self->swap_disabled) { + swap_ret =swapon_restore (self->swap_dev, self->swap_prio); + swap_err = errno; + } +if (self->uffd >=0 ) +close (self->uffd); +if (self->aux !=MAP_FAILED ) +munmap (self->aux, self->pmd_size); +if (self->mem !=MAP_FAILED ) +munmap (self->mem, self->mem_len); +if (self->pagemap_fd >=0 ) +close (self->pagemap_fd); + + EXPECT_EQ(swap_ret, 0) { +TH_LOG ("swapon(%s) failed: %s" , self->swap_dev, +strerror (swap_err)); + } +} + +TEST_F(pmd_swap, basic) +{ +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +} + +/* +* With the swap cache evicted, the fault cannot be served by remapping a +* cached folio, so this covers the PMD-order swapin_sync() read. +*/ +TEST_F(pmd_swap, swapin_sync) +{ +if (!self->swapcache_dropped) +SKIP (return,"Could not evict the folio from the swap cache\n" ); + +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->mem, self->pmd_size,1 , + self->pmd_size)); +} + +TEST_F(pmd_swap, fork) +{ +pid_t pid; +int status; + + pid =fork (); +ASSERT_GE (pid,0 ); + +if (pid ==0 ) +_exit (verify_pattern (self->mem, self->pmd_size, + self->seed) ?0 :1 ); + +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); + +ASSERT_EQ (waitpid (pid, &status,0 ), pid); +ASSERT_TRUE (WIFEXITED (status)); +ASSERT_EQ (WEXITSTATUS (status),0 ); +} + +TEST_F(pmd_swap, fork_cow) +{ +unsigned int parent_seed = self->seed; +unsigned int child_seed = ~self->seed; +unsigned int new_seed = self->seed ^0xa5a5a5a5 ; +int release_child[2 ]; +bool parent_ok; +char c =0 ; +pid_t pid; +int status, ret; + +ASSERT_EQ (pipe (release_child),0 ); + + pid =fork (); +ASSERT_GE (pid,0 ); + +if (pid ==0 ) { +close (release_child[1 ]); +if (read (release_child[0 ], &c,1 ) !=1 ) +_exit (1 ); +if (!verify_pattern (self->mem, self->pmd_size, parent_seed)) +_exit (2 ); +fill_pattern (self->mem, self->pmd_size, child_seed); +if (!verify_pattern (self->mem, self->pmd_size, child_seed)) +_exit (3 ); +_exit (0 ); + } + +close (release_child[0 ]); +fill_pattern (self->mem, self->pmd_size, new_seed); + parent_ok =verify_pattern (self->mem, self->pmd_size, new_seed); + ret =write (release_child[1 ], &c,1 ); +close (release_child[1 ]); +ASSERT_EQ (waitpid (pid, &status,0 ), pid); +ASSERT_EQ (ret,1 ); +ASSERT_TRUE (parent_ok); +ASSERT_TRUE (WIFEXITED (status)); +ASSERT_EQ (WEXITSTATUS (status),0 ); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, new_seed)); +} + +TEST_F(pmd_swap, write) +{ + self->mem[0 ] =0xbb ; +ASSERT_EQ (self->mem[0 ], (char )0xbb ); +ASSERT_TRUE (verify_pattern_range (self->mem +1 , self->pmd_size -1 , + self->seed,1 )); +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->mem, self->pmd_size,1 , + self->pmd_size)); +} + +TEST_F(pmd_swap, rwp_swapin) +{ + self->uffd =register_rwp (self->mem, self->pmd_size, true); +if (self->uffd <0 ) +SKIP (return,"Userfaultfd RWP unsupported\n" ); + +ASSERT_TRUE (expect_rwp_fault (self->uffd, self->mem, self->pmd_size, +pattern_byte (self->seed,0 ), false)); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, munmap) +{ +unsigned long swap_before, swap_after; +int ret; + + swap_before =read_vmswap (); +ASSERT_GE (swap_before, self->pmd_size); + + ret =munmap (self->mem, self->pmd_size); +if (!ret) { + self->mem =MAP_FAILED ; + self->mem_len =0 ; + } +ASSERT_EQ (ret,0 ); + + swap_after =read_vmswap (); +ASSERT_LE (swap_after, swap_before - self->pmd_size); +} + +TEST_F(pmd_swap, mprotect) +{ +ASSERT_EQ (mprotect (self->mem, self->pmd_size,PROT_READ ),0 ); +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +ASSERT_EQ (mprotect (self->mem, self->pmd_size, +PROT_READ |PROT_WRITE ),0 ); +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, split_mprotect) +{ +unsigned long half = self->pmd_size /2 ; + +ASSERT_EQ (mprotect (self->mem, half,PROT_READ ),0 ); +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +ASSERT_EQ (mprotect (self->mem, half,PROT_READ |PROT_WRITE ),0 ); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, split_munmap) +{ +unsigned long half = self->pmd_size /2 ; +unsigned long swap_before =read_vmswap (); +unsigned long i; +char *base = self->mem; +int ret; + +ASSERT_GE (swap_before, half); + ret =munmap (base, half); +if (!ret) { + self->mem = base + half; + self->mem_len = half; + } +ASSERT_EQ (ret,0 ); +ASSERT_LE (read_vmswap (), swap_before - half); + +for (i =0 ; i < half; i +=getpagesize ()) +ASSERT_TRUE (pagemap_is_swapped (self->pagemap_fd, + self->mem + i)); +ASSERT_TRUE (verify_pattern_range (self->mem, half, self->seed, half)); +} + +TEST_F(pmd_swap, uffdio_move) +{ +struct uffdio_register reg = {}; +struct uffdio_move move = {}; +struct uffdio_api api = {}; +bool rwp; + + self->aux =mmap_pmd_aligned (self->pmd_size); +if (self->aux ==MAP_FAILED ) +SKIP (return,"Could not mmap aligned dst\n" ); +ASSERT_EQ (madvise (self->aux, self->pmd_size,MADV_HUGEPAGE ),0 ); + + self->uffd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK ); +if (self->uffd <0 ) +SKIP (return,"userfaultfd unavailable\n" ); + + api.api =UFFD_API ; + api.features =UFFD_FEATURE_MOVE |UFFD_FEATURE_RWP ; +if (ioctl (self->uffd,UFFDIO_API , &api) || + !(api.features &UFFD_FEATURE_MOVE )) +SKIP (return,"UFFD_FEATURE_MOVE unsupported\n" ); + rwp = api.features &UFFD_FEATURE_RWP ; + + reg.range.start = (unsigned long )self->aux; + reg.range.len = self->pmd_size; + reg.mode =UFFDIO_REGISTER_MODE_MISSING | + (rwp ?UFFDIO_REGISTER_MODE_RWP :0 ); +ASSERT_EQ (ioctl (self->uffd,UFFDIO_REGISTER , ®),0 ); + + move.dst = (unsigned long )self->aux; + move.src = (unsigned long )self->mem; + move.len = self->pmd_size; +ASSERT_EQ (ioctl (self->uffd,UFFDIO_MOVE , &move),0 ); +ASSERT_EQ (move.move, self->pmd_size); + +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->aux, + self->pmd_size)); +if (rwp) +ASSERT_TRUE (expect_rwp_fault (self->uffd, self->aux, + self->pmd_size, +pattern_byte (self->seed,0 ), false)); +ASSERT_TRUE (verify_pattern (self->aux, self->pmd_size, self->seed)); +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->aux, self->pmd_size,1 , + self->pmd_size)); +} + +TEST_F(pmd_swap, mremap) +{ +char *new_mem, *dst; + + self->aux =mmap_pmd_aligned (self->pmd_size); +if (self->aux ==MAP_FAILED ) +SKIP (return,"Could not mmap aligned dst\n" ); + dst = self->aux; + + new_mem =mremap (self->mem, self->pmd_size, self->pmd_size, +MREMAP_MAYMOVE |MREMAP_FIXED , dst); +if (new_mem !=MAP_FAILED ) { + self->mem = new_mem; + self->aux =MAP_FAILED ; + } +ASSERT_NE (new_mem,MAP_FAILED ); +ASSERT_EQ (new_mem, dst); + +ASSERT_TRUE (check_swapped (self->pagemap_fd, new_mem, self->pmd_size)); +ASSERT_TRUE (verify_pattern (new_mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, pagemap) +{ +uint64_t entry, first; +unsigned long off; + + entry =pagemap_get_entry (self->pagemap_fd, self->mem); +ASSERT_TRUE (entry & (1ULL <<62 )); +ASSERT_FALSE (entry & (1ULL <<63 )); + first = entry &PM_PFRAME_MASK ; + +/* +* The kernel zeroes the swap type/offset payload for readers without +* CAP_SYS_ADMIN, so the slot-ordering check below would silently test +* nothing. Skip instead of passing vacuously. +*/ +if (!first) +SKIP (return,"pagemap swap offsets need CAP_SYS_ADMIN\n" ); + +for (off =getpagesize (); off < self->pmd_size; off +=getpagesize ()) { +uint64_t idx = off /getpagesize (); + + entry =pagemap_get_entry (self->pagemap_fd, self->mem + off); +ASSERT_TRUE (entry & (1ULL <<62 )); +ASSERT_FALSE (entry & (1ULL <<63 )); +ASSERT_EQ (entry &PM_PFRAME_MASK , + first + (idx <<MAX_SWAPFILES_SHIFT )); + } +} + +TEST_F(pmd_swap, mincore) +{ +unsigned long pages = self->pmd_size /getpagesize (); +unsigned char vec[pages]; +unsigned long i; + +ASSERT_EQ (mincore (self->mem, self->pmd_size, vec),0 ); +/* +* Nothing in this test faults the range, and the fixture already +* evicted the swap cache, so every slot must report not-resident. The +* other direction is not stable - reclaim can drop the cached folio +* between fixture setup and here, and a split cache reports per slot - +* so only check that mincore() answered. +*/ +if (self->swapcache_dropped) { +for (i =0 ; i < pages; i++) +ASSERT_EQ (vec[i] &1 ,0 ); + } +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +} + +TEST_F(pmd_swap, madvise_free) +{ +unsigned long swap_before =read_vmswap (); +unsigned long i; + +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +ASSERT_GE (swap_before, self->pmd_size); +ASSERT_EQ (madvise (self->mem, self->pmd_size,MADV_FREE ),0 ); +for (i =0 ; i < self->pmd_size; i +=getpagesize ()) +ASSERT_FALSE (pagemap_is_swapped (self->pagemap_fd, + self->mem + i)); +ASSERT_LE (read_vmswap (), swap_before - self->pmd_size); +ASSERT_TRUE (verify_zero (self->mem, self->pmd_size)); +} + +TEST_F(pmd_swap, madvise_willneed) +{ +ASSERT_EQ (madvise (self->mem, self->pmd_size,MADV_WILLNEED ),0 ); +ASSERT_TRUE (check_swapped (self->pagemap_fd, self->mem, + self->pmd_size)); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->mem, self->pmd_size,1 , + self->pmd_size)); +} + +TEST_F(pmd_swap, swapoff) +{ +int ret, err; + + self->swap_dev =getenv ("PMD_SWAP_DEVICE" ); +if (!self->swap_dev) +SKIP (return,"PMD_SWAP_DEVICE env var not set\n" ); +/* +* Otherwise a higher-priority device may have taken the PMD swap +* entry and swapoff() would operate on the wrong backend. +*/ +if (!swap_device_is_only_active (self->swap_dev, &self->swap_prio)) +SKIP (return,"PMD_SWAP_DEVICE must be the only active swap device\n" ); + + self->uffd =register_rwp (self->mem, self->pmd_size, true); + + ret =swapoff (self->swap_dev); + err = errno; +if (!ret) + self->swap_disabled = true; + ASSERT_EQ(ret, 0) { +TH_LOG ("swapoff(%s) failed: %s" , self->swap_dev,strerror (err)); + } + +/* +* Check residency before touching the memory. If we read +* first, a bug that left a PMD swap entry in place after swapoff +* would silently trigger do_huge_pmd_swap_page() and reinstall a +* PMD mapping, masking the regression. +*/ +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->mem, self->pmd_size,1 , + self->pmd_size)); +if (self->uffd >=0 ) +ASSERT_TRUE (expect_rwp_fault (self->uffd, self->mem, + self->pmd_size, +pattern_byte (self->seed,0 ), false)); +ASSERT_TRUE (verify_pattern (self->mem, self->pmd_size, self->seed)); + + ret =swapon_restore (self->swap_dev, self->swap_prio); + err = errno; +if (!ret) + self->swap_disabled = false; + ASSERT_EQ(ret, 0) { +TH_LOG ("swapon(%s) failed: %s" , self->swap_dev,strerror (err)); + } +} + +TEST_HARNESS_MAIN tools/testing/selftests/mm/run_vmtests.sh +4
-0@@ -69,6 +69,8 @@ separated by spaces: test pagemap_scan IOCTL - pfnmap tests for VM_PFNMAP handling +- pmd_swap + tests for PMD-level swap entries - process_madv test for process_madv - cow@@ -414,6 +416,8 @@ CATEGORY="pagemap" run_test ./pagemap_ioctl CATEGORY="pfnmap" run_test ./pfnmap +CATEGORY="pmd_swap" run_test ./pmd_swap + # COW tests CATEGORY="cow" run_test ./cowtools/testing/selftests/mm/vm_util.c +24
-0@@ -180,6 +180,30 @@ uint64_t read_pmd_pagesize(void)return strtoul (buf,NULL ,10 ); } +long read_vmstat (const char *name) +{ +char line[MAX_LINE_LENGTH ]; +size_t name_len =strlen (name); +long val =-1 ; +FILE *f; + + f =fopen ("/proc/vmstat" ,"r" ); +if (!f) +return -1 ; +/* +* Match the whole field name: "thp_swpout" is a prefix of +* "thp_swpout_fallback" and "thp_swpout_pmd". +*/ +while (fgets (line,sizeof (line), f)) { +if (!strncmp (line, name, name_len) && line[name_len] ==' ' ) { + val =strtol (line + name_len +1 ,NULL ,10 ); +break ; + } + } +fclose (f); +return val; +} +unsigned long rss_anon (void ) {unsigned long rss_anon =0 ;tools/testing/selftests/mm/vm_util.h +2
-0@@ -89,6 +89,8 @@ unsigned long pagemap_get_pfn(int fd, char *start);void clear_softdirty (void );bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void ); +/* Returns the counter value, or -1 if /proc/vmstat has no such field. */ +long read_vmstat (const char *name);unsigned long rss_anon (void );bool check_huge_anon (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);bool check_huge_file (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);
Zi Yan
reviewerRe: [PATCH v8 10/30] mm: make PMD migration-entry splitting explicit
In reply to Usama ArifQuoted 6 lines · Usama Arif
__split_huge_pmd()and friends take a "freeze" boolean that every caller has to pass and almost every caller passes as false. The name says nothing about what it selects, and the one thing it does select - PTE migration entries instead of PTE mappings - is only ever wanted by the rmap migration path.
Quoted 6 lines
Rename it to
to_migration_entries, keep it private tomm/huge_memory.c, and addsplit_pmd_to_migration_entries()fortry_to_migrate_one(), the only caller that wants it.
migrate_vma_split_unmapped_folio()also passed freeze=true, but only ever runs on a PMD that is already a migration entry, which the generic helper
Is it always a PMD migration entry? Can a concurrent MADV_DONTNEED zap the PMD migration entry or split it with a partial zap? Although it does not affect the correctness of the patch.
If that concurrent zap is possible, the existing code can leak a ref, causing the folio to be not freed, since split_huge_pmd_address(freeze=true) does not drop the folio refcount when the PMD is zapped or becomes a PTE page.
We might want a separate fix for it.
Actually, Mika Penttilä's "migrate on fault for device pages"[1] also talks about the race.
[1] https://lore.kernel.org/r/[email protected]
Quoted 21 lines
expands into PTE migration entries either way. Its
folio_get()only existed to balance theput_page()that freeze=true performs, so both go.
split_pmd_to_migration_entries()is only ever handed a present or device-private PMD, so assert that in__split_huge_pmd_locked()instead of silently skipping anything else.Other than that assertion, no functional change intended.
Suggested-by David Hildenbrand (Arm) <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>--- include/linux/huge_mm.h | 21 ++++++------ mm/huge_memory.c | 74 ++++++++++++++++++++++++++--------------- mm/memory.c | 4 +-- mm/migrate_device.c | 7 +--- mm/mprotect.c | 2 +- mm/rmap.c | 8 ++--- 6 files changed, 65 insertions(+), 51 deletions(-)
<snip>
Quoted 8 lines
mm/migrate_device.c +0
-0@@ -918,12 +918,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ;
mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 924
Quoted 10 lines
mm/migrate_device.c +1
-6-/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
This part can be a separate fix for the refcount leak.
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Quoted 78 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into
HPAGE_PMD_NRPTE-level swap entries viaTTU_SPLIT_HUGE_PMDbefore unmap. This series introduces a PMD-level swap entry so the huge mapping can survive the swap round-trip anddo_huge_pmd_swap_page()can restore the PMD mapping directly on swap-in, without waiting for khugepaged to collapse the range later.The PMD swap entry is a compact page-table encoding for
HPAGE_PMD_NRconsecutive swap slots.swap_mapaccounting remains per-slot and is unchanged. Importantly, a PMD swap entry does not promise that the swap cache always contains one PMD-sized folio. While the cache is empty or contains one PMD-sized folio, PMD-level handling can proceed. Once the cache has split/per-slot state, users either inspect the individual slots directly (mincore) or split the PMD swap entry and retry through the PTE path (fault, swapoff,MADV_WILLNEED,UFFDIO_MOVE).MADV_FREEdoes not consult the cache at all: it frees a whole PMD swap entry in place and only splits when the advised range covers part of the PMD. Likewise, if any slot is still backed by zswap's per-page store, PMD-order swap-in consumers split and let the PTE path load the range page by page; an all-on-disk range can still be read back as one PMD-sized folio.The series is ordered so every consumer can handle PMD swap entries before the swap-out producer starts installing them. The swap-out patch is the last functional change.
Performance:
Measured with vm-scalability's case-swap-w-seq benchmark [1]. Four pinned workers repeatedly write a 6 GiB anonymous working set on a 4 vCPU / 4 GiB guest, forcing the set out to swap and back in. Swap is an 8 GiB NOCOW raw virtio device (cache=none, aio=native), zswap is disabled, and THP enabled/defrag are both "always". The numbers below are medians of five interleaved runs per kernel after one warm-up run:
Metric Baseline Patched Change Aggregate throughput 584.3 MiB/s 2408.9 MiB/s +312.2% (4.12x) Elapsed time 85.43 s 20.72 s -75.7% Major faults 1,814,699 228,466 -87.4% Swap I/O rate 1.02 GiB/s 4.10 GiB/s +303.9%This is a swap-intensive synthetic workload, so it mostly demonstrates the reduction in swap-fault and page-table overhead from preserving PMD mappings. I don't want to use sythetic workloads to show benefits of the series. IMHO, the main advantage comes from long-running workloads I expect the bigger win to come from fewer TLB misses, less khugepaged work, and less kernel churn from larger folios, although I have not found a benchmark that captures that well. PMD swap entries also move us closer to eliminating page-table deposits for anonymous THPs, which would provide memory savings.
Sashiko reviews on intermediate patches:
Because the swap-out producer is the last functional patch, the consumer code added by the patches before it is unreachable at the point it is introduced. Every previous sashiko review has reported that code as broken on the basis of a state it cannot yet be in; the series is ordered this way deliberately. See [2].
Notes on zswap:
Native PMD-order zswap load/store is intentionally left for a follow-up. Alexandre Ghiti is currently working on this.
This series can still preserve PMD swap entries while zswap is enabled:
zswap stores the THP as order-0 entries, and PMD-order swap-in consumers split any range that has zswap entries before reading it. If zswap has written the whole range back to disk, or the swap cache still contains one PMD-sized folio, PMD-level handling can proceed.Testing:
The 17
pmd_swapselftests pass onx86_64with zswap both disabled and enabled.PMD_SWAP_DEVICEwas the sole active swap device, so the swapoff test ran in both configurations. Note that with zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so in the log; the zswap-disabled run is the one that proves PMD restoration.
Quoted 6 lines
hmm-tests was run with
CONFIG_DEBUG_VM=yand panic_on_warn=1, both on this series and on the base commit: identical results either way (pass:35 fail:3 skip:40). The three failures areO_TMPFILEon the test VM's 9p /tmp, not kernel behaviour, and the THP paths (migrate_anon_huge_*,migrate_partial_unmap_fault,benchmark_thp_migration) pass.
I ran the MM kselftests, including pmd_swap with PMD_SWAP_DEVICE set, on x86-64 and s390 (both are KVM guests). Both passed. I may test more if I have the cycles once this reaches the MM tree (maybe running the kselftests on other archs or a stress test), but for now:
Quoted 357 lines
[1] https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
[2] https://lore.kernel.org/all/[email protected]/v7 -> v8: https://lore.kernel.org/all/[email protected]/
- Cover letter: add the case-swap-w-seq performance results. (Andrew Morton)
- Patches 2-8: reword the opening paragraph as "Prepare for ..." instead of referring to a later patch. (David Hildenbrand)
- Patch 4 (powerpc): move the PMD exclusive helpers next to the soft-dirty PMD helpers and express them the same way, as pte_pmd()/pmd_pte() wrappers over the PTE helpers. (David Hildenbrand)
- Patch 6 (s390): drop the comment above the helpers, keep the explanation in the RSTE swap layout above__SWP_OFFSET_MASK_RSTE, and move the helpers into the existingCONFIG_ARCH_HAS_PMD_SOFTLEAVESblock abovepmd_swp_soft_dirty(). (David Hildenbrand)
- Patch 7 (x86): addstatic_assert(_PAGE_SWP_EXCLUSIVE != _PAGE_PSE)so a 32-bit build that ever selectsARCH_HAS_PMD_SOFTLEAVESfails to compile rather than producingpmd_present()swap entries. (Kiryl Shutsemau)
- Patch 10: rename the flag toto_migration_entries, use two-tab continuation indentation, reword the split comment, turn the comment onsplit_pmd_to_migration_entries()into kerneldoc, and fold thetry_to_migrate_one()call onto one line. Also drop that helper'spmd_trans_huge()||pmd_is_valid_softleaf()test, which was both too loose and silently skipped the split, in favour of aVM_WARN_ON_ONCE()at the top of__split_huge_pmd_locked()asserting thatto_migration_entriesimplies a present or device-private PMD. (David Hildenbrand)
- Patch 11: drop the VM_WARN_ON_ONCE()/force and its comment from the swap decode arm - 10/29 now asserts theto_migration_entriescontract at the top of__split_huge_pmd_locked()instead - and build the replacement PTEs by advancingpte_next_swp_offset()rather than rebuilding each entry, which hoists the soft-dirty/uffd/exclusive tests out of the loop. (David Hildenbrand).
- Patch 12 (new): split the swap-side changes out of the fork patch, so the swap-entry range duplication gets its own patch for the swap maintainers. Keeps the single-slot names as inline wrappers so existing callers are untouched. (David Hildenbrand)
- Patch 13 (fork): report a failed swap dup as-EIOand letcopy_pmd_range()own theGFP_KERNELretry, as the PTE path does, dropping the open-coded retry loop; move the mm counter update into each entry-type arm. (David Hildenbrand)
- Patch 16 (smaps):smaps_account_swap()takesnr_pagesrather than a byte size, and the local is called swapcount. (David Hildenbrand)
- No other functional change. All twelve sashiko findings on v7 were analysed and none are defects of this series.v6 -> v7: https://lore.kernel.org/all/[email protected]/
- Rebase onto akpm/mm-new atbaa8de2f3448and adapt to the newget_swap_device()contract andlinear_anon_page_index(). The series grows from 12 to 29 patches.
- Patch 1: code unchanged; reword the commit message and collect review tags.
- Patches 2-9: split the six architecture helpers from generic detection and adddebug_vm_pgtablecoverage. Use the s390 RSTE exclusive bit, make the x86 helper return bool, clear all PMD swap overlay bits before softleaf decoding, and add the architecture maintainers to Cc.
- Patches 10-11: add a preparatory no-functional-change cleanup of the migration-splitting API and keep the PMD swap split separate.
- Patch 12: reject a multi-slot duplication range that crosses a swap-cluster boundary.
- Patch 13: drop thezswap_load()change, now upstream as1a904e0d3c43, and check for zswap after swap-cache insertion while all slots are pinned.
- Patch 14: scan every subpage for hardware poison, discard a failed or newly allocated unmapped PMD-sized folio before PTE fallback, and recheck the PMD before splitting it.
- Patches 15-23: split the non-present PMD walkers by subsystem. Add the guard-advice patch so MADV_GUARD_INSTALL/REMOVE leaves PMD swap entries whole; the other split patches preserve v6 behaviour.
- Patch 24: honour the current THP policy, discard a failed PMD-sized folio, and recheck the PMD before PTE fallback.
- Patch 25: do not split after cached-folio revalidation loses a race; retry so a restored present THP remains whole.
- Patches 26-27: separate the independent PTE-batching hardware-poison fix, scan subpages directly, and discard failed or never-mapped PMD-sized folios before PTE retry instead of making the whole range fail with SIGBUS.
- Patch 28: keep the normal swap-out path unchanged, but make failed producer preconditions warn and return-EBUSYrather than BUG or return-EINVAL.
- Patch 29: grow the selftests from 16 to 17, adding swapin_sync/cache-residency coverage and more robust feature, fallback, privilege, data-integrity, and swap-device-priority handling.v5 -> v6: https://lore.kernel.org/all/[email protected]/
- Add patch 1 to renamepmd_to_softleaf_folio()topmd_softleaf_to_folio(). No functional change. (Dev Jain)
- Patch 2: warn whenpmd_softleaf_to_folio()is given a non-PFN softleaf rather than silently returning NULL. (Dev Jain)
- Patch 4: bound the fork extend-table fallback to one retry, re-read the PMD under its lock, normalize unrecoverablecopy_huge_pmd()errors to-ENOMEMsocopy_pmd_range()cannot clear and leak the source swap PMD, and drop a redundantthp_migration_supported()gate.
- Patch 5: check multi-page swap-cache insertions for zswap-backed slots in__swap_cache_add_check()under the cluster lock, both before allocation and before insertion, and reject mixed zswap/disk state with-EBUSY. (Yosry Ahmed, Nhat Pham)
- Patch 6: on a failed non-uptodate PMD-order read, remove the large folio from swap cache before splitting so order-0
fallback retries individual slots rather than poisoning the whole
2 MiB range; retain hardware-poisoned folios for per-subpage handling.
- Patch 7: make HMM snapshot mode report a PMD swap entry as non-resident, matching PTE swap entries, rather thanHMM_PFN_ERROR. Drop redundantthp_migration_supported()gates and simplify non-present PMD handling.
- Patch 8: factor PMDMADV_WILLNEEDprefetch intoswapin_pmd_swap_entry(), split and retry through PTEs after any PMD-order swapin failure, and replace the racyfolio_test_locked()plusfolio_lock()sequence withfolio_trylock().
- Patch 9: guard PMD-swapUFFDIO_MOVEcode withCONFIG_THP_SWAP, clarify RWP marker propagation, and reject a PMD swap entry at the destination with-EEXISTsoUFFDIO_MOVEcannot loop forever on-EAGAIN.
- Patch 10: honor current THP/VMA policy before PMD-order swap-in, recheck that the PMD is still the original swap entry before splitting for PTE fallback, and provide theCONFIG_TRANSPARENT_HUGEPAGEwp_huge_pmd()declaration/stub needed by THP=n builds.
- Patch 11: make an invalidset_pmd_swap_entry()walk context warn and return-EINVALinstead of falsely reporting success and corrupting the MM_ANONPAGES/MM_SWAPENTS accounting, and add an exact PMD-size folio precondition check. (Luiz Capitulino)
- Patch 12: use/proc/swapsfor prerequisite detection, checkMADV_HUGEPAGE, and distinguish an environment that cannot allocate a PMD THP (SKIP) from a swap-out validation failure (FAIL). Add partial-mprotect and partial-munmap split coverage. Strengthen munmap/MADV_FREE VmSwap accounting, pagemap slot-offset checks, and mprotect/mremap swapped-state checks; force mremap to move, check munmap()'s return, and mark theUFFDIO_MOVEdestinationMADV_HUGEPAGEbefore asserting PMD restoration. Move common setup and cleanup into one fixture, merge the swapoff fixture, remove the redundant cycles test, and make the data pattern differ between base pages so the split tests can detect incorrect slot ordering. Order fork-COW so the parent writes while the child still holds the untouched shared swap entry. (Luiz Capitulino)
- Clarify commit messages throughout. RetainTTU_SPLIT_HUGE_PMDafter prototyping its removal: removing it here requires an extra rmap walk and broadens the series beyond PMD swap entries. (Matthew Wilcox)
- Rebase onto akpm/mm-new from 15 August (4b65683fd25f).v4 -> v5: https://lore.kernel.org/all/[email protected]/
- Commit message improvements for almost all patches (Yosry for zswap patch)
- Patch 1: makepmd_to_softleaf_folio()reject softleaf entries that do not encode a PFN, so a PMD swap offset is never interpreted as one. PMD swap entries remain valid softleaf entries for classification. (sashiko)
- Patch 2: use the existingpmd_swp_uffd()helper and force freeze=false for PMD swap entries, which have no struct page for the migration-entry freeze path. (sashiko)
- Patch 3: document that the caller's page-table or swap-cache reference pins every source slot while a partial PMD-sized duplication is rolled back. Keep the pre-existing PTE fork retry behavior outside this series. (sashiko)
- Patch 5: split to the PTE path rather than mapping a PMD-sized folio containing a hardware-poisoned subpage, and restorePAGE_NONEwhen swapoff restores a UFFD marker in an RWP VMA. (sashiko)
- Patch 6: account SwapPss for a PMD swap entry one slot at a time because the slots can have different swap reference counts.
- Patch 7: if PMD-orderMADV_WILLNEEDencounters newly populated per-page zswap state, revalidate and remove the failed clean PMD-sized cache folio before retrying through PTEs.
- Patch 8: mark a moved PMD swap entry for UFFD when theUFFDIO_MOVEdestination VMA is RWP-registered. (sashiko)
- Patch 9: restorePAGE_NONEfor UFFD RWP swap-in, preserve the original write-fault state through swap-slot release and COW handling, remove the unnecessary LRU drain, and prevent PTE batching from mapping a poisoned subpage. (sashiko)
- Patch 10: add and documentthp_swpout_pmd, which counts PMD mappings replaced by PMD-level swap entries rather than swapped folios.
- Patch 11: registerpmd_swapwith the default mm selftest runner, preserve errno acrossUFFDIO_MOVEcleanup, check swapoff residency before the first memory access, add a parent-side write and verification to the fork+COW test, and add RWP regression coverage for swap-in,UFFDIO_MOVE, and swapoff. (sashiko)
- Keepdo_huge_pmd_swap_page()in patch 9. Patches 6 and 7 only add consumers; patch 10 remains the first producer, so no PMD swap entry can reach those paths before the fault handler is present. (sashiko)
- Rebase onto latest akpm/mm-new from 22 July (5e0603ba185a)v3 -> v4: https://lore.kernel.org/all/[email protected]/
- Patch 1: guard the new arch-specificpmd_swp_mkexclusive/pmd_swp_exclusive/pmd_swp_clear_exclusivehelpers on arm64, loongarch, powerpc, riscv, s390, and x86 withCONFIG_ARCH_HAS_PMD_SOFTLEAVES, matching the pattern already
used forpmd_swp_soft_dirty. Also fixes the redefinition-vs- generic-fallback build errors kernel test robot reported on i386-allnoconfig-bpf and riscv-allnoconfig-bpf, and rewraps
the patch 1 commit message paragraphs to ~75 columns.
(sashiko, kernel test robot, Usama Arif)
- Patch 2: switch the trailingfolio_remove_rmap_pmd()gate in__split_huge_pmd_locked()from *pmd toold_pmd,old_pmdretains
the original present-or-non-present classification for every
branch above. (sashiko)
- Patch 3: teachswap_retry_table_alloc()(and the underlyingswap_extend_table_alloc()) to accept an nr parameter and scan
every slot in [ci_off,ci_off+ nr) before committing an extend-table allocation. (sashiko)
- Patch 4: renamezswap_range_has_entry()tozswap_is_present()so
the same helper serves both single-slot (nr=1) and range queries, and switch the implementation fromXA_STATE+xas_find()toxa_find(), which handles RCU locking and internal-retry markers itself. Rename the callers in patches 5, 7, 9. (Yosry)
- Patch 9: refuse to map a swap-cache folio indo_huge_pmd_swap_page()whenfolio_contain_hwpoisoned_page()reports a poisoned subpage; split the PMD swap entry sodo_swap_page()can returnVM_FAULT_HWPOISONper subpage instead of the PMD handler mapping the corrupted memory as one THP. Mirrors the PageHWPoison check the PTE swap-in path already performs. (sashiko)
- Patch 9: note explicitly in the commit message that PMD-order swap-in deliberately skips the order-0 readahead paths, order-0 readahead would populate per-page swap-cache state and force the PMD swap entry to split before the fault could finish. (Kairui)
- Patch 10: movemm_prepare_for_swap_entries()intoset_pmd_swap_entry()betweenfolio_dup_swap()andset_pmd_at()
so this mm is on init_mm.mmlist before any swap PMD referencing slots with a non-zeroswap_mapbecomes visible. Matches the PTE swap-out ordering. (sashiko)
- rebase onto latest akpm/mm-new (61cccb8363fcc282d4ae0555b8739dd227f5ad0b)v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Clarified the PMD swap entry rule: it is a compact encoding forHPAGE_PMD_NRswap slots, not a guarantee that swap cache always has one PMD-sized folio. (Lance Yang)
- Swapoff, fault,MADV_WILLNEED, andUFFDIO_MOVEnow classify the whole PMD swap-cache range and split/retry through the PTE path for split/per-slot cache state. (Lance Yang)
- mincore handles PMD swap entries without assuming one lookup covers a split swap-cache range. (Lance Yang)
-UFFDIO_MOVErechecks allHPAGE_PMD_NRslots before moving an empty PMD swap-cache range, avoiding stale rmap metadata for per-slot cached folios.
- Added a standalone zswap prerequisite patch from Alexandre that distinguishes all-on-disk large-folio ranges from ranges with per-page zswap entries.
- Replaced the global zswap-ever-enabled policy with per-range zswap
checks: PMD swap entries can still be installed while zswap is enabled, and PMD-order swap-in consumers split when the range has per-page zswap state.
- Added a mincore selftest and updatedMADV_WILLNEEDcoverage so the test checks that the PMD swap entry remains in place until first touch. Totalpmd_swapcoverage is now 14 tests.v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Patch 1: convert two additionalsoftleaf_to_pmd()callers that landed in mm-unstable since v1 (mm/debug_vm_pgtable.c,mm/migrate_device.c) (Dev)
- Patch 2: rename helperensure_on_mmlist()tomm_prepare_for_swap_entries()to better describe its purpose (David)
- Patch 3: dropVM_WARN_ON_ONCE(!pmd_is_migration_entry)as
Dev posted it as a separate patch.
- Patch 5 (new): movesoftleaf_to_folio()inside the device-private branch inmigrate_vma_collect_pmd(); same class of fix as patch 4 but for the migrate-device PMD walker.
- Patch 6 (new): renameCONFIG_ARCH_ENABLE_THP_MIGRATIONtoCONFIG_ARCH_HAS_PMD_SOFTLEAVESso the gate that now drives swap-entry support too is named for what it actually controls
(PMD softleaf entries), not just migration. (Dev)
- Patch 7: add the missingpmd_swp_exclusive/ mkexclusive /clear_exclusivehelpers for powerpc.
- Patches 10 and 14: use upstreamswapin_sync()(bundlesswap_cache_alloc_folio+swap_read_folio+ the-EEXISTrace
retry) instead of the bespokeswapin_alloc_pmd_folio()helper
from v1;do_swap_pageandshmem_swapin_foliouse the same
helper (Kairui)
- Patch 10: construct a stackvm_faultfor the swapoff swap-in so
the allocator can resolve a mempolicy, mirroring how the PTE swapoff path (unuse_pte_range) already does it.
- Patch 11: extend coverage tocheck_pmd_state()in khugepaged so a swapped-out PMD-mapped THP is treated asSCAN_PMD_MAPPED(matches the existing migration-entry handling). Route PMD swap entries in thepmd_trans_huge_lock()branch ofmincore_pte_range()throughmincore_pmd_swap()so a swapped-out PMD-mapped THP isn't reported as resident.
- Patch 12 (new): handle PMD swap entries inMADV_WILLNEEDvia swapin_sync(BIT(HPAGE_PMD_ORDER)); a naive order-0 read-ahead
would force the subsequent fault to split.
- Patch 13: refuseUFFDIO_MOVEwith-EBUSYif the swap-cache folio
was split between swap-out and the move, matching move_pages_pte()'s rejection of large folios; otherwise only one
of the 512 anon-rmaps would be re-anchored todst_vma.
- Patch 16:alloc_fill_swap_thp()now uses the existingmmap_pmd_aligned()helper so tests don't flake/skip based on VA placement; newMADV_WILLNEEDtest that watches the PMD-order
mTHP swpin counter; swapoff test restructured to use thekselftest_harnessASSERT cleanup blocks (no double swapoff, no verify-after-munmap).
- Collected Acks and ReviewsUsama Arif (30):
mm: renamepmd_to_softleaf_folio()topmd_softleaf_to_folio()
arm64: mm: add PMD swap-exclusive helpers
loongarch: mm: add PMD swap-exclusive helpers
powerpc: mm: add PMD swap-exclusive helpers
riscv: mm: add PMD swap-exclusive helpers
s390: mm: add PMD swap-exclusive helpers
x86: mm: add PMD swap-exclusive helpers
mm: recognize PMD swap entries in the softleaf layermm/debug_vm_pgtable: test PMD swap-exclusive helpers
mm: make PMD migration-entry splitting explicit
mm: split PMD swap entries into PTE swap entries
mm/swap: allow duplicating a range of swap entries
mm: handle PMD swap entries in fork path
mm: zswap: reject high-order swap cache allocations backed by zswap
mm: swap in PMD swap entries as whole THPs during swapoff fs/proc: account PMD swap entries in smaps
mm: handle soft-dirty and uffd-wp on PMD swap entries mm/hmm: fault PMD swap entries on demand
mm: free PMD swap entries inzap_huge_pmd()
mm/madvise: free PMD swap entries withMADV_FREEmm/madvise: skip PMD swap entries forMADV_COLDandMADV_PAGEOUTmm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE mm/mincore: report PMD swap-cache residency mm/khugepaged: treat PMD swap entries as mapped THPs
mm: handle PMD swap entries inMADV_WILLNEED
mm: handle PMD swap entries inUFFDIO_MOVE
mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
mm: handle PMD swap entry faults on swap-in
mm: install PMD swap entries on swap-out
selftests/mm: add PMD swap entry testsDocumentation/admin-guide/mm/transhuge.rst | 5 + arch/arm64/include/asm/pgtable.h | 6 + arch/loongarch/include/asm/pgtable.h | 19 + arch/powerpc/include/asm/book3s/64/pgtable.h | 6 + arch/riscv/include/asm/pgtable.h | 15 + arch/s390/include/asm/pgtable.h | 20 +- arch/x86/include/asm/pgtable.h | 20 + fs/proc/task_mmu.c | 45 +- include/linux/huge_mm.h | 40 +- include/linux/leafops.h | 44 +- include/linux/pgtable.h | 17 + include/linux/swap.h | 12 +- include/linux/vm_event_item.h | 1 + include/linux/zswap.h | 6 + mm/debug_vm_pgtable.c | 40 + mm/hmm.c | 11 +- mm/huge_memory.c | 714 +++++++++++-- mm/internal.h | 58 ++ mm/khugepaged.c | 6 + mm/madvise.c | 169 +++- mm/memory.c | 65 +- mm/migrate_device.c | 7 +- mm/mincore.c | 47 +- mm/mprotect.c | 2 +- mm/rmap.c | 27 +- mm/swap.h | 30 +- mm/swap_state.c | 83 +- mm/swapfile.c | 245 ++++- mm/userfaultfd.c | 14 + mm/vmscan.c | 9 +- mm/vmstat.c | 1 + mm/zswap.c | 12 +- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 +++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 38 files changed, 2630 insertions(+), 191 deletions(-)create mode 100755
tools/testing/selftests/mm/ksft_pmd_swap.shcreate mode 100644tools/testing/selftests/mm/pmd_swap.c
Re: [PATCH v8 11/30] mm: split PMD swap entries into PTE swap entries
In reply to Usama ArifQuoted 20 lines · Usama Arif
Once a PMD can hold a swap entry, everything that splits a PMD -
mprotect()ormunmap()over part of the range,MADV_FREE, a pagewalk with no PMD handler - has to be able to split that entry too. A swap PMD already passes thepmd_is_valid_softleaf()gate, so without this it reaches__split_huge_pmd_locked()and falls through to the present-PMD path, which pmdp_invalidate()s it and callspmd_page()on a non-present entry.No reference counting is needed: a swap entry pins no folio, and
swap_mapis already one per slot, so the PTEs simply take over what the PMD held.The migration-only entry point cannot reach the new branch:
page_vma_mapped_walk()never hands back a swap PMD, and__split_huge_pmd_locked()already asserts thatto_migration_entriesimplies a present or device-private PMD.Test the pre-split
old_pmdrather than re-reading *pmd in the trailingfolio_remove_rmap_pmd()gate, so every entry-type test in the function interrogates the same snapshot. That part is cosmetic:pmdp_invalidate()leaves the PMD present as far as software is concerned.
Quoted 5 lines
Signed-off-by Usama Arif <[email protected]>--- mm/huge_memory.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v8 13/30] mm: handle PMD swap entries in fork path
In reply to Usama ArifQuoted 74 lines · Usama Arif
copy_huge_pmd()only knows about migration and device-private PMDs, so a PMD swap entry would fall through to the present-PMD path andfork()would duplicate it without taking a reference on the slots it points at.Copy it the way
copy_nonpresent_pte()copies a PTE swap entry: duplicate the swap references, clear the exclusive marker on the source, put the destination mm on mmlist, and account the child's slots toMM_SWAPENTS.The
GFP_ATOMICextend-table allocation inside the dup can fail. Report that as-EIOand letcopy_pmd_range()retry withGFP_KERNEL, ascopy_nonpresent_pte()andcopy_pte_range()already do for a PTE swap entry.copy_huge_pmd()hands the entry back so the caller knows which range to allocate for.Only
-ENOMEMis reported that way. The other failures mean the entry itself is bad, andswap_retry_table_alloc_nr()returns 0 for those, so collapsing them into-EIOas the PTE path does would spin in the caller's retry rather than failing the fork.While here, move the mm counter update into each entry-type arm, as the PTE version does, so the swap arm can account
MM_SWAPENTSinstead ofMM_ANONPAGES.Signed-off-by Usama Arif <[email protected]>--- include/linux/huge_mm.h | 3 +- mm/huge_memory.c | 62 ++++++++++++++++++++++++++++++----------- mm/memory.c | 12 +++++++- 3 files changed, 58 insertions(+), 19 deletions(-)include/linux/huge_mm.h +2
-1@@ -10,7 +10,8 @@vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf);int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm,pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, -struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma); +struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, +softleaf_t *entryp);bool huge_pmd_set_accessed (struct vm_fault *vmf); int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr,mm/huge_memory.c +2
-10@@ -1894,7 +1894,7 @@ bool touch_pmd(struct vm_area_struct *vma, unsigned long addr,return false; } -static void copy_huge_non_present_pmd( +static int copy_huge_non_present_pmd( struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,@@ -1902,18 +1902,41 @@ static void copy_huge_non_present_pmd( {softleaf_t entry =softleaf_from_pmd (pmd);struct folio *src_folio; +int err;VM_WARN_ON_ONCE (!pmd_is_valid_softleaf (pmd)); -if (softleaf_is_migration_write (entry) || -softleaf_is_migration_read_exclusive (entry)) { - entry =make_readable_migration_entry (swp_offset (entry)); - pmd =softleaf_to_pmd (entry); -if (pmd_swp_soft_dirty (*src_pmd)) - pmd =pmd_swp_mksoft_dirty (pmd); -if (pmd_swp_uffd (*src_pmd)) - pmd =pmd_swp_mkuffd (pmd); -set_pmd_at (src_mm, addr, src_pmd, pmd);
mm/huge_memory.c static void copy_huge_non_present_pmd( · line 1918
Quoted 10 lines
mm/huge_memory.c +10
-0+if (softleaf_is_swap (entry)) { +/* +* A PMD swap entry only exists under CONFIG_THP_SWAP, where +* SWAPFILE_CLUSTER == HPAGE_PMD_NR, and it is cluster aligned, +* so these HPAGE_PMD_NR slots are exactly one cluster - which +* is what swap_dup_entries_direct() requires. +*/ + err =swap_dup_entries_direct (entry,HPAGE_PMD_NR ); +if (err) +/* Only -ENOMEM is worth a GFP_KERNEL retry. */
Is it better to say "if swap_dup_entries_direct() cannot allocate memory, return -EIO so that the caller can retry with GFP_KERNEL. Fail the fork on any other error"? It clarifies why EIO is needed.
mm/huge_memory.c static void copy_huge_non_present_pmd( · line 1920
Quoted 2 lines
mm/huge_memory.c +2
-0+return err == -ENOMEM ? -EIO : -ENOMEM ; +
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Zi Yan
reviewerRe: [PATCH v8 15/30] mm: swap in PMD swap entries as whole THPs during swapoff
In reply to Usama ArifQuoted 35 lines · Usama Arif
swapoff walks every mm and faults every slot of the device back in.
unuse_pte_range()only understands PTEs, so a PMD swap entry would never be found and swapoff would never complete.A PMD swap entry is a compact encoding for
HPAGE_PMD_NRslots, not a promise that the swap cache holds one folio for them. Addswap_pmd_cache_lookup()to classify the covered range as empty, one PMD-sized folio, or already split, andunuse_pmd()to map the first two cases back in as one THP, preserving soft-dirty, exclusive and UFFD state.Everything else falls back to PTEs: a split cache, per-page zswap state, a failed PMD-order allocation or read, or a poisoned subpage. Check PageHWPoison on every subpage rather than the folio-level flag, which
memory_failure()only sets after taking the folio lock.All the fallback reasons are observed without the PMD lock and possibly after sleeping, so they share one exit that re-checks the PMD is still the entry we were called for before splitting it. That exit also drops a folio that is not uptodate, or that has never been mapped, from the swap cache:
the PTE path cannot re-read the first, and would add a single-page rmap to the second.Signed-off-by Usama Arif <[email protected]>--- mm/internal.h | 16 ++++ mm/swap.h | 17 +++++ mm/swap_state.c | 44 +++++++++++ mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 266 insertions(+)mm/internal.h +0
-0
mm/internal.h line 27
Quoted 4 lines
mm/internal.h +1
-0@@ -24,6 +24,22 @@struct folio_batch ; +/*
A high level description like below might be better.
Check every page in a folio for HWPoison flag. Return true if there is any.
mm/internal.h line 32
Quoted 5 lines
mm/internal.h +5
-0+* Unlike folio_contain_hwpoisoned_page(), this does not rely on the folio-level +* PG_has_hwpoisoned, which memory_failure() only sets after taking the folio +* lock and so can lag a tail-page poison. +*/ +static inline bool folio_has_hwpoisoned_subpage (const struct folio *folio)
s/subpage/page, IIRC, we are moving away from subpage.
Maybe folio_has_any_hwpoisoned_page()?
Quoted 10 lines
mm/internal.h +10
-0+{ +long nr =folio_nr_pages (folio); +long i; + +for (i =0 ; i < nr; i++) +if (PageHWPoison (folio_page (folio, i))) +return true; +return false; +} +
Signature · 2 lines
Best Regards, Yan, Zi
Usama Arif
reviewerRe: [PATCH v8 03/30] loongarch: mm: add PMD swap-exclusive helpers
In reply to Huacai ChenQuoted 47 lines · On Fri, 2 Oct 2026 22:17:51 +0800 Huacai Chen <[email protected]> wrote:
Hi, Usama,
Quoted 31 lines · Usama Arif
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.A LoongArch swap PMD is the swap PTE value plus
_PAGE_HUGE, and_PAGE_SWP_EXCLUSIVEsits outside both the type and the offset field, so the PMD helpers can use the same bit.Cc Huacai Chen <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by David Hildenbrand (Arm) <[email protected]>--- arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)arch/loongarch/include/asm/pgtable.h +7
-0@@ -351,6 +351,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SWP_EXCLUSIVE; +return pmd; +} +arch/loongarch/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 370
Quoted 10 lines
arch/loongarch/include/asm/pgtable.h +10
-0+static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_SWP_EXCLUSIVE; +return pmd; +}I think the order should be
pmd_swp_exclusive,pmd_swp_mkexclusiveandpmd_swp_clear_exclusive. That looks more natural.
Quoted 1 line
The riscv version is already like this, but others are not.
Done. The LoongArch helpers now use test/mk/clear. I also reordered s390's PMD helpers to match its adjacent PTE helpers. RISC-V and arm64 already use test/mk/clear for their PMD helpers. Powerpc and x86 retain the order used by their adjacent PTE helpers.
Quoted 13 lines
Huacai
Quoted 8 lines
arch/loongarch/include/asm/pgtable.h +2
-1+#endif +#define pte_none (pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_no_exec (pte) (pte_val(pte) & _PAGE_NO_EXEC) --2.53.0-Meta
Usama Arif
reviewerQuoted 11 lines · On Sun, 4 Oct 2026 11:08:44 +0800 Lance Yang <[email protected]> wrote:
Hey Usama,
Staring at the swap-entry bits across architectures, and RISC-V looks a bit off in that regard ...
With Svrsw60t59b and
CONFIG_MEM_SOFT_DIRTY=y, setting swap soft-dirty also sets_PAGE_EXEC. A migration PMD then looks present ... so the fault handler skips the migration wait :(And yeah, this is a pre-existing bug. I've sent a separate fix for it.
Thanks for the fix! PMD swap entries also preserve soft-dirty, so the same encoding issue affects this series. The dependency note lists your separate fix for RISC-V with Svrsw60t59b and soft-dirty enabled. It applies cleanly and preserves the exclusive marker at bit 6.
Quoted 5 lines
https://lore.kernel.org/linux-mm/[email protected]/
BTW, could we also add more checks to
pmd_leaf_soft_dirty_tests()so we catch this kind of issue early? Setting a swap flag shouldn't make a non-present PMD present, IIUC.
Done. The pgtable test now sets swap soft-dirty, checks that the flag is set, and checks that the PMD remains non-present.
Quoted 23 lines
---8<---
mm/debug_vm_pgtable.c +3
-2@@ -762,7 +762,9 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_is_huge (pmd));WARN_ON (!pmd_is_valid_softleaf (pmd)); -WARN_ON (!pmd_swp_soft_dirty (pmd_swp_mksoft_dirty (pmd))); + pmd =pmd_swp_mksoft_dirty (pmd); +WARN_ON (!pmd_swp_soft_dirty (pmd)); +WARN_ON (pmd_present (pmd)); WARN_ON(pmd_swp_soft_dirty(pmd_swp_clear_soft_dirty(pmd))); }#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ --Cheers, Lance
Usama Arif
reviewerRe: [PATCH v8 26/30] mm: handle PMD swap entries in UFFDIO_MOVE
In reply to Lance YangQuoted 19 lines · On Sun, 4 Oct 2026 16:19:17 +0800 Lance Yang <[email protected]> wrote:
Hey Usama,
On Fri, Oct 02, 2026 at 02:52:40AM -0700, Usama Arif wrote:
[...]Quoted 7 lines
mm/huge_memory.c +7
-0+static int move_swap_pmd (struct mm_struct *mm,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +pmd_t *dst_pmd,pmd_t *src_pmd, +pmd_t orig_dst_pmd,pmd_t orig_src_pmd, +spinlock_t *dst_ptl,spinlock_t *src_ptl, +struct folio *src_folio,swp_entry_t entry) +{[...]
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3010
Quoted 4 lines
mm/huge_memory.c +4
-0+ + moved_pmd =pmdp_huge_get_and_clear (mm, src_addr, src_pmd); +if (pgtable_supports_soft_dirty ()) + moved_pmd =pmd_swp_mksoft_dirty (moved_pmd);Shouldn't we clear the source UFFD marker here?
Quoted 3 lines
moved_pmd = pmd_swp_clear_uffd(moved_pmd);just like
move_swap_pte(), no?
Yes, good catch, Thanks! I have added clear for next revision.
Quoted 110 lines
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3014
Quoted 5 lines
mm/huge_memory.c +4
-0+/* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ +if (userfaultfd_rwp (dst_vma)) + moved_pmd =pmd_swp_mkuffd (moved_pmd); +set_pmd_at (mm, dst_addr, dst_pmd, moved_pmd);static int move_swap_pte(struct
mm_struct*mm, structvm_area_struct*dst_vma,
unsigned longdst_addr, unsigned longsrc_addr,pte_t*dst_pte,pte_t*src_pte,
pte_torig_dst_pte,pte_torig_src_pte,pmd_t*dst_pmd,pmd_tdst_pmdval,spinlock_t*dst_ptl,spinlock_t*src_ptl, struct folio*src_folio,
structswap_info_struct*si,swp_entry_tentry)
{
...orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte); ... orig_src_pte = pte_swp_clear_uffd(orig_src_pte); /* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ if (userfaultfd_rwp(dst_vma)) orig_src_pte = pte_swp_mkuffd(orig_src_pte); set_pte_at(mm, dst_addr, dst_pte, orig_src_pte);...
}Suppose the source is an exclusive swap PMD carrying a UFFD-WP marker, and the destination is an empty hole registered for synchronous UFFD-WP without RWP. We haven't write-protected the destination, but a successful whole-PMD MOVE carries the source's UFFD-WP marker into it.
On swap-in, that marker is restored in the present PMD, leaving it write-protected. The write fault then reaches
handle_userfault(vmf, VM_UFFD_WP), even though we never write-protected the destination ...That would be quite a surprise for userspace, and could leave the writer stuck if the unexpected WP event goes unhandled :)
vm_fault_tdo_huge_pmd_swap_page(struct vm_fault *vmf){ struct vm_area_struct *vma = vmf->vma; ... bool exclusive, stable_writes, rwp_restore = false; bool write = vmf->flags & FAULT_FLAG_WRITE; ... pmd = folio_mk_pmd(folio, vma->vm_page_prot); ... if (pmd_swp_uffd(vmf->orig_pmd)) pmd = pmd_mkuffd(pmd); if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) { pmd = pmd_modify(pmd, PAGE_NONE); rwp_restore = true; } ... if (exclusive) { if (!rwp_restore && (vma->vm_flags & VM_WRITE) && !userfaultfd_huge_pmd_wp(vma, pmd) && !pmd_needs_soft_dirty_wp(vma, pmd)) { pmd = pmd_mkwrite(pmd, vma); if (write) pmd = pmd_mkdirty(pmd); } rmap_flags |= RMAP_EXCLUSIVE; } ... vmf->orig_pmd = pmd; ... if (write && !pmd_write(pmd) && !rwp_restore) { vm_fault_t wp_ret = wp_huge_pmd(vmf); ... }...
}
vm_fault_twp_huge_pmd(struct vm_fault *vmf){ struct vm_area_struct *vma = vmf->vma; const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; ... if (vma_is_anonymous(vma)) { if (likely(!unshare) && userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) { if (userfaultfd_wp_async(vmf->vma)) goto split; return handle_userfault(vmf, VM_UFFD_WP); } return do_huge_pmd_wp_page(vmf); }...
}So, clearing the source UFFD-WP marker before the destination RWP check would match the PTE helper and keep the RWP re-arm :)
Am I missing something?
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 3023
Quoted 8 lines
mm/huge_memory.c +8
-0+ src_pgtable =pgtable_trans_huge_withdraw (mm, src_pmd); +pgtable_trans_huge_deposit (mm, dst_pmd, src_pgtable); + +double_pt_unlock (dst_ptl, src_ptl); + return 0; +} +#endif /* CONFIG_THP_SWAP */ +[...]
Cheers, Lance
Usama Arif
reviewerRe: [PATCH v8 28/30] mm: handle PMD swap entry faults on swap-in
In reply to Lance YangQuoted 240 lines · On Sun, 4 Oct 2026 18:19:42 +0800 Lance Yang <[email protected]> wrote:
Hey Usama,
Not an expert on swap, so the below may be naive ...
On Fri, Oct 02, 2026 at 02:52:42AM -0700, Usama Arif wrote:
[...]Quoted 234 lines
(quoted diff) +234
-0+#ifdef CONFIG_THP_SWAP +/** + * do_huge_pmd_swap_page() - Handle a fault on a PMD-level swap entry. + * @vmf: Fault context. vmf->orig_pmd contains the swap PMD. + * + * A PMD swap entry is a compact encoding for HPAGE_PMD_NR consecutive swap + * slots. If the swap cache still has one PMD-sized folio covering the range, + * map it directly at PMD level. If the range has been split into per-page + * cache state, or zswap may have per-page state for it, split the PMD swap + * entry and retry at PTE granularity. + * + * Return: VM_FAULT_* flags. + */ +vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; + struct folio *folio; + struct page *page; + struct swap_info_struct *si; + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + softleaf_t entry; + swp_entry_t swp_entry; + pmd_t pmd; + vm_fault_t ret = 0; + bool exclusive, stable_writes, rwp_restore = false; + bool write = vmf->flags & FAULT_FLAG_WRITE; + rmap_t rmap_flags = RMAP_NONE; + enum swap_pmd_cache cache_state; + + entry = softleaf_from_pmd(vmf->orig_pmd); + if (unlikely(!softleaf_is_swap(entry))) + return 0; + + if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT, + HPAGE_PMD_ORDER)) { + __split_huge_pmd(vma, vmf->pmd, haddr); + return 0; + } + + swp_entry = entry; + + /* Prevent swapoff from happening to us. */ + si = get_swap_device(swp_entry); + if (IS_ERR_OR_NULL(si)) { + if (IS_ERR(si)) + return VM_FAULT_SIGBUS; + return 0; + } + + cache_state = swap_pmd_cache_lookup(swp_entry, &folio); + if (cache_state == SWAP_PMD_CACHE_SPLIT) + goto split_fallback; + if (!folio) { + /* + * PMD swap entries encode ordinary per-page swap slots. If any + * slot is in zswap, split and let the PTE swap path load the + * range per page. Otherwise the range is all on disk and can be + * read back as one PMD-sized folio. + */ + if (zswap_is_present(swp_entry, HPAGE_PMD_NR)) + goto split_fallback; + + folio = swapin_sync(swp_entry, GFP_HIGHUSER_MOVABLE, + BIT(HPAGE_PMD_ORDER), vmf, NULL, 0); + if (IS_ERR_OR_NULL(folio)) + goto split_fallback; + + /* Had to read from swap area: Major fault */ + ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + } + + ret |= folio_lock_or_retry(folio, vmf); + if (ret & VM_FAULT_RETRY) + goto out_release; + + /* Verify the folio is still in swap cache and matches our entry */ + if (unlikely(!folio_matches_swap_entry(folio, swp_entry))) + goto out_page; + + /* + * Folio should be PMD-sized; if not (e.g. split in swap cache), + * split the PMD swap entry and retry at PTE level. + */ + if (folio_nr_pages(folio) != HPAGE_PMD_NR) + goto unlock_split_fallback; + + /* + * A read that failed - a PMD-order zswap load that found per-page + * state, or an I/O error - leaves the folio clean and not uptodate. + * Fall back so the PTE retry reads each slot again rather than + * returning SIGBUS for the whole range. + */ + if (unlikely(!folio_test_uptodate(folio))) + goto unlock_split_fallback; + + /* + * If any subpage is hardware-poisoned, split the PMD swap entry and + * let the PTE swap-in path handle each page individually so + * do_swap_page() can return VM_FAULT_HWPOISON for the poisoned + * subpage rather than mapping the corrupted memory as one THP. + */ + if (unlikely(folio_has_hwpoisoned_subpage(folio))) + goto unlock_split_fallback; + + page = folio_page(folio, 0); + arch_swap_restore(folio_swap(swp_entry, folio), folio); + + folio_throttle_swaprate(folio, GFP_KERNEL); + + /* Lock the PMD and verify it hasn't changed */ + vmf->ptl = pmd_lock(mm, vmf->pmd); + if (unlikely(!pmd_same(vmf->orig_pmd, pmdp_get(vmf->pmd)))) { + spin_unlock(vmf->ptl); + goto out_page; + } + + exclusive = pmd_swp_exclusive(vmf->orig_pmd); + + /* + * Some swap backends (e.g. zram) don't support concurrent page + * modifications while under writeback. If we map exclusive on such + * a backend while the folio is still under writeback, the writeback + * may see partial modifications and corrupt the swap slot. Drop the + * exclusive marker and only map R/O for that case; further GUP + * references can't appear once the page is fully unmapped, so this + * is safe. + */ + /* Lockless like do_swap_page(): SWP_STABLE_WRITES never changes. */ + stable_writes = data_race(si->flags & SWP_STABLE_WRITES); + if (exclusive && folio_test_writeback(folio) && stable_writes) + exclusive = false; + + /* + * Set up the PMD mapping. Similar to do_swap_page() but at PMD level. + */ + add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + + pmd = folio_mk_pmd(folio, vma->vm_page_prot); + pmd = pmd_mkyoung(pmd); + + if (pmd_swp_soft_dirty(vmf->orig_pmd)) + pmd = pmd_mksoft_dirty(pmd); + if (pmd_swp_uffd(vmf->orig_pmd)) + pmd = pmd_mkuffd(pmd); + if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) { + pmd = pmd_modify(pmd, PAGE_NONE); + rwp_restore = true; + } + + /* + * Check exclusivity to determine if we can map writable. + */ + if (exclusive) { + if (!rwp_restore && (vma->vm_flags & VM_WRITE) && + !userfaultfd_huge_pmd_wp(vma, pmd) && + !pmd_needs_soft_dirty_wp(vma, pmd)) { + pmd = pmd_mkwrite(pmd, vma); + if (write) + pmd = pmd_mkdirty(pmd); + } + rmap_flags |= RMAP_EXCLUSIVE; + } + + flush_icache_pages(vma, page, HPAGE_PMD_NR); + + if (!folio_test_anon(folio)) + folio_add_new_anon_rmap(folio, vma, haddr, rmap_flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, haddr, rmap_flags); + + folio_put_swap(folio, NULL); + + set_pmd_at(mm, haddr, vmf->pmd, pmd); + update_mmu_cache_pmd(vma, haddr, vmf->pmd); + + /* Update orig_pmd for any follow-up wp_huge_pmd() below. */ + vmf->orig_pmd = pmd; + + /* + * Conditionally try to free up the swap cache. Do it after mapping, + * so raced page faults will likely see the folio in swap cache and + * wait on the folio lock. + */ + if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags)) + folio_free_swap(folio); + + spin_unlock(vmf->ptl); + + folio_unlock(folio); + put_swap_device(si); + + /* + * If the write fault wasn't satisfied above (folio is shared without + * exclusivity), call wp_huge_pmd() to handle COW or + * userfaultfd-wp without forcing a second fault. + * + * wp_huge_pmd() may return VM_FAULT_FALLBACK if it had to split the + * PMD; that's a normal outcome, and the natural PTE-level refault will + * complete the COW. Mask it so callers (and the arch fault handler) + * don't see VM_FAULT_FALLBACK as a fatal VM_FAULT_ERROR. + */ + if (write && !pmd_write(pmd) && !rwp_restore) { + vm_fault_t wp_ret = wp_huge_pmd(vmf); + + wp_ret &= ~VM_FAULT_FALLBACK; + ret |= wp_ret; + if (ret & VM_FAULT_ERROR) + ret &= VM_FAULT_ERROR; + } + + return ret; + +out_page: + folio_unlock(folio); +out_release: + folio_put(folio); + put_swap_device(si); + return ret; + +unlock_split_fallback: + /* + * PTE fallback cannot add a single-page rmap to a PMD-sized folio that + * has never been mapped: do_swap_page() would hand the whole folio to + * folio_add_new_anon_rmap() while installing one PTE. Nor can it do + * anything useful with a folio that failed to read. Remove either from + * the swap cache so each slot is read back into its own order-0 folio. + * An uptodate anon swap-cache folio can be mapped one PTE at a time and + * must stay cached, so that any poisoned subpage stays visible to + * do_swap_page(). This mirrors unuse_pmd_entry(). + */
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (folio_matches_swap_entry(folio, swp_entry) && + (!folio_test_uptodate(folio) || !folio_test_anon(folio))) + swap_cache_del_folio(folio); + folio_unlock(folio); + folio_put(folio);Requesting
BIT(HPAGE_PMD_ORDER)doesn't preventswapin_sync()from returning a cached order-0 folio ...
Agreed. swapin_sync() returns an existing cached folio without restricting its order. A small zswap writeback folio can still need its cache reference while I/O is pending.
This base retains zswap's allocation reference until after swap_write_submit(), rather than dropping it before __swap_writeout(). After that reference is dropped, the cache reference can still be needed for pending I/O, so the size-mismatch fix is still required.
Quoted 75 lines
struct folio *swapin_sync(swp_entry_t entry,
gfp_tgfp, unsigned long orders,
structvm_fault*vmf, struct mempolicy *mpol,pgoff_tilx)
{
...do { folio = swap_cache_get_folio(entry); if (folio) return folio; folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx); } while (PTR_ERR(folio) == -EEXIST);...
}static int zswap_writeback_entry(struct
zswap_entry*entry,
swp_entry_tswpentry)
{
...folio = __swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol, NO_INTERLEAVE_INDEX); ... if (IS_ERR(folio)) return PTR_ERR(folio); ... if (!zswap_decompress(entry, folio)) { ret = -EIO; goto err; } xa_erase(tree, offset); ... /* folio is up to date */ folio_mark_uptodate(folio); folio_set_dropbehind(folio); ... folio_put(folio); /* start writeback */ __swap_writeout(&ctx, folio); swap_write_submit(&ctx); return 0;...
}void
__swap_writeout(struct swap_io_ctx *ctx, struct folio *folio){ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); ... folio_start_writeback(folio); folio_unlock(folio); swap_add_folio(ctx, folio, WRITE); }Say we hit the following race:
1)
swap_pmd_cache_lookup()finds the range empty, before zswap writeback inserts its order-0 folio.2) zswap writeback puts that small folio in the swap cache at the first slot, then erases the last zswap entry in the range.
3) The fault thread checks zswap and gets false from
zswap_is_present(). It then callsswapin_sync(), which returns that cached small folio.4) After
__swap_writeout()unlocks the folio, the fault thread can lock it while writeback is still pending. The folio isn't PMD-sized, so we reach that cleanup ...zswap has already dropped its allocation reference, and writeback itself doesn't hold one. So, without any other references, we're left with the swap-cache reference and our fault reference.
Quoted 8 lines
The folio is still !anon, so
swap_cache_del_folio()removes the cache reference, thenfolio_put()drops ours. That can drop the last reference while I/O is still using the folio ... :(So ... could we just unlock and put it on the size mismatch, then jump to
split_fallback? That would keep the folio in the swap cache fordo_swap_page().
Done for next revision. Thanks for catching this!
A size mismatch now unlocks and puts the folio before jumping to split_fallback. It stays in the swap cache for do_swap_page(). Only PMD-sized folios reach the cleanup that may delete a folio.
Quoted 56 lines
Something like:
---8<---
mm/huge_memory.c +8
-3@@ -2507,9 +2507,14 @@ vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)/* * Folio should be PMD-sized; if not (e.g. split in swap cache), * split the PMD swap entry and retry at PTE level. +* Keep the folio cached: zswap writeback may still be in flight and +* relies on the swap-cache reference to keep it alive. */ -if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) - goto unlock_split_fallback; +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { +folio_unlock (folio); +folio_put (folio); + goto split_fallback; + }/* * A read that failed - a PMD-order zswap load that found per-page @@ -2647,6 +2652,7 @@ vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) unlock_split_fallback:/* +* Only PMD-sized folios reach this cleanup. * PTE fallback cannot add a single-page rmap to a PMD-sized folio that * has never been mapped: do_swap_page() would hand the whole folio to * folio_add_new_anon_rmap() while installing one PTE. Nor can it do -- Hopefully I didn't miss anything :D
Quoted 5 lines
(quoted diff) +5
-0+split_fallback: + /* + * Only split if the PMD is still the swap entry we were called for. + * All the reasons we get here (allocation failure, zswap state, a + * split or poisoned cached folio) were observed without the PMD lock,Review
Quoted 10 lines
(quoted diff) +10
-0+ * so a racing thread may already have swapped the range back in as a + * THP -- splitting that would silently demote a perfectly good huge + * mapping. + */ + if (pmd_same(vmf->orig_pmd, pmdp_get_lockless(vmf->pmd))) + __split_huge_pmd(vma, vmf->pmd, haddr); + put_swap_device(si); + return 0; +} +#endif /* CONFIG_THP_SWAP */[...]
Cheers, Lance
Usama Arif
reviewerRe: [PATCH v8 07/30] x86: mm: add PMD swap-exclusive helpers
In reply to Zi YanQuoted 31 lines · On Mon, 05 Oct 2026 16:14:52 -0400 "Zi Yan" <[email protected]> wrote:
Quoted 30 lines · Usama Arif
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse
_PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves forPG_anon_exclusive. 32-bit x86 aliases that bit to_PAGE_PSEand does not selectARCH_HAS_PMD_SOFTLEAVES; assert that, because an exclusive swap PMD would otherwise bepmd_present().Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>Reviewed-by David Hildenbrand (Arm) <[email protected]>--- arch/x86/include/asm/pgtable.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)arch/x86/include/asm/pgtable.h +0
-0@@ -1525,6 +1525,26 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_EXCLUSIVE); }
Quoted 8 lines
arch/x86/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 1529
Quoted 2 lines
arch/x86/include/asm/pgtable.h +2
-0+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +/* 32-bit aliases this to _PAGE_PSE, making an exclusive swap PMD present. */ Is it better to say? Since the above comment sounds like 32-bit specific, but the actual constraint is about not being
_PAGE_PSE.
_PAGE_SWP_EXCLUSIVEcannot be_PAGE_PSE, otherwise making an exclusive swap PMD present.
Updated the comment to:
/*
* _PAGE_SWP_EXCLUSIVE must not alias _PAGE_PSE, which would make an
* exclusive swap PMD present.
*/
which I think is clearer.
Quoted 25 lines
Quoted 12 lines
arch/x86/include/asm/pgtable.h +11
-0+static_assert (_PAGE_SWP_EXCLUSIVE != _PAGE_PSE); + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_EXCLUSIVE; +}arch/x86/include/asm/pgtable.h static inline pte_t pte_swp_clear_exclusive(pte_t pte) · line 1550
Quoted 10 lines
arch/x86/include/asm/pgtable.h +7
-0+ +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty (pte_t pte) {Otherwise, LGTM.
Quoted 1 line
Reviewed-by Zi Yan <[email protected]>
Thanks for the review! And apologies for the late reply, I was away.
Quoted 7 lines
Signature · 3 lines
-- Best Regards, Yan, Zi
Usama Arif
reviewerRe: [PATCH v8 14/30] mm: zswap: reject high-order swap cache allocations backed by zswap
In reply to Yosry AhmedQuoted 12 lines · On Mon, 5 Oct 2026 13:18:24 -0700 Yosry Ahmed <[email protected]> wrote:
Quoted 11 lines · Usama Arif
zswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Reject that in
__swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, whichzswap_load()handles per page.
Quoted 8 lines
Quoted 5 lines
Export
zswap_is_present()for the PMD-order swap-in users added later.Co-developed-by Alexandre Ghiti <[email protected]>Signed-off-by Alexandre Ghiti <[email protected]>Signed-off-by Usama Arif <[email protected]>Overall looks good, but I think it can be split into 3 patches (one of which doesn't have to be part of this series), see below.
Thanks for the review! I have followed your suggestion and split it into 3 patches now.
Quoted 152 lines
Quoted 90 lines
--- include/linux/zswap.h | 6 ++++++ mm/swap_state.c | 39 +++++++++++++++++++++++++-------------- mm/zswap.c | 12 +++++++++--- 3 files changed, 40 insertions(+), 17 deletions(-)include/linux/zswap.h +6
-0@@ -35,6 +35,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);void zswap_folio_swapin (struct folio *folio);bool zswap_is_enabled (void );bool zswap_never_enabled (void ); +bool zswap_is_present (swp_entry_t entry,unsigned int nr);#else struct zswap_lruvec_state {};@@ -73,6 +74,11 @@ static inline bool zswap_never_enabled(void)return true; } +static inline bool zswap_is_present (swp_entry_t entry,unsigned int nr) +{ +return false; +} +#endif #endif /* _LINUX_ZSWAP_H */ mm/swap_state.c +16
-14@@ -12,6 +12,7 @@#include <linux/kernel_stat.h> #include <linux/mempolicy.h> #include <linux/swap.h> +#include <linux/zswap.h> #include <linux/leafops.h> #include <linux/init.h> #include <linux/pagemap.h> @@ -466,26 +467,27 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,__swap_cache_do_add_folio (ci, folio, entry);spin_unlock (&ci->lock); +/* +* Now that the folio is in the swap cache, zswap can no longer start +* storing or writing back any slot in the range, so this is a stable +* answer. Reject a high-order allocation over a range that already +* has per-page zswap entries. +*/ +if (order &&zswap_is_present (entry, nr_pages)) { + err = -EBUSY ; + goto delete_folio; + } +if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); + err = -ENOMEM ;count_mthp_stat (order,MTHP_STAT_SWPIN_FALLBACK_CHARGE ); -return ERR_PTR (-ENOMEM ); + goto delete_folio; }if (order >1 &&folio_memcg_alloc_deferred (folio)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); -return ERR_PTR (-ENOMEM ); + err = -ENOMEM ; + goto delete_folio; }/* memsw uncharges swap when folio is added to swap cache */ @@ -499,6 +501,15 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,/* Caller will initiate read into locked new_folio */ folio_add_lru (folio);mm/swap_state.c static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, · line 512
Quoted 10 lines
mm/swap_state.c +9
-0return folio; + +delete_folio: +spin_lock (&ci->lock); +__swap_cache_do_del_folio (ci, folio, entry, shadow); +spin_unlock (&ci->lock); +folio_unlock (folio); +/* nr_pages refs from swap cache, 1 from allocation */ +folio_put_refs (folio, nr_pages +1 ); +return ERR_PTR (err);Nit: You can pull the refactoring into a separate patch (as well as most zswap changes, see below), so that this patch only has the logical change to bail for higher orders if any entry is in zswap.
Quoted 8 lines
mm/swap_state.c +0
-0}/** mm/zswap.c +0
-0mm/zswap.c bool zswap_store(struct folio *folio) · line 1594
Quoted 10 lines
mm/zswap.c +5
-2@@ -1587,11 +1587,14 @@ bool zswap_store(struct folio *folio)* @entry: base swap entry of the range * @nr: number of contiguous slots to check * -* Context: The caller must keep the range pinned, otherwise the answer can -* change under it. +* Context: Unless the caller keeps the range pinned, the answer is only a +* hint: zswap can store or write back a slot right after this returns. A +* caller that needs a stable answer must pin the range first, as +* __swap_cache_alloc() does by inserting the folio into the swap cache before +* asking. I would rather avoid naming specific functions as it will go stale quickly, and instead list the ways of "pinning" a range.
Also, generally, the changes to
mm/zswap.cseem orthogonal to this patch and can be a separate patch (which would be clearer if__swap_cache_alloc()isn't named here).Quoted 20 lines
mm/zswap.c +4
-2* Return: true if at least one slot in the range is in zswap. */ -static bool zswap_is_present (swp_entry_t entry,unsigned int nr) +bool zswap_is_present (swp_entry_t entry,unsigned int nr) {pgoff_t offset =swp_offset (entry);struct xarray *tree =swap_zswap_tree (entry);@@ -1605,6 +1608,9 @@ static bool zswap_is_present(swp_entry_t entry, unsigned int nr)*/ BUILD_BUG_ON (SWAPFILE_CLUSTER >ZSWAP_ADDRESS_SPACE_PAGES ); +if (zswap_never_enabled ()) +return false; +return xa_find (tree, &index, offset + nr -1 ,XA_PRESENT ); } --2.53.0-Meta
Usama Arif
reviewerRe: [PATCH v8 08/30] mm: recognize PMD swap entries in the softleaf layer
In reply to Zi YanQuoted 31 lines · On Wed, 07 Oct 2026 22:06:57 -0400 "Zi Yan" <[email protected]> wrote:
Quoted 30 lines · Usama Arif
Reclaim splits a PMD-mapped anonymous THP into PTE-level swap entries before unmapping it, so an ordinary swap entry has never had to appear in a PMD. Prepare for installing one there instead: the softleaf layer is where every consumer decodes non-present PMDs.
Accept swap entries as valid PMD softleaves and add
pmd_is_swap_entry(). A swap entry carries no PFN, so makepmd_softleaf_to_folio()warn and return NULL rather than interpret a swap offset as a page frame number.Unlike migration and device-private entries, a PMD swap entry can also carry the swap-exclusive marker, which
softleaf_from_pmd()has to strip before decoding. Strip all three overlays unconditionally while we are
here: each clear is a plain bit clear, so testing first only buys a branch.Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>--- include/linux/leafops.h | 40 ++++++++++++++++++++++++++++------------ include/linux/pgtable.h | 17 +++++++++++++++++ 2 files changed, 45 insertions(+), 12 deletions(-)include/linux/leafops.h +0
-0@@ -98,10 +98,9 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)if (pmd_present (pmd) ||pmd_none (pmd))return softleaf_mk_none ();
Quoted 10 lines
include/linux/leafops.h static inline softleaf_t softleaf_from_pmd(pmd_t pmd) · line 103
Quoted 7 lines
include/linux/leafops.h +3
-4-if (pmd_swp_soft_dirty (pmd)) - pmd =pmd_swp_clear_soft_dirty (pmd); -if (pmd_swp_uffd (pmd)) - pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_soft_dirty (pmd); + pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_exclusive (pmd);I notice there is
pte_swp_clear_flags()and wonder if we wantpmd_swp_clear_flags()to match it.
Added pmd_swp_clear_flags() alongside pte_swp_clear_flags() and used it in softleaf_from_pmd() for the next revision.
Quoted 19 lines
include/linux/leafops.h static inline bool pmd_is_migration_entry(pmd_t pmd) · line 638
Quoted 13 lines
include/linux/leafops.h +2
-2arch_entry =__pmd_to_swp_entry (pmd);/* Temporary until swp_entry_t eliminated. */ @@ -634,18 +633,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)*/ static inline bool softleaf_is_valid_pmd_entry (softleaf_t entry) { -/* Only device private, migration entries valid for PMD. */ return softleaf_is_device_private (entry) || -softleaf_is_migration (entry); +softleaf_is_migration (entry) || +softleaf_is_swap (entry);Using
softleaf_is_swap()here makessoftleaf_is_valid_pmd_entry()return false positive when pmd swap entry is not possible at the moment. Adding it only after pmd swap entry is possible can close the gap. But the whole series should be applied together, so it should be fine.
Quoted 2 lines
Otherwise, LGTM.
The predicate accepts the new encoding before the producer is enabled, but no PMD swap entry can be installed at that point. Reclaim still uses TTU_SPLIT_HUGE_PMD until the final swap-out patch. I kept the ordering so each consumer is ready before entries can be produced. As you said, there is no reachable intermediate PMD swap entry to mishandle.
Quoted 2 lines
Reviewed-by Zi Yan <[email protected]>
Thanks for the reviews!
Quoted 5 lines
Signature · 2 lines
Best Regards, Yan, Zi
Usama Arif
reviewerRe: [PATCH v8 10/30] mm: make PMD migration-entry splitting explicit
In reply to Zi YanQuoted 11 lines · On Wed, 07 Oct 2026 22:53:51 -0400 "Zi Yan" <[email protected]> wrote:
Quoted 10 lines · Usama Arif
__split_huge_pmd()and friends take a "freeze" boolean that every caller has to pass and almost every caller passes as false. The name says nothing about what it selects, and the one thing it does select - PTE migration entries instead of PTE mappings - is only ever wanted by the rmap migration path.Rename it to
to_migration_entries, keep it private tomm/huge_memory.c, and addsplit_pmd_to_migration_entries()fortry_to_migrate_one(), the only caller that wants it.
Quoted 7 lines
Quoted 2 lines
migrate_vma_split_unmapped_folio()also passed freeze=true, but only ever runs on a PMD that is already a migration entry, which the generic helperIs it always a PMD migration entry? Can a concurrent
MADV_DONTNEEDzap the PMD migration entry or split it with a partial zap? Although it does not affect the correctness of the patch.
Yes. Migration holds mmap_lock for reading, and MADV_DONTNEED can clear or split the migration PMD without taking the source folio lock. The PMD need not still be a migration entry when the split helper checks it.
Quoted 5 lines
If that concurrent zap is possible, the existing code can leak a ref, causing the folio to be not freed, since
split_huge_pmd_address(freeze=true)does not drop the folio refcount when the PMD is zapped or becomes a PTE page.
Agreed. The old extra folio_get() leaks if the helper finds an empty PMD or a PTE table and returns without consuming that reference. folio_split_unmapped() can then fail because of the unexpected ref.
Quoted 2 lines
We might want a separate fix for it.
Extracted it into a separate prerequisite fix, with your Reported-by, a Fixes tag and stable Cc. It removes the extra reference and passes freeze=false using the old API. An existing migration PMD becomes PTE migration entries either way; a concurrent zap or split no longer leaves an extra reference behind.
Quoted 57 lines
Actually, Mika Penttilä's "migrate on fault for device pages"[1] also talks about the race.
[1] https://lore.kernel.org/r/[email protected]
Quoted 21 lines
expands into PTE migration entries either way. Its
folio_get()only existed to balance theput_page()that freeze=true performs, so both go.
split_pmd_to_migration_entries()is only ever handed a present or device-private PMD, so assert that in__split_huge_pmd_locked()instead of silently skipping anything else.Other than that assertion, no functional change intended.
Suggested-by David Hildenbrand (Arm) <[email protected]>Signed-off-by Usama Arif <[email protected]>Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>--- include/linux/huge_mm.h | 21 ++++++------ mm/huge_memory.c | 74 ++++++++++++++++++++++++++--------------- mm/memory.c | 4 +-- mm/migrate_device.c | 7 +--- mm/mprotect.c | 2 +- mm/rmap.c | 8 ++--- 6 files changed, 65 insertions(+), 51 deletions(-)<snip>
Quoted 8 lines
mm/migrate_device.c +0
-0@@ -918,12 +918,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ;mm/migrate_device.c static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, · line 924
Quoted 10 lines
mm/migrate_device.c +1
-6-/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr); ret =folio_split_unmapped (folio,0 );if (ret)return ret;This part can be a separate fix for the refcount leak.
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi