mm: PMD-level swap entries for anonymous THPs
Subsystems (24)
Usama Arif
reviewerv9 cover mm: PMD-level swap entries for anonymous THPs
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into HPAGE_PMD_NR PTE-level swap entries via TTU_SPLIT_HUGE_PMD before unmap. This series introduces a PMD-level swap entry so the huge mapping can survive the swap round-trip and do_huge_pmd_swap_page() can restore the PMD mapping directly on swap-in, without waiting for khugepaged to collapse the range later.
The PMD swap entry is a compact page-table encoding for HPAGE_PMD_NR consecutive swap slots. swap_map accounting remains per-slot and is unchanged. A PMD swap entry does not promise that the swap cache always contains one PMD-sized folio. While the cache is empty or contains one PMD-sized folio, PMD-level handling can proceed. Once the cache has split/per-slot state, users either inspect the individual slots directly (mincore) or split the PMD swap entry and retry through the PTE path (fault, swapoff, MADV_WILLNEED, UFFDIO_MOVE). MADV_FREE does not consult the cache: it frees a whole PMD swap entry in place and only splits when the advised range covers part of the PMD. Likewise, if any slot is still backed by zswap's per-page store, PMD-order swap-in consumers split and let the PTE path load the range page by page; an all-on-disk range can still be read back as one PMD-sized folio.
The series is ordered so every consumer can handle PMD swap entries before the swap-out producer starts installing them. The swap-out patch is the last functional change.
Performance (measurements from v8):
Measured with vm-scalability's case-swap-w-seq benchmark [1]. Four pinned workers repeatedly write a 6 GiB anonymous working set on a 4 vCPU / 4 GiB guest, forcing the set out to swap and back in. Swap is an 8 GiB NOCOW raw virtio device (cache=none, aio=native), zswap is disabled, and THP enabled/defrag are both "always". The numbers below are medians of five interleaved runs per kernel after one warm-up run:
Metric Baseline Patched Change Aggregate throughput 584.3 MiB/s 2408.9 MiB/s +312.2% (4.12x) Elapsed time 85.43 s 20.72 s -75.7% Major faults 1,814,699 228,466 -87.4% Swap I/O rate 1.02 GiB/s 4.10 GiB/s +303.9%
This swap-intensive synthetic workload measures the reduction in swap-fault and page-table overhead from preserving PMD mappings. For long-running workloads, I expect benefits from fewer TLB misses, less khugepaged work and retaining larger folios. I have not yet found a benchmark that captures those benefits well. PMD swap entries also move us closer to eliminating page-table deposits for anonymous THPs, which would save memory.
Intermediate patches and Sashiko:
Earlier patches prepare consumers while reclaim still requests TTU_SPLIT_HUGE_PMD. PMD swap entries cannot reach those consumers until the swap-out producer is enabled. Concerns requiring live PMD swap entries before that patch describe an unreachable state. See [2]. The v8 Sashiko report failed to apply and supplied no findings.
Notes on zswap:
Native PMD-order zswap load/store is left for a follow-up. Alexandre Ghiti is currently working on this.
This series can still preserve PMD swap entries while zswap is enabled:
zswap stores the THP as order-0 entries, and PMD-order swap-in consumers split any range that has zswap entries before reading it. If zswap has written the whole range back to disk, or the swap cache still contains one PMD-sized folio, PMD-level handling can proceed.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
[2] https://lore.kernel.org/all/[email protected]/
v8 -> v9: https://lore.kernel.org/all/[email protected]/
Patch numbers below refer to v9.
- Rebase onto akpm/mm-new at dba8aba0a9a0.
- Grow from 30 to 32 patches by extracting the migration reference fix and splitting the zswap preparatory changes. (Zi Yan, Yosry Ahmed)
- Patches 3 (loongarch) and 6 (s390): order the PMD exclusive helpers as test/mk/clear, matching their adjacent PTE helpers. (Huacai Chen)
- Patch 7: describe the exclusive-bit assertion in terms of not aliasing _PAGE_PSE, without making the constraint specific to 32-bit. (Zi Yan)
- Patch 8: add pmd_swp_clear_flags() alongside the PTE helper and use it in softleaf_from_pmd(). Keep consumers before the swap-out producer:
no PMD swap entry can be installed in the intermediate patches. (Zi Yan)
- Patch 9: check that setting swap soft-dirty leaves the PMD non-present. (Lance Yang)
- Patch 10 (new): extract the reference fix already present in v8 from the migration-splitting API cleanup, now patch 11. A concurrent MADV_DONTNEED can zap or partially split the migration PMD and cause the old freeze=true path to strand the extra folio reference. Remove that reference and use freeze=false with the old API. Drop patch 11's claim that the PMD must still be a migration entry. (Zi Yan)
- Patch 14: explain that -EIO requests a GFP_KERNEL allocation retry, while other errors fail the fork. (Zi Yan)
- Patches 15-16: keep shared swap-cache allocation error cleanup and high-order zswap rejection separate. The new base already exports zswap_is_present(), so drop the separate locking documentation patch. (Yosry Ahmed)
- Patch 17: rename the helper to folio_has_any_hwpoisoned_page(), add a high level description and update all three callers, including the PTE-batching and PMD-fault paths in patches 29 and 30. (Zi Yan)
- Patch 28: clear the source UFFD marker before destination RWP re-arming, matching move_swap_pte(). (Lance Yang)
- Patch 30: leave a cached folio that is not PMD-sized in the swap cache on PTE fallback; it may still be a zswap writeback folio under I/O. (Lance Yang)
v7 -> v8: https://lore.kernel.org/all/[email protected]/
- Cover letter: add the case-swap-w-seq performance results. (Andrew Morton)
- Patches 2-8: reword the opening paragraph as "Prepare for ..." instead of referring to a later patch. (David Hildenbrand)
- Patch 4 (powerpc): move the PMD exclusive helpers next to the soft-dirty PMD helpers and express them the same way, as pte_pmd()/pmd_pte() wrappers over the PTE helpers. (David Hildenbrand)
- Patch 6 (s390): drop the comment above the helpers, keep the explanation in the RSTE swap layout above __SWP_OFFSET_MASK_RSTE, and move the helpers into the existing CONFIG_ARCH_HAS_PMD_SOFTLEAVES block above pmd_swp_soft_dirty(). (David Hildenbrand)
- Patch 7 (x86): add static_assert(_PAGE_SWP_EXCLUSIVE != _PAGE_PSE) so a 32-bit build that ever selects ARCH_HAS_PMD_SOFTLEAVES fails to compile rather than producing pmd_present() swap entries. (Kiryl Shutsemau)
- Patch 10: rename the flag to to_migration_entries, use two-tab continuation indentation, reword the split comment, turn the comment on split_pmd_to_migration_entries() into kerneldoc, and fold the try_to_migrate_one() call onto one line. Also drop that helper's pmd_trans_huge() || pmd_is_valid_softleaf() test, which was both too loose and silently skipped the split, in favour of a VM_WARN_ON_ONCE() at the top of __split_huge_pmd_locked() asserting that to_migration_entries implies a present or device-private PMD. (David Hildenbrand)
- Patch 11: drop the VM_WARN_ON_ONCE()/force and its comment from the swap decode arm - 10/29 now asserts the to_migration_entries contract at the top of __split_huge_pmd_locked() instead - and build the replacement PTEs by advancing pte_next_swp_offset() rather than rebuilding each entry, which hoists the soft-dirty/uffd/exclusive tests out of the loop. (David Hildenbrand).
- Patch 12 (new): split the swap-side changes out of the fork patch, so the swap-entry range duplication gets its own patch for the swap maintainers. Keeps the single-slot names as inline wrappers so existing callers are untouched. (David Hildenbrand)
- Patch 13 (fork): report a failed swap dup as -EIO and let copy_pmd_range() own the GFP_KERNEL retry, as the PTE path does, dropping the open-coded retry loop; move the mm counter update into each entry-type arm. (David Hildenbrand)
- Patch 16 (smaps): smaps_account_swap() takes nr_pages rather than a byte size, and the local is called swapcount. (David Hildenbrand)
- No other functional change. All twelve sashiko findings on v7 were analysed and none are defects of this series.
v6 -> v7: https://lore.kernel.org/all/[email protected]/
- Rebase onto akpm/mm-new at baa8de2f3448 and adapt to the new get_swap_device() contract and linear_anon_page_index(). The series grows from 12 to 29 patches.
- Patch 1: code unchanged; reword the commit message and collect review tags.
- Patches 2-9: split the six architecture helpers from generic detection and add debug_vm_pgtable coverage. Use the s390 RSTE exclusive bit, make the x86 helper return bool, clear all PMD swap overlay bits before softleaf decoding, and add the architecture maintainers to Cc.
- Patches 10-11: add a preparatory no-functional-change cleanup of the migration-splitting API and keep the PMD swap split separate.
- Patch 12: reject a multi-slot duplication range that crosses a swap-cluster boundary.
- Patch 13: drop the zswap_load() change, now upstream as 1a904e0d3c43, and check for zswap after swap-cache insertion while all slots are pinned.
- Patch 14: scan every subpage for hardware poison, discard a failed or newly allocated unmapped PMD-sized folio before PTE fallback, and recheck the PMD before splitting it.
- Patches 15-23: split the non-present PMD walkers by subsystem. Add the guard-advice patch so MADV_GUARD_INSTALL/REMOVE leaves PMD swap entries whole; the other split patches preserve v6 behaviour.
- Patch 24: honour the current THP policy, discard a failed PMD-sized folio, and recheck the PMD before PTE fallback.
- Patch 25: do not split after cached-folio revalidation loses a race; retry so a restored present THP remains whole.
- Patches 26-27: separate the independent PTE-batching hardware-poison fix, scan subpages directly, and discard failed or never-mapped PMD-sized folios before PTE retry instead of making the whole range fail with SIGBUS.
- Patch 28: keep the normal swap-out path unchanged, but make failed producer preconditions warn and return -EBUSY rather than BUG or return -EINVAL.
- Patch 29: grow the selftests from 16 to 17, adding swapin_sync/cache-residency coverage and more robust feature, fallback, privilege, data-integrity, and swap-device-priority handling.
v5 -> v6: https://lore.kernel.org/all/[email protected]/
- Add patch 1 to rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio(). No functional change. (Dev Jain)
- Patch 2: warn when pmd_softleaf_to_folio() is given a non-PFN softleaf rather than silently returning NULL. (Dev Jain)
- Patch 4: bound the fork extend-table fallback to one retry, re-read the PMD under its lock, normalize unrecoverable copy_huge_pmd() errors to -ENOMEM so copy_pmd_range() cannot clear and leak the source swap PMD, and drop a redundant thp_migration_supported() gate.
- Patch 5: check multi-page swap-cache insertions for zswap-backed slots in __swap_cache_add_check() under the cluster lock, both before allocation and before insertion, and reject mixed zswap/disk state with -EBUSY. (Yosry Ahmed, Nhat Pham)
- Patch 6: on a failed non-uptodate PMD-order read, remove the large folio from swap cache before splitting so order-0
fallback retries individual slots rather than poisoning the whole
2 MiB range; retain hardware-poisoned folios for per-subpage handling.
- Patch 7: make HMM snapshot mode report a PMD swap entry as non-resident, matching PTE swap entries, rather than HMM_PFN_ERROR. Drop redundant thp_migration_supported() gates and simplify non-present PMD handling.
- Patch 8: factor PMD MADV_WILLNEED prefetch into swapin_pmd_swap_entry(), split and retry through PTEs after any PMD-order swapin failure, and replace the racy folio_test_locked() plus folio_lock() sequence with folio_trylock().
- Patch 9: guard PMD-swap UFFDIO_MOVE code with CONFIG_THP_SWAP, clarify RWP marker propagation, and reject a PMD swap entry at the destination with -EEXIST so UFFDIO_MOVE cannot loop forever on -EAGAIN.
- Patch 10: honor current THP/VMA policy before PMD-order swap-in, recheck that the PMD is still the original swap entry before splitting for PTE fallback, and provide the CONFIG_TRANSPARENT_HUGEPAGE wp_huge_pmd() declaration/stub needed by THP=n builds.
- Patch 11: make an invalid set_pmd_swap_entry() walk context warn and return -EINVAL instead of falsely reporting success and corrupting the MM_ANONPAGES/MM_SWAPENTS accounting, and add an exact PMD-size folio precondition check. (Luiz Capitulino)
- Patch 12: use /proc/swaps for prerequisite detection, check MADV_HUGEPAGE, and distinguish an environment that cannot allocate a PMD THP (SKIP) from a swap-out validation failure (FAIL). Add partial-mprotect and partial-munmap split coverage. Strengthen munmap/MADV_FREE VmSwap accounting, pagemap slot-offset checks, and mprotect/mremap swapped-state checks; force mremap to move, check munmap()'s return, and mark the UFFDIO_MOVE destination MADV_HUGEPAGE before asserting PMD restoration. Move common setup and cleanup into one fixture, merge the swapoff fixture, remove the redundant cycles test, and make the data pattern differ between base pages so the split tests can detect incorrect slot ordering. Order fork-COW so the parent writes while the child still holds the untouched shared swap entry. (Luiz Capitulino)
- Clarify commit messages throughout. Retain TTU_SPLIT_HUGE_PMD after prototyping its removal: removing it here requires an extra rmap walk and broadens the series beyond PMD swap entries. (Matthew Wilcox)
- Rebase onto akpm/mm-new from 15 August (4b65683fd25f).
v4 -> v5: https://lore.kernel.org/all/[email protected]/
- Commit message improvements for almost all patches (Yosry for zswap patch)
- Patch 1: make pmd_to_softleaf_folio() reject softleaf entries that do not encode a PFN, so a PMD swap offset is never interpreted as one. PMD swap entries remain valid softleaf entries for classification. (sashiko)
- Patch 2: use the existing pmd_swp_uffd() helper and force freeze=false for PMD swap entries, which have no struct page for the migration-entry freeze path. (sashiko)
- Patch 3: document that the caller's page-table or swap-cache reference pins every source slot while a partial PMD-sized duplication is rolled back. Keep the pre-existing PTE fork retry behavior outside this series. (sashiko)
- Patch 5: split to the PTE path rather than mapping a PMD-sized folio containing a hardware-poisoned subpage, and restore PAGE_NONE when swapoff restores a UFFD marker in an RWP VMA. (sashiko)
- Patch 6: account SwapPss for a PMD swap entry one slot at a time because the slots can have different swap reference counts.
- Patch 7: if PMD-order MADV_WILLNEED encounters newly populated per-page zswap state, revalidate and remove the failed clean PMD-sized cache folio before retrying through PTEs.
- Patch 8: mark a moved PMD swap entry for UFFD when the UFFDIO_MOVE destination VMA is RWP-registered. (sashiko)
- Patch 9: restore PAGE_NONE for UFFD RWP swap-in, preserve the original write-fault state through swap-slot release and COW handling, remove the unnecessary LRU drain, and prevent PTE batching from mapping a poisoned subpage. (sashiko)
- Patch 10: add and document thp_swpout_pmd, which counts PMD mappings replaced by PMD-level swap entries rather than swapped folios.
- Patch 11: register pmd_swap with the default mm selftest runner, preserve errno across UFFDIO_MOVE cleanup, check swapoff residency before the first memory access, add a parent-side write and verification to the fork+COW test, and add RWP regression coverage for swap-in, UFFDIO_MOVE, and swapoff. (sashiko)
- Keep do_huge_pmd_swap_page() in patch 9. Patches 6 and 7 only add consumers; patch 10 remains the first producer, so no PMD swap entry can reach those paths before the fault handler is present. (sashiko)
- Rebase onto latest akpm/mm-new from 22 July (5e0603ba185a)
v3 -> v4: https://lore.kernel.org/all/[email protected]/
- Patch 1: guard the new arch-specific pmd_swp_mkexclusive / pmd_swp_exclusive / pmd_swp_clear_exclusive helpers on arm64, loongarch, powerpc, riscv, s390, and x86 with CONFIG_ARCH_HAS_PMD_SOFTLEAVES, matching the pattern already
used for pmd_swp_soft_dirty. Also fixes the redefinition-vs- generic-fallback build errors kernel test robot reported on i386-allnoconfig-bpf and riscv-allnoconfig-bpf, and rewraps
the patch 1 commit message paragraphs to ~75 columns.
(sashiko, kernel test robot, Usama Arif)
- Patch 2: switch the trailing folio_remove_rmap_pmd() gate in __split_huge_pmd_locked() from *pmd to old_pmd, old_pmd retains
the original present-or-non-present classification for every
branch above. (sashiko)
- Patch 3: teach swap_retry_table_alloc() (and the underlying swap_extend_table_alloc()) to accept an nr parameter and scan
every slot in [ci_off, ci_off + nr) before committing an extend-table allocation. (sashiko)
- Patch 4: rename zswap_range_has_entry() to zswap_is_present() so
the same helper serves both single-slot (nr=1) and range queries, and switch the implementation from XA_STATE + xas_find() to xa_find(), which handles RCU locking and internal-retry markers itself. Rename the callers in patches 5, 7, 9. (Yosry)
- Patch 9: refuse to map a swap-cache folio in do_huge_pmd_swap_page() when folio_contain_hwpoisoned_page() reports a poisoned subpage; split the PMD swap entry so do_swap_page() can return VM_FAULT_HWPOISON per subpage instead of the PMD handler mapping
the corrupted memory as one THP. Mirrors the PageHWPoison check
the PTE swap-in path already performs. (sashiko)
- Patch 9: note explicitly in the commit message that PMD-order swap-in deliberately skips the order-0 readahead paths, order-0 readahead would populate per-page swap-cache state and force the
PMD swap entry to split before the fault could finish. (Kairui)
- Patch 10: move mm_prepare_for_swap_entries() into set_pmd_swap_entry() between folio_dup_swap() and set_pmd_at()
so this mm is on init_mm.mmlist before any swap PMD referencing slots with a non-zero swap_map becomes visible. Matches the PTE swap-out ordering. (sashiko)
- rebase onto latest akpm/mm-new (61cccb8363fcc282d4ae0555b8739dd227f5ad0b)
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Clarified the PMD swap entry rule: it is a compact encoding for HPAGE_PMD_NR swap slots, not a guarantee that swap cache always has one PMD-sized folio. (Lance Yang)
- Swapoff, fault, MADV_WILLNEED, and UFFDIO_MOVE now classify the whole PMD swap-cache range and split/retry through the PTE path for split/per-slot cache state. (Lance Yang)
- mincore handles PMD swap entries without assuming one lookup covers a split swap-cache range. (Lance Yang)
- UFFDIO_MOVE rechecks all HPAGE_PMD_NR slots before moving an empty PMD swap-cache range, avoiding stale rmap metadata for per-slot cached folios.
- Added a standalone zswap prerequisite patch from Alexandre that distinguishes all-on-disk large-folio ranges from ranges with per-page zswap entries.
- Replaced the global zswap-ever-enabled policy with per-range zswap
checks: PMD swap entries can still be installed while zswap is enabled, and PMD-order swap-in consumers split when the range has per-page zswap state.
- Added a mincore selftest and updated MADV_WILLNEED coverage so the test checks that the PMD swap entry remains in place until first touch. Total pmd_swap coverage is now 14 tests.
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Patch 1: convert two additional softleaf_to_pmd() callers that landed in mm-unstable since v1 (mm/debug_vm_pgtable.c, mm/migrate_device.c) (Dev)
- Patch 2: rename helper ensure_on_mmlist() to mm_prepare_for_swap_entries() to better describe its purpose
(David)
- Patch 3: drop VM_WARN_ON_ONCE(!pmd_is_migration_entry) as
Dev posted it as a separate patch.
- Patch 5 (new): move softleaf_to_folio() inside the device-private branch in migrate_vma_collect_pmd(); same class of fix as patch 4 but for the migrate-device PMD walker.
- Patch 6 (new): rename CONFIG_ARCH_ENABLE_THP_MIGRATION to CONFIG_ARCH_HAS_PMD_SOFTLEAVES so the gate that now drives swap-entry support too is named for what it actually controls
(PMD softleaf entries), not just migration. (Dev)
- Patch 7: add the missing pmd_swp_exclusive / mkexclusive / clear_exclusive helpers for powerpc.
- Patches 10 and 14: use upstream swapin_sync() (bundles swap_cache_alloc_folio + swap_read_folio + the -EEXIST race
retry) instead of the bespoke swapin_alloc_pmd_folio() helper
from v1; do_swap_page and shmem_swapin_folio use the same
helper (Kairui)
- Patch 10: construct a stack vm_fault for the swapoff swap-in so
the allocator can resolve a mempolicy, mirroring how the PTE
swapoff path (unuse_pte_range) already does it.
- Patch 11: extend coverage to check_pmd_state() in khugepaged so a swapped-out PMD-mapped THP is treated as SCAN_PMD_MAPPED (matches the existing migration-entry handling). Route PMD swap entries in the pmd_trans_huge_lock() branch of mincore_pte_range() through mincore_pmd_swap() so a swapped-out PMD-mapped THP isn't reported as resident.
- Patch 12 (new): handle PMD swap entries in MADV_WILLNEED via swapin_sync(BIT(HPAGE_PMD_ORDER)); a naive order-0 read-ahead
would force the subsequent fault to split.
- Patch 13: refuse UFFDIO_MOVE with -EBUSY if the swap-cache folio
was split between swap-out and the move, matching
move_pages_pte()'s rejection of large folios; otherwise only one
of the 512 anon-rmaps would be re-anchored to dst_vma.
- Patch 16: alloc_fill_swap_thp() now uses the existing mmap_pmd_aligned() helper so tests don't flake/skip based on VA placement; new MADV_WILLNEED test that watches the PMD-order
mTHP swpin counter; swapoff test restructured to use the kselftest_harness ASSERT cleanup blocks (no double swapoff, no verify-after-munmap).
- Collected Acks and Reviews
Usama Arif (32):
mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
arm64: mm: add PMD swap-exclusive helpers
loongarch: mm: add PMD swap-exclusive helpers
powerpc: mm: add PMD swap-exclusive helpers
riscv: mm: add PMD swap-exclusive helpers
s390: mm: add PMD swap-exclusive helpers
x86: mm: add PMD swap-exclusive helpers
mm: recognize PMD swap entries in the softleaf layer mm/debug_vm_pgtable: test PMD swap-exclusive helpers mm/migrate_device: avoid leaking folio references during THP splitting
mm: make PMD migration-entry splitting explicit
mm: split PMD swap entries into PTE swap entries
mm/swap: allow duplicating a range of swap entries
mm: handle PMD swap entries in fork path
mm: swap: share swap-cache allocation error cleanup
mm: zswap: reject high-order swap cache allocations backed by zswap
mm: swap in PMD swap entries as whole THPs during swapoff fs/proc: account PMD swap entries in smaps
mm: handle soft-dirty and uffd-wp on PMD swap entries mm/hmm: fault PMD swap entries on demand
mm: free PMD swap entries in zap_huge_pmd()
mm/madvise: free PMD swap entries with MADV_FREE mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE mm/mincore: report PMD swap-cache residency
mm/khugepaged: treat PMD swap entries as mapped THPs
mm: handle PMD swap entries in MADV_WILLNEED
mm: handle PMD swap entries in UFFDIO_MOVE
mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
mm: handle PMD swap entry faults on swap-in
mm: install PMD swap entries on swap-out
selftests/mm: add PMD swap entry tests
Documentation/admin-guide/mm/transhuge.rst | 5 + arch/arm64/include/asm/pgtable.h | 6 + arch/loongarch/include/asm/pgtable.h | 19 + arch/powerpc/include/asm/book3s/64/pgtable.h | 6 + arch/riscv/include/asm/pgtable.h | 15 + arch/s390/include/asm/pgtable.h | 20 +- arch/x86/include/asm/pgtable.h | 23 + fs/proc/task_mmu.c | 45 +- include/linux/huge_mm.h | 40 +- include/linux/leafops.h | 42 +- include/linux/pgtable.h | 17 + include/linux/swap.h | 12 +- include/linux/swapops.h | 10 + include/linux/vm_event_item.h | 1 + mm/debug_vm_pgtable.c | 44 +- mm/hmm.c | 11 +- mm/huge_memory.c | 722 ++++++++++++-- mm/internal.h | 61 ++ mm/khugepaged.c | 6 + mm/madvise.c | 169 +++- mm/memory.c | 65 +- mm/migrate_device.c | 7 +- mm/mincore.c | 47 +- mm/mprotect.c | 2 +- mm/rmap.c | 27 +- mm/swap.h | 30 +- mm/swap_state.c | 83 +- mm/swapfile.c | 245 ++++- mm/userfaultfd.c | 14 + mm/vmscan.c | 9 +- mm/vmstat.c | 1 + tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 +++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 37 files changed, 2640 insertions(+), 189 deletions(-) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 1/32 mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What it does is decode the softleaf entry stored in the PMD and return the folio that entry references - the direction softleaf_to_folio() already spells out.
No functional change intended.
--- include/linux/leafops.h | 4 ++-- mm/huge_memory.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
include/linux/leafops.h +2 -2
@@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd) }/** -* pmd_to_softleaf_folio() - Convert the PMD entry to a folio. +* pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * * The PMD entry is expected to be a valid PMD softleaf entry.
@@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)* Returns: the folio the softleaf entry references if this is a valid softleaf * entry, otherwise NULL. */ -static inline struct folio *pmd_to_softleaf_folio (pmd_t pmd) +static inline struct folio *pmd_softleaf_to_folio (pmd_t pmd) {const softleaf_t entry =softleaf_from_pmd (pmd);
mm/huge_memory.c +1 -1
@@ -2516,7 +2516,7 @@ static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,if (!thp_migration_supported ())WARN_ONCE (1 ,"Non present huge pmd without pmd migration enabled!" ); -return pmd_to_softleaf_folio (pmdval); +return pmd_softleaf_to_folio (pmdval); }static bool vma_has_deposited_pgtable (struct vm_area_struct *vma)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 2/32 arm64: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
arm64 encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse PTE_SWP_EXCLUSIVE.
--- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/include/asm/pgtable.h +6 -0
@@ -560,6 +560,12 @@ static inline int pmd_protnone(pmd_t pmd)#define pmd_swp_clear_uffd (pmd) \ pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive (pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif #define pmd_write (pmd) pte_write(pmd_pte(pmd))
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 3/32 loongarch: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
A LoongArch swap PMD is the swap PTE value plus _PAGE_HUGE, and _PAGE_SWP_EXCLUSIVE sits outside both the type and the offset field, so the PMD helpers can use the same bit.
--- arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/loongarch/include/asm/pgtable.h +19 -0
@@ -352,6 +352,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SWP_EXCLUSIVE; +return pmd; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_SWP_EXCLUSIVE; +return pmd; +} +#endif +#define pte_none (pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_no_exec (pte) (pte_val(pte) & _PAGE_NO_EXEC)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 4/32 powerpc: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
book3s64 builds a swap PMD by running the PTE encoding over pmd_pte(), so the PMD helpers wrap the PTE ones and reuse _PAGE_SWP_EXCLUSIVE, exactly as the soft-dirty PMD helpers next to them do. It is also the only powerpc variant that selects ARCH_HAS_PMD_SOFTLEAVES, via PPC_THP.
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/powerpc/include/asm/book3s/64/pgtable.h +6 -0
@@ -1060,6 +1060,12 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#endif #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_clear_exclusive (pmd) pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif +#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE static inline int pmd_protnone (pmd_t pmd) {
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 5/32 riscv: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
riscv encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse _PAGE_SWP_EXCLUSIVE.
--- arch/riscv/include/asm/pgtable.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/riscv/include/asm/pgtable.h +15 -0
@@ -1219,6 +1219,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) }#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pte_swp_exclusive (pmd_pte (pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_mkexclusive (pmd_pte (pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_clear_exclusive (pmd_pte (pmd))); +} +#define __pmd_to_swp_entry (pmd) ((swp_entry_t) { pmd_val(pmd) })#define __swp_entry_to_pmd (swp) __pmd((swp).val)#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 6/32 s390: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap entry for the common code. Give it its own exclusive bit rather than borrowing the PTE-format macro, and document it in the RSTE swap layout, where bit 52 was listed as unused.
--- arch/s390/include/asm/pgtable.h | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
arch/s390/include/asm/pgtable.h +18 -2
@@ -333,6 +333,7 @@ void setup_protection_map(void);/* Common bits in region and segment table entries, for swap entries */ #define _RST_ENTRY_COMM 0x0010/* Common-Region/Segment, marks swap entry */ #define _RST_ENTRY_INVALID 0x0020/* invalid region/segment table entry */ +#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800/* SW exclusive swap bit, see mk_swap_rste() */ #define _CRST_ENTRIES 2048/* number of region/segment table entries */ #define _PAGE_ENTRIES 256/* number of page table entries */
@@ -893,6 +894,21 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd) }#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _RST_ENTRY_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return set_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return clear_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} +#define pmd_swp_soft_dirty (pmd) pmd_soft_dirty(pmd)#define pmd_swp_mksoft_dirty (pmd) pmd_mksoft_dirty(pmd)#define pmd_swp_clear_soft_dirty (pmd) pmd_clear_soft_dirty(pmd)
@@ -1900,15 +1916,15 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)* Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste * as invalid. * A swap entry is indicated by bit pattern (rste & 0x011) == 0x010 -* | offset |Xtype |11TT|S0| +* | offset |Etype |11TT|S0| * |0000000000111111111122222222223333333333444444444455|555555|5566|66| * |0123456789012345678901234567890123456789012345678901|234567|8901|23| * * Bits 0-51 store the offset. +* Bit 52 (E) is used to remember PG_anon_exclusive (_RST_ENTRY_SWP_EXCLUSIVE). * Bits 53-57 store the type. * Bit 62 (S) is used for softdirty tracking. * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT. -* Bit 52 (X) is unused. */ #define __SWP_OFFSET_MASK_RSTE ((1UL << 52) - 1)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap entry for the common code. Give it its own exclusive bit rather than borrowing the PTE-format macro, and document it in the RSTE swap layout, where bit 52 was listed as unused.
Quoted 6 lines
Cc Alexander Gordeev <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Vasily Gorbik <[email protected]>Signed-off-by Usama Arif <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers
In reply to Usama ArifQuoted 62 lines · On Thu, 5 Nov 2026 10:38:50 -0800 Usama Arif <[email protected]> wrote:
Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes
PG_anon_exclusivehave to survive in a swap PMD and not just in a swap PTE.s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap entry for the common code. Give it its own exclusive bit rather than borrowing the PTE-format macro, and document it in the RSTE swap layout, where bit 52 was listed as unused.Cc Alexander Gordeev <[email protected]>Cc Gerald Schaefer <[email protected]>Cc Heiko Carstens <[email protected]>Cc Vasily Gorbik <[email protected]>Signed-off-by Usama Arif <[email protected]>--- arch/s390/include/asm/pgtable.h | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)arch/s390/include/asm/pgtable.h +17
-1@@ -333,6 +333,7 @@ void setup_protection_map(void);/* Common bits in region and segment table entries, for swap entries */ #define _RST_ENTRY_COMM 0x0010/* Common-Region/Segment, marks swap entry */ #define _RST_ENTRY_INVALID 0x0020/* invalid region/segment table entry */ +#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800/* SW exclusive swap bit, see mk_swap_rste() */ #define _CRST_ENTRIES 2048/* number of region/segment table entries */ #define _PAGE_ENTRIES 256/* number of page table entries */ @@ -893,6 +894,21 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd) }#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _RST_ENTRY_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return set_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return clear_pmd_bit (pmd,__pgprot (_RST_ENTRY_SWP_EXCLUSIVE)); +} +#define pmd_swp_soft_dirty (pmd) pmd_soft_dirty(pmd)#define pmd_swp_mksoft_dirty (pmd) pmd_mksoft_dirty(pmd)#define pmd_swp_clear_soft_dirty (pmd) pmd_clear_soft_dirty(pmd)@@ -1900,15 +1916,15 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)* Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste * as invalid. * A swap entry is indicated by bit pattern (rste & 0x011) == 0x010 -* | offset |Xtype |11TT|S0| +* | offset |Etype |11TT|S0| * |0000000000111111111122222222223333333333444444444455|555555|5566|66| * |0123456789012345678901234567890123456789012345678901|234567|8901|23|
arch/s390/include/asm/pgtable.h static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset) · line 1930
Quoted 10 lines
arch/s390/include/asm/pgtable.h +1
-1* * Bits 0-51 store the offset. +* Bit 52 (E) is used to remember PG_anon_exclusive (_RST_ENTRY_SWP_EXCLUSIVE). * Bits 53-57 store the type. * Bit 62 (S) is used for softdirty tracking. * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT. -* Bit 52 (X) is unused. */ #define __SWP_OFFSET_MASK_RSTE ((1UL << 52) - 1)
Nice, you found the last free bit in s390 RST swap entry, and made good use of it. Bit 52 would also be free in regular RST entries, but as far as I remember from PTE SWP_EXCLUSIVE, this will only ever be checked or set for swap entries, right? Otherwise we should also document and reserve for regular RST entries, but it should be safe for now in any case.
Also did some basic tests with your series on s390. Couldn't see an increase in throughput yet, but my swap device was way too slow an a bottleneck itself. Verified that thp_swpout_pmd counter was increased, system runs stable, found no problems so far.
I think this series could give s390 some considerable benefit, besides swap throughput, by saving the 256 IPTEs (PTE invalidations) for each 1MB THP that doesn't need to get split before unmap and swap out. This could have a positive system-wide impact, not just for one guest.
Great job, thanks for the effort!
Usama Arif
reviewerPATCH v9 7/32 x86: mm: add PMD swap-exclusive helpers
In reply to Usama ArifPrepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, which makes PG_anon_exclusive have to survive in a swap PMD and not just in a swap PTE.
x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse _PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves for PG_anon_exclusive. 32-bit x86 aliases that bit to _PAGE_PSE and does not select ARCH_HAS_PMD_SOFTLEAVES; assert that, because an exclusive swap PMD would otherwise be pmd_present().
--- arch/x86/include/asm/pgtable.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
arch/x86/include/asm/pgtable.h +23 -0
@@ -1529,6 +1529,29 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_EXCLUSIVE); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +/* +* _PAGE_SWP_EXCLUSIVE must not alias _PAGE_PSE, which would make an +* exclusive swap PMD present. +*/ +static_assert (_PAGE_SWP_EXCLUSIVE != _PAGE_PSE); + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty (pte_t pte) {
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 8/32 mm: recognize PMD swap entries in the softleaf layer
In reply to Usama ArifReclaim splits a PMD-mapped anonymous THP into PTE-level swap entries before unmapping it, so an ordinary swap entry has never had to appear in a PMD. Prepare for installing one there instead: the softleaf layer is where every consumer decodes non-present PMDs.
Accept swap entries as valid PMD softleaves and add pmd_is_swap_entry(). A swap entry carries no PFN, so make pmd_softleaf_to_folio() warn and return NULL rather than interpret a swap offset as a page frame number.
Unlike migration and device-private entries, a PMD swap entry can also carry the swap-exclusive marker, which softleaf_from_pmd() has to strip before decoding. Strip all three overlays unconditionally while we are
here: each clear is a plain bit clear, so testing first only buys a branch.
--- include/linux/leafops.h | 38 ++++++++++++++++++++++++++------------ include/linux/pgtable.h | 17 +++++++++++++++++ include/linux/swapops.h | 10 ++++++++++ 3 files changed, 53 insertions(+), 12 deletions(-)
include/linux/leafops.h +26 -12
@@ -98,10 +98,7 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)if (pmd_present (pmd) ||pmd_none (pmd))return softleaf_mk_none (); -if (pmd_swp_soft_dirty (pmd)) - pmd =pmd_swp_clear_soft_dirty (pmd); -if (pmd_swp_uffd (pmd)) - pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_flags (pmd); arch_entry =__pmd_to_swp_entry (pmd);/* Temporary until swp_entry_t eliminated. */
@@ -634,18 +631,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)*/ static inline bool softleaf_is_valid_pmd_entry (softleaf_t entry) { -/* Only device private, migration entries valid for PMD. */ return softleaf_is_device_private (entry) || -softleaf_is_migration (entry); +softleaf_is_migration (entry) || +softleaf_is_swap (entry); +} + +/** +* pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? +* @pmd: PMD entry. +* +* Returns: true if the PMD encodes a swap entry, otherwise false. +*/ +static inline bool pmd_is_swap_entry (pmd_t pmd) +{ +return softleaf_is_swap (softleaf_from_pmd (pmd)); }/** * pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry? * @pmd: PMD entry. * -* PMD leaf entries are valid only if they are device private or migration -* entries. This function asserts that a PMD leaf entry is valid in this -* respect. +* PMD leaf entries are valid only if they are device private, migration, +* or swap entries. This function asserts that a PMD leaf entry is valid +* in this respect. * * Returns: true if the PMD entry is a valid leaf entry, otherwise false. */
@@ -660,10 +668,12 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)* pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * -* The PMD entry is expected to be a valid PMD softleaf entry. +* The PMD entry is expected to be a valid PMD softleaf entry that references a +* PFN, that is a migration or device private entry. A PMD swap entry is a valid +* softleaf entry but encodes swap slots rather than a PFN, so it has no folio. * -* Returns: the folio the softleaf entry references if this is a valid softleaf -* entry, otherwise NULL. +* Returns: the folio the softleaf entry references, or NULL if the entry is not +* a valid PMD softleaf entry or does not reference a PFN. */ static inline struct folio *pmd_softleaf_to_folio (pmd_t pmd) {
@@ -673,6 +683,10 @@ static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)VM_WARN_ON_ONCE (true);return NULL ; } +if (!softleaf_has_pfn (entry)) { +VM_WARN_ON_ONCE (true); +return NULL ; + }return softleaf_to_folio (entry); }
include/linux/pgtable.h +17 -0
@@ -1917,6 +1917,23 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) }#endif +#ifndef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd; +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return false; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd; +} +#endif +#ifndef __HAVE_PFNMAP_TRACKING/* * Interfaces that can be used by architecture code to keep track of
include/linux/swapops.h +10 -0
@@ -79,6 +79,16 @@ static inline pte_t pte_swp_clear_flags(pte_t pte)return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_clear_flags (pmd_t pmd) +{ + pmd =pmd_swp_clear_soft_dirty (pmd); + pmd =pmd_swp_clear_uffd (pmd); + pmd =pmd_swp_clear_exclusive (pmd); +return pmd; +} +#endif +/* * Store a type+offset into a swp_entry_t in an arch-independent format */
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 9/32 mm/debug_vm_pgtable: test PMD swap-exclusive helpers
In reply to Usama ArifAn architecture that picked a PMD exclusive bit overlapping the swap type or offset field would otherwise only be caught by data corruption at runtime. Mirror pte_swap_exclusive_tests() at PMD level.
Also check that setting swap soft-dirty leaves the PMD non-present.
--- mm/debug_vm_pgtable.c | 44 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-)
mm/debug_vm_pgtable.c +43 -1
@@ -762,7 +762,9 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_is_huge (pmd));WARN_ON (!pmd_is_valid_softleaf (pmd)); -WARN_ON (!pmd_swp_soft_dirty (pmd_swp_mksoft_dirty (pmd))); + pmd =pmd_swp_mksoft_dirty (pmd); +WARN_ON (!pmd_swp_soft_dirty (pmd)); +WARN_ON (pmd_present (pmd)); WARN_ON(pmd_swp_soft_dirty(pmd_swp_clear_soft_dirty(pmd))); }#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
@@ -802,6 +804,45 @@ static void __init pte_swap_exclusive_tests(struct pgtable_debug_args *args)WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static void __init pmd_swap_exclusive_tests (struct pgtable_debug_args *args) +{ +swp_entry_t entry; +softleaf_t softleaf; +pmd_t pmd; + +if (!has_transparent_hugepage ()) +return ; + +pr_debug ("Validating PMD swap exclusive\n" ); + entry = args->swp_entry; + + pmd =softleaf_to_pmd (entry); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); + + pmd =pmd_swp_mkexclusive (pmd); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (!pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (pmd_swp_soft_dirty (pmd)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); + + pmd =pmd_swp_clear_exclusive (pmd); + softleaf =softleaf_from_pmd (pmd); + +WARN_ON (pmd_swp_exclusive (pmd)); +WARN_ON (!softleaf_is_swap (softleaf)); +WARN_ON (memcmp (&entry, &softleaf,sizeof (entry))); +} +#else /* !CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ +static void __init pmd_swap_exclusive_tests (struct pgtable_debug_args *args) { } +#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ +static void __init pte_swap_tests (struct pgtable_debug_args *args) {swp_entry_t arch_entry;
@@ -1322,6 +1363,7 @@ static int __init debug_vm_pgtable(void)pmd_leaf_soft_dirty_tests (&args);pte_swap_exclusive_tests (&args); +pmd_swap_exclusive_tests (&args);pte_swap_tests (&args);pmd_softleaf_tests (&args);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 10/32 mm/migrate_device: avoid leaking folio references during THP splitting
In reply to Usama Arifmigrate_vma_split_unmapped_folio() takes an extra folio reference because split_huge_pmd_address(..., true) drops one when it splits a PMD.
However, migrate_vma only holds mmap_lock for reading. A concurrent MADV_DONTNEED can clear the migration PMD, or split it into PTE migration entries, before the split helper takes the PMD lock. The helper then does nothing and leaves the extra reference behind. folio_split_unmapped() fails with -EAGAIN due to the unexpected reference, and the reference is never dropped.
An existing migration PMD is expanded into PTE migration entries even with freeze=false. Use that mode and remove the extra folio_get(). The reference already held by migration keeps the source folio alive whether or not its PMD migration entry still exists.
--- mm/migrate_device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
mm/migrate_device.c +1 -6
@@ -920,12 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; -/* -* take a reference, since split_huge_pmd_address() with freeze = true -* drops a reference at the end. -*/ -folio_get (folio); -split_huge_pmd_address (migrate->vma, addr, true); +split_huge_pmd_address (migrate->vma, addr, false); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 11/32 mm: make PMD migration-entry splitting explicit
In reply to Usama Arif__split_huge_pmd() and friends take a "freeze" boolean that every caller has to pass and almost every caller passes as false. The name says nothing about what it selects, and the one thing it does select - PTE migration entries instead of PTE mappings - is only ever wanted by the rmap migration path.
Rename it to to_migration_entries, keep it private to mm/huge_memory.c, and add split_pmd_to_migration_entries() for try_to_migrate_one(), the only caller that wants it.
split_pmd_to_migration_entries() is only ever handed a present or device-private PMD, so assert that in __split_huge_pmd_locked() instead of silently skipping anything else.
Other than that assertion, no functional change intended.
--- include/linux/huge_mm.h | 21 ++++++------ mm/huge_memory.c | 74 ++++++++++++++++++++++++++--------------- mm/memory.c | 4 +-- mm/migrate_device.c | 2 +- mm/mprotect.c | 2 +- mm/rmap.c | 8 ++--- 6 files changed, 65 insertions(+), 46 deletions(-)
include/linux/huge_mm.h +11 -10
@@ -431,7 +431,7 @@ int folio_memcg_alloc_deferred(struct folio *folio);void deferred_split_folio (struct folio *folio,bool partially_mapped);void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze); +unsigned long address);/** * pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf entry?
@@ -463,12 +463,10 @@ static inline bool pmd_is_huge(pmd_t pmd)do { \pmd_t *____pmd = (__pmd); \if (pmd_is_huge (*____pmd)) \ -__split_huge_pmd (__vma, __pmd, __address, \ - false); \ +__split_huge_pmd (__vma, __pmd, __address); \ } while (0) -void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address, -bool freeze); +void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address);void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud,unsigned long address);
@@ -593,7 +591,9 @@ static inline bool thp_migration_supported(void) }void split_huge_pmd_locked (struct vm_area_struct *vma,unsigned long address, -pmd_t *pmd,bool freeze); +pmd_t *pmd); +void split_pmd_to_migration_entries (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd);bool unmap_huge_pmd_locked (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmdp,struct folio *folio); void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,
@@ -693,12 +693,13 @@ static inline void deferred_split_folio(struct folio *folio, bool partially_mapp do { } while (0)static inline void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long address,bool freeze) {} +unsigned long address) {}static inline void split_huge_pmd_address (struct vm_area_struct *vma, -unsigned long address,bool freeze) {} +unsigned long address) {}static inline void split_huge_pmd_locked (struct vm_area_struct *vma, -unsigned long address,pmd_t *pmd, -bool freeze) {} +unsigned long address,pmd_t *pmd) {} +static inline void split_pmd_to_migration_entries (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmd) {} static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp,
mm/huge_memory.c +47 -27
171 lines not on this page · raw message
mm/memory.c +2 -2
@@ -2099,7 +2099,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, next =pmd_addr_end (addr, end);if (pmd_is_huge (*pmd)) {if (next - addr !=HPAGE_PMD_SIZE ) -__split_huge_pmd (vma, pmd, addr, false); +__split_huge_pmd (vma, pmd, addr);else if (zap_huge_pmd (tlb, vma, pmd, addr)) { addr = next;continue ;
@@ -6491,7 +6491,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) split:/* COW or write-notify handled on pte level: split pmd. */ -__split_huge_pmd (vma, vmf->pmd, vmf->address, false); +__split_huge_pmd (vma, vmf->pmd, vmf->address);return VM_FAULT_FALLBACK ; }
mm/migrate_device.c +1 -1
@@ -920,7 +920,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,unsigned long flags;int ret =0 ; -split_huge_pmd_address (migrate->vma, addr, false); +split_huge_pmd_address (migrate->vma, addr); ret =folio_split_unmapped (folio,0 );if (ret)return ret;
mm/mprotect.c +1 -1
@@ -530,7 +530,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb,if (pmd_is_huge (_pmd)) {if ((next - addr !=HPAGE_PMD_SIZE ) ||pgtable_split_needed (vma, cp_flags)) { -__split_huge_pmd (vma, pmd, addr, false); +__split_huge_pmd (vma, pmd, addr);/* * For file-backed, the pmd could have been * cleared; make sure pmd populated if
mm/rmap.c +3 -5
@@ -2280,8 +2280,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* We temporarily have to drop the PTL and * restart so we can process the PTE-mapped THP. */ -split_huge_pmd_locked (vma, pvmw.address, - pvmw.pmd, false); +split_huge_pmd_locked (vma, pvmw.address, pvmw.pmd); flags &= ~TTU_SPLIT_HUGE_PMD ;page_vma_mapped_walk_restart (&pvmw);continue ;
@@ -2506,13 +2505,12 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,if (flags &TTU_SPLIT_HUGE_PMD ) {/* -* split_huge_pmd_locked() might leave the +* split_pmd_to_migration_entries() might leave the * folio mapped through PTEs. Retry the walk * so we can detect this scenario and properly * abort the walk. */ -split_huge_pmd_locked (vma, pvmw.address, - pvmw.pmd, true); +split_pmd_to_migration_entries (vma, pvmw.address, pvmw.pmd); flags &= ~TTU_SPLIT_HUGE_PMD ;page_vma_mapped_walk_restart (&pvmw);continue ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 12/32 mm: split PMD swap entries into PTE swap entries
In reply to Usama ArifOnce a PMD can hold a swap entry, everything that splits a PMD - mprotect() or munmap() over part of the range, MADV_FREE, a pagewalk with no PMD handler - has to be able to split that entry too. A swap PMD already passes the pmd_is_valid_softleaf() gate, so without this it reaches __split_huge_pmd_locked() and falls through to the present-PMD path, which pmdp_invalidate()s it and calls pmd_page() on a non-present entry.
No reference counting is needed: a swap entry pins no folio, and swap_map is already one per slot, so the PTEs simply take over what the PMD held.
The migration-only entry point cannot reach the new branch:page_vma_mapped_walk() never hands back a swap PMD, and __split_huge_pmd_locked() already asserts that to_migration_entries implies a present or device-private PMD.
Test the pre-split old_pmd rather than re-reading *pmd in the trailing folio_remove_rmap_pmd() gate, so every entry-type test in the function interrogates the same snapshot. That part is cosmetic: pmdp_invalidate() leaves the PMD present as far as software is concerned.
--- mm/huge_memory.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
mm/huge_memory.c +21 -1
@@ -3292,6 +3292,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,folio_add_anon_rmap_ptes (folio, page,HPAGE_PMD_NR , vma, haddr, rmap_flags); } + }else if (pmd_is_swap_entry (*pmd)) { + old_pmd = *pmd; + soft_dirty =pmd_swp_soft_dirty (old_pmd); + uffd_wp =pmd_swp_uffd (old_pmd); + anon_exclusive =pmd_swp_exclusive (old_pmd); }else {/* * Up to this point the pmd is present and huge and userland has
@@ -3429,6 +3434,21 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,VM_WARN_ON (!pte_none (ptep_get (pte + i)));set_pte_at (mm, addr, pte + i, entry); } + }else if (pmd_is_swap_entry (old_pmd)) { +pte_t entry =softleaf_to_pte (softleaf_from_pmd (old_pmd)); + +if (soft_dirty) + entry =pte_swp_mksoft_dirty (entry); +if (uffd_wp) + entry =pte_swp_mkuffd (entry); +if (anon_exclusive) + entry =pte_swp_mkexclusive (entry); + +for (i =0 , addr = haddr; i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE ) { +VM_WARN_ON (!pte_none (ptep_get (pte + i))); +set_pte_at (mm, addr, pte + i, entry); + entry =pte_next_swp_offset (entry); + } }else {pte_t entry;
@@ -3456,7 +3476,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, }pte_unmap (pte); -if (!pmd_is_migration_entry (*pmd)) +if (!pmd_is_migration_entry (old_pmd) && !pmd_is_swap_entry (old_pmd))folio_remove_rmap_pmd (folio, page, vma);if (to_migration_entries)put_page (page);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 13/32 mm/swap: allow duplicating a range of swap entries
In reply to Usama Arifswap_dup_entry_direct() duplicates one slot at a time. The PMD swap entry fork path needs HPAGE_PMD_NR of them, and doing that one slot at a time would take and drop the cluster lock HPAGE_PMD_NR times.
Give it an @nr argument and rename it swap_dup_entries_direct(), and do the same for swap_retry_table_alloc(), whose GFP_KERNEL retry has to cover the same range - the caller does not know which slot in it overflowed. Keep the old single-slot names as inline wrappers so existing callers are untouched.
Unlike the put side, @nr is handed straight to the per-cluster helper, so the range has to sit inside one cluster. That holds for the only caller passing nr > 1: a PMD swap entry only exists under CONFIG_THP_SWAP, where SWAPFILE_CLUSTER == HPAGE_PMD_NR, and a PMD-order folio's slots are only ever allocated at a cluster head (see alloc_swap_scan_cluster()), so the range is exactly one cluster. Reject a crossing range with -EINVAL so a future caller cannot walk off the end of the swap table.
No functional change intended.
--- include/linux/swap.h | 12 +++++++++- mm/swap.h | 13 +++++++++- mm/swapfile.c | 56 +++++++++++++++++++++++++++++++++----------- 3 files changed, 65 insertions(+), 16 deletions(-)
include/linux/swap.h +11 -1
@@ -417,9 +417,14 @@ sector_t swap_folio_sector(struct folio *folio);* All entries must be allocated by folio_alloc_swap(). And they must have * a swap count > 1. See comments of folio_*_swap helpers for more info. */ -int swap_dup_entry_direct (swp_entry_t entry); +int swap_dup_entries_direct (swp_entry_t entry,int nr);void swap_put_entries_direct (swp_entry_t entry,int nr); +static inline int swap_dup_entry_direct (swp_entry_t entry) +{ +return swap_dup_entries_direct (entry,1 ); +} +/* * folio_free_swap tries to free the swap entries pinned by a swap cache * folio, it has to be here to be called by other components.
@@ -464,6 +469,11 @@ static inline void free_swap_cache(struct folio *folio) { } +static inline int swap_dup_entries_direct (swp_entry_t ent,int nr) +{ +return 0 ; +} +static inline int swap_dup_entry_direct (swp_entry_t ent) {return 0 ;
mm/swap.h +12 -1
@@ -223,7 +223,12 @@ static inline void swap_cluster_unlock_irq(struct swap_cluster_info *ci)spin_unlock_irq (&ci->lock); } -extern int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp); +int swap_retry_table_alloc_nr (swp_entry_t entry,unsigned int nr,gfp_t gfp); + +static inline int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp) +{ +return swap_retry_table_alloc_nr (entry,1 , gfp); +}/* * Below are the core routines for doing swap for a folio.
@@ -453,6 +458,12 @@ static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)return 0 ; } +static inline int swap_retry_table_alloc_nr (swp_entry_t entry,unsigned int nr, +gfp_t gfp) +{ +return -EINVAL ; +} +static inline int swap_retry_table_alloc (swp_entry_t entry,gfp_t gfp) {return -EINVAL ;
mm/swapfile.c +42 -14
141 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 14/32 mm: handle PMD swap entries in fork path
In reply to Usama Arifcopy_huge_pmd() only knows about migration and device-private PMDs, so a PMD swap entry would fall through to the present-PMD path and fork() would duplicate it without taking a reference on the slots it points at.
Copy it the way copy_nonpresent_pte() copies a PTE swap entry: duplicate the swap references, clear the exclusive marker on the source, put the destination mm on mmlist, and account the child's slots to MM_SWAPENTS.
The GFP_ATOMIC extend-table allocation inside the dup can fail. Report that as -EIO and let copy_pmd_range() retry with GFP_KERNEL, as copy_nonpresent_pte() and copy_pte_range() already do for a PTE swap entry. copy_huge_pmd() hands the entry back so the caller knows which range to allocate for.
Only -ENOMEM is reported that way. The other failures mean the entry itself is bad, and swap_retry_table_alloc_nr() returns 0 for those, so collapsing them into -EIO as the PTE path does would spin in the caller's retry rather than failing the fork.
While here, move the mm counter update into each entry-type arm, as the PTE version does, so the swap arm can account MM_SWAPENTS instead of MM_ANONPAGES.
--- include/linux/huge_mm.h | 3 +- mm/huge_memory.c | 65 ++++++++++++++++++++++++++++++----------- mm/memory.c | 12 +++++++- 3 files changed, 61 insertions(+), 19 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -10,7 +10,8 @@vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf);int copy_huge_pmd (struct mm_struct *dst_mm,struct mm_struct *src_mm,pmd_t *dst_pmd,pmd_t *src_pmd,unsigned long addr, -struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma); +struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, +softleaf_t *entryp);bool huge_pmd_set_accessed (struct vm_fault *vmf); int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr,
mm/huge_memory.c +48 -17
103 lines not on this page · raw message
mm/memory.c +11 -1
@@ -1437,11 +1437,21 @@ copy_pmd_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, do { next = pmd_addr_end(addr, end); if (pmd_is_huge(*src_pmd)) { + softleaf_t entry = softleaf_mk_none(); int err; VM_BUG_ON_VMA(next-addr != HPAGE_PMD_SIZE, src_vma); +again: err = copy_huge_pmd(dst_mm, src_mm, dst_pmd, src_pmd, - addr, dst_vma, src_vma); + addr, dst_vma, src_vma, &entry); + if (err == -EIO) { + VM_WARN_ON_ONCE(!entry.val); + if (swap_retry_table_alloc_nr(entry, + HPAGE_PMD_NR, + GFP_KERNEL) < 0) + return -ENOMEM; + goto again; + } if (err == -ENOMEM) return -ENOMEM; if (!err)
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 15/32 mm: swap: share swap-cache allocation error cleanup
In reply to Usama ArifBoth post-insertion allocation failures remove the folio from the swap cache, restore its shadow and drop the folio references. Factor the common cleanup into one error path so additional checks can use it.
--- mm/swap_state.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
mm/swap_state.c +13 -14
@@ -484,24 +484,14 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); + err = -ENOMEM ;count_mthp_stat (order,MTHP_STAT_SWPIN_FALLBACK_CHARGE ); -return ERR_PTR (-ENOMEM ); + goto delete_folio; }if (order >1 &&folio_memcg_alloc_deferred (folio)) { -spin_lock (&ci->lock); -__swap_cache_do_del_folio (ci, folio, entry, shadow); -spin_unlock (&ci->lock); -folio_unlock (folio); -/* nr_pages refs from swap cache, 1 from allocation */ -folio_put_refs (folio, nr_pages +1 ); -return ERR_PTR (-ENOMEM ); + err = -ENOMEM ; + goto delete_folio; }/* memsw uncharges swap when folio is added to swap cache */
@@ -513,6 +503,15 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,lruvec_stat_mod_folio (folio,NR_SWAPCACHE , nr_pages);return folio; + +delete_folio: +spin_lock (&ci->lock); +__swap_cache_do_del_folio (ci, folio, entry, shadow); +spin_unlock (&ci->lock); +folio_unlock (folio); +/* nr_pages refs from swap cache, 1 from allocation */ +folio_put_refs (folio, nr_pages +1 ); +return ERR_PTR (err); }/**
Signature · 1 line
2.53.0-Meta
Re: [PATCH v9 15/32] mm: swap: share swap-cache allocation error cleanup
Quoted 5 lines · Usama Arif
Both post-insertion allocation failures remove the folio from the swap cache, restore its shadow and drop the folio references. Factor the common cleanup into one error path so additional checks can use it.
Signed-off-by Usama Arif <[email protected]>
Re: [PATCH v9 15/32] mm: swap: share swap-cache allocation error cleanup
In reply to Usama ArifQuoted 6 lines · Usama Arif
Both post-insertion allocation failures remove the folio from the swap cache, restore its shadow and drop the folio references. Factor the common cleanup into one error path so additional checks can use it.
Signed-off-by Usama Arif <[email protected]>
Nice cleanup.
Usama Arif
reviewerPATCH v9 16/32 mm: zswap: reject high-order swap cache allocations backed by zswap
In reply to Usama Arifzswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Reject that in __swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, which zswap_load() handles per page.
--- mm/swap_state.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
mm/swap_state.c +12 -0
@@ -12,6 +12,7 @@#include <linux/kernel_stat.h> #include <linux/mempolicy.h> #include <linux/swap.h> +#include <linux/zswap.h> #include <linux/leafops.h> #include <linux/init.h> #include <linux/pagemap.h>
@@ -482,6 +483,17 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,__swap_cache_do_add_folio (ci, folio, entry);spin_unlock (&ci->lock); +/* +* Now that the folio is in the swap cache, zswap can no longer start +* storing or writing back any slot in the range, so this is a stable +* answer. Reject a high-order allocation over a range that already +* has per-page zswap entries. +*/ +if (order &&zswap_is_present (entry, nr_pages)) { + err = -EBUSY ; + goto delete_folio; + } +if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { err = -ENOMEM ;
Signature · 1 line
2.53.0-Meta
Re: [PATCH v9 16/32] mm: zswap: reject high-order swap cache allocations backed by zswap
Quoted 6 lines · Usama Arif
zswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Quoted 8 lines
Reject that in
__swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, whichzswap_load()handles per page.Co-developed-by Alexandre Ghiti <[email protected]>Signed-off-by Alexandre Ghiti <[email protected]>Signed-off-by Usama Arif <[email protected]>
Re: [PATCH v9 16/32] mm: zswap: reject high-order swap cache allocations backed by zswap
In reply to Usama ArifQuoted 30 lines · Usama Arif
zswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns.
Reject that in
__swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, whichzswap_load()handles per page.Co-developed-by Alexandre Ghiti <[email protected]>Signed-off-by Alexandre Ghiti <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/swap_state.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)mm/swap_state.c +1
-0@@ -12,6 +12,7 @@#include <linux/kernel_stat.h> #include <linux/mempolicy.h> #include <linux/swap.h> +#include <linux/zswap.h> #include <linux/leafops.h> #include <linux/init.h> #include <linux/pagemap.h>
mm/swap_state.c static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, · line 490
Quoted 8 lines
mm/swap_state.c +5
-0@@ -482,6 +483,17 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,__swap_cache_do_add_folio (ci, folio, entry);spin_unlock (&ci->lock); +/* +* Now that the folio is in the swap cache, zswap can no longer start +* storing or writing back any slot in the range, so this is a stable +* answer. Reject a high-order allocation over a range that already +* has per-page zswap entries.
Probably worth mentioning *why* we reject the folio is part of it is in zswap.
mm/swap_state.c static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, · line 492
Quoted 2 lines
mm/swap_state.c +2
-0+*/ +if (order &&zswap_is_present (entry, nr_pages)) {
I am curious how often this fails in practice when zswap is enabled and how often do we end up churning the lock and swap allocation for no reason. I am mainly wondering if a lockless check at the top of the function would be useful.
Anyway, this looks good:
Quoted 10 lines
mm/swap_state.c +4
-0+ err = -EBUSY ; + goto delete_folio; + } +if (mem_cgroup_swapin_charge_folio (folio, memcg_id, vmf ? vmf->vma->vm_mm :NULL , gfp)) { err = -ENOMEM ;Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 17/32 mm: swap in PMD swap entries as whole THPs during swapoff
In reply to Usama Arifswapoff walks every mm and faults every slot of the device back in. unuse_pte_range() only understands PTEs, so a PMD swap entry would never be found and swapoff would never complete.
A PMD swap entry is a compact encoding for HPAGE_PMD_NR slots, not a promise that the swap cache holds one folio for them. Add swap_pmd_cache_lookup() to classify the covered range as empty, one PMD-sized folio, or already split, and unuse_pmd() to map the first two cases back in as one THP, preserving soft-dirty, exclusive and UFFD state.
Everything else falls back to PTEs: a split cache, per-page zswap state, a failed PMD-order allocation or read, or a poisoned subpage. Check PageHWPoison on every subpage rather than the folio-level flag, which memory_failure() only sets after taking the folio lock.
All the fallback reasons are observed without the PMD lock and possibly after sleeping, so they share one exit that re-checks the PMD is still the entry we were called for before splitting it. That exit also drops a folio that is not uptodate, or that has never been mapped, from the swap cache:
the PTE path cannot re-read the first, and would add a single-page rmap to the second.
--- mm/internal.h | 19 +++++ mm/swap.h | 17 +++++ mm/swap_state.c | 44 +++++++++++ mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 269 insertions(+)
mm/internal.h +19 -0
@@ -28,6 +28,25 @@ struct folio_batch;int __copy_remote_mm_str (struct mm_struct *mm,unsigned long addr,void *buf,int len,unsigned int gup_flags); +/* +* Check every page in the folio for HWPoison and return true if any is +* poisoned. +* +* Unlike folio_contain_hwpoisoned_page(), this does not rely on the folio-level +* PG_has_hwpoisoned, which memory_failure() only sets after taking the folio +* lock and so can lag a tail-page poison. +*/ +static inline bool folio_has_any_hwpoisoned_page (const struct folio *folio) +{ +long nr =folio_nr_pages (folio); +long i; + +for (i =0 ; i < nr; i++) +if (PageHWPoison (folio_page (folio, i))) +return true; +return false; +} +/* mm/workingset.c */ bool workingset_test_recent (void *shadow,bool file,bool *workingset,bool flush);
mm/swap.h +17 -0
@@ -317,6 +317,23 @@ static inline bool folio_matches_swap_entry(const struct folio *folio,bool swap_cache_has_folio (swp_entry_t entry);struct folio *swap_cache_get_folio (swp_entry_t entry);void *swap_cache_get_shadow (swp_entry_t entry); +enum swap_pmd_cache { +SWAP_PMD_CACHE_EMPTY , +SWAP_PMD_CACHE_HUGE , +SWAP_PMD_CACHE_SPLIT , +}; + +#ifdef CONFIG_THP_SWAP +enum swap_pmd_cache swap_pmd_cache_lookup (swp_entry_t entry, +struct folio **foliop); +#else +static inline enum swap_pmd_cache swap_pmd_cache_lookup (swp_entry_t entry, +struct folio **foliop) +{ + *foliop =NULL ; +return SWAP_PMD_CACHE_EMPTY ; +} +#endif void swap_cache_del_folio (struct folio *folio); struct folio *__swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask, unsigned long orders, struct vm_fault *vmf,
mm/swap_state.c +44 -0
50 lines not on this page · raw message
mm/swapfile.c +189 -0
211 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 17/32] mm: swap in PMD swap entries as whole THPs during swapoff
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Critical] mm/swapfile: race between unuse_pmd_entry() and do_swap_page() corrupts rmap --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Critical] mm/swapfile: race between unuse_pmd_entry() and do_swap_page() corrupts rmap
--
commit 2a3a8bc71439dccac7a6dee3671d26f7c3dba876
Author: Usama Arif <[email protected]>
mm: swap in PMD swap entries as whole THPs during swapoff
This commit adds swap_pmd_cache_lookup(), unuse_pmd(), and unuse_pmd_entry() to swap in PMD swap entries as whole THPs during swapoff when possible, falling back to splitting the PMD into PTEs otherwise.
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
Quoted 1 line
mm/swapfile.c +0
-0@@ -2695,6 +2696,182 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
[ ... ]
Quoted 51 lines
mm/swapfile.c +51
-0+static int unuse_pmd_entry (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry) +{ +enum swap_pmd_cache cache_state; +struct folio *folio; +pmd_t pmdval; +int ret; + + cache_state =swap_pmd_cache_lookup (entry, &folio); +if (cache_state ==SWAP_PMD_CACHE_SPLIT ) { + ret = -EAGAIN ; + goto split_fallback; + } +if (!folio) { +struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; + +if (zswap_is_present (entry,HPAGE_PMD_NR )) { + ret = -EAGAIN ; + goto split_fallback; + } + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE , +BIT (HPAGE_PMD_ORDER ), &vmf,NULL ,0 ); +if (IS_ERR_OR_NULL (folio)) { + ret = folio ?PTR_ERR (folio) : -ENOMEM ; + goto split_fallback; + } + } + +folio_lock (folio); +folio_wait_writeback (folio); +/* +* If the cached folio is no longer PMD-sized (e.g. split in the +* swap cache by deferred_split_scan() or memory_failure() while +* the PMD swap entry was installed), the PMD swap entry no longer +* maps a single contiguous folio. Split the PMD swap entry so +* unuse_pte_range() can swap the per-slot folios in individually. +*/ +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { +folio_unlock (folio); +folio_put (folio); + ret = -EAGAIN ; + goto split_fallback; + } + ret =unuse_pmd (vma, pmd, addr, entry, folio); +/*
mm/swapfile.c static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, · line 2847
Quoted 10 lines
mm/swapfile.c +10
-0+* PTE fallback cannot add a single-page rmap to a newly allocated, +* PMD-sized !anon folio, and it cannot do anything useful with a folio +* that failed to read. Remove either from the swap cache so each slot +* is read back into an order-0 folio. An uptodate anon swap-cache folio +* can be mapped one PTE at a time and must stay cached so poisoned +* subpages remain visible. +*/ +if (ret &&folio_matches_swap_entry (folio, entry) && + (!folio_test_uptodate (folio) || !folio_test_anon (folio))) +swap_cache_del_folio (folio);
[Severity: Critical]
Can a concurrent page fault in do_swap_page() race with unuse_pmd_entry() and map this newly allocated PMD-sized !anon folio via PTEs before swap_cache_del_folio() runs, corrupting the folio's rmap state?
In mm/swapfile.c, try_to_unuse() -> unuse_mm() -> unuse_pmd_range() -> unuse_pmd_entry() calls swapin_sync() with BIT(HPAGE_PMD_ORDER) without holding pmd_lock (and with only mmap_read_lock(mm) held). swapin_sync() allocates an order HPAGE_PMD_ORDER folio with !folio_test_anon(folio) and folio_test_pmd_mappable(folio) == true, inserts it into the swap cache covering [entry, entry + HPAGE_PMD_NR), and initiates swap_read_folio().
When the swap read completes, the folio is unlocked before unuse_pmd_entry() acquires folio_lock(folio). During that window, a concurrent page fault on split PTEs covering the same swap range (either in the same mm after a concurrent PMD split such as madvise or mprotect, or in a forked mm whose PMD swap entry was split into HPAGE_PMD_NR contiguous PTE swap entries) can enter do_swap_page(), find the folio via swap_cache_get_folio(entry), and lock it first.
In mm/memory.c:do_swap_page(), when all HPAGE_PMD_NR PTEs in the split page table are still contiguous swap entries, folio_pte_range() and swap_pte_batch() compute nr_pages == HPAGE_PMD_NR == folio_nr_pages(folio), so the size mismatch check does not trigger:
if (!folio_test_anon(folio) && folio_test_large(folio) &&
nr_pages != folio_nr_pages(folio)) {
if (!WARN_ON_ONCE(folio_test_dirty(folio)))
swap_cache_del_folio(folio);
goto out_nomap;
}do_swap_page() then calls folio_add_new_anon_rmap(folio, vma, address, rmap_flags) and maps the folio across HPAGE_PMD_NR PTEs with set_ptes(). However, mm/rmap.c:folio_add_new_anon_rmap() assumes any folio_test_pmd_mappable(folio) folio is being mapped at the PMD level:
} else if (!folio_test_pmd_mappable(folio)) {
...
} else {
nr = folio_large_nr_pages(folio);
/* increment count (starts at -1) */
atomic_set(&folio->_entire_mapcount, 0);
folio_set_large_mapcount(folio, 1, vma);
if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT))
atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED);
if (exclusive)
SetPageAnonExclusive(&folio->page);
nr_pmdmapped = nr;
}This initializes PMD-mapped rmap accounting (_entire_mapcount = 0, _large_mapcount = 1, per-page _mapcount left at -1, and PageAnonExclusive set only on page 0) on a folio that is actually mapped by HPAGE_PMD_NR PTEs, which will corrupt mapcounts and trigger BUG checks when those PTEs are later unmapped via folio_remove_rmap_ptes().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Usama Arif
reviewerPATCH v9 18/32 fs/proc: account PMD swap entries in smaps
In reply to Usama Arifsmaps_pmd_entry() derives a page from a non-present PMD and gives up if it cannot. A PMD swap entry carries no PFN, so a swapped-out THP is missing from both Swap and SwapPss.
The slots of a PMD swap entry can have different swap reference counts, so SwapPss has to be computed one slot at a time rather than divided once. Factor that loop out of smaps_pte_entry() and call it from both levels, then return early: a swapped-out THP is not resident and must not be added to AnonHugePages or Rss.
While touching PMD softleaf accounting, match the PTE path and treat device-private PMD entries as fake-present. This lets smaps use mapcount-based PSS and private/shared accounting instead of treating them like migration entries.
--- fs/proc/task_mmu.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
fs/proc/task_mmu.c +28 -13
@@ -965,6 +965,25 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)#endif } +static void smaps_account_swap (struct mem_size_stats *mss, +softleaf_t entry,unsigned long nr_pages) +{ + mss->swap += nr_pages <<PAGE_SHIFT ; +do { +int swapcount =swp_swapcount (entry); + +if (swapcount >=2 ) { +u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; + +do_div (pss_delta, swapcount); + mss->swap_pss += pss_delta; + }else { + mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; + } + entry.val++; + }while (--nr_pages); +} +static void smaps_pte_entry (pte_t *pte,unsigned long addr,struct mm_walk *walk) {
@@ -986,18 +1005,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,const softleaf_t entry =softleaf_from_pte (ptent);if (softleaf_is_swap (entry)) { -int mapcount; - - mss->swap +=PAGE_SIZE ; - mapcount =swp_swapcount (entry); -if (mapcount >=2 ) { -u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; - -do_div (pss_delta, mapcount); - mss->swap_pss += pss_delta; - }else { - mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; - } +smaps_account_swap (mss, entry,1 ); }else if (softleaf_has_pfn (entry)) {if (softleaf_is_device_private (entry)) present = true;
@@ -1030,8 +1038,15 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, }else if (unlikely (thp_migration_supported ())) {const softleaf_t entry =softleaf_from_pmd (*pmd); -if (softleaf_has_pfn (entry)) +if (softleaf_is_swap (entry)) { +smaps_account_swap (mss, entry,HPAGE_PMD_NR ); +return ; + } +if (softleaf_has_pfn (entry)) { +if (softleaf_is_device_private (entry)) + present = true; page =softleaf_to_page (entry); + } }if (IS_ERR_OR_NULL (page))return ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 19/32 mm: handle soft-dirty and uffd-wp on PMD swap entries
In reply to Usama ArifSoft-dirty and uffd-wp live in the PMD, not in the swap entry, so every path that manipulates them for a PMD migration entry has to do the same for a PMD swap entry. Otherwise clearing soft-dirty skips a swapped-out THP, UFFDIO_WRITEPROTECT does not arm it, and mremap does not mark the destination.
change_huge_pmd() needs nothing beyond dropping its thp_migration_supported() gate: a PMD swap entry encodes no write permission, so it falls into the default arm of change_non_present_huge_pmd() and picks up only the uffd-wp update.
--- fs/proc/task_mmu.c | 4 ++-- mm/huge_memory.c | 11 +++++++++-- 2 files changed, 11 insertions(+), 4 deletions(-)
fs/proc/task_mmu.c +2 -2
@@ -1663,7 +1663,7 @@ static void clear_soft_dirty_pmd(struct vm_area_struct *vma, pmd =pmd_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
@@ -2491,7 +2491,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma, old =pmdp_invalidate_ad (vma, addr, pmdp); pmd =pmd_mkuffd (old);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_mkuffd (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
mm/huge_memory.c +9 -2
@@ -2626,7 +2626,8 @@ static inline int pmd_move_must_withdraw(spinlock_t *new_pmd_ptl,static pmd_t move_soft_dirty_pmd (pmd_t pmd) {if (pgtable_supports_soft_dirty ()) { -if (unlikely (pmd_is_migration_entry (pmd))) +if (unlikely (pmd_is_migration_entry (pmd) || +pmd_is_swap_entry (pmd))) pmd =pmd_swp_mksoft_dirty (pmd);else if (pmd_present (pmd)) pmd =pmd_mksoft_dirty (pmd);
@@ -2718,6 +2719,12 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,pmd_t newpmd;VM_WARN_ON (!pmd_is_valid_softleaf (*pmd)); + +/* +* Note that a PMD swap entry falls into the default branch below: it +* does not encode write permission in the entry type, so only the +* uffd_wp flag update at the end applies to it. +*/ if (softleaf_is_migration_write (entry)) {const struct folio *folio =softleaf_to_folio (entry);
@@ -2778,7 +2785,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,if (!ptl)return 0 ; -if (thp_migration_supported () &&pmd_is_valid_softleaf (*pmd)) { +if (pmd_is_valid_softleaf (*pmd)) {change_non_present_huge_pmd (mm, addr, pmd, uffd_prot, uffd_prot_resolve); goto unlock;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 20/32 mm/hmm: fault PMD swap entries on demand
In reply to Usama Arifhmm_vma_handle_absent_pmd() reports anything non-present that is not device-private as HMM_PFN_ERROR, so a swapped-out THP would look like a permanent failure rather than memory that needs paging in.
Treat it like a PTE swap entry: record the fault when one was asked for, and otherwise report the range as non-resident with no flags, which is what hmm_vma_handle_pte() already does in snapshot mode.
--- mm/hmm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/hmm.c +10 -1
@@ -377,12 +377,21 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start, required_fault =hmm_range_need_fault (hmm_vma_walk, hmm_pfns, npages,0 );if (required_fault) { -if (softleaf_is_device_private (entry)) +if (softleaf_is_device_private (entry) || +softleaf_is_swap (entry))return hmm_record_fault (addr, end, required_fault, walk);else return -EFAULT ; } +/* +* A swapped-out THP is not resident. Report it as not-valid, +* matching what hmm_vma_handle_pte() does for a PTE swap entry when +* no fault was requested. +*/ +if (softleaf_is_swap (entry)) +return hmm_pfns_fill (start, end, range,0 ); + return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); }#else
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 21/32 mm: free PMD swap entries in zap_huge_pmd()
In reply to Usama Arifzap_huge_pmd() assumes any non-present PMD it clears references a folio, so a PMD swap entry would reach normal_or_softleaf_folio_pmd() and warn. Worse, nothing would release the slots it points at: unmapping a swapped-out THP would leak the whole PMD's worth of swap.
Handle it before the folio lookup, as zap_pte_range() already does for a PTE swap entry.
--- mm/huge_memory.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/huge_memory.c +10 -0
@@ -2594,6 +2594,16 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,arch_check_zapped_pmd (vma, orig_pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr); +if (pmd_is_swap_entry (orig_pmd)) { +softleaf_t entry =softleaf_from_pmd (orig_pmd); + +zap_deposited_table (mm, pmd); +spin_unlock (ptl); +swap_put_entries_direct (entry,HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , -HPAGE_PMD_NR ); +return true; + } + is_present =pmd_present (orig_pmd); folio =normal_or_softleaf_folio_pmd (vma, addr, orig_pmd, is_present); has_deposit =vma_has_deposited_pgtable (vma);
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 22/32 mm/madvise: free PMD swap entries with MADV_FREE
In reply to Usama Arifmadvise_free_pte_range() only calls madvise_free_huge_pmd() for a present THP, and its PTE loop cannot descend into a non-present PMD, so MADV_FREE over a swapped-out THP silently does nothing. Over the same range as PTE swap entries it frees every slot, which is what userspace gets today.
Free it at PMD granularity when the advice covers the whole PMD, and split and let the PTE path handle the subrange when it does not.
zap_deposited_table() moves above its new caller; it is otherwise unchanged.
--- mm/huge_memory.c | 34 +++++++++++++++++++++++++--------- mm/madvise.c | 2 +- 2 files changed, 26 insertions(+), 10 deletions(-)
mm/huge_memory.c +25 -9
@@ -2425,6 +2425,15 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)return 0 ; } +static inline void zap_deposited_table (struct mm_struct *mm,pmd_t *pmd) +{ +pgtable_t pgtable; + + pgtable =pgtable_trans_huge_withdraw (mm, pmd); +pte_free (mm, pgtable); +mm_dec_nr_ptes (mm); +} +/* * Return true if we do MADV_FREE successfully on entire pmd page. * Otherwise, return false.
@@ -2449,6 +2458,22 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, goto out;if (unlikely (!pmd_present (orig_pmd))) { +if (pmd_is_swap_entry (orig_pmd)) { +softleaf_t entry =softleaf_from_pmd (orig_pmd); + +if (next - addr !=HPAGE_PMD_SIZE ) { +spin_unlock (ptl); +__split_huge_pmd (vma, pmd, addr); + goto out_unlocked; + } + +pmdp_huge_get_and_clear (mm, addr, pmd); +zap_deposited_table (mm, pmd); +spin_unlock (ptl); +swap_put_entries_direct (entry,HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , -HPAGE_PMD_NR ); +return true; + }VM_WARN_ON_ONCE (!pmd_is_migration_entry (orig_pmd) && !pmd_is_device_private_entry (orig_pmd)); goto out;
@@ -2503,15 +2528,6 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,return ret; } -static inline void zap_deposited_table (struct mm_struct *mm,pmd_t *pmd) -{ -pgtable_t pgtable; - - pgtable =pgtable_trans_huge_withdraw (mm, pmd); -pte_free (mm, pgtable); -mm_dec_nr_ptes (mm); -} -static void zap_huge_pmd_folio (struct mm_struct *mm,struct vm_area_struct *vma,pmd_t pmdval,struct folio *folio,bool is_present) {
mm/madvise.c +1 -1
@@ -677,7 +677,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,int nr, max_nr; next =pmd_addr_end (addr, end); -if (pmd_trans_huge (*pmd)) +if (pmd_trans_huge (*pmd) ||pmd_is_swap_entry (*pmd))if (madvise_free_huge_pmd (tlb, vma, pmd, addr, next))return 0 ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 23/32 mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT
In reply to Usama ArifA swapped-out THP has no resident folio to deactivate or reclaim, which is also the conclusion the PTE path reaches when it walks the swap entries and skips every one.
madvise_cold_or_pageout_pte_range() gets there the long way round:pmd_trans_huge() is false for a PMD swap entry, so it falls out of the huge branch and tries to map a PTE table that a non-present PMD does not have.
The entry can also appear after that first, lockless test, so re-check under the PMD lock; otherwise a PMD that turned into a swap entry while pmd_trans_huge_lock() was being taken trips the VM_WARN_ON_ONCE() that expects a migration or device-private entry.
--- mm/madvise.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
mm/madvise.c +12 -0
@@ -384,6 +384,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, !can_do_file_pageout (vma);#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* +* Swapped-out THPs have no resident folio to deactivate or reclaim. +* Avoid descending into or splitting a PMD swap entry. +*/ +if (pmd_is_swap_entry (*pmd)) { + walk->action =ACTION_CONTINUE ; +return 0 ; + } +if (pmd_trans_huge (*pmd)) {pmd_t orig_pmd;unsigned long next =pmd_addr_end (addr, end);
@@ -394,6 +403,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ; orig_pmd = *pmd; +if (pmd_is_swap_entry (orig_pmd)) + goto huge_unlock; +if (unlikely (!pmd_present (orig_pmd))) {VM_WARN_ON_ONCE (!pmd_is_migration_entry (orig_pmd) && !pmd_is_device_private_entry (orig_pmd));
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 24/32 mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE
In reply to Usama ArifBoth callbacks test bare pmd_trans_huge(), which is false for a PMD swap entry, so they return without consuming the PMD and walk_pmd_range() splits it - only for the PTE callbacks to find the range populated and abort, or to find no guard markers and do nothing.
Both already short-circuit at PMD level for a present THP. Do the same for a swapped-out one: it is just as populated, and just as free of guard markers.
--- mm/madvise.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
mm/madvise.c +12 -4
@@ -1109,8 +1109,12 @@ static int guard_install_pmd_entry(pmd_t *pmd, unsigned long addr, {pmd_t pmdval =pmdp_get (pmd); -/* If huge return >0 so we abort the operation + zap. */ -return pmd_trans_huge (pmdval); +/* +* If huge return >0 so we abort the operation + zap. A PMD swap entry +* is a swapped-out THP: also populated, and splitting it here would +* only demote it before the zap. +*/ +return pmd_trans_huge (pmdval) ||pmd_is_swap_entry (pmdval); } static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
@@ -1249,8 +1253,12 @@ static int guard_remove_pmd_entry(pmd_t *pmd, unsigned long addr, {pmd_t pmdval =pmdp_get (pmd); -/* If huge, cannot have guard pages present, so no-op - skip. */ -if (pmd_trans_huge (pmdval)) +/* +* If huge, cannot have guard pages present, so no-op - skip. The same +* is true of a PMD swap entry, which must not be split just to discover +* there is nothing to remove. +*/ +if (pmd_trans_huge (pmdval) ||pmd_is_swap_entry (pmdval)) walk->action =ACTION_CONTINUE ;return 0 ;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 25/32 mm/mincore: report PMD swap-cache residency
In reply to Usama Arifmincore_pte_range() reports every page under a huge PMD as resident. That is wrong for a swapped-out one: mincore() would tell userspace that memory it will have to fault in is already in core.
A PMD swap entry is only a compact encoding for HPAGE_PMD_NR slots, so answer from the swap cache instead. One PMD-sized folio covering the range makes every page resident exactly when that folio is uptodate; an empty cache makes none of them; and if the folio was split while the entry stayed in place, fall back to looking up each covered slot as mincore_swap() does for a PTE swap entry.
--- mm/mincore.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)
mm/mincore.c +46 -1
@@ -85,6 +85,48 @@ static unsigned char mincore_swap(swp_entry_t entry, bool shmem)return present; } +#ifdef CONFIG_THP_SWAP +static void mincore_pmd_swap (swp_entry_t entry,unsigned long addr, +unsigned long end,unsigned char *vec) +{ +unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long start = (addr - haddr) >>PAGE_SHIFT ; +unsigned long nr = (end - addr) >>PAGE_SHIFT ; +struct folio *folio; +enum swap_pmd_cache state; +int i; + + state =swap_pmd_cache_lookup (entry, &folio); +if (state ==SWAP_PMD_CACHE_HUGE ) { +memset (vec,folio_test_uptodate (folio), nr); +folio_put (folio); +return ; + } + +if (state ==SWAP_PMD_CACHE_EMPTY ) { +memset (vec,0 , nr); +return ; + } + +/* +* The PMD swap entry is only a compact encoding for consecutive swap +* slots. If the PMD-sized swapcache folio was split, report residency +* from the individual slots covered by this mincore() range. +*/ +for (i =0 ; i < nr; i++) + vec[i] =mincore_swap (swp_entry (swp_type (entry), +swp_offset (entry) + start + i), + false); +} +#else /* !CONFIG_THP_SWAP */ +static void mincore_pmd_swap (swp_entry_t entry,unsigned long addr, +unsigned long end,unsigned char *vec) +{ +/* Nothing produces PMD swap entries without CONFIG_THP_SWAP. */ +memset (vec,0 , (end - addr) >>PAGE_SHIFT ); +} +#endif /* CONFIG_THP_SWAP */ +/* * Later we can get more picky about what "in core" means precisely. * For now, simply check to see if the page is in the page cache,
@@ -171,7 +213,10 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_is_swap_entry (*pmd)) +mincore_pmd_swap (softleaf_from_pmd (*pmd), addr, end, vec); +else +memset (vec,1 , nr);spin_unlock (ptl); goto out; }
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 26/32 mm/khugepaged: treat PMD swap entries as mapped THPs
In reply to Usama Arifcheck_pmd_state() would report a PMD swap entry as SCAN_NO_PTE_TABLE - it is not present and, on x86, is also pmd_bad(). The range is already a THP; it just happens to be on disk.
Return SCAN_PMD_MAPPED next to the existing migration-entry case and before the pmd_present() and pmd_bad() tests.
This does change MADV_COLLAPSE. Today a swapped-out THP is HPAGE_PMD_NR PTE swap entries and MADV_COLLAPSE, which ignores khugepaged_max_ptes_swap, swaps them all in. With a PMD swap entry it reports SCAN_PMD_MAPPED, which madvise_collapse() counts as success, so the call returns 0 without making the range resident. It still faults back in as a whole THP on first touch; only the point at which the I/O happens moves.
--- mm/khugepaged.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/khugepaged.c +6 -0
@@ -1065,6 +1065,12 @@ static inline enum scan_result check_pmd_state(pmd_t *pmd)*/ if (pmd_is_migration_entry (pmde))return SCAN_PMD_MAPPED ; +/* +* A PMD-mapped THP that has been swapped out is still a THP from +* khugepaged's perspective; treat it like a present huge PMD. +*/ +if (pmd_is_swap_entry (pmde)) +return SCAN_PMD_MAPPED ;if (!pmd_present (pmde))return SCAN_NO_PTE_TABLE ;if (pmd_trans_huge (pmde))
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 27/32 mm: handle PMD swap entries in MADV_WILLNEED
In reply to Usama Arifswapin_walk_pmd_entry() walks PTEs and cannot descend into a non-present PMD, so MADV_WILLNEED is a no-op on a PMD swap entry. Reading the slots back one at a time would not be much better: order-0 readahead populates per-page swap-cache state, and the fault that follows would then have to split the entry the prefetch was supposed to help.
Prefetch the whole range as one PMD-order folio instead, leaving the entry in place so the eventual fault still takes do_huge_pmd_swap_page(). Check the THP policy first: the range was PMD-mapped when it was swapped out, but MADV_NOHUGEPAGE or the sysfs knob may have changed since, and reading at PMD order would then hand do_swap_page() a PMD-sized folio to map with PTEs.
A split cache, per-page zswap state, or a failed PMD-order read or allocation falls back to PTEs. Leaving the entry alone would prefetch nothing at all, while the PTE path can still read the slots at order 0. A folio that failed to read is dropped from the swap cache first, so the PTE retry re-reads each slot rather than every later fault returning SIGBUS.
Only split if the PMD still looks like the entry that was observed - the decision is taken after the PMD lock has been dropped, so a racing fault may have swapped the range back in as a THP. The test is lockless, so it narrows that window rather than closing it.
--- mm/madvise.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+)
mm/madvise.c +139 -0
@@ -33,6 +33,7 @@#include <linux/shmem_fs.h> #include <linux/mmu_notifier.h> #include <linux/swap_ops.h> +#include <linux/zswap.h> #include <asm/tlb.h>
@@ -193,6 +194,108 @@ static int madvise_update_vma(vm_flags_t new_flags, }#ifdef CONFIG_SWAP +/* +* Prefetch a whole PMD swap entry as one PMD-order folio. +* +* Called with the PMD lock held; always drops it. Returns true when the +* caller should ask the walker to retry so the PTE path can handle the +* covered slots individually. +*/ +static bool swapin_pmd_swap_entry (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry, +spinlock_t *ptl) +{ +struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; +enum swap_pmd_cache cache_state; +struct swap_info_struct *si; +struct folio *folio; +bool split = false; + +/* +* The range was PMD-mapped when it was swapped out, but the policy may +* have changed since: MADV_NOHUGEPAGE, or the sysfs knob. Reading it +* back at PMD order would then hand do_swap_page() a PMD-sized folio to +* map with PTEs, which is exactly what the policy forbids. Split now +* and let the PTE path prefetch at order 0 - the next fault would split +* the entry anyway, so nothing is lost that the VMA still permits. +*/ +if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_PAGEFAULT , +HPAGE_PMD_ORDER )) { +spin_unlock (ptl); +return true; + } + + cache_state =swap_pmd_cache_lookup (entry, &folio); +if (cache_state ==SWAP_PMD_CACHE_HUGE ) { +/* Already cached as one PMD-sized folio, nothing to do. */ +folio_put (folio); +spin_unlock (ptl); +return false; + } +if (cache_state ==SWAP_PMD_CACHE_SPLIT || +zswap_is_present (entry,HPAGE_PMD_NR )) { +spin_unlock (ptl); +return true; + } + +/* +* Pin the swap device under the PMD lock so the PMD-swap-entry +* observation keeps the entry valid for swapin_sync(). +*/ + si =get_swap_device (entry); +spin_unlock (ptl); +if (IS_ERR_OR_NULL (si)) +return false; + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE ,BIT (HPAGE_PMD_ORDER ), + &vmf,NULL ,0 ); + +/* +* Fall back to PTE-order swapin: a PMD-order failure does not mean +* that individual slots cannot be read. +*/ +if (IS_ERR_OR_NULL (folio)) { + split = true; + goto out; + } + +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { + split = true; + goto out_put; + } + +/* +* A trylock only succeeds once the read has completed, so this never +* blocks MADV_WILLNEED on in-flight I/O. A read that failed - a +* PMD-order zswap load that found per-page state, or an I/O error - +* leaves the folio clean and not uptodate. Drop it from the swap cache +* so the PTE retry reads each slot again; leaving it there would make +* the next fault return VM_FAULT_SIGBUS. Another thread may have +* removed it already, so revalidate the association first. +*/ +if (!folio_trylock (folio)) + goto out_put; + +if (!folio_test_uptodate (folio)) { +if (folio_matches_swap_entry (folio, entry)) +swap_cache_del_folio (folio); + split = true; + } +folio_unlock (folio); + +out_put: +folio_put (folio); +out: +/* Keep the device pinned until the last use of @entry. */ +put_swap_device (si); +return split; +} +static int swapin_walk_pmd_entry (pmd_t *pmd,unsigned long start,unsigned long end,struct mm_walk *walk) {
@@ -202,6 +305,41 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,spinlock_t *ptl;unsigned long addr; + ptl =pmd_trans_huge_lock (pmd, vma); +if (ptl) { +pmd_t pmdval = *pmd; + +if (pmd_is_swap_entry (pmdval)) { +/* swapin_pmd_swap_entry() always drops the PMD lock. */ +if (!swapin_pmd_swap_entry (vma, pmd, start, +softleaf_from_pmd (pmdval), + ptl)) + goto ret; +/* +* Only split if this still looks like the entry we +* observed. The fallback was decided after the PMD lock +* was dropped, so a racing fault may have swapped the +* range back in as a THP, and splitting that would +* demote a perfectly good huge mapping for an advisory +* hint. The test is lockless, so it narrows that window +* rather than closing it. +*/ +if (pmd_same (pmdval,pmdp_get_lockless (pmd))) { +__split_huge_pmd (vma, pmd, start); + walk->action =ACTION_AGAIN ; + goto ret; + } +/* +* Somebody else changed the PMD. Leave it alone and let +* the PTE loop below deal with whatever is there now; +* it simply finds no page table if the range came back +* as a THP. +*/ + }else { +spin_unlock (ptl); + } + } +for (addr = start; addr < end; addr +=PAGE_SIZE ) {pte_t pte;softleaf_t entry;
@@ -230,6 +368,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,if (ptep)pte_unmap_unlock (ptep, ptl);swap_read_submit (&ctx); +ret:cond_resched ();return 0 ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 27/32] mm: handle PMD swap entries in MADV_WILLNEED
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Critical] mm: rmap corruption from unmapped PMD folio cached by swapin_pmd_swap_entry() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Critical] mm: rmap corruption from unmapped PMD folio cached by swapin_pmd_swap_entry()
--
commit 14ed08337de0c0909e93ae1d8b999993fb793692
Author: Usama Arif <[email protected]>
mm: handle PMD swap entries in MADV_WILLNEED
This commit teaches MADV_WILLNEED to prefetch PMD swap entries as a single PMD-order folio when THP policy allows it, falling back to splitting the PMD into PTEs when PMD-order swapin is disallowed or fails.
Quoted 4 lines
mm/madvise.c +0
-0
[ ... ]
Quoted 15 lines
mm/madvise.c +11
-0@@ -193,6 +194,108 @@ static int madvise_update_vma(vm_flags_t new_flags, }#ifdef CONFIG_SWAP +/* +* Prefetch a whole PMD swap entry as one PMD-order folio. +* +* Called with the PMD lock held; always drops it. Returns true when the +* caller should ask the walker to retry so the PTE path can handle the +* covered slots individually. +*/ +static bool swapin_pmd_swap_entry (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry, +spinlock_t *ptl) +{
[ ... ]
Quoted 28 lines
mm/madvise.c +28
-0+/* +* The range was PMD-mapped when it was swapped out, but the policy may +* have changed since: MADV_NOHUGEPAGE, or the sysfs knob. Reading it +* back at PMD order would then hand do_swap_page() a PMD-sized folio to +* map with PTEs, which is exactly what the policy forbids. Split now +* and let the PTE path prefetch at order 0 - the next fault would split +* the entry anyway, so nothing is lost that the VMA still permits. +*/ +if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_PAGEFAULT , +HPAGE_PMD_ORDER )) { +spin_unlock (ptl); +return true; + } + + cache_state =swap_pmd_cache_lookup (entry, &folio); +if (cache_state ==SWAP_PMD_CACHE_HUGE ) { +/* Already cached as one PMD-sized folio, nothing to do. */ +folio_put (folio); +spin_unlock (ptl); +return false; + } +if (cache_state ==SWAP_PMD_CACHE_SPLIT || +zswap_is_present (entry,HPAGE_PMD_NR )) { +spin_unlock (ptl); +return true; + } + +/*
mm/madvise.c static int madvise_update_vma(vm_flags_t new_flags, · line 256
Quoted 10 lines
mm/madvise.c +10
-0+* Pin the swap device under the PMD lock so the PMD-swap-entry +* observation keeps the entry valid for swapin_sync(). +*/ + si =get_swap_device (entry); +spin_unlock (ptl); +if (IS_ERR_OR_NULL (si)) +return false; + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE ,BIT (HPAGE_PMD_ORDER ), + &vmf,NULL ,0 );
[Severity: Critical]
Can leaving a newly allocated PMD-order folio unmapped in the swap cache corrupt rmap and mapcount state if the PMD swap entry is split before a PMD fault maps it?
When madvise_willneed() -> swapin_walk_pmd_entry() -> swapin_pmd_swap_entry() calls swapin_sync() with BIT(HPAGE_PMD_ORDER), it inserts a fresh HPAGE_PMD_ORDER folio into the swap cache and leaves it unmapped (!folio_test_anon(folio)).
Before a PMD page fault maps that folio via do_huge_pmd_swap_page(), the PMD swap entry can be split into 512 contiguous PTE swap entries without removing the cached folio. For example, if userspace applies MADV_NOHUGEPAGE (or the sysfs THP policy changes) and then either calls MADV_WILLNEED again (hitting the !thp_vma_allowable_order() check above and splitting via __split_huge_pmd()) or faults on the range (causing do_huge_pmd_swap_page() to split the PMD for PTE fallback), or performs a partial munmap(), mprotect(), MADV_DONTNEED, or MADV_FREE, the unmapped PMD-order folio remains in the swap cache while the page table now holds PTE swap entries.
When do_swap_page() then faults on one of those 512 contiguous PTE swap entries, swap_cache_get_folio() returns the HPAGE_PMD_ORDER folio and swap_pte_batch() batches all 512 PTEs so nr_pages == folio_nr_pages(folio), bypassing the partial mapping check:
mm/memory.c:do_swap_page() {
...
if (!folio_test_anon(folio) && folio_test_large(folio) &&
nr_pages != folio_nr_pages(folio)) {
if (!WARN_ON_ONCE(folio_test_dirty(folio)))
swap_cache_del_folio(folio);
goto out_nomap;
}
...
} else if (!folio_test_anon(folio)) {
VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) != nr_pages, folio);
VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
folio_put_swap(folio, NULL);
}
...
set_ptes(vma->vm_mm, addr, ptep, pte, nr_pages);
}Because folio_test_pmd_mappable(folio) is true for an HPAGE_PMD_ORDER folio, folio_add_new_anon_rmap() initializes the folio as PMD-mapped rather than PTE-mapped:
mm/rmap.c:folio_add_new_anon_rmap() {
...
} else if (!folio_test_pmd_mappable(folio)) {
...
} else {
nr = folio_large_nr_pages(folio);
/* increment count (starts at -1) */
atomic_set(&folio->_entire_mapcount, 0);
folio_set_large_mapcount(folio, 1, vma);
if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT))
atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED);
if (exclusive)
SetPageAnonExclusive(&folio->page);
nr_pmdmapped = nr;
}
}This maps the folio with 512 PTEs while setting _entire_mapcount = 0, _large_mapcount = 1, nr_pmdmapped = 512, leaving per-page _mapcount at -1, and only setting PageAnonExclusive on the head page even when tail pages are mapped by writable PTEs.
Similarly, if swapoff runs on the split PTE swap entries, unuse_pte() will find the unmapped HPAGE_PMD_ORDER folio in the swap cache:
mm/swapfile.c:unuse_pte() {
...
if (!folio_test_anon(folio)) {
VM_WARN_ON_ONCE(folio_test_large(folio));
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
folio_add_new_anon_rmap(folio, vma, addr, rmap_flags);
}
}Would it make sense to ensure that unmapped PMD-order folios in the swap cache are either removed when a PMD swap entry is split or rejected by PTE swapin paths such as do_swap_page() and unuse_pte()?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=27
Usama Arif
reviewerPATCH v9 28/32 mm: handle PMD swap entries in UFFDIO_MOVE
In reply to Usama Arifmove_pages_huge_pmd() returns -ENOENT for any PMD that is neither trans_huge nor a migration entry, so an aligned UFFDIO_MOVE over a swapped-out THP fails even though a PMD swap entry is a perfectly good mapping to move. Falling back to the PTE path is no help either: splitting yields PTE swap entries pointing at the same swap-cache folio, and move_pages_ptes() refuses any swap-cache folio that is still large.
move_swap_pmd() is modelled on move_swap_pte(): it moves the entry under both PMD locks, propagates soft-dirty, clears an inherited UFFD marker and re-arms it only for an RWP-registered destination, carries the deposited page table across, and requires pmd_swp_exclusive() for the same single-owner semantics.
The entry can only be moved whole while the covered swap cache is empty or holds one PMD-sized folio. A cached folio is locked and revalidated, then its anon rmap is re-anchored to the destination VMA; an empty cache is re-checked slot by slot under both PMD locks, because a per-slot folio that appeared meanwhile would need the PTE path to fix up its rmap metadata. A range that is already split is split and retried through PTEs. Revalidation failure just returns -EAGAIN: its usual cause is a racing fault that made src_pmd a healthy present THP, which must not be shattered.
Finally, reject a PMD swap entry at the *destination* with -EEXIST. It is not a hole, and unlike a migration entry it does not resolve on its own:pte_alloc() skips a !pmd_none PMD, pte_offset_map_rw_nolock() then fails, and the resulting -EAGAIN would be retried forever.
--- mm/huge_memory.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++- mm/userfaultfd.c | 14 +++++ 2 files changed, 172 insertions(+), 1 deletion(-)
mm/huge_memory.c +158 -1
@@ -2938,6 +2938,79 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,#endif #ifdef CONFIG_USERFAULTFD +#ifdef CONFIG_THP_SWAP +/* +* Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are +* acquired here; src_folio (if present) must already be locked. The deposited +* page table backing the source THP is moved across with the entry. +*/ +static int move_swap_pmd (struct mm_struct *mm,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +pmd_t *dst_pmd,pmd_t *src_pmd, +pmd_t orig_dst_pmd,pmd_t orig_src_pmd, +spinlock_t *dst_ptl,spinlock_t *src_ptl, +struct folio *src_folio,swp_entry_t entry) +{ +pgtable_t src_pgtable; +pmd_t moved_pmd; + +/* +* The folio may have been freed and reused for a different swap entry +* while it was unlocked. Re-verify the association. +*/ +if (src_folio &&unlikely (!folio_matches_swap_entry (src_folio, entry) || +folio_nr_pages (src_folio) !=HPAGE_PMD_NR )) +return -EAGAIN ; + +double_pt_lock (dst_ptl, src_ptl); + +if (!pmd_same (*src_pmd, orig_src_pmd) || + !pmd_same (*dst_pmd, orig_dst_pmd)) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + +/* +* If the folio is in the swap cache, re-anchor its anon rmap to the +* destination VMA so a future swap-in fault at dst_addr finds it. +* Otherwise, re-check the whole PMD swap range: a PMD swap entry is +* only a compact encoding for HPAGE_PMD_NR swap slots, and any per-slot +* cached folio would need the PTE move path to update its rmap +* metadata. +*/ +if (src_folio) { +folio_move_anon_rmap (src_folio, dst_vma); + src_folio->index =linear_anon_page_index (dst_vma, dst_addr); + }else { +unsigned int type =swp_type (entry); +pgoff_t offset =swp_offset (entry); +int i; + +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +if (swap_cache_has_folio (swp_entry (type, offset + i))) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + } + } + + moved_pmd =pmdp_huge_get_and_clear (mm, src_addr, src_pmd); +if (pgtable_supports_soft_dirty ()) + moved_pmd =pmd_swp_mksoft_dirty (moved_pmd); + moved_pmd =pmd_swp_clear_uffd (moved_pmd); +/* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ +if (userfaultfd_rwp (dst_vma)) + moved_pmd =pmd_swp_mkuffd (moved_pmd); +set_pmd_at (mm, dst_addr, dst_pmd, moved_pmd); + + src_pgtable =pgtable_trans_huge_withdraw (mm, src_pmd); +pgtable_trans_huge_deposit (mm, dst_pmd, src_pgtable); + +double_pt_unlock (dst_ptl, src_ptl); +return 0 ; +} +#endif /* CONFIG_THP_SWAP */ +/* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by * the caller, but it must return after releasing the page_table_lock.
@@ -2972,11 +3045,95 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }if (!pmd_trans_huge (src_pmdval)) { -spin_unlock (src_ptl);if (pmd_is_migration_entry (src_pmdval)) { +spin_unlock (src_ptl);pmd_migration_entry_wait (mm, src_pmd);return -EAGAIN ; } +#ifdef CONFIG_THP_SWAP +if (pmd_is_swap_entry (src_pmdval)) { +swp_entry_t entry; +struct swap_info_struct *si; +enum swap_pmd_cache cache_state; + +/* +* UFFDIO_MOVE on anon mappings requires single-owner +* semantics; refuse to move a shared swap entry. +*/ +if (!pmd_swp_exclusive (src_pmdval)) { +spin_unlock (src_ptl); +return -EBUSY ; + } + + entry =softleaf_from_pmd (src_pmdval); +spin_unlock (src_ptl); + +/* +* Pin the swap device against a racing swapoff. NULL +* means swapoff is in progress, which resolves on its +* own, so ask the caller to retry. An error pointer +* means the entry names no swap device at all: that +* never resolves, so report it instead of spinning in +* the caller's -EAGAIN loop. +*/ + si =get_swap_device (entry); +if (!si) +return -EAGAIN ; +if (IS_ERR (si)) +return PTR_ERR (si); + + src_folio =NULL ; + cache_state =swap_pmd_cache_lookup (entry, &src_folio); +if (cache_state ==SWAP_PMD_CACHE_SPLIT ) { +put_swap_device (si); +__split_huge_pmd (src_vma, src_pmd, src_addr); +return -EAGAIN ; + } + +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , + mm, src_addr, + src_addr +HPAGE_PMD_SIZE ); +mmu_notifier_invalidate_range_start (&range); + +if (src_folio) { +folio_lock (src_folio); +/* +* Do not split on failure here. The usual cause +* is that a racing fault swapped the range back +* in and dropped the folio from the swap cache, +* so src_pmd is now a healthy present THP; +* splitting it would destroy the very mapping +* UFFDIO_MOVE is trying to move whole. The +* caller's -EAGAIN retry re-reads src_pmd and +* picks the right path, exactly as +* move_swap_pte() relies on for the PTE case. +*/ +if (!folio_matches_swap_entry (src_folio, entry) || +folio_nr_pages (src_folio) !=HPAGE_PMD_NR ) { +folio_unlock (src_folio); +folio_put (src_folio); +mmu_notifier_invalidate_range_end (&range); +put_swap_device (si); +return -EAGAIN ; + } + } + + dst_ptl =pmd_lockptr (mm, dst_pmd); + err =move_swap_pmd (mm, dst_vma, dst_addr, src_addr, + dst_pmd, src_pmd, dst_pmdval, + src_pmdval, dst_ptl, src_ptl, + src_folio, entry); + +mmu_notifier_invalidate_range_end (&range); +if (src_folio) { +folio_unlock (src_folio); +folio_put (src_folio); + } +put_swap_device (si); +return err; + } +#endif /* CONFIG_THP_SWAP */ +spin_unlock (src_ptl);return -ENOENT ; }
mm/userfaultfd.c +14 -0
@@ -2072,6 +2072,20 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,break ; } +/* +* A PMD swap entry at dst is a swapped-out THP, not a hole, +* and unlike a PMD migration entry it will not resolve on its +* own. Nothing below faults it back in: pte_alloc() skips a +* !pmd_none PMD, pte_offset_map_rw_nolock() then fails on the +* non-present PMD, and the -EAGAIN that produces would be +* retried forever by the loop below. Be strict, exactly as for +* a present THP. +*/ +if (unlikely (pmd_is_swap_entry (dst_pmdval))) { + err = -EEXIST ; +break ; + } + ptl =pmd_trans_huge_lock (src_pmd, src_vma);if (ptl) {/* Check if we can move the pmd without splitting it. */
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 28/32] mm: handle PMD swap entries in UFFDIO_MOVE
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] mm: move_swap_pmd() sets anon rmap on unmapped swap-cache folios - [Medium] mm…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm: move_swap_pmd() sets anon rmap on unmapped swap-cache folios
- [Medium] mm: move_pages_huge_pmd() drops src_ptl before swap lookup and split
- [Medium] mm: move_pages() splits PMD swap entries before checking movability
--
commit 0e469173a6b536c3bd6b664124f92ddaf042afb0
Author: Usama Arif <[email protected]>
mm: handle PMD swap entries in UFFDIO_MOVE
This commit adds support for moving PMD-level swap entries in UFFDIO_MOVE via a new move_swap_pmd() helper in move_pages_huge_pmd(), while rejecting PMD swap entries at the destination in move_pages() with -EEXIST.
Quoted 21 lines
mm/huge_memory.c +13
-0@@ -2938,6 +2938,79 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,#endif #ifdef CONFIG_USERFAULTFD +#ifdef CONFIG_THP_SWAP +/* + * Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are + * acquired here; src_folio (if present) must already be locked. The deposited + * page table backing the source THP is moved across with the entry. + */ +static int move_swap_pmd(struct mm_struct *mm, struct vm_area_struct *dst_vma, + unsigned long dst_addr, unsigned long src_addr, + pmd_t *dst_pmd, pmd_t *src_pmd, + pmd_t orig_dst_pmd, pmd_t orig_src_pmd, + spinlock_t *dst_ptl, spinlock_t *src_ptl, + struct folio *src_folio, swp_entry_t entry) +{
[ ... ]
mm/huge_memory.c int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2983
Quoted 3 lines
mm/huge_memory.c +3
-0+if (src_folio) { +folio_move_anon_rmap (src_folio, dst_vma); + src_folio->index =linear_anon_page_index (dst_vma, dst_addr);
[Severity: Medium]
Can src_folio be a newly allocated swap-cache folio that has not yet been marked anonymous with folio_add_new_anon_rmap()?
If a swapped-out THP had its folio evicted from the swap cache and a fresh PMD-order folio was subsequently allocated into the swap cache (for example, via MADV_WILLNEED in swapin_walk_pmd_entry() -> swapin_pmd_swap_entry() -> swapin_sync() -> __swap_cache_alloc_folio(), or a racing swap-in fault before the folio is mapped), src_folio->mapping is still NULL (!folio_test_anon(src_folio)).
Calling folio_move_anon_rmap(src_folio, dst_vma) unconditionally in move_swap_pmd() writes dst_vma->anon_vma + FOLIO_MAPPING_ANON into src_folio->mapping without going through folio_add_new_anon_rmap():
mm/rmap.c:folio_add_new_anon_rmap() {
...
__folio_mod_stat(folio, nr, nr_pmdmapped);
mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1);
}When the folio is later freed, free_pages_prepare() sees folio_test_anon(folio) and decrements MTHP_STAT_NR_ANON without a matching increment:
mm/page_alloc.c:free_pages_prepare() {
...
if (folio_test_anon(folio)) {
mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1);
folio->mapping = NULL;
}
}Stamping FOLIO_MAPPING_ANON early also defeats the !folio_test_anon(folio) checks in unuse_pmd_entry(), do_huge_pmd_swap_page(), and do_swap_page() that delete newly allocated swap-cache folios on PTE fallback or read failure:
mm/swapfile.c:unuse_pmd_entry() {
...
if (ret && folio_matches_swap_entry(folio, entry) &&
(!folio_test_uptodate(folio) || !folio_test_anon(folio)))
swap_cache_del_folio(folio);
}Should move_swap_pmd() check folio_test_anon(src_folio) before calling folio_move_anon_rmap() and updating src_folio->index?
[ ... ]
Quoted 39 lines
mm/huge_memory.c +30
-1@@ -2972,11 +3045,95 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }if (!pmd_trans_huge (src_pmdval)) { -spin_unlock (src_ptl);if (pmd_is_migration_entry (src_pmdval)) { +spin_unlock (src_ptl);pmd_migration_entry_wait (mm, src_pmd);return -EAGAIN ; } +#ifdef CONFIG_THP_SWAP +if (pmd_is_swap_entry (src_pmdval)) { +swp_entry_t entry; +struct swap_info_struct *si; +enum swap_pmd_cache cache_state; + +/* +* UFFDIO_MOVE on anon mappings requires single-owner +* semantics; refuse to move a shared swap entry. +*/ +if (!pmd_swp_exclusive (src_pmdval)) { +spin_unlock (src_ptl); +return -EBUSY ; + } + + entry =softleaf_from_pmd (src_pmdval); +spin_unlock (src_ptl); + +/* +* Pin the swap device against a racing swapoff. NULL +* means swapoff is in progress, which resolves on its +* own, so ask the caller to retry. An error pointer +* means the entry names no swap device at all: that +* never resolves, so report it instead of spinning in +* the caller's -EAGAIN loop. +*/ + si =get_swap_device (entry); +if (!si) +return -EAGAIN ;
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 3091
Quoted 10 lines
mm/huge_memory.c +10
-0+if (IS_ERR (si)) +return PTR_ERR (si); + + src_folio =NULL ; + cache_state =swap_pmd_cache_lookup (entry, &src_folio); +if (cache_state ==SWAP_PMD_CACHE_SPLIT ) { +put_swap_device (si); +__split_huge_pmd (src_vma, src_pmd, src_addr); +return -EAGAIN ; + }
[Severity: Medium]
What happens if src_pmd changes after spin_unlock(src_ptl) before get_swap_device(), swap_pmd_cache_lookup(), and __split_huge_pmd() run in move_pages_huge_pmd()?
Because src_ptl is released before calling get_swap_device(entry) and swap_pmd_cache_lookup(entry, &src_folio), and __split_huge_pmd() is called without verifying that src_pmd still matches src_pmdval:
1. A racing swap-in fault, MADV_WILLNEED, or swapoff can swap the THP back in (making src_pmd a present THP) or populate a PMD-sized swap-cache folio while those swap slots are reused or partially populated as order-0 folios. If swap_pmd_cache_lookup() then sees SWAP_PMD_CACHE_SPLIT, move_pages_huge_pmd() unconditionally calls __split_huge_pmd(src_vma, src_pmd, src_addr), shattering a present THP (whose folio will then be split in move_pages_ptes() on retry) or splitting a PMD swap entry backed by a PMD-sized swap-cache folio so the retry in move_pages_ptes() fails with -EBUSY.
2. A racing swapoff followed by swapon of a smaller swap device on the same swap type can complete after spin_unlock(src_ptl) and before get_swap_device(entry), causing get_swap_device() to log a false "Bad swap offset" error and return ERR_PTR(-EIO), failing UFFDIO_MOVE with -EIO instead of retrying.
By comparison, swapin_pmd_swap_entry() in mm/madvise.c performs swap_pmd_cache_lookup() and get_swap_device() while still holding the PMD lock, and swapin_walk_pmd_entry() rechecks pmd_same() before calling __split_huge_pmd():
mm/madvise.c:swapin_pmd_swap_entry() {
...
cache_state = swap_pmd_cache_lookup(entry, &folio);
...
si = get_swap_device(entry);
spin_unlock(ptl);
}Could move_pages_huge_pmd() keep src_ptl held across get_swap_device() and swap_pmd_cache_lookup(), and split under src_ptl via split_huge_pmd_locked() (or recheck pmd_same() before splitting)?
[ ... ]
Quoted 15 lines
mm/userfaultfd.c +7
-0@@ -2072,6 +2072,20 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,break ; } +/* +* A PMD swap entry at dst is a swapped-out THP, not a hole, +* and unlike a PMD migration entry it will not resolve on its +* own. Nothing below faults it back in: pte_alloc() skips a +* !pmd_none PMD, pte_offset_map_rw_nolock() then fails on the +* non-present PMD, and the -EAGAIN that produces would be +* retried forever by the loop below. Be strict, exactly as for
mm/userfaultfd.c static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2091
Quoted 10 lines
mm/userfaultfd.c +7
-0+* a present THP. +*/ +if (unlikely (pmd_is_swap_entry (dst_pmdval))) { + err = -EEXIST ; +break ; + } + ptl =pmd_trans_huge_lock (src_pmd, src_vma);if (ptl) {/* Check if we can move the pmd without splitting it. */
[Severity: Medium]
When move_splits_huge_pmd() or !pmd_none(dst_pmdval) is true in move_pages(), doesn't the fallback path unconditionally call split_huge_pmd() on a source PMD swap entry before checking whether the resulting PTE swap entries can be moved?
In mm/userfaultfd.c:move_pages(), the split branch only checks exclusivity when pmd_present(*src_pmd) is true:
mm/userfaultfd.c:move_pages() {
...
if (move_splits_huge_pmd(dst_addr, src_addr, src_start + len) ||
!pmd_none(dst_pmdval)) {
/* Can be a migration entry */
if (pmd_present(*src_pmd)) {
struct folio *folio = pmd_folio(*src_pmd);
if (!is_huge_zero_folio(folio) &&
!PageAnonExclusive(&folio->page)) {
spin_unlock(ptl);
err = -EBUSY;
break;
}
}
spin_unlock(ptl);
split_huge_pmd(src_vma, src_pmd, src_addr);
/* The folio will be split by move_pages_pte() */
continue;
}
}If *src_pmd is a PMD swap entry and UFFDIO_MOVE is called with a sub-PMD length, an unaligned address, or a destination PMD that already has an empty PTE page table (!pmd_none(dst_pmdval)):
1. If !pmd_swp_exclusive(*src_pmd), split_huge_pmd() shatters the shared PMD swap entry into PTE swap entries before move_pages_ptes() rejects them with -EBUSY.
2. If *src_pmd is exclusive and backed by a PMD-sized folio in the swap cache (SWAP_PMD_CACHE_HUGE), split_huge_pmd() shatters the PMD swap entry into PTE swap entries that still point to the large swap-cache folio. On the next iteration, move_pages_ptes() immediately fails with -EBUSY because folio_test_large(folio) is true:
mm/userfaultfd.c:move_pages_ptes() {
...
if (folio) {
if (folio_test_large(folio)) {
ret = -EBUSY;
folio_put(folio);
goto out;
}
}That leaves src_pmd permanently split into PTE swap entries that even a subsequent PMD-aligned UFFDIO_MOVE cannot move while the large folio remains in the swap cache.
Should move_pages() check pmd_is_swap_entry(*src_pmd) before splitting, and refuse with -EBUSY if !pmd_swp_exclusive(*src_pmd) or if the swap cache holds a large folio?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=28
Usama Arif
reviewerPATCH v9 29/32 mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
In reply to Usama Arifdo_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at the other subpages. A poisoned subpage that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens.
Skip batching for such a folio and let each page take its own fault, where the existing check catches it. Test every subpage rather than the folio-level PG_has_hwpoisoned: memory_failure() sets PageHWPoison() on the subpage before it acquires the folio lock, which we hold here.
--- mm/memory.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/memory.c +8 -1
@@ -5143,7 +5143,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_idx =0 ; address = vmf->address; ptep = vmf->pte; -if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +/* +* Scan every subpage rather than testing the folio-level +* PG_has_hwpoisoned: memory_failure() sets PageHWPoison on the subpage +* before it takes the folio lock, and we hold that lock, so the +* folio-level flag can still be clear here. +*/ +if (folio_test_large (folio) &&folio_test_swapcache (folio) && + !folio_has_any_hwpoisoned_page (folio)) {const int nr =folio_nr_pages (folio);const unsigned long idx =folio_page_idx (folio, page);unsigned long folio_start, folio_end;
Signature · 1 line
2.53.0-Meta
Usama Arif
reviewerPATCH v9 30/32 mm: handle PMD swap entry faults on swap-in
In reply to Usama ArifNothing faults a PMD swap entry back in. __handle_mm_fault() recognises a non-present PMD as device-private or a migration entry and returns 0 for anything else, so a swapped-out THP would refault forever.
do_huge_pmd_swap_page() resolves the whole mapping in one go, mirroring do_swap_page() at PMD granularity, and restores soft-dirty, uffd-wp and write permission the same way. It deliberately skips the order-0 readahead
paths: the fault already asks for the whole range, and readahead would populate per-page swap-cache state and force the entry to split before the fault could finish.
A PMD swap entry only promises HPAGE_PMD_NR consecutive slots, not that the cache still holds one folio for them. When it does not - the VMA no longer permits PMD-order THPs, the cache has been split, a slot is in zswap, the allocation or read fails, the memcg charge fails, or the cache insertion loses a race - split the entry and return 0 so the retry lands in do_swap_page(). That keeps a transient PMD-order allocation failure from becoming VM_FAULT_OOM. Only split if the PMD is still the entry we were called for, since every reason to fall back was observed without the PMD lock.
Before falling back with a locked PMD-sized folio, drop it from the swap cache if it has never been mapped or is not uptodate: do_swap_page() would hand an unmapped PMD-sized folio to folio_add_new_anon_rmap() as a whole while installing one PTE, and a folio that failed to read would make every later fault return VM_FAULT_SIGBUS instead of re-reading the slots. An uptodate anon folio stays cached so a poisoned subpage stays visible. A smaller folio is always left cached: it may be a zswap writeback folio still under I/O, which only the swap cache keeps alive. This mirrors unuse_pmd_entry().
Refuse to map a folio with a poisoned subpage so do_swap_page() can report VM_FAULT_HWPOISON for the offending page, and drop the exclusive marker when the folio is under writeback to an SWP_STABLE_WRITES backend such as zram, so a later write COWs rather than corrupting the writeback.
When the PMD ends up read-only but the fault was a write, call wp_huge_pmd() from the same handler so the COW does not need a second fault. Mask VM_FAULT_FALLBACK out of what it returns: splitting to PTE level is a normal outcome, but the bit is part of VM_FAULT_ERROR and arch fault handlers BUG() on it without an accompanying signal.
--- include/linux/huge_mm.h | 14 +++ mm/huge_memory.c | 261 ++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 42 +++++++ mm/memory.c | 40 +----- 4 files changed, 323 insertions(+), 34 deletions(-)
include/linux/huge_mm.h +14 -0
@@ -552,6 +552,15 @@ vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf);vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf); +#ifdef CONFIG_THP_SWAP +vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +#else +static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +extern struct folio *huge_zero_folio;extern unsigned long huge_zero_pfn;
@@ -757,6 +766,11 @@ static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf)return 0 ; } +static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{ +return 0 ; +} +static inline bool is_huge_zero_folio (const struct folio *folio) {return false;
mm/huge_memory.c +261 -0
273 lines not on this page · raw message
mm/internal.h +42 -0
48 lines not on this page · raw message
mm/memory.c +6 -34
@@ -4688,38 +4688,6 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)return 0 ; } -/* -* Check if we should call folio_free_swap to free the swap cache. -* folio_free_swap only frees the swap cache to release the slot if swap -* count is zero, so we don't need to check the swap count here. -*/ -static inline bool should_try_to_free_swap (struct swap_info_struct *si, -struct folio *folio, -struct vm_area_struct *vma, -bool exclusive, -unsigned int fault_flags) -{ -if (!folio_test_swapcache (folio)) -return false; -/* -* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap -* cache can help save some IO or memory overhead, but these devices -* are fast, and meanwhile, swap cache pinning the slot deferring the -* release of metadata or fragmentation is a more critical issue. -*/ -if (data_race (si->flags &SWP_SYNCHRONOUS_IO )) -return true; -if (mem_cgroup_swap_full (folio) || (vma->vm_flags &VM_LOCKED ) || -folio_test_mlocked (folio)) -return true; - -/* -* Free the swapcache only if we are the exclusive user and -* this is a write fault. -*/ -return (fault_flags &FAULT_FLAG_WRITE ) && exclusive; -} -static vm_fault_t pte_marker_clear (struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
@@ -6481,8 +6449,8 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)return VM_FAULT_FALLBACK ; } -/* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +vm_fault_t wp_huge_pmd (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;
@@ -6512,6 +6480,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) return VM_FAULT_FALLBACK; } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static vm_fault_t create_huge_pud (struct vm_fault *vmf) {
@@ -6778,6 +6747,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pmd_is_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd); +else if (IS_ENABLED (CONFIG_THP_SWAP ) && +pmd_is_swap_entry (vmf.orig_pmd)) +return do_huge_pmd_swap_page (&vmf);return 0 ; }if (pmd_trans_huge (vmf.orig_pmd)) {
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 30/32] mm: handle PMD swap entry faults on swap-in
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Critical] mm: use-after-free in do_huge_pmd_swap_page() from unlocking PTL first - [Critical] mm: unmapped…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Critical] mm: use-after-free in do_huge_pmd_swap_page() from unlocking PTL first
- [Critical] mm: unmapped PMD swap-cache folio in do_huge_pmd_swap_page() corrupts rmap
- [Medium] mm: missing PTE fault fallback after PMD split in do_huge_pmd_swap_page()
--
commit 651cc7048660407a18f57e406abe979d3a9ccbfe
Author: Usama Arif <[email protected]>
mm: handle PMD swap entry faults on swap-in
This commit adds do_huge_pmd_swap_page() and hooks it into __handle_mm_fault() to fault in PMD swap entries at PMD granularity when a PMD-sized folio is available, or split the PMD swap entry and fall back to PTE swap-in otherwise.
Quoted 4 lines
mm/huge_memory.c +0
-0
[ ... ]
Quoted 19 lines
mm/huge_memory.c +15
-0@@ -2425,6 +2426,266 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_THP_SWAP +/** +* do_huge_pmd_swap_page() - Handle a fault on a PMD-level swap entry. +* @vmf: Fault context. vmf->orig_pmd contains the swap PMD. +* +* A PMD swap entry is a compact encoding for HPAGE_PMD_NR consecutive swap +* slots. If the swap cache still has one PMD-sized folio covering the range, +* map it directly at PMD level. If the range has been split into per-page +* cache state, or zswap may have per-page state for it, split the PMD swap +* entry and retry at PTE granularity. +* +* Return: VM_FAULT_* flags. +*/ +vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{
[ ... ]
Quoted 47 lines
mm/huge_memory.c +47
-0+if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_PAGEFAULT , +HPAGE_PMD_ORDER )) { +__split_huge_pmd (vma, vmf->pmd, haddr); +return 0 ; + } + + swp_entry = entry; + +/* Prevent swapoff from happening to us. */ + si =get_swap_device (swp_entry); +if (IS_ERR_OR_NULL (si)) { +if (IS_ERR (si)) +return VM_FAULT_SIGBUS ; +return 0 ; + } + + cache_state =swap_pmd_cache_lookup (swp_entry, &folio); +if (cache_state ==SWAP_PMD_CACHE_SPLIT ) + goto split_fallback; +if (!folio) { +/* +* PMD swap entries encode ordinary per-page swap slots. If any +* slot is in zswap, split and let the PTE swap path load the +* range per page. Otherwise the range is all on disk and can be +* read back as one PMD-sized folio. +*/ +if (zswap_is_present (swp_entry,HPAGE_PMD_NR )) + goto split_fallback; + + folio =swapin_sync (swp_entry,GFP_HIGHUSER_MOVABLE , +BIT (HPAGE_PMD_ORDER ), vmf,NULL ,0 ); +if (IS_ERR_OR_NULL (folio)) + goto split_fallback; + +/* Had to read from swap area: Major fault */ + ret =VM_FAULT_MAJOR ; +count_vm_event (PGMAJFAULT ); +count_memcg_event_mm (mm,PGMAJFAULT ); + } + + ret |=folio_lock_or_retry (folio, vmf); +if (ret &VM_FAULT_RETRY ) + goto out_release; + +/* Verify the folio is still in swap cache and matches our entry */ +if (unlikely (!folio_matches_swap_entry (folio, swp_entry))) + goto out_page;
[ ... ]
Quoted 6 lines
mm/huge_memory.c +6
-0+/* Lock the PMD and verify it hasn't changed */ + vmf->ptl =pmd_lock (mm, vmf->pmd); +if (unlikely (!pmd_same (vmf->orig_pmd,pmdp_get (vmf->pmd)))) { +spin_unlock (vmf->ptl); + goto out_page; + }
[ ... ]
Quoted 8 lines
mm/huge_memory.c +8
-0+set_pmd_at (mm, haddr, vmf->pmd, pmd); +update_mmu_cache_pmd (vma, haddr, vmf->pmd); + +/* Update orig_pmd for any follow-up wp_huge_pmd() below. */ + vmf->orig_pmd = pmd; + +/* +* Conditionally try to free up the swap cache. Do it after mapping,
mm/huge_memory.c vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · line 2626
Quoted 10 lines
mm/huge_memory.c +10
-0+* so raced page faults will likely see the folio in swap cache and +* wait on the folio lock. +*/ +if (should_try_to_free_swap (si, folio, vma, exclusive, vmf->flags)) +folio_free_swap (folio); + +spin_unlock (vmf->ptl); + +folio_unlock (folio); +put_swap_device (si);
[Severity: Critical]
Can releasing vmf->ptl before calling folio_unlock(folio) in do_huge_pmd_swap_page() lead to a use-after-free on folio?
do_huge_pmd_swap_page() holds a single reference to folio from swap_pmd_cache_lookup() or swapin_sync(), and set_pmd_at() transfers that reference to the PMD mapping. When should_try_to_free_swap() is true (such as on SWP_SYNCHRONOUS_IO swap devices, exclusive write faults, or VM_LOCKED VMAs), folio_free_swap(folio) drops the swap-cache references as well.
At that point the faulting thread no longer holds its own reference on folio; only vmf->ptl prevents the PMD mapping reference from going away. In mm/memory.c:do_swap_page(), folio_unlock(folio) is called before pte_unmap_unlock(vmf->pte, vmf->ptl):
if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags)) folio_free_swap(folio); folio_unlock(folio); ... pte_unmap_unlock(vmf->pte, vmf->ptl);
With spin_unlock(vmf->ptl) happening first here, a racing MADV_DONTNEED or munmap() in zap_huge_pmd() (or move_folios_to_lru() dropping a transient LRU isolation reference) can unmap and free the folio while PG_locked is still set before folio_unlock(folio) runs on this CPU. Should folio_unlock(folio) be called before spin_unlock(vmf->ptl)?
Quoted 35 lines
mm/huge_memory.c +35
-0+ +/* +* If the write fault wasn't satisfied above (folio is shared without +* exclusivity), call wp_huge_pmd() to handle COW or +* userfaultfd-wp without forcing a second fault. +* +* wp_huge_pmd() may return VM_FAULT_FALLBACK if it had to split the +* PMD; that's a normal outcome, and the natural PTE-level refault will +* complete the COW. Mask it so callers (and the arch fault handler) +* don't see VM_FAULT_FALLBACK as a fatal VM_FAULT_ERROR. +*/ +if (write && !pmd_write (pmd) && !rwp_restore) { +vm_fault_t wp_ret =wp_huge_pmd (vmf); + + wp_ret &= ~VM_FAULT_FALLBACK ; + ret |= wp_ret; +if (ret &VM_FAULT_ERROR ) + ret &=VM_FAULT_ERROR ; + } + +return ret; + +out_page: +folio_unlock (folio); +out_release: +folio_put (folio); +put_swap_device (si); +return ret; + +unlock_split_fallback: +/* +* PTE fallback cannot add a single-page rmap to a PMD-sized folio that +* has never been mapped: do_swap_page() would hand the whole folio to +* folio_add_new_anon_rmap() while installing one PTE. Nor can it do +* anything useful with a folio that failed to read. Remove either from
mm/huge_memory.c vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · line 2671
Quoted 10 lines
mm/huge_memory.c +10
-0+* the swap cache so each slot is read back into its own order-0 folio. +* An uptodate anon swap-cache folio can be mapped one PTE at a time and +* must stay cached, so that any poisoned subpage stays visible to +* do_swap_page(). This mirrors unuse_pmd_entry(). +*/ +if (folio_matches_swap_entry (folio, swp_entry) && + (!folio_test_uptodate (folio) || !folio_test_anon (folio))) +swap_cache_del_folio (folio); +folio_unlock (folio); +folio_put (folio);
[Severity: Critical]
What happens if a newly allocated PMD-sized swap-cache folio (!folio_test_anon(folio) or !folio_test_uptodate(folio)) exits via out_page or out_release instead of unlock_split_fallback?
There appear to be a few paths where an unmapped PMD-sized folio remains in the swap cache after the PMD swap entry is split or zapped:
1. If the PMD changed while swapin_sync() or folio_lock_or_retry() was in progress (for example, a racing thread split the PMD at split_fallback, or mprotect/madvise split the PMD), the !pmd_same() check jumps to out_page without deleting the !folio_test_anon(folio) (or !folio_test_uptodate(folio)) folio from the swap cache.
2. If the PMD changed because a racing zap_huge_pmd() or madvise_free_huge_pmd() zapped the swap entry while this thread held the folio lock, swap_put_entries_direct() -> __try_to_reclaim_swap() fails folio_trylock(folio). Unlike mm/memory.c:do_swap_page(), which calls folio_free_swap(folio) under out_page before unlocking the folio, out_page here does not call folio_free_swap(folio), leaving the folio and swap slots stranded in the swap cache.
3. On asynchronous swap devices, swapin_sync() inserts a locked !folio_test_anon(folio) PMD-sized folio into the swap cache, and folio_lock_or_retry() drops the fault lock and jumps to out_release with VM_FAULT_RETRY (and MADV_WILLNEED's swapin_pmd_swap_entry() similarly populates an unmapped PMD-sized folio). If the PMD is subsequently split before the folio is mapped at PMD level (including via the !thp_vma_allowable_order() check at the top of do_huge_pmd_swap_page(), which splits the PMD without checking the swap cache), the unmapped PMD-sized folio stays in the swap cache.
When a subsequent PTE fault enters mm/memory.c:do_swap_page() on the HPAGE_PMD_NR contiguous split PTE swap entries, the large-folio PTE batching logic batches all nr_pages = HPAGE_PMD_NR PTEs. Because nr_pages equals folio_nr_pages(folio), the check in do_swap_page():
if (!folio_test_anon(folio) && folio_test_large(folio) &&
nr_pages != folio_nr_pages(folio)) {
if (!WARN_ON_ONCE(folio_test_dirty(folio)))
swap_cache_del_folio(folio);
goto out_nomap;
}does not trigger, and do_swap_page() passes the PMD-mappable folio to folio_add_new_anon_rmap() while installing PTEs. Since folio_add_new_anon_rmap() initializes any folio_test_pmd_mappable(folio) for a PMD mapping (_entire_mapcount = 0, _large_mapcount = 1, _nr_pages_mapped = ENTIRELY_MAPPED, and PageAnonExclusive only on subpage 0), wouldn't PTE-mapping it corrupt the rmap/mapcounts and PageAnonExclusive state across subpages?
Quoted 4 lines
mm/memory.c +0
-0
[ ... ]
mm/memory.c static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, · line 6755
Quoted 10 lines
mm/memory.c +3
-0@@ -6778,6 +6747,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pmd_is_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd); +else if (IS_ENABLED (CONFIG_THP_SWAP ) && +pmd_is_swap_entry (vmf.orig_pmd)) +return do_huge_pmd_swap_page (&vmf);return 0 ; }if (pmd_trans_huge (vmf.orig_pmd)) {
[Severity: Medium]
When do_huge_pmd_swap_page() splits the PMD (either at the !thp_vma_allowable_order() check, at split_fallback, or inside wp_huge_pmd() when a non-exclusive swapped-in THP must be split into PTEs for COW), should __handle_mm_fault() fall through to the fallback label and call handle_pte_fault(&vmf) rather than returning 0 (or masking out VM_FAULT_FALLBACK)?
Other PMD handlers in __handle_mm_fault() such as create_huge_pmd() and wp_huge_pmd() return VM_FAULT_FALLBACK so __handle_mm_fault() jumps to
fallback: and resolves the PTE fault in the same handle_mm_fault() call.
By returning 0 after splitting the PMD swap entry (or clearing VM_FAULT_FALLBACK when wp_huge_pmd() splits the read-only PMD into read-only PTEs without copying), callers of handle_mm_fault() that do not loop on 0 - such as fixup_user_fault(), fault_in_safe_writeable(), and user_event_mm_fault_in() - see a successful return while the target PTE is still a non-present swap entry or read-only PTE, causing subsequent pagefault-disabled user accesses to fail with -EFAULT.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=30
Usama Arif
reviewerPATCH v9 31/32 mm: install PMD swap entries on swap-out
In reply to Usama ArifWhen reclaim swaps out a PMD-mapped anonymous THP it first splits the PMD via TTU_SPLIT_HUGE_PMD. The huge mapping is lost for the whole swap
round-trip: swapping the range back in takes HPAGE_PMD_NR faults and leaves as many small mappings, and the process waits for khugepaged to collapse them again.
The PMD does not have to be split. A contiguous run of slots was already secured when the folio was added to the swap cache - a non-contiguous allocation would have split the folio first - so the whole mapping can be replaced by one PMD-level swap entry encoding the first slot.
shrink_folio_list() therefore stops asking for TTU_SPLIT_HUGE_PMD for a PMD-mappable folio already in the swap cache, and try_to_unmap_one() grows a PMD branch. TTU_SPLIT_HUGE_PMD remains the fallback for everything else.
set_pmd_swap_entry() is deliberately close in shape to set_pmd_migration_entry(): invalidate the mapping while keeping the original for rollback, take a swap reference on every slot, transfer the exclusive state, propagate the dirty bit to the folio so writeback is not lost, add the mm to mmlist before the entry becomes visible, and carry over soft-dirty and uffd-wp. Any step that can fail restores the mapping first.
The entry encodes exactly what the PTE entries would, so swap_map accounting is unchanged: each slot carries a count of one, released individually on a later split or together on swap-in.
zswap needs no handling here. It stores the folio as order-0 entries, and the PMD-order swap-in users split and fall back to PTEs if any covered slot turns out to have a zswap entry.
thp_swpout_pmd counts PMD mappings replaced this way. Unlike thp_swpout it counts mappings rather than folios, so a fork-shared THP can increment it once per mapping.
--- Documentation/admin-guide/mm/transhuge.rst | 5 ++ include/linux/huge_mm.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 84 ++++++++++++++++++++++ mm/rmap.c | 19 +++++ mm/vmscan.c | 9 ++- mm/vmstat.c | 1 + 7 files changed, 120 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/transhuge.rst +5 -0
@@ -632,6 +632,11 @@ thp_swpout is incremented every time a huge page is swapout in one piece without splitting. +thp_swpout_pmd + is incremented every time a PMD mapping is replaced by a PMD-level + swap entry. A fork-shared THP can increment this counter once for each + PMD mapping that is swapped out. + thp_swpout_fallback is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space
include/linux/huge_mm.h +2 -0
@@ -554,6 +554,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);#ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio);#else static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) {
include/linux/vm_event_item.h +1 -0
@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_ZERO_PAGE_ALLOC_FAILED ,THP_SWPOUT ,THP_SWPOUT_FALLBACK , +THP_SWPOUT_PMD , #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE,
mm/huge_memory.c +84 -0
87 lines not on this page · raw message
mm/rmap.c +19 -0
@@ -2275,6 +2275,25 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, }if (!pvmw.pte) { +#ifdef CONFIG_THP_SWAP +/* +* If the folio is in the swap cache and we're not +* asked to split, install a PMD-level swap entry. +*/ +if (!(flags &TTU_SPLIT_HUGE_PMD ) && +folio_test_anon (folio) && +folio_test_swapcache (folio)) { +if (set_pmd_swap_entry (&pvmw, folio)) + goto walk_abort; + +add_mm_counter (mm,MM_ANONPAGES , + -HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , +HPAGE_PMD_NR ); + goto walk_done; + } +#endif +if (flags &TTU_SPLIT_HUGE_PMD ) {/* * We temporarily have to drop the PTL and
mm/vmscan.c +8 -1
@@ -1330,7 +1330,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio); -if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the +* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio))) flags |=TTU_SPLIT_HUGE_PMD ;/* * Without TTU_SYNC, try_to_unmap will only begin to
mm/vmstat.c +1 -0
@@ -1435,6 +1435,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I (THP_SWPOUT_PMD )] ="thp_swpout_pmd" , #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 31/32] mm: install PMD swap entries on swap-out
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Critical] mm: rmap corruption in do_swap_page() on unmapped PMD swapcache folio - [Critical] mm: stale…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Critical] mm: rmap corruption in do_swap_page() on unmapped PMD swapcache folio
- [Critical] mm: stale swap data via set_pmd_swap_entry() after madvise_free_huge_pmd()
- [High] mm: swapped-out PMD THPs omitted from core dumps in follow_pmd_mask()
--
commit 5fc4ff682d479e59aca8b75c2c237658535938ae
Author: Usama Arif <[email protected]>
mm: install PMD swap entries on swap-out
This commit avoids splitting a PMD-mapped anonymous THP during swap-out when the folio is already in the swap cache, replacing the PMD mapping directly with a PMD-level swap entry via set_pmd_swap_entry().
Quoted 24 lines
mm/huge_memory.c +16
-0@@ -5778,3 +5778,87 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *foliotrace_remove_migration_pmd (address,pmd_val (pmde)); }#endif + +#ifdef CONFIG_THP_SWAP +/** +* set_pmd_swap_entry() - Replace a PMD mapping with a PMD-level swap entry. +* @pvmw: Page vma mapped walk context, must have pvmw->pmd set and +* pvmw->pte NULL (i.e. PMD-mapped). +* @folio: The folio being swapped out. Must be in the swap cache. +* +* This installs a PMD-level swap entry in place of a present PMD mapping, +* avoiding the need to split the PMD into PTE-level swap entries. +* +* Return: 0 on success, negative error code on failure. +*/ +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio) +{
[ ... ]
Quoted 4 lines
mm/huge_memory.c +4
-0+mm_prepare_for_swap_entries (mm); + +if (pmd_dirty (pmdval)) +folio_mark_dirty (folio);
mm/huge_memory.c void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio · line 5856
Quoted 10 lines
mm/huge_memory.c +10
-0+ + entry = folio->swap; + pmdswp =softleaf_to_pmd (entry); +if (pmd_soft_dirty (pmdval)) + pmdswp =pmd_swp_mksoft_dirty (pmdswp); +if (pmd_uffd (pmdval)) + pmdswp =pmd_swp_mkuffd (pmdswp); +if (anon_exclusive) + pmdswp =pmd_swp_mkexclusive (pmdswp); +set_pmd_at (mm, haddr, pvmw->pmd, pmdswp);
[Severity: Critical]
Can installing a PMD swap entry here lead to rmap corruption in do_swap_page() if the PMD swap entry is later split while a freshly allocated PMD-order folio is in the swap cache?
Once set_pmd_swap_entry() installs a PMD swap entry, swapin_pmd_swap_entry() (via MADV_WILLNEED) or do_huge_pmd_swap_page() calls swapin_sync() with BIT(HPAGE_PMD_ORDER), which allocates an unmapped PMD-order folio in the swap cache with !folio_test_anon(folio).
If the PMD swap entry is then split into HPAGE_PMD_NR contiguous PTE swap entries while that unmapped PMD-order folio remains in the swap cache, for example when madvise(MADV_WILLNEED) is followed by madvise(MADV_NOHUGEPAGE) or prctl(PR_SET_THP_DISABLE) and a page fault enters do_huge_pmd_swap_page():
mm/huge_memory.c:do_huge_pmd_swap_page()
if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT,
HPAGE_PMD_ORDER)) {
__split_huge_pmd(vma, vmf->pmd, haddr);
return 0;
}(or when do_huge_pmd_swap_page() backs out at folio_lock_or_retry() or !pmd_same() after a concurrent split), the subsequent fault falls back to do_swap_page().
In do_swap_page(), swap_pte_batch() batches all HPAGE_PMD_NR contiguous PTE swap entries, so nr_pages == folio_nr_pages(folio) and the partial-mapping check does not remove the folio from the swap cache. do_swap_page() then installs PTEs via set_ptes() and calls folio_add_new_anon_rmap():
mm/memory.c:do_swap_page()
} else if (!folio_test_anon(folio)) {
VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) != nr_pages, folio);
VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
folio_put_swap(folio, NULL);
}However, folio_add_new_anon_rmap() assumes any folio_test_pmd_mappable(folio) folio is being PMD-mapped rather than PTE-mapped:
mm/rmap.c:folio_add_new_anon_rmap()
} else if (!folio_test_pmd_mappable(folio)) {
...
} else {
nr = folio_large_nr_pages(folio);
/* increment count (starts at -1) */
atomic_set(&folio->_entire_mapcount, 0);
folio_set_large_mapcount(folio, 1, vma);
if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT))
atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED);
if (exclusive)
SetPageAnonExclusive(&folio->page);
nr_pmdmapped = nr;
}Because the folio was mapped with PTEs via set_ptes(), this leaves page->_mapcount at -1 on all subpages, sets _large_mapcount to 1 instead of HPAGE_PMD_NR, sets _entire_mapcount to 0, and only sets PageAnonExclusive on the head page. When zap_pte_range() later unmaps those PTEs via folio_remove_rmap_ptes(), page->_mapcount underflows to -2, _large_mapcount underflows, and _entire_mapcount is never decremented, triggering a Bad page state bug when the folio is freed. (Similarly, if swapoff runs after the PMD swap entry is split, unuse_pte() hits VM_WARN_ON_ONCE(folio_test_large(folio)) and calls folio_add_new_anon_rmap() while installing a single PTE.)
Quoted 8 lines
mm/rmap.c +0
-0@@ -2275,6 +2275,25 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, }if (!pvmw.pte) {
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2287
Quoted 10 lines
mm/rmap.c +10
-0+#ifdef CONFIG_THP_SWAP +/* +* If the folio is in the swap cache and we're not +* asked to split, install a PMD-level swap entry. +*/ +if (!(flags &TTU_SPLIT_HUGE_PMD ) && +folio_test_anon (folio) && +folio_test_swapcache (folio)) { +if (set_pmd_swap_entry (&pvmw, folio)) + goto walk_abort;
[Severity: High]
Does replacing the PMD with a non-present PMD swap entry cause swapped-out THPs to be silently omitted from core dumps?
Previously, reclaiming an anonymous THP split the PMD into PTE swap entries. When dump_user_range() (or mte_dump_tag_range() on arm64) calls get_dump_page() with FOLL_FORCE | FOLL_DUMP | FOLL_GET, follow_page_pte() returns NULL for a non-none, non-present PTE, which causes __get_user_pages() to call faultin_page() and fault the swapped-out page back in.
With a non-present PMD swap entry, follow_pmd_mask() instead calls no_page_table():
mm/gup.c:follow_pmd_mask()
if (pmd_none(pmdval)) return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) return no_page_table(vma, flags, address);
And no_page_table() treats FOLL_DUMP on an anonymous VMA as an unpopulated hole, returning ERR_PTR(-EFAULT):
mm/gup.c:no_page_table()
if (flags & FOLL_DUMP) {
if (vma->vm_ops && vma->vm_ops->may_split)
return ERR_PTR(-ENOSYS);
/*
* For anonymous or fault-less VMA, return -EFAULT to judge
* there is no page Mapped, so we can skip dumping them.
*/
if (vma_is_anonymous(vma) || !vma->vm_ops->fault)
return ERR_PTR(-EFAULT);
}Because __get_user_pages() sees IS_ERR(page), it aborts without calling faultin_page(), so get_dump_page() returns NULL and dump_user_range() skips the entire 2MB THP range in the core dump. Should follow_pmd_mask() only call no_page_table() when pmd_none(pmdval) and return NULL for non-present PMD swap entries?
Quoted 8 lines
mm/vmscan.c +0
-0@@ -1330,7 +1330,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio);
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1341
Quoted 10 lines
mm/vmscan.c +8
-1-if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the +* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio))) flags |=TTU_SPLIT_HUGE_PMD ;
[Severity: Critical]
Can skipping TTU_SPLIT_HUGE_PMD for a PMD-mapped THP that is already in the swap cache result in reclaiming the folio without writeback after madvise(MADV_FREE), leaving the PMD swap entry pointing to stale or unwritten swap data?
With this change and do_huge_pmd_swap_page(), an anonymous THP can be both PMD-mapped and in the swap cache (for instance, after a read swap-in in do_huge_pmd_swap_page() when should_try_to_free_swap() is false, where the process then writes to the THP in memory while the swap slot holds older data; or if an earlier reclaim pass called folio_alloc_swap() and try_to_unmap() aborted before unmapping all mappings).
If userspace then calls madvise(MADV_FREE) on that range, madvise_free_huge_pmd() clears both the folio dirty bit and the PMD dirty bit without freeing the swap cache entry (unlike madvise_free_pte_range(), which checks folio_test_swapcache(folio) and calls folio_free_swap()):
mm/huge_memory.c:madvise_free_huge_pmd()
if (folio_test_dirty(folio))
folio_clear_dirty(folio);
folio_unlock(folio);
if (pmd_young(orig_pmd) || pmd_dirty(orig_pmd)) {
pmdp_invalidate(vma, addr, pmd);
orig_pmd = pmd_mkold(orig_pmd);
orig_pmd = pmd_mkclean(orig_pmd);
set_pmd_at(mm, addr, pmd, orig_pmd);
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
}
folio_mark_lazyfree(folio);Because folio_test_swapcache(folio) is still true, folio_mark_lazyfree() returns early without clearing PG_swapbacked:
mm/folio.c:folio_mark_lazyfree()
if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) || !folio_test_lru(folio) || folio_test_swapcache(folio) || folio_test_unevictable(folio)) return;
When shrink_folio_list() later reclaims this folio, folio_test_swapcache() is already true, so it skips the !folio_test_swapcache(folio) block (which would have called folio_mark_dirty(folio)), skips TTU_SPLIT_HUGE_PMD here, and calls try_to_unmap_one() -> set_pmd_swap_entry().
In set_pmd_swap_entry(), pmd_dirty(pmdval) is false, so the folio is not marked dirty while a PMD swap entry is installed for folio->swap. Back in shrink_folio_list(), folio_test_dirty(folio) is false, so pageout() is skipped and the folio is freed without writing it out, leaving subsequent faults to read stale pre-modification data or unwritten swap blocks from disk.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=31
Usama Arif
reviewerPATCH v9 32/32 selftests/mm: add PMD swap entry tests
In reply to Usama ArifEach test gets a fresh PMD-mapped THP from fixture setup, fills it with a pattern that differs between base pages so a reordered split is detectable, swaps it out with MADV_PAGEOUT, and verifies thp_swpout_pmd increased before the test body runs.
The tests are basic, swapin_sync, fork, fork_cow, write, rwp_swapin, munmap, mprotect, split_mprotect, split_munmap, uffdio_move, mremap, pagemap, mincore, madvise_free, madvise_willneed and swapoff.
MADV_PAGEOUT leaves the folio in the swap cache on an asynchronous device, so a fault would only remap it and never reach swapin_sync(). Fixture setup therefore asks the current cgroup to reclaim afterwards, and uses mincore()
- which reports a PMD swap entry as resident exactly while the cache holds it - to tell whether that worked. Tests run either way; swapin_sync skips if the cache survived. The swapoff test runs only when PMD_SWAP_DEVICE is the sole active swap device, and restores it at its original priority.
Skip rather than fail where the kernel never promised anything. thp_swpout_pmd and thp_swpout_fallback are system-wide, so test the fallback counter first: if reclaim split any THP we cannot be sure it was not ours. That also covers a kernel without CONFIG_THP_SWAP. A kernel with no thp_swpout_pmd counter does not implement the feature at all.
With zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so.
Add a /proc/vmstat field reader to vm_util.c so other tests can use it too, and register pmd_swap with run_vmtests.sh and the default runner.
--- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 6 files changed, 1025 insertions(+) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c
tools/testing/selftests/mm/Makefile +2 -0
@@ -109,6 +109,7 @@ TEST_GEN_FILES += folio_split_race_test TEST_GEN_FILES += folio_order_check TEST_GEN_FILES += khugepaged_sync_check TEST_GEN_FILES += khugepaged_race +TEST_GEN_FILES += pmd_swap TEST_GEN_FILES += soft-dirty ifeq ($(ARCH),x86_64)
@@ -168,6 +169,7 @@ TEST_PROGS += ksft_mremap.sh TEST_PROGS += ksft_pagemap.sh TEST_PROGS += ksft_pfnmap.sh TEST_PROGS += ksft_pkey.sh +TEST_PROGS += ksft_pmd_swap.sh TEST_PROGS += ksft_process_madv.sh TEST_PROGS += ksft_process_mrelease.sh TEST_PROGS += ksft_rmap.sh
tools/testing/selftests/mm/ksft_pmd_swap.sh new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0 + +./run_vmtests.sh -t pmd_swap
tools/testing/selftests/mm/pmd_swap.c new file +989 -0
989 lines not on this page · raw message
tools/testing/selftests/mm/run_vmtests.sh +4 -0
@@ -71,6 +71,8 @@ separated by spaces: test pagemap_scan IOCTL - pfnmap tests for VM_PFNMAP handling +- pmd_swap + tests for PMD-level swap entries - process_madv test for process_madv - cow
@@ -397,6 +399,8 @@ CATEGORY="pagemap" run_test ./pagemap_ioctl CATEGORY="pfnmap" run_test ./pfnmap +CATEGORY="pmd_swap" run_test ./pmd_swap + # COW tests CATEGORY="cow" run_test ./cow
tools/testing/selftests/mm/vm_util.c +24 -0
@@ -199,6 +199,30 @@ uint64_t pmd_pshift(void)return __pmd_pshift; } +long read_vmstat (const char *name) +{ +char line[MAX_LINE_LENGTH ]; +size_t name_len =strlen (name); +long val =-1 ; +FILE *f; + + f =fopen ("/proc/vmstat" ,"r" ); +if (!f) +return -1 ; +/* +* Match the whole field name: "thp_swpout" is a prefix of +* "thp_swpout_fallback" and "thp_swpout_pmd". +*/ +while (fgets (line,sizeof (line), f)) { +if (!strncmp (line, name, name_len) && line[name_len] ==' ' ) { + val =strtol (line + name_len +1 ,NULL ,10 ); +break ; + } + } +fclose (f); +return val; +} +unsigned long rss_anon (void ) {unsigned long rss_anon =0 ;
tools/testing/selftests/mm/vm_util.h +2 -0
@@ -107,6 +107,8 @@ void clear_softdirty(void);bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t pmd_psize (void );uint64_t pmd_pshift (void ); +/* Returns the counter value, or -1 if /proc/vmstat has no such field. */ +long read_vmstat (const char *name);unsigned long rss_anon (void );bool check_huge_anon (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);bool check_huge_file (void *addr,size_t len,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v9 32/32] selftests/mm: add PMD swap entry tests
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] selftests/mm: check_huge_anon() in swapoff test passes on swapped PMDs - [Medium] selftests/mm…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] selftests/mm: check_huge_anon() in swapoff test passes on swapped PMDs
- [Medium] selftests/mm: uffdio_move test fails UFFDIO_API when RWP is unsupported
--
commit 19ef341b36a063e4b63f435180149741746c0988
Author: Usama Arif <[email protected]>
selftests/mm: add PMD swap entry tests
This commit adds a selftest suite for PMD-level swap entries covering basic fault-in, swapin_sync, fork, COW, mprotect, munmap, userfaultfd move/RWP, mremap, pagemap, mincore, madvise, and swapoff operations. It also adds a read_vmstat() helper to vm_util.c and wires the test into run_vmtests.sh.
Quoted 6 lines
tools/testing/selftests/mm/pmd_swap.c new file +0
-0@@ -0,0 +1,989 @@
[ ... ]
Quoted 12 lines
tools/testing/selftests/mm/pmd_swap.c new file +12
-0+TEST_F(pmd_swap, uffdio_move) +{ +struct uffdio_register reg = {}; +struct uffdio_move move = {}; +struct uffdio_api api = {}; +bool rwp; + + self->aux =mmap_pmd_aligned (self->pmd_size); +if (self->aux ==MAP_FAILED ) +SKIP (return,"Could not mmap aligned dst\n" ); +ASSERT_EQ (madvise (self->aux, self->pmd_size,MADV_HUGEPAGE ),0 ); +
tools/testing/selftests/mm/pmd_swap.c line 809
Quoted 10 lines
tools/testing/selftests/mm/pmd_swap.c new file +10
-0+ self->uffd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK ); +if (self->uffd <0 ) +SKIP (return,"userfaultfd unavailable\n" ); + + api.api =UFFD_API ; + api.features =UFFD_FEATURE_MOVE |UFFD_FEATURE_RWP ; +if (ioctl (self->uffd,UFFDIO_API , &api) || + !(api.features &UFFD_FEATURE_MOVE )) +SKIP (return,"UFFD_FEATURE_MOVE unsupported\n" ); + rwp = api.features &UFFD_FEATURE_RWP ;
[Severity: Medium]
In TEST_F(pmd_swap, uffdio_move) in tools/testing/selftests/mm/pmd_swap.c, will requesting UFFD_FEATURE_MOVE | UFFD_FEATURE_RWP in a single UFFDIO_API ioctl cause the test to skip whenever UFFD_FEATURE_RWP is unsupported?
In mm/userfaultfd.c, userfaultfd_api() rejects the call with -EINVAL if any requested feature bit is unavailable rather than masking off unsupported bits:
mm/userfaultfd.c:userfaultfd_api() {
...
uffdio_api.features = uffd_api_available_features(ctx, features);
ret = -EINVAL;
if (features & ~uffdio_api.features)
goto err_out;
...
}On architectures or configurations where UFFD_FEATURE_MOVE is supported but UFFD_FEATURE_RWP is cleared by uffd_api_available_features() (because VM_UFFD_RWP == VM_NONE || !pgtable_supports_uffd()), ioctl(self->uffd, UFFDIO_API, &api) fails with -EINVAL, skipping the test with "UFFD_FEATURE_MOVE unsupported" and leaving the !rwp fallback path unreachable.
[ ... ]
Quoted 25 lines
tools/testing/selftests/mm/pmd_swap.c new file +25
-0+TEST_F(pmd_swap, swapoff) +{ +int ret, err; + + self->swap_dev =getenv ("PMD_SWAP_DEVICE" ); +if (!self->swap_dev) +SKIP (return,"PMD_SWAP_DEVICE env var not set\n" ); +/* +* Otherwise a higher-priority device may have taken the PMD swap +* entry and swapoff() would operate on the wrong backend. +*/ +if (!swap_device_is_only_active (self->swap_dev, &self->swap_prio)) +SKIP (return,"PMD_SWAP_DEVICE must be the only active swap device\n" ); + + self->uffd =register_rwp (self->mem, self->pmd_size, true); + + ret =swapoff (self->swap_dev); + err = errno; +if (!ret) + self->swap_disabled = true; + ASSERT_EQ(ret, 0) { +TH_LOG ("swapoff(%s) failed: %s" , self->swap_dev,strerror (err)); + } + +/*
tools/testing/selftests/mm/pmd_swap.c line 973
Quoted 10 lines
tools/testing/selftests/mm/pmd_swap.c new file +10
-0+* Check residency before touching the memory. If we read +* first, a bug that left a PMD swap entry in place after swapoff +* would silently trigger do_huge_pmd_swap_page() and reinstall a +* PMD mapping, masking the regression. +*/ +if (self->zswap_enabled) +ksft_print_msg ("zswap enabled: PMD restoration not checked\n" ); +else +ASSERT_TRUE (check_huge_anon (self->mem, self->pmd_size,1 , + self->pmd_size));
[Severity: Medium]
Does check_huge_anon() in TEST_F(pmd_swap, swapoff) in tools/testing/selftests/mm/pmd_swap.c actually verify that the PMD swap entry is no longer swapped out before the test reads self->mem?
In tools/testing/selftests/mm/vm_util.c, check_huge_anon() calls __check_huge() followed by __check_type():
tools/testing/selftests/mm/vm_util.c:__check_huge() {
...
for (; start < end; start += hpage_size) {
categories = pagemap_scan_get_categories(pagemap_fd, start);
if (categories & PAGE_IS_HUGE)
nr_pmd_mappings++;
}
...
}tools/testing/selftests/mm/vm_util.c:__check_type() {
...
for (; start < end; start += page_size) {
categories = pagemap_scan_get_categories(pagemap_fd, start);
if ((categories & PAGE_IS_PRESENT) != PAGE_IS_PRESENT)
continue;
if ((type == CHECK_TYPE_FILE) != !!(categories & PAGE_IS_FILE))
goto out;
}
ret = true;
...
}In fs/proc/task_mmu.c, pagemap_scan_thp_entry()->pagemap_thp_category() sets PAGE_IS_HUGE | PAGE_IS_SWAPPED on a non-present PMD swap entry. Because __check_huge() only tests PAGE_IS_HUGE without PAGE_IS_PRESENT, and __check_type() skips non-present entries via continue, check_huge_anon() returns true even if swapoff() left the non-present PMD swap entry in place.
Additionally, when self->zswap_enabled is true, no residency or swap check is performed at all before expect_rwp_fault() and verify_pattern() read self->mem. Would checking that none of the pages in the range remain swapped in pagemap (or checking residency directly) before reading self->mem catch a left-behind swap entry in both cases?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=32