mm: PMD-level swap entries for anonymous THPs
Subsystems (23)
Usama Arif
reviewerv2 cover mm: PMD-level swap entries for anonymous THPs
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via TTU_SPLIT_HUGE_PMD before unmap.
This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and do_huge_pmd_swap_page() resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed. swap_map metadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.
This work was brought about after Hugh reported that one of the major blockers for having lazy page table deposit is the lack of PMD swap entries [1]. However, this series has benefits of its own:
- The huge mapping is restored on swap-in. Today even when the folio is still in swap cache as a single 2 MB folio, the swap-in path installs 512 PTE mappings -- the PMD mapping is gone, the freshly-materialised PTE table sticks around, and only khugepaged can later collapse the range back into a THP. do_huge_pmd_swap_page() reinstalls the PMD mapping directly in one fault, no khugepaged involvement.
- Memory saved per swapped-out THP *once lazy page table deposit is merged* [2]. With lazy page table deposit [2], splitting a PMD into 512 PTE swap entries forces allocation of a 4 KB PTE table page. The new path leaves the pgtable hierarchy at PMD level and avoids that allocation entirely.
This will save memory when swapping, which is likely when there is memory pressure and exactly when allocations are most likely to fail.
- Walkers (zap, mprotect, smaps, pagemap, soft-dirty, uffd-wp) visit one PMD entry instead of 512 PTEs, reducing traversal time and lock-hold windows.
The swap entry value is identical to 512 PTE swap entries (same type, same starting offset), so swap_map refcounting is unchanged. Only the page-table representation differs; the swap slot allocator, swap I/O, and swap cache are untouched. The new path falls back to the existing PTE-split path whenever a PMD-order resource is
unavailable: zswap enabled, non-contiguous swap allocation (THP_SWPOUT_FALLBACK), PMD-order folio allocation failure on swap-in or fork, racing folio split, or rmap-driven split on a swapcache folio. Walkers that previously assumed every non-present PMD encodes a PFN (migration / device_private) are taught to recognise PMD swap entries.
Patch breakdown:
The series is ordered to preserve git bisectability: every consumer of a PMD swap entry (split, fork, swapoff, walkers, MADV_WILLNEED, UFFDIO_MOVE, swap-in fault) lands before the producer. The swap-out path that actually installs PMD swap entries is the very last functional patch (15), so no intermediate commit can leave the kernel handling a PMD swap entry it does not yet understand.
The first 6 patches are preparatory patches. Some of them (like softleaf_to_pmd() change in patch 1) are not exactly needed but its done to hopefully improve code quality and so that the PMD swap entry changes look well integrated with the rest of mm.
Prep patches:
1. mm: add softleaf_to_pmd() and convert existing callers PMD counterpart to softleaf_to_pte(); needed to construct a PMD from a swap entry in later patches.
2. mm: extract mm_prepare_for_swap_entries() helper
Hoists the "register mm with swapoff" double-checked-locking pattern out of try_to_unmap_one() / copy_nonpresent_pte() so the PMD swap-out and PMD fork paths can reuse it without a third open-coded copy.
3. fs/proc: use softleaf_has_pfn() in pagemap PMD walker pagemap_pmd_range_thp() today calls softleaf_to_page() unconditionally; a PMD swap entry has no PFN and would crash it.
4. mm/huge_memory: move softleaf_to_folio() inside migration branch change_non_present_huge_pmd() today calls softleaf_to_folio() before branching on entry type, so a PMD swap entry would produce a bogus folio pointer that the migration-only code below would then dereference.
5. mm/migrate_device: move softleaf_to_folio() inside device-private branch
migrate_vma_collect_pmd() has the same pre-check ordering issue as patch 4 in the migrate-device PMD walker; move the folio lookup inside the device-private check.
6. mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_SUPPORTS_PMD_SOFTLEAF The config gates the entire PMD softleaf machinery (migration, device-private, and now swap), not just migration; rename to match. Pure rename, no behavioural change.
Core patches:
7. PMD swap entry detection (pmd_is_swap_entry, softleaf_is_valid_pmd_entry) and per-arch pmd_swp_*exclusive helpers (x86/arm64/s390/riscv/loongarch/powerpc).
8. __split_huge_pmd_locked() learns to split a PMD swap entry into 512 PTE swap entries, used as the fallback when a PMD-order resource is unavailable.
9. Fork: copy_huge_non_present_pmd() duplicates the PMD swap entry in one swap_dup_entries_direct(HPAGE_PMD_NR) call, with GFP_KERNEL retry on per-cluster table-allocation failure mirroring copy_pte_range().
10. Swapoff: unuse_pmd() reads the whole 2 MB folio and reinstalls the PMD; falls back to PTE-split + unuse_pte_range() on error.
11. Walker updates: zap_huge_pmd, change_huge_pmd, change_non_present_huge_pmd, move_soft_dirty_pmd, clear_soft_dirty_pmd, make_uffd_wp_pmd, smaps_pmd_entry, queue_folios_pmd (mempolicy), check_pmd_state (khugepaged), mincore_pte_range, and the madvise_cold_or_pageout_pte_range / madvise_free_huge_pmd VM_BUG_ON extensions.
12. MADV_WILLNEED: swapin_walk_pmd_entry() reads the whole 2 MB folio in at PMD order via swapin_sync(BIT(HPAGE_PMD_ORDER)), so the subsequent fault hits do_huge_pmd_swap_page() and restores the THP mapping. A naive order-0 read-ahead would force the fault to split.
13. UFFDIO_MOVE: move_pages_huge_pmd() learns to move a PMD swap entry whole via a new move_swap_pmd() helper modeled on move_swap_pte().
14. Swap-in: do_huge_pmd_swap_page() resolves a PMD swap fault in one shot. Handles racing splits, SWP_STABLE_WRITES read-only mapping, immediate COW for write faults; falls back to PTE-split on any PMD-order resource shortfall.
15. Swap-out: shrink_folio_list() drops TTU_SPLIT_HUGE_PMD for PMD-mappable swapcache folios (when zswap is disabled), and try_to_unmap_one() installs one PMD swap entry via set_pmd_swap_entry() instead of splitting.
Testing:
16. selftests/mm: 13 tests covering swap-out/in, fork, fork+COW, repeated cycles, write fault, munmap, mprotect, mremap, pagemap, MADV_FREE, MADV_WILLNEED, UFFDIO_MOVE, swapoff.
Making PMD swap entries work with zswap is another project on its own and should be in a separate follow up series.
The patches are on top of mm-new from 31 May
(415489ef1cdfe586b4992662bee65286d50232e6).
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Patch 1: convert two additional softleaf_to_pmd() callers that landed in mm-unstable since v1 (mm/debug_vm_pgtable.c, mm/migrate_device.c) (Dev)
- Patch 2: rename helper ensure_on_mmlist() to mm_prepare_for_swap_entries() to better describe its purpose
(David)
- Patch 3: drop VM_WARN_ON_ONCE(!pmd_is_migration_entry) as
Dev posted it as a separate patch.
- Patch 5 (new): move softleaf_to_folio() inside the device-private branch in migrate_vma_collect_pmd(); same class of fix as patch 4 but for the migrate-device PMD walker.
- Patch 6 (new): rename CONFIG_ARCH_ENABLE_THP_MIGRATION to CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF so the gate that now drives swap-entry support too is named for what it actually controls
(PMD softleaf entries), not just migration. (Dev)
- Patch 7: add the missing pmd_swp_exclusive / mkexclusive / clear_exclusive helpers for powerpc.
- Patches 10 and 14: use upstream swapin_sync() (bundles swap_cache_alloc_folio + swap_read_folio + the -EEXIST race
retry) instead of the bespoke swapin_alloc_pmd_folio() helper
from v1; do_swap_page and shmem_swapin_folio use the same
helper (Kairui)
- Patch 10: construct a stack vm_fault for the swapoff swap-in so
the allocator can resolve a mempolicy, mirroring how the PTE
swapoff path (unuse_pte_range) already does it.
- Patch 11: extend coverage to check_pmd_state() in khugepaged so a swapped-out PMD-mapped THP is treated as SCAN_PMD_MAPPED (matches the existing migration-entry handling). Route the pmd_trans_huge_lock() branch of mincore_pte_range() through mincore_swap() so a swapped-out PMD-mapped THP isn't reported as resident.
- Patch 12 (new): handle PMD swap entries in MADV_WILLNEED via swapin_sync(BIT(HPAGE_PMD_ORDER)); a naive order-0 read-ahead
would force the subsequent fault to split.
- Patch 13: refuse UFFDIO_MOVE with -EBUSY if the swap-cache folio
was split between swap-out and the move, matching
move_pages_pte()'s rejection of large folios; otherwise only one
of the 512 anon-rmaps would be re-anchored to dst_vma.
- Patch 16: alloc_fill_swap_thp() now uses the existing mmap_pmd_aligned() helper so tests don't flake/skip based on VA placement; new MADV_WILLNEED test that watches the PMD-order
mTHP swpin counter; swapoff test restructured to use the kselftest_harness ASSERT cleanup blocks (no double swapoff, no verify-after-munmap).
- Collected Acks and Reviews.
Usama Arif (16):
mm: add softleaf_to_pmd() and convert existing callers
mm: extract mm_prepare_for_swap_entries() helper fs/proc: use softleaf_has_pfn() in pagemap PMD walker mm/huge_memory: move softleaf_to_folio() inside migration branch mm/migrate_device: move softleaf_to_folio() inside device-private
branch
mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_SUPPORTS_PMD_SOFTLEAF
mm: add PMD swap entry detection support
mm: add PMD swap entry splitting support
mm: handle PMD swap entries in fork path
mm: swap in PMD swap entries as whole THPs during swapoff
mm: handle PMD swap entries in non-present PMD walkers
mm: handle PMD swap entries in MADV_WILLNEED
mm: handle PMD swap entries in UFFDIO_MOVE
mm: handle PMD swap entry faults on swap-in
mm: install PMD swap entries on swap-out
selftests/mm: add PMD swap entry tests
arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/pgtable.h | 8 +- arch/loongarch/Kconfig | 2 +- arch/loongarch/include/asm/pgtable.h | 17 + arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +- arch/powerpc/platforms/Kconfig.cputype | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/pgtable.h | 23 +- arch/s390/Kconfig | 2 +- arch/s390/include/asm/pgtable.h | 17 +- arch/x86/Kconfig | 2 +- arch/x86/include/asm/pgtable.h | 17 +- fs/proc/task_mmu.c | 46 +- include/linux/huge_mm.h | 13 +- include/linux/leafops.h | 52 +- include/linux/pgtable.h | 2 +- include/linux/swap.h | 4 +- include/linux/swapops.h | 6 +- include/linux/vm_event_item.h | 1 + mm/Kconfig | 2 +- mm/debug_vm_pgtable.c | 12 +- mm/hmm.c | 7 +- mm/huge_memory.c | 543 ++++++++++++++- mm/internal.h | 49 ++ mm/khugepaged.c | 6 + mm/madvise.c | 45 +- mm/memory.c | 51 +- mm/mempolicy.c | 2 + mm/migrate.c | 4 +- mm/migrate_device.c | 19 +- mm/mincore.c | 14 +- mm/rmap.c | 29 +- mm/swapfile.c | 152 ++++- mm/vmscan.c | 14 +- mm/vmstat.c | 1 + tools/testing/selftests/mm/Makefile | 1 + tools/testing/selftests/mm/pmd_swap.c | 672 +++++++++++++++++++ 37 files changed, 1702 insertions(+), 156 deletions(-) create mode 100644 tools/testing/selftests/mm/pmd_swap.c
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 1/16 mm: add softleaf_to_pmd() and convert existing callers
Add softleaf_to_pmd() as the PMD counterpart to softleaf_to_pte(), completing the symmetry of the softleaf abstraction for page table leaf entries.
The upcoming PMD swap entry support needs to construct PMD entries from swap entries. Converting existing swp_entry_to_pmd() callers to softleaf_to_pmd() in a prep patch keeps the feature patches focused on new functionality rather than mixing refactoring with new code.
--- include/linux/leafops.h | 20 ++++++++++++++++++++ mm/debug_vm_pgtable.c | 4 ++-- mm/huge_memory.c | 12 ++++++------ mm/migrate_device.c | 2 +- 4 files changed, 29 insertions(+), 9 deletions(-)
include/linux/leafops.h +20 -0
@@ -108,6 +108,21 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) return swp_entry(__swp_type(arch_entry), __swp_offset(arch_entry)); } +/** +* softleaf_to_pmd() - Obtain a PMD entry from a leaf entry. +* @entry: Leaf entry. +* +* This generates an architecture-specific PMD entry that can be utilised to +* encode the metadata the leaf entry encodes. +* +* Returns: Architecture-specific PMD entry encoding leaf entry. +*/ +static inline pmd_t softleaf_to_pmd (softleaf_t entry) +{ +/* Temporary until swp_entry_t eliminated. */ +return swp_entry_to_pmd (entry); +} +#else static inline softleaf_t softleaf_from_pmd (pmd_t pmd)
@@ -115,6 +130,11 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) return softleaf_mk_none(); } +static inline pmd_t softleaf_to_pmd (softleaf_t entry) +{ +return __pmd (0 ); +} +#endif /**
mm/debug_vm_pgtable.c +2 -2
@@ -758,7 +758,7 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args)return ;pr_debug ("Validating PMD swap soft dirty\n" ); - pmd =swp_entry_to_pmd (args->leaf_entry); + pmd =softleaf_to_pmd (args->leaf_entry);WARN_ON (!pmd_is_huge (pmd));WARN_ON (!pmd_is_valid_softleaf (pmd));
@@ -829,7 +829,7 @@ static void __init pmd_softleaf_tests(struct pgtable_debug_args *args)return ;pr_debug ("Validating PMD swap\n" ); - pmd1 =swp_entry_to_pmd (args->leaf_entry); + pmd1 =softleaf_to_pmd (args->leaf_entry);WARN_ON (!pmd_is_huge (pmd1));WARN_ON (!pmd_is_valid_softleaf (pmd1));
mm/huge_memory.c +6 -6
@@ -1820,7 +1820,7 @@ static void copy_huge_non_present_pmd(if (softleaf_is_migration_write (entry) ||softleaf_is_migration_read_exclusive (entry)) { entry =make_readable_migration_entry (swp_offset (entry)); - pmd =swp_entry_to_pmd (entry); + pmd =softleaf_to_pmd (entry);if (pmd_swp_soft_dirty (*src_pmd)) pmd =pmd_swp_mksoft_dirty (pmd);if (pmd_swp_uffd_wp (*src_pmd))
@@ -1833,7 +1833,7 @@ static void copy_huge_non_present_pmd(*/ if (softleaf_is_device_private_write (entry)) { entry =make_readable_device_private_entry (swp_offset (entry)); - pmd =swp_entry_to_pmd (entry); + pmd =softleaf_to_pmd (entry);if (pmd_swp_soft_dirty (*src_pmd)) pmd =pmd_swp_mksoft_dirty (pmd);
@@ -2571,12 +2571,12 @@ static void change_non_present_huge_pmd(struct mm_struct *mm, entry =make_readable_exclusive_migration_entry (swp_offset (entry)); else entry =make_readable_migration_entry (swp_offset (entry)); - newpmd =swp_entry_to_pmd (entry); + newpmd =softleaf_to_pmd (entry);if (pmd_swp_soft_dirty (*pmd)) newpmd =pmd_swp_mksoft_dirty (newpmd); }else if (softleaf_is_device_private_write (entry)) { entry =make_readable_device_private_entry (swp_offset (entry)); - newpmd =swp_entry_to_pmd (entry); + newpmd =softleaf_to_pmd (entry);if (pmd_swp_uffd_wp (*pmd)) newpmd =pmd_swp_mkuffd_wp (newpmd); }else {
@@ -4901,7 +4901,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, entry =make_migration_entry_young (entry);if (pmd_dirty (pmdval)) entry =make_migration_entry_dirty (entry); - pmdswp =swp_entry_to_pmd (entry); + pmdswp =softleaf_to_pmd (entry);if (pmd_soft_dirty (pmdval)) pmdswp =pmd_swp_mksoft_dirty (pmdswp);if (pmd_uffd_wp (pmdval))
@@ -4952,7 +4952,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)else entry =make_readable_device_private_entry (page_to_pfn (new)); - pmde =swp_entry_to_pmd (entry); + pmde =softleaf_to_pmd (entry);if (pmd_swp_soft_dirty (*pvmw->pmd)) pmde =pmd_swp_mksoft_dirty (pmde);
mm/migrate_device.c +1 -1
@@ -835,7 +835,7 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate,else swp_entry =make_readable_device_private_entry (page_to_pfn (page)); - entry =swp_entry_to_pmd (swp_entry); + entry =softleaf_to_pmd (swp_entry); }else {if (folio_is_zone_device (folio) && !folio_is_device_coherent (folio)) {
Signature · 1 line
2.52.0
Re: [v2 01/16] mm: add softleaf_to_pmd() and convert existing callers
Quoted 10 lines · Usama Arif
Add
softleaf_to_pmd()as the PMD counterpart tosoftleaf_to_pte(), completing the symmetry of the softleaf abstraction for page table leaf entries.The upcoming PMD swap entry support needs to construct PMD entries from swap entries. Converting existing
swp_entry_to_pmd()callers tosoftleaf_to_pmd()in a prep patch keeps the feature patches focused on new functionality rather than mixing refactoring with new code.
Quoted 9 lines
Signed-off-by Usama Arif <[email protected]>--- include/linux/leafops.h | 20 ++++++++++++++++++++ mm/debug_vm_pgtable.c | 4 ++-- mm/huge_memory.c | 12 ++++++------ mm/migrate_device.c | 2 +- 4 files changed, 29 insertions(+), 9 deletions(-)
LGTM.
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 2/16 mm: extract mm_prepare_for_swap_entries() helper
In reply to Usama ArifWhen a swap entry is installed in a page table, the mm must be added to init_mm.mmlist so that swapoff can find and unuse its swap entries. This double-checked locking pattern is currently open-coded in try_to_unmap_one() and copy_nonpresent_pte().
Move it into mm_prepare_for_swap_entries() in mm/internal.h and convert both callers so it can be reused by upcoming PMD-level swap entry code paths that also need to register the mm with swapoff.
copy_nonpresent_pte() previously inserted into &src_mm->mmlist rather than &init_mm.mmlist, but the insertion point is irrelevant, mmlist is a circular list and swapoff walks it entirely from init_mm.mmlist, so only membership matters, not position.
--- mm/internal.h | 13 +++++++++++++ mm/memory.c | 9 +-------- mm/rmap.c | 7 +------ 3 files changed, 15 insertions(+), 14 deletions(-)
mm/internal.h +13 -0
@@ -1951,4 +1951,17 @@ static inline int get_sysctl_max_map_count(void)bool may_expand_vm (struct mm_struct *mm,const vma_flags_t *vma_flags,unsigned long npages); +/* +* Ensure @mm is on the init_mm.mmlist so swapoff can find it. +*/ +static inline void mm_prepare_for_swap_entries (struct mm_struct *mm) +{ +if (list_empty (&mm->mmlist)) { +spin_lock (&mmlist_lock); +if (list_empty (&mm->mmlist)) +list_add (&mm->mmlist, &init_mm.mmlist); +spin_unlock (&mmlist_lock); + } +} +#endif /* __MM_INTERNAL_H */
mm/memory.c +1 -8
@@ -953,14 +953,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,if (swap_dup_entry_direct (entry) <0 )return -EIO ; -/* make sure dst_mm is on swapoff's mmlist. */ -if (unlikely (list_empty (&dst_mm->mmlist))) { -spin_lock (&mmlist_lock); -if (list_empty (&dst_mm->mmlist)) -list_add (&dst_mm->mmlist, - &src_mm->mmlist); -spin_unlock (&mmlist_lock); - } +mm_prepare_for_swap_entries (dst_mm);/* Mark the swap entry as shared. */ if (pte_swp_exclusive (orig_pte)) { pte =pte_swp_clear_exclusive (orig_pte);
mm/rmap.c +1 -6
@@ -2304,12 +2304,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,set_pte_at (mm, address, pvmw.pte, pteval); goto walk_abort; } -if (list_empty (&mm->mmlist)) { -spin_lock (&mmlist_lock); -if (list_empty (&mm->mmlist)) -list_add (&mm->mmlist, &init_mm.mmlist); -spin_unlock (&mmlist_lock); - } +mm_prepare_for_swap_entries (mm);dec_mm_counter (mm,MM_ANONPAGES );inc_mm_counter (mm,MM_SWAPENTS ); swp_pte =swp_entry_to_pte (entry);
Signature · 1 line
2.52.0
Re: [v2 02/16] mm: extract mm_prepare_for_swap_entries() helper
Quoted 14 lines · Usama Arif
When a swap entry is installed in a page table, the mm must be added to init_mm.mmlist so that swapoff can find and unuse its swap entries. This double-checked locking pattern is currently open-coded in
try_to_unmap_one()andcopy_nonpresent_pte().Move it into
mm_prepare_for_swap_entries()inmm/internal.hand convert both callers so it can be reused by upcoming PMD-level swap entry code paths that also need to register the mm with swapoff.
copy_nonpresent_pte()previously inserted into&src_mm->mmlistrather than &init_mm.mmlist, but the insertion point is irrelevant, mmlist is a circular list and swapoff walks it entirely from init_mm.mmlist, so only membership matters, not position.
Quoted 8 lines
Reviewed-by Dev Jain <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/internal.h | 13 +++++++++++++ mm/memory.c | 9 +-------- mm/rmap.c | 7 +------ 3 files changed, 15 insertions(+), 14 deletions(-)
LGTM.
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 3/16 fs/proc: use softleaf_has_pfn() in pagemap PMD walker
In reply to Usama Arifpagemap_pmd_range_thp() assumes that every non-present PMD is a migration entry and unconditionally calls softleaf_to_page(). This will crash on any non-present PMD type that does not encode a PFN, such as the upcoming PMD-level swap entries.
Guard the page lookup with softleaf_has_pfn(), matching how pte_to_pagemap_entry() already handles non-present PTEs.
--- fs/proc/task_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +2 -1
@@ -2129,7 +2129,8 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr, flags |=PM_SOFT_DIRTY ;if (pmd_swp_uffd_wp (pmd)) flags |=PM_UFFD_WP ; - page =softleaf_to_page (entry); +if (softleaf_has_pfn (entry)) + page =softleaf_to_page (entry); }if (page) {
Signature · 1 line
2.52.0
Re: [v2 03/16] fs/proc: use softleaf_has_pfn() in pagemap PMD walker
Quoted 5 lines · Usama Arif
pagemap_pmd_range_thp()assumes that every non-present PMD is a migration entry and unconditionally callssoftleaf_to_page(). This will crash on any non-present PMD type that does not encode a PFN, such as the upcoming PMD-level swap entries.
Quoted 9 lines
Guard the page lookup with
softleaf_has_pfn(), matching howpte_to_pagemap_entry()already handles non-present PTEs.Signed-off-by Usama Arif <[email protected]>--- fs/proc/task_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
LGTM. I wonder if we want to change softleaf_to_page() and softleaf_to_page() to return NULL if softleaf_has_pfn() returns false. That might reduce such errors.
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 4/16 mm/huge_memory: move softleaf_to_folio() inside migration branch
In reply to Usama Arifchange_non_present_huge_pmd() calls softleaf_to_folio() unconditionally at the top of the function. softleaf_to_folio() extracts a PFN from the entry and converts it to a folio pointer, which is only meaningful for migration and device_private entries that encode a real PFN.
A swap entry encodes a swap offset instead, so softleaf_to_folio() would produce a bogus pointer and crash on mprotect() when a PMD swap entry is present.
Move the call into the migration_write branch where the folio is actually used, so the function is safe for any non-present PMD type.
--- mm/huge_memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/huge_memory.c +2 -1
@@ -2558,11 +2558,12 @@ static void change_non_present_huge_pmd(struct mm_struct *mm, bool uffd_wp_resolve) {softleaf_t entry =softleaf_from_pmd (*pmd); -const struct folio *folio =softleaf_to_folio (entry);pmd_t newpmd;VM_WARN_ON (!pmd_is_valid_softleaf (*pmd));if (softleaf_is_migration_write (entry)) { +const struct folio *folio =softleaf_to_folio (entry); +/* * A protection check is difficult so * just be safe and disable write
Signature · 1 line
2.52.0
Re: [v2 04/16] mm/huge_memory: move softleaf_to_folio() inside migration branch
Quoted 12 lines · Usama Arif
change_non_present_huge_pmd()callssoftleaf_to_folio()unconditionally at the top of the function.softleaf_to_folio()extracts a PFN from the entry and converts it to a folio pointer, which is only meaningful for migration anddevice_privateentries that encode a real PFN.A swap entry encodes a swap offset instead, so
softleaf_to_folio()would produce a bogus pointer and crash onmprotect()when a PMD swap entry is present.Move the call into the
migration_writebranch where the folio is actually used, so the function is safe for any non-present PMD type.
Quoted 7 lines
Reviewed-by Dev Jain <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/huge_memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
LGTM.
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 5/16 mm/migrate_device: move softleaf_to_folio() inside device-private branch
In reply to Usama Arifmigrate_vma_collect_pmd() calls softleaf_to_folio() on a non-present PMD before checking the entry's type. softleaf_to_folio() converts the entry's offset to a PFN, which is only meaningful for migration or device-private entries.
A PMD swap entry's offset is a swap offset, not a PFN, so the lookup would either return a bogus folio pointer or trip pfn_to_page validation on a debug kernel. In the non-device-private path the returned folio is then unused (the OR short-circuits to migrate_vma_collect_skip()), but the lookup itself is already unsafe.
Move the softleaf_to_folio() call inside the device-private branch where the folio is actually needed, mirroring the equivalent change_non_present_huge_pmd() fix.
--- mm/migrate_device.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
mm/migrate_device.c +7 -4
@@ -166,11 +166,14 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, }else if (!pmd_present (*pmdp)) {const softleaf_t entry =softleaf_from_pmd (*pmdp); - folio =softleaf_to_folio (entry); -if (!softleaf_is_device_private (entry) || - !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE ) || - (folio->pgmap->owner != migrate->pgmap_owner)) { + !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE )) { +spin_unlock (ptl); +return migrate_vma_collect_skip (start, end, walk); + } + + folio =softleaf_to_folio (entry); +if (folio->pgmap->owner != migrate->pgmap_owner) {spin_unlock (ptl);return migrate_vma_collect_skip (start, end, walk); }
Signature · 1 line
2.52.0
Re: [v2 05/16] mm/migrate_device: move softleaf_to_folio() inside device-private branch
Quoted 46 lines · Usama Arif
migrate_vma_collect_pmd()callssoftleaf_to_folio()on a non-present PMD before checking the entry's type.softleaf_to_folio()converts the entry's offset to a PFN, which is only meaningful for migration or device-private entries.A PMD swap entry's offset is a swap offset, not a PFN, so the lookup would either return a bogus folio pointer or trip
pfn_to_pagevalidation on a debug kernel. In the non-device-private path the returned folio is then unused (the OR short-circuits tomigrate_vma_collect_skip()), but the lookup itself is already unsafe.Move the
softleaf_to_folio()call inside the device-private branch where the folio is actually needed, mirroring the equivalentchange_non_present_huge_pmd()fix.Signed-off-by Usama Arif <[email protected]>--- mm/migrate_device.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)mm/migrate_device.c +7
-4@@ -166,11 +166,14 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, }else if (!pmd_present (*pmdp)) {const softleaf_t entry =softleaf_from_pmd (*pmdp); - folio =softleaf_to_folio (entry); -if (!softleaf_is_device_private (entry) || - !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE ) || - (folio->pgmap->owner != migrate->pgmap_owner)) { + !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE )) { +spin_unlock (ptl); +return migrate_vma_collect_skip (start, end, walk); + } + + folio =softleaf_to_folio (entry); +if (folio->pgmap->owner != migrate->pgmap_owner) {spin_unlock (ptl);return migrate_vma_collect_skip (start, end, walk); }Signature · 1 line
2.52.0
Is the patch below a simpler alternative?
mm/migrate_device.c +7 -4
@@ -166,11 +166,14 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, }else if (!pmd_present (*pmdp)) {const softleaf_t entry =softleaf_from_pmd (*pmdp); - folio =softleaf_to_folio (entry); + folio =NULL ; -if (!softleaf_is_device_private (entry) || - !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE ) || - (folio->pgmap->owner != migrate->pgmap_owner)) { +if (softleaf_is_device_private (entry)) + folio =softleaf_to_folio (entry); + +if (!folio || + !(migrate->flags &MIGRATE_VMA_SELECT_DEVICE_PRIVATE ) || + (folio->pgmap->owner != migrate->pgmap_owner)) {spin_unlock (ptl);return migrate_vma_collect_skip (start, end, walk); }
I have no strong opinion here. No matter what you choose, feel free to add:
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 6/16 mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_SUPPORTS_PMD_SOFTLEAF
In reply to Usama ArifCONFIG_ARCH_ENABLE_THP_MIGRATION started life gating just PMD-level migration entries, but has grown to gate the entire PMD-level softleaf
machinery: migration entries, device-private entries, and soon swap entries.
Rename CONFIG_ARCH_ENABLE_THP_MIGRATION to CONFIG_ARCH_SUPPORTS_PMD _SOFTLEAF to make this clear. This is a pure rename: the set of selecting architectures (x86, arm64, s390, riscv, loongarch, and powerpc on PPC_BOOK3S_64) and the gating semantics are unchanged.
No functional change intended.
--- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/loongarch/Kconfig | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 2 +- arch/powerpc/platforms/Kconfig.cputype | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/pgtable.h | 8 ++++---- arch/s390/Kconfig | 2 +- arch/s390/include/asm/pgtable.h | 2 +- arch/x86/Kconfig | 2 +- arch/x86/include/asm/pgtable.h | 2 +- include/linux/huge_mm.h | 2 +- include/linux/leafops.h | 8 ++++---- include/linux/pgtable.h | 2 +- include/linux/swapops.h | 6 +++--- mm/Kconfig | 2 +- mm/debug_vm_pgtable.c | 8 ++++---- mm/hmm.c | 4 ++-- mm/huge_memory.c | 2 +- mm/migrate.c | 4 ++-- mm/migrate_device.c | 6 +++--- mm/rmap.c | 2 +- 22 files changed, 38 insertions(+), 38 deletions(-)
arch/arm64/Kconfig +1 -1
@@ -17,7 +17,7 @@ config ARM64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 - select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CC_PLATFORM select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
arch/arm64/include/asm/pgtable.h +2 -2
@@ -1534,10 +1534,10 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#define __pte_to_swp_entry (pte) ((swp_entry_t) { pte_val(pte) })#define __swp_entry_to_pte (swp) ((pte_t) { (swp).val }) -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF #define __pmd_to_swp_entry (pmd) ((swp_entry_t) { pmd_val(pmd) })#define __swp_entry_to_pmd (swp) __pmd((swp).val) -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ /* * Ensure that there are not more swap files than can be encoded in the kernel
arch/loongarch/Kconfig +1 -1
@@ -12,7 +12,7 @@ config LOONGARCH select ARCH_NEEDS_DEFER_KASAN select ARCH_DISABLE_KASAN_INLINE select ARCH_ENABLE_MEMORY_HOTPLUG - select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if TRANSPARENT_HUGEPAGE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CURRENT_STACK_POINTER
arch/powerpc/include/asm/book3s/64/pgtable.h +1 -1
@@ -1060,7 +1060,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#define pmd_mksoft_dirty (pmd) pte_pmd(pte_mksoft_dirty(pmd_pte(pmd)))#define pmd_clear_soft_dirty (pmd) pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd))) -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF #define pmd_swp_mksoft_dirty (pmd) pte_pmd(pte_swp_mksoft_dirty(pmd_pte(pmd)))#define pmd_swp_soft_dirty (pmd) pte_swp_soft_dirty(pmd_pte(pmd))#define pmd_swp_clear_soft_dirty (pmd) pte_pmd(pte_swp_clear_soft_dirty(pmd_pte(pmd)))
arch/powerpc/platforms/Kconfig.cputype +1 -1
@@ -112,7 +112,7 @@ config PPC_THP depends on PPC_RADIX_MMU || (PPC_64S_HASH_MMU && PAGE_SIZE_64KB) select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD - select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if TRANSPARENT_HUGEPAGE choice prompt "CPU selection"
arch/riscv/Kconfig +1 -1
@@ -22,7 +22,7 @@ config RISCV select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 - select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU
arch/riscv/include/asm/pgtable.h +4 -4
@@ -936,7 +936,7 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)return pte_pmd (pte_clear_soft_dirty (pmd_pte (pmd))); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF static inline bool pmd_swp_soft_dirty (pmd_t pmd) {return pte_swp_soft_dirty (pmd_pte (pmd));
@@ -951,7 +951,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) {return pte_pmd (pte_swp_clear_soft_dirty (pmd_pte (pmd))); } -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
@@ -1198,10 +1198,10 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return __pte (pte_val (pte) & ~_PAGE_SWP_EXCLUSIVE); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF #define __pmd_to_swp_entry (pmd) ((swp_entry_t) { pmd_val(pmd) })#define __swp_entry_to_pmd (swp) __pmd((swp).val) -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ /* * In the RV64 Linux scheme, we give the user half of the virtual-address space
arch/s390/Kconfig +1 -1
@@ -85,7 +85,7 @@ config S390 select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 - select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if TRANSPARENT_HUGEPAGE select ARCH_HAS_CC_CAN_LINK select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CURRENT_STACK_POINTER
arch/s390/include/asm/pgtable.h +1 -1
@@ -903,7 +903,7 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)return clear_pmd_bit (pmd,__pgprot (_SEGMENT_ENTRY_SOFT_DIRTY)); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF #define pmd_swp_soft_dirty (pmd) pmd_soft_dirty(pmd)#define pmd_swp_mksoft_dirty (pmd) pmd_mksoft_dirty(pmd)#define pmd_swp_clear_soft_dirty (pmd) pmd_clear_soft_dirty(pmd)
arch/x86/Kconfig +1 -1
@@ -70,7 +70,7 @@ config X86 select ARCH_ENABLE_HUGEPAGE_MIGRATION if X86_64 && HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if (PGTABLE_LEVELS > 2) && (X86_64 || X86_PAE) - select ARCH_ENABLE_THP_MIGRATION if X86_64 && TRANSPARENT_HUGEPAGE + select ARCH_SUPPORTS_PMD_SOFTLEAF if X86_64 && TRANSPARENT_HUGEPAGE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_CPU_ATTACK_VECTORS if CPU_MITIGATIONS select ARCH_HAS_CACHE_LINE_SIZE
arch/x86/include/asm/pgtable.h +1 -1
@@ -1533,7 +1533,7 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_SOFT_DIRTY); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF static inline pmd_t pmd_swp_mksoft_dirty (pmd_t pmd) {return pmd_set_flags (pmd, _PAGE_SWP_SOFT_DIRTY);
include/linux/huge_mm.h +1 -1
@@ -567,7 +567,7 @@ static inline struct folio *get_persistent_huge_zero_folio(void)static inline bool thp_migration_supported (void ) { -return IS_ENABLED (CONFIG_ARCH_ENABLE_THP_MIGRATION ); +return IS_ENABLED (CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF ); } void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
include/linux/leafops.h +4 -4
@@ -81,7 +81,7 @@ static inline pte_t softleaf_to_pte(softleaf_t entry)return swp_entry_to_pte (entry); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF /** * softleaf_from_pmd() - Obtain a leaf entry from a PMD entry. * @pmd: PMD entry.
@@ -587,7 +587,7 @@ static inline bool pte_is_uffd_marker(pte_t pte)return false; } -#if defined(CONFIG_ZONE_DEVICE ) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION ) +#if defined(CONFIG_ZONE_DEVICE ) && defined(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF )/** * pmd_is_device_private_entry() - Check if PMD contains a device private swap
@@ -606,14 +606,14 @@ static inline bool pmd_is_device_private_entry(pmd_t pmd) return softleaf_is_device_private(softleaf_from_pmd(pmd)); } -#else /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#else /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static inline bool pmd_is_device_private_entry (pmd_t pmd) {return false; } -#endif /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ /** * pmd_is_migration_entry() - Does this PMD entry encode a migration entry?
include/linux/pgtable.h +1 -1
@@ -1781,7 +1781,7 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot)#endif #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY -#ifndef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifndef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd) { return pmd;
include/linux/swapops.h +3 -3
@@ -321,7 +321,7 @@ static inline swp_entry_t make_guard_swp_entry(void)struct page_vma_mapped_walk ; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF extern int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page);
@@ -338,7 +338,7 @@ static inline pmd_t swp_entry_to_pmd(swp_entry_t entry) return __swp_entry_to_pmd(arch_entry); } -#else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#else /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static inline int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page) {
@@ -358,7 +358,7 @@ static inline pmd_t swp_entry_to_pmd(swp_entry_t entry)return __pmd (0 ); } -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ #endif /* CONFIG_MMU */ #endif /* _LINUX_SWAPOPS_H */
mm/Kconfig +1 -1
@@ -650,7 +650,7 @@ config DEVICE_MIGRATION config ARCH_ENABLE_HUGEPAGE_MIGRATION bool -config ARCH_ENABLE_THP_MIGRATION +config ARCH_SUPPORTS_PMD_SOFTLEAF bool config HUGETLB_PAGE_SIZE_VARIABLE
mm/debug_vm_pgtable.c +4 -4
@@ -751,7 +751,7 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args)pmd_t pmd;if (!pgtable_supports_soft_dirty () || - !IS_ENABLED (CONFIG_ARCH_ENABLE_THP_MIGRATION )) + !IS_ENABLED (CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF ))return ;if (!has_transparent_hugepage ())
@@ -819,7 +819,7 @@ static void __init pte_swap_tests(struct pgtable_debug_args *args)WARN_ON (memcmp (&pte1, &pte2,sizeof (pte1))); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF static void __init pmd_softleaf_tests (struct pgtable_debug_args *args) {swp_entry_t arch_entry;
@@ -837,9 +837,9 @@ static void __init pmd_softleaf_tests(struct pgtable_debug_args *args) pmd2 =__swp_entry_to_pmd (arch_entry); WARN_ON(memcmp(&pmd1, &pmd2, sizeof(pmd1))); } -#else /* !CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#else /* !CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static void __init pmd_softleaf_tests (struct pgtable_debug_args *args) { } -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static void __init swap_migration_tests (struct pgtable_debug_args *args) {
mm/hmm.c +2 -2
@@ -331,7 +331,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,return hmm_vma_fault (addr, end, required_fault, walk); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF static int hmm_vma_handle_absent_pmd (struct mm_walk *walk,unsigned long start,unsigned long end,unsigned long *hmm_pfns,pmd_t pmd)
@@ -391,7 +391,7 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,return -EFAULT ;return hmm_pfns_fill (start, end, range,HMM_PFN_ERROR ); } -#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#endif /* CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static int hmm_vma_walk_pmd(pmd_t *pmdp, unsigned long start,
mm/huge_memory.c +1 -1
@@ -4861,7 +4861,7 @@ static int __init split_huge_pages_debugfs(void)late_initcall (split_huge_pages_debugfs);#endif -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF int set_pmd_migration_entry (struct page_vma_mapped_walk *pvmw,struct page *page) {
mm/migrate.c +2 -2
@@ -362,7 +362,7 @@ static bool remove_migration_pte(struct folio *folio, idx =linear_page_index (vma, pvmw.address) - pvmw.pgoff; new =folio_page (folio, idx); -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF /* PMD-mapped THP migration entry */ if (!pvmw.pte) { VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) ||
@@ -545,7 +545,7 @@ void migration_entry_wait_huge(struct vm_area_struct *vma, unsigned long addr, p }#endif -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF void pmd_migration_entry_wait (struct mm_struct *mm,pmd_t *pmd) {spinlock_t *ptl;
mm/migrate_device.c +3 -3
@@ -771,7 +771,7 @@ int migrate_vma_setup(struct migrate_vma *args) }EXPORT_SYMBOL (migrate_vma_setup); -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF /** * migrate_vma_insert_huge_pmd_page: Insert a huge folio into @migrate->vma->vm_mm * at @addr. folio is already allocated as a part of the migration process with
@@ -926,7 +926,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate, migrate->src[i+idx] =migrate_pfn (pfn + i) | flags;return ret; } -#else /* !CONFIG_ARCH_ENABLE_THP_MIGRATION */ +#else /* !CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF */ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, unsigned long addr, struct page *page,
@@ -947,7 +947,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,static unsigned long migrate_vma_nr_pages (unsigned long *src) {unsigned long nr =1 ; -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF if (*src &MIGRATE_PFN_COMPOUND ) nr =HPAGE_PMD_NR ;#else
mm/rmap.c +1 -1
@@ -2472,7 +2472,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_restart (&pvmw);continue ; } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF pmdval =pmdp_get (pvmw.pmd);if (likely (pmd_present (pmdval))) pfn =pmd_pfn (pmdval);
Signature · 1 line
2.52.0
Re: [v2 06/16] mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_SUPPORTS_PMD_SOFTLEAF
Quoted 29 lines · Usama Arif
CONFIG_ARCH_ENABLE_THP_MIGRATIONstarted life gating just PMD-level migration entries, but has grown to gate the entire PMD-level softleaf
machinery: migration entries, device-private entries, and soon swap entries.Rename
CONFIG_ARCH_ENABLE_THP_MIGRATIONtoCONFIG_ARCH_SUPPORTS_PMD_SOFTLEAFto make this clear. This is a pure rename: the set of selecting architectures (x86, arm64, s390, riscv, loongarch, and powerpc onPPC_BOOK3S_64) and the gating semantics are unchanged.No functional change intended.
Signed-off-by Usama Arif <[email protected]>--- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/loongarch/Kconfig | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 2 +- arch/powerpc/platforms/Kconfig.cputype | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/pgtable.h | 8 ++++---- arch/s390/Kconfig | 2 +- arch/s390/include/asm/pgtable.h | 2 +- arch/x86/Kconfig | 2 +- arch/x86/include/asm/pgtable.h | 2 +- include/linux/huge_mm.h | 2 +- include/linux/leafops.h | 8 ++++---- include/linux/pgtable.h | 2 +- include/linux/swapops.h | 6 +++---
Quoted 9 lines
mm/Kconfig | 2 +- mm/debug_vm_pgtable.c | 8 ++++---- mm/hmm.c | 4 ++-- mm/huge_memory.c | 2 +- mm/migrate.c | 4 ++-- mm/migrate_device.c | 6 +++--- mm/rmap.c | 2 +- 22 files changed, 38 insertions(+), 38 deletions(-)
Makes sense to me.
--
Best Regards,
Yan, Zi
Usama Arif
reviewerPATCH v2 7/16 mm: add PMD swap entry detection support
In reply to Usama ArifCurrently when a PMD-mapped THP is swapped out, the PMD is always split into 512 PTE-level swap entries. To preserve huge page information across swap cycles, later patches will install a single PMD-level swap entry instead. This patch adds the infrastructure to detect those entries.
Teach the softleaf layer to recognise PMD swap entries:pmd_is_swap_entry() detects them and softleaf_is_valid_pmd_entry() accepts them as a valid non-present type. Clear the exclusive overlay bit in softleaf_from_pmd() before decoding, matching how soft_dirty and uffd_wp bits are already stripped.
Add pmd_swp_mkexclusive(), pmd_swp_exclusive(), and pmd_swp_clear_exclusive() helpers to each architecture that supports THP migration (x86, arm64, s390, riscv, loongarch, powerpc), mirroring the existing PTE swap exclusive helpers in each arch's pgtable.h.
--- arch/arm64/include/asm/pgtable.h | 4 ++++ arch/loongarch/include/asm/pgtable.h | 17 ++++++++++++++ arch/powerpc/include/asm/book3s/64/pgtable.h | 15 ++++++++++++ arch/riscv/include/asm/pgtable.h | 15 ++++++++++++ arch/s390/include/asm/pgtable.h | 15 ++++++++++++ arch/x86/include/asm/pgtable.h | 15 ++++++++++++ include/linux/leafops.h | 24 ++++++++++++++++---- 7 files changed, 100 insertions(+), 5 deletions(-)
arch/arm64/include/asm/pgtable.h +4 -0
@@ -601,6 +601,10 @@ static inline int pmd_protnone(pmd_t pmd)#define pmd_swp_clear_uffd_wp (pmd) \ pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)))#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#define pmd_swp_exclusive (pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive (pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive (pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd)))#define pmd_write (pmd) pte_write(pmd_pte(pmd))
arch/loongarch/include/asm/pgtable.h +17 -0
@@ -357,6 +357,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte; } +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +pmd_val (pmd) |= _PAGE_SWP_EXCLUSIVE; +return pmd; +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +pmd_val (pmd) &= ~_PAGE_SWP_EXCLUSIVE; +return pmd; +} +#define pte_none (pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))#define pte_present (pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))#define pte_no_exec (pte) (pte_val(pte) & _PAGE_NO_EXEC)
arch/powerpc/include/asm/book3s/64/pgtable.h +15 -0
@@ -699,6 +699,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return __pte_raw (pte_raw (pte) &cpu_to_be64 (~_PAGE_SWP_EXCLUSIVE)); } +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return __pmd_raw (pmd_raw (pmd) |cpu_to_be64 (_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_SWP_EXCLUSIVE)); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return __pmd_raw (pmd_raw (pmd) &cpu_to_be64 (~_PAGE_SWP_EXCLUSIVE)); +} +static inline bool check_pte_access (unsigned long access,unsigned long ptev) {/*
arch/riscv/include/asm/pgtable.h +15 -0
@@ -920,6 +920,21 @@ static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd) }#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pte_swp_exclusive (pmd_pte (pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_mkexclusive (pmd_pte (pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pte_pmd (pte_swp_clear_exclusive (pmd_pte (pmd))); +} +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline bool pmd_soft_dirty (pmd_t pmd) {
arch/s390/include/asm/pgtable.h +15 -0
@@ -870,6 +870,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return clear_pte_bit (pte,__pgprot (_PAGE_SWP_EXCLUSIVE)); } +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return set_pmd_bit (pmd,__pgprot (_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return clear_pmd_bit (pmd,__pgprot (_PAGE_SWP_EXCLUSIVE)); +} +static inline int pte_soft_dirty (pte_t pte) {return pte_val (pte) & _PAGE_SOFT_DIRTY;
arch/x86/include/asm/pgtable.h +15 -0
@@ -1517,6 +1517,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)return pte_clear_flags (pte, _PAGE_SWP_EXCLUSIVE); } +static inline pmd_t pmd_swp_mkexclusive (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline int pmd_swp_exclusive (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive (pmd_t pmd) +{ +return pmd_clear_flags (pmd, _PAGE_SWP_EXCLUSIVE); +} +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty (pte_t pte) {
include/linux/leafops.h +19 -5
@@ -102,6 +102,8 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) pmd =pmd_swp_clear_soft_dirty (pmd);if (pmd_swp_uffd_wp (pmd)) pmd =pmd_swp_clear_uffd_wp (pmd); +if (pmd_swp_exclusive (pmd)) + pmd =pmd_swp_clear_exclusive (pmd); arch_entry =__pmd_to_swp_entry (pmd);/* Temporary until swp_entry_t eliminated. */
@@ -634,18 +636,30 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)*/ static inline bool softleaf_is_valid_pmd_entry (softleaf_t entry) { -/* Only device private, migration entries valid for PMD. */ +/* Device private, migration, and swap entries valid for PMD. */ return softleaf_is_device_private (entry) || -softleaf_is_migration (entry); +softleaf_is_migration (entry) || +softleaf_is_swap (entry); +} + +/** +* pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? +* @pmd: PMD entry. +* +* Returns: true if the PMD encodes a swap entry, otherwise false. +*/ +static inline bool pmd_is_swap_entry (pmd_t pmd) +{ +return softleaf_is_swap (softleaf_from_pmd (pmd)); }/** * pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry? * @pmd: PMD entry. * -* PMD leaf entries are valid only if they are device private or migration -* entries. This function asserts that a PMD leaf entry is valid in this -* respect. +* PMD leaf entries are valid only if they are device private, migration, +* or swap entries. This function asserts that a PMD leaf entry is valid +* in this respect. * * Returns: true if the PMD entry is a valid leaf entry, otherwise false. */
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 8/16 mm: add PMD swap entry splitting support
In reply to Usama ArifAdd a swap branch in __split_huge_pmd_locked() that splits a PMD swap entry into 512 PTE swap entries. Unlike migration splits, no folio reference is needed because swap entries point to swap slots, not pages. Each PTE inherits the correct sub-slot offset and preserves soft_dirty, uffd_wp, and exclusive flags.
This branch is reached from the explicit __split_huge_pmd() callers that hit a non-present PMD: partial-range mprotect / munmap, the wp_huge_pmd() PMD-COW fallback, and the swap-in / swapoff fallbacks added in later patches when the cached folio is no longer PMD-sized. page_vma_mapped_walk() does not iterate PMD swap entries, so try_to_unmap_one() and try_to_migrate_one() do not reach this branch and freeze=true cannot occur in this branch today. page and folio are therefore left uninitialized in the swap branch; a VM_WARN_ON_ONCE(freeze) catches any future caller that breaks this invariant before the freeze path dereferences page_to_pfn(page + i) or put_page(page).
--- mm/huge_memory.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
mm/huge_memory.c +26 -1
@@ -3144,6 +3144,12 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,folio_add_anon_rmap_ptes (folio, page,HPAGE_PMD_NR , vma, haddr, rmap_flags); } + }else if (pmd_is_swap_entry (*pmd)) { +VM_WARN_ON_ONCE (freeze); + old_pmd = *pmd; + soft_dirty =pmd_swp_soft_dirty (old_pmd); + uffd_wp =pmd_swp_uffd_wp (old_pmd); + anon_exclusive =pmd_swp_exclusive (old_pmd); }else {/* * Up to this point the pmd is present and huge and userland has
@@ -3280,6 +3286,25 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,VM_WARN_ON (!pte_none (ptep_get (pte + i)));set_pte_at (mm, addr, pte + i, entry); } + }else if (pmd_is_swap_entry (old_pmd)) { +softleaf_t sl_entry =softleaf_from_pmd (old_pmd); +pte_t swp_pte; +swp_entry_t sub_entry; + +for (i =0 , addr = haddr; i <HPAGE_PMD_NR ; + i++, addr +=PAGE_SIZE ) { + sub_entry =swp_entry (swp_type (sl_entry), +swp_offset (sl_entry) + i); + swp_pte =swp_entry_to_pte (sub_entry); +if (soft_dirty) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (uffd_wp) + swp_pte =pte_swp_mkuffd_wp (swp_pte); +if (anon_exclusive) + swp_pte =pte_swp_mkexclusive (swp_pte); +VM_WARN_ON (!pte_none (ptep_get (pte + i))); +set_pte_at (mm, addr, pte + i, swp_pte); + } }else {pte_t entry;
@@ -3303,7 +3328,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, }pte_unmap (pte); -if (!pmd_is_migration_entry (*pmd)) +if (!pmd_is_migration_entry (*pmd) && !pmd_is_swap_entry (*pmd))folio_remove_rmap_pmd (folio, page, vma);if (freeze)put_page (page);
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 9/16 mm: handle PMD swap entries in fork path
In reply to Usama ArifTeach copy_huge_pmd()/copy_huge_non_present_pmd() about swap entries, mirroring copy_nonpresent_pte().
swap_dup_entry_direct() gains a nr parameter (and is renamed to swap_dup_entries_direct()) so it can duplicate a contiguous range of swap slots in one call, matching the existing swap_put_entries_direct(entry, nr) API. Existing callers pass 1.
copy_huge_non_present_pmd() "copies" PMD swap entries during fork instead of splitting, preserving the THP. This mirrors copy_nonpresent_pte() which duplicates the swap slot refcount, clears the exclusive bit on the source, and adds the destination mm to mmlist. If swap_dup_entries_direct() fails (GFP_ATOMIC table alloc), copy_huge_pmd() retries after swap_retry_table_alloc() with GFP_KERNEL, matching the PTE retry in copy_pte_range(). The PMD is stable across the retry because dup_mmap() holds write mmap_lock on both mm_structs.
--- include/linux/swap.h | 4 ++-- mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++----- mm/memory.c | 2 +- mm/swapfile.c | 7 +++--- 4 files changed, 53 insertions(+), 12 deletions(-)
include/linux/swap.h +2 -2
@@ -458,7 +458,7 @@ sector_t swap_folio_sector(struct folio *folio);* All entries must be allocated by folio_alloc_swap(). And they must have * a swap count > 1. See comments of folio_*_swap helpers for more info. */ -int swap_dup_entry_direct (swp_entry_t entry); +int swap_dup_entries_direct (swp_entry_t entry,int nr);void swap_put_entries_direct (swp_entry_t entry,int nr);/*
@@ -502,7 +502,7 @@ static inline void free_swap_cache(struct folio *folio) { } -static inline int swap_dup_entry_direct (swp_entry_t ent) +static inline int swap_dup_entries_direct (swp_entry_t ent,int nr) {return 0 ; }
mm/huge_memory.c +46 -6
83 lines not on this page · raw message
mm/memory.c +1 -1
@@ -950,7 +950,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,struct page *page;if (likely (softleaf_is_swap (entry))) { -if (swap_dup_entry_direct (entry) <0 ) +if (swap_dup_entries_direct (entry,1 ) <0 )return -EIO ;mm_prepare_for_swap_entries (dst_mm);
mm/swapfile.c +4 -3
@@ -3899,8 +3899,9 @@ void si_swapinfo(struct sysinfo *val) }/* -* swap_dup_entry_direct() - Increase reference count of a swap entry by one. +* swap_dup_entries_direct() - Increase reference count of swap entries by one. * @entry: first swap entry from which we want to increase the refcount. +* @nr: number of contiguous swap entries to duplicate. * * Returns 0 for success, or -ENOMEM if the extend table is required * but could not be atomically allocated. Returns -EINVAL if the swap
@@ -3912,7 +3913,7 @@ void si_swapinfo(struct sysinfo *val)* Also the swap entry must have a count >= 1. Otherwise folio_dup_swap should * be used. */ -int swap_dup_entry_direct (swp_entry_t entry) +int swap_dup_entries_direct (swp_entry_t entry,int nr) {struct swap_info_struct *si;
@@ -3929,7 +3930,7 @@ int swap_dup_entry_direct(swp_entry_t entry)*/ VM_WARN_ON_ONCE (!swap_entry_swapped (si, entry)); -return swap_dup_entries_cluster (si,swp_offset (entry),1 ); +return swap_dup_entries_cluster (si,swp_offset (entry), nr); }#if defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP )
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 10/16 mm: swap in PMD swap entries as whole THPs during swapoff
In reply to Usama ArifAdd unuse_pmd() and call it from unuse_pmd_range() to swap in PMD-level swap entries as whole THPs during swapoff. This mirrors the existing unuse_pte_range() but operates at PMD granularity.
If the PMD-order folio cannot be allocated, the cached folio is no longer PMD-sized (e.g. split in the swap cache by deferred_split_scan() or memory_failure() while the PMD swap entry was installed), or the folio is not uptodate, the PMD swap entry is split into PTE-level entries via __split_huge_pmd() and a non-zero error is returned so unuse_pmd_range() falls through to unuse_pte_range(), which handles the individual entries at order-0.
--- mm/swapfile.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+)
mm/swapfile.c +145 -0
@@ -42,6 +42,7 @@#include <linux/suspend.h> #include <linux/zswap.h> #include <linux/plist.h> +#include <linux/huge_mm.h> #include <asm/tlbflush.h> #include <linux/leafops.h>
@@ -2641,6 +2642,138 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,return 0 ; } +/* +* unuse_pmd - Map a locked folio at PMD granularity during swapoff. +* +* The caller provides a locked, swapped-in folio. Returns 0 on success +* (PMD was mapped). Returns -EAGAIN if the swap cache folio no longer +* matches the entry or the PMD changed under the lock (try_to_unuse will +* rescan). Returns -EIO if the folio is not uptodate; in that case the +* PMD is split so unuse_pte_range() can handle individual pages. +*/ +static int unuse_pmd (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry, +struct folio *folio) +{ +struct mm_struct *mm = vma->vm_mm; +struct page *page; +pmd_t new_pmd, old_pmd; +spinlock_t *ptl; +rmap_t rmap_flags =RMAP_NONE ; +bool exclusive; + +if (unlikely (!folio_matches_swap_entry (folio, entry))) +return -EAGAIN ; + +if (unlikely (!folio_test_uptodate (folio))) { +__split_huge_pmd (vma, pmd, addr, false); +return -EIO ; + } + + page =folio_page (folio,0 ); + + ptl =pmd_lock (mm, pmd); + old_pmd =pmdp_get (pmd); + +if (!pmd_is_swap_entry (old_pmd) || +softleaf_from_pmd (old_pmd).val != entry.val) { +spin_unlock (ptl); +return -EAGAIN ; + } + + exclusive =pmd_swp_exclusive (old_pmd); + +/* +* Some architectures may have to restore extra metadata to the folio +* when reading from swap. This metadata may be indexed by swap entry +* so this must be called before folio_put_swap(). +*/ +arch_swap_restore (folio_swap (entry, folio), folio); + +add_mm_counter (mm,MM_ANONPAGES ,HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , -HPAGE_PMD_NR ); + + new_pmd =folio_mk_pmd (folio, vma->vm_page_prot); + new_pmd =pmd_mkold (new_pmd); +if (pmd_swp_soft_dirty (old_pmd)) + new_pmd =pmd_mksoft_dirty (new_pmd); +if (pmd_swp_uffd_wp (old_pmd)) + new_pmd =pmd_mkuffd_wp (new_pmd); + +if (exclusive) + rmap_flags |=RMAP_EXCLUSIVE ; + +folio_get (folio); +if (!folio_test_anon (folio)) +folio_add_new_anon_rmap (folio, vma, addr, rmap_flags); +else +folio_add_anon_rmap_pmd (folio, page, vma, addr, rmap_flags); + +set_pmd_at (mm, addr, pmd, new_pmd); +folio_put_swap (folio,NULL ); + +spin_unlock (ptl); + +folio_free_swap (folio); +return 0 ; +} + +/* +* Try to swap in a PMD swap entry as a whole THP. Returns 0 on success. +* Returns -ENOMEM if the PMD-order folio could not be allocated/charged, +* -EIO if swap-in failed, or -EAGAIN if the cached folio is no longer +* PMD-sized; in all of these the PMD is split so the caller can fall +* back to unuse_pte_range(). Otherwise propagates the error from +* unuse_pmd(). +*/ +static int unuse_pmd_entry (struct vm_area_struct *vma,pmd_t *pmd, +unsigned long addr,softleaf_t entry) +{ +struct folio *folio; +int ret; + + folio =swap_cache_get_folio (entry); +if (!folio) { +struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE , +BIT (HPAGE_PMD_ORDER ), &vmf,NULL ,0 ); +if (IS_ERR_OR_NULL (folio)) { + ret = -ENOMEM ; + goto split_fallback; + } + } + +folio_lock (folio); +folio_wait_writeback (folio); +/* +* If the cached folio is no longer PMD-sized (e.g. split in the +* swap cache by deferred_split_scan() or memory_failure() while +* the PMD swap entry was installed), the PMD swap entry no longer +* maps a single contiguous folio. Split the PMD swap entry so +* unuse_pte_range() can swap the per-slot folios in individually. +*/ +if (folio_nr_pages (folio) !=HPAGE_PMD_NR ) { +folio_unlock (folio); +folio_put (folio); + ret = -EAGAIN ; + goto split_fallback; + } + ret =unuse_pmd (vma, pmd, addr, entry, folio); +folio_unlock (folio); +folio_put (folio); +return ret; + +split_fallback: +__split_huge_pmd (vma, pmd, addr, false); +return ret; +} +static inline int unuse_pmd_range (struct vm_area_struct *vma,pud_t *pud,unsigned long addr,unsigned long end,unsigned int type)
@@ -2653,6 +2786,18 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, do { cond_resched(); next = pmd_addr_end(addr, end); + + pmd_t pmdval = pmdp_get(pmd); + + if (pmd_is_swap_entry(pmdval)) { + softleaf_t sl = softleaf_from_pmd(pmdval); + + if (swp_type(sl) == type) { + if (!unuse_pmd_entry(vma, pmd, addr, sl)) + continue; + } + } + ret = unuse_pte_range(vma, pmd, addr, next, type); if (ret) return ret;
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 11/16 mm: handle PMD swap entries in non-present PMD walkers
In reply to Usama ArifTeach the remaining non-present PMD walkers about swap entries, mirroring the PTE-level equivalents.
smaps_pmd_entry() accounts swap and swap_pss via a new shared smaps_account_swap() helper used by both PTE and PMD paths.
zap_huge_pmd() frees swap slots via swap_put_entries_direct(), matching zap_nonpresent_ptes().
change_non_present_huge_pmd() skips write-permission changes for swap entries and only updates uffd_wp, matching change_softleaf_pte().
move_soft_dirty_pmd(), clear_soft_dirty_pmd(), and make_uffd_wp_pmd(), pagemap_pmd_range_thp() and change_huge_pmd() handle swap entries alongside migration entries.
madvise_cold_or_pageout_pte_range() extends its non-present PMD VM_BUG_ON to allow swap entries; without this, hitting a PMD swap entry on a DEBUG_VM kernel would BUG().
mincore_pte_range() routes the pmd_trans_huge_lock() branch through mincore_swap() for non-present PMDs, matching how the PTE path already calls mincore_swap() for non-present PTEs. Without this a swapped-out PMD-mapped THP would be reported as resident, because pmd_is_huge() (and therefore pmd_trans_huge_lock()) accepts any non-present non-none PMD and the old branch unconditionally did memset(vec, 1, nr). mincore_swap() returns 1 for migration / device-private entries (preserving the prior behavior for those) and checks swap-cache residency for swap entries.
queue_folios_pmd() in mempolicy silently skips swap entries, matching the PTE walker which only counts migration entries as failures. Without this, mbind(MPOL_MF_STRICT) would spuriously return -EIO on a swapped-out THP.
madvise_free_huge_pmd() handles PMD swap entries directly: for a full-range MADV_FREE it clears the PMD, frees the deposited page table, and releases the swap slots; for a partial range it splits to PTE swap entries. Without this, MADV_FREE silently becomes a no-op on swapped-out THPs, leaking swap slots.
check_pmd_state() in khugepaged returns SCAN_PMD_MAPPED for PMD swap entries, treating a swapped-out THP as still being a THP from khugepaged's perspective and matching the existing migration-entry handling.
hmm_vma_handle_absent_pmd() faults in PMD swap entries via hmm_vma_fault() instead of returning -EFAULT. The first per-page handle_mm_fault() call triggers do_huge_pmd_swap_page(), which maps the entire folio; subsequent calls become harmless huge_pmd_set_accessed() and the walker retries with a present PMD.
--- fs/proc/task_mmu.c | 43 +++++++++++++++++++++------------- mm/hmm.c | 3 ++- mm/huge_memory.c | 58 +++++++++++++++++++++++++++++++++++----------- mm/khugepaged.c | 6 +++++ mm/madvise.c | 5 ++-- mm/mempolicy.c | 2 ++ mm/mincore.c | 14 ++++++++++- 7 files changed, 98 insertions(+), 33 deletions(-)
fs/proc/task_mmu.c +27 -16
@@ -1046,6 +1046,23 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)#endif } +static void smaps_account_swap (struct mem_size_stats *mss, +softleaf_t entry,unsigned long size) +{ +int mapcount; + + mss->swap += size; + mapcount =swp_swapcount (entry); +if (mapcount >=2 ) { +u64 pss_delta = (u64 )size <<PSS_SHIFT ; + +do_div (pss_delta, mapcount); + mss->swap_pss += pss_delta; + }else { + mss->swap_pss += (u64 )size <<PSS_SHIFT ; + } +} +static void smaps_pte_entry (pte_t *pte,unsigned long addr,struct mm_walk *walk) {
@@ -1067,18 +1084,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,const softleaf_t entry =softleaf_from_pte (ptent);if (softleaf_is_swap (entry)) { -int mapcount; - - mss->swap +=PAGE_SIZE ; - mapcount =swp_swapcount (entry); -if (mapcount >=2 ) { -u64 pss_delta = (u64 )PAGE_SIZE <<PSS_SHIFT ; - -do_div (pss_delta, mapcount); - mss->swap_pss += pss_delta; - }else { - mss->swap_pss += (u64 )PAGE_SIZE <<PSS_SHIFT ; - } +smaps_account_swap (mss, entry,PAGE_SIZE ); }else if (softleaf_has_pfn (entry)) {if (softleaf_is_device_private (entry)) present = true;
@@ -1108,9 +1114,13 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,if (pmd_present (*pmd)) { page =vm_normal_page_pmd (vma, addr, *pmd); present = true; - }else if (unlikely (thp_migration_supported ())) { + }else {const softleaf_t entry =softleaf_from_pmd (*pmd); +if (softleaf_is_swap (entry)) { +smaps_account_swap (mss, entry,HPAGE_PMD_SIZE ); +return ; + }if (softleaf_has_pfn (entry)) page =softleaf_to_page (entry); }
@@ -1752,7 +1762,7 @@ static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma, pmd =pmd_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_clear_soft_dirty (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
@@ -2112,7 +2122,8 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr, flags |=PM_UFFD_WP ;if (pm->show_pfn) frame =pmd_pfn (pmd) + idx; - }else if (thp_migration_supported ()) { + }else if (pmd_is_swap_entry (pmd) || + (thp_migration_supported () &&pmd_is_migration_entry (pmd))) {const softleaf_t entry =softleaf_from_pmd (pmd);unsigned long offset;
@@ -2550,7 +2561,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma, old =pmdp_invalidate_ad (vma, addr, pmdp); pmd =pmd_mkuffd_wp (old);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); - }else if (pmd_is_migration_entry (pmd)) { + }else if (pmd_is_migration_entry (pmd) ||pmd_is_swap_entry (pmd)) { pmd =pmd_swp_mkuffd_wp (pmd);set_pmd_at (vma->vm_mm, addr, pmdp, pmd); }
mm/hmm.c +2 -1
@@ -370,7 +370,8 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start, required_fault =hmm_range_need_fault (hmm_vma_walk, hmm_pfns, npages,0 );if (required_fault) { -if (softleaf_is_device_private (entry)) +if (softleaf_is_device_private (entry) || +softleaf_is_swap (entry))return hmm_vma_fault (addr, end, required_fault, walk);else return -EFAULT ;
mm/huge_memory.c +45 -13
101 lines not on this page · raw message
mm/khugepaged.c +6 -0
@@ -1111,6 +1111,12 @@ static inline enum scan_result check_pmd_state(pmd_t *pmd)*/ if (pmd_is_migration_entry (pmde))return SCAN_PMD_MAPPED ; +/* +* A PMD-mapped THP that has been swapped out is still a THP from +* khugepaged's perspective; treat it like a present huge PMD. +*/ +if (pmd_is_swap_entry (pmde)) +return SCAN_PMD_MAPPED ;if (!pmd_present (pmde))return SCAN_NO_PTE_TABLE ;if (pmd_trans_huge (pmde))
mm/madvise.c +3 -2
@@ -390,7 +390,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (unlikely (!pmd_present (orig_pmd))) {VM_BUG_ON (thp_migration_supported () && - !pmd_is_migration_entry (orig_pmd)); + !pmd_is_migration_entry (orig_pmd) && + !pmd_is_swap_entry (orig_pmd)); goto huge_unlock; }
@@ -666,7 +667,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,int nr, max_nr; next =pmd_addr_end (addr, end); -if (pmd_trans_huge (*pmd)) +if (pmd_trans_huge (*pmd) ||pmd_is_swap_entry (*pmd))if (madvise_free_huge_pmd (tlb, vma, pmd, addr, next))return 0 ;
mm/mempolicy.c +2 -0
@@ -658,6 +658,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) qp->nr_failed++;return ; } +if (unlikely (pmd_is_swap_entry (*pmd))) +return ; folio =pmd_folio (*pmd);if (is_huge_zero_folio (folio)) { walk->action =ACTION_CONTINUE ;
mm/mincore.c +13 -1
@@ -172,7 +172,19 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_present (*pmd)) { +memset (vec,1 , nr); + }else { +/* +* Non-present PMD: migration, device-private, or PMD +* swap entry. Route through mincore_swap() the same way +* the PTE path does -- the swap entry covers all 512 +* slots, so the whole vec gets the same answer. +*/ +softleaf_t entry =softleaf_from_pmd (*pmd); + +memset (vec,mincore_swap (entry, false), nr); + }spin_unlock (ptl); goto out; }
Signature · 1 line
2.52.0
Lance Yang
reviewerRe: [v2 11/16] mm: handle PMD swap entries in non-present PMD walkers
+Cc linux-mm
Please Cc linux-mm next time. Pretty clearly MM work ...
On Tue, Jun 02, 2026 at 07:24:19AM -0700, Usama Arif wrote:
[...]
Quoted 11 lines
mm/mincore.c +2
-1@@ -172,7 +172,19 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_present (*pmd)) { +memset (vec,1 , nr);
mm/mincore.c static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · line 186
Quoted 10 lines
mm/mincore.c +10
-0+ }else { +/* +* Non-present PMD: migration, device-private, or PMD +* swap entry. Route through mincore_swap() the same way +* the PTE path does -- the swap entry covers all 512 +* slots, so the whole vec gets the same answer. +*/ +softleaf_t entry =softleaf_from_pmd (*pmd); + +memset (vec,mincore_swap (entry, false), nr);
Looks buggy ...
That assumes one swap-cache lookup is enough for whole PMD-sized range. I don't think that always holds ...
See do_huge_pmd_swap_page():
---8<---
folio = swap_cache_get_folio(swp_entry);
[...]
/*
* Folio should be PMD-sized; if not (e.g. split in swap cache),
* split the PMD swap entry and retry at PTE level.
*/
if (folio_nr_pages(folio) != HPAGE_PMD_NR) {
folio_unlock(folio);
folio_put(folio);
goto split_fallback;
}---
it handles the case where swap_cache_get_folio() returns a folio that is no longer PMD-sized. E.g. because it was split in the swap cache while the PMD swap entry was installed. Then it split the PMD swap entry and retries at PTE level :)
unuse_pmd_entry() has the same fallback. Can mincore hit that case?
Maybe the comment right above should say something like:
"
One lookup is enough for a PMD-sized swapcache folio. If the swapcache was split, check the per-page swap slots.
"
Hopefully, I'm not missing something here :D
Cheers, Lance
Usama Arif
reviewerRe: [v2 11/16] mm: handle PMD swap entries in non-present PMD walkers
Quoted 3 lines · Lance Yang
+Cc linux-mm
Please Cc linux-mm next time. Pretty clearly MM work ...
Yes, thanks for this! I forgot, will be careful in v3.
Quoted 55 lines · Usama Arif
[...]
Quoted 11 lines
mm/mincore.c +2
-1@@ -172,7 +172,19 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_present (*pmd)) { +memset (vec,1 , nr);mm/mincore.c static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · line 186
Quoted 10 lines
mm/mincore.c +10
-0+ }else { +/* +* Non-present PMD: migration, device-private, or PMD +* swap entry. Route through mincore_swap() the same way +* the PTE path does -- the swap entry covers all 512 +* slots, so the whole vec gets the same answer. +*/ +softleaf_t entry =softleaf_from_pmd (*pmd); + +memset (vec,mincore_swap (entry, false), nr);Looks buggy ...
That assumes one swap-cache lookup is enough for whole PMD-sized range. I don't think that always holds ...
See
do_huge_pmd_swap_page():---8<---
folio = swap_cache_get_folio(swp_entry); [...] /* * Folio should be PMD-sized; if not (e.g. split in swap cache), * split the PMD swap entry and retry at PTE level. */ if (folio_nr_pages(folio) != HPAGE_PMD_NR) { folio_unlock(folio); folio_put(folio); goto split_fallback; }---
it handles the case where
swap_cache_get_folio()returns a folio that is no longer PMD-sized. E.g. because it was split in the swap cache while the PMD swap entry was installed. Then it split the PMD swap entry and retries at PTE level :)
unuse_pmd_entry()has the same fallback. Can mincore hit that case?Maybe the comment right above should say something like:
Quoted 8 lines
"
One lookup is enough for a PMD-sized swapcache folio. If the swapcache was split, check the per-page swap slots.
"Hopefully, I'm not missing something here :D
Cheers, Lance
Good catch! Thanks for pointing this out.
I think the below diff over this commit should be ok. I will add it to the next revision. Its slower, but it shouldn't be an issue as its just mincore:
mm/mincore.c +34 -7
@@ -175,15 +175,42 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (pmd_present (*pmd)) {memset (vec,1 , nr); }else { -/* -* Non-present PMD: migration, device-private, or PMD -* swap entry. Route through mincore_swap() the same way -* the PTE path does -- the swap entry covers all 512 -* slots, so the whole vec gets the same answer. -*/ softleaf_t entry =softleaf_from_pmd (*pmd); -memset (vec,mincore_swap (entry, false), nr); +/* +* Non-present PMD: migration, device-private, or +* PMD swap entry. Migration / device-private cover +* the whole PMD range with a single answer. +*/ +if (!softleaf_is_swap (entry)) { +memset (vec,mincore_swap (entry, false), nr); + }else { +struct folio *folio =swap_cache_get_folio (entry); + +/* +* One lookup is enough for a PMD-sized +* swapcache folio. If the swapcache was split +* (e.g. by deferred_split_scan() or +* memory_failure()) while the PMD swap entry +* was installed, check the per-page swap slots. +*/ +if (folio &&folio_nr_pages (folio) ==HPAGE_PMD_NR ) { +memset (vec,folio_test_uptodate (folio), nr); +folio_put (folio); + }else { +unsigned long haddr = addr &HPAGE_PMD_MASK ; +pgoff_t off =swp_offset (entry) + + ((addr - haddr) >>PAGE_SHIFT ); + +if (folio) +folio_put (folio); +for (i =0 ; i < nr; i++) + vec[i] =mincore_swap ( +swp_entry (swp_type (entry), + off + i), + false); + } + } }spin_unlock (ptl); goto out;
Lance Yang
reviewerRe: [v2 11/16] mm: handle PMD swap entries in non-present PMD walkers
Quoted 8 lines · Usama Arif
Quoted 3 lines · Lance Yang
+Cc linux-mm
Please Cc linux-mm next time. Pretty clearly MM work ...
Yes, thanks for this! I forgot, will be careful in v3.
Cool.
Quoted 62 lines
Quoted 62 lines · Usama Arif
[...]
Quoted 11 lines
mm/mincore.c +2
-1@@ -172,7 +172,19 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { -memset (vec,1 , nr); +if (pmd_present (*pmd)) { +memset (vec,1 , nr);mm/mincore.c static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · line 186
Quoted 10 lines
mm/mincore.c +10
-0+ }else { +/* +* Non-present PMD: migration, device-private, or PMD +* swap entry. Route through mincore_swap() the same way +* the PTE path does -- the swap entry covers all 512 +* slots, so the whole vec gets the same answer. +*/ +softleaf_t entry =softleaf_from_pmd (*pmd); + +memset (vec,mincore_swap (entry, false), nr);Looks buggy ...
That assumes one swap-cache lookup is enough for whole PMD-sized range. I don't think that always holds ...
See
do_huge_pmd_swap_page():---8<---
folio = swap_cache_get_folio(swp_entry); [...] /* * Folio should be PMD-sized; if not (e.g. split in swap cache), * split the PMD swap entry and retry at PTE level. */ if (folio_nr_pages(folio) != HPAGE_PMD_NR) { folio_unlock(folio); folio_put(folio); goto split_fallback; }---
it handles the case where
swap_cache_get_folio()returns a folio that is no longer PMD-sized. E.g. because it was split in the swap cache while the PMD swap entry was installed. Then it split the PMD swap entry and retries at PTE level :)
unuse_pmd_entry()has the same fallback. Can mincore hit that case?Maybe the comment right above should say something like:
"
One lookup is enough for a PMD-sized swapcache folio. If the swapcache was split, check the per-page swap slots.
"Hopefully, I'm not missing something here :D
Quoted 7 lines
Quoted 1 line
Cheers, Lance
Good catch! Thanks for pointing this out.
I think the below diff over this commit should be ok. I will add it to the next revision. Its slower, but it shouldn't be an issue as its just mincore:
Just skimmed it. That should do the trick. Will go through it properly in v3 :)
Thanks, Lance
Quoted 56 lines
mm/mincore.c +34
-7@@ -175,15 +175,42 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (pmd_present (*pmd)) {memset (vec,1 , nr); }else { -/* -* Non-present PMD: migration, device-private, or PMD -* swap entry. Route through mincore_swap() the same way -* the PTE path does -- the swap entry covers all 512 -* slots, so the whole vec gets the same answer. -*/ softleaf_t entry =softleaf_from_pmd (*pmd); -memset (vec,mincore_swap (entry, false), nr); +/* +* Non-present PMD: migration, device-private, or +* PMD swap entry. Migration / device-private cover +* the whole PMD range with a single answer. +*/ +if (!softleaf_is_swap (entry)) { +memset (vec,mincore_swap (entry, false), nr); + }else { +struct folio *folio =swap_cache_get_folio (entry); + +/* +* One lookup is enough for a PMD-sized +* swapcache folio. If the swapcache was split +* (e.g. by deferred_split_scan() or +* memory_failure()) while the PMD swap entry +* was installed, check the per-page swap slots. +*/ +if (folio &&folio_nr_pages (folio) ==HPAGE_PMD_NR ) { +memset (vec,folio_test_uptodate (folio), nr); +folio_put (folio); + }else { +unsigned long haddr = addr &HPAGE_PMD_MASK ; +pgoff_t off =swp_offset (entry) + + ((addr - haddr) >>PAGE_SHIFT ); + +if (folio) +folio_put (folio); +for (i =0 ; i < nr; i++) + vec[i] =mincore_swap ( +swp_entry (swp_type (entry), + off + i), + false); + } + } }spin_unlock (ptl); goto out;
Usama Arif
reviewerPATCH v2 12/16 mm: handle PMD swap entries in MADV_WILLNEED
In reply to Usama Arifswapin_walk_pmd_entry() walks PTEs and skips non-present PMDs, so MADV_WILLNEED is a no-op on a PMD swap entry. Read the whole 2 MB folio in at PMD order via swapin_sync(BIT(HPAGE_PMD_ORDER)) so the subsequent fault hits do_huge_pmd_swap_page() and restores the THP mapping; an order-0 read-ahead would force the fault to split.
--- mm/madvise.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
mm/madvise.c +40 -0
@@ -193,6 +193,46 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,spinlock_t *ptl;unsigned long addr; + ptl =pmd_trans_huge_lock (pmd, vma); +if (ptl) { +pmd_t pmdval = *pmd; + +if (pmd_is_swap_entry (pmdval)) { +softleaf_t entry =softleaf_from_pmd (pmdval); +struct vm_fault vmf = { + .vma = vma, + .address = start, + .real_address = start, + .pmd = pmd, + }; +struct swap_info_struct *si; +struct folio *folio; + +/* +* Pin the swap device under the PMD lock so the +* lookup is atomic with the PMD-swap-entry +* observation; swapin_sync() requires its caller to +* keep the device valid for the duration of the call. +*/ + si =get_swap_device (entry); +spin_unlock (ptl); +if (!si) { +cond_resched (); +return 0 ; + } + + folio =swapin_sync (entry,GFP_HIGHUSER_MOVABLE , +BIT (HPAGE_PMD_ORDER ), &vmf, +NULL ,0 ); +if (!IS_ERR_OR_NULL (folio)) +folio_put (folio); +put_swap_device (si); +cond_resched (); +return 0 ; + } +spin_unlock (ptl); + } +for (addr = start; addr < end; addr +=PAGE_SIZE ) {pte_t pte;softleaf_t entry;
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 13/16 mm: handle PMD swap entries in UFFDIO_MOVE
In reply to Usama Arifmove_pages_huge_pmd() returned -ENOENT for any non-trans_huge, non-migration PMD, which fails aligned UFFDIO_MOVE on a swapped-out THP -- the PMD swap entry is a perfectly valid mapping that should move whole. Splitting via the move_pages_ptes() fallback isn't a substitute either: __split_huge_pmd_locked() splits a PMD swap entry into HPAGE_PMD_NR PTE swap entries pointing at the same swap-cache folio, but move_swap_pte() refuses any swap-cache folio that is still large and returns -EBUSY.
Add move_swap_pmd(), modeled on move_swap_pte(), that moves the swap entry whole-PMD and re-anchors the swap-cache folio's anon rmap to the destination VMA. Reject !pmd_swp_exclusive() entries with -EBUSY to preserve UFFDIO_MOVE's single-owner semantics, propagate soft-dirty, and carry the deposited page table across with the entry.
The dispatcher in move_pages_huge_pmd() now waits for migration on a PMD migration entry (matching the PTE path) and routes PMD swap entries through move_swap_pmd() after pinning the swap device, fetching and locking any cached folio, and arming an mmu_notifier range so secondary MMUs see the move.
If the swap-cache folio was split (e.g. by deferred_split_scan or memory_failure) between swap-out and UFFDIO_MOVE, src_folio is no longer PMD-sized but the PMD swap entry still covers all 512 slots. Moving the entry whole would only re-anchor one folio's anon rmap, leaving the other 511 with a stale anon_vma. Return -EBUSY in this case, matching move_pages_pte()'s rejection of large folios, so the caller falls back to PTE-level moves.
--- mm/huge_memory.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 112 insertions(+), 1 deletion(-)
mm/huge_memory.c +112 -1
@@ -2812,6 +2812,62 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,#endif #ifdef CONFIG_USERFAULTFD +/* +* Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are +* acquired here; src_folio (if present) must already be locked. The deposited +* page table backing the source THP is moved across with the entry. +*/ +static int move_swap_pmd (struct mm_struct *mm,struct vm_area_struct *dst_vma, +unsigned long dst_addr,unsigned long src_addr, +pmd_t *dst_pmd,pmd_t *src_pmd, +pmd_t orig_dst_pmd,pmd_t orig_src_pmd, +spinlock_t *dst_ptl,spinlock_t *src_ptl, +struct folio *src_folio,swp_entry_t entry) +{ +pgtable_t src_pgtable; +pmd_t moved_pmd; + +/* +* The folio may have been freed and reused for a different swap entry +* while it was unlocked. Re-verify the association. +*/ +if (src_folio &&unlikely (!folio_test_swapcache (src_folio) || + entry.val != src_folio->swap.val)) +return -EAGAIN ; + +double_pt_lock (dst_ptl, src_ptl); + +if (!pmd_same (*src_pmd, orig_src_pmd) || + !pmd_same (*dst_pmd, orig_dst_pmd)) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + +/* +* If the folio is in the swap cache, re-anchor its anon rmap to the +* destination VMA so a future swap-in fault at dst_addr finds it. +* Otherwise, re-check that no folio was newly inserted under us. +*/ +if (src_folio) { +folio_move_anon_rmap (src_folio, dst_vma); + src_folio->index =linear_page_index (dst_vma, dst_addr); + }else if (swap_cache_has_folio (entry)) { +double_pt_unlock (dst_ptl, src_ptl); +return -EAGAIN ; + } + + moved_pmd =pmdp_huge_get_and_clear (mm, src_addr, src_pmd); +if (pgtable_supports_soft_dirty ()) + moved_pmd =pmd_swp_mksoft_dirty (moved_pmd); +set_pmd_at (mm, dst_addr, dst_pmd, moved_pmd); + + src_pgtable =pgtable_trans_huge_withdraw (mm, src_pmd); +pgtable_trans_huge_deposit (mm, dst_pmd, src_pgtable); + +double_pt_unlock (dst_ptl, src_ptl); +return 0 ; +} +/* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by * the caller, but it must return after releasing the page_table_lock.
@@ -2846,11 +2902,66 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }if (!pmd_trans_huge (src_pmdval)) { -spin_unlock (src_ptl);if (pmd_is_migration_entry (src_pmdval)) { +spin_unlock (src_ptl);pmd_migration_entry_wait (mm, &src_pmdval);return -EAGAIN ; } +if (pmd_is_swap_entry (src_pmdval)) { +swp_entry_t entry; +struct swap_info_struct *si; + +/* +* UFFDIO_MOVE on anon mappings requires single-owner +* semantics; refuse to move a shared swap entry. +*/ +if (!pmd_swp_exclusive (src_pmdval)) { +spin_unlock (src_ptl); +return -EBUSY ; + } + + entry =softleaf_from_pmd (src_pmdval); +spin_unlock (src_ptl); + +/* Pin the swap device against a racing swapoff. */ + si =get_swap_device (entry); +if (unlikely (!si)) +return -EAGAIN ; + + src_folio =swap_cache_get_folio (entry); + +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , + mm, src_addr, + src_addr +HPAGE_PMD_SIZE ); +mmu_notifier_invalidate_range_start (&range); + +if (src_folio) { +folio_lock (src_folio); +if (folio_nr_pages (src_folio) !=HPAGE_PMD_NR ) { + err = -EBUSY ; +folio_unlock (src_folio); +folio_put (src_folio); +mmu_notifier_invalidate_range_end (&range); +put_swap_device (si); +return err; + } + } + + dst_ptl =pmd_lockptr (mm, dst_pmd); + err =move_swap_pmd (mm, dst_vma, dst_addr, src_addr, + dst_pmd, src_pmd, dst_pmdval, + src_pmdval, dst_ptl, src_ptl, + src_folio, entry); + +mmu_notifier_invalidate_range_end (&range); +if (src_folio) { +folio_unlock (src_folio); +folio_put (src_folio); + } +put_swap_device (si); +return err; + } +spin_unlock (src_ptl);return -ENOENT ; }
Signature · 1 line
2.52.0
Lance Yang
reviewerRe: [v2 13/16] mm: handle PMD swap entries in UFFDIO_MOVE
+Cc linux-mm
On Tue, Jun 02, 2026 at 07:24:21AM -0700, Usama Arif wrote:
[...]
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2910
Quoted 11 lines
mm/huge_memory.c +2
-1@@ -2846,11 +2902,66 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }if (!pmd_trans_huge (src_pmdval)) { -spin_unlock (src_ptl);if (pmd_is_migration_entry (src_pmdval)) { +spin_unlock (src_ptl);pmd_migration_entry_wait (mm, &src_pmdval);return -EAGAIN ; } +if (pmd_is_swap_entry (src_pmdval)) {
Looks buggy ... unless I missed something ...
Quoted 11 lines
mm/huge_memory.c +11
-0+swp_entry_t entry; +struct swap_info_struct *si; + +/* +* UFFDIO_MOVE on anon mappings requires single-owner +* semantics; refuse to move a shared swap entry. +*/ +if (!pmd_swp_exclusive (src_pmdval)) { +spin_unlock (src_ptl); +return -EBUSY ; + }
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2931
Quoted 10 lines
mm/huge_memory.c +10
-0+ + entry =softleaf_from_pmd (src_pmdval); +spin_unlock (src_ptl); + +/* Pin the swap device against a racing swapoff. */ + si =get_swap_device (entry); +if (unlikely (!si)) +return -EAGAIN ; + + src_folio =swap_cache_get_folio (entry);
We only check the first swap slot. Imagine we have something like this after the PMD-sized swapcache folio was split while the PMD swap entry was installed:
page table:
src PMD -> swap entry S
swap cache:
S + 0 -> no folio S + 1 -> order-0 folio in the swap cache S + 2 -> no folio S + 3 -> order-0 folio in the swap cache
...
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2940
Quoted 9 lines
mm/huge_memory.c +9
-0+ +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , + mm, src_addr, + src_addr +HPAGE_PMD_SIZE ); +mmu_notifier_invalidate_range_start (&range); + +if (src_folio) { +folio_lock (src_folio); +if (folio_nr_pages (src_folio) !=HPAGE_PMD_NR ) {
If S has a non-PMD-sized folio, this returns -EBUSY.
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2950
Quoted 10 lines
mm/huge_memory.c +10
-0+ err = -EBUSY ; +folio_unlock (src_folio); +folio_put (src_folio); +mmu_notifier_invalidate_range_end (&range); +put_swap_device (si); +return err; + } + } + + dst_ptl =pmd_lockptr (mm, dst_pmd);
But if S has no folio, the initial lookup passes src_folio == NULL to move_swap_pmd(), , which only rechecks S:
if (src_folio) {
[...]
} else if (swap_cache_has_folio(entry)) {
double_pt_unlock(dst_ptl, src_ptl);
return -EAGAIN;
}So if S is empty, the move can still go ahead even if S + 1 ... S + 511 contain folios in the swap cache.
mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2955
Quoted 5 lines
mm/huge_memory.c +5
-0+ err =move_swap_pmd (mm, dst_vma, dst_addr, src_addr, + dst_pmd, src_pmd, dst_pmdval, + src_pmdval, dst_ptl, src_ptl, + src_folio, entry); +
In that case, checking only S misses the order-0 folios in later slots. move_swap_pmd() can then move the PMD swap entry whole without calling folio_move_anon_rmap() or updating folio->index for those later folios.
Note that move_swap_pte() already does this for PTE-mapped swap entries, because a folio in the swap cache needs its index and mapping updated to align with dst_vma.
If those folios are later faulted in at dst, their rmap metadata still points at the old anon_vma/index. Later rmap users derive the virtual address from folio->mapping and folio->index, so they can look at the wrong VMA/address ...
Should check the whole PMD swap range before deciding there is no folio in the swap cache to update?
Am I reading that code right?
Quoted 9 lines
mm/huge_memory.c +9
-0+mmu_notifier_invalidate_range_end (&range); +if (src_folio) { +folio_unlock (src_folio); +folio_put (src_folio); + } +put_swap_device (si); +return err; + } +spin_unlock (src_ptl);
Quoted 6 lines
return -ENOENT; }Signature · 1 line
2.52.0
Cheers, Lance
Usama Arif
reviewerPATCH v2 14/16 mm: handle PMD swap entry faults on swap-in
In reply to Usama ArifAdd do_huge_pmd_swap_page() and dispatch to it from __handle_mm_fault() when vmf->orig_pmd encodes a swap entry. The handler resolves the entire 2 MB mapping in one shot, mirroring do_swap_page() (PTE path) at PMD granularity:
- Look up the folio in the swap cache; on a miss, allocate a PMD-order folio via swap_cache_alloc_folio() and read from swap.
- After locking, re-validate that the folio still corresponds to our entry and is still PMD-sized. Between the unlocked cache lookup and the lock, a racing swap-in on the same entry may have removed it from the cache via folio_free_swap(), or reclaim / memory_failure / deferred-split may have split the folio into smaller folios.
- Restore soft_dirty and uffd_wp from the swap PMD. Map writable only when the entry was exclusive, the VMA permits writes, and uffd-wp is not armed. Drop the exclusive marker when the cached folio is under writeback to an SWP_STABLE_WRITES backend (zram, encrypted) so the PMD is mapped read-only; a later write COWs into a fresh folio rather than corrupting the in-flight writeback. Mirrors do_swap_page().
- When the resulting PMD is read-only but the fault was a write, update vmf->orig_pmd and call wp_huge_pmd() in the same handler to COW immediately rather than forcing a second fault. Mask VM_FAULT_FALLBACK from its return: a PMD-COW that splits to PTE-level is normal, but the bit is part of VM_FAULT_ERROR and arch fault handlers BUG() on it without SIGBUS/HWPOISON/SIGSEGV. Requires exposing wp_huge_pmd() via mm/internal.h.
- Free the swap slot via should_try_to_free_swap() (hoisted from mm/memory.c into mm/internal.h so PTE- and PMD-level swap-in share the heuristic).
When PMD-order resources are unavailable (folio allocation fails, the cached folio was split, memcg charge fails, or swapin_folio() races) split the PMD swap entry into 512 PTE swap entries via __split_huge_pmd() and return 0. The fault retries and do_swap_page() takes over per-PTE. This avoids returning VM_FAULT_OOM for transient PMD-order allocation failures.
--- include/linux/huge_mm.h | 9 ++ mm/huge_memory.c | 198 ++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 36 ++++++++ mm/memory.c | 40 +------- 4 files changed, 247 insertions(+), 36 deletions(-)
include/linux/huge_mm.h +9 -0
@@ -531,6 +531,15 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);vm_fault_t do_huge_pmd_device_private (struct vm_fault *vmf); +#ifdef CONFIG_THP_SWAP +vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +#else +static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) +{ +return 0 ; +} +#endif +extern struct folio *huge_zero_folio;extern unsigned long huge_zero_pfn;
mm/huge_memory.c +198 -0
204 lines not on this page · raw message
mm/internal.h +36 -0
@@ -499,6 +499,42 @@ static inline vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) }vm_fault_t do_swap_page (struct vm_fault *vmf); +vm_fault_t wp_huge_pmd (struct vm_fault *vmf); + +/* +* Check if we should call folio_free_swap to free the swap cache. +* folio_free_swap only frees the swap cache to release the slot if swap +* count is zero, so we don't need to check the swap count here. +*/ +static inline bool should_try_to_free_swap (struct swap_info_struct *si, +struct folio *folio, +struct vm_area_struct *vma, +unsigned int extra_refs, +unsigned int fault_flags) +{ +if (!folio_test_swapcache (folio)) +return false; +/* +* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap +* cache can help save some IO or memory overhead, but these devices +* are fast, and meanwhile, swap cache pinning the slot deferring the +* release of metadata or fragmentation is a more critical issue. +*/ +if (data_race (si->flags &SWP_SYNCHRONOUS_IO )) +return true; +if (mem_cgroup_swap_full (folio) || (vma->vm_flags &VM_LOCKED ) || +folio_test_mlocked (folio)) +return true; +/* +* If we want to map a page that's in the swapcache writable, we +* have to detect via the refcount if we're really the exclusive +* user. Try freeing the swapcache to get rid of the swapcache +* reference only in case it's likely that we'll be the exclusive user. +*/ +return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && +folio_ref_count (folio) == (extra_refs +folio_nr_pages (folio)); +} +void folio_rotate_reclaimable (struct folio *folio);bool __folio_end_writeback (struct folio *folio);void deactivate_file_folio (struct folio *folio);
mm/memory.c +4 -36
@@ -4497,40 +4497,6 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)return 0 ; } -/* -* Check if we should call folio_free_swap to free the swap cache. -* folio_free_swap only frees the swap cache to release the slot if swap -* count is zero, so we don't need to check the swap count here. -*/ -static inline bool should_try_to_free_swap (struct swap_info_struct *si, -struct folio *folio, -struct vm_area_struct *vma, -unsigned int extra_refs, -unsigned int fault_flags) -{ -if (!folio_test_swapcache (folio)) -return false; -/* -* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap -* cache can help save some IO or memory overhead, but these devices -* are fast, and meanwhile, swap cache pinning the slot deferring the -* release of metadata or fragmentation is a more critical issue. -*/ -if (data_race (si->flags &SWP_SYNCHRONOUS_IO )) -return true; -if (mem_cgroup_swap_full (folio) || (vma->vm_flags &VM_LOCKED ) || -folio_test_mlocked (folio)) -return true; -/* -* If we want to map a page that's in the swapcache writable, we -* have to detect via the refcount if we're really the exclusive -* user. Try freeing the swapcache to get rid of the swapcache -* reference only in case it's likely that we'll be the exclusive user. -*/ -return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && -folio_ref_count (folio) == (extra_refs +folio_nr_pages (folio)); -} -static vm_fault_t pte_marker_clear (struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
@@ -6200,8 +6166,7 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)return VM_FAULT_FALLBACK ; } -/* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd (struct vm_fault *vmf) +vm_fault_t wp_huge_pmd (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;const bool unshare = vmf->flags &FAULT_FLAG_UNSHARE ;
@@ -6486,6 +6451,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,if (pmd_is_migration_entry (vmf.orig_pmd))pmd_migration_entry_wait (mm, vmf.pmd); +else if (IS_ENABLED (CONFIG_THP_SWAP ) && +pmd_is_swap_entry (vmf.orig_pmd)) +return do_huge_pmd_swap_page (&vmf);return 0 ; }if (pmd_trans_huge (vmf.orig_pmd)) {
Signature · 1 line
2.52.0
Usama Arif
reviewerPATCH v2 15/16 mm: install PMD swap entries on swap-out
In reply to Usama ArifReclaim today splits a PMD-mapped anonymous THP into 512 PTE swap entries before unmap, losing the huge mapping across the swap round-trip and forcing khugepaged to rebuild it later. The contiguous swap range was already secured when the folio was added to the swap cache (a non-contiguous allocation would have split the folio earlier), so the PMD can be replaced by a single PMD-level swap entry instead.
This patch mirrors the existing PTE swap-out path at PMD granularity:
- shrink_folio_list() drops TTU_SPLIT_HUGE_PMD for PMD-mappable swapcache folios, gated on zswap_never_enabled() since zswap cannot reconstruct a 2 MB folio from per-page blobs (Best to handle zswap case separately).
- try_to_unmap_one() now has a PMD branch that calls set_pmd_swap_entry() and adjusts MM_ANONPAGES / MM_SWAPENTS by HPAGE_PMD_NR before walk_done. TTU_SPLIT_HUGE_PMD remains the fallback.
- set_pmd_swap_entry() is the installer. Mirroring the PTE swap-out sequence at PMD granularity, it clears the present mapping (keeping the original for rollback), bumps the swap_map refcount for the folio's 512 slots, drops the exclusive mark if the page was anon-exclusive, propagates the dirty bit to the folio so writeback is not lost, and installs a swap PMD that preserves the original soft-dirty / uffd-wp / exclusive bits. Any failing step rolls back the present mapping.
The swap entry value matches what 512 PTE swap entries would encode, so swap_map refcounting is unchanged: each of the 512 slots carries a count of 1, released individually on later split or together on swap-in.
--- include/linux/huge_mm.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 78 +++++++++++++++++++++++++++++++++++ mm/rmap.c | 20 +++++++++ mm/vmscan.c | 14 ++++++- mm/vmstat.c | 1 + 6 files changed, 115 insertions(+), 1 deletion(-)
include/linux/huge_mm.h +2 -0
@@ -533,6 +533,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);#ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio);#else static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) {
include/linux/vm_event_item.h +1 -0
@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_ZERO_PAGE_ALLOC_FAILED ,THP_SWPOUT ,THP_SWPOUT_FALLBACK , +THP_SWPOUT_PMD , #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE,
mm/huge_memory.c +78 -0
81 lines not on this page · raw message
mm/rmap.c +20 -0
@@ -2079,6 +2079,26 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_abort; } +#ifdef CONFIG_THP_SWAP +/* +* If the folio is in the swap cache and we're not +* asked to split, install a PMD-level swap entry. +*/ +if (!(flags &TTU_SPLIT_HUGE_PMD ) && +folio_test_anon (folio) && +folio_test_swapcache (folio)) { +if (set_pmd_swap_entry (&pvmw, folio)) + goto walk_abort; + +mm_prepare_for_swap_entries (mm); +add_mm_counter (mm,MM_ANONPAGES , + -HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , +HPAGE_PMD_NR ); + goto walk_done; + } +#endif +if (flags &TTU_SPLIT_HUGE_PMD ) {/* * We temporarily have to drop the PTL and
mm/vmscan.c +13 -1
@@ -64,6 +64,7 @@#include <linux/swapops.h> #include <linux/sched/sysctl.h> +#include <linux/zswap.h> #include "internal.h" #include "swap.h"
@@ -1332,7 +1333,18 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio); -if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the +* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +* Skip this when zswap has been enabled because +* zswap stores pages individually and cannot +* reconstruct a large folio on swap-in. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio) && +zswap_never_enabled ())) flags |=TTU_SPLIT_HUGE_PMD ;/* * Without TTU_SYNC, try_to_unmap will only begin to
mm/vmstat.c +1 -0
@@ -1421,6 +1421,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I (THP_SWPOUT_PMD )] ="thp_swpout_pmd" , #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",
Signature · 1 line
2.52.0
Lance Yang
reviewerRe: [v2 15/16] mm: install PMD swap entries on swap-out
+Cc linux-mm
On Tue, Jun 02, 2026 at 07:24:23AM -0700, Usama Arif wrote:
[...]
Quoted 19 lines
mm/vmscan.c +3
-1@@ -64,6 +64,7 @@#include <linux/swapops.h> #include <linux/sched/sysctl.h> +#include <linux/zswap.h> #include "internal.h" #include "swap.h" @@ -1332,7 +1333,18 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio); -if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1347
Quoted 10 lines
mm/vmscan.c +10
-0+* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +* Skip this when zswap has been enabled because +* zswap stores pages individually and cannot +* reconstruct a large folio on swap-in. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio) && +zswap_never_enabled ()))
There may be a race here ...
1) zswap_never_enabled() passes, 2) try_to_unmap() installs the PMD swap entry, and 3) zswap can still be enabled before the later pageout() -> swap_writeout() -> zswap_store().
zswap_store() loops over each page of the folio:
for (index = 0; index < nr_pages; ++index) {
struct page *page = folio_page(folio, index);
if (!zswap_store_page(page, objcg, pool))
goto put_pool;
}So still one PMD swap entry, while zswap has 512 entries, one for each page of the folio ...
If the swapcache is reclaimed later, a PMD fault will try PMD-order swapin again:
do_huge_pmd_swap_page()
swap_cache_get_folio() swapin_sync(..., BIT(HPAGE_PMD_ORDER)) swap_read_folio() zswap_load()
zswap_load() rejects large folios with -EINVAL and leaves the folio not uptodate:
/*
* Large folios should not be swapped in while zswap is being used, as
* they are not properly handled. Zswap does not properly load large
* folios, and a large folio may only be partially in zswap.
*/
if (WARN_ON_ONCE(folio_test_large(folio))) {
folio_unlock(folio);
return -EINVAL;
}swap_read_folio() jumps to finish and does not try a normal swap read:
if (zswap_load(folio) != -ENOENT) goto finish;
And the awkward part is that no error really gets propagated ... swap_read_folio() is void, and swapin_sync() just hands the same folio back to do_huge_pmd_swap_page().
At that point the folio is still !uptodate, so the fault would just end up:
if (unlikely(!folio_test_uptodate(folio))) {
ret = VM_FAULT_SIGBUS;
goto out_page;
}Looks race, but possible?
Cheers, Lance
Quoted 19 lines
mm/vmscan.c +0
-0flags |=TTU_SPLIT_HUGE_PMD ;/* * Without TTU_SYNC, try_to_unmap will only begin to mm/vmstat.c +1
-0@@ -1421,6 +1421,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I (THP_SWPOUT_PMD )] ="thp_swpout_pmd" , #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",Signature · 1 line
2.52.0
Alexandre Ghiti
reviewerRe: [v2 15/16] mm: install PMD swap entries on swap-out
In reply to Usama ArifHi Usama,
Quoted 202 lines · Usama Arif
Reclaim today splits a PMD-mapped anonymous THP into 512 PTE swap entries before unmap, losing the huge mapping across the swap round-trip and forcing khugepaged to rebuild it later. The contiguous swap range was already secured when the folio was added to the swap cache (a non-contiguous allocation would have split the folio earlier), so the PMD can be replaced by a single PMD-level swap entry instead.
This patch mirrors the existing PTE swap-out path at PMD granularity:
-shrink_folio_list()dropsTTU_SPLIT_HUGE_PMDfor PMD-mappable swapcache folios, gated onzswap_never_enabled()since zswap cannot reconstruct a 2 MB folio from per-page blobs (Best to handle zswap case separately).
-try_to_unmap_one()now has a PMD branch that callsset_pmd_swap_entry()and adjustsMM_ANONPAGES/MM_SWAPENTSbyHPAGE_PMD_NRbeforewalk_done.TTU_SPLIT_HUGE_PMDremains the fallback.
-set_pmd_swap_entry()is the installer. Mirroring the PTE swap-out sequence at PMD granularity, it clears the present mapping (keeping the original for rollback), bumps theswap_maprefcount for the folio's 512 slots, drops the exclusive mark if the page was anon-exclusive, propagates the dirty bit to the folio so writeback is not lost, and installs a swap PMD that preserves the original soft-dirty / uffd-wp / exclusive bits. Any failing step rolls back the present mapping.The swap entry value matches what 512 PTE swap entries would encode, so
swap_maprefcounting is unchanged: each of the 512 slots carries a count of 1, released individually on later split or together on swap-in.Signed-off-by Usama Arif <[email protected]>--- include/linux/huge_mm.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 78 +++++++++++++++++++++++++++++++++++ mm/rmap.c | 20 +++++++++ mm/vmscan.c | 14 ++++++- mm/vmstat.c | 1 + 6 files changed, 115 insertions(+), 1 deletion(-)include/linux/huge_mm.h +2
-0@@ -533,6 +533,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);#ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf); +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio);#else static inline vm_fault_t do_huge_pmd_swap_page (struct vm_fault *vmf) {include/linux/vm_event_item.h +1
-0@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_ZERO_PAGE_ALLOC_FAILED ,THP_SWPOUT ,THP_SWPOUT_FALLBACK , +THP_SWPOUT_PMD , #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE,mm/huge_memory.c +78
-0@@ -5385,3 +5385,81 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)trace_remove_migration_pmd (address,pmd_val (pmde)); }#endif + +#ifdef CONFIG_THP_SWAP +/** +* set_pmd_swap_entry() - Replace a PMD mapping with a PMD-level swap entry. +* @pvmw: Page vma mapped walk context, must have pvmw->pmd set and +* pvmw->pte NULL (i.e. PMD-mapped). +* @folio: The folio being swapped out. Must be in the swap cache. +* +* This installs a PMD-level swap entry in place of a present PMD mapping, +* avoiding the need to split the PMD into PTE-level swap entries. +* +* Return: 0 on success, negative error code on failure. +*/ +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, +struct folio *folio) +{ +struct vm_area_struct *vma = pvmw->vma; +struct mm_struct *mm = vma->vm_mm; +unsigned long address = pvmw->address; +unsigned long haddr = address &HPAGE_PMD_MASK ; +struct page *page =folio_page (folio,0 ); +bool anon_exclusive; +pmd_t pmdval; +swp_entry_t entry; +pmd_t pmdswp; + +if (!(pvmw->pmd && !pvmw->pte)) +return 0 ; + +VM_BUG_ON_FOLIO (!folio_test_swapcache (folio), folio); +VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); + +if (unlikely (folio_test_swapbacked (folio) != +folio_test_swapcache (folio))) { +WARN_ON_ONCE (1 ); +return -EBUSY ; + } + +flush_cache_range (vma, haddr, haddr +HPAGE_PMD_SIZE ); + + pmdval =pmdp_invalidate (vma, haddr, pvmw->pmd); + +/* Update high watermark before we lower rss */ +update_hiwater_rss (mm); + +if (folio_dup_swap (folio,NULL ) <0 ) { +set_pmd_at (mm, haddr, pvmw->pmd, pmdval); +return -ENOMEM ; + } + +/* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ + anon_exclusive =PageAnonExclusive (page); +if (anon_exclusive &&folio_try_share_anon_rmap_pmd (folio, page)) { +folio_put_swap (folio,NULL ); +set_pmd_at (mm, haddr, pvmw->pmd, pmdval); +return -EBUSY ; + } + +if (pmd_dirty (pmdval)) +folio_mark_dirty (folio); + + entry = folio->swap; + pmdswp =softleaf_to_pmd (entry); +if (pmd_soft_dirty (pmdval)) + pmdswp =pmd_swp_mksoft_dirty (pmdswp); +if (pmd_uffd_wp (pmdval)) + pmdswp =pmd_swp_mkuffd_wp (pmdswp); +if (anon_exclusive) + pmdswp =pmd_swp_mkexclusive (pmdswp); +set_pmd_at (mm, haddr, pvmw->pmd, pmdswp); + +folio_remove_rmap_pmd (folio, page, vma); +folio_put (folio); + +count_vm_event (THP_SWPOUT_PMD ); +return 0 ; +} +#endif /* CONFIG_THP_SWAP */ mm/rmap.c +20
-0@@ -2079,6 +2079,26 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_abort; } +#ifdef CONFIG_THP_SWAP +/* +* If the folio is in the swap cache and we're not +* asked to split, install a PMD-level swap entry. +*/ +if (!(flags &TTU_SPLIT_HUGE_PMD ) && +folio_test_anon (folio) && +folio_test_swapcache (folio)) { +if (set_pmd_swap_entry (&pvmw, folio)) + goto walk_abort; + +mm_prepare_for_swap_entries (mm); +add_mm_counter (mm,MM_ANONPAGES , + -HPAGE_PMD_NR ); +add_mm_counter (mm,MM_SWAPENTS , +HPAGE_PMD_NR ); + goto walk_done; + } +#endif +if (flags &TTU_SPLIT_HUGE_PMD ) {/* * We temporarily have to drop the PTL and mm/vmscan.c +3
-1@@ -64,6 +64,7 @@#include <linux/swapops.h> #include <linux/sched/sysctl.h> +#include <linux/zswap.h> #include "internal.h" #include "swap.h" @@ -1332,7 +1333,18 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,enum ttu_flags flags =TTU_BATCH_FLUSH ;bool was_swapbacked =folio_test_swapbacked (folio); -if (folio_test_pmd_mappable (folio)) +/* +* With THP_SWAP, PMD-mappable folios already in the
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1347
Quoted 10 lines
mm/vmscan.c +10
-0+* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +* Skip this when zswap has been enabled because +* zswap stores pages individually and cannot +* reconstruct a large folio on swap-in. +*/ +if (folio_test_pmd_mappable (folio) && + !(IS_ENABLED (CONFIG_THP_SWAP ) && +folio_test_swapcache (folio) && +zswap_never_enabled ()))
While working on the PMD zswap support, I noticed the following (small) problem: if zswap is enabled at runtime, there could be PMD entries created before, then in zswap_load(), after zswap enablement, it would fail with -EINVAL and finally swap_read_folio() would not even try to read it from the disk.
I fixed it with the following patch, let me know what you think:
mm/zswap.c +16 -10
@@ -1571,10 +1571,9 @@ bool zswap_store(struct folio *folio)* NOT marked up-to-date, so that an IO error is emitted (e.g. do_swap_page() * will SIGBUS). * -* -EINVAL: if the swapped out content was in zswap, but the page belongs -* to a large folio, which is not supported by zswap. The folio is unlocked, -* but NOT marked up-to-date, so that an IO error is emitted (e.g. -* do_swap_page() will SIGBUS). +* -EIO: if a slot in a large-folio range is unexpectedly still in zswap. +* The folio is unlocked, but NOT marked up-to-date, so that an IO error is +* emitted (e.g. do_swap_page() will SIGBUS). * * -ENOENT: if the swapped out content was not in zswap. The folio remains * locked on return.
@@ -1593,13 +1592,29 @@ int zswap_load(struct folio *folio)return -ENOENT ;/* -* Large folios should not be swapped in while zswap is being used, as -* they are not properly handled. Zswap does not properly load large -* folios, and a large folio may only be partially in zswap. +* A large (PMD) folio reaches zswap_load() only when its whole range +* is on disk: do_huge_pmd_swap_page() splits the PMD swap entry to +* PTEs and faults order-0 whenever any slot is still in zswap, so +* zswap never reconstructs a large folio. Confirm the range is +* entirely absent from zswap and return -ENOENT so the caller reads it +* from disk; if a slot is unexpectedly still in zswap, fail the read +* rather than return partially-initialised data. */ -if (WARN_ON_ONCE (folio_test_large (folio))) { -folio_unlock (folio); -return -EINVAL ; +if (folio_test_large (folio)) { +unsigned long nr_pages =folio_nr_pages (folio); +XA_STATE (xas, tree, offset); +bool any; + +/* One xa_state walk over the range, not a per-slot
xa_load(). */
+ rcu_read_lock(); + any = xas_find(&xas, offset + nr_pages - 1) != NULL; + rcu_read_unlock();
+
+ if (any) {
+ folio_unlock(folio);
+ return -EIO;
+ }
+ return -ENOENT;
}
entry = xa_load(tree, offset);Thanks,
Alex
Quoted 15 lines
mm/vmscan.c +0
-0flags |=TTU_SPLIT_HUGE_PMD ;/* * Without TTU_SYNC, try_to_unmap will only begin to mm/vmstat.c +1
-0@@ -1421,6 +1421,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I (THP_SWPOUT_PMD )] ="thp_swpout_pmd" , #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",
Usama Arif
reviewerRe: [v2 15/16] mm: install PMD swap entries on swap-out
Quoted 205 lines · Alexandre Ghiti
Hi Usama,
Quoted 202 lines · Usama Arif
Reclaim today splits a PMD-mapped anonymous THP into 512 PTE swap entries before unmap, losing the huge mapping across the swap round-trip and forcing khugepaged to rebuild it later. The contiguous swap range was already secured when the folio was added to the swap cache (a non-contiguous allocation would have split the folio earlier), so the PMD can be replaced by a single PMD-level swap entry instead.
This patch mirrors the existing PTE swap-out path at PMD granularity:
-shrink_folio_list()dropsTTU_SPLIT_HUGE_PMDfor PMD-mappable swapcache folios, gated onzswap_never_enabled()since zswap cannot reconstruct a 2 MB folio from per-page blobs (Best to handle zswap case separately).
-try_to_unmap_one()now has a PMD branch that callsset_pmd_swap_entry()and adjustsMM_ANONPAGES/MM_SWAPENTSbyHPAGE_PMD_NRbefore walk_done.TTU_SPLIT_HUGE_PMDremains the fallback.
-set_pmd_swap_entry()is the installer. Mirroring the PTE swap-out sequence at PMD granularity, it clears the present mapping (keeping the original for rollback), bumps theswap_maprefcount for the folio's 512 slots, drops the exclusive mark if the page was anon-exclusive, propagates the dirty bit to the folio so writeback is not lost, and installs a swap PMD that preserves the original soft-dirty / uffd-wp / exclusive bits. Any failing step rolls back the present mapping.The swap entry value matches what 512 PTE swap entries would encode, so
swap_maprefcounting is unchanged: each of the 512 slots carries a count of 1, released individually on later split or together on swap-in.Signed-off-by Usama Arif <[email protected]>---
include/linux/huge_mm.h | 2 +include/linux/vm_event_item.h| 1 +
mm/huge_memory.c | 78 +++++++++++++++++++++++++++++++++++ mm/rmap.c | 20 +++++++++
mm/vmscan.c | 14 ++++++-
mm/vmstat.c | 1 +6 files changed, 115 insertions(+), 1 deletion(-)include/linux/huge_mm.h +2
-0@@ -533,6 +533,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf); #ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf); +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, + struct folio *folio); #else static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) {include/linux/vm_event_item.h +1
-0@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, THP_ZERO_PAGE_ALLOC_FAILED, THP_SWPOUT, THP_SWPOUT_FALLBACK, + THP_SWPOUT_PMD, #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE,mm/huge_memory.c +78
-0@@ -5385,3 +5385,81 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) trace_remove_migration_pmd(address, pmd_val(pmde)); } #endif + +#ifdef CONFIG_THP_SWAP +/** +* set_pmd_swap_entry() - Replace a PMD mapping with a PMD-level swap entry. +* @pvmw: Page vma mapped walk context, must have pvmw->pmd set and +* pvmw->pte NULL (i.e. PMD-mapped). +* @folio: The folio being swapped out. Must be in the swap cache. +* +* This installs a PMD-level swap entry in place of a present PMD mapping, +* avoiding the need to split the PMD into PTE-level swap entries. +* +* Return: 0 on success, negative error code on failure. +*/ +int set_pmd_swap_entry (struct page_vma_mapped_walk *pvmw, + struct folio *folio) +{ + struct vm_area_struct *vma = pvmw->vma; + struct mm_struct *mm = vma->vm_mm; + unsigned long address = pvmw->address; + unsigned long haddr = address & HPAGE_PMD_MASK; + struct page *page = folio_page(folio, 0); + bool anon_exclusive; + pmd_t pmdval; + swp_entry_t entry; + pmd_t pmdswp; + + if (!(pvmw->pmd && !pvmw->pte)) + return 0; + + VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); + + if (unlikely(folio_test_swapbacked(folio) != + folio_test_swapcache(folio))) { + WARN_ON_ONCE(1); + return -EBUSY; + } + + flush_cache_range(vma, haddr, haddr + HPAGE_PMD_SIZE); + + pmdval = pmdp_invalidate(vma, haddr, pvmw->pmd); + + /* Update high watermark before we lower rss */ + update_hiwater_rss(mm); + + if (folio_dup_swap(folio, NULL) < 0) { + set_pmd_at(mm, haddr, pvmw->pmd, pmdval); + return -ENOMEM; + } + + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ + anon_exclusive = PageAnonExclusive(page); + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { + folio_put_swap(folio, NULL); + set_pmd_at(mm, haddr, pvmw->pmd, pmdval); + return -EBUSY; + } + + if (pmd_dirty(pmdval)) + folio_mark_dirty(folio); + + entry = folio->swap; + pmdswp = softleaf_to_pmd(entry); + if (pmd_soft_dirty(pmdval)) + pmdswp = pmd_swp_mksoft_dirty(pmdswp); + if (pmd_uffd_wp(pmdval)) + pmdswp = pmd_swp_mkuffd_wp(pmdswp); + if (anon_exclusive) + pmdswp = pmd_swp_mkexclusive(pmdswp); + set_pmd_at(mm, haddr, pvmw->pmd, pmdswp); + + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); + + count_vm_event(THP_SWPOUT_PMD); + return 0; +} +#endif /* CONFIG_THP_SWAP */ mm/rmap.c +0
-0@@ -2079,6 +2079,26 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_abort; }+#ifdef
CONFIG_THP_SWAP
+ /*
+ * If the folio is in the swap cache and we're not + * asked to split, install a PMD-level swap entry. + */
+ if (!(flags &TTU_SPLIT_HUGE_PMD) &&
+folio_test_anon(folio)&&
+folio_test_swapcache(folio)) {
+ if (set_pmd_swap_entry(&pvmw, folio)) + gotowalk_abort;mm/rmap.c +10
-0+ + mm_prepare_for_swap_entries(mm); + add_mm_counter(mm, MM_ANONPAGES, + -HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, + HPAGE_PMD_NR); + goto walk_done; + } +#endif +if (flags &
TTU_SPLIT_HUGE_PMD) {
/*
* We temporarily have to drop the PTL andmm/vmscan.c +8
-0@@ -64,6 +64,7 @@ #include <linux/swapops.h> #include <linux/sched/sysctl.h> +#include <linux/zswap.h> #include "internal.h" #include "swap.h"@@ -1332,7 +1333,18 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, enum ttu_flags flags = TTU_BATCH_FLUSH; bool was_swapbacked = folio_test_swapbacked(folio); - if (folio_test_pmd_mappable(folio)) + /* +* With THP_SWAP, PMD-mappable folios already in the +* swap cache can be unmapped with a PMD-level swap +* entry, avoiding the cost of splitting the PMD. +* Skip this when zswap has been enabled because +* zswap stores pages individually and cannot +* reconstruct a large folio on swap-in.
Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1347
Quoted 5 lines
mm/vmscan.c +5
-0+*/ + if (folio_test_pmd_mappable(folio) && + !(IS_ENABLED(CONFIG_THP_SWAP) && + folio_test_swapcache(folio) && + zswap_never_enabled()))While working on the PMD zswap support, I noticed the following (small) problem: if zswap is enabled at runtime, there could be PMD entries created before, then in
zswap_load(), after zswap enablement, it would fail with-EINVALand finallyswap_read_folio()would not even try to read it from the disk.
Thanks Alexandre!
I think its similar to what Lance raised in https://lore.kernel.org/all/[email protected]/.
What I have locally is at the end, but I need to test it and spend more time into it.
I will look at your change as well before sending the next revision (looks better at a first glance)! The current strategy for this series is in: https://lore.kernel.org/all/[email protected]/
mm/zswap.c +26 -3
@@ -1592,17 +1592,40 @@ int zswap_load(struct folio *folio)if (zswap_never_enabled ())return -ENOENT ; + entry =xa_load (tree, offset); +/* * Large folios should not be swapped in while zswap is being used, as * they are not properly handled. Zswap does not properly load large * folios, and a large folio may only be partially in zswap. +* +* If no zswap entry exists for the folio, however, the caller can read +* it from the backing swap device. Scan the covered slots so a PMD +* swapin racing with zswap enable does not SIGBUS solely because zswap +* became available after the PMD swap entry was installed. */ -if (WARN_ON_ONCE (folio_test_large (folio))) { +if (folio_test_large (folio)) { +unsigned int type =swp_type (swp); +bool found = !!entry; +long index; + +if (!found) { +for (index =1 ; index <folio_nr_pages (folio); index++) { +swp_entry_t cur =swp_entry (type, offset + index); + +if (xa_load (swap_zswap_tree (cur), offset + index)) { + found = true; +break ; + } + } + } +if (!found) +return -ENOENT ; + +WARN_ON_ONCE (1 );folio_unlock (folio);return -EINVAL ; } - - entry =xa_load (tree, offset);if (!entry)return -ENOENT ;
Quoted 80 lines
I fixed it with the following patch, let me know what you think:
mm/zswap.c +16
-10@@ -1571,10 +1571,9 @@ bool zswap_store(struct folio *folio) * NOT marked up-to-date, so that an IO error is emitted (e.g. do_swap_page() * will SIGBUS). * - * -EINVAL: if the swapped out content was in zswap, but the page belongs - * to a large folio, which is not supported by zswap. The folio is unlocked, - * but NOT marked up-to-date, so that an IO error is emitted (e.g. - * do_swap_page() will SIGBUS). + * -EIO: if a slot in a large-folio range is unexpectedly still in zswap. + * The folio is unlocked, but NOT marked up-to-date, so that an IO error is + * emitted (e.g. do_swap_page() will SIGBUS). * * -ENOENT: if the swapped out content was not in zswap. The folio remains * locked on return.@@ -1593,13 +1592,29 @@ int zswap_load(struct folio *folio) return -ENOENT; /* - * Large folios should not be swapped in while zswap is being used, as - * they are not properly handled. Zswap does not properly load large - * folios, and a large folio may only be partially in zswap. + * A large (PMD) folio reaches zswap_load() only when its whole range + * is on disk: do_huge_pmd_swap_page() splits the PMD swap entry to + * PTEs and faults order-0 whenever any slot is still in zswap, so + * zswap never reconstructs a large folio. Confirm the range is + * entirely absent from zswap and return -ENOENT so the caller reads it + * from disk; if a slot is unexpectedly still in zswap, fail the read + * rather than return partially-initialised data. */ - if (WARN_ON_ONCE(folio_test_large(folio))) { - folio_unlock(folio); - return -EINVAL; + if (folio_test_large(folio)) { + unsigned long nr_pages = folio_nr_pages(folio); + XA_STATE(xas, tree, offset); + bool any; + + /* One xa_state walk over the range, not a per-slot
xa_load(). */
+rcu_read_lock();
+ any =xas_find(&xas, offset + nr_pages - 1)!= NULL; +rcu_read_unlock();
+
+ if (any) {
+folio_unlock(folio);
+ return-EIO;
+ }
+ return-ENOENT;
}entry =
xa_load(tree, offset);Thanks,
Alex
Quoted 15 lines
mm/vmscan.c +0
-0flags |= TTU_SPLIT_HUGE_PMD; /** Without TTU_SYNC, try_to_unmap will only begin to mm/vmstat.c +1
-0@@ -1421,6 +1421,7 @@ const char * const vmstat_text[] = { [I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] = "thp_swpout", [I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback", + [I(THP_SWPOUT_PMD)] = "thp_swpout_pmd", #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] = "balloon_inflate",
Usama Arif
reviewerPATCH v2 16/16 selftests/mm: add PMD swap entry tests
In reply to Usama ArifExercise the PMD swap entry paths. The tests allocate a PMD-mapped THP, write a known pattern, swap it out via MADV_PAGEOUT, and then exercise different code paths:
- swap-out / swap-in round-trip with data verification
- fork with read-only access from both parent and child
- fork with writes in both processes to verify COW isolation
- repeated swap cycles to try and catch reference counting issues
- write fault on a swapped PMD to verify dirty handling
- munmap of a swapped PMD (zap_huge_pmd swap slot cleanup)
- mprotect on a swapped PMD (change_non_present_huge_pmd)
- mremap of a swapped PMD (move_soft_dirty_pmd)
- pagemap reading (pagemap_pmd_range_thp softleaf_has_pfn guard)
- MADV_FREE on a swapped PMD: verifies swap slots are freed via pagemap and the memory reads back as zero
- UFFDIO_MOVE on a swapped PMD (move_pages_huge_pmd swap path); verifies the entry transfers without splitting and that the destination faults back in as a THP
- swapoff with active PMD swap entries (unuse_pmd_range split)
--- tools/testing/selftests/mm/Makefile | 1 + tools/testing/selftests/mm/pmd_swap.c | 672 ++++++++++++++++++++++++++ 2 files changed, 673 insertions(+) create mode 100644 tools/testing/selftests/mm/pmd_swap.c
tools/testing/selftests/mm/Makefile +1 -0
@@ -105,6 +105,7 @@ TEST_GEN_FILES += guard-regions TEST_GEN_FILES += merge TEST_GEN_FILES += rmap TEST_GEN_FILES += folio_split_race_test +TEST_GEN_FILES += pmd_swap ifneq ($(ARCH),arm64) TEST_GEN_FILES += soft-dirty
tools/testing/selftests/mm/pmd_swap.c new file +672 -0
672 lines not on this page · raw message
Signature · 1 line
2.52.0
Usama Arif
reviewerQuoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Thanks!
Usama
David Hildenbrand (Arm)
maintainerQuoted 15 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.
Quoted 6 lines
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Signature · 3 lines
Cheers, David
Lance Yang
reviewerQuoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.Hello!
Quoted 7 lines
Quoted 4 lines
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
Right, mTHP khugepaged was a rough one. Glad we got it over the line, but yeah, there's just been a lot of THP work lately. pretty nonstop ...
Quoted 5 lines
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Usama, I'll try to make this one a priority too. Looks interesting :P
Cheers, Lance
David Hildenbrand (Arm)
maintainerQuoted 20 lines · Lance Yang
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 5 lines · Usama Arif
Hello!
Quoted 7 lines
Quoted 4 lines
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
Right, mTHP khugepaged was a rough one. Glad we got it over the line, but yeah, there's just been a lot of THP work lately. pretty nonstop ...
Quoted 7 lines
Quoted 5 lines
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Usama, I'll try to make this one a priority too. Looks interesting :P
I have two other bigger series to review, but I should soon get to this as well.
Signature · 3 lines
Cheers, David
Usama Arif
reviewerQuoted 15 lines · David Hildenbrand (Arm)
Quoted 14 lines · Lance Yang
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 9 lines · Usama Arif
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
Right, mTHP khugepaged was a rough one. Glad we got it over the line, but yeah, there's just been a lot of THP work lately. pretty nonstop ...
Yeah its definitely a lot. I have set a target of leaving review comments on atleast 2 patches from mm per day myself, but even that can sometimes be difficult! I will try and help out more in reviews.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Usama, I'll try to make this one a priority too. Looks interesting :P
Thanks Lance!
Quoted 3 lines
I have two other bigger series to review, but I should soon get to this as well.
No worries at all! Thanks for the reviews! and yeah definitely 7.3.
I will send this out again when 7.3-rc1 opens (rebased), so that the reviews wont be on outdated code which could cause some confusion.
Lance Yang
reviewerQuoted 21 lines · Usama Arif
Quoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · Lance Yang
Quoted 13 lines · David Hildenbrand (Arm)
Quoted 9 lines · Usama Arif
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Right, mTHP khugepaged was a rough one. Glad we got it over the line, but yeah, there's just been a lot of THP work lately. pretty nonstop ...
Yeah its definitely a lot. I have set a target of leaving review comments on atleast 2 patches from mm per day myself, but even that can sometimes be difficult! I will try and help out more in reviews.
Awesome!
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Usama, I'll try to make this one a priority too. Looks interesting :P
Thanks Lance!
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
I have two other bigger series to review, but I should soon get to this as well.
No worries at all! Thanks for the reviews! and yeah definitely 7.3.
I will send this out again when 7.3-rc1 opens (rebased), so that the reviews wont be on outdated code which could cause some confusion.
After skimming through the whole series, probably PMD swap entries need one bigger rethink ...
Emm ... same tricky bit keeps showing up ...
One PMD swap entry is easy to handle while the swapcache still has one PMD-sized folio behind it. Once taht folio got split and reclaimed, the 512 swap slots need per-page handling :)
Maybe worth first pinning down the rule here.
Is a PMD swap entry supposed to mean "there is, or soon will be, one PMD- sized folio behnid it", or is just a compact page-table encoding for 512 swap slot?
Without that rule being very clear, every caller has to guess how much it can assume, and it is easy to miss one ...
So I stopped staring at the details for now, because the same issue keeps popping up wearing a slightly different hat :)
Anyway, no clever answer from me here, not a swap expect :( Just pointing out the pattern I keep runing into.
Thanks, Lance
Usama Arif
reviewerQuoted 67 lines · Lance Yang
Quoted 21 lines · Usama Arif
Quoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · Lance Yang
Quoted 13 lines · David Hildenbrand (Arm)
Quoted 9 lines · Usama Arif
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
It shall be reviewed. We just finished the mTHP khugepaged review to get it into 7.2, so we've all been rather busy.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Right, mTHP khugepaged was a rough one. Glad we got it over the line, but yeah, there's just been a lot of THP work lately. pretty nonstop ...
Yeah its definitely a lot. I have set a target of leaving review comments on atleast 2 patches from mm per day myself, but even that can sometimes be difficult! I will try and help out more in reviews.
Awesome!
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
(I mean, just take a look at the THP-related flood of patches we are fighting with on a daily basis, it's not funny anymore)
This is clearly going to be 7.3 material, so there is plenty of time given that the merge window is about to open soon.
Usama, I'll try to make this one a priority too. Looks interesting :P
Thanks Lance!
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
I have two other bigger series to review, but I should soon get to this as well.
No worries at all! Thanks for the reviews! and yeah definitely 7.3.
I will send this out again when 7.3-rc1 opens (rebased), so that the reviews wont be on outdated code which could cause some confusion.
After skimming through the whole series, probably PMD swap entries need one bigger rethink ...
Emm ... same tricky bit keeps showing up ...
One PMD swap entry is easy to handle while the swapcache still has one PMD-sized folio behind it. Once taht folio got split and reclaimed, the 512 swap slots need per-page handling :)
Maybe worth first pinning down the rule here.
Is a PMD swap entry supposed to mean "there is, or soon will be, one PMD- sized folio behnid it", or is just a compact page-table encoding for 512 swap slot?
Quoted 9 lines
Without that rule being very clear, every caller has to guess how much it can assume, and it is easy to miss one ...
So I stopped staring at the details for now, because the same issue keeps popping up wearing a slightly different hat :)
Anyway, no clever answer from me here, not a swap expect :( Just pointing out the pattern I keep runing into.
Thanks for the amazing reviews!
For the next revision I’m going to treat a PMD swap entry as just a compact page-table encoding for 512 ordinary swap slots. It does not mean that the swapcache still has, or will soon have, one PMD-sized folio behind it.
With that rule, whole-PMD handling is only valid when either:
1. the swapcache still has one PMD-sized folio for the range, or
2. the whole PMD swap range has no cached folios, so the caller can try a PMD-sized swapin and still fall back if that is not possible.
If any slot in the range has per-page cache state, the PMD entry has to be split and the existing PTE paths need to handle the individual slots.
I an reworking the next revision around that. I added a shared helper to classify the swapcache behind a PMD swap entry as empty, PMD-sized, or split, then used it in the places where this assumption mattered:
mincore, UFFDIO_MOVE, swapoff, MADV_WILLNEED, and the PMD swap fault path. UFFDIO_MOVE now checks the whole 512-slot range before moving a PMD swap entry without a cached folio, and falls back to PTE handling if per-page cached folios exist.
Thanks!
Usama
Quoted 46 lines · Usama Arif
Quoted 17 lines · Lance Yang
After skimming through the whole series, probably PMD swap entries need one bigger rethink ...
Emm ... same tricky bit keeps showing up ...
One PMD swap entry is easy to handle while the swapcache still has one PMD-sized folio behind it. Once taht folio got split and reclaimed, the 512 swap slots need per-page handling :)
Maybe worth first pinning down the rule here.
Is a PMD swap entry supposed to mean "there is, or soon will be, one PMD- sized folio behnid it", or is just a compact page-table encoding for 512 swap slot?
Quoted 9 lines
Without that rule being very clear, every caller has to guess how much it can assume, and it is easy to miss one ...
So I stopped staring at the details for now, because the same issue keeps popping up wearing a slightly different hat :)
Anyway, no clever answer from me here, not a swap expect :( Just pointing out the pattern I keep runing into.
Thanks for the amazing reviews!
For the next revision I’m going to treat a PMD swap entry as just a compact page-table encoding for 512 ordinary swap slots. It does not mean that the swapcache still has, or will soon have, one PMD-sized folio behind it.
With that rule, whole-PMD handling is only valid when either:
1. the swapcache still has one PMD-sized folio for the range, or
2. the whole PMD swap range has no cached folios, so the caller can try a PMD-sized swapin and still fall back if that is not possible.If any slot in the range has per-page cache state, the PMD entry has to be split and the existing PTE paths need to handle the individual slots.
Quoted 7 lines
I an reworking the next revision around that. I added a shared helper to classify the swapcache behind a PMD swap entry as empty, PMD-sized, or split, then used it in the places where this assumption mattered:
mincore,UFFDIO_MOVE, swapoff,MADV_WILLNEED, and the PMD swap fault path.UFFDIO_MOVEnow checks the whole 512-slot range before moving a PMD swap entry without a cached folio, and falls back to PTE handling if per-page cached folios exist.
Looks interesting, sorry I haven't go though the whole series in detail yet, one question from me is how much different is it compared to the current THP handling? We are already dealing with THP fallback where sub-pages (of any order) could exist in the swap cache range covered by an incoming (swapin) (m)THP, the only thing different here is we need to do split, right? Meanwhile shmem already has similar split logic (split of large shmem mapping xarray entry), maybe they are somehow similiar so some routine can be shared or reused?
Zi Yan
reviewerQuoted 13 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.
Quoted 6 lines
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Maybe send first 6 clean-up patches separately to get them merged first?
--
Best Regards,
Yan, Zi
Usama Arif
reviewerQuoted 15 lines · Zi Yan
Quoted 13 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.
Quoted 8 lines
Quoted 6 lines
Hello!
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Maybe send first 6 clean-up patches separately to get them merged first?
Good idea! Will do that. Thanks!
Quoted 4 lines
--
Best Regards,
Yan, Zi
Usama Arif
reviewerQuoted 20 lines · Usama Arif
Quoted 17 lines · Zi Yan
Quoted 15 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.Hello!
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Maybe send first 6 clean-up patches separately to get them merged first?
Good idea! Will do that. Thanks!
So what I will do is once the merge window closes, send the first 6 patches for review, then when they make it into mm-new (so that we get sashiko feedback), send the core patches that I have reworked based on Lances' feedback. [1] [2]
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Lance Yang
reviewerQuoted 20 lines · Usama Arif
Quoted 17 lines · Zi Yan
Quoted 15 lines · Usama Arif
Quoted 11 lines · Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is split into 512 PTE-level swap entries via
TTU_SPLIT_HUGE_PMDbefore unmap.This series introduces a PMD-level swap entry. The huge mapping is preserved across the swap round-trip, and
do_huge_pmd_swap_page()resolves the entire 2 MB region in a single fault on swap-in, no khugepaged involvement is needed.swap_mapmetadata is identical either way (512 single-slot counts), so the PTE split buys nothing on the swap side, it is purely a page-table representation change.Hello!
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Just following up if there were any reviews/comments on this series!
I know its a large series but was just checking if there was any feedback?
Maybe send first 6 clean-up patches separately to get them merged first?
Good idea! Will do that. Thanks!
Yeah, sounds good. Less churn, and the cleanups can land first, and life is easier for everyone :D