Skip to content

mm: batch rmap walks during large folio migration

Shivank Garg linux-mm lkml 2 versions – In review

Get series with b4
b4 am [email protected] change-id 20260701-migrate-rmap-batch-8ecc04de1905

Versions newest first

  1. v2 Latest7 patches 16 repliesIn review
  2. v16 patches 6 repliesSuperseded

Changes in v2 from the cover letter

- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes, and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David) Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(), which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison() once and skip batching when mapped page is poisoned, and use same result for hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/[email protected]

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7mm: factor out generic PTE batch detection from swap_pte_batch()mm/internal.h+21 -153 replies
2/7mm/migrate: factor out migration PTE constructionmm/migrate.c+32 -210 replies
3/7mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte()mm/migrate.c+63 -342 replies
4/7mm/migrate: batch the restore-side migration rmap walkmm/migrate.c+42 -70 replies
5/7mm/rmap: factor out migration PTE constructionmm/rmap.c+38 -270 replies
6/7mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one()mm/rmap.c+144 -1070 replies
7/7mm/rmap: batch the unmap of large folios in try_to_migrate_one()mm/rmap.c+86 -297 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/7] mm: batch rmap walks during large folio migration

Migrating a PTE-mapped large folio currently walks the folio's rmap one page at a time. It does so twice per migration:
1. try_to_migrate_one(): Replace present PTEs with migration entries
2. remove_migration_pte(): Restore migration entries to working PTEs.

For a folio with N contiguous PTEs, each walk repeats page_vma_mapped_walk() and the associated operation per subpage of the folio. (256 iterations for 1M folio).

This series enables both walks to process a batch of consecutive PTEs that map the same large folio and do some refactoring and clean up on the way.

Performance:

AMD EPYC ZEN 3, 2-socket system (1 node per socket). move_pages() migrating 1 GB from node 0 -> node 1, throughput in GB/s:
batch- batch-restore

folio     vanilla       restore-only   + unmap         restore%   restore+unmap%
4K        2.96±0.15     3.02±0.14      2.99±0.17         +2.2%    +1.0%
64K       5.99±0.10     6.12±0.08      7.65±0.16         +2.2%   +27.7%
256K      6.48±0.09     6.67±0.08      9.23±0.14         +2.9%   +42.4%
1M        6.76±0.10     6.98±0.10      9.74±0.19         +3.1%   +44.0%

2M 10.50±0.20 10.44±0.16 10.67±0.21 -0.5% +1.6%

In plain migration, the folio copy itself dominates the cost, so these rmap walk savings are only partly visible end-to-end.

When the folio copy is batched-copy and offloaded to DMA engine [1], the copy is no longer the bottleneck and rmap-walk batching translates into much larger gains:

This series + [1], DMA Offload on PTDMA (DCBM), 16 channels:

batch-copy dcbm+restore

folio    vanilla    -offload(dcbm)  dcbm+restore    +unmap          (rmap batch impact)
4K       3.05±0.13   2.76±0.11       2.77±0.12        2.74±0.09
64K      5.85±0.11  10.66±0.27      11.56±1.45       17.55±0.56
256K     6.26±0.08  13.64±2.73      13.68±0.16       34.76±0.57       (2.54x)
1M       6.58±0.08  14.25±0.07      15.27±0.77       44.28±0.27       (3.11x)

2M 10.61±0.24 65.26±0.96 65.25±1.65 65.66±0.87

Applies cleanly on mm-new+[2].

[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]

---
Changes in v2:

- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes, and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David) Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(), which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison() once and skip batching when mapped page is poisoned, and use same result for hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Shivank Garg (7):
mm: factor out generic PTE batch detection from swap_pte_batch() mm/migrate: factor out migration PTE construction
mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte() mm/migrate: batch the restore-side migration rmap walk
mm/rmap: factor out migration PTE construction
mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one() mm/rmap: batch the unmap of large folios in try_to_migrate_one()

 mm/internal.h |  36 ++---
 mm/migrate.c  | 191 ++++++++++++++++++--------
 mm/rmap.c     | 419 ++++++++++++++++++++++++++++++++++++----------------------
 3 files changed, 416 insertions(+), 230 deletions(-)

---
base-commit: 480a31230b426efb005b6e71a14ef80f405f18b6
change-id: 20260701-migrate-rmap-batch-8ecc04de1905
prerequisite-message-id: [email protected]
prerequisite-patch-id: 5fdb03c3ea4a037f66fbd1c9b5d04e24e86406eb
prerequisite-patch-id: 033980575a1ef89d70e43a67c5530e2cf6cf97aa
prerequisite-patch-id: 6f7508334fc2bfc0df104ac50d36ed88072487e4
prerequisite-patch-id: 3f1aaae36e02d8b9343f7e719f732f10c1767572
prerequisite-patch-id: 98bc99c6bc274594946d022b3998f5f1dc70012f
prerequisite-patch-id: dd35ec00b82bf235f2e41e286d16eab6a3fd1e9e
prerequisite-patch-id: 552ee939dc6a7b3cce51373f4e75fa01a9d2b979
prerequisite-patch-id: 485cdbbe4e65b13494dc5104d4b8b7d0d23911a4

Best regards,

Signature · 1 line
Shivank Garg <[email protected]>