mm: batch rmap walks during large folio migration
Get series with b4
b4 am [email protected] change-id 20260701-migrate-rmap-batch-8ecc04de1905Versions newest first
- v2 Latest7 patches 16 repliesIn review
- v16 patches 6 repliesSuperseded
Changes in v2 from the cover letter
- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes, and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David) Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(), which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison() once and skip batching when mapped page is poisoned, and use same result for hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/[email protected]
Patches in v2 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/7 | mm: factor out generic PTE batch detection from swap_pte_batch()mm/ | +21 | 3 replies | |
| 2/7 | mm/migrate: factor out migration PTE constructionmm/ | +32 | 0 replies | |
| 3/7 | mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte()mm/ | +63 | 2 replies | |
| 4/7 | mm/migrate: batch the restore-side migration rmap walkmm/ | +42 | 0 replies | |
| 5/7 | mm/rmap: factor out migration PTE constructionmm/ | +38 | 0 replies | |
| 6/7 | mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one()mm/ | +144 | 0 replies | |
| 7/7 | mm/rmap: batch the unmap of large folios in try_to_migrate_one()mm/ | +86 | 7 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.
Cover letter [PATCH v2 0/7] mm: batch rmap walks during large folio migration
Migrating a PTE-mapped large folio currently walks the folio's rmap one page at a time. It does so twice per migration:
1. try_to_migrate_one(): Replace present PTEs with migration entries
2. remove_migration_pte(): Restore migration entries to working PTEs.
For a folio with N contiguous PTEs, each walk repeats page_vma_mapped_walk() and the associated operation per subpage of the folio. (256 iterations for 1M folio).
This series enables both walks to process a batch of consecutive PTEs that map the same large folio and do some refactoring and clean up on the way.
Performance:
AMD EPYC ZEN 3, 2-socket system (1 node per socket). move_pages() migrating 1 GB from node 0 -> node 1, throughput in GB/s:
batch- batch-restore
folio vanilla restore-only + unmap restore% restore+unmap% 4K 2.96±0.15 3.02±0.14 2.99±0.17 +2.2% +1.0% 64K 5.99±0.10 6.12±0.08 7.65±0.16 +2.2% +27.7% 256K 6.48±0.09 6.67±0.08 9.23±0.14 +2.9% +42.4% 1M 6.76±0.10 6.98±0.10 9.74±0.19 +3.1% +44.0%
2M 10.50±0.20 10.44±0.16 10.67±0.21 -0.5% +1.6%
In plain migration, the folio copy itself dominates the cost, so these rmap walk savings are only partly visible end-to-end.
When the folio copy is batched-copy and offloaded to DMA engine [1], the copy is no longer the bottleneck and rmap-walk batching translates into much larger gains:
This series + [1], DMA Offload on PTDMA (DCBM), 16 channels:
batch-copy dcbm+restore
folio vanilla -offload(dcbm) dcbm+restore +unmap (rmap batch impact) 4K 3.05±0.13 2.76±0.11 2.77±0.12 2.74±0.09 64K 5.85±0.11 10.66±0.27 11.56±1.45 17.55±0.56 256K 6.26±0.08 13.64±2.73 13.68±0.16 34.76±0.57 (2.54x) 1M 6.58±0.08 14.25±0.07 15.27±0.77 44.28±0.27 (3.11x)
2M 10.61±0.24 65.26±0.96 65.25±1.65 65.66±0.87
Applies cleanly on mm-new+[2].
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
---
Changes in v2:
- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes, and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David) Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(), which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison() once and skip batching when mapped page is poisoned, and use same result for hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Shivank Garg (7):
mm: factor out generic PTE batch detection from swap_pte_batch() mm/migrate: factor out migration PTE construction
mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte() mm/migrate: batch the restore-side migration rmap walk
mm/rmap: factor out migration PTE construction
mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one() mm/rmap: batch the unmap of large folios in try_to_migrate_one()
mm/internal.h | 36 ++--- mm/migrate.c | 191 ++++++++++++++++++-------- mm/rmap.c | 419 ++++++++++++++++++++++++++++++++++++---------------------- 3 files changed, 416 insertions(+), 230 deletions(-)
---
base-commit: 480a31230b426efb005b6e71a14ef80f405f18b6
change-id: 20260701-migrate-rmap-batch-8ecc04de1905
prerequisite-message-id: [email protected]
prerequisite-patch-id: 5fdb03c3ea4a037f66fbd1c9b5d04e24e86406eb
prerequisite-patch-id: 033980575a1ef89d70e43a67c5530e2cf6cf97aa
prerequisite-patch-id: 6f7508334fc2bfc0df104ac50d36ed88072487e4
prerequisite-patch-id: 3f1aaae36e02d8b9343f7e719f732f10c1767572
prerequisite-patch-id: 98bc99c6bc274594946d022b3998f5f1dc70012f
prerequisite-patch-id: dd35ec00b82bf235f2e41e286d16eab6a3fd1e9e
prerequisite-patch-id: 552ee939dc6a7b3cce51373f4e75fa01a9d2b979
prerequisite-patch-id: 485cdbbe4e65b13494dc5104d4b8b7d0d23911a4
Best regards,
Signature · 1 line
Shivank Garg <[email protected]>