Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA
Subsystems (3)
v2 cover Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA
This patchset introduces enhancements to the page migration by batching folio-copy operations and using multiple CPU threads for copying or offloading the copy to DMA hardware.
It builds upon Zi's work on accelerating page migration via multi-threading[1] and my previous work on enhancing page migration with batch offloading via DMA[2].
MOTIVATION:
-----------
Page migration costs have become increasingly critical in modern systems with memory-tiers and NUMA nodes:
1. Batching folio copies increases throughput, especially for base page migrations where kernel activities (moving folio metadata, updating page table entries) create overhead between individual copies. This is particularly important for smaller page-sizes (4KB on x86_64/ARM64, 64KB on ARM64).
2. Current simple serial copy patterns underutilize modern hardware capabilities, leaving memory migration bandwidth capped by single-threaded CPU-bound operations.
These improvements are particularly valuable in:
- Large-scale tiered-memory systems with CXL nodes and HBM
- CPU-GPU coherent systems with GPU memory exposed as NUMA nodes
- Systems where frequent page promotion/demotion occurs
Following the trend of batching operations in the memory migration core path (batch migration, batch TLB flush), batch copying folio content is the logical next step. Modern systems equipped with powerful hardware accelerators (DMA engines), GPUs, and high CPU core counts offer untapped potential for hardware acceleration.
DESIGN:
-------
The patchset implements three key enhancements:
1. Batching:
- Current approach: Process each folio individually for_each_folio() {
Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio
}
- New approach: Process in batches
for_each_folio() {
Copy folio metadata like flags and mappings
}
Batch copy all src folios to dst
for_each_folio() {
Update page tables with dst folios
}
2. Multi-Threading:
- Distribute folio batch copy operations across multiple CPU threads.
3. DMA Offload:
- Leverage DMA engines designed for high copy throughput.
- Distribute folio batch-copy across mutliple DMA channels.
PERFORMANCE RESULTS:
-------------------
System Info:
Testing environment: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, Linux Kernel 6.14.0-rc7+, DVFS set to Performance, PTDMA hardware.
Measurement: Throughput (GB/s)
1. Varying folio-size with different parallel threads/channels:
Move different sized folios (mTHP - 4KB, 16KB,..., 2MB) such that total transfer size is constant (1GB), with different number of parallel threads/channels.
a. Multi-Threaded CPU
Folio Size--> Thread Cnt | 4K | 16K | 32K | 64K | 128K | 256K | 512K | 1M | 2M | =============================================================================================================== 1 | 1.72±0.05| 3.55±0.14| 4.44±0.07| 5.19±0.37| 5.57±0.47| 6.27±0.02 | 6.43±0.09 | 6.59±0.05 | 10.73±0.07| 2 | 1.93±0.06| 3.91±0.24| 5.22±0.03| 5.76±0.62| 7.42±0.16| 7.30±0.93 | 8.08±0.85 | 8.67±0.09 | 17.21±0.28| 4 | 2.00±0.03| 4.30±0.22| 6.02±0.10| 7.61±0.26| 8.60±0.92| 9.54±1.11 | 10.03±1.12| 10.98±0.14| 29.61±0.43| 8 | 2.07±0.08| 4.60±0.32| 6.06±0.85| 7.52±0.96| 7.98±1.83| 8.66±1.94 | 10.99±1.40| 11.22±1.49| 37.42±0.70| 16 | 2.04±0.04| 4.74±0.31| 6.20±0.39| 7.51±0.86| 8.26±1.47| 10.99±0.11| 9.72±1.51 | 12.07±0.02| 37.08±0.53|
b. DMA Offload
Folio Size--> Channel Cnt| 4K | 16K | 32K | 64K | 128K | 256K | 512K | 1M | 2M | ============================================================================================================ 1 | 0.46±0.01| 1.35±0.02| 1.99±0.02| 2.76±0.02| 3.44±0.17| 3.87±0.20| 3.98±0.29| 4.36±0.01| 11.79±0.05| 2 | 0.66±0.02| 1.84±0.07| 2.89±0.10| 4.02±0.30| 4.27±0.53| 5.98±0.05| 6.15±0.50| 5.83±0.64| 13.39±0.08| 4 | 0.91±0.01| 2.62±0.13| 3.98±0.17| 5.57±0.41| 6.55±0.70| 8.32±0.04| 8.91±0.05| 8.82±0.96| 24.52±0.22| 8 | 1.14±0.00| 3.21±0.07| 4.21±1.09| 6.07±0.81| 8.80±0.08| 8.91±1.38|11.03±0.02|10.68±1.38| 39.17±0.58| 16 | 1.19±0.11| 3.33±0.20| 4.98±0.33| 7.65±0.10| 7.85±1.50| 8.38±1.35| 8.94±3.23|12.85±0.06| 55.45±1.20|
Inference:
- Throughput increases with folio size. Higher Size folios benefit more from DMA.
- Multi-threading and DMA offloading both provide significant gains.
2. Varying folio count (total transfer size)
2MB folio-size, use only 1 thread
a. CPU Multi-Threaded
Folio Count| GB/s
======================
1 | 7.56±3.23 8 | 9.54±1.34 64 | 9.57±0.39 256 | 10.09±0.17 512 | 10.61±0.17 1024 | 10.77±0.07 2048 | 10.81±0.08 8192 | 10.84±0.05
b. DMA offload
Folio Count| GB/s
======================
1 | 8.21±3.68 8 | 9.92±2.12 64 | 9.90±0.31 256 | 11.51±0.32 512 | 11.67±0.11 1024 | 11.89±0.06 2048 | 11.92±0.08 8192 | 12.03±0.05
Inference:
- Throughput increase with folios count but plateaus after a threshold. (The migrate_pages function uses a folio batch size of 512)
3. CPU Threads scheduling
Analyze effect of CPU topology
a. Spread Across different CCDs
Threads | GB/s
========================
1 | 10.60±0.06 2 | 17.21±0.12 4 | 29.94±0.16 8 | 37.07±1.62 16 | 36.19±0.97
b. Fill one CCD completely before moving to next one
Threads | GB/s
========================
1 | 10.44±0.47 2 | 10.93±0.11 4 | 10.99±0.04 8 | 11.08±0.03 16 | 17.91±0.12
Inference:
- Hardware topology matters. On AMD systems, distributing copy threads across CCDs utilizes bandwidth better
TODOs:
We can further experiments to:
- Characterize system behavior and develop heuristics
- Analyze remote/local CPU scheduling impacts
- Measure DMA setup overheads
- Evaluate costs to userspace
- Study cache hotness/pollution effects
- DMA cost with different system I/O load
[1] https://lore.kernel.org/linux-mm/[email protected]
[2] https://lore.kernel.org/linux-mm/[email protected]
[3] LSFMM Proposal: https://lore.kernel.org/all/[email protected]
Mike Day (1):
mm: add support for copy offload for folio Migration
Shivank Garg (4):
mm: batch folio copying during migration
mm/migrate: add migrate_folios_batch_move to batch the folio move
operations
dcbm: add dma core batch migrator for batch page offloading
mtcopy: spread threads across die for testing
Zi Yan (4):
mm/migrate: factor out code in move_to_new_folio() and
migrate_folio_move()
mm/migrate: revive MIGRATE_NO_COPY in migrate_mode. mm/migrate: introduce multi-threaded page copy routine adjust NR_MAX_BATCHED_MIGRATION for testing
drivers/Kconfig | 2 + drivers/Makefile | 3 + drivers/migoffcopy/Kconfig | 17 ++ drivers/migoffcopy/Makefile | 2 + drivers/migoffcopy/dcbm/Makefile | 1 + drivers/migoffcopy/dcbm/dcbm.c | 393 ++++++++++++++++++++++++ drivers/migoffcopy/mtcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/copy_pages.c | 408 +++++++++++++++++++++++++ include/linux/migrate_mode.h | 2 + include/linux/migrate_offc.h | 36 +++ include/linux/mm.h | 4 + mm/Kconfig | 8 + mm/Makefile | 1 + mm/migrate.c | 351 ++++++++++++++++++--- mm/migrate_offc.c | 51 ++++ mm/util.c | 41 +++ 16 files changed, 1275 insertions(+), 46 deletions(-) create mode 100644 drivers/migoffcopy/Kconfig create mode 100644 drivers/migoffcopy/Makefile create mode 100644 drivers/migoffcopy/dcbm/Makefile create mode 100644 drivers/migoffcopy/dcbm/dcbm.c create mode 100644 drivers/migoffcopy/mtcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/copy_pages.c create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c
Signature · 1 line
2.34.1
RFC PATCH v2 1/9 mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move()
From: Zi Yan <[email protected]>
No function change is intended. The factored out code will be reused in an upcoming batched folio move function.
--- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-)
mm/migrate.c +65 -36
@@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_space *mapping,return migrate_folio (mapping, dst, src, mode); } -/* -* Move a page to a newly allocated page -* The page is locked and all ptes have been successfully removed. -* -* The new page will have replaced the old page if this function -* is successful. -* -* Return value: -* < 0 - error code -* MIGRATEPAGE_SUCCESS - success -*/ -static int move_to_new_folio (struct folio *dst,struct folio *src, +static int _move_to_new_folio_prep (struct folio *dst,struct folio *src,enum migrate_mode mode) {int rc = -EAGAIN ;
@@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,WARN_ON_ONCE (rc ==MIGRATEPAGE_SUCCESS && !folio_test_isolated (src)); } +out: +return rc; +} +static void _move_to_new_folio_finalize (struct folio *dst,struct folio *src, +int rc) +{/* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon.
@@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,if (likely (!folio_is_zone_device (dst)))flush_dcache_folio (dst); } -out: +} + + +/* +* Move a page to a newly allocated page +* The page is locked and all ptes have been successfully removed. +* +* The new page will have replaced the old page if this function +* is successful. +* +* Return value: +* < 0 - error code +* MIGRATEPAGE_SUCCESS - success +*/ +static int move_to_new_folio (struct folio *dst,struct folio *src, +enum migrate_mode mode) +{ +int rc; + + rc =_move_to_new_folio_prep (dst, src, mode); + +_move_to_new_folio_finalize (dst, src, rc); +return rc; }
@@ -1341,29 +1358,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,return rc; } -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move (free_folio_t put_new_folio,unsigned long private, -struct folio *src,struct folio *dst, -enum migrate_mode mode,enum migrate_reason reason, -struct list_head *ret) +static void _migrate_folio_move_finalize1 (struct folio *src,struct folio *dst, +int old_page_state) { -int rc; -int old_page_state =0 ; -struct anon_vma *anon_vma =NULL ; -bool is_lru = !__folio_test_movable (src); -struct list_head *prev; - -__migrate_folio_extract (dst, &old_page_state, &anon_vma); - prev = dst->lru.prev; -list_del (&dst->lru); - - rc =move_to_new_folio (dst, src, mode); -if (rc) - goto out; - -if (unlikely (!is_lru)) - goto out_unlock_both; -/* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will
@@ -1379,8 +1376,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +} -out_unlock_both: +static void _migrate_folio_move_finalize2 (struct folio *src,struct folio *dst, +enum migrate_reason reason, +struct anon_vma *anon_vma) +{folio_unlock (dst);set_page_owner_migrate_reason (&dst->page, reason);/*
@@ -1400,6 +1401,34 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,put_anon_vma (anon_vma);folio_unlock (src);migrate_folio_done (src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move (free_folio_t put_new_folio,unsigned long private, +struct folio *src,struct folio *dst, +enum migrate_mode mode,enum migrate_reason reason, +struct list_head *ret) +{ +int rc; +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; +bool is_lru = !__folio_test_movable (src); +struct list_head *prev; + +__migrate_folio_extract (dst, &old_page_state, &anon_vma); + prev = dst->lru.prev; +list_del (&dst->lru); + + rc =move_to_new_folio (dst, src, mode); +if (rc) + goto out; + +if (unlikely (!is_lru)) + goto out_unlock_both; + +_migrate_folio_move_finalize1 (src, dst, old_page_state); +out_unlock_both: +_migrate_folio_move_finalize2 (src, dst, reason, anon_vma);return rc; out:
Signature · 1 line
2.34.1
RFC PATCH v2 2/9 mm/migrate: revive MIGRATE_NO_COPY in migrate_mode.
In reply to Shivank GargFrom: Zi Yan <[email protected]>
It is a preparation patch. The added MIGRATE_NO_COPY will be used by the following patches to implement batched page copy functions by skipping folio copy process in __migrate_folio() and copying folios in one shot at the end.
--- include/linux/migrate_mode.h | 2 ++ mm/migrate.c | 8 +++++--- 2 files changed, 7 insertions(+), 3 deletions(-)
include/linux/migrate_mode.h +2 -0
@@ -7,11 +7,13 @@* on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages +* MIGRATE_NO_COPY will not copy page content */ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_NO_COPY , };enum migrate_reason {
mm/migrate.c +5 -3
@@ -757,9 +757,11 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,if (folio_ref_count (src) != expected_count)return -EAGAIN ; - rc =folio_mc_copy (dst, src); -if (unlikely (rc)) -return rc; +if (mode !=MIGRATE_NO_COPY ) { + rc =folio_mc_copy (dst, src); +if (unlikely (rc)) +return rc; + } rc =__folio_migrate_mapping (mapping, dst, src, expected_count);if (rc !=MIGRATEPAGE_SUCCESS )
Signature · 1 line
2.34.1
RFC PATCH v2 3/9 mm: batch folio copying during migration
In reply to Shivank GargIntroduce the folios_copy() and folios_mc_copy() to copy the folio content from the list of src folios to the list of dst folios.
This is preparatory patch for batch page migration offloading.
--- include/linux/mm.h | 4 ++++ mm/util.c | 41 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)
include/linux/mm.h +4 -0
@@ -1301,7 +1301,11 @@ void __folio_put(struct folio *folio);void split_page (struct page *page,unsigned int order);void folio_copy (struct folio *dst,struct folio *src); +void folios_copy (struct list_head *dst_list,struct list_head *src_list, +int __maybe_unused folios_cnt);int folio_mc_copy (struct folio *dst,struct folio *src); +int folios_mc_copy (struct list_head *dst_list,struct list_head *src_list, +int __maybe_unused folios_cnt);unsigned long nr_free_buffer_pages (void );
mm/util.c +41 -0
47 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v2 4/9 mm/migrate: add migrate_folios_batch_move to batch the folio move operations
In reply to Shivank GargThis is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It uses MIGRATE_NO_COPY to skip folio copy during metadata copy process and performed the copies in a batch later.
Currently, the folio move operation is performed individually for each folio in sequential manner:for_each_folio() {
Copy folio metadata like flags and mappings
Copy the folio content from src to dst
Update page tables with dst folio
}
With this patch, we transition to a batch processing approach as shown below:for_each_folio() {
Copy folio metadata like flags and mappings
}
Batch copy all src folios to dstfor_each_folio() {
Update page tables with dst folios
}
dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used.
[Zi Yan: Refactor the patch. Improved the original patch by removing the need for an extra mig_info allocation (for storing anon_vma and old page state). Instead, reuse src->private to store the data, making the implementation simpler and efficient.]
--- mm/migrate.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 6 deletions(-)
mm/migrate.c +198 -6
@@ -752,12 +752,15 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, enum migrate_mode mode) {int rc, expected_count =folio_expected_refs (mapping, src); +unsigned long dst_private = (unsigned long )dst->private;/* Check whether src does not have extra refs before we do more work */ if (folio_ref_count (src) != expected_count)return -EAGAIN ; -if (mode !=MIGRATE_NO_COPY ) { +if (mode ==MIGRATE_NO_COPY ) + dst->private =NULL ; +else { rc =folio_mc_copy (dst, src);if (unlikely (rc))return rc;
@@ -771,6 +774,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,folio_attach_private (dst,folio_detach_private (src));folio_migrate_flags (dst, src); + +if (mode ==MIGRATE_NO_COPY ) + src->private = (void *)dst_private; +return MIGRATEPAGE_SUCCESS ; }
@@ -1044,7 +1051,7 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, mode); else rc =fallback_migrate_folio (mapping, dst, src, mode); - }else { + }else if (mode !=MIGRATE_NO_COPY ) {const struct movable_operations *mops;/*
@@ -1062,7 +1069,8 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, rc = mops->migrate_page (&dst->page, &src->page, mode);WARN_ON_ONCE (rc ==MIGRATEPAGE_SUCCESS && !folio_test_isolated (src)); - } + }else + rc = -EAGAIN ; out:return rc; }
@@ -1140,7 +1148,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private = (void *)anon_vma + old_page_state; } -static void __migrate_folio_extract (struct folio *dst, +static void __migrate_folio_read (struct folio *dst,int *old_page_state,struct anon_vma **anon_vmap) {
@@ -1148,6 +1156,13 @@ static void __migrate_folio_extract(struct folio *dst, *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES ); *old_page_state = private &PAGE_OLD_STATES ; +} + +static void __migrate_folio_extract (struct folio *dst, +int *old_page_state, +struct anon_vma **anon_vmap) +{ +__migrate_folio_read (dst, old_page_state, anon_vmap); dst->private =NULL ; }
@@ -1770,6 +1785,178 @@ static void migrate_folios_move(struct list_head *src_folios, } } +static void migrate_folios_batch_move (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +enum migrate_mode mode,int reason, +struct list_head *ret_folios, +struct migrate_pages_stats *stats, +int *retry,int *thp_retry,int *nr_failed, +int *nr_retry_pages) +{ +struct folio *folio, *folio2, *dst, *dst2; +int rc, nr_pages =0 , nr_batched_folios =0 ; +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; +int is_thp =0 ; +LIST_HEAD (err_src); +LIST_HEAD (err_dst); + +if (mode !=MIGRATE_ASYNC ) { + *retry +=1 ; +return ; + } + +/* +* Iterate over the list of locked src/dst folios to copy the metadata +*/ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + +/* +* dst->private is not cleared here. It is cleared and moved to +* src->private in __migrate_folio(). +*/ +__migrate_folio_read (dst, &old_page_state, &anon_vma); + +/* +* Use MIGRATE_NO_COPY mode in migrate_folio family functions +* to copy the flags, mapping and some other ancillary information. +* This does everything except the page copy. The actual page copy +* is handled later in a batch manner. +*/ + rc =_move_to_new_folio_prep (dst, folio,MIGRATE_NO_COPY ); + +/* +* The rules are: +* Success: folio will be copied in batch +* -EAGAIN: move src/dst folios to tmp lists for +* non-batch retry +* Other errno: put src folio on ret_folios list, restore +* the dst folio +*/ +if (rc == -EAGAIN ) { + *retry +=1 ; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + +list_move_tail (&folio->lru, &err_src); +list_move_tail (&dst->lru, &err_dst); +__migrate_folio_record (dst, old_page_state, anon_vma); + }else if (rc !=MIGRATEPAGE_SUCCESS ) { + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + +list_del (&dst->lru); +migrate_folio_undo_src (folio, + old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +migrate_folio_undo_dst (dst, true, put_new_folio, private); + }else /* MIGRATEPAGE_SUCCESS */ + nr_batched_folios++; + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + +/* Exit if folio list for batch migration is empty */ +if (!nr_batched_folios) + goto out; + +/* Batch copy the folios */ + rc =folios_mc_copy (dst_folios, src_folios, nr_batched_folios); + +/* TODO: Is there a better way of handling the poison +* recover for batch copy, instead of falling back to serial copy? +*/ +/* fallback to serial page copy if needed */ +if (rc) { + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) && +folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + rc =folio_mc_copy (dst, folio); + +if (rc) { +/* +* dst->private is moved to src->private in +* __migrate_folio(), so page state and anon_vma +* values can be extracted from (src) folio. +*/ +__migrate_folio_extract (folio, &old_page_state, + &anon_vma); +migrate_folio_undo_src (folio, + old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +list_del (&dst->lru); +migrate_folio_undo_dst (dst, true, put_new_folio, + private); + } + +switch (rc) { +case MIGRATEPAGE_SUCCESS : + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; +break ; +default : + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; +break ; + } + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + } + +/* +* Iterate the folio lists to remove migration pte and restore them +* as working pte. Unlock the folios, add/remove them to LRU lists (if +* applicable) and release the src folios. +*/ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); +/* +* dst->private is moved to src->private in __migrate_folio(), +* so page state and anon_vma values can be extracted from +* (src) folio. +*/ +__migrate_folio_extract (folio, &old_page_state, &anon_vma); +list_del (&dst->lru); + +_move_to_new_folio_finalize (dst, folio,MIGRATEPAGE_SUCCESS ); + +/* +* Below few steps are only applicable for lru pages which is +* ensured as we have removed the non-lru pages from our list. +*/ +_migrate_folio_move_finalize1 (folio, dst, old_page_state); + +_migrate_folio_move_finalize2 (folio, dst, reason, anon_vma); + +/* Page migration successful, increase stat counter */ + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +out: +/* Add tmp folios back to the list to re-attempt migration. */ +list_splice (&err_src, src_folios); +list_splice (&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private,
@@ -1980,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *from,/* Flush TLBs for all unmapped folios */ try_to_unmap_flush (); - retry =1 ; + retry =0 ; +/* Batch move the unmapped folios */ +migrate_folios_batch_move (&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); +for (pass =0 ; pass < nr_pass && retry; pass++) { retry =0 ; thp_retry =0 ; nr_retry_pages =0 ; -/* Move the unmapped folios */ +/* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry,
Signature · 1 line
2.34.1
RFC PATCH v2 5/9 mm: add support for copy offload for folio Migration
In reply to Shivank GargFrom: Mike Day <[email protected]>
Offload-Copy drivers should implement following functions to enable folio migration offloading:migrate_offc() - This function takes src and dst folios list undergoing migration. It is responsible for transfer of page content between the src and dst folios.can_migrate_offc() - It performs necessary checks if offload copying migration is supported for the give src and dst folios.
Offload-Copy driver should include a mechanism to call start_offloading and stop_offloading for enabling and disabling migration offload respectively.
[Shivank: Rename the APIs and files to generalize the original DMA-specific offload implementation to support various copy offloading mechanisms such as DMA engines, CPU multi-threading, or other
hardware accelerators.]
--- include/linux/migrate_offc.h | 36 +++++++++++++++++++++++++ mm/Kconfig | 8 ++++++ mm/Makefile | 1 + mm/migrate.c | 43 ++++++++++++++++++++++++++++-- mm/migrate_offc.c | 51 ++++++++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c
include/linux/migrate_offc.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _MIGRATE_OFFC_H +#define _MIGRATE_OFFC_H +#include <linux/migrate_mode.h> + +#define MIGRATOR_NAME_LEN 32 +struct migrator { +char name[MIGRATOR_NAME_LEN ]; +int (*migrate_offc)(struct list_head *dst_list,struct list_head *src_list,int folio_cnt); +bool (*can_migrate_offc)(struct folio *dst,struct folio *src); +struct rcu_head srcu_head; +struct module *owner; +}; + +extern struct migrator migrator; +extern struct mutex migrator_mut; +extern struct srcu_struct mig_srcu; + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb (struct rcu_head *head); +void offc_update_migrator (struct migrator *mig); +unsigned char *get_active_migrator_name (void ); +bool can_offc_migrate (struct folio *dst,struct folio *src); +void start_offloading (struct migrator *migrator); +void stop_offloading (void ); +#else +static inline void srcu_mig_cb (struct rcu_head *head) { }; +static inline void offc_update_migrator (struct migrator *mig) { }; +static inline unsigned char *get_active_migrator_name (void ) {return NULL ; }; +static inline bool can_offc_migrate (struct folio *dst,struct folio *src) {return true; }; +static inline void start_offloading (struct migrator *migrator) { }; +static inline void stop_offloading (void ) { }; +#endif /* CONFIG_OFFC_MIGRATION */ + +#endif /* _MIGRATE_OFFC_H */
mm/Kconfig +8 -0
@@ -722,6 +722,14 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE +config OFFC_MIGRATION + bool "Migrate Pages offloading copy" + def_bool n + depends on MIGRATION + help + An interface allowing external modules or driver to offload + page copying in page migration. + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool
mm/Makefile +1 -0
@@ -93,6 +93,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_OFFC_MIGRATION) += migrate_offc.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
mm/migrate.c +41 -2
71 lines not on this page · raw message
mm/migrate_offc.c new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v2 6/9 mm/migrate: introduce multi-threaded page copy routine
In reply to Shivank GargFrom: Zi Yan <[email protected]>
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput.
Enable using:
echo 1 > /sys/kernel/cpu_mt/offloading
echo NR_THREADS > /sys/kernel/cpu_mt/threads
Disable:
echo 0 > /sys/kernel/cpu_mt/offloading
[Shivank: Convert the original MT copy_pages implementation into a module, leveraging migrate offload infrastructure and sysfs interface.]
--- drivers/Kconfig | 2 + drivers/Makefile | 3 + drivers/migoffcopy/Kconfig | 9 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/copy_pages.c | 337 +++++++++++++++++++++++++ mm/migrate.c | 11 +- 7 files changed, 357 insertions(+), 7 deletions(-) create mode 100644 drivers/migoffcopy/Kconfig create mode 100644 drivers/migoffcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/copy_pages.c
drivers/Kconfig +2 -0
@@ -245,4 +245,6 @@ source "drivers/cdx/Kconfig" source "drivers/dpll/Kconfig" +source "drivers/migoffcopy/Kconfig" + endmenu
drivers/Makefile +3 -0
@@ -42,6 +42,9 @@ obj-y += clk/ # really early. obj-$(CONFIG_DMADEVICES) += dma/ +# Migration copy Offload +obj-$(CONFIG_OFFC_MIGRATION) += migoffcopy/ + # SOC specific infrastructure drivers. obj-y += soc/ obj-$(CONFIG_PM_GENERIC_DOMAINS) += pmdomain/
drivers/migoffcopy/Kconfig new file +9 -0
@@ -0,0 +1,9 @@ +config MTCOPY_CPU + bool "Multi-Threaded Copy with CPU" + depends on OFFC_MIGRATION + default n + help + Interface MT COPY CPU driver for batch page migration + offloading. Say Y if you want to try offloading with + MultiThreaded CPU copy APIs. +
drivers/migoffcopy/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) += mtcopy/
drivers/migoffcopy/mtcopy/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) += copy_pages.o
drivers/migoffcopy/mtcopy/copy_pages.c new file +337 -0
337 lines not on this page · raw message
mm/migrate.c +4 -7
@@ -1831,18 +1831,13 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) {struct folio *folio, *folio2, *dst, *dst2; -int rc, nr_pages =0 , nr_batched_folios =0 ; +int rc, nr_pages =0 , total_nr_pages =0 , nr_batched_folios =0 ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;int is_thp =0 ;LIST_HEAD (err_src);LIST_HEAD (err_dst); -if (mode !=MIGRATE_ASYNC ) { - *retry +=1 ; -return ; - } -/* * Iterate over the list of locked src/dst folios to copy the metadata */
@@ -1892,8 +1887,10 @@ static void migrate_folios_batch_move(struct list_head *src_folios, old_page_state &PAGE_WAS_MAPPED , anon_vma, true, ret_folios);migrate_folio_undo_dst (dst, true, put_new_folio, private); - }else /* MIGRATEPAGE_SUCCESS */ + }else {/* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; nr_batched_folios++; + } dst = dst2; dst2 =list_next_entry (dst, lru);
Signature · 1 line
2.34.1
RFC PATCH v2 7/9 dcbm: add dma core batch migrator for batch page offloading
In reply to Shivank GargThe dcbm (DMA core batch migrator) provides a generic interface using DMAEngine for end-to-end testing of the batch page migration offload feature.
Enable DCBM offload:
echo 1 > /sys/kernel/dcbm/offloading
echo NR_DMA_CHAN_TO_USE > /sys/kernel/dcbm/nr_dma_chan
Disable DCBM offload:
echo 0 > /sys/kernel/dcbm/offloading
--- drivers/migoffcopy/Kconfig | 8 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/dcbm/Makefile | 1 + drivers/migoffcopy/dcbm/dcbm.c | 393 +++++++++++++++++++++++++++++++ 4 files changed, 403 insertions(+) create mode 100644 drivers/migoffcopy/dcbm/Makefile create mode 100644 drivers/migoffcopy/dcbm/dcbm.c
drivers/migoffcopy/Kconfig +8 -0
@@ -6,4 +6,12 @@ config MTCOPY_CPU Interface MT COPY CPU driver for batch page migration offloading. Say Y if you want to try offloading with MultiThreaded CPU copy APIs. +config DCBM_DMA + bool "DMA Core Batch Migrator" + depends on OFFC_MIGRATION && DMA_ENGINE + default n + help + Interface DMA driver for batch page migration offloading. + Say Y if you want to try offloading with DMAEngine APIs + based driver.
drivers/migoffcopy/Makefile +1 -0
@@ -1 +1,2 @@ obj-$(CONFIG_MTCOPY_CPU) += mtcopy/ +obj-$(CONFIG_DCBM_DMA) += dcbm/
drivers/migoffcopy/dcbm/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_DCBM_DMA) += dcbm.o
drivers/migoffcopy/dcbm/dcbm.c new file +393 -0
393 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v2 8/9 adjust NR_MAX_BATCHED_MIGRATION for testing
In reply to Shivank GargFrom: Zi Yan <[email protected]>
change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies.
These are for testing purpose only.
--- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
@@ -1653,7 +1653,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR#else #define NR_MAX_BATCHED_MIGRATION 512#endif
Signature · 1 line
2.34.1
RFC PATCH v2 9/9 mtcopy: spread threads across die for testing
In reply to Shivank GargSelect CPUs using sysfs
For testing purpose only.
--- drivers/migoffcopy/mtcopy/copy_pages.c | 77 +++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 3 deletions(-)
drivers/migoffcopy/mtcopy/copy_pages.c +74 -3
@@ -15,11 +15,37 @@#include <linux/migrate.h> #include <linux/migrate_offc.h> -#define MAX_NUM_COPY_THREADS 64 +#define MAX_NUM_COPY_THREADS 32unsigned int limit_mt_num =4 ;static int is_dispatching; +static int cpuselect =0 ; + +// spread across die +static const int cpu_id_list_0[] = + {0 ,8 ,16 ,24 , +32 ,40 ,48 ,56 , +64 ,72 ,80 ,88 , +96 ,104 ,112 ,120 , +128 ,136 ,144 ,152 , +160 ,168 ,176 ,184 , +192 ,200 ,208 ,216 , +224 ,232 ,240 ,248 }; + +// don't spread, fill the die +static const int cpu_id_list_1[] = + {0 ,1 ,2 ,3 , +4 ,5 ,6 ,7 , +8 ,9 ,10 ,11 , +12 ,13 ,14 ,15 , +16 ,17 ,18 ,19 , +20 ,21 ,22 ,23 , +24 ,25 ,26 ,27 , +28 ,29 ,30 ,31 }; + +int cpu_id_list[32 ] = {0 }; +static int copy_page_lists_mt (struct list_head *dst_folios,struct list_head *src_folios,int nr_items);static bool can_migrate_mt (struct folio *dst,struct folio *src);
@@ -143,6 +169,40 @@ static ssize_t mt_threads_show(struct kobject *kobj,return sysfs_emit (buf,"%u\n" , limit_mt_num); } +static ssize_t mt_cpuselect_set (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t count) +{ +int ccode; +unsigned int cpuconfig; + + ccode =kstrtouint (buf,0 , &cpuconfig); +if (ccode) { +pr_debug ("(%s:) error parsing input %s\n" , __func__, buf); +return ccode; + } +mutex_lock (&migratecfg_mutex); + cpuselect = cpuconfig; +switch (cpuselect) { +case 1 : +memcpy (cpu_id_list, cpu_id_list_1,MAX_NUM_COPY_THREADS *sizeof (int )); +break ; +default : +memcpy (cpu_id_list, cpu_id_list_0,MAX_NUM_COPY_THREADS *sizeof (int )); +break ; + } + +mutex_unlock (&migratecfg_mutex); + +return count; +} + + +static ssize_t mt_cpuselect_show (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%u\n" , cpuselect); +} +static bool can_migrate_mt (struct folio *dst,struct folio *src) {return true;
@@ -218,7 +278,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, }for (cpu =0 ; cpu < total_mt_num; ++cpu) -queue_work (system_unbound_wq, +queue_work_on (cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); }else {int num_xfer_per_thread = nr_items / total_mt_num;
@@ -255,7 +315,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, dst2 =list_next_entry (dst, lru);if (per_cpu_item_idx == work_items[cpu]->num_items) { -queue_work (system_unbound_wq, +queue_work_on (cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); per_cpu_item_idx =0 ; cpu++;
@@ -286,6 +346,8 @@ static struct kobj_attribute mt_offloading_attribute = __ATTR(offloading, 0664, mt_offloading_show, mt_offloading_set);static struct kobj_attribute mt_threads_attribute =__ATTR (threads,0664 , mt_threads_show, mt_threads_set); +static struct kobj_attribute mt_cpuselect_attribute =__ATTR (cpuselect,0664 , + mt_cpuselect_show, mt_cpuselect_set);static int __init cpu_mt_module_init (void ) {
@@ -303,10 +365,18 @@ static int __init cpu_mt_module_init(void)if (ret) goto out_threads; + ret =sysfs_create_file (mt_kobj_ref, &mt_cpuselect_attribute.attr); +if (ret) + goto out_cpuselect; + +memcpy (cpu_id_list, cpu_id_list_0,MAX_NUM_COPY_THREADS *sizeof (int )); + is_dispatching =0 ;return 0 ; +out_cpuselect: +sysfs_remove_file (mt_kobj_ref, &mt_threads_attribute.attr); out_threads:sysfs_remove_file (mt_kobj_ref, &mt_offloading_attribute.attr); out_offloading:
@@ -324,6 +394,7 @@ static void __exit cpu_mt_module_exit(void) }mutex_unlock (&migratecfg_mutex); +sysfs_remove_file (mt_kobj_ref, &mt_cpuselect_attribute.attr);sysfs_remove_file (mt_kobj_ref, &mt_threads_attribute.attr);sysfs_remove_file (mt_kobj_ref, &mt_offloading_attribute.attr);kobject_put (mt_kobj_ref);
Signature · 1 line
2.34.1