Skip to contentSkip to conversation

Accelerate page migration with batch copying and hardware offload

PATCH In review Shivank Garg 8 participants 67 messages Apr 28 – Jun 22

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover Accelerate page migration with batch copying and hardware offload

This is the fifth RFC of the patchset to enhance page migration by batching folio-copy operations and enabling acceleration via DMA offload.

Single-threaded, folio-by-folio copying bottlenecks page migration in modern systems with deep memory hierarchies, especially for large folios where copy overhead dominates, leaving significant hardware potential untapped.

By batching the copy phase, we create an opportunity for hardware acceleration. This series builds the framework and provides a DMA offload driver (dcbm) as a reference implementation, targeting bulk migration workloads where offloading the copy improves throughput and latency while freeing the CPU cycles.

See the RFC V3 cover letter [2] for motivation.

Changelog since V4:
-------------------

1. Renamed PAGE_* migration state flags to FOLIO_*. (David)
2. Use the new folio->migrate_info field instead of folio->private for migration state. (David)
3. Fold folios_mc_copy patch in batch-copy implementation patch. (David)
3. Renamed migrate_offload_start()/stop() to register()/unregister(). (Huang, Ying)
4. Dropped should_batch() callback from struct migrator. Reason-based policy now lives in migrate_pages_batch(). Migrators can still skip a batch they don't want (size based policy). (Huang, Ying)
5. CONFIG_MIGRATION_COPY_OFFLOAD is now hidden and selected by the migrator driver. CONFIG_DCBM_DMA is tristate. (Huang Ying, Gregory Price).
6. Wrapped the SRCU + static_call dispatch in a small helper. (Huang, Ying)
7. Requir m->owner in migrate_offload_register(), SRCU sync at unregister relies on it. Counters are atomic_long_t to avoid lock-order issue.
9. Moved DCBM sysfs from /sys/kernel/dcbm to /sys/module/dcbm (Huang, Ying)
10. Rebased on v7.1-rc1.

DESIGN:
-------

New Migration Flow:

[ migrate_pages_batch() ]

    |
    |--> do_batch = migrate_offload_do_batch(reason)  // core filters by migration reason
    |
    |--> for each folio:
    |      migrate_folio_unmap()        // unmap the folio
    |      |
    |      +--> (success):
    |           if do_batch && folio_supports_batch_copy():
    |               -> unmap_batch / dst_batch  // batch list for copy offloading
    |           else:
    |               -> unmap_single / dst_single // single lists for per-folio CPU copy
    |
    |--> try_to_unmap_flush()                   // single batched TLB flush
    |
    |--> Batch copy (if unmap_batch not empty):
    |    - Migrator is configurable at runtime via sysfs.
    |
    |      static_call(migrate_offload_copy)    // Pluggable Migrators
    |              /          |            \
    |             v           v             v
    |     [ Default ]  [ DMA Offload ]  [ ... ]
    |
    |      On -EOPNOTSUPP or other error, batch falls back to per-folio CPU copy.
    |

+--> migrate_folios_move() // metadata, update PTEs, finalize
(batch list with already_copied=true, single list with false)

Offload Registration:

Driver fills struct migrator { .name, .offload_copy, .owner } and calls migrate_offload_register(). This:
- Pins the module via try_module_get()
- Patches the migrate_offload_copy() static_call target
- Enables the migrate_offload_enabled static branch

migrate_offload_unregister() disables the static branch and reverts the static_call, then synchronize_srcu() waits for in-flight migrations before module_put().

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

PLAN:
-----

Patches 1-4 (the batching infrastructure) don't depend on the migrator interface, so if it helps I can split them off and post them ahead of the migrator and DCBM bits, which still have a few open questions to work through.

I would appreciate guidance on splitting the infrastructure portion ahead of the migrator interface if that matches maintainers' preference.

OPEN QUESTIONS:
---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

4. Cgroup integration: How should per-cgroup be accounted for different migrators (e.g.: any accounting for DMA-busy time)?

5. Tuning migrate_pages callers for offloading. For instance, in compaction COMPACT_CLUSTER_MAX = 32 caps DMA's payoff for compaction (V4 experiment).

6. Where do batch-size thresholds live, and how are they tuned? Per Huang Ying's split, that policy lives in the migrator. DCBM has no threshold today. Open whether it should later be a per-migrator sysfs knob or hard-coded; probably clearer once a second migrator (SDXI, mtcopy) shows the trade-off.

FOLLOW-UPS:
--------------

1. dmaengine_prep_dma_memcpy_sg() in DCBM (Vinod Koul). The SG-prep variant cuts per-batch prep/submit cost (=CPU savings), but ptdma does not implement the SG hook yet [10]. The end-to-end migration throughput delta is small because per-descriptor execute time dominates. I'll post the ptdma SG hook + DCBM switch as a follow-up.

2. SDXI as a second migrator. The SDXI series [11] is in review. SDXI is a generic memcpy engine without DMA_PRIVATE, so channel acquisition goes through dma_find_channel() or async_tx rather than dma_request_chan_by_mask(). I have a local DCBM variant working on top of the SDXI driver. I'm planning to send it as a follow-up once the SDXI series settles.

3. IOMMU SG merging in DCBM (Gregory). dma_map_sgtable() may merge contiguous PFNs unevenly, so src.nents != dst.nents. DCBM falls back to CPU for safety. Though I haven't seen it on Zen3 + PTDMA. I'll understand this and address it a follow-up.

4. Revisit Multi-threaded CPU copy migrator once the infra is settled.

EARLIER POSTINGS:
-----------------
[1] RFC V4: https://lore.kernel.org/all/[email protected]
[2] RFC V3: https://lore.kernel.org/all/[email protected]
[3] RFC V2: https://lore.kernel.org/all/[email protected]
[4] RFC V1: https://lore.kernel.org/all/[email protected]
[5] RFC from Zi Yan: https://lore.kernel.org/all/[email protected]

RELATED DISCUSSIONS:
--------------------
[6] MM-alignment Session [Nov 12, 2025]:

    https://lore.kernel.org/linux-mm/[email protected]
[7] Linux Memory Hotness and Promotion call [Nov 6, 2025]:
    https://lore.kernel.org/linux-mm/[email protected]
[8] LSFMM 2025:
    https://lore.kernel.org/all/[email protected]
[9] OSS India:
    https://ossindia2025.sched.com/event/23Jk1
[10] DMA_MEMCPY_SG comparison:
     https://lore.kernel.org/linux-mm/[email protected]
[11] SDXI V1:
     https://lore.kernel.org/all/[email protected]

Thanks to everyone who reviewed, tested or participated in discussions around this series. Your feedback helped me throughout the development process.

Best Regards,
Shivank

Shivank Garg (6):
mm/migrate: rename PAGE_ migration flags to FOLIO_ mm/migrate: use migrate_info field instead of private mm/migrate: skip data copy for already-copied folios mm/migrate: add batch-copy path in migrate_pages_batch mm/migrate: add copy offload registration infrastructure drivers/migrate_offload: add DMA batch copy driver (dcbm)

Zi Yan (1):
mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 440 ++++++++++++++++++++++++++
 include/linux/migrate_copy_offload.h  |  44 +++
 include/linux/mm.h                    |   2 +
 include/linux/mm_types.h              |   1 +
 mm/Kconfig                            |   6 +
 mm/Makefile                           |   1 +
 mm/migrate.c                          | 211 ++++++++----
 mm/migrate_copy_offload.c             |  94 ++++++
 mm/util.c                             |  30 ++
 14 files changed, 784 insertions(+), 60 deletions(-)
 create mode 100644 drivers/migrate_offload/Kconfig
 create mode 100644 drivers/migrate_offload/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/dcbm.c
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c

base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731

Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/7 mm/migrate: rename PAGE_ migration flags to FOLIO_

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

No functional change.

---
 mm/migrate.c | 48 +++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 25 deletions(-)
mm/migrate.c +23 -25
index 8a64291ab5b4..0c6a0ab6ecce 100644
@@ -1135,26 +1135,24 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
  * This is safe because nobody is using it except us.
  */
 enum {
-	PAGE_WAS_MAPPED = BIT(0),
-	PAGE_WAS_MLOCKED = BIT(1),
-	PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
 };
 
 static void __migrate_folio_record(struct folio *dst,
-				   int old_page_state,
-				   struct anon_vma *anon_vma)
+		int old_folio_state, struct anon_vma *anon_vma)
 {
-	dst->private = (void *)anon_vma + old_page_state;
+	dst->private = (void *)anon_vma + old_folio_state;
 }
 
 static void __migrate_folio_extract(struct folio *dst,
-				   int *old_page_state,
-				   struct anon_vma **anon_vmap)
+		int *old_folio_state, struct anon_vma **anon_vmap)
 {
 	unsigned long private = (unsigned long)dst->private;
 
-	*anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES);
-	*old_page_state = private & PAGE_OLD_STATES;
+	*anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
+	*old_folio_state = private & FOLIO_OLD_STATES;
 	dst->private = NULL;
 }
 
@@ -1209,7 +1207,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
-	int old_page_state = 0;
+	int old_folio_state = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool locked = false;
 	bool dst_locked = false;
@@ -1253,7 +1251,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	}
 	locked = true;
 	if (folio_test_mlocked(src))
-		old_page_state |= PAGE_WAS_MLOCKED;
+		old_folio_state |= FOLIO_WAS_MLOCKED;
 
 	if (folio_test_writeback(src)) {
 		/*
@@ -1302,7 +1300,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst_locked = true;
 
 	if (unlikely(page_has_movable_ops(&src->page))) {
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return 0;
 	}
 
@@ -1328,11 +1326,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
 		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
-		old_page_state |= PAGE_WAS_MAPPED;
+		old_folio_state |= FOLIO_WAS_MAPPED;
 	}
 
 	if (!folio_mapped(src)) {
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return 0;
 	}
 
@@ -1344,7 +1342,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	if (rc == -EAGAIN)
 		ret = NULL;
 
-	migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
+	migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
 			       anon_vma, locked, ret);
 	migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
 
@@ -1358,13 +1356,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct list_head *ret)
 {
 	int rc;
-	int old_page_state = 0;
+	int old_folio_state = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool src_deferred_split = false;
 	bool src_partially_mapped = false;
 	struct list_head *prev;
 
-	__migrate_folio_extract(dst, &old_page_state, &anon_vma);
+	__migrate_folio_extract(dst, &old_folio_state, &anon_vma);
 	prev = dst->lru.prev;
 	list_del(&dst->lru);
 
@@ -1404,10 +1402,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	 * isolated from the unevictable LRU: but this case is the easiest.
 	 */
 	folio_add_lru(dst);
-	if (old_page_state & PAGE_WAS_MLOCKED)
+	if (old_folio_state & FOLIO_WAS_MLOCKED)
 		lru_add_drain();
 
-	if (old_page_state & PAGE_WAS_MAPPED)
+	if (old_folio_state & FOLIO_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 
 out_unlock_both:
@@ -1439,11 +1437,11 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	 */
 	if (rc == -EAGAIN) {
 		list_add(&dst->lru, prev);
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return rc;
 	}
 
-	migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
+	migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
 			       anon_vma, true, ret);
 	migrate_folio_undo_dst(dst, true, put_new_folio, private);
 
@@ -1777,11 +1775,11 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	dst = list_first_entry(dst_folios, struct folio, lru);
 	dst2 = list_next_entry(dst, lru);
 	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
-		int old_page_state = 0;
+		int old_folio_state = 0;
 		struct anon_vma *anon_vma = NULL;
 
-		__migrate_folio_extract(dst, &old_page_state, &anon_vma);
-		migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
+		__migrate_folio_extract(dst, &old_folio_state, &anon_vma);
+		migrate_folio_undo_src(folio, old_folio_state & FOLIO_WAS_MAPPED,
 				anon_vma, true, ret_folios);
 		list_del(&dst->lru);
 		migrate_folio_undo_dst(dst, true, put_new_folio, private);
Signature · 1 line
2.43.0
reviewer Reviewed-by on v1 1/7
#2 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

Quoted 4 lines · Shivank Garg

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

Quoted 8 lines

No functional change.

LGTM, Thanks! Feel free to add my

in the future versions.

[snip]

---
Best Regards,
Huang, Ying

#3 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

In reply to Shivank Garg
Quoted 4 lines · On Tue, 28 Apr 2026 15:50:39 +0000 Shivank Garg <[email protected]> wrote:

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

Quoted 8 lines

No functional change.

Hi.

A few trivial things inline.

Thanks,

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1144

Quoted 10 lines
---
 mm/migrate.c | 48 +++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 25 deletions(-)
mm/migrate.c +1 -2
 
 static void __migrate_folio_record(struct folio *dst,
-				   int old_page_state,
-				   struct anon_vma *anon_vma)
+		int old_folio_state, struct anon_vma *anon_vma)

Trivial but why the formatting change?
This is all well under 80 chars and it makes the diff a tiny bit worse than it would otherwise be.

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1150

Quoted 9 lines

{

mm/migrate.c +2 -3
-	dst->private = (void *)anon_vma + old_page_state;
+	dst->private = (void *)anon_vma + old_folio_state;
 }
 
 static void __migrate_folio_extract(struct folio *dst,
-				   int *old_page_state,
-				   struct anon_vma **anon_vmap)
+		int *old_folio_state, struct anon_vma **anon_vmap)

Same as above - avoid the reformat.

Quoted 3 lines

{
unsigned long private = (unsigned long)dst->private;

reviewer
#4 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

Quoted 40 lines · Jonathan Cameron
Quoted 4 lines · On Tue, 28 Apr 2026 15:50:39 +0000 Shivank Garg <[email protected]> wrote:

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

Quoted 8 lines

No functional change.

Hi.

A few trivial things inline.

Thanks,

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1144

Quoted 10 lines
---
 mm/migrate.c | 48 +++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 25 deletions(-)
mm/migrate.c +1 -2
 
 static void __migrate_folio_record(struct folio *dst,
-				   int old_page_state,
-				   struct anon_vma *anon_vma)
+		int old_folio_state, struct anon_vma *anon_vma)

Trivial but why the formatting change?
This is all well under 80 chars and it makes the diff a tiny bit worse than it would otherwise be.

Quoted 5 lines

{

mm/migrate.c +1 -1
-	dst->private = (void *)anon_vma + old_page_state;
+	dst->private = (void *)anon_vma + old_folio_state;
 }
Quoted 6 lines

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1150

Quoted 4 lines

static void __migrate_folio_extract(struct folio *dst,

mm/migrate.c +1 -2
-				   int *old_page_state,
-				   struct anon_vma **anon_vmap)
+		int *old_folio_state, struct anon_vma **anon_vmap)

Same as above - avoid the reformat.

This format is preferred by mm and it is different from kernel’s clang-format.

Quoted 4 lines
Quoted 4 lines

{
unsigned long private = (unsigned long)dst->private;

Best Regards,
Yan, Zi

#5 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

In reply to Shivank Garg
Quoted 6 lines · Shivank Garg

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

No functional change.

Quoted 7 lines

---

migrate_folio_undo_src() has a parameter "page_was_mapped", change that too?

Quoted 150 lines
 mm/migrate.c | 48 +++++++++++++++++++++++-------------------------
 1 file changed, 23 insertions(+), 25 deletions(-)
mm/migrate.c +23 -25
index 8a64291ab5b4..0c6a0ab6ecce 100644
@@ -1135,26 +1135,24 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
  * This is safe because nobody is using it except us.
  */
 enum {
-	PAGE_WAS_MAPPED = BIT(0),
-	PAGE_WAS_MLOCKED = BIT(1),
-	PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
 };
 
 static void __migrate_folio_record(struct folio *dst,
-				   int old_page_state,
-				   struct anon_vma *anon_vma)
+		int old_folio_state, struct anon_vma *anon_vma)
 {
-	dst->private = (void *)anon_vma + old_page_state;
+	dst->private = (void *)anon_vma + old_folio_state;
 }
 
 static void __migrate_folio_extract(struct folio *dst,
-				   int *old_page_state,
-				   struct anon_vma **anon_vmap)
+		int *old_folio_state, struct anon_vma **anon_vmap)
 {
 	unsigned long private = (unsigned long)dst->private;
 
-	*anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES);
-	*old_page_state = private & PAGE_OLD_STATES;
+	*anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
+	*old_folio_state = private & FOLIO_OLD_STATES;
 	dst->private = NULL;
 }
 
@@ -1209,7 +1207,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
-	int old_page_state = 0;
+	int old_folio_state = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool locked = false;
 	bool dst_locked = false;
@@ -1253,7 +1251,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	}
 	locked = true;
 	if (folio_test_mlocked(src))
-		old_page_state |= PAGE_WAS_MLOCKED;
+		old_folio_state |= FOLIO_WAS_MLOCKED;
 
 	if (folio_test_writeback(src)) {
 		/*
@@ -1302,7 +1300,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst_locked = true;
 
 	if (unlikely(page_has_movable_ops(&src->page))) {
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return 0;
 	}
 
@@ -1328,11 +1326,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
 		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
-		old_page_state |= PAGE_WAS_MAPPED;
+		old_folio_state |= FOLIO_WAS_MAPPED;
 	}
 
 	if (!folio_mapped(src)) {
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return 0;
 	}
 
@@ -1344,7 +1342,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	if (rc == -EAGAIN)
 		ret = NULL;
 
-	migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
+	migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
 			       anon_vma, locked, ret);
 	migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
 
@@ -1358,13 +1356,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct list_head *ret)
 {
 	int rc;
-	int old_page_state = 0;
+	int old_folio_state = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool src_deferred_split = false;
 	bool src_partially_mapped = false;
 	struct list_head *prev;
 
-	__migrate_folio_extract(dst, &old_page_state, &anon_vma);
+	__migrate_folio_extract(dst, &old_folio_state, &anon_vma);
 	prev = dst->lru.prev;
 	list_del(&dst->lru);
 
@@ -1404,10 +1402,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	 * isolated from the unevictable LRU: but this case is the easiest.
 	 */
 	folio_add_lru(dst);
-	if (old_page_state & PAGE_WAS_MLOCKED)
+	if (old_folio_state & FOLIO_WAS_MLOCKED)
 		lru_add_drain();
 
-	if (old_page_state & PAGE_WAS_MAPPED)
+	if (old_folio_state & FOLIO_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 
 out_unlock_both:
@@ -1439,11 +1437,11 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	 */
 	if (rc == -EAGAIN) {
 		list_add(&dst->lru, prev);
-		__migrate_folio_record(dst, old_page_state, anon_vma);
+		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return rc;
 	}
 
-	migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
+	migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
 			       anon_vma, true, ret);
 	migrate_folio_undo_dst(dst, true, put_new_folio, private);
 
@@ -1777,11 +1775,11 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	dst = list_first_entry(dst_folios, struct folio, lru);
 	dst2 = list_next_entry(dst, lru);
 	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
-		int old_page_state = 0;
+		int old_folio_state = 0;
 		struct anon_vma *anon_vma = NULL;
 
-		__migrate_folio_extract(dst, &old_page_state, &anon_vma);
-		migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
+		__migrate_folio_extract(dst, &old_folio_state, &anon_vma);
+		migrate_folio_undo_src(folio, old_folio_state & FOLIO_WAS_MAPPED,
 				anon_vma, true, ret_folios);
 		list_del(&dst->lru);
 		migrate_folio_undo_dst(dst, true, put_new_folio, private);
#6 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

Quoted 17 lines · Dev Jain
Quoted 6 lines · Shivank Garg

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

No functional change.

Quoted 7 lines

---

Quoted 4 lines

migrate_folio_undo_src() has a parameter "page_was_mapped", change that too?

Thanks, Dev. Right.

This patch is already in mm-stable tree and it's fairly late in this cycle. So, not sure if the last minute changes are okay for this?

Given this, I'll leave the timing to the maintainers:
I can fold this and resend now, or send it early next cycle?

Best Regards,
Shivank

---
 mm/migrate.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
mm/migrate.c +3 -6
index 0c6a0ab6ecce..fd524f9e6fae 100644
@@ -1157,13 +1157,10 @@ static void __migrate_folio_extract(struct folio *dst,
 }
 
 /* Restore the source folio to the original state upon failure */
-static void migrate_folio_undo_src(struct folio *src,
-				   int page_was_mapped,
-				   struct anon_vma *anon_vma,
-				   bool locked,
-				   struct list_head *ret)
+static void migrate_folio_undo_src(struct folio *src, int folio_was_mapped,
+		struct anon_vma *anon_vma, bool locked, struct list_head *ret)
 {
-	if (page_was_mapped)
+	if (folio_was_mapped)
 		remove_migration_ptes(src, src, 0);
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH 1/7] mm/migrate: rename PAGE_ migration flags to FOLIO_

Quoted 25 lines · Shivank
Quoted 17 lines · Dev Jain
Quoted 6 lines · Shivank Garg

These flags only track folio-specific state during migration and are not used for movable_ops pages. Rename the enum values and the old_page_state variable to match.

No functional change.

Quoted 7 lines

---

Quoted 4 lines

migrate_folio_undo_src() has a parameter "page_was_mapped", change that too?

Quoted 4 lines

Thanks, Dev. Right.

This patch is already in mm-stable tree and it's fairly late in this cycle. So, not sure if the last minute changes are okay for this?

Oh I didn't realize this is already in. No worries then, it can be sent later too.

Quoted 31 lines

Given this, I'll leave the timing to the maintainers:
I can fold this and resend now, or send it early next cycle?

Best Regards,
Shivank

---
 mm/migrate.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
mm/migrate.c +3 -6
index 0c6a0ab6ecce..fd524f9e6fae 100644
@@ -1157,13 +1157,10 @@ static void __migrate_folio_extract(struct folio *dst,
 }
 
 /* Restore the source folio to the original state upon failure */
-static void migrate_folio_undo_src(struct folio *src,
-				   int page_was_mapped,
-				   struct anon_vma *anon_vma,
-				   bool locked,
-				   struct list_head *ret)
+static void migrate_folio_undo_src(struct folio *src, int folio_was_mapped,
+		struct anon_vma *anon_vma, bool locked, struct list_head *ret)
 {
-	if (page_was_mapped)
+	if (folio_was_mapped)
 		remove_migration_ptes(src, src, 0);
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
#8 lore raw

PATCH 2/7 mm/migrate: use migrate_info field instead of private

In reply to Shivank Garg

Add an unsigned long migrate_info member to the struct folio union and use it to store migration state (anon_vma pointer and FOLIO_WAS_* flags) instead of using folio->private.

No functional change.

---
 include/linux/mm_types.h |  1 +
 mm/migrate.c             | 14 +++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)
include/linux/mm_types.h +1 -0
index a308e2c23b82..f52818dcf4d2 100644
@@ -426,6 +426,7 @@ struct folio {
 			union {
 				void *private;
 				swp_entry_t swap;
+				unsigned long migrate_info;
 			};
 			atomic_t _mapcount;
 			atomic_t _refcount;
mm/migrate.c +7 -7
index 0c6a0ab6ecce..03c2a6f7e5e4 100644
@@ -1130,7 +1130,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 }
 
 /*
- * To record some information during migration, we use unused private
+ * To record some information during migration, we use the migrate_info
  * field of struct folio of the newly allocated destination folio.
  * This is safe because nobody is using it except us.
  */
@@ -1143,17 +1143,17 @@ enum {
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
-	dst->private = (void *)anon_vma + old_folio_state;
+	dst->migrate_info = (unsigned long)anon_vma | old_folio_state;
 }
 
 static void __migrate_folio_extract(struct folio *dst,
 		int *old_folio_state, struct anon_vma **anon_vmap)
 {
-	unsigned long private = (unsigned long)dst->private;
+	unsigned long info = dst->migrate_info;
 
-	*anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
-	*old_folio_state = private & FOLIO_OLD_STATES;
-	dst->private = NULL;
+	*anon_vmap = (struct anon_vma *)(info & ~FOLIO_OLD_STATES);
+	*old_folio_state = info & FOLIO_OLD_STATES;
+	dst->migrate_info = 0;
 }
 
 /* Restore the source folio to the original state upon failure */
@@ -1217,7 +1217,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		return -ENOMEM;
 	*dstp = dst;
 
-	dst->private = NULL;
+	dst->migrate_info = 0;
 
 	if (!folio_trylock(src)) {
 		if (mode == MIGRATE_ASYNC)
Signature · 1 line
2.43.0
reviewer Reviewed-by on v1 2/7
#9 lore raw

Re: [PATCH 2/7] mm/migrate: use migrate_info field instead of private

Quoted 62 lines · Shivank Garg

Add an unsigned long migrate_info member to the struct folio union and use it to store migration state (anon_vma pointer and FOLIO_WAS_* flags) instead of using folio->private.

No functional change.

---
 include/linux/mm_types.h |  1 +
 mm/migrate.c             | 14 +++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)
include/linux/mm_types.h +1 -0
index a308e2c23b82..f52818dcf4d2 100644
@@ -426,6 +426,7 @@ struct folio {
 			union {
 				void *private;
 				swp_entry_t swap;
+				unsigned long migrate_info;
 			};
 			atomic_t _mapcount;
 			atomic_t _refcount;
mm/migrate.c +6 -6
index 0c6a0ab6ecce..03c2a6f7e5e4 100644
@@ -1130,7 +1130,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 }
 
 /*
- * To record some information during migration, we use unused private
+ * To record some information during migration, we use the migrate_info
  * field of struct folio of the newly allocated destination folio.
  * This is safe because nobody is using it except us.
  */
@@ -1143,17 +1143,17 @@ enum {
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
-	dst->private = (void *)anon_vma + old_folio_state;
+	dst->migrate_info = (unsigned long)anon_vma | old_folio_state;
 }
 
 static void __migrate_folio_extract(struct folio *dst,
 		int *old_folio_state, struct anon_vma **anon_vmap)
 {
-	unsigned long private = (unsigned long)dst->private;
+	unsigned long info = dst->migrate_info;
 
-	*anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
-	*old_folio_state = private & FOLIO_OLD_STATES;
-	dst->private = NULL;
+	*anon_vmap = (struct anon_vma *)(info & ~FOLIO_OLD_STATES);
+	*old_folio_state = info & FOLIO_OLD_STATES;
+	dst->migrate_info = 0;
 }
 
 /* Restore the source folio to the original state upon failure */

mm/migrate.c static int migrate_folio_unmap(new_folio_t get_new_folio, · line 1223

Quoted 8 lines
mm/migrate.c +1 -1
@@ -1217,7 +1217,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		return -ENOMEM;
 	*dstp = dst;
 
-	dst->private = NULL;
+	dst->migrate_info = 0;
 
 	if (!folio_trylock(src)) {
 		if (mode == MIGRATE_ASYNC)

LGTM! Feel free to add my

in the future versions.

---
Best Regards,
Huang, Ying

maintainer Acked-by on v1 2/7
#10 lore raw

Re: [PATCH 2/7] mm/migrate: use migrate_info field instead of private

In reply to Shivank Garg
Quoted 9 lines · Shivank Garg

Add an unsigned long migrate_info member to the struct folio union and use it to store migration state (anon_vma pointer and FOLIO_WAS_* flags) instead of using folio->private.

No functional change.

---

Signature · 3 lines
Cheers,

David
Reviewed-by on v1 2/7
#11 lore raw

Re: [PATCH 2/7] mm/migrate: use migrate_info field instead of private

In reply to Shivank Garg
Quoted 8 lines · On Tue, 28 Apr 2026 15:50:41 +0000 Shivank Garg <[email protected]> wrote:

Add an unsigned long migrate_info member to the struct folio union and use it to store migration state (anon_vma pointer and FOLIO_WAS_* flags) instead of using folio->private.

No functional change.

Nice to get rid of the hammering of integers into a pointer.

#12 lore raw

PATCH 3/7 mm/migrate: skip data copy for already-copied folios

In reply to Shivank Garg

Add a FOLIO_ALREADY_COPIED flag to the dst->migrate_info migration state. When set, __migrate_folio() skips folio_mc_copy() and performs metadata-only migration. All callers currently pass already_copied=false. The batch-copy path enables it later in a subsequent patch.

Move the dst->migrate_info state enum earlier in the file so __migrate_folio() and move_to_new_folio() can see FOLIO_ALREADY_COPIED.

---
 mm/migrate.c | 53 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 21 deletions(-)
mm/migrate.c +32 -21
index 03c2a6f7e5e4..c493e67e359d 100644
@@ -850,6 +850,19 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 }
 EXPORT_SYMBOL(folio_migrate_flags);
 
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio.
+ * This is safe because nobody is using it except us.
+ */
+enum {
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_ALREADY_COPIED = BIT(2),
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED |
+			   FOLIO_ALREADY_COPIED,
+};
+
 /************************************************************
  *                    Migration functions
  ***********************************************************/
@@ -859,14 +872,20 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   enum migrate_mode mode)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
+	bool already_copied = (dst->migrate_info & FOLIO_ALREADY_COPIED);
+
+	if (already_copied)
+		dst->migrate_info = 0;
 
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (!already_copied) {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -1090,7 +1109,7 @@ static int fallback_migrate_folio(struct address_space *mapping,
  *     0 - success
  */
 static int move_to_new_folio(struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+		enum migrate_mode mode, bool already_copied)
 {
 	struct address_space *mapping = folio_mapping(src);
 	int rc = -EAGAIN;
@@ -1098,6 +1117,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
+	if (already_copied)
+		dst->migrate_info = FOLIO_ALREADY_COPIED;
+
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
 	else if (mapping_inaccessible(mapping))
@@ -1129,17 +1151,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
-	FOLIO_WAS_MAPPED = BIT(0),
-	FOLIO_WAS_MLOCKED = BIT(1),
-	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
@@ -1353,7 +1364,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct folio *src, struct folio *dst,
 			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
+			      struct list_head *ret, bool already_copied)
 {
 	int rc;
 	int old_folio_state = 0;
@@ -1379,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		src_partially_mapped = folio_test_partially_mapped(src);
 	}
 
-	rc = move_to_new_folio(dst, src, mode);
+	rc = move_to_new_folio(dst, src, mode, already_copied);
 	if (rc)
 		goto out;
 
@@ -1536,7 +1547,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 	}
 
 	if (!folio_mapped(src))
-		rc = move_to_new_folio(dst, src, mode);
+		rc = move_to_new_folio(dst, src, mode, false);
 
 	if (page_was_mapped)
 		remove_migration_ptes(src, !rc ? dst : src, ttu);
@@ -1720,7 +1731,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats,
 		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		int *nr_retry_pages, bool already_copied)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
 	bool is_thp;
@@ -1737,7 +1748,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 
 		rc = migrate_folio_move(put_new_folio, private,
 				folio, dst, mode,
-				reason, ret_folios);
+				reason, ret_folios, already_copied);
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
@@ -1994,7 +2005,7 @@ static int migrate_pages_batch(struct list_head *from,
 		migrate_folios_move(&unmap_folios, &dst_folios,
 				put_new_folio, private, mode, reason,
 				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
+				&nr_failed, &nr_retry_pages, false);
 	}
 	nr_failed += retry;
 	stats->nr_thp_failed += thp_retry;
Signature · 1 line
2.43.0
#13 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 21 lines · Shivank Garg

Add a FOLIO_ALREADY_COPIED flag to the dst->migrate_info migration state. When set, __migrate_folio() skips folio_mc_copy() and performs metadata-only migration. All callers currently pass already_copied=false. The batch-copy path enables it later in a subsequent patch.

Move the dst->migrate_info state enum earlier in the file so __migrate_folio() and move_to_new_folio() can see FOLIO_ALREADY_COPIED.

---
 mm/migrate.c | 53 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 21 deletions(-)
mm/migrate.c +0 -0
index 03c2a6f7e5e4..c493e67e359d 100644
@@ -850,6 +850,19 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 }
 EXPORT_SYMBOL(folio_migrate_flags);

mm/migrate.c void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · line 861

Quoted 10 lines
mm/migrate.c +9 -0
 
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio.
+ * This is safe because nobody is using it except us.
+ */
+enum {
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_ALREADY_COPIED = BIT(2),

I wonder whether we want to talk about "folio content copied", to not confuse it with folio flags copied etc.

FOLIO_CONTENT_COPIED.

Thoughts?

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 875

Quoted 12 lines
mm/migrate.c +5 -0
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED |
+			   FOLIO_ALREADY_COPIED,
+};
+
 /************************************************************
  *                    Migration functions
  ***********************************************************/
@@ -859,14 +872,20 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   enum migrate_mode mode)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
+	bool already_copied = (dst->migrate_info & FOLIO_ALREADY_COPIED);

const, and no need for ().

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 878

Quoted 3 lines
mm/migrate.c +3 -0
+
+	if (already_copied)
+		dst->migrate_info = 0;

Hm, why is that required? Might deserve a comment.

Likely you want to clear the "already copied" marker?

	dst->migrate_info &= ~FOLIO_ALREADY_COPIED;

?

But I wonder if this really belongs exactly here.

Quoted 26 lines
mm/migrate.c +6 -4
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (!already_copied) {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -1090,7 +1109,7 @@ static int fallback_migrate_folio(struct address_space *mapping,
  *     0 - success
  */
 static int move_to_new_folio(struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+		enum migrate_mode mode, bool already_copied)
 {
 	struct address_space *mapping = folio_mapping(src);
 	int rc = -EAGAIN;

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1121

Quoted 5 lines
mm/migrate.c +2 -0
@@ -1098,6 +1117,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
+	if (already_copied)
+		dst->migrate_info = FOLIO_ALREADY_COPIED;

|= ?

Quoted 41 lines
mm/migrate.c +3 -13
+
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
 	else if (mapping_inaccessible(mapping))
@@ -1129,17 +1151,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
-	FOLIO_WAS_MAPPED = BIT(0),
-	FOLIO_WAS_MLOCKED = BIT(1),
-	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
@@ -1353,7 +1364,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct folio *src, struct folio *dst,
 			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
+			      struct list_head *ret, bool already_copied)
 {
 	int rc;
 	int old_folio_state = 0;
@@ -1379,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		src_partially_mapped = folio_test_partially_mapped(src);
 	}
 
-	rc = move_to_new_folio(dst, src, mode);
+	rc = move_to_new_folio(dst, src, mode, already_copied);
 	if (rc)
 		goto out;
 

mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1550

Quoted 5 lines
mm/migrate.c +1 -1
@@ -1536,7 +1547,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 	}
 
 	if (!folio_mapped(src))
-		rc = move_to_new_folio(dst, src, mode);
+		rc = move_to_new_folio(dst, src, mode, false);

... mode, /* already_copied = */ false

Quoted 22 lines
mm/migrate.c +2 -2
 	if (page_was_mapped)
 		remove_migration_ptes(src, !rc ? dst : src, ttu);
@@ -1720,7 +1731,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats,
 		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		int *nr_retry_pages, bool already_copied)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
 	bool is_thp;
@@ -1737,7 +1748,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 
 		rc = migrate_folio_move(put_new_folio, private,
 				folio, dst, mode,
-				reason, ret_folios);
+				reason, ret_folios, already_copied);
 		/*
 		 * The rules are:
 		 *	0: folio will be freed

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2009

Quoted 6 lines
mm/migrate.c +1 -1
@@ -1994,7 +2005,7 @@ static int migrate_pages_batch(struct list_head *from,
 		migrate_folios_move(&unmap_folios, &dst_folios,
 				put_new_folio, private, mode, reason,
 				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
+				&nr_failed, &nr_retry_pages, false);
 	}

dito.

Signature · 3 lines
Cheers,

David
#14 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 36 lines · David Hildenbrand (Arm)
Quoted 21 lines · Shivank Garg

Add a FOLIO_ALREADY_COPIED flag to the dst->migrate_info migration state. When set, __migrate_folio() skips folio_mc_copy() and performs metadata-only migration. All callers currently pass already_copied=false. The batch-copy path enables it later in a subsequent patch.

Move the dst->migrate_info state enum earlier in the file so __migrate_folio() and move_to_new_folio() can see FOLIO_ALREADY_COPIED.

---
 mm/migrate.c | 53 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 21 deletions(-)
mm/migrate.c +0 -0
index 03c2a6f7e5e4..c493e67e359d 100644
@@ -850,6 +850,19 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 }
 EXPORT_SYMBOL(folio_migrate_flags);

mm/migrate.c void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · line 861

Quoted 10 lines
mm/migrate.c +9 -0
 
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio.
+ * This is safe because nobody is using it except us.
+ */
+enum {
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_ALREADY_COPIED = BIT(2),

I wonder whether we want to talk about "folio content copied", to not confuse it with folio flags copied etc.

Quoted 3 lines

FOLIO_CONTENT_COPIED.

Thoughts?

Good point, this is more precise.

Quoted 6 lines
Quoted 6 lines
mm/migrate.c +4 -0
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED |
+			   FOLIO_ALREADY_COPIED,
+};
+
 /************************************************************
  *                    Migration functions
Quoted 8 lines

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 875

Quoted 6 lines
mm/migrate.c +1 -0
  ***********************************************************/
@@ -859,14 +872,20 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   enum migrate_mode mode)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
+	bool already_copied = (dst->migrate_info & FOLIO_ALREADY_COPIED);

const, and no need for ().

Will fix.

Quoted 11 lines

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 878

Quoted 3 lines
mm/migrate.c +3 -0
+
+	if (already_copied)
+		dst->migrate_info = 0;

Hm, why is that required? Might deserve a comment.

Likely you want to clear the "already copied" marker?

	dst->migrate_info &= ~FOLIO_ALREADY_COPIED;

?

Yes,
In current code, migrate_info only carry FOLIO_ALREADY_COPIED. So, =0 and &=~ are equivalent and similar for |= case. I mirrored the how anon_vma/old_folio_state get handled __migrate_folio_extract().

But with your other suggestion of marking already_copied immediately after folio_mc_copy(), then bitwise clear and set will be required. I'll do this.

Quoted 2 lines

But I wonder if this really belongs exactly here.

I did not understand this.

Quoted 29 lines
Quoted 29 lines
mm/migrate.c +6 -4
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (!already_copied) {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -1090,7 +1109,7 @@ static int fallback_migrate_folio(struct address_space *mapping,
  *     0 - success
  */
 static int move_to_new_folio(struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+		enum migrate_mode mode, bool already_copied)
 {
 	struct address_space *mapping = folio_mapping(src);
 	int rc = -EAGAIN;
@@ -1098,6 +1117,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
Quoted 5 lines

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1121

Quoted 2 lines
mm/migrate.c +2 -0
+	if (already_copied)
+		dst->migrate_info = FOLIO_ALREADY_COPIED;

|= ?

Will do.

Quoted 40 lines
Quoted 40 lines
mm/migrate.c +3 -13
+
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
 	else if (mapping_inaccessible(mapping))
@@ -1129,17 +1151,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
-	FOLIO_WAS_MAPPED = BIT(0),
-	FOLIO_WAS_MLOCKED = BIT(1),
-	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
@@ -1353,7 +1364,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct folio *src, struct folio *dst,
 			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
+			      struct list_head *ret, bool already_copied)
 {
 	int rc;
 	int old_folio_state = 0;
@@ -1379,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		src_partially_mapped = folio_test_partially_mapped(src);
 	}
 
-	rc = move_to_new_folio(dst, src, mode);
+	rc = move_to_new_folio(dst, src, mode, already_copied);
 	if (rc)
 		goto out;
 
Quoted 9 lines

mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1550

Quoted 6 lines
mm/migrate.c +1 -1
@@ -1536,7 +1547,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 	}
 
 	if (!folio_mapped(src))
-		rc = move_to_new_folio(dst, src, mode);
+		rc = move_to_new_folio(dst, src, mode, false);

... mode, /* already_copied = */ false

will fix

Quoted 14 lines
Quoted 14 lines
mm/migrate.c +1 -1
 	if (page_was_mapped)
 		remove_migration_ptes(src, !rc ? dst : src, ttu);
@@ -1720,7 +1731,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats,
 		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		int *nr_retry_pages, bool already_copied)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
 	bool is_thp;
@@ -1737,7 +1748,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 
Quoted 17 lines
Quoted 8 lines
mm/migrate.c +1 -1
 		rc = migrate_folio_move(put_new_folio, private,
 				folio, dst, mode,
-				reason, ret_folios);
+				reason, ret_folios, already_copied);
 		/*
 		 * The rules are:
 		 *	0: folio will be freed

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2009

Quoted 6 lines
mm/migrate.c +1 -1
@@ -1994,7 +2005,7 @@ static int migrate_pages_batch(struct list_head *from,
 		migrate_folios_move(&unmap_folios, &dst_folios,
 				put_new_folio, private, mode, reason,
 				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
+				&nr_failed, &nr_retry_pages, false);
 	}

dito.

will fix.

Thanks,
Shivank

#15 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 39 lines · Shivank
Quoted 36 lines · David Hildenbrand (Arm)
Quoted 21 lines · Shivank Garg

Add a FOLIO_ALREADY_COPIED flag to the dst->migrate_info migration state. When set, __migrate_folio() skips folio_mc_copy() and performs metadata-only migration. All callers currently pass already_copied=false. The batch-copy path enables it later in a subsequent patch.

Move the dst->migrate_info state enum earlier in the file so __migrate_folio() and move_to_new_folio() can see FOLIO_ALREADY_COPIED.

---
 mm/migrate.c | 53 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 21 deletions(-)
mm/migrate.c +0 -0
index 03c2a6f7e5e4..c493e67e359d 100644
@@ -850,6 +850,19 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
 }
 EXPORT_SYMBOL(folio_migrate_flags);

mm/migrate.c void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · line 861

Quoted 10 lines
mm/migrate.c +9 -0
 
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio.
+ * This is safe because nobody is using it except us.
+ */
+enum {
+	FOLIO_WAS_MAPPED = BIT(0),
+	FOLIO_WAS_MLOCKED = BIT(1),
+	FOLIO_ALREADY_COPIED = BIT(2),

I wonder whether we want to talk about "folio content copied", to not confuse it with folio flags copied etc.

Quoted 6 lines
Quoted 3 lines

FOLIO_CONTENT_COPIED.

Thoughts?

Good point, this is more precise.

Sashiko Comment:
Does expanding FOLIO_OLD_STATES to include BIT(2) silently corrupt anon_vma pointers on 32-bit architectures?
The anon_vma pointer is packed with FOLIO_OLD_STATES inside dst->migrate_info. To separate them, __migrate_folio_extract() uses info & ~FOLIO_OLD_STATES. Since anon_vma objects are allocated from a kmem_cache with align=0, ARCH_SLAB_MINALIGN defaults to 4 bytes on many 32-bit architectures. This means a valid anon_vma pointer can legitimately end in 0x4 or 0xC (meaning bit 2 is set).
When __migrate_folio_extract() masks the pointer with ~7, it will silently clear bit 2 from the anon_vma pointer. Any subsequent call to put_anon_vma() with this corrupted pointer could cause a use-after-free or a kernel panic. --

#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)

I initially assumed this to be always 8, confusing it to be same as size of unsigned long long.
But the GCC docs note that alignment can be smaller in size:

https://gcc.gnu.org/onlinedocs/gcc/Alignment.html
"For example, if the target machine requires a double value to be aligned on an 8-byte boundary, then __alignof__ (double) is 8. This is true on many RISC machines. On more traditional machine designs, __alignof__ (double) is 4 or even 2."

If my understanding is right, Sashiko concern is valid, and I can't safely use BIT(2).
I see few option from here. Either I can gate batch copy for CONFIG_64BIT, or I can force anon_vma_cachep to 8-byte alignment via kmem_cache_create() align arg. Or I can change the migrate_folio() callback to pass already_copied info to change to dst->migrate_info enum.

I'm not sure if touching the anon_vma cache would be preferred? Gating the change to CONFIG_64BIT feels less risky to me.

Thanks,
Shivank

Quoted 171 lines
Quoted 6 lines
Quoted 6 lines
mm/migrate.c +4 -0
+	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED |
+			   FOLIO_ALREADY_COPIED,
+};
+
 /************************************************************
  *                    Migration functions
Quoted 8 lines

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 875

Quoted 6 lines
mm/migrate.c +1 -0
  ***********************************************************/
@@ -859,14 +872,20 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   enum migrate_mode mode)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
+	bool already_copied = (dst->migrate_info & FOLIO_ALREADY_COPIED);

const, and no need for ().

Will fix.

Quoted 11 lines

mm/migrate.c static int __migrate_folio(struct address_space *mapping, struct folio *dst, · line 878

Quoted 3 lines
mm/migrate.c +3 -0
+
+	if (already_copied)
+		dst->migrate_info = 0;

Hm, why is that required? Might deserve a comment.

Likely you want to clear the "already copied" marker?

	dst->migrate_info &= ~FOLIO_ALREADY_COPIED;

?

Yes,
In current code, migrate_info only carry FOLIO_ALREADY_COPIED. So, =0 and &=~ are equivalent and similar for |= case. I mirrored the how anon_vma/old_folio_state get handled __migrate_folio_extract().

But with your other suggestion of marking already_copied immediately after folio_mc_copy(), then bitwise clear and set will be required. I'll do this.

Quoted 2 lines

But I wonder if this really belongs exactly here.

I did not understand this.

Quoted 29 lines
Quoted 29 lines
mm/migrate.c +6 -4
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (!already_copied) {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -1090,7 +1109,7 @@ static int fallback_migrate_folio(struct address_space *mapping,
  *     0 - success
  */
 static int move_to_new_folio(struct folio *dst, struct folio *src,
-				enum migrate_mode mode)
+		enum migrate_mode mode, bool already_copied)
 {
 	struct address_space *mapping = folio_mapping(src);
 	int rc = -EAGAIN;
@@ -1098,6 +1117,9 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
 	VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
 
Quoted 5 lines

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 1121

Quoted 2 lines
mm/migrate.c +2 -0
+	if (already_copied)
+		dst->migrate_info = FOLIO_ALREADY_COPIED;

|= ?

Will do.

Quoted 40 lines
Quoted 40 lines
mm/migrate.c +3 -13
+
 	if (!mapping)
 		rc = migrate_folio(mapping, dst, src, mode);
 	else if (mapping_inaccessible(mapping))
@@ -1129,17 +1151,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
-	FOLIO_WAS_MAPPED = BIT(0),
-	FOLIO_WAS_MLOCKED = BIT(1),
-	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
@@ -1353,7 +1364,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 			      struct folio *src, struct folio *dst,
 			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
+			      struct list_head *ret, bool already_copied)
 {
 	int rc;
 	int old_folio_state = 0;
@@ -1379,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		src_partially_mapped = folio_test_partially_mapped(src);
 	}
 
-	rc = move_to_new_folio(dst, src, mode);
+	rc = move_to_new_folio(dst, src, mode, already_copied);
 	if (rc)
 		goto out;
 
Quoted 9 lines

mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1550

Quoted 6 lines
mm/migrate.c +1 -1
@@ -1536,7 +1547,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 	}
 
 	if (!folio_mapped(src))
-		rc = move_to_new_folio(dst, src, mode);
+		rc = move_to_new_folio(dst, src, mode, false);

... mode, /* already_copied = */ false

will fix

Quoted 14 lines
Quoted 14 lines
mm/migrate.c +1 -1
 	if (page_was_mapped)
 		remove_migration_ptes(src, !rc ? dst : src, ttu);
@@ -1720,7 +1731,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats,
 		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		int *nr_retry_pages, bool already_copied)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
 	bool is_thp;
@@ -1737,7 +1748,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 
Quoted 17 lines
Quoted 8 lines
mm/migrate.c +1 -1
 		rc = migrate_folio_move(put_new_folio, private,
 				folio, dst, mode,
-				reason, ret_folios);
+				reason, ret_folios, already_copied);
 		/*
 		 * The rules are:
 		 *	0: folio will be freed

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2009

Quoted 6 lines
mm/migrate.c +1 -1
@@ -1994,7 +2005,7 @@ static int migrate_pages_batch(struct list_head *from,
 		migrate_folios_move(&unmap_folios, &dst_folios,
 				put_new_folio, private, mode, reason,
 				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
+				&nr_failed, &nr_retry_pages, false);
 	}

dito.

will fix.

Thanks,
Shivank

#16 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 20 lines

Sashiko Comment:
Does expanding FOLIO_OLD_STATES to include BIT(2) silently corrupt anon_vma pointers on 32-bit architectures?
The anon_vma pointer is packed with FOLIO_OLD_STATES inside dst->migrate_info. To separate them, __migrate_folio_extract() uses info & ~FOLIO_OLD_STATES. Since anon_vma objects are allocated from a kmem_cache with align=0, ARCH_SLAB_MINALIGN defaults to 4 bytes on many 32-bit architectures. This means a valid anon_vma pointer can legitimately end in 0x4 or 0xC (meaning bit 2 is set).
When __migrate_folio_extract() masks the pointer with ~7, it will silently clear bit 2 from the anon_vma pointer. Any subsequent call to put_anon_vma() with this corrupted pointer could cause a use-after-free or a kernel panic. --

#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)

I initially assumed this to be always 8, confusing it to be same as size of unsigned long long.
But the GCC docs note that alignment can be smaller in size:

Quoted 8 lines

https://gcc.gnu.org/onlinedocs/gcc/Alignment.html
"For example, if the target machine requires a double value to be aligned on an 8-byte boundary, then __alignof__ (double) is 8. This is true on many RISC machines. On more traditional machine designs, __alignof__ (double) is 4 or even 2."

If my understanding is right, Sashiko concern is valid, and I can't safely use BIT(2).

32bit makes this tricky indeed. And that's also the reason why FOLIO_MAPPING_FLAGS is currently limited to 2 bits.

Quoted 1 line

I see few option from here. Either I can gate batch copy for CONFIG_64BIT,

That's a bit nasty as we'll have to special case 32bit vs 64bit.

Quoted 1 line

or I can force anon_vma_cachep to 8-byte alignment via kmem_cache_create()

That's an option, but I would not do it just for this optimization.

Quoted 2 lines

align arg. Or I can change the migrate_folio() callback to pass already_copied info to change to dst->migrate_info enum.

Can you elaborate how that would look like?

Signature · 3 lines
Cheers,

David
reviewer
#17 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 29 lines · David Hildenbrand (Arm)
Quoted 20 lines

Sashiko Comment:
Does expanding FOLIO_OLD_STATES to include BIT(2) silently corrupt anon_vma pointers on 32-bit architectures?
The anon_vma pointer is packed with FOLIO_OLD_STATES inside dst->migrate_info. To separate them, __migrate_folio_extract() uses info & ~FOLIO_OLD_STATES. Since anon_vma objects are allocated from a kmem_cache with align=0, ARCH_SLAB_MINALIGN defaults to 4 bytes on many 32-bit architectures. This means a valid anon_vma pointer can legitimately end in 0x4 or 0xC (meaning bit 2 is set).
When __migrate_folio_extract() masks the pointer with ~7, it will silently clear bit 2 from the anon_vma pointer. Any subsequent call to put_anon_vma() with this corrupted pointer could cause a use-after-free or a kernel panic. --

#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)

I initially assumed this to be always 8, confusing it to be same as size of unsigned long long.
But the GCC docs note that alignment can be smaller in size:

Quoted 8 lines

https://gcc.gnu.org/onlinedocs/gcc/Alignment.html
"For example, if the target machine requires a double value to be aligned on an 8-byte boundary, then __alignof__ (double) is 8. This is true on many RISC machines. On more traditional machine designs, __alignof__ (double) is 4 or even 2."

If my understanding is right, Sashiko concern is valid, and I can't safely use BIT(2).

Quoted 6 lines

32bit makes this tricky indeed. And that's also the reason why FOLIO_MAPPING_FLAGS is currently limited to 2 bits.

Quoted 1 line

I see few option from here. Either I can gate batch copy for CONFIG_64BIT,

That's a bit nasty as we'll have to special case 32bit vs 64bit.

IIRC, multithreaded copy is already gated by CONFIG_HIGHMEM, otherwise it needs to perform kmap_local() at each copying CPU, which complicates the process. Then, this code will only used for 32bit without highmem, I assume there will no page copy DMA on 32bit platform. Maybe it is not too bad to limit this to 64bit.

Quoted 9 lines
Quoted 2 lines

or I can force anon_vma_cachep to 8-byte alignment via kmem_cache_create()

That's an option, but I would not do it just for this optimization.

Quoted 2 lines

align arg. Or I can change the migrate_folio() callback to pass already_copied info to change to dst->migrate_info enum.

Can you elaborate how that would look like?

Best Regards,
Yan, Zi

#18 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 36 lines · Zi Yan
Quoted 34 lines · David Hildenbrand (Arm)
Quoted 20 lines

Sashiko Comment:
Does expanding FOLIO_OLD_STATES to include BIT(2) silently corrupt anon_vma pointers on 32-bit architectures?
The anon_vma pointer is packed with FOLIO_OLD_STATES inside dst->migrate_info. To separate them, __migrate_folio_extract() uses info & ~FOLIO_OLD_STATES. Since anon_vma objects are allocated from a kmem_cache with align=0, ARCH_SLAB_MINALIGN defaults to 4 bytes on many 32-bit architectures. This means a valid anon_vma pointer can legitimately end in 0x4 or 0xC (meaning bit 2 is set).
When __migrate_folio_extract() masks the pointer with ~7, it will silently clear bit 2 from the anon_vma pointer. Any subsequent call to put_anon_vma() with this corrupted pointer could cause a use-after-free or a kernel panic. --

#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)

I initially assumed this to be always 8, confusing it to be same as size of unsigned long long.
But the GCC docs note that alignment can be smaller in size:

Quoted 8 lines

https://gcc.gnu.org/onlinedocs/gcc/Alignment.html
"For example, if the target machine requires a double value to be aligned on an 8-byte boundary, then __alignof__ (double) is 8. This is true on many RISC machines. On more traditional machine designs, __alignof__ (double) is 4 or even 2."

If my understanding is right, Sashiko concern is valid, and I can't safely use BIT(2).

32bit makes this tricky indeed. And that's also the reason why FOLIO_MAPPING_FLAGS is currently limited to 2 bits.

Quoted 1 line

I see few option from here. Either I can gate batch copy for CONFIG_64BIT,

Quoted 7 lines
Quoted 1 line

That's a bit nasty as we'll have to special case 32bit vs 64bit.

IIRC, multithreaded copy is already gated by CONFIG_HIGHMEM, otherwise it needs to perform kmap_local() at each copying CPU, which complicates the process. Then, this code will only used for 32bit without highmem, I assume there will no page copy DMA on 32bit platform. Maybe it is not too bad to limit this to 64bit.

I'm more concerned of CONFIG_64BIT handling in the code, but if that can be avoided easily, fine with me.

Signature · 3 lines
Cheers,

David
#19 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 37 lines · David Hildenbrand (Arm)
Quoted 36 lines · Zi Yan
Quoted 34 lines · David Hildenbrand (Arm)
Quoted 20 lines

Sashiko Comment:
Does expanding FOLIO_OLD_STATES to include BIT(2) silently corrupt anon_vma pointers on 32-bit architectures?
The anon_vma pointer is packed with FOLIO_OLD_STATES inside dst->migrate_info. To separate them, __migrate_folio_extract() uses info & ~FOLIO_OLD_STATES. Since anon_vma objects are allocated from a kmem_cache with align=0, ARCH_SLAB_MINALIGN defaults to 4 bytes on many 32-bit architectures. This means a valid anon_vma pointer can legitimately end in 0x4 or 0xC (meaning bit 2 is set).
When __migrate_folio_extract() masks the pointer with ~7, it will silently clear bit 2 from the anon_vma pointer. Any subsequent call to put_anon_vma() with this corrupted pointer could cause a use-after-free or a kernel panic. --

#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)

I initially assumed this to be always 8, confusing it to be same as size of unsigned long long.
But the GCC docs note that alignment can be smaller in size:

Quoted 8 lines

https://gcc.gnu.org/onlinedocs/gcc/Alignment.html
"For example, if the target machine requires a double value to be aligned on an 8-byte boundary, then __alignof__ (double) is 8. This is true on many RISC machines. On more traditional machine designs, __alignof__ (double) is 4 or even 2."

If my understanding is right, Sashiko concern is valid, and I can't safely use BIT(2).

32bit makes this tricky indeed. And that's also the reason why FOLIO_MAPPING_FLAGS is currently limited to 2 bits.

Quoted 1 line

I see few option from here. Either I can gate batch copy for CONFIG_64BIT,

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

That's a bit nasty as we'll have to special case 32bit vs 64bit.

IIRC, multithreaded copy is already gated by CONFIG_HIGHMEM, otherwise it needs to perform kmap_local() at each copying CPU, which complicates the process. Then, this code will only used for 32bit without highmem, I assume there will no page copy DMA on 32bit platform. Maybe it is not too bad to limit this to 64bit.

I agree.

Quoted 3 lines

I'm more concerned of CONFIG_64BIT handling in the code, but if that can be avoided easily, fine with me.

MIGRATION_COPY_OFFLOAD will not be enabled for !64BIT at config. config MIGRATION_COPY_OFFLOAD

	bool "Page migration copy offload"
	depends on MIGRATION && 64BIT

I think only place in code we need #ifdef is:

enum {
	FOLIO_WAS_MAPPED	= BIT(0),
	FOLIO_WAS_MLOCKED	= BIT(1),
	FOLIO_OLD_STATES	= FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
	FOLIO_CONTENT_COPIED	= BIT(2),

#else
FOLIO_CONTENT_COPIED = 0,
#endif
};

With FOLIO_CONTENT_COPIED=0, this will be no operation for its bit operation.

eg:
const bool already_copied = dst->migrate_info & FOLIO_CONTENT_COPIED; evaluates 0 for !64BIT.

Git tree (Work-in-progress):
https://github.com/AMDESE/linux-mm/commits/shivank/batch-migrate-offload-v6-wip: 4cd9324a

Quoted 4 lines · David Hildenbrand (Arm)
Quoted 2 lines

align arg. Or I can change the migrate_folio() callback to pass already_copied info to change to dst->migrate_info enum.

Can you elaborate how that would look like?

migrate_folios_move(.., already_copied) -> migrate_folio_move(.., already_copied) ->
-> move_to_new_folio(dst, src, mode, already_copied) -> a_ops->migrate_folio(mapping, dst, src, mode, already_copied)
= migrate_folio / filemap_migrate_folio / buffer_migrate_folio (many other sites in fs to change)

Thanks,
Shivank

#20 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

Quoted 15 lines · Shivank
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · Zi Yan

IIRC, multithreaded copy is already gated by CONFIG_HIGHMEM, otherwise it needs to perform kmap_local() at each copying CPU, which complicates the process. Then, this code will only used for 32bit without highmem, I assume there will no page copy DMA on 32bit platform. Maybe it is not too bad to limit this to 64bit.

I agree.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

I'm more concerned of CONFIG_64BIT handling in the code, but if that can be avoided easily, fine with me.

MIGRATION_COPY_OFFLOAD will not be enabled for !64BIT at config. config MIGRATION_COPY_OFFLOAD

	bool "Page migration copy offload"
	depends on MIGRATION && 64BIT

Might we worth documenting somewhere why, like ...

Quoted 10 lines

I think only place in code we need #ifdef is:

enum {
	FOLIO_WAS_MAPPED	= BIT(0),
	FOLIO_WAS_MLOCKED	= BIT(1),
	FOLIO_OLD_STATES	= FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
	FOLIO_CONTENT_COPIED	= BIT(2),

#else
FOLIO_CONTENT_COPIED = 0,

... here

/* On 32bit we do not have a spare bit, migration-copy offload is disabled. */

Quoted 8 lines

#endif
};

With FOLIO_CONTENT_COPIED=0, this will be no operation for its bit operation.

eg:
const bool already_copied = dst->migrate_info & FOLIO_CONTENT_COPIED; evaluates 0 for !64BIT.

If that does the trick, good!

Signature · 3 lines
Cheers,

David
#21 lore raw

Re: [PATCH 3/7] mm/migrate: skip data copy for already-copied folios

In reply to Garg, Shivank
Quoted 13 lines

Yes,
In current code, migrate_info only carry FOLIO_ALREADY_COPIED. So, =0 and &=~ are equivalent and similar for |= case. I mirrored the how anon_vma/old_folio_state get handled __migrate_folio_extract().

But with your other suggestion of marking already_copied immediately after folio_mc_copy(), then bitwise clear and set will be required. I'll do this.

Quoted 4 lines
Quoted 2 lines

But I wonder if this really belongs exactly here.

I did not understand this.

I was wondering whether this was really the right place to reset/clear it. Your comment above addresses this.

Signature · 3 lines
Cheers,

David
#22 lore raw

PATCH 4/7 mm/migrate: add batch-copy path in migrate_pages_batch

In reply to Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)
include/linux/mm.h +2 -0
index 0b776907152e..e6ab9bc3de8f 100644
@@ -1917,6 +1917,8 @@ void __folio_put(struct folio *folio);
 void split_page(struct page *page, unsigned int order);
 void folio_copy(struct folio *dst, struct folio *src);
 int folio_mc_copy(struct folio *dst, struct folio *src);
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt);
 
 unsigned long nr_free_buffer_pages(void);
 
mm/migrate.c +47 -14

127 lines not on this page · raw message

mm/util.c +30 -0
index 232c3930a662..77eeb285def1 100644
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(folios_mc_copy);
+
 int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS;
 static int sysctl_overcommit_ratio __read_mostly = 50;
 static unsigned long sysctl_overcommit_kbytes __read_mostly;
Signature · 1 line
2.43.0
#23 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 24 lines · Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)

[...]

Quoted 7 lines
mm/migrate.c +2 -0
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;

mm/migrate.c static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, · line 1734

Quoted 8 lines
mm/migrate.c +5 -0
@@ -1724,6 +1727,12 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/* movable_ops folios have their own migrate path */
+static bool folio_supports_batch_copy(struct folio *folio)
+{
+	return likely(!page_has_movable_ops(&folio->page));
+}

As these things are not actually folios (and callers will have to be taught to distinguish them way, way earlier), I guess you should make this

/* movable_ops pages have a separate migration path */ static bool page_supports_batch_copy(struct page *page)
...

Quoted 13 lines
mm/migrate.c +2 -1
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1752,7 +1761,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the unmap_folios list
+		 *	-EAGAIN: stay on the src_folios list
 		 *	Other errno: put on ret_folios list
 		 */
 		switch (rc) {

[...]

Quoted 5 lines
mm/util.c +0 -0
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 789

Quoted 10 lines
mm/util.c +9 -0
 
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.

This *function*

Quoted 6 lines
mm/util.c +6 -0
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 805

Quoted 10 lines
mm/util.c +10 -0
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}

Wouldn't it be cleaner to remember "already copied" immediately after we ... performed the copy? (succeeded with folio_mc_copy)

Signature · 3 lines
Cheers,

David
#24 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 47 lines · David Hildenbrand (Arm)
Quoted 24 lines · Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)

[...]

Quoted 7 lines
mm/migrate.c +2 -0
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;

mm/migrate.c static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, · line 1734

Quoted 8 lines
mm/migrate.c +5 -0
@@ -1724,6 +1727,12 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/* movable_ops folios have their own migrate path */
+static bool folio_supports_batch_copy(struct folio *folio)
+{
+	return likely(!page_has_movable_ops(&folio->page));
+}

As these things are not actually folios (and callers will have to be taught to distinguish them way, way earlier), I guess you should make this

Quoted 3 lines

/* movable_ops pages have a separate migration path */ static bool page_supports_batch_copy(struct page *page)
...

I'll change this.

Quoted 24 lines
Quoted 14 lines
mm/migrate.c +2 -1
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1752,7 +1761,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the unmap_folios list
+		 *	-EAGAIN: stay on the src_folios list
 		 *	Other errno: put on ret_folios list
 		 */
 		switch (rc) {

[...]

Quoted 6 lines
mm/util.c +0 -0
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
Quoted 12 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 789

Quoted 9 lines
mm/util.c +9 -0
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.

This *function*

Will add function.

Quoted 11 lines
Quoted 11 lines
mm/util.c +11 -0
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
Quoted 8 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 805

Quoted 5 lines
mm/util.c +5 -0
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}

Wouldn't it be cleaner to remember "already copied" immediately after we ... performed the copy? (succeeded with folio_mc_copy)

IIUC, you mean something like:

	list_for_each_entry(src, src_list, lru) {
		ret = folio_mc_copy(dst, src);
		if (ret)
			return ret;
		dst->migrate_info |= FOLIO_CONTENT_COPIED;
		dst = list_next_entry(dst, lru);
	}

This will be cleaner. However, it will make the folios_mc_copy() specific for page migration and can't be reused by other potential callers. And offload driver (like DCBM) must set the already copied marker themselves for successful copies.
This will have an upside that on partial-success (even a single failed copy), it will no longer forces re-copy of the whole batch.
If that's the intention, I'll move this to migrate.c itself.

Thanks,
Shivank

#25 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 124 lines · Shivank
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 24 lines · Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)

[...]

Quoted 7 lines
mm/migrate.c +2 -0
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;

mm/migrate.c static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, · line 1734

Quoted 8 lines
mm/migrate.c +5 -0
@@ -1724,6 +1727,12 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/* movable_ops folios have their own migrate path */
+static bool folio_supports_batch_copy(struct folio *folio)
+{
+	return likely(!page_has_movable_ops(&folio->page));
+}

As these things are not actually folios (and callers will have to be taught to distinguish them way, way earlier), I guess you should make this

Quoted 3 lines

/* movable_ops pages have a separate migration path */ static bool page_supports_batch_copy(struct page *page)
...

I'll change this.

Quoted 24 lines
Quoted 14 lines
mm/migrate.c +2 -1
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1752,7 +1761,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the unmap_folios list
+		 *	-EAGAIN: stay on the src_folios list
 		 *	Other errno: put on ret_folios list
 		 */
 		switch (rc) {

[...]

Quoted 6 lines
mm/util.c +0 -0
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
Quoted 12 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 789

Quoted 9 lines
mm/util.c +9 -0
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.

This *function*

Will add function.

Quoted 11 lines
Quoted 11 lines
mm/util.c +11 -0
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
Quoted 8 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 805

Quoted 5 lines
mm/util.c +5 -0
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}

Wouldn't it be cleaner to remember "already copied" immediately after we ... performed the copy? (succeeded with folio_mc_copy)

IIUC, you mean something like:

	list_for_each_entry(src, src_list, lru) {
		ret = folio_mc_copy(dst, src);
		if (ret)
			return ret;
		dst->migrate_info |= FOLIO_CONTENT_COPIED;
		dst = list_next_entry(dst, lru);
	}
Quoted 4 lines

This will be cleaner. However, it will make the folios_mc_copy() specific for page migration and can't be reused by other potential callers. And offload driver (like DCBM) must set the already copied marker themselves for successful copies.

Right, maybe some helper for that would be possible?

Quoted 3 lines

This will have an upside that on partial-success (even a single failed copy), it will no longer forces re-copy of the whole batch.
If that's the intention, I'll move this to migrate.c itself.

True. I'd probably have to take a look at the end result to see if it looks better that way.

Signature · 3 lines
Cheers,

David
reviewer
#26 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 130 lines · David Hildenbrand (Arm)
Quoted 124 lines · Shivank
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 24 lines · Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)

[...]

Quoted 7 lines
mm/migrate.c +2 -0
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;

mm/migrate.c static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, · line 1734

Quoted 8 lines
mm/migrate.c +5 -0
@@ -1724,6 +1727,12 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/* movable_ops folios have their own migrate path */
+static bool folio_supports_batch_copy(struct folio *folio)
+{
+	return likely(!page_has_movable_ops(&folio->page));
+}

As these things are not actually folios (and callers will have to be taught to distinguish them way, way earlier), I guess you should make this

Quoted 3 lines

/* movable_ops pages have a separate migration path */ static bool page_supports_batch_copy(struct page *page)
...

I'll change this.

Quoted 24 lines
Quoted 14 lines
mm/migrate.c +2 -1
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1752,7 +1761,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the unmap_folios list
+		 *	-EAGAIN: stay on the src_folios list
 		 *	Other errno: put on ret_folios list
 		 */
 		switch (rc) {

[...]

Quoted 6 lines
mm/util.c +0 -0
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
Quoted 12 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 789

Quoted 9 lines
mm/util.c +9 -0
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.

This *function*

Will add function.

Quoted 11 lines
Quoted 11 lines
mm/util.c +11 -0
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
Quoted 8 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 805

Quoted 5 lines
mm/util.c +5 -0
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}

Wouldn't it be cleaner to remember "already copied" immediately after we ... performed the copy? (succeeded with folio_mc_copy)

IIUC, you mean something like:

	list_for_each_entry(src, src_list, lru) {
		ret = folio_mc_copy(dst, src);
		if (ret)
			return ret;
		dst->migrate_info |= FOLIO_CONTENT_COPIED;
		dst = list_next_entry(dst, lru);
	}
Quoted 4 lines

This will be cleaner. However, it will make the folios_mc_copy() specific for page migration and can't be reused by other potential callers. And offload driver (like DCBM) must set the already copied marker themselves for successful copies.

Quoted 8 lines

Right, maybe some helper for that would be possible?

Quoted 3 lines

This will have an upside that on partial-success (even a single failed copy), it will no longer forces re-copy of the whole batch.
If that's the intention, I'll move this to migrate.c itself.

True. I'd probably have to take a look at the end result to see if it looks better that way.

Maybe do this optimization in a separate patch to make it simpler to review.

BTW, the whole migration code has become more and more complicated due to batching at different levels, I wonder if it is worth the complication to optimize for the rare memory error case.

Best Regards,
Yan, Zi

#27 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 127 lines · Zi Yan
Quoted 125 lines · David Hildenbrand (Arm)
Quoted 124 lines · Shivank
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 24 lines · Shivank Garg

Add folios_mc_copy() which walks list of src and dst folios in lockstep, and copies folio content via folio_mc_copy(). folios_cnt parameter is unused here, but is part of the offload_copy callback signature used by later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and standard (unmap_single/dst_single) lists, gated by the migrate_offload_enabled which is off by default. So, when no offload driver is active, the branch is never taken and everything goes through the standard path.

After TLB flush, batch copy the eligible folios via folios_mc_copy() and pass already_copied=true into migrate_folios_move() so __migrate_folio() skips the per-folio copy.

On batch copy failure, already_copied flag stays false and each folio fall back to individual copy.

---
 include/linux/mm.h |  2 ++
 mm/migrate.c       | 61 +++++++++++++++++++++++++++++++++++-----------
 mm/util.c          | 30 +++++++++++++++++++++++
 3 files changed, 79 insertions(+), 14 deletions(-)

[...]

Quoted 7 lines
mm/migrate.c +2 -0
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;

mm/migrate.c static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, · line 1734

Quoted 8 lines
mm/migrate.c +5 -0
@@ -1724,6 +1727,12 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/* movable_ops folios have their own migrate path */
+static bool folio_supports_batch_copy(struct folio *folio)
+{
+	return likely(!page_has_movable_ops(&folio->page));
+}

As these things are not actually folios (and callers will have to be taught to distinguish them way, way earlier), I guess you should make this

Quoted 3 lines

/* movable_ops pages have a separate migration path */ static bool page_supports_batch_copy(struct page *page)
...

I'll change this.

Quoted 24 lines
Quoted 14 lines
mm/migrate.c +2 -1
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1752,7 +1761,7 @@ static void migrate_folios_move(struct list_head *src_folios,
 		/*
 		 * The rules are:
 		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the unmap_folios list
+		 *	-EAGAIN: stay on the src_folios list
 		 *	Other errno: put on ret_folios list
 		 */
 		switch (rc) {

[...]

Quoted 6 lines
mm/util.c +0 -0
@@ -778,6 +778,36 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
Quoted 12 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 789

Quoted 9 lines
mm/util.c +9 -0
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.

This *function*

Will add function.

Quoted 11 lines
Quoted 11 lines
mm/util.c +11 -0
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
Quoted 8 lines

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 805

Quoted 5 lines
mm/util.c +5 -0
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}

Wouldn't it be cleaner to remember "already copied" immediately after we ... performed the copy? (succeeded with folio_mc_copy)

IIUC, you mean something like:

	list_for_each_entry(src, src_list, lru) {
		ret = folio_mc_copy(dst, src);
		if (ret)
			return ret;
		dst->migrate_info |= FOLIO_CONTENT_COPIED;
		dst = list_next_entry(dst, lru);
	}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

This will be cleaner. However, it will make the folios_mc_copy() specific for page migration and can't be reused by other potential callers. And offload driver (like DCBM) must set the already copied marker themselves for successful copies.

Right, maybe some helper for that would be possible?

I'm adding a migrate.c specific implementation: migrate_folios_mc_copy().

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

This will have an upside that on partial-success (even a single failed copy), it will no longer forces re-copy of the whole batch.
If that's the intention, I'll move this to migrate.c itself.

Quoted 8 lines
Quoted 2 lines

True. I'd probably have to take a look at the end result to see if it looks better that way.

Maybe do this optimization in a separate patch to make it simpler to review.

BTW, the whole migration code has become more and more complicated due to batching at different levels, I wonder if it is worth the complication to optimize for the rare memory error case.

I agree on complication, but I think this change is actually simpler than my previous patch.

Using per-folio marker lets me drop the already_copied arg that was passed through different function:
migrate_folios_move() -> migrate_folio_move() -> move_to_new_folio(). And there is no separate error path with this.

Whoever performs the batch copy (migrate_folios_mc_copy() or a driver callback) sets FOLIO_CONTENT_COPIED on each dst it successfully copied, and __migrate_folio() reads that marker to skip folio_mc_copy().

If copy is unsuccessful (e.g. memory error), FOLIO_CONTENT_COPIED is NOT marked, and __migrate_folio() will not skip the folio_mc_copy().

Since the per-folio marking is the natural implementation for this, I kept it in this patch (af44801ee).

Git Tree for reference: https://github.com/AMDESE/linux-mm/commits/shivank/batch-migrate-offload-v6-wip

Thanks,
Shivank

#28 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

Quoted 18 lines · Shivank
Quoted 6 lines · Zi Yan

Maybe do this optimization in a separate patch to make it simpler to review.

BTW, the whole migration code has become more and more complicated due to batching at different levels, I wonder if it is worth the complication to optimize for the rare memory error case.

I agree on complication, but I think this change is actually simpler than my previous patch.

Using per-folio marker lets me drop the already_copied arg that was passed through different function:
migrate_folios_move() -> migrate_folio_move() -> move_to_new_folio(). And there is no separate error path with this.

Quoted 6 lines

Whoever performs the batch copy (migrate_folios_mc_copy() or a driver callback) sets FOLIO_CONTENT_COPIED on each dst it successfully copied, and __migrate_folio() reads that marker to skip folio_mc_copy().

If copy is unsuccessful (e.g. memory error), FOLIO_CONTENT_COPIED is NOT marked, and __migrate_folio() will not skip the folio_mc_copy().

I fully agree on this as well. It's much simpler and cleaner than passing down already_copied flag.

Quoted 8 lines

Since the per-folio marking is the natural implementation for this, I kept it in this patch (af44801ee).

Git Tree for reference: https://github.com/AMDESE/linux-mm/commits/shivank/batch-migrate-offload-v6-wip

Thanks,
Shivank

Signature · 1 line
~karim
#29 lore raw

Re: [PATCH 4/7] mm/migrate: add batch-copy path in migrate_pages_batch

In reply to Shivank Garg

On 4/28/2026 9:20 PM, Shivank Garg wrote:

Sashiko review comment:
https://sashiko.dev/#/patchset/20260428155043.39251-2-shivankg%40amd.com

[...]

Quoted 16 lines
mm/migrate.c +5 -5
+		rc = folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+		batch_copied = (rc == 0);
+	}
+
 	retry = 1;
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;
 		nr_retry_pages = 0;
 
-		/* Move the unmapped folios */
-		migrate_folios_move(&unmap_folios, &dst_folios,
-				put_new_folio, private, mode, reason,
-				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages, false);
+		if (!list_empty(&unmap_batch))

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2040

Quoted 10 lines
mm/migrate.c +9 -0
+			migrate_folios_move(&unmap_batch, &dst_batch, put_new_folio,
+					private, mode, reason, ret_folios, stats,
+					&retry, &thp_retry, &nr_failed,
+					&nr_retry_pages, batch_copied);
+		if (!list_empty(&unmap_single))
+			migrate_folios_move(&unmap_single, &dst_single,	put_new_folio,
+					private, mode, reason, ret_folios, stats,
+					&retry, &thp_retry, &nr_failed,
+					&nr_retry_pages, false);
 	}

Sashiko:
Could reusing the batch_copied flag on retry cause data loss? The early batch copy captures the data before the exclusivity check (folio_ref_count() == expected_count) in the move phase. If a folio is pinned by DMA, the early copy might capture stale or partial data while the move phase correctly detects the elevated refcount and returns -EAGAIN. During the retry loop, migrate_folios_move() is called again with the original batch_copied value (true). Once the DMA pin is dropped, the move phase succeeds but skips the data copy, leaving the destination folio with the stale data from the first pass.
--

IIUC, This is real race and original code avoids this by placing the copy after the refcount check, so by the time we copy, no extra ref are held by writer.

To fix this, I plan to avoid DMA pinned folios from batch path, by adding refcount check when creating the unmap_batch list.

if (... page_supports_batch_copy(folio) &&

    folio_ref_count(folio) == folio_expected_ref_count(folio) + 1) {
    /* add to dst_batch/src_batch list */
} else {
    /* dst_single/unmap_single list */
}

[...]

Quoted 18 lines
mm/util.c +18 -0
+/**
+ * folios_mc_copy - Copy the contents of list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walks list of src and dst folios in lockstep and copies folio
+ * content via folio_mc_copy(). The caller must ensure both lists have
+ * the same number of entries. This may sleep.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+

mm/util.c int folio_mc_copy(struct folio *dst, struct folio *src) · line 808

Quoted 10 lines
mm/util.c +10 -0
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst = list_next_entry(dst, lru);
+	}
+
+	return 0;
+}

Sashiko:
Should this loop include a cond_resched()?
The original folio migration loop in migrate_folios_move() explicitly calls cond_resched() on every iteration. While folio_mc_copy() calls cond_resched() internally, it only does so between subpages of a large folio:

mm/util.c:folio_mc_copy() {
    ...
    if (++i == nr)
        break;
    cond_resched();
}

For order-0 folios (nr == 1), the loop breaks before reaching the reschedule point. Processing a large batch of order-0 folios here will hold the CPU in an uninterrupted tight loop without yielding.
--

This concern for large batch of order-0 folios looks valid. But I think the current direction is to remove cond_resched() eventually for lazy preemption. So should I take Sashiko's suggestion to add cond_resched()?

Thanks,
Shivank

#30 lore raw

PATCH 5/7 mm/migrate: add copy offload registration infrastructure

In reply to Shivank Garg

Add a registration interface that lets a single offload provider (DMA, multi-threaded CPU copy, etc) take over the batch folio copy performed by migrate_pages_batch().

The provider fills in a struct migrator with an offload_copy() callback and calls migrate_offload_register(). Registration patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_unregister() reverts both.

Whether a migration reason is batch-copy eligible is decided by the core in migrate_offload_do_batch(). A migrator may decline a particular batch (e.g. when nr_batch is too small to amortize setup) by returning -EOPNOTSUPP, and the move phase falls back to per-folio CPU copy.

Only one migrator can be active at a time. A second registration returns -EBUSY, and only the active migrator can unregister itself. The static_call dispatch is protected by SRCU so that the synchronize_srcu() in unregister waits for all in-flight copy before the module reference is dropped.

---
 include/linux/migrate_copy_offload.h | 44 +++++++++++++
 mm/Kconfig                           |  6 ++
 mm/Makefile                          |  1 +
 mm/migrate.c                         | 57 +++++++++++++++--
 mm/migrate_copy_offload.c            | 94 ++++++++++++++++++++++++++++
 5 files changed, 198 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c
include/linux/migrate_copy_offload.h new file +44 -0
new file mode 100644
index 000000000000..d68b10a84743
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H
+#define _LINUX_MIGRATE_COPY_OFFLOAD_H
+
+#include <linux/errno.h>
+#include <linux/jump_label.h>
+#include <linux/srcu.h>
+#include <linux/types.h>
+
+struct list_head;
+struct module;
+
+#define MIGRATOR_NAME_LEN 32
+
+/**
+ * struct migrator - batch-copy provider for page migration.
+ * @name: name of the provider.
+ * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list.
+ *
+ *	The migrator may inspect @folio_cnt to decide whether the batch
+ * 	is worth offloading, e.g. skip when the batch is too small to
+ * 	amortize setup cost. If returns error, the core falls back to CPU copy.
+ *
+ * @owner: module providing the migrator.
+ */
+struct migrator {
+	char name[MIGRATOR_NAME_LEN];
+	int (*offload_copy)(struct list_head *dst_list,
+			    struct list_head *src_list,
+			    unsigned int folio_cnt);
+	struct module *owner;
+};
+
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+extern struct static_key_false migrate_offload_enabled;
+extern struct srcu_struct migrate_offload_srcu;
+int migrate_offload_register(struct migrator *m);
+int migrate_offload_unregister(struct migrator *m);
+#else
+static inline int migrate_offload_register(struct migrator *m) { return -EOPNOTSUPP; }
+static inline int migrate_offload_unregister(struct migrator *m) { return -EOPNOTSUPP; }
+#endif
+
+#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */
mm/Kconfig +6 -0
index e8bf1e9e6ad9..325d79619680 100644
@@ -647,6 +647,12 @@ config MIGRATION
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
 
+# Page-migration batch-copy offload infrastructure.
+# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA).
+config MIGRATION_COPY_OFFLOAD
+	bool
+	depends on MIGRATION
+
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
 
mm/Makefile +1 -0
index 8ad2ab08244e..db1ac8097089 100644
@@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o
 obj-$(CONFIG_MEMTEST)		+= memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o
 obj-$(CONFIG_NUMA) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
mm/migrate.c +53 -4

93 lines not on this page · raw message

mm/migrate_copy_offload.c new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.43.0
#31 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Quoted 21 lines · Shivank Garg

Add a registration interface that lets a single offload provider (DMA, multi-threaded CPU copy, etc) take over the batch folio copy performed by migrate_pages_batch().

The provider fills in a struct migrator with an offload_copy() callback and calls migrate_offload_register(). Registration patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_unregister() reverts both.

Whether a migration reason is batch-copy eligible is decided by the core in migrate_offload_do_batch(). A migrator may decline a particular batch (e.g. when nr_batch is too small to amortize setup) by returning -EOPNOTSUPP, and the move phase falls back to per-folio CPU copy.

Only one migrator can be active at a time. A second registration returns -EBUSY, and only the active migrator can unregister itself. The static_call dispatch is protected by SRCU so that the synchronize_srcu() in unregister waits for all in-flight copy before the module reference is dropped.

Quoted 3 lines

MAINTAINERS file deserves some love (likely put the new files under the migration section)

Quoted 140 lines
---
 include/linux/migrate_copy_offload.h | 44 +++++++++++++
 mm/Kconfig                           |  6 ++
 mm/Makefile                          |  1 +
 mm/migrate.c                         | 57 +++++++++++++++--
 mm/migrate_copy_offload.c            | 94 ++++++++++++++++++++++++++++
 5 files changed, 198 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c
include/linux/migrate_copy_offload.h new file +44 -0
new file mode 100644
index 000000000000..d68b10a84743
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H
+#define _LINUX_MIGRATE_COPY_OFFLOAD_H
+
+#include <linux/errno.h>
+#include <linux/jump_label.h>
+#include <linux/srcu.h>
+#include <linux/types.h>
+
+struct list_head;
+struct module;
+
+#define MIGRATOR_NAME_LEN 32
+
+/**
+ * struct migrator - batch-copy provider for page migration.
+ * @name: name of the provider.
+ * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list.
+ *
+ *	The migrator may inspect @folio_cnt to decide whether the batch
+ * 	is worth offloading, e.g. skip when the batch is too small to
+ * 	amortize setup cost. If returns error, the core falls back to CPU copy.
+ *
+ * @owner: module providing the migrator.
+ */
+struct migrator {
+	char name[MIGRATOR_NAME_LEN];
+	int (*offload_copy)(struct list_head *dst_list,
+			    struct list_head *src_list,
+			    unsigned int folio_cnt);
+	struct module *owner;
+};
+
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+extern struct static_key_false migrate_offload_enabled;
+extern struct srcu_struct migrate_offload_srcu;
+int migrate_offload_register(struct migrator *m);
+int migrate_offload_unregister(struct migrator *m);
+#else
+static inline int migrate_offload_register(struct migrator *m) { return -EOPNOTSUPP; }
+static inline int migrate_offload_unregister(struct migrator *m) { return -EOPNOTSUPP; }
+#endif
+
+#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */
mm/Kconfig +6 -0
index e8bf1e9e6ad9..325d79619680 100644
@@ -647,6 +647,12 @@ config MIGRATION
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
 
+# Page-migration batch-copy offload infrastructure.
+# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA).
+config MIGRATION_COPY_OFFLOAD
+	bool
+	depends on MIGRATION
+
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
 
mm/Makefile +1 -0
index 8ad2ab08244e..db1ac8097089 100644
@@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o
 obj-$(CONFIG_MEMTEST)		+= memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o
 obj-$(CONFIG_NUMA) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
mm/migrate.c +36 -0
index 6c2f1cb66f96..9af070f9a1f2 100644
@@ -44,6 +44,8 @@
 #include <linux/memory-tiers.h>
 #include <linux/pagewalk.h>
 #include <linux/jump_label.h>
+#include <linux/static_call.h>
+#include <linux/migrate_copy_offload.h>
 
 #include <asm/tlbflush.h>
 
@@ -54,6 +56,51 @@
 
 DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
 
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+DEFINE_SRCU(migrate_offload_srcu);
+DEFINE_STATIC_CALL(migrate_offload_copy, folios_mc_copy);
+
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static int migrate_offload_batch_copy(struct list_head *dst_batch,
+				      struct list_head *src_batch,
+				      unsigned int nr_batch)
+{
+	int idx, rc;
+
+	idx = srcu_read_lock(&migrate_offload_srcu);
+	rc = static_call(migrate_offload_copy)(dst_batch, src_batch, nr_batch);
+	srcu_read_unlock(&migrate_offload_srcu, idx);
+	return rc;
+}
+#else
+static bool migrate_offload_do_batch(int reason)
+{

mm/migrate.c line 102

Quoted 10 lines
mm/migrate.c +10 -0
+	return false;
+}
+
+static int migrate_offload_batch_copy(struct list_head *dst_batch,
+				      struct list_head *src_batch,
+				      unsigned int nr_batch)
+{
+	return -EOPNOTSUPP;
+}
+#endif

Can't all that go to migrate_copy_offload.(hc) Looks like a clean interface form migrate.c, no?

Quoted 23 lines
mm/migrate.c +4 -1
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;
 
@@ -1833,7 +1880,7 @@ static int migrate_pages_batch(struct list_head *from,
 	struct folio *folio, *folio2, *dst = NULL;
 	int rc, rc_saved = 0, nr_pages;
 	unsigned int nr_batch = 0;
-	bool batch_copied = false;
+	bool do_batch = false, batch_copied = false;
 	LIST_HEAD(unmap_batch);
 	LIST_HEAD(dst_batch);
 	LIST_HEAD(unmap_single);
@@ -1843,6 +1890,8 @@ static int migrate_pages_batch(struct list_head *from,
 	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
 			!list_empty(from) && !list_is_singular(from));
 
+	do_batch = migrate_offload_do_batch(reason);
+
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2036

Quoted 6 lines
mm/migrate.c +1 -2
@@ -1984,8 +2033,7 @@ static int migrate_pages_batch(struct list_head *from,
 				nr_retry_pages += nr_pages;
 				break;
 			case 0:
-				if (static_branch_unlikely(&migrate_offload_enabled) &&
-				    folio_supports_batch_copy(folio)) {
+				if (do_batch && folio_supports_batch_copy(folio)) {

I was about to say, the migrate_offload_enabled usage in the previous patch looks off.

Can you move that to this patch here, and use in the previous patch simply a default migrate_offload_do_batch() that simply returns "false" ?

Quoted 14 lines
mm/migrate.c +2 -1
 					list_move_tail(&folio->lru, &unmap_batch);
 					list_add_tail(&dst->lru, &dst_batch);
 					nr_batch++;
@@ -2017,7 +2065,8 @@ static int migrate_pages_batch(struct list_head *from,
 
 	/* Batch-copy eligible folios before the move phase */
 	if (!list_empty(&unmap_batch)) {
-		rc = folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+		rc = migrate_offload_batch_copy(&dst_batch, &unmap_batch,
+						nr_batch);
 		batch_copied = (rc == 0);
 	}
 
mm/migrate_copy_offload.c +0 -0
Signature · 3 lines
Cheers,

David
#32 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Quoted 22 lines · David Hildenbrand (Arm)
Quoted 21 lines · Shivank Garg

Add a registration interface that lets a single offload provider (DMA, multi-threaded CPU copy, etc) take over the batch folio copy performed by migrate_pages_batch().

The provider fills in a struct migrator with an offload_copy() callback and calls migrate_offload_register(). Registration patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_unregister() reverts both.

Whether a migration reason is batch-copy eligible is decided by the core in migrate_offload_do_batch(). A migrator may decline a particular batch (e.g. when nr_batch is too small to amortize setup) by returning -EOPNOTSUPP, and the move phase falls back to per-folio CPU copy.

Only one migrator can be active at a time. A second registration returns -EBUSY, and only the active migrator can unregister itself. The static_call dispatch is protected by SRCU so that the synchronize_srcu() in unregister waits for all in-flight copy before the module reference is dropped.

Quoted 6 lines
Quoted 3 lines

MAINTAINERS file deserves some love (likely put the new files under the migration section)

Sure, will add this in the next posting.

[...]

Quoted 53 lines
Quoted 53 lines

+++ b/mm/migrate.c

mm/migrate.c +41 -0
@@ -44,6 +44,8 @@
 #include <linux/memory-tiers.h>
 #include <linux/pagewalk.h>
 #include <linux/jump_label.h>
+#include <linux/static_call.h>
+#include <linux/migrate_copy_offload.h>
 
 #include <asm/tlbflush.h>
 
@@ -54,6 +56,51 @@
 
 DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
 
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+DEFINE_SRCU(migrate_offload_srcu);
+DEFINE_STATIC_CALL(migrate_offload_copy, folios_mc_copy);
+
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static int migrate_offload_batch_copy(struct list_head *dst_batch,
+				      struct list_head *src_batch,
+				      unsigned int nr_batch)
+{
+	int idx, rc;
+
+	idx = srcu_read_lock(&migrate_offload_srcu);
+	rc = static_call(migrate_offload_copy)(dst_batch, src_batch, nr_batch);
+	srcu_read_unlock(&migrate_offload_srcu, idx);
+	return rc;
+}
+#else
+static bool migrate_offload_do_batch(int reason)
+{
+	return false;
+}
+
+static int migrate_offload_batch_copy(struct list_head *dst_batch,
+				      struct list_head *src_batch,
Quoted 9 lines

mm/migrate.c line 102

Quoted 5 lines
mm/migrate.c +5 -0
+				      unsigned int nr_batch)
+{
+	return -EOPNOTSUPP;
+}
+#endif

Can't all that go to migrate_copy_offload.(hc) Looks like a clean interface form migrate.c, no?

Yes, Will do.

Quoted 33 lines
Quoted 23 lines
mm/migrate.c +4 -1
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;
 
@@ -1833,7 +1880,7 @@ static int migrate_pages_batch(struct list_head *from,
 	struct folio *folio, *folio2, *dst = NULL;
 	int rc, rc_saved = 0, nr_pages;
 	unsigned int nr_batch = 0;
-	bool batch_copied = false;
+	bool do_batch = false, batch_copied = false;
 	LIST_HEAD(unmap_batch);
 	LIST_HEAD(dst_batch);
 	LIST_HEAD(unmap_single);
@@ -1843,6 +1890,8 @@ static int migrate_pages_batch(struct list_head *from,
 	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
 			!list_empty(from) && !list_is_singular(from));
 
+	do_batch = migrate_offload_do_batch(reason);
+
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;

mm/migrate.c static int migrate_pages_batch(struct list_head *from, · line 2036

Quoted 6 lines
mm/migrate.c +1 -2
@@ -1984,8 +2033,7 @@ static int migrate_pages_batch(struct list_head *from,
 				nr_retry_pages += nr_pages;
 				break;
 			case 0:
-				if (static_branch_unlikely(&migrate_offload_enabled) &&
-				    folio_supports_batch_copy(folio)) {
+				if (do_batch && folio_supports_batch_copy(folio)) {

I was about to say, the migrate_offload_enabled usage in the previous patch looks off.

Quoted 3 lines

Can you move that to this patch here, and use in the previous patch simply a default migrate_offload_do_batch() that simply returns "false" ?

Yes, will do.
This is more clean.

Thanks,
Shivank

#33 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

In reply to Shivank Garg
Quoted 137 lines · Shivank Garg

Add a registration interface that lets a single offload provider (DMA, multi-threaded CPU copy, etc) take over the batch folio copy performed by migrate_pages_batch().

The provider fills in a struct migrator with an offload_copy() callback and calls migrate_offload_register(). Registration patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_unregister() reverts both.

Whether a migration reason is batch-copy eligible is decided by the core in migrate_offload_do_batch(). A migrator may decline a particular batch (e.g. when nr_batch is too small to amortize setup) by returning -EOPNOTSUPP, and the move phase falls back to per-folio CPU copy.

Only one migrator can be active at a time. A second registration returns -EBUSY, and only the active migrator can unregister itself. The static_call dispatch is protected by SRCU so that the synchronize_srcu() in unregister waits for all in-flight copy before the module reference is dropped.

---
 include/linux/migrate_copy_offload.h | 44 +++++++++++++
 mm/Kconfig                           |  6 ++
 mm/Makefile                          |  1 +
 mm/migrate.c                         | 57 +++++++++++++++--
 mm/migrate_copy_offload.c            | 94 ++++++++++++++++++++++++++++
 5 files changed, 198 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c
include/linux/migrate_copy_offload.h new file +44 -0
new file mode 100644
index 000000000000..d68b10a84743
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H
+#define _LINUX_MIGRATE_COPY_OFFLOAD_H
+
+#include <linux/errno.h>
+#include <linux/jump_label.h>
+#include <linux/srcu.h>
+#include <linux/types.h>
+
+struct list_head;
+struct module;
+
+#define MIGRATOR_NAME_LEN 32
+
+/**
+ * struct migrator - batch-copy provider for page migration.
+ * @name: name of the provider.
+ * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list.
+ *
+ *	The migrator may inspect @folio_cnt to decide whether the batch
+ * 	is worth offloading, e.g. skip when the batch is too small to
+ * 	amortize setup cost. If returns error, the core falls back to CPU copy.
+ *
+ * @owner: module providing the migrator.
+ */
+struct migrator {
+	char name[MIGRATOR_NAME_LEN];
+	int (*offload_copy)(struct list_head *dst_list,
+			    struct list_head *src_list,
+			    unsigned int folio_cnt);
+	struct module *owner;
+};
+
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+extern struct static_key_false migrate_offload_enabled;
+extern struct srcu_struct migrate_offload_srcu;
+int migrate_offload_register(struct migrator *m);
+int migrate_offload_unregister(struct migrator *m);
+#else
+static inline int migrate_offload_register(struct migrator *m) { return -EOPNOTSUPP; }
+static inline int migrate_offload_unregister(struct migrator *m) { return -EOPNOTSUPP; }
+#endif
+
+#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */
mm/Kconfig +6 -0
index e8bf1e9e6ad9..325d79619680 100644
@@ -647,6 +647,12 @@ config MIGRATION
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
 
+# Page-migration batch-copy offload infrastructure.
+# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA).
+config MIGRATION_COPY_OFFLOAD
+	bool
+	depends on MIGRATION
+
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
 
mm/Makefile +1 -0
index 8ad2ab08244e..db1ac8097089 100644
@@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o
 obj-$(CONFIG_MEMTEST)		+= memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o
 obj-$(CONFIG_NUMA) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
mm/migrate.c +9 -0
index 6c2f1cb66f96..9af070f9a1f2 100644
@@ -44,6 +44,8 @@
 #include <linux/memory-tiers.h>
 #include <linux/pagewalk.h>
 #include <linux/jump_label.h>
+#include <linux/static_call.h>
+#include <linux/migrate_copy_offload.h>
 
 #include <asm/tlbflush.h>
 
@@ -54,6 +56,51 @@
 
 DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
 
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+DEFINE_SRCU(migrate_offload_srcu);
+DEFINE_STATIC_CALL(migrate_offload_copy, folios_mc_copy);
+
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))

mm/migrate.c line 75

Quoted 10 lines
mm/migrate.c +10 -0
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Signature · 3 lines
Cheers,

David
#34 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Quoted 1 line · David Hildenbrand (Arm)

On 4/28/26 17:50, Shivank Garg wrote:

[...]

Quoted 9 lines
Quoted 9 lines
mm/migrate.c +9 -0
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
Quoted 8 lines

mm/migrate.c line 75

Quoted 4 lines
mm/migrate.c +4 -0
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Reason-based filtering could be a requirement for some users who want only specific use cases to go through DMA offload.

For the RFC, I introduced a placeholder to enable further discussion on which use cases should allow migration offload and whether offload users actually need this control?

Your other point also makes sense: "If someone migrates a handful of folios, latency is likely more important (and batching less beneficial)."
Based on this, we could either fully rely on batch size. I'll think more about this.

Thanks,
Shivank

#35 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Quoted 18 lines · Shivank
Quoted 9 lines
Quoted 9 lines
mm/migrate.c +9 -0
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
Quoted 8 lines

mm/migrate.c line 75

Quoted 4 lines
mm/migrate.c +4 -0
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Quoted 3 lines

Reason-based filtering could be a requirement for some users who want only specific use cases to go through DMA offload.

+1, I think this makes a lot of sense; not all DMA offloads are created equally and we may prefer to unburden them from being contended by migrations that they are not intending to accelerate.

Quoted 7 lines

For the RFC, I introduced a placeholder to enable further discussion on which use cases should allow migration offload and whether offload users actually need this control?

Your other point also makes sense: "If someone migrates a handful of folios, latency is likely more important (and batching less beneficial)."
Based on this, we could either fully rely on batch size. I'll think more about this.

There are, or will be, some offloads that must be used for certain types of page migrations, like Confidential Computing. That's for functional reasons, not a heuristic.

We want to use certain hardware assists solely for promotion and demotion of memory for tiering. We certainly wouldn't want those hardware assists to be inundated by users doing tons of move_pages(2) on their own or by best effort memory compaction in the kernel.

I think the use cases should be configurable by the user if at all possible so we can control what has access to being offloaded. These are often shared system resources and can be contended like any other resource, so configuring which migrations can use them vs not use them seems important.

#36 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Received a bounce for Jonathan, so adding in his other email address.

Quoted 51 lines · David Rientjes
Quoted 18 lines · Shivank
Quoted 9 lines
Quoted 9 lines
mm/migrate.c +9 -0
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
Quoted 8 lines

mm/migrate.c line 75

Quoted 4 lines
mm/migrate.c +4 -0
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Quoted 3 lines

Reason-based filtering could be a requirement for some users who want only specific use cases to go through DMA offload.

+1, I think this makes a lot of sense; not all DMA offloads are created equally and we may prefer to unburden them from being contended by migrations that they are not intending to accelerate.

Quoted 7 lines

For the RFC, I introduced a placeholder to enable further discussion on which use cases should allow migration offload and whether offload users actually need this control?

Your other point also makes sense: "If someone migrates a handful of folios, latency is likely more important (and batching less beneficial)."
Based on this, we could either fully rely on batch size. I'll think more about this.

There are, or will be, some offloads that must be used for certain types of page migrations, like Confidential Computing. That's for functional reasons, not a heuristic.

We want to use certain hardware assists solely for promotion and demotion of memory for tiering. We certainly wouldn't want those hardware assists to be inundated by users doing tons of move_pages(2) on their own or by best effort memory compaction in the kernel.

I think the use cases should be configurable by the user if at all possible so we can control what has access to being offloaded. These are often shared system resources and can be contended like any other resource, so configuring which migrations can use them vs not use them seems important.

#37 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

In reply to David Rientjes
Quoted 36 lines · David Rientjes
Quoted 18 lines · Shivank
Quoted 9 lines
Quoted 9 lines
mm/migrate.c +9 -0
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
Quoted 8 lines

mm/migrate.c line 75

Quoted 4 lines
mm/migrate.c +4 -0
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Quoted 3 lines

Reason-based filtering could be a requirement for some users who want only specific use cases to go through DMA offload.

+1, I think this makes a lot of sense; not all DMA offloads are created equally and we may prefer to unburden them from being contended by migrations that they are not intending to accelerate.

Quoted 7 lines

For the RFC, I introduced a placeholder to enable further discussion on which use cases should allow migration offload and whether offload users actually need this control?

Your other point also makes sense: "If someone migrates a handful of folios, latency is likely more important (and batching less beneficial)."
Based on this, we could either fully rely on batch size. I'll think more about this.

Quoted 8 lines

There are, or will be, some offloads that must be used for certain types of page migrations, like Confidential Computing. That's for functional reasons, not a heuristic.

We want to use certain hardware assists solely for promotion and demotion of memory for tiering. We certainly wouldn't want those hardware assists to be inundated by users doing tons of move_pages(2) on their own or by best effort memory compaction in the kernel.

One of the biggest arguments against using memory-offload accelerators for generic page migration is that user-mapped memory migration is dominated by rmap (unmapping before migration) and remove_migration_ptes() after memory copy. In my experiments with Intel DSA on Sapphire Rapid, assuming you're copying 4KiB pages at a deep queue depth (64/128 pages).

1. A single instance can achieve up to 30GB/s when counting only descriptor submission and polling for completion.

2. With DMA mapping overhead, which you usually have to do when you are behind an IOMMU, it's about 23GB/s in IOMMU passthrough mode, and 17GB/s in translated mode. That's still more than double the throughput achieved by a single CPU core at around 9GB/s.

3. The throughput drops significantly when measuring the completetion of the entire migrate_pages() at barely 1GB/s.

Unmapping is about 50% of the total time and remove_migration_ptes() is about 40/45% in my experiments, which leaves only 5/10% on the table for the actual page data copy. You're not bottlenecked by memory bandwidth to justify using a memory-offload accelerators and the CPU cycles you save from offloading is too small to matter.

So any discussion on accelerating promotion/demotion using these accelerators is useless to me, unless I am missing something.

Like what kind of use cases do you have in mind? What kind of workloads and pages you're thinking? Especially, via migrate_pages() which is largely involved in scenarios involving user-mapped memory (anonymous and file pages) and thus heavy rmap involvment.

Quoted 5 lines

I think the use cases should be configurable by the user if at all possible so we can control what has access to being offloaded. These are often shared system resources and can be contended like any other resource, so configuring which migrations can use them vs not use them seems important.

Signature · 1 line
~karim
reviewer
#38 lore raw

Re: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure

Quoted 60 lines · Karim Manaouil
Quoted 36 lines · David Rientjes
Quoted 18 lines · Shivank
Quoted 9 lines
Quoted 9 lines
mm/migrate.c +9 -0
+static bool migrate_offload_do_batch(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	switch (reason) {
+	case MR_COMPACTION:
+	case MR_SYSCALL:
+	case MR_DEMOTION:
Quoted 8 lines

mm/migrate.c line 75

Quoted 4 lines
mm/migrate.c +4 -0
+	case MR_NUMA_MISPLACED:
+		return true;
+	default:
+		return false;

What's the exact reason we don't do this for hotunplug etc? IOW, why do we make this depend on a reason?

Quoted 3 lines

Reason-based filtering could be a requirement for some users who want only specific use cases to go through DMA offload.

+1, I think this makes a lot of sense; not all DMA offloads are created equally and we may prefer to unburden them from being contended by migrations that they are not intending to accelerate.

Quoted 7 lines

For the RFC, I introduced a placeholder to enable further discussion on which use cases should allow migration offload and whether offload users actually need this control?

Your other point also makes sense: "If someone migrates a handful of folios, latency is likely more important (and batching less beneficial)."
Based on this, we could either fully rely on batch size. I'll think more about this.

Quoted 8 lines

There are, or will be, some offloads that must be used for certain types of page migrations, like Confidential Computing. That's for functional reasons, not a heuristic.

We want to use certain hardware assists solely for promotion and demotion of memory for tiering. We certainly wouldn't want those hardware assists to be inundated by users doing tons of move_pages(2) on their own or by best effort memory compaction in the kernel.

One of the biggest arguments against using memory-offload accelerators for generic page migration is that user-mapped memory migration is dominated by rmap (unmapping before migration) and remove_migration_ptes() after memory copy. In my experiments with Intel DSA on Sapphire Rapid, assuming you're copying 4KiB pages at a deep queue depth (64/128 pages).

1. A single instance can achieve up to 30GB/s when counting only descriptor submission and polling for completion.

2. With DMA mapping overhead, which you usually have to do when you are behind an IOMMU, it's about 23GB/s in IOMMU passthrough mode, and 17GB/s in translated mode. That's still more than double the throughput achieved by a single CPU core at around 9GB/s.

Quoted 5 lines

3. The throughput drops significantly when measuring the completetion of the entire migrate_pages() at barely 1GB/s.

Unmapping is about 50% of the total time and remove_migration_ptes() is about 40/45% in my experiments, which leaves only 5/10% on the table for

Do you have a breakdown of unmapping and remove_migration_ptes()? Are these 4KB pages shared or private? I can see that a shared page can cost more during rmap. BTW, I assume your measurements are done with migrate_pages_batch(), which performs TLB shootdown once for all these pages, otherwise TLB shootdown cost would dominate.

Quoted 11 lines

the actual page data copy. You're not bottlenecked by memory bandwidth to justify using a memory-offload accelerators and the CPU cycles you save from offloading is too small to matter.

So any discussion on accelerating promotion/demotion using these accelerators is useless to me, unless I am missing something.

Like what kind of use cases do you have in mind? What kind of workloads and pages you're thinking? Especially, via migrate_pages() which is largely involved in scenarios involving user-mapped memory (anonymous and file pages) and thus heavy rmap involvment.

A paper[1] shows that using DSA can improve memory compaction speed and memory compaction should migrate <2MB folios.

[1] https://ieeexplore.ieee.org/document/10841986

Quoted 9 lines
Quoted 6 lines

I think the use cases should be configurable by the user if at all possible so we can control what has access to being offloaded. These are often shared system resources and can be contended like any other resource, so configuring which migrations can use them vs not use them seems important.

Signature · 1 line
~karim

Best Regards,
Yan, Zi

#39 lore raw

PATCH 6/7 drivers/migrate_offload: add DMA batch copy driver (dcbm)

In reply to Shivank Garg

Simple DMAEngine-based migrator that plugs into the page migration copy offload infrastructure to batch-copy folios via DMA memcpy channels. It is intended for testing the offload plumbing and as a template for future migrators (SDXI, multi-threaded CPU copy, etc.).

When DMA fails, the callback returns an error and the migration path falls back to per-folio CPU copy.

Loading the module exposes attributes under /sys/module/dcbm/:

offloading - enable/disable DMA offload
nr_dma_chan - max DMA channels to use
folios_migrated - folios copied via DMA
folios_failures - fallback count

CONFIG_DCBM_DMA selects MIGRATION_COPY_OFFLOAD so enabling the driver pulls in the infrastructure automatically.

Channel acquisition uses dma_request_chan_by_mask(DMA_MEMCPY), which works for providers that set DMA_PRIVATE (e.g. AMD PTDMA). Generic mem-to-mem engines that do not set DMA_PRIVATE (e.g. SDXI) should acquire channels via dma_find_channel(DMA_MEMCPY) or the async_tx APIs, which can be added in a follow-up.

---
 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 440 ++++++++++++++++++++++++++
 6 files changed, 455 insertions(+)
 create mode 100644 drivers/migrate_offload/Kconfig
 create mode 100644 drivers/migrate_offload/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/dcbm.c
drivers/Kconfig +2 -0
index f2bed2ddeb66..3e83a1475cbc 100644
@@ -253,4 +253,6 @@ source "drivers/cdx/Kconfig"
 
 source "drivers/resctrl/Kconfig"
 
+source "drivers/migrate_offload/Kconfig"
+
 endmenu
drivers/Makefile +2 -0
index 0841ea851847..88cb8e3e88df 100644
@@ -42,6 +42,8 @@ obj-y				+= clk/
 # really early.
 obj-$(CONFIG_DMADEVICES)	+= dma/
 
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD)	+= migrate_offload/
+
 # SOC specific infrastructure drivers.
 obj-y				+= soc/
 obj-$(CONFIG_PM_GENERIC_DOMAINS)	+= pmdomain/
drivers/migrate_offload/Kconfig new file +9 -0
new file mode 100644
index 000000000000..930d8605c15d
@@ -0,0 +1,9 @@
+config DCBM_DMA
+	tristate "DMA Core Batch Migrator"
+	depends on MIGRATION && DMA_ENGINE
+	select MIGRATION_COPY_OFFLOAD
+	help
+	  DMA-based batch copy engine for page migration. Uses
+	  DMAEngine memcpy channels to offload folio data copies
+	  during migration. Primarily intended for testing the copy
+	  offload infrastructure.
drivers/migrate_offload/Makefile new file +1 -0
new file mode 100644
index 000000000000..9e16018beb15
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA)		+= dcbm/
drivers/migrate_offload/dcbm/Makefile new file +1 -0
new file mode 100644
index 000000000000..56ba47cce0f1
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA) += dcbm.o
drivers/migrate_offload/dcbm/dcbm.c new file +440 -0

440 lines not on this page · raw message

Signature · 1 line
2.43.0
Reviewed-by on v1 6/7
#40 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

In my opinion the driver should be at drivers/dma/dcbm, other DMA drivers in the same directory like idxd and sdxi can just export a struct dma_migrator. It makes it easy to have everything in the driver's own directory. For example, idxd has some internals headers as well as an already existing dmaengine implementation in drivers/dma/idxd.

PS. I am writing an implementation of this based on idxd. I'll try to post it soon with numbers.

You can add

Signature · 1 line
~karim
#41 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

Quoted 8 lines · Karim Manaouil

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

In my opinion the driver should be at drivers/dma/dcbm, other DMA drivers in the same directory like idxd and sdxi can just export a struct dma_migrator. It makes it easy to have everything in the driver's own directory. For example, idxd has some internals headers as well as an already existing dmaengine implementation in drivers/dma/idxd.

Quoted 6 lines

PS. I am writing an implementation of this based on idxd. I'll try to post it soon with numbers.

You can add

Hi Karim,

Thanks for the review.

I'm excited to see your idxd based implementation.

For plain memory to memory copy, which is all migration needs right now, the public dmaengine API is good enough, and idxd/sdxi/ptdma ... can all work through it. I'm curious which features you're planning to use from idxd internal headers that dmaengine can't provide?

Also, my understanding is that driver/dma houses DMA providers, whereas DCBM would be consumer of DMAengine APIs. So is that right place for it?

Best regards,
Shivank

#42 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

Quoted 20 lines · Shivank
Quoted 8 lines · Karim Manaouil

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

In my opinion the driver should be at drivers/dma/dcbm, other DMA drivers in the same directory like idxd and sdxi can just export a struct dma_migrator. It makes it easy to have everything in the driver's own directory. For example, idxd has some internals headers as well as an already existing dmaengine implementation in drivers/dma/idxd.

Quoted 6 lines

PS. I am writing an implementation of this based on idxd. I'll try to post it soon with numbers.

You can add

Hi Karim,

Quoted 8 lines

Thanks for the review.

I'm excited to see your idxd based implementation.

For plain memory to memory copy, which is all migration needs right now, the public dmaengine API is good enough, and idxd/sdxi/ptdma ... can all work through it. I'm curious which features you're planning to use from idxd internal headers that dmaengine can't provide?

Hi Shivank,

I'm still testing, but the initial implementation I wrote with DMAEngine had too much overhead because of the sgtable allocations and the conversion between kernel scatterlists to device descriptors. So I entirely bypassed the DMAEngine API by directly passing the folios lists to the driver.

I know it depends on the use case. If you just want to offload with no latency requirements, then DMAEngine is fine, but if the goal is to achieve high bandwidth with minimal latency, then it's a problem.

Another example, if you have to do several independent copies of 256 or 512 4KiB pages in a short period of time, there will to much stress on sgtable allocations.

Another problem for low latency is DMA mapping.

Anyway, I need to collect more numbers. I will try to share my insights with idxd asap.

Quoted 3 lines

Also, my understanding is that driver/dma houses DMA providers, whereas DCBM would be consumer of DMAengine APIs. So is that right place for it?

I think you're right. It makes more sense when viewed from that perspective. In fact, maybe at some point we will have some more generic memory offload infrastructure to support sharing, load balancing across several acceelrator instances, numa-awareness, handling batching for drivers and asynchronous offload (to workqueues with completions).

Signature · 1 line
~karim
#43 lore raw

...

Quoted 10 lines

I'm still testing, but the initial implementation I wrote with DMAEngine had too much overhead because of the sgtable allocations and the conversion between kernel scatterlists to device descriptors. So I entirely bypassed the DMAEngine API by directly passing the folios lists to the driver.

I know it depends on the use case. If you just want to offload with no latency requirements, then DMAEngine is fine, but if the goal is to achieve high bandwidth with minimal latency, then it's a problem.

Quoted 8 lines

Another example, if you have to do several independent copies of 256 or 512 4KiB pages in a short period of time, there will to much stress on sgtable allocations.

Another problem for low latency is DMA mapping.

Anyway, I need to collect more numbers. I will try to share my insights with idxd asap.

Thanks, looking forward to those insights and numbers.

An IDXD specific implementation is good for experimentation, but for upstream path, I think this would be hard to maintain and add duplicate logic. The cleanest approch is the DMA_MEMCPY_SG API. So, a single offload driver can drive any engine that implements it. dmaengine_prep_dma_memcpy_sg() submits a whole src/dst scatterlist as one transaction, which cuts the per-descriptor setup overhead that dominates for 4KB pages.

I've added a patch for dmaengine_prep_dma_memcpy_sg(), Could you look into wiring up device_prep_dma_memcpy_sg hook in the IDXD?

This will keep it generic and address the bandwidth/latency problem for small transfers.

Best Regards,
Shivank

---

From d60e4dcd58552cbc6c7ecee37df9e216ab0c5a4f Mon Sep 17 00:00:00 2001
From: Shivank Garg <[email protected]>
Date: Sun, 17 May 2026 18:19:10 +0000
Subject: [PATCH] dmaengine: add DMA_MEMCPY_SG transaction type

Currently, a client that wants to copy N (src, dst, len) tuples call dmaengine_prep_dma_memcpy() N times, allocating N independent dma_async_tx_descriptor. The provider has to treat them as independent transactions, even when the underlying hardware can program them as one descriptor group with a single completion. This overheads are dominant for page size like 4KB.

To reduce this overheads, add DMA_MEMCPY_SG and the matching device_prep_dma_memcpy_sg() callback taking paired src and dst scatterlists, plus the dmaengine_prep_dma_memcpy_sg() inline wrapper. The provider walks both lists in lockstep and retires the whole batch as one async tx.

This API was removed by commit 0cae04373b77 ("dmaengine: remove DMA_MEMCPY_SG once again") for lack of in-tree users. The user this time is the page-migration copy offload driver.

---
 .../driver-api/dmaengine/provider.rst         |  7 ++++
 drivers/dma/dmaengine.c                       |  1 +
 include/linux/dmaengine.h                     | 36 +++++++++++++++++++
 3 files changed, 44 insertions(+)
Documentation/driver-api/dmaengine/provider.rst +7 -0
index f4ed98f701c9..fc3cab78e61a 100644
@@ -175,6 +175,13 @@ Currently, the types available are:
     ``glReadPixels()``, which might require a verbatim copy of a huge
     framebuffer from local device memory onto host memory.
 
+- DMA_MEMCPY_SG
+
+  - The device is able to do  memory to memory scatter-gather transfers.
+
+  - This takes pair of src and dst scatterlists and retires the whole batch
+    as one async tx.
+
 - DMA_XOR
 
   - The device is able to perform XOR operations on memory areas
drivers/dma/dmaengine.c +1 -0
index 405bd2fbb4a3..665e00a96c8f 100644
@@ -1197,6 +1197,7 @@ int dma_async_device_register(struct dma_device *device)
 }
 
 	CHECK_CAP(dma_memcpy,      DMA_MEMCPY);
+	CHECK_CAP(dma_memcpy_sg,   DMA_MEMCPY_SG);
 	CHECK_CAP(dma_xor,         DMA_XOR);
 	CHECK_CAP(dma_xor_val,     DMA_XOR_VAL);
 	CHECK_CAP(dma_pq,          DMA_PQ);
include/linux/dmaengine.h +36 -0
index b3d251c9734e..9378e7a7803a 100644
@@ -50,6 +50,7 @@ enum dma_status {
  */
 enum dma_transaction_type {
 	DMA_MEMCPY,
+	DMA_MEMCPY_SG,
 	DMA_XOR,
 	DMA_PQ,
 	DMA_XOR_VAL,
@@ -824,6 +825,7 @@ struct dma_filter {
  * @device_router_config: optional callback for DMA router configuration
  * @device_free_chan_resources: release DMA channel's resources
  * @device_prep_dma_memcpy: prepares a memcpy operation
+ * @device_prep_dma_memcpy_sg: prepares a memcpy operation over a scatter list
  * @device_prep_dma_xor: prepares a xor operation
  * @device_prep_dma_xor_val: prepares a xor validation operation
  * @device_prep_dma_pq: prepares a pq operation
@@ -903,6 +905,11 @@ struct dma_device {
 	struct dma_async_tx_descriptor *(*device_prep_dma_memcpy)(
 		struct dma_chan *chan, dma_addr_t dst, dma_addr_t src,
 		size_t len, unsigned long flags);
+	struct dma_async_tx_descriptor *(*device_prep_dma_memcpy_sg)(
+		struct dma_chan *chan,
+		struct scatterlist *dst_sg, unsigned int dst_nents,
+		struct scatterlist *src_sg, unsigned int src_nents,
+		unsigned long flags);
 	struct dma_async_tx_descriptor *(*device_prep_dma_xor)(
 		struct dma_chan *chan, dma_addr_t dst, dma_addr_t *src,
 		unsigned int src_cnt, size_t len, unsigned long flags);
@@ -1091,6 +1098,35 @@ static inline struct dma_async_tx_descriptor *dmaengine_prep_dma_memcpy(
 						    len, flags);
 }
 
+/**
+ * dmaengine_prep_dma_memcpy_sg - Prepare a scatter-gather memcpy
+ * @chan: The channel to prepare on
+ * @dst_sg: Destination scatterlist
+ * @dst_nents: Number of mapped entries in @dst_sg
+ * @src_sg: Source scatterlist
+ * @src_nents: Number of mapped entries in @src_sg
+ * @flags: Dmaengine flags (e.g. DMA_PREP_INTERRUPT, DMA_CTRL_ACK)
+ *
+ * Submit a batch of memcpy operations described by two scatterlists as
+ * a single async transaction. All segments retire as one tx with one
+ * completion.
+ *
+ * Returns NULL if the channel does not implement this operation.
+ */
+static inline struct dma_async_tx_descriptor *dmaengine_prep_dma_memcpy_sg(
+		struct dma_chan *chan,
+		struct scatterlist *dst_sg, unsigned int dst_nents,
+		struct scatterlist *src_sg, unsigned int src_nents,
+		unsigned long flags)
+{
+	if (!chan || !chan->device ||
+	    !chan->device->device_prep_dma_memcpy_sg)
+		return NULL;
+
+	return chan->device->device_prep_dma_memcpy_sg(chan,
+			dst_sg, dst_nents, src_sg, src_nents, flags);
+}
+
 static inline bool dmaengine_is_metadata_mode_supported(struct dma_chan *chan,
 		enum dma_desc_metadata_mode mode)
 {
Signature · 1 line
2.43.0
#44 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

In reply to Shivank Garg

Hi again Shivank,

I just got some time to resume testing this on Intel Sapphire Rapids and something caught my attention, below

Quoted 20 lines · Shivank Garg
drivers/migrate_offload/dcbm/dcbm.c new file +20 -0
+static int submit_dma_transfers(struct dma_work *work)
+{
+	struct scatterlist *sg_src, *sg_dst;
+	struct dma_async_tx_descriptor *tx;
+	unsigned long flags = DMA_CTRL_ACK;
+	dma_cookie_t cookie;
+	int i;
+
+	atomic_set(&work->pending, 1);
+
+	sg_src = work->src_sgt->sgl;
+	sg_dst = work->dst_sgt->sgl;
+	for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) {
+		if (i == work->src_sgt->nents - 1)
+			flags |= DMA_PREP_INTERRUPT;
+
+		tx = dmaengine_prep_dma_memcpy(work->chan,
+				sg_dma_address(sg_dst),
+				sg_dma_address(sg_src),
+				sg_dma_len(sg_src), flags);

drivers/migrate_offload/dcbm/dcbm.c line 199

Quoted 10 lines
drivers/migrate_offload/dcbm/dcbm.c new file +10 -0
+		if (!tx) {
+			atomic_set(&work->pending, 0);
+			return -EIO;
+		}
+
+		if (i == work->src_sgt->nents - 1) {
+			tx->callback = dma_completion_callback;
+			tx->callback_param = work;
+		}
+

Here, you are submitting the descriptors one after the other and only the last descriptor has a callback, which in theory sounds correct as you expect the DMA engine to complete the descriptors in the same order they were submitted. However, in reality that's not really gauranteed.

Intel DSA in particular can complete descriptors out of order. That means, the last descriptor submitted may not necessarily be the last descriptors that completes. In that case, you will return in folios_copy_dma() before the copy truly completes for all the folios.

For correctness, we have to add a callback to every descriptor and initialize work->pending to the number of descriptors submitted then every time a descriptor completes, you call atomic_dec(&work->pending) and only complete the completion the moment it reaches zero.

Btw, waiting for an interrupt adds massive scheduling overhead. If we also add the logic above, it'll get even worse. In my measurements, this can easily add up to 6ms, by which CPU page copy have easily completed the entire copy, which again adds to the list of latency concerns I raised in my other reply.

Quoted 98 lines
drivers/migrate_offload/dcbm/dcbm.c new file +98 -0
+		cookie = dmaengine_submit(tx);
+		if (dma_submit_error(cookie)) {
+			atomic_set(&work->pending, 0);
+			return -EIO;
+		}
+		sg_dst = sg_next(sg_dst);
+	}
+	return 0;
+}
+
+/**
+ * folios_copy_dma - copy a batch of folios via DMA memcpy
+ * @dst_list: destination folio list
+ * @src_list: source folio list
+ * @nr_folios: number of folios in each list
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+static int folios_copy_dma(struct list_head *dst_list,
+		struct list_head *src_list, unsigned int nr_folios)
+{
+	struct dma_work *works;
+	struct list_head *src_pos = src_list->next;
+	struct list_head *dst_pos = dst_list->next;
+	int i, folios_per_chan, ret;
+	dma_cap_mask_t mask;
+	int actual_channels = 0;
+	unsigned int max_channels;
+
+	max_channels = min3(nr_dma_channels, nr_folios,
+			(unsigned int)MAX_DMA_CHANNELS);
+
+	works = kcalloc(max_channels, sizeof(*works), GFP_KERNEL);
+	if (!works)
+		return -ENOMEM;
+
+	dma_cap_zero(mask);
+	dma_cap_set(DMA_MEMCPY, mask);
+
+	for (i = 0; i < max_channels; i++) {
+		works[actual_channels].chan = dma_request_chan_by_mask(&mask);
+		if (IS_ERR(works[actual_channels].chan))
+			break;
+		init_completion(&works[actual_channels].done);
+		actual_channels++;
+	}
+
+	if (actual_channels == 0) {
+		kfree(works);
+		return -ENODEV;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		folios_per_chan = nr_folios * (i + 1) / actual_channels -
+				(nr_folios * i) / actual_channels;
+		if (folios_per_chan == 0)
+			continue;
+
+		ret = setup_sg_tables(&works[i], &src_pos, &dst_pos,
+				folios_per_chan);
+		if (ret)
+			goto err_cleanup;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		ret = submit_dma_transfers(&works[i]);
+		if (ret)
+			goto err_cleanup;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		if (atomic_read(&works[i].pending) > 0)
+			dma_async_issue_pending(works[i].chan);
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		if (atomic_read(&works[i].pending) == 0)
+			continue;
+		if (!wait_for_completion_timeout(&works[i].done,
+				msecs_to_jiffies(10000))) {
+			ret = -ETIMEDOUT;
+			goto err_cleanup;
+		}
+	}
+
+	cleanup_dma_work(works, actual_channels);
+
+	atomic_long_add(nr_folios, &folios_migrated);
+	return 0;
+
+err_cleanup:
+	pr_warn_ratelimited("dcbm: DMA copy failed (%d), falling back to CPU\n",
+			ret);
+	cleanup_dma_work(works, actual_channels);
+
+	atomic_long_add(nr_folios, &folios_failures);
+	return ret;
+}
Signature · 1 line
~karim
#45 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

Quoted 5 lines · Karim Manaouil

Btw, waiting for an interrupt adds massive scheduling overhead. If we also add the logic above, it'll get even worse. In my measurements, this can easily add up to 6ms, by which CPU page copy have easily completed the entire copy, which again adds to the list of latency concerns I raised in my other reply.

Sorry, I meant 6us.

Signature · 1 line
~karim
#46 lore raw

Re: [PATCH 6/7] drivers/migrate_offload: add DMA batch copy driver (dcbm)

In reply to Karim Manaouil
Quoted 54 lines · Karim Manaouil

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi again Shivank,

I just got some time to resume testing this on Intel Sapphire Rapids and something caught my attention, below

Quoted 20 lines · Shivank Garg
drivers/migrate_offload/dcbm/dcbm.c new file +20 -0
+static int submit_dma_transfers(struct dma_work *work)
+{
+     struct scatterlist *sg_src, *sg_dst;
+     struct dma_async_tx_descriptor *tx;
+     unsigned long flags = DMA_CTRL_ACK;
+     dma_cookie_t cookie;
+     int i;
+
+     atomic_set(&work->pending, 1);
+
+     sg_src = work->src_sgt->sgl;
+     sg_dst = work->dst_sgt->sgl;
+     for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) {
+             if (i == work->src_sgt->nents - 1)
+                     flags |= DMA_PREP_INTERRUPT;
+
+             tx = dmaengine_prep_dma_memcpy(work->chan,
+                             sg_dma_address(sg_dst),
+                             sg_dma_address(sg_src),
+                             sg_dma_len(sg_src), flags);

drivers/migrate_offload/dcbm/dcbm.c line 199

Quoted 10 lines
drivers/migrate_offload/dcbm/dcbm.c new file +10 -0
+             if (!tx) {
+                     atomic_set(&work->pending, 0);
+                     return -EIO;
+             }
+
+             if (i == work->src_sgt->nents - 1) {
+                     tx->callback = dma_completion_callback;
+                     tx->callback_param = work;
+             }
+

Here, you are submitting the descriptors one after the other and only the last descriptor has a callback, which in theory sounds correct as you expect the DMA engine to complete the descriptors in the same order they were submitted. However, in reality that's not really gauranteed.

Intel DSA in particular can complete descriptors out of order. That means, the last descriptor submitted may not necessarily be the last descriptors that completes. In that case, you will return in folios_copy_dma() before the copy truly completes for all the folios.

For correctness, we have to add a callback to every descriptor and initialize work->pending to the number of descriptors submitted then every time a descriptor completes, you call atomic_dec(&work->pending) and only complete the completion the moment it reaches zero.

Quoted 5 lines

Btw, waiting for an interrupt adds massive scheduling overhead. If we also add the logic above, it'll get even worse. In my measurements, this can easily add up to 6ms, by which CPU page copy have easily completed the entire copy, which again adds to the list of latency concerns I raised in my other reply.

Thanks Karim for catching this.
I was not aware that descriptor chaining was not applicable for DSA.

Going forward, implementing the device_prep_dma_memcpy_sg() fixes this broken assumption. So client will issue single transaction and see single completion for whole batch. The ordering/correctness will become provider's responsibility.

Thanks,
Shivank

#47 lore raw

PATCH 7/7 mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

In reply to Shivank Garg

From: Zi Yan <[email protected]>

Change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies.

These are for testing purpose only.

---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
index 9af070f9a1f2..a16c009d31d0 100644
@@ -1658,7 +1658,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
 }
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_MAX_BATCHED_MIGRATION	HPAGE_PMD_NR
+#define NR_MAX_BATCHED_MIGRATION	HPAGE_PUD_NR
 #else
 #define NR_MAX_BATCHED_MIGRATION	512
 #endif
Signature · 1 line
2.43.0
#48 lore raw
In reply to Shivank Garg

Hi all,

Apologies. The subject prefix should have been [RFC PATCH v5 0/7].

This is the fifth RFC, as mentioned in the cover letter, but I missed the prefix while formatting the patches. Please treat this round as RFC v5.

Thanks,
Shivank

#49 lore raw
Quoted 7 lines · Shivank

Hi all,

Apologies. The subject prefix should have been [RFC PATCH v5 0/7].

This is the fifth RFC, as mentioned in the cover letter, but I missed the prefix while formatting the patches. Please treat this round as RFC v5.

Ever since I switched to b4 for patch management, the quality of my life improved :)

$ b4 prep -n SERIES -f mm/mm-unstable
$ b4 prep --set-prefixes RFC
... add patches
$ b4 prep --auto-to-cc
$ b4 prep --edit-cover
$ b4 send  --no-sign
Signature · 3 lines
Cheers,

David
#50 lore raw
Quoted 9 lines · David Hildenbrand (Arm)
Quoted 7 lines · Shivank

Hi all,

Apologies. The subject prefix should have been [RFC PATCH v5 0/7].

This is the fifth RFC, as mentioned in the cover letter, but I missed the prefix while formatting the patches. Please treat this round as RFC v5.

Quoted 9 lines

Ever since I switched to b4 for patch management, the quality of my life improved :)

$ b4 prep -n SERIES -f mm/mm-unstable
$ b4 prep --set-prefixes RFC
... add patches
$ b4 prep --auto-to-cc
$ b4 prep --edit-cover
$ b4 send  --no-sign

Thanks, appreciate the pointers. :)
I'll switch to b4.

Best regards,
Shivank

reviewer
#51 lore raw
In reply to Shivank Garg
Quoted 83 lines · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by batching folio-copy operations and enabling acceleration via DMA offload.

Single-threaded, folio-by-folio copying bottlenecks page migration in modern systems with deep memory hierarchies, especially for large folios where copy overhead dominates, leaving significant hardware potential untapped.

By batching the copy phase, we create an opportunity for hardware acceleration. This series builds the framework and provides a DMA offload driver (dcbm) as a reference implementation, targeting bulk migration workloads where offloading the copy improves throughput and latency while freeing the CPU cycles.

See the RFC V3 cover letter [2] for motivation.

Changelog since V4:
-------------------

1. Renamed PAGE_* migration state flags to FOLIO_*. (David)
2. Use the new folio->migrate_info field instead of folio->private for migration state. (David)
3. Fold folios_mc_copy patch in batch-copy implementation patch. (David)
3. Renamed migrate_offload_start()/stop() to register()/unregister(). (Huang, Ying)
4. Dropped should_batch() callback from struct migrator. Reason-based policy now lives in migrate_pages_batch(). Migrators can still skip a batch they don't want (size based policy). (Huang, Ying)
5. CONFIG_MIGRATION_COPY_OFFLOAD is now hidden and selected by the migrator driver. CONFIG_DCBM_DMA is tristate. (Huang Ying, Gregory Price).
6. Wrapped the SRCU + static_call dispatch in a small helper. (Huang, Ying)
7. Requir m->owner in migrate_offload_register(), SRCU sync at unregister relies on it. Counters are atomic_long_t to avoid lock-order issue.
9. Moved DCBM sysfs from /sys/kernel/dcbm to /sys/module/dcbm (Huang, Ying)
10. Rebased on v7.1-rc1.

DESIGN:
-------

New Migration Flow:

[ migrate_pages_batch() ]

    |
    |--> do_batch = migrate_offload_do_batch(reason)  // core filters by migration reason
    |
    |--> for each folio:
    |      migrate_folio_unmap()        // unmap the folio
    |      |
    |      +--> (success):
    |           if do_batch && folio_supports_batch_copy():
    |               -> unmap_batch / dst_batch  // batch list for copy offloading
    |           else:
    |               -> unmap_single / dst_single // single lists for per-folio CPU copy
    |
    |--> try_to_unmap_flush()                   // single batched TLB flush
    |
    |--> Batch copy (if unmap_batch not empty):
    |    - Migrator is configurable at runtime via sysfs.
    |
    |      static_call(migrate_offload_copy)    // Pluggable Migrators
    |              /          |            \
    |             v           v             v
    |     [ Default ]  [ DMA Offload ]  [ ... ]
    |
    |      On -EOPNOTSUPP or other error, batch falls back to per-folio CPU copy.
    |

+--> migrate_folios_move() // metadata, update PTEs, finalize
(batch list with already_copied=true, single list with false)

Offload Registration:

Driver fills struct migrator { .name, .offload_copy, .owner } and calls migrate_offload_register(). This:
- Pins the module via try_module_get()
- Patches the migrate_offload_copy() static_call target
- Enables the migrate_offload_enabled static branch

migrate_offload_unregister() disables the static branch and reverts the static_call, then synchronize_srcu() waits for in-flight migrations before module_put().

Quoted 7 lines

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

Quoted 131 lines

PLAN:
-----

Patches 1-4 (the batching infrastructure) don't depend on the migrator interface, so if it helps I can split them off and post them ahead of the migrator and DCBM bits, which still have a few open questions to work through.

I would appreciate guidance on splitting the infrastructure portion ahead of the migrator interface if that matches maintainers' preference.

OPEN QUESTIONS:
---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

4. Cgroup integration: How should per-cgroup be accounted for different migrators (e.g.: any accounting for DMA-busy time)?

5. Tuning migrate_pages callers for offloading. For instance, in compaction COMPACT_CLUSTER_MAX = 32 caps DMA's payoff for compaction (V4 experiment).

6. Where do batch-size thresholds live, and how are they tuned? Per Huang Ying's split, that policy lives in the migrator. DCBM has no threshold today. Open whether it should later be a per-migrator sysfs knob or hard-coded; probably clearer once a second migrator (SDXI, mtcopy) shows the trade-off.

FOLLOW-UPS:
--------------

1. dmaengine_prep_dma_memcpy_sg() in DCBM (Vinod Koul). The SG-prep variant cuts per-batch prep/submit cost (=CPU savings), but ptdma does not implement the SG hook yet [10]. The end-to-end migration throughput delta is small because per-descriptor execute time dominates. I'll post the ptdma SG hook + DCBM switch as a follow-up.

2. SDXI as a second migrator. The SDXI series [11] is in review. SDXI is a generic memcpy engine without DMA_PRIVATE, so channel acquisition goes through dma_find_channel() or async_tx rather than dma_request_chan_by_mask(). I have a local DCBM variant working on top of the SDXI driver. I'm planning to send it as a follow-up once the SDXI series settles.

3. IOMMU SG merging in DCBM (Gregory). dma_map_sgtable() may merge contiguous PFNs unevenly, so src.nents != dst.nents. DCBM falls back to CPU for safety. Though I haven't seen it on Zen3 + PTDMA. I'll understand this and address it a follow-up.

4. Revisit Multi-threaded CPU copy migrator once the infra is settled.

EARLIER POSTINGS:
-----------------
[1] RFC V4: https://lore.kernel.org/all/[email protected]
[2] RFC V3: https://lore.kernel.org/all/[email protected]
[3] RFC V2: https://lore.kernel.org/all/[email protected]
[4] RFC V1: https://lore.kernel.org/all/[email protected]
[5] RFC from Zi Yan: https://lore.kernel.org/all/[email protected]

RELATED DISCUSSIONS:
--------------------
[6] MM-alignment Session [Nov 12, 2025]:

    https://lore.kernel.org/linux-mm/[email protected]
[7] Linux Memory Hotness and Promotion call [Nov 6, 2025]:
    https://lore.kernel.org/linux-mm/[email protected]
[8] LSFMM 2025:
    https://lore.kernel.org/all/[email protected]
[9] OSS India:
    https://ossindia2025.sched.com/event/23Jk1
[10] DMA_MEMCPY_SG comparison:
     https://lore.kernel.org/linux-mm/[email protected]
[11] SDXI V1:
     https://lore.kernel.org/all/[email protected]

Thanks to everyone who reviewed, tested or participated in discussions around this series. Your feedback helped me throughout the development process.

Best Regards,
Shivank

Shivank Garg (6):
mm/migrate: rename PAGE_ migration flags to FOLIO_ mm/migrate: use migrate_info field instead of private mm/migrate: skip data copy for already-copied folios mm/migrate: add batch-copy path in migrate_pages_batch mm/migrate: add copy offload registration infrastructure drivers/migrate_offload: add DMA batch copy driver (dcbm)

Zi Yan (1):
mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 440 ++++++++++++++++++++++++++
 include/linux/migrate_copy_offload.h  |  44 +++
 include/linux/mm.h                    |   2 +
 include/linux/mm_types.h              |   1 +
 mm/Kconfig                            |   6 +
 mm/Makefile                           |   1 +
 mm/migrate.c                          | 211 ++++++++----
 mm/migrate_copy_offload.c             |  94 ++++++
 mm/util.c                             |  30 ++
 14 files changed, 784 insertions(+), 60 deletions(-)
 create mode 100644 drivers/migrate_offload/Kconfig
 create mode 100644 drivers/migrate_offload/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/dcbm.c
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c
Quoted 1 line

base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731

---
Best Regards,
Huang, Ying

#52 lore raw
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |

2). First-folio latency: Instrumented with custom tracepoints to measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB. Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

Thanks,
Shivank

reviewer
#53 lore raw

Hi, Shivank,

Quoted 49 lines · Shivank
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |
Quoted 2 lines

2). First-folio latency: Instrumented with custom tracepoints to measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

Thanks for detailed data. Per my understanding, the run time of migrate_pages_batch() may be not good enough for measuring first folio latency. IIUC, the migration procedure is something like,

for each folio
unmap
flush
for each folio

        copy
        remap ===> first folio migrated

Some tracepoint should be better to measure it.

Quoted 40 lines

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

Quoted 3 lines

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB.

Emm, so NR_MAX_BATCHED_MIGRATION could be very large? I think that it needs to be bounded. If it is too large, too many pages may be in an inaccessible state for a longer time. That will hurt the workload performance, although it is optimal for migration performance.

Quoted 36 lines

Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

Quoted 8 lines

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

---
Best Regards,
Huang, Ying

#54 lore raw
Quoted 60 lines · Ying

Hi, Shivank,

Quoted 49 lines · Shivank
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |
Quoted 2 lines

2). First-folio latency: Instrumented with custom tracepoints to measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

Thanks for detailed data. Per my understanding, the run time of migrate_pages_batch() may be not good enough for measuring first folio latency. IIUC, the migration procedure is something like,

Quoted 8 lines

for each folio
unmap
flush
for each folio

        copy
        remap ===> first folio migrated

Some tracepoint should be better to measure it.

Sorry, my earlier write-up was unclear.
For first folio latency, I add two tracepoints: one at the start of migrate_pages_batch() and one in migrate_folio_done().

I agree that the user-accessible point tracepoint should be right after remove_migration_ptes(). Though, migrate_folio_done() runs only a few operations later, and will have a constant offset, so it's unlikely to change the shape of the trade-off curve.
I'll move the tracepoint right after remove_migration_ptes() for new posting.

Quoted 41 lines
Quoted 41 lines

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

Quoted 9 lines
Quoted 3 lines

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB.

Emm, so NR_MAX_BATCHED_MIGRATION could be very large? I think that it needs to be bounded. If it is too large, too many pages may be in an inaccessible state for a longer time. That will hurt the workload performance, although it is optimal for migration performance.

Agreed, it must be bounded.

Quoted 48 lines
Quoted 36 lines

Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

Quoted 8 lines

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

---
Best Regards,
Huang, Ying

reviewer
#55 lore raw
Quoted 71 lines · Shivank
Quoted 61 lines · Ying

Hi, Shivank,

Quoted 49 lines · Shivank
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |
Quoted 3 lines

2). First-folio latency: Instrumented with custom tracepoints to
measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

Thanks for detailed data. Per my understanding, the run time of migrate_pages_batch() may be not good enough for measuring first folio latency. IIUC, the migration procedure is something like,

Quoted 8 lines

for each folio
unmap
flush
for each folio

        copy
        remap ===> first folio migrated

Some tracepoint should be better to measure it.

Quoted 8 lines

Sorry, my earlier write-up was unclear.
For first folio latency, I add two tracepoints: one at the start of migrate_pages_batch() and one in migrate_folio_done().

I agree that the user-accessible point tracepoint should be right after remove_migration_ptes(). Though, migrate_folio_done() runs only a few operations later, and will have a constant offset, so it's unlikely to change the shape of the trade-off curve.
I'll move the tracepoint right after remove_migration_ptes() for new posting.

Thanks for explanation. Trace point in migrate_folio_done() should be OK.

Quoted 45 lines
Quoted 45 lines
Quoted 44 lines

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB.

Quoted 7 lines
Quoted 5 lines

Emm, so NR_MAX_BATCHED_MIGRATION could be very large? I think that it needs to be bounded. If it is too large, too many pages may be in an inaccessible state for a longer time. That will hurt the workload performance, although it is optimal for migration performance.

Agreed, it must be bounded.

Thanks! Could you retest with bounded NR_MAX_BATCHED_MIGRATION. If the upstream default doesn't work well for you. We can find a better one that balances throughput and latency well.

Quoted 36 lines
Quoted 36 lines
Quoted 36 lines

Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

---
Best Regards,
Huang, Ying

#56 lore raw
Quoted 135 lines · Ying
Quoted 71 lines · Shivank
Quoted 61 lines · Ying

Hi, Shivank,

Quoted 49 lines · Shivank
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |
Quoted 3 lines

2). First-folio latency: Instrumented with custom tracepoints to
measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

Thanks for detailed data. Per my understanding, the run time of migrate_pages_batch() may be not good enough for measuring first folio latency. IIUC, the migration procedure is something like,

Quoted 8 lines

for each folio
unmap
flush
for each folio

        copy
        remap ===> first folio migrated

Some tracepoint should be better to measure it.

Quoted 8 lines

Sorry, my earlier write-up was unclear.
For first folio latency, I add two tracepoints: one at the start of migrate_pages_batch() and one in migrate_folio_done().

I agree that the user-accessible point tracepoint should be right after remove_migration_ptes(). Though, migrate_folio_done() runs only a few operations later, and will have a constant offset, so it's unlikely to change the shape of the trade-off curve.
I'll move the tracepoint right after remove_migration_ptes() for new posting.

Thanks for explanation. Trace point in migrate_folio_done() should be OK.

Quoted 51 lines
Quoted 50 lines
Quoted 41 lines

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

Quoted 3 lines

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB.

Emm, so NR_MAX_BATCHED_MIGRATION could be very large? I think that it needs to be bounded. If it is too large, too many pages may be in an inaccessible state for a longer time. That will hurt the workload performance, although it is optimal for migration performance.

Quoted 6 lines
Quoted 1 line

Agreed, it must be bounded.

Thanks! Could you retest with bounded NR_MAX_BATCHED_MIGRATION. If the upstream default doesn't work well for you. We can find a better one that balances throughput and latency well.

Thanks. Below tables sweep NR_MAX_BATCHED_MIGRATION from 512 up to 262144. On 2M folios, 16-channel PTDMA, the knee is at N=8192-16384 (= {16 to 32} * 512 ).

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

8192 12.56 | 2424 26.57 | 1118 58.72 | 470 *

One thing worth flagging on the "bounded default": at the upstream cap of 512 pages, migrate_pages_batch() receives at most one 2M folio per call, so PTDMA can only use one of its 16 channels per batch and the offload reduces to vanilla. (DCBM offloads one 2M folio to each channel).
The larger-N rows are what exercise the channel parallelism for PTDMA case.

"SDXI"[1] like memory-to-memory data movers should reach good throughput with just 1 channel, and thus may not require increasing the NR_MAX_BATCHED_MIGRATION for good throughput.

I'm not tying series this to specific perf default for now, the design review (batch-copy path, migrator interface, registration, static_call dispatch) is the part I'd like to converge on first, then tune the threshold after it. Does that ordering work?

[1] https://lore.kernel.org/all/[email protected]

Best regards,
Shivank

Quoted 48 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines

Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

---
Best Regards,
Huang, Ying

reviewer
#57 lore raw
Quoted 138 lines · Shivank
Quoted 137 lines · Ying
Quoted 71 lines · Shivank
Quoted 61 lines · Ying

Hi, Shivank,

Quoted 49 lines · Shivank
Quoted 10 lines · Ying
Quoted 7 lines · Shivank Garg

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

Quoted 8 lines

IMHO, it's better to copy performance data here.

In addition to the performance benefit, I want to know the downside as well. For example, the migration latency of the first folio may be longer. If so, by how much? Can you measure the batch number vs. total migration time (benefit) and first folio migration time (downside)? That can be used to determine the optimal batch number.

System Info: AMD Zen 3 EPYC server (2-sockets, 32 cores, SMT Enabled), 1 NUMA node per socket, v7.1-rc1, DVFS set to Performance, PTDMA hardware.

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant (1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):

================================================================================
4K | 16K | 64K | 256K | 1M | 2M |
================================================================================
3.31±0.18 | 5.61±0.07 | 6.66±0.03 | 7.01±0.03 | 7.13±0.08 | 11.02±0.17 |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):

============================================================================================
N channel| 4K | 16K | 64K | 256K | 1M | 2M |
============================================================================================

1      | 2.16±0.14   | 2.58±0.02   | 3.00±0.04   | 4.56±0.28   | 4.62±0.02   | 12.65±0.08  |
2      | 2.68±0.09   | 3.69±0.15   | 4.52±0.04   | 6.75±0.06   | 7.19±0.19   | 14.38±0.06  |
4      | 3.07±0.13   | 4.62±0.09   | 6.47±0.56   | 9.22±0.15   | 10.24±0.47  | 27.01±0.11  |
8      | 3.43±0.09   | 5.40±0.16   | 7.67±0.08   | 11.25±0.17  | 12.60±0.60  | 45.62±0.52  |
12     | 3.50±0.11   | 5.66±0.16   | 8.12±0.10   | 11.97±0.19  | 13.43±0.08  | 61.02±0.92  |
16     | 3.54±0.12   | 5.79±0.14   | 8.50±0.13   | 12.59±0.15  | 17.21±6.40  | 65.23±1.70  |
Quoted 3 lines

2). First-folio latency: Instrumented with custom tracepoints to
measure latency per migrate_pages_batch() call.
Result: throughput (GB/s) and first-folio latency (in microseconds), median of 10 runs.

Thanks for detailed data. Per my understanding, the run time of migrate_pages_batch() may be not good enough for measuring first folio latency. IIUC, the migration procedure is something like,

Quoted 8 lines

for each folio
unmap
flush
for each folio

        copy
        remap ===> first folio migrated

Some tracepoint should be better to measure it.

Quoted 8 lines

Sorry, my earlier write-up was unclear.
For first folio latency, I add two tracepoints: one at the start of migrate_pages_batch() and one in migrate_folio_done().

I agree that the user-accessible point tracepoint should be right after remove_migration_ptes(). Though, migrate_folio_done() runs only a few operations later, and will have a constant offset, so it's unlikely to change the shape of the trade-off curve.
I'll move the tracepoint right after remove_migration_ptes() for new posting.

Thanks for explanation. Trace point in migrate_folio_done() should be OK.

Quoted 52 lines
Quoted 41 lines
Quoted 41 lines

A). Vanilla Kernel:

Here, n = workload size passed to move_pages() in folios. Move n number of folios with move_pages(). NR_MAX_BATCHED_MIGRATION is upstream default value 512.

--- Order 0 (4K folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       0.04 |     24
     4       0.16 |     25
     8       0.29 |     31
    16       0.54 |     27
    64       1.15 |     68
   256       1.86 |    162
   512       2.21 |    264
  2048       2.62 |    208
  4096       2.74 |    182
 16384       2.73 |    173
 65536       3.28 |    166

262144 3.20 | 167

--- Order 9 (2M folios) ---
n vanilla/cpu
(folios) GB/s | first(us)
--------------------------

     1       7.05 |    194
     4       8.78 |    186
     8       8.47 |    188
    16       7.20 |    193
    64       8.23 |    191
   256      10.51 |    180
   512      10.88 |    173

Takeaway:
In each migrate_pages_batch() call, folios are first unmapped, then try_to_unmap_flush(), and only then folios enter move_to_new_folio(). So first-folio latency is bounded by the per-batch unmap+flush cost, and then plateaus once workload is large enough.

Quoted 9 lines
Quoted 3 lines

B). Patched kernel:

Here, N = NR_MAX_BATCHED_MIGRATION (in page). Total migrated data is fixed at 1 GB.

Emm, so NR_MAX_BATCHED_MIGRATION could be very large? I think that it needs to be bounded. If it is too large, too many pages may be in an inaccessible state for a longer time. That will hurt the workload performance, although it is optimal for migration performance.

Agreed, it must be bounded.

Quoted 9 lines
Quoted 4 lines

Thanks! Could you retest with bounded NR_MAX_BATCHED_MIGRATION. If the upstream default doesn't work well for you. We can find a better one that balances throughput and latency well.

Thanks. Below tables sweep NR_MAX_BATCHED_MIGRATION from 512 up to 262144. On 2M folios, 16-channel PTDMA, the knee is at N=8192-16384 (= {16 to 32} * 512 ).

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

8192 12.56 | 2424 26.57 | 1118 58.72 | 470 *

        2048         12.30 |    613        25.47 |    290        25.48 |    291

IIUC, N=2048 already helps dma4. And, the latency looks OK too. The good batch size is hardware configuration dependent too? If so, we may need to add another migrator callback for that.

Quoted 6 lines

One thing worth flagging on the "bounded default": at the upstream cap of 512 pages, migrate_pages_batch() receives at most one 2M folio per call, so PTDMA can only use one of its 16 channels per batch and the offload reduces to vanilla. (DCBM offloads one 2M folio to each channel).
The larger-N rows are what exercise the channel parallelism for PTDMA case.

Quoted 6 lines

"SDXI"[1] like memory-to-memory data movers should reach good throughput with just 1 channel, and thus may not require increasing the NR_MAX_BATCHED_MIGRATION for good throughput.

I'm not tying series this to specific perf default for now, the design review (batch-copy path, migrator interface, registration, static_call dispatch) is the part I'd like to converge on first, then tune the threshold after it. Does that ordering work?

IMHO, we need some performance data to justify the added complexity. So, threshold tuning isn't the goal, whether we can get better throughput with some bounded latency is.

Quoted 50 lines

[1] https://lore.kernel.org/all/[email protected]

Best regards,
Shivank

Quoted 45 lines
Quoted 44 lines
Quoted 44 lines
Quoted 44 lines

Change N with a knob to measure impact of different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.13 |    639         3.23 |    290         3.27 |    253
  1024         2.17 |   1261         3.44 |    582         3.58 |    536
  2048         2.01 |   2769         3.09 |   1360         3.45 |   1083
  4096         2.10 |   5059         3.13 |   2737         3.58 |   2115 
  8192         2.21 |   9320         3.17 |   5015         3.75 |   3617 
 16384         2.15 |  18689         3.31 |   9623         3.87 |   6937
 32768         2.12 |  42692         3.38 |  18893         3.83 |  14255
 65536         2.09 |  81956         3.38 |  38556         3.64 |  29003
131072         2.02 | 169563         3.22 |  81082         3.63 |  62236
262144         2.21 | 318424         3.12 | 170174         3.50 | 129413

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
-------------------------------------------------------------------------
   512         11.66 |    160        11.68 |    160        11.65 |    160
  1024         12.16 |    310        13.67 |    275        13.64 |    276
  2048         12.30 |    613        25.47 |    290        25.48 |    291
  4096         12.48 |   1215        26.19 |    566        42.59 |    335
  8192         12.56 |   2424        26.57 |   1118        58.72 |    470 *
 16384         12.61 |   4839        26.77 |   2218        61.94 |    896
 32768         12.60 |   9667        26.98 |   4422        63.75 |   1748
 65536         12.63 |  19318        26.99 |   8838        60.66 |   3543
131072         12.64 |  38935        27.02 |  17935        61.06 |   7178
262144         12.66 |  77694        26.85 |  35871        65.06 |  14129

In the batch-copy offload approach, DMA copy phase is inserted between unmap/flush and move, So larger N increases first-folio wall clock latency. Throughput improves but with diminishing returns.

For DCBM+PTDMA setup, the optimal batch for 2M folios sits around N=8192-16384, because a larger batch allows the driver to distribute more folios across available DMA channels. This is where we get most throughput while keeping the first folio latency in check.

This optimal batch value is hardware-specific. Other engines (eg. SDXI) and memory tier (eg. CXL) will likely have different curves.

Does this approach and experiment look good to you?

---
Best Regards,
Huang, Ying

#58 lore raw
Quoted 18 lines · Ying
Quoted 16 lines · Shivank
Quoted 10 lines · Ying
Quoted 8 lines · Shivank
Quoted 7 lines · Ying

Hi, Shivank,

Quoted 3 lines · Shivank
Quoted 1 line · Ying

Shivank Garg <[email protected]> writes:

Thanks. Below tables sweep NR_MAX_BATCHED_MIGRATION from 512 up to 262144. On 2M folios, 16-channel PTDMA, the knee is at N=8192-16384 (= {16 to 32} * 512 ).

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

8192 12.56 | 2424 26.57 | 1118 58.72 | 470 *

        2048         12.30 |    613        25.47 |    290        25.48 |    291

IIUC, N=2048 already helps dma4. And, the latency looks OK too. The good batch size is hardware configuration dependent too? If so, we may need to add another migrator callback for that.

Yeah, right.

Quoted 9 lines
Quoted 9 lines

One thing worth flagging on the "bounded default": at the upstream cap of 512 pages, migrate_pages_batch() receives at most one 2M folio per call, so PTDMA can only use one of its 16 channels per batch and the offload reduces to vanilla. (DCBM offloads one 2M folio to each channel).
The larger-N rows are what exercise the channel parallelism for PTDMA case.

"SDXI"[1] like memory-to-memory data movers should reach good throughput with just 1 channel, and thus may not require increasing the NR_MAX_BATCHED_MIGRATION for good throughput.

Quoted 8 lines
Quoted 3 lines

I'm not tying series this to specific perf default for now, the design review (batch-copy path, migrator interface, registration, static_call dispatch) is the part I'd like to converge on first, then tune the threshold after it. Does that ordering work?

IMHO, we need some performance data to justify the added complexity. So, threshold tuning isn't the goal, whether we can get better throughput with some bounded latency is.

Fair point.

As you pointed PTDMA-4chan gives both tput and inaccessible time improvement. PTDMA 16-chan could be preferred by some who value throuhgput more. And this is different for different hardware. So, I think another callback for this sounds like good idea.

Thanks,
Shivank

reviewer
#59 lore raw
In reply to Shivank Garg
Quoted 112 lines · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by batching folio-copy operations and enabling acceleration via DMA offload.

Single-threaded, folio-by-folio copying bottlenecks page migration in modern systems with deep memory hierarchies, especially for large folios where copy overhead dominates, leaving significant hardware potential untapped.

By batching the copy phase, we create an opportunity for hardware acceleration. This series builds the framework and provides a DMA offload driver (dcbm) as a reference implementation, targeting bulk migration workloads where offloading the copy improves throughput and latency while freeing the CPU cycles.

See the RFC V3 cover letter [2] for motivation.

Changelog since V4:
-------------------

1. Renamed PAGE_* migration state flags to FOLIO_*. (David)
2. Use the new folio->migrate_info field instead of folio->private for migration state. (David)
3. Fold folios_mc_copy patch in batch-copy implementation patch. (David)
3. Renamed migrate_offload_start()/stop() to register()/unregister(). (Huang, Ying)
4. Dropped should_batch() callback from struct migrator. Reason-based policy now lives in migrate_pages_batch(). Migrators can still skip a batch they don't want (size based policy). (Huang, Ying)
5. CONFIG_MIGRATION_COPY_OFFLOAD is now hidden and selected by the migrator driver. CONFIG_DCBM_DMA is tristate. (Huang Ying, Gregory Price).
6. Wrapped the SRCU + static_call dispatch in a small helper. (Huang, Ying)
7. Requir m->owner in migrate_offload_register(), SRCU sync at unregister relies on it. Counters are atomic_long_t to avoid lock-order issue.
9. Moved DCBM sysfs from /sys/kernel/dcbm to /sys/module/dcbm (Huang, Ying)
10. Rebased on v7.1-rc1.

DESIGN:
-------

New Migration Flow:

[ migrate_pages_batch() ]

    |
    |--> do_batch = migrate_offload_do_batch(reason)  // core filters by migration reason
    |
    |--> for each folio:
    |      migrate_folio_unmap()        // unmap the folio
    |      |
    |      +--> (success):
    |           if do_batch && folio_supports_batch_copy():
    |               -> unmap_batch / dst_batch  // batch list for copy offloading
    |           else:
    |               -> unmap_single / dst_single // single lists for per-folio CPU copy
    |
    |--> try_to_unmap_flush()                   // single batched TLB flush
    |
    |--> Batch copy (if unmap_batch not empty):
    |    - Migrator is configurable at runtime via sysfs.
    |
    |      static_call(migrate_offload_copy)    // Pluggable Migrators
    |              /          |            \
    |             v           v             v
    |     [ Default ]  [ DMA Offload ]  [ ... ]
    |
    |      On -EOPNOTSUPP or other error, batch falls back to per-folio CPU copy.
    |

+--> migrate_folios_move() // metadata, update PTEs, finalize
(batch list with already_copied=true, single list with false)

Offload Registration:

Driver fills struct migrator { .name, .offload_copy, .owner } and calls migrate_offload_register(). This:
- Pins the module via try_module_get()
- Patches the migrate_offload_copy() static_call target
- Enables the migrate_offload_enabled static branch

migrate_offload_unregister() disables the static branch and reverts the static_call, then synchronize_srcu() waits for in-flight migrations before module_put().

PERFORMANCE RESULTS:
--------------------

Re-ran the V4 workload on v7.1-rc1 with this series; relative speedups match V4 (~6x for 2MB folios at 16 DMA channels). No design change in V5 alters this picture; please refer to the V4 cover letter for the throughput tables [1].

PLAN:
-----

Patches 1-4 (the batching infrastructure) don't depend on the migrator interface, so if it helps I can split them off and post them ahead of the migrator and DCBM bits, which still have a few open questions to work through.

I would appreciate guidance on splitting the infrastructure portion ahead of the migrator interface if that matches maintainers' preference.

OPEN QUESTIONS:
---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

Quoted 4 lines

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

Personally, I still prefer to change migrate_folio() callbacks for better readability.

Quoted 106 lines

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

4. Cgroup integration: How should per-cgroup be accounted for different migrators (e.g.: any accounting for DMA-busy time)?

5. Tuning migrate_pages callers for offloading. For instance, in compaction COMPACT_CLUSTER_MAX = 32 caps DMA's payoff for compaction (V4 experiment).

6. Where do batch-size thresholds live, and how are they tuned? Per Huang Ying's split, that policy lives in the migrator. DCBM has no threshold today. Open whether it should later be a per-migrator sysfs knob or hard-coded; probably clearer once a second migrator (SDXI, mtcopy) shows the trade-off.

FOLLOW-UPS:
--------------

1. dmaengine_prep_dma_memcpy_sg() in DCBM (Vinod Koul). The SG-prep variant cuts per-batch prep/submit cost (=CPU savings), but ptdma does not implement the SG hook yet [10]. The end-to-end migration throughput delta is small because per-descriptor execute time dominates. I'll post the ptdma SG hook + DCBM switch as a follow-up.

2. SDXI as a second migrator. The SDXI series [11] is in review. SDXI is a generic memcpy engine without DMA_PRIVATE, so channel acquisition goes through dma_find_channel() or async_tx rather than dma_request_chan_by_mask(). I have a local DCBM variant working on top of the SDXI driver. I'm planning to send it as a follow-up once the SDXI series settles.

3. IOMMU SG merging in DCBM (Gregory). dma_map_sgtable() may merge contiguous PFNs unevenly, so src.nents != dst.nents. DCBM falls back to CPU for safety. Though I haven't seen it on Zen3 + PTDMA. I'll understand this and address it a follow-up.

4. Revisit Multi-threaded CPU copy migrator once the infra is settled.

EARLIER POSTINGS:
-----------------
[1] RFC V4: https://lore.kernel.org/all/[email protected]
[2] RFC V3: https://lore.kernel.org/all/[email protected]
[3] RFC V2: https://lore.kernel.org/all/[email protected]
[4] RFC V1: https://lore.kernel.org/all/[email protected]
[5] RFC from Zi Yan: https://lore.kernel.org/all/[email protected]

RELATED DISCUSSIONS:
--------------------
[6] MM-alignment Session [Nov 12, 2025]:

    https://lore.kernel.org/linux-mm/[email protected]
[7] Linux Memory Hotness and Promotion call [Nov 6, 2025]:
    https://lore.kernel.org/linux-mm/[email protected]
[8] LSFMM 2025:
    https://lore.kernel.org/all/[email protected]
[9] OSS India:
    https://ossindia2025.sched.com/event/23Jk1
[10] DMA_MEMCPY_SG comparison:
     https://lore.kernel.org/linux-mm/[email protected]
[11] SDXI V1:
     https://lore.kernel.org/all/[email protected]

Thanks to everyone who reviewed, tested or participated in discussions around this series. Your feedback helped me throughout the development process.

Best Regards,
Shivank

Shivank Garg (6):
mm/migrate: rename PAGE_ migration flags to FOLIO_ mm/migrate: use migrate_info field instead of private mm/migrate: skip data copy for already-copied folios mm/migrate: add batch-copy path in migrate_pages_batch mm/migrate: add copy offload registration infrastructure drivers/migrate_offload: add DMA batch copy driver (dcbm)

Zi Yan (1):
mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 440 ++++++++++++++++++++++++++
 include/linux/migrate_copy_offload.h  |  44 +++
 include/linux/mm.h                    |   2 +
 include/linux/mm_types.h              |   1 +
 mm/Kconfig                            |   6 +
 mm/Makefile                           |   1 +
 mm/migrate.c                          | 211 ++++++++----
 mm/migrate_copy_offload.c             |  94 ++++++
 mm/util.c                             |  30 ++
 14 files changed, 784 insertions(+), 60 deletions(-)
 create mode 100644 drivers/migrate_offload/Kconfig
 create mode 100644 drivers/migrate_offload/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/Makefile
 create mode 100644 drivers/migrate_offload/dcbm/dcbm.c
 create mode 100644 include/linux/migrate_copy_offload.h
 create mode 100644 mm/migrate_copy_offload.c
Quoted 1 line

base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731

---
Best Regards,
Huang, Ying

#60 lore raw
Quoted 9 lines
Quoted 9 lines

---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

Quoted 7 lines
Quoted 4 lines

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

Personally, I still prefer to change migrate_folio() callbacks for better readability.

Can that be added as a cleanup on top?

Signature · 3 lines
Cheers,

David
reviewer
#61 lore raw
Quoted 13 lines · David Hildenbrand (Arm)
Quoted 13 lines
Quoted 12 lines

---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

Quoted 4 lines
Quoted 2 lines

Personally, I still prefer to change migrate_folio() callbacks for better readability.

Can that be added as a cleanup on top?

It sounds good to me.

---
Best Regards,
Huang, Ying

#62 lore raw
In reply to Shivank Garg
Quoted 1 line · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by

Ah, this is an RFC ...

... I suggest b4 for patch series management :P

That also explains why patch #7 is still in there.

Quoted 36 lines

batching folio-copy operations and enabling acceleration via DMA offload.

Single-threaded, folio-by-folio copying bottlenecks page migration in modern systems with deep memory hierarchies, especially for large folios where copy overhead dominates, leaving significant hardware potential untapped.

By batching the copy phase, we create an opportunity for hardware acceleration. This series builds the framework and provides a DMA offload driver (dcbm) as a reference implementation, targeting bulk migration workloads where offloading the copy improves throughput and latency while freeing the CPU cycles.

See the RFC V3 cover letter [2] for motivation.

Changelog since V4:
-------------------

1. Renamed PAGE_* migration state flags to FOLIO_*. (David)
2. Use the new folio->migrate_info field instead of folio->private for migration state. (David)
3. Fold folios_mc_copy patch in batch-copy implementation patch. (David)
3. Renamed migrate_offload_start()/stop() to register()/unregister(). (Huang, Ying)
4. Dropped should_batch() callback from struct migrator. Reason-based policy now lives in migrate_pages_batch(). Migrators can still skip a batch they don't want (size based policy). (Huang, Ying)
5. CONFIG_MIGRATION_COPY_OFFLOAD is now hidden and selected by the migrator driver. CONFIG_DCBM_DMA is tristate. (Huang Ying, Gregory Price).
6. Wrapped the SRCU + static_call dispatch in a small helper. (Huang, Ying)
7. Requir m->owner in migrate_offload_register(), SRCU sync at unregister relies on it. Counters are atomic_long_t to avoid lock-order issue.
9. Moved DCBM sysfs from /sys/kernel/dcbm to /sys/module/dcbm (Huang, Ying)
10. Rebased on v7.1-rc1.

[...]

Quoted 9 lines

OPEN QUESTIONS:
---------------

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

Hiding that detail from migrate.c sounds interesting.

Quoted 9 lines

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

Isn't it sufficient to just do it based on the #folios or sth like that?

If someone migrates a handful of folios, latency is likely more important (and batching less beneficial).

I'd assume when migrating many folios, batching could just always be done. Or what's the concern?

Quoted 3 lines

4. Cgroup integration: How should per-cgroup be accounted for different migrators (e.g.: any accounting for DMA-busy time)?

Oh. Do we even have to mess with that?

Quoted 4 lines

5. Tuning migrate_pages callers for offloading. For instance, in compaction COMPACT_CLUSTER_MAX = 32 caps DMA's payoff for compaction (V4 experiment).

Is that HW dependent?

Quoted 6 lines

6. Where do batch-size thresholds live, and how are they tuned? Per Huang Ying's split, that policy lives in the migrator. DCBM has no threshold today. Open whether it should later be a per-migrator sysfs knob or hard-coded; probably clearer once a second migrator (SDXI, mtcopy) shows the trade-off.

Again, sounds like being HW dependent, no?

Signature · 3 lines
Cheers,

David
reviewer
#63 lore raw
Quoted 2 lines · David Hildenbrand (Arm)
Quoted 1 line · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by

[snip]

Quoted 6 lines
Quoted 5 lines

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

Quoted 7 lines

Isn't it sufficient to just do it based on the #folios or sth like that?

If someone migrates a handful of folios, latency is likely more important (and batching less beneficial).

I'd assume when migrating many folios, batching could just always be done. Or what's the concern?

IIUC, for callers like migrate_pages syscall, it's possible that almost all folios of a process are passed to migrate_pages(). However, I think that we still need to keep the folio inaccessible time reasonable.

[snip]

---
Best Regards,
Huang, Ying

#64 lore raw
Quoted 17 lines · Ying
Quoted 2 lines · David Hildenbrand (Arm)
Quoted 1 line · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by

[snip]

Quoted 11 lines
Quoted 5 lines

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

Isn't it sufficient to just do it based on the #folios or sth like that?

If someone migrates a handful of folios, latency is likely more important (and batching less beneficial).

Quoted 6 lines
Quoted 2 lines

I'd assume when migrating many folios, batching could just always be done. Or what's the concern?

IIUC, for callers like migrate_pages syscall, it's possible that almost all folios of a process are passed to migrate_pages(). However, I think that we still need to keep the folio inaccessible time reasonable.

Wouldn't we still want to process them in batches, only affecting folios in a batch at one point in time, not the whole address space?

Signature · 3 lines
Cheers,

David
reviewer
#65 lore raw
Quoted 21 lines · David Hildenbrand (Arm)
Quoted 20 lines · Ying
Quoted 2 lines · David Hildenbrand (Arm)
Quoted 1 line · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by

[snip]

Quoted 13 lines
Quoted 5 lines

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

Isn't it sufficient to just do it based on the #folios or sth like that?

If someone migrates a handful of folios, latency is likely more important (and batching less beneficial).

I'd assume when migrating many folios, batching could just always be done. Or what's the concern?

Quoted 6 lines
Quoted 3 lines

IIUC, for callers like migrate_pages syscall, it's possible that almost all folios of a process are passed to migrate_pages(). However, I think that we still need to keep the folio inaccessible time reasonable.

Wouldn't we still want to process them in batches, only affecting folios in a batch at one point in time, not the whole address space?

Sorry, my previous reply was confusing. Let me try to be clearer.

I think we need to distinguish between two kinds of latency. One is the latency to migrate folios; the other is the latency during which a core cannot access memory (while unmapped during migration).

IIUC, most users want the second one to be as short as possible. One possible user that does not care as much is the folio demoter, which migrates cold higher-tier folios to lower tier after they have not been accessed for some time.

---
Best Regards,
Huang, Ying

#66 lore raw
In reply to David Hildenbrand (Arm)
Quoted 9 lines · David Hildenbrand (Arm)
Quoted 1 line · Shivank Garg

This is the fifth RFC of the patchset to enhance page migration by

Ah, this is an RFC ...

... I suggest b4 for patch series management :P

That also explains why patch #7 is still in there.

yes, started using it :)

Patch 7 is for testing only but I need to think on optimum batch-size for offload which depends on HW, or have a callback as per Huang Ying's suggestion.

Quoted 43 lines
Quoted 36 lines

batching folio-copy operations and enabling acceleration via DMA offload.

Single-threaded, folio-by-folio copying bottlenecks page migration in modern systems with deep memory hierarchies, especially for large folios where copy overhead dominates, leaving significant hardware potential untapped.

By batching the copy phase, we create an opportunity for hardware acceleration. This series builds the framework and provides a DMA offload driver (dcbm) as a reference implementation, targeting bulk migration workloads where offloading the copy improves throughput and latency while freeing the CPU cycles.

See the RFC V3 cover letter [2] for motivation.

Changelog since V4:
-------------------

1. Renamed PAGE_* migration state flags to FOLIO_*. (David)
2. Use the new folio->migrate_info field instead of folio->private for migration state. (David)
3. Fold folios_mc_copy patch in batch-copy implementation patch. (David)
3. Renamed migrate_offload_start()/stop() to register()/unregister(). (Huang, Ying)
4. Dropped should_batch() callback from struct migrator. Reason-based policy now lives in migrate_pages_batch(). Migrators can still skip a batch they don't want (size based policy). (Huang, Ying)
5. CONFIG_MIGRATION_COPY_OFFLOAD is now hidden and selected by the migrator driver. CONFIG_DCBM_DMA is tristate. (Huang Ying, Gregory Price).
6. Wrapped the SRCU + static_call dispatch in a small helper. (Huang, Ying)
7. Requir m->owner in migrate_offload_register(), SRCU sync at unregister relies on it. Counters are atomic_long_t to avoid lock-order issue.
9. Moved DCBM sysfs from /sys/kernel/dcbm to /sys/module/dcbm (Huang, Ying)
10. Rebased on v7.1-rc1.

[...]

Quoted 4 lines

OPEN QUESTIONS:
---------------

Quoted 8 lines
Quoted 5 lines

1. Should the batch path run without a registered migrator? Patches 1-4 are self-contained and use folios_mc_copy() (CPU). I have several options like making batch path always-on for eligible folios, or giving admin an option to flip the static branch, or keep the gate. I'm leaning toward always-on.

Hiding that detail from migrate.c sounds interesting.

Yes, will do that.

Quoted 9 lines
Quoted 8 lines

2. Carrying already_copied via folio->migrate_info vs changing the migrate_folio() callback signature (Huang, Ying). I went with the field for now to avoid touching every fs callback before the design settles. Happy to revisit.

3. Per-caller offload selection: Today eligibility is by migrate_reason only. Some are latency-tolerant, others may be not. Is reason the right granularity, or do we want a per-caller hint?

Quoted 8 lines

Isn't it sufficient to just do it based on the #folios or sth like that?

If someone migrates a handful of folios, latency is likely more important (and batching less beneficial).

I'd assume when migrating many folios, batching could just always be done. Or what's the concern?

It could be a requirement for some users who want only specific use cases to go through DMA offload.

I agree with your point, and will discuss more on it.

Quoted 5 lines
Quoted 3 lines

4. Cgroup integration: How should per-cgroup be accounted for different migrators (e.g.: any accounting for DMA-busy time)?

Oh. Do we even have to mess with that?

Probably not for the intial series.
Will drop this question.

Quoted 8 lines
Quoted 4 lines

5. Tuning migrate_pages callers for offloading. For instance, in compaction COMPACT_CLUSTER_MAX = 32 caps DMA's payoff for compaction (V4 experiment).

Is that HW dependent?

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

6. Where do batch-size thresholds live, and how are they tuned? Per Huang Ying's split, that policy lives in the migrator. DCBM has no threshold today. Open whether it should later be a per-migrator sysfs knob or hard-coded; probably clearer once a second migrator (SDXI, mtcopy) shows the trade-off.

Again, sounds like being HW dependent, no?

Yes, both are HW dependent.
Batch-size gating fits naturally in the migrator.
For something like COMPACT_CLUSTER_MAX, would a callback from compaction to registered migrator is right thought? or do you have something else in mind? For initial series, I think I need not mess with it.

Thanks,
Shivank