Skip to contentSkip to conversation

Batch migration for NUMA balancing

RFC PATCH In review Bharata B Rao 9 participants 42 messages May 21, 2025 – May 28, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Batch migration for NUMA balancing

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

As a first step, this is an attempt to batch and perform NUMAB migrations in async manner. Separate migration threads aren't yet implemented but I am using Gregory's patch[7] that provides migrate_misplaced_folio_batch() API to do batch migration of misplaced folios.

Some points for discussion
--------------------------
1. To isolate the misplaced folios or not?

To do batch migration, the misplaced folios need to be stored in some manner. I thought isolating them and using the folio->lru field to link them up would be the most straight-forward way. But then there were concerns expressed about folios remaining isolated for long until they get migrated.

Or should we just maintain the PFNs instead of folios and isolate them only just prior to migrating them?

2. Managing target_nid for misplaced pages

NUMAB provides the accurate target_nid for each folio that is detected as misplaced. However when we don't migrate the folio right away, but instead want to batch and do asyn migration later, then where do we keep track of target_nid for each folio?

In this implementation, I am using last_cpupid field as it appeared that this field could be reused (with some challenges mentioned in 2/2) for isolated folios. This approach may be specific to NUMAB but then each sub-system that hands over pages to the migrator thread should also provide a target_nid and hence each sub-system should be free to maintain and track the target_nid of folios that it has isolated/batched for migration in its own specific manner.

3. How many folios to batch?

Currently I have a fixed threshold for number of folios to batch. It could be a sysctl to allow a setting between a min and max. It could also be auto-tuned if required.

The state of the patchset
-------------------------
* Still raw and very lightly tested
* Just posted to serve as base for subsequent discussions here and in MM alignment calls.

References
----------
[1] LSFMM LWN summary - https://lwn.net/Articles/1016519/
[2] MM alignment call summary - https://lore.kernel.org/linux-mm/[email protected]/
[3] kpromoted patchset - https://lore.kernel.org/linux-mm/[email protected]/
[4] Kmmscand: PTE A bit scanning - https://lore.kernel.org/linux-mm/[email protected]/
[5] MGLRU scanning for page promotion - https://lore.kernel.org/lkml/[email protected]/
[6] IBS base hot page promotion - https://lore.kernel.org/linux-mm/[email protected]/
[7] Unmapped page cache folio promotion patchset - https://lore.kernel.org/linux-mm/[email protected]/

Bharata B Rao (1):
mm: sched: Batch-migrate misplaced pages

Gregory Price (1):
migrate: implement migrate_misplaced_folio_batch

 include/linux/migrate.h |  6 ++++
 include/linux/sched.h   |  4 +++
 init/init_task.c        |  2 ++
 kernel/sched/fair.c     | 64 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             | 44 ++++++++++++++--------------
 mm/migrate.c            | 31 ++++++++++++++++++++
 6 files changed, 130 insertions(+), 21 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/2 migrate: implement migrate_misplaced_folio_batch

From: Gregory Price <[email protected]>

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index aaa2114498d6..c9496adcf192 100644
@@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struct page *page)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folio_batch(struct list_head *foliolist, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -155,6 +156,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folio_batch(struct list_head *foliolist,
+						int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +31 -0
index 676d9cfc7059..32cc2eafb037 100644
@@ -2733,5 +2733,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/*
+ * Batch variant of migrate_misplaced_folio. Attempts to migrate
+ * a folio list to the specified destination.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, dereference them, and
+ * remove them from the list before returning.
+ */
+int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	unsigned int nr_succeeded;
+	int nr_remaining;
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	BUG_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
#2 lore raw

Re: [RFC PATCH v0 1/2] migrate: implement migrate_misplaced_folio_batch

Quoted 49 lines · Bharata B Rao

From: Gregory Price <[email protected]>

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

---
  include/linux/migrate.h |  6 ++++++
  mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
  2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index aaa2114498d6..c9496adcf192 100644
@@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struct page *page)
  int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folio_batch(struct list_head *foliolist, int node);
  #else
  static inline int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node)
@@ -155,6 +156,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
  {
  	return -EAGAIN; /* can't migrate now */
  }
+static inline int migrate_misplaced_folio_batch(struct list_head *foliolist,
+						int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  #endif /* CONFIG_NUMA_BALANCING */
 
  #ifdef CONFIG_MIGRATION
mm/migrate.c +3 -0
index 676d9cfc7059..32cc2eafb037 100644
@@ -2733,5 +2733,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
  	BUG_ON(!list_empty(&migratepages));
  	return nr_remaining ? -EAGAIN : 0;
  }
+
+/*
+ * Batch variant of migrate_misplaced_folio. Attempts to migrate

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2748

Quoted 10 lines
mm/migrate.c +10 -0
+ * a folio list to the specified destination.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, dereference them, and
+ * remove them from the list before returning.
+ */
+int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)

"migrate_misplaced_folios" ?

:)

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#3 lore raw

Re: [RFC PATCH v0 1/2] migrate: implement migrate_misplaced_folio_batch

Quoted 5 lines · David Hildenbrand
Quoted 1 line

+int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)

"migrate_misplaced_folios" ?

:)

something something brevity is the soul of wit

I think i went with _batch to match surrounding code (been a while since I wrote this), but I don't have strong feelings either way.

~Gregory

#4 lore raw

Re: [RFC PATCH v0 1/2] migrate: implement migrate_misplaced_folio_batch

Quoted 11 lines · Gregory Price
Quoted 5 lines · David Hildenbrand
Quoted 1 line

+int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)

"migrate_misplaced_folios" ?

:)

something something brevity is the soul of wit

I think i went with _batch to match surrounding code (been a while since I wrote this), but I don't have strong feelings either way.

I think we have migrate_pages_batch() and migrate_pages_sync() because ... they are called from migrate_pages() :)

For something that "simply" calls migrate_pages() right now, probably we should just call migrate_folios().

But maybe you were referring to yet another set of "_batch" functions.

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#5 lore raw

Re: [RFC PATCH v0 1/2] migrate: implement migrate_misplaced_folio_batch

In reply to Bharata B Rao
Quoted 69 lines · Bharata B Rao

From: Gregory Price <[email protected]>

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index aaa2114498d6..c9496adcf192 100644
@@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struct page *page)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folio_batch(struct list_head *foliolist, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -155,6 +156,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folio_batch(struct list_head *foliolist,
+						int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +23 -0
index 676d9cfc7059..32cc2eafb037 100644
@@ -2733,5 +2733,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/*
+ * Batch variant of migrate_misplaced_folio. Attempts to migrate
+ * a folio list to the specified destination.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, dereference them, and
+ * remove them from the list before returning.
+ */
+int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	unsigned int nr_succeeded;
+	int nr_remaining;
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2768

Quoted 10 lines
mm/migrate.c +8 -0
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	BUG_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */

migrate_misplaced_folio_batch() looks quite similar as migrate_misplaced_folio(), can we merge them?

---
Best Regards,
Huang, Ying

#6 lore raw

RFC PATCH 2/2 mm: sched: Batch-migrate misplaced pages

In reply to Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

The PTEs for the isolated folios are restored to PRESENT state right after isolation.

The last_cpupid field of isolated folios is used to store the target_nid to which the folios need to be migrated to. This needs changes to (at least) a couple of places where last_cpupid field is updated/reset which now should happen conditionally. The updation in folio_migrate_flags() isn't handled yet but the reset in write page fault case is handled.

The failed migration count isn't fed back to the scan period update heuristics currently.

---
 include/linux/sched.h |  4 +++
 init/init_task.c      |  2 ++
 kernel/sched/fair.c   | 64 +++++++++++++++++++++++++++++++++++++++++++
 mm/memory.c           | 44 +++++++++++++++--------------
 4 files changed, 93 insertions(+), 21 deletions(-)
include/linux/sched.h +4 -0
index f96ac1982893..4177ecf53633 100644
@@ -1360,6 +1360,8 @@ struct task_struct {
 	u64				last_task_numa_placement;
 	u64				last_sum_exec_runtime;
 	struct callback_head		numa_work;
+	struct callback_head		numa_mig_work;
+	unsigned long			numa_mig_interval;
 
 	/*
 	 * This pointer is only modified for current in syscall and
@@ -1397,6 +1399,8 @@ struct task_struct {
 	unsigned long			numa_faults_locality[3];
 
 	unsigned long			numa_pages_migrated;
+	struct list_head		migrate_list;
+	unsigned long			migrate_count;
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_RSEQ
init/init_task.c +2 -0
index e557f622bd90..997af6ab67a7 100644
@@ -187,6 +187,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 	.numa_preferred_nid = NUMA_NO_NODE,
 	.numa_group	= NULL,
 	.numa_faults	= NULL,
+	.migrate_count	= 0,
+	.migrate_list	= LIST_HEAD_INIT(init_task.migrate_list),
 #endif
 #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
 	.kasan_depth	= 1,
kernel/sched/fair.c +64 -0

108 lines not on this page · raw message

mm/memory.c +23 -21

77 lines not on this page · raw message

Signature · 1 line
2.34.1
#7 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 98 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

The PTEs for the isolated folios are restored to PRESENT state right after isolation.

The last_cpupid field of isolated folios is used to store the target_nid to which the folios need to be migrated to. This needs changes to (at least) a couple of places where last_cpupid field is updated/reset which now should happen conditionally. The updation in folio_migrate_flags() isn't handled yet but the reset in write page fault case is handled.

The failed migration count isn't fed back to the scan period update heuristics currently.

---
  include/linux/sched.h |  4 +++
  init/init_task.c      |  2 ++
  kernel/sched/fair.c   | 64 +++++++++++++++++++++++++++++++++++++++++++
  mm/memory.c           | 44 +++++++++++++++--------------
  4 files changed, 93 insertions(+), 21 deletions(-)
include/linux/sched.h +4 -0
index f96ac1982893..4177ecf53633 100644
@@ -1360,6 +1360,8 @@ struct task_struct {
  	u64				last_task_numa_placement;
  	u64				last_sum_exec_runtime;
  	struct callback_head		numa_work;
+	struct callback_head		numa_mig_work;
+	unsigned long			numa_mig_interval;
 
  	/*
  	 * This pointer is only modified for current in syscall and
@@ -1397,6 +1399,8 @@ struct task_struct {
  	unsigned long			numa_faults_locality[3];
 
  	unsigned long			numa_pages_migrated;
+	struct list_head		migrate_list;
+	unsigned long			migrate_count;
  #endif /* CONFIG_NUMA_BALANCING */
 
  #ifdef CONFIG_RSEQ
init/init_task.c +2 -0
index e557f622bd90..997af6ab67a7 100644
@@ -187,6 +187,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
  	.numa_preferred_nid = NUMA_NO_NODE,
  	.numa_group	= NULL,
  	.numa_faults	= NULL,
+	.migrate_count	= 0,
+	.migrate_list	= LIST_HEAD_INIT(init_task.migrate_list),
  #endif
  #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
  	.kasan_depth	= 1,
kernel/sched/fair.c +5 -0
index 0fb9bf995a47..d6cbf8be76e1 100644
@@ -49,6 +49,7 @@
  #include <linux/ratelimit.h>
  #include <linux/task_work.h>
  #include <linux/rbtree_augmented.h>
+#include <linux/migrate.h>
 
  #include <asm/switch_to.h>
 
@@ -1463,6 +1464,8 @@ unsigned int sysctl_numa_balancing_scan_delay = 1000;
  /* The page with hint page fault latency < threshold in ms is considered hot */
  unsigned int sysctl_numa_balancing_hot_threshold = MSEC_PER_SEC;
 
+#define NUMAB_BATCH_MIGRATION_THRESHOLD	512
+
  struct numa_group {
  	refcount_t refcount;
 
@@ -3297,6 +3300,46 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
 
  #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay)
 
+/*
+ * TODO: Feed failed migration count back to scan period update

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3314

Quoted 10 lines
kernel/sched/fair.c +10 -0
+ * mechanism.
+ */
+static void migrate_queued_pages(struct list_head *migrate_list)
+{
+	int cur_nid, nid;
+	struct folio *folio, *tmp;
+	LIST_HEAD(nid_list);
+
+	folio = list_entry(migrate_list, struct folio, lru);
+	cur_nid = folio_last_cpupid(folio);

Hi Bharatha,

This is target node ID right?

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3317

Quoted 3 lines
kernel/sched/fair.c +3 -0
+
+	list_for_each_entry_safe(folio, tmp, migrate_list, lru) {
+		nid = folio_xchg_last_cpupid(folio, -1);

Just one doubt: to get the last CPU ID (target node ID) here, folio_xchg_last_cpupid()

is used, whereas earlier folio_last_cpupid() was used. Is there a specific reason for

using different functions?

Thanks
Donet

Quoted 163 lines
kernel/sched/fair.c +46 -0
+
+		if (cur_nid != nid) {
+			migrate_misplaced_folio_batch(&nid_list, cur_nid);
+			cur_nid = nid;
+		}
+		list_move(&folio->lru, &nid_list);
+	}
+	migrate_misplaced_folio_batch(&nid_list, cur_nid);
+}
+
+static void task_migration_work(struct callback_head *work)
+{
+	struct task_struct *p = current;
+
+	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_mig_work));
+
+	work->next = work;
+
+	if (list_empty(&p->migrate_list))
+		return;
+
+	migrate_queued_pages(&p->migrate_list);
+	p->migrate_count = 0;
+}
+
  /*
   * The expensive part of numa migration is done from task_work context.
   * Triggered from task_tick_numa().
@@ -3567,14 +3610,19 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p)
  	p->numa_migrate_retry		= 0;
  	/* Protect against double add, see task_tick_numa and task_numa_work */
  	p->numa_work.next		= &p->numa_work;
+	p->numa_mig_work.next		= &p->numa_mig_work;
+	p->numa_mig_interval			= 0;
  	p->numa_faults			= NULL;
  	p->numa_pages_migrated		= 0;
  	p->total_numa_faults		= 0;
  	RCU_INIT_POINTER(p->numa_group, NULL);
  	p->last_task_numa_placement	= 0;
  	p->last_sum_exec_runtime	= 0;
+	p->migrate_count		= 0;
+	INIT_LIST_HEAD(&p->migrate_list);
 
  	init_task_work(&p->numa_work, task_numa_work);
+	init_task_work(&p->numa_mig_work, task_migration_work);
 
  	/* New address space, reset the preferred nid */
  	if (!(clone_flags & CLONE_VM)) {
@@ -3596,6 +3644,20 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p)
  	}
  }
 
+static void task_check_pending_migrations(struct task_struct *curr)
+{
+	struct callback_head *work = &curr->numa_mig_work;
+
+	if (work->next != work)
+		return;
+
+	if (time_after(jiffies, curr->numa_mig_interval) ||
+	    (curr->migrate_count > NUMAB_BATCH_MIGRATION_THRESHOLD)) {
+		curr->numa_mig_interval = jiffies + HZ;
+		task_work_add(curr, work, TWA_RESUME);
+	}
+}
+
  /*
   * Drive the periodic memory faults..
   */
@@ -3610,6 +3672,8 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
  	if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work)
  		return;
 
+	task_check_pending_migrations(curr);
+
  	/*
  	 * Using runtime rather than walltime has the dual advantage that
  	 * we (mostly) drive the selection from busy threads and that the
mm/memory.c +23 -21
index 49199410805c..11d07004cb04 100644
@@ -3375,8 +3375,13 @@ static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio)
  		 * Clear the folio's cpupid information as the existing
  		 * information potentially belongs to a now completely
  		 * unrelated process.
+		 *
+		 * If the page is found to be isolated pending migration,
+		 * then don't reset as last_cpupid will be holding the
+		 * target_nid information.
  		 */
-		folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
+		if (folio_test_lru(folio))
+			folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
  	}
 
  	flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
@@ -5766,12 +5771,13 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru
 
  static vm_fault_t do_numa_page(struct vm_fault *vmf)
  {
+	struct task_struct *task = current;
  	struct vm_area_struct *vma = vmf->vma;
  	struct folio *folio = NULL;
  	int nid = NUMA_NO_NODE;
  	bool writable = false, ignore_writable = false;
  	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
-	int last_cpupid;
+	int last_cpupid = (-1 & LAST_CPUPID_MASK);
  	int target_nid;
  	pte_t pte, old_pte;
  	int flags = 0, nr_pages;
@@ -5807,6 +5813,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	nid = folio_nid(folio);
  	nr_pages = folio_nr_pages(folio);
 
+	/*
+	 * If it is a non-LRU folio, it has been already
+	 * isolated and is in migration list.
+	 */
+	if (!folio_test_lru(folio))
+		goto out_map;
+
  	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
  					writable, &last_cpupid);
  	if (target_nid == NUMA_NO_NODE)
@@ -5815,28 +5828,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-	/* The folio is isolated and isolation code holds a folio reference. */
-	pte_unmap_unlock(vmf->pte, vmf->ptl);
  	writable = false;
  	ignore_writable = true;
+	nid = target_nid;
 
-	/* Migrate to the requested node */
-	if (!migrate_misplaced_folio(folio, target_nid)) {
-		nid = target_nid;
-		flags |= TNF_MIGRATED;
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
-		return 0;
-	}
-
-	flags |= TNF_MIGRATE_FAIL;
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
-				       vmf->address, &vmf->ptl);
-	if (unlikely(!vmf->pte))
-		return 0;
-	if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
-		pte_unmap_unlock(vmf->pte, vmf->ptl);
-		return 0;
-	}
+	/*
+	 * Store target_nid in last_cpupid field for the isolated
+	 * folios.
+	 */
+	folio_xchg_last_cpupid(folio, target_nid);
+	list_add_tail(&folio->lru, &task->migrate_list);
+	task->migrate_count += nr_pages;
  out_map:
  	/*
  	 * Make it present again, depending on how arch implements
reviewer
#8 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 103 lines · Donet Tom
Quoted 89 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

The PTEs for the isolated folios are restored to PRESENT state right after isolation.

The last_cpupid field of isolated folios is used to store the target_nid to which the folios need to be migrated to. This needs changes to (at least) a couple of places where last_cpupid field is updated/reset which now should happen conditionally. The updation in folio_migrate_flags() isn't handled yet but the reset in write page fault case is handled.

The failed migration count isn't fed back to the scan period update heuristics currently.

---
  include/linux/sched.h |  4 +++
  init/init_task.c      |  2 ++
  kernel/sched/fair.c   | 64 +++++++++++++++++++++++++++++++++++++++++++
  mm/memory.c           | 44 +++++++++++++++--------------
  4 files changed, 93 insertions(+), 21 deletions(-)
include/linux/sched.h +4 -0
index f96ac1982893..4177ecf53633 100644
@@ -1360,6 +1360,8 @@ struct task_struct {
  	u64				last_task_numa_placement;
  	u64				last_sum_exec_runtime;
  	struct callback_head		numa_work;
+	struct callback_head		numa_mig_work;
+	unsigned long			numa_mig_interval;
   	/*
  	 * This pointer is only modified for current in syscall and
@@ -1397,6 +1399,8 @@ struct task_struct {
  	unsigned long			numa_faults_locality[3];
   	unsigned long			numa_pages_migrated;
+	struct list_head		migrate_list;
+	unsigned long			migrate_count;
  #endif /* CONFIG_NUMA_BALANCING */
   #ifdef CONFIG_RSEQ
init/init_task.c +2 -0
index e557f622bd90..997af6ab67a7 100644
@@ -187,6 +187,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
  	.numa_preferred_nid = NUMA_NO_NODE,
  	.numa_group	= NULL,
  	.numa_faults	= NULL,
+	.migrate_count	= 0,
+	.migrate_list	= LIST_HEAD_INIT(init_task.migrate_list),
  #endif
  #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
  	.kasan_depth	= 1,
kernel/sched/fair.c +1 -0
index 0fb9bf995a47..d6cbf8be76e1 100644
@@ -49,6 +49,7 @@
  #include <linux/ratelimit.h>
  #include <linux/task_work.h>
  #include <linux/rbtree_augmented.h>
+#include <linux/migrate.h>
   #include <asm/switch_to.h>
 @@ -1463,6 +1464,8 @@ unsigned int sysctl_numa_balancing_scan_delay = 1000;
  /* The page with hint page fault latency < threshold in ms is considered hot */

unsigned int sysctl_numa_balancing_hot_threshold = MSEC_PER_SEC;
+#define NUMAB_BATCH_MIGRATION_THRESHOLD 512

kernel/sched/fair.c +2 -0
+
  struct numa_group {
  	refcount_t refcount;
 @@ -3297,6 +3300,46 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
   #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay)
 +/*
+ * TODO: Feed failed migration count back to scan period update

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3314

Quoted 10 lines
kernel/sched/fair.c +10 -0
+ * mechanism.
+ */
+static void migrate_queued_pages(struct list_head *migrate_list)
+{
+	int cur_nid, nid;
+	struct folio *folio, *tmp;
+	LIST_HEAD(nid_list);
+
+	folio = list_entry(migrate_list, struct folio, lru);
+	cur_nid = folio_last_cpupid(folio);

Hi Bharatha,

Quoted 1 line

This is target node ID right?

In memory tiering mode, folio_last_cpupid() gives page access time for slow memory folios. In !folio_use_access_time() case, folio_last_cpupid() gives last cpupid. Now it is reused for node id. It is too confusing. At least, a new function like folio_get_target_nid() should be added to return a nid only if folio is isolated.

Quoted 163 lines

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3317

Quoted 5 lines
kernel/sched/fair.c +3 -0
+
+	list_for_each_entry_safe(folio, tmp, migrate_list, lru) {
+		nid = folio_xchg_last_cpupid(folio, -1);

Just one doubt: to get the last CPU ID (target node ID) here, folio_xchg_last_cpupid()

is used, whereas earlier folio_last_cpupid() was used. Is there a specific reason for

using different functions?

Thanks
Donet

Quoted 147 lines
kernel/sched/fair.c +44 -0
+
+		if (cur_nid != nid) {
+			migrate_misplaced_folio_batch(&nid_list, cur_nid);
+			cur_nid = nid;
+		}
+		list_move(&folio->lru, &nid_list);
+	}
+	migrate_misplaced_folio_batch(&nid_list, cur_nid);
+}
+
+static void task_migration_work(struct callback_head *work)
+{
+	struct task_struct *p = current;
+
+	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_mig_work));
+
+	work->next = work;
+
+	if (list_empty(&p->migrate_list))
+		return;
+
+	migrate_queued_pages(&p->migrate_list);
+	p->migrate_count = 0;
+}
+
  /*
   * The expensive part of numa migration is done from task_work context.
   * Triggered from task_tick_numa().
@@ -3567,14 +3610,19 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p)
  	p->numa_migrate_retry		= 0;
  	/* Protect against double add, see task_tick_numa and task_numa_work */
  	p->numa_work.next		= &p->numa_work;
+	p->numa_mig_work.next		= &p->numa_mig_work;
+	p->numa_mig_interval			= 0;
  	p->numa_faults			= NULL;
  	p->numa_pages_migrated		= 0;
  	p->total_numa_faults		= 0;
  	RCU_INIT_POINTER(p->numa_group, NULL);
  	p->last_task_numa_placement	= 0;
  	p->last_sum_exec_runtime	= 0;
+	p->migrate_count		= 0;
+	INIT_LIST_HEAD(&p->migrate_list);
   	init_task_work(&p->numa_work, task_numa_work);
+	init_task_work(&p->numa_mig_work, task_migration_work);
   	/* New address space, reset the preferred nid */
  	if (!(clone_flags & CLONE_VM)) {
@@ -3596,6 +3644,20 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p)
  	}
  }
 +static void task_check_pending_migrations(struct task_struct *curr)
+{
+	struct callback_head *work = &curr->numa_mig_work;
+
+	if (work->next != work)
+		return;
+
+	if (time_after(jiffies, curr->numa_mig_interval) ||
+	    (curr->migrate_count > NUMAB_BATCH_MIGRATION_THRESHOLD)) {
+		curr->numa_mig_interval = jiffies + HZ;
+		task_work_add(curr, work, TWA_RESUME);
+	}
+}
+
  /*
   * Drive the periodic memory faults..
   */
@@ -3610,6 +3672,8 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
  	if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work)
  		return;
 +	task_check_pending_migrations(curr);
+
  	/*
  	 * Using runtime rather than walltime has the dual advantage that
  	 * we (mostly) drive the selection from busy threads and that the
mm/memory.c +17 -20
index 49199410805c..11d07004cb04 100644
@@ -3375,8 +3375,13 @@ static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio)
  		 * Clear the folio's cpupid information as the existing
  		 * information potentially belongs to a now completely
  		 * unrelated process.
+		 *
+		 * If the page is found to be isolated pending migration,
+		 * then don't reset as last_cpupid will be holding the
+		 * target_nid information.
  		 */
-		folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
+		if (folio_test_lru(folio))
+			folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
  	}
   	flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
@@ -5766,12 +5771,13 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru
   static vm_fault_t do_numa_page(struct vm_fault *vmf)
  {
+	struct task_struct *task = current;
  	struct vm_area_struct *vma = vmf->vma;
  	struct folio *folio = NULL;
  	int nid = NUMA_NO_NODE;
  	bool writable = false, ignore_writable = false;
  	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
-	int last_cpupid;
+	int last_cpupid = (-1 & LAST_CPUPID_MASK);
  	int target_nid;
  	pte_t pte, old_pte;
  	int flags = 0, nr_pages;
@@ -5807,6 +5813,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	nid = folio_nid(folio);
  	nr_pages = folio_nr_pages(folio);
 +	/*
+	 * If it is a non-LRU folio, it has been already
+	 * isolated and is in migration list.
+	 */
+	if (!folio_test_lru(folio))
+		goto out_map;
+
  	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
  					writable, &last_cpupid);
  	if (target_nid == NUMA_NO_NODE)
@@ -5815,28 +5828,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-	/* The folio is isolated and isolation code holds a folio reference. */
-	pte_unmap_unlock(vmf->pte, vmf->ptl);
  	writable = false;
  	ignore_writable = true;
+	nid = target_nid;
 -	/* Migrate to the requested node */
-	if (!migrate_misplaced_folio(folio, target_nid)) {
-		nid = target_nid;
-		flags |= TNF_MIGRATED;
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
-		return 0;
-	}
-
-	flags |= TNF_MIGRATE_FAIL;
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
-				       vmf->address, &vmf->ptl);
-	if (unlikely(!vmf->pte))
-		return 0;
-	if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
-		pte_unmap_unlock(vmf->pte, vmf->ptl);
-		return 0;
-	}
+	/*
+	 * Store target_nid in last_cpupid field for the isolated
Quoted 8 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5844

Quoted 8 lines
mm/memory.c +5 -0
+	 * folios.
+	 */
+	folio_xchg_last_cpupid(folio, target_nid);
+	list_add_tail(&folio->lru, &task->migrate_list);
+	task->migrate_count += nr_pages;
  out_map:
  	/*
  	 * Make it present again, depending on how arch implements

--
Best Regards,
Yan, Zi

reviewer
#9 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 11 lines · Zi Yan
Quoted 3 lines · Donet Tom

Hi Bharatha,

This is target node ID right?

In memory tiering mode, folio_last_cpupid() gives page access time for slow memory folios. In !folio_use_access_time() case, folio_last_cpupid() gives last cpupid. Now it is reused for node id. It is too confusing. At least, a new function like folio_get_target_nid() should be added to return a nid only if folio is isolated.

The really annoying part of all of this is

#ifdef CONFIG_NUMA_BALANCING
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
static inline int folio_last_cpupid(struct folio *folio)

{
        return folio->_last_cpupid;
}
#else
static inline int folio_last_cpupid(struct folio *folio)
{
        return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
}

#endif
#else /* !CONFIG_NUMA_BALANCING */
static inline int folio_last_cpupid(struct folio *folio) {
return folio_nid(folio); /* XXX */
}
...
#endif

Obviously we don't have to care about the !NUMAB case, but what a silly muxing we have going on here (I get it, space is tight - the interfaces are just confusing is all).

My question is whether there's some kind of race condition here if the mode changes between isolate and fetch. Can we "fetch a node id" and end up with a cpupid because someone toggled the between tiering and balancing?

If we can answer that, then implementing folio_last_cpupid and folio_last_access_nid can return -1 if called in the wrong mode (assuming this check isn't too expensive). That would be a nice cleanup for readability sake.

~Gregory

#10 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 23 lines · Gregory Price

The really annoying part of all of this is

#ifdef CONFIG_NUMA_BALANCING
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
static inline int folio_last_cpupid(struct folio *folio)

{
         return folio->_last_cpupid;
}
#else
static inline int folio_last_cpupid(struct folio *folio)
{
         return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
}

#endif
#else /* !CONFIG_NUMA_BALANCING */
static inline int folio_last_cpupid(struct folio *folio) {
return folio_nid(folio); /* XXX */
}
...
#endif

Quoted 3 lines

Obviously we don't have to care about the !NUMAB case, but what a silly muxing we have going on here (I get it, space is tight - the interfaces are just confusing is all).

I really didn't realize the usage in !NUMAB case, thanks.

Quoted 5 lines

My question is whether there's some kind of race condition here if the mode changes between isolate and fetch. Can we "fetch a node id" and end up with a cpupid because someone toggled the between tiering and balancing?

Good question. I need to check all such cases where inadvertent reset or reuse/repurposing of last_cpupid field of an isolated folio becomes possible.

Quoted 5 lines

If we can answer that, then implementing folio_last_cpupid and folio_last_access_nid can return -1 if called in the wrong mode (assuming this check isn't too expensive). That would be a nice cleanup for readability sake.

Right.

Regards,
Bharata.

#11 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Zi Yan
Quoted 7 lines · Zi Yan

On 21 May 2025, at 14:25, Donet Tom wrote:

In memory tiering mode, folio_last_cpupid() gives page access time for slow memory folios. In !folio_use_access_time() case, folio_last_cpupid() gives last cpupid. Now it is reused for node id. It is too confusing. At least, a new function like folio_get_target_nid() should be added to return a nid only if folio is isolated.

Yes, it can be confusing. If this approach of using last_cpupid field to store the target_nid is found to be feasible, I will cleanup the names as you suggest.

Regards,
Bharata.

#12 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Donet Tom

Hi Donet,

Quoted 12 lines · Donet Tom

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3314

Quoted 9 lines
kernel/sched/fair.c +8 -0
+static void migrate_queued_pages(struct list_head *migrate_list)
+{
+    int cur_nid, nid;
+    struct folio *folio, *tmp;
+    LIST_HEAD(nid_list);
+
+    folio = list_entry(migrate_list, struct folio, lru);
+    cur_nid = folio_last_cpupid(folio);

Hi Bharatha,

Quoted 1 line

This is target node ID right?

Correct.

Quoted 11 lines
Quoted 5 lines

+
+    list_for_each_entry_safe(folio, tmp, migrate_list, lru) { +        nid = folio_xchg_last_cpupid(folio, -1);

Just one doubt: to get the last CPU ID (target node ID) here, folio_xchg_last_cpupid()

is used, whereas earlier folio_last_cpupid() was used. Is there a specific reason for

using different functions?

This function iterates over the isolated folios looking for the same target_nid so that all of them can be migrated at once to the given target_nid. Hence the first call just reads the target_nid from last_cpupid field to note which nid is of interest in the current iteration and the next call actually reads target_nid and resets the last_cpupid field.

Regards,
Bharata.

#13 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 25 lines · Bharata B Rao

Hi Donet,

Quoted 12 lines · Donet Tom

kernel/sched/fair.c static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma) · line 3314

Quoted 9 lines
kernel/sched/fair.c +8 -0
+static void migrate_queued_pages(struct list_head *migrate_list)
+{
+    int cur_nid, nid;
+    struct folio *folio, *tmp;
+    LIST_HEAD(nid_list);
+
+    folio = list_entry(migrate_list, struct folio, lru);
+    cur_nid = folio_last_cpupid(folio);

Hi Bharatha,

Quoted 1 line

This is target node ID right?

Correct.

Quoted 6 lines
Quoted 5 lines

+
+    list_for_each_entry_safe(folio, tmp, migrate_list, lru) { +        nid = folio_xchg_last_cpupid(folio, -1);

Quoted 7 lines
Quoted 5 lines

Just one doubt: to get the last CPU ID (target node ID) here, folio_xchg_last_cpupid()

is used, whereas earlier folio_last_cpupid() was used. Is there a specific reason for

using different functions?

This function iterates over the isolated folios looking for the same target_nid so that all of them can be migrated at once to the given target_nid. Hence the first call just reads the target_nid from last_cpupid field to note which nid is of interest in the current iteration and the next call actually reads target_nid and resets the last_cpupid field.

Thank you, Bharata, for the clarification.

Quoted 3 lines

Regards,
Bharata.

#14 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Bharata B Rao
kernel test robot noticed the following build errors:

Hi Bharata,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on tip/sched/core peterz-queue/sched/core linus/master v6.15-rc7 next-20250521] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/migrate-implement-migrate_misplaced_folio_batch/20250521-160958
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250521080238.209678-3-bharata%40amd.com
patch subject: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages
config: arm-randconfig-002-20250522 (https://download.01.org/0day-ci/archive/20250522/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250522/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> mm/memory.c:5930:36: error: no member named 'migrate_list' in 'struct task_struct'
    5930 |         list_add_tail(&folio->lru, &task->migrate_list);
         |                                     ~~~~  ^
>> mm/memory.c:5931:8: error: no member named 'migrate_count' in 'struct task_struct'
    5931 |         task->migrate_count += nr_pages;
         |         ~~~~  ^

2 errors generated.

vim +5930 mm/memory.c

5861
5862 static vm_fault_t do_numa_page(struct vm_fault *vmf)
5863 {
5864 struct task_struct *task = current;
5865 struct vm_area_struct *vma = vmf->vma;
5866 struct folio *folio = NULL;
5867 int nid = NUMA_NO_NODE;
5868 bool writable = false, ignore_writable = false;
5869 bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
5870 int last_cpupid = (-1 & LAST_CPUPID_MASK);
5871 int target_nid;
5872 pte_t pte, old_pte;
5873 int flags = 0, nr_pages;
5874
5875 /*
5876 * The pte cannot be used safely until we verify, while holding the page
5877 * table lock, that its contents have not changed during fault handling.
5878 */
5879 spin_lock(vmf->ptl);
5880 /* Read the live PTE from the page tables: */
5881 old_pte = ptep_get(vmf->pte);
5882
5883 if (unlikely(!pte_same(old_pte, vmf->orig_pte))) {
5884 pte_unmap_unlock(vmf->pte, vmf->ptl);
5885 return 0;
5886 }
5887
5888 pte = pte_modify(old_pte, vma->vm_page_prot);
5889
5890 /*
5891 * Detect now whether the PTE could be writable; this information
5892 * is only valid while holding the PT lock.
5893 */
5894 writable = pte_write(pte);
5895 if (!writable && pte_write_upgrade &&
5896 can_change_pte_writable(vma, vmf->address, pte))
5897 writable = true;
5898
5899 folio = vm_normal_folio(vma, vmf->address, pte);
5900 if (!folio || folio_is_zone_device(folio))
5901 goto out_map;
5902
5903 nid = folio_nid(folio);
5904 nr_pages = folio_nr_pages(folio);
5905
5906 /*
5907 * If it is a non-LRU folio, it has been already
5908 * isolated and is in migration list.
5909 */
5910 if (!folio_test_lru(folio))
5911 goto out_map;
5912
5913 target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
5914 writable, &last_cpupid);
5915 if (target_nid == NUMA_NO_NODE)
5916 goto out_map;
5917 if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
5918 flags |= TNF_MIGRATE_FAIL;
5919 goto out_map;
5920 }
5921 writable = false;
5922 ignore_writable = true;
5923 nid = target_nid;
5924
5925 /*
5926 * Store target_nid in last_cpupid field for the isolated
5927 * folios.
5928 */
5929 folio_xchg_last_cpupid(folio, target_nid);

> 5930		list_add_tail(&folio->lru, &task->migrate_list);
> 5931		task->migrate_count += nr_pages;

5932 out_map:
5933 /*
5934 * Make it present again, depending on how arch implements
5935 * non-accessible ptes, some can allow access by kernel mode.
5936 */
5937 if (folio && folio_test_large(folio))
5938 numa_rebuild_large_mapping(vmf, vma, folio, pte, ignore_writable,
5939 pte_write_upgrade);
5940 else
5941 numa_rebuild_single_mapping(vmf, vma, vmf->address, vmf->pte,
5942 writable);
5943 pte_unmap_unlock(vmf->pte, vmf->ptl);
5944
5945 if (nid != NUMA_NO_NODE)
5946 task_numa_fault(last_cpupid, nid, nr_pages, flags);
5947 return 0;
5948 }
5949

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reviewer
#15 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Bharata B Rao
Quoted 19 lines · Bharata B Rao
kernel/sched/fair.c +14 -0
+static void task_check_pending_migrations(struct task_struct *curr)
+{
+	struct callback_head *work = &curr->numa_mig_work;
+
+	if (work->next != work)
+		return;
+
+	if (time_after(jiffies, curr->numa_mig_interval) ||
+	    (curr->migrate_count > NUMAB_BATCH_MIGRATION_THRESHOLD)) {
+		curr->numa_mig_interval = jiffies + HZ;
+		task_work_add(curr, work, TWA_RESUME);
+	}
+}
+
 /*
  * Drive the periodic memory faults..
  */

kernel/sched/fair.c static void task_tick_numa(struct rq *rq, struct task_struct *curr) · line 3676

Quoted 5 lines
kernel/sched/fair.c +2 -0
@@ -3610,6 +3672,8 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
 	if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work)
 		return;
 
+	task_check_pending_migrations(curr);
+

So I know this was discussed in the cover leter a bit and alluded to in the patch, but I want to add my 2cents from work on the unmapped page cache set.

In that set, I chose to always schedule the task work on the next return to user-space, rather than defer to a tick like the current numa-balance code. This was for two concerns:

1) I didn't want to leave a potentially large number of isolated folios on a list that may not be reaped for an unknown period of time.

I don't know the real limitations on the number of isolated folios, but given what we have here I think we can represent a mathematical worst case on the nubmer of stranded folios.

If (N=1,000,000, and M=511) then we could have ~1.8TB of pages stranded on these lists - never to be migrated because it never hits the threshhold. In practice this won't happen to that extreme, but in practice it absolutely will happen for some chunk of tasks.

So I chose to never leave kernel space with isolated folios on the task numa_mig_list.

This discussion changes if the numa_mig_list is not on the task_struct and instead some per-cpu list routinely reaped by a kthread (kpromoted or whatever).

2) I was not confident I could measure the performance implications of the migrations directly when it was deferred. When would I even know it happened? The actual goal is to *not* know it happened, right?

But now it might happen during a page fault, or any random syscall.

This concerned me - so i just didn't defer. That was largely out of lack of confidence in my own understanding of the task_work system.

So i think this, as presented, is a half-measure - and I don't think it's a good half-measure. I think we might need to go all the way to a set of per-cpu migration lists that a kernel work can pluck the head of on some interval. That would bound the number of isolated folios to the number of CPUs rather than the number of tasks.

~Gregory

#16 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 37 lines · Gregory Price
Quoted 19 lines · Bharata B Rao
kernel/sched/fair.c +14 -0
+static void task_check_pending_migrations(struct task_struct *curr)
+{
+	struct callback_head *work = &curr->numa_mig_work;
+
+	if (work->next != work)
+		return;
+
+	if (time_after(jiffies, curr->numa_mig_interval) ||
+	    (curr->migrate_count > NUMAB_BATCH_MIGRATION_THRESHOLD)) {
+		curr->numa_mig_interval = jiffies + HZ;
+		task_work_add(curr, work, TWA_RESUME);
+	}
+}
+
  /*
   * Drive the periodic memory faults..
   */

kernel/sched/fair.c static void task_tick_numa(struct rq *rq, struct task_struct *curr) · line 3676

Quoted 5 lines
kernel/sched/fair.c +2 -0
@@ -3610,6 +3672,8 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
  	if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work)
  		return;
 
+	task_check_pending_migrations(curr);
+

So I know this was discussed in the cover leter a bit and alluded to in the patch, but I want to add my 2cents from work on the unmapped page cache set.

In that set, I chose to always schedule the task work on the next return to user-space, rather than defer to a tick like the current numa-balance code. This was for two concerns:

1) I didn't want to leave a potentially large number of isolated folios on a list that may not be reaped for an unknown period of time.

Quoted 8 lines

I don't know the real limitations on the number of isolated folios, but given what we have here I think we can represent a mathematical worst case on the nubmer of stranded folios.

If (N=1,000,000, and M=511) then we could have ~1.8TB of pages stranded on these lists - never to be migrated because it never hits the threshhold. In practice this won't happen to that extreme, but in practice it absolutely will happen for some chunk of tasks.

In addition to the threshold, I have a time limit too and hence at the end of that period, the isolated folios do get migrated even if the threshold isn't hit.

The other thing I haven't taken care yet is to put back the isolated folios if the task exits with pending isolated folios.

Quoted 19 lines

So I chose to never leave kernel space with isolated folios on the task numa_mig_list.

This discussion changes if the numa_mig_list is not on the task_struct and instead some per-cpu list routinely reaped by a kthread (kpromoted or whatever).

2) I was not confident I could measure the performance implications of the migrations directly when it was deferred. When would I even know it happened? The actual goal is to *not* know it happened, right?

But now it might happen during a page fault, or any random syscall.

This concerned me - so i just didn't defer. That was largely out of lack of confidence in my own understanding of the task_work system.

Quoted 5 lines

So i think this, as presented, is a half-measure - and I don't think it's a good half-measure. I think we might need to go all the way to a set of per-cpu migration lists that a kernel work can pluck the head of on some interval. That would bound the number of isolated folios to the number of CPUs rather than the number of tasks.

Why per-cpu and not per-node? All folios that are targeted for a node can be in that node's list.

I think if we are leaving the migration to be done by the migrator thread later, then isolating them beforehand may not be ideal. In such cases tracking the hot pages via PFNs like I did in kpromoted may be better.

Even when we have per-node migrator threads that would handle migration requests from multiple hot page sources (or single unified layer), I still think that there should be a "migrate now" kind of interface (which is essentially what your migrate_misplaced_folio_batch() is). That will be more suitable for handling migration request originating from locality-based NUMA balancing (NUMAB=1 case).

Regards,
Bharata.

reviewer
#17 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 11 lines · Bharata B Rao
Quoted 6 lines · Gregory Price

So i think this, as presented, is a half-measure - and I don't think it's a good half-measure. I think we might need to go all the way to a set of per-cpu migration lists that a kernel work can pluck the head of on some interval. That would bound the number of isolated folios to the number of CPUs rather than the number of tasks.

Why per-cpu and not per-node? All folios that are targeted for a node can be in that node's list.

On systems with significant number of threads (512-1024), these lists may be highly contended. I suppose we can start with per-node, but I would not be surprised if this went straight to per-cpu.

Quoted 4 lines

I think if we are leaving the migration to be done by the migrator thread later, then isolating them beforehand may not be ideal. In such cases tracking the hot pages via PFNs like I did in kpromoted may be better.

This seems like not a bad idea, you could do hot-swapped buffers to prevent infinite growth / contention. One of the problems with PFNs is that the state of that page can change between candidacy and promotion. I suppose the devil is the details there.

~Gregory

#18 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Bharata B Rao
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#19 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 15 lines · David Hildenbrand
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

--
Best Regards,
Yan, Zi

#20 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#21 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 26 lines · David Hildenbrand
Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

Quoted 5 lines

How do you envision that working when CMA needs to migrate this exact page to a different location?

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

--
Best Regards,
Yan, Zi

#22 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 30 lines · Zi Yan
Quoted 28 lines · David Hildenbrand
Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

b) The one responsible for the isolation is numa code (could be someone else)

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#23 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 45 lines · David Hildenbrand
Quoted 30 lines · Zi Yan
Quoted 28 lines · David Hildenbrand
Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

Quoted 6 lines

b) The one responsible for the isolation is numa code (could be someone else)

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page. In addition, periodically, the migration task would do a PFN scanning and migrate any migration candidate. I remember Willy did some experiments showing that PFN scanning is very fast.

--
Best Regards,
Yan, Zi

reviewer
#24 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 56 lines · Zi Yan
Quoted 47 lines · David Hildenbrand
Quoted 32 lines · Zi Yan
Quoted 30 lines · David Hildenbrand
Quoted 21 lines · Zi Yan
Quoted 19 lines · David Hildenbrand
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Quoted 5 lines
Quoted 1 line

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

Quoted 6 lines

b) The one responsible for the isolation is numa code (could be someone else)

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Quoted 7 lines

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page. In addition, periodically, the migration task would do a PFN scanning and migrate any migration candidate. I remember Willy did some experiments showing that PFN scanning is very fast.

I think that this could be a second step optimization after the simple implementation has been done.

---
Best Regards,
Huang, Ying

#25 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Zi Yan
Quoted 50 lines · Zi Yan
Quoted 48 lines · David Hildenbrand
Quoted 30 lines · Zi Yan
Quoted 28 lines · David Hildenbrand
Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

b) The one responsible for the isolation is numa code (could be someone else)

Quoted 8 lines
Quoted 3 lines

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page.

I mean, all these will migrate the page either way, no need to add another flag for that.

I guess what you mean, indicating that the migration destination should be on a different node than the current one.

Well, and for the NUMA scanner (below) to find which pages to migrate.

... to be this raises some questions: like, if we don't migrate immediately, could that information ("migrate this page") actually now be wrong? I guess a way to obtain the destination node would suffice: if the destination node matches, no need to migrate from that NUMA scanner.

In addition,

Quoted 3 lines

periodically, the migration task would do a PFN scanning and migrate any migration candidate. I remember Willy did some experiments showing that PFN scanning is very fast.

PFN scanning can be faster than walking lists, but I suspect it depends on how many pages there really are to be migrated ... and some other factors :)

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#26 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 55 lines · David Hildenbrand
Quoted 54 lines · Zi Yan
Quoted 51 lines · David Hildenbrand
Quoted 30 lines · Zi Yan
Quoted 28 lines · David Hildenbrand
Quoted 17 lines · Zi Yan
Quoted 15 lines · David Hildenbrand
Quoted 13 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 7 lines
Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

b) The one responsible for the isolation is numa code (could be someone else)

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Quoted 8 lines
Quoted 4 lines

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page.

I mean, all these will migrate the page either way, no need to add another flag for that.

I guess what you mean, indicating that the migration destination should be on a different node than the current one.

Yes.

Quoted 4 lines

Well, and for the NUMA scanner (below) to find which pages to migrate.

... to be this raises some questions: like, if we don't migrate immediately, could that information ("migrate this page") actually now be wrong? I guess a way to

Could be. So it is better to evaluate the page before the actual migration, in case the page is no longer needed in a remote node.

Quoted 1 line

obtain the destination node would suffice: if the destination node matches, no need to migrate from that NUMA scanner.

Right. The destination node could be calculated by certain metric like most recent accesses or last remote node access time. If most recent accesses are still coming from a remote node and/or last remote node access time is within a short time frame, the page should be migrated. Since it is possible that the page is frequently accessed by a remote node but when it comes to migration, it is no longer needed by a remote node and the access pattern would look like 1) a lot of remote node accesses, but
2) the last remote node access is long time ago.

Quoted 7 lines

In addition,

Quoted 3 lines

periodically, the migration task would do a PFN scanning and migrate any migration candidate. I remember Willy did some experiments showing that PFN scanning is very fast.

PFN scanning can be faster than walking lists, but I suspect it depends on how many pages there really are to be migrated ... and some other factors :)

Yes. LRU list is good since it restricts the scanning range, but PFN scanning itself does not have it. PFN scanning with some filter mechanism might work and that filter mechanism is a way of marking to-be-migrated pages. Of course, a quick re-evaluation of the to-be-migrated pages right before a migration would avoid unnecessary work like we discussed above.

--
Best Regards,
Yan, Zi

reviewer
#27 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 78 lines · Zi Yan
Quoted 62 lines · David Hildenbrand
Quoted 52 lines · Zi Yan
Quoted 50 lines · David Hildenbrand
Quoted 32 lines · Zi Yan
Quoted 30 lines · David Hildenbrand
Quoted 21 lines · Zi Yan
Quoted 19 lines · David Hildenbrand
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Quoted 5 lines
Quoted 1 line

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

b) The one responsible for the isolation is numa code (could be someone else)

Quoted 8 lines
Quoted 3 lines

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page.

Quoted 4 lines

I mean, all these will migrate the page either way, no need to add another flag for that.

I guess what you mean, indicating that the migration destination should be on a different node than the current one.

Yes.

Quoted 6 lines

Well, and for the NUMA scanner (below) to find which pages to migrate.

... to be this raises some questions: like, if we don't migrate immediately, could that information ("migrate this page") actually now be wrong? I guess a way to

Quoted 8 lines

Could be. So it is better to evaluate the page before the actual migration, in case the page is no longer needed in a remote node.

Quoted 2 lines

obtain the destination node would suffice: if the destination node matches, no need to migrate from that NUMA scanner.

Right. The destination node could be calculated by certain metric like most recent accesses or last remote node access time.

Do we have the necessary information available? last_cpupid have either last accessing CPU or last scanning timestamp, not both. Any other information source?

---
Best Regards,
Huang, Ying

Quoted 26 lines

If most recent accesses are still coming
from a remote node and/or last remote node access time is within a short time frame, the page should be migrated. Since it is possible that the page is frequently accessed by a remote node but when it comes to migration, it is no longer needed by a remote node and the access pattern would look like 1) a lot of remote node accesses, but
2) the last remote node access is long time ago.

Quoted 9 lines

In addition,

Quoted 3 lines

periodically, the migration task would do a PFN scanning and migrate any migration candidate. I remember Willy did some experiments showing that PFN scanning is very fast.

PFN scanning can be faster than walking lists, but I suspect it depends on how many pages there really are to be migrated ... and some other factors :)

Yes. LRU list is good since it restricts the scanning range, but PFN scanning itself does not have it. PFN scanning with some filter mechanism might work and that filter mechanism is a way of marking to-be-migrated pages. Of course, a quick re-evaluation of the to-be-migrated pages right before a migration would avoid unnecessary work like we discussed above.

--
Best Regards,
Yan, Zi

reviewer
#28 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

Quoted 86 lines · Ying
Quoted 84 lines · Zi Yan
Quoted 62 lines · David Hildenbrand
Quoted 52 lines · Zi Yan
Quoted 50 lines · David Hildenbrand
Quoted 32 lines · Zi Yan
Quoted 30 lines · David Hildenbrand
Quoted 21 lines · Zi Yan
Quoted 19 lines · David Hildenbrand
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Quoted 5 lines
Quoted 1 line

Bad.

Probably we can mark these pages and when others want to migrate the page, get_new_page() just looks at the page's target node and get a new page from the target node.

How do you envision that working when CMA needs to migrate this exact page to a different location?

Quoted 8 lines
Quoted 3 lines

It cannot isolate it for migration because ... it's already isolated ... so it will give up.

Marking might not be easy I assume ...

I guess you mean we do not have any extra bit to indicate this page is isolated, but it can be migrated. My point is that if this page is going to be migrated due to other reasons, like CMA, compaction, why not migrate it to the target node instead of moving it around within the same node.

I think we'd have to identify that

a) This page is isolate for migration (could be isolated for other reasons)

b) The one responsible for the isolation is numa code (could be someone else)

Quoted 8 lines
Quoted 3 lines

c) We're allowed to grab that page from that list (IOW sync against others, and especially also against), to essentially "steal" the isolated page.

Right. c) sounds like adding more contention to the candidate list. I wonder if we can just mark the page as migration candidate (using a page flag or something else), then migrate it whenever CMA, compaction, long-term pinning and more look at the page.

Quoted 4 lines

I mean, all these will migrate the page either way, no need to add another flag for that.

I guess what you mean, indicating that the migration destination should be on a different node than the current one.

Yes.

Quoted 6 lines

Well, and for the NUMA scanner (below) to find which pages to migrate.

... to be this raises some questions: like, if we don't migrate immediately, could that information ("migrate this page") actually now be wrong? I guess a way to

Could be. So it is better to evaluate the page before the actual migration, in case the page is no longer needed in a remote node.

Quoted 2 lines

obtain the destination node would suffice: if the destination node matches, no need to migrate from that NUMA scanner.

Quoted 6 lines
Quoted 2 lines

Right. The destination node could be calculated by certain metric like most recent accesses or last remote node access time.

Do we have the necessary information available? last_cpupid have either last accessing CPU or last scanning timestamp, not both. Any other information source?

Not at the moment. A unified page access information framework is probably needed. The recent LSFMM has a related discussion[1]. We also have a biweekly discussion on it[2].

[1] https://lwn.net/Articles/1016722/
[2] https://lore.kernel.org/linux-mm/[email protected]/

Best Regards,
Yan, Zi

#29 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to Zi Yan
Quoted 8 lines · Zi Yan
Quoted 1 line · David Hildenbrand

PFN scanning can be faster than walking lists, but I suspect it depends on how many pages there really are to be migrated ... and some other factors :)

Yes. LRU list is good since it restricts the scanning range, but PFN scanning itself does not have it. PFN scanning with some filter mechanism might work and that filter mechanism is a way of marking to-be-migrated pages. Of course, a quick re-evaluation of the to-be-migrated pages right before a migration would avoid unnecessary work like we discussed above.

PFN scanning could be faster because of prefetching, but it pollutes the caches, which may not be nice to the application running on that cpu core before we transitioned to kernel space.

Signature · 1 line
~karim
#30 lore raw

Re: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages

In reply to David Hildenbrand
Quoted 15 lines · David Hildenbrand
Quoted 5 lines · Bharata B Rao

Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.

Quoted 8 lines

Instead of such singe folio migrations, batch them and migrate them at once.

Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold.

Quoted 3 lines

That means that these pages are effectively unmovable for other purposes (CMA, compaction, long-term pinning, whatever) until that list was drained.

Bad.

During the last week's MM alignment call on this subject, it was decided not to isolate and batch from fault context and migrate from task_work context like this.

However since the amount of time the folios stay in isolated state was
bounded both by time(1s in my patchset) and the number of folios isolated, I thought it should be okay, but I may be wrong.

Regards,
Bharata.

#31 lore raw
In reply to Bharata B Rao

Hi Bharata,

Quoted 8 lines · On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 11 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

Also in LSFMM, I proposed damos_add_folio() for a case that callers want to utilize DAMON worker thread (kdamond) as an asynchronous memory management operations execution thread while using its other features such as [auto-tuned] quotas. I think this has a potential to serve something similar to your migration threads. I haven't started damos_add_folio() development yet, though.

I remember we discussed about DAMON on mailing list and in LSFMM a bit, on your session. IIRC, you were also looking for a time to see if there is a chance to use DAMON in some way. Due to the technical issue, we were unable to discuss on the two new proposals on my LSFMM session, and it has been a bit while since our last discussion. So if you don't mind, I'd like to ask if you have some opinions or comments about these.

[1] https://lwn.net/Articles/1016525/

Thanks,
SJ

[...]

reviewer
#32 lore raw
Quoted 28 lines · SeongJae Park

Hi Bharata,

Quoted 8 lines · On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 11 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

Quoted 6 lines

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

It seems to me that DAMON might make use of the batch migration interface, so if you need any changes or extensions, it might be good for you (SJ) to take a look at that for us.

~Gregory

#33 lore raw
Quoted 36 lines · On Wed, 21 May 2025 23:08:16 -0400 Gregory Price <[email protected]> wrote:
Quoted 28 lines · SeongJae Park

Hi Bharata,

Quoted 8 lines · On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 11 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

Quoted 6 lines

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

Quoted 3 lines

It seems to me that DAMON might make use of the batch migration interface, so if you need any changes or extensions, it might be good for you (SJ) to take a look at that for us.

I started this subthread not for batch migration but the long term goal. I took only a glance on the migration batching part, and I'm still trying to find time to take more deep look on batch migration.

Nonetheless, yes, basically I believe DAMON and Bharata's works have great opportunities to collaborate and use each other in a very productive ways. I'm especially very intersted in kpromoted's AMD IBS code, and trying to make DAMON easier to be used for Bharata's works.

For batch migration interface, though, to be honest I don't find very clear DAMON's usage of it, since DAMON does region-based sort of batched migration. Again, I took only a glance on migration batching part and gonna take more time to the details.

Meanwhile, if you saw some opportunities, and if you don't mind, it would be very helpful if you can share your detailed view of the opportunity (how DAMON could be better by using the migration batching in what way?).

Thanks,
SJ

Quoted 2 lines

~Gregory

reviewer
#34 lore raw
Quoted 12 lines · SeongJae Park
Quoted 4 lines · On Wed, 21 May 2025 23:08:16 -0400 Gregory Price <[email protected]> wrote:

It seems to me that DAMON might make use of the batch migration interface, so if you need any changes or extensions, it might be good for you (SJ) to take a look at that for us.

For batch migration interface, though, to be honest I don't find very clear DAMON's usage of it, since DAMON does region-based sort of batched migration. Again, I took only a glance on migration batching part and gonna take more time to the details.

DAMON would identify a set of PFNs or Folios to migrate, at some point, wouldn't it be beneficial to DAMON to simply re-use:

int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)

If not, then why?

That's why I mean by what would DAMON want out of such an interface. Not the async part, but the underlying migration functions.

~Gregory

#35 lore raw
Quoted 14 lines · On Thu, 22 May 2025 13:40:59 -0400 Gregory Price <[email protected]> wrote:
Quoted 12 lines · SeongJae Park
Quoted 4 lines · On Wed, 21 May 2025 23:08:16 -0400 Gregory Price <[email protected]> wrote:

It seems to me that DAMON might make use of the batch migration interface, so if you need any changes or extensions, it might be good for you (SJ) to take a look at that for us.

For batch migration interface, though, to be honest I don't find very clear DAMON's usage of it, since DAMON does region-based sort of batched migration. Again, I took only a glance on migration batching part and gonna take more time to the details.

Quoted 4 lines

DAMON would identify a set of PFNs or Folios to migrate, at some point, wouldn't it be beneficial to DAMON to simply re-use:

int migrate_misplaced_folio_batch(struct list_head *folio_list, int node)

Good idea. Actually we implemnted DAMOS_MIGRATE_HOT and DAMOS_MIGRATE_COLD instead of DAMOS_PROMOTE and DAMOS_DEMOTE since we didn't sure the promotion path logic is somewhat everyone agreed upon. FYI, I'm planning to revisit the promotion path logic later (don't wait for me though ;) ).

Quoted 2 lines

If not, then why?

I'll need to look into the detail, but from a high level glance I think it is a good idea.

Quoted 3 lines

That's why I mean by what would DAMON want out of such an interface. Not the async part, but the underlying migration functions.

Makes sense, thank you!

Thanks,
SJ

[...]

#36 lore raw

Apologies and clarifications on DAMON-disruptions (was Re: [RFC PATCH v0 0/2] Batch migration for NUMA balancing)

In reply to SeongJae Park

On Wed, 21 May 2025 11:45:52 -0700 SeongJae Park <[email protected]> wrote:

[...]

Quoted 9 lines

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

[...]

I heard some people are feeling uncomfortable about patterns on my mails like this. I understand the pattern is that I suddenly replying to a thread saying "hey, by the way DAMON is ...", and I understand it bothers people when they wanted to discuss about something more than DAMON.

I never intended to make others feel uncomfortable but was doing that with only good faith to make discussions be made with full information. But if you felt so, you felt so. I sincerely feel sorry if you felt so, and will try my best to not bother you next time.

But, I'm a human and I cannot do something more than my best effort. I hence expect I will unintentionally continue making people upset. I think I migt be able to reduce such cases by explaining why and what I'm doing, and how you can avoid it.

Yes, I might bothering you with exactly the pattern now, but let me do this hopefully as the last time, to reduce next recurrences.

TL; DR: please briefly mention DAMON and optionally clarify DAMON discussion and/or clarifications are unwelcome for now, if you don't want to be bothered by DAMON.

Why and What I'm Doing
======================

I sometimes find threads of works that seem related with DAMON. From such mails, I find opportunities to help the work using DAMON, shiny features of the work that DAMON could adopt, or whatever makes me believe so. Such mails sometimes contain explanations of such relations, and sometimes not.

In some cases such relations description has some details that look important on the context but missed, outdated, or wrong. I believe I have a responsibility to add or fix those as a maintainer of DAMON for making Linux kernel healthier with discussions under full information, and so jump in.

If there is no such relations description at all, I have no ability to know if they just don't know about DAMON, mistakenly didn't add the part, or want to ignore DAMON at the moment. Hence again I believe it is a responsibility of DAMON maintainer to tell about it, to help the original author and other reviewers' understanding.

I think sometimes you didn't want to discuss or know such details of DAMON but I was jumping in and waste your time since I misread your intention.

How You Can Prevent I Bothering You with DAMON
==============================================

Please help me better understand your intentions. If you knwo DAMON and you feel I may find your work is related with DAMON, please take a moment to add a brief paragraph explaining it. If you don't want me adding more details that I believe worthy to be added there, or don't want to discuss in that detail of DAMON, please mention so. E.g., "this may have potential relations with DAMON, but that's out of the scope at the moment."

I will also do my best to read implicit intention, but again, the best effort is the best effort and I cannot promise something I cannot do.

Thanks,
SJ

[...]

#37 lore raw
In reply to SeongJae Park

Hi SJ,

Quoted 41 lines · SeongJae Park

Hi Bharata,

Quoted 8 lines · On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 11 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

Also in LSFMM, I proposed damos_add_folio() for a case that callers want to utilize DAMON worker thread (kdamond) as an asynchronous memory management operations execution thread while using its other features such as [auto-tuned] quotas. I think this has a potential to serve something similar to your migration threads. I haven't started damos_add_folio() development yet, though.

Quoted 8 lines

I remember we discussed about DAMON on mailing list and in LSFMM a bit, on your session. IIRC, you were also looking for a time to see if there is a chance to use DAMON in some way. Due to the technical issue, we were unable to discuss on the two new proposals on my LSFMM session, and it has been a bit while since our last discussion. So if you don't mind, I'd like to ask if you have some opinions or comments about these.

[1] https://lwn.net/Articles/1016525/

Since this patchset was just about making the migration batched and async for NUMAB, I didn't mention DAMON as an alternative here.

One of the concerns I always had about DAMON when it is considered as replacement for existing hot page migration is its current inability to gather and maintain hot page info at per-folio granularity. How much that eventually matters to the workloads has to be really seen.

Regards,
Bharata.

Regards,
Bharata.

#38 lore raw
Quoted 51 lines · On Mon, 26 May 2025 10:50:02 +0530 Bharata B Rao <[email protected]> wrote:

Hi SJ,

Quoted 48 lines · SeongJae Park

Hi Bharata,

Quoted 8 lines · On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 11 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads. Or if we manage to have a single source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

I think (or, hope) it would also be not very worthless or rude to mention other existing and ongoing works that have potentials to serve for similar purpose or collaborate in future, here.

DAMON is designed for a sort of multi-source access information handling. In LSFMM, I proposed[1] damon_report_access() interface for making it easier to be extended for more types of access information. Currenlty damon_report_access() is under early development. I think this has a potential to serve something similar to your single source goal.

Also in LSFMM, I proposed damos_add_folio() for a case that callers want to utilize DAMON worker thread (kdamond) as an asynchronous memory management operations execution thread while using its other features such as [auto-tuned] quotas. I think this has a potential to serve something similar to your migration threads. I haven't started damos_add_folio() development yet, though.

I remember we discussed about DAMON on mailing list and in LSFMM a bit, on your session. IIRC, you were also looking for a time to see if there is a chance to use DAMON in some way. Due to the technical issue, we were unable to discuss on the two new proposals on my LSFMM session, and it has been a bit while since our last discussion. So if you don't mind, I'd like to ask if you have some opinions or comments about these.

Quoted 4 lines
Quoted 1 line

[1] https://lwn.net/Articles/1016525/

Since this patchset was just about making the migration batched and async for NUMAB, I didn't mention DAMON as an alternative here.

I was thinking a clarification like this could be useful for readers though, since you were mentioning the future work together. Thank you for clarifying.

Quoted 4 lines

One of the concerns I always had about DAMON when it is considered as replacement for existing hot page migration is its current inability to gather and maintain hot page info at per-folio granularity.

I think this is a very valid concern. But I don't think DAMON should be a _replacement_. Rather, I'm looking for a chance to make existing approaches help each other. For example, I recommend running DAMON-based memory tiering[1] together with the LRU-based demotion. I think there is no reason to discourage using it together with NUMAB-2 based promotion, if the folio granularity is a real issue. That is, still NUMAB-2 will do synchronous promotion, but DAMON will do it asynchronously, so the amount of synchronous promotions and its overhead will reduce.

I didn't encourage using NUMB-2 based promotion together with DAMON-based memory tiering[1] not because I show a problem at such co-usage, but just because I found no clear benefit of that from my test setup. In theory, I think running those together makes sense.

That said, we're also making efforts for overcoming the folio-granularity issue on DAMON side, too. We implemented page-level filters that motivated by SK hynix' test results, and developed monitoring intervals auto-tuning for overall monitoring results accuracy. We proposed damon_report_access() and damos_add_folios() as yet another opportunitis to better deal with the issue. I was curious about your opinion to damon_report_access() and damos_add_folios() for the reason. I understand that could be out of the scope of this patch series, though.

Quoted 2 lines

How much
that eventually matters to the workloads has to be really seen.

Cannot agree more. Nonetheless, as mentioned abovely, my test setup[1] didn't show the problem. That said, I'm not really convinced with my test setup, and I don't think the test setup is good for verifying the problem. Hence I'm trying to make a better test setup for this. I'll share more of the new setup if I make some progress. I will also be more than happy to learn about other's test setup if they have a good one or suggestions.

[1] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

reviewer
#39 lore raw
In reply to Bharata B Rao

Hi, Bharata,

Quoted 8 lines · Bharata B Rao

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 5 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads.

What is the expected benefit of the change?

For code reuse, we can use migrate_misplaced_folio() or migrate_misplaced_folio_batch() in various promotion path.

For workload latency influence, per my understanding, PTE scanning is much more serious than migration. Why not start from that?

Quoted 70 lines

Or if we manage to have a single
source of hotness truth like kpromoted[3], then that too can hand over migration requests to the migration threads. I am envisaging that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] and CXL HMU would push hot page info to kpromoted, which would then isolate and push the folios to be promoted to the migrator thread.

As a first step, this is an attempt to batch and perform NUMAB migrations in async manner. Separate migration threads aren't yet implemented but I am using Gregory's patch[7] that provides migrate_misplaced_folio_batch() API to do batch migration of misplaced folios.

Some points for discussion
--------------------------
1. To isolate the misplaced folios or not?

To do batch migration, the misplaced folios need to be stored in some manner. I thought isolating them and using the folio->lru field to link them up would be the most straight-forward way. But then there were concerns expressed about folios remaining isolated for long until they get migrated.

Or should we just maintain the PFNs instead of folios and isolate them only just prior to migrating them?

2. Managing target_nid for misplaced pages

NUMAB provides the accurate target_nid for each folio that is detected as misplaced. However when we don't migrate the folio right away, but instead want to batch and do asyn migration later, then where do we keep track of target_nid for each folio?

In this implementation, I am using last_cpupid field as it appeared that this field could be reused (with some challenges mentioned in 2/2) for isolated folios. This approach may be specific to NUMAB but then each sub-system that hands over pages to the migrator thread should also provide a target_nid and hence each sub-system should be free to maintain and track the target_nid of folios that it has isolated/batched for migration in its own specific manner.

3. How many folios to batch?

Currently I have a fixed threshold for number of folios to batch. It could be a sysctl to allow a setting between a min and max. It could also be auto-tuned if required.

The state of the patchset
-------------------------
* Still raw and very lightly tested
* Just posted to serve as base for subsequent discussions here and in MM alignment calls.

References
----------
[1] LSFMM LWN summary - https://lwn.net/Articles/1016519/
[2] MM alignment call summary - https://lore.kernel.org/linux-mm/[email protected]/
[3] kpromoted patchset - https://lore.kernel.org/linux-mm/[email protected]/
[4] Kmmscand: PTE A bit scanning - https://lore.kernel.org/linux-mm/[email protected]/
[5] MGLRU scanning for page promotion - https://lore.kernel.org/lkml/[email protected]/
[6] IBS base hot page promotion - https://lore.kernel.org/linux-mm/[email protected]/
[7] Unmapped page cache folio promotion patchset - https://lore.kernel.org/linux-mm/[email protected]/

Bharata B Rao (1):
mm: sched: Batch-migrate misplaced pages

Gregory Price (1):
migrate: implement migrate_misplaced_folio_batch

Quoted 7 lines
 include/linux/migrate.h |  6 ++++
 include/linux/sched.h   |  4 +++
 init/init_task.c        |  2 ++
 kernel/sched/fair.c     | 64 +++++++++++++++++++++++++++++++++++++++++
 mm/memory.c             | 44 ++++++++++++++--------------
 mm/migrate.c            | 31 ++++++++++++++++++++
 6 files changed, 130 insertions(+), 21 deletions(-)

---
Best Regards,
Huang, Ying

#40 lore raw
Quoted 12 lines · Ying

Hi, Bharata,

Quoted 8 lines · Bharata B Rao

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 7 lines
Quoted 5 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads.

What is the expected benefit of the change?

Initially it is about cleanliness and separation of migration into its own thread/sub-system.

Quoted 3 lines

For code reuse, we can use migrate_misplaced_folio() or migrate_misplaced_folio_batch() in various promotion path.

That's what I have done in this patchset at least. We thought we could go full length and off-load migration to its own thread.

Quoted 3 lines

For workload latency influence, per my understanding, PTE scanning is much more serious than migration. Why not start from that?

Raghu's PTE A bit scanning is one effort towards that (Removing PTE scanning from task context.

Regards,
Bharata.

reviewer
#41 lore raw
Quoted 19 lines · Bharata B Rao
Quoted 11 lines · Ying

Hi, Bharata,

Quoted 8 lines · Bharata B Rao

Hi,

This is an attempt to convert the NUMA balancing to do batched migration instead of migrating one folio at a time. The basic idea is to collect (from hint fault handler) the folios to be migrated in a list and batch-migrate them from task_work context. More details about the specifics are present in patch 2/2.

Quoted 6 lines
Quoted 5 lines

During LSFMM[1] and subsequent discussions in MM alignment calls[2], it was suggested that separate migration threads to handle migration or promotion request may be desirable. Existing NUMA balancing, hot page promotion and other future promotion techniques could off-load migration part to these threads.

What is the expected benefit of the change?

Quoted 8 lines

Initially it is about cleanliness and separation of migration into its own thread/sub-system.

Quoted 2 lines

For code reuse, we can use migrate_misplaced_folio() or migrate_misplaced_folio_batch() in various promotion path.

That's what I have done in this patchset at least. We thought we could go full length and off-load migration to its own thread.

Even if we migrate pages in another thread, the migrated pages will be unmapped, copied, remapped during migrating. That is, the workload threads may be stalled to wait for migrating. So, we need to measure the real benefit firstly.

Quoted 6 lines
Quoted 3 lines

For workload latency influence, per my understanding, PTE scanning is
much more serious than migration. Why not start from that?

Raghu's PTE A bit scanning is one effort towards that (Removing PTE scanning from task context.

---
Best Regards,
Huang, Ying