sched/numa: Introduce per cgroup numa balance control

Introduce a per-cgroup interface to enable NUMA balancing for specific cgroups. The system administrator needs to set the NUMA balancing mode to NUMA_BALANCING_CGROUP=4 to enable this feature. When in the NUMA_BALANCING_CGROUP mode, all cgroups' NUMA balancing is disabled by default. After the administrator enables this feature for a specific cgroup, NUMA balancing for that cgroup is enabled.

This per-cgroup NUMA balancing control was once proposed in 2019 by Yun Wang[1]. Then, in 2024, Kaiyang Zhao mentioned that he was working with Meta on per-cgroup NUMA control[2] during a discussion with David Rientjes.

I could not find further discussion regarding per-cgroup NUMA balancing from that point on. This set of RFC patches is a rough and compile-passed version, and may have unhandled cases (for example, THP). It has not been thoroughly tested and is intended to initiate or resume the discussion on the topic of per-cgroup NUMA load balancing.

The first patch is a NUMA load balancing statistics enhancement. The second patch introduces per-cgroup NUMA balancing. The third one enhances NUMA load balancing for the MPOL_INTERLEAVE policy.

Any feedback would be appreciated.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/T/

Chen Yu (3):
sched/numa: Introduce numa balance task migration and swap in
schedstats
sched/numa: Introduce per cgroup numa balance control sched/numa: Allow intervale memory allocation for numa balance

 include/linux/numa.h           |  1 +
 include/linux/sched.h          |  4 ++++
 include/linux/sched/sysctl.h   |  1 +
 include/linux/vm_event_item.h  |  2 ++
 include/uapi/linux/mempolicy.h |  1 +
 kernel/sched/core.c            | 42 ++++++++++++++++++++++++++++++++--
 kernel/sched/debug.c           |  4 ++++
 kernel/sched/fair.c            | 18 +++++++++++++++
 kernel/sched/sched.h           |  3 +++
 mm/memcontrol.c                |  2 ++
 mm/memory.c                    |  2 +-
 mm/mempolicy.c                 |  7 ++++++
 mm/mprotect.c                  |  5 ++--
 mm/vmstat.c                    |  2 ++
 14 files changed, 89 insertions(+), 5 deletions(-)

2.25.1

There is a requirement to track task activities during NUMA balancing. NUMA balancing has two mechanisms for task migration:
one is to migrate the task to an idle CPU in its preferred node, and the other is to swap tasks on different nodes if they are on each other's preferred node. The kernel already has NUMA page migration statistics. Add the task migration and swap count described above in the per-task/cgroup scope. The data will be displayed at

/sys/fs/cgroup/mytest/memory.stat and
/proc/{PID}/sched.

---
 include/linux/sched.h         |  4 ++++
 include/linux/vm_event_item.h |  2 ++
 kernel/sched/core.c           | 10 ++++++++--
 kernel/sched/debug.c          |  4 ++++
 mm/memcontrol.c               |  2 ++
 mm/vmstat.c                   |  2 ++
 6 files changed, 22 insertions(+), 2 deletions(-)

index 9632e3318e0d..01faa608ed7c 100644
@@ -527,6 +527,10 @@ struct sched_statistics {
 	u64				nr_failed_migrations_running;
 	u64				nr_failed_migrations_hot;
 	u64				nr_forced_migrations;
+#ifdef CONFIG_NUMA_BALANCING
+	u64				nr_numa_migrations;
+	u64				nr_numa_swap;
+#endif
 
 	u64				nr_wakeups;
 	u64				nr_wakeups_sync;

index f70d0958095c..aef817474781 100644
@@ -64,6 +64,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		NUMA_HINT_FAULTS,
 		NUMA_HINT_FAULTS_LOCAL,
 		NUMA_PAGE_MIGRATE,
+		NUMA_TASK_MIGRATE,
+		NUMA_TASK_SWAP,
 #endif
 #ifdef CONFIG_MIGRATION
 		PGMIGRATE_SUCCESS, PGMIGRATE_FAIL,

index 165c90ba64ea..44efc725054a 100644
@@ -3348,6 +3348,11 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 #ifdef CONFIG_NUMA_BALANCING
 static void __migrate_swap_task(struct task_struct *p, int cpu)
 {
+	__schedstat_inc(p->stats.nr_numa_swap);
+
+	if (p->mm)
+		count_memcg_events_mm(p->mm, NUMA_TASK_SWAP, 1);
+
 	if (task_on_rq_queued(p)) {
 		struct rq *src_rq, *dst_rq;
 		struct rq_flags srf, drf;

@@ -7901,8 +7906,9 @@ int migrate_task_to(struct task_struct *p, int target_cpu)
 	if (!cpumask_test_cpu(target_cpu, p->cpus_ptr))
 		return -EINVAL;
 
-	/* TODO: This is not properly updating schedstats */
-
+	__schedstat_inc(p->stats.nr_numa_migrations);
+	if (p->mm)
+		count_memcg_events_mm(p->mm, NUMA_TASK_MIGRATE, 1);
 	trace_sched_move_numa(p, curr_cpu, target_cpu);
 	return stop_one_cpu(curr_cpu, migration_cpu_stop, &arg);
 }

index ef047add7f9e..ed801cc00bf1 100644
@@ -1204,6 +1204,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		P_SCHEDSTAT(nr_failed_migrations_running);
 		P_SCHEDSTAT(nr_failed_migrations_hot);
 		P_SCHEDSTAT(nr_forced_migrations);
+#ifdef CONFIG_NUMA_BALANCING
+		P_SCHEDSTAT(nr_numa_migrations);
+		P_SCHEDSTAT(nr_numa_swap);
+#endif
 		P_SCHEDSTAT(nr_wakeups);
 		P_SCHEDSTAT(nr_wakeups_sync);
 		P_SCHEDSTAT(nr_wakeups_migrate);

index 46f8b372d212..496b5edc3db6 100644
@@ -460,6 +460,8 @@ static const unsigned int memcg_vm_event_stat[] = {
 	NUMA_PAGE_MIGRATE,
 	NUMA_PTE_UPDATES,
 	NUMA_HINT_FAULTS,
+	NUMA_TASK_MIGRATE,
+	NUMA_TASK_SWAP,
 #endif
 };
 

index 16bfe1c694dd..d6651778e4bf 100644
@@ -1339,6 +1339,8 @@ const char * const vmstat_text[] = {
 	"numa_hint_faults",
 	"numa_hint_faults_local",
 	"numa_pages_migrated",
+	"numa_task_migrated",
+	"numa_task_swaped",
 #endif
 #ifdef CONFIG_MIGRATION
 	"pgmigrate_success",

2.25.1

[Problem Statement]
Currently, NUMA balancing is configured system-wide. However, in some production environments, different containers may have varying requirements for NUMA balancing. Some containers are CPU-intensive, while others are memory-intensive. Some do not benefit from NUMA balancing due to the overhead associated with VMA scanning, while the other prefers NUMA balancing as it helps improve memory locality. In this case, system-wide NUMA balancing is usually disabled to produce stable results.

[Proposal]
Introduce a per-cgroup interface to enable NUMA balancing for specific cgroups. The system administrator must set the NUMA balancing mode to NUMA_BALANCING_CGROUP=4 to enable this feature. When in global NUMA_BALANCING_CGROUP mode, all cgroups' NUMA balancing is disabled by default. After the administrator enables this feature for a specific cgroup, NUMA balancing for that cgroup is enabled.

A simple example to show how to use per-cgroup Numa balancing:

Step1
//switch to global per cgroup Numa balancing, //All cgroup's Numa balance is disabled by default. echo 4 > /proc/sys/kernel/numa_balancing

Step2
//created a cgroup named mytest, enable its Numa balancing echo 1 > /sys/fs/cgroup/mytest/cpu.numa_load_balance

[Benchmark]
Tested on two systems. Both systems have 4 nodes. Created a cgroup mytest which is bind to node0 and node1(cpu affinity as well as memory allocation policy). Launched autonumabench NUMA01_THREADLOCAL via mmtests.

echo 0 > /sys/fs/cgroup/mytest/cpu.numa_load_balance cgexec -g cpuset:mytest ./run-mmtests.sh --no-monitor \
--config config-numa baseline
echo 1 > /sys/fs/cgroup/mytest/cpu.numa_load_balance \
cgexec -g cpuset:mytest ./run-mmtests.sh --no-monitor
--config config-numa nb_cgroup

system1: 4 nodes, 24 Cores(48 CPUs)/node.
baseline took a total of 191.32 seconds to finish, while cgroup numa balancing took a total of 104.46 seconds. There is around 45% improvement.

baselin nb_cgrou
baseline nb_cgroup

Min       syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 ( -53.24%)
Min       elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 (  45.40%)
Amean     syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 * -53.24%*
Amean     elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 *  45.40%*  <---
Stddev    syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Stddev    elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Max       syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 ( -53.24%)
Max       elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 (  45.40%)
BAmean-50 syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 ( -53.24%)
BAmean-50 elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 (  45.40%)
BAmean-95 syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 ( -53.24%)
BAmean-95 elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 (  45.40%)
BAmean-99 syst-NUMA01_THREADLOCAL       69.65 (   0.00%)      106.73 ( -53.24%)
BAmean-99 elsp-NUMA01_THREADLOCAL      191.32 (   0.00%)      104.46 (  45.40%)

The run-to-run deviation downgrading occurs because sometimes the per-cgroup NUMA balancing does not improve the score, although no performance downgrading is observed.

delta of /sys/fs/cgroup/mytest/memory.stat during the test:

numa_pages_migrated: 979933
numa_pte_updates:    21007548  <-- introduced in previous patch
numa_hint_faults:    19663982  <-- introduced in previous patch

system1: 4 nodes, 40 Cores(80 CPUs)/node.
baseline took a total of 212.94 seconds to finish, while cgroup numa balance took a total of 127.05 second, which is of 40.34% improvment.

baselin nb_cgrou
baseline nb_cgroup

Min       syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
Min       elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)
Amean     syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
Amean     elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)  <---
Stddev    syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Stddev    elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Max       syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
Max       elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)
BAmean-50 syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
BAmean-50 elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)
BAmean-95 syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
BAmean-95 elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)
BAmean-99 syst-NUMA01_THREADLOCAL     8356.05 (   0.00%)     8921.84 (  -6.77%)
BAmean-99 elsp-NUMA01_THREADLOCAL      212.94 (   0.00%)      127.05 (  40.34%)

The Numa statistics delta during the test:

numa_pages_migrated:  785848
numa_pte_updates:     2359714
numa_hint_faults:     2349857

[Shortage]
It has been observed that even with per-cgroup NUMA balancing enabled, there is still remote node access, and the benchmark score does not increase compared to the baseline. According to the NUMA statistics, not much NUMA page migration is detected. Further testing shows that global NUMA balancing has the same issue—sometimes NUMA balancing does not help. This could be a generic issue in the current kernel code, possibly due to either the NUMA page migration or task migration strategy, and it needs to be further investigated.

---
 include/linux/sched/sysctl.h |  1 +
 kernel/sched/core.c          | 32 ++++++++++++++++++++++++++++++++
 kernel/sched/fair.c          | 18 ++++++++++++++++++
 kernel/sched/sched.h         |  3 +++
 mm/mprotect.c                |  5 +++--
 5 files changed, 57 insertions(+), 2 deletions(-)

index 5a64582b086b..1e4d5a9ddb26 100644
@@ -22,6 +22,7 @@ enum sched_tunable_scaling {
 #define NUMA_BALANCING_DISABLED		0x0
 #define NUMA_BALANCING_NORMAL		0x1
 #define NUMA_BALANCING_MEMORY_TIERING	0x2
+#define NUMA_BALANCING_CGROUP		0x4
 
 #ifdef CONFIG_NUMA_BALANCING
 extern int sysctl_numa_balancing_mode;

index 44efc725054a..f4f048b3da68 100644
@@ -10023,6 +10023,31 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of,
 }
 #endif
 
+#ifdef CONFIG_NUMA_BALANCING
+static DEFINE_MUTEX(numa_balance_mutex);
+static int numa_balance_write_u64(struct cgroup_subsys_state *css,
+				  struct cftype *cftype, u64 enable)
+{
+	struct task_group *tg;
+	int ret;
+
+	guard(mutex)(&numa_balance_mutex);
+	tg = css_tg(css);
+	if (tg->nlb_enabled == enable)
+		return 0;
+
+	tg->nlb_enabled = enable;
+
+	return ret;
+}
+
+static u64 numa_balance_read_u64(struct cgroup_subsys_state *css,
+				 struct cftype *cft)
+{
+	return css_tg(css)->nlb_enabled;
+}
+#endif /* CONFIG_NUMA_BALANCING */
+
 static struct cftype cpu_files[] = {
 #ifdef CONFIG_GROUP_SCHED_WEIGHT
 	{

@@ -10071,6 +10096,13 @@ static struct cftype cpu_files[] = {
 		.seq_show = cpu_uclamp_max_show,
 		.write = cpu_uclamp_max_write,
 	},
+#endif
+#ifdef CONFIG_NUMA_BALANCING
+	{
+		.name = "numa_load_balance",
+		.read_u64 = numa_balance_read_u64,
+		.write_u64 = numa_balance_write_u64,
+	},
 #endif
 	{ }	/* terminate */
 };

index 1c0ef435a7aa..526cb33b007c 100644
@@ -3146,6 +3146,18 @@ void task_numa_free(struct task_struct *p, bool final)
 	}
 }
 
+/* return true if the task group has enabled the numa balance */
+static bool tg_numa_balance_enabled(struct task_struct *p)
+{
+	struct task_group *tg = task_group(p);
+
+	if (tg && (sysctl_numa_balancing_mode & NUMA_BALANCING_CGROUP) &&
+	    !tg->nlb_enabled)
+		return false;
+
+	return true;
+}
+
 /*
  * Got a PROT_NONE fault for a page on @node.
  */

@@ -3174,6 +3186,9 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	     !cpupid_valid(last_cpupid)))
 		return;
 
+	if (!tg_numa_balance_enabled(p))
+		return;
+
 	/* Allocate buffer to track faults on a per-node basis */
 	if (unlikely(!p->numa_faults)) {
 		int size = sizeof(*p->numa_faults) *

@@ -3596,6 +3611,9 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)
 	if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next != work)
 		return;
 
+	if (!tg_numa_balance_enabled(curr))
+		return;
+
 	/*
 	 * Using runtime rather than walltime has the dual advantage that
 	 * we (mostly) drive the selection from busy threads and that the

index 38e0e323dda2..9f478fb2c03a 100644
@@ -491,6 +491,9 @@ struct task_group {
 	/* Effective clamp values used for a task group */
 	struct uclamp_se	uclamp[UCLAMP_CNT];
 #endif
+#ifdef CONFIG_NUMA_BALANCING
+	u64			nlb_enabled;
+#endif
 
 };
 

index 516b1d847e2c..ddaaf20ef94c 100644
@@ -155,10 +155,11 @@ static long change_pte_range(struct mmu_gather *tlb,
 				toptier = node_is_toptier(nid);
 
 				/*
-				 * Skip scanning top tier node if normal numa
+				 * Skip scanning top tier node if normal/cgroup numa
 				 * balancing is disabled
 				 */
-				if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
+				if (!(sysctl_numa_balancing_mode &
+				    (NUMA_BALANCING_CGROUP | NUMA_BALANCING_NORMAL)) &&
 				    toptier)
 					continue;
 				if (folio_use_access_time(folio))

2.25.1

Can you add documentation of this additional feature for numa_balancing in admin-guide/sysctl/kernel.rst

Should you make NUMA_BALANCING_NORMAL and NUMA_BALANCING_CGROUP mutually exclusive in? In other words
echo 5 > /proc/sys/kernel/numa_balancing should result in numa_balancing to be 1?

Otherwise tg_numa_balance_enabled() can return 0 with NUMA_BALANCING_CGROUP bit turned on even though you have NUMA_BALANCING_NORMAL bit on.

Tim

OK, will refine in next version.

I see, will fix tg_numa_balance_enabled() in next version, thanks!

Best,
Chenyu

MPOL_INTERLEAVE is used to allocate pages interleaved across different NUMA nodes to make the best use of memory bandwidth. Under MPOL_INTERLEAVE mode, NUMA load balance page migration does not occur because the page is already in its designated place. Similarly, NUMA load task migration does not occur either—mpol_misplaced() returns NUMA_NO_NODE, which instructs do_numa_page() to skip page/task migration. However, there is a scenario in the production environment where NUMA balance could benefit MPOL_INTERLEAVE. This typical scenario involves tasks within cgroup g_A being bound to two SNC (Sub-NUMA Cluster) nodes via cpuset, with their pages allocated only on these two SNC nodes in an interleaved manner using MPOL_INTERLEAVE. This setup allows g_A to achieve good resource isolation while effectively utilizing the memory bandwidth of the two SNC nodes. However, it is possible that tasks t1 and t2 in g_A could experience remote access patterns:

	Node 0		Node 1
	t1		t1.page
	t2.page		t2

Ideally, a NUMA balance task swap would be beneficial:

	Node 0		Node 1
	t2		t1.page
	t2.page		t1

In other words, NUMA balancing can help swap t1 and t2 to improve NUMA locality without migrating pages, thereby still honoring the MPOL_INTERLEAVE policy. To enable NUMA balancing to manage MPOL_INTERLEAVE, add MPOL_F_MOF to the MPOL_INTERLEAVE policy if the user has requested it via MPOL_F_NUMA_BALANCING (similar to MPOL_BIND). In summary, pages will not be migrated for MPOL_INTERLEAVE, but tasks will be migrated to their preferred nodes.

Tested on a system with 4 nodes, 40 Cores(80 CPUs)/node, using autonumabench NUMA01_THREADLOCAL, with some minor changes to support MPOL_INTERLEAVE:

  p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, \
		MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  set_mempolicy(MPOL_INTERLEAVE | MPOL_F_NUMA_BALANCING, \
		&nodemask_global, max_nodes);

...
//each thread accesses 4K of data every 8K,
//1 thread should access the pages on 1 node.

No obvious score difference was observed, but noticed some Numa balance task migration:

baseline_nocg_interleav nb_nocg_interlave
baseline_nocg_interleave nb_nocg_interlave/

Min       syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
Min       elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)
Amean     syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
Amean     elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)
Stddev    syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Stddev    elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  syst-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
CoeffVar  elsp-NUMA01_THREADLOCAL        0.00 (   0.00%)        0.00 (   0.00%)
Max       syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
Max       elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)
BAmean-50 syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
BAmean-50 elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)
BAmean-95 syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
BAmean-95 elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)
BAmean-99 syst-NUMA01_THREADLOCAL     7156.34 (   0.00%)     7267.28 (  -1.55%)
BAmean-99 elsp-NUMA01_THREADLOCAL       90.73 (   0.00%)       90.88 (  -0.17%)

delta of /sys/fs/cgroup/mytest/memory.stat during the test:

numa_pages_migrated: 0
numa_pte_updates:    9156154
numa_hint_faults:    8659673
numa_task_migrated:  282  <--- introduced in previous patch
numa_task_swaped:    114  <---- introduced in previous patch

More tests to come.

---
 include/linux/numa.h           | 1 +
 include/uapi/linux/mempolicy.h | 1 +
 mm/memory.c                    | 2 +-
 mm/mempolicy.c                 | 7 +++++++
 4 files changed, 10 insertions(+), 1 deletion(-)

index 3567e40329eb..6c3f2d839c76 100644
@@ -14,6 +14,7 @@
 
 #define	NUMA_NO_NODE	(-1)
 #define	NUMA_NO_MEMBLK	(-1)
+#define	NUMA_TASK_MIG	(1)
 
 static inline bool numa_valid_node(int nid)
 {

index 1f9bb10d1a47..2081365612ac 100644
@@ -64,6 +64,7 @@ enum {
 #define MPOL_F_SHARED  (1 << 0)	/* identify shared policies */
 #define MPOL_F_MOF	(1 << 3) /* this policy wants migrate on fault */
 #define MPOL_F_MORON	(1 << 4) /* Migrate On protnone Reference On Node */
+#define MPOL_F_MOFT	(1 << 5) /* allow task but no page migrate on fault */
 
 /*
  * These bit locations are exposed in the vm.zone_reclaim_mode sysctl

index 539c0f7c6d54..4013bbcbf40f 100644
@@ -5683,7 +5683,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
-	if (target_nid == NUMA_NO_NODE)
+	if (target_nid == NUMA_NO_NODE || target_nid == NUMA_TASK_MIG)
 		goto out_map;
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;

index bbaadbeeb291..0b88601ec22d 100644
@@ -1510,6 +1510,8 @@ static inline int sanitize_mpol_flags(int *mode, unsigned short *flags)
 	if (*flags & MPOL_F_NUMA_BALANCING) {
 		if (*mode == MPOL_BIND || *mode == MPOL_PREFERRED_MANY)
 			*flags |= (MPOL_F_MOF | MPOL_F_MORON);
+		else if (*mode == MPOL_INTERLEAVE)
+			*flags |= (MPOL_F_MOF | MPOL_F_MOFT);
 		else
 			return -EINVAL;
 	}

@@ -2779,6 +2781,11 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf,
 	if (!(pol->flags & MPOL_F_MOF))
 		goto out;
 
+	if (pol->flags & MPOL_F_MOFT) {
+		ret = NUMA_TASK_MIG;
+		goto out;
+	}
+
 	switch (pol->mode) {
 	case MPOL_INTERLEAVE:
 		polnid = interleave_nid(pol, ilx);

2.25.1

Hello Chen,

It's nice to see people interested in this. I posted a set of RFC patches later[1] that focuses on the fairness issue in memory tiering. It mostly concerns the demotion side of things, and the promotion / NUMA balancing side of things was left out of the patch set.

I don't work for Meta now, but my understanding is that they'll attempt to push through a solution for per-cgroup control of memory tiering that is in the same vein as my RFC patches, and it may include controls for per-group NUMA balancing in the context of tiered memory.

Best,
Kaiyang

[1] https://lore.kernel.org/linux-mm/[email protected]/

Hi Kaiyang,

I see, thanks for the information.

OK, it would be nice to see that patch set. We can continue the disscussion on this basic per-cgroup Numa balancing control, the tiered memory promotion could be on top of that IMO.

thanks,
Chenyu