Skip to contentSkip to conversation

cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

PATCH Superseded Waiman Long 2 participants 14 messages May 27 – May 29 3 messages not in the archive

Subsystems (2)

v3 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Waiman Long

author maintainer
#0 lore raw

v3 cover cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

v3:
- Rebased to the lastest linux-next tree.
- Keep cpuset_attach_old_cs as suggested by Chen Ridong and replace patch 3 by a new one to make it track task group leader.

Sashiko AI review of another cpuset patch had found that cpuset_attach() and cpuset_can_attach() can be passed a cgroup_taskset with tasks migrating from one source cpuset to multiple destination cpusets and vice versa. Further testing of the cpuset code indicates that this is indeed the case when the v2 cpuset controller is enabled or disabled.

Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that there will be one source and one destinaton cpuset which may result in inocrrect behavior.

This patch series is created to fix this issue. The first 2 patches are just preparatory patches to make the remaining patches easier to review.

Patch 3 makes cpuset_attach_old_cs to track group leader for use by cpuset_migrate_mm().

Patch 4 moves mpol_rebind_mm() and cpuset_migrate_mm() inside cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works more like moving task from one cpuset to another one, while also make supporting multiple source and destination cpusets easier.

Patch 5 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.

Waiman Long (5):
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper cgroup/cpuset: Expand the scope of cpuset_can_attach_check() cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()

 kernel/cgroup/cpuset-internal.h |   6 +
 kernel/cgroup/cpuset.c          | 375 ++++++++++++++++++++++----------
 2 files changed, 263 insertions(+), 118 deletions(-)
Signature · 1 line
2.54.0
author maintainer
#1 lore raw

PATCH v3 1/5 cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper

Extract the DL bandwidth allocation code in cpuset_attach() to a new cpuset_reserve_dl_bw() helper to simplify code.

No functional change is expected.

---
 kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------
 1 file changed, 30 insertions(+), 23 deletions(-)
kernel/cgroup/cpuset.c +30 -23
index 51327333980a..d720bcc7ef83 100644
@@ -2980,6 +2980,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)
 	return 0;
 }
 
+static int cpuset_reserve_dl_bw(struct cpuset *cs)
+{
+	int cpu, ret;
+
+	if (!cs->sum_migrate_dl_bw)
+		return 0;
+
+	cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+	if (unlikely(cpu >= nr_cpu_ids))
+		return -EINVAL;
+
+	ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+	if (ret)
+		return ret;
+
+	cs->dl_bw_cpu = cpu;
+	return 0;
+}
+
 static void reset_migrate_dl_data(struct cpuset *cs)
 {
 	cs->nr_migrate_dl_tasks = 0;
@@ -2994,7 +3013,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	struct cpuset *cs, *oldcs;
 	struct task_struct *task;
 	bool setsched_check;
-	int cpu, ret;
+	int ret;
 
 	/* used later by cpuset_attach() */
 	cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
@@ -3050,31 +3069,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 		}
 	}
 
-	if (!cs->sum_migrate_dl_bw)
-		goto out_success;
-
-	cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
-	if (unlikely(cpu >= nr_cpu_ids)) {
-		ret = -EINVAL;
-		goto out_unlock;
-	}
-
-	ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
-	if (ret)
-		goto out_unlock;
-
-	cs->dl_bw_cpu = cpu;
-
-out_success:
-	/*
-	 * Mark attach is in progress.  This makes validate_change() fail
-	 * changes which zero cpus/mems_allowed.
-	 */
-	cs->attach_in_progress++;
+	ret = cpuset_reserve_dl_bw(cs);
 
 out_unlock:
-	if (ret)
+	if (ret) {
 		reset_migrate_dl_data(cs);
+	} else {
+		/*
+		 * Mark attach is in progress.  This makes validate_change() fail
+		 * changes which zero cpus/mems_allowed.
+		 */
+		cs->attach_in_progress++;
+	}
+
 	mutex_unlock(&cpuset_mutex);
 	return ret;
 }
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v3 2/5 cgroup/cpuset: Expand the scope of cpuset_can_attach_check()

In reply to Waiman Long

Expand the scope of cpuset_can_attach_check() by including the setting of setsched flag inside cpuset_can_attach_check() with the new @oldcs and @psetsched argument. As cpuset_can_attach_check() is also called from cpuset_can_fork(), set the new arguments to NULL from that caller.

While at it, expose the source and destination cpuset cpu/memory check results in the new attach_cpus_updated and attach_mems_updated static flags so that these flags can be used directly from cpuset_attach() without the need to do the same computations again.

No functional change is expected.

---
 kernel/cgroup/cpuset.c | 70 +++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 28 deletions(-)
kernel/cgroup/cpuset.c +42 -28
index d720bcc7ef83..4457c4f11fce 100644
@@ -2964,19 +2964,56 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 	return 0;
 }
 
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
 static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
 
 /*
  * Check to see if a cpuset can accept a new task
  * For v1, cpus_allowed and mems_allowed can't be empty.
  * For v2, effective_cpus can't be empty.
  * Note that in v1, effective_cpus = cpus_allowed.
+ *
+ * Also set the boolean flag passed in by @psetsched depending on if
+ * security_task_setscheduler() call is needed and @oldcs is not NULL.
  */
-static int cpuset_can_attach_check(struct cpuset *cs)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
+				   bool *psetsched)
 {
 	if (cpumask_empty(cs->effective_cpus) ||
 	   (!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
 		return -ENOSPC;
+
+	if (!oldcs)
+		return 0;
+
+	/*
+	 * Update attach specific data
+	 */
+	attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+	attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+	/*
+	 * Skip rights over task setsched check in v2 when nothing changes,
+	 * migration permission derives from hierarchy ownership in
+	 * cgroup_procs_write_permission()).
+	 */
+	*psetsched = !cpuset_v2() || attach_cpus_updated || attach_mems_updated;
+
+	/*
+	 * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
+	 * brings the last online CPU offline as users are not allowed to empty
+	 * cpuset.cpus when there are active tasks inside. When that happens,
+	 * we should allow tasks to migrate out without security check to make
+	 * sure they will be able to run after migration.
+	 */
+	if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
+		*psetsched = false;
+
 	return 0;
 }
 
@@ -3023,29 +3060,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	mutex_lock(&cpuset_mutex);
 
 	/* Check to see if task is allowed in the cpuset */
-	ret = cpuset_can_attach_check(cs);
+	ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
 	if (ret)
 		goto out_unlock;
 
-	/*
-	 * Skip rights over task setsched check in v2 when nothing changes,
-	 * migration permission derives from hierarchy ownership in
-	 * cgroup_procs_write_permission()).
-	 */
-	setsched_check = !cpuset_v2() ||
-		!cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
-		!nodes_equal(cs->effective_mems, oldcs->effective_mems);
-
-	/*
-	 * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
-	 * brings the last online CPU offline as users are not allowed to empty
-	 * cpuset.cpus when there are active tasks inside. When that happens,
-	 * we should allow tasks to migrate out without security check to make
-	 * sure they will be able to run after migration.
-	 */
-	if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
-		setsched_check = false;
-
 	cgroup_taskset_for_each(task, css, tset) {
 		ret = task_can_attach(task);
 		if (ret)
@@ -3140,7 +3158,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	struct cgroup_subsys_state *css;
 	struct cpuset *cs;
 	struct cpuset *oldcs = cpuset_attach_old_cs;
-	bool cpus_updated, mems_updated;
 	bool queue_task_work = false;
 
 	cgroup_taskset_first(tset, &css);
@@ -3148,9 +3165,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 
 	lockdep_assert_cpus_held();	/* see cgroup_attach_lock() */
 	mutex_lock(&cpuset_mutex);
-	cpus_updated = !cpumask_equal(cs->effective_cpus,
-				      oldcs->effective_cpus);
-	mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
 
 	/*
 	 * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3158,7 +3172,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	 * in effective cpus and mems. In that case, we can optimize out
 	 * by skipping the task iteration and update.
 	 */
-	if (cpuset_v2() && !cpus_updated && !mems_updated) {
+	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
 		cpuset_attach_nodemask_to = cs->effective_mems;
 		goto out;
 	}
@@ -3175,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	 * not set.
 	 */
 	cpuset_attach_nodemask_to = cs->effective_mems;
-	if (!is_memory_migrate(cs) && !mems_updated)
+	if (!is_memory_migrate(cs) && !attach_mems_updated)
 		goto out;
 
 	cgroup_taskset_for_each_leader(leader, css, tset) {
@@ -3590,7 +3604,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
 	mutex_lock(&cpuset_mutex);
 
 	/* Check to see if task is allowed in the cpuset */
-	ret = cpuset_can_attach_check(cs);
+	ret = cpuset_can_attach_check(cs, NULL, NULL);
 	if (ret)
 		goto out_unlock;
 
Signature · 1 line
2.54.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#4 lore raw

Re: [PATCH-next v3 2/5] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()

Quoted 67 lines · On 5/27/26 12:33 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Caching attach_cpus_updated and attach_mems_updated in global variables during can_attach introduces a Time-Of-Check to Time-Of-Use (TOCTOU) race condition that can allow tasks to escape CPU/memory isolation boundaries. --

commit 3fc8c4e50c216b4d34d25e7c56636a5ec10e50a6
Author: Waiman Long <[email protected]>

cgroup/cpuset: Expand the scope of cpuset_can_attach_check()

This commit expands cpuset_can_attach_check() to include setsched flag calculations. It also caches the source and destination mask update results in static variables to avoid recomputing them in cpuset_attach().

Quoted 15 lines
kernel/cgroup/cpuset.c +6 -0
index d720bcc7ef83c..4457c4f11fcea 100644
@@ -2964,19 +2964,56 @@ static int update_prstate(struct cpuset *cs, int new_prs)
  	return 0;
  }
 
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
  static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;

[ ... ]

Quoted 19 lines
kernel/cgroup/cpuset.c +0 -4
@@ -3140,7 +3158,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
  	struct cgroup_subsys_state *css;
  	struct cpuset *cs;
  	struct cpuset *oldcs = cpuset_attach_old_cs;
-	bool cpus_updated, mems_updated;
  	bool queue_task_work = false;
 
  	cgroup_taskset_first(tset, &css);
@@ -3148,9 +3165,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 
  	lockdep_assert_cpus_held();	/* see cgroup_attach_lock() */
  	mutex_lock(&cpuset_mutex);
-	cpus_updated = !cpumask_equal(cs->effective_cpus,
-				      oldcs->effective_cpus);
-	mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
 
  	/*
  	 * In the default hierarchy, enabling cpuset in the child cgroups

kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3175

Quoted 5 lines
kernel/cgroup/cpuset.c +1 -1
@@ -3158,7 +3172,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
  	 * in effective cpus and mems. In that case, we can optimize out
  	 * by skipping the task iteration and update.
  	 */
-	if (cpuset_v2() && !cpus_updated && !mems_updated) {
+	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {

[Severity: High]
Does caching the attach_cpus_updated and attach_mems_updated flags during cpuset_can_attach() introduce a Time-Of-Check to Time-Of-Use race condition?

The cgroup migration process separates the can_attach and attach phases, dropping locks like cpuset_mutex between them. If a concurrent update to the destination cpuset's masks (e.g., via a CPU hotplug event or writing to cpuset.cpus) occurs while the lock is dropped:

1. The destination's effective_cpus are modified, updating tasks already in the cpuset.

2. The migrating task is not yet in the new cpuset's task list, so it isn't updated.

Quoted 7 lines

3. When cpuset_attach() re-acquires cpuset_mutex, it evaluates the stale cached flags instead of rechecking the masks.

If the masks were identical during can_attach, the flags remain false, causing cpuset_attach() to take the fast path and entirely skip cpuset_attach_task(). Could this leave the migrating task with its old, incorrect masks, allowing it to escape its designated isolation boundary?

Yes, it can be a valid concern. One workaround is to check the attach_in_progress flag in the cpuset. If it is set, we just have to clear the corresponding global flag when either the memory nodes or CPUs are changed.

Cheers,
Longman

Quoted 4 lines
Quoted 4 lines
kernel/cgroup/cpuset.c +0 -0
  		cpuset_attach_nodemask_to = cs->effective_mems;
  		goto out;
  	}
author maintainer
#5 lore raw

Re: [PATCH-next v3 2/5] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()

Quoted 18 lines · Waiman Long
Quoted 18 lines

Severity: High]
Does caching the attach_cpus_updated and attach_mems_updated flags during cpuset_can_attach() introduce a Time-Of-Check to Time-Of-Use race condition?

The cgroup migration process separates the can_attach and attach phases, dropping locks like cpuset_mutex between them. If a concurrent update to the destination cpuset's masks (e.g., via a CPU hotplug event or writing to cpuset.cpus) occurs while the lock is dropped:

1. The destination's effective_cpus are modified, updating tasks already in the cpuset.

2. The migrating task is not yet in the new cpuset's task list, so it isn't updated.

3. When cpuset_attach() re-acquires cpuset_mutex, it evaluates the stale cached flags instead of rechecking the masks.

Quoted 6 lines
Quoted 4 lines

If the masks were identical during can_attach, the flags remain false, causing cpuset_attach() to take the fast path and entirely skip cpuset_attach_task(). Could this leave the migrating task with its old, incorrect masks, allowing it to escape its designated isolation boundary?

Yes, it can be a valid concern. One workaround is to check the attach_in_progress flag in the cpuset. If it is set, we just have to clear the corresponding global flag when either the memory nodes or CPUs are changed.

Sorry, I mean to set the flag to indicate there are changes instead of clearing it.

-Longman

author maintainer
#6 lore raw

PATCH v3 3/5 cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders

In reply to Waiman Long

There are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.

In the former case, t is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.

IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach().

---
 kernel/cgroup/cpuset.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
kernel/cgroup/cpuset.c +18 -0
index 4457c4f11fce..b233a71f9b7c 100644
@@ -2967,6 +2967,20 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 /*
  * cpuset_can_attach() and cpuset_attach() specific internal data
  * Protected by cpuset_mutex
+ *
+ * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() tp get the
+ * old_mems_allowed value. There are two ways that many-to-one cpuset migration
+ * can happen:
+ * 1) A multithread application with threads in different cpusets is wholely
+ *    moved to a new cpuset.
+ * 2) Disabling v2 cpuset controller will move all the tasks in child cpusets
+ *    to the parent cpuset.
+ *
+ * In the former case, it is the mm setting of the group leader that really
+ * matters. So cpuset_attach_old_cs should track the oldcs of the thread
+ * leader. In the latter case, effective_mems of child cpusets must always
+ * be a subset of the parent. So no real page migration will be necessary no
+ * matter which child cpuset is selected as cpuset_attach_old_cs.
  */
 static struct cpuset *cpuset_attach_old_cs;
 static bool attach_cpus_updated;
@@ -3069,6 +3083,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 		if (ret)
 			goto out_unlock;
 
+		/* Update cpuset_attach_old_cs to the latest group leader */
+		if (task == task->group_leader)
+			cpuset_attach_old_cs = task_cs(task);
+
 		if (setsched_check) {
 			ret = security_task_setscheduler(task);
 			if (ret)
Signature · 1 line
2.54.0
#7 lore raw

Re: [PATCH-next v3 3/5] cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders

Quoted 26 lines · Waiman Long

There are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.

In the former case, t is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.

IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach().

---
 kernel/cgroup/cpuset.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
kernel/cgroup/cpuset.c +0 -0
index 4457c4f11fce..b233a71f9b7c 100644

kernel/cgroup/cpuset.c static int update_prstate(struct cpuset *cs, int new_prs) · line 2973

Quoted 7 lines
kernel/cgroup/cpuset.c +4 -0
@@ -2967,6 +2967,20 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 /*
  * cpuset_can_attach() and cpuset_attach() specific internal data
  * Protected by cpuset_mutex
+ *
+ * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() tp get the
+ * old_mems_allowed value. There are two ways that many-to-one cpuset migration
+ * can happen:

Hi Waiman,

I applied this series locally and ran some of my test cases. I didn't observe any issue so far.

While doing a static/checkpatch pass, I noticed a few minor issues in patches 3, 4 and 5. They are all non-functional nits.

For this patch, I only noticed a couple of small wording/typo nits in the new comment:

s/tp get/to get/

Best,
Guopeng

Quoted 24 lines
kernel/cgroup/cpuset.c +14 -0
+ * 1) A multithread application with threads in different cpusets is wholely
+ *    moved to a new cpuset.
+ * 2) Disabling v2 cpuset controller will move all the tasks in child cpusets
+ *    to the parent cpuset.
+ *
+ * In the former case, it is the mm setting of the group leader that really
+ * matters. So cpuset_attach_old_cs should track the oldcs of the thread
+ * leader. In the latter case, effective_mems of child cpusets must always
+ * be a subset of the parent. So no real page migration will be necessary no
+ * matter which child cpuset is selected as cpuset_attach_old_cs.
  */
 static struct cpuset *cpuset_attach_old_cs;
 static bool attach_cpus_updated;
@@ -3069,6 +3083,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 		if (ret)
 			goto out_unlock;
 
+		/* Update cpuset_attach_old_cs to the latest group leader */
+		if (task == task->group_leader)
+			cpuset_attach_old_cs = task_cs(task);
+
 		if (setsched_check) {
 			ret = security_task_setscheduler(task);
 			if (ret)
author maintainer
#8 lore raw

Re: [PATCH-next v3 3/5] cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders

Quoted 40 lines · Guopeng Zhang
Quoted 26 lines · Waiman Long

There are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.

In the former case, t is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.

IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach().

---
  kernel/cgroup/cpuset.c | 18 ++++++++++++++++++
  1 file changed, 18 insertions(+)
kernel/cgroup/cpuset.c +0 -0
index 4457c4f11fce..b233a71f9b7c 100644

kernel/cgroup/cpuset.c static int update_prstate(struct cpuset *cs, int new_prs) · line 2973

Quoted 7 lines
kernel/cgroup/cpuset.c +4 -0
@@ -2967,6 +2967,20 @@ static int update_prstate(struct cpuset *cs, int new_prs)
  /*
   * cpuset_can_attach() and cpuset_attach() specific internal data
   * Protected by cpuset_mutex
+ *
+ * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() tp get the
+ * old_mems_allowed value. There are two ways that many-to-one cpuset migration
+ * can happen:

Hi Waiman,

I applied this series locally and ran some of my test cases. I didn't observe any issue so far.

Quoted 7 lines

While doing a static/checkpatch pass, I noticed a few minor issues in patches 3, 4 and 5. They are all non-functional nits.

For this patch, I only noticed a couple of small wording/typo nits in the new comment:

s/tp get/to get/

Thanks for the review, will fix the typo in the next version.

Cheers,
Longman

author maintainer
#9 lore raw

PATCH v3 4/5 cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

In reply to Waiman Long

The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().

When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, cpuset_attach_old_cs, attach_cpus_updated and attach_mems_updated will also need to be updated in cpuset_fork().

Besides, the original code use cpuset_attach_nodemask_to for both nodemask returned by guarantee_online_mems() used only by cpuset_change_task_nodemask() and cs->effective_mems in all other cases. Such dual use is now impractical by merging the two task iteration loops into one. So keep cpuset_attach_nodemask_to for the nodemask returned by guarantee_online_mems() and reference cs->effective_mems directly in all the other cases.

---
 kernel/cgroup/cpuset.c | 90 ++++++++++++++++++++++--------------------
 1 file changed, 47 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +47 -43
index b233a71f9b7c..7100575927f6 100644
@@ -3149,9 +3149,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
  */
 static cpumask_var_t cpus_attach;
 static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
 
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
+	struct mm_struct *mm;
+
 	lockdep_assert_cpuset_lock_held();
 
 	if (cs != &top_cpuset)
@@ -3165,24 +3168,56 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	 */
 	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
+	if (cpuset_v2() && !attach_mems_updated)
+		return;
+
 	cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
 	cpuset1_update_task_spread_flags(cs, task);
+
+	if (task != task->group_leader)
+		return;
+
+	/*
+	 * Change mm for threadgroup leader. This is expensive and may
+	 * sleep and should be moved outside migration path proper.
+	 */
+	mm = get_task_mm(task);
+	if (mm) {
+		struct cpuset *oldcs = cpuset_attach_old_cs;
+
+		mpol_rebind_mm(mm, &cs->effective_mems);
+
+		/*
+		 * old_mems_allowed is the same with mems_allowed
+		 * here, except if this task is being moved
+		 * automatically due to hotplug.  In that case
+		 * @mems_allowed has been updated and is empty, so
+		 * @old_mems_allowed is the right nodesets that we
+		 * migrate mm from.
+		 */
+		if (is_memory_migrate(cs)) {
+			cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+					  &cs->effective_mems);
+			queue_task_work = true;
+		} else {
+			mmput(mm);
+		}
+	}
 }
 
 static void cpuset_attach(struct cgroup_taskset *tset)
 {
 	struct task_struct *task;
-	struct task_struct *leader;
 	struct cgroup_subsys_state *css;
 	struct cpuset *cs;
 	struct cpuset *oldcs = cpuset_attach_old_cs;
-	bool queue_task_work = false;
 
 	cgroup_taskset_first(tset, &css);
 	cs = css_cs(css);
 
 	lockdep_assert_cpus_held();	/* see cgroup_attach_lock() */
 	mutex_lock(&cpuset_mutex);
+	queue_task_work = false;
 
 	/*
 	 * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3190,53 +3225,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	 * in effective cpus and mems. In that case, we can optimize out
 	 * by skipping the task iteration and update.
 	 */
-	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
-		cpuset_attach_nodemask_to = cs->effective_mems;
+	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
 		goto out;
-	}
 
 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
 
 	cgroup_taskset_for_each(task, css, tset)
 		cpuset_attach_task(cs, task);
 
-	/*
-	 * Change mm for all threadgroup leaders. This is expensive and may
-	 * sleep and should be moved outside migration path proper. Skip it
-	 * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
-	 * not set.
-	 */
-	cpuset_attach_nodemask_to = cs->effective_mems;
-	if (!is_memory_migrate(cs) && !attach_mems_updated)
-		goto out;
-
-	cgroup_taskset_for_each_leader(leader, css, tset) {
-		struct mm_struct *mm = get_task_mm(leader);
-
-		if (mm) {
-			mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
-
-			/*
-			 * old_mems_allowed is the same with mems_allowed
-			 * here, except if this task is being moved
-			 * automatically due to hotplug.  In that case
-			 * @mems_allowed has been updated and is empty, so
-			 * @old_mems_allowed is the right nodesets that we
-			 * migrate mm from.
-			 */
-			if (is_memory_migrate(cs)) {
-				cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
-						  &cpuset_attach_nodemask_to);
-				queue_task_work = true;
-			} else
-				mmput(mm);
-		}
-	}
-
 out:
 	if (queue_task_work)
 		schedule_flush_migrate_mm();
-	cs->old_mems_allowed = cpuset_attach_nodemask_to;
+	cs->old_mems_allowed = cs->effective_mems;
 
 	if (cs->nr_migrate_dl_tasks) {
 		cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
@@ -3666,15 +3666,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)
  */
 static void cpuset_fork(struct task_struct *task)
 {
-	struct cpuset *cs;
-	bool same_cs;
+	struct cpuset *cs, *oldcs;
 
 	rcu_read_lock();
 	cs = task_cs(task);
-	same_cs = (cs == task_cs(current));
+	oldcs = task_cs(current);
 	rcu_read_unlock();
 
-	if (same_cs) {
+	if (cs == oldcs) {
 		if (cs == &top_cpuset)
 			return;
 
@@ -3686,7 +3685,12 @@ static void cpuset_fork(struct task_struct *task)
 	/* CLONE_INTO_CGROUP */
 	mutex_lock(&cpuset_mutex);
 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+	/* Assume CPUs and memory nodes are updated */
+	attach_cpus_updated = attach_mems_updated = true;
+	cpuset_attach_old_cs = oldcs;
+	oldcs->old_mems_allowed = oldcs->effective_mems;
 	cpuset_attach_task(cs, task);
+	attach_cpus_updated = attach_mems_updated = false;
 
 	dec_attach_in_progress_locked(cs);
 	mutex_unlock(&cpuset_mutex);
Signature · 1 line
2.54.0
#10 lore raw

Re: [PATCH-next v3 4/5] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

Quoted 37 lines · Waiman Long

The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().

When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, cpuset_attach_old_cs, attach_cpus_updated and attach_mems_updated will also need to be updated in cpuset_fork().

Besides, the original code use cpuset_attach_nodemask_to for both nodemask returned by guarantee_online_mems() used only by cpuset_change_task_nodemask() and cs->effective_mems in all other cases. Such dual use is now impractical by merging the two task iteration loops into one. So keep cpuset_attach_nodemask_to for the nodemask returned by guarantee_online_mems() and reference cs->effective_mems directly in all the other cases.

---
 kernel/cgroup/cpuset.c | 90 ++++++++++++++++++++++--------------------
 1 file changed, 47 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +1 -0
index b233a71f9b7c..7100575927f6 100644
@@ -3149,9 +3149,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
  */
 static cpumask_var_t cpus_attach;
 static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;

...

kernel/cgroup/cpuset.c static void cpuset_fork(struct task_struct *task) · line 3696

Quoted 13 lines
kernel/cgroup/cpuset.c +5 -0
@@ -3686,7 +3685,12 @@ static void cpuset_fork(struct task_struct *task)
 	/* CLONE_INTO_CGROUP */
 	mutex_lock(&cpuset_mutex);
 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+	/* Assume CPUs and memory nodes are updated */
+	attach_cpus_updated = attach_mems_updated = true;
+	cpuset_attach_old_cs = oldcs;
+	oldcs->old_mems_allowed = oldcs->effective_mems;
 	cpuset_attach_task(cs, task);
+	attach_cpus_updated = attach_mems_updated = false;
 
 	dec_attach_in_progress_locked(cs);
 	mutex_unlock(&cpuset_mutex);

Just a minor nit while running checkpatch --strict on this patch:

checkpatch reports:

CHECK: multiple assignments should be avoided

Perhaps the multiple assignments can be split to keep the patch checkpatch-clean?

attach_cpus_updated = true;
attach_mems_updated = true;

and later:

attach_cpus_updated = false;
attach_mems_updated = false;

Just a style nit.

Best,
Guopeng

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#12 lore raw

Re: [PATCH-next v3 4/5] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

Quoted 50 lines · On 5/27/26 1:54 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] cpuset_fork() misses the required task work flush for memory migration and leaks the static queue_task_work flag.
- [Low] cpuset_fork() improperly mutates the parent cpuset's internal state (old_mems_allowed) to satisfy the migration API. --

commit f676d612fb8888338177a57d117e451242f1ee60
Author: Waiman Long <[email protected]>

cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

This commit moves the mpol_rebind_mm() and cpuset_migrate_mm() calls for the group leader into cpuset_attach_task(). This makes the CLONE_INTO_CGROUP flag of clone(2) behave more like moving a task from one cpuset into another when used without CLONE_THREAD.

Quoted 4 lines
kernel/cgroup/cpuset.c +0 -0
index b233a71f9b7c2..7100575927f6a 100644

[ ... ]

Quoted 8 lines
kernel/cgroup/cpuset.c +1 -0
@@ -3149,9 +3149,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
   */
  static cpumask_var_t cpus_attach;
  static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
 
  static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
  {

[ ... ]

Quoted 9 lines
kernel/cgroup/cpuset.c +8 -0
+		if (is_memory_migrate(cs)) {
+			cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+					  &cs->effective_mems);
+			queue_task_work = true;
+		} else {
+			mmput(mm);
+		}
+	}
  }

[ ... ]

kernel/cgroup/cpuset.c static void cpuset_fork(struct task_struct *task) · line 3691

Quoted 8 lines
kernel/cgroup/cpuset.c +4 -0
@@ -3686,7 +3685,12 @@ static void cpuset_fork(struct task_struct *task)
  	/* CLONE_INTO_CGROUP */
  	mutex_lock(&cpuset_mutex);
  	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+	/* Assume CPUs and memory nodes are updated */
+	attach_cpus_updated = attach_mems_updated = true;
+	cpuset_attach_old_cs = oldcs;
+	oldcs->old_mems_allowed = oldcs->effective_mems;

[Severity: Low]
Does this code improperly mutate the parent cpuset's internal state?

Quoted 4 lines

By setting oldcs->old_mems_allowed to oldcs->effective_mems, we are modifying a shared cpuset's configuration state simply to accommodate the migration API parameters for the child's fork operation. Could this pollute the cpuset abstraction by manipulating the parent's state instead of keeping it isolated?

On second thought, we don't really need to touch old_mems_allowed which should be change in the last node mask modification operation. So I will remove it from the next version.

Quoted 11 lines

kernel/cgroup/cpuset.c static void cpuset_fork(struct task_struct *task) · line 3692

Quoted 1 line
kernel/cgroup/cpuset.c +0 -0
  	cpuset_attach_task(cs, task);

[Severity: Medium]
Is it possible that we miss the task work flush for memory migration here?

Since cpuset_attach_task() can set the global static flag queue_task_work to true, does cpuset_fork() need to check queue_task_work and call schedule_flush_migrate_mm() before returning?

If this is missed, it seems the static flag could be leaked as true, and memory migration might execute asynchronously in the background instead of completing synchronously before returning to userspace.

Yes, cpuset_fork() should check should check queue_task_work and call schedule_flush_migrate_mm() before returning if set.

Cheers,
Longman

author maintainer
#13 lore raw

PATCH v3 5/5 cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

In reply to Waiman Long

With cgroup v2, the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple destination or source cpusets when the cpuset controller is enabled or disabled respectively.

Since cpuset is threaded in both v1 and v2, another possible way to cause many-to-one migration is to move the whole process with multiple threads in different cpuset enabled threaded cgroups into another cpuset enabled cgroup.

The current cpuset_can_attach() and cpuset_attach() functions still expect task migration is from one source cpuset to one destination cpuset. This has been the case since cpuset was enabled for cgroup v2 in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy").

This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.

It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.

Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.

To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both the source and destination cpusets are decremented/incremented with their values added to nr_deadline_tasks when the migration is successful.

Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy")
---
 kernel/cgroup/cpuset-internal.h |   6 +
 kernel/cgroup/cpuset.c          | 206 +++++++++++++++++++++++---------
 2 files changed, 157 insertions(+), 55 deletions(-)
kernel/cgroup/cpuset-internal.h +6 -0
index f7aaf01f7cd5..4c2772a7fd5e 100644
@@ -161,6 +161,12 @@ struct cpuset {
 	 */
 	bool remote_partition;
 
+	/*
+	 * cpuset_can_attach() and cpuset_attach() specific data
+	 */
+	bool			attach_node_in_llist;
+	struct llist_node	attach_node;
+
 	/*
 	 * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
 	 * know when to rebuild associated root domain bandwidth information.
kernel/cgroup/cpuset.c +151 -55

305 lines not on this page · raw message

Signature · 1 line
2.54.0
#14 lore raw
Quoted 4 lines · Waiman Long

With cgroup v2, the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple destination or source cpusets when the cpuset controller is enabled or disabled respectively.

...

Quoted 28 lines
kernel/cgroup/cpuset.c +20 -1
-/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */
+/*
+ * Called by cgroups to determine if a cpuset is usable; cpuset_mutex held.
+ *
+ * With cgroup v2, enabling of cpuset controller in a cgroup subtree can
+ * cause @tset to contain task migration data from one parent cpuset to multiple
+ * child cpusets. Not much is needed to be done here other than tracking the
+ * number of DL tasks in each cpuset as the CPUs and memory nodes of the child
+ * cpusets are exactly the same as the parent.
+ *
+ * Conversely, disabling of cpuset controller can cause @tset to contain task
+ * migration data from multiple child cpusets to one parent cpuset. Here, the
+ * CPUs and memory nodes of the child cpusets may be different from the parent,
+ * but must be a subset of its parent.
+ *
+ * Another possible many-to-one migration is the moving of the whole
+ * multithreaded process with threads in different cpusets to another cpuset.
+ *
+ * For all other use cases, @tset task migration data should be from one source
+ * cpuset to one destination cpuset.
+ */
 static int cpuset_can_attach(struct cgroup_taskset *tset)
 {
 	struct cgroup_subsys_state *css;
@@ -3079,6 +3172,16 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 		goto out_unlock;
 
 	cgroup_taskset_for_each(task, css, tset) {

kernel/cgroup/cpuset.c static int cpuset_can_attach(struct cgroup_taskset *tset) · line 3184

Quoted 10 lines
kernel/cgroup/cpuset.c +10 -0
+		struct cpuset *newcs = css_cs(css);
+		struct cpuset *new_oldcs = task_cs(task);
+
+		if ((newcs != cs) || (new_oldcs != oldcs)) {
+			cs = newcs;
+			oldcs = new_oldcs;
+			ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
+			if (ret)
+				goto out_unlock;
+		}

Just a minor nit while running checkpatch --strict on this patch:

checkpatch reports unnecessary parentheses here:

if ((newcs != cs) || (new_oldcs != oldcs)) {

Perhaps this can be simplified to:

if (newcs != cs || new_oldcs != oldcs) {

Quoted 2 lines
kernel/cgroup/cpuset.c +0 -0
 		ret = task_can_attach(task);
 		if (ret)

...

kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3319

Quoted 10 lines
kernel/cgroup/cpuset.c +4 -3
 	/*
 	 * In the default hierarchy, enabling cpuset in the child cgroups
-	 * will trigger a number of cpuset_attach() calls with no change
-	 * in effective cpus and mems. In that case, we can optimize out
-	 * by skipping the task iteration and update.
+	 * will trigger a cpuset_attach() call with no change in effective cpus
+	 * and mems. In that case, we can optimize out by skipping the task
+	 * iteration and update, but the destination cpuset list is iterated to
+	 * set old_mems_sllowed.
 	 */

I also noticed one small typo in the added comment:

s/old_mems_sllowed/old_mems_allowed/

Best,
Guopeng

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#16 lore raw
Quoted 60 lines · On 5/27/26 2:30 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Incorrect memory migration due to global cpuset_attach_old_cs tracking only a single source cpuset for multi-source tasksets. --

commit e9778f4a1eb98851e3298f795f8a88874b0e43c1
Author: Waiman Long <[email protected]>

cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

This commit updates the cpuset attach functions to handle multiple source and destination cpusets during task migration, maintaining singly linked lists to track them. It also updates deadline task accounting to reflect the possibility of many-to-one migrations.

Quoted 4 lines
kernel/cgroup/cpuset.c +0 -0
index 7100575927f6a..98ee001ef950a 100644

[ ... ]

kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3308

Quoted 8 lines
kernel/cgroup/cpuset.c +0 -1
@@ -3210,42 +3303,45 @@ static void cpuset_attach(struct cgroup_taskset *tset)
  	struct task_struct *task;
  	struct cgroup_subsys_state *css;
  	struct cpuset *cs;
-	struct cpuset *oldcs = cpuset_attach_old_cs;
 
  	cgroup_taskset_first(tset, &css);
  	cs = css_cs(css);

[Severity: High]
This is a pre-existing issue, but does this multi-source migration logic miss the memory migration path?

During cpuset_can_attach(), the global variable cpuset_attach_old_cs is overwritten with the task_cs(task) of each group leader:

cpuset_can_attach() {
     ...
     cgroup_taskset_for_each(task, css, tset) {
         ...
         /* Update cpuset_attach_old_cs to the latest group leader */
         if (task == task->group_leader)
             cpuset_attach_old_cs = task_cs(task);

For a taskset containing tasks from multiple different source cpusets, cpuset_attach_old_cs ends up pointing only to the old cpuset of the last group leader.

Later, cpuset_attach_task() relies on this global variable to determine the source memory node mask for cpuset_migrate_mm():

cpuset_attach_task() {
     ...
     if (mm) {
         struct cpuset *oldcs = cpuset_attach_old_cs;
         ...
         if (is_memory_migrate(cs)) {
             cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
                               &cs->effective_mems);
Quoted 6 lines

Will this result in all tasks in a heterogeneous migration batch using the old_mems_allowed of the last processed task's old cpuset?

If so, could this use the wrong source mask for tasks originating from other cpusets, potentially leaving pages stranded on incorrect NUMA nodes and violating cpuset memory isolation boundaries?

The reason why that is not a problem is documented in patch 3. So this shouldn't be a problem.

Cheers,
Longman

Quoted 1 line