Skip to contentSkip to conversation

cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

PATCH Superseded Waiman Long 3 participants 18 messages Jun 21 – Jun 25

Subsystems (3)

v7 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Waiman Long

author maintainer
#0 lore raw

v7 cover cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()

v7:
- Include the fix patch from Farhad Alemi to fix a div/0 crash that was part of the old patch 1.
- Integrated v6 patch 7 into earlier patches.
- Add a new "cgroup/cpuset: Prevent race between task attach and cpuset state change" patch to ensure that there will be no cpuset state change between cpuset_can_attach() and cpuset_attach() or cpuset_cancel_attach().
- Break v6 patch 6 into 2 separate patches for supporting multiple source cpusets and multiple destination cpusets respectively and further simplify and streamline the code.

v6:
- Make guarantee_online_mems() to only return cs->effective_mems with v2 in patch 1.
- Remove obsolete commit description text from patch 3.
- Add Reviewed-by tags.
- In patch 6, add WARN_ON_ONCE() test in cpuset_can_attach() to

    confirm that cs != oldcs.

v5:
- Remove the WARN_ON() call as it can be triggered in a corner case.
- Instead of passing an attach_cpus_updated and attach_mems_updated flags from cpuset_can_attach() to cpuset_attach(), re-evaluate the flags at the beginning of cpuset_attach() based on data in the source & destination cpusets in the singly linked lists to eliminate the Time-of-Check to Time-of-Use (TOCTOU) race condition & simplify the code changes.
- Add back the dropped optimization in patch 5.

Sashiko AI review of another cpuset patch had found that cpuset_attach() and cpuset_can_attach() can be passed a cgroup_taskset with tasks migrating from one source cpuset to multiple destination cpusets and vice versa. Further testing of the cpuset code indicates that this is indeed the case when the v2 cpuset controller is enabled or disabled.

Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that there will be one source and one destinaton cpuset which may result in inocrrect behavior.

This patch series is created to fix this issue.

Patch 1 is a fix that fix a cgroup v2 div/0 crash due to bug in cpuset_update_tasks_nodemask().

Patch 2 is to fix an inconsistency in the way node mask update is being handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that they match each other.

Patch 3 makes any cpuset state change to wait for the completion of the pending cpuset attach operation, if any.

Patches 4 and 5 are just preparatory patches to make the remaining patches easier to review.

Patch 6 makes cpuset_attach_old_cs to track group leader for use by cpuset_migrate_mm().

Patch 7 moves mpol_rebind_mm() and cpuset_migrate_mm() inside cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works more like moving task from one cpuset to another one, while also make supporting multiple source and destination cpusets easier.

Patch 8 makes the necessary changes to enable the support of multiple source cpusets by keeping all the source cpusets found during task iterations in a singly linked lists.

Patch 9 enables the support of multiple destination cpusets during the the enabling of cpuset v2 controller.

Farhad Alemi (1):
cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed

Waiman Long (8):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Prevent race between task attach and cpuset state
change
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper cgroup/cpuset: Expand the scope of cpuset_can_attach_check() cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source cpusets for cpuset_*attach() cgroup/cpuset: Support multiple destination cpusets for
cpuset_*attach()

 kernel/cgroup/cpuset-internal.h |   2 +
 kernel/cgroup/cpuset.c          | 400 ++++++++++++++++++++++----------
 2 files changed, 277 insertions(+), 125 deletions(-)
Signature · 1 line
2.54.0
author maintainer
#1 lore raw

PATCH v7 1/9 cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed

From: Farhad Alemi <[email protected]>

Creating a child cpuset where cpuset.mems is never set leads to a div/0 when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a CPU hotplug event.

Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the call to __nodes_fold()

The cpuset code passes (cs->mems_allowed) which is not guaranteed to have nodes to the rebind routine. Use cs->effective_mems instead, which is guaranteed to have a non-empty nodemask.

---
 kernel/cgroup/cpuset.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/cgroup/cpuset.c +1 -1
index 591e3aa487fc..b21c31650583 100644
@@ -2653,7 +2653,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
 
 		migrate = is_memory_migrate(cs);
 
-		mpol_rebind_mm(mm, &cs->mems_allowed);
+		mpol_rebind_mm(mm, &cs->effective_mems);
 		if (migrate)
 			cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
 		else
Signature · 1 line
2.54.0
reviewer Reviewed-by on v7 1/9
#2 lore raw

Re: [PATCH v7 1/9] cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed

Quoted 37 lines · Waiman Long

From: Farhad Alemi <[email protected]>

Creating a child cpuset where cpuset.mems is never set leads to a div/0 when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a CPU hotplug event.

Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the call to __nodes_fold()

The cpuset code passes (cs->mems_allowed) which is not guaranteed to have nodes to the rebind routine. Use cs->effective_mems instead, which is guaranteed to have a non-empty nodemask.

---
  kernel/cgroup/cpuset.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/cgroup/cpuset.c +0 -0
index 591e3aa487fc..b21c31650583 100644

kernel/cgroup/cpuset.c void cpuset_update_tasks_nodemask(struct cpuset *cs) · line 2659

Quoted 8 lines
kernel/cgroup/cpuset.c +1 -1
@@ -2653,7 +2653,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
 
  		migrate = is_memory_migrate(cs);
 
-		mpol_rebind_mm(mm, &cs->mems_allowed);
+		mpol_rebind_mm(mm, &cs->effective_mems);
  		if (migrate)
  			cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
  		else
Signature · 2 lines
Best regards
Ridong
Reviewed-by on v7 1/9
#3 lore raw

Re: [PATCH v7 1/9] cgroup/cpuset: rebind mm mempolicy to effective_mems, not mems_allowed

In reply to Waiman Long
Quoted 20 lines · Waiman Long

From: Farhad Alemi <[email protected]>

Creating a child cpuset where cpuset.mems is never set leads to a div/0 when a VMA mempolicy with MPOL_F_RELATIVE_NODES rebinds in response to a CPU hotplug event.

Reproduction steps:
1) Create a cgroup w/ cpuset controls (do not set cpuset.mems)
2) Move the task into the child cpuset
3) Create a VMA mempolicy for that task with MPOL_F_RELATIVE_NODES
4) unplug and hotplug a cpu
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu1/online
5) mempolicy rebind does a div/0 in mpol_relative_nodemask on the call to __nodes_fold()

The cpuset code passes (cs->mems_allowed) which is not guaranteed to have nodes to the rebind routine. Use cs->effective_mems instead, which is guaranteed to have a non-empty nodemask.

Quoted 8 lines
Quoted 20 lines
---
 kernel/cgroup/cpuset.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/cgroup/cpuset.c +1 -1
index 591e3aa487fc..b21c31650583 100644
@@ -2653,7 +2653,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
 
 		migrate = is_memory_migrate(cs);
 
-		mpol_rebind_mm(mm, &cs->mems_allowed);
+		mpol_rebind_mm(mm, &cs->effective_mems);
 		if (migrate)
 			cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
 		else
Signature · 1 line
2.54.0
author maintainer
#4 lore raw

PATCH v7 2/9 cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()

In reply to Waiman Long

Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed

These memory actions are done in cpuset_update_tasks_nodemask() and cpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.

In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()

In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems

These inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.

The guarantee_online_mems() function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].

The guarantee_online_mems() function should mostly be useful for v1 where mems_allowed is the same as effective_mems. With v2, the memory nodes in effective_mems should be a subset of node_states[N_MEMORY] except when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected in the effective_mems's as cpuset_handle_hotplug() has not been called from cpuset_track_online_nodes().

Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()

So for v2, it is effectively all effective_mems most of the time. For v1, mpol_rebind_mm() uses mems_allowed which may differ from what guarantee_online_mems() returns, but it conforms to what the cpuset v1 documentation says with respect to setting memory policy.

---
 kernel/cgroup/cpuset.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)
kernel/cgroup/cpuset.c +18 -12
index b21c31650583..a1c8890d3519 100644
@@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,
  * Return in *pmask the portion of a cpusets's mems_allowed that
  * are online, with memory.  If none are online with memory, walk
  * up the cpuset hierarchy until we find one that does have some
- * online mems.  The top cpuset always has some mems online.
+ * online mems.  The top cpuset always has some mems online. With v2,
+ * effective_mems should always contain online memory nodes except
+ * during the transition period where a memory node hotunplug operation
+ * is in progress.
  *
  * One way or another, we guarantee to return some non-empty subset
  * of node_states[N_MEMORY].
@@ -2619,6 +2622,14 @@ static void *cpuset_being_rebound;
  * Iterate through each task of @cs updating its mems_allowed to the
  * effective cpuset's.  As this function is called with cpuset_mutex held,
  * cpuset membership stays stable.
+ *
+ * - cpuset_change_task_nodemask(): guarantee_online_mems()
+ * - mpol_rebind_mm(): effective_mems
+ * - cpuset_migrate_mm(): guarantee_online_mems()
+ * - old_mems_allowed: guarantee_online_mems()
+ *
+ * For v2, guarantee_online_mems() should return a node mask that is the same
+ * as the effective_mems of current cpuset.
  */
 void cpuset_update_tasks_nodemask(struct cpuset *cs)
 {
@@ -2627,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
 	struct task_struct *task;
 
 	cpuset_being_rebound = cs;		/* causes mpol_dup() rebind */
-
 	guarantee_online_mems(cs, &newmems);
 
 	/*
@@ -3148,19 +3158,16 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	cpus_updated = !cpumask_equal(cs->effective_cpus,
 				      oldcs->effective_cpus);
 	mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
 
 	/*
 	 * In the default hierarchy, enabling cpuset in the child cgroups
-	 * will trigger a number of cpuset_attach() calls with no change
-	 * in effective cpus and mems. In that case, we can optimize out
-	 * by skipping the task iteration and update.
+	 * will trigger a cpuset_attach() call with no change in effective cpus
+	 * and mems. In that case, we can optimize out by skipping the task
+	 * iteration and update.
 	 */
-	if (cpuset_v2() && !cpus_updated && !mems_updated) {
-		cpuset_attach_nodemask_to = cs->effective_mems;
+	if (cpuset_v2() && !cpus_updated && !mems_updated)
 		goto out;
-	}
-
-	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
 
 	cgroup_taskset_for_each(task, css, tset)
 		cpuset_attach_task(cs, task);
@@ -3171,7 +3178,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	 * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
 	 * not set.
 	 */
-	cpuset_attach_nodemask_to = cs->effective_mems;
 	if (!is_memory_migrate(cs) && !mems_updated)
 		goto out;
 
@@ -3179,7 +3185,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 		struct mm_struct *mm = get_task_mm(leader);
 
 		if (mm) {
-			mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
+			mpol_rebind_mm(mm, &cs->effective_mems);
 
 			/*
 			 * old_mems_allowed is the same with mems_allowed
Signature · 1 line
2.54.0
Reviewed-by on v7 2/9
#5 lore raw

Re: [PATCH v7 2/9] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()

Quoted 52 lines · Waiman Long

Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed

These memory actions are done in cpuset_update_tasks_nodemask() and cpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.

In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()

In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems

These inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.

The guarantee_online_mems() function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].

The guarantee_online_mems() function should mostly be useful for v1 where mems_allowed is the same as effective_mems. With v2, the memory nodes in effective_mems should be a subset of node_states[N_MEMORY] except when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected in the effective_mems's as cpuset_handle_hotplug() has not been called from cpuset_track_online_nodes().

Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()

So for v2, it is effectively all effective_mems most of the time. For v1, mpol_rebind_mm() uses mems_allowed which may differ from what guarantee_online_mems() returns, but it conforms to what the cpuset v1 documentation says with respect to setting memory policy.

Quoted 6 lines
---
 kernel/cgroup/cpuset.c | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)
author maintainer
#6 lore raw

PATCH v7 3/9 cgroup/cpuset: Prevent race between task attach and cpuset state change

In reply to Waiman Long

Commit e44193d39e8d ("cpuset: let hotplug propagation work wait for task attaching") was introduced to let hotplug operation to wait until the completion of task attaching operation. However, it is still possible that the states of the source or destination cpuset can be changed between the cpuset_can_attach() call and the subsequent cpuset_attach()/cpuset_cacnel_attach() call.

As a result, data gathered during cpuset_can_attach() cannot be reliably used in the subsequent cpuset_attach()/cpuset_cacnel_attach() call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and cpuset_attach()/cpuset_cacnel_attach() by making cpuset_write_resmask() and cpuset_partition_write() wait for the completion of task attach and set the attach_in_progress flag in the source cpuset as well.

The comments about validate_change() are no longer valid as it won't be called at all if an attach operation is in progress. So the comments can be removed.

---
 kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
kernel/cgroup/cpuset.c +20 -8
index a1c8890d3519..65d095dcada1 100644
@@ -3080,11 +3080,8 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	cs->dl_bw_cpu = cpu;
 
 out_success:
-	/*
-	 * Mark attach is in progress.  This makes validate_change() fail
-	 * changes which zero cpus/mems_allowed.
-	 */
 	cs->attach_in_progress++;
+	oldcs->attach_in_progress++;
 
 out_unlock:
 	if (ret)
@@ -3235,10 +3232,19 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 		return -EACCES;
 
 	buf = strstrip(buf);
+retry:
+	wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
 	cpuset_full_lock();
 	if (!is_cpuset_online(cs))
 		goto out_unlock;
 
+	/* Don't race with task attach */
+	if (cs->attach_in_progress) {
+		cpuset_full_unlock();
+		goto retry;
+	}
+
 	trialcs = dup_or_alloc_cpuset(cs);
 	if (!trialcs) {
 		retval = -ENOMEM;
@@ -3366,7 +3372,17 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,
 	else
 		return -EINVAL;
 
+retry:
+	wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
 	cpuset_full_lock();
+
+	/* Don't race with task attach */
+	if (cs->attach_in_progress) {
+		cpuset_full_unlock();
+		goto retry;
+	}
+
 	if (is_cpuset_online(cs))
 		retval = update_prstate(cs, val);
 	cpuset_update_sd_hk_unlock();
@@ -3605,10 +3621,6 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
 	if (ret)
 		goto out_unlock;
 
-	/*
-	 * Mark attach is in progress.  This makes validate_change() fail
-	 * changes which zero cpus/mems_allowed.
-	 */
 	cs->attach_in_progress++;
 out_unlock:
 	mutex_unlock(&cpuset_mutex);
Signature · 1 line
2.54.0
reviewer
#7 lore raw

Re: [PATCH v7 3/9] cgroup/cpuset: Prevent race between task attach and cpuset state change

Quoted 29 lines · Waiman Long

Commit e44193d39e8d ("cpuset: let hotplug propagation work wait for task attaching") was introduced to let hotplug operation to wait until the completion of task attaching operation. However, it is still possible that the states of the source or destination cpuset can be changed between the cpuset_can_attach() call and the subsequent cpuset_attach()/cpuset_cacnel_attach() call.

As a result, data gathered during cpuset_can_attach() cannot be reliably used in the subsequent cpuset_attach()/cpuset_cacnel_attach() call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and cpuset_attach()/cpuset_cacnel_attach() by making cpuset_write_resmask() and cpuset_partition_write() wait for the completion of task attach and set the attach_in_progress flag in the source cpuset as well.

The comments about validate_change() are no longer valid as it won't be called at all if an attach operation is in progress. So the comments can be removed.

---
  kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
kernel/cgroup/cpuset.c +0 -0
index a1c8890d3519..65d095dcada1 100644

kernel/cgroup/cpuset.c static int cpuset_can_attach(struct cgroup_taskset *tset) · line 3085

Quoted 10 lines
kernel/cgroup/cpuset.c +1 -4
@@ -3080,11 +3080,8 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
  	cs->dl_bw_cpu = cpu;
 
  out_success:
-	/*
-	 * Mark attach is in progress.  This makes validate_change() fail
-	 * changes which zero cpus/mems_allowed.
-	 */
  	cs->attach_in_progress++;
+	oldcs->attach_in_progress++;
 

I only see oldcs->attach_in_progress being incremented here — the matching decrement seems to land in a later patch. That makes this one unbalanced on its own (the count would leak, and a later write to the source cpuset would block on the new wait_event()), so it's not bisect-safe.

Let's either keep the patch self-contained or fold it into the patch that adds the decrement.

Quoted 51 lines
kernel/cgroup/cpuset.c +19 -4
  out_unlock:
  	if (ret)
@@ -3235,10 +3232,19 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
  		return -EACCES;
 
  	buf = strstrip(buf);
+retry:
+	wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
  	cpuset_full_lock();
  	if (!is_cpuset_online(cs))
  		goto out_unlock;
 
+	/* Don't race with task attach */
+	if (cs->attach_in_progress) {
+		cpuset_full_unlock();
+		goto retry;
+	}
+
  	trialcs = dup_or_alloc_cpuset(cs);
  	if (!trialcs) {
  		retval = -ENOMEM;
@@ -3366,7 +3372,17 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,
  	else
  		return -EINVAL;
 
+retry:
+	wait_event(cpuset_attach_wq, cs->attach_in_progress == 0);
+
  	cpuset_full_lock();
+
+	/* Don't race with task attach */
+	if (cs->attach_in_progress) {
+		cpuset_full_unlock();
+		goto retry;
+	}
+
  	if (is_cpuset_online(cs))
  		retval = update_prstate(cs, val);
  	cpuset_update_sd_hk_unlock();
@@ -3605,10 +3621,6 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
  	if (ret)
  		goto out_unlock;
 
-	/*
-	 * Mark attach is in progress.  This makes validate_change() fail
-	 * changes which zero cpus/mems_allowed.
-	 */
  	cs->attach_in_progress++;
  out_unlock:
  	mutex_unlock(&cpuset_mutex);
Signature · 2 lines
Best regards
Ridong
author maintainer
#8 lore raw

Re: [PATCH v7 3/9] cgroup/cpuset: Prevent race between task attach and cpuset state change

Quoted 44 lines · Ridong Chen
Quoted 31 lines · Waiman Long

Commit e44193d39e8d ("cpuset: let hotplug propagation work wait for task attaching") was introduced to let hotplug operation to wait until the completion of task attaching operation. However, it is still possible that the states of the source or destination cpuset can be changed between the cpuset_can_attach() call and the subsequent cpuset_attach()/cpuset_cacnel_attach() call.

As a result, data gathered during cpuset_can_attach() cannot be reliably used in the subsequent cpuset_attach()/cpuset_cacnel_attach() call at all. Make the task attach operation more robust
and allow the sharing of data between cpuset_can_attach() and cpuset_attach()/cpuset_cacnel_attach() by making cpuset_write_resmask() and cpuset_partition_write() wait for the completion of task attach and set the attach_in_progress flag in the source cpuset as well.

The comments about validate_change() are no longer valid as it won't be called at all if an attach operation is in progress. So the comments can be removed.

---
  kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++--------

  1 file changed, 20 insertions(+), 8 deletions(-)
kernel/cgroup/cpuset.c +0 -0
index a1c8890d3519..65d095dcada1 100644
@@ -3080,11 +3080,8 @@ static int cpuset_can_attach(struct 
 cgroup_taskset *tset)
       cs->dl_bw_cpu = cpu;
Quoted 7 lines

    out_success:
-    /*
-     * Mark attach is in progress.  This makes validate_change() fail -     * changes which zero cpus/mems_allowed.
-     */
      cs->attach_in_progress++;
+    oldcs->attach_in_progress++;

I only see oldcs->attach_in_progress being incremented here — the matching decrement seems to land in a later patch. That makes this one unbalanced on its own (the count would leak, and a later write to the source cpuset would block on the new wait_event()), so it's not bisect-safe.

Quoted 1 line

Let's either keep the patch self-contained or fold it into the patch that adds the decrement.

Yes, it should have a matching decrement there. Anyway, I am going to revamp this patch to make attach_in_progress flag global instead of per cpuset. This is to make sure that a task attach operation will not race with any cpuset control file write operation.

Cheers,
Longman

author maintainer
#9 lore raw

PATCH v7 4/9 cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper

In reply to Waiman Long

Extract the DL bandwidth allocation code in cpuset_attach() to a new cpuset_reserve_dl_bw() helper to simplify code.

No functional change is expected.

---
 kernel/cgroup/cpuset.c | 47 ++++++++++++++++++++++++------------------
 1 file changed, 27 insertions(+), 20 deletions(-)
kernel/cgroup/cpuset.c +27 -20
index 65d095dcada1..2ffc66baedf3 100644
@@ -2994,6 +2994,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)
 	return 0;
 }
 
+static int cpuset_reserve_dl_bw(struct cpuset *cs)
+{
+	int cpu, ret;
+
+	if (!cs->sum_migrate_dl_bw)
+		return 0;
+
+	cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
+	if (unlikely(cpu >= nr_cpu_ids))
+		return -EINVAL;
+
+	ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
+	if (ret)
+		return ret;
+
+	cs->dl_bw_cpu = cpu;
+	return 0;
+}
+
 static void reset_migrate_dl_data(struct cpuset *cs)
 {
 	cs->nr_migrate_dl_tasks = 0;
@@ -3008,7 +3027,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	struct cpuset *cs, *oldcs;
 	struct task_struct *task;
 	bool setsched_check;
-	int cpu, ret;
+	int ret;
 
 	/* used later by cpuset_attach() */
 	cpuset_attach_old_cs = task_cs(cgroup_taskset_first(tset, &css));
@@ -3064,28 +3083,16 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 		}
 	}
 
-	if (!cs->sum_migrate_dl_bw)
-		goto out_success;
-
-	cpu = cpumask_any_and(cpu_active_mask, cs->effective_cpus);
-	if (unlikely(cpu >= nr_cpu_ids)) {
-		ret = -EINVAL;
-		goto out_unlock;
-	}
-
-	ret = dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
-	if (ret)
-		goto out_unlock;
-
-	cs->dl_bw_cpu = cpu;
-
-out_success:
-	cs->attach_in_progress++;
-	oldcs->attach_in_progress++;
+	ret = cpuset_reserve_dl_bw(cs);
 
 out_unlock:
-	if (ret)
+	if (ret) {
 		reset_migrate_dl_data(cs);
+	} else {
+		cs->attach_in_progress++;
+		oldcs->attach_in_progress++;
+	}
+
 	mutex_unlock(&cpuset_mutex);
 	return ret;
 }
Signature · 1 line
2.54.0
author maintainer
#11 lore raw

PATCH v7 5/9 cgroup/cpuset: Expand the scope of cpuset_can_attach_check()

In reply to Waiman Long

Expand the scope of cpuset_can_attach_check() by including the setting of setsched flag inside cpuset_can_attach_check() with the new @oldcs and @psetsched argument. As cpuset_can_attach_check() is also called from cpuset_can_fork(), set the new arguments to NULL from that caller.

---
 kernel/cgroup/cpuset.c | 52 ++++++++++++++++++++++++------------------
 1 file changed, 30 insertions(+), 22 deletions(-)
kernel/cgroup/cpuset.c +30 -22
index 2ffc66baedf3..b7b5072f2fdd 100644
@@ -2985,12 +2985,39 @@ static struct cpuset *cpuset_attach_old_cs;
  * For v1, cpus_allowed and mems_allowed can't be empty.
  * For v2, effective_cpus can't be empty.
  * Note that in v1, effective_cpus = cpus_allowed.
+ *
+ * Also set the boolean flag passed in by @psetsched depending on if
+ * security_task_setscheduler() call is needed and @oldcs is not NULL.
  */
-static int cpuset_can_attach_check(struct cpuset *cs)
+static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,
+				   bool *psetsched)
 {
 	if (cpumask_empty(cs->effective_cpus) ||
 	   (!is_in_v2_mode() && nodes_empty(cs->mems_allowed)))
 		return -ENOSPC;
+
+	if (!oldcs)
+		return 0;
+
+	/*
+	 * Skip rights over task setsched check in v2 when nothing changes,
+	 * migration permission derives from hierarchy ownership in
+	 * cgroup_procs_write_permission()).
+	 */
+	*psetsched = !cpuset_v2() ||
+		!cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
+		!nodes_equal(cs->effective_mems, oldcs->effective_mems);
+
+	/*
+	 * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
+	 * brings the last online CPU offline as users are not allowed to empty
+	 * cpuset.cpus when there are active tasks inside. When that happens,
+	 * we should allow tasks to migrate out without security check to make
+	 * sure they will be able to run after migration.
+	 */
+	if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
+		*psetsched = false;
+
 	return 0;
 }
 
@@ -3037,29 +3064,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	mutex_lock(&cpuset_mutex);
 
 	/* Check to see if task is allowed in the cpuset */
-	ret = cpuset_can_attach_check(cs);
+	ret = cpuset_can_attach_check(cs, oldcs, &setsched_check);
 	if (ret)
 		goto out_unlock;
 
-	/*
-	 * Skip rights over task setsched check in v2 when nothing changes,
-	 * migration permission derives from hierarchy ownership in
-	 * cgroup_procs_write_permission()).
-	 */
-	setsched_check = !cpuset_v2() ||
-		!cpumask_equal(cs->effective_cpus, oldcs->effective_cpus) ||
-		!nodes_equal(cs->effective_mems, oldcs->effective_mems);
-
-	/*
-	 * A v1 cpuset with tasks will have no CPU left only when CPU hotplug
-	 * brings the last online CPU offline as users are not allowed to empty
-	 * cpuset.cpus when there are active tasks inside. When that happens,
-	 * we should allow tasks to migrate out without security check to make
-	 * sure they will be able to run after migration.
-	 */
-	if (!is_in_v2_mode() && cpumask_empty(oldcs->effective_cpus))
-		setsched_check = false;
-
 	cgroup_taskset_for_each(task, css, tset) {
 		ret = task_can_attach(task);
 		if (ret)
@@ -3616,7 +3624,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)
 	mutex_lock(&cpuset_mutex);
 
 	/* Check to see if task is allowed in the cpuset */
-	ret = cpuset_can_attach_check(cs);
+	ret = cpuset_can_attach_check(cs, NULL, NULL);
 	if (ret)
 		goto out_unlock;
 
Signature · 1 line
2.54.0
author maintainer
#12 lore raw

PATCH v7 6/9 cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders

In reply to Waiman Long

There are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.

In the former case, it is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.

IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach(), but fall back to that of the first task if there is no group leader in the taskset.

---
 kernel/cgroup/cpuset.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
kernel/cgroup/cpuset.c +25 -0
index b7b5072f2fdd..0375dae26d0b 100644
@@ -2978,6 +2978,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 	return 0;
 }
 
+/*
+ * cpuset_can_attach() and cpuset_attach() specific internal data
+ * Protected by cpuset_mutex
+ */
 static struct cpuset *cpuset_attach_old_cs;
 
 /*
@@ -3068,11 +3072,32 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)
 	if (ret)
 		goto out_unlock;
 
+	/*
+	 * The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get
+	 * the old_mems_allowed value. There are two ways that many-to-one
+	 * cpuset migration can happen:
+	 * 1) A multithread application with threads in different cpusets is
+	 *    wholely migrated to a new cpuset.
+	 * 2) Disabling v2 cpuset controller will move all the tasks in child
+	 *    cpusets to the parent cpuset.
+	 *
+	 * In the former case, it is the mm setting of the group leader that
+	 * really matters. So cpuset_attach_old_cs should track the oldcs of the
+	 * group leader. It falls back to the oldcs of the first task if there
+	 * is no group leader in the taskset. In the latter case, effective_mems
+	 * of child cpusets must always be a subset of the parent. So no real
+	 * page migration will be necessary no matter which child cpuset is
+	 * selected as cpuset_attach_old_cs.
+	 */
 	cgroup_taskset_for_each(task, css, tset) {
 		ret = task_can_attach(task);
 		if (ret)
 			goto out_unlock;
 
+		/* Update cpuset_attach_old_cs to the latest group leader */
+		if (task == task->group_leader)
+			cpuset_attach_old_cs = task_cs(task);
+
 		if (setsched_check) {
 			ret = security_task_setscheduler(task);
 			if (ret)
Signature · 1 line
2.54.0
author maintainer
#13 lore raw

PATCH v7 7/9 cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

In reply to Waiman Long

The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().

When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work

---
 kernel/cgroup/cpuset.c | 105 ++++++++++++++++++++++++-----------------
 1 file changed, 62 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +62 -43
index 0375dae26d0b..511afb077e2d 100644
@@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 /*
  * cpuset_can_attach() and cpuset_attach() specific internal data
  * Protected by cpuset_mutex
+ *
+ * The attach_cpus_updated/attach_mems_updated flags are set in either
+ * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task().
  */
 static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
 
 /*
  * Check to see if a cpuset can accept a new task
@@ -3157,9 +3162,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
  */
 static cpumask_var_t cpus_attach;
 static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
 
 static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 {
+	struct mm_struct *mm;
+
 	lockdep_assert_cpuset_lock_held();
 
 	if (cs != &top_cpuset)
@@ -3173,28 +3181,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
 	 */
 	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
+	if (cpuset_v2() && !attach_mems_updated)
+		return;
+
 	cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
 	cpuset1_update_task_spread_flags(cs, task);
+
+	if ((task != task->group_leader) ||
+	    (!is_memory_migrate(cs) && !attach_mems_updated))
+		return;
+
+	/*
+	 * Change mm for threadgroup leader. This is expensive and may
+	 * sleep and should be moved outside migration path proper.
+	 */
+	mm = get_task_mm(task);
+	if (mm) {
+		struct cpuset *oldcs = cpuset_attach_old_cs;
+
+		mpol_rebind_mm(mm, &cs->effective_mems);
+
+		/*
+		 * old_mems_allowed is the same with mems_allowed
+		 * here, except if this task is being moved
+		 * automatically due to hotplug.  In that case
+		 * @mems_allowed has been updated and is empty, so
+		 * @old_mems_allowed is the right nodesets that we
+		 * migrate mm from.
+		 */
+		if (is_memory_migrate(cs)) {
+			cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+					  &cpuset_attach_nodemask_to);
+			queue_task_work = true;
+		} else {
+			mmput(mm);
+		}
+	}
 }
 
 static void cpuset_attach(struct cgroup_taskset *tset)
 {
 	struct task_struct *task;
-	struct task_struct *leader;
 	struct cgroup_subsys_state *css;
 	struct cpuset *cs;
 	struct cpuset *oldcs = cpuset_attach_old_cs;
-	bool cpus_updated, mems_updated;
-	bool queue_task_work = false;
 
 	cgroup_taskset_first(tset, &css);
 	cs = css_cs(css);
 
 	lockdep_assert_cpus_held();	/* see cgroup_attach_lock() */
 	mutex_lock(&cpuset_mutex);
-	cpus_updated = !cpumask_equal(cs->effective_cpus,
-				      oldcs->effective_cpus);
-	mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+	queue_task_work = false;
+
+	attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+	attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
 
 	/*
@@ -3203,44 +3243,12 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	 * and mems. In that case, we can optimize out by skipping the task
 	 * iteration and update.
 	 */
-	if (cpuset_v2() && !cpus_updated && !mems_updated)
+	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
 		goto out;
 
 	cgroup_taskset_for_each(task, css, tset)
 		cpuset_attach_task(cs, task);
 
-	/*
-	 * Change mm for all threadgroup leaders. This is expensive and may
-	 * sleep and should be moved outside migration path proper. Skip it
-	 * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
-	 * not set.
-	 */
-	if (!is_memory_migrate(cs) && !mems_updated)
-		goto out;
-
-	cgroup_taskset_for_each_leader(leader, css, tset) {
-		struct mm_struct *mm = get_task_mm(leader);
-
-		if (mm) {
-			mpol_rebind_mm(mm, &cs->effective_mems);
-
-			/*
-			 * old_mems_allowed is the same with mems_allowed
-			 * here, except if this task is being moved
-			 * automatically due to hotplug.  In that case
-			 * @mems_allowed has been updated and is empty, so
-			 * @old_mems_allowed is the right nodesets that we
-			 * migrate mm from.
-			 */
-			if (is_memory_migrate(cs)) {
-				cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
-						  &cpuset_attach_nodemask_to);
-				queue_task_work = true;
-			} else
-				mmput(mm);
-		}
-	}
-
 out:
 	if (queue_task_work)
 		schedule_flush_migrate_mm();
@@ -3689,15 +3697,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)
  */
 static void cpuset_fork(struct task_struct *task)
 {
-	struct cpuset *cs;
-	bool same_cs;
+	struct cpuset *cs, *oldcs;
 
 	rcu_read_lock();
 	cs = task_cs(task);
-	same_cs = (cs == task_cs(current));
+	oldcs = task_cs(current);
 	rcu_read_unlock();
 
-	if (same_cs) {
+	if (cs == oldcs) {
 		if (cs == &top_cpuset)
 			return;
 
@@ -3709,7 +3716,19 @@ static void cpuset_fork(struct task_struct *task)
 	/* CLONE_INTO_CGROUP */
 	mutex_lock(&cpuset_mutex);
 	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+	cs->old_mems_allowed = cpuset_attach_nodemask_to;
+
+	/*
+	 * Assume CPUs and memory nodes are updated
+	 * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
+	 * and so there shouldn't be a competing cpuset_attach() operation.
+	 */
+	attach_cpus_updated = attach_mems_updated = true;
+	queue_task_work = false;
+	cpuset_attach_old_cs = oldcs;
 	cpuset_attach_task(cs, task);
+	if (queue_task_work)
+		schedule_flush_migrate_mm();
 
 	dec_attach_in_progress_locked(cs);
 	mutex_unlock(&cpuset_mutex);
Signature · 1 line
2.54.0
reviewer
#14 lore raw

Re: [PATCH v7 7/9] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

Quoted 42 lines · Waiman Long

The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().

When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work

---
  kernel/cgroup/cpuset.c | 105 ++++++++++++++++++++++++-----------------
  1 file changed, 62 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +5 -0
index 0375dae26d0b..511afb077e2d 100644
@@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, int new_prs)
  /*
   * cpuset_can_attach() and cpuset_attach() specific internal data
   * Protected by cpuset_mutex
+ *
+ * The attach_cpus_updated/attach_mems_updated flags are set in either
+ * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task().
   */
  static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
 
  /*
   * Check to see if a cpuset can accept a new task

kernel/cgroup/cpuset.c static void cpuset_cancel_attach(struct cgroup_taskset *tset) · line 3166

Quoted 5 lines
kernel/cgroup/cpuset.c +1 -0
@@ -3157,9 +3162,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)
   */
  static cpumask_var_t cpus_attach;
  static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;
 

There are more and more of these standalone state variables now, and it's getting harder to maintain. Could we group them into a struct and manage them together rather than keep adding globals?

Just like:

```
struct cpuset_attach_ctx {
	struct cpuset     *old_cs;
	struct llist_head src_cs, dst_cs;
	bool              cpus_updated, mems_updated, queue_work;
	nodemask_t        nodemask_to;
};
```
Quoted 159 lines
kernel/cgroup/cpuset.c +56 -43
  static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
  {
+	struct mm_struct *mm;
+
  	lockdep_assert_cpuset_lock_held();
 
  	if (cs != &top_cpuset)
@@ -3173,28 +3181,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)
  	 */
  	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
 
+	if (cpuset_v2() && !attach_mems_updated)
+		return;
+
  	cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
  	cpuset1_update_task_spread_flags(cs, task);
+
+	if ((task != task->group_leader) ||
+	    (!is_memory_migrate(cs) && !attach_mems_updated))
+		return;
+
+	/*
+	 * Change mm for threadgroup leader. This is expensive and may
+	 * sleep and should be moved outside migration path proper.
+	 */
+	mm = get_task_mm(task);
+	if (mm) {
+		struct cpuset *oldcs = cpuset_attach_old_cs;
+
+		mpol_rebind_mm(mm, &cs->effective_mems);
+
+		/*
+		 * old_mems_allowed is the same with mems_allowed
+		 * here, except if this task is being moved
+		 * automatically due to hotplug.  In that case
+		 * @mems_allowed has been updated and is empty, so
+		 * @old_mems_allowed is the right nodesets that we
+		 * migrate mm from.
+		 */
+		if (is_memory_migrate(cs)) {
+			cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
+					  &cpuset_attach_nodemask_to);
+			queue_task_work = true;
+		} else {
+			mmput(mm);
+		}
+	}
  }
 
  static void cpuset_attach(struct cgroup_taskset *tset)
  {
  	struct task_struct *task;
-	struct task_struct *leader;
  	struct cgroup_subsys_state *css;
  	struct cpuset *cs;
  	struct cpuset *oldcs = cpuset_attach_old_cs;
-	bool cpus_updated, mems_updated;
-	bool queue_task_work = false;
 
  	cgroup_taskset_first(tset, &css);
  	cs = css_cs(css);
 
  	lockdep_assert_cpus_held();	/* see cgroup_attach_lock() */
  	mutex_lock(&cpuset_mutex);
-	cpus_updated = !cpumask_equal(cs->effective_cpus,
-				      oldcs->effective_cpus);
-	mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
+	queue_task_work = false;
+
+	attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus);
+	attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems);
  	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
 
  	/*
@@ -3203,44 +3243,12 @@ static void cpuset_attach(struct cgroup_taskset *tset)
  	 * and mems. In that case, we can optimize out by skipping the task
  	 * iteration and update.
  	 */
-	if (cpuset_v2() && !cpus_updated && !mems_updated)
+	if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated)
  		goto out;
 
  	cgroup_taskset_for_each(task, css, tset)
  		cpuset_attach_task(cs, task);
 
-	/*
-	 * Change mm for all threadgroup leaders. This is expensive and may
-	 * sleep and should be moved outside migration path proper. Skip it
-	 * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
-	 * not set.
-	 */
-	if (!is_memory_migrate(cs) && !mems_updated)
-		goto out;
-
-	cgroup_taskset_for_each_leader(leader, css, tset) {
-		struct mm_struct *mm = get_task_mm(leader);
-
-		if (mm) {
-			mpol_rebind_mm(mm, &cs->effective_mems);
-
-			/*
-			 * old_mems_allowed is the same with mems_allowed
-			 * here, except if this task is being moved
-			 * automatically due to hotplug.  In that case
-			 * @mems_allowed has been updated and is empty, so
-			 * @old_mems_allowed is the right nodesets that we
-			 * migrate mm from.
-			 */
-			if (is_memory_migrate(cs)) {
-				cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
-						  &cpuset_attach_nodemask_to);
-				queue_task_work = true;
-			} else
-				mmput(mm);
-		}
-	}
-
  out:
  	if (queue_task_work)
  		schedule_flush_migrate_mm();
@@ -3689,15 +3697,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)
   */
  static void cpuset_fork(struct task_struct *task)
  {
-	struct cpuset *cs;
-	bool same_cs;
+	struct cpuset *cs, *oldcs;
 
  	rcu_read_lock();
  	cs = task_cs(task);
-	same_cs = (cs == task_cs(current));
+	oldcs = task_cs(current);
  	rcu_read_unlock();
 
-	if (same_cs) {
+	if (cs == oldcs) {
  		if (cs == &top_cpuset)
  			return;
 
@@ -3709,7 +3716,19 @@ static void cpuset_fork(struct task_struct *task)
  	/* CLONE_INTO_CGROUP */
  	mutex_lock(&cpuset_mutex);
  	guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+	cs->old_mems_allowed = cpuset_attach_nodemask_to;
+
+	/*
+	 * Assume CPUs and memory nodes are updated
+	 * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
+	 * and so there shouldn't be a competing cpuset_attach() operation.
+	 */
+	attach_cpus_updated = attach_mems_updated = true;
+	queue_task_work = false;
+	cpuset_attach_old_cs = oldcs;
  	cpuset_attach_task(cs, task);
+	if (queue_task_work)
+		schedule_flush_migrate_mm();
 
  	dec_attach_in_progress_locked(cs);
  	mutex_unlock(&cpuset_mutex);
Signature · 2 lines
Best regards
Ridong
author maintainer
#15 lore raw

Re: [PATCH v7 7/9] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()

Quoted 55 lines · Ridong Chen
Quoted 42 lines · Waiman Long

The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().

When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
  - cpuset_attach_old_cs
  - attach_cpus_updated
  - attach_mems_updated
  - queue_task_work

---
  kernel/cgroup/cpuset.c | 105 ++++++++++++++++++++++++-----------------

  1 file changed, 62 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +5 -0
index 0375dae26d0b..511afb077e2d 100644
@@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, 
 int new_prs)
   /*
    * cpuset_can_attach() and cpuset_attach() specific internal data
    * Protected by cpuset_mutex
+ *
+ * The attach_cpus_updated/attach_mems_updated flags are set in either
+ * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task().
    */
   static struct cpuset *cpuset_attach_old_cs;
+static bool attach_cpus_updated;
+static bool attach_mems_updated;
     /*
    * Check to see if a cpuset can accept a new task

kernel/cgroup/cpuset.c static void cpuset_cancel_attach(struct · line 3166

Quoted 5 lines
kernel/cgroup/cpuset.c +1 -0
@@ -3157,9 +3162,12 @@ static void cpuset_cancel_attach(struct 
 cgroup_taskset *tset)
    */
   static cpumask_var_t cpus_attach;
   static nodemask_t cpuset_attach_nodemask_to;
+static bool queue_task_work;

There are more and more of these standalone state variables now, and it's getting harder to maintain. Could we group them into a struct and manage them together rather than keep adding globals?

Just like:

Quoted 8 lines
```
struct cpuset_attach_ctx {
    struct cpuset     *old_cs;
    struct llist_head src_cs, dst_cs;
    bool              cpus_updated, mems_updated, queue_work;
    nodemask_t        nodemask_to;
};
```

Yes, that makes sense. Will incorporate that in the next version.

Cheers,
Longman

author maintainer
#16 lore raw

PATCH v7 8/9 cgroup/cpuset: Support multiple source cpusets for cpuset_*attach()

In reply to Waiman Long

There are 2 possible scenarios where the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple source cpusets.

- A multithread application with threads in different cpusets is fully migrated into a new cpuset.
- Disabling v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.

The current cpuset_can_attach() and cpuset_attach() functions still expect task migration is from one source cpuset to one destination cpuset.

Fix that by tracking the set of source (old) cpusets in singly linked lists with the setting of attach_in_progress flag associated with the insertion into the list. The list will be iterated when necessary to properly update the internal data.

To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both the source and destination cpusets are decremented/incremented with their values added to nr_deadline_tasks when the migration is successful.

The setting of the global attach_cpus_updated and attach_mems_updated flags are also moved from cpuset_attach() to cpuset_can_attach() as the correct source cpuset can no longer be determined in cpuset_attach() and cpuset states will not be changed between cpuset_attach() and cpuset_can_attach() with an earlier patch.

---
 kernel/cgroup/cpuset-internal.h |  1 +
 kernel/cgroup/cpuset.c          | 66 ++++++++++++++++++++++++++++-----
 2 files changed, 57 insertions(+), 10 deletions(-)
kernel/cgroup/cpuset-internal.h +1 -0
index f7aaf01f7cd5..011993b1f756 100644
@@ -149,6 +149,7 @@ struct cpuset {
 	 * Tasks are being attached to this cpuset.  Used to prevent
 	 * zeroing cpus/mems_allowed between ->can_attach() and ->attach().
 	 */
+	struct llist_node attach_node;
 	int attach_in_progress;
 
 	/* partition root state */
kernel/cgroup/cpuset.c +56 -10

168 lines not on this page · raw message

Signature · 1 line
2.54.0
author maintainer
#17 lore raw

PATCH v7 9/9 cgroup/cpuset: Support multiple destination cpusets for cpuset_*attach()

In reply to Waiman Long

The only case where the cgroup_taskset structure requires task migration to multiple cpusets is when enabling a cpuset controller in cgroup v2 where the newly created child cpusets inherits the same effective CPUs and memory nodes from the parent. In that case, task migration can happen directly with no update to tasks' CPU and memory nodes assignment and no further work needed from the cpuset side exact updating nr_deadline_tasks when DL tasks are involved and setting old_mems_allowed in the child cpusets.

Do that by tracking all the destination cpusets with a new dst_cs_head singly linked list again with the setting of attach_in_progress associated with the insertion into the list.

It is assumed that a given cpuset cannot be both a source and a destination cpuset. If such condition happens or when there are multiple destination cpusets with CPU or memory nodes changes, the current code will not handle it correctly. So it will print a warning and fail the attach operation in these unexpected cases as we will have to enhance the code to support this if such use cases are valid and not coding bugs.

---
 kernel/cgroup/cpuset-internal.h |   1 +
 kernel/cgroup/cpuset.c          | 121 +++++++++++++++++++-------------
 2 files changed, 75 insertions(+), 47 deletions(-)
kernel/cgroup/cpuset-internal.h +1 -0
index 011993b1f756..900e74ac3538 100644
@@ -151,6 +151,7 @@ struct cpuset {
 	 */
 	struct llist_node attach_node;
 	int attach_in_progress;
+	bool attach_source;
 
 	/* partition root state */
 	int partition_root_state;
kernel/cgroup/cpuset.c +74 -47

215 lines not on this page · raw message

Signature · 1 line
2.54.0