cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
Subsystems (2)
Waiman Long
maintainerv4 cover cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
v4:
- Add a new patch 1 to fix inconsistency in node mask usage in cpuset_update_tasks_nodemask() and cpuset_attach() and adjust the subsequent patches accordingly.
- Update patch 3 to set the update flags whenever the CPU or node mask is updated to address issue reported by Sashiko.
- Update patch 5 to remove unneeded setting of old_mems_allowed as well as calling schedule_flush_migrate_mm() if queue_task_work is set.
v3:
- Rebased to the lastest linux-next tree.
- Keep cpuset_attach_old_cs as suggested by Chen Ridong and replace patch 3 by a new one to make it track task group leader.
Sashiko AI review of another cpuset patch had found that cpuset_attach() and cpuset_can_attach() can be passed a cgroup_taskset with tasks migrating from one source cpuset to multiple destination cpusets and vice versa. Further testing of the cpuset code indicates that this is indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that there will be one source and one destinaton cpuset which may result in inocrrect behavior.
This patch series is created to fix this issue.
Patch 1 is to fix an inconsistency in the way node mask update is being handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that they match each other.
Patches 2 and 3 are just preparatory patches to make the remaining patches easier to review.
Patch 4 makes cpuset_attach_old_cs to track group leader for use by cpuset_migrate_mm().
Patch 5 moves mpol_rebind_mm() and cpuset_migrate_mm() inside cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works more like moving task from one cpuset to another one, while also make supporting multiple source and destination cpusets easier.
Patch 6 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.
Waiman Long (6):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper cgroup/cpuset: Expand the scope of cpuset_can_attach_check() cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 424 +++++++++++++++++++++++--------- 2 files changed, 308 insertions(+), 122 deletions(-)
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v4 1/6 cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and cpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should only be useful for v1 where mems_allowed is the same as effective_mems. With v2, the memory nodes in effective_mems should always be a subset of node_states[N_MEMORY], so guarantee_online_mems() should just return cs->effective_mems.
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems. For v1, mpol_rebind_mm() uses cpus_allowed which may differ from what guarantee_online_mems() returns.
--- kernel/cgroup/cpuset.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-)
kernel/cgroup/cpuset.c +23 -11
@@ -2615,6 +2615,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask (struct cpuset *cs) {
@@ -2624,7 +2631,10 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems); +if (cpuset_v2 ()) + newmems = cs->effective_mems; +else +guarantee_online_mems (cs, &newmems);/* * The mpol_rebind_mm() call takes mmap_lock, which we couldn't
@@ -2649,7 +2659,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else
@@ -2713,6 +2723,8 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems)WARN_ON (!is_in_v2_mode () && !nodes_equal (cp->mems_allowed, cp->effective_mems)); +WARN_ON (cpuset_v2 () && + !nodes_subset (cp->effective_mems, node_states[N_MEMORY ]));cpuset_update_tasks_nodemask (cp);
@@ -3147,17 +3159,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);
@@ -3167,7 +3180,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;
@@ -3175,7 +3187,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Signature · 1 line
2.54.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Waiman Long
maintainerRe: [PATCH-next v4 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 31 lines · On 5/29/26 5:56 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] SpuriousWARN_ONduring memory hot-unplug racing with cpuset updates. --commit
050c4ab3304f0ad9eb91a3e6654d7a2dbb7cd92f
Author: Waiman Long <[email protected]>cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask()andcpuset_attach()This commit standardizes memory node mask usage in cpuset updates and attach operations to use
effective_memsfor cgroups v2. It also adds warning checks to ensureeffective_memsremains a subset of online memory nodes.Quoted 4 lines
kernel/cgroup/cpuset.c +0
-0[ ... ]
kernel/cgroup/cpuset.c static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems) · line 2728
Quoted 6 lines
kernel/cgroup/cpuset.c +2
-0@@ -2714,6 +2724,8 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems)WARN_ON (!is_in_v2_mode () && !nodes_equal (cp->mems_allowed, cp->effective_mems)); +WARN_ON (cpuset_v2 () && + !nodes_subset (cp->effective_mems, node_states[N_MEMORY ]));[Severity: Medium]
Can thisWARN_ONtrigger spuriously during a memory hot-unplug event?When a node is hot-unplugged, the memory hotplug subsystem removes the node from node_states[N_MEMORY] before invoking the notifier chain.
Quoted 8 lines
If
cpuset_track_online_nodes()callscpuset_handle_hotplug()but is delayed waiting forcpuset_mutex, and a concurrent write to cpuset.mems acquires the mutex first, wouldn'tcp->effective_memsstill contain the offlined node while node_states[N_MEMORY] does not?This race condition might cause unexpected panics on systems where
panic_on_warnis enabled.
OK, will remove the WARN_ON() test to avoid problem.
Cheers,
Longman
Michal Koutný
maintainerRe: [PATCH-next v4 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Hi.
Quoted 2 lines · Waiman Long
Quoted 2 lines
When a node is hot-unplugged, the memory hotplug subsystem removes the node from node_states[N_MEMORY] before invoking the notifier chain.
O.K.
Quoted 5 lines
Quoted 5 lines
If
cpuset_track_online_nodes()callscpuset_handle_hotplug()but is delayed waiting forcpuset_mutex, and a concurrent write to cpuset.mems acquires the mutex first, wouldn'tcp->effective_memsstill contain the offlined node while node_states[N_MEMORY] does not?
Not sure this condition could happen -- rewriting cpuset's effective_mems to a value containing an offlined node.
Quoted 3 lines
Quoted 3 lines
This race condition might cause unexpected panics on systems where
panic_on_warnis enabled.
That AI reasoning is disputable -- with panic_on_warn=1 one kind of _expects_ panics.
The warning upon a privileged action (memory hotplug) is IMO justified (and helpful for potential troubleshooting).
The question is whether the warning bears any information and what could go wrong when the predicate is violated.
Here I see the warning is right before cpuset_update_tasks_nodemask() so the invalid value may write through to tasks (only assuming the race is real) -- the warning may be useful.
Quoted 1 line
OK, will remove the
WARN_ON()test to avoid problem.
IOW, if you thought the warning was useful (IMO it's not clear cut), question that reasoning.
Michal
Waiman Long
maintainerRe: [PATCH-next v4 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 28 lines · Michal Koutný
Hi.
Quoted 2 lines · Waiman Long
Quoted 2 lines
When a node is hot-unplugged, the memory hotplug subsystem removes the node from node_states[N_MEMORY] before invoking the notifier chain.
O.K.
Quoted 4 lines
Quoted 4 lines
If
cpuset_track_online_nodes()callscpuset_handle_hotplug()but is delayed waiting forcpuset_mutex, and a concurrent write to cpuset.mems acquires the mutex first, wouldn'tcp->effective_memsstill contain the offlined node while node_states[N_MEMORY] does not?Not sure this condition could happen -- rewriting cpuset's
effective_memsto a value containing an offlined node.Quoted 2 lines
Quoted 2 lines
This race condition might cause unexpected panics on systems where
panic_on_warnis enabled.That AI reasoning is disputable -- with panic_on_warn=1 one kind of _expects_ panics.
The warning upon a privileged action (memory hotplug) is IMO justified (and helpful for potential troubleshooting).The question is whether the warning bears any information and what could go wrong when the predicate is violated.
Here I see the warning is right beforecpuset_update_tasks_nodemask()so the invalid value may write through to tasks (only assuming the race is real) -- the warning may be useful.
Quoted 3 lines
Quoted 1 line
OK, will remove the
WARN_ON()test to avoid problem.IOW, if you thought the warning was useful (IMO it's not clear cut), question that reasoning.
The warning is added so as to confirm that that all the nodes specified in effective_mems in v2 are online and effective_mems should be equivalent to the values returned by guarantee_online_mems(). As Sashiko has described, there is a corner case where the warning may get triggered even though the use of effective_mems in v2 should still be OK. I think it will be easier to just remove the WARN_ON() call instead of adding code to check if a hotplug operation has just happened.
Cheers,
Longman
Waiman Long
maintainerPATCH v4 2/6 cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
In reply to Waiman LongExtract the DL bandwidth allocation code in cpuset_attach() to a new cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
--- kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 23 deletions(-)
kernel/cgroup/cpuset.c +30 -23
@@ -2992,6 +2992,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)return 0 ; } +static int cpuset_reserve_dl_bw (struct cpuset *cs) +{ +int cpu, ret; + +if (!cs->sum_migrate_dl_bw) +return 0 ; + + cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) +return -EINVAL ; + + ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); +if (ret) +return ret; + + cs->dl_bw_cpu = cpu; +return 0 ; +} +static void reset_migrate_dl_data (struct cpuset *cs) { cs->nr_migrate_dl_tasks =0 ;
@@ -3006,7 +3025,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)struct cpuset *cs, *oldcs;struct task_struct *task;bool setsched_check; -int cpu, ret; +int ret;/* used later by cpuset_attach() */ cpuset_attach_old_cs =task_cs (cgroup_taskset_first (tset, &css));
@@ -3062,31 +3081,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) } } -if (!cs->sum_migrate_dl_bw) - goto out_success; - - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) { - ret = -EINVAL ; - goto out_unlock; - } - - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) - goto out_unlock; - - cs->dl_bw_cpu = cpu; - -out_success: -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ - cs->attach_in_progress++; + ret =cpuset_reserve_dl_bw (cs); out_unlock: -if (ret) +if (ret) {reset_migrate_dl_data (cs); + }else { +/* +* Mark attach is in progress. This makes validate_change() fail +* changes which zero cpus/mems_allowed. +*/ + cs->attach_in_progress++; + } +mutex_unlock (&cpuset_mutex);return ret; }
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v4 3/6 cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
In reply to Waiman LongExpand the scope of cpuset_can_attach_check() by including the setting of setsched flag inside cpuset_can_attach_check() with the new @oldcs and @psetsched argument. As cpuset_can_attach_check() is also called from cpuset_can_fork(), set the new arguments to NULL from that caller.
While at it, expose the source and destination cpuset cpu/memory check results in the new attach_cpus_updated and attach_mems_updated static flags so that these flags can be used directly from cpuset_attach() without the need to do the same computations again.
Two new global attach related flags are added (attach_cpus_updated & attach_mems_updated) which are set to indicate that CPUs or memory nodes are updated. These 2 flags are set in cpuset_can_attach() and are used in cpuset_attach() for optimization. Since cpuset_mutex will be released between the 2 calls, it is possible that an intervening cpuset action may change the CPU or node mask of the relevant cpusets, so check is added to set these flags if the effective_cpus or effective_mems of those cpusets is changed.
--- kernel/cgroup/cpuset.c | 90 ++++++++++++++++++++++++++++-------------- 1 file changed, 60 insertions(+), 30 deletions(-)
kernel/cgroup/cpuset.c +60 -30
@@ -1108,6 +1108,14 @@ enum partition_cmd {static void update_sibling_cpumasks (struct cpuset *parent,struct cpuset *cs,struct tmpmasks *tmp); +/* +* cpuset_can_attach() and cpuset_attach() specific internal data +* Protected by cpuset_mutex +*/ +static struct cpuset *cpuset_attach_old_cs; +static bool attach_cpus_updated; +static bool attach_mems_updated; +/* * Update partition exclusive flag *
@@ -1192,6 +1200,8 @@ static void reset_partition_data(struct cpuset *cs) }if (!cpumask_and (cs->effective_cpus, parent->effective_cpus, cs->cpus_allowed))cpumask_copy (cs->effective_cpus, parent->effective_cpus); +if (cs->attach_in_progress) + attach_cpus_updated = true; }/*
@@ -1242,6 +1252,8 @@ static void partition_xcpus_add(int new_prs, struct cpuset *parent, xcpus);cpumask_andnot (parent->effective_cpus, parent->effective_cpus, xcpus); +if (parent->attach_in_progress) + attach_cpus_updated = true; }/*
@@ -1269,6 +1281,8 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,cpumask_or (parent->effective_cpus, parent->effective_cpus, xcpus);cpumask_and (parent->effective_cpus, parent->effective_cpus, cpu_active_mask); +if (parent->attach_in_progress) + attach_cpus_updated = true; }/*
@@ -2217,6 +2231,8 @@ static void update_cpumasks_hier(struct cpuset *cs, struct tmpmasks *tmp,if (new_prs <=0 )reset_partition_data (cp);spin_unlock_irq (&callback_lock); +if (cp->attach_in_progress) + attach_cpus_updated = true;notify_partition_change (cp, old_prs);
@@ -2720,6 +2736,8 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems)spin_lock_irq (&callback_lock); cp->effective_mems = *new_mems;spin_unlock_irq (&callback_lock); +if (cp->attach_in_progress) + attach_mems_updated = true;WARN_ON (!is_in_v2_mode () && !nodes_equal (cp->mems_allowed, cp->effective_mems));
@@ -2976,19 +2994,48 @@ static int update_prstate(struct cpuset *cs, int new_prs)return 0 ; } -static struct cpuset *cpuset_attach_old_cs; -/* * Check to see if a cpuset can accept a new task * For v1, cpus_allowed and mems_allowed can't be empty. * For v2, effective_cpus can't be empty. * Note that in v1, effective_cpus = cpus_allowed. +* +* Also set the boolean flag passed in by @psetsched depending on if +* security_task_setscheduler() call is needed and @oldcs is not NULL. */ -static int cpuset_can_attach_check (struct cpuset *cs) +static int cpuset_can_attach_check (struct cpuset *cs,struct cpuset *oldcs, +bool *psetsched) {if (cpumask_empty (cs->effective_cpus) || (!is_in_v2_mode () &&nodes_empty (cs->mems_allowed)))return -ENOSPC ; + +if (!oldcs) +return 0 ; + +/* +* Update attach specific data +*/ + attach_cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); + attach_mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems); + +/* +* Skip rights over task setsched check in v2 when nothing changes, +* migration permission derives from hierarchy ownership in +* cgroup_procs_write_permission()). +*/ + *psetsched = !cpuset_v2 () || attach_cpus_updated || attach_mems_updated; + +/* +* A v1 cpuset with tasks will have no CPU left only when CPU hotplug +* brings the last online CPU offline as users are not allowed to empty +* cpuset.cpus when there are active tasks inside. When that happens, +* we should allow tasks to migrate out without security check to make +* sure they will be able to run after migration. +*/ +if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) + *psetsched = false; +return 0 ; }
@@ -3035,29 +3082,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* Skip rights over task setsched check in v2 when nothing changes, -* migration permission derives from hierarchy ownership in -* cgroup_procs_write_permission()). -*/ - setsched_check = !cpuset_v2 () || - !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || - !nodes_equal (cs->effective_mems, oldcs->effective_mems); - -/* -* A v1 cpuset with tasks will have no CPU left only when CPU hotplug -* brings the last online CPU offline as users are not allowed to empty -* cpuset.cpus when there are active tasks inside. When that happens, -* we should allow tasks to migrate out without security check to make -* sure they will be able to run after migration. -*/ -if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) - setsched_check = false; -cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret)
@@ -3152,7 +3180,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct cgroup_subsys_state *css;struct cpuset *cs;struct cpuset *oldcs = cpuset_attach_old_cs; -bool cpus_updated, mems_updated;bool queue_task_work = false;cgroup_taskset_first (tset, &css);
@@ -3160,9 +3187,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)lockdep_assert_cpus_held ();/* see cgroup_attach_lock() */ mutex_lock (&cpuset_mutex); - cpus_updated = !cpumask_equal (cs->effective_cpus, - oldcs->effective_cpus); - mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems);/* * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3172,7 +3196,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)*/ if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; -if (!cpus_updated && !mems_updated) +if (!attach_cpus_updated && !attach_mems_updated) goto out; }else {guarantee_online_mems (cs, &cpuset_attach_nodemask_to);
@@ -3187,7 +3211,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ -if (!is_memory_migrate (cs) && !mems_updated) +if (!is_memory_migrate (cs) && !attach_mems_updated) goto out;cgroup_taskset_for_each_leader (leader, css, tset) {
@@ -3602,7 +3626,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs,NULL ,NULL );if (ret) goto out_unlock;
@@ -3742,6 +3766,8 @@ hotplug_update_tasks(struct cpuset *cs,cpumask_copy (cs->effective_cpus, new_cpus); cs->effective_mems = *new_mems;spin_unlock_irq (&callback_lock); +if (cs->attach_in_progress) + attach_cpus_updated = attach_mems_updated = true;if (cpus_updated)cpuset_update_tasks_cpumask (cs, new_cpus);
@@ -3927,6 +3953,8 @@ static void cpuset_handle_hotplug(void) }cpumask_copy (top_cpuset.effective_cpus, &new_cpus);spin_unlock_irq (&callback_lock); +if (top_cpuset.attach_in_progress) + attach_cpus_updated = true;/* we don't mess with cpumasks of tasks in top_cpuset */ }
@@ -3937,6 +3965,8 @@ static void cpuset_handle_hotplug(void) top_cpuset.mems_allowed = new_mems; top_cpuset.effective_mems = new_mems;spin_unlock_irq (&callback_lock); +if (top_cpuset.attach_in_progress) + attach_mems_updated = true;cpuset_update_tasks_nodemask (&top_cpuset); }
Signature · 1 line
2.54.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Waiman Long
maintainerRe: [PATCH-next v4 3/6] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
Quoted 74 lines · On 5/29/26 6:33 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Cachingattach_cpus_updatedandattach_mems_updatedglobally introduces a TOCTOU race with CPU/memory hotplug. --commit
86181172df4d07555b90ca27dbd4515786dc4970
Author: Waiman Long <[email protected]>cgroup/cpuset: Expand the scope of
cpuset_can_attach_check()Expand the scope of
cpuset_can_attach_check()by including the setting of setsched flag insidecpuset_can_attach_check()with the new @oldcs and @psetsched argument.Quoted 4 lines
kernel/cgroup/cpuset.c +0
-0[ ... ]
Quoted 19 lines
kernel/cgroup/cpuset.c +0
-4@@ -3153,7 +3181,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct cgroup_subsys_state *css;struct cpuset *cs;struct cpuset *oldcs = cpuset_attach_old_cs; -bool cpus_updated, mems_updated;bool queue_task_work = false;cgroup_taskset_first (tset, &css);@@ -3161,9 +3188,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)lockdep_assert_cpus_held ();/* see cgroup_attach_lock() */ mutex_lock (&cpuset_mutex); - cpus_updated = !cpumask_equal (cs->effective_cpus, - oldcs->effective_cpus); - mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems);/* * In the default hierarchy, enabling cpuset in the child cgroups kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3201
Quoted 6 lines
kernel/cgroup/cpuset.c +1
-1@@ -3173,7 +3197,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)*/ if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; -if (!cpus_updated && !mems_updated) +if (!attach_cpus_updated && !attach_mems_updated) goto out;[Severity: High]
Can caching these flags globally incpuset_can_attach_check()introduce a Time-of-Check to Time-of-Use (TOCTOU) race condition with CPU/memory hotplug?While
cgroup_mutexprevents concurrent userspace cpuset.cpus writes, CPU and memory hotplug events (cpuset_handle_hotplug) only requirecpuset_mutex, which is dropped betweencpuset_can_attach()andcpuset_attach().In
cpuset_can_attach(), we incrementcs->attach_in_progress, but notoldcs->attach_in_progress.If a hotplug event occurs during the unlocked window and affects oldcs, the hotplug handler iterates over all cpusets. For oldcs, it proceeds without blocking, updating oldcs's masks and the migrating task's masks (since the task is still in oldcs).
When the hotplug handler reaches cs, it sees
cs->attach_in_progress> 0, dropscpuset_mutex, and blocks.Then
cpuset_attach()resumes, re-acquirescpuset_mutex, and reads the stale globalattach_cpus_updatedflag (which is still false).Does this cause
cpuset_attach()to incorrectly take the goto out path, skipping the update of the task's masks to cs's masks?If cs's configuration wasn't affected by the hotplug event, its masks won't change when the hotplug handler resumes. The task would remain in cs but retain oldcs's hotplug-updated masks, potentially violating cgroup isolation boundaries.
[ ... ]
Quoted 11 lines
kernel/cgroup/cpuset.c hotplug_update_tasks(struct cpuset *cs, · line 3771
Quoted 6 lines
kernel/cgroup/cpuset.c +2
-0@@ -3743,6 +3767,8 @@ hotplug_update_tasks(struct cpuset *cs,cpumask_copy (cs->effective_cpus, new_cpus); cs->effective_mems = *new_mems;spin_unlock_irq (&callback_lock); +if (cs->attach_in_progress) + attach_cpus_updated = attach_mems_updated = true;[Severity: High]
Becauseoldcs->attach_in_progressis not incremented during migration, this check will evaluate to false for oldcs. A concurrent hotplug update to oldcs will be missed, leaving the globalattach_cpus_updatedandattach_mems_updatedflags incorrectly set to false.
Will add a global attach_in_progress flag to disable optimization (set updated flags to true) whenever there is an intervening hotplug events or cpuset files update.
Cheers,
Longman
Waiman Long
maintainerPATCH v4 4/6 cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders
In reply to Waiman LongThere are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.
IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach().
--- kernel/cgroup/cpuset.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/cgroup/cpuset.c +18 -0
@@ -1111,6 +1111,20 @@ static void update_sibling_cpumasks(struct cpuset *parent, struct cpuset *cs,/* * cpuset_can_attach() and cpuset_attach() specific internal data * Protected by cpuset_mutex +* +* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get the +* old_mems_allowed value. There are two ways that many-to-one cpuset migration +* can happen: +* 1) A multithread application with threads in different cpusets is wholely +* moved to a new cpuset. +* 2) Disabling v2 cpuset controller will move all the tasks in child cpusets +* to the parent cpuset. +* +* In the former case, it is the mm setting of the group leader that really +* matters. So cpuset_attach_old_cs should track the oldcs of the thread +* leader. In the latter case, effective_mems of child cpusets must always +* be a subset of the parent. So no real page migration will be necessary no +* matter which child cpuset is selected as cpuset_attach_old_cs. */ static struct cpuset *cpuset_attach_old_cs;static bool attach_cpus_updated;
@@ -3091,6 +3105,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)if (ret) goto out_unlock; +/* Update cpuset_attach_old_cs to the latest group leader */ +if (task == task->group_leader) + cpuset_attach_old_cs =task_cs (task); +if (setsched_check) { ret =security_task_setscheduler (task);if (ret)
Signature · 1 line
2.54.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Waiman Long
maintainerRe: [PATCH-next v4 4/6] cgroup/cpuset: Made cpuset_attach_old_cs track task group leaders
Quoted 41 lines · On 5/29/26 7:19 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Pre-existing issues:
- [High] Global boolean flags are evaluated based solely on the first task's source cpuset during multi-cpuset migrations, skipping updates for subsequent tasks.
- [High]SCHED_DEADLINEtask accounting (nr_deadline_tasks) underflows and leaks when migrating tasks from multiple source cpusets.
- [High] When migrating a process whose threads reside in different cpusets,cpuset_migrate_mm()only migrates pages from the leader's old nodes, bypassing other threads' nodes.
- [High] If a migrating process's thread group leader has exited (e.g., is a zombie), page migration and memory policy rebinding are entirely bypassed for surviving threads. --commit
80e6ff469724f377b7de8a5fa03f60d067749c38
Author: Waiman Long <[email protected]>cgroup/cpuset: Made
cpuset_attach_old_cstrack task group leadersThis commit modifies
cpuset_can_attach()to updatecpuset_attach_old_csto track the old cpuset of the task group leader. This ensures that when a multithreaded application spans multiple cpusets, the mm setting of the group leader is used during migration.kernel/cgroup/cpuset.c static int cpuset_can_attach(struct cgroup_taskset *tset) · line 3112
Quoted 12 lines
kernel/cgroup/cpuset.c +4
-0@@ -3092,6 +3106,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)if (ret) goto out_unlock; +/* Update cpuset_attach_old_cs to the latest group leader */ +if (task == task->group_leader) + cpuset_attach_old_cs =task_cs (task); +[Severity: High]
This isn't a bug introduced by this patch, but does relying on a singlecpuset_attach_old_cscause global boolean flags to be evaluated incorrectly during multi-cpuset migrations?In
cpuset_can_attach_check(),attach_cpus_updatedandattach_mems_updatedare evaluated by comparing the target cpuset only against the oldcs of the first task. If the first task originates from a cpuset with masks matching the target, these flags become false.
Quoted 7 lines
This optimizes out the v2 task mask updates in
cpuset_attach():if (!attach_cpus_updated && !attach_mems_updated) goto out;Does this mean subsequent tasks migrating from different cpusets are skipped and left with incorrect affinities?
I am going to remove the global update flags and re-evaluate the local update flags in cpuset_attach().
Quoted 10 lines
[Severity: High]
This is a pre-existing issue, but does tracking a single oldcs causeSCHED_DEADLINEtask accounting to underflow?In
cpuset_can_attach(), the count of migratingSCHED_DEADLINEtasks across all source cpusets is aggregated into a single variable:cs->nr_migrate_dl_tasks++;
Quoted 8 lines
In
cpuset_attach(), this total count is subtracted from thenr_deadline_tasksof the single selected oldcs:oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;If tasks are migrating from multiple cpusets, this would cause an underflow in the selected oldcs and leak the deadline task count in the other source cpusets.
This is a known problem that has been fixed in a later patch of the series.
Quoted 10 lines
[Severity: High]
This isn't a bug introduced by this patch, but sincecpuset_migrate_mm()uses only the leader'sold_mems_allowed, does it bypass page migration for pages allocated by other threads?
cpuset_migrate_mm()is invoked with the from mask pointing to&oldcs->old_mems_allowed. If the process has threads in other cpusets that allocated pages on different memory nodes, those nodes are excluded from the from mask, leaving pages on nodes that may be disallowed in the new cpuset.
For best performance, the node mask of all the threads of a multithreaded application should be the same, though it is common for different threads to be associated with different CPUs. Anyway, this is out of scope for this patch series and will not be addressed.
Quoted 11 lines
[Severity: High]
This is a pre-existing issue, but if a migrating process's thread group leader has exited (e.g., is a zombie), does this fail to updatecpuset_attach_old_csand bypass migration for surviving threads?Exiting tasks are excluded from the migration taskset. If the leader is absent from tset,
cpuset_attach()never executes its loop body for surviving threads because it iterates usingcgroup_taskset_for_each_leader(). This leaves surviving threads' memory unmigrated and their memory policies unbound to the new cpuset.
It is out of scope for this patch series and will not be addressed here.
Cheers,
Longman
Waiman Long
maintainerPATCH v4 5/6 cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
In reply to Waiman LongThe cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work
--- kernel/cgroup/cpuset.c | 89 ++++++++++++++++++++++++------------------ 1 file changed, 51 insertions(+), 38 deletions(-)
kernel/cgroup/cpuset.c +51 -38
@@ -3171,9 +3171,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)*/ static cpumask_var_t cpus_attach;static nodemask_t cpuset_attach_nodemask_to; +static bool queue_task_work;static void cpuset_attach_task (struct cpuset *cs,struct task_struct *task) { +struct mm_struct *mm; +lockdep_assert_cpuset_lock_held ();if (cs != &top_cpuset)
@@ -3187,24 +3190,56 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)*/ WARN_ON_ONCE (set_cpus_allowed_ptr (task, cpus_attach)); +if (cpuset_v2 () && !attach_mems_updated) +return ; +cpuset_change_task_nodemask (task, &cpuset_attach_nodemask_to);cpuset1_update_task_spread_flags (cs, task); + +if (task != task->group_leader) +return ; + +/* +* Change mm for threadgroup leader. This is expensive and may +* sleep and should be moved outside migration path proper. +*/ + mm =get_task_mm (task); +if (mm) { +struct cpuset *oldcs = cpuset_attach_old_cs; + +mpol_rebind_mm (mm, &cs->effective_mems); + +/* +* old_mems_allowed is the same with mems_allowed +* here, except if this task is being moved +* automatically due to hotplug. In that case +* @mems_allowed has been updated and is empty, so +* @old_mems_allowed is the right nodesets that we +* migrate mm from. +*/ +if (is_memory_migrate (cs)) { +cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, + &cpuset_attach_nodemask_to); + queue_task_work = true; + }else { +mmput (mm); + } + } }static void cpuset_attach (struct cgroup_taskset *tset) {struct task_struct *task; -struct task_struct *leader;struct cgroup_subsys_state *css;struct cpuset *cs;struct cpuset *oldcs = cpuset_attach_old_cs; -bool queue_task_work = false;cgroup_taskset_first (tset, &css); cs =css_cs (css);lockdep_assert_cpus_held ();/* see cgroup_attach_lock() */ mutex_lock (&cpuset_mutex); + queue_task_work = false;/* * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3223,38 +3258,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task); -/* -* Change mm for all threadgroup leaders. This is expensive and may -* sleep and should be moved outside migration path proper. Skip it -* if there is no change in effective_mems and CS_MEMORY_MIGRATE is -* not set. -*/ -if (!is_memory_migrate (cs) && !attach_mems_updated) - goto out; - -cgroup_taskset_for_each_leader (leader, css, tset) { -struct mm_struct *mm =get_task_mm (leader); - -if (mm) { -mpol_rebind_mm (mm, &cs->effective_mems); - -/* -* old_mems_allowed is the same with mems_allowed -* here, except if this task is being moved -* automatically due to hotplug. In that case -* @mems_allowed has been updated and is empty, so -* @old_mems_allowed is the right nodesets that we -* migrate mm from. -*/ -if (is_memory_migrate (cs)) { -cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, - &cpuset_attach_nodemask_to); - queue_task_work = true; - }else -mmput (mm); - } - } - out:if (queue_task_work)schedule_flush_migrate_mm ();
@@ -3688,15 +3691,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)*/ static void cpuset_fork (struct task_struct *task) { -struct cpuset *cs; -bool same_cs; +struct cpuset *cs, *oldcs;rcu_read_lock (); cs =task_cs (task); - same_cs = (cs ==task_cs (current)); + oldcs =task_cs (current);rcu_read_unlock (); -if (same_cs) { +if (cs == oldcs) {if (cs == &top_cpuset)return ;
@@ -3708,7 +3710,18 @@ static void cpuset_fork(struct task_struct *task)/* CLONE_INTO_CGROUP */ mutex_lock (&cpuset_mutex);guarantee_online_mems (cs, &cpuset_attach_nodemask_to); +/* +* Assume CPUs and memory nodes are updated +* A CLONE_INTO_CGROUP operation should have taken the cgroup mutex +* and so there shouldn't be a competing cpuset_attach() operation. +*/ + attach_cpus_updated = attach_mems_updated = true; + queue_task_work = false; + cpuset_attach_old_cs = oldcs;cpuset_attach_task (cs, task); + attach_cpus_updated = attach_mems_updated = false; +if (queue_task_work) +schedule_flush_migrate_mm ();dec_attach_in_progress_locked (cs);mutex_unlock (&cpuset_mutex);
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v4 6/6 cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
In reply to Waiman LongWith cgroup v2, the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple destination or source cpusets when the cpuset controller is enabled or disabled respectively.
Since cpuset is threaded in both v1 and v2, another possible way to cause many-to-one migration is to move the whole process with multiple threads in different cpuset enabled threaded cgroups into another cpuset enabled cgroup.
The current cpuset_can_attach() and cpuset_attach() functions still expect task migration is from one source cpuset to one destination cpuset. This has been the case since cpuset was enabled for cgroup v2 in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy").
This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.
Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both the source and destination cpusets are decremented/incremented with their values added to nr_deadline_tasks when the migration is successful.
--- kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 204 ++++++++++++++++++++++++-------- 2 files changed, 158 insertions(+), 52 deletions(-)
kernel/cgroup/cpuset-internal.h +6 -0
@@ -161,6 +161,12 @@ struct cpuset {*/ bool remote_partition; +/* +* cpuset_can_attach() and cpuset_attach() specific data +*/ +bool attach_node_in_llist; +struct llist_node attach_node; +/* * number of SCHED_DEADLINE tasks attached to this cpuset, so that we * know when to rebuild associated root domain bandwidth information.
kernel/cgroup/cpuset.c +152 -52
303 lines not on this page · raw message
Signature · 1 line
2.54.0