cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
Subsystems (5)
Waiman Long
maintainerv5 cover cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
v5:
- Remove the WARN_ON() call as it can be triggered in a corner case.
- Instead of passing an attach_cpus_updated and attach_mems_updated flags from cpuset_can_attach() to cpuset_attach(), re-evaluate the flags at the beginning of cpuset_attach() based on data in the source & destination cpusets in the singly linked lists to eliminate the Time-of-Check to Time-of-Use (TOCTOU) race condition & simplify the code changes.
- Add back the dropped optimization in patch 5.
v4:
- Add a new patch 1 to fix inconsistency in node mask usage in cpuset_update_tasks_nodemask() and cpuset_attach() and adjust the subsequent patches accordingly.
- Update patch 3 to set the update flags whenever the CPU or node mask is updated to address issue reported by Sashiko.
- Update patch 5 to remove unneeded setting of old_mems_allowed as well as calling schedule_flush_migrate_mm() if queue_task_work is set.
v3:
- Rebased to the lastest linux-next tree.
- Keep cpuset_attach_old_cs as suggested by Chen Ridong and replace patch 3 by a new one to make it track task group leader.
Sashiko AI review of another cpuset patch had found that cpuset_attach() and cpuset_can_attach() can be passed a cgroup_taskset with tasks migrating from one source cpuset to multiple destination cpusets and vice versa. Further testing of the cpuset code indicates that this is indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that there will be one source and one destinaton cpuset which may result in inocrrect behavior.
This patch series is created to fix this issue.
Patch 1 is to fix an inconsistency in the way node mask update is being handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that they match each other.
Patches 2 and 3 are just preparatory patches to make the remaining patches easier to review.
Patch 4 makes cpuset_attach_old_cs to track group leader for use by cpuset_migrate_mm().
Patch 5 moves mpol_rebind_mm() and cpuset_migrate_mm() inside cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works more like moving task from one cpuset to another one, while also make supporting multiple source and destination cpusets easier.
Patch 6 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.
Waiman Long (6):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper cgroup/cpuset: Expand the scope of cpuset_can_attach_check() cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 411 +++++++++++++++++++++++--------- 2 files changed, 299 insertions(+), 118 deletions(-)
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v5 1/6 cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and cpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should only be useful for v1 where mems_allowed is the same as effective_mems. With v2, the memory nodes in effective_mems should always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected in effective_mems as cpuset_handle_hotplug() has not yet been called from cpuset_track_online_nodes(). When cpuset_handle_hotplug() is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing the guarantee_online_mems() call by just using cs->effective_mems should be fine.
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems. For v1, mpol_rebind_mm() uses cpus_allowed which may differ from what guarantee_online_mems() returns.
--- kernel/cgroup/cpuset.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
kernel/cgroup/cpuset.c +21 -11
@@ -2616,6 +2616,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask (struct cpuset *cs) {
@@ -2625,7 +2632,10 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems); +if (cpuset_v2 ()) + newmems = cs->effective_mems; +else +guarantee_online_mems (cs, &newmems);/* * The mpol_rebind_mm() call takes mmap_lock, which we couldn't
@@ -2650,7 +2660,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else
@@ -3148,17 +3158,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);
@@ -3168,7 +3179,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;
@@ -3176,7 +3186,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Signature · 1 line
2.54.0
Ridong Chen
reviewerRe: [PATCH-next v5 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 33 lines · Waiman Long
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask viacpuset_change_task_nodemask()
2) memory policy binding viampol_rebind_mm()
3) if memory migration is enabled, migrate fromold_mems_allowedto the new node mask viacpuset_migrate_mm().
4) settingold_mems_allowedThese memory actions are done in
cpuset_update_tasks_nodemask()andcpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.In
cpuset_update_tasks_nodemask(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():mems_allowed
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()In
cpuset_attach(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():effective_mems
-old_mems_allowed:effective_memsThese inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The
guarantee_online_mems()function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
Quoted 12 lines
The
guarantee_online_mems()function should only be useful for v1 wheremems_allowedis the same aseffective_mems. With v2, the memory nodes ineffective_memsshould always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected ineffective_memsascpuset_handle_hotplug()has not yet been called fromcpuset_track_online_nodes(). Whencpuset_handle_hotplug()is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing theguarantee_online_mems()call by just usingcs->effective_memsshould be fine.
I noticed this pattern in several places:
``` if (cpuset_v2()) newmems = cs->effective_mems; else guarantee_online_mems(cs, &newmems); ```
Would it be simpler to move the v2 logic into guarantee_online_mems?
```
static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
{
if (cpuset_v2()) {
*pmask = cs->effective_mems;
return;
}
while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]))
cs = parent_cs(cs);
}
```Quoted 8 lines
Let use the following setup for both of them and make them consistent.
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()So for v2, it is effectively all
effective_mems. For v1,mpol_rebind_mm()usescpus_allowedwhich may differ from whatguarantee_online_mems()
^
mems_allowed?
Quoted 12 lines
returns.
Signed-off-by Waiman Long <[email protected]>--- kernel/cgroup/cpuset.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)kernel/cgroup/cpuset.c +0
-0
kernel/cgroup/cpuset.c static void *cpuset_being_rebound; · line 2625
Quoted 10 lines
kernel/cgroup/cpuset.c +7
-0@@ -2616,6 +2616,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems.
I agree, but the implementation is not as simple as what I mentioned above.
Quoted 67 lines
kernel/cgroup/cpuset.c +14
-11*/ void cpuset_update_tasks_nodemask (struct cpuset *cs) {@@ -2625,7 +2632,10 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems); +if (cpuset_v2 ()) + newmems = cs->effective_mems; +else +guarantee_online_mems (cs, &newmems);/* * The mpol_rebind_mm() call takes mmap_lock, which we couldn't @@ -2650,7 +2660,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else @@ -3148,17 +3158,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);@@ -3168,7 +3179,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;@@ -3176,7 +3186,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerRe: [PATCH-next v5 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 58 lines · Ridong Chen
Quoted 33 lines · Waiman Long
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask viacpuset_change_task_nodemask()
2) memory policy binding viampol_rebind_mm()
3) if memory migration is enabled, migrate fromold_mems_allowedto the new node mask viacpuset_migrate_mm().
4) settingold_mems_allowedThese memory actions are done in
cpuset_update_tasks_nodemask()andcpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.In
cpuset_update_tasks_nodemask(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():mems_allowed
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()In
cpuset_attach(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():effective_mems
-old_mems_allowed:effective_memsThese inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The
guarantee_online_mems()function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].Quoted 12 lines
The
guarantee_online_mems()function should only be useful for v1 wheremems_allowedis the same aseffective_mems. With v2, the memory nodes ineffective_memsshould always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected ineffective_memsascpuset_handle_hotplug()has not yet been called fromcpuset_track_online_nodes(). Whencpuset_handle_hotplug()is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing theguarantee_online_mems()call by just usingcs->effective_memsshould be fine.I noticed this pattern in several places:
``` if (cpuset_v2()) newmems = cs->effective_mems; else guarantee_online_mems(cs, &newmems); ```Would it be simpler to move the v2 logic into
guarantee_online_mems?
Quoted 11 lines
```
static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
{
if (cpuset_v2()) {
*pmask = cs->effective_mems;
return;
}
while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]))
cs = parent_cs(cs);
}
```Yes, it makes sense to put it directly into guarantee_online_mems().
Quoted 10 lines
Quoted 8 lines
Let use the following setup for both of them and make them consistent.
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()So for v2, it is effectively all
effective_mems. For v1,mpol_rebind_mm()usescpus_allowedwhich may differ from whatguarantee_online_mems()^
mems_allowed?
Thanks for catching it. Will fix that and update your email address in the next version.
Cheers,
Longman
Quoted 91 lines
Quoted 12 lines
returns.
Signed-off-by Waiman Long <[email protected]>--- kernel/cgroup/cpuset.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)kernel/cgroup/cpuset.c +0
-0kernel/cgroup/cpuset.c static void *cpuset_being_rebound; · line 2625
Quoted 10 lines
kernel/cgroup/cpuset.c +7
-0@@ -2616,6 +2616,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. I agree, but the implementation is not as simple as what I mentioned above.
Quoted 67 lines
kernel/cgroup/cpuset.c +14
-11*/ void cpuset_update_tasks_nodemask (struct cpuset *cs) {@@ -2625,7 +2632,10 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems); +if (cpuset_v2 ()) + newmems = cs->effective_mems; +else +guarantee_online_mems (cs, &newmems);/* * The mpol_rebind_mm() call takes mmap_lock, which we couldn't @@ -2650,7 +2660,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else @@ -3148,17 +3158,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);@@ -3168,7 +3179,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;@@ -3176,7 +3186,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Waiman Long
maintainerPATCH v5 2/6 cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
In reply to Waiman LongExtract the DL bandwidth allocation code in cpuset_attach() to a new cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
--- kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 23 deletions(-)
kernel/cgroup/cpuset.c +30 -23
@@ -2991,6 +2991,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)return 0 ; } +static int cpuset_reserve_dl_bw (struct cpuset *cs) +{ +int cpu, ret; + +if (!cs->sum_migrate_dl_bw) +return 0 ; + + cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) +return -EINVAL ; + + ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); +if (ret) +return ret; + + cs->dl_bw_cpu = cpu; +return 0 ; +} +static void reset_migrate_dl_data (struct cpuset *cs) { cs->nr_migrate_dl_tasks =0 ;
@@ -3005,7 +3024,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)struct cpuset *cs, *oldcs;struct task_struct *task;bool setsched_check; -int cpu, ret; +int ret;/* used later by cpuset_attach() */ cpuset_attach_old_cs =task_cs (cgroup_taskset_first (tset, &css));
@@ -3061,31 +3080,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) } } -if (!cs->sum_migrate_dl_bw) - goto out_success; - - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) { - ret = -EINVAL ; - goto out_unlock; - } - - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) - goto out_unlock; - - cs->dl_bw_cpu = cpu; - -out_success: -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ - cs->attach_in_progress++; + ret =cpuset_reserve_dl_bw (cs); out_unlock: -if (ret) +if (ret) {reset_migrate_dl_data (cs); + }else { +/* +* Mark attach is in progress. This makes validate_change() fail +* changes which zero cpus/mems_allowed. +*/ + cs->attach_in_progress++; + } +mutex_unlock (&cpuset_mutex);return ret; }
Signature · 1 line
2.54.0
Re: [PATCH-next v5 2/6] cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
Quoted 6 lines · Waiman Long
Extract the DL bandwidth allocation code in
cpuset_attach()to a newcpuset_reserve_dl_bw()helper to simplify code.No functional change is expected.
Signed-off-by Waiman Long <[email protected]>
LGTM.
Quoted 85 lines
--- kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 23 deletions(-)kernel/cgroup/cpuset.c +30
-23@@ -2991,6 +2991,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)return 0 ; } +static int cpuset_reserve_dl_bw (struct cpuset *cs) +{ +int cpu, ret; + +if (!cs->sum_migrate_dl_bw) +return 0 ; + + cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) +return -EINVAL ; + + ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); +if (ret) +return ret; + + cs->dl_bw_cpu = cpu; +return 0 ; +} +static void reset_migrate_dl_data (struct cpuset *cs) { cs->nr_migrate_dl_tasks =0 ;@@ -3005,7 +3024,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)struct cpuset *cs, *oldcs;struct task_struct *task;bool setsched_check; -int cpu, ret; +int ret;/* used later by cpuset_attach() */ cpuset_attach_old_cs =task_cs (cgroup_taskset_first (tset, &css));@@ -3061,31 +3080,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) } } -if (!cs->sum_migrate_dl_bw) - goto out_success; - - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) { - ret = -EINVAL ; - goto out_unlock; - } - - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) - goto out_unlock; - - cs->dl_bw_cpu = cpu; - -out_success: -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ - cs->attach_in_progress++; + ret =cpuset_reserve_dl_bw (cs); out_unlock: -if (ret) +if (ret) {reset_migrate_dl_data (cs); + }else { +/* +* Mark attach is in progress. This makes validate_change() fail +* changes which zero cpus/mems_allowed. +*/ + cs->attach_in_progress++; + } +mutex_unlock (&cpuset_mutex);return ret; }
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerPATCH v5 3/6 cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
In reply to Waiman LongExpand the scope of cpuset_can_attach_check() by including the setting of setsched flag inside cpuset_can_attach_check() with the new @oldcs and @psetsched argument. As cpuset_can_attach_check() is also called from cpuset_can_fork(), set the new arguments to NULL from that caller.
While at it, expose the source and destination cpuset cpu/memory check results in the new attach_cpus_updated and attach_mems_updated static flags so that these flags can be used directly from cpuset_attach() without the need to do the same computations again.
Two new global attach related flags are added (attach_cpus_updated & attach_mems_updated) which are set to indicate that CPUs or memory nodes are updated. These 2 flags are set in cpuset_can_attach() and are used in cpuset_attach() for optimization. Since cpuset_mutex will be released between the 2 calls, it is possible that an intervening cpuset action may change the CPU or node mask of the relevant cpusets, so check is added to set these flags if the effective_cpus or effective_mems of those cpusets is changed.
--- kernel/cgroup/cpuset.c | 52 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 22 deletions(-)
kernel/cgroup/cpuset.c +30 -22
@@ -2982,12 +2982,39 @@ static struct cpuset *cpuset_attach_old_cs;* For v1, cpus_allowed and mems_allowed can't be empty. * For v2, effective_cpus can't be empty. * Note that in v1, effective_cpus = cpus_allowed. +* +* Also set the boolean flag passed in by @psetsched depending on if +* security_task_setscheduler() call is needed and @oldcs is not NULL. */ -static int cpuset_can_attach_check (struct cpuset *cs) +static int cpuset_can_attach_check (struct cpuset *cs,struct cpuset *oldcs, +bool *psetsched) {if (cpumask_empty (cs->effective_cpus) || (!is_in_v2_mode () &&nodes_empty (cs->mems_allowed)))return -ENOSPC ; + +if (!oldcs) +return 0 ; + +/* +* Skip rights over task setsched check in v2 when nothing changes, +* migration permission derives from hierarchy ownership in +* cgroup_procs_write_permission()). +*/ + *psetsched = !cpuset_v2 () || + !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || + !nodes_equal (cs->effective_mems, oldcs->effective_mems); + +/* +* A v1 cpuset with tasks will have no CPU left only when CPU hotplug +* brings the last online CPU offline as users are not allowed to empty +* cpuset.cpus when there are active tasks inside. When that happens, +* we should allow tasks to migrate out without security check to make +* sure they will be able to run after migration. +*/ +if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) + *psetsched = false; +return 0 ; }
@@ -3034,29 +3061,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* Skip rights over task setsched check in v2 when nothing changes, -* migration permission derives from hierarchy ownership in -* cgroup_procs_write_permission()). -*/ - setsched_check = !cpuset_v2 () || - !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || - !nodes_equal (cs->effective_mems, oldcs->effective_mems); - -/* -* A v1 cpuset with tasks will have no CPU left only when CPU hotplug -* brings the last online CPU offline as users are not allowed to empty -* cpuset.cpus when there are active tasks inside. When that happens, -* we should allow tasks to migrate out without security check to make -* sure they will be able to run after migration. -*/ -if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) - setsched_check = false; -cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret)
@@ -3601,7 +3609,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs,NULL ,NULL );if (ret) goto out_unlock;
Signature · 1 line
2.54.0
Re: [PATCH-next v5 3/6] cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
Quoted 5 lines · Waiman Long
Expand the scope of
cpuset_can_attach_check()by including the setting of setsched flag insidecpuset_can_attach_check()with the new @oldcs and @psetsched argument. Ascpuset_can_attach_check()is also called fromcpuset_can_fork(), set the new arguments to NULL from that caller.
Hi Waiman,
The code change itself looks good to me. However, the commit message has two paragraphs that don't match this patch:
Quoted 14 lines
While at it, expose the source and destination cpuset cpu/memory check results in the new
attach_cpus_updatedandattach_mems_updatedstatic flags so that these flags can be used directly fromcpuset_attach()without the need to do the same computations again.Two new global attach related flags are added (
attach_cpus_updated&attach_mems_updated) which are set to indicate that CPUs or memory nodes are updated. These 2 flags are set incpuset_can_attach()and are used incpuset_attach()for optimization. Sincecpuset_mutexwill be released between the 2 calls, it is possible that an intervening cpuset action may change the CPU or node mask of the relevant cpusets, so check is added to set these flags if theeffective_cpusoreffective_memsof those cpusets is changed.
Quoted 1 line
Signed-off-by Waiman Long <[email protected]>
Other than that:
Quoted 89 lines
--- kernel/cgroup/cpuset.c | 52 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 22 deletions(-)kernel/cgroup/cpuset.c +30
-22@@ -2982,12 +2982,39 @@ static struct cpuset *cpuset_attach_old_cs;* For v1, cpus_allowed and mems_allowed can't be empty. * For v2, effective_cpus can't be empty. * Note that in v1, effective_cpus = cpus_allowed. +* +* Also set the boolean flag passed in by @psetsched depending on if +* security_task_setscheduler() call is needed and @oldcs is not NULL. */ -static int cpuset_can_attach_check (struct cpuset *cs) +static int cpuset_can_attach_check (struct cpuset *cs,struct cpuset *oldcs, +bool *psetsched) {if (cpumask_empty (cs->effective_cpus) || (!is_in_v2_mode () &&nodes_empty (cs->mems_allowed)))return -ENOSPC ; + +if (!oldcs) +return 0 ; + +/* +* Skip rights over task setsched check in v2 when nothing changes, +* migration permission derives from hierarchy ownership in +* cgroup_procs_write_permission()). +*/ + *psetsched = !cpuset_v2 () || + !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || + !nodes_equal (cs->effective_mems, oldcs->effective_mems); + +/* +* A v1 cpuset with tasks will have no CPU left only when CPU hotplug +* brings the last online CPU offline as users are not allowed to empty +* cpuset.cpus when there are active tasks inside. When that happens, +* we should allow tasks to migrate out without security check to make +* sure they will be able to run after migration. +*/ +if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) + *psetsched = false; +return 0 ; }@@ -3034,29 +3061,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* Skip rights over task setsched check in v2 when nothing changes, -* migration permission derives from hierarchy ownership in -* cgroup_procs_write_permission()). -*/ - setsched_check = !cpuset_v2 () || - !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || - !nodes_equal (cs->effective_mems, oldcs->effective_mems); - -/* -* A v1 cpuset with tasks will have no CPU left only when CPU hotplug -* brings the last online CPU offline as users are not allowed to empty -* cpuset.cpus when there are active tasks inside. When that happens, -* we should allow tasks to migrate out without security check to make -* sure they will be able to run after migration. -*/ -if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) - setsched_check = false; -cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret)@@ -3601,7 +3609,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs,NULL ,NULL );if (ret) goto out_unlock;
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerPATCH v5 4/6 cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders
In reply to Waiman LongThere are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.
IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach(), but fall back to that of the first task if there is no group leader in the taskset.
--- kernel/cgroup/cpuset.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
kernel/cgroup/cpuset.c +25 -0
@@ -2975,6 +2975,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)return 0 ; } +/* +* cpuset_can_attach() and cpuset_attach() specific internal data +* Protected by cpuset_mutex +*/ static struct cpuset *cpuset_attach_old_cs;/*
@@ -3065,11 +3069,32 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)if (ret) goto out_unlock; +/* +* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get +* the old_mems_allowed value. There are two ways that many-to-one +* cpuset migration can happen: +* 1) A multithread application with threads in different cpusets is +* wholely migrated to a new cpuset. +* 2) Disabling v2 cpuset controller will move all the tasks in child +* cpusets to the parent cpuset. +* +* In the former case, it is the mm setting of the group leader that +* really matters. So cpuset_attach_old_cs should track the oldcs of the +* group leader. It falls back to the oldcs of the first task if there +* is no group leader in the taskset. In the latter case, effective_mems +* of child cpusets must always be a subset of the parent. So no real +* page migration will be necessary no matter which child cpuset is +* selected as cpuset_attach_old_cs. +*/ cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret) goto out_unlock; +/* Update cpuset_attach_old_cs to the latest group leader */ +if (task == task->group_leader) + cpuset_attach_old_cs =task_cs (task); +if (setsched_check) { ret =security_task_setscheduler (task);if (ret)
Signature · 1 line
2.54.0
Re: [PATCH-next v5 4/6] cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders
Quoted 3 lines · Waiman Long
There are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset
^ wholly
Quoted 9 lines
or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really matters. So
cpuset_attach_old_csshould track the oldcs of the thread leader. In the latter case,effective_memsof child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected ascpuset_attach_old_cs.
Quoted 6 lines
IOW,
cpuset_attach_old_csshould be updated to match the latest task group leader incpuset_can_attach(), but fall back to that of the first task if there is no group leader in the taskset.Suggested-by Ridong Chen <[email protected]>Signed-off-by Waiman Long <[email protected]>
Quoted 52 lines
--- kernel/cgroup/cpuset.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)kernel/cgroup/cpuset.c +25
-0@@ -2975,6 +2975,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)return 0 ; } +/* +* cpuset_can_attach() and cpuset_attach() specific internal data +* Protected by cpuset_mutex +*/ static struct cpuset *cpuset_attach_old_cs;/* @@ -3065,11 +3069,32 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)if (ret) goto out_unlock; +/* +* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get +* the old_mems_allowed value. There are two ways that many-to-one +* cpuset migration can happen: +* 1) A multithread application with threads in different cpusets is +* wholely migrated to a new cpuset. +* 2) Disabling v2 cpuset controller will move all the tasks in child +* cpusets to the parent cpuset. +* +* In the former case, it is the mm setting of the group leader that +* really matters. So cpuset_attach_old_cs should track the oldcs of the +* group leader. It falls back to the oldcs of the first task if there +* is no group leader in the taskset. In the latter case, effective_mems +* of child cpusets must always be a subset of the parent. So no real +* page migration will be necessary no matter which child cpuset is +* selected as cpuset_attach_old_cs. +*/ cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret) goto out_unlock; +/* Update cpuset_attach_old_cs to the latest group leader */ +if (task == task->group_leader) + cpuset_attach_old_cs =task_cs (task); +if (setsched_check) { ret =security_task_setscheduler (task);if (ret)
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerPATCH v5 5/6 cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
In reply to Waiman LongThe cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work
--- kernel/cgroup/cpuset.c | 103 ++++++++++++++++++++++++----------------- 1 file changed, 60 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +60 -43
@@ -2978,8 +2978,13 @@ static int update_prstate(struct cpuset *cs, int new_prs)/* * cpuset_can_attach() and cpuset_attach() specific internal data * Protected by cpuset_mutex +* +* The attach_cpus_updated/attach_mems_updated flags are set in either +* cpuset_attach() or cpuset_fork() and used in cpuset_attach_task(). */ static struct cpuset *cpuset_attach_old_cs; +static bool attach_cpus_updated; +static bool attach_mems_updated;/* * Check to see if a cpuset can accept a new task
@@ -3157,9 +3162,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)*/ static cpumask_var_t cpus_attach;static nodemask_t cpuset_attach_nodemask_to; +static bool queue_task_work;static void cpuset_attach_task (struct cpuset *cs,struct task_struct *task) { +struct mm_struct *mm; +lockdep_assert_cpuset_lock_held ();if (cs != &top_cpuset)
@@ -3173,28 +3181,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)*/ WARN_ON_ONCE (set_cpus_allowed_ptr (task, cpus_attach)); +if (cpuset_v2 () && !attach_mems_updated) +return ; +cpuset_change_task_nodemask (task, &cpuset_attach_nodemask_to);cpuset1_update_task_spread_flags (cs, task); + +if ((task != task->group_leader) || + (!is_memory_migrate (cs) && !attach_mems_updated)) +return ; + +/* +* Change mm for threadgroup leader. This is expensive and may +* sleep and should be moved outside migration path proper. +*/ + mm =get_task_mm (task); +if (mm) { +struct cpuset *oldcs = cpuset_attach_old_cs; + +mpol_rebind_mm (mm, &cs->effective_mems); + +/* +* old_mems_allowed is the same with mems_allowed +* here, except if this task is being moved +* automatically due to hotplug. In that case +* @mems_allowed has been updated and is empty, so +* @old_mems_allowed is the right nodesets that we +* migrate mm from. +*/ +if (is_memory_migrate (cs)) { +cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, + &cpuset_attach_nodemask_to); + queue_task_work = true; + }else { +mmput (mm); + } + } }static void cpuset_attach (struct cgroup_taskset *tset) {struct task_struct *task; -struct task_struct *leader;struct cgroup_subsys_state *css;struct cpuset *cs;struct cpuset *oldcs = cpuset_attach_old_cs; -bool cpus_updated, mems_updated; -bool queue_task_work = false;cgroup_taskset_first (tset, &css); cs =css_cs (css);lockdep_assert_cpus_held ();/* see cgroup_attach_lock() */ mutex_lock (&cpuset_mutex); - cpus_updated = !cpumask_equal (cs->effective_cpus, - oldcs->effective_cpus); - mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems); + queue_task_work = false; + + attach_cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); + attach_mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems);/* * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3204,7 +3244,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)*/ if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; -if (!cpus_updated && !mems_updated) +if (!attach_cpus_updated && !attach_mems_updated) goto out; }else {guarantee_online_mems (cs, &cpuset_attach_nodemask_to);
@@ -3213,38 +3253,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task); -/* -* Change mm for all threadgroup leaders. This is expensive and may -* sleep and should be moved outside migration path proper. Skip it -* if there is no change in effective_mems and CS_MEMORY_MIGRATE is -* not set. -*/ -if (!is_memory_migrate (cs) && !mems_updated) - goto out; - -cgroup_taskset_for_each_leader (leader, css, tset) { -struct mm_struct *mm =get_task_mm (leader); - -if (mm) { -mpol_rebind_mm (mm, &cs->effective_mems); - -/* -* old_mems_allowed is the same with mems_allowed -* here, except if this task is being moved -* automatically due to hotplug. In that case -* @mems_allowed has been updated and is empty, so -* @old_mems_allowed is the right nodesets that we -* migrate mm from. -*/ -if (is_memory_migrate (cs)) { -cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, - &cpuset_attach_nodemask_to); - queue_task_work = true; - }else -mmput (mm); - } - } - out:if (queue_task_work)schedule_flush_migrate_mm ();
@@ -3678,15 +3686,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)*/ static void cpuset_fork (struct task_struct *task) { -struct cpuset *cs; -bool same_cs; +struct cpuset *cs, *oldcs;rcu_read_lock (); cs =task_cs (task); - same_cs = (cs ==task_cs (current)); + oldcs =task_cs (current);rcu_read_unlock (); -if (same_cs) { +if (cs == oldcs) {if (cs == &top_cpuset)return ;
@@ -3698,7 +3705,17 @@ static void cpuset_fork(struct task_struct *task)/* CLONE_INTO_CGROUP */ mutex_lock (&cpuset_mutex);guarantee_online_mems (cs, &cpuset_attach_nodemask_to); +/* +* Assume CPUs and memory nodes are updated +* A CLONE_INTO_CGROUP operation should have taken the cgroup mutex +* and so there shouldn't be a competing cpuset_attach() operation. +*/ + attach_cpus_updated = attach_mems_updated = true; + queue_task_work = false; + cpuset_attach_old_cs = oldcs;cpuset_attach_task (cs, task); +if (queue_task_work) +schedule_flush_migrate_mm ();dec_attach_in_progress_locked (cs);mutex_unlock (&cpuset_mutex);
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v5 6/6 cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
In reply to Waiman LongWith cgroup v2, the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple destination or source cpusets when the cpuset controller is enabled or disabled respectively.
Since cpuset is threaded in both v1 and v2, another possible way to cause many-to-one migration is to move the whole process with multiple threads in different cpuset enabled threaded cgroups into another cpuset enabled cgroup.
The current cpuset_can_attach() and cpuset_attach() functions still expect task migration is from one source cpuset to one destination cpuset. This has been the case since cpuset was enabled for cgroup v2 in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy").
This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.
Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both the source and destination cpusets are decremented/incremented with their values added to nr_deadline_tasks when the migration is successful.
--- kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 208 ++++++++++++++++++++++++-------- 2 files changed, 164 insertions(+), 50 deletions(-)
kernel/cgroup/cpuset-internal.h +6 -0
@@ -161,6 +161,12 @@ struct cpuset {*/ bool remote_partition; +/* +* cpuset_can_attach() and cpuset_attach() specific data +*/ +bool attach_node_in_llist; +struct llist_node attach_node; +/* * number of SCHED_DEADLINE tasks attached to this cpuset, so that we * know when to rebuild associated root domain bandwidth information.
kernel/cgroup/cpuset.c +158 -50
300 lines not on this page · raw message
Signature · 1 line
2.54.0
Ridong Chen
reviewerPATCH 1/1 cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
The current cpuset_can_attach() and cpuset_attach() functions assume task migration is from one source cpuset to one destination cpuset. This can be wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)
Fix this by adopting the pids subsystem's per-task accounting pattern. In cpuset_can_attach(), use task_cs(task) to get the correct source cpuset for each task (like pids_can_attach uses task_css), adjust nr_deadline_tasks and reserve DL bandwidth per-task, and increment attach_in_progress per-task on the destination cpuset. In cpuset_attach(), handle destination cpuset changes within the task iteration loop.
A shared helper cpuset_undo_attach() reverses the per-task operations for both partial rollback in cpuset_can_attach() and full reversal in cpuset_cancel_attach().
When multiple source cpusets are detected in can_attach(), set attach_many_sources so that cpuset_attach() forces cpus_updated and mems_updated to true, ensuring all tasks get properly updated regardless of which source cpuset cpuset_attach_old_cs points to.
This eliminates the need for nr_migrate_dl_tasks, sum_migrate_dl_bw, and dl_bw_cpu fields in struct cpuset.
--- kernel/cgroup/cpuset-internal.h | 8 -- kernel/cgroup/cpuset.c | 177 ++++++++++++++++---------------- 2 files changed, 89 insertions(+), 96 deletions(-)
kernel/cgroup/cpuset-internal.h +0 -8
@@ -166,14 +166,6 @@ struct cpuset {* know when to rebuild associated root domain bandwidth information. */ int nr_deadline_tasks; -int nr_migrate_dl_tasks; -/* DL bandwidth that needs destination reservation for this attach. */ -u64 sum_migrate_dl_bw; -/* -* CPU used for temporary DL bandwidth allocation during attach; -* -1 if no DL bandwidth was allocated in the current attach. -*/ -int dl_bw_cpu;/* Invalid partition error code, not lock protected */ enum prs_errcode prs_err;
kernel/cgroup/cpuset.c +89 -88
265 lines not on this page · raw message
Signature · 1 line
2.43.0
Ridong Chen
reviewerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
Hi Longman,
I used AI to generate a patch that fixes this issue, following the same approach as the pids subsystem. I think this may be much simpler. Just a heads-up — this patch is only for discussion and hasn't been tested.
Quoted 328 lines · Ridong Chen
The current
cpuset_can_attach()andcpuset_attach()functions assume task migration is from one source cpuset to one destination cpuset. This can be wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)Fix this by adopting the pids subsystem's per-task accounting pattern. In
cpuset_can_attach(), usetask_cs(task)to get the correct source cpuset for each task (likepids_can_attachusestask_css), adjustnr_deadline_tasksand reserve DL bandwidth per-task, and incrementattach_in_progressper-task on the destination cpuset. Incpuset_attach(), handle destination cpuset changes within the task iteration loop.A shared helper
cpuset_undo_attach()reverses the per-task operations for both partial rollback incpuset_can_attach()and full reversal incpuset_cancel_attach().When multiple source cpusets are detected in
can_attach(), setattach_many_sourcesso thatcpuset_attach()forcescpus_updatedandmems_updatedto true, ensuring all tasks get properly updated regardless of which source cpusetcpuset_attach_old_cspoints to.This eliminates the need for
nr_migrate_dl_tasks,sum_migrate_dl_bw, anddl_bw_cpufields in struct cpuset.Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy")Signed-off-by Ridong Chen <[email protected]>--- kernel/cgroup/cpuset-internal.h | 8 -- kernel/cgroup/cpuset.c | 177 ++++++++++++++++---------------- 2 files changed, 89 insertions(+), 96 deletions(-)kernel/cgroup/cpuset-internal.h +0
-8@@ -166,14 +166,6 @@ struct cpuset {* know when to rebuild associated root domain bandwidth information. */ int nr_deadline_tasks; -int nr_migrate_dl_tasks; -/* DL bandwidth that needs destination reservation for this attach. */ -u64 sum_migrate_dl_bw; -/* -* CPU used for temporary DL bandwidth allocation during attach; -* -1 if no DL bandwidth was allocated in the current attach. -*/ -int dl_bw_cpu;/* Invalid partition error code, not lock protected */ enum prs_errcode prs_err;kernel/cgroup/cpuset.c +89
-88@@ -288,7 +288,6 @@ struct cpuset top_cpuset = { .flags = BIT(CS_CPU_EXCLUSIVE) | BIT(CS_MEM_EXCLUSIVE) | BIT(CS_SCHED_LOAD_BALANCE), .partition_root_state = PRS_ROOT, - .dl_bw_cpu = -1, };/** @@ -580,8 +579,6 @@ static struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)if (!trial)return NULL ; - trial->dl_bw_cpu =-1 ; -/* Setup cpumask pointer array */ cpumask_var_t *pmask[4 ] = { &trial->cpus_allowed,@@ -2984,6 +2981,7 @@ static int update_prstate(struct cpuset *cs, int new_prs)static struct cpuset *cpuset_attach_old_cs;static bool attach_cpus_updated;static bool attach_mems_updated; +static bool attach_many_sources;/* * Check to see if a cpuset can accept a new task @@ -3026,30 +3024,36 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,return 0 ; } -static int cpuset_reserve_dl_bw (struct cpuset *cs) +/* +* Reverse per-task operations done in cpuset_can_attach(). +* If @stop_at is non-NULL, only undo tasks before it (partial rollback). +* If @stop_at is NULL, undo all tasks (full reversal for cancel_attach). +* Must be called with cpuset_mutex held. +*/ +static void cpuset_undo_attach (struct cgroup_taskset *tset, +struct task_struct *stop_at) { -int cpu, ret; - -if (!cs->sum_migrate_dl_bw) -return 0 ; - - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) -return -EINVAL ; +struct cgroup_subsys_state *css; +struct task_struct *task; - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) -return ret; +cgroup_taskset_for_each (task, css, tset) { +struct cpuset *cs =css_cs (css); +struct cpuset *oldcs =task_cs (task); - cs->dl_bw_cpu = cpu; -return 0 ; -} +if (task == stop_at) +break ; -static void reset_migrate_dl_data (struct cpuset *cs) -{ - cs->nr_migrate_dl_tasks =0 ; - cs->sum_migrate_dl_bw =0 ; - cs->dl_bw_cpu =-1 ; +if (dl_task (task)) { + cs->nr_deadline_tasks--; + oldcs->nr_deadline_tasks++; +if (dl_task_needs_bw_move (task, cs->effective_cpus)) { +int cpu =cpumask_any_and (cpu_active_mask, + cs->effective_cpus); +dl_bw_free (cpu, task->dl.dl_bw); + } + } +dec_attach_in_progress_locked (cs); + } }/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */ @@ -3061,96 +3065,79 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)bool setsched_check;int ret; -/* used later by cpuset_attach() */ cpuset_attach_old_cs =task_cs (cgroup_taskset_first (tset, &css)); oldcs = cpuset_attach_old_cs; cs =css_cs (css);mutex_lock (&cpuset_mutex); + attach_many_sources = false; -/* Check to see if task is allowed in the cpuset */ ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get -* the old_mems_allowed value. There are two ways that many-to-one -* cpuset migration can happen: -* 1) A multithread application with threads in different cpusets is -* wholely migrated to a new cpuset. -* 2) Disabling v2 cpuset controller will move all the tasks in child -* cpusets to the parent cpuset. -* -* In the former case, it is the mm setting of the group leader that -* really matters. So cpuset_attach_old_cs should track the oldcs of the -* group leader. It falls back to the oldcs of the first task if there -* is no group leader in the taskset. In the latter case, effective_mems -* of child cpusets must always be a subset of the parent. So no real -* page migration will be necessary no matter which child cpuset is -* selected as cpuset_attach_old_cs. -*/ cgroup_taskset_for_each (task, css, tset) { +struct cpuset *newcs =css_cs (css); +struct cpuset *new_oldcs =task_cs (task); + +if (newcs != cs || new_oldcs != oldcs) { +if (new_oldcs != oldcs) + attach_many_sources = true; + cs = newcs; + oldcs = new_oldcs; + ret =cpuset_can_attach_check (cs, oldcs, + &setsched_check); +if (ret) + goto out_rollback; + } + ret =task_can_attach (task);if (ret) - goto out_unlock; + goto out_rollback; -/* Update cpuset_attach_old_cs to the latest group leader */ if (task == task->group_leader) cpuset_attach_old_cs =task_cs (task);if (setsched_check) { ret =security_task_setscheduler (task);if (ret) - goto out_unlock; + goto out_rollback; }if (dl_task (task)) { -/* -* Count all migrating DL tasks for cpuset task accounting. -* Only tasks that need a root-domain bandwidth move -* contribute to sum_migrate_dl_bw. -*/ - cs->nr_migrate_dl_tasks++; -if (dl_task_needs_bw_move (task, cs->effective_cpus)) - cs->sum_migrate_dl_bw += task->dl.dl_bw; + cs->nr_deadline_tasks++; + oldcs->nr_deadline_tasks--; + +if (dl_task_needs_bw_move (task, cs->effective_cpus)) { +int cpu =cpumask_any_and (cpu_active_mask, + cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) { + ret = -EINVAL ; + goto out_rollback; + } + ret =dl_bw_alloc (cpu, task->dl.dl_bw); +if (ret) + goto out_rollback; + } } - } - - ret =cpuset_reserve_dl_bw (cs); -out_unlock: -if (ret) { -reset_migrate_dl_data (cs); - }else { -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ cs->attach_in_progress++; } + goto out_unlock; + +out_rollback: +cpuset_undo_attach (tset, task); + +out_unlock:mutex_unlock (&cpuset_mutex);return ret; }static void cpuset_cancel_attach (struct cgroup_taskset *tset) { -struct cgroup_subsys_state *css; -struct cpuset *cs; - -cgroup_taskset_first (tset, &css); - cs =css_cs (css); -mutex_lock (&cpuset_mutex); -dec_attach_in_progress_locked (cs); - -if (cs->dl_bw_cpu >=0 ) -dl_bw_free (cs->dl_bw_cpu, cs->sum_migrate_dl_bw); - -if (cs->nr_migrate_dl_tasks) -reset_migrate_dl_data (cs); - +cpuset_undo_attach (tset,NULL );mutex_unlock (&cpuset_mutex); }@@ -3232,8 +3219,15 @@ static void cpuset_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex); queue_task_work = false; - attach_cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); - attach_mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems); +if (attach_many_sources) { + attach_cpus_updated = true; + attach_mems_updated = true; + }else { + attach_cpus_updated = !cpumask_equal (cs->effective_cpus, + oldcs->effective_cpus); + attach_mems_updated = !nodes_equal (cs->effective_mems, + oldcs->effective_mems); + }/* * In the default hierarchy, enabling cpuset in the child cgroups @@ -3249,21 +3243,28 @@ static void cpuset_attach(struct cgroup_taskset *tset)guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -cgroup_taskset_for_each (task, css, tset) +cgroup_taskset_for_each (task, css, tset) { +struct cpuset *newcs =css_cs (css); + +if (newcs != cs) { + cs->old_mems_allowed = cpuset_attach_nodemask_to; + cs = newcs; +if (cpuset_v2 ()) + cpuset_attach_nodemask_to = cs->effective_mems; +else +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); + }cpuset_attach_task (cs, task); + } out:if (queue_task_work)schedule_flush_migrate_mm (); cs->old_mems_allowed = cpuset_attach_nodemask_to; -if (cs->nr_migrate_dl_tasks) { - cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks; - oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks; -reset_migrate_dl_data (cs); - } - -dec_attach_in_progress_locked (cs); +/* Decrement per-task attach_in_progress */ +cgroup_taskset_for_each (task, css, tset) +dec_attach_in_progress_locked (css_cs (css));mutex_unlock (&cpuset_mutex); }
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
In reply to Ridong ChenQuoted 23 lines · Ridong Chen
The current
cpuset_can_attach()andcpuset_attach()functions assume task migration is from one source cpuset to one destination cpuset. This can be wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)Fix this by adopting the pids subsystem's per-task accounting pattern. In
cpuset_can_attach(), usetask_cs(task)to get the correct source cpuset for each task (likepids_can_attachusestask_css), adjustnr_deadline_tasksand reserve DL bandwidth per-task, and incrementattach_in_progressper-task on the destination cpuset. Incpuset_attach(), handle destination cpuset changes within the task iteration loop.A shared helper
cpuset_undo_attach()reverses the per-task operations for both partial rollback incpuset_can_attach()and full reversal incpuset_cancel_attach().When multiple source cpusets are detected in
can_attach(), setattach_many_sourcesso thatcpuset_attach()forcescpus_updatedandmems_updatedto true, ensuring all tasks get properly updated regardless of which source cpusetcpuset_attach_old_cspoints to.
Quoted 5 lines
This eliminates the need for
nr_migrate_dl_tasks,sum_migrate_dl_bw, anddl_bw_cpufields in struct cpuset.Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy")Signed-off-by Ridong Chen <[email protected]>
It is not a problem doing per-task DL BW allocation and eliminating the *dl_bw* fields. However, updating nr_deadline_tasks before it is committed can be problematic.
nr_deadline_tasks is used in dl_rebuild_rd_accounting() which is called by partition_sched_domains_locked(). After the release of cpuset_mutex at the end of cpuset_can_attach() and before cpuset_attach() or cpuset_cancel_attach() is called, it is possible that partition_sched_domains_locked() can be called and dl_rebuild_rd_accounting() is not getting the right DL BW accounting information. So unless there is a way to confirm that this situation cannot happen, we can't change nr_deadline_tasks before the attach is commited.
Cheers,
Longman
Ridong Chen
reviewerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
Quoted 31 lines · Waiman Long
Quoted 30 lines · Ridong Chen
The current
cpuset_can_attach()andcpuset_attach()functions assume task migration is from one source cpuset to one destination cpuset. This can be
wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)Fix this by adopting the pids subsystem's per-task accounting pattern. In
cpuset_can_attach(), usetask_cs(task)to get the correct source cpuset
for each task (likepids_can_attachusestask_css), adjustnr_deadline_tasks
and reserve DL bandwidth per-task, and incrementattach_in_progressper-task
on the destination cpuset. Incpuset_attach(), handle destination cpuset changes within the task iteration loop.A shared helper
cpuset_undo_attach()reverses the per-task operations for both partial rollback incpuset_can_attach()and full reversal incpuset_cancel_attach().When multiple source cpusets are detected in
can_attach(), setattach_many_sourcesso thatcpuset_attach()forcescpus_updatedandmems_updatedto true, ensuring all tasks get properly updated regardless of which source cpusetcpuset_attach_old_cspoints to.This eliminates the need for
nr_migrate_dl_tasks,sum_migrate_dl_bw, anddl_bw_cpufields in struct cpuset.
Quoted 8 lines
Quoted 3 lines
Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in defaulthierarchy")
Signed-off-by Ridong Chen <[email protected]>It is not a problem doing per-task DL BW allocation and eliminating the *dl_bw* fields. However, updating
nr_deadline_tasksbefore it is committed can be problematic.
Good to hear that.
Quoted 10 lines
nr_deadline_tasksis used indl_rebuild_rd_accounting()which is called bypartition_sched_domains_locked(). After the release ofcpuset_mutexat the end ofcpuset_can_attach()and beforecpuset_attach()orcpuset_cancel_attach()is called, it is possible
that partition_sched_domains_locked() can be called
and dl_rebuild_rd_accounting() is not getting the right DL BW accounting information. So unless there is a way to confirm that this situation cannot happen, we can't change nr_deadline_tasks before the attach is commited.
We can keep the nr_migrate_dl_tasks field and update nr_deadline_tasks once migration is complete. I think this will be much simpler than fixing the issue using lists.
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
Quoted 52 lines · Ridong Chen
Quoted 31 lines · Waiman Long
Quoted 30 lines · Ridong Chen
The current
cpuset_can_attach()andcpuset_attach()functions assume task migration is from one source cpuset to one destination cpuset. This can be
wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)Fix this by adopting the pids subsystem's per-task accounting pattern. In
cpuset_can_attach(), usetask_cs(task)to get the correct source cpuset
for each task (likepids_can_attachusestask_css), adjustnr_deadline_tasks
and reserve DL bandwidth per-task, and incrementattach_in_progressper-task
on the destination cpuset. Incpuset_attach(), handle destination cpuset changes within the task iteration loop.A shared helper
cpuset_undo_attach()reverses the per-task operations for both partial rollback incpuset_can_attach()and full reversal incpuset_cancel_attach().When multiple source cpusets are detected in
can_attach(), setattach_many_sourcesso thatcpuset_attach()forcescpus_updatedandmems_updatedto true, ensuring all tasks get properly updated regardless of which source cpusetcpuset_attach_old_cspoints to.This eliminates the need for
nr_migrate_dl_tasks,sum_migrate_dl_bw, anddl_bw_cpufields in struct cpuset.Quoted 7 lines
Quoted 3 lines
Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in defaulthierarchy")
Signed-off-by Ridong Chen <[email protected]>It is not a problem doing per-task DL BW allocation and eliminating the *dl_bw* fields. However, updating
nr_deadline_tasksbefore it is committed can be problematic.Good to hear that.
Quoted 10 lines
nr_deadline_tasksis used indl_rebuild_rd_accounting()which is called bypartition_sched_domains_locked(). After the release ofcpuset_mutexat the end ofcpuset_can_attach()and beforecpuset_attach()orcpuset_cancel_attach()is called, it is possible
that partition_sched_domains_locked() can be called
and dl_rebuild_rd_accounting() is not getting the right DL BW accounting information. So unless there is a way to confirm that this situation cannot happen, we can't change nr_deadline_tasks before the attach is commited.
Quoted 4 lines
We can keep the
nr_migrate_dl_tasksfield and updatenr_deadline_tasksonce migration is complete. I think this will be much simpler than fixing the issue using lists.
But we still need to track the set of source and destination cpusets to commit or cancel the change. Doing it task-by-task will add code in the cpuset_attach() and cpuset_cancel_attach() to check if a task is a DL task and act accordingly. So we are just trading task-by-task code with code to handle the lists.
Cheers,
Longman
Ridong Chen
reviewerPATCH 1/1 cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
When the cpuset controller is enabled/disabled in a parent cgroup, tasks from multiple child cpusets need to be migrated. The current code only handles a single source/destination pair.
Support multiple source/destination cpusets by adopting the per-task processing pattern similar to the pids controller:
1) Perform per-task DL bandwidth reservation (dl_bw_alloc) directly in cpuset_can_attach() instead of batching into sum_migrate_dl_bw. This eliminates the sum_migrate_dl_bw and dl_bw_cpu fields from the cpuset struct.
2) Track attach_in_progress per-task per-destination cpuset to properly guard all involved cpusets from having their cpus/mems zeroed.
3) Use a shared cpuset_undo_attach() helper for both rollback-on-error in cpuset_can_attach() and for cpuset_cancel_attach().
4) Detect many-source migrations and force cpus_updated/mems_updated to true so all tasks get properly updated during attach.
5) Defer nr_deadline_tasks updates to cpuset_attach() (after migration is committed) to avoid a race with dl_rebuild_rd_accounting() that could see inconsistent values between can_attach and attach.
--- kernel/cgroup/cpuset-internal.h | 7 -- kernel/cgroup/cpuset.c | 167 ++++++++++++++++---------------- 2 files changed, 84 insertions(+), 90 deletions(-)
kernel/cgroup/cpuset-internal.h +0 -7
@@ -167,13 +167,6 @@ struct cpuset {*/ int nr_deadline_tasks;int nr_migrate_dl_tasks; -/* DL bandwidth that needs destination reservation for this attach. */ -u64 sum_migrate_dl_bw; -/* -* CPU used for temporary DL bandwidth allocation during attach; -* -1 if no DL bandwidth was allocated in the current attach. -*/ -int dl_bw_cpu;/* Invalid partition error code, not lock protected */ enum prs_errcode prs_err;
kernel/cgroup/cpuset.c +84 -83
248 lines not on this page · raw message
Signature · 1 line
2.43.0
Waiman Long
maintainerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
Quoted 294 lines · Ridong Chen
When the cpuset controller is enabled/disabled in a parent cgroup, tasks from multiple child cpusets need to be migrated. The current code only handles a single source/destination pair.
Support multiple source/destination cpusets by adopting the per-task processing pattern similar to the pids controller:
1) Perform per-task DL bandwidth reservation (
dl_bw_alloc) directly incpuset_can_attach()instead of batching intosum_migrate_dl_bw. This eliminates thesum_migrate_dl_bwanddl_bw_cpufields from the cpuset struct.2) Track
attach_in_progressper-task per-destination cpuset to properly guard all involved cpusets from having their cpus/mems zeroed.3) Use a shared
cpuset_undo_attach()helper for both rollback-on-error incpuset_can_attach()and forcpuset_cancel_attach().4) Detect many-source migrations and force cpus_updated/mems_updated to true so all tasks get properly updated during attach.
5) Defer
nr_deadline_tasksupdates tocpuset_attach()(after migration is committed) to avoid a race withdl_rebuild_rd_accounting()that could see inconsistent values betweencan_attachand attach.Signed-off-by Ridong Chen <[email protected]>--- kernel/cgroup/cpuset-internal.h | 7 -- kernel/cgroup/cpuset.c | 167 ++++++++++++++++---------------- 2 files changed, 84 insertions(+), 90 deletions(-)kernel/cgroup/cpuset-internal.h +0
-7@@ -167,13 +167,6 @@ struct cpuset {*/ int nr_deadline_tasks;int nr_migrate_dl_tasks; -/* DL bandwidth that needs destination reservation for this attach. */ -u64 sum_migrate_dl_bw; -/* -* CPU used for temporary DL bandwidth allocation during attach; -* -1 if no DL bandwidth was allocated in the current attach. -*/ -int dl_bw_cpu;/* Invalid partition error code, not lock protected */ enum prs_errcode prs_err;kernel/cgroup/cpuset.c +76
-82@@ -288,7 +288,6 @@ struct cpuset top_cpuset = { .flags = BIT(CS_CPU_EXCLUSIVE) | BIT(CS_MEM_EXCLUSIVE) | BIT(CS_SCHED_LOAD_BALANCE), .partition_root_state = PRS_ROOT, - .dl_bw_cpu = -1, };/** @@ -580,8 +579,6 @@ static struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs)if (!trial)return NULL ; - trial->dl_bw_cpu =-1 ; -/* Setup cpumask pointer array */ cpumask_var_t *pmask[4 ] = { &trial->cpus_allowed,@@ -3026,31 +3023,36 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,return 0 ; } -static int cpuset_reserve_dl_bw (struct cpuset *cs) +/* +* Undo DL bandwidth reservations and attach_in_progress increments done +* in cpuset_can_attach(). Used for both rollback on error and cancel_attach. +* If @stop_at is non-NULL, undo only for tasks before @stop_at in the tset. +*/ +static void cpuset_undo_attach (struct cgroup_taskset *tset, +struct task_struct *stop_at) { -int cpu, ret; - -if (!cs->sum_migrate_dl_bw) -return 0 ; +struct cgroup_subsys_state *css; +struct task_struct *task; - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) -return -EINVAL ; +cgroup_taskset_for_each (task, css, tset) { +struct cpuset *cs =css_cs (css); - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) -return ret; +if (task == stop_at) +break ; - cs->dl_bw_cpu = cpu; -return 0 ; +if (dl_task (task)) { + cs->nr_migrate_dl_tasks--; +if (dl_task_needs_bw_move (task, cs->effective_cpus)) { +int cpu =cpumask_any_and (cpu_active_mask, + cs->effective_cpus); +dl_bw_free (cpu, task->dl.dl_bw); + } + } +dec_attach_in_progress_locked (cs); + } } -static void reset_migrate_dl_data (struct cpuset *cs) -{ - cs->nr_migrate_dl_tasks =0 ; - cs->sum_migrate_dl_bw =0 ; - cs->dl_bw_cpu =-1 ; -} +static bool attach_many_sources;/* Called by cgroups to determine if a cpuset is usable; cpuset_mutex held */ static int cpuset_can_attach (struct cgroup_taskset *tset)@@ -3067,90 +3069,73 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) cs =css_cs (css);mutex_lock (&cpuset_mutex); + attach_many_sources = false;/* Check to see if task is allowed in the cpuset */ ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get -* the old_mems_allowed value. There are two ways that many-to-one -* cpuset migration can happen: -* 1) A multithread application with threads in different cpusets is -* wholely migrated to a new cpuset. -* 2) Disabling v2 cpuset controller will move all the tasks in child -* cpusets to the parent cpuset. -* -* In the former case, it is the mm setting of the group leader that -* really matters. So cpuset_attach_old_cs should track the oldcs of the -* group leader. It falls back to the oldcs of the first task if there -* is no group leader in the taskset. In the latter case, effective_mems -* of child cpusets must always be a subset of the parent. So no real -* page migration will be necessary no matter which child cpuset is -* selected as cpuset_attach_old_cs. -*/ cgroup_taskset_for_each (task, css, tset) { +struct cpuset *newcs =css_cs (css); +struct cpuset *new_oldcs =task_cs (task); + +if (newcs != cs || new_oldcs != oldcs) { +if (new_oldcs != oldcs) + attach_many_sources = true; + cs = newcs; + oldcs = new_oldcs; + ret =cpuset_can_attach_check (cs, oldcs, + &setsched_check); +if (ret) + goto out_rollback; + } + ret =task_can_attach (task);if (ret) - goto out_unlock; + goto out_rollback; -/* Update cpuset_attach_old_cs to the latest group leader */ if (task == task->group_leader) cpuset_attach_old_cs =task_cs (task);if (setsched_check) { ret =security_task_setscheduler (task);if (ret) - goto out_unlock; + goto out_rollback; }if (dl_task (task)) { -/* -* Count all migrating DL tasks for cpuset task accounting. -* Only tasks that need a root-domain bandwidth move -* contribute to sum_migrate_dl_bw. -*/ cs->nr_migrate_dl_tasks++; -if (dl_task_needs_bw_move (task, cs->effective_cpus)) - cs->sum_migrate_dl_bw += task->dl.dl_bw; +if (dl_task_needs_bw_move (task, cs->effective_cpus)) { +int cpu =cpumask_any_and (cpu_active_mask, + cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) { + ret = -EINVAL ; + goto out_rollback; + } + ret =dl_bw_alloc (cpu, task->dl.dl_bw); +if (ret) + goto out_rollback; + } } - } - - ret =cpuset_reserve_dl_bw (cs); -out_unlock: -if (ret) { -reset_migrate_dl_data (cs); - }else { -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ cs->attach_in_progress++; } + goto out_unlock; + +out_rollback: +cpuset_undo_attach (tset, task); + +out_unlock:mutex_unlock (&cpuset_mutex);return ret; }static void cpuset_cancel_attach (struct cgroup_taskset *tset) { -struct cgroup_subsys_state *css; -struct cpuset *cs; - -cgroup_taskset_first (tset, &css); - cs =css_cs (css); -mutex_lock (&cpuset_mutex); -dec_attach_in_progress_locked (cs); - -if (cs->dl_bw_cpu >=0 ) -dl_bw_free (cs->dl_bw_cpu, cs->sum_migrate_dl_bw); - -if (cs->nr_migrate_dl_tasks) -reset_migrate_dl_data (cs); - +cpuset_undo_attach (tset,NULL );mutex_unlock (&cpuset_mutex); }@@ -3232,8 +3217,15 @@ static void cpuset_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex); queue_task_work = false; - attach_cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); - attach_mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems); +if (attach_many_sources) { + attach_cpus_updated = true; + attach_mems_updated = true; + }else { + attach_cpus_updated = !cpumask_equal (cs->effective_cpus, + oldcs->effective_cpus); + attach_mems_updated = !nodes_equal (cs->effective_mems, + oldcs->effective_mems); + }/* * In the default hierarchy, enabling cpuset in the child cgroups @@ -3250,20 +3242,29 @@ static void cpuset_attach(struct cgroup_taskset *tset) }cgroup_taskset_for_each (task, css, tset) -cpuset_attach_task (cs, task); +cpuset_attach_task (css_cs (css), task); out:if (queue_task_work)schedule_flush_migrate_mm (); cs->old_mems_allowed = cpuset_attach_nodemask_to; -if (cs->nr_migrate_dl_tasks) { - cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks; - oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks; -reset_migrate_dl_data (cs); - } +/* +* Update nr_deadline_tasks now that migration is committed. +* nr_migrate_dl_tasks was accumulated per-dst in can_attach but +* nr_deadline_tasks is deferred to here to avoid a race with +* dl_rebuild_rd_accounting() between can_attach and attach. +*/ +cgroup_taskset_for_each (task, css, tset) {
kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3267
Quoted 10 lines
kernel/cgroup/cpuset.c +8
-1+struct cpuset *dst_cs =css_cs (css); -dec_attach_in_progress_locked (cs); +if (dst_cs->nr_migrate_dl_tasks) { + dst_cs->nr_deadline_tasks += dst_cs->nr_migrate_dl_tasks; + oldcs->nr_deadline_tasks -= dst_cs->nr_migrate_dl_tasks; + dst_cs->nr_migrate_dl_tasks =0 ; + } +dec_attach_in_progress_locked (dst_cs); + }
You are assuming that there is only one source cpuset. That may not be true. I suppose that we may not track the set of destination cpusets and they will be hit during task iteration. We will still need to track all the source cpusets.
Cheers,
Longman
Quoted 3 lines
kernel/cgroup/cpuset.c +0
-0mutex_unlock (&cpuset_mutex); }
Ridong Chen
reviewerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
Quoted 301 lines · Waiman Long
Quoted 288 lines · Ridong Chen
When the cpuset controller is enabled/disabled in a parent cgroup, tasks from multiple child cpusets need to be migrated. The current code only handles a single source/destination pair.
Support multiple source/destination cpusets by adopting the per-task processing pattern similar to the pids controller:
1) Perform per-task DL bandwidth reservation (
dl_bw_alloc) directly incpuset_can_attach()instead of batching intosum_migrate_dl_bw. This eliminates thesum_migrate_dl_bwanddl_bw_cpufields from the cpuset struct.2) Track
attach_in_progressper-task per-destination cpuset to properly guard all involved cpusets from having their cpus/mems zeroed.3) Use a shared
cpuset_undo_attach()helper for both rollback-on-error incpuset_can_attach()and forcpuset_cancel_attach().4) Detect many-source migrations and force cpus_updated/mems_updated to true so all tasks get properly updated during attach.
5) Defer
nr_deadline_tasksupdates tocpuset_attach()(after migration is committed) to avoid a race withdl_rebuild_rd_accounting()that could see inconsistent values betweencan_attachand attach.Signed-off-by Ridong Chen <[email protected]>---
kernel/cgroup/cpuset-internal.h| 7 -- kernel/cgroup/cpuset.c | 167 ++++++++++++++++----------------2 files changed, 84 insertions(+), 90 deletions(-)kernel/cgroup/cpuset-internal.h +0
-1@@ -167,13 +167,6 @@ struct cpuset {*/ int nr_deadline_tasks; int nr_migrate_dl_tasks; - /* DL bandwidth that needs destination reservation for thisattach. */
- u64sum_migrate_dl_bw;
- /*
- * CPU used for temporary DL bandwidth allocation during attach; - * -1 if no DL bandwidth was allocated in the current attach. - */
- intdl_bw_cpu;
/* Invalid partition error code, not lock protected */ enumprs_errcodeprs_err;kernel/cgroup/cpuset.c +0
-0@@ -288,7 +288,6 @@ struct cpuset top_cpuset = {.flags =
BIT(CS_CPU_EXCLUSIVE)|BIT(CS_MEM_EXCLUSIVE)|BIT(CS_SCHED_LOAD_BALANCE), .partition_root_state =PRS_ROOT,
- .dl_bw_cpu = -1,
};
/**kernel/cgroup/cpuset.c +3
-3@@ -580,8 +579,6 @@ static struct cpuset *dup_or_alloc_cpuset(struct cpuset *cs) if (!trial) return NULL; - trial->dl_bw_cpu = -1; - /* Setup cpumask pointer array */ cpumask_var_t *pmask[4] = { &trial->cpus_allowed,@@ -3026,31 +3023,36 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs, return 0; } -static int cpuset_reserve_dl_bw(struct cpuset *cs) +/* +* Undo DL bandwidth reservations and attach_in_progress increments done +* in cpuset_can_attach(). Used for both rollback on error and
cancel_attach.
+ * If @stop_at is non-NULL, undo only for tasks before @stop_at in the tset.
+ */
+static voidcpuset_undo_attach(struct cgroup_taskset *tset, + struct task_struct *stop_at)
{
- int cpu, ret;
-
- if (!cs->sum_migrate_dl_bw)
- return 0;
+ structcgroup_subsys_state*css;
+ structtask_struct*task;
- cpu =cpumask_any_and(cpu_active_mask, cs->effective_cpus); - if (unlikely(cpu >= nr_cpu_ids))
- return-EINVAL;
+cgroup_taskset_for_each(task, css, tset){
+ struct cpuset *cs =css_cs(css);
- ret =dl_bw_alloc(cpu, cs->sum_migrate_dl_bw);
- if (ret)
- return ret;
+ if (task ==stop_at)
+ break;
-cs->dl_bw_cpu= cpu;
- return 0;
+ if (dl_task(task)) {
+ cs->nr_migrate_dl_tasks--;
+ if (dl_task_needs_bw_move(task, cs->effective_cpus)) { + int cpu =cpumask_any_and(cpu_active_mask, + cs->effective_cpus);
+dl_bw_free(cpu, task->dl.dl_bw);
+ }
+ }
+dec_attach_in_progress_locked(cs);
+ }
}kernel/cgroup/cpuset.c +1
-6-static void reset_migrate_dl_data (struct cpuset *cs) -{ - cs->nr_migrate_dl_tasks = 0; - cs->sum_migrate_dl_bw = 0; - cs->dl_bw_cpu = -1; -} +static bool attach_many_sources;/* Called by cgroups to determine if a cpuset is usable;
cpuset_mutexheld */ static intcpuset_can_attach(struct cgroup_taskset *tset)kernel/cgroup/cpuset.c +1
-2@@ -3067,90 +3069,73 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) cs = css_cs(css); mutex_lock(&cpuset_mutex); + attach_many_sources = false; /* Check to see if task is allowed in the cpuset */ ret = cpuset_can_attach_check(cs, oldcs, &setsched_check); if (ret) goto out_unlock; - /* -* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get
- * theold_mems_allowedvalue. There are two ways that many-to-one - * cpuset migration can happen:
- * 1) A multithread application with threads in different cpusets is - * wholely migrated to a new cpuset.
- * 2) Disabling v2 cpuset controller will move all the tasks in child
- * cpusets to the parent cpuset.
- *
- * In the former case, it is the mm setting of the group leader that - * really matters. Socpuset_attach_old_csshould track the oldcs of the
- * group leader. It falls back to the oldcs of the first task if there
- * is no group leader in the taskset. In the latter case,effective_mems
- * of child cpusets must always be a subset of the parent. So no real
- * page migration will be necessary no matter which child cpuset is - * selected ascpuset_attach_old_cs.
- */
cgroup_taskset_for_each(task, css, tset){
+ struct cpuset *newcs =css_cs(css);
+ struct cpuset*new_oldcs=task_cs(task);kernel/cgroup/cpuset.c +12
-0+ + if (newcs != cs || new_oldcs != oldcs) { + if (new_oldcs != oldcs) + attach_many_sources = true; + cs = newcs; + oldcs = new_oldcs; + ret = cpuset_can_attach_check(cs, oldcs, + &setsched_check); + if (ret) + goto out_rollback; + } +ret =
task_can_attach(task);
if (ret)
- gotoout_unlock;
+ gotoout_rollback;
- /* Updatecpuset_attach_old_csto the latest group leader */ if (task ==task->group_leader)cpuset_attach_old_cs=task_cs(task);
if (setsched_check) {
ret =security_task_setscheduler(task);
if (ret)
- gotoout_unlock;
+ gotoout_rollback;
}
if (dl_task(task)) {
- /*
- * Count all migrating DL tasks for cpuset task accounting. - * Only tasks that need a root-domain bandwidth move - * contribute tosum_migrate_dl_bw.
- */
cs->nr_migrate_dl_tasks++;
- if (dl_task_needs_bw_move(task, cs->effective_cpus)) -cs->sum_migrate_dl_bw+=task->dl.dl_bw;
+ if (dl_task_needs_bw_move(task, cs->effective_cpus)) { + int cpu =cpumask_any_and(cpu_active_mask, + cs->effective_cpus);
+ if (unlikely(cpu >= nr_cpu_ids)) {
+ ret =-EINVAL;
+ gotoout_rollback;
+ }
+ ret =dl_bw_alloc(cpu, task->dl.dl_bw);
+ if (ret)
+ gotoout_rollback;
+ }
}
- }kernel/cgroup/cpuset.c +0
-8- - ret = cpuset_reserve_dl_bw(cs); -out_unlock: - if (ret) { - reset_migrate_dl_data(cs); - } else { - /* -* Mark attach is in progress. This makes validate_change() fail
- * changes which zero cpus/mems_allowed.
- */
cs->attach_in_progress++;
}
+ gotoout_unlock;kernel/cgroup/cpuset.c +5
-0+ +out_rollback: + cpuset_undo_attach(tset, task); + +out_unlock:
mutex_unlock(&cpuset_mutex);
return ret;
}
static voidcpuset_cancel_attach(struct cgroup_taskset *tset){
- structcgroup_subsys_state*css;
- struct cpuset *cs;kernel/cgroup/cpuset.c +0
-4- - cgroup_taskset_first(tset, &css); - cs = css_cs(css); -
mutex_lock(&cpuset_mutex);
-dec_attach_in_progress_locked(cs);kernel/cgroup/cpuset.c +1
-7- - if (cs->dl_bw_cpu >= 0) - dl_bw_free(cs->dl_bw_cpu, cs->sum_migrate_dl_bw); - - if (cs->nr_migrate_dl_tasks) - reset_migrate_dl_data(cs); - + cpuset_undo_attach(tset, NULL);
mutex_unlock(&cpuset_mutex);
}kernel/cgroup/cpuset.c +9
-2@@ -3232,8 +3217,15 @@ static void cpuset_attach(struct cgroup_taskset *tset) mutex_lock(&cpuset_mutex); queue_task_work = false; - attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs- >effective_cpus); - attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs- >effective_mems); + if (attach_many_sources) { + attach_cpus_updated = true; + attach_mems_updated = true; + } else { + attach_cpus_updated = !cpumask_equal(cs->effective_cpus, + oldcs->effective_cpus); + attach_mems_updated = !nodes_equal(cs->effective_mems, + oldcs->effective_mems); + } /** In the default hierarchy, enabling cpuset in the child cgroups
kernel/cgroup/cpuset.c +1
-1@@ -3250,20 +3242,29 @@ static void cpuset_attach(struct cgroup_taskset *tset) } cgroup_taskset_for_each(task, css, tset) - cpuset_attach_task(cs, task); + cpuset_attach_task(css_cs(css), task);out:
if (queue_task_work)
schedule_flush_migrate_mm();cs->old_mems_allowed=cpuset_attach_nodemask_to;
- if (cs->nr_migrate_dl_tasks) {
-cs->nr_deadline_tasks+=cs->nr_migrate_dl_tasks; -oldcs->nr_deadline_tasks-=cs->nr_migrate_dl_tasks; -reset_migrate_dl_data(cs);
- }
+ /*
+ * Updatenr_deadline_tasksnow that migration is committed. + *nr_migrate_dl_taskswas accumulated per-dst incan_attachbut + *nr_deadline_tasksis deferred to here to avoid a race with + *dl_rebuild_rd_accounting()betweencan_attachand attach. + */
+cgroup_taskset_for_each(task, css, tset){
+ struct cpuset*dst_cs=css_cs(css);Quoted 8 lines
-
dec_attach_in_progress_locked(cs);
+ if (dst_cs->nr_migrate_dl_tasks) {
+dst_cs->nr_deadline_tasks+=dst_cs->nr_migrate_dl_tasks; +oldcs->nr_deadline_tasks-=dst_cs->nr_migrate_dl_tasks; +dst_cs->nr_migrate_dl_tasks= 0;
+ }
+dec_attach_in_progress_locked(dst_cs);
+ }You are assuming that there is only one source cpuset. That may not be true. I suppose that we may not track the set of destination cpusets and they will be hit during task iteration. We will still need to track all the source cpusets.
Quoted 6 lines
Cheers,
LongmanQuoted 2 lines
mutex_unlock(&cpuset_mutex);
}
Yeah, we still need the per-src here. I agree that we can just track the source cpusets.
Signature · 2 lines
Best regards Ridong
Ridong Chen
reviewerRe: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
In reply to Waiman LongQuoted 62 lines · Waiman Long
Quoted 57 lines · Ridong Chen
Quoted 36 lines · Waiman Long
Quoted 35 lines · Ridong Chen
The current
cpuset_can_attach()andcpuset_attach()functions assume task
migration is from one source cpuset to one destination cpuset. This can be
wrong in several scenarios:
- Moving a multi-threaded process with threads in different cpusets
- Disabling the cpuset controller (many children to one parent)
- Enabling the cpuset controller (one parent to many children)Fix this by adopting the pids subsystem's per-task accounting pattern. In
cpuset_can_attach(), usetask_cs(task)to get the correct source cpuset
for each task (likepids_can_attachusestask_css), adjustnr_deadline_tasks
and reserve DL bandwidth per-task, and incrementattach_in_progressper-task
on the destination cpuset. Incpuset_attach(), handle destination cpuset
changes within the task iteration loop.A shared helper
cpuset_undo_attach()reverses the per-task operations for
both partial rollback incpuset_can_attach()and full reversal incpuset_cancel_attach().When multiple source cpusets are detected in
can_attach(), setattach_many_sourcesso thatcpuset_attach()forcescpus_updatedandmems_updatedto true, ensuring all tasks get properly updated regardless
of which source cpusetcpuset_attach_old_cspoints to.This eliminates the need for
nr_migrate_dl_tasks,sum_migrate_dl_bw, anddl_bw_cpufields in struct cpuset.Quoted 7 lines
Quoted 3 lines
Fixes 4ec22e9c5a90 ("cpuset: Enable cpuset controller in defaulthierarchy")
Signed-off-by Ridong Chen <[email protected]>It is not a problem doing per-task DL BW allocation and eliminating the *dl_bw* fields. However, updating
nr_deadline_tasksbefore it is committed can be problematic.Good to hear that.
Quoted 10 lines
nr_deadline_tasksis used indl_rebuild_rd_accounting()which is called bypartition_sched_domains_locked(). After the release ofcpuset_mutexat the end ofcpuset_can_attach()and beforecpuset_attach()orcpuset_cancel_attach()is called, it is possible
that partition_sched_domains_locked() can be called
and dl_rebuild_rd_accounting() is not getting the right DL BW accounting information. So unless there is a way to confirm that this situation cannot happen, we can't change nr_deadline_tasks before the attach is commited.Quoted 4 lines
We can keep the
nr_migrate_dl_tasksfield and updatenr_deadline_tasksonce migration is complete. I think this will be much simpler than fixing the issue using lists.
Quoted 6 lines
But we still need to track the set of source and destination cpusets to commit or cancel the change. Doing it task-by-task will add code in the
cpuset_attach()andcpuset_cancel_attach()to check if a task is a DL task and act accordingly. So we are just trading task-by-task code with code to handle the lists.
I resend a patch [1] that keeps nr_migrate_dl_tasks but eliminates sum_migrate_dl_bw, dl_bw_cpu, attach_node, and attach_llist_head from the cpuset structure by doing per-task dl_bw_alloc directly in cpuset_can_attach().
I just offer a way to discuss whether we can solve this issue in a simpler way.
[1]
https://lore.kernel.org/cgroups/[email protected]/T/#mb2c6a3ae44f34f571db5dffa888212eaeaaea17a
Signature · 2 lines
Best regards, Ridong
A message that is not in the archive (sent off-list or not yet received): [email protected]
Waiman Long
maintainerQuoted 30 lines · On 6/1/26 11:39 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Cpuset CPU and memory mask updates are silently skipped for tasks when migrating a process with threads in multiple cpusets. --commit
a6d20cd30ffff4c71ea67a488e69d137c1a06525
Author: Waiman Long <[email protected]>cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
This commit updates
cpuset_can_attach()andcpuset_attach()to support task migration data with multiple destination or source cpusets. It uses singly linked lists to track the source and destination cpusets.kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3310
Quoted 10 lines
kernel/cgroup/cpuset.c +1
-1@@ -3227,48 +3307,76 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct task_struct *task;struct cgroup_subsys_state *css;struct cpuset *cs; -struct cpuset *oldcs = cpuset_attach_old_cs; +bool many_cs_to_one = !!src_cs_head.first->next;[Severity: High]
Does this logic correctly handle migrating a process that spans multiple cpusets (e.g., A and C) into one of those existing cpusets (e.g., C)?If
cpuset_can_attach_check()processes a task migrating from A to C, it adds C todst_cs_headand sets itsattach_node_in_llistflag to true.
Quoted 8 lines
When it subsequently processes a task migrating from C to C, C is the oldcs. Since C's
attach_node_in_llistflag is already true, it is not added tosrc_cs_head.If
src_cs_headonly ends up containing A,src_cs_head.first->nextis NULL, andmany_cs_to_oneincorrectly evaluates to false.[ ... ]
My own testing indicates that the cgroup core will only put tasks that need migration into the cgroup_taskset passed to cpuset_can_attach(). IOW, existing tasks in C won't show up in the task iteration process. To be safe, safeguard will be added to guard against this situation.
Cheers,
Longman
Quoted 28 lines
Quoted 8 lines
kernel/cgroup/cpuset.c +1
-1cs =css_cs (css);/* * In the default hierarchy, enabling cpuset in the child cgroups * will trigger a cpuset_attach() call with no change in effective cpus * and mems. In that case, we can optimize out by skipping the task -* iteration and update. +* iteration and update, but the destination cpuset list is iterated to kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3352
Quoted 10 lines
kernel/cgroup/cpuset.c +5
-1+* set old_mems_sllowed. */ if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; -if (!attach_cpus_updated && !attach_mems_updated) +if (!attach_cpus_updated && !attach_mems_updated) { +llist_for_each_entry (cs, dst_cs_head.first, attach_node) + cs->old_mems_allowed = cs->effective_mems; goto out; + }[Severity: High]
Ifmany_cs_to_oneevaluated to false in the scenario above, could this cause threads migrating from A to incorrectly hit this early exit?If the destination (C) matches the group leader's origin (C),
attach_cpus_updatedandattach_mems_updatedwould be false.This would skip
cpuset_attach_task()for the thread migrating from A, causing it to retain A's affinity parameters while logically residing in C.
Michal Koutný
maintainerHello Waiman.
Quoted 8 lines · Waiman Long
This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.
When I generalize that it can be an issue for any threaded controller that somehow relies on the _difference_ between old and new thread membership.
So I checked some: pids and perf_events look alright (no diff-dependency) but I noticed the very same issue is tackled in sched_change_group/scx_cgroup_move_task and that there is a member inside task_struct allocated for this state tracking already:
task_struct::scx::cgrp_moving_from
Quoted 4 lines
Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.
So there would be more than a single use for something conceptually like:
include/linux/sched.h +3 -0
@@ -1326,6 +1326,9 @@ struct task_struct {#ifdef CONFIG_PREEMPT_RT struct llist_node cg_dead_lnode;#endif /* CONFIG_PREEMPT_RT */ +#ifdef CONFIG_CGROUPS_MOVING_FROM +struct cgroup *cgrp_moving_from; +#endif #endif /* CONFIG_CGROUPS */ #ifdef CONFIG_X86_CPU_RESCTRL u32 closid;
include/linux/sched/ext.h +0 -3
@@ -240,9 +240,6 @@ struct sched_ext_entity {bool disallow;/* reject switching into SCX */ /* cold fields */ -#ifdef CONFIG_EXT_GROUP_SCHED -struct cgroup *cgrp_moving_from; -#endif struct list_head tasks_node; };
init/Kconfig +2 -0
@@ -1186,6 +1186,7 @@ config EXT_GROUP_SCHED depends on SCHED_CLASS_EXT && CGROUP_SCHED select GROUP_SCHED_WEIGHT select GROUP_SCHED_BANDWIDTH + select CGROUPS_MOVING_FROM default y endif #CGROUP_SCHED
@@ -1288,6 +1289,7 @@ config CPUSETS depends on SMP select UNION_FIND select CPU_ISOLATION + select CGROUPS_MOVING_FROM help This option will let you create and manage CPUSETs which allow dynamically partitioning a system into sets of CPUs and
I think this could simplify the before-after state tracking generally, WDYT?
Michal
Waiman Long
maintainerQuoted 77 lines · Michal Koutný
Hello Waiman.
Quoted 8 lines · Waiman Long
This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.
When I generalize that it can be an issue for any threaded controller that somehow relies on the _difference_ between old and new thread membership.
So I checked some: pids and
perf_eventslook alright (no diff-dependency) but I noticed the very same issue is tackled in sched_change_group/scx_cgroup_move_task and that there is a member insidetask_structallocated for this state tracking already:
task_struct::scx::cgrp_moving_fromQuoted 4 lines
Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.
So there would be more than a single use for something conceptually like:
include/linux/sched.h +3
-0@@ -1326,6 +1326,9 @@ struct task_struct {#ifdef CONFIG_PREEMPT_RT struct llist_node cg_dead_lnode;#endif /* CONFIG_PREEMPT_RT */ +#ifdef CONFIG_CGROUPS_MOVING_FROM +struct cgroup *cgrp_moving_from; +#endif #endif /* CONFIG_CGROUPS */ #ifdef CONFIG_X86_CPU_RESCTRL u32 closid;include/linux/sched/ext.h +0
-3@@ -240,9 +240,6 @@ struct sched_ext_entity {bool disallow;/* reject switching into SCX */ /* cold fields */ -#ifdef CONFIG_EXT_GROUP_SCHED -struct cgroup *cgrp_moving_from; -#endif struct list_head tasks_node; };init/Kconfig +2
-0@@ -1186,6 +1186,7 @@ config EXT_GROUP_SCHED depends on SCHED_CLASS_EXT && CGROUP_SCHED select GROUP_SCHED_WEIGHT select GROUP_SCHED_BANDWIDTH + select CGROUPS_MOVING_FROM default y endif #CGROUP_SCHED@@ -1288,6 +1289,7 @@ config CPUSETS depends on SMP select UNION_FIND select CPU_ISOLATION + select CGROUPS_MOVING_FROM help This option will let you create and manage CPUSETs which allow dynamically partitioning a system into sets of CPUs and
Quoted 2 lines
I think this could simplify the before-after state tracking generally, WDYT?
I had actually introduced a new task_struct field in an early version to track the old cpuset to handle memory migration. However, Chen Ridong had shown me that we may not really need such granular detail. So I drop it in the newer versions. Also sharing a common field between cpuset and sched_ext can introduce complication as we have to make sure that we won't step into each other.
Thank for the suggestion anyway and I will reconsider it in case it is found that we really need such information to do the right thing.
Cheers,
Longman
Michal Koutný
maintainerQuoted 4 lines · Waiman Long
Patch 6 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.
Thanks for looking into this!
I've played with a coding assistant and produced the following selftest (it (expectedly) fails on my machine), feel free to include in the series (if it validates the fix).
-- 8< --
From ed4e6cf91413bb4b64befb1c15412c8cfd205d73 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Michal=20Koutn=C3=BD?= <[email protected]>
Date: Wed, 24 Jun 2026 16:39:30 +0200
Subject: [PATCH] selftests/cgroup: Add test for cpuset affinity on controller
disable
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Add a new selftest that exposes a bug in cpuset_attach() where thread CPU affinity is not properly updated when the cpuset controller is disabled in a threaded cgroup hierarchy.
The test creates a threaded cgroup hierarchy with two child cgroups (A and B) having different cpuset.cpus constraints:
- Parent: cpuset.cpus=0-1
- Child A: cpuset.cpus=0-1
- Child B: cpuset.cpus=1 (restricted to CPU 1 only)
A multithreaded process is created with threads placed in different cgroups. When the cpuset controller is disabled on the parent, thread affinities should be updated to match the parent's cpuset.
Expected behavior:
- thread_a affinity: {0-1} before and after (unchanged)
- thread_b affinity: {1} before, {0-1} after (expanded)
Current buggy behavior:
- thread_b affinity remains {1} after controller disable
--- tools/testing/selftests/cgroup/test_cpuset.c | 243 +++++++++++++++++++ 1 file changed, 243 insertions(+)
tools/testing/selftests/cgroup/test_cpuset.c +243 -0
@@ -1,7 +1,13 @@// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <assert.h> #include <linux/limits.h> +#include <pthread.h> +#include <sched.h> #include <signal.h> +#include <sys/syscall.h> +#include <unistd.h> #include "kselftest.h" #include "cgroup_util.h"
@@ -232,6 +238,242 @@ static int test_cpuset_perms_subtree(const char *root)return ret; } +static int get_cpu_affinity (cpu_set_t *mask) +{ +CPU_ZERO (mask); +return sched_getaffinity (0 ,sizeof (*mask), mask); +} + +static int cpu_set_equal (cpu_set_t *dst,unsigned long mask) +{ +cpu_set_t expected; + +CPU_ZERO (&expected); +assert (sizeof (mask) <CPU_SETSIZE ); + +for (int cpu =0 ; cpu <sizeof (mask); ++cpu) +if ((1UL << cpu) & mask) +CPU_SET (cpu, &expected); + +return CPU_EQUAL (&expected, dst); +} + +enum test_phase { +AFFINITY_SETUP , +AFFINITY_THREAD_A_READY , +AFFINITY_THREADS_READY , +AFFINITY_CONTROLLER_DISABLED , +AFFINITY_COMPLETE , +AFFINITY_ERROR +}; + +struct thread_args { +const char *cgroup; +cpu_set_t *affinity_before; +cpu_set_t *affinity_after; +enum test_phase ready_phase; +}; + +static pthread_mutex_t test_mutex =PTHREAD_MUTEX_INITIALIZER ; +static pthread_cond_t test_cond =PTHREAD_COND_INITIALIZER ; +static enum test_phase test_phase; + +static void *affinity_thread_fn (void *arg) +{ +struct thread_args *args = (struct thread_args *)arg; + +if (cg_enter_current_thread (args->cgroup)) + goto fail; + +if (get_cpu_affinity (args->affinity_before) !=0 ) + goto fail; + +pthread_mutex_lock (&test_mutex); +if (test_phase < args->ready_phase) + test_phase = args->ready_phase; +pthread_cond_broadcast (&test_cond); + +while (test_phase <AFFINITY_CONTROLLER_DISABLED ) +pthread_cond_wait (&test_cond, &test_mutex); +pthread_mutex_unlock (&test_mutex); + +if (get_cpu_affinity (args->affinity_after) !=0 ) + goto fail; + + +return NULL ; + +fail: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_ERROR ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); +return NULL ; +} + +/* +* Test that disabling cpuset controller properly updates thread affinity. +* +* This test exposes a bug in cpuset_attach() where threads in child cgroups +* don't get their affinity updated when the cpuset controller is disabled. +* +* Setup: +* - Create parent cgroup with cpuset.cpus=0-1 +* - Create child A with cpuset.cpus=0-1 +* - Create child B with cpuset.cpus=1 +* - Place multithreaded process: group leader + thread_a in A, thread_b in B +* - Disable cpuset controller on parent +* +* Expected: thread_b's affinity should expand from {1} to {0-1} +* Buggy: thread_b's affinity remains {1} +*/ +static int test_cpuset_affinity_on_controller_disable (const char *root) +{ +char *parent =NULL , *child_a =NULL , *child_b =NULL ; +pthread_t thread_a, thread_b; +int thread_a_created =0 , thread_b_created =0 ; +cpu_set_t affinity_a_before, affinity_a_after; +cpu_set_t affinity_b_before, affinity_b_after; +int ret =KSFT_FAIL ; + + parent =cg_name (root,"cpuset_affinity_test" ); +if (!parent) + goto cleanup; +if (cg_create (parent)) + goto cleanup; +if (cg_write (parent,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_a =cg_name (parent,"A" ); +if (!child_a) + goto cleanup; +if (cg_create (child_a)) + goto cleanup; +if (cg_write (child_a,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_b =cg_name (parent,"B" ); +if (!child_b) + goto cleanup; +if (cg_create (child_b)) + goto cleanup; +if (cg_write (child_b,"cgroup.type" ,"threaded" )) + goto cleanup; + +/* Now enable cpuset controller in parent */ +if (cg_write (parent,"cgroup.subtree_control" ,"+cpuset" )) { + ret =KSFT_SKIP ; + goto cleanup; + } + +/* Set CPU affinity constraints */ +if (cg_write (parent,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_a,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_b,"cpuset.cpus" ,"1" )) + goto cleanup; + +/* Move group leader (main thread) to child A */ +if (cg_enter_current (child_a)) + goto cleanup; + +/* Create threads - they will move themselves to their respective cgroups */ + test_phase =AFFINITY_SETUP ; + +struct thread_args args_a = { + .cgroup = child_a, + .affinity_before = &affinity_a_before, + .affinity_after = &affinity_a_after, + .ready_phase =AFFINITY_THREAD_A_READY , + }; +if (pthread_create (&thread_a,NULL , affinity_thread_fn, &args_a)) + goto cleanup; + thread_a_created =1 ; + +struct thread_args args_b = { + .cgroup = child_b, + .affinity_before = &affinity_b_before, + .affinity_after = &affinity_b_after, + .ready_phase =AFFINITY_THREADS_READY , + }; +if (pthread_create (&thread_b,NULL , affinity_thread_fn, &args_b)) + goto cleanup_threads; + thread_b_created =1 ; + +pthread_mutex_lock (&test_mutex); +while (test_phase <AFFINITY_THREADS_READY ) +pthread_cond_wait (&test_cond, &test_mutex); + +/* If a thread failed during setup, bail out */ +if (test_phase ==AFFINITY_ERROR ) { +pthread_mutex_unlock (&test_mutex); + goto cleanup_threads; + } +pthread_mutex_unlock (&test_mutex); + +if (!cpu_set_equal (&affinity_a_before,0x3 )) { +ksft_print_msg ("FAIL: thread_a initial affinity incorrect\n" ); + goto cleanup_threads; + } + +if (!cpu_set_equal (&affinity_b_before,0x2 )) { +ksft_print_msg ("FAIL: thread_b initial affinity incorrect\n" ); + goto cleanup_threads; + } + +/* Disable cpuset controller - this should trigger affinity update */ +if (cg_write (parent,"cgroup.subtree_control" ,"-cpuset" )) + goto cleanup_threads; + +/* Signal threads to save their final affinity and exit */ +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_CONTROLLER_DISABLED ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +pthread_join (thread_a,NULL ); +pthread_join (thread_b,NULL ); + +/* Verify thread affinities AFTER disabling controller */ +if (!cpu_set_equal (&affinity_a_after,0x3 )) { +ksft_print_msg ("FAIL: thread_a final affinity incorrect\n" ); + goto cleanup; + } + +if (!cpu_set_equal (&affinity_b_after,0x3 )) { +ksft_print_msg ("FAIL: thread_b affinity did not expand to {0-1}\n" ); + goto cleanup; + } + + ret =KSFT_PASS ; + goto cleanup; + +cleanup_threads: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_COMPLETE ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +if (thread_a_created) +pthread_join (thread_a,NULL ); +if (thread_b_created) +pthread_join (thread_b,NULL ); + +cleanup: +/* Move back to root before cleanup */ +cg_enter_current (root); + +cg_destroy (child_b); +free (child_b); +cg_destroy (child_a); +free (child_a); +cg_destroy (parent); +free (parent); + +return ret; +} +#define T (x) { x, #x }struct cpuset_test {
@@ -241,6 +483,7 @@ struct cpuset_test {T (test_cpuset_perms_object_allow),T (test_cpuset_perms_object_deny),T (test_cpuset_perms_subtree), + T(test_cpuset_affinity_on_controller_disable), };#undef T
Signature · 1 line
2.54.0
Waiman Long
maintainerQuoted 9 lines · Michal Koutný
Quoted 4 lines · Waiman Long
Patch 6 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.
Thanks for looking into this!
I've played with a coding assistant and produced the following selftest (it (expectedly) fails on my machine), feel free to include in the series (if it validates the fix).
Thank for the provided selftest update. Will include that in the next version.
Cheers,
Longman
Quoted 307 lines
-- 8< --
Fromed4e6cf91413bb4b64befb1c15412c8cfd205d73Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Michal=20Koutn=C3=BD?= <[email protected]>
Date: Wed, 24 Jun 2026 16:39:30 +0200
Subject: [PATCH] selftests/cgroup: Add test for cpuset affinity on controller
disable
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitAdd a new selftest that exposes a bug in
cpuset_attach()where thread CPU affinity is not properly updated when the cpuset controller is disabled in a threaded cgroup hierarchy.The test creates a threaded cgroup hierarchy with two child cgroups (A and B) having different cpuset.cpus constraints:
- Parent: cpuset.cpus=0-1
- Child A: cpuset.cpus=0-1
- Child B: cpuset.cpus=1 (restricted to CPU 1 only)A multithreaded process is created with threads placed in different cgroups. When the cpuset controller is disabled on the parent, thread affinities should be updated to match the parent's cpuset.
Expected behavior:
-thread_aaffinity: {0-1} before and after (unchanged)
-thread_baffinity: {1} before, {0-1} after (expanded)Current buggy behavior:
-thread_baffinity remains {1} after controller disableAssisted-by Claude:claude-sonnet-4-5Signed-off-by Michal Koutný <[email protected]>--- tools/testing/selftests/cgroup/test_cpuset.c | 243 +++++++++++++++++++ 1 file changed, 243 insertions(+)tools/testing/selftests/cgroup/test_cpuset.c +243
-0@@ -1,7 +1,13 @@// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <assert.h> #include <linux/limits.h> +#include <pthread.h> +#include <sched.h> #include <signal.h> +#include <sys/syscall.h> +#include <unistd.h> #include "kselftest.h" #include "cgroup_util.h" @@ -232,6 +238,242 @@ static int test_cpuset_perms_subtree(const char *root)return ret; } +static int get_cpu_affinity (cpu_set_t *mask) +{ +CPU_ZERO (mask); +return sched_getaffinity (0 ,sizeof (*mask), mask); +} + +static int cpu_set_equal (cpu_set_t *dst,unsigned long mask) +{ +cpu_set_t expected; + +CPU_ZERO (&expected); +assert (sizeof (mask) <CPU_SETSIZE ); + +for (int cpu =0 ; cpu <sizeof (mask); ++cpu) +if ((1UL << cpu) & mask) +CPU_SET (cpu, &expected); + +return CPU_EQUAL (&expected, dst); +} + +enum test_phase { +AFFINITY_SETUP , +AFFINITY_THREAD_A_READY , +AFFINITY_THREADS_READY , +AFFINITY_CONTROLLER_DISABLED , +AFFINITY_COMPLETE , +AFFINITY_ERROR +}; + +struct thread_args { +const char *cgroup; +cpu_set_t *affinity_before; +cpu_set_t *affinity_after; +enum test_phase ready_phase; +}; + +static pthread_mutex_t test_mutex =PTHREAD_MUTEX_INITIALIZER ; +static pthread_cond_t test_cond =PTHREAD_COND_INITIALIZER ; +static enum test_phase test_phase; + +static void *affinity_thread_fn (void *arg) +{ +struct thread_args *args = (struct thread_args *)arg; + +if (cg_enter_current_thread (args->cgroup)) + goto fail; + +if (get_cpu_affinity (args->affinity_before) !=0 ) + goto fail; + +pthread_mutex_lock (&test_mutex); +if (test_phase < args->ready_phase) + test_phase = args->ready_phase; +pthread_cond_broadcast (&test_cond); + +while (test_phase <AFFINITY_CONTROLLER_DISABLED ) +pthread_cond_wait (&test_cond, &test_mutex); +pthread_mutex_unlock (&test_mutex); + +if (get_cpu_affinity (args->affinity_after) !=0 ) + goto fail; + + +return NULL ; + +fail: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_ERROR ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); +return NULL ; +} + +/* +* Test that disabling cpuset controller properly updates thread affinity. +* +* This test exposes a bug in cpuset_attach() where threads in child cgroups +* don't get their affinity updated when the cpuset controller is disabled. +* +* Setup: +* - Create parent cgroup with cpuset.cpus=0-1 +* - Create child A with cpuset.cpus=0-1 +* - Create child B with cpuset.cpus=1 +* - Place multithreaded process: group leader + thread_a in A, thread_b in B +* - Disable cpuset controller on parent +* +* Expected: thread_b's affinity should expand from {1} to {0-1} +* Buggy: thread_b's affinity remains {1} +*/ +static int test_cpuset_affinity_on_controller_disable (const char *root) +{ +char *parent =NULL , *child_a =NULL , *child_b =NULL ; +pthread_t thread_a, thread_b; +int thread_a_created =0 , thread_b_created =0 ; +cpu_set_t affinity_a_before, affinity_a_after; +cpu_set_t affinity_b_before, affinity_b_after; +int ret =KSFT_FAIL ; + + parent =cg_name (root,"cpuset_affinity_test" ); +if (!parent) + goto cleanup; +if (cg_create (parent)) + goto cleanup; +if (cg_write (parent,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_a =cg_name (parent,"A" ); +if (!child_a) + goto cleanup; +if (cg_create (child_a)) + goto cleanup; +if (cg_write (child_a,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_b =cg_name (parent,"B" ); +if (!child_b) + goto cleanup; +if (cg_create (child_b)) + goto cleanup; +if (cg_write (child_b,"cgroup.type" ,"threaded" )) + goto cleanup; + +/* Now enable cpuset controller in parent */ +if (cg_write (parent,"cgroup.subtree_control" ,"+cpuset" )) { + ret =KSFT_SKIP ; + goto cleanup; + } + +/* Set CPU affinity constraints */ +if (cg_write (parent,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_a,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_b,"cpuset.cpus" ,"1" )) + goto cleanup; + +/* Move group leader (main thread) to child A */ +if (cg_enter_current (child_a)) + goto cleanup; + +/* Create threads - they will move themselves to their respective cgroups */ + test_phase =AFFINITY_SETUP ; + +struct thread_args args_a = { + .cgroup = child_a, + .affinity_before = &affinity_a_before, + .affinity_after = &affinity_a_after, + .ready_phase =AFFINITY_THREAD_A_READY , + }; +if (pthread_create (&thread_a,NULL , affinity_thread_fn, &args_a)) + goto cleanup; + thread_a_created =1 ; + +struct thread_args args_b = { + .cgroup = child_b, + .affinity_before = &affinity_b_before, + .affinity_after = &affinity_b_after, + .ready_phase =AFFINITY_THREADS_READY , + }; +if (pthread_create (&thread_b,NULL , affinity_thread_fn, &args_b)) + goto cleanup_threads; + thread_b_created =1 ; + +pthread_mutex_lock (&test_mutex); +while (test_phase <AFFINITY_THREADS_READY ) +pthread_cond_wait (&test_cond, &test_mutex); + +/* If a thread failed during setup, bail out */ +if (test_phase ==AFFINITY_ERROR ) { +pthread_mutex_unlock (&test_mutex); + goto cleanup_threads; + } +pthread_mutex_unlock (&test_mutex); + +if (!cpu_set_equal (&affinity_a_before,0x3 )) { +ksft_print_msg ("FAIL: thread_a initial affinity incorrect\n" ); + goto cleanup_threads; + } + +if (!cpu_set_equal (&affinity_b_before,0x2 )) { +ksft_print_msg ("FAIL: thread_b initial affinity incorrect\n" ); + goto cleanup_threads; + } + +/* Disable cpuset controller - this should trigger affinity update */ +if (cg_write (parent,"cgroup.subtree_control" ,"-cpuset" )) + goto cleanup_threads; + +/* Signal threads to save their final affinity and exit */ +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_CONTROLLER_DISABLED ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +pthread_join (thread_a,NULL ); +pthread_join (thread_b,NULL ); + +/* Verify thread affinities AFTER disabling controller */ +if (!cpu_set_equal (&affinity_a_after,0x3 )) { +ksft_print_msg ("FAIL: thread_a final affinity incorrect\n" ); + goto cleanup; + } + +if (!cpu_set_equal (&affinity_b_after,0x3 )) { +ksft_print_msg ("FAIL: thread_b affinity did not expand to {0-1}\n" ); + goto cleanup; + } + + ret =KSFT_PASS ; + goto cleanup; + +cleanup_threads: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_COMPLETE ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +if (thread_a_created) +pthread_join (thread_a,NULL ); +if (thread_b_created) +pthread_join (thread_b,NULL ); + +cleanup: +/* Move back to root before cleanup */ +cg_enter_current (root); + +cg_destroy (child_b); +free (child_b); +cg_destroy (child_a); +free (child_a); +cg_destroy (parent); +free (parent); + +return ret; +} +#define T (x) { x, #x }struct cpuset_test {@@ -241,6 +483,7 @@ struct cpuset_test {T (test_cpuset_perms_object_allow),T (test_cpuset_perms_object_deny),T (test_cpuset_perms_subtree), + T(test_cpuset_affinity_on_controller_disable), };#undef T