cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
Subsystems (2)
Waiman Long
maintainerv6 cover cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
v6:
- Make guarantee_online_mems() to only return cs->effective_mems with v2 in patch 1.
- Remove obsolete commit description text from patch 3.
- Add Reviewed-by tags.
- In patch 6, add WARN_ON_ONCE() test in cpuset_can_attach() to
confirm that cs != oldcs.
v5:
- Remove the WARN_ON() call as it can be triggered in a corner case.
- Instead of passing an attach_cpus_updated and attach_mems_updated flags from cpuset_can_attach() to cpuset_attach(), re-evaluate the flags at the beginning of cpuset_attach() based on data in the source & destination cpusets in the singly linked lists to eliminate the Time-of-Check to Time-of-Use (TOCTOU) race condition & simplify the code changes.
- Add back the dropped optimization in patch 5.
v4:
- Add a new patch 1 to fix inconsistency in node mask usage in cpuset_update_tasks_nodemask() and cpuset_attach() and adjust the subsequent patches accordingly.
- Update patch 3 to set the update flags whenever the CPU or node mask is updated to address issue reported by Sashiko.
- Update patch 5 to remove unneeded setting of old_mems_allowed as well as calling schedule_flush_migrate_mm() if queue_task_work is set.
Sashiko AI review of another cpuset patch had found that cpuset_attach() and cpuset_can_attach() can be passed a cgroup_taskset with tasks migrating from one source cpuset to multiple destination cpusets and vice versa. Further testing of the cpuset code indicates that this is indeed the case when the v2 cpuset controller is enabled or disabled.
Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that there will be one source and one destinaton cpuset which may result in inocrrect behavior.
This patch series is created to fix this issue.
Patch 1 is to fix an inconsistency in the way node mask update is being handled in cpuset_update_tasks_nodemask() and cpuset_attach() so that they match each other.
Patches 2 and 3 are just preparatory patches to make the remaining patches easier to review.
Patch 4 makes cpuset_attach_old_cs to track group leader for use by cpuset_migrate_mm().
Patch 5 moves mpol_rebind_mm() and cpuset_migrate_mm() inside cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works more like moving task from one cpuset to another one, while also make supporting multiple source and destination cpusets easier.
Patch 6 makes the necessary changes to enable the support of multiple source and destination cpusets by keeping all the source and destination cpusets found during task iterations in two singly linked lists for source and destination cpusets respectively.
Waiman Long (6):
cgroup/cpuset: Fix node inconsistencies between
cpuset_update_tasks_nodemask() and cpuset_attach()
cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper cgroup/cpuset: Expand the scope of cpuset_can_attach_check() cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
cpuset_attach_task()
cgroup/cpuset: Support multiple source/destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 424 +++++++++++++++++++++++--------- 2 files changed, 311 insertions(+), 119 deletions(-)
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v6 1/6 cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask via cpuset_change_task_nodemask()
2) memory policy binding via mpol_rebind_mm()
3) if memory migration is enabled, migrate from old_mems_allowed to the new node mask via cpuset_migrate_mm().
4) setting old_mems_allowed
These memory actions are done in cpuset_update_tasks_nodemask() and cpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.
In cpuset_update_tasks_nodemask(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): mems_allowed
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
In cpuset_attach(),
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): effective_mems
- old_mems_allowed: effective_mems
These inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The guarantee_online_mems() function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
The guarantee_online_mems() function should only be useful for v1 where mems_allowed is the same as effective_mems. With v2, the memory nodes in effective_mems should always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected in effective_mems as cpuset_handle_hotplug() has not yet been called from cpuset_track_online_nodes(). When cpuset_handle_hotplug() is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing the guarantee_online_mems() call by just using cs->effective_mems should be fine.
Let use the following setup for both of them and make them consistent.
- cpuset_change_task_nodemask(): guarantee_online_mems()
- mpol_rebind_mm(): effective_mems
- cpuset_migrate_mm(): guarantee_online_mems()
- old_mems_allowed: guarantee_online_mems()
So for v2, it is effectively all effective_mems. For v1, mpol_rebind_mm() uses mems_allowed which may differ from what guarantee_online_mems() returns.
--- kernel/cgroup/cpuset.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)
kernel/cgroup/cpuset.c +25 -12
@@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,* Return in *pmask the portion of a cpusets's mems_allowed that * are online, with memory. If none are online with memory, walk * up the cpuset hierarchy until we find one that does have some -* online mems. The top cpuset always has some mems online. +* online mems. The top cpuset always has some mems online. With v2, +* effective_mems should always contain online memory nodes except +* during the transition period where a memory node hotunplug operation +* is in progress. * * One way or another, we guarantee to return some non-empty subset * of node_states[N_MEMORY].
@@ -498,6 +501,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,*/ static void guarantee_online_mems (struct cpuset *cs,nodemask_t *pmask) { +if (cpuset_v2 ()) { + *pmask = cs->effective_mems; +return ; + }while (!nodes_and (*pmask, cs->effective_mems, node_states[N_MEMORY ])) cs =parent_cs (cs); }
@@ -2616,6 +2623,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask (struct cpuset *cs) {
@@ -2624,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)struct task_struct *task; cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems);/*
@@ -2650,7 +2663,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else
@@ -3148,17 +3161,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);
@@ -3168,7 +3182,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;
@@ -3176,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Signature · 1 line
2.54.0
Re: [PATCH-next v6 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 33 lines · Waiman Long
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask viacpuset_change_task_nodemask()
2) memory policy binding viampol_rebind_mm()
3) if memory migration is enabled, migrate fromold_mems_allowedto the new node mask viacpuset_migrate_mm().
4) settingold_mems_allowedThese memory actions are done in
cpuset_update_tasks_nodemask()andcpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.In
cpuset_update_tasks_nodemask(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():mems_allowed
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()In
cpuset_attach(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():effective_mems
-old_mems_allowed:effective_memsThese inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The
guarantee_online_mems()function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
Quoted 12 lines
The
guarantee_online_mems()function should only be useful for v1 wheremems_allowedis the same aseffective_mems. With v2, the memory nodes ineffective_memsshould always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected ineffective_memsascpuset_handle_hotplug()has not yet been called fromcpuset_track_online_nodes(). Whencpuset_handle_hotplug()is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing theguarantee_online_mems()call by just usingcs->effective_memsshould be fine.
The message should be updated.
Quoted 86 lines
Let use the following setup for both of them and make them consistent.
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()So for v2, it is effectively all
effective_mems. For v1,mpol_rebind_mm()usesmems_allowedwhich may differ from whatguarantee_online_mems()returns.Signed-off-by Waiman Long <[email protected]>--- kernel/cgroup/cpuset.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)kernel/cgroup/cpuset.c +20
-7@@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,* Return in *pmask the portion of a cpusets's mems_allowed that * are online, with memory. If none are online with memory, walk * up the cpuset hierarchy until we find one that does have some -* online mems. The top cpuset always has some mems online. +* online mems. The top cpuset always has some mems online. With v2, +* effective_mems should always contain online memory nodes except +* during the transition period where a memory node hotunplug operation +* is in progress. * * One way or another, we guarantee to return some non-empty subset * of node_states[N_MEMORY]. @@ -498,6 +501,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,*/ static void guarantee_online_mems (struct cpuset *cs,nodemask_t *pmask) { +if (cpuset_v2 ()) { + *pmask = cs->effective_mems; +return ; + }while (!nodes_and (*pmask, cs->effective_mems, node_states[N_MEMORY ])) cs =parent_cs (cs); }@@ -2616,6 +2623,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask (struct cpuset *cs) {@@ -2624,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)struct task_struct *task; cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems);/* @@ -2650,7 +2663,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else @@ -3148,17 +3161,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) {
kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3161
Quoted 10 lines
kernel/cgroup/cpuset.c +4
-3cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -
Nit.
I prefer:
guarantee_online_mems(cs, &cpuset_attach_nodemask_to); if (cpuset_v2() && !cpus_updated && !mems_updated) goto out;
Quoted 12 lines
kernel/cgroup/cpuset.c +0
-1cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);@@ -3168,7 +3182,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;
kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3195
Quoted 8 lines
kernel/cgroup/cpuset.c +1
-1@@ -3176,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed
Other than that, looks good to me.
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerRe: [PATCH-next v6 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 47 lines · Ridong Chen
Quoted 33 lines · Waiman Long
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask viacpuset_change_task_nodemask()
2) memory policy binding viampol_rebind_mm()
3) if memory migration is enabled, migrate fromold_mems_allowedto the new node mask viacpuset_migrate_mm().
4) settingold_mems_allowedThese memory actions are done in
cpuset_update_tasks_nodemask()andcpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.In
cpuset_update_tasks_nodemask(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():mems_allowed
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()In
cpuset_attach(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():effective_mems
-old_mems_allowed:effective_memsThese inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The
guarantee_online_mems()function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].Quoted 12 lines
The
guarantee_online_mems()function should only be useful for v1 wheremems_allowedis the same aseffective_mems. With v2, the memory nodes ineffective_memsshould always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected ineffective_memsascpuset_handle_hotplug()has not yet been called fromcpuset_track_online_nodes(). Whencpuset_handle_hotplug()is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing theguarantee_online_mems()call by just usingcs->effective_memsshould be fine.
Quoted 1 line
The message should be updated.
Could you be more specific about what message are you referring to?
Quoted 99 lines
Quoted 87 lines
Let use the following setup for both of them and make them consistent.
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()So for v2, it is effectively all
effective_mems. For v1,mpol_rebind_mm()usesmems_allowedwhich may differ from whatguarantee_online_mems()returns.Signed-off-by Waiman Long <[email protected]>--- kernel/cgroup/cpuset.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)kernel/cgroup/cpuset.c +20
-7@@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,* Return in *pmask the portion of a cpusets's mems_allowed that * are online, with memory. If none are online with memory, walk * up the cpuset hierarchy until we find one that does have some -* online mems. The top cpuset always has some mems online. +* online mems. The top cpuset always has some mems online. With v2, +* effective_mems should always contain online memory nodes except +* during the transition period where a memory node hotunplug operation +* is in progress. * * One way or another, we guarantee to return some non-empty subset * of node_states[N_MEMORY]. @@ -498,6 +501,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,*/ static void guarantee_online_mems (struct cpuset *cs,nodemask_t *pmask) { +if (cpuset_v2 ()) { + *pmask = cs->effective_mems; +return ; + }while (!nodes_and (*pmask, cs->effective_mems, node_states[N_MEMORY ])) cs =parent_cs (cs); }@@ -2616,6 +2623,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask (struct cpuset *cs) {@@ -2624,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)struct task_struct *task; cpuset_being_rebound = cs;/* causes mpol_dup() rebind */ -guarantee_online_mems (cs, &newmems);/* @@ -2650,7 +2663,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate =is_memory_migrate (cs); -mpol_rebind_mm (mm, &cs->mems_allowed); +mpol_rebind_mm (mm, &cs->effective_mems);if (migrate)cpuset_migrate_mm (mm, &cs->old_mems_allowed, &newmems);else @@ -3148,17 +3161,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus +* and mems. In that case, we can optimize out by skipping the task +* iteration and update. */ -if (cpuset_v2 () && !cpus_updated && !mems_updated) { +if (cpuset_v2 ()) {kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3161
Quoted 10 lines
kernel/cgroup/cpuset.c +4
-3cpuset_attach_nodemask_to = cs->effective_mems; - goto out; +if (!cpus_updated && !mems_updated) + goto out; + }else { +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); } -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); -Nit.
Quoted 5 lines
I prefer:
guarantee_online_mems(cs, &cpuset_attach_nodemask_to); if (cpuset_v2() && !cpus_updated && !mems_updated) goto out;
This is fixed in patch 7. I may have to send a new version of the patch 7 integrated back into individual patches.
Cheers,
Longman
Quoted 24 lines
Quoted 12 lines
kernel/cgroup/cpuset.c +0
-1cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task);@@ -3168,7 +3182,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)* if there is no change in effective_mems and CS_MEMORY_MIGRATE is * not set. */ - cpuset_attach_nodemask_to = cs->effective_mems;if (!is_memory_migrate (cs) && !mems_updated) goto out;kernel/cgroup/cpuset.c static void cpuset_attach(struct cgroup_taskset *tset) · line 3195
Quoted 8 lines
kernel/cgroup/cpuset.c +1
-1@@ -3176,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)struct mm_struct *mm =get_task_mm (leader);if (mm) { -mpol_rebind_mm (mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm (mm, &cs->effective_mems);/* * old_mems_allowed is the same with mems_allowed Other than that, looks good to me.
Reviewed-by Ridong Chen <[email protected]>
Ridong Chen
reviewerRe: [PATCH-next v6 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
Quoted 48 lines · Waiman Long
Quoted 47 lines · Ridong Chen
Quoted 45 lines · Waiman Long
Whenever memory node mask is changed, there are 4 places where the node mask has to be updated or used.
1) task's node mask viacpuset_change_task_nodemask()
2) memory policy binding viampol_rebind_mm()
3) if memory migration is enabled, migrate fromold_mems_allowedto the new node mask viacpuset_migrate_mm().
4) settingold_mems_allowedThese memory actions are done in
cpuset_update_tasks_nodemask()andcpuset_attach(). However there are inconsistencies in what node masks are being used in these 2 functions.In
cpuset_update_tasks_nodemask(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():mems_allowed
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()In
cpuset_attach(),
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():effective_mems
-old_mems_allowed:effective_memsThese inconsistencies dates back to quite a long time ago and it is hard to say what should be the correct values.
The
guarantee_online_mems()function returns a node mask from current or an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE]. However, node in node_states[N_ONLINE] may not have memory. So node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].The
guarantee_online_mems()function should only be useful for v1 wheremems_allowedis the same aseffective_mems. With v2, the memory nodes ineffective_memsshould always be a subset of node_states[N_MEMORY]. The only time that may not be true is when a memory hot-unplug operation is in progress and a memory node is removed from node_states[N_MEMORY] but not yet reflected ineffective_memsascpuset_handle_hotplug()has not yet been called fromcpuset_track_online_nodes(). Whencpuset_handle_hotplug()is called later, the memory node setting of the relevant cpusets and tasks will be updated. So replacing theguarantee_online_mems()call by just usingcs->effective_memsshould be fine.
Quoted 2 lines
Quoted 1 line
The message should be updated.
Could you be more specific about what message are you referring to?
After this patch, guarantee_online_mems() for v2 simply returns cs->effective_mems directly. So the paragraph:
"The guarantee_online_mems() function should only be useful for v1
where mems_allowed is the same as effective_mems."
is no longer accurate — guarantee_online_mems() is now useful for both v1 and v2, just with different behavior. For v2 it returns effective_mems directly; for v1 it walks up the hierarchy to find an ancestor with online memory nodes.
Similarly, "So replacing the guarantee_online_mems() call by just using cs->effective_mems should be fine" describes the old reasoning, but the patch actually keeps guarantee_online_mems() calls in place and makes the function itself return effective_mems for v2. The commit message should reflect what the patch actually does.
Quoted 139 lines
Quoted 103 lines
Quoted 93 lines
Let use the following setup for both of them and make them consistent.
-cpuset_change_task_nodemask():guarantee_online_mems()
-mpol_rebind_mm():effective_mems
-cpuset_migrate_mm():guarantee_online_mems()
-old_mems_allowed:guarantee_online_mems()So for v2, it is effectively all
effective_mems. For v1,mpol_rebind_mm()
usesmems_allowedwhich may differ from whatguarantee_online_mems()returns.Signed-off-by Waiman Long <[email protected]>---
kernel/cgroup/cpuset.c| 37 +++++++++++++++++++++++++------------1 file changed, 25 insertions(+), 12 deletions(-)kernel/cgroup/cpuset.c +4
-1@@ -489,7 +489,10 @@ static void guarantee_active_cpus(structtask_struct *tsk, * Return in *pmask the portion of a cpusets's mems_allowed that * are online, with memory. If none are online with memory, walk * up the cpuset hierarchy until we find one that does have some - * online mems. The top cpuset always has some mems online. + * online mems. The top cpuset always has some mems online. With v2, + * effective_mems should always contain online memory nodes except + * during the transition period where a memory node hotunplug operation + * is in progress. * * One way or another, we guarantee to return some non-empty subset* of node_states[N_MEMORY].
kernel/cgroup/cpuset.c +0
-0@@ -498,6 +501,10 @@ static void guarantee_active_cpus(structtask_struct *tsk, */ static void
guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
{
+ if (cpuset_v2()) {
+ *pmask =cs->effective_mems;
+ return;
+ }
while (!nodes_and(*pmask,cs->effective_mems, node_states[N_MEMORY]))
cs =parent_cs(cs);
}kernel/cgroup/cpuset.c +7
-0@@ -2616,6 +2623,13 @@ static void *cpuset_being_rebound;* Iterate through each task of @cs updating its mems_allowed to the * effective cpuset's. As this function is called with cpuset_mutex held, * cpuset membership stays stable. +* +* - cpuset_change_task_nodemask(): guarantee_online_mems() +* - mpol_rebind_mm(): effective_mems +* - cpuset_migrate_mm(): guarantee_online_mems() +* - old_mems_allowed: guarantee_online_mems() +* +* For v2, guarantee_online_mems() should just return effective_mems. */ void cpuset_update_tasks_nodemask(struct cpuset *cs){
kernel/cgroup/cpuset.c +0
-0@@ -2624,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) struct task_struct *task;
cpuset_being_rebound= cs; /* causesmpol_dup()rebind */ -
guarantee_online_mems(cs, &newmems);
/*kernel/cgroup/cpuset.c +1
-0@@ -2650,7 +2663,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs) migrate = is_memory_migrate(cs); - mpol_rebind_mm(mm, &cs->mems_allowed); + mpol_rebind_mm(mm, &cs->effective_mems); if (migrate) cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);else
kernel/cgroup/cpuset.c +1
-3@@ -3148,17 +3161,18 @@ static void cpuset_attach(struct cgroup_taskset *tset) /** In the default hierarchy, enabling cpuset in the child cgroups -* will trigger a number of cpuset_attach() calls with no change -* in effective cpus and mems. In that case, we can optimize out -* by skipping the task iteration and update. +* will trigger a cpuset_attach() call with no change in effective cpus
+ * and mems. In that case, we can optimize out by skipping the task + * iteration and update.
*/
- if (cpuset_v2()&& !cpus_updated && !mems_updated) {
+ if (cpuset_v2()) {
cpuset_attach_nodemask_to=cs->effective_mems;Quoted 8 lines
- goto out;
+ if (!cpus_updated && !mems_updated)
+ goto out;
+ } else {
+guarantee_online_mems(cs, &cpuset_attach_nodemask_to); }
-guarantee_online_mems(cs, &cpuset_attach_nodemask_to); -Nit.
Quoted 5 lines
I prefer:
guarantee_online_mems(cs,
&cpuset_attach_nodemask_to); if (cpuset_v2()&& !cpus_updated && !mems_updated) goto out;This is fixed in patch 7. I may have to send a new version of the patch 7 integrated back into individual patches.
Cheers,
LongmanQuoted 23 lines
Quoted 11 lines
cgroup_taskset_for_each(task, css, tset)cpuset_attach_task(cs, task);
@@ -3168,7 +3182,6 @@ static voidcpuset_attach(struct cgroup_taskset *tset)
* if there is no change ineffective_memsandCS_MEMORY_MIGRATEis
* not set.
*/
-cpuset_attach_nodemask_to=cs->effective_mems; if (!is_memory_migrate(cs) && !mems_updated) goto out;Quoted 8 lines
@@ -3176,7 +3189,7 @@ static void
cpuset_attach(struct cgroup_taskset *tset)
structmm_struct*mm =get_task_mm(leader); if (mm) {
-mpol_rebind_mm(mm, &cpuset_attach_nodemask_to); +mpol_rebind_mm(mm, &cs->effective_mems); /*
*old_mems_allowedis the same withmems_allowedOther than that, looks good to me.
Reviewed-by Ridong Chen <[email protected]>
Signature · 2 lines
Best regards, Ridong
Waiman Long
maintainerPATCH v6 2/6 cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
In reply to Waiman LongExtract the DL bandwidth allocation code in cpuset_attach() to a new cpuset_reserve_dl_bw() helper to simplify code.
No functional change is expected.
--- kernel/cgroup/cpuset.c | 53 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 23 deletions(-)
kernel/cgroup/cpuset.c +30 -23
@@ -2994,6 +2994,25 @@ static int cpuset_can_attach_check(struct cpuset *cs)return 0 ; } +static int cpuset_reserve_dl_bw (struct cpuset *cs) +{ +int cpu, ret; + +if (!cs->sum_migrate_dl_bw) +return 0 ; + + cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) +return -EINVAL ; + + ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); +if (ret) +return ret; + + cs->dl_bw_cpu = cpu; +return 0 ; +} +static void reset_migrate_dl_data (struct cpuset *cs) { cs->nr_migrate_dl_tasks =0 ;
@@ -3008,7 +3027,7 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)struct cpuset *cs, *oldcs;struct task_struct *task;bool setsched_check; -int cpu, ret; +int ret;/* used later by cpuset_attach() */ cpuset_attach_old_cs =task_cs (cgroup_taskset_first (tset, &css));
@@ -3064,31 +3083,19 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) } } -if (!cs->sum_migrate_dl_bw) - goto out_success; - - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) { - ret = -EINVAL ; - goto out_unlock; - } - - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) - goto out_unlock; - - cs->dl_bw_cpu = cpu; - -out_success: -/* -* Mark attach is in progress. This makes validate_change() fail -* changes which zero cpus/mems_allowed. -*/ - cs->attach_in_progress++; + ret =cpuset_reserve_dl_bw (cs); out_unlock: -if (ret) +if (ret) {reset_migrate_dl_data (cs); + }else { +/* +* Mark attach is in progress. This makes validate_change() fail +* changes which zero cpus/mems_allowed. +*/ + cs->attach_in_progress++; + } +mutex_unlock (&cpuset_mutex);return ret; }
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v6 3/6 cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
In reply to Waiman LongExpand the scope of cpuset_can_attach_check() by including the setting of setsched flag inside cpuset_can_attach_check() with the new @oldcs and @psetsched argument. As cpuset_can_attach_check() is also called from cpuset_can_fork(), set the new arguments to NULL from that caller.
--- kernel/cgroup/cpuset.c | 52 ++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 22 deletions(-)
kernel/cgroup/cpuset.c +30 -22
@@ -2985,12 +2985,39 @@ static struct cpuset *cpuset_attach_old_cs;* For v1, cpus_allowed and mems_allowed can't be empty. * For v2, effective_cpus can't be empty. * Note that in v1, effective_cpus = cpus_allowed. +* +* Also set the boolean flag passed in by @psetsched depending on if +* security_task_setscheduler() call is needed and @oldcs is not NULL. */ -static int cpuset_can_attach_check (struct cpuset *cs) +static int cpuset_can_attach_check (struct cpuset *cs,struct cpuset *oldcs, +bool *psetsched) {if (cpumask_empty (cs->effective_cpus) || (!is_in_v2_mode () &&nodes_empty (cs->mems_allowed)))return -ENOSPC ; + +if (!oldcs) +return 0 ; + +/* +* Skip rights over task setsched check in v2 when nothing changes, +* migration permission derives from hierarchy ownership in +* cgroup_procs_write_permission()). +*/ + *psetsched = !cpuset_v2 () || + !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || + !nodes_equal (cs->effective_mems, oldcs->effective_mems); + +/* +* A v1 cpuset with tasks will have no CPU left only when CPU hotplug +* brings the last online CPU offline as users are not allowed to empty +* cpuset.cpus when there are active tasks inside. When that happens, +* we should allow tasks to migrate out without security check to make +* sure they will be able to run after migration. +*/ +if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) + *psetsched = false; +return 0 ; }
@@ -3037,29 +3064,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; -/* -* Skip rights over task setsched check in v2 when nothing changes, -* migration permission derives from hierarchy ownership in -* cgroup_procs_write_permission()). -*/ - setsched_check = !cpuset_v2 () || - !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus) || - !nodes_equal (cs->effective_mems, oldcs->effective_mems); - -/* -* A v1 cpuset with tasks will have no CPU left only when CPU hotplug -* brings the last online CPU offline as users are not allowed to empty -* cpuset.cpus when there are active tasks inside. When that happens, -* we should allow tasks to migrate out without security check to make -* sure they will be able to run after migration. -*/ -if (!is_in_v2_mode () &&cpumask_empty (oldcs->effective_cpus)) - setsched_check = false; -cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret)
@@ -3604,7 +3612,7 @@ static int cpuset_can_fork(struct task_struct *task, struct css_set *cset)mutex_lock (&cpuset_mutex);/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs); + ret =cpuset_can_attach_check (cs,NULL ,NULL );if (ret) goto out_unlock;
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v6 4/6 cgroup/cpuset: Make cpuset_attach_old_cs track task group leaders
In reply to Waiman LongThere are two possible ways that migration of tasks from multiple source cpusets to a target cpuset can happen. Either a multithread application with threads in different cpusets is wholely moved to a new cpuset or disabling of v2 cpuset controller will move all the tasks in child cpusets to the parent cpuset.
In the former case, it is the mm setting of the group leader that really matters. So cpuset_attach_old_cs should track the oldcs of the thread leader. In the latter case, effective_mems of child cpusets must always be a subset of the parent. So no real page migration will be necessary no matter which child cpuset is selected as cpuset_attach_old_cs.
IOW, cpuset_attach_old_cs should be updated to match the latest task group leader in cpuset_can_attach(), but fall back to that of the first task if there is no group leader in the taskset.
--- kernel/cgroup/cpuset.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
kernel/cgroup/cpuset.c +25 -0
@@ -2978,6 +2978,10 @@ static int update_prstate(struct cpuset *cs, int new_prs)return 0 ; } +/* +* cpuset_can_attach() and cpuset_attach() specific internal data +* Protected by cpuset_mutex +*/ static struct cpuset *cpuset_attach_old_cs;/*
@@ -3068,11 +3072,32 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)if (ret) goto out_unlock; +/* +* The cpuset_attach_old_cs is used mainly by cpuset_migrate_mm() to get +* the old_mems_allowed value. There are two ways that many-to-one +* cpuset migration can happen: +* 1) A multithread application with threads in different cpusets is +* wholely migrated to a new cpuset. +* 2) Disabling v2 cpuset controller will move all the tasks in child +* cpusets to the parent cpuset. +* +* In the former case, it is the mm setting of the group leader that +* really matters. So cpuset_attach_old_cs should track the oldcs of the +* group leader. It falls back to the oldcs of the first task if there +* is no group leader in the taskset. In the latter case, effective_mems +* of child cpusets must always be a subset of the parent. So no real +* page migration will be necessary no matter which child cpuset is +* selected as cpuset_attach_old_cs. +*/ cgroup_taskset_for_each (task, css, tset) { ret =task_can_attach (task);if (ret) goto out_unlock; +/* Update cpuset_attach_old_cs to the latest group leader */ +if (task == task->group_leader) + cpuset_attach_old_cs =task_cs (task); +if (setsched_check) { ret =security_task_setscheduler (task);if (ret)
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v6 5/6 cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task()
In reply to Waiman LongThe cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task().
When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork().
- cpuset_attach_old_cs
- attach_cpus_updated
- attach_mems_updated
- queue_task_work
--- kernel/cgroup/cpuset.c | 103 ++++++++++++++++++++++++----------------- 1 file changed, 60 insertions(+), 43 deletions(-)
kernel/cgroup/cpuset.c +60 -43
@@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, int new_prs)/* * cpuset_can_attach() and cpuset_attach() specific internal data * Protected by cpuset_mutex +* +* The attach_cpus_updated/attach_mems_updated flags are set in either +* cpuset_attach() or cpuset_fork() and used in cpuset_attach_task(). */ static struct cpuset *cpuset_attach_old_cs; +static bool attach_cpus_updated; +static bool attach_mems_updated;/* * Check to see if a cpuset can accept a new task
@@ -3160,9 +3165,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset)*/ static cpumask_var_t cpus_attach;static nodemask_t cpuset_attach_nodemask_to; +static bool queue_task_work;static void cpuset_attach_task (struct cpuset *cs,struct task_struct *task) { +struct mm_struct *mm; +lockdep_assert_cpuset_lock_held ();if (cs != &top_cpuset)
@@ -3176,28 +3184,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task)*/ WARN_ON_ONCE (set_cpus_allowed_ptr (task, cpus_attach)); +if (cpuset_v2 () && !attach_mems_updated) +return ; +cpuset_change_task_nodemask (task, &cpuset_attach_nodemask_to);cpuset1_update_task_spread_flags (cs, task); + +if ((task != task->group_leader) || + (!is_memory_migrate (cs) && !attach_mems_updated)) +return ; + +/* +* Change mm for threadgroup leader. This is expensive and may +* sleep and should be moved outside migration path proper. +*/ + mm =get_task_mm (task); +if (mm) { +struct cpuset *oldcs = cpuset_attach_old_cs; + +mpol_rebind_mm (mm, &cs->effective_mems); + +/* +* old_mems_allowed is the same with mems_allowed +* here, except if this task is being moved +* automatically due to hotplug. In that case +* @mems_allowed has been updated and is empty, so +* @old_mems_allowed is the right nodesets that we +* migrate mm from. +*/ +if (is_memory_migrate (cs)) { +cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, + &cpuset_attach_nodemask_to); + queue_task_work = true; + }else { +mmput (mm); + } + } }static void cpuset_attach (struct cgroup_taskset *tset) {struct task_struct *task; -struct task_struct *leader;struct cgroup_subsys_state *css;struct cpuset *cs;struct cpuset *oldcs = cpuset_attach_old_cs; -bool cpus_updated, mems_updated; -bool queue_task_work = false;cgroup_taskset_first (tset, &css); cs =css_cs (css);lockdep_assert_cpus_held ();/* see cgroup_attach_lock() */ mutex_lock (&cpuset_mutex); - cpus_updated = !cpumask_equal (cs->effective_cpus, - oldcs->effective_cpus); - mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems); + queue_task_work = false; + + attach_cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); + attach_mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems);/* * In the default hierarchy, enabling cpuset in the child cgroups
@@ -3207,7 +3247,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)*/ if (cpuset_v2 ()) { cpuset_attach_nodemask_to = cs->effective_mems; -if (!cpus_updated && !mems_updated) +if (!attach_cpus_updated && !attach_mems_updated) goto out; }else {guarantee_online_mems (cs, &cpuset_attach_nodemask_to);
@@ -3216,38 +3256,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task); -/* -* Change mm for all threadgroup leaders. This is expensive and may -* sleep and should be moved outside migration path proper. Skip it -* if there is no change in effective_mems and CS_MEMORY_MIGRATE is -* not set. -*/ -if (!is_memory_migrate (cs) && !mems_updated) - goto out; - -cgroup_taskset_for_each_leader (leader, css, tset) { -struct mm_struct *mm =get_task_mm (leader); - -if (mm) { -mpol_rebind_mm (mm, &cs->effective_mems); - -/* -* old_mems_allowed is the same with mems_allowed -* here, except if this task is being moved -* automatically due to hotplug. In that case -* @mems_allowed has been updated and is empty, so -* @old_mems_allowed is the right nodesets that we -* migrate mm from. -*/ -if (is_memory_migrate (cs)) { -cpuset_migrate_mm (mm, &oldcs->old_mems_allowed, - &cpuset_attach_nodemask_to); - queue_task_work = true; - }else -mmput (mm); - } - } - out:if (queue_task_work)schedule_flush_migrate_mm ();
@@ -3681,15 +3689,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset)*/ static void cpuset_fork (struct task_struct *task) { -struct cpuset *cs; -bool same_cs; +struct cpuset *cs, *oldcs;rcu_read_lock (); cs =task_cs (task); - same_cs = (cs ==task_cs (current)); + oldcs =task_cs (current);rcu_read_unlock (); -if (same_cs) { +if (cs == oldcs) {if (cs == &top_cpuset)return ;
@@ -3701,7 +3708,17 @@ static void cpuset_fork(struct task_struct *task)/* CLONE_INTO_CGROUP */ mutex_lock (&cpuset_mutex);guarantee_online_mems (cs, &cpuset_attach_nodemask_to); +/* +* Assume CPUs and memory nodes are updated +* A CLONE_INTO_CGROUP operation should have taken the cgroup mutex +* and so there shouldn't be a competing cpuset_attach() operation. +*/ + attach_cpus_updated = attach_mems_updated = true; + queue_task_work = false; + cpuset_attach_old_cs = oldcs;cpuset_attach_task (cs, task); +if (queue_task_work) +schedule_flush_migrate_mm ();dec_attach_in_progress_locked (cs);mutex_unlock (&cpuset_mutex);
Signature · 1 line
2.54.0
Waiman Long
maintainerPATCH v6 6/6 cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
In reply to Waiman LongWith cgroup v2, the cgroup_taskset structure passed into the cgroup can_attach() and attach() methods can contain task migration data with multiple destination or source cpusets when the cpuset controller is enabled or disabled respectively.
Since cpuset is threaded in both v1 and v2, another possible way to cause many-to-one migration is to move the whole process with multiple threads in different cpuset enabled threaded cgroups into another cpuset enabled cgroup.
The current cpuset_can_attach() and cpuset_attach() functions still expect task migration is from one source cpuset to one destination cpuset. This has been the case since cpuset was enabled for cgroup v2 in commit 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default hierarchy").
This problem is less an issue when enabling the cpuset controller as all the newly created child cpusets will have exactly the same set of CPUs and memory nodes except when deadline tasks are involved in migration as the deadline task accounting data can be off.
It can be more problematic when the cpuset controller is disabled as their set of CPUs and memory nodes may differ from their parent or with the moving of multi-threaded process from different threaded cgroups.
Fix that by tracking the set of source (old) and destination cpusets in singly linked lists and iterating them all to properly update the internal data. Also keep the current cs and oldcs variables up-to-date with the css and task iterators.
This commit assumes that the set of source and destination cpusets are distnct. IOW, the cgroup core shouldn't move a task from a given cpuset back to itself. So a given cpuset cannot be both a source and a destination cpuset. Running an experiment by moving a multithreaded process with threads in multiple cpusets in threaded cgroups back to its domain cgroup confirms that only threads that need to be moved are included into the cgroup_taskset passed to cpuset_can_attach(). A WARN_ON_ONCE() is added to cpuset_can_attach_check() to make sure that this should always be the case.
To ensure proper DL tasks accounting, the nr_migrate_dl_tasks in both the source and destination cpusets are decremented/incremented with their values added to nr_deadline_tasks when the migration is successful.
--- kernel/cgroup/cpuset-internal.h | 6 + kernel/cgroup/cpuset.c | 216 ++++++++++++++++++++++++-------- 2 files changed, 172 insertions(+), 50 deletions(-)
kernel/cgroup/cpuset-internal.h +6 -0
@@ -161,6 +161,12 @@ struct cpuset {*/ bool remote_partition; +/* +* cpuset_can_attach() and cpuset_attach() specific data +*/ +bool attach_node_in_llist; +struct llist_node attach_node; +/* * number of SCHED_DEADLINE tasks attached to this cpuset, so that we * know when to rebuild associated root domain bandwidth information.
kernel/cgroup/cpuset.c +166 -50
316 lines not on this page · raw message
Signature · 1 line
2.54.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Waiman Long
maintainerQuoted 35 lines · On 6/4/26 11:27 AM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Unprivileged users can trigger a kernelWARN_ON_ONCE(local DoS) during a valid cgroup migration. --commit
5ed00e2c5b52acfe513b327768730d3015a032f0
Author: Waiman Long <[email protected]>
cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()This commit updates cpuset migration to track multiple source and destination cpusets using singly linked lists. It handles many-to-many migrations that can occur when moving multithreaded processes across threaded cgroups. Proper accounting for deadline tasks is also ensured across multiple cpusets.
Quoted 3 lines
kernel/cgroup/cpuset.c +0
-0[ ... ]
kernel/cgroup/cpuset.c static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs, · line 3023
Quoted 10 lines
kernel/cgroup/cpuset.c +6
-0@@ -3011,6 +3015,23 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,if (!oldcs)return 0 ; +/* +* The cgroup core shouldn't migrate a task to its original cpuset. +* In the unlikely event that it happens, it will be treated as a +* destination cpuset only and the attach_conflict flag will be set. +*/ +if (WARN_ON_ONCE (cs == oldcs))[Severity: High]
Can thisWARN_ON_ONCEbe triggered by unprivileged users during a valid multithreaded process migration?In cgroup v2, a multithreaded process can be spread across multiple threaded cgroups. If the entire process is migrated via cgroup.procs, the migration taskset will contain all threads.
Quoted 7 lines
If a thread moves between two cgroups that share the same cpuset css (for example, if neither explicitly enables the cpuset controller and they share an ancestor's cpuset css), its oldcs and newcs will be identical. When
cpuset_can_attach_check()evaluates this thread, cs will equal oldcs.If an unprivileged user with cgroup delegation triggers this, wouldn't it cause a local DoS on systems with
panic_on_warnenabled?
panic_on_warn is for debugging purpose only. It shouldn't be enabled in a production system. AFAICS, no further change will be needed.
Cheers,
Longman
Quoted 7 lines
Quoted 7 lines
kernel/cgroup/cpuset.c +6
-0+ attach_conflict = true; + +if (!cs->attach_node_in_llist) { +llist_add (&cs->attach_node, &dst_cs_head); + cs->attach_node_in_llist = true; + }
Waiman Long
maintainer[PATCH-next v6 7/6] cgroup/cpuset: Set old_mems_allowed from guarantee_online_mems() consistently
In reply to Waiman LongAn earlier patch has added an optimization in guarantee_online_mems() to just return effective_mems for v2. However there is a short window during memory hotunplug operation that it can return a nodemask with no online node leading to possible memory OOM. To avoid this scenario, though highly unlikely, the optimization is dropped.
Also set old_mems_allowed of the cpuset structure consistently with the output of guarantee_online_mems() whenever an attach or a related operation is in progress.
--- kernel/cgroup/cpuset.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/cgroup/cpuset.c +14 -14
@@ -502,10 +502,6 @@ static void guarantee_active_cpus(struct task_struct *tsk,*/ static void guarantee_online_mems (struct cpuset *cs,nodemask_t *pmask) { -if (cpuset_v2 ()) { - *pmask = cs->effective_mems; -return ; - }while (!nodes_and (*pmask, cs->effective_mems, node_states[N_MEMORY ])) cs =parent_cs (cs); }
@@ -3350,22 +3346,24 @@ static void cpuset_attach(struct cgroup_taskset *tset) } cs =css_cs (css); +guarantee_online_mems (cs, &cpuset_attach_nodemask_to); +/* * In the default hierarchy, enabling cpuset in the child cgroups * will trigger a cpuset_attach() call with no change in effective cpus * and mems. In that case, we can optimize out by skipping the task * iteration and update, but the destination cpuset list is iterated to -* set old_mems_sllowed. +* set old_mems_allowed. */ -if (cpuset_v2 ()) { - cpuset_attach_nodemask_to = cs->effective_mems; -if (!attach_cpus_updated && !attach_mems_updated) { -llist_for_each_entry (cs, dst_cs_head.first, attach_node) - cs->old_mems_allowed = cs->effective_mems; - goto out; +if (cpuset_v2 () && !attach_cpus_updated && !attach_mems_updated) { +struct cpuset *tcs; + +llist_for_each_entry (tcs, dst_cs_head.first, attach_node) { +if (tcs == cs) +continue ; +guarantee_online_mems (tcs, &tcs->old_mems_allowed); } - }else { -guarantee_online_mems (cs, &cpuset_attach_nodemask_to); + goto out; }cgroup_taskset_for_each (task, css, tset) {
@@ -3381,8 +3379,8 @@ static void cpuset_attach(struct cgroup_taskset *tset)if (queue_task_work)schedule_flush_migrate_mm (); - cs->old_mems_allowed = cpuset_attach_nodemask_to; out: + cs->old_mems_allowed = cpuset_attach_nodemask_to;reset_attach_in_progress ();clear_attach_data (false);mutex_unlock (&cpuset_mutex);
@@ -3824,6 +3822,8 @@ static void cpuset_fork(struct task_struct *task)/* CLONE_INTO_CGROUP */ mutex_lock (&cpuset_mutex);guarantee_online_mems (cs, &cpuset_attach_nodemask_to); + cs->old_mems_allowed = cpuset_attach_nodemask_to; +/* * Assume CPUs and memory nodes are updated * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
Signature · 1 line
2.54.0