sched, cgroup/cpuset: Keep user set cpus affinity
Subsystems (3)
Waiman Long
maintainerv3 cover sched, cgroup/cpuset: Keep user set cpus affinity
v3:
- Attach a new patch 2 to introduce a copy_user_cpus_mask() to copy out user masks with lock protection & use it in patch 3.
v2:
- Rework the v1 patch by extending the semantics of user_cpus_ptr to store user set cpus affinity and keeping to it as much as possible.
The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity") which uses it narrowly to allow keeping cpus affinity intact with asymmetric cpu setup.
This patchset extends user_cpus_ptr to store user set cpus affinity via sched_setaffinity() API. With that information avaiable, it will enable cpuset to keep cpus afinity as close to what the user wants as possible within the cpu list constraint of the current cpuset. Otherwise some change to the cpuset hierarchy may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before.
It also means that once sched_setaffinity() is called, user_cpus_ptr will remain allocated until the task exits.
Waiman Long (3):
sched: Use user_cpus_ptr for saving user provided cpumask in
sched_setaffinity()
sched: Provide copy_user_cpus_mask() to copy out user mask cgroup/cpuset: Keep user set cpus affinity
include/linux/sched.h | 2 + kernel/cgroup/cpuset.c | 28 ++++++++++++- kernel/sched/core.c | 89 ++++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 1 - 4 files changed, 97 insertions(+), 23 deletions(-)
Signature · 1 line
2.31.1
Waiman Long
maintainerPATCH v3 1/3 sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric cpu setup. This patch extends its usage to save user provided cpumask when sched_setaffinity() is called for all arches.
To preserve the existing arm64 use case, a new cpus_affinity_set flag is added to differentiate if user_cpus_ptr is set up by sched_setaffinity() or by force_compatible_cpus_allowed_ptr(). user_cpus_ptr
set by sched_setaffinity() has priority and won't be overwritten by force_compatible_cpus_allowed_ptr() or relax_compatible_cpus_allowed_ptr().
As a call to sched_setaffinity() will no longer clear user_cpus_ptr but set it instead, the SCA_USER flag is no longer necessary and can be removed.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 71 +++++++++++++++++++++++++++++++------------ kernel/sched/sched.h | 1 - 3 files changed, 52 insertions(+), 21 deletions(-)
include/linux/sched.h +1 -0
@@ -816,6 +816,7 @@ struct task_struct {unsigned int policy;int nr_cpus_allowed; +int cpus_affinity_set;const cpumask_t *cpus_ptr;cpumask_t *user_cpus_ptr;cpumask_t cpus_mask;
kernel/sched/core.c +51 -20
144 lines not on this page · raw message
kernel/sched/sched.h +0 -1
@@ -2283,7 +2283,6 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);#define SCA_CHECK 0x01#define SCA_MIGRATE_DISABLE 0x02#define SCA_MIGRATE_ENABLE 0x04 -#define SCA_USER 0x08#ifdef CONFIG_SMP
Signature · 1 line
2.31.1
Peter Zijlstra
maintainerRe: [PATCH v3 1/3] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
Quoted 6 lines · Waiman Long
The
user_cpus_ptrfield is added by commitb90ca8badbd1("sched:
Introducetask_struct::user_cpus_ptrto track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric cpu setup. This patch extends its usage to save user provided cpumask whensched_setaffinity()is called for all arches.
Quoted 6 lines
To preserve the existing arm64 use case, a new
cpus_affinity_setflag is added to differentiate ifuser_cpus_ptris set up bysched_setaffinity()or byforce_compatible_cpus_allowed_ptr().user_cpus_ptr
set bysched_setaffinity()has priority and won't be overwritten byforce_compatible_cpus_allowed_ptr()orrelax_compatible_cpus_allowed_ptr().
What why ?! The only possible case where restrict_cpus_allowed_ptr() will now need that weird new state is when the affinity has never been set before, in that case cpus_ptr should be possible_mask.
Please just make a single consistent rule and don't make weird corner cases like this.
Waiman Long
maintainerRe: [PATCH v3 1/3] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
Quoted 13 lines · Peter Zijlstra
Quoted 6 lines · Waiman Long
The
user_cpus_ptrfield is added by commitb90ca8badbd1("sched:
Introducetask_struct::user_cpus_ptrto track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric cpu setup. This patch extends its usage to save user provided cpumask whensched_setaffinity()is called for all arches.Quoted 6 lines
To preserve the existing arm64 use case, a new
cpus_affinity_setflag is added to differentiate ifuser_cpus_ptris set up bysched_setaffinity()or byforce_compatible_cpus_allowed_ptr().user_cpus_ptr
set bysched_setaffinity()has priority and won't be overwritten byforce_compatible_cpus_allowed_ptr()orrelax_compatible_cpus_allowed_ptr().
Quoted 4 lines
What why ?! The only possible case where
restrict_cpus_allowed_ptr()will now need that weird new state is when the affinity has never been set before, in that casecpus_ptrshould bepossible_mask.
Since I don't have a full history for this particular patch series that add user_cpus_ptr, I am hesitant to change the current behavior for arm64 systems. However, given the statement that user_cpus_ptr is for tracking "requested affinity" which I assume is when user applications call sched_setaffinity(). It does make sense we may not really need this if sched_setaffinity() is never called.
Quoted 2 lines
Please just make a single consistent rule and don't make weird corner cases like this.
I will take a closer look to try to simplify the rule here.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v3 1/3] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
Quoted 20 lines · Waiman Long
Quoted 13 lines · Peter Zijlstra
Quoted 6 lines · Waiman Long
The
user_cpus_ptrfield is added by commitb90ca8badbd1("sched:
Introducetask_struct::user_cpus_ptrto track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric cpu setup. This patch extends its usage to save user provided cpumask whensched_setaffinity()is called for all arches.Quoted 6 lines
To preserve the existing arm64 use case, a new
cpus_affinity_setflag is added to differentiate ifuser_cpus_ptris set up bysched_setaffinity()or byforce_compatible_cpus_allowed_ptr().user_cpus_ptr
set bysched_setaffinity()has priority and won't be overwritten byforce_compatible_cpus_allowed_ptr()orrelax_compatible_cpus_allowed_ptr().Quoted 4 lines
What why ?! The only possible case where
restrict_cpus_allowed_ptr()will now need that weird new state is when the affinity has never been set before, in that casecpus_ptrshould bepossible_mask.
Quoted 6 lines
Since I don't have a full history for this particular patch series that add
user_cpus_ptr, I am hesitant to change the current behavior for arm64 systems. However, given the statement thatuser_cpus_ptris for tracking "requested affinity" which I assume is when user applications callsched_setaffinity(). It does make sense we may not really need this ifsched_setaffinity()is never called.
So it comes from the asymmetric arm stuff, where only little cores can still run arm32 code. This means that on those machines, 32bit code needs to be contrained so a subset of CPUs.
A direct consequence of that was that if you have any 32bit program in your process hierarchy, you loose the big cores from you affinity mask.
For some reason that wasn't popular.. Hence the save/restore of cpumasks.
Quoted 4 lines
Quoted 2 lines
Please just make a single consistent rule and don't make weird corner cases like this.
I will take a closer look to try to simplify the rule here.
I think something like:
mask = p->user_cpus_ptr; if (!mask) mask = &init_task.cpus_mask; // impose cpuset masks
should 'just-work'.
Waiman Long
maintainerRe: [PATCH v3 1/3] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
Quoted 25 lines · Peter Zijlstra
Quoted 18 lines · Waiman Long
Quoted 13 lines · Peter Zijlstra
Quoted 6 lines · Waiman Long
The
user_cpus_ptrfield is added by commitb90ca8badbd1("sched:
Introducetask_struct::user_cpus_ptrto track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric cpu setup. This patch extends its usage to save user provided cpumask whensched_setaffinity()is called for all arches.Quoted 6 lines
To preserve the existing arm64 use case, a new
cpus_affinity_setflag is added to differentiate ifuser_cpus_ptris set up bysched_setaffinity()or byforce_compatible_cpus_allowed_ptr().user_cpus_ptr
set bysched_setaffinity()has priority and won't be overwritten byforce_compatible_cpus_allowed_ptr()orrelax_compatible_cpus_allowed_ptr().Quoted 4 lines
What why ?! The only possible case where
restrict_cpus_allowed_ptr()will now need that weird new state is when the affinity has never been set before, in that casecpus_ptrshould bepossible_mask.Quoted 6 lines
Since I don't have a full history for this particular patch series that add
user_cpus_ptr, I am hesitant to change the current behavior for arm64 systems. However, given the statement thatuser_cpus_ptris for tracking "requested affinity" which I assume is when user applications callsched_setaffinity(). It does make sense we may not really need this ifsched_setaffinity()is never called.
Quoted 8 lines
So it comes from the asymmetric arm stuff, where only little cores can still run arm32 code. This means that on those machines, 32bit code needs to be contrained so a subset of CPUs.
A direct consequence of that was that if you have any 32bit program in your process hierarchy, you loose the big cores from you affinity mask.
For some reason that wasn't popular.. Hence the save/restore of cpumasks.
I am aware of that part of the patch series.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Please just make a single consistent rule and don't make weird corner cases like this.
I will take a closer look to try to simplify the rule here.
I think something like:
Quoted 7 lines
mask = p->user_cpus_ptr; if (!mask) mask = &init_task.cpus_mask; // impose cpuset masksshould 'just-work'.
I think that should work in relax_compatible_cpus_allowed_ptr().
Thanks,
Longman
Waiman Long
maintainerPATCH v3 2/3 sched: Provide copy_user_cpus_mask() to copy out user mask
In reply to Waiman LongSince accessing the content of the user_cpus_ptr requires lock protection to ensure its validity, provide a helper function copy_user_cpus_mask() to facilitate its reading.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
include/linux/sched.h +1 -0
@@ -1831,6 +1831,7 @@ extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_effecextern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node); +extern struct cpumask *copy_user_cpus_mask (struct task_struct *p,struct cpumask *user_mask);extern void release_user_cpus_ptr (struct task_struct *p);extern int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask);extern void force_compatible_cpus_allowed_ptr (struct task_struct *p);
kernel/sched/core.c +18 -0
@@ -2619,6 +2619,24 @@ void release_user_cpus_ptr(struct task_struct *p)kfree (clear_user_cpus_ptr (p)); } +/* +* Return the copied mask pointer or NULL if user mask not available. +*/ +struct cpumask *copy_user_cpus_mask (struct task_struct *p, +struct cpumask *user_mask) +{ +struct rq_flags rf; +struct rq *rq =task_rq_lock (p, &rf); +struct cpumask *mask =NULL ; + +if (p->user_cpus_ptr) { +cpumask_copy (user_mask, p->user_cpus_ptr); + mask = user_mask; + } +task_rq_unlock (rq, p, &rf); +return mask; +} +/* * This function is wildly self concurrent; here be dragons. *
Signature · 1 line
2.31.1
Peter Zijlstra
maintainerRe: [PATCH v3 2/3] sched: Provide copy_user_cpus_mask() to copy out user mask
Quoted 3 lines · Waiman Long
Since accessing the content of the
user_cpus_ptrrequires lock protection to ensure its validity, provide a helper functioncopy_user_cpus_mask()to facilitate its reading.
Sure, but this is atrocious.
Quoted 13 lines
kernel/sched/core.c +7
-0@@ -2619,6 +2619,24 @@ void release_user_cpus_ptr(struct task_struct *p)kfree (clear_user_cpus_ptr (p)); } +/* +* Return the copied mask pointer or NULL if user mask not available. +*/ +struct cpumask *copy_user_cpus_mask (struct task_struct *p, +struct cpumask *user_mask) +{ +struct rq_flags rf;
kernel/sched/core.c void release_user_cpus_ptr(struct task_struct *p) · line 2638
Quoted 10 lines
kernel/sched/core.c +10
-0+struct rq *rq =task_rq_lock (p, &rf); +struct cpumask *mask =NULL ; + +if (p->user_cpus_ptr) { +cpumask_copy (user_mask, p->user_cpus_ptr); + mask = user_mask; + } +task_rq_unlock (rq, p, &rf); +return mask; +}
For reading the mask you only need one of those locks, and I would suggest p->pi_lock is much less contended than rq->lock.
Waiman Long
maintainerRe: [PATCH v3 2/3] sched: Provide copy_user_cpus_mask() to copy out user mask
Quoted 23 lines · Peter Zijlstra
Quoted 3 lines · Waiman Long
Since accessing the content of the
user_cpus_ptrrequires lock protection to ensure its validity, provide a helper functioncopy_user_cpus_mask()to facilitate its reading.Sure, but this is atrocious.
Quoted 17 lines
kernel/sched/core.c +11
-0@@ -2619,6 +2619,24 @@ void release_user_cpus_ptr(struct task_struct *p)kfree (clear_user_cpus_ptr (p)); } +/* +* Return the copied mask pointer or NULL if user mask not available. +*/ +struct cpumask *copy_user_cpus_mask (struct task_struct *p, +struct cpumask *user_mask) +{ +struct rq_flags rf; +struct rq *rq =task_rq_lock (p, &rf); +struct cpumask *mask =NULL ; + +if (p->user_cpus_ptr) {
Quoted 9 lines
kernel/sched/core.c void release_user_cpus_ptr(struct task_struct *p) · line 2638
Quoted 6 lines
kernel/sched/core.c +6
-0+cpumask_copy (user_mask, p->user_cpus_ptr); + mask = user_mask; + } +task_rq_unlock (rq, p, &rf); +return mask; +}For reading the mask you only need one of those locks, and I would suggest
p->pi_lockis much less contended thanrq->lock.
Right. pi_lock should be enough for read access. Will make the change.
Thanks,
Longman
Waiman Long
maintainerPATCH v3 3/3 cgroup/cpuset: Keep user set cpus affinity
In reply to Waiman LongIt was found that any change to the current cpuset hierarchy may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.
That is problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.
Fix this problem by looking at user_cpus_ptr which will be set if cpus affinity have been explicitly set before and use it to restrcit the given cpumask unless there is no overlap. In that case, it will fallback to the given one.
With that change in place, it was verified that tasks that have its cpus affinity explicitly set will not be affected by changes made to the v2 cgroup.subtree_control files.
--- kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
kernel/cgroup/cpuset.c +26 -2
@@ -704,6 +704,30 @@ static int validate_change(struct cpuset *cur, struct cpuset *trial)return ret; } +/* +* Preserve user provided cpumask (if set) as much as possible unless there +* is no overlap with the given mask. +*/ +static int cpuset_set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *mask) +{ +if (p->user_cpus_ptr) { +cpumask_var_t new_mask; + +if (alloc_cpumask_var (&new_mask,GFP_KERNEL ) && +copy_user_cpus_mask (p, new_mask) && +cpumask_and (new_mask, new_mask, mask)) { +int ret =set_cpus_allowed_ptr (p, new_mask); + +free_cpumask_var (new_mask); +return ret; + } +free_cpumask_var (new_mask); + } + +return set_cpus_allowed_ptr (p, mask); +} +#ifdef CONFIG_SMP /* * Helper routine for generate_sched_domains().
@@ -1130,7 +1154,7 @@ static void update_tasks_cpumask(struct cpuset *cs)css_task_iter_start (&cs->css,0 , &it);while ((task =css_task_iter_next (&it))) -set_cpus_allowed_ptr (task, cs->effective_cpus); +cpuset_set_cpus_allowed_ptr (task, cs->effective_cpus);css_task_iter_end (&it); }
@@ -2303,7 +2327,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)* can_attach beforehand should guarantee that this doesn't * fail. TODO: have a better way to handle failure here */ -WARN_ON_ONCE (set_cpus_allowed_ptr (task, cpus_attach)); +WARN_ON_ONCE (cpuset_set_cpus_allowed_ptr (task, cpus_attach));cpuset_change_task_nodemask (task, &cpuset_attach_nodemask_to);cpuset_update_task_spread_flag (cs, task);
Signature · 1 line
2.31.1