sched, cgroup/cpuset: Keep user set cpus affinity
Subsystems (3)
Waiman Long
maintainerv4 cover sched, cgroup/cpuset: Keep user set cpus affinity
v4:
- Update patch 1 to make sched_setaffinity() the only function to update user_cpus_ptr to make the logic simpler and easier to understand. restrict_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() will just use it if present.
v3:
- Attach a new patch 2 to introduce a copy_user_cpus_mask() to copy out user masks with lock protection & use it in patch 3.
v2:
- Rework the v1 patch by extending the semantics of user_cpus_ptr to store user set cpus affinity and keeping to it as much as possible.
The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity") which uses it narrowly to allow keeping cpus affinity intact with asymmetric cpu setup.
This patchset extends user_cpus_ptr to store user set cpus affinity via sched_setaffinity() API. With that information avaiable, it will enable cpuset to keep cpus afinity as close to what the user wants as possible within the cpu list constraint of the current cpuset. Otherwise some change to the cpuset hierarchy may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before.
It also means that once sched_setaffinity() is called, user_cpus_ptr will remain allocated until the task exits.
Waiman Long (3):
sched: Use user_cpus_ptr for saving user provided cpumask in
sched_setaffinity()
sched: Provide copy_user_cpus_mask() to copy out user mask cgroup/cpuset: Keep user set cpus affinity
include/linux/sched.h | 1 + kernel/cgroup/cpuset.c | 28 +++++++++- kernel/sched/core.c | 119 ++++++++++++++++++++++++----------------- kernel/sched/sched.h | 1 - 4 files changed, 98 insertions(+), 51 deletions(-)
Signature · 1 line
2.31.1
Waiman Long
maintainerPATCH v4 1/3 sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()
The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric CPU setup. This patch extends its usage to save user provided cpumask when sched_setaffinity() is called for all arches. With this patch applied, user_cpus_ptr, once allocated after a call to sched_setaffinity(), will only be freed when the task exits.
Since user_cpus_ptr is supposed to be used for "requested affinity", there is actually no point to save current cpu affinity in restrict_cpus_allowed_ptr() if sched_setaffinity() has never been called. Modify the logic to set user_cpus_ptr only in sched_setaffinity() and use it in restrict_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() if defined but not changing it.
This will be some changes in behavior for arm64 systems with asymmetric CPUs in some corner cases. For instance, if sched_setaffinity() has never been called and there is a cpuset change before relax_compatible_cpus_allowed_ptr() is called, its subsequent call will follow what the cpuset allows but not what the previous cpu affinity setting allows.
As a call to sched_setaffinity() will no longer clear user_cpus_ptr but set it instead, the SCA_USER flag is no longer necessary and can be removed.
--- kernel/sched/core.c | 100 ++++++++++++++++++++++--------------------- kernel/sched/sched.h | 1 - 2 files changed, 52 insertions(+), 49 deletions(-)
kernel/sched/core.c +52 -48
@@ -2848,7 +2848,6 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,const struct cpumask *cpu_allowed_mask =task_cpu_possible_mask (p);const struct cpumask *cpu_valid_mask = cpu_active_mask;bool kthread = p->flags &PF_KTHREAD ; -struct cpumask *user_mask =NULL ;unsigned int dest_cpu;int ret =0 ;
@@ -2907,14 +2906,7 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,__do_set_cpus_allowed (p, new_mask, flags); -if (flags &SCA_USER ) - user_mask =clear_user_cpus_ptr (p); - - ret =affine_move_task (rq, p, rf, dest_cpu, flags); - -kfree (user_mask); - -return ret; +return affine_move_task (rq, p, rf, dest_cpu, flags); out:task_rq_unlock (rq, p, rf);
@@ -2949,8 +2941,10 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);/* * Change a given task's CPU affinity to the intersection of its current -* affinity mask and @subset_mask, writing the resulting mask to @new_mask -* and pointing @p->user_cpus_ptr to a copy of the old mask. +* affinity mask and @subset_mask, writing the resulting mask to @new_mask. +* If user_cpus_ptr is defined, use it as the the basis for restricting CPU +* affinity or use cpu_online_mask instead. +* * If the resulting mask is empty, leave the affinity unchanged and return * -EINVAL. */
@@ -2958,16 +2952,10 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p,struct cpumask *new_mask, const struct cpumask *subset_mask) { - struct cpumask *user_mask = NULL; struct rq_flags rf;struct rq *rq;int err; - -if (!p->user_cpus_ptr) { - user_mask =kmalloc (cpumask_size (),GFP_KERNEL ); -if (!user_mask) -return -ENOMEM ; - } +bool not_empty; rq =task_rq_lock (p, &rf);
@@ -2981,25 +2969,21 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p, goto err_unlock; } -if (!cpumask_and (new_mask, &p->cpus_mask, subset_mask)) { + +if (p->user_cpus_ptr) + not_empty =cpumask_and (new_mask, p->user_cpus_ptr, subset_mask); +else + not_empty =cpumask_and (new_mask, cpu_online_mask, subset_mask); + +if (!not_empty) { err = -EINVAL ; goto err_unlock; } -/* -* We're about to butcher the task affinity, so keep track of what -* the user asked for in case we're able to restore it later on. -*/ -if (user_mask) { -cpumask_copy (user_mask, p->cpus_ptr); - p->user_cpus_ptr = user_mask; - } -return __set_cpus_allowed_ptr_locked (p, new_mask,0 , rq, &rf); err_unlock:task_rq_unlock (rq, p, &rf); -kfree (user_mask);return err; }
@@ -3049,34 +3033,27 @@ void force_compatible_cpus_allowed_ptr(struct task_struct *p) }static int -__sched_setaffinity (struct task_struct *p,const struct cpumask *mask); +__sched_setaffinity (struct task_struct *p,const struct cpumask *mask,bool save_mask);/* * Restore the affinity of a task @p which was previously restricted by a -* call to force_compatible_cpus_allowed_ptr(). This will clear (and free) -* @p->user_cpus_ptr. +* call to force_compatible_cpus_allowed_ptr(). * * It is the caller's responsibility to serialise this with any calls to * force_compatible_cpus_allowed_ptr(@p). */ void relax_compatible_cpus_allowed_ptr (struct task_struct *p) { -struct cpumask *user_mask = p->user_cpus_ptr; -unsigned long flags; +const struct cpumask *user_mask = p->user_cpus_ptr; + +if (!user_mask) + user_mask = cpu_online_mask;/* -* Try to restore the old affinity mask. If this fails, then -* we free the mask explicitly to avoid it being inherited across -* a subsequent fork(). +* Try to restore the old affinity mask with __sched_setaffinity(). +* Cpuset masking will be done there too. */ -if (!user_mask || !__sched_setaffinity (p, user_mask)) -return ; - -raw_spin_lock_irqsave (&p->pi_lock, flags); - user_mask =clear_user_cpus_ptr (p); -raw_spin_unlock_irqrestore (&p->pi_lock, flags); - -kfree (user_mask); +__sched_setaffinity (p, user_mask, false); }void set_task_cpu (struct task_struct *p,unsigned int new_cpu)
@@ -8079,10 +8056,11 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)#endif static int -__sched_setaffinity (struct task_struct *p,const struct cpumask *mask) +__sched_setaffinity (struct task_struct *p,const struct cpumask *mask,bool save_mask) {int retval;cpumask_var_t cpus_allowed, new_mask; +struct cpumask *user_mask =NULL ;if (!alloc_cpumask_var (&cpus_allowed,GFP_KERNEL ))return -ENOMEM ;
@@ -8098,8 +8076,33 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask) retval =dl_task_check_affinity (p, new_mask);if (retval) goto out_free_new_mask; + +/* +* Save the user requested mask into user_cpus_ptr if save_mask set. +* pi_lock is used for protecting user_cpus_ptr. +*/ +if (save_mask && !p->user_cpus_ptr) { + user_mask =kmalloc (cpumask_size (),GFP_KERNEL ); + +if (!user_mask) { + retval = -ENOMEM ; + goto out_free_new_mask; + } + } +if (save_mask) { +unsigned long flags; + +raw_spin_lock_irqsave (&p->pi_lock, flags); +if (!p->user_cpus_ptr) { + p->user_cpus_ptr = user_mask; + user_mask =NULL ; + } + +cpumask_copy (p->user_cpus_ptr, mask); +raw_spin_unlock_irqrestore (&p->pi_lock, flags); + } again: - retval =__set_cpus_allowed_ptr (p, new_mask,SCA_CHECK |SCA_USER ); + retval =__set_cpus_allowed_ptr (p, new_mask,SCA_CHECK );if (retval) goto out_free_new_mask;
@@ -8113,6 +8116,7 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask) goto again; } +kfree (user_mask); out_free_new_mask:free_cpumask_var (new_mask); out_free_cpus_allowed:
@@ -8156,7 +8160,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)if (retval) goto out_put_task; - retval =__sched_setaffinity (p, in_mask); + retval =__sched_setaffinity (p, in_mask, true); out_put_task:put_task_struct (p);return retval;
kernel/sched/sched.h +0 -1
@@ -2283,7 +2283,6 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);#define SCA_CHECK 0x01#define SCA_MIGRATE_DISABLE 0x02#define SCA_MIGRATE_ENABLE 0x04 -#define SCA_USER 0x08#ifdef CONFIG_SMP
Signature · 1 line
2.31.1
Waiman Long
maintainerPATCH v4 2/3 sched: Provide copy_user_cpus_mask() to copy out user mask
In reply to Waiman LongSince accessing the content of the user_cpus_ptr requires lock protection to ensure its validity, provide a helper function copy_user_cpus_mask() to facilitate its reading.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+)
include/linux/sched.h +1 -0
@@ -1830,6 +1830,7 @@ extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_effecextern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);extern int dup_user_cpus_ptr (struct task_struct *dst,struct task_struct *src,int node); +extern struct cpumask *copy_user_cpus_mask (struct task_struct *p,struct cpumask *user_mask);extern void release_user_cpus_ptr (struct task_struct *p);extern int dl_task_check_affinity (struct task_struct *p,const struct cpumask *mask);extern void force_compatible_cpus_allowed_ptr (struct task_struct *p);
kernel/sched/core.c +19 -0
@@ -2618,6 +2618,25 @@ void release_user_cpus_ptr(struct task_struct *p)kfree (clear_user_cpus_ptr (p)); } +/* +* Return the copied mask pointer or NULL if user mask not available. +* Acquiring pi_lock for read access protection. +*/ +struct cpumask *copy_user_cpus_mask (struct task_struct *p, +struct cpumask *user_mask) +{ +struct cpumask *mask =NULL ; +unsigned long flags; + +raw_spin_lock_irqsave (&p->pi_lock, flags); +if (p->user_cpus_ptr) { +cpumask_copy (user_mask, p->user_cpus_ptr); + mask = user_mask; + } +raw_spin_unlock_irqrestore (&p->pi_lock, flags); +return mask; +} +/* * This function is wildly self concurrent; here be dragons. *
Signature · 1 line
2.31.1
Waiman Long
maintainerPATCH v4 3/3 cgroup/cpuset: Keep user set cpus affinity
In reply to Waiman LongIt was found that any change to the current cpuset hierarchy may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.
That is problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.
Fix this problem by looking at user_cpus_ptr which will be set if cpus affinity have been explicitly set before and use it to restrcit the given cpumask unless there is no overlap. In that case, it will fallback to the given one.
With that change in place, it was verified that tasks that have its cpus affinity explicitly set will not be affected by changes made to the v2 cgroup.subtree_control files.
--- kernel/cgroup/cpuset.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
kernel/cgroup/cpuset.c +26 -2
@@ -704,6 +704,30 @@ static int validate_change(struct cpuset *cur, struct cpuset *trial)return ret; } +/* +* Preserve user provided cpumask (if set) as much as possible unless there +* is no overlap with the given mask. +*/ +static int cpuset_set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *mask) +{ +if (p->user_cpus_ptr) { +cpumask_var_t new_mask; + +if (alloc_cpumask_var (&new_mask,GFP_KERNEL ) && +copy_user_cpus_mask (p, new_mask) && +cpumask_and (new_mask, new_mask, mask)) { +int ret =set_cpus_allowed_ptr (p, new_mask); + +free_cpumask_var (new_mask); +return ret; + } +free_cpumask_var (new_mask); + } + +return set_cpus_allowed_ptr (p, mask); +} +#ifdef CONFIG_SMP /* * Helper routine for generate_sched_domains().
@@ -1130,7 +1154,7 @@ static void update_tasks_cpumask(struct cpuset *cs)css_task_iter_start (&cs->css,0 , &it);while ((task =css_task_iter_next (&it))) -set_cpus_allowed_ptr (task, cs->effective_cpus); +cpuset_set_cpus_allowed_ptr (task, cs->effective_cpus);css_task_iter_end (&it); }
@@ -2303,7 +2327,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)* can_attach beforehand should guarantee that this doesn't * fail. TODO: have a better way to handle failure here */ -WARN_ON_ONCE (set_cpus_allowed_ptr (task, cpus_attach)); +WARN_ON_ONCE (cpuset_set_cpus_allowed_ptr (task, cpus_attach));cpuset_change_task_nodemask (task, &cpuset_attach_nodemask_to);cpuset_update_task_spread_flag (cs, task);
Signature · 1 line
2.31.1
Tejun Heo
maintainerRe: [PATCH v4 3/3] cgroup/cpuset: Keep user set cpus affinity
Hello,
So, overall I think this is the right direction.
Quoted 9 lines
kernel/cgroup/cpuset.c +9
-0+static int cpuset_set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *mask) +{ +if (p->user_cpus_ptr) { +cpumask_var_t new_mask; + +if (alloc_cpumask_var (&new_mask,GFP_KERNEL ) && +copy_user_cpus_mask (p, new_mask) && +cpumask_and (new_mask, new_mask, mask)) {
kernel/cgroup/cpuset.c static int validate_change(struct cpuset *cur, struct cpuset *trial) · line 729
Quoted 10 lines
kernel/cgroup/cpuset.c +10
-0+int ret =set_cpus_allowed_ptr (p, new_mask); + +free_cpumask_var (new_mask); +return ret; + } +free_cpumask_var (new_mask); + } + +return set_cpus_allowed_ptr (p, mask); +}
But this seems racy to me. Let's say attach and setaffinity race. The expectation should be that we'd end up with the same eventual mask no matter what the operation order may be. The above code wouldn't do that, right? There's nothing synchronizing the two and if setaffinity takes place between the user_cpus_ptr test and set_cpus_allowed_ptr(), it'd get ignored.
This gotta be more integrated. There is what the user requested and there are restrictions from CPU hotplug state and cpuset. All three should be synchronized so that there is one synchronzied way to obtain and apply the current effective mask.
Thanks.
Signature · 1 line
tejun
Waiman Long
maintainerRe: [PATCH v4 3/3] cgroup/cpuset: Keep user set cpus affinity
Quoted 20 lines · Tejun Heo
Hello,
So, overall I think this is the right direction.
Quoted 16 lines
kernel/cgroup/cpuset.c +16
-0+static int cpuset_set_cpus_allowed_ptr (struct task_struct *p, +const struct cpumask *mask) +{ +if (p->user_cpus_ptr) { +cpumask_var_t new_mask; + +if (alloc_cpumask_var (&new_mask,GFP_KERNEL ) && +copy_user_cpus_mask (p, new_mask) && +cpumask_and (new_mask, new_mask, mask)) { +int ret =set_cpus_allowed_ptr (p, new_mask); + +free_cpumask_var (new_mask); +return ret; + } +free_cpumask_var (new_mask); + }
Quoted 8 lines
kernel/cgroup/cpuset.c static int validate_change(struct cpuset *cur, struct cpuset *trial) · line 729
Quoted 3 lines
kernel/cgroup/cpuset.c +3
-0+ +return set_cpus_allowed_ptr (p, mask); +}But this seems racy to me. Let's say attach and setaffinity race. The expectation should be that we'd end up with the same eventual mask no matter what the operation order may be. The above code wouldn't do that, right? There's nothing synchronizing the two and if setaffinity takes place between the
user_cpus_ptrtest andset_cpus_allowed_ptr(), it'd get ignored.
Yes, a race like this is possible. To completely eliminate the race may require taking task_rq_lock() and then calling __set_cpus_allowed_ptr_locked() which is internal to kernel/sched/core.c.
Alternatively, we can check user_cpus_ptr again after the scond set_cpus_allowed_ptr() and retry it with the other path if set. That will probably address your concern. Please let me know if you are OK with that.
Cheers,
Longman
Quoted 8 lines
This gotta be more integrated. There is what the user requested and there are restrictions from CPU hotplug state and cpuset. All three should be synchronized so that there is one synchronzied way to obtain and apply the current effective mask.
Thanks.
Tejun Heo
maintainerRe: [PATCH v4 3/3] cgroup/cpuset: Keep user set cpus affinity
Hello,
Quoted 7 lines · Waiman Long
Yes, a race like this is possible. To completely eliminate the race may require taking
task_rq_lock()and then calling__set_cpus_allowed_ptr_locked()which is internal tokernel/sched/core.c.Alternatively, we can check
user_cpus_ptragain after the scondset_cpus_allowed_ptr()and retry it with the other path if set. That will probably address your concern. Please let me know if you are OK with that.
I think this would look better if structured the other way around - make the scheduler side call out to cpuset to query the current restrictions and apply it atomically.
Thanks.
Signature · 1 line
tejun
Waiman Long
maintainerRe: [PATCH v4 3/3] cgroup/cpuset: Keep user set cpus affinity
Quoted 7 lines · Tejun Heo
Hello,
Quoted 4 lines · Waiman Long
Yes, a race like this is possible. To completely eliminate the race may require taking
task_rq_lock()and then calling__set_cpus_allowed_ptr_locked()which is internal tokernel/sched/core.c.
Quoted 6 lines
Quoted 3 lines
Alternatively, we can check
user_cpus_ptragain after the scondset_cpus_allowed_ptr()and retry it with the other path if set. That will probably address your concern. Please let me know if you are OK with that.I think this would look better if structured the other way around - make the scheduler side call out to cpuset to query the current restrictions and apply it atomically.
The sched_setaffinity() function does call cpuset_cpus_allowed() to apply the cpuset constraint. However, making set_cpus_allowed_ptr() call cpuset function is a major change. It will disturb the current locking sequences and may cause circular locking dependency problem. We certainly need more time to figure out if that is feasible.
In the mean time, I would prefer to do a retry if user_cpus_ptr status changes for now. We can then do a follow up patch to make this structural change if there is a consensus of doing so.
Cheers,
Longman