Skip to contentSkip to conversation

sched: Persistent user requested affinity

PATCH Superseded Waiman Long 2 participants 17 messages Aug 26, 2022 – Aug 31, 2022 Scheduler

v6 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v6 cover sched: Persistent user requested affinity

v6:
- Drop the cpuset patch and make user_cpus_ptr masking applicable to all callers of set_cpus_allowed_ptr().
- Make user_cpus_ptr and cpus_mask update in a single lock critical section to avoid race problems.

v5:
- Update patch 3 to handle race with concurrent sched_setaffinity() by rechecking a previously cleared user_cpus_ptr afterward.

v4:
- Update patch 1 to make sched_setaffinity() the only function to update user_cpus_ptr to make the logic simpler and easier to understand. restrict_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() will just use it if present.

The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity") which uses it narrowly to allow keeping cpus affinity intact with asymmetric cpu setup.

This patchset extends user_cpus_ptr to store user requested cpus affinity via the sched_setaffinity() API. With that information available, it will enable cpuset and other callers of set_cpus_allowed_ptr() like hotplug to keep cpus afinity as close to what the user wants as possible within the cpu list constraint of the current cpuset. Otherwise some change to the cpuset hierarchy or a hotplug event may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before.

It also means that once sched_setaffinity() is called, user_cpus_ptr will remain allocated until the task exits.

Waiman Long (5):
sched: Add __releases annotations to affine_move_task()
sched: Use user_cpus_ptr for saving user provided cpumask in
sched_setaffinity()
sched: Enforce user requested affinity
sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race
sched: Fix sched_setaffinity() and fork/clone() race

 kernel/sched/core.c  | 174 +++++++++++++++++++++++++++----------------
 kernel/sched/sched.h |  11 ++-
 2 files changed, 121 insertions(+), 64 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v6 1/5 sched: Add __releases annotations to affine_move_task()

affine_move_task() assumes task_rq_lock() has been called and it does an implicit task_rq_unlock() before returning. Add the appropriate __releases annotations to make this clear.

A typo error in comment is also fixed.

---
 kernel/sched/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/core.c +3 -1
index ee28253c9ac0..b351e6d173b7 100644
@@ -2696,6 +2696,8 @@ void release_user_cpus_ptr(struct task_struct *p)
  */
 static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flags *rf,
 			    int dest_cpu, unsigned int flags)
+	__releases(rq->lock)
+	__releases(p->pi_lock)
 {
 	struct set_affinity_pending my_pending = { }, *pending = NULL;
 	bool stop_pending, complete = false;
@@ -3005,7 +3007,7 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p,
 
 /*
  * Restrict the CPU affinity of task @p so that it is a subset of
- * task_cpu_possible_mask() and point @p->user_cpu_ptr to a copy of the
+ * task_cpu_possible_mask() and point @p->user_cpus_ptr to a copy of the
  * old affinity mask. If the resulting mask is empty, we warn and walk
  * up the cpuset hierarchy until we find a suitable mask.
  */
Signature · 1 line
2.31.1
#2 lore raw

PATCH v6 2/5 sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()

In reply to Waiman Long

The user_cpus_ptr field is added by commit b90ca8badbd1 ("sched:
Introduce task_struct::user_cpus_ptr to track requested affinity"). It is currently used only by arm64 arch due to possible asymmetric CPU setup. This patch extends its usage to save user provided cpumask when sched_setaffinity() is called for all arches. With this patch applied, user_cpus_ptr, once allocated after a successful call to sched_setaffinity(), will only be freed when the task exits.

Since user_cpus_ptr is supposed to be used for "requested affinity", there is actually no point to save current cpu affinity in restrict_cpus_allowed_ptr() if sched_setaffinity() has never been called. Modify the logic to set user_cpus_ptr only in sched_setaffinity() and use it in restrict_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() if defined but not changing it.

This will be some changes in behavior for arm64 systems with asymmetric CPUs in some corner cases. For instance, if sched_setaffinity() has never been called and there is a cpuset change before relax_compatible_cpus_allowed_ptr() is called, its subsequent call will follow what the cpuset allows but not what the previous cpu affinity setting allows.

As a call to sched_setaffinity() will no longer clear user_cpus_ptr but set it instead, the SCA_USER flag is no longer necessary and can be removed.

---
 kernel/sched/core.c  | 87 +++++++++++++++++++-------------------------
 kernel/sched/sched.h |  8 +++-
 2 files changed, 45 insertions(+), 50 deletions(-)
kernel/sched/core.c +38 -49
index b351e6d173b7..ac2b103d69dc 100644
@@ -2850,7 +2850,6 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 	const struct cpumask *cpu_allowed_mask = task_cpu_possible_mask(p);
 	const struct cpumask *cpu_valid_mask = cpu_active_mask;
 	bool kthread = p->flags & PF_KTHREAD;
-	struct cpumask *user_mask = NULL;
 	unsigned int dest_cpu;
 	int ret = 0;
 
@@ -2909,14 +2908,7 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 
 	__do_set_cpus_allowed(p, new_mask, flags);
 
-	if (flags & SCA_USER)
-		user_mask = clear_user_cpus_ptr(p);
-
-	ret = affine_move_task(rq, p, rf, dest_cpu, flags);
-
-	kfree(user_mask);
-
-	return ret;
+	return affine_move_task(rq, p, rf, dest_cpu, flags);
 
 out:
 	task_rq_unlock(rq, p, rf);
@@ -2951,8 +2943,10 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
 /*
  * Change a given task's CPU affinity to the intersection of its current
- * affinity mask and @subset_mask, writing the resulting mask to @new_mask
- * and pointing @p->user_cpus_ptr to a copy of the old mask.
+ * affinity mask and @subset_mask, writing the resulting mask to @new_mask.
+ * If user_cpus_ptr is defined, use it as the basis for restricting CPU
+ * affinity or use cpu_online_mask instead.
+ *
  * If the resulting mask is empty, leave the affinity unchanged and return
  * -EINVAL.
  */
@@ -2960,17 +2954,10 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p,
 				     struct cpumask *new_mask,
 				     const struct cpumask *subset_mask)
 {
-	struct cpumask *user_mask = NULL;
 	struct rq_flags rf;
 	struct rq *rq;
 	int err;
 
-	if (!p->user_cpus_ptr) {
-		user_mask = kmalloc(cpumask_size(), GFP_KERNEL);
-		if (!user_mask)
-			return -ENOMEM;
-	}
-
 	rq = task_rq_lock(p, &rf);
 
 	/*
@@ -2983,25 +2970,15 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p,
 		goto err_unlock;
 	}
 
-	if (!cpumask_and(new_mask, &p->cpus_mask, subset_mask)) {
+	if (!cpumask_and(new_mask, task_user_cpus(p), subset_mask)) {
 		err = -EINVAL;
 		goto err_unlock;
 	}
 
-	/*
-	 * We're about to butcher the task affinity, so keep track of what
-	 * the user asked for in case we're able to restore it later on.
-	 */
-	if (user_mask) {
-		cpumask_copy(user_mask, p->cpus_ptr);
-		p->user_cpus_ptr = user_mask;
-	}
-
 	return __set_cpus_allowed_ptr_locked(p, new_mask, 0, rq, &rf);
 
 err_unlock:
 	task_rq_unlock(rq, p, &rf);
-	kfree(user_mask);
 	return err;
 }
 
@@ -3051,34 +3028,22 @@ void force_compatible_cpus_allowed_ptr(struct task_struct *p)
 }
 
 static int
-__sched_setaffinity(struct task_struct *p, const struct cpumask *mask);
+__sched_setaffinity(struct task_struct *p, const struct cpumask *mask, bool save_mask);
 
 /*
  * Restore the affinity of a task @p which was previously restricted by a
- * call to force_compatible_cpus_allowed_ptr(). This will clear (and free)
- * @p->user_cpus_ptr.
+ * call to force_compatible_cpus_allowed_ptr().
  *
  * It is the caller's responsibility to serialise this with any calls to
  * force_compatible_cpus_allowed_ptr(@p).
  */
 void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
 {
-	struct cpumask *user_mask = p->user_cpus_ptr;
-	unsigned long flags;
-
 	/*
-	 * Try to restore the old affinity mask. If this fails, then
-	 * we free the mask explicitly to avoid it being inherited across
-	 * a subsequent fork().
+	 * Try to restore the old affinity mask with __sched_setaffinity().
+	 * Cpuset masking will be done there too.
 	 */
-	if (!user_mask || !__sched_setaffinity(p, user_mask))
-		return;
-
-	raw_spin_lock_irqsave(&p->pi_lock, flags);
-	user_mask = clear_user_cpus_ptr(p);
-	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
-
-	kfree(user_mask);
+	__sched_setaffinity(p, task_user_cpus(p), false);
 }
 
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
@@ -8081,10 +8046,11 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
 #endif
 
 static int
-__sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
+__sched_setaffinity(struct task_struct *p, const struct cpumask *mask, bool save_mask)
 {
 	int retval;
 	cpumask_var_t cpus_allowed, new_mask;
+	struct cpumask *user_mask = NULL;
 
 	if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL))
 		return -ENOMEM;
@@ -8100,8 +8066,22 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
 	retval = dl_task_check_affinity(p, new_mask);
 	if (retval)
 		goto out_free_new_mask;
+
+	/*
+	 * Save the user requested mask internally now and then update
+	 * user_cpus_ptr later after making sure this call will be
+	 * successful, i.e. retval == 0.
+	 */
+	if (save_mask) {
+		user_mask = kmalloc(cpumask_size(), GFP_KERNEL);
+		if (!user_mask) {
+			retval = -ENOMEM;
+			goto out_free_new_mask;
+		}
+		cpumask_copy(user_mask, mask);
+	}
 again:
-	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK | SCA_USER);
+	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK);
 	if (retval)
 		goto out_free_new_mask;
 
@@ -8115,7 +8095,16 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
 		goto again;
 	}
 
+	if (save_mask) {
+		unsigned long flags;
+
+		/* Use pi_lock to synchronize changes to user_cpus_ptr */
+		raw_spin_lock_irqsave(&p->pi_lock, flags);
+		swap(p->user_cpus_ptr, user_mask);
+		raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	}
 out_free_new_mask:
+	kfree(user_mask);
 	free_cpumask_var(new_mask);
 out_free_cpus_allowed:
 	free_cpumask_var(cpus_allowed);
@@ -8158,7 +8147,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
 	if (retval)
 		goto out_put_task;
 
-	retval = __sched_setaffinity(p, in_mask);
+	retval = __sched_setaffinity(p, in_mask, true);
 out_put_task:
 	put_task_struct(p);
 	return retval;
kernel/sched/sched.h +7 -1
index e26688d387ae..a49c17e1c7ea 100644
@@ -1881,6 +1881,13 @@ static inline void dirty_sched_domain_sysctl(int cpu)
 #endif
 
 extern int sched_update_scaling(void);
+
+static inline const struct cpumask *task_user_cpus(struct task_struct *p)
+{
+	if (!p->user_cpus_ptr)
+		return cpu_possible_mask; /* &init_task.cpus_mask */
+	return p->user_cpus_ptr;
+}
 #endif /* CONFIG_SMP */
 
 #include "stats.h"
@@ -2283,7 +2290,6 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);
 #define SCA_CHECK		0x01
 #define SCA_MIGRATE_DISABLE	0x02
 #define SCA_MIGRATE_ENABLE	0x04
-#define SCA_USER		0x08
 
 #ifdef CONFIG_SMP
 
Signature · 1 line
2.31.1
maintainer
#3 lore raw

Re: [PATCH v6 2/5] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()

Quoted 12 lines · Waiman Long

void relax_compatible_cpus_allowed_ptr(struct task_struct *p) {

kernel/sched/core.c +2 -6
-	struct cpumask *user_mask = p->user_cpus_ptr;
-	unsigned long flags;
-
 	/*
-	 * Try to restore the old affinity mask. If this fails, then
-	 * we free the mask explicitly to avoid it being inherited across
-	 * a subsequent fork().
+	 * Try to restore the old affinity mask with __sched_setaffinity().
+	 * Cpuset masking will be done there too.
 	 */

kernel/sched/core.c void force_compatible_cpus_allowed_ptr(struct task_struct *p) · line 3047

Quoted 10 lines
kernel/sched/core.c +1 -8
-	if (!user_mask || !__sched_setaffinity(p, user_mask))
-		return;
-
-	raw_spin_lock_irqsave(&p->pi_lock, flags);
-	user_mask = clear_user_cpus_ptr(p);
-	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
-
-	kfree(user_mask);
+	__sched_setaffinity(p, task_user_cpus(p), false);
 }

We have an issue with __sched_setaffinity() failing here. I'm not sure ignoring the failure is the right thing -- but I'm also not enturely sure what is.

Quoted 45 lines

void set_task_cpu(struct task_struct *p, unsigned int new_cpu)

kernel/sched/core.c +20 -2
@@ -8081,10 +8046,11 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
 #endif
 
 static int
-__sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
+__sched_setaffinity(struct task_struct *p, const struct cpumask *mask, bool save_mask)
 {
 	int retval;
 	cpumask_var_t cpus_allowed, new_mask;
+	struct cpumask *user_mask = NULL;
 
 	if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL))
 		return -ENOMEM;
@@ -8100,8 +8066,22 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
 	retval = dl_task_check_affinity(p, new_mask);
 	if (retval)
 		goto out_free_new_mask;
+
+	/*
+	 * Save the user requested mask internally now and then update
+	 * user_cpus_ptr later after making sure this call will be
+	 * successful, i.e. retval == 0.
+	 */
+	if (save_mask) {
+		user_mask = kmalloc(cpumask_size(), GFP_KERNEL);
+		if (!user_mask) {
+			retval = -ENOMEM;
+			goto out_free_new_mask;
+		}
+		cpumask_copy(user_mask, mask);
+	}
 again:
-	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK | SCA_USER);
+	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK);
 	if (retval)
 		goto out_free_new_mask;
 
@@ -8115,7 +8095,16 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
 		goto again;
 	}
 
+	if (save_mask) {
+		unsigned long flags;
+

kernel/sched/core.c __sched_setaffinity(struct task_struct *p, const struct cpumask *mask) · line 8110

Quoted 10 lines
kernel/sched/core.c +6 -0
+		/* Use pi_lock to synchronize changes to user_cpus_ptr */
+		raw_spin_lock_irqsave(&p->pi_lock, flags);
+		swap(p->user_cpus_ptr, user_mask);
+		raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	}
 out_free_new_mask:
+	kfree(user_mask);
 	free_cpumask_var(new_mask);
 out_free_cpus_allowed:
 	free_cpumask_var(cpus_allowed);

I'm confused as to why it's put in this function and not in the one caller that actually sets the new @save_mask true, here:

Quoted 9 lines
kernel/sched/core.c +1 -1
@@ -8158,7 +8147,7 @@ long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
 	if (retval)
 		goto out_put_task;
 
-	retval = __sched_setaffinity(p, in_mask);
+	retval = __sched_setaffinity(p, in_mask, true);
 out_put_task:
 	put_task_struct(p);
 	return retval;
#4 lore raw

Re: [PATCH v6 2/5] sched: Use user_cpus_ptr for saving user provided cpumask in sched_setaffinity()

Quoted 20 lines · Peter Zijlstra
Quoted 17 lines · Waiman Long
  void relax_compatible_cpus_allowed_ptr(struct task_struct *p)
  {
kernel/sched/core.c +2 -11
-	struct cpumask *user_mask = p->user_cpus_ptr;
-	unsigned long flags;
-
  	/*
-	 * Try to restore the old affinity mask. If this fails, then
-	 * we free the mask explicitly to avoid it being inherited across
-	 * a subsequent fork().
+	 * Try to restore the old affinity mask with __sched_setaffinity().
+	 * Cpuset masking will be done there too.
  	 */
-	if (!user_mask || !__sched_setaffinity(p, user_mask))
-		return;
-
-	raw_spin_lock_irqsave(&p->pi_lock, flags);
-	user_mask = clear_user_cpus_ptr(p);
Quoted 8 lines

kernel/sched/core.c void force_compatible_cpus_allowed_ptr(struct task_struct *p) · line 3047

Quoted 5 lines
kernel/sched/core.c +1 -3
-	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
-
-	kfree(user_mask);
+	__sched_setaffinity(p, task_user_cpus(p), false);
  }

We have an issue with __sched_setaffinity() failing here. I'm not sure ignoring the failure is the right thing -- but I'm also not enturely sure what is.

I am not sure what we can do in case __sched_setaffinity() fails. Maybe we can print a warning when this happen. What do you think?

Quoted 42 lines
Quoted 42 lines

void set_task_cpu(struct task_struct *p, unsigned int new_cpu)

kernel/sched/core.c +17 -2
@@ -8081,10 +8046,11 @@ int dl_task_check_affinity(struct task_struct *p, const struct cpumask *mask)
  #endif
 
  static int
-__sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
+__sched_setaffinity(struct task_struct *p, const struct cpumask *mask, bool save_mask)
  {
  	int retval;
  	cpumask_var_t cpus_allowed, new_mask;
+	struct cpumask *user_mask = NULL;
 
  	if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL))
  		return -ENOMEM;
@@ -8100,8 +8066,22 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
  	retval = dl_task_check_affinity(p, new_mask);
  	if (retval)
  		goto out_free_new_mask;
+
+	/*
+	 * Save the user requested mask internally now and then update
+	 * user_cpus_ptr later after making sure this call will be
+	 * successful, i.e. retval == 0.
+	 */
+	if (save_mask) {
+		user_mask = kmalloc(cpumask_size(), GFP_KERNEL);
+		if (!user_mask) {
+			retval = -ENOMEM;
+			goto out_free_new_mask;
+		}
+		cpumask_copy(user_mask, mask);
+	}
  again:
-	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK | SCA_USER);
+	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK);
  	if (retval)
  		goto out_free_new_mask;
 
@@ -8115,7 +8095,16 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
  		goto again;
  	}
 
Quoted 15 lines

kernel/sched/core.c __sched_setaffinity(struct task_struct *p, const struct cpumask *mask) · line 8110

Quoted 13 lines
kernel/sched/core.c +9 -0
+	if (save_mask) {
+		unsigned long flags;
+
+		/* Use pi_lock to synchronize changes to user_cpus_ptr */
+		raw_spin_lock_irqsave(&p->pi_lock, flags);
+		swap(p->user_cpus_ptr, user_mask);
+		raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	}
  out_free_new_mask:
+	kfree(user_mask);
  	free_cpumask_var(new_mask);
  out_free_cpus_allowed:
  	free_cpumask_var(cpus_allowed);

I'm confused as to why it's put in this function and not in the one caller that actually sets the new @save_mask true, here:

Looking at this patch alone, we can certainly put mask saving in sched_setaffinity(). In later patches, however, I have to make user_cpus_ptr update in the same lock critical section as cpus_mask. That is the reason why it is done this way here. I can certainly make your suggested change in this patch and then move the saving inside in a later patch.

Cheers,
Longman

#5 lore raw

PATCH v6 3/5 sched: Enforce user requested affinity

In reply to Waiman Long

It was found that the user requested affinity via sched_setaffinity() can be easily overwritten by other kernel subsystems without an easy way to reset it back to what the user requested. For example, any change to the current cpuset hierarchy may reset the cpumask of the tasks in the affected cpusets to the default cpuset value even if those tasks have pre-existing user requested affinity. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

That is problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

Fix this problem by looking at user_cpus_ptr in __set_cpus_allowed_ptr() and use it to restrcit the given cpumask unless there is no overlap. In that case, it will fallback to the given one.

All callers of set_cpus_allowed_ptr() will be affected by this change. A scratch cpumask is added to percpu runqueues structure for doing additional masking when user_cpus_ptr is set. The scratch cpumask should get allocated during cpu activation. A fallback atomic memory allocation in __set_cpus_allowed_ptr() is also added in case set_cpus_allowed_ptr() is called before the scratch cpumask is properly allocated.

---
 kernel/sched/core.c  | 36 +++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h |  3 +++
 2 files changed, 38 insertions(+), 1 deletion(-)
kernel/sched/core.c +35 -1
index ac2b103d69dc..1c2f548e5369 100644
@@ -2928,11 +2928,40 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 static int __set_cpus_allowed_ptr(struct task_struct *p,
 				  const struct cpumask *new_mask, u32 flags)
 {
+	struct cpumask *alloc_mask = NULL;
 	struct rq_flags rf;
 	struct rq *rq;
+	int ret;
 
 	rq = task_rq_lock(p, &rf);
-	return __set_cpus_allowed_ptr_locked(p, new_mask, flags, rq, &rf);
+	if (p->user_cpus_ptr) {
+
+		/*
+		 * A scratch cpumask is allocated on the percpu runqueues
+		 * to enable additional masking with user_cpus_ptr. This
+		 * cpumask, once allocated, will not be freed.
+		 */
+		if (unlikely(!rq->scratch_mask)) {
+			alloc_mask = kmalloc(cpumask_size(), GFP_ATOMIC);
+			if (!rq->scratch_mask && alloc_mask) {
+				rq->scratch_mask = alloc_mask;
+				alloc_mask = NULL;
+			}
+		}
+		/*
+		 * Ignore user_cpus_ptr if atomic memory allocation fails
+		 * or it doesn't intersect new_mask.
+		 */
+		if (rq->scratch_mask &&
+		    cpumask_and(rq->scratch_mask, new_mask, p->user_cpus_ptr))
+			new_mask = rq->scratch_mask;
+	}
+
+
+	ret = __set_cpus_allowed_ptr_locked(p, new_mask, flags, rq, &rf);
+	if (unlikely(alloc_mask))
+		kfree(alloc_mask);
+	return ret;
 }
 
 int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
@@ -9352,6 +9381,11 @@ int sched_cpu_activate(unsigned int cpu)
 		sched_update_numa(cpu, true);
 		sched_domains_numa_masks_set(cpu);
 		cpuset_cpu_active();
+		/*
+		 * Preallocated scratch cpumask
+		 */
+		if (!rq->scratch_mask)
+			rq->scratch_mask = kmalloc(cpumask_size(), GFP_KERNEL);
 	}
 
 	/*
kernel/sched/sched.h +3 -0
index a49c17e1c7ea..66a6bfddd716 100644
@@ -1159,6 +1159,9 @@ struct rq {
 	unsigned int		core_forceidle_occupation;
 	u64			core_forceidle_start;
 #endif
+
+	/* Scratch cpumask to be temporarily used under rq_lock */
+	struct cpumask		*scratch_mask;
 };
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.31.1
maintainer
#6 lore raw

Re: [PATCH v6 3/5] sched: Enforce user requested affinity

Quoted 14 lines · Waiman Long
kernel/sched/core.c +2 -0
index ac2b103d69dc..1c2f548e5369 100644
@@ -2928,11 +2928,40 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 static int __set_cpus_allowed_ptr(struct task_struct *p,
 				  const struct cpumask *new_mask, u32 flags)
 {
+	struct cpumask *alloc_mask = NULL;
 	struct rq_flags rf;
 	struct rq *rq;
+	int ret;
 
 	rq = task_rq_lock(p, &rf);

kernel/sched/core.c static int __set_cpus_allowed_ptr_locked(struct task_struct *p, · line 2945

Quoted 10 lines
kernel/sched/core.c +9 -1
-	return __set_cpus_allowed_ptr_locked(p, new_mask, flags, rq, &rf);
+	if (p->user_cpus_ptr) {
+
+		/*
+		 * A scratch cpumask is allocated on the percpu runqueues
+		 * to enable additional masking with user_cpus_ptr. This
+		 * cpumask, once allocated, will not be freed.
+		 */
+		if (unlikely(!rq->scratch_mask)) {
+			alloc_mask = kmalloc(cpumask_size(), GFP_ATOMIC);

This -- absolutely not. You can't have allocations under a raw_spinlock_t.

maintainer
#7 lore raw

Re: [PATCH v6 3/5] sched: Enforce user requested affinity

In reply to Waiman Long

kernel/sched/core.c int sched_cpu_activate(unsigned int cpu) · line 9389

Quoted 10 lines · Waiman Long
kernel/sched/core.c +5 -0
@@ -9352,6 +9381,11 @@ int sched_cpu_activate(unsigned int cpu)
 		sched_update_numa(cpu, true);
 		sched_domains_numa_masks_set(cpu);
 		cpuset_cpu_active();
+		/*
+		 * Preallocated scratch cpumask
+		 */
+		if (!rq->scratch_mask)
+			rq->scratch_mask = kmalloc(cpumask_size(), GFP_KERNEL);
 	}

this is too late; I think you'll have to add a sched_cpu_prepare() and simply fail the cpu-up when the allocation fails.

maintainer
#8 lore raw

Re: [PATCH v6 3/5] sched: Enforce user requested affinity

Quoted 15 lines · Peter Zijlstra

kernel/sched/core.c int sched_cpu_activate(unsigned int cpu) · line 9389

Quoted 10 lines · Waiman Long
kernel/sched/core.c +5 -0
@@ -9352,6 +9381,11 @@ int sched_cpu_activate(unsigned int cpu)
 		sched_update_numa(cpu, true);
 		sched_domains_numa_masks_set(cpu);
 		cpuset_cpu_active();
+		/*
+		 * Preallocated scratch cpumask
+		 */
+		if (!rq->scratch_mask)
+			rq->scratch_mask = kmalloc(cpumask_size(), GFP_KERNEL);
 	}

this is too late; I think you'll have to add a sched_cpu_prepare() and simply fail the cpu-up when the allocation fails.

Alternatively, waste some more memory and add yet another per-cpu cpumask.

#9 lore raw

Re: [PATCH v6 3/5] sched: Enforce user requested affinity

Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra

kernel/sched/core.c int sched_cpu_activate(unsigned int cpu) · line 9389

Quoted 10 lines · Waiman Long
kernel/sched/core.c +5 -0
@@ -9352,6 +9381,11 @@ int sched_cpu_activate(unsigned int cpu)
  		sched_update_numa(cpu, true);
  		sched_domains_numa_masks_set(cpu);
  		cpuset_cpu_active();
+		/*
+		 * Preallocated scratch cpumask
+		 */
+		if (!rq->scratch_mask)
+			rq->scratch_mask = kmalloc(cpumask_size(), GFP_KERNEL);
  	}

this is too late; I think you'll have to add a sched_cpu_prepare() and simply fail the cpu-up when the allocation fails.

Quoted 2 lines

Alternatively, waste some more memory and add yet another per-cpu cpumask.

A percpu cpumask is probably a better idea. However, I don't need that as early as the other cpumasks like __cpu_possible_mask. Maybe I can do a percpu memory allocation early in the pre-smp boot.

Cheers,
Longman

#10 lore raw

Re: [PATCH v6 3/5] sched: Enforce user requested affinity

In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra

kernel/sched/core.c int sched_cpu_activate(unsigned int cpu) · line 9389

Quoted 10 lines · Waiman Long
kernel/sched/core.c +5 -0
@@ -9352,6 +9381,11 @@ int sched_cpu_activate(unsigned int cpu)
  		sched_update_numa(cpu, true);
  		sched_domains_numa_masks_set(cpu);
  		cpuset_cpu_active();
+		/*
+		 * Preallocated scratch cpumask
+		 */
+		if (!rq->scratch_mask)
+			rq->scratch_mask = kmalloc(cpumask_size(), GFP_KERNEL);
  	}

this is too late; I think you'll have to add a sched_cpu_prepare() and simply fail the cpu-up when the allocation fails.

That is the reason why I have a fallback allocation in __set_cpus_allowed_ptr(). Thanks for the suggestion, I will looking into doing that.

Cheers,
Longman

#11 lore raw

PATCH v6 4/5 sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race

In reply to Waiman Long

Racing is possible between set_cpus_allowed_ptr() and sched_setaffinity() or between multiple sched_setaffinity() calls from different CPUs. To resolve these race conditions, we need to update both user_cpus_ptr and cpus_mask in a single lock critical section instead of separated ones. This requires moving the user_cpus_ptr update to affine_move_task() before doing task_rq_unlock().

A new argument puser_mask is added to affine_move_task(), __set_cpus_allowed_ptr_locked() and __set_cpus_allowed_ptr() to do that.

Ideally, user_cpus_ptr should only be updated if the sched_setaffinity() is successful. However, this patch will update user_cpus_ptr when the first call to __set_cpus_allowed_ptr() is successful. However, if there is racing between sched_setaffinity() and cpuset update, the subsequent calls to __set_cpus_allowed_ptr() may fail but the user_cpus_ptr will still be updated in this corner case.

---
 kernel/sched/core.c | 66 ++++++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 24 deletions(-)
kernel/sched/core.c +42 -24
index 1c2f548e5369..6cd1177fbcea 100644
@@ -2199,7 +2199,7 @@ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32
 
 static int __set_cpus_allowed_ptr(struct task_struct *p,
 				  const struct cpumask *new_mask,
-				  u32 flags);
+				  u32 flags, struct cpumask **puser_mask);
 
 static void migrate_disable_switch(struct rq *rq, struct task_struct *p)
 {
@@ -2249,7 +2249,7 @@ void migrate_enable(void)
 	 */
 	preempt_disable();
 	if (p->cpus_ptr != &p->cpus_mask)
-		__set_cpus_allowed_ptr(p, &p->cpus_mask, SCA_MIGRATE_ENABLE);
+		__set_cpus_allowed_ptr(p, &p->cpus_mask, SCA_MIGRATE_ENABLE, NULL);
 	/*
 	 * Mustn't clear migration_disabled() until cpus_ptr points back at the
 	 * regular cpus_mask, otherwise things that race (eg.
@@ -2618,6 +2618,15 @@ void release_user_cpus_ptr(struct task_struct *p)
 	kfree(clear_user_cpus_ptr(p));
 }
 
+static inline void swap_user_cpus_ptr(struct task_struct *p,
+				      struct cpumask **puser_mask)
+{
+	if (!puser_mask)
+		return;
+
+	swap(p->user_cpus_ptr, *puser_mask);
+}
+
 /*
  * This function is wildly self concurrent; here be dragons.
  *
@@ -2693,9 +2702,12 @@ void release_user_cpus_ptr(struct task_struct *p)
  * Note that the above is safe vs a concurrent migrate_enable(), as any
  * pending affinity completion is preceded by an uninstallation of
  * p->migration_pending done with p->pi_lock held.
+ *
+ * The puser_mask pointer, if defined, will cause its swap with the current
+ * user_cpus_ptr value if operation succeeds.
  */
 static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flags *rf,
-			    int dest_cpu, unsigned int flags)
+			    int dest_cpu, unsigned int flags, struct cpumask **puser_mask)
 	__releases(rq->lock)
 	__releases(p->pi_lock)
 {
@@ -2722,6 +2734,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 			complete = true;
 		}
 
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (push_task) {
@@ -2793,6 +2806,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 		if (flags & SCA_MIGRATE_ENABLE)
 			p->migration_flags &= ~MDF_PUSH;
 
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (!stop_pending) {
@@ -2813,6 +2827,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 				complete = true;
 			}
 		}
+
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (complete)
@@ -2843,7 +2859,8 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 					 const struct cpumask *new_mask,
 					 u32 flags,
 					 struct rq *rq,
-					 struct rq_flags *rf)
+					 struct rq_flags *rf,
+					 struct cpumask **puser_mask)
 	__releases(rq->lock)
 	__releases(p->pi_lock)
 {
@@ -2908,7 +2925,7 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
 
 	__do_set_cpus_allowed(p, new_mask, flags);
 
-	return affine_move_task(rq, p, rf, dest_cpu, flags);
+	return affine_move_task(rq, p, rf, dest_cpu, flags, puser_mask);
 
 out:
 	task_rq_unlock(rq, p, rf);
@@ -2926,7 +2943,8 @@ static int __set_cpus_allowed_ptr_locked(struct task_struct *p,
  * call is not atomic; no spinlocks may be held.
  */
 static int __set_cpus_allowed_ptr(struct task_struct *p,
-				  const struct cpumask *new_mask, u32 flags)
+				  const struct cpumask *new_mask, u32 flags,
+				  struct cpumask **puser_mask)
 {
 	struct cpumask *alloc_mask = NULL;
 	struct rq_flags rf;
@@ -2934,8 +2952,11 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
 	int ret;
 
 	rq = task_rq_lock(p, &rf);
-	if (p->user_cpus_ptr) {
 
+	/*
+	 * user_cpus_ptr masking is skipped if puser_mask is defined.
+	 */
+	if (p->user_cpus_ptr && !puser_mask) {
 		/*
 		 * A scratch cpumask is allocated on the percpu runqueues
 		 * to enable additional masking with user_cpus_ptr. This
@@ -2958,7 +2979,8 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
 	}
 
 
-	ret = __set_cpus_allowed_ptr_locked(p, new_mask, flags, rq, &rf);
+	ret = __set_cpus_allowed_ptr_locked(p, new_mask, flags, rq, &rf,
+					    puser_mask);
 	if (unlikely(alloc_mask))
 		kfree(alloc_mask);
 	return ret;
@@ -2966,7 +2988,7 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
 
 int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 {
-	return __set_cpus_allowed_ptr(p, new_mask, 0);
+	return __set_cpus_allowed_ptr(p, new_mask, 0, NULL);
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
@@ -3004,7 +3026,7 @@ static int restrict_cpus_allowed_ptr(struct task_struct *p,
 		goto err_unlock;
 	}
 
-	return __set_cpus_allowed_ptr_locked(p, new_mask, 0, rq, &rf);
+	return __set_cpus_allowed_ptr_locked(p, new_mask, 0, rq, &rf, NULL);
 
 err_unlock:
 	task_rq_unlock(rq, p, &rf);
@@ -3551,7 +3573,7 @@ void sched_set_stop_task(int cpu, struct task_struct *stop)
 
 static inline int __set_cpus_allowed_ptr(struct task_struct *p,
 					 const struct cpumask *new_mask,
-					 u32 flags)
+					 u32 flags, struct cpumask **puser_mask)
 {
 	return set_cpus_allowed_ptr(p, new_mask);
 }
@@ -8109,29 +8131,25 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask, bool save
 		}
 		cpumask_copy(user_mask, mask);
 	}
-again:
-	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK);
+
+	retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK,
+					user_mask ? &user_mask : NULL);
 	if (retval)
 		goto out_free_new_mask;
 
-	cpuset_cpus_allowed(p, cpus_allowed);
-	if (!cpumask_subset(new_mask, cpus_allowed)) {
+	for (;;) {
+		cpuset_cpus_allowed(p, cpus_allowed);
+		if (cpumask_subset(new_mask, cpus_allowed))
+			break;
+
 		/*
 		 * We must have raced with a concurrent cpuset update.
 		 * Just reset the cpumask to the cpuset's cpus_allowed.
 		 */
 		cpumask_copy(new_mask, cpus_allowed);
-		goto again;
+		retval = __set_cpus_allowed_ptr(p, new_mask, SCA_CHECK, NULL);
 	}
 
-	if (save_mask) {
-		unsigned long flags;
-
-		/* Use pi_lock to synchronize changes to user_cpus_ptr */
-		raw_spin_lock_irqsave(&p->pi_lock, flags);
-		swap(p->user_cpus_ptr, user_mask);
-		raw_spin_unlock_irqrestore(&p->pi_lock, flags);
-	}
 out_free_new_mask:
 	kfree(user_mask);
 	free_cpumask_var(new_mask);
Signature · 1 line
2.31.1
maintainer
#12 lore raw

Re: [PATCH v6 4/5] sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race

Quoted 10 lines · Waiman Long

Racing is possible between set_cpus_allowed_ptr() and sched_setaffinity() or between multiple sched_setaffinity() calls from different CPUs. To resolve these race conditions, we need to update both user_cpus_ptr and cpus_mask in a single lock critical section instead of separated ones. This requires moving the user_cpus_ptr update to affine_move_task() before doing task_rq_unlock().

A new argument puser_mask is added to affine_move_task(), __set_cpus_allowed_ptr_locked() and __set_cpus_allowed_ptr() to do that.

Quoted 6 lines

Ideally, user_cpus_ptr should only be updated if the sched_setaffinity() is successful. However, this patch will update user_cpus_ptr when the first call to __set_cpus_allowed_ptr() is successful. However, if there is racing between sched_setaffinity() and cpuset update, the subsequent calls to __set_cpus_allowed_ptr() may fail but the user_cpus_ptr will still be updated in this corner case.

Urgh, this is a bit weird, to have a fix for a patch in the same series.

#13 lore raw

Re: [PATCH v6 4/5] sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race

Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Waiman Long

Racing is possible between set_cpus_allowed_ptr() and sched_setaffinity() or between multiple sched_setaffinity() calls from different CPUs. To resolve these race conditions, we need to update both user_cpus_ptr and cpus_mask in a single lock critical section instead of separated ones. This requires moving the user_cpus_ptr update to affine_move_task() before doing task_rq_unlock().

A new argument puser_mask is added to affine_move_task(), __set_cpus_allowed_ptr_locked() and __set_cpus_allowed_ptr() to do that.

Quoted 7 lines
Quoted 6 lines

Ideally, user_cpus_ptr should only be updated if the sched_setaffinity() is successful. However, this patch will update user_cpus_ptr when the first call to __set_cpus_allowed_ptr() is successful. However, if there is racing between sched_setaffinity() and cpuset update, the subsequent calls to __set_cpus_allowed_ptr() may fail but the user_cpus_ptr will still be updated in this corner case.

Urgh, this is a bit weird, to have a fix for a patch in the same series.

This is just to make each patch simpler and easier to read.

Cheers,
Longman

maintainer
#14 lore raw

Re: [PATCH v6 4/5] sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race

In reply to Waiman Long
Quoted 17 lines · Waiman Long
kernel/sched/core.c +2 -0
@@ -2722,6 +2734,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 			complete = true;
 		}
 
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (push_task) {
@@ -2793,6 +2806,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 		if (flags & SCA_MIGRATE_ENABLE)
 			p->migration_flags &= ~MDF_PUSH;
 
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (!stop_pending) {

kernel/sched/core.c static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag · line 2834

Quoted 8 lines
kernel/sched/core.c +2 -0
@@ -2813,6 +2827,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 				complete = true;
 			}
 		}
+
+		swap_user_cpus_ptr(p, puser_mask);
 		task_rq_unlock(rq, p, rf);
 
 		if (complete)

I'm not at all sure about those.

Would it not be much simpler to keep the update of cpus_mask and cpus_user_mask together, always ensuring that cpus_user_mask is a strict superset of cpus_mask ? That is, set_cpus_allowed_common() seems like the right place to me.

I'm thinking this also means blowing away user_mask when we do a full reset of the cpus_mask when we do an affnity break.

#15 lore raw

Re: [PATCH v6 4/5] sched: Handle set_cpus_allowed_ptr() & sched_setaffinity() race

Quoted 28 lines · Peter Zijlstra
Quoted 17 lines · Waiman Long
kernel/sched/core.c +2 -0
@@ -2722,6 +2734,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
  			complete = true;
  		}
 
+		swap_user_cpus_ptr(p, puser_mask);
  		task_rq_unlock(rq, p, rf);
 
  		if (push_task) {
@@ -2793,6 +2806,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
  		if (flags & SCA_MIGRATE_ENABLE)
  			p->migration_flags &= ~MDF_PUSH;
 
+		swap_user_cpus_ptr(p, puser_mask);
  		task_rq_unlock(rq, p, rf);
 
  		if (!stop_pending) {

kernel/sched/core.c static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag · line 2834

Quoted 8 lines
kernel/sched/core.c +2 -0
@@ -2813,6 +2827,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
  				complete = true;
  			}
  		}
+
+		swap_user_cpus_ptr(p, puser_mask);
  		task_rq_unlock(rq, p, rf);
 
  		if (complete)

I'm not at all sure about those.

Quoted 7 lines

Would it not be much simpler to keep the update of cpus_mask and cpus_user_mask together, always ensuring that cpus_user_mask is a strict superset of cpus_mask ? That is, set_cpus_allowed_common() seems like the right place to me.

I'm thinking this also means blowing away user_mask when we do a full reset of the cpus_mask when we do an affnity break.

Thanks for the suggestion. I will need to think about what will be best way to pass in those additional parameters there.

Cheers,
Longman

#16 lore raw

PATCH v6 5/5 sched: Fix sched_setaffinity() and fork/clone() race

In reply to Waiman Long

sched_setaffinity() can also race with a concurrent fork/clone() syscall calling dup_user_cpus_ptr(). That may lead to a use after free problem. Fix that by protecting the cpumask copying using pi_lock of the source task.

---
 kernel/sched/core.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/sched/core.c +5 -0
index 6cd1177fbcea..71a3fdcb6b6b 100644
@@ -2593,6 +2593,8 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)
 int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src,
 		      int node)
 {
+	unsigned long flags;
+
 	if (!src->user_cpus_ptr)
 		return 0;
 
@@ -2600,7 +2602,10 @@ int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src,
 	if (!dst->user_cpus_ptr)
 		return -ENOMEM;
 
+	/* Use pi_lock to protect content of user_cpus_ptr */
+	raw_spin_lock_irqsave(&src->pi_lock, flags);
 	cpumask_copy(dst->user_cpus_ptr, src->user_cpus_ptr);
+	raw_spin_unlock_irqrestore(&src->pi_lock, flags);
 	return 0;
 }
 
Signature · 1 line
2.31.1