Skip to contentSkip to conversation

sched/core: Mask out offline CPUs when user_cpus_ptr is used

PATCH In review Waiman Long 4 participants 14 messages Jul 18, 2025 – Jan 12

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched/core: Mask out offline CPUs when user_cpus_ptr is used

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +4 -3
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
Signature · 1 line
2.50.0
#1 lore raw
Quoted 19 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Quoted 7 lines
Fixes da019032819a ("sched: Enforce user requested affinity")
---
  kernel/sched/core.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)

Sorry, I forgot to change the patch title. Will send out a v3.

Cheers,
Longman

Quoted 21 lines
kernel/sched/core.c +4 -3
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
  	rq = task_rq_lock(p, &rf);
  	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
  	 */
  	if (p->user_cpus_ptr &&
  	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
  		ctx->new_mask = rq->scratch_mask;
 
  	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
#2 lore raw

PATCH v3 1/1 sched/core: Skip user_cpus_ptr masking if no online CPU left

In reply to Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +4 -3
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
Signature · 1 line
2.50.0
maintainer
#3 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Quoted 7 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Does this make sense for cpuset v2 (or no cpuset at all for that matter)? I'm asking whether this mask modification could only be extracted into cpuset-v1.c (like cgroup_tranfer_tasks() or a new function)

Thanks,
Michal

#4 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Quoted 11 lines · Michal Koutný
Quoted 7 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Does this make sense for cpuset v2 (or no cpuset at all for that matter)? I'm asking whether this mask modification could only be extracted into cpuset-v1.c (like cgroup_tranfer_tasks() or a new function)

This corner case as specified in Chen Ridong's patch only happens with a cpuset v1 environment, but it is still the case that the default cpu affinity of the root cgroup (with or without CONFIG_CGROUPS) will include offline CPUs, if present. So it still make senses to skip the sched_setaffinity() setting if there is no online CPU left, though it will be much harder to have such a condition without using cpuset v1.

Cheers,
Longman

maintainer
#5 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Hi.

I had a look after a while (thanks for reminders Ridong).

Quoted 4 lines · Waiman Long

This corner case as specified in Chen Ridong's patch only happens with a cpuset v1 environment, but it is still the case that the default cpu affinity of the root cgroup (with or without CONFIG_CGROUPS) will include offline CPUs, if present.

IIUC, the generic sched_setaffinity(2) is ready for that, simply returning an EINVAL.

Quoted 3 lines

So it still make senses to skip the sched_setaffinity() setting if there is no online CPU left, though it will be much harder to have such a condition without using cpuset v1.

That sounds like there'd be no issue without cpuset v1 and the source of the warning has quite a telling comment:

	 * fail.  TODO: have a better way to handle failure here
	 */
	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));

The trouble is that this is from cpuset_attach() (cgroup_subsys.attach) where no errors are expected. So I'd say the place for the check should be earlier in cpuset_can_attach() [1]. I'm not sure if that's universally immune against cpu offlining but it'd be sufficient for the reported sequential offlining.

HTH,
Michal

[1] Although the error propagates, it ends up without recovery in remove_tasks_in_empty_cpuset() "only" as an error message. But that's likely all what can be done in this workfn context -- it's better than silently skipping the migration as consequence of this patch.

#6 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Quoted 11 lines · Michal Koutný

Hi.

I had a look after a while (thanks for reminders Ridong).

Quoted 4 lines · Waiman Long

This corner case as specified in Chen Ridong's patch only happens with a cpuset v1 environment, but it is still the case that the default cpu affinity of the root cgroup (with or without CONFIG_CGROUPS) will include offline CPUs, if present.

IIUC, the generic sched_setaffinity(2) is ready for that, simply returning an EINVAL.

The modified code will not be executed when called from sched_setaffiity() as the SCA_USER flag will be set.

In the described scenario, sched_setaffinity() was called without failure as the request was valid at the time.

Quoted 11 lines
Quoted 4 lines

So it still make senses to skip the sched_setaffinity() setting if there is no online CPU left, though it will be much harder to have such a condition without using cpuset v1.

That sounds like there'd be no issue without cpuset v1 and the source of the warning has quite a telling comment:

	 * fail.  TODO: have a better way to handle failure here
	 */
	WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
Quoted 5 lines

The trouble is that this is from cpuset_attach() (cgroup_subsys.attach) where no errors are expected. So I'd say the place for the check should be earlier in cpuset_can_attach() [1]. I'm not sure if that's universally immune against cpu offlining but it'd be sufficient for the reported sequential offlining.

Cpuset1 has no concept of effective cpumask  that excludes offline CPUs unless "cpuset_v2_mode" mount option is used. So when the cpuset has no CPU left, it will force migrate the tasks to its parent and the __set_cpus_allowed_ptr() function will be invoked. The parent will likely have those offline CPUs in their cpus_allowed list and __set_cpus_allowed_ptr_locked() will be called with only the offline CPUs causing the warning. Migrating to the top_cpuset is probably not needed to illustrate the problem.

Cheers,
Longman

Quoted 7 lines

HTH,
Michal

[1] Although the error propagates, it ends up without recovery in remove_tasks_in_empty_cpuset() "only" as an error message. But that's likely all what can be done in this workfn context -- it's better than silently skipping the migration as consequence of this patch.

Tested-by on v3 1/1
#7 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

In reply to Waiman Long
Quoted 38 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -2
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*

kernel/sched/core.c int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx) · line 3182

Quoted 10 lines
kernel/sched/core.c +3 -1
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
#8 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Quoted 50 lines · Chen Ridong
Quoted 47 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +4 -3
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
Quoted 4 lines

kernel/sched/core.c int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx) · line 3182

Quoted 1 line
kernel/sched/core.c +0 -0
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);

Friendly ping.

Best regards,
Ridong

#9 lore raw

Re: [PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Quoted 58 lines · Chen Ridong
Quoted 50 lines · Chen Ridong
Quoted 47 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to the top cpuset.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +4 -3
index 81c6df746df1..208f8af73134 100644
@@ -3170,12 +3170,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
Quoted 4 lines

kernel/sched/core.c int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx) · line 3182

Quoted 1 line
kernel/sched/core.c +0 -0
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
Quoted 5 lines

Friendly ping.

Best regards,
Ridong

Could someone please review this patch?

Signature · 2 lines
Best regards,
Ridong
#10 lore raw

PATCH RESEND v3 1/1 sched/core: Skip user_cpus_ptr masking if no online CPU left

In reply to Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to top_cpuset. Task migration is not needed for cpuset v2.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +4 -3
index f1ebf67b48e2..66cd21582822 100644
@@ -3146,12 +3146,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
Signature · 1 line
2.51.0
maintainer Reviewed-by on v3 1/1
#11 lore raw

Re: [RESEND PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

Hi Waiman.

On Wed, Oct 29, 2025 at 05:27:24PM -0400, Waiman Long <[email protected]> wrote:
...

Quoted 7 lines
Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

(I had looked at v2 back then and reached the conclusion such a fix had made sense.)

Reviewed-by on v3 1/1
#12 lore raw

Re: [RESEND PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

In reply to Waiman Long
Quoted 39 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to top_cpuset. Task migration is not needed for cpuset v2.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -2
index f1ebf67b48e2..66cd21582822 100644
@@ -3146,12 +3146,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*

kernel/sched/core.c int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx) · line 3158

Quoted 10 lines
kernel/sched/core.c +3 -1
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);

This patch have been tested with our test project.

Signature · 2 lines
Best regards,
Ridong
#13 lore raw

Re: [RESEND PATCH v3] sched/core: Skip user_cpus_ptr masking if no online CPU left

In reply to Waiman Long
Quoted 39 lines · Waiman Long

Chen Ridong reported that cpuset could report a kernel warning for a task due to set_cpus_allowed_ptr() returning failure in the corner case that:

1) the task used sched_setaffinity(2) to set its CPU affinity mask to be the same as the cpuset.cpus of its cpuset,
2) all the CPUs assigned to that cpuset were taken offline, and
3) cpuset v1 is in use and the task had to be migrated to top_cpuset. Task migration is not needed for cpuset v2.

Due to the fact that CPU affinity of the tasks in the top cpuset are not updated when a CPU hotplug online/offline event happens, offline CPUs are included in CPU affinity of those tasks. It is possible that further masking with user_cpus_ptr set by sched_setaffinity(2) in __set_cpus_allowed_ptr() will leave only offline CPUs in the new mask causing the subsequent call to __set_cpus_allowed_ptr_locked() to return failure with an empty CPU affinity.

Fix this failure by skipping user_cpus_ptr masking if there is no online CPU left.

Fixes da019032819a ("sched: Enforce user requested affinity")
---
 kernel/sched/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -2
index f1ebf67b48e2..66cd21582822 100644
@@ -3146,12 +3146,13 @@ int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
-	 * flags are set.
+	 * Masking should be skipped if SCA_USER, any of the SCA_MIGRATE_*

kernel/sched/core.c int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx) · line 3158

Quoted 10 lines
kernel/sched/core.c +3 -1
+	 * flags are set or no online CPU left.
 	 */
 	if (p->user_cpus_ptr &&
 	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
-	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
+	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) &&
+	    cpumask_intersects(rq->scratch_mask, cpu_active_mask))
 		ctx->new_mask = rq->scratch_mask;
 
 	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);

Hi, Peter,

This is an issue that should be addressed. Should we proceed with applying this patch, or is more review needed?

Signature · 2 lines
Best regards,
Ridong