Skip to contentSkip to conversation

sched: Store restrict_cpus_allowed_ptr() call state

PATCH Superseded Waiman Long 2 participants 11 messages Jan 21, 2023 – Jan 30, 2023

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched: Store restrict_cpus_allowed_ptr() call state

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

In order to fix this regression, a new scheduler flag task_struct::cpus_allowed_restricted is now added to track if force_compatible_cpus_allowed_ptr() has been called before or not. This patch also updates the comments in force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() and handles their interaction with sched_setaffinity().

Fixes 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask")
---
 include/linux/sched.h |  3 +++
 kernel/sched/core.c   | 46 ++++++++++++++++++++++++++++++++++---------
 kernel/sched/sched.h  |  2 ++
 3 files changed, 42 insertions(+), 9 deletions(-)
include/linux/sched.h +3 -0
index 853d08f7562b..f93f62a1f858 100644
@@ -886,6 +886,9 @@ struct task_struct {
 	unsigned			sched_contributes_to_load:1;
 	unsigned			sched_migrated:1;
 
+	/* restrict_cpus_allowed_ptr() bit, serialized by scheduler locks */
+	unsigned			cpus_allowed_restricted:1;
+
 	/* Force alignment to the next boundary: */
 	unsigned			:0;
 
kernel/sched/core.c +37 -9

86 lines not on this page · raw message

kernel/sched/sched.h +2 -0
index 771f8ddb7053..adcef29d5479 100644
@@ -2293,6 +2293,8 @@ extern struct task_struct *pick_next_task_idle(struct rq *rq);
 #define SCA_MIGRATE_DISABLE	0x02
 #define SCA_MIGRATE_ENABLE	0x04
 #define SCA_USER		0x08
+#define SCA_CLR_RESTRICT	0x10
+#define SCA_SET_RESTRICT	0x20
 
 #ifdef CONFIG_SMP
 
Signature · 1 line
2.31.1
#1 lore raw

Hi Waiman,

[+Thorsten given where we are in the release cycle]

Quoted 9 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

Quoted 7 lines

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Quoted 7 lines

In order to fix this regression, a new scheduler flag task_struct::cpus_allowed_restricted is now added to track if force_compatible_cpus_allowed_ptr() has been called before or not. This patch also updates the comments in force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() and handles their interaction with sched_setaffinity().

Quoted 8 lines
Fixes 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask")
---
 include/linux/sched.h |  3 +++
 kernel/sched/core.c   | 46 ++++++++++++++++++++++++++++++++++---------
 kernel/sched/sched.h  |  2 ++
 3 files changed, 42 insertions(+), 9 deletions(-)

I find this pretty invasive, but I guess it's up to Peter and Ingo. It also doesn't the whole problem for me; see below.

Quoted 19 lines
include/linux/sched.h +3 -0
index 853d08f7562b..f93f62a1f858 100644
@@ -886,6 +886,9 @@ struct task_struct {
 	unsigned			sched_contributes_to_load:1;
 	unsigned			sched_migrated:1;
 
+	/* restrict_cpus_allowed_ptr() bit, serialized by scheduler locks */
+	unsigned			cpus_allowed_restricted:1;
+
 	/* Force alignment to the next boundary: */
 	unsigned			:0;
 
kernel/sched/core.c +0 -0
index bb1ee6d7bdde..48234dc9005b 100644

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3004

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -2999,15 +2999,40 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
 	struct rq *rq;
 
 	rq = task_rq_lock(p, &rf);
+
+	if (ctx->flags & (SCA_CLR_RESTRICT | SCA_SET_RESTRICT)) {
+		p->cpus_allowed_restricted = 0;

I don't think this is ever called on the SCA_SET_RESTRICT path, as restrict_cpus_allowed_ptr() calls __set_cpus_allowed_ptr_locked() directly. In my testing, we see a failure in the following sequence:

1. A 64-bit task has an affinity of 0xf
2. It exec()s a 32-bit program and is forcefully restricted to the set

     of 32-bit-capable cores. Let's say that new mask is 0xc

3. The 32-bit task now exec()s a 64-bit program again

And now we're still stuck with 0xc after step 3 whereas we should restore 0xf.

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3014

Quoted 10 lines
kernel/sched/core.c +10 -0
+	} else if (p->cpus_allowed_restricted) {
+		/*
+		 * If force_compatible_cpus_allowed_ptr() has been called,
+		 * we can't extend cpumask to beyond what is in cpus_mask.
+		 */
+		if (!cpumask_and(rq->scratch_mask, ctx->new_mask,
+				 &p->cpus_mask)) {
+			task_rq_unlock(rq, p, &rf);
+			return -EINVAL;
+		}

Why is this masking actually needed? __sched_setaffinity() already takes into account the task_cpu_possible_mask(), which is why I asked you before [1] about cases where the saved affinity is not simply a superset of the effective affinity.

Thanks,

Will

[1] https://lore.kernel.org/r/20230120175931.GA22417@willie-the-truck

#2 lore raw
Quoted 72 lines · Will Deacon

Hi Waiman,

[+Thorsten given where we are in the release cycle]

Quoted 9 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

Quoted 7 lines

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Quoted 7 lines

In order to fix this regression, a new scheduler flag task_struct::cpus_allowed_restricted is now added to track if force_compatible_cpus_allowed_ptr() has been called before or not. This patch also updates the comments in force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() and handles their interaction with sched_setaffinity().

Quoted 8 lines
Fixes 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask")
---
  include/linux/sched.h |  3 +++
  kernel/sched/core.c   | 46 ++++++++++++++++++++++++++++++++++---------
  kernel/sched/sched.h  |  2 ++
  3 files changed, 42 insertions(+), 9 deletions(-)

I find this pretty invasive, but I guess it's up to Peter and Ingo. It also doesn't the whole problem for me; see below.

Quoted 19 lines
include/linux/sched.h +3 -0
index 853d08f7562b..f93f62a1f858 100644
@@ -886,6 +886,9 @@ struct task_struct {
  	unsigned			sched_contributes_to_load:1;
  	unsigned			sched_migrated:1;
 
+	/* restrict_cpus_allowed_ptr() bit, serialized by scheduler locks */
+	unsigned			cpus_allowed_restricted:1;
+
  	/* Force alignment to the next boundary: */
  	unsigned			:0;
 
kernel/sched/core.c +0 -0
index bb1ee6d7bdde..48234dc9005b 100644

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3004

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -2999,15 +2999,40 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
  	struct rq *rq;
 
  	rq = task_rq_lock(p, &rf);
+
+	if (ctx->flags & (SCA_CLR_RESTRICT | SCA_SET_RESTRICT)) {
+		p->cpus_allowed_restricted = 0;

I don't think this is ever called on the SCA_SET_RESTRICT path, as restrict_cpus_allowed_ptr() calls __set_cpus_allowed_ptr_locked() directly. In my testing, we see a failure in the following sequence:

Quoted 7 lines

1. A 64-bit task has an affinity of 0xf
2. It exec()s a 32-bit program and is forcefully restricted to the set

      of 32-bit-capable cores. Let's say that new mask is 0xc

3. The 32-bit task now exec()s a 64-bit program again

And now we're still stuck with 0xc after step 3 whereas we should restore 0xf.

I am sorry that missed it. You are right. For setting the cpus_allowed_restricted bit, it should be done directly in restrict_cpus_allowed_ptr().

Quoted 6 lines
Quoted 6 lines
kernel/sched/core.c +6 -0
+	} else if (p->cpus_allowed_restricted) {
+		/*
+		 * If force_compatible_cpus_allowed_ptr() has been called,
+		 * we can't extend cpumask to beyond what is in cpus_mask.
+		 */
+		if (!cpumask_and(rq->scratch_mask, ctx->new_mask,
Quoted 8 lines

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3014

Quoted 4 lines
kernel/sched/core.c +4 -0
+				 &p->cpus_mask)) {
+			task_rq_unlock(rq, p, &rf);
+			return -EINVAL;
+		}

Why is this masking actually needed? __sched_setaffinity() already takes into account the task_cpu_possible_mask(), which is why I asked you before [1] about cases where the saved affinity is not simply a superset of the effective affinity.

I kind of overlook the use of task_cpu_possible_mask() in __set_cpus_allowed_ptr_locked. So we don't really need that masking. That make the patch even simpler then. I will send out a v3.

Cheers,
Longman

#3 lore raw
Quoted 53 lines · Waiman Long
Quoted 27 lines · Will Deacon
Quoted 17 lines · Waiman Long
include/linux/sched.h +3 -0
index 853d08f7562b..f93f62a1f858 100644
@@ -886,6 +886,9 @@ struct task_struct {
  	unsigned			sched_contributes_to_load:1;
  	unsigned			sched_migrated:1;
+	/* restrict_cpus_allowed_ptr() bit, serialized by scheduler locks */
+	unsigned			cpus_allowed_restricted:1;
+
  	/* Force alignment to the next boundary: */
  	unsigned			:0;
kernel/sched/core.c +0 -0
index bb1ee6d7bdde..48234dc9005b 100644

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3003

Quoted 5 lines
kernel/sched/core.c +3 -0
@@ -2999,15 +2999,40 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
  	struct rq *rq;
  	rq = task_rq_lock(p, &rf);
+
+	if (ctx->flags & (SCA_CLR_RESTRICT | SCA_SET_RESTRICT)) {
+		p->cpus_allowed_restricted = 0;

I don't think this is ever called on the SCA_SET_RESTRICT path, as restrict_cpus_allowed_ptr() calls __set_cpus_allowed_ptr_locked() directly. In my testing, we see a failure in the following sequence:

Quoted 7 lines

1. A 64-bit task has an affinity of 0xf
2. It exec()s a 32-bit program and is forcefully restricted to the set

      of 32-bit-capable cores. Let's say that new mask is 0xc

3. The 32-bit task now exec()s a 64-bit program again

And now we're still stuck with 0xc after step 3 whereas we should restore 0xf.

I am sorry that missed it. You are right. For setting the cpus_allowed_restricted bit, it should be done directly in restrict_cpus_allowed_ptr().

Quoted 6 lines
Quoted 6 lines
kernel/sched/core.c +6 -0
+	} else if (p->cpus_allowed_restricted) {
+		/*
+		 * If force_compatible_cpus_allowed_ptr() has been called,
+		 * we can't extend cpumask to beyond what is in cpus_mask.
+		 */
+		if (!cpumask_and(rq->scratch_mask, ctx->new_mask,
Quoted 8 lines

kernel/sched/core.c static int __set_cpus_allowed_ptr(struct task_struct *p, · line 3014

Quoted 4 lines
kernel/sched/core.c +4 -0
+				 &p->cpus_mask)) {
+			task_rq_unlock(rq, p, &rf);
+			return -EINVAL;
+		}

Why is this masking actually needed? __sched_setaffinity() already takes into account the task_cpu_possible_mask(), which is why I asked you before [1] about cases where the saved affinity is not simply a superset of the effective affinity.

Quoted 3 lines

I kind of overlook the use of task_cpu_possible_mask() in __set_cpus_allowed_ptr_locked. So we don't really need that masking. That make the patch even simpler then. I will send out a v3.

Thanks; I'll give it a spin when I see it.

Will

#4 lore raw
In reply to Will Deacon
Quoted 21 lines · Will Deacon

Hi Waiman,

[+Thorsten given where we are in the release cycle]

Quoted 9 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

Quoted 7 lines

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

sched_getaffinity() just return whatever is in cpus_mask. Normally, it should be whatever cpus are allowed by the current cpuset unless sched_setaffinity() has been called before. So after a call to relax_compatible_cpus_allowed_ptr(), it should revert back to the cpu_allowed set in the cpuset. If sched_setaffinity() has been called, it should revert back to the intersection of the current cpuset and user_cpus_ptr.

Cheers,
Longman

#5 lore raw
Quoted 18 lines · Waiman Long
Quoted 17 lines · Will Deacon
Quoted 16 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 7 lines
Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 > /sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 > /sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f

but with v6.2-rc5 that last taskset invocation gives:

pid 334's current affinity mask: 1

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

Will

#6 lore raw
Quoted 37 lines · Will Deacon
Quoted 18 lines · Waiman Long
Quoted 17 lines · Will Deacon
Quoted 16 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 6 lines
Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 > /sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 > /sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f
Quoted 6 lines

but with v6.2-rc5 that last taskset invocation gives:

pid 334's current affinity mask: 1

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

I see what you mean now. Hotplug doesn't work quite well now because user_cpus_ptr has been repurposed to store the value set of sched_setaffinity() but not the previous cpus_mask before force_compatible_cpus_allowed_ptr().

One possible solution is to modify the hotplug related code to check for the cpus_allowed_restricted, and if set, check task_cpu_possible_mask() to see if the cpu can be added back to its cpus_mask. I will take a further look at that later.

Cheers,
Longman

#7 lore raw
Quoted 40 lines · Waiman Long
Quoted 39 lines · Will Deacon
Quoted 18 lines · Waiman Long
Quoted 17 lines · Will Deacon
Quoted 16 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 6 lines
Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 > /sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 > /sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f

but with v6.2-rc5 that last taskset invocation gives:

Quoted 8 lines
Quoted 4 lines

pid 334's current affinity mask: 1

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

I see what you mean now. Hotplug doesn't work quite well now because user_cpus_ptr has been repurposed to store the value set of sched_setaffinity() but not the previous cpus_mask before force_compatible_cpus_allowed_ptr().

One possible solution is to modify the hotplug related code to check for the cpus_allowed_restricted, and if set, check task_cpu_possible_mask() to see if the cpu can be added back to its cpus_mask. I will take a further look at that later.

Wait, I think the cpuset hotplug code should be able to restore the right cpumask since task_cpu_possible_mask() is used there. Is cpuset enabled? Does the test works without allow_mismatched_32bit_el0?

I think there may be a bug somewhere.

Cheers,
Longman

#8 lore raw
Quoted 43 lines · Waiman Long
Quoted 42 lines · Waiman Long
Quoted 41 lines · Will Deacon
Quoted 18 lines · Waiman Long
Quoted 17 lines · Will Deacon
Quoted 16 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 6 lines
Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 > /sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 > /sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f

but with v6.2-rc5 that last taskset invocation gives:

pid 334's current affinity mask: 1

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

I see what you mean now. Hotplug doesn't work quite well now because user_cpus_ptr has been repurposed to store the value set of sched_setaffinity() but not the previous cpus_mask before force_compatible_cpus_allowed_ptr().

One possible solution is to modify the hotplug related code to check for the cpus_allowed_restricted, and if set, check task_cpu_possible_mask() to see if the cpu can be added back to its cpus_mask. I will take a further look at that later.

Wait, I think the cpuset hotplug code should be able to restore the right cpumask since task_cpu_possible_mask() is used there. Is cpuset enabled? Does the test works without allow_mismatched_32bit_el0?

BTW, if the test result is from running on a kernel built with the v2 patch, it is the unexpected result. That should be fixed in the v3 patch.

Cheers,
Longman

#9 lore raw
Quoted 67 lines · Waiman Long
Quoted 66 lines · Waiman Long
Quoted 64 lines · Waiman Long
Quoted 47 lines · Will Deacon
Quoted 24 lines · Waiman Long
Quoted 23 lines · Will Deacon
Quoted 22 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric
CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve
the user requested
cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted
with "allow_mismatched_32bit_el0" on the
command-line. The arch code will
(amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing
a 32-bit or a 64-bit
task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a
previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 7 lines
Quoted 4 lines

I'd argue it's more than just a performance regression -- the affinity
masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following
with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 >
/sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 >
/sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f
Quoted 6 lines

but with v6.2-rc5 that last taskset invocation gives:

pid 334's current affinity mask: 1

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

I see what you mean now. Hotplug doesn't work quite well now because user_cpus_ptr has been repurposed to store the value set of sched_setaffinity() but not the previous cpus_mask before force_compatible_cpus_allowed_ptr().

One possible solution is to modify the hotplug related code to check for the cpus_allowed_restricted, and if set, check task_cpu_possible_mask() to see if the cpu can be added back to its cpus_mask. I will take a further look at that later.

Quoted 6 lines
Quoted 3 lines

Wait, I think the cpuset hotplug code should be able to restore the right cpumask since task_cpu_possible_mask() is used there. Is cpuset enabled? Does the test works without allow_mismatched_32bit_el0?

BTW, if the test result is from running on a kernel built with the v2 patch, it is the unexpected result. That should be fixed in the v3 patch.

The failure listed above is on vanilla v6.2-rc5. Your v2 has other issues, as described in:

https://lore.kernel.org/r/20230124194805.GA27257@willie-the-truck

Will

#10 lore raw
In reply to Will Deacon
Quoted 37 lines · Will Deacon
Quoted 18 lines · Waiman Long
Quoted 17 lines · Will Deacon
Quoted 16 lines · Waiman Long

The user_cpus_ptr field was originally added by commit b90ca8badbd1 ("sched: Introduce task_struct::user_cpus_ptr to track requested affinity"). It was used only by arm64 arch due to possible asymmetric CPU setup.

Since commit 8f9ea86fdf99 ("sched: Always preserve the user requested cpumask"), task_struct::user_cpus_ptr is repurposed to store user requested cpu affinity specified in the sched_setaffinity().

This results in a performance regression in an arm64 system when booted with "allow_mismatched_32bit_el0" on the command-line. The arch code will (amongst other things) calls force_compatible_cpus_allowed_ptr() and relax_compatible_cpus_allowed_ptr() when exec()'ing a 32-bit or a 64-bit task respectively. Now a call to relax_compatible_cpus_allowed_ptr() will always result in a __sched_setaffinity() call whether there is a previous force_compatible_cpus_allowed_ptr() call or not.

Quoted 6 lines
Quoted 3 lines

I'd argue it's more than just a performance regression -- the affinity masks are set incorrectly, which is a user visible thing (i.e. sched_getaffinity() gives unexpected values).

Can your elaborate a bit more on what you mean by getting unexpected sched_getaffinity() results? You mean the result is wrong after a relax_compatible_cpus_allowed_ptr(). Right?

Yes, as in the original report. If, on a 4-CPU system, I do the following with v6.1 and "allow_mismatched_32bit_el0" on the kernel cmdline:

# for c in `seq 1 3`; do echo 0 > /sys/devices/system/cpu/cpu$c/online; done
# yes > /dev/null &
[1] 334
# taskset -p 334
pid 334's current affinity mask: 1
# for c in `seq 1 3`; do echo 1 > /sys/devices/system/cpu/cpu$c/online; done
# taskset -p 334
pid 334's current affinity mask: f
Quoted 6 lines

but with v6.2-rc5 that last taskset invocation gives:

pid 334's current affinity mask: 1

so, yes, the performance definitely regresses, but that's because the affinity mask is wrong!

Are you using cgroup v1 or v2? Are your process in the root cgroup/cpuset or a child cgroup/cpuset under root?

If you are using cgroup v1 in a child cpuset, cpuset.cpus works more like cpuset.cpus_effective. IOW, with an offline and then online hotplug event, the cpu will be permanently lost from the cpuset. So the above is an expected result. If you using cgroup v2, the cpuset should be able to recover the lost cpu after the online event. If your process is in the root cpuset, the cpu will not be lost too. Alternatively, if you mount the v1 cpuset with the "cpuset_v2_mode" flag, it will behave more like v2 and recover the lost cpu.

I ran a similar cpu offline/online test with cgroup v1 and v2 and confirm that v1 has the above behavior and v2 is fine.

I believe what you have observed above may not be related to my sched patch as I can't see how it will affect cpu hotplug at all.

Cheers,
Longman