Skip to contentSkip to conversation

cgroup/cpuset: Keep current cpus list if cpus affinity was explicitly set

PATCH Applied Waiman Long 4 participants 22 messages Jul 28, 2022 – Jul 29, 2022

Subsystems (3)
lore
Share & tools
lore

Waiman Long

author maintainer
#0 lore raw

PATCH 1/2 cgroup/cpuset: Keep current cpus list if cpus affinity was explicitly set

It was found that any change to the current cpuset hierarchy may reset the cpus_allowed list of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

That is especially problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

Fix this problem by adding a flag in the task structure to indicate that a task has their cpus affinity explicitly set before and make cpuset code not to change their cpus_allowed list unless the user chosen cpu list is no longer a subset of the cpus_allowed list of the cpuset itself.

With that change in place, it was verified that tasks that have its cpus affinity explicitly set will not be affected by changes made to the v2 cgroup.subtree_control files.

---
 include/linux/sched.h  |  1 +
 kernel/cgroup/cpuset.c | 18 ++++++++++++++++--
 kernel/sched/core.c    |  1 +
 3 files changed, 18 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index c46f3a63b758..60ae022fa842 100644
@@ -815,6 +815,7 @@ struct task_struct {
 
 	unsigned int			policy;
 	int				nr_cpus_allowed;
+	int				cpus_affinity_set;
 	const cpumask_t			*cpus_ptr;
 	cpumask_t			*user_cpus_ptr;
 	cpumask_t			cpus_mask;
kernel/cgroup/cpuset.c +16 -2
index 71a418858a5e..c47757c61f39 100644
@@ -704,6 +704,20 @@ static int validate_change(struct cpuset *cur, struct cpuset *trial)
 	return ret;
 }
 
+/*
+ * Don't change the cpus_allowed list if cpus affinity has been explicitly
+ * set before unless the current cpu list is not a subset of the new cpu list.
+ */
+static int cpuset_set_cpus_allowed_ptr(struct task_struct *p,
+				       const struct cpumask *new_mask)
+{
+	if (p->cpus_affinity_set && cpumask_subset(p->cpus_ptr, new_mask))
+		return 0;
+
+	p->cpus_affinity_set = 0;
+	return set_cpus_allowed_ptr(p, new_mask);
+}
+
 #ifdef CONFIG_SMP
 /*
  * Helper routine for generate_sched_domains().
@@ -1130,7 +1144,7 @@ static void update_tasks_cpumask(struct cpuset *cs)
 
 	css_task_iter_start(&cs->css, 0, &it);
 	while ((task = css_task_iter_next(&it)))
-		set_cpus_allowed_ptr(task, cs->effective_cpus);
+		cpuset_set_cpus_allowed_ptr(task, cs->effective_cpus);
 	css_task_iter_end(&it);
 }
 
@@ -2303,7 +2317,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 		 * can_attach beforehand should guarantee that this doesn't
 		 * fail.  TODO: have a better way to handle failure here
 		 */
-		WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach));
+		WARN_ON_ONCE(cpuset_set_cpus_allowed_ptr(task, cpus_attach));
 
 		cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to);
 		cpuset_update_task_spread_flag(cs, task);
kernel/sched/core.c +1 -0
index da0bf6fe9ecd..ab8ea6fa92db 100644
@@ -8034,6 +8034,7 @@ __sched_setaffinity(struct task_struct *p, const struct cpumask *mask)
 	if (retval)
 		goto out_free_new_mask;
 
+	p->cpus_affinity_set = 1;
 	cpuset_cpus_allowed(p, cpus_allowed);
 	if (!cpumask_subset(new_mask, cpus_allowed)) {
 		/*
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH 2/2 cgroup: Skip subtree root in cgroup_update_dfl_csses()

The cgroup_update_dfl_csses() function updates css associations when a cgroup's subtree_control file is modified. Any changes made to a cgroup's subtree_control file, however, will only affect its descendants but not the cgroup itself. So there is no point in migrating csses associated with that cgroup. We can skip them instead.

---
 kernel/cgroup/cgroup.c | 9 +++++++++
 1 file changed, 9 insertions(+)
kernel/cgroup/cgroup.c +9 -0
index 13c8e91d7862..1151ff44d578 100644
@@ -2960,6 +2960,15 @@ static int cgroup_update_dfl_csses(struct cgroup *cgrp)
 	cgroup_for_each_live_descendant_pre(dsct, d_css, cgrp) {
 		struct cgrp_cset_link *link;
 
+		/*
+		 * As cgroup_update_dfl_csses() is only called by
+		 * cgroup_apply_control(). The csses associated with the
+		 * given cgrp will not be affected by changes made to
+		 * its subtree_control file. We can skip them.
+		 */
+		if (dsct == cgrp)
+			continue;
+
 		list_for_each_entry(link, &dsct->cset_links, cset_link)
 			cgroup_migrate_add_src(link->cset, dsct, &mgctx);
 	}
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] cgroup: Skip subtree root in cgroup_update_dfl_csses()

Quoted 4 lines · Waiman Long

The cgroup_update_dfl_csses() function updates css associations when a cgroup's subtree_control file is modified. Any changes made to a cgroup's subtree_control file, however, will only affect its descendants but not the cgroup itself.

I find this correct.

Quoted 2 lines

So there is no point in migrating csses associated with that cgroup. We can skip them instead.

Alone it's not such a big win but it componds with the recent Tejun's threadgroup_rwsem elision.

Quoted 3 lines
---
 kernel/cgroup/cgroup.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Feel free to have

author maintainer
#3 lore raw

Re: [PATCH 2/2] cgroup: Skip subtree root in cgroup_update_dfl_csses()

Quoted 5 lines · Michal Koutný
Quoted 4 lines · Waiman Long

The cgroup_update_dfl_csses() function updates css associations when a cgroup's subtree_control file is modified. Any changes made to a cgroup's subtree_control file, however, will only affect its descendants but not the cgroup itself.

Quoted 7 lines

I find this correct.

Quoted 2 lines

So there is no point in migrating csses associated with that cgroup. We can skip them instead.

Alone it's not such a big win but it componds with the recent Tejun's threadgroup_rwsem elision.

It is more an optimization patch trying to not waste cpu time doing unnecessary work.

Quoted 6 lines
Quoted 3 lines
---
  kernel/cgroup/cgroup.c | 9 +++++++++
  1 file changed, 9 insertions(+)

Feel free to have

Thanks for the review.

Cheers,
Longman

maintainer
#4 lore raw

Re: [PATCH 2/2] cgroup: Skip subtree root in cgroup_update_dfl_csses()

In reply to Michal Koutný
Quoted 6 lines · Michal Koutný
Quoted 4 lines · Waiman Long

The cgroup_update_dfl_csses() function updates css associations when a cgroup's subtree_control file is modified. Any changes made to a cgroup's subtree_control file, however, will only affect its descendants but not the cgroup itself.

Quoted 7 lines

I find this correct.

Quoted 2 lines

So there is no point in migrating csses associated with that cgroup. We can skip them instead.

Alone it's not such a big win but it componds with the recent Tejun's threadgroup_rwsem elision.

The chance is that if you're writing to a cgroup's subtree_control, that cgroup is gonna be empty. The only case I can think of that this would make a difference is w/ threaded cgroups, but it does make sense.

Thanks.

Signature · 1 line
tejun
maintainer Applied
#5 lore raw

Re: [PATCH 2/2] cgroup: Skip subtree root in cgroup_update_dfl_csses()

In reply to Waiman Long
Quoted 7 lines · Waiman Long

The cgroup_update_dfl_csses() function updates css associations when a cgroup's subtree_control file is modified. Any changes made to a cgroup's subtree_control file, however, will only affect its descendants but not the cgroup itself. So there is no point in migrating csses associated with that cgroup. We can skip them instead.

Applied to cgroup/for-5.20.

Thanks.

Signature · 1 line
tejun
maintainer
#6 lore raw
In reply to Waiman Long

Hello.

Quoted 4 lines · Waiman Long

It was found that any change to the current cpuset hierarchy may reset the cpus_allowed list of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before.

I'm surprised this went so long unnoticed / unreported.

Could it be users relied on that implicit affinity reset?

Quoted 3 lines

That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

This should apply only to tasks that were extracted out of the root cgroup, no? (OK, those are all processes practically.)

(Even without your second patch, the scope should be limited because of src_cset==dst_cset check in cgroup_migrate_prepare_dst().)

Quoted 3 lines

That is especially problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

One could also argue that for such processes, cgroup hierarchy should be first configured and only then they start and set own affinity.

Quoted 4 lines

Fix this problem by adding a flag in the task structure to indicate that a task has their cpus affinity explicitly set before and make cpuset code not to change their cpus_allowed list unless the user chosen cpu list is no longer a subset of the cpus_allowed list of the cpuset itself.

I'm uneasy with the occasional revert of this flag, i.e. the task who set their affinity would sometimes have it overwritten and sometimes not (which might have been relied on, especially with writes into cpuset.cpus).
(But I have no better answer than the counter-argument above since there's no easier way to detect the implicit migrations.)

Also, is there similar effect with memory binding?

Thanks,
Michal

author maintainer
#7 lore raw
Quoted 10 lines · Michal Koutný

Hello.

Quoted 4 lines · Waiman Long

It was found that any change to the current cpuset hierarchy may reset the cpus_allowed list of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before.

I'm surprised this went so long unnoticed / unreported.

Could it be users relied on that implicit affinity reset?

As said, it is more easily triggered in a cgroup v2 environment. Systemd, on a cgroup v2 environment, will write "+cpuset" to the root cgroup's subtree_control file when a new container is instantiated. I don't know why it is doing that, but that cause problem as it resets all the cpus_allowed list assignment. Cgroup v1 doesn't have this problem.

Quoted 5 lines
Quoted 3 lines

That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

This should apply only to tasks that were extracted out of the root cgroup, no? (OK, those are all processes practically.)

The reset is done on all cgroups in a particular subtree. In the case of cgroup root, it is all the processes in the system.

Quoted 14 lines

(Even without your second patch, the scope should be limited because of src_cset==dst_cset check in cgroup_migrate_prepare_dst().)

Quoted 3 lines

That is especially problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

One could also argue that for such processes, cgroup hierarchy should be first configured and only then they start and set own affinity.

Quoted 4 lines

Fix this problem by adding a flag in the task structure to indicate that a task has their cpus affinity explicitly set before and make cpuset code not to change their cpus_allowed list unless the user chosen cpu list is no longer a subset of the cpus_allowed list of the cpuset itself.

Quoted 6 lines

I'm uneasy with the occasional revert of this flag, i.e. the task who set their affinity would sometimes have it overwritten and sometimes not (which might have been relied on, especially with writes into cpuset.cpus).
(But I have no better answer than the counter-argument above since there's no easier way to detect the implicit migrations.)

I also thought about that. Another possible alternative is to use the intersection of cpuset's cpu list and task's own cpu list as long as it is not empty. Reducing the number of cpus assigned to a task may produce some unexpected behavior too.

Quoted 2 lines

Also, is there similar effect with memory binding?

I think so, but memory binding is less frequently used and its effect is less noticeable.

Cheers,
Longman

Quoted 4 lines

Thanks,
Michal

maintainer
#8 lore raw
Quoted 1 line · Waiman Long

Cgroup v1 doesn't have this problem.

v1 analogy would be:

echo 2-3 >$dst/cpuset.cpus
# job runs in $dst
# one task T in $dst sets affinity just to one cpu # I rethink my config, I want to allow $dst more space echo 2-5 >$dst/cpuset.cpus

Most tasks in $dst happily utilize the new cpus but it breaks affinity for T -- this must have been broken since ever.

(Or I'd argue that per-thread affinities are just recommendations, if I have a task for nohz CPU, I should enforce its placement with cpuset from the beginning.)

Michal

author maintainer
#9 lore raw
Quoted 10 lines · Michal Koutný
Quoted 1 line · Waiman Long

Cgroup v1 doesn't have this problem.

v1 analogy would be:

echo 2-3 >$dst/cpuset.cpus
# job runs in $dst
# one task T in $dst sets affinity just to one cpu # I rethink my config, I want to allow $dst more space echo 2-5 >$dst/cpuset.cpus

Quoted 6 lines

Most tasks in $dst happily utilize the new cpus but it breaks affinity for T -- this must have been broken since ever.

(Or I'd argue that per-thread affinities are just recommendations, if I have a task for nohz CPU, I should enforce its placement with cpuset from the beginning.)

I should have clarified that what I meant is systemd on a cgroup v1 environment doesn't cause this cpu list reset to happen. It doesn't mean that cgroup v1 has no similar problem. Sorry for the confusion.

Cheers,
Longman

#10 lore raw
In reply to Waiman Long
Quoted 6 lines · Waiman Long
Quoted 2 lines · Michal Koutný

This should apply only to tasks that were extracted out of the root cgroup, no? (OK, those are all processes practically.)

The reset is done on all cgroups in a particular subtree. In the case of cgroup root, it is all the processes in the system.

Quoted 1 line

I've been briefly playing with this, tasks in the cgroup root don't seem affected on my end (QEMU + buildroot + latest tip/sched/core):

  $ mount -t cgroup2 none /sys/fs/cgroup
  $ /root/loop.sh &
  $ PID=$!
  $ taskset -pc 2-3 $PID
  pid 177's current affinity list: 0-3
  pid 177's new affinity list: 2,3
  $ echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
  $ taskset -pc $PID
  pid 177's current affinity list: 2,3

However tasks extracted out as mentioned by Michal definitely are:

  $ mount -t cgroup2 none /sys/fs/cgroup
  $ /root/loop.sh &
  $ PID=$!
  $ taskset -pc 2-3 $PID
  pid 172's current affinity list: 0-3
  pid 172's new affinity list: 2,3
  $ mkdir /sys/fs/cgroup/foobar
  $ echo $PID > /sys/fs/cgroup/foobar/cgroup.procs
  $ taskset -pc $PID
  pid 172's current affinity list: 2,3
  $ echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
  $ taskset -pc $PID
  pid 172's current affinity list: 0-3

IIUC this is just what happens anytime a task gets migrated to a new cpuset. Initially loop.sh remains attached to the root cpuset, and the echo +cpuset migrates it to the /foobar one.

Does that match what you're seeing?

author maintainer
#11 lore raw
Quoted 35 lines · Valentin Schneider
Quoted 5 lines · Waiman Long
Quoted 2 lines · Michal Koutný

This should apply only to tasks that were extracted out of the root cgroup, no? (OK, those are all processes practically.)

The reset is done on all cgroups in a particular subtree. In the case of cgroup root, it is all the processes in the system.

I've been briefly playing with this, tasks in the cgroup root don't seem affected on my end (QEMU + buildroot + latest tip/sched/core):

   $ mount -t cgroup2 none /sys/fs/cgroup
   $ /root/loop.sh &
   $ PID=$!
   $ taskset -pc 2-3 $PID
   pid 177's current affinity list: 0-3
   pid 177's new affinity list: 2,3
   $ echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
   $ taskset -pc $PID
   pid 177's current affinity list: 2,3

However tasks extracted out as mentioned by Michal definitely are:

   $ mount -t cgroup2 none /sys/fs/cgroup
   $ /root/loop.sh &
   $ PID=$!
   $ taskset -pc 2-3 $PID
   pid 172's current affinity list: 0-3
   pid 172's new affinity list: 2,3
   $ mkdir /sys/fs/cgroup/foobar
   $ echo $PID > /sys/fs/cgroup/foobar/cgroup.procs
   $ taskset -pc $PID
   pid 172's current affinity list: 2,3
   $ echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
   $ taskset -pc $PID
   pid 172's current affinity list: 0-3
Quoted 6 lines

IIUC this is just what happens anytime a task gets migrated to a new cpuset. Initially loop.sh remains attached to the root cpuset, and the echo +cpuset migrates it to the /foobar one.

Does that match what you're seeing?

Yes. echo "+cpuset" to subtree_control means tasks in the child cgroups will move to new cpusets. Those new cpusets will have the same cpu lists as the parent unless the cpuset.cpus files are explicitly written to. This patch will ensure that tasks that have explicitly set their cpu affinity won't be affected by this change.

Cheers,
Longman

maintainer
#12 lore raw
In reply to Waiman Long

Hello,

Quoted 12 lines · Waiman Long

It was found that any change to the current cpuset hierarchy may reset the cpus_allowed list of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

That is especially problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

Quoted 8 lines

Fix this problem by adding a flag in the task structure to indicate that a task has their cpus affinity explicitly set before and make cpuset code not to change their cpus_allowed list unless the user chosen cpu list is no longer a subset of the cpus_allowed list of the cpuset itself.

With that change in place, it was verified that tasks that have its cpus affinity explicitly set will not be affected by changes made to the v2 cgroup.subtree_control files.

I think the underlying cause here is cpuset overwriting the cpumask the user configured but that's a longer discussion.

kernel/cgroup/cpuset.c static int validate_change(struct cpuset *cur, struct cpuset *trial) · line 719

Quoted 13 lines
kernel/cgroup/cpuset.c +13 -0
+/*
+ * Don't change the cpus_allowed list if cpus affinity has been explicitly
+ * set before unless the current cpu list is not a subset of the new cpu list.
+ */
+static int cpuset_set_cpus_allowed_ptr(struct task_struct *p,
+				       const struct cpumask *new_mask)
+{
+	if (p->cpus_affinity_set && cpumask_subset(p->cpus_ptr, new_mask))
+		return 0;
+
+	p->cpus_affinity_set = 0;
+	return set_cpus_allowed_ptr(p, new_mask);
+}

I wonder whether the more predictable behavior would be always not resetting the cpumask if it's a subset of the new_mask. Also, shouldn't this check p->cpus_mask instead of p->cpus_ptr?

Thanks.

Signature · 1 line
tejun
author maintainer
#13 lore raw
Quoted 26 lines · Tejun Heo

Hello,

Quoted 12 lines · Waiman Long

It was found that any change to the current cpuset hierarchy may reset the cpus_allowed list of the tasks in the affected cpusets to the default cpuset value even if those tasks have cpus affinity explicitly set by the users before. That is especially easy to trigger under a cgroup v2 environment where writing "+cpuset" to the root cgroup's cgroup.subtree_control file will reset the cpus affinity of all the processes in the system.

That is especially problematic in a nohz_full environment where the tasks running in the nohz_full CPUs usually have their cpus affinity explicitly set and will behave incorrectly if cpus affinity changes.

Quoted 8 lines

Fix this problem by adding a flag in the task structure to indicate that a task has their cpus affinity explicitly set before and make cpuset code not to change their cpus_allowed list unless the user chosen cpu list is no longer a subset of the cpus_allowed list of the cpuset itself.

With that change in place, it was verified that tasks that have its cpus affinity explicitly set will not be affected by changes made to the v2 cgroup.subtree_control files.

I think the underlying cause here is cpuset overwriting the cpumask the user configured but that's a longer discussion.

Quoted 15 lines

kernel/cgroup/cpuset.c static int validate_change(struct cpuset *cur, struct cpuset *trial) · line 719

Quoted 13 lines
kernel/cgroup/cpuset.c +13 -0
+/*
+ * Don't change the cpus_allowed list if cpus affinity has been explicitly
+ * set before unless the current cpu list is not a subset of the new cpu list.
+ */
+static int cpuset_set_cpus_allowed_ptr(struct task_struct *p,
+				       const struct cpumask *new_mask)
+{
+	if (p->cpus_affinity_set && cpumask_subset(p->cpus_ptr, new_mask))
+		return 0;
+
+	p->cpus_affinity_set = 0;
+	return set_cpus_allowed_ptr(p, new_mask);
+}

I wonder whether the more predictable behavior would be always not resetting the cpumask if it's a subset of the new_mask.

There can be a counter argument that if a user found out that there is not enough cpus in a cpuset to meet its performance target, one can always increase the number of cpus in the cpuset. Generalizing this behavior to all the tasks irrespective if they have explicitly set cpus affinity before will disallow this use case.

Quoted 2 lines

Also, shouldn't this check
p->cpus_mask instead of p->cpus_ptr?

You are right. I should have used cpus_mask instead. Will send out a v2.

Cheers,
Longman

maintainer
#14 lore raw

Hello,

Quoted 5 lines · Waiman Long

There can be a counter argument that if a user found out that there is not enough cpus in a cpuset to meet its performance target, one can always increase the number of cpus in the cpuset. Generalizing this behavior to all the tasks irrespective if they have explicitly set cpus affinity before will disallow this use case.

This is nasty. The real solution here is separating out what user requested and the mask that cpuset (or cpu hotplug) needs to apply on top. ie. remember what the user requested in a separate cpumask and compute the intersection into p->cpus_maks whenever something changes and apply fallbacks on that final mask. Multiple parties updating the same variable is never gonna lead to anything consistent and we're patching up for whatever the immediate use case seems to need at the moment. That said, I'm not necessarily against patching it up but if you're interested in delving into it deeper, that'd be great.

Thanks.

Signature · 1 line
tejun
author maintainer
#15 lore raw
Quoted 9 lines · Tejun Heo

Hello,

Quoted 5 lines · Waiman Long

There can be a counter argument that if a user found out that there is not enough cpus in a cpuset to meet its performance target, one can always increase the number of cpus in the cpuset. Generalizing this behavior to all the tasks irrespective if they have explicitly set cpus affinity before will disallow this use case.

This is nasty.

That is a nasty example, I know. There may be users depending on the existing behavior even if they don't know it. So I am a bit hesitant to change the default behavior like that. On the other hand, tasks that have explicitly set its cpu affinity certainly don't want to have unexpected change to that.

Quoted 9 lines

The real solution here is separating out what user requested
and the mask that cpuset (or cpu hotplug) needs to apply on top. ie. remember what the user requested in a separate cpumask and compute the intersection into p->cpus_maks whenever something changes and apply fallbacks on that final mask. Multiple parties updating the same variable is never gonna lead to anything consistent and we're patching up for whatever the immediate use case seems to need at the moment. That said, I'm not necessarily against patching it up but if you're interested in delving into it deeper, that'd be great.

I believe the current code is already restricting what cpu affinity that a user can request by limiting to those allowed by the current cpuset. Hotplug is another issue that may need to be addressed. I will update my patch to make it handle hotplug in a more graceful way.

Thanks,
Longman

maintainer
#16 lore raw

Hello,

Quoted 7 lines · Waiman Long
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Waiman Long

There can be a counter argument that if a user found out that there is not enough cpus in a cpuset to meet its performance target, one can always increase the number of cpus in the cpuset. Generalizing this behavior to all the tasks irrespective if they have explicitly set cpus affinity before will disallow this use case.

Quoted 7 lines
Quoted 1 line

This is nasty.

That is a nasty example, I know. There may be users depending on the existing behavior even if they don't know it. So I am a bit hesitant to change the default behavior like that. On the other hand, tasks that have explicitly set its cpu affinity certainly don't want to have unexpected change to that.

Yeah, I hear you. I'm on the same page.

Quoted 10 lines
Quoted 9 lines

The real solution here is separating out what user requested
and the mask that cpuset (or cpu hotplug) needs to apply on top. ie. remember what the user requested in a separate cpumask and compute the intersection into p->cpus_maks whenever something changes and apply fallbacks on that final mask. Multiple parties updating the same variable is never gonna lead to anything consistent and we're patching up for whatever the immediate use case seems to need at the moment. That said, I'm not necessarily against patching it up but if you're interested in delving into it deeper, that'd be great.

Quoted 4 lines

I believe the current code is already restricting what cpu affinity that a user can request by limiting to those allowed by the current cpuset. Hotplug is another issue that may need to be addressed. I will update my patch to make it handle hotplug in a more graceful way.

So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

Thanks.

Signature · 1 line
tejun
author maintainer
#17 lore raw
Quoted 31 lines · Tejun Heo

Hello,

Quoted 7 lines · Waiman Long
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Waiman Long

There can be a counter argument that if a user found out that there is not enough cpus in a cpuset to meet its performance target, one can always increase the number of cpus in the cpuset. Generalizing this behavior to all the tasks irrespective if they have explicitly set cpus affinity before will disallow this use case.

Quoted 6 lines
Quoted 1 line

This is nasty.

That is a nasty example, I know. There may be users depending on the existing behavior even if they don't know it. So I am a bit hesitant to change the default behavior like that. On the other hand, tasks that have explicitly set its cpu affinity certainly don't want to have unexpected change to that.

Yeah, I hear you. I'm on the same page.

Quoted 9 lines
Quoted 9 lines

The real solution here is separating out what user requested
and the mask that cpuset (or cpu hotplug) needs to apply on top. ie. remember what the user requested in a separate cpumask and compute the intersection into p->cpus_maks whenever something changes and apply fallbacks on that final mask. Multiple parties updating the same variable is never gonna lead to anything consistent and we're patching up for whatever the immediate use case seems to need at the moment. That said, I'm not necessarily against patching it up but if you're interested in delving into it deeper, that'd be great.

Quoted 4 lines

I believe the current code is already restricting what cpu affinity that a user can request by limiting to those allowed by the current cpuset. Hotplug is another issue that may need to be addressed. I will update my patch to make it handle hotplug in a more graceful way.

Quoted 11 lines

af
So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

OK, I see what you want to accomplish. To fully address this issue, we will need to have a new cpumask variable in the the task structure which will be allocated if sched_setaffinity() is ever called. I can rework my patch to use this approach.

Thanks,
Longman

maintainer
#18 lore raw

Hello, Waiman.

Quoted 11 lines · Waiman Long
Quoted 10 lines

So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

Quoted 4 lines

OK, I see what you want to accomplish. To fully address this issue, we will need to have a new cpumask variable in the the task structure which will be allocated if sched_setaffinity() is ever called. I can rework my patch to use this approach.

Yeah, we'd need to track what user requested separately from the currently effective cpumask. Let's make sure that the scheduler folks are on board before committing to the idea tho. Peter, Ingo, what do you guys think?

Thanks.

Signature · 1 line
tejun
#19 lore raw
Quoted 14 lines · Tejun Heo

Hello, Waiman.

Quoted 11 lines · Waiman Long
Quoted 10 lines

So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

Quoted 9 lines
Quoted 4 lines

OK, I see what you want to accomplish. To fully address this issue, we will need to have a new cpumask variable in the the task structure which will be allocated if sched_setaffinity() is ever called. I can rework my patch to use this approach.

Yeah, we'd need to track what user requested separately from the currently effective cpumask. Let's make sure that the scheduler folks are on board before committing to the idea tho. Peter, Ingo, what do you guys think?

FWIW on a runtime overhead side of things I think it'll be OK as that should be just an extra mask copy in sched_setaffinity() and a subset check / cpumask_and() in set_cpus_allowed_ptr(). The policy side is a bit less clear (when, if ever, do we clear the user-defined mask? Will it keep haunting us even after moving a task to a disjoint cpuset partition?).

There's also if/how that new mask should be exposed, because attaching a task to a cpuset will now yield a not-necessarily-obvious affinity - e.g. in the thread affinity example above, if the initial affinity setting was done ages ago by some system tool, IMO the user needs a way to be able to expect/understand the result of 0x555 rather than 0xfff.

While I'm saying this, I don't think anything exposes p->user_cpus_ptr, but then again that one is for "special" hardware...

Quoted 4 lines

Thanks.

--
tejun

author maintainer
#20 lore raw
Quoted 22 lines · Valentin Schneider
Quoted 13 lines · Tejun Heo

Hello, Waiman.

Quoted 10 lines · Waiman Long
Quoted 10 lines

So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

Quoted 8 lines
Quoted 4 lines

OK, I see what you want to accomplish. To fully address this issue, we will need to have a new cpumask variable in the the task structure which will be allocated if sched_setaffinity() is ever called. I can rework my patch to use this approach.

Yeah, we'd need to track what user requested separately from the currently effective cpumask. Let's make sure that the scheduler folks are on board before committing to the idea tho. Peter, Ingo, what do you guys think?

Quoted 5 lines

FWIW on a runtime overhead side of things I think it'll be OK as that should be just an extra mask copy in sched_setaffinity() and a subset check / cpumask_and() in set_cpus_allowed_ptr(). The policy side is a bit less clear (when, if ever, do we clear the user-defined mask? Will it keep haunting us even after moving a task to a disjoint cpuset partition?).

The runtime overhead should be minimal. It is the behavioral side that we should be careful about. It is a change in existing behavior and we don't want to cause surprise to the users. Currently, a task that set its cpu affinity explicitly will have its affinity reset whenever there is any change to the cpuset it belongs to or a hotplug event touch any cpu in the current cpuset. The new behavior we are proposing here is that it will try its best to keep the cpu affinity that the user requested within the constraint of the current cpuset as well as the cpu hotplug state.

Quoted 6 lines

There's also if/how that new mask should be exposed, because attaching a task to a cpuset will now yield a not-necessarily-obvious affinity - e.g. in the thread affinity example above, if the initial affinity setting was done ages ago by some system tool, IMO the user needs a way to be able to expect/understand the result of 0x555 rather than 0xfff.

Users can use sched_getaffinity(2) to retrieve the current cpu affinity. It is up to users to set another one if they don't like the current one. I don't think we need to return what the previous requested cpu affinity is. They are suppose to know that or they can set their own if they don't like it. \

Cheers,
Longman

Quoted 8 lines

While I'm saying this, I don't think anything exposes p->user_cpus_ptr, but then again that one is for "special" hardware...

Quoted 4 lines

Thanks.

--
tejun

author maintainer
#21 lore raw
Quoted 33 lines · Waiman Long
Quoted 22 lines · Valentin Schneider
Quoted 13 lines · Tejun Heo

Hello, Waiman.

Quoted 10 lines · Waiman Long
Quoted 10 lines

So, the patch you proposed is making the code remember one special aspect of user requested configuration - whether it configured it or not, and trying to preserve that particular state as cpuset state changes. It addresses the immediate problem but it is a very partial approach. Let's say a task wanna be affined to one logical thread of each core and set its mask to 0x5555. Now, let's say cpuset got enabled and enforced 0xff and affined the task to 0xff. After a while, the cgroup got more cpus allocated and its cpuset now has 0xfff. Ideally, what should happen is the task now having the effective mask of 0x555. In practice, tho, it either would get 0xf55 or 0x55 depending on which way we decide to misbehave.

Quoted 8 lines
Quoted 4 lines

OK, I see what you want to accomplish. To fully address this issue, we will need to have a new cpumask variable in the the task structure which will be allocated if sched_setaffinity() is ever called. I can rework my patch to use this approach.

Yeah, we'd need to track what user requested separately from the currently effective cpumask. Let's make sure that the scheduler folks are on board before committing to the idea tho. Peter, Ingo, what do you guys think?

Quoted 5 lines

FWIW on a runtime overhead side of things I think it'll be OK as that should be just an extra mask copy  in sched_setaffinity() and a subset check / cpumask_and() in set_cpus_allowed_ptr(). The policy side is a bit less clear (when, if ever, do we clear the user-defined mask? Will it keep haunting us even after moving a task to a disjoint cpuset partition?).

The runtime overhead should be minimal. It is the behavioral side that we should be careful about. It is a change in existing behavior and we don't want to cause surprise to the users. Currently, a task that set its cpu affinity explicitly will have its affinity reset whenever there is any change to the cpuset it belongs to or a hotplug event touch any cpu in the current cpuset. The new behavior we are proposing here is that it will try its best to keep the cpu affinity that the user requested within the constraint of the current cpuset as well as the cpu hotplug state.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

There's also if/how that new mask should be exposed, because attaching a task to a cpuset will now yield a not-necessarily-obvious affinity - e.g. in the thread affinity example above, if the initial affinity setting was done ages ago by some system tool, IMO the user needs a way to be able to expect/understand the result of 0x555 rather than 0xfff.

Users can use sched_getaffinity(2) to retrieve the current cpu affinity. It is up to users to set another one if they don't like the current one. I don't think we need to return what the previous requested cpu affinity is. They are suppose to know that or they can set their own if they don't like it. \

Looking at Will's series that introduced user_cpus_ptr, I think we can overlay our proposal on top of that. So calling sched_setaffinity() will also update user_cpus_ptr. We may still need a flag to indicate whether user_cpus_ptr is set up because of sched_setaffinity() or due to a call to force_compatible_cpus_allowed_ptr() from arm64 arch code. That will make our work easier as some of the infrastructure is already there. I am looking forward for your feedback.

Thanks,
Longman