question about cpusets vs sched_setaffinity()
Hi,
I've got a question about the interaction between cpusets and sched_setaffinity().
If I put a task into a cpuset and then call sched_setaffinity() on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)
However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original sched_setaffinity() call.
Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original sched_setaffinity() call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.
Thanks,
Chris
PS: Not subscribed to the list, please CC me on replies.
Quoted 20 lines · Chris Friesen
Hi,
I've got a question about the interaction between cpusets and
sched_setaffinity().If I put a task into a cpuset and then call
sched_setaffinity()on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original
sched_setaffinity()call.Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original
sched_setaffinity()call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.
Quoted 4 lines
Thanks,
ChrisPS: Not subscribed to the list, please CC me on replies.
Hi,
This behavior seems a bit odd to me as well - if you've done a sched_setaffinity() call to a subset of the cpus of a cpuset that the task in contained within, any change to the cpuset cpus will wipe away the sched_setaffinity() settings even if they continue to be a subset of the cpuset cpus.
To add the behavior you are describing, I think requires another cpumask_t field in the task_struct. Where we could store the last requested mask value for sched_setaffinity() and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out any sched_setaffinity() settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.
You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?
Thanks,
-Jason
Quoted 16 lines · Jason Baron
Quoted 14 lines · Chris Friesen
If I put a task into a cpuset and then call
sched_setaffinity()on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original
sched_setaffinity()call.Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original
sched_setaffinity()call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.
<snip>
Quoted 7 lines
To add the behavior you are describing, I think requires another
cpumask_tfield in thetask_struct. Where we could store the last requested mask value forsched_setaffinity()and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out anysched_setaffinity()settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.
Agreed, the additional field in the task_struct makes sense. Personally I don't think that adding a task to a cpuset should wipe out any previously-set affinity, I think it should take the intersection for that case as well.
In this environment it might make sense to have separate queries to return the requested and actual affinity.
Quoted 5 lines
You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using
sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?
I ended up doing something like this where I had a top-level cpuset and a number of child cpusets, each with an exclusive subset of the CPUs assigned to it. But it meant that I needed more complicated code to figure out which tasks needed to go into which child cpusets, and more complicated code to handle removing a CPU from the top-level cpuset (since you have to remove it from any children first).
Chris
Quoted 28 lines · Chris Friesen
Quoted 16 lines · Jason Baron
Quoted 14 lines · Chris Friesen
If I put a task into a cpuset and then call
sched_setaffinity()on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original
sched_setaffinity()call.Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original
sched_setaffinity()call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.<snip>
Quoted 7 lines
To add the behavior you are describing, I think requires another
cpumask_tfield in thetask_struct. Where we could store the last requested mask value forsched_setaffinity()and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out anysched_setaffinity()settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.
Quoted 8 lines
Agreed, the additional field in the
task_structmakes sense. Personally I don't think that adding a task to a cpuset should wipe out any previously-set affinity, I think it should take the intersection for that case as well.In this environment it might make sense to have separate queries to return the requested and actual affinity.
So because cpumask_t is dimensioned by NR_CPUS, I think we would need a pointer to the cpumask_t field. And we could allocate it when we want the cpus set by sched_setaffinity() to persist across the cgroup cpuset cpu changes. I think you are right that a flag to sched_[set|get]affinity() for this case might be nice - but that would require a new syscall...
Quoted 6 lines
Quoted 5 lines
You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using
sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?
Quoted 8 lines
I ended up doing something like this where I had a top-level cpuset and a number of child cpusets, each with an exclusive subset of the CPUs assigned to it. But it meant that I needed more complicated code to figure out which tasks needed to go into which child cpusets, and more complicated code to handle removing a CPU from the top-level cpuset (since you have to remove it from any children first).
Chris
I agree that it would be nice to improve this interface, since you are creating extra cgroups here just to sort of work around this.
Thanks,
-Jason