Skip to contentSkip to conversation

question about cpusets vs sched_setaffinity()

DISCUSSION Chris Friesen 2 participants 4 messages Dec 10, 2015 – Dec 14, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

I've got a question about the interaction between cpusets and sched_setaffinity().

If I put a task into a cpuset and then call sched_setaffinity() on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)

However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original sched_setaffinity() call.

Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original sched_setaffinity() call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.

Thanks,
Chris

PS: Not subscribed to the list, please CC me on replies.

#1 lore raw
Quoted 20 lines · Chris Friesen

Hi,

I've got a question about the interaction between cpusets and sched_setaffinity().

If I put a task into a cpuset and then call sched_setaffinity() on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)

However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original sched_setaffinity() call.

Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original sched_setaffinity() call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.

Quoted 4 lines

Thanks,
Chris

PS: Not subscribed to the list, please CC me on replies.

Hi,

This behavior seems a bit odd to me as well - if you've done a sched_setaffinity() call to a subset of the cpus of a cpuset that the task in contained within, any change to the cpuset cpus will wipe away the sched_setaffinity() settings even if they continue to be a subset of the cpuset cpus.

To add the behavior you are describing, I think requires another cpumask_t field in the task_struct. Where we could store the last requested mask value for sched_setaffinity() and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out any sched_setaffinity() settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.

You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?

Thanks,

-Jason

#2 lore raw
Quoted 16 lines · Jason Baron
Quoted 14 lines · Chris Friesen

If I put a task into a cpuset and then call sched_setaffinity() on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)

However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original sched_setaffinity() call.

Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original sched_setaffinity() call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.

<snip>

Quoted 7 lines

To add the behavior you are describing, I think requires another cpumask_t field in the task_struct. Where we could store the last requested mask value for sched_setaffinity() and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out any sched_setaffinity() settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.

Agreed, the additional field in the task_struct makes sense. Personally I don't think that adding a task to a cpuset should wipe out any previously-set affinity, I think it should take the intersection for that case as well.

In this environment it might make sense to have separate queries to return the requested and actual affinity.

Quoted 5 lines

You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?

I ended up doing something like this where I had a top-level cpuset and a number of child cpusets, each with an exclusive subset of the CPUs assigned to it. But it meant that I needed more complicated code to figure out which tasks needed to go into which child cpusets, and more complicated code to handle removing a CPU from the top-level cpuset (since you have to remove it from any children first).

Chris

#3 lore raw
Quoted 28 lines · Chris Friesen
Quoted 16 lines · Jason Baron
Quoted 14 lines · Chris Friesen

If I put a task into a cpuset and then call sched_setaffinity() on it, it will be affined to the intersection of the two sets of cpus. (Those specified on the set, and those specified in the syscall.)

However, if I then change the cpus in the cpuset the process affinity will simply be overwritten by the new cpuset affinity. It does not seem to take into account any restrictions from the original sched_setaffinity() call.

Wouldn't it make more sense to affine the process to the intersection between the new set of cpus from the cpuset, and the current process affinity? That way if I explicitly masked out certain CPUs in the original sched_setaffinity() call then they would remain masked out regardless of changes to the set of cpus assigned to the cpuset.

<snip>

Quoted 7 lines

To add the behavior you are describing, I think requires another cpumask_t field in the task_struct. Where we could store the last requested mask value for sched_setaffinity() and use that when updating the cpus for a cpuset via an intersection as you described. I think adding a task to a cpuset still should wipe out any sched_setaffinity() settings - but that would depend on the desired semantics here. It would also require a knob so as not to break existing behavior by default.

Quoted 8 lines

Agreed, the additional field in the task_struct makes sense. Personally I don't think that adding a task to a cpuset should wipe out any previously-set affinity, I think it should take the intersection for that case as well.

In this environment it might make sense to have separate queries to return the requested and actual affinity.

So because cpumask_t is dimensioned by NR_CPUS, I think we would need a pointer to the cpumask_t field. And we could allocate it when we want the cpus set by sched_setaffinity() to persist across the cgroup cpuset cpu changes. I think you are right that a flag to sched_[set|get]affinity() for this case might be nice - but that would require a new syscall...

Quoted 6 lines
Quoted 5 lines

You could also create a child cgroup for the process that you don't want to change and set the cpus on that cgroup instead of using sched_setaffinity(). Then you change the cpus for the parent cgroup and that shouldn't affect the child as long as the child cgroup is a subset. But its not entirely clear to me if that addresses your use-case?

Quoted 8 lines

I ended up doing something like this where I had a top-level cpuset and a number of child cpusets, each with an exclusive subset of the CPUs assigned to it. But it meant that I needed more complicated code to figure out which tasks needed to go into which child cpusets, and more complicated code to handle removing a CPU from the top-level cpuset (since you have to remove it from any children first).

Chris

I agree that it would be nice to improve this interface, since you are creating extra cgroups here just to sort of work around this.

Thanks,

-Jason