Kernel thread's CPU affinity with isolcpus kernel boot argument
Hi all,
we use an Intel Core i7 machine (64bit) to use the cores > 0 for real time tasks.
For this the kernel boot parameter “isolcpus=1-31” is provided. That works fine for all user space tasks.
However, I noticed that most kernel threads use the affinity mask 0xff.
I can change this by using “taskset”. However, I am wondering if there is mechanism that forces the kernel to consider the value of “isolcpus” also for kernel threads.
Kernel version in use is: 3.18.20-rt18.
Thanks for any feedback on this.
Best regards
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · * Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:
However, I noticed that most kernel threads use the affinity mask
0xff. I can change this by using “taskset”. However, I am wondering if there is mechanism that forces the kernel to consider the value of “isolcpus” also for kernel threads.
not that I am aware of. There are a few per-CPU threads which have to stay the way they are. I would have expected that others like kworker/u* respect the isol CPUs and stay away.
Quoted 3 lines
Best regards
Mathias
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 4 lines · * Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:
However, I noticed that most kernel threads use the affinity mask
0xff. I can change this by using ???taskset???. However, I am wondering if there is mechanism that forces the kernel to consider the value of ???isolcpus??? also for kernel threads.not that I am aware of. There are a few per-CPU threads which have to stay the way they are. I would have expected that others like kworker/u* respect the isol CPUs and stay away.
I suppose the higher level question to Mathias is: are you seeing these threads perturb your application? Or, are you just observing they are affinitized to isolcpus?
Josh
Am 25.02.2016 um 15:43 schrieb Josh Cartwright:
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 7 lines · * Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:
Quoted 5 lines
However, I noticed that most kernel threads use the affinity mask
0xff. I can change this by using ???taskset???. However, I am wondering if there is mechanism that forces the kernel to consider the value of ???isolcpus??? also for kernel threads.
Quoted 7 lines
Quoted 3 lines
not that I am aware of. There are a few per-CPU threads which have to stay the way they are. I would have expected that others like kworker/u* respect the isol CPUs and stay away.
I suppose the higher level question to Mathias is: are you seeing these threads perturb your application? Or, are you just observing they are affinitized to isolcpus?
We use isolcpus=1-31 to reserve all cores but core 0 for real time stuff. I just noticed, that some kernel threads have an affinity to cores > 0. So far I did not observe any disturbance from these threads.
Thanks
Mathias
Quoted 3 lines · Sebastian Andrzej Siewior
not that I am aware of. There are a few per-CPU threads which have to stay the way they are. I would have expected that others like kworker/u* respect the isol CPUs and stay away.
Isolcpus is applied only for user-space threads. For instance, you still need to move rcu offload callbacks away from isolated CPUs, and all other kernel threads as well.
In the special case of kworker/u* threads, you need to use a special interface to do it. The interface is the following file:
/sys/devices/virtual/workqueue/cpumask
It was implemented on the following commit:
042f7df workqueue: Allow modifying low level unbound workqueue cpumask
But the default CPU mask is the "cpu_possible_mask".
Sometime ago I sent a patch to modify the default mask of the kworker/u* threads, to avoid them on isolated CPUs by default, but it was not acked. It was not explicitly nacked, though.
http://www.gossamer-threads.com/lists/linux/kernel/2218495
Should I try it again?
-- Daniel
Quoted 8 lines · Daniel Bristot de Oliveira
Quoted 3 lines · Sebastian Andrzej Siewior
not that I am aware of. There are a few per-CPU threads which have to stay the way they are. I would have expected that others like kworker/u* respect the isol CPUs and stay away.
Isolcpus is applied only for user-space threads. For instance, you still need to move rcu offload callbacks away from isolated CPUs, and all other kernel threads as well.
RCU is special since this is a per-CPU thread. However Paul did make an option where you can move those to another CPU.
Quoted 4 lines
In the special case of kworker/u* threads, you need to use a special interface to do it. The interface is the following file:
/sys/devices/virtual/workqueue/cpumask
I see it documented in "Documentation/ABI". Oh wait…
Quoted 6 lines
It was implemented on the following commit:
042f7df workqueue: Allow modifying low level unbound workqueue cpumask
But the default CPU mask is the "
cpu_possible_mask".
Quoted 7 lines
Sometime ago I sent a patch to modify the default mask of the kworker/u* threads, to avoid them on isolated CPUs by default, but it was not acked. It was not explicitly nacked, though.
http://www.gossamer-threads.com/lists/linux/kernel/2218495
Should I try it again?
Atleast they do have a default cpumask which you can modify. I don't see why the kworker should be special here and avoid the isolcpu mask. I *do* support the default mask being !isolcpus and not cpu_possible. If you have the time to try again, please do so.
Quoted 3 lines
-- Daniel
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html