question about rcuc/X tasks
Hi,
I'm testing the CentOS 7.3 RT kernel, 3.10.0-514.rt56.420
(I realize this is a heavily-patched kernel, but I'm hoping general information will still apply.)
My kernel is booted with "isolcpus=1-15 rcu_nocbs=1-15" as kernel parameters, and /dev/cpu_dma_latency is set to 3. The relevent config options are set as follows:
CONFIG_PREEMPT=yCONFIG_PREEMPT_RT_BASE=yCONFIG_HAVE_PREEMPT_LAZY=yCONFIG_PREEMPT_LAZY=yCONFIG_PREEMPT_RT_FULL=yCONFIG_RCU_BOOST=yCONFIG_RCU_KTHREAD_PRIO=2CONFIG_RCU_BOOST_DELAY=500CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y
The issue I'm seeing is that when I run cyclictest, ftrace shows rcuc/<cpu> running for 5-6 usec quite a few times per second.
Based on the fact that Documentation/kernel-per-CPU-kthreads.txt describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.
Does anyone have any idea what's going on here, and how I can prevent the rcuc/<cpu> tasks from waking up?
Thanks,
Chris
Hi Chris,
Quoted 4 lines · Chris Friesen
Based on the fact that
Documentation/kernel-per-CPU-kthreads.txtdescribesCONFIG_RCU_NOCB_CPU_ALL=yas a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.
As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU callbacks from running on rcuc/X threads (which are pinned to the CPU X), but the rcuc/X thread is still in charge of awaking the ruco* threads of the CPU X.
Quoted 2 lines
Does anyone have any idea what's going on here, and how I can prevent the rcuc/<cpu> tasks from waking up?
You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X thread of the task of awaking its respective ruco*s threads.
Quoted 1 line
From [1]:
rcu_nocb_poll [KNL]
Rather than requiring that offloaded CPUs (specified by rcu_nocbs= above) explicitly awaken the corresponding "rcuoN" kthreads, make these kthreads poll for callbacks. This improves the real-time response for the offloaded CPUs by relieving them of the need to wake up the corresponding kthread, but degrades energy efficiency by requiring that the kthreads periodically wake up to do the polling.
[1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt
-- Daniel
Quoted 7 lines · On Thu, 15 Dec 2016 14:47:37 +0100 Daniel Bristot de Oliveira…
Hi Chris,
Quoted 4 lines · Chris Friesen
Based on the fact that
Documentation/kernel-per-CPU-kthreads.txtdescribesCONFIG_RCU_NOCB_CPU_ALL=yas a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.
Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming this is on a 16 CPUs box, in which case I don't see much of a difference for not just using ALL as it is almost there anyway ;-)
-- Steve
Quoted 32 lines
As far as I recall, using
CONFIG_RCU_NOCB_CPU_ALL=ywill prevent the RCU callbacks from running on rcuc/X threads (which are pinned to the CPU X), but the rcuc/X thread is still in charge of awaking the ruco* threads of the CPU X.Quoted 2 lines
Does anyone have any idea what's going on here, and how I can prevent the rcuc/<cpu> tasks from waking up?
You can try to use
rcu_nocb_pollkernel parameter to unload the rcuc/X thread of the task of awaking its respective ruco*s threads.Quoted 1 line
From [1]:
rcu_nocb_poll[KNL]
Rather than requiring that offloaded CPUs (specified by rcu_nocbs= above) explicitly awaken the corresponding "rcuoN" kthreads, make these kthreads poll for callbacks. This improves the real-time response for the offloaded CPUs by relieving them of the need to wake up the corresponding kthread, but degrades energy efficiency by requiring that the kthreads periodically wake up to do the polling.[1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt
-- Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · On Thu, 15 Dec 2016 14:47:37 +0100 Daniel Bristot de Oliveira…
Hi Chris,
Quoted 4 lines · Chris Friesen
Based on the fact that
Documentation/kernel-per-CPU-kthreads.txtdescribesCONFIG_RCU_NOCB_CPU_ALL=yas a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.
Quoted 5 lines
Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming this is on a 16 CPUs box, in which case I don't see much of a difference for not just using ALL as it is almost there anyway ;-)
-- Steve
Yes, this was a 16 CPU box.
The blocker for CONFIG_RCU_NOCB_CPU_ALL is that the set of management/housekeeping CPUs is configurable by the end-user, so while it defaults to only CPU0 as management it's not guaranteed that it will always be that way.
On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.)
Thanks,
Chris
Quoted 25 lines · Chris Friesen
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · On Thu, 15 Dec 2016 14:47:37 +0100 Daniel Bristot de Oliveira…
Hi Chris,
Quoted 4 lines · Chris Friesen
Based on the fact that
Documentation/kernel-per-CPU-kthreads.txtdescribesCONFIG_RCU_NOCB_CPU_ALL=yas a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.Quoted 5 lines
Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming this is on a 16 CPUs box, in which case I don't see much of a difference for not just using ALL as it is almost there anyway ;-)
-- Steve
Yes, this was a 16 CPU box.
The blocker for
CONFIG_RCU_NOCB_CPU_ALLis that the set of management/housekeeping CPUs is configurable by the end-user, so while it defaults to only CPU0 as management it's not guaranteed that it will always be that way.
Quoted 6 lines
On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.)
The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on the corresponding CPU for correctness reasons -- they communicate with RCU core using protocols that are only single-CPU-safe. But if you are running NO_HZ_FULL, these kthreads should never run unless your user threads are doing syscalls.
So, are they actually running in your setup?
Thanx, Paul
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Chris Friesen
On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.)
Quoted 7 lines
The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on the corresponding CPU for correctness reasons -- they communicate with RCU core using protocols that are only single-CPU-safe. But if you are running
NO_HZ_FULL, these kthreads should never run unless your user threads are doing syscalls.So, are they actually running in your setup?
Yes, but I wasn't setting nohz_full. With "rcu_nocb_poll isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the rcuc/X kthreads running.
So in the non-nohz_full case, what are they waking up to do? Something timer-related?
Thanks,
Chris
Quoted 16 lines · Chris Friesen
Quoted 15 lines · Paul E. McKenney
Quoted 6 lines · Chris Friesen
On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.)
The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on the corresponding CPU for correctness reasons -- they communicate with RCU core using protocols that are only single-CPU-safe. But if you are running
NO_HZ_FULL, these kthreads should never run unless your user threads are doing syscalls.
Quoted 8 lines
Quoted 1 line
So, are they actually running in your setup?
Yes, but I wasn't setting
nohz_full. With "rcu_nocb_pollisolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the rcuc/X kthreads running.So in the non-nohz_full case, what are they waking up to do? Something timer-related?
Interesting. I need to look into this a bit. I would not expect that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up. (They are created by a per-CPU facility that creates a kthread per CPU no matter what.)
Thanx, Paul
Quoted 23 lines · Paul E. McKenney
Quoted 22 lines · Chris Friesen
Quoted 9 lines · Paul E. McKenney
Quoted 6 lines · Chris Friesen
On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.)
Quoted 7 lines
The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on the corresponding CPU for correctness reasons -- they communicate with RCU core using protocols that are only single-CPU-safe. But if you are running
NO_HZ_FULL, these kthreads should never run unless your user threads are doing syscalls.So, are they actually running in your setup?
Yes, but I wasn't setting
nohz_full. With "rcu_nocb_pollisolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the rcuc/X kthreads running.
Quoted 7 lines
Quoted 2 lines
So in the non-nohz_full case, what are they waking up to do? Something timer-related?
Interesting. I need to look into this a bit. I would not expect that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up. (They are created by a per-CPU facility that creates a kthread per CPU no matter what.)
Just be aware that this is Centos 7.3, so who knows what mishmash they've got going on. :)
This is a typical function trace of rcuc/9, the only thing running on CPU 9 is a qemu thread corresponding to a virtual CPU that is pinned to CPU 9.
<idle>-0 [009] dN..2.. 3335.422089: pick_next_task_dl <-__schedule <idle>-0 [009] dN..2.. 3335.422089: pick_next_task_rt <-__schedule rcuc/9-97 [009] d...2.. 3335.422089: __switch_to_xtra <-__switch_to rcuc/9-97 [009] d...2.. 3335.422089: finish_task_switch <-__schedule rcuc/9-97 [009] d...2.. 3335.422089: _raw_spin_unlock_irq <-finish_task_switch rcuc/9-97 [009] ....1.. 3335.422090: kthread_should_stop <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422090: kthread_should_park <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422090: rcu_cpu_kthread_should_run <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422090: rcu_cpu_kthread <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422090: local_bh_disable <-rcu_cpu_kthread rcuc/9-97 [009] ....... 3335.422090: migrate_disable <-local_bh_disable rcuc/9-97 [009] ....11. 3335.422090: pin_current_cpu <-migrate_disable rcuc/9-97 [009] .....11 3335.422090: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422090: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422090: _raw_spin_lock_irqsave <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: rcu_accelerate_cbs <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: rcu_report_qs_rnp <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: _raw_spin_unlock_irqrestore <-rcu_report_qs_rnp rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: local_bh_enable <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422092: migrate_enable <-local_bh_enable rcuc/9-97 [009] ....11. 3335.422092: unpin_current_cpu <-migrate_enable rcuc/9-97 [009] ....... 3335.422092: _raw_spin_lock_irq <-rcu_cpu_kthread rcuc/9-97 [009] d...1.. 3335.422092: rt_mutex_getprio <-rcu_cpu_kthread rcuc/9-97 [009] d...1.. 3335.422092: _raw_spin_unlock_irq <-rcu_cpu_kthread rcuc/9-97 [009] ....1.. 3335.422092: kthread_should_stop <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422092: kthread_should_park <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422092: rcu_cpu_kthread_should_run <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422092: schedule <-smpboot_thread_fn
Does this give any useful clues as to why it's waking up?
Looking at the code, rcu_cpu_kthread() is calling rcu_process_callbacks(), which will loop calling __rcu_process_callbacks() for each rcu flavor.
The fact that rcu_accelerate_cbs() and rcu_report_qs_rnp() are called within the spinlock for the first rcu flavor processed indicates that (rnp->qsmask & rdp->grpmask) is nonzero in rcu_report_qs_rdp(). I'm not sure what that actually means real-world though.
Then we loop through the other two rcu flavors and it doesn't look like we really do anything for them.
Then we return from rcu_process_callbacks() and *workp is 0 so we set the priority and return to the caller.
Chris
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Thu, 15 Dec 2016 14:47:37 +0100 Daniel Bristot de Oliveira…
Hi Chris,
On 12/12/2016 11:42 PM, Chris Friesen wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Based on the fact that
Documentation/kernel-per-CPU-kthreads.txtdescribesCONFIG_RCU_NOCB_CPU_ALL=yas a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect.Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming this is on a 16 CPUs box, in which case I don't see much of a difference for not just using ALL as it is almost there anyway ;-)
Using ALL is equivalent to rcu_nocbs=0-15 (as opposed to rcu_nocbs=1-15) on a 16-CPU box, assuming that the CPUs are numbered consecutively. The only case where I could see a difference is if you were running something like Rik's nasty Java workload on CPU 0, in which case rcu_nocbs=1-15 would be the better choice.
Thanx, Paul
Quoted 35 lines
-- Steve
Quoted 32 lines
As far as I recall, using
CONFIG_RCU_NOCB_CPU_ALL=ywill prevent the RCU callbacks from running on rcuc/X threads (which are pinned to the CPU X), but the rcuc/X thread is still in charge of awaking the ruco* threads of the CPU X.Quoted 2 lines
Does anyone have any idea what's going on here, and how I can prevent the rcuc/<cpu> tasks from waking up?
You can try to use
rcu_nocb_pollkernel parameter to unload the rcuc/X thread of the task of awaking its respective ruco*s threads.Quoted 1 line
From [1]:
rcu_nocb_poll[KNL]
Rather than requiring that offloaded CPUs (specified by rcu_nocbs= above) explicitly awaken the corresponding "rcuoN" kthreads, make these kthreads poll for callbacks. This improves the real-time response for the offloaded CPUs by relieving them of the need to wake up the corresponding kthread, but degrades energy efficiency by requiring that the kthreads periodically wake up to do the polling.[1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt
-- Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html