Add callback-free CPUs
RFC PATCH 1/1 Add callback-free CPUs
RCU callback execution can add significant OS jitter and also can degrade scheduling latency. This commit therefore adds the ability for selected CPUs ("rcu_nocbs=" boot parameter) to have their callbacks offloaded to kthreads. If the "rcu_nocb_poll" boot parameter is also specified, these kthreads will do polling, removing the need for the offloaded CPUs to do wakeups. At least one CPU must be doing normal callback processing:
currently CPU 0 cannot be selected as a no-CBs CPU. In addition, attempts to offline the last normal-CBs CPU will fail.
This is an experimental patch, so just FYI for the moment. Known shortcomings include:
o The counters should be atomic_long_t rather than atomic_t.
o No-CBs CPUs can be configured only at boot time.
o Only a modest number of CPUs can be configured as no-CBs CPUs.
Definitely a few tens, perhaps a few hundred, but no way thousands.
o At least one CPU must remain a normal-CBs CPU.
o Not much in the way of energy-efficiency features, though there
are some natural energy savings inherent in the implementation
o The per-no-CBs-CPU kthreads are not subject to RCU priority boosting.
o Care is required when setting the kthreads to RT priority.
Later versions will address some of them, but others are likely to remain.
include/trace/events/rcu.h +1 -0
@@ -549,6 +549,7 @@ TRACE_EVENT(rcu_torture_read,* "EarlyExit": rcu_barrier_callback() piggybacked, thus early exit. * "Inc1": rcu_barrier_callback() piggyback check counter incremented. * "Offline": rcu_barrier_callback() found offline CPU +* "OnlineNoCB": rcu_barrier_callback() found online no-CBs CPU. * "OnlineQ": rcu_barrier_callback() found online CPU with callbacks. * "OnlineNQ": rcu_barrier_callback() found online CPU, no callbacks. * "IRQ": An rcu_barrier_callback() callback posted on remote CPU.
init/Kconfig +19 -0
@@ -598,6 +598,25 @@ config RCU_BOOST_DELAY Accept the default if unsure. +config RCU_NOCB_CPU + bool "Offload RCU callback processing from boot-selected CPUs" + depends on TREE_RCU || TREE_PREEMPT_RCU + default n + help + Use this option to reduce OS jitter for aggressive HPC or + real-time workloads. + + This option offloads callback invocation from the set of CPUs + specified at boot time by the rcu_nocbs parameter. For each + such CPU, a kthread ("rcuoN") will be created to invoke callbacks. + Nothing prevents this kthread from running on of of the specified + CPUs, but (1) the kthreads may be preempted between each callback + and (2) affinity or cgroups can be used to force the kthreads off + of those CPUs if desired. + + Say Y here if you want reduced OS jitter on selected CPUs. + Say N here if you are unsure. + endmenu # "RCU Subsystem" config IKCONFIG
kernel/rcutree.c +51 -12
163 lines not on this page · raw message
kernel/rcutree.h +47 -0
70 lines not on this page · raw message
kernel/rcutree_plugin.h +394 -3
430 lines not on this page · raw message
kernel/rcutree_trace.c +12 -2
@@ -84,6 +84,8 @@ static char convert_kthread_status(unsigned int kthread_status)static void print_one_rcu_data (struct seq_file *m,struct rcu_data *rdp) { +long ql, qll; +if (!rdp->beenonline)return ; seq_printf(m, "%3d%cc=%lu g=%lu pq=%d qp=%d",
@@ -97,8 +99,11 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp) rdp->dynticks->dynticks_nmi_nesting, rdp->dynticks_fqs);seq_printf (m," of=%lu" , rdp->offline_fqs); +rcu_nocb_q_lengths (rdp, &ql, &qll); + qll += rdp->qlen_lazy; + ql += rdp->qlen; seq_printf(m, " ql=%ld/%ld qs=%c%c%c%c", - rdp->qlen_lazy, rdp->qlen, + qll, ql, ".N"[rdp->nxttail[RCU_NEXT_READY_TAIL] != rdp->nxttail[RCU_NEXT_TAIL]], ".R"[rdp->nxttail[RCU_WAIT_TAIL] !=
@@ -147,6 +152,8 @@ static const struct file_operations rcudata_fops = {static void print_one_rcu_data_csv (struct seq_file *m,struct rcu_data *rdp) { +long ql, qll; +if (!rdp->beenonline)return ; seq_printf(m, "%d,%s,%lu,%lu,%d,%d",
@@ -160,7 +167,10 @@ static void print_one_rcu_data_csv(struct seq_file *m, struct rcu_data *rdp) rdp->dynticks->dynticks_nmi_nesting, rdp->dynticks_fqs);seq_printf (m,",%lu" , rdp->offline_fqs); - seq_printf(m, ",%ld,%ld,\"%c%c%c%c\"", rdp->qlen_lazy, rdp->qlen, +rcu_nocb_q_lengths (rdp, &ql, &qll); + qll += rdp->qlen_lazy; + ql += rdp->qlen; + seq_printf(m, ",%ld,%ld,\"%c%c%c%c\"", qll, ql, ".N"[rdp->nxttail[RCU_NEXT_READY_TAIL] != rdp->nxttail[RCU_NEXT_TAIL]], ".R"[rdp->nxttail[RCU_WAIT_TAIL] !=
Quoted 21 lines · Paul E. McKenney
RCU callback execution can add significant OS jitter and also can degrade scheduling latency. This commit therefore adds the ability for selected CPUs ("rcu_nocbs=" boot parameter) to have their callbacks offloaded to kthreads. If the "
rcu_nocb_poll" boot parameter is also specified, these kthreads will do polling, removing the need for the offloaded CPUs to do wakeups. At least one CPU must be doing normal callback processing:
currently CPU 0 cannot be selected as a no-CBs CPU. In addition, attempts to offline the last normal-CBs CPU will fail.This is an experimental patch, so just FYI for the moment. Known shortcomings include:
o The counters should be
atomic_long_trather thanatomic_t.o No-CBs CPUs can be configured only at boot time.
o Only a modest number of CPUs can be configured as no-CBs CPUs.
Definitely a few tens, perhaps a few hundred, but no way thousands.o At least one CPU must remain a normal-CBs CPU.
Quoted 8 lines
o Not much in the way of energy-efficiency features, though there
are some natural energy savings inherent in the implementationo The per-no-CBs-CPU kthreads are not subject to RCU priority boosting.
o Care is required when setting the kthreads to RT priority.
Later versions will address some of them, but others are likely to remain.
My LPC feedback in writing...
So I see RCU as consisting of two parts:
A) Grace period tracking,
2) Running the callbacks.
This series seems to conflate the two, it talks of doing the callbacks elsewhere (kthread), but it also moves the grace period detectoring into the same kthread.
The latter part is what complicates the thing. I'd suggest doing the very simple callbacks only implementation first and leaving the grace period machinery in the tick.
Its typically the callbacks that consume most CPU time, whereas the grace period computations, while tricky and subtle, are relatively cheap.
In particular, it solves the need to wait for grace periods from the kthread (and bounce that no-nocb cpu to make progress), and it makes the atomic list operations stuff a lot easier.
Quoted 45 lines · Peter Zijlstra
Quoted 21 lines · Paul E. McKenney
RCU callback execution can add significant OS jitter and also can degrade scheduling latency. This commit therefore adds the ability for selected CPUs ("rcu_nocbs=" boot parameter) to have their callbacks offloaded to kthreads. If the "
rcu_nocb_poll" boot parameter is also specified, these kthreads will do polling, removing the need for the offloaded CPUs to do wakeups. At least one CPU must be doing normal callback processing:
currently CPU 0 cannot be selected as a no-CBs CPU. In addition, attempts to offline the last normal-CBs CPU will fail.This is an experimental patch, so just FYI for the moment. Known shortcomings include:
o The counters should be
atomic_long_trather thanatomic_t.o No-CBs CPUs can be configured only at boot time.
o Only a modest number of CPUs can be configured as no-CBs CPUs.
Definitely a few tens, perhaps a few hundred, but no way thousands.o At least one CPU must remain a normal-CBs CPU.
Quoted 8 lines
o Not much in the way of energy-efficiency features, though there
are some natural energy savings inherent in the implementationo The per-no-CBs-CPU kthreads are not subject to RCU priority boosting.
o Care is required when setting the kthreads to RT priority.
Later versions will address some of them, but others are likely to remain.
My LPC feedback in writing...
So I see RCU as consisting of two parts:
A) Grace period tracking,
2) Running the callbacks.This series seems to conflate the two, it talks of doing the callbacks elsewhere (kthread), but it also moves the grace period detectoring into the same kthread.
The latter part is what complicates the thing. I'd suggest doing the very simple callbacks only implementation first and leaving the grace period machinery in the tick.
Quoted 7 lines
Its typically the callbacks that consume most CPU time, whereas the grace period computations, while tricky and subtle, are relatively cheap.
In particular, it solves the need to wait for grace periods from the kthread (and bounce that no-nocb cpu to make progress), and it makes the atomic list operations stuff a lot easier.
I was excited by this possibility when you first mentioned it, but the low-OS-jitter fans are going to need the grace-period computation to be offloaded as well. So if I use your (admittedly much simpler) approach, I get to rewrite it when Frederic's adaptive-ticks work goes in. Given that this is probably happening relatively soon, it would be better if I just did the implementation that will be needed long-term, rather than rewriting.
Though I am sure that people will be sad about fewer RCU patches. ;-)
Thanx, Paul
Quoted 3 lines · Paul E. McKenney
I was excited by this possibility when you first mentioned it, but the low-OS-jitter fans are going to need the grace-period computation to be offloaded as well.
Sure, but it seems to me pulling the grace period machinery out is a much harder feat and should be a patch (series) on its own. Also..
Quoted 3 lines
So if I use your (admittedly much simpler)
approach, I get to rewrite it when Frederic's adaptive-ticks work goes in.
I don't see how Frederic's work affects any of this, that would simple put RCU into extended quiescent state (aka. idle) while in userspace. In that state the grace period machinery is stopped all together, so it doesn't matter who would've ran it.
Quoted 5 lines
Given that this is probably happening relatively soon, it would be
better if I just did the implementation that will be needed long-term, rather than rewriting.Though I am sure that people will be sad about fewer RCU patches. ;-)
Always...
Now thinking about this grace machinery stuff a little more, would it be possible to stick the entire state machine in a kthread and replace all current hooks, like the tick and rcu_read_unlock_special with a message passing construct such that they pass their event on to the kthread?
That way you could run the entire state thing from a kthread with random affinity, all 'per-cpu' data would still be fine since only the one kthread will access it, even though locality might suffer somewhat.
This would also not suffer from the having to keep one cpu special and the ugly bouncing etc..
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
I was excited by this possibility when you first mentioned it, but the low-OS-jitter fans are going to need the grace-period computation to be offloaded as well.
Sure, but it seems to me pulling the grace period machinery out is a much harder feat and should be a patch (series) on its own. Also..
Actually, I believe that I can resolve this reasonably simply.
Quoted 8 lines
Quoted 3 lines
So if I use your (admittedly much simpler)
approach, I get to rewrite it when Frederic's adaptive-ticks work goes in.I don't see how Frederic's work affects any of this, that would simple put RCU into extended quiescent state (aka. idle) while in userspace. In that state the grace period machinery is stopped all together, so it doesn't matter who would've ran it.
The key point is "would simple put RCU into extended quiescent state". This can only happen if the CPU has no callbacks. If the CPU does have callbacks, then RCU will need to do some work to advance the callbacks. Advancing the callbacks requires that RCU periodically do work on that CPU, resulting in OS jitter.
Quoted 8 lines
Quoted 5 lines
Given that this is probably happening relatively soon, it would be
better if I just did the implementation that will be needed long-term, rather than rewriting.Though I am sure that people will be sad about fewer RCU patches. ;-)
Always...
Quoted 8 lines
Now thinking about this grace machinery stuff a little more, would it be possible to stick the entire state machine in a kthread and replace all current hooks, like the tick and
rcu_read_unlock_specialwith a message passing construct such that they pass their event on to the kthread?That way you could run the entire state thing from a kthread with random affinity, all 'per-cpu' data would still be fine since only the one kthread will access it, even though locality might suffer somewhat.
Well, the current patch set does move much of the grace-period machinery to a kthread. Much of the remaining work needs to remain on the CPUs (at least those not in an extended quiescent state) in order to keep the overhead of the read-side primitives and scheduler hooks inexpensive.
Quoted 2 lines
This would also not suffer from the having to keep one cpu special and the ugly bouncing etc..
The CPU-0 bounce thing was just a short-term hack to get the prototype working quickly. It absolutely will -not- go mainline. I have not yet worked out the best thing to replace it, but it will be replaced!
Thanx, Paul
Quoted 5 lines · Paul E. McKenney
The key point is "would simple put RCU into extended quiescent state". This can only happen if the CPU has no callbacks. If the CPU does have callbacks, then RCU will need to do some work to advance the callbacks. Advancing the callbacks requires that RCU periodically do work on that CPU, resulting in OS jitter.
But since its then not actually in adaptive-tick mode (the tick is still running) who cares? It will only disable the tick once all preconditions are met, this includes RCU being in extended qs, so until that time...
Quoted 8 lines
Quoted 3 lines
That way you could run the entire state thing from a kthread with random affinity, all 'per-cpu' data would still be fine since only the one kthread will access it, even though locality might suffer somewhat.
Well, the current patch set does move much of the grace-period machinery to a kthread. Much of the remaining work needs to remain on the CPUs (at least those not in an extended quiescent state) in order to keep the overhead of the read-side primitives and scheduler hooks inexpensive.
Ah indeed, what you're saying is that the data required by those hooks needs to be accessed locally in order to avoid proper atomic ops.
So then you do indeed need to break the state machine into two parts, and I guess that's the bit you're struggling with.
Still I would not make this more complex than it needs to be, if the tick is running we can use this to drive the state machine, if its not, we are in extended qs and we don't need to drive the tick.
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Paul E. McKenney
The key point is "would simple put RCU into extended quiescent state". This can only happen if the CPU has no callbacks. If the CPU does have callbacks, then RCU will need to do some work to advance the callbacks. Advancing the callbacks requires that RCU periodically do work on that CPU, resulting in OS jitter.
But since its then not actually in adaptive-tick mode (the tick is still running) who cares? It will only disable the tick once all preconditions are met, this includes RCU being in extended qs, so until that time...
The fact that it is then not actually in adaptive-tick mode is exactly the problem. In other words, if the grace-period processing is offloaded along with the callbacks, then no-CBs CPUs can get into adaptive-tick mode more quickly than CPUs processing their own CBs. Getting these CPUs into adaptive-tick mode more quickly reduces OS jitter, which is one big expected benefit of adaptive-tick mode.
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
That way you could run the entire state thing from a kthread with random affinity, all 'per-cpu' data would still be fine since only the one kthread will access it, even though locality might suffer somewhat.
Well, the current patch set does move much of the grace-period machinery to a kthread. Much of the remaining work needs to remain on the CPUs (at least those not in an extended quiescent state) in order to keep the overhead of the read-side primitives and scheduler hooks inexpensive.
Ah indeed, what you're saying is that the data required by those hooks needs to be accessed locally in order to avoid proper atomic ops.
Yep, that is it!
Quoted 2 lines
So then you do indeed need to break the state machine into two parts, and I guess that's the bit you're struggling with.
Exactly! I should be able to work something out without too much trouble, but it was not going to happen in time for Plumbers, hence the crude prototype.
Quoted 3 lines
Still I would not make this more complex than it needs to be, if the tick is running we can use this to drive the state machine, if its not, we are in extended qs and we don't need to drive the tick.
True, but an important goal of no-CBs CPUs is to spend more time in tickless mode, thus reducing OS jitter.
Thanx, Paul
Quoted 3 lines · Paul E. McKenney
Getting these
CPUs into adaptive-tick mode more quickly reduces OS jitter, which is one big expected benefit of adaptive-tick mode.
I'm not sure I agree with that statement. Its a transition thing and therefore statistically irrelevant on the whole -- assuming we transition rarely.
And for those who cannot deal with the transition effects, a barrier was proposed which would wait until the system hit this state -- although the specific implementation of this isn't clear yet afaik.
I very much think we should focus on getting adaptive tick working as simple as possible and worry about transition effects later, if at all.
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
Getting these
CPUs into adaptive-tick mode more quickly reduces OS jitter, which is one big expected benefit of adaptive-tick mode.I'm not sure I agree with that statement. Its a transition thing and therefore statistically irrelevant on the whole -- assuming we transition rarely.
Quoted 6 lines
And for those who cannot deal with the transition effects, a barrier was proposed which would wait until the system hit this state -- although the specific implementation of this isn't clear yet afaik.
I very much think we should focus on getting adaptive tick working as simple as possible and worry about transition effects later, if at all.
I expect that the RT folks will be interested as well, and they will likely care about the transition effects because they degrade worst-case behavior.
Thanx, Paul