dyntick-idle CPU and node's qsmask
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the rnp->qsmask corresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.
rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0
That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
thanks,
- Joel
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
We could, but that would require consulting the rcu_data structure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where each CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the rcu_data structures would be wasted motion.
Now, this does increase overhead on mostly idle systems, but the theory is that mostly idle systems are most able to absorb this extra overhead.
Thoughts?
Thanx, Paul
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where each
When I traced, it appears to me that rcu_data structure of a remote CPU was being consulted anyway by the rcu_sched thread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU's rcu_data happens anyway.
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.
If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then rcu_implicit_dynticks_qs (called from force_qs_rnp) wouldn't be called and so there would no cache misses on rcu_data right?
Quoted 2 lines
Now, this does increase overhead on mostly idle systems, but the theory is that mostly idle systems are most able to absorb this extra overhead.
Yes. Could we use rcuperf to check the impact of such change?
Anyway it was just an idea that popped up when I was going through traces :) Thanks for the discussion and happy to discuss further or try out anything.
- Joel
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where each
Quoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.
Hmmm...
The rcu_gp_init() function does access an rcu_data structure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.
Or are you seeing these cross-CPU rcu_data accesses in rcu_gp_fqs() or functions that it calls? In that case, please see below.
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?
Yes, but assuming that all CPUs report their quiescent states before the first call to rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels, cond_resched() is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels, cond_resched() was never an RCU quiescent state unless it actually scheduled.)
Why wait 100 milliseconds? Because otherwise the increase in cond_resched() overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.
Ah, did you build with NO_HZ_FULL, boot with nohz_full CPUs, and then run CPU-bound usermode workloads on those CPUs? Such CPUs would appear to be idle from an RCU perspective. But these CPUs would never touch their rcu_data structures, so they would likely remain in the RCU grace-period kthread's cache. So this should work well also. Give or take that other work would likely eject them from the cache, but in that case they would be capacity cache misses rather than the aforementioned communications cache misses. Not that this distinction matters to whoever is measuring performance. ;-)
Quoted 4 lines
Quoted 2 lines
Now, this does increase overhead on mostly idle systems, but the theory is that mostly idle systems are most able to absorb this extra overhead.
Yes. Could we use rcuperf to check the impact of such change?
I would be very surprised if the overhead was large enough for rcuperf to be able to see it.
Quoted 2 lines
Anyway it was just an idea that popped up when I was going through traces :) Thanks for the discussion and happy to discuss further or try out anything.
Either way, I do appreciate your going through this. People have found RCU bugs this way, one of which involved RCU uselessly calling a particular function twice in quick succession. ;-)
Thanx, Paul
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.
Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.
Yes, it was rcu_implicit_dynticks_qs called from rcu_gp_fqs.
Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)
Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
Quoted 9 lines
Ah, did you build with
NO_HZ_FULL, boot withnohz_fullCPUs, and then run CPU-bound usermode workloads on those CPUs? Such CPUs would appear to be idle from an RCU perspective. But these CPUs would never touch theirrcu_datastructures, so they would likely remain in the RCU grace-period kthread's cache. So this should work well also. Give or take that other work would likely eject them from the cache, but in that case they would be capacity cache misses rather than the aforementioned communications cache misses. Not that this distinction matters to whoever is measuring performance. ;-)
Ah ok :-) I had booted with !CONFIG_NO_HZ_FULL for this test.
Quoted 6 lines
Quoted 2 lines
Anyway it was just an idea that popped up when I was going through traces :) Thanks for the discussion and happy to discuss further or try out anything.
Either way, I do appreciate your going through this. People have found RCU bugs this way, one of which involved RCU uselessly calling a particular function twice in quick succession. ;-)
Thanks. It is my pleasure and happy to help :) I'll keep digging into it.
- Joel
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.
Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500))
So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out RCU_JIFFIES_FQS_DIV for more info on this.
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqs
You can even set the first one to zero if you want the effect of pre-scanning for idle CPUs. ;-)
The second must be set to one or greater.
Both are capped at one second (HZ).
Quoted 11 lines
Quoted 9 lines
Ah, did you build with
NO_HZ_FULL, boot withnohz_fullCPUs, and then run CPU-bound usermode workloads on those CPUs? Such CPUs would appear to be idle from an RCU perspective. But these CPUs would never touch theirrcu_datastructures, so they would likely remain in the RCU grace-period kthread's cache. So this should work well also. Give or take that other work would likely eject them from the cache, but in that case they would be capacity cache misses rather than the aforementioned communications cache misses. Not that this distinction matters to whoever is measuring performance. ;-)Ah ok :-) I had booted with !CONFIG_NO_HZ_FULL for this test.
Never mind, then. ;-)
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Anyway it was just an idea that popped up when I was going through traces :) Thanks for the discussion and happy to discuss further or try out anything.
Either way, I do appreciate your going through this. People have found RCU bugs this way, one of which involved RCU uselessly calling a particular function twice in quick succession. ;-)
Thanks. It is my pleasure and happy to help :) I'll keep digging into it.
Looking forward to further questions and patches. ;-)
Thanx, Paul
On Sun, Nov 11, 2018 at 10:36:18AM -0800, Paul E. McKenney wrote:
[..]
Quoted 36 lines
Quoted 22 lines
Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))
Quoted 8 lines
So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.
Got it. I agree with that heuristic.
Quoted 6 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqs
Quoted 6 lines
You can even set the first one to zero if you want the effect of pre-scanning for idle CPUs. ;-)
The second must be set to one or greater.
Both are capped at one second (HZ).
Got it. Thanks a lot for the explanations.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Anyway it was just an idea that popped up when I was going through traces :) Thanks for the discussion and happy to discuss further or try out anything.
Either way, I do appreciate your going through this. People have found RCU bugs this way, one of which involved RCU uselessly calling a particular function twice in quick succession. ;-)
Thanks. It is my pleasure and happy to help :) I'll keep digging into it.
Looking forward to further questions and patches. ;-)
Will do! thanks,
- Joel
Quoted 93 lines · Paul E. McKenney
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.
Quoted 5 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqs
Slightly related, I was just going through your patch in the dev branch "doc:
Now jiffies_till_sched_qs solicits from cond_resched()".
If I understand correctly, what you're trying to do is set rcu_data.rcu_urgent_qs if you've not heard from the CPU long enough from rcu_implicit_dynticks_qs.
Then in the other paths, you are reading this value and similuating a dyntick idle transition even though you may not be really going into dyntick-idle. Actually in the scheduler-tick, you are also using it to set NEED_RESCHED appropriately.
Did I get it right so far?
I was thinking if we could simplify rcu_note_context_switch (the parts that call rcu_momentary_dyntick_idle), if we did the following in rcu_implicit_dynticks_qs.
Since we already call rcu_qs in rcu_note_context_switch, that would clear the rdp->cpu_no_qs flag. Then there should be no need to call rcu_momentary_dyntick_idle from rcu_note_context switch.
I think this would simplify cond_resched as well. Could this avoid the need for having an rcu_all_qs at all? Hopefully I didn't some Tasks-RCU corner cases..
Basically for some background, I was thinking can we simplify the code that calls "rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resetting cpu_no_qs).
I should probably start drawing some pictures to make sense of everything, but do let me know if I have a point ;-) Thanks for your time.
- Joel
kernel/rcu/tree.c +1 -1
@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. */ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {trace_rcu_fqs (rcu_state.name, rdp->gp_seq, rdp->cpu,TPS ("dti" ));rcu_gpnum_ovf (rnp, rdp);return 1 ;
Quoted 107 lines · Joel Fernandes
Quoted 93 lines · Paul E. McKenney
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.Quoted 5 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqsSlightly related, I was just going through your patch in the dev branch "doc:
Nowjiffies_till_sched_qssolicits fromcond_resched()".If I understand correctly, what you're trying to do is set rcu_data.rcu_urgent_qs if you've not heard from the CPU long enough from
rcu_implicit_dynticks_qs.
Quoted 6 lines
Then in the other paths, you are reading this value and similuating a dyntick idle transition even though you may not be really going into dyntick-idle. Actually in the scheduler-tick, you are also using it to set
NEED_RESCHEDappropriately.Did I get it right so far?
Partially.
The simulated dyntick-idle transition happens if the grace period extends for even longer, so that ->rcu_need_heavy_qs gets set. Up to that point, all that is asked for is a local-to-the-CPU report of a quiescent state.
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.
But does this also work for the rcu_all_qs() code path?
Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..
There is also the code path from cond_resched() in PREEMPT=n kernels. This needs rcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.
Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).
One complication is that rcu_all_qs() is invoked with interrupts and preemption enabled, while rcu_note_context_switch() is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plus rcu_all_qs() wants to exit immediately if there is nothing to do, while rcu_note_context_switch() must unconditionally do rcu_qs() -- yes, it could check, but that would be redundant with the checks within rcu_qs(). The one function traces and the other one doesn't, but it would be OK if both traced. (I hope, anyway: The cond_resched() performance requirements are surprisingly severe.) Aside from that, the two functions are quite similar.
It would of course be possible to create a common helper function that rcu_all_qs() and rcu_note_context_switch() both became simple wrappers for, but it is not clear that this would actually be shorter or simpler.
Quoted 2 lines
I should probably start drawing some pictures to make sense of everything, but do let me know if I have a point ;-) Thanks for your time.
This stuff is admittedly a bit fiddly. Again, it took some serious work to avoid cond_resched() performance regressions.
kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 12 lines
- Joel
kernel/rcu/tree.c +1
-1@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. */ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {
If I am not too confused, this change could cause trouble for nohz_full CPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.
Or am I missing something here?
Thanx, Paul
Quoted 4 lines
kernel/rcu/tree.c +0
-0trace_rcu_fqs (rcu_state.name, rdp->gp_seq, rdp->cpu,TPS ("dti" ));rcu_gpnum_ovf (rnp, rdp);return 1 ;
Quoted 142 lines · Paul E. McKenney
Quoted 107 lines · Joel Fernandes
Quoted 93 lines · Paul E. McKenney
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.Quoted 5 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqsSlightly related, I was just going through your patch in the dev branch "doc:
Nowjiffies_till_sched_qssolicits fromcond_resched()".If I understand correctly, what you're trying to do is set rcu_data.rcu_urgent_qs if you've not heard from the CPU long enough from
rcu_implicit_dynticks_qs.Quoted 6 lines
Then in the other paths, you are reading this value and similuating a dyntick idle transition even though you may not be really going into dyntick-idle. Actually in the scheduler-tick, you are also using it to set
NEED_RESCHEDappropriately.Did I get it right so far?
Partially.
The simulated dyntick-idle transition happens if the grace period extends for even longer, so that ->rcu_need_heavy_qs gets set. Up to that point, all that is asked for is a local-to-the-CPU report of a quiescent state.
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.But does this also work for the
rcu_all_qs()code path?Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..There is also the code path from
cond_resched()in PREEMPT=n kernels. This needsrcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).
Quoted 10 lines
One complication is that
rcu_all_qs()is invoked with interrupts and preemption enabled, whilercu_note_context_switch()is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plusrcu_all_qs()wants to exit immediately if there is nothing to do, whilercu_note_context_switch()must unconditionally dorcu_qs()-- yes, it could check, but that would be redundant with the checks withinrcu_qs(). The one function traces and the other one doesn't, but it would be OK if both traced. (I hope, anyway: Thecond_resched()performance requirements are surprisingly severe.) Aside from that, the two functions are quite similar.
Plus there are two sets of rcu_qs() and rcu_note_context_switch(), on for PREEMPT=y and the other for PREEMPT=n. And cond_resched() is nothingness for PREEMPT=y. And currently rcu_implicit_dynticks_qs() needs to work with both sets.
Thanx, Paul
Quoted 36 lines
It would of course be possible to create a common helper function that
rcu_all_qs()andrcu_note_context_switch()both became simple wrappers for, but it is not clear that this would actually be shorter or simpler.Quoted 2 lines
I should probably start drawing some pictures to make sense of everything, but do let me know if I have a point ;-) Thanks for your time.
This stuff is admittedly a bit fiddly. Again, it took some serious work to avoid
cond_resched()performance regressions.kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 12 lines
- Joel
kernel/rcu/tree.c +1
-1@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. */ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {If I am not too confused, this change could cause trouble for
nohz_fullCPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.Or am I missing something here?
Thanx, Paul
Quoted 4 lines
kernel/rcu/tree.c +0
-0trace_rcu_fqs (rcu_state.name, rdp->gp_seq, rdp->cpu,TPS ("dti" ));rcu_gpnum_ovf (rnp, rdp);return 1 ;
Quoted 113 lines · Paul E. McKenney
Quoted 112 lines · Joel Fernandes
Quoted 93 lines · Paul E. McKenney
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.Quoted 5 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqsSlightly related, I was just going through your patch in the dev branch "doc:
Nowjiffies_till_sched_qssolicits fromcond_resched()".If I understand correctly, what you're trying to do is set rcu_data.rcu_urgent_qs if you've not heard from the CPU long enough from
rcu_implicit_dynticks_qs.Then in the other paths, you are reading this value and similuating a dyntick idle transition even though you may not be really going into dyntick-idle. Actually in the scheduler-tick, you are also using it to set
NEED_RESCHEDappropriately.
Quoted 7 lines
Quoted 1 line
Did I get it right so far?
Partially.
The simulated dyntick-idle transition happens if the grace period extends for even longer, so that ->rcu_need_heavy_qs gets set. Up to that point, all that is asked for is a local-to-the-CPU report of a quiescent state.
Right, that's true. My feeling was the whole "fake a dyntick idle transition" seems to me a bit of a hack and I was thinking if not depending on that would simplify the code so we don't need the rcu_momentary_dyntick_idle.
Quoted 9 lines
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.But does this also work for the
rcu_all_qs()code path?
Could we not do something like this in rcu_all_qs? as some over-simplified pseudo code:
rcu_all_qs() {
if (!urgent_qs || !heavy_qs)
return;
rcu_qs(); // This clears the rdp->cpu_no_qs flags which we can monitor in
// the diff in my last email (from rcu_implicit_dynticks_qs) }
Quoted 11 lines
Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..There is also the code path from
cond_resched()in PREEMPT=n kernels. This needsrcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).
Quoted 6 lines
One complication is that
rcu_all_qs()is invoked with interrupts and preemption enabled, whilercu_note_context_switch()is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plusrcu_all_qs()wants to exit immediately if there is nothing to do, whilercu_note_context_switch()must unconditionally dorcu_qs()-- yes, it could check, but that would be redundant with the checks
This immediate exit is taken care off in the above psuedo code, would that help the cond_resched performance?
Quoted 7 lines
Quoted 7 lines
kernel/rcu/tree.c +0
-0@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period.
Quoted 8 lines
kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 3 lines
kernel/rcu/tree.c +1
-1*/ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {If I am not too confused, this change could cause trouble for
nohz_fullCPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.
Would that still be a problem if rcu_all_qs called rcu_qs? Also the above diff is an OR condition so it is more relaxed than before.
Assuming the NOHZ_FULL CPUs call cond_resched during their looping, that would trigger the rcu_all_qs -> rcu_qs path which would clear cpu_no_qs flag for that CPU right? This would result in the above diff causing a return of 1 for that CPU (from rcu_implicit_dynticks_qs).
Quoted 1 line
Or am I missing something here?
I think I might be the one missing something but I'm glad we are almost on the same page ;-)
thanks,
- Joel
Quoted 118 lines · Joel Fernandes
Quoted 117 lines · Paul E. McKenney
Quoted 107 lines · Joel Fernandes
Quoted 93 lines · Paul E. McKenney
Quoted 69 lines · Joel Fernandes
Quoted 35 lines · Paul E. McKenney
Quoted 28 lines · Joel Fernandes
Quoted 22 lines · Paul E. McKenney
Quoted 16 lines · Joel Fernandes
Hi Paul and everyone,
I was tracing/studying the RCU code today in paul/dev branch and noticed that for dyntick-idle CPUs, the RCU GP thread is clearing the
rnp->qsmaskcorresponding to the leaf node for the idle CPU, and reporting a QS on their behalf.rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 792 0 dti rcu_sched-10 [003] 40.008039: rcu_fqs: rcu_sched 801 2 dti rcu_sched-10 [003] 40.008041: rcu_quiescent_state_report: rcu_sched 805 5>0 0 0 3 0That's all good but I was wondering if we can do better for the idle CPUs if we can some how not set the qsmask of the node in the first place. Then no reporting would be needed of quiescent state is needed for idle CPUs right? And we would also not need to acquire the rnp lock I think.
Quoted 4 lines
At least for a single node tree RCU system, it seems that would avoid needing to acquire the lock without complications. Anyway let me know your thoughts and happy to discuss this at the hallways of the LPC as well for folks attending :)
Quoted 4 lines
We could, but that would require consulting the
rcu_datastructure for each CPU while initializing the grace period, thus increasing the number of cache misses during grace-period initialization and also shortly after for any non-idle CPUs. This seems backwards on busy systems where eachQuoted 5 lines
When I traced, it appears to me that
rcu_datastructure of a remote CPU was being consulted anyway by thercu_schedthread. So it seems like such cache miss would happen anyway whether it is during grace-period initialization or during the fqs stage? I guess I'm trying to say, the consultation of remote CPU'srcu_datahappens anyway.Quoted 8 lines
Hmmm...
The
rcu_gp_init()function does access anrcu_datastructure, but it is that of the current CPU, so shouldn't involve a communications cache miss, at least not in the common case.Or are you seeing these cross-CPU
rcu_dataaccesses inrcu_gp_fqs()or functions that it calls? In that case, please see below.Yes, it was
rcu_implicit_dynticks_qscalled fromrcu_gp_fqs.Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
CPU will with high probability report its own quiescent state before three jiffies pass, in which case the cache misses on the
rcu_datastructures would be wasted motion.If all the CPUs are busy and reporting their QS themselves, then I think the qsmask is likely 0 so then
rcu_implicit_dynticks_qs(called fromforce_qs_rnp) wouldn't be called and so there would no cache misses onrcu_dataright?Yes, but assuming that all CPUs report their quiescent states before the first call to
rcu_gp_fqs(). One exception is when some CPU is looping in the kernel for many milliseconds without passing through a quiescent state. This is because for recent kernels,cond_resched()is not a quiescent state until the grace period is something like 100 milliseconds old. (For older kernels,cond_resched()was never an RCU quiescent state unless it actually scheduled.)Quoted 4 lines
Why wait 100 milliseconds? Because otherwise the increase in
cond_resched()overhead shows up all too well, causing 0day test robot to complain bitterly. Besides, I would expect that in the common case, CPUs would be executing usermode code.Quoted 9 lines
Makes sense. I was also wondering about this other thing you mentioned about waiting for 3 jiffies before reporting the idle CPU's quiescent state. Does that mean that even if a single CPU is dyntick-idle for a long period of time, then the minimum grace period duration would be atleast 3 jiffies? In our mobile embedded devices, jiffies is set to 3.33ms (HZ=300) to keep power consumption low. Not that I'm saying its an issue or anything (since IIUC if someone wants shorter grace periods, they should just use expedited GPs), but it sounds like it would be shorter GP if we just set the qsmask early on some how and we can manage the overhead of doing so.
First, there is some autotuning of the delay based on HZ:
#define
RCU_JIFFIES_TILL_FORCE_QS(1 + (HZ > 250) + (HZ > 500))So at HZ=300, you should be seeing a two-jiffy delay rather than the usual HZ=1000 three-jiffy delay. Of course, this means that the delay is 6.67ms rather than the usual 3ms, but the theory is that lower HZ rates often mean slower instruction execution and thus a desire for lower RCU overhead. There is further autotuning based on number of CPUs, but this does not kick in until you have 256 CPUs on your system, and I bet that smartphones aren't there yet. Nevertheless, check out
RCU_JIFFIES_FQS_DIVfor more info on this.Quoted 5 lines
But you can always override this autotuning using the following kernel boot paramters:
rcutree.jiffies_till_first_fqs
rcutree.jiffies_till_next_fqsSlightly related, I was just going through your patch in the dev branch "doc:
Nowjiffies_till_sched_qssolicits fromcond_resched()".If I understand correctly, what you're trying to do is set rcu_data.rcu_urgent_qs if you've not heard from the CPU long enough from
rcu_implicit_dynticks_qs.Quoted 6 lines
Then in the other paths, you are reading this value and similuating a dyntick idle transition even though you may not be really going into dyntick-idle. Actually in the scheduler-tick, you are also using it to set
NEED_RESCHEDappropriately.Did I get it right so far?
Partially.
Quoted 7 lines
Quoted 3 lines
The simulated dyntick-idle transition happens if the grace period extends for even longer, so that ->rcu_need_heavy_qs gets set. Up to that point, all that is asked for is a local-to-the-CPU report of a quiescent state.
Right, that's true. My feeling was the whole "fake a dyntick idle transition" seems to me a bit of a hack and I was thinking if not depending on that would simplify the code so we don't need the
rcu_momentary_dyntick_idle.
It is there in case scheduling-clock interrupts are for whatever reason not happening on that CPU.
Quoted 13 lines
Quoted 9 lines
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.But does this also work for the
rcu_all_qs()code path?Could we not do something like this in
rcu_all_qs? as some over-simplified pseudo code:
Quoted 7 lines
rcu_all_qs(){
if (!urgent_qs || !heavy_qs)return;
rcu_qs(); // This clears therdp->cpu_no_qsflags which we can monitor in// the diff in my last email (from rcu_implicit_dynticks_qs) }
Except that rcu_qs() doesn't necessarily report the quiescent state to the RCU core. Keeping down context-switch overhead and all that.
Quoted 18 lines
Quoted 11 lines
Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..There is also the code path from
cond_resched()in PREEMPT=n kernels. This needsrcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).Quoted 6 lines
One complication is that
rcu_all_qs()is invoked with interrupts and preemption enabled, whilercu_note_context_switch()is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plusrcu_all_qs()wants to exit immediately if there is nothing to do, whilercu_note_context_switch()must unconditionally dorcu_qs()-- yes, it could check, but that would be redundant with the checks
Quoted 2 lines
This immediate exit is taken care off in the above psuedo code, would that help the
cond_reschedperformance?
It look like you are cautiously edging towards the two wrapper functions calling common code, relying on inlining and simplification. Why not just try doing it? ;-)
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
kernel/rcu/tree.c +0
-0@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. Quoted 8 lines
kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 3 lines
kernel/rcu/tree.c +1
-1*/ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {If I am not too confused, this change could cause trouble for
nohz_fullCPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.
Quoted 2 lines
Would that still be a problem if
rcu_all_qscalledrcu_qs? Also the above diff is an OR condition so it is more relaxed than before.
Yes, because rcu_qs() is only guaranteed to capture the quiescent state on the current CPU, not necessarily report it to the RCU core.
Quoted 4 lines
Assuming the
NOHZ_FULLCPUs callcond_reschedduring their looping, that would trigger thercu_all_qs->rcu_qspath which would clearcpu_no_qsflag for that CPU right? This would result in the above diff causing a return of 1 for that CPU (fromrcu_implicit_dynticks_qs).
One problem is that cond_resched() is a complete no-op in PREEMPT=y kernels.
Quoted 4 lines
Quoted 1 line
Or am I missing something here?
I think I might be the one missing something but I'm glad we are almost on the same page ;-)
There are a lot of moving parts in this area, to be sure.
Thanx, Paul
On Tue, Nov 20, 2018 at 06:41:07PM -0800, Paul E. McKenney wrote:
[...]
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.But does this also work for the
rcu_all_qs()code path?Could we not do something like this in
rcu_all_qs? as some over-simplified pseudo code:
rcu_all_qs(){
if (!urgent_qs || !heavy_qs)return;
Quoted 6 lines
Quoted 3 lines
rcu_qs(); // This clears therdp->cpu_no_qsflags which we can monitor in// the diff in my last email (from rcu_implicit_dynticks_qs) }Except that
rcu_qs()doesn't necessarily report the quiescent state to the RCU core. Keeping down context-switch overhead and all that.
Sure yeah, but I think the QS will be indirectly anyway by the force_qs_rnp() path if we detect that rcu_qs() happened on the CPU?
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..There is also the code path from
cond_resched()in PREEMPT=n kernels. This needsrcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).One complication is that
rcu_all_qs()is invoked with interrupts and preemption enabled, whilercu_note_context_switch()is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plusrcu_all_qs()wants to exit immediately if there is nothing to do, whilercu_note_context_switch()must unconditionally dorcu_qs()-- yes, it could check, but that would be redundant with the checks
Quoted 6 lines
Quoted 2 lines
This immediate exit is taken care off in the above psuedo code, would that help the
cond_reschedperformance?It look like you are cautiously edging towards the two wrapper functions calling common code, relying on inlining and simplification. Why not just try doing it? ;-)
Sure yeah. I was more thinking of the ambitious goal of getting rid of the complexity and exploring the general design idea, than containing/managing the complexity with reducing code duplication. :D
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 10 lines
kernel/rcu/tree.c +1
-1@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. */ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {If I am not too confused, this change could cause trouble for
nohz_fullCPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.
Quoted 5 lines
Quoted 2 lines
Would that still be a problem if
rcu_all_qscalledrcu_qs? Also the above diff is an OR condition so it is more relaxed than before.Yes, because
rcu_qs()is only guaranteed to capture the quiescent state on the current CPU, not necessarily report it to the RCU core.
The reporting to the core is necessary to call rcu_report_qs_rnp so that the QS information is propogating up the tree, right?
Wouldn't that reporting be done anyway by:
force_qs_rnp
-> rcu_implicit_dynticks_qs (which returns 1 because rdp->cpu_no_qs.b.norm
was cleared by rcu_qs() and we detect that with help of above diff)
-> rcu_report_qs_rnp is called with mask bit set for corresponding CPU that
has the !rdp->cpu_no_qs.b.norm
I think that's what I am missing - that why wouldn't the above scheme work. The only difference is reporting to the RCU core might invoke pending callbacks but I'm not sure if that matters for this. I'll these changes, and try tracing it out and study it more. thanks for the patience,
- Joel
Quoted 23 lines · Joel Fernandes
On Tue, Nov 20, 2018 at 06:41:07PM -0800, Paul E. McKenney wrote:
[...]Quoted 21 lines
Quoted 20 lines
Quoted 9 lines
Quoted 7 lines
I was thinking if we could simplify
rcu_note_context_switch(the parts that callrcu_momentary_dyntick_idle), if we did the following inrcu_implicit_dynticks_qs.Since we already call
rcu_qsinrcu_note_context_switch, that would clear therdp->cpu_no_qsflag. Then there should be no need to callrcu_momentary_dyntick_idlefromrcu_note_contextswitch.But does this also work for the
rcu_all_qs()code path?Could we not do something like this in
rcu_all_qs? as some over-simplified pseudo code:
rcu_all_qs(){
if (!urgent_qs || !heavy_qs)return;
rcu_qs(); // This clears therdp->cpu_no_qsflags which we can monitor in// the diff in my last email (from rcu_implicit_dynticks_qs) }
Quoted 5 lines
Quoted 2 lines
Except that
rcu_qs()doesn't necessarily report the quiescent state to the RCU core. Keeping down context-switch overhead and all that.Sure yeah, but I think the QS will be indirectly anyway by the
force_qs_rnp()path if we detect thatrcu_qs()happened on the CPU?
The force_qs_rnp() path won't see anything that has not already been reported to the RCU core.
Quoted 54 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 2 lines
I think this would simplify
cond_reschedas well. Could this avoid the need for having anrcu_all_qsat all? Hopefully I didn't some Tasks-RCU corner cases..There is also the code path from
cond_resched()in PREEMPT=n kernels. This needsrcu_all_qs(). Though it is quite possible that some additional code collapsing is possible.Quoted 3 lines
Basically for some background, I was thinking can we simplify the code that calls "
rcu_momentary_dyntick_idle" since we already register a qs in other ways (like by resettingcpu_no_qs).One complication is that
rcu_all_qs()is invoked with interrupts and preemption enabled, whilercu_note_context_switch()is invoked with interrupts disabled. Also, as you say, Tasks RCU. Plusrcu_all_qs()wants to exit immediately if there is nothing to do, whilercu_note_context_switch()must unconditionally dorcu_qs()-- yes, it could check, but that would be redundant with the checksQuoted 6 lines
Quoted 2 lines
This immediate exit is taken care off in the above psuedo code, would that help the
cond_reschedperformance?It look like you are cautiously edging towards the two wrapper functions calling common code, relying on inlining and simplification. Why not just try doing it? ;-)
Sure yeah. I was more thinking of the ambitious goal of getting rid of the complexity and exploring the general design idea, than containing/managing the complexity with reducing code duplication. :D
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
kernel/rcu/tree.c static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · line 1066
Quoted 10 lines
kernel/rcu/tree.c +1
-1@@ -1063,7 +1063,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)* read-side critical section that started before the beginning * of the current RCU grace period. */ -if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap)) { +if (rcu_dynticks_in_eqs_since (rdp, rdp->dynticks_snap) || !rdp->cpu_no_qs.b.norm) {If I am not too confused, this change could cause trouble for
nohz_fullCPUs looping in the kernel. Such CPUs don't necessarily take scheduler-clock interrupts, last I checked, and this could prevent the CPU from reporting its quiescent state to core RCU.Quoted 5 lines
Quoted 2 lines
Would that still be a problem if
rcu_all_qscalledrcu_qs? Also the above diff is an OR condition so it is more relaxed than before.Yes, because
rcu_qs()is only guaranteed to capture the quiescent state on the current CPU, not necessarily report it to the RCU core.The reporting to the core is necessary to call
rcu_report_qs_rnpso that the QS information is propogating up the tree, right?
Quoted 6 lines
Wouldn't that reporting be done anyway by:
force_qs_rnp
->rcu_implicit_dynticks_qs(which returns 1 becauserdp->cpu_no_qs.b.norm
was cleared byrcu_qs()and we detect that with help of above diff)
Ah. It is not safe to sample rdp->cpu_no_qs.b.norm off-CPU, and that is what your patch would do. This is intentional -- if it were safe to sample off-CPU, then it would be more expensive to read/update on-CPU.
Quoted 8 lines
->
rcu_report_qs_rnpis called with mask bit set for corresponding CPU that
has the !rdp->cpu_no_qs.b.normI think that's what I am missing - that why wouldn't the above scheme work. The only difference is reporting to the RCU core might invoke pending callbacks but I'm not sure if that matters for this. I'll these changes, and try tracing it out and study it more. thanks for the patience,
There are a lot of moving parts and you have not yet gotten to all of them. I suggest next taking a look at the relationship between rcu_check_callbacks() and rcu_process_callbacks(), including the open_softirq(). These have old names -- they handle the interface between the CPU and RCU code, among other things. Including invoking callbacks, but only for some configurations. :-/
Thanx, Paul