tracepoint: rcuidle: use rcu_is_watching() and tree-rcu
Mathieu Desnoyers
reviewerPATCH 1/1 tracepoint: rcuidle: use rcu_is_watching() and tree-rcu
commit e6753f23d961 ("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.
commit 865e63b04e9b ("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.
It turns out that by using "rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.
I suspect that the original benchmarks justifying the introduction of SRCU to handle rcuidle tracepoints was caused by preempt/irq tracepoints, which are typically invoked from contexts that have RCU watching.
--- include/linux/tracepoint.h | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-)
include/linux/tracepoint.h +10 -15
@@ -165,25 +165,22 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)void *it_func; \void *__data; \ int __maybe_unused __idx = 0; \ +bool __exit_rcu = false; \ \if (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ - \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ - \/* \ -* For rcuidle callers, use srcu since sched-rcu \ -* doesn't work from the idle path. \ +* For rcuidle callers, temporarily enable RCU if \ +* it is not currently watching. \ */ \ -if (rcuidle) { \ - __idx =srcu_read_lock_notrace (&tracepoint_srcu);\ +if (rcuidle && !rcu_is_watching ()) { \rcu_irq_enter_irqson (); \ + __exit_rcu = true; \ } \ \ +preempt_disable_notrace (); \ + \ it_func_ptr =rcu_dereference_raw ((tp)->funcs); \ \if (it_func_ptr) { \
@@ -194,12 +191,10 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) }while ((++it_func_ptr)->func); \ } \ \ -if (rcuidle) { \ -rcu_irq_exit_irqson (); \ -srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ - } \ - \preempt_enable_notrace (); \ + \ +if (__exit_rcu) \ +rcu_irq_exit_irqson (); \ }while (0 ) #ifndef MODULE
Signature · 1 line
2.17.1
Quoted 4 lines · Mathieu Desnoyers
commit
e6753f23d961("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.
Quoted 8 lines
commit
865e63b04e9b("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.It turns out that by using "
rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.
Adding Alexei on CC for his thoughts, given that these were his benchmarks. I believe that he also has additional use cases.
But given the use cases you describe, this seems plausible. This does mean that tracepoints cannot be attached to the CPU-hotplug code that runs on the incoming/outgoing CPU early/late in that process, though that might be OK.
Thanx, Paul
Quoted 73 lines
I suspect that the original benchmarks justifying the introduction of SRCU to handle rcuidle tracepoints was caused by preempt/irq tracepoints, which are typically invoked from contexts that have RCU watching.
Signed-off-by Mathieu Desnoyers <[email protected]>CC Thomas Gleixner <[email protected]>CC Steven Rostedt <[email protected]>CC Joel Fernandes <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Peter Zijlstra <[email protected]>CC Frederic Weisbecker <[email protected]>CC Ingo Molnar <[email protected]>--- include/linux/tracepoint.h | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-)include/linux/tracepoint.h +10
-15@@ -165,25 +165,22 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)void *it_func; \void *__data; \ int __maybe_unused __idx = 0; \ +bool __exit_rcu = false; \ \if (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ - \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ - \/* \ -* For rcuidle callers, use srcu since sched-rcu \ -* doesn't work from the idle path. \ +* For rcuidle callers, temporarily enable RCU if \ +* it is not currently watching. \ */ \ -if (rcuidle) { \ - __idx =srcu_read_lock_notrace (&tracepoint_srcu);\ +if (rcuidle && !rcu_is_watching ()) { \rcu_irq_enter_irqson (); \ + __exit_rcu = true; \ } \ \ +preempt_disable_notrace (); \ + \ it_func_ptr =rcu_dereference_raw ((tp)->funcs); \ \if (it_func_ptr) { \@@ -194,12 +191,10 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) }while ((++it_func_ptr)->func); \ } \ \ -if (rcuidle) { \ -rcu_irq_exit_irqson (); \ -srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ - } \ - \preempt_enable_notrace (); \ + \ +if (__exit_rcu) \ +rcu_irq_exit_irqson (); \ }while (0 ) #ifndef MODULESignature · 1 line
2.17.1
Mathieu Desnoyers
reviewerQuoted 10 lines · ----- On Mar 10, 2020, at 4:53 PM, paulmck [email protected] wrote:
Quoted 9 lines · Mathieu Desnoyers
commit
e6753f23d961("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.commit
865e63b04e9b("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.
Quoted 6 lines
Quoted 3 lines
It turns out that by using "
rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.Adding Alexei on CC for his thoughts, given that these were his benchmarks. I believe that he also has additional use cases.
Good point! Sorry I forgot to add Alexei to my CC list for that patch.
Quoted 4 lines
But given the use cases you describe, this seems plausible. This does mean that tracepoints cannot be attached to the CPU-hotplug code that runs on the incoming/outgoing CPU early/late in that process, though that might be OK.
Do you mean standard tracepoints or rcuidle tracepoints ?
Are there any such tracepoints early/late in the cpu hotplug code today ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 22 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Mar 10, 2020, at 4:53 PM, paulmck [email protected] wrote:
Quoted 9 lines · Mathieu Desnoyers
commit
e6753f23d961("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.commit
865e63b04e9b("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.Quoted 6 lines
Quoted 3 lines
It turns out that by using "
rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.Adding Alexei on CC for his thoughts, given that these were his benchmarks. I believe that he also has additional use cases.
Good point! Sorry I forgot to add Alexei to my CC list for that patch.
Quoted 8 lines
Quoted 4 lines
But given the use cases you describe, this seems plausible. This does mean that tracepoints cannot be attached to the CPU-hotplug code that runs on the incoming/outgoing CPU early/late in that process, though that might be OK.
Do you mean standard tracepoints or rcuidle tracepoints ?
Are there any such tracepoints early/late in the cpu hotplug code today ?
I have no idea. You would know better than I. However, I would expect that the same common-code issue that applies to exception-entry/exit code might also apply to the CPU hotplug code.
Thanx, Paul
Mathieu Desnoyers
reviewerQuoted 28 lines · ----- On Mar 10, 2020, at 7:52 PM, paulmck [email protected] wrote:
Quoted 27 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Mar 10, 2020, at 4:53 PM, paulmck [email protected] wrote:
Quoted 9 lines · Mathieu Desnoyers
commit
e6753f23d961("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.commit
865e63b04e9b("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.Quoted 6 lines
Quoted 3 lines
It turns out that by using "
rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.Adding Alexei on CC for his thoughts, given that these were his benchmarks. I believe that he also has additional use cases.
Good point! Sorry I forgot to add Alexei to my CC list for that patch.
Quoted 4 lines
But given the use cases you describe, this seems plausible. This does mean that tracepoints cannot be attached to the CPU-hotplug code that runs on the incoming/outgoing CPU early/late in that process, though that might be OK.
Quoted 7 lines
Quoted 3 lines
Do you mean standard tracepoints or rcuidle tracepoints ?
Are there any such tracepoints early/late in the cpu hotplug code today ?
I have no idea. You would know better than I. However, I would expect that the same common-code issue that applies to exception-entry/exit code might also apply to the CPU hotplug code.
I would also expect early/late CPU hotplug states to fall into the same category of "low-level entry/exit" code which Thomas would like to hide from instrumentation.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 34 lines · Mathieu Desnoyers
Quoted 32 lines · ----- On Mar 10, 2020, at 7:52 PM, paulmck [email protected] wrote:
Quoted 30 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Mar 10, 2020, at 4:53 PM, paulmck [email protected] wrote:
Quoted 9 lines · Mathieu Desnoyers
commit
e6753f23d961("tracepoint: Make rcuidle tracepoint callers use SRCU") aimed at improving performance of rcuidle tracepoints by using SRCU rather than temporarily enabling tree-rcu every time.commit
865e63b04e9b("tracing: Add back in rcu_irq_enter/exit_irqson() for rcuidle tracepoints") adds back the high-overhead enabling of tree-rcu because perf expects RCU to be watching when called from rcuidle tracepoints.Quoted 6 lines
Quoted 3 lines
It turns out that by using "
rcu_is_watching()" and conditionally calling the high-overhead rcu_irq_enter/exit_irqson(), the original motivation for using SRCU in the first place disappears.Adding Alexei on CC for his thoughts, given that these were his benchmarks. I believe that he also has additional use cases.
Good point! Sorry I forgot to add Alexei to my CC list for that patch.
Quoted 4 lines
But given the use cases you describe, this seems plausible. This does mean that tracepoints cannot be attached to the CPU-hotplug code that runs on the incoming/outgoing CPU early/late in that process, though that might be OK.
Do you mean standard tracepoints or rcuidle tracepoints ?
Are there any such tracepoints early/late in the cpu hotplug code today ?
Quoted 7 lines
Quoted 3 lines
I have no idea. You would know better than I. However, I would expect that the same common-code issue that applies to exception-entry/exit code might also apply to the CPU hotplug code.
I would also expect early/late CPU hotplug states to fall into the same category of "low-level entry/exit" code which Thomas would like to hide from instrumentation.
That decision I must leave to you guys.
Thanx, Paul