RCU dyntick nesting counter cleanups
Subsystems (3)
v2 cover RCU dyntick nesting counter cleanups
These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.
It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.
rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.
v1->v2:
- Rebase on v5.6-rc6
Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter
.../Data-Structures/Data-Structures.rst | 31 +-- Documentation/RCU/stallwarn.txt | 6 +- include/linux/rcutiny.h | 3 - include/trace/events/rcu.h | 29 +-- kernel/rcu/rcu.h | 4 - kernel/rcu/tree.c | 188 +++++++----------- kernel/rcu/tree.h | 4 +- kernel/rcu/tree_stall.h | 4 +- 8 files changed, 105 insertions(+), 164 deletions(-)
--
2.26.0.rc2.310.g2932bb562d-goog
PATCH v2 1/4 Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
This code is unused and can be removed now. Revert was straightforward.
Tested with rcutorture on all TREE configurations.
--- include/linux/rcutiny.h | 3 -- kernel/rcu/tree.c | 82 ++++++++++------------------------------- 2 files changed, 19 insertions(+), 66 deletions(-)
include/linux/rcutiny.h +0 -3
@@ -14,9 +14,6 @@#include <asm/param.h> /* for HZ */ -/* Never flag non-existent other CPUs! */ -static inline bool rcu_eqs_special_set (int cpu) {return false; } -static inline unsigned long get_state_synchronize_rcu (void ) {return 0 ;
kernel/rcu/tree.c +19 -63
152 lines not on this page · raw message
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
PATCH v2 2/4 rcu/tree: Add better tracing for dyntick-idle
In reply to Joel Fernandes (Google)The dyntick-idle traces are a bit confusing. This patch makes it simpler and adds some missing cases such as EQS-enter due to user vs idle mode.
Following are the changes:
(1) Add a new context field to trace_rcu_dyntick tracepoint. This context field can be "USER", "IDLE" or "IRQ".
(2) Remove the "++=" and "--=" strings and replace them with "StillNonIdle". This is much easier on the eyes, and the -- and ++ are easily apparent in the dynticks_nesting counters we are printing anyway.
--- include/trace/events/rcu.h | 29 ++++++++++++++++------------- kernel/rcu/tree.c | 20 +++++++++++++------- 2 files changed, 29 insertions(+), 20 deletions(-)
include/trace/events/rcu.h +16 -13
@@ -435,26 +435,28 @@ TRACE_EVENT_RCU(rcu_fqs,#endif /* #if defined(CONFIG_TREE_RCU) */ /* -* Tracepoint for dyntick-idle entry/exit events. These take a string -* as argument: "Start" for entering dyntick-idle mode, "Startirq" for -* entering it from irq/NMI, "End" for leaving it, "Endirq" for leaving it -* to irq/NMI, "--=" for events moving towards idle, and "++=" for events -* moving away from idle. +* Tracepoint for dyntick-idle entry/exit events. These take 2 strings +* as argument: +* polarilty: "Start", "End", "StillIdle" for entering, exiting or still being +* in dyntick-idle mode. +* context: "USER" or "KERNEL" or "IRQ". +* NMIs nested in IRQs are inferred with dynticks_nesting > 1 in IRQ context. * * These events also take a pair of numbers, which indicate the nesting * depth before and after the event of interest, and a third number that is -* the ->dynticks counter. Note that task-related and interrupt-related -* events use two separate counters, and that the "++=" and "--=" events -* for irq/NMI will change the counter by two, otherwise by one. +* the ->dynticks counter. During NMI nesting within IRQs, the dynticks_nesting +* counter changes by two, otherwise one. */ TRACE_EVENT_RCU(rcu_dyntick, - TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks), + TP_PROTO(const char *polarity, const char *context, long oldnesting, + long newnesting, int dynticks), - TP_ARGS(polarity, oldnesting, newnesting, dynticks), + TP_ARGS(polarity, context, oldnesting, newnesting, dynticks), TP_STRUCT__entry( __field(const char *, polarity) + __field(const char *, context) __field(long, oldnesting) __field(long, newnesting) __field(int, dynticks)
@@ -462,14 +464,15 @@ TRACE_EVENT_RCU(rcu_dyntick,TP_fast_assign ( __entry->polarity = polarity; + __entry->context = context; __entry->oldnesting = oldnesting; __entry->newnesting = newnesting; __entry->dynticks = dynticks; ), -TP_printk ("%s %lx %lx %#3x" , __entry->polarity, - __entry->oldnesting, __entry->newnesting, - __entry->dynticks &0xfff ) +TP_printk ("%s %s %lx %lx %#3x" , __entry->polarity, + __entry->context, __entry->oldnesting, __entry->newnesting, + __entry->dynticks &0xfff ) );/*
kernel/rcu/tree.c +13 -7
@@ -523,7 +523,8 @@ static void rcu_eqs_enter(bool user) }lockdep_assert_irqs_disabled (); -trace_rcu_dyntick (TPS ("Start" ), rdp->dynticks_nesting,0 ,atomic_read (&rdp->dynticks)); +trace_rcu_dyntick (TPS ("Start" ), (user ?TPS ("USER" ) :TPS ("IDLE" )), + rdp->dynticks_nesting,0 ,atomic_read (&rdp->dynticks));WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && !user && !is_idle_task (current)); rdp =this_cpu_ptr (&rcu_data);do_nocb_deferred_wakeup (rdp);
@@ -596,15 +597,17 @@ static __always_inline void rcu_nmi_exit_common(bool irq)* leave it in non-RCU-idle state. */ if (rdp->dynticks_nmi_nesting !=1 ) { -trace_rcu_dyntick (TPS ("--=" ), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting -2 , -atomic_read (&rdp->dynticks)); +trace_rcu_dyntick (TPS ("StillNonIdle" ),TPS ("IRQ" ), + rdp->dynticks_nmi_nesting, + rdp->dynticks_nmi_nesting -2 ,atomic_read (&rdp->dynticks));WRITE_ONCE (rdp->dynticks_nmi_nesting,/* No store tearing. */ rdp->dynticks_nmi_nesting -2 );return ; }/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ -trace_rcu_dyntick (TPS ("Startirq" ), rdp->dynticks_nmi_nesting,0 ,atomic_read (&rdp->dynticks)); +trace_rcu_dyntick (TPS ("Start" ),TPS ("IRQ" ), rdp->dynticks_nmi_nesting, +0 ,atomic_read (&rdp->dynticks));WRITE_ONCE (rdp->dynticks_nmi_nesting,0 );/* Avoid store tearing. */ if (irq)
@@ -691,7 +694,8 @@ static void rcu_eqs_exit(bool user)rcu_dynticks_task_exit ();rcu_dynticks_eqs_exit ();rcu_cleanup_after_idle (); -trace_rcu_dyntick (TPS ("End" ), rdp->dynticks_nesting,1 ,atomic_read (&rdp->dynticks)); +trace_rcu_dyntick (TPS ("End" ), (user ?TPS ("USER" ) :TPS ("IDLE" )), + rdp->dynticks_nesting,1 ,atomic_read (&rdp->dynticks));WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && !user && !is_idle_task (current));WRITE_ONCE (rdp->dynticks_nesting,1 );WARN_ON_ONCE (rdp->dynticks_nmi_nesting);
@@ -783,9 +787,11 @@ static __always_inline void rcu_nmi_enter_common(bool irq) }raw_spin_unlock_rcu_node (rdp->mynode); } -trace_rcu_dyntick (incby ==1 ?TPS ("Endirq" ) :TPS ("++=" ), - rdp->dynticks_nmi_nesting, + +trace_rcu_dyntick (incby ==1 ?TPS ("End" ) :TPS ("StillNonIdle" ), +TPS ("IRQ" ), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting + incby,atomic_read (&rdp->dynticks)); +WRITE_ONCE (rdp->dynticks_nmi_nesting,/* Prevent store tearing. */ rdp->dynticks_nmi_nesting + incby);barrier ();
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
PATCH v2 3/4 rcu/tree: Clean up dynticks counter usage
In reply to Joel Fernandes (Google)The dynticks counter are confusing due to crowbar writes of DYNTICK_IRQ_NONIDLE whose purpose is to detect half-interrupts (i.e. we see rcu_irq_enter() but not rcu_irq_exit() due to a usermode upcall) and if so then do a reset of the dyntick_nmi_nesting counters. This patch tries to get rid of DYNTICK_IRQ_NONIDLE while still keeping the code working, fully functional, and less confusing. The confusion recently has even led to patches forgetting that DYNTICK_IRQ_NONIDLE was written to which wasted lots of time.
The patch has the following changes:
(1) Use dynticks_nesting instead of dynticks_nmi_nesting for determining outer most "EQS exit". This is needed to detect in rcu_nmi_enter_common() if we have already EQS-exited, such as because of a syscall. Currently we rely on a forced write of DYNTICK_IRQ_NONIDLE from rcu_eqs_exit() for this purpose. This is one purpose of the DYNTICK_IRQ_NONIDLE write (other than detecting half-interrupts). However, we do not need to do that. dyntick_nesting already tells us that we have EQS-exited so just use that thus removing the dependence of dynticks_nmi_nesting for this purpose.
(2) Keep dynticks_nmi_nesting around because:
(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.
(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.
(3) Since we got rid of DYNTICK_IRQ_NONIDLE, we also do cheaper comparisons with zero instead for the code that keeps the tick on in rcu_nmi_enter_common().
In the next patch, both of the concerns of (2) will be addressed and then we can get rid of dynticks_nmi_nesting, however one step at a time.
--- kernel/rcu/rcu.h | 4 ---- kernel/rcu/tree.c | 60 +++++++++++++++++++++++++++-------------------- 2 files changed, 35 insertions(+), 29 deletions(-)
kernel/rcu/rcu.h +0 -4
@@ -12,10 +12,6 @@#include <trace/events/rcu.h> -/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ -#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) - -/* * Grace-period counter management. */
kernel/rcu/tree.c +35 -25
128 lines not on this page · raw message
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
PATCH v2 4/4 rcu/tree: Remove dynticks_nmi_nesting counter
In reply to Joel Fernandes (Google)The dynticks_nmi_nesting counter serves 3 purposes:
(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.
(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.
(c) When a quiescent state report is needed from a nohz_full CPU. The nesting counter detects we are a first level interrupt.
For (a), we can just use dyntick_nesting == 1 to determine this. Only the outermost interrupt that interrupted an RCU-idle state can set it to 1.
For (b), this warning condition has not occurred for several kernel releases. But we still keep the warning but change it to use in_irq() instead of the nesting counter. In a later year, we can remove the warning.
For (c), the nest check is not really necessary since forced_tick would have been set to true in the outermost interrupt, so the nested/NMI interrupts will check forced_tick anyway, and bail.
--- .../Data-Structures/Data-Structures.rst | 31 ++++------ Documentation/RCU/stallwarn.txt | 6 +- kernel/rcu/tree.c | 56 +++++++------------ kernel/rcu/tree.h | 4 +- kernel/rcu/tree_stall.h | 4 +- 5 files changed, 37 insertions(+), 64 deletions(-)
Documentation/RCU/Design/Data-Structures/Data-Structures.rst +10 -21
@@ -936,10 +936,9 @@ This portion of the rcu_data structure is declared as follows: :: 1 long dynticks_nesting; - 2 long dynticks_nmi_nesting; - 3 atomic_t dynticks; - 4 bool rcu_need_heavy_qs; - 5 bool rcu_urgent_qs; + 2 atomic_t dynticks; + 3 bool rcu_need_heavy_qs; + 4 bool rcu_urgent_qs; These fields in the rcu_data structure maintain the per-CPU dyntick-idle state for the corresponding CPU. The fields may be accessed only from
@@ -948,26 +947,14 @@ the corresponding CPU (and from tracing) unless otherwise stated. The ``->dynticks_nesting`` field counts the nesting depth of process execution, so that in normal circumstances this counter has value zero or one. NMIs, irqs, and tracers are counted by the -``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes +``->dynticks_nesting`` field as well. Because NMIs cannot be masked, changes to this variable have to be undertaken carefully using an algorithm provided by Andy Lutomirski. The initial transition from idle adds one, and nested transitions add two, so that a nesting level of five is -represented by a ``->dynticks_nmi_nesting`` value of nine. This counter +represented by a ``->dynticks_nesting`` value of nine. This counter can therefore be thought of as counting the number of reasons why this -CPU cannot be permitted to enter dyntick-idle mode, aside from -process-level transitions. - -However, it turns out that when running in non-idle kernel context, the -Linux kernel is fully capable of entering interrupt handlers that never -exit and perhaps also vice versa. Therefore, whenever the -``->dynticks_nesting`` field is incremented up from zero, the -``->dynticks_nmi_nesting`` field is set to a large positive number, and -whenever the ``->dynticks_nesting`` field is decremented down to zero, -the the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that -the number of misnested interrupts is not sufficient to overflow the -counter, this approach corrects the ``->dynticks_nmi_nesting`` field -every time the corresponding CPU enters the idle loop from process -context. +CPU cannot be permitted to enter dyntick-idle mode. It counts both the +process-level and interrupt transitions. The ``->dynticks`` field counts the corresponding CPU's transitions to and from either dyntick-idle or user mode, so that this counter has an
@@ -1000,7 +987,9 @@ code. +-----------------------------------------------------------------------+ | Because this would fail in the presence of interrupts whose handlers | | never return and of handlers that manage to return from a made-up | -| interrupt. | +| interrupt. NOTE: The counters have now been combined however | +| a temporary warning has been left to make sure this condition never | +| occurs. | +-----------------------------------------------------------------------+ Additional fields are present for some special-purpose builds, and are
Documentation/RCU/stallwarn.txt +3 -3
@@ -173,8 +173,8 @@ For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling, it will print a message similar to the following: INFO: rcu_sched detected stalls on CPUs/tasks: - 2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0 - 16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0 + 2-...: (3 GPs behind) idle=06c/0 softirq=1453/1455 fqs=0 + 16-...: (0 ticks this GP) idle=81c/0 softirq=764/764 fqs=0 (detected by 32, t=2603 jiffies, g=7075, q=625) This message indicates that CPU 32 detected that CPUs 2 and 16 were both
@@ -225,7 +225,7 @@ an estimate of the total number of RCU callbacks queued across all CPUs In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed for each CPU: - 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1 + 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1 The "last_accelerate:" prints the low-order 16 bits (in hex) of the jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
kernel/rcu/tree.c +20 -36
122 lines not on this page · raw message
kernel/rcu/tree.h +2 -2
@@ -175,8 +175,8 @@ struct rcu_data {/* 3) dynticks interface. */ int dynticks_snap;/* Per-GP tracking for dynticks. */ -long dynticks_nesting;/* Track process nesting level. */ -long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ +long dynticks_nesting;/* Track dyntick (non-IDLE) nesting +* level for kernel entry and interrupt. */ atomic_t dynticks;/* Even value for idle, else odd. */ bool rcu_need_heavy_qs;/* GP old, so heavy quiescent state! */ bool rcu_urgent_qs;/* GP old need light quiescent state. */
kernel/rcu/tree_stall.h +2 -2
@@ -333,7 +333,7 @@ static void print_cpu_stall_info(int cpu) }print_cpu_stall_fast_no_hz (fast_no_hz, cpu); delta =rcu_seq_ctr (rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); - pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld %s\n", + pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld softirq=%u/%u fqs=%ld %s\n", cpu, "O."[!!cpu_online(cpu)], "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)],
@@ -343,7 +343,7 @@ static void print_cpu_stall_info(int cpu)"!." [!delta], ticks_value, ticks_title,rcu_dynticks_snap (rdp) &0xfff , - rdp->dynticks_nesting, rdp->dynticks_nmi_nesting, + rdp->dynticks_nesting, rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu),READ_ONCE (rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, fast_no_hz);
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
Paul E. McKenney
maintainerQuoted 10 lines · Joel Fernandes (Google)
These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.
It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the
dynticks_nmi_nestingcounter is not a simple counter and can be "crowbarred" in common situations.rcutorture testing with all TREE RCU configurations succeed with
CONFIG_RCU_EQS_DEBUG=yandCONFIG_PROVE_LOCKING=y.
Heh! We now have a three-way collision between Thomas's and Peter's series, the RCU Tasks Trace series, and this series. ;-)
Remind me once v5.7-rc1 comes out and let's see what fits where.
Thanx, Paul
Quoted 23 lines
v1->v2:
- Rebase on v5.6-rc6Joel Fernandes (Google) (4):
Revertb8c17e6664c4("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Removedynticks_nmi_nestingcounter.../Data-Structures/Data-Structures.rst | 31 +-- Documentation/RCU/stallwarn.txt | 6 +- include/linux/rcutiny.h | 3 - include/trace/events/rcu.h | 29 +-- kernel/rcu/rcu.h | 4 - kernel/rcu/tree.c | 188 +++++++----------- kernel/rcu/tree.h | 4 +- kernel/rcu/tree_stall.h | 4 +- 8 files changed, 105 insertions(+), 164 deletions(-)--
2.26.0.rc2.310.g2932bb562d-goog
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Joel Fernandes (Google)
These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.
It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the
dynticks_nmi_nestingcounter is not a simple counter and can be "crowbarred" in common situations.
Quoted 7 lines
Quoted 2 lines
rcutorture testing with all TREE RCU configurations succeed with
CONFIG_RCU_EQS_DEBUG=yandCONFIG_PROVE_LOCKING=y.Heh! We now have a three-way collision between Thomas's and Peter's series, the RCU Tasks Trace series, and this series. ;-)
Remind me once v5.7-rc1 comes out and let's see what fits where.
Ok, no problem, I will resend at 5.7-rc1 time then. I believe I did a lot of the work to make the series catch up with the tip especially after the KCSAN changes, so it should be relatively easy (hopefully) for me to rebase at -rc1.
thanks!
- Joel
Quoted 23 lines
Quoted 23 lines
v1->v2:
- Rebase on v5.6-rc6Joel Fernandes (Google) (4):
Revertb8c17e6664c4("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Removedynticks_nmi_nestingcounter.../Data-Structures/Data-Structures.rst | 31 +-- Documentation/RCU/stallwarn.txt | 6 +- include/linux/rcutiny.h | 3 - include/trace/events/rcu.h | 29 +-- kernel/rcu/rcu.h | 4 - kernel/rcu/tree.c | 188 +++++++----------- kernel/rcu/tree.h | 4 +- kernel/rcu/tree_stall.h | 4 +- 8 files changed, 105 insertions(+), 164 deletions(-)--
2.26.0.rc2.310.g2932bb562d-goog