Skip to contentSkip to conversation

RCU dyntick nesting counter cleanups

PATCH Superseded Joel Fernandes (Google) 2 participants 7 messages Mar 28, 2020

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover RCU dyntick nesting counter cleanups

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.txt               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 188 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 105 insertions(+), 164 deletions(-)

--
2.26.0.rc2.310.g2932bb562d-goog

#1 lore raw

PATCH v2 1/4 Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")

This code is unused and can be removed now. Revert was straightforward.

Tested with rcutorture on all TREE configurations.

---
 include/linux/rcutiny.h |  3 --
 kernel/rcu/tree.c       | 82 ++++++++++-------------------------------
 2 files changed, 19 insertions(+), 66 deletions(-)
include/linux/rcutiny.h +0 -3
index b2b2dc990da99..18ee3539a79dd 100644
@@ -14,9 +14,6 @@
 
 #include <asm/param.h> /* for HZ */
 
-/* Never flag non-existent other CPUs! */
-static inline bool rcu_eqs_special_set(int cpu) { return false; }
-
 static inline unsigned long get_state_synchronize_rcu(void)
 {
 	return 0;
kernel/rcu/tree.c +19 -63

152 lines not on this page · raw message

Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
#2 lore raw

PATCH v2 2/4 rcu/tree: Add better tracing for dyntick-idle

In reply to Joel Fernandes (Google)

The dyntick-idle traces are a bit confusing. This patch makes it simpler and adds some missing cases such as EQS-enter due to user vs idle mode.

Following are the changes:
(1) Add a new context field to trace_rcu_dyntick tracepoint. This context field can be "USER", "IDLE" or "IRQ".

(2) Remove the "++=" and "--=" strings and replace them with "StillNonIdle". This is much easier on the eyes, and the -- and ++ are easily apparent in the dynticks_nesting counters we are printing anyway.

---
 include/trace/events/rcu.h | 29 ++++++++++++++++-------------
 kernel/rcu/tree.c          | 20 +++++++++++++-------
 2 files changed, 29 insertions(+), 20 deletions(-)
include/trace/events/rcu.h +16 -13
index 5e49b06e81044..c9ac71e2afd46 100644
@@ -435,26 +435,28 @@ TRACE_EVENT_RCU(rcu_fqs,
 #endif /* #if defined(CONFIG_TREE_RCU) */
 
 /*
- * Tracepoint for dyntick-idle entry/exit events.  These take a string
- * as argument: "Start" for entering dyntick-idle mode, "Startirq" for
- * entering it from irq/NMI, "End" for leaving it, "Endirq" for leaving it
- * to irq/NMI, "--=" for events moving towards idle, and "++=" for events
- * moving away from idle.
+ * Tracepoint for dyntick-idle entry/exit events.  These take 2 strings
+ * as argument:
+ * polarilty: "Start", "End", "StillIdle" for entering, exiting or still being
+ * in dyntick-idle mode.
+ * context: "USER" or "KERNEL" or "IRQ".
+ * NMIs nested in IRQs are inferred with dynticks_nesting > 1 in IRQ context.
  *
  * These events also take a pair of numbers, which indicate the nesting
  * depth before and after the event of interest, and a third number that is
- * the ->dynticks counter.  Note that task-related and interrupt-related
- * events use two separate counters, and that the "++=" and "--=" events
- * for irq/NMI will change the counter by two, otherwise by one.
+ * the ->dynticks counter. During NMI nesting within IRQs, the dynticks_nesting
+ * counter changes by two, otherwise one.
  */
 TRACE_EVENT_RCU(rcu_dyntick,
 
-	TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks),
+	TP_PROTO(const char *polarity, const char *context, long oldnesting,
+		 long newnesting, int dynticks),
 
-	TP_ARGS(polarity, oldnesting, newnesting, dynticks),
+	TP_ARGS(polarity, context, oldnesting, newnesting, dynticks),
 
 	TP_STRUCT__entry(
 		__field(const char *, polarity)
+		__field(const char *, context)
 		__field(long, oldnesting)
 		__field(long, newnesting)
 		__field(int, dynticks)
@@ -462,14 +464,15 @@ TRACE_EVENT_RCU(rcu_dyntick,
 
 	TP_fast_assign(
 		__entry->polarity = polarity;
+		__entry->context = context;
 		__entry->oldnesting = oldnesting;
 		__entry->newnesting = newnesting;
 		__entry->dynticks = dynticks;
 	),
 
-	TP_printk("%s %lx %lx %#3x", __entry->polarity,
-		  __entry->oldnesting, __entry->newnesting,
-		  __entry->dynticks & 0xfff)
+	TP_printk("%s %s %lx %lx %#3x", __entry->polarity,
+		__entry->context, __entry->oldnesting, __entry->newnesting,
+		__entry->dynticks & 0xfff)
 );
 
 /*
kernel/rcu/tree.c +13 -7
index a011bebe7d0e0..0e5304bad705a 100644
@@ -523,7 +523,8 @@ static void rcu_eqs_enter(bool user)
 	}
 
 	lockdep_assert_irqs_disabled();
-	trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("Start"), (user ? TPS("USER") : TPS("IDLE")),
+			  rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	rdp = this_cpu_ptr(&rcu_data);
 	do_nocb_deferred_wakeup(rdp);
@@ -596,15 +597,17 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	 * leave it in non-RCU-idle state.
 	 */
 	if (rdp->dynticks_nmi_nesting != 1) {
-		trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2,
-				  atomic_read(&rdp->dynticks));
+		trace_rcu_dyntick(TPS("StillNonIdle"), TPS("IRQ"),
+				  rdp->dynticks_nmi_nesting,
+				  rdp->dynticks_nmi_nesting - 2, atomic_read(&rdp->dynticks));
 		WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
 			   rdp->dynticks_nmi_nesting - 2);
 		return;
 	}
 
 	/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */
-	trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("Start"), TPS("IRQ"), rdp->dynticks_nmi_nesting,
+			  0, atomic_read(&rdp->dynticks));
 	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
 
 	if (irq)
@@ -691,7 +694,8 @@ static void rcu_eqs_exit(bool user)
 	rcu_dynticks_task_exit();
 	rcu_dynticks_eqs_exit();
 	rcu_cleanup_after_idle();
-	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("End"), (user ? TPS("USER") : TPS("IDLE")),
+			  rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	WRITE_ONCE(rdp->dynticks_nesting, 1);
 	WARN_ON_ONCE(rdp->dynticks_nmi_nesting);
@@ -783,9 +787,11 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 		}
 		raw_spin_unlock_rcu_node(rdp->mynode);
 	}
-	trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="),
-			  rdp->dynticks_nmi_nesting,
+
+	trace_rcu_dyntick(incby == 1 ? TPS("End") : TPS("StillNonIdle"),
+			  TPS("IRQ"), rdp->dynticks_nmi_nesting,
 			  rdp->dynticks_nmi_nesting + incby, atomic_read(&rdp->dynticks));
+
 	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */
 		   rdp->dynticks_nmi_nesting + incby);
 	barrier();
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
#3 lore raw

PATCH v2 3/4 rcu/tree: Clean up dynticks counter usage

In reply to Joel Fernandes (Google)

The dynticks counter are confusing due to crowbar writes of DYNTICK_IRQ_NONIDLE whose purpose is to detect half-interrupts (i.e. we see rcu_irq_enter() but not rcu_irq_exit() due to a usermode upcall) and if so then do a reset of the dyntick_nmi_nesting counters. This patch tries to get rid of DYNTICK_IRQ_NONIDLE while still keeping the code working, fully functional, and less confusing. The confusion recently has even led to patches forgetting that DYNTICK_IRQ_NONIDLE was written to which wasted lots of time.

The patch has the following changes:

(1) Use dynticks_nesting instead of dynticks_nmi_nesting for determining outer most "EQS exit". This is needed to detect in rcu_nmi_enter_common() if we have already EQS-exited, such as because of a syscall. Currently we rely on a forced write of DYNTICK_IRQ_NONIDLE from rcu_eqs_exit() for this purpose. This is one purpose of the DYNTICK_IRQ_NONIDLE write (other than detecting half-interrupts). However, we do not need to do that. dyntick_nesting already tells us that we have EQS-exited so just use that thus removing the dependence of dynticks_nmi_nesting for this purpose.

(2) Keep dynticks_nmi_nesting around because:

(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.

(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.

(3) Since we got rid of DYNTICK_IRQ_NONIDLE, we also do cheaper comparisons with zero instead for the code that keeps the tick on in rcu_nmi_enter_common().

In the next patch, both of the concerns of (2) will be addressed and then we can get rid of dynticks_nmi_nesting, however one step at a time.

---
 kernel/rcu/rcu.h  |  4 ----
 kernel/rcu/tree.c | 60 +++++++++++++++++++++++++++--------------------
 2 files changed, 35 insertions(+), 29 deletions(-)
kernel/rcu/rcu.h +0 -4
index 05f936ed167a7..429065a75e05f 100644
@@ -12,10 +12,6 @@
 
 #include <trace/events/rcu.h>
 
-/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */
-#define DYNTICK_IRQ_NONIDLE	((LONG_MAX / 2) + 1)
-
-
 /*
  * Grace-period counter management.
  */
kernel/rcu/tree.c +35 -25

128 lines not on this page · raw message

Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
#4 lore raw

PATCH v2 4/4 rcu/tree: Remove dynticks_nmi_nesting counter

In reply to Joel Fernandes (Google)

The dynticks_nmi_nesting counter serves 3 purposes:

(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.

(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.

(c) When a quiescent state report is needed from a nohz_full CPU. The nesting counter detects we are a first level interrupt.

For (a), we can just use dyntick_nesting == 1 to determine this. Only the outermost interrupt that interrupted an RCU-idle state can set it to 1.

For (b), this warning condition has not occurred for several kernel releases. But we still keep the warning but change it to use in_irq() instead of the nesting counter. In a later year, we can remove the warning.

For (c), the nest check is not really necessary since forced_tick would have been set to true in the outermost interrupt, so the nested/NMI interrupts will check forced_tick anyway, and bail.

---
 .../Data-Structures/Data-Structures.rst       | 31 ++++------
 Documentation/RCU/stallwarn.txt               |  6 +-
 kernel/rcu/tree.c                             | 56 +++++++------------
 kernel/rcu/tree.h                             |  4 +-
 kernel/rcu/tree_stall.h                       |  4 +-
 5 files changed, 37 insertions(+), 64 deletions(-)
Documentation/RCU/Design/Data-Structures/Data-Structures.rst +10 -21
index 4a48e20a46f2b..a5a907f434a1a 100644
@@ -936,10 +936,9 @@ This portion of the rcu_data structure is declared as follows:
 ::
 
      1   long dynticks_nesting;
-     2   long dynticks_nmi_nesting;
-     3   atomic_t dynticks;
-     4   bool rcu_need_heavy_qs;
-     5   bool rcu_urgent_qs;
+     2   atomic_t dynticks;
+     3   bool rcu_need_heavy_qs;
+     4   bool rcu_urgent_qs;
 
 These fields in the rcu_data structure maintain the per-CPU dyntick-idle
 state for the corresponding CPU. The fields may be accessed only from
@@ -948,26 +947,14 @@ the corresponding CPU (and from tracing) unless otherwise stated.
 The ``->dynticks_nesting`` field counts the nesting depth of process
 execution, so that in normal circumstances this counter has value zero
 or one. NMIs, irqs, and tracers are counted by the
-``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes
+``->dynticks_nesting`` field as well. Because NMIs cannot be masked, changes
 to this variable have to be undertaken carefully using an algorithm
 provided by Andy Lutomirski. The initial transition from idle adds one,
 and nested transitions add two, so that a nesting level of five is
-represented by a ``->dynticks_nmi_nesting`` value of nine. This counter
+represented by a ``->dynticks_nesting`` value of nine. This counter
 can therefore be thought of as counting the number of reasons why this
-CPU cannot be permitted to enter dyntick-idle mode, aside from
-process-level transitions.
-
-However, it turns out that when running in non-idle kernel context, the
-Linux kernel is fully capable of entering interrupt handlers that never
-exit and perhaps also vice versa. Therefore, whenever the
-``->dynticks_nesting`` field is incremented up from zero, the
-``->dynticks_nmi_nesting`` field is set to a large positive number, and
-whenever the ``->dynticks_nesting`` field is decremented down to zero,
-the the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that
-the number of misnested interrupts is not sufficient to overflow the
-counter, this approach corrects the ``->dynticks_nmi_nesting`` field
-every time the corresponding CPU enters the idle loop from process
-context.
+CPU cannot be permitted to enter dyntick-idle mode. It counts both the
+process-level and interrupt transitions.
 
 The ``->dynticks`` field counts the corresponding CPU's transitions to
 and from either dyntick-idle or user mode, so that this counter has an
@@ -1000,7 +987,9 @@ code.
 +-----------------------------------------------------------------------+
 | Because this would fail in the presence of interrupts whose handlers  |
 | never return and of handlers that manage to return from a made-up     |
-| interrupt.                                                            |
+| interrupt. NOTE: The counters have now been combined however          |
+| a temporary warning has been left to make sure this condition never   |
+| occurs.                                                               |
 +-----------------------------------------------------------------------+
 
 Additional fields are present for some special-purpose builds, and are
Documentation/RCU/stallwarn.txt +3 -3
index a360a8796710a..a8c068d82373e 100644
@@ -173,8 +173,8 @@ For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling,
 it will print a message similar to the following:
 
 	INFO: rcu_sched detected stalls on CPUs/tasks:
-	2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0
-	16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0
+	2-...: (3 GPs behind) idle=06c/0 softirq=1453/1455 fqs=0
+	16-...: (0 ticks this GP) idle=81c/0 softirq=764/764 fqs=0
 	(detected by 32, t=2603 jiffies, g=7075, q=625)
 
 This message indicates that CPU 32 detected that CPUs 2 and 16 were both
@@ -225,7 +225,7 @@ an estimate of the total number of RCU callbacks queued across all CPUs
 In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
 for each CPU:
 
-	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
+	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
 
 The "last_accelerate:" prints the low-order 16 bits (in hex) of the
 jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
kernel/rcu/tree.c +20 -36

122 lines not on this page · raw message

kernel/rcu/tree.h +2 -2
index 0c87e4c161c2f..b2e611ef6d43b 100644
@@ -175,8 +175,8 @@ struct rcu_data {
 
 	/* 3) dynticks interface. */
 	int dynticks_snap;		/* Per-GP tracking for dynticks. */
-	long dynticks_nesting;		/* Track process nesting level. */
-	long dynticks_nmi_nesting;	/* Track irq/NMI nesting level. */
+	long dynticks_nesting;		/* Track dyntick (non-IDLE) nesting
+					 * level for kernel entry and interrupt. */
 	atomic_t dynticks;		/* Even value for idle, else odd. */
 	bool rcu_need_heavy_qs;		/* GP old, so heavy quiescent state! */
 	bool rcu_urgent_qs;		/* GP old need light quiescent state. */
kernel/rcu/tree_stall.h +2 -2
index 55f9b84790d3f..f0f179f21cb2d 100644
@@ -333,7 +333,7 @@ static void print_cpu_stall_info(int cpu)
 	}
 	print_cpu_stall_fast_no_hz(fast_no_hz, cpu);
 	delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq);
-	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld %s\n",
+	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld softirq=%u/%u fqs=%ld %s\n",
 	       cpu,
 	       "O."[!!cpu_online(cpu)],
 	       "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)],
@@ -343,7 +343,7 @@ static void print_cpu_stall_info(int cpu)
 				"!."[!delta],
 	       ticks_value, ticks_title,
 	       rcu_dynticks_snap(rdp) & 0xfff,
-	       rdp->dynticks_nesting, rdp->dynticks_nmi_nesting,
+	       rdp->dynticks_nesting,
 	       rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu),
 	       READ_ONCE(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,
 	       fast_no_hz);
Signature · 1 line
2.26.0.rc2.310.g2932bb562d-goog
maintainer
#5 lore raw
In reply to Joel Fernandes (Google)
Quoted 10 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

Heh! We now have a three-way collision between Thomas's and Peter's series, the RCU Tasks Trace series, and this series. ;-)

Remind me once v5.7-rc1 comes out and let's see what fits where.

Thanx, Paul

Quoted 23 lines

v1->v2:
- Rebase on v5.6-rc6

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.txt               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 188 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 105 insertions(+), 164 deletions(-)

--
2.26.0.rc2.310.g2932bb562d-goog

#6 lore raw
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

Quoted 7 lines
Quoted 2 lines

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

Heh! We now have a three-way collision between Thomas's and Peter's series, the RCU Tasks Trace series, and this series. ;-)

Remind me once v5.7-rc1 comes out and let's see what fits where.

Ok, no problem, I will resend at 5.7-rc1 time then. I believe I did a lot of the work to make the series catch up with the tip especially after the KCSAN changes, so it should be relatively easy (hopefully) for me to rebase at -rc1.

thanks!

- Joel

Quoted 23 lines
Quoted 23 lines

v1->v2:
- Rebase on v5.6-rc6

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.txt               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 188 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 105 insertions(+), 164 deletions(-)

--
2.26.0.rc2.310.g2932bb562d-goog