Skip to contentSkip to conversation

RCU dyntick nesting counter cleanups for rcu -dev

PATCH In review Joel Fernandes (Google) 4 participants 14 messages May 4, 2020 – May 17, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover RCU dyntick nesting counter cleanups for rcu -dev

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

Madhuparna Bhowmik (1):
trace: events: rcu: Change description of rcu_dyntick trace event

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.rst               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 199 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 110 insertions(+), 170 deletions(-)

--
2.26.2.526.g744177e7f7-goog

#1 lore raw

PATCH v3 1/5 Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")

This code is unused and can be removed now. Revert was straightforward.

Tested with rcutorture on all TREE configurations.

---
 include/linux/rcutiny.h |  3 --
 kernel/rcu/tree.c       | 93 +++++++++++------------------------------
 2 files changed, 24 insertions(+), 72 deletions(-)
include/linux/rcutiny.h +0 -3
index 3465ba704a111..dbcddc7b26b94 100644
@@ -14,9 +14,6 @@
 
 #include <asm/param.h> /* for HZ */
 
-/* Never flag non-existent other CPUs! */
-static inline bool rcu_eqs_special_set(int cpu) { return false; }
-
 static inline unsigned long get_state_synchronize_rcu(void)
 {
 	return 0;
kernel/rcu/tree.c +24 -69

172 lines not on this page · raw message

Signature · 1 line
2.26.2.526.g744177e7f7-goog
#2 lore raw

PATCH v3 2/5 rcu/tree: Add better tracing for dyntick-idle

In reply to Joel Fernandes (Google)

The dyntick-idle traces are a bit confusing. This patch makes it simpler and adds some missing cases such as EQS-enter due to user vs idle mode.

Following are the changes:
(1) Add a new context field to trace_rcu_dyntick tracepoint. This context field can be "USER", "IDLE" or "IRQ".

(2) Remove the "++=" and "--=" strings and replace them with "StillNonIdle". This is much easier on the eyes, and the -- and ++ are easily apparent in the dynticks_nesting counters we are printing anyway.

---
 include/trace/events/rcu.h | 29 ++++++++++++++++-------------
 kernel/rcu/tree.c          | 20 +++++++++++++-------
 2 files changed, 29 insertions(+), 20 deletions(-)
include/trace/events/rcu.h +16 -13
index f9a7811148e2a..164c43b72ca29 100644
@@ -435,26 +435,28 @@ TRACE_EVENT_RCU(rcu_fqs,
 #endif /* #if defined(CONFIG_TREE_RCU) */
 
 /*
- * Tracepoint for dyntick-idle entry/exit events.  These take a string
- * as argument: "Start" for entering dyntick-idle mode, "Startirq" for
- * entering it from irq/NMI, "End" for leaving it, "Endirq" for leaving it
- * to irq/NMI, "--=" for events moving towards idle, and "++=" for events
- * moving away from idle.
+ * Tracepoint for dyntick-idle entry/exit events.  These take 2 strings
+ * as argument:
+ * polarilty: "Start", "End", "StillIdle" for entering, exiting or still being
+ * in dyntick-idle mode.
+ * context: "USER" or "KERNEL" or "IRQ".
+ * NMIs nested in IRQs are inferred with dynticks_nesting > 1 in IRQ context.
  *
  * These events also take a pair of numbers, which indicate the nesting
  * depth before and after the event of interest, and a third number that is
- * the ->dynticks counter.  Note that task-related and interrupt-related
- * events use two separate counters, and that the "++=" and "--=" events
- * for irq/NMI will change the counter by two, otherwise by one.
+ * the ->dynticks counter. During NMI nesting within IRQs, the dynticks_nesting
+ * counter changes by two, otherwise one.
  */
 TRACE_EVENT_RCU(rcu_dyntick,
 
-	TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks),
+	TP_PROTO(const char *polarity, const char *context, long oldnesting,
+		 long newnesting, int dynticks),
 
-	TP_ARGS(polarity, oldnesting, newnesting, dynticks),
+	TP_ARGS(polarity, context, oldnesting, newnesting, dynticks),
 
 	TP_STRUCT__entry(
 		__field(const char *, polarity)
+		__field(const char *, context)
 		__field(long, oldnesting)
 		__field(long, newnesting)
 		__field(int, dynticks)
@@ -462,14 +464,15 @@ TRACE_EVENT_RCU(rcu_dyntick,
 
 	TP_fast_assign(
 		__entry->polarity = polarity;
+		__entry->context = context;
 		__entry->oldnesting = oldnesting;
 		__entry->newnesting = newnesting;
 		__entry->dynticks = dynticks;
 	),
 
-	TP_printk("%s %lx %lx %#3x", __entry->polarity,
-		  __entry->oldnesting, __entry->newnesting,
-		  __entry->dynticks & 0xfff)
+	TP_printk("%s %s %lx %lx %#3x", __entry->polarity,
+		__entry->context, __entry->oldnesting, __entry->newnesting,
+		__entry->dynticks & 0xfff)
 );
 
 /*
kernel/rcu/tree.c +13 -7
index 1ec7b1d4a03c4..eb7a4d90b3b91 100644
@@ -573,7 +573,8 @@ static void rcu_eqs_enter(bool user)
 	}
 
 	lockdep_assert_irqs_disabled();
-	trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("Start"), (user ? TPS("USER") : TPS("IDLE")),
+			  rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	rdp = this_cpu_ptr(&rcu_data);
 	do_nocb_deferred_wakeup(rdp);
@@ -648,15 +649,17 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	 * leave it in non-RCU-idle state.
 	 */
 	if (rdp->dynticks_nmi_nesting != 1) {
-		trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2,
-				  atomic_read(&rdp->dynticks));
+		trace_rcu_dyntick(TPS("StillNonIdle"), TPS("IRQ"),
+				  rdp->dynticks_nmi_nesting,
+				  rdp->dynticks_nmi_nesting - 2, atomic_read(&rdp->dynticks));
 		WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
 			   rdp->dynticks_nmi_nesting - 2);
 		return;
 	}
 
 	/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */
-	trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("Start"), TPS("IRQ"), rdp->dynticks_nmi_nesting,
+			  0, atomic_read(&rdp->dynticks));
 	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
 
 	if (irq)
@@ -748,7 +751,8 @@ static void rcu_eqs_exit(bool user)
 	rcu_dynticks_eqs_exit();
 	// ... but is watching here.
 	rcu_cleanup_after_idle();
-	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
+	trace_rcu_dyntick(TPS("End"), (user ? TPS("USER") : TPS("IDLE")),
+			  rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	WRITE_ONCE(rdp->dynticks_nesting, 1);
 	WARN_ON_ONCE(rdp->dynticks_nmi_nesting);
@@ -850,9 +854,11 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 		}
 		raw_spin_unlock_rcu_node(rdp->mynode);
 	}
-	trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="),
-			  rdp->dynticks_nmi_nesting,
+
+	trace_rcu_dyntick(incby == 1 ? TPS("End") : TPS("StillNonIdle"),
+			  TPS("IRQ"), rdp->dynticks_nmi_nesting,
 			  rdp->dynticks_nmi_nesting + incby, atomic_read(&rdp->dynticks));
+
 	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */
 		   rdp->dynticks_nmi_nesting + incby);
 	barrier();
Signature · 1 line
2.26.2.526.g744177e7f7-goog
#3 lore raw

PATCH v3 3/5 rcu/tree: Clean up dynticks counter usage

In reply to Joel Fernandes (Google)

The dynticks counter are confusing due to crowbar writes of DYNTICK_IRQ_NONIDLE whose purpose is to detect half-interrupts (i.e. we see rcu_irq_enter() but not rcu_irq_exit() due to a usermode upcall) and if so then do a reset of the dyntick_nmi_nesting counters. This patch tries to get rid of DYNTICK_IRQ_NONIDLE while still keeping the code working, fully functional, and less confusing. The confusion recently has even led to patches forgetting that DYNTICK_IRQ_NONIDLE was written to which wasted lots of time.

The patch has the following changes:

(1) Use dynticks_nesting instead of dynticks_nmi_nesting for determining outer most "EQS exit". This is needed to detect in rcu_nmi_enter_common() if we have already EQS-exited, such as because of a syscall. Currently we rely on a forced write of DYNTICK_IRQ_NONIDLE from rcu_eqs_exit() for this purpose. This is one purpose of the DYNTICK_IRQ_NONIDLE write (other than detecting half-interrupts). However, we do not need to do that. dyntick_nesting already tells us that we have EQS-exited so just use that thus removing the dependence of dynticks_nmi_nesting for this purpose.

(2) Keep dynticks_nmi_nesting around because:

(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.

(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.

(3) Since we got rid of DYNTICK_IRQ_NONIDLE, we also do cheaper comparisons with zero instead for the code that keeps the tick on in rcu_nmi_enter_common().

In the next patch, both of the concerns of (2) will be addressed and then we can get rid of dynticks_nmi_nesting, however one step at a time.

---
 kernel/rcu/rcu.h  |  4 ----
 kernel/rcu/tree.c | 58 +++++++++++++++++++++++++++--------------------
 2 files changed, 34 insertions(+), 28 deletions(-)
kernel/rcu/rcu.h +0 -4
index cf66a3ccd7573..b9f64abc48b85 100644
@@ -12,10 +12,6 @@
 
 #include <trace/events/rcu.h>
 
-/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */
-#define DYNTICK_IRQ_NONIDLE	((LONG_MAX / 2) + 1)
-
-
 /*
  * Grace-period counter management.
  */
kernel/rcu/tree.c +34 -24

126 lines not on this page · raw message

Signature · 1 line
2.26.2.526.g744177e7f7-goog
#4 lore raw

PATCH v3 4/5 rcu/tree: Remove dynticks_nmi_nesting counter

In reply to Joel Fernandes (Google)

The dynticks_nmi_nesting counter serves 3 purposes:

(a) rcu_is_cpu_rrupt_from_idle() needs to be able to detect first interrupt nesting level.

(b) We need to detect half-interrupts till we are sure they're not an issue. However, change the comparison to DYNTICK_IRQ_NONIDLE with 0.

(c) When a quiescent state report is needed from a nohz_full CPU. The nesting counter detects we are a first level interrupt.

For (a), we can just use dyntick_nesting == 1 to determine this. Only the outermost interrupt that interrupted an RCU-idle state can set it to 1.

For (b), this warning condition has not occurred for several kernel releases. But we still keep the warning but change it to use in_irq() instead of the nesting counter. In a later year, we can remove the warning.

For (c), the nest check is not really necessary since forced_tick would have been set to true in the outermost interrupt, so the nested/NMI interrupts will check forced_tick anyway, and bail.

---
 .../Data-Structures/Data-Structures.rst       | 31 ++++-------
 Documentation/RCU/stallwarn.rst               |  6 +--
 kernel/rcu/tree.c                             | 54 +++++++------------
 kernel/rcu/tree.h                             |  4 +-
 kernel/rcu/tree_stall.h                       |  4 +-
 5 files changed, 36 insertions(+), 63 deletions(-)
Documentation/RCU/Design/Data-Structures/Data-Structures.rst +10 -21
index 4a48e20a46f2b..a5a907f434a1a 100644
@@ -936,10 +936,9 @@ This portion of the rcu_data structure is declared as follows:
 ::
 
      1   long dynticks_nesting;
-     2   long dynticks_nmi_nesting;
-     3   atomic_t dynticks;
-     4   bool rcu_need_heavy_qs;
-     5   bool rcu_urgent_qs;
+     2   atomic_t dynticks;
+     3   bool rcu_need_heavy_qs;
+     4   bool rcu_urgent_qs;
 
 These fields in the rcu_data structure maintain the per-CPU dyntick-idle
 state for the corresponding CPU. The fields may be accessed only from
@@ -948,26 +947,14 @@ the corresponding CPU (and from tracing) unless otherwise stated.
 The ``->dynticks_nesting`` field counts the nesting depth of process
 execution, so that in normal circumstances this counter has value zero
 or one. NMIs, irqs, and tracers are counted by the
-``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes
+``->dynticks_nesting`` field as well. Because NMIs cannot be masked, changes
 to this variable have to be undertaken carefully using an algorithm
 provided by Andy Lutomirski. The initial transition from idle adds one,
 and nested transitions add two, so that a nesting level of five is
-represented by a ``->dynticks_nmi_nesting`` value of nine. This counter
+represented by a ``->dynticks_nesting`` value of nine. This counter
 can therefore be thought of as counting the number of reasons why this
-CPU cannot be permitted to enter dyntick-idle mode, aside from
-process-level transitions.
-
-However, it turns out that when running in non-idle kernel context, the
-Linux kernel is fully capable of entering interrupt handlers that never
-exit and perhaps also vice versa. Therefore, whenever the
-``->dynticks_nesting`` field is incremented up from zero, the
-``->dynticks_nmi_nesting`` field is set to a large positive number, and
-whenever the ``->dynticks_nesting`` field is decremented down to zero,
-the the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that
-the number of misnested interrupts is not sufficient to overflow the
-counter, this approach corrects the ``->dynticks_nmi_nesting`` field
-every time the corresponding CPU enters the idle loop from process
-context.
+CPU cannot be permitted to enter dyntick-idle mode. It counts both the
+process-level and interrupt transitions.
 
 The ``->dynticks`` field counts the corresponding CPU's transitions to
 and from either dyntick-idle or user mode, so that this counter has an
@@ -1000,7 +987,9 @@ code.
 +-----------------------------------------------------------------------+
 | Because this would fail in the presence of interrupts whose handlers  |
 | never return and of handlers that manage to return from a made-up     |
-| interrupt.                                                            |
+| interrupt. NOTE: The counters have now been combined however          |
+| a temporary warning has been left to make sure this condition never   |
+| occurs.                                                               |
 +-----------------------------------------------------------------------+
 
 Additional fields are present for some special-purpose builds, and are
Documentation/RCU/stallwarn.rst +3 -3
index 08bc9aec4606a..d7042d893b167 100644
@@ -184,8 +184,8 @@ For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling,
 it will print a message similar to the following::
 
 	INFO: rcu_sched detected stalls on CPUs/tasks:
-	2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0
-	16-...: (0 ticks this GP) idle=81c/0/0 softirq=764/764 fqs=0
+	2-...: (3 GPs behind) idle=06c/0 softirq=1453/1455 fqs=0
+	16-...: (0 ticks this GP) idle=81c/0 softirq=764/764 fqs=0
 	(detected by 32, t=2603 jiffies, g=7075, q=625)
 
 This message indicates that CPU 32 detected that CPUs 2 and 16 were both
@@ -236,7 +236,7 @@ an estimate of the total number of RCU callbacks queued across all CPUs
 In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
 for each CPU::
 
-	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
+	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
 
 The "last_accelerate:" prints the low-order 16 bits (in hex) of the
 jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
kernel/rcu/tree.c +19 -35

120 lines not on this page · raw message

kernel/rcu/tree.h +2 -2
index 43991a40b0849..2803f85d9bd21 100644
@@ -177,8 +177,8 @@ struct rcu_data {
 
 	/* 3) dynticks interface. */
 	int dynticks_snap;		/* Per-GP tracking for dynticks. */
-	long dynticks_nesting;		/* Track process nesting level. */
-	long dynticks_nmi_nesting;	/* Track irq/NMI nesting level. */
+	long dynticks_nesting;		/* Track dyntick (non-IDLE) nesting
+					 * level for kernel entry and interrupt. */
 	atomic_t dynticks;		/* Even value for idle, else odd. */
 	bool rcu_need_heavy_qs;		/* GP old, so heavy quiescent state! */
 	bool rcu_urgent_qs;		/* GP old need light quiescent state. */
kernel/rcu/tree_stall.h +2 -2
index b899883a42478..f8521e9564fc7 100644
@@ -412,7 +412,7 @@ static void print_cpu_stall_info(int cpu)
 	delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq);
 	falsepositive = rcu_is_gp_kthread_starving(NULL) &&
 			rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp));
-	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld %s%s\n",
+	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld softirq=%u/%u fqs=%ld %s%s\n",
 	       cpu,
 	       "O."[!!cpu_online(cpu)],
 	       "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)],
@@ -422,7 +422,7 @@ static void print_cpu_stall_info(int cpu)
 				"!."[!delta],
 	       ticks_value, ticks_title,
 	       rcu_dynticks_snap(rdp) & 0xfff,
-	       rdp->dynticks_nesting, rdp->dynticks_nmi_nesting,
+	       rdp->dynticks_nesting,
 	       rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu),
 	       data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,
 	       fast_no_hz,
Signature · 1 line
2.26.2.526.g744177e7f7-goog
#5 lore raw

PATCH v3 5/5 trace: events: rcu: Change description of rcu_dyntick trace event

In reply to Joel Fernandes (Google)

From: Madhuparna Bhowmik <[email protected]>

The different strings used for describing the polarity are Start, End and StillNonIdle. Since StillIdle is not used in any trace point for rcu_dyntick, it can be removed and StillNonIdle can be added in the description. Because StillNonIdle is used in a few tracepoints for rcu_dyntick.

Similarly, USER, IDLE and IRQ are used for describing context in the rcu_dyntick tracepoints. Since, "KERNEL" is not used for any of the rcu_dyntick tracepoints, remove it from the description.

---
 include/trace/events/rcu.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
include/trace/events/rcu.h +3 -3
index 164c43b72ca29..02dcd119f3263 100644
@@ -437,9 +437,9 @@ TRACE_EVENT_RCU(rcu_fqs,
 /*
  * Tracepoint for dyntick-idle entry/exit events.  These take 2 strings
  * as argument:
- * polarilty: "Start", "End", "StillIdle" for entering, exiting or still being
- * in dyntick-idle mode.
- * context: "USER" or "KERNEL" or "IRQ".
+ * polarilty: "Start", "End", "StillNonIdle" for entering, exiting or still not
+ *            being in dyntick-idle mode.
+ * context: "USER" or "IDLE" or "IRQ".
  * NMIs nested in IRQs are inferred with dynticks_nesting > 1 in IRQ context.
  *
  * These events also take a pair of numbers, which indicate the nesting
Signature · 1 line
2.26.2.526.g744177e7f7-goog
maintainer
#6 lore raw
In reply to Joel Fernandes (Google)
Quoted 8 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

Quoted 8 lines

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Thank you!

But this does not apply to any of v5.6-rc6, v5.7-rc1, or v5.7-rc2.

Where should I be trying to apply it?

Thanx, Paul

Quoted 23 lines

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

Madhuparna Bhowmik (1):
trace: events: rcu: Change description of rcu_dyntick trace event

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.rst               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 199 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 110 insertions(+), 170 deletions(-)

--
2.26.2.526.g744177e7f7-goog

#7 lore raw
Quoted 16 lines · Paul E. McKenney
Quoted 14 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

Quoted 9 lines
Quoted 2 lines

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Thank you!

But this does not apply to any of v5.6-rc6, v5.7-rc1, or v5.7-rc2.

Where should I be trying to apply it?

Hi Paul,
It is for the rcu/dev branch as mentioned in the cover letter subject line. I thought it would reduce conflicts for you if I rebased on that. There are conflicts with trace-RCU for example.

What do you suggest?

Thanks,

- Joel

maintainer
#8 lore raw
In reply to Paul E. McKenney
Quoted 15 lines · Paul E. McKenney
Quoted 14 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

Quoted 8 lines
Quoted 2 lines

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Thank you!

But this does not apply to any of v5.6-rc6, v5.7-rc1, or v5.7-rc2.

Where should I be trying to apply it?

OK, morning blindness overcome. I new see the "rcu/dev" in v2->v3.

Please accept my apologies for the noise.

Thanx, Paul

Quoted 23 lines
Quoted 23 lines

Joel Fernandes (Google) (4):
Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter")
rcu/tree: Add better tracing for dyntick-idle rcu/tree: Clean up dynticks counter usage
rcu/tree: Remove dynticks_nmi_nesting counter

Madhuparna Bhowmik (1):
trace: events: rcu: Change description of rcu_dyntick trace event

.../Data-Structures/Data-Structures.rst       |  31 +--
Documentation/RCU/stallwarn.rst               |   6 +-
include/linux/rcutiny.h                       |   3 -
include/trace/events/rcu.h                    |  29 +--
kernel/rcu/rcu.h                              |   4 -
kernel/rcu/tree.c                             | 199 +++++++-----------
kernel/rcu/tree.h                             |   4 +-
kernel/rcu/tree_stall.h                       |   4 +-
8 files changed, 110 insertions(+), 170 deletions(-)

--
2.26.2.526.g744177e7f7-goog

#9 lore raw
Quoted 22 lines · Paul E. McKenney
Quoted 20 lines · Paul E. McKenney
Quoted 8 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

Quoted 8 lines

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Thank you!

Quoted 7 lines
Quoted 3 lines

But this does not apply to any of v5.6-rc6, v5.7-rc1, or v5.7-rc2.

Where should I be trying to apply it?

OK, morning blindness overcome. I new see the "rcu/dev" in v2->v3.

Please accept my apologies for the noise.

No problem, glad it got resolved. Thanks!

- Joel

maintainer
#10 lore raw
In reply to Paul E. McKenney
Quoted 21 lines · Paul E. McKenney
Quoted 20 lines · Paul E. McKenney
Quoted 8 lines · Joel Fernandes (Google)

These patches clean up the usage of dynticks nesting counters simplifying the code, while preserving the usecases.

It is a much needed simplification, makes the code less confusing, and prevents future bugs such as those that arise from forgetting that the dynticks_nmi_nesting counter is not a simple counter and can be "crowbarred" in common situations.

Quoted 8 lines

rcutorture testing with all TREE RCU configurations succeed with CONFIG_RCU_EQS_DEBUG=y and CONFIG_PROVE_LOCKING=y.

v1->v2:
- Rebase on v5.6-rc6

v2->v3:
- Rebase on rcu/dev with adjustments for tasks-RCU.

Thank you!

Quoted 7 lines
Quoted 3 lines

But this does not apply to any of v5.6-rc6, v5.7-rc1, or v5.7-rc2.

Where should I be trying to apply it?

OK, morning blindness overcome. I new see the "rcu/dev" in v2->v3.

Please accept my apologies for the noise.

And I have allegedly resolved the conflicts with Thomas's and Peter's series, or at least the noinstr-rcu-nmi-2020-05-15 branch of that series. At least one conflict was completely invisible to "git rebase" (but fortunately blindingly obvious to the compiler). This naturally makes me suspect that additional adjustments might be needed. Especially given that misplaced instrumentation_begin() and instrumentation_end() calls are invisible not only to the compiler, but also to rcutorture in my setup. (For example, tracing before instrumentation_begin() or after instrumentation_end() is a bug.)

So could you please carefully check these commits? They are now on the -rcu tree's "dev" branch.

Thanx, Paul

#11 lore raw

Paul,

Quoted 9 lines · Paul E. McKenney

And I have allegedly resolved the conflicts with Thomas's and Peter's series, or at least the noinstr-rcu-nmi-2020-05-15 branch of that series. At least one conflict was completely invisible to "git rebase" (but fortunately blindingly obvious to the compiler). This naturally makes me suspect that additional adjustments might be needed. Especially given that misplaced instrumentation_begin() and instrumentation_end() calls are invisible not only to the compiler, but also to rcutorture in my setup. (For example, tracing before instrumentation_begin() or after instrumentation_end() is a bug.)

there is tooling to verify that. If you merge

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v7-full

into your dev branch (trivial conflict) and after building it, run:

objtool check -fal $BUILDIR/vmlinux.o

Note: This only covers built-in code. So if you have code you want to check make sure it's not built as module.

Just did that and it complains about:

../build/vmlinux.o: warning: objtool: rcu_eqs_enter.constprop.0()+0xa9: call to rcu_preempt_deferred_qs() leaves .noinstr.text section

But that's even the case on my branch without your changes.

The call to rcu_preempt_deferred_qs() is definitely before instrumentation_end() so this needs to be investigated on the objtool side. Peter?

Thanks,

tglx

maintainer
#12 lore raw
Quoted 13 lines · Thomas Gleixner

Paul,

Quoted 9 lines · Paul E. McKenney

And I have allegedly resolved the conflicts with Thomas's and Peter's series, or at least the noinstr-rcu-nmi-2020-05-15 branch of that series. At least one conflict was completely invisible to "git rebase" (but fortunately blindingly obvious to the compiler). This naturally makes me suspect that additional adjustments might be needed. Especially given that misplaced instrumentation_begin() and instrumentation_end() calls are invisible not only to the compiler, but also to rcutorture in my setup. (For example, tracing before instrumentation_begin() or after instrumentation_end() is a bug.)

Quoted 7 lines

there is tooling to verify that. If you merge

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v7-full

into your dev branch (trivial conflict) and after building it, run:

objtool check -fal $BUILDIR/vmlinux.o

The conflict was trivial, but for future reference, where should I be getting the proper objtool from?

Thanx, Paul

#13 lore raw

Paul,

Quoted 5 lines · Paul E. McKenney
Quoted 1 line · Thomas Gleixner

objtool check -fal $BUILDIR/vmlinux.o

The conflict was trivial, but for future reference, where should I be getting the proper objtool from?

It's in the objtool/core branch of tip and targeted for 5.8 upstream.

Thanks,

tglx