Fix RCU idle-exit problem and comment

RFC PATCH In review Paul E. McKenney 2 participants 5 messages Mar 13, 2020

Subsystems (2)

Stable branch RCU

lore

Share & tools

lore

Paul E. McKenney

author maintainer

2020-03-13 02:40 UTC #0 lore raw

cover Fix RCU idle-exit problem and comment

Hello!

This series fixes a brown-paper-bag NMI self-deadlock bug located by Thomas Gleixner and adds comments:

1. Don't acquire lock in NMI handler in rcu_nmi_enter_common().

2. Add comments marking transitions between RCU watching and not.

These pass light rcutorture testing, and seem like v5.8 material.

Thanx, Paul

------------------------------------------------------------------------

 tree.c |   31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)

[email protected]

author maintainer

2020-03-13 02:40 UTC #1 lore raw

RFC PATCH 1/2 rcu: Don't acquire lock in NMI handler in rcu_nmi_enter_common()

From: "Paul E. McKenney" <[email protected]>

The rcu_nmi_enter_common() function can be invoked both in interrupt and NMI handlers. If it is invoked from process context (as opposed to userspace or idle context) on a nohz_full CPU, it might acquire the CPU's leaf rcu_node structure's ->lock. Because this lock is held only with interrupts disabled, this is safe from an interrupt handler, but doing so from an NMI handler can result in self-deadlock.

This commit therefore adds "irq" to the "if" condition so as to only acquire the ->lock from irq handlers or process context, never from an NMI handler.

Fixes 5b14557b073c ("rcu: Avoid tick_dep_set_cpu() misordering")

Reported-by Thomas Gleixner <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Cc <[email protected]> # 5.5.x

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/rcu/tree.c +1 -1

index d3f52c3..f7d3e48 100644
@@ -825,7 +825,7 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 			rcu_cleanup_after_idle();
 
 		incby = 1;
-	} else if (tick_nohz_full_cpu(rdp->cpu) &&
+	} else if (irq && tick_nohz_full_cpu(rdp->cpu) &&
 		   rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
 		   READ_ONCE(rdp->rcu_urgent_qs) &&
 		   !READ_ONCE(rdp->rcu_forced_tick)) {

Signature · 1 line

2.9.5

Joel Fernandes

Reviewed-by on v1 1/2

2020-03-13 03:52 UTC #2 lore raw

Re: [PATCH RFC tip/core/rcu 1/2] rcu: Don't acquire lock in NMI handler in rcu_nmi_enter_common()

Quoted 9 lines · On Thu, Mar 12, 2020 at 07:40:45PM -0700, [email protected] wrote:

From: "Paul E. McKenney" <[email protected]>
The rcu_nmi_enter_common() function can be invoked both in interrupt and NMI handlers. If it is invoked from process context (as opposed to userspace or idle context) on a nohz_full CPU, it might acquire the CPU's leaf rcu_node structure's ->lock. Because this lock is held only with interrupts disabled, this is safe from an interrupt handler, but doing so from an NMI handler can result in self-deadlock.

Quoted 3 lines

This commit therefore adds "irq" to the "if" condition so as to only acquire the ->lock from irq handlers or process context, never from an NMI handler.

I think Peter's new lockdep changes for NMI would also catch this issue.

Quoted 2 lines

Fixes 5b14557b073c ("rcu: Avoid tick_dep_set_cpu() misordering")

Reviewed-by Joel Fernandes (Google) <[email protected]>

thanks,

- Joel

Quoted 23 lines

Reported-by Thomas Gleixner <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Cc <[email protected]> # 5.5.x

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/rcu/tree.c +1 -1

index d3f52c3..f7d3e48 100644
@@ -825,7 +825,7 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 			rcu_cleanup_after_idle();
 
 		incby = 1;
-	} else if (tick_nohz_full_cpu(rdp->cpu) &&
+	} else if (irq && tick_nohz_full_cpu(rdp->cpu) &&
 		   rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
 		   READ_ONCE(rdp->rcu_urgent_qs) &&
 		   !READ_ONCE(rdp->rcu_forced_tick)) {

Signature · 1 line

2.9.5

Paul E. McKenney

author maintainer Applied

2020-03-13 21:47 UTC #3 lore raw

Re: [PATCH RFC tip/core/rcu 1/2] rcu: Don't acquire lock in NMI handler in rcu_nmi_enter_common()

Quoted 14 lines · Joel Fernandes

Quoted 9 lines · On Thu, Mar 12, 2020 at 07:40:45PM -0700, [email protected] wrote:
From: "Paul E. McKenney" <[email protected]>
The rcu_nmi_enter_common() function can be invoked both in interrupt and NMI handlers. If it is invoked from process context (as opposed to userspace or idle context) on a nohz_full CPU, it might acquire the CPU's leaf rcu_node structure's ->lock. Because this lock is held only with interrupts disabled, this is safe from an interrupt handler, but doing so from an NMI handler can result in self-deadlock.
Quoted 3 lines
This commit therefore adds "irq" to the "if" condition so as to only acquire the ->lock from irq handlers or process context, never from an NMI handler.

Quoted 6 lines

I think Peter's new lockdep changes for NMI would also catch this issue.
Quoted 2 lines
Fixes 5b14557b073c ("rcu: Avoid tick_dep_set_cpu() misordering")
Reviewed-by Joel Fernandes (Google) <[email protected]>

Applied, thank you!!!

Thanx, Paul

Quoted 28 lines

thanks,

- Joel

Quoted 23 lines

Reported-by Thomas Gleixner <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Cc <[email protected]> # 5.5.x

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/rcu/tree.c +1 -1

index d3f52c3..f7d3e48 100644
@@ -825,7 +825,7 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 			rcu_cleanup_after_idle();
 
 		incby = 1;
-	} else if (tick_nohz_full_cpu(rdp->cpu) &&
+	} else if (irq && tick_nohz_full_cpu(rdp->cpu) &&
 		   rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
 		   READ_ONCE(rdp->rcu_urgent_qs) &&
 		   !READ_ONCE(rdp->rcu_forced_tick)) {

Signature · 1 line

2.9.5

[email protected]

author maintainer

2020-03-13 02:40 UTC #4 lore raw

RFC PATCH 2/2 rcu: Add comments marking transitions between RCU watching and not

In reply to Paul E. McKenney

From: "Paul E. McKenney" <[email protected]>

It is not as clear as it might be just where in RCU's idle entry/exit code RCU stops and starts watching the current CPU. This commit therefore adds comments calling out the transitions.

Reported-by Thomas Gleixner <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

---
 kernel/rcu/tree.c | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

kernel/rcu/tree.c +25 -4

index f7d3e48..2f4c91a 100644
@@ -225,7 +225,9 @@ void rcu_softirq_qs(void)
 
 /*
  * Record entry into an extended quiescent state.  This is only to be
- * called when not already in an extended quiescent state.
+ * called when not already in an extended quiescent state, that is,
+ * RCU is watching prior to the call to this function and is no longer
+ * watching upon return.
  */
 static void rcu_dynticks_eqs_enter(void)
 {

@@ -238,7 +240,7 @@ static void rcu_dynticks_eqs_enter(void)
 	 * next idle sojourn.
 	 */
 	seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdp->dynticks);
-	/* Better be in an extended quiescent state! */
+	// RCU is no longer watching.  Better be in extended quiescent state!
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
 		     (seq & RCU_DYNTICK_CTRL_CTR));
 	/* Better not have special action (TLB flush) pending! */

@@ -248,7 +250,8 @@ static void rcu_dynticks_eqs_enter(void)
 
 /*
  * Record exit from an extended quiescent state.  This is only to be
- * called from an extended quiescent state.
+ * called from an extended quiescent state, that is, RCU is not watching
+ * prior to the call to this function and is watching upon return.
  */
 static void rcu_dynticks_eqs_exit(void)
 {

@@ -261,6 +264,7 @@ static void rcu_dynticks_eqs_exit(void)
 	 * critical section.
 	 */
 	seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdp->dynticks);
+	// RCU is now watching.  Better not be in an extended quiescent state!
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
 		     !(seq & RCU_DYNTICK_CTRL_CTR));
 	if (seq & RCU_DYNTICK_CTRL_MASK) {

@@ -571,6 +575,7 @@ static void rcu_eqs_enter(bool user)
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
 		     rdp->dynticks_nesting == 0);
 	if (rdp->dynticks_nesting != 1) {
+		// RCU will still be watching, so just do accounting and leave.
 		rdp->dynticks_nesting--;
 		return;
 	}

@@ -583,7 +588,9 @@ static void rcu_eqs_enter(bool user)
 	rcu_prepare_for_idle();
 	rcu_preempt_deferred_qs(current);
 	WRITE_ONCE(rdp->dynticks_nesting, 0); /* Avoid irq-access tearing. */
+	// RCU is watching here ...
 	rcu_dynticks_eqs_enter();
+	// ... but is no longer watching here.
 	rcu_dynticks_task_enter();
 }

@@ -663,7 +670,9 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	if (irq)
 		rcu_prepare_for_idle();
 
+	// RCU is watching here ...
 	rcu_dynticks_eqs_enter();
+	// ... but is no longer watching here.
 
 	if (irq)
 		rcu_dynticks_task_enter();

@@ -738,11 +747,14 @@ static void rcu_eqs_exit(bool user)
 	oldval = rdp->dynticks_nesting;
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0);
 	if (oldval) {
+		// RCU was already watching, so just do accounting and leave.
 		rdp->dynticks_nesting++;
 		return;
 	}
 	rcu_dynticks_task_exit();
+	// RCU is not watching here ...
 	rcu_dynticks_eqs_exit();
+	// ... but is watching here.
 	rcu_cleanup_after_idle();
 	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));

@@ -819,7 +831,9 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 		if (irq)
 			rcu_dynticks_task_exit();
 
+		// RCU is not watching here ...
 		rcu_dynticks_eqs_exit();
+		// ... but is watching here.
 
 		if (irq)
 			rcu_cleanup_after_idle();

@@ -829,9 +843,16 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
 		   rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
 		   READ_ONCE(rdp->rcu_urgent_qs) &&
 		   !READ_ONCE(rdp->rcu_forced_tick)) {
+		// We get here only if we had already exited the extended
+		// quiescent state and this was an interrupt (not an NMI).
+		// Therefore, (1) RCU is already watching and (2) The fact
+		// that we are in an interrupt handler and that the rcu_node
+		// lock is an irq-disabled lock prevents self-deadlock.
+		// So we can safely recheck under the lock.
 		raw_spin_lock_rcu_node(rdp->mynode);
-		// Recheck under lock.
 		if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+			// A nohz_full CPU is in the kernel and RCU
+			// needs a quiescent state.  Turn on the tick!
 			WRITE_ONCE(rdp->rcu_forced_tick, true);
 			tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
 		}

Signature · 1 line

2.9.5