Skip to contentSkip to conversation

srcu: A few NMI-safe debugging updates

PATCH In review Frederic Weisbecker 3 participants 7 messages Oct 13, 2022 – Oct 20, 2022

Subsystems (2)
lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

cover srcu: A few NMI-safe debugging updates

Hi,

This has passed SRCU-N, SRCU-P and SRCU-T so far.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
srcu/nmisafe

HEAD: 3cfdc2c6b8e89ca3c33954ea9b0d69e8cd141412

Thanks,
Frederic
---

Frederic Weisbecker (3):
srcu: Warn when NMI-unsafe API is used in NMI
srcu: Explain the reason behind the read side critical section on GP start
srcu: Debug NMI safety even on archs that don't require it

 include/linux/srcu.h     | 44 ++++++++++++++++++++++++++++++++++----------
 include/linux/srcutiny.h | 12 ------------
 include/linux/srcutree.h |  7 -------
 kernel/rcu/srcutree.c    | 31 ++++++++++++++++---------------
 4 files changed, 50 insertions(+), 44 deletions(-)
author maintainer
#1 lore raw

PATCH 1/3 srcu: Warn when NMI-unsafe API is used in NMI

Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:

1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.

2) If the only reader site for a given ssp is in an NMI, RCU is definetly a better choice over SRCU.

3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.

For all these reasons, warn if an nmi unsafe reader API is used from an NMI.

---
 kernel/rcu/srcutree.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/srcutree.c +2 -0
index c54142374793..8b7ef1031d89 100644
@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
 
 	if (!IS_ENABLED(CONFIG_PROVE_RCU))
 		return;
+	/* NMI-unsafe use in NMI is a bad sign */
+	WARN_ON_ONCE(!nmi_safe && in_nmi());
 	sdp = raw_cpu_ptr(ssp->sda);
 	old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety);
 	if (!old_nmi_safe_mask) {
Signature · 1 line
2.25.1
author maintainer
#4 lore raw

PATCH 2/3 srcu: Explain the reason behind the read side critical section on GP start

In reply to Frederic Weisbecker

Tell about the need to protect against concurrent updaters who may overflow the GP counter behind the current update.

---
 kernel/rcu/srcutree.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/rcu/srcutree.c +5 -0
index 8b7ef1031d89..5bf67d997796 100644
@@ -1154,6 +1154,11 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
 	int ss_state;
 
 	check_init_srcu_struct(ssp);
+	/*
+	 * While starting the new gp if needed, make sure we are in an SRCU read
+	 * side critical section so that the gp sequence can't wrap around in
+	 * the middle.
+	 */
 	idx = __srcu_read_lock_nmisafe(ssp, false);
 	ss_state = smp_load_acquire(&ssp->srcu_size_state);
 	if (ss_state < SRCU_SIZE_WAIT_CALL)
Signature · 1 line
2.25.1
author maintainer
#5 lore raw

PATCH 3/3 srcu: Debug NMI safety even on archs that don't require it

In reply to Frederic Weisbecker

Currently the NMI safety debugging is only performed on architectures that don't support NMI-safe this_cpu_inc().

Reorder the code so that other architectures like x86 also detect bad uses.

---
 include/linux/srcu.h     | 44 +++++++++++++++++++++++++++++++---------
 include/linux/srcutiny.h | 12 -----------
 include/linux/srcutree.h |  7 -------
 kernel/rcu/srcutree.c    | 24 ++++++++--------------
 4 files changed, 43 insertions(+), 44 deletions(-)
include/linux/srcu.h +34 -10
index 565f60d57484..f0814ffca34b 100644
@@ -52,8 +52,6 @@ int init_srcu_struct(struct srcu_struct *ssp);
 #else
 /* Dummy definition for things like notifiers.  Actual use gets link error. */
 struct srcu_struct { };
-int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp);
-void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp);
 #endif
 
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
@@ -66,6 +64,20 @@ unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp);
 unsigned long start_poll_synchronize_srcu(struct srcu_struct *ssp);
 bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie);
 
+#ifdef CONFIG_NEED_SRCU_NMI_SAFE
+int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp);
+void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp);
+#else
+static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp)
+{
+	return __srcu_read_lock(ssp);
+}
+static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)
+{
+	__srcu_read_unlock(ssp, idx);
+}
+#endif /* CONFIG_NEED_SRCU_NMI_SAFE */
+
 #ifdef CONFIG_SRCU
 void srcu_init(void);
 #else /* #ifdef CONFIG_SRCU */
@@ -106,6 +118,18 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp)
 
 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
 
+#define SRCU_NMI_UNKNOWN	0x0
+#define SRCU_NMI_UNSAFE		0x1
+#define SRCU_NMI_SAFE		0x2
+
+#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU)
+void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe);
+#else
+static inline void srcu_check_nmi_safety(struct srcu_struct *ssp,
+					 bool nmi_safe) { }
+#endif
+
+
 /**
  * srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing
  * @p: the pointer to fetch and protect for later dereferencing
@@ -163,6 +187,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp)
 {
 	int retval;
 
+	srcu_check_nmi_safety(ssp, false);
 	retval = __srcu_read_lock(ssp);
 	rcu_lock_acquire(&(ssp)->dep_map);
 	return retval;
@@ -179,10 +204,8 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp
 {
 	int retval;
 
-	if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
-		retval = __srcu_read_lock_nmisafe(ssp, true);
-	else
-		retval = __srcu_read_lock(ssp);
+	srcu_check_nmi_safety(ssp, true);
+	retval = __srcu_read_lock_nmisafe(ssp);
 	rcu_lock_acquire(&(ssp)->dep_map);
 	return retval;
 }
@@ -193,6 +216,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp)
 {
 	int retval;
 
+	srcu_check_nmi_safety(ssp, false);
 	retval = __srcu_read_lock(ssp);
 	return retval;
 }
@@ -208,6 +232,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)
 	__releases(ssp)
 {
 	WARN_ON_ONCE(idx & ~0x1);
+	srcu_check_nmi_safety(ssp, false);
 	rcu_lock_release(&(ssp)->dep_map);
 	__srcu_read_unlock(ssp, idx);
 }
@@ -223,17 +248,16 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)
 	__releases(ssp)
 {
 	WARN_ON_ONCE(idx & ~0x1);
+	srcu_check_nmi_safety(ssp, true);
 	rcu_lock_release(&(ssp)->dep_map);
-	if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
-		__srcu_read_unlock_nmisafe(ssp, idx, true);
-	else
-		__srcu_read_unlock(ssp, idx);
+	__srcu_read_unlock_nmisafe(ssp, idx);
 }
 
 /* Used by tracing, cannot be traced and cannot call lockdep. */
 static inline notrace void
 srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp)
 {
+	srcu_check_nmi_safety(ssp, false);
 	__srcu_read_unlock(ssp, idx);
 }
 
include/linux/srcutiny.h +0 -12
index efd808a23f8d..a2f620f8c559 100644
@@ -87,16 +87,4 @@ static inline void srcu_torture_stats_print(struct srcu_struct *ssp,
 		 data_race(READ_ONCE(ssp->srcu_lock_nesting[!idx])),
 		 data_race(READ_ONCE(ssp->srcu_lock_nesting[idx])));
 }
-
-static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe)
-{
-	BUG();
-	return 0;
-}
-
-static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe)
-{
-	BUG();
-}
-
 #endif
include/linux/srcutree.h +0 -7
index 35ffdedf86cc..c689a81752c9 100644
@@ -43,10 +43,6 @@ struct srcu_data {
 	struct srcu_struct *ssp;
 };
 
-#define SRCU_NMI_UNKNOWN	0x0
-#define SRCU_NMI_NMI_UNSAFE	0x1
-#define SRCU_NMI_NMI_SAFE	0x2
-
 /*
  * Node in SRCU combining tree, similar in function to rcu_data.
  */
@@ -159,7 +155,4 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp);
 void srcu_barrier(struct srcu_struct *ssp);
 void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf);
 
-int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp);
-void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp);
-
 #endif
kernel/rcu/srcutree.c +9 -15
index 5bf67d997796..f6425076779c 100644
@@ -631,17 +631,16 @@ void cleanup_srcu_struct(struct srcu_struct *ssp)
 }
 EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
 
+#ifdef CONFIG_PROVE_RCU
 /*
  * Check for consistent NMI safety.
  */
-static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
+void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
 {
 	int nmi_safe_mask = 1 << nmi_safe;
 	int old_nmi_safe_mask;
 	struct srcu_data *sdp;
 
-	if (!IS_ENABLED(CONFIG_PROVE_RCU))
-		return;
 	/* NMI-unsafe use in NMI is a bad sign */
 	WARN_ON_ONCE(!nmi_safe && in_nmi());
 	sdp = raw_cpu_ptr(ssp->sda);
@@ -652,6 +651,7 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
 	}
 	WARN_ONCE(old_nmi_safe_mask != nmi_safe_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_nmi_safe_mask, nmi_safe_mask);
 }
+#endif /* CONFIG_PROVE_RCU */
 
 /*
  * Counts the new reader in the appropriate per-CPU element of the
@@ -665,7 +665,6 @@ int __srcu_read_lock(struct srcu_struct *ssp)
 	idx = READ_ONCE(ssp->srcu_idx) & 0x1;
 	this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter);
 	smp_mb(); /* B */  /* Avoid leaking the critical section. */
-	srcu_check_nmi_safety(ssp, false);
 	return idx;
 }
 EXPORT_SYMBOL_GPL(__srcu_read_lock);
@@ -679,7 +678,6 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx)
 {
 	smp_mb(); /* C */  /* Avoid leaking the critical section. */
 	this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter);
-	srcu_check_nmi_safety(ssp, false);
 }
 EXPORT_SYMBOL_GPL(__srcu_read_unlock);
 
@@ -688,7 +686,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);
  * srcu_struct, but in an NMI-safe manner using RMW atomics.
  * Returns an index that must be passed to the matching srcu_read_unlock().
  */
-int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe)
+int __srcu_read_lock_nmisafe(struct srcu_struct *ssp)
 {
 	int idx;
 	struct srcu_data *sdp = raw_cpu_ptr(ssp->sda);
@@ -696,8 +694,6 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe)
 	idx = READ_ONCE(ssp->srcu_idx) & 0x1;
 	atomic_long_inc(&sdp->srcu_lock_count[idx]);
 	smp_mb__after_atomic(); /* B */  /* Avoid leaking the critical section. */
-	if (chknmisafe)
-		srcu_check_nmi_safety(ssp, true);
 	return idx;
 }
 EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe);
@@ -707,14 +703,12 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe);
  * element of the srcu_struct.  Note that this may well be a different
  * CPU than that which was incremented by the corresponding srcu_read_lock().
  */
-void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe)
+void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)
 {
 	struct srcu_data *sdp = raw_cpu_ptr(ssp->sda);
 
 	smp_mb__before_atomic(); /* C */  /* Avoid leaking the critical section. */
 	atomic_long_inc(&sdp->srcu_unlock_count[idx]);
-	if (chknmisafe)
-		srcu_check_nmi_safety(ssp, true);
 }
 EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe);
 
@@ -1159,7 +1153,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
 	 * side critical section so that the gp sequence can't wrap around in
 	 * the middle.
 	 */
-	idx = __srcu_read_lock_nmisafe(ssp, false);
+	idx = __srcu_read_lock_nmisafe(ssp);
 	ss_state = smp_load_acquire(&ssp->srcu_size_state);
 	if (ss_state < SRCU_SIZE_WAIT_CALL)
 		sdp = per_cpu_ptr(ssp->sda, 0);
@@ -1192,7 +1186,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
 		srcu_funnel_gp_start(ssp, sdp, s, do_norm);
 	else if (needexp)
 		srcu_funnel_exp_start(ssp, sdp_mynode, s);
-	__srcu_read_unlock_nmisafe(ssp, idx, false);
+	__srcu_read_unlock_nmisafe(ssp, idx);
 	return s;
 }
 
@@ -1496,13 +1490,13 @@ void srcu_barrier(struct srcu_struct *ssp)
 	/* Initial count prevents reaching zero until all CBs are posted. */
 	atomic_set(&ssp->srcu_barrier_cpu_cnt, 1);
 
-	idx = __srcu_read_lock_nmisafe(ssp, false);
+	idx = __srcu_read_lock_nmisafe(ssp);
 	if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER)
 		srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0));
 	else
 		for_each_possible_cpu(cpu)
 			srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu));
-	__srcu_read_unlock_nmisafe(ssp, idx, false);
+	__srcu_read_unlock_nmisafe(ssp, idx);
 
 	/* Remove the initial count, at which point reaching zero can happen. */
 	if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt))
Signature · 1 line
2.25.1
maintainer
#6 lore raw
In reply to Frederic Weisbecker
Quoted 8 lines · Frederic Weisbecker

Hi,

This has passed SRCU-N, SRCU-P and SRCU-T so far.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
srcu/nmisafe

HEAD: 3cfdc2c6b8e89ca3c33954ea9b0d69e8cd141412

Thank you, Frederic!

I have pulled this is for review and testing.

Thanx, Paul

Quoted 15 lines

Thanks,
Frederic
---

Frederic Weisbecker (3):
srcu: Warn when NMI-unsafe API is used in NMI
srcu: Explain the reason behind the read side critical section on GP start
srcu: Debug NMI safety even on archs that don't require it

 include/linux/srcu.h     | 44 ++++++++++++++++++++++++++++++++++----------
 include/linux/srcutiny.h | 12 ------------
 include/linux/srcutree.h |  7 -------
 kernel/rcu/srcutree.c    | 31 ++++++++++++++++---------------
 4 files changed, 50 insertions(+), 44 deletions(-)
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] srcu: Warn when NMI-unsafe API is used in NMI

In reply to Frederic Weisbecker
Quoted 10 lines · Frederic Weisbecker

Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:

1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.

2) If the only reader site for a given ssp is in an NMI, RCU is definetly

definitely

Quoted 1 line

a better choice over SRCU.

Just checking - because NMI are by definition not-preemptibe, so SRCU over RCU doesn't make much sense right?

thanks,

- Joel

Quoted 29 lines

3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.

For all these reasons, warn if an nmi unsafe reader API is used from an NMI.

---
 kernel/rcu/srcutree.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/srcutree.c +2 -0
index c54142374793..8b7ef1031d89 100644
@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
 
 	if (!IS_ENABLED(CONFIG_PROVE_RCU))
 		return;
+	/* NMI-unsafe use in NMI is a bad sign */
+	WARN_ON_ONCE(!nmi_safe && in_nmi());
 	sdp = raw_cpu_ptr(ssp->sda);
 	old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety);
 	if (!old_nmi_safe_mask) {
Signature · 1 line
2.25.1
maintainer
#3 lore raw

Re: [PATCH 1/3] srcu: Warn when NMI-unsafe API is used in NMI

Quoted 10 lines · Joel Fernandes
Quoted 9 lines · Frederic Weisbecker

Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:

1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.

Quoted 6 lines
Quoted 1 line

2) If the only reader site for a given ssp is in an NMI, RCU is definetly

definitely

Quoted 1 line

a better choice over SRCU.

Just checking - because NMI are by definition not-preemptibe, so SRCU over RCU doesn't make much sense right?

Agreed. But you never know...

Quoted 1 line

I will apply on the next rebase (after today's rebase), thank you!

Thanx, Paul

Quoted 33 lines

thanks,

- Joel

Quoted 29 lines

3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.

For all these reasons, warn if an nmi unsafe reader API is used from an NMI.

---
 kernel/rcu/srcutree.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/srcutree.c +2 -0
index c54142374793..8b7ef1031d89 100644
@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)
 
 	if (!IS_ENABLED(CONFIG_PROVE_RCU))
 		return;
+	/* NMI-unsafe use in NMI is a bad sign */
+	WARN_ON_ONCE(!nmi_safe && in_nmi());
 	sdp = raw_cpu_ptr(ssp->sda);
 	old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety);
 	if (!old_nmi_safe_mask) {
Signature · 1 line
2.25.1