srcu: A few NMI-safe debugging updates
Subsystems (2)
Frederic Weisbecker
maintainercover srcu: A few NMI-safe debugging updates
Hi,
This has passed SRCU-N, SRCU-P and SRCU-T so far.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
srcu/nmisafe
HEAD: 3cfdc2c6b8e89ca3c33954ea9b0d69e8cd141412
Thanks,
Frederic
---
Frederic Weisbecker (3):
srcu: Warn when NMI-unsafe API is used in NMI
srcu: Explain the reason behind the read side critical section on GP start
srcu: Debug NMI safety even on archs that don't require it
include/linux/srcu.h | 44 ++++++++++++++++++++++++++++++++++---------- include/linux/srcutiny.h | 12 ------------ include/linux/srcutree.h | 7 ------- kernel/rcu/srcutree.c | 31 ++++++++++++++++--------------- 4 files changed, 50 insertions(+), 44 deletions(-)
Frederic Weisbecker
maintainerPATCH 1/3 srcu: Warn when NMI-unsafe API is used in NMI
Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:
1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.
2) If the only reader site for a given ssp is in an NMI, RCU is definetly a better choice over SRCU.
3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.
For all these reasons, warn if an nmi unsafe reader API is used from an NMI.
--- kernel/rcu/srcutree.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/rcu/srcutree.c +2 -0
@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)if (!IS_ENABLED (CONFIG_PROVE_RCU ))return ; +/* NMI-unsafe use in NMI is a bad sign */ +WARN_ON_ONCE (!nmi_safe &&in_nmi ()); sdp =raw_cpu_ptr (ssp->sda); old_nmi_safe_mask =READ_ONCE (sdp->srcu_nmi_safety);if (!old_nmi_safe_mask) {
Signature · 1 line
2.25.1
Re: [PATCH 1/3] srcu: Warn when NMI-unsafe API is used in NMI
Quoted 10 lines · Frederic Weisbecker
Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:
1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.
2) If the only reader site for a given ssp is in an NMI, RCU is definetly
definitely
Quoted 1 line
a better choice over SRCU.
Just checking - because NMI are by definition not-preemptibe, so SRCU over RCU doesn't make much sense right?
thanks,
- Joel
Quoted 29 lines
3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.
For all these reasons, warn if an nmi unsafe reader API is used from an NMI.
Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/rcu/srcutree.c | 2 ++ 1 file changed, 2 insertions(+)kernel/rcu/srcutree.c +2
-0@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)if (!IS_ENABLED (CONFIG_PROVE_RCU ))return ; +/* NMI-unsafe use in NMI is a bad sign */ +WARN_ON_ONCE (!nmi_safe &&in_nmi ()); sdp =raw_cpu_ptr (ssp->sda); old_nmi_safe_mask =READ_ONCE (sdp->srcu_nmi_safety);if (!old_nmi_safe_mask) {Signature · 1 line
2.25.1
Paul E. McKenney
maintainerRe: [PATCH 1/3] srcu: Warn when NMI-unsafe API is used in NMI
Quoted 10 lines · Joel Fernandes
Quoted 9 lines · Frederic Weisbecker
Using the NMI-unsafe reader API from within NMIs is very likely to be buggy for three reasons:
1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen.
Quoted 6 lines
Quoted 1 line
2) If the only reader site for a given ssp is in an NMI, RCU is definetly
definitely
Quoted 1 line
a better choice over SRCU.
Just checking - because NMI are by definition not-preemptibe, so SRCU over RCU doesn't make much sense right?
Agreed. But you never know...
Quoted 1 line
Reviewed-by Joel Fernandes (Google) <[email protected]>
I will apply on the next rebase (after today's rebase), thank you!
Thanx, Paul
Quoted 33 lines
thanks,
- Joel
Quoted 29 lines
3) Because of the previous reason (2), an ssp having an SRCU read side critical section in an NMI is likely to have another one from a task context.
For all these reasons, warn if an nmi unsafe reader API is used from an NMI.
Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/rcu/srcutree.c | 2 ++ 1 file changed, 2 insertions(+)kernel/rcu/srcutree.c +2
-0@@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe)if (!IS_ENABLED (CONFIG_PROVE_RCU ))return ; +/* NMI-unsafe use in NMI is a bad sign */ +WARN_ON_ONCE (!nmi_safe &&in_nmi ()); sdp =raw_cpu_ptr (ssp->sda); old_nmi_safe_mask =READ_ONCE (sdp->srcu_nmi_safety);if (!old_nmi_safe_mask) {Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 2/3 srcu: Explain the reason behind the read side critical section on GP start
In reply to Frederic WeisbeckerTell about the need to protect against concurrent updaters who may overflow the GP counter behind the current update.
--- kernel/rcu/srcutree.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/rcu/srcutree.c +5 -0
@@ -1154,6 +1154,11 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,int ss_state;check_init_srcu_struct (ssp); +/* +* While starting the new gp if needed, make sure we are in an SRCU read +* side critical section so that the gp sequence can't wrap around in +* the middle. +*/ idx =__srcu_read_lock_nmisafe (ssp, false); ss_state =smp_load_acquire (&ssp->srcu_size_state);if (ss_state <SRCU_SIZE_WAIT_CALL )
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 3/3 srcu: Debug NMI safety even on archs that don't require it
In reply to Frederic WeisbeckerCurrently the NMI safety debugging is only performed on architectures that don't support NMI-safe this_cpu_inc().
Reorder the code so that other architectures like x86 also detect bad uses.
--- include/linux/srcu.h | 44 +++++++++++++++++++++++++++++++--------- include/linux/srcutiny.h | 12 ----------- include/linux/srcutree.h | 7 ------- kernel/rcu/srcutree.c | 24 ++++++++-------------- 4 files changed, 43 insertions(+), 44 deletions(-)
include/linux/srcu.h +34 -10
@@ -52,8 +52,6 @@ int init_srcu_struct(struct srcu_struct *ssp);#else /* Dummy definition for things like notifiers. Actual use gets link error. */ struct srcu_struct { }; -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp);#endif void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
@@ -66,6 +64,20 @@ unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp);unsigned long start_poll_synchronize_srcu (struct srcu_struct *ssp);bool poll_state_synchronize_srcu (struct srcu_struct *ssp,unsigned long cookie); +#ifdef CONFIG_NEED_SRCU_NMI_SAFE +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp); +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp); +#else +static inline int __srcu_read_lock_nmisafe (struct srcu_struct *ssp) +{ +return __srcu_read_lock (ssp); +} +static inline void __srcu_read_unlock_nmisafe (struct srcu_struct *ssp,int idx) +{ +__srcu_read_unlock (ssp, idx); +} +#endif /* CONFIG_NEED_SRCU_NMI_SAFE */ +#ifdef CONFIG_SRCU void srcu_init (void );#else /* #ifdef CONFIG_SRCU */
@@ -106,6 +118,18 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp)#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#define SRCU_NMI_UNKNOWN 0x0 +#define SRCU_NMI_UNSAFE 0x1 +#define SRCU_NMI_SAFE 0x2 + +#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_TREE_SRCU ) +void srcu_check_nmi_safety (struct srcu_struct *ssp,bool nmi_safe); +#else +static inline void srcu_check_nmi_safety (struct srcu_struct *ssp, +bool nmi_safe) { } +#endif + +/** * srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing * @p: the pointer to fetch and protect for later dereferencing
@@ -163,6 +187,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) {int retval; +srcu_check_nmi_safety (ssp, false); retval =__srcu_read_lock (ssp);rcu_lock_acquire (&(ssp)->dep_map);return retval;
@@ -179,10 +204,8 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp {int retval; -if (IS_ENABLED (CONFIG_NEED_SRCU_NMI_SAFE )) - retval =__srcu_read_lock_nmisafe (ssp, true); -else - retval =__srcu_read_lock (ssp); +srcu_check_nmi_safety (ssp, true); + retval =__srcu_read_lock_nmisafe (ssp);rcu_lock_acquire (&(ssp)->dep_map);return retval; }
@@ -193,6 +216,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) {int retval; +srcu_check_nmi_safety (ssp, false); retval =__srcu_read_lock (ssp);return retval; }
@@ -208,6 +232,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); +srcu_check_nmi_safety (ssp, false);rcu_lock_release (&(ssp)->dep_map);__srcu_read_unlock (ssp, idx); }
@@ -223,17 +248,16 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); +srcu_check_nmi_safety (ssp, true);rcu_lock_release (&(ssp)->dep_map); -if (IS_ENABLED (CONFIG_NEED_SRCU_NMI_SAFE )) -__srcu_read_unlock_nmisafe (ssp, idx, true); -else -__srcu_read_unlock (ssp, idx); +__srcu_read_unlock_nmisafe (ssp, idx); }/* Used by tracing, cannot be traced and cannot call lockdep. */ static inline notrace voidsrcu_read_unlock_notrace (struct srcu_struct *ssp,int idx)__releases (ssp) { +srcu_check_nmi_safety (ssp, false);__srcu_read_unlock (ssp, idx); }
include/linux/srcutiny.h +0 -12
@@ -87,16 +87,4 @@ static inline void srcu_torture_stats_print(struct srcu_struct *ssp,data_race (READ_ONCE (ssp->srcu_lock_nesting[!idx])), data_race(READ_ONCE(ssp->srcu_lock_nesting[idx]))); } - -static inline int __srcu_read_lock_nmisafe (struct srcu_struct *ssp,bool chknmisafe) -{ -BUG (); -return 0 ; -} - -static inline void __srcu_read_unlock_nmisafe (struct srcu_struct *ssp,int idx,bool chknmisafe) -{ -BUG (); -} -#endif
include/linux/srcutree.h +0 -7
@@ -43,10 +43,6 @@ struct srcu_data {struct srcu_struct *ssp; }; -#define SRCU_NMI_UNKNOWN 0x0 -#define SRCU_NMI_NMI_UNSAFE 0x1 -#define SRCU_NMI_NMI_SAFE 0x2 -/* * Node in SRCU combining tree, similar in function to rcu_data. */
@@ -159,7 +155,4 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp);void srcu_barrier (struct srcu_struct *ssp);void srcu_torture_stats_print (struct srcu_struct *ssp,char *tt,char *tf); -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp); -#endif
kernel/rcu/srcutree.c +9 -15
@@ -631,17 +631,16 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) }EXPORT_SYMBOL_GPL (cleanup_srcu_struct); +#ifdef CONFIG_PROVE_RCU /* * Check for consistent NMI safety. */ -static void srcu_check_nmi_safety (struct srcu_struct *ssp,bool nmi_safe) +void srcu_check_nmi_safety (struct srcu_struct *ssp,bool nmi_safe) {int nmi_safe_mask =1 << nmi_safe;int old_nmi_safe_mask;struct srcu_data *sdp; -if (!IS_ENABLED (CONFIG_PROVE_RCU )) -return ;/* NMI-unsafe use in NMI is a bad sign */ WARN_ON_ONCE (!nmi_safe &&in_nmi ()); sdp =raw_cpu_ptr (ssp->sda);
@@ -652,6 +651,7 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) }WARN_ONCE (old_nmi_safe_mask != nmi_safe_mask,"CPU %d old state %d new state %d\n" , sdp->cpu, old_nmi_safe_mask, nmi_safe_mask); } +#endif /* CONFIG_PROVE_RCU */ /* * Counts the new reader in the appropriate per-CPU element of the
@@ -665,7 +665,6 @@ int __srcu_read_lock(struct srcu_struct *ssp) idx =READ_ONCE (ssp->srcu_idx) &0x1 ;this_cpu_inc (ssp->sda->srcu_lock_count[idx].counter);smp_mb ();/* B */ /* Avoid leaking the critical section. */ -srcu_check_nmi_safety (ssp, false);return idx; }EXPORT_SYMBOL_GPL (__srcu_read_lock);
@@ -679,7 +678,6 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) {smp_mb ();/* C */ /* Avoid leaking the critical section. */ this_cpu_inc (ssp->sda->srcu_unlock_count[idx].counter); -srcu_check_nmi_safety (ssp, false); }EXPORT_SYMBOL_GPL (__srcu_read_unlock);
@@ -688,7 +686,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);* srcu_struct, but in an NMI-safe manner using RMW atomics. * Returns an index that must be passed to the matching srcu_read_unlock(). */ -int __srcu_read_lock_nmisafe (struct srcu_struct *ssp,bool chknmisafe) +int __srcu_read_lock_nmisafe (struct srcu_struct *ssp) {int idx;struct srcu_data *sdp =raw_cpu_ptr (ssp->sda);
@@ -696,8 +694,6 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) idx =READ_ONCE (ssp->srcu_idx) &0x1 ;atomic_long_inc (&sdp->srcu_lock_count[idx]);smp_mb__after_atomic ();/* B */ /* Avoid leaking the critical section. */ -if (chknmisafe) -srcu_check_nmi_safety (ssp, true);return idx; }EXPORT_SYMBOL_GPL (__srcu_read_lock_nmisafe);
@@ -707,14 +703,12 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe);* element of the srcu_struct. Note that this may well be a different * CPU than that which was incremented by the corresponding srcu_read_lock(). */ -void __srcu_read_unlock_nmisafe (struct srcu_struct *ssp,int idx,bool chknmisafe) +void __srcu_read_unlock_nmisafe (struct srcu_struct *ssp,int idx) {struct srcu_data *sdp =raw_cpu_ptr (ssp->sda);smp_mb__before_atomic ();/* C */ /* Avoid leaking the critical section. */ atomic_long_inc (&sdp->srcu_unlock_count[idx]); -if (chknmisafe) -srcu_check_nmi_safety (ssp, true); }EXPORT_SYMBOL_GPL (__srcu_read_unlock_nmisafe);
@@ -1159,7 +1153,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,* side critical section so that the gp sequence can't wrap around in * the middle. */ - idx =__srcu_read_lock_nmisafe (ssp, false); + idx =__srcu_read_lock_nmisafe (ssp); ss_state =smp_load_acquire (&ssp->srcu_size_state);if (ss_state <SRCU_SIZE_WAIT_CALL ) sdp =per_cpu_ptr (ssp->sda,0 );
@@ -1192,7 +1186,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,srcu_funnel_gp_start (ssp, sdp, s, do_norm); else if (needexp) srcu_funnel_exp_start(ssp, sdp_mynode, s); -__srcu_read_unlock_nmisafe (ssp ,idx ,false ); +__srcu_read_unlock_nmisafe (ssp ,idx );return s; }
@@ -1496,13 +1490,13 @@ void srcu_barrier(struct srcu_struct *ssp)/* Initial count prevents reaching zero until all CBs are posted. */ atomic_set (&ssp->srcu_barrier_cpu_cnt,1 ); - idx =__srcu_read_lock_nmisafe (ssp, false); + idx =__srcu_read_lock_nmisafe (ssp);if (smp_load_acquire (&ssp->srcu_size_state) <SRCU_SIZE_WAIT_BARRIER )srcu_barrier_one_cpu (ssp,per_cpu_ptr (ssp->sda,0 ));else for_each_possible_cpu (cpu)srcu_barrier_one_cpu (ssp,per_cpu_ptr (ssp->sda, cpu)); -__srcu_read_unlock_nmisafe (ssp, idx, false); +__srcu_read_unlock_nmisafe (ssp, idx);/* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test (&ssp->srcu_barrier_cpu_cnt))
Signature · 1 line
2.25.1
Paul E. McKenney
maintainerQuoted 8 lines · Frederic Weisbecker
Hi,
This has passed SRCU-N, SRCU-P and SRCU-T so far.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
srcu/nmisafeHEAD:
3cfdc2c6b8e89ca3c33954ea9b0d69e8cd141412
Thank you, Frederic!
I have pulled this is for review and testing.
Thanx, Paul
Quoted 15 lines
Thanks,
Frederic
---Frederic Weisbecker (3):
srcu: Warn when NMI-unsafe API is used in NMI
srcu: Explain the reason behind the read side critical section on GP start
srcu: Debug NMI safety even on archs that don't require itinclude/linux/srcu.h | 44 ++++++++++++++++++++++++++++++++++---------- include/linux/srcutiny.h | 12 ------------ include/linux/srcutree.h | 7 ------- kernel/rcu/srcutree.c | 31 ++++++++++++++++--------------- 4 files changed, 50 insertions(+), 44 deletions(-)