SRCU for v6.13
Subsystems (7)
Frederic Weisbecker
maintainercover SRCU for v6.13
Hello,
Please find below the SRCU patches targeted for the upcoming merge window.
Paul E. McKenney (15):
srcu: Rename srcu_might_be_idle() to srcu_should_expedite()
srcu: Introduce srcu_gp_is_expedited() helper function
srcu: Renaming in preparation for additional reader flavor
srcu: Bit manipulation changes for additional reader flavor
srcu: Standardize srcu_data pointers to "sdp" and similar
srcu: Improve srcu_read_lock{,_nmisafe}() comments
srcu: Create CPP macros for normal and NMI-safe SRCU readers
srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()
srcu: Allow inlining of __srcu_read_{,un}lock_lite()
rcutorture: Expand RCUTORTURE_RDR_MASK_[12] to eight bits
rcutorture: Add reader_flavor parameter for SRCU readers
rcutorture: Add srcu_read_lock_lite() support to
rcutorture.reader_flavor
rcutorture: Add light-weight SRCU scenario
refscale: Add srcu_read_lock_lite() support using "srcu-lite"
srcu: Improve srcu_read_lock_lite() kernel-doc comment
Zhen Lei (1):
srcu: Replace WARN_ON_ONCE() with BUILD_BUG_ON() if possible
.../admin-guide/kernel-parameters.txt | 8 ++ include/linux/srcu.h | 90 ++++++++++--- include/linux/srcutree.h | 46 ++++++- kernel/rcu/rcutorture.c | 65 +++++---- kernel/rcu/refscale.c | 37 +++++- kernel/rcu/srcutree.c | 123 +++++++++++------- .../selftests/rcutorture/configs/rcu/CFLIST | 1 + .../selftests/rcutorture/configs/rcu/SRCU-L | 10 ++ .../rcutorture/configs/rcu/SRCU-L.boot | 3 + .../rcutorture/configs/rcu/SRCU-N.boot | 1 + .../selftests/rcutorture/configs/rcu/TREE10 | 2 +- 11 files changed, 295 insertions(+), 91 deletions(-) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot
Thanks.
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 1/16 srcu: Replace WARN_ON_ONCE() with BUILD_BUG_ON() if possible
From: Zhen Lei <[email protected]>
The value of ARRAY_SIZE() can be determined at compile time, so if both sides of the equation are ARRAY_SIZE(), using BUILD_BUG_ON() can help us catch the problem earlier.
While there are cases where unequal array sizes will work, there is no point in allowing them, so it makes more sense to force them to be equal using BUILD_BUG_ON().
--- kernel/rcu/srcutree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/srcutree.c +2 -2
@@ -128,7 +128,7 @@ static void init_srcu_struct_data(struct srcu_struct *ssp)* Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ -WARN_ON_ONCE (ARRAY_SIZE (sdp->srcu_lock_count) != +BUILD_BUG_ON (ARRAY_SIZE (sdp->srcu_lock_count) !=ARRAY_SIZE (sdp->srcu_unlock_count));for_each_possible_cpu (cpu) { sdp =per_cpu_ptr (ssp->sda, cpu);
@@ -187,7 +187,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags)/* Each pass through this loop initializes one srcu_node structure. */ srcu_for_each_node_breadth_first (ssp, snp) {spin_lock_init (&ACCESS_PRIVATE (snp, lock)); -WARN_ON_ONCE (ARRAY_SIZE (snp->srcu_have_cbs) != +BUILD_BUG_ON (ARRAY_SIZE (snp->srcu_have_cbs) !=ARRAY_SIZE (snp->srcu_data_have_cbs));for (i =0 ; i <ARRAY_SIZE (snp->srcu_have_cbs); i++) { snp->srcu_have_cbs[i] =SRCU_SNP_INIT_SEQ ;
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 2/16 srcu: Rename srcu_might_be_idle() to srcu_should_expedite()
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
SRCU auto-expedites grace periods that follow a sufficiently long idle period, and the srcu_might_be_idle() function is used to make this decision. However, the upcoming light-weight SRCU readers will not do auto-expediting because doing so would cause the grace-period machinery to invoke synchronize_rcu_expedited() twice, with IPIs all around. However, software-engineering considerations force this determination to remain in srcu_might_be_idle().
This commit therefore changes the name of srcu_might_be_idle() to srcu_should_expedite(), thus moving from what it currently does to why it does it, this latter being more future-proof.
--- kernel/rcu/srcutree.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
kernel/rcu/srcutree.c +9 -7
@@ -1139,7 +1139,8 @@ static void srcu_flip(struct srcu_struct *ssp) }/* -* If SRCU is likely idle, return true, otherwise return false. +* If SRCU is likely idle, in other words, the next SRCU grace period +* should be expedited, return true, otherwise return false. * * Note that it is OK for several current from-idle requests for a new * grace period from idle to specify expediting because they will all end
@@ -1159,7 +1160,7 @@ static void srcu_flip(struct srcu_struct *ssp)* negligible when amortized over that time period, and the extra latency * of a needlessly non-expedited grace period is similarly negligible. */ -static bool srcu_might_be_idle (struct srcu_struct *ssp) +static bool srcu_should_expedite (struct srcu_struct *ssp) {unsigned long curseq;unsigned long flags;
@@ -1469,14 +1470,15 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_expedited);* Implementation of these memory-ordering guarantees is similar to * that of synchronize_rcu(). * -* If SRCU is likely idle, expedite the first request. This semantic -* was provided by Classic SRCU, and is relied upon by its users, so TREE -* SRCU must also provide it. Note that detecting idleness is heuristic -* and subject to both false positives and negatives. +* If SRCU is likely idle as determined by srcu_should_expedite(), +* expedite the first request. This semantic was provided by Classic SRCU, +* and is relied upon by its users, so TREE SRCU must also provide it. +* Note that detecting idleness is heuristic and subject to both false +* positives and negatives. */ void synchronize_srcu (struct srcu_struct *ssp) { -if (srcu_might_be_idle (ssp) ||rcu_gp_is_expedited ()) +if (srcu_should_expedite (ssp) ||rcu_gp_is_expedited ())synchronize_srcu_expedited (ssp);else __synchronize_srcu (ssp, true);
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 3/16 srcu: Introduce srcu_gp_is_expedited() helper function
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
Even though the open-coded expressions usually fit on one line, this commit replaces them with a call to a new srcu_gp_is_expedited() helper function in order to improve readability.
--- kernel/rcu/srcutree.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/rcu/srcutree.c +12 -2
@@ -418,6 +418,16 @@ static void check_init_srcu_struct(struct srcu_struct *ssp)spin_unlock_irqrestore_rcu_node (ssp->srcu_sup, flags); } +/* +* Is the current or any upcoming grace period to be expedited? +*/ +static bool srcu_gp_is_expedited (struct srcu_struct *ssp) +{ +struct srcu_usage *sup = ssp->srcu_sup; + +return ULONG_CMP_LT (READ_ONCE (sup->srcu_gp_seq),READ_ONCE (sup->srcu_gp_seq_needed_exp)); +} +/* * Returns approximate total of the readers' ->srcu_lock_count[] values * for the rank of per-CPU counters specified by idx.
@@ -622,7 +632,7 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp)unsigned long jbase =SRCU_INTERVAL ;struct srcu_usage *sup = ssp->srcu_sup; -if (ULONG_CMP_LT (READ_ONCE (sup->srcu_gp_seq),READ_ONCE (sup->srcu_gp_seq_needed_exp))) +if (srcu_gp_is_expedited (ssp)) jbase =0 ;if (rcu_seq_state (READ_ONCE (sup->srcu_gp_seq))) { j = jiffies -1 ;
@@ -867,7 +877,7 @@ static void srcu_gp_end(struct srcu_struct *ssp)spin_lock_irq_rcu_node (sup); idx =rcu_seq_state (sup->srcu_gp_seq);WARN_ON_ONCE (idx !=SRCU_STATE_SCAN2 ); -if (ULONG_CMP_LT (READ_ONCE (sup->srcu_gp_seq),READ_ONCE (sup->srcu_gp_seq_needed_exp))) +if (srcu_gp_is_expedited (ssp)) cbdelay =0 ;WRITE_ONCE (sup->srcu_last_gp_end,ktime_get_mono_fast_ns ());
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 4/16 srcu: Renaming in preparation for additional reader flavor
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
Currently, there are only two flavors of readers, normal and NMI-safe. A number of fields, functions, and types reflect this restriction. This renaming-only commit prepares for the addition of light-weight (as in memory-barrier-free) readers. OK, OK, there is also a drive-by white-space fixeup!
--- include/linux/srcu.h | 21 ++++++++++----------- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 22 +++++++++++----------- 3 files changed, 22 insertions(+), 23 deletions(-)
include/linux/srcu.h +10 -11
@@ -181,10 +181,9 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp)#define SRCU_NMI_SAFE 0x2#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_TREE_SRCU ) -void srcu_check_nmi_safety (struct srcu_struct *ssp,bool nmi_safe); +void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor);#else -static inline void srcu_check_nmi_safety (struct srcu_struct *ssp, -bool nmi_safe) { } +static inline void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor) { }#endif
@@ -245,7 +244,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) {int retval; -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false); retval =__srcu_read_lock (ssp);srcu_lock_acquire (&ssp->dep_map);return retval;
@@ -262,7 +261,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp {int retval; -srcu_check_nmi_safety (ssp, true); +srcu_check_read_flavor (ssp, true); retval =__srcu_read_lock_nmisafe (ssp);rcu_try_lock_acquire (&ssp->dep_map);return retval;
@@ -274,7 +273,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) {int retval; -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false); retval =__srcu_read_lock (ssp);return retval; }
@@ -303,7 +302,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp)static inline int srcu_down_read (struct srcu_struct *ssp)__acquires (ssp) {WARN_ON_ONCE (in_nmi ()); -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false);return __srcu_read_lock (ssp); }
@@ -318,7 +317,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false);srcu_lock_release (&ssp->dep_map);__srcu_read_unlock (ssp, idx); }
@@ -334,7 +333,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); -srcu_check_nmi_safety (ssp, true); +srcu_check_read_flavor (ssp, true);rcu_lock_release (&ssp->dep_map);__srcu_read_unlock_nmisafe (ssp, idx); }
@@ -343,7 +342,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) static inline notrace voidsrcu_read_unlock_notrace (struct srcu_struct *ssp,int idx)__releases (ssp) { -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false);__srcu_read_unlock (ssp, idx); }
@@ -360,7 +359,7 @@ static inline void srcu_up_read(struct srcu_struct *ssp, int idx) {WARN_ON_ONCE (idx & ~0x1 );WARN_ON_ONCE (in_nmi ()); -srcu_check_nmi_safety (ssp, false); +srcu_check_read_flavor (ssp, false);__srcu_read_unlock (ssp, idx); }
include/linux/srcutree.h +1 -1
@@ -25,7 +25,7 @@ struct srcu_data {/* Read-side state. */ atomic_long_t srcu_lock_count[2 ];/* Locks per CPU. */ atomic_long_t srcu_unlock_count[2 ];/* Unlocks per CPU. */ -int srcu_nmi_safety;/* NMI-safe srcu_struct structure? */ +int srcu_reader_flavor;/* Reader flavor for srcu_struct structure? */ /* Update-side state. */ spinlock_t __private lock____cacheline_internodealigned_in_smp ;
kernel/rcu/srcutree.c +11 -11
@@ -460,7 +460,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) sum +=atomic_long_read (&cpuc->srcu_unlock_count[idx]);if (IS_ENABLED (CONFIG_PROVE_RCU )) - mask = mask |READ_ONCE (cpuc->srcu_nmi_safety); + mask = mask |READ_ONCE (cpuc->srcu_reader_flavor); }WARN_ONCE (IS_ENABLED (CONFIG_PROVE_RCU ) && (mask & (mask >>1 )),"Mixed NMI-safe readers for srcu_struct at %ps.\n" , ssp);
@@ -699,25 +699,25 @@ EXPORT_SYMBOL_GPL(cleanup_srcu_struct);#ifdef CONFIG_PROVE_RCU /* -* Check for consistent NMI safety. +* Check for consistent reader flavor. */ -void srcu_check_nmi_safety (struct srcu_struct *ssp,bool nmi_safe) +void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor) { -int nmi_safe_mask =1 << nmi_safe; -int old_nmi_safe_mask; +int reader_flavor_mask =1 << read_flavor; +int old_reader_flavor_mask;struct srcu_data *sdp;/* NMI-unsafe use in NMI is a bad sign */ -WARN_ON_ONCE (!nmi_safe &&in_nmi ()); +WARN_ON_ONCE (!read_flavor &&in_nmi ()); sdp =raw_cpu_ptr (ssp->sda); - old_nmi_safe_mask =READ_ONCE (sdp->srcu_nmi_safety); -if (!old_nmi_safe_mask) { -WRITE_ONCE (sdp->srcu_nmi_safety, nmi_safe_mask); + old_reader_flavor_mask =READ_ONCE (sdp->srcu_reader_flavor); +if (!old_reader_flavor_mask) { +WRITE_ONCE (sdp->srcu_reader_flavor, reader_flavor_mask);return ; } -WARN_ONCE (old_nmi_safe_mask != nmi_safe_mask,"CPU %d old state %d new state %d\n" , sdp->cpu, old_nmi_safe_mask, nmi_safe_mask); +WARN_ONCE (old_reader_flavor_mask != reader_flavor_mask,"CPU %d old state %d new state %d\n" , sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); } -EXPORT_SYMBOL_GPL (srcu_check_nmi_safety); +EXPORT_SYMBOL_GPL (srcu_check_read_flavor);#endif /* CONFIG_PROVE_RCU */ /*
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 5/16 srcu: Bit manipulation changes for additional reader flavor
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
Currently, there are only two flavors of readers, normal and NMI-safe. Very straightforward state updates suffice to check for erroneous mixing of reader flavors on a given srcu_struct structure. This commit upgrades the checking in preparation for the addition of light-weight (as in memory-barrier-free) readers.
--- kernel/rcu/srcutree.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/rcu/srcutree.c +4 -3
@@ -462,7 +462,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx)if (IS_ENABLED (CONFIG_PROVE_RCU )) mask = mask |READ_ONCE (cpuc->srcu_reader_flavor); } -WARN_ONCE (IS_ENABLED (CONFIG_PROVE_RCU ) && (mask & (mask >>1 )), +WARN_ONCE (IS_ENABLED (CONFIG_PROVE_RCU ) && (mask & (mask -1 )),"Mixed NMI-safe readers for srcu_struct at %ps.\n" , ssp);return sum; }
@@ -712,8 +712,9 @@ void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) sdp =raw_cpu_ptr (ssp->sda); old_reader_flavor_mask =READ_ONCE (sdp->srcu_reader_flavor);if (!old_reader_flavor_mask) { -WRITE_ONCE (sdp->srcu_reader_flavor, reader_flavor_mask); -return ; + old_reader_flavor_mask =cmpxchg (&sdp->srcu_reader_flavor,0 , reader_flavor_mask); +if (!old_reader_flavor_mask) +return ; }WARN_ONCE (old_reader_flavor_mask != reader_flavor_mask,"CPU %d old state %d new state %d\n" , sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); }
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 6/16 srcu: Standardize srcu_data pointers to "sdp" and similar
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit changes a few "cpuc" variables to "sdp" to align with usage elsewhere.
[ paulmck: Apply Neeraj Upadhyay feedback. ]
--- kernel/rcu/srcutree.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
kernel/rcu/srcutree.c +10 -10
@@ -438,9 +438,9 @@ static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx)unsigned long sum =0 ;for_each_possible_cpu (cpu) { -struct srcu_data *cpuc =per_cpu_ptr (ssp->sda, cpu); +struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); - sum +=atomic_long_read (&cpuc->srcu_lock_count[idx]); + sum +=atomic_long_read (&sdp->srcu_lock_count[idx]); }return sum; }
@@ -456,11 +456,11 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx)unsigned long sum =0 ;for_each_possible_cpu (cpu) { -struct srcu_data *cpuc =per_cpu_ptr (ssp->sda, cpu); +struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); - sum +=atomic_long_read (&cpuc->srcu_unlock_count[idx]); + sum +=atomic_long_read (&sdp->srcu_unlock_count[idx]);if (IS_ENABLED (CONFIG_PROVE_RCU )) - mask = mask |READ_ONCE (cpuc->srcu_reader_flavor); + mask = mask |READ_ONCE (sdp->srcu_reader_flavor); }WARN_ONCE (IS_ENABLED (CONFIG_PROVE_RCU ) && (mask & (mask -1 )),"Mixed NMI-safe readers for srcu_struct at %ps.\n" , ssp);
@@ -564,12 +564,12 @@ static bool srcu_readers_active(struct srcu_struct *ssp)unsigned long sum =0 ;for_each_possible_cpu (cpu) { -struct srcu_data *cpuc =per_cpu_ptr (ssp->sda, cpu); +struct srcu_data *sdp =per_cpu_ptr (ssp->sda, cpu); - sum +=atomic_long_read (&cpuc->srcu_lock_count[0 ]); - sum +=atomic_long_read (&cpuc->srcu_lock_count[1 ]); - sum -=atomic_long_read (&cpuc->srcu_unlock_count[0 ]); - sum -=atomic_long_read (&cpuc->srcu_unlock_count[1 ]); + sum +=atomic_long_read (&sdp->srcu_lock_count[0 ]); + sum +=atomic_long_read (&sdp->srcu_lock_count[1 ]); + sum -=atomic_long_read (&sdp->srcu_unlock_count[0 ]); + sum -=atomic_long_read (&sdp->srcu_unlock_count[1 ]); }return sum; }
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 7/16 srcu: Improve srcu_read_lock{,_nmisafe}() comments
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit adds some additional usage constraints to the kernel-doc headers of srcu_read_lock() and srcu_read_lock_nmi_safe().
--- include/linux/srcu.h | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
include/linux/srcu.h +11 -4
@@ -235,10 +235,13 @@ static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flav* a mutex that is held elsewhere while calling synchronize_srcu() or * synchronize_srcu_expedited(). * -* Note that srcu_read_lock() and the matching srcu_read_unlock() must -* occur in the same context, for example, it is illegal to invoke -* srcu_read_unlock() in an irq handler if the matching srcu_read_lock() -* was invoked in process context. +* The return value from srcu_read_lock() must be passed unaltered +* to the matching srcu_read_unlock(). Note that srcu_read_lock() and +* the matching srcu_read_unlock() must occur in the same context, for +* example, it is illegal to invoke srcu_read_unlock() in an irq handler +* if the matching srcu_read_lock() was invoked in process context. Or, +* for that matter to invoke srcu_read_unlock() from one task and the +* matching srcu_read_lock() from another. */ static inline int srcu_read_lock (struct srcu_struct *ssp)__acquires (ssp) {
@@ -256,6 +259,10 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp)* * Enter an SRCU read-side critical section, but in an NMI-safe manner. * See srcu_read_lock() for more information. +* +* If srcu_read_lock_nmisafe() is ever used on an srcu_struct structure, +* then none of the other flavors may be used, whether before, during, +* or after. */ static inline int srcu_read_lock_nmisafe (struct srcu_struct *ssp)__acquires (ssp) {
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 8/16 srcu: Create CPP macros for normal and NMI-safe SRCU readers
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit creates SRCU_READ_FLAVOR_NORMAL and SRCU_READ_FLAVOR_NMI C-preprocessor macros for srcu_read_lock() and srcu_read_lock_nmisafe(), respectively. These replace the old true/false values that were previously passed to srcu_check_read_flavor(). In addition, the srcu_check_read_flavor() function itself requires a bit of rework to handle bitmasks instead of true/false values.
--- include/linux/srcu.h | 20 ++++++++------------ include/linux/srcutree.h | 4 ++++ kernel/rcu/srcutree.c | 21 +++++++++++---------- 3 files changed, 23 insertions(+), 22 deletions(-)
include/linux/srcu.h +8 -12
@@ -176,10 +176,6 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp)#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ -#define SRCU_NMI_UNKNOWN 0x0 -#define SRCU_NMI_UNSAFE 0x1 -#define SRCU_NMI_SAFE 0x2 -#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_TREE_SRCU )void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor);#else
@@ -247,7 +243,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) {int retval; -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL ); retval =__srcu_read_lock (ssp);srcu_lock_acquire (&ssp->dep_map);return retval;
@@ -268,7 +264,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp {int retval; -srcu_check_read_flavor (ssp, true); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NMI ); retval =__srcu_read_lock_nmisafe (ssp);rcu_try_lock_acquire (&ssp->dep_map);return retval;
@@ -280,7 +276,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) {int retval; -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL ); retval =__srcu_read_lock (ssp);return retval; }
@@ -309,7 +305,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp)static inline int srcu_down_read (struct srcu_struct *ssp)__acquires (ssp) {WARN_ON_ONCE (in_nmi ()); -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL );return __srcu_read_lock (ssp); }
@@ -324,7 +320,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL );srcu_lock_release (&ssp->dep_map);__srcu_read_unlock (ssp, idx); }
@@ -340,7 +336,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)__releases (ssp) {WARN_ON_ONCE (idx & ~0x1 ); -srcu_check_read_flavor (ssp, true); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NMI );rcu_lock_release (&ssp->dep_map);__srcu_read_unlock_nmisafe (ssp, idx); }
@@ -349,7 +345,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) static inline notrace voidsrcu_read_unlock_notrace (struct srcu_struct *ssp,int idx)__releases (ssp) { -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL );__srcu_read_unlock (ssp, idx); }
@@ -366,7 +362,7 @@ static inline void srcu_up_read(struct srcu_struct *ssp, int idx) {WARN_ON_ONCE (idx & ~0x1 );WARN_ON_ONCE (in_nmi ()); -srcu_check_read_flavor (ssp, false); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_NORMAL );__srcu_read_unlock (ssp, idx); }
include/linux/srcutree.h +4 -0
@@ -43,6 +43,10 @@ struct srcu_data {struct srcu_struct *ssp; }; +/* Values for ->srcu_reader_flavor. */ +#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). +#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). +/* * Node in SRCU combining tree, similar in function to rcu_data. */
kernel/rcu/srcutree.c +11 -10
@@ -463,7 +463,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) mask = mask |READ_ONCE (sdp->srcu_reader_flavor); }WARN_ONCE (IS_ENABLED (CONFIG_PROVE_RCU ) && (mask & (mask -1 )), -"Mixed NMI-safe readers for srcu_struct at %ps.\n" , ssp); +"Mixed reader flavors for srcu_struct at %ps.\n" , ssp);return sum; }
@@ -703,20 +703,21 @@ EXPORT_SYMBOL_GPL(cleanup_srcu_struct);*/ void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor) { -int reader_flavor_mask =1 << read_flavor; -int old_reader_flavor_mask; +int old_read_flavor;struct srcu_data *sdp; -/* NMI-unsafe use in NMI is a bad sign */ -WARN_ON_ONCE (!read_flavor &&in_nmi ()); +/* NMI-unsafe use in NMI is a bad sign, as is multi-bit read_flavor values. */ +WARN_ON_ONCE ((read_flavor !=SRCU_READ_FLAVOR_NMI ) &&in_nmi ()); +WARN_ON_ONCE (read_flavor & (read_flavor -1 )); + sdp =raw_cpu_ptr (ssp->sda); - old_reader_flavor_mask =READ_ONCE (sdp->srcu_reader_flavor); -if (!old_reader_flavor_mask) { - old_reader_flavor_mask =cmpxchg (&sdp->srcu_reader_flavor,0 , reader_flavor_mask); -if (!old_reader_flavor_mask) + old_read_flavor =READ_ONCE (sdp->srcu_reader_flavor); +if (!old_read_flavor) { + old_read_flavor =cmpxchg (&sdp->srcu_reader_flavor,0 , read_flavor); +if (!old_read_flavor)return ; } -WARN_ONCE (old_reader_flavor_mask != reader_flavor_mask,"CPU %d old state %d new state %d\n" , sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); +WARN_ONCE (old_read_flavor != read_flavor,"CPU %d old state %d new state %d\n" , sdp->cpu, old_read_flavor, read_flavor); }EXPORT_SYMBOL_GPL (srcu_check_read_flavor);#endif /* CONFIG_PROVE_RCU */
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 9/16 srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This patch adds srcu_read_lock_lite() and srcu_read_unlock_lite(), which dispense with the read-side smp_mb() but also are restricted to code regions that RCU is watching. If a given srcu_struct structure uses srcu_read_lock_lite() and srcu_read_unlock_lite(), it is not permitted to use any other SRCU read-side marker, before, during, or after.
Another price of light-weight readers is heavier weight grace periods. Such readers mean that SRCU grace periods on srcu_struct structures used by light-weight readers will incur at least two calls to synchronize_rcu(). In addition, normal SRCU grace periods for light-weight-reader srcu_struct structures never auto-expedite. Note that expedited SRCU grace periods for light-weight-reader srcu_struct structures still invoke synchronize_rcu(), not synchronize_srcu_expedited(). Something about wishing to keep the IPIs down to a dull roar.
The srcu_read_lock_lite() and srcu_read_unlock_lite() functions may not (repeat, *not*) be used from NMI handlers, but if this is needed, an additional flavor of SRCU reader can be added by some future commit.
[ paulmck: Apply Alexei Starovoitov expediting feedback. ] [ paulmck: Apply kernel test robot feedback. ]
--- include/linux/srcu.h | 51 ++++++++++++++++++++++++- include/linux/srcutree.h | 1 + kernel/rcu/srcutree.c | 82 ++++++++++++++++++++++++++++++++++------ 3 files changed, 122 insertions(+), 12 deletions(-)
include/linux/srcu.h +50 -1
@@ -56,6 +56,13 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,void cleanup_srcu_struct (struct srcu_struct *ssp); int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); +#ifdef CONFIG_TINY_SRCU +#define __srcu_read_lock_lite __srcu_read_lock +#define __srcu_read_unlock_lite __srcu_read_unlock +#else // #ifdef CONFIG_TINY_SRCU +int __srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp); +void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) __releases(ssp); +#endif // #else // #ifdef CONFIG_TINY_SRCU void synchronize_srcu (struct srcu_struct *ssp);#define SRCU_GET_STATE_COMPLETED 0x1
@@ -179,7 +186,7 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp)#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_TREE_SRCU )void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor);#else -static inline void srcu_check_read_flavor (struct srcu_struct *ssp,int read_flavor) { } +#define srcu_check_read_flavor (ssp, read_flavor) do { } while (0)#endif
@@ -249,6 +256,32 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp)return retval; } +/** +* srcu_read_lock_lite - register a new reader for an SRCU-protected structure. +* @ssp: srcu_struct in which to register the new reader. +* +* Enter an SRCU read-side critical section, but for a light-weight +* smp_mb()-free reader. See srcu_read_lock() for more information. +* +* If srcu_read_lock_lite() is ever used on an srcu_struct structure, +* then none of the other flavors may be used, whether before, during, +* or after. Note that grace-period auto-expediting is disabled for _lite +* srcu_struct structures because auto-expedited grace periods invoke +* synchronize_rcu_expedited(), IPIs and all. +* +* Note that srcu_read_lock_lite() can be invoked only from those contexts +* where RCU is watching. Otherwise, lockdep will complain. +*/ +static inline int srcu_read_lock_lite (struct srcu_struct *ssp)__acquires (ssp) +{ +int retval; + +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_LITE ); + retval =__srcu_read_lock_lite (ssp); +rcu_try_lock_acquire (&ssp->dep_map); +return retval; +} +/** * srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure. * @ssp: srcu_struct in which to register the new reader.
@@ -325,6 +358,22 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)__srcu_read_unlock (ssp, idx); } +/** +* srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure. +* @ssp: srcu_struct in which to unregister the old reader. +* @idx: return value from corresponding srcu_read_lock(). +* +* Exit a light-weight SRCU read-side critical section. +*/ +static inline void srcu_read_unlock_lite (struct srcu_struct *ssp,int idx) +__releases (ssp) +{ +WARN_ON_ONCE (idx & ~0x1 ); +srcu_check_read_flavor (ssp,SRCU_READ_FLAVOR_LITE ); +srcu_lock_release (&ssp->dep_map); +__srcu_read_unlock (ssp, idx); +} +/** * srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader.
include/linux/srcutree.h +1 -0
@@ -46,6 +46,7 @@ struct srcu_data {/* Values for ->srcu_reader_flavor. */ #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). +#define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite()./* * Node in SRCU combining tree, similar in function to rcu_data.
kernel/rcu/srcutree.c +71 -11
142 lines not on this page · raw message
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 10/16 srcu: Allow inlining of __srcu_read_{,un}lock_lite()
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit moves __srcu_read_lock_lite() and __srcu_read_unlock_lite() into include/linux/srcu.h and marks them "static inline" so that they can be inlined into srcu_read_lock_lite() and srcu_read_unlock_lite(), respectively. They are not hand-inlined due to Tree SRCU and Tiny SRCU having different implementations.
The earlier removal of smp_mb() combined with the inlining produce significant single-percentage performance wins.
--- include/linux/srcutree.h | 39 ++++++++++++++++++++++++++++++++++++++ kernel/rcu/srcutree.c | 41 ---------------------------------------- 2 files changed, 39 insertions(+), 41 deletions(-)
include/linux/srcutree.h +39 -0
@@ -209,4 +209,43 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp);void srcu_barrier (struct srcu_struct *ssp);void srcu_torture_stats_print (struct srcu_struct *ssp,char *tt,char *tf); +/* +* Counts the new reader in the appropriate per-CPU element of the +* srcu_struct. Returns an index that must be passed to the matching +* srcu_read_unlock_lite(). +* +* Note that this_cpu_inc() is an RCU read-side critical section either +* because it disables interrupts, because it is a single instruction, +* or because it is a read-modify-write atomic operation, depending on +* the whims of the architecture. +*/ +static inline int __srcu_read_lock_lite (struct srcu_struct *ssp) +{ +int idx; + +RCU_LOCKDEP_WARN (!rcu_is_watching (),"RCU must be watching srcu_read_lock_lite()." ); + idx =READ_ONCE (ssp->srcu_idx) &0x1 ; +this_cpu_inc (ssp->sda->srcu_lock_count[idx].counter);/* Y */ +barrier ();/* Avoid leaking the critical section. */ +return idx; +} + +/* +* Removes the count for the old reader from the appropriate +* per-CPU element of the srcu_struct. Note that this may well be a +* different CPU than that which was incremented by the corresponding +* srcu_read_lock_lite(), but it must be within the same task. +* +* Note that this_cpu_inc() is an RCU read-side critical section either +* because it disables interrupts, because it is a single instruction, +* or because it is a read-modify-write atomic operation, depending on +* the whims of the architecture. +*/ +static inline void __srcu_read_unlock_lite (struct srcu_struct *ssp,int idx) +{ +barrier ();/* Avoid leaking the critical section. */ +this_cpu_inc (ssp->sda->srcu_unlock_count[idx].counter);/* Z */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"RCU must be watching srcu_read_unlock_lite()." ); +} +#endif
kernel/rcu/srcutree.c +0 -41
47 lines not on this page · raw message
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 11/16 rcutorture: Expand RCUTORTURE_RDR_MASK_[12] to eight bits
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit prepares for testing of multiple SRCU reader flavors by expanding RCUTORTURE_RDR_MASK_1 and RCUTORTURE_RDR_MASK_2 from a single bit to eight bits, allowing them to accommodate the return values from multiple calls to srcu_read_lock*(). This will in turn permit better testing coverage for these SRCU reader flavors, including testing of the diagnostics for inproper use of mixed reader flavors.
--- kernel/rcu/rcutorture.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/rcu/rcutorture.c +14 -14
@@ -57,9 +57,9 @@ MODULE_AUTHOR("Paul E. McKenney <[email protected]> and Josh Triplett <josh@/* Bits for ->extendables field, extendables param, and related definitions. */ #define RCUTORTURE_RDR_SHIFT_1 8/* Put SRCU index in upper bits. */ -#define RCUTORTURE_RDR_MASK_1 (1 << RCUTORTURE_RDR_SHIFT_1) -#define RCUTORTURE_RDR_SHIFT_2 9/* Put SRCU index in upper bits. */ -#define RCUTORTURE_RDR_MASK_2 (1 << RCUTORTURE_RDR_SHIFT_2) +#define RCUTORTURE_RDR_MASK_1 (0xff << RCUTORTURE_RDR_SHIFT_1) +#define RCUTORTURE_RDR_SHIFT_2 16/* Put SRCU index in upper bits. */ +#define RCUTORTURE_RDR_MASK_2 (0xff << RCUTORTURE_RDR_SHIFT_2)#define RCUTORTURE_RDR_BH 0x01/* Extend readers by disabling bh. */ #define RCUTORTURE_RDR_IRQ 0x02/* ... disabling interrupts. */ #define RCUTORTURE_RDR_PREEMPT 0x04/* ... disabling preemption. */
@@ -71,6 +71,9 @@ MODULE_AUTHOR("Paul E. McKenney <[email protected]> and Josh Triplett <josh@#define RCUTORTURE_MAX_EXTEND \ (RCUTORTURE_RDR_BH | RCUTORTURE_RDR_IRQ | RCUTORTURE_RDR_PREEMPT | \ RCUTORTURE_RDR_RBH | RCUTORTURE_RDR_SCHED) +#define RCUTORTURE_RDR_ALLBITS \ + (RCUTORTURE_MAX_EXTEND | RCUTORTURE_RDR_RCU_1 | RCUTORTURE_RDR_RCU_2 | \ + RCUTORTURE_RDR_MASK_1 | RCUTORTURE_RDR_MASK_2)#define RCUTORTURE_RDR_MAX_LOOPS 0x7/* Maximum reader extensions. */ /* Must be power of two minus one. */ #define RCUTORTURE_RDR_MAX_SEGS (RCUTORTURE_RDR_MAX_LOOPS + 3)
@@ -1820,7 +1823,7 @@ static void rcutorture_one_extend(int *readstate, int newstate,int statesold = *readstate & ~newstate;WARN_ON_ONCE (idxold2 <0 ); -WARN_ON_ONCE ((idxold2 >>RCUTORTURE_RDR_SHIFT_2 ) >1 ); +WARN_ON_ONCE (idxold2 & ~RCUTORTURE_RDR_ALLBITS ); rtrsp->rt_readstate = newstate;/* First, put new protection in place to avoid critical-section gap. */
@@ -1835,9 +1838,9 @@ static void rcutorture_one_extend(int *readstate, int newstate,if (statesnew &RCUTORTURE_RDR_SCHED )rcu_read_lock_sched ();if (statesnew &RCUTORTURE_RDR_RCU_1 ) - idxnew1 = (cur_ops->readlock () &0x1 ) <<RCUTORTURE_RDR_SHIFT_1 ; + idxnew1 = (cur_ops->readlock () <<RCUTORTURE_RDR_SHIFT_1 ) &RCUTORTURE_RDR_MASK_1 ;if (statesnew &RCUTORTURE_RDR_RCU_2 ) - idxnew2 = (cur_ops->readlock () &0x1 ) <<RCUTORTURE_RDR_SHIFT_2 ; + idxnew2 = (cur_ops->readlock () <<RCUTORTURE_RDR_SHIFT_2 ) &RCUTORTURE_RDR_MASK_2 ;/* * Next, remove old protection, in decreasing order of strength
@@ -1857,7 +1860,7 @@ static void rcutorture_one_extend(int *readstate, int newstate,if (statesold &RCUTORTURE_RDR_RBH )rcu_read_unlock_bh ();if (statesold &RCUTORTURE_RDR_RCU_2 ) { - cur_ops->readunlock ((idxold2 >>RCUTORTURE_RDR_SHIFT_2 ) &0x1 ); + cur_ops->readunlock ((idxold2 &RCUTORTURE_RDR_MASK_2 ) >>RCUTORTURE_RDR_SHIFT_2 );WARN_ON_ONCE (idxnew2 !=-1 ); idxold2 =0 ; }
@@ -1867,7 +1870,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, lockit = !cur_ops->no_pi_lock && !statesnew && !(torture_random (trsp) &0xffff );if (lockit)raw_spin_lock_irqsave (¤t->pi_lock, flags); - cur_ops->readunlock ((idxold1 >>RCUTORTURE_RDR_SHIFT_1 ) &0x1 ); + cur_ops->readunlock ((idxold1 &RCUTORTURE_RDR_MASK_1 ) >>RCUTORTURE_RDR_SHIFT_1 );WARN_ON_ONCE (idxnew1 !=-1 ); idxold1 =0 ;if (lockit)
@@ -1882,16 +1885,13 @@ static void rcutorture_one_extend(int *readstate, int newstate,if (idxnew1 ==-1 ) idxnew1 = idxold1 &RCUTORTURE_RDR_MASK_1 ;WARN_ON_ONCE (idxnew1 <0 ); -if (WARN_ON_ONCE ((idxnew1 >>RCUTORTURE_RDR_SHIFT_1 ) >1 )) -pr_info ("Unexpected idxnew1 value of %#x\n" , idxnew1);if (idxnew2 ==-1 ) idxnew2 = idxold2 &RCUTORTURE_RDR_MASK_2 ;WARN_ON_ONCE (idxnew2 <0 ); -WARN_ON_ONCE ((idxnew2 >>RCUTORTURE_RDR_SHIFT_2 ) >1 ); *readstate = idxnew1 | idxnew2 | newstate;WARN_ON_ONCE (*readstate <0 ); -if (WARN_ON_ONCE ((*readstate >>RCUTORTURE_RDR_SHIFT_2 ) >1 )) -pr_info ("Unexpected idxnew2 value of %#x\n" , idxnew2); +if (WARN_ON_ONCE (*readstate & ~RCUTORTURE_RDR_ALLBITS )) +pr_info ("Unexpected readstate value of %#x\n" , *readstate); }/* Return the biggest extendables mask given current RCU and boot parameters. */
@@ -1916,7 +1916,7 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp)unsigned long preempts_irq = preempts |RCUTORTURE_RDR_IRQ ;unsigned long bhs =RCUTORTURE_RDR_BH |RCUTORTURE_RDR_RBH ; -WARN_ON_ONCE (mask >>RCUTORTURE_RDR_SHIFT_1 ); +WARN_ON_ONCE (mask >>RCUTORTURE_RDR_SHIFT_1 );// Can't have reader idx bits. /* Mostly only one bit (need preemption!), sometimes lots of bits. */ if (!(randmask1 &0x7 )) mask = mask & randmask2;
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 12/16 rcutorture: Add reader_flavor parameter for SRCU readers
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit adds an rcutorture.reader_flavor parameter whose bits correspond to reader flavors. For example, SRCU's readers are 0x1 for normal and 0x2 for NMI-safe.
--- .../admin-guide/kernel-parameters.txt | 8 +++++ kernel/rcu/rcutorture.c | 30 ++++++++++++++----- 2 files changed, 30 insertions(+), 8 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -5426,6 +5426,14 @@ The delay, in seconds, between successive read-then-exit testing episodes. + rcutorture.reader_flavor= [KNL] + A bit mask indicating which readers to use. + If there is more than one bit set, the readers + are entered from low-order bit up, and are + exited in the opposite order. For SRCU, the + 0x1 bit is normal readers and the 0x2 bit is + for NMI-safe readers. + rcutorture.shuffle_interval= [KNL] Set task-shuffle interval (s). Shuffling tasks allows some CPUs to go into dyntick-idle mode
kernel/rcu/rcutorture.c +22 -8
@@ -111,6 +111,7 @@ torture_param(int, nocbs_nthreads, 0, "Number of NOCB toggle threads, 0 to disabtorture_param (int, nocbs_toggle,1000 ,"Time between toggling nocb state (ms)" );torture_param (int, read_exit_delay,13 ,"Delay between read-then-exit episodes (s)" );torture_param (int, read_exit_burst,16 ,"# of read-then-exit bursts per episode, zero to disable" ); +torture_param (int, reader_flavor,0x1 ,"Reader flavors to use, one per bit." );torture_param (int, shuffle_interval,3 ,"Number of seconds between shuffles" );torture_param (int, shutdown_secs,0 ,"Shutdown time (s), <= zero to disable." );torture_param (int, stall_cpu,0 ,"Stall duration (s), zero to disable." );
@@ -644,10 +645,20 @@ static void srcu_get_gp_data(int *flags, unsigned long *gp_seq)static int srcu_torture_read_lock (void ) { -if (cur_ops == &srcud_ops) -return srcu_read_lock_nmisafe (srcu_ctlp); -else -return srcu_read_lock (srcu_ctlp); +int idx; +int ret =0 ; + +if ((reader_flavor &0x1 ) || !(reader_flavor &0x7 )) { + idx =srcu_read_lock (srcu_ctlp); +WARN_ON_ONCE (idx & ~0x1 ); + ret += idx; + } +if (reader_flavor &0x2 ) { + idx =srcu_read_lock_nmisafe (srcu_ctlp); +WARN_ON_ONCE (idx & ~0x1 ); + ret += idx <<1 ; + } +return ret; } static void
@@ -671,10 +682,11 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)static void srcu_torture_read_unlock (int idx) { -if (cur_ops == &srcud_ops) -srcu_read_unlock_nmisafe (srcu_ctlp, idx); -else -srcu_read_unlock (srcu_ctlp, idx); +WARN_ON_ONCE ((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1 ))); +if (reader_flavor &0x2 ) +srcu_read_unlock_nmisafe (srcu_ctlp, (idx &0x2 ) >>1 ); +if ((reader_flavor &0x1 ) || !(reader_flavor &0x7 )) +srcu_read_unlock (srcu_ctlp, idx &0x1 ); }static int torture_srcu_read_lock_held (void )
@@ -2389,6 +2401,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag)"n_barrier_cbs=%d " "onoff_interval=%d onoff_holdoff=%d " "read_exit_delay=%d read_exit_burst=%d " +"reader_flavor=%x " "nocbs_nthreads=%d nocbs_toggle=%d " "test_nmis=%d\n" , torture_type, tag, nrealreaders, nfakewriters,
@@ -2401,6 +2414,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag) n_barrier_cbs, onoff_interval, onoff_holdoff, read_exit_delay, read_exit_burst, + reader_flavor, nocbs_nthreads, nocbs_toggle, test_nmis); }
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 13/16 rcutorture: Add srcu_read_lock_lite() support to rcutorture.reader_flavor
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit causes bit 0x4 of rcutorture.reader_flavor to select the new srcu_read_lock_lite() and srcu_read_unlock_lite() functions.
--- Documentation/admin-guide/kernel-parameters.txt | 4 ++-- kernel/rcu/rcutorture.c | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -2
@@ -5431,8 +5431,8 @@ If there is more than one bit set, the readers are entered from low-order bit up, and are exited in the opposite order. For SRCU, the - 0x1 bit is normal readers and the 0x2 bit is - for NMI-safe readers. + 0x1 bit is normal readers, 0x2 NMI-safe readers, + and 0x4 light-weight readers. rcutorture.shuffle_interval= [KNL] Set task-shuffle interval (s). Shuffling tasks
kernel/rcu/rcutorture.c +7 -0
@@ -658,6 +658,11 @@ static int srcu_torture_read_lock(void)WARN_ON_ONCE (idx & ~0x1 ); ret += idx <<1 ; } +if (reader_flavor &0x4 ) { + idx =srcu_read_lock_lite (srcu_ctlp); +WARN_ON_ONCE (idx & ~0x1 ); + ret += idx <<2 ; + }return ret; }
@@ -683,6 +688,8 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp)static void srcu_torture_read_unlock (int idx) {WARN_ON_ONCE ((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1 ))); +if (reader_flavor &0x4 ) +srcu_read_unlock_lite (srcu_ctlp, (idx &0x4 ) >>2 );if (reader_flavor &0x2 )srcu_read_unlock_nmisafe (srcu_ctlp, (idx &0x2 ) >>1 );if ((reader_flavor &0x1 ) || !(reader_flavor &0x7 ))
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 14/16 rcutorture: Add light-weight SRCU scenario
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit adds an rcutorture scenario that tests light-weight SRCU readers. While in the area, it adjusts the size of the TREE10 scenario.
--- tools/testing/selftests/rcutorture/configs/rcu/CFLIST | 1 + tools/testing/selftests/rcutorture/configs/rcu/SRCU-L | 10 ++++++++++ .../selftests/rcutorture/configs/rcu/SRCU-L.boot | 3 +++ .../selftests/rcutorture/configs/rcu/SRCU-N.boot | 1 + tools/testing/selftests/rcutorture/configs/rcu/TREE10 | 2 +- 5 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot
tools/testing/selftests/rcutorture/configs/rcu/CFLIST +1 -0
@@ -5,6 +5,7 @@ TREE04 TREE05 TREE07 TREE09 +SRCU-L SRCU-N SRCU-P SRCU-T
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L new file +10 -0
@@ -0,0 +1,10 @@ +CONFIG_RCU_TRACE=n +CONFIG_SMP=y +CONFIG_NR_CPUS=6 +CONFIG_HOTPLUG_CPU=y +CONFIG_PREEMPT_NONE=y +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=n +#CHECK#CONFIG_RCU_EXPERT=n +CONFIG_KPROBES=n +CONFIG_FTRACE=n
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot new file +3 -0
@@ -0,0 +1,3 @@ +rcutorture.torture_type=srcu +rcutorture.reader_flavor=0x4 +rcutorture.fwd_progress=3
tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot +1 -0
@@ -1,2 +1,3 @@ rcutorture.torture_type=srcu +rcutorture.reader_flavor=0x2 rcutorture.fwd_progress=3
tools/testing/selftests/rcutorture/configs/rcu/TREE10 +1 -1
@@ -1,5 +1,5 @@ CONFIG_SMP=y -CONFIG_NR_CPUS=56 +CONFIG_NR_CPUS=74 CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 15/16 refscale: Add srcu_read_lock_lite() support using "srcu-lite"
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit creates a new srcu-lite option for the refscale.scale_type module parameter that selects srcu_read_lock_lite() and srcu_read_unlock_lite().
[ paulmck: Apply Dan Carpenter feedback. ]
--- kernel/rcu/refscale.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)
kernel/rcu/refscale.c +34 -3
@@ -212,6 +212,36 @@ static const struct ref_scale_ops srcu_ops = { .name = "srcu" }; +static void srcu_lite_ref_scale_read_section (const int nloops) +{ +int i; +int idx; + +for (i = nloops; i >=0 ; i--) { + idx =srcu_read_lock_lite (srcu_ctlp); +srcu_read_unlock_lite (srcu_ctlp, idx); + } +} + +static void srcu_lite_ref_scale_delay_section (const int nloops,const int udl,const int ndl) +{ +int i; +int idx; + +for (i = nloops; i >=0 ; i--) { + idx =srcu_read_lock_lite (srcu_ctlp); +un_delay (udl, ndl); +srcu_read_unlock_lite (srcu_ctlp, idx); + } +} + +static const struct ref_scale_ops srcu_lite_ops = { + .init = rcu_sync_scale_init, + .readsection = srcu_lite_ref_scale_read_section, + .delaysection = srcu_lite_ref_scale_delay_section, + .name ="srcu-lite" +}; +#ifdef CONFIG_TASKS_RCU // Definitions for RCU Tasks ref scale testing: Empty read markers.
@@ -1082,9 +1112,10 @@ ref_scale_init(void)long i;int firsterr =0 ;static const struct ref_scale_ops *scale_ops[] = { - &rcu_ops, &srcu_ops, RCU_TRACE_OPS RCU_TASKS_OPS &refcnt_ops, &rwlock_ops, - &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &clock_ops, &jiffies_ops, - &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, + &rcu_ops, &srcu_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS + &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, + &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, + &typesafe_seqlock_ops, };if (!torture_init_begin (scale_type, verbose))
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 16/16 srcu: Improve srcu_read_lock_lite() kernel-doc comment
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
Where RCU is watching is where it is OK to invoke rcu_read_lock().
--- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/srcu.h +2 -1
@@ -270,7 +270,8 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp)* synchronize_rcu_expedited(), IPIs and all. * * Note that srcu_read_lock_lite() can be invoked only from those contexts -* where RCU is watching. Otherwise, lockdep will complain. +* where RCU is watching, that is, from contexts where it would be legal +* to invoke rcu_read_lock(). Otherwise, lockdep will complain. */ static inline int srcu_read_lock_lite (struct srcu_struct *ssp)__acquires (ssp) {
Signature · 1 line
2.46.0