rcu: Avoid unneeded function call in rcu_read_unlock()
PATCH 1/1 rcu: Avoid unneeded function call in rcu_read_unlock()
Since commit aa40c138cc8f3 ("rcu: Report QS for outermost PREEMPT=n rcu_read_unlock() for strict GPs"). A real function call rcu_read_unlock_strict() is added to the inlined rcu_read_unlock(). The rcu_read_unlock_strict() call is only needed if the performance sagging CONFIG_RCU_STRICT_GRACE_PERIOD option is set. This config option isn't set for most production kernels while the function call overhead remains.
To provide a slight performance improvement, the CONFIG_RCU_STRICT_GRACE_PERIOD config check is moved from rcu_read_unlock_strict() to __rcu_read_unlock() so that the function call can be compiled out in most cases.
Besides, the GPL exported rcu_read_unlock_strict() also impact the the compilation of non-GPL kernel modules as rcu_read_unlock() is a frequently used kernel API.
--- include/linux/rcupdate.h | 3 ++- kernel/rcu/tree_plugin.h | 3 +-- 2 files changed, 3 insertions(+), 3 deletions(-)
include/linux/rcupdate.h +2 -1
@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )
kernel/rcu/tree_plugin.h +1 -2
@@ -844,8 +844,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);
Signature · 1 line
2.18.1
Paul E. McKenney
maintainerQuoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.
Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>
Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the EXPORT_SYMBOL_GPL() discussion because:
1. Kernels built with CONFIG_PREEMPT=y have the same issue
with the __rcu_read_lock() and __rcu_read_unlock() functions.
2. Many other RCU functions are EXPORT_SYMBOL_GPL() and have
been for almost two decades.
But if someone does use RCU readers within CONFIG_PREEMPT=n kernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)
I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
Thanx, Paul
------------------------------------------------------------------------
commit 4a9f53b997b809c0256838e31c604aeeded2345a
Author: Waiman Long <[email protected]> Date: Thu Aug 26 22:21:22 2021 -0400
rcu: Avoid unneeded function call in rcu_read_unlock()
Since commit aa40c138cc8f3 ("rcu: Report QS for outermost PREEMPT=n rcu_read_unlock() for strict GPs") the function rcu_read_unlock_strict() is invoked by the inlined rcu_read_unlock() function. However, rcu_read_unlock_strict() is an empty function in production kernels, which are built with CONFIG_RCU_STRICT_GRACE_PERIOD=n.
There is a mention of rcu_read_unlock_strict() in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whether rcu_read_unlock_strict() is ever called.
This commit therefore provides a slight performance improvement by hoisting the check of CONFIG_RCU_STRICT_GRACE_PERIOD from rcu_read_unlock_strict() into rcu_read_unlock(), thus avoiding the pointless call to an empty function.
include/linux/rcupdate.h +2 -1
@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )
kernel/rcu/tree_plugin.h +1 -2
@@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);
Quoted 34 lines · Paul E. McKenney
Quoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the
EXPORT_SYMBOL_GPL()discussion because:1. Kernels built with
CONFIG_PREEMPT=yhave the same issue
with the__rcu_read_lock()and__rcu_read_unlock()functions.2. Many other RCU functions are
EXPORT_SYMBOL_GPL()and have
been for almost two decades.
Quoted 9 lines
But if someone does use RCU readers within
CONFIG_PREEMPT=nkernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
Thanx, Paul
It looks good to me. Thanks for the rewording. I did regret mentioning about about the GPL export symbol in the commit log and it is good that you had taken it out.
Cheers,
Longman
Quoted 36 lines · Paul E. McKenney
Quoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the
EXPORT_SYMBOL_GPL()discussion because:1. Kernels built with
CONFIG_PREEMPT=yhave the same issue
with the__rcu_read_lock()and__rcu_read_unlock()functions.2. Many other RCU functions are
EXPORT_SYMBOL_GPL()and have
been for almost two decades.
Quoted 7 lines
But if someone does use RCU readers within
CONFIG_PREEMPT=nkernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
LGTM from the BPF side, nothing really changed about when rcu_read_unlock_strict is an actual function vs no-op macro. It's also important to minimize the number of function calls in the context of recent LBR on-demand work done by Song, so this is a great improvement!
Quoted 58 lines
Thanx, Paul
------------------------------------------------------------------------
commit
4a9f53b997b809c0256838e31c604aeeded2345aAuthor: Waiman Long <[email protected]> Date: Thu Aug 26 22:21:22 2021 -0400rcu: Avoid unneeded function call in
rcu_read_unlock()Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs") the functionrcu_read_unlock_strict()is invoked by the inlinedrcu_read_unlock()function. However,rcu_read_unlock_strict()is an empty function in production kernels, which are built withCONFIG_RCU_STRICT_GRACE_PERIOD=n.There is a mention of
rcu_read_unlock_strict()in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whetherrcu_read_unlock_strict()is ever called.This commit therefore provides a slight performance improvement by hoisting the check of
CONFIG_RCU_STRICT_GRACE_PERIODfromrcu_read_unlock_strict()intorcu_read_unlock(), thus avoiding the pointless call to an empty function.Cc Alexei Starovoitov <[email protected]>Cc Andrii Nakryiko <[email protected]>Signed-off-by Waiman Long <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>include/linux/rcupdate.h +2
-1@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )kernel/rcu/tree_plugin.h +1
-2@@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);
Paul E. McKenney
maintainerQuoted 45 lines · Andrii Nakryiko
Quoted 36 lines · Paul E. McKenney
Quoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the
EXPORT_SYMBOL_GPL()discussion because:1. Kernels built with
CONFIG_PREEMPT=yhave the same issue
with the__rcu_read_lock()and__rcu_read_unlock()functions.2. Many other RCU functions are
EXPORT_SYMBOL_GPL()and have
been for almost two decades.Quoted 7 lines
But if someone does use RCU readers within
CONFIG_PREEMPT=nkernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
Quoted 5 lines
LGTM from the BPF side, nothing really changed about when
rcu_read_unlock_strictis an actual function vs no-op macro. It's also important to minimize the number of function calls in the context of recent LBR on-demand work done by Song, so this is a great improvement!
Thank you for looking this over! May I add your Acked-by or similar?
Thanx, Paul
Quoted 56 lines
Quoted 56 lines
------------------------------------------------------------------------
commit
4a9f53b997b809c0256838e31c604aeeded2345aAuthor: Waiman Long <[email protected]> Date: Thu Aug 26 22:21:22 2021 -0400rcu: Avoid unneeded function call in
rcu_read_unlock()Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs") the functionrcu_read_unlock_strict()is invoked by the inlinedrcu_read_unlock()function. However,rcu_read_unlock_strict()is an empty function in production kernels, which are built withCONFIG_RCU_STRICT_GRACE_PERIOD=n.There is a mention of
rcu_read_unlock_strict()in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whetherrcu_read_unlock_strict()is ever called.This commit therefore provides a slight performance improvement by hoisting the check of
CONFIG_RCU_STRICT_GRACE_PERIODfromrcu_read_unlock_strict()intorcu_read_unlock(), thus avoiding the pointless call to an empty function.Cc Alexei Starovoitov <[email protected]>Cc Andrii Nakryiko <[email protected]>Signed-off-by Waiman Long <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>include/linux/rcupdate.h +2
-1@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )kernel/rcu/tree_plugin.h +1
-2@@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);
Andrii Nakryiko
Acked-byQuoted 47 lines · Paul E. McKenney
Quoted 45 lines · Andrii Nakryiko
Quoted 43 lines · Paul E. McKenney
Quoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the
EXPORT_SYMBOL_GPL()discussion because:1. Kernels built with
CONFIG_PREEMPT=yhave the same issue
with the__rcu_read_lock()and__rcu_read_unlock()functions.2. Many other RCU functions are
EXPORT_SYMBOL_GPL()and have
been for almost two decades.But if someone does use RCU readers within
CONFIG_PREEMPT=nkernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
Quoted 8 lines
Quoted 5 lines
LGTM from the BPF side, nothing really changed about when
rcu_read_unlock_strictis an actual function vs no-op macro. It's also important to minimize the number of function calls in the context of recent LBR on-demand work done by Song, so this is a great improvement!Thank you for looking this over! May I add your Acked-by or similar?
Sure.
Quoted 58 lines
Thanx, Paul
Quoted 56 lines
Quoted 56 lines
------------------------------------------------------------------------
commit
4a9f53b997b809c0256838e31c604aeeded2345aAuthor: Waiman Long <[email protected]> Date: Thu Aug 26 22:21:22 2021 -0400rcu: Avoid unneeded function call in
rcu_read_unlock()Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs") the functionrcu_read_unlock_strict()is invoked by the inlinedrcu_read_unlock()function. However,rcu_read_unlock_strict()is an empty function in production kernels, which are built withCONFIG_RCU_STRICT_GRACE_PERIOD=n.There is a mention of
rcu_read_unlock_strict()in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whetherrcu_read_unlock_strict()is ever called.This commit therefore provides a slight performance improvement by hoisting the check of
CONFIG_RCU_STRICT_GRACE_PERIODfromrcu_read_unlock_strict()intorcu_read_unlock(), thus avoiding the pointless call to an empty function.Cc Alexei Starovoitov <[email protected]>Cc Andrii Nakryiko <[email protected]>Signed-off-by Waiman Long <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>include/linux/rcupdate.h +2
-1@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )kernel/rcu/tree_plugin.h +1
-2@@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);
Paul E. McKenney
maintainerQuoted 54 lines · Andrii Nakryiko
Quoted 53 lines · Paul E. McKenney
Quoted 50 lines · Andrii Nakryiko
Quoted 36 lines · Paul E. McKenney
Quoted 13 lines · Waiman Long
Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs"). A real function callrcu_read_unlock_strict()is added to the inlinedrcu_read_unlock(). Thercu_read_unlock_strict()call is only needed if the performance saggingCONFIG_RCU_STRICT_GRACE_PERIODoption is set. This config option isn't set for most production kernels while the function call overhead remains.To provide a slight performance improvement, the
CONFIG_RCU_STRICT_GRACE_PERIODconfig check is moved fromrcu_read_unlock_strict()to__rcu_read_unlock()so that the function call can be compiled out in most cases.Quoted 5 lines
Besides, the GPL exported
rcu_read_unlock_strict()also impact the the compilation of non-GPL kernel modules asrcu_read_unlock()is a frequently used kernel API.Signed-off-by Waiman Long <[email protected]>Nice, and good eyes!!!
I have queued this for v5.16, that is, not the upcoming merge window but the one after that.
I did my usual wordsmithing, so please check the following in case I messed something up. I intentionally omitted the
EXPORT_SYMBOL_GPL()discussion because:1. Kernels built with
CONFIG_PREEMPT=yhave the same issue
with the__rcu_read_lock()and__rcu_read_unlock()functions.2. Many other RCU functions are
EXPORT_SYMBOL_GPL()and have
been for almost two decades.Quoted 7 lines
But if someone does use RCU readers within
CONFIG_PREEMPT=nkernels from a binary module, I will happily refer them to you for any RCU issues that they encounter. ;-)I am also CCing the BPF guys in case my interpretation of the code in the BPF verifier is incorrect.
LGTM from the BPF side, nothing really changed about when
rcu_read_unlock_strictis an actual function vs no-op macro. It's also important to minimize the number of function calls in the context of recent LBR on-demand work done by Song, so this is a great improvement!
Quoted 6 lines
Quoted 2 lines
Thank you for looking this over! May I add your Acked-by or similar?
Sure.
Acked-by Andrii Nakryiko <[email protected]>
Thank you! I will add this on the next rebase.
Thanx, Paul
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
------------------------------------------------------------------------
commit
4a9f53b997b809c0256838e31c604aeeded2345aAuthor: Waiman Long <[email protected]> Date: Thu Aug 26 22:21:22 2021 -0400rcu: Avoid unneeded function call in
rcu_read_unlock()Since commit
aa40c138cc8f3("rcu: Report QS for outermost PREEMPT=nrcu_read_unlock()for strict GPs") the functionrcu_read_unlock_strict()is invoked by the inlinedrcu_read_unlock()function. However,rcu_read_unlock_strict()is an empty function in production kernels, which are built withCONFIG_RCU_STRICT_GRACE_PERIOD=n.There is a mention of
rcu_read_unlock_strict()in the BPF verifier, but this is in a deny-list, meaning that BPF does not care whetherrcu_read_unlock_strict()is ever called.This commit therefore provides a slight performance improvement by hoisting the check of
CONFIG_RCU_STRICT_GRACE_PERIODfromrcu_read_unlock_strict()intorcu_read_unlock(), thus avoiding the pointless call to an empty function.Cc Alexei Starovoitov <[email protected]>Cc Andrii Nakryiko <[email protected]>Signed-off-by Waiman Long <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>include/linux/rcupdate.h +2
-1@@ -71,7 +71,8 @@ static inline void __rcu_read_lock(void)static inline void __rcu_read_unlock (void ) {preempt_enable (); -rcu_read_unlock_strict (); +if (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD )) +rcu_read_unlock_strict (); }static inline int rcu_preempt_depth (void )kernel/rcu/tree_plugin.h +1
-2@@ -814,8 +814,7 @@ void rcu_read_unlock_strict(void) {struct rcu_data *rdp; -if (!IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || -irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread) +if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data);rcu_report_qs_rdp (rdp);