jump_label: use atomic_try_cmpxchg in static_key_slow_inc_cpuslocked
PATCH 1/1 jump_label: use atomic_try_cmpxchg in static_key_slow_inc_cpuslocked
Use atomic_try_cmpxchg instead of atomic_cmpxchg (*ptr, old, new) == old in static_key_slow_inc_cpuslocked. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).
Also, atomic_try_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.
No functional change intended.
--- kernel/jump_label.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/jump_label.c +2 -6
@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);void static_key_slow_inc_cpuslocked (struct static_key *key) { -int v, v1; -STATIC_KEY_CHECK_USE (key);lockdep_assert_cpus_held ();
@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)* so it counts as "enabled" in jump_label_update(). Note that * atomic_inc_unless_negative() checks >= 0, so roll our own. */ -for (v =atomic_read (&key->enabled); v >0 ; v = v1) { - v1 =atomic_cmpxchg (&key->enabled, v, v +1 ); -if (likely (v1 == v)) +for (int v =atomic_read (&key->enabled); v >0 ; ) +if (likely (atomic_try_cmpxchg (&key->enabled, &v, v +1 )))return ; - }jump_label_lock ();if (atomic_read (&key->enabled) ==0 ) {
Signature · 1 line
2.37.3
[tip: locking/core] jump_label: Use atomic_try_cmpxchg() in static_key_slow_inc_cpuslocked()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: d0c006402e7941558e5283ae434e2847c7999378 Gitweb: https://git.kernel.org/tip/d0c006402e7941558e5283ae434e2847c7999378 Author: Uros Bizjak <[email protected]> AuthorDate: Wed, 19 Oct 2022 16:08:50 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 27 Oct 2022 10:35:41 +02:00
jump_label: Use atomic_try_cmpxchg() in static_key_slow_inc_cpuslocked()
Use atomic_try_cmpxchg() instead of atomic_cmpxchg (*ptr, old, new) == old in static_key_slow_inc_cpuslocked(). x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).
Also, atomic_try_cmpxchg() implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.
No functional change intended.
--- kernel/jump_label.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/jump_label.c +2 -6
@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);void static_key_slow_inc_cpuslocked (struct static_key *key) { -int v, v1; -STATIC_KEY_CHECK_USE (key);lockdep_assert_cpus_held ();
@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)* so it counts as "enabled" in jump_label_update(). Note that * atomic_inc_unless_negative() checks >= 0, so roll our own. */ -for (v =atomic_read (&key->enabled); v >0 ; v = v1) { - v1 =atomic_cmpxchg (&key->enabled, v, v +1 ); -if (likely (v1 == v)) +for (int v =atomic_read (&key->enabled); v >0 ; ) +if (likely (atomic_try_cmpxchg (&key->enabled, &v, v +1 )))return ; - }jump_label_lock ();if (atomic_read (&key->enabled) ==0 ) {
Steven Rostedt
reviewerQuoted 34 lines · On Wed, 19 Oct 2022 16:08:50 +0200 Uros Bizjak <[email protected]> wrote:
Use
atomic_try_cmpxchginstead ofatomic_cmpxchg(*ptr, old, new) == old instatic_key_slow_inc_cpuslocked. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).Also,
atomic_try_cmpxchgimplicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.No functional change intended.
Cc Peter Zijlstra <[email protected]>Cc Josh Poimboeuf <[email protected]>Cc Jason Baron <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ard Biesheuvel <[email protected]>Signed-off-by Uros Bizjak <[email protected]>--- kernel/jump_label.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)kernel/jump_label.c +0
-2@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);void static_key_slow_inc_cpuslocked (struct static_key *key) { -int v, v1; -STATIC_KEY_CHECK_USE (key);lockdep_assert_cpus_held ();
kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133
Quoted 7 lines
kernel/jump_label.c +1
-3@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)* so it counts as "enabled" in jump_label_update(). Note that * atomic_inc_unless_negative() checks >= 0, so roll our own. */ -for (v =atomic_read (&key->enabled); v >0 ; v = v1) { - v1 =atomic_cmpxchg (&key->enabled, v, v +1 ); -if (likely (v1 == v)) +for (int v =atomic_read (&key->enabled); v >0 ; )
Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.
Please keep the "int v;" at the beginning.
kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 134
Quoted 1 line
kernel/jump_label.c +1
-0+if (likely (atomic_try_cmpxchg (&key->enabled, &v, v +1 )))
I'm fine with this change.
-- Steve
Quoted 5 lines
kernel/jump_label.c +0
-1return ; - }jump_label_lock ();if (atomic_read (&key->enabled) ==0 ) {
Peter Zijlstra
maintainerQuoted 4 lines · Steven Rostedt
kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133
Quoted 1 line
kernel/jump_label.c +1
-0+for (int v =atomic_read (&key->enabled); v >0 ; )Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.
I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt
kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133
Quoted 1 line
kernel/jump_label.c +1
-0+for (int v =atomic_read (&key->enabled); v >0 ; )Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.
I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.
That was one reason, yes. Marco and I wanted to be able to use C99-style declarations in for loops to make it easier/possible to build macros with locally-scoped control variables.
I personally prefer using C99-style declarations in for loops, but I don't have a strong feeling that we *must* do so.
Thanks,
Mark.
Steven Rostedt
reviewerQuoted 9 lines · On Wed, 23 Nov 2022 10:08:59 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt
kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133
Quoted 1 line
kernel/jump_label.c +1
-0+for (int v =atomic_read (&key->enabled); v >0 ; )Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.
I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.
OK, if you are fine with it then sure. I personally like seeing all variables declared in one place. Maybe because I've been trained that way, and I can easily be confused when I see a variable somewhere and don't see it in the beginning declarations.
-- Steve