Skip to contentSkip to conversation

jump_label: use atomic_try_cmpxchg in static_key_slow_inc_cpuslocked

PATCH Applied Uros Bizjak 5 participants 6 messages Oct 19, 2022 – Nov 23, 2022 Static branch/call

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 jump_label: use atomic_try_cmpxchg in static_key_slow_inc_cpuslocked

Use atomic_try_cmpxchg instead of atomic_cmpxchg (*ptr, old, new) == old in static_key_slow_inc_cpuslocked. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).

Also, atomic_try_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.

No functional change intended.

Cc Peter Zijlstra <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Jason Baron <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ard Biesheuvel <[email protected]>
---
 kernel/jump_label.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/jump_label.c +2 -6
index 714ac4c3b556..4d6c6f5f60db 100644
@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);
 
 void static_key_slow_inc_cpuslocked(struct static_key *key)
 {
-	int v, v1;
-
 	STATIC_KEY_CHECK_USE(key);
 	lockdep_assert_cpus_held();
 
@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)
 	 * so it counts as "enabled" in jump_label_update().  Note that
 	 * atomic_inc_unless_negative() checks >= 0, so roll our own.
 	 */
-	for (v = atomic_read(&key->enabled); v > 0; v = v1) {
-		v1 = atomic_cmpxchg(&key->enabled, v, v + 1);
-		if (likely(v1 == v))
+	for (int v = atomic_read(&key->enabled); v > 0; )
+		if (likely(atomic_try_cmpxchg(&key->enabled, &v, v + 1)))
 			return;
-	}
 
 	jump_label_lock();
 	if (atomic_read(&key->enabled) == 0) {
Signature · 1 line
2.37.3
merge bot Applied
#1 lore raw

[tip: locking/core] jump_label: Use atomic_try_cmpxchg() in static_key_slow_inc_cpuslocked()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     d0c006402e7941558e5283ae434e2847c7999378
Gitweb:        https://git.kernel.org/tip/d0c006402e7941558e5283ae434e2847c7999378
Author:        Uros Bizjak <[email protected]>
AuthorDate:    Wed, 19 Oct 2022 16:08:50 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 27 Oct 2022 10:35:41 +02:00

jump_label: Use atomic_try_cmpxchg() in static_key_slow_inc_cpuslocked()

Use atomic_try_cmpxchg() instead of atomic_cmpxchg (*ptr, old, new) == old in static_key_slow_inc_cpuslocked(). x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).

Also, atomic_try_cmpxchg() implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.

No functional change intended.

---
 kernel/jump_label.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/jump_label.c +2 -6
index 714ac4c..4d6c6f5 100644
@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);
 
 void static_key_slow_inc_cpuslocked(struct static_key *key)
 {
-	int v, v1;
-
 	STATIC_KEY_CHECK_USE(key);
 	lockdep_assert_cpus_held();
 
@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)
 	 * so it counts as "enabled" in jump_label_update().  Note that
 	 * atomic_inc_unless_negative() checks >= 0, so roll our own.
 	 */
-	for (v = atomic_read(&key->enabled); v > 0; v = v1) {
-		v1 = atomic_cmpxchg(&key->enabled, v, v + 1);
-		if (likely(v1 == v))
+	for (int v = atomic_read(&key->enabled); v > 0; )
+		if (likely(atomic_try_cmpxchg(&key->enabled, &v, v + 1)))
 			return;
-	}
 
 	jump_label_lock();
 	if (atomic_read(&key->enabled) == 0) {
#2 lore raw
In reply to Uros Bizjak
Quoted 34 lines · On Wed, 19 Oct 2022 16:08:50 +0200 Uros Bizjak <[email protected]> wrote:

Use atomic_try_cmpxchg instead of atomic_cmpxchg (*ptr, old, new) == old in static_key_slow_inc_cpuslocked. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).

Also, atomic_try_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications.

No functional change intended.

Cc Peter Zijlstra <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Jason Baron <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ard Biesheuvel <[email protected]>
---
 kernel/jump_label.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/jump_label.c +0 -2
index 714ac4c3b556..4d6c6f5f60db 100644
@@ -115,8 +115,6 @@ EXPORT_SYMBOL_GPL(static_key_count);
 
 void static_key_slow_inc_cpuslocked(struct static_key *key)
 {
-	int v, v1;
-
 	STATIC_KEY_CHECK_USE(key);
 	lockdep_assert_cpus_held();
 

kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133

Quoted 7 lines
kernel/jump_label.c +1 -3
@@ -132,11 +130,9 @@ void static_key_slow_inc_cpuslocked(struct static_key *key)
 	 * so it counts as "enabled" in jump_label_update().  Note that
 	 * atomic_inc_unless_negative() checks >= 0, so roll our own.
 	 */
-	for (v = atomic_read(&key->enabled); v > 0; v = v1) {
-		v1 = atomic_cmpxchg(&key->enabled, v, v + 1);
-		if (likely(v1 == v))
+	for (int v = atomic_read(&key->enabled); v > 0; )

Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.

Please keep the "int v;" at the beginning.

kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 134

Quoted 1 line
kernel/jump_label.c +1 -0
+		if (likely(atomic_try_cmpxchg(&key->enabled, &v, v + 1)))

I'm fine with this change.

-- Steve

Quoted 5 lines
kernel/jump_label.c +0 -1
 			return;
-	}
 
 	jump_label_lock();
 	if (atomic_read(&key->enabled) == 0) {
maintainer
#3 lore raw
Quoted 4 lines · Steven Rostedt

kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133

Quoted 1 line
kernel/jump_label.c +1 -0
+	for (int v = atomic_read(&key->enabled); v > 0; )

Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.

I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.

#4 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt

kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133

Quoted 1 line
kernel/jump_label.c +1 -0
+	for (int v = atomic_read(&key->enabled); v > 0; )

Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.

I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.

That was one reason, yes. Marco and I wanted to be able to use C99-style declarations in for loops to make it easier/possible to build macros with locally-scoped control variables.

I personally prefer using C99-style declarations in for loops, but I don't have a strong feeling that we *must* do so.

Thanks,
Mark.

#5 lore raw
In reply to Peter Zijlstra
Quoted 9 lines · On Wed, 23 Nov 2022 10:08:59 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt

kernel/jump_label.c void static_key_slow_inc_cpuslocked(struct static_key *key) · line 133

Quoted 1 line
kernel/jump_label.c +1 -0
+	for (int v = atomic_read(&key->enabled); v > 0; )  

Although it's permitted by the compiler, the kernel style is to not add declarations in conditionals.

I'm thinking the whole motivation for upping to C99 was exactly so that we could start using this pattern.

OK, if you are fine with it then sure. I personally like seeing all variables declared in one place. Maybe because I've been trained that way, and I can easily be confused when I see a variable somewhere and don't see it in the beginning declarations.

-- Steve