locking/rtmutex: Use try_cmpxchg_relaxed in mark_rt_mutex_waiters()
PATCH 1/1 locking/rtmutex: Use try_cmpxchg_relaxed in mark_rt_mutex_waiters()
Use try_cmpxchg() instead of cmpxchg(*ptr, old, new) == old.
X86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after CMPXCHG (and related move instruction in front of CMPXCHG).
Also, try_cmpxchg() implicitly assigns old *ptr value to "old" when CMPXCHG fails. There is no need to re-read the value in the loop.
Note that the value from *ptr should be read using READ_ONCE to prevent the compiler from merging, refetching or reordering the read.
No functional change intended.
--- kernel/locking/rtmutex.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/locking/rtmutex.c +5 -4
@@ -237,12 +237,13 @@ static __always_inline bool rt_mutex_cmpxchg_release(struct rt_mutex_base *lock,*/ static __always_inline void mark_rt_mutex_waiters(struct rt_mutex_base *lock) { -unsigned long owner, *p = (unsigned long *) &lock->owner; +unsigned long *p = (unsigned long *) &lock->owner; +unsigned long owner, new; + owner =READ_ONCE (*p);do { - owner = *p; - }while (cmpxchg_relaxed (p, owner, - owner |RT_MUTEX_HAS_WAITERS ) != owner); + new = owner |RT_MUTEX_HAS_WAITERS ; + }while (!try_cmpxchg_relaxed (p, &owner, new));/* * The cmpxchg loop above is relaxed to avoid back-to-back ACQUIRE
Signature · 1 line
2.31.1
[tip: locking/core] locking/rtmutex: Use try_cmpxchg_relaxed() in mark_rt_mutex_waiters()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: ce3576ebd62d99f79c1dc98824e2ef6d6ab68434 Gitweb: https://git.kernel.org/tip/ce3576ebd62d99f79c1dc98824e2ef6d6ab68434 Author: Uros Bizjak <[email protected]> AuthorDate: Wed, 24 Jan 2024 11:49:53 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 01 Mar 2024 13:02:05 +01:00
locking/rtmutex: Use try_cmpxchg_relaxed() in mark_rt_mutex_waiters()
Use try_cmpxchg() instead of cmpxchg(*ptr, old, new) == old.
The x86 CMPXCHG instruction returns success in the ZF flag, so this change saves a compare after CMPXCHG (and related move instruction in front of CMPXCHG).
Also, try_cmpxchg() implicitly assigns old *ptr value to "old" when CMPXCHG fails. There is no need to re-read the value in the loop.
Note that the value from *ptr should be read using READ_ONCE() to prevent the compiler from merging, refetching or reordering the read.
No functional change intended.
--- kernel/locking/rtmutex.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/locking/rtmutex.c +5 -4
@@ -237,12 +237,13 @@ static __always_inline bool rt_mutex_cmpxchg_release(struct rt_mutex_base *lock,*/ static __always_inline void mark_rt_mutex_waiters(struct rt_mutex_base *lock) { -unsigned long owner, *p = (unsigned long *) &lock->owner; +unsigned long *p = (unsigned long *) &lock->owner; +unsigned long owner, new; + owner =READ_ONCE (*p);do { - owner = *p; - }while (cmpxchg_relaxed (p, owner, - owner |RT_MUTEX_HAS_WAITERS ) != owner); + new = owner |RT_MUTEX_HAS_WAITERS ; + }while (!try_cmpxchg_relaxed (p, &owner, new));/* * The cmpxchg loop above is relaxed to avoid back-to-back ACQUIRE