locking/barriers: Introduce smp_cond_load_mask_relaxed & acquire
RFC PATCH 1/1 locking/barriers: Introduce smp_cond_load_mask_relaxed & acquire
From: Guo Ren <[email protected]>
The current cond_load primitive contains two parts (condition expression and load value), but the usage of cond_load may require the sub-size condition expression of the load size. That means hardware could utilize a mask argument to optimize the wait condition. If the mask argument size is less than the hardware minimum wait size, the hardware uses its minimum size.
The patch contains a qspinlock example: When it is at the head of the waitqueue, it waits for the owner & pending to go away. The forward progress condition only cares locked_pending part, but it needs to load the 32-bit lock value as a return.
That also means WFE-liked instruction would need a mask argument of the load reservation set.
--- include/asm-generic/barrier.h | 22 ++++++++++++++++++++++ include/linux/atomic.h | 4 ++++ kernel/locking/qspinlock.c | 3 ++- 3 files changed, 28 insertions(+), 1 deletion(-)
include/asm-generic/barrier.h +22 -0
@@ -275,6 +275,28 @@ do { \ })#endif +/** +* smp_cond_load_mask_relaxed() - (Spin) wait for cond with ACQUIRE ordering +* @ptr: pointer to the variable to wait on +* @cond: boolean expression to wait for +* @mask: mask *ptr to wait for (effect of 0 is the same with -1) +*/ +#ifndef smp_cond_load_mask_relaxed +#define smp_cond_load_mask_relaxed (ptr, cond_expr, mask) \ + smp_cond_load_relaxed(ptr, cond_expr) +#endif + +/** +* smp_cond_load_mask_acquire() - (Spin) wait for cond with ACQUIRE ordering +* @ptr: pointer to the variable to wait on +* @cond: boolean expression to wait for +* @mask: mask *ptr to wait for (effect of 0 is the same with -1) +*/ +#ifndef smp_cond_load_mask_acquire +#define smp_cond_load_mask_acquire (ptr, cond_expr, mask) \ + smp_cond_load_acquire(ptr, cond_expr) +#endif +/* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have
include/linux/atomic.h +4 -0
@@ -27,9 +27,13 @@#define atomic_cond_read_acquire (v, c) smp_cond_load_acquire(&(v)->counter, (c))#define atomic_cond_read_relaxed (v, c) smp_cond_load_relaxed(&(v)->counter, (c)) +#define atomic_cond_read_mask_acquire (v, c, m) smp_cond_load_mask_acquire(&(v)->counter, (c), (m)) +#define atomic_cond_read_mask_relaxed (v, c, m) smp_cond_load_mask_relaxed(&(v)->counter, (c), (m))#define atomic64_cond_read_acquire (v, c) smp_cond_load_acquire(&(v)->counter, (c))#define atomic64_cond_read_relaxed (v, c) smp_cond_load_relaxed(&(v)->counter, (c)) +#define atomic64_cond_read_mask_acquire (v, c, m) smp_cond_load_mask_acquire(&(v)->counter, (c), (m)) +#define atomic64_cond_read_mask_relaxed (v, c, m) smp_cond_load_mask_relaxed(&(v)->counter, (c), (m))/* * The idea here is to build acquire/release variants by adding explicit
kernel/locking/qspinlock.c +2 -1
@@ -511,7 +511,8 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if ((val =pv_wait_head_or_lock (lock, node))) goto locked; - val =atomic_cond_read_acquire (&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK)); + val =atomic_cond_read_mask_acquire (&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK), + _Q_LOCKED_PENDING_MASK); locked:/*
Signature · 1 line
2.36.1
Waiman Long
reviewerQuoted 86 lines · On 12/25/22 06:55, [email protected] wrote:
From: Guo Ren <[email protected]>
The current
cond_loadprimitive contains two parts (condition expression and load value), but the usage ofcond_loadmay require the sub-size condition expression of the load size. That means hardware could utilize a mask argument to optimize the wait condition. If the mask argument size is less than the hardware minimum wait size, the hardware uses its minimum size.The patch contains a qspinlock example: When it is at the head of the waitqueue, it waits for the owner & pending to go away. The forward progress condition only cares
locked_pendingpart, but it needs to load the 32-bit lock value as a return.That also means WFE-liked instruction would need a mask argument of the load reservation set.
Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Waiman Long <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Boqun Feng <[email protected]>Cc Will Deacon <[email protected]>Cc Ingo Molnar <[email protected]>--- include/asm-generic/barrier.h | 22 ++++++++++++++++++++++ include/linux/atomic.h | 4 ++++ kernel/locking/qspinlock.c | 3 ++- 3 files changed, 28 insertions(+), 1 deletion(-)include/asm-generic/barrier.h +22
-0@@ -275,6 +275,28 @@ do { \ })#endif +/** +* smp_cond_load_mask_relaxed() - (Spin) wait for cond with ACQUIRE ordering +* @ptr: pointer to the variable to wait on +* @cond: boolean expression to wait for +* @mask: mask *ptr to wait for (effect of 0 is the same with -1) +*/ +#ifndef smp_cond_load_mask_relaxed +#define smp_cond_load_mask_relaxed (ptr, cond_expr, mask) \ + smp_cond_load_relaxed(ptr, cond_expr) +#endif + +/** +* smp_cond_load_mask_acquire() - (Spin) wait for cond with ACQUIRE ordering +* @ptr: pointer to the variable to wait on +* @cond: boolean expression to wait for +* @mask: mask *ptr to wait for (effect of 0 is the same with -1) +*/ +#ifndef smp_cond_load_mask_acquire +#define smp_cond_load_mask_acquire (ptr, cond_expr, mask) \ + smp_cond_load_acquire(ptr, cond_expr) +#endif +/* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have include/linux/atomic.h +4
-0@@ -27,9 +27,13 @@#define atomic_cond_read_acquire (v, c) smp_cond_load_acquire(&(v)->counter, (c))#define atomic_cond_read_relaxed (v, c) smp_cond_load_relaxed(&(v)->counter, (c)) +#define atomic_cond_read_mask_acquire (v, c, m) smp_cond_load_mask_acquire(&(v)->counter, (c), (m)) +#define atomic_cond_read_mask_relaxed (v, c, m) smp_cond_load_mask_relaxed(&(v)->counter, (c), (m))#define atomic64_cond_read_acquire (v, c) smp_cond_load_acquire(&(v)->counter, (c))#define atomic64_cond_read_relaxed (v, c) smp_cond_load_relaxed(&(v)->counter, (c)) +#define atomic64_cond_read_mask_acquire (v, c, m) smp_cond_load_mask_acquire(&(v)->counter, (c), (m)) +#define atomic64_cond_read_mask_relaxed (v, c, m) smp_cond_load_mask_relaxed(&(v)->counter, (c), (m))/* * The idea here is to build acquire/release variants by adding explicit kernel/locking/qspinlock.c +0
-0
kernel/locking/qspinlock.c void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) · line 518
Quoted 9 lines
kernel/locking/qspinlock.c +2
-1@@ -511,7 +511,8 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if ((val =pv_wait_head_or_lock (lock, node))) goto locked; - val =atomic_cond_read_acquire (&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK)); + val =atomic_cond_read_mask_acquire (&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK), + _Q_LOCKED_PENDING_MASK); locked:/*
This patch is essentially a no-op. You will have to have at least one arch that has its own version of smp_cond_load_mask*() and get some benefit out of it. Otherwise, it is not likely to be merged.
Cheers,
Longman