powerpc: alternate queued spinlock implementation
Subsystems (2)
Nicholas Piggin
reviewerv2 cover powerpc: alternate queued spinlock implementation
This replaces the generic queued spinlock code (like s390 does) with our own implementation. There is an extra shim patch 1a to get the series to apply.
Generic PV qspinlock code is causing latency / starvation regressions on large systems that are resulting in hard lockups reported (mostly in pathoogical cases). The generic qspinlock code has a number of issues important for powerpc hardware and hypervisors that aren't easily solved without changing code that would impact other architectures. Follow s390's lead and implement our own for now.
Issues for powerpc using generic qspinlocks:
- The previous lock value should not be loaded with simple loads, and need not be passed around from previous loads or cmpxchg results, because powerpc uses ll/sc-style atomics which can perform more complex operations that do not require this. powerpc implementations tend to prefer loads use larx for improved coherency performance.
- The queueing process should absolutely minimise the number of stores to the lock word to reduce exclusive coherency probes, important for large system scalability. The pending logic is counter productive here.
- Non-atomic unlock for paravirt locks is important (atomic instructions tend to still be more expensive than x86 CPUs).
- Yielding to the lock owner is important in the oversubscribed paravirt case, which requires storing the owner CPU in the lock word.
- More control of lock stealing for the paravirt case is important to keep latency down on large systems.
- The lock acquisition operation should always be made with a special variant of atomic instructions with the lock hint bit set, including (especially) in the queueing paths. This is more a matter of adding more arch lock helpers so not an insurmountable problem for generic code.
So far this still has some work to test and tune performance. It does improve some of the latency and starvation issues, it also has some throughput regressions in some cases, but I already left it too long since Jordan's really nice review including two subtle bugs found, so I'm posting the current state of things...
Since v1:
- Change most 'if (cond) return 1 ; return 0;'
- Bug fix: was testing count == MAX, but reentrant NMIs could bring that
Quoted 1 line
MAX and crash.
- Fix missing memory barrier lost in asm conversion patch.
- Seperate the release barrier in publish_tail from the acquire barrier in get_tail_qnode.
- Moving a few minor things into their logically correct change.
- Make encode_tail_cpu take a cpu argument to match get_tail_cpu.
- Rename get_tail_cpu to decode_tail_cpu to match encode_tail_cpu.
- Rename lock_set_locked to set_locked.
- IS_ENABLED(x) ? 1 : 0 -> IS_ENABLED(x)
- Fix some comments inside inline asm.
- Change tunable names to lowercase.
- Consolidate asm for trylock_clear_tail_cpu and trylock_with_tail_cpu
- Restructure steal/wait loops to be more readable
- Count a failed cmpxchg as an iteration in steal/wait loops to avoid theoretical livelock/latency concern.
Nicholas Piggin (17):
powerpc/qspinlock: powerpc qspinlock implementation powerpc/qspinlock: add mcs queueing for contended waiters powerpc/qspinlock: use a half-word store to unlock to avoid larx/stcx. powerpc/qspinlock: convert atomic operations to assembly powerpc/qspinlock: allow new waiters to steal the lock before queueing powerpc/qspinlock: theft prevention to control latency powerpc/qspinlock: store owner CPU in lock word powerpc/qspinlock: paravirt yield to lock owner powerpc/qspinlock: implement option to yield to previous node powerpc/qspinlock: allow stealing when head of queue yields powerpc/qspinlock: allow propagation of yield CPU down the queue powerpc/qspinlock: add ability to prod new queue head CPU powerpc/qspinlock: trylock and initial lock attempt may steal powerpc/qspinlock: use spin_begin/end API powerpc/qspinlock: reduce remote node steal spins powerpc/qspinlock: allow indefinite spinning on a preempted owner powerpc/qspinlock: provide accounting and options for sleepy locks
arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/qspinlock.h | 133 ++- arch/powerpc/include/asm/qspinlock_types.h | 70 ++ arch/powerpc/include/asm/spinlock_types.h | 2 +- arch/powerpc/lib/Makefile | 4 +- arch/powerpc/lib/qspinlock.c | 1008 ++++++++++++++++++++ 6 files changed, 1174 insertions(+), 44 deletions(-) create mode 100644 arch/powerpc/include/asm/qspinlock_types.h create mode 100644 arch/powerpc/lib/qspinlock.c
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 1/1 powerpc/qspinlock: prepare powerpc qspinlock implementation
This is a merge placeholder with a conflicting series of patches to generic qspinlocks. Not intended to be standalone, this should be applied before patch 1.
arch/powerpc/include/asm/Kbuild +0 -1
@@ -4,7 +4,6 @@ generated-y += syscall_table_64.h generated-y += syscall_table_spu.h generic-y += export.h generic-y += kvm_types.h -generic-y += mcs_spinlock.h generic-y += qrwlock.h generic-y += vtime.h generic-y += early_ioremap.h
arch/powerpc/include/asm/qspinlock.h +14 -31
69 lines not on this page · raw message
arch/powerpc/include/asm/qspinlock_paravirt.h deleted +0 -7
@@ -1,7 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -#ifndef _ASM_POWERPC_QSPINLOCK_PARAVIRT_H -#define _ASM_POWERPC_QSPINLOCK_PARAVIRT_H - -EXPORT_SYMBOL (__pv_queued_spin_unlock); - -#endif /* _ASM_POWERPC_QSPINLOCK_PARAVIRT_H */
arch/powerpc/include/asm/spinlock.h +1 -1
@@ -13,7 +13,7 @@/* See include/linux/spinlock.h */ #define smp_mb__after_spinlock () smp_mb() -#ifndef CONFIG_PARAVIRT_SPINLOCKS +#ifndef CONFIG_PPC_QUEUED_SPINLOCKS static inline void pv_spinlocks_init (void ) { }#endif
Nicholas Piggin
reviewerPATCH v2 1/17 powerpc/qspinlock: powerpc qspinlock implementation
In reply to Nicholas PigginAdd a powerpc specific implementation of queued spinlocks. This is the build framework with a very simple (non-queued) spinlock implementation to begin with. Later changes add queueing, and other features and optimisations one-at-a-time. It is done this way to more easily see how the queued spinlocks are built, and to make performance and correctness bisects more useful.
--- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/qspinlock.h | 76 +++++++++------------- arch/powerpc/include/asm/qspinlock_types.h | 13 ++++ arch/powerpc/include/asm/spinlock_types.h | 2 +- arch/powerpc/lib/Makefile | 4 +- arch/powerpc/lib/qspinlock.c | 17 +++++ 6 files changed, 66 insertions(+), 47 deletions(-) create mode 100644 arch/powerpc/include/asm/qspinlock_types.h create mode 100644 arch/powerpc/lib/qspinlock.c
arch/powerpc/Kconfig +0 -1
@@ -155,7 +155,6 @@ config PPC select ARCH_USE_CMPXCHG_LOCKREF if PPC64 select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS - select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM
arch/powerpc/include/asm/qspinlock.h +32 -44
98 lines not on this page · raw message
arch/powerpc/include/asm/qspinlock_types.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _ASM_POWERPC_QSPINLOCK_TYPES_H +#define _ASM_POWERPC_QSPINLOCK_TYPES_H + +#include <linux/types.h> + +typedef struct qspinlock { +atomic_t val; +}arch_spinlock_t ; + +#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } + +#endif /* _ASM_POWERPC_QSPINLOCK_TYPES_H */
arch/powerpc/include/asm/spinlock_types.h +1 -1
@@ -7,7 +7,7 @@#endif #ifdef CONFIG_PPC_QUEUED_SPINLOCKS -#include <asm-generic/qspinlock_types.h> +#include <asm/qspinlock_types.h> #include <asm-generic/qrwlock_types.h> #else #include <asm/simple_spinlock_types.h>
arch/powerpc/lib/Makefile +3 -1
@@ -52,7 +52,9 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ memcpy_64.o copy_mc_64.o -ifndef CONFIG_PPC_QUEUED_SPINLOCKS +ifdef CONFIG_PPC_QUEUED_SPINLOCKS +obj64-$(CONFIG_SMP) += qspinlock.o +else obj64-$(CONFIG_SMP) += locks.o endif
arch/powerpc/lib/qspinlock.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include <linux/export.h> +#include <linux/processor.h> +#include <asm/qspinlock.h> + +void queued_spin_lock_slowpath (struct qspinlock *lock) +{ +while (!queued_spin_trylock (lock)) +cpu_relax (); +} +EXPORT_SYMBOL (queued_spin_lock_slowpath); + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +void pv_spinlocks_init (void ) +{ +} +#endif
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 2/17 powerpc/qspinlock: add mcs queueing for contended waiters
In reply to Nicholas PigginThis forms the basis of the qspinlock slow path.
Like generic qspinlocks and unlike the vanilla MCS algorithm, the lock owner does not participate in the queue, only waiters. The first waiter spins on the lock word, then when the lock is released it takes ownership and unqueues the next waiter. This is how qspinlocks can be implemented with the spinlock API -- lock owners don't need a node, only waiters do.
--- arch/powerpc/include/asm/qspinlock.h | 10 +- arch/powerpc/include/asm/qspinlock_types.h | 21 +++ arch/powerpc/lib/qspinlock.c | 180 ++++++++++++++++++++- 3 files changed, 205 insertions(+), 6 deletions(-)
arch/powerpc/include/asm/qspinlock.h +7 -3
@@ -18,12 +18,12 @@ static __always_inline int queued_spin_value_unlocked(struct qspinlock lock) static __always_inline int queued_spin_is_contended(struct qspinlock *lock) { -return 0 ; +return !!(atomic_read (&lock->val) & _Q_TAIL_CPU_MASK); } static __always_inline int queued_spin_trylock(struct qspinlock *lock) { -return atomic_cmpxchg_acquire (&lock->val,0 ,1 ) ==0 ; +return atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ; }void queued_spin_lock_slowpath (struct qspinlock *lock);
@@ -36,7 +36,11 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock)static inline void queued_spin_unlock (struct qspinlock *lock) { -atomic_set_release (&lock->val,0 ); +for (;;) { +int val =atomic_read (&lock->val); +if (atomic_cmpxchg_release (&lock->val, val, val & ~_Q_LOCKED_VAL) == val) +return ; + } }#define arch_spin_is_locked (l) queued_spin_is_locked(l)
arch/powerpc/include/asm/qspinlock_types.h +21 -0
@@ -10,4 +10,25 @@ typedef struct qspinlock {#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } +/* +* Bitfields in the atomic value: +* +* 0: locked bit +* 16-31: tail cpu (+1) +*/ +#define _Q_SET_MASK (type) (((1U << _Q_ ## type ## _BITS) - 1)\ + << _Q_ ## type ## _OFFSET) +#define _Q_LOCKED_OFFSET 0 +#define _Q_LOCKED_BITS 1 +#define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED) +#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) + +#define _Q_TAIL_CPU_OFFSET 16 +#define _Q_TAIL_CPU_BITS (32 - _Q_TAIL_CPU_OFFSET) +#define _Q_TAIL_CPU_MASK _Q_SET_MASK(TAIL_CPU) + +#if CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS) +#error "qspinlock does not support such large CONFIG_NR_CPUS" +#endif +#endif /* _ASM_POWERPC_QSPINLOCK_TYPES_H */
arch/powerpc/lib/qspinlock.c +177 -3
189 lines not on this page · raw message
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 3/17 powerpc/qspinlock: use a half-word store to unlock to avoid larx/stcx.
In reply to Nicholas PigginThe first 16 bits of the lock are only modified by the owner, and other modifications always use atomic operations on the entire 32 bits, so unlocks can use plain stores on the 16 bits. This is the same kind of optimisation done by core qspinlock code.
--- arch/powerpc/include/asm/qspinlock.h | 6 +----- arch/powerpc/include/asm/qspinlock_types.h | 19 +++++++++++++++++-- 2 files changed, 18 insertions(+), 7 deletions(-)
arch/powerpc/include/asm/qspinlock.h +1 -5
@@ -36,11 +36,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock)static inline void queued_spin_unlock (struct qspinlock *lock) { -for (;;) { -int val =atomic_read (&lock->val); -if (atomic_cmpxchg_release (&lock->val, val, val & ~_Q_LOCKED_VAL) == val) -return ; - } +smp_store_release (&lock->locked,0 ); }#define arch_spin_is_locked (l) queued_spin_is_locked(l)
arch/powerpc/include/asm/qspinlock_types.h +17 -2
@@ -3,12 +3,27 @@#define _ASM_POWERPC_QSPINLOCK_TYPES_H#include <linux/types.h> +#include <asm/byteorder.h> typedef struct qspinlock { -atomic_t val; +union { +atomic_t val; + +#ifdef __LITTLE_ENDIAN +struct { +u16 locked; +u8 reserved[2 ]; + }; +#else +struct { +u8 reserved[2 ]; +u16 locked; + }; +#endif + }; }arch_spinlock_t ; -#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } +#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } }/* * Bitfields in the atomic value:
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 4/17 powerpc/qspinlock: convert atomic operations to assembly
In reply to Nicholas PigginThis uses more optimal ll/sc style access patterns (rather than cmpxchg), and also sets the EH=1 lock hint on those operations which acquire ownership of the lock.
--- arch/powerpc/include/asm/qspinlock.h | 24 +++++-- arch/powerpc/include/asm/qspinlock_types.h | 6 +- arch/powerpc/lib/qspinlock.c | 81 +++++++++++++++------- 3 files changed, 77 insertions(+), 34 deletions(-)
arch/powerpc/include/asm/qspinlock.h +19 -5
@@ -2,28 +2,42 @@#ifndef _ASM_POWERPC_QSPINLOCK_H#define _ASM_POWERPC_QSPINLOCK_H -#include <linux/atomic.h> #include <linux/compiler.h> #include <asm/qspinlock_types.h> static __always_inline int queued_spin_is_locked(struct qspinlock *lock) { -return atomic_read (&lock->val); +return READ_ONCE (lock->val); } static __always_inline int queued_spin_value_unlocked(struct qspinlock lock) { -return !atomic_read (&lock.val); +return !lock.val; } static __always_inline int queued_spin_is_contended(struct qspinlock *lock) { -return !!(atomic_read (&lock->val) & _Q_TAIL_CPU_MASK); +return !!(READ_ONCE (lock->val) & _Q_TAIL_CPU_MASK); } static __always_inline int queued_spin_trylock(struct qspinlock *lock) { -return atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ; +u32 prev; + + asmvolatile ( +"1: lwarx %0,0,%1,%3 # queued_spin_trylock \n" +" cmpwi 0,%0,0 \n" +" bne- 2f \n" +" stwcx. %2,0,%1 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + :"=&r" (prev) + :"r" (&lock->val),"r" (_Q_LOCKED_VAL), +"i" (IS_ENABLED (CONFIG_PPC64 )) + :"cr0" ,"memory" ); + +return likely (prev ==0 ); }void queued_spin_lock_slowpath (struct qspinlock *lock);
arch/powerpc/include/asm/qspinlock_types.h +3 -3
@@ -7,7 +7,7 @@ typedef struct qspinlock { union { - atomic_t val; + u32 val; #ifdef __LITTLE_ENDIAN struct {
@@ -23,10 +23,10 @@ typedef struct qspinlock { }; } arch_spinlock_t; -#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } } +#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = 0 } }/* -* Bitfields in the atomic value: +* Bitfields in the lock word: * * 0: locked bit * 16-31: tail cpu (+1)
arch/powerpc/lib/qspinlock.c +55 -26
125 lines not on this page · raw message
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 5/17 powerpc/qspinlock: allow new waiters to steal the lock before queueing
In reply to Nicholas PigginAllow new waiters a number of spins on the lock word before queueing, which particularly helps paravirt performance when physical CPUs are oversubscribed.
--- arch/powerpc/lib/qspinlock.c | 159 ++++++++++++++++++++++++++++++----- 1 file changed, 140 insertions(+), 19 deletions(-)
arch/powerpc/lib/qspinlock.c +140 -19
@@ -19,8 +19,17 @@ struct qnodes {struct qnode nodes[MAX_NODES ]; }; +/* Tuning parameters */ +static int steal_spins__read_mostly = (1 <<5 ); +static bool maybe_stealers__read_mostly = true; + static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); +static __always_inline int get_steal_spins(void) +{ +return steal_spins; +} +static inline u32 encode_tail_cpu (int cpu) {return (cpu +1 ) << _Q_TAIL_CPU_OFFSET;
@@ -50,15 +59,14 @@ static __always_inline void set_locked(struct qspinlock *lock)BUG_ON (prev & _Q_LOCKED_VAL); } -/* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ -static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 old) +static __always_inline u32 __trylock_cmpxchg(struct qspinlock *lock, u32 old, u32 new) {u32 prev;BUG_ON (old & _Q_LOCKED_VAL); asmvolatile ( -"1: lwarx %0,0,%1,%4 # trylock_clear_tail_cpu \n" +"1: lwarx %0,0,%1,%4 # __trylock_cmpxchg \n" " cmpw 0,%0,%2 \n" " bne- 2f \n" " stwcx. %3,0,%1 \n"
@@ -66,13 +74,27 @@ static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 ol"\t" PPC_ACQUIRE_BARRIER " \n" "2: \n" : "=&r" (prev) - : "r" (&lock->val), "r"(old), "r" (_Q_LOCKED_VAL), + : "r" (&lock->val), "r"(old), "r" (new)," i" (IS_ENABLED(CONFIG_PPC64)) : "cr0", "memory"); return likely(prev == old); } +/* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ +static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 val) +{ +return __trylock_cmpxchg (lock, val, _Q_LOCKED_VAL); +} + +/* Take lock, preserving tail, cmpxchg with val (which must not be locked) */ +static __always_inline int trylock_with_tail_cpu(struct qspinlock *lock, u32 val) +{ +u32 newval = _Q_LOCKED_VAL | (val & _Q_TAIL_CPU_MASK); + +return __trylock_cmpxchg (lock, val, newval); +} +/* * Publish our tail, replacing previous tail. Return previous value. *
@@ -122,6 +144,30 @@ static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)BUG (); } +static inline bool try_to_steal_lock (struct qspinlock *lock) +{ +int iters =0 ; + +if (!maybe_stealers) +return false; + +/* Attempt to steal the lock */ +do { +u32 val =READ_ONCE (lock->val); + +if (unlikely (!(val & _Q_LOCKED_VAL))) { +if (trylock_with_tail_cpu (lock, val)) +return true; + }else { +cpu_relax (); + } + + iters++; + }while (iters <get_steal_spins ()); + +return false; +} +static inline void queued_spin_lock_mcs_queue (struct qspinlock *lock) {struct qnodes *qnodesp;
@@ -171,25 +217,49 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock)smp_rmb ();/* acquire barrier for the mcs lock */ } -/* We're at the head of the waitqueue, wait for the lock. */ -for (;;) { - val =READ_ONCE (lock->val); -if (!(val & _Q_LOCKED_VAL)) -break ; +if (!maybe_stealers) { +/* We're at the head of the waitqueue, wait for the lock. */ +for (;;) { + val =READ_ONCE (lock->val); +if (!(val & _Q_LOCKED_VAL)) +break ; -cpu_relax (); - } +cpu_relax (); + } + +/* If we're the last queued, must clean up the tail. */ +if ((val & _Q_TAIL_CPU_MASK) == tail) { +if (trylock_clear_tail_cpu (lock, val)) + goto release; +/* Another waiter must have enqueued. */ + } + +/* We must be the owner, just set the lock bit and acquire */ +set_locked (lock); + }else { +again: +/* We're at the head of the waitqueue, wait for the lock. */ +for (;;) { + val =READ_ONCE (lock->val); +if (!(val & _Q_LOCKED_VAL)) +break ; -/* If we're the last queued, must clean up the tail. */ -if ((val & _Q_TAIL_CPU_MASK) == tail) { -if (trylock_clear_tail_cpu (lock, val)) - goto release; -/* Another waiter must have enqueued */ +cpu_relax (); + } + +/* If we're the last queued, must clean up the tail. */ +if ((val & _Q_TAIL_CPU_MASK) == tail) { +if (trylock_clear_tail_cpu (lock, val)) + goto release; +/* Another waiter must have enqueued, or lock stolen. */ + }else { +if (trylock_with_tail_cpu (lock, val)) + goto unlock_next; + } + goto again; } -/* We must be the owner, just set the lock bit and acquire */ -set_locked (lock); - +unlock_next:/* contended path; must wait for next != NULL (MCS protocol) */ while (!(next =READ_ONCE (node->next)))cpu_relax ();
@@ -209,6 +279,9 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock)void queued_spin_lock_slowpath (struct qspinlock *lock) { +if (try_to_steal_lock (lock)) +return ; +queued_spin_lock_mcs_queue (lock); }EXPORT_SYMBOL (queued_spin_lock_slowpath);
@@ -218,3 +291,51 @@ void pv_spinlocks_init(void) { }#endif + +#include <linux/debugfs.h> +static int steal_spins_set (void *data,u64 val) +{ +static DEFINE_MUTEX (lock); + +/* +* The lock slow path has a !maybe_stealers case that can assume +* the head of queue will not see concurrent waiters. That waiter +* is unsafe in the presence of stealers, so must keep them away +* from one another. +*/ + +mutex_lock (&lock); +if (val && !steal_spins) { + maybe_stealers = true; +/* wait for queue head waiter to go away */ +synchronize_rcu (); + steal_spins = val; + }else if (!val && steal_spins) { + steal_spins = val; +/* wait for all possible stealers to go away */ +synchronize_rcu (); + maybe_stealers = false; + }else { + steal_spins = val; + } +mutex_unlock (&lock); + +return 0 ; +} + +static int steal_spins_get (void *data,u64 *val) +{ + *val = steal_spins; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_steal_spins, steal_spins_get, steal_spins_set,"%llu\n" ); + +static __init int spinlock_debugfs_init (void ) +{ +debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins); + +return 0 ; +} +device_initcall (spinlock_debugfs_init);
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 6/17 powerpc/qspinlock: theft prevention to control latency
In reply to Nicholas PigginGive the queue head the ability to stop stealers. After a number of spins without sucessfully acquiring the lock, the queue head employs this, which will assure it is the next owner.
--- arch/powerpc/include/asm/qspinlock_types.h | 10 ++++- arch/powerpc/lib/qspinlock.c | 52 ++++++++++++++++++++++ 2 files changed, 60 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/qspinlock_types.h +8 -2
@@ -29,7 +29,8 @@ typedef struct qspinlock {* Bitfields in the lock word: * * 0: locked bit -* 16-31: tail cpu (+1) +* 16: must queue bit +* 17-31: tail cpu (+1) */ #define _Q_SET_MASK (type) (((1U << _Q_ ## type ## _BITS) - 1)\ << _Q_ ## type ## _OFFSET)
@@ -38,7 +39,12 @@ typedef struct qspinlock {#define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED)#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) -#define _Q_TAIL_CPU_OFFSET 16 +#define _Q_MUST_Q_OFFSET 16 +#define _Q_MUST_Q_BITS 1 +#define _Q_MUST_Q_MASK _Q_SET_MASK(MUST_Q) +#define _Q_MUST_Q_VAL (1U << _Q_MUST_Q_OFFSET) + +#define _Q_TAIL_CPU_OFFSET 17#define _Q_TAIL_CPU_BITS (32 - _Q_TAIL_CPU_OFFSET)#define _Q_TAIL_CPU_MASK _Q_SET_MASK(TAIL_CPU)
arch/powerpc/lib/qspinlock.c +52 -0
97 lines not on this page · raw message
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 7/17 powerpc/qspinlock: store owner CPU in lock word
In reply to Nicholas PigginStore the owner CPU number in the lock word so it may be yielded to, as powerpc's paravirtualised simple spinlocks do.
--- arch/powerpc/include/asm/qspinlock.h | 9 ++++++++- arch/powerpc/include/asm/qspinlock_types.h | 10 ++++++++++ arch/powerpc/lib/qspinlock.c | 9 ++++++--- 3 files changed, 24 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/qspinlock.h +8 -1
@@ -20,8 +20,15 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock)return !!(READ_ONCE (lock->val) & _Q_TAIL_CPU_MASK); } +static __always_inline u32 queued_spin_encode_locked_val(void) +{ +/* XXX: make this use lock value in paca like simple spinlocks? */ +return _Q_LOCKED_VAL | (smp_processor_id () << _Q_OWNER_CPU_OFFSET); +} + static __always_inline int queued_spin_trylock(struct qspinlock *lock) { +u32 new =queued_spin_encode_locked_val ();u32 prev; asm volatile(
@@ -33,7 +40,7 @@ static __always_inline int queued_spin_trylock(struct qspinlock *lock)"\t" PPC_ACQUIRE_BARRIER " \n" "2: \n" : "=&r" (prev) - : "r" (&lock->val), "r" (_Q_LOCKED_VAL), + : "r" (&lock->val), "r" (new)," i" (IS_ENABLED(CONFIG_PPC64)) : "cr0", "memory");
arch/powerpc/include/asm/qspinlock_types.h +10 -0
@@ -29,6 +29,8 @@ typedef struct qspinlock {* Bitfields in the lock word: * * 0: locked bit +* 1-14: lock holder cpu +* 15: unused bit * 16: must queue bit * 17-31: tail cpu (+1) */
@@ -39,6 +41,14 @@ typedef struct qspinlock {#define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED)#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) +#define _Q_OWNER_CPU_OFFSET 1 +#define _Q_OWNER_CPU_BITS 14 +#define _Q_OWNER_CPU_MASK _Q_SET_MASK(OWNER_CPU) + +#if CONFIG_NR_CPUS > (1U << _Q_OWNER_CPU_BITS) +#error "qspinlock does not support such large CONFIG_NR_CPUS" +#endif +#define _Q_MUST_Q_OFFSET 16#define _Q_MUST_Q_BITS 1#define _Q_MUST_Q_MASK _Q_SET_MASK(MUST_Q)
arch/powerpc/lib/qspinlock.c +6 -3
@@ -49,6 +49,7 @@ static inline int decode_tail_cpu(u32 val)/* Take the lock by setting the lock bit, no other CPUs will touch it. */ static __always_inline void set_locked(struct qspinlock *lock) { +u32 new =queued_spin_encode_locked_val ();u32 prev, tmp; asm volatile(
@@ -58,7 +59,7 @@ static __always_inline void set_locked(struct qspinlock *lock)" bne- 1b \n" "\t" PPC_ACQUIRE_BARRIER " \n" : "=&r" (prev), "=&r" (tmp) - : "r" (&lock->val), "i" (_Q_LOCKED_VAL), + : "r" (&lock->val), "r" (new)," i" (IS_ENABLED(CONFIG_PPC64)) : "cr0", "memory");
@@ -90,13 +91,15 @@ static __always_inline u32 __trylock_cmpxchg(struct qspinlock *lock, u32 old, u3/* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 val) { -return __trylock_cmpxchg (lock, val, _Q_LOCKED_VAL); +u32 newval =queued_spin_encode_locked_val (); + +return __trylock_cmpxchg (lock, val, newval); }/* Take lock, preserving tail, cmpxchg with val (which must not be locked) */ static __always_inline int trylock_with_tail_cpu(struct qspinlock *lock, u32 val) { -u32 newval = _Q_LOCKED_VAL | (val & _Q_TAIL_CPU_MASK); +u32 newval =queued_spin_encode_locked_val () | (val & _Q_TAIL_CPU_MASK);return __trylock_cmpxchg (lock, val, newval); }
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 8/17 powerpc/qspinlock: paravirt yield to lock owner
In reply to Nicholas PigginWaiters spinning on the lock word should yield to the lock owner if the vCPU is preempted. This improves performance when the hypervisor has oversubscribed physical CPUs.
--- arch/powerpc/lib/qspinlock.c | 101 ++++++++++++++++++++++++++++++----- 1 file changed, 88 insertions(+), 13 deletions(-)
arch/powerpc/lib/qspinlock.c +88 -13
@@ -5,6 +5,7 @@#include <linux/percpu.h> #include <linux/smp.h> #include <asm/qspinlock.h> +#include <asm/paravirt.h> #define MAX_NODES 4
@@ -24,14 +25,16 @@ static int steal_spins __read_mostly = (1<<5);static bool maybe_stealers__read_mostly = true;static int head_spins__read_mostly = (1 <<8 ); +static bool pv_yield_owner__read_mostly = true; + static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); -static __always_inline int get_steal_spins(void) +static __always_inline int get_steal_spins(bool paravirt) {return steal_spins; } -static __always_inline int get_head_spins(void) +static __always_inline int get_head_spins(bool paravirt) {return head_spins; }
@@ -46,6 +49,11 @@ static inline int decode_tail_cpu(u32 val)return (val >> _Q_TAIL_CPU_OFFSET) -1 ; } +static inline int get_owner_cpu (u32 val) +{ +return (val & _Q_OWNER_CPU_MASK) >> _Q_OWNER_CPU_OFFSET; +} +/* Take the lock by setting the lock bit, no other CPUs will touch it. */ static __always_inline void set_locked(struct qspinlock *lock) {
@@ -169,7 +177,45 @@ static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)BUG (); } -static inline bool try_to_steal_lock (struct qspinlock *lock) +static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +{ +int owner; +u32 yield_count; + +BUG_ON (!(val & _Q_LOCKED_VAL)); + +if (!paravirt) + goto relax; + +if (!pv_yield_owner) + goto relax; + + owner =get_owner_cpu (val); + yield_count =yield_count_of (owner); + +if ((yield_count &1 ) ==0 ) + goto relax;/* owner vcpu is running */ + +/* +* Read the lock word after sampling the yield count. On the other side +* there may a wmb because the yield count update is done by the +* hypervisor preemption and the value update by the OS, however this +* ordering might reduce the chance of out of order accesses and +* improve the heuristic. +*/ +smp_rmb (); + +if (READ_ONCE (lock->val) == val) { +yield_to_preempted (owner, yield_count); +/* Don't relax if we yielded. Maybe we should? */ +return ; + } +relax: +cpu_relax (); +} + + +static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt) {int iters =0 ;
@@ -187,16 +233,16 @@ static inline bool try_to_steal_lock(struct qspinlock *lock)if (trylock_with_tail_cpu (lock, val))return true; }else { -cpu_relax (); +yield_to_locked_owner (lock, val, paravirt); } iters++; - }while (iters <get_steal_spins ()); + }while (iters <get_steal_spins (paravirt));return false; } -static inline void queued_spin_lock_mcs_queue (struct qspinlock *lock) +static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bool paravirt) {struct qnodes *qnodesp;struct qnode *next, *node;
@@ -252,7 +298,7 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock)if (!(val & _Q_LOCKED_VAL))break ; -cpu_relax (); +yield_to_locked_owner (lock, val, paravirt); }/* If we're the last queued, must clean up the tail. */
@@ -275,10 +321,10 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock)if (!(val & _Q_LOCKED_VAL))break ; -cpu_relax (); +yield_to_locked_owner (lock, val, paravirt); iters++; -if (!mustq && iters >=get_head_spins ()) { +if (!mustq && iters >=get_head_spins (paravirt)) { mustq = true;set_mustq (lock); val |= _Q_MUST_Q_VAL;
@@ -317,10 +363,20 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock)void queued_spin_lock_slowpath (struct qspinlock *lock) { -if (try_to_steal_lock (lock)) -return ; - -queued_spin_lock_mcs_queue (lock); +/* +* This looks funny, but it induces the compiler to inline both +* sides of the branch rather than share code as when the condition +* is passed as the paravirt argument to the functions. +*/ +if (IS_ENABLED (CONFIG_PARAVIRT_SPINLOCKS ) &&is_shared_processor ()) { +if (try_to_steal_lock (lock, true)) +return ; +queued_spin_lock_mcs_queue (lock, true); + }else { +if (try_to_steal_lock (lock, false)) +return ; +queued_spin_lock_mcs_queue (lock, false); + } }EXPORT_SYMBOL (queued_spin_lock_slowpath);
@@ -386,10 +442,29 @@ static int head_spins_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_head_spins, head_spins_get, head_spins_set,"%llu\n" ); +static int pv_yield_owner_set (void *data,u64 val) +{ + pv_yield_owner = !!val; + +return 0 ; +} + +static int pv_yield_owner_get (void *data,u64 *val) +{ + *val = pv_yield_owner; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set,"%llu\n" ); +static __init int spinlock_debugfs_init (void ) {debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins);debugfs_create_file ("qspl_head_spins" ,0600 , arch_debugfs_dir,NULL , &fops_head_spins); +if (is_shared_processor ()) { +debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner); + }return 0 ; }
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 9/17 powerpc/qspinlock: implement option to yield to previous node
In reply to Nicholas PigginQueued waiters which are not at the head of the queue don't spin on the lock word but their qnode lock word, waiting for the previous queued CPU to release them. Add an option which allows these waiters to yield to the previous CPU if its vCPU is preempted.
--- arch/powerpc/lib/qspinlock.c | 46 +++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-)
arch/powerpc/lib/qspinlock.c +45 -1
@@ -26,6 +26,7 @@ static bool maybe_stealers __read_mostly = true;static int head_spins__read_mostly = (1 <<8 );static bool pv_yield_owner__read_mostly = true; +static bool pv_yield_prev__read_mostly = true; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes);
@@ -214,6 +215,32 @@ static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 vacpu_relax (); } +static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt) +{ +int prev_cpu =decode_tail_cpu (val); +u32 yield_count; + +if (!paravirt) + goto relax; + +if (!pv_yield_prev) + goto relax; + + yield_count =yield_count_of (prev_cpu); +if ((yield_count &1 ) ==0 ) + goto relax;/* owner vcpu is running */ + +smp_rmb ();/* See yield_to_locked_owner comment */ + +if (!node->locked) { +yield_to_preempted (prev_cpu, yield_count); +return ; + } + +relax: +cpu_relax (); +} + static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt) {
@@ -286,7 +313,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b/* Wait for mcs node lock to be released */ while (!node->locked) -cpu_relax (); +yield_to_prev (lock, node, old, paravirt);smp_rmb ();/* acquire barrier for the mcs lock */ }
@@ -458,12 +485,29 @@ static int pv_yield_owner_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set,"%llu\n" ); +static int pv_yield_prev_set (void *data,u64 val) +{ + pv_yield_prev = !!val; + +return 0 ; +} + +static int pv_yield_prev_get (void *data,u64 *val) +{ + *val = pv_yield_prev; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set,"%llu\n" ); +static __init int spinlock_debugfs_init (void ) {debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins);debugfs_create_file ("qspl_head_spins" ,0600 , arch_debugfs_dir,NULL , &fops_head_spins);if (is_shared_processor ()) {debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner); +debugfs_create_file ("qspl_pv_yield_prev" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_prev); }return 0 ;
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 10/17 powerpc/qspinlock: allow stealing when head of queue yields
In reply to Nicholas PigginIf the head of queue is preventing stealing but it finds the owner vCPU is preempted, it will yield its cycles to the owner which could cause it to become preempted. Add an option to re-allow stealers before yielding, and disallow them again after returning from the yield.
Disable this option by default for now, i.e., no logical change.
--- arch/powerpc/lib/qspinlock.c | 61 +++++++++++++++++++++++++++++++++--- 1 file changed, 57 insertions(+), 4 deletions(-)
arch/powerpc/lib/qspinlock.c +57 -4
@@ -26,6 +26,7 @@ static bool maybe_stealers __read_mostly = true;static int head_spins__read_mostly = (1 <<8 );static bool pv_yield_owner__read_mostly = true; +static bool pv_yield_allow_steal__read_mostly = false;static bool pv_yield_prev__read_mostly = true; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes);
@@ -154,6 +155,22 @@ static __always_inline u32 set_mustq(struct qspinlock *lock)return prev; } +static __always_inline u32 clear_mustq(struct qspinlock *lock) +{ +u32 prev; + + asmvolatile ( +"1: lwarx %0,0,%1 # clear_mustq \n" +" andc %0,%0,%2 \n" +" stwcx. %0,0,%1 \n" +" bne- 1b \n" + :"=&r" (prev) + :"r" (&lock->val),"r" (_Q_MUST_Q_VAL) + :"cr0" ,"memory" ); + +return prev; +} +static struct qnode *get_tail_qnode (struct qspinlock *lock,u32 val) {int cpu =decode_tail_cpu (val);
@@ -178,7 +195,7 @@ static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)BUG (); } -static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool mustq) {int owner;u32 yield_count;
@@ -207,7 +224,11 @@ static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 vasmp_rmb ();if (READ_ONCE (lock->val) == val) { +if (mustq) +clear_mustq (lock);yield_to_preempted (owner, yield_count); +if (mustq) +set_mustq (lock);/* Don't relax if we yielded. Maybe we should? */ return ; }
@@ -215,6 +236,21 @@ static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 vacpu_relax (); } +static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +{ +__yield_to_locked_owner (lock, val, paravirt, false); +} + +static __always_inline void yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +{ +bool mustq = false; + +if ((val & _Q_MUST_Q_VAL) && pv_yield_allow_steal) + mustq = true; + +__yield_to_locked_owner (lock, val, paravirt, mustq); +} + static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt) {int prev_cpu =decode_tail_cpu (val);
@@ -230,7 +266,7 @@ static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *if ((yield_count &1 ) ==0 ) goto relax;/* owner vcpu is running */ -smp_rmb ();/* See yield_to_locked_owner comment */ +smp_rmb ();/* See __yield_to_locked_owner comment */ if (!node->locked) {yield_to_preempted (prev_cpu, yield_count);
@@ -325,7 +361,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bif (!(val & _Q_LOCKED_VAL))break ; -yield_to_locked_owner (lock, val, paravirt); +yield_head_to_locked_owner (lock, val, paravirt); }/* If we're the last queued, must clean up the tail. */
@@ -348,7 +384,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bif (!(val & _Q_LOCKED_VAL))break ; -yield_to_locked_owner (lock, val, paravirt); +yield_head_to_locked_owner (lock, val, paravirt); iters++;if (!mustq && iters >=get_head_spins (paravirt)) {
@@ -485,6 +521,22 @@ static int pv_yield_owner_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set,"%llu\n" ); +static int pv_yield_allow_steal_set (void *data,u64 val) +{ + pv_yield_allow_steal = !!val; + +return 0 ; +} + +static int pv_yield_allow_steal_get (void *data,u64 *val) +{ + *val = pv_yield_allow_steal; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_allow_steal, pv_yield_allow_steal_get, pv_yield_allow_steal_set,"%llu\n" ); +static int pv_yield_prev_set (void *data,u64 val) { pv_yield_prev = !!val;
@@ -507,6 +559,7 @@ static __init int spinlock_debugfs_init(void)debugfs_create_file ("qspl_head_spins" ,0600 , arch_debugfs_dir,NULL , &fops_head_spins);if (is_shared_processor ()) {debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner); +debugfs_create_file ("qspl_pv_yield_allow_steal" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_allow_steal);debugfs_create_file ("qspl_pv_yield_prev" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_prev); }
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 11/17 powerpc/qspinlock: allow propagation of yield CPU down the queue
In reply to Nicholas PigginHaving all CPUs poll the lock word for the owner CPU that should be yielded to defeats most of the purpose of using MCS queueing for scalability. Yet it may be desirable for queued waiters to to yield to a preempted owner.
s390 addreses this problem by having queued waiters sample the lock word to find the owner much less frequently. In this approach, the waiters never sample it directly, but the queue head propagates the owner CPU back to the next waiter if it ever finds the owner has been preempted. Queued waiters then subsequently propagate the owner CPU back to the next waiter, and so on.
--- arch/powerpc/lib/qspinlock.c | 82 ++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)
arch/powerpc/lib/qspinlock.c +82 -0
@@ -12,6 +12,7 @@struct qnode {struct qnode *next;struct qspinlock *lock; +int yield_cpu;u8 locked;/* 1 if lock acquired */ };
@@ -28,6 +29,7 @@ static int head_spins __read_mostly = (1<<8);static bool pv_yield_owner__read_mostly = true;static bool pv_yield_allow_steal__read_mostly = false;static bool pv_yield_prev__read_mostly = true; +static bool pv_yield_propagate_owner__read_mostly = true; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes);
@@ -251,14 +253,67 @@ static __always_inline void yield_head_to_locked_owner(struct qspinlock *lock, u__yield_to_locked_owner (lock, val, paravirt, mustq); } +static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int *set_yield_cpu, bool paravirt) +{ +struct qnode *next; +int owner; + +if (!paravirt) +return ; +if (!pv_yield_propagate_owner) +return ; + + owner =get_owner_cpu (val); +if (*set_yield_cpu == owner) +return ; + + next =READ_ONCE (node->next); +if (!next) +return ; + +if (vcpu_is_preempted (owner)) { + next->yield_cpu = owner; + *set_yield_cpu = owner; + }else if (*set_yield_cpu !=-1 ) { + next->yield_cpu = owner; + *set_yield_cpu = owner; + } +} + static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt) {int prev_cpu =decode_tail_cpu (val);u32 yield_count; +int yield_cpu;if (!paravirt) goto relax; +if (!pv_yield_propagate_owner) + goto yield_prev; + + yield_cpu =READ_ONCE (node->yield_cpu); +if (yield_cpu ==-1 ) { +/* Propagate back the -1 CPU */ +if (node->next && node->next->yield_cpu !=-1 ) + node->next->yield_cpu = yield_cpu; + goto yield_prev; + } + + yield_count =yield_count_of (yield_cpu); +if ((yield_count &1 ) ==0 ) + goto yield_prev;/* owner vcpu is running */ + +smp_rmb (); + +if (yield_cpu == node->yield_cpu) { +if (node->next && node->next->yield_cpu != yield_cpu) + node->next->yield_cpu = yield_cpu; +yield_to_preempted (yield_cpu, yield_count); +return ; + } + +yield_prev:if (!pv_yield_prev) goto relax;
@@ -331,6 +386,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node = &qnodesp->nodes[idx]; node->next =NULL ; node->lock = lock; + node->yield_cpu =-1 ; node->locked =0 ; tail =encode_tail_cpu (smp_processor_id ());
@@ -351,16 +407,23 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bwhile (!node->locked)yield_to_prev (lock, node, old, paravirt); +/* Clear out stale propagated yield_cpu */ +if (paravirt && pv_yield_propagate_owner && node->yield_cpu !=-1 ) + node->yield_cpu =-1 ; +smp_rmb ();/* acquire barrier for the mcs lock */ }if (!maybe_stealers) { +int set_yield_cpu =-1 ; +/* We're at the head of the waitqueue, wait for the lock. */ for (;;) { val =READ_ONCE (lock->val);if (!(val & _Q_LOCKED_VAL))break ; +propagate_yield_cpu (node, val, &set_yield_cpu, paravirt);yield_head_to_locked_owner (lock, val, paravirt); }
@@ -374,6 +437,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b/* We must be the owner, just set the lock bit and acquire */ set_locked (lock); }else { +int set_yield_cpu =-1 ;int iters =0 ;bool mustq = false;
@@ -384,6 +448,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bif (!(val & _Q_LOCKED_VAL))break ; +propagate_yield_cpu (node, val, &set_yield_cpu, paravirt);yield_head_to_locked_owner (lock, val, paravirt); iters++;
@@ -553,6 +618,22 @@ static int pv_yield_prev_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set,"%llu\n" ); +static int pv_yield_propagate_owner_set (void *data,u64 val) +{ + pv_yield_propagate_owner = !!val; + +return 0 ; +} + +static int pv_yield_propagate_owner_get (void *data,u64 *val) +{ + *val = pv_yield_propagate_owner; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set,"%llu\n" ); +static __init int spinlock_debugfs_init (void ) {debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins);
@@ -561,6 +642,7 @@ static __init int spinlock_debugfs_init(void)debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner);debugfs_create_file ("qspl_pv_yield_allow_steal" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_allow_steal);debugfs_create_file ("qspl_pv_yield_prev" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_prev); +debugfs_create_file ("qspl_pv_yield_propagate_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_propagate_owner); }return 0 ;
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 12/17 powerpc/qspinlock: add ability to prod new queue head CPU
In reply to Nicholas PigginAfter the head of the queue acquires the lock, it releases the next waiter in the queue to become the new head. Add an option to prod the new head if its vCPU was preempted. This may only have an effect if queue waiters are yielding.
Disable this option by default for now, i.e., no logical change.
--- arch/powerpc/lib/qspinlock.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
arch/powerpc/lib/qspinlock.c +28 -1
@@ -12,6 +12,7 @@struct qnode {struct qnode *next;struct qspinlock *lock; +int cpu;int yield_cpu;u8 locked;/* 1 if lock acquired */ };
@@ -30,6 +31,7 @@ static bool pv_yield_owner __read_mostly = true;static bool pv_yield_allow_steal__read_mostly = false;static bool pv_yield_prev__read_mostly = true;static bool pv_yield_propagate_owner__read_mostly = true; +static bool pv_prod_head__read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes);
@@ -386,6 +388,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node = &qnodesp->nodes[idx]; node->next =NULL ; node->lock = lock; + node->cpu =smp_processor_id (); node->yield_cpu =-1 ; node->locked =0 ;
@@ -483,7 +486,14 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b* this store to locked. The corresponding barrier is the smp_rmb() * acquire barrier for mcs lock, above. */ -WRITE_ONCE (next->locked,1 ); +if (paravirt && pv_prod_head) { +int next_cpu = next->cpu; +WRITE_ONCE (next->locked,1 ); +if (vcpu_is_preempted (next_cpu)) +prod_cpu (next_cpu); + }else { +WRITE_ONCE (next->locked,1 ); + } release: qnodesp->count--;/* release the node */
@@ -634,6 +644,22 @@ static int pv_yield_propagate_owner_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set,"%llu\n" ); +static int pv_prod_head_set (void *data,u64 val) +{ + pv_prod_head = !!val; + +return 0 ; +} + +static int pv_prod_head_get (void *data,u64 *val) +{ + *val = pv_prod_head; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_prod_head, pv_prod_head_get, pv_prod_head_set,"%llu\n" ); +static __init int spinlock_debugfs_init (void ) {debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins);
@@ -643,6 +669,7 @@ static __init int spinlock_debugfs_init(void)debugfs_create_file ("qspl_pv_yield_allow_steal" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_allow_steal);debugfs_create_file ("qspl_pv_yield_prev" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_prev);debugfs_create_file ("qspl_pv_yield_propagate_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_propagate_owner); +debugfs_create_file ("qspl_pv_prod_head" ,0600 , arch_debugfs_dir,NULL , &fops_pv_prod_head); }return 0 ;
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 13/17 powerpc/qspinlock: trylock and initial lock attempt may steal
In reply to Nicholas PigginThis gives trylock slightly more strength, and it also gives most of the benefit of passing 'val' back through the slowpath without the complexity.
--- arch/powerpc/include/asm/qspinlock.h | 44 +++++++++++++++++++++++++++- arch/powerpc/lib/qspinlock.c | 9 ++++++ 2 files changed, 52 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/qspinlock.h +43 -1
@@ -5,6 +5,15 @@#include <linux/compiler.h> #include <asm/qspinlock_types.h> +/* +* The trylock itself may steal. This makes trylocks slightly stronger, and +* might make spin locks slightly more efficient when stealing. +* +* This is compile-time, so if true then there may always be stealers, so the +* nosteal paths become unused. +*/ +#define _Q_SPIN_TRY_LOCK_STEAL 1 + static __always_inline int queued_spin_is_locked(struct qspinlock *lock) {return READ_ONCE (lock->val);
@@ -26,11 +35,12 @@ static __always_inline u32 queued_spin_encode_locked_val(void)return _Q_LOCKED_VAL | (smp_processor_id () << _Q_OWNER_CPU_OFFSET); } -static __always_inline int queued_spin_trylock(struct qspinlock *lock) +static __always_inline int __queued_spin_trylock_nosteal(struct qspinlock *lock) {u32 new =queued_spin_encode_locked_val ();u32 prev; +/* Trylock succeeds only when unlocked and no queued nodes */ asmvolatile ("1: lwarx %0,0,%1,%3 # queued_spin_trylock \n" " cmpwi 0,%0,0 \n"
@@ -47,6 +57,38 @@ static __always_inline int queued_spin_trylock(struct qspinlock *lock)return likely (prev ==0 ); } +static __always_inline int __queued_spin_trylock_steal(struct qspinlock *lock) +{ +u32 new =queued_spin_encode_locked_val (); +u32 prev, tmp; + +/* Trylock may get ahead of queued nodes if it finds unlocked */ + asmvolatile ( +"1: lwarx %0,0,%2,%5 # queued_spin_trylock \n" +" andc. %1,%0,%4 \n" +" bne- 2f \n" +" and %1,%0,%4 \n" +" or %1,%1,%3 \n" +" stwcx. %1,0,%2 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + :"=&r" (prev),"=&r" (tmp) + :"r" (&lock->val),"r" (new),"r" (_Q_TAIL_CPU_MASK), +"i" (IS_ENABLED (CONFIG_PPC64 )) + :"cr0" ,"memory" ); + +return likely (!(prev & ~_Q_TAIL_CPU_MASK)); +} + +static __always_inline int queued_spin_trylock(struct qspinlock *lock) +{ +if (!_Q_SPIN_TRY_LOCK_STEAL) +return __queued_spin_trylock_nosteal (lock); +else +return __queued_spin_trylock_steal (lock); +} +void queued_spin_lock_slowpath (struct qspinlock *lock); static __always_inline void queued_spin_lock(struct qspinlock *lock)
arch/powerpc/lib/qspinlock.c +9 -0
@@ -24,7 +24,11 @@ struct qnodes {/* Tuning parameters */ static int steal_spins__read_mostly = (1 <<5 ); +#if _Q_SPIN_TRY_LOCK_STEAL ==1 +static const bool maybe_stealers = true; +#else static bool maybe_stealers__read_mostly = true; +#endif static int head_spins__read_mostly = (1 <<8 );static bool pv_yield_owner__read_mostly = true;
@@ -527,6 +531,10 @@ void pv_spinlocks_init(void)#include <linux/debugfs.h> static int steal_spins_set (void *data,u64 val) { +#if _Q_SPIN_TRY_LOCK_STEAL ==1 +/* MAYBE_STEAL remains true */ + steal_spins = val; +#else static DEFINE_MUTEX (lock);/*
@@ -551,6 +559,7 @@ static int steal_spins_set(void *data, u64 val) steal_spins = val; }mutex_unlock (&lock); +#endif return 0 ; }
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 14/17 powerpc/qspinlock: use spin_begin/end API
In reply to Nicholas PigginUse the spin_begin/spin_cpu_relax/spin_end APIs in qspinlock, which helps to prevent threads issuing a lot of expensive priority nops which may not have much effect due to immediately executing low then medium priority.
--- arch/powerpc/lib/qspinlock.c | 41 ++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-)
arch/powerpc/lib/qspinlock.c +37 -4
@@ -203,6 +203,7 @@ static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)BUG (); } +/* Called inside spin_begin() */ static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool mustq) {int owner;
@@ -222,6 +223,8 @@ static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32if ((yield_count &1 ) ==0 ) goto relax;/* owner vcpu is running */ +spin_end (); +/* * Read the lock word after sampling the yield count. On the other side * there may a wmb because the yield count update is done by the
@@ -237,18 +240,22 @@ static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32yield_to_preempted (owner, yield_count);if (mustq)set_mustq (lock); +spin_begin ();/* Don't relax if we yielded. Maybe we should? */ return ; } +spin_begin (); relax: -cpu_relax (); +spin_cpu_relax (); } +/* Called inside spin_begin() */ static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) {__yield_to_locked_owner (lock, val, paravirt, false); } +/* Called inside spin_begin() */ static __always_inline void yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) {bool mustq = false;
@@ -286,6 +293,7 @@ static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int } } +/* Called inside spin_begin() */ static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt) {int prev_cpu =decode_tail_cpu (val);
@@ -310,14 +318,18 @@ static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *if ((yield_count &1 ) ==0 ) goto yield_prev;/* owner vcpu is running */ +spin_end (); +smp_rmb ();if (yield_cpu == node->yield_cpu) {if (node->next && node->next->yield_cpu != yield_cpu) node->next->yield_cpu = yield_cpu;yield_to_preempted (yield_cpu, yield_count); +spin_begin ();return ; } +spin_begin (); yield_prev:if (!pv_yield_prev)
@@ -327,15 +339,19 @@ static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *if ((yield_count &1 ) ==0 ) goto relax;/* owner vcpu is running */ +spin_end (); +smp_rmb ();/* See __yield_to_locked_owner comment */ if (!node->locked) {yield_to_preempted (prev_cpu, yield_count); +spin_begin ();return ; } +spin_begin (); relax: -cpu_relax (); +spin_cpu_relax (); }
@@ -347,6 +363,8 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravreturn false;/* Attempt to steal the lock */ +spin_begin (); + do { u32 val = READ_ONCE(lock->val);
@@ -354,8 +372,10 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravbreak ;if (unlikely (!(val & _Q_LOCKED_VAL))) { +spin_end ();if (trylock_with_tail_cpu (lock, val))return true; +spin_begin (); }else {yield_to_locked_owner (lock, val, paravirt); }
@@ -363,6 +383,8 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav iters++; }while (iters <get_steal_spins (paravirt)); +spin_end (); +return false; }
@@ -411,8 +433,10 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bWRITE_ONCE (prev->next, node);/* Wait for mcs node lock to be released */ +spin_begin ();while (!node->locked)yield_to_prev (lock, node, old, paravirt); +spin_end ();/* Clear out stale propagated yield_cpu */ if (paravirt && pv_yield_propagate_owner && node->yield_cpu !=-1 )
@@ -425,6 +449,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bint set_yield_cpu =-1 ;/* We're at the head of the waitqueue, wait for the lock. */ +spin_begin ();for (;;) { val =READ_ONCE (lock->val);if (!(val & _Q_LOCKED_VAL))
@@ -433,6 +458,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bpropagate_yield_cpu (node, val, &set_yield_cpu, paravirt);yield_head_to_locked_owner (lock, val, paravirt); } +spin_end ();/* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) {
@@ -450,6 +476,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b again:/* We're at the head of the waitqueue, wait for the lock. */ +spin_begin ();for (;;) { val =READ_ONCE (lock->val);if (!(val & _Q_LOCKED_VAL))
@@ -465,6 +492,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b val |= _Q_MUST_Q_VAL; } } +spin_end ();/* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) {
@@ -480,8 +508,13 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b unlock_next:/* contended path; must wait for next != NULL (MCS protocol) */ -while (!(next =READ_ONCE (node->next))) -cpu_relax (); + next =READ_ONCE (node->next); +if (!next) { +spin_begin (); +while (!(next =READ_ONCE (node->next))) +cpu_relax (); +spin_end (); + }/* * Unlock the next mcs waiter node. Release barrier is not required
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 15/17 powerpc/qspinlock: reduce remote node steal spins
In reply to Nicholas PigginAllow for a reduction in the number of times a CPU from a different node than the owner can attempt to steal the lock before queueing. This could bias the transfer behaviour of the lock across the machine and reduce NUMA crossings.
--- arch/powerpc/lib/qspinlock.c | 43 +++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-)
arch/powerpc/lib/qspinlock.c +40 -3
@@ -4,6 +4,7 @@#include <linux/export.h> #include <linux/percpu.h> #include <linux/smp.h> +#include <linux/topology.h> #include <asm/qspinlock.h> #include <asm/paravirt.h>
@@ -24,6 +25,7 @@ struct qnodes {/* Tuning parameters */ static int steal_spins__read_mostly = (1 <<5 ); +static int remote_steal_spins__read_mostly = (1 <<2 );#if _Q_SPIN_TRY_LOCK_STEAL ==1 static const bool maybe_stealers = true;#else
@@ -44,6 +46,11 @@ static __always_inline int get_steal_spins(bool paravirt)return steal_spins; } +static __always_inline int get_remote_steal_spins(bool paravirt) +{ +return remote_steal_spins; +} + static __always_inline int get_head_spins(bool paravirt) {return head_spins;
@@ -354,10 +361,24 @@ static __always_inline void yield_to_prev(struct qspinlock *lock, struct qnode *spin_cpu_relax (); } +static __always_inline bool steal_break(u32 val, int iters, bool paravirt) +{ +if (iters >=get_steal_spins (paravirt)) +return true; + +if (IS_ENABLED (CONFIG_NUMA ) && + (iters >=get_remote_steal_spins (paravirt))) { +int cpu =get_owner_cpu (val); +if (numa_node_id () !=cpu_to_node (cpu)) +return true; + } +return false; +} static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt) {int iters =0 ; +u32 val;if (!maybe_stealers)return false;
@@ -366,8 +387,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravspin_begin (); do { - u32 val = READ_ONCE(lock->val); - + val = READ_ONCE(lock->val); if (val & _Q_MUST_Q_VAL) break;
@@ -381,7 +401,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav } iters++; - }while (iters <get_steal_spins (paravirt)); + }while (!steal_break (val, iters, paravirt));spin_end ();
@@ -606,6 +626,22 @@ static int steal_spins_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_steal_spins, steal_spins_get, steal_spins_set,"%llu\n" ); +static int remote_steal_spins_set (void *data,u64 val) +{ + remote_steal_spins = val; + +return 0 ; +} + +static int remote_steal_spins_get (void *data,u64 *val) +{ + *val = remote_steal_spins; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_remote_steal_spins, remote_steal_spins_get, remote_steal_spins_set,"%llu\n" ); +static int head_spins_set (void *data,u64 val) { head_spins = val;
@@ -705,6 +741,7 @@ DEFINE_SIMPLE_ATTRIBUTE(fops_pv_prod_head, pv_prod_head_get, pv_prod_head_set, "static __init int spinlock_debugfs_init (void ) {debugfs_create_file ("qspl_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_steal_spins); +debugfs_create_file ("qspl_remote_steal_spins" ,0600 , arch_debugfs_dir,NULL , &fops_remote_steal_spins);debugfs_create_file ("qspl_head_spins" ,0600 , arch_debugfs_dir,NULL , &fops_head_spins);if (is_shared_processor ()) {debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner);
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 16/17 powerpc/qspinlock: allow indefinite spinning on a preempted owner
In reply to Nicholas PigginProvide an option that holds off queueing indefinitely while the lock owner is preempted. This could reduce queueing latencies for very overcommitted vcpu situations.
This is disabled by default.
--- arch/powerpc/lib/qspinlock.c | 74 ++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 15 deletions(-)
arch/powerpc/lib/qspinlock.c +59 -15
@@ -35,6 +35,7 @@ static int head_spins __read_mostly = (1<<8);static bool pv_yield_owner__read_mostly = true;static bool pv_yield_allow_steal__read_mostly = false; +static bool pv_spin_on_preempted_owner__read_mostly = false;static bool pv_yield_prev__read_mostly = true;static bool pv_yield_propagate_owner__read_mostly = true;static bool pv_prod_head__read_mostly = false;
@@ -210,11 +211,12 @@ static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)BUG (); } -/* Called inside spin_begin() */ -static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool mustq) +/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */ +static __always_inline bool __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool mustq) {int owner;u32 yield_count; +bool preempted = false;BUG_ON (!(val & _Q_LOCKED_VAL));
@@ -232,6 +234,8 @@ static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32spin_end (); + preempted = true; +/* * Read the lock word after sampling the yield count. On the other side * there may a wmb because the yield count update is done by the
@@ -248,29 +252,32 @@ static __always_inline void __yield_to_locked_owner(struct qspinlock *lock, u32if (mustq)set_mustq (lock);spin_begin (); +/* Don't relax if we yielded. Maybe we should? */ -return ; +return preempted; }spin_begin (); relax:spin_cpu_relax (); + +return preempted; } -/* Called inside spin_begin() */ -static __always_inline void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */ +static __always_inline bool yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) { -__yield_to_locked_owner (lock, val, paravirt, false); +return __yield_to_locked_owner (lock, val, paravirt, false); } -/* Called inside spin_begin() */ -static __always_inline void yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */ +static __always_inline bool yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) {bool mustq = false;if ((val & _Q_MUST_Q_VAL) && pv_yield_allow_steal) mustq = true; -__yield_to_locked_owner (lock, val, paravirt, mustq); +return __yield_to_locked_owner (lock, val, paravirt, mustq); } static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int *set_yield_cpu, bool paravirt)
@@ -380,13 +387,16 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravint iters =0 ;u32 val; -if (!maybe_stealers) +if (!maybe_stealers) { +/* XXX: should spin_on_preempted_owner do anything here? */ return false; + }/* Attempt to steal the lock */ spin_begin (); - do { + bool preempted = false; + val = READ_ONCE(lock->val); if (val & _Q_MUST_Q_VAL) break;
@@ -397,10 +407,23 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravreturn true;spin_begin (); }else { -yield_to_locked_owner (lock, val, paravirt); + preempted =yield_to_locked_owner (lock, val, paravirt); } - iters++; +if (preempted) { +if (!pv_spin_on_preempted_owner) + iters++; +/* +* pv_spin_on_preempted_owner don't increase iters +* while the owner is preempted -- we won't interfere +* with it by definition. This could introduce some +* latency issue if we continually observe preempted +* owners, but hopefully that's a rare corner case of +* a badly oversubscribed system. +*/ + }else { + iters++; + } }while (!steal_break (val, iters, paravirt));spin_end ();
@@ -503,9 +526,13 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bbreak ;propagate_yield_cpu (node, val, &set_yield_cpu, paravirt); -yield_head_to_locked_owner (lock, val, paravirt); +if (yield_head_to_locked_owner (lock, val, paravirt)) { +if (!pv_spin_on_preempted_owner) + iters++; + }else { + iters++; + } - iters++;if (!mustq && iters >=get_head_spins (paravirt)) { mustq = true;set_mustq (lock);
@@ -690,6 +717,22 @@ static int pv_yield_allow_steal_get(void *data, u64 *val)DEFINE_SIMPLE_ATTRIBUTE (fops_pv_yield_allow_steal, pv_yield_allow_steal_get, pv_yield_allow_steal_set,"%llu\n" ); +static int pv_spin_on_preempted_owner_set (void *data,u64 val) +{ + pv_spin_on_preempted_owner = !!val; + +return 0 ; +} + +static int pv_spin_on_preempted_owner_get (void *data,u64 *val) +{ + *val = pv_spin_on_preempted_owner; + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (fops_pv_spin_on_preempted_owner, pv_spin_on_preempted_owner_get, pv_spin_on_preempted_owner_set,"%llu\n" ); +static int pv_yield_prev_set (void *data,u64 val) { pv_yield_prev = !!val;
@@ -746,6 +789,7 @@ static __init int spinlock_debugfs_init(void)if (is_shared_processor ()) {debugfs_create_file ("qspl_pv_yield_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_owner);debugfs_create_file ("qspl_pv_yield_allow_steal" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_allow_steal); +debugfs_create_file ("qspl_pv_spin_on_preempted_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_spin_on_preempted_owner);debugfs_create_file ("qspl_pv_yield_prev" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_prev);debugfs_create_file ("qspl_pv_yield_propagate_owner" ,0600 , arch_debugfs_dir,NULL , &fops_pv_yield_propagate_owner);debugfs_create_file ("qspl_pv_prod_head" ,0600 , arch_debugfs_dir,NULL , &fops_pv_prod_head);
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v2 17/17 powerpc/qspinlock: provide accounting and options for sleepy locks
In reply to Nicholas PigginFinding the owner or a queued waiter on a lock with a preempted vcpu is indicative of an oversubscribed guest causing the lock to get into trouble. Provide some options to detect this situation and have new CPUs avoid queueing for a longer time (more steal iterations) to minimise the problems caused by vcpu preemption on the queue.
--- arch/powerpc/include/asm/qspinlock_types.h | 7 +- arch/powerpc/lib/qspinlock.c | 244 +++++++++++++++++++-- 2 files changed, 232 insertions(+), 19 deletions(-)
arch/powerpc/include/asm/qspinlock_types.h +6 -1
@@ -30,7 +30,7 @@ typedef struct qspinlock {* * 0: locked bit * 1-14: lock holder cpu -* 15: unused bit +* 15: lock owner or queuer vcpus observed to be preempted bit * 16: must queue bit * 17-31: tail cpu (+1) */
@@ -49,6 +49,11 @@ typedef struct qspinlock {#error "qspinlock does not support such large CONFIG_NR_CPUS"#endif +#define _Q_SLEEPY_OFFSET 15 +#define _Q_SLEEPY_BITS 1 +#define _Q_SLEEPY_MASK _Q_SET_MASK(SLEEPY_OWNER) +#define _Q_SLEEPY_VAL (1U << _Q_SLEEPY_OFFSET) +#define _Q_MUST_Q_OFFSET 16#define _Q_MUST_Q_BITS 1#define _Q_MUST_Q_MASK _Q_SET_MASK(MUST_Q)
arch/powerpc/lib/qspinlock.c +226 -18
394 lines not on this page · raw message
Signature · 1 line
2.37.2