locking/qspinlock: Enhance pvqspinlock performance
Subsystems (3)
v7 cover locking/qspinlock: Enhance pvqspinlock performance
v6->v7:
- Removed arch/x86/include/asm/qspinlock.h from patch 1.
- Removed the unconditional PV kick patch as it has been merged into tip.
- Changed the pvstat_inc() API to add a new condition parameter.
- Added comments and rearrange code in patch 4 to clarify where lock stealing happened.
- In patch 5, removed the check for pv_wait count when deciding when to wait early.
- Updated copyrights and email address.
v5->v6:
- Added a new patch 1 to relax the cmpxchg and xchg operations in the native code path to reduce performance overhead on non-x86 architectures.
- Updated the unconditional PV kick patch as suggested by PeterZ.
- Added a new patch to allow one lock stealing attempt at slowpath entry point to reduce performance penalty due to lock waiter preemption.
- Removed the pending bit and kick-ahead patches as they didn't show any noticeable performance improvement on top of the lock stealing patch.
- Simplified the adaptive spinning patch as the lock stealing patch allows more aggressive pv_wait() without much performance penalty in non-overcommitted VMs.
v4->v5:
- Rebased the patch to the latest tip tree.
- Corrected the comments and commit log for patch 1.
- Removed the v4 patch 5 as PV kick deferment is no longer needed with the new tip tree.
- Simplified the adaptive spinning patch (patch 6) & improve its performance a bit further.
- Re-ran the benchmark test with the new patch.
v3->v4:
- Patch 1: add comment about possible racing condition in PV unlock.
- Patch 2: simplified the pv_pending_lock() function as suggested by Davidlohr.
- Move PV unlock optimization patch forward to patch 4 & rerun performance test.
v2->v3:
- Moved deferred kicking enablement patch forward & move back the kick-ahead patch to make the effect of kick-ahead more visible.
- Reworked patch 6 to make it more readable.
- Reverted back to use state as a tri-state variable instead of adding an additional bistate variable.
- Added performance data for different values of PV_KICK_AHEAD_MAX.
- Add a new patch to optimize PV unlock code path performance.
v1->v2:
- Take out the queued unfair lock patches
- Add a patch to simplify the PV unlock code
- Move pending bit and statistics collection patches to the front
- Keep vCPU kicking in pv_kick_node(), but defer it to unlock time when appropriate.
- Change the wait-early patch to use adaptive spinning to better balance the difference effect on normal and over-committed guests.
- Add patch-to-patch performance changes in the patch commit logs.
This patchset tries to improve the performance of both regular and over-commmitted VM guests. The adaptive spinning patch was inspired by the "Do Virtual Machines Really Scale?" blog from Sanidhya Kashyap.
Patch 1 relaxes the memory order restriction of atomic operations by using less restrictive _acquire and _release variants of cmpxchg() and xchg(). This will reduce performance overhead when ported to other non-x86 architectures.
Patch 2 optimizes the PV unlock code path performance for x86-64 architecture.
Patch 3 allows the collection of various count data that are useful to see what is happening in the system. They do add a bit of overhead when enabled slowing performance a tiny bit.
Patch 4 allows one lock stealing attempt at slowpath entry. This causes a pretty big performance improvement for over-committed VM guests.
Patch 5 enables adaptive spinning in the queue nodes. This patch leads to further performance improvement in over-committed guest, though it is not as big as the previous patch.
Waiman Long (5):
locking/qspinlock: relaxes cmpxchg & xchg ops in native code locking/pvqspinlock, x86: Optimize PV unlock code path locking/pvqspinlock: Collect slowpath lock statistics locking/pvqspinlock: Allow 1 lock stealing attempt locking/pvqspinlock: Queue node adaptive spinning
arch/x86/Kconfig | 9 + arch/x86/include/asm/qspinlock_paravirt.h | 59 +++++ include/asm-generic/qspinlock.h | 9 +- kernel/locking/qspinlock.c | 48 +++-- kernel/locking/qspinlock_paravirt.h | 376 +++++++++++++++++++++++++---- 5 files changed, 437 insertions(+), 64 deletions(-)
PATCH v7 1/5 locking/qspinlock: relaxes cmpxchg & xchg ops in native code
This patch replaces the cmpxchg() and xchg() calls in the native qspinlock code with more relaxed versions of those calls to enable other architectures to adopt queued spinlocks with less performance overhead.
--- include/asm-generic/qspinlock.h | 9 +++++---- kernel/locking/qspinlock.c | 24 +++++++++++++++++++----- 2 files changed, 24 insertions(+), 9 deletions(-)
include/asm-generic/qspinlock.h +5 -4
@@ -12,8 +12,9 @@* GNU General Public License for more details. * * (C) Copyright 2013-2015 Hewlett-Packard Development Company, L.P. +* (C) Copyright 2015 Hewlett-Packard Enterprise Development LP * -* Authors: Waiman Long <[email protected]> +* Authors: Waiman Long <[email protected]> */ #ifndef __ASM_GENERIC_QSPINLOCK_H#define __ASM_GENERIC_QSPINLOCK_H
@@ -62,7 +63,7 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) static __always_inline int queued_spin_trylock(struct qspinlock *lock) {if (!atomic_read (&lock->val) && - (atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL) ==0 )) + (atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ))return 1 ;return 0 ; }
@@ -77,7 +78,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) {u32 val; - val =atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL); + val =atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL);if (likely (val ==0 ))return ;queued_spin_lock_slowpath (lock, val);
@@ -93,7 +94,7 @@ static __always_inline void queued_spin_unlock(struct qspinlock *lock)/* * smp_mb__before_atomic() in order to guarantee release semantics */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic (); atomic_sub(_Q_LOCKED_VAL, &lock->val); }#endif
kernel/locking/qspinlock.c +19 -5
@@ -14,8 +14,9 @@* (C) Copyright 2013-2015 Hewlett-Packard Development Company, L.P. * (C) Copyright 2013-2014 Red Hat, Inc. * (C) Copyright 2015 Intel Corp. +* (C) Copyright 2015 Hewlett-Packard Enterprise Development LP * -* Authors: Waiman Long <[email protected]> +* Authors: Waiman Long <[email protected]> * Peter Zijlstra <[email protected]> */
@@ -176,7 +177,12 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) {struct __qspinlock *l = (void *)lock; -return (u32 )xchg (&l->tail, tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; +/* +* Use release semantics to make sure that the MCS node is properly +* initialized before changing the tail code. +*/ +return (u32 )xchg_release (&l->tail, + tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET; }#else /* _Q_PENDING_BITS == 8 */
@@ -208,7 +214,11 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)for (;;) { new = (val & _Q_LOCKED_PENDING_MASK) | tail; - old =atomic_cmpxchg (&lock->val, val, new); +/* +* Use release semantics to make sure that the MCS node is +* properly initialized before changing the tail code. +*/ + old =atomic_cmpxchg_release (&lock->val, val, new);if (old == val)break ;
@@ -319,7 +329,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if (val == new) new |= _Q_PENDING_VAL; - old =atomic_cmpxchg (&lock->val, val, new); + old =atomic_cmpxchg_acquire (&lock->val, val, new);if (old == val)break ;
@@ -426,7 +436,11 @@ queue:set_locked (lock);break ; } - old =atomic_cmpxchg (&lock->val, val, _Q_LOCKED_VAL); +/* +* The smp_load_acquire() call above has provided the necessary +* acquire semantics required for locking. +*/ + old =atomic_cmpxchg_relaxed (&lock->val, val, _Q_LOCKED_VAL);if (old == val) goto release;/* No contention */
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerRe: [PATCH v7 1/5] locking/qspinlock: relaxes cmpxchg & xchg ops in native code
Quoted 25 lines · Waiman Long
This patch replaces the
cmpxchg()andxchg()calls in the native qspinlock code with more relaxed versions of those calls to enable other architectures to adopt queued spinlocks with less performance overhead.include/asm-generic/qspinlock.h +2
-2@@ -62,7 +63,7 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) static __always_inline int queued_spin_trylock(struct qspinlock *lock) {if (!atomic_read (&lock->val) && - (atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL) ==0 )) + (atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ))return 1 ;return 0 ; }@@ -77,7 +78,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) {u32 val; - val =atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL); + val =atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL);if (likely (val ==0 ))return ;queued_spin_lock_slowpath (lock, val);
include/asm-generic/qspinlock.h void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) · line 335
Quoted 8 lines
include/asm-generic/qspinlock.h +1
-1@@ -319,7 +329,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if (val == new) new |= _Q_PENDING_VAL; - old =atomic_cmpxchg (&lock->val, val, new); + old =atomic_cmpxchg_acquire (&lock->val, val, new);if (old == val)break ;
So given recent discussion, all this _release/_acquire stuff is starting to worry me.
So we've not declared if they should be RCsc or RCpc, and given this patch (and the previous ones) these lock primitives turn into RCpc if the atomic primitives are RCpc.
So far only the proposed PPC implementation is RCpc -- and their current spinlock implementation is also RCpc, but that is a point of discussion.
Just saying..
Also, I think we should annotate the control dependencies in these things.
Re: [PATCH v7 1/5] locking/qspinlock: relaxes cmpxchg & xchg ops in native code
Quoted 35 lines · Peter Zijlstra
Quoted 23 lines · Waiman Long
This patch replaces the
cmpxchg()andxchg()calls in the native qspinlock code with more relaxed versions of those calls to enable other architectures to adopt queued spinlocks with less performance overhead.include/asm-generic/qspinlock.h +2
-2@@ -62,7 +63,7 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) static __always_inline int queued_spin_trylock(struct qspinlock *lock) {if (!atomic_read (&lock->val)&& - (atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL) ==0 )) + (atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ))return 1 ;return 0 ; }@@ -77,7 +78,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) {u32 val; - val =atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL); + val =atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL);if (likely (val ==0 ))return ;queued_spin_lock_slowpath (lock, val);include/asm-generic/qspinlock.h void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) · line 335
Quoted 8 lines
include/asm-generic/qspinlock.h +1
-1@@ -319,7 +329,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if (val == new) new |= _Q_PENDING_VAL; - old =atomic_cmpxchg (&lock->val, val, new); + old =atomic_cmpxchg_acquire (&lock->val, val, new);if (old == val)break ;So given recent discussion, all this _release/_acquire stuff is starting to worry me.
Quoted 8 lines
So we've not declared if they should be RCsc or RCpc, and given this patch (and the previous ones) these lock primitives turn into RCpc if the atomic primitives are RCpc.
So far only the proposed PPC implementation is RCpc -- and their current spinlock implementation is also RCpc, but that is a point of discussion.
Just saying..
Davidlohr's patches to make similar changes in other locking code will also have this issue. Anyway, the goal of this patch is to make the generic qspinlock code less costly when ported to other architectures. This change will have no effect on the x86 architecture which is the only one using qspinlock at the moment.
Quoted 3 lines
Also, I think we should annotate the control dependencies in these things.
Will do.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v7 1/5] locking/qspinlock: relaxes cmpxchg & xchg ops in native code
Quoted 40 lines · Waiman Long
Quoted 39 lines · Peter Zijlstra
Quoted 23 lines · Waiman Long
This patch replaces the
cmpxchg()andxchg()calls in the native qspinlock code with more relaxed versions of those calls to enable other architectures to adopt queued spinlocks with less performance overhead.include/asm-generic/qspinlock.h +2
-2@@ -62,7 +63,7 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) static __always_inline int queued_spin_trylock(struct qspinlock *lock) {if (!atomic_read (&lock->val)&& - (atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL) ==0 )) + (atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ))return 1 ;return 0 ; }@@ -77,7 +78,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) {u32 val; - val =atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL); + val =atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL);if (likely (val ==0 ))return ;queued_spin_lock_slowpath (lock, val);include/asm-generic/qspinlock.h void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) · line 335
Quoted 8 lines
include/asm-generic/qspinlock.h +1
-1@@ -319,7 +329,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if (val == new) new |= _Q_PENDING_VAL; - old =atomic_cmpxchg (&lock->val, val, new); + old =atomic_cmpxchg_acquire (&lock->val, val, new);if (old == val)break ;So given recent discussion, all this _release/_acquire stuff is starting to worry me.
So we've not declared if they should be RCsc or RCpc, and given this patch (and the previous ones) these lock primitives turn into RCpc if the atomic primitives are RCpc.
Quoted 7 lines
Quoted 4 lines
So far only the proposed PPC implementation is RCpc -- and their current spinlock implementation is also RCpc, but that is a point of discussion.
Just saying..
Davidlohr's patches to make similar changes in other locking code will also have this issue.
Yes, I only fully appreciated the RCpc pain last week :/
Quoted 2 lines
Anyway, the goal of this patch is to make the generic qspinlock code less costly when ported to other architectures.
As long as we stay away from PPC this will be fine ;-) Luckily they are unlikely to start using it since their LPAR hypervisor thingy isn't really co-operative.
Quoted 2 lines
This change
will have no effect on the x86 architecture which is the only one using
I know ARM and ARGH64 will want to start using this, luckily both are RCsc so no worries there.
Re: [PATCH v7 1/5] locking/qspinlock: relaxes cmpxchg & xchg ops in native code
In reply to Peter ZijlstraQuoted 38 lines · Peter Zijlstra
Quoted 25 lines · Waiman Long
This patch replaces the
cmpxchg()andxchg()calls in the native qspinlock code with more relaxed versions of those calls to enable other architectures to adopt queued spinlocks with less performance overhead.include/asm-generic/qspinlock.h +2
-2@@ -62,7 +63,7 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) static __always_inline int queued_spin_trylock(struct qspinlock *lock) {if (!atomic_read (&lock->val) && - (atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL) ==0 )) + (atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL) ==0 ))return 1 ;return 0 ; }@@ -77,7 +78,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) {u32 val; - val =atomic_cmpxchg (&lock->val,0 , _Q_LOCKED_VAL); + val =atomic_cmpxchg_acquire (&lock->val,0 , _Q_LOCKED_VAL);if (likely (val ==0 ))return ;queued_spin_lock_slowpath (lock, val);include/asm-generic/qspinlock.h void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) · line 335
Quoted 8 lines
include/asm-generic/qspinlock.h +1
-1@@ -319,7 +329,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)if (val == new) new |= _Q_PENDING_VAL; - old =atomic_cmpxchg (&lock->val, val, new); + old =atomic_cmpxchg_acquire (&lock->val, val, new);if (old == val)break ;So given recent discussion, all this _release/_acquire stuff is starting to worry me.
Quoted 3 lines
So we've not declared if they should be RCsc or RCpc, and given this patch (and the previous ones) these lock primitives turn into RCpc if the atomic primitives are RCpc.
Our spinlocks are currently RCpc and I think these really should match.
Quoted 4 lines
So far only the proposed PPC implementation is RCpc -- and their current spinlock implementation is also RCpc, but that is a point of discussion.
Just saying..
Well, PPC already made that choice for their spinlocks, so I don't necessarily think we should worry too much here. They get to choose between a ~5% performance hit (iirc) or a higher potential for subtle locking issues. That said, I don't believe the RCpc/RCsc choice actually affects common locking paradigms (so far, we're only aware of RCU needing to do something different on PPC).
If we end up getting swamped with horrible bug reports from PPC users, then we can add a TAINT_RCPC ;) (or just strengthen their implementation).
Quoted 2 lines
Also, I think we should annotate the control dependencies in these things.
Yes, please.
Will
PATCH v7 2/5 locking/pvqspinlock, x86: Optimize PV unlock code path
In reply to Waiman LongThe unlock function in queued spinlocks was optimized for better performance on bare metal systems at the expense of virtualized guests.
For x86-64 systems, the unlock call needs to go through a PV_CALLEE_SAVE_REGS_THUNK() which saves and restores 8 64-bit registers before calling the real __pv_queued_spin_unlock() function. The thunk code may also be in a separate cacheline from __pv_queued_spin_unlock().
This patch optimizes the PV unlock code path by:
1) Moving the unlock slowpath code from the fastpath into a separate __pv_queued_spin_unlock_slowpath() function to make the fastpath as simple as possible..
2) For x86-64, hand-coded an assembly function to combine the register saving thunk code with the fastpath code. Only registers that are used in the fastpath will be saved and restored. If the fastpath fails, the slowpath function will be called via another PV_CALLEE_SAVE_REGS_THUNK(). For 32-bit, it falls back to the C __pv_queued_spin_unlock() code as the thunk saves and restores only one 32-bit register.
With a microbenchmark of 5M lock-unlock loop, the table below shows the execution times before and after the patch with different number of threads in a VM running on a 32-core Westmere-EX box with x86-64 4.2-rc1 based kernels:
Threads Before patch After patch % Change
------- ------------ ----------- --------
1 134.1 ms 119.3 ms -11%
2 1286 ms 953 ms -26%
3 3715 ms 3480 ms -6.3%
4 4092 ms 3764 ms -8.0%--- arch/x86/include/asm/qspinlock_paravirt.h | 59 +++++++++++++++++++++++++++++ kernel/locking/qspinlock_paravirt.h | 43 +++++++++++++-------- 2 files changed, 86 insertions(+), 16 deletions(-)
arch/x86/include/asm/qspinlock_paravirt.h +59 -0
@@ -1,6 +1,65 @@#ifndef __ASM_QSPINLOCK_PARAVIRT_H#define __ASM_QSPINLOCK_PARAVIRT_H +/* +* For x86-64, PV_CALLEE_SAVE_REGS_THUNK() saves and restores 8 64-bit +* registers. For i386, however, only 1 32-bit register needs to be saved +* and restored. So an optimized version of __pv_queued_spin_unlock() is +* hand-coded for 64-bit, but it isn't worthwhile to do it for 32-bit. +*/ +#ifdef CONFIG_64BIT + +PV_CALLEE_SAVE_REGS_THUNK (__pv_queued_spin_unlock_slowpath); +#define __pv_queued_spin_unlock __pv_queued_spin_unlock +#define PV_UNLOCK "__raw_callee_save___pv_queued_spin_unlock" +#define PV_UNLOCK_SLOWPATH "__raw_callee_save___pv_queued_spin_unlock_slowpath" + +/* +* Optimized assembly version of __raw_callee_save___pv_queued_spin_unlock +* which combines the registers saving trunk and the body of the following +* C code: +* +* void __pv_queued_spin_unlock(struct qspinlock *lock) +* { +* struct __qspinlock *l = (void *)lock; +* u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0); +* +* if (likely(lockval == _Q_LOCKED_VAL)) +* return; +* pv_queued_spin_unlock_slowpath(lock, lockval); +* } +* +* For x86-64, +* rdi = lock (first argument) +* rsi = lockval (second argument) +* rdx = internal variable (set to 0) +*/ +asm (".pushsection .text;" +".globl " PV_UNLOCK ";" +".align 4,0x90;" +PV_UNLOCK ": " +"push %rdx;" +"mov $0x1,%eax;" +"xor %edx,%edx;" +"lock cmpxchg %dl,(%rdi);" +"cmp $0x1,%al;" +"jne .slowpath;" +"pop %rdx;" +"ret;" +".slowpath: " +"push %rsi;" +"movzbl %al,%esi;" +"call " PV_UNLOCK_SLOWPATH ";" +"pop %rsi;" +"pop %rdx;" +"ret;" +".size " PV_UNLOCK ", .-" PV_UNLOCK ";" +".popsection" ); + +#else /* CONFIG_64BIT */ + +extern void __pv_queued_spin_unlock (struct qspinlock *lock);PV_CALLEE_SAVE_REGS_THUNK (__pv_queued_spin_unlock); +#endif /* CONFIG_64BIT */ #endif
kernel/locking/qspinlock_paravirt.h +27 -16
62 lines not on this page · raw message
Signature · 1 line
1.7.1
PATCH v7 3/5 locking/pvqspinlock: Collect slowpath lock statistics
In reply to Waiman LongThis patch enables the accumulation of kicking and waiting related PV qspinlock statistics when the new QUEUED_LOCK_STAT configuration option is selected. It also enables the collection of data which enable us to calculate the kicking and wakeup latencies which have a heavy dependency on the CPUs being used.
Debugfs is used for managing the statistics counts for its simplicity. It has the limitation that per-cpu statistic counters cannot be used. Therefore, atomic instructions will be needed to update the counters. This introduces a certain amount of performance overhead when this feature is enabled.
The measured latencies for different CPUs are:
CPU Wakeup Kicking --- ------ ------- Haswell-EX 63.6us 7.4us Westmere-EX 67.6us 9.3us
The measured latencies varied a bit from run-to-run. The wakeup latency is much higher than the kicking latency.
A sample of statistics counts after system bootup (with vCPU overcommit) was:
hash_hops_count=9001
kick_latencies=138047878
kick_unlock_count=9001
kick_wait_count=9000
spurious_wakeup=3
wait_again_count=2
wait_head_count=10
wait_node_count=8994
wake_latencies=713195944
--- arch/x86/Kconfig | 9 ++ kernel/locking/qspinlock_paravirt.h | 168 +++++++++++++++++++++++++++++++++- 2 files changed, 172 insertions(+), 5 deletions(-)
arch/x86/Kconfig +9 -0
@@ -721,6 +721,15 @@ config PARAVIRT_SPINLOCKS If you are unsure how to answer this question, answer Y. +config QUEUED_LOCK_STAT + bool "Paravirt queued lock statistics" + depends on PARAVIRT && DEBUG_FS && QUEUED_SPINLOCKS + ---help--- + Enable the collection of statistical data on the behavior of + paravirtualized queued spinlocks and report them on debugfs. + There will be a slight performance overhead when this option + is enabled. + source "arch/x86/xen/Kconfig" config KVM_GUEST
kernel/locking/qspinlock_paravirt.h +163 -5
239 lines not on this page · raw message
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerRe: [PATCH v7 3/5] locking/pvqspinlock: Collect slowpath lock statistics
Quoted 36 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +6
-1@@ -100,10 +242,13 @@ static struct qspinlock **pv_hash(struct qspinlock *lock, struct pv_node *node) {unsigned long offset, hash =hash_ptr (lock, pv_lock_hash_bits);struct pv_hash_entry *he; +int hopcnt =0 ;for_each_hash_entry (he, offset, hash) { + hopcnt++;if (!cmpxchg (&he->lock,NULL , lock)) {WRITE_ONCE (he->node, node); +pvstat_hop (hopcnt);return &he->lock; } }@@ -164,9 +309,10 @@ static void pv_init_node(struct mcs_spinlock *node)static void pv_wait_node (struct mcs_spinlock *node) {struct pv_node *pn = (struct pv_node *)node; +int waitcnt =0 ;int loop; -for (;;) { +for (;; waitcnt++) {for (loop =SPIN_THRESHOLD ; loop; loop--) {if (READ_ONCE (node->locked))return ;@@ -250,6 +401,7 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)struct pv_node *pn = (struct pv_node *)node;struct __qspinlock *l = (void *)lock;struct qspinlock **lp =NULL ; +int waitcnt =0 ;int loop;/*
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 417
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h +1
-1@@ -259,7 +411,7 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)if (READ_ONCE (pn->state) == vcpu_hashed) lp = (struct qspinlock **)1 ; -for (;;) { +for (;; waitcnt++) {for (loop =SPIN_THRESHOLD ; loop; loop--) {if (!READ_ONCE (l->locked))return ;
These things are ugly; did you verify that they compile away for the !stats case?
Re: [PATCH v7 3/5] locking/pvqspinlock: Collect slowpath lock statistics
Quoted 40 lines · Peter Zijlstra
Quoted 38 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +6
-1@@ -100,10 +242,13 @@ static struct qspinlock **pv_hash(struct qspinlock *lock, struct pv_node *node) {unsigned long offset, hash =hash_ptr (lock, pv_lock_hash_bits);struct pv_hash_entry *he; +int hopcnt =0 ;for_each_hash_entry (he, offset, hash) { + hopcnt++;if (!cmpxchg (&he->lock,NULL , lock)) {WRITE_ONCE (he->node, node); +pvstat_hop (hopcnt);return &he->lock; } }@@ -164,9 +309,10 @@ static void pv_init_node(struct mcs_spinlock *node)static void pv_wait_node (struct mcs_spinlock *node) {struct pv_node *pn = (struct pv_node *)node; +int waitcnt =0 ;int loop; -for (;;) { +for (;; waitcnt++) {for (loop =SPIN_THRESHOLD ; loop; loop--) {if (READ_ONCE (node->locked))return ;@@ -250,6 +401,7 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)struct pv_node *pn = (struct pv_node *)node;struct __qspinlock *l = (void *)lock;struct qspinlock **lp =NULL ; +int waitcnt =0 ;int loop;/* @@ -259,7 +411,7 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)if (READ_ONCE (pn->state) == vcpu_hashed) lp = (struct qspinlock **)1 ;
Quoted 7 lines
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 417
Quoted 5 lines
kernel/locking/qspinlock_paravirt.h +1
-1-for (;;) { +for (;; waitcnt++) {for (loop =SPIN_THRESHOLD ; loop; loop--) {if (!READ_ONCE (l->locked))return ;These things are ugly; did you verify that they compile away for the !stats case?
The waitcnt was added to track if a vCPU was suspended again without getting the lock after being kicked. I will double check if it will be compiled away in the !stat case.
Cheers,
Longman
PATCH v7 4/5 locking/pvqspinlock: Allow 1 lock stealing attempt
In reply to Waiman LongThis patch allows one attempt for the lock waiter to steal the lock when entering the PV slowpath. This helps to reduce the performance penalty caused by lock waiter preemption while not having much of the downsides of a real unfair lock.
Linux kernel builds were run in KVM guest on an 8-socket, 4 cores/socket Westmere-EX system and a 4-socket, 8 cores/socket Haswell-EX system. Both systems are configured to have 32 physical CPUs. The kernel build times before and after the patch were:
Westmere Haswell
Patch 32 vCPUs 48 vCPUs 32 vCPUs 48 vCPUs ----- -------- -------- -------- -------- Before patch 3m15.6s 10m56.1s 1m44.1s 5m29.1s
After patch 3m02.3s 5m00.2s 1m43.7s 3m03.5s
For the overcommited case (48 vCPUs), this patch is able to reduce kernel build time by more than 54% for Westmere and 44% for Haswell.
--- kernel/locking/qspinlock.c | 19 ++--- kernel/locking/qspinlock_paravirt.h | 128 ++++++++++++++++++++++++++++------- 2 files changed, 112 insertions(+), 35 deletions(-)
kernel/locking/qspinlock.c +8 -11
@@ -249,17 +249,15 @@ static __always_inline void set_locked(struct qspinlock *lock) static __always_inline void __pv_init_node(struct mcs_spinlock *node) { } static __always_inline void __pv_wait_node(struct mcs_spinlock *node) { } -static __always_inline void __pv_kick_node(struct qspinlock *lock, -struct mcs_spinlock *node) { } -static __always_inline void __pv_wait_head(struct qspinlock *lock, -struct mcs_spinlock *node) { } - +static __always_inline bool __pv_wait_head_and_lock(struct qspinlock *lock, +struct mcs_spinlock *node, +u32 tail) + {return false; }#define pv_enabled () false#define pv_init_node __pv_init_node#define pv_wait_node __pv_wait_node -#define pv_kick_node __pv_kick_node -#define pv_wait_head __pv_wait_head +#define pv_wait_head_and_lock __pv_wait_head_and_lock#ifdef CONFIG_PARAVIRT_SPINLOCKS #define queued_spin_lock_slowpath native_queued_spin_lock_slowpath
@@ -417,7 +415,8 @@ queue:* does not imply a full barrier. * */ -pv_wait_head (lock, node); +if (pv_wait_head_and_lock (lock, node, tail)) + goto release;while ((val =smp_load_acquire (&lock->val.counter)) & _Q_LOCKED_PENDING_MASK)cpu_relax ();
@@ -454,7 +453,6 @@ queue:cpu_relax ();arch_mcs_spin_unlock_contended (&next->locked); -pv_kick_node (lock, next); release:/*
@@ -475,8 +473,7 @@ EXPORT_SYMBOL(queued_spin_lock_slowpath);#undef pv_init_node#undef pv_wait_node -#undef pv_kick_node -#undef pv_wait_head +#undef pv_wait_head_and_lock#undef queued_spin_lock_slowpath#define queued_spin_lock_slowpath __pv_queued_spin_lock_slowpath
kernel/locking/qspinlock_paravirt.h +104 -24
210 lines not on this page · raw message
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerRe: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 9 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +8
-0for (;; waitcnt++) { + loop =SPIN_THRESHOLD ; +while (loop) { +/* +* Spin until the lock is free +*/ +for (; loop &&READ_ONCE (l->locked); loop--) +cpu_relax (); +/*
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 475
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +9
-0+* Seeing the lock is free, this queue head vCPU is +* the rightful next owner of the lock. However, the +* lock may have just been stolen by another task which +* has entered the slowpath. So we need to use atomic +* operation to make sure that we really get the lock. +* Otherwise, we have to wait again. +*/ +if (cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ) + goto gotlock; }
for (loop = SPIN_THRESHOLD; loop; --loop) {
if (!READ_ONCE(l->locked) &&
cmpxchg(&l->locked, 0, _Q_LOCKED_VA) == 0)
goto gotlock;
cpu_relax();
}Re: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Waiman Long
for (;; waitcnt++) {kernel/locking/qspinlock_paravirt.h +17
-0+ loop =SPIN_THRESHOLD ; +while (loop) { +/* +* Spin until the lock is free +*/ +for (; loop&&READ_ONCE (l->locked); loop--) +cpu_relax (); +/* +* Seeing the lock is free, this queue head vCPU is +* the rightful next owner of the lock. However, the +* lock may have just been stolen by another task which +* has entered the slowpath. So we need to use atomic +* operation to make sure that we really get the lock. +* Otherwise, we have to wait again. +*/ +if (cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ) + goto gotlock;
Quoted 9 lines
Quoted 1 line
}for (loop = SPIN_THRESHOLD; loop; --loop) { if (!READ_ONCE(l->locked)&& cmpxchg(&l->locked, 0, _Q_LOCKED_VA) == 0) goto gotlock; cpu_relax(); }
This was the code that I used in my original patch, but it seems to confuse you about doing too many lock stealing. So I separated it out to make my intention more explicit. I will change it back to the old code.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 26 lines · Waiman Long
Quoted 25 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +8
-0for (;; waitcnt++) { + loop =SPIN_THRESHOLD ; +while (loop) { +/* +* Spin until the lock is free +*/ +for (; loop&&READ_ONCE (l->locked); loop--) +cpu_relax (); +/* kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 475
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +9
-0+* Seeing the lock is free, this queue head vCPU is +* the rightful next owner of the lock. However, the +* lock may have just been stolen by another task which +* has entered the slowpath. So we need to use atomic +* operation to make sure that we really get the lock. +* Otherwise, we have to wait again. +*/ +if (cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ) + goto gotlock; }for (loop = SPIN_THRESHOLD; loop; --loop) { if (!READ_ONCE(l->locked)&& cmpxchg(&l->locked, 0, _Q_LOCKED_VA) == 0) goto gotlock;
Quoted 7 lines
Quoted 3 lines
cpu_relax(); }This was the code that I used in my original patch, but it seems to confuse you about doing too many lock stealing. So I separated it out to make my intention more explicit. I will change it back to the old code.
Code should be compact; its the purpose of Changelogs and comments to explain it if its subtle.
Here you made weird code and the comments still don't explain how its starvation proof and the Changelog is almost empty of useful.
Re: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Waiman Long
Quoted 28 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
for (;; waitcnt++) {kernel/locking/qspinlock_paravirt.h +8
-0+ loop =SPIN_THRESHOLD ; +while (loop) { +/* +* Spin until the lock is free +*/ +for (; loop&&READ_ONCE (l->locked); loop--) +cpu_relax (); +/* kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 475
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +9
-0+* Seeing the lock is free, this queue head vCPU is +* the rightful next owner of the lock. However, the +* lock may have just been stolen by another task which +* has entered the slowpath. So we need to use atomic +* operation to make sure that we really get the lock. +* Otherwise, we have to wait again. +*/ +if (cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ) + goto gotlock; }for (loop = SPIN_THRESHOLD; loop; --loop) { if (!READ_ONCE(l->locked)&& cmpxchg(&l->locked, 0, _Q_LOCKED_VA) == 0) goto gotlock; cpu_relax(); }
Quoted 8 lines
Quoted 3 lines
This was the code that I used in my original patch, but it seems to confuse you about doing too many lock stealing. So I separated it out to make my intention more explicit. I will change it back to the old code.
Code should be compact; its the purpose of Changelogs and comments to explain it if its subtle.
Here you made weird code and the comments still don't explain how its starvation proof and the Changelog is almost empty of useful.
You are right. The current patch can't guarantee that there will be no lock starvation. I will make some code changes to make sure that lock starvation won't happen. I will also try to clean up the code and the comments at the same time.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
In reply to Waiman LongQuoted 5 lines · Waiman Long
This patch allows one attempt for the lock waiter to steal the lock when entering the PV slowpath. This helps to reduce the performance penalty caused by lock waiter preemption while not having much of the downsides of a real unfair lock.
Changelog does not explain the implementation, which is subtle enough to warrant a few words.
kernel/locking/qspinlock.c queue: · line 419
Quoted 7 lines
kernel/locking/qspinlock.c +2
-1@@ -417,7 +415,8 @@ queue:* does not imply a full barrier. * */ -pv_wait_head (lock, node); +if (pv_wait_head_and_lock (lock, node, tail)) + goto release;
That's very much: pv_wait_head_or_lock(), maybe _or_steal() is even better.
kernel/locking/qspinlock.c queue: · line 457
Quoted 8 lines
kernel/locking/qspinlock.c +0
-1while ((val =smp_load_acquire (&lock->val.counter)) & _Q_LOCKED_PENDING_MASK)cpu_relax ();@@ -454,7 +453,6 @@ queue:cpu_relax ();arch_mcs_spin_unlock_contended (&next->locked); -pv_kick_node (lock, next);
Not sure about removing that, breaks symmetry.
Quoted 14 lines
kernel/locking/qspinlock_paravirt.h +13
-0/* +* Allow one unfair trylock when entering the PV slowpath to reduce the +* performance impact of lock waiter preemption (either explicitly via +* pv_wait or implicitly via PLE). This function will be called once when +* a lock waiter enter the slowpath before being queued. +* +* A little bit of unfairness here can improve performance without many +* of the downsides of a real unfair lock. +*/ +#define queued_spin_trylock (l) pv_queued_spin_trylock_unfair(l) +static inline bool pv_queued_spin_trylock_unfair (struct qspinlock *lock) +{ +struct __qspinlock *l = (void *)lock; +
kernel/locking/qspinlock_paravirt.h struct pv_node { · line 66
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +10
-0+if (READ_ONCE (l->locked)) +return 0 ; +/* +* Wait a bit here to ensure that an actively spinning queue head vCPU +* has a fair chance of getting the lock. +*/ +cpu_relax (); + +return cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ; +}
This doesn't seem to make any sense.. Its also very much distinct from the rest of the patch and can easily be added in a separate patch with separate performance numbers to show it does (or does not) make a difference.
Re: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Waiman Long
This patch allows one attempt for the lock waiter to steal the lock when entering the PV slowpath. This helps to reduce the performance penalty caused by lock waiter preemption while not having much of the downsides of a real unfair lock.
Changelog does not explain the implementation, which is subtle enough to warrant a few words.
Will add more information into the changelog.
Quoted 10 lines
kernel/locking/qspinlock.c queue: · line 419
Quoted 8 lines
kernel/locking/qspinlock.c +2
-1@@ -417,7 +415,8 @@ queue:* does not imply a full barrier. * */ -pv_wait_head (lock, node); +if (pv_wait_head_and_lock (lock, node, tail)) + goto release;That's very much:
pv_wait_head_or_lock(), maybe_or_steal()is even better.
I am not very good at naming function. Changing it to _or_steal() is fine for me.
Quoted 30 lines
kernel/locking/qspinlock.c queue: · line 457
Quoted 8 lines
kernel/locking/qspinlock.c +0
-1while ((val =smp_load_acquire (&lock->val.counter))& _Q_LOCKED_PENDING_MASK)cpu_relax ();@@ -454,7 +453,6 @@ queue:cpu_relax ();arch_mcs_spin_unlock_contended (&next->locked); -pv_kick_node (lock, next);Not sure about removing that, breaks symmetry.
Quoted 20 lines
kernel/locking/qspinlock_paravirt.h +19
-0/* +* Allow one unfair trylock when entering the PV slowpath to reduce the +* performance impact of lock waiter preemption (either explicitly via +* pv_wait or implicitly via PLE). This function will be called once when +* a lock waiter enter the slowpath before being queued. +* +* A little bit of unfairness here can improve performance without many +* of the downsides of a real unfair lock. +*/ +#define queued_spin_trylock (l) pv_queued_spin_trylock_unfair(l) +static inline bool pv_queued_spin_trylock_unfair (struct qspinlock *lock) +{ +struct __qspinlock *l = (void *)lock; + +if (READ_ONCE (l->locked)) +return 0 ; +/* +* Wait a bit here to ensure that an actively spinning queue head vCPU +* has a fair chance of getting the lock. +*/
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h struct pv_node { · line 66
Quoted 4 lines
kernel/locking/qspinlock_paravirt.h +4
-0+cpu_relax (); + +return cmpxchg (&l->locked,0 , _Q_LOCKED_VAL) ==0 ; +}This doesn't seem to make any sense.. Its also very much distinct from the rest of the patch and can easily be added in a separate patch with separate performance numbers to show it does (or does not) make a difference.
If you mean I don't need an extra cpu_relax() here, I can take that out. It was there to make the active queue head vCPU having a higher chance of getting the lock, but it is not essential.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
In reply to Waiman LongQuoted 10 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +8
-0+gotlock:/* +* We now have the lock. We need to either clear the tail code or +* notify the next one in queue as the new queue head. */ + old =atomic_read (&lock->val); +while ((old & _Q_TAIL_MASK) == tail) { +int val; +int new = old & ~_Q_TAIL_MASK; +
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 534
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +10
-0+/* +* We are the only one in the queue, so clear the tail code +* and return. +*/ + val =atomic_cmpxchg (&lock->val, old, new); +if (old == val) + goto done; + old = val; + } +
This i need to think about a wee bit; its almost the same...
So the below is exactly duplicated from the normal slowpath, so why don't you keep that there?
It would get you something like:
if (pv_wait_head_or_steal(..)) goto stolen;
stolen:
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 542
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h +8
-0+/* +* contended path; wait for next, release. +*/ +while (!(next =READ_ONCE (node->next))) +cpu_relax (); + +arch_mcs_spin_unlock_contended (&next->locked); +pv_kick_node (lock, next);
release:
...
Re: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 44 lines · Peter Zijlstra
Quoted 10 lines · Waiman Long
kernel/locking/qspinlock_paravirt.h +8
-0+gotlock:/* +* We now have the lock. We need to either clear the tail code or +* notify the next one in queue as the new queue head. */ + old =atomic_read (&lock->val); +while ((old& _Q_TAIL_MASK) == tail) { +int val; +int new = old& ~_Q_TAIL_MASK; +kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 534
Quoted 10 lines
kernel/locking/qspinlock_paravirt.h +10
-0+/* +* We are the only one in the queue, so clear the tail code +* and return. +*/ + val =atomic_cmpxchg (&lock->val, old, new); +if (old == val) + goto done; + old = val; + } +This i need to think about a wee bit; its almost the same...
So the below is exactly duplicated from the normal slowpath, so why don't you keep that there?
It would get you something like:
if (pv_wait_head_or_steal(..)) goto stolen;stolen:
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 542
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h +8
-0+/* +* contended path; wait for next, release. +*/ +while (!(next =READ_ONCE (node->next))) +cpu_relax (); + +arch_mcs_spin_unlock_contended (&next->locked); +pv_kick_node (lock, next);release:
Quoted 1 line
...
Yes, it is largely the same. I thought that you don't like too much change in the logic flow of the generic qspinlock code. I will make the change in the next revision.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 23 lines · Waiman Long
Quoted 19 lines · Peter Zijlstra
So the below is exactly duplicated from the normal slowpath, so why don't you keep that there?
It would get you something like:
if (pv_wait_head_or_steal(..)) goto stolen;stolen:
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 542
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h +8
-0+/* +* contended path; wait for next, release. +*/ +while (!(next =READ_ONCE (node->next))) +cpu_relax (); + +arch_mcs_spin_unlock_contended (&next->locked); +pv_kick_node (lock, next);release:
Quoted 1 line
...
Quoted 3 lines
Yes, it is largely the same. I thought that you don't like too much change in the logic flow of the generic qspinlock code. I will make the change in the next revision.
Well, you already put the branch in there, the only difference here is an 'extra' label. OTOH that extra label avoids duplicating some hairy code. So over all I would say its a definite win.
And its easy to see it will compile away on the native case where:
#define pv_wait_head_or_steal(l, n, t) (false)
Re: [PATCH v7 4/5] locking/pvqspinlock: Allow 1 lock stealing attempt
Quoted 25 lines · Peter Zijlstra
Quoted 21 lines · Waiman Long
Quoted 19 lines · Peter Zijlstra
So the below is exactly duplicated from the normal slowpath, so why don't you keep that there?
It would get you something like:
if (pv_wait_head_or_steal(..)) goto stolen;stolen:
kernel/locking/qspinlock_paravirt.h static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) · line 542
Quoted 8 lines
kernel/locking/qspinlock_paravirt.h +8
-0+/* +* contended path; wait for next, release. +*/ +while (!(next =READ_ONCE (node->next))) +cpu_relax (); + +arch_mcs_spin_unlock_contended (&next->locked); +pv_kick_node (lock, next);release:
Quoted 1 line
...Quoted 3 lines
Yes, it is largely the same. I thought that you don't like too much change in the logic flow of the generic qspinlock code. I will make the change in the next revision.
Quoted 9 lines
Well, you already put the branch in there, the only difference here is an 'extra' label. OTOH that extra label avoids duplicating some hairy code. So over all I would say its a definite win.
And its easy to see it will compile away on the native case where:
#define pv_wait_head_or_steal(l, n, t) (false)
I have done it in a different way to avoid duplicating code. I think I am not going to use the pv_wait_head_or_steal name after all as I don't consider acquiring the lock by the queue head CPU is lock stealing. I will skip "and" and name it as pv_wait_head_lock instead. Please let me know if you have a better idea.
Cheers,
Longman
PATCH v7 5/5 locking/pvqspinlock: Queue node adaptive spinning
In reply to Waiman LongIn an overcommitted guest where some vCPUs have to be halted to make forward progress in other areas, it is highly likely that a vCPU later in the spinlock queue will be spinning while the ones earlier in the queue would have been halted. The spinning in the later vCPUs is then just a waste of precious CPU cycles because they are not going to get the lock soon as the earlier ones have to be woken up and take their turn to get the lock.
This patch implements an adaptive spinning mechanism where the vCPU will call pv_wait() if the previous vCPU is not running.
Linux kernel builds were run in KVM guest on an 8-socket, 4 cores/socket Westmere-EX system and a 4-socket, 8 cores/socket Haswell-EX system. Both systems are configured to have 32 physical CPUs. The kernel build times before and after the patch were:
Westmere Haswell
Patch 32 vCPUs 48 vCPUs 32 vCPUs 48 vCPUs
----- -------- -------- -------- --------
Before patch 3m02.3s 5m00.2s 1m43.7s 3m03.5s After patch 3m03.0s 4m37.5s 1m43.0s 2m47.2s
For 32 vCPUs, this patch doesn't cause any noticeable change in performance. For 48 vCPUs (over-committed), there is about 8% performance improvement.
--- kernel/locking/qspinlock.c | 5 ++- kernel/locking/qspinlock_paravirt.h | 47 +++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 4 deletions(-)
kernel/locking/qspinlock.c +3 -2
@@ -248,7 +248,8 @@ static __always_inline void set_locked(struct qspinlock *lock)*/ static __always_inline void __pv_init_node(struct mcs_spinlock *node) { } -static __always_inline void __pv_wait_node(struct mcs_spinlock *node) { } +static __always_inline void __pv_wait_node(struct mcs_spinlock *node, +struct mcs_spinlock *prev) { } static __always_inline bool __pv_wait_head_and_lock(struct qspinlock *lock,struct mcs_spinlock *node,u32 tail)
@@ -399,7 +400,7 @@ queue: prev =decode_tail (old);WRITE_ONCE (prev->next, node); -pv_wait_node (node); +pv_wait_node (node, prev);arch_mcs_spin_lock_contended (&node->locked); }
kernel/locking/qspinlock_paravirt.h +45 -2
103 lines not on this page · raw message
Signature · 1 line
1.7.1