locking/spinlock_debug: Change it to a mostly fair lock
Subsystems (6)
Waiman Long
reviewerv3 cover locking/spinlock_debug: Change it to a mostly fair lock
v2->v3:
- Keep the original v1 patches but move patch 3 of v2 in front so as to disable GENERIC_LOCKBREAK when DEBUG_LOCK_ALLOC is on.
v1->v2:
- Pack lockup and break_lock into a single 4-byte slot so as not to in increase spinlock size when GENERIC_LOCKBREAK is on. Hopefully that will be enough to fix a frame size too large warning in 0-day build.
- Add a new patch to disable GENERIC_LOCKBREAK when DEBUG_LOCK_ALLOC is on.
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock. It also includes a patch that disable GENERIC_LOCKBREAK when DEBUG_LOCK_ALLOC is on.
Waiman Long (3):
locking/spinlock: Disable GENERIC_LOCKBREAK when DEBUG_LOCK_ALLOC is
on
locking/spinlock_debug: Reduce lockup suspected message clutter locking/spinlock_debug: Reduce lock cacheline contention
arch/m32r/Kconfig | 2 +- arch/parisc/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/sparc/Kconfig | 2 +- include/linux/spinlock_types.h | 8 +++-- kernel/locking/spinlock_debug.c | 73 ++++++++++++++++++++++++++++++----------- 8 files changed, 64 insertions(+), 29 deletions(-)
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH v3 1/3 locking/spinlock: Disable GENERIC_LOCKBREAK when DEBUG_LOCK_ALLOC is on
The break_lock variable defined when GENERIC_LOCKBREAK is on is used only in kernel/locking/spinlock.c when
defined(CONFIG_GENERIC_LOCKBREAK) && !defined(CONFIG_DEBUG_LOCK_ALLOC).
As a result, there is no point in enabling GENERIC_LOCKBREAK to define one more variable in the spinlock structure that is not going to be used when DEBUG_LOCK_ALLOC is also on. This patch disables GENERIC_LOCKBREAK under this circumstance.
--- arch/m32r/Kconfig | 2 +- arch/parisc/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/sparc/Kconfig | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-)
arch/m32r/Kconfig +1 -1
@@ -242,7 +242,7 @@ config IRAM_SIZE config GENERIC_LOCKBREAK bool default y - depends on SMP && PREEMPT + depends on SMP && PREEMPT && !DEBUG_LOCK_ALLOC config RWSEM_GENERIC_SPINLOCK bool
arch/parisc/Kconfig +1 -1
@@ -56,7 +56,7 @@ config STACK_GROWSUP config GENERIC_LOCKBREAK bool default y - depends on SMP && PREEMPT + depends on SMP && PREEMPT && !DEBUG_LOCK_ALLOC config RWSEM_GENERIC_SPINLOCK def_bool y
arch/powerpc/Kconfig +1 -1
@@ -60,7 +60,7 @@ config RWSEM_XCHGADD_ALGORITHM config GENERIC_LOCKBREAK bool default y - depends on SMP && PREEMPT + depends on SMP && PREEMPT && !DEBUG_LOCK_ALLOC config ARCH_HAS_ILOG2_U32 bool
arch/s390/Kconfig +1 -1
@@ -38,7 +38,7 @@ config ARCH_DMA_ADDR_T_64BIT def_bool y config GENERIC_LOCKBREAK - def_bool y if SMP && PREEMPT + def_bool y if SMP && PREEMPT && !DEBUG_LOCK_ALLOC config PGSTE def_bool y if KVM
arch/sh/Kconfig +1 -1
@@ -105,7 +105,7 @@ config GENERIC_CALIBRATE_DELAY config GENERIC_LOCKBREAK def_bool y - depends on SMP && PREEMPT + depends on SMP && PREEMPT && !DEBUG_LOCK_ALLOC config ARCH_SUSPEND_POSSIBLE def_bool n
arch/sparc/Kconfig +1 -1
@@ -286,7 +286,7 @@ config US3_MC config GENERIC_LOCKBREAK bool default y - depends on SPARC64 && SMP && PREEMPT + depends on SPARC64 && SMP && PREEMPT && !DEBUG_LOCK_ALLOC config NUMA bool "NUMA support"
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH v3 2/3 locking/spinlock_debug: Reduce lockup suspected message clutter
In reply to Waiman LongWhen the debug spinlock code detects a lockup, it will print out an error messages as well as the backtraces of all the CPUs. However, if more than one CPUs are waiting on that lock, multiple lockup messages will be printed leading to garbled output.
To reduce clutter in the console log, now only one of the lock waiters will be allowed to print out the CPU backtraces.
Since break_lock, like lockup, can only have a value of 0 or 1, its size is now reduced so that on a 64-bit architecture, the size of the raw_spinlock structure won't increase whether CONFIG_GENERIC_LOCKBREAK is defined or not.
--- include/linux/spinlock_types.h | 3 ++- kernel/locking/spinlock_debug.c | 26 +++++++++++++++++++++----- 2 files changed, 23 insertions(+), 6 deletions(-)
include/linux/spinlock_types.h +2 -1
@@ -23,7 +23,7 @@unsigned int break_lock;#endif #ifdef CONFIG_DEBUG_SPINLOCK -unsigned int magic, owner_cpu; +unsigned int magic, owner_cpu, lockup;void *owner;#endif #ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -45,6 +45,7 @@# define SPIN_DEBUG_INIT (lockname) \ .magic = SPINLOCK_MAGIC, \ .owner_cpu = -1, \ + .lockup = 0, \ .owner = SPINLOCK_OWNER_INIT,#else # define SPIN_DEBUG_INIT (lockname)
kernel/locking/spinlock_debug.c +21 -5
@@ -27,6 +27,7 @@ void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name, lock->magic =SPINLOCK_MAGIC ; lock->owner =SPINLOCK_OWNER_INIT ; lock->owner_cpu =-1 ; + lock->lockup =0 ; }EXPORT_SYMBOL (__raw_spin_lock_init);
@@ -101,6 +102,24 @@ static inline void debug_spin_unlock(raw_spinlock_t *lock) lock, "wrong CPU"); lock->owner =SPINLOCK_OWNER_INIT ; lock->owner_cpu =-1 ; + lock->lockup =0 ; +} + +static inline void __spin_lockup (raw_spinlock_t *lock) +{ +/* +* lockup suspected: +* +* Only one of the lock waiters will be allowed to print the lockup +* message in order to avoid an avalanche of lockup and backtrace +* messages from different lock waiters of the same lock. +*/ +if (!xchg (&lock->lockup,1 )) { +spin_dump (lock,"lockup suspected" ); +#ifdef CONFIG_SMP +trigger_all_cpu_backtrace (); +#endif + } }static void __spin_lock_debug (raw_spinlock_t *lock)
@@ -113,11 +132,8 @@ static void __spin_lock_debug(raw_spinlock_t *lock)return ;__delay (1 ); } -/* lockup suspected: */ -spin_dump (lock,"lockup suspected" ); -#ifdef CONFIG_SMP -trigger_all_cpu_backtrace (); -#endif + +__spin_lockup (lock);/* * The trylock above was causing a livelock. Give the lower level arch
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH v3 3/3 locking/spinlock_debug: Reduce lock cacheline contention
In reply to Waiman LongThe debug spinlock code is a basic TATAS unfair lock irrespective of what the underlying architecture specific spinlock implementation is. As a result, it is sometimes possible to trigger a false positive "lockup suspected" warning with all the cpu backtraces.
This patch re-implements the debug spinlock as a fair MCS lock. This reduces the chance of false positive warning messages. At the same time, it also improves performance by reducing lock cacheline contention.
Because there is a trylock before entering the MCS queue, this new debug spinlock code also perform pretty well in a virtual machine even if its vCPUSs are over-committed.
On a 4-socket 32-core 64-thread system, the performance of a locking microbenchmark (locking rate and standard deviation) on a 4.9.6 based debug kernel with and without the patch was as follows:
32 locking threads:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 263.1 Mop/s 1.39s with patch 917.6 Mop/s 0.07s
64 locking threads:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 368.3 Mop/s 6.88s with patch 733.0 Mop/s 0.09s
On a 2-socket 24-core 48-thread system, the performance of the same locking microbenchmark (# of locking threads = # of vCPUs) on a KVM guest are as follows:
24 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 746.4 Mop/s 1.07s
with patch 1323.6 Mop/s 0.20s
48 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 1077.8 Mop/s 3.34s with patch 1090.4 Mop/s 0.29s
72 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 944.5 Mop/s 3.96s
with patch 1176.7 Mop/s 0.44s
96 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 878.0 Mop/s 5.19s
with patch 1017.0 Mop/s 0.83s
--- include/linux/spinlock_types.h | 5 ++-- kernel/locking/spinlock_debug.c | 53 +++++++++++++++++++++++++++-------------- 2 files changed, 38 insertions(+), 20 deletions(-)
include/linux/spinlock_types.h +3 -2
@@ -24,7 +24,7 @@#endif #ifdef CONFIG_DEBUG_SPINLOCK unsigned int magic, owner_cpu, lockup; -void *owner; +void *owner, *tail;#endif #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map;
@@ -46,7 +46,8 @@ .magic = SPINLOCK_MAGIC, \ .owner_cpu = -1, \ .lockup = 0, \ - .owner = SPINLOCK_OWNER_INIT, + .owner = SPINLOCK_OWNER_INIT, \ + .tail = NULL,#else # define SPIN_DEBUG_INIT (lockname)#endif
kernel/locking/spinlock_debug.c +35 -18
91 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerQuoted 12 lines · Waiman Long
v2->v3:
- Keep the original v1 patches but move patch 3 of v2 in front so as to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
v1->v2:
- Pack lockup andbreak_lockinto a single 4-byte slot so as not to in increase spinlock size whenGENERIC_LOCKBREAKis on. Hopefully that will be enough to fix a frame size too large warning in 0-day build.
- Add a new patch to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
Quoted 8 lines
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock. It also includes a patch that disable
GENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
An alternative is to just delete the thing entirely..
Ingo, much of what this thing does seems to be superseded by both lockdep and a reliable NMI watchdog. Is there still value in spinlock_debug?
Quoted 23 lines · Peter Zijlstra
Quoted 12 lines · Waiman Long
v2->v3:
- Keep the original v1 patches but move patch 3 of v2 in front so as to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
v1->v2:
- Pack lockup andbreak_lockinto a single 4-byte slot so as not to in increase spinlock size whenGENERIC_LOCKBREAKis on. Hopefully that will be enough to fix a frame size too large warning in 0-day build.
- Add a new patch to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.Quoted 8 lines
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock. It also includes a patch that disable
GENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
Quoted 5 lines
An alternative is to just delete the thing entirely..
Ingo, much of what this thing does seems to be superseded by both lockdep and a reliable NMI watchdog. Is there still value in
spinlock_debug?
So there's still early stages when the NMI watchdog is not running, and spinlock-debug can detect lockups in raw locks that lockdep does not cover, right?
But yeah ... it would simplify things all around, so I'm not unsympathetic to the idea...
I've Cc:-ed a few other locking gents.
Thanks,
Ingo
Waiman Long
reviewerQuoted 32 lines · Ingo Molnar
Quoted 22 lines · Peter Zijlstra
Quoted 12 lines · Waiman Long
v2->v3:
- Keep the original v1 patches but move patch 3 of v2 in front so as to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
v1->v2:
- Pack lockup andbreak_lockinto a single 4-byte slot so as not to in increase spinlock size whenGENERIC_LOCKBREAKis on. Hopefully that will be enough to fix a frame size too large warning in 0-day build.
- Add a new patch to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.Quoted 8 lines
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock. It also includes a patch that disable
GENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.Quoted 5 lines
An alternative is to just delete the thing entirely..
Ingo, much of what this thing does seems to be superseded by both lockdep and a reliable NMI watchdog. Is there still value in
spinlock_debug?So there's still early stages when the NMI watchdog is not running, and spinlock-debug can detect lockups in raw locks that lockdep does not cover, right?
Quoted 8 lines
But yeah ... it would simplify things all around, so I'm not unsympathetic to the idea...
I've Cc:-ed a few other locking gents.
Thanks,
Ingo
I have no problem deleting the debug_spinlock code entirely. I can update the patch to delete the code if you guys think that is the right thing to do.
Cheers,
Longman
Quoted 40 lines · Waiman Long
Quoted 39 lines · Ingo Molnar
Quoted 22 lines · Peter Zijlstra
Quoted 12 lines · Waiman Long
v2->v3:
- Keep the original v1 patches but move patch 3 of v2 in front so as to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.
v1->v2:
- Pack lockup andbreak_lockinto a single 4-byte slot so as not to in increase spinlock size whenGENERIC_LOCKBREAKis on. Hopefully that will be enough to fix a frame size too large warning in 0-day build.
- Add a new patch to disableGENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.Quoted 8 lines
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock. It also includes a patch that disable
GENERIC_LOCKBREAKwhenDEBUG_LOCK_ALLOCis on.Quoted 5 lines
An alternative is to just delete the thing entirely..
Ingo, much of what this thing does seems to be superseded by both lockdep and a reliable NMI watchdog. Is there still value in
spinlock_debug?So there's still early stages when the NMI watchdog is not running, and spinlock-debug can detect lockups in raw locks that lockdep does not cover, right?
But yeah ... it would simplify things all around, so I'm not unsympathetic to the idea...
I've Cc:-ed a few other locking gents.
Thanks,
Quoted 8 lines
Quoted 1 line
Ingo
I have no problem deleting the
debug_spinlockcode entirely. I can update the patch to delete the code if you guys think that is the right thing to do.Cheers,
Longman
My usual question is "how often does the spinlock_debug code find a problem that would be hard to find otherwise?" Probably unanswerable given the nature of Linux-kernel development, but I figured I would ask anyway. ;-)
Thanx, Paul
Peter Zijlstra
maintainerQuoted 4 lines · Paul E. McKenney
My usual question is "how often does the
spinlock_debugcode find a problem that would be hard to find otherwise?" Probably unanswerable given the nature of Linux-kernel development, but I figured I would ask anyway. ;-)
So I've not found it useful in many years, and quite to the contrary, its proven prone to generate false positives because the lock timeout gets hit because of various reasons.
But that's just me of course..
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Paul E. McKenney
My usual question is "how often does the
spinlock_debugcode find a problem that would be hard to find otherwise?" Probably unanswerable given the nature of Linux-kernel development, but I figured I would ask anyway. ;-)So I've not found it useful in many years, and quite to the contrary, its proven prone to generate false positives because the lock timeout gets hit because of various reasons.
But that's just me of course..
I have seen neither useful information nor false positives from it, nor have I heard of anyone else having recently done so. So I have nothing indicating that I should either defend it on the one hand or advocate for its removal on the other. But I was asked, so...
One approach would be to push the patch and see if anyone complained. But there are quite a few people who wouldn't complain about anything until it bit them, which wouldn't happen for some time. Which -might- be OK, but...
Another would be to announce the patch widely, via social media, LWN articles, presentations, etc., in order to increase the probability of finding out if anyone relies on it sooner rather than later.
My recommendation is to do both in parallel. Get the patch out there for testing, get it headed for mainline, and make noise about it along the way. If no one complains, it goes in. If someone does complain, we can see what they really need, which would hopefully point the way to an improved mechanism.
But that's just me of course... ;-)
Thanx, Paul