locking/spinlock_debug: Change it to a mostly fair lock
Waiman Long
reviewercover locking/spinlock_debug: Change it to a mostly fair lock
The current debug spinlock implementation is a TATAS unfair lock. This can occasionally lead to system lockup with a debug kernel because of the unfairness of the lock rather than inherent locking problem.
This patch set changes the debug spinlock implementation to a mostly fair spinlock based on the MCS lock similar to what is done in qspinlock.
Waiman Long (2):
locking/spinlock_debug: Reduce lockup suspected message clutter locking/spinlock_debug: Reduce lock cacheline contention
include/linux/spinlock_types.h | 8 +++-- kernel/locking/spinlock_debug.c | 73 ++++++++++++++++++++++++++++++----------- 2 files changed, 58 insertions(+), 23 deletions(-)
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 1/2 locking/spinlock_debug: Reduce lockup suspected message clutter
When the debug spinlock code detects a lockup, it will print out an error messages as well as the backtraces of all the CPUs. However, if more than one CPUs are waiting on that lock, multiple lockup messages will be printed leading to garbled output.
To reduce clutter in the console log, now only one of the lock waiters will be allowed to print out the CPU backtraces.
--- include/linux/spinlock_types.h | 3 ++- kernel/locking/spinlock_debug.c | 26 +++++++++++++++++++++----- 2 files changed, 23 insertions(+), 6 deletions(-)
include/linux/spinlock_types.h +2 -1
@@ -23,7 +23,7 @@unsigned int break_lock;#endif #ifdef CONFIG_DEBUG_SPINLOCK -unsigned int magic, owner_cpu; +unsigned int magic, owner_cpu, lockup;void *owner;#endif #ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -45,6 +45,7 @@# define SPIN_DEBUG_INIT (lockname) \ .magic = SPINLOCK_MAGIC, \ .owner_cpu = -1, \ + .lockup = 0, \ .owner = SPINLOCK_OWNER_INIT,#else # define SPIN_DEBUG_INIT (lockname)
kernel/locking/spinlock_debug.c +21 -5
@@ -27,6 +27,7 @@ void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name, lock->magic =SPINLOCK_MAGIC ; lock->owner =SPINLOCK_OWNER_INIT ; lock->owner_cpu =-1 ; + lock->lockup =0 ; }EXPORT_SYMBOL (__raw_spin_lock_init);
@@ -101,6 +102,24 @@ static inline void debug_spin_unlock(raw_spinlock_t *lock) lock, "wrong CPU"); lock->owner =SPINLOCK_OWNER_INIT ; lock->owner_cpu =-1 ; + lock->lockup =0 ; +} + +static inline void __spin_lockup (raw_spinlock_t *lock) +{ +/* +* lockup suspected: +* +* Only one of the lock waiters will be allowed to print the lockup +* message in order to avoid an avalanche of lockup and backtrace +* messages from different lock waiters of the same lock. +*/ +if (!xchg (&lock->lockup,1 )) { +spin_dump (lock,"lockup suspected" ); +#ifdef CONFIG_SMP +trigger_all_cpu_backtrace (); +#endif + } }static void __spin_lock_debug (raw_spinlock_t *lock)
@@ -113,11 +132,8 @@ static void __spin_lock_debug(raw_spinlock_t *lock)return ;__delay (1 ); } -/* lockup suspected: */ -spin_dump (lock,"lockup suspected" ); -#ifdef CONFIG_SMP -trigger_all_cpu_backtrace (); -#endif + +__spin_lockup (lock);/* * The trylock above was causing a livelock. Give the lower level arch
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 2/2 locking/spinlock_debug: Reduce lock cacheline contention
In reply to Waiman LongThe debug spinlock code is a basic TATAS unfair lock irrespective of what the underlying architecture specific spinlock implementation is. As a result, it is sometimes possible to trigger a false positive "lockup suspected" warning with all the cpu backtraces.
This patch re-implements the debug spinlock as a fair MCS lock. This reduces the chance of false positive warning messages. At the same time, it also improves performance by reducing lock cacheline contention.
Because there is a trylock before entering the MCS queue, this new debug spinlock code also perform pretty well in a virtual machine even if its vCPUSs are over-committed.
On a 4-socket 32-core 64-thread system, the performance of a locking microbenchmark (locking rate and standard deviation) on a 4.9.6 based debug kernel with and without the patch was as follows:
32 locking threads:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 263.1 Mop/s 1.39s with patch 917.6 Mop/s 0.07s
64 locking threads:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 368.3 Mop/s 6.88s with patch 733.0 Mop/s 0.09s
On a 2-socket 24-core 48-thread system, the performance of the same locking microbenchmark (# of locking threads = # of vCPUs) on a KVM guest are as follows:
24 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 746.4 Mop/s 1.07s
with patch 1323.6 Mop/s 0.20s
48 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ ------------------- w/o patch 1077.8 Mop/s 3.34s with patch 1090.4 Mop/s 0.29s
72 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 944.5 Mop/s 3.96s
with patch 1176.7 Mop/s 0.44s
96 vCPUs:
Kernel Locking Rate SD (execution time) ------ ------------ -------------------
w/o patch 878.0 Mop/s 5.19s
with patch 1017.0 Mop/s 0.83s
--- include/linux/spinlock_types.h | 5 ++-- kernel/locking/spinlock_debug.c | 53 +++++++++++++++++++++++++++-------------- 2 files changed, 38 insertions(+), 20 deletions(-)
include/linux/spinlock_types.h +3 -2
@@ -24,7 +24,7 @@#endif #ifdef CONFIG_DEBUG_SPINLOCK unsigned int magic, owner_cpu, lockup; -void *owner; +void *owner, *tail;#endif #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map;
@@ -46,7 +46,8 @@ .magic = SPINLOCK_MAGIC, \ .owner_cpu = -1, \ .lockup = 0, \ - .owner = SPINLOCK_OWNER_INIT, + .owner = SPINLOCK_OWNER_INIT, \ + .tail = NULL,#else # define SPIN_DEBUG_INIT (lockname)#endif
kernel/locking/spinlock_debug.c +35 -18
91 lines not on this page · raw message
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH 2/2] locking/spinlock_debug: Reduce lock cacheline contention
url: https://github.com/0day-ci/linux/commits/Waiman-Long/locking-spinlock_debug-Change-it-to-a-mostly-fair-lock/20170202-024626 config: s390-default_defconfig (attached as .config) compiler…
Hi Waiman,
[auto build test WARNING on tip/locking/core]
[also build test WARNING on v4.10-rc6 next-20170201]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Waiman-Long/locking-spinlock_debug-Change-it-to-a-mostly-fair-lock/20170202-024626
config: s390-default_defconfig (attached as .config)
compiler: s390x-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=s390
All warnings (new ones prefixed by >>):
block/blk-cgroup.c: In function 'blkg_destroy':
>> block/blk-cgroup.c:353:1: warning: the frame size of 1032 bytes is larger than 1024 bytes [-Wframe-larger-than=]
}
^
--
fs/lockd/svcsubs.c: In function 'nlmsvc_mark_resources':
>> fs/lockd/svcsubs.c:376:1: warning: the frame size of 1032 bytes is larger than 1024 bytes [-Wframe-larger-than=]
}
^vim +353 block/blk-cgroup.c
e8989fae Tejun Heo 2012-03-05 337 list_del_init(&blkg->q_node); 9f13ef67 Tejun Heo 2012-03-05 338 hlist_del_init_rcu(&blkg->blkcg_node); 03aa264a Tejun Heo 2012-03-05 339 03aa264a Tejun Heo 2012-03-05 340 /* a637120e Tejun Heo 2012-04-19 341 * Both setting lookup hint to and clearing it from @blkg are done a637120e Tejun Heo 2012-04-19 342 * under queue_lock. If it's not pointing to @blkg now, it never a637120e Tejun Heo 2012-04-19 343 * will. Hint assignment itself can race safely. a637120e Tejun Heo 2012-04-19 344 */ ec6c676a Paul E. McKenney 2014-02-17 345 if (rcu_access_pointer(blkcg->blkg_hint) == blkg) a637120e Tejun Heo 2012-04-19 346 rcu_assign_pointer(blkcg->blkg_hint, NULL); a637120e Tejun Heo 2012-04-19 347 a637120e Tejun Heo 2012-04-19 348 /* 03aa264a Tejun Heo 2012-03-05 349 * Put the reference taken at the time of creation so that when all 03aa264a Tejun Heo 2012-03-05 350 * queues are gone, group can be destroyed. 03aa264a Tejun Heo 2012-03-05 351 */ 03aa264a Tejun Heo 2012-03-05 352 blkg_put(blkg); 03aa264a Tejun Heo 2012-03-05 @353 } 03aa264a Tejun Heo 2012-03-05 354 9f13ef67 Tejun Heo 2012-03-05 355 /** 9f13ef67 Tejun Heo 2012-03-05 356 * blkg_destroy_all - destroy all blkgs associated with a request_queue 9f13ef67 Tejun Heo 2012-03-05 357 * @q: request_queue of interest 9f13ef67 Tejun Heo 2012-03-05 358 * 3c96cb32 Tejun Heo 2012-04-13 359 * Destroy all blkgs associated with @q. 9f13ef67 Tejun Heo 2012-03-05 360 */ 3c96cb32 Tejun Heo 2012-04-13 361 static void blkg_destroy_all(struct request_queue *q)
:::::: The code at line 353 was first introduced by commit
:::::: 03aa264ac15637b6f98374270bcdf31400965505 blkcg: let blkcg core manage per-queue blkg list and counter
:::::: TO: Tejun Heo <[email protected]>
:::::: CC: Jens Axboe <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation