locking/rwsem: Rework rwsem-xadd & enable new rwsem features
Subsystems (12)
Waiman Long
reviewercover locking/rwsem: Rework rwsem-xadd & enable new rwsem features
This patchset revamps the current rwsem-xadd implementation to make it saner and easier to work with. This patchset removes all the architecture specific assembly code and uses generic C code for all architectures. This eases maintenance and enables us to enhance the code more easily.
This patchset also implements the following 3 new features:
1) Waiter lock handoff
2) Reader optimistic spinning
3) Store write-lock owner in the atomic count (x86-64 only)
Waiter lock handoff is similar to the mechanism currently in the mutex code. This ensures that lock starvation won't happen.
Reader optimistic spinning enables readers to acquire the lock more quickly. So workloads that use a mix of readers and writers should see an increase in performance.
Finally, storing the write-lock owner into the count will allow optimistic spinners to get to the lock holder's task structure more quickly and eliminating the timing gap where the write lock is acquired but the owner isn't known yet. This is important for RT tasks where spinning on a lock with an unknown owner is not allowed.
Because of the fact that multiple readers can share the same lock, there is a natural preference for readers when measuring in term of locking throughput as more readers are likely to get into the locking fast path than the writers. With waiter lock handoff, we are not going to starve the writers.
Patches 1-2 reworks the qspinlock_stat code to make it generic (lock event counting) so that it can be used by all architectures and all locking code.
Patch 3 reloctes the rwsem_down_read_failed() and associated functions to below the optimistic spinning functions.
Patch 4 eliminates all architecture specific code and use generic C code for all.
Patch 5 moves code that manages the owner field closer to the rwsem lock fast path as it is not needed by the rwsem-spinlock code.
Patch 6 renames rwsem.h to rwsem-xadd.h as it is now specific to rwsem-xadd.c only.
Patch 7 hides the internal rwsem-xadd functions from the public.
Patch 8 moves the DEBUG_RWSEMS_WARN_ON checks from rwsem.c to kernel/locking/rwsem-xadd.h and adds some new ones.
Patch 9 enhances the DEBUG_RWSEMS_WARN_ON macro to print out rwsem internal states that can be useful for debugging purpose.
Patch 10 enables lock event countings in the rwsem code.
Patch 11 implements a new rwsem locking scheme similar to what qrwlock is current doing. Write lock is done by atomic_cmpxchg() while read lock is still being done by atomic_add().
Patch 12 implments lock handoff to prevent lock starvation.
Patch 13 removes rwsem_wake() wakeup optimization as it doesn't work with lock handoff.
Patch 14 adds some new rwsem owner access helper functions.
Patch 15 merges the write-lock owner task pointer into the count. Only 64-bit count has enough space to provide a reasonable number of bits for reader count. ARM64 seems to have problem with the current encoding scheme. So this owner merging is currently limited to x86-64 only.
Patch 16 eliminates redundant computation of the merged owner-count.
Patch 17 reduces the chance of missed optimistic spinning opportunity because of some race conditions.
Patch 18 makes rwsem_spin_on_owner() returns a tri-state value.
Patch 19 enables reader to spin on a writer-owned rwsem.
Patch 20 enables lock waiters to spin on a reader-owned rwsem with limited number of tries.
Patch 21 makes reader wakeup to wake all the readers in the wait queue instead of just those in the front.
Patch 22 disallows RT tasks to spin on a rwsem with unknown owner.
In term of performance, eliminating architecture specific assembly code and using generic code doesn't seem to have any impact on performance.
Supporting lock handoff does have a minor performance impact on highly contended rwsem, but it is a price worth paying for preventing lock starvation.
Reader optimistic spinning is generally good for performance. Of course, there will be some corner cases where performance may suffer.
Merging owner into count does have a minor performance impact. We can discuss if this is a feature we want to have in the rwsem code.
There are also some performance data scattered in some of the patches.
Waiman Long (22):
locking/qspinlock_stat: Introduce a generic lockevent counting APIs locking/lock_events: Make lock_events available for all archs & other
locks
locking/rwsem: Relocate rwsem_down_read_failed() locking/rwsem: Remove arch specific rwsem files locking/rwsem: Move owner setting code from rwsem.c to rwsem.h locking/rwsem: Rename kernel/locking/rwsem.h locking/rwsem: Move rwsem internal function declarations to
rwsem-xadd.h
locking/rwsem: Add debug check for __down_read*() locking/rwsem: Enhance DEBUG_RWSEMS_WARN_ON() macro locking/rwsem: Enable lock event counting
locking/rwsem: Implement a new locking scheme locking/rwsem: Implement lock handoff to prevent lock starvation locking/rwsem: Remove rwsem_wake() wakeup optimization locking/rwsem: Add more rwsem owner access helpers locking/rwsem: Merge owner into count on x86-64 locking/rwsem: Remove redundant computation of writer lock word locking/rwsem: Recheck owner if it is not on cpu locking/rwsem: Make rwsem_spin_on_owner() return a tri-state value locking/rwsem: Enable readers spinning on writer locking/rwsem: Enable count-based spinning on reader locking/rwsem: Wake up all readers in wait queue locking/rwsem: Ensure an RT task will not spin on reader
MAINTAINERS | 1 - arch/Kconfig | 10 + arch/alpha/include/asm/rwsem.h | 211 ----------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 --------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/Kconfig | 8 - arch/x86/include/asm/rwsem.h | 237 ------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 --------- arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 -------- include/linux/rwsem.h | 11 +- kernel/locking/Makefile | 1 + kernel/locking/lock_events.c | 153 ++++++++ kernel/locking/lock_events.h | 55 +++ kernel/locking/lock_events_list.h | 71 ++++ kernel/locking/percpu-rwsem.c | 4 + kernel/locking/qspinlock.c | 8 +- kernel/locking/qspinlock_paravirt.h | 19 +- kernel/locking/qspinlock_stat.h | 242 +++---------- kernel/locking/rwsem-xadd.c | 682 +++++++++++++++++++++--------------- kernel/locking/rwsem-xadd.h | 436 +++++++++++++++++++++++ kernel/locking/rwsem.c | 31 +- kernel/locking/rwsem.h | 134 ------- 30 files changed, 1197 insertions(+), 1594 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h create mode 100644 kernel/locking/lock_events.c create mode 100644 kernel/locking/lock_events.h create mode 100644 kernel/locking/lock_events_list.h create mode 100644 kernel/locking/rwsem-xadd.h delete mode 100644 kernel/locking/rwsem.h
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 1/22 locking/qspinlock_stat: Introduce a generic lockevent counting APIs
The percpu event counts used by qspinlock code can be useful for other locking code as well. So a new set of lockevent_* counting APIs is introduced with the lock event names extracted out into the new lock_events_list.h header file for easier addition in the future.
The existing qstat_inc() calls are replaced by either lockevent_inc() or lockevent_cond_inc() calls.
The qstat_hop() call is renamed to lockevent_pv_hop(). The "reset_counters" debugfs file is also renamed to ".reset_counts".
--- kernel/locking/lock_events.h | 55 ++++++++++++ kernel/locking/lock_events_list.h | 50 +++++++++++ kernel/locking/qspinlock.c | 8 +- kernel/locking/qspinlock_paravirt.h | 19 +++-- kernel/locking/qspinlock_stat.h | 163 ++++++++++++++---------------------- 5 files changed, 181 insertions(+), 114 deletions(-) create mode 100644 kernel/locking/lock_events.h create mode 100644 kernel/locking/lock_events_list.h
kernel/locking/lock_events.h new file +55 -0
@@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* Authors: Waiman Long <[email protected]> +*/ + +enum lock_events { + +#include "lock_events_list.h" + + lockevent_num,/* Total number of lock event counts */ + LOCKEVENT_reset_cnts = lockevent_num, +}; + +#ifdef CONFIG_QUEUED_LOCK_STAT +/* +* Per-cpu counters +*/ +DECLARE_PER_CPU(unsigned long, lockevents[lockevent_num]); + +/* +* Increment the PV qspinlock statistical counters +*/ +static inline void __lockevent_inc (enum lock_events event,bool cond) +{ +if (cond) +__this_cpu_inc (lockevents[event]); +} + +#define lockevent_inc (ev) __lockevent_inc(LOCKEVENT_ ##ev, true) +#define lockevent_cond_inc (ev, c) __lockevent_inc(LOCKEVENT_ ##ev, c) + +static inline void __lockevent_add (enum lock_events event,int inc) +{ +__this_cpu_add (lockevents[event], inc); +} + +#define lockevent_add (ev, c) __lockevent_add(LOCKEVENT_ ##ev, c) + +#else /* CONFIG_QUEUED_LOCK_STAT */ + +#define lockevent_inc (ev) +#define lockevent_add (ev, c) +#define lockevent_cond_inc (ev, c) + +#endif /* CONFIG_QUEUED_LOCK_STAT */
kernel/locking/lock_events_list.h new file +50 -0
50 lines not on this page · raw message
kernel/locking/qspinlock.c +4 -4
@@ -398,7 +398,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)* 0,1,0 -> 0,0,1 */ clear_pending_set_locked (lock); -qstat_inc (qstat_lock_pending, true); +lockevent_inc (lock_pending);return ;/*
@@ -406,7 +406,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)* queuing. */ queue: -qstat_inc (qstat_lock_slowpath, true); +lockevent_inc (lock_slowpath); pv_queue: node =this_cpu_ptr (&qnodes[0 ].mcs); idx = node->count++;
@@ -422,7 +422,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)* simple enough. */ if (unlikely (idx >=MAX_NODES )) { -qstat_inc (qstat_lock_no_node, true); +lockevent_inc (lock_no_node);while (!queued_spin_trylock (lock))cpu_relax (); goto release;
@@ -433,7 +433,7 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)/* * Keep counts of non-zero index values: */ -qstat_inc (qstat_lock_use_node2 + idx -1 , idx); +lockevent_cond_inc (lock_use_node2 + idx -1 , idx);/* * Ensure that we increment the head node->count before initialising
kernel/locking/qspinlock_paravirt.h +10 -9
@@ -89,7 +89,7 @@ static inline bool pv_hybrid_queued_unfair_trylock(struct qspinlock *lock)if (!(val & _Q_LOCKED_PENDING_MASK) && (cmpxchg_acquire (&lock->locked,0 , _Q_LOCKED_VAL) ==0 )) { -qstat_inc (qstat_pv_lock_stealing, true); +lockevent_inc (pv_lock_stealing);return true; }if (!(val & _Q_TAIL_MASK) || (val & _Q_PENDING_MASK))
@@ -219,7 +219,7 @@ static struct qspinlock **pv_hash(struct qspinlock *lock, struct pv_node *node) hopcnt++;if (!cmpxchg (&he->lock,NULL , lock)) {WRITE_ONCE (he->node, node); -qstat_hop (hopcnt); +lockevent_pv_hop (hopcnt);return &he->lock; } }
@@ -320,8 +320,8 @@ static void pv_wait_node(struct mcs_spinlock *node, struct mcs_spinlock *prev)smp_store_mb (pn->state, vcpu_halted);if (!READ_ONCE (node->locked)) { -qstat_inc (qstat_pv_wait_node, true); -qstat_inc (qstat_pv_wait_early, wait_early); +lockevent_inc (pv_wait_node); +lockevent_cond_inc (pv_wait_early, wait_early);pv_wait (&pn->state, vcpu_halted); }
@@ -339,7 +339,8 @@ static void pv_wait_node(struct mcs_spinlock *node, struct mcs_spinlock *prev)* So it is better to spin for a while in the hope that the * MCS lock will be released soon. */ -qstat_inc (qstat_pv_spurious_wakeup, !READ_ONCE (node->locked)); +lockevent_cond_inc (pv_spurious_wakeup, + !READ_ONCE (node->locked)); }/*
@@ -416,7 +417,7 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node)/* * Tracking # of slowpath locking operations */ -qstat_inc (qstat_lock_slowpath, true); +lockevent_inc (lock_slowpath);for (;; waitcnt++) {/*
@@ -464,8 +465,8 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node) } }WRITE_ONCE (pn->state, vcpu_hashed); -qstat_inc (qstat_pv_wait_head, true); -qstat_inc (qstat_pv_wait_again, waitcnt); +lockevent_inc (pv_wait_head); +lockevent_cond_inc (pv_wait_again, waitcnt);pv_wait (&lock->locked, _Q_SLOW_VAL);/*
@@ -528,7 +529,7 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node)* vCPU is harmless other than the additional latency in completing * the unlock. */ -qstat_inc (qstat_pv_kick_unlock, true); +lockevent_inc (pv_kick_unlock);pv_kick (node->cpu); }
kernel/locking/qspinlock_stat.h +62 -101
301 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 2/22 locking/lock_events: Make lock_events available for all archs & other locks
In reply to Waiman LongThe QUEUED_LOCK_STAT option to report queued spinlocks event counts was previously allowed only on x86 architecture. To make the locking event counting code more useful, it is now renamed to a more generic LOCK_EVENT_COUNTS config option. This new option will be available to all the architectures that use qspinlock at the moment.
Other locking code can now start to use the generic locking event counting code by including lock_events.h and put the new locking event names into the lock_events_list.h header file.
My experience with lock event counting is that it gives valuable insight on how the locking code works and what can be done to make it better. I would like to extend this benefit to other locking code like mutex and rwsem in the near future.
The PV qspinlock specific code will stay in qspinlock_stat.h. The locking event counters will now reside in the <debugfs>/lock_event_counts directory.
--- arch/Kconfig | 10 +++ arch/x86/Kconfig | 8 --- kernel/locking/Makefile | 1 + kernel/locking/lock_events.c | 153 ++++++++++++++++++++++++++++++++++++++++ kernel/locking/lock_events.h | 6 +- kernel/locking/qspinlock_stat.h | 141 ++++-------------------------------- 6 files changed, 179 insertions(+), 140 deletions(-) create mode 100644 kernel/locking/lock_events.c
arch/Kconfig +10 -0
@@ -888,6 +888,16 @@ config HAVE_ARCH_PREL32_RELOCATIONS config ARCH_USE_MEMREMAP_PROT bool +config LOCK_EVENT_COUNTS + bool "Locking event counts collection" + depends on DEBUG_FS + depends on QUEUED_SPINLOCKS + ---help--- + Enable light-weight counting of various locking related events + in the system with minimal performance impact. This reduces + the chance of application behavior change because of timing + differences. The counts are reported via debugfs. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"
arch/x86/Kconfig +0 -8
@@ -784,14 +784,6 @@ config PARAVIRT_SPINLOCKS If you are unsure how to answer this question, answer Y. -config QUEUED_LOCK_STAT - bool "Paravirt queued spinlock statistics" - depends on PARAVIRT_SPINLOCKS && DEBUG_FS - ---help--- - Enable the collection of statistical data on the slowpath - behavior of paravirtualized queued spinlocks and report - them on debugfs. - source "arch/x86/xen/Kconfig" config KVM_GUEST
kernel/locking/Makefile +1 -0
@@ -30,3 +30,4 @@ obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o +obj-$(CONFIG_LOCK_EVENT_COUNTS) += lock_events.o
kernel/locking/lock_events.c new file +153 -0
153 lines not on this page · raw message
kernel/locking/lock_events.h +3 -3
@@ -21,7 +21,7 @@ enum lock_events { LOCKEVENT_reset_cnts = lockevent_num, }; -#ifdef CONFIG_QUEUED_LOCK_STAT +#ifdef CONFIG_LOCK_EVENT_COUNTS /* * Per-cpu counters */
@@ -46,10 +46,10 @@ static inline void __lockevent_add(enum lock_events event, int inc)#define lockevent_add (ev, c) __lockevent_add(LOCKEVENT_ ##ev, c) -#else /* CONFIG_QUEUED_LOCK_STAT */ +#else /* CONFIG_LOCK_EVENT_COUNTS */ #define lockevent_inc (ev)#define lockevent_add (ev, c)#define lockevent_cond_inc (ev, c) -#endif /* CONFIG_QUEUED_LOCK_STAT */ +#endif /* CONFIG_LOCK_EVENT_COUNTS */
kernel/locking/qspinlock_stat.h +12 -129
182 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 3/22 locking/rwsem: Relocate rwsem_down_read_failed()
In reply to Waiman LongThe rwsem_down_read_failed*() functions were relocted from above the optimistic spinning section to below that section. This enables the reader functions to use optimisitic spinning in future patches. There is no code change.
--- kernel/locking/rwsem-xadd.c | 172 ++++++++++++++++++++++---------------------- 1 file changed, 86 insertions(+), 86 deletions(-)
kernel/locking/rwsem-xadd.c +86 -86
@@ -225,92 +225,6 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, }/* -* Wait for the read lock to be granted -*/ -static inline struct rw_semaphore __sched * -__rwsem_down_read_failed_common (struct rw_semaphore *sem,int state) -{ -long count, adjustment = -RWSEM_ACTIVE_READ_BIAS ; -struct rwsem_waiter waiter; -DEFINE_WAKE_Q (wake_q); - - waiter.task = current; - waiter.type =RWSEM_WAITING_FOR_READ ; - -raw_spin_lock_irq (&sem->wait_lock); -if (list_empty (&sem->wait_list)) { -/* -* In case the wait queue is empty and the lock isn't owned -* by a writer, this reader can exit the slowpath and return -* immediately as its RWSEM_ACTIVE_READ_BIAS has already -* been set in the count. -*/ -if (atomic_long_read (&sem->count) >=0 ) { -raw_spin_unlock_irq (&sem->wait_lock); -return sem; - } - adjustment +=RWSEM_WAITING_BIAS ; - } -list_add_tail (&waiter.list, &sem->wait_list); - -/* we're now waiting on the lock, but no longer actively locking */ - count =atomic_long_add_return (adjustment, &sem->count); - -/* -* If there are no active locks, wake the front queued process(es). -* -* If there are no writers and we are first in the queue, -* wake our own waiter to join the existing active readers ! -*/ -if (count ==RWSEM_WAITING_BIAS || - (count >RWSEM_WAITING_BIAS && - adjustment != -RWSEM_ACTIVE_READ_BIAS )) -__rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q); - -raw_spin_unlock_irq (&sem->wait_lock); -wake_up_q (&wake_q); - -/* wait to be given the lock */ -while (true) { -set_current_state (state); -if (!waiter.task) -break ; -if (signal_pending_state (state, current)) { -raw_spin_lock_irq (&sem->wait_lock); -if (waiter.task) - goto out_nolock; -raw_spin_unlock_irq (&sem->wait_lock); -break ; - } -schedule (); - } - -__set_current_state (TASK_RUNNING ); -return sem; -out_nolock: -list_del (&waiter.list); -if (list_empty (&sem->wait_list)) -atomic_long_add (-RWSEM_WAITING_BIAS , &sem->count); -raw_spin_unlock_irq (&sem->wait_lock); -__set_current_state (TASK_RUNNING ); -return ERR_PTR (-EINTR ); -} - -__visible struct rw_semaphore * __sched -rwsem_down_read_failed(struct rw_semaphore *sem) -{ -return __rwsem_down_read_failed_common (sem,TASK_UNINTERRUPTIBLE ); -} -EXPORT_SYMBOL (rwsem_down_read_failed); - -__visible struct rw_semaphore * __sched -rwsem_down_read_failed_killable(struct rw_semaphore *sem) -{ -return __rwsem_down_read_failed_common (sem,TASK_KILLABLE ); -} -EXPORT_SYMBOL (rwsem_down_read_failed_killable); - -/* * This function must be called with the sem->wait_lock held to prevent * race conditions between checking the rwsem wait list and setting the * sem->count accordingly.
@@ -505,6 +419,92 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)#endif /* +* Wait for the read lock to be granted +*/ +static inline struct rw_semaphore __sched * +__rwsem_down_read_failed_common (struct rw_semaphore *sem,int state) +{ +long count, adjustment = -RWSEM_ACTIVE_READ_BIAS ; +struct rwsem_waiter waiter; +DEFINE_WAKE_Q (wake_q); + + waiter.task = current; + waiter.type =RWSEM_WAITING_FOR_READ ; + +raw_spin_lock_irq (&sem->wait_lock); +if (list_empty (&sem->wait_list)) { +/* +* In case the wait queue is empty and the lock isn't owned +* by a writer, this reader can exit the slowpath and return +* immediately as its RWSEM_ACTIVE_READ_BIAS has already +* been set in the count. +*/ +if (atomic_long_read (&sem->count) >=0 ) { +raw_spin_unlock_irq (&sem->wait_lock); +return sem; + } + adjustment +=RWSEM_WAITING_BIAS ; + } +list_add_tail (&waiter.list, &sem->wait_list); + +/* we're now waiting on the lock, but no longer actively locking */ + count =atomic_long_add_return (adjustment, &sem->count); + +/* +* If there are no active locks, wake the front queued process(es). +* +* If there are no writers and we are first in the queue, +* wake our own waiter to join the existing active readers ! +*/ +if (count ==RWSEM_WAITING_BIAS || + (count >RWSEM_WAITING_BIAS && + adjustment != -RWSEM_ACTIVE_READ_BIAS )) +__rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q); + +raw_spin_unlock_irq (&sem->wait_lock); +wake_up_q (&wake_q); + +/* wait to be given the lock */ +while (true) { +set_current_state (state); +if (!waiter.task) +break ; +if (signal_pending_state (state, current)) { +raw_spin_lock_irq (&sem->wait_lock); +if (waiter.task) + goto out_nolock; +raw_spin_unlock_irq (&sem->wait_lock); +break ; + } +schedule (); + } + +__set_current_state (TASK_RUNNING ); +return sem; +out_nolock: +list_del (&waiter.list); +if (list_empty (&sem->wait_list)) +atomic_long_add (-RWSEM_WAITING_BIAS , &sem->count); +raw_spin_unlock_irq (&sem->wait_lock); +__set_current_state (TASK_RUNNING ); +return ERR_PTR (-EINTR ); +} + +__visible struct rw_semaphore * __sched +rwsem_down_read_failed(struct rw_semaphore *sem) +{ +return __rwsem_down_read_failed_common (sem,TASK_UNINTERRUPTIBLE ); +} +EXPORT_SYMBOL (rwsem_down_read_failed); + +__visible struct rw_semaphore * __sched +rwsem_down_read_failed_killable(struct rw_semaphore *sem) +{ +return __rwsem_down_read_failed_common (sem,TASK_KILLABLE ); +} +EXPORT_SYMBOL (rwsem_down_read_failed_killable); + +/* * Wait until we successfully acquire the write lock */ static inline struct rw_semaphore *
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 4/22 locking/rwsem: Remove arch specific rwsem files
In reply to Waiman LongAs the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions.
--- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ 18 files changed, 133 insertions(+), 929 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h
MAINTAINERS +0 -1
@@ -8922,7 +8922,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/
arch/alpha/include/asm/rwsem.h deleted +0 -211
211 lines not on this page · raw message
arch/arm/include/asm/Kbuild +0 -1
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h
arch/hexagon/include/asm/Kbuild +0 -1
@@ -26,7 +26,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h
arch/ia64/include/asm/rwsem.h deleted +0 -172
172 lines not on this page · raw message
arch/powerpc/include/asm/Kbuild +0 -1
@@ -9,6 +9,5 @@ generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h
arch/s390/include/asm/Kbuild +0 -1
@@ -21,7 +21,6 @@ generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h
arch/sparc/include/asm/Kbuild +0 -1
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/x86/include/asm/rwsem.h deleted +0 -237
237 lines not on this page · raw message
arch/x86/lib/Makefile +0 -1
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
arch/x86/lib/rwsem.S deleted +0 -156
156 lines not on this page · raw message
arch/xtensa/include/asm/Kbuild +0 -1
@@ -23,7 +23,6 @@ generic-y += mm-arch-hooks.h generic-y += param.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += topology.h generic-y += trace_clock.h
include/asm-generic/rwsem.h deleted +0 -140
140 lines not on this page · raw message
include/linux/rwsem.h +1 -3
@@ -57,15 +57,13 @@ struct rw_semaphore {extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {return atomic_long_read (&sem->count) !=0 ; } +#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)#endif
kernel/locking/percpu-rwsem.c +2 -0
@@ -7,6 +7,8 @@#include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" +int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key) {
kernel/locking/rwsem.h +130 -0
139 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerRe: [PATCH-tip 04/22] locking/rwsem: Remove arch specific rwsem files
kernel/locking/rwsem.h static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) · line 185
Quoted 10 lines · Waiman Long
kernel/locking/rwsem.h +10
-0+static inline int __down_read_trylock (struct rw_semaphore *sem) +{ +long tmp; + +while ((tmp =atomic_long_read (&sem->count)) >=0 ) { +if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) { +return 1 ; + } + }
Nah, you're supposed to write that like:
for (;;) {
val = atomic_long_cond_read_relaxed(&sem->count, VAL < 0);
if (atomic_long_try_cmpxchg_acquire(&sem->count, &val,
val + RWSEM_ACTIVE_READ_BIAS))
break;
}Review
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}
Anyway, yuck, you're keeping all that BIAS nonsense :/ I was so hoping for a rwsem implementation without that impenetrable crap.
Waiman Long
reviewerRe: [PATCH-tip 04/22] locking/rwsem: Remove arch specific rwsem files
Quoted 14 lines · Peter Zijlstra
kernel/locking/rwsem.h static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) · line 185
Quoted 10 lines · Waiman Long
kernel/locking/rwsem.h +10
-0+static inline int __down_read_trylock (struct rw_semaphore *sem) +{ +long tmp; + +while ((tmp =atomic_long_read (&sem->count)) >=0 ) { +if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) { +return 1 ; + } + }Nah, you're supposed to write that like:
Quoted 7 lines
for (;;) {
val = atomic_long_cond_read_relaxed(&sem->count, VAL < 0);
if (atomic_long_try_cmpxchg_acquire(&sem->count, &val,
val + RWSEM_ACTIVE_READ_BIAS))
break;
}Ah, you are right. I just took it directly from the current generic asm.h file and didn't pay much attention to it. I will fix that in the next iteration.
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}Anyway, yuck, you're keeping all that BIAS nonsense :/ I was so hoping for a rwsem implementation without that impenetrable crap.
Well, I am going to take out all the BIAS handling in a later patch. This one is just for removing the architecture specific files.
I can't change the rwsem count encoding before I take out all the arch specific files.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH-tip 04/22] locking/rwsem: Remove arch specific rwsem files
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
kernel/locking/rwsem.h static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) · line 185
Quoted 10 lines · Waiman Long
kernel/locking/rwsem.h +10
-0+static inline int __down_read_trylock (struct rw_semaphore *sem) +{ +long tmp; + +while ((tmp =atomic_long_read (&sem->count)) >=0 ) { +if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) { +return 1 ; + } + }Nah, you're supposed to write that like:
Quoted 6 lines
for (;;) {
val = atomic_long_cond_read_relaxed(&sem->count, VAL < 0);
if (atomic_long_try_cmpxchg_acquire(&sem->count, &val,
val + RWSEM_ACTIVE_READ_BIAS))
break;
}N/m that's not in fact the same...
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}Anyway, yuck, you're keeping all that BIAS nonsense :/ I was so hoping for a rwsem implementation without that impenetrable crap.
Waiman Long
reviewerPATCH 5/22 locking/rwsem: Move owner setting code from rwsem.c to rwsem.h
In reply to Waiman LongThe setting of owner field is specific to rwsem-xadd code, it is not needed for rwsem-spinlock. This patch moves all the owner setting code closer to the rwsem-xadd fast paths directly within rwsem.h file.
For __down_read() and __down_read_killable(), it is assumed that rwsem will be marked as reader-owned when the functions return. That is currently true except in the transient case that the waiter queue just becomes empty. So a rwsem_set_reader_owned() call is added for this case. The __rwsem_set_reader_owned() call in __rwsem_mark_wake() is now necessary.
--- kernel/locking/rwsem-xadd.c | 6 +++--- kernel/locking/rwsem.c | 19 ++----------------- kernel/locking/rwsem.h | 17 +++++++++++++++-- 3 files changed, 20 insertions(+), 22 deletions(-)
kernel/locking/rwsem-xadd.c +3 -3
@@ -176,9 +176,8 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, goto try_reader_grant; }/* -* It is not really necessary to set it to reader-owned here, -* but it gives the spinners an early indication that the -* readers now have the lock. +* Set it to reader-owned to give spinners an early +* indication that readers now have the lock. */ __rwsem_set_reader_owned (sem, waiter->task); }
@@ -441,6 +440,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)*/ if (atomic_long_read (&sem->count) >=0 ) {raw_spin_unlock_irq (&sem->wait_lock); +rwsem_set_reader_owned (sem);return sem; } adjustment +=RWSEM_WAITING_BIAS ;
kernel/locking/rwsem.c +2 -17
@@ -24,7 +24,6 @@ void __sched down_read(struct rw_semaphore *sem)rwsem_acquire_read (&sem->dep_map,0 ,0 , _RET_IP_);LOCK_CONTENDED (sem, __down_read_trylock, __down_read); -rwsem_set_reader_owned (sem); }EXPORT_SYMBOL (down_read);
@@ -39,7 +38,6 @@ int __sched down_read_killable(struct rw_semaphore *sem)return -EINTR ; } -rwsem_set_reader_owned (sem);return 0 ; }
@@ -52,10 +50,8 @@ int down_read_trylock(struct rw_semaphore *sem) {int ret =__down_read_trylock (sem); -if (ret ==1 ) { +if (ret ==1 )rwsem_acquire_read (&sem->dep_map,0 ,1 , _RET_IP_); -rwsem_set_reader_owned (sem); - }return ret; }
@@ -70,7 +66,6 @@ void __sched down_write(struct rw_semaphore *sem)rwsem_acquire (&sem->dep_map,0 ,0 , _RET_IP_);LOCK_CONTENDED (sem, __down_write_trylock, __down_write); -rwsem_set_owner (sem); }EXPORT_SYMBOL (down_write);
@@ -88,7 +83,6 @@ int __sched down_write_killable(struct rw_semaphore *sem)return -EINTR ; } -rwsem_set_owner (sem);return 0 ; }
@@ -101,10 +95,8 @@ int down_write_trylock(struct rw_semaphore *sem) {int ret =__down_write_trylock (sem); -if (ret ==1 ) { +if (ret ==1 )rwsem_acquire (&sem->dep_map,0 ,1 , _RET_IP_); -rwsem_set_owner (sem); - }return ret; }
@@ -119,7 +111,6 @@ void up_read(struct rw_semaphore *sem)rwsem_release (&sem->dep_map,1 , _RET_IP_);DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED )); -rwsem_clear_reader_owned (sem);__up_read (sem); }
@@ -133,7 +124,6 @@ void up_write(struct rw_semaphore *sem)rwsem_release (&sem->dep_map,1 , _RET_IP_);DEBUG_RWSEMS_WARN_ON (sem->owner != current); -rwsem_clear_owner (sem);__up_write (sem); }
@@ -147,7 +137,6 @@ void downgrade_write(struct rw_semaphore *sem)lock_downgrade (&sem->dep_map, _RET_IP_);DEBUG_RWSEMS_WARN_ON (sem->owner != current); -rwsem_set_reader_owned (sem);__downgrade_write (sem); }
@@ -161,7 +150,6 @@ void down_read_nested(struct rw_semaphore *sem, int subclass)rwsem_acquire_read (&sem->dep_map, subclass,0 , _RET_IP_);LOCK_CONTENDED (sem, __down_read_trylock, __down_read); -rwsem_set_reader_owned (sem); }EXPORT_SYMBOL (down_read_nested);
@@ -172,7 +160,6 @@ void _down_write_nest_lock(struct rw_semaphore *sem, struct lockdep_map *nest)rwsem_acquire_nest (&sem->dep_map,0 ,0 , nest, _RET_IP_);LOCK_CONTENDED (sem, __down_write_trylock, __down_write); -rwsem_set_owner (sem); }EXPORT_SYMBOL (_down_write_nest_lock);
@@ -193,7 +180,6 @@ void down_write_nested(struct rw_semaphore *sem, int subclass)rwsem_acquire (&sem->dep_map, subclass,0 , _RET_IP_);LOCK_CONTENDED (sem, __down_write_trylock, __down_write); -rwsem_set_owner (sem); }EXPORT_SYMBOL (down_write_nested);
@@ -208,7 +194,6 @@ int __sched down_write_killable_nested(struct rw_semaphore *sem, int subclass)return -EINTR ; } -rwsem_set_owner (sem);return 0 ; }
kernel/locking/rwsem.h +15 -2
@@ -161,6 +161,8 @@ static inline void __down_read(struct rw_semaphore *sem) {if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 ))rwsem_down_read_failed (sem); +else +rwsem_set_reader_owned (sem); }static inline int __down_read_killable (struct rw_semaphore *sem)
@@ -168,8 +170,9 @@ static inline int __down_read_killable(struct rw_semaphore *sem)if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 )) {if (IS_ERR (rwsem_down_read_failed_killable (sem)))return -EINTR ; + }else { +rwsem_set_reader_owned (sem); } -return 0 ; }
@@ -180,6 +183,7 @@ static inline int __down_read_trylock(struct rw_semaphore *sem)while ((tmp =atomic_long_read (&sem->count)) >=0 ) {if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, tmp +RWSEM_ACTIVE_READ_BIAS )) { +rwsem_set_reader_owned (sem);return 1 ; } }
@@ -197,6 +201,7 @@ static inline void __down_write(struct rw_semaphore *sem) &sem->count);if (unlikely (tmp !=RWSEM_ACTIVE_WRITE_BIAS ))rwsem_down_write_failed (sem); +rwsem_set_owner (sem); }static inline int __down_write_killable (struct rw_semaphore *sem)
@@ -208,6 +213,7 @@ static inline int __down_write_killable(struct rw_semaphore *sem)if (unlikely (tmp !=RWSEM_ACTIVE_WRITE_BIAS ))if (IS_ERR (rwsem_down_write_failed_killable (sem)))return -EINTR ; +rwsem_set_owner (sem);return 0 ; }
@@ -217,7 +223,11 @@ static inline int __down_write_trylock(struct rw_semaphore *sem) tmp =atomic_long_cmpxchg_acquire (&sem->count,RWSEM_UNLOCKED_VALUE ,RWSEM_ACTIVE_WRITE_BIAS ); -return tmp ==RWSEM_UNLOCKED_VALUE ; +if (tmp ==RWSEM_UNLOCKED_VALUE ) { +rwsem_set_owner (sem); +return true; + } +return false; }/*
@@ -227,6 +237,7 @@ static inline void __up_read(struct rw_semaphore *sem) {long tmp; +rwsem_clear_reader_owned (sem); tmp =atomic_long_dec_return_release (&sem->count);if (unlikely (tmp <-1 && (tmp &RWSEM_ACTIVE_MASK ) ==0 ))rwsem_wake (sem);
@@ -237,6 +248,7 @@ static inline void __up_read(struct rw_semaphore *sem)*/ static inline void __up_write (struct rw_semaphore *sem) { +rwsem_clear_owner (sem);if (unlikely (atomic_long_sub_return_release (RWSEM_ACTIVE_WRITE_BIAS , &sem->count) <0 ))rwsem_wake (sem);
@@ -257,6 +269,7 @@ static inline void __downgrade_write(struct rw_semaphore *sem)* write side. As such, rely on RELEASE semantics. */ tmp =atomic_long_add_return_release (-RWSEM_WAITING_BIAS , &sem->count); +rwsem_set_reader_owned (sem);if (tmp <0 )rwsem_downgrade_wake (sem); }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 6/22 locking/rwsem: Rename kernel/locking/rwsem.h
In reply to Waiman LongThe content of kernel/locking/rwsem.h is now specific to rwsem-xadd only. Rename it to rwsem-xadd.h to indicate that it is specific to rwsem-xadd and include it only when CONFIG_RWSEM_XCHGADD_ALGORITHM is set.
--- kernel/locking/percpu-rwsem.c | 4 +- kernel/locking/rwsem-xadd.c | 2 +- kernel/locking/rwsem-xadd.h | 274 +++++++++++++++++++++++++++++++++++++++++ kernel/locking/rwsem.c | 7 +- kernel/locking/rwsem.h | 277 ------------------------------------------ 5 files changed, 284 insertions(+), 280 deletions(-) create mode 100644 kernel/locking/rwsem-xadd.h delete mode 100644 kernel/locking/rwsem.h
kernel/locking/percpu-rwsem.c +3 -1
@@ -7,7 +7,9 @@#include <linux/sched.h> #include <linux/errno.h> -#include "rwsem.h" +#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM +#include "rwsem-xadd.h" +#endif int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key)
kernel/locking/rwsem-xadd.c +1 -1
@@ -19,7 +19,7 @@#include <linux/sched/debug.h> #include <linux/osq_lock.h> -#include "rwsem.h" +#include "rwsem-xadd.h" /* * Guide to the rw_semaphore's count field for common values.
kernel/locking/rwsem-xadd.h new file +274 -0
274 lines not on this page · raw message
kernel/locking/rwsem.c +6 -1
@@ -13,7 +13,12 @@#include <linux/rwsem.h> #include <linux/atomic.h> -#include "rwsem.h" +#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM +#include "rwsem-xadd.h" +#else +#define __rwsem_set_reader_owned (a, b) +#define DEBUG_RWSEMS_WARN_ON (a) +#endif /* * lock for reading
kernel/locking/rwsem.h deleted +0 -277
277 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 7/22 locking/rwsem: Move rwsem internal function declarations to rwsem-xadd.h
In reply to Waiman LongWe don't need to expose rwsem internal functions which are not supposed to be called directly from other kernel code.
--- include/linux/rwsem.h | 7 ------- kernel/locking/rwsem-xadd.h | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-)
include/linux/rwsem.h +0 -7
@@ -50,13 +50,6 @@ struct rw_semaphore {*/ #define RWSEM_OWNER_UNKNOWN ((struct task_struct *)-2L) -extern struct rw_semaphore *rwsem_down_read_failed (struct rw_semaphore *sem); -extern struct rw_semaphore *rwsem_down_read_failed_killable (struct rw_semaphore *sem); -extern struct rw_semaphore *rwsem_down_write_failed (struct rw_semaphore *sem); -extern struct rw_semaphore *rwsem_down_write_failed_killable (struct rw_semaphore *sem); -extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *); -extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {
kernel/locking/rwsem-xadd.h +7 -0
@@ -153,6 +153,13 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) }#endif +extern struct rw_semaphore *rwsem_down_read_failed (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_down_read_failed_killable (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_down_write_failed (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_down_write_failed_killable (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); +/* * lock for reading */
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 8/22 locking/rwsem: Add debug check for __down_read*()
In reply to Waiman LongWhen rwsem_down_read_failed*() return, the read lock is acquired indirectly by others. So debug checks are added in __down_read() and __down_read_killable() to make sure the rwsem is really reader-owned.
The other debug check calls in kernel/locking/rwsem.c except the one in up_read_non_owner() are also moved over to rwsem-xadd.h.
--- kernel/locking/rwsem-xadd.h | 12 ++++++++++-- kernel/locking/rwsem.c | 3 --- 2 files changed, 10 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.h +10 -2
@@ -165,10 +165,13 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)*/ static inline void __down_read (struct rw_semaphore *sem) { -if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 )) +if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 )) {rwsem_down_read_failed (sem); -else +DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner & +RWSEM_READER_OWNED )); + }else {rwsem_set_reader_owned (sem); + } }static inline int __down_read_killable (struct rw_semaphore *sem)
@@ -176,6 +179,8 @@ static inline int __down_read_killable(struct rw_semaphore *sem)if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 )) {if (IS_ERR (rwsem_down_read_failed_killable (sem)))return -EINTR ; +DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner & +RWSEM_READER_OWNED )); }else {rwsem_set_reader_owned (sem); }
@@ -243,6 +248,7 @@ static inline void __up_read(struct rw_semaphore *sem) {long tmp; +DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ));rwsem_clear_reader_owned (sem); tmp =atomic_long_dec_return_release (&sem->count);if (unlikely (tmp <-1 && (tmp &RWSEM_ACTIVE_MASK ) ==0 ))
@@ -254,6 +260,7 @@ static inline void __up_read(struct rw_semaphore *sem)*/ static inline void __up_write (struct rw_semaphore *sem) { +DEBUG_RWSEMS_WARN_ON (sem->owner != current);rwsem_clear_owner (sem);if (unlikely (atomic_long_sub_return_release (RWSEM_ACTIVE_WRITE_BIAS , &sem->count) <0 ))
@@ -274,6 +281,7 @@ static inline void __downgrade_write(struct rw_semaphore *sem)* read-locked region is ok to be re-ordered into the * write side. As such, rely on RELEASE semantics. */ +DEBUG_RWSEMS_WARN_ON (sem->owner != current); tmp =atomic_long_add_return_release (-RWSEM_WAITING_BIAS , &sem->count);rwsem_set_reader_owned (sem);if (tmp <0 )
kernel/locking/rwsem.c +0 -3
@@ -114,7 +114,6 @@ int down_write_trylock(struct rw_semaphore *sem)void up_read (struct rw_semaphore *sem) {rwsem_release (&sem->dep_map,1 , _RET_IP_); -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ));__up_read (sem); }
@@ -127,7 +126,6 @@ void up_read(struct rw_semaphore *sem)void up_write (struct rw_semaphore *sem) {rwsem_release (&sem->dep_map,1 , _RET_IP_); -DEBUG_RWSEMS_WARN_ON (sem->owner != current);__up_write (sem); }
@@ -140,7 +138,6 @@ void up_write(struct rw_semaphore *sem)void downgrade_write (struct rw_semaphore *sem) {lock_downgrade (&sem->dep_map, _RET_IP_); -DEBUG_RWSEMS_WARN_ON (sem->owner != current);__downgrade_write (sem); }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 9/22 locking/rwsem: Enhance DEBUG_RWSEMS_WARN_ON() macro
In reply to Waiman LongCurrently, the DEBUG_RWSEMS_WARN_ON() macro just dumps a stack trace when the rwsem isn't in the right state. It does not show the actual states of the rwsem. This may not be that helpful in the debugging process.
Enhance the DEBUG_RWSEMS_WARN_ON() macro to also show the current content of the rwsem count and owner fields to give more information about what is wrong with the rwsem.
--- kernel/locking/rwsem-xadd.h | 19 ++++++++++++------- kernel/locking/rwsem.c | 5 +++-- 2 files changed, 15 insertions(+), 9 deletions(-)
kernel/locking/rwsem-xadd.h +12 -7
@@ -27,9 +27,13 @@#define RWSEM_ANONYMOUSLY_OWNED (1UL << 1)#ifdef CONFIG_DEBUG_RWSEMS -# define DEBUG_RWSEMS_WARN_ON (c) DEBUG_LOCKS_WARN_ON(c) +# define DEBUG_RWSEMS_WARN_ON (c, sem) \ + WARN_ONCE(c, "DEBUG_RWSEMS_WARN_ON(%s): count = 0x%lx, owner = 0x%lx, curr 0x%lx, list %sempty\n",\ + #c, atomic_long_read(&(sem)->count), \ + (long)((sem)->owner), (long)current, \ + list_empty(&(sem)->wait_list) ? "" : "not ")#else -# define DEBUG_RWSEMS_WARN_ON (c) +# define DEBUG_RWSEMS_WARN_ON (c, sem)#endif /*
@@ -168,7 +172,7 @@ static inline void __down_read(struct rw_semaphore *sem)if (unlikely (atomic_long_inc_return_acquire (&sem->count) <=0 )) {rwsem_down_read_failed (sem);DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner & -RWSEM_READER_OWNED )); +RWSEM_READER_OWNED ), sem); }else {rwsem_set_reader_owned (sem); }
@@ -180,7 +184,7 @@ static inline int __down_read_killable(struct rw_semaphore *sem)if (IS_ERR (rwsem_down_read_failed_killable (sem)))return -EINTR ;DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner & -RWSEM_READER_OWNED )); +RWSEM_READER_OWNED ), sem); }else {rwsem_set_reader_owned (sem); }
@@ -248,7 +252,8 @@ static inline void __up_read(struct rw_semaphore *sem) {long tmp; -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED )); +DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ), + sem);rwsem_clear_reader_owned (sem); tmp =atomic_long_dec_return_release (&sem->count);if (unlikely (tmp <-1 && (tmp &RWSEM_ACTIVE_MASK ) ==0 ))
@@ -260,7 +265,7 @@ static inline void __up_read(struct rw_semaphore *sem)*/ static inline void __up_write (struct rw_semaphore *sem) { -DEBUG_RWSEMS_WARN_ON (sem->owner != current); +DEBUG_RWSEMS_WARN_ON (sem->owner != current, sem);rwsem_clear_owner (sem);if (unlikely (atomic_long_sub_return_release (RWSEM_ACTIVE_WRITE_BIAS , &sem->count) <0 ))
@@ -281,7 +286,7 @@ static inline void __downgrade_write(struct rw_semaphore *sem)* read-locked region is ok to be re-ordered into the * write side. As such, rely on RELEASE semantics. */ -DEBUG_RWSEMS_WARN_ON (sem->owner != current); +DEBUG_RWSEMS_WARN_ON (sem->owner != current, sem); tmp =atomic_long_add_return_release (-RWSEM_WAITING_BIAS , &sem->count);rwsem_set_reader_owned (sem);if (tmp <0 )
kernel/locking/rwsem.c +3 -2
@@ -17,7 +17,7 @@#include "rwsem-xadd.h" #else #define __rwsem_set_reader_owned (a, b) -#define DEBUG_RWSEMS_WARN_ON (a) +#define DEBUG_RWSEMS_WARN_ON (a, b)#endif /*
@@ -203,7 +203,8 @@ int __sched down_write_killable_nested(struct rw_semaphore *sem, int subclass)void up_read_non_owner (struct rw_semaphore *sem) { -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED )); +DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ), + sem);__up_read (sem); }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 10/22 locking/rwsem: Enable lock event counting
In reply to Waiman LongAdd lock event counting calls so that we can track the number of lock events happening in the rwsem code.
With CONFIG_LOCK_EVENT_COUNTS on and booting a 1-socket 22-core 44-thread x86-64 system, the non-zero rwsem counts after system bootup were as follows:
rwsem_opt_fail=113
rwsem_opt_wlock=13647
rwsem_rlock=176
rwsem_rlock_fast=10
rwsem_wake_reader=153
rwsem_wake_writer=139
rwsem_wlock=113
It can be seen that most of the lock acquisitions in the slowpath were writer-locks in the optimistic spinning code path with no sleeping at all. Only about 4% of locks were acquired after sleeping.
--- arch/Kconfig | 2 +- kernel/locking/lock_events_list.h | 17 +++++++++++++++++ kernel/locking/rwsem-xadd.c | 12 ++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-)
arch/Kconfig +1 -1
@@ -891,7 +891,7 @@ config ARCH_USE_MEMREMAP_PROT config LOCK_EVENT_COUNTS bool "Locking event counts collection" depends on DEBUG_FS - depends on QUEUED_SPINLOCKS + depends on (QUEUED_SPINLOCKS || RWSEM_XCHGADD_ALGORITHM) ---help--- Enable light-weight counting of various locking related events in the system with minimal performance impact. This reduces
kernel/locking/lock_events_list.h +17 -0
@@ -48,3 +48,20 @@ LOCK_EVENT(lock_use_node4) /* # of locking ops that use 4th percpu node */ LOCK_EVENT(lock_no_node) /* # of locking ops w/o using percpu node */ #endif /* CONFIG_QUEUED_SPINLOCKS */ + +#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM +/* + * Locking events for rwsem + */ +LOCK_EVENT(rwsem_sleep_reader) /* # of reader sleeps */ +LOCK_EVENT(rwsem_sleep_writer) /* # of writer sleeps */ +LOCK_EVENT(rwsem_wake_reader) /* # of reader wakeups */ +LOCK_EVENT(rwsem_wake_writer) /* # of writer wakeups */ +LOCK_EVENT(rwsem_opt_wlock) /* # of write locks opt-spin acquired */ +LOCK_EVENT(rwsem_opt_fail) /* # of failed opt-spinnings */ +LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ +LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ +LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */ +LOCK_EVENT(rwsem_wlock) /* # of write locks acquired */ +LOCK_EVENT(rwsem_wlock_fail) /* # of failed write lock acquisitions */ +#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
kernel/locking/rwsem-xadd.c +12 -0
@@ -20,6 +20,7 @@#include <linux/osq_lock.h> #include "rwsem-xadd.h" +#include "lock_events.h" /* * Guide to the rw_semaphore's count field for common values.
@@ -147,6 +148,7 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,* will notice the queued writer. */ wake_q_add (wake_q, waiter->task); +lockevent_inc (rwsem_wake_writer); }return ;
@@ -214,6 +216,7 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, } adjustment = woken *RWSEM_ACTIVE_READ_BIAS - adjustment; +lockevent_cond_inc (rwsem_wake_reader, woken);if (list_empty (&sem->wait_list)) {/* hit end of list above */ adjustment -=RWSEM_WAITING_BIAS ;
@@ -269,6 +272,7 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) count + RWSEM_ACTIVE_WRITE_BIAS);if (old == count) {rwsem_set_owner (sem); +lockevent_inc (rwsem_opt_wlock);return true; }
@@ -394,6 +398,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)osq_unlock (&sem->osq); done:preempt_enable (); +lockevent_cond_inc (rwsem_opt_fail, !taken);return taken; }
@@ -441,6 +446,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)if (atomic_long_read (&sem->count) >=0 ) {raw_spin_unlock_irq (&sem->wait_lock);rwsem_set_reader_owned (sem); +lockevent_inc (rwsem_rlock_fast);return sem; } adjustment +=RWSEM_WAITING_BIAS ;
@@ -477,9 +483,11 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)break ; }schedule (); +lockevent_inc (rwsem_sleep_reader); }__set_current_state (TASK_RUNNING ); +lockevent_inc (rwsem_rlock);return sem; out_nolock:list_del (&waiter.list);
@@ -487,6 +495,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)atomic_long_add (-RWSEM_WAITING_BIAS , &sem->count);raw_spin_unlock_irq (&sem->wait_lock);__set_current_state (TASK_RUNNING ); +lockevent_inc (rwsem_rlock_fail);return ERR_PTR (-EINTR ); }
@@ -580,6 +589,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem) goto out_nolock;schedule (); +lockevent_inc (rwsem_sleep_writer);set_current_state (state); }while ((count =atomic_long_read (&sem->count)) &RWSEM_ACTIVE_MASK );
@@ -588,6 +598,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)__set_current_state (TASK_RUNNING );list_del (&waiter.list);raw_spin_unlock_irq (&sem->wait_lock); +lockevent_inc (rwsem_wlock);return ret;
@@ -601,6 +612,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)__rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q);raw_spin_unlock_irq (&sem->wait_lock);wake_up_q (&wake_q); +lockevent_inc (rwsem_wlock_fail);return ERR_PTR (-EINTR ); }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 11/22 locking/rwsem: Implement a new locking scheme
In reply to Waiman LongThe current way of using various reader, writer and waiting biases in the rwsem code are confusing and hard to understand. I have to reread the rwsem count guide in the rwsem-xadd.c file from time to time to remind myself how this whole thing works. It also makes the rwsem code harder to be optimized.
To make rwsem more sane, a new locking scheme similar to the one in qrwlock is now being used. The count is now a 32-bit atomic value in all architectures. The current bit definitions are:
Bit 0 - writer locked bit Bit 1 - waiters present bit
Bits 2-7 - reserved for future extension
Bits 8-X - reader count (24/56 bits)
Now the cmpxchg instruction is used to acquire the write lock. The read lock is still acquired with xadd instruction, so there is no change here. This scheme will allow up to 16M active readers which should be more than enough. We can always use some more reserved bits if necessary.
With that change, we can deterministically know if a rwsem has been write-locked. Looking at the count alone, however, one cannot determine for certain if a rwsem is owned by readers or not as the readers that set the reader count bits may be in the process of backing out. So we still need the reader-owned bit in the owner field to be sure.
With a locking microbenchmark running on 5.0 based kernel, the total locking rates (in kops/s) of the benchmark on a 4-socket 56-core x86-64 system before and after the patch were as follows:
Before Patch After Patch
# of Threads wlock rlock wlock rlock
------------ ----- ----- ----- -----
1 28,773 30,164 29,063 29,967
2 7,435 15,167 6,746 13,198
4 7,181 14,438 6,515 12,763
8 6,918 13,796 6,747 13,419
16 6,554 16,030 6,653 15,534The locking rates of the benchmark on a Power9 system were as follows:
Before Patch After Patch
# of Threads wlock rlock wlock rlock
------------ ----- ----- ----- -----
1 11,527 12,378 11,527 12,195
2 6,318 15,116 7,690 15,065
4 3,957 16,812 4,495 16,884
8 3,617 16,496 3,993 16,588The locking rates of the benchmark on a 2-socket ARM64 system were as follows:
Before Patch After Patch
# of Threads wlock rlock wlock rlock
------------ ----- ----- ----- -----
1 23,703 23,208 23,923 23,167
2 6,206 11,996 6,756 13,268
4 6,098 12,343 6,928 15,064
8 6,007 13,152 7,117 15,335For x86, the locking rate dropped somewhat at low contention level with the patch. At high contention level, however, the performance recovered. For both Power9 and ARM64, the patch caused no performance drop. In fact, it improved a little bit after the patch.
--- kernel/locking/rwsem-xadd.c | 145 +++++++++++++++----------------------------- kernel/locking/rwsem-xadd.h | 85 +++++++++++++------------- 2 files changed, 89 insertions(+), 141 deletions(-)
kernel/locking/rwsem-xadd.c +48 -97
@@ -9,6 +9,8 @@* * Optimistic spinning by Tim Chen <[email protected]> * and Davidlohr Bueso <[email protected]>. Based on mutexes. +* +* Rwsem count bit fields re-definition by Waiman Long <[email protected]>. */ #include <linux/rwsem.h> #include <linux/init.h>
@@ -23,52 +25,20 @@#include "lock_events.h" /* -* Guide to the rw_semaphore's count field for common values. -* (32-bit case illustrated, similar for 64-bit) -* -* 0x0000000X (1) X readers active or attempting lock, no writer waiting -* X = #active_readers + #readers attempting to lock -* (X*ACTIVE_BIAS) -* -* 0x00000000 rwsem is unlocked, and no one is waiting for the lock or -* attempting to read lock or write lock. -* -* 0xffff000X (1) X readers active or attempting lock, with waiters for lock -* X = #active readers + # readers attempting lock -* (X*ACTIVE_BIAS + WAITING_BIAS) -* (2) 1 writer attempting lock, no waiters for lock -* X-1 = #active readers + #readers attempting lock -* ((X-1)*ACTIVE_BIAS + ACTIVE_WRITE_BIAS) -* (3) 1 writer active, no waiters for lock -* X-1 = #active readers + #readers attempting lock -* ((X-1)*ACTIVE_BIAS + ACTIVE_WRITE_BIAS) -* -* 0xffff0001 (1) 1 reader active or attempting lock, waiters for lock -* (WAITING_BIAS + ACTIVE_BIAS) -* (2) 1 writer active or attempting lock, no waiters for lock -* (ACTIVE_WRITE_BIAS) +* Guide to the rw_semaphore's count field. * -* 0xffff0000 (1) There are writers or readers queued but none active -* or in the process of attempting lock. -* (WAITING_BIAS) -* Note: writer can attempt to steal lock for this count by adding -* ACTIVE_WRITE_BIAS in cmpxchg and checking the old count +* When the RWSEM_WRITER_LOCKED bit in count is set, the lock is owned +* by a writer. * -* 0xfffe0001 (1) 1 writer active, or attempting lock. Waiters on queue. -* (ACTIVE_WRITE_BIAS + WAITING_BIAS) -* -* Note: Readers attempt to lock by adding ACTIVE_BIAS in down_read and checking -* the count becomes more than 0 for successful lock acquisition, -* i.e. the case where there are only readers or nobody has lock. -* (1st and 2nd case above). -* -* Writers attempt to lock by adding ACTIVE_WRITE_BIAS in down_write and -* checking the count becomes ACTIVE_WRITE_BIAS for successful lock -* acquisition (i.e. nobody else has lock or attempts lock). If -* unsuccessful, in rwsem_down_write_failed, we'll check to see if there -* are only waiters but none active (5th case above), and attempt to -* steal the lock. +* The lock is owned by readers when +* (1) the RWSEM_WRITER_LOCKED isn't set in count, +* (2) some of the reader bits are set in count, and +* (3) the owner field has RWSEM_READ_OWNED bit set. * +* Having some reader bits set is not enough to guarantee a readers owned +* lock as the readers may be in the process of backing out from the count +* and a writer has just released the lock. So another writer may steal +* the lock immediately after that. */ /*
@@ -114,9 +84,8 @@ enum rwsem_wake_type {/* * handle the lock release when processes blocked on it that can now run -* - if we come here from up_xxxx(), then: -* - the 'active part' of count (&0x0000ffff) reached 0 (but may have changed) -* - the 'waiting part' of count (&0xffff0000) is -ve (and will still be so) +* - if we come here from up_xxxx(), then the RWSEM_FLAG_WAITERS bit must +* have been set. * - there must be someone on the queue * - the wait_lock must be held by the caller * - tasks are marked for wakeup, the caller must later invoke wake_up_q()
@@ -160,22 +129,11 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,* so we can bail out early if a writer stole the lock. */ if (wake_type !=RWSEM_WAKE_READ_OWNED ) { - adjustment =RWSEM_ACTIVE_READ_BIAS ; - try_reader_grant: + adjustment =RWSEM_READER_BIAS ; oldcount =atomic_long_fetch_add (adjustment, &sem->count); -if (unlikely (oldcount <RWSEM_WAITING_BIAS )) { -/* -* If the count is still less than RWSEM_WAITING_BIAS -* after removing the adjustment, it is assumed that -* a writer has stolen the lock. We have to undo our -* reader grant. -*/ -if (atomic_long_add_return (-adjustment, &sem->count) < -RWSEM_WAITING_BIAS ) -return ; - -/* Last active locker left. Retry waking readers. */ - goto try_reader_grant; +if (unlikely (oldcount &RWSEM_WRITER_MASK )) { +atomic_long_sub (adjustment, &sem->count); +return ; }/* * Set it to reader-owned to give spinners an early
@@ -215,11 +173,11 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,wake_q_add_safe (wake_q, tsk); } - adjustment = woken *RWSEM_ACTIVE_READ_BIAS - adjustment; + adjustment = woken *RWSEM_READER_BIAS - adjustment;lockevent_cond_inc (rwsem_wake_reader, woken);if (list_empty (&sem->wait_list)) {/* hit end of list above */ - adjustment -=RWSEM_WAITING_BIAS ; + adjustment -=RWSEM_FLAG_WAITERS ; }if (adjustment)
@@ -233,22 +191,15 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,*/ static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem) { -/* -* Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS. -*/ -if (count !=RWSEM_WAITING_BIAS ) +long new; + +if (RWSEM_COUNT_LOCKED (count))return false; -/* -* Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there -* are other tasks on the wait list, we need to add on WAITING_BIAS. -*/ - count =list_is_singular (&sem->wait_list) ? -RWSEM_ACTIVE_WRITE_BIAS : -RWSEM_ACTIVE_WRITE_BIAS +RWSEM_WAITING_BIAS ; + new = count +RWSEM_WRITER_LOCKED - + (list_is_singular (&sem->wait_list) ?RWSEM_FLAG_WAITERS :0 ); -if (atomic_long_cmpxchg_acquire (&sem->count,RWSEM_WAITING_BIAS , count) - ==RWSEM_WAITING_BIAS ) { +if (atomic_long_cmpxchg_acquire (&sem->count, count, new) == count) {rwsem_set_owner (sem);return true; }
@@ -265,11 +216,11 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)long old, count =atomic_long_read (&sem->count);while (true) { -if (!(count ==0 || count ==RWSEM_WAITING_BIAS )) +if (RWSEM_COUNT_LOCKED (count))return false; old =atomic_long_cmpxchg_acquire (&sem->count, count, - count +RWSEM_ACTIVE_WRITE_BIAS ); + count +RWSEM_WRITER_LOCKED );if (old == count) {rwsem_set_owner (sem);lockevent_inc (rwsem_opt_wlock);
@@ -428,7 +379,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem) static inline struct rw_semaphore __sched *__rwsem_down_read_failed_common (struct rw_semaphore *sem,int state) { -long count, adjustment = -RWSEM_ACTIVE_READ_BIAS ; +long count, adjustment = -RWSEM_READER_BIAS ;struct rwsem_waiter waiter;DEFINE_WAKE_Q (wake_q);
@@ -440,16 +391,16 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)/* * In case the wait queue is empty and the lock isn't owned * by a writer, this reader can exit the slowpath and return -* immediately as its RWSEM_ACTIVE_READ_BIAS has already -* been set in the count. +* immediately as its RWSEM_READER_BIAS has already been +* set in the count. */ -if (atomic_long_read (&sem->count) >=0 ) { +if (!(atomic_long_read (&sem->count) &RWSEM_WRITER_MASK )) {raw_spin_unlock_irq (&sem->wait_lock);rwsem_set_reader_owned (sem);lockevent_inc (rwsem_rlock_fast);return sem; } - adjustment +=RWSEM_WAITING_BIAS ; + adjustment +=RWSEM_FLAG_WAITERS ; }list_add_tail (&waiter.list, &sem->wait_list);
@@ -462,9 +413,8 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)* If there are no writers and we are first in the queue, * wake our own waiter to join the existing active readers ! */ -if (count ==RWSEM_WAITING_BIAS || - (count >RWSEM_WAITING_BIAS && - adjustment != -RWSEM_ACTIVE_READ_BIAS )) +if (!RWSEM_COUNT_LOCKED (count) || + (!(count &RWSEM_WRITER_MASK ) && (adjustment &RWSEM_FLAG_WAITERS )))__rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q);raw_spin_unlock_irq (&sem->wait_lock);
@@ -492,7 +442,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem) out_nolock:list_del (&waiter.list);if (list_empty (&sem->wait_list)) -atomic_long_add (-RWSEM_WAITING_BIAS , &sem->count); +atomic_long_add (-RWSEM_FLAG_WAITERS , &sem->count);raw_spin_unlock_irq (&sem->wait_lock);__set_current_state (TASK_RUNNING );lockevent_inc (rwsem_rlock_fail);
@@ -525,9 +475,6 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)struct rw_semaphore *ret = sem;DEFINE_WAKE_Q (wake_q); -/* undo write bias from down_write operation, stop active locking */ - count =atomic_long_sub_return (RWSEM_ACTIVE_WRITE_BIAS , &sem->count); -/* do optimistic spinning and steal lock if possible */ if (rwsem_optimistic_spin (sem))return sem;
@@ -553,10 +500,12 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)/* * If there were already threads queued before us and there are -* no active writers, the lock must be read owned; so we try to -* wake any read locks that were queued ahead of us. +* no active writers and some readers, the lock must be read +* owned; so we try to any read locks that were queued ahead +* of us. */ -if (count >RWSEM_WAITING_BIAS ) { +if (!(count &RWSEM_WRITER_MASK ) && + (count &RWSEM_READER_MASK )) {__rwsem_mark_wake (sem,RWSEM_WAKE_READERS , &wake_q);/* * The wakeup is normally called _after_ the wait_lock
@@ -573,8 +522,9 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)wake_q_init (&wake_q); } - }else - count =atomic_long_add_return (RWSEM_WAITING_BIAS , &sem->count); + }else { + count =atomic_long_add_return (RWSEM_FLAG_WAITERS , &sem->count); + }/* wait until we successfully acquire the lock */ set_current_state (state);
@@ -591,7 +541,8 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)schedule ();lockevent_inc (rwsem_sleep_writer);set_current_state (state); - }while ((count =atomic_long_read (&sem->count)) &RWSEM_ACTIVE_MASK ); + count =atomic_long_read (&sem->count); + }while (RWSEM_COUNT_LOCKED (count));raw_spin_lock_irq (&sem->wait_lock); }
@@ -607,7 +558,7 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)raw_spin_lock_irq (&sem->wait_lock);list_del (&waiter.list);if (list_empty (&sem->wait_list)) -atomic_long_add (-RWSEM_WAITING_BIAS , &sem->count); +atomic_long_add (-RWSEM_FLAG_WAITERS , &sem->count);else __rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q);raw_spin_unlock_irq (&sem->wait_lock);
kernel/locking/rwsem-xadd.h +41 -44
149 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 12/22 locking/rwsem: Implement lock handoff to prevent lock starvation
In reply to Waiman LongBecause of writer lock stealing, it is possible that a constant stream of incoming writers will cause a waiting writer or reader to wait indefinitely leading to lock starvation.
The mutex code has a lock handoff mechanism to prevent lock starvation. This patch implements a similar lock handoff mechanism to disable lock stealing and force lock handoff to the first waiter in the queue after at least a 5ms waiting period. The waiting period is used to avoid discouraging lock stealing too much to affect performance.
A rwsem microbenchmark was run for 8 seconds on a 4-socket 56-core 128-thread x86-64 system with a v5.0 based kernel and 112 write_lock threads with 5us sleep critical section.
Before the patch, the min/mean/max numbers of locking operations for the locking threads were 1/7,708/542,988. After the patch, the figures became 6,031/7,267/9,476. It can be seen that the rwsem became much more fair, though there was a slight drop of about 6% in the mean locking operations done which was a tradeoff of having better fairness.
--- kernel/locking/lock_events_list.h | 2 + kernel/locking/rwsem-xadd.c | 110 +++++++++++++++++++++++++++++++------- kernel/locking/rwsem-xadd.h | 23 +++++--- 3 files changed, 109 insertions(+), 26 deletions(-)
kernel/locking/lock_events_list.h +2 -0
@@ -62,6 +62,8 @@ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */ +LOCK_EVENT(rwsem_rlock_handoff) /* # of read lock handoffs */ LOCK_EVENT(rwsem_wlock) /* # of write locks acquired */ LOCK_EVENT(rwsem_wlock_fail) /* # of failed write lock acquisitions */ +LOCK_EVENT(rwsem_wlock_handoff) /* # of write lock handoffs */#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
kernel/locking/rwsem-xadd.c +91 -19
231 lines not on this page · raw message
kernel/locking/rwsem-xadd.h +16 -7
@@ -41,7 +41,8 @@* * Bit 0 - writer locked bit * Bit 1 - waiters present bit -* Bits 2-7 - reserved +* Bit 2 - lock handoff bit +* Bits 3-7 - reserved * Bits 8-X - 24-bit (32-bit) or 56-bit reader count * * atomic_long_fetch_add() is used to obtain reader lock, whereas
@@ -49,14 +50,20 @@*/ #define RWSEM_WRITER_LOCKED (1UL << 0)#define RWSEM_FLAG_WAITERS (1UL << 1) +#define RWSEM_FLAG_HANDOFF (1UL << 2) +#define RWSEM_READER_SHIFT 8#define RWSEM_READER_BIAS (1UL << RWSEM_READER_SHIFT)#define RWSEM_READER_MASK (~(RWSEM_READER_BIAS - 1))#define RWSEM_WRITER_MASK RWSEM_WRITER_LOCKED#define RWSEM_LOCK_MASK (RWSEM_WRITER_MASK|RWSEM_READER_MASK) -#define RWSEM_READ_FAILED_MASK (RWSEM_WRITER_MASK|RWSEM_FLAG_WAITERS) +#define RWSEM_READ_FAILED_MASK (RWSEM_WRITER_MASK|RWSEM_FLAG_WAITERS|\ + RWSEM_FLAG_HANDOFF)#define RWSEM_COUNT_LOCKED (c) ((c) & RWSEM_LOCK_MASK) +#define RWSEM_COUNT_HANDOFF (c) ((c) & RWSEM_FLAG_HANDOFF) +#define RWSEM_COUNT_LOCKED_OR_HANDOFF (c) \ + ((c) & (RWSEM_LOCK_MASK|RWSEM_FLAG_HANDOFF))#ifdef CONFIG_RWSEM_SPIN_ON_OWNER /*
@@ -163,7 +170,7 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)extern struct rw_semaphore *rwsem_down_read_failed_killable (struct rw_semaphore *sem);extern struct rw_semaphore *rwsem_down_write_failed (struct rw_semaphore *sem);extern struct rw_semaphore *rwsem_down_write_failed_killable (struct rw_semaphore *sem); -extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *sem); +extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *sem,long count);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem);/*
@@ -253,7 +260,7 @@ static inline void __up_read(struct rw_semaphore *sem) tmp =atomic_long_add_return_release (-RWSEM_READER_BIAS , &sem->count);if (unlikely ((tmp & (RWSEM_LOCK_MASK |RWSEM_FLAG_WAITERS )) ==RWSEM_FLAG_WAITERS )) -rwsem_wake (sem); +rwsem_wake (sem, tmp); }/*
@@ -261,11 +268,13 @@ static inline void __up_read(struct rw_semaphore *sem)*/ static inline void __up_write (struct rw_semaphore *sem) { +long tmp; +DEBUG_RWSEMS_WARN_ON (sem->owner != current, sem);rwsem_clear_owner (sem); -if (unlikely (atomic_long_fetch_add_release (-RWSEM_WRITER_LOCKED , - &sem->count) &RWSEM_FLAG_WAITERS )) -rwsem_wake (sem); + tmp =atomic_long_fetch_add_release (-RWSEM_WRITER_LOCKED , &sem->count); +if (unlikely (tmp &RWSEM_FLAG_WAITERS )) +rwsem_wake (sem, tmp); }/*
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 13/22 locking/rwsem: Remove rwsem_wake() wakeup optimization
In reply to Waiman LongWith the commit 59aabfc7e959 ("locking/rwsem: Reduce spinlock contention in wakeup after up_read()/up_write()"), the rwsem_wake() forgoes doing a wakeup if the wait_lock cannot be directly acquired and an optimistic spinning locker is present. This can help performance by avoiding spinning on the wait_lock when it is contended.
With the later commit 133e89ef5ef3 ("locking/rwsem: Enable lockless waiter wakeup(s)"), the performance advantage of the above optimization diminishes as the average wait_lock hold time become much shorter.
By supporting rwsem lock handoff, we can no longer relies on the fact that the presence of an optimistic spinning locker will ensure that the lock will be acquired by a task soon. This can lead to missed wakeup and application hang. So the commit 59aabfc7e959 ("locking/rwsem:
Reduce spinlock contention in wakeup after up_read()/up_write()") will have to be reverted.
--- kernel/locking/rwsem-xadd.c | 74 --------------------------------------------- 1 file changed, 74 deletions(-)
kernel/locking/rwsem-xadd.c +0 -74
@@ -378,25 +378,11 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)lockevent_cond_inc (rwsem_opt_fail, !taken); return taken; } - -/* -* Return true if the rwsem has active spinner -*/ -static inline bool rwsem_has_spinner (struct rw_semaphore *sem) -{ -return osq_is_locked (&sem->osq); -} -#else static bool rwsem_optimistic_spin (struct rw_semaphore *sem) {return false; } - -static inline bool rwsem_has_spinner (struct rw_semaphore *sem) -{ -return false; -}#endif /*
@@ -662,67 +648,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem, long count)unsigned long flags;DEFINE_WAKE_Q (wake_q); -/* -* __rwsem_down_write_failed_common(sem) -* rwsem_optimistic_spin(sem) -* osq_unlock(sem->osq) -* ... -* atomic_long_add_return(&sem->count) -* -* - VS - -* -* __up_write() -* if (atomic_long_sub_return_release(&sem->count) < 0) -* rwsem_wake(sem) -* osq_is_locked(&sem->osq) -* -* And __up_write() must observe !osq_is_locked() when it observes the -* atomic_long_add_return() in order to not miss a wakeup. -* -* This boils down to: -* -* [S.rel] X = 1 [RmW] r0 = (Y += 0) -* MB RMB -* [RmW] Y += 1 [L] r1 = X -* -* exists (r0=1 /\ r1=0) -*/ -smp_rmb (); - -/* -* If a spinner is present and the handoff flag isn't set, it is -* not necessary to do the wakeup. -* -* Try to do wakeup only if the trylock succeeds to minimize -* spinlock contention which may introduce too much delay in the -* unlock operation. -* -* spinning writer up_write/up_read caller -* --------------- ----------------------- -* [S] osq_unlock() [L] osq -* MB RMB -* [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock) -* -* Here, it is important to make sure that there won't be a missed -* wakeup while the rwsem is free and the only spinning writer goes -* to sleep without taking the rwsem. Even when the spinning writer -* is just going to break out of the waiting loop, it will still do -* a trylock in rwsem_down_write_failed() before sleeping. IOW, if -* rwsem_has_spinner() is true, it will guarantee at least one -* trylock attempt on the rwsem later on. -*/ -if (rwsem_has_spinner (sem) && !RWSEM_COUNT_HANDOFF (count)) { -/* -* The smp_rmb() here is to make sure that the spinner -* state is consulted before reading the wait_lock. -*/ -smp_rmb (); -if (!raw_spin_trylock_irqsave (&sem->wait_lock, flags)) -return sem; - goto locked; - }raw_spin_lock_irqsave (&sem->wait_lock, flags); -locked:if (!list_empty (&sem->wait_list))__rwsem_mark_wake (sem,RWSEM_WAKE_ANY , &wake_q);
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 14/22 locking/rwsem: Add more rwsem owner access helpers
In reply to Waiman LongBefore combining owner and count, we are adding two new helpers for accessing the owner value in the rwsem.
1) struct task_struct *rwsem_get_owner(struct rw_semaphore *sem)
2) bool is_rwsem_reader_owned(struct rw_semaphore *sem)
--- kernel/locking/rwsem-xadd.c | 11 ++++++----- kernel/locking/rwsem-xadd.h | 32 ++++++++++++++++++++++++++------ kernel/locking/rwsem.c | 3 +-- 3 files changed, 33 insertions(+), 13 deletions(-)
kernel/locking/rwsem-xadd.c +6 -5
@@ -277,7 +277,7 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)return false;rcu_read_lock (); - owner =READ_ONCE (sem->owner); + owner =rwsem_get_owner (sem);if (owner) { ret =is_rwsem_owner_spinnable (owner) &&owner_on_cpu (owner);
@@ -291,13 +291,13 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)*/ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem) { -struct task_struct *owner =READ_ONCE (sem->owner); +struct task_struct *owner =rwsem_get_owner (sem);if (!is_rwsem_owner_spinnable (owner))return false;rcu_read_lock (); -while (owner && (READ_ONCE (sem->owner) == owner)) { +while (owner && (rwsem_get_owner (sem) == owner)) {/* * Ensure we emit the owner->on_cpu, dereference _after_ * checking sem->owner still matches owner, if that fails,
@@ -323,7 +323,7 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)* If there is a new owner or the owner is not set, we continue * spinning. */ -return is_rwsem_owner_spinnable (READ_ONCE (sem->owner)); +return is_rwsem_owner_spinnable (rwsem_get_owner (sem)); }static bool rwsem_optimistic_spin (struct rw_semaphore *sem)
@@ -361,7 +361,8 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)* we're an RT task that will live-lock because we won't let * the owner complete. */ -if (!sem->owner && (need_resched () ||rt_task (current))) +if (!rwsem_get_owner (sem) && + (need_resched () ||rt_task (current)))break ;/*
kernel/locking/rwsem-xadd.h +26 -6
@@ -83,6 +83,11 @@ static inline void rwsem_clear_owner(struct rw_semaphore *sem)WRITE_ONCE (sem->owner,NULL ); } +static inline struct task_struct *rwsem_get_owner (struct rw_semaphore *sem) +{ +return READ_ONCE (sem->owner); +} +/* * The task_struct pointer of the last owning reader will be left in * the owner field.
@@ -116,6 +121,23 @@ static inline bool is_rwsem_owner_spinnable(struct task_struct *owner) }/* +* Return true if the rwsem is owned by a reader. +*/ +static inline bool is_rwsem_reader_owned (struct rw_semaphore *sem) +{ +#ifdef CONFIG_DEBUG_RWSEMS +/* +* Check the count to see if it is write-locked. +*/ +long count =atomic_long_read (&sem->count); + +if (count &RWSEM_WRITER_MASK ) +return false; +#endif +return (unsigned long )sem->owner &RWSEM_READER_OWNED ; +} + +/* * Return true if rwsem is owned by an anonymous writer or readers. */ static inline bool rwsem_has_anonymous_owner (struct task_struct *owner)
@@ -135,6 +157,7 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) {unsigned long val = (unsigned long )current |RWSEM_READER_OWNED |RWSEM_ANONYMOUSLY_OWNED ; +if (READ_ONCE (sem->owner) == (struct task_struct *)val)cmpxchg_relaxed ((unsigned long *)&sem->owner, val,RWSEM_READER_OWNED |RWSEM_ANONYMOUSLY_OWNED );
@@ -181,8 +204,7 @@ static inline void __down_read(struct rw_semaphore *sem)if (unlikely (atomic_long_fetch_add_acquire (RWSEM_READER_BIAS , &sem->count) &RWSEM_READ_FAILED_MASK )) {rwsem_down_read_failed (sem); -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner & -RWSEM_READER_OWNED ), sem); +DEBUG_RWSEMS_WARN_ON (!is_rwsem_reader_owned (sem), sem); }else {rwsem_set_reader_owned (sem); }
@@ -194,8 +216,7 @@ static inline int __down_read_killable(struct rw_semaphore *sem) &sem->count) & RWSEM_READ_FAILED_MASK)) { if (IS_ERR(rwsem_down_read_failed_killable(sem))) return -EINTR; - DEBUG_RWSEMS_WARN_ON(!((unsigned long)sem->owner & - RWSEM_READER_OWNED), sem); + DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem); } else {rwsem_set_reader_owned (sem); }
@@ -254,8 +275,7 @@ static inline void __up_read(struct rw_semaphore *sem) {long tmp; -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ), - sem); +DEBUG_RWSEMS_WARN_ON (!is_rwsem_reader_owned (sem), sem);rwsem_clear_reader_owned (sem); tmp =atomic_long_add_return_release (-RWSEM_READER_BIAS , &sem->count); if (unlikely((tmp & (RWSEM_LOCK_MASK|RWSEM_FLAG_WAITERS))
kernel/locking/rwsem.c +1 -2
@@ -203,8 +203,7 @@ int __sched down_write_killable_nested(struct rw_semaphore *sem, int subclass)void up_read_non_owner (struct rw_semaphore *sem) { -DEBUG_RWSEMS_WARN_ON (!((unsigned long )sem->owner &RWSEM_READER_OWNED ), - sem); +DEBUG_RWSEMS_WARN_ON (!is_rwsem_reader_owned (sem), sem);__up_read (sem); }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 15/22 locking/rwsem: Merge owner into count on x86-64
In reply to Waiman LongWith separate count and owner, there are timing windows where the two values are inconsistent. That can cause problem when trying to figure out the exact state of the rwsem. For instance, a RT task will stop optimistic spinning if the lock is acquired by a writer but the owner field isn't set yet. That can be solved by combining the count and owner together in a single atomic value.
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
The owner value will still be duplicated in the owner field for the purpose of signalling that the task is in the process of acquiring or releasing a rwsem.
This change is currently for x86-64 only. Other 64-bit architectures may be enabled in the future if the need arises.
With a locking microbenchmark running on 5.0 based kernel, the total locking rates (in kops/s) of the benchmark on a 4-socket 56-core x86-64 system before and after the patch were as follows:
Before Patch After Patch
# of Threads wlock rlock wlock rlock
------------ ----- ----- ----- -----
1 29,085 30,179 27,892 29,514
2 7,341 14,084 6,240 14,304
4 7,393 14,246 5,216 11,754
8 7,139 13,860 5,400 11,308
16 6,650 15,773 5,744 15,405This change does have an impact on both read and write lock performance.
--- kernel/locking/rwsem-xadd.c | 20 +++++++-- kernel/locking/rwsem-xadd.h | 105 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 110 insertions(+), 15 deletions(-)
kernel/locking/rwsem-xadd.c +16 -4
@@ -27,11 +27,11 @@/* * Guide to the rw_semaphore's count field. * -* When the RWSEM_WRITER_LOCKED bit in count is set, the lock is owned -* by a writer. +* When any of the RWSEM_WRITER_MASK bits in count is set, the lock is +* owned by a writer. * * The lock is owned by readers when -* (1) the RWSEM_WRITER_LOCKED isn't set in count, +* (1) none of the RWSEM_WRITER_MASK bits is set in count, * (2) some of the reader bits are set in count, and * (3) the owner field has RWSEM_READ_OWNED bit set. *
@@ -47,6 +47,11 @@void __init_rwsem (struct rw_semaphore *sem,const char *name,struct lock_class_key *key) { +/* +* We should support at least (4k-1) concurrent readers +*/ +BUILD_BUG_ON (sizeof (long ) *8 -RWSEM_READER_SHIFT <12 ); +#ifdef CONFIG_DEBUG_LOCK_ALLOC /* * Make sure we are not reinitializing a held semaphore:
@@ -297,7 +302,14 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)return false;rcu_read_lock (); -while (owner && (rwsem_get_owner (sem) == owner)) { +/* +* In case the owner task pointer is also stored in the count, +* checking the sem->owner value alone will give an early indication +* if the owner is about to release the lock (sem->owner cleared). +* This enables the spinner to move forward and do a trylock +* earlier. +*/ +while (owner && (READ_ONCE (sem->owner) == owner)) {/* * Ensure we emit the owner->on_cpu, dereference _after_ * checking sem->owner still matches owner, if that fails,
kernel/locking/rwsem-xadd.h +94 -11
150 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerRe: [PATCH-tip 15/22] locking/rwsem: Merge owner into count on x86-64
Quoted 8 lines · Waiman Long
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
*groan*...
So take qrwlock's idea for a queue, then make the count value (similar to the new mutex); that is have a bit0 be a r/w bit, when w bits 6-N are owner, when r they are reader-count. bit1 can be a pending bit, bit2 a handoff bit etc..
That should fit and work on 32bit and 64bit without issue.
I have a half-arsed rwsem-atomic.c somewhere that does just that. I just never got around to doing all the optimistic spin and steal crap that makes our current rwsem fly.
And that nicely gets rid of that mind bending BIAS crud.
Waiman Long
reviewerRe: [PATCH-tip 15/22] locking/rwsem: Merge owner into count on x86-64
Quoted 16 lines · Peter Zijlstra
Quoted 8 lines · Waiman Long
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
*groan*...
So take qrwlock's idea for a queue, then make the count value (similar to the new mutex); that is have a bit0 be a r/w bit, when w bits 6-N are owner, when r they are reader-count. bit1 can be a pending bit, bit2 a handoff bit etc..
Quoted 7 lines
That should fit and work on 32bit and 64bit without issue.
I have a half-arsed
rwsem-atomic.csomewhere that does just that. I just never got around to doing all the optimistic spin and steal crap that makes our current rwsem fly.And that nicely gets rid of that mind bending BIAS crud.
Well, the reason for this compromise is to keep using xadd for readers. Your scheme will certainly work, but we have to use cmpxchg for readers too. That will have a performance impact especially with multiple readers contending which I am trying to avoid.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH-tip 15/22] locking/rwsem: Merge owner into count on x86-64
In reply to Waiman LongQuoted 8 lines · Waiman Long
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
64k readers sounds like a number that is fairly 'easy' to reach, esp. on 64bit. These are preemptible locks after all, all we need to do is get 64k tasks nested on enough CPUs.
I'm sure there's some willing Java proglet around that spawns more than 64k threads just because it can. Run it on a big enough machine (ISTR there's a number of >1k CPU systems out there) and voila.
Waiman Long
reviewerRe: [PATCH-tip 15/22] locking/rwsem: Merge owner into count on x86-64
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Waiman Long
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
Quoted 7 lines
64k readers sounds like a number that is fairly 'easy' to reach, esp. on 64bit. These are preemptible locks after all, all we need to do is get 64k tasks nested on enough CPUs.
I'm sure there's some willing Java proglet around that spawns more than 64k threads just because it can. Run it on a big enough machine (ISTR there's a number of >1k CPU systems out there) and voila.
Yes, that can be a problem.
One possible solution is to check if the count goes negative. If so, fail the read lock and make the readers wait in the wait queue until the count is in positive territory. That effectively reduces the reader count to 15 bits, but it will avoid the overflow situation. I will try to add that support into the next version.
Cheers,
Longman
Waiman Long
reviewerRe: [PATCH-tip 15/22] locking/rwsem: Merge owner into count on x86-64
Quoted 19 lines · Waiman Long
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Waiman Long
On 32-bit architectures, there aren't enough bits to hold both. 64-bit architectures, however, can have enough bits to do that. For x86-64, the physical address can use up to 52 bits. That is 4PB of memory. That leaves 12 bits available for other use. The task structure pointer is also aligned to the L1 cache size. That means another 6 bits (64 bytes cacheline) will be available. Reserving 2 bits for status flags, we will have 16 bits for the reader count. That can supports up to (64k-1) readers.
Quoted 7 lines
64k readers sounds like a number that is fairly 'easy' to reach, esp. on 64bit. These are preemptible locks after all, all we need to do is get 64k tasks nested on enough CPUs.
I'm sure there's some willing Java proglet around that spawns more than 64k threads just because it can. Run it on a big enough machine (ISTR there's a number of >1k CPU systems out there) and voila.
Yes, that can be a problem.
Quoted 8 lines
One possible solution is to check if the count goes negative. If so, fail the read lock and make the readers wait in the wait queue until the count is in positive territory. That effectively reduces the reader count to 15 bits, but it will avoid the overflow situation. I will try to add that support into the next version.
Cheers,
Longman
Something like the attached patch.
Cheers,
Longman
From 746913e7d14e874eeace1e146e63bdaea4dfd4a5 Mon Sep 17 00:00:00 2001
From: Waiman Long <[email protected]>
Date: Fri, 8 Feb 2019 08:58:10 -0500
Subject: [PATCH 23/23] locking/rwsem: Make MSbit of count as guard bit to fail
readlock
With the merging of owner into count for x86-64, there is only 16 bits left for reader count. It is theoretically possible for an application to cause more than 64k readers to acquire a rwsem leading to count overflow.
To prevent this dire situation, the most significant bit of the count is now treated as a guard bit (RWSEM_FLAG_READFAIL). Read-lock will now fails for both the fast and optimistic spinning paths whenever this bit is set. So all those extra readers will be put to sleep in the wait queue. Wakeup will not happen until the reader count reaches 0.
A limit of 256 is also imposed on the number of readers that can be woken up in one wakeup function call. This will eliminate the possibility of waking up more than 64k readers and overflowing the count.
--- kernel/locking/lock_events_list.h | 1 + kernel/locking/rwsem-xadd.c | 40 ++++++++++++++++++++++++++++++++------ kernel/locking/rwsem-xadd.h | 41 ++++++++++++++++++++++++++------------- 3 files changed, 62 insertions(+), 20 deletions(-)
kernel/locking/lock_events_list.h +1 -0
@@ -60,6 +60,7 @@ LOCK_EVENT(rwsem_opt_rlock) /* # of read locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_wlock) /* # of write locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_fail) /* # of failed opt-spinnings */ +LOCK_EVENT(rwsem_opt_rfail) /* # of failed reader-owned readlocks */ LOCK_EVENT(rwsem_opt_nospin) /* # of disabled reader opt-spinnings */ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */
kernel/locking/rwsem-xadd.c +34 -6
@@ -110,6 +110,8 @@ enum rwsem_wake_type {# define RWSEM_RSPIN_MAX (1 << 12)#endif +#define MAX_READERS_WAKEUP 0x100 +/* * handle the lock release when processes blocked on it that can now run * - if we come here from up_xxxx(), then the RWSEM_FLAG_WAITERS bit must
@@ -208,6 +210,12 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,* after setting the reader waiter to nil. */ wake_q_add_safe (wake_q, tsk); + +/* +* Limit # of readers that can be woken up per wakeup call. +*/ +if (woken >=MAX_READERS_WAKEUP ) +break ; } adjustment = woken *RWSEM_READER_BIAS - adjustment;
@@ -445,6 +453,16 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem, const long wlock)break ;/* +* If a reader cannot acquire a reader-owned lock, we +* have to quit. It is either the handoff bit just got +* set or (unlikely) readfail bit is somehow set. +*/ +if (unlikely (!wlock && (owner_state ==OWNER_READER ))) { +lockevent_inc (rwsem_opt_rfail); +break ; + } + +/* * An RT task cannot do optimistic spinning if it cannot * be sure the lock holder is running. When there's no owner * or is reader-owned, an RT task has to stop spinning or
@@ -526,12 +544,22 @@ static inline bool rwsem_optimistic_spin(struct rw_semaphore *sem,* Wait for the read lock to be granted */ static inline struct rw_semaphore __sched * -__rwsem_down_read_failed_common (struct rw_semaphore *sem,int state) +__rwsem_down_read_failed_common (struct rw_semaphore *sem,int state,long count) { -long count, adjustment = -RWSEM_READER_BIAS ; +long adjustment = -RWSEM_READER_BIAS ;struct rwsem_waiter waiter;DEFINE_WAKE_Q (wake_q); +if (unlikely (count <0 )) { +/* +* Too many active readers, decrement count & +* enter the wait queue. +*/ +atomic_long_add (-RWSEM_READER_BIAS , &sem->count); + adjustment =0 ; + goto queue; + } +if (!rwsem_can_spin_on_owner (sem)) goto queue;
@@ -635,16 +663,16 @@ static inline bool rwsem_optimistic_spin(struct rw_semaphore *sem, } __visible struct rw_semaphore * __sched -rwsem_down_read_failed(struct rw_semaphore *sem) +rwsem_down_read_failed(struct rw_semaphore *sem, long cnt) { -return __rwsem_down_read_failed_common (sem,TASK_UNINTERRUPTIBLE ); +return __rwsem_down_read_failed_common (sem,TASK_UNINTERRUPTIBLE , cnt); }EXPORT_SYMBOL (rwsem_down_read_failed); __visible struct rw_semaphore * __sched -rwsem_down_read_failed_killable(struct rw_semaphore *sem) +rwsem_down_read_failed_killable(struct rw_semaphore *sem, long cnt) { -return __rwsem_down_read_failed_common (sem,TASK_KILLABLE ); +return __rwsem_down_read_failed_common (sem,TASK_KILLABLE , cnt); }EXPORT_SYMBOL (rwsem_down_read_failed_killable);
kernel/locking/rwsem-xadd.h +27 -14
89 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 16/22 locking/rwsem: Remove redundant computation of writer lock word
In reply to Waiman LongOn 64-bit architectures, each rwsem writer will have its unique lock word for acquiring the lock. Right now, the writer code recomputes the lock word every time it tries to acquire the lock. This is a waste of time. The lock word is now cached and reused when it is needed.
On 32-bit architectures, the extra constant argument to rwsem_try_write_lock() and rwsem_try_write_lock_unqueued() should be optimized out by the compiler.
--- kernel/locking/rwsem-xadd.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
kernel/locking/rwsem-xadd.c +12 -10
@@ -216,8 +216,8 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,* race conditions between checking the rwsem wait list and setting the * sem->count accordingly. */ -static inline bool -rwsem_try_write_lock (long count,struct rw_semaphore *sem,bool first) +static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem, +const long wlock,bool first) {long new;
@@ -227,7 +227,7 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,if (!first &&RWSEM_COUNT_HANDOFF (count))return false; - new = (count & ~RWSEM_FLAG_HANDOFF ) +RWSEM_WRITER_LOCKED - + new = (count & ~RWSEM_FLAG_HANDOFF ) + wlock - (list_is_singular (&sem->wait_list) ?RWSEM_FLAG_WAITERS :0 );if (atomic_long_cmpxchg_acquire (&sem->count, count, new) == count) {
@@ -242,7 +242,8 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,/* * Try to acquire write lock before the writer has been put on wait queue. */ -static inline bool rwsem_try_write_lock_unqueued (struct rw_semaphore *sem) +static inline bool rwsem_try_write_lock_unqueued (struct rw_semaphore *sem, +const long wlock) {long old, count =atomic_long_read (&sem->count);
@@ -251,7 +252,7 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)return false; old =atomic_long_cmpxchg_acquire (&sem->count, count, - count +RWSEM_WRITER_LOCKED ); + count + wlock);if (old == count) {rwsem_set_owner (sem);lockevent_inc (rwsem_opt_wlock);
@@ -338,7 +339,7 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)return is_rwsem_owner_spinnable (rwsem_get_owner (sem)); } -static bool rwsem_optimistic_spin (struct rw_semaphore *sem) +static bool rwsem_optimistic_spin (struct rw_semaphore *sem,const long wlock) {bool taken = false;
@@ -362,7 +363,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)/* * Try to acquire the lock */ -if (rwsem_try_write_lock_unqueued (sem)) { +if (rwsem_try_write_lock_unqueued (sem, wlock)) { taken = true;break ; }
@@ -392,7 +393,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem) return taken; }#else -static bool rwsem_optimistic_spin (struct rw_semaphore *sem) +static bool rwsem_optimistic_spin (struct rw_semaphore *sem,const long wlock) {return false; }
@@ -514,9 +515,10 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)struct rwsem_waiter waiter;struct rw_semaphore *ret = sem;DEFINE_WAKE_Q (wake_q); +const long wlock =RWSEM_WRITER_LOCKED ;/* do optimistic spinning and steal lock if possible */ -if (rwsem_optimistic_spin (sem)) +if (rwsem_optimistic_spin (sem, wlock))return sem;/*
@@ -569,7 +571,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem)/* wait until we successfully acquire the lock */ set_current_state (state);while (true) { -if (rwsem_try_write_lock (count, sem, first)) +if (rwsem_try_write_lock (count, sem, wlock, first))break ;raw_spin_unlock_irq (&sem->wait_lock);
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 17/22 locking/rwsem: Recheck owner if it is not on cpu
In reply to Waiman LongAfter merging the owner value directly into the count field, it was found that the number of failed optimistic spinning operations increased significantly during the boot up process.
The cause of this increased failures was tracked down to the condition that a lock holder might have just released the lock and gone to sleep right after its owner value was fetched by a spinner. So the task might have slept, but it was no longer the lock holder. The merging of owner into count increases the chance this condition can happen.
To close this failure mode, we are now rechecking the owner value again to see if it has been changed in case it is not on cpu.
On a 1-socket x86-64 system, the lock event counts before the patch were:
rwsem_opt_fail=5847
rwsem_opt_wlock=7880
rwsem_wlock=5847
After the patch, the counts were:
rwsem_opt_fail=225
rwsem_opt_wlock=8541
rwsem_wlock=225
--- kernel/locking/rwsem-xadd.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
kernel/locking/rwsem-xadd.c +16 -4
@@ -263,13 +263,25 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem, } } -static inline bool owner_on_cpu (struct task_struct *owner) +static inline bool owner_on_cpu (struct task_struct *owner, +struct rw_semaphore *sem) {/* * As lock holder preemption issue, we both skip spinning if * task is not on cpu or its cpu is preempted */ -return owner->on_cpu && !vcpu_is_preempted (task_cpu (owner)); +bool oncpu = owner->on_cpu && !vcpu_is_preempted (task_cpu (owner)); + +/* +* There is a slight chance that the lock holder might have +* just release the rwsem and gone to sleep right after we +* fetched the owner value. So we double-check the sem->owner +* field again to see if it has been changed. The sem->owner +* would have been cleared right before the lock was released. +*/ +if (!oncpu && (READ_ONCE (sem->owner) != owner)) +return true;/* Assume the new owner is on cpu */ +return oncpu; }static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem)
@@ -286,7 +298,7 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem) owner =rwsem_get_owner (sem);if (owner) { ret =is_rwsem_owner_spinnable (owner) && -owner_on_cpu (owner); +owner_on_cpu (owner, sem); }rcu_read_unlock ();return ret;
@@ -323,7 +335,7 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)* abort spinning when need_resched or owner is not running or * owner's cpu is preempted. */ -if (need_resched () || !owner_on_cpu (owner)) { +if (need_resched () || !owner_on_cpu (owner, sem)) {rcu_read_unlock ();return false; }
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 18/22 locking/rwsem: Make rwsem_spin_on_owner() return a tri-state value
In reply to Waiman LongThis patch modifies rwsem_spin_on_owner() to return a tri-state value to better reflect the state of lock holder which enables us to make a better decision of what to do next.
--- kernel/locking/rwsem-xadd.c | 25 +++++++++++++++++++------ kernel/locking/rwsem-xadd.h | 5 +++++ 2 files changed, 24 insertions(+), 6 deletions(-)
kernel/locking/rwsem-xadd.c +19 -6
@@ -305,14 +305,24 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem) }/* -* Return true only if we can still spin on the owner field of the rwsem. +* Return the folowing three values depending on the lock owner state. +* 1 when owner changes and no reader is detected yet. +* 0 when owner changes and the new owner may be a reader. +* -1 when optimistic spinning has to stop because either the owner stops +* running, is unknown, or its timeslice has been used up. */ -static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem) +enum owner_state { +OWNER_SPINNABLE =1 , +OWNER_READER =0 , +OWNER_NONSPINNABLE =-1 , +}; + +static noinline enum owner_state rwsem_spin_on_owner(struct rw_semaphore *sem) {struct task_struct *owner =rwsem_get_owner (sem);if (!is_rwsem_owner_spinnable (owner)) -return false; +return OWNER_NONSPINNABLE ;rcu_read_lock ();/*
@@ -337,7 +347,7 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)*/ if (need_resched () || !owner_on_cpu (owner, sem)) {rcu_read_unlock (); -return false; +return OWNER_NONSPINNABLE ; }cpu_relax ();
@@ -348,7 +358,10 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem)* If there is a new owner or the owner is not set, we continue * spinning. */ -return is_rwsem_owner_spinnable (rwsem_get_owner (sem)); + owner =rwsem_get_owner (sem); +if (!is_rwsem_owner_spinnable (owner)) +return OWNER_NONSPINNABLE ; +return is_rwsem_owner_reader (owner) ?OWNER_READER :OWNER_SPINNABLE ; }static bool rwsem_optimistic_spin (struct rw_semaphore *sem,const long wlock)
@@ -371,7 +384,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem, const long wlock)* 2) readers own the lock as we can't determine if they are * actively running or not. */ -while (rwsem_spin_on_owner (sem)) { +while (rwsem_spin_on_owner (sem) ==OWNER_SPINNABLE ) {/* * Try to acquire the lock */
kernel/locking/rwsem-xadd.h +5 -0
@@ -200,6 +200,11 @@ static inline bool is_rwsem_owner_spinnable(struct task_struct *owner)return !((unsigned long )owner &RWSEM_ANONYMOUSLY_OWNED ); } +static inline bool is_rwsem_owner_reader (struct task_struct *owner) +{ +return (unsigned long )owner &RWSEM_READER_OWNED ; +} +/* * Return true if the rwsem is owned by a reader. */
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 19/22 locking/rwsem: Enable readers spinning on writer
In reply to Waiman LongThis patch enables readers to optimistically spin on a
rwsem when it is owned by a writer instead of going to sleep directly. The rwsem_can_spin_on_owner() function is extracted out of rwsem_optimistic_spin() and is called directly by __rwsem_down_read_failed_common() and __rwsem_down_write_failed_common().
This patch may actually reduce performance under certain circumstances for reader-mostly workload as the readers may not be grouped together in the wait queue anymore. So we may have a number of small reader groups among writers instead of a large reader group. However, this change is needed for some of the subsequent patches.
With a locking microbenchmark running on 5.0 based kernel, the total locking rates (in kops/s) of the benchmark on a 4-socket 56-core x86-64 system with equal numbers of readers and writers before and after the patch were as follows:
# of Threads Pre-patch Post-patch
------------ --------- ----------
2 1,926 2,120
4 1,391 1,320
8 716 694
16 618 606
32 501 487
64 61 57--- kernel/locking/lock_events_list.h | 1 + kernel/locking/rwsem-xadd.c | 80 ++++++++++++++++++++++++++++++++++----- kernel/locking/rwsem-xadd.h | 3 ++ 3 files changed, 74 insertions(+), 10 deletions(-)
kernel/locking/lock_events_list.h +1 -0
@@ -57,6 +57,7 @@ LOCK_EVENT(rwsem_sleep_writer) /* # of writer sleeps */ LOCK_EVENT(rwsem_wake_reader) /* # of reader wakeups */ LOCK_EVENT(rwsem_wake_writer) /* # of writer wakeups */ +LOCK_EVENT(rwsem_opt_rlock) /* # of read locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_wlock) /* # of write locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_fail) /* # of failed opt-spinnings */ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */
kernel/locking/rwsem-xadd.c +70 -10
136 lines not on this page · raw message
kernel/locking/rwsem-xadd.h +3 -0
@@ -109,9 +109,12 @@ RWSEM_FLAG_HANDOFF)#define RWSEM_COUNT_LOCKED (c) ((c) & RWSEM_LOCK_MASK) +#define RWSEM_COUNT_WLOCKED (c) ((c) & RWSEM_WRITER_MASK)#define RWSEM_COUNT_HANDOFF (c) ((c) & RWSEM_FLAG_HANDOFF)#define RWSEM_COUNT_LOCKED_OR_HANDOFF (c) \ ((c) & (RWSEM_LOCK_MASK|RWSEM_FLAG_HANDOFF)) +#define RWSEM_COUNT_WLOCKED_OR_HANDOFF (c) \ + ((c) & (RWSEM_WRITER_MASK | RWSEM_FLAG_HANDOFF))/* * Task structure pointer compression (64-bit only):
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 20/22 locking/rwsem: Enable count-based spinning on reader
In reply to Waiman LongWhen the rwsem is owned by reader, writers stop optimistic spinning simply because there is no easy way to figure out if all the readers are actively running or not. However, there are scenarios where the readers are unlikely to sleep and optimistic spinning can help performance.
This patch provides a simple mechanism for spinning on a reader-owned rwsem. It is a loop count threshold based spinning where the count will get reset whenenver the rwsem reader count value changes indicating that the rwsem is still active. There is another maximum count value that limits that maximum number of spinnings that can happen.
When the loop or max counts reach 0, a bit will be set in the owner field to indicate that no more optimistic spinning should be done on this rwsem until it becomes writer owned again. Not even readers is allowed to acquire the reader-locked rwsem for better fairness.
The spinning threshold and maximum values can be overridden by architecture specific header file, if necessary. The current default threshold value is 512 iterations.
With a locking microbenchmark running on 5.0 based kernel, the total locking rates (in kops/s) of the benchmark on a 4-socket 56-core x86-64 system with equal numbers of readers and writers before all the reader spining patches, before this patch and after this patch were as follows:
# of Threads Pre-rspin Pre-Patch Post-patch
------------ --------- --------- ----------
2 1,926 2,120 8,057
4 1,391 1,320 7,680
8 716 694 7,284
16 618 606 6,542
32 501 487 1,449
64 61 57 480This patch gives a big boost in performance for mixed reader/writer workloads.
--- kernel/locking/lock_events_list.h | 1 + kernel/locking/rwsem-xadd.c | 63 +++++++++++++++++++++++++++++++++++---- kernel/locking/rwsem-xadd.h | 45 +++++++++++++++++++++------- 3 files changed, 94 insertions(+), 15 deletions(-)
kernel/locking/lock_events_list.h +1 -0
@@ -60,6 +60,7 @@ LOCK_EVENT(rwsem_opt_rlock) /* # of read locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_wlock) /* # of write locks opt-spin acquired */ LOCK_EVENT(rwsem_opt_fail) /* # of failed opt-spinnings */ +LOCK_EVENT(rwsem_opt_nospin) /* # of disabled reader opt-spinnings */ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */
kernel/locking/rwsem-xadd.c +58 -5
104 lines not on this page · raw message
kernel/locking/rwsem-xadd.h +35 -10
73 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 21/22 locking/rwsem: Wake up all readers in wait queue
In reply to Waiman LongWhen the front of the wait queue is a reader, other readers immediately following the first reader will also be woken up at the same time. However, if there is a writer in between. Those readers behind the writer will not be woken up.
Because of optimistic spinning, the lock acquisition order is not FIFO anyway. The lock handoff mechanism will ensure that lock starvation will not happen.
Assuming that the lock hold times of the other readers still in the queue will be about the same as the readers that are being woken up, there is really not much additional cost other than the additional latency due to the wakeup of additional tasks by the waker. Therefore all the readers in the queue are woken up when the first waiter is a reader to improve reader throughput.
With a locking microbenchmark running on 5.0 based kernel, the total locking rates (in kops/s) of the benchmark on a 4-socket 56-core x86-64 system with equal numbers of readers and writers before all the reader spining patches, before this patch and after this patch were as follows:
# of Threads Pre-rspin Pre-Patch Post-patch
------------ --------- --------- ----------
2 1,926 8,057 7,397
4 1,391 7,680 6,161
8 716 7,284 6,405
16 618 6,542 6,768
32 501 1,449 6,550
64 61 480 5,548
112 75 769 5,216At low contention level, there is a slight drop in performance. At high contention level, however, this patch gives a big performance boost.
--- kernel/locking/rwsem-xadd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/locking/rwsem-xadd.c +4 -4
@@ -180,16 +180,16 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, }/* -* Grant an infinite number of read locks to the readers at the front -* of the queue. We know that woken will be at least 1 as we accounted -* for above. Note we increment the 'active part' of the count by the +* Grant an infinite number of read locks to all the readers in the +* queue. We know that woken will be at least 1 as we accounted for +* above. Note we increment the 'active part' of the count by the * number of readers before waking any processes up. */ list_for_each_entry_safe (waiter, tmp, &sem->wait_list, list) {struct task_struct *tsk;if (waiter->type ==RWSEM_WAITING_FOR_WRITE ) -break ; +continue ; woken++; tsk = waiter->task;
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH 22/22 locking/rwsem: Ensure an RT task will not spin on reader
In reply to Waiman LongAn RT task can do optimistic spinning only if the lock holder is actually running. If the state of the lock holder isn't known, there is a possibility that high priority of the RT task may block forward progress of the lock holder if they happen to reside on the same CPU. This will lead to deadlock. So we have to make sure that an RT task will not spin on a reader-owned rwsem.
--- kernel/locking/rwsem-xadd.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
kernel/locking/rwsem-xadd.c +10 -6
@@ -349,12 +349,14 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)/* * Return the folowing three values depending on the lock owner state. +* 2 when owner is currently NULL * 1 when owner changes and no reader is detected yet. * 0 when owner changes and the new owner may be a reader. * -1 when optimistic spinning has to stop because either the owner stops * running, is unknown, or its timeslice has been used up. */ enum owner_state { +OWNER_NULL =2 ,OWNER_SPINNABLE =1 ,OWNER_READER =0 ,OWNER_NONSPINNABLE =-1 ,
@@ -405,7 +407,8 @@ static noinline enum owner_state rwsem_spin_on_owner(struct rw_semaphore *sem) owner =rwsem_get_owner (sem);if (!is_rwsem_owner_spinnable (owner))return OWNER_NONSPINNABLE ; -return is_rwsem_owner_reader (owner) ?OWNER_READER :OWNER_SPINNABLE ; +return !owner ?OWNER_NULL : +is_rwsem_owner_reader (owner) ?OWNER_READER :OWNER_SPINNABLE ; }static bool rwsem_optimistic_spin (struct rw_semaphore *sem,const long wlock)
@@ -442,12 +445,13 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem, const long wlock)break ;/* -* When there's no owner, we might have preempted between the -* owner acquiring the lock and setting the owner field. If -* we're an RT task that will live-lock because we won't let -* the owner complete. +* An RT task cannot do optimistic spinning if it cannot +* be sure the lock holder is running. When there's no owner +* or is reader-owned, an RT task has to stop spinning or +* live-lock may happen if the current task and the lock +* holder happen to run in the same CPU. */ -if (!rwsem_get_owner (sem) && +if ((owner_state !=OWNER_SPINNABLE ) && (need_resched () ||rt_task (current)))break ;
Signature · 1 line
1.8.3.1
Quoted 1 line · Waiman Long
30 files changed, 1197 insertions(+), 1594 deletions(-)
Performance numbers on numerous workloads, pretty please.
I'll go and throw this at my mmap_sem intensive workloads I've collected.
Thanks,
Davidlohr
Waiman Long
reviewerQuoted 10 lines · Davidlohr Bueso
Quoted 1 line · Waiman Long
30 files changed, 1197 insertions(+), 1594 deletions(-)Performance numbers on numerous workloads, pretty please.
I'll go and throw this at my
mmap_semintensive workloads I've collected.Thanks,
Davidlohr
Thanks for getting some of the performance numbers. This is the initial draft after more than 1 years of hibernation. I will also get other performance numbers in subsequent revision of the patch.
Cheers,
Longman
Quoted 9 lines · Waiman Long
Quoted 8 lines · Davidlohr Bueso
Quoted 1 line · Waiman Long
30 files changed, 1197 insertions(+), 1594 deletions(-)Performance numbers on numerous workloads, pretty please.
I'll go and throw this at my
mmap_semintensive workloads I've collected.
Quoted 6 lines
Quoted 2 lines
Thanks,
DavidlohrThanks for getting some of the performance numbers. This is the initial draft after more than 1 years of hibernation. I will also get other performance numbers in subsequent revision of the patch.
If you could sort all the invariant preparatory patches to the head of the series I can merge them to reduce overall complexity and simplify performance testing and review of the rest.
Thanks,
Ingo
Quoted 7 lines · Waiman Long
This patchset revamps the current rwsem-xadd implementation to make it saner and easier to work with. This patchset removes all the architecture specific assembly code and uses generic C code for all architectures. This eases maintenance and enables us to enhance the code more easily.
Quoted 5 lines
This patchset also implements the following 3 new features:
1) Waiter lock handoff
2) Reader optimistic spinning
3) Store write-lock owner in the atomic count (x86-64 only)
The patches are kind of hard to read, with most of them just doing prep-work that doesn't necessarily matter to the big picture.
What I'd really like to see is
(a) an overview of the new locking logic
(b) what's the new fastpath case
(c) some performance numbers
to explain the changes from a "this is the point of the whole exercise" standpoint.
And yes, I realize that the lock handoff and optimistic spinning is a big deal, since I've seen the same regression numbers that presumably caused this effort to be resurrected. So it's not that I don't find this intriguing and worthwhile, it's literally that I'd like a summary not so much of the individual patches, but of the new model.
Please?
Linus
Waiman Long
reviewerQuoted 12 lines · Linus Torvalds
Quoted 11 lines · Waiman Long
This patchset revamps the current rwsem-xadd implementation to make it saner and easier to work with. This patchset removes all the architecture specific assembly code and uses generic C code for all architectures. This eases maintenance and enables us to enhance the code more easily.
This patchset also implements the following 3 new features:
1) Waiter lock handoff
2) Reader optimistic spinning
3) Store write-lock owner in the atomic count (x86-64 only)
Quoted 6 lines
The patches are kind of hard to read, with most of them just doing prep-work that doesn't necessarily matter to the big picture.
What I'd really like to see is
(a) an overview of the new locking logic
The new locking logic is similar to qrwlock (see patch 11). Cmpxchg is used to acquire the write lock, while xadd is still used for read lock. Some of the bits in the count are also reserved for special purpose like has waiter or lock handoff. Patch 15 tries to compress the write-lock owner task pointer and put it into the count field for x86-64 at the expense of less bits available for reader count. I have sent out an additional patch this morning to make sure that the reader count won't overflow.
In term of performance, there isn't much change with respect to read-lock performance. For write-lock, I saw a slight drop in some cases, but nothing significant. The merging of owner task pointer into the count field does impose a slightly bigger drop than I would have liked which I am going to look into a bit more.
Quoted 2 lines
(b) what's the new fastpath case
The only change in the fastpath is the use of cmpxchg for writer lock.
Quoted 2 lines
(c) some performance numbers
There are performance data at patches 11, 12, 15, 19, 20, 21. There was performance data for patch 4 as well for eliminating the arch specific file. Apparently, I might have deleted it accidentally. Anyway, no noticeable performance difference was observed when switching to use generic C code for x86, ppc and ARM64.
The major gain in performance is due to reader optimistic spinning patches. The microbenchmark that I used shown an order of magnitude of performance improvement for mixed reader-writer workloads. Of course, we will see less performance gain with real world benchmarks.
I am planning to run more performance test and post the data sometimes next week. Davidlohr is also going to run some of his rwsem performance test on this patchset.
Quoted 11 lines
to explain the changes from a "this is the point of the whole exercise" standpoint.
And yes, I realize that the lock handoff and optimistic spinning is a big deal, since I've seen the same regression numbers that presumably caused this effort to be resurrected. So it's not that I don't find this intriguing and worthwhile, it's literally that I'd like a summary not so much of the individual patches, but of the new model.
Please?
Maybe I should break this patchset into a few smaller ones to make it easier to review. Any suggestion is welcome.
Cheers,
Longman
Quoted 4 lines · Waiman Long
Quoted 2 lines
(b) what's the new fastpath case
The only change in the fastpath is the use of cmpxchg for writer lock.
.. since a big deal here was about using the generic atomic accessor functions, I really was looking forward to seeing the *actual* fast path code generation.
In other words, right now I have very little visibility in how it actually affects the code. Looking at the patches themselves doesn't make it obvious. I was hoping for the overview to really explain the whole "before and after" situation, and it didn't. Not at the high level, and not at a low level. And no performance numbers in the overview either.
And yes, I see the numbers in the patches, but what I really hoped for was some real load numbers. In particular, I would have loved to see numbers from th ekernel test robot "will-it-scale.per_thread_ops" case, which is the one that had a 65% regression due to the lack of reader spinning.
So I was kind of hoping to hear whether that regression is basically entirely gone with this patch series, or if we still have a regression due to the extra downgrade, or what?
Linus
Quoted 3 lines · Waiman Long
I am planning to run more performance test and post the data sometimes next week. Davidlohr is also going to run some of his rwsem performance test on this patchset.
So I ran this series on a 40-core IB 2 socket with various worklods in mmtests. Below are some of the interesting ones; full numbers and curves at https://linux-scalability.org/rwsem-reader-spinner/
All workloads are with increasing number of threads.
-- pagefault timings: pft is an artificial pf benchmark (thus reader stress). metric is faults/cpu and faults/sec
v5.0-rc6 v5.0-rc6
dirty
Hmean faults/cpu-1 624224.9815 ( 0.00%) 618847.5201 * -0.86%*
Hmean faults/cpu-4 539550.3509 ( 0.00%) 547407.5738 * 1.46%*
Hmean faults/cpu-7 401470.3461 ( 0.00%) 381157.9830 * -5.06%*
Hmean faults/cpu-12 267617.0353 ( 0.00%) 271098.5441 * 1.30%*
Hmean faults/cpu-21 176194.4641 ( 0.00%) 175151.3256 * -0.59%*
Hmean faults/cpu-30 119927.3862 ( 0.00%) 120610.1348 * 0.57%*
Hmean faults/cpu-40 91203.6820 ( 0.00%) 91832.7489 * 0.69%*
Hmean faults/sec-1 623292.3467 ( 0.00%) 617992.0795 * -0.85%*
Hmean faults/sec-4 2113364.6898 ( 0.00%) 2140254.8238 * 1.27%*
Hmean faults/sec-7 2557378.4385 ( 0.00%) 2450945.7060 * -4.16%*
Hmean faults/sec-12 2696509.8975 ( 0.00%) 2747968.9819 * 1.91%*
Hmean faults/sec-21 2902892.5639 ( 0.00%) 2905923.3881 * 0.10%*
Hmean faults/sec-30 2956696.5793 ( 0.00%) 2990583.5147 * 1.15%*
Hmean faults/sec-40 3422806.4806 ( 0.00%) 3352970.3082 * -2.04%*
Stddev faults/cpu-1 2949.5159 ( 0.00%) 2802.2712 ( 4.99%)
Stddev faults/cpu-4 24165.9454 ( 0.00%) 15841.1232 ( 34.45%)
Stddev faults/cpu-7 20914.8351 ( 0.00%) 22744.3294 ( -8.75%)
Stddev faults/cpu-12 11274.3490 ( 0.00%) 14733.3152 ( -30.68%)
Stddev faults/cpu-21 2500.1950 ( 0.00%) 2200.9518 ( 11.97%)
Stddev faults/cpu-30 1599.5346 ( 0.00%) 1414.0339 ( 11.60%)
Stddev faults/cpu-40 1473.0181 ( 0.00%) 3004.1209 (-103.94%)
Stddev faults/sec-1 2655.2581 ( 0.00%) 2405.1625 ( 9.42%)
Stddev faults/sec-4 84042.7234 ( 0.00%) 57996.7158 ( 30.99%)
Stddev faults/sec-7 123656.7901 ( 0.00%) 135591.1087 ( -9.65%)
Stddev faults/sec-12 97135.6091 ( 0.00%) 127054.4926 ( -30.80%)
Stddev faults/sec-21 69564.6264 ( 0.00%) 65922.6381 ( 5.24%)
Stddev faults/sec-30 51524.4027 ( 0.00%) 56109.4159 ( -8.90%)
Stddev faults/sec-40 101927.5280 ( 0.00%) 160117.0093 ( -57.09%)With the exception of the hicup at 7 threads, things are pretty much in the noise region for both metrics.
-- git checkout
First metric is total runtime for runs with incremental threads.
v5.0-rc6 v5.0-rc6
dirty
User 218.95 219.07
System 149.29 146.82Elapsed 1574.10 1427.08
In this case there's a non trivial improvement (11%) in overall elapsed time.
-- reaim (which is always succeptible to rwsem changes for both mmap_sem and i_mmap)
v5.0-rc6 v5.0-rc6
dirty
Hmean compute-1 6674.01 ( 0.00%) 6544.28 * -1.94%*
Hmean compute-21 85294.91 ( 0.00%) 85524.20 * 0.27%*
Hmean compute-41 149674.70 ( 0.00%) 149494.58 * -0.12%*
Hmean compute-61 177721.15 ( 0.00%) 170507.38 * -4.06%*
Hmean compute-81 181531.07 ( 0.00%) 180463.24 * -0.59%*
Hmean compute-101 189024.09 ( 0.00%) 187288.86 * -0.92%*
Hmean compute-121 200673.24 ( 0.00%) 195327.65 * -2.66%*
Hmean compute-141 213082.29 ( 0.00%) 211290.80 * -0.84%*
Hmean compute-161 207764.06 ( 0.00%) 204626.68 * -1.51%*The 'compute' workload overall takes a small hit.
Hmean new_dbase-1 60.48 ( 0.00%) 60.63 * 0.25%* Hmean new_dbase-21 6590.49 ( 0.00%) 6671.81 * 1.23%* Hmean new_dbase-41 14202.91 ( 0.00%) 14470.59 * 1.88%* Hmean new_dbase-61 21207.24 ( 0.00%) 21067.40 * -0.66%* Hmean new_dbase-81 25542.40 ( 0.00%) 25542.40 * 0.00%* Hmean new_dbase-101 30165.28 ( 0.00%) 30046.21 * -0.39%* Hmean new_dbase-121 33638.33 ( 0.00%) 33219.90 * -1.24%* Hmean new_dbase-141 36723.70 ( 0.00%) 37504.52 * 2.13%* Hmean new_dbase-161 42242.51 ( 0.00%) 42117.34 * -0.30%* Hmean shared-1 76.54 ( 0.00%) 76.09 * -0.59%* Hmean shared-21 7535.51 ( 0.00%) 5518.75 * -26.76%* Hmean shared-41 17207.81 ( 0.00%) 14651.94 * -14.85%* Hmean shared-61 20716.98 ( 0.00%) 18667.52 * -9.89%* Hmean shared-81 27603.83 ( 0.00%) 23466.45 * -14.99%* Hmean shared-101 26008.59 ( 0.00%) 29536.96 * 13.57%* Hmean shared-121 28354.76 ( 0.00%) 43139.39 * 52.14%* Hmean shared-141 38509.25 ( 0.00%) 41619.35 * 8.08%* Hmean shared-161 40496.07 ( 0.00%) 44303.46 * 9.40%*
Overall there is a small hit (in the noise level but consistent throughout many workloads), except git-checkout which does quite well.
Thanks,
Davidlohr
Waiman Long
reviewerQuoted 97 lines · Davidlohr Bueso
Quoted 3 lines · Waiman Long
I am planning to run more performance test and post the data sometimes next week. Davidlohr is also going to run some of his rwsem performance test on this patchset.
So I ran this series on a 40-core IB 2 socket with various worklods in mmtests. Below are some of the interesting ones; full numbers and curves at https://linux-scalability.org/rwsem-reader-spinner/
All workloads are with increasing number of threads.
-- pagefault timings: pft is an artificial pf benchmark (thus reader stress).
metric is faults/cpu and faults/sec
v5.0-rc6 v5.0-rc6 dirty Hmean faults/cpu-1 624224.9815 ( 0.00%) 618847.5201 * -0.86%* Hmean faults/cpu-4 539550.3509 ( 0.00%) 547407.5738 * 1.46%* Hmean faults/cpu-7 401470.3461 ( 0.00%) 381157.9830 * -5.06%* Hmean faults/cpu-12 267617.0353 ( 0.00%) 271098.5441 * 1.30%* Hmean faults/cpu-21 176194.4641 ( 0.00%) 175151.3256 * -0.59%* Hmean faults/cpu-30 119927.3862 ( 0.00%) 120610.1348 * 0.57%* Hmean faults/cpu-40 91203.6820 ( 0.00%) 91832.7489 * 0.69%* Hmean faults/sec-1 623292.3467 ( 0.00%) 617992.0795 * -0.85%* Hmean faults/sec-4 2113364.6898 ( 0.00%) 2140254.8238 * 1.27%* Hmean faults/sec-7 2557378.4385 ( 0.00%) 2450945.7060 * -4.16%* Hmean faults/sec-12 2696509.8975 ( 0.00%) 2747968.9819 * 1.91%* Hmean faults/sec-21 2902892.5639 ( 0.00%) 2905923.3881 * 0.10%* Hmean faults/sec-30 2956696.5793 ( 0.00%) 2990583.5147 * 1.15%* Hmean faults/sec-40 3422806.4806 ( 0.00%) 3352970.3082 * -2.04%* Stddev faults/cpu-1 2949.5159 ( 0.00%) 2802.2712 ( 4.99%) Stddev faults/cpu-4 24165.9454 ( 0.00%) 15841.1232 ( 34.45%) Stddev faults/cpu-7 20914.8351 ( 0.00%) 22744.3294 ( -8.75%) Stddev faults/cpu-12 11274.3490 ( 0.00%) 14733.3152 ( -30.68%) Stddev faults/cpu-21 2500.1950 ( 0.00%) 2200.9518 ( 11.97%) Stddev faults/cpu-30 1599.5346 ( 0.00%) 1414.0339 ( 11.60%) Stddev faults/cpu-40 1473.0181 ( 0.00%) 3004.1209 (-103.94%) Stddev faults/sec-1 2655.2581 ( 0.00%) 2405.1625 ( 9.42%) Stddev faults/sec-4 84042.7234 ( 0.00%) 57996.7158 ( 30.99%) Stddev faults/sec-7 123656.7901 ( 0.00%) 135591.1087 ( -9.65%) Stddev faults/sec-12 97135.6091 ( 0.00%) 127054.4926 ( -30.80%) Stddev faults/sec-21 69564.6264 ( 0.00%) 65922.6381 ( 5.24%) Stddev faults/sec-30 51524.4027 ( 0.00%) 56109.4159 ( -8.90%) Stddev faults/sec-40 101927.5280 ( 0.00%) 160117.0093 ( -57.09%)With the exception of the hicup at 7 threads, things are pretty much in the noise region for both metrics.
-- git checkout
First metric is total runtime for runs with incremental threads.
v5.0-rc6 v5.0-rc6
dirty
User 218.95 219.07
System 149.29 146.82
Elapsed 1574.10 1427.08In this case there's a non trivial improvement (11%) in overall elapsed time.
-- reaim (which is always succeptible to rwsem changes for both
mmap_semandi_mmap)
v5.0-rc6 v5.0-rc6 dirty Hmean compute-1 6674.01 ( 0.00%) 6544.28 * -1.94%* Hmean compute-21 85294.91 ( 0.00%) 85524.20 * 0.27%* Hmean compute-41 149674.70 ( 0.00%) 149494.58 * -0.12%* Hmean compute-61 177721.15 ( 0.00%) 170507.38 * -4.06%* Hmean compute-81 181531.07 ( 0.00%) 180463.24 * -0.59%* Hmean compute-101 189024.09 ( 0.00%) 187288.86 * -0.92%* Hmean compute-121 200673.24 ( 0.00%) 195327.65 * -2.66%* Hmean compute-141 213082.29 ( 0.00%) 211290.80 * -0.84%* Hmean compute-161 207764.06 ( 0.00%) 204626.68 * -1.51%*The 'compute' workload overall takes a small hit.
Hmean new_dbase-1 60.48 ( 0.00%) 60.63 * 0.25%* Hmean new_dbase-21 6590.49 ( 0.00%) 6671.81 * 1.23%* Hmean new_dbase-41 14202.91 ( 0.00%) 14470.59 * 1.88%* Hmean new_dbase-61 21207.24 ( 0.00%) 21067.40 * -0.66%* Hmean new_dbase-81 25542.40 ( 0.00%) 25542.40 * 0.00%* Hmean new_dbase-101 30165.28 ( 0.00%) 30046.21 * -0.39%* Hmean new_dbase-121 33638.33 ( 0.00%) 33219.90 * -1.24%* Hmean new_dbase-141 36723.70 ( 0.00%) 37504.52 * 2.13%* Hmean new_dbase-161 42242.51 ( 0.00%) 42117.34 * -0.30%* Hmean shared-1 76.54 ( 0.00%) 76.09 * -0.59%* Hmean shared-21 7535.51 ( 0.00%) 5518.75 * -26.76%* Hmean shared-41 17207.81 ( 0.00%) 14651.94 * -14.85%* Hmean shared-61 20716.98 ( 0.00%) 18667.52 * -9.89%* Hmean shared-81 27603.83 ( 0.00%) 23466.45 * -14.99%* Hmean shared-101 26008.59 ( 0.00%) 29536.96 * 13.57%* Hmean shared-121 28354.76 ( 0.00%) 43139.39 * 52.14%* Hmean shared-141 38509.25 ( 0.00%) 41619.35 * 8.08%* Hmean shared-161 40496.07 ( 0.00%) 44303.46 * 9.40%*
Quoted 6 lines
Overall there is a small hit (in the noise level but consistent throughout
many workloads), except git-checkout which does quite well.Thanks,
Davidlohr
Thanks for running the patch through your performance tests.
Cheers,
Longman
Hi all,
Kernel test robot reported a will-it-scale.per_thread_ops -64.1% regression on IVB-desktop for v4.20-rc1.
The first bad commit is: 9bc8039e715da3b53dbac89525323a9f2f69b7b5, Yang Shi <[email protected]>: mm: brk: downgrade mmap_sem to read when shrinking
(https://lists.01.org/pipermail/lkp/2018-November/009335.html).
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.2/thread/100%/debian-x86_64-2018-04-03.cgz/lkp-ivb-d01/brk1/will-it-scale/0x20
commit:
85a06835f6 ("mm: mremap: downgrade mmap_sem to read when shrinking") 9bc8039e71 ("mm: brk: downgrade mmap_sem to read when shrinking")
85a06835f6f1ba79 9bc8039e715da3b53dbac89525
---------------- --------------------------
%stddev %change %stddev
\ | \
196250 ± 8% -64.1% 70494 will-it-scale.per_thread_ops
127330 ± 19% -98.0% 2525 ± 24% will-it-scale.time.involuntary_context_switches
727.50 ± 2% -77.0% 167.25 will-it-scale.time.percent_of_cpu_this_job_got
2141 ± 2% -77.6% 479.12 will-it-scale.time.system_time
50.48 ± 7% -48.5% 25.98 will-it-scale.time.user_time
34925294 ± 18% +270.3% 1.293e+08 ± 4% will-it-scale.time.voluntary_context_switches
1570007 ± 8% -64.1% 563958 will-it-scale.workload
6435 ± 2% -6.4% 6024 proc-vmstat.nr_shmem
1298 ± 16% -44.5% 721.00 ± 18% proc-vmstat.pgactivate
2341 +16.4% 2724 slabinfo.kmalloc-96.active_objs
2341 +16.4% 2724 slabinfo.kmalloc-96.num_objs
6346 ±150% -87.8% 776.25 ± 9% softirqs.NET_RX
160107 ± 8% +151.9% 403273 softirqs.SCHED
1097999 -13.0% 955526 softirqs.TIMER
5.50 ± 9% -81.8% 1.00 vmstat.procs.r
230700 ± 19% +269.9% 853292 ± 4% vmstat.system.cs
26706 ± 3% +15.7% 30910 ± 5% vmstat.system.in
11.24 ± 23% +72.2 83.39 mpstat.cpu.idle%
0.00 ±131% +0.0 0.04 ± 99% mpstat.cpu.iowait%
86.32 ± 2% -70.8 15.54 mpstat.cpu.sys%
2.44 ± 7% -1.4 1.04 ± 8% mpstat.cpu.usr%
20610709 ± 15% +2376.0% 5.103e+08 ± 34% cpuidle.C1.time
3233399 ± 8% +241.5% 11042785 ± 25% cpuidle.C1.usage
36172040 ± 6% +931.3% 3.73e+08 ± 15% cpuidle.C1E.time
783605 ± 4% +548.7% 5083041 ± 18% cpuidle.C1E.usage
28753819 ± 39% +1054.5% 3.319e+08 ± 49% cpuidle.C3.time
283912 ± 25% +688.4% 2238225 ± 34% cpuidle.C3.usage
1.507e+08 ± 47% +292.3% 5.913e+08 ± 28% cpuidle.C6.time
339861 ± 37% +549.7% 2208222 ± 24% cpuidle.C6.usage
2709719 ± 5% +824.2% 25043444 cpuidle.POLL.time
28602864 ± 18% +173.7% 78276116 ± 10% cpuidle.POLL.usageWe found that the patchset could fix the regression.
tests: 1
testcase/path_params/tbox_group/run: will-it-scale/performance-thread-100%-brk1-ucode=0x20/lkp-ivb-d01
commit:
85a06835f6 ("mm: mremap: downgrade mmap_sem to read when shrinking") fb835fe7f0 ("locking/rwsem: Ensure an RT task will not spin on reader")
85a06835f6f1ba79 fb835fe7f0adbd7c2c074b98ec
---------------- --------------------------
%stddev change %stddev
\ | \
120736 ± 22% 56% 188019 ± 6% will-it-scale.time.involuntary_context_switches
2126 ± 3% 4% 2215 will-it-scale.time.system_time
722 ± 3% 4% 752 will-it-scale.time.percent_of_cpu_this_job_got
36256485 ± 27% -35% 23682989 ± 3% will-it-scale.time.voluntary_context_switches
3151 ± 9% 11% 3504 turbostat.Avg_MHz
229285 ± 32% -30% 160660 ± 3% vmstat.system.cs
120736 ± 22% 56% 188019 ± 6% time.involuntary_context_switches
2126 ± 3% 4% 2215 time.system_time
722 ± 3% 4% 752 time.percent_of_cpu_this_job_got
36256485 ± 27% -35% 23682989 ± 3% time.voluntary_context_switches
23 643% 171 ± 3% proc-vmstat.nr_zone_inactive_file
23 643% 171 ± 3% proc-vmstat.nr_inactive_file
3664 12% 4121 proc-vmstat.nr_kernel_stack
6392 6% 6785 proc-vmstat.nr_slab_unreclaimable
9991 10176 proc-vmstat.nr_slab_reclaimable
63938 62394 proc-vmstat.nr_zone_active_anon
63938 62394 proc-vmstat.nr_active_anon
386388 ± 9% -6% 362272 proc-vmstat.pgfree
368296 ± 9% -10% 333074 proc-vmstat.numa_hit
368296 ± 9% -10% 333074 proc-vmstat.numa_local
5169 ± 13% -28% 3745 proc-vmstat.nr_shmem
1801 ± 21% -83% 309 proc-vmstat.pgactivate
0 1e+04 11441 latency_stats.avg.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
13165 ±222% -1e+04 0 latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup_revalidate.lookup_fast.walk_component.link_path_walk.path_lookupat.filename_lookup
22499 ±151% -2e+04 657 ± 7% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
117414 ±181% -9e+04 24418 ± 44% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
666005 ±218% -7e+05 198 ±141% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_access.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat.filename_lookup
2600097 ±132% -3e+06 572 latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
34391390 ±150% -3e+07 21807 ±141% latency_stats.avg.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
34624774 ±149% -3e+07 37668 ± 58% latency_stats.avg.max
0 1e+04 11441 latency_stats.max.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
22499 ±151% -2e+04 657 ± 7% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
37845 ±222% -4e+04 0 latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup_revalidate.lookup_fast.walk_component.link_path_walk.path_lookupat.filename_lookup
80096 ± 59% -8e+04 0 latency_stats.max.call_rwsem_down_write_failed_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
177149 ±195% -2e+05 24418 ± 44% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
689417 ±209% -7e+05 200 ±141% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_access.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat.filename_lookup
18679699 ±129% -2e+07 656 latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
83587334 ±129% -8e+07 43457 ±141% latency_stats.max.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
84867236 ±126% -8e+07 59318 ± 86% latency_stats.max.max
0 1e+04 11441 latency_stats.sum.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
22499 ±151% -2e+04 657 ± 7% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
39431 ±222% -4e+04 0 latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup_revalidate.lookup_fast.walk_component.link_path_walk.path_lookupat.filename_lookup
216448 ±200% -2e+05 24418 ± 44% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
691960 ±208% -7e+05 397 ±141% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_access.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat.filename_lookup
24239011 ±140% -2e+07 4768 ± 10% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
1.771e+08 ±122% -2e+08 43614 ±141% latency_stats.sum.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.939e+08 ± 36% -2e+08 0 latency_stats.sum.call_rwsem_down_write_failed_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.943e+08 ± 51% -2e+08 51929782 latency_stats.sum.max
407463 ± 10% -100% 0 perf-stat.total.page-faults
74225651 ± 26% -100% 0 perf-stat.total.context-switches
55293 ± 25% -100% 0 perf-stat.total.cpu-migrations
407463 ± 10% -100% 0 perf-stat.total.minor-faultstests: 1
testcase/path_params/tbox_group/run: will-it-scale/performance-thread-100%-brk1-ucode=0x20/lkp-ivb-d01
commit:
9bc8039e71 ("mm: brk: downgrade mmap_sem to read when shrinking") fb835fe7f0 ("locking/rwsem: Ensure an RT task will not spin on reader")
9bc8039e715da3b5 fb835fe7f0adbd7c2c074b98ec
---------------- --------------------------
%stddev change %stddev
\ | \
3500 ± 36% 5272% 188019 ± 6% will-it-scale.time.involuntary_context_switches
483 358% 2215 will-it-scale.time.system_time
168 346% 752 will-it-scale.time.percent_of_cpu_this_job_got
71190 180% 199232 ± 4% will-it-scale.per_thread_ops
569524 180% 1593862 ± 4% will-it-scale.workload
25.85 93% 49.95 ± 3% will-it-scale.time.user_time
1.314e+08 ± 3% -82% 23682989 ± 3% will-it-scale.time.voluntary_context_switches
30501 ± 9% -15% 25813 ± 4% vmstat.system.in
799593 ± 10% -80% 160660 ± 3% vmstat.system.cs
887 ± 11% 295% 3504 turbostat.Avg_MHz
23.60 ± 10% 68% 39.54 turbostat.CorWatt
28.38 ± 8% 57% 44.43 turbostat.PkgWatt
3500 ± 36% 5272% 188019 ± 6% time.involuntary_context_switches
483 358% 2215 time.system_time
168 346% 752 time.percent_of_cpu_this_job_got
25.85 93% 49.95 ± 3% time.user_time
1.314e+08 ± 3% -82% 23682989 ± 3% time.voluntary_context_switches
0 ± 44% 46220% 386 proc-vmstat.nr_zone_active_file
0 ± 44% 46220% 386 proc-vmstat.nr_active_file
23 643% 171 ± 3% proc-vmstat.nr_zone_inactive_file
23 643% 171 ± 3% proc-vmstat.nr_inactive_file
3690 12% 4121 proc-vmstat.nr_kernel_stack
6419 6% 6785 proc-vmstat.nr_slab_unreclaimable
9961 10176 proc-vmstat.nr_slab_reclaimable
229251 231278 proc-vmstat.nr_zone_unevictable
229251 231278 proc-vmstat.nr_unevictable
1008 1005 proc-vmstat.nr_page_table_pages
63178 62394 proc-vmstat.nr_zone_active_anon
63178 62394 proc-vmstat.nr_active_anon
432061 ± 12% -11% 385372 proc-vmstat.pgfault
408099 ± 10% -11% 362272 proc-vmstat.pgfree
422206 ± 9% -11% 373690 proc-vmstat.pgalloc_normal
382357 ± 11% -13% 333074 proc-vmstat.numa_hit
382357 ± 11% -13% 333074 proc-vmstat.numa_local
4428 ± 17% -15% 3745 proc-vmstat.nr_shmem
0 1e+04 11441 latency_stats.avg.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
11180 ±168% -1e+04 657 ± 7% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
19239 ±223% -2e+04 0 latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_get_acl.get_acl.posix_acl_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open
63702 ±169% -4e+04 24418 ± 44% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
77617 ±205% -8e+04 510 ± 11% latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
3043762 ±124% -3e+06 572 latency_stats.avg.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
11630441 ±139% -1e+07 21807 ±141% latency_stats.avg.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
12242832 ±129% -1e+07 37668 ± 58% latency_stats.avg.max
0 1e+04 11441 latency_stats.max.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
11180 ±168% -1e+04 657 ± 7% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
19239 ±223% -2e+04 0 latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_get_acl.get_acl.posix_acl_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open
29152 ± 11% -3e+04 0 latency_stats.max.call_rwsem_down_write_failed_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
65909 ±164% -4e+04 24418 ± 44% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
77617 ±205% -8e+04 510 ± 11% latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
17301268 ±125% -2e+07 656 latency_stats.max.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
44248611 ±140% -4e+07 43457 ±141% latency_stats.max.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
46380610 ±130% -5e+07 59318 ± 86% latency_stats.max.max
0 1e+04 11441 latency_stats.sum.msleep.cpuinfo_open.proc_reg_open.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
11180 ±168% -1e+04 657 ± 7% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.__lookup_slow.lookup_slow.walk_component.path_lookupat.filename_lookup
19239 ±223% -2e+04 0 latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_get_acl.get_acl.posix_acl_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open
74047 ±148% -5e+04 24418 ± 44% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_do_create.nfs3_proc_create.nfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
77617 ±205% -8e+04 510 ± 11% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_lookup.nfs_lookup.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
26043088 ±130% -3e+07 4768 ± 10% latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_run_task.rpc_call_sync.nfs3_rpc_wrapper.nfs3_proc_getattr.__nfs_revalidate_inode.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_lookupat
82480038 ±152% -8e+07 43614 ±141% latency_stats.sum.io_schedule.nfs_lock_and_join_requests.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.771e+09 -2e+09 51929782 latency_stats.sum.max
1.771e+09 -2e+09 0 latency_stats.sum.call_rwsem_down_write_failed_killable.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
420016 ± 12% -100% 0 perf-stat.total.page-faults
2.648e+08 ± 3% -100% 0 perf-stat.total.context-switches
52212 ± 18% -100% 0 perf-stat.total.cpu-migrations
420016 ± 12% -100% 0 perf-stat.total.minor-faultsBest Regards,
Rong Chen
Ok, those test robot reports are hard to read, but trying to distill it down:
Quoted 4 lines · Chen Rong
%stddev %change %stddev
\ | \
196250 ± 8% -64.1% 70494 will-it-scale.per_thread_opsThat's the original 64% regression..
And then with the patch set:
Quoted 3 lines
%stddev change %stddev
\ | \
71190 180% 199232 ± 4% will-it-scale.per_thread_opslooks like it's back up where it used to be.
So I guess we have numbers for the regression now. Thanks.
And that closes my biggest question for the new model, and with the new organization that gets ird of the arch-specific asm separately first and makes it a bit more legible that way, I guess I'll just Ack the whole series.
Linus
Hi, Waiman,
What's the status of this patchset? And its merging plan?
Best Regards,
Huang, Ying
Waiman Long
reviewerQuoted 6 lines · huang ying
Hi, Waiman,
What's the status of this patchset? And its merging plan?
Best Regards,
Huang, Ying
I have broken the patch into 3 parts (0/1/2) and rewritten some of them. Part 0 has been merged into tip. Parts 1 and 2 are still under testing.
Cheers,
Longman
Quoted 11 lines · Waiman Long
Quoted 6 lines · huang ying
Hi, Waiman,
What's the status of this patchset? And its merging plan?
Best Regards,
Huang, YingI have broken the patch into 3 parts (0/1/2) and rewritten some of them. Part 0 has been merged into tip. Parts 1 and 2 are still under testing.
Thanks! Please keep me updated!
Best Regards,
Huang, Ying
Quoted 3 lines
Cheers,
Longman