locking/rwsem: Rwsem rearchitecture part 0
Subsystems (22)
Waiman Long
reviewerv5 cover locking/rwsem: Rwsem rearchitecture part 0
v5:
- Rebase to the latest v5.1 tree and fix conflicts in arch/{xtensa,s390}/include/asm/Kbuild.
v4:
- Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c.
v3:
- Optimize __down_read_trylock() for the uncontended case as suggested by Linus.
v2:
- Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ.
- Update performance test data in patch 1.
The goal of this patchset is to remove the architecture specific files for rwsem-xadd to make it easer to add enhancements in the later rwsem patches. It also removes the legacy rwsem-spinlock.c file and make all the architectures use one single implementation of rwsem - rwsem-xadd.c.
Waiman Long (3):
locking/rwsem: Remove arch specific rwsem files locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all
archs
locking/rwsem: Optimize down_read_trylock()
MAINTAINERS | 1 - arch/alpha/Kconfig | 7 - arch/alpha/include/asm/rwsem.h | 211 -------------------- arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm/include/asm/Kbuild | 1 - arch/arm64/Kconfig | 3 - arch/arm64/include/asm/Kbuild | 1 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/Kconfig | 4 - arch/ia64/include/asm/rwsem.h | 172 ---------------- arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/powerpc/include/asm/Kbuild | 1 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/s390/include/asm/Kbuild | 1 - arch/sh/Kconfig | 6 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/Kconfig | 8 - arch/sparc/include/asm/Kbuild | 1 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/include/asm/rwsem.h | 237 ---------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 --------------- arch/x86/um/Kconfig | 6 - arch/x86/um/Makefile | 1 - arch/xtensa/Kconfig | 3 - arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------- include/linux/rwsem-spinlock.h | 47 ----- include/linux/rwsem.h | 9 +- kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem-spinlock.c | 339 -------------------------------- kernel/locking/rwsem.h | 130 ++++++++++++ 48 files changed, 135 insertions(+), 1447 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h delete mode 100644 include/linux/rwsem-spinlock.h delete mode 100644 kernel/locking/rwsem-spinlock.c
Signature · 1 line
2.18.1
Waiman Long
reviewerPATCH v5 1/3 locking/rwsem: Remove arch specific rwsem files
As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort.
Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway.
By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks):
Before Patch After Patch
# of Threads wlock rlock mixed wlock rlock mixed
------------ ----- ----- ----- ----- ----- -----
1 29,201 30,143 29,458 28,615 30,172 29,201
2 6,807 13,299 1,171 7,725 15,025 1,804
4 6,504 12,755 1,520 7,127 14,286 1,345
8 6,762 13,412 764 6,826 13,652 726
16 6,693 15,408 662 6,599 15,938 626
32 6,145 15,286 496 5,549 15,487 511
64 5,812 15,495 60 5,858 15,572 60There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions.
--- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ---------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 -------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 --------------------- arch/x86/um/Makefile | 1 - arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------- include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++ 19 files changed, 133 insertions(+), 930 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h
MAINTAINERS +0 -1
@@ -9089,7 +9089,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/
arch/alpha/include/asm/rwsem.h deleted +0 -211
211 lines not on this page · raw message
arch/arm/include/asm/Kbuild +0 -1
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h
arch/hexagon/include/asm/Kbuild +0 -1
@@ -26,7 +26,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h
arch/ia64/include/asm/rwsem.h deleted +0 -172
172 lines not on this page · raw message
arch/powerpc/include/asm/Kbuild +0 -1
@@ -8,6 +8,5 @@ generic-y += irq_regs.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h
arch/s390/include/asm/Kbuild +0 -1
@@ -20,7 +20,6 @@ generic-y += local.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h
arch/sparc/include/asm/Kbuild +0 -1
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/x86/include/asm/rwsem.h deleted +0 -237
237 lines not on this page · raw message
arch/x86/lib/Makefile +0 -1
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
arch/x86/lib/rwsem.S deleted +0 -156
156 lines not on this page · raw message
arch/x86/um/Makefile +0 -1
@@ -21,7 +21,6 @@ obj-y += checksum_32.o syscalls_32.o obj-$(CONFIG_ELF_CORE) += elfcore.o subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o -subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o else
arch/xtensa/include/asm/Kbuild +0 -1
@@ -24,7 +24,6 @@ generic-y += percpu.h generic-y += preempt.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += sections.h generic-y += socket.h generic-y += topology.h
include/asm-generic/rwsem.h deleted +0 -140
140 lines not on this page · raw message
include/linux/rwsem.h +1 -3
@@ -57,15 +57,13 @@ extern struct rw_semaphore *rwsem_down_write_failed_killable(struct rw_semaphoreextern struct rw_semaphore *rwsem_wake (struct rw_semaphore *);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {return atomic_long_read (&sem->count) !=0 ; } +#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)#endif
kernel/locking/percpu-rwsem.c +2 -0
@@ -7,6 +7,8 @@#include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" +int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key) {
kernel/locking/rwsem.h +130 -0
139 lines not on this page · raw message
Signature · 1 line
2.18.1
Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
Quoted 2 lines · Waiman Long
19 files changed, 133 insertions(+), 930 deletions(-)
Lovely. And it all looks sane to me.
So ack.
The only comment I have is about __down_read_trylock(), which probably isn't critical enough to actually care about, but:
kernel/locking/rwsem.h static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) · line 187
Quoted 12 lines
kernel/locking/rwsem.h +12
-0+static inline int __down_read_trylock (struct rw_semaphore *sem) +{ +long tmp; + +while ((tmp =atomic_long_read (&sem->count)) >=0 ) { +if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) { +return 1 ; + } + } +return 0 ; +}
So this seems to
(a) read the line early (the whole cacheline in shared state issue)
(b) read the line again unnecessarily in the while loop
Now, (a) might be explained by "well, maybe we do trylock even with existing readers", although I continue to think that the case we should optimize for is simply the uncontended one, where we don't even have multiple readers.
But (b) just seems silly.
So I wonder if it shouldn't just be
long tmp = 0;
do {
long new = atomic_long_cmpxchg_acquire(&sem->count, tmp,
tmp + RWSEM_ACTIVE_READ_BIAS);
if (likely(new == tmp))
return 1;
tmp = new;
} while (tmp >= 0);
return 0;which would seem simpler and solve both issues. Hmm?
But honestly, I didn't check what our uses of down_read_trylock() look like. We have more of them than I expected, and I _think_ the normal case is the "nobody else holds the lock", but that's just a gut feeling.
Some of them _might_ be performance-critical. There's the one on mmap_sem in the fault handling path, for example. And yes, I'd expect the normal case to very much be "no other readers or writers" for that one.
NOTE! The above code snippet is absolutely untested, and might be completely wrong. Take it as a "something like this" rather than anything else.
Linus
Waiman Long
reviewerRe: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
Quoted 59 lines · Linus Torvalds
Quoted 1 line · Waiman Long
19 files changed, 133 insertions(+), 930 deletions(-)Lovely. And it all looks sane to me.
So ack.
The only comment I have is about
__down_read_trylock(), which probably isn't critical enough to actually care about, but:kernel/locking/rwsem.h static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) · line 187
Quoted 12 lines
kernel/locking/rwsem.h +12
-0+static inline int __down_read_trylock (struct rw_semaphore *sem) +{ +long tmp; + +while ((tmp =atomic_long_read (&sem->count)) >=0 ) { +if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) { +return 1 ; + } + } +return 0 ; +}So this seems to
(a) read the line early (the whole cacheline in shared state issue)
(b) read the line again unnecessarily in the while loop
Now, (a) might be explained by "well, maybe we do trylock even with existing readers", although I continue to think that the case we should optimize for is simply the uncontended one, where we don't even have multiple readers.
But (b) just seems silly.
So I wonder if it shouldn't just be
long tmp = 0; do { long new = atomic_long_cmpxchg_acquire(&sem->count, tmp, tmp + RWSEM_ACTIVE_READ_BIAS); if (likely(new == tmp)) return 1; tmp = new; } while (tmp >= 0); return 0;which would seem simpler and solve both issues. Hmm?
But honestly, I didn't check what our uses of
down_read_trylock()look like. We have more of them than I expected, and I _think_ the normal case is the "nobody else holds the lock", but that's just a gut feeling.Some of them _might_ be performance-critical. There's the one on
mmap_semin the fault handling path, for example. And yes, I'd expect the normal case to very much be "no other readers or writers" for that one.
Quoted 5 lines
NOTE! The above code snippet is absolutely untested, and might be completely wrong. Take it as a "something like this" rather than anything else.
Linus
As you have noticed already, this patch is just for moving code around without changing it. I optimize __down_read_trylock() in patch 3.
Cheers,
Longman
Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
In reply to Linus TorvaldsQuoted 4 lines · Linus Torvalds
Some of them _might_ be performance-critical. There's the one on
mmap_semin the fault handling path, for example. And yes, I'd expect the normal case to very much be "no other readers or writers" for that one.
Yeah, the mmap_sem case in the fault path is really expecting an unlocked state. To the point that four archs have added branch predictions, ie:
92181f190b6 (x86: optimise x86's do_page_fault (C entry point for the page fault path)) b15021d994f (powerpc/mm: Add a bunch of (un)likely annotations to do_page_fault)
And using PROFILE_ANNOTATED_BRANCHES shows pretty clearly:
(without resetting the counters)
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 4603685 34 0 do_user_addr_fault fault.c 1416 (bootup) 382327745 449 0 do_user_addr_fault fault.c 1416 (kernel build) 399446159 461 0 do_user_addr_fault fault.c 1416 (redis benchmark)
It would probably wouldn't harm doing the unlikely() for all archs, or alternatively, add likely() to the atomic_long_try_cmpxchg_acquire in patch 3 and do it implicitly but maybe that would be less flexible(?)
Thanks,
Davidlohr
Waiman Long
reviewerRe: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
Quoted 26 lines · Davidlohr Bueso
Quoted 4 lines · Linus Torvalds
Some of them _might_ be performance-critical. There's the one on
mmap_semin the fault handling path, for example. And yes, I'd expect the normal case to very much be "no other readers or writers" for that one.Yeah, the
mmap_semcase in the fault path is really expecting an unlocked state. To the point that four archs have added branch predictions, ie:
92181f190b6(x86: optimise x86'sdo_page_fault(C entry point for the page fault path))b15021d994f(powerpc/mm: Add a bunch of (un)likely annotations todo_page_fault)And using
PROFILE_ANNOTATED_BRANCHESshows pretty clearly:
(without resetting the counters)correct incorrect % Function File Line ------- --------- - -------- ---- ---- 4603685 34 0 do_user_addr_fault fault.c 1416 (bootup)
382327745 449 0 do_user_addr_faultfault.c1416 (kernel build)
399446159 461 0 do_user_addr_faultfault.c1416 (redis benchmark)
Quoted 6 lines
It would probably wouldn't harm doing the
unlikely()for all archs, or alternatively, addlikely()to theatomic_long_try_cmpxchg_acquirein patch 3 and do it implicitly but maybe that would be less flexible(?)Thanks,
Davidlohr
I had used the my lock event counting code to count the number of contended and uncontended trylocks. I tested both bootup and kernel build. I think I saw less than 1% were contended, the rests were all uncontended. That is similar to what you got. I thought I had sent the data out previously, but I couldn't find the email. That was the main reason why I took Linus' suggestion to optimize it for the uncontended case.
Thanks,
Longman
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Remove arch specific rwsem files
In reply to Waiman Longlocking/rwsem: Remove arch specific rwsem files
Commit-ID: 701fd16f3b4e3e5f317a051b36962b8cc756c138 Gitweb: https://git.kernel.org/tip/701fd16f3b4e3e5f317a051b36962b8cc756c138 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:06 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 11:42:33 +0200
locking/rwsem: Remove arch specific rwsem files
As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort.
Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway.
By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks):
Before Patch After Patch
# of Threads wlock rlock mixed wlock rlock mixed
------------ ----- ----- ----- ----- ----- -----
1 29,201 30,143 29,458 28,615 30,172 29,201
2 6,807 13,299 1,171 7,725 15,025 1,804
4 6,504 12,755 1,520 7,127 14,286 1,345
8 6,762 13,412 764 6,826 13,652 726
16 6,693 15,408 662 6,599 15,938 626
32 6,145 15,286 496 5,549 15,487 511
64 5,812 15,495 60 5,858 15,572 60There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions.
--- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/x86/um/Makefile | 1 - arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ 19 files changed, 133 insertions(+), 930 deletions(-)
MAINTAINERS +0 -1
@@ -9098,7 +9098,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/
arch/alpha/include/asm/rwsem.h deleted +0 -211
211 lines not on this page · raw message
arch/arm/include/asm/Kbuild +0 -1
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h
arch/hexagon/include/asm/Kbuild +0 -1
@@ -27,7 +27,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h
arch/ia64/include/asm/rwsem.h deleted +0 -172
172 lines not on this page · raw message
arch/powerpc/include/asm/Kbuild +0 -1
@@ -8,6 +8,5 @@ generic-y += irq_regs.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h
arch/s390/include/asm/Kbuild +0 -1
@@ -20,7 +20,6 @@ generic-y += local.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +0 -1
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h
arch/sparc/include/asm/Kbuild +0 -1
@@ -18,7 +18,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/x86/include/asm/rwsem.h deleted +0 -237
237 lines not on this page · raw message
arch/x86/lib/Makefile +0 -1
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
arch/x86/lib/rwsem.S deleted +0 -156
156 lines not on this page · raw message
arch/x86/um/Makefile +0 -1
@@ -21,7 +21,6 @@ obj-y += checksum_32.o syscalls_32.o obj-$(CONFIG_ELF_CORE) += elfcore.o subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o -subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o else
arch/xtensa/include/asm/Kbuild +0 -1
@@ -25,7 +25,6 @@ generic-y += percpu.h generic-y += preempt.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += sections.h generic-y += socket.h generic-y += topology.h
include/asm-generic/rwsem.h deleted +0 -140
140 lines not on this page · raw message
include/linux/rwsem.h +1 -3
@@ -57,15 +57,13 @@ extern struct rw_semaphore *rwsem_down_write_failed_killable(struct rw_semaphoreextern struct rw_semaphore *rwsem_wake (struct rw_semaphore *);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {return atomic_long_read (&sem->count) !=0 ; } +#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)#endif
kernel/locking/percpu-rwsem.c +2 -0
@@ -7,6 +7,8 @@#include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" +int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key) {
kernel/locking/rwsem.h +130 -0
139 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [tip:locking/core] locking/rwsem: Remove arch specific rwsem files
arch/x86/um/Makefile obj-y += checksum_32.o syscalls_32.o · line 26
Quoted 12 lines · tip-bot for Waiman Long
arch/x86/um/Makefile +0
-1@@ -21,7 +21,6 @@ obj-y += checksum_32.o syscalls_32.o obj-$(CONFIG_ELF_CORE) += elfcore.o subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o -subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o else
The x86_64 um build needs this on top:
arch/x86/um/Makefile +1 -2
@@ -26,8 +26,7 @@ else obj-y += syscalls_64.o vdso/ -subarch-y = ../lib/csum-partial_64.o ../lib/memcpy_64.o ../entry/thunk_64.o \ - ../lib/rwsem.o +subarch-y = ../lib/csum-partial_64.o ../lib/memcpy_64.o ../entry/thunk_64.o endif
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Remove arch specific rwsem files
In reply to Waiman Longlocking/rwsem: Remove arch specific rwsem files
Commit-ID: 46ad0840b1584b92b5ff2cc3ed0b011dd6b8e0f1 Gitweb: https://git.kernel.org/tip/46ad0840b1584b92b5ff2cc3ed0b011dd6b8e0f1 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:06 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 14:50:50 +0200
locking/rwsem: Remove arch specific rwsem files
As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort.
Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway.
By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks):
Before Patch After Patch
# of Threads wlock rlock mixed wlock rlock mixed
------------ ----- ----- ----- ----- ----- -----
1 29,201 30,143 29,458 28,615 30,172 29,201
2 6,807 13,299 1,171 7,725 15,025 1,804
4 6,504 12,755 1,520 7,127 14,286 1,345
8 6,762 13,412 764 6,826 13,652 726
16 6,693 15,408 662 6,599 15,938 626
32 6,145 15,286 496 5,549 15,487 511
64 5,812 15,495 60 5,858 15,572 60There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions.
--- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/x86/um/Makefile | 4 +- arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ 19 files changed, 134 insertions(+), 932 deletions(-)
MAINTAINERS +0 -1
@@ -9098,7 +9098,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/
arch/alpha/include/asm/rwsem.h deleted +0 -211
211 lines not on this page · raw message
arch/arm/include/asm/Kbuild +0 -1
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h
arch/hexagon/include/asm/Kbuild +0 -1
@@ -27,7 +27,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h
arch/ia64/include/asm/rwsem.h deleted +0 -172
172 lines not on this page · raw message
arch/powerpc/include/asm/Kbuild +0 -1
@@ -8,6 +8,5 @@ generic-y += irq_regs.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h
arch/s390/include/asm/Kbuild +0 -1
@@ -20,7 +20,6 @@ generic-y += local.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +0 -1
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h
arch/sparc/include/asm/Kbuild +0 -1
@@ -18,7 +18,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/x86/include/asm/rwsem.h deleted +0 -237
237 lines not on this page · raw message
arch/x86/lib/Makefile +0 -1
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
arch/x86/lib/rwsem.S deleted +0 -156
156 lines not on this page · raw message
arch/x86/um/Makefile +1 -3
@@ -21,14 +21,12 @@ obj-y += checksum_32.o syscalls_32.o obj-$(CONFIG_ELF_CORE) += elfcore.o subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o -subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o else obj-y += syscalls_64.o vdso/ -subarch-y = ../lib/csum-partial_64.o ../lib/memcpy_64.o ../entry/thunk_64.o \ - ../lib/rwsem.o +subarch-y = ../lib/csum-partial_64.o ../lib/memcpy_64.o ../entry/thunk_64.o endif
arch/xtensa/include/asm/Kbuild +0 -1
@@ -25,7 +25,6 @@ generic-y += percpu.h generic-y += preempt.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += sections.h generic-y += socket.h generic-y += topology.h
include/asm-generic/rwsem.h deleted +0 -140
140 lines not on this page · raw message
include/linux/rwsem.h +1 -3
@@ -57,15 +57,13 @@ extern struct rw_semaphore *rwsem_down_write_failed_killable(struct rw_semaphoreextern struct rw_semaphore *rwsem_wake (struct rw_semaphore *);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {return atomic_long_read (&sem->count) !=0 ; } +#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)#endif
kernel/locking/percpu-rwsem.c +2 -0
@@ -7,6 +7,8 @@#include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" +int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key) {
kernel/locking/rwsem.h +130 -0
139 lines not on this page · raw message
Waiman Long
reviewerPATCH v5 2/3 locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
In reply to Waiman LongCurrently, we have two different implementation of rwsem:
1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c)
2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c)
As we are going to use a single generic implementation for rwsem-xadd.c and no architecture-specific code will be needed, there is no point in keeping two different implementations of rwsem. In most cases, the performance of rwsem-spinlock.c will be worse. It also doesn't get all the performance tuning and optimizations that had been implemented in rwsem-xadd.c over the years.
For simplication, we are going to remove rwsem-spinlock.c and make all architectures use a single implementation of rwsem - rwsem-xadd.c.
All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM in the code are removed.
--- arch/alpha/Kconfig | 7 - arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm64/Kconfig | 3 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/ia64/Kconfig | 4 - arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/sh/Kconfig | 6 - arch/sparc/Kconfig | 8 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/um/Kconfig | 6 - arch/xtensa/Kconfig | 3 - include/linux/rwsem-spinlock.h | 47 ----- include/linux/rwsem.h | 5 - kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/rwsem-spinlock.c | 339 -------------------------------- kernel/locking/rwsem.h | 3 - 31 files changed, 2 insertions(+), 520 deletions(-) delete mode 100644 include/linux/rwsem-spinlock.h delete mode 100644 kernel/locking/rwsem-spinlock.c
arch/alpha/Kconfig +0 -7
@@ -49,13 +49,6 @@ config MMU bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool default n
arch/arc/Kconfig +0 -3
@@ -63,9 +63,6 @@ config SCHED_OMIT_FRAME_POINTER config GENERIC_CSUM def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ARCH_DISCONTIGMEM_ENABLE def_bool n
arch/arm/Kconfig +0 -4
@@ -178,10 +178,6 @@ config TRACE_IRQFLAGS_SUPPORT bool default !CPU_V7M -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool
arch/arm64/Kconfig +0 -3
@@ -237,9 +237,6 @@ config LOCKDEP_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/c6x/Kconfig +0 -3
@@ -27,9 +27,6 @@ config MMU config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/csky/Kconfig +0 -3
@@ -92,9 +92,6 @@ config GENERIC_HWEIGHT config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config STACKTRACE_SUPPORT def_bool y
arch/h8300/Kconfig +0 -3
@@ -27,9 +27,6 @@ config H8300 config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/hexagon/Kconfig +0 -6
@@ -65,12 +65,6 @@ config GENERIC_CSUM config GENERIC_IRQ_PROBE def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool n - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/ia64/Kconfig +0 -4
@@ -83,10 +83,6 @@ config STACKTRACE_SUPPORT config GENERIC_LOCKBREAK def_bool n -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config HUGETLB_PAGE_SIZE_VARIABLE bool depends on HUGETLB_PAGE
arch/m68k/Kconfig +0 -7
@@ -32,13 +32,6 @@ config M68K config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/microblaze/Kconfig +0 -6
@@ -58,15 +58,9 @@ config CPU_LITTLE_ENDIAN endchoice -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ZONE_DMA def_bool y -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 def_bool n
arch/mips/Kconfig +0 -7
@@ -1037,13 +1037,6 @@ source "arch/mips/paravirt/Kconfig" endmenu -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_HWEIGHT bool default y
arch/nds32/Kconfig +0 -3
@@ -60,9 +60,6 @@ config GENERIC_LOCKBREAK def_bool y depends on PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool y
arch/nios2/Kconfig +0 -3
@@ -40,9 +40,6 @@ config NO_IOPORT_MAP config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool n
arch/openrisc/Kconfig +0 -6
@@ -43,12 +43,6 @@ config CPU_BIG_ENDIAN config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - def_bool n - config GENERIC_HWEIGHT def_bool y
arch/parisc/Kconfig +0 -6
@@ -75,12 +75,6 @@ config GENERIC_LOCKBREAK default y depends on SMP && PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool default n
arch/powerpc/Kconfig +0 -7
@@ -103,13 +103,6 @@ config LOCKDEP_SUPPORT bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config GENERIC_LOCKBREAK bool default y
arch/riscv/Kconfig +0 -3
@@ -69,9 +69,6 @@ config STACKTRACE_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/s390/Kconfig +0 -6
@@ -14,12 +14,6 @@ config LOCKDEP_SUPPORT config STACKTRACE_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config ARCH_HAS_ILOG2_U32 def_bool n
arch/sh/Kconfig +0 -6
@@ -90,12 +90,6 @@ config ARCH_DEFCONFIG default "arch/sh/configs/shx3_defconfig" if SUPERH32 default "arch/sh/configs/cayman_defconfig" if SUPERH64 -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_BUG def_bool y depends on BUG && SUPERH32
arch/sparc/Kconfig +0 -8
@@ -191,14 +191,6 @@ config NR_CPUS source "kernel/Kconfig.hz" -config RWSEM_GENERIC_SPINLOCK - bool - default y if SPARC32 - -config RWSEM_XCHGADD_ALGORITHM - bool - default y if SPARC64 - config GENERIC_HWEIGHT bool default y
arch/unicore32/Kconfig +0 -6
@@ -38,12 +38,6 @@ config STACKTRACE_SUPPORT config LOCKDEP_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/x86/Kconfig +0 -3
@@ -268,9 +268,6 @@ config ARCH_MAY_HAVE_PC_FDC def_bool y depends on ISA_DMA_API -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/x86/um/Kconfig +0 -6
@@ -32,12 +32,6 @@ config ARCH_DEFCONFIG default "arch/um/configs/i386_defconfig" if X86_32 default "arch/um/configs/x86_64_defconfig" if X86_64 -config RWSEM_XCHGADD_ALGORITHM - def_bool 64BIT - -config RWSEM_GENERIC_SPINLOCK - def_bool !RWSEM_XCHGADD_ALGORITHM - config 3_LEVEL_PGTABLES bool "Three-level pagetables" if !64BIT default 64BIT
arch/xtensa/Kconfig +0 -3
@@ -46,9 +46,6 @@ config XTENSA with reasonable minimum requirements. The Xtensa Linux project has a home page at <http://www.linux-xtensa.org/>. -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
include/linux/rwsem-spinlock.h deleted +0 -47
47 lines not on this page · raw message
include/linux/rwsem.h +0 -5
@@ -22,10 +22,6 @@struct rw_semaphore ; -#ifdef CONFIG_RWSEM_GENERIC_SPINLOCK -#include <linux/rwsem-spinlock.h> /* use a generic implementation */ -#define __RWSEM_INIT_COUNT(name) .count = RWSEM_UNLOCKED_VALUE -#else/* All arch specific implementations share the same struct */ struct rw_semaphore {atomic_long_t count;
@@ -65,7 +61,6 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE) -#endif /* Common initializer macros and functions */
kernel/Kconfig.locks +1 -1
@@ -229,7 +229,7 @@ config MUTEX_SPIN_ON_OWNER config RWSEM_SPIN_ON_OWNER def_bool y - depends on SMP && RWSEM_XCHGADD_ALGORITHM && ARCH_SUPPORTS_ATOMIC_RMW + depends on SMP && ARCH_SUPPORTS_ATOMIC_RMW config LOCK_SPIN_ON_OWNER def_bool y
kernel/locking/Makefile +1 -3
@@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT := n -obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o rwsem-xadd.o ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)
@@ -25,8 +25,6 @@ obj-$(CONFIG_RT_MUTEXES) += rtmutex.o obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
kernel/locking/rwsem-spinlock.c deleted +0 -339
339 lines not on this page · raw message
kernel/locking/rwsem.h +0 -3
@@ -153,7 +153,6 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) }#endif -#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM /* * lock for reading */
@@ -260,5 +259,3 @@ static inline void __downgrade_write(struct rw_semaphore *sem)if (tmp <0 )rwsem_downgrade_wake (sem); } - -#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
Signature · 1 line
2.18.1
Re: [PATCH v5 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
Quoted 3 lines · Waiman Long
For simplication, we are going to remove
rwsem-spinlock.cand make all architectures use a single implementation of rwsem -rwsem-xadd.c.
Ack.
Linus
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
In reply to Waiman Longlocking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
Commit-ID: 79407a77fe0ea11c0d38c5f4a3936bf35a994965 Gitweb: https://git.kernel.org/tip/79407a77fe0ea11c0d38c5f4a3936bf35a994965 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:07 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 11:42:34 +0200
locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
Currently, we have two different implementation of rwsem:
1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c)
2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c)
As we are going to use a single generic implementation for rwsem-xadd.c and no architecture-specific code will be needed, there is no point in keeping two different implementations of rwsem. In most cases, the performance of rwsem-spinlock.c will be worse. It also doesn't get all the performance tuning and optimizations that had been implemented in rwsem-xadd.c over the years.
For simplication, we are going to remove rwsem-spinlock.c and make all architectures use a single implementation of rwsem - rwsem-xadd.c.
All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM in the code are removed.
--- arch/alpha/Kconfig | 7 - arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm64/Kconfig | 3 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/ia64/Kconfig | 4 - arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/sh/Kconfig | 6 - arch/sparc/Kconfig | 8 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/um/Kconfig | 6 - arch/xtensa/Kconfig | 3 - include/linux/rwsem-spinlock.h | 47 ------ include/linux/rwsem.h | 5 - kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/rwsem-spinlock.c | 339 ---------------------------------------- kernel/locking/rwsem.h | 3 - 31 files changed, 2 insertions(+), 520 deletions(-)
arch/alpha/Kconfig +0 -7
@@ -49,13 +49,6 @@ config MMU bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool default n
arch/arc/Kconfig +0 -3
@@ -63,9 +63,6 @@ config SCHED_OMIT_FRAME_POINTER config GENERIC_CSUM def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ARCH_DISCONTIGMEM_ENABLE def_bool n
arch/arm/Kconfig +0 -4
@@ -178,10 +178,6 @@ config TRACE_IRQFLAGS_SUPPORT bool default !CPU_V7M -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool
arch/arm64/Kconfig +0 -3
@@ -237,9 +237,6 @@ config LOCKDEP_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/c6x/Kconfig +0 -3
@@ -27,9 +27,6 @@ config MMU config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/csky/Kconfig +0 -3
@@ -92,9 +92,6 @@ config GENERIC_HWEIGHT config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config STACKTRACE_SUPPORT def_bool y
arch/h8300/Kconfig +0 -3
@@ -27,9 +27,6 @@ config H8300 config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/hexagon/Kconfig +0 -6
@@ -65,12 +65,6 @@ config GENERIC_CSUM config GENERIC_IRQ_PROBE def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool n - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/ia64/Kconfig +0 -4
@@ -83,10 +83,6 @@ config STACKTRACE_SUPPORT config GENERIC_LOCKBREAK def_bool n -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config HUGETLB_PAGE_SIZE_VARIABLE bool depends on HUGETLB_PAGE
arch/m68k/Kconfig +0 -7
@@ -32,13 +32,6 @@ config M68K config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/microblaze/Kconfig +0 -6
@@ -58,15 +58,9 @@ config CPU_LITTLE_ENDIAN endchoice -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ZONE_DMA def_bool y -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 def_bool n
arch/mips/Kconfig +0 -7
@@ -1037,13 +1037,6 @@ source "arch/mips/paravirt/Kconfig" endmenu -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_HWEIGHT bool default y
arch/nds32/Kconfig +0 -3
@@ -60,9 +60,6 @@ config GENERIC_LOCKBREAK def_bool y depends on PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool y
arch/nios2/Kconfig +0 -3
@@ -40,9 +40,6 @@ config NO_IOPORT_MAP config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool n
arch/openrisc/Kconfig +0 -6
@@ -43,12 +43,6 @@ config CPU_BIG_ENDIAN config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - def_bool n - config GENERIC_HWEIGHT def_bool y
arch/parisc/Kconfig +0 -6
@@ -75,12 +75,6 @@ config GENERIC_LOCKBREAK default y depends on SMP && PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool default n
arch/powerpc/Kconfig +0 -7
@@ -103,13 +103,6 @@ config LOCKDEP_SUPPORT bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config GENERIC_LOCKBREAK bool default y
arch/riscv/Kconfig +0 -3
@@ -69,9 +69,6 @@ config STACKTRACE_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/s390/Kconfig +0 -6
@@ -14,12 +14,6 @@ config LOCKDEP_SUPPORT config STACKTRACE_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config ARCH_HAS_ILOG2_U32 def_bool n
arch/sh/Kconfig +0 -6
@@ -90,12 +90,6 @@ config ARCH_DEFCONFIG default "arch/sh/configs/shx3_defconfig" if SUPERH32 default "arch/sh/configs/cayman_defconfig" if SUPERH64 -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_BUG def_bool y depends on BUG && SUPERH32
arch/sparc/Kconfig +0 -8
@@ -191,14 +191,6 @@ config NR_CPUS source "kernel/Kconfig.hz" -config RWSEM_GENERIC_SPINLOCK - bool - default y if SPARC32 - -config RWSEM_XCHGADD_ALGORITHM - bool - default y if SPARC64 - config GENERIC_HWEIGHT bool default y
arch/unicore32/Kconfig +0 -6
@@ -38,12 +38,6 @@ config STACKTRACE_SUPPORT config LOCKDEP_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/x86/Kconfig +0 -3
@@ -268,9 +268,6 @@ config ARCH_MAY_HAVE_PC_FDC def_bool y depends on ISA_DMA_API -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/x86/um/Kconfig +0 -6
@@ -32,12 +32,6 @@ config ARCH_DEFCONFIG default "arch/um/configs/i386_defconfig" if X86_32 default "arch/um/configs/x86_64_defconfig" if X86_64 -config RWSEM_XCHGADD_ALGORITHM - def_bool 64BIT - -config RWSEM_GENERIC_SPINLOCK - def_bool !RWSEM_XCHGADD_ALGORITHM - config 3_LEVEL_PGTABLES bool "Three-level pagetables" if !64BIT default 64BIT
arch/xtensa/Kconfig +0 -3
@@ -46,9 +46,6 @@ config XTENSA with reasonable minimum requirements. The Xtensa Linux project has a home page at <http://www.linux-xtensa.org/>. -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
include/linux/rwsem-spinlock.h deleted +0 -47
47 lines not on this page · raw message
include/linux/rwsem.h +0 -5
@@ -22,10 +22,6 @@struct rw_semaphore ; -#ifdef CONFIG_RWSEM_GENERIC_SPINLOCK -#include <linux/rwsem-spinlock.h> /* use a generic implementation */ -#define __RWSEM_INIT_COUNT(name) .count = RWSEM_UNLOCKED_VALUE -#else/* All arch specific implementations share the same struct */ struct rw_semaphore {atomic_long_t count;
@@ -65,7 +61,6 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE) -#endif /* Common initializer macros and functions */
kernel/Kconfig.locks +1 -1
@@ -229,7 +229,7 @@ config MUTEX_SPIN_ON_OWNER config RWSEM_SPIN_ON_OWNER def_bool y - depends on SMP && RWSEM_XCHGADD_ALGORITHM && ARCH_SUPPORTS_ATOMIC_RMW + depends on SMP && ARCH_SUPPORTS_ATOMIC_RMW config LOCK_SPIN_ON_OWNER def_bool y
kernel/locking/Makefile +1 -3
@@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT := n -obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o rwsem-xadd.o ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)
@@ -25,8 +25,6 @@ obj-$(CONFIG_RT_MUTEXES) += rtmutex.o obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
kernel/locking/rwsem-spinlock.c deleted +0 -339
339 lines not on this page · raw message
kernel/locking/rwsem.h +0 -3
@@ -153,7 +153,6 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) }#endif -#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM /* * lock for reading */
@@ -260,5 +259,3 @@ static inline void __downgrade_write(struct rw_semaphore *sem)if (tmp <0 )rwsem_downgrade_wake (sem); } - -#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
In reply to Waiman Longlocking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
Commit-ID: 390a0c62c23cb026cd4664a66f6f45fed3a215f6 Gitweb: https://git.kernel.org/tip/390a0c62c23cb026cd4664a66f6f45fed3a215f6 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:07 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 14:50:52 +0200
locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
Currently, we have two different implementation of rwsem:
1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c)
2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c)
As we are going to use a single generic implementation for rwsem-xadd.c and no architecture-specific code will be needed, there is no point in keeping two different implementations of rwsem. In most cases, the performance of rwsem-spinlock.c will be worse. It also doesn't get all the performance tuning and optimizations that had been implemented in rwsem-xadd.c over the years.
For simplication, we are going to remove rwsem-spinlock.c and make all architectures use a single implementation of rwsem - rwsem-xadd.c.
All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM in the code are removed.
--- arch/alpha/Kconfig | 7 - arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm64/Kconfig | 3 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/ia64/Kconfig | 4 - arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/sh/Kconfig | 6 - arch/sparc/Kconfig | 8 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/um/Kconfig | 6 - arch/xtensa/Kconfig | 3 - include/linux/rwsem-spinlock.h | 47 ------ include/linux/rwsem.h | 5 - kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/rwsem-spinlock.c | 339 ---------------------------------------- kernel/locking/rwsem.h | 3 - 31 files changed, 2 insertions(+), 520 deletions(-)
arch/alpha/Kconfig +0 -7
@@ -49,13 +49,6 @@ config MMU bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool default n
arch/arc/Kconfig +0 -3
@@ -63,9 +63,6 @@ config SCHED_OMIT_FRAME_POINTER config GENERIC_CSUM def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ARCH_DISCONTIGMEM_ENABLE def_bool n
arch/arm/Kconfig +0 -4
@@ -178,10 +178,6 @@ config TRACE_IRQFLAGS_SUPPORT bool default !CPU_V7M -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool
arch/arm64/Kconfig +0 -3
@@ -237,9 +237,6 @@ config LOCKDEP_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/c6x/Kconfig +0 -3
@@ -27,9 +27,6 @@ config MMU config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/csky/Kconfig +0 -3
@@ -92,9 +92,6 @@ config GENERIC_HWEIGHT config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config STACKTRACE_SUPPORT def_bool y
arch/h8300/Kconfig +0 -3
@@ -27,9 +27,6 @@ config H8300 config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/hexagon/Kconfig +0 -6
@@ -65,12 +65,6 @@ config GENERIC_CSUM config GENERIC_IRQ_PROBE def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool n - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
arch/ia64/Kconfig +0 -4
@@ -83,10 +83,6 @@ config STACKTRACE_SUPPORT config GENERIC_LOCKBREAK def_bool n -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config HUGETLB_PAGE_SIZE_VARIABLE bool depends on HUGETLB_PAGE
arch/m68k/Kconfig +0 -7
@@ -32,13 +32,6 @@ config M68K config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/microblaze/Kconfig +0 -6
@@ -58,15 +58,9 @@ config CPU_LITTLE_ENDIAN endchoice -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ZONE_DMA def_bool y -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 def_bool n
arch/mips/Kconfig +0 -7
@@ -1037,13 +1037,6 @@ source "arch/mips/paravirt/Kconfig" endmenu -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_HWEIGHT bool default y
arch/nds32/Kconfig +0 -3
@@ -60,9 +60,6 @@ config GENERIC_LOCKBREAK def_bool y depends on PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool y
arch/nios2/Kconfig +0 -3
@@ -40,9 +40,6 @@ config NO_IOPORT_MAP config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool n
arch/openrisc/Kconfig +0 -6
@@ -43,12 +43,6 @@ config CPU_BIG_ENDIAN config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - def_bool n - config GENERIC_HWEIGHT def_bool y
arch/parisc/Kconfig +0 -6
@@ -75,12 +75,6 @@ config GENERIC_LOCKBREAK default y depends on SMP && PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool default n
arch/powerpc/Kconfig +0 -7
@@ -103,13 +103,6 @@ config LOCKDEP_SUPPORT bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config GENERIC_LOCKBREAK bool default y
arch/riscv/Kconfig +0 -3
@@ -69,9 +69,6 @@ config STACKTRACE_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_BUG def_bool y depends on BUG
arch/s390/Kconfig +0 -6
@@ -14,12 +14,6 @@ config LOCKDEP_SUPPORT config STACKTRACE_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config ARCH_HAS_ILOG2_U32 def_bool n
arch/sh/Kconfig +0 -6
@@ -90,12 +90,6 @@ config ARCH_DEFCONFIG default "arch/sh/configs/shx3_defconfig" if SUPERH32 default "arch/sh/configs/cayman_defconfig" if SUPERH64 -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_BUG def_bool y depends on BUG && SUPERH32
arch/sparc/Kconfig +0 -8
@@ -191,14 +191,6 @@ config NR_CPUS source "kernel/Kconfig.hz" -config RWSEM_GENERIC_SPINLOCK - bool - default y if SPARC32 - -config RWSEM_XCHGADD_ALGORITHM - bool - default y if SPARC64 - config GENERIC_HWEIGHT bool default y
arch/unicore32/Kconfig +0 -6
@@ -38,12 +38,6 @@ config STACKTRACE_SUPPORT config LOCKDEP_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool
arch/x86/Kconfig +0 -3
@@ -268,9 +268,6 @@ config ARCH_MAY_HAVE_PC_FDC def_bool y depends on ISA_DMA_API -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y
arch/x86/um/Kconfig +0 -6
@@ -32,12 +32,6 @@ config ARCH_DEFCONFIG default "arch/um/configs/i386_defconfig" if X86_32 default "arch/um/configs/x86_64_defconfig" if X86_64 -config RWSEM_XCHGADD_ALGORITHM - def_bool 64BIT - -config RWSEM_GENERIC_SPINLOCK - def_bool !RWSEM_XCHGADD_ALGORITHM - config 3_LEVEL_PGTABLES bool "Three-level pagetables" if !64BIT default 64BIT
arch/xtensa/Kconfig +0 -3
@@ -46,9 +46,6 @@ config XTENSA with reasonable minimum requirements. The Xtensa Linux project has a home page at <http://www.linux-xtensa.org/>. -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y
include/linux/rwsem-spinlock.h deleted +0 -47
47 lines not on this page · raw message
include/linux/rwsem.h +0 -5
@@ -22,10 +22,6 @@struct rw_semaphore ; -#ifdef CONFIG_RWSEM_GENERIC_SPINLOCK -#include <linux/rwsem-spinlock.h> /* use a generic implementation */ -#define __RWSEM_INIT_COUNT(name) .count = RWSEM_UNLOCKED_VALUE -#else/* All arch specific implementations share the same struct */ struct rw_semaphore {atomic_long_t count;
@@ -65,7 +61,6 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE) -#endif /* Common initializer macros and functions */
kernel/Kconfig.locks +1 -1
@@ -229,7 +229,7 @@ config MUTEX_SPIN_ON_OWNER config RWSEM_SPIN_ON_OWNER def_bool y - depends on SMP && RWSEM_XCHGADD_ALGORITHM && ARCH_SUPPORTS_ATOMIC_RMW + depends on SMP && ARCH_SUPPORTS_ATOMIC_RMW config LOCK_SPIN_ON_OWNER def_bool y
kernel/locking/Makefile +1 -3
@@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT := n -obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o rwsem-xadd.o ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)
@@ -25,8 +25,6 @@ obj-$(CONFIG_RT_MUTEXES) += rtmutex.o obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
kernel/locking/rwsem-spinlock.c deleted +0 -339
339 lines not on this page · raw message
kernel/locking/rwsem.h +0 -3
@@ -153,7 +153,6 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) }#endif -#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM /* * lock for reading */
@@ -260,5 +259,3 @@ static inline void __downgrade_write(struct rw_semaphore *sem)if (tmp <0 )rwsem_downgrade_wake (sem); } - -#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
Waiman Long
reviewerPATCH v5 3/3 locking/rwsem: Optimize down_read_trylock()
In reply to Waiman LongModify __down_read_trylock() to optimize for an unlocked rwsem and make it generate slightly better code.
Before this patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retq
After patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030 <+48>: retq
By using a rwsem microbenchmark, the down_read_trylock() rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 14,496 14,716
2 8,644 8,453
4 6,799 6,983
8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 23,676 24,488
2 7,697 9,502
4 4,945 3,440
8 2,641 1,603For the uncontended case (1 thread), the new down_read_trylock() is a little bit faster. For the contended cases, the new down_read_trylock() perform pretty well in x86-64, but performance degrades at high contention level on ARM64.
--- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/locking/rwsem.h +8 -5
@@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)static inline int __down_read_trylock (struct rw_semaphore *sem) { -long tmp; +/* +* Optimize for the case when the rwsem is not locked at all. +*/ +long tmp =RWSEM_UNLOCKED_VALUE ; -while ((tmp =atomic_long_read (&sem->count)) >=0 ) { -if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, - tmp +RWSEM_ACTIVE_READ_BIAS )) { +do { +if (atomic_long_try_cmpxchg_acquire (&sem->count, &tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) {return 1 ; } - } + }while (tmp >=0 );return 0 ; }
Signature · 1 line
2.18.1
Re: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
Quoted 3 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.
Oh, that should teach me to read all patches in the series before starting to comment on them.
So ignore my comment on #1.
Linus
Russell King - ARM Linux admin
maintainerRe: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
In reply to Waiman LongQuoted 62 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603
Quoted 4 lines
For the uncontended case (1 thread), the new
down_read_trylock()is a little bit faster. For the contended cases, the newdown_read_trylock()perform pretty well in x86-64, but performance degrades at high contention level on ARM64.
So, 70% for 4 threads, 61% for 4 threads - does this trend continue tailing off as the number of threads (and cores) increase?
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
Waiman Long
reviewerRe: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
Quoted 63 lines · Russell King - ARM Linux admin
Quoted 62 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603
Quoted 8 lines
Quoted 4 lines
For the uncontended case (1 thread), the new
down_read_trylock()is a little bit faster. For the contended cases, the newdown_read_trylock()perform pretty well in x86-64, but performance degrades at high contention level on ARM64.So, 70% for 4 threads, 61% for 4 threads - does this trend continue tailing off as the number of threads (and cores) increase?
I didn't try higher number of contending threads. I won't worry too much about contention as trylock is a one-off event. The chance of having more than one trylock happening simultaneously is very small.
Cheers,
Longman
Re: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
In reply to Waiman LongHi,
Could you share the microbenchmark you are using ?
I'd like to test the series on powerpc.
Thanks
Christophe
Le 22/03/2019 à 15:30, Waiman Long a écrit :
Quoted 101 lines
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax 0x0000000000000030 <+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch # of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch # of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603For the uncontended case (1 thread), the new
down_read_trylock()is a little bit faster. For the contended cases, the newdown_read_trylock()perform pretty well in x86-64, but performance degrades at high contention level on ARM64.Suggested-by Linus Torvalds <[email protected]>Signed-off-by Waiman Long <[email protected]>--- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)kernel/locking/rwsem.h +8
-5@@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)static inline int __down_read_trylock (struct rw_semaphore *sem) { -long tmp; +/* +* Optimize for the case when the rwsem is not locked at all. +*/ +long tmp =RWSEM_UNLOCKED_VALUE ; -while ((tmp =atomic_long_read (&sem->count)) >=0 ) { -if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, - tmp +RWSEM_ACTIVE_READ_BIAS )) { +do { +if (atomic_long_try_cmpxchg_acquire (&sem->count, &tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) {return 1 ; } - } + }while (tmp >=0 );return 0 ; }
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Optimize down_read_trylock()
In reply to Waiman Longlocking/rwsem: Optimize down_read_trylock()
Commit-ID: 0975e3df30eb5849284c01be66c2ec16d8a48114 Gitweb: https://git.kernel.org/tip/0975e3df30eb5849284c01be66c2ec16d8a48114 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:08 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 11:42:35 +0200
locking/rwsem: Optimize down_read_trylock()
Modify __down_read_trylock() to optimize for an unlocked rwsem and make it generate slightly better code.
Before this patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retq
After patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030 <+48>: retq
By using a rwsem microbenchmark, the down_read_trylock() rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 14,496 14,716
2 8,644 8,453
4 6,799 6,983
8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 23,676 24,488
2 7,697 9,502
4 4,945 3,440
8 2,641 1,603For the uncontended case (1 thread), the new down_read_trylock() is a little bit faster. For the contended cases, the new down_read_trylock() perform pretty well in x86-64, but performance degrades at high contention level on ARM64.
--- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/locking/rwsem.h +8 -5
@@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)static inline int __down_read_trylock (struct rw_semaphore *sem) { -long tmp; +/* +* Optimize for the case when the rwsem is not locked at all. +*/ +long tmp =RWSEM_UNLOCKED_VALUE ; -while ((tmp =atomic_long_read (&sem->count)) >=0 ) { -if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, - tmp +RWSEM_ACTIVE_READ_BIAS )) { +do { +if (atomic_long_try_cmpxchg_acquire (&sem->count, &tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) {return 1 ; } - } + }while (tmp >=0 );return 0 ; }
tip-bot for Waiman Long
merge bot[tip:locking/core] locking/rwsem: Optimize down_read_trylock()
In reply to Waiman Longlocking/rwsem: Optimize down_read_trylock()
Commit-ID: ddb20d1d3aed8f130519c0a29cd5392efcc067b8 Gitweb: https://git.kernel.org/tip/ddb20d1d3aed8f130519c0a29cd5392efcc067b8 Author: Waiman Long <[email protected]> AuthorDate: Fri, 22 Mar 2019 10:30:08 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 3 Apr 2019 14:50:52 +0200
locking/rwsem: Optimize down_read_trylock()
Modify __down_read_trylock() to optimize for an unlocked rwsem and make it generate slightly better code.
Before this patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retq
After patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030 <+48>: retq
By using a rwsem microbenchmark, the down_read_trylock() rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 14,496 14,716
2 8,644 8,453
4 6,799 6,983
8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 23,676 24,488
2 7,697 9,502
4 4,945 3,440
8 2,641 1,603For the uncontended case (1 thread), the new down_read_trylock() is a little bit faster. For the contended cases, the new down_read_trylock() perform pretty well in x86-64, but performance degrades at high contention level on ARM64.
--- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/locking/rwsem.h +8 -5
@@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)static inline int __down_read_trylock (struct rw_semaphore *sem) { -long tmp; +/* +* Optimize for the case when the rwsem is not locked at all. +*/ +long tmp =RWSEM_UNLOCKED_VALUE ; -while ((tmp =atomic_long_read (&sem->count)) >=0 ) { -if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, - tmp +RWSEM_ACTIVE_READ_BIAS )) { +do { +if (atomic_long_try_cmpxchg_acquire (&sem->count, &tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) {return 1 ; } - } + }while (tmp >=0 );return 0 ; }