locking/rwsem: Remove arch specific rwsem files
Subsystems (12)
Waiman Long
reviewerv3 cover locking/rwsem: Remove arch specific rwsem files
v3:
- Optimize __down_read_trylock() for the uncontended case as suggested by Linus.
v2:
- Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ.
- Update performance test data in patch 1.
This is part 0 of my rwsem patchset. It just removes the architecture specific files to make it easer to add enhancements in the upcoming rwsem patches.
Since the two ll/sc platforms that I can tested on (arm64 & ppc) are both using the generic C codes, the rwsem performance shouldn't be affected by this patch except the down_read_trylock() code which was included in patch 2 for arm64.
Waiman Long (2):
locking/rwsem: Remove arch specific rwsem files
locking/rwsem: Optimize down_read_trylock()
MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 133 ++++++++++++++++++++++ 18 files changed, 136 insertions(+), 929 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH v3 1/2 locking/rwsem: Remove arch specific rwsem files
As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort.
Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway.
By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks):
Before Patch After Patch
# of Threads wlock rlock mixed wlock rlock mixed
------------ ----- ----- ----- ----- ----- -----
1 29,201 30,143 29,458 28,615 30,172 29,201
2 6,807 13,299 1,171 7,725 15,025 1,804
4 6,504 12,755 1,520 7,127 14,286 1,345
8 6,762 13,412 764 6,826 13,652 726
16 6,693 15,408 662 6,599 15,938 626
32 6,145 15,286 496 5,549 15,487 511
64 5,812 15,495 60 5,858 15,572 60There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions.
--- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ 18 files changed, 133 insertions(+), 929 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h
MAINTAINERS +0 -1
@@ -8926,7 +8926,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/
arch/alpha/include/asm/rwsem.h deleted +0 -211
211 lines not on this page · raw message
arch/arm/include/asm/Kbuild +0 -1
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h
arch/hexagon/include/asm/Kbuild +0 -1
@@ -26,7 +26,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h
arch/ia64/include/asm/rwsem.h deleted +0 -172
172 lines not on this page · raw message
arch/powerpc/include/asm/Kbuild +0 -1
@@ -9,6 +9,5 @@ generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h
arch/s390/include/asm/Kbuild +0 -1
@@ -21,7 +21,6 @@ generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h
arch/sh/include/asm/Kbuild +0 -1
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h
arch/sparc/include/asm/Kbuild +0 -1
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h
arch/x86/include/asm/rwsem.h deleted +0 -237
237 lines not on this page · raw message
arch/x86/lib/Makefile +0 -1
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
arch/x86/lib/rwsem.S deleted +0 -156
156 lines not on this page · raw message
arch/xtensa/include/asm/Kbuild +0 -1
@@ -23,7 +23,6 @@ generic-y += mm-arch-hooks.h generic-y += param.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += topology.h generic-y += trace_clock.h
include/asm-generic/rwsem.h deleted +0 -140
140 lines not on this page · raw message
include/linux/rwsem.h +1 -3
@@ -57,15 +57,13 @@ struct rw_semaphore {extern struct rw_semaphore *rwsem_wake (struct rw_semaphore *);extern struct rw_semaphore *rwsem_downgrade_wake (struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> -/* In all implementations count != 0 means locked */ static inline int rwsem_is_locked (struct rw_semaphore *sem) {return atomic_long_read (&sem->count) !=0 ; } +#define RWSEM_UNLOCKED_VALUE 0L#define __RWSEM_INIT_COUNT (name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)#endif
kernel/locking/percpu-rwsem.c +2 -0
@@ -7,6 +7,8 @@#include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" +int __percpu_init_rwsem (struct percpu_rw_semaphore *sem,const char *name,struct lock_class_key *rwsem_key) {
kernel/locking/rwsem.h +130 -0
139 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Waiman Long
reviewerPATCH v3 2/2 locking/rwsem: Optimize down_read_trylock()
In reply to Waiman LongModify __down_read_trylock() to optimize for an unlocked rwsem and make it generate slightly better code.
Before this patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retq
After patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030 <+48>: retq
By using a rwsem microbenchmark, the down_read_trylock() rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 14,496 14,716
2 8,644 8,453
4 6,799 6,983
8 5,664 7,190On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 23,676 24,488
2 7,697 9,502
4 4,945 3,440
8 2,641 1,603For the uncontended case (1 thread), the new down_read_trylock() is a little bit faster. For the contended cases, the new down_read_trylock() perform pretty well in x86-64, but performance degrades at high contention level on ARM64.
--- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/locking/rwsem.h +8 -5
@@ -175,14 +175,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)static inline int __down_read_trylock (struct rw_semaphore *sem) { -long tmp; +/* +* Optimize for the case when the rwsem is not locked at all. +*/ +long tmp =RWSEM_UNLOCKED_VALUE ; -while ((tmp =atomic_long_read (&sem->count)) >=0 ) { -if (tmp ==atomic_long_cmpxchg_acquire (&sem->count, tmp, - tmp +RWSEM_ACTIVE_READ_BIAS )) { +do { +if (atomic_long_try_cmpxchg_acquire (&sem->count, &tmp, + tmp +RWSEM_ACTIVE_READ_BIAS )) {return 1 ; } - } + }while (tmp >=0 );return 0 ; }
Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerRe: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 54 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Quoted 7 lines
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603
Urgh, yes LL/SC is the obvious exception that can actually do better here :/
Will, what say you?
Waiman Long
reviewerRe: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 62 lines · Peter Zijlstra
Quoted 54 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Quoted 7 lines
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603
Quoted 4 lines
Urgh, yes LL/SC is the obvious exception that can actually do better here :/
Will, what say you?
The ARM64 result is what I would have expected given that the change was to optimize for the uncontended case. The x86-64 result is kind of an anomaly to me, but I haven't bothered to dig into that.
Cheers,
Longman
Re: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 4 lines · Waiman Long
The ARM64 result is what I would have expected given that the change was to optimize for the uncontended case. The x86-64 result is kind of an anomaly to me, but I haven't bothered to dig into that.
I would say that the ARM result is what I'd expect from something that scales badly to begin with.
The x86-64 result is the expected one: yes, the cmpxchg is done one extra time, but it results in fewer cache transitions (the cacheline never goes into "shared" state), and cache transitions are what matter.
The cost of re-doing the instruction should be low. The cacheline ping-pong and the cache coherency messages is what hurts.
So I actually think both are very easily explained.
The x86-64 number improves, because there is less cache coherency traffic.
The arm64 numbers scaled horribly even before, and that's because there is too much ping-pong, and it's probably because there is no "stickiness" to the cacheline to the core, and thus adding the extra loop can make the ping-pong issue even worse because now there is more of it.
The cachelines not sticking at all to a core probably is good for fairness issues (in particular, sticking *too* much can cause horrible issues), but it's absolutely horrible if it means that you lose the cacheline even before you get to complete the second cmpxchg.
Linus
Re: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 6 lines · Linus Torvalds
The arm64 numbers scaled horribly even before, and that's because there is too much ping-pong, and it's probably because there is no "stickiness" to the cacheline to the core, and thus adding the extra loop can make the ping-pong issue even worse because now there is more of it.
Actually, if it's using the ll/sc, then I don't see why arm64 should even change. It doesn't really even change the pattern: the initial load of the value is just replaced with a "ll" that gets a non-zero value, and then we re-try without even doing the "sc" part.
End result: exact same "load once, then do ll/sc to update". Just using a slightly different instruction pattern.
But maybe "ll" does something different to the cacheline than a regular "ld"?
Alternatively, the machine you used is using LSE, and the "swp" thing has some horrid behavior when it fails.
So I take it back. I'm actually surprised that arm64 performs worse. I don't think it should. But numbers walk, bullshit talks, and it clearly does make for worse numbers on arm64.
Linus
Re: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Linus Torvalds
The arm64 numbers scaled horribly even before, and that's because there is too much ping-pong, and it's probably because there is no "stickiness" to the cacheline to the core, and thus adding the extra loop can make the ping-pong issue even worse because now there is more of it.
Quoted 4 lines
Actually, if it's using the ll/sc, then I don't see why arm64 should even change. It doesn't really even change the pattern: the initial load of the value is just replaced with a "ll" that gets a non-zero value, and then we re-try without even doing the "sc" part.
So our cmpxchg() has a prefetch-with-intent-to-modify instruction before the 'll' part, which will attempt to grab the line unique the first time round. The 'll' also has acquire semantics, so there's the chance for the micro-architecture to handle that badly too.
I think that the problem with the proposed changed change is that whenever a reader tries to acquire an rwsem that is already held for read, it will always fail the first cmpxchg(), so in this situation the read path is considerably slower than before.
Quoted 7 lines
End result: exact same "load once, then do ll/sc to update". Just using a slightly different instruction pattern.
But maybe "ll" does something different to the cacheline than a regular "ld"?
Alternatively, the machine you used is using LSE, and the "swp" thing has some horrid behavior when it fails.
Depending on where the data is, the LSE instructions may execute outside of the CPU (e.g. in a cache controller) and so could add latency to a failing CAS.
Will
Re: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
In reply to Peter ZijlstraQuoted 63 lines · Peter Zijlstra
Quoted 54 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Quoted 7 lines
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603
Quoted 4 lines
Urgh, yes LL/SC is the obvious exception that can actually do better here :/
Will, what say you?
What machine were these numbers generated on and is it using LL/SC or LSE atomics for arm64? If you stick the microbenchmark somewhere, I can go play with a broader variety of h/w.
Will
Waiman Long
reviewerRe: [PATCH v3 2/2] locking/rwsem: Optimize down_read_trylock()
Quoted 66 lines · Will Deacon
Quoted 65 lines · Peter Zijlstra
Quoted 54 lines · Waiman Long
Modify
__down_read_trylock()to optimize for an unlocked rwsem and make it generate slightly better code.Before this patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retqAfter patch,
down_read_trylock:0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030<+48>: retqBy using a rwsem microbenchmark, the
down_read_trylock()rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were:Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190On a ARM64 system, the performance results were:
Quoted 7 lines
Before Patch After Patch
# of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603Urgh, yes LL/SC is the obvious exception that can actually do better here :/
Quoted 6 lines
Quoted 1 line
Will, what say you?
What machine were these numbers generated on and is it using LL/SC or LSE atomics for arm64? If you stick the microbenchmark somewhere, I can go play with a broader variety of h/w.
Will
The machine is a 2-socket Cavium ThunderX2 99xx system with 64 cores and 256 threads. I was just using threads from the first socket for this test. The microbenchmark that I used is attached. I used the command "./run-locktest -ltryrwsem -r100 -i-10 -c10 -n<threads>" to generate the locking rates.
The lscpu flags were:
fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
Cheers,
Longman