arm64: support for 8.1 LSE atomic instructions
Subsystems (5)
v2 cover arm64: support for 8.1 LSE atomic instructions
Hello,
This is version two of the patches I originally posted here:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/355980.html
Changes since v1 include:
* Fixed WFE usage in rwlock code
* Fixed rwlock trylock retry semantics
* Added documentation update for failed cmpxchg semantics
* Replaced explicit save/restore of LR with a clobber
Tested on Juno and Fastmodel.
Peter -- do you want to pick up the documentation update, or shall I
keep it with the rest of this series?
Will
--->8
Will Deacon (20):
arm64: rwlocks: don't fail trylock purely due to contention
documentation: Clarify failed cmpxchg memory ordering semantics
arm64: cpufeature.h: add missing #include of kernel.h
arm64: atomics: move ll/sc atomics into separate header file
arm64: elf: advertise 8.1 atomic instructions as new hwcap
arm64: alternatives: add cpu feature for lse atomics
arm64: introduce CONFIG_ARM64_LSE_ATOMICS as fallback to ll/sc atomics
arm64: atomics: patch in lse instructions when supported by the CPU
arm64: locks: patch in lse instructions when supported by the CPU
arm64: bitops: patch in lse instructions when supported by the CPU
arm64: xchg: patch in lse instructions when supported by the CPU
arm64: cmpxchg: patch in lse instructions when supported by the CPU
arm64: cmpxchg_dbl: patch in lse instructions when supported by the
CPU
arm64: cmpxchg: avoid "cc" clobber in ll/sc routines
arm64: cmpxchg: avoid memory barrier on comparison failure
arm64: atomics: tidy up common atomic{,64}_* macros
arm64: atomics: prefetch the destination word for write prior to stxr
arm64: atomics: implement atomic{,64}_cmpxchg using cmpxchg
arm64: atomic64_dec_if_positive: fix incorrect branch condition
arm64: kconfig: select HAVE_CMPXCHG_LOCAL
Documentation/atomic_ops.txt | 4 +- Documentation/memory-barriers.txt | 6 +- arch/arm64/Kconfig | 13 ++ arch/arm64/Makefile | 13 +- arch/arm64/include/asm/atomic.h | 262 ++++++------------------------- arch/arm64/include/asm/atomic_ll_sc.h | 237 ++++++++++++++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 285 ++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 192 +++++++++-------------- arch/arm64/include/asm/cpufeature.h | 5 +- arch/arm64/include/asm/futex.h | 2 + arch/arm64/include/asm/lse.h | 53 +++++++ arch/arm64/include/asm/spinlock.h | 147 ++++++++++++++---- arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/setup.c | 18 +++ arch/arm64/lib/Makefile | 13 ++ arch/arm64/lib/atomic_ll_sc.c | 3 + arch/arm64/lib/bitops.S | 45 +++--- 17 files changed, 911 insertions(+), 388 deletions(-) create mode 100644 arch/arm64/include/asm/atomic_ll_sc.h create mode 100644 arch/arm64/include/asm/atomic_lse.h create mode 100644 arch/arm64/include/asm/lse.h create mode 100644 arch/arm64/lib/atomic_ll_sc.c
Signature · 1 line
2.1.4
PATCH v2 1/20 arm64: rwlocks: don't fail trylock purely due to contention
STXR can fail for a number of reasons, so don't fail an rwlock trylock operation simply because the STXR reported failure.
I'm not aware of any issues with the current code, but this makes it consistent with spin_trylock and also other architectures (e.g. arch/arm).
--- arch/arm64/include/asm/spinlock.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/arm64/include/asm/spinlock.h +8 -6
@@ -140,10 +140,11 @@ static inline int arch_write_trylock(arch_rwlock_t *rw)unsigned int tmp; asmvolatile ( -" ldaxr %w0, %1\n" -" cbnz %w0, 1f\n" +"1: ldaxr %w0, %1\n" +" cbnz %w0, 2f\n" " stxr %w0, %w2, %1\n" -"1:\n" +" cbnz %w0, 1b\n" +"2:\n" :"=&r" (tmp),"+Q" (rw->lock) :"r" (0x80000000 ) :"memory" );
@@ -209,11 +210,12 @@ static inline int arch_read_trylock(arch_rwlock_t *rw)unsigned int tmp, tmp2 =1 ; asmvolatile ( -" ldaxr %w0, %2\n" +"1: ldaxr %w0, %2\n" " add %w0, %w0, #1\n" -" tbnz %w0, #31, 1f\n" +" tbnz %w0, #31, 2f\n" " stxr %w1, %w0, %2\n" -"1:\n" +" cbnz %w1, 1b\n" +"2:\n" :"=&r" (tmp),"+r" (tmp2),"+Q" (rw->lock) : :"memory" );
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 1/20 arm64: rwlocks: don't fail trylock purely due to contention
Quoted 8 lines · Will Deacon
STXR can fail for a number of reasons, so don't fail an rwlock trylock operation simply because the STXR reported failure.
I'm not aware of any issues with the current code, but this makes it consistent with
spin_trylockand also other architectures (e.g.arch/arm).Reported-by Catalin Marinas <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 2/20 documentation: Clarify failed cmpxchg memory ordering semantics
In reply to Will DeaconA failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64.
This patch updates atomic_ops.txt and memory-barriers.txt to reflect this.
--- Documentation/atomic_ops.txt | 4 +++- Documentation/memory-barriers.txt | 6 +++--- 2 files changed, 6 insertions(+), 4 deletions(-)
Documentation/atomic_ops.txt +3 -1
@@ -266,7 +266,9 @@ with the given old and new values. Like all atomic_xxx operations, atomic_cmpxchg will only satisfy its atomicity semantics as long as all other accesses of *v are performed through atomic_xxx operations. -atomic_cmpxchg must provide explicit memory barriers around the operation. +atomic_cmpxchg must provide explicit memory barriers around the operation, +although if the comparison fails then no memory ordering guarantees are +required. The semantics for atomic_cmpxchg are the same as those defined for 'cas' below.
Documentation/memory-barriers.txt +3 -3
@@ -2383,9 +2383,7 @@ about the state (old or new) implies an SMP-conditional general memory barrier explicit lock operations, described later). These include: xchg(); - cmpxchg(); atomic_xchg(); atomic_long_xchg(); - atomic_cmpxchg(); atomic_long_cmpxchg(); atomic_inc_return(); atomic_long_inc_return(); atomic_dec_return(); atomic_long_dec_return(); atomic_add_return(); atomic_long_add_return();
@@ -2398,7 +2396,9 @@ explicit lock operations, described later). These include: test_and_clear_bit(); test_and_change_bit(); - /* when succeeds (returns 1) */ + /* when succeeds */ + cmpxchg(); + atomic_cmpxchg(); atomic_long_cmpxchg(); atomic_add_unless(); atomic_long_add_unless(); These are used for such things as implementing ACQUIRE-class and RELEASE-class
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 2/20 documentation: Clarify failed cmpxchg memory ordering semantics
Quoted 9 lines · Will Deacon
A failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64.
This patch updates atomic_ops.txt and memory-barriers.txt to reflect this.
Cc Peter Zijlstra <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 2/20 documentation: Clarify failed cmpxchg memory ordering semantics
In reply to Will DeaconQuoted 7 lines · Will Deacon
A failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64.
This patch updates atomic_ops.txt and memory-barriers.txt to reflect this.
Quoted 6 lines
Cc Peter Zijlstra <[email protected]>Signed-off-by Will Deacon <[email protected]>--- Documentation/atomic_ops.txt | 4 +++- Documentation/memory-barriers.txt | 6 +++--- 2 files changed, 6 insertions(+), 4 deletions(-)
Peter: are you ok with me taking this via the arm64 tree (along with the rest of the series), or would you prefer this patch routed through -tip?
Will
Quoted 43 lines
Documentation/atomic_ops.txt +3
-1@@ -266,7 +266,9 @@ with the given old and new values. Like all atomic_xxx operations, atomic_cmpxchg will only satisfy its atomicity semantics as long as all other accesses of *v are performed through atomic_xxx operations. -atomic_cmpxchg must provide explicit memory barriers around the operation. +atomic_cmpxchg must provide explicit memory barriers around the operation, +although if the comparison fails then no memory ordering guarantees are +required. The semantics for atomic_cmpxchg are the same as those defined for 'cas' below.Documentation/memory-barriers.txt +3
-3@@ -2383,9 +2383,7 @@ about the state (old or new) implies an SMP-conditional general memory barrier explicit lock operations, described later). These include: xchg(); - cmpxchg(); atomic_xchg(); atomic_long_xchg(); - atomic_cmpxchg(); atomic_long_cmpxchg(); atomic_inc_return(); atomic_long_inc_return(); atomic_dec_return(); atomic_long_dec_return(); atomic_add_return(); atomic_long_add_return();@@ -2398,7 +2396,9 @@ explicit lock operations, described later). These include: test_and_clear_bit(); test_and_change_bit(); - /* when succeeds (returns 1) */ + /* when succeeds */ + cmpxchg(); + atomic_cmpxchg(); atomic_long_cmpxchg(); atomic_add_unless(); atomic_long_add_unless(); These are used for such things as implementing ACQUIRE-class and RELEASE-classSignature · 1 line
2.1.4
Peter Zijlstra
maintainerPATCH v2 2/20 documentation: Clarify failed cmpxchg memory ordering semantics
Quoted 15 lines · Will Deacon
Quoted 7 lines · Will Deacon
A failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64.
This patch updates atomic_ops.txt and memory-barriers.txt to reflect this.
Quoted 6 lines
Cc Peter Zijlstra <[email protected]>Signed-off-by Will Deacon <[email protected]>--- Documentation/atomic_ops.txt | 4 +++- Documentation/memory-barriers.txt | 6 +++--- 2 files changed, 6 insertions(+), 4 deletions(-)
Quoted 2 lines
Peter: are you ok with me taking this via the arm64 tree (along with the rest of the series), or would you prefer this patch routed through -tip?
So I have this one queued, and typically these changes go through tip because the RCU tree also ends up there, and Paul is the typical source of patches there.
So to minimize collisions on memory-barriers.txt i'd like to keep it if its not too much hassle.
PATCH v2 2/20 documentation: Clarify failed cmpxchg memory ordering semantics
Quoted 19 lines · Peter Zijlstra
Quoted 15 lines · Will Deacon
Quoted 7 lines · Will Deacon
A failed cmpxchg does not provide any memory ordering guarantees, a property that is used to optimise the cmpxchg implementations on Alpha, PowerPC and arm64.
This patch updates atomic_ops.txt and memory-barriers.txt to reflect this.
Quoted 6 lines
Cc Peter Zijlstra <[email protected]>Signed-off-by Will Deacon <[email protected]>--- Documentation/atomic_ops.txt | 4 +++- Documentation/memory-barriers.txt | 6 +++--- 2 files changed, 6 insertions(+), 4 deletions(-)Quoted 2 lines
Peter: are you ok with me taking this via the arm64 tree (along with the rest of the series), or would you prefer this patch routed through -tip?
Quoted 6 lines
So I have this one queued, and typically these changes go through tip because the RCU tree also ends up there, and Paul is the typical source of patches there.
So to minimize collisions on memory-barriers.txt i'd like to keep it if its not too much hassle.
No problem at all, just didn't want it to get dropped. Minimising conflicts in memory-barriers.txt is definitely a good idea ;)
Thanks!
Will
PATCH v2 3/20 arm64: cpufeature.h: add missing #include of kernel.h
In reply to Will Deaconcpufeature.h makes use of DECLARE_BITMAP, which in turn relies on the BITS_TO_LONGS and DIV_ROUND_UP macros.
This patch includes kernel.h in cpufeature.h to prevent all users having to do the same thing.
--- arch/arm64/include/asm/cpufeature.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/include/asm/cpufeature.h +2 -0
@@ -30,6 +30,8 @@ #ifndef __ASSEMBLY__ +#include <linux/kernel.h> + struct arm64_cpu_capabilities { const char *desc; u16 capability;
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 3/20 arm64: cpufeature.h: add missing #include of kernel.h
Quoted 7 lines · Will Deacon
cpufeature.hmakes use ofDECLARE_BITMAP, which in turn relies on theBITS_TO_LONGSandDIV_ROUND_UPmacros.This patch includes
kernel.hincpufeature.hto prevent all users having to do the same thing.Signed-off-by Will Deacon <[email protected]>
PATCH v2 4/20 arm64: atomics: move ll/sc atomics into separate header file
In reply to Will DeaconIn preparation for the Large System Extension (LSE) atomic instructions introduced by ARM v8.1, move the current exclusive load/store (LL/SC) atomics into their own header file.
--- arch/arm64/include/asm/atomic.h | 162 +-------------------------- arch/arm64/include/asm/atomic_ll_sc.h | 205 ++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+), 160 deletions(-) create mode 100644 arch/arm64/include/asm/atomic_ll_sc.h
arch/arm64/include/asm/atomic.h +2 -160
@@ -30,6 +30,8 @@#ifdef __KERNEL__ +#include <asm/atomic_ll_sc.h> +/* * On ARM, ordinary assignment (str instruction) doesn't clear the local * strex/ldrex monitor on some implementations. The reason we can use it for
@@ -38,79 +40,6 @@#define atomic_read (v) ACCESS_ONCE((v)->counter)#define atomic_set (v,i) (((v)->counter) = (i)) -/* -* AArch64 UP and SMP safe atomic ops. We use load exclusive and -* store exclusive to ensure that these are atomic. We may loop -* to ensure that the update happens. -*/ - -#define ATOMIC_OP (op, asm_op) \ -static inline void atomic_##op(int i, atomic_t *v) \ -{ \ - unsigned long tmp; \ - int result; \ - \ - asm volatile("// atomic_" #op "\n" \ -"1: ldxr %w0, %2\n" \ -" " #asm_op " %w0, %w0, %w3\n" \ -" stxr %w1, %w0, %2\n" \ -" cbnz %w1, 1b" \ - : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ - : "Ir" (i)); \ -} \ - -#define ATOMIC_OP_RETURN (op, asm_op) \ -static inline int atomic_##op##_return(int i, atomic_t *v) \ -{ \ - unsigned long tmp; \ - int result; \ - \ - asm volatile("// atomic_" #op "_return\n" \ -"1: ldxr %w0, %2\n" \ -" " #asm_op " %w0, %w0, %w3\n" \ -" stlxr %w1, %w0, %2\n" \ -" cbnz %w1, 1b" \ - : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ - : "Ir" (i) \ - : "memory"); \ - \ - smp_mb(); \ - return result; \ -} - -#define ATOMIC_OPS (op, asm_op) \ - ATOMIC_OP(op, asm_op) \ - ATOMIC_OP_RETURN(op, asm_op) - -ATOMIC_OPS(add, add) -ATOMIC_OPS(sub, sub) - -#undef ATOMIC_OPS -#undef ATOMIC_OP_RETURN -#undef ATOMIC_OP - -static inline int atomic_cmpxchg (atomic_t *ptr,int old,int new) -{ -unsigned long tmp; -int oldval; - -smp_mb (); - - asmvolatile ("// atomic_cmpxchg\n" -"1: ldxr %w1, %2\n" -" cmp %w1, %w3\n" -" b.ne 2f\n" -" stxr %w0, %w4, %2\n" -" cbnz %w0, 1b\n" -"2:" - :"=&r" (tmp),"=&r" (oldval),"+Q" (ptr->counter) - :"Ir" (old),"r" (new) - :"cc" ); - -smp_mb (); -return oldval; -} -#define atomic_xchg (v, new) (xchg(&((v)->counter), new))static inline int __atomic_add_unless (atomic_t *v,int a,int u)
@@ -142,95 +71,8 @@ static inline int __atomic_add_unless(atomic_t *v, int a, int u)#define atomic64_read (v) ACCESS_ONCE((v)->counter)#define atomic64_set (v,i) (((v)->counter) = (i)) -#define ATOMIC64_OP (op, asm_op) \ -static inline void atomic64_##op(long i, atomic64_t *v) \ -{ \ - long result; \ - unsigned long tmp; \ - \ - asm volatile("// atomic64_" #op "\n" \ -"1: ldxr %0, %2\n" \ -" " #asm_op " %0, %0, %3\n" \ -" stxr %w1, %0, %2\n" \ -" cbnz %w1, 1b" \ - : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ - : "Ir" (i)); \ -} \ - -#define ATOMIC64_OP_RETURN (op, asm_op) \ -static inline long atomic64_##op##_return(long i, atomic64_t *v) \ -{ \ - long result; \ - unsigned long tmp; \ - \ - asm volatile("// atomic64_" #op "_return\n" \ -"1: ldxr %0, %2\n" \ -" " #asm_op " %0, %0, %3\n" \ -" stlxr %w1, %0, %2\n" \ -" cbnz %w1, 1b" \ - : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ - : "Ir" (i) \ - : "memory"); \ - \ - smp_mb(); \ - return result; \ -} - -#define ATOMIC64_OPS (op, asm_op) \ - ATOMIC64_OP(op, asm_op) \ - ATOMIC64_OP_RETURN(op, asm_op) - -ATOMIC64_OPS(add, add) -ATOMIC64_OPS(sub, sub) - -#undef ATOMIC64_OPS -#undef ATOMIC64_OP_RETURN -#undef ATOMIC64_OP - -static inline long atomic64_cmpxchg (atomic64_t *ptr,long old,long new) -{ -long oldval; -unsigned long res; - -smp_mb (); - - asmvolatile ("// atomic64_cmpxchg\n" -"1: ldxr %1, %2\n" -" cmp %1, %3\n" -" b.ne 2f\n" -" stxr %w0, %4, %2\n" -" cbnz %w0, 1b\n" -"2:" - :"=&r" (res),"=&r" (oldval),"+Q" (ptr->counter) - :"Ir" (old),"r" (new) - :"cc" ); - -smp_mb (); -return oldval; -} -#define atomic64_xchg (v, new) (xchg(&((v)->counter), new)) -static inline long atomic64_dec_if_positive (atomic64_t *v) -{ -long result; -unsigned long tmp; - - asmvolatile ("// atomic64_dec_if_positive\n" -"1: ldxr %0, %2\n" -" subs %0, %0, #1\n" -" b.mi 2f\n" -" stlxr %w1, %0, %2\n" -" cbnz %w1, 1b\n" -" dmb ish\n" -"2:" - :"=&r" (result),"=&r" (tmp),"+Q" (v->counter) - : - :"cc" ,"memory" ); - -return result; -} -static inline int atomic64_add_unless (atomic64_t *v,long a,long u) {long c, old;
arch/arm64/include/asm/atomic_ll_sc.h new file +205 -0
205 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 4/20 arm64: atomics: move ll/sc atomics into separate header file
Quoted 6 lines · Will Deacon
In preparation for the Large System Extension (LSE) atomic instructions introduced by ARM v8.1, move the current exclusive load/store (LL/SC) atomics into their own header file.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 5/20 arm64: elf: advertise 8.1 atomic instructions as new hwcap
In reply to Will DeaconThe ARM v8.1 architecture introduces new atomic instructions to the A64 instruction set for things like cmpxchg, so advertise their availability to userspace using a hwcap.
--- arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/setup.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+)
arch/arm64/include/uapi/asm/hwcap.h +1 -0
@@ -27,5 +27,6 @@#define HWCAP_SHA1 (1 << 5)#define HWCAP_SHA2 (1 << 6)#define HWCAP_CRC32 (1 << 7) +#define HWCAP_ATOMICS (1 << 8)#endif /* _UAPI__ASM_HWCAP_H */
arch/arm64/kernel/setup.c +14 -0
@@ -280,6 +280,19 @@ static void __init setup_processor(void)if (block && !(block &0x8 )) elf_hwcap |=HWCAP_CRC32 ; + block = (features >>20 ) &0xf ; +if (!(block &0x8 )) { +switch (block) { +default : +case 2 : + elf_hwcap |=HWCAP_ATOMICS ; +case 1 : +/* RESERVED */ +case 0 : +break ; + } + } +#ifdef CONFIG_COMPAT /* * ID_ISAR5_EL1 carries similar information as above, but pertaining to
@@ -456,6 +469,7 @@ static const char *hwcap_str[] = {"sha1" ,"sha2" ,"crc32" , +"atomics" , NULL };
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 5/20 arm64: elf: advertise 8.1 atomic instructions as new hwcap
Quoted 6 lines · Will Deacon
The ARM v8.1 architecture introduces new atomic instructions to the A64 instruction set for things like cmpxchg, so advertise their availability to userspace using a hwcap.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
Quoted 21 lines
arch/arm64/include/uapi/asm/hwcap.h +1
-0@@ -27,5 +27,6 @@#define HWCAP_SHA1 (1 << 5)#define HWCAP_SHA2 (1 << 6)#define HWCAP_CRC32 (1 << 7) +#define HWCAP_ATOMICS (1 << 8)#endif /* _UAPI__ASM_HWCAP_H */ arch/arm64/kernel/setup.c +2
-0@@ -280,6 +280,19 @@ static void __init setup_processor(void)if (block && !(block &0x8 )) elf_hwcap |=HWCAP_CRC32 ; + block = (features >>20 ) &0xf ; +if (!(block &0x8 )) {
arch/arm64/kernel/setup.c static void __init setup_processor(void) · line 294
Quoted 10 lines
arch/arm64/kernel/setup.c +10
-0+switch (block) { +default : +case 2 : + elf_hwcap |=HWCAP_ATOMICS ; +case 1 : +/* RESERVED */ +case 0 : +break ; + } + }
After we merge Robin's patch for cpuid_feature_extract_field(), we can change this function (subsequent patch though).
Signature · 1 line
Catalin
PATCH v2 6/20 arm64: alternatives: add cpu feature for lse atomics
In reply to Will DeaconAdd a CPU feature for the LSE atomic instructions, so that they can be patched in at runtime when we detect that they are supported.
--- arch/arm64/include/asm/cpufeature.h | 3 ++- arch/arm64/kernel/setup.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpufeature.h +2 -1
@@ -25,8 +25,9 @@#define ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE 1#define ARM64_WORKAROUND_845719 2#define ARM64_HAS_SYSREG_GIC_CPUIF 3 +#define ARM64_CPU_FEAT_LSE_ATOMICS 4 -#define ARM64_NCAPS 4 +#define ARM64_NCAPS 5#ifndef __ASSEMBLY__
arch/arm64/kernel/setup.c +1 -0
@@ -286,6 +286,7 @@ static void __init setup_processor(void)default :case 2 : elf_hwcap |=HWCAP_ATOMICS ; +cpus_set_cap (ARM64_CPU_FEAT_LSE_ATOMICS );case 1 :/* RESERVED */ case 0 :
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 6/20 arm64: alternatives: add cpu feature for lse atomics
Quoted 5 lines · Will Deacon
Add a CPU feature for the LSE atomic instructions, so that they can be patched in at runtime when we detect that they are supported.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 7/20 arm64: introduce CONFIG_ARM64_LSE_ATOMICS as fallback to ll/sc atomics
In reply to Will DeaconIn order to patch in the new atomic instructions at runtime, we need to generate wrappers around the out-of-line exclusive load/store atomics.
This patch adds a new Kconfig option, CONFIG_ARM64_LSE_ATOMICS. which causes our atomic functions to branch to the out-of-line ll/sc implementations. To avoid the register spill overhead of the PCS, the out-of-line functions are compiled with specific compiler flags to force out-of-line save/restore of any registers that are usually caller-saved.
--- arch/arm64/Kconfig | 12 +++ arch/arm64/include/asm/atomic.h | 9 ++ arch/arm64/include/asm/atomic_ll_sc.h | 19 +++- arch/arm64/include/asm/atomic_lse.h | 160 ++++++++++++++++++++++++++++++++++ arch/arm64/lib/Makefile | 13 +++ arch/arm64/lib/atomic_ll_sc.c | 3 + 6 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/atomic_lse.h create mode 100644 arch/arm64/lib/atomic_ll_sc.c
arch/arm64/Kconfig +12 -0
@@ -664,6 +664,18 @@ config SETEND_EMULATION If unsure, say Y endif +config ARM64_LSE_ATOMICS + bool "ARMv8.1 atomic instructions" + help + As part of the Large System Extensions, ARMv8.1 introduces new + atomic instructions that are designed specifically to scale in + very large systems. + + Say Y here to make use of these instructions for the in-kernel + atomic routines. This incurs a small overhead on CPUs that do + not support these instructions and requires the kernel to be + built with binutils >= 2.25. + endmenu menu "Boot options"
arch/arm64/include/asm/atomic.h +9 -0
@@ -21,6 +21,7 @@#define __ASM_ATOMIC_H#include <linux/compiler.h> +#include <linux/stringify.h> #include <linux/types.h> #include <asm/barrier.h>
@@ -30,7 +31,15 @@#ifdef __KERNEL__ +#define __ARM64_IN_ATOMIC_IMPL + +#ifdef CONFIG_ARM64_LSE_ATOMICS +#include <asm/atomic_lse.h> +#else #include <asm/atomic_ll_sc.h> +#endif + +#undef __ARM64_IN_ATOMIC_IMPL/* * On ARM, ordinary assignment (str instruction) doesn't clear the local
arch/arm64/include/asm/atomic_ll_sc.h +17 -2
@@ -21,6 +21,10 @@#ifndef __ASM_ATOMIC_LL_SC_H#define __ASM_ATOMIC_LL_SC_H +#ifndef __ARM64_IN_ATOMIC_IMPL +#error "please don't include this file directly" +#endif +/* * AArch64 UP and SMP safe atomic ops. We use load exclusive and * store exclusive to ensure that these are atomic. We may loop
@@ -41,6 +45,10 @@#define __LL_SC_PREFIX (x) x#endif +#ifndef __LL_SC_EXPORT +#define __LL_SC_EXPORT (x) +#endif +#define ATOMIC_OP (op, asm_op) \ __LL_SC_INLINE void \ __LL_SC_PREFIX(atomic_##op(int i, atomic_t *v)) \
@@ -56,6 +64,7 @@ __LL_SC_PREFIX(atomic_##op(int i, atomic_t *v)) \ : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ : "Ir" (i)); \ } \ +__LL_SC_EXPORT(atomic_##op);#define ATOMIC_OP_RETURN (op, asm_op) \ __LL_SC_INLINE int \
@@ -75,7 +84,8 @@ __LL_SC_PREFIX(atomic_##op##_return(int i, atomic_t *v)) \ \smp_mb (); \return result; \ -} +} \ +__LL_SC_EXPORT(atomic_##op##_return);#define ATOMIC_OPS (op, asm_op) \ ATOMIC_OP(op, asm_op) \
@@ -110,6 +120,7 @@ __LL_SC_PREFIX(atomic_cmpxchg(atomic_t *ptr, int old, int new))smp_mb ();return oldval; } +__LL_SC_EXPORT (atomic_cmpxchg);#define ATOMIC64_OP (op, asm_op) \ __LL_SC_INLINE void \
@@ -126,6 +137,7 @@ __LL_SC_PREFIX(atomic64_##op(long i, atomic64_t *v)) \ : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) \ : "Ir" (i)); \ } \ +__LL_SC_EXPORT(atomic64_##op);#define ATOMIC64_OP_RETURN (op, asm_op) \ __LL_SC_INLINE long \
@@ -145,7 +157,8 @@ __LL_SC_PREFIX(atomic64_##op##_return(long i, atomic64_t *v)) \ \smp_mb (); \return result; \ -} +} \ +__LL_SC_EXPORT(atomic64_##op##_return);#define ATOMIC64_OPS (op, asm_op) \ ATOMIC64_OP(op, asm_op) \
@@ -180,6 +193,7 @@ __LL_SC_PREFIX(atomic64_cmpxchg(atomic64_t *ptr, long old, long new))smp_mb ();return oldval; } +__LL_SC_EXPORT (atomic64_cmpxchg);__LL_SC_INLINE long __LL_SC_PREFIX (atomic64_dec_if_positive (atomic64_t *v))
@@ -201,5 +215,6 @@ __LL_SC_PREFIX(atomic64_dec_if_positive(atomic64_t *v)) return result; } +__LL_SC_EXPORT (atomic64_dec_if_positive);#endif /* __ASM_ATOMIC_LL_SC_H */
arch/arm64/include/asm/atomic_lse.h new file +160 -0
160 lines not on this page · raw message
arch/arm64/lib/Makefile +13 -0
@@ -3,3 +3,16 @@ lib-y := bitops.o clear_user.o delay.o copy_from_user.o \ clear_page.o memchr.o memcpy.o memmove.o memset.o \ memcmp.o strcmp.o strncmp.o strlen.o strnlen.o \ strchr.o strrchr.o + +# Tell the compiler to treat all general purpose registers as +# callee-saved, which allows for efficient runtime patching of the bl +# instruction in the caller with an atomic instruction when supported by +# the CPU. Result and argument registers are handled correctly, based on +# the function prototype. +lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o +CFLAGS_atomic_ll_sc.o := -fcall-used-x0 -ffixed-x1 -ffixed-x2 \ + -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6 \ + -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9 \ + -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12 \ + -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15 \ + -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18
arch/arm64/lib/atomic_ll_sc.c new file +3 -0
@@ -0,0 +1,3 @@ +#include <asm/atomic.h> +#define __ARM64_IN_ATOMIC_IMPL +#include <asm/atomic_ll_sc.h>
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 7/20 arm64: introduce CONFIG_ARM64_LSE_ATOMICS as fallback to ll/sc atomics
Quoted 11 lines · Will Deacon
In order to patch in the new atomic instructions at runtime, we need to generate wrappers around the out-of-line exclusive load/store atomics.
This patch adds a new
Kconfigoption,CONFIG_ARM64_LSE_ATOMICS. which causes our atomic functions to branch to the out-of-line ll/sc implementations. To avoid the register spill overhead of the PCS, the out-of-line functions are compiled with specific compiler flags to force out-of-line save/restore of any registers that are usually caller-saved.Signed-off-by Will Deacon <[email protected]>
Slightly easier to read now ;).
PATCH v2 8/20 arm64: atomics: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of atomic_t and atomic64_t routines so that the call-site for the out-of-line ll/sc sequences is patched with an LSE atomic instruction when we detect that the CPU supports it.
If binutils is not recent enough to assemble the LSE instructions, then the ll/sc sequences are inlined as though CONFIG_ARM64_LSE_ATOMICS=n.
--- arch/arm64/Makefile | 13 +- arch/arm64/include/asm/atomic.h | 4 +- arch/arm64/include/asm/atomic_ll_sc.h | 12 -- arch/arm64/include/asm/atomic_lse.h | 270 ++++++++++++++++++++++------------ arch/arm64/include/asm/lse.h | 34 +++++ arch/arm64/kernel/setup.c | 3 + 6 files changed, 229 insertions(+), 107 deletions(-) create mode 100644 arch/arm64/include/asm/lse.h
arch/arm64/Makefile +12 -1
@@ -17,7 +17,18 @@ GZFLAGS :=-9 KBUILD_DEFCONFIG := defconfig -KBUILD_CFLAGS += -mgeneral-regs-only +# Check for binutils support for specific extensions +lseinstr := $(call as-instr,.arch_extension lse,-DCONFIG_AS_LSE=1) + +ifeq ($(CONFIG_ARM64_LSE_ATOMICS), y) + ifeq ($(lseinstr),) +$(warning LSE atomics not supported by binutils) + endif +endif + +KBUILD_CFLAGS += -mgeneral-regs-only $(lseinstr) +KBUILD_AFLAGS += $(lseinstr) + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian AS += -EB
arch/arm64/include/asm/atomic.h +2 -2
@@ -21,11 +21,11 @@#define __ASM_ATOMIC_H#include <linux/compiler.h> -#include <linux/stringify.h> #include <linux/types.h> #include <asm/barrier.h> #include <asm/cmpxchg.h> +#include <asm/lse.h> #define ATOMIC_INIT (i) { (i) }
@@ -33,7 +33,7 @@#define __ARM64_IN_ATOMIC_IMPL -#ifdef CONFIG_ARM64_LSE_ATOMICS +#if defined(CONFIG_ARM64_LSE_ATOMICS ) && defined(CONFIG_AS_LSE )#include <asm/atomic_lse.h> #else #include <asm/atomic_ll_sc.h>
arch/arm64/include/asm/atomic_ll_sc.h +0 -12
@@ -37,18 +37,6 @@* (the optimize attribute silently ignores these options). */ -#ifndef __LL_SC_INLINE -#define __LL_SC_INLINE static inline -#endif - -#ifndef __LL_SC_PREFIX -#define __LL_SC_PREFIX (x) x -#endif - -#ifndef __LL_SC_EXPORT -#define __LL_SC_EXPORT (x) -#endif -#define ATOMIC_OP (op, asm_op) \ __LL_SC_INLINE void \ __LL_SC_PREFIX(atomic_##op(int i, atomic_t *v)) \
arch/arm64/include/asm/atomic_lse.h +178 -92
312 lines not on this page · raw message
arch/arm64/include/asm/lse.h new file +34 -0
@@ -0,0 +1,34 @@ +#ifndef __ASM_LSE_H +#define __ASM_LSE_H + +#if defined(CONFIG_AS_LSE ) && defined(CONFIG_ARM64_LSE_ATOMICS ) + +#include <linux/stringify.h> + +#include <asm/alternative.h> +#include <asm/cpufeature.h> + +__asm__(".arch_extension lse" ); + +/* Move the ll/sc atomics out-of-line */ +#define __LL_SC_INLINE +#define __LL_SC_PREFIX (x) __ll_sc_##x +#define __LL_SC_EXPORT (x) EXPORT_SYMBOL(__LL_SC_PREFIX(x)) + +/* Macro for constructing calls to out-of-line ll/sc atomics */ +#define __LL_SC_CALL (op) "bl\t" __stringify(__LL_SC_PREFIX(op)) "\n" + +/* In-line patching at runtime */ +#define ARM64_LSE_ATOMIC_INSN (llsc, lse) \ + ALTERNATIVE(llsc, lse, ARM64_CPU_FEAT_LSE_ATOMICS) + +#else + +#define __LL_SC_INLINE static inline +#define __LL_SC_PREFIX (x) x +#define __LL_SC_EXPORT (x) + +#define ARM64_LSE_ATOMIC_INSN (llsc, lse) llsc + +#endif /* CONFIG_AS_LSE && CONFIG_ARM64_LSE_ATOMICS */ +#endif /* __ASM_LSE_H */
arch/arm64/kernel/setup.c +3 -0
@@ -287,6 +287,9 @@ static void __init setup_processor(void)case 2 : elf_hwcap |=HWCAP_ATOMICS ;cpus_set_cap (ARM64_CPU_FEAT_LSE_ATOMICS ); +if (IS_ENABLED (CONFIG_AS_LSE ) && +IS_ENABLED (CONFIG_ARM64_LSE_ATOMICS )) +pr_info ("LSE atomics supported\n" );case 1 :/* RESERVED */ case 0 :
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 8/20 arm64: atomics: patch in lse instructions when supported by the CPU
Quoted 8 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of
atomic_tandatomic64_troutines so that the call-site for the out-of-line ll/sc sequences is patched with an LSE atomic instruction when we detect that the CPU supports it.
Quoted 4 lines
If binutils is not recent enough to assemble the LSE instructions, then the ll/sc sequences are inlined as though
CONFIG_ARM64_LSE_ATOMICS=n.Signed-off-by Will Deacon <[email protected]>
PATCH v2 9/20 arm64: locks: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our locking functions so that LSE atomic instructions are used for spinlocks and rwlocks.
--- arch/arm64/include/asm/spinlock.h | 137 ++++++++++++++++++++++++++++++-------- 1 file changed, 108 insertions(+), 29 deletions(-)
arch/arm64/include/asm/spinlock.h +108 -29
@@ -16,6 +16,7 @@#ifndef __ASM_SPINLOCK_H#define __ASM_SPINLOCK_H +#include <asm/lse.h> #include <asm/spinlock_types.h> #include <asm/processor.h>
@@ -38,11 +39,21 @@ static inline void arch_spin_lock(arch_spinlock_t *lock) asmvolatile (/* Atomically increment the next ticket. */ + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ " prfm pstl1strm, %3\n" "1: ldaxr %w0, %3\n" " add %w1, %w0, %w5\n" " stxr %w2, %w1, %3\n" -" cbnz %w2, 1b\n" +" cbnz %w2, 1b\n", +/* LSE atomics */ +" mov %w2, %w5\n" +" ldadda %w2, %w0, %3\n" +" nop\n" +" nop\n" +" nop\n" + ) + /* Did we get the lock? */" eor %w1, %w0, %w0, ror #16\n" " cbz %w1, 3f\n"
@@ -67,15 +78,25 @@ static inline int arch_spin_trylock(arch_spinlock_t *lock)unsigned int tmp;arch_spinlock_t lockval; - asmvolatile ( -" prfm pstl1strm, %2\n" -"1: ldaxr %w0, %2\n" -" eor %w1, %w0, %w0, ror #16\n" -" cbnz %w1, 2f\n" -" add %w0, %w0, %3\n" -" stxr %w1, %w0, %2\n" -" cbnz %w1, 1b\n" -"2:" + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " prfm pstl1strm, %2\n" + "1: ldaxr %w0, %2\n" + " eor %w1, %w0, %w0, ror #16\n" + " cbnz %w1, 2f\n" + " add %w0, %w0, %3\n" + " stxr %w1, %w0, %2\n" + " cbnz %w1, 1b\n" + "2:", +/* LSE atomics */ +" ldr %w0, %2\n" +" eor %w1, %w0, %w0, ror #16\n" +" cbnz %w1, 1f\n" +" add %w1, %w0, %3\n" +" casa %w0, %w1, %2\n" +" and %w1, %w1, #0xffff\n" +" eor %w1, %w1, %w0, lsr #16\n" +"1:" ) : "=&r" (lockval), "=&r" (tmp), "+Q" (*lock) : "I" (1 << TICKET_SHIFT) : "memory");
@@ -85,10 +106,19 @@ static inline int arch_spin_trylock(arch_spinlock_t *lock)static inline void arch_spin_unlock (arch_spinlock_t *lock) { - asmvolatile ( -" stlrh %w1, %0\n" - :"=Q" (lock->owner) - :"r" (lock->owner +1 ) +unsigned long tmp; + + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " ldr %w1, %0\n" + " add %w1, %w1, #1\n" + " stlrh %w1, %0", + /* LSE atomics */ + " mov %w1, #1\n" + " nop\n" + " staddlh %w1, %0") + : "=Q" (lock->owner), "=&r" (tmp) + : : "memory"); }
@@ -123,13 +153,24 @@ static inline void arch_write_lock(arch_rwlock_t *rw) {unsigned int tmp; - asmvolatile ( + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ " sevl\n" "1: wfe\n" "2: ldaxr %w0, %1\n" " cbnz %w0, 1b\n" " stxr %w0, %w2, %1\n" " cbnz %w0, 2b\n" + " nop", + /* LSE atomics */ + "1: mov %w0, wzr\n" + "2: casa %w0, %w2, %1\n" + " cbz %w0, 3f\n" + " ldxr %w0, %1\n" + " cbz %w0, 2b\n" + " wfe\n" + " b 1b\n" + "3:") : "=&r" (tmp), "+Q" (rw->lock) : "r" (0x80000000) : "memory");
@@ -139,12 +180,18 @@ static inline int arch_write_trylock(arch_rwlock_t *rw) {unsigned int tmp; - asmvolatile ( + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldaxr %w0, %1\n" " cbnz %w0, 2f\n" " stxr %w0, %w2, %1\n" " cbnz %w0, 1b\n" -"2:\n" + "2:", + /* LSE atomics */ + " mov %w0, wzr\n" + " casa %w0, %w2, %1\n" + " nop\n" + " nop") : "=&r" (tmp), "+Q" (rw->lock) : "r" (0x80000000) : "memory");
@@ -154,9 +201,10 @@ static inline int arch_write_trylock(arch_rwlock_t *rw)static inline void arch_write_unlock (arch_rwlock_t *rw) { - asmvolatile ( -" stlr %w1, %0\n" - :"=Q" (rw->lock) :"r" (0 ) :"memory" ); + asm volatile(ARM64_LSE_ATOMIC_INSN( + " stlr wzr, %0", + " swpl wzr, wzr, %0") + : "=Q" (rw->lock) :: "memory"); }/* write_can_lock - would write_trylock() succeed? */
@@ -173,6 +221,10 @@ static inline void arch_write_unlock(arch_rwlock_t *rw)* * The memory barriers are implicit with the load-acquire and store-release * instructions. +* +* Note that in UNDEFINED cases, such as unlocking a lock twice, the LL/SC +* and LSE implementations may exhibit different behaviour (although this +* will have no effect on lockdep). */ static inline void arch_read_lock (arch_rwlock_t *rw) {
@@ -180,26 +232,43 @@ static inline void arch_read_lock(arch_rwlock_t *rw) asmvolatile (" sevl\n" + ARM64_LSE_ATOMIC_INSN( +/* LL/SC */ "1: wfe\n" "2: ldaxr %w0, %2\n" " add %w0, %w0, #1\n" " tbnz %w0, #31, 1b\n" " stxr %w1, %w0, %2\n" -" cbnz %w1, 2b\n" +" nop\n" +" cbnz %w1, 2b" , +/* LSE atomics */ +"1: wfe\n" +"2: ldxr %w0, %2\n" +" adds %w1, %w0, #1\n" +" tbnz %w1, #31, 1b\n" +" casa %w0, %w1, %2\n" +" sbc %w0, %w1, %w0\n" +" cbnz %w0, 2b" ) : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) : - :"memory" ); + : "cc", "memory"); } static inline void arch_read_unlock(arch_rwlock_t *rw) {unsigned int tmp, tmp2; - asmvolatile ( + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldxr %w0, %2\n" " sub %w0, %w0, #1\n" " stlxr %w1, %w0, %2\n" -" cbnz %w1, 1b\n" + " cbnz %w1, 1b", + /* LSE atomics */ + " movn %w0, #0\n" + " nop\n" + " nop\n" + " staddl %w0, %2") : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) : : "memory");
@@ -207,18 +276,28 @@ static inline void arch_read_unlock(arch_rwlock_t *rw)static inline int arch_read_trylock (arch_rwlock_t *rw) { -unsigned int tmp, tmp2 =1 ; +unsigned int tmp, tmp2; - asmvolatile ( + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " mov %w1, #1\n" "1: ldaxr %w0, %2\n" " add %w0, %w0, #1\n" " tbnz %w0, #31, 2f\n" " stxr %w1, %w0, %2\n" " cbnz %w1, 1b\n" -"2:\n" - :"=&r" (tmp),"+r" (tmp2),"+Q" (rw->lock) + "2:", + /* LSE atomics */ + " ldr %w0, %2\n" + " adds %w1, %w0, #1\n" + " tbnz %w1, #31, 1f\n" + " casa %w0, %w1, %2\n" + " sbc %w1, %w1, %w0\n" + " nop\n" + "1:") + : "=&r" (tmp), "=&r" (tmp2), "+Q" (rw->lock) : - :"memory" ); + : "cc", "memory");return !tmp2; }
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 9/20 arm64: locks: patch in lse instructions when supported by the CPU
Quoted 7 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our locking functions so that LSE atomic instructions are used for spinlocks and rwlocks.
Signed-off-by Will Deacon <[email protected]>
PATCH v2 10/20 arm64: bitops: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our bitops functions so that LSE atomic instructions are used instead.
--- arch/arm64/include/asm/lse.h | 23 +++++++++++++++++++++-- arch/arm64/lib/bitops.S | 43 ++++++++++++++++++++++++------------------- 2 files changed, 45 insertions(+), 21 deletions(-)
arch/arm64/include/asm/lse.h +21 -2
@@ -4,10 +4,19 @@#if defined(CONFIG_AS_LSE ) && defined(CONFIG_ARM64_LSE_ATOMICS )#include <linux/stringify.h> -#include <asm/alternative.h> #include <asm/cpufeature.h> +#ifdef __ASSEMBLER__ + +.arch_extension lse + +.macro alt_lse, llsc, lse + alternative_insn "\llsc", "\lse", ARM64_CPU_FEAT_LSE_ATOMICS +.endm + +#else /* __ASSEMBLER__ */ + __asm__(".arch_extension lse" );/* Move the ll/sc atomics out-of-line */
@@ -22,7 +31,16 @@ __asm__(".arch_extension lse");#define ARM64_LSE_ATOMIC_INSN (llsc, lse) \ ALTERNATIVE(llsc, lse, ARM64_CPU_FEAT_LSE_ATOMICS) -#else +#endif /* __ASSEMBLER__ */ +#else /* CONFIG_AS_LSE && CONFIG_ARM64_LSE_ATOMICS */ + +#ifdef __ASSEMBLER__ + +.macro alt_lse, llsc, lse + \llsc +.endm + +#else /* __ASSEMBLER__ */ #define __LL_SC_INLINE static inline#define __LL_SC_PREFIX (x) x
@@ -30,5 +48,6 @@ __asm__(".arch_extension lse");#define ARM64_LSE_ATOMIC_INSN (llsc, lse) llsc +#endif /* __ASSEMBLER__ */ #endif /* CONFIG_AS_LSE && CONFIG_ARM64_LSE_ATOMICS */ #endif /* __ASM_LSE_H */
arch/arm64/lib/bitops.S +24 -19
76 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 10/20 arm64: bitops: patch in lse instructions when supported by the CPU
Quoted 8 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our bitops functions so that LSE atomic instructions are used instead.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 11/20 arm64: xchg: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our xchg primitives so that the LSE swp instruction (yes, you read right!) is used instead.
--- arch/arm64/include/asm/cmpxchg.h | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-)
arch/arm64/include/asm/cmpxchg.h +33 -5
@@ -22,6 +22,7 @@#include <linux/mmdebug.h> #include <asm/barrier.h> +#include <asm/lse.h> static inline unsigned long __xchg (unsigned long x,volatile void *ptr,int size) {
@@ -29,37 +30,65 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizeswitch (size) { case 1: - asmvolatile ("// __xchg1\n" + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldxrb %w0, %2\n" " stlxrb %w1, %w3, %2\n" " cbnz %w1, 1b\n" + " dmb ish", + /* LSE atomics */ + " nop\n" + " swpalb %w3, %w0, %2\n" + " nop\n" + " nop") : "=&r" (ret), "=&r" (tmp), "+Q" (*(u8 *)ptr) : "r" (x) : "memory");break ; case 2: - asmvolatile ("// __xchg2\n" + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldxrh %w0, %2\n" " stlxrh %w1, %w3, %2\n" " cbnz %w1, 1b\n" + " dmb ish", + /* LSE atomics */ + " nop\n" + " swpalh %w3, %w0, %2\n" + " nop\n" + " nop") : "=&r" (ret), "=&r" (tmp), "+Q" (*(u16 *)ptr) : "r" (x) : "memory");break ; case 4: - asmvolatile ("// __xchg4\n" + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldxr %w0, %2\n" " stlxr %w1, %w3, %2\n" " cbnz %w1, 1b\n" + " dmb ish", + /* LSE atomics */ + " nop\n" + " swpal %w3, %w0, %2\n" + " nop\n" + " nop") : "=&r" (ret), "=&r" (tmp), "+Q" (*(u32 *)ptr) : "r" (x) : "memory");break ; case 8: - asmvolatile ("// __xchg8\n" + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ "1: ldxr %0, %2\n" " stlxr %w1, %3, %2\n" " cbnz %w1, 1b\n" + " dmb ish", + /* LSE atomics */ + " nop\n" + " swpal %3, %0, %2\n" + " nop\n" + " nop") : "=&r" (ret), "=&r" (tmp), "+Q" (*(u64 *)ptr) : "r" (x) : "memory");
@@ -68,7 +97,6 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizeBUILD_BUG (); } -smp_mb ();return ret; }
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 11/20 arm64: xchg: patch in lse instructions when supported by the CPU
Quoted 8 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our xchg primitives so that the LSE swp instruction (yes, you read right!) is used instead.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 12/20 arm64: cmpxchg: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our cmpxchg primitives so that the LSE cas instruction is used instead.
--- arch/arm64/include/asm/atomic.h | 3 +- arch/arm64/include/asm/atomic_ll_sc.h | 38 ++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 39 ++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 84 ++++++++--------------------------- 4 files changed, 98 insertions(+), 66 deletions(-)
arch/arm64/include/asm/atomic.h +2 -1
@@ -24,7 +24,6 @@#include <linux/types.h> #include <asm/barrier.h> -#include <asm/cmpxchg.h> #include <asm/lse.h> #define ATOMIC_INIT (i) { (i) }
@@ -41,6 +40,8 @@#undef __ARM64_IN_ATOMIC_IMPL +#include <asm/cmpxchg.h> +/* * On ARM, ordinary assignment (str instruction) doesn't clear the local * strex/ldrex monitor on some implementations. The reason we can use it for
arch/arm64/include/asm/atomic_ll_sc.h +38 -0
@@ -205,4 +205,42 @@ __LL_SC_PREFIX(atomic64_dec_if_positive(atomic64_t *v)) }__LL_SC_EXPORT (atomic64_dec_if_positive); +#define __CMPXCHG_CASE (w, sz, name, mb, cl) \ +__LL_SC_INLINE unsigned long \ +__LL_SC_PREFIX(__cmpxchg_case_##name(volatile void *ptr, \ + unsigned long old, \ + unsigned long new)) \ +{ \ + unsigned long tmp, oldval; \ + \ + asm volatile( \ + " " #mb "\n" \ + "1: ldxr" #sz "\t%" #w "[oldval], %[v]\n" \ + " eor %" #w "[tmp], %" #w "[oldval], %" #w "[old]\n" \ + " cbnz %" #w "[tmp], 2f\n" \ + " stxr" #sz "\t%w[tmp], %" #w "[new], %[v]\n" \ + " cbnz %w[tmp], 1b\n" \ + " " #mb "\n" \ + " mov %" #w "[oldval], %" #w "[old]\n" \ + "2:" \ + : [tmp] "=&r" (tmp), [oldval] "=&r" (oldval), \ + [v] "+Q" (*(unsigned long *)ptr) \ + : [old] "Lr" (old), [new] "r" (new) \ + : cl); \ + \ + return oldval; \ +} \ +__LL_SC_EXPORT(__cmpxchg_case_##name); + +__CMPXCHG_CASE(w, b, 1, , ) +__CMPXCHG_CASE(w, h, 2, , ) +__CMPXCHG_CASE(w, , 4, , ) +__CMPXCHG_CASE( , , 8, , ) +__CMPXCHG_CASE(w, b, mb_1, dmb ish, "memory") +__CMPXCHG_CASE(w, h, mb_2, dmb ish, "memory") +__CMPXCHG_CASE(w, , mb_4, dmb ish, "memory") +__CMPXCHG_CASE( , , mb_8, dmb ish, "memory") + +#undef __CMPXCHG_CASE +#endif /* __ASM_ATOMIC_LL_SC_H */
arch/arm64/include/asm/atomic_lse.h +39 -0
@@ -243,4 +243,43 @@ static inline long atomic64_dec_if_positive(atomic64_t *v) #undef __LL_SC_ATOMIC64 +#define __LL_SC_CMPXCHG(op) __LL_SC_CALL(__cmpxchg_case_##op) + +#define __CMPXCHG_CASE(w, sz, name, mb, cl...) \ +static inline unsigned long __cmpxchg_case_##name(volatile void *ptr, \ + unsigned long old, \ + unsigned long new) \ +{ \ + register unsigned long x0 asm ("x0") = (unsigned long)ptr; \ + register unsigned long x1 asm ("x1") = old; \ + register unsigned long x2 asm ("x2") = new; \ + \ + asm volatile(ARM64_LSE_ATOMIC_INSN( \ + /* LL/SC */ \ + "nop\n" \ + __LL_SC_CMPXCHG(name) \ + "nop", \ + /* LSE atomics */ \ + " mov " #w "30, %" #w "[old]\n" \ + " cas" #mb #sz "\t" #w "30, %" #w "[new], %[v]\n" \ + " mov %" #w "[ret], " #w "30") \ + : [ret] "+r" (x0), [v] "+Q" (*(unsigned long *)ptr) \ + : [old] "r" (x1), [new] "r" (x2) \ + : "x30" , ##cl); \ + \ + return x0; \ +} + +__CMPXCHG_CASE(w, b, 1, ) +__CMPXCHG_CASE(w, h, 2, ) +__CMPXCHG_CASE(w, , 4, ) +__CMPXCHG_CASE(x, , 8, ) +__CMPXCHG_CASE(w, b, mb_1, al, "memory") +__CMPXCHG_CASE(w, h, mb_2, al, "memory") +__CMPXCHG_CASE(w, , mb_4, al, "memory") +__CMPXCHG_CASE(x, , mb_8, al, "memory") + +#undef __LL_SC_CMPXCHG +#undef __CMPXCHG_CASE + #endif /* __ASM_ATOMIC_LSE_H */
arch/arm64/include/asm/cmpxchg.h +19 -65
112 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 12/20 arm64: cmpxchg: patch in lse instructions when supported by the CPU
Quoted 7 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our cmpxchg primitives so that the LSE cas instruction is used instead.
Signed-off-by Will Deacon <[email protected]>
PATCH v2 13/20 arm64: cmpxchg_dbl: patch in lse instructions when supported by the CPU
In reply to Will DeaconOn CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our cmpxchg_double primitives so that the LSE casp instruction is used instead.
--- arch/arm64/include/asm/atomic_ll_sc.h | 34 ++++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 43 ++++++++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 68 +++++++++-------------------------- 3 files changed, 94 insertions(+), 51 deletions(-)
arch/arm64/include/asm/atomic_ll_sc.h +34 -0
@@ -243,4 +243,38 @@ __CMPXCHG_CASE( , , mb_8, dmb ish, "memory")#undef __CMPXCHG_CASE +#define __CMPXCHG_DBL (name, mb, cl) \ +__LL_SC_INLINE int \ +__LL_SC_PREFIX(__cmpxchg_double##name(unsigned long old1, \ + unsigned long old2, \ + unsigned long new1, \ + unsigned long new2, \ + volatile void *ptr)) \ +{ \ + unsigned long tmp, ret; \ + \ + asm volatile("// __cmpxchg_double" #name "\n" \ + " " #mb "\n" \ + "1: ldxp %0, %1, %2\n" \ + " eor %0, %0, %3\n" \ + " eor %1, %1, %4\n" \ + " orr %1, %0, %1\n" \ + " cbnz %1, 2f\n" \ + " stxp %w0, %5, %6, %2\n" \ + " cbnz %w0, 1b\n" \ + " " #mb "\n" \ + "2:" \ + : "=&r" (tmp), "=&r" (ret), "+Q" (*(unsigned long *)ptr) \ + : "r" (old1), "r" (old2), "r" (new1), "r" (new2) \ + : cl); \ + \ + return ret; \ +} \ +__LL_SC_EXPORT(__cmpxchg_double##name); + +__CMPXCHG_DBL( , , ) +__CMPXCHG_DBL(_mb, dmb ish, "memory") + +#undef __CMPXCHG_DBL +#endif /* __ASM_ATOMIC_LL_SC_H */
arch/arm64/include/asm/atomic_lse.h +43 -0
47 lines not on this page · raw message
arch/arm64/include/asm/cmpxchg.h +17 -51
89 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 13/20 arm64: cmpxchg_dbl: patch in lse instructions when supported by the CPU
Quoted 7 lines · Will Deacon
On CPUs which support the LSE atomic instructions introduced in ARMv8.1, it makes sense to use them in preference to ll/sc sequences.
This patch introduces runtime patching of our
cmpxchg_doubleprimitives so that the LSE casp instruction is used instead.Signed-off-by Will Deacon <[email protected]>
PATCH v2 14/20 arm64: cmpxchg: avoid "cc" clobber in ll/sc routines
In reply to Will DeaconWe can perform the cmpxchg comparison using eor and cbnz which avoids the "cc" clobber for the ll/sc case and consequently for the LSE case where we may have to fall-back on the ll/sc code at runtime.
--- arch/arm64/include/asm/atomic_ll_sc.h | 14 ++++++-------- arch/arm64/include/asm/atomic_lse.h | 4 ++-- 2 files changed, 8 insertions(+), 10 deletions(-)
arch/arm64/include/asm/atomic_ll_sc.h +6 -8
@@ -96,14 +96,13 @@ __LL_SC_PREFIX(atomic_cmpxchg(atomic_t *ptr, int old, int new)) asmvolatile ("// atomic_cmpxchg\n" "1: ldxr %w1, %2\n" -" cmp %w1, %w3\n" -" b.ne 2f\n" +" eor %w0, %w1, %w3\n" +" cbnz %w0, 2f\n" " stxr %w0, %w4, %2\n" " cbnz %w0, 1b\n" "2:" :"=&r" (tmp),"=&r" (oldval),"+Q" (ptr->counter) - :"Ir" (old),"r" (new) - :"cc" ); + :"Lr" (old),"r" (new));smp_mb ();return oldval;
@@ -169,14 +168,13 @@ __LL_SC_PREFIX(atomic64_cmpxchg(atomic64_t *ptr, long old, long new)) asmvolatile ("// atomic64_cmpxchg\n" "1: ldxr %1, %2\n" -" cmp %1, %3\n" -" b.ne 2f\n" +" eor %0, %1, %3\n" +" cbnz %w0, 2f\n" " stxr %w0, %4, %2\n" " cbnz %w0, 1b\n" "2:" :"=&r" (res),"=&r" (oldval),"+Q" (ptr->counter) - :"Ir" (old),"r" (new) - :"cc" ); + :"Lr" (old),"r" (new));smp_mb ();return oldval;
arch/arm64/include/asm/atomic_lse.h +2 -2
@@ -113,7 +113,7 @@ static inline int atomic_cmpxchg(atomic_t *ptr, int old, int new) " mov %w[ret], w30") : [ret] "+r" (x0), [v] "+Q" (ptr->counter) : [old] "r" (w1), [new] "r" (w2) - : "x30", "cc", "memory"); + : "x30", "memory"); return x0; }
@@ -207,7 +207,7 @@ static inline long atomic64_cmpxchg(atomic64_t *ptr, long old, long new) " mov %[ret], x30") : [ret] "+r" (x0), [v] "+Q" (ptr->counter) : [old] "r" (x1), [new] "r" (x2) - : "x30", "cc", "memory"); + : "x30", "memory"); return x0; }
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 14/20 arm64: cmpxchg: avoid "cc" clobber in ll/sc routines
Quoted 6 lines · Will Deacon
We can perform the cmpxchg comparison using eor and cbnz which avoids the "cc" clobber for the ll/sc case and consequently for the LSE case where we may have to fall-back on the ll/sc code at runtime.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 15/20 arm64: cmpxchg: avoid memory barrier on comparison failure
In reply to Will Deaconcmpxchg doesn't require memory barrier semantics when the value comparison fails, so make the barrier conditional on success.
--- arch/arm64/include/asm/atomic_ll_sc.h | 48 ++++++++++++++++------------------- 1 file changed, 22 insertions(+), 26 deletions(-)
arch/arm64/include/asm/atomic_ll_sc.h +22 -26
@@ -92,19 +92,18 @@ __LL_SC_PREFIX(atomic_cmpxchg(atomic_t *ptr, int old, int new))unsigned long tmp;int oldval; -smp_mb (); - asmvolatile ("// atomic_cmpxchg\n" "1: ldxr %w1, %2\n" " eor %w0, %w1, %w3\n" " cbnz %w0, 2f\n" -" stxr %w0, %w4, %2\n" +" stlxr %w0, %w4, %2\n" " cbnz %w0, 1b\n" +" dmb ish\n" "2:" :"=&r" (tmp),"=&r" (oldval),"+Q" (ptr->counter) - :"Lr" (old),"r" (new)); + :"Lr" (old),"r" (new) + :"memory" ); -smp_mb ();return oldval; }__LL_SC_EXPORT (atomic_cmpxchg);
@@ -164,19 +163,18 @@ __LL_SC_PREFIX(atomic64_cmpxchg(atomic64_t *ptr, long old, long new))long oldval;unsigned long res; -smp_mb (); - asmvolatile ("// atomic64_cmpxchg\n" "1: ldxr %1, %2\n" " eor %0, %1, %3\n" " cbnz %w0, 2f\n" -" stxr %w0, %4, %2\n" +" stlxr %w0, %4, %2\n" " cbnz %w0, 1b\n" +" dmb ish\n" "2:" :"=&r" (res),"=&r" (oldval),"+Q" (ptr->counter) - :"Lr" (old),"r" (new)); + :"Lr" (old),"r" (new) + :"memory" ); -smp_mb ();return oldval; }__LL_SC_EXPORT (atomic64_cmpxchg);
@@ -203,7 +201,7 @@ __LL_SC_PREFIX(atomic64_dec_if_positive(atomic64_t *v)) }__LL_SC_EXPORT (atomic64_dec_if_positive); -#define __CMPXCHG_CASE (w, sz, name, mb, cl) \ +#define __CMPXCHG_CASE (w, sz, name, mb, rel, cl) \ __LL_SC_INLINE unsigned long \ __LL_SC_PREFIX(__cmpxchg_case_##name(volatile void *ptr, \ unsigned long old, \
@@ -212,11 +210,10 @@ __LL_SC_PREFIX(__cmpxchg_case_##name(volatile void *ptr, \unsigned long tmp, oldval; \ \ asm volatile( \ - " " #mb "\n" \ "1: ldxr" #sz "\t%" #w "[oldval], %[v]\n" \ " eor %" #w "[tmp], %" #w "[oldval], %" #w "[old]\n" \ " cbnz %" #w "[tmp], 2f\n" \ - " stxr" #sz "\t%w[tmp], %" #w "[new], %[v]\n" \ + " st" #rel "xr" #sz "\t%w[tmp], %" #w "[new], %[v]\n" \ " cbnz %w[tmp], 1b\n" \ " " #mb "\n" \ " mov %" #w "[oldval], %" #w "[old]\n" \
@@ -230,18 +227,18 @@ __LL_SC_PREFIX(__cmpxchg_case_##name(volatile void *ptr, \ } \ __LL_SC_EXPORT(__cmpxchg_case_##name); -__CMPXCHG_CASE(w, b, 1, , ) -__CMPXCHG_CASE(w, h, 2, , ) -__CMPXCHG_CASE(w, , 4, , ) -__CMPXCHG_CASE( , , 8, , ) -__CMPXCHG_CASE(w, b, mb_1, dmb ish, "memory") -__CMPXCHG_CASE(w, h, mb_2, dmb ish, "memory") -__CMPXCHG_CASE(w, , mb_4, dmb ish, "memory") -__CMPXCHG_CASE( , , mb_8, dmb ish, "memory") +__CMPXCHG_CASE(w, b, 1, , , ) +__CMPXCHG_CASE(w, h, 2, , , ) +__CMPXCHG_CASE(w, , 4, , , ) +__CMPXCHG_CASE( , , 8, , , ) +__CMPXCHG_CASE(w, b, mb_1, dmb ish, l, "memory") +__CMPXCHG_CASE(w, h, mb_2, dmb ish, l, "memory") +__CMPXCHG_CASE(w, , mb_4, dmb ish, l, "memory") +__CMPXCHG_CASE( , , mb_8, dmb ish, l, "memory")#undef __CMPXCHG_CASE -#define __CMPXCHG_DBL (name, mb, cl) \ +#define __CMPXCHG_DBL (name, mb, rel, cl) \ __LL_SC_INLINE int \ __LL_SC_PREFIX(__cmpxchg_double##name(unsigned long old1, \ unsigned long old2, \
@@ -252,13 +249,12 @@ __LL_SC_PREFIX(__cmpxchg_double##name(unsigned long old1, \unsigned long tmp, ret; \ \ asm volatile("// __cmpxchg_double" #name "\n" \ - " " #mb "\n" \ "1: ldxp %0, %1, %2\n" \ " eor %0, %0, %3\n" \ " eor %1, %1, %4\n" \ " orr %1, %0, %1\n" \ " cbnz %1, 2f\n" \ - " stxp %w0, %5, %6, %2\n" \ + " st" #rel "xp %w0, %5, %6, %2\n" \ " cbnz %w0, 1b\n" \ " " #mb "\n" \ "2:" \
@@ -270,8 +266,8 @@ __LL_SC_PREFIX(__cmpxchg_double##name(unsigned long old1, \ } \ __LL_SC_EXPORT(__cmpxchg_double##name); -__CMPXCHG_DBL( , , ) -__CMPXCHG_DBL(_mb, dmb ish, "memory") +__CMPXCHG_DBL( , , , ) +__CMPXCHG_DBL(_mb, dmb ish, l, "memory")#undef __CMPXCHG_DBL
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 15/20 arm64: cmpxchg: avoid memory barrier on comparison failure
Quoted 5 lines · Will Deacon
cmpxchg doesn't require memory barrier semantics when the value comparison fails, so make the barrier conditional on success.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 16/20 arm64: atomics: tidy up common atomic{,64}_* macros
In reply to Will DeaconThe common (i.e. identical for ll/sc and lse) atomic macros in atomic.h are needlessley different for atomic_t and atomic64_t.
This patch tidies up the definitions to make them consistent across the two atomic types and factors out common code such as the add_unless implementation based on cmpxchg.
--- arch/arm64/include/asm/atomic.h | 92 +++++++++++++++++------------------------ 1 file changed, 38 insertions(+), 54 deletions(-)
arch/arm64/include/asm/atomic.h +38 -54
@@ -26,8 +26,6 @@#include <asm/barrier.h> #include <asm/lse.h> -#define ATOMIC_INIT (i) { (i) } -#ifdef __KERNEL__#define __ARM64_IN_ATOMIC_IMPL
@@ -42,67 +40,53 @@#include <asm/cmpxchg.h> -/* -* On ARM, ordinary assignment (str instruction) doesn't clear the local -* strex/ldrex monitor on some implementations. The reason we can use it for -* atomic_set() is the clrex or dummy strex done on every exception return. -*/ -#define atomic_read (v) ACCESS_ONCE((v)->counter) -#define atomic_set (v,i) (((v)->counter) = (i)) - -#define atomic_xchg (v, new) (xchg(&((v)->counter), new)) - -static inline int __atomic_add_unless (atomic_t *v,int a,int u) -{ -int c, old; - - c =atomic_read (v); -while (c != u && (old =atomic_cmpxchg ((v), c, c + a)) != c) - c = old; -return c; -} +#define ___atomic_add_unless (v, a, u, sfx) \ +({ \ + typeof((v)->counter) c, old; \ + \ + c = atomic##sfx##_read(v); \ + while (c != (u) && \ + (old = atomic##sfx##_cmpxchg((v), c, c + (a))) != c) \ + c = old; \ + c; \ + }) -#define atomic_inc (v) atomic_add(1, v) -#define atomic_dec (v) atomic_sub(1, v) +#define ATOMIC_INIT (i) { (i) } -#define atomic_inc_and_test (v) (atomic_add_return(1, v) == 0) -#define atomic_dec_and_test (v) (atomic_sub_return(1, v) == 0) -#define atomic_inc_return (v) (atomic_add_return(1, v)) -#define atomic_dec_return (v) (atomic_sub_return(1, v)) -#define atomic_sub_and_test (i, v) (atomic_sub_return(i, v) == 0) +#define atomic_read (v) READ_ONCE((v)->counter) +#define atomic_set (v, i) (((v)->counter) = (i)) +#define atomic_xchg (v, new) xchg(&((v)->counter), (new)) -#define atomic_add_negative (i,v) (atomic_add_return(i, v) < 0) +#define atomic_inc (v) atomic_add(1, (v)) +#define atomic_dec (v) atomic_sub(1, (v)) +#define atomic_inc_return (v) atomic_add_return(1, (v)) +#define atomic_dec_return (v) atomic_sub_return(1, (v)) +#define atomic_inc_and_test (v) (atomic_inc_return(v) == 0) +#define atomic_dec_and_test (v) (atomic_dec_return(v) == 0) +#define atomic_sub_and_test (i, v) (atomic_sub_return((i), (v)) == 0) +#define atomic_add_negative (i, v) (atomic_add_return((i), (v)) < 0) +#define __atomic_add_unless (v, a, u) ___atomic_add_unless(v, a, u,)/* * 64-bit atomic operations. */ -#define ATOMIC64_INIT (i) { (i) } - -#define atomic64_read (v) ACCESS_ONCE((v)->counter) -#define atomic64_set (v,i) (((v)->counter) = (i)) - -#define atomic64_xchg (v, new) (xchg(&((v)->counter), new)) - -static inline int atomic64_add_unless (atomic64_t *v,long a,long u) -{ -long c, old; - - c =atomic64_read (v); -while (c != u && (old =atomic64_cmpxchg ((v), c, c + a)) != c) - c = old; +#define ATOMIC64_INIT ATOMIC_INIT +#define atomic64_read atomic_read +#define atomic64_set atomic_set +#define atomic64_xchg atomic_xchg + +#define atomic64_inc (v) atomic64_add(1, (v)) +#define atomic64_dec (v) atomic64_sub(1, (v)) +#define atomic64_inc_return (v) atomic64_add_return(1, (v)) +#define atomic64_dec_return (v) atomic64_sub_return(1, (v)) +#define atomic64_inc_and_test (v) (atomic64_inc_return(v) == 0) +#define atomic64_dec_and_test (v) (atomic64_dec_return(v) == 0) +#define atomic64_sub_and_test (i, v) (atomic64_sub_return((i), (v)) == 0) +#define atomic64_add_negative (i, v) (atomic64_add_return((i), (v)) < 0) +#define atomic64_add_unless (v, a, u) (___atomic_add_unless(v, a, u, 64) != u) -return c != u; -} +#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1, 0) -#define atomic64_add_negative (a, v) (atomic64_add_return((a), (v)) < 0) -#define atomic64_inc (v) atomic64_add(1LL, (v)) -#define atomic64_inc_return (v) atomic64_add_return(1LL, (v)) -#define atomic64_inc_and_test (v) (atomic64_inc_return(v) == 0) -#define atomic64_sub_and_test (a, v) (atomic64_sub_return((a), (v)) == 0) -#define atomic64_dec (v) atomic64_sub(1LL, (v)) -#define atomic64_dec_return (v) atomic64_sub_return(1LL, (v)) -#define atomic64_dec_and_test (v) (atomic64_dec_return((v)) == 0) -#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1LL, 0LL)#endif #endif
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 16/20 arm64: atomics: tidy up common atomic{,64}_* macros
Quoted 9 lines · Will Deacon
The common (i.e. identical for ll/sc and lse) atomic macros in
atomic.hare needlessley different foratomic_tandatomic64_t.This patch tidies up the definitions to make them consistent across the two atomic types and factors out common code such as the
add_unlessimplementation based on cmpxchg.Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 17/20 arm64: atomics: prefetch the destination word for write prior to stxr
In reply to Will DeaconThe cost of changing a cacheline from shared to exclusive state can be significant, especially when this is triggered by an exclusive store, since it may result in having to retry the transaction.
This patch makes use of prfm to prefetch cachelines for write prior to ldxr/stxr loops when using the ll/sc atomic routines.
--- arch/arm64/include/asm/atomic_ll_sc.h | 9 +++++++++ arch/arm64/include/asm/cmpxchg.h | 8 ++++++++ arch/arm64/include/asm/futex.h | 2 ++ arch/arm64/lib/bitops.S | 2 ++ 4 files changed, 21 insertions(+)
arch/arm64/include/asm/atomic_ll_sc.h +9 -0
@@ -45,6 +45,7 @@ __LL_SC_PREFIX(atomic_##op(int i, atomic_t *v)) \int result; \ \ asm volatile("// atomic_" #op "\n" \ +" prfm pstl1strm, %2\n" \ "1: ldxr %w0, %2\n" \ " " #asm_op " %w0, %w0, %w3\n" \ " stxr %w1, %w0, %2\n" \
@@ -62,6 +63,7 @@ __LL_SC_PREFIX(atomic_##op##_return(int i, atomic_t *v)) \int result; \ \ asm volatile("// atomic_" #op "_return\n" \ +" prfm pstl1strm, %2\n" \ "1: ldxr %w0, %2\n" \ " " #asm_op " %w0, %w0, %w3\n" \ " stlxr %w1, %w0, %2\n" \
@@ -93,6 +95,7 @@ __LL_SC_PREFIX(atomic_cmpxchg(atomic_t *ptr, int old, int new))int oldval; asmvolatile ("// atomic_cmpxchg\n" +" prfm pstl1strm, %2\n" "1: ldxr %w1, %2\n" " eor %w0, %w1, %w3\n" " cbnz %w0, 2f\n"
@@ -116,6 +119,7 @@ __LL_SC_PREFIX(atomic64_##op(long i, atomic64_t *v)) \unsigned long tmp; \ \ asm volatile("// atomic64_" #op "\n" \ +" prfm pstl1strm, %2\n" \ "1: ldxr %0, %2\n" \ " " #asm_op " %0, %0, %3\n" \ " stxr %w1, %0, %2\n" \
@@ -133,6 +137,7 @@ __LL_SC_PREFIX(atomic64_##op##_return(long i, atomic64_t *v)) \unsigned long tmp; \ \ asm volatile("// atomic64_" #op "_return\n" \ +" prfm pstl1strm, %2\n" \ "1: ldxr %0, %2\n" \ " " #asm_op " %0, %0, %3\n" \ " stlxr %w1, %0, %2\n" \
@@ -164,6 +169,7 @@ __LL_SC_PREFIX(atomic64_cmpxchg(atomic64_t *ptr, long old, long new))unsigned long res; asmvolatile ("// atomic64_cmpxchg\n" +" prfm pstl1strm, %2\n" "1: ldxr %1, %2\n" " eor %0, %1, %3\n" " cbnz %w0, 2f\n"
@@ -186,6 +192,7 @@ __LL_SC_PREFIX(atomic64_dec_if_positive(atomic64_t *v))unsigned long tmp; asmvolatile ("// atomic64_dec_if_positive\n" +" prfm pstl1strm, %2\n" "1: ldxr %0, %2\n" " subs %0, %0, #1\n" " b.mi 2f\n"
@@ -210,6 +217,7 @@ __LL_SC_PREFIX(__cmpxchg_case_##name(volatile void *ptr, \unsigned long tmp, oldval; \ \ asm volatile( \ + " prfm pstl1strm, %2\n" \ "1: ldxr" #sz "\t%" #w "[oldval], %[v]\n" \ " eor %" #w "[tmp], %" #w "[oldval], %" #w "[old]\n" \ " cbnz %" #w "[tmp], 2f\n" \
@@ -249,6 +257,7 @@ __LL_SC_PREFIX(__cmpxchg_double##name(unsigned long old1, \unsigned long tmp, ret; \ \ asm volatile("// __cmpxchg_double" #name "\n" \ + " prfm pstl1strm, %2\n" \ "1: ldxp %0, %1, %2\n" \ " eor %0, %0, %3\n" \ " eor %1, %1, %4\n" \
arch/arm64/include/asm/cmpxchg.h +8 -0
@@ -33,12 +33,14 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizecase 1 : asm volatile(ARM64_LSE_ATOMIC_INSN(/* LL/SC */ +" prfm pstl1strm, %2\n" "1: ldxrb %w0, %2\n" " stlxrb %w1, %w3, %2\n" " cbnz %w1, 1b\n" " dmb ish",/* LSE atomics */ " nop\n" +" nop\n" " swpalb %w3, %w0, %2\n" " nop\n" " nop")
@@ -49,12 +51,14 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizecase 2 : asm volatile(ARM64_LSE_ATOMIC_INSN(/* LL/SC */ +" prfm pstl1strm, %2\n" "1: ldxrh %w0, %2\n" " stlxrh %w1, %w3, %2\n" " cbnz %w1, 1b\n" " dmb ish",/* LSE atomics */ " nop\n" +" nop\n" " swpalh %w3, %w0, %2\n" " nop\n" " nop")
@@ -65,12 +69,14 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizecase 4 : asm volatile(ARM64_LSE_ATOMIC_INSN(/* LL/SC */ +" prfm pstl1strm, %2\n" "1: ldxr %w0, %2\n" " stlxr %w1, %w3, %2\n" " cbnz %w1, 1b\n" " dmb ish",/* LSE atomics */ " nop\n" +" nop\n" " swpal %w3, %w0, %2\n" " nop\n" " nop")
@@ -81,12 +87,14 @@ static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int sizecase 8 : asm volatile(ARM64_LSE_ATOMIC_INSN(/* LL/SC */ +" prfm pstl1strm, %2\n" "1: ldxr %0, %2\n" " stlxr %w1, %3, %2\n" " cbnz %w1, 1b\n" " dmb ish",/* LSE atomics */ " nop\n" +" nop\n" " swpal %3, %0, %2\n" " nop\n" " nop")
arch/arm64/include/asm/futex.h +2 -0
@@ -24,6 +24,7 @@#define __futex_atomic_op (insn, ret, oldval, uaddr, tmp, oparg) \ asm volatile( \ +" prfm pstl1strm, %2\n" \ "1: ldxr %w1, %2\n" \ insn "\n" \ "2: stlxr %w3, %w0, %2\n" \
@@ -112,6 +113,7 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,return -EFAULT ; asmvolatile ("// futex_atomic_cmpxchg_inatomic\n" +" prfm pstl1strm, %2\n" "1: ldxr %w1, %2\n" " sub %w3, %w1, %w4\n" " cbnz %w3, 3f\n"
arch/arm64/lib/bitops.S +2 -0
@@ -31,6 +31,7 @@ ENTRY( \name ) eor w0, w0, w3 // Clear low bits mov x2, #1 add x1, x1, x0, lsr #3 // Get word offset +alt_lse " prfm pstl1strm, [x1]", "nop" lsl x3, x2, x3 // Create mask alt_lse "1: ldxr x2, [x1]", "\lse x3, [x1]"
@@ -48,6 +49,7 @@ ENTRY( \name ) eor w0, w0, w3 // Clear low bits mov x2, #1 add x1, x1, x0, lsr #3 // Get word offset +alt_lse " prfm pstl1strm, [x1]", "nop" lsl x4, x2, x3 // Create mask alt_lse "1: ldxr x2, [x1]", "\lse x4, x2, [x1]"
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 17/20 arm64: atomics: prefetch the destination word for write prior to stxr
Quoted 8 lines · Will Deacon
The cost of changing a cacheline from shared to exclusive state can be significant, especially when this is triggered by an exclusive store, since it may result in having to retry the transaction.
This patch makes use of prfm to prefetch cachelines for write prior to ldxr/stxr loops when using the ll/sc atomic routines.
Signed-off-by Will Deacon <[email protected]>
PATCH v2 18/20 arm64: atomics: implement atomic{, 64}_cmpxchg using cmpxchg
In reply to Will DeaconWe don't need duplicate cmpxchg implementations, so use cmpxchg to implement atomic{,64}_cmpxchg, like we do for xchg already.
--- arch/arm64/include/asm/atomic.h | 2 ++ arch/arm64/include/asm/atomic_ll_sc.h | 46 ----------------------------------- arch/arm64/include/asm/atomic_lse.h | 43 -------------------------------- 3 files changed, 2 insertions(+), 89 deletions(-)
arch/arm64/include/asm/atomic.h +2 -0
@@ -56,6 +56,7 @@#define atomic_read (v) READ_ONCE((v)->counter)#define atomic_set (v, i) (((v)->counter) = (i))#define atomic_xchg (v, new) xchg(&((v)->counter), (new)) +#define atomic_cmpxchg (v, old, new) cmpxchg(&((v)->counter), (old), (new))#define atomic_inc (v) atomic_add(1, (v))#define atomic_dec (v) atomic_sub(1, (v))
@@ -74,6 +75,7 @@#define atomic64_read atomic_read#define atomic64_set atomic_set#define atomic64_xchg atomic_xchg +#define atomic64_cmpxchg atomic_cmpxchg#define atomic64_inc (v) atomic64_add(1, (v))#define atomic64_dec (v) atomic64_sub(1, (v))
arch/arm64/include/asm/atomic_ll_sc.h +0 -46
58 lines not on this page · raw message
arch/arm64/include/asm/atomic_lse.h +0 -43
55 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 18/20 arm64: atomics: implement atomic{, 64}_cmpxchg using cmpxchg
Quoted 6 lines · Will Deacon
We don't need duplicate cmpxchg implementations, so use cmpxchg to implement atomic{,64}_cmpxchg, like we do for xchg already.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>---
PATCH v2 19/20 arm64: atomic64_dec_if_positive: fix incorrect branch condition
In reply to Will DeaconIf we attempt to atomic64_dec_if_positive on INT_MIN, we will underflow and incorrectly decide that the original parameter was positive.
This patches fixes the broken condition code so that we handle this corner case correctly.
--- arch/arm64/include/asm/atomic_ll_sc.h | 2 +- arch/arm64/include/asm/atomic_lse.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/atomic_ll_sc.h +1 -1
@@ -149,7 +149,7 @@ __LL_SC_PREFIX(atomic64_dec_if_positive(atomic64_t *v))" prfm pstl1strm, %2\n" "1: ldxr %0, %2\n" " subs %0, %0, #1\n" -" b.mi 2f\n" +" b.lt 2f\n" " stlxr %w1, %0, %2\n" " cbnz %w1, 1b\n" " dmb ish\n"
arch/arm64/include/asm/atomic_lse.h +1 -1
@@ -185,7 +185,7 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)/* LSE atomics */ "1: ldr x30, %[v]\n" " subs %[ret], x30, #1\n" -" b.mi 2f\n" +" b.lt 2f\n" " casal x30, %[ret], %[v]\n" " sub x30, x30, #1\n" " sub x30, x30, %[ret]\n"
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 19/20 arm64: atomic64_dec_if_positive: fix incorrect branch condition
Quoted 8 lines · Will Deacon
If we attempt to
atomic64_dec_if_positiveonINT_MIN, we will underflow and incorrectly decide that the original parameter was positive.This patches fixes the broken condition code so that we handle this corner case correctly.
Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>
PATCH v2 20/20 arm64: kconfig: select HAVE_CMPXCHG_LOCAL
In reply to Will DeaconWe implement an optimised cmpxchg_local macro, so let the kernel know.
--- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
@@ -53,6 +53,7 @@ config ARM64 select HAVE_C_RECORDMCOUNT select HAVE_CC_STACKPROTECTOR select HAVE_CMPXCHG_DOUBLE + select HAVE_CMPXCHG_LOCAL select HAVE_DEBUG_BUGVERBOSE select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_API_DEBUG
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 20/20 arm64: kconfig: select HAVE_CMPXCHG_LOCAL
Quoted 4 lines · Will Deacon
We implement an optimised
cmpxchg_localmacro, so let the kernel know.Reviewed-by Steve Capper <[email protected]>Signed-off-by Will Deacon <[email protected]>