riscv: atomic: Optimize AMO instructions usage
Subsystems (3)
v2 cover riscv: atomic: Optimize AMO instructions usage
From: Guo Ren <[email protected]>
These patch series contain one cleanup and some optimizations for atomic operations.
Changes in V2:
- Fixup LR/SC memory barrier semantic problems which pointed by Rutland
- Combine patches into one patchset series
- Separate AMO optimization & LRSC optimization for convenience patch review
Guo Ren (3):
riscv: atomic: Cleanup unnecessary definition
riscv: atomic: Optimize acquire and release for AMO operations
riscv: atomic: Optimize memory barrier semantics of LRSC-pairs
arch/riscv/include/asm/atomic.h | 70 ++++++++++++++++++++++++++++++-- arch/riscv/include/asm/cmpxchg.h | 42 +++++-------------- 2 files changed, 76 insertions(+), 36 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/3 riscv: atomic: Cleanup unnecessary definition
From: Guo Ren <[email protected]>
The cmpxchg32 & cmpxchg32_local have been never used in linux, so remove them from cmpxchg.h.
--- arch/riscv/include/asm/cmpxchg.h | 12 ------------ 1 file changed, 12 deletions(-)
arch/riscv/include/asm/cmpxchg.h +0 -12
@@ -348,18 +348,6 @@#define arch_cmpxchg_local (ptr, o, n) \ (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) -#define cmpxchg32 (ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - arch_cmpxchg((ptr), (o), (n)); \ -}) - -#define cmpxchg32_local (ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - arch_cmpxchg_relaxed((ptr), (o), (n)) \ -}) -#define arch_cmpxchg64 (ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
Signature · 1 line
2.25.1
PATCH v2 2/3 riscv: atomic: Optimize acquire and release for AMO operations
In reply to [email protected]From: Guo Ren <[email protected]>
Current acquire & release implementations from atomic-arch- fallback.h are using __atomic_acquire/release_fence(), it cause another extra "fence r, rw/fence rw,w" instruction after/before AMO instruction. RISC-V AMO instructions could combine acquire and release in the instruction self, we could gain two benefits by using this feature:
- Reduce an instruction
- Prevent strong "fence r/fence ,w" which are used to protect "lr/sc".
--- arch/riscv/include/asm/atomic.h | 64 ++++++++++++++++++++++++++++++++ arch/riscv/include/asm/cmpxchg.h | 12 ++---- 2 files changed, 68 insertions(+), 8 deletions(-)
arch/riscv/include/asm/atomic.h +64 -0
@@ -99,6 +99,30 @@ c_type arch_atomic##prefix##_fetch_##op##_relaxed(c_type i, \return ret; \ } \ static __always_inline \ +c_type arch_atomic##prefix##_fetch_##op##_acquire(c_type i, \ + atomic##prefix##_t *v) \ +{ \ + register c_type ret; \ + __asm__ __volatile__ ( \ + " amo" #asm_op "." #asm_type ".aq %1, %2, %0" \ + : "+A" (v->counter), "=r" (ret) \ + : "r" (I) \ + : "memory"); \ + return ret; \ +} \ +static __always_inline \ +c_type arch_atomic##prefix##_fetch_##op##_release(c_type i, \ + atomic##prefix##_t *v) \ +{ \ + register c_type ret; \ + __asm__ __volatile__ ( \ + " amo" #asm_op "." #asm_type ".rl %1, %2, %0" \ + : "+A" (v->counter), "=r" (ret) \ + : "r" (I) \ + : "memory"); \ + return ret; \ +} \ +static __always_inline \ c_type arch_atomic##prefix##_fetch_##op(c_type i, atomic##prefix##_t *v) \ { \ register c_type ret; \
@@ -118,6 +142,18 @@ c_type arch_atomic##prefix##_##op##_return_relaxed(c_type i, \ return arch_atomic##prefix##_fetch_##op##_relaxed(i, v) c_op I; \ } \ static __always_inline \ +c_type arch_atomic##prefix##_##op##_return_acquire(c_type i, \ + atomic##prefix##_t *v) \ +{ \ + return arch_atomic##prefix##_fetch_##op##_acquire(i, v) c_op I; \ +} \ +static __always_inline \ +c_type arch_atomic##prefix##_##op##_return_release(c_type i, \ + atomic##prefix##_t *v) \ +{ \ + return arch_atomic##prefix##_fetch_##op##_release(i, v) c_op I; \ +} \ +static __always_inline \ c_type arch_atomic##prefix##_##op##_return(c_type i, atomic##prefix##_t *v) \ { \ return arch_atomic##prefix##_fetch_##op(i, v) c_op I; \
@@ -140,22 +176,38 @@ ATOMIC_OPS(sub, add, +, -i)#define arch_atomic_add_return_relaxed arch_atomic_add_return_relaxed#define arch_atomic_sub_return_relaxed arch_atomic_sub_return_relaxed +#define arch_atomic_add_return_acquire arch_atomic_add_return_acquire +#define arch_atomic_sub_return_acquire arch_atomic_sub_return_acquire +#define arch_atomic_add_return_release arch_atomic_add_return_release +#define arch_atomic_sub_return_release arch_atomic_sub_return_release#define arch_atomic_add_return arch_atomic_add_return#define arch_atomic_sub_return arch_atomic_sub_return#define arch_atomic_fetch_add_relaxed arch_atomic_fetch_add_relaxed#define arch_atomic_fetch_sub_relaxed arch_atomic_fetch_sub_relaxed +#define arch_atomic_fetch_add_acquire arch_atomic_fetch_add_acquire +#define arch_atomic_fetch_sub_acquire arch_atomic_fetch_sub_acquire +#define arch_atomic_fetch_add_release arch_atomic_fetch_add_release +#define arch_atomic_fetch_sub_release arch_atomic_fetch_sub_release#define arch_atomic_fetch_add arch_atomic_fetch_add#define arch_atomic_fetch_sub arch_atomic_fetch_sub#ifndef CONFIG_GENERIC_ATOMIC64 #define arch_atomic64_add_return_relaxed arch_atomic64_add_return_relaxed#define arch_atomic64_sub_return_relaxed arch_atomic64_sub_return_relaxed +#define arch_atomic64_add_return_acquire arch_atomic64_add_return_acquire +#define arch_atomic64_sub_return_acquire arch_atomic64_sub_return_acquire +#define arch_atomic64_add_return_release arch_atomic64_add_return_release +#define arch_atomic64_sub_return_release arch_atomic64_sub_return_release#define arch_atomic64_add_return arch_atomic64_add_return#define arch_atomic64_sub_return arch_atomic64_sub_return#define arch_atomic64_fetch_add_relaxed arch_atomic64_fetch_add_relaxed#define arch_atomic64_fetch_sub_relaxed arch_atomic64_fetch_sub_relaxed +#define arch_atomic64_fetch_add_acquire arch_atomic64_fetch_add_acquire +#define arch_atomic64_fetch_sub_acquire arch_atomic64_fetch_sub_acquire +#define arch_atomic64_fetch_add_release arch_atomic64_fetch_add_release +#define arch_atomic64_fetch_sub_release arch_atomic64_fetch_sub_release#define arch_atomic64_fetch_add arch_atomic64_fetch_add#define arch_atomic64_fetch_sub arch_atomic64_fetch_sub#endif
@@ -178,6 +230,12 @@ ATOMIC_OPS(xor, xor, i)#define arch_atomic_fetch_and_relaxed arch_atomic_fetch_and_relaxed#define arch_atomic_fetch_or_relaxed arch_atomic_fetch_or_relaxed#define arch_atomic_fetch_xor_relaxed arch_atomic_fetch_xor_relaxed +#define arch_atomic_fetch_and_acquire arch_atomic_fetch_and_acquire +#define arch_atomic_fetch_or_acquire arch_atomic_fetch_or_acquire +#define arch_atomic_fetch_xor_acquire arch_atomic_fetch_xor_acquire +#define arch_atomic_fetch_and_release arch_atomic_fetch_and_release +#define arch_atomic_fetch_or_release arch_atomic_fetch_or_release +#define arch_atomic_fetch_xor_release arch_atomic_fetch_xor_release#define arch_atomic_fetch_and arch_atomic_fetch_and#define arch_atomic_fetch_or arch_atomic_fetch_or#define arch_atomic_fetch_xor arch_atomic_fetch_xor
@@ -186,6 +244,12 @@ ATOMIC_OPS(xor, xor, i)#define arch_atomic64_fetch_and_relaxed arch_atomic64_fetch_and_relaxed#define arch_atomic64_fetch_or_relaxed arch_atomic64_fetch_or_relaxed#define arch_atomic64_fetch_xor_relaxed arch_atomic64_fetch_xor_relaxed +#define arch_atomic64_fetch_and_acquire arch_atomic64_fetch_and_acquire +#define arch_atomic64_fetch_or_acquire arch_atomic64_fetch_or_acquire +#define arch_atomic64_fetch_xor_acquire arch_atomic64_fetch_xor_acquire +#define arch_atomic64_fetch_and_release arch_atomic64_fetch_and_release +#define arch_atomic64_fetch_or_release arch_atomic64_fetch_or_release +#define arch_atomic64_fetch_xor_release arch_atomic64_fetch_xor_release#define arch_atomic64_fetch_and arch_atomic64_fetch_and#define arch_atomic64_fetch_or arch_atomic64_fetch_or#define arch_atomic64_fetch_xor arch_atomic64_fetch_xor
arch/riscv/include/asm/cmpxchg.h +4 -8
@@ -52,16 +52,14 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -" amoswap.w %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.w.aq %0, %2, %1\n" \ :"=r" (__ret),"+A" (*__ptr) \ :"r" (__new) \ :"memory" ); \break ; \case 8 : \ __asm__ __volatile__ ( \ -" amoswap.d %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.d.aq %0, %2, %1\n" \ :"=r" (__ret),"+A" (*__ptr) \ :"r" (__new) \ :"memory" ); \
@@ -87,16 +85,14 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ -" amoswap.w %0, %2, %1\n" \ +" amoswap.w.rl %0, %2, %1\n" \ :"=r" (__ret),"+A" (*__ptr) \ :"r" (__new) \ :"memory" ); \break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ -" amoswap.d %0, %2, %1\n" \ +" amoswap.d.rl %0, %2, %1\n" \ :"=r" (__ret),"+A" (*__ptr) \ :"r" (__new) \ :"memory" ); \
Signature · 7 lines
2.25.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v2 3/3 riscv: atomic: Optimize memory barrier semantics of LRSC-pairs
In reply to [email protected]From: Guo Ren <[email protected]>
The current implementation is the same with 8e86f0b409a4 ("arm64:
atomics: fix use of acquire + release for full barrier semantics"). RISC-V could combine acquire and release into the AMO instructions and it could reduce the cost of instruction in performance. Here are the reasons for optimization:
- Reduce one extra fence instruction
- The "LR/SC" instruction with "acquire and release" operation is less cost than ACQUIRE_BARRIER/RELEASE_BARRIER which used precedes-loads/subsequent-stores prohibit to protect only LR/SC self-instruction.
- Putting acquire/release barrier into the loop shouldn't cost extra performance problems from the micro-arch design view. Because LR and SC are sequential in the loop by RVWMO rules.
--- arch/riscv/include/asm/atomic.h | 6 ++---- arch/riscv/include/asm/cmpxchg.h | 18 ++++++------------ 2 files changed, 8 insertions(+), 16 deletions(-)
arch/riscv/include/asm/atomic.h +2 -4
@@ -382,9 +382,8 @@ static __always_inline int arch_atomic_sub_if_positive(atomic_t *v, int offset)"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n" -" sc.w.rl %[rc], %[rc], %[c]\n" +" sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" -" fence rw, rw\n" "1:\n" : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) : [o]"r" (offset)
@@ -404,9 +403,8 @@ static __always_inline s64 arch_atomic64_sub_if_positive(atomic64_t *v, s64 offs"0: lr.d %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n" -" sc.d.rl %[rc], %[rc], %[c]\n" +" sc.d.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" -" fence rw, rw\n" "1:\n" : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) : [o]"r" (offset)
arch/riscv/include/asm/cmpxchg.h +6 -12
@@ -215,9 +215,8 @@ __asm__ __volatile__ ( \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ -" sc.w %1, %z4, %2\n" \ +" sc.w.aq %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \"1:\n" \ :"=&r" (__ret),"=&r" (__rc),"+A" (*__ptr) \ : "rJ" ((long)__old), "rJ" (__new) \
@@ -227,9 +226,8 @@ __asm__ __volatile__ ( \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ -" sc.d %1, %z4, %2\n" \ +" sc.d.aq %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \"1:\n" \ :"=&r" (__ret),"=&r" (__rc),"+A" (*__ptr) \ : "rJ" (__old), "rJ" (__new) \
@@ -259,8 +257,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ -"0: lr.w %0, %2\n" \ +"0: lr.w.rl %0, %2\n" \" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \ " bnez %1, 0b\n" \
@@ -271,8 +268,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ -"0: lr.d %0, %2\n" \ +"0: lr.d.rl %0, %2\n" \" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \ " bnez %1, 0b\n" \
@@ -307,9 +303,8 @@ __asm__ __volatile__ ( \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ -" sc.w.rl %1, %z4, %2\n" \ +" sc.w.aqrl %1, %z4, %2\n" \" bnez %1, 0b\n" \ -" fence rw, rw\n" \"1:\n" \ :"=&r" (__ret),"=&r" (__rc),"+A" (*__ptr) \ : "rJ" ((long)__old), "rJ" (__new) \
@@ -319,9 +314,8 @@ __asm__ __volatile__ ( \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ -" sc.d.rl %1, %z4, %2\n" \ +" sc.d.aqrl %1, %z4, %2\n" \" bnez %1, 0b\n" \ -" fence rw, rw\n" \"1:\n" \ :"=&r" (__ret),"=&r" (__rc),"+A" (*__ptr) \ : "rJ" (__old), "rJ" (__new) \
Signature · 7 lines
2.25.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
[Cc Andrea]
Quoted 5 lines · On Tue, Apr 12, 2022 at 11:49:54AM +0800, [email protected] wrote:
From: Guo Ren <[email protected]>
These patch series contain one cleanup and some optimizations for atomic operations.
Seems to me that you are basically reverting 5ce6c1f3535f ("riscv/atomic: Strengthen implementations with fences"). That commit fixed an memory ordering issue, could you explain why the issue no longer needs a fix?
Regards,
Boqun
Quoted 19 lines
Changes in V2:
- Fixup LR/SC memory barrier semantic problems which pointed by Rutland
- Combine patches into one patchset series
- Separate AMO optimization & LRSC optimization for convenience patch reviewGuo Ren (3):
riscv: atomic: Cleanup unnecessary definition
riscv: atomic: Optimize acquire and release for AMO operations
riscv: atomic: Optimize memory barrier semantics of LRSC-pairsarch/riscv/include/asm/atomic.h | 70 ++++++++++++++++++++++++++++++-- arch/riscv/include/asm/cmpxchg.h | 42 +++++-------------- 2 files changed, 76 insertions(+), 36 deletions(-)Signature · 1 line
2.25.1
Hi Boqun,
Quoted 6 lines · Boqun Feng
[Cc Andrea]
Quoted 2 lines · On Tue, Apr 12, 2022 at 11:49:54AM +0800, [email protected] wrote:
From: Guo Ren <[email protected]>
Quoted 8 lines
Quoted 3 lines
These patch series contain one cleanup and some optimizations for atomic operations.
Seems to me that you are basically reverting
5ce6c1f3535f("riscv/atomic: Strengthen implementations with fences"). That commit fixed an memory ordering issue, could you explain why the issue no longer needs a fix?
I'm not reverting the prior patch, just optimizing it.
In RISC-V “A” Standard Extension for Atomic Instructions spec, it said:
If only the aq bit is set, the atomic memory operation is treated as an acquire access, i.e., no following memory operations on this RISC-V hart can be observed to take place before the acquire memory operation.
- " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ + " amoswap.w.aq %0, %2, %1\n" \
So RISCV_ACQUIRE_BARRIER is "fence r, rw" and "fence r" is over constraints to protect amoswap.w. Here using amoswap.w.aq is more proper.
If only the rl bit is set, the atomic memory operation is treated as a release access, i.e., the release memory operation cannot be observed to take place before any earlier memory operations on this RISC-V hart.
- RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ + " amoswap.w.rl %0, %2, %1\n" \
So RISCV_RELEASE_BARRIER is "fence rw, w" and "fence ,w" is over constraints to protect amoswap.w. Here using amoswap.w.rl is more proper.
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n"
" sub %[rc], %[p], %[o]\n"
" bltz %[rc], 1f\n".
- " sc.w.rl %[rc], %[rc], %[c]\n"
+ " sc.w.aqrl %[rc], %[rc], %[c]\n"
" bnez %[rc], 0b\n"
- " fence rw, rw\n"
"1:\n"So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 23 lines
Regards,
BoqunQuoted 19 lines
Changes in V2:
- Fixup LR/SC memory barrier semantic problems which pointed by Rutland
- Combine patches into one patchset series
- Separate AMO optimization & LRSC optimization for convenience patch reviewGuo Ren (3):
riscv: atomic: Cleanup unnecessary definition
riscv: atomic: Optimize acquire and release for AMO operations
riscv: atomic: Optimize memory barrier semantics of LRSC-pairsarch/riscv/include/asm/atomic.h | 70 ++++++++++++++++++++++++++++++-- arch/riscv/include/asm/cmpxchg.h | 42 +++++-------------- 2 files changed, 76 insertions(+), 36 deletions(-)--
2.25.1
--
Best Regards
Guo Ren
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]
Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"
Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier
{}
P0(int *x, int *y, atomic_t *u)
{
int r0;
int r1;
WRITE_ONCE(*x, 1);
r0 = atomic_cmpxchg(u, 0, 1);
r1 = READ_ONCE(*y);
}
P1(int *x, int *y, atomic_t *v)
{
int r0;
int r1;
WRITE_ONCE(*y, 1);
r0 = atomic_cmpxchg(v, 0, 1);
r1 = READ_ONCE(*x);
}
exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)Regards,
Boqun
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should be A->B->C->D when sc.w.aq is executed. For the amoswap
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w.rl %1, %z4, %2\n" \
" bnez %1, 0b\n" \
" fence rw, rw\n" \
"1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;
Quoted 8 lines
int r1;
WRITE_ONCE(*y, 1);
r0 = atomic_cmpxchg(v, 0, 1);
r1 = READ_ONCE(*x);
}
exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
- "0: lr.w %0, %2\n" \
+ "0: lr.w.rl %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w.rl %1, %z4, %2\n" \
" bnez %1, 0b\n" \
- " fence rw, rw\n" \
"1:\n" \+ " fence w, rw\n" \
To give an unconditional RSsc for atomic_cmpxchg.
Quoted 3 lines
Regards,
Boqun
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 22 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 15 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 5 lines
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Thanks for the confirmation, btw, where can I find the RVWMO spec?
Quoted 65 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThe purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \
Quoted 2 lines
To give an unconditional RSsc for
atomic_cmpxchg.
Note that Linux kernel doesn't require cmpxchg() to provide any order if cmpxchg() fails to update the memory location. So you won't need to strengthen the atomic_cmpxchg().
Regards,
Boqun
Quoted 11 lines
Quoted 3 lines
Regards,
BoqunSignature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 24 lines · Boqun Feng
Quoted 22 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 15 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Thanks for the confirmation, btw, where can I find the RVWMO spec?
RVWMO section:
https://five-embeddev.com/riscv-isa-manual/latest/rvwmo.html#ch:memorymodel
ATOMIC instructions:
https://five-embeddev.com/riscv-isa-manual/latest/a.html#atomics
Quoted 66 lines
Quoted 66 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThe purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \
Quoted 6 lines
Quoted 2 lines
To give an unconditional RSsc for
atomic_cmpxchg.Note that Linux kernel doesn't require
cmpxchg()to provide any order ifcmpxchg()fails to update the memory location. So you won't need to strengthen theatomic_cmpxchg().
Thx for the clarification.
Quoted 15 lines
Regards,
BoqunQuoted 11 lines
Quoted 3 lines
Regards,
Boqun--
Best Regards
Guo Ren
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswap
These opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun
Thx Dan,
Quoted 40 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswap
Quoted 5 lines
These opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
"0: lr.w %[p], %[c]\n"
" sub %[rc], %[p], %[o]\n"
" bltz %[rc], 1f\n".
- " sc.w.rl %[rc], %[rc], %[c]\n"
+ " sc.w.aqrl %[rc], %[rc], %[c]\n"
" bnez %[rc], 0b\n"
- " fence rw, rw\n"Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 43 lines · Guo Ren
Thx Dan,
Quoted 40 lines · Dan Lustig
Quoted 37 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswap
Quoted 8 lines
Quoted 5 lines
These opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no smp_store_acquire() or smp_load_release(), nor are there equivalents in C/C++ atomics.
Quoted 9 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"
Yes, using .aqrl is valid.
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun
Hi Dan,
Quoted 50 lines · Dan Lustig
Quoted 48 lines · Guo Ren
Thx Dan,
Quoted 45 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThese opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Quoted 8 lines
Quoted 3 lines
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no
smp_store_acquire()orsmp_load_release(), nor are there equivalents in C/C++ atomics.
First, thx for pointing out that my patch violates the rules defined in the ISA manual. I've abandoned these parts in v3.
It's easy to let hw support lr.rl & sc.aq (eg: our hardware supports them). I agree there are no equivalents in C/C++ atomics. But they are useful for LR/SC pairs to implement atomic_acqurie/release semantics. Compare below:
A): fence rw, r; lr
B): lr.rl
The A has another "fence ,r" effect in semantics, it's over commit from a software design view.
ps: Current definition has problems:
#define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"
#define RISCV_RELEASE_BARRIER "\tfence rw, w\n"
#define __cmpxchg_release(ptr, old, new, size) \
...
__asm__ __volatile__ ( \
RISCV_RELEASE_BARRIER \
"0: lr.w %0, %2\n" \That means "fence rw, w" can't prevent lr.w beyond the fence, we need a "fence.rw. r" here. Here is the Fixup patch which I'm preparing:
From 14c93aca0c3b10cf134791cf491b459972a36ec4 Mon Sep 17 00:00:00 2001
From: Guo Ren <[email protected]>
Date: Thu, 21 Apr 2022 16:44:48 +0800
Subject: [PATCH] riscv: atomic: Fixup wrong __atomic_acquire/release_fence
implementation
Current RISCV_ACQUIRE/RELEASE_BARRIER is for spin_lock not atomic.
__cmpxchg_release(ptr, old, new, size)
...
__asm__ __volatile__ (
RISCV_RELEASE_BARRIER
"0: lr.w %0, %2\n"The "fence rw, w -> lr.w" is invalid and lr would beyond fence, so we need "fence rw, r -> lr.w" here. Atomic acquire is the same.
--- arch/riscv/include/asm/atomic.h | 4 ++-- arch/riscv/include/asm/cmpxchg.h | 8 ++++---- arch/riscv/include/asm/fence.h | 4 ++++ 3 files changed, 10 insertions(+), 6 deletions(-)
arch/riscv/include/asm/atomic.h +2 -2
@@ -20,10 +20,10 @@#include <asm/barrier.h> #define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + __asm__ __volatile__(RISCV_ATOMIC_ACQUIRE_BARRIER "":::"memory")#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); + __asm__ __volatile__(RISCV_ATOMIC_RELEASE_BARRIER"" ::: "memory"); static __always_inline int arch_atomic_read(const atomic_t *v) {
arch/riscv/include/asm/cmpxchg.h +4 -4
@@ -217,7 +217,7 @@" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" ((long)__old), " rJ" (__new) \
@@ -229,7 +229,7 @@" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" (__old), " rJ" (__new) \
@@ -259,7 +259,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.w %1, %z4, %2\n" \
@@ -271,7 +271,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.d %1, %z4, %2\n" \
arch/riscv/include/asm/fence.h +4 -0
@@ -4,9 +4,13 @@#ifdef CONFIG_SMP #define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"#define RISCV_RELEASE_BARRIER "\tfence rw, w\n" +#define RISCV_ATOMIC_ACQUIRE_BARRIER "\tfence w , rw\n" +#define RISCV_ATOMIC_RELEASE_BARRIER "\tfence rw, r\n"#else #define RISCV_ACQUIRE_BARRIER #define RISCV_RELEASE_BARRIER +#define RISCV_ATOMIC_ACQUIRE_BARRIER +#define RISCV_ATOMIC_RELEASE_BARRIER #endif #endif /* _ASM_RISCV_FENCE_H */
Quoted 11 lines
Quoted 10 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"
Quoted 1 line
Yes, using .aqrl is valid.
Thx and I think the below is also valid, right?
- RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ + " amoswap.w.rl %0, %2, %1\n" \
- " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ + " amoswap.d.aq %0, %2, %1\n" \
Quoted 71 lines
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun
--
Best Regards
Guo Ren
Quoted 77 lines · Guo Ren
Hi Dan,
Quoted 50 lines · Dan Lustig
Quoted 48 lines · Guo Ren
Thx Dan,
Quoted 45 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThese opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Quoted 8 lines
Quoted 3 lines
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no
smp_store_acquire()orsmp_load_release(), nor are there equivalents in C/C++ atomics.First, thx for pointing out that my patch violates the rules defined in the ISA manual. I've abandoned these parts in v3.
It's easy to let hw support lr.rl & sc.aq (eg: our hardware supports them). I agree there are no equivalents in C/C++ atomics. But they are useful for LR/SC pairs to implement atomic_acqurie/release semantics. Compare below:
A): fence rw, r; lr
B): lr.rl
The A has another "fence ,r" effect in semantics, it's over commit from a software design view.ps: Current definition has problems:
#defineRISCV_ACQUIRE_BARRIER"\tfence r , rw\n"
#defineRISCV_RELEASE_BARRIER"\tfence rw, w\n"
Quoted 9 lines
#define
__cmpxchg_release(ptr, old, new, size)\
...__asm__ __volatile__ ( \ RISCV_RELEASE_BARRIER \ "0: lr.w %0, %2\n" \That means "fence rw, w" can't prevent lr.w beyond the fence, we need a "fence.rw. r" here. Here is the Fixup patch which I'm preparing:
That's not true. Note that RELEASE semantics only applies to the write/store part of a read-modify-write atomic, similarly, ACQUIRE only applies to the read/load part. For example, the following litmus test can observe the exists clause being true.
{}
P0(int *x, int *y)
{
int r0;
int r1;
r0 = cmpxchg_acquire(x, 0, 1);
r1 = READ_ONCE(*y);
}
P1(int *x, int *y)
{
int r0;
WRITE_ONCE(*y, 1);
smp_mb();
r0 = READ_ONCE(*x);
}
exists (0:r0=0 /\ 0:r1=0 /\ 1:r0=0)Regards,
Boqun
Quoted 209 lines
From
14c93aca0c3b10cf134791cf491b459972a36ec4Mon Sep 17 00:00:00 2001
From: Guo Ren <[email protected]>
Date: Thu, 21 Apr 2022 16:44:48 +0800
Subject: [PATCH] riscv: atomic: Fixup wrong __atomic_acquire/release_fence
implementationCurrent RISCV_ACQUIRE/RELEASE_BARRIER is for
spin_locknot atomic.
__cmpxchg_release(ptr, old, new, size)
...__asm__ __volatile__ ( RISCV_RELEASE_BARRIER "0: lr.w %0, %2\n"The "fence rw, w -> lr.w" is invalid and lr would beyond fence, so we need "fence rw, r -> lr.w" here. Atomic acquire is the same.
Fixes 0123f4d76ca6 ("riscv/spinlock: Strengthen implementations with fences")Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Palmer Dabbelt <[email protected]>Cc Mark Rutland <[email protected]>Cc Andrea Parri <[email protected]>Cc Dan Lustig <[email protected]>--- arch/riscv/include/asm/atomic.h | 4 ++-- arch/riscv/include/asm/cmpxchg.h | 8 ++++---- arch/riscv/include/asm/fence.h | 4 ++++ 3 files changed, 10 insertions(+), 6 deletions(-)arch/riscv/include/asm/atomic.h +2
-2@@ -20,10 +20,10 @@#include <asm/barrier.h> #define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + __asm__ __volatile__(RISCV_ATOMIC_ACQUIRE_BARRIER "":::"memory")#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); + __asm__ __volatile__(RISCV_ATOMIC_RELEASE_BARRIER"" ::: "memory"); static __always_inline int arch_atomic_read(const atomic_t *v) {arch/riscv/include/asm/cmpxchg.h +4
-4@@ -217,7 +217,7 @@" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" ((long)__old), " rJ" (__new) \ @@ -229,7 +229,7 @@" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" (__old), " rJ" (__new) \ @@ -259,7 +259,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.w %1, %z4, %2\n" \@@ -271,7 +271,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.d %1, %z4, %2\n" \arch/riscv/include/asm/fence.h +4
-0@@ -4,9 +4,13 @@#ifdef CONFIG_SMP #define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"#define RISCV_RELEASE_BARRIER "\tfence rw, w\n" +#define RISCV_ATOMIC_ACQUIRE_BARRIER "\tfence w , rw\n" +#define RISCV_ATOMIC_RELEASE_BARRIER "\tfence rw, r\n"#else #define RISCV_ACQUIRE_BARRIER #define RISCV_RELEASE_BARRIER +#define RISCV_ATOMIC_ACQUIRE_BARRIER +#define RISCV_ATOMIC_RELEASE_BARRIER #endif #endif /* _ASM_RISCV_FENCE_H */ Quoted 13 lines
Quoted 5 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
Quoted 7 lines
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"Quoted 1 line
Yes, using .aqrl is valid.
Thx and I think the below is also valid, right?
arch/riscv/include/asm/fence.h +2
-4-RISCV_RELEASE_BARRIER \ -" amoswap.w %0, %2, %1\n" \ +" amoswap.w.rl %0, %2, %1\n" \ -" amoswap.d %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.d.aq %0, %2, %1\n" \Quoted 72 lines
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun--
Best Regards
Guo Ren
Quoted 85 lines · Boqun Feng
Quoted 83 lines · Guo Ren
Hi Dan,
Quoted 50 lines · Dan Lustig
Quoted 48 lines · Guo Ren
Thx Dan,
Quoted 45 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThese opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Quoted 8 lines
Quoted 3 lines
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no
smp_store_acquire()orsmp_load_release(), nor are there equivalents in C/C++ atomics.First, thx for pointing out that my patch violates the rules defined in the ISA manual. I've abandoned these parts in v3.
It's easy to let hw support lr.rl & sc.aq (eg: our hardware supports them). I agree there are no equivalents in C/C++ atomics. But they are useful for LR/SC pairs to implement atomic_acqurie/release semantics. Compare below:
A): fence rw, r; lr
B): lr.rl
The A has another "fence ,r" effect in semantics, it's over commit from a software design view.ps: Current definition has problems:
#defineRISCV_ACQUIRE_BARRIER"\tfence r , rw\n"
#defineRISCV_RELEASE_BARRIER"\tfence rw, w\n"#define
__cmpxchg_release(ptr, old, new, size)\
...__asm__ __volatile__ ( \ RISCV_RELEASE_BARRIER \ "0: lr.w %0, %2\n" \
Quoted 6 lines
Quoted 3 lines
That means "fence rw, w" can't prevent lr.w beyond the fence, we need a "fence.rw. r" here. Here is the Fixup patch which I'm preparing:
That's not true. Note that RELEASE semantics only applies to the write/store part of a read-modify-write atomic, similarly, ACQUIRE only
I just want to point out that the "atomic" mentioned here is only for RISC-V LR/SC AMO instructions. It has been clarified to tread AMO instruction as the whole part for other AMO instructions.
- .aq: If the aq bit is set, then no later memory operations
in this RISC-V hart can be observed to take place before the AMO.
- .rl: If the rl bit is set, then other RISC-V harts will not
observe the AMO before memory accesses preceding the AMO in this RISC-V hart.
- .aqrl: Setting both the aq and the rl bit on an AMO makes the sequence sequentially consistent, meaning that it cannot be reordered with earlier or later memory operations from the same hart.
Quoted 2 lines
applies to the read/load part. For example, the following litmus test can observe the exists clause being true.
Thx for pointing out, that means changing "fence rw, w" to "fence rw. r" is more strict and it would lower performance, right?
Quoted 10 lines
{}
P0(int *x, int *y)
{
int r0;
int r1;
r0 = cmpxchg_acquire(x, 0, 1);
r1 = READ_ONCE(*y);Oh, READ_ONCE could be beyond the write/store part of cmpxchg_acquire, right? We shouldn't prevent it.
Quoted 225 lines
} P1(int *x, int *y) { int r0; WRITE_ONCE(*y, 1); smp_mb(); r0 = READ_ONCE(*x); } exists (0:r0=0 /\ 0:r1=0 /\ 1:r0=0)Regards,
BoqunQuoted 209 lines
From
14c93aca0c3b10cf134791cf491b459972a36ec4Mon Sep 17 00:00:00 2001
From: Guo Ren <[email protected]>
Date: Thu, 21 Apr 2022 16:44:48 +0800
Subject: [PATCH] riscv: atomic: Fixup wrong __atomic_acquire/release_fence
implementationCurrent RISCV_ACQUIRE/RELEASE_BARRIER is for
spin_locknot atomic.
__cmpxchg_release(ptr, old, new, size)
...__asm__ __volatile__ ( RISCV_RELEASE_BARRIER "0: lr.w %0, %2\n"The "fence rw, w -> lr.w" is invalid and lr would beyond fence, so we need "fence rw, r -> lr.w" here. Atomic acquire is the same.
Fixes 0123f4d76ca6 ("riscv/spinlock: Strengthen implementations with fences")Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Palmer Dabbelt <[email protected]>Cc Mark Rutland <[email protected]>Cc Andrea Parri <[email protected]>Cc Dan Lustig <[email protected]>--- arch/riscv/include/asm/atomic.h | 4 ++-- arch/riscv/include/asm/cmpxchg.h | 8 ++++---- arch/riscv/include/asm/fence.h | 4 ++++ 3 files changed, 10 insertions(+), 6 deletions(-)arch/riscv/include/asm/atomic.h +2
-2@@ -20,10 +20,10 @@#include <asm/barrier.h> #define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + __asm__ __volatile__(RISCV_ATOMIC_ACQUIRE_BARRIER "":::"memory")#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); + __asm__ __volatile__(RISCV_ATOMIC_RELEASE_BARRIER"" ::: "memory"); static __always_inline int arch_atomic_read(const atomic_t *v) {arch/riscv/include/asm/cmpxchg.h +4
-4@@ -217,7 +217,7 @@" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" ((long)__old), " rJ" (__new) \ @@ -229,7 +229,7 @@" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" (__old), " rJ" (__new) \ @@ -259,7 +259,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.w %1, %z4, %2\n" \@@ -271,7 +271,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.d %1, %z4, %2\n" \arch/riscv/include/asm/fence.h +4
-0@@ -4,9 +4,13 @@#ifdef CONFIG_SMP #define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"#define RISCV_RELEASE_BARRIER "\tfence rw, w\n" +#define RISCV_ATOMIC_ACQUIRE_BARRIER "\tfence w , rw\n" +#define RISCV_ATOMIC_RELEASE_BARRIER "\tfence rw, r\n"#else #define RISCV_ACQUIRE_BARRIER #define RISCV_RELEASE_BARRIER +#define RISCV_ATOMIC_ACQUIRE_BARRIER +#define RISCV_ATOMIC_RELEASE_BARRIER #endif #endif /* _ASM_RISCV_FENCE_H */ Quoted 13 lines
Quoted 5 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
Quoted 7 lines
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"Quoted 1 line
Yes, using .aqrl is valid.
Thx and I think the below is also valid, right?
arch/riscv/include/asm/fence.h +2
-4-RISCV_RELEASE_BARRIER \ -" amoswap.w %0, %2, %1\n" \ +" amoswap.w.rl %0, %2, %1\n" \ -" amoswap.d %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.d.aq %0, %2, %1\n" \Quoted 72 lines
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun--
Best Regards
Guo Ren
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 107 lines · Guo Ren
Quoted 85 lines · Boqun Feng
Quoted 83 lines · Guo Ren
Hi Dan,
Quoted 50 lines · Dan Lustig
Quoted 48 lines · Guo Ren
Thx Dan,
Quoted 45 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThese opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Quoted 8 lines
Quoted 3 lines
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no
smp_store_acquire()orsmp_load_release(), nor are there equivalents in C/C++ atomics.First, thx for pointing out that my patch violates the rules defined in the ISA manual. I've abandoned these parts in v3.
It's easy to let hw support lr.rl & sc.aq (eg: our hardware supports them). I agree there are no equivalents in C/C++ atomics. But they are useful for LR/SC pairs to implement atomic_acqurie/release semantics. Compare below:
A): fence rw, r; lr
B): lr.rl
The A has another "fence ,r" effect in semantics, it's over commit from a software design view.ps: Current definition has problems:
#defineRISCV_ACQUIRE_BARRIER"\tfence r , rw\n"
#defineRISCV_RELEASE_BARRIER"\tfence rw, w\n"#define
__cmpxchg_release(ptr, old, new, size)\
...__asm__ __volatile__ ( \ RISCV_RELEASE_BARRIER \ "0: lr.w %0, %2\n" \Quoted 6 lines
Quoted 3 lines
That means "fence rw, w" can't prevent lr.w beyond the fence, we need a "fence.rw. r" here. Here is the Fixup patch which I'm preparing:
That's not true. Note that RELEASE semantics only applies to the write/store part of a read-modify-write atomic, similarly, ACQUIRE only
I just want to point out that the "atomic" mentioned here is only for RISC-V LR/SC AMO instructions. It has been clarified to tread AMO instruction as the whole part for other AMO instructions.
- .aq: If the aq bit is set, then no later memory operations
in this RISC-V hart can be observed to take place before the AMO.
- .rl: If the rl bit is set, then other RISC-V harts will not
observe the AMO before memory accesses preceding the AMO in this RISC-V hart.
- .aqrl: Setting both the aq and the rl bit on an AMO makes the sequence sequentially consistent, meaning that it cannot be reordered with earlier or later memory operations from the same hart.
Quoted 4 lines
Quoted 2 lines
applies to the read/load part. For example, the following litmus test can observe the exists clause being true.
Thx for pointing out, that means changing "fence rw, w" to "fence rw. r" is more strict and it would lower performance, right?
Yes, I think it's more strict but honestly I don't know the performance impact ;-)
Quoted 9 lines
Quoted 9 lines
{} P0(int *x, int *y) { int r0; int r1;
Quoted 4 lines
Quoted 2 lines
r0 = cmpxchg_acquire(x, 0, 1); r1 = READ_ONCE(*y);Oh,
READ_ONCEcould be beyond the write/store part ofcmpxchg_acquire, right? We shouldn't prevent it.
Right, the reordering is allowed by the API of Linux atomics and you don't have to prevent it.
Regards,
Boqun
Quoted 234 lines
Quoted 226 lines
} P1(int *x, int *y) { int r0; WRITE_ONCE(*y, 1); smp_mb(); r0 = READ_ONCE(*x); } exists (0:r0=0 /\ 0:r1=0 /\ 1:r0=0)Regards,
BoqunQuoted 209 lines
From
14c93aca0c3b10cf134791cf491b459972a36ec4Mon Sep 17 00:00:00 2001
From: Guo Ren <[email protected]>
Date: Thu, 21 Apr 2022 16:44:48 +0800
Subject: [PATCH] riscv: atomic: Fixup wrong __atomic_acquire/release_fence
implementationCurrent RISCV_ACQUIRE/RELEASE_BARRIER is for
spin_locknot atomic.
__cmpxchg_release(ptr, old, new, size)
...__asm__ __volatile__ ( RISCV_RELEASE_BARRIER "0: lr.w %0, %2\n"The "fence rw, w -> lr.w" is invalid and lr would beyond fence, so we need "fence rw, r -> lr.w" here. Atomic acquire is the same.
Fixes 0123f4d76ca6 ("riscv/spinlock: Strengthen implementations with fences")Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Palmer Dabbelt <[email protected]>Cc Mark Rutland <[email protected]>Cc Andrea Parri <[email protected]>Cc Dan Lustig <[email protected]>--- arch/riscv/include/asm/atomic.h | 4 ++-- arch/riscv/include/asm/cmpxchg.h | 8 ++++---- arch/riscv/include/asm/fence.h | 4 ++++ 3 files changed, 10 insertions(+), 6 deletions(-)arch/riscv/include/asm/atomic.h +2
-2@@ -20,10 +20,10 @@#include <asm/barrier.h> #define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + __asm__ __volatile__(RISCV_ATOMIC_ACQUIRE_BARRIER "":::"memory")#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); + __asm__ __volatile__(RISCV_ATOMIC_RELEASE_BARRIER"" ::: "memory"); static __always_inline int arch_atomic_read(const atomic_t *v) {arch/riscv/include/asm/cmpxchg.h +4
-4@@ -217,7 +217,7 @@" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" ((long)__old), " rJ" (__new) \ @@ -229,7 +229,7 @@" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" (__old), " rJ" (__new) \ @@ -259,7 +259,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.w %1, %z4, %2\n" \@@ -271,7 +271,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.d %1, %z4, %2\n" \arch/riscv/include/asm/fence.h +4
-0@@ -4,9 +4,13 @@#ifdef CONFIG_SMP #define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"#define RISCV_RELEASE_BARRIER "\tfence rw, w\n" +#define RISCV_ATOMIC_ACQUIRE_BARRIER "\tfence w , rw\n" +#define RISCV_ATOMIC_RELEASE_BARRIER "\tfence rw, r\n"#else #define RISCV_ACQUIRE_BARRIER #define RISCV_RELEASE_BARRIER +#define RISCV_ATOMIC_ACQUIRE_BARRIER +#define RISCV_ATOMIC_RELEASE_BARRIER #endif #endif /* _ASM_RISCV_FENCE_H */ Quoted 13 lines
Quoted 5 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
Quoted 7 lines
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"Quoted 1 line
Yes, using .aqrl is valid.
Thx and I think the below is also valid, right?
arch/riscv/include/asm/fence.h +2
-4-RISCV_RELEASE_BARRIER \ -" amoswap.w %0, %2, %1\n" \ +" amoswap.w.rl %0, %2, %1\n" \ -" amoswap.d %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.d.aq %0, %2, %1\n" \Quoted 72 lines
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun--
Best Regards
Guo RenSignature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 126 lines · Boqun Feng
Quoted 107 lines · Guo Ren
Quoted 85 lines · Boqun Feng
Quoted 83 lines · Guo Ren
Hi Dan,
Quoted 50 lines · Dan Lustig
Quoted 48 lines · Guo Ren
Thx Dan,
Quoted 45 lines · Dan Lustig
Quoted 27 lines · Guo Ren
Hi Boqun & Andrea,
Quoted 19 lines · Boqun Feng
On Sun, Apr 17, 2022 at 12:49:44AM +0800, Guo Ren wrote:
[...]Quoted 13 lines
If both the aq and rl bits are set, the atomic memory operation is sequentially consistent and cannot be observed to happen before any earlier memory operations or after any later memory operations in the same RISC-V hart and to the same address domain.
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n".(quoted diff) +1
-2- " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n" "1:\n"Quoted 2 lines
So .rl + fence rw, rw is over constraints, only using sc.w.aqrl is more proper.
Quoted 2 lines
Can .aqrl order memory accesses before and after it (not against itself, against each other), i.e. act as a full memory barrier? For example, can
From the RVWMO spec description, the .aqrl annotation appends the same effect with "fence rw, rw" to the AMO instruction, so it's RCsc.
Quoted 10 lines
Not only .aqrl, and I think the below also could be an RCsc when sc.w.aq is executed:
A: Pre-Access
B: lr.w.rl ADDR-0
...
C: sc.w.aq ADDR-0
D: Post-Acess
Because sc.w.aq has overlap address & data dependency on lr.w.rl, the global memory order should beA->B->C->Dwhen sc.w.aq is executed. For the amoswapThese opcodes aren't actually meaningful, unfortunately.
Quoting the ISA manual chapter 10.2: "Software should not set the rl bit on an LR instruction unless the aq bit is also set, nor should software set the aq bit on an SC instruction unless the rl bit is also set."
Quoted 8 lines
Quoted 3 lines
1. Oh, I've missed the behind half of the ISA manual. But why can't we utilize lr.rl & sc.aq in software programming to guarantee the sequence?
lr.aq and sc.rl map more naturally to hardware than lr.rl and sc.aq. Plus, they just aren't common operations to begin with, e.g., there is no
smp_store_acquire()orsmp_load_release(), nor are there equivalents in C/C++ atomics.First, thx for pointing out that my patch violates the rules defined in the ISA manual. I've abandoned these parts in v3.
It's easy to let hw support lr.rl & sc.aq (eg: our hardware supports them). I agree there are no equivalents in C/C++ atomics. But they are useful for LR/SC pairs to implement atomic_acqurie/release semantics. Compare below:
A): fence rw, r; lr
B): lr.rl
The A has another "fence ,r" effect in semantics, it's over commit from a software design view.ps: Current definition has problems:
#defineRISCV_ACQUIRE_BARRIER"\tfence r , rw\n"
#defineRISCV_RELEASE_BARRIER"\tfence rw, w\n"#define
__cmpxchg_release(ptr, old, new, size)\
...__asm__ __volatile__ ( \ RISCV_RELEASE_BARRIER \ "0: lr.w %0, %2\n" \Quoted 6 lines
Quoted 3 lines
That means "fence rw, w" can't prevent lr.w beyond the fence, we need a "fence.rw. r" here. Here is the Fixup patch which I'm preparing:
That's not true. Note that RELEASE semantics only applies to the write/store part of a read-modify-write atomic, similarly, ACQUIRE only
I just want to point out that the "atomic" mentioned here is only for RISC-V LR/SC AMO instructions. It has been clarified to tread AMO instruction as the whole part for other AMO instructions.
- .aq: If the aq bit is set, then no later memory operations
in this RISC-V hart can be observed to take place before the AMO.
- .rl: If the rl bit is set, then other RISC-V harts will not
observe the AMO before memory accesses preceding the AMO in this RISC-V hart.
- .aqrl: Setting both the aq and the rl bit on an AMO makes the sequence sequentially consistent, meaning that it cannot be reordered with earlier or later memory operations from the same hart.Quoted 4 lines
Quoted 2 lines
applies to the read/load part. For example, the following litmus test can observe the exists clause being true.
Thx for pointing out, that means changing "fence rw, w" to "fence rw. r" is more strict and it would lower performance, right?
Yes, I think it's more strict but honestly I don't know the performance impact ;-)
Quoted 9 lines
Quoted 9 lines
{} P0(int *x, int *y) { int r0; int r1;
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
r0 = cmpxchg_acquire(x, 0, 1); r1 = READ_ONCE(*y);Oh,
READ_ONCEcould be beyond the write/store part ofcmpxchg_acquire, right? We shouldn't prevent it.Right, the reordering is allowed by the API of Linux atomics and you don't have to prevent it.
Thx, you are right, I got it.
Quoted 238 lines
Regards,
BoqunQuoted 234 lines
Quoted 14 lines
} P1(int *x, int *y) { int r0; WRITE_ONCE(*y, 1); smp_mb(); r0 = READ_ONCE(*x); } exists (0:r0=0 /\ 0:r1=0 /\ 1:r0=0)Quoted 212 lines
Regards,
BoqunQuoted 209 lines
From
14c93aca0c3b10cf134791cf491b459972a36ec4Mon Sep 17 00:00:00 2001
From: Guo Ren <[email protected]>
Date: Thu, 21 Apr 2022 16:44:48 +0800
Subject: [PATCH] riscv: atomic: Fixup wrong __atomic_acquire/release_fence
implementationCurrent RISCV_ACQUIRE/RELEASE_BARRIER is for
spin_locknot atomic.
__cmpxchg_release(ptr, old, new, size)
...__asm__ __volatile__ ( RISCV_RELEASE_BARRIER "0: lr.w %0, %2\n"The "fence rw, w -> lr.w" is invalid and lr would beyond fence, so we need "fence rw, r -> lr.w" here. Atomic acquire is the same.
Fixes 0123f4d76ca6 ("riscv/spinlock: Strengthen implementations with fences")Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Palmer Dabbelt <[email protected]>Cc Mark Rutland <[email protected]>Cc Andrea Parri <[email protected]>Cc Dan Lustig <[email protected]>--- arch/riscv/include/asm/atomic.h | 4 ++-- arch/riscv/include/asm/cmpxchg.h | 8 ++++---- arch/riscv/include/asm/fence.h | 4 ++++ 3 files changed, 10 insertions(+), 6 deletions(-)arch/riscv/include/asm/atomic.h +2
-2@@ -20,10 +20,10 @@#include <asm/barrier.h> #define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + __asm__ __volatile__(RISCV_ATOMIC_ACQUIRE_BARRIER "":::"memory")#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); + __asm__ __volatile__(RISCV_ATOMIC_RELEASE_BARRIER"" ::: "memory"); static __always_inline int arch_atomic_read(const atomic_t *v) {arch/riscv/include/asm/cmpxchg.h +4
-4@@ -217,7 +217,7 @@" bne %0, %z3, 1f\n" \" sc.w %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" ((long)__old), " rJ" (__new) \ @@ -229,7 +229,7 @@" bne %0, %z3, 1f\n" \" sc.d %1, %z4, %2\n" \" bnez %1, 0b\n" \ -RISCV_ACQUIRE_BARRIER \ +RISCV_ATOMIC_ACQUIRE_BARRIER \"1:\n" \ : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \: " rJ" (__old), " rJ" (__new) \ @@ -259,7 +259,7 @@switch (size) { \case 4 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.w %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.w %1, %z4, %2\n" \@@ -271,7 +271,7 @@break ; \case 8 : \ __asm__ __volatile__ ( \ -RISCV_RELEASE_BARRIER \ +RISCV_ATOMIC_RELEASE_BARRIER \"0: lr.d %0, %2\n" \" bne %0, %z3, 1f\n" \ " sc.d %1, %z4, %2\n" \arch/riscv/include/asm/fence.h +4
-0@@ -4,9 +4,13 @@#ifdef CONFIG_SMP #define RISCV_ACQUIRE_BARRIER "\tfence r , rw\n"#define RISCV_RELEASE_BARRIER "\tfence rw, w\n" +#define RISCV_ATOMIC_ACQUIRE_BARRIER "\tfence w , rw\n" +#define RISCV_ATOMIC_RELEASE_BARRIER "\tfence rw, r\n"#else #define RISCV_ACQUIRE_BARRIER #define RISCV_RELEASE_BARRIER +#define RISCV_ATOMIC_ACQUIRE_BARRIER +#define RISCV_ATOMIC_RELEASE_BARRIER #endif #endif /* _ASM_RISCV_FENCE_H */ Quoted 13 lines
Quoted 5 lines
2. Using .aqrl to replace the fence rw, rw is okay to ISA manual, right? And reducing a fence instruction to gain better performance:
Quoted 7 lines
"0: lr.w %[p], %[c]\n" " sub %[rc], %[p], %[o]\n" " bltz %[rc], 1f\n". - " sc.w.rl %[rc], %[rc], %[c]\n" + " sc.w.aqrl %[rc], %[rc], %[c]\n" " bnez %[rc], 0b\n" - " fence rw, rw\n"Quoted 1 line
Yes, using .aqrl is valid.
Thx and I think the below is also valid, right?
arch/riscv/include/asm/fence.h +2
-4-RISCV_RELEASE_BARRIER \ -" amoswap.w %0, %2, %1\n" \ +" amoswap.w.rl %0, %2, %1\n" \ -" amoswap.d %0, %2, %1\n" \ -RISCV_ACQUIRE_BARRIER \ +" amoswap.d.aq %0, %2, %1\n" \Quoted 72 lines
Dan
Quoted 68 lines
Quoted 65 lines
Dan
Quoted 62 lines
The purpose of the whole patchset is to reduce the usage of independent fence rw, rw instructions, and maximize the usage of the .aq/.rl/.aqrl aonntation of RISC-V.
__asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ " fence rw, rw\n" \ "1:\n" \Quoted 20 lines
we end up with u == 1, v == 1, r1 on P0 is 0 and r1 on P1 is 0, for the following litmus test?
C lr-sc-aqrl-pair-vs-full-barrier {} P0(int *x, int *y, atomic_t *u) { int r0; int r1; WRITE_ONCE(*x, 1); r0 = atomic_cmpxchg(u, 0, 1); r1 = READ_ONCE(*y); } P1(int *x, int *y, atomic_t *v) { int r0;Quoted 8 lines
int r1; WRITE_ONCE(*y, 1); r0 = atomic_cmpxchg(v, 0, 1); r1 = READ_ONCE(*x); } exists (u=1 /\ v=1 /\ 0:r1=0 /\ 1:r1=0)I think my patchset won't affect the above sequence guarantee. Current RISC-V implementation only gives RCsc when the original value is the same at least once. So I prefer RISC-V cmpxchg should be:
(quoted diff) +2
-2- "0: lr.w %0, %2\n" \ + "0: lr.w.rl %0, %2\n" \ " bne %0, %z3, 1f\n" \ " sc.w.rl %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - " fence rw, rw\n" \ "1:\n" \ + " fence w, rw\n" \To give an unconditional RSsc for
atomic_cmpxchg.Quoted 3 lines
Regards,
Boqun--
Best Regards
Guo Ren--
Best Regards
Guo Ren
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 8 lines
Quoted 4 lines
Seems to me that you are basically reverting
5ce6c1f3535f("riscv/atomic: Strengthen implementations with fences"). That commit fixed an memory ordering issue, could you explain why the issue no longer needs a fix?I'm not reverting the prior patch, just optimizing it.
In RISC-V “A” Standard Extension for Atomic Instructions spec, it said:
With reference to the RISC-V herd specification at:
https://github.com/riscv/riscv-isa-manual.git
the issue, better, lr-sc-aqrl-pair-vs-full-barrier seems to _no longer_ need a fix since commit:
03a5e722fc0f ("Updates to the memory consistency model spec")
(here a template, to double check:
I defer to Daniel/others for a "bi-section" of the prose specification. ;-)
Andrea
Quoted 16 lines · Andrea Parri
Quoted 8 lines
Quoted 4 lines
Seems to me that you are basically reverting
5ce6c1f3535f("riscv/atomic: Strengthen implementations with fences"). That commit fixed an memory ordering issue, could you explain why the issue no longer needs a fix?I'm not reverting the prior patch, just optimizing it.
In RISC-V “A” Standard Extension for Atomic Instructions spec, it said:
With reference to the RISC-V herd specification at:
https://github.com/riscv/riscv-isa-manual.git
the issue, better, lr-sc-aqrl-pair-vs-full-barrier seems to
_nolonger_need a fix since commit:
Quoted 8 lines
03a5e722fc0f("Updates to the memory consistency model spec")(here a template, to double check:
I defer to Daniel/others for a "bi-section" of the prose specification. ;-)
What is the question exactly?
Dan
Quoted 2 lines
Andrea
Quoted 8 lines · Andrea Parri
Quoted 8 lines
Quoted 5 lines
Seems to me that you are basically reverting
5ce6c1f3535f("riscv/atomic: Strengthen implementations with fences"). That commit fixed an memory ordering issue, could you explain why the issue no longer needs a fix?I'm not reverting the prior patch, just optimizing it.
Quoted 8 lines
Quoted 1 line
In RISC-V “A” Standard Extension for Atomic Instructions spec, it said:
With reference to the RISC-V herd specification at:
https://github.com/riscv/riscv-isa-manual.git
the issue, better, lr-sc-aqrl-pair-vs-full-barrier seems to
_nolonger_need a fix since commit:
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w.rl %1, %z4, %2\n" \
" bnez %1, 0b\n" \
" fence rw, rw\n" \
Above is the current implementation, and the logic is in conflict. If
we want full-barrier, we should implement like below:
fence rw, w
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w %1, %z4, %2\n" \
" bnez %1, 0b\n" \
" fence rw, rw\n" \
Above we could let lr.w & sc.w executed fastest. If we think .aq/.rl
won't affect forward guarantee, we should implement like below:
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w.aqrl %1, %z4, %2\n" \
" bnez %1, 0b\n" \Using .aqrl is better than sc.w.rl + fence rw, rw, because lr/sc.rl pair forward guarantee is the same with lr/sw.aqrl and only sc.rl part would affect the speed of lr/sc speed. Second, it could reduce one fence rw, rw overhead. So for riscv, we needn't put a full-barrier after sc like arm64 and use .aqrl instead.
Quoted 11 lines
03a5e722fc0f("Updates to the memory consistency model spec")(here a template, to double check:
I defer to Daniel/others for a "bi-section" of the prose specification. ;-)
Andrea
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/