LoongArch: Fixup cmpxchg sematic for memory barrier
PATCH 1/1 LoongArch: Fixup cmpxchg sematic for memory barrier
From: Guo Ren <[email protected]>
When cmpxchg failed, no memory barrier was needed to take. Only when cmpxchg success and the new value is written, then the memory barriers needed.
- cmpxchg_asm: Unnecessary __WEAK_LLSC_WB for the fail path would
reduce the performance of the cmpxchg loop trying.
- cmpxchg_small: Miss an necessary __WEAK_LLSC_WB when sc succeeds. It's a bug because there is no memory synchronization when sc succeeds.
--- arch/loongarch/include/asm/cmpxchg.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/loongarch/include/asm/cmpxchg.h +2 -4
@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size)" move $t0, %z4 \n" \" " st" $t0, %1 \n" \" beqz $t0, 1b \n" \ -"2: \n" \ __WEAK_LLSC_MB \ +"2: \n" \ : "=&r" (__ret), "=ZB"(*m) \: " ZB "(*m), " Jr" (old), " Jr" (new) \ : "t0", "memory"); \
@@ -148,10 +148,8 @@ static inline unsigned int __cmpxchg_small(volatile void *ptr, unsigned int old," or %1, %1, %z6 \n" " sc.w %1, %2 \n" " beqz %1, 1b \n" -" b 3f \n" -"2: \n" __WEAK_LLSC_MB -"3: \n" +"2: \n" : "=&r" (old32), "=&r" (temp), "=ZC" (*ptr32) : "ZC" (*ptr32), "Jr" (mask), "Jr" (old), "Jr" (new) : "memory");
Signature · 1 line
2.36.1
Hello,
arch/loongarch/include/asm/cmpxchg.h __arch_xchg(volatile void *ptr, unsigned long x, int size) · line 106
Quoted 11 lines · On Tue, Aug 1, 2023 at 9:16 AM <[email protected]> wrote:
arch/loongarch/include/asm/cmpxchg.h +1
-1@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size)" move $t0, %z4 \n" \" " st" $t0, %1 \n" \" beqz $t0, 1b \n" \ -"2: \n" \ __WEAK_LLSC_MB \ +"2: \n" \
Thanks for the patch.
This would look pretty good if it weren't for the special memory barrier semantics of the LoongArch's LL and SC instructions.
The LL/SC memory barrier behavior of LoongArch:
* LL: <memory-barrier> + <load-exclusive>
* SC: <store-conditional> + <memory-barrier>
and the LoongArch's weak memory model allows load/load reorder for the same address.
So, the __WEAK_LLSC_MB[1] is used to prevent load/load reorder and no explicit barrier instruction is required after SC.
[1] https://lore.kernel.org/loongarch/[email protected]/
Regards,
--
WANG Rui
Quoted 18 lines · WANG Rui
arch/loongarch/include/asm/cmpxchg.h __arch_xchg(volatile void *ptr, unsigned long x, int size) · line 106
Quoted 11 lines · On Tue, Aug 1, 2023 at 9:16 AM <[email protected]> wrote:
arch/loongarch/include/asm/cmpxchg.h +1
-1@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size)" move $t0, %z4 \n" \" " st" $t0, %1 \n" \" beqz $t0, 1b \n" \ -"2: \n" \ __WEAK_LLSC_MB \ +"2: \n" \Thanks for the patch.
This would look pretty good if it weren't for the special memory barrier semantics of the LoongArch's LL and SC instructions.
Quoted 7 lines
The LL/SC memory barrier behavior of LoongArch:
* LL: <memory-barrier> + <load-exclusive>
* SC: <store-conditional> + <memory-barrier>and the LoongArch's weak memory model allows load/load reorder for the same address.
Hmm, somehow this one passed me by, but I think that puts you in the naughty corner with Itanium. It probably also means your READ_ONCE() is broken, unless the compiler emits barriers for volatile reads (like ia64)?
Will
Hello,
Quoted 4 lines · Will Deacon
Hmm, somehow this one passed me by, but I think that puts you in the naughty corner with Itanium. It probably also means your
READ_ONCE()is broken, unless the compiler emits barriers for volatile reads (like ia64)?
Hmm, I agree with your perspective. Allowing out-of-order loads for the same address in the memory model provides certain performance benefits, but it also poses challenges to software. Fortunately, hardware supports software to disable this feature when needed.
Regards,
Signature · 1 line
WANG Rui
Quoted 21 lines · WANG Rui
Hello,
arch/loongarch/include/asm/cmpxchg.h __arch_xchg(volatile void *ptr, unsigned long x, int size) · line 106
Quoted 11 lines · On Tue, Aug 1, 2023 at 9:16 AM <[email protected]> wrote:
arch/loongarch/include/asm/cmpxchg.h +1
-1@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size)" move $t0, %z4 \n" \" " st" $t0, %1 \n" \" beqz $t0, 1b \n" \ -"2: \n" \ __WEAK_LLSC_MB \ +"2: \n" \Thanks for the patch.
This would look pretty good if it weren't for the special memory barrier semantics of the LoongArch's LL and SC instructions.
Quoted 7 lines
The LL/SC memory barrier behavior of LoongArch:
* LL: <memory-barrier> + <load-exclusive>
* SC: <store-conditional> + <memory-barrier>and the LoongArch's weak memory model allows load/load reorder for the same address.
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
CPU 0
CPU1
LL(2) (set ex-monitor)
store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successed ?)
Quoted 10 lines
So, the __WEAK_LLSC_MB[1] is used to prevent load/load reorder and no explicit barrier instruction is required after SC.
[1] https://lore.kernel.org/loongarch/[email protected]/
Regards,
--
WANG Rui
Signature · 2 lines
Best Regards Guo Ren
Quoted 33 lines · Guo Ren
Quoted 21 lines · WANG Rui
Hello,
arch/loongarch/include/asm/cmpxchg.h __arch_xchg(volatile void *ptr, unsigned long x, int size) · line 106
Quoted 11 lines · On Tue, Aug 1, 2023 at 9:16 AM <[email protected]> wrote:
arch/loongarch/include/asm/cmpxchg.h +1
-1@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size)" move $t0, %z4 \n" \" " st" $t0, %1 \n" \" beqz $t0, 1b \n" \ -"2: \n" \ __WEAK_LLSC_MB \ +"2: \n" \Thanks for the patch.
This would look pretty good if it weren't for the special memory barrier semantics of the LoongArch's LL and SC instructions.
Quoted 7 lines
The LL/SC memory barrier behavior of LoongArch:
* LL: <memory-barrier> + <load-exclusive>
* SC: <store-conditional> + <memory-barrier>and the LoongArch's weak memory model allows load/load reorder for the same address.
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
Quoted 7 lines
CPU 0
CPU1
LL(2) (set ex-monitor)store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successes ?)
Sorry for the mail client reformat, I mean:
CPU0 LL(2) (set ex-monitor) CPU1 STORE (break the ex-monitor) CPU0 LL(1) (reordered instruction set ex-monitor CPU0 SC(3) (success?)
Quoted 16 lines
Quoted 11 lines
So, the __WEAK_LLSC_MB[1] is used to prevent load/load reorder and no explicit barrier instruction is required after SC.
[1] https://lore.kernel.org/loongarch/[email protected]/
Regards,
--
WANG Rui--
Best Regards
Guo Ren
Signature · 2 lines
Best Regards Guo Ren
Hello,
Quoted 11 lines · Guo Ren
Quoted 11 lines
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
CPU 0
CPU1
LL(2) (set ex-monitor)store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successes ?)
Quoted 6 lines
Sorry for the mail client reformat, I mean:
CPU0 LL(2) (set ex-monitor) CPU1 STORE (break the ex-monitor) CPU0 LL(1) (reordered instruction set ex-monitor CPU0 SC(3) (success?)
No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
Regards,
Signature · 1 line
WANG Rui
Quoted 17 lines · WANG Rui
Hello,
Quoted 13 lines · Guo Ren
Quoted 11 lines
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
CPU 0
CPU1
LL(2) (set ex-monitor)store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successes ?)Sorry for the mail client reformat, I mean:
Quoted 7 lines
Quoted 4 lines
CPU0 LL(2) (set ex-monitor) CPU1 STORE (break the ex-monitor) CPU0 LL(1) (reordered instruction set ex-monitor CPU0 SC(3) (success?)No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
That means we could remove __WEAK_LLSC_MB totally, right?
Quoted 5 lines
Regards,
--
WANG Rui
Signature · 2 lines
Best Regards Guo Ren
Quoted 14 lines · Guo Ren
Quoted 14 lines · Guo Ren
Quoted 13 lines
Quoted 11 lines
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
CPU 0
CPU1
LL(2) (set ex-monitor)store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successes ?)Sorry for the mail client reformat, I mean:
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
CPU0 LL(2) (set ex-monitor)
CPU1 STORE (break the ex-monitor)
CPU0 LL(1) (reordered instruction set ex-monitor
CPU0 SC(3) (success?)No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
That means we could remove
__WEAK_LLSC_MBtotally, right?
As I've said, to implement CAS on LA464 this barrier is *really* needed. I initially didn't believe it then I spent one night (from 11 PM to 04 AM) debugging GCC libgomp test failures.
On LA664 (3A6000) things may change though.
Signature · 2 lines
Xi Ruoyao <[email protected]> School of Aerospace Science and Technology, Xidian University
Quoted 22 lines · Xi Ruoyao
Quoted 21 lines · Guo Ren
Quoted 21 lines · Guo Ren
Quoted 11 lines
Quoted 11 lines
The CoRR problem would cause wider problems than this.For this case, do you mean your LL -> LL would be reordered?
CPU 0
CPU1
LL(2) (set ex-monitor)store (break the ex-monitor)
LL(1) (reordered instruction set ex-monitor
SC(3) (successes ?)Quoted 6 lines
Sorry for the mail client reformat, I mean:
CPU0 LL(2) (set ex-monitor)
CPU1 STORE (break the ex-monitor)
CPU0 LL(1) (reordered instruction set ex-monitor
CPU0 SC(3) (success?)No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
Quoted 7 lines
Quoted 1 line
That means we could remove
__WEAK_LLSC_MBtotally, right?As I've said, to implement CAS on LA464 this barrier is *really* needed. I initially didn't believe it then I spent one night (from 11 PM to 04 AM) debugging GCC libgomp test failures.
On LA664 (3A6000) things may change though.
And I consider this a hardware bug, so it's out of the general concept of general memory orders. We are basically just applying a workaround provided by uarch designers.
Signature · 2 lines
Xi Ruoyao <[email protected]> School of Aerospace Science and Technology, Xidian University
Quoted 5 lines · Guo Ren
Quoted 2 lines · WANG Rui
No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
That means we could remove
__WEAK_LLSC_MBtotally, right?
More precisely, __WEAK_LLSC_MB is intended to prevent reordering between LL and normal LD used to fetch the expected value for cmpxchg.
Regards,
Signature · 1 line
WANG Rui
Quoted 10 lines · WANG Rui
Quoted 5 lines · Guo Ren
Quoted 2 lines · WANG Rui
No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
That means we could remove
__WEAK_LLSC_MBtotally, right?More precisely,
__WEAK_LLSC_MBis intended to prevent reordering between LL and normal LD used to fetch the expected value for cmpxchg.
Oh, that's unnecessary when cmpxchg fails.
Maybe you treat cmpxchg as a CoRR antidote in coincidence. Please solve the CoRR problem by READ_ONCE.
See alpha architecture.
Quoted 5 lines
Regards,
--
WANG Rui
Signature · 2 lines
Best Regards Guo Ren
Quoted 10 lines · Guo Ren
Quoted 8 lines · WANG Rui
Quoted 5 lines · Guo Ren
Quoted 2 lines · WANG Rui
No. LL and LL won't reorder because LL implies a memory barrier(though not acquire semantics).
That means we could remove
__WEAK_LLSC_MBtotally, right?
Quoted 8 lines
Quoted 2 lines
More precisely,
__WEAK_LLSC_MBis intended to prevent reordering between LL and normal LD used to fetch the expected value for cmpxchg.Oh, that's unnecessary when cmpxchg fails.
Maybe you treat cmpxchg as a CoRR antidote in coincidence. Please solve the CoRR problem by
READ_ONCE.See alpha architecture.
Unfortunately, the LL instruction has no acquire semantics. Even if our kernel team improves READ_ONCE, it cannot prevent reordering between LL and READ_ONCE after cmpxchg fails.
LL (<memory-barrier> + <load-exclusive>); WEAK_LLSC_MB; READ_ONCE (<normal-load>); ...
vs
LL (<memory-barrier> + <load-exclusive>); READ_ONCE (<normal-load> + <memory-barrier>); ...
Improving READ_ONCE is really important.
Regards,
Signature · 1 line
WANG Rui
Quoted 8 lines · On Mon, 2023-07-31 at 21:15 -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
When cmpxchg failed, no memory barrier was needed to take. Only when cmpxchg success and the new value is written, then the memory barriers needed.
- cmpxchg_asm: Unnecessary
__WEAK_LLSC_WBfor the fail path would reduce the performance of the cmpxchg loop trying.
I'm not an expert in memory models, but in practice this barrier is really needed or cmpxchg will be "not atomic". See
https://lore.kernel.org/linux-mips/[email protected]/.
Quoted 36 lines
-
cmpxchg_small: Miss an necessary__WEAK_LLSC_WBwhen sc succeeds. It's a bug because there is no memory synchronization when sc succeeds.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>---
arch/loongarch/include/asm/cmpxchg.h | 6 ++----1 file changed, 2 insertions(+), 4 deletions(-)arch/loongarch/include/asm/cmpxchg.h +1
-1@@ -102,8 +102,8 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) " move $t0, %z4 \n" \ " " st " $t0, %1 \n" \ " beqz $t0, 1b \n" \ - "2: \n" \ __WEAK_LLSC_MB \ + "2: \n" \ : "=&r" (__ret), "=ZB"(*m) \: " ZB "(*m), " Jr" (old), " Jr" (new) \ : " t0", " memory"); \ @@ -148,10 +148,8 @@ static inline unsigned int __cmpxchg_small(volatile void *ptr, unsigned int old," or %1, %1, %z6 \n" " sc.w %1, %2 \n" " beqz %1, 1b \n" - " b 3f \n" - "2: \n"
__WEAK_LLSC_MB
- "3: \n" + "2: \n" : "=&r" (old32), "=&r" (temp), "=ZC" (*ptr32) : "ZC" (*ptr32), "Jr" (mask), "Jr" (old), "Jr" (new) : "memory");
Signature · 2 lines
Xi Ruoyao <[email protected]> School of Aerospace Science and Technology, Xidian University