Skip to contentSkip to conversation

atomic: Fixes to smp_mb__{before,after}_atomic() and mips...

RFC PATCH Superseded Peter Zijlstra 5 participants 31 messages Apr 24, 2019 – May 15, 2019

Subsystems (6)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover atomic: Fixes to smp_mb__{before,after}_atomic() and mips...

Hi,

This all started when Andrea Parri found a 'surprising' behaviour for x86:

http://lkml.kernel.org/r/20190418125412.GA10817@andrea

Basically we fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it (surprisingly) does not provide a compiler barrier. This then allows the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended.

This had me audit all the (strong) architectures that had weak smp_mb__{before,after}_atomic: ia64, mips, sparc, s390, x86, xtensa.

Of those, only x86 and mips were affected. Looking at MIPS to solve this, led to the other MIPS patches.

Only the x86 patch is actually compile tested, the MIPS things are fresh from the keyboard.

author maintainer
#1 lore raw

RFC PATCH 1/5 mips/atomic: Fix cmpxchg64 barriers

There were no memory barriers on cmpxchg64() _at_all_. Fix this.

---
 arch/mips/include/asm/cmpxchg.h |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/mips/include/asm/cmpxchg.h +4 -2
@@ -290,9 +290,11 @@ static inline unsigned long __cmpxchg64(
 	 * will cause a build error unless cpu_has_64bits is a		\
 	 * compile-time constant 1.					\
 	 */								\
-	if (cpu_has_64bits && kernel_uses_llsc)				\
+	if (cpu_has_64bits && kernel_uses_llsc) {			\
+		smp_mb__before_llsc();					\
 		__res = __cmpxchg64((ptr), __old, __new);		\
-	else								\
+		smp_llsc_mb();						\
+	} else								\
 		__res = __cmpxchg64_unsupported();			\
 									\
 	__res;								\
#2 lore raw

Re: [RFC][PATCH 1/5] mips/atomic: Fix cmpxchg64 barriers

Hi Peter,

Quoted 1 line · Peter Zijlstra

There were no memory barriers on cmpxchg64() _at_all_. Fix this.

This does looks problematic, but it's worth noting that this code path is only applicable to 32b kernels running on 64b CPUs which is pretty rare. The commit message as-is suggests to me that all configurations are broken, which isn't the case (at least, not in this respect :) ).

Quoted 11 lines
---
 arch/mips/include/asm/cmpxchg.h |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/mips/include/asm/cmpxchg.h +0 -0
@@ -290,9 +290,11 @@ static inline unsigned long __cmpxchg64(
 	 * will cause a build error unless cpu_has_64bits is a		\

arch/mips/include/asm/cmpxchg.h static inline unsigned long __cmpxchg64( · line 298

Quoted 10 lines
arch/mips/include/asm/cmpxchg.h +4 -2
 	 * compile-time constant 1.					\
 	 */								\
-	if (cpu_has_64bits && kernel_uses_llsc)				\
+	if (cpu_has_64bits && kernel_uses_llsc) {			\
+		smp_mb__before_llsc();					\
 		__res = __cmpxchg64((ptr), __old, __new);		\
-	else								\
+		smp_llsc_mb();						\
+	} else								\
 		__res = __cmpxchg64_unsupported();			\

It would be good to also add braces around the else block, and I believe checkpatch should be complaining about that ("braces {} should be used on all arms of this statement").

Thanks,
Paul

Quoted 4 lines
arch/mips/include/asm/cmpxchg.h +0 -0
 									\
 	__res;								\
author maintainer
#3 lore raw

Re: [RFC][PATCH 1/5] mips/atomic: Fix cmpxchg64 barriers

Quoted 9 lines · Paul Burton

Hi Peter,

Quoted 1 line · Peter Zijlstra

There were no memory barriers on cmpxchg64() _at_all_. Fix this.

This does looks problematic, but it's worth noting that this code path is only applicable to 32b kernels running on 64b CPUs which is pretty rare. The commit message as-is suggests to me that all configurations are broken, which isn't the case (at least, not in this respect :) ).

OK, I hadn't gone through the ifdef selection process. I just encountered this cmpxchg implementation and noted a significant lack of barriers.

Quoted 17 lines
Quoted 17 lines
---
 arch/mips/include/asm/cmpxchg.h |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/mips/include/asm/cmpxchg.h +2 -1
@@ -290,9 +290,11 @@ static inline unsigned long __cmpxchg64(
 	 * will cause a build error unless cpu_has_64bits is a		\
 	 * compile-time constant 1.					\
 	 */								\
-	if (cpu_has_64bits && kernel_uses_llsc)				\
+	if (cpu_has_64bits && kernel_uses_llsc) {			\
+		smp_mb__before_llsc();					\
 		__res = __cmpxchg64((ptr), __old, __new);		\
Quoted 8 lines

arch/mips/include/asm/cmpxchg.h static inline unsigned long __cmpxchg64( · line 298

Quoted 4 lines
arch/mips/include/asm/cmpxchg.h +2 -1
-	else								\
+		smp_llsc_mb();						\
+	} else								\
 		__res = __cmpxchg64_unsupported();			\

It would be good to also add braces around the else block, and I believe checkpatch should be complaining about that ("braces {} should be used on all arms of this statement").

You're right, I'll fix up.

author maintainer
#4 lore raw

RFC PATCH 2/5 mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Peter Zijlstra

The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. This means that _every_ LL/SC loop needs this barrier right in front to avoid the CPU from leaking a memop inside it.

For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asmg() implementation does too. We cannot rely on the barriers from cmpxchg() because cmpxchg_local() is implemented with the same macro, and branch prediction and speculation are, too, CPU local.

Fixes e02e07e3127d ("MIPS: Loongson: Introduce and use loongson_llsc_mb()")
Cc Huacai Chen <[email protected]>
Cc Huang Pei <[email protected]>
Cc Paul Burton <[email protected]>
---
 arch/mips/include/asm/atomic.h  |   24 ++++++++++++++++++++----
 arch/mips/include/asm/barrier.h |    7 ++-----
 arch/mips/include/asm/bitops.h  |    5 +++++
 arch/mips/include/asm/cmpxchg.h |    5 +++++
 arch/mips/include/asm/local.h   |    2 ++
 arch/mips/kernel/syscall.c      |    1 +
 6 files changed, 35 insertions(+), 9 deletions(-)
arch/mips/include/asm/atomic.h +20 -4
@@ -193,6 +193,7 @@ static __inline__ int atomic_sub_if_posi
 	if (kernel_uses_llsc) {
 		int temp;
 
+		loongson_llsc_mb();
 		__asm__ __volatile__(
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_LEVEL"			\n"
@@ -200,16 +201,23 @@ static __inline__ int atomic_sub_if_posi
 		"	.set	pop					\n"
 		"	subu	%0, %1, %3				\n"
 		"	move	%1, %0					\n"
-		"	bltz	%0, 1f					\n"
+		"	bltz	%0, 2f					\n"
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_LEVEL"			\n"
 		"	sc	%1, %2					\n"
 		"\t" __scbeqz "	%1, 1b					\n"
-		"1:							\n"
+		"2:							\n"
 		"	.set	pop					\n"
 		: "=&r" (result), "=&r" (temp),
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)
 		: "Ir" (i));
+
+		if (!IS_ENABLED(CONFIG_SMP))
+			loongson_llsc_mb();
+		/*
+		 * Otherwise the loongson_llsc_mb() for the bltz target is
+		 * implied by the smp_llsc_mb() below.
+		 */
 	} else {
 		unsigned long flags;
 
@@ -395,20 +403,28 @@ static __inline__ long atomic64_sub_if_p
 	if (kernel_uses_llsc) {
 		long temp;
 
+		loongson_llsc_mb();
 		__asm__ __volatile__(
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_LEVEL"			\n"
 		"1:	lld	%1, %2		# atomic64_sub_if_positive\n"
 		"	dsubu	%0, %1, %3				\n"
 		"	move	%1, %0					\n"
-		"	bltz	%0, 1f					\n"
+		"	bltz	%0, 2f					\n"
 		"	scd	%1, %2					\n"
 		"\t" __scbeqz "	%1, 1b					\n"
-		"1:							\n"
+		"2:							\n"
 		"	.set	pop					\n"
 		: "=&r" (result), "=&r" (temp),
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)
 		: "Ir" (i));
+
+		if (!IS_ENABLED(CONFIG_SMP))
+			loongson_llsc_mb();
+		/*
+		 * Otherwise the loongson_llsc_mb() for the bltz target is
+		 * implied by the smp_llsc_mb() below.
+		 */
 	} else {
 		unsigned long flags;
 
arch/mips/include/asm/barrier.h +2 -5
@@ -248,10 +248,7 @@
  *
  *    In order to avoid this we need to place a memory barrier (ie. a sync
  *    instruction) prior to every ll instruction, in between it & any earlier
- *    memory access instructions. Many of these cases are already covered by
- *    smp_mb__before_llsc() but for the remaining cases, typically ones in
- *    which multiple CPUs may operate on a memory location but ordering is not
- *    usually guaranteed, we use loongson_llsc_mb() below.
+ *    memory access instructions.
  *
  *    This reordering case is fixed by 3A R2 CPUs, ie. 3A2000 models and later.
  *
@@ -267,7 +264,7 @@
  *    This case affects all current Loongson 3 CPUs.
  */
 #ifdef CONFIG_CPU_LOONGSON3_WORKAROUNDS /* Loongson-3's LLSC workaround */
-#define loongson_llsc_mb()	__asm__ __volatile__(__WEAK_LLSC_MB : : :"memory")
+#define loongson_llsc_mb()	__asm__ __volatile__("sync" : : :"memory")
 #else
 #define loongson_llsc_mb()	do { } while (0)
 #endif
arch/mips/include/asm/bitops.h +5 -0
@@ -249,6 +249,7 @@ static inline int test_and_set_bit(unsig
 		unsigned long *m = ((unsigned long *) addr) + (nr >> SZLONG_LOG);
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		do {
 			__asm__ __volatile__(
 			"	.set	push				\n"
@@ -305,6 +306,7 @@ static inline int test_and_set_bit_lock(
 		unsigned long *m = ((unsigned long *) addr) + (nr >> SZLONG_LOG);
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		do {
 			__asm__ __volatile__(
 			"	.set	push				\n"
@@ -364,6 +366,7 @@ static inline int test_and_clear_bit(uns
 		unsigned long *m = ((unsigned long *) addr) + (nr >> SZLONG_LOG);
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		do {
 			__asm__ __volatile__(
 			"	" __LL	"%0, %1 # test_and_clear_bit	\n"
@@ -379,6 +382,7 @@ static inline int test_and_clear_bit(uns
 		unsigned long *m = ((unsigned long *) addr) + (nr >> SZLONG_LOG);
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		do {
 			__asm__ __volatile__(
 			"	.set	push				\n"
@@ -438,6 +442,7 @@ static inline int test_and_change_bit(un
 		unsigned long *m = ((unsigned long *) addr) + (nr >> SZLONG_LOG);
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		do {
 			__asm__ __volatile__(
 			"	.set	push				\n"
arch/mips/include/asm/cmpxchg.h +5 -0
@@ -46,6 +46,7 @@ extern unsigned long __xchg_called_with_
 	__typeof(*(m)) __ret;						\
 									\
 	if (kernel_uses_llsc) {						\
+		loongson_llsc_mb();					\
 		__asm__ __volatile__(					\
 		"	.set	push				\n"	\
 		"	.set	noat				\n"	\
@@ -117,6 +118,7 @@ static inline unsigned long __xchg(volat
 	__typeof(*(m)) __ret;						\
 									\
 	if (kernel_uses_llsc) {						\
+		loongson_llsc_mb();					\
 		__asm__ __volatile__(					\
 		"	.set	push				\n"	\
 		"	.set	noat				\n"	\
@@ -134,6 +136,7 @@ static inline unsigned long __xchg(volat
 		: "=&r" (__ret), "=" GCC_OFF_SMALL_ASM() (*m)		\
 		: GCC_OFF_SMALL_ASM() (*m), "Jr" (old), "Jr" (new)		\
 		: "memory");						\
+		loongson_llsc_mb();					\
 	} else {							\
 		unsigned long __flags;					\
 									\
@@ -229,6 +232,7 @@ static inline unsigned long __cmpxchg64(
 	 */
 	local_irq_save(flags);
 
+	loongson_llsc_mb();
 	asm volatile(
 	"	.set	push				\n"
 	"	.set	" MIPS_ISA_ARCH_LEVEL "		\n"
@@ -274,6 +278,7 @@ static inline unsigned long __cmpxchg64(
 	  "r" (old),
 	  "r" (new)
 	: "memory");
+	loongson_llsc_mb();
 
 	local_irq_restore(flags);
 	return ret;
arch/mips/include/asm/local.h +2 -0
@@ -49,6 +49,7 @@ static __inline__ long local_add_return(
 	} else if (kernel_uses_llsc) {
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		__asm__ __volatile__(
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_ARCH_LEVEL"			\n"
@@ -96,6 +97,7 @@ static __inline__ long local_sub_return(
 	} else if (kernel_uses_llsc) {
 		unsigned long temp;
 
+		loongson_llsc_mb();
 		__asm__ __volatile__(
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_ARCH_LEVEL"			\n"
arch/mips/kernel/syscall.c +1 -0
@@ -132,6 +132,7 @@ static inline int mips_atomic_set(unsign
 		  [efault] "i" (-EFAULT)
 		: "memory");
 	} else if (cpu_has_llsc) {
+		loongson_llsc_mb();
 		__asm__ __volatile__ (
 		"	.set	push					\n"
 		"	.set	"MIPS_ISA_ARCH_LEVEL"			\n"
author maintainer
#5 lore raw

Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 6 lines · Peter Zijlstra

The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. This means that _every_ LL/SC loop needs this barrier right in front to avoid the CPU from leaking a memop inside it.

Quoted 6 lines

For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asmg() implementation does too. We cannot rely on the barriers from cmpxchg() because cmpxchg_local() is implemented with the same macro, and branch prediction and speculation are, too, CPU local.

Also; just doing them all makes for much simpler rules and less mistakes.

#6 lore raw

Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Peter Zijlstra

Hi Peter,

Quoted 5 lines · Peter Zijlstra

The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. This means that _every_ LL/SC loop needs this barrier right in front to avoid the CPU from leaking a memop inside it.

Does it?

The Loongson bug being described here causes an sc to succeed erroneously if certain loads or stores are executed between the ll & associated sc, including speculatively. On a UP system there's no code running on other cores to race with us & cause our sc to fail - ie. sc should always succeed anyway, so if the bug hits & the sc succeeds what's the big deal? It would have succeeded anyway. At least that's my understanding based on discussions with Loongson engineers a while ago.

Having said that, if you have a strong preference for adding the barrier in UP systems anyway then I don't really object. It's not like anyone's likely to want to run a UP kernel on the affected systems, nevermind care about a miniscule performance impact.

One possibility your change could benefit would be if someone ran Linux on a subset of cores & some non-Linux code on other cores, in which case there could be something to cause the sc to fail. I've no idea if that's something these Loongson systems ever do though.

Quoted 3 lines

For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asmg() implementation does too. We cannot rely on the

s/cmpxch_asmg/cmpxchg_asm/

Quoted 3 lines

barriers from cmpxchg() because cmpxchg_local() is implemented with the same macro, and branch prediction and speculation are, too, CPU local.

Similar story - cmpxchg_local() ought not have have CPUs racing for access to the memory in question. Having said that I don't know the details of when Loongson clears LLBit (ie. causes an sc to fail), so if it does that on based upon access to memory at a larger granularity than the 32b or 64b value being operated on then that could be a problem so I'm pretty happy with adding these barriers.

Thanks,
Paul

#7 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In my opinion. patch 2/3 is about Loongson's bug, and patch 4/5 is another theme.

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

#. speculative memory access from A cause A erroneously succeed sc operation, since the
erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

PS:

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

Quoted 53 lines

-----原始邮件-----
发件人: "Paul Burton" <[email protected]>
发送时间: 2019-04-25 05:18:04 (星期四)
收件人: "Peter Zijlstra" <[email protected]>
抄送: "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "Huacai Chen" <[email protected]>, "Huang Pei" <[email protected]> 主题: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Hi Peter,

Quoted 5 lines · Peter Zijlstra

The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. This means that _every_ LL/SC loop needs this barrier right in front to avoid the CPU from leaking a memop inside it.

Does it?

The Loongson bug being described here causes an sc to succeed erroneously if certain loads or stores are executed between the ll & associated sc, including speculatively. On a UP system there's no code running on other cores to race with us & cause our sc to fail - ie. sc should always succeed anyway, so if the bug hits & the sc succeeds what's the big deal? It would have succeeded anyway. At least that's my understanding based on discussions with Loongson engineers a while ago.

Having said that, if you have a strong preference for adding the barrier in UP systems anyway then I don't really object. It's not like anyone's likely to want to run a UP kernel on the affected systems, nevermind care about a miniscule performance impact.

One possibility your change could benefit would be if someone ran Linux on a subset of cores & some non-Linux code on other cores, in which case there could be something to cause the sc to fail. I've no idea if that's something these Loongson systems ever do though.

Quoted 3 lines

For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asmg() implementation does too. We cannot rely on the

s/cmpxch_asmg/cmpxchg_asm/

Quoted 3 lines

barriers from cmpxchg() because cmpxchg_local() is implemented with the same macro, and branch prediction and speculation are, too, CPU local.

Similar story - cmpxchg_local() ought not have have CPUs racing for access to the memory in question. Having said that I don't know the details of when Loongson clears LLBit (ie. causes an sc to fail), so if it does that on based upon access to memory at a larger granularity than the 32b or 64b value being operated on then that could be a problem so I'm pretty happy with adding these barriers.

Quoted 2 lines

Thanks,
Paul

北京市海淀区中关村环保科技示范园龙芯产业园2号楼 100095电话: +86 (10) 62546668传真: +86 (10) 62600826www.loongson.cn本邮件及其附件含有龙芯中科技术有限公司的商业秘密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部 分地泄露、复制或散发)本邮件及其附件中的信息。如果您错收本邮件,请您立即电话或邮件通知发件人并删除本邮件。

This email and its attachments contain confidential information from Loongson Technology Corporation Limited, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

author maintainer
#8 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Quoted 1 line · On Thu, Apr 25, 2019 at 12:58:50PM +0800, [email protected] wrote:

In my opinion. patch 2/3 is about Loongson's bug, and patch 4/5 is another theme.

Agreed; it's just that looking at the MIPS code to fix 4/5 made me trip over this stuff.

Quoted 7 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

Quoted 8 lines

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

(I know your coherence protocol is probably more complicated than MESI, but bear with me)

So A speculatively gets V's line in Exclusive mode, speculates the Lock flag is still there and completes the Store. This speculative store then leaks out and violates MESI because there _should_ only be one Exclusive owner of a line (B).

Something like that?

Quoted 3 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

  {
	  atomic_set(v, 0);

}

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

Quoted 2 lines

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

Because it is a coherence issue, triggered by a reorder. OK.

Quoted 3 lines

PS:

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

It _should_ be CPU local, but this was not at all clear from reading the original changelog nor the comment with loongson_llsc_mb().

author maintainer
#9 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 19 lines · Peter Zijlstra
Quoted 7 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

Quoted 8 lines

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

(I know your coherence protocol is probably more complicated than MESI, but bear with me)

Quoted 6 lines

So A speculatively gets V's line in Exclusive mode, speculates the Lock flag is still there and completes the Store. This speculative store then leaks out and violates MESI because there _should_ only be one Exclusive owner of a line (B).

Something like that?

So B gets E (from LL), does I on A, then SC succeeds and get M. A got I, speculates E, speculates M and lets the M escape.

That gets us with 2 competing Ms (which is of course completely insane), one wins one looses (at random I presume).

And this violates atomic guarantees because one operation got lost.

Quoted 18 lines
Quoted 3 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

Quoted 8 lines
  {
	  atomic_set(v, 0);

}

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

But then in this case, P1 has E from LL, P2 does M from the STORE, which should cause I on P1. P1 speculates E, speculates M and lets M escape.

We again have two competing Ms, one wins at random, and v==2 if P1 wins. This again violates the atomic guarantees and would invalidate your claim of it only mattering for competing LL/SC.

Or am I missing something? (quite likely, I always get confused with these things)

#10 lore raw

Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 38 lines

-----原始邮件-----
发件人: "Peter Zijlstra" <[email protected]>
发送时间: 2019-04-25 17:09:07 (星期四)
收件人: [email protected]
抄送: "Paul Burton" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "Huacai Chen" <[email protected]> 主题: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 19 lines · Peter Zijlstra
Quoted 7 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

Quoted 8 lines

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

(I know your coherence protocol is probably more complicated than MESI, but bear with me)

Quoted 6 lines

So A speculatively gets V's line in Exclusive mode, speculates the Lock flag is still there and completes the Store. This speculative store then leaks out and violates MESI because there _should_ only be one Exclusive owner of a line (B).

Something like that?

Quoted 7 lines

So B gets E (from LL), does I on A, then SC succeeds and get M. A got I, speculates E, speculates M and lets the M escape.

That gets us with 2 competing Ms (which is of course completely insane), one wins one looses (at random I presume).

And this violates atomic guarantees because one operation got lost.

Based on what I was told:

#. A get E from LL, previous speculative access from A kick V out, so A should get I, but A still thought A get E ;

#. and B get E from LL,so B get sc done right; V get sc atomically by B;

#. A still thought it get E, so A get sc done, V get sc by A, but not atomically.

Quoted 31 lines
Quoted 19 lines
Quoted 4 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

Quoted 8 lines
  {
	  atomic_set(v, 0);

}

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

But then in this case, P1 has E from LL, P2 does M from the STORE, which should cause I on P1. P1 speculates E, speculates M and lets M escape.

Quoted 6 lines

We again have two competing Ms, one wins at random, and v==2 if P1 wins. This again violates the atomic guarantees and would invalidate your claim of it only mattering for competing LL/SC.

Or am I missing something? (quite likely, I always get confused with these things)

北京市海淀区中关村环保科技示范园龙芯产业园2号楼 100095电话: +86 (10) 62546668传真: +86 (10) 62600826www.loongson.cn本邮件及其附件含有龙芯中科技术有限公司的商业秘密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部 分地泄露、复制或散发)本邮件及其附件中的信息。如果您错收本邮件,请您立即电话或邮件通知发件人并删除本邮件。

This email and its attachments contain confidential information from Loongson Technology Corporation Limited, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

author maintainer
#11 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Peter Zijlstra
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

Quoted 6 lines
Quoted 3 lines

PS:

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

It _should_ be CPU local, but this was not at all clear from reading the original changelog nor the comment with loongson_llsc_mb().

However, if it is a coherence issue, the thing is at the cacheline level, and there is nothing that says the line isn't shared, just the one variable isn't.

Ideally there should be minimal false sharing, but....

author maintainer
#12 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

I think this thread got 'lost'

Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

PS:

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

Quoted 8 lines
Quoted 2 lines

It _should_ be CPU local, but this was not at all clear from reading the original changelog nor the comment with loongson_llsc_mb().

However, if it is a coherence issue, the thing is at the cacheline level, and there is nothing that says the line isn't shared, just the one variable isn't.

Ideally there should be minimal false sharing, but....

So if two variables share a line, and one is local while the other is shared atomic, can contention on the line, but not the variable, cause issues for the local variable?

If not; why not? Because so far the issue is line granular due to the coherence aspect.

#13 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 7 lines · Peter Zijlstra

So if two variables share a line, and one is local while the other is shared atomic, can contention on the line, but not the variable, cause issues for the local variable?

If not; why not? Because so far the issue is line granular due to the coherence aspect.

If I understood the issue correctly, it's not that cache coherence doesn't work, it's literally that the sc succeeds when it shouldn't.

In other words, it's not going to affect anything else, but it means that "ll/sc" isn't actually truly atomic, because the cacheline could have bounced around to another CPU in the meantime.

So we *think* we got an atomic update, but didn't, and the "ll/sc" pair ends up incorrectly working as a regular "load -> store" pair, because the "sc' incorrectly thought it still had exclusive access to the line from the "ll".

The added memory barrier isn't because it's a memory barrier, it's just keeping the subsequent speculative instructions from getting the cacheline back and causing that "sc" confusion.

But note how from a cache coherency standpoint, it's not about the cache coherency being wrong, it's literally just about the ll/sc not giving the atomicity guarantees that the sequence is *supposed* to give. So an "atomic_inc()" can basically (under just the wrong circumstances) essentially turn into just a non-atomic "*p++".

NOTE! I have no actual inside knowledge of what is going on. The above is purely my reading of this thread, and maybe I have mis-understood.

Linus

author maintainer
#14 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 25 lines · Linus Torvalds
Quoted 7 lines · Peter Zijlstra

So if two variables share a line, and one is local while the other is shared atomic, can contention on the line, but not the variable, cause issues for the local variable?

If not; why not? Because so far the issue is line granular due to the coherence aspect.

If I understood the issue correctly, it's not that cache coherence doesn't work, it's literally that the sc succeeds when it shouldn't.

In other words, it's not going to affect anything else, but it means that "ll/sc" isn't actually truly atomic, because the cacheline could have bounced around to another CPU in the meantime.

So we *think* we got an atomic update, but didn't, and the "ll/sc" pair ends up incorrectly working as a regular "load -> store" pair, because the "sc' incorrectly thought it still had exclusive access to the line from the "ll".

The added memory barrier isn't because it's a memory barrier, it's just keeping the subsequent speculative instructions from getting the cacheline back and causing that "sc" confusion.

Quoted 5 lines

But note how from a cache coherency standpoint, it's not about the cache coherency being wrong, it's literally just about the ll/sc not giving the atomicity guarantees that the sequence is *supposed* to give. So an "atomic_inc()" can basically (under just the wrong circumstances) essentially turn into just a non-atomic "*p++".

Understood; the problem is that "*p++" is not good enough for local_t either (on load-store architectures), since it needs to be "atomic" wrt all other instructions on that CPU, most notably exceptions.

The issue has come up before in this thread; but I don't think it was clearly answered before.

#15 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 4 lines · Peter Zijlstra

Understood; the problem is that "*p++" is not good enough for local_t either (on load-store architectures), since it needs to be "atomic" wrt all other instructions on that CPU, most notably exceptions.

Right. But I don't think that's the issue here, since if it was then it would be a problem even on UP.

And while the CPU-local ones want atomicity, they *shouldn't* have the issue of another CPU modifying them, so even if you were to lose exclusive ownership of the cacheline (because some other CPU is reading your per-cpu data for statistics of whatever), the final end result should be fine.

End result: I suspect ll/sc still works for cpu-local stuff without any extra loongson hacks.

But I agree that it would be good to verify.

Linus

#16 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Linus Torvalds
Quoted 26 lines · Peter Zijlstra
Quoted 7 lines

So if two variables share a line, and one is local while the other is shared atomic, can contention on the line, but not the variable, cause issues for the local variable?

If not; why not? Because so far the issue is line granular due to the coherence aspect.

If I understood the issue correctly, it's not that cache coherence doesn't work, it's literally that the sc succeeds when it shouldn't.

In other words, it's not going to affect anything else, but it means that "ll/sc" isn't actually truly atomic, because the cacheline could have bounced around to another CPU in the meantime.

So we *think* we got an atomic update, but didn't, and the "ll/sc" pair ends up incorrectly working as a regular "load -> store" pair, because the "sc' incorrectly thought it still had exclusive access to the line from the "ll".

The added memory barrier isn't because it's a memory barrier, it's just keeping the subsequent speculative instructions from getting the cacheline back and causing that "sc" confusion.

Quoted 6 lines

But note how from a cache coherency standpoint, it's not about the cache coherency being wrong, it's literally just about the ll/sc not giving the atomicity guarantees that the sequence is *supposed* to give. So an "atomic_inc()" can basically (under just the wrong circumstances) essentially turn into just a non-atomic "*p++".

Agreed,that is exactly what I was learned.

Quoted 3 lines

NOTE! I have no actual inside knowledge of what is going on. The above is purely my reading of this thread, and maybe I have mis-understood.

you got it right.

Quoted 1 line

                  Linus

#17 lore raw

Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Peter Zijlstra
Quoted 34 lines

-----原始邮件-----
发件人: "Peter Zijlstra" <[email protected]>
发送时间: 2019-04-25 15:33:48 (星期四)
收件人: [email protected]
抄送: "Paul Burton" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "Huacai Chen" <[email protected]> 主题: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Quoted 1 line · On Thu, Apr 25, 2019 at 12:58:50PM +0800, [email protected] wrote:

In my opinion. patch 2/3 is about Loongson's bug, and patch 4/5 is another theme.

Agreed; it's just that looking at the MIPS code to fix 4/5 made me trip over this stuff.

Quoted 12 lines

Let me explain the bug more specific:

the bug ONLY matters in following situation:

#. more than one cpu (assume cpu A and B) doing ll/sc on same shared var V

#. speculative memory access from A cause A erroneously succeed sc operation, since the erroneously successful sc operation violate the coherence protocol. (here coherence protocol means the rules that CPU follow to implement ll/sc right)

Quoted 6 lines
Quoted 3 lines

#. B succeed sc operation too, but this sc operation is right both logically and follow the coherence protocol, and makes A's sc wrong logically since only ONE sc operation can succeed.

(I know your coherence protocol is probably more complicated than MESI, but bear with me)

our coherentce protocal is simpler than MESI.

Quoted 7 lines

So A speculatively gets V's line in Exclusive mode, speculates the Lock flag is still there and completes the Store. This speculative store then leaks out and violates MESI because there _should_ only be one Exclusive owner of a line (B).

Something like that?

Yes

Quoted 21 lines
Quoted 4 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

Quoted 8 lines
  {
	  atomic_set(v, 0);

}

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

you mean C,P1, P2 on 3 different CPU? I do not know much about LKMM, can explain the test case more explicit?

Quoted 5 lines
Quoted 3 lines

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

Because it is a coherence issue, triggered by a reorder. OK.

Not exactly, it is a ll/sc issue, triggered by speculative memory access from local cpu
based on what I was told.

Quoted 7 lines
Quoted 4 lines

PS:

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

It _should_ be CPU local, but this was not at all clear from reading the original changelog nor the comment with loongson_llsc_mb().

北京市海淀区中关村环保科技示范园龙芯产业园2号楼 100095电话: +86 (10) 62546668传真: +86 (10) 62600826www.loongson.cn本邮件及其附件含有龙芯中科技术有限公司的商业秘密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部 分地泄露、复制或散发)本邮件及其附件中的信息。如果您错收本邮件,请您立即电话或邮件通知发件人并删除本邮件。

This email and its attachments contain confidential information from Loongson Technology Corporation Limited, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

author maintainer
#18 lore raw

Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 11 lines · On Thu, Apr 25, 2019 at 07:32:59PM +0800, [email protected] wrote:
Quoted 11 lines
Quoted 3 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

This is the initial state.

Quoted 12 lines
Quoted 12 lines

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

  {
	  atomic_set(v, 0);

}

Quoted 6 lines
Quoted 4 lines

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

you mean C,P1, P2 on 3 different CPU? I do not know much about LKMM, can explain the test case more explicit?

The 'C' is the language identifier, the 'atomic-set' is the litmus name.

The unnamed block give the initial conditions.

Pn blocks give code sequences for CPU n

The 'exists' clause is evaluated after all Pn blocks are done.

There's others in this thread that can point you to many papers and resources on these litmus test thingies.

So basically the initial value of @v is set to 1.

Then CPU-1 does atomic_add_unless(v, 1, 0)

     CPU-2 does atomic_set(v, 0)

If CPU1 goes first, it will see 1, which is not 0 and thus add 1 to 1 and obtains 2. Then CPU2 goes and writes 0, so the exist clause sees v==0 and doesn't observe 2.

The other way around, CPU-2 goes first, writes a 0, then CPU-1 goes and observes the 0, finds it matches 0 and doesn't add. Again, the exist clause will find 0 doesn't match 2.

This all goes unstuck if interleaved like:

	CPU-1			CPU-2

				xor	t0, t0
1:	ll	t0, v
	bez	t0, 2f
				sw	t0, v
	add	t0, t1
	sc	t0, v
	beqz t0, 1b

(sorry if I got the MIPS asm wrong; it's not something I normally write)

And the store-word from CPU-2 doesn't make the SC from CPU-1 fail.

#19 lore raw

Re: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 82 lines

-----原始邮件-----
发件人: "Peter Zijlstra" <[email protected]>
发送时间: 2019-04-25 20:26:11 (星期四)
收件人: [email protected]
抄送: "Paul Burton" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "Huacai Chen" <[email protected]> 主题: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 11 lines · On Thu, Apr 25, 2019 at 07:32:59PM +0800, [email protected] wrote:
Quoted 11 lines
Quoted 3 lines

If it is not LL/SC but other memory access from B on V, A's ll/sc can follow the atomic semantics even if A violate the coherence protocol in the same situation.

*shudder*...

C atomic-set

  {
	  atomic_set(v, 1);

}

This is the initial state.

Quoted 12 lines
Quoted 12 lines

P1(atomic_t *v)

  {
	  atomic_add_unless(v, 1, 0);

}

P2(atomic_t *v)

  {
	  atomic_set(v, 0);

}

Quoted 6 lines
Quoted 4 lines

exists
(v=2)

So that one will still work? (that is, v=2 is forbidden)

you mean C,P1, P2 on 3 different CPU? I do not know much about LKMM, can explain the test case more explicit?

The 'C' is the language identifier, the 'atomic-set' is the litmus name.

The unnamed block give the initial conditions.

Pn blocks give code sequences for CPU n

The 'exists' clause is evaluated after all Pn blocks are done.

There's others in this thread that can point you to many papers and resources on these litmus test thingies.

So basically the initial value of @v is set to 1.

Then CPU-1 does atomic_add_unless(v, 1, 0)

     CPU-2 does atomic_set(v, 0)

If CPU1 goes first, it will see 1, which is not 0 and thus add 1 to 1 and obtains 2. Then CPU2 goes and writes 0, so the exist clause sees v==0 and doesn't observe 2.

The other way around, CPU-2 goes first, writes a 0, then CPU-1 goes and observes the 0, finds it matches 0 and doesn't add. Again, the exist clause will find 0 doesn't match 2.

This all goes unstuck if interleaved like:

	CPU-1			CPU-2

				xor	t0, t0
1:	ll	t0, v
	bez	t0, 2f
				sw	t0, v
	add	t0, t1
	sc	t0, v
	beqz t0, 1b
Quoted 4 lines

(sorry if I got the MIPS asm wrong; it's not something I normally write)

And the store-word from CPU-2 doesn't make the SC from CPU-1 fail.

loongson's llsc bug DOES NOT fail this litmus( we will not get V=2);

only speculative memory access from CPU-1 can "blind" CPU-1(here blind means do ll/sc
wrong), this speculative memory access can be observed corrently by CPU2. In this
case, sw from CPU-2 can get I , which can be observed by CPU-1, and clear llbit,then failed sc.

北京市海淀区中关村环保科技示范园龙芯产业园2号楼 100095电话: +86 (10) 62546668传真: +86 (10) 62600826www.loongson.cn本邮件及其附件含有龙芯中科技术有限公司的商业秘密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部 分地泄露、复制或散发)本邮件及其附件中的信息。如果您错收本邮件,请您立即电话或邮件通知发件人并删除本邮件。

This email and its attachments contain confidential information from Loongson Technology Corporation Limited, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

author maintainer
#20 lore raw

Re: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 31 lines · On Thu, Apr 25, 2019 at 08:51:17PM +0800, [email protected] wrote:
Quoted 26 lines

So basically the initial value of @v is set to 1.

Then CPU-1 does atomic_add_unless(v, 1, 0)

     CPU-2 does atomic_set(v, 0)

If CPU1 goes first, it will see 1, which is not 0 and thus add 1 to 1 and obtains 2. Then CPU2 goes and writes 0, so the exist clause sees v==0 and doesn't observe 2.

The other way around, CPU-2 goes first, writes a 0, then CPU-1 goes and observes the 0, finds it matches 0 and doesn't add. Again, the exist clause will find 0 doesn't match 2.

This all goes unstuck if interleaved like:

	CPU-1			CPU-2

				xor	t0, t0
1:	ll	t0, v
	bez	t0, 2f
				sw	t0, v
	add	t0, t1
	sc	t0, v
	beqz t0, 1b
Quoted 4 lines

(sorry if I got the MIPS asm wrong; it's not something I normally write)

And the store-word from CPU-2 doesn't make the SC from CPU-1 fail.

Quoted 6 lines

loongson's llsc bug DOES NOT fail this litmus( we will not get V=2);

only speculative memory access from CPU-1 can "blind" CPU-1(here blind means do ll/sc
wrong), this speculative memory access can be observed corrently by CPU2. In this
case, sw from CPU-2 can get I , which can be observed by CPU-1, and clear llbit,then failed sc.

I'm not following, suppose CPU-1 happens as a speculation (imagine whatever code is required to make that happen before). CPU-2 sw will cause I on CPU-1's ll but, as in the previous email, CPU-1 will continue as if it still has E and complete the SC.

That is; I'm just not seeing why this case would be different from two competing LL/SCs.

#21 lore raw

Re: Re: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 50 lines

-----原始邮件-----
发件人: "Peter Zijlstra" <[email protected]>
发送时间: 2019-04-25 21:31:05 (星期四)
收件人: [email protected]
抄送: "Paul Burton" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "[email protected]" <[email protected]>, "Huacai Chen" <[email protected]> 主题: Re: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

Quoted 31 lines · On Thu, Apr 25, 2019 at 08:51:17PM +0800, [email protected] wrote:
Quoted 26 lines

So basically the initial value of @v is set to 1.

Then CPU-1 does atomic_add_unless(v, 1, 0)

     CPU-2 does atomic_set(v, 0)

If CPU1 goes first, it will see 1, which is not 0 and thus add 1 to 1 and obtains 2. Then CPU2 goes and writes 0, so the exist clause sees v==0 and doesn't observe 2.

The other way around, CPU-2 goes first, writes a 0, then CPU-1 goes and observes the 0, finds it matches 0 and doesn't add. Again, the exist clause will find 0 doesn't match 2.

This all goes unstuck if interleaved like:

	CPU-1			CPU-2

				xor	t0, t0
1:	ll	t0, v
	bez	t0, 2f
				sw	t0, v
	add	t0, t1
	sc	t0, v
	beqz t0, 1b
Quoted 4 lines

(sorry if I got the MIPS asm wrong; it's not something I normally write)

And the store-word from CPU-2 doesn't make the SC from CPU-1 fail.

Quoted 6 lines

loongson's llsc bug DOES NOT fail this litmus( we will not get V=2);

only speculative memory access from CPU-1 can "blind" CPU-1(here blind means do ll/sc
wrong), this speculative memory access can be observed corrently by CPU2. In this
case, sw from CPU-2 can get I , which can be observed by CPU-1, and clear llbit,then failed sc.

Quoted 8 lines

I'm not following, suppose CPU-1 happens as a speculation (imagine whatever code is required to make that happen before). CPU-2 sw will cause I on CPU-1's ll but, as in the previous email, CPU-1 will continue as if it still has E and complete the SC.

That is; I'm just not seeing why this case would be different from two competing LL/SCs.

I get your point. I kept my eye on the sw from CPU-2, but forgot the speculative
mem access from CPU-1.

There is no difference bewteen this one and the former case.

=========================================================================

                       V = 1

    CPU-1                       CPU-2

                                xor  t0, t0

1: ll t0, V
beqz t0, 2f

/* if speculative mem
access kick cacheline of
V out, it can blind CPU-1
and make CPU-1 believe it
still hold E on V, and can
NOT see the sw from CPU-2
actually invalid V, which
should clear LLBit of CPU-1,
but not */
sw t0, V // just after sw, V = 0
addiu t0, t0, 1

sc t0, V
/* oops, sc write t0(2)
into V with LLBit */

/* get V=2 */
beqz t0, 1b
nop
2:
================================================================================

if speculative mem access *does not* kick out cache line of V, CPU-1 can see sw from CPU-2, and clear LLBit, which cause sc fail and retry, That's OK

北京市海淀区中关村环保科技示范园龙芯产业园2号楼 100095电话: +86 (10) 62546668传真: +86 (10) 62600826www.loongson.cn本邮件及其附件含有龙芯中科技术有限公司的商业秘密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部 分地泄露、复制或散发)本邮件及其附件中的信息。如果您错收本邮件,请您立即电话或邮件通知发件人并删除本邮件。

This email and its attachments contain confidential information from Loongson Technology Corporation Limited, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

author maintainer
#22 lore raw

Re: Re: Re: Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

(sorry for the delay, I got sidetracked elsewhere)

Quoted 80 lines · On Fri, Apr 26, 2019 at 10:57:20AM +0800, [email protected] wrote:
Quoted 41 lines

-----原始邮件-----

Quoted 31 lines · On Thu, Apr 25, 2019 at 08:51:17PM +0800, [email protected] wrote:
Quoted 26 lines

So basically the initial value of @v is set to 1.

Then CPU-1 does atomic_add_unless(v, 1, 0)

     CPU-2 does atomic_set(v, 0)

If CPU1 goes first, it will see 1, which is not 0 and thus add 1 to 1 and obtains 2. Then CPU2 goes and writes 0, so the exist clause sees v==0 and doesn't observe 2.

The other way around, CPU-2 goes first, writes a 0, then CPU-1 goes and observes the 0, finds it matches 0 and doesn't add. Again, the exist clause will find 0 doesn't match 2.

This all goes unstuck if interleaved like:

	CPU-1			CPU-2

				xor	t0, t0
1:	ll	t0, v
	bez	t0, 2f
				sw	t0, v
	add	t0, t1
	sc	t0, v
	beqz t0, 1b
Quoted 4 lines

(sorry if I got the MIPS asm wrong; it's not something I normally write)

And the store-word from CPU-2 doesn't make the SC from CPU-1 fail.

Quoted 6 lines

loongson's llsc bug DOES NOT fail this litmus( we will not get V=2);

only speculative memory access from CPU-1 can "blind" CPU-1(here blind means do ll/sc
wrong), this speculative memory access can be observed corrently by CPU2. In this
case, sw from CPU-2 can get I , which can be observed by CPU-1, and clear llbit,then failed sc.

Quoted 8 lines

I'm not following, suppose CPU-1 happens as a speculation (imagine whatever code is required to make that happen before). CPU-2 sw will cause I on CPU-1's ll but, as in the previous email, CPU-1 will continue as if it still has E and complete the SC.

That is; I'm just not seeing why this case would be different from two competing LL/SCs.

I get your point. I kept my eye on the sw from CPU-2, but forgot the speculative
mem access from CPU-1.

There is no difference bewteen this one and the former case.

=========================================================================

                       V = 1

    CPU-1                       CPU-2

                                xor  t0, t0

1: ll t0, V
beqz t0, 2f

/* if speculative mem
access kick cacheline of
V out, it can blind CPU-1
and make CPU-1 believe it
still hold E on V, and can
NOT see the sw from CPU-2
actually invalid V, which
should clear LLBit of CPU-1,
but not */
sw t0, V // just after sw, V = 0
addiu t0, t0, 1

sc t0, V
/* oops, sc write t0(2)
into V with LLBit */

Quoted 8 lines

/* get V=2 */
beqz t0, 1b
nop
2:
================================================================================

if speculative mem access *does not* kick out cache line of V, CPU-1 can see sw from CPU-2, and clear LLBit, which cause sc fail and retry, That's OK

OK; so do I understand it correctly that your CPU _can_ in fact fail that test and result in 2? If so I think I'm (finally) understanding :-)

#23 lore raw

Re: Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to [email protected]
Quoted 4 lines · On Wed, Apr 24, 2019 at 9:59 PM <[email protected]> wrote:

In one word, the bug only affect local cpu‘s ll/sc operation, and affect MP system.

If local_t is only ll/sc manipulated by current CPU, then no need fix it.

As to the whole MP vs UP issue:

Is this "guaranteed no problem on UP" true even in the presence of DMA? I'm _hoping_ some MIPS chips are starting to be coherent with DMA (but maybe DMA never participates in any coherency traffic that could trigger the bug even if the DMA _were_ to be coherent?).

Also, as Peter mentioned, we do depend on ll/sc also reacting to interrupts - again including on UP systems, of course. I assume that's always handled correctly, and that an interrupt will set the right state that the SC will not succeed?

Finally, it worries me a bit that the loongson_llsc_mb() things are all at the C level, so who knows what compiler behavior you'll see between the ll/sc inline asm, and the loongson_llsc_mb() generation. Hopefully not a lot (and presumably mostly just stack reloads etc that probably don't change any cache state), but I do wonder if the loongson quirk should be *inside* the asm rather than separate.

Linus

author maintainer
#24 lore raw

Re: [RFC][PATCH 2/5] mips/atomic: Fix loongson_llsc_mb() wreckage

In reply to Paul Burton
Quoted 10 lines · Paul Burton

Hi Peter,

Quoted 5 lines · Peter Zijlstra

The comment describing the loongson_llsc_mb() reorder case doesn't make any sense what so ever. Instruction re-ordering is not an SMP artifact, but rather a CPU local phenomenon. This means that _every_ LL/SC loop needs this barrier right in front to avoid the CPU from leaking a memop inside it.

Does it?

It does, however..

Quoted 7 lines

The Loongson bug being described here causes an sc to succeed erroneously if certain loads or stores are executed between the ll & associated sc, including speculatively. On a UP system there's no code running on other cores to race with us & cause our sc to fail - ie. sc should always succeed anyway, so if the bug hits & the sc succeeds what's the big deal? It would have succeeded anyway. At least that's my understanding based on discussions with Loongson engineers a while ago.

Ah! So that wasn't spelled out as such. This basically says that: Yes, it also screws with SC on UP, however the failure case is harmless.

(Also the comment with loongson_llsc_mb() seems incomplete in that it doesn't mention the SC can also erroneously fail; typically that isn't a problem because we'll just get an extra loop around and succeed eventually.)

That said; I'm not entirely sure. The reason we use LL/SC even for CPU-local variables is because of interrupts and the like. Would not a false positive be a problem if it _should_ fail because of an interrupt?

Quoted 4 lines

Having said that, if you have a strong preference for adding the barrier in UP systems anyway then I don't really object. It's not like anyone's likely to want to run a UP kernel on the affected systems, nevermind care about a miniscule performance impact.

It mostly all didn't make sense to me; and having a consistent recipie for LL/SC loops is easier on the person occasionally looking at all this (me, mostly :-).

(also, you should probably have a look at
include/asm-generic/bitops/atomic.h)

Quoted 4 lines

One possibility your change could benefit would be if someone ran Linux on a subset of cores & some non-Linux code on other cores, in which case there could be something to cause the sc to fail. I've no idea if that's something these Loongson systems ever do though.

Or a bunch of UP guests ?

Quoted 5 lines
Quoted 3 lines

For the branch speculation case; if futex_atomic_cmpxchg_inatomic() needs one at the bne branch target, then surely the normal __cmpxch_asmg() implementation does too. We cannot rely on the

s/cmpxch_asmg/cmpxchg_asm/

Typing hard :-)

author maintainer
#25 lore raw

RFC PATCH 3/5 mips/atomic: Optimize loongson3_llsc_mb()

In reply to Peter Zijlstra

Now that every single LL/SC loop has loongson_llsc_mb() in front, we can NO-OP smp_mb__before_llsc() in that case.

While there, remove the superfluous __smp_mb__before_llsc().

---
 arch/mips/include/asm/barrier.h |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/mips/include/asm/barrier.h +8 -3
@@ -221,15 +221,12 @@
 
 #ifdef CONFIG_CPU_CAVIUM_OCTEON
 #define smp_mb__before_llsc() smp_wmb()
-#define __smp_mb__before_llsc() __smp_wmb()
 /* Cause previous writes to become visible on all CPUs as soon as possible */
 #define nudge_writes() __asm__ __volatile__(".set push\n\t"		\
 					    ".set arch=octeon\n\t"	\
 					    "syncw\n\t"			\
 					    ".set pop" : : : "memory")
 #else
-#define smp_mb__before_llsc() smp_llsc_mb()
-#define __smp_mb__before_llsc() smp_llsc_mb()
 #define nudge_writes() mb()
 #endif
 
@@ -264,11 +261,19 @@
  *    This case affects all current Loongson 3 CPUs.
  */
 #ifdef CONFIG_CPU_LOONGSON3_WORKAROUNDS /* Loongson-3's LLSC workaround */
+#define smp_mb__before_llsc()	do { } while (0)
 #define loongson_llsc_mb()	__asm__ __volatile__("sync" : : :"memory")
 #else
 #define loongson_llsc_mb()	do { } while (0)
 #endif
 
+#ifndef smp_mb__before_llsc
+#define smp_mb__before_llsc() smp_llsc_mb()
+#endif
+
+#define __smp_mb__before_atomic()	smp_mb__before_llsc()
+#define __smp_mb__after_atomic()	smp_llsc_mb()
+
 static inline void sync_ginv(void)
 {
 	asm volatile("sync\t%0" :: "i"(STYPE_GINV));
author maintainer
#26 lore raw

RFC PATCH 4/5 mips/atomic: Fix smp_mb__{before,after}_atomic()

In reply to Peter Zijlstra

Recent probing at the Linux Kernel Memory Model uncovered a 'surprise'. Strongly ordered architectures where the atomic RmW primitive implies full memory ordering and smp_mb__{before,after}_atomic() are a simple barrier() (such as MIPS without WEAK_REORDERING_BEYOND_LLSC) fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it (surprisingly) does not provide a compiler barrier. This then allows the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended. Therefore strengthen the atomic RmW ops to include a compiler barrier (when they provide order).

---
 arch/mips/include/asm/atomic.h  |   16 +++++++--------
 arch/mips/include/asm/barrier.h |   15 +++++++++-----
 arch/mips/include/asm/bitops.h  |   42 +++++++++++++++++++++++-----------------
 arch/mips/include/asm/cmpxchg.h |    6 ++---
 4 files changed, 46 insertions(+), 33 deletions(-)
arch/mips/include/asm/atomic.h +8 -8
@@ -68,7 +68,7 @@ static __inline__ void atomic_##op(int i
 		"\t" __scbeqz "	%0, 1b					\n"   \
 		"	.set	pop					\n"   \
 		: "=&r" (temp), "+" GCC_OFF_SMALL_ASM() (v->counter)	      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -98,7 +98,7 @@ static __inline__ int atomic_##op##_retu
 		"	.set	pop					\n"   \
 		: "=&r" (result), "=&r" (temp),				      \
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)			      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -132,7 +132,7 @@ static __inline__ int atomic_fetch_##op#
 		"	move	%0, %1					\n"   \
 		: "=&r" (result), "=&r" (temp),				      \
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)			      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -210,7 +210,7 @@ static __inline__ int atomic_sub_if_posi
 		"	.set	pop					\n"
 		: "=&r" (result), "=&r" (temp),
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)
-		: "Ir" (i));
+		: "Ir" (i) : __LLSC_CLOBBER);
 
 		if (!IS_ENABLED(CONFIG_SMP))
 			loongson_llsc_mb();
@@ -277,7 +277,7 @@ static __inline__ void atomic64_##op(lon
 		"\t" __scbeqz "	%0, 1b					\n"   \
 		"	.set	pop					\n"   \
 		: "=&r" (temp), "+" GCC_OFF_SMALL_ASM() (v->counter)	      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -307,7 +307,7 @@ static __inline__ long atomic64_##op##_r
 		"	.set	pop					\n"   \
 		: "=&r" (result), "=&r" (temp),				      \
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)			      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -341,7 +341,7 @@ static __inline__ long atomic64_fetch_##
 		"	.set	pop					\n"   \
 		: "=&r" (result), "=&r" (temp),				      \
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)			      \
-		: "Ir" (i));						      \
+		: "Ir" (i) : __LLSC_CLOBBER);				      \
 	} else {							      \
 		unsigned long flags;					      \
 									      \
@@ -417,7 +417,7 @@ static __inline__ long atomic64_sub_if_p
 		"	.set	pop					\n"
 		: "=&r" (result), "=&r" (temp),
 		  "+" GCC_OFF_SMALL_ASM() (v->counter)
-		: "Ir" (i));
+		: "Ir" (i) : __LLSC_CLOBBER);
 
 		if (!IS_ENABLED(CONFIG_SMP))
 			loongson_llsc_mb();
arch/mips/include/asm/barrier.h +10 -5
@@ -211,14 +211,22 @@
 #define __smp_wmb()	barrier()
 #endif
 
+/*
+ * When LL/SC does imply order, it must also be a compiler barrier to avoid the
+ * compiler from reordering where the CPU will not. When it does not imply
+ * order, the compiler is also free to reorder across the LL/SC loop and
+ * ordering will be done by smp_llsc_mb() and friends.
+ */
 #if defined(CONFIG_WEAK_REORDERING_BEYOND_LLSC) && defined(CONFIG_SMP)
 #define __WEAK_LLSC_MB		"	sync	\n"
+#define smp_llsc_mb()		__asm__ __volatile__(__WEAK_LLSC_MB : : :"memory")
+#define __LLSC_CLOBBER
 #else
 #define __WEAK_LLSC_MB		"		\n"
+#define smp_llsc_mb()		do { } while (0)
+#define __LLSC_CLOBBER		"memory"
 #endif
 
-#define smp_llsc_mb()	__asm__ __volatile__(__WEAK_LLSC_MB : : :"memory")
-
 #ifdef CONFIG_CPU_CAVIUM_OCTEON
 #define smp_mb__before_llsc() smp_wmb()
 /* Cause previous writes to become visible on all CPUs as soon as possible */
@@ -230,9 +238,6 @@
 #define nudge_writes() mb()
 #endif
 
-#define __smp_mb__before_atomic()	__smp_mb__before_llsc()
-#define __smp_mb__after_atomic()	smp_llsc_mb()
-
 /*
  * Some Loongson 3 CPUs have a bug wherein execution of a memory access (load,
  * store or pref) in between an ll & sc can cause the sc instruction to
arch/mips/include/asm/bitops.h +25 -17

144 lines not on this page · raw message

arch/mips/include/asm/cmpxchg.h +3 -3
@@ -61,7 +61,7 @@ extern unsigned long __xchg_called_with_
 		"	.set	pop				\n"	\
 		: "=&r" (__ret), "=" GCC_OFF_SMALL_ASM() (*m)		\
 		: GCC_OFF_SMALL_ASM() (*m), "Jr" (val)			\
-		: "memory");						\
+		: __LLSC_COBBER);					\
 	} else {							\
 		unsigned long __flags;					\
 									\
@@ -134,8 +134,8 @@ static inline unsigned long __xchg(volat
 		"	.set	pop				\n"	\
 		"2:						\n"	\
 		: "=&r" (__ret), "=" GCC_OFF_SMALL_ASM() (*m)		\
-		: GCC_OFF_SMALL_ASM() (*m), "Jr" (old), "Jr" (new)		\
-		: "memory");						\
+		: GCC_OFF_SMALL_ASM() (*m), "Jr" (old), "Jr" (new)	\
+		: __LLSC_CLOBBER);					\
 		loongson_llsc_mb();					\
 	} else {							\
 		unsigned long __flags;					\
#27 lore raw

Re: [RFC][PATCH 4/5] mips/atomic: Fix smp_mb__{before,after}_atomic()

Hi Peter,

arch/mips/include/asm/barrier.h line 243

Quoted 12 lines · Peter Zijlstra
arch/mips/include/asm/barrier.h +0 -3
@@ -230,9 +238,6 @@
 #define nudge_writes() mb()
 #endif
 
-#define __smp_mb__before_atomic()	__smp_mb__before_llsc()
-#define __smp_mb__after_atomic()	smp_llsc_mb()
-
 /*
  * Some Loongson 3 CPUs have a bug wherein execution of a memory access (load,
  * store or pref) in between an ll & sc can cause the sc instruction to

I think this bit should be part of patch 3, where you currently add a second definition of these 2 macros.

Otherwise this one looks reasonable to me.

Thanks,
Paul

author maintainer
#28 lore raw

Re: [RFC][PATCH 4/5] mips/atomic: Fix smp_mb__{before,after}_atomic()

Quoted 9 lines · Paul Burton

Hi Peter,

Quoted 6 lines · Peter Zijlstra
arch/mips/include/asm/barrier.h +0 -0
@@ -230,9 +238,6 @@
 #define nudge_writes() mb()
 #endif
 
Quoted 9 lines

arch/mips/include/asm/barrier.h line 243

Quoted 6 lines
arch/mips/include/asm/barrier.h +0 -3
-#define __smp_mb__before_atomic()	__smp_mb__before_llsc()
-#define __smp_mb__after_atomic()	smp_llsc_mb()
-
 /*
  * Some Loongson 3 CPUs have a bug wherein execution of a memory access (load,
  * store or pref) in between an ll & sc can cause the sc instruction to

I think this bit should be part of patch 3, where you currently add a second definition of these 2 macros.

Whoops, indeed.

Quoted 1 line

Otherwise this one looks reasonable to me.

Great, thanks!

author maintainer
#29 lore raw

RFC PATCH 5/5 x86/atomic: Fix smp_mb__{before,after}_atomic()

In reply to Peter Zijlstra

Recent probing at the Linux Kernel Memory Model uncovered a 'surprise'. Strongly ordered architectures where the atomic RmW primitive implies full memory ordering and smp_mb__{before,after}_atomic() are a simple barrier() (such as x86) fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it (surprisingly) does not provide a compiler barrier. This then allows the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended. Therefore strengthen the atomic RmW ops to include a compiler barrier.

NOTE: atomic_{or,and,xor} and the bitops already had the compiler barrier.

---
 Documentation/atomic_t.txt         |    3 +++
 arch/x86/include/asm/atomic.h      |    8 ++++----
 arch/x86/include/asm/atomic64_64.h |    8 ++++----
 arch/x86/include/asm/barrier.h     |    4 ++--
 4 files changed, 13 insertions(+), 10 deletions(-)
Documentation/atomic_t.txt +3 -0
@@ -194,6 +194,9 @@ These helper barriers exist because arch
 ordering on their SMP atomic primitives. For example our TSO architectures
 provide full ordered atomics and these barriers are no-ops.
 
+NOTE: when the atomic RmW ops are fully ordered, they should also imply a
+compiler barrier.
+
 Thus:
 
   atomic_fetch_add();
arch/x86/include/asm/atomic.h +4 -4
@@ -54,7 +54,7 @@ static __always_inline void arch_atomic_
 {
 	asm volatile(LOCK_PREFIX "addl %1,%0"
 		     : "+m" (v->counter)
-		     : "ir" (i));
+		     : "ir" (i) : "memory");
 }
 
 /**
@@ -68,7 +68,7 @@ static __always_inline void arch_atomic_
 {
 	asm volatile(LOCK_PREFIX "subl %1,%0"
 		     : "+m" (v->counter)
-		     : "ir" (i));
+		     : "ir" (i) : "memory");
 }
 
 /**
@@ -95,7 +95,7 @@ static __always_inline bool arch_atomic_
 static __always_inline void arch_atomic_inc(atomic_t *v)
 {
 	asm volatile(LOCK_PREFIX "incl %0"
-		     : "+m" (v->counter));
+		     : "+m" (v->counter) :: "memory");
 }
 #define arch_atomic_inc arch_atomic_inc
 
@@ -108,7 +108,7 @@ static __always_inline void arch_atomic_
 static __always_inline void arch_atomic_dec(atomic_t *v)
 {
 	asm volatile(LOCK_PREFIX "decl %0"
-		     : "+m" (v->counter));
+		     : "+m" (v->counter) :: "memory");
 }
 #define arch_atomic_dec arch_atomic_dec
 
arch/x86/include/asm/atomic64_64.h +4 -4
@@ -45,7 +45,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "addq %1,%0"
 		     : "=m" (v->counter)
-		     : "er" (i), "m" (v->counter));
+		     : "er" (i), "m" (v->counter) : "memory");
 }
 
 /**
@@ -59,7 +59,7 @@ static inline void arch_atomic64_sub(lon
 {
 	asm volatile(LOCK_PREFIX "subq %1,%0"
 		     : "=m" (v->counter)
-		     : "er" (i), "m" (v->counter));
+		     : "er" (i), "m" (v->counter) : "memory");
 }
 
 /**
@@ -87,7 +87,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "incq %0"
 		     : "=m" (v->counter)
-		     : "m" (v->counter));
+		     : "m" (v->counter) : "memory");
 }
 #define arch_atomic64_inc arch_atomic64_inc
 
@@ -101,7 +101,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "decq %0"
 		     : "=m" (v->counter)
-		     : "m" (v->counter));
+		     : "m" (v->counter) : "memory");
 }
 #define arch_atomic64_dec arch_atomic64_dec
 
arch/x86/include/asm/barrier.h +2 -2
@@ -80,8 +80,8 @@ do {									\
 })
 
 /* Atomic operations are already serializing on x86 */
-#define __smp_mb__before_atomic()	barrier()
-#define __smp_mb__after_atomic()	barrier()
+#define __smp_mb__before_atomic()	do { } while (0)
+#define __smp_mb__after_atomic()	do { } while (0)
 
 #include <asm-generic/barrier.h>
 
Acked-by on v1 5/5
#30 lore raw

Re: [RFC][PATCH 5/5] x86/atomic: Fix smp_mb__{before,after}_atomic()

Quoted 43 lines · Peter Zijlstra

Recent probing at the Linux Kernel Memory Model uncovered a 'surprise'. Strongly ordered architectures where the atomic RmW primitive implies full memory ordering and smp_mb__{before,after}_atomic() are a simple barrier() (such as x86) fail for:

	*x = 1;
	atomic_inc(u);
	smp_mb__after_atomic();
	r0 = *y;

Because, while the atomic_inc() implies memory order, it (surprisingly) does not provide a compiler barrier. This then allows the compiler to re-order like so:

	atomic_inc(u);
	*x = 1;
	smp_mb__after_atomic();
	r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

	atomic_inc(u);
	r0 = *y;
	*x = 1;

And this very much was not intended. Therefore strengthen the atomic RmW ops to include a compiler barrier.

NOTE: atomic_{or,and,xor} and the bitops already had the compiler barrier.

---
 Documentation/atomic_t.txt         |    3 +++
 arch/x86/include/asm/atomic.h      |    8 ++++----
 arch/x86/include/asm/atomic64_64.h |    8 ++++----
 arch/x86/include/asm/barrier.h     |    4 ++--
 4 files changed, 13 insertions(+), 10 deletions(-)

Documentation/atomic_t.txt These helper barriers exist because arch · line 198

Quoted 5 lines
Documentation/atomic_t.txt +2 -0
@@ -194,6 +194,9 @@ These helper barriers exist because arch
 ordering on their SMP atomic primitives. For example our TSO architectures
 provide full ordered atomics and these barriers are no-ops.
 
+NOTE: when the atomic RmW ops are fully ordered, they should also imply a
+compiler barrier.

This is also the case for acquire and release variants, so we should probably include those as well to avoid the implication that they don't need the "memory" clobber.

Quoted 83 lines
Documentation/atomic_t.txt +1 -0
+
 Thus:
 
   atomic_fetch_add();
arch/x86/include/asm/atomic.h +4 -4
@@ -54,7 +54,7 @@ static __always_inline void arch_atomic_
 {
 	asm volatile(LOCK_PREFIX "addl %1,%0"
 		     : "+m" (v->counter)
-		     : "ir" (i));
+		     : "ir" (i) : "memory");
 }
 
 /**
@@ -68,7 +68,7 @@ static __always_inline void arch_atomic_
 {
 	asm volatile(LOCK_PREFIX "subl %1,%0"
 		     : "+m" (v->counter)
-		     : "ir" (i));
+		     : "ir" (i) : "memory");
 }
 
 /**
@@ -95,7 +95,7 @@ static __always_inline bool arch_atomic_
 static __always_inline void arch_atomic_inc(atomic_t *v)
 {
 	asm volatile(LOCK_PREFIX "incl %0"
-		     : "+m" (v->counter));
+		     : "+m" (v->counter) :: "memory");
 }
 #define arch_atomic_inc arch_atomic_inc
 
@@ -108,7 +108,7 @@ static __always_inline void arch_atomic_
 static __always_inline void arch_atomic_dec(atomic_t *v)
 {
 	asm volatile(LOCK_PREFIX "decl %0"
-		     : "+m" (v->counter));
+		     : "+m" (v->counter) :: "memory");
 }
 #define arch_atomic_dec arch_atomic_dec
 
arch/x86/include/asm/atomic64_64.h +4 -4
@@ -45,7 +45,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "addq %1,%0"
 		     : "=m" (v->counter)
-		     : "er" (i), "m" (v->counter));
+		     : "er" (i), "m" (v->counter) : "memory");
 }
 
 /**
@@ -59,7 +59,7 @@ static inline void arch_atomic64_sub(lon
 {
 	asm volatile(LOCK_PREFIX "subq %1,%0"
 		     : "=m" (v->counter)
-		     : "er" (i), "m" (v->counter));
+		     : "er" (i), "m" (v->counter) : "memory");
 }
 
 /**
@@ -87,7 +87,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "incq %0"
 		     : "=m" (v->counter)
-		     : "m" (v->counter));
+		     : "m" (v->counter) : "memory");
 }
 #define arch_atomic64_inc arch_atomic64_inc
 
@@ -101,7 +101,7 @@ static __always_inline void arch_atomic6
 {
 	asm volatile(LOCK_PREFIX "decq %0"
 		     : "=m" (v->counter)
-		     : "m" (v->counter));
+		     : "m" (v->counter) : "memory");
 }
 #define arch_atomic64_dec arch_atomic64_dec
 

arch/x86/include/asm/barrier.h do { \ · line 84

Quoted 7 lines
arch/x86/include/asm/barrier.h +2 -2
@@ -80,8 +80,8 @@ do {									\
 })
 
 /* Atomic operations are already serializing on x86 */
-#define __smp_mb__before_atomic()	barrier()
-#define __smp_mb__after_atomic()	barrier()
+#define __smp_mb__before_atomic()	do { } while (0)
+#define __smp_mb__after_atomic()	do { } while (0)

I do wonder whether or not it would be cleaner to have a Kconfig option such as ARCH_HAS_ORDERED_ATOMIC_RMW which, when selected, NOPs these barrier macros out in the generic code. Then it's a requirement that you have the "memory" clobber on your relaxed/non-returning atomics if you select the option.

But that needn't hold this up, because your patch looks good to me modulo the earlier, minor documentation nit.

Will