Skip to contentSkip to conversation

sparc64 cmpxchg is not a full memory barrier anymore ?

DISCUSSION Mathieu Desnoyers 2 participants 7 messages Oct 22, 2009 – Nov 16, 2009

lore
Share & tools
lore
#0 lore raw

Hi David,

I took a look at the current sparc64 cmpxchg implementation in the Linux kernel and stumbled on this commit:

commit 293666b7a17cb7a389fc274980439212386a19c4

Author: David S. Miller <[email protected]>
Date:   Sat Nov 15 13:33:25 2008 -0800

sparc64: Stop using memory barriers for atomics and locks.

The kernel always executes in the TSO memory model now, so none of this stuff is necessary any more.

With helpful feedback from Nick Piggin.

Reading p. 152 A.9 Compare and Swap, I see that the cas instruction does not imply a memory barrier.

Reading http://docs.sun.com/app/docs/doc/801-6678/6i11oelck?a=view

I see that:

"TSO guarantees that the store, FLUSH, and atomic load-store instructions of all processors appear to be executed by memory serially in a single order called the memory order. Furthermore, the sequence of store, FLUSH, and atomic load-store instructions in the memory order for a given processor is identical to the sequence in which they were issued by the processor."

So it provides no guarantee whatsoever wrt loads. However, reading Documentation/memory-barriers.txt:

"Whilst they are technically interprocessor interaction considerations, atomic operations are noted specially as some of them imply full memory barriers and some don't, but they're very heavily relied on as a group throughout the kernel." -> cmpxchg is part of them.

The same applies to the other atomic instructions we find in this list. How is the correct ordering of loads wrt to cmxchg (and other atomic ops) still ensured by this modification?

Thanks,

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#1 lore raw

Sparc64 support added to Userspace RCU

By the way, I just added sparc64 support to the Userspace RCU library, available at: git://lttng.org/userspace-rcu.git

Interesting files concerning sparc64 are:

urcu/arch_sparc64.h
urcu/uatomic_arch_sparc64.h

Direct links:
http://www.lttng.org/cgi-bin/gitweb.cgi?p=userspace-rcu.git;a=blob;f=urcu/arch_sparc64.h;hb=HEAD
http://www.lttng.org/cgi-bin/gitweb.cgi?p=userspace-rcu.git;a=blob;f=urcu/uatomic_arch_sparc64.h;hb=HEAD

Please note that this code targets user-space, not kernel-space.

Feedback is welcome,

Thanks,

Mathieu

Quoted 51 lines · * Mathieu Desnoyers ([email protected]) wrote:

Hi David,

I took a look at the current sparc64 cmpxchg implementation in the Linux kernel and stumbled on this commit:

commit 293666b7a17cb7a389fc274980439212386a19c4

Author: David S. Miller <[email protected]>
Date:   Sat Nov 15 13:33:25 2008 -0800

sparc64: Stop using memory barriers for atomics and locks.

The kernel always executes in the TSO memory model now, so none of this stuff is necessary any more.

With helpful feedback from Nick Piggin.

Reading p. 152 A.9 Compare and Swap, I see that the cas instruction does not imply a memory barrier.

Reading http://docs.sun.com/app/docs/doc/801-6678/6i11oelck?a=view

I see that:

"TSO guarantees that the store, FLUSH, and atomic load-store instructions of all processors appear to be executed by memory serially in a single order called the memory order. Furthermore, the sequence of store, FLUSH, and atomic load-store instructions in the memory order for a given processor is identical to the sequence in which they were issued by the processor."

So it provides no guarantee whatsoever wrt loads. However, reading Documentation/memory-barriers.txt:

"Whilst they are technically interprocessor interaction considerations, atomic operations are noted specially as some of them imply full memory barriers and some don't, but they're very heavily relied on as a group throughout the kernel." -> cmpxchg is part of them.

The same applies to the other atomic instructions we find in this list. How is the correct ordering of loads wrt to cmxchg (and other atomic ops) still ensured by this modification?

Thanks,

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#2 lore raw

Re: Sparc64 support added to Userspace RCU

From: Mathieu Desnoyers <[email protected]>
Date: Thu, 22 Oct 2009 15:57:53 -0400

Quoted 1 line

Feedback is welcome,

__sparc_v8__ does not mean 64-bit longs or pointers as implied here:

#ifndef __SIZEOF_LONG__
#if (defined(__sparc_v8__) || defined(__sparc_v9__)) #define __SIZEOF_LONG__ 8
#else
#define __SIZEOF_LONG__ 4
#endif
#endif

"v8" was the last 32-bit standard of the cpu architecture.

Otherwise looks fine.

Although this port won't be very useful until it can be built in 32-bit mode. As most applications on sparc64 systems are still 32-bit. Like on powerpc, people only rarely build things 64-bit on sparc64.

You need to build with "-Wa,-Av9a" on the GCC command line for that case so that the assembler allows the v9 instructions emitted by your macros, even when building for 32-bit.

Hope this helps.

#3 lore raw

Re: Sparc64 support added to Userspace RCU

Quoted 24 lines · * David Miller ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>
Date: Thu, 22 Oct 2009 15:57:53 -0400

Quoted 1 line

Feedback is welcome,

__sparc_v8__ does not mean 64-bit longs or pointers as implied here:

#ifndef __SIZEOF_LONG__
#if (defined(__sparc_v8__) || defined(__sparc_v9__)) #define __SIZEOF_LONG__ 8
#else
#define __SIZEOF_LONG__ 4
#endif
#endif

"v8" was the last 32-bit standard of the cpu architecture.

Otherwise looks fine.

Although this port won't be very useful until it can be built in 32-bit mode. As most applications on sparc64 systems are still 32-bit. Like on powerpc, people only rarely build things 64-bit on sparc64.

Quoted 5 lines

You need to build with "-Wa,-Av9a" on the GCC command line for that case so that the assembler allows the v9 instructions emitted by your macros, even when building for 32-bit.

Hope this helps.

Yes ! Thanks a lot for this information. I'll push the following fix in my tree.

Mathieu

README +4 -1
index 883b808..ca23d43 100644
@@ -18,11 +18,14 @@ BUILDING
 		Forcing a 32-bit build with 386 backward compatibility:
 		* CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu
 
+		Forcing a 32-bit build for Sparcv9 (typical for Sparc v9)
+		* CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure
+
 ARCHITECTURES SUPPORTED
 -----------------------
 
 Currently, x86 (i386, i486, i586, i686), x86 64-bit, PowerPC 32/64, S390, S390x
-and Sparc64 are supported. Only tested on Linux so far, but should
+and Sparcv9 32/64 are supported. Only tested on Linux so far, but should
 theoretically work on other operating systems.
 
 QUICK START GUIDE
urcu/uatomic_arch_sparc64.h +1 -1
index 35b9ac2..28829d2 100644
@@ -24,7 +24,7 @@
 #include <urcu/system.h>
 
 #ifndef __SIZEOF_LONG__
-#if (defined(__sparc_v8__) || defined(__sparc_v9__))
+#ifdef __LP64__
 #define __SIZEOF_LONG__ 8
 #else
 #define __SIZEOF_LONG__ 4
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#4 lore raw
In reply to Mathieu Desnoyers

From: Mathieu Desnoyers <[email protected]>
Date: Thu, 22 Oct 2009 14:32:42 -0400

Quoted 3 lines

The same applies to the other atomic instructions we find in this list. How is the correct ordering of loads wrt to cmxchg (and other atomic ops) still ensured by this modification?

All actual sparc64 chips implement more strict ordering than the V9 specification permits. The memory barriers were just nops and actually not doing anything more than the chip already does for us.

#5 lore raw
Quoted 11 lines · * David Miller ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>
Date: Thu, 22 Oct 2009 14:32:42 -0400

Quoted 3 lines

The same applies to the other atomic instructions we find in this list. How is the correct ordering of loads wrt to cmxchg (and other atomic ops) still ensured by this modification?

All actual sparc64 chips implement more strict ordering than the V9 specification permits. The memory barriers were just nops and actually not doing anything more than the chip already does for us.

OK. Perhaps adding a comment to that effect near sparc mb() implementation would be appropriate ?

Thanks,

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#6 lore raw

From: Mathieu Desnoyers <[email protected]>
Date: Fri, 23 Oct 2009 08:33:27 -0400

Quoted 2 lines

OK. Perhaps adding a comment to that effect near sparc mb() implementation would be appropriate ?

Sure, I'll take care of that when I get back from Japan.