Skip to contentSkip to conversation

Intel Memory Ordering White Paper

DISCUSSION Jesse Barnes 10 participants 22 messages Sep 7, 2007 – Sep 19, 2007

lore
Share & tools
lore
#1 lore raw
Quoted 6 lines · Jesse Barnes

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

So, can we finally noop smp_rmb and smp_wmb on x86?

include/asm-i386/system.h +1 -1
Index: linux-2.6/include/asm-i386/system.h
===================================================================
@@ -286,7 +286,7 @@ static inline unsigned long get_limit(un
 
 #ifdef CONFIG_SMP
 #define smp_mb()	mb()
-#define smp_rmb()	rmb()
+#define smp_rmb()	barrier()
 #define smp_wmb()	wmb()
 #define smp_read_barrier_depends()	read_barrier_depends()
 #define set_mb(var, value) do { (void) xchg(&var, value); } while (0)
include/asm-x86_64/system.h +2 -2
Index: linux-2.6/include/asm-x86_64/system.h
===================================================================
@@ -141,8 +141,8 @@ static inline void write_cr8(unsigned lo
 
 #ifdef CONFIG_SMP
 #define smp_mb()	mb()
-#define smp_rmb()	rmb()
-#define smp_wmb()	wmb()
+#define smp_rmb()	barrier()
+#define smp_wmb()	barrier()
 #define smp_read_barrier_depends()	do {} while(0)
 #else
 #define smp_mb()	barrier()
#2 lore raw
Quoted 2 lines · Nick Piggin

So, can we finally noop smp_rmb and smp_wmb on x86?

Did AMD already release their version? If so, we should probably add a commit that does that in somewhat early 2.6.24 rc, and add the pointers to the whitepapers in the commit message.

Linus

#3 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 1 line · Nick Piggin

So, can we finally noop smp_rmb and smp_wmb on x86?

Did AMD already release their version? If so, we should probably add a commit that does that in somewhat early 2.6.24 rc, and add the pointers to the whitepapers in the commit message.

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24593.pdf

AMD64 Architecture Programmer's Manual Volume 2: System Programming section 7.2: Multiprocessor Memory Access Ordering, a paragraph on the first page says

"Loads do not pass previous loads (loads are not re-ordered). Stores do not pass previous stores (stores are not re-ordered)"

So, yes, it should be easy to do.

#4 lore raw
Quoted 11 lines · Nick Piggin
Quoted 6 lines · Linus Torvalds
Quoted 1 line · Nick Piggin

So, can we finally noop smp_rmb and smp_wmb on x86?

Did AMD already release their version? If so, we should probably add a commit that does that in somewhat early 2.6.24 rc, and add the pointers to the whitepapers in the commit message.

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/245
93.pdf

Quoted 8 lines

AMD64 Architecture Programmer's Manual Volume 2: System Programming section 7.2: Multiprocessor Memory Access Ordering, a paragraph on the first page says

"Loads do not pass previous loads (loads are not re-ordered). Stores do not pass previous stores (stores are not re-ordered)"

So, yes, it should be easy to do.

There is some suggestion in the source code that non-temporal stores (movntq) are weakly ordered. But AFAIKS from the documents, it is ordered when operating on wb memory. What's the situation there?

I've also heard that string operations do not follow the normal ordering, but that's just with respect to individual loads/stores in the one operation, I hope? And they will still follow ordering rules WRT surrounding loads and stores?

#5 lore raw
Quoted 3 lines · Nick Piggin

There is some suggestion in the source code that non-temporal stores (movntq) are weakly ordered. But AFAIKS from the documents, it is ordered when operating on wb memory. What's the situation there?

Sorry, it looks from the AMD document like nontemporal stores to wb memory can go out of order. It is a bit hard to decipher what the types mean.

If this is the case, we can either retain the sfence in smp_wmb(), or noop it, and put explicit sfences around any place that performs nontemporal stores...

Anyway, the lfence should be able to go away without so much trouble.

#6 lore raw
Quoted 8 lines · On Friday 07 September 2007 20:13:12 Nick Piggin wrote:
Quoted 3 lines · Nick Piggin

There is some suggestion in the source code that non-temporal stores (movntq) are weakly ordered. But AFAIKS from the documents, it is ordered when operating on wb memory. What's the situation there?

Sorry, it looks from the AMD document like nontemporal stores to wb memory can go out of order.

Yes, that is how NT stores are defined.

Quoted 3 lines

If this is the case, we can either retain the sfence in smp_wmb(), or noop it, and put explicit sfences around any place that performs nontemporal stores...

We do this already, but in most cases it doesn't matter anyways. We AFAIK do not rely on any ordering for copy_*_user for example. There are not that many users of nt so it's not a huge issue.

Quoted 2 lines

Anyway, the lfence should be able to go away without so much trouble.

You mean sfence? lfence in rmb is definitely needed.

sfence on x86-64 is not strictly needed, but also shouldn't hurt very much so I always kept it in.

-Andi

#7 lore raw
Quoted 11 lines · Andi Kleen
Quoted 7 lines · On Friday 07 September 2007 20:13:12 Nick Piggin wrote:
Quoted 3 lines · Nick Piggin

There is some suggestion in the source code that non-temporal stores (movntq) are weakly ordered. But AFAIKS from the documents, it is ordered when operating on wb memory. What's the situation there?

Sorry, it looks from the AMD document like nontemporal stores to wb memory can go out of order.

Yes, that is how NT stores are defined.

Quoted 7 lines
Quoted 3 lines

If this is the case, we can either retain the sfence in smp_wmb(), or noop it, and put explicit sfences around any place that performs nontemporal stores...

We do this already, but in most cases it doesn't matter anyways. We AFAIK do not rely on any ordering for copy_*_user for example. There are not that many users of nt so it's not a huge issue.

OK, but we just don't want to be making lots of little exceptions. For bulk copies, I don't see it being a big issue to always sfence around them (it would be a relatively minor cost).

Quoted 3 lines
Quoted 1 line

Anyway, the lfence should be able to go away without so much trouble.

You mean sfence? lfence in rmb is definitely needed.

I mean lfence in smp_rmb().

Quoted 4 lines

sfence on x86-64 is not strictly needed, but also shouldn't hurt very much so I always kept it in.

-Andi

#8 lore raw
Quoted 6 lines · On Friday 07 September 2007 21:57:35 Nick Piggin wrote:
Quoted 4 lines
Quoted 2 lines

Anyway, the lfence should be able to go away without so much trouble.

You mean sfence? lfence in rmb is definitely needed.

I mean lfence in smp_rmb().

One point of rmb is to stop speculative loads and I don't think we can get that without lfence.

-Andi

#9 lore raw
Quoted 9 lines · Andi Kleen
Quoted 5 lines · On Friday 07 September 2007 21:57:35 Nick Piggin wrote:
Quoted 3 lines
Quoted 1 line

Anyway, the lfence should be able to go away without so much trouble.

You mean sfence? lfence in rmb is definitely needed.

I mean lfence in smp_rmb().

One point of rmb is to stop speculative loads and I don't think we can get that without lfence.

smp_rmb() should not need to do anything because loads are done in order anyway. Both AMD and Intel have committed to this now.

The important point is that they *appear* to be done in order. AFAIK, the CPUs can still do speculative and out of order loads, but throw out the results if they could be wrong.

#10 lore raw
Quoted 6 lines · Nick Piggin

smp_rmb() should not need to do anything because loads are done in order anyway. Both AMD and Intel have committed to this now.

The important point is that they *appear* to be done in order. AFAIK, the CPUs can still do speculative and out of order loads, but throw out the results if they could be wrong.

Is there anything even semiofficial from VIA? Not that the x86 architecture isn't pretty much definable as the AMD-Intel consensus...

-hpa

#11 lore raw
In reply to Nick Piggin
Quoted 4 lines · Nick Piggin

I've also heard that string operations do not follow the normal ordering, but that's just with respect to individual loads/stores in the one operation, I hope? And they will still follow ordering rules WRT surrounding loads and stores?

see section 7.2.3 of intel volume 3A...

"Code dependent upon sequential store ordering should not use the string operations for the entire data structure to be stored. Data and semaphores should be separated. Order dependent code should use a discrete semaphore uniquely stored to after any string operations to allow correctly ordered data to be seen by all processors."

i think we need sfence after things like copy_page, clear_page, and possibly copy_user... at least on intel processors with fast strings option enabled.

-dean

#12 lore raw
Quoted 7 lines · dean gaudet
Quoted 4 lines · Nick Piggin

I've also heard that string operations do not follow the normal ordering, but that's just with respect to individual loads/stores in the one operation, I hope? And they will still follow ordering rules WRT surrounding loads and stores?

Quoted 5 lines

see section 7.2.3 of intel volume 3A...

"Code dependent upon sequential store ordering should not use the string operations for the entire data structure to be stored. Data and semaphores should be separated. Order dependent code should use a discrete semaphore uniquely stored to after any string operations to allow correctly ordered data to be seen by all processors."

i think we need sfence after things like copy_page, clear_page, and possibly copy_user... at least on intel processors with fast strings option enabled.

I do not think. I believe that authors are trying to say that

struct { uint8 lock; uint8 data; } x;

lea (x.data),%edi
mov $2,%ecx
std
rep movsb

to set both data and lock does not guarantee that x.lock will be set after x.data and that you should do

lea (x.data),%edi
std
movsb
movsb # or mov (%esi),%al; mov %al,(%edi), but movsb looks discrete enough to me

instead (and yes, I know that my example is silly).
Petr

#13 lore raw
Quoted 33 lines · Petr Vandrovec
Quoted 11 lines · dean gaudet
Quoted 6 lines · Nick Piggin

I've also heard that string operations do not follow the normal ordering, but
that's just with respect to individual loads/stores in the one operation, I
hope? And they will still follow ordering rules WRT surrounding loads and stores?

see section 7.2.3 of intel volume 3A...

Quoted 8 lines

"Code dependent upon sequential store ordering should not use the string operations for the entire data structure to be stored. Data and semaphores should be separated. Order dependent code should use a discrete semaphore uniquely stored to after any string operations to allow correctly ordered data to be seen by all processors."

i think we need sfence after things like copy_page, clear_page, and possibly copy_user... at least on intel processors with fast strings option enabled.

I do not think. I believe that authors are trying to say that

struct { uint8 lock; uint8 data; } x;

lea (x.data),%edi
mov $2,%ecx
std
rep movsb

to set both data and lock does not guarantee that x.lock will be set after x.data and that you should do

Quoted 7 lines

lea (x.data),%edi
std
movsb
movsb # or mov (%esi),%al; mov %al,(%edi), but movsb looks discrete enough to me

instead (and yes, I know that my example is silly).

no it's worse than that -- intel fast string stores can become globally visible in any order at all w.r.t. normal loads or stores... so take all those great examples in their recent whitepaper and throw out all the ordering guarantees for addresses on different cachelines if any of the stores are rep string.

for example transitive store ordering for locations on multiple cachelines is not guaranteed at all. the kernel could return a zero page and one core could see the zeroes out of order with another core performing some sort of lockless data structure operation.

fast strings don't break ordering from the point of view of the core performing the rep string operation, but externally there are no guarantees (it's right there in the docs).

-dean

#14 lore raw
In reply to Nick Piggin
Quoted 9 lines · On Sat, 8 Sep 2007 18:54:57 +1000 Nick Piggin <[email protected]> wrote:
Quoted 6 lines · Jesse Barnes

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

So, can we finally noop smp_rmb and smp_wmb on x86?

You can only no-op it on 64bit Intel processors. On 32bit it needs to be conditional on whether your processor family (or back compat for it) as the Pentium Pro has some serious store ordering errata (hence the way it needs lock decb for spin_unlock)

Alan

#15 lore raw
Quoted 11 lines · Alan Cox

On Sat, 8 Sep 2007 18:54:57 +1000

Quoted 8 lines · Nick Piggin
Quoted 6 lines · Jesse Barnes

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

Quoted 8 lines
Quoted 1 line

So, can we finally noop smp_rmb and smp_wmb on x86?

You can only no-op it on 64bit Intel processors. On 32bit it needs to be conditional on whether your processor family (or back compat for it) as the Pentium Pro has some serious store ordering errata (hence the way it needs lock decb for spin_unlock)

We already noop smp_wmb on i386 even when CONFIG_X86_PPRO_FENCE.

I'm not sure if either errata can be solved completely by adding lock ops in barrier instructions anyway: they both seem to involve situations where there is just a single problematic cacheline in question.

#16 lore raw
In reply to Jesse Barnes
Quoted 6 lines · On Fri, 7 Sep 2007 15:26:50 -0700 Jesse Barnes <[email protected]> wrote:

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

Nice - but it appears to be 64bit only - and indeed it appears to be untrue for real 32bit because of the Pentium Pro fencing errata.

The kernel also runs on IDT Winchip, Cyrix and AMD processors not all of which have exactly the same behaviour (the IDT Winchip as we run it profoundly differs)

Alan

#17 lore raw
Quoted 7 lines · Alan Cox

On Fri, 7 Sep 2007 15:26:50 -0700

Quoted 4 lines · Jesse Barnes

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Quoted 5 lines
Quoted 2 lines

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

Nice - but it appears to be 64bit only - and indeed it appears to be untrue for real 32bit because of the Pentium Pro fencing errata.

As I said, we're not doing anything special in barriers for the ppro errata today anyway.

Quoted 3 lines

The kernel also runs on IDT Winchip, Cyrix and AMD processors not all of which have exactly the same behaviour (the IDT Winchip as we run it profoundly differs)

AMD processors guarantee loads are ordered and stores are ordered (with exceptions of non-temporal, and non-wb policy).

As for the others that do out of order stores, are any of them SMP?

#18 lore raw
Quoted 4 lines

AMD processors guarantee loads are ordered and stores are ordered (with exceptions of non-temporal, and non-wb policy).

As for the others that do out of order stores, are any of them SMP?

IDT winchip isn't, Geode isn't

#19 lore raw
In reply to Jesse Barnes
Quoted 6 lines · * Jesse Barnes ([email protected]) wrote:

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

Hi Jesse,
Thanks for letting everyone know about that paper, however - it
has confused me somewhat; there seem to be differences in that description and that described in the 'Intel 64 and IA-32 Architectures Software Developer's Manual' and I'd like to understand whether this paper is designed just to explain points or is actually intended to change what can be expected of the processor.

That ordering doc states:
'Loads are not reordered with other loads'

Vol3a section 7.2.1 of the architecture manual states:

'Reads can be carried out speculatively and in any order.'

Is this a:
1) Change in the definition of the architecture that existing
processors actually follow anyway.
2) A difference between what the processor does and what is visible
to the software (the intro to this paper does seem to emphasize software visibility more than the architecture manual).
3) Some other difference I haven't spotted.

The other thing that made me think about it was that the Itanium Architecture Software Dev Manul vol2 2.1.2 states that the Itanium uses ld.acq/st.rel (acquire/release) references to 'operate according to the IA-32 ordering model.' which I think means that all those loads are in order relative to all the other acquire loads?

Dave

Signature · 4 lines
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
#20 lore raw
Quoted 23 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Jesse Barnes ([email protected]) wrote:

FYI, we just released a new white paper describing memory ordering for Intel processors:
http://developer.intel.com/products/processor/manuals/index.htm

Should help answer some questions about some of the ordering primitives we use on i386 and x86_64.

Hi Jesse,
Thanks for letting everyone know about that paper, however - it
has confused me somewhat; there seem to be differences in that description and that described in the 'Intel 64 and IA-32 Architectures Software Developer's Manual' and I'd like to understand whether this paper is designed just to explain points or is actually intended to change what can be expected of the processor.

That ordering doc states:
'Loads are not reordered with other loads'

Vol3a section 7.2.1 of the architecture manual states:

'Reads can be carried out speculatively and in any order.'

Quoted 7 lines

Is this a:
1) Change in the definition of the architecture that existing
processors actually follow anyway.
2) A difference between what the processor does and what is visible
to the software (the intro to this paper does seem to emphasize software visibility more than the architecture manual).
3) Some other difference I haven't spotted.

It's really both (1) and (2). This document will become part of the regular manuals when the next version is published. And yes, processors may do something different internally, but software can rely on the behavior described by the rules in the document.

Jesse

#21 lore raw
Quoted 5 lines · Jesse Barnes

It's really both (1) and (2). This document will become part of the regular manuals when the next version is published. And yes, processors may do something different internally, but software can rely on the behavior described by the rules in the document.

... until the first erratum comes around. With the multitude of x86 cores being introduced all the time (how many did only Intel just announce at IDF?@) that is going to happen sooner or later.

i386 with full legacy enabled already has to care about old PPros and those seriously violate write ordering.

-Andi