x86_64: fence nontemproal stores
RFC PATCH 1/3 x86_64: fence nontemproal stores
Hi,
Here's a couple of patches to improve the memory barrier situation on x86. They probably aren't going upstream until after the x86 merge, however I'm posting them here for RFC, and in case anybody wants to backport into stable trees.
---
movnt* instructions are not strongly ordered with respect to other stores, so if we are to assume stores are strongly ordered in the rest of the x86_64 kernel, we must fence these off (see similar examples in i386 kernel).
[ The AMD memory ordering document seems to say that nontemporal stores can
also pass earlier regular stores, so maybe we need sfences _before_ movnt* everywhere too? ]
arch/x86_64/lib/copy_user_nocache.S +1 -0
@@ -117,6 +117,7 @@ ENTRY(__copy_user_nocache) popq %rbx CFI_ADJUST_CFA_OFFSET -8 CFI_RESTORE rbx + sfence ret CFI_RESTORE_STATE
RFC PATCH 2/3 x86: fix IO write barriers
wmb() on x86 must always include a barrier, because stores can go out of order in many cases when dealing with devices (eg. WC memory).
include/asm-i386/system.h +6 -9
@@ -216,6 +216,7 @@ static inline unsigned long get_limit(un#define mb () alternative("lock; addl $0,0(%%esp)", "mfence", X86_FEATURE_XMM2)#define rmb () alternative("lock; addl $0,0(%%esp)", "lfence", X86_FEATURE_XMM2) +#define wmb () alternative("lock; addl $0,0(%%esp)", "sfence", X86_FEATURE_XMM)/** * read_barrier_depends - Flush all pending reads that subsequents reads
@@ -271,18 +272,14 @@ static inline unsigned long get_limit(un#define read_barrier_depends () do { } while(0) -#ifdef CONFIG_X86_OOSTORE -/* Actually there are no OOO store capable CPUs for now that do SSE, -but make it already an possibility. */ -#define wmb () alternative("lock; addl $0,0(%%esp)", "sfence", X86_FEATURE_XMM) -#else -#define wmb () __asm__ __volatile__ ("": : :"memory") -#endif -#ifdef CONFIG_SMP #define smp_mb () mb()#define smp_rmb () rmb() -#define smp_wmb () wmb() +#ifdef CONFIG_X86_OOSTORE +# define smp_wmb () wmb() +#else +# define smp_wmb () barrier() +#endif #define smp_read_barrier_depends () read_barrier_depends()#define set_mb (var, value) do { (void) xchg(&var, value); } while (0)#else
include/asm-x86_64/system.h +1 -5
@@ -159,12 +159,8 @@ static inline void write_cr8(unsigned lo*/ #define mb () asm volatile("mfence":::"memory")#define rmb () asm volatile("lfence":::"memory") - -#ifdef CONFIG_UNORDERED_IO #define wmb () asm volatile("sfence" ::: "memory") -#else -#define wmb () asm volatile("" ::: "memory") -#endif +#define read_barrier_depends () do {} while(0)#define set_mb (var, value) do { (void) xchg(&var, value); } while (0)
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 7 lines · Nick Piggin
include/asm-i386/system.h +0
-7-#ifdef CONFIG_X86_OOSTORE -/* Actually there are no OOO store capable CPUs for now that do SSE, -but make it already an possibility. */ -#define wmb () alternative("lock; addl $0,0(%%esp)", "sfence", X86_FEATURE_XMM) -#else -#define wmb () __asm__ __volatile__ ("": : :"memory") -#endif
include/asm-i386/system.h static inline unsigned long get_limit(un · line 282
Quoted 10 lines
include/asm-i386/system.h +5
-2-#ifdef CONFIG_SMP #define smp_mb () mb()#define smp_rmb () rmb() -#define smp_wmb () wmb() +#ifdef CONFIG_X86_OOSTORE +# define smp_wmb () wmb() +#else +# define smp_wmb () barrier() +#endif
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
So we can probably just drop that "just in case" clause above, and just do..
#define smp_wmb() barrier()
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Do you know if it made a big performance difference?
But yes we should probably just remove this special case to make maintenance easier.
-Andi
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 8 lines · Andi Kleen
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Do you know if it made a big performance difference?
On the winchip, it was a huge win. I can't remember exact numbers, but pretty much every benchmark I threw at it at the time showed significant improvement.
Quoted 2 lines
But yes we should probably just remove this special case to make maintenance easier.
It's CONFIG_SMP anyway, which none of the winchips were. SMP+OOSTORE just didn't happen, and I'd be surprised if any vendor makes it happen any time soon.
(Even if so, it's likely we'd need to make additional changes
anyway, so adding it back shouldn't be a big deal.)
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 8 lines · On Thursday 04 October 2007 20:10:44 Dave Jones wrote:
Quoted 7 lines · Andi Kleen
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Quoted 5 lines
Quoted 1 line
Do you know if it made a big performance difference?
On the winchip, it was a huge win. I can't remember exact numbers, but pretty much every benchmark I threw at it at the time showed significant improvement.
Significant as in >10%?
Quoted 4 lines
Quoted 2 lines
But yes we should probably just remove this special case to make maintenance easier.
It's
CONFIG_SMPanyway, which none of the winchips were.
It's not. And we need memory barriers even without SMP when talking to device drivers. Only the smp_*b()s get noped on UP.
-Andi
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 9 lines · Andi Kleen
Quoted 8 lines · On Thursday 04 October 2007 20:10:44 Dave Jones wrote:
Quoted 7 lines · Andi Kleen
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Do you know if it made a big performance difference?
On the winchip, it was a huge win. I can't remember exact numbers, but pretty much every benchmark I threw at it at the time showed significant improvement.
Significant as in >10%?
"Worth about 10-20% performance" according to the 2.4.18pre9-ac4 release notes: http://www.linuxtoday.com/news_story.php3?ltsn=2002-02-14-015-20-NW-KN
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
But yes we should probably just remove this special case to make maintenance easier.
It's
CONFIG_SMPanyway, which none of the winchips were.It's not.
You're right it isn't now, but Nicks patch seems to change it so that it is.
...
#ifdef CONFIG_SMP
#define smp_mb() mb() #define smp_rmb() rmb() -#define smp_wmb() wmb()
+#ifdef CONFIG_X86_OOSTORE
+# define smp_wmb() wmb()
+#else
+# define smp_wmb() barrier()
+#endif
Quoted 3 lines
And we need memory barriers even without SMP
when talking to device drivers. Only the smp_*b()s get noped on UP.
Good point.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 12 lines · On Thursday 04 October 2007 20:41:07 Dave Jones wrote:
Quoted 11 lines · Andi Kleen
Quoted 10 lines · On Thursday 04 October 2007 20:10:44 Dave Jones wrote:
Quoted 8 lines · Andi Kleen
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Do you know if it made a big performance difference?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
On the winchip, it was a huge win. I can't remember exact numbers, but pretty much every benchmark I threw at it at the time showed significant improvement.
Significant as in >10%?
"Worth about 10-20% performance" according to the 2.4.18pre9-ac4 release notes: http://www.linuxtoday.com/news_story.php3?ltsn=2002-02-14-015-20-NW-KN
Are there numbers for a newer kernel available too?
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
But yes we should probably just remove this special case to make maintenance easier.
It's
CONFIG_SMPanyway, which none of the winchips were.It's not.
You're right it isn't now, but Nicks patch seems to change it so that it is.
...
#ifdef
CONFIG_SMP
#definesmp_mb()mb()
#definesmp_rmb()rmb()
include/asm-i386/system.h static inline unsigned long get_limit(un · line 282
Quoted 6 lines
include/asm-i386/system.h +5
-1-#define smp_wmb () wmb() +#ifdef CONFIG_X86_OOSTORE +# define smp_wmb () wmb() +#else +# define smp_wmb () barrier() +#endif
That is only for smp_wmb() which are always SMP only
-Andi
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 17 lines · Andi Kleen
Quoted 16 lines · On Thursday 04 October 2007 20:41:07 Dave Jones wrote:
Quoted 15 lines · Andi Kleen
Quoted 13 lines · On Thursday 04 October 2007 20:10:44 Dave Jones wrote:
Quoted 8 lines · Andi Kleen
Quoted 6 lines
The only vendor that ever implemented OOSTOREs was Centaur, and they only did in the Winchip generation of the CPUs. When they dropped it from the C3, I asked whether they intended to bring it back, and the answer was "extremely unlikely".
Do you know if it made a big performance difference?
On the winchip, it was a huge win. I can't remember exact numbers, but pretty much every benchmark I threw at it at the time showed significant improvement.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Significant as in >10%?
"Worth about 10-20% performance" according to the 2.4.18pre9-ac4 release notes: http://www.linuxtoday.com/news_story.php3?ltsn=2002-02-14-015-20-NW-KN
Are there numbers for a newer kernel available too?
no idea, my winchips died about 5 years ago.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Re: [rfc][patch 2/3] x86: fix IO write barriers
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
"Worth about 10-20% performance" according to the 2.4.18pre9-ac4 release notes: http://www.linuxtoday.com/news_story.php3?ltsn=2002-02-14-015-20-NW-KN
Are there numbers for a newer kernel available too?
no idea, my winchips died about 5 years ago
Got a couple here just need a mainboard 8)
RFC PATCH 3/3 x86: optimise barriers
In reply to Nick PigginAccording to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence, smp_rmb() may be a simple barrier.
Also according to those documents, and according to existing practice in Linux (eg. spin_unlock doesn't enforce ordering), stores to cacheable memory are visible in program order too. Special string stores are safe -- their constituent stores may be out of order, but they must complete in order WRT surrounding stores. Nontemporal stores to WB memory can go out of order, and so they should be fenced explicitly to make them appear in-order WRT other stores. Hence, smp_wmb() may be a simple barrier.
http://developer.intel.com/products/processor/manuals/318147.pdf
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24593.pdf
In userspace microbenchmarks on a core2 system, fence instructions range anywhere from around 15 cycles to 50, which may not be totally insignificant in performance critical paths (code size will go down too).
However the primary motivation for this is to have the canonical barrier implementation for x86 architecture.
smp_rmb on buggy pentium pros remains a locked op, which is apparently required.
---
include/asm-i386/system.h +5 -1
@@ -274,7 +274,11 @@ static inline unsigned long get_limit(un#ifdef CONFIG_SMP #define smp_mb () mb() -#define smp_rmb () rmb() +#ifdef CONFIG_X86_PPRO_FENCE +# define smp_rmb () rmb() +#else +# define smp_rmb () barrier() +#endif #ifdef CONFIG_X86_OOSTORE # define smp_wmb () wmb()#else
include/asm-x86_64/system.h +2 -2
@@ -141,8 +141,8 @@ static inline void write_cr8(unsigned lo#ifdef CONFIG_SMP #define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() +#define smp_rmb () barrier() +#define smp_wmb () barrier()#define smp_read_barrier_depends () do {} while(0)#else #define smp_mb () barrier()
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 3 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence,
smp_rmb()may be a simple barrier.
...
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
First again, we could try to blame Intel etc. But then, wait a minute:
is it such a mystery knowledge? If this reordering is done there are some easy rules broken (just like in examples from these manuals). And if somebody cared to do this for optimization, then this is probably noticeable optimization, let's say 5 or 10%. Then any test shouldn't need to take very long to tell the truth in less than 100 loops!
So, maybe linux needs something like this, instead of waiting few years with each new model for vendors goodwill? IMHO, even for less popular processors, this could be checked under some debugging option at the system start (after disabling suspicios barrier for a while plus some WARN_ONs).
Thanks,
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 6 lines · Jarek Poplawski
Quoted 4 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence,
smp_rmb()may be a simple barrier.
Quoted 8 lines
...
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
You could have tried the optimization before, and
gotten better performance. But if without solid knowledge that the optimization is _valid_, you risk having a kernel
that performs great but suffer the occational glitch and therefore is unstable and crash the machine "now and then". This sort of thing can't really be figured out by experimentation, because the bad cases might happen only with some processors, some combinations of memory/chipsets, or with some minimum
number of processors. Such problems can be very hard
to find, especially considering that other plain bugs also cause crashes.
Therefore, the "ineffective code" was used because it was the only safe alternative. Now we know, so now we may optimize.
Helge Hafting
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 28 lines · Helge Hafting
Quoted 7 lines · Jarek Poplawski
Quoted 5 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory
for the x86 architecture. Hence,smp_rmb()may be a simple barrier.Quoted 8 lines
...
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
You could have tried the optimization before, and
gotten better performance. But if without solid knowledge that the optimization is _valid_, you risk having a kernel
that performs great but suffer the occational glitch and therefore is unstable and crash the machine "now and then". This sort of thing can't really be figured out by experimentation, because the bad cases might happen only with some processors, some combinations of memory/chipsets, or with some minimum
number of processors. Such problems can be very hard
to find, especially considering that other plain bugs also cause crashes.
Quoted 2 lines
Therefore, the "ineffective code" was used because it was the only safe alternative. Now we know, so now we may optimize.
Sorry, I don't understand this logic at all. Since bad cases happen independently from any specifications and Intel doesn't take any legal responsibility for such information, it seems we should better still not optimize?
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 29 lines · Jarek Poplawski
Quoted 28 lines · Helge Hafting
Quoted 7 lines · Jarek Poplawski
Quoted 5 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory
for the x86 architecture. Hence,smp_rmb()may be a simple barrier.Quoted 8 lines
...
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
You could have tried the optimization before, and
gotten better performance. But if without solid knowledge that the optimization is _valid_, you risk having a kernel
that performs great but suffer the occational glitch and therefore is unstable and crash the machine "now and then". This sort of thing can't really be figured out by experimentation, because the bad cases might happen only with some processors, some combinations of memory/chipsets, or with some minimum
number of processors. Such problems can be very hard
to find, especially considering that other plain bugs also cause crashes.
Quoted 7 lines
Quoted 2 lines
Therefore, the "ineffective code" was used because it was the only safe alternative. Now we know, so now we may optimize.
Sorry, I don't understand this logic at all. Since bad cases happen independently from any specifications and Intel doesn't take any legal responsibility for such information, it seems we should better still not optimize?
We already do in probably more critical and lible to be problematic cases (notably, spin_unlock).
So unless there is reasonable information for us to believe this will be a problem, IMO the best thing to do is stick with the specs. Intel is pretty reasonable with documenting errata I think.
With memory barriers specifically, I'm sure we have many more bugs in the kernel than AMD or Intel have in their chips ;)
Re: [rfc][patch 3/3] x86: optimise barriers
On Fri, Oct 12, 2007 at 11:44:27AM +0200, Nick Piggin wrote:
...
Quoted 3 lines
So unless there is reasonable information for us to believe this will be a problem, IMO the best thing to do is stick with the specs. Intel is pretty reasonable with documenting errata I think.
100% right - if there are any specs. But it seems for a few years this spec was missing or there is some change of mind, I presume?
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
In reply to Jarek PoplawskiQuoted 33 lines · Jarek Poplawski
Quoted 31 lines · Helge Hafting
Quoted 8 lines · Jarek Poplawski
Quoted 5 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence,
smp_rmb()may be a simple barrier.Quoted 9 lines
...
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
You could have tried the optimization before, and
gotten better performance. But if without solid knowledge that the optimization is _valid_, you risk having a kernel
that performs great but suffer the occational glitch and therefore is unstable and crash the machine "now and then". This sort of thing can't really be figured out by experimentation, because the bad cases might happen only with some processors, some combinations of memory/chipsets, or with some minimum
number of processors. Such problems can be very hard
to find, especially considering that other plain bugs also cause crashes.
Quoted 9 lines
Quoted 3 lines
Therefore, the "ineffective code" was used because it was the only safe alternative. Now we know, so now we may optimize.
Sorry, I don't understand this logic at all. Since bad cases happen independently from any specifications and Intel doesn't take any legal responsibility for such information, it seems we should better still not optimize?
The point is that we _trust_ intel when they says "this will work". Therefore, we can use the optimizations. It was never about legal matters. If we didn't trust intel, then we couldn't use their processors at all.
We couldn't take the chance before. It was not documented to work, verification by testing would not be trivial at all for this case.
Linux is about "stability first, then performance". Now we _know_ that we can have this optimization without compromising stability. Nobody knew before!
Helge Hafting
Re: [rfc][patch 3/3] x86: optimise barriers
On Fri, Oct 12, 2007 at 02:44:51PM +0200, Helge Hafting wrote:
...
Quoted 4 lines
The point is that we _trust_ intel when they says "this will work". Therefore, we can use the optimizations. It was never about legal matters. If we didn't trust intel, then we couldn't use their processors at all.
But there was nothing about trust. Usually you don't trust somebody but somebody's opinions. The problem is there was no valid opinion, or this opinion has been changed now (no reason to not trust yet...).
Quoted 6 lines
We couldn't take the chance before. It was not documented to work, verification by testing would not be trivial at all for this case.
Linux is about "stability first, then performance". Now we _know_ that we can have this optimization without compromising stability. Nobody knew before!
So, you think this would be the first or the least credibly verified undocumented feature used in linux? Then, it seems I can try to install this linux on my laptop at last! (... And, I can trust you, it will not break anything...?)
Thanks,
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 9 lines · Jarek Poplawski
On Fri, Oct 12, 2007 at 02:44:51PM +0200, Helge Hafting wrote:
...Quoted 5 lines
The point is that we _trust_ intel when they says "this will work". Therefore, we can use the optimizations. It was never about legal matters. If we didn't trust intel, then we couldn't use their processors at all.
Quoted 4 lines
But there was nothing about trust. Usually you don't trust somebody but somebody's opinions. The problem is there was no valid opinion, or this opinion has been changed now (no reason to not trust yet...).
"Trusting people or their opinions" is only about use of the english language, and not that intersting to bring up here. Surely you know that lots of people here have english as a secondary language only. Intersting for me to know, but probably not for everybody else.
Quoted 8 lines
Quoted 7 lines
We couldn't take the chance before. It was not documented to work, verification by testing would not be trivial at all for this case.
Linux is about "stability first, then performance". Now we _know_ that we can have this optimization without compromising stability. Nobody knew before!
Quoted 5 lines
So, you think this would be the first or the least credibly verified undocumented feature used in linux? Then, it seems I can try to install this linux on my laptop at last! (... And, I can trust you, it will not break anything...?)
I never claimed that linux will work on your laptop, so no:
You can't take my word for that, because I never gave it! It is well known that some laptops don't work with linux, I have no idea if yours will work, I don't even know what kind it is.
I told you the reasoning behind using _this particular optimization_, the same does _not_ apply to everything else. If you think every kernel decision is made the same way, then you are mistaken. Things don't work that way.
First, several people are involved - they think differently. Second, "what kind of tricks to use" is not an all-or-nothing approach. If linux were to use every undocumented trick that might or might not work, then linux would fail on lots of hardware. It would not be useful.
If linux took the other approach and never used any "tricks", then it'd be slow and boring.
Some things are much easier to test - you construct a testcase or just build a test kernel and benchmark it. If all is ok, then the "trick" is useable. Some cases are a clear win for lots of machines, and the possible failure cases involves
very rare hardware. So it might get used. Some tricks have a failure mode that is rare but completely obvious when it happens. So it gets used, and "troublesome hardware" is added to a blacklist as needed.
Some "tricks" however, are hard to figure out without docs. There may be no good way to test. The tricks
may cause instability that will be very hard to track down, and this could happen on a wide range of hardware. So such don't get used, until adequate documentation appear. In this case, it seems like intel, who make and design the processors in question and therefore know them well enough, provided such documentation. That
makes a previously dubious optimization safe.
Helge Hafting
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 14 lines · Helge Hafting
Quoted 9 lines · Jarek Poplawski
On Fri, Oct 12, 2007 at 02:44:51PM +0200, Helge Hafting wrote:
...Quoted 5 lines
The point is that we _trust_ intel when they says "this will work". Therefore, we can use the optimizations. It was never about legal matters. If we didn't trust intel, then we couldn't use their processors at all.
Quoted 4 lines
But there was nothing about trust. Usually you don't trust somebody but somebody's opinions. The problem is there was no valid opinion, or this opinion has been changed now (no reason to not trust yet...).
Quoted 5 lines
"Trusting people or their opinions" is only about use of the english language, and not that intersting to bring up here. Surely you know that lots of people here have english as a secondary language only. Intersting for me to know, but probably not for everybody else.
Of curse, I know this problem: sometimes it's very hard to make people believe it's my secondary language! But this time I didn't see any language problem. I simply poined out that sometimes trusting could be not enough - not necessarily in this case.
Quoted 13 lines
Quoted 8 lines
Quoted 7 lines
We couldn't take the chance before. It was not documented to work, verification by testing would not be trivial at all for this case.
Linux is about "stability first, then performance". Now we _know_ that we can have this optimization without compromising stability. Nobody knew before!Quoted 5 lines
So, you think this would be the first or the least credibly verified undocumented feature used in linux? Then, it seems I can try to install this linux on my laptop at last! (... And, I can trust you, it will not break anything...?)
Quoted 4 lines
I never claimed that linux will work on your laptop, so no:
You can't take my word for that, because I never gave it! It is well known that some laptops don't work with linux, I have no idea if yours will work, I don't even know what kind it is.
OK, this was supposed to be a joke... (Btw, can you remember burning linux laptops?) I thought this "stability first" a bit funny, but this was a really bad joke, sorry.
Thanks for these additional explanations - you are completely right!
Regards,
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
In reply to Jarek PoplawskiQuoted 11 lines · Jarek Poplawski
Quoted 3 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence,
smp_rmb()may be a simple barrier....
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
I'm not sure exactly what the situation is with the manufacturers, but maybe they (at least Intel) wanted to keep their options open WRT their barrier semantics, even if current implementations were not taking full liberty of them.
Quoted 6 lines
First again, we could try to blame Intel etc. But then, wait a minute:
is it such a mystery knowledge? If this reordering is done there are some easy rules broken (just like in examples from these manuals). And if somebody cared to do this for optimization, then this is probably noticeable optimization, let's say 5 or 10%. Then any test shouldn't need to take very long to tell the truth in less than 100 loops!
I don't know quite what you're saying... the CPUs could probably get performance by having weakly ordered loads, OTOH I think the Intel ones might already do this speculatively so they appear in order but essentially have the performance of weak order.
If you're just talking about this patch, then it probably isn't much performance gain. I'm guessing you'd be lucky to measure it from userspace.
Quoted 5 lines
So, maybe linux needs something like this, instead of waiting few years with each new model for vendors goodwill? IMHO, even for less popular processors, this could be checked under some debugging option at the system start (after disabling suspicios barrier for a while plus some
WARN_ONs).
I don't know if that would be worthwhile. It actually isn't always trivial to trigger reordering. For example, on my dual-core core2, in order to see reads pass writes, I have to do work on a set that exceeds the cache size and does a huge amount of work to ensure it is going to trigger that. If you can actually come up with a test case that triggers load/load or store/store reordering, I'm sure Intel / AMD would like to see it ;)
All existing processors as far as we know are in-order WRT loads vs loads and stores vs stores. It was just a matter of getting the docs clarified, which gives us more confidence that we're correct and a reasonable guarnatee of forward compatibility.
So, I think the plan is just to merge these 3 patches during the current window.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 26 lines · Nick Piggin
Quoted 11 lines · Jarek Poplawski
Quoted 3 lines · Nick Piggin
According to latest memory ordering specification documents from Intel and AMD, both manufacturers are committed to in-order loads from cacheable memory for the x86 architecture. Hence,
smp_rmb()may be a simple barrier....
Great news!
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
I'm not sure exactly what the situation is with the manufacturers, but maybe they (at least Intel) wanted to keep their options open WRT their barrier semantics, even if current implementations were not taking full liberty of them.
Quoted 6 lines
First again, we could try to blame Intel etc. But then, wait a minute:
is it such a mystery knowledge? If this reordering is done there are some easy rules broken (just like in examples from these manuals). And if somebody cared to do this for optimization, then this is probably noticeable optimization, let's say 5 or 10%. Then any test shouldn't need to take very long to tell the truth in less than 100 loops!
Quoted 4 lines
I don't know quite what you're saying... the CPUs could probably get performance by having weakly ordered loads, OTOH I think the Intel ones might already do this speculatively so they appear in order but essentially have the performance of weak order.
I meant: if there is any reordering possible this should be quite distinctly visible, because why would any vendor enable such nasty things if not for performance. But now I start to doubt: of course there is such a possibility someone makes this reordering for some other reasons which could be so rare it's hard to check. And this someone knows it's processors are seen less efficient because of eg. mostly unneeded read barriers used by operating systems...
Quoted 4 lines
If you're just talking about this patch, then it probably isn't much performance gain. I'm guessing you'd be lucky to measure it from userspace.
No, it's only about the comment to this patch: "Hence, smp_rmb() may be a simple barrier".
Quoted 8 lines
Quoted 7 lines
So, maybe linux needs something like this, instead of waiting few years with each new model for vendors goodwill? IMHO, even for less popular processors, this could be checked under some debugging option at the system start (after disabling suspicios barrier for a while plus some
WARN_ONs).
Quoted 7 lines
I don't know if that would be worthwhile. It actually isn't always trivial to trigger reordering. For example, on my dual-core core2, in order to see reads pass writes, I have to do work on a set that exceeds the cache size and does a huge amount of work to ensure it is going to trigger that. If you can actually come up with a test case that triggers load/load or store/store reordering, I'm sure Intel / AMD would like to see it ;)
Anyway, it seems any heavy testing such as yours, should give us the same informations years earlier than any vendors manual and then any gain is multiplied by millions of users. Then only still doubtful cases could be treated with additional caution and some debugging code.
Quoted 5 lines
All existing processors as far as we know are in-order WRT loads vs loads and stores vs stores. It was just a matter of getting the docs clarified, which gives us more confidence that we're correct and a reasonable guarnatee of forward compatibility.
After reading this Intel's legal information I don't think you should feel so much more forward confident...
Quoted 4 lines
So, I think the plan is just to merge these 3 patches during the current window.
And they really should be!
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 9 lines · Jarek Poplawski
Quoted 5 lines · Nick Piggin
I don't know quite what you're saying... the CPUs could probably get performance by having weakly ordered loads, OTOH I think the Intel ones might already do this speculatively so they appear in order but essentially have the performance of weak order.
I meant: if there is any reordering possible this should be quite distinctly visible.
It's not. Not in the cases where it is explicitly allowed and actively exploited (loads passing stores), but most definitely not distinctly visible in errata cases that have slipped through all the V&V.
Quoted 4 lines
because why would any vendor enable such nasty
things if not for performance. But now I start to doubt: of course there is such a possibility someone makes this reordering for some other reasons which could be so rare it's hard to check.
Yes: it isn't the explicitly allowed reorderings that we care about here (because obviously we're retaining the barriers for those). It would be cases of bugs in the CPUs meaning they don't follow the standard. But how far do you take your mistrust of a CPU? You could ask gcc to insert locked ops between every load and store operation? Or keep it switched off to ensure no bugs ;)
Quoted 5 lines
Anyway, it seems any heavy testing such as yours, should give us the same informations years earlier than any vendors manual and then any gain is multiplied by millions of users. Then only still doubtful cases could be treated with additional caution and some debugging code.
Firstly, while it can be possible to write a code to show up reordering, it is really hard (ie. impossible) to guarantee no reordering happens. For example, it may have only showed up on SMT+SMP P4 CPUs with some obscure interactions between threads and cores involving more than 2 threads.
Secondly, even if we were sure that no current implementations reordered loads, we don't want to go outside the bounds of the specification because we might break on some future CPUs. This isn't a big performance win.
Quoted 7 lines
Quoted 4 lines
All existing processors as far as we know are in-order WRT loads vs loads and stores vs stores. It was just a matter of getting the docs clarified, which gives us more confidence that we're correct and a reasonable guarnatee of forward compatibility.
After reading this Intel's legal information I don't think you should feel so much more forward confident...
Yes, but that's the same way I feel after reading *any* legal "information" ;)
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 21 lines · Nick Piggin
Quoted 9 lines · Jarek Poplawski
Quoted 5 lines · Nick Piggin
I don't know quite what you're saying... the CPUs could probably get performance by having weakly ordered loads, OTOH I think the Intel ones might already do this speculatively so they appear in order but essentially have the performance of weak order.
I meant: if there is any reordering possible this should be quite distinctly visible.
It's not. Not in the cases where it is explicitly allowed and actively exploited (loads passing stores), but most definitely not distinctly visible in errata cases that have slipped through all the V&V.
Quoted 4 lines
because why would any vendor enable such nasty
things if not for performance. But now I start to doubt: of course there is such a possibility someone makes this reordering for some other reasons which could be so rare it's hard to check.
Quoted 6 lines
Yes: it isn't the explicitly allowed reorderings that we care about here (because obviously we're retaining the barriers for those). It would be cases of bugs in the CPUs meaning they don't follow the standard. But how far do you take your mistrust of a CPU? You could ask gcc to insert locked ops between every load and store operation? Or keep it switched off to ensure no bugs ;)
I'm not sure of your point, but it seems we don't differ here, and after all there is quirks code for such things.
Quoted 8 lines
Quoted 7 lines
Anyway, it seems any heavy testing such as yours, should give us the same informations years earlier than any vendors manual and then any gain is multiplied by millions of users. Then only still doubtful cases could be treated with additional caution and some debugging code.
Quoted 4 lines
Firstly, while it can be possible to write a code to show up reordering, it is really hard (ie. impossible) to guarantee no reordering happens. For example, it may have only showed up on SMT+SMP P4 CPUs with some obscure interactions between threads and cores involving more than 2 threads.
I'm not sure how much this all above is consistent wrt. this earlier opinion:
Quoted 3 lines
[...] If you can actually come up with a test
case that triggers load/load or store/store reordering, I'm sure Intel / AMD would like to see it ;)
It seems, after testing only (plus no official spec against this idea), you could be almost sure there is no such test possible. And, if it were done a few years ago, you think it still should be not enough to make a decision on changing this smp_rmb because of lack of official specs? Besides, there is probably so much features guessing in arch and drivers sections, this reorder testing should look as solid as a math proof wrt. them.
Quoted 5 lines
Secondly, even if we were sure that no current implementations reordered loads, we don't want to go outside the bounds of the specification because we might break on some future CPUs. This isn't a big performance win.
I don't agree with this - IMO we should care only about currently used CPUs, and test each time the new ones.
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
All existing processors as far as we know are in-order WRT loads vs loads and stores vs stores. It was just a matter of getting the docs clarified, which gives us more confidence that we're correct and a reasonable guarnatee of forward compatibility.
After reading this Intel's legal information I don't think you should feel so much more forward confident...
Yes, but that's the same way I feel after reading *any* legal "information" ;)
Strange... I feel exactly opposite. Are you sure you've chosen the right job (...and the right system)?
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 1 line · Jarek Poplawski
On Fri, Oct 12, 2007 at 12:42:38PM +0200, Nick Piggin wrote:
...
Quoted 5 lines
Quoted 3 lines
[...] If you can actually come up with a test
case that triggers load/load or store/store reordering, I'm sure Intel / AMD would like to see it ;)It seems, after testing only (plus no official spec against this idea),
(...plus of course proper smp_rmb & smp_wmb vs. smp_mb interpretation probably available from Paul McKenney or Davide Libenzi before this Intel spec, as well...)
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
In reply to Jarek PoplawskiQuoted 4 lines · Jarek Poplawski
First it looks like a really great thing that it's revealed at last. But then... there is probably some confusion: did we have to use ineffective code for so long?
I think the chip manufacturers really wanted to keep their options open.
Having the option to re-order loads in architecturally visible ways was something that they probably felt they really wanted to have. On the other hand:
- I bet they had noticed that things break, and some applications depend on fairly strong ordering (not necessarily in Linux-land, but..)
I suspect hw manufacturers go through life hoping that "software improves". They probably thought that getting rid of the old 16-bit windows would mean that less people depended on undefined behaviour.
And I suspect that they started noticing that no, with threads and JVM's and things, *more* people started depending on fairly strong memory ordering.
- I suspect Intel in particular noticed that they can do a lot of very aggressive re-ordering at a microarchitectural level, but can still guarantee that *architecturally* they never show it (dynamic detection of reordered loads being replayed on cache dirty events etc).
IOW, I suspect that both Intel and AMD noticed that while they had wanted to keep their options open, those options weren't really realistic, and not something that the market wanted (aggressive use of threading wants *stricter* memory ordering, not looser), and they could work well enough with a fairly strict memory model.
Quoted 5 lines
So, maybe linux needs something like this, instead of waiting few years with each new model for vendors goodwill? IMHO, even for less popular processors, this could be checked under some debugging option at the system start (after disabling suspicios barrier for a while plus some
WARN_ONs).
Quite frankly, even *within* Intel and AMD, there are damn few people who understand exactly what the memory ordering requirements and guarantees are and historically were for the different CPU's.
I would bet that had you asked a random (but still competent) Intel/AMD engineer that wasn't really intimately involved with the actual design of the cache protocols and memory pipelines, they would absolutely not have been able to tell you how the CPU actually worked.
So no, there's no way a software person could have afforded to say "it seems to work on my setup even without the barrier". On a dual-socket setup with s shared bus, that says absolutely *nothing* about the behaviour of the exact same CPU when used with a multi-bus chipset. Not to mention another revisions of the same CPU - much less a whole other microarchitecture.
So yes, I've personally been aware for about a year that the memory ordering was going to likely be documented, but no way was I going to depend on it until Intel and AMD were ready to state so *publicly*. Because before that happens, they may have noticed errata etc that made it not work out.
Also, please note that we didn't even just change the barriers immediately when the docs came out. I want to do it soon - still *early* in the 2.6.24 development cycle - exactly because bugs happen, and if somebody notices something strange, we'll have more time to perhaps decide that "oops, there's something bad going on, let's undo this for the real 2.6.24 release until we can figure out the exact pattern".
Linus
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 3 lines · Linus Torvalds
On Fri, 12 Oct 2007, Jarek Poplawski wrote:
...
Quoted 6 lines
So no, there's no way a software person could have afforded to say "it seems to work on my setup even without the barrier". On a dual-socket setup with s shared bus, that says absolutely *nothing* about the behaviour of the exact same CPU when used with a multi-bus chipset. Not to mention another revisions of the same CPU - much less a whole other microarchitecture.
Yes, I still can't believe this, but after some more reading I start to admit such things can happen in computer "science" too... I've mentioned a lost performance, but as a matter of fact I've been more concerned with the problem of truth:
From: Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3A:
"7.2.2 Memory Ordering in P6 and More Recent Processor Families
...
1. Reads can be carried out speculatively and in any order.
..."So, it looks to me like almost the 1-st Commandment. Some people (like me) did believe this, others tried to check, and it was respected for years notwithstanding nobody had ever seen such an event.
And then, a few years later, we have this:
From: Intel(R) 64 Architecture Memory Ordering White Paper
"2 Memory ordering for write-back (WB) memory
...
Intel 64 memory ordering obeys the following principles:
1. Loads are not reordered with other loads. ..."
I know, technically this doesn't have to be a contradiction (for not WB), but to me it's something like: "OK, Elvis lives and this guy is not real Paul McCartney too" in an official CIA statement!
...
Quoted 6 lines
Also, please note that we didn't even just change the barriers immediately when the docs came out. I want to do it soon - still *early* in the 2.6.24 development cycle - exactly because bugs happen, and if somebody notices something strange, we'll have more time to perhaps decide that "oops, there's something bad going on, let's undo this for the real 2.6.24 release until we can figure out the exact pattern".
I'm still so "dazed and confused" that I can't tell this (or anything) is right...
Thanks very much for so extensive and sound explanation,
Jarek P.
PS: Btw, I apologize Helge for not trusting her: "verification by testing would not be trivial" words.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 20 lines · Jarek Poplawski
Quoted 3 lines · Linus Torvalds
On Fri, 12 Oct 2007, Jarek Poplawski wrote:
...
Quoted 6 lines
So no, there's no way a software person could have afforded to say "it seems to work on my setup even without the barrier". On a dual-socket setup with s shared bus, that says absolutely *nothing* about the behaviour of the exact same CPU when used with a multi-bus chipset. Not to mention another revisions of the same CPU - much less a whole other microarchitecture.
Yes, I still can't believe this, but after some more reading I start to admit such things can happen in computer "science" too... I've mentioned a lost performance, but as a matter of fact I've been more concerned with the problem of truth:
From: Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3A:
Quoted 8 lines
"7.2.2 Memory Ordering in P6 and More Recent Processor Families
... 1. Reads can be carried out speculatively and in any order. ..."So, it looks to me like almost the 1-st Commandment. Some people (like me) did believe this, others tried to check, and it was respected for years notwithstanding nobody had ever seen such an event.
I'd say that's exactly what Intel wanted. It's pretty common (we do it all the time in the kernel too) to create an API which places a stronger requirement on the caller than is actually required. It can make changes much less painful.
Has performance really been much problem for you? (even before the lfence instruction, when you theoretically had to use a locked op)? I mean, I'd struggle to find a place in the Linux kernel where there is actually a measurable difference anywhere... and we're pretty performance critical and I think we have a reasonable amount of lockless code (I guess we may not have a lot of tight computational loops, though). I'd be interested to know what, if any, application had found these barriers to be problematic...
Quoted 10 lines
And then, a few years later, we have this:
From: Intel(R) 64 Architecture Memory Ordering White Paper
"2 Memory ordering for write-back (WB) memory
...
Intel 64 memory ordering obeys the following principles:
1. Loads are not reordered with other loads. ..."
Quoted 3 lines
I know, technically this doesn't have to be a contradiction (for not WB), but to me it's something like: "OK, Elvis lives and this guy is not real Paul McCartney too" in an official CIA statement!
The thing is that those documents are not defining what a particular implementation does, but how the architecture is defined (ie. what must some arbitrary software/hardware provide and what may it expect).
It's pretty natural that Intel started out with a weaker guarantee than their CPUs of the time actually supported, and tightened it up after (presumably) deciding not to implement such relaxed semantics for the forseeable future.
Re: [rfc][patch 3/3] x86: optimise barriers
On Mon, Oct 15, 2007 at 10:09:24AM +0200, Nick Piggin wrote:
...
Quoted 8 lines
Has performance really been much problem for you? (even before the lfence instruction, when you theoretically had to use a locked op)? I mean, I'd struggle to find a place in the Linux kernel where there is actually a measurable difference anywhere... and we're pretty performance critical and I think we have a reasonable amount of lockless code (I guess we may not have a lot of tight computational loops, though). I'd be interested to know what, if any, application had found these barriers to be problematic...
I'm not performance-words at all, so I can't help you, sorry. But, I understand people who care about this, and think there is a popular conviction barriers and locked instructions are costly, so I'm surprised there is any "problem" now with finding these gains...
Quoted 3 lines
The thing is that those documents are not defining what a particular implementation does, but how the architecture is defined (ie. what must some arbitrary software/hardware provide and what may it expect).
I'm not sure this is the right way to tell it. If there is no distinction between what is and what could be, how can I believe in similar Alpha or Itanium stuff? IMHO, these manuals sometimes look like they describe some real hardware mechanisms, and sometimes they mention about possible changes and reserved features too. So, when they don't mention you could think it's a present behavior.
Quoted 4 lines
It's pretty natural that Intel started out with a weaker guarantee than their CPUs of the time actually supported, and tightened it up after (presumably) deciding not to implement such relaxed semantics for the forseeable future.
As a matter of fact it's not natural for me at all. I expected the other direction, and I still doubt programmers' intentions could be "automatically" predicted good enough, so IMHO, it's not for long. Of course, it doesn't seem to be any help for linux or bsd programmers, which still have to think about different architectures.
Regards,
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
On Mon, Oct 15, 2007 at 11:09:59AM +0200, Jarek Poplawski wrote:
...
Quoted 1 line
I'm not performance-words at all, so I can't help you, sorry. But, I
...performance-wards?!
Looks like serious: I don't even now who I'm not now!
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
In reply to Jarek PoplawskiQuoted 11 lines · Jarek Poplawski
On Mon, Oct 15, 2007 at 10:09:24AM +0200, Nick Piggin wrote:
...Quoted 8 lines
Has performance really been much problem for you? (even before the lfence instruction, when you theoretically had to use a locked op)? I mean, I'd struggle to find a place in the Linux kernel where there is actually a measurable difference anywhere... and we're pretty performance critical and I think we have a reasonable amount of lockless code (I guess we may not have a lot of tight computational loops, though). I'd be interested to know what, if any, application had found these barriers to be problematic...
Quoted 4 lines
I'm not performance-words at all, so I can't help you, sorry. But, I understand people who care about this, and think there is a popular conviction barriers and locked instructions are costly, so I'm surprised there is any "problem" now with finding these gains...
It's more expensive than nothing, sure. However in real code, algorithmic complexity, cache misses and cacheline bouncing tend to be much bigger issues.
I can't think of a place in the kernel where smp_rmb matters _that_ much. seqlocks maybe (timers, dcache lookup), vmscan... Obviously removing the lfence is not going to hurt. Maybe we even gain 0.01% performance in someone's workload.
Also, remember: if loads are already in-order, then lfence is a noop, right? (in practice it seems to have to do a _little_ bit of work, but it's like a dozen cycles).
Quoted 10 lines
Quoted 3 lines
The thing is that those documents are not defining what a particular implementation does, but how the architecture is defined (ie. what must some arbitrary software/hardware provide and what may it expect).
I'm not sure this is the right way to tell it. If there is no distinction between what is and what could be, how can I believe in similar Alpha or Itanium stuff? IMHO, these manuals sometimes look like they describe some real hardware mechanisms, and sometimes they mention about possible changes and reserved features too. So, when they don't mention you could think it's a present behavior.
No. Why are you reading that much into it? I know for a fact that some non-x86 architectures actual implementations have stronger ordering than their ISA allows. It's nothing to do with you "believing" how the hardware works. That's not what the document is describing (directly).
Quoted 8 lines
Quoted 4 lines
It's pretty natural that Intel started out with a weaker guarantee than their CPUs of the time actually supported, and tightened it up after (presumably) deciding not to implement such relaxed semantics for the forseeable future.
As a matter of fact it's not natural for me at all. I expected the other direction, and I still doubt programmers' intentions could be "automatically" predicted good enough, so IMHO, it's not for long.
Really? Consider the consequences if, instead of releasing this latest document tightening consistency, Intel found that out of order loads were worth 5% more performance and implemented them in their next chip. The chip could be completely backwards compatible, but all your old code would break, because it was broken to begin with (because it was outside the spec).
IMO Intel did exactly the right thing from an engineering perspective, and so did Linux to always follow the spec.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 1 line · Nick Piggin
On Mon, Oct 15, 2007 at 11:10:00AM +0200, Jarek Poplawski wrote:
...
Quoted 9 lines
Quoted 4 lines
I'm not performance-words at all, so I can't help you, sorry. But, I understand people who care about this, and think there is a popular conviction barriers and locked instructions are costly, so I'm surprised there is any "problem" now with finding these gains...
It's more expensive than nothing, sure. However in real code, algorithmic complexity, cache misses and cacheline bouncing tend to be much bigger issues.
Quoted 8 lines
I can't think of a place in the kernel where
smp_rmbmatters _that_ much. seqlocks maybe (timers, dcache lookup), vmscan... Obviously removing the lfence is not going to hurt. Maybe we even gain 0.01% performance in someone's workload.Also, remember: if loads are already in-order, then lfence is a noop, right? (in practice it seems to have to do a _little_ bit of work, but it's like a dozen cycles).
You are right: considering current CPUs there could be no performance problem at all. Removing LOCKs for older ones should probably matter more, but as a matter of fact, now I wouldn't bet even on this - it could be mostly noop as well.
Quoted 10 lines
Quoted 3 lines
As a matter of fact it's not natural for me at all. I expected the other direction, and I still doubt programmers' intentions could be "automatically" predicted good enough, so IMHO, it's not for long.
Really? Consider the consequences if, instead of releasing this latest document tightening consistency, Intel found that out of order loads were worth 5% more performance and implemented them in their next chip. The chip could be completely backwards compatible, but all your old code would break, because it was broken to begin with (because it was outside the spec).
I've different opinion on this: I expect any spec to describe current implementation. Before issuing new models any changes of implementation should be made public with proper margin of time. Then system could be optimally adjusted to a real hardware, instead of planned only, but possibly never realized (plus doing such not used things with old means is usually more costly: lock vs. lfence). There is still problem of specs' completness: there are probably often some things unspecified which could brake on a new model, so never 100% guarantee anyway.
Quoted 2 lines
IMO Intel did exactly the right thing from an engineering perspective, and so did Linux to always follow the spec.
But, if you follow the spec - you don't follow the spec! Why do you ignore so much this part of Intel's spec:
"This document contains information which Intel may change at any
time without notice. Do not finalize a design with this information."
Maybe it's a real Intel intention and not for lawyers only? (Btw, it seems we have an example.)
Regards,
Jarek P.
Re: [rfc][patch 3/3] x86: optimise barriers
Quoted 15 lines · Jarek Poplawski
Quoted 1 line · Nick Piggin
On Mon, Oct 15, 2007 at 11:10:00AM +0200, Jarek Poplawski wrote:
...
Quoted 10 lines
Quoted 3 lines
As a matter of fact it's not natural for me at all. I expected the other direction, and I still doubt programmers' intentions could be "automatically" predicted good enough, so IMHO, it's not for long.
Really? Consider the consequences if, instead of releasing this latest document tightening consistency, Intel found that out of order loads were worth 5% more performance and implemented them in their next chip. The chip could be completely backwards compatible, but all your old code would break, because it was broken to begin with (because it was outside the spec).
Quoted 9 lines
I've different opinion on this: I expect any spec to describe current implementation. Before issuing new models any changes of implementation should be made public with proper margin of time. Then system could be optimally adjusted to a real hardware, instead of planned only, but possibly never realized (plus doing such not used things with old means is usually more costly: lock vs. lfence). There is still problem of specs' completness: there are probably often some things unspecified which could brake on a new model, so never 100% guarantee anyway.
what you don't realize is that Intel (and AMD) have built their business on makeing sure that their new CPU's run existing software with no modifications, (and almost always faster then the old versions). remember that for most of the world, getting the software modified would mean buying a new version, if the vendor bothered to make a different version for the new chip.
if they required everyone to buy new software to use a new chip it wouldn't work well. In fact Intel tried to do exactly withat with the itanium and it has been a spectacular failure (or t the very least, not a spectacular sucess)
Quoted 11 lines
Quoted 2 lines
IMO Intel did exactly the right thing from an engineering perspective, and so did Linux to always follow the spec.
But, if you follow the spec - you don't follow the spec! Why do you ignore so much this part of Intel's spec:
"This document contains information which Intel may change at any
time without notice. Do not finalize a design with this information."Maybe it's a real Intel intention and not for lawyers only? (Btw, it seems we have an example.)
in theory they could change anything at any time, in practice if they break old software they won't sell the chips, so the modifications tend to be along the lines of this one, adding detail to the specifications so that programmers can get more performance.
David Lang
Re: [rfc][patch 3/3] x86: optimise barriers
On Tue, Oct 16, 2007 at 02:14:17AM -0700, [email protected] wrote:
...
Quoted 6 lines
what you don't realize is that Intel (and AMD) have built their business on makeing sure that their new CPU's run existing software with no modifications, (and almost always faster then the old versions). remember that for most of the world, getting the software modified would mean buying a new version, if the vendor bothered to make a different version for the new chip.
It's a good point to always consider when you analyze how something new should work if it's used with older programs too. But with newer things like SMP or multithreading they probably have more choice, and you can only guess what it's done 'officially'.
Quoted 4 lines
if they required everyone to buy new software to use a new chip it wouldn't work well. In fact Intel tried to do exactly withat with the itanium and it has been a spectacular failure (or t the very least, not a spectacular sucess)
The failure of an architecture doesn't mean all specific new technologies used in itanium were failure too, so they could be back when needed (and nothing better in reserve) yet.
...
Quoted 4 lines
in theory they could change anything at any time, in practice if they break old software they won't sell the chips, so the modifications tend to be along the lines of this one, adding detail to the specifications so that programmers can get more performance.
I don't think 'not breaking' is much problem here, rather how to use all new features (which you seem to ignore a bit) to get maximum of performance without breaking older things. Or, like current problem:
go rational and remove useless (acording to new specs) things, even without performance gain, or stay 'safe'?
Jarek P.
RE: [rfc][patch 3/3] x86: optimise barriers
In reply to Jarek PoplawskiQuoted 4 lines
From: Intel(R) 64 and IA-32 Architectures Software Developer's Manual Volume 3A:
Quoted 8 lines
"7.2.2 Memory Ordering in P6 and More Recent Processor Families
... 1. Reads can be carried out speculatively and in any order. ..."So, it looks to me like almost the 1-st Commandment. Some people (like me) did believe this, others tried to check, and it was respected for years notwithstanding nobody had ever seen such an event.
When Intel first added speculative loads to the x86 family, they pegged the speculative load to the cache line. If the cache line is invalidated, so is the speculative load. As a result, out-of-order reads to normal memory are invisible to software. If a write to the same memory location on another CPU would make the fetched value invalid, it will make the cache line invalid, which invalidates the fetch.
I think it's extremely unlikely that any x86 CPU will do this any differently. It's hard to imagine Intel and AMD would go to all this trouble for so long just to stop so late in the line's lifetime.
DS