barriers using data dependency
Subsystems (9)
Michael S. Tsirkin
maintainercover barriers using data dependency
So as explained in Documentation/memory-barriers.txt e.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.
Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.
This adds an API for this and uses it in virtio.
Written over the holiday and build tested only so far.
This patchset is also suboptimal on e.g. x86 where e.g. smp_rmb is a nop.
Sending out for early feedback/flames.
Michael S. Tsirkin (4):
include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR include/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: use dependent_ptr_mb
Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + drivers/virtio/virtio_ring.c | 6 ++++-- include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 8 +++++++- 8 files changed, 53 insertions(+), 13 deletions(-)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 1/4 include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.
Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.
Also fix up comments to match reality since we are no longer overriding any macros.
Build-tested with gcc and clang.
--- include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 4 +++- 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
@@ -3,9 +3,8 @@#error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."#endif -/* Some compiler specific definitions are overwritten here -* for Clang compiler -*/ +/* Compiler specific definitions for Clang compiler */ +#define uninitialized_var (x) x = *(&(x))/* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
@@ -58,10 +58,6 @@ (typeof (ptr)) (__ptr + (off)); \ }) -/* Make the optimizer believe the variable can be manipulated arbitrarily. */ -#define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) -/* * A trick to suppress uninitialized variable warning without generating any * code
include/linux/compiler-intel.h +1 -3
@@ -5,9 +5,7 @@#ifdef __ECC -/* Some compiler specific definitions are overwritten here -* for Intel ECC compiler -*/ +/* Compiler specific definitions for Intel ECC compiler */ #include <asm/intrinsics.h>
include/linux/compiler.h +3 -1
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR -#define OPTIMIZER_HIDE_VAR (var) barrier() +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ +#define OPTIMIZER_HIDE_VAR (var) \ + __asm__ ("" : "=r" (var) : "0" (var))#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.
Quoted 6 lines · Michael S. Tsirkin
Since commit
815f0ddb346c("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses theOPTIMIZER_HIDE_VARmacro from compiler-gcc - instead it gets the version ininclude/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.
Good catch. Did you find this via eyeballing the code, a test, or some other way?
Quoted 8 lines
Fix up by moving the macro out from
compiler-gcc.htocompiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.Also fix up comments to match reality since we are no longer overriding any macros.
Quoted 8 lines
Build-tested with gcc and clang.
Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")Cc Eli Friedman <[email protected]>Cc Joe Perches <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Linus Torvalds <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>
Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")
Quoted 58 lines
--- include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 4 +++- 4 files changed, 6 insertions(+), 11 deletions(-)include/linux/compiler-clang.h +2
-3@@ -3,9 +3,8 @@#error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."#endif -/* Some compiler specific definitions are overwritten here -* for Clang compiler -*/ +/* Compiler specific definitions for Clang compiler */ +#define uninitialized_var (x) x = *(&(x))/* same as gcc, this was present in clang-2.6 so we can assume it works include/linux/compiler-gcc.h +0
-4@@ -58,10 +58,6 @@ (typeof (ptr)) (__ptr + (off)); \ }) -/* Make the optimizer believe the variable can be manipulated arbitrarily. */ -#define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) -/* * A trick to suppress uninitialized variable warning without generating any * code include/linux/compiler-intel.h +1
-3@@ -5,9 +5,7 @@#ifdef __ECC -/* Some compiler specific definitions are overwritten here -* for Intel ECC compiler -*/ +/* Compiler specific definitions for Intel ECC compiler */ #include <asm/intrinsics.h> include/linux/compiler.h +0
-0
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158
Quoted 8 lines
include/linux/compiler.h +3
-1@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR -#define OPTIMIZER_HIDE_VAR (var) barrier() +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ +#define OPTIMIZER_HIDE_VAR (var) \ + __asm__ ("" : "=r" (var) : "0" (var))#endif
This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.
+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?
Signature · 2 lines
Thanks, ~Nick Desaulniers
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 4 lines · Nick Desaulniers
Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.
Quoted 1 line · Michael S. Tsirkin
Quoted 8 lines
Quoted 5 lines
Since commit
815f0ddb346c("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses theOPTIMIZER_HIDE_VARmacro from compiler-gcc - instead it gets the version ininclude/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.Good catch. Did you find this via eyeballing the code, a test, or some other way?
eyeballing
Quoted 17 lines
Quoted 8 lines
Fix up by moving the macro out from
compiler-gcc.htocompiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.Also fix up comments to match reality since we are no longer overriding any macros.
Quoted 8 lines
Build-tested with gcc and clang.
Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")Cc Eli Friedman <[email protected]>Cc Joe Perches <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Linus Torvalds <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>
Quoted 3 lines
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")
Interesting. That added this text:
* while gcc behavior gets along with a normal
* barrier(), llvm needs an explicit input variable to be assumed
* clobbered.
however:
#define barrier_data(ptr) __asm__ __volatile__("": :"r"(ptr) :"memory")
So no explicit variable is clobbered.
Weird isn't it?
Quoted 81 lines
Quoted 58 lines
--- include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 4 +++- 4 files changed, 6 insertions(+), 11 deletions(-)include/linux/compiler-clang.h +2
-3@@ -3,9 +3,8 @@#error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."#endif -/* Some compiler specific definitions are overwritten here -* for Clang compiler -*/ +/* Compiler specific definitions for Clang compiler */ +#define uninitialized_var (x) x = *(&(x))/* same as gcc, this was present in clang-2.6 so we can assume it works include/linux/compiler-gcc.h +0
-4@@ -58,10 +58,6 @@ (typeof (ptr)) (__ptr + (off)); \ }) -/* Make the optimizer believe the variable can be manipulated arbitrarily. */ -#define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) -/* * A trick to suppress uninitialized variable warning without generating any * code include/linux/compiler-intel.h +1
-3@@ -5,9 +5,7 @@#ifdef __ECC -/* Some compiler specific definitions are overwritten here -* for Intel ECC compiler -*/ +/* Compiler specific definitions for Intel ECC compiler */ #include <asm/intrinsics.h> include/linux/compiler.h +0
-0include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158
Quoted 8 lines
include/linux/compiler.h +3
-1@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR -#define OPTIMIZER_HIDE_VAR (var) barrier() +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ +#define OPTIMIZER_HIDE_VAR (var) \ + __asm__ ("" : "=r" (var) : "0" (var))#endif This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Signature · 2 lines
Thanks, ~Nick Desaulniers
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Nick DesaulniersQuoted 4 lines · Nick Desaulniers
Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.
Quoted 1 line · Michael S. Tsirkin
Quoted 8 lines
Quoted 5 lines
Since commit
815f0ddb346c("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses theOPTIMIZER_HIDE_VARmacro from compiler-gcc - instead it gets the version ininclude/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.Good catch. Did you find this via eyeballing the code, a test, or some other way?
eyeballing
Quoted 17 lines
Quoted 8 lines
Fix up by moving the macro out from
compiler-gcc.htocompiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.Also fix up comments to match reality since we are no longer overriding any macros.
Quoted 8 lines
Build-tested with gcc and clang.
Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")Cc Eli Friedman <[email protected]>Cc Joe Perches <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Linus Torvalds <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>
Quoted 3 lines
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")
Interesting. That added this text:
* while gcc behavior gets along with a normal
* barrier(), llvm needs an explicit input variable to be assumed
* clobbered.
however:
#define barrier_data(ptr) __asm__ __volatile__("": :"r"(ptr) :"memory")
So no explicit variable is clobbered.
Weird isn't it?
Quoted 81 lines
Quoted 58 lines
--- include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 4 +++- 4 files changed, 6 insertions(+), 11 deletions(-)include/linux/compiler-clang.h +2
-3@@ -3,9 +3,8 @@#error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."#endif -/* Some compiler specific definitions are overwritten here -* for Clang compiler -*/ +/* Compiler specific definitions for Clang compiler */ +#define uninitialized_var (x) x = *(&(x))/* same as gcc, this was present in clang-2.6 so we can assume it works include/linux/compiler-gcc.h +0
-4@@ -58,10 +58,6 @@ (typeof (ptr)) (__ptr + (off)); \ }) -/* Make the optimizer believe the variable can be manipulated arbitrarily. */ -#define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) -/* * A trick to suppress uninitialized variable warning without generating any * code include/linux/compiler-intel.h +1
-3@@ -5,9 +5,7 @@#ifdef __ECC -/* Some compiler specific definitions are overwritten here -* for Intel ECC compiler -*/ +/* Compiler specific definitions for Intel ECC compiler */ #include <asm/intrinsics.h> include/linux/compiler.h +0
-0include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158
Quoted 8 lines
include/linux/compiler.h +3
-1@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR -#define OPTIMIZER_HIDE_VAR (var) barrier() +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ +#define OPTIMIZER_HIDE_VAR (var) \ + __asm__ ("" : "=r" (var) : "0" (var))#endif This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Nick DesaulniersQuoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")
By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?
Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?
Sure, at least we get quickly some linux-next time.
Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().
Cheers,
Miguel
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 14 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?
Quoted 8 lines
Sure, at least we get quickly some linux-next time.
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().Cheers,
Miguel
Okay, I will try.
Signature · 1 line
MST
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Hi Michael,
Quoted 7 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().Okay, I will try.
Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the barrier_data later.
Cheers,
Miguel
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 9 lines · Miguel Ojeda
Hi Michael,
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().
Quoted 8 lines
Quoted 1 line
Okay, I will try.
Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the
barrier_datalater.Cheers,
Miguel
No not yet. Sorry! Pls send this one in, barrier_data will likely miss the next merge window.
Signature · 1 line
MSR
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 3 lines · Michael S. Tsirkin
No not yet. Sorry! Pls send this one in,
barrier_datawill likely miss the next merge window.
No worries! Done.
Cheers,
Miguel
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Miguel OjedaQuoted 9 lines · Miguel Ojeda
Hi Michael,
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().
Quoted 8 lines
Quoted 1 line
Okay, I will try.
Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the
barrier_datalater.Cheers,
Miguel
No not yet. Sorry! Pls send this one in, barrier_data will likely miss the next merge window.
Signature · 1 line
MSR
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Miguel OjedaQuoted 14 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?
Quoted 8 lines
Sure, at least we get quickly some linux-next time.
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().Cheers,
Miguel
Okay, I will try.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Miguel OjedaQuoted 9 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?
Quoted 6 lines
Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Sure, at least we get quickly some linux-next time.
BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.
Quoted 6 lines
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().Cheers,
Miguel
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?Quoted 2 lines
Reviewed-by Nick Desaulniers <[email protected]>
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Sure, at least we get quickly some linux-next time.
BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.
It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.
regards,
dan carpenter
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
Quoted 19 lines · Dan Carpenter
Quoted 18 lines · Michael S. Tsirkin
Quoted 15 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Sure, at least we get quickly some linux-next time.
Quoted 8 lines
Quoted 2 lines
BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.
It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.
regards,
dan carpenter
I misunderstood. Sure that makes sense.
Signature · 1 line
MST
Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Michael S. TsirkinQuoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?Quoted 2 lines
Reviewed-by Nick Desaulniers <[email protected]>
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Sure, at least we get quickly some linux-next time.
BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.
It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.
regards,
dan carpenter
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Miguel OjedaQuoted 9 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers
Also for more context, see:
commit7829fb09a2b4("lib: makememzero_explicitmore robust against dead store elimination")By the way, shouldn't that
barrier_data()be directly incompiler.htoo, since it is for both gcc & clang?
Quoted 6 lines
Quoted 4 lines
Reviewed-by Nick Desaulniers <[email protected]>+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?Sure, at least we get quickly some linux-next time.
BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.
Quoted 6 lines
Note it would be nice to separate the patch into two (one for the comments, another for
OPTIMIZER_HIDE_VAR), and also possibly another forbarrier_data().Cheers,
Miguel
Michael S. Tsirkin
maintainerRFC PATCH 1/4 include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR
In reply to Michael S. TsirkinSince commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.
Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.
Also fix up comments to match reality since we are no longer overriding any macros.
Build-tested with gcc and clang.
--- include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 4 +++- 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
@@ -3,9 +3,8 @@#error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."#endif -/* Some compiler specific definitions are overwritten here -* for Clang compiler -*/ +/* Compiler specific definitions for Clang compiler */ +#define uninitialized_var (x) x = *(&(x))/* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
@@ -58,10 +58,6 @@ (typeof (ptr)) (__ptr + (off)); \ }) -/* Make the optimizer believe the variable can be manipulated arbitrarily. */ -#define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) -/* * A trick to suppress uninitialized variable warning without generating any * code
include/linux/compiler-intel.h +1 -3
@@ -5,9 +5,7 @@#ifdef __ECC -/* Some compiler specific definitions are overwritten here -* for Intel ECC compiler -*/ +/* Compiler specific definitions for Intel ECC compiler */ #include <asm/intrinsics.h>
include/linux/compiler.h +3 -1
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR -#define OPTIMIZER_HIDE_VAR (var) barrier() +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ +#define OPTIMIZER_HIDE_VAR (var) \ + __asm__ ("" : "=r" (var) : "0" (var))#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 2/4 include/linux/compiler.h: allow memory operands
In reply to Michael S. TsirkinWe don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.
--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +1 -1
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#ifndef OPTIMIZER_HIDE_VAR /* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) + __asm__ ("" : "=rm" (var) : "0" (var))#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Re: [PATCH RFC 2/4] include/linux/compiler.h: allow memory operands
Quoted 13 lines · Michael S. Tsirkin
We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.
Signed-off-by Michael S. Tsirkin <[email protected]>--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/compiler.h +0
-0
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158
Quoted 6 lines
include/linux/compiler.h +1
-1@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#ifndef OPTIMIZER_HIDE_VAR /* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) + __asm__ ("" : "=rm" (var) : "0" (var))#endif
I think this can break for architectures with write-back addressing modes such as arm, where the "m" constraint is assumed to be evaluated precisely once in the asm block.
Will
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/4] include/linux/compiler.h: allow memory operands
Quoted 25 lines · Will Deacon
Quoted 13 lines · Michael S. Tsirkin
We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.
Signed-off-by Michael S. Tsirkin <[email protected]>--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/compiler.h +0
-0include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158
Quoted 6 lines
include/linux/compiler.h +1
-1@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#ifndef OPTIMIZER_HIDE_VAR /* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) + __asm__ ("" : "=rm" (var) : "0" (var))#endif I think this can break for architectures with write-back addressing modes such as arm, where the "m" constraint is assumed to be evaluated precisely once in the asm block.
Quoted 1 line
Will
Thanks, I'll drop this patch.
Michael S. Tsirkin
maintainerRFC PATCH 2/4 include/linux/compiler.h: allow memory operands
In reply to Michael S. TsirkinWe don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.
--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +1 -1
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#ifndef OPTIMIZER_HIDE_VAR /* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ - __asm__ ("" : "=r" (var) : "0" (var)) + __asm__ ("" : "=rm" (var) : "0" (var))#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 3/4 barriers: convert a control to a data dependency
In reply to Michael S. TsirkinIt's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.
However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.
This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.
--- Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler.h | 4 ++++ 4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
@@ -691,6 +691,18 @@ case what's actually required is: p = READ_ONCE(b); } +Alternatively, a control dependency can be converted to a data dependency, +e.g.: + + q = READ_ONCE(a); + if (q) { + b = dependent_ptr_mb(b, q); + p = READ_ONCE(b); + } + +Note how the result of dependent_ptr_mb must be used with the following +accesses in order to have an effect. + However, stores are not speculated. This means that ordering -is- provided for load-store control dependencies, as in the following example:
@@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force the compiler to actually emit code for a given load, it does not force the compiler to use the results. +Converting to a data dependency helps with this too: + + q = READ_ONCE(a); + b = dependent_ptr_mb(b, q); + WRITE_ONCE(b, 1); + In addition, control dependencies apply only to the then-clause and else-clause of the if-statement in question. In particular, it does not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
@@ -59,6 +59,7 @@* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() * in cases like this where there are no data dependencies. */ +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory")#ifdef CONFIG_SMP
include/asm-generic/barrier.h +18 -0
@@ -70,6 +70,24 @@#define __smp_read_barrier_depends () read_barrier_depends()#endif +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \ + long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \ + \ + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \ + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \ + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); }) + +#endif +#ifdef CONFIG_SMP #ifndef smp_mb
include/linux/compiler.h +4 -0
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ __asm__ ("" : "=rm" (var) : "0" (var)) + +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1 +#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900
Quoted 9 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +2
-0@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.
Was this hunk intentional?
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +0
-0@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information.
Quoted 7 lines
Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900
Quoted 5 lines
Documentation/memory-barriers.txt +2
-0+ + In summary: (*) Control dependencies can order prior loads against later stores.Was this hunk intentional?
Nope, thanks for catching this.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Matthew WilcoxQuoted 5 lines · Matthew Wilcox
Quoted 4 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +0
-0@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information.
Quoted 7 lines
Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900
Quoted 5 lines
Documentation/memory-barriers.txt +2
-0+ + In summary: (*) Control dependencies can order prior loads against later stores.Was this hunk intentional?
Nope, thanks for catching this.
Signature · 1 line
MST
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Michael S. TsirkinDocumentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900
Quoted 9 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +2
-0@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.
Was this hunk intentional?
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Michael S. TsirkinQuoted 88 lines · Michael S. Tsirkin
It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.
However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.
This adds a new interface:
dependent_ptr_mbwhich does exactly this: it returns a pointer with a data dependency on the supplied value.Signed-off-by Michael S. Tsirkin <[email protected]>--- Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler.h | 4 ++++ 4 files changed, 43 insertions(+)Documentation/memory-barriers.txt +20
-0@@ -691,6 +691,18 @@ case what's actually required is: p = READ_ONCE(b); } +Alternatively, a control dependency can be converted to a data dependency, +e.g.: + + q = READ_ONCE(a); + if (q) { + b = dependent_ptr_mb(b, q); + p = READ_ONCE(b); + } + +Note how the result of dependent_ptr_mb must be used with the following +accesses in order to have an effect. + However, stores are not speculated. This means that ordering -is- provided for load-store control dependencies, as in the following example:@@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force the compiler to actually emit code for a given load, it does not force the compiler to use the results. +Converting to a data dependency helps with this too: + + q = READ_ONCE(a); + b = dependent_ptr_mb(b, q); + WRITE_ONCE(b, 1); + In addition, control dependencies apply only to the then-clause and else-clause of the if-statement in question. In particular, it does not necessarily apply to code following the if-statement:@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.arch/alpha/include/asm/barrier.h +1
-0@@ -59,6 +59,7 @@* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() * in cases like this where there are no data dependencies. */ +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory")#ifdef CONFIG_SMP include/asm-generic/barrier.h +5
-0@@ -70,6 +70,24 @@#define __smp_read_barrier_depends () read_barrier_depends()#endif +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \
include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })
So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?
Thanks
Quoted 14 lines
include/asm-generic/barrier.h +3
-0+ +#endif +#ifdef CONFIG_SMP #ifndef smp_mbinclude/linux/compiler.h +0
-0@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 164
Quoted 10 lines
include/linux/compiler.h +4
-0+/* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ __asm__ ("" : "=rm" (var) : "0" (var)) + +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1 +#endif /* Not-quite-unique ID. */
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 97 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin
It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.
However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.
This adds a new interface:
dependent_ptr_mbwhich does exactly this: it returns a pointer with a data dependency on the supplied value.Signed-off-by Michael S. Tsirkin <[email protected]>--- Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler.h | 4 ++++ 4 files changed, 43 insertions(+)Documentation/memory-barriers.txt +20
-0@@ -691,6 +691,18 @@ case what's actually required is: p = READ_ONCE(b); } +Alternatively, a control dependency can be converted to a data dependency, +e.g.: + + q = READ_ONCE(a); + if (q) { + b = dependent_ptr_mb(b, q); + p = READ_ONCE(b); + } + +Note how the result of dependent_ptr_mb must be used with the following +accesses in order to have an effect. + However, stores are not speculated. This means that ordering -is- provided for load-store control dependencies, as in the following example:@@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force the compiler to actually emit code for a given load, it does not force the compiler to use the results. +Converting to a data dependency helps with this too: + + q = READ_ONCE(a); + b = dependent_ptr_mb(b, q); + WRITE_ONCE(b, 1); + In addition, control dependencies apply only to the then-clause and else-clause of the if-statement in question. In particular, it does not necessarily apply to code following the if-statement:@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.arch/alpha/include/asm/barrier.h +1
-0@@ -59,6 +59,7 @@* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() * in cases like this where there are no data dependencies. */ +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory")#ifdef CONFIG_SMP include/asm-generic/barrier.h +5
-0@@ -70,6 +70,24 @@#define __smp_read_barrier_depends () read_barrier_depends()#endif +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 86
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?
Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.
mb variant is unused right now so I'll remove it.
Quoted 14 lines
Quoted 14 lines
include/asm-generic/barrier.h +3
-0+ +#endif +#ifdef CONFIG_SMP #ifndef smp_mbinclude/linux/compiler.h +1
-0@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR +
Quoted 8 lines
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 164
Quoted 8 lines
include/linux/compiler.h +3
-0/* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ __asm__ ("" : "=rm" (var) : "0" (var)) + +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1 +#endif /* Not-quite-unique ID. */
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 95 lines · Michael S. Tsirkin
Quoted 94 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin
It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.
However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.
This adds a new interface:
dependent_ptr_mbwhich does exactly this: it returns a pointer with a data dependency on the supplied value.Signed-off-by Michael S. Tsirkin<[email protected]>--- Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler.h | 4 ++++ 4 files changed, 43 insertions(+)Documentation/memory-barriers.txt +20
-0@@ -691,6 +691,18 @@ case what's actually required is: p = READ_ONCE(b); } +Alternatively, a control dependency can be converted to a data dependency, +e.g.: + + q = READ_ONCE(a); + if (q) { + b = dependent_ptr_mb(b, q); + p = READ_ONCE(b); + } + +Note how the result of dependent_ptr_mb must be used with the following +accesses in order to have an effect. + However, stores are not speculated. This means that ordering -is- provided for load-store control dependencies, as in the following example:@@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force the compiler to actually emit code for a given load, it does not force the compiler to use the results. +Converting to a data dependency helps with this too: + + q = READ_ONCE(a); + b = dependent_ptr_mb(b, q); + WRITE_ONCE(b, 1); + In addition, control dependencies apply only to the then-clause and else-clause of the if-statement in question. In particular, it does not necessarily apply to code following the if-statement:@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.arch/alpha/include/asm/barrier.h +1
-0@@ -59,6 +59,7 @@* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() * in cases like this where there are no data dependencies. */ +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory")#ifdef CONFIG_SMP include/asm-generic/barrier.h +5
-0@@ -70,6 +70,24 @@#define __smp_read_barrier_depends () read_barrier_depends()#endif +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 86
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?
Quoted 8 lines
Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.mb variant is unused right now so I'll remove it.
Yes.
Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Peter Zijlstra
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Michael S. TsirkinQuoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?
Quoted 7 lines
Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.mb variant is unused right now so I'll remove it.
How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.
Quoted 6 lines
Quoted 1 line
mb variant is unused right now so I'll remove it.
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.
So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?
Thanks,
Signature · 5 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Peter Zijlstra
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.
Quoted 8 lines
So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.
The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.
Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?
Not sure; what is the actual overhead of the construct on x86 vs the NOP?
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.Quoted 8 lines
So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.
Quoted 2 lines
The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.
But not on intel right? On intel loads are ordered so it can be a nop.
Quoted 8 lines
Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?Not sure; what is the actual overhead of the construct on x86 vs the NOP?
I'll have to check. There's a pipeline stall almost for sure - that's why we put it there after all :).
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Peter ZijlstraQuoted 17 lines · Peter Zijlstra
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.Quoted 8 lines
So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.
Quoted 2 lines
The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.
But not on intel right? On intel loads are ordered so it can be a nop.
Quoted 8 lines
Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?Not sure; what is the actual overhead of the construct on x86 vs the NOP?
I'll have to check. There's a pipeline stall almost for sure - that's why we put it there after all :).
Signature · 1 line
MST
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Michael S. TsirkinQuoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.Quoted 6 lines
Quoted 1 line
mb variant is unused right now so I'll remove it.
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.
Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?
But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?
Thanx, Paul
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.Quoted 6 lines
Quoted 1 line
mb variant is unused right now so I'll remove it.
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?
Quoted 5 lines
But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?
Thanx, Paul
Oh you are right. Stores are not reordered with older loads on x86.
So point 2 is moot. Sorry about the noise.
I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an mb() there, seeing as it doesn't even have spectre protection enabled. Slow but safe.
Signature · 1 line
MST
Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency
Quoted 61 lines · Michael S. Tsirkin
Quoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
(quoted diff) +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS) + +#define dependent_ptr_mb(ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \Review
Quoted 10 lines
(quoted diff) +10
-0+ long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \ + \ + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \ + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \ + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.Quoted 6 lines
Quoted 1 line
mb variant is unused right now so I'll remove it.
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?Quoted 5 lines
But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?
Thanx, Paul
Quoted 7 lines
Oh you are right. Stores are not reordered with older loads on x86.
So point 2 is moot. Sorry about the noise.
I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an
mb()there, seeing as it doesn't even have spectre protection enabled. Slow but safe.
Well, there is a mention of DMA above, which on some systems throws in a wild card. I would certainly hope that DMA would integrate nicely with the cache-coherence protocols these days, unlike 25 years ago, but who knows?
Thanx, Paul
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/4] barriers: convert a control to a data dependency
In reply to Paul E. McKenneyQuoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5
-0+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \include/asm-generic/barrier.h line 87
Quoted 10 lines
include/asm-generic/barrier.h +10
-0+long dependent_ptr_mb_ptr = (long )(ptr) - dependent_ptr_mb_val; \ + \ +BUILD_BUG_ON (sizeof (val) >sizeof (long )); \ +OPTIMIZER_HIDE_VAR (dependent_ptr_mb_val); \ + (typeof (ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); })So for the example of patch 4, we'd better fall back to
rmb()or need adependent_ptr_rmb()?Quoted 1 line
Thanks
You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have
dependent_ptr_smp_rmb,dependent_ptr_dma_rmbanddependent_ptr_virt_rmb.Quoted 6 lines
Quoted 1 line
mb variant is unused right now so I'll remove it.
How about naming the thing:
dependent_ptr()? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.So I'm fine with this as it's enough for virtio, but I would like to point out two things:
1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want
smp_dependent_ptr(),dma_dependent_ptr(),dependent_ptr()or we just will specify thatdependent_ptr()works for both DMA and SMP.Quoted 5 lines
2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make
dependent_ptra NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to renamedependent_ptrtodependent_ptr_rmbat that point?
Quoted 5 lines
But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?
Thanx, Paul
Oh you are right. Stores are not reordered with older loads on x86.
So point 2 is moot. Sorry about the noise.
I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an mb() there, seeing as it doesn't even have spectre protection enabled. Slow but safe.
Signature · 5 lines
MST _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRFC PATCH 3/4 barriers: convert a control to a data dependency
In reply to Michael S. TsirkinIt's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.
However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.
This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.
--- Documentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler.h | 4 ++++ 4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
@@ -691,6 +691,18 @@ case what's actually required is: p = READ_ONCE(b); } +Alternatively, a control dependency can be converted to a data dependency, +e.g.: + + q = READ_ONCE(a); + if (q) { + b = dependent_ptr_mb(b, q); + p = READ_ONCE(b); + } + +Note how the result of dependent_ptr_mb must be used with the following +accesses in order to have an effect. + However, stores are not speculated. This means that ordering -is- provided for load-store control dependencies, as in the following example:
@@ -836,6 +848,12 @@ out-guess your code. More generally, although READ_ONCE() does force the compiler to actually emit code for a given load, it does not force the compiler to use the results. +Converting to a data dependency helps with this too: + + q = READ_ONCE(a); + b = dependent_ptr_mb(b, q); + WRITE_ONCE(b, 1); + In addition, control dependencies apply only to the then-clause and else-clause of the if-statement in question. In particular, it does not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it. See the section on "Multicopy atomicity" for more information. + + In summary: (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
@@ -59,6 +59,7 @@* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() * in cases like this where there are no data dependencies. */ +#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory")#ifdef CONFIG_SMP
include/asm-generic/barrier.h +18 -0
@@ -70,6 +70,24 @@#define __smp_read_barrier_depends () read_barrier_depends()#endif +#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR ) && \ + !defined(ARCH_NEEDS_READ_BARRIER_DEPENDS ) + +#define dependent_ptr_mb (ptr, val) ({ \ + long dependent_ptr_mb_val = (long)(val); \ + long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val; \ + \ + BUILD_BUG_ON(sizeof(val) > sizeof(long)); \ + OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val); \ + (typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val); \ +}) + +#else + +#define dependent_ptr_mb (ptr, val) ({ mb(); (ptr); }) + +#endif +#ifdef CONFIG_SMP #ifndef smp_mb
include/linux/compiler.h +4 -0
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,#endif #ifndef OPTIMIZER_HIDE_VAR +/* Make the optimizer believe the variable can be manipulated arbitrarily. */ #define OPTIMIZER_HIDE_VAR (var) \ __asm__ ("" : "=rm" (var) : "0" (var)) + +#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1 +#endif /* Not-quite-unique ID. */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 4/4 virtio: use dependent_ptr_mb
In reply to Michael S. TsirkinUse dependent_ptr_mb which is - on some architectures - more light-weight than an rmb.
--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +4 -2
@@ -702,6 +702,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,void *ret;unsigned int i;u16 last_used; +bool more;START_USE (vq);
@@ -710,14 +711,15 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,return NULL ; } -if (!more_used (vq)) { + more =more_used (vq); +if (!more) {pr_debug ("No more buffers in queue\n" );END_USE (vq);return NULL ; }/* Only get used array entries after they have been exposed by host. */ -virtio_rmb (vq->weak_barriers); + vq =dependent_ptr_mb (vq, more); last_used = (vq->last_used_idx & (vq->vring.num -1 )); i =virtio32_to_cpu (_vq->vdev, vq->vring.used->ring[last_used].id);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 4/4 virtio: use dependent_ptr_mb
In reply to Michael S. TsirkinUse dependent_ptr_mb which is - on some architectures - more light-weight than an rmb.
--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +4 -2
@@ -702,6 +702,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,void *ret;unsigned int i;u16 last_used; +bool more;START_USE (vq);
@@ -710,14 +711,15 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,return NULL ; } -if (!more_used (vq)) { + more =more_used (vq); +if (!more) {pr_debug ("No more buffers in queue\n" );END_USE (vq);return NULL ; }/* Only get used array entries after they have been exposed by host. */ -virtio_rmb (vq->weak_barriers); + vq =dependent_ptr_mb (vq, more); last_used = (vq->last_used_idx & (vq->vring.num -1 )); i =virtio32_to_cpu (_vq->vdev, vq->vring.used->ring[last_used].id);
Signature · 1 line
MST
Alan Stern
maintainerQuoted 5 lines · Michael S. Tsirkin
So as explained in
Documentation/memory-barriers.txte.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.
Quoted 7 lines
Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.
This adds an API for this and uses it in virtio.
Written over the holiday and build tested only so far.
You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However, tools/memory-model/* uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?
Quoted 1 line
This patchset is also suboptimal on e.g. x86 where e.g.
smp_rmbis a nop.
This should be easy to fix with an architecture-specific override.
Alan Stern
Quoted 17 lines
Sending out for early feedback/flames.
Michael S. Tsirkin (4):
include/linux/compiler*.h: fixOPTIMIZER_HIDE_VARinclude/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: usedependent_ptr_mbDocumentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + drivers/virtio/virtio_ring.c | 6 ++++-- include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 8 +++++++- 8 files changed, 53 insertions(+), 13 deletions(-)
Michael S. Tsirkin
maintainerQuoted 13 lines · Alan Stern
Quoted 11 lines · Michael S. Tsirkin
So as explained in
Documentation/memory-barriers.txte.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.
This adds an API for this and uses it in virtio.
Quoted 7 lines
Quoted 1 line
Written over the holiday and build tested only so far.
You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However,
tools/memory-model/*uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?
Sure, sounds good. While I'm at it, should memory-barriers.txt be switched over too?
Quoted 5 lines
Quoted 1 line
This patchset is also suboptimal on e.g. x86 where e.g.
smp_rmbis a nop.This should be easy to fix with an architecture-specific override.
Alan Stern
Absolutely. It does however mean that we'll need several
variants: mb/rmb, smp/dma/virt/mandatory.
I am still trying to decide whether it's good since it documents the kind of barrier that we are trying to use - or bad since it's more verbose and makes you choose one where they are all pretty cheap.
Quoted 17 lines
Quoted 17 lines
Sending out for early feedback/flames.
Michael S. Tsirkin (4):
include/linux/compiler*.h: fixOPTIMIZER_HIDE_VARinclude/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: usedependent_ptr_mbDocumentation/memory-barriers.txt | 20 ++++++++++++++++++++ arch/alpha/include/asm/barrier.h | 1 + drivers/virtio/virtio_ring.c | 6 ++++-- include/asm-generic/barrier.h | 18 ++++++++++++++++++ include/linux/compiler-clang.h | 5 ++--- include/linux/compiler-gcc.h | 4 ---- include/linux/compiler-intel.h | 4 +--- include/linux/compiler.h | 8 +++++++- 8 files changed, 53 insertions(+), 13 deletions(-)
Alan Stern
maintainerQuoted 16 lines · Michael S. Tsirkin
Quoted 15 lines · Alan Stern
Quoted 12 lines · Michael S. Tsirkin
So as explained in
Documentation/memory-barriers.txte.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.
This adds an API for this and uses it in virtio.
Written over the holiday and build tested only so far.
Quoted 8 lines
Quoted 5 lines
You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However,
tools/memory-model/*uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?Sure, sounds good. While I'm at it, should memory-barriers.txt be switched over too?
If you want to take care of that, great! I never seem to get around to doing it.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
This patchset is also suboptimal on e.g. x86 where e.g.
smp_rmbis a nop.This should be easy to fix with an architecture-specific override.
Quoted 8 lines
Quoted 1 line
Alan Stern
Absolutely. It does however mean that we'll need several
variants: mb/rmb, smp/dma/virt/mandatory.I am still trying to decide whether it's good since it documents the kind of barrier that we are trying to use - or bad since it's more verbose and makes you choose one where they are all pretty cheap.
How many places can these things be used? My guess is not very many, or at least, there aren't very many different _types_ of usage. So start only with variants you know will be used. More can be added later if we want.
Alan Stern