Skip to contentSkip to conversation

barriers using data dependency

RFC PATCH In review Michael S. Tsirkin 10 participants 45 messages Jan 2, 2019 – Jan 20, 2019

Subsystems (9)
lore
Share & tools
lore

Michael S. Tsirkin

author maintainer
#0 lore raw

cover barriers using data dependency

So as explained in Documentation/memory-barriers.txt e.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.

Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.

This adds an API for this and uses it in virtio.

Written over the holiday and build tested only so far.

This patchset is also suboptimal on e.g. x86 where e.g. smp_rmb is a nop.

Sending out for early feedback/flames.

Michael S. Tsirkin (4):
include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR include/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: use dependent_ptr_mb

 Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
 arch/alpha/include/asm/barrier.h  |  1 +
 drivers/virtio/virtio_ring.c      |  6 ++++--
 include/asm-generic/barrier.h     | 18 ++++++++++++++++++
 include/linux/compiler-clang.h    |  5 ++---
 include/linux/compiler-gcc.h      |  4 ----
 include/linux/compiler-intel.h    |  4 +---
 include/linux/compiler.h          |  8 +++++++-
 8 files changed, 53 insertions(+), 13 deletions(-)
Signature · 1 line
MST
author maintainer
#1 lore raw

RFC PATCH 1/4 include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.

Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.

Also fix up comments to match reality since we are no longer overriding any macros.

Build-tested with gcc and clang.

Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")
Cc Eli Friedman <[email protected]>
Cc Joe Perches <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Linus Torvalds <[email protected]>
---
 include/linux/compiler-clang.h | 5 ++---
 include/linux/compiler-gcc.h   | 4 ----
 include/linux/compiler-intel.h | 4 +---
 include/linux/compiler.h       | 4 +++-
 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
index 3e7dafb3ea80..7ddaeb5182e3 100644
@@ -3,9 +3,8 @@
 #error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."
 #endif
 
-/* Some compiler specific definitions are overwritten here
- * for Clang compiler
- */
+/* Compiler specific definitions for Clang compiler */
+
 #define uninitialized_var(x) x = *(&(x))
 
 /* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
index 2010493e1040..72054d9f0eaa 100644
@@ -58,10 +58,6 @@
 	(typeof(ptr)) (__ptr + (off));					\
 })
 
-/* Make the optimizer believe the variable can be manipulated arbitrarily. */
-#define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
-
 /*
  * A trick to suppress uninitialized variable warning without generating any
  * code
include/linux/compiler-intel.h +1 -3
index 517bd14e1222..b17f3cd18334 100644
@@ -5,9 +5,7 @@
 
 #ifdef __ECC
 
-/* Some compiler specific definitions are overwritten here
- * for Intel ECC compiler
- */
+/* Compiler specific definitions for Intel ECC compiler */
 
 #include <asm/intrinsics.h>
 
include/linux/compiler.h +3 -1
index 06396c1cf127..1ad367b4cd8d 100644
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
-#define OPTIMIZER_HIDE_VAR(var) barrier()
+/* Make the optimizer believe the variable can be manipulated arbitrarily. */
+#define OPTIMIZER_HIDE_VAR(var)						\
+	__asm__ ("" : "=r" (var) : "0" (var))
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
reviewer Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.

Quoted 6 lines · Michael S. Tsirkin

Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.

Good catch. Did you find this via eyeballing the code, a test, or some other way?

Quoted 8 lines

Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.

Also fix up comments to match reality since we are no longer overriding any macros.

Quoted 8 lines

Build-tested with gcc and clang.

Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")
Cc Eli Friedman <[email protected]>
Cc Joe Perches <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Linus Torvalds <[email protected]>

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

Quoted 58 lines
---
 include/linux/compiler-clang.h | 5 ++---
 include/linux/compiler-gcc.h   | 4 ----
 include/linux/compiler-intel.h | 4 +---
 include/linux/compiler.h       | 4 +++-
 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
index 3e7dafb3ea80..7ddaeb5182e3 100644
@@ -3,9 +3,8 @@
 #error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."
 #endif
 
-/* Some compiler specific definitions are overwritten here
- * for Clang compiler
- */
+/* Compiler specific definitions for Clang compiler */
+
 #define uninitialized_var(x) x = *(&(x))
 
 /* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
index 2010493e1040..72054d9f0eaa 100644
@@ -58,10 +58,6 @@
        (typeof(ptr)) (__ptr + (off));                                  \
 })
 
-/* Make the optimizer believe the variable can be manipulated arbitrarily. */
-#define OPTIMIZER_HIDE_VAR(var)                                                \
-       __asm__ ("" : "=r" (var) : "0" (var))
-
 /*
  * A trick to suppress uninitialized variable warning without generating any
  * code
include/linux/compiler-intel.h +1 -3
index 517bd14e1222..b17f3cd18334 100644
@@ -5,9 +5,7 @@
 
 #ifdef __ECC
 
-/* Some compiler specific definitions are overwritten here
- * for Intel ECC compiler
- */
+/* Compiler specific definitions for Intel ECC compiler */
 
 #include <asm/intrinsics.h>
 
include/linux/compiler.h +0 -0
index 06396c1cf127..1ad367b4cd8d 100644

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158

Quoted 8 lines
include/linux/compiler.h +3 -1
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
-#define OPTIMIZER_HIDE_VAR(var) barrier()
+/* Make the optimizer believe the variable can be manipulated arbitrarily. */
+#define OPTIMIZER_HIDE_VAR(var)                                                \
+       __asm__ ("" : "=r" (var) : "0" (var))
 #endif

This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Signature · 2 lines
Thanks,
~Nick Desaulniers
author maintainer
#3 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 4 lines · Nick Desaulniers

Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.

Quoted 1 line · Michael S. Tsirkin
Quoted 8 lines
Quoted 5 lines

Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.

Good catch. Did you find this via eyeballing the code, a test, or some other way?

eyeballing

Quoted 17 lines
Quoted 8 lines

Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.

Also fix up comments to match reality since we are no longer overriding any macros.

Quoted 8 lines

Build-tested with gcc and clang.

Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")
Cc Eli Friedman <[email protected]>
Cc Joe Perches <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Linus Torvalds <[email protected]>
Quoted 3 lines

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

Interesting. That added this text:

* while gcc behavior gets along with a normal
* barrier(), llvm needs an explicit input variable to be assumed
* clobbered.

however:
#define barrier_data(ptr) __asm__ __volatile__("": :"r"(ptr) :"memory")

So no explicit variable is clobbered.
Weird isn't it?

Quoted 81 lines
Quoted 58 lines
---
 include/linux/compiler-clang.h | 5 ++---
 include/linux/compiler-gcc.h   | 4 ----
 include/linux/compiler-intel.h | 4 +---
 include/linux/compiler.h       | 4 +++-
 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
index 3e7dafb3ea80..7ddaeb5182e3 100644
@@ -3,9 +3,8 @@
 #error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."
 #endif
 
-/* Some compiler specific definitions are overwritten here
- * for Clang compiler
- */
+/* Compiler specific definitions for Clang compiler */
+
 #define uninitialized_var(x) x = *(&(x))
 
 /* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
index 2010493e1040..72054d9f0eaa 100644
@@ -58,10 +58,6 @@
        (typeof(ptr)) (__ptr + (off));                                  \
 })
 
-/* Make the optimizer believe the variable can be manipulated arbitrarily. */
-#define OPTIMIZER_HIDE_VAR(var)                                                \
-       __asm__ ("" : "=r" (var) : "0" (var))
-
 /*
  * A trick to suppress uninitialized variable warning without generating any
  * code
include/linux/compiler-intel.h +1 -3
index 517bd14e1222..b17f3cd18334 100644
@@ -5,9 +5,7 @@
 
 #ifdef __ECC
 
-/* Some compiler specific definitions are overwritten here
- * for Intel ECC compiler
- */
+/* Compiler specific definitions for Intel ECC compiler */
 
 #include <asm/intrinsics.h>
 
include/linux/compiler.h +0 -0
index 06396c1cf127..1ad367b4cd8d 100644

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158

Quoted 8 lines
include/linux/compiler.h +3 -1
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
-#define OPTIMIZER_HIDE_VAR(var) barrier()
+/* Make the optimizer believe the variable can be manipulated arbitrarily. */
+#define OPTIMIZER_HIDE_VAR(var)                                                \
+       __asm__ ("" : "=r" (var) : "0" (var))
 #endif

This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Signature · 2 lines
Thanks,
~Nick Desaulniers
author maintainer
#4 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Nick Desaulniers
Quoted 4 lines · Nick Desaulniers

Thanks for the patch and sorry for the delay; was totally unplugged for the holidays.

Quoted 1 line · Michael S. Tsirkin
Quoted 8 lines
Quoted 5 lines

Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.

Good catch. Did you find this via eyeballing the code, a test, or some other way?

eyeballing

Quoted 17 lines
Quoted 8 lines

Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.

Also fix up comments to match reality since we are no longer overriding any macros.

Quoted 8 lines

Build-tested with gcc and clang.

Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")
Cc Eli Friedman <[email protected]>
Cc Joe Perches <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Linus Torvalds <[email protected]>
Quoted 3 lines

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

Interesting. That added this text:

* while gcc behavior gets along with a normal
* barrier(), llvm needs an explicit input variable to be assumed
* clobbered.

however:
#define barrier_data(ptr) __asm__ __volatile__("": :"r"(ptr) :"memory")

So no explicit variable is clobbered.
Weird isn't it?

Quoted 81 lines
Quoted 58 lines
---
 include/linux/compiler-clang.h | 5 ++---
 include/linux/compiler-gcc.h   | 4 ----
 include/linux/compiler-intel.h | 4 +---
 include/linux/compiler.h       | 4 +++-
 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
index 3e7dafb3ea80..7ddaeb5182e3 100644
@@ -3,9 +3,8 @@
 #error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."
 #endif
 
-/* Some compiler specific definitions are overwritten here
- * for Clang compiler
- */
+/* Compiler specific definitions for Clang compiler */
+
 #define uninitialized_var(x) x = *(&(x))
 
 /* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
index 2010493e1040..72054d9f0eaa 100644
@@ -58,10 +58,6 @@
        (typeof(ptr)) (__ptr + (off));                                  \
 })
 
-/* Make the optimizer believe the variable can be manipulated arbitrarily. */
-#define OPTIMIZER_HIDE_VAR(var)                                                \
-       __asm__ ("" : "=r" (var) : "0" (var))
-
 /*
  * A trick to suppress uninitialized variable warning without generating any
  * code
include/linux/compiler-intel.h +1 -3
index 517bd14e1222..b17f3cd18334 100644
@@ -5,9 +5,7 @@
 
 #ifdef __ECC
 
-/* Some compiler specific definitions are overwritten here
- * for Intel ECC compiler
- */
+/* Compiler specific definitions for Intel ECC compiler */
 
 #include <asm/intrinsics.h>
 
include/linux/compiler.h +0 -0
index 06396c1cf127..1ad367b4cd8d 100644

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158

Quoted 8 lines
include/linux/compiler.h +3 -1
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
-#define OPTIMIZER_HIDE_VAR(var) barrier()
+/* Make the optimizer believe the variable can be manipulated arbitrarily. */
+#define OPTIMIZER_HIDE_VAR(var)                                                \
+       __asm__ ("" : "=r" (var) : "0" (var))
 #endif

This should be fine, thanks for the cleanup! For now, we're not yet confident to turn on Clang's integrated assembler for the kernel, but I'll make sure to revisit this should we, in case Clang is then able to optimize this out.
+ Eric, who might know of a better trick for what we're trying to accomplish with this macro.

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Signature · 2 lines
Thanks,
~Nick Desaulniers
#5 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Nick Desaulniers
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Cheers,
Miguel

author maintainer
#6 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 14 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Quoted 8 lines

Sure, at least we get quickly some linux-next time.

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Cheers,
Miguel

Okay, I will try.

Signature · 1 line
MST
#7 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Hi Michael,

Quoted 7 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Okay, I will try.

Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the barrier_data later.

Cheers,
Miguel

author maintainer
#8 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 9 lines · Miguel Ojeda

Hi Michael,

Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Quoted 8 lines
Quoted 1 line

Okay, I will try.

Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the barrier_data later.

Cheers,
Miguel

No not yet. Sorry! Pls send this one in, barrier_data will likely miss the next merge window.

Signature · 1 line
MSR
#9 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 3 lines · Michael S. Tsirkin

No not yet. Sorry! Pls send this one in, barrier_data will likely miss the next merge window.

No worries! Done.

Cheers,
Miguel

author maintainer
#10 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Miguel Ojeda
Quoted 9 lines · Miguel Ojeda

Hi Michael,

Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Miguel Ojeda

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Quoted 8 lines
Quoted 1 line

Okay, I will try.

Did you have a chance to do it (or maybe I missed the patches)? If not, no worries, I can send this to Linus as it is and get it in already, then we can do the barrier_data later.

Cheers,
Miguel

No not yet. Sorry! Pls send this one in, barrier_data will likely miss the next merge window.

Signature · 1 line
MSR
author maintainer
#11 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Miguel Ojeda
Quoted 14 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Quoted 8 lines

Sure, at least we get quickly some linux-next time.

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Cheers,
Miguel

Okay, I will try.

Signature · 1 line
MST
author maintainer
#12 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Miguel Ojeda
Quoted 9 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 6 lines
Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.

Quoted 6 lines

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Cheers,
Miguel

#13 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.

It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.

regards,
dan carpenter

author maintainer
#14 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

Quoted 19 lines · Dan Carpenter
Quoted 18 lines · Michael S. Tsirkin
Quoted 15 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

Quoted 8 lines
Quoted 2 lines

BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.

It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.

regards,
dan carpenter

I misunderstood. Sure that makes sense.

Signature · 1 line
MST
#15 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Michael S. Tsirkin
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.

It doesn't hurt to put things in linux-next for a week and then 5.0 and -stable. Not a lot of testing happens on linux-next, but some does.

regards,
dan carpenter

author maintainer
#16 lore raw

Re: [PATCH RFC 1/4] include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Miguel Ojeda
Quoted 9 lines · Miguel Ojeda
Quoted 4 lines · Nick Desaulniers

Also for more context, see:
commit 7829fb09a2b4 ("lib: make memzero_explicit more robust against dead store elimination")

By the way, shouldn't that barrier_data() be directly in compiler.h too, since it is for both gcc & clang?

Quoted 6 lines
Quoted 4 lines

+ Miguel
Miguel, would you mind taking this into your compiler-attributes tree?

Sure, at least we get quickly some linux-next time.

BTW why linux-next? shouldn't this go into 5.0 and stable? It's a bugfix after all.

Quoted 6 lines

Note it would be nice to separate the patch into two (one for the comments, another for OPTIMIZER_HIDE_VAR), and also possibly another for barrier_data().

Cheers,
Miguel

author maintainer
#17 lore raw

RFC PATCH 1/4 include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR

In reply to Michael S. Tsirkin

Since commit 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive") clang no longer reuses the OPTIMIZER_HIDE_VAR macro from compiler-gcc - instead it gets the version in include/linux/compiler.h. Unfortunately that version doesn't actually prevent compiler from optimizing out the variable.

Fix up by moving the macro out from compiler-gcc.h to compiler.h. Compilers without incline asm support will keep working since it's protected by an ifdef.

Also fix up comments to match reality since we are no longer overriding any macros.

Build-tested with gcc and clang.

Fixes 815f0ddb346c ("include/linux/compiler*.h: make compiler-*.h mutually exclusive")
Cc Eli Friedman <[email protected]>
Cc Joe Perches <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Linus Torvalds <[email protected]>
---
 include/linux/compiler-clang.h | 5 ++---
 include/linux/compiler-gcc.h   | 4 ----
 include/linux/compiler-intel.h | 4 +---
 include/linux/compiler.h       | 4 +++-
 4 files changed, 6 insertions(+), 11 deletions(-)
include/linux/compiler-clang.h +2 -3
index 3e7dafb3ea80..7ddaeb5182e3 100644
@@ -3,9 +3,8 @@
 #error "Please don't include <linux/compiler-clang.h> directly, include <linux/compiler.h> instead."
 #endif
 
-/* Some compiler specific definitions are overwritten here
- * for Clang compiler
- */
+/* Compiler specific definitions for Clang compiler */
+
 #define uninitialized_var(x) x = *(&(x))
 
 /* same as gcc, this was present in clang-2.6 so we can assume it works
include/linux/compiler-gcc.h +0 -4
index 2010493e1040..72054d9f0eaa 100644
@@ -58,10 +58,6 @@
 	(typeof(ptr)) (__ptr + (off));					\
 })
 
-/* Make the optimizer believe the variable can be manipulated arbitrarily. */
-#define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
-
 /*
  * A trick to suppress uninitialized variable warning without generating any
  * code
include/linux/compiler-intel.h +1 -3
index 517bd14e1222..b17f3cd18334 100644
@@ -5,9 +5,7 @@
 
 #ifdef __ECC
 
-/* Some compiler specific definitions are overwritten here
- * for Intel ECC compiler
- */
+/* Compiler specific definitions for Intel ECC compiler */
 
 #include <asm/intrinsics.h>
 
include/linux/compiler.h +3 -1
index 06396c1cf127..1ad367b4cd8d 100644
@@ -152,7 +152,9 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
-#define OPTIMIZER_HIDE_VAR(var) barrier()
+/* Make the optimizer believe the variable can be manipulated arbitrarily. */
+#define OPTIMIZER_HIDE_VAR(var)						\
+	__asm__ ("" : "=r" (var) : "0" (var))
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
author maintainer
#18 lore raw

RFC PATCH 2/4 include/linux/compiler.h: allow memory operands

In reply to Michael S. Tsirkin

We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.

---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +1 -1
index 1ad367b4cd8d..6601d39e8c48 100644
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #ifndef OPTIMIZER_HIDE_VAR
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
+	__asm__ ("" : "=rm" (var) : "0" (var))
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
#19 lore raw

Re: [PATCH RFC 2/4] include/linux/compiler.h: allow memory operands

Quoted 13 lines · Michael S. Tsirkin

We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.

---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +0 -0
index 1ad367b4cd8d..6601d39e8c48 100644

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158

Quoted 6 lines
include/linux/compiler.h +1 -1
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #ifndef OPTIMIZER_HIDE_VAR
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
+	__asm__ ("" : "=rm" (var) : "0" (var))
 #endif

I think this can break for architectures with write-back addressing modes such as arm, where the "m" constraint is assumed to be evaluated precisely once in the asm block.

Will

author maintainer
#20 lore raw

Re: [PATCH RFC 2/4] include/linux/compiler.h: allow memory operands

Quoted 25 lines · Will Deacon
Quoted 13 lines · Michael S. Tsirkin

We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.

---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +0 -0
index 1ad367b4cd8d..6601d39e8c48 100644

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 158

Quoted 6 lines
include/linux/compiler.h +1 -1
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #ifndef OPTIMIZER_HIDE_VAR
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
+	__asm__ ("" : "=rm" (var) : "0" (var))
 #endif

I think this can break for architectures with write-back addressing modes such as arm, where the "m" constraint is assumed to be evaluated precisely once in the asm block.

Quoted 1 line

Will

Thanks, I'll drop this patch.

author maintainer
#21 lore raw

RFC PATCH 2/4 include/linux/compiler.h: allow memory operands

In reply to Michael S. Tsirkin

We don't really care whether the variable is in-register or in-memory. Relax the constraint accordingly.

---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +1 -1
index 1ad367b4cd8d..6601d39e8c48 100644
@@ -154,7 +154,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #ifndef OPTIMIZER_HIDE_VAR
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
-	__asm__ ("" : "=r" (var) : "0" (var))
+	__asm__ ("" : "=rm" (var) : "0" (var))
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
author maintainer
#22 lore raw

RFC PATCH 3/4 barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin

It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.

However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.

This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.

---
 Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
 arch/alpha/include/asm/barrier.h  |  1 +
 include/asm-generic/barrier.h     | 18 ++++++++++++++++++
 include/linux/compiler.h          |  4 ++++
 4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
index c1d913944ad8..9dbaa2e1dbf6 100644
@@ -691,6 +691,18 @@ case what's actually required is:
 		p = READ_ONCE(b);
 	}
 
+Alternatively, a control dependency can be converted to a data dependency,
+e.g.:
+
+	q = READ_ONCE(a);
+	if (q) {
+		b = dependent_ptr_mb(b, q);
+		p = READ_ONCE(b);
+	}
+
+Note how the result of dependent_ptr_mb must be used with the following
+accesses in order to have an effect.
+
 However, stores are not speculated.  This means that ordering -is- provided
 for load-store control dependencies, as in the following example:
 
@@ -836,6 +848,12 @@ out-guess your code.  More generally, although READ_ONCE() does force
 the compiler to actually emit code for a given load, it does not force
 the compiler to use the results.
 
+Converting to a data dependency helps with this too:
+
+	q = READ_ONCE(a);
+	b = dependent_ptr_mb(b, q);
+	WRITE_ONCE(b, 1);
+
 In addition, control dependencies apply only to the then-clause and
 else-clause of the if-statement in question.  In particular, it does
 not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
index 92ec486a4f9e..b4934e8c551b 100644
@@ -59,6 +59,7 @@
  * as Alpha, "y" could be set to 3 and "x" to 0.  Use rmb()
  * in cases like this where there are no data dependencies.
  */
+#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
 #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
 
 #ifdef CONFIG_SMP
include/asm-generic/barrier.h +18 -0
index 2cafdbb9ae4c..fa2e2ef72b68 100644
@@ -70,6 +70,24 @@
 #define __smp_read_barrier_depends()	read_barrier_depends()
 #endif
 
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })
+
+#endif
+
 #ifdef CONFIG_SMP
 
 #ifndef smp_mb
include/linux/compiler.h +4 -0
index 6601d39e8c48..f599c30f1b28 100644
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
+
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
 	__asm__ ("" : "=rm" (var) : "0" (var))
+
+#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
+
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
#23 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900

Quoted 9 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +2 -0
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.

Was this hunk intentional?

author maintainer
#24 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +0 -0
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
Quoted 7 lines

Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900

Quoted 5 lines
Documentation/memory-barriers.txt +2 -0
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.

Was this hunk intentional?

Nope, thanks for catching this.

Signature · 1 line
MST
author maintainer
#25 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Matthew Wilcox
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +0 -0
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
Quoted 7 lines

Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900

Quoted 5 lines
Documentation/memory-barriers.txt +2 -0
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.

Was this hunk intentional?

Nope, thanks for catching this.

Signature · 1 line
MST
#26 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin

Documentation/memory-barriers.txt to the CPU containing it. See the section on "Multicopy atomicity" · line 900

Quoted 9 lines · Michael S. Tsirkin
Documentation/memory-barriers.txt +2 -0
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.

Was this hunk intentional?

#27 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin
Quoted 88 lines · Michael S. Tsirkin

It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.

However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.

This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.

---
  Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
  arch/alpha/include/asm/barrier.h  |  1 +
  include/asm-generic/barrier.h     | 18 ++++++++++++++++++
  include/linux/compiler.h          |  4 ++++
  4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
index c1d913944ad8..9dbaa2e1dbf6 100644
@@ -691,6 +691,18 @@ case what's actually required is:
  		p = READ_ONCE(b);
  	}
 
+Alternatively, a control dependency can be converted to a data dependency,
+e.g.:
+
+	q = READ_ONCE(a);
+	if (q) {
+		b = dependent_ptr_mb(b, q);
+		p = READ_ONCE(b);
+	}
+
+Note how the result of dependent_ptr_mb must be used with the following
+accesses in order to have an effect.
+
  However, stores are not speculated.  This means that ordering -is- provided
  for load-store control dependencies, as in the following example:
 
@@ -836,6 +848,12 @@ out-guess your code.  More generally, although READ_ONCE() does force
  the compiler to actually emit code for a given load, it does not force
  the compiler to use the results.
 
+Converting to a data dependency helps with this too:
+
+	q = READ_ONCE(a);
+	b = dependent_ptr_mb(b, q);
+	WRITE_ONCE(b, 1);
+
  In addition, control dependencies apply only to the then-clause and
  else-clause of the if-statement in question.  In particular, it does
  not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
  for more information.
 
 
+
+
  In summary:
 
    (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
index 92ec486a4f9e..b4934e8c551b 100644
@@ -59,6 +59,7 @@
   * as Alpha, "y" could be set to 3 and "x" to 0.  Use rmb()
   * in cases like this where there are no data dependencies.
   */
+#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
  #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
 
  #ifdef CONFIG_SMP
include/asm-generic/barrier.h +5 -0
index 2cafdbb9ae4c..fa2e2ef72b68 100644
@@ -70,6 +70,24 @@
  #define __smp_read_barrier_depends()	read_barrier_depends()
  #endif
 
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Thanks

Quoted 14 lines
include/asm-generic/barrier.h +3 -0
+
+#endif
+
  #ifdef CONFIG_SMP
 
  #ifndef smp_mb
include/linux/compiler.h +0 -0
index 6601d39e8c48..f599c30f1b28 100644
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
  #endif
 
  #ifndef OPTIMIZER_HIDE_VAR

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 164

Quoted 10 lines
include/linux/compiler.h +4 -0
+
  /* Make the optimizer believe the variable can be manipulated arbitrarily. */
  #define OPTIMIZER_HIDE_VAR(var)						\
  	__asm__ ("" : "=rm" (var) : "0" (var))
+
+#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
+
  #endif
 
  /* Not-quite-unique ID. */

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#28 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 97 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin

It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.

However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.

This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.

---
  Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
  arch/alpha/include/asm/barrier.h  |  1 +
  include/asm-generic/barrier.h     | 18 ++++++++++++++++++
  include/linux/compiler.h          |  4 ++++
  4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
index c1d913944ad8..9dbaa2e1dbf6 100644
@@ -691,6 +691,18 @@ case what's actually required is:
  		p = READ_ONCE(b);
  	}
+Alternatively, a control dependency can be converted to a data dependency,
+e.g.:
+
+	q = READ_ONCE(a);
+	if (q) {
+		b = dependent_ptr_mb(b, q);
+		p = READ_ONCE(b);
+	}
+
+Note how the result of dependent_ptr_mb must be used with the following
+accesses in order to have an effect.
+
  However, stores are not speculated.  This means that ordering -is- provided
  for load-store control dependencies, as in the following example:
@@ -836,6 +848,12 @@ out-guess your code.  More generally, although READ_ONCE() does force
  the compiler to actually emit code for a given load, it does not force
  the compiler to use the results.
+Converting to a data dependency helps with this too:
+
+	q = READ_ONCE(a);
+	b = dependent_ptr_mb(b, q);
+	WRITE_ONCE(b, 1);
+
  In addition, control dependencies apply only to the then-clause and
  else-clause of the if-statement in question.  In particular, it does
  not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
  for more information.
+
+
  In summary:
    (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
index 92ec486a4f9e..b4934e8c551b 100644
@@ -59,6 +59,7 @@
   * as Alpha, "y" could be set to 3 and "x" to 0.  Use rmb()
   * in cases like this where there are no data dependencies.
   */
+#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
  #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
  #ifdef CONFIG_SMP
include/asm-generic/barrier.h +5 -0
index 2cafdbb9ae4c..fa2e2ef72b68 100644
@@ -70,6 +70,24 @@
  #define __smp_read_barrier_depends()	read_barrier_depends()
  #endif
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 86

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

mb variant is unused right now so I'll remove it.

Quoted 14 lines
Quoted 14 lines
include/asm-generic/barrier.h +3 -0
+
+#endif
+
  #ifdef CONFIG_SMP
  #ifndef smp_mb
include/linux/compiler.h +1 -0
index 6601d39e8c48..f599c30f1b28 100644
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
  #endif
  #ifndef OPTIMIZER_HIDE_VAR
+
Quoted 8 lines

include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 164

Quoted 8 lines
include/linux/compiler.h +3 -0
  /* Make the optimizer believe the variable can be manipulated arbitrarily. */
  #define OPTIMIZER_HIDE_VAR(var)						\
  	__asm__ ("" : "=rm" (var) : "0" (var))
+
+#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
+
  #endif
  /* Not-quite-unique ID. */

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#29 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 95 lines · Michael S. Tsirkin
Quoted 94 lines · Jason Wang
Quoted 80 lines · Michael S. Tsirkin

It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.

However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.

This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.

---
   Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
   arch/alpha/include/asm/barrier.h  |  1 +
   include/asm-generic/barrier.h     | 18 ++++++++++++++++++
   include/linux/compiler.h          |  4 ++++
   4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
index c1d913944ad8..9dbaa2e1dbf6 100644
@@ -691,6 +691,18 @@ case what's actually required is:
   		p = READ_ONCE(b);
   	}
+Alternatively, a control dependency can be converted to a data dependency,
+e.g.:
+
+	q = READ_ONCE(a);
+	if (q) {
+		b = dependent_ptr_mb(b, q);
+		p = READ_ONCE(b);
+	}
+
+Note how the result of dependent_ptr_mb must be used with the following
+accesses in order to have an effect.
+
   However, stores are not speculated.  This means that ordering -is- provided
   for load-store control dependencies, as in the following example:
@@ -836,6 +848,12 @@ out-guess your code.  More generally, although READ_ONCE() does force
   the compiler to actually emit code for a given load, it does not force
   the compiler to use the results.
+Converting to a data dependency helps with this too:
+
+	q = READ_ONCE(a);
+	b = dependent_ptr_mb(b, q);
+	WRITE_ONCE(b, 1);
+
   In addition, control dependencies apply only to the then-clause and
   else-clause of the if-statement in question.  In particular, it does
   not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
   for more information.
+
+
   In summary:
     (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
index 92ec486a4f9e..b4934e8c551b 100644
@@ -59,6 +59,7 @@
    * as Alpha, "y" could be set to 3 and "x" to 0.  Use rmb()
    * in cases like this where there are no data dependencies.
    */
+#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
   #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
   #ifdef CONFIG_SMP
include/asm-generic/barrier.h +5 -0
index 2cafdbb9ae4c..fa2e2ef72b68 100644
@@ -70,6 +70,24 @@
   #define __smp_read_barrier_depends()	read_barrier_depends()
   #endif
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 86

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 8 lines
Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

mb variant is unused right now so I'll remove it.

Yes.

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

maintainer
#30 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 7 lines
Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#31 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

Quoted 6 lines
Quoted 1 line

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Thanks,

Signature · 5 lines
MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
maintainer
#32 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

Quoted 8 lines

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.

Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Not sure; what is the actual overhead of the construct on x86 vs the NOP?

author maintainer
#33 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

Quoted 8 lines

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 2 lines

The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.

But not on intel right? On intel loads are ordered so it can be a nop.

Quoted 8 lines
Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Not sure; what is the actual overhead of the construct on x86 vs the NOP?

I'll have to check. There's a pipeline stall almost for sure - that's why we put it there after all :).

Signature · 1 line
MST
author maintainer
#34 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Peter Zijlstra
Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines · Peter Zijlstra

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

Quoted 8 lines

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 2 lines

The latter; the construct simply generates dependent loads. It is up to the CPU as to what all that works for.

But not on intel right? On intel loads are ordered so it can be a nop.

Quoted 8 lines
Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Not sure; what is the actual overhead of the construct on x86 vs the NOP?

I'll have to check. There's a pipeline stall almost for sure - that's why we put it there after all :).

Signature · 1 line
MST
#35 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

Quoted 6 lines
Quoted 1 line

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?

Thanx, Paul

author maintainer
#36 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

Quoted 6 lines
Quoted 1 line

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Quoted 5 lines

But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?

Thanx, Paul

Oh you are right. Stores are not reordered with older loads on x86.

So point 2 is moot. Sorry about the noise.

I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an mb() there, seeing as it doesn't even have spectre protection enabled. Slow but safe.

Signature · 1 line
MST
#37 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

Quoted 61 lines · Michael S. Tsirkin
Quoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
(quoted diff) +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

Review

Quoted 10 lines
(quoted diff) +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

Quoted 6 lines
Quoted 1 line

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Quoted 5 lines

But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?

Thanx, Paul

Quoted 7 lines

Oh you are right. Stores are not reordered with older loads on x86.

So point 2 is moot. Sorry about the noise.

I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an mb() there, seeing as it doesn't even have spectre protection enabled. Slow but safe.

Well, there is a mention of DMA above, which on some systems throws in a wild card. I would certainly hope that DMA would integrate nicely with the cache-coherence protocols these days, unlike 25 years ago, but who knows?

Thanx, Paul

author maintainer
#38 lore raw

Re: [PATCH RFC 3/4] barriers: convert a control to a data dependency

In reply to Paul E. McKenney
Quoted 54 lines · Paul E. McKenney
Quoted 47 lines · Michael S. Tsirkin
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 5 lines · Michael S. Tsirkin
include/asm-generic/barrier.h +5 -0
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\

include/asm-generic/barrier.h line 87

Quoted 10 lines
include/asm-generic/barrier.h +10 -0
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })

So for the example of patch 4, we'd better fall back to rmb() or need a dependent_ptr_rmb()?

Quoted 1 line

Thanks

You mean for strongly ordered architectures like Intel? Yes, maybe it makes sense to have dependent_ptr_smp_rmb, dependent_ptr_dma_rmb and dependent_ptr_virt_rmb.

Quoted 6 lines
Quoted 1 line

mb variant is unused right now so I'll remove it.

How about naming the thing: dependent_ptr() ? That is without any (r)mb implications at all. The address dependency is strictly weaker than an rmb in that it will only order the two loads in qestion and not, like rmb, any prior to any later load.

So I'm fine with this as it's enough for virtio, but I would like to point out two things:

1. E.g. on x86 both SMP and DMA variants can be NOPs but the madatory one can't, so assuming we do not want it to be stronger than rmp then either we want smp_dependent_ptr(), dma_dependent_ptr(), dependent_ptr() or we just will specify that dependent_ptr() works for both DMA and SMP.

Quoted 5 lines

2. Down the road, someone might want to order a store after a load. Address dependency does that for us too. Assuming we make dependent_ptr a NOP on x86, we will want an mb variant which isn't a NOP on x86. Will we want to rename dependent_ptr to dependent_ptr_rmb at that point?

Quoted 5 lines

But x86 preserves store-after-load orderings anyway, and even Alpha respects ordering from loads to dependent stores. So what am I missing here?

Thanx, Paul

Oh you are right. Stores are not reordered with older loads on x86.

So point 2 is moot. Sorry about the noise.

I guess at this point the only sticking point is the ECC compiler. I'm inclined to stick an mb() there, seeing as it doesn't even have spectre protection enabled. Slow but safe.

Signature · 5 lines
MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
author maintainer
#39 lore raw

RFC PATCH 3/4 barriers: convert a control to a data dependency

In reply to Michael S. Tsirkin

It's not uncommon to have two access two unrelated memory locations in a specific order. At the moment one has to use a memory barrier for this.

However, if the first access was a read and the second used an address depending on the first one we would have a data dependency and no barrier would be necessary.

This adds a new interface: dependent_ptr_mb which does exactly this: it returns a pointer with a data dependency on the supplied value.

---
 Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
 arch/alpha/include/asm/barrier.h  |  1 +
 include/asm-generic/barrier.h     | 18 ++++++++++++++++++
 include/linux/compiler.h          |  4 ++++
 4 files changed, 43 insertions(+)
Documentation/memory-barriers.txt +20 -0
index c1d913944ad8..9dbaa2e1dbf6 100644
@@ -691,6 +691,18 @@ case what's actually required is:
 		p = READ_ONCE(b);
 	}
 
+Alternatively, a control dependency can be converted to a data dependency,
+e.g.:
+
+	q = READ_ONCE(a);
+	if (q) {
+		b = dependent_ptr_mb(b, q);
+		p = READ_ONCE(b);
+	}
+
+Note how the result of dependent_ptr_mb must be used with the following
+accesses in order to have an effect.
+
 However, stores are not speculated.  This means that ordering -is- provided
 for load-store control dependencies, as in the following example:
 
@@ -836,6 +848,12 @@ out-guess your code.  More generally, although READ_ONCE() does force
 the compiler to actually emit code for a given load, it does not force
 the compiler to use the results.
 
+Converting to a data dependency helps with this too:
+
+	q = READ_ONCE(a);
+	b = dependent_ptr_mb(b, q);
+	WRITE_ONCE(b, 1);
+
 In addition, control dependencies apply only to the then-clause and
 else-clause of the if-statement in question.  In particular, it does
 not necessarily apply to code following the if-statement:
@@ -875,6 +893,8 @@ to the CPU containing it.  See the section on "Multicopy atomicity"
 for more information.
 
 
+
+
 In summary:
 
   (*) Control dependencies can order prior loads against later stores.
arch/alpha/include/asm/barrier.h +1 -0
index 92ec486a4f9e..b4934e8c551b 100644
@@ -59,6 +59,7 @@
  * as Alpha, "y" could be set to 3 and "x" to 0.  Use rmb()
  * in cases like this where there are no data dependencies.
  */
+#define ARCH_NEEDS_READ_BARRIER_DEPENDS 1
 #define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
 
 #ifdef CONFIG_SMP
include/asm-generic/barrier.h +18 -0
index 2cafdbb9ae4c..fa2e2ef72b68 100644
@@ -70,6 +70,24 @@
 #define __smp_read_barrier_depends()	read_barrier_depends()
 #endif
 
+#if defined(COMPILER_HAS_OPTIMIZER_HIDE_VAR) && \
+	!defined(ARCH_NEEDS_READ_BARRIER_DEPENDS)
+
+#define dependent_ptr_mb(ptr, val) ({					\
+	long dependent_ptr_mb_val = (long)(val);			\
+	long dependent_ptr_mb_ptr = (long)(ptr) - dependent_ptr_mb_val;	\
+									\
+	BUILD_BUG_ON(sizeof(val) > sizeof(long));			\
+	OPTIMIZER_HIDE_VAR(dependent_ptr_mb_val);			\
+	(typeof(ptr))(dependent_ptr_mb_ptr + dependent_ptr_mb_val);	\
+})
+
+#else
+
+#define dependent_ptr_mb(ptr, val) ({ mb(); (ptr); })
+
+#endif
+
 #ifdef CONFIG_SMP
 
 #ifndef smp_mb
include/linux/compiler.h +4 -0
index 6601d39e8c48..f599c30f1b28 100644
@@ -152,9 +152,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 #endif
 
 #ifndef OPTIMIZER_HIDE_VAR
+
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var)						\
 	__asm__ ("" : "=rm" (var) : "0" (var))
+
+#define COMPILER_HAS_OPTIMIZER_HIDE_VAR 1
+
 #endif
 
 /* Not-quite-unique ID. */
Signature · 1 line
MST
author maintainer
#40 lore raw

RFC PATCH 4/4 virtio: use dependent_ptr_mb

In reply to Michael S. Tsirkin

Use dependent_ptr_mb which is - on some architectures - more light-weight than an rmb.

---
 drivers/virtio/virtio_ring.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +4 -2
index 814b395007b2..2d320396eff8 100644
@@ -702,6 +702,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
 	void *ret;
 	unsigned int i;
 	u16 last_used;
+	bool more;
 
 	START_USE(vq);
 
@@ -710,14 +711,15 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
 		return NULL;
 	}
 
-	if (!more_used(vq)) {
+	more = more_used(vq);
+	if (!more) {
 		pr_debug("No more buffers in queue\n");
 		END_USE(vq);
 		return NULL;
 	}
 
 	/* Only get used array entries after they have been exposed by host. */
-	virtio_rmb(vq->weak_barriers);
+	vq = dependent_ptr_mb(vq, more);
 
 	last_used = (vq->last_used_idx & (vq->vring.num - 1));
 	i = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].id);
Signature · 1 line
MST
author maintainer
#41 lore raw

RFC PATCH 4/4 virtio: use dependent_ptr_mb

In reply to Michael S. Tsirkin

Use dependent_ptr_mb which is - on some architectures - more light-weight than an rmb.

---
 drivers/virtio/virtio_ring.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +4 -2
index 814b395007b2..2d320396eff8 100644
@@ -702,6 +702,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
 	void *ret;
 	unsigned int i;
 	u16 last_used;
+	bool more;
 
 	START_USE(vq);
 
@@ -710,14 +711,15 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
 		return NULL;
 	}
 
-	if (!more_used(vq)) {
+	more = more_used(vq);
+	if (!more) {
 		pr_debug("No more buffers in queue\n");
 		END_USE(vq);
 		return NULL;
 	}
 
 	/* Only get used array entries after they have been exposed by host. */
-	virtio_rmb(vq->weak_barriers);
+	vq = dependent_ptr_mb(vq, more);
 
 	last_used = (vq->last_used_idx & (vq->vring.num - 1));
 	i = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].id);
Signature · 1 line
MST
maintainer
#42 lore raw
In reply to Michael S. Tsirkin
Quoted 5 lines · Michael S. Tsirkin

So as explained in Documentation/memory-barriers.txt e.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.

Quoted 7 lines

Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.

This adds an API for this and uses it in virtio.

Written over the holiday and build tested only so far.

You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However, tools/memory-model/* uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?

Quoted 1 line

This patchset is also suboptimal on e.g. x86 where e.g. smp_rmb is a nop.

This should be easy to fix with an architecture-specific override.

Alan Stern

Quoted 17 lines

Sending out for early feedback/flames.

Michael S. Tsirkin (4):
include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR include/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: use dependent_ptr_mb

 Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
 arch/alpha/include/asm/barrier.h  |  1 +
 drivers/virtio/virtio_ring.c      |  6 ++++--
 include/asm-generic/barrier.h     | 18 ++++++++++++++++++
 include/linux/compiler-clang.h    |  5 ++---
 include/linux/compiler-gcc.h      |  4 ----
 include/linux/compiler-intel.h    |  4 +---
 include/linux/compiler.h          |  8 +++++++-
 8 files changed, 53 insertions(+), 13 deletions(-)
author maintainer
#43 lore raw
Quoted 13 lines · Alan Stern
Quoted 11 lines · Michael S. Tsirkin

So as explained in Documentation/memory-barriers.txt e.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.

Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.

This adds an API for this and uses it in virtio.

Quoted 7 lines
Quoted 1 line

Written over the holiday and build tested only so far.

You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However, tools/memory-model/* uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?

Sure, sounds good. While I'm at it, should memory-barriers.txt be switched over too?

Quoted 5 lines
Quoted 1 line

This patchset is also suboptimal on e.g. x86 where e.g. smp_rmb is a nop.

This should be easy to fix with an architecture-specific override.

Alan Stern

Absolutely. It does however mean that we'll need several
variants: mb/rmb, smp/dma/virt/mandatory.

I am still trying to decide whether it's good since it documents the kind of barrier that we are trying to use - or bad since it's more verbose and makes you choose one where they are all pretty cheap.

Quoted 17 lines
Quoted 17 lines

Sending out for early feedback/flames.

Michael S. Tsirkin (4):
include/linux/compiler*.h: fix OPTIMIZER_HIDE_VAR include/linux/compiler.h: allow memory operands
barriers: convert a control to a data dependency
virtio: use dependent_ptr_mb

 Documentation/memory-barriers.txt | 20 ++++++++++++++++++++
 arch/alpha/include/asm/barrier.h  |  1 +
 drivers/virtio/virtio_ring.c      |  6 ++++--
 include/asm-generic/barrier.h     | 18 ++++++++++++++++++
 include/linux/compiler-clang.h    |  5 ++---
 include/linux/compiler-gcc.h      |  4 ----
 include/linux/compiler-intel.h    |  4 +---
 include/linux/compiler.h          |  8 +++++++-
 8 files changed, 53 insertions(+), 13 deletions(-)
maintainer
#44 lore raw
Quoted 16 lines · Michael S. Tsirkin
Quoted 15 lines · Alan Stern
Quoted 12 lines · Michael S. Tsirkin

So as explained in Documentation/memory-barriers.txt e.g. a load followed by a store require a full memory barrier, to avoid store being ordered before the load. Similarly load-load requires a read memory barrier.

Thinking about it, we can actually create a data dependency by mixing the first loaded value into the pointer being accessed.

This adds an API for this and uses it in virtio.

Written over the holiday and build tested only so far.

Quoted 8 lines
Quoted 5 lines

You are using the terminology from memory-barriers.txt, referring to the new dependency you create as a data dependency. However, tools/memory-model/* uses a more precise name, calling it an address dependency. Could you change the comments in the patches to use this name instead?

Sure, sounds good. While I'm at it, should memory-barriers.txt be switched over too?

If you want to take care of that, great! I never seem to get around to doing it.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

This patchset is also suboptimal on e.g. x86 where e.g. smp_rmb is a nop.

This should be easy to fix with an architecture-specific override.

Quoted 8 lines
Quoted 1 line

Alan Stern

Absolutely. It does however mean that we'll need several
variants: mb/rmb, smp/dma/virt/mandatory.

I am still trying to decide whether it's good since it documents the kind of barrier that we are trying to use - or bad since it's more verbose and makes you choose one where they are all pretty cheap.

How many places can these things be used? My guess is not very many, or at least, there aren't very many different _types_ of usage. So start only with variants you know will be used. More can be added later if we want.

Alan Stern