Skip to contentSkip to conversation

rust: more memory barriers bindings

PATCH In review Gary Guo 4 participants 16 messages Apr 2 – Apr 13

Subsystems (8)
lore
Share & tools
lore

Gary Guo

author
#0 lore raw

cover rust: more memory barriers bindings

From: Gary Guo <[email protected]>

This expands the existing smp barriers to also mandatory barriers and DMA barriers.

The API looks like:
mb(Ordering)/smp_mb(Ordering)/dma_mb(Ordering), where Ordering is one of Full, Read, Write and also Acquire, Release.

The Acquire and Release barriers are mapped to Full barriers for now and they only serve the purpose of documenting what ordering is needed without codegen optimizations, but they could be improved later to produce better codegen compared to full barriers. More on them in the commit message of patch 2.

A user of these introduced API is included in patch 3, which is a concurrency bug that exists in Nova code today due to missing barriers.

To: Miguel Ojeda <[email protected]>
To: Boqun Feng <[email protected]>
To: Danilo Krummrich <[email protected]>
To: Alexandre Courbot <[email protected]>
Cc: Björn Roy Baron <[email protected]>
Cc: Benno Lossin <[email protected]>
Cc: Andreas Hindborg <[email protected]>
Cc: Alice Ryhl <[email protected]>
Cc: Trevor Gross <[email protected]>
Cc: Alan Stern <[email protected]>
Cc: Andrea Parri <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: David Howells <[email protected]>
Cc: Jade Alglave <[email protected]>
Cc: Luc Maranget <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Akira Yokosawa <[email protected]>
Cc: Daniel Lustig <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]

Gary Guo (3):
rust: sync: add helpers for mb, dma_mb and friends
rust: sync: generic memory barriers
gpu: nova-core: fix wrong use of barriers in GSP code

 drivers/gpu/nova-core/gsp/cmdq.rs   |  19 +++
 drivers/gpu/nova-core/gsp/fw.rs     |  12 --
 rust/helpers/barrier.c              |  30 +++++
 rust/kernel/sync/atomic/ordering.rs |   2 +-
 rust/kernel/sync/barrier.rs         | 194 ++++++++++++++++++++++++----
 5 files changed, 217 insertions(+), 40 deletions(-)

base-commit: 36ece9697e89016181e5ae87510e40fb31d86f2b

Signature · 1 line
2.51.2
author
#1 lore raw

PATCH 1/3 rust: sync: add helpers for mb, dma_mb and friends

From: Gary Guo <[email protected]>

They supplement the existing smp_mb, smp_rmb and smp_wmb.

---
 rust/helpers/barrier.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
rust/helpers/barrier.c +30 -0
index fed8853745c8..dbc7a3017c78 100644
@@ -2,6 +2,36 @@
 
 #include <asm/barrier.h>
 
+__rust_helper void rust_helper_mb(void)
+{
+	mb();
+}
+
+__rust_helper void rust_helper_rmb(void)
+{
+	rmb();
+}
+
+__rust_helper void rust_helper_wmb(void)
+{
+	wmb();
+}
+
+__rust_helper void rust_helper_dma_mb(void)
+{
+	dma_mb();
+}
+
+__rust_helper void rust_helper_dma_rmb(void)
+{
+	dma_rmb();
+}
+
+__rust_helper void rust_helper_dma_wmb(void)
+{
+	dma_wmb();
+}
+
 __rust_helper void rust_helper_smp_mb(void)
 {
 	smp_mb();
Signature · 1 line
2.51.2
author
#3 lore raw

PATCH 2/3 rust: sync: generic memory barriers

In reply to Gary Guo

From: Gary Guo <[email protected]>

Implement a generic interface for memory barriers (full system/DMA/SMP). The interface uses a parameter to force user to specify their intent with barriers.

It provides Read, Write, Full orderings which map to the existing rmb(), wmb() and mb(), but also Acquire and Release which is documented to have LOAD->{LOAD,STORE} ordering and {LOAD,STORE}->WRITE ordering, although for now they're still mapped to a full mb(). But in the future it could be mapped to a more efficient form depending on the architecture. I included them as many users do not need the STORE->LOAD ordering, and having them use Acquire/Release is more clear on their intent in what reordering is to be prevented.

Generic is used here instead of providing individual standalone functions to reduce code duplication. For example, the Acquire -> Full upgrade here is uniformly implemented for all three types. The CONFIG_SMP check in smp_mb is uniformly implemented for all SMP barriers. This could extend to virt_mb's if they're introduced in the future.

---
 rust/kernel/sync/atomic/ordering.rs |   2 +-
 rust/kernel/sync/barrier.rs         | 194 ++++++++++++++++++++++++----
 2 files changed, 168 insertions(+), 28 deletions(-)
rust/kernel/sync/atomic/ordering.rs +1 -1
index 3f103aa8db99..c4e732e7212f 100644
@@ -15,7 +15,7 @@
 //!   - It provides ordering between the annotated operation and all the following memory accesses.
 //!   - It provides ordering between all the preceding memory accesses and all the following memory
 //!     accesses.
-//!   - All the orderings are the same strength as a full memory barrier (i.e. `smp_mb()`).
+//!   - All the orderings are the same strength as a full memory barrier (i.e. `smp_mb(Full)`).
 //! - [`Relaxed`] provides no ordering except the dependency orderings. Dependency orderings are
 //!   described in "DEPENDENCY RELATIONS" in [`LKMM`]'s [`explanation`].
 //!
rust/kernel/sync/barrier.rs +167 -27

216 lines not on this page · raw message

Signature · 1 line
2.51.2
#4 lore raw

Re: [PATCH 2/3] rust: sync: generic memory barriers

Hi Gary,

Quoted 21 lines · Gary Guo

From: Gary Guo <[email protected]>

Implement a generic interface for memory barriers (full system/DMA/SMP). The interface uses a parameter to force user to specify their intent with barriers.

It provides Read, Write, Full orderings which map to the existing rmb(), wmb() and mb(), but also Acquire and Release which is documented to have LOAD->{LOAD,STORE} ordering and {LOAD,STORE}->WRITE ordering, although for now they're still mapped to a full mb(). But in the future it could be mapped to a more efficient form depending on the architecture. I included them as many users do not need the STORE->LOAD ordering, and having them use Acquire/Release is more clear on their intent in what reordering is to be prevented.

Generic is used here instead of providing individual standalone functions to reduce code duplication. For example, the Acquire -> Full upgrade here is uniformly implemented for all three types. The CONFIG_SMP check in smp_mb is uniformly implemented for all SMP barriers. This could extend to virt_mb's if they're introduced in the future.

Quoted 4 lines
---
 rust/kernel/sync/atomic/ordering.rs |   2 +-
 rust/kernel/sync/barrier.rs         | 194 ++++++++++++++++++++++++----

IMO this patch should be split up into different patches for CPU vs IO, and perhaps even more patches separating out different barrier types.

rust/kernel/sync/atomic/ordering.rs

Quoted 6 lines
 2 files changed, 168 insertions(+), 28 deletions(-)
rust/kernel/sync/atomic/ordering.rs +0 -0
index 3f103aa8db99..c4e732e7212f 100644

[...]> +// Currently kernel only support rmb, wmb and full mb.

Quoted 33 lines
rust/kernel/sync/atomic/ordering.rs +20 -7
+impl MemoryBarrier<Smp> for Read {
+    #[inline]
+    fn run() {
+        // SAFETY: `smp_rmb()` is safe to call.
+        unsafe { bindings::smp_rmb() };
+    }
+}
+
+impl MemoryBarrier<Smp> for Write {
+    #[inline]
+    fn run() {
         // SAFETY: `smp_wmb()` is safe to call.
         unsafe { bindings::smp_wmb() };
-    } else {
-        barrier();
     }
 }
 
-/// A read-read memory barrier.
+impl MemoryBarrier<Smp> for Full {
+    #[inline]
+    fn run() {
+        // SAFETY: `smp_mb()` is safe to call.
+        unsafe { bindings::smp_mb() };
+    }
+}
+
+/// Memory barrier.
 ///
-/// A barrier that prevents compiler and CPU from reordering memory read accesses across the
-/// barrier.
-#[inline(always)]
-pub fn smp_rmb() {

rust/kernel/sync/atomic/ordering.rs

Quoted 10 lines
rust/kernel/sync/atomic/ordering.rs +10 -0
+/// A barrier that prevents compiler and CPU from reordering memory accesses across the barrier.
+///
+/// The specific forms of reordering can be specified using the parameter.
+/// - `mb(Read)` provides a read-read barrier.
+/// - `mb(Write)` provides a write-write barrier.
+/// - `mb(Full)` provides a full barrier.
+/// - `mb(Acquire)` prevents preceding read from being ordered against succeeding memory
+///    operations.
+/// - `mb(Release)` prevents preceding memory operations from being ordered against succeeding
+///    writes.

I don't agree with this definition of Release. Release is always associated with a specific store, likewise acquire with a load. The definition above also doesn't make sense 'prevents preceding memory operations from being ordered against succeeding writes', that's not what Release semantics are. Release orders memory operations with a specific memory operation associated with Release. Same for Acquire.

See also in Documentation/memory-barriers.txt, ACQUIRE and RELEASE are defined as being tied to specific memory operations.

Or am I missing something subtle?

thanks,

Signature · 1 line
Joel Fernandes
maintainer
#5 lore raw

Re: [PATCH 2/3] rust: sync: generic memory barriers

Quoted 24 lines · Joel Fernandes

Hi Gary,

Quoted 21 lines · Gary Guo

From: Gary Guo <[email protected]>

Implement a generic interface for memory barriers (full system/DMA/SMP). The interface uses a parameter to force user to specify their intent with barriers.

It provides Read, Write, Full orderings which map to the existing rmb(), wmb() and mb(), but also Acquire and Release which is documented to have LOAD->{LOAD,STORE} ordering and {LOAD,STORE}->WRITE ordering, although for now they're still mapped to a full mb(). But in the future it could be mapped to a more efficient form depending on the architecture. I included them as many users do not need the STORE->LOAD ordering, and having them use Acquire/Release is more clear on their intent in what reordering is to be prevented.

Generic is used here instead of providing individual standalone functions to reduce code duplication. For example, the Acquire -> Full upgrade here is uniformly implemented for all three types. The CONFIG_SMP check in smp_mb is uniformly implemented for all SMP barriers. This could extend to virt_mb's if they're introduced in the future.

Quoted 7 lines
Quoted 4 lines
---
 rust/kernel/sync/atomic/ordering.rs |   2 +-
 rust/kernel/sync/barrier.rs         | 194 ++++++++++++++++++++++++----

IMO this patch should be split up into different patches for CPU vs IO, and perhaps even more patches separating out different barrier types.

Given the different barrier types are quite closely related, I don't want to create a bunch of small patches adding one each. But splitting out the SMP change and then add DMA/mandatory barrier as another patch does sound reasonable.

Quoted 50 lines

rust/kernel/sync/atomic/ordering.rs

Quoted 7 lines
 2 files changed, 168 insertions(+), 28 deletions(-)
rust/kernel/sync/atomic/ordering.rs +0 -0
index 3f103aa8db99..c4e732e7212f 100644

[...]> +// Currently kernel only support rmb, wmb and full mb.

Quoted 42 lines
rust/kernel/sync/atomic/ordering.rs +29 -7
+impl MemoryBarrier<Smp> for Read {
+    #[inline]
+    fn run() {
+        // SAFETY: `smp_rmb()` is safe to call.
+        unsafe { bindings::smp_rmb() };
+    }
+}
+
+impl MemoryBarrier<Smp> for Write {
+    #[inline]
+    fn run() {
         // SAFETY: `smp_wmb()` is safe to call.
         unsafe { bindings::smp_wmb() };
-    } else {
-        barrier();
     }
 }
 
-/// A read-read memory barrier.
+impl MemoryBarrier<Smp> for Full {
+    #[inline]
+    fn run() {
+        // SAFETY: `smp_mb()` is safe to call.
+        unsafe { bindings::smp_mb() };
+    }
+}
+
+/// Memory barrier.
 ///
-/// A barrier that prevents compiler and CPU from reordering memory read accesses across the
-/// barrier.
-#[inline(always)]
-pub fn smp_rmb() {
+/// A barrier that prevents compiler and CPU from reordering memory accesses across the barrier.
+///
+/// The specific forms of reordering can be specified using the parameter.
+/// - `mb(Read)` provides a read-read barrier.
+/// - `mb(Write)` provides a write-write barrier.
+/// - `mb(Full)` provides a full barrier.
+/// - `mb(Acquire)` prevents preceding read from being ordered against succeeding memory
+///    operations.
+/// - `mb(Release)` prevents preceding memory operations from being ordered against succeeding
Quoted 8 lines
Quoted 1 line

+/// writes.

I don't agree with this definition of Release. Release is always associated with a specific store, likewise acquire with a load. The definition above also doesn't make sense 'prevents preceding memory operations from being ordered against succeeding writes', that's not what Release semantics are. Release orders memory operations with a specific memory operation associated with Release. Same for Acquire.

I worded my commit message to say that these are about *intentions* and not semantics. I do want to change the semantics too, but it's not fully ready yet. But looks like you're interested, so let me share them too (perhaps a bit prematurely).

Quoted 3 lines

See also in Documentation/memory-barriers.txt, ACQUIRE and RELEASE are defined as being tied to specific memory operations.

That's what we have today, hence the implementation upgrades them to full memory barriers. But ACQUIRE and RELEASE orderings doesn't *need* to be tied to specific memory operations and they can still make conceptual sense as barriers.

C11 memory model defines Acquire and Release fences, and it looks to me it's relatively easy to add it to LKMM. I was playing with Herd7 and I think I've got it working, see the attached diff.

Another thing that I'd like to note is that in all architectures that we have today except ARM and PARISC, the smp_load_acquire and smp_store_release are actually implemented as READ_ONCE + ACQUIRE barrier and RELEASE barrier + WRITE_ONCE. I'm planning to propose C API and corresponding memory model change too, but I want to gather some more concrete numbers (the performance benefit of having dma_mb_acquire/dma_mb_release compared to full dma_mb) before proposing so.

Note that marked dma_load_acquire/dma_store_release (and their mandatory versions) don't make too much sense, as AFAIK no architectures have instructions for them so you're implementing these as fence instructions anyway.

Best,
Gary

tools/memory-model/linux-kernel.bell +2 -0
index fe65998002b9..9b3322fc5b9c 100644
@@ -25,6 +25,8 @@ instructions RMW[Accesses]
 enum Barriers = 'wmb (*smp_wmb*) ||
 		'rmb (*smp_rmb*) ||
 		'MB (*smp_mb*) ||
+		'mb-acquire (*smp_mb_acquire*) ||
+		'mb-release (*smp_mb_release*) ||
 		'barrier (*barrier*) ||
 		'rcu-lock (*rcu_read_lock*)  ||
 		'rcu-unlock (*rcu_read_unlock*) ||
tools/memory-model/linux-kernel.cat +6 -3
index d7e7bf13c831..d7be7c03bb25 100644
@@ -33,6 +33,8 @@ let po-unlock-lock-po = po ; [UL] ; (po|rf) ; [LKR] ; po
 let R4rmb = R \ Noreturn	(* Reads for which rmb works *)
 let rmb = [R4rmb] ; fencerel(Rmb) ; [R4rmb]
 let wmb = [W] ; fencerel(Wmb) ; [W]
+let mb-acquire = [R4rmb] ; fencerel(Mb-acquire) ; [M]
+let mb-release = [M] ; fencerel(Mb-release) ; [W]
 let mb = ([M] ; fencerel(Mb) ; [M]) |
 	(*
 	 * full-barrier RMWs (successful cmpxchg(), xchg(), etc.) act as
@@ -64,9 +66,10 @@ let mb = ([M] ; fencerel(Mb) ; [M]) |
 let gp = po ; [Sync-rcu | Sync-srcu] ; po?
 let strong-fence = mb | gp
 
-let nonrw-fence = strong-fence | po-rel | acq-po
+let nonrw-fence = strong-fence | po-rel | acq-po | mb-acquire | mb-release
 let fence = nonrw-fence | wmb | rmb
-let barrier = fencerel(Barrier | Rmb | Wmb | Mb | Sync-rcu | Sync-srcu |
+let barrier = fencerel(Barrier | Rmb | Wmb | Mb | Mb-acquire | Mb-release |
+        Sync-rcu | Sync-srcu |
 		Before-atomic | After-atomic | Acquire | Release |
 		Rcu-lock | Rcu-unlock | Srcu-lock | Srcu-unlock) |
 	(po ; [Release]) | ([Acquire] ; po)
@@ -97,7 +100,7 @@ let ppo = to-r | to-w | (fence & int) | (po-unlock-lock-po & int)
 (* Propagation: Ordering from release operations and strong fences. *)
 let A-cumul(r) = (rfe ; [Marked])? ; r
 let rmw-sequence = (rf ; rmw)*
-let cumul-fence = [Marked] ; (A-cumul(strong-fence | po-rel) | wmb |
+let cumul-fence = [Marked] ; (A-cumul(strong-fence | po-rel | mb-release) | wmb |
 	po-unlock-lock-po) ; [Marked] ; rmw-sequence
 let prop = [Marked] ; (overwrite & ext)? ; cumul-fence* ;
 	[Marked] ; rfe? ; [Marked]
tools/memory-model/linux-kernel.def +2 -0
index 49e402782e49..e32aea2c01a9 100644
@@ -20,6 +20,8 @@ smp_store_mb(X,V) { __store{ONCE}(X,V); __fence{MB}; }
 smp_mb() { __fence{MB}; }
 smp_rmb() { __fence{rmb}; }
 smp_wmb() { __fence{wmb}; }
+smp_mb_acquire() { __fence{mb-acquire}; }
+smp_mb_release() { __fence{mb-release}; }
 smp_mb__before_atomic() { __fence{before-atomic}; }
 smp_mb__after_atomic() { __fence{after-atomic}; }
 smp_mb__after_spinlock() { __fence{after-spinlock}; }
tools/memory-model/litmus-tests/MP+fencembreleaseonceonce+fencembacquireonceonce.litmus new file +30 -0
new file mode 100644
index 000000000000..d53b848c5687
@@ -0,0 +1,30 @@
+C MP+fencembrelonceonce+fencembacquireonceonce
+
+(*
+ * Result: Never
+ *
+ * This litmus test demonstrates that smp_mb_release() and
+ * smp_mb_acquire() provide sufficient ordering for the message-passing
+ * pattern.
+ *)
+
+{}
+
+P0(int *buf, int *flag) // Producer
+{
+	WRITE_ONCE(*buf, 1);
+	smp_mb_release();
+	WRITE_ONCE(*flag, 1);
+}
+
+P1(int *buf, int *flag) // Consumer
+{
+	int r0;
+	int r1;
+
+	r0 = READ_ONCE(*flag);
+	smp_mb_acquire();
+	r1 = READ_ONCE(*buf);
+}
+
+exists (1:r0=1 /\ 1:r1=0) (* Bad outcome. *)
#6 lore raw

Re: [PATCH 2/3] rust: sync: generic memory barriers

Quoted 1 line · Gary Guo

On Thu Apr 2, 2026 at 10:49 PM BST, Joel Fernandes wrote:

[...]

Quoted 18 lines
Quoted 2 lines

See also in Documentation/memory-barriers.txt, ACQUIRE and RELEASE are defined as being tied to specific memory operations.

That's what we have today, hence the implementation upgrades them to full memory barriers. But ACQUIRE and RELEASE orderings doesn't *need* to be tied to specific memory operations and they can still make conceptual sense as barriers.

C11 memory model defines Acquire and Release fences, and it looks to me it's relatively easy to add it to LKMM. I was playing with Herd7 and I think I've got it working, see the attached diff.

Another thing that I'd like to note is that in all architectures that we have today except ARM and PARISC, the smp_load_acquire and smp_store_release are actually implemented as READ_ONCE + ACQUIRE barrier and RELEASE barrier + WRITE_ONCE. I'm planning to propose C API and corresponding memory model change too, but I want to gather some more concrete numbers (the performance benefit of having dma_mb_acquire/dma_mb_release compared to full dma_mb) before proposing so.

Quoted 3 lines

Note that marked dma_load_acquire/dma_store_release (and their mandatory versions) don't make too much sense, as AFAIK no architectures have instructions for them so you're implementing these as fence instructions anyway.

Ah, so you're proposing new memory barrier types that don't exist anywhere in the kernel yet. I thought you were just wrapping existing ones so it got a bit confusing.

My suggestion: fix the nova-core issue in patch 3 using dma_mb() directly, without the new barrier types. Adding a new standalone release/acquire fence semantic to the kernel is something that will need broad consensus from the LKMM maintainers IMO, independent of this series. It's a bigger conversation that could delay your nova bug fix.

Once smp_mb_release() / smp_mb_acquire() (or the DMA equivalents) land properly, updating nova to use them will be trivial. But the correctness of the fix shouldn't depend on semantics that don't exist in the kernel yet, even if they currently degrade to a full barrier.

That said, I find the idea genuinely interesting, and thanks for keeping me in the loop. Standalone acquire/release fences that are formally modeled in LKMM and A-cumulative is interesting as you showed in the herd7 code. You would want to update Documentation/memory-barriers.txt too.

thanks,

--
Joel Fernandes

maintainer
#7 lore raw

Re: [PATCH 2/3] rust: sync: generic memory barriers

Quoted 25 lines · Joel Fernandes
Quoted 1 line · Gary Guo

On Thu Apr 2, 2026 at 10:49 PM BST, Joel Fernandes wrote:

[...]

Quoted 18 lines
Quoted 2 lines

See also in Documentation/memory-barriers.txt, ACQUIRE and RELEASE are defined as being tied to specific memory operations.

That's what we have today, hence the implementation upgrades them to full memory barriers. But ACQUIRE and RELEASE orderings doesn't *need* to be tied to specific memory operations and they can still make conceptual sense as barriers.

C11 memory model defines Acquire and Release fences, and it looks to me it's relatively easy to add it to LKMM. I was playing with Herd7 and I think I've got it working, see the attached diff.

Another thing that I'd like to note is that in all architectures that we have today except ARM and PARISC, the smp_load_acquire and smp_store_release are actually implemented as READ_ONCE + ACQUIRE barrier and RELEASE barrier + WRITE_ONCE. I'm planning to propose C API and corresponding memory model change too, but I want to gather some more concrete numbers (the performance benefit of having dma_mb_acquire/dma_mb_release compared to full dma_mb) before proposing so.

Quoted 7 lines
Quoted 3 lines

Note that marked dma_load_acquire/dma_store_release (and their mandatory versions) don't make too much sense, as AFAIK no architectures have instructions for them so you're implementing these as fence instructions anyway.

Ah, so you're proposing new memory barrier types that don't exist anywhere in the kernel yet. I thought you were just wrapping existing ones so it got a bit confusing.

Well, I am not proposing new memory barrier types in this change, hence I mentioned that these are annotations about intention and documentation and not about their *semantics* currently. Perhaps I didn't phrase this clear enough?

How about change the wording in documentation like this:

    /// `Acquire` is a full memory barrier, but caller indicates that it only
    /// uses it to provide LOAD->{LOAD,STORE} ordering. 

or do you think even that is too much?

Quoted 6 lines

My suggestion: fix the nova-core issue in patch 3 using dma_mb() directly, without the new barrier types. Adding a new standalone release/acquire fence semantic to the kernel is something that will need broad consensus from the LKMM maintainers IMO, independent of this series. It's a bigger conversation that could delay your nova bug fix.

Something like this?

    // TODO: Replace with `dma_mb(Release)` when it's available.
    dma_mb(Full);

Or not even mentioning that? I really like the Acquire/Release because they serve themselves a documentation on why the barrier is there to exist.

When I read a full barrier, I always need to think whether the code really relies on STORE->LOAD ordering or a weaker ordering that could be used. There's a conventional wisdom in userspace C/Rust where when you see seq_cst, 99% of the time it's wrong.

Quoted 10 lines

Once smp_mb_release() / smp_mb_acquire() (or the DMA equivalents) land properly, updating nova to use them will be trivial. But the correctness of the fix shouldn't depend on semantics that don't exist in the kernel yet, even if they currently degrade to a full barrier.

That said, I find the idea genuinely interesting, and thanks for keeping me in the loop. Standalone acquire/release fences that are formally modeled in LKMM and A-cumulative is interesting as you showed in the herd7 code. You would want to update Documentation/memory-barriers.txt too.

Yeah, there're more works to do on changing the semantics, hence I want to limit this to not be about semantics.

Best,
Gary

#8 lore raw

Re: [PATCH 2/3] rust: sync: generic memory barriers

Quoted 38 lines · Gary Guo
Quoted 25 lines · Joel Fernandes
Quoted 1 line · Gary Guo

On Thu Apr 2, 2026 at 10:49 PM BST, Joel Fernandes wrote:

[...]

Quoted 18 lines
Quoted 2 lines

See also in Documentation/memory-barriers.txt, ACQUIRE and RELEASE are defined as being tied to specific memory operations.

That's what we have today, hence the implementation upgrades them to full memory barriers. But ACQUIRE and RELEASE orderings doesn't *need* to be tied to specific memory operations and they can still make conceptual sense as barriers.

C11 memory model defines Acquire and Release fences, and it looks to me it's relatively easy to add it to LKMM. I was playing with Herd7 and I think I've got it working, see the attached diff.

Another thing that I'd like to note is that in all architectures that we have today except ARM and PARISC, the smp_load_acquire and smp_store_release are actually implemented as READ_ONCE + ACQUIRE barrier and RELEASE barrier + WRITE_ONCE. I'm planning to propose C API and corresponding memory model change too, but I want to gather some more concrete numbers (the performance benefit of having dma_mb_acquire/dma_mb_release compared to full dma_mb) before proposing so.

Quoted 7 lines
Quoted 3 lines

Note that marked dma_load_acquire/dma_store_release (and their mandatory versions) don't make too much sense, as AFAIK no architectures have instructions for them so you're implementing these as fence instructions anyway.

Ah, so you're proposing new memory barrier types that don't exist anywhere in the kernel yet. I thought you were just wrapping existing ones so it got a bit confusing.

Well, I am not proposing new memory barrier types in this change, hence I mentioned that these are annotations about intention and documentation and not about their *semantics* currently. Perhaps I didn't phrase this clear enough?

Quoted 6 lines

How about change the wording in documentation like this:

    /// `Acquire` is a full memory barrier, but caller indicates that it only
    /// uses it to provide LOAD->{LOAD,STORE} ordering. 

or do you think even that is too much?

You are conflating lots of terms here, a "full memory barrier" is much stronger, has stronger cumulativity properties etc (even stronger than Acquire/Release).

And I do think it is too much, because you are using the name of existing memory barrier types (Acquire/Release) with specific definition, but saying "This is an acquire lets pretend its a full memory barrier". So you are in effect defining new types IMO.

Quoted 8 lines
Quoted 7 lines

My suggestion: fix the nova-core issue in patch 3 using dma_mb() directly, without the new barrier types. Adding a new standalone release/acquire fence semantic to the kernel is something that will need broad consensus from the LKMM maintainers IMO, independent of this series. It's a bigger conversation that could delay your nova bug fix.

Quoted 7 lines

Something like this?

    // TODO: Replace with `dma_mb(Release)` when it's available.
    dma_mb(Full);

Or not even mentioning that? I really like the Acquire/Release because they serve themselves a documentation on why the barrier is there to exist.

Sure that's certainly better, once the Release stuff that you're sort of re-defining is accepted by LKMM maintainers and others, it can be replaced for sure. But I wouldn't even want to mention a comment until this type of "Standalone Release" is something everyone (LKMM reviewers) agree with and is upstream-bound. Just split the patch series into "Bug fix" and "New memory barrier things".

Quoted 12 lines

When I read a full barrier, I always need to think whether the code really relies on STORE->LOAD ordering or a weaker ordering that could be used. There's a conventional wisdom in userspace C/Rust where when you see seq_cst, 99% of the time it's wrong.

Quoted 6 lines

Once smp_mb_release() / smp_mb_acquire() (or the DMA equivalents) land properly, updating nova to use them will be trivial. But the correctness of the fix shouldn't depend on semantics that don't exist in the kernel yet, even if they currently degrade to a full barrier.

Quoted 7 lines
Quoted 4 lines

That said, I find the idea genuinely interesting, and thanks for keeping me in the loop. Standalone acquire/release fences that are formally modeled in LKMM and A-cumulative is interesting as you showed in the herd7 code. You would want to update Documentation/memory-barriers.txt too.

Yeah, there're more works to do on changing the semantics, hence I want to limit this to not be about semantics.

My gut feeling is, by calling it a Release (something that already has specific definition), the marketing of this is probably going to be one of any hurdles. :) But I'm curious about the take of others on this thread.

thanks,

--
Joel Fernandes

author
#9 lore raw

PATCH 3/3 gpu: nova-core: fix wrong use of barriers in GSP code

In reply to Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +19 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -408,6 +420,10 @@ fn cpu_read_ptr(&self) -> u32 {
 
     // Informs the GSP that it can send `elem_count` new pages into the message queue.
     fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure read pointer is properly ordered.
+        //
+        dma_mb(Release);
+
         super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count)
     }
 
@@ -422,6 +438,9 @@ fn cpu_write_ptr(&self) -> u32 {
 
     // Informs the GSP that it can process `elem_count` new pages from the command queue.
     fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure all command data is visible before updateing ring buffer pointer.
+        dma_mb(Write);
+
         super::fw::gsp_mem::advance_cpu_write_ptr(&self.0, elem_count)
     }
 }
drivers/gpu/nova-core/gsp/fw.rs +0 -12
index 0c8a74f0e8ac..62c2cf1b030c 100644
@@ -42,11 +42,6 @@
 
 // TODO: Replace with `IoView` projections once available.
 pub(super) mod gsp_mem {
-    use core::sync::atomic::{
-        fence,
-        Ordering, //
-    };
-
     use kernel::{
         dma::Coherent,
         dma_read,
@@ -72,10 +67,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 {
 
     pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) {
         let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
-
-        // Ensure read pointer is properly ordered.
-        fence(Ordering::SeqCst);
-
         dma_write!(qs, .cpuq.rx.0.readPtr, rptr);
     }
 
@@ -87,9 +78,6 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
         let wptr = cpu_write_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
 
         dma_write!(qs, .cpuq.tx.0.writePtr, wptr);
-
-        // Ensure all command data is visible before triggering the GSP read.
-        fence(Ordering::SeqCst);
     }
 }
 
Signature · 1 line
2.51.2
#10 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

Hi Gary,

Quoted 39 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +6 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,

drivers/gpu/nova-core/gsp/cmdq.rs fn new(dev: &device::Device<device::Bound>) -> Result<Self> { · line 268

Quoted 5 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.

Just checking, does control dependency on CPU side really apply to ordering for IO (what the device perceives?). IOW, the loads are stores might be ordered on the CPU side, but the device might be seeing these operations out of order. If that is the case, perhaps the control dependency comment is misleading.

drivers/gpu/nova-core/gsp/cmdq.rs fn driver_write_area_size(&self) -> usize { · line 325

Quoted 11 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+

I suggest taking it on a case by case basis, and splitting the patch for each case, for easier review. There are many patterns AFAICS, load-store, store-store etc.

I do acknowledge the issue you find here though. thanks,

--
Joel Fernandes

Quoted 61 lines
drivers/gpu/nova-core/gsp/cmdq.rs +7 -0
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -408,6 +420,10 @@ fn cpu_read_ptr(&self) -> u32 {
 
     // Informs the GSP that it can send `elem_count` new pages into the message queue.
     fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure read pointer is properly ordered.
+        //
+        dma_mb(Release);
+
         super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count)
     }
 
@@ -422,6 +438,9 @@ fn cpu_write_ptr(&self) -> u32 {
 
     // Informs the GSP that it can process `elem_count` new pages from the command queue.
     fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure all command data is visible before updateing ring buffer pointer.
+        dma_mb(Write);
+
         super::fw::gsp_mem::advance_cpu_write_ptr(&self.0, elem_count)
     }
 }
drivers/gpu/nova-core/gsp/fw.rs +0 -12
index 0c8a74f0e8ac..62c2cf1b030c 100644
@@ -42,11 +42,6 @@
 
 // TODO: Replace with `IoView` projections once available.
 pub(super) mod gsp_mem {
-    use core::sync::atomic::{
-        fence,
-        Ordering, //
-    };
-
     use kernel::{
         dma::Coherent,
         dma_read,
@@ -72,10 +67,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 {
 
     pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) {
         let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
-
-        // Ensure read pointer is properly ordered.
-        fence(Ordering::SeqCst);
-
         dma_write!(qs, .cpuq.rx.0.readPtr, rptr);
     }
 
@@ -87,9 +78,6 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
         let wptr = cpu_write_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
 
         dma_write!(qs, .cpuq.tx.0.writePtr, wptr);
-
-        // Ensure all command data is visible before triggering the GSP read.
-        fence(Ordering::SeqCst);
     }
 }
 
Signature · 1 line
Joel Fernandes
#11 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

Btw, nouveau@ list has issues at the moment, I suggest CC Nvidia nova folks directly. Adding some ++ and dropping Nouveau. In case my reply didn't make it, it is below.

Quoted 139 lines · Joel Fernandes

Hi Gary,

Quoted 39 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +6 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,

drivers/gpu/nova-core/gsp/cmdq.rs fn new(dev: &device::Device<device::Bound>) -> Result<Self> { · line 268

Quoted 5 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.

Just checking, does control dependency on CPU side really apply to ordering for IO (what the device perceives?). IOW, the loads are stores might be ordered on the CPU side, but the device might be seeing these operations out of order. If that is the case, perhaps the control dependency comment is misleading.

drivers/gpu/nova-core/gsp/cmdq.rs fn driver_write_area_size(&self) -> usize { · line 325

Quoted 11 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+

I suggest taking it on a case by case basis, and splitting the patch for each case, for easier review. There are many patterns AFAICS, load-store, store-store etc.

I do acknowledge the issue you find here though. thanks,

--
Joel Fernandes

Quoted 61 lines
drivers/gpu/nova-core/gsp/cmdq.rs +7 -0
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -408,6 +420,10 @@ fn cpu_read_ptr(&self) -> u32 {
 
     // Informs the GSP that it can send `elem_count` new pages into the message queue.
     fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure read pointer is properly ordered.
+        //
+        dma_mb(Release);
+
         super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count)
     }
 
@@ -422,6 +438,9 @@ fn cpu_write_ptr(&self) -> u32 {
 
     // Informs the GSP that it can process `elem_count` new pages from the command queue.
     fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure all command data is visible before updateing ring buffer pointer.
+        dma_mb(Write);
+
         super::fw::gsp_mem::advance_cpu_write_ptr(&self.0, elem_count)
     }
 }
drivers/gpu/nova-core/gsp/fw.rs +0 -12
index 0c8a74f0e8ac..62c2cf1b030c 100644
@@ -42,11 +42,6 @@
 
 // TODO: Replace with `IoView` projections once available.
 pub(super) mod gsp_mem {
-    use core::sync::atomic::{
-        fence,
-        Ordering, //
-    };
-
     use kernel::{
         dma::Coherent,
         dma_read,
@@ -72,10 +67,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 {
 
     pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) {
         let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
-
-        // Ensure read pointer is properly ordered.
-        fence(Ordering::SeqCst);
-
         dma_write!(qs, .cpuq.rx.0.readPtr, rptr);
     }
 
@@ -87,9 +78,6 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
         let wptr = cpu_write_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
 
         dma_write!(qs, .cpuq.tx.0.writePtr, wptr);
-
-        // Ensure all command data is visible before triggering the GSP read.
-        fence(Ordering::SeqCst);
     }
 }
 
Signature · 1 line
Joel Fernandes
maintainer
#12 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

In reply to Joel Fernandes
Quoted 45 lines · Joel Fernandes

Hi Gary,

Quoted 42 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +6 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
Quoted 7 lines

drivers/gpu/nova-core/gsp/cmdq.rs fn new(dev: &device::Device<device::Bound>) -> Result<Self> { · line 268

Quoted 2 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.

Just checking, does control dependency on CPU side really apply to ordering for IO (what the device perceives?). IOW, the loads are stores might be ordered on the CPU side, but the device might be seeing these operations out of order. If that is the case, perhaps the control dependency comment is misleading.

Given that CPU cannot speculate store, I don't see how dependency ordering can be broken even if the other side is a bus-mastering device.

For this to be broken, the device would be able to see the dependency-ordered STORE to be propagated to it before it does it own STORE that propagates to the CPU to trigger the CPU STORE in the first place? That'll just break casuality.

I'll say that control dependency is sufficient here. I'm not worried that the compiler will break this particular control dependency given that if gsp_read_ptr == cpu_write_ptr will imply command allocation failure and this condition cannot possibly be optimized out by the compiler.

Best,
Gary

Quoted 18 lines

drivers/gpu/nova-core/gsp/cmdq.rs fn driver_write_area_size(&self) -> usize { · line 325

Quoted 13 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+

I suggest taking it on a case by case basis, and splitting the patch for each case, for easier review. There are many patterns AFAICS, load-store, store-store etc.

Quoted 1 line

I do acknowledge the issue you find here though. thanks,

This is pretty much just a common MP pattern, where there's even an example in Documentation/core-api/circular-buffers.rst (except we're using dma barriers here as we're talking to devices rather than another SMP CPU), so I think there's no real need of breaking this into small parts.

(That documentation makes use of smp_load_acquire/smp_store_release which is another reason I want to at least document the barrrier to be of ACQUIRE/RELEASE ordering).

Perhaps I can move the removal of redundant barrier after write pointer update to another patch.

Best,
Gary

#13 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

Quoted 57 lines · Gary Guo
Quoted 45 lines · Joel Fernandes

Hi Gary,

Quoted 42 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +6 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
Quoted 7 lines

drivers/gpu/nova-core/gsp/cmdq.rs fn new(dev: &device::Device<device::Bound>) -> Result<Self> { · line 268

Quoted 2 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.

Just checking, does control dependency on CPU side really apply to ordering for IO (what the device perceives?). IOW, the loads are stores might be ordered on the CPU side, but the device might be seeing these operations out of order. If that is the case, perhaps the control dependency comment is misleading.

Given that CPU cannot speculate store, I don't see how dependency ordering can be broken even if the other side is a bus-mastering device.

Quoted 8 lines

For this to be broken, the device would be able to see the dependency-ordered STORE to be propagated to it before it does it own STORE that propagates to the CPU to trigger the CPU STORE in the first place? That'll just break casuality.

I'll say that control dependency is sufficient here. I'm not worried that the compiler will break this particular control dependency given that if gsp_read_ptr == cpu_write_ptr will imply command allocation failure and this condition cannot possibly be optimized out by the compiler.

Yep, ok, fair enough.

thanks,

--
Joel Fernandes

#14 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

In reply to Gary Guo
Quoted 10 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

Quoted 6 lines

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

nit: should this have

Fixes 75f6b1de8133 ("gpu: nova-core: gsp: Add GSP command queue bindings and handling")

?

Quoted 43 lines
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +12 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.

drivers/gpu/nova-core/gsp/cmdq.rs fn cpu_read_ptr(&self) -> u32 { · line 423

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
@@ -408,6 +420,10 @@ fn cpu_read_ptr(&self) -> u32 {
 
     // Informs the GSP that it can send `elem_count` new pages into the message queue.
     fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure read pointer is properly ordered.

What about a more specific comment that describes exactly what is ordered, e.g. something like:
Ensure all reads of message data by the CPU have completed before writing the updated read pointer to the GSP, since it may overwrite that data.

Maybe this is just me but it's a lot easier for me to think of the orderings as a pair of (load? store? -> load? store?) which works for everything hw actually supports except for ll+ls+ss, rather than mapping 'Release' to (load+store -> store) in my head. e.g. here IIUC we need to make sure all loads by the CPU are done before we do the store for the pointer, so we need to make sure loads don't cross ahead of this barrier but also that stores don't cross behind it, so (load -> store) should be sufficient? So, depending on what you want to do with the memory model, this could be tightened IMO. Unlike the one below that only needs to order stores with eachother (ss).

drivers/gpu/nova-core/gsp/cmdq.rs fn cpu_read_ptr(&self) -> u32 { · line 424

Quoted 1 line
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
+        //

nit: stray //

Quoted 52 lines
drivers/gpu/nova-core/gsp/cmdq.rs +5 -0
+        dma_mb(Release);
+
         super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count)
     }
 
@@ -422,6 +438,9 @@ fn cpu_write_ptr(&self) -> u32 {
 
     // Informs the GSP that it can process `elem_count` new pages from the command queue.
     fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure all command data is visible before updateing ring buffer pointer.
+        dma_mb(Write);
+
         super::fw::gsp_mem::advance_cpu_write_ptr(&self.0, elem_count)
     }
 }
drivers/gpu/nova-core/gsp/fw.rs +0 -12
index 0c8a74f0e8ac..62c2cf1b030c 100644
@@ -42,11 +42,6 @@
 
 // TODO: Replace with `IoView` projections once available.
 pub(super) mod gsp_mem {
-    use core::sync::atomic::{
-        fence,
-        Ordering, //
-    };
-
     use kernel::{
         dma::Coherent,
         dma_read,
@@ -72,10 +67,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 {
 
     pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) {
         let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
-
-        // Ensure read pointer is properly ordered.
-        fence(Ordering::SeqCst);
-
         dma_write!(qs, .cpuq.rx.0.readPtr, rptr);
     }
 
@@ -87,9 +78,6 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
         let wptr = cpu_write_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
 
         dma_write!(qs, .cpuq.tx.0.writePtr, wptr);
-
-        // Ensure all command data is visible before triggering the GSP read.
-        fence(Ordering::SeqCst);
     }
 }
 
maintainer
#15 lore raw

Re: [PATCH 3/3] gpu: nova-core: fix wrong use of barriers in GSP code

Quoted 76 lines · Eliot Courtney
Quoted 10 lines · Gary Guo

From: Gary Guo <[email protected]>

Currently, in the GSP->CPU messaging path, the current code misses a read barrier before data read. The barrier after read is updated to a DMA barrier (with release ordering desired), instead of the existing (Rust) SeqCst SMP barrier; the location of barrier is also moved to the beginning of function, because the barrier is needed to synchronizing between data and ring-buffer pointer, the RMW operation does not internally need a barrier (nor it has to be atomic, as CPU pointers are updated by CPU only).

Quoted 6 lines

In the CPU->GSP messaging path, the current code misses a write barrier after data write and before updating the CPU write pointer. Barrier is not needed before data write due to control dependency, this fact is documented explicitly. This could be replaced with an acquire barrier if needed.

nit: should this have

Fixes 75f6b1de8133 ("gpu: nova-core: gsp: Add GSP command queue bindings and handling")

?

Quoted 43 lines
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 19 +++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ------------
 2 files changed, 19 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +12 -0
index 2224896ccc89..7e4315b13984 100644
@@ -19,6 +19,12 @@
     prelude::*,
     sync::{
         aref::ARef,
+        barrier::{
+            dma_mb,
+            Read,
+            Release,
+            Write, //
+        },
         Mutex, //
     },
     time::Delta,
@@ -258,6 +264,9 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
         let tx = self.cpu_write_ptr() as usize;
         let rx = self.gsp_read_ptr() as usize;
 
+        // ORDERING: control dependency provides necessary LOAD->STORE ordering.
+        // `dma_mb(Acquire)` may be used here if we don't want to rely on control dependency.
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.
@@ -311,6 +320,9 @@ fn driver_write_area_size(&self) -> usize {
         let tx = self.gsp_write_ptr() as usize;
         let rx = self.cpu_read_ptr() as usize;
 
+        // ORDERING: Ensure data load is ordered after load of GSP write pointer.
+        dma_mb(Read);
+
         // SAFETY:
         // - We will only access the driver-owned part of the shared memory.
         // - Per the safety statement of the function, no concurrent access will be performed.

drivers/gpu/nova-core/gsp/cmdq.rs fn cpu_read_ptr(&self) -> u32 { · line 423

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
@@ -408,6 +420,10 @@ fn cpu_read_ptr(&self) -> u32 {
 
     // Informs the GSP that it can send `elem_count` new pages into the message queue.
     fn advance_cpu_read_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure read pointer is properly ordered.

What about a more specific comment that describes exactly what is ordered, e.g. something like:
Ensure all reads of message data by the CPU have completed before writing the updated read pointer to the GSP, since it may overwrite that data.

Quoted 10 lines

Maybe this is just me but it's a lot easier for me to think of the orderings as a pair of (load? store? -> load? store?) which works for everything hw actually supports except for ll+ls+ss, rather than mapping 'Release' to (load+store -> store) in my head. e.g. here IIUC we need to make sure all loads by the CPU are done before we do the store for the pointer, so we need to make sure loads don't cross ahead of this barrier but also that stores don't cross behind it, so (load -> store) should be sufficient? So, depending on what you want to do with the memory model, this could be tightened IMO. Unlike the one below that only needs to order stores with eachother (ss).

The pair of (access, access) is actually the first iteration of the API that I've designed. You can see it implemented here: https://github.com/nbdd0121/lkmm-rs/blob/trunk/src/lib.rs
A good thing is that it maps nicely to loongarch and RISC-V. A bad thing is that it misses ll+ls+ss as you pointed out.

However, I think the more tricky part of this approach is now the semantics is harder to fit into LKMM; it is now became 9 types of barrier that we need to reason about, and that's a huge addition. I am not sure it is even feasible to implement, given the KCSAN only have 5 types of barriers that it could encode about.

On the other hand, acquire/barriers are much easier to reason about (and they already exist in C11/Rust memory models), and KCSAN already implements marked release accesses as ONCE + release barrier (with acquire accesses being implicit), so it probably doesn't even need changing.

Best,
Gary

Quoted 57 lines

drivers/gpu/nova-core/gsp/cmdq.rs fn cpu_read_ptr(&self) -> u32 { · line 424

Quoted 2 lines
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
+        //

nit: stray //

Quoted 52 lines
drivers/gpu/nova-core/gsp/cmdq.rs +5 -0
+        dma_mb(Release);
+
         super::fw::gsp_mem::advance_cpu_read_ptr(&self.0, elem_count)
     }
 
@@ -422,6 +438,9 @@ fn cpu_write_ptr(&self) -> u32 {
 
     // Informs the GSP that it can process `elem_count` new pages from the command queue.
     fn advance_cpu_write_ptr(&mut self, elem_count: u32) {
+        // ORDERING: Ensure all command data is visible before updateing ring buffer pointer.
+        dma_mb(Write);
+
         super::fw::gsp_mem::advance_cpu_write_ptr(&self.0, elem_count)
     }
 }
drivers/gpu/nova-core/gsp/fw.rs +0 -12
index 0c8a74f0e8ac..62c2cf1b030c 100644
@@ -42,11 +42,6 @@
 
 // TODO: Replace with `IoView` projections once available.
 pub(super) mod gsp_mem {
-    use core::sync::atomic::{
-        fence,
-        Ordering, //
-    };
-
     use kernel::{
         dma::Coherent,
         dma_read,
@@ -72,10 +67,6 @@ pub(in crate::gsp) fn cpu_read_ptr(qs: &Coherent<GspMem>) -> u32 {
 
     pub(in crate::gsp) fn advance_cpu_read_ptr(qs: &Coherent<GspMem>, count: u32) {
         let rptr = cpu_read_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
-
-        // Ensure read pointer is properly ordered.
-        fence(Ordering::SeqCst);
-
         dma_write!(qs, .cpuq.rx.0.readPtr, rptr);
     }
 
@@ -87,9 +78,6 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
         let wptr = cpu_write_ptr(qs).wrapping_add(count) % MSGQ_NUM_PAGES;
 
         dma_write!(qs, .cpuq.tx.0.writePtr, wptr);
-
-        // Ensure all command data is visible before triggering the GSP read.
-        fence(Ordering::SeqCst);
     }
 }