Skip to contentSkip to conversation

gpu: nova-core: gsp: add locking to Cmdq

PATCH Applied Eliot Courtney 4 participants 13 messages Mar 18 – Mar 20

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v5 cover gpu: nova-core: gsp: add locking to Cmdq

Add locking to Cmdq. This is required e.g. for unloading the driver, which needs to send the UnloadingGuestDriver via the command queue on unbind which may be on a different thread.

We have commands that need a reply and commands that don't. For commands with a reply we want to make sure that they don't get the reply of a different command back. The approach this patch series takes is by making those commands block until they get a response. For now this should be ok, and we expect GSP to be fast anyway.

To do this, we need to know which commands expect a reply and which don't. John's existing series[1] adds IS_ASYNC which solves part of the problem, but we need to know a bit more. So instead, add an associated type called Reply which tells us what the reply is.

An alternative would be to define traits inheriting CommandToGsp, e.g. CommandWithReply and CommandWithoutReply, instead of using the associated type. I implemented the associated type version because it feels more compositional rather than inherity so seemed a bit better to me. But both of these approaches work and are fine, IMO.

In summary, this patch series has three steps:
1. Add the type infrastructure to know what replies are expected for a command and update each caller to explicitly wait for the reply or not.
2. Make Cmdq pinned so we can use Mutex
3. Add a Mutex to protect Cmdq by moving the relevant state to an inner struct.

[1]: https://lore.kernel.org/all/[email protected]/

---
Changes in v5:
- tweak documentation
- rebase on drm-rust-next
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- Change RECEIVE_TIMEOUT from 10s to 5s
- Update NoReply doc comment
- Move CommandToGsp doc comment update
- Fix import formatting in continuation.rs
- Move CmdqInner after Cmdq to reduce diff size
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- Rename send_sync_command/send_async_command to send_command/send_command_no_wait.
- Move dev field into CmdqInner to avoid passing it through method parameters.
- Add RECEIVE_TIMEOUT constant for the 10s receive timeout.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- Rebase on drm-rust-next
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Eliot Courtney (5):
gpu: nova-core: gsp: fix stale doc comments on command queue methods
gpu: nova-core: gsp: add RECEIVE_TIMEOUT constant for command queue
gpu: nova-core: gsp: add reply/no-reply info to CommandToGsp
gpu: nova-core: gsp: make Cmdq a pinned type
gpu: nova-core: gsp: add mutex locking to Cmdq

 drivers/gpu/nova-core/gsp.rs                   |   5 +-
 drivers/gpu/nova-core/gsp/boot.rs              |  13 +-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 159 +++++++++++++++++++------
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |   8 +-
 drivers/gpu/nova-core/gsp/commands.rs          |  23 ++--
 drivers/gpu/nova-core/gsp/sequencer.rs         |   4 +-
 6 files changed, 151 insertions(+), 61 deletions(-)

---
base-commit: d19ab42867ae7c68be84ed957d95712b7934773f
change-id: 20260225-cmdq-locking-d32928a2c2cf

Best regards,

Signature · 1 line
Eliot Courtney <[email protected]>
#1 lore raw

PATCH v5 1/5 gpu: nova-core: gsp: fix stale doc comments on command queue methods

Fix some inaccuracies / old doc comments.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +8 -9
index efa1aab1568f..f7ca6856ff35 100644
@@ -502,6 +502,7 @@ fn notify_gsp(bar: &Bar0) {
     ///
     /// # Errors
     ///
+    /// - `EMSGSIZE` if the command exceeds the maximum queue element size.
     /// - `ETIMEDOUT` if space does not become available within the timeout.
     /// - `EIO` if the variable payload requested by the command has not been entirely
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
@@ -682,22 +683,20 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
     /// Receive a message from the GSP.
     ///
-    /// `init` is a closure tasked with processing the message. It receives a reference to the
-    /// message in the message queue, and a [`SBufferIter`] pointing to its variable-length
-    /// payload, if any.
+    /// The expected message type is specified using the `M` generic parameter. If the pending
+    /// message has a different function code, `ERANGE` is returned and the message is consumed.
     ///
-    /// The expected message is specified using the `M` generic parameter. If the pending message
-    /// is different, `EAGAIN` is returned and the unexpected message is dropped.
-    ///
-    /// This design is by no means final, but it is simple and will let us go through GSP
-    /// initialization.
+    /// The read pointer is always advanced past the message, regardless of whether it matched.
     ///
     /// # Errors
     ///
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `EINVAL` if the function of the message was unrecognized.
+    /// - `EINVAL` if the function code of the message was not recognized.
+    /// - `ERANGE` if the message had a recognized but non-matching function code.
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     pub(crate) fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
     where
         // This allows all error types, including `Infallible`, to be used for `M::InitError`.
Signature · 1 line
2.53.0
#2 lore raw

PATCH v5 2/5 gpu: nova-core: gsp: add `RECEIVE_TIMEOUT` constant for command queue

In reply to Eliot Courtney

Remove magic numbers and add a default timeout for callers to use.

---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 3 +++
 drivers/gpu/nova-core/gsp/commands.rs  | 5 ++---
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +3 -0
index f7ca6856ff35..c62db727a2a9 100644
@@ -467,6 +467,9 @@ impl Cmdq {
     /// Timeout for waiting for space on the command queue.
     const ALLOCATE_TIMEOUT: Delta = Delta::from_secs(1);
 
+    /// Default timeout for receiving a message from the GSP.
+    pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(5);
+
     /// Creates a new command queue for `dev`.
     pub(crate) fn new(dev: &device::Device<device::Bound>) -> Result<Cmdq> {
         let gsp_mem = DmaGspMem::new(dev)?;
drivers/gpu/nova-core/gsp/commands.rs +2 -3
index 8f270eca33be..88df117ba575 100644
@@ -11,7 +11,6 @@
     device,
     pci,
     prelude::*,
-    time::Delta,
     transmute::{
         AsBytes,
         FromBytes, //
@@ -165,7 +164,7 @@ fn read(
 /// Waits for GSP initialization to complete.
 pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
     loop {
-        match cmdq.receive_msg::<GspInitDone>(Delta::from_secs(10)) {
+        match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
             Err(ERANGE) => continue,
             Err(e) => break Err(e),
@@ -235,7 +234,7 @@ pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticIn
     cmdq.send_command(bar, GetGspStaticInfo)?;
 
     loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Delta::from_secs(5)) {
+        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(info) => return Ok(info),
             Err(ERANGE) => continue,
             Err(e) => return Err(e),
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
index 0cfbedc47fcf..ce2b3bb05d22 100644
@@ -358,7 +358,7 @@ pub(crate) struct GspSequencerParams<'a> {
 impl<'a> GspSequencer<'a> {
     pub(crate) fn run(cmdq: &mut Cmdq, params: GspSequencerParams<'a>) -> Result {
         let seq_info = loop {
-            match cmdq.receive_msg::<GspSequence>(Delta::from_secs(10)) {
+            match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
                 Err(ERANGE) => continue,
                 Err(e) => return Err(e),
Signature · 1 line
2.53.0
#3 lore raw

PATCH v5 3/5 gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

In reply to Eliot Courtney

Add type infrastructure to know what reply is expected from each CommandToGsp. Uses a marker type NoReply which does not implement MessageFromGsp to mark commands which don't expect a response.

Update send_command to wait for a reply and add send_command_no_wait which sends a command that has no reply, without blocking.

This prepares for adding locking to the queue.

---
 drivers/gpu/nova-core/gsp/boot.rs              |  5 ++-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 62 ++++++++++++++++++++++++--
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |  8 +++-
 drivers/gpu/nova-core/gsp/commands.rs          | 16 +++----
 4 files changed, 75 insertions(+), 16 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +3 -2
index 6db2decbc6f5..ffc478b33640 100644
@@ -169,8 +169,9 @@ pub(crate) fn boot(
         dma_write!(wpr_meta, [0]?, GspFwWprMeta::new(&gsp_fw, &fb_layout));
 
         self.cmdq
-            .send_command(bar, commands::SetSystemInfo::new(pdev))?;
-        self.cmdq.send_command(bar, commands::SetRegistry::new())?;
+            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev))?;
+        self.cmdq
+            .send_command_no_wait(bar, commands::SetRegistry::new())?;
 
         gsp_falcon.reset(bar)?;
         let libos_handle = self.libos.dma_handle();
drivers/gpu/nova-core/gsp/cmdq.rs +59 -3

88 lines not on this page · raw message

drivers/gpu/nova-core/gsp/cmdq/continuation.rs +7 -1
index 2aa17caac2e0..05e904f18097 100644
@@ -6,7 +6,10 @@
 
 use kernel::prelude::*;
 
-use super::CommandToGsp;
+use super::{
+    CommandToGsp,
+    NoReply, //
+};
 
 use crate::{
     gsp::fw::{
@@ -63,6 +66,7 @@ fn new(data: &'a [u8]) -> Self {
 impl<'a> CommandToGsp for ContinuationRecord<'a> {
     const FUNCTION: MsgFunction = MsgFunction::ContinuationRecord;
     type Command = ();
+    type Reply = NoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -144,6 +148,7 @@ fn new(command: C, payload: KVVec<u8>) -> Self {
 impl<C: CommandToGsp> CommandToGsp for SplitCommand<C> {
     const FUNCTION: MsgFunction = C::FUNCTION;
     type Command = C::Command;
+    type Reply = C::Reply;
     type InitError = C::InitError;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -206,6 +211,7 @@ fn new(len: usize) -> Result<Self> {
     impl CommandToGsp for TestPayload {
         const FUNCTION: MsgFunction = MsgFunction::Nop;
         type Command = TestHeader;
+        type Reply = NoReply;
         type InitError = Infallible;
 
         fn init(&self) -> impl Init<Self::Command, Self::InitError> {
drivers/gpu/nova-core/gsp/commands.rs +6 -10
index 88df117ba575..77054c92fcc2 100644
@@ -23,7 +23,8 @@
         cmdq::{
             Cmdq,
             CommandToGsp,
-            MessageFromGsp, //
+            MessageFromGsp,
+            NoReply, //
         },
         fw::{
             commands::*,
@@ -48,6 +49,7 @@ pub(crate) fn new(pdev: &'a pci::Device<device::Bound>) -> Self {
 impl<'a> CommandToGsp for SetSystemInfo<'a> {
     const FUNCTION: MsgFunction = MsgFunction::GspSetSystemInfo;
     type Command = GspSetSystemInfo;
+    type Reply = NoReply;
     type InitError = Error;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -99,6 +101,7 @@ pub(crate) fn new() -> Self {
 impl CommandToGsp for SetRegistry {
     const FUNCTION: MsgFunction = MsgFunction::SetRegistry;
     type Command = PackedRegistryTable;
+    type Reply = NoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -178,6 +181,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
 impl CommandToGsp for GetGspStaticInfo {
     const FUNCTION: MsgFunction = MsgFunction::GetGspStaticInfo;
     type Command = GspStaticConfigInfo;
+    type Reply = GetGspStaticInfoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -231,13 +235,5 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
 
 /// Send the [`GetGspInfo`] command and awaits for its reply.
 pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
-    cmdq.send_command(bar, GetGspStaticInfo)?;
-
-    loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(info) => return Ok(info),
-            Err(ERANGE) => continue,
-            Err(e) => return Err(e),
-        }
-    }
+    cmdq.send_command(bar, GetGspStaticInfo)
 }
Signature · 1 line
2.53.0
#4 lore raw

PATCH v5 4/5 gpu: nova-core: gsp: make `Cmdq` a pinned type

In reply to Eliot Courtney

Make Cmdq a pinned type. This is needed to use Mutex, which is needed to add locking to Cmdq.

---
 drivers/gpu/nova-core/gsp.rs      | 5 +++--
 drivers/gpu/nova-core/gsp/cmdq.rs | 9 ++++-----
 2 files changed, 7 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/gsp.rs +3 -2
index c69adaa92bbe..72f173726f87 100644
@@ -114,6 +114,7 @@ pub(crate) struct Gsp {
     /// RM log buffer.
     logrm: LogBuffer,
     /// Command queue.
+    #[pin]
     pub(crate) cmdq: Cmdq,
     /// RM arguments.
     rmargs: CoherentAllocation<GspArgumentsPadded>,
@@ -134,7 +135,7 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
                 loginit: LogBuffer::new(dev)?,
                 logintr: LogBuffer::new(dev)?,
                 logrm: LogBuffer::new(dev)?,
-                cmdq: Cmdq::new(dev)?,
+                cmdq <- Cmdq::new(dev),
                 rmargs: CoherentAllocation::<GspArgumentsPadded>::alloc_coherent(
                     dev,
                     1,
@@ -151,7 +152,7 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
                         libos, [1]?, LibosMemoryRegionInitArgument::new("LOGINTR", &logintr.0)
                     );
                     dma_write!(libos, [2]?, LibosMemoryRegionInitArgument::new("LOGRM", &logrm.0));
-                    dma_write!(rmargs, [0]?.inner, fw::GspArgumentsCached::new(cmdq));
+                    dma_write!(rmargs, [0]?.inner, fw::GspArgumentsCached::new(&cmdq));
                     dma_write!(libos, [3]?, LibosMemoryRegionInitArgument::new("RMARGS", rmargs));
                 },
             }))
drivers/gpu/nova-core/gsp/cmdq.rs +4 -5
index 4fc14689d38e..86ff9a3d1732 100644
@@ -446,6 +446,7 @@ struct GspMessage<'a> {
 ///
 /// Provides the ability to send commands and receive messages from the GSP using a shared memory
 /// area.
+#[pin_data]
 pub(crate) struct Cmdq {
     /// Device this command queue belongs to.
     dev: ARef<device::Device>,
@@ -479,13 +480,11 @@ impl Cmdq {
     pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(5);
 
     /// Creates a new command queue for `dev`.
-    pub(crate) fn new(dev: &device::Device<device::Bound>) -> Result<Cmdq> {
-        let gsp_mem = DmaGspMem::new(dev)?;
-
-        Ok(Cmdq {
+    pub(crate) fn new(dev: &device::Device<device::Bound>) -> impl PinInit<Self, Error> + '_ {
+        try_pin_init!(Self {
+            gsp_mem: DmaGspMem::new(dev)?,
             dev: dev.into(),
             seq: 0,
-            gsp_mem,
         })
     }
 
Signature · 1 line
2.53.0
#5 lore raw

PATCH v5 5/5 gpu: nova-core: gsp: add mutex locking to Cmdq

In reply to Eliot Courtney

Wrap Cmdq's mutable state in a new struct CmdqInner and wrap that in a Mutex. This lets Cmdq methods take &self instead of &mut self, which lets required commands be sent e.g. while unloading the driver.

The mutex is held over both send and receive in send_command to make sure that it doesn't get the reply of some other command that could have been sent just beforehand.

---
 drivers/gpu/nova-core/gsp/boot.rs      |   8 +-
 drivers/gpu/nova-core/gsp/cmdq.rs      | 170 +++++++++++++++++++--------------
 drivers/gpu/nova-core/gsp/commands.rs  |   4 +-
 drivers/gpu/nova-core/gsp/sequencer.rs |   2 +-
 4 files changed, 107 insertions(+), 77 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +4 -4
index ffc478b33640..5e73bd769dcc 100644
@@ -137,7 +137,7 @@ fn run_fwsec_frts(
     ///
     /// Upon return, the GSP is up and running, and its runtime object given as return value.
     pub(crate) fn boot(
-        mut self: Pin<&mut Self>,
+        self: Pin<&mut Self>,
         pdev: &pci::Device<device::Bound>,
         bar: &Bar0,
         chipset: Chipset,
@@ -223,13 +223,13 @@ pub(crate) fn boot(
             dev: pdev.as_ref().into(),
             bar,
         };
-        GspSequencer::run(&mut self.cmdq, seq_params)?;
+        GspSequencer::run(&self.cmdq, seq_params)?;
 
         // Wait until GSP is fully initialized.
-        commands::wait_gsp_init_done(&mut self.cmdq)?;
+        commands::wait_gsp_init_done(&self.cmdq)?;
 
         // Obtain and display basic GPU information.
-        let info = commands::get_gsp_info(&mut self.cmdq, bar)?;
+        let info = commands::get_gsp_info(&self.cmdq, bar)?;
         match info.gpu_name() {
             Ok(name) => dev_info!(pdev, "GPU name: {}\n", name),
             Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e),
drivers/gpu/nova-core/gsp/cmdq.rs +100 -70

235 lines not on this page · raw message

drivers/gpu/nova-core/gsp/commands.rs +2 -2
index 77054c92fcc2..c89c7b57a751 100644
@@ -165,7 +165,7 @@ fn read(
 }
 
 /// Waits for GSP initialization to complete.
-pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
+pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq) -> Result {
     loop {
         match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
@@ -234,6 +234,6 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
 }
 
 /// Send the [`GetGspInfo`] command and awaits for its reply.
-pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
+pub(crate) fn get_gsp_info(cmdq: &Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
     cmdq.send_command(bar, GetGspStaticInfo)
 }
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
index ce2b3bb05d22..474e4c8021db 100644
@@ -356,7 +356,7 @@ pub(crate) struct GspSequencerParams<'a> {
 }
 
 impl<'a> GspSequencer<'a> {
-    pub(crate) fn run(cmdq: &mut Cmdq, params: GspSequencerParams<'a>) -> Result {
+    pub(crate) fn run(cmdq: &Cmdq, params: GspSequencerParams<'a>) -> Result {
         let seq_info = loop {
             match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
Signature · 1 line
2.53.0
#6 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

No need to resend, I will apply this patch on top of mine and merge.

maintainer
#7 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 10 lines · Alexandre Courbot

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Quoted 7 lines

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

No need to resend, I will apply this patch on top of mine and merge.

It is a minor inconvinience, but it may indicate that dma::Coherent should probaly support locking, i.e. you want to protect the data within the dma::Coherent allocation, not the dma::Coherent object itself.

#8 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 17 lines · Danilo Krummrich
Quoted 16 lines · Alexandre Courbot

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

Quoted 5 lines
Quoted 1 line

No need to resend, I will apply this patch on top of mine and merge.

It is a minor inconvinience, but it may indicate that dma::Coherent should probaly support locking, i.e. you want to protect the data within the dma::Coherent allocation, not the dma::Coherent object itself.

Not quite sure I understand what you mean here - can you elaborate? This sounds to me like you want to add a Mutex to every dma::Coherent, so I am likely misunderstanding. :)

#9 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

In reply to Alexandre Courbot
Quoted 10 lines · Alexandre Courbot

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Quoted 7 lines

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

No need to resend, I will apply this patch on top of mine and merge.

Actually let's fix that afterwards, what I proposed above would introduce code that hasn't undergone review, and doesn't reduce the number of patches in the series so doesn't bring any benefit. I'll merge the series as-is for now.

#10 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 17 lines · Alexandre Courbot
Quoted 16 lines · Alexandre Courbot

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

Quoted 6 lines
Quoted 1 line

No need to resend, I will apply this patch on top of mine and merge.

Actually let's fix that afterwards, what I proposed above would introduce code that hasn't undergone review, and doesn't reduce the number of patches in the series so doesn't bring any benefit. I'll merge the series as-is for now.

For what it's worth, this avoided an email from me, to the effect of "there is no such thing as a quick locking fixup, don't do this".

So, good. :)

thanks,

Signature · 1 line
John Hubbard
#11 lore raw

Re: [PATCH v5 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

In reply to Alexandre Courbot
Quoted 10 lines · Alexandre Courbot

On Wed Mar 18, 2026 at 1:07 PM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 577

Quoted 4 lines
drivers/gpu/nova-core/gsp/cmdq.rs +4 -0
+    /// Returns the DMA handle of the command queue's shared memory region.
+    pub(crate) fn dma_handle(&self) -> DmaAddress {
+        self.inner.lock().gsp_mem.0.dma_handle()
+    }

Just noticed that we now need to lock to get the DMA handle. It's inconsequential in practice but a bit inelegant.

Quoted 7 lines

Since the DMA handle never changes, and is only ever needed during initialization, I think I will just insert a patch before this one that adds a pub(super) dma_handle member to Cmdq. That way we only need to obtain the handle at construction time and can get rid of this method, which keeps the public API focused on message handling.

No need to resend, I will apply this patch on top of mine and merge.

Yerp it's not ideal, but we only call this method in one location, once, when creating the command queue. The locking in this series is very basic and we probably want to improve it later to be more performant, so I think it's simpler and better to just leave this as is for now.

maintainer Applied
#12 lore raw
In reply to Eliot Courtney
Quoted 29 lines · Eliot Courtney

Add locking to Cmdq. This is required e.g. for unloading the driver, which needs to send the UnloadingGuestDriver via the command queue on unbind which may be on a different thread.

We have commands that need a reply and commands that don't. For commands with a reply we want to make sure that they don't get the reply of a different command back. The approach this patch series takes is by making those commands block until they get a response. For now this should be ok, and we expect GSP to be fast anyway.

To do this, we need to know which commands expect a reply and which don't. John's existing series[1] adds IS_ASYNC which solves part of the problem, but we need to know a bit more. So instead, add an associated type called Reply which tells us what the reply is.

An alternative would be to define traits inheriting CommandToGsp, e.g. CommandWithReply and CommandWithoutReply, instead of using the associated type. I implemented the associated type version because it feels more compositional rather than inherity so seemed a bit better to me. But both of these approaches work and are fine, IMO.

In summary, this patch series has three steps:
1. Add the type infrastructure to know what replies are expected for a command and update each caller to explicitly wait for the reply or not.
2. Make Cmdq pinned so we can use Mutex
3. Add a Mutex to protect Cmdq by moving the relevant state to an inner struct.

Quoted 3 lines

[1]: https://lore.kernel.org/all/[email protected]/

Merged into drm-rust-next, thanks!