Skip to contentSkip to conversation

gpu: nova-core: gsp: add locking to Cmdq

PATCH Superseded Eliot Courtney 3 participants 24 messages Mar 4 – Mar 5

Subsystems (4)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover gpu: nova-core: gsp: add locking to Cmdq

Add locking to Cmdq. This is required e.g. for unloading the driver, which needs to send the UnloadingGuestDriver via the command queue on unbind which may be on a different thread.

We have commands that need a reply and commands that don't. For commands with a reply we want to make sure that they don't get the reply of a different command back. The approach this patch series takes is by making those commands block until they get a response. For now this should be ok, and we expect GSP to be fast anyway.

To do this, we need to know which commands expect a reply and which don't. John's existing series[1] adds IS_ASYNC which solves part of the problem, but we need to know a bit more. So instead, add an associated type called Reply which tells us what the reply is.

An alternative would be to define traits inheriting CommandToGsp, e.g. CommandWithReply and CommandWithoutReply, instead of using the associated type. I implemented the associated type version because it feels more compositional rather than inherity so seemed a bit better to me. But both of these approaches work and are fine, IMO.

In summary, this patch series has three steps:
1. Add the type infrastructure to know what replies are expected for a command and update each caller to explicitly wait for the reply or not.
2. Make Cmdq pinned so we can use Mutex
3. Add a Mutex to protect Cmdq by moving the relevant state to an inner struct.

[1]: https://lore.kernel.org/all/[email protected]/

---
Changes in v3:
- Rename send_sync_command/send_async_command to send_command/send_command_no_wait.
- Move dev field into CmdqInner to avoid passing it through method parameters.
- Add RECEIVE_TIMEOUT constant for the 10s receive timeout.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- Rebase on drm-rust-next
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Eliot Courtney (5):
gpu: nova-core: gsp: fix stale doc comments on command queue methods
gpu: nova-core: gsp: add RECEIVE_TIMEOUT constant for command queue
gpu: nova-core: gsp: add reply/no-reply info to CommandToGsp
gpu: nova-core: gsp: make Cmdq a pinned type
gpu: nova-core: gsp: add mutex locking to Cmdq

 drivers/gpu/nova-core/gsp.rs                   |   5 +-
 drivers/gpu/nova-core/gsp/boot.rs              |  13 +-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 225 +++++++++++++++++--------
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |   5 +-
 drivers/gpu/nova-core/gsp/commands.rs          |  23 +--
 drivers/gpu/nova-core/gsp/sequencer.rs         |   4 +-
 6 files changed, 180 insertions(+), 95 deletions(-)

---
base-commit: 4a49fe23e357b48845e31fe9c28a802c05458198
change-id: 20260225-cmdq-locking-d32928a2c2cf
prerequisite-message-id: <[email protected]>
prerequisite-patch-id: fd45bc5b8eda5e2b54a052dddb1a1c363107f0a7
prerequisite-patch-id: 06fe65f900206c44b5ba52286ca4ce1ca42b55d5
prerequisite-patch-id: 8844970d0e387488c70979a73732579ba174b46c
prerequisite-patch-id: e138a94ed48fa8d50d5ed1eb36524f98923ed478
prerequisite-patch-id: dccf2b12b176947e89b44baafda9c5a0aa0a93bc
prerequisite-patch-id: 30ed64c398e541d6efbcb2e46ed9a9e6cf953f4f
prerequisite-patch-id: ba1c8da0cbdb4682b879633a94a172d1b2b6bc8e
prerequisite-patch-id: 081d4a4198a0bf09f3480cb8baf296db585decce
prerequisite-patch-id: 56c8c25e7362178cd019c8f03954a6bcdb72b1b5

Best regards,

Signature · 1 line
Eliot Courtney <[email protected]>
#1 lore raw

PATCH v3 1/5 gpu: nova-core: gsp: fix stale doc comments on command queue methods

Fix some inaccuracies / old doc comments.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +8 -9
index 492e9489e808..4829830b6921 100644
@@ -531,6 +531,7 @@ fn notify_gsp(bar: &Bar0) {
     ///
     /// # Errors
     ///
+    /// - `EMSGSIZE` if the command exceeds the maximum queue element size.
     /// - `ETIMEDOUT` if space does not become available within the timeout.
     /// - `EIO` if the variable payload requested by the command has not been entirely
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
@@ -711,22 +712,20 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
     /// Receive a message from the GSP.
     ///
-    /// `init` is a closure tasked with processing the message. It receives a reference to the
-    /// message in the message queue, and a [`SBufferIter`] pointing to its variable-length
-    /// payload, if any.
+    /// The expected message type is specified using the `M` generic parameter. If the pending
+    /// message has a different function code, `ERANGE` is returned and the message is consumed.
     ///
-    /// The expected message is specified using the `M` generic parameter. If the pending message
-    /// is different, `EAGAIN` is returned and the unexpected message is dropped.
-    ///
-    /// This design is by no means final, but it is simple and will let us go through GSP
-    /// initialization.
+    /// The read pointer is always advanced past the message, regardless of whether it matched.
     ///
     /// # Errors
     ///
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `EINVAL` if the function of the message was unrecognized.
+    /// - `EINVAL` if the function code of the message was not recognized.
+    /// - `ERANGE` if the message had a recognized but non-matching function code.
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     pub(crate) fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
     where
         // This allows all error types, including `Infallible`, to be used for `M::InitError`.
Signature · 1 line
2.53.0
reviewer
#2 lore raw

Re: [PATCH v3 1/5] gpu: nova-core: gsp: fix stale doc comments on command queue methods

Quoted 14 lines · Eliot Courtney

Fix some inaccuracies / old doc comments.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +0 -0
index 492e9489e808..4829830b6921 100644

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 536

Quoted 6 lines
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
@@ -531,6 +531,7 @@ fn notify_gsp(bar: &Bar0) {
     ///
     /// # Errors
     ///
+    /// - `EMSGSIZE` if the command exceeds the maximum queue element size.
     /// - `ETIMEDOUT` if space does not become available within the timeout.
     /// - `EIO` if the variable payload requested by the command has not been entirely

What's the benefit of enumerating all the error codes like this? Unless all the mentioned error code here is supposed to be handled, then it doesn't gain much for them to be mentioned, no?

For the errors that do need special handling, we probably want to use enums to force their handling.

Best,
Gary

Quoted 31 lines
drivers/gpu/nova-core/gsp/cmdq.rs +7 -9
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
@@ -711,22 +712,20 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
     /// Receive a message from the GSP.
     ///
-    /// `init` is a closure tasked with processing the message. It receives a reference to the
-    /// message in the message queue, and a [`SBufferIter`] pointing to its variable-length
-    /// payload, if any.
+    /// The expected message type is specified using the `M` generic parameter. If the pending
+    /// message has a different function code, `ERANGE` is returned and the message is consumed.
     ///
-    /// The expected message is specified using the `M` generic parameter. If the pending message
-    /// is different, `EAGAIN` is returned and the unexpected message is dropped.
-    ///
-    /// This design is by no means final, but it is simple and will let us go through GSP
-    /// initialization.
+    /// The read pointer is always advanced past the message, regardless of whether it matched.
     ///
     /// # Errors
     ///
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `EINVAL` if the function of the message was unrecognized.
+    /// - `EINVAL` if the function code of the message was not recognized.
+    /// - `ERANGE` if the message had a recognized but non-matching function code.
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     pub(crate) fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
     where
         // This allows all error types, including `Infallible`, to be used for `M::InitError`.
#3 lore raw

Re: [PATCH v3 1/5] gpu: nova-core: gsp: fix stale doc comments on command queue methods

Quoted 26 lines · Gary Guo
Quoted 14 lines · Eliot Courtney

Fix some inaccuracies / old doc comments.

---
 drivers/gpu/nova-core/gsp/cmdq.rs | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +0 -0
index 492e9489e808..4829830b6921 100644

drivers/gpu/nova-core/gsp/cmdq.rs fn notify_gsp(bar: &Bar0) { · line 536

Quoted 6 lines
drivers/gpu/nova-core/gsp/cmdq.rs +1 -0
@@ -531,6 +531,7 @@ fn notify_gsp(bar: &Bar0) {
     ///
     /// # Errors
     ///
+    /// - `EMSGSIZE` if the command exceeds the maximum queue element size.
     /// - `ETIMEDOUT` if space does not become available within the timeout.
     /// - `EIO` if the variable payload requested by the command has not been entirely

What's the benefit of enumerating all the error codes like this? Unless all the mentioned error code here is supposed to be handled, then it doesn't gain much for them to be mentioned, no?

Quoted 2 lines

For the errors that do need special handling, we probably want to use enums to force their handling.

Agreed about using enums, that's a change we will want to do driver-wide at some point. Preferably once we have an equivalent of thiserror to make the transition easier.

Meanwhile I think it is expected to document the returned error codes, as
(1) they provide an easy way to lookup the reason for runtime errors instead of grepping the code, and (2) they can eventually be converted into the doccomments of the enum error types once we switch to them.

#4 lore raw

PATCH v3 2/5 gpu: nova-core: gsp: add `RECEIVE_TIMEOUT` constant for command queue

In reply to Eliot Courtney

Remove magic numbers and add a default timeout for callers to use.

---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 3 +++
 drivers/gpu/nova-core/gsp/commands.rs  | 5 ++---
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +3 -0
index 4829830b6921..0192c85ddd75 100644
@@ -496,6 +496,9 @@ impl Cmdq {
     /// Timeout for waiting for space on the command queue.
     const ALLOCATE_TIMEOUT: Delta = Delta::from_secs(1);
 
+    /// Default timeout for receiving a message from the GSP.
+    pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(10);
+
     /// Creates a new command queue for `dev`.
     pub(crate) fn new(dev: &device::Device<device::Bound>) -> Result<Cmdq> {
         let gsp_mem = DmaGspMem::new(dev)?;
drivers/gpu/nova-core/gsp/commands.rs +2 -3
index 8f270eca33be..88df117ba575 100644
@@ -11,7 +11,6 @@
     device,
     pci,
     prelude::*,
-    time::Delta,
     transmute::{
         AsBytes,
         FromBytes, //
@@ -165,7 +164,7 @@ fn read(
 /// Waits for GSP initialization to complete.
 pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
     loop {
-        match cmdq.receive_msg::<GspInitDone>(Delta::from_secs(10)) {
+        match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
             Err(ERANGE) => continue,
             Err(e) => break Err(e),
@@ -235,7 +234,7 @@ pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticIn
     cmdq.send_command(bar, GetGspStaticInfo)?;
 
     loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Delta::from_secs(5)) {
+        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(info) => return Ok(info),
             Err(ERANGE) => continue,
             Err(e) => return Err(e),
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
index 0cfbedc47fcf..ce2b3bb05d22 100644
@@ -358,7 +358,7 @@ pub(crate) struct GspSequencerParams<'a> {
 impl<'a> GspSequencer<'a> {
     pub(crate) fn run(cmdq: &mut Cmdq, params: GspSequencerParams<'a>) -> Result {
         let seq_info = loop {
-            match cmdq.receive_msg::<GspSequence>(Delta::from_secs(10)) {
+            match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
                 Err(ERANGE) => continue,
                 Err(e) => return Err(e),
Signature · 1 line
2.53.0
reviewer Reviewed-by on v3 2/5
#5 lore raw

Re: [PATCH v3 2/5] gpu: nova-core: gsp: add `RECEIVE_TIMEOUT` constant for command queue

Quoted 4 lines · Eliot Courtney

Remove magic numbers and add a default timeout for callers to use.

Quoted 5 lines
---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 3 +++
 drivers/gpu/nova-core/gsp/commands.rs  | 5 ++---
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
#6 lore raw

Re: [PATCH v3 2/5] gpu: nova-core: gsp: add `RECEIVE_TIMEOUT` constant for command queue

In reply to Eliot Courtney
Quoted 15 lines · Eliot Courtney

Remove magic numbers and add a default timeout for callers to use.

---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 3 +++
 drivers/gpu/nova-core/gsp/commands.rs  | 5 ++---
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +0 -0
index 4829830b6921..0192c85ddd75 100644

drivers/gpu/nova-core/gsp/cmdq.rs impl Cmdq { · line 500

Quoted 5 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
@@ -496,6 +496,9 @@ impl Cmdq {
     /// Timeout for waiting for space on the command queue.
     const ALLOCATE_TIMEOUT: Delta = Delta::from_secs(1);
 
+    /// Default timeout for receiving a message from the GSP.
+    pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(10);

Agree with the idea, but let's settle on 5 seconds - GSP replies are supposed to be fast and there should be no good reason for any command to take longer than that (and potentially hang the whole system for the same time).

#7 lore raw

PATCH v3 3/5 gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

In reply to Eliot Courtney

Add type infrastructure to know what reply is expected from each CommandToGsp. Uses a marker type NoReply which does not implement MessageFromGsp to mark commands which don't expect a response.

Update send_command to wait for a reply and add send_command_no_wait which sends a command that has no reply, without blocking.

This prepares for adding locking to the queue.

---
 drivers/gpu/nova-core/gsp/boot.rs              |  5 ++-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 55 +++++++++++++++++++++++++-
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |  5 ++-
 drivers/gpu/nova-core/gsp/commands.rs          | 16 +++-----
 4 files changed, 67 insertions(+), 14 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +3 -2
index c56029f444cb..991eb5957e3d 100644
@@ -160,8 +160,9 @@ pub(crate) fn boot(
         dma_write!(wpr_meta[0] = GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
         self.cmdq
-            .send_command(bar, commands::SetSystemInfo::new(pdev))?;
-        self.cmdq.send_command(bar, commands::SetRegistry::new())?;
+            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev))?;
+        self.cmdq
+            .send_command_no_wait(bar, commands::SetRegistry::new())?;
 
         gsp_falcon.reset(bar)?;
         let libos_handle = self.libos.dma_handle();
drivers/gpu/nova-core/gsp/cmdq.rs +54 -1

79 lines not on this page · raw message

drivers/gpu/nova-core/gsp/cmdq/continuation.rs +4 -1
index 637942917237..8a6bb8fa7e60 100644
@@ -6,7 +6,7 @@
 
 use kernel::prelude::*;
 
-use super::CommandToGsp;
+use super::{CommandToGsp, NoReply};
 
 use crate::{
     gsp::fw::{
@@ -63,6 +63,7 @@ fn new(data: &'a [u8]) -> Self {
 impl<'a> CommandToGsp for ContinuationRecord<'a> {
     const FUNCTION: MsgFunction = MsgFunction::ContinuationRecord;
     type Command = ();
+    type Reply = NoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -144,6 +145,7 @@ fn new(command: C, payload: KVVec<u8>) -> Self {
 impl<C: CommandToGsp> CommandToGsp for SplitCommand<C> {
     const FUNCTION: MsgFunction = C::FUNCTION;
     type Command = C::Command;
+    type Reply = C::Reply;
     type InitError = C::InitError;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -206,6 +208,7 @@ fn new(len: usize) -> Result<Self> {
     impl CommandToGsp for TestPayload {
         const FUNCTION: MsgFunction = MsgFunction::Nop;
         type Command = TestHeader;
+        type Reply = NoReply;
         type InitError = Infallible;
 
         fn init(&self) -> impl Init<Self::Command, Self::InitError> {
drivers/gpu/nova-core/gsp/commands.rs +6 -10
index 88df117ba575..77054c92fcc2 100644
@@ -23,7 +23,8 @@
         cmdq::{
             Cmdq,
             CommandToGsp,
-            MessageFromGsp, //
+            MessageFromGsp,
+            NoReply, //
         },
         fw::{
             commands::*,
@@ -48,6 +49,7 @@ pub(crate) fn new(pdev: &'a pci::Device<device::Bound>) -> Self {
 impl<'a> CommandToGsp for SetSystemInfo<'a> {
     const FUNCTION: MsgFunction = MsgFunction::GspSetSystemInfo;
     type Command = GspSetSystemInfo;
+    type Reply = NoReply;
     type InitError = Error;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -99,6 +101,7 @@ pub(crate) fn new() -> Self {
 impl CommandToGsp for SetRegistry {
     const FUNCTION: MsgFunction = MsgFunction::SetRegistry;
     type Command = PackedRegistryTable;
+    type Reply = NoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -178,6 +181,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
 impl CommandToGsp for GetGspStaticInfo {
     const FUNCTION: MsgFunction = MsgFunction::GetGspStaticInfo;
     type Command = GspStaticConfigInfo;
+    type Reply = GetGspStaticInfoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
@@ -231,13 +235,5 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
 
 /// Send the [`GetGspInfo`] command and awaits for its reply.
 pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
-    cmdq.send_command(bar, GetGspStaticInfo)?;
-
-    loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(info) => return Ok(info),
-            Err(ERANGE) => continue,
-            Err(e) => return Err(e),
-        }
-    }
+    cmdq.send_command(bar, GetGspStaticInfo)
 }
Signature · 1 line
2.53.0
reviewer Reviewed-by on v3 3/5
#8 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 11 lines · Eliot Courtney

Add type infrastructure to know what reply is expected from each CommandToGsp. Uses a marker type NoReply which does not implement MessageFromGsp to mark commands which don't expect a response.

Update send_command to wait for a reply and add send_command_no_wait which sends a command that has no reply, without blocking.

This prepares for adding locking to the queue.

Quoted 6 lines
---
 drivers/gpu/nova-core/gsp/boot.rs              |  5 ++-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 55 +++++++++++++++++++++++++-
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |  5 ++-
 drivers/gpu/nova-core/gsp/commands.rs          | 16 +++-----
 4 files changed, 67 insertions(+), 14 deletions(-)
#9 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

In reply to Eliot Courtney
Quoted 39 lines · Eliot Courtney

Add type infrastructure to know what reply is expected from each CommandToGsp. Uses a marker type NoReply which does not implement MessageFromGsp to mark commands which don't expect a response.

Update send_command to wait for a reply and add send_command_no_wait which sends a command that has no reply, without blocking.

This prepares for adding locking to the queue.

---
 drivers/gpu/nova-core/gsp/boot.rs              |  5 ++-
 drivers/gpu/nova-core/gsp/cmdq.rs              | 55 +++++++++++++++++++++++++-
 drivers/gpu/nova-core/gsp/cmdq/continuation.rs |  5 ++-
 drivers/gpu/nova-core/gsp/commands.rs          | 16 +++-----
 4 files changed, 67 insertions(+), 14 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +3 -2
index c56029f444cb..991eb5957e3d 100644
@@ -160,8 +160,9 @@ pub(crate) fn boot(
         dma_write!(wpr_meta[0] = GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
         self.cmdq
-            .send_command(bar, commands::SetSystemInfo::new(pdev))?;
-        self.cmdq.send_command(bar, commands::SetRegistry::new())?;
+            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev))?;
+        self.cmdq
+            .send_command_no_wait(bar, commands::SetRegistry::new())?;
 
         gsp_falcon.reset(bar)?;
         let libos_handle = self.libos.dma_handle();
drivers/gpu/nova-core/gsp/cmdq.rs +0 -0
index 0192c85ddd75..7750f5792b21 100644

drivers/gpu/nova-core/gsp/cmdq.rs line 55

Quoted 5 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -0
@@ -51,6 +51,10 @@
     sbuffer::SBufferIter, //
 };
 
+/// Marker type representing the absence of a reply for a command. This does not implement
+/// `MessageFromGsp`.

This is giving either too much or not enough implementation detail. :)

Without knowing why MessageFromGsp is not implemented, this can be confusing to users who will wonder why we give them this information. I'd remove that sentence and instead say something like "commands using this as their reply type are sent using send_command_no_wait" or something like that.

Quoted 45 lines
drivers/gpu/nova-core/gsp/cmdq.rs +23 -1
+pub(crate) struct NoReply;
+
 /// Trait implemented by types representing a command to send to the GSP.
 ///
 /// The main purpose of this trait is to provide [`Cmdq::send_command`] with the information it
@@ -69,6 +73,10 @@ pub(crate) trait CommandToGsp {
     /// Type generated by [`CommandToGsp::init`], to be written into the command queue buffer.
     type Command: FromBytes + AsBytes;
 
+    /// Type of the reply expected from the GSP, or [`NoReply`] for commands that don't
+    /// have a reply.
+    type Reply;
+
     /// Error type returned by [`CommandToGsp::init`].
     type InitError;
 
@@ -610,7 +618,7 @@ fn send_single_command<M>(&mut self, bar: &Bar0, command: M) -> Result
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
     ///
     /// Error codes returned by the command initializers are propagated as-is.
-    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result
+    fn send_command_internal<M>(&mut self, bar: &Bar0, command: M) -> Result
     where
         M: CommandToGsp,
         Error: From<M::InitError>,
@@ -630,6 +638,51 @@ pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result
         }
     }
 
+    /// Sends `command` to the GSP and waits for the reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
+    ///   not received within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command and reply initializers are propagated as-is.
+    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result<M::Reply>
+    where
+        M: CommandToGsp,
+        M::Reply: MessageFromGsp,
+        Error: From<M::InitError>,
+        Error: From<<M::Reply as MessageFromGsp>::InitError>,

drivers/gpu/nova-core/gsp/cmdq.rs pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result · line 666

Quoted 10 lines
drivers/gpu/nova-core/gsp/cmdq.rs +10 -0
+    {
+        self.send_command_internal(bar, command)?;
+
+        loop {
+            match self.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
+                Ok(reply) => break Ok(reply),
+                Err(ERANGE) => continue,
+                Err(e) => break Err(e),
+            }
+        }

There is an opportunity for factorize some more code here.

Notice how the other callers of receive_msg (wait_gsp_init_done and GspSequencer::run) both use the same kind of loop, down to the same error handling. We could move that loop logic here and do it in a single place.

In the future, we will probably want to add handlers for unexpected messages from the GSP and it will be easier if we receive all messages from a single place.

This can be a separate patch from this one, but I think it makes sense to have that in this series.

I expect the last patch to change a bit as a consequence of that - maybe we will need a receive_msg_loop or something in CmdqInner.

Quoted 27 lines
drivers/gpu/nova-core/gsp/cmdq.rs +19 -0
+    }
+
+    /// Sends `command` to the GSP without waiting for a reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command initializers are propagated as-is.
+    pub(crate) fn send_command_no_wait<M>(&mut self, bar: &Bar0, command: M) -> Result
+    where
+        M: CommandToGsp<Reply = NoReply>,
+        Error: From<M::InitError>,
+    {
+        self.send_command_internal(bar, command)
+    }
+
     /// Wait for a message to become available on the message queue.
     ///
     /// This works purely at the transport layer and does not interpret or validate the message
drivers/gpu/nova-core/gsp/cmdq/continuation.rs +0 -0
index 637942917237..8a6bb8fa7e60 100644

drivers/gpu/nova-core/gsp/cmdq/continuation.rs line 9

Quoted 5 lines
drivers/gpu/nova-core/gsp/cmdq/continuation.rs +1 -1
@@ -6,7 +6,7 @@
 
 use kernel::prelude::*;
 
-use super::CommandToGsp;
+use super::{CommandToGsp, NoReply};

Nit: let's follow the formatting convention for imports.

#10 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 38 lines · Alexandre Courbot
Quoted 16 lines
drivers/gpu/nova-core/gsp/cmdq.rs +16 -0
+    /// Sends `command` to the GSP and waits for the reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
+    ///   not received within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command and reply initializers are propagated as-is.
+    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result<M::Reply>
+    where
+        M: CommandToGsp,
+        M::Reply: MessageFromGsp,
+        Error: From<M::InitError>,
+        Error: From<<M::Reply as MessageFromGsp>::InitError>,

drivers/gpu/nova-core/gsp/cmdq.rs pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result · line 666

Quoted 10 lines
drivers/gpu/nova-core/gsp/cmdq.rs +10 -0
+    {
+        self.send_command_internal(bar, command)?;
+
+        loop {
+            match self.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
+                Ok(reply) => break Ok(reply),
+                Err(ERANGE) => continue,
+                Err(e) => break Err(e),
+            }
+        }

There is an opportunity for factorize some more code here.

Notice how the other callers of receive_msg (wait_gsp_init_done and GspSequencer::run) both use the same kind of loop, down to the same error handling. We could move that loop logic here and do it in a single place.

In the future, we will probably want to add handlers for unexpected messages from the GSP and it will be easier if we receive all messages from a single place.

Quoted 5 lines

This can be a separate patch from this one, but I think it makes sense to have that in this series.

I expect the last patch to change a bit as a consequence of that - maybe we will need a receive_msg_loop or something in CmdqInner.

I agree we should migrate all callers and make Cmdq responsible for draining / handling spontaneous messages from the GSP, but I was planning on doing it in a separate patch series until now. I can put it into this one though if you want though no worries.

#11 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 42 lines · Eliot Courtney
Quoted 41 lines · Alexandre Courbot
Quoted 16 lines
drivers/gpu/nova-core/gsp/cmdq.rs +16 -0
+    /// Sends `command` to the GSP and waits for the reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
+    ///   not received within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command and reply initializers are propagated as-is.
+    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result<M::Reply>
+    where
+        M: CommandToGsp,
+        M::Reply: MessageFromGsp,
+        Error: From<M::InitError>,
+        Error: From<<M::Reply as MessageFromGsp>::InitError>,

drivers/gpu/nova-core/gsp/cmdq.rs pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result · line 666

Quoted 10 lines
drivers/gpu/nova-core/gsp/cmdq.rs +10 -0
+    {
+        self.send_command_internal(bar, command)?;
+
+        loop {
+            match self.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
+                Ok(reply) => break Ok(reply),
+                Err(ERANGE) => continue,
+                Err(e) => break Err(e),
+            }
+        }

There is an opportunity for factorize some more code here.

Notice how the other callers of receive_msg (wait_gsp_init_done and GspSequencer::run) both use the same kind of loop, down to the same error handling. We could move that loop logic here and do it in a single place.

In the future, we will probably want to add handlers for unexpected messages from the GSP and it will be easier if we receive all messages from a single place.

This can be a separate patch from this one, but I think it makes sense to have that in this series.

Quoted 7 lines
Quoted 2 lines

I expect the last patch to change a bit as a consequence of that - maybe we will need a receive_msg_loop or something in CmdqInner.

I agree we should migrate all callers and make Cmdq responsible for draining / handling spontaneous messages from the GSP, but I was planning on doing it in a separate patch series until now. I can put it into this one though if you want though no worries.

If it ends up being convulated, let's do that afterwards but since it looks like a quick and easy win I thought it would make sense to have it here. Your call though.

#12 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 46 lines · Alexandre Courbot
Quoted 45 lines · Eliot Courtney
Quoted 43 lines · Alexandre Courbot
Quoted 16 lines
drivers/gpu/nova-core/gsp/cmdq.rs +16 -0
+    /// Sends `command` to the GSP and waits for the reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
+    ///   not received within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command and reply initializers are propagated as-is.
+    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result<M::Reply>
+    where
+        M: CommandToGsp,
+        M::Reply: MessageFromGsp,
+        Error: From<M::InitError>,
+        Error: From<<M::Reply as MessageFromGsp>::InitError>,

drivers/gpu/nova-core/gsp/cmdq.rs pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result · line 666

Quoted 10 lines
drivers/gpu/nova-core/gsp/cmdq.rs +10 -0
+    {
+        self.send_command_internal(bar, command)?;
+
+        loop {
+            match self.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
+                Ok(reply) => break Ok(reply),
+                Err(ERANGE) => continue,
+                Err(e) => break Err(e),
+            }
+        }

There is an opportunity for factorize some more code here.

Notice how the other callers of receive_msg (wait_gsp_init_done and GspSequencer::run) both use the same kind of loop, down to the same error handling. We could move that loop logic here and do it in a single place.

In the future, we will probably want to add handlers for unexpected messages from the GSP and it will be easier if we receive all messages from a single place.

This can be a separate patch from this one, but I think it makes sense to have that in this series.

I expect the last patch to change a bit as a consequence of that - maybe we will need a receive_msg_loop or something in CmdqInner.

Quoted 8 lines
Quoted 4 lines

I agree we should migrate all callers and make Cmdq responsible for draining / handling spontaneous messages from the GSP, but I was planning on doing it in a separate patch series until now. I can put it into this one though if you want though no worries.

If it ends up being convulated, let's do that afterwards but since it looks like a quick and easy win I thought it would make sense to have it here. Your call though.

Another consideration is that if the GSP has some issue, it could cause this receive loop to run forever. I'm not sure if that practically can happen or if we want to guard against it, but personally I think we should and taking care of:

1. Some loop level timeout
2. Future considerations for how to handle spontaneous messages / draining
3. Migrating the callers

Seems large enough to do as a follow up series to me. But LMK if you feel otherwise. Thanks!

#13 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 61 lines · Eliot Courtney
Quoted 46 lines · Alexandre Courbot
Quoted 45 lines · Eliot Courtney
Quoted 43 lines · Alexandre Courbot
Quoted 16 lines
drivers/gpu/nova-core/gsp/cmdq.rs +16 -0
+    /// Sends `command` to the GSP and waits for the reply.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
+    ///   not received within the timeout.
+    /// - `EIO` if the variable payload requested by the command has not been entirely
+    ///   written to by its [`CommandToGsp::init_variable_payload`] method.
+    ///
+    /// Error codes returned by the command and reply initializers are propagated as-is.
+    pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result<M::Reply>
+    where
+        M: CommandToGsp,
+        M::Reply: MessageFromGsp,
+        Error: From<M::InitError>,
+        Error: From<<M::Reply as MessageFromGsp>::InitError>,

drivers/gpu/nova-core/gsp/cmdq.rs pub(crate) fn send_command<M>(&mut self, bar: &Bar0, command: M) -> Result · line 666

Quoted 10 lines
drivers/gpu/nova-core/gsp/cmdq.rs +10 -0
+    {
+        self.send_command_internal(bar, command)?;
+
+        loop {
+            match self.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
+                Ok(reply) => break Ok(reply),
+                Err(ERANGE) => continue,
+                Err(e) => break Err(e),
+            }
+        }

There is an opportunity for factorize some more code here.

Notice how the other callers of receive_msg (wait_gsp_init_done and GspSequencer::run) both use the same kind of loop, down to the same error handling. We could move that loop logic here and do it in a single place.

In the future, we will probably want to add handlers for unexpected messages from the GSP and it will be easier if we receive all messages from a single place.

This can be a separate patch from this one, but I think it makes sense to have that in this series.

I expect the last patch to change a bit as a consequence of that - maybe we will need a receive_msg_loop or something in CmdqInner.

Quoted 8 lines
Quoted 4 lines

I agree we should migrate all callers and make Cmdq responsible for draining / handling spontaneous messages from the GSP, but I was planning on doing it in a separate patch series until now. I can put it into this one though if you want though no worries.

If it ends up being convulated, let's do that afterwards but since it looks like a quick and easy win I thought it would make sense to have it here. Your call though.

Another consideration is that if the GSP has some issue, it could cause this receive loop to run forever. I'm not sure if that practically can happen or if we want to guard against it, but personally I think we should and taking care of:

Quoted 7 lines

1. Some loop level timeout
2. Future considerations for how to handle spontaneous messages / draining
3. Migrating the callers

Seems large enough to do as a follow up series to me. But LMK if you feel otherwise. Thanks!

Follow-up is fine, especially if the fix ends up being a bit more involved than I first expected.

#14 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

In reply to Eliot Courtney

On Wed Mar 4, 2026 at 11:46 AM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/commands.rs pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> { · line 238

Quoted 12 lines

/// Send the [GetGspInfo] command and awaits for its reply.
pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {

drivers/gpu/nova-core/gsp/commands.rs +1 -9
-    cmdq.send_command(bar, GetGspStaticInfo)?;
-
-    loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(info) => return Ok(info),
-            Err(ERANGE) => continue,
-            Err(e) => return Err(e),
-        }
-    }
+    cmdq.send_command(bar, GetGspStaticInfo)

Also noticed something cool while rebasing the unload series on top of this one: all these command helpers become one-liners and are not really needed anymore! Here we can just make GetGspStaticInfo public and call cmdq.send_command on it directly in boot. It removes a layer of black magic and makes it easier to understand what is going on.

#15 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 15 lines · Alexandre Courbot

On Wed Mar 4, 2026 at 11:46 AM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/commands.rs pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> { · line 238

Quoted 12 lines

/// Send the [GetGspInfo] command and awaits for its reply.
pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {

drivers/gpu/nova-core/gsp/commands.rs +1 -9
-    cmdq.send_command(bar, GetGspStaticInfo)?;
-
-    loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(info) => return Ok(info),
-            Err(ERANGE) => continue,
-            Err(e) => return Err(e),
-        }
-    }
+    cmdq.send_command(bar, GetGspStaticInfo)
Quoted 5 lines

Also noticed something cool while rebasing the unload series on top of this one: all these command helpers become one-liners and are not really needed anymore! Here we can just make GetGspStaticInfo public and call cmdq.send_command on it directly in boot. It removes a layer of black magic and makes it easier to understand what is going on.

I think this is ok, but we will still need some helpers (e.g. for RM control RPCs) that mimic the structure we have here. But there are a bunch of simple RPCs like this that can just be called without the helper. Do you think it's better to do this and have mixed helper vs direct call, or just go all helpers?

#16 lore raw

Re: [PATCH v3 3/5] gpu: nova-core: gsp: add reply/no-reply info to `CommandToGsp`

Quoted 22 lines · Eliot Courtney
Quoted 15 lines · Alexandre Courbot

On Wed Mar 4, 2026 at 11:46 AM JST, Eliot Courtney wrote:
<snip>

drivers/gpu/nova-core/gsp/commands.rs pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> { · line 238

Quoted 12 lines

/// Send the [GetGspInfo] command and awaits for its reply.
pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {

drivers/gpu/nova-core/gsp/commands.rs +1 -9
-    cmdq.send_command(bar, GetGspStaticInfo)?;
-
-    loop {
-        match cmdq.receive_msg::<GetGspStaticInfoReply>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(info) => return Ok(info),
-            Err(ERANGE) => continue,
-            Err(e) => return Err(e),
-        }
-    }
+    cmdq.send_command(bar, GetGspStaticInfo)
Quoted 5 lines

Also noticed something cool while rebasing the unload series on top of this one: all these command helpers become one-liners and are not really needed anymore! Here we can just make GetGspStaticInfo public and call cmdq.send_command on it directly in boot. It removes a layer of black magic and makes it easier to understand what is going on.

Quoted 5 lines

I think this is ok, but we will still need some helpers (e.g. for RM control RPCs) that mimic the structure we have here. But there are a bunch of simple RPCs like this that can just be called without the helper. Do you think it's better to do this and have mixed helper vs direct call, or just go all helpers?

Mmm ok, let's keep the current helpers and reconsider once we have RM controls and more visibility.

#17 lore raw

PATCH v3 4/5 gpu: nova-core: gsp: make `Cmdq` a pinned type

In reply to Eliot Courtney

Make Cmdq a pinned type. This is needed to use Mutex, which is needed to add locking to Cmdq.

---
 drivers/gpu/nova-core/gsp.rs      | 5 +++--
 drivers/gpu/nova-core/gsp/cmdq.rs | 9 ++++-----
 2 files changed, 7 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/gsp.rs +3 -2
index 174feaca0a6b..a6f3918c20b1 100644
@@ -112,6 +112,7 @@ pub(crate) struct Gsp {
     /// RM log buffer.
     logrm: LogBuffer,
     /// Command queue.
+    #[pin]
     pub(crate) cmdq: Cmdq,
     /// RM arguments.
     rmargs: CoherentAllocation<GspArgumentsPadded>,
@@ -132,7 +133,7 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
                 loginit: LogBuffer::new(dev)?,
                 logintr: LogBuffer::new(dev)?,
                 logrm: LogBuffer::new(dev)?,
-                cmdq: Cmdq::new(dev)?,
+                cmdq <- Cmdq::new(dev),
                 rmargs: CoherentAllocation::<GspArgumentsPadded>::alloc_coherent(
                     dev,
                     1,
@@ -149,7 +150,7 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
                         libos[1] = LibosMemoryRegionInitArgument::new("LOGINTR", &logintr.0)
                     )?;
                     dma_write!(libos[2] = LibosMemoryRegionInitArgument::new("LOGRM", &logrm.0))?;
-                    dma_write!(rmargs[0].inner = fw::GspArgumentsCached::new(cmdq))?;
+                    dma_write!(rmargs[0].inner = fw::GspArgumentsCached::new(&cmdq))?;
                     dma_write!(libos[3] = LibosMemoryRegionInitArgument::new("RMARGS", rmargs))?;
                 },
             }))
drivers/gpu/nova-core/gsp/cmdq.rs +4 -5
index 7750f5792b21..de84a298909f 100644
@@ -475,6 +475,7 @@ struct GspMessage<'a> {
 ///
 /// Provides the ability to send commands and receive messages from the GSP using a shared memory
 /// area.
+#[pin_data]
 pub(crate) struct Cmdq {
     /// Device this command queue belongs to.
     dev: ARef<device::Device>,
@@ -508,13 +509,11 @@ impl Cmdq {
     pub(super) const RECEIVE_TIMEOUT: Delta = Delta::from_secs(10);
 
     /// Creates a new command queue for `dev`.
-    pub(crate) fn new(dev: &device::Device<device::Bound>) -> Result<Cmdq> {
-        let gsp_mem = DmaGspMem::new(dev)?;
-
-        Ok(Cmdq {
+    pub(crate) fn new(dev: &device::Device<device::Bound>) -> impl PinInit<Self, Error> + '_ {
+        try_pin_init!(Self {
+            gsp_mem: DmaGspMem::new(dev)?,
             dev: dev.into(),
             seq: 0,
-            gsp_mem,
         })
     }
 
Signature · 1 line
2.53.0
#18 lore raw

PATCH v3 5/5 gpu: nova-core: gsp: add mutex locking to Cmdq

In reply to Eliot Courtney

Wrap Cmdq's mutable state in a new struct CmdqInner and wrap that in a Mutex. This lets Cmdq methods take &self instead of &mut self, which lets required commands be sent e.g. while unloading the driver.

The mutex is held over both send and receive in send_command to make sure that it doesn't get the reply of some other command that could have been sent just beforehand.

---
 drivers/gpu/nova-core/gsp/boot.rs      |   8 +-
 drivers/gpu/nova-core/gsp/cmdq.rs      | 247 +++++++++++++++++++--------------
 drivers/gpu/nova-core/gsp/commands.rs  |   4 +-
 drivers/gpu/nova-core/gsp/sequencer.rs |   2 +-
 4 files changed, 146 insertions(+), 115 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +4 -4
index 991eb5957e3d..bc53e667cd9e 100644
@@ -128,7 +128,7 @@ fn run_fwsec_frts(
     ///
     /// Upon return, the GSP is up and running, and its runtime object given as return value.
     pub(crate) fn boot(
-        mut self: Pin<&mut Self>,
+        self: Pin<&mut Self>,
         pdev: &pci::Device<device::Bound>,
         bar: &Bar0,
         chipset: Chipset,
@@ -214,13 +214,13 @@ pub(crate) fn boot(
             dev: pdev.as_ref().into(),
             bar,
         };
-        GspSequencer::run(&mut self.cmdq, seq_params)?;
+        GspSequencer::run(&self.cmdq, seq_params)?;
 
         // Wait until GSP is fully initialized.
-        commands::wait_gsp_init_done(&mut self.cmdq)?;
+        commands::wait_gsp_init_done(&self.cmdq)?;
 
         // Obtain and display basic GPU information.
-        let info = commands::get_gsp_info(&mut self.cmdq, bar)?;
+        let info = commands::get_gsp_info(&self.cmdq, bar)?;
         match info.gpu_name() {
             Ok(name) => dev_info!(pdev, "GPU name: {}\n", name),
             Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e),
drivers/gpu/nova-core/gsp/cmdq.rs +139 -108

313 lines not on this page · raw message

drivers/gpu/nova-core/gsp/commands.rs +2 -2
index 77054c92fcc2..c89c7b57a751 100644
@@ -165,7 +165,7 @@ fn read(
 }
 
 /// Waits for GSP initialization to complete.
-pub(crate) fn wait_gsp_init_done(cmdq: &mut Cmdq) -> Result {
+pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq) -> Result {
     loop {
         match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
@@ -234,6 +234,6 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
 }
 
 /// Send the [`GetGspInfo`] command and awaits for its reply.
-pub(crate) fn get_gsp_info(cmdq: &mut Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
+pub(crate) fn get_gsp_info(cmdq: &Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
     cmdq.send_command(bar, GetGspStaticInfo)
 }
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
index ce2b3bb05d22..474e4c8021db 100644
@@ -356,7 +356,7 @@ pub(crate) struct GspSequencerParams<'a> {
 }
 
 impl<'a> GspSequencer<'a> {
-    pub(crate) fn run(cmdq: &mut Cmdq, params: GspSequencerParams<'a>) -> Result {
+    pub(crate) fn run(cmdq: &Cmdq, params: GspSequencerParams<'a>) -> Result {
         let seq_info = loop {
             match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
Signature · 1 line
2.53.0
#19 lore raw

Re: [PATCH v3 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 11 lines · Eliot Courtney

Wrap Cmdq's mutable state in a new struct CmdqInner and wrap that in a Mutex. This lets Cmdq methods take &self instead of &mut self, which lets required commands be sent e.g. while unloading the driver.

The mutex is held over both send and receive in send_command to make sure that it doesn't get the reply of some other command that could have been sent just beforehand.

Quoted 6 lines
---
 drivers/gpu/nova-core/gsp/boot.rs      |   8 +-
 drivers/gpu/nova-core/gsp/cmdq.rs      | 247 +++++++++++++++++++--------------
 drivers/gpu/nova-core/gsp/commands.rs  |   4 +-
 drivers/gpu/nova-core/gsp/sequencer.rs |   2 +-
 4 files changed, 146 insertions(+), 115 deletions(-)

Most of the diff is code moving around. I've been able to reduce it to +107 -76 by moving CmdqInner *after* Cmdq - this makes this patch easier to review imho.

#20 lore raw

Re: [PATCH v3 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 19 lines · Alexandre Courbot
Quoted 11 lines · Eliot Courtney

Wrap Cmdq's mutable state in a new struct CmdqInner and wrap that in a Mutex. This lets Cmdq methods take &self instead of &mut self, which lets required commands be sent e.g. while unloading the driver.

The mutex is held over both send and receive in send_command to make sure that it doesn't get the reply of some other command that could have been sent just beforehand.

Quoted 6 lines
---
 drivers/gpu/nova-core/gsp/boot.rs      |   8 +-
 drivers/gpu/nova-core/gsp/cmdq.rs      | 247 +++++++++++++++++++--------------
 drivers/gpu/nova-core/gsp/commands.rs  |   4 +-
 drivers/gpu/nova-core/gsp/sequencer.rs |   2 +-
 4 files changed, 146 insertions(+), 115 deletions(-)
Quoted 3 lines

Most of the diff is code moving around. I've been able to reduce it to +107 -76 by moving CmdqInner *after* Cmdq - this makes this patch easier to review imho.

Thanks, will move this then since I think the ordering here is probably not too important to readability.

#21 lore raw

Re: [PATCH v3 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

Quoted 20 lines · Eliot Courtney
Quoted 19 lines · Alexandre Courbot
Quoted 17 lines · Eliot Courtney

Wrap Cmdq's mutable state in a new struct CmdqInner and wrap that in a Mutex. This lets Cmdq methods take &self instead of &mut self, which lets required commands be sent e.g. while unloading the driver.

The mutex is held over both send and receive in send_command to make sure that it doesn't get the reply of some other command that could have been sent just beforehand.

---
 drivers/gpu/nova-core/gsp/boot.rs      |   8 +-
 drivers/gpu/nova-core/gsp/cmdq.rs      | 247 +++++++++++++++++++--------------
 drivers/gpu/nova-core/gsp/commands.rs  |   4 +-
 drivers/gpu/nova-core/gsp/sequencer.rs |   2 +-
 4 files changed, 146 insertions(+), 115 deletions(-)
Quoted 6 lines
Quoted 3 lines

Most of the diff is code moving around. I've been able to reduce it to +107 -76 by moving CmdqInner *after* Cmdq - this makes this patch easier to review imho.

Thanks, will move this then since I think the ordering here is probably not too important to readability.

Yes, actually one could even argue in favor of having the public struct before its inner private counterpart for readability purposes.

#22 lore raw

Re: [PATCH v3 5/5] gpu: nova-core: gsp: add mutex locking to Cmdq

In reply to Eliot Courtney
Quoted 19 lines · Eliot Courtney
drivers/gpu/nova-core/gsp/cmdq.rs +5 -1
index de84a298909f..94cb2aa6568d 100644
@@ -18,8 +18,12 @@
     },
     dma_write,
     io::poll::read_poll_timeout,
+    new_mutex,
     prelude::*,
-    sync::aref::ARef,
+    sync::{
+        aref::ARef,
+        Mutex, //
+    },
     time::Delta,
     transmute::{
         AsBytes,

drivers/gpu/nova-core/gsp/cmdq.rs line 65

Quoted 7 lines
drivers/gpu/nova-core/gsp/cmdq.rs +2 -2
@@ -57,8 +61,8 @@
 
 /// Trait implemented by types representing a command to send to the GSP.
 ///
-/// The main purpose of this trait is to provide [`Cmdq::send_command`] with the information it
-/// needs to send a given command.
+/// The main purpose of this trait is to provide [`Cmdq`] with the information it needs to send
+/// a given command.

This looks unrelated - should this chunk be merged with the reply/no_reply patch?

#23 lore raw
In reply to Eliot Courtney
Quoted 15 lines · Eliot Courtney

Add locking to Cmdq. This is required e.g. for unloading the driver, which needs to send the UnloadingGuestDriver via the command queue on unbind which may be on a different thread.

We have commands that need a reply and commands that don't. For commands with a reply we want to make sure that they don't get the reply of a different command back. The approach this patch series takes is by making those commands block until they get a response. For now this should be ok, and we expect GSP to be fast anyway.

To do this, we need to know which commands expect a reply and which don't. John's existing series[1] adds IS_ASYNC which solves part of the problem, but we need to know a bit more. So instead, add an associated type called Reply which tells us what the reply is.

Quoted 5 lines

An alternative would be to define traits inheriting CommandToGsp, e.g. CommandWithReply and CommandWithoutReply, instead of using the associated type. I implemented the associated type version because it feels more compositional rather than inherity so seemed a bit better to me. But both of these approaches work and are fine, IMO.

The associated type seems to work just fine. I was wondering whether we could mess with it by using other types that do not implement MessageFromGsp, but in that case we cannot call any of the send_command methods so this approach looks good to me.