Skip to contentSkip to conversation

gpu: nova-core: run unload sequence upon unbinding

PATCH Applied Alexandre Courbot 2 participants 13 messages May 21 – May 29

Subsystems (5)

v6 of 7 Applied Latest is v7 →

lore
Share & tools
lore

Alexandre Courbot

author maintainer
#0 lore raw

v6 cover gpu: nova-core: run unload sequence upon unbinding

Currently the GSP is left running and the WPR2 memory region untouched when the driver is unbound. This is obviously not ideal for at least two reasons:

- Probing requires setting up the WPR2 region, which cannot be done if there is already one in place. Hence the current requirement to reset the GPU (using e.g. echo 1 >/sys/bus/pci/devices/.../reset) before the driver can be probed again after removal.
- The running GSP may still attempt to access shared memory regions which the kernel might recycle.

On top of that, there is a nasty bug in the Blackwell VBIOS that sometimes borks the GPU upon PCI reset, requiring a reboot. So relying on the PCI reset to unload/reload Nova is really not practical here.

This series does what is needed to leave the GPU in a clean state after unbind, for all currently supported GPUs. Blackwell support is basic and will be added alongside the Blackwell series if this can be merged first.

This revision rebases on top of the Device HRT series [1] and moves the unload bundle from Gsp into NovaCore. This makes GSP unload initiation only possible at the driver module level, which is the only place that can consume the unload bundle.

A branch with the series and its required dependencies is available at [2].

[1] https://lore.kernel.org/[email protected]
[2] https://github.com/Gnurou/linux/tree/b4/nova-unload

---
Changes in v6:
- Inline TU102 local run_booter method in its unique call site.
- Rename unload bundle field to unload_bundle.
- Make Sec2UnloadBundle private.
- Continue GSP teardown upon partial failure.
- Store the unload bundle into NovaCore.
- Take the unload bundle by value to make it one-shot.
- Link to v5: https://patch.msgid.link/[email protected]

Changes in v5:
- Rebase on top of the Device HRT series.
- Drop the now unneeded "gpu: nova-core: split BAR acquisition in unbind()".
- Link to v4: https://patch.msgid.link/[email protected]

Changes in v4:
- Remove warn_on_err macro as it isn't performing as expected and distracts from the goal of the series.
- Add John's patch from the Blackwell series refactoring the Booter
Loader runner code.
- Add a GSP HAL and move the existing TU102/SEC2 boot sequence into it
in preparation for the Hopper/Blackwell FSP boot path.
- Prepare the firmware required for unloading at probe time and save it into an unload bundle, as we cannot guarantee filesystem access at unload time.
- Constrain UNLOADING_GUEST_DRIVER's visibility to the parent module.
- Also write the sentinel value 0xff into mbox1 when running Booter Unloader to align with OpenRM.
- Link to v3: https://patch.msgid.link/[email protected]

Changes in v3:
- Disambiguate doccomment for warn_on_err.
- Test the correct bit instead of the whole register value to determine that the GSP has stopped.
- Use an enum instead of a boolean to encode the power level when
shutting down the GSP.
- Add missing newline to dev_err.
- Add missing doccomments for new types.
- Use values from bindings instead of magic numbers.
- Remove the redundant get_gsp_info function.
- Better document Booter Unloader mailbox sentinel value, and check the value of mbox0 upon return.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Rebase on top of master and remove unneeded/obsolete preparatory patches.
- Tidy up the imports of commands from the fw module in the gsp module.
- Link to v1: https://patch.msgid.link/[email protected]

---
Alexandre Courbot (6):
gpu: nova-core: remove unneeded get_gsp_info proxy function
gpu: nova-core: do not import firmware commands into GSP command module
gpu: nova-core: send UNLOADING_GUEST_DRIVER GSP command upon unloading
gpu: nova-core: gsp: shuffle boot code a bit to keep chipset-specific parts close
gpu: nova-core: gsp: move chipset-specific parts of the boot process into a HAL
gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

John Hubbard (1):
gpu: nova-core: refactor SEC2 booter loading into BooterFirmware::run()

 drivers/gpu/nova-core/driver.rs                   |  23 +-
 drivers/gpu/nova-core/firmware/booter.rs          |  32 +-
 drivers/gpu/nova-core/firmware/fwsec.rs           |   1 -
 drivers/gpu/nova-core/gpu.rs                      |  38 ++-
 drivers/gpu/nova-core/gsp.rs                      |   4 +
 drivers/gpu/nova-core/gsp/boot.rs                 | 262 ++++++-----------
 drivers/gpu/nova-core/gsp/commands.rs             |  72 +++--
 drivers/gpu/nova-core/gsp/fw.rs                   |   4 +
 drivers/gpu/nova-core/gsp/fw/commands.rs          |  45 +++
 drivers/gpu/nova-core/gsp/fw/r570_144/bindings.rs |  11 +
 drivers/gpu/nova-core/gsp/hal.rs                  |  93 ++++++
 drivers/gpu/nova-core/gsp/hal/gh100.rs            |  53 ++++
 drivers/gpu/nova-core/gsp/hal/tu102.rs            | 341 ++++++++++++++++++++++
 drivers/gpu/nova-core/regs.rs                     |   5 +
 14 files changed, 783 insertions(+), 201 deletions(-)

---
base-commit: 293c8393b49c9fc017168ddb46aa2012d508c921
change-id: 20251216-nova-unload-4029b3b76950

Best regards,

Signature · 1 line
Alexandre Courbot <[email protected]>
author maintainer
#1 lore raw

PATCH v6 1/7 gpu: nova-core: remove unneeded get_gsp_info proxy function

This function was useful before the generic command-queue send methods got merged, but it is just boilerplate now. Replace it with the correct sequence to queue the GetGspStaticInfo command directly.

---
 drivers/gpu/nova-core/gsp/boot.rs     | 2 +-
 drivers/gpu/nova-core/gsp/commands.rs | 8 +-------
 2 files changed, 2 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +1 -1
index df105ef4b371..e838d61bef50 100644
@@ -243,7 +243,7 @@ pub(crate) fn boot(
         commands::wait_gsp_init_done(&self.cmdq)?;
 
         // Obtain and display basic GPU information.
-        let info = commands::get_gsp_info(&self.cmdq, bar)?;
+        let info = self.cmdq.send_command(bar, commands::GetGspStaticInfo)?;
         match info.gpu_name() {
             Ok(name) => dev_info!(pdev, "GPU name: {}\n", name),
             Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e),
drivers/gpu/nova-core/gsp/commands.rs +1 -7
index 0da5b92f4b27..74a8a79bd2d6 100644
@@ -18,7 +18,6 @@
 };
 
 use crate::{
-    driver::Bar0,
     gsp::{
         cmdq::{
             Cmdq,
@@ -176,7 +175,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq) -> Result {
 }
 
 /// The `GetGspStaticInfo` command.
-struct GetGspStaticInfo;
+pub(crate) struct GetGspStaticInfo;
 
 impl CommandToGsp for GetGspStaticInfo {
     const FUNCTION: MsgFunction = MsgFunction::GetGspStaticInfo;
@@ -232,8 +231,3 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
             .map_err(GpuNameError::InvalidUtf8)
     }
 }
-
-/// Send the [`GetGspInfo`] command and awaits for its reply.
-pub(crate) fn get_gsp_info(cmdq: &Cmdq, bar: &Bar0) -> Result<GetGspStaticInfoReply> {
-    cmdq.send_command(bar, GetGspStaticInfo)
-}
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v6 2/7 gpu: nova-core: do not import firmware commands into GSP command module

In reply to Alexandre Courbot

Importing all the firmware commands like we did is a bit confusing, as the layer of a command type (fw or GSP) cannot be inferred from looking at its name alone. Furthermore it makes it impossible to create commands that have the same name as their firmware command.

Thus, stop importing all commands and refer to them from the fw module instead.

---
 drivers/gpu/nova-core/gsp/commands.rs | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/gsp/commands.rs +13 -12
index 74a8a79bd2d6..cc8df448ff39 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
 use core::{
     array,
@@ -26,7 +27,7 @@
             NoReply, //
         },
         fw::{
-            commands::*,
+            self,
             MsgFunction, //
         },
     },
@@ -47,12 +48,12 @@ pub(crate) fn new(pdev: &'bound pci::Device<device::Bound>) -> Self {
 
 impl<'bound> CommandToGsp for SetSystemInfo<'bound> {
     const FUNCTION: MsgFunction = MsgFunction::GspSetSystemInfo;
-    type Command = GspSetSystemInfo;
+    type Command = fw::commands::GspSetSystemInfo;
     type Reply = NoReply;
     type InitError = Error;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
-        GspSetSystemInfo::init(self.pdev)
+        Self::Command::init(self.pdev)
     }
 }
 
@@ -99,12 +100,12 @@ pub(crate) fn new() -> Self {
 
 impl CommandToGsp for SetRegistry {
     const FUNCTION: MsgFunction = MsgFunction::SetRegistry;
-    type Command = PackedRegistryTable;
+    type Command = fw::commands::PackedRegistryTable;
     type Reply = NoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
-        PackedRegistryTable::init(Self::NUM_ENTRIES as u32, self.variable_payload_len() as u32)
+        Self::Command::init(Self::NUM_ENTRIES as u32, self.variable_payload_len() as u32)
     }
 
     fn variable_payload_len(&self) -> usize {
@@ -112,22 +113,22 @@ fn variable_payload_len(&self) -> usize {
         for i in 0..Self::NUM_ENTRIES {
             key_size += self.entries[i].key.len() + 1; // +1 for NULL terminator
         }
-        Self::NUM_ENTRIES * size_of::<PackedRegistryEntry>() + key_size
+        Self::NUM_ENTRIES * size_of::<fw::commands::PackedRegistryEntry>() + key_size
     }
 
     fn init_variable_payload(
         &self,
         dst: &mut SBufferIter<core::array::IntoIter<&mut [u8], 2>>,
     ) -> Result {
-        let string_data_start_offset =
-            size_of::<PackedRegistryTable>() + Self::NUM_ENTRIES * size_of::<PackedRegistryEntry>();
+        let string_data_start_offset = size_of::<Self::Command>()
+            + Self::NUM_ENTRIES * size_of::<fw::commands::PackedRegistryEntry>();
 
         // Array for string data.
         let mut string_data = KVec::new();
 
         for entry in self.entries.iter().take(Self::NUM_ENTRIES) {
             dst.write_all(
-                PackedRegistryEntry::new(
+                fw::commands::PackedRegistryEntry::new(
                     (string_data_start_offset + string_data.len()) as u32,
                     entry.value,
                 )
@@ -179,12 +180,12 @@ pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq) -> Result {
 
 impl CommandToGsp for GetGspStaticInfo {
     const FUNCTION: MsgFunction = MsgFunction::GetGspStaticInfo;
-    type Command = GspStaticConfigInfo;
+    type Command = fw::commands::GspStaticConfigInfo;
     type Reply = GetGspStaticInfoReply;
     type InitError = Infallible;
 
     fn init(&self) -> impl Init<Self::Command, Self::InitError> {
-        GspStaticConfigInfo::init_zeroed()
+        Self::Command::init_zeroed()
     }
 }
 
@@ -195,7 +196,7 @@ pub(crate) struct GetGspStaticInfoReply {
 
 impl MessageFromGsp for GetGspStaticInfoReply {
     const FUNCTION: MsgFunction = MsgFunction::GetGspStaticInfo;
-    type Message = GspStaticConfigInfo;
+    type Message = fw::commands::GspStaticConfigInfo;
     type InitError = Infallible;
 
     fn read(
Signature · 1 line
2.54.0
author maintainer
#3 lore raw

PATCH v6 3/7 gpu: nova-core: send UNLOADING_GUEST_DRIVER GSP command upon unloading

In reply to Alexandre Courbot

Currently, the GSP is left running after the driver is unbound. This is not great for several reasons, notably that it can still access shared memory areas that the kernel will now reclaim (especially problematic on setups without an IOMMU).

Fix this by sending the UNLOADING_GUEST_DRIVER GSP command when unbinding. This stops the GSP and lets us proceed with the rest of the unbind sequence in a later patch.

We make use of the pci::Driver::unbind() hook as we need access to the device in order to properly unbind.

---
 drivers/gpu/nova-core/driver.rs                   |  7 ++++
 drivers/gpu/nova-core/gpu.rs                      |  7 ++++
 drivers/gpu/nova-core/gsp/boot.rs                 | 45 +++++++++++++++++++++++
 drivers/gpu/nova-core/gsp/commands.rs             | 43 ++++++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw.rs                   |  4 ++
 drivers/gpu/nova-core/gsp/fw/commands.rs          | 45 +++++++++++++++++++++++
 drivers/gpu/nova-core/gsp/fw/r570_144/bindings.rs | 11 ++++++
 7 files changed, 162 insertions(+)
drivers/gpu/nova-core/driver.rs +7 -0
index 6571a8c59d09..ced1d38d206a 100644
@@ -114,4 +114,11 @@ fn probe<'bound>(
             }))
         })
     }
+
+    fn unbind<'bound>(
+        dev: &'bound pci::Device<kernel::device::Core>,
+        this: Pin<&'bound Self::Data<'bound>>,
+    ) {
+        this.gpu.unbind(dev)
+    }
 }
drivers/gpu/nova-core/gpu.rs +7 -0
index 4c329ab40955..75fe1bdb80fe 100644
@@ -293,4 +293,11 @@ pub(crate) fn new(
             _: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? },
         })
     }
+
+    pub(crate) fn unbind(&self, pdev: &'bound pci::Device<device::Core>) {
+        let _ = self
+            .gsp
+            .unload(pdev.as_ref(), self.bar, &self.gsp_falcon)
+            .inspect_err(|e| dev_err!(pdev, "failed to unload GSP: {:?}\n", e));
+    }
 }
drivers/gpu/nova-core/gsp/boot.rs +45 -0

61 lines not on this page · raw message

drivers/gpu/nova-core/gsp/commands.rs +43 -0

46 lines not on this page · raw message

drivers/gpu/nova-core/gsp/fw.rs +4 -0
index 3245793bbe42..33c9f5860771 100644
@@ -279,6 +279,7 @@ pub(crate) enum MsgFunction {
     Nop = bindings::NV_VGPU_MSG_FUNCTION_NOP,
     SetGuestSystemInfo = bindings::NV_VGPU_MSG_FUNCTION_SET_GUEST_SYSTEM_INFO,
     SetRegistry = bindings::NV_VGPU_MSG_FUNCTION_SET_REGISTRY,
+    UnloadingGuestDriver = bindings::NV_VGPU_MSG_FUNCTION_UNLOADING_GUEST_DRIVER,
 
     // Event codes
     GspInitDone = bindings::NV_VGPU_MSG_EVENT_GSP_INIT_DONE,
@@ -323,6 +324,9 @@ fn try_from(value: u32) -> Result<MsgFunction> {
                 Ok(MsgFunction::SetGuestSystemInfo)
             }
             bindings::NV_VGPU_MSG_FUNCTION_SET_REGISTRY => Ok(MsgFunction::SetRegistry),
+            bindings::NV_VGPU_MSG_FUNCTION_UNLOADING_GUEST_DRIVER => {
+                Ok(MsgFunction::UnloadingGuestDriver)
+            }
 
             // Event codes
             bindings::NV_VGPU_MSG_EVENT_GSP_INIT_DONE => Ok(MsgFunction::GspInitDone),
drivers/gpu/nova-core/gsp/fw/commands.rs +45 -0

52 lines not on this page · raw message

drivers/gpu/nova-core/gsp/fw/r570_144/bindings.rs +11 -0
index 334e8be5fde8..f82ed097b283 100644
@@ -30,6 +30,9 @@ fn fmt(&self, fmt: &mut ::core::fmt::Formatter<'_>) -> ::core::fmt::Result {
         fmt.write_str("__IncompleteArrayField")
     }
 }
+pub const NV2080_CTRL_GPU_SET_POWER_STATE_GPU_LEVEL_0: u32 = 0;
+pub const NV2080_CTRL_GPU_SET_POWER_STATE_GPU_LEVEL_3: u32 = 3;
+pub const NV2080_CTRL_GPU_SET_POWER_STATE_GPU_LEVEL_7: u32 = 7;
 pub const NV_VGPU_MSG_SIGNATURE_VALID: u32 = 1129337430;
 pub const GSP_FW_HEAP_PARAM_OS_SIZE_LIBOS2: u32 = 0;
 pub const GSP_FW_HEAP_PARAM_OS_SIZE_LIBOS3_BAREMETAL: u32 = 23068672;
@@ -880,6 +883,14 @@ fn default() -> Self {
     }
 }
 #[repr(C)]
+#[derive(Debug, Default, Copy, Clone, MaybeZeroable)]
+pub struct rpc_unloading_guest_driver_v1F_07 {
+    pub bInPMTransition: u8_,
+    pub bGc6Entering: u8_,
+    pub __bindgen_padding_0: [u8; 2usize],
+    pub newLevel: u32_,
+}
+#[repr(C)]
 #[derive(Debug, Default, MaybeZeroable)]
 pub struct rpc_run_cpu_sequencer_v17_00 {
     pub bufferSizeDWord: u32_,
Signature · 1 line
2.54.0
author maintainer
#4 lore raw

PATCH v6 4/7 gpu: nova-core: refactor SEC2 booter loading into BooterFirmware::run()

In reply to Alexandre Courbot

From: John Hubbard <[email protected]>

Move the SEC2 reset/load/boot sequence into a BooterFirmware::run() method This is mostly refactoring, with no significant behavior change, done in preparation for adding an alternative FSP boot path.

---
 drivers/gpu/nova-core/firmware/booter.rs | 31 +++++++++++++++++++++++++++++++
 drivers/gpu/nova-core/gsp/boot.rs        | 30 ++++++++----------------------
 2 files changed, 39 insertions(+), 22 deletions(-)
drivers/gpu/nova-core/firmware/booter.rs +31 -0
index de2a4536b532..e45e5dc8d5d2 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
 //! Support for loading and patching the `Booter` firmware. `Booter` is a Heavy Secured firmware
 //! running on [`Sec2`], that is used on Turing/Ampere to load the GSP firmware into the GSP falcon
@@ -8,6 +9,7 @@
 
 use kernel::{
     device,
+    dma::Coherent,
     prelude::*,
     transmute::FromBytes, //
 };
@@ -396,6 +398,35 @@ pub(crate) fn new(
             ucode: ucode_signed,
         })
     }
+
+    /// Load and run the booter firmware on SEC2.
+    ///
+    /// Resets SEC2, loads this firmware image, then boots with the WPR metadata
+    /// address passed via the SEC2 mailboxes.
+    pub(crate) fn run<T>(
+        &self,
+        dev: &device::Device<device::Bound>,
+        bar: &Bar0,
+        sec2_falcon: &Falcon<Sec2>,
+        wpr_meta: &Coherent<T>,
+    ) -> Result {
+        sec2_falcon.reset(bar)?;
+        sec2_falcon.load(dev, bar, self)?;
+        let wpr_handle = wpr_meta.dma_handle();
+        let (mbox0, mbox1) = sec2_falcon.boot(
+            bar,
+            Some(wpr_handle as u32),
+            Some((wpr_handle >> 32) as u32),
+        )?;
+        dev_dbg!(dev, "SEC2 MBOX0: {:#x}, MBOX1: {:#x}\n", mbox0, mbox1);
+
+        if mbox0 != 0 {
+            dev_err!(dev, "Booter-load failed with error {:#x}\n", mbox0);
+            return Err(ENODEV);
+        }
+
+        Ok(())
+    }
 }
 
 impl FalconDmaLoadable for BooterFirmware {
drivers/gpu/nova-core/gsp/boot.rs +8 -22
index c123080148f1..4f654e10259a 100644
@@ -174,15 +174,6 @@ pub(crate) fn boot(
             Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
         }
 
-        let booter_loader = BooterFirmware::new(
-            dev,
-            BooterKind::Loader,
-            chipset,
-            FIRMWARE_VERSION,
-            sec2_falcon,
-            bar,
-        )?;
-
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
         self.cmdq
@@ -204,20 +195,15 @@ pub(crate) fn boot(
             "Using SEC2 to load and run the booter_load firmware...\n"
         );
 
-        sec2_falcon.reset(bar)?;
-        sec2_falcon.load(dev, bar, &booter_loader)?;
-        let wpr_handle = wpr_meta.dma_handle();
-        let (mbox0, mbox1) = sec2_falcon.boot(
+        BooterFirmware::new(
+            dev,
+            BooterKind::Loader,
+            chipset,
+            FIRMWARE_VERSION,
+            sec2_falcon,
             bar,
-            Some(wpr_handle as u32),
-            Some((wpr_handle >> 32) as u32),
-        )?;
-        dev_dbg!(pdev, "SEC2 MBOX0: {:#x}, MBOX1: {:#x}\n", mbox0, mbox1);
-
-        if mbox0 != 0 {
-            dev_err!(pdev, "Booter-load failed with error {:#x}\n", mbox0);
-            return Err(ENODEV);
-        }
+        )?
+        .run(dev, bar, sec2_falcon, &wpr_meta)?;
 
         gsp_falcon.write_os_version(bar, gsp_fw.bootloader.app_version);
 
Signature · 1 line
2.54.0
author maintainer
#5 lore raw

PATCH v6 5/7 gpu: nova-core: gsp: shuffle boot code a bit to keep chipset-specific parts close

In reply to Alexandre Courbot

Some parts of the GSP boot process are chip-specific actions, whereas others (like sending the initial post-boot messages) deal directly with the working GSP.

Reorganize the boot code a bit so the chipset-specific parts are clumped together, which will make their extraction into a HAL easier.

This has no effect on the GSP boot process.

---
 drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +7 -7
index 4f654e10259a..11b1cd2db8cb 100644
@@ -169,18 +169,13 @@ pub(crate) fn boot(
         let fb_layout = FbLayout::new(chipset, bar, &gsp_fw)?;
         dev_dbg!(dev, "{:#x?}\n", fb_layout);
 
+        let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
+
         // FWSEC-FRTS is not executed on chips where the FRTS region size is 0 (e.g. GA100).
         if !fb_layout.frts.is_empty() {
             Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
         }
 
-        let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
-
-        self.cmdq
-            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev))?;
-        self.cmdq
-            .send_command_no_wait(bar, commands::SetRegistry::new())?;
-
         gsp_falcon.reset(bar)?;
         let libos_handle = self.libos.dma_handle();
         let (mbox0, mbox1) = gsp_falcon.boot(
@@ -217,6 +212,11 @@ pub(crate) fn boot(
 
         dev_dbg!(pdev, "RISC-V active? {}\n", gsp_falcon.is_riscv_active(bar),);
 
+        self.cmdq
+            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev))?;
+        self.cmdq
+            .send_command_no_wait(bar, commands::SetRegistry::new())?;
+
         // Create and run the GSP sequencer.
         let seq_params = GspSequencerParams {
             bootloader_app_version: gsp_fw.bootloader.app_version,
Signature · 1 line
2.54.0
author maintainer
#6 lore raw

PATCH v6 6/7 gpu: nova-core: gsp: move chipset-specific parts of the boot process into a HAL

In reply to Alexandre Courbot

Booting the GSP is done differently depending on the architecture. Move the parts that are chipset-specific under a HAL.

This does not change much at the moment, since the differences between Turing and Ampere are rather benign, but will become critical to properly support the FSP boot process used by Hopper and Blackwell.

The Hopper/Blackwell support is not merged yet, so their HAL is a stub for now.

This patch is intended to be a mechanical code extraction with no behavioral changes.

---
 drivers/gpu/nova-core/gsp.rs           |   1 +
 drivers/gpu/nova-core/gsp/boot.rs      | 168 +++------------------------
 drivers/gpu/nova-core/gsp/hal.rs       |  74 ++++++++++++
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  50 ++++++++
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 206 +++++++++++++++++++++++++++++++++
 5 files changed, 345 insertions(+), 154 deletions(-)
drivers/gpu/nova-core/gsp.rs +1 -0
index ba5b7f990031..38378f104068 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
 mod boot;
+mod hal;
 
 use kernel::{
     debugfs,
drivers/gpu/nova-core/gsp/boot.rs +14 -154

221 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal.rs new file +74 -0

74 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal/gh100.rs new file +50 -0

50 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal/tu102.rs new file +206 -0

206 lines not on this page · raw message

Signature · 1 line
2.54.0
author maintainer
#7 lore raw

PATCH v6 7/7 gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

In reply to Alexandre Courbot

When probing the driver, the FWSEC-FRTS firmware creates a WPR2 secure memory region to store the GSP firmware, and the Booter Loader loads and starts that firmware into the GSP, making it run in RISC-V mode.

These operations need to be reverted upon unloading, particularly the WPR2 secure region creation, as its presence prevents the driver from subsequently probing.

Thus, prepare the Booter Unloader and FWSEC-SB firmwares when booting the GSP, so they can be executed at unbind time to put the GPU into a state where it can be probed again.

---
 drivers/gpu/nova-core/driver.rs          |  18 +++-
 drivers/gpu/nova-core/firmware/booter.rs |   1 -
 drivers/gpu/nova-core/firmware/fwsec.rs  |   1 -
 drivers/gpu/nova-core/gpu.rs             |  35 ++++++--
 drivers/gpu/nova-core/gsp.rs             |   3 +
 drivers/gpu/nova-core/gsp/boot.rs        |  35 ++++++--
 drivers/gpu/nova-core/gsp/hal.rs         |  21 ++++-
 drivers/gpu/nova-core/gsp/hal/gh100.rs   |   7 +-
 drivers/gpu/nova-core/gsp/hal/tu102.rs   | 141 ++++++++++++++++++++++++++++++-
 drivers/gpu/nova-core/regs.rs            |   5 ++
 10 files changed, 242 insertions(+), 25 deletions(-)
drivers/gpu/nova-core/driver.rs +15 -3
index ced1d38d206a..20d38a64dcc7 100644
@@ -1,5 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+use core::cell::Cell;
+
 use kernel::{
     auxiliary,
     device::Core,
@@ -21,7 +23,10 @@
     types::ForLt,
 };
 
-use crate::gpu::Gpu;
+use crate::{
+    gpu::Gpu,
+    gsp, //
+};
 
 /// Counter for generating unique auxiliary device IDs.
 static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
@@ -33,6 +38,10 @@ pub(crate) struct NovaCore<'bound> {
     bar: pci::Bar<'bound, BAR0_SIZE>,
     #[allow(clippy::type_complexity)]
     _reg: Devres<auxiliary::Registration<ForLt!(())>>,
+    /// GSP unload bundle, if any.
+    ///
+    /// Stored into a `Cell` so it can be [taken](Cell::take) without a mutable reference.
+    unload_bundle: Cell<Option<gsp::UnloadBundle>>,
 }
 
 const BAR0_SIZE: usize = SZ_16M;
@@ -94,6 +103,8 @@ fn probe<'bound>(
             // other threads of execution.
             unsafe { pdev.dma_set_mask_and_coherent(DmaMask::new::<GPU_DMA_BITS>())? };
 
+            let mut unload_bundle = None;
+
             Ok(try_pin_init!(NovaCore {
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 // TODO: Use `&bar` self-referential pin-init syntax once available.
@@ -101,7 +112,7 @@ fn probe<'bound>(
                 // SAFETY: `bar` is initialized before this expression is evaluated
                 // (`try_pin_init!()` initializes fields in declaration order), lives at a pinned
                 // stable address, and is dropped after `gpu` (struct field drop order).
-                gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }),
+                gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }, &mut unload_bundle),
                 _reg: auxiliary::Registration::new(
                     pdev.as_ref(),
                     c"nova-drm",
@@ -111,6 +122,7 @@ fn probe<'bound>(
                     crate::MODULE_NAME,
                     (),
                 )?,
+                unload_bundle: Cell::new(unload_bundle),
             }))
         })
     }
@@ -119,6 +131,6 @@ fn unbind<'bound>(
         dev: &'bound pci::Device<kernel::device::Core>,
         this: Pin<&'bound Self::Data<'bound>>,
     ) {
-        this.gpu.unbind(dev)
+        this.gpu.unbind(dev, this.unload_bundle.take())
     }
 }
drivers/gpu/nova-core/firmware/booter.rs +0 -1
index e45e5dc8d5d2..c5e17605e1a3 100644
@@ -282,7 +282,6 @@ fn new_booter(data: &[u8]) -> Result<Self> {
 #[derive(Copy, Clone, Debug, PartialEq)]
 pub(crate) enum BooterKind {
     Loader,
-    #[expect(unused)]
     Unloader,
 }
 
drivers/gpu/nova-core/firmware/fwsec.rs +0 -1
index 8810cb49db67..4108f28cd338 100644
@@ -144,7 +144,6 @@ pub(crate) enum FwsecCommand {
     /// image into it.
     Frts { frts_addr: u64, frts_size: u64 },
     /// Asks [`FwsecFirmware`] to load pre-OS apps on the PMU.
-    #[expect(dead_code)]
     Sb,
 }
 
drivers/gpu/nova-core/gpu.rs +29 -6
index 75fe1bdb80fe..5af04901b512 100644
@@ -18,7 +18,10 @@
         Falcon, //
     },
     fb::SysmemFlush,
-    gsp::Gsp,
+    gsp::{
+        self,
+        Gsp, //
+    },
     regs,
 };
 
@@ -261,10 +264,20 @@ pub(crate) struct Gpu<'bound> {
 }
 
 impl<'bound> Gpu<'bound> {
-    pub(crate) fn new(
+    /// Create a new [`Gpu`] instance.
+    ///
+    /// `pdev` is the PCI device for the GPU, `bar` is its `Bar0` mapping.
+    ///
+    /// `unload_bundle` is an output parameter, where the [GSP unload bundle](gsp::UnloadBundle) is
+    /// to be written. The driver layer will pass the written value back to [`Gpu::unbind`].
+    pub(crate) fn new<'init>(
         pdev: &'bound pci::Device<device::Bound>,
         bar: &'bound Bar0,
-    ) -> impl PinInit<Self, Error> + 'bound {
+        unload_bundle: &'init mut Option<gsp::UnloadBundle>,
+    ) -> impl PinInit<Self, Error> + 'init
+    where
+        'bound: 'init,
+    {
         try_pin_init!(Self {
             spec: Spec::new(pdev.as_ref(), bar).inspect(|spec| {
                 dev_info!(pdev,"NVIDIA ({})\n", spec);
@@ -290,14 +303,24 @@ pub(crate) fn new(
 
             gsp <- Gsp::new(pdev),
 
-            _: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? },
+            _: { *unload_bundle = gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? },
         })
     }
 
-    pub(crate) fn unbind(&self, pdev: &'bound pci::Device<device::Core>) {
+    pub(crate) fn unbind(
+        &self,
+        pdev: &'bound pci::Device<device::Core>,
+        unload_bundle: Option<gsp::UnloadBundle>,
+    ) {
         let _ = self
             .gsp
-            .unload(pdev.as_ref(), self.bar, &self.gsp_falcon)
+            .unload(
+                pdev.as_ref(),
+                self.bar,
+                &self.gsp_falcon,
+                &self.sec2_falcon,
+                unload_bundle,
+            )
             .inspect_err(|e| dev_err!(pdev, "failed to unload GSP: {:?}\n", e));
     }
 }
drivers/gpu/nova-core/gsp.rs +3 -0
index 38378f104068..1885cfa5cb38 100644
@@ -185,3 +185,6 @@ pub(crate) fn new(pdev: &pci::Device<device::Bound>) -> impl PinInit<Self, Error
         })
     }
 }
+
+/// Opaque bundle required to unload the GSP. Created by [`Gsp::boot`], consumed by [`Gsp::unload`].
+pub(crate) struct UnloadBundle(KBox<dyn hal::UnloadBundle>);
drivers/gpu/nova-core/gsp/boot.rs +27 -8
index 447c9b083039..2968178d0c6d 100644
@@ -46,7 +46,7 @@ pub(crate) fn boot(
         chipset: Chipset,
         gsp_falcon: &Falcon<Gsp>,
         sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    ) -> Result<Option<super::UnloadBundle>> {
         let dev = pdev.as_ref();
         let hal = super::hal::gsp_hal(chipset);
 
@@ -57,8 +57,8 @@ pub(crate) fn boot(
 
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
-        // Perform the chipset-specific boot sequence.
-        hal.boot(
+        // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
+        let unload_bundle = hal.boot(
             self.as_mut(),
             dev,
             bar,
@@ -98,7 +98,7 @@ pub(crate) fn boot(
             Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e),
         }
 
-        Ok(())
+        Ok(unload_bundle.map(super::UnloadBundle))
     }
 
     /// Shut down the GSP and wait until it is offline.
@@ -130,16 +130,35 @@ pub(crate) fn unload(
         dev: &device::Device<device::Bound>,
         bar: &Bar0,
         gsp_falcon: &Falcon<Gsp>,
+        sec2_falcon: &Falcon<Sec2>,
+        unload_bundle: Option<super::UnloadBundle>,
     ) -> Result {
-        // Shut down the GSP.
-        Self::shutdown_gsp(
+        // Shut down the GSP. Keep going even in case of error.
+        let mut res = Self::shutdown_gsp(
             &self.cmdq,
             bar,
             gsp_falcon,
             commands::PowerStateLevel::Level0,
         )
-        .inspect_err(|e| dev_err!(dev, "Unload guest driver failed: {:?}\n", e))?;
+        .inspect_err(|e| dev_err!(dev, "GSP shutdown failed: {:?}\n", e));
 
-        Ok(())
+        // Run the unload bundle to reset the GSP so it can be booted again.
+        if let Some(unload_bundle) = unload_bundle {
+            res = res.and(
+                unload_bundle
+                    .0
+                    .run(dev, bar, gsp_falcon, sec2_falcon)
+                    .inspect_err(|e| dev_err!(dev, "Unload bundle failed: {:?}\n", e)),
+            );
+        } else {
+            dev_warn!(
+                dev,
+                "Unload bundle is missing, GSP won't be properly reset.\n"
+            );
+
+            res = Err(EAGAIN);
+        }
+
+        res.inspect(|()| dev_info!(dev, "GSP successfully unloaded\n"))
     }
 }
drivers/gpu/nova-core/gsp/hal.rs +20 -1
index ae16fd6ba7fb..fe591c124a94 100644
@@ -30,9 +30,28 @@
     },
 };
 
+/// Trait for types containing the resources and code required to fully reset the GSP.
+///
+/// The GSP unload code might run in a situation where we cannot load firmware dynamically (e.g.
+/// because we are in shutdown and the file system is not accessible anymore). Thus, the firmware
+/// required for unloading is prepared at load time, and stored here until it needs to be run.
+pub(super) trait UnloadBundle: Send {
+    /// Performs the steps required to properly reset the GSP after it has been stopped.
+    fn run(
+        &self,
+        dev: &device::Device<device::Bound>,
+        bar: &Bar0,
+        gsp_falcon: &Falcon<GspEngine>,
+        sec2_falcon: &Falcon<Sec2>,
+    ) -> Result;
+}
+
 /// Trait implemented by GSP HALs.
 pub(super) trait GspHal: Send {
     /// Performs the GSP boot process, loading and running the required firmwares as needed.
+    ///
+    /// Upon success, returns the [`UnloadBundle`] to be run (if any) in order to properly reset the
+    /// GSP after it has been stopped.
     #[allow(clippy::too_many_arguments)]
     fn boot(
         &self,
@@ -44,7 +63,7 @@ fn boot(
         wpr_meta: &Coherent<GspFwWprMeta>,
         gsp_falcon: &Falcon<GspEngine>,
         sec2_falcon: &Falcon<Sec2>,
-    ) -> Result;
+    ) -> Result<Option<KBox<dyn UnloadBundle>>>;
 
     /// Performs HAL-specific post-GSP boot tasks.
     ///
drivers/gpu/nova-core/gsp/hal/gh100.rs +5 -2
index 187fb7dbe40a..46ffc51dc385 100644
@@ -18,7 +18,10 @@
     fb::FbLayout,
     gpu::Chipset,
     gsp::{
-        hal::GspHal,
+        hal::{
+            GspHal,
+            UnloadBundle, //
+        },
         Gsp,
         GspFwWprMeta, //
     },
@@ -41,7 +44,7 @@ fn boot(
         _wpr_meta: &Coherent<GspFwWprMeta>,
         _gsp_falcon: &Falcon<GspEngine>,
         _sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    ) -> Result<Option<KBox<dyn UnloadBundle>>> {
         Err(ENOTSUPP)
     }
 }
drivers/gpu/nova-core/gsp/hal/tu102.rs +138 -3

165 lines not on this page · raw message

drivers/gpu/nova-core/regs.rs +5 -0
index 6faeed73901d..356fbf364ea5 100644
@@ -175,6 +175,11 @@ impl NV_PFB_PRI_MMU_WPR2_ADDR_HI {
     pub(crate) fn higher_bound(self) -> u64 {
         u64::from(self.hi_val()) << 12
     }
+
+    /// Returns whether the WPR2 region is currently set.
+    pub(crate) fn is_wpr2_set(self) -> bool {
+        self.hi_val() != 0
+    }
 }
 
 // PGSP
Signature · 1 line
2.54.0
Reviewed-by on v6 7/7
#8 lore raw

Re: [PATCH v6 7/7] gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

Quoted 8 lines · Alexandre Courbot

When probing the driver, the FWSEC-FRTS firmware creates a WPR2 secure memory region to store the GSP firmware, and the Booter Loader loads and starts that firmware into the GSP, making it run in RISC-V mode.

These operations need to be reverted upon unloading, particularly the WPR2 secure region creation, as its presence prevents the driver from subsequently probing.

Quoted 6 lines

Thus, prepare the Booter Unloader and FWSEC-SB firmwares when booting the GSP, so they can be executed at unbind time to put the GPU into a state where it can be probed again.

---

After seeing the bundle moved outwards, I realised that it has the same issue that SysmemFlush does, i.e. if probe fails it does not reset the GSP. A lot of the time during development I will break things badly enough that probe fails, so it would be nice if this is supported. OTOH, this gets the probe suceed and unload case working which is important and this is a definite improvement, so for this version and the previous version of the patch:

I also had a brief go at making this work on Drop, here is the diff on top of this series. I can send this as a follow up if you would like after cleaning it up, or lmk wdyt:

drivers/gpu/nova-core/driver.rs +2 -21
index 20d38a64dcc7..6571a8c59d09 100644
@@ -1,7 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
 
-use core::cell::Cell;
-
 use kernel::{
     auxiliary,
     device::Core,
@@ -23,10 +21,7 @@
     types::ForLt,
 };
 
-use crate::{
-    gpu::Gpu,
-    gsp, //
-};
+use crate::gpu::Gpu;
 
 /// Counter for generating unique auxiliary device IDs.
 static AUXILIARY_ID_COUNTER: Atomic<u32> = Atomic::new(0);
@@ -38,10 +33,6 @@ pub(crate) struct NovaCore<'bound> {
     bar: pci::Bar<'bound, BAR0_SIZE>,
     #[allow(clippy::type_complexity)]
     _reg: Devres<auxiliary::Registration<ForLt!(())>>,
-    /// GSP unload bundle, if any.
-    ///
-    /// Stored into a `Cell` so it can be [taken](Cell::take) without a mutable reference.
-    unload_bundle: Cell<Option<gsp::UnloadBundle>>,
 }
 
 const BAR0_SIZE: usize = SZ_16M;
@@ -103,8 +94,6 @@ fn probe<'bound>(
             // other threads of execution.
             unsafe { pdev.dma_set_mask_and_coherent(DmaMask::new::<GPU_DMA_BITS>())? };
 
-            let mut unload_bundle = None;
-
             Ok(try_pin_init!(NovaCore {
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 // TODO: Use `&bar` self-referential pin-init syntax once available.
@@ -112,7 +101,7 @@ fn probe<'bound>(
                 // SAFETY: `bar` is initialized before this expression is evaluated
                 // (`try_pin_init!()` initializes fields in declaration order), lives at a pinned
                 // stable address, and is dropped after `gpu` (struct field drop order).
-                gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }, &mut unload_bundle),
+                gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }),
                 _reg: auxiliary::Registration::new(
                     pdev.as_ref(),
                     c"nova-drm",
@@ -122,15 +111,7 @@ fn probe<'bound>(
                     crate::MODULE_NAME,
                     (),
                 )?,
-                unload_bundle: Cell::new(unload_bundle),
             }))
         })
     }
-
-    fn unbind<'bound>(
-        dev: &'bound pci::Device<kernel::device::Core>,
-        this: Pin<&'bound Self::Data<'bound>>,
-    ) {
-        this.gpu.unbind(dev, this.unload_bundle.take())
-    }
 }
drivers/gpu/nova-core/gpu.rs +21 -20

77 lines not on this page · raw message

author maintainer
#9 lore raw

Re: [PATCH v6 7/7] gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

Quoted 24 lines · Eliot Courtney
Quoted 8 lines · Alexandre Courbot

When probing the driver, the FWSEC-FRTS firmware creates a WPR2 secure memory region to store the GSP firmware, and the Booter Loader loads and starts that firmware into the GSP, making it run in RISC-V mode.

These operations need to be reverted upon unloading, particularly the WPR2 secure region creation, as its presence prevents the driver from subsequently probing.

Quoted 6 lines

Thus, prepare the Booter Unloader and FWSEC-SB firmwares when booting the GSP, so they can be executed at unbind time to put the GPU into a state where it can be probed again.

---

After seeing the bundle moved outwards, I realised that it has the same issue that SysmemFlush does, i.e. if probe fails it does not reset the GSP. A lot of the time during development I will break things badly enough that probe fails, so it would be nice if this is supported. OTOH, this gets the probe suceed and unload case working which is important and this is a definite improvement, so for this version and the previous version of the patch:

Quoted 5 lines

I also had a brief go at making this work on Drop, here is the diff on top of this series. I can send this as a follow up if you would like after cleaning it up, or lmk wdyt:

This is clearly better. It guarantees that the unload sequence is run when the Gpu is dropped, while preserving its one-shot nature. Also, no Cell and no awkward output parameter to Gpu::new.

The only blind spot remaining would be to also cover the case where a failure occurs during Gsp::boot, but that's for Gsp::boot to handle itself imho.

Would you be ok if I folded this into this patch for v7, with your Co-developed-by and Signed-off-by? Then I'll also try to tackle the Gsp::boot failure scenario using a drop wrapper or something similar.

#10 lore raw

Re: [PATCH v6 7/7] gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

Quoted 35 lines · Alexandre Courbot
Quoted 24 lines · Eliot Courtney
Quoted 8 lines · Alexandre Courbot

When probing the driver, the FWSEC-FRTS firmware creates a WPR2 secure memory region to store the GSP firmware, and the Booter Loader loads and starts that firmware into the GSP, making it run in RISC-V mode.

These operations need to be reverted upon unloading, particularly the WPR2 secure region creation, as its presence prevents the driver from subsequently probing.

Quoted 6 lines

Thus, prepare the Booter Unloader and FWSEC-SB firmwares when booting the GSP, so they can be executed at unbind time to put the GPU into a state where it can be probed again.

---

After seeing the bundle moved outwards, I realised that it has the same issue that SysmemFlush does, i.e. if probe fails it does not reset the GSP. A lot of the time during development I will break things badly enough that probe fails, so it would be nice if this is supported. OTOH, this gets the probe suceed and unload case working which is important and this is a definite improvement, so for this version and the previous version of the patch:

Quoted 5 lines

I also had a brief go at making this work on Drop, here is the diff on top of this series. I can send this as a follow up if you would like after cleaning it up, or lmk wdyt:

This is clearly better. It guarantees that the unload sequence is run when the Gpu is dropped, while preserving its one-shot nature. Also, no Cell and no awkward output parameter to Gpu::new.

Quoted 7 lines

The only blind spot remaining would be to also cover the case where a failure occurs during Gsp::boot, but that's for Gsp::boot to handle itself imho.

Would you be ok if I folded this into this patch for v7, with your Co-developed-by and Signed-off-by? Then I'll also try to tackle the Gsp::boot failure scenario using a drop wrapper or something similar.

Yerp that is fine of course. For Gsp::boot it should handle unwinding if there is a failure but I agree it is orthogonal to this case, so addressing in a follow-up SGTM.

author maintainer
#11 lore raw

Re: [PATCH v6 7/7] gpu: nova-core: run Booter Unloader and FWSEC-SB upon unbinding

Quoted 40 lines · Eliot Courtney
Quoted 39 lines · Alexandre Courbot
Quoted 24 lines · Eliot Courtney
Quoted 8 lines · Alexandre Courbot

When probing the driver, the FWSEC-FRTS firmware creates a WPR2 secure memory region to store the GSP firmware, and the Booter Loader loads and starts that firmware into the GSP, making it run in RISC-V mode.

These operations need to be reverted upon unloading, particularly the WPR2 secure region creation, as its presence prevents the driver from subsequently probing.

Quoted 6 lines

Thus, prepare the Booter Unloader and FWSEC-SB firmwares when booting the GSP, so they can be executed at unbind time to put the GPU into a state where it can be probed again.

---

After seeing the bundle moved outwards, I realised that it has the same issue that SysmemFlush does, i.e. if probe fails it does not reset the GSP. A lot of the time during development I will break things badly enough that probe fails, so it would be nice if this is supported. OTOH, this gets the probe suceed and unload case working which is important and this is a definite improvement, so for this version and the previous version of the patch:

Quoted 5 lines

I also had a brief go at making this work on Drop, here is the diff on top of this series. I can send this as a follow up if you would like after cleaning it up, or lmk wdyt:

This is clearly better. It guarantees that the unload sequence is run when the Gpu is dropped, while preserving its one-shot nature. Also, no Cell and no awkward output parameter to Gpu::new.

The only blind spot remaining would be to also cover the case where a failure occurs during Gsp::boot, but that's for Gsp::boot to handle itself imho.

Quoted 7 lines
Quoted 3 lines

Would you be ok if I folded this into this patch for v7, with your Co-developed-by and Signed-off-by? Then I'll also try to tackle the Gsp::boot failure scenario using a drop wrapper or something similar.

Yerp that is fine of course. For Gsp::boot it should handle unwinding if there is a failure but I agree it is orthogonal to this case, so addressing in a follow-up SGTM.

I will add a patch that takes care of that using a drop guard in v7.

While doing so, I noticed that with the current state of the code we could also perfectly store that drop guard into Gpu itself, and get the behavior we want without implementing PinnedDrop. However, this is only possible because Falcon::run and Falcon::load do not take mutable references, which they should do eventually for soundness - once we do the switch, storing the drop guard won't be possible anymore as that would mean keeping mutable references for the full life of the GPU. So PinnedDrop is necessary.

I mention this here preemptively lest it is raised in v7. :)

author maintainer Applied
#12 lore raw
In reply to Alexandre Courbot

These pre-requisite patches:

Quoted 4 lines

gpu: nova-core: remove unneeded get_gsp_info proxy function
gpu: nova-core: do not import firmware commands into GSP command module
gpu: nova-core: refactor SEC2 booter loading into BooterFirmware::run()
gpu: nova-core: gsp: shuffle boot code a bit to keep chipset-specific parts close

Pushed to drm-rust-next, thanks!

I will make a quick respin for the remaining patches, with the intent of merging v7 quickly if no new issue is raised.