Skip to contentSkip to conversation

gpu: nova-core: consolidate and streamline GSP boot process

PATCH Superseded Alexandre Courbot 4 participants 21 messages Jun 22 – Jun 29

Subsystems (5)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Alexandre Courbot

author maintainer
#0 lore raw

v2 cover gpu: nova-core: consolidate and streamline GSP boot process

The recently introduced GspBootContext is useful in at least two other places:

- As a replacement for the sequencer's GspSequencerParams,
- As the main parameter of Gsp::unload, for symmetry with Gsp::boot.

This symmetry can be further exploited to make Gsp::boot and Gsp::unload generic over the exact boot context required for the current chip. Doing so requires some more preliminary work that goes beyond this series, but this groundwork is needed regardless.

Another cleanup that can be performed as a consequence is the removal of the BootUnloadGuard. This type is cumbersome as it holds extra references to the boot context, making it difficult to make some of these references mutable (as we want to do with the Falcons that should require exclusive access to load and run firmware). Furthermore, it is only needed a single time in Gsp::boot.

By running the boot process in a local closure, we can catch errors and call Gsp::unload on the unload bundle just as efficiently, and with less code.

Building on this, the series also introduces a proper type for identifying the GSP boot method, and moves the Fsp instance to Gpu to make it available outside of the GSP boot process. This will be notably useful for vGPU support [1].

This series applies on drm-rust-next.

[1] https://lore.kernel.org/all/[email protected]/

---
Changes in v2:
- Replace nonexistent ENODATA with the more appropriate EAGAIN.
- Add patches introducing GspBootMethod.
- Add patches moving Fsp to Gpu.
- Link to v1: https://patch.msgid.link/[email protected]

---
Alexandre Courbot (13):
gpu: nova-core: gsp: sequencer: use GspBootContext
gpu: nova-core: gsp: sequencer: do not store sequence into GspSequencer
gpu: nova-core: gsp: move boot code into local closure
gpu: nova-core: gsp: replace BootUnloadGuard with local handler
gpu: nova-core: gsp: move unload bundle error handling to Gsp::boot
gpu: nova-core: gsp: make unload take GspBootContext
gpu: nova-core: gsp: fold TU102 unload bundle construction into HAL method
gpu: nova-core: gsp: turn FWSEC execution into HAL method
gpu: nova-core: gsp: make use of FWSEC bootloader a property of the TU102 HAL
gpu: nova-core: introduce GspBootMethod
gpu: nova-core: avoid repeated calls to pci::Device::as_ref
gpu: nova-core: gsp: pass GspBootContext mutably
gpu: nova-core: store Fsp instance in Gpu

 drivers/gpu/nova-core/firmware.rs       |  25 ++-
 drivers/gpu/nova-core/firmware/fwsec.rs |   5 +-
 drivers/gpu/nova-core/gpu.rs            |  68 +++---
 drivers/gpu/nova-core/gsp.rs            |  44 +++-
 drivers/gpu/nova-core/gsp/boot.rs       | 142 +++++-------
 drivers/gpu/nova-core/gsp/hal.rs        |  63 +++---
 drivers/gpu/nova-core/gsp/hal/ga102.rs  |  14 ++
 drivers/gpu/nova-core/gsp/hal/gh100.rs  |  62 +++--
 drivers/gpu/nova-core/gsp/hal/tu102.rs  | 385 +++++++++++++++-----------------
 drivers/gpu/nova-core/gsp/sequencer.rs  |  71 +++---
 10 files changed, 436 insertions(+), 443 deletions(-)

---
base-commit: f0c1bb8ead8a9790e7d0339354598b50a9c6789a
change-id: 20260619-nova-bootcontext-401d6107e8fb

Best regards,

Signature · 1 line
Alexandre Courbot <[email protected]>
author maintainer
#1 lore raw

PATCH v2 1/13 gpu: nova-core: gsp: sequencer: use GspBootContext

GspBootContext contains all the resources currently carried by GspSequencerParams, so replace the latter with the former for better integration with the boot process and less code.

---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 21 +++++++-------------
 drivers/gpu/nova-core/gsp/sequencer.rs | 36 ++++++++++++----------------------
 2 files changed, 20 insertions(+), 37 deletions(-)
drivers/gpu/nova-core/gsp/hal/tu102.rs +7 -14
index f8a8541704ee..6ed4ee268086 100644
@@ -37,10 +37,7 @@
             GspHal,
             UnloadBundle, //
         },
-        sequencer::{
-            GspSequencer,
-            GspSequencerParams, //
-        },
+        sequencer::GspSequencer,
         Gsp,
         GspBootContext,
         GspFwWprMeta, //
@@ -336,16 +333,12 @@ fn boot<'a>(
     }
 
     fn post_boot(&self, gsp: &Gsp, ctx: &GspBootContext<'_>, gsp_fw: &GspFirmware) -> Result {
-        // Create and run the GSP sequencer.
-        let seq_params = GspSequencerParams {
-            bootloader_app_version: gsp_fw.bootloader.app_version,
-            libos_dma_handle: gsp.libos.dma_handle(),
-            gsp_falcon: ctx.gsp_falcon,
-            sec2_falcon: ctx.sec2_falcon,
-            dev: ctx.dev(),
-            bar: ctx.bar,
-        };
-        GspSequencer::run(&gsp.cmdq, seq_params)?;
+        GspSequencer::run(
+            &gsp.cmdq,
+            ctx,
+            gsp.libos.dma_handle(),
+            gsp_fw.bootloader.app_version,
+        )?;
 
         Ok(())
     }
drivers/gpu/nova-core/gsp/sequencer.rs +13 -23
index e0850d21adca..0da3c3531886 100644
@@ -31,6 +31,7 @@
             MessageFromGsp, //
         },
         fw,
+        GspBootContext, //
     },
     num::FromSafeCast,
     sbuffer::SBufferIter,
@@ -338,24 +339,13 @@ fn iter(&self) -> GspSeqIter<'_> {
     }
 }
 
-/// Parameters for running the GSP sequencer.
-pub(crate) struct GspSequencerParams<'a> {
-    /// Bootloader application version.
-    pub(crate) bootloader_app_version: u32,
-    /// LibOS DMA handle address.
-    pub(crate) libos_dma_handle: u64,
-    /// GSP falcon for core operations.
-    pub(crate) gsp_falcon: &'a Falcon<Gsp>,
-    /// SEC2 falcon for core operations.
-    pub(crate) sec2_falcon: &'a Falcon<Sec2>,
-    /// Device for logging.
-    pub(crate) dev: &'a device::Device,
-    /// BAR0 for register access.
-    pub(crate) bar: Bar0<'a>,
-}
-
 impl<'a> GspSequencer<'a> {
-    pub(crate) fn run(cmdq: &Cmdq, params: GspSequencerParams<'a>) -> Result {
+    pub(crate) fn run(
+        cmdq: &Cmdq,
+        ctx: &'a GspBootContext<'_>,
+        libos_dma_handle: u64,
+        bootloader_app_version: u32,
+    ) -> Result {
         let seq_info = loop {
             match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
@@ -366,12 +356,12 @@ pub(crate) fn run(cmdq: &Cmdq, params: GspSequencerParams<'a>) -> Result {
 
         let sequencer = GspSequencer {
             seq_info,
-            bar: params.bar,
-            sec2_falcon: params.sec2_falcon,
-            gsp_falcon: params.gsp_falcon,
-            libos_dma_handle: params.libos_dma_handle,
-            bootloader_app_version: params.bootloader_app_version,
-            dev: params.dev,
+            bar: ctx.bar,
+            sec2_falcon: ctx.sec2_falcon,
+            gsp_falcon: ctx.gsp_falcon,
+            libos_dma_handle,
+            bootloader_app_version,
+            dev: ctx.dev(),
         };
 
         dev_dbg!(sequencer.dev, "Running CPU Sequencer commands\n");
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v2 2/13 gpu: nova-core: gsp: sequencer: do not store sequence into GspSequencer

In reply to Alexandre Courbot

The sequence is currently stored in the GspSequencer even though its lifetime is limited to GspSequencer::run. This object-oriented design does not play well with the borrow-checker, as GspSequencer::iter borrows a reference to the GspSequencer, which makes it difficult to introduce mutable references in GspBootContext, as we want to do in order to make the Falcon references mutable.

Thus, store the sequence locally in GspSequencer::run, and move iterator creation to GspSeqIter::new so it no longer needs to borrow the whole GspSequencer.

---
 drivers/gpu/nova-core/gsp/sequencer.rs | 35 +++++++++++++++-------------------
 1 file changed, 15 insertions(+), 20 deletions(-)
drivers/gpu/nova-core/gsp/sequencer.rs +15 -20
index 0da3c3531886..b5e049f76c28 100644
@@ -129,8 +129,6 @@ pub(crate) fn new(data: &[u8], dev: &device::Device) -> Result<(Self, usize)> {
 
 /// GSP Sequencer for executing firmware commands during boot.
 pub(crate) struct GspSequencer<'a> {
-    /// Sequencer information with command data.
-    seq_info: GspSequence,
     /// `Bar0` for register access.
     bar: Bar0<'a>,
     /// SEC2 falcon for core operations.
@@ -271,7 +269,7 @@ fn run(&self, seq: &GspSequencer<'_>) -> Result {
 }
 
 /// Iterator over GSP sequencer commands.
-pub(crate) struct GspSeqIter<'a> {
+struct GspSeqIter<'a> {
     /// Command data buffer.
     cmd_data: &'a [u8],
     /// Current position in the buffer.
@@ -284,6 +282,18 @@ pub(crate) struct GspSeqIter<'a> {
     dev: &'a device::Device,
 }
 
+impl<'a> GspSeqIter<'a> {
+    fn new(seq: &'a GspSequence, dev: &'a device::Device) -> Self {
+        Self {
+            cmd_data: &seq.cmd_data,
+            current_offset: 0,
+            total_cmds: seq.cmd_index,
+            cmds_processed: 0,
+            dev,
+        }
+    }
+}
+
 impl<'a> Iterator for GspSeqIter<'a> {
     type Item = Result<GspSeqCmd>;
 
@@ -325,20 +335,6 @@ fn next(&mut self) -> Option<Self::Item> {
     }
 }
 
-impl<'a> GspSequencer<'a> {
-    fn iter(&self) -> GspSeqIter<'_> {
-        let cmd_data = &self.seq_info.cmd_data[..];
-
-        GspSeqIter {
-            cmd_data,
-            current_offset: 0,
-            total_cmds: self.seq_info.cmd_index,
-            cmds_processed: 0,
-            dev: self.dev,
-        }
-    }
-}
-
 impl<'a> GspSequencer<'a> {
     pub(crate) fn run(
         cmdq: &Cmdq,
@@ -355,7 +351,6 @@ pub(crate) fn run(
         };
 
         let sequencer = GspSequencer {
-            seq_info,
             bar: ctx.bar,
             sec2_falcon: ctx.sec2_falcon,
             gsp_falcon: ctx.gsp_falcon,
@@ -366,14 +361,14 @@ pub(crate) fn run(
 
         dev_dbg!(sequencer.dev, "Running CPU Sequencer commands\n");
 
-        for cmd_result in sequencer.iter() {
+        for cmd_result in GspSeqIter::new(&seq_info, sequencer.dev) {
             match cmd_result {
                 Ok(cmd) => cmd.run(&sequencer)?,
                 Err(e) => {
                     dev_err!(
                         sequencer.dev,
                         "Error running command at index {}\n",
-                        sequencer.seq_info.cmd_index
+                        seq_info.cmd_index
                     );
                     return Err(e);
                 }
Signature · 1 line
2.54.0
#3 lore raw

Re: [PATCH v2 02/13] gpu: nova-core: gsp: sequencer: do not store sequence into GspSequencer

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the iterator silently terminates on command parsing errors, causing the GSP boot sequence to incorrectly report success instead of aborting. --

commit 67a42d6a50b9bd7a1977aebced3e9453d53bb90e
Author: Alexandre Courbot <[email protected]>

gpu: nova-core: gsp: sequencer: do not store sequence into GspSequencer

This commit refactors the GSP sequencer by moving the command sequence data out of the GspSequencer struct and instead passing it directly when creating the iterator. This resolves borrow-checker limitations and allows for future mutable borrows within the boot context.

Quoted 4 lines
drivers/gpu/nova-core/gsp/sequencer.rs +0 -0
index 0da3c35318866..b5e049f76c282 100644

[ ... ]

Quoted 9 lines
drivers/gpu/nova-core/gsp/sequencer.rs +0 -1
@@ -355,7 +351,6 @@ pub(crate) fn run(
         };
 
         let sequencer = GspSequencer {
-            seq_info,
             bar: ctx.bar,
             sec2_falcon: ctx.sec2_falcon,
             gsp_falcon: ctx.gsp_falcon,

drivers/gpu/nova-core/gsp/sequencer.rs pub(crate) fn run( · line 364

Quoted 5 lines
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
@@ -366,14 +361,14 @@ pub(crate) fn run(
 
         dev_dbg!(sequencer.dev, "Running CPU Sequencer commands\n");
 
-        for cmd_result in sequencer.iter() {
+        for cmd_result in GspSeqIter::new(&seq_info, sequencer.dev) {

[Severity: High]
This isn't a bug introduced by this patch, but doesn't the iterator silently terminate on command parsing errors, causing the GSP boot sequence to incorrectly report success instead of aborting?

Looking at GspSeqIter::next(), when a command fails to parse, the error is swallowed and translated to None:

drivers/gpu/nova-core/gsp/sequencer.rs:GspSeqIter::next() {
    ...
    cmd_result.map_or_else(
        |_err| {
            dev_err!(self.dev, "Error parsing command at offset {}\n", offset);
            None
        },
    ...
}

Returning None signals normal exhaustion of the iterator. This causes the loop here in GspSequencer::run() to cleanly exit without ever hitting the Err(e) match arm below.

Consequently, GspSequencer::run() will incorrectly log success and return Ok(()), completely masking the hardware initialization failure and leaving the hardware in a malformed state.

Quoted 11 lines
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
             match cmd_result {
                 Ok(cmd) => cmd.run(&sequencer)?,
                 Err(e) => {
                     dev_err!(
                         sequencer.dev,
                         "Error running command at index {}\n",
-                        sequencer.seq_info.cmd_index
+                        seq_info.cmd_index
                     );
                     return Err(e);
                 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author maintainer
#4 lore raw

PATCH v2 3/13 gpu: nova-core: gsp: move boot code into local closure

In reply to Alexandre Courbot

The next patch aims at replacing the cumbersome BootUnloadGuard with a more local and less intrusive mechanism to run the GSP unload sequence upon GSP boot failure. Doing so requires running the boot code in a local closure, which changes its indentation and would make other changes difficult to track in the diff. Thus, this preparatory patch moves said boot code into a local closure that is run upon construction, so the next patch does not need to re-indent code that changes.

This is a mechanical preparatory patch to make the next patch easier to read. No functional change intended.

---
 drivers/gpu/nova-core/gsp/boot.rs      | 42 ++++++++-------
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 38 ++++++++------
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 96 ++++++++++++++++++----------------
 3 files changed, 95 insertions(+), 81 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +24 -18
index bb2000b7a78b..9eccfd634b61 100644
@@ -119,30 +119,36 @@ pub(crate) fn boot(
 
         // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
         let unload_guard = hal.boot(&self, &ctx, &fb_layout, &wpr_meta)?;
+        // Run from a closure so we can retrieve the result, and run the unload sequence of the GSP
+        // in case of error.
+        let res = (|| {
+            gsp_falcon.write_os_version(bar, gsp_fw.bootloader.app_version);
 
-        gsp_falcon.write_os_version(bar, gsp_fw.bootloader.app_version);
+            // Poll for RISC-V to become active before continuing.
+            read_poll_timeout(
+                || Ok(gsp_falcon.is_riscv_active(bar)),
+                |val: &bool| *val,
+                Delta::from_millis(10),
+                Delta::from_secs(5),
+            )?;
 
-        // Poll for RISC-V to become active before continuing.
-        read_poll_timeout(
-            || Ok(gsp_falcon.is_riscv_active(bar)),
-            |val: &bool| *val,
-            Delta::from_millis(10),
-            Delta::from_secs(5),
-        )?;
+            dev_dbg!(pdev, "RISC-V active? {}\n", gsp_falcon.is_riscv_active(bar),);
 
-        dev_dbg!(pdev, "RISC-V active? {}\n", gsp_falcon.is_riscv_active(bar),);
+            self.cmdq
+                .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev, chipset))?;
+            self.cmdq
+                .send_command_no_wait(bar, commands::SetRegistry::new())?;
 
-        self.cmdq
-            .send_command_no_wait(bar, commands::SetSystemInfo::new(pdev, chipset))?;
-        self.cmdq
-            .send_command_no_wait(bar, commands::SetRegistry::new())?;
+            hal.post_boot(&self, &ctx, &gsp_fw)?;
 
-        hal.post_boot(&self, &ctx, &gsp_fw)?;
+            // Wait until GSP is fully initialized.
+            commands::wait_gsp_init_done(&self.cmdq)
+        })();
 
-        // Wait until GSP is fully initialized.
-        commands::wait_gsp_init_done(&self.cmdq)?;
-
-        Ok(unload_guard.dismiss())
+        match res {
+            Err(e) => Err(e),
+            Ok(()) => Ok(unload_guard.dismiss()),
+        }
     }
 
     /// Shut down the GSP and wait until it is offline.
drivers/gpu/nova-core/gsp/hal/gh100.rs +21 -17
index 2187e11168b2..776f61b8b9d7 100644
@@ -158,29 +158,33 @@ fn boot<'a>(
         let gsp_falcon = ctx.gsp_falcon;
         let sec2_falcon = ctx.sec2_falcon;
 
-        let unload_bundle = crate::gsp::UnloadBundle(
-            KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>
-        );
+        let res = (|| {
+            let unload_bundle = crate::gsp::UnloadBundle(
+                KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>
+            );
 
-        // Wrap the unload bundle into a drop guard so it is automatically run upon failure.
-        let unload_guard =
-            BootUnloadGuard::new(gsp, dev, bar, gsp_falcon, sec2_falcon, Some(unload_bundle));
+            // Wrap the unload bundle into a drop guard so it is automatically run upon failure.
+            let unload_guard =
+                BootUnloadGuard::new(gsp, dev, bar, gsp_falcon, sec2_falcon, Some(unload_bundle));
 
-        let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset)?;
+            let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset)?;
 
-        let args = FmcBootArgs::new(
-            dev,
-            chipset,
-            wpr_meta.dma_handle(),
-            gsp.libos.dma_handle(),
-            false,
-        )?;
+            let args = FmcBootArgs::new(
+                dev,
+                chipset,
+                wpr_meta.dma_handle(),
+                gsp.libos.dma_handle(),
+                false,
+            )?;
 
-        fsp.boot_fmc(dev, bar, fb_layout, &args)?;
+            fsp.boot_fmc(dev, bar, fb_layout, &args)?;
 
-        wait_for_gsp_lockdown_release(dev, bar, gsp_falcon, args.boot_params_dma_handle())?;
+            wait_for_gsp_lockdown_release(dev, bar, gsp_falcon, args.boot_params_dma_handle())?;
 
-        Ok(unload_guard)
+            Ok(unload_guard)
+        })();
+
+        res
     }
 }
 
drivers/gpu/nova-core/gsp/hal/tu102.rs +50 -46

109 lines not on this page · raw message

Signature · 1 line
2.54.0
#5 lore raw

Re: [PATCH v2 03/13] gpu: nova-core: gsp: move boot code into local closure

Quoted 8 lines · Alexandre Courbot

The next patch aims at replacing the cumbersome BootUnloadGuard with a more local and less intrusive mechanism to run the GSP unload sequence upon GSP boot failure. Doing so requires running the boot code in a local closure, which changes its indentation and would make other changes difficult to track in the diff. Thus, this preparatory patch moves said boot code into a local closure that is run upon construction, so the next patch does not need to re-indent code that changes.

Quoted 5 lines

This is a mechanical preparatory patch to make the next patch easier to read. No functional change intended.

---

See my response at [1].

[1]: https://lore.kernel.org/all/[email protected]/

#6 lore raw

Re: [PATCH v2 03/13] gpu: nova-core: gsp: move boot code into local closure

Quoted 12 lines · Eliot Courtney
Quoted 11 lines · Alexandre Courbot

The next patch aims at replacing the cumbersome BootUnloadGuard with a more local and less intrusive mechanism to run the GSP unload sequence upon GSP boot failure. Doing so requires running the boot code in a local closure, which changes its indentation and would make other changes difficult to track in the diff. Thus, this preparatory patch moves said boot code into a local closure that is run upon construction, so the next patch does not need to re-indent code that changes.

This is a mechanical preparatory patch to make the next patch easier to read. No functional change intended.

Quoted 6 lines
Quoted 2 lines

---

See my response at [1].

[1]: https://lore.kernel.org/all/[email protected]/

I'm relieved to see us going with RAII scope guards instead of a pile of closures. But now I'm tempted to wait for a v3 that does that, before reviewing, on the assumption that the rest of the series changes quite a bit as a result.

thanks,

Signature · 1 line
John Hubbard
author maintainer
#7 lore raw

PATCH v2 4/13 gpu: nova-core: gsp: replace BootUnloadGuard with local handler

In reply to Alexandre Courbot

When adding the GSP unload capability, we introduced new types to support what is effectively an ad-hoc behavior: that Gsp::unload must be run if any error occurs during the boot sequence.

Furthermore, BootUnloadGuard is problematic because it holds additional references to the boot context, notably the Falcons. These extra references stand in the way of making some of the Falcon's methods mutable, since those methods would require exclusive access. As this behavior is only needed in one place, introducing dedicated types for it is distracting and unnecessary.

Thus, take advantage of the local closures introduced in the preceding patch to run the unload sequence in Gsp::boot if an error occurred at any step of the boot process.

This slightly broadens the failure cleanup path, as Gsp::boot now attempts a best-effort Gsp::unload even when the failure happened before the unload bundle was prepared.

---
 drivers/gpu/nova-core/gsp/boot.rs      | 81 ++++++----------------------------
 drivers/gpu/nova-core/gsp/hal.rs       | 20 ++++++---
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 28 +++++-------
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 23 ++++------
 4 files changed, 47 insertions(+), 105 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +14 -67
index 9eccfd634b61..3574f1a87344 100644
@@ -7,8 +7,7 @@
     dma::Coherent,
     io::poll::read_poll_timeout,
     prelude::*,
-    time::Delta,
-    types::ScopeGuard, //
+    time::Delta, //
 };
 
 use crate::{
@@ -30,66 +29,6 @@
     },
 };
 
-/// Arguments required to call [`Gsp::unload`](super::Gsp::unload).
-///
-/// Stored as their own type to avoid repeating a long and tedious list in [`BootUnloadGuard`].
-pub(super) struct BootUnloadArgs<'a> {
-    gsp: &'a super::Gsp,
-    dev: &'a device::Device<device::Bound>,
-    bar: Bar0<'a>,
-    gsp_falcon: &'a Falcon<Gsp>,
-    sec2_falcon: &'a Falcon<Sec2>,
-    unload_bundle: Option<super::UnloadBundle>,
-}
-
-/// Guard that calls [`Gsp::unload`](super::Gsp::unload) with a
-/// [`UnloadBundle`](super::UnloadBundle) when dropped.
-///
-/// Used to ensure the `UnloadBundle` is run during failure paths.
-pub(super) struct BootUnloadGuard<'a> {
-    guard: ScopeGuard<BootUnloadArgs<'a>, fn(BootUnloadArgs<'a>)>,
-}
-
-impl<'a> BootUnloadGuard<'a> {
-    /// Wraps `unload_bundle` into a guard that executes it when dropped.
-    pub(super) fn new(
-        gsp: &'a super::Gsp,
-        dev: &'a device::Device<device::Bound>,
-        bar: Bar0<'a>,
-        gsp_falcon: &'a Falcon<Gsp>,
-        sec2_falcon: &'a Falcon<Sec2>,
-        unload_bundle: Option<super::UnloadBundle>,
-    ) -> Self {
-        Self {
-            guard: ScopeGuard::new_with_data(
-                BootUnloadArgs {
-                    gsp,
-                    dev,
-                    bar,
-                    gsp_falcon,
-                    sec2_falcon,
-                    unload_bundle,
-                },
-                |args| {
-                    let _ = super::Gsp::unload(
-                        args.gsp,
-                        args.dev,
-                        args.bar,
-                        args.gsp_falcon,
-                        args.sec2_falcon,
-                        args.unload_bundle,
-                    );
-                },
-            ),
-        }
-    }
-
-    /// Disarms the guard and returns the [`UnloadBundle`](super::UnloadBundle) it contains.
-    pub(super) fn dismiss(self) -> Option<super::UnloadBundle> {
-        self.guard.dismiss().unload_bundle
-    }
-}
-
 impl super::Gsp {
     /// Attempt to boot the GSP.
     ///
@@ -118,10 +57,11 @@ pub(crate) fn boot(
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
         // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
-        let unload_guard = hal.boot(&self, &ctx, &fb_layout, &wpr_meta)?;
+        let (res, unload_bundle) = hal.boot(&self, &ctx, &fb_layout, &wpr_meta);
+
         // Run from a closure so we can retrieve the result, and run the unload sequence of the GSP
         // in case of error.
-        let res = (|| {
+        let res = res.and_then(|()| {
             gsp_falcon.write_os_version(bar, gsp_fw.bootloader.app_version);
 
             // Poll for RISC-V to become active before continuing.
@@ -143,11 +83,18 @@ pub(crate) fn boot(
 
             // Wait until GSP is fully initialized.
             commands::wait_gsp_init_done(&self.cmdq)
-        })();
+        });
 
         match res {
-            Err(e) => Err(e),
-            Ok(()) => Ok(unload_guard.dismiss()),
+            Err(e) => {
+                dev_err!(dev, "GSP boot failed with error {:?}\n", e);
+
+                // Ignore errors during unload; we will return the error that happened during boot.
+                let _ = self.unload(dev, bar, gsp_falcon, ctx.sec2_falcon, unload_bundle);
+
+                Err(e)
+            }
+            Ok(()) => Ok(unload_bundle),
         }
     }
 
drivers/gpu/nova-core/gsp/hal.rs +13 -7
index 51a277fe97bb..a76be4e43272 100644
@@ -24,7 +24,6 @@
         Chipset, //
     },
     gsp::{
-        boot::BootUnloadGuard,
         Gsp,
         GspBootContext,
         GspFwWprMeta, //
@@ -51,15 +50,22 @@ fn run(
 pub(super) trait GspHal: Send {
     /// Performs the GSP boot process, loading and running the required firmwares as needed.
     ///
-    /// Upon success, returns a guard that runs the GSP unload sequence if GSP boot does not
-    /// complete.
-    fn boot<'a>(
+    /// Returns two things:
+    ///
+    /// - The `Result` of the boot process itself,
+    /// - The `UnloadBundle` to use with [`Gsp::unload`], or `None` if the bundle could not be
+    ///   created.
+    ///
+    /// Note that the two returned values are independent: it is possible for the boot process to
+    /// succeed while the unload bundle couldn't be created. In this case, the GSP won't be able to
+    /// unload properly and a full GPU reset is required before the GSP can be booted again.
+    fn boot(
         &self,
-        gsp: &'a Gsp,
-        ctx: &GspBootContext<'a>,
+        gsp: &Gsp,
+        ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> Result<BootUnloadGuard<'a>>;
+    ) -> (Result, Option<crate::gsp::UnloadBundle>);
 
     /// Performs HAL-specific post-GSP boot tasks.
     ///
drivers/gpu/nova-core/gsp/hal/gh100.rs +11 -17
index 776f61b8b9d7..ece40403b831 100644
@@ -23,7 +23,6 @@
         Fsp, //
     },
     gsp::{
-        boot::BootUnloadGuard,
         hal::{
             GspHal,
             UnloadBundle, //
@@ -145,27 +144,24 @@ impl GspHal for Gh100 {
     ///
     /// This path uses FSP to establish a chain of trust and boot GSP-FMC. FSP handles
     /// the GSP boot internally - no manual GSP reset/boot is needed.
-    fn boot<'a>(
+    fn boot(
         &self,
-        gsp: &'a Gsp,
-        ctx: &GspBootContext<'a>,
+        gsp: &Gsp,
+        ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> Result<BootUnloadGuard<'a>> {
+    ) -> (Result, Option<crate::gsp::UnloadBundle>) {
         let dev = ctx.dev();
         let bar = ctx.bar;
         let chipset = ctx.chipset;
         let gsp_falcon = ctx.gsp_falcon;
-        let sec2_falcon = ctx.sec2_falcon;
+
+        let mut unload_bundle = None;
 
         let res = (|| {
-            let unload_bundle = crate::gsp::UnloadBundle(
-                KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>
-            );
-
-            // Wrap the unload bundle into a drop guard so it is automatically run upon failure.
-            let unload_guard =
-                BootUnloadGuard::new(gsp, dev, bar, gsp_falcon, sec2_falcon, Some(unload_bundle));
+            unload_bundle = Some(crate::gsp::UnloadBundle(
+                KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>,
+            ));
 
             let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset)?;
 
@@ -179,12 +175,10 @@ fn boot<'a>(
 
             fsp.boot_fmc(dev, bar, fb_layout, &args)?;
 
-            wait_for_gsp_lockdown_release(dev, bar, gsp_falcon, args.boot_params_dma_handle())?;
-
-            Ok(unload_guard)
+            wait_for_gsp_lockdown_release(dev, bar, gsp_falcon, args.boot_params_dma_handle())
         })();
 
-        res
+        (res, unload_bundle)
     }
 }
 
drivers/gpu/nova-core/gsp/hal/tu102.rs +9 -14
index 9b24361f924b..fca8da281e16 100644
@@ -32,7 +32,6 @@
     },
     gpu::Chipset,
     gsp::{
-        boot::BootUnloadGuard,
         hal::{
             GspHal,
             UnloadBundle, //
@@ -264,19 +263,21 @@ fn run_fwsec_frts(
 struct Tu102;
 
 impl GspHal for Tu102 {
-    fn boot<'a>(
+    fn boot(
         &self,
-        gsp: &'a Gsp,
-        ctx: &GspBootContext<'a>,
+        gsp: &Gsp,
+        ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> Result<BootUnloadGuard<'a>> {
+    ) -> (Result, Option<crate::gsp::UnloadBundle>) {
         let dev = ctx.dev();
         let bar = ctx.bar;
         let chipset = ctx.chipset;
         let gsp_falcon = ctx.gsp_falcon;
         let sec2_falcon = ctx.sec2_falcon;
 
+        let mut unload_bundle = None;
+
         let res = (|| {
             let bios = Vbios::new(dev, bar)?;
 
@@ -284,7 +285,7 @@ fn boot<'a>(
             //
             // If the unload bundle creation fails, the GPU will need to be reset before the driver
             // can be probed again.
-            let unload_bundle =
+            unload_bundle =
                 Sec2UnloadBundle::build(dev, bar, chipset, &bios, gsp_falcon, sec2_falcon)
                     .inspect_err(|e| {
                         dev_warn!(dev, "Failed to prepare unload firmware: {:?}\n", e);
@@ -297,10 +298,6 @@ fn boot<'a>(
                     .ok()
                     .map(crate::gsp::UnloadBundle);
 
-            // Wrap the unload bundle into a drop guard so it is automatically run upon failure.
-            let unload_guard =
-                BootUnloadGuard::new(gsp, dev, bar, gsp_falcon, sec2_falcon, unload_bundle);
-
             // FWSEC-FRTS is not executed on chips where the FRTS region size is 0 (e.g. GA100).
             if !fb_layout.frts.is_empty() {
                 run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, fb_layout)?;
@@ -328,12 +325,10 @@ fn boot<'a>(
                 sec2_falcon,
                 bar,
             )?
-            .run(dev, bar, sec2_falcon, wpr_meta)?;
-
-            Ok(unload_guard)
+            .run(dev, bar, sec2_falcon, wpr_meta)
         })();
 
-        res
+        (res, unload_bundle)
     }
 
     fn post_boot(&self, gsp: &Gsp, ctx: &GspBootContext<'_>, gsp_fw: &GspFirmware) -> Result {
Signature · 1 line
2.54.0
author maintainer
#8 lore raw

PATCH v2 5/13 gpu: nova-core: gsp: move unload bundle error handling to Gsp::boot

In reply to Alexandre Courbot

The warning emitted when the unload bundle cannot be constructed is valid regardless of the boot method, but it was local to Tu102. Move it to Gsp::boot so it applies to all boot methods.

This requires the HAL's boot method to return the Result for the unload bundle's creation, so Gsp::boot can display the warning.

---
 drivers/gpu/nova-core/gsp/boot.rs      | 12 ++++++++++++
 drivers/gpu/nova-core/gsp/hal.rs       |  4 ++--
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  6 +++---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 13 ++-----------
 4 files changed, 19 insertions(+), 16 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +12 -0
index 3574f1a87344..3f11eaec26c7 100644
@@ -59,6 +59,18 @@ pub(crate) fn boot(
         // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
         let (res, unload_bundle) = hal.boot(&self, &ctx, &fb_layout, &wpr_meta);
 
+        // Display error for unload bundle if any, and convert to `Option`.
+        let unload_bundle = unload_bundle
+            .inspect_err(|e| {
+                dev_warn!(dev, "Failed to prepare unload firmware: {:?}\n", e);
+                dev_warn!(dev, "The GSP won't be able to unload properly on unbind.\n");
+                dev_warn!(
+                    dev,
+                    "The GPU will need to be reset before the driver can bind again.\n"
+                );
+            })
+            .ok();
+
         // Run from a closure so we can retrieve the result, and run the unload sequence of the GSP
         // in case of error.
         let res = res.and_then(|()| {
drivers/gpu/nova-core/gsp/hal.rs +2 -2
index a76be4e43272..00e39cc1fbde 100644
@@ -53,7 +53,7 @@ pub(super) trait GspHal: Send {
     /// Returns two things:
     ///
     /// - The `Result` of the boot process itself,
-    /// - The `UnloadBundle` to use with [`Gsp::unload`], or `None` if the bundle could not be
+    /// - The `UnloadBundle` to use with [`Gsp::unload`], or `Err` if the bundle could not be
     ///   created.
     ///
     /// Note that the two returned values are independent: it is possible for the boot process to
@@ -65,7 +65,7 @@ fn boot(
         ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> (Result, Option<crate::gsp::UnloadBundle>);
+    ) -> (Result, Result<crate::gsp::UnloadBundle>);
 
     /// Performs HAL-specific post-GSP boot tasks.
     ///
drivers/gpu/nova-core/gsp/hal/gh100.rs +3 -3
index ece40403b831..02203dfd3584 100644
@@ -150,16 +150,16 @@ fn boot(
         ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> (Result, Option<crate::gsp::UnloadBundle>) {
+    ) -> (Result, Result<crate::gsp::UnloadBundle>) {
         let dev = ctx.dev();
         let bar = ctx.bar;
         let chipset = ctx.chipset;
         let gsp_falcon = ctx.gsp_falcon;
 
-        let mut unload_bundle = None;
+        let mut unload_bundle = Err(EAGAIN);
 
         let res = (|| {
-            unload_bundle = Some(crate::gsp::UnloadBundle(
+            unload_bundle = Ok(crate::gsp::UnloadBundle(
                 KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>,
             ));
 
drivers/gpu/nova-core/gsp/hal/tu102.rs +2 -11
index fca8da281e16..984716fc0bf9 100644
@@ -269,14 +269,14 @@ fn boot(
         ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> (Result, Option<crate::gsp::UnloadBundle>) {
+    ) -> (Result, Result<crate::gsp::UnloadBundle>) {
         let dev = ctx.dev();
         let bar = ctx.bar;
         let chipset = ctx.chipset;
         let gsp_falcon = ctx.gsp_falcon;
         let sec2_falcon = ctx.sec2_falcon;
 
-        let mut unload_bundle = None;
+        let mut unload_bundle = Err(EAGAIN);
 
         let res = (|| {
             let bios = Vbios::new(dev, bar)?;
@@ -287,15 +287,6 @@ fn boot(
             // can be probed again.
             unload_bundle =
                 Sec2UnloadBundle::build(dev, bar, chipset, &bios, gsp_falcon, sec2_falcon)
-                    .inspect_err(|e| {
-                        dev_warn!(dev, "Failed to prepare unload firmware: {:?}\n", e);
-                        dev_warn!(dev, "The GSP won't be able to unload properly on unbind.\n");
-                        dev_warn!(
-                            dev,
-                            "The GPU will need to be reset before the driver can bind again.\n"
-                        );
-                    })
-                    .ok()
                     .map(crate::gsp::UnloadBundle);
 
             // FWSEC-FRTS is not executed on chips where the FRTS region size is 0 (e.g. GA100).
Signature · 1 line
2.54.0
#9 lore raw

Re: [PATCH v2 05/13] gpu: nova-core: gsp: move unload bundle error handling to Gsp::boot

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Early boot failures unconditionally trigger false warnings about unload firmware preparation and…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Early boot failures unconditionally trigger false warnings about unload firmware preparation and incorrectly state a GPU reset is required. --

commit 273d5dbf9e1e0ce85069f0026ccecdadd0a2d4fd
Author: Alexandre Courbot <[email protected]>

gpu: nova-core: gsp: move unload bundle error handling to Gsp::boot

This commit moves the warning for unload bundle creation failure from the Tu102 specific implementation to Gsp::boot so it applies to all boot methods. It changes the HAL boot method to return a Result for the unload bundle.

Quoted 3 lines
drivers/gpu/nova-core/gsp/boot.rs +0 -0

[ ... ]

drivers/gpu/nova-core/gsp/boot.rs pub(crate) fn boot( · line 64

Quoted 7 lines
drivers/gpu/nova-core/gsp/boot.rs +3 -0
@@ -59,6 +59,18 @@ pub(crate) fn boot(
         // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
         let (res, unload_bundle) = hal.boot(&self, &ctx, &fb_layout, &wpr_meta);
 
+        // Display error for unload bundle if any, and convert to Option.
+        let unload_bundle = unload_bundle
+            .inspect_err(|e| {

[Severity: Medium]
If the hal.boot() implementation fails early (for example, if Vbios::new() fails in Tu102::boot()), unload_bundle will retain its initial dummy error value of Err(EAGAIN).

Won't inspect_err() then unconditionally print these warnings about firmware preparation and the need for a GPU reset, even though the actual failure occurred before bundle preparation was attempted?

Quoted 12 lines
drivers/gpu/nova-core/gsp/boot.rs +9 -0
+                dev_warn!(dev, "Failed to prepare unload firmware: {:?}\n", e);
+                dev_warn!(dev, "The GSP won't be able to unload properly on unbind.\n");
+                dev_warn!(
+                    dev,
+                    "The GPU will need to be reset before the driver can bind again.\n"
+                );
+            })
+            .ok();
+
         // Run from a closure so we can retrieve the result, and run the unload sequence of the GSP
         // in case of error.
         let res = res.and_then(|()| {

[ ... ]

Quoted 3 lines
drivers/gpu/nova-core/gsp/hal/tu102.rs +0 -0

[ ... ]

Quoted 7 lines
drivers/gpu/nova-core/gsp/hal/tu102.rs +1 -1
@@ -269,14 +269,14 @@ fn boot(
         ctx: &GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-    ) -> (Result, Option<crate::gsp::UnloadBundle>) {
+    ) -> (Result, Result<crate::gsp::UnloadBundle>) {
         let dev = ctx.dev();

drivers/gpu/nova-core/gsp/hal/tu102.rs fn boot( · line 282

Quoted 10 lines
drivers/gpu/nova-core/gsp/hal/tu102.rs +1 -1
         let bar = ctx.bar;
         let chipset = ctx.chipset;
         let gsp_falcon = ctx.gsp_falcon;
         let sec2_falcon = ctx.sec2_falcon;
 
-        let mut unload_bundle = None;
+        let mut unload_bundle = Err(EAGAIN);
 
         let res = (|| {
             let bios = Vbios::new(dev, bar)?;

[Severity: Medium]
Here is where unload_bundle defaults to Err(EAGAIN). Is there a way to differentiate between an early boot failure where bundle creation wasn't reached, versus an actual failure during bundle creation, so that Gsp::boot() doesn't emit false warnings?

Quoted 1 line
drivers/gpu/nova-core/gsp/hal/tu102.rs +0 -0
@@ -287,15 +287,6 @@ fn boot(
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author maintainer
#10 lore raw

PATCH v2 6/13 gpu: nova-core: gsp: make unload take GspBootContext

In reply to Alexandre Courbot

GspBootContext contains the resources required to boot the GSP. As it turns out, this is also the context required for unloading it.

Reflect that fact by replacing the arguments of Gsp::unload with the GspBootContext. This symmetry between Gsp::boot and Gsp::unload will also be convenient when we want to make these methods generic over the boot context corresponding to the boot method used.

---
 drivers/gpu/nova-core/gpu.rs           | 19 ++++++++++++++++---
 drivers/gpu/nova-core/gsp/boot.rs      | 17 +++++++----------
 drivers/gpu/nova-core/gsp/hal.rs       | 15 +--------------
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 13 +++----------
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 20 +++++++++-----------
 5 files changed, 36 insertions(+), 48 deletions(-)
drivers/gpu/nova-core/gpu.rs +16 -3
index a34114d3afcb..7918ebb508f9 100644
@@ -268,7 +268,9 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
 #[pin_data(PinnedDrop)]
 struct GspResources<'gpu> {
     /// Device owning the GPU.
-    device: &'gpu device::Device<device::Bound>,
+    device: &'gpu pci::Device<device::Bound>,
+    /// Details about the chipset.
+    spec: Spec,
     /// MMIO mapping of PCI BAR 0.
     bar: Bar0<'gpu>,
     /// GSP falcon instance, used for GSP boot up and cleanup.
@@ -311,7 +313,16 @@ fn drop(self: Pin<&mut Self>) {
             .gsp
             .as_ref()
             .get_ref()
-            .unload(device, bar, &*this.gsp_falcon, &*this.sec2_falcon, bundle)
+            .unload(
+                GspBootContext {
+                    pdev: device,
+                    bar,
+                    chipset: this.spec.chipset,
+                    gsp_falcon: &*this.gsp_falcon,
+                    sec2_falcon: &*this.sec2_falcon,
+                },
+                bundle,
+            )
             .inspect_err(|e| dev_err!(device, "failed to unload GSP: {:?}\n", e));
     }
 }
@@ -343,7 +354,9 @@ pub(crate) fn new(
             sysmem_flush: SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?,
 
             gsp_resources <- try_pin_init!(GspResources {
-                device: pdev.as_ref(),
+                device: pdev,
+
+                spec: *spec,
 
                 bar,
 
drivers/gpu/nova-core/gsp/boot.rs +7 -10
index 3f11eaec26c7..336ad23c96f9 100644
@@ -3,7 +3,6 @@
 
 use kernel::{
     bits,
-    device,
     dma::Coherent,
     io::poll::read_poll_timeout,
     prelude::*,
@@ -14,7 +13,6 @@
     driver::Bar0,
     falcon::{
         gsp::Gsp,
-        sec2::Sec2,
         Falcon, //
     },
     fb::FbLayout,
@@ -102,7 +100,7 @@ pub(crate) fn boot(
                 dev_err!(dev, "GSP boot failed with error {:?}\n", e);
 
                 // Ignore errors during unload; we will return the error that happened during boot.
-                let _ = self.unload(dev, bar, gsp_falcon, ctx.sec2_falcon, unload_bundle);
+                let _ = self.unload(ctx, unload_bundle);
 
                 Err(e)
             }
@@ -136,17 +134,16 @@ fn shutdown_gsp(
     /// This stops all activity on the GSP.
     pub(crate) fn unload(
         &self,
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        gsp_falcon: &Falcon<Gsp>,
-        sec2_falcon: &Falcon<Sec2>,
+        ctx: super::GspBootContext<'_>,
         unload_bundle: Option<super::UnloadBundle>,
     ) -> Result {
+        let dev = ctx.dev();
+
         // Shut down the GSP. Keep going even in case of error.
         let mut res = Self::shutdown_gsp(
             &self.cmdq,
-            bar,
-            gsp_falcon,
+            ctx.bar,
+            ctx.gsp_falcon,
             commands::PowerStateLevel::Level0,
         )
         .inspect_err(|e| dev_err!(dev, "GSP shutdown failed: {:?}\n", e));
@@ -156,7 +153,7 @@ pub(crate) fn unload(
             res = res.and(
                 unload_bundle
                     .0
-                    .run(dev, bar, gsp_falcon, sec2_falcon)
+                    .run(&ctx)
                     .inspect_err(|e| dev_err!(dev, "Unload bundle failed: {:?}\n", e)),
             );
         } else {
drivers/gpu/nova-core/gsp/hal.rs +1 -14
index 00e39cc1fbde..113d445239b9 100644
@@ -5,18 +5,11 @@
 mod tu102;
 
 use kernel::{
-    device,
     dma::Coherent,
     prelude::*, //
 };
 
 use crate::{
-    driver::Bar0,
-    falcon::{
-        gsp::Gsp as GspEngine,
-        sec2::Sec2,
-        Falcon, //
-    },
     fb::FbLayout,
     firmware::gsp::GspFirmware,
     gpu::{
@@ -37,13 +30,7 @@
 /// required for unloading is prepared at load time, and stored here until it needs to be run.
 pub(super) trait UnloadBundle: Send {
     /// Performs the steps required to properly reset the GSP after it has been stopped.
-    fn run(
-        &self,
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        gsp_falcon: &Falcon<GspEngine>,
-        sec2_falcon: &Falcon<Sec2>,
-    ) -> Result;
+    fn run(&self, ctx: &GspBootContext<'_>) -> Result;
 }
 
 /// Trait implemented by GSP HALs.
drivers/gpu/nova-core/gsp/hal/gh100.rs +3 -10
index 02203dfd3584..a87d526d2310 100644
@@ -14,7 +14,6 @@
     driver::Bar0,
     falcon::{
         gsp::Gsp as GspEngine,
-        sec2::Sec2,
         Falcon, //
     },
     fb::FbLayout,
@@ -118,22 +117,16 @@ fn wait_for_gsp_lockdown_release(
 struct FspUnloadBundle;
 
 impl UnloadBundle for FspUnloadBundle {
-    fn run(
-        &self,
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        gsp_falcon: &Falcon<GspEngine>,
-        _sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    fn run(&self, ctx: &GspBootContext<'_>) -> Result {
         // GSP falcon does most of the work of resetting, so just wait for it to finish.
         read_poll_timeout(
-            || Ok(gsp_falcon.is_riscv_active(bar)),
+            || Ok(ctx.gsp_falcon.is_riscv_active(ctx.bar)),
             |&active| !active,
             Delta::from_millis(10),
             Delta::from_secs(5),
         )
         .map(|_| ())
-        .inspect_err(|_| dev_err!(dev, "GSP falcon failed to halt\n"))
+        .inspect_err(|_| dev_err!(ctx.dev(), "GSP falcon failed to halt\n"))
     }
 }
 
drivers/gpu/nova-core/gsp/hal/tu102.rs +9 -11
index 984716fc0bf9..7cb322b0e31d 100644
@@ -123,18 +123,15 @@ fn build(
 }
 
 impl UnloadBundle for Sec2UnloadBundle {
-    fn run(
-        &self,
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        gsp_falcon: &Falcon<GspEngine>,
-        sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    fn run(&self, ctx: &GspBootContext<'_>) -> Result {
+        let dev = ctx.dev();
+        let bar = ctx.bar;
+
         // Run FWSEC-SB to reset the GSP falcon to its pre-libos state.
         // Log errors but keep going if it fails.
         let fwsec_sb_res = self
             .fwsec_sb
-            .run(dev, bar, gsp_falcon)
+            .run(dev, bar, ctx.gsp_falcon)
             .inspect_err(|e| dev_err!(dev, "FWSEC-SB failed to run: {:?}\n", e));
 
         // Remove WPR2 region if set.
@@ -144,13 +141,14 @@ fn run(
                 return Ok(());
             }
 
-            sec2_falcon.reset(bar)?;
-            sec2_falcon.load(dev, bar, &self.booter_unloader)?;
+            ctx.sec2_falcon.reset(bar)?;
+            ctx.sec2_falcon.load(dev, bar, &self.booter_unloader)?;
 
             // Sentinel value to confirm that Booter Unloader has run.
             const MAILBOX_SENTINEL: u32 = 0xff;
             let (mbox0, _) =
-                sec2_falcon.boot(bar, Some(MAILBOX_SENTINEL), Some(MAILBOX_SENTINEL))?;
+                ctx.sec2_falcon
+                    .boot(bar, Some(MAILBOX_SENTINEL), Some(MAILBOX_SENTINEL))?;
             if mbox0 != 0 {
                 dev_err!(dev, "Booter Unloader returned error 0x{:x}\n", mbox0);
                 return Err(EINVAL);
Signature · 1 line
2.54.0
#11 lore raw

Re: [PATCH v2 06/13] gpu: nova-core: gsp: make unload take GspBootContext

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Early failures in `GspHal::boot` return a dummy `EAGAIN` error for the unload bundle, triggering…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Early failures in GspHal::boot return a dummy EAGAIN error for the unload bundle, triggering misleading warnings in Gsp::boot. --

commit 59b7c0f9ed708bf2f09f8548faf948cfa1adfc96
Author: Alexandre Courbot <[email protected]>

gpu: nova-core: gsp: make unload take GspBootContext

This patch refactors Gsp::unload to accept GspBootContext instead of individual resource arguments. This reflects that the resources required to boot the GSP are also needed for unloading it.

Quoted 4 lines
drivers/gpu/nova-core/gsp/boot.rs +0 -0
index 3f11eaec26c78..336ad23c96f9b 100644

[ ... ]

drivers/gpu/nova-core/gsp/boot.rs pub(crate) fn boot( · line 103

Quoted 6 lines
drivers/gpu/nova-core/gsp/boot.rs +1 -1
@@ -102,7 +100,7 @@ pub(crate) fn boot(
                 dev_err!(dev, "GSP boot failed with error {:?}\n", e);
 
                 // Ignore errors during unload; we will return the error that happened during boot.
-                let _ = self.unload(dev, bar, gsp_falcon, ctx.sec2_falcon, unload_bundle);
+                let _ = self.unload(ctx, unload_bundle);

[Severity: Low]
If an early operation in the HAL boot paths fails before unload_bundle is properly initialized (such as Vbios::new() failing in Tu102::boot(), or missing fsp in Gh100::boot()), does unload_bundle retain a dummy Err(EAGAIN) value?

If so, does this cause the inspect_err() on unload_bundle to unconditionally trigger and log misleading warnings that falsely claim the GPU will need a reset even for these early initialization failures?

Quoted 3 lines
drivers/gpu/nova-core/gsp/boot.rs +0 -0
                 Err(e)
             }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author maintainer
#12 lore raw

PATCH v2 7/13 gpu: nova-core: gsp: fold TU102 unload bundle construction into HAL method

In reply to Alexandre Courbot

The construction of the unload bundle is currently a bit convoluted and could be done in one function instead of two.

Additionally, turn that function into a method of Tu102. A following patch will turn the "use FWSEC bootloader" property into a flag of the TU102 HAL itself, and making this a method will allow the code to access it instead of querying Chipset.

---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 90 ++++++++++++++++------------------
 1 file changed, 42 insertions(+), 48 deletions(-)
drivers/gpu/nova-core/gsp/hal/tu102.rs +42 -48
index 7cb322b0e31d..230145146540 100644
@@ -55,23 +55,6 @@ enum FwsecUnloadFirmware {
 }
 
 impl FwsecUnloadFirmware {
-    /// Loads the FWSEC SB firmware, as well as its bootloader if `chipset` requires it.
-    fn new(
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        chipset: Chipset,
-        bios: &Vbios,
-        gsp_falcon: &Falcon<GspEngine>,
-    ) -> Result<Self> {
-        let fwsec_sb = FwsecFirmware::new(dev, gsp_falcon, bar, bios, FwsecCommand::Sb)?;
-
-        Ok(if chipset.needs_fwsec_bootloader() {
-            Self::WithBl(FwsecFirmwareWithBl::new(fwsec_sb, dev, chipset)?)
-        } else {
-            Self::WithoutBl(fwsec_sb)
-        })
-    }
-
     /// Runs the FWSEC SB firmware.
     fn run(
         &self,
@@ -93,35 +76,6 @@ struct Sec2UnloadBundle {
     booter_unloader: BooterFirmware,
 }
 
-impl Sec2UnloadBundle {
-    /// Load and prepare the resources required to properly reset the GSP after it has been stopped.
-    fn build(
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        chipset: Chipset,
-        bios: &Vbios,
-        gsp_falcon: &Falcon<GspEngine>,
-        sec2_falcon: &Falcon<Sec2>,
-    ) -> Result<KBox<dyn UnloadBundle>> {
-        KBox::new(
-            Self {
-                fwsec_sb: FwsecUnloadFirmware::new(dev, bar, chipset, bios, gsp_falcon)?,
-                booter_unloader: BooterFirmware::new(
-                    dev,
-                    BooterKind::Unloader,
-                    chipset,
-                    FIRMWARE_VERSION,
-                    sec2_falcon,
-                    bar,
-                )?,
-            },
-            GFP_KERNEL,
-        )
-        .map(|b| b as KBox<dyn UnloadBundle>)
-        .map_err(Into::into)
-    }
-}
-
 impl UnloadBundle for Sec2UnloadBundle {
     fn run(&self, ctx: &GspBootContext<'_>) -> Result {
         let dev = ctx.dev();
@@ -260,6 +214,47 @@ fn run_fwsec_frts(
 
 struct Tu102;
 
+impl Tu102 {
+    /// Load and prepare the resources required to properly reset the GSP after it has been stopped.
+    fn build_unload_bundle(
+        &self,
+        dev: &device::Device<device::Bound>,
+        bar: Bar0<'_>,
+        chipset: Chipset,
+        bios: &Vbios,
+        gsp_falcon: &Falcon<GspEngine>,
+        sec2_falcon: &Falcon<Sec2>,
+    ) -> Result<crate::gsp::UnloadBundle> {
+        // Load the FWSEC SB firmware, as well as its bootloader if required.
+        let fwsec_sb = FwsecFirmware::new(dev, gsp_falcon, bar, bios, FwsecCommand::Sb).and_then(
+            |fwsec_sb| {
+                Ok(if chipset.needs_fwsec_bootloader() {
+                    FwsecUnloadFirmware::WithBl(FwsecFirmwareWithBl::new(fwsec_sb, dev, chipset)?)
+                } else {
+                    FwsecUnloadFirmware::WithoutBl(fwsec_sb)
+                })
+            },
+        )?;
+
+        KBox::new(
+            Sec2UnloadBundle {
+                fwsec_sb,
+                booter_unloader: BooterFirmware::new(
+                    dev,
+                    BooterKind::Unloader,
+                    chipset,
+                    FIRMWARE_VERSION,
+                    sec2_falcon,
+                    bar,
+                )?,
+            },
+            GFP_KERNEL,
+        )
+        .map(|b| crate::gsp::UnloadBundle(b))
+        .map_err(Into::into)
+    }
+}
+
 impl GspHal for Tu102 {
     fn boot(
         &self,
@@ -284,8 +279,7 @@ fn boot(
             // If the unload bundle creation fails, the GPU will need to be reset before the driver
             // can be probed again.
             unload_bundle =
-                Sec2UnloadBundle::build(dev, bar, chipset, &bios, gsp_falcon, sec2_falcon)
-                    .map(crate::gsp::UnloadBundle);
+                self.build_unload_bundle(dev, bar, chipset, &bios, gsp_falcon, sec2_falcon);
 
             // FWSEC-FRTS is not executed on chips where the FRTS region size is 0 (e.g. GA100).
             if !fb_layout.frts.is_empty() {
Signature · 1 line
2.54.0
author maintainer
#13 lore raw

PATCH v2 8/13 gpu: nova-core: gsp: turn FWSEC execution into HAL method

In reply to Alexandre Courbot

Turn the run_fwsec_frts function into a method of Tu102. A following patch will turn the "use FWSEC bootloader" property into a flag of the TU102 HAL itself, and making this a method will allow the code to access it instead of querying Chipset.

---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 175 +++++++++++++++++----------------
 1 file changed, 88 insertions(+), 87 deletions(-)
drivers/gpu/nova-core/gsp/hal/tu102.rs +88 -87
index 230145146540..fb0fc99b492b 100644
@@ -126,95 +126,96 @@ fn run(&self, ctx: &GspBootContext<'_>) -> Result {
     }
 }
 
-/// Helper function to load and run the FWSEC-FRTS firmware and confirm that it has properly
-/// created the WPR2 region.
-fn run_fwsec_frts(
-    dev: &device::Device<device::Bound>,
-    chipset: Chipset,
-    falcon: &Falcon<GspEngine>,
-    bar: Bar0<'_>,
-    bios: &Vbios,
-    fb_layout: &FbLayout,
-) -> Result {
-    // Check that the WPR2 region does not already exist - if it does, we cannot run
-    // FWSEC-FRTS until the GPU is reset.
-    if bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound() != 0 {
-        dev_err!(
-            dev,
-            "WPR2 region already exists - GPU needs to be reset to proceed\n"
-        );
-        return Err(EBUSY);
-    }
-
-    // FWSEC-FRTS will create the WPR2 region.
-    let fwsec_frts = FwsecFirmware::new(
-        dev,
-        falcon,
-        bar,
-        bios,
-        FwsecCommand::Frts {
-            frts_addr: fb_layout.frts.start,
-            frts_size: fb_layout.frts.len(),
-        },
-    )?;
-
-    if chipset.needs_fwsec_bootloader() {
-        let fwsec_frts_bl = FwsecFirmwareWithBl::new(fwsec_frts, dev, chipset)?;
-        // Load and run the bootloader, which will load FWSEC-FRTS and run it.
-        fwsec_frts_bl.run(dev, falcon, bar)?;
-    } else {
-        // Load and run FWSEC-FRTS directly.
-        fwsec_frts.run(dev, falcon, bar)?;
-    }
-
-    // SCRATCH_E contains the error code for FWSEC-FRTS.
-    let frts_status = bar
-        .read(regs::NV_PBUS_SW_SCRATCH_0E_FRTS_ERR)
-        .frts_err_code();
-    if frts_status != 0 {
-        dev_err!(
-            dev,
-            "FWSEC-FRTS returned with error code {:#x}\n",
-            frts_status
-        );
-
-        return Err(EIO);
-    }
-
-    // Check that the WPR2 region has been created as we requested.
-    let (wpr2_lo, wpr2_hi) = (
-        bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_LO).lower_bound(),
-        bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound(),
-    );
-
-    match (wpr2_lo, wpr2_hi) {
-        (_, 0) => {
-            dev_err!(dev, "WPR2 region not created after running FWSEC-FRTS\n");
-
-            Err(EIO)
-        }
-        (wpr2_lo, _) if wpr2_lo != fb_layout.frts.start => {
-            dev_err!(
-                dev,
-                "WPR2 region created at unexpected address {:#x}; expected {:#x}\n",
-                wpr2_lo,
-                fb_layout.frts.start,
-            );
-
-            Err(EIO)
-        }
-        (wpr2_lo, wpr2_hi) => {
-            dev_dbg!(dev, "WPR2: {:#x}-{:#x}\n", wpr2_lo, wpr2_hi);
-            dev_dbg!(dev, "GPU instance built\n");
-
-            Ok(())
-        }
-    }
-}
-
 struct Tu102;
 
 impl Tu102 {
+    /// Helper method to load and run the FWSEC-FRTS firmware and confirm that it has properly
+    /// created the WPR2 region.
+    fn run_fwsec_frts(
+        &self,
+        dev: &device::Device<device::Bound>,
+        chipset: Chipset,
+        falcon: &Falcon<GspEngine>,
+        bar: Bar0<'_>,
+        bios: &Vbios,
+        fb_layout: &FbLayout,
+    ) -> Result {
+        // Check that the WPR2 region does not already exist - if it does, we cannot run
+        // FWSEC-FRTS until the GPU is reset.
+        if bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound() != 0 {
+            dev_err!(
+                dev,
+                "WPR2 region already exists - GPU needs to be reset to proceed\n"
+            );
+            return Err(EBUSY);
+        }
+
+        // FWSEC-FRTS will create the WPR2 region.
+        let fwsec_frts = FwsecFirmware::new(
+            dev,
+            falcon,
+            bar,
+            bios,
+            FwsecCommand::Frts {
+                frts_addr: fb_layout.frts.start,
+                frts_size: fb_layout.frts.len(),
+            },
+        )?;
+
+        if chipset.needs_fwsec_bootloader() {
+            let fwsec_frts_bl = FwsecFirmwareWithBl::new(fwsec_frts, dev, chipset)?;
+            // Load and run the bootloader, which will load FWSEC-FRTS and run it.
+            fwsec_frts_bl.run(dev, falcon, bar)?;
+        } else {
+            // Load and run FWSEC-FRTS directly.
+            fwsec_frts.run(dev, falcon, bar)?;
+        }
+
+        // SCRATCH_E contains the error code for FWSEC-FRTS.
+        let frts_status = bar
+            .read(regs::NV_PBUS_SW_SCRATCH_0E_FRTS_ERR)
+            .frts_err_code();
+        if frts_status != 0 {
+            dev_err!(
+                dev,
+                "FWSEC-FRTS returned with error code {:#x}\n",
+                frts_status
+            );
+
+            return Err(EIO);
+        }
+
+        // Check that the WPR2 region has been created as we requested.
+        let (wpr2_lo, wpr2_hi) = (
+            bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_LO).lower_bound(),
+            bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound(),
+        );
+
+        match (wpr2_lo, wpr2_hi) {
+            (_, 0) => {
+                dev_err!(dev, "WPR2 region not created after running FWSEC-FRTS\n");
+
+                Err(EIO)
+            }
+            (wpr2_lo, _) if wpr2_lo != fb_layout.frts.start => {
+                dev_err!(
+                    dev,
+                    "WPR2 region created at unexpected address {:#x}; expected {:#x}\n",
+                    wpr2_lo,
+                    fb_layout.frts.start,
+                );
+
+                Err(EIO)
+            }
+            (wpr2_lo, wpr2_hi) => {
+                dev_dbg!(dev, "WPR2: {:#x}-{:#x}\n", wpr2_lo, wpr2_hi);
+                dev_dbg!(dev, "GPU instance built\n");
+
+                Ok(())
+            }
+        }
+    }
+
     /// Load and prepare the resources required to properly reset the GSP after it has been stopped.
     fn build_unload_bundle(
         &self,
@@ -283,7 +284,7 @@ fn boot(
 
             // FWSEC-FRTS is not executed on chips where the FRTS region size is 0 (e.g. GA100).
             if !fb_layout.frts.is_empty() {
-                run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, fb_layout)?;
+                self.run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, fb_layout)?;
             }
 
             gsp_falcon.reset(bar)?;
Signature · 1 line
2.54.0
author maintainer
#14 lore raw

PATCH v2 9/13 gpu: nova-core: gsp: make use of FWSEC bootloader a property of the TU102 HAL

In reply to Alexandre Courbot

By being in the TU102 HAL, we already know that the GSP boot method is the SEC2 Booter, so the only variable is whether the FWSEC bootloader is used or not. Since Chipset also includes the variants that boot FSP, querying it for that information introduces a potential code path where the chipset actually boots via FSP that the current code doesn't handle.

Turn the use of FWSEC bootloader into a property of the Tu102 HAL, and give Ampere chipsets their own instance with that property set to false. This removes the invalid code path and the only use of Chipset is now to load the correct firmware files.

This also removes some uses of the Chipset::needs_fwsec_bootloader method and prepares the ground for removing it.

---
 drivers/gpu/nova-core/gsp/hal.rs       |  5 ++++-
 drivers/gpu/nova-core/gsp/hal/ga102.rs | 14 ++++++++++++++
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 15 +++++++++++----
 3 files changed, 29 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/gsp/hal.rs +4 -1
index 113d445239b9..ae4c44aeddaa 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
+mod ga102;
 mod gh100;
 mod tu102;
 
@@ -66,7 +67,9 @@ fn post_boot(&self, _gsp: &Gsp, _ctx: &GspBootContext<'_>, _gsp_fw: &GspFirmware
 /// Returns the GSP HAL to be used for `chipset`.
 pub(super) fn gsp_hal(chipset: Chipset) -> &'static dyn GspHal {
     match chipset.arch() {
-        Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
+        Architecture::Turing => tu102::TU102_HAL,
+        Architecture::Ampere if matches!(chipset, Chipset::GA100) => tu102::TU102_HAL,
+        Architecture::Ampere | Architecture::Ada => ga102::GA102_HAL,
         Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
             gh100::GH100_HAL
         }
drivers/gpu/nova-core/gsp/hal/ga102.rs new file +14 -0
new file mode 100644
index 000000000000..ceb3eb39d138
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use crate::gsp::hal::{
+    tu102::Tu102,
+    GspHal, //
+};
+
+/// The GA102 HAL is like the TU102 one, except it doesn't use the bootloader.
+const GA102: Tu102 = Tu102 {
+    needs_fwsec_bootloader: false,
+};
+
+pub(super) const GA102_HAL: &dyn GspHal = &GA102;
drivers/gpu/nova-core/gsp/hal/tu102.rs +11 -4
index fb0fc99b492b..8e732f540af2 100644
@@ -126,7 +126,10 @@ fn run(&self, ctx: &GspBootContext<'_>) -> Result {
     }
 }
 
-struct Tu102;
+pub(super) struct Tu102 {
+    /// If `true`, then the FWSEC-FRTS bootloader will be used to load the actual firmware.
+    pub(super) needs_fwsec_bootloader: bool,
+}
 
 impl Tu102 {
     /// Helper method to load and run the FWSEC-FRTS firmware and confirm that it has properly
@@ -162,7 +165,7 @@ fn run_fwsec_frts(
             },
         )?;
 
-        if chipset.needs_fwsec_bootloader() {
+        if self.needs_fwsec_bootloader {
             let fwsec_frts_bl = FwsecFirmwareWithBl::new(fwsec_frts, dev, chipset)?;
             // Load and run the bootloader, which will load FWSEC-FRTS and run it.
             fwsec_frts_bl.run(dev, falcon, bar)?;
@@ -229,7 +232,7 @@ fn build_unload_bundle(
         // Load the FWSEC SB firmware, as well as its bootloader if required.
         let fwsec_sb = FwsecFirmware::new(dev, gsp_falcon, bar, bios, FwsecCommand::Sb).and_then(
             |fwsec_sb| {
-                Ok(if chipset.needs_fwsec_bootloader() {
+                Ok(if self.needs_fwsec_bootloader {
                     FwsecUnloadFirmware::WithBl(FwsecFirmwareWithBl::new(fwsec_sb, dev, chipset)?)
                 } else {
                     FwsecUnloadFirmware::WithoutBl(fwsec_sb)
@@ -327,5 +330,9 @@ fn post_boot(&self, gsp: &Gsp, ctx: &GspBootContext<'_>, gsp_fw: &GspFirmware) -
     }
 }
 
-const TU102: Tu102 = Tu102;
+/// The TU102 HAL requires the use of the FWSEC bootloader.
+const TU102: Tu102 = Tu102 {
+    needs_fwsec_bootloader: true,
+};
+
 pub(super) const TU102_HAL: &dyn GspHal = &TU102;
Signature · 1 line
2.54.0
author maintainer
#15 lore raw

PATCH v2 10/13 gpu: nova-core: introduce GspBootMethod

In reply to Alexandre Courbot

The GSP boot method is currently determined by two ad-hoc methods of Chipset: uses_fsp (a boolean telling whether to use the FSP boot path or the Sec2 Booter one) and needs_fwsec_bootloader (another boolean valid only for the Sec2 Booter method that tells whether the FWSEC bootloader must be used).

This is neither extensible nor sound: the combination uses_fsp && needs_fwsec_bootloader is invalid, but can still be expressed.

Thus, unify these two predicates into a single gsp_boot_method method that returns an enum type unambiguously describing the boot method to use. This ensures that no invalid combination can be expressed, which makes matching sounder.

---
 drivers/gpu/nova-core/firmware.rs       | 25 ++++++++++++--------
 drivers/gpu/nova-core/firmware/fwsec.rs |  5 ++--
 drivers/gpu/nova-core/gpu.rs            | 16 -------------
 drivers/gpu/nova-core/gsp.rs            | 42 ++++++++++++++++++++++++++++++++-
 drivers/gpu/nova-core/gsp/hal.rs        | 22 ++++++++---------
 5 files changed, 70 insertions(+), 40 deletions(-)
drivers/gpu/nova-core/firmware.rs +15 -10
index 15a61edaaa82..0b7047646bf2 100644
@@ -21,6 +21,7 @@
         FalconFirmware, //
     },
     gpu,
+    gsp::GspBootMethod,
     num::{
         FromSafeCast,
         IntoSafeCast, //
@@ -432,17 +433,21 @@ const fn make_entry_chipset(self, chipset: gpu::Chipset) -> Self {
 
         // FSP-based chipsets (Hopper, Blackwell and later) boot the GSP via the FMC image loaded by
         // FSP. Older chipsets use the SEC2 booter instead.
-        let this = if chipset.uses_fsp() {
-            this.make_entry_file(name, "fmc")
-        } else {
-            this.make_entry_file(name, "booter_load")
-                .make_entry_file(name, "booter_unload")
-        };
+        match chipset.gsp_boot_method() {
+            GspBootMethod::Sec2 {
+                needs_fwsec_bootloader,
+            } => {
+                let mut this = this
+                    .make_entry_file(name, "booter_load")
+                    .make_entry_file(name, "booter_unload");
 
-        if chipset.needs_fwsec_bootloader() {
-            this.make_entry_file(name, "gen_bootloader")
-        } else {
-            this
+                if needs_fwsec_bootloader {
+                    this = this.make_entry_file(name, "gen_bootloader")
+                }
+
+                this
+            }
+            GspBootMethod::Fsp => this.make_entry_file(name, "fmc"),
         }
     }
 
drivers/gpu/nova-core/firmware/fwsec.rs +3 -2
index 199ae2adb664..a1cdf1aa8773 100644
@@ -388,8 +388,9 @@ pub(crate) fn new(
     /// Loads the FWSEC firmware into `falcon` and execute it.
     ///
     /// This must only be called on chipsets that do not need the FWSEC bootloader (i.e., where
-    /// [`Chipset::needs_fwsec_bootloader()`](crate::gpu::Chipset::needs_fwsec_bootloader) returns
-    /// `false`). On chipsets that do, use [`bootloader::FwsecFirmwareWithBl`] instead.
+    /// [`Chipset::gsp_boot_method()`](crate::gpu::Chipset::gsp_boot_method) returns
+    /// `GspBootMethod::Sec2 { needs_fwsec_bootloader: false }`). On chipsets where
+    /// `needs_fwsec_bootloader` is `true` do, use [`bootloader::FwsecFirmwareWithBl`] instead.
     pub(crate) fn run(
         &self,
         dev: &Device<device::Bound>,
drivers/gpu/nova-core/gpu.rs +0 -16
index 7918ebb508f9..d80c6dbb4cba 100644
@@ -132,22 +132,6 @@ pub(crate) const fn arch(self) -> Architecture {
         }
     }
 
-    /// Returns `true` if this chipset requires the PIO-loaded bootloader in order to boot FWSEC.
-    ///
-    /// This includes all chipsets < GA102.
-    pub(crate) const fn needs_fwsec_bootloader(self) -> bool {
-        matches!(self.arch(), Architecture::Turing) || matches!(self, Self::GA100)
-    }
-
-    /// Returns `true` if this chipset boots via FSP (Hopper and later), which requires the FMC
-    /// firmware image.
-    pub(crate) const fn uses_fsp(self) -> bool {
-        matches!(
-            self.arch(),
-            Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x
-        )
-    }
-
     /// Returns the address range of the PCI config mirror space.
     pub(crate) fn pci_config_mirror_range(self) -> Range<u32> {
         hal::gpu_hal(self).pci_config_mirror_range()
drivers/gpu/nova-core/gsp.rs +41 -1

53 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal.rs +11 -11
index ae4c44aeddaa..9abdafbdbb57 100644
@@ -13,13 +13,11 @@
 use crate::{
     fb::FbLayout,
     firmware::gsp::GspFirmware,
-    gpu::{
-        Architecture,
-        Chipset, //
-    },
+    gpu::Chipset,
     gsp::{
         Gsp,
         GspBootContext,
+        GspBootMethod,
         GspFwWprMeta, //
     },
 };
@@ -66,12 +64,14 @@ fn post_boot(&self, _gsp: &Gsp, _ctx: &GspBootContext<'_>, _gsp_fw: &GspFirmware
 
 /// Returns the GSP HAL to be used for `chipset`.
 pub(super) fn gsp_hal(chipset: Chipset) -> &'static dyn GspHal {
-    match chipset.arch() {
-        Architecture::Turing => tu102::TU102_HAL,
-        Architecture::Ampere if matches!(chipset, Chipset::GA100) => tu102::TU102_HAL,
-        Architecture::Ampere | Architecture::Ada => ga102::GA102_HAL,
-        Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
-            gh100::GH100_HAL
-        }
+    // The GSP HAL is entirely determined by the boot method.
+    match chipset.gsp_boot_method() {
+        GspBootMethod::Sec2 {
+            needs_fwsec_bootloader: true,
+        } => tu102::TU102_HAL,
+        GspBootMethod::Sec2 {
+            needs_fwsec_bootloader: false,
+        } => ga102::GA102_HAL,
+        GspBootMethod::Fsp => gh100::GH100_HAL,
     }
 }
Signature · 1 line
2.54.0
author maintainer
#16 lore raw

PATCH v2 11/13 gpu: nova-core: avoid repeated calls to pci::Device::as_ref

In reply to Alexandre Courbot

Add a local Device reference created from the pci::Device in the Gpu constructor to avoid repeatedly calling as_ref.

---
 drivers/gpu/nova-core/gpu.rs | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/gpu.rs +10 -8
index d80c6dbb4cba..2e76e4bf79b2 100644
@@ -316,9 +316,11 @@ pub(crate) fn new(
         pdev: &'gpu pci::Device<device::Core<'_>>,
         bar: Bar0<'gpu>,
     ) -> impl PinInit<Self, Error> + 'gpu {
+        let dev = pdev.as_ref();
+
         try_pin_init!(Self {
-            spec: Spec::new(pdev.as_ref(), bar).inspect(|spec| {
-                dev_info!(pdev,"NVIDIA ({})\n", spec);
+            spec: Spec::new(dev, bar).inspect(|spec| {
+                dev_info!(dev,"NVIDIA ({})\n", spec);
             })?,
 
             // We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
@@ -331,11 +333,11 @@ pub(crate) fn new(
                 unsafe { pdev.dma_set_mask_and_coherent(dma_mask)? };
 
                 hal.wait_gfw_boot_completion(bar)
-                    .inspect_err(|_| dev_err!(pdev, "GFW boot did not complete\n"))?;
+                    .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))?;
             },
 
             // Initialize this early because `gsp_resources` depends on it.
-            sysmem_flush: SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?,
+            sysmem_flush: SysmemFlush::register(dev, bar, spec.chipset)?,
 
             gsp_resources <- try_pin_init!(GspResources {
                 device: pdev,
@@ -345,12 +347,12 @@ pub(crate) fn new(
                 bar,
 
                 gsp_falcon: Falcon::new(
-                    pdev.as_ref(),
+                    dev,
                     spec.chipset,
                 )
                 .inspect(|falcon| falcon.clear_swgen0_intr(bar))?,
 
-                sec2_falcon: Falcon::new(pdev.as_ref(), spec.chipset)?,
+                sec2_falcon: Falcon::new(dev, spec.chipset)?,
 
                 gsp <- Gsp::new(pdev),
 
@@ -370,8 +372,8 @@ pub(crate) fn new(
                 // Obtain and display basic GPU information.
                 let info = gsp_resources.gsp.get_static_info(bar)?;
                 match info.gpu_name() {
-                    Ok(name) => dev_info!(pdev, "GPU name: {}\n", name),
-                    Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e),
+                    Ok(name) => dev_info!(dev, "GPU name: {}\n", name),
+                    Err(e) => dev_warn!(dev, "GPU name unavailable: {:?}\n", e),
                 }
 
                 info
Signature · 1 line
2.54.0
author maintainer
#17 lore raw

PATCH v2 12/13 gpu: nova-core: gsp: pass GspBootContext mutably

In reply to Alexandre Courbot

We want to move the Fsp instance into Gpu, which will require passing it as part of the GspBootContext as Fsp::boot_fmc is a mutable method.

Additionally, we will also want to make some methods of the Falcons mutable, which will also require passing them as mutable references.

To prepare for this, make the passed GspBootContext mutable, and pass mutable references to it to the GSP boot HAL methods.

---
 drivers/gpu/nova-core/gsp/boot.rs      | 6 +++---
 drivers/gpu/nova-core/gsp/hal.rs       | 9 +++++++--
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 2 +-
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 4 ++--
 4 files changed, 13 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +3 -3
index 336ad23c96f9..1e0d4793e96d 100644
@@ -38,7 +38,7 @@ impl super::Gsp {
     /// [`Self::unload`]) returned.
     pub(crate) fn boot(
         self: Pin<&mut Self>,
-        ctx: super::GspBootContext<'_>,
+        mut ctx: super::GspBootContext<'_>,
     ) -> Result<Option<super::UnloadBundle>> {
         let pdev = ctx.pdev;
         let bar = ctx.bar;
@@ -55,7 +55,7 @@ pub(crate) fn boot(
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
 
         // Perform the chipset-specific boot sequence, and retrieve the unload bundle.
-        let (res, unload_bundle) = hal.boot(&self, &ctx, &fb_layout, &wpr_meta);
+        let (res, unload_bundle) = hal.boot(&self, &mut ctx, &fb_layout, &wpr_meta);
 
         // Display error for unload bundle if any, and convert to `Option`.
         let unload_bundle = unload_bundle
@@ -89,7 +89,7 @@ pub(crate) fn boot(
             self.cmdq
                 .send_command_no_wait(bar, commands::SetRegistry::new())?;
 
-            hal.post_boot(&self, &ctx, &gsp_fw)?;
+            hal.post_boot(&self, &mut ctx, &gsp_fw)?;
 
             // Wait until GSP is fully initialized.
             commands::wait_gsp_init_done(&self.cmdq)
drivers/gpu/nova-core/gsp/hal.rs +7 -2
index 9abdafbdbb57..15c6d86c0d51 100644
@@ -48,7 +48,7 @@ pub(super) trait GspHal: Send {
     fn boot(
         &self,
         gsp: &Gsp,
-        ctx: &GspBootContext<'_>,
+        ctx: &mut GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
     ) -> (Result, Result<crate::gsp::UnloadBundle>);
@@ -57,7 +57,12 @@ fn boot(
     ///
     /// This method is called by the GSP boot code after the GSP is confirmed to be running, and
     /// after the initialization commands have been pushed onto its queue.
-    fn post_boot(&self, _gsp: &Gsp, _ctx: &GspBootContext<'_>, _gsp_fw: &GspFirmware) -> Result {
+    fn post_boot(
+        &self,
+        _gsp: &Gsp,
+        _ctx: &mut GspBootContext<'_>,
+        _gsp_fw: &GspFirmware,
+    ) -> Result {
         Ok(())
     }
 }
drivers/gpu/nova-core/gsp/hal/gh100.rs +1 -1
index a87d526d2310..7bba18ba2f75 100644
@@ -140,7 +140,7 @@ impl GspHal for Gh100 {
     fn boot(
         &self,
         gsp: &Gsp,
-        ctx: &GspBootContext<'_>,
+        ctx: &mut GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
     ) -> (Result, Result<crate::gsp::UnloadBundle>) {
drivers/gpu/nova-core/gsp/hal/tu102.rs +2 -2
index 8e732f540af2..90c734c2f63e 100644
@@ -263,7 +263,7 @@ impl GspHal for Tu102 {
     fn boot(
         &self,
         gsp: &Gsp,
-        ctx: &GspBootContext<'_>,
+        ctx: &mut GspBootContext<'_>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
     ) -> (Result, Result<crate::gsp::UnloadBundle>) {
@@ -318,7 +318,7 @@ fn boot(
         (res, unload_bundle)
     }
 
-    fn post_boot(&self, gsp: &Gsp, ctx: &GspBootContext<'_>, gsp_fw: &GspFirmware) -> Result {
+    fn post_boot(&self, gsp: &Gsp, ctx: &mut GspBootContext<'_>, gsp_fw: &GspFirmware) -> Result {
         GspSequencer::run(
             &gsp.cmdq,
             ctx,
Signature · 1 line
2.54.0
author maintainer
#18 lore raw

PATCH v2 13/13 gpu: nova-core: store Fsp instance in Gpu

In reply to Alexandre Courbot

The Fsp instance was only used in the Hopper+ boot path, and consequently built locally (and immediately dropped) in it.

This worked well as a temporary measure, but the FSP is also needed in other parts of the driver, for instance vGPU.

Thus, create the Fsp instance in the Gpu constructor and store it there, passing it to the GSP boot as a mutable reference using GspBootContext. This makes the Fsp available even after the GSP is booted.

---
 drivers/gpu/nova-core/gpu.rs           | 15 ++++++++++++++-
 drivers/gpu/nova-core/gsp.rs           |  2 ++
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  9 +++------
 3 files changed, 19 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/gpu.rs +14 -1
index 2e76e4bf79b2..e5ebd79c9020 100644
@@ -21,11 +21,13 @@
         Falcon, //
     },
     fb::SysmemFlush,
+    fsp::Fsp,
     gsp::{
         self,
         commands::GetGspStaticInfoReply,
         Gsp,
-        GspBootContext, //
+        GspBootContext,
+        GspBootMethod, //
     },
     regs,
 };
@@ -261,6 +263,10 @@ struct GspResources<'gpu> {
     gsp_falcon: Falcon<GspFalcon>,
     /// SEC2 falcon instance, used for GSP boot up and cleanup.
     sec2_falcon: Falcon<Sec2Falcon>,
+    /// FSP instance, if on an arch that supports it.
+    // TODO: use different resource types for each boot method, and make the relevant Gsp methods
+    // generic against them.
+    fsp: Option<Fsp>,
     /// GSP runtime data.
     #[pin]
     gsp: Gsp,
@@ -304,6 +310,7 @@ fn drop(self: Pin<&mut Self>) {
                     chipset: this.spec.chipset,
                     gsp_falcon: &*this.gsp_falcon,
                     sec2_falcon: &*this.sec2_falcon,
+                    fsp: this.fsp.as_mut(),
                 },
                 bundle,
             )
@@ -354,6 +361,11 @@ pub(crate) fn new(
 
                 sec2_falcon: Falcon::new(dev, spec.chipset)?,
 
+                fsp: match spec.chipset.gsp_boot_method() {
+                    GspBootMethod::Sec2 { .. } => None,
+                    GspBootMethod::Fsp => Some(Fsp::wait_secure_boot(dev, bar, spec.chipset)?),
+                },
+
                 gsp <- Gsp::new(pdev),
 
                 // This member must be initialized last, so the `UnloadBundle` can never be dropped
@@ -365,6 +377,7 @@ pub(crate) fn new(
                     chipset: spec.chipset,
                     gsp_falcon,
                     sec2_falcon,
+                    fsp: fsp.as_mut(),
                 })?,
             }),
 
drivers/gpu/nova-core/gsp.rs +2 -0
index 771b38e6335d..ff438506070a 100644
@@ -38,6 +38,7 @@
         sec2::Sec2 as Sec2Falcon,
         Falcon, //
     },
+    fsp::Fsp,
     gpu::{
         Architecture,
         Chipset, //
@@ -62,6 +63,7 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) chipset: Chipset,
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
+    pub(crate) fsp: Option<&'a mut Fsp>,
 }
 
 impl<'a> GspBootContext<'a> {
drivers/gpu/nova-core/gsp/hal/gh100.rs +3 -6
index 7bba18ba2f75..8673a749dbac 100644
@@ -17,10 +17,7 @@
         Falcon, //
     },
     fb::FbLayout,
-    fsp::{
-        FmcBootArgs,
-        Fsp, //
-    },
+    fsp::FmcBootArgs,
     gsp::{
         hal::{
             GspHal,
@@ -152,12 +149,12 @@ fn boot(
         let mut unload_bundle = Err(EAGAIN);
 
         let res = (|| {
+            let fsp = ctx.fsp.as_mut().ok_or(ENODEV)?;
+
             unload_bundle = Ok(crate::gsp::UnloadBundle(
                 KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>,
             ));
 
-            let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset)?;
-
             let args = FmcBootArgs::new(
                 dev,
                 chipset,
Signature · 1 line
2.54.0
#19 lore raw

Re: [PATCH v2 13/13] gpu: nova-core: store Fsp instance in Gpu

Quoted 5 lines · Alexandre Courbot

The Fsp instance was only used in the Hopper+ boot path, and consequently built locally (and immediately dropped) in it.

This worked well as a temporary measure, but the FSP is also needed in other parts of the driver, for instance vGPU.

Or: "will be needed, in future patchsets", but not in this one. May I suggest that we put this patch in one of those future patchsets, instead of trying to do it here, in advance of anything using it?

thanks,

Signature · 1 line
John Hubbard
Quoted 123 lines

Thus, create the Fsp instance in the Gpu constructor and store it there, passing it to the GSP boot as a mutable reference using GspBootContext. This makes the Fsp available even after the GSP is booted.

---
 drivers/gpu/nova-core/gpu.rs           | 15 ++++++++++++++-
 drivers/gpu/nova-core/gsp.rs           |  2 ++
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  9 +++------
 3 files changed, 19 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/gpu.rs +14 -1
index 2e76e4bf79b2..e5ebd79c9020 100644
@@ -21,11 +21,13 @@
         Falcon, //
     },
     fb::SysmemFlush,
+    fsp::Fsp,
     gsp::{
         self,
         commands::GetGspStaticInfoReply,
         Gsp,
-        GspBootContext, //
+        GspBootContext,
+        GspBootMethod, //
     },
     regs,
 };
@@ -261,6 +263,10 @@ struct GspResources<'gpu> {
     gsp_falcon: Falcon<GspFalcon>,
     /// SEC2 falcon instance, used for GSP boot up and cleanup.
     sec2_falcon: Falcon<Sec2Falcon>,
+    /// FSP instance, if on an arch that supports it.
+    // TODO: use different resource types for each boot method, and make the relevant Gsp methods
+    // generic against them.
+    fsp: Option<Fsp>,
     /// GSP runtime data.
     #[pin]
     gsp: Gsp,
@@ -304,6 +310,7 @@ fn drop(self: Pin<&mut Self>) {
                     chipset: this.spec.chipset,
                     gsp_falcon: &*this.gsp_falcon,
                     sec2_falcon: &*this.sec2_falcon,
+                    fsp: this.fsp.as_mut(),
                 },
                 bundle,
             )
@@ -354,6 +361,11 @@ pub(crate) fn new(
 
                 sec2_falcon: Falcon::new(dev, spec.chipset)?,
 
+                fsp: match spec.chipset.gsp_boot_method() {
+                    GspBootMethod::Sec2 { .. } => None,
+                    GspBootMethod::Fsp => Some(Fsp::wait_secure_boot(dev, bar, spec.chipset)?),
+                },
+
                 gsp <- Gsp::new(pdev),
 
                 // This member must be initialized last, so the `UnloadBundle` can never be dropped
@@ -365,6 +377,7 @@ pub(crate) fn new(
                     chipset: spec.chipset,
                     gsp_falcon,
                     sec2_falcon,
+                    fsp: fsp.as_mut(),
                 })?,
             }),
 
drivers/gpu/nova-core/gsp.rs +2 -0
index 771b38e6335d..ff438506070a 100644
@@ -38,6 +38,7 @@
         sec2::Sec2 as Sec2Falcon,
         Falcon, //
     },
+    fsp::Fsp,
     gpu::{
         Architecture,
         Chipset, //
@@ -62,6 +63,7 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) chipset: Chipset,
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
+    pub(crate) fsp: Option<&'a mut Fsp>,
 }
 
 impl<'a> GspBootContext<'a> {
drivers/gpu/nova-core/gsp/hal/gh100.rs +3 -6
index 7bba18ba2f75..8673a749dbac 100644
@@ -17,10 +17,7 @@
         Falcon, //
     },
     fb::FbLayout,
-    fsp::{
-        FmcBootArgs,
-        Fsp, //
-    },
+    fsp::FmcBootArgs,
     gsp::{
         hal::{
             GspHal,
@@ -152,12 +149,12 @@ fn boot(
         let mut unload_bundle = Err(EAGAIN);
 
         let res = (|| {
+            let fsp = ctx.fsp.as_mut().ok_or(ENODEV)?;
+
             unload_bundle = Ok(crate::gsp::UnloadBundle(
                 KBox::new(FspUnloadBundle, GFP_KERNEL)? as KBox<dyn UnloadBundle>,
             ));
 
-            let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset)?;
-
             let args = FmcBootArgs::new(
                 dev,
                 chipset,
author maintainer
#20 lore raw

Re: [PATCH v2 13/13] gpu: nova-core: store Fsp instance in Gpu

Quoted 11 lines · John Hubbard
Quoted 5 lines · Alexandre Courbot

The Fsp instance was only used in the Hopper+ boot path, and consequently built locally (and immediately dropped) in it.

This worked well as a temporary measure, but the FSP is also needed in other parts of the driver, for instance vGPU.

Or: "will be needed, in future patchsets", but not in this one. May I suggest that we put this patch in one of those future patchsets, instead of trying to do it here, in advance of anything using it?

So we have two patchsets (from Zhi and Eliot) that eventually depend on this, and I am not sure which one would take this patch. :)

Besides, I'd argue that this change also stands on its own; FSP is a sub-device of the GPU, we ephemerally instanciated it in the GH100 HAL because that's where we needed it, but from an architectural perspective its scope should be larger.