Skip to contentSkip to conversation

gpu: nova-core: boot GSP with vGPU enabled on

PATCH Applied Zhi Wang 2 participants 30 messages Jun 4 – Jun 17

Subsystems (9)
lore
Share & tools
lore

Zhi Wang

author
#0 lore raw

cover gpu: nova-core: boot GSP with vGPU enabled on

Booting up GSP with vGPU enabled is part of the first milestone (M1) together with rust fwctl abstraction [1] and nova-core fwctl driver [2] for upstream vGPU support, allowing us to validate the basic GSP boot flow with vGPU enabled, upload vGPU types even before the remaining nova-core dependencies are ready. The nova-vgpu WIP patches for all milestones can be found at [3].

v3:

- Consolidate GSP boot parameters into GspBootContext struct, passed through the HAL trait methods instead of individual arguments.
- Add vGPU preludes patch (VgpuManager, Architecture::supports_vgpu, GspBootContext fields) as preparation for the vGPU boot changes.
- Rebase on top of the latest drm-rust-next.

Dropped patches:

Dependencies still in progress:

- "populate GSP_VF_INFO when vGPU is enabled" - depends on the ExtSriovCapability abstraction [4] which is still under review and Gary's on-going io projection work [5].

Ada specific:

- "introduce vgpu_support module param" - select vGPU support based on module params.

- "load the scrubber ucode when vGPU support is enabled"

v2:

- Adopt early-return style (Dirk).
- Add #ifndef CONFIG_PCI_IOV helper to fix compilation when CONFIG_PCI_IOV is disabled, per (Alex).
- Change return type from Result<i32> to Result<u16> to match the PCI spec field width, avoiding try_from at call sites.
- GspVfInfo changed to tuple struct (Alex).
- Use unconditional constructor with Option wrapping instead of bool parameter. (Alex)
- Use full initialization expression instead of mutating a zeroed value.
- Use .chain() pattern in GspSetSystemInfo::init() for optional vGPU info. (Alex)
- Eliminate all magic numbers: add vf_bar_is_64bit() and read_vf_bar64_addr() to ExtSriovCapability using PCI bindings constants (PCI_BASE_ADDRESS_MEM_TYPE_MASK, etc.).
- Use KVec<RegistryEntry> for dynamic registry entry construction instead of hardcoded array (Timur, Joel, Alexandre).
- Replace magic numbers 32/48 with named binding constants MAX_PARTITIONS_WITH_GFID_32VM / MAX_PARTITIONS_WITH_GFID from OpenRM (Alex).
- Use read_poll_timeout() instead of single read for scrubber completion check (Joel).
- Use dev instead of pdev.as_ref() in dev_dbg! (Dirk).
- Change scrubber trigger condition from vgpu_requested to fb_layout.wpr2_heap.len() > SZ_256M, checking actual heap size instead of vGPU flag. (Alex).

[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/
[3] https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-wip
[4] https://lore.kernel.org/rust-for-linux/[email protected]/
[5] https://lore.kernel.org/rust-for-linux/[email protected]/

Zhi Wang (9):
rust: pci: expose sriov_get_totalvfs() helper
gpu: nova-core: factor out common FSP message header
gpu: nova-core: return FSP response buffer to caller
gpu: nova-core: read vGPU mode from FSP via PRC protocol
gpu: nova-core: add FSP and PRC protocol documentation
gpu: nova-core: consolidate GSP boot parameters into GspBootContext
gpu: nova-core: add vGPU preludes
gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled
gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

 Documentation/gpu/nova/core/fsp.rst    | 142 +++++++++++++++++
 Documentation/gpu/nova/index.rst       |   1 +
 drivers/gpu/nova-core/fb.rs            |  17 +-
 drivers/gpu/nova-core/fsp.rs           | 207 ++++++++++++++++++++++---
 drivers/gpu/nova-core/gpu.rs           |  38 ++++-
 drivers/gpu/nova-core/gsp.rs           |  27 ++++
 drivers/gpu/nova-core/gsp/boot.rs      |  60 ++++---
 drivers/gpu/nova-core/gsp/commands.rs  |  93 +++++++----
 drivers/gpu/nova-core/gsp/fw.rs        |  16 ++
 drivers/gpu/nova-core/gsp/hal.rs       |  23 +--
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  22 ++-
 drivers/gpu/nova-core/gsp/hal/tu102.rs |  31 ++--
 drivers/gpu/nova-core/mctp.rs          |   3 +
 drivers/gpu/nova-core/nova_core.rs     |   1 +
 drivers/gpu/nova-core/vgpu.rs          |  47 ++++++
 rust/kernel/pci.rs                     |  12 ++
 16 files changed, 608 insertions(+), 132 deletions(-)
 create mode 100644 Documentation/gpu/nova/core/fsp.rst
 create mode 100644 drivers/gpu/nova-core/vgpu.rs
Signature · 1 line
2.51.0
author
#1 lore raw

PATCH 1/9 rust: pci: expose sriov_get_totalvfs() helper

Add a wrapper for the pci_sriov_get_totalvfs() helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.

---
 rust/kernel/pci.rs | 12 ++++++++++++
 1 file changed, 12 insertions(+)
rust/kernel/pci.rs +12 -0
index 5071cae6543f..d04e5f6841f2 100644
@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {
         // SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`.
         Class::from_raw(unsafe { (*self.as_raw()).class })
     }
+
+    /// Returns total number of VFs, or `Err(ENODEV)` if none available.
+    pub fn sriov_get_totalvfs(&self) -> Result<i32> {
+        // SAFETY: `self.as_raw()` is a valid pointer to a `struct pci_dev`.
+        let vfs = unsafe { bindings::pci_sriov_get_totalvfs(self.as_raw()) };
+
+        if vfs == 0 {
+            return Err(ENODEV);
+        }
+
+        Ok(vfs)
+    }
 }
 
 impl<'a> Device<device::Core<'a>> {
Signature · 1 line
2.51.0
#2 lore raw

Re: [PATCH 1/9] rust: pci: expose sriov_get_totalvfs() helper

Quoted 16 lines · Zhi Wang

Add a wrapper for the pci_sriov_get_totalvfs() helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.

---
 rust/kernel/pci.rs | 12 ++++++++++++
 1 file changed, 12 insertions(+)
rust/kernel/pci.rs +0 -0
index 5071cae6543f..d04e5f6841f2 100644

rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 455

Quoted 6 lines
rust/kernel/pci.rs +3 -0
@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {
         // SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`.
         Class::from_raw(unsafe { (*self.as_raw()).class })
     }
+
+    /// Returns total number of VFs, or `Err(ENODEV)` if none available.
+    pub fn sriov_get_totalvfs(&self) -> Result<i32> {

I mentioned it in a previous review [1], but I think there is an opportunity to improve the C API (and by transition the Rust one) by making it return a u16, which is the type the number of total VFs is ultimately stored in anyway.

IIRC there is also no need to even update any caller, just changing the prototype of pci_sriov_get_totalvfs would be enough as callers ultimately compare the return value against u16s. So if anything it would make the C API more sound.

[1] https://lore.kernel.org/all/[email protected]/

rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 461

Quoted 6 lines
rust/kernel/pci.rs +6 -0
+        // SAFETY: `self.as_raw()` is a valid pointer to a `struct pci_dev`.
+        let vfs = unsafe { bindings::pci_sriov_get_totalvfs(self.as_raw()) };
+
+        if vfs == 0 {
+            return Err(ENODEV);
+        }

Having 0 VFs does not necessarily look like an error - it's quite a valid answer to the question "how many VFs do we have?" which this method tries to answer. In patch 7 you even do unwrap_or(0) on the result of this method. So unless there is a good reason to treat this as an error, maybe we can just return a u16 here.

author
#3 lore raw

Re: [PATCH 1/9] rust: pci: expose sriov_get_totalvfs() helper

Quoted 45 lines · On Fri, 05 Jun 2026 23:08:38 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 21 lines · Zhi Wang

Add a wrapper for the pci_sriov_get_totalvfs() helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.

---
 rust/kernel/pci.rs | 12 ++++++++++++
 1 file changed, 12 insertions(+)
rust/kernel/pci.rs +2 -0
index 5071cae6543f..d04e5f6841f2 100644
@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {
         // SAFETY: `self.as_raw` is a valid pointer to a `struct
 pci_dev`. Class::from_raw(unsafe { (*self.as_raw()).class })
     }
+
+    /// Returns total number of VFs, or `Err(ENODEV)` if none
Quoted 2 lines

available.
+ pub fn sriov_get_totalvfs(&self) -> Result<i32> {

I mentioned it in a previous review [1], but I think there is an opportunity to improve the C API (and by transition the Rust one) by making it return a u16, which is the type the number of total VFs is ultimately stored in anyway.

IIRC there is also no need to even update any caller, just changing the prototype of pci_sriov_get_totalvfs would be enough as callers ultimately compare the return value against u16s. So if anything it would make the C API more sound.

[1] https://lore.kernel.org/all/[email protected]/

rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 461

Quoted 7 lines

+ // SAFETY: self.as_raw() is a valid pointer to a `struct
pci_dev`.
+ let vfs = unsafe {
bindings::pci_sriov_get_totalvfs(self.as_raw()) }; +

rust/kernel/pci.rs +3 -0
+        if vfs == 0 {
+            return Err(ENODEV);
+        }
Quoted 5 lines

Having 0 VFs does not necessarily look like an error - it's quite a valid answer to the question "how many VFs do we have?" which this method tries to answer. In patch 7 you even do unwrap_or(0) on the result of this method. So unless there is a good reason to treat this as an error, maybe we can just return a u16 here.

I will add it in the next re-spin. Sorry that I missed it.

author
#4 lore raw

PATCH 2/9 gpu: nova-core: factor out common FSP message header

In reply to Zhi Wang

Extract common MCTP + NVDM headers into FspMessageHeader, rename FspMessage to FspCotMessage, and update FspResponse to use the shared header. This prepares for adding new FSP message types.

---
 drivers/gpu/nova-core/fsp.rs | 56 +++++++++++++++++++++++++-----------
 1 file changed, 40 insertions(+), 16 deletions(-)
drivers/gpu/nova-core/fsp.rs +40 -16
index 8fc243c66e35..78b90bfbfba4 100644
@@ -57,12 +57,35 @@ struct NvdmPayloadCommandResponse {
     error_code: u32,
 }
 
-/// Complete FSP response structure with MCTP and NVDM headers.
+/// Common MCTP and NVDM headers shared by all FSP messages.
 #[repr(C, packed)]
 #[derive(Clone, Copy)]
-struct FspResponse {
+struct FspMessageHeader {
     mctp_header: MctpHeader,
     nvdm_header: NvdmHeader,
+}
+
+// SAFETY: FspMessageHeader is a packed C struct with only integral fields.
+unsafe impl AsBytes for FspMessageHeader {}
+
+// SAFETY: FspMessageHeader is a packed C struct with only integral fields.
+unsafe impl FromBytes for FspMessageHeader {}
+
+impl FspMessageHeader {
+    /// Construct a standard FSP message header for the given NVDM type.
+    fn new(nvdm_type: NvdmType) -> Self {
+        Self {
+            mctp_header: MctpHeader::single_packet(),
+            nvdm_header: NvdmHeader::new(nvdm_type),
+        }
+    }
+}
+
+/// Complete FSP response structure with MCTP and NVDM headers.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspResponse {
+    header: FspMessageHeader,
     response: NvdmPayloadCommandResponse,
 }
 
@@ -94,17 +117,16 @@ struct NvdmPayloadCot {
     gsp_boot_args_sysmem_offset: u64,
 }
 
-/// Complete FSP message structure with MCTP and NVDM headers.
+/// Complete FSP COT (Chain of Trust) message structure.
 #[repr(C)]
 #[derive(Clone, Copy)]
-struct FspMessage {
-    mctp_header: MctpHeader,
-    nvdm_header: NvdmHeader,
+struct FspCotMessage {
+    header: FspMessageHeader,
     cot: NvdmPayloadCot,
 }
 
-impl FspMessage {
-    /// Returns an in-place initializer for [`FspMessage`].
+impl FspCotMessage {
+    /// Returns an in-place initializer for [`FspCotMessage`].
     fn new<'a>(
         fb_layout: &FbLayout,
         fsp_fw: &'a FspFirmware,
@@ -131,8 +153,7 @@ fn new<'a>(
         let size = num::usize_into_u16::<{ core::mem::size_of::<NvdmPayloadCot>() }>();
 
         Ok(init!(Self {
-            mctp_header: MctpHeader::single_packet(),
-            nvdm_header: NvdmHeader::new(NvdmType::Cot),
+            header: FspMessageHeader::new(NvdmType::Cot),
             // The payload is packed, so we cannot use `init!`. Initialize it member-by-member using
             // `chain`.
             cot <- pin_init::init_zeroed(),
@@ -153,11 +174,11 @@ fn new<'a>(
     }
 }
 
-// SAFETY: `FspMessage` is `#[repr(C)]` with no padding, so all of its
+// SAFETY: `FspCotMessage` is `#[repr(C)]` with no padding, so all of its
 // bytes are initialized.
-unsafe impl AsBytes for FspMessage {}
+unsafe impl AsBytes for FspCotMessage {}
 
-impl MessageToFsp for FspMessage {
+impl MessageToFsp for FspCotMessage {
     const NVDM_TYPE: NvdmType = NvdmType::Cot;
 }
 
@@ -251,8 +272,8 @@ fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) ->
             EIO
         })?;
 
-        let mctp_header = response.mctp_header;
-        let nvdm_header = response.nvdm_header;
+        let mctp_header = response.header.mctp_header;
+        let nvdm_header = response.header.nvdm_header;
         let command_nvdm_type = response.response.command_nvdm_type;
         let error_code = response.response.error_code;
 
@@ -310,7 +331,10 @@ pub(crate) fn boot_fmc(
     ) -> Result {
         dev_dbg!(dev, "Starting FSP boot sequence for {}\n", args.chipset);
 
-        let msg = KBox::init(FspMessage::new(fb_layout, &self.fsp_fw, args)?, GFP_KERNEL)?;
+        let msg = KBox::init(
+            FspCotMessage::new(fb_layout, &self.fsp_fw, args)?,
+            GFP_KERNEL,
+        )?;
 
         self.send_sync_fsp(dev, bar, &*msg)?;
 
Signature · 1 line
2.51.0
#5 lore raw

Re: [PATCH 2/9] gpu: nova-core: factor out common FSP message header

Quoted 5 lines · Zhi Wang

Extract common MCTP + NVDM headers into FspMessageHeader, rename FspMessage to FspCotMessage, and update FspResponse to use the shared header. This prepares for adding new FSP message types.

This patch is a good follow-up to the FSP series irrespective of vGPU support, so I'd like to merge it rapidly. Leaving it a couple of days for reviews but I think it makes sense to take it early and avoid rebasing churn.

author
#6 lore raw

PATCH 3/9 gpu: nova-core: return FSP response buffer to caller

In reply to Zhi Wang

Change send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.

For the existing COT caller, the response buffer is unused.

---
 drivers/gpu/nova-core/fsp.rs | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/fsp.rs +4 -3
index 78b90bfbfba4..5fd2e9e277b1 100644
@@ -257,7 +257,8 @@ pub(crate) fn wait_secure_boot(
     }
 
     /// Sends a message to FSP and waits for the response.
-    fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) -> Result
+    /// Returns the full response buffer on success.
+    fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) -> Result<KVec<u8>>
     where
         M: MessageToFsp,
     {
@@ -315,7 +316,7 @@ fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) ->
             return Err(EIO);
         }
 
-        Ok(())
+        Ok(response_buf)
     }
 
     /// Boots GSP FMC via FSP Chain of Trust.
@@ -336,7 +337,7 @@ pub(crate) fn boot_fmc(
             GFP_KERNEL,
         )?;
 
-        self.send_sync_fsp(dev, bar, &*msg)?;
+        let _response_buf = self.send_sync_fsp(dev, bar, &*msg)?;
 
         dev_dbg!(dev, "FSP Chain of Trust completed successfully\n");
         Ok(())
Signature · 1 line
2.51.0
#7 lore raw

Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller

Quoted 8 lines · Zhi Wang

Change send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.

For the existing COT caller, the response buffer is unused.

Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.

author
#8 lore raw

Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller

Quoted 6 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 5 lines · Zhi Wang

Change send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.

Quoted 6 lines
Quoted 3 lines

For the existing COT caller, the response buffer is unused.

Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.

Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.

#9 lore raw

Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller

Quoted 9 lines · Zhi Wang
Quoted 6 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 5 lines · Zhi Wang

Change send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

For the existing COT caller, the response buffer is unused.

Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.

Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.

Patches 2 and 3 have been pushed to drm-rust-next!

author
#10 lore raw

Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller

Quoted 15 lines · On Tue, 09 Jun 2026 15:07:21 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 14 lines · Zhi Wang
Quoted 11 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 9 lines · Zhi Wang

Change send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.

For the existing COT caller, the response buffer is unused.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.

Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.

Patches 2 and 3 have been pushed to drm-rust-next!

Thanks so much. :)

author
#11 lore raw

PATCH 4/9 gpu: nova-core: read vGPU mode from FSP via PRC protocol

In reply to Zhi Wang

Add support for querying the vGPU mode configuration from FSP using the PRC (Product Reconfiguration Control) protocol. PRC is an API system exposed through FSP's Management Partition that allows querying device configuration "knobs" without firmware updates.

Add a VgpuMode enum that validates the raw PRC response value, returning an error for unexpected values.

---
 drivers/gpu/nova-core/fsp.rs  | 145 ++++++++++++++++++++++++++++++++++
 drivers/gpu/nova-core/mctp.rs |   3 +
 2 files changed, 148 insertions(+)
drivers/gpu/nova-core/fsp.rs +145 -0
index 5fd2e9e277b1..ce11efeba37e 100644
@@ -48,6 +48,44 @@
 
 mod hal;
 
+/// PRC (Product Reconfiguration Control) protocol constants.
+///
+/// PRC is an API system exposed through FSP's Management Partition that allows
+/// querying and modifying device configuration "knobs" without firmware updates.
+/// Each knob is identified by a unique object ID and controls a specific device
+/// behavior (e.g., vGPU mode, ECC, confidential computing).
+mod prc {
+    /// Sub-command to read a PRC knob value.
+    pub(super) const SUBCMD_READ: u8 = 0x0c;
+
+    /// PRC object ID for vGPU mode configuration (knob ID 41).
+    pub(super) const OBJECT_VGPU_MODE: u8 = 0x29;
+
+    /// Request the active knob value (currently effective this boot).
+    pub(super) const FLAG_ACTIVE: u8 = 1 << 1;
+}
+
+/// vGPU operating mode as reported by FSP via the PRC protocol.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum VgpuMode {
+    /// vGPU support is disabled on this GPU.
+    Disabled = 0,
+    /// vGPU support is enabled on this GPU.
+    Enabled = 1,
+}
+
+impl TryFrom<u16> for VgpuMode {
+    type Error = kernel::error::Error;
+
+    fn try_from(value: u16) -> Result<Self> {
+        match value {
+            0 => Ok(VgpuMode::Disabled),
+            1 => Ok(VgpuMode::Enabled),
+            _ => Err(EINVAL),
+        }
+    }
+}
+
 /// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`).
 #[repr(C, packed)]
 #[derive(Clone, Copy)]
@@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse {
     error_code: u32,
 }
 
+// SAFETY: NvdmPayloadCommandResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for NvdmPayloadCommandResponse {}
+
+/// PRC message payload.
+///
+/// Sent to FSP to query or modify a device configuration knob.
+/// The response includes the common FSP response header followed by
+/// a [`NvdmPayloadPrcResponse`] with the knob's current state value.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrc {
+    sub_message_id: u8,
+    flags: u8,
+    object_id: u8,
+    reserved: u8,
+}
+
+// SAFETY: NvdmPayloadPrc is a packed C struct with only integral fields.
+unsafe impl AsBytes for NvdmPayloadPrc {}
+
+/// PRC response payload containing the knob state value.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrcResponse {
+    value_low: u8,
+    value_high: u8,
+    reserved1: u8,
+    reserved2: u8,
+}
+
+// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for NvdmPayloadPrcResponse {}
+
 /// Common MCTP and NVDM headers shared by all FSP messages.
 #[repr(C, packed)]
 #[derive(Clone, Copy)]
@@ -92,6 +163,18 @@ struct FspResponse {
 // SAFETY: FspResponse is a packed C struct with only integral fields.
 unsafe impl FromBytes for FspResponse {}
 
+/// Complete FSP PRC response including the knob state payload.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcResponse {
+    header: FspMessageHeader,
+    response: NvdmPayloadCommandResponse,
+    prc_data: NvdmPayloadPrcResponse,
+}
+
+// SAFETY: FspPrcResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for FspPrcResponse {}
+
 /// Trait implemented by types representing a message to send to FSP.
 ///
 /// This provides [`Fsp::send_sync_fsp`] with the information it needs to send
@@ -178,10 +261,25 @@ fn new<'a>(
 // bytes are initialized.
 unsafe impl AsBytes for FspCotMessage {}
 
+/// Complete FSP PRC message.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcMessage {
+    header: FspMessageHeader,
+    prc: NvdmPayloadPrc,
+}
+
+// SAFETY: FspPrcMessage is a packed C struct with only integral fields.
+unsafe impl AsBytes for FspPrcMessage {}
+
 impl MessageToFsp for FspCotMessage {
     const NVDM_TYPE: NvdmType = NvdmType::Cot;
 }
 
+impl MessageToFsp for FspPrcMessage {
+    const NVDM_TYPE: NvdmType = NvdmType::Prc;
+}
+
 /// Bundled arguments for FMC boot via FSP Chain of Trust.
 pub(crate) struct FmcBootArgs {
     chipset: Chipset,
@@ -226,6 +324,53 @@ pub(crate) struct Fsp {
 }
 
 impl Fsp {
+    /// Read vGPU mode from FSP using the PRC protocol.
+    ///
+    /// Queries FSP's Management Partition for the active vGPU mode knob value.
+    /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU,
+    /// [`VgpuMode::Disabled`] otherwise.
+    #[expect(dead_code)]
+    pub(crate) fn read_vgpu_mode(
+        &mut self,
+        dev: &device::Device<device::Bound>,
+        bar: Bar0<'_>,
+    ) -> Result<VgpuMode> {
+        let msg = KBox::new(
+            FspPrcMessage {
+                header: FspMessageHeader::new(NvdmType::Prc),
+                prc: NvdmPayloadPrc {
+                    sub_message_id: prc::SUBCMD_READ,
+                    flags: prc::FLAG_ACTIVE,
+                    object_id: prc::OBJECT_VGPU_MODE,
+                    reserved: 0,
+                },
+            },
+            GFP_KERNEL,
+        )?;
+
+        let response_buf = self.send_sync_fsp(dev, bar, &*msg)?;
+
+        let prc_resp_size = core::mem::size_of::<FspPrcResponse>();
+        if response_buf.len() < prc_resp_size {
+            dev_err!(
+                dev,
+                "PRC response too small: {} bytes (expected {})\n",
+                response_buf.len(),
+                prc_resp_size
+            );
+            return Err(EIO);
+        }
+
+        let (prc_response, _) = FspPrcResponse::from_bytes_prefix(&response_buf[..]).ok_or(EIO)?;
+
+        let raw_value = u16::from(prc_response.prc_data.value_low)
+            | (u16::from(prc_response.prc_data.value_high) << 8);
+
+        VgpuMode::try_from(raw_value).inspect_err(|_| {
+            dev_err!(dev, "unexpected vGPU mode value: {:#x}\n", raw_value);
+        })
+    }
+
     /// Waits for FSP secure boot completion, then returns the [`Fsp`] interface.
     ///
     /// Polls the thermal scratch register until FSP signals boot completion or the timeout
drivers/gpu/nova-core/mctp.rs +3 -0
index 482786e07bc7..b203c632bf20 100644
@@ -13,6 +13,8 @@
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 #[repr(u8)]
 pub(crate) enum NvdmType {
+    /// PRC (Product Reconfiguration Control) message.
+    Prc = 0x13,
     #[default]
     /// Chain of Trust boot message.
     Cot = 0x14,
@@ -25,6 +27,7 @@ impl TryFrom<u8> for NvdmType {
 
     fn try_from(value: u8) -> Result<Self, Self::Error> {
         match value {
+            x if x == u8::from(Self::Prc) => Ok(Self::Prc),
             x if x == u8::from(Self::Cot) => Ok(Self::Cot),
             x if x == u8::from(Self::FspResponse) => Ok(Self::FspResponse),
             _ => Err(value),
Signature · 1 line
2.51.0
#12 lore raw

Re: [PATCH 4/9] gpu: nova-core: read vGPU mode from FSP via PRC protocol

Quoted 28 lines · Zhi Wang

Add support for querying the vGPU mode configuration from FSP using the PRC (Product Reconfiguration Control) protocol. PRC is an API system exposed through FSP's Management Partition that allows querying device configuration "knobs" without firmware updates.

Add a VgpuMode enum that validates the raw PRC response value, returning an error for unexpected values.

---
 drivers/gpu/nova-core/fsp.rs  | 145 ++++++++++++++++++++++++++++++++++
 drivers/gpu/nova-core/mctp.rs |   3 +
 2 files changed, 148 insertions(+)
drivers/gpu/nova-core/fsp.rs +6 -0
index 5fd2e9e277b1..ce11efeba37e 100644
@@ -48,6 +48,44 @@
 
 mod hal;
 
+/// PRC (Product Reconfiguration Control) protocol constants.
+///
+/// PRC is an API system exposed through FSP's Management Partition that allows
+/// querying and modifying device configuration "knobs" without firmware updates.
+/// Each knob is identified by a unique object ID and controls a specific device
+/// behavior (e.g., vGPU mode, ECC, confidential computing).

drivers/gpu/nova-core/fsp.rs line 66

Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10 -0
+mod prc {
+    /// Sub-command to read a PRC knob value.
+    pub(super) const SUBCMD_READ: u8 = 0x0c;
+
+    /// PRC object ID for vGPU mode configuration (knob ID 41).
+    pub(super) const OBJECT_VGPU_MODE: u8 = 0x29;
+
+    /// Request the active knob value (currently effective this boot).
+    pub(super) const FLAG_ACTIVE: u8 = 1 << 1;
+}

These 3 should be defined as their own enum types, to make sure we cannot mix them. For instance:

    enum PrcMessageSubcmd {
      Read = 0x0c,
    }

    enum PrcObjectId {
      VgpuMode = 0x29,
    }

    (for the flags I guess you will want to use `kernel::impl_flags!`?)

Then, NvdmPayloadPrc can have a constructor that takes these 3 as parameters and returns the constructed value - that way no risk of, say, using the subcmd as an object identifier.

Btw, is there a public source for these values?

I like the idea of using a prc sub-module - to the point where I'd suggest moving all PRC-related types to it, and all CoT messages to a cot sub-module. But doing it in this series would distract from the goal, and the fsp module is not that large, so let's keep everything in it for now and do this as a follow-up.

Quoted 11 lines
drivers/gpu/nova-core/fsp.rs +11 -0
+
+/// vGPU operating mode as reported by FSP via the PRC protocol.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum VgpuMode {
+    /// vGPU support is disabled on this GPU.
+    Disabled = 0,
+    /// vGPU support is enabled on this GPU.
+    Enabled = 1,
+}
+
+impl TryFrom<u16> for VgpuMode {

drivers/gpu/nova-core/fsp.rs line 87

Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10 -0
+    type Error = kernel::error::Error;
+
+    fn try_from(value: u16) -> Result<Self> {
+        match value {
+            0 => Ok(VgpuMode::Disabled),
+            1 => Ok(VgpuMode::Enabled),
+            _ => Err(EINVAL),
+        }
+    }
+}

You can even do TryFrom<NvdmPayloadPrcResponse> for VgpuMode for a better fit.

Quoted 45 lines
drivers/gpu/nova-core/fsp.rs +34 -0
+
 /// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`).
 #[repr(C, packed)]
 #[derive(Clone, Copy)]
@@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse {
     error_code: u32,
 }
 
+// SAFETY: NvdmPayloadCommandResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for NvdmPayloadCommandResponse {}
+
+/// PRC message payload.
+///
+/// Sent to FSP to query or modify a device configuration knob.
+/// The response includes the common FSP response header followed by
+/// a [`NvdmPayloadPrcResponse`] with the knob's current state value.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrc {
+    sub_message_id: u8,
+    flags: u8,
+    object_id: u8,
+    reserved: u8,
+}
+
+// SAFETY: NvdmPayloadPrc is a packed C struct with only integral fields.
+unsafe impl AsBytes for NvdmPayloadPrc {}
+
+/// PRC response payload containing the knob state value.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrcResponse {
+    value_low: u8,
+    value_high: u8,
+    reserved1: u8,
+    reserved2: u8,
+}
+
+// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for NvdmPayloadPrcResponse {}
+
 /// Common MCTP and NVDM headers shared by all FSP messages.
 #[repr(C, packed)]
 #[derive(Clone, Copy)]

drivers/gpu/nova-core/fsp.rs struct FspResponse { · line 171

Quoted 9 lines
drivers/gpu/nova-core/fsp.rs +6 -0
@@ -92,6 +163,18 @@ struct FspResponse {
 // SAFETY: FspResponse is a packed C struct with only integral fields.
 unsafe impl FromBytes for FspResponse {}
 
+/// Complete FSP PRC response including the knob state payload.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcResponse {
+    header: FspMessageHeader,
+    response: NvdmPayloadCommandResponse,

Since you introduced FspMessageHeader to factor out mctp_header and nvdm_header, can you do the same for the reponse? I guess in our case this is as simple as renaming FspResponse to FspResponseHeader and using it in FspPrcResponse.

Quoted 65 lines
drivers/gpu/nova-core/fsp.rs +47 -0
+    prc_data: NvdmPayloadPrcResponse,
+}
+
+// SAFETY: FspPrcResponse is a packed C struct with only integral fields.
+unsafe impl FromBytes for FspPrcResponse {}
+
 /// Trait implemented by types representing a message to send to FSP.
 ///
 /// This provides [`Fsp::send_sync_fsp`] with the information it needs to send
@@ -178,10 +261,25 @@ fn new<'a>(
 // bytes are initialized.
 unsafe impl AsBytes for FspCotMessage {}
 
+/// Complete FSP PRC message.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcMessage {
+    header: FspMessageHeader,
+    prc: NvdmPayloadPrc,
+}
+
+// SAFETY: FspPrcMessage is a packed C struct with only integral fields.
+unsafe impl AsBytes for FspPrcMessage {}
+
 impl MessageToFsp for FspCotMessage {
     const NVDM_TYPE: NvdmType = NvdmType::Cot;
 }
 
+impl MessageToFsp for FspPrcMessage {
+    const NVDM_TYPE: NvdmType = NvdmType::Prc;
+}
+
 /// Bundled arguments for FMC boot via FSP Chain of Trust.
 pub(crate) struct FmcBootArgs {
     chipset: Chipset,
@@ -226,6 +324,53 @@ pub(crate) struct Fsp {
 }
 
 impl Fsp {
+    /// Read vGPU mode from FSP using the PRC protocol.
+    ///
+    /// Queries FSP's Management Partition for the active vGPU mode knob value.
+    /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU,
+    /// [`VgpuMode::Disabled`] otherwise.
+    #[expect(dead_code)]
+    pub(crate) fn read_vgpu_mode(
+        &mut self,
+        dev: &device::Device<device::Bound>,
+        bar: Bar0<'_>,
+    ) -> Result<VgpuMode> {
+        let msg = KBox::new(
+            FspPrcMessage {
+                header: FspMessageHeader::new(NvdmType::Prc),
+                prc: NvdmPayloadPrc {
+                    sub_message_id: prc::SUBCMD_READ,
+                    flags: prc::FLAG_ACTIVE,
+                    object_id: prc::OBJECT_VGPU_MODE,
+                    reserved: 0,
+                },
+            },
+            GFP_KERNEL,
+        )?;
+
+        let response_buf = self.send_sync_fsp(dev, bar, &*msg)?;
+

drivers/gpu/nova-core/fsp.rs pub(crate) struct Fsp { · line 362

Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10 -0
+        let prc_resp_size = core::mem::size_of::<FspPrcResponse>();
+        if response_buf.len() < prc_resp_size {
+            dev_err!(
+                dev,
+                "PRC response too small: {} bytes (expected {})\n",
+                response_buf.len(),
+                prc_resp_size
+            );
+            return Err(EIO);
+        }

IIUC from_bytes_prefix takes care of checking the size, so this check looks redundant.

author
#13 lore raw

Re: [PATCH 4/9] gpu: nova-core: read vGPU mode from FSP via PRC protocol

Quoted 1 line · On Tue, 16 Jun 2026 17:35:27 +0900 "Alexandre Courbot" <[email protected]>…

On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:

snip

Quoted 14 lines
Quoted 1 line

+}

These 3 should be defined as their own enum types, to make sure we cannot mix them. For instance:

    enum PrcMessageSubcmd {
      Read = 0x0c,
    }

    enum PrcObjectId {
      VgpuMode = 0x29,
    }

    (for the flags I guess you will want to use
Quoted 6 lines

kernel::impl_flags!?)

Then, NvdmPayloadPrc can have a constructor that takes these 3 as parameters and returns the constructed value - that way no risk of, say, using the subcmd as an object identifier.

Great idea. Actually, this is what I am looking for - how to organized those values.

Quoted 1 line

Btw, is there a public source for these values?

Production reconfiguration knobs are public knowledge where folks can find from [1]. vGPU mode is a newly added one.

[1] https://github.com/NVIDIA/gpu-admin-tools

Quoted 7 lines

I like the idea of using a prc sub-module - to the point where I'd suggest moving all PRC-related types to it, and all CoT messages to a cot sub-module. But doing it in this series would distract from the goal, and the fsp module is not that large, so let's keep everything in it for now and do this as a follow-up.

I see. I can send another series to address above after this one is done.

Quoted 165 lines
Quoted 11 lines
drivers/gpu/nova-core/fsp.rs +11 -0
+
+/// vGPU operating mode as reported by FSP via the PRC protocol.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum VgpuMode {
+    /// vGPU support is disabled on this GPU.
+    Disabled = 0,
+    /// vGPU support is enabled on this GPU.
+    Enabled = 1,
+}
+
+impl TryFrom<u16> for VgpuMode {

drivers/gpu/nova-core/fsp.rs line 87

Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10 -0
+    type Error = kernel::error::Error;
+
+    fn try_from(value: u16) -> Result<Self> {
+        match value {
+            0 => Ok(VgpuMode::Disabled),
+            1 => Ok(VgpuMode::Enabled),
+            _ => Err(EINVAL),
+        }
+    }
+}  

You can even do TryFrom<NvdmPayloadPrcResponse> for VgpuMode for a better fit.

Quoted 45 lines
drivers/gpu/nova-core/fsp.rs +2 -0
+
 /// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`).
 #[repr(C, packed)]
 #[derive(Clone, Copy)]
@@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse {
     error_code: u32,
 }
 
+// SAFETY: NvdmPayloadCommandResponse is a packed C struct with

only integral fields. +unsafe impl FromBytes for
NvdmPayloadCommandResponse {} +

drivers/gpu/nova-core/fsp.rs +4 -0
+/// PRC message payload.
+///
+/// Sent to FSP to query or modify a device configuration knob.
+/// The response includes the common FSP response header followed

by +/// a [NvdmPayloadPrcResponse] with the knob's current state value. +#[repr(C, packed)]

drivers/gpu/nova-core/fsp.rs +9 -0
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrc {
+    sub_message_id: u8,
+    flags: u8,
+    object_id: u8,
+    reserved: u8,
+}
+
+// SAFETY: NvdmPayloadPrc is a packed C struct with only integral

fields. +unsafe impl AsBytes for NvdmPayloadPrc {}

drivers/gpu/nova-core/fsp.rs +12 -0
+
+/// PRC response payload containing the knob state value.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct NvdmPayloadPrcResponse {
+    value_low: u8,
+    value_high: u8,
+    reserved1: u8,
+    reserved2: u8,
+}
+
+// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only

integral fields. +unsafe impl FromBytes for NvdmPayloadPrcResponse {} +
/// Common MCTP and NVDM headers shared by all FSP messages. #[repr(C, packed)]
#[derive(Clone, Copy)]

drivers/gpu/nova-core/fsp.rs struct FspResponse { · line 171

Quoted 9 lines
drivers/gpu/nova-core/fsp.rs +6 -0
@@ -92,6 +163,18 @@ struct FspResponse {
 // SAFETY: FspResponse is a packed C struct with only integral
 fields. unsafe impl FromBytes for FspResponse {}
 
+/// Complete FSP PRC response including the knob state payload.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcResponse {
+    header: FspMessageHeader,
+    response: NvdmPayloadCommandResponse,  

Since you introduced FspMessageHeader to factor out mctp_header and nvdm_header, can you do the same for the reponse? I guess in our case this is as simple as renaming FspResponse to FspResponseHeader and using it in FspPrcResponse.

Quoted 67 lines
drivers/gpu/nova-core/fsp.rs +5 -0
+    prc_data: NvdmPayloadPrcResponse,
+}
+
+// SAFETY: FspPrcResponse is a packed C struct with only integral
 fields. +unsafe impl FromBytes for FspPrcResponse {}
+
 /// Trait implemented by types representing a message to send to

FSP. ///
/// This provides [Fsp::send_sync_fsp] with the information it
needs to send @@ -178,10 +261,25 @@ fn new<'a>(
// bytes are initialized.
unsafe impl AsBytes for FspCotMessage {}

drivers/gpu/nova-core/fsp.rs +9 -0
+/// Complete FSP PRC message.
+#[repr(C, packed)]
+#[derive(Clone, Copy)]
+struct FspPrcMessage {
+    header: FspMessageHeader,
+    prc: NvdmPayloadPrc,
+}
+
+// SAFETY: FspPrcMessage is a packed C struct with only integral

fields. +unsafe impl AsBytes for FspPrcMessage {}

drivers/gpu/nova-core/fsp.rs +8 -0
+
 impl MessageToFsp for FspCotMessage {
     const NVDM_TYPE: NvdmType = NvdmType::Cot;
 }
 
+impl MessageToFsp for FspPrcMessage {
+    const NVDM_TYPE: NvdmType = NvdmType::Prc;
+}
+
 /// Bundled arguments for FMC boot via FSP Chain of Trust.
 pub(crate) struct FmcBootArgs {
     chipset: Chipset,
@@ -226,6 +324,53 @@ pub(crate) struct Fsp {
 }
 
 impl Fsp {
+    /// Read vGPU mode from FSP using the PRC protocol.
+    ///
+    /// Queries FSP's Management Partition for the active vGPU

mode knob value.
+ /// Returns [VgpuMode::Enabled] if vGPU support is active on
this GPU,

drivers/gpu/nova-core/fsp.rs +22 -0
+    /// [`VgpuMode::Disabled`] otherwise.
+    #[expect(dead_code)]
+    pub(crate) fn read_vgpu_mode(
+        &mut self,
+        dev: &device::Device<device::Bound>,
+        bar: Bar0<'_>,
+    ) -> Result<VgpuMode> {
+        let msg = KBox::new(
+            FspPrcMessage {
+                header: FspMessageHeader::new(NvdmType::Prc),
+                prc: NvdmPayloadPrc {
+                    sub_message_id: prc::SUBCMD_READ,
+                    flags: prc::FLAG_ACTIVE,
+                    object_id: prc::OBJECT_VGPU_MODE,
+                    reserved: 0,
+                },
+            },
+            GFP_KERNEL,
+        )?;
+
+        let response_buf = self.send_sync_fsp(dev, bar, &*msg)?;
+

drivers/gpu/nova-core/fsp.rs pub(crate) struct Fsp { · line 362

Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10 -0
+        let prc_resp_size = core::mem::size_of::<FspPrcResponse>();
+        if response_buf.len() < prc_resp_size {
+            dev_err!(
+                dev,
+                "PRC response too small: {} bytes (expected {})\n",
+                response_buf.len(),
+                prc_resp_size
+            );
+            return Err(EIO);
+        }  

IIUC from_bytes_prefix takes care of checking the size, so this check looks redundant.

author
#14 lore raw

PATCH 5/9 gpu: nova-core: add FSP and PRC protocol documentation

In reply to Zhi Wang

Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.

---
 Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst    |   1 +
 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/fsp.rst
Documentation/gpu/nova/core/fsp.rst new file +142 -0
new file mode 100644
index 000000000000..52d618d22bb8
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+FSP (Foundation Security Processor) and Secure Boot
+===================================================
+This document describes the role of the FSP in the GPU boot sequence on
+Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot
+flow. It also provides a brief overview of the PRC (Product Reconfiguration
+Control) protocol used to query device configuration through FSP. As with
+other documents in this directory, the information is subject to change and
+is intended to help developers understand the corresponding kernel code.
+
+What is FSP?
+============
+The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust
+(IROT). It is a dedicated security processor that boots from immutable ROM
+(Boot ROM) inside the GPU and is responsible for establishing the Chain of
+Trust before any other firmware is allowed to run.
+
+FSP runs independently of the host CPU and starts executing as soon as the
+GPU is powered on. By the time the nova-core driver is loaded, FSP has
+already completed its own secure boot and is ready to accept commands from
+the driver.
+
+Simplified boot flow (Hopper/Blackwell)
+=======================================
+Starting with Hopper, the boot flow is significantly simplified compared to
+earlier GPU generations like Ampere.
+
+On an **Ampere** GPU, the boot verification chain involves multiple Falcon
+engines and multiple ucode stages (see falcon.rst for details)::
+
+     Hardware BROM (SEC2)
+          -> HS Booter (SEC2)
+               -> LS GSP-RM (GSP)
+
+The driver must extract ucode from VBIOS, manage SEC2 and GSP, and
+orchestrate the Booter to load GSP-RM. This involves FWSEC-FRTS, devinit,
+and the Booter stages.
+
+On **Hopper/Blackwell** GPUs, FSP replaces this multi-stage process with a
+single message-driven interface::
+
+     FSP (hardware root of trust, boots from ROM)
+          -> FMC (Falcon Microcontroller, verified by FSP)
+               -> GSP-RM (verified and loaded by FMC)
+
+The driver only needs to:
+
+1. Wait for FSP to complete its own secure boot (polling a scratch register).
+2. Send a Chain of Trust (COT) message to FSP with the FMC firmware location,
+   cryptographic signatures, and GSP boot parameters.
+3. FSP authenticates the FMC firmware and boots it, FMC in turn loads GSP-RM.
+
+There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage. The
+entire secure boot is driven by a single FSP message exchange.
+
+Chain of Trust (COT) protocol
+=============================
+The Chain of Trust establishes a cryptographically enforced boot sequence,
+ensuring the GPU reaches a known, trusted state.
+
+The driver communicates with FSP using a message queue (Falcon MSGQ
+interface). Each message consists of an MCTP (Management Component Transport
+Protocol) transport header and an NVDM (NVIDIA Vendor Defined Message) header,
+followed by a protocol-specific payload.
+
+For Chain of Trust, the payload includes:
+
+- The system memory address of the FMC firmware image.
+- Cryptographic material: a SHA-384 hash, RSA-3K public key, and RSA-3K
+  signature extracted from the FMC ELF firmware.
+- FRTS (Firmware Runtime Services) region information (vidmem offset and size).
+- The system memory address of the GSP boot arguments structure.
+
+FSP verifies the signature against the provided public key and hash, and if
+verification succeeds, boots the FMC. The FMC then authenticates and launches
+GSP-RM.
+
+The message flow is::
+
+     nova-core                          FSP
+        |                                |
+        |  1. Poll scratch register      |
+        |  (wait for FSP boot complete)  |
+        |                                |
+        |  2. COT message  ------------> |
+        |     (FMC addr, signatures,     |
+        |      boot params)              |
+        |                                |
+        |                                |--- Verify FMC signature
+        |                                |--- Boot FMC
+        |                                |--- FMC loads GSP-RM
+        |                                |
+        |  3. COT response <------------ |
+        |     (success/error)            |
+        |                                |
+
+FSP message format
+==================
+All FSP messages share a common header format consisting of two 32-bit words:
+
+**MCTP header** (Management Component Transport Protocol):
+
+- Bit 31: SOM (Start of Message)
+- Bit 30: EOM (End of Message)
+- Bits 29:28: Packet sequence number
+- Bits 23:16: Source Endpoint ID
+
+**NVDM header** (NVIDIA Vendor Defined Message):
+
+- Bits 6:0: MCTP message type (0x7e = vendor-defined PCI)
+- Bits 23:8: PCI vendor ID (0x10de = NVIDIA)
+- Bits 31:24: NVDM type (0x14 = COT, 0x13 = PRC, 0x15 = FSP response)
+
+PRC (Product Reconfiguration Control) protocol
+===============================================
+PRC is an API system exposed through FSP's Management Partition that allows
+querying and modifying device configuration without firmware updates.
+
+Configuration parameters are called "knobs". Each knob has a unique object
+ID and controls a specific device behavior. Examples include vGPU mode, ECC
+enable, confidential computing mode, and NVLINK configuration.
+
+Each knob has two values:
+
+- **Active**: the currently effective value for this boot cycle.
+- **Persistent**: the value stored in InfoROM, applied on subsequent boots.
+
+The nova-core driver uses PRC to read the vGPU mode knob (object ID 0x29)
+during early boot, before firmware loading, to determine whether the GPU
+should operate in vGPU mode.
+
+The PRC message format follows the same MCTP/NVDM header structure as COT,
+with NVDM type 0x13. The payload contains:
+
+- A sub-command (e.g., 0x0c for read).
+- Flags indicating which value to read (bit 0 = persistent, bit 1 = active).
+- The knob object ID.
+
+The response includes the common FSP response header (with error status)
+followed by the knob's 16-bit state value.
Documentation/gpu/nova/index.rst +1 -0
index e39cb3163581..1783513cbd05 100644
@@ -30,5 +30,6 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/todo
    core/vbios
    core/devinit
+   core/fsp
    core/fwsec
    core/falcon
Signature · 1 line
2.51.0
#15 lore raw

Re: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation

Quoted 7 lines · Zhi Wang

Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.

This is super useful, thanks! One question below:

Quoted 35 lines
---
 Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst    |   1 +
 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/fsp.rst
Documentation/gpu/nova/core/fsp.rst new file +23 -0
new file mode 100644
index 000000000000..52d618d22bb8
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+FSP (Foundation Security Processor) and Secure Boot
+===================================================
+This document describes the role of the FSP in the GPU boot sequence on
+Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot
+flow. It also provides a brief overview of the PRC (Product Reconfiguration
+Control) protocol used to query device configuration through FSP. As with
+other documents in this directory, the information is subject to change and
+is intended to help developers understand the corresponding kernel code.
+
+What is FSP?
+============
+The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust
+(IROT). It is a dedicated security processor that boots from immutable ROM
+(Boot ROM) inside the GPU and is responsible for establishing the Chain of
+Trust before any other firmware is allowed to run.
+
+FSP runs independently of the host CPU and starts executing as soon as the
+GPU is powered on. By the time the nova-core driver is loaded, FSP has
+already completed its own secure boot and is ready to accept commands from
+the driver.

Documentation/gpu/nova/core/fsp.rst line 33

Quoted 10 lines
Documentation/gpu/nova/core/fsp.rst new file +10 -0
+
+Simplified boot flow (Hopper/Blackwell)
+=======================================
+Starting with Hopper, the boot flow is significantly simplified compared to
+earlier GPU generations like Ampere.
+
+On an **Ampere** GPU, the boot verification chain involves multiple Falcon
+engines and multiple ucode stages (see falcon.rst for details)::
+
+     Hardware BROM (SEC2)

Is this referring to FWSEC-FRTS? If so, should this be GSP instead of SEC2?

author
#16 lore raw

Re: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation

Quoted 51 lines · On Tue, 16 Jun 2026 17:17:34 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 7 lines · Zhi Wang

Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.

This is super useful, thanks! One question below:

Quoted 40 lines
---
 Documentation/gpu/nova/core/fsp.rst | 142
++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst    |
 1 + 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/fsp.rst
(quoted diff) +0 -0

b/Documentation/gpu/nova/core/fsp.rst new file mode 100644 index 000000000000..52d618d22bb8

Documentation/gpu/nova/core/fsp.rst new file +6 -0
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+FSP (Foundation Security Processor) and Secure Boot
+===================================================
+This document describes the role of the FSP in the GPU boot

sequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. +

Documentation/gpu/nova/core/fsp.rst new file +3 -0
+What is FSP?
+============
+The Foundation Security Processor (FSP) is the GPU's Internal Root

of Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. +
+FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver.

Documentation/gpu/nova/core/fsp.rst new file +4 -0
+
+Simplified boot flow (Hopper/Blackwell)
+=======================================
+Starting with Hopper, the boot flow is significantly simplified

compared to +earlier GPU generations like Ampere.

Quoted 8 lines
Quoted 5 lines
Documentation/gpu/nova/core/fsp.rst new file +2 -0
+
+On an **Ampere** GPU, the boot verification chain involves

multiple Falcon +engines and multiple ucode stages (see falcon.rst for details):: +
+ Hardware BROM (SEC2)

Is this referring to FWSEC-FRTS? If so, should this be GSP instead of SEC2?

Hi Alex:

This refers to the BootROM in the SEC2 Peregrine. From the timeline perspective, FWSEC-FRTS should be positioned as below. I was thinking to show the brief differences of relationship between internal processors so I omit the FWSEC-FRTS in the sequence. Do you think we should add it?

+ Hardware BROM (SEC2)

                    here  < ----- FWSEC-FRTS (GSP)

+ -> HS Booter (SEC2)
+ -> LS GSP-RM (GSP)

Z.

#17 lore raw

Re: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation

Quoted 70 lines · Zhi Wang
Quoted 51 lines · On Tue, 16 Jun 2026 17:17:34 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 7 lines · Zhi Wang

Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.

This is super useful, thanks! One question below:

Quoted 40 lines
---
 Documentation/gpu/nova/core/fsp.rst | 142
++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst    |
 1 + 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/fsp.rst
(quoted diff) +0 -0

b/Documentation/gpu/nova/core/fsp.rst new file mode 100644 index 000000000000..52d618d22bb8

Documentation/gpu/nova/core/fsp.rst new file +6 -0
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+FSP (Foundation Security Processor) and Secure Boot
+===================================================
+This document describes the role of the FSP in the GPU boot

sequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. +

Documentation/gpu/nova/core/fsp.rst new file +3 -0
+What is FSP?
+============
+The Foundation Security Processor (FSP) is the GPU's Internal Root

of Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. +
+FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver.

Documentation/gpu/nova/core/fsp.rst new file +4 -0
+
+Simplified boot flow (Hopper/Blackwell)
+=======================================
+Starting with Hopper, the boot flow is significantly simplified

compared to +earlier GPU generations like Ampere.

Quoted 8 lines
Quoted 5 lines
Documentation/gpu/nova/core/fsp.rst new file +2 -0
+
+On an **Ampere** GPU, the boot verification chain involves

multiple Falcon +engines and multiple ucode stages (see falcon.rst for details):: +
+ Hardware BROM (SEC2)

Is this referring to FWSEC-FRTS? If so, should this be GSP instead of SEC2?

Hi Alex:

This refers to the BootROM in the SEC2 Peregrine. From the timeline perspective, FWSEC-FRTS should be positioned as below. I was thinking to show the brief differences of relationship between internal processors so I omit the FWSEC-FRTS in the sequence. Do you think we should add it?

Documentation/gpu/nova/core/fsp.rst line 35

Quoted 4 lines
Documentation/gpu/nova/core/fsp.rst new file +3 -0
+     Hardware BROM (SEC2)
                    here  < ----- FWSEC-FRTS (GSP)
+          -> HS Booter (SEC2)
+               -> LS GSP-RM (GSP)

Ah, I understand now - the boot ROM that verifies the HS Booter. No, what you wrote makes perfect sense, I just misunderstood.

Pushed as-is to drm-rust-next! Thanks for taking the time to put this together.

author
#18 lore raw

PATCH 6/9 gpu: nova-core: consolidate GSP boot parameters into GspBootContext

In reply to Zhi Wang

The GspHal trait methods boot() and post_boot() accept a long list of individual parameters (dev, bar, chipset, gsp_falcon, sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.

Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a dev() helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.

Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in Gsp::boot() accordingly.

---
 drivers/gpu/nova-core/gpu.rs           | 14 ++++++-
 drivers/gpu/nova-core/gsp.rs           | 22 +++++++++++
 drivers/gpu/nova-core/gsp/boot.rs      | 55 ++++++++++++--------------
 drivers/gpu/nova-core/gsp/hal.rs       | 23 +++--------
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++---------
 6 files changed, 85 insertions(+), 74 deletions(-)
drivers/gpu/nova-core/gpu.rs +12 -2
index b3c91731db45..69569e218d9b 100644
@@ -23,7 +23,8 @@
     fb::SysmemFlush,
     gsp::{
         self,
-        Gsp, //
+        Gsp,
+        GspBootContext, //
     },
     regs,
 };
@@ -323,7 +324,16 @@ pub(crate) fn new(
             // This member must be initialized last, so the `UnloadBundle` can never be dropped from
             // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run
             // in case of failure.
-            unload_bundle: gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)?,
+            unload_bundle: {
+                let ctx = GspBootContext {
+                    pdev,
+                    bar,
+                    chipset: spec.chipset,
+                    gsp_falcon,
+                    sec2_falcon,
+                };
+                gsp.boot(&ctx)?
+            },
             bar,
         })
     }
drivers/gpu/nova-core/gsp.rs +22 -0
index 69175ca3315c..f0901bf86258 100644
@@ -31,6 +31,13 @@
 };
 
 use crate::{
+    driver::Bar0,
+    falcon::{
+        gsp::Gsp as GspFalcon,
+        sec2::Sec2 as Sec2Falcon,
+        Falcon, //
+    },
+    gpu::Chipset,
     gsp::cmdq::Cmdq,
     gsp::fw::{
         GspArgumentsPadded,
@@ -42,6 +49,21 @@
 pub(crate) const GSP_PAGE_SHIFT: usize = 12;
 pub(crate) const GSP_PAGE_SIZE: usize = 1 << GSP_PAGE_SHIFT;
 
+/// Common context for the GSP boot process.
+pub(crate) struct GspBootContext<'a> {
+    pub(crate) pdev: &'a pci::Device<device::Bound>,
+    pub(crate) bar: Bar0<'a>,
+    pub(crate) chipset: Chipset,
+    pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
+    pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
+}
+
+impl<'a> GspBootContext<'a> {
+    pub(crate) fn dev(&self) -> &'a device::Device<device::Bound> {
+        self.pdev.as_ref()
+    }
+}
+
 /// Number of GSP pages to use in a RM log buffer.
 const RM_LOG_BUFFER_NUM_PAGES: usize = 0x10;
 const LOG_BUFFER_SIZE: usize = RM_LOG_BUFFER_NUM_PAGES * GSP_PAGE_SIZE;
drivers/gpu/nova-core/gsp/boot.rs +25 -30

100 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal.rs +5 -18
index 04f004856c60..51a277fe97bb 100644
@@ -4,11 +4,10 @@
 mod gh100;
 mod tu102;
 
-use kernel::prelude::*;
-
 use kernel::{
     device,
-    dma::Coherent, //
+    dma::Coherent,
+    prelude::*, //
 };
 
 use crate::{
@@ -27,6 +26,7 @@
     gsp::{
         boot::BootUnloadGuard,
         Gsp,
+        GspBootContext,
         GspFwWprMeta, //
     },
 };
@@ -53,32 +53,19 @@ pub(super) trait GspHal: Send {
     ///
     /// Upon success, returns a guard that runs the GSP unload sequence if GSP boot does not
     /// complete.
-    #[allow(clippy::too_many_arguments)]
     fn boot<'a>(
         &self,
         gsp: &'a Gsp,
-        dev: &'a device::Device<device::Bound>,
-        bar: Bar0<'a>,
-        chipset: Chipset,
+        ctx: &GspBootContext<'a>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-        gsp_falcon: &'a Falcon<GspEngine>,
-        sec2_falcon: &'a Falcon<Sec2>,
     ) -> Result<BootUnloadGuard<'a>>;
 
     /// Performs HAL-specific post-GSP boot tasks.
     ///
     /// This method is called by the GSP boot code after the GSP is confirmed to be running, and
     /// after the initialization commands have been pushed onto its queue.
-    fn post_boot(
-        &self,
-        _gsp: &Gsp,
-        _dev: &device::Device<device::Bound>,
-        _bar: Bar0<'_>,
-        _gsp_fw: &GspFirmware,
-        _gsp_falcon: &Falcon<GspEngine>,
-        _sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    fn post_boot(&self, _gsp: &Gsp, _ctx: &GspBootContext<'_>, _gsp_fw: &GspFirmware) -> Result {
         Ok(())
     }
 }
drivers/gpu/nova-core/gsp/hal/gh100.rs +8 -6
index 98f5ce197d13..c9fdc8cacedc 100644
@@ -26,7 +26,6 @@
         FmcBootArgs,
         Fsp, //
     },
-    gpu::Chipset,
     gsp::{
         boot::BootUnloadGuard,
         hal::{
@@ -34,6 +33,7 @@
             UnloadBundle, //
         },
         Gsp,
+        GspBootContext,
         GspFwWprMeta, //
     },
 };
@@ -152,14 +152,16 @@ impl GspHal for Gh100 {
     fn boot<'a>(
         &self,
         gsp: &'a Gsp,
-        dev: &'a device::Device<device::Bound>,
-        bar: Bar0<'a>,
-        chipset: Chipset,
+        ctx: &GspBootContext<'a>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-        gsp_falcon: &'a Falcon<GspEngine>,
-        sec2_falcon: &'a Falcon<Sec2>,
     ) -> Result<BootUnloadGuard<'a>> {
+        let dev = ctx.dev();
+        let bar = ctx.bar;
+        let chipset = ctx.chipset;
+        let gsp_falcon = ctx.gsp_falcon;
+        let sec2_falcon = ctx.sec2_falcon;
+
         let fsp_fw = FspFirmware::new(dev, chipset, FIRMWARE_VERSION)?;
 
         let unload_bundle = crate::gsp::UnloadBundle(
drivers/gpu/nova-core/gsp/hal/tu102.rs +13 -18
index 2f6301af7113..5b4325f16930 100644
@@ -42,6 +42,7 @@
             GspSequencerParams, //
         },
         Gsp,
+        GspBootContext,
         GspFwWprMeta, //
     },
     regs,
@@ -258,14 +259,16 @@ impl GspHal for Tu102 {
     fn boot<'a>(
         &self,
         gsp: &'a Gsp,
-        dev: &'a device::Device<device::Bound>,
-        bar: Bar0<'a>,
-        chipset: Chipset,
+        ctx: &GspBootContext<'a>,
         fb_layout: &FbLayout,
         wpr_meta: &Coherent<GspFwWprMeta>,
-        gsp_falcon: &'a Falcon<GspEngine>,
-        sec2_falcon: &'a Falcon<Sec2>,
     ) -> Result<BootUnloadGuard<'a>> {
+        let dev = ctx.dev();
+        let bar = ctx.bar;
+        let chipset = ctx.chipset;
+        let gsp_falcon = ctx.gsp_falcon;
+        let sec2_falcon = ctx.sec2_falcon;
+
         let bios = Vbios::new(dev, bar)?;
 
         // Try and prepare the unload bundle.
@@ -321,23 +324,15 @@ fn boot<'a>(
         Ok(unload_guard)
     }
 
-    fn post_boot(
-        &self,
-        gsp: &Gsp,
-        dev: &device::Device<device::Bound>,
-        bar: Bar0<'_>,
-        gsp_fw: &GspFirmware,
-        gsp_falcon: &Falcon<GspEngine>,
-        sec2_falcon: &Falcon<Sec2>,
-    ) -> Result {
+    fn post_boot(&self, gsp: &Gsp, ctx: &GspBootContext<'_>, gsp_fw: &GspFirmware) -> Result {
         // Create and run the GSP sequencer.
         let seq_params = GspSequencerParams {
             bootloader_app_version: gsp_fw.bootloader.app_version,
             libos_dma_handle: gsp.libos.dma_handle(),
-            gsp_falcon,
-            sec2_falcon,
-            dev,
-            bar,
+            gsp_falcon: ctx.gsp_falcon,
+            sec2_falcon: ctx.sec2_falcon,
+            dev: ctx.dev(),
+            bar: ctx.bar,
         };
         GspSequencer::run(&gsp.cmdq, seq_params)?;
 
Signature · 1 line
2.51.0
#19 lore raw

Re: [PATCH 6/9] gpu: nova-core: consolidate GSP boot parameters into GspBootContext

Quoted 18 lines · Zhi Wang

The GspHal trait methods boot() and post_boot() accept a long list of individual parameters (dev, bar, chipset, gsp_falcon, sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.

Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a dev() helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.

Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in Gsp::boot() accordingly.

Quoted 8 lines
---
 drivers/gpu/nova-core/gpu.rs           | 14 ++++++-
 drivers/gpu/nova-core/gsp.rs           | 22 +++++++++++
 drivers/gpu/nova-core/gsp/boot.rs      | 55 ++++++++++++--------------
 drivers/gpu/nova-core/gsp/hal.rs       | 23 +++--------
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++---------
 6 files changed, 85 insertions(+), 74 deletions(-)

I think this one is also good to take independently, it groups all the context required for booting the GSP in a dedicated type, which is something we might want to do as the SEC2 and FSP paths require different resources (and thus could use different associated types).

So I'm leaning towards merging this early as it is independent from the preceding patches in the series and makes the first step in that direction.

maintainer Applied
#20 lore raw

Re: [PATCH 6/9] gpu: nova-core: consolidate GSP boot parameters into GspBootContext

Quoted 28 lines · Alexandre Courbot
Quoted 18 lines · Zhi Wang

The GspHal trait methods boot() and post_boot() accept a long list of individual parameters (dev, bar, chipset, gsp_falcon, sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.

Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a dev() helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.

Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in Gsp::boot() accordingly.

Quoted 8 lines
---
 drivers/gpu/nova-core/gpu.rs           | 14 ++++++-
 drivers/gpu/nova-core/gsp.rs           | 22 +++++++++++
 drivers/gpu/nova-core/gsp/boot.rs      | 55 ++++++++++++--------------
 drivers/gpu/nova-core/gsp/hal.rs       | 23 +++--------
 drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++---
 drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++---------
 6 files changed, 85 insertions(+), 74 deletions(-)
Quoted 8 lines

I think this one is also good to take independently, it groups all the context required for booting the GSP in a dedicated type, which is something we might want to do as the SEC2 and FSP paths require different resources (and thus could use different associated types).

So I'm leaning towards merging this early as it is independent from the preceding patches in the series and makes the first step in that direction.

Merged into drm-rust-next, thanks! With two minor edits:

[acourbot: pass GspBootContext by value to Gsp::boot.] │

So the boot context can be modified by Gsp::boot if needed (although that shouldn't be necessary, but it is more natural to pass by value here regardless).

[acourbot: deconstruct GspBootContext in Gsp::boot to simplify diff.]

To stay consistent with what was done in the HALs. This also makes the diff considerably smaller and easier to parse.

author
#21 lore raw

PATCH 7/9 gpu: nova-core: add vGPU preludes

In reply to Zhi Wang

The driver needs to detect vGPU capability before firmware loading so that the GSP boot flow can be adjusted accordingly.

vGPU capability is determined by two sources: the PCI SR-IOV totalvfs count (whether the device advertises VFs) and the FSP PRC (Product Reconfiguration Control) knob (whether vGPU mode is actively enabled on this boot cycle). Both must agree for vGPU to proceed.

Introduce VgpuManager to encapsulate vGPU state detection and tracking. On creation it queries totalvfs via sriov_get_totalvfs(); during GSP boot the PRC knob is read from FSP to refine the initial estimate. Extend GspBootContext with vgpu_requested and total_vfs fields to carry this state across the boot sequence. The FSP PRC read is performed inside the GH100 HAL boot method, where the FSP falcon is already available.

---
 drivers/gpu/nova-core/fsp.rs           |  1 -
 drivers/gpu/nova-core/gpu.rs           | 24 +++++++++++--
 drivers/gpu/nova-core/gsp.rs           |  5 +++
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  8 ++++-
 drivers/gpu/nova-core/nova_core.rs     |  1 +
 drivers/gpu/nova-core/vgpu.rs          | 47 ++++++++++++++++++++++++++
 6 files changed, 82 insertions(+), 4 deletions(-)
 create mode 100644 drivers/gpu/nova-core/vgpu.rs
drivers/gpu/nova-core/fsp.rs +0 -1
index ce11efeba37e..c775e12c5451 100644
@@ -329,7 +329,6 @@ impl Fsp {
     /// Queries FSP's Management Partition for the active vGPU mode knob value.
     /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU,
     /// [`VgpuMode::Disabled`] otherwise.
-    #[expect(dead_code)]
     pub(crate) fn read_vgpu_mode(
         &mut self,
         dev: &device::Device<device::Bound>,
drivers/gpu/nova-core/gpu.rs +22 -2
index 69569e218d9b..6d8e60dd5292 100644
@@ -9,7 +9,11 @@
     io::Io,
     num::Bounded,
     pci,
-    prelude::*, //
+    prelude::*,
+    sync::{
+        new_mutex,
+        Mutex, //
+    },
 };
 
 use crate::{
@@ -27,6 +31,7 @@
         GspBootContext, //
     },
     regs,
+    vgpu::VgpuManager,
 };
 
 mod hal;
@@ -180,6 +185,13 @@ pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> {
     }
 }
 
+impl Architecture {
+    /// Returns true for architectures that support vGPU (RTX PRO 6000 Blackwell Server Edition).
+    pub(crate) const fn supports_vgpu(&self) -> bool {
+        matches!(self, Self::BlackwellGB20x)
+    }
+}
+
 #[derive(Clone, Copy)]
 pub(crate) struct Revision {
     major: Bounded<u8, 4>,
@@ -278,9 +290,12 @@ pub(crate) struct Gpu<'gpu> {
     gsp_falcon: Falcon<GspFalcon>,
     /// SEC2 falcon instance, used for GSP boot up and cleanup.
     sec2_falcon: Falcon<Sec2Falcon>,
-    /// GSP runtime data. Temporarily an empty placeholder.
+    /// GSP runtime data.
     #[pin]
     gsp: Gsp,
+    /// vGPU state (SR-IOV / FSP PRC), behind Mutex for FFI concurrency.
+    #[pin]
+    vgpu: Mutex<VgpuManager>,
     /// GSP unload firmware bundle, if any.
     unload_bundle: Option<gsp::UnloadBundle>,
 }
@@ -319,18 +334,23 @@ pub(crate) fn new(
 
             sec2_falcon: Falcon::new(pdev.as_ref(), spec.chipset)?,
 
+            vgpu <- new_mutex!(VgpuManager::new(pdev, spec.chipset.arch())?, "vgpu_manager"),
+
             gsp <- Gsp::new(pdev),
 
             // This member must be initialized last, so the `UnloadBundle` can never be dropped from
             // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run
             // in case of failure.
             unload_bundle: {
+                let mgr = vgpu.lock();
                 let ctx = GspBootContext {
                     pdev,
                     bar,
                     chipset: spec.chipset,
                     gsp_falcon,
                     sec2_falcon,
+                    vgpu_requested: core::cell::Cell::new(mgr.vgpu_requested),
+                    total_vfs: mgr.total_vfs,
                 };
                 gsp.boot(&ctx)?
             },
drivers/gpu/nova-core/gsp.rs +5 -0
index f0901bf86258..94cd4a784b79 100644
@@ -3,6 +3,8 @@
 mod boot;
 mod hal;
 
+use core::cell::Cell;
+
 use kernel::{
     debugfs,
     device,
@@ -56,6 +58,9 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) chipset: Chipset,
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
+    pub(crate) vgpu_requested: Cell<bool>,
+    #[expect(dead_code)]
+    pub(crate) total_vfs: u16,
 }
 
 impl<'a> GspBootContext<'a> {
drivers/gpu/nova-core/gsp/hal/gh100.rs +7 -1
index c9fdc8cacedc..e133a92fb67f 100644
@@ -24,7 +24,8 @@
     },
     fsp::{
         FmcBootArgs,
-        Fsp, //
+        Fsp,
+        VgpuMode, //
     },
     gsp::{
         boot::BootUnloadGuard,
@@ -174,6 +175,11 @@ fn boot<'a>(
 
         let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset, fsp_fw)?;
 
+        let vgpu_mode = fsp.read_vgpu_mode(dev, bar)?;
+        dev_dbg!(dev, "vGPU mode: {:?}\n", vgpu_mode);
+        ctx.vgpu_requested
+            .set(ctx.vgpu_requested.get() && vgpu_mode == VgpuMode::Enabled);
+
         let args = FmcBootArgs::new(
             dev,
             chipset,
drivers/gpu/nova-core/nova_core.rs +1 -0
index 9f0199f7b38c..6f55a9242027 100644
@@ -26,6 +26,7 @@
 mod regs;
 mod sbuffer;
 mod vbios;
+mod vgpu;
 
 pub(crate) const MODULE_NAME: &core::ffi::CStr = <LocalModule as kernel::ModuleMetadata>::NAME;
 
drivers/gpu/nova-core/vgpu.rs new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.51.0
#22 lore raw

Re: [PATCH 7/9] gpu: nova-core: add vGPU preludes

Quoted 65 lines · Zhi Wang

The driver needs to detect vGPU capability before firmware loading so that the GSP boot flow can be adjusted accordingly.

vGPU capability is determined by two sources: the PCI SR-IOV totalvfs count (whether the device advertises VFs) and the FSP PRC (Product Reconfiguration Control) knob (whether vGPU mode is actively enabled on this boot cycle). Both must agree for vGPU to proceed.

Introduce VgpuManager to encapsulate vGPU state detection and tracking. On creation it queries totalvfs via sriov_get_totalvfs(); during GSP boot the PRC knob is read from FSP to refine the initial estimate. Extend GspBootContext with vgpu_requested and total_vfs fields to carry this state across the boot sequence. The FSP PRC read is performed inside the GH100 HAL boot method, where the FSP falcon is already available.

---
 drivers/gpu/nova-core/fsp.rs           |  1 -
 drivers/gpu/nova-core/gpu.rs           | 24 +++++++++++--
 drivers/gpu/nova-core/gsp.rs           |  5 +++
 drivers/gpu/nova-core/gsp/hal/gh100.rs |  8 ++++-
 drivers/gpu/nova-core/nova_core.rs     |  1 +
 drivers/gpu/nova-core/vgpu.rs          | 47 ++++++++++++++++++++++++++
 6 files changed, 82 insertions(+), 4 deletions(-)
 create mode 100644 drivers/gpu/nova-core/vgpu.rs
drivers/gpu/nova-core/fsp.rs +0 -1
index ce11efeba37e..c775e12c5451 100644
@@ -329,7 +329,6 @@ impl Fsp {
     /// Queries FSP's Management Partition for the active vGPU mode knob value.
     /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU,
     /// [`VgpuMode::Disabled`] otherwise.
-    #[expect(dead_code)]
     pub(crate) fn read_vgpu_mode(
         &mut self,
         dev: &device::Device<device::Bound>,
drivers/gpu/nova-core/gpu.rs +6 -1
index 69569e218d9b..6d8e60dd5292 100644
@@ -9,7 +9,11 @@
     io::Io,
     num::Bounded,
     pci,
-    prelude::*, //
+    prelude::*,
+    sync::{
+        new_mutex,
+        Mutex, //
+    },
 };
 
 use crate::{
@@ -27,6 +31,7 @@
         GspBootContext, //
     },
     regs,
+    vgpu::VgpuManager,
 };
 
 mod hal;

drivers/gpu/nova-core/gpu.rs pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> { · line 193

Quoted 9 lines
drivers/gpu/nova-core/gpu.rs +6 -0
@@ -180,6 +185,13 @@ pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> {
     }
 }
 
+impl Architecture {
+    /// Returns true for architectures that support vGPU (RTX PRO 6000 Blackwell Server Edition).
+    pub(crate) const fn supports_vgpu(&self) -> bool {
+        matches!(self, Self::BlackwellGB20x)
+    }
+}

Is this needed at all? It is only used in VgpuManager::new to condition the call to sriov_get_totalvfs. But sriov_get_totalvfs should return 0 if the device does not support virtual functions, which is the value used on devices for which this method returns false. So unless we can have virtual functions without vGPU support, it seems like we can rely solely on the result of sriov_get_totalvfs.

Another potential gate is read_vgpu_mode - if this always returns false on architectures that don't support vGPU, then we can also rely on that instead of hardcoding supported archs in the driver.

In case we need to keep this method, it should use a HAL method (see pci_config_mirror_range for an example). But I'd rather just remove it if that's possible.

Quoted 5 lines
drivers/gpu/nova-core/gpu.rs +1 -0
+
 #[derive(Clone, Copy)]
 pub(crate) struct Revision {
     major: Bounded<u8, 4>,

drivers/gpu/nova-core/gpu.rs pub(crate) struct Gpu<'gpu> { · line 298

Quoted 10 lines
drivers/gpu/nova-core/gpu.rs +4 -1
@@ -278,9 +290,12 @@ pub(crate) struct Gpu<'gpu> {
     gsp_falcon: Falcon<GspFalcon>,
     /// SEC2 falcon instance, used for GSP boot up and cleanup.
     sec2_falcon: Falcon<Sec2Falcon>,
-    /// GSP runtime data. Temporarily an empty placeholder.
+    /// GSP runtime data.
     #[pin]
     gsp: Gsp,
+    /// vGPU state (SR-IOV / FSP PRC), behind Mutex for FFI concurrency.
+    #[pin]
+    vgpu: Mutex<VgpuManager>,

The Mutex should not be needed - afaict vgpu is not modified, and we can get a mutable reference to it while we are contructing the GPU anyway.

Quoted 41 lines
drivers/gpu/nova-core/gpu.rs +5 -0
     /// GSP unload firmware bundle, if any.
     unload_bundle: Option<gsp::UnloadBundle>,
 }
@@ -319,18 +334,23 @@ pub(crate) fn new(
 
             sec2_falcon: Falcon::new(pdev.as_ref(), spec.chipset)?,
 
+            vgpu <- new_mutex!(VgpuManager::new(pdev, spec.chipset.arch())?, "vgpu_manager"),
+
             gsp <- Gsp::new(pdev),
 
             // This member must be initialized last, so the `UnloadBundle` can never be dropped from
             // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run
             // in case of failure.
             unload_bundle: {
+                let mgr = vgpu.lock();
                 let ctx = GspBootContext {
                     pdev,
                     bar,
                     chipset: spec.chipset,
                     gsp_falcon,
                     sec2_falcon,
+                    vgpu_requested: core::cell::Cell::new(mgr.vgpu_requested),
+                    total_vfs: mgr.total_vfs,
                 };
                 gsp.boot(&ctx)?
             },
drivers/gpu/nova-core/gsp.rs +2 -0
index f0901bf86258..94cd4a784b79 100644
@@ -3,6 +3,8 @@
 mod boot;
 mod hal;
 
+use core::cell::Cell;
+
 use kernel::{
     debugfs,
     device,

drivers/gpu/nova-core/gsp.rs pub(crate) struct GspBootContext<'a> { · line 61

Quoted 4 lines
drivers/gpu/nova-core/gsp.rs +1 -0
@@ -56,6 +58,9 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) chipset: Chipset,
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
+    pub(crate) vgpu_requested: Cell<bool>,

I think we can do without the Cell. Since we own the GspBootContext, we can pass it by value to Gsp::boot, which can them modify it, and pass mutable references to it to the other methods is calls. But it would be even better if we didn't need to modify this at all.

The problem seems to come from the fact that we only build Fsp in the GSP boot code. But vGPU would ideally query it outside of the boot sequence. So a better design would probably be to make Fsp a member of Gpu (behind an Option, as all GPUs don't have one) so we can call read_vgpu_mode outside of the GSP boot sequence and get the definitive enabled state of vGPU earlier in the GPU construction.

I think this is going to be needed anyway because patch 9 also wants to pass the vGPU enabled status to build FbLayout, but can only do so *before* we query the FSP about the PRC knob. So it doesn't necessarily have the correct state.

Another benefit of having the Fsp moved to Gpu will be that you can then pass a reference to it to the VgpuManager::new and have all the vGPU stuff done in vgpu.rs.

Once we have that, we should also be able to add a reference to the (now read-only) VgpuManager to GspBootContext, and get the parameters directly from the source instead of duplicating them.

I can work on a patch to move Fsp if my suggestion is not clear, or let me know if you prefer to do it yourself.

Quoted 85 lines
drivers/gpu/nova-core/gsp.rs +2 -0
+    #[expect(dead_code)]
+    pub(crate) total_vfs: u16,
 }
 
 impl<'a> GspBootContext<'a> {
drivers/gpu/nova-core/gsp/hal/gh100.rs +7 -1
index c9fdc8cacedc..e133a92fb67f 100644
@@ -24,7 +24,8 @@
     },
     fsp::{
         FmcBootArgs,
-        Fsp, //
+        Fsp,
+        VgpuMode, //
     },
     gsp::{
         boot::BootUnloadGuard,
@@ -174,6 +175,11 @@ fn boot<'a>(
 
         let mut fsp = Fsp::wait_secure_boot(dev, bar, chipset, fsp_fw)?;
 
+        let vgpu_mode = fsp.read_vgpu_mode(dev, bar)?;
+        dev_dbg!(dev, "vGPU mode: {:?}\n", vgpu_mode);
+        ctx.vgpu_requested
+            .set(ctx.vgpu_requested.get() && vgpu_mode == VgpuMode::Enabled);
+
         let args = FmcBootArgs::new(
             dev,
             chipset,
drivers/gpu/nova-core/nova_core.rs +1 -0
index 9f0199f7b38c..6f55a9242027 100644
@@ -26,6 +26,7 @@
 mod regs;
 mod sbuffer;
 mod vbios;
+mod vgpu;
 
 pub(crate) const MODULE_NAME: &core::ffi::CStr = <LocalModule as kernel::ModuleMetadata>::NAME;
 
drivers/gpu/nova-core/vgpu.rs new file +36 -0
new file mode 100644
index 000000000000..c8f1b74037c7
@@ -0,0 +1,47 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::{
+    device,
+    pci,
+    prelude::*, //
+};
+
+use crate::gpu::Architecture;
+
+/// vGPU manager.
+///
+/// On creation, performs platform detection to determine whether vGPU is
+/// requested (PRC knob + totalvfs for Blackwell). The `vgpu_requested`
+/// flag may be further refined during boot (e.g. FSP PRC knob read).
+pub(crate) struct VgpuManager {
+    pub(crate) vgpu_requested: bool,
+    pub(crate) vgpu_enabled: bool,
+    pub(crate) total_vfs: u16,
+}
+
+impl VgpuManager {
+    pub(crate) fn new(
+        pdev: &pci::Device<device::Bound>,
+        arch: Architecture,
+    ) -> Result<VgpuManager> {
+        let total_vfs: u16 = if arch.supports_vgpu() {
+            pdev.sriov_get_totalvfs()
+                .ok()
+                .and_then(|n| n.try_into().ok())
+                .unwrap_or(0)
+        } else {
+            0
+        };
+
+        Ok(VgpuManager {

drivers/gpu/nova-core/vgpu.rs line 46

Quoted 10 lines
drivers/gpu/nova-core/vgpu.rs new file +10 -0
+            vgpu_requested: total_vfs > 0,
+            vgpu_enabled: false,
+            total_vfs,
+        })
+    }
+
+    #[expect(dead_code)]
+    pub(crate) fn set_vgpu_enabled(&mut self, enabled: bool) {
+        self.vgpu_enabled = enabled;
+    }

This method doesn't seem to be called anywhere in the series - let's remove it if it is not needed.

author
#23 lore raw

PATCH 8/9 gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled

In reply to Zhi Wang

The registry object "RMSetSriovMode" is required to be set when vGPU is enabled.

Convert SetRegistry to use KVec<RegistryEntry> for dynamic construction, allowing entries to be added conditionally at runtime.

Set "RMSetSriovMode" to 1 when nova-core is loading the GSP firmware and initialize the GSP registry objects, if vGPU is enabled.

---
 drivers/gpu/nova-core/gsp/boot.rs     |  2 +-
 drivers/gpu/nova-core/gsp/commands.rs | 93 ++++++++++++++++++---------
 drivers/gpu/nova-core/gsp/fw.rs       |  4 ++
 3 files changed, 67 insertions(+), 32 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +1 -1
index 6e170401d616..2981d02d15ad 100644
@@ -139,7 +139,7 @@ pub(crate) fn boot(
         self.cmdq
             .send_command_no_wait(ctx.bar, commands::SetSystemInfo::new(ctx.pdev, ctx.chipset))?;
         self.cmdq
-            .send_command_no_wait(ctx.bar, commands::SetRegistry::new())?;
+            .send_command_no_wait(ctx.bar, commands::SetRegistry::new(ctx.vgpu_requested.get())?)?;
 
         hal.post_boot(&self, ctx, &gsp_fw)?;
 
drivers/gpu/nova-core/gsp/commands.rs +62 -31

134 lines not on this page · raw message

drivers/gpu/nova-core/gsp/fw.rs +4 -0
index 4db0cfa4dc4d..14424a2c2d83 100644
@@ -299,6 +299,7 @@ pub(crate) enum MsgFunction {
     OsErrorLog = bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG,
     PostEvent = bindings::NV_VGPU_MSG_EVENT_POST_EVENT,
     RcTriggered = bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED,
+    GpuacctPerfmonUtilSamples = bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES,
     UcodeLibOsPrint = bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT,
 }
 
@@ -348,6 +349,9 @@ fn try_from(value: u32) -> Result<MsgFunction> {
             bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG => Ok(MsgFunction::OsErrorLog),
             bindings::NV_VGPU_MSG_EVENT_POST_EVENT => Ok(MsgFunction::PostEvent),
             bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED => Ok(MsgFunction::RcTriggered),
+            bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES => {
+                Ok(MsgFunction::GpuacctPerfmonUtilSamples)
+            }
             bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT => Ok(MsgFunction::UcodeLibOsPrint),
             _ => Err(EINVAL),
         }
Signature · 1 line
2.51.0
#24 lore raw

Re: [PATCH 8/9] gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled

Quoted 10 lines · Zhi Wang

The registry object "RMSetSriovMode" is required to be set when vGPU is enabled.

Convert SetRegistry to use KVec<RegistryEntry> for dynamic construction, allowing entries to be added conditionally at runtime.

Set "RMSetSriovMode" to 1 when nova-core is loading the GSP firmware and initialize the GSP registry objects, if vGPU is enabled.

Can you split the conversion to KVec and the addition of RMSetSriovMode into two separate patches? The conversion to KVec will likely be useful on its own, and splitting also makes review easier as the concerns are well-separated.

<...>

Quoted 9 lines
drivers/gpu/nova-core/gsp/fw.rs +1 -0
@@ -299,6 +299,7 @@ pub(crate) enum MsgFunction {
     OsErrorLog = bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG,
     PostEvent = bindings::NV_VGPU_MSG_EVENT_POST_EVENT,
     RcTriggered = bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED,
+    GpuacctPerfmonUtilSamples = bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES,
     UcodeLibOsPrint = bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT,
 }
 

drivers/gpu/nova-core/gsp/fw.rs fn try_from(value: u32) -> Result<MsgFunction> { · line 354

Quoted 6 lines
drivers/gpu/nova-core/gsp/fw.rs +3 -0
@@ -348,6 +349,9 @@ fn try_from(value: u32) -> Result<MsgFunction> {
             bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG => Ok(MsgFunction::OsErrorLog),
             bindings::NV_VGPU_MSG_EVENT_POST_EVENT => Ok(MsgFunction::PostEvent),
             bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED => Ok(MsgFunction::RcTriggered),
+            bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES => {
+                Ok(MsgFunction::GpuacctPerfmonUtilSamples)
+            }

Why is this needed?

author
#25 lore raw

PATCH 1/1 gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

In reply to Zhi Wang

GSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.

The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.

Select a pre-calibrated static heap size based on total_vfs (174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and set vf_partition_count accordingly. Extend FbLayout::new() and GspBootContext to propagate total_vfs through the boot path.

---
 drivers/gpu/nova-core/fb.rs       | 17 +++++++++++++----
 drivers/gpu/nova-core/gsp.rs      |  2 +-
 drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++---
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ++++++++++++
 4 files changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/fb.rs +13 -4
index 725e428154cf..fb4e6aa9fda4 100644
@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {
 
 impl FbLayout {
     /// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware.
-    pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Result<Self> {
+    pub(crate) fn new(
+        chipset: Chipset,
+        bar: Bar0<'_>,
+        gsp_fw: &GspFirmware,
+        vgpu_requested: bool,
+        total_vfs: u16,
+    ) -> Result<Self> {
         let hal = hal::fb_hal(chipset);
 
         let fb = {
@@ -236,8 +242,11 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu
 
         let wpr2_heap = {
             const WPR2_HEAP_DOWN_ALIGN: Alignment = Alignment::new::<SZ_1M>();
-            let wpr2_heap_size =
-                gsp::LibosParams::from_chipset(chipset).wpr_heap_size(chipset, fb.end)?;
+            let wpr2_heap_size = if vgpu_requested {
+                gsp::vgpu_fw_heap_size(u32::from(total_vfs))
+            } else {
+                gsp::LibosParams::from_chipset(chipset).wpr_heap_size(chipset, fb.end)?
+            };
             let wpr2_heap_addr = (elf.start - wpr2_heap_size).align_down(WPR2_HEAP_DOWN_ALIGN);
 
             FbRange(wpr2_heap_addr..(elf.start).align_down(WPR2_HEAP_DOWN_ALIGN))
@@ -265,7 +274,7 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu
             wpr2_heap,
             wpr2,
             heap,
-            vf_partition_count: 0,
+            vf_partition_count: if vgpu_requested { total_vfs as u8 } else { 0 },
             pmu_reserved_size: hal.pmu_reserved_size(),
         })
     }
drivers/gpu/nova-core/gsp.rs +1 -1
index 94cd4a784b79..921b92c9eb92 100644
@@ -27,6 +27,7 @@
 mod sequencer;
 
 pub(crate) use fw::{
+    vgpu_fw_heap_size,
     GspFmcBootParams,
     GspFwWprMeta,
     LibosParams, //
@@ -59,7 +60,6 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
     pub(crate) vgpu_requested: Cell<bool>,
-    #[expect(dead_code)]
     pub(crate) total_vfs: u16,
 }
 
drivers/gpu/nova-core/gsp/boot.rs +11 -3
index 2981d02d15ad..7c1f3f962fbe 100644
@@ -111,7 +111,13 @@ pub(crate) fn boot(
             GFP_KERNEL,
         )?;
 
-        let fb_layout = FbLayout::new(ctx.chipset, ctx.bar, &gsp_fw)?;
+        let fb_layout = FbLayout::new(
+            ctx.chipset,
+            ctx.bar,
+            &gsp_fw,
+            ctx.vgpu_requested.get(),
+            ctx.total_vfs,
+        )?;
         dev_dbg!(dev, "{:#x?}\n", fb_layout);
 
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
@@ -138,8 +144,10 @@ pub(crate) fn boot(
 
         self.cmdq
             .send_command_no_wait(ctx.bar, commands::SetSystemInfo::new(ctx.pdev, ctx.chipset))?;
-        self.cmdq
-            .send_command_no_wait(ctx.bar, commands::SetRegistry::new(ctx.vgpu_requested.get())?)?;
+        self.cmdq.send_command_no_wait(
+            ctx.bar,
+            commands::SetRegistry::new(ctx.vgpu_requested.get())?,
+        )?;
 
         hal.post_boot(&self, ctx, &gsp_fw)?;
 
drivers/gpu/nova-core/gsp/fw.rs +12 -0
index 14424a2c2d83..2f3cbc5d5114 100644
@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
 pub(crate) const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize =
     num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);
 
+const GSP_FW_HEAP_SIZE_VGPU_1VM: u64 = 174 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_DEFAULT: u64 = 581 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_48VMS: u64 = 1370 * u64::SZ_1M;
+
+pub(crate) fn vgpu_fw_heap_size(total_vfs: u32) -> u64 {
+    match total_vfs {
+        1 => GSP_FW_HEAP_SIZE_VGPU_1VM,
+        2..=32 => GSP_FW_HEAP_SIZE_VGPU_DEFAULT,
+        _ => GSP_FW_HEAP_SIZE_VGPU_48VMS,
+    }
+}
+
 /// Empty type to group methods related to heap parameters for running the GSP firmware.
 enum GspFwHeapParams {}
 
Signature · 1 line
2.51.0
#26 lore raw

Re: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

Quoted 28 lines · Zhi Wang

GSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.

The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.

Select a pre-calibrated static heap size based on total_vfs (174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and set vf_partition_count accordingly. Extend FbLayout::new() and GspBootContext to propagate total_vfs through the boot path.

---
 drivers/gpu/nova-core/fb.rs       | 17 +++++++++++++----
 drivers/gpu/nova-core/gsp.rs      |  2 +-
 drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++---
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ++++++++++++
 4 files changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/fb.rs +0 -0
index 725e428154cf..fb4e6aa9fda4 100644

drivers/gpu/nova-core/fb.rs pub(crate) struct FbLayout { · line 179

Quoted 10 lines
drivers/gpu/nova-core/fb.rs +6 -1
@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {
 
 impl FbLayout {
     /// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware.
-    pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Result<Self> {
+    pub(crate) fn new(
+        chipset: Chipset,
+        bar: Bar0<'_>,
+        gsp_fw: &GspFirmware,
+        vgpu_requested: bool,
+        total_vfs: u16,

total_vfs is only meaningful if vgpu_requested is true. So these two parameters would be better modeled using an Option, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):

    enum VgpuRequest {
        Disabled,
        Requested {
            total_vfs: u16,
        }
    }
Quoted 84 lines
drivers/gpu/nova-core/fb.rs +7 -3
+    ) -> Result<Self> {
         let hal = hal::fb_hal(chipset);
 
         let fb = {
@@ -236,8 +242,11 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu
 
         let wpr2_heap = {
             const WPR2_HEAP_DOWN_ALIGN: Alignment = Alignment::new::<SZ_1M>();
-            let wpr2_heap_size =
-                gsp::LibosParams::from_chipset(chipset).wpr_heap_size(chipset, fb.end)?;
+            let wpr2_heap_size = if vgpu_requested {
+                gsp::vgpu_fw_heap_size(u32::from(total_vfs))
+            } else {
+                gsp::LibosParams::from_chipset(chipset).wpr_heap_size(chipset, fb.end)?
+            };
             let wpr2_heap_addr = (elf.start - wpr2_heap_size).align_down(WPR2_HEAP_DOWN_ALIGN);
 
             FbRange(wpr2_heap_addr..(elf.start).align_down(WPR2_HEAP_DOWN_ALIGN))
@@ -265,7 +274,7 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu
             wpr2_heap,
             wpr2,
             heap,
-            vf_partition_count: 0,
+            vf_partition_count: if vgpu_requested { total_vfs as u8 } else { 0 },
             pmu_reserved_size: hal.pmu_reserved_size(),
         })
     }
drivers/gpu/nova-core/gsp.rs +1 -1
index 94cd4a784b79..921b92c9eb92 100644
@@ -27,6 +27,7 @@
 mod sequencer;
 
 pub(crate) use fw::{
+    vgpu_fw_heap_size,
     GspFmcBootParams,
     GspFwWprMeta,
     LibosParams, //
@@ -59,7 +60,6 @@ pub(crate) struct GspBootContext<'a> {
     pub(crate) gsp_falcon: &'a Falcon<GspFalcon>,
     pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>,
     pub(crate) vgpu_requested: Cell<bool>,
-    #[expect(dead_code)]
     pub(crate) total_vfs: u16,
 }
 
drivers/gpu/nova-core/gsp/boot.rs +11 -3
index 2981d02d15ad..7c1f3f962fbe 100644
@@ -111,7 +111,13 @@ pub(crate) fn boot(
             GFP_KERNEL,
         )?;
 
-        let fb_layout = FbLayout::new(ctx.chipset, ctx.bar, &gsp_fw)?;
+        let fb_layout = FbLayout::new(
+            ctx.chipset,
+            ctx.bar,
+            &gsp_fw,
+            ctx.vgpu_requested.get(),
+            ctx.total_vfs,
+        )?;
         dev_dbg!(dev, "{:#x?}\n", fb_layout);
 
         let wpr_meta = Coherent::init(dev, GFP_KERNEL, GspFwWprMeta::new(&gsp_fw, &fb_layout))?;
@@ -138,8 +144,10 @@ pub(crate) fn boot(
 
         self.cmdq
             .send_command_no_wait(ctx.bar, commands::SetSystemInfo::new(ctx.pdev, ctx.chipset))?;
-        self.cmdq
-            .send_command_no_wait(ctx.bar, commands::SetRegistry::new(ctx.vgpu_requested.get())?)?;
+        self.cmdq.send_command_no_wait(
+            ctx.bar,
+            commands::SetRegistry::new(ctx.vgpu_requested.get())?,
+        )?;
 
         hal.post_boot(&self, ctx, &gsp_fw)?;
 
drivers/gpu/nova-core/gsp/fw.rs +0 -0
index 14424a2c2d83..2f3cbc5d5114 100644

drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106

Quoted 6 lines
drivers/gpu/nova-core/gsp/fw.rs +3 -0
@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {
 pub(crate) const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize =
     num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);
 
+const GSP_FW_HEAP_SIZE_VGPU_1VM: u64 = 174 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_DEFAULT: u64 = 581 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_48VMS: u64 = 1370 * u64::SZ_1M;

Do we have a source of truth for these values? I can see definitions for GSP_FW_HEAP_SIZE_VGPU_DEFAULT and GSP_FW_HEAP_SIZE_VGPU_48VMS in OpenRM (meaning we could generate bindings for them), but cannot find a reference for GSP_FW_HEAP_SIZE_VGPU_1VM...

#27 lore raw

Re: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

Quoted 45 lines · Alexandre Courbot
Quoted 28 lines · Zhi Wang

GSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.

The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.

Select a pre-calibrated static heap size based on total_vfs (174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and set vf_partition_count accordingly. Extend FbLayout::new() and GspBootContext to propagate total_vfs through the boot path.

---
 drivers/gpu/nova-core/fb.rs       | 17 +++++++++++++----
 drivers/gpu/nova-core/gsp.rs      |  2 +-
 drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++---
 drivers/gpu/nova-core/gsp/fw.rs   | 12 ++++++++++++
 4 files changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/fb.rs +0 -0
index 725e428154cf..fb4e6aa9fda4 100644

drivers/gpu/nova-core/fb.rs pub(crate) struct FbLayout { · line 179

Quoted 10 lines
drivers/gpu/nova-core/fb.rs +6 -1
@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {
 
 impl FbLayout {
     /// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware.
-    pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Result<Self> {
+    pub(crate) fn new(
+        chipset: Chipset,
+        bar: Bar0<'_>,
+        gsp_fw: &GspFirmware,
+        vgpu_requested: bool,
+        total_vfs: u16,

total_vfs is only meaningful if vgpu_requested is true. So these two parameters would be better modeled using an Option, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):

Quoted 6 lines
    enum VgpuRequest {
        Disabled,
        Requested {
            total_vfs: u16,
        }
    }

If we can implement my suggestion of moving Fsp into Gpu of patch 7, then this can take a &VgpuManager and not require any new type to express the vGPU state.

author
#28 lore raw

Re: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

In reply to Alexandre Courbot
Quoted 1 line · On Tue, 16 Jun 2026 23:20:41 +0900 "Alexandre Courbot" <[email protected]>…

On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:

snip

Quoted 6 lines

total_vfs is only meaningful if vgpu_requested is true. So these two parameters would be better modeled using an Option, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):

Quoted 7 lines
    enum VgpuRequest {
        Disabled,
        Requested {
            total_vfs: u16,
        }
    }

Make sense. Let me try.

Quoted 1 line
Quoted 1 line

+ ) -> Result<Self> {

snip

Quoted 8 lines
Quoted 8 lines
(quoted diff) +0 -0

b/drivers/gpu/nova-core/gsp/fw.rs index 14424a2c2d83..2f3cbc5d5114 100644 --- a/drivers/gpu/nova-core/gsp/fw.rs
+++ b/drivers/gpu/nova-core/gsp/fw.rs

(quoted diff) +0 -0
@@ -101,6 +101,18 @@ pub(in crate::gsp) fn
 advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {

pub(crate) const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize = num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);

Quoted 8 lines

drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106

Quoted 3 lines
drivers/gpu/nova-core/gsp/fw.rs +3 -0
+const GSP_FW_HEAP_SIZE_VGPU_1VM: u64 = 174 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_DEFAULT: u64 = 581 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_48VMS: u64 = 1370 * u64::SZ_1M;  

Do we have a source of truth for these values? I can see definitions for GSP_FW_HEAP_SIZE_VGPU_DEFAULT and GSP_FW_HEAP_SIZE_VGPU_48VMS in OpenRM (meaning we could generate bindings for them), but cannot find a reference for GSP_FW_HEAP_SIZE_VGPU_1VM...

I see. Let me remove this one and we can add this when OpenRM has it.

Z.

#29 lore raw

Re: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled

Quoted 21 lines · Zhi Wang
Quoted 1 line · On Tue, 16 Jun 2026 23:20:41 +0900 "Alexandre Courbot" <[email protected]>…

On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:

snip

Quoted 6 lines

total_vfs is only meaningful if vgpu_requested is true. So these two parameters would be better modeled using an Option, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):

Quoted 7 lines
    enum VgpuRequest {
        Disabled,
        Requested {
            total_vfs: u16,
        }
    }
Quoted 1 line

Make sense. Let me try.

Actually, please check my later reply to this patch [1]. We should be able to completely get rid of this if we move the Fsp into Gpu as described in [2].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 22 lines
Quoted 2 lines
Quoted 2 lines

+ ) -> Result<Self> {

snip

Quoted 8 lines
Quoted 8 lines
(quoted diff) +0 -0

b/drivers/gpu/nova-core/gsp/fw.rs index 14424a2c2d83..2f3cbc5d5114 100644 --- a/drivers/gpu/nova-core/gsp/fw.rs
+++ b/drivers/gpu/nova-core/gsp/fw.rs

(quoted diff) +0 -0
@@ -101,6 +101,18 @@ pub(in crate::gsp) fn
 advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {

pub(crate) const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize = num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);

Quoted 8 lines

drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106

Quoted 3 lines
drivers/gpu/nova-core/gsp/fw.rs +3 -0
+const GSP_FW_HEAP_SIZE_VGPU_1VM: u64 = 174 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_DEFAULT: u64 = 581 * u64::SZ_1M;
+const GSP_FW_HEAP_SIZE_VGPU_48VMS: u64 = 1370 * u64::SZ_1M;  

Do we have a source of truth for these values? I can see definitions for GSP_FW_HEAP_SIZE_VGPU_DEFAULT and GSP_FW_HEAP_SIZE_VGPU_48VMS in OpenRM (meaning we could generate bindings for them), but cannot find a reference for GSP_FW_HEAP_SIZE_VGPU_1VM...

Quoted 1 line

I see. Let me remove this one and we can add this when OpenRM has it.

I am fine with keeping it if it is useful, but would like to know how its value was inferred. :)

And also, how stable these values are, since they will need to remain stable across firmware releases. If they turn out to be firmware-dependent, maybe we can turn them into TLV properties of the firmware.