gpu: nova-core: boot GSP with vGPU enabled on
Subsystems (9)
cover gpu: nova-core: boot GSP with vGPU enabled on
Booting up GSP with vGPU enabled is part of the first milestone (M1) together with rust fwctl abstraction [1] and nova-core fwctl driver [2] for upstream vGPU support, allowing us to validate the basic GSP boot flow with vGPU enabled, upload vGPU types even before the remaining nova-core dependencies are ready. The nova-vgpu WIP patches for all milestones can be found at [3].
v3:
- Consolidate GSP boot parameters into GspBootContext struct, passed through the HAL trait methods instead of individual arguments.
- Add vGPU preludes patch (VgpuManager, Architecture::supports_vgpu, GspBootContext fields) as preparation for the vGPU boot changes.
- Rebase on top of the latest drm-rust-next.
Dropped patches:
Dependencies still in progress:
- "populate GSP_VF_INFO when vGPU is enabled" - depends on the ExtSriovCapability abstraction [4] which is still under review and Gary's on-going io projection work [5].
Ada specific:
- "introduce vgpu_support module param" - select vGPU support based on module params.
- "load the scrubber ucode when vGPU support is enabled"
v2:
- Adopt early-return style (Dirk).
- Add #ifndef CONFIG_PCI_IOV helper to fix compilation when CONFIG_PCI_IOV is disabled, per (Alex).
- Change return type from Result<i32> to Result<u16> to match the PCI spec field width, avoiding try_from at call sites.
- GspVfInfo changed to tuple struct (Alex).
- Use unconditional constructor with Option wrapping instead of bool parameter. (Alex)
- Use full initialization expression instead of mutating a zeroed value.
- Use .chain() pattern in GspSetSystemInfo::init() for optional vGPU info. (Alex)
- Eliminate all magic numbers: add vf_bar_is_64bit() and read_vf_bar64_addr() to ExtSriovCapability using PCI bindings constants (PCI_BASE_ADDRESS_MEM_TYPE_MASK, etc.).
- Use KVec<RegistryEntry> for dynamic registry entry construction instead of hardcoded array (Timur, Joel, Alexandre).
- Replace magic numbers 32/48 with named binding constants MAX_PARTITIONS_WITH_GFID_32VM / MAX_PARTITIONS_WITH_GFID from OpenRM (Alex).
- Use read_poll_timeout() instead of single read for scrubber completion check (Joel).
- Use dev instead of pdev.as_ref() in dev_dbg! (Dirk).
- Change scrubber trigger condition from vgpu_requested to fb_layout.wpr2_heap.len() > SZ_256M, checking actual heap size instead of vGPU flag. (Alex).
[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/
[3] https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-wip
[4] https://lore.kernel.org/rust-for-linux/[email protected]/
[5] https://lore.kernel.org/rust-for-linux/[email protected]/
Zhi Wang (9):
rust: pci: expose sriov_get_totalvfs() helper
gpu: nova-core: factor out common FSP message header
gpu: nova-core: return FSP response buffer to caller
gpu: nova-core: read vGPU mode from FSP via PRC protocol
gpu: nova-core: add FSP and PRC protocol documentation
gpu: nova-core: consolidate GSP boot parameters into GspBootContext
gpu: nova-core: add vGPU preludes
gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled
gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
Documentation/gpu/nova/core/fsp.rst | 142 +++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + drivers/gpu/nova-core/fb.rs | 17 +- drivers/gpu/nova-core/fsp.rs | 207 ++++++++++++++++++++++--- drivers/gpu/nova-core/gpu.rs | 38 ++++- drivers/gpu/nova-core/gsp.rs | 27 ++++ drivers/gpu/nova-core/gsp/boot.rs | 60 ++++--- drivers/gpu/nova-core/gsp/commands.rs | 93 +++++++---- drivers/gpu/nova-core/gsp/fw.rs | 16 ++ drivers/gpu/nova-core/gsp/hal.rs | 23 +-- drivers/gpu/nova-core/gsp/hal/gh100.rs | 22 ++- drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++-- drivers/gpu/nova-core/mctp.rs | 3 + drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/vgpu.rs | 47 ++++++ rust/kernel/pci.rs | 12 ++ 16 files changed, 608 insertions(+), 132 deletions(-) create mode 100644 Documentation/gpu/nova/core/fsp.rst create mode 100644 drivers/gpu/nova-core/vgpu.rs
Signature · 1 line
2.51.0
PATCH 1/9 rust: pci: expose sriov_get_totalvfs() helper
Add a wrapper for the pci_sriov_get_totalvfs() helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.
--- rust/kernel/pci.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+)
rust/kernel/pci.rs +12 -0
@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {// SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`. Class ::from_raw (unsafe { (*self.as_raw ()).class }) } + +/// Returns total number of VFs, or `Err(ENODEV)` if none available. +pub fn sriov_get_totalvfs (&self) ->Result <i32 > { +// SAFETY: `self.as_raw()` is a valid pointer to a `struct pci_dev`. +let vfs =unsafe { bindings::pci_sriov_get_totalvfs (self.as_raw ()) }; + +if vfs ==0 { +return Err (ENODEV ); + } + +Ok (vfs) + } }impl <'a>Device <device::Core <'a>> {
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 1/9] rust: pci: expose sriov_get_totalvfs() helper
Quoted 16 lines · Zhi Wang
Add a wrapper for the
pci_sriov_get_totalvfs()helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.--- rust/kernel/pci.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+)rust/kernel/pci.rs +0
-0
rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 455
Quoted 6 lines
rust/kernel/pci.rs +3
-0@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {// SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`. Class ::from_raw (unsafe { (*self.as_raw ()).class }) } + +/// Returns total number of VFs, or `Err(ENODEV)` if none available. +pub fn sriov_get_totalvfs (&self) ->Result <i32 > {
I mentioned it in a previous review [1], but I think there is an opportunity to improve the C API (and by transition the Rust one) by making it return a u16, which is the type the number of total VFs is ultimately stored in anyway.
IIRC there is also no need to even update any caller, just changing the prototype of pci_sriov_get_totalvfs would be enough as callers ultimately compare the return value against u16s. So if anything it would make the C API more sound.
rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 461
Quoted 6 lines
rust/kernel/pci.rs +6
-0+// SAFETY: `self.as_raw()` is a valid pointer to a `struct pci_dev`. +let vfs =unsafe { bindings::pci_sriov_get_totalvfs (self.as_raw ()) }; + +if vfs ==0 { +return Err (ENODEV ); + }
Having 0 VFs does not necessarily look like an error - it's quite a valid answer to the question "how many VFs do we have?" which this method tries to answer. In patch 7 you even do unwrap_or(0) on the result of this method. So unless there is a good reason to treat this as an error, maybe we can just return a u16 here.
Re: [PATCH 1/9] rust: pci: expose sriov_get_totalvfs() helper
Quoted 45 lines · On Fri, 05 Jun 2026 23:08:38 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 21 lines · Zhi Wang
Add a wrapper for the
pci_sriov_get_totalvfs()helper, allowing drivers to query the number of total SR-IOV virtual functions a PCI device supports.--- rust/kernel/pci.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+)rust/kernel/pci.rs +2
-0@@ -450,6 +450,18 @@ pub fn pci_class(&self) -> Class {// SAFETY: `self.as_raw` is a valid pointer to a `struct pci_dev`. Class::from_raw(unsafe { (*self.as_raw()).class }) } + +/// Returns total number of VFs, or `Err(ENODEV)` if none Quoted 2 lines
available.
+ pub fnsriov_get_totalvfs(&self)-> Result<i32> {I mentioned it in a previous review [1], but I think there is an opportunity to improve the C API (and by transition the Rust one) by making it return a
u16, which is the type the number of total VFs is ultimately stored in anyway.IIRC there is also no need to even update any caller, just changing the prototype of
pci_sriov_get_totalvfswould be enough as callers ultimately compare the return value against u16s. So if anything it would make the C API more sound.[1] https://lore.kernel.org/all/[email protected]/
rust/kernel/pci.rs pub fn pci_class(&self) -> Class { · line 461
Quoted 7 lines
+ // SAFETY:
self.as_raw()is a valid pointer to a `struct
pci_dev`.
+ let vfs = unsafe {
bindings::pci_sriov_get_totalvfs(self.as_raw()) }; +rust/kernel/pci.rs +3
-0+if vfs ==0 { +return Err (ENODEV ); + }
Quoted 5 lines
Having 0 VFs does not necessarily look like an error - it's quite a valid answer to the question "how many VFs do we have?" which this method tries to answer. In patch 7 you even do
unwrap_or(0)on the result of this method. So unless there is a good reason to treat this as an error, maybe we can just return au16here.
I will add it in the next re-spin. Sorry that I missed it.
PATCH 2/9 gpu: nova-core: factor out common FSP message header
In reply to Zhi WangExtract common MCTP + NVDM headers into FspMessageHeader, rename FspMessage to FspCotMessage, and update FspResponse to use the shared header. This prepares for adding new FSP message types.
--- drivers/gpu/nova-core/fsp.rs | 56 +++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 16 deletions(-)
drivers/gpu/nova-core/fsp.rs +40 -16
@@ -57,12 +57,35 @@ struct NvdmPayloadCommandResponse { error_code: u32, } -/// Complete FSP response structure with MCTP and NVDM headers. +/// Common MCTP and NVDM headers shared by all FSP messages. #[repr( C , packed)] #[derive( Clone , Copy )] -struct FspResponse { +struct FspMessageHeader { mctp_header:MctpHeader , nvdm_header:NvdmHeader , +} + +// SAFETY: FspMessageHeader is a packed C struct with only integral fields. +unsafe impl AsBytes for FspMessageHeader {} + +// SAFETY: FspMessageHeader is a packed C struct with only integral fields. +unsafe impl FromBytes for FspMessageHeader {} + +impl FspMessageHeader { +/// Construct a standard FSP message header for the given NVDM type. +fn new (nvdm_type:NvdmType ) ->Self { +Self { + mctp_header:MctpHeader ::single_packet (), + nvdm_header:NvdmHeader ::new (nvdm_type), + } + } +} + +/// Complete FSP response structure with MCTP and NVDM headers. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspResponse { + header:FspMessageHeader , response:NvdmPayloadCommandResponse , }
@@ -94,17 +117,16 @@ struct NvdmPayloadCot { gsp_boot_args_sysmem_offset: u64, } -/// Complete FSP message structure with MCTP and NVDM headers. +/// Complete FSP COT (Chain of Trust) message structure. #[repr( C )] #[derive( Clone , Copy )] -struct FspMessage { - mctp_header:MctpHeader , - nvdm_header:NvdmHeader , +struct FspCotMessage { + header:FspMessageHeader , cot:NvdmPayloadCot , } -impl FspMessage { -/// Returns an in-place initializer for [`FspMessage`]. +impl FspCotMessage { +/// Returns an in-place initializer for [`FspCotMessage`]. fn new <'a>( fb_layout: &FbLayout , fsp_fw: &'a FspFirmware,
@@ -131,8 +153,7 @@ fn new<'a>(let size = num::usize_into_u16 ::<{ core::mem::size_of ::<NvdmPayloadCot >() }>(); Ok(init!(Self { - mctp_header: MctpHeader::single_packet(), - nvdm_header: NvdmHeader::new(NvdmType::Cot), + header: FspMessageHeader::new(NvdmType::Cot), // The payload is packed, so we cannot use `init!`. Initialize it member-by-member using // `chain`. cot <- pin_init::init_zeroed(),
@@ -153,11 +174,11 @@ fn new<'a>( } } -// SAFETY: `FspMessage` is `#[repr(C)]` with no padding, so all of its +// SAFETY: `FspCotMessage` is `#[repr(C)]` with no padding, so all of its // bytes are initialized. -unsafe impl AsBytes for FspMessage {} +unsafe impl AsBytes for FspCotMessage {} -impl MessageToFsp for FspMessage { +impl MessageToFsp for FspCotMessage {const NVDM_TYPE :NvdmType =NvdmType ::Cot ; }
@@ -251,8 +272,8 @@ fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) ->EIO })?; -let mctp_header = response.mctp_header; -let nvdm_header = response.nvdm_header; +let mctp_header = response.header.mctp_header; +let nvdm_header = response.header.nvdm_header;let command_nvdm_type = response.response.command_nvdm_type;let error_code = response.response.error_code;
@@ -310,7 +331,10 @@ pub(crate) fn boot_fmc( ) -> Result {dev_dbg! (dev,"Starting FSP boot sequence for {}\n" , args.chipset); -let msg =KBox ::init (FspMessage ::new (fb_layout, &self.fsp_fw, args)?,GFP_KERNEL )?; +let msg =KBox ::init ( +FspCotMessage ::new (fb_layout, &self.fsp_fw, args)?, +GFP_KERNEL , + )?; self.send_sync_fsp (dev, bar, &*msg)?;
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 2/9] gpu: nova-core: factor out common FSP message header
Quoted 5 lines · Zhi Wang
Extract common MCTP + NVDM headers into FspMessageHeader, rename FspMessage to FspCotMessage, and update FspResponse to use the shared header. This prepares for adding new FSP message types.
Signed-off-by Zhi Wang <[email protected]>
This patch is a good follow-up to the FSP series irrespective of vGPU support, so I'd like to merge it rapidly. Leaving it a couple of days for reviews but I think it makes sense to take it early and avoid rebasing churn.
PATCH 3/9 gpu: nova-core: return FSP response buffer to caller
In reply to Zhi WangChange send_sync_fsp() to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.
For the existing COT caller, the response buffer is unused.
--- drivers/gpu/nova-core/fsp.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/fsp.rs +4 -3
@@ -257,7 +257,8 @@ pub(crate) fn wait_secure_boot( }/// Sends a message to FSP and waits for the response. -fn send_sync_fsp <M >(&mut self, dev: &device::Device , bar:Bar0 <'_>, msg: &M ) ->Result +/// Returns the full response buffer on success. +fn send_sync_fsp <M >(&mut self, dev: &device::Device , bar:Bar0 <'_>, msg: &M ) ->Result <KVec <u8 >>where M :MessageToFsp , {
@@ -315,7 +316,7 @@ fn send_sync_fsp<M>(&mut self, dev: &device::Device, bar: Bar0<'_>, msg: &M) ->return Err (EIO ); } -Ok (()) +Ok (response_buf) }/// Boots GSP FMC via FSP Chain of Trust.
@@ -336,7 +337,7 @@ pub(crate) fn boot_fmc( GFP_KERNEL, )?; - self.send_sync_fsp (dev, bar, &*msg)?; +let _response_buf = self.send_sync_fsp (dev, bar, &*msg)?;dev_dbg! (dev,"FSP Chain of Trust completed successfully\n" );Ok (())
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller
Quoted 8 lines · Zhi Wang
Change
send_sync_fsp()to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.For the existing COT caller, the response buffer is unused.
Signed-off-by Zhi Wang <[email protected]>
Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.
Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller
Quoted 6 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 5 lines · Zhi Wang
Change
send_sync_fsp()to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.
Quoted 6 lines
Quoted 3 lines
For the existing COT caller, the response buffer is unused.
Signed-off-by Zhi Wang <[email protected]>Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.
Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.
Alexandre Courbot
maintainerRe: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller
Quoted 9 lines · Zhi Wang
Quoted 6 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 5 lines · Zhi Wang
Change
send_sync_fsp()to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
For the existing COT caller, the response buffer is unused.
Signed-off-by Zhi Wang <[email protected]>Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.
Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.
Patches 2 and 3 have been pushed to drm-rust-next!
Re: [PATCH 3/9] gpu: nova-core: return FSP response buffer to caller
Quoted 15 lines · On Tue, 09 Jun 2026 15:07:21 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 14 lines · Zhi Wang
Quoted 11 lines · On Fri, 05 Jun 2026 22:25:27 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 9 lines · Zhi Wang
Change
send_sync_fsp()to return the raw response buffer after validating the common MCTP/NVDM headers and error code. This allows callers to perform protocol-specific parsing on the response payload, which is needed for the upcoming PRC protocol support.For the existing COT caller, the response buffer is unused.
Signed-off-by Zhi Wang <[email protected]>
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Same as the previous patch, I intend to merge this quickly as it is a local follow-up to the FSP support.
Thanks. I will drop patch 2 and 3 in the next re-spin if you take them.
Patches 2 and 3 have been pushed to
drm-rust-next!
Thanks so much. :)
PATCH 4/9 gpu: nova-core: read vGPU mode from FSP via PRC protocol
In reply to Zhi WangAdd support for querying the vGPU mode configuration from FSP using the PRC (Product Reconfiguration Control) protocol. PRC is an API system exposed through FSP's Management Partition that allows querying device configuration "knobs" without firmware updates.
Add a VgpuMode enum that validates the raw PRC response value, returning an error for unexpected values.
--- drivers/gpu/nova-core/fsp.rs | 145 ++++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/mctp.rs | 3 + 2 files changed, 148 insertions(+)
drivers/gpu/nova-core/fsp.rs +145 -0
@@ -48,6 +48,44 @@mod hal; +/// PRC (Product Reconfiguration Control) protocol constants. +/// +/// PRC is an API system exposed through FSP's Management Partition that allows +/// querying and modifying device configuration "knobs" without firmware updates. +/// Each knob is identified by a unique object ID and controls a specific device +/// behavior (e.g., vGPU mode, ECC, confidential computing). +mod prc { +/// Sub-command to read a PRC knob value. +pub (super )const SUBCMD_READ :u8 =0x0c ; + +/// PRC object ID for vGPU mode configuration (knob ID 41). +pub (super )const OBJECT_VGPU_MODE :u8 =0x29 ; + +/// Request the active knob value (currently effective this boot). +pub (super )const FLAG_ACTIVE :u8 =1 <<1 ; +} + +/// vGPU operating mode as reported by FSP via the PRC protocol. +#[derive( Debug , Clone , Copy , PartialEq , Eq )] +pub (crate )enum VgpuMode { +/// vGPU support is disabled on this GPU. +Disabled =0 , +/// vGPU support is enabled on this GPU. +Enabled =1 , +} + +impl TryFrom <u16 >for VgpuMode { +type Error = kernel::error::Error ; + +fn try_from (value:u16 ) ->Result <Self > { +match value { +0 =>Ok (VgpuMode ::Disabled ), +1 =>Ok (VgpuMode ::Enabled ), + _ =>Err (EINVAL ), + } + } +} +/// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`). #[repr( C , packed)] #[derive( Clone , Copy )]
@@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse { error_code: u32, } +// SAFETY: NvdmPayloadCommandResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for NvdmPayloadCommandResponse {} + +/// PRC message payload. +/// +/// Sent to FSP to query or modify a device configuration knob. +/// The response includes the common FSP response header followed by +/// a [`NvdmPayloadPrcResponse`] with the knob's current state value. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct NvdmPayloadPrc { + sub_message_id:u8 , + flags:u8 , + object_id:u8 , + reserved:u8 , +} + +// SAFETY: NvdmPayloadPrc is a packed C struct with only integral fields. +unsafe impl AsBytes for NvdmPayloadPrc {} + +/// PRC response payload containing the knob state value. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct NvdmPayloadPrcResponse { + value_low:u8 , + value_high:u8 , + reserved1:u8 , + reserved2:u8 , +} + +// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for NvdmPayloadPrcResponse {} +/// Common MCTP and NVDM headers shared by all FSP messages. #[repr( C , packed)] #[derive( Clone , Copy )]
@@ -92,6 +163,18 @@ struct FspResponse {// SAFETY: FspResponse is a packed C struct with only integral fields. unsafe impl FromBytes for FspResponse {} +/// Complete FSP PRC response including the knob state payload. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcResponse { + header:FspMessageHeader , + response:NvdmPayloadCommandResponse , + prc_data:NvdmPayloadPrcResponse , +} + +// SAFETY: FspPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for FspPrcResponse {} +/// Trait implemented by types representing a message to send to FSP. /// /// This provides [`Fsp::send_sync_fsp`] with the information it needs to send
@@ -178,10 +261,25 @@ fn new<'a>(// bytes are initialized. unsafe impl AsBytes for FspCotMessage {} +/// Complete FSP PRC message. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcMessage { + header:FspMessageHeader , + prc:NvdmPayloadPrc , +} + +// SAFETY: FspPrcMessage is a packed C struct with only integral fields. +unsafe impl AsBytes for FspPrcMessage {} +impl MessageToFsp for FspCotMessage {const NVDM_TYPE :NvdmType =NvdmType ::Cot ; } +impl MessageToFsp for FspPrcMessage { +const NVDM_TYPE :NvdmType =NvdmType ::Prc ; +} +/// Bundled arguments for FMC boot via FSP Chain of Trust. pub (crate )struct FmcBootArgs { chipset:Chipset ,
@@ -226,6 +324,53 @@ pub(crate) struct Fsp { }impl Fsp { +/// Read vGPU mode from FSP using the PRC protocol. +/// +/// Queries FSP's Management Partition for the active vGPU mode knob value. +/// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU, +/// [`VgpuMode::Disabled`] otherwise. +#[expect(dead_code)] +pub (crate )fn read_vgpu_mode ( + &mut self, + dev: &device::Device <device::Bound >, + bar:Bar0 <'_>, + ) ->Result <VgpuMode > { +let msg =KBox ::new ( +FspPrcMessage { + header:FspMessageHeader ::new (NvdmType ::Prc ), + prc:NvdmPayloadPrc { + sub_message_id: prc::SUBCMD_READ , + flags: prc::FLAG_ACTIVE , + object_id: prc::OBJECT_VGPU_MODE , + reserved:0 , + }, + }, +GFP_KERNEL , + )?; + +let response_buf = self.send_sync_fsp (dev, bar, &*msg)?; + +let prc_resp_size = core::mem::size_of ::<FspPrcResponse >(); +if response_buf.len () < prc_resp_size { +dev_err! ( + dev, +"PRC response too small: {} bytes (expected {})\n" , + response_buf.len (), + prc_resp_size + ); +return Err (EIO ); + } + +let (prc_response, _) =FspPrcResponse ::from_bytes_prefix (&response_buf[..]).ok_or (EIO )?; + +let raw_value = u16::from (prc_response.prc_data.value_low) + | (u16::from (prc_response.prc_data.value_high) <<8 ); + +VgpuMode ::try_from (raw_value).inspect_err (|_| { +dev_err! (dev,"unexpected vGPU mode value: {:#x}\n" , raw_value); + }) + } +/// Waits for FSP secure boot completion, then returns the [`Fsp`] interface. /// /// Polls the thermal scratch register until FSP signals boot completion or the timeout
drivers/gpu/nova-core/mctp.rs +3 -0
@@ -13,6 +13,8 @@#[derive( Debug , Clone , Copy , Default , PartialEq , Eq )] #[repr( u8 )] pub (crate )enum NvdmType { +/// PRC (Product Reconfiguration Control) message. +Prc =0x13 ,#[default] /// Chain of Trust boot message. Cot =0x14 ,
@@ -25,6 +27,7 @@ impl TryFrom<u8> for NvdmType {fn try_from (value:u8 ) ->Result <Self ,Self ::Error > {match value { + xif x == u8::from (Self ::Prc ) =>Ok (Self ::Prc ), xif x == u8::from (Self ::Cot ) =>Ok (Self ::Cot ), xif x == u8::from (Self ::FspResponse ) =>Ok (Self ::FspResponse ), _ =>Err (value),
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 4/9] gpu: nova-core: read vGPU mode from FSP via PRC protocol
Quoted 28 lines · Zhi Wang
Add support for querying the vGPU mode configuration from FSP using the PRC (Product Reconfiguration Control) protocol. PRC is an API system exposed through FSP's Management Partition that allows querying device configuration "knobs" without firmware updates.
Add a VgpuMode enum that validates the raw PRC response value, returning an error for unexpected values.
Signed-off-by Zhi Wang <[email protected]>--- drivers/gpu/nova-core/fsp.rs | 145 ++++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/mctp.rs | 3 + 2 files changed, 148 insertions(+)drivers/gpu/nova-core/fsp.rs +6
-0@@ -48,6 +48,44 @@mod hal; +/// PRC (Product Reconfiguration Control) protocol constants. +/// +/// PRC is an API system exposed through FSP's Management Partition that allows +/// querying and modifying device configuration "knobs" without firmware updates. +/// Each knob is identified by a unique object ID and controls a specific device +/// behavior (e.g., vGPU mode, ECC, confidential computing).
drivers/gpu/nova-core/fsp.rs line 66
Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10
-0+mod prc { +/// Sub-command to read a PRC knob value. +pub (super )const SUBCMD_READ :u8 =0x0c ; + +/// PRC object ID for vGPU mode configuration (knob ID 41). +pub (super )const OBJECT_VGPU_MODE :u8 =0x29 ; + +/// Request the active knob value (currently effective this boot). +pub (super )const FLAG_ACTIVE :u8 =1 <<1 ; +}
These 3 should be defined as their own enum types, to make sure we cannot mix them. For instance:
enum PrcMessageSubcmd {
Read = 0x0c,
}
enum PrcObjectId {
VgpuMode = 0x29,
}
(for the flags I guess you will want to use `kernel::impl_flags!`?)Then, NvdmPayloadPrc can have a constructor that takes these 3 as parameters and returns the constructed value - that way no risk of, say, using the subcmd as an object identifier.
Btw, is there a public source for these values?
I like the idea of using a prc sub-module - to the point where I'd suggest moving all PRC-related types to it, and all CoT messages to a cot sub-module. But doing it in this series would distract from the goal, and the fsp module is not that large, so let's keep everything in it for now and do this as a follow-up.
Quoted 11 lines
drivers/gpu/nova-core/fsp.rs +11
-0+ +/// vGPU operating mode as reported by FSP via the PRC protocol. +#[derive( Debug , Clone , Copy , PartialEq , Eq )] +pub (crate )enum VgpuMode { +/// vGPU support is disabled on this GPU. +Disabled =0 , +/// vGPU support is enabled on this GPU. +Enabled =1 , +} + +impl TryFrom <u16 >for VgpuMode {
drivers/gpu/nova-core/fsp.rs line 87
Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10
-0+type Error = kernel::error::Error ; + +fn try_from (value:u16 ) ->Result <Self > { +match value { +0 =>Ok (VgpuMode ::Disabled ), +1 =>Ok (VgpuMode ::Enabled ), + _ =>Err (EINVAL ), + } + } +}
You can even do TryFrom<NvdmPayloadPrcResponse> for VgpuMode for a better fit.
Quoted 45 lines
drivers/gpu/nova-core/fsp.rs +34
-0+/// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`). #[repr( C , packed)] #[derive( Clone , Copy )] @@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse { error_code: u32, } +// SAFETY: NvdmPayloadCommandResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for NvdmPayloadCommandResponse {} + +/// PRC message payload. +/// +/// Sent to FSP to query or modify a device configuration knob. +/// The response includes the common FSP response header followed by +/// a [`NvdmPayloadPrcResponse`] with the knob's current state value. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct NvdmPayloadPrc { + sub_message_id:u8 , + flags:u8 , + object_id:u8 , + reserved:u8 , +} + +// SAFETY: NvdmPayloadPrc is a packed C struct with only integral fields. +unsafe impl AsBytes for NvdmPayloadPrc {} + +/// PRC response payload containing the knob state value. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct NvdmPayloadPrcResponse { + value_low:u8 , + value_high:u8 , + reserved1:u8 , + reserved2:u8 , +} + +// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for NvdmPayloadPrcResponse {} +/// Common MCTP and NVDM headers shared by all FSP messages. #[repr( C , packed)] #[derive( Clone , Copy )]
drivers/gpu/nova-core/fsp.rs struct FspResponse { · line 171
Quoted 9 lines
drivers/gpu/nova-core/fsp.rs +6
-0@@ -92,6 +163,18 @@ struct FspResponse {// SAFETY: FspResponse is a packed C struct with only integral fields. unsafe impl FromBytes for FspResponse {} +/// Complete FSP PRC response including the knob state payload. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcResponse { + header:FspMessageHeader , + response:NvdmPayloadCommandResponse ,
Since you introduced FspMessageHeader to factor out mctp_header and nvdm_header, can you do the same for the reponse? I guess in our case this is as simple as renaming FspResponse to FspResponseHeader and using it in FspPrcResponse.
Quoted 65 lines
drivers/gpu/nova-core/fsp.rs +47
-0+ prc_data: NvdmPayloadPrcResponse, +} + +// SAFETY: FspPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for FspPrcResponse {} +/// Trait implemented by types representing a message to send to FSP. /// /// This provides [`Fsp::send_sync_fsp`] with the information it needs to send @@ -178,10 +261,25 @@ fn new<'a>(// bytes are initialized. unsafe impl AsBytes for FspCotMessage {} +/// Complete FSP PRC message. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcMessage { + header:FspMessageHeader , + prc:NvdmPayloadPrc , +} + +// SAFETY: FspPrcMessage is a packed C struct with only integral fields. +unsafe impl AsBytes for FspPrcMessage {} +impl MessageToFsp for FspCotMessage {const NVDM_TYPE :NvdmType =NvdmType ::Cot ; } +impl MessageToFsp for FspPrcMessage { +const NVDM_TYPE :NvdmType =NvdmType ::Prc ; +} +/// Bundled arguments for FMC boot via FSP Chain of Trust. pub (crate )struct FmcBootArgs { chipset:Chipset ,@@ -226,6 +324,53 @@ pub(crate) struct Fsp { }impl Fsp { +/// Read vGPU mode from FSP using the PRC protocol. +/// +/// Queries FSP's Management Partition for the active vGPU mode knob value. +/// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU, +/// [`VgpuMode::Disabled`] otherwise. +#[expect(dead_code)] +pub (crate )fn read_vgpu_mode ( + &mut self, + dev: &device::Device <device::Bound >, + bar:Bar0 <'_>, + ) ->Result <VgpuMode > { +let msg =KBox ::new ( +FspPrcMessage { + header:FspMessageHeader ::new (NvdmType ::Prc ), + prc:NvdmPayloadPrc { + sub_message_id: prc::SUBCMD_READ , + flags: prc::FLAG_ACTIVE , + object_id: prc::OBJECT_VGPU_MODE , + reserved:0 , + }, + }, +GFP_KERNEL , + )?; + +let response_buf = self.send_sync_fsp (dev, bar, &*msg)?; +
drivers/gpu/nova-core/fsp.rs pub(crate) struct Fsp { · line 362
Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10
-0+let prc_resp_size = core::mem::size_of ::<FspPrcResponse >(); +if response_buf.len () < prc_resp_size { +dev_err! ( + dev, +"PRC response too small: {} bytes (expected {})\n" , + response_buf.len (), + prc_resp_size + ); +return Err (EIO ); + }
IIUC from_bytes_prefix takes care of checking the size, so this check looks redundant.
Re: [PATCH 4/9] gpu: nova-core: read vGPU mode from FSP via PRC protocol
Quoted 1 line · On Tue, 16 Jun 2026 17:35:27 +0900 "Alexandre Courbot" <[email protected]>…
On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:
snip
Quoted 14 lines
Quoted 1 line
+}
These 3 should be defined as their own enum types, to make sure we cannot mix them. For instance:
enum PrcMessageSubcmd { Read = 0x0c, } enum PrcObjectId { VgpuMode = 0x29, } (for the flags I guess you will want to use
Quoted 6 lines
kernel::impl_flags!?)Then,
NvdmPayloadPrccan have a constructor that takes these 3 as parameters and returns the constructed value - that way no risk of, say, using the subcmd as an object identifier.
Great idea. Actually, this is what I am looking for - how to organized those values.
Quoted 1 line
Btw, is there a public source for these values?
Production reconfiguration knobs are public knowledge where folks can find from [1]. vGPU mode is a newly added one.
[1] https://github.com/NVIDIA/gpu-admin-tools
Quoted 7 lines
I like the idea of using a
prcsub-module - to the point where I'd suggest moving all PRC-related types to it, and all CoT messages to acotsub-module. But doing it in this series would distract from the goal, and thefspmodule is not that large, so let's keep everything in it for now and do this as a follow-up.
I see. I can send another series to address above after this one is done.
Quoted 165 lines
Quoted 11 lines
drivers/gpu/nova-core/fsp.rs +11
-0+ +/// vGPU operating mode as reported by FSP via the PRC protocol. +#[derive( Debug , Clone , Copy , PartialEq , Eq )] +pub (crate )enum VgpuMode { +/// vGPU support is disabled on this GPU. +Disabled =0 , +/// vGPU support is enabled on this GPU. +Enabled =1 , +} + +impl TryFrom <u16 >for VgpuMode {drivers/gpu/nova-core/fsp.rs line 87
Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10
-0+type Error = kernel::error::Error ; + +fn try_from (value:u16 ) ->Result <Self > { +match value { +0 =>Ok (VgpuMode ::Disabled ), +1 =>Ok (VgpuMode ::Enabled ), + _ =>Err (EINVAL ), + } + } +}You can even do
TryFrom<NvdmPayloadPrcResponse> for VgpuModefor a better fit.Quoted 45 lines
drivers/gpu/nova-core/fsp.rs +2
-0+/// FSP command response payload (`NVDM_PAYLOAD_COMMAND_RESPONSE`). #[repr( C , packed)] #[derive( Clone , Copy )] @@ -57,6 +95,39 @@ struct NvdmPayloadCommandResponse { error_code: u32, } +// SAFETY: NvdmPayloadCommandResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for
NvdmPayloadCommandResponse {} +drivers/gpu/nova-core/fsp.rs +4
-0+/// PRC message payload. +/// +/// Sent to FSP to query or modify a device configuration knob. +/// The response includes the common FSP response header followed by +/// a [
NvdmPayloadPrcResponse] with the knob's current state value. +#[repr(C, packed)]drivers/gpu/nova-core/fsp.rs +9
-0+#[derive( Clone , Copy )] +struct NvdmPayloadPrc { + sub_message_id:u8 , + flags:u8 , + object_id:u8 , + reserved:u8 , +} + +// SAFETY: NvdmPayloadPrc is a packed C struct with only integral fields. +unsafe impl AsBytes for NvdmPayloadPrc {}
drivers/gpu/nova-core/fsp.rs +12
-0+ +/// PRC response payload containing the knob state value. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct NvdmPayloadPrcResponse { + value_low:u8 , + value_high:u8 , + reserved1:u8 , + reserved2:u8 , +} + +// SAFETY: NvdmPayloadPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for NvdmPayloadPrcResponse {} +
/// Common MCTP and NVDM headers shared by all FSP messages. #[repr(C, packed)]
#[derive(Clone, Copy)]drivers/gpu/nova-core/fsp.rs struct FspResponse { · line 171
Quoted 9 lines
drivers/gpu/nova-core/fsp.rs +6
-0@@ -92,6 +163,18 @@ struct FspResponse {// SAFETY: FspResponse is a packed C struct with only integral fields. unsafe impl FromBytes for FspResponse {} +/// Complete FSP PRC response including the knob state payload. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcResponse { + header:FspMessageHeader , + response:NvdmPayloadCommandResponse ,Since you introduced
FspMessageHeaderto factor outmctp_headerandnvdm_header, can you do the same for the reponse? I guess in our case this is as simple as renamingFspResponsetoFspResponseHeaderand using it inFspPrcResponse.Quoted 67 lines
drivers/gpu/nova-core/fsp.rs +5
-0+ prc_data: NvdmPayloadPrcResponse, +} + +// SAFETY: FspPrcResponse is a packed C struct with only integral fields. +unsafe impl FromBytes for FspPrcResponse {} +/// Trait implemented by types representing a message to send to FSP. ///
/// This provides [Fsp::send_sync_fsp] with the information it
needs to send @@ -178,10 +261,25 @@ fn new<'a>(
// bytes are initialized.
unsafe impl AsBytes for FspCotMessage {}drivers/gpu/nova-core/fsp.rs +9
-0+/// Complete FSP PRC message. +#[repr( C , packed)] +#[derive( Clone , Copy )] +struct FspPrcMessage { + header:FspMessageHeader , + prc:NvdmPayloadPrc , +} + +// SAFETY: FspPrcMessage is a packed C struct with only integral fields. +unsafe impl AsBytes for FspPrcMessage {}
drivers/gpu/nova-core/fsp.rs +8
-0+impl MessageToFsp for FspCotMessage {const NVDM_TYPE :NvdmType =NvdmType ::Cot ; } +impl MessageToFsp for FspPrcMessage { +const NVDM_TYPE :NvdmType =NvdmType ::Prc ; +} +/// Bundled arguments for FMC boot via FSP Chain of Trust. pub (crate )struct FmcBootArgs { chipset:Chipset ,@@ -226,6 +324,53 @@ pub(crate) struct Fsp { }impl Fsp { +/// Read vGPU mode from FSP using the PRC protocol. +/// +/// Queries FSP's Management Partition for the active vGPU mode knob value.
+ /// Returns [VgpuMode::Enabled] if vGPU support is active on
this GPU,drivers/gpu/nova-core/fsp.rs +22
-0+/// [`VgpuMode::Disabled`] otherwise. +#[expect(dead_code)] +pub (crate )fn read_vgpu_mode ( + &mut self, + dev: &device::Device <device::Bound >, + bar:Bar0 <'_>, + ) ->Result <VgpuMode > { +let msg =KBox ::new ( +FspPrcMessage { + header:FspMessageHeader ::new (NvdmType ::Prc ), + prc:NvdmPayloadPrc { + sub_message_id: prc::SUBCMD_READ , + flags: prc::FLAG_ACTIVE , + object_id: prc::OBJECT_VGPU_MODE , + reserved:0 , + }, + }, +GFP_KERNEL , + )?; + +let response_buf = self.send_sync_fsp (dev, bar, &*msg)?; +drivers/gpu/nova-core/fsp.rs pub(crate) struct Fsp { · line 362
Quoted 10 lines
drivers/gpu/nova-core/fsp.rs +10
-0+let prc_resp_size = core::mem::size_of ::<FspPrcResponse >(); +if response_buf.len () < prc_resp_size { +dev_err! ( + dev, +"PRC response too small: {} bytes (expected {})\n" , + response_buf.len (), + prc_resp_size + ); +return Err (EIO ); + }IIUC
from_bytes_prefixtakes care of checking the size, so this check looks redundant.
PATCH 5/9 gpu: nova-core: add FSP and PRC protocol documentation
In reply to Zhi WangAdd documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.
--- Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 143 insertions(+) create mode 100644 Documentation/gpu/nova/core/fsp.rst
Documentation/gpu/nova/core/fsp.rst new file +142 -0
@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +FSP (Foundation Security Processor) and Secure Boot +=================================================== +This document describes the role of the FSP in the GPU boot sequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. + +What is FSP? +============ +The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. + +FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver. + +Simplified boot flow (Hopper/Blackwell) +======================================= +Starting with Hopper, the boot flow is significantly simplified compared to +earlier GPU generations like Ampere. + +On an **Ampere** GPU, the boot verification chain involves multiple Falcon +engines and multiple ucode stages (see falcon.rst for details):: + + Hardware BROM (SEC2) + -> HS Booter (SEC2) + -> LS GSP-RM (GSP) + +The driver must extract ucode from VBIOS, manage SEC2 and GSP, and +orchestrate the Booter to load GSP-RM. This involves FWSEC-FRTS, devinit, +and the Booter stages. + +On **Hopper/Blackwell** GPUs, FSP replaces this multi-stage process with a +single message-driven interface:: + + FSP (hardware root of trust, boots from ROM) + -> FMC (Falcon Microcontroller, verified by FSP) + -> GSP-RM (verified and loaded by FMC) + +The driver only needs to: + +1. Wait for FSP to complete its own secure boot (polling a scratch register). +2. Send a Chain of Trust (COT) message to FSP with the FMC firmware location, + cryptographic signatures, and GSP boot parameters. +3. FSP authenticates the FMC firmware and boots it, FMC in turn loads GSP-RM. + +There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage. The +entire secure boot is driven by a single FSP message exchange. + +Chain of Trust (COT) protocol +============================= +The Chain of Trust establishes a cryptographically enforced boot sequence, +ensuring the GPU reaches a known, trusted state. + +The driver communicates with FSP using a message queue (Falcon MSGQ +interface). Each message consists of an MCTP (Management Component Transport +Protocol) transport header and an NVDM (NVIDIA Vendor Defined Message) header, +followed by a protocol-specific payload. + +For Chain of Trust, the payload includes: + +- The system memory address of the FMC firmware image. +- Cryptographic material: a SHA-384 hash, RSA-3K public key, and RSA-3K + signature extracted from the FMC ELF firmware. +- FRTS (Firmware Runtime Services) region information (vidmem offset and size). +- The system memory address of the GSP boot arguments structure. + +FSP verifies the signature against the provided public key and hash, and if +verification succeeds, boots the FMC. The FMC then authenticates and launches +GSP-RM. + +The message flow is:: + + nova-core FSP + | | + | 1. Poll scratch register | + | (wait for FSP boot complete) | + | | + | 2. COT message ------------> | + | (FMC addr, signatures, | + | boot params) | + | | + | |--- Verify FMC signature + | |--- Boot FMC + | |--- FMC loads GSP-RM + | | + | 3. COT response <------------ | + | (success/error) | + | | + +FSP message format +================== +All FSP messages share a common header format consisting of two 32-bit words: + +**MCTP header** (Management Component Transport Protocol): + +- Bit 31: SOM (Start of Message) +- Bit 30: EOM (End of Message) +- Bits 29:28: Packet sequence number +- Bits 23:16: Source Endpoint ID + +**NVDM header** (NVIDIA Vendor Defined Message): + +- Bits 6:0: MCTP message type (0x7e = vendor-defined PCI) +- Bits 23:8: PCI vendor ID (0x10de = NVIDIA) +- Bits 31:24: NVDM type (0x14 = COT, 0x13 = PRC, 0x15 = FSP response) + +PRC (Product Reconfiguration Control) protocol +=============================================== +PRC is an API system exposed through FSP's Management Partition that allows +querying and modifying device configuration without firmware updates. + +Configuration parameters are called "knobs". Each knob has a unique object +ID and controls a specific device behavior. Examples include vGPU mode, ECC +enable, confidential computing mode, and NVLINK configuration. + +Each knob has two values: + +- **Active**: the currently effective value for this boot cycle. +- **Persistent**: the value stored in InfoROM, applied on subsequent boots. + +The nova-core driver uses PRC to read the vGPU mode knob (object ID 0x29) +during early boot, before firmware loading, to determine whether the GPU +should operate in vGPU mode. + +The PRC message format follows the same MCTP/NVDM header structure as COT, +with NVDM type 0x13. The payload contains: + +- A sub-command (e.g., 0x0c for read). +- Flags indicating which value to read (bit 0 = persistent, bit 1 = active). +- The knob object ID. + +The response includes the common FSP response header (with error status) +followed by the knob's 16-bit state value.
Documentation/gpu/nova/index.rst +1 -0
@@ -30,5 +30,6 @@ vGPU manager VFIO driver and the nova-drm driver. core/todo core/vbios core/devinit + core/fsp core/fwsec core/falcon
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation
Quoted 7 lines · Zhi Wang
Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.
Signed-off-by Zhi Wang <[email protected]>
This is super useful, thanks! One question below:
Quoted 35 lines
--- Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 143 insertions(+) create mode 100644 Documentation/gpu/nova/core/fsp.rstDocumentation/gpu/nova/core/fsp.rst new file +23
-0@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +FSP (Foundation Security Processor) and Secure Boot +=================================================== +This document describes the role of the FSP in the GPU boot sequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. + +What is FSP? +============ +The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. + +FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver.
Documentation/gpu/nova/core/fsp.rst line 33
Quoted 10 lines
Documentation/gpu/nova/core/fsp.rst new file +10
-0+ +Simplified boot flow (Hopper/Blackwell) +======================================= +Starting with Hopper, the boot flow is significantly simplified compared to +earlier GPU generations like Ampere. + +On an **Ampere** GPU, the boot verification chain involves multiple Falcon +engines and multiple ucode stages (see falcon.rst for details):: + + Hardware BROM (SEC2)
Is this referring to FWSEC-FRTS? If so, should this be GSP instead of SEC2?
Re: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation
Quoted 51 lines · On Tue, 16 Jun 2026 17:17:34 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 7 lines · Zhi Wang
Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.
Signed-off-by Zhi Wang <[email protected]>This is super useful, thanks! One question below:
Quoted 40 lines
--- Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 143 insertions(+) create mode 100644 Documentation/gpu/nova/core/fsp.rst(quoted diff) +0
-0
b/Documentation/gpu/nova/core/fsp.rstnew file mode 100644 index 000000000000..52d618d22bb8Documentation/gpu/nova/core/fsp.rst new file +6
-0@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +FSP (Foundation Security Processor) and Secure Boot +=================================================== +This document describes the role of the FSP in the GPU bootsequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. +
Documentation/gpu/nova/core/fsp.rst new file +3
-0+What is FSP? +============ +The Foundation Security Processor (FSP) is the GPU's Internal Rootof Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. +
+FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver.Documentation/gpu/nova/core/fsp.rst new file +4
-0+ +Simplified boot flow (Hopper/Blackwell) +======================================= +Starting with Hopper, the boot flow is significantly simplifiedcompared to +earlier GPU generations like Ampere.
Quoted 8 lines
Quoted 5 lines
Documentation/gpu/nova/core/fsp.rst new file +2
-0+ +On an **Ampere** GPU, the boot verification chain involvesmultiple Falcon +engines and multiple ucode stages (see
falcon.rstfor details):: +
+ Hardware BROM (SEC2)Is this referring to FWSEC-FRTS? If so, should this be
GSPinstead ofSEC2?
Hi Alex:
This refers to the BootROM in the SEC2 Peregrine. From the timeline perspective, FWSEC-FRTS should be positioned as below. I was thinking to show the brief differences of relationship between internal processors so I omit the FWSEC-FRTS in the sequence. Do you think we should add it?
+ Hardware BROM (SEC2)
here < ----- FWSEC-FRTS (GSP)
+ -> HS Booter (SEC2)
+ -> LS GSP-RM (GSP)
Z.
Alexandre Courbot
maintainerRe: [PATCH 5/9] gpu: nova-core: add FSP and PRC protocol documentation
Quoted 70 lines · Zhi Wang
Quoted 51 lines · On Tue, 16 Jun 2026 17:17:34 +0900 "Alexandre Courbot" <[email protected]>…
Quoted 7 lines · Zhi Wang
Add documentation for the Foundation Security Processor (FSP) interface covering the simplified Hopper/Blackwell boot flow, the Chain of Trust (COT) message protocol, the MCTP/NVDM message format, and the Product Reconfiguration Control (PRC) protocol used to query device configuration knobs such as vGPU mode.
Signed-off-by Zhi Wang <[email protected]>This is super useful, thanks! One question below:
Quoted 40 lines
--- Documentation/gpu/nova/core/fsp.rst | 142 ++++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 143 insertions(+) create mode 100644 Documentation/gpu/nova/core/fsp.rst(quoted diff) +0
-0
b/Documentation/gpu/nova/core/fsp.rstnew file mode 100644 index 000000000000..52d618d22bb8Documentation/gpu/nova/core/fsp.rst new file +6
-0@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +FSP (Foundation Security Processor) and Secure Boot +=================================================== +This document describes the role of the FSP in the GPU bootsequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. +
Documentation/gpu/nova/core/fsp.rst new file +3
-0+What is FSP? +============ +The Foundation Security Processor (FSP) is the GPU's Internal Rootof Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. +
+FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver.Documentation/gpu/nova/core/fsp.rst new file +4
-0+ +Simplified boot flow (Hopper/Blackwell) +======================================= +Starting with Hopper, the boot flow is significantly simplifiedcompared to +earlier GPU generations like Ampere.
Quoted 8 lines
Quoted 5 lines
Documentation/gpu/nova/core/fsp.rst new file +2
-0+ +On an **Ampere** GPU, the boot verification chain involvesmultiple Falcon +engines and multiple ucode stages (see
falcon.rstfor details):: +
+ Hardware BROM (SEC2)Is this referring to FWSEC-FRTS? If so, should this be
GSPinstead ofSEC2?Hi Alex:
This refers to the BootROM in the SEC2 Peregrine. From the timeline perspective, FWSEC-FRTS should be positioned as below. I was thinking to show the brief differences of relationship between internal processors so I omit the FWSEC-FRTS in the sequence. Do you think we should add it?
Documentation/gpu/nova/core/fsp.rst line 35
Quoted 4 lines
Documentation/gpu/nova/core/fsp.rst new file +3
-0+ Hardware BROM (SEC2) here < ----- FWSEC-FRTS (GSP) + -> HS Booter (SEC2) + -> LS GSP-RM (GSP)
Ah, I understand now - the boot ROM that verifies the HS Booter. No, what you wrote makes perfect sense, I just misunderstood.
Pushed as-is to drm-rust-next! Thanks for taking the time to put this together.
PATCH 6/9 gpu: nova-core: consolidate GSP boot parameters into GspBootContext
In reply to Zhi WangThe GspHal trait methods boot() and post_boot() accept a long list of individual parameters (dev, bar, chipset, gsp_falcon, sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.
Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a dev() helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.
Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in Gsp::boot() accordingly.
--- drivers/gpu/nova-core/gpu.rs | 14 ++++++- drivers/gpu/nova-core/gsp.rs | 22 +++++++++++ drivers/gpu/nova-core/gsp/boot.rs | 55 ++++++++++++-------------- drivers/gpu/nova-core/gsp/hal.rs | 23 +++-------- drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++--- drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++--------- 6 files changed, 85 insertions(+), 74 deletions(-)
drivers/gpu/nova-core/gpu.rs +12 -2
@@ -23,7 +23,8 @@ fb::SysmemFlush, gsp::{ self, - Gsp, // + Gsp, + GspBootContext, // }, regs, };
@@ -323,7 +324,16 @@ pub(crate) fn new(// This member must be initialized last, so the `UnloadBundle` can never be dropped from // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run // in case of failure. - unload_bundle: gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)?, + unload_bundle: { + let ctx = GspBootContext { + pdev, + bar, + chipset: spec.chipset, + gsp_falcon, + sec2_falcon, + }; + gsp.boot (&ctx)? + }, bar, }) }
drivers/gpu/nova-core/gsp.rs +22 -0
@@ -31,6 +31,13 @@ };use crate ::{ + driver::Bar0 , + falcon::{ + gsp::Gsp as GspFalcon , + sec2::Sec2 as Sec2Falcon , +Falcon ,// + }, + gpu::Chipset , gsp::cmdq::Cmdq , gsp::fw::{GspArgumentsPadded ,
@@ -42,6 +49,21 @@pub (crate )const GSP_PAGE_SHIFT :usize =12 ;pub (crate )const GSP_PAGE_SIZE :usize =1 <<GSP_PAGE_SHIFT ; +/// Common context for the GSP boot process. +pub (crate )struct GspBootContext <'a> { +pub (crate ) pdev: &'a pci::Device <device::Bound >, +pub (crate ) bar:Bar0 <'a>, +pub (crate ) chipset:Chipset , +pub (crate ) gsp_falcon: &'aFalcon <GspFalcon >, +pub (crate ) sec2_falcon: &'aFalcon <Sec2Falcon >, +} + +impl <'a>GspBootContext <'a> { +pub (crate )fn dev (&self) -> &'a device::Device <device::Bound > { + self.pdev.as_ref () + } +} +/// Number of GSP pages to use in a RM log buffer. const RM_LOG_BUFFER_NUM_PAGES :usize =0x10 ;const LOG_BUFFER_SIZE :usize =RM_LOG_BUFFER_NUM_PAGES *GSP_PAGE_SIZE ;
drivers/gpu/nova-core/gsp/boot.rs +25 -30
100 lines not on this page · raw message
drivers/gpu/nova-core/gsp/hal.rs +5 -18
@@ -4,11 +4,10 @@mod gh100;mod tu102; -use kernel::prelude::*; -use kernel::{ device, - dma::Coherent ,// + dma::Coherent , + prelude::*,// };use crate ::{
@@ -27,6 +26,7 @@ gsp::{ boot::BootUnloadGuard, Gsp, + GspBootContext, GspFwWprMeta, // }, };
@@ -53,32 +53,19 @@ pub(super) trait GspHal: Send {/// /// Upon success, returns a guard that runs the GSP unload sequence if GSP boot does not /// complete. -#[allow(clippy::too_many_arguments)] fn boot <'a>( &self, gsp: &'aGsp , - dev: &'a device::Device <device::Bound >, - bar:Bar0 <'a>, - chipset:Chipset , + ctx: &GspBootContext <'a>, fb_layout: &FbLayout , wpr_meta: &Coherent <GspFwWprMeta >, - gsp_falcon: &'aFalcon <GspEngine >, - sec2_falcon: &'aFalcon <Sec2 >, ) ->Result <BootUnloadGuard <'a>>;/// Performs HAL-specific post-GSP boot tasks. /// /// This method is called by the GSP boot code after the GSP is confirmed to be running, and /// after the initialization commands have been pushed onto its queue. -fn post_boot ( - &self, - _gsp: &Gsp , - _dev: &device::Device <device::Bound >, - _bar:Bar0 <'_>, - _gsp_fw: &GspFirmware , - _gsp_falcon: &Falcon <GspEngine >, - _sec2_falcon: &Falcon <Sec2 >, - ) ->Result { +fn post_boot (&self, _gsp: &Gsp , _ctx: &GspBootContext <'_>, _gsp_fw: &GspFirmware ) ->Result {Ok (()) } }
drivers/gpu/nova-core/gsp/hal/gh100.rs +8 -6
@@ -26,7 +26,6 @@ FmcBootArgs, Fsp, // }, - gpu::Chipset, gsp::{ boot::BootUnloadGuard, hal::{
@@ -34,6 +33,7 @@ UnloadBundle, // }, Gsp, + GspBootContext, GspFwWprMeta, // }, };
@@ -152,14 +152,16 @@ impl GspHal for Gh100 {fn boot <'a>( &self, gsp: &'aGsp , - dev: &'a device::Device <device::Bound >, - bar:Bar0 <'a>, - chipset:Chipset , + ctx: &GspBootContext <'a>, fb_layout: &FbLayout , wpr_meta: &Coherent <GspFwWprMeta >, - gsp_falcon: &'aFalcon <GspEngine >, - sec2_falcon: &'aFalcon <Sec2 >, ) ->Result <BootUnloadGuard <'a>> { +let dev = ctx.dev (); +let bar = ctx.bar; +let chipset = ctx.chipset; +let gsp_falcon = ctx.gsp_falcon; +let sec2_falcon = ctx.sec2_falcon; +let fsp_fw =FspFirmware ::new (dev, chipset,FIRMWARE_VERSION )?; let unload_bundle = crate::gsp::UnloadBundle(
drivers/gpu/nova-core/gsp/hal/tu102.rs +13 -18
@@ -42,6 +42,7 @@ GspSequencerParams, // }, Gsp, + GspBootContext, GspFwWprMeta, // }, regs,
@@ -258,14 +259,16 @@ impl GspHal for Tu102 {fn boot <'a>( &self, gsp: &'aGsp , - dev: &'a device::Device <device::Bound >, - bar:Bar0 <'a>, - chipset:Chipset , + ctx: &GspBootContext <'a>, fb_layout: &FbLayout , wpr_meta: &Coherent <GspFwWprMeta >, - gsp_falcon: &'aFalcon <GspEngine >, - sec2_falcon: &'aFalcon <Sec2 >, ) ->Result <BootUnloadGuard <'a>> { +let dev = ctx.dev (); +let bar = ctx.bar; +let chipset = ctx.chipset; +let gsp_falcon = ctx.gsp_falcon; +let sec2_falcon = ctx.sec2_falcon; +let bios =Vbios ::new (dev, bar)?;// Try and prepare the unload bundle.
@@ -321,23 +324,15 @@ fn boot<'a>(Ok (unload_guard) } -fn post_boot ( - &self, - gsp: &Gsp , - dev: &device::Device <device::Bound >, - bar:Bar0 <'_>, - gsp_fw: &GspFirmware , - gsp_falcon: &Falcon <GspEngine >, - sec2_falcon: &Falcon <Sec2 >, - ) ->Result { +fn post_boot (&self, gsp: &Gsp , ctx: &GspBootContext <'_>, gsp_fw: &GspFirmware ) ->Result {// Create and run the GSP sequencer. let seq_params =GspSequencerParams { bootloader_app_version: gsp_fw.bootloader.app_version, libos_dma_handle: gsp.libos.dma_handle (), - gsp_falcon, - sec2_falcon, - dev, - bar, + gsp_falcon: ctx.gsp_falcon, + sec2_falcon: ctx.sec2_falcon, + dev: ctx.dev (), + bar: ctx.bar, };GspSequencer ::run (&gsp.cmdq, seq_params)?;
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 6/9] gpu: nova-core: consolidate GSP boot parameters into GspBootContext
Quoted 18 lines · Zhi Wang
The GspHal trait methods
boot()andpost_boot()accept a long list of individual parameters (dev, bar, chipset,gsp_falcon,sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a
dev()helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in
Gsp::boot()accordingly.Signed-off-by Zhi Wang <[email protected]>
Quoted 8 lines
--- drivers/gpu/nova-core/gpu.rs | 14 ++++++- drivers/gpu/nova-core/gsp.rs | 22 +++++++++++ drivers/gpu/nova-core/gsp/boot.rs | 55 ++++++++++++-------------- drivers/gpu/nova-core/gsp/hal.rs | 23 +++-------- drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++--- drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++--------- 6 files changed, 85 insertions(+), 74 deletions(-)
I think this one is also good to take independently, it groups all the context required for booting the GSP in a dedicated type, which is something we might want to do as the SEC2 and FSP paths require different resources (and thus could use different associated types).
So I'm leaning towards merging this early as it is independent from the preceding patches in the series and makes the first step in that direction.
Re: [PATCH 6/9] gpu: nova-core: consolidate GSP boot parameters into GspBootContext
Quoted 28 lines · Alexandre Courbot
Quoted 18 lines · Zhi Wang
The GspHal trait methods
boot()andpost_boot()accept a long list of individual parameters (dev, bar, chipset,gsp_falcon,sec2_falcon) that are threaded through the entire GSP boot call chain. This makes the signatures unwieldy and difficult to extend as new boot-time context (e.g. vGPU state) is introduced.Introduce a GspBootContext struct that bundles the common boot parameters into a single object, and refactor the GspHal trait to accept &GspBootContext instead of individual arguments. The struct also exposes a
dev()helper with proper lifetime annotation so that HAL implementations can extract the device reference without reborrowing constraints.Update both TU102 and GH100 HAL implementations to extract their required parameters from the context struct, and simplify the call sites in
Gsp::boot()accordingly.Signed-off-by Zhi Wang <[email protected]>Quoted 8 lines
--- drivers/gpu/nova-core/gpu.rs | 14 ++++++- drivers/gpu/nova-core/gsp.rs | 22 +++++++++++ drivers/gpu/nova-core/gsp/boot.rs | 55 ++++++++++++-------------- drivers/gpu/nova-core/gsp/hal.rs | 23 +++-------- drivers/gpu/nova-core/gsp/hal/gh100.rs | 14 ++++--- drivers/gpu/nova-core/gsp/hal/tu102.rs | 31 ++++++--------- 6 files changed, 85 insertions(+), 74 deletions(-)
Quoted 8 lines
I think this one is also good to take independently, it groups all the context required for booting the GSP in a dedicated type, which is something we might want to do as the SEC2 and FSP paths require different resources (and thus could use different associated types).
So I'm leaning towards merging this early as it is independent from the preceding patches in the series and makes the first step in that direction.
Merged into drm-rust-next, thanks! With two minor edits:
[acourbot: pass GspBootContext by value to Gsp::boot.] │
So the boot context can be modified by Gsp::boot if needed (although that shouldn't be necessary, but it is more natural to pass by value here regardless).
[acourbot: deconstruct GspBootContext in Gsp::boot to simplify diff.]
To stay consistent with what was done in the HALs. This also makes the diff considerably smaller and easier to parse.
PATCH 7/9 gpu: nova-core: add vGPU preludes
In reply to Zhi WangThe driver needs to detect vGPU capability before firmware loading so that the GSP boot flow can be adjusted accordingly.
vGPU capability is determined by two sources: the PCI SR-IOV totalvfs count (whether the device advertises VFs) and the FSP PRC (Product Reconfiguration Control) knob (whether vGPU mode is actively enabled on this boot cycle). Both must agree for vGPU to proceed.
Introduce VgpuManager to encapsulate vGPU state detection and tracking. On creation it queries totalvfs via sriov_get_totalvfs(); during GSP boot the PRC knob is read from FSP to refine the initial estimate. Extend GspBootContext with vgpu_requested and total_vfs fields to carry this state across the boot sequence. The FSP PRC read is performed inside the GH100 HAL boot method, where the FSP falcon is already available.
--- drivers/gpu/nova-core/fsp.rs | 1 - drivers/gpu/nova-core/gpu.rs | 24 +++++++++++-- drivers/gpu/nova-core/gsp.rs | 5 +++ drivers/gpu/nova-core/gsp/hal/gh100.rs | 8 ++++- drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/vgpu.rs | 47 ++++++++++++++++++++++++++ 6 files changed, 82 insertions(+), 4 deletions(-) create mode 100644 drivers/gpu/nova-core/vgpu.rs
drivers/gpu/nova-core/fsp.rs +0 -1
@@ -329,7 +329,6 @@ impl Fsp {/// Queries FSP's Management Partition for the active vGPU mode knob value. /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU, /// [`VgpuMode::Disabled`] otherwise. -#[expect(dead_code)] pub (crate )fn read_vgpu_mode ( &mut self, dev: &device::Device<device::Bound>,
drivers/gpu/nova-core/gpu.rs +22 -2
@@ -9,7 +9,11 @@ io::Io, num::Bounded, pci, - prelude::*, // + prelude::*, + sync::{ + new_mutex, +Mutex ,// + }, };use crate ::{
@@ -27,6 +31,7 @@ GspBootContext, // }, regs, + vgpu::VgpuManager, };mod hal;
@@ -180,6 +185,13 @@ pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> { } } +impl Architecture { +/// Returns true for architectures that support vGPU (RTX PRO 6000 Blackwell Server Edition). +pub (crate )const fn supports_vgpu (&self) ->bool { +matches! (self,Self ::BlackwellGB20x ) + } +} +#[derive( Clone , Copy )] pub (crate )struct Revision { major:Bounded <u8 ,4 >,
@@ -278,9 +290,12 @@ pub(crate) struct Gpu<'gpu> { gsp_falcon: Falcon<GspFalcon>, /// SEC2 falcon instance, used for GSP boot up and cleanup. sec2_falcon: Falcon<Sec2Falcon>, -/// GSP runtime data. Temporarily an empty placeholder. +/// GSP runtime data. #[pin] gsp: Gsp, +/// vGPU state (SR-IOV / FSP PRC), behind Mutex for FFI concurrency. +#[pin] + vgpu: Mutex<VgpuManager>,/// GSP unload firmware bundle, if any. unload_bundle: Option<gsp::UnloadBundle>, }
@@ -319,18 +334,23 @@ pub(crate) fn new( sec2_falcon: Falcon::new(pdev.as_ref(), spec.chipset)?, + vgpu <- new_mutex!(VgpuManager::new(pdev, spec.chipset.arch())?, "vgpu_manager"), + gsp <- Gsp::new(pdev),// This member must be initialized last, so the `UnloadBundle` can never be dropped from // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run // in case of failure. unload_bundle: { + let mgr = vgpu.lock();let ctx =GspBootContext { pdev, bar, chipset: spec.chipset, gsp_falcon, sec2_falcon, + vgpu_requested: core::cell::Cell ::new (mgr.vgpu_requested), + total_vfs: mgr.total_vfs, }; gsp.boot (&ctx)? },
drivers/gpu/nova-core/gsp.rs +5 -0
@@ -3,6 +3,8 @@mod boot;mod hal; +use core::cell::Cell ; +use kernel::{ debugfs, device,
@@ -56,6 +58,9 @@ pub(crate) struct GspBootContext<'a> { pub(crate) chipset: Chipset, pub(crate) gsp_falcon: &'a Falcon<GspFalcon>, pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>, + pub(crate) vgpu_requested: Cell<bool>, +#[expect(dead_code)] + pub(crate) total_vfs: u16, }impl <'a>GspBootContext <'a> {
drivers/gpu/nova-core/gsp/hal/gh100.rs +7 -1
@@ -24,7 +24,8 @@ }, fsp::{ FmcBootArgs, - Fsp, // + Fsp, + VgpuMode, // }, gsp::{ boot::BootUnloadGuard,
@@ -174,6 +175,11 @@ fn boot<'a>(let mut fsp =Fsp ::wait_secure_boot (dev, bar, chipset, fsp_fw)?; +let vgpu_mode = fsp.read_vgpu_mode (dev, bar)?; +dev_dbg! (dev,"vGPU mode: {:?}\n" , vgpu_mode); + ctx.vgpu_requested + .set (ctx.vgpu_requested.get () && vgpu_mode ==VgpuMode ::Enabled ); +let args =FmcBootArgs ::new ( dev, chipset,
drivers/gpu/nova-core/nova_core.rs +1 -0
@@ -26,6 +26,7 @@mod regs;mod sbuffer;mod vbios; +mod vgpu;pub (crate )const MODULE_NAME : &core::ffi::CStr = <LocalModule as kernel::ModuleMetadata >::NAME ;
drivers/gpu/nova-core/vgpu.rs new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 7/9] gpu: nova-core: add vGPU preludes
Quoted 65 lines · Zhi Wang
The driver needs to detect vGPU capability before firmware loading so that the GSP boot flow can be adjusted accordingly.
vGPU capability is determined by two sources: the PCI SR-IOV totalvfs count (whether the device advertises VFs) and the FSP PRC (Product Reconfiguration Control) knob (whether vGPU mode is actively enabled on this boot cycle). Both must agree for vGPU to proceed.
Introduce VgpuManager to encapsulate vGPU state detection and tracking. On creation it queries totalvfs via
sriov_get_totalvfs(); during GSP boot the PRC knob is read from FSP to refine the initial estimate. Extend GspBootContext withvgpu_requestedandtotal_vfsfields to carry this state across the boot sequence. The FSP PRC read is performed inside the GH100 HAL boot method, where the FSP falcon is already available.Signed-off-by Zhi Wang <[email protected]>--- drivers/gpu/nova-core/fsp.rs | 1 - drivers/gpu/nova-core/gpu.rs | 24 +++++++++++-- drivers/gpu/nova-core/gsp.rs | 5 +++ drivers/gpu/nova-core/gsp/hal/gh100.rs | 8 ++++- drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/vgpu.rs | 47 ++++++++++++++++++++++++++ 6 files changed, 82 insertions(+), 4 deletions(-) create mode 100644 drivers/gpu/nova-core/vgpu.rsdrivers/gpu/nova-core/fsp.rs +0
-1@@ -329,7 +329,6 @@ impl Fsp {/// Queries FSP's Management Partition for the active vGPU mode knob value. /// Returns [`VgpuMode::Enabled`] if vGPU support is active on this GPU, /// [`VgpuMode::Disabled`] otherwise. -#[expect(dead_code)] pub (crate )fn read_vgpu_mode ( &mut self, dev: &device::Device<device::Bound>,drivers/gpu/nova-core/gpu.rs +6
-1@@ -9,7 +9,11 @@ io::Io, num::Bounded, pci, - prelude::*, // + prelude::*, + sync::{ + new_mutex, +Mutex ,// + }, };use crate ::{@@ -27,6 +31,7 @@ GspBootContext, // }, regs, + vgpu::VgpuManager, };mod hal;
drivers/gpu/nova-core/gpu.rs pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> { · line 193
Quoted 9 lines
drivers/gpu/nova-core/gpu.rs +6
-0@@ -180,6 +185,13 @@ pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> { } } +impl Architecture { +/// Returns true for architectures that support vGPU (RTX PRO 6000 Blackwell Server Edition). +pub (crate )const fn supports_vgpu (&self) ->bool { +matches! (self,Self ::BlackwellGB20x ) + } +}
Is this needed at all? It is only used in VgpuManager::new to condition the call to sriov_get_totalvfs. But sriov_get_totalvfs should return 0 if the device does not support virtual functions, which is the value used on devices for which this method returns false. So unless we can have virtual functions without vGPU support, it seems like we can rely solely on the result of sriov_get_totalvfs.
Another potential gate is read_vgpu_mode - if this always returns false on architectures that don't support vGPU, then we can also rely on that instead of hardcoding supported archs in the driver.
In case we need to keep this method, it should use a HAL method (see pci_config_mirror_range for an example). But I'd rather just remove it if that's possible.
Quoted 5 lines
drivers/gpu/nova-core/gpu.rs +1
-0+#[derive( Clone , Copy )] pub (crate )struct Revision { major:Bounded <u8 ,4 >,
drivers/gpu/nova-core/gpu.rs pub(crate) struct Gpu<'gpu> { · line 298
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs +4
-1@@ -278,9 +290,12 @@ pub(crate) struct Gpu<'gpu> { gsp_falcon: Falcon<GspFalcon>, /// SEC2 falcon instance, used for GSP boot up and cleanup. sec2_falcon: Falcon<Sec2Falcon>, -/// GSP runtime data. Temporarily an empty placeholder. +/// GSP runtime data. #[pin] gsp: Gsp, +/// vGPU state (SR-IOV / FSP PRC), behind Mutex for FFI concurrency. +#[pin] + vgpu: Mutex<VgpuManager>,
The Mutex should not be needed - afaict vgpu is not modified, and we can get a mutable reference to it while we are contructing the GPU anyway.
Quoted 41 lines
drivers/gpu/nova-core/gpu.rs +5
-0/// GSP unload firmware bundle, if any. unload_bundle: Option<gsp::UnloadBundle>, }@@ -319,18 +334,23 @@ pub(crate) fn new( sec2_falcon: Falcon::new(pdev.as_ref(), spec.chipset)?, + vgpu <- new_mutex!(VgpuManager::new(pdev, spec.chipset.arch())?, "vgpu_manager"), + gsp <- Gsp::new(pdev),// This member must be initialized last, so the `UnloadBundle` can never be dropped from // outside of the constructed `Gpu`, ensuring that the unload sequence is properly run // in case of failure. unload_bundle: { + let mgr = vgpu.lock();let ctx =GspBootContext { pdev, bar, chipset: spec.chipset, gsp_falcon, sec2_falcon, + vgpu_requested: core::cell::Cell ::new (mgr.vgpu_requested), + total_vfs: mgr.total_vfs, }; gsp.boot (&ctx)? },drivers/gpu/nova-core/gsp.rs +2
-0@@ -3,6 +3,8 @@mod boot;mod hal; +use core::cell::Cell ; +use kernel::{ debugfs, device,
drivers/gpu/nova-core/gsp.rs pub(crate) struct GspBootContext<'a> { · line 61
Quoted 4 lines
drivers/gpu/nova-core/gsp.rs +1
-0@@ -56,6 +58,9 @@ pub(crate) struct GspBootContext<'a> { pub(crate) chipset: Chipset, pub(crate) gsp_falcon: &'a Falcon<GspFalcon>, pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>, + pub(crate) vgpu_requested: Cell<bool>,
I think we can do without the Cell. Since we own the GspBootContext, we can pass it by value to Gsp::boot, which can them modify it, and pass mutable references to it to the other methods is calls. But it would be even better if we didn't need to modify this at all.
The problem seems to come from the fact that we only build Fsp in the GSP boot code. But vGPU would ideally query it outside of the boot sequence. So a better design would probably be to make Fsp a member of Gpu (behind an Option, as all GPUs don't have one) so we can call read_vgpu_mode outside of the GSP boot sequence and get the definitive enabled state of vGPU earlier in the GPU construction.
I think this is going to be needed anyway because patch 9 also wants to pass the vGPU enabled status to build FbLayout, but can only do so *before* we query the FSP about the PRC knob. So it doesn't necessarily have the correct state.
Another benefit of having the Fsp moved to Gpu will be that you can then pass a reference to it to the VgpuManager::new and have all the vGPU stuff done in vgpu.rs.
Once we have that, we should also be able to add a reference to the (now read-only) VgpuManager to GspBootContext, and get the parameters directly from the source instead of duplicating them.
I can work on a patch to move Fsp if my suggestion is not clear, or let me know if you prefer to do it yourself.
Quoted 85 lines
drivers/gpu/nova-core/gsp.rs +2
-0+#[expect(dead_code)] + pub(crate) total_vfs: u16, }impl <'a>GspBootContext <'a> {drivers/gpu/nova-core/gsp/hal/gh100.rs +7
-1@@ -24,7 +24,8 @@ }, fsp::{ FmcBootArgs, - Fsp, // + Fsp, + VgpuMode, // }, gsp::{ boot::BootUnloadGuard,@@ -174,6 +175,11 @@ fn boot<'a>(let mut fsp =Fsp ::wait_secure_boot (dev, bar, chipset, fsp_fw)?; +let vgpu_mode = fsp.read_vgpu_mode (dev, bar)?; +dev_dbg! (dev,"vGPU mode: {:?}\n" , vgpu_mode); + ctx.vgpu_requested + .set (ctx.vgpu_requested.get () && vgpu_mode ==VgpuMode ::Enabled ); +let args =FmcBootArgs ::new ( dev, chipset,drivers/gpu/nova-core/nova_core.rs +1
-0@@ -26,6 +26,7 @@mod regs;mod sbuffer;mod vbios; +mod vgpu;pub (crate )const MODULE_NAME : &core::ffi::CStr = <LocalModule as kernel::ModuleMetadata >::NAME ;drivers/gpu/nova-core/vgpu.rs new file +36
-0@@ -0,0 +1,47 @@ +// SPDX-License-Identifier: GPL-2.0 + +use kernel::{ + device, + pci, + prelude::*,// +}; + +use crate ::gpu::Architecture ; + +/// vGPU manager. +/// +/// On creation, performs platform detection to determine whether vGPU is +/// requested (PRC knob + totalvfs for Blackwell). The `vgpu_requested` +/// flag may be further refined during boot (e.g. FSP PRC knob read). +pub (crate )struct VgpuManager { +pub (crate ) vgpu_requested:bool , +pub (crate ) vgpu_enabled:bool , +pub (crate ) total_vfs:u16 , +} + +impl VgpuManager { +pub (crate )fn new ( + pdev: &pci::Device <device::Bound >, + arch:Architecture , + ) ->Result <VgpuManager > { +let total_vfs:u16 =if arch.supports_vgpu () { + pdev.sriov_get_totalvfs () + .ok () + .and_then (|n| n.try_into ().ok ()) + .unwrap_or (0 ) + }else { +0 + }; + +Ok (VgpuManager {
drivers/gpu/nova-core/vgpu.rs line 46
Quoted 10 lines
drivers/gpu/nova-core/vgpu.rs new file +10
-0+ vgpu_requested: total_vfs > 0, + vgpu_enabled: false, + total_vfs, + }) + } + +#[expect(dead_code)] +pub (crate )fn set_vgpu_enabled (&mut self, enabled:bool ) { + self.vgpu_enabled = enabled; + }
This method doesn't seem to be called anywhere in the series - let's remove it if it is not needed.
PATCH 8/9 gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled
In reply to Zhi WangThe registry object "RMSetSriovMode" is required to be set when vGPU is enabled.
Convert SetRegistry to use KVec<RegistryEntry> for dynamic construction, allowing entries to be added conditionally at runtime.
Set "RMSetSriovMode" to 1 when nova-core is loading the GSP firmware and initialize the GSP registry objects, if vGPU is enabled.
--- drivers/gpu/nova-core/gsp/boot.rs | 2 +- drivers/gpu/nova-core/gsp/commands.rs | 93 ++++++++++++++++++--------- drivers/gpu/nova-core/gsp/fw.rs | 4 ++ 3 files changed, 67 insertions(+), 32 deletions(-)
drivers/gpu/nova-core/gsp/boot.rs +1 -1
@@ -139,7 +139,7 @@ pub(crate) fn boot( self.cmdq .send_command_no_wait (ctx.bar, commands::SetSystemInfo ::new (ctx.pdev, ctx.chipset))?; self.cmdq - .send_command_no_wait (ctx.bar, commands::SetRegistry ::new ())?; + .send_command_no_wait (ctx.bar, commands::SetRegistry ::new (ctx.vgpu_requested.get ())?)?; hal.post_boot (&self, ctx, &gsp_fw)?;
drivers/gpu/nova-core/gsp/commands.rs +62 -31
134 lines not on this page · raw message
drivers/gpu/nova-core/gsp/fw.rs +4 -0
@@ -299,6 +299,7 @@ pub(crate) enum MsgFunction { OsErrorLog = bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG, PostEvent = bindings::NV_VGPU_MSG_EVENT_POST_EVENT, RcTriggered = bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED, + GpuacctPerfmonUtilSamples = bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES, UcodeLibOsPrint = bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT, }
@@ -348,6 +349,9 @@ fn try_from(value: u32) -> Result<MsgFunction> { bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG => Ok(MsgFunction::OsErrorLog), bindings::NV_VGPU_MSG_EVENT_POST_EVENT => Ok(MsgFunction::PostEvent), bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED => Ok(MsgFunction::RcTriggered), + bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES => { + Ok(MsgFunction::GpuacctPerfmonUtilSamples) + } bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT => Ok(MsgFunction::UcodeLibOsPrint), _ => Err(EINVAL), }
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH 8/9] gpu: nova-core: set RMSetSriovMode when NVIDIA vGPU is enabled
Quoted 10 lines · Zhi Wang
The registry object "RMSetSriovMode" is required to be set when vGPU is enabled.
Convert SetRegistry to use KVec<RegistryEntry> for dynamic construction, allowing entries to be added conditionally at runtime.
Set "RMSetSriovMode" to 1 when nova-core is loading the GSP firmware and initialize the GSP registry objects, if vGPU is enabled.
Signed-off-by Zhi Wang <[email protected]>
Can you split the conversion to KVec and the addition of RMSetSriovMode into two separate patches? The conversion to KVec will likely be useful on its own, and splitting also makes review easier as the concerns are well-separated.
<...>
Quoted 9 lines
drivers/gpu/nova-core/gsp/fw.rs +1
-0@@ -299,6 +299,7 @@ pub(crate) enum MsgFunction { OsErrorLog = bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG, PostEvent = bindings::NV_VGPU_MSG_EVENT_POST_EVENT, RcTriggered = bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED, + GpuacctPerfmonUtilSamples = bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES, UcodeLibOsPrint = bindings::NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT, }
drivers/gpu/nova-core/gsp/fw.rs fn try_from(value: u32) -> Result<MsgFunction> { · line 354
Quoted 6 lines
drivers/gpu/nova-core/gsp/fw.rs +3
-0@@ -348,6 +349,9 @@ fn try_from(value: u32) -> Result<MsgFunction> { bindings::NV_VGPU_MSG_EVENT_OS_ERROR_LOG => Ok(MsgFunction::OsErrorLog), bindings::NV_VGPU_MSG_EVENT_POST_EVENT => Ok(MsgFunction::PostEvent), bindings::NV_VGPU_MSG_EVENT_RC_TRIGGERED => Ok(MsgFunction::RcTriggered), + bindings::NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES => { +Ok (MsgFunction ::GpuacctPerfmonUtilSamples ) + }
Why is this needed?
PATCH 1/1 gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
In reply to Zhi WangGSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.
The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.
Select a pre-calibrated static heap size based on total_vfs (174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and set vf_partition_count accordingly. Extend FbLayout::new() and GspBootContext to propagate total_vfs through the boot path.
--- drivers/gpu/nova-core/fb.rs | 17 +++++++++++++---- drivers/gpu/nova-core/gsp.rs | 2 +- drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++--- drivers/gpu/nova-core/gsp/fw.rs | 12 ++++++++++++ 4 files changed, 37 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/fb.rs +13 -4
@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {impl FbLayout {/// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware. -pub (crate )fn new (chipset:Chipset , bar:Bar0 <'_>, gsp_fw: &GspFirmware ) ->Result <Self > { +pub (crate )fn new ( + chipset:Chipset , + bar:Bar0 <'_>, + gsp_fw: &GspFirmware , + vgpu_requested:bool , + total_vfs:u16 , + ) ->Result <Self > {let hal = hal::fb_hal (chipset);let fb = {
@@ -236,8 +242,11 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resulet wpr2_heap = {const WPR2_HEAP_DOWN_ALIGN :Alignment =Alignment ::new ::<SZ_1M >(); -let wpr2_heap_size = - gsp::LibosParams ::from_chipset (chipset).wpr_heap_size (chipset, fb.end)?; +let wpr2_heap_size =if vgpu_requested { + gsp::vgpu_fw_heap_size (u32::from (total_vfs)) + }else { + gsp::LibosParams ::from_chipset (chipset).wpr_heap_size (chipset, fb.end)? + };let wpr2_heap_addr = (elf.start - wpr2_heap_size).align_down (WPR2_HEAP_DOWN_ALIGN );FbRange (wpr2_heap_addr..(elf.start).align_down (WPR2_HEAP_DOWN_ALIGN ))
@@ -265,7 +274,7 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu wpr2_heap, wpr2, heap, - vf_partition_count: 0, + vf_partition_count: if vgpu_requested { total_vfs as u8 } else { 0 }, pmu_reserved_size: hal.pmu_reserved_size(), }) }
drivers/gpu/nova-core/gsp.rs +1 -1
@@ -27,6 +27,7 @@mod sequencer;pub (crate )use fw::{ + vgpu_fw_heap_size,GspFmcBootParams ,GspFwWprMeta ,LibosParams ,//
@@ -59,7 +60,6 @@ pub(crate) struct GspBootContext<'a> { pub(crate) gsp_falcon: &'a Falcon<GspFalcon>, pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>, pub(crate) vgpu_requested: Cell<bool>, -#[expect(dead_code)] pub(crate) total_vfs: u16, }
drivers/gpu/nova-core/gsp/boot.rs +11 -3
@@ -111,7 +111,13 @@ pub(crate) fn boot( GFP_KERNEL, )?; -let fb_layout =FbLayout ::new (ctx.chipset, ctx.bar, &gsp_fw)?; +let fb_layout =FbLayout ::new ( + ctx.chipset, + ctx.bar, + &gsp_fw, + ctx.vgpu_requested.get (), + ctx.total_vfs, + )?;dev_dbg! (dev,"{:#x?}\n" , fb_layout);let wpr_meta =Coherent ::init (dev,GFP_KERNEL ,GspFwWprMeta ::new (&gsp_fw, &fb_layout))?;
@@ -138,8 +144,10 @@ pub(crate) fn boot( self.cmdq .send_command_no_wait (ctx.bar, commands::SetSystemInfo ::new (ctx.pdev, ctx.chipset))?; - self.cmdq - .send_command_no_wait (ctx.bar, commands::SetRegistry ::new (ctx.vgpu_requested.get ())?)?; + self.cmdq.send_command_no_wait ( + ctx.bar, + commands::SetRegistry ::new (ctx.vgpu_requested.get ())?, + )?; hal.post_boot (&self, ctx, &gsp_fw)?;
drivers/gpu/nova-core/gsp/fw.rs +12 -0
@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {pub (crate )const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX :usize = num::u32_as_usize (bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX ); +const GSP_FW_HEAP_SIZE_VGPU_1VM :u64 =174 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_DEFAULT :u64 =581 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_48VMS :u64 =1370 * u64::SZ_1M ; + +pub (crate )fn vgpu_fw_heap_size (total_vfs:u32 ) ->u64 { +match total_vfs { +1 =>GSP_FW_HEAP_SIZE_VGPU_1VM , +2 ..=32 =>GSP_FW_HEAP_SIZE_VGPU_DEFAULT , + _ =>GSP_FW_HEAP_SIZE_VGPU_48VMS , + } +} +/// Empty type to group methods related to heap parameters for running the GSP firmware. enum GspFwHeapParams {}
Signature · 1 line
2.51.0
Alexandre Courbot
maintainerRe: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
Quoted 28 lines · Zhi Wang
GSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.
The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.
Select a pre-calibrated static heap size based on
total_vfs(174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and setvf_partition_countaccordingly. ExtendFbLayout::new()and GspBootContext to propagatetotal_vfsthrough the boot path.Signed-off-by Zhi Wang <[email protected]>--- drivers/gpu/nova-core/fb.rs | 17 +++++++++++++---- drivers/gpu/nova-core/gsp.rs | 2 +- drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++--- drivers/gpu/nova-core/gsp/fw.rs | 12 ++++++++++++ 4 files changed, 37 insertions(+), 8 deletions(-)drivers/gpu/nova-core/fb.rs +0
-0
drivers/gpu/nova-core/fb.rs pub(crate) struct FbLayout { · line 179
Quoted 10 lines
drivers/gpu/nova-core/fb.rs +6
-1@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {impl FbLayout {/// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware. -pub (crate )fn new (chipset:Chipset , bar:Bar0 <'_>, gsp_fw: &GspFirmware ) ->Result <Self > { +pub (crate )fn new ( + chipset:Chipset , + bar:Bar0 <'_>, + gsp_fw: &GspFirmware , + vgpu_requested:bool , + total_vfs: u16,
total_vfs is only meaningful if vgpu_requested is true. So these two parameters would be better modeled using an Option, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):
enum VgpuRequest {
Disabled,
Requested {
total_vfs: u16,
}
}Quoted 84 lines
drivers/gpu/nova-core/fb.rs +7
-3+ ) -> Result<Self> {let hal = hal::fb_hal (chipset);let fb = {@@ -236,8 +242,11 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resulet wpr2_heap = {const WPR2_HEAP_DOWN_ALIGN :Alignment =Alignment ::new ::<SZ_1M >(); -let wpr2_heap_size = - gsp::LibosParams ::from_chipset (chipset).wpr_heap_size (chipset, fb.end)?; +let wpr2_heap_size =if vgpu_requested { + gsp::vgpu_fw_heap_size (u32::from (total_vfs)) + }else { + gsp::LibosParams ::from_chipset (chipset).wpr_heap_size (chipset, fb.end)? + };let wpr2_heap_addr = (elf.start - wpr2_heap_size).align_down (WPR2_HEAP_DOWN_ALIGN );FbRange (wpr2_heap_addr..(elf.start).align_down (WPR2_HEAP_DOWN_ALIGN ))@@ -265,7 +274,7 @@ pub(crate) fn new(chipset: Chipset, bar: Bar0<'_>, gsp_fw: &GspFirmware) -> Resu wpr2_heap, wpr2, heap, - vf_partition_count: 0, + vf_partition_count: if vgpu_requested { total_vfs as u8 } else { 0 }, pmu_reserved_size: hal.pmu_reserved_size(), }) }drivers/gpu/nova-core/gsp.rs +1
-1@@ -27,6 +27,7 @@mod sequencer;pub (crate )use fw::{ + vgpu_fw_heap_size,GspFmcBootParams ,GspFwWprMeta ,LibosParams ,// @@ -59,7 +60,6 @@ pub(crate) struct GspBootContext<'a> { pub(crate) gsp_falcon: &'a Falcon<GspFalcon>, pub(crate) sec2_falcon: &'a Falcon<Sec2Falcon>, pub(crate) vgpu_requested: Cell<bool>, -#[expect(dead_code)] pub(crate) total_vfs: u16, }drivers/gpu/nova-core/gsp/boot.rs +11
-3@@ -111,7 +111,13 @@ pub(crate) fn boot( GFP_KERNEL, )?; -let fb_layout =FbLayout ::new (ctx.chipset, ctx.bar, &gsp_fw)?; +let fb_layout =FbLayout ::new ( + ctx.chipset, + ctx.bar, + &gsp_fw, + ctx.vgpu_requested.get (), + ctx.total_vfs, + )?;dev_dbg! (dev,"{:#x?}\n" , fb_layout);let wpr_meta =Coherent ::init (dev,GFP_KERNEL ,GspFwWprMeta ::new (&gsp_fw, &fb_layout))?;@@ -138,8 +144,10 @@ pub(crate) fn boot( self.cmdq .send_command_no_wait (ctx.bar, commands::SetSystemInfo ::new (ctx.pdev, ctx.chipset))?; - self.cmdq - .send_command_no_wait (ctx.bar, commands::SetRegistry ::new (ctx.vgpu_requested.get ())?)?; + self.cmdq.send_command_no_wait ( + ctx.bar, + commands::SetRegistry ::new (ctx.vgpu_requested.get ())?, + )?; hal.post_boot (&self, ctx, &gsp_fw)?;drivers/gpu/nova-core/gsp/fw.rs +0
-0
drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106
Quoted 6 lines
drivers/gpu/nova-core/gsp/fw.rs +3
-0@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {pub (crate )const GSP_MSG_QUEUE_ELEMENT_SIZE_MAX :usize = num::u32_as_usize (bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX ); +const GSP_FW_HEAP_SIZE_VGPU_1VM :u64 =174 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_DEFAULT :u64 =581 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_48VMS :u64 =1370 * u64::SZ_1M ;
Do we have a source of truth for these values? I can see definitions for GSP_FW_HEAP_SIZE_VGPU_DEFAULT and GSP_FW_HEAP_SIZE_VGPU_48VMS in OpenRM (meaning we could generate bindings for them), but cannot find a reference for GSP_FW_HEAP_SIZE_VGPU_1VM...
Alexandre Courbot
maintainerRe: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
Quoted 45 lines · Alexandre Courbot
Quoted 28 lines · Zhi Wang
GSP-RM allocates independent RM sub-heaps for each VF partition inside the WPR2 region. The default baremetal heap sizing is far too small for vGPU instance, causing GSP-RM to hit out-of-memory failures during VF initialization.
The host driver must reserve the correct heap size before GSP boots, because the WPR2 region is locked down by the hardware after boot and cannot be resized at runtime. The firmware determines the per-VF carve from the gspFwHeapVfPartitionCount field in the WPR2 metadata header.
Select a pre-calibrated static heap size based on
total_vfs(174 MB for 1 VM, 581 MB for 2-32 VFs, 1370 MB for 48 VFs) and setvf_partition_countaccordingly. ExtendFbLayout::new()and GspBootContext to propagatetotal_vfsthrough the boot path.Signed-off-by Zhi Wang <[email protected]>--- drivers/gpu/nova-core/fb.rs | 17 +++++++++++++---- drivers/gpu/nova-core/gsp.rs | 2 +- drivers/gpu/nova-core/gsp/boot.rs | 14 +++++++++++--- drivers/gpu/nova-core/gsp/fw.rs | 12 ++++++++++++ 4 files changed, 37 insertions(+), 8 deletions(-)drivers/gpu/nova-core/fb.rs +0
-0drivers/gpu/nova-core/fb.rs pub(crate) struct FbLayout { · line 179
Quoted 10 lines
drivers/gpu/nova-core/fb.rs +6
-1@@ -171,7 +171,13 @@ pub(crate) struct FbLayout {impl FbLayout {/// Computes the FB layout for `chipset` required to run the `gsp_fw` GSP firmware. -pub (crate )fn new (chipset:Chipset , bar:Bar0 <'_>, gsp_fw: &GspFirmware ) ->Result <Self > { +pub (crate )fn new ( + chipset:Chipset , + bar:Bar0 <'_>, + gsp_fw: &GspFirmware , + vgpu_requested:bool , + total_vfs: u16,
total_vfsis only meaningful ifvgpu_requestedis true. So these two parameters would be better modeled using anOption, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):
Quoted 6 lines
enum VgpuRequest {
Disabled,
Requested {
total_vfs: u16,
}
}If we can implement my suggestion of moving Fsp into Gpu of patch 7, then this can take a &VgpuManager and not require any new type to express the vGPU state.
Re: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
In reply to Alexandre CourbotQuoted 1 line · On Tue, 16 Jun 2026 23:20:41 +0900 "Alexandre Courbot" <[email protected]>…
On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:
snip
Quoted 6 lines
total_vfsis only meaningful ifvgpu_requestedis true. So these two parameters would be better modeled using anOption, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):
Quoted 7 lines
enum VgpuRequest {
Disabled,
Requested {
total_vfs: u16,
}
}Make sense. Let me try.
Quoted 1 line
Quoted 1 line
+ ) -> Result<Self> {
snip
Quoted 8 lines
Quoted 8 lines
(quoted diff) +0
-0
b/drivers/gpu/nova-core/gsp/fw.rsindex14424a2c2d83..2f3cbc5d5114100644 ---a/drivers/gpu/nova-core/gsp/fw.rs
+++b/drivers/gpu/nova-core/gsp/fw.rs(quoted diff) +0
-0@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {pub(crate) const
GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize =num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);
Quoted 8 lines
drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106
Quoted 3 lines
drivers/gpu/nova-core/gsp/fw.rs +3
-0+const GSP_FW_HEAP_SIZE_VGPU_1VM :u64 =174 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_DEFAULT :u64 =581 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_48VMS :u64 =1370 * u64::SZ_1M ;Do we have a source of truth for these values? I can see definitions for
GSP_FW_HEAP_SIZE_VGPU_DEFAULTandGSP_FW_HEAP_SIZE_VGPU_48VMSin OpenRM (meaning we could generate bindings for them), but cannot find a reference forGSP_FW_HEAP_SIZE_VGPU_1VM...
I see. Let me remove this one and we can add this when OpenRM has it.
Z.
Alexandre Courbot
maintainerRe: [PATCH] gpu: nova-core: reserve a larger GSP WPR2 heap when vGPU is enabled
Quoted 21 lines · Zhi Wang
Quoted 1 line · On Tue, 16 Jun 2026 23:20:41 +0900 "Alexandre Courbot" <[email protected]>…
On Thu Jun 4, 2026 at 8:43 PM JST, Zhi Wang wrote:
snip
Quoted 6 lines
total_vfsis only meaningful ifvgpu_requestedis true. So these two parameters would be better modeled using anOption, or maybe even better a dedicated parameter type for vGPU settings? Something like (please pick a better name if any):Quoted 7 lines
enum VgpuRequest { Disabled, Requested { total_vfs: u16, } }
Quoted 1 line
Make sense. Let me try.
Actually, please check my later reply to this patch [1]. We should be able to completely get rid of this if we move the Fsp into Gpu as described in [2].
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 22 lines
Quoted 2 lines
Quoted 2 lines
+ ) -> Result<Self> {
snip
Quoted 8 lines
Quoted 8 lines
(quoted diff) +0
-0
b/drivers/gpu/nova-core/gsp/fw.rsindex14424a2c2d83..2f3cbc5d5114100644 ---a/drivers/gpu/nova-core/gsp/fw.rs
+++b/drivers/gpu/nova-core/gsp/fw.rs(quoted diff) +0
-0@@ -101,6 +101,18 @@ pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) {pub(crate) const
GSP_MSG_QUEUE_ELEMENT_SIZE_MAX: usize =num::u32_as_usize(bindings::GSP_MSG_QUEUE_ELEMENT_SIZE_MAX);Quoted 8 lines
drivers/gpu/nova-core/gsp/fw.rs pub(in crate::gsp) fn advance_cpu_write_ptr(qs: &Coherent<GspMem>, count: u32) { · line 106
Quoted 3 lines
drivers/gpu/nova-core/gsp/fw.rs +3
-0+const GSP_FW_HEAP_SIZE_VGPU_1VM :u64 =174 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_DEFAULT :u64 =581 * u64::SZ_1M ; +const GSP_FW_HEAP_SIZE_VGPU_48VMS :u64 =1370 * u64::SZ_1M ;Do we have a source of truth for these values? I can see definitions for
GSP_FW_HEAP_SIZE_VGPU_DEFAULTandGSP_FW_HEAP_SIZE_VGPU_48VMSin OpenRM (meaning we could generate bindings for them), but cannot find a reference forGSP_FW_HEAP_SIZE_VGPU_1VM...
Quoted 1 line
I see. Let me remove this one and we can add this when OpenRM has it.
I am fine with keeping it if it is useful, but would like to know how its value was inferred. :)
And also, how stable these values are, since they will need to remain stable across firmware releases. If they turn out to be firmware-dependent, maybe we can turn them into TLV properties of the firmware.