Skip to contentSkip to conversation

gpu: nova-core: add two prerequisites for the r000 GSP firmware

PATCH Applied John Hubbard 4 participants 8 messages Nov 10 – Nov 11

Subsystems (7)
DRM drivers PCI Rust DRM Rust +3 more subsystems
lore
Share & tools
lore
#0 lore raw

cover gpu: nova-core: add two prerequisites for the r000 GSP firmware

Here are two preparatory changes for (or more accurately, from) the r000 GSP firmware series [1], which enables nova-core to work with the GSP firmware for r615+ releases.

Both of these patches stand on their own, so posting them now helps avoid reviewing a huge-er r000 series. :)

* For the Rust PCI abstraction, add an accessor for the PCI domain number.

* Factor out the shutdown path's check for a suspended GSP. The r000 boot sequence needs the same check, so it gets factored out and cleaned up a bit at the same time, by using register!() for the bit 31 access.

This applies to today's drm-rust-next: commit a7cdb17ca0d7.

An LLM coding assistant (Claude Code) helped write both patches, but I have fussed over every line of both the code and the commit messages. The LLM also drafted the first version of this letter, which I've completely rewritten, because it totally missed the point. All of the points. :)

Testing: booted nova-core on my 3-GPU test system (Turing, Ampere and Blackwell GPUs) with all KUnit and built-in Nova tests enabled, as usual.

[1] https://lore.kernel.org/all/[email protected]/

John Hubbard (2):
rust: pci: add domain_nr() accessor
gpu: nova-core: factor out and expose Falcon::is_processor_suspended()

 drivers/gpu/nova-core/falcon/gsp.rs |  7 +++++++
 drivers/gpu/nova-core/gsp/boot.rs   |  6 ++----
 drivers/gpu/nova-core/regs.rs       |  5 +++++
 rust/helpers/pci.c                  |  5 +++++
 rust/kernel/pci.rs                  | 14 ++++++++++++++
 5 files changed, 33 insertions(+), 4 deletions(-)

base-commit: a7cdb17ca0d7ce67e9f3a52d39edd60f03223729

Signature · 1 line
2.56.0
#1 lore raw

PATCH 1/2 rust: pci: add domain_nr() accessor

The nova-core driver has to send the GPU's PCI location to the GPU's firmware as one word that holds the domain, the bus and the device number.

The PCI abstraction has dev_id(), which packs the bus, the device and the function into one value. It has no accessor that reports the domain.

Add an accessor for the domain number.

---
 rust/helpers/pci.c |  5 +++++
 rust/kernel/pci.rs | 14 ++++++++++++++
 2 files changed, 19 insertions(+)
rust/helpers/pci.c +5 -0
index a8540f90b13c..49f2c089d236 100644
@@ -7,6 +7,11 @@ __rust_helper u16 rust_helper_pci_dev_id(struct pci_dev *dev)
 	return PCI_DEVID(dev->bus->number, dev->devfn);
 }
 
+__rust_helper int rust_helper_pci_domain_nr(struct pci_dev *dev)
+{
+	return pci_domain_nr(dev->bus);
+}
+
 __rust_helper resource_size_t
 rust_helper_pci_resource_start(struct pci_dev *pdev, int bar)
 {
rust/kernel/pci.rs +14 -0
index 769d01d23429..57718e48208d 100644
@@ -408,6 +408,20 @@ pub fn dev_id(&self) -> u16 {
         unsafe { bindings::pci_dev_id(self.as_raw()) }
     }
 
+    /// Returns the PCI domain number of the bus that this device is on.
+    #[inline]
+    pub fn domain_nr(&self) -> u32 {
+        // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+        // `struct pci_dev`.
+        let domain_nr = unsafe { bindings::pci_domain_nr(self.as_raw()) };
+
+        // CAST: `pci_domain_nr()` returns an `int`, but a registered bus never has a negative
+        // domain, because `pci_register_host_bridge()` fails when the domain allocation fails.
+        // A negative value here is a kernel bug rather than a condition for a caller to handle.
+        debug_assert!(domain_nr >= 0);
+        domain_nr as u32
+    }
+
     /// Returns the PCI subsystem vendor ID.
     #[inline]
     pub fn subsystem_vendor_id(&self) -> u16 {
Signature · 1 line
2.56.0
#2 lore raw

Re: [PATCH 1/2] rust: pci: add domain_nr() accessor

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] rust: pci: signature mismatch in rust_helper_pci_domain_nr() on ppc and s390 --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] rust: pci: signature mismatch in rust_helper_pci_domain_nr() on ppc and s390
--

commit 40ebd2410b826b9813de5dae1bbe6075723a08bc
Author: John Hubbard <[email protected]>

rust: pci: add domain_nr() accessor

This commit adds a domain_nr() accessor to pci::Device and a corresponding rust_helper_pci_domain_nr() C helper to retrieve the PCI domain number.

Quoted 21 lines
rust/helpers/pci.c +5 -0
index a8540f90b13ce..49f2c089d2369 100644
@@ -7,6 +7,11 @@ __rust_helper u16 rust_helper_pci_dev_id(struct pci_dev *dev)
 	return PCI_DEVID(dev->bus->number, dev->devfn);
 }
 
+__rust_helper int rust_helper_pci_domain_nr(struct pci_dev *dev)
+{
+	return pci_domain_nr(dev->bus);
+}
+
 __rust_helper resource_size_t
 rust_helper_pci_resource_start(struct pci_dev *pdev, int bar)
 {
rust/kernel/pci.rs +0 -0
index 769d01d234296..57718e48208d5 100644

rust/kernel/pci.rs pub fn dev_id(&self) -> u16 { · line 416

Quoted 9 lines
rust/kernel/pci.rs +6 -0
@@ -408,6 +408,20 @@ pub fn dev_id(&self) -> u16 {
         unsafe { bindings::pci_dev_id(self.as_raw()) }
     }
 
+    /// Returns the PCI domain number of the bus that this device is on.
+    #[inline]
+    pub fn domain_nr(&self) -> u32 {
+        // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+        // `struct pci_dev`.
+        let domain_nr = unsafe { bindings::pci_domain_nr(self.as_raw()) };

[Severity: Medium]
Does bindings::pci_domain_nr(self.as_raw()) fail to compile on PowerPC and s390 due to a pointer type mismatch?

In rust/bindings/lib.rs, bindings_generated.rs is included after pub use super::bindings_helper::*, so any non-inline C function emitted by bindgen shadows a rust_helper_* wrapper of the same stripped name. On PowerPC (arch/powerpc/include/asm/pci.h) and s390 (arch/s390/include/asm/pci.h), pci_domain_nr() is declared as a non-inline extern function taking struct pci_bus * rather than a static inline function:

arch/powerpc/include/asm/pci.h:
extern int pci_domain_nr(struct pci_bus *bus);

On those architectures, bindings::pci_domain_nr resolves to the C declaration expecting *mut bindings::pci_bus instead of the helper's
*mut bindings::pci_dev, breaking the build when CONFIG_RUST and CONFIG_PCI are enabled. Either rust_helper_pci_domain_nr() should take struct pci_bus *bus to match the C signature (with domain_nr() passing (*self.as_raw()).bus), or the helper should use a distinct name that does not collide with C pci_domain_nr().

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#3 lore raw

Re: [PATCH 1/2] rust: pci: add domain_nr() accessor

In reply to John Hubbard
Quoted 10 lines · John Hubbard

The nova-core driver has to send the GPU's PCI location to the GPU's firmware as one word that holds the domain, the bus and the device number.

The PCI abstraction has dev_id(), which packs the bus, the device and the function into one value. It has no accessor that reports the domain.

Add an accessor for the domain number.

Is this tag intentional? (Just asking since my tooling flagged it as it wasn't present in the first submission of this patch and some other criteria.)

Quoted 11 lines
---
 rust/helpers/pci.c |  5 +++++
 rust/kernel/pci.rs | 14 ++++++++++++++
 2 files changed, 19 insertions(+)
rust/helpers/pci.c +0 -0
index a8540f90b13c..49f2c089d236 100644

rust/helpers/pci.c __rust_helper u16 rust_helper_pci_dev_id(struct pci_dev *dev) · line 12

Quoted 6 lines
rust/helpers/pci.c +3 -0
@@ -7,6 +7,11 @@ __rust_helper u16 rust_helper_pci_dev_id(struct pci_dev *dev)
 	return PCI_DEVID(dev->bus->number, dev->devfn);
 }
 
+__rust_helper int rust_helper_pci_domain_nr(struct pci_dev *dev)
+{
+	return pci_domain_nr(dev->bus);

Rust helpers should generally be transparent wrappers and not provide any semantics, even if it is just to resolve a pointer.

Quoted 15 lines
rust/helpers/pci.c +2 -0
+}
+
 __rust_helper resource_size_t
 rust_helper_pci_resource_start(struct pci_dev *pdev, int bar)
 {
rust/kernel/pci.rs +2 -0
index 769d01d23429..57718e48208d 100644
@@ -408,6 +408,20 @@ pub fn dev_id(&self) -> u16 {
         unsafe { bindings::pci_dev_id(self.as_raw()) }
     }
 
+    /// Returns the PCI domain number of the bus that this device is on.
+    #[inline]

rust/kernel/pci.rs pub fn dev_id(&self) -> u16 { · line 422

Quoted 10 lines
rust/kernel/pci.rs +10 -0
+    pub fn domain_nr(&self) -> u32 {
+        // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+        // `struct pci_dev`.
+        let domain_nr = unsafe { bindings::pci_domain_nr(self.as_raw()) };
+
+        // CAST: `pci_domain_nr()` returns an `int`, but a registered bus never has a negative
+        // domain, because `pci_register_host_bridge()` fails when the domain allocation fails.
+        // A negative value here is a kernel bug rather than a condition for a caller to handle.
+        debug_assert!(domain_nr >= 0);
+        domain_nr as u32

NIT: Please move the comment directly above the actual cast statement.

Quoted 1 line
rust/kernel/pci.rs +1 -0
+    }
#4 lore raw

PATCH 2/2 gpu: nova-core: factor out and expose Falcon::is_processor_suspended()

In reply to John Hubbard

The r000 boot protocol requires the driver to wait for the GSP suspend bit after each load-and-execute event. The shutdown path already does this.

The boot sequence will soon need this same functionality, so factor it out into a method, and put it in it's logical location as well, as a new method of Falcon<Gsp>.

Also, use register!() for this, instead of manually masking off bit 31. As usual, that results in much more readable code.

No functional changes.

---
 drivers/gpu/nova-core/falcon/gsp.rs | 7 +++++++
 drivers/gpu/nova-core/gsp/boot.rs   | 6 ++----
 drivers/gpu/nova-core/regs.rs       | 5 +++++
 3 files changed, 14 insertions(+), 4 deletions(-)
drivers/gpu/nova-core/falcon/gsp.rs +7 -0
index 1faefaede9fe..54247691b53c 100644
@@ -47,6 +47,13 @@ fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
 }
 
 impl<'a> Falcon<'a, Gsp> {
+    /// Returns `true` if GSP-RM has suspended the RISC-V core.
+    pub(crate) fn is_processor_suspended(&self) -> bool {
+        self.pfalcon
+            .read(regs::NV_PFALCON_FALCON_MAILBOX0_LIBOS)
+            .processor_suspended()
+    }
+
     /// Clears the SWGEN0 latch in the GSP falcon.
     ///
     /// While the latch is set, no later message signals the tree, so a caller that consumed a
drivers/gpu/nova-core/gsp/boot.rs +2 -4
index 1887806dc330..9d5da5b165ac 100644
@@ -2,7 +2,6 @@
 // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
 use kernel::{
-    bits,
     io::poll::read_poll_timeout,
     prelude::*,
     time::Delta,
@@ -98,10 +97,9 @@ fn shutdown_gsp(
         cmdq.send_rpc_command(commands::rpc::UnloadingGuestDriver::new(mode))?;
 
         // Wait until GSP signals it is suspended.
-        const LIBOS_INTERRUPT_PROCESSOR_SUSPENDED: u32 = bits::bit_u32(31);
         read_poll_timeout(
-            || Ok(gsp_falcon.read_mailbox0()),
-            |&mb0| mb0 & LIBOS_INTERRUPT_PROCESSOR_SUSPENDED != 0,
+            || Ok(gsp_falcon.is_processor_suspended()),
+            |suspended| *suspended,
             Delta::from_millis(10),
             Delta::from_secs(5),
         )
drivers/gpu/nova-core/regs.rs +5 -0
index df2eddc58efb..3a0c7159fa7e 100644
@@ -147,6 +147,11 @@ pub(crate) fn usable_fb_size(self) -> u64 {
         31:0    value => u32;
     }
 
+    /// `MAILBOX0` as LibOS, the kernel that GSP-RM runs on, writes it.
+    pub(crate) NV_PFALCON_FALCON_MAILBOX0_LIBOS(u32) => NV_PFALCON_FALCON_MAILBOX0 {
+        31:31   processor_suspended => bool;
+    }
+
     pub(crate) NV_PFALCON_FALCON_MAILBOX1(u32) @ 0x00000044 {
         31:0    value => u32;
     }
Signature · 1 line
2.56.0
#5 lore raw

Re: [PATCH 2/2] gpu: nova-core: factor out and expose Falcon::is_processor_suspended()

Sashiko has reviewed this patch and found no issues.
Quoted 8 lines

The r000 boot protocol requires the driver to wait for the GSP suspend bit after each load-and-execute event. The shutdown path already does this.

The boot sequence will soon need this same functionality, so factor it out into a method, and put it in it's logical location as well, as a new method of Falcon<Gsp>.

Quoted 8 lines

Also, use register!() for this, instead of manually masking off bit 31. As usual, that results in much more readable code.

No functional changes.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] gpu: nova-core: factor out and expose Falcon::is_processor_suspended()

In reply to John Hubbard
Quoted 11 lines · John Hubbard

The r000 boot protocol requires the driver to wait for the GSP suspend bit after each load-and-execute event. The shutdown path already does this.

The boot sequence will soon need this same functionality, so factor it out into a method, and put it in it's logical location as well, as a new method of Falcon<Gsp>.

Also, use register!() for this, instead of manually masking off bit 31. As usual, that results in much more readable code.

Quoted 6 lines

No functional changes.

---

maintainer Applied
#7 lore raw

Re: (subset) [PATCH 0/2] gpu: nova-core: add two prerequisites for the r000 GSP firmware

In reply to John Hubbard
Quoted 1 line · John Hubbard

[PATCH 0/2] gpu: nova-core: add two prerequisites for the r000 GSP firmware

Applied, thanks!

Branch: drm-rust-next
Tree: https://gitlab.freedesktop.org/drm/rust/kernel.git

[2/2] gpu: nova-core: factor out and expose Falcon::is_processor_suspended()

      commit: 6e557330b874

The patch will appear in the next linux-next integration (typically within 24 hours on weekdays).

The patch is queued up for the upcoming merge window for the next major kernel release.