Skip to contentSkip to conversation

nova-core: GPU interrupt support and GSP event delivery

PATCH Applied John Hubbard 6 participants 33 messages Sep 30 – Oct 2

Subsystems (9)
lore
Share & tools
lore
#0 lore raw

v5 cover nova-core: GPU interrupt support and GSP event delivery

This series adds GPU interrupt support to nova-core, so that GSP events reach the driver as interrupts instead of only when the driver polls. v5 applies Alex's review of v4, and makes three of the design changes that he had set aside for after the merge. The series is 15 patches, down from 17. The two rust/ patches are unchanged.

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller, and it latches pending interrupts in a two-level register tree. TOP, the tree's top-level enable register, holds one enable bit per subtree.

The design uses a threaded IRQ handler. The top half reads and writes only GPU registers, while the threaded bottom half drains the message queue.

Still TODO after this: retrieve the dynamic interrupt routing tables from GSP, and use those, instead of the simple, hard-coded approach that was done here for these first two interrupts.

The series is based on drm-rust-next at commit 658c5f0042e6 ("gpu:
nova-core: use a single try_pin_init!() block in probe()"), and it carries no prerequisite commits.

The patches, as applied to drm-rust-next, are on a git branch:

https://github.com/johnhubbard/linux/tree/nova-core-gin-interrupt-tree-v5/

Changes in v5:

* Dropped the patch that poisoned the message queue after a framing or checksum failure. (Alex)

* The GSP interrupt handler borrows the GSP falcon and the command queue instead of holding a BAR and a chipset. (Alex)

* The per-chipset falcon interrupt operations are methods of the existing falcon HAL rather than of a separate falcon interrupt HAL. (Alex)

* The PCI vector allocation and the interrupt tree are one type, which the Gpu owns and the GSP handler borrows. (Alex)

* The tree reset is a method of the tree, and it disables the serviced subtrees at TOP last. (Alex)

* The Gpu holds the TOP enable of the GSP subtree, and the GSP handler's registration holds only the leaf enable of its own vector. (Alex)

* The interrupt tree's trigger and the message queue's drain are added by the patches that first call them, so the separate message queue drain patch is dropped. (Alex)

* The two KUnit suites that tested the HAL dispatchers are removed, and the tree suite's chipset check covers every chipset. (Alex)

* Smaller items from the same review (Alex):

* repr(u32) on the leaf count, and an if/else in the vector check.

* The interrupt HAL performs the rearm, and the tree holds the HAL instead of a leaf count and a rearm method.

* The GFW comment restored at the wait in probe, where GFW is the GPU's boot firmware, and one import group in probe.

* The self-test handler's clear and rearm grouped ahead of the count, and the superfluous dev local removed.

* A comment at the MSI fallback in the vector allocation.

* "chipset" for "part", and "OpenRM" for "Open RM".

* The falcon HAL patch's message no longer claims that a HAL call allocates, since there is no separate HAL left to justify.

TESTING: Tested on Turing (TU117), Ampere (GA104), and Blackwell (GB202), covering probe and chipset identification, the interrupt self-test, a GSP name query over the command queue, driver unload and reload, and the KUnit suites. No test produced a GSP fault on any of the three chips, so the fault path is untested.

Joel Fernandes (2):
rust: sync: completion: add wait_for_completion_timeout()
gpu: nova-core: add the GIN interrupt tree and allocate its vectors

John Hubbard (13):
rust: pci: declare IrqType and IrqTypes with impl_flags
gpu: nova-core: add the GIN vector, leaf and subtree types
gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers
gpu: nova-core: add the per-architecture GIN CPU interrupt HAL
gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
gpu: nova-core: add an interrupt delivery self-test
gpu: nova-core: log GSP events instead of discarding them
gpu: nova-core: return ENOMSG for an unmatched GSP message
gpu: nova-core: bound a GSP wait by a single deadline
gpu: nova-core: add the falcon interrupt registers and HAL methods
gpu: nova-core: service GSP events from the SWGEN0 interrupt
gpu: nova-core: add KUnit tests for the interrupt tree
gpu: nova-core: document the GIN interrupt controller and GSP events

 Documentation/gpu/nova/core/interrupts.rst  | 679 +++++++++++++++++++
 Documentation/gpu/nova/index.rst            |   1 +
 drivers/gpu/nova-core/Kconfig               |   5 +
 drivers/gpu/nova-core/driver.rs             |  19 +-
 drivers/gpu/nova-core/falcon/gsp.rs         |  47 +-
 drivers/gpu/nova-core/falcon/hal.rs         |  19 +-
 drivers/gpu/nova-core/falcon/hal/ga102.rs   |  16 +
 drivers/gpu/nova-core/falcon/hal/tu102.rs   |  49 +-
 drivers/gpu/nova-core/gpu.rs                | 122 +++-
 drivers/gpu/nova-core/gsp/cmdq.rs           | 166 ++++-
 drivers/gpu/nova-core/gsp/commands.rs       |   8 +-
 drivers/gpu/nova-core/gsp/sequencer.rs      |   8 +-
 drivers/gpu/nova-core/irq.rs                |  29 +
 drivers/gpu/nova-core/irq/doorbell_test.rs  | 257 ++++++++
 drivers/gpu/nova-core/irq/gsp.rs            | 196 ++++++
 drivers/gpu/nova-core/irq/hal.rs            |  67 ++
 drivers/gpu/nova-core/irq/hal/gh100.rs      |  40 ++
 drivers/gpu/nova-core/irq/hal/tu102.rs      |  43 ++
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 682 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/regs.rs           |  88 +++
 drivers/gpu/nova-core/nova_core.rs          |   1 +
 drivers/gpu/nova-core/regs.rs               |  69 ++
 rust/kernel/pci/irq.rs                      |  68 +-
 rust/kernel/sync/completion.rs              |  23 +-
 24 files changed, 2594 insertions(+), 108 deletions(-)
 create mode 100644 Documentation/gpu/nova/core/interrupts.rst
 create mode 100644 drivers/gpu/nova-core/irq.rs
 create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs
 create mode 100644 drivers/gpu/nova-core/irq/gsp.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs
 create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs
 create mode 100644 drivers/gpu/nova-core/irq/regs.rs

base-commit: 658c5f0042e6e897338dd04330a363aa7022c85d

Signature · 1 line
2.55.0
#1 lore raw

PATCH v5 1/15 rust: pci: declare IrqType and IrqTypes with impl_flags

The kernel provides impl_flags! for declaring a bitmask type alongside the enum of its individual flags, generating the bit operators and the containment queries.

IrqTypes open-coded that pattern with a with() builder, so a caller naming two interrupt types chained two calls onto IrqTypes::default().

Declare both types through impl_flags!, so the same set reads as IrqType::Msi | IrqType::MsiX.

---
 rust/kernel/pci/irq.rs | 68 +++++++++++++-----------------------------
 1 file changed, 21 insertions(+), 47 deletions(-)
rust/kernel/pci/irq.rs +21 -47
index 6741046ec1c0..f074aad7f1d8 100644
@@ -13,27 +13,26 @@
 };
 use core::num::NonZero;
 
-/// IRQ type flags for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy)]
-pub enum IrqType {
-    /// INTx interrupts.
-    Intx,
-    /// Message Signaled Interrupts (MSI).
-    Msi,
-    /// Extended Message Signaled Interrupts (MSI-X).
-    MsiX,
-}
-
-impl IrqType {
-    /// Convert to the corresponding kernel flags.
-    const fn as_raw(self) -> u32 {
-        match self {
-            IrqType::Intx => bindings::PCI_IRQ_INTX,
-            IrqType::Msi => bindings::PCI_IRQ_MSI,
-            IrqType::MsiX => bindings::PCI_IRQ_MSIX,
-        }
+crate::impl_flags!(
+    /// Set of IRQ types that can be used for PCI interrupt allocation.
+    #[derive(Debug, Clone, Copy, Default)]
+    pub struct IrqTypes(u32);
+
+    /// IRQ type flags for PCI interrupt allocation.
+    #[derive(Debug, Clone, Copy)]
+    pub enum IrqType {
+        /// INTx interrupts.
+        Intx = bindings::PCI_IRQ_INTX,
+
+        /// Message Signaled Interrupts (MSI).
+        Msi = bindings::PCI_IRQ_MSI,
+
+        /// Extended Message Signaled Interrupts (MSI-X).
+        MsiX = bindings::PCI_IRQ_MSIX,
     }
+);
 
+impl IrqType {
     /// Construct from raw value.
     #[inline]
     const fn from_raw(raw: u32) -> Self {
@@ -45,33 +44,10 @@ const fn from_raw(raw: u32) -> Self {
     }
 }
 
-/// Set of IRQ types that can be used for PCI interrupt allocation.
-#[derive(Debug, Clone, Copy, Default)]
-pub struct IrqTypes(u32);
-
 impl IrqTypes {
     /// Create a set containing all IRQ types (MSI-X, MSI, and INTx).
     pub const fn all() -> Self {
-        Self(bindings::PCI_IRQ_ALL_TYPES)
-    }
-
-    /// Build a set of IRQ types.
-    ///
-    /// # Examples
-    ///
-    /// ```ignore
-    /// // Create a set with only MSI and MSI-X (no INTx interrupts).
-    /// let msi_only = IrqTypes::default()
-    ///     .with(IrqType::Msi)
-    ///     .with(IrqType::MsiX);
-    /// ```
-    pub const fn with(self, irq_type: IrqType) -> Self {
-        Self(self.0 | irq_type.as_raw())
-    }
-
-    /// Get the raw flags value.
-    const fn as_raw(self) -> u32 {
-        self.0
+        Self(Self::all_bits())
     }
 }
 
@@ -203,9 +179,7 @@ impl Device<device::Bound> {
     /// let vectors = dev.alloc_irq_vectors(1, 32, pci::IrqTypes::all())?;
     ///
     /// // Allocate MSI or MSI-X only (no INTx interrupts).
-    /// let msi_only = pci::IrqTypes::default()
-    ///     .with(pci::IrqType::Msi)
-    ///     .with(pci::IrqType::MsiX);
+    /// let msi_only = pci::IrqType::Msi | pci::IrqType::MsiX;
     /// let vectors = dev.alloc_irq_vectors(4, 16, msi_only)?;
     /// # Ok(())
     /// # }
@@ -222,7 +196,7 @@ pub fn alloc_irq_vectors(
         // - `pci_alloc_irq_vectors` internally validates all other parameters
         //   and returns error codes.
         let ret = unsafe {
-            bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
+            bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, u32::from(irq_types))
         };
         to_result(ret)?;
 
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH v5 01/15] rust: pci: declare IrqType and IrqTypes with impl_flags

Quoted 7 lines · John Hubbard

The kernel provides impl_flags! for declaring a bitmask type alongside the enum of its individual flags, generating the bit operators and the containment queries.

IrqTypes open-coded that pattern with a with() builder, so a caller naming two interrupt types chained two calls onto IrqTypes::default().

Quoted 6 lines

Declare both types through impl_flags!, so the same set reads as IrqType::Msi | IrqType::MsiX.

Danilo, is this ok if we take this patch through drm-rust-next?

#3 lore raw

PATCH v5 2/15 rust: sync: completion: add wait_for_completion_timeout()

In reply to John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

[jhubbard: return the remaining jiffies]
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
rust/kernel/sync/completion.rs +22 -1
index 35ff049ff078..7e8b3c1c880e 100644
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {
+        // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+        match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
+            0 => None,
+            remaining => Some(remaining),
+        }
+    }
 }
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] rust: sync: wait_for_completion_timeout stack dump on large timeouts --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] rust: sync: wait_for_completion_timeout stack dump on large timeouts --

commit f89eb4d12071b7e3c61766cd1bbbd5c2cfee1ea5
Author: Joel Fernandes <[email protected]>

rust: sync: completion: add wait_for_completion_timeout()

This commit adds wait_for_completion_timeout() to the Rust sync completion primitives. It provides a way to wait for a completion with a specified timeout in jiffies, preventing indefinite stalls in drivers.

Quoted 3 lines
rust/kernel/sync/completion.rs +0 -0

[ ... ]

Quoted 6 lines
rust/kernel/sync/completion.rs +2 -0
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }
+
+    /// Wait for completion of a task, with a timeout.

[ ... ]

rust/kernel/sync/completion.rs pub fn wait_for_completion(&self) { · line 130

Quoted 4 lines
rust/kernel/sync/completion.rs +4 -0
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {
+        // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+        match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {

[Severity: Medium]
Does this code overflow the timeout parameter if large values are provided?

The timeout parameter is passed directly to the C API without bounds checking. Because C implicitly casts it to a signed long downstream, values greater than LONG_MAX can lead to an immediate unhandled timeout and a kernel stack dump.

For example, passing Jiffies::MAX causes the unsigned timeout to overflow into a negative long inside the C wait_for_common() and schedule_timeout() functions. This triggers a kernel stack dump (printk(KERN_ERR ...); dump_stack();) and causes the function to return immediately without waiting, violating the safe function's contract.

Could this be clamped to MAX_SCHEDULE_TIMEOUT to prevent unexpected behavior from excessively large timeouts?

Quoted 5 lines
rust/kernel/sync/completion.rs +4 -0
+            0 => None,
+            remaining => Some(remaining),
+        }
+    }
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

In reply to John Hubbard
Quoted 11 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

[jhubbard: return the remaining jiffies]

Sync maintainers, is this ok if we push this patch through drm-rust-next?

maintainer Acked-by on v5 2/15
#6 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

In reply to John Hubbard
Quoted 8 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

Quoted 4 lines
[jhubbard: return the remaining jiffies]

With the timeout clamped to LONG_MAX as sashiko says:

Feel free to take through drm-rust-next.

#7 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Quoted 9 lines · Alice Ryhl
Quoted 8 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

Quoted 6 lines
Quoted 4 lines
[jhubbard: return the remaining jiffies]

With the timeout clamped to LONG_MAX as sashiko says:

Clearly Sashiko is improving its Rust-for-Linux review capabilities. It missed this on the first four revisions of the series, but now it spotted it. Progress!>

Quoted 3 lines

Feel free to take through drm-rust-next.

Thanks for the review, Alice!

thanks,

Signature · 1 line
John Hubbard
maintainer
#8 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

In reply to John Hubbard
Quoted 11 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

[jhubbard: return the remaining jiffies]

This email should have RfL list CC'ed.

Quoted 26 lines
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
rust/kernel/sync/completion.rs +6 -1
index 35ff049ff078..7e8b3c1c880e 100644
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }

rust/kernel/sync/completion.rs pub fn wait_for_completion(&self) { · line 128

Quoted 10 lines
rust/kernel/sync/completion.rs +10 -0
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {

NACK. Please use the typed Delta<Jiffy>, not this old type alias for c_ulong. I asked Miguel to take that late last cycle because I know there'll be new users that want to use Jiffies this cycle!

This would avoid you having to do clamping on the max side (but you still need to clamp the value to be at least 0). There'll be a helper method for this, but it isn't available yet.
https://lore.kernel.org/rust-for-linux/[email protected]/

Best,
Gary

Quoted 7 lines
rust/kernel/sync/completion.rs +6 -0
+        // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+        match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
+            0 => None,
+            remaining => Some(remaining),
+        }
+    }
 }
#9 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Quoted 56 lines · Gary Guo
Quoted 11 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

[jhubbard: return the remaining jiffies]

This email should have RfL list CC'ed.

Quoted 26 lines
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
rust/kernel/sync/completion.rs +6 -1
index 35ff049ff078..7e8b3c1c880e 100644
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }

rust/kernel/sync/completion.rs pub fn wait_for_completion(&self) { · line 128

Quoted 10 lines
rust/kernel/sync/completion.rs +10 -0
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {

NACK. Please use the typed Delta<Jiffy>, not this old type alias for c_ulong. I asked Miguel to take that late last cycle because I know there'll be new users that want to use Jiffies this cycle!

Quoted 4 lines

This would avoid you having to do clamping on the max side (but you still need to clamp the value to be at least 0). There'll be a helper method for this, but it isn't available yet.
https://lore.kernel.org/rust-for-linux/[email protected]/

I will drop this patch and replace the code using this new method (which is just an optional probe-time self-test) by a polling loop on an atomic. That way we can wait for the helper method to be available and revisit this with the proper type.

#10 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Quoted 9 lines · Alexandre Courbot
Quoted 8 lines · Gary Guo
Quoted 7 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
[jhubbard: return the remaining jiffies]

This email should have RfL list CC'ed.

Hi Gary! Apologies, my send-email scripts had drifted into a nova-centric direction. Will fix.

Quoted 47 lines
Quoted 42 lines
Quoted 27 lines
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
rust/kernel/sync/completion.rs +6 -1
index 35ff049ff078..7e8b3c1c880e 100644
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }

rust/kernel/sync/completion.rs pub fn wait_for_completion(&self) { · line 128

Quoted 10 lines
rust/kernel/sync/completion.rs +10 -0
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {

NACK. Please use the typed Delta<Jiffy>, not this old type alias for c_ulong. I asked Miguel to take that late last cycle because I know there'll be new users that want to use Jiffies this cycle!

Quoted 4 lines

This would avoid you having to do clamping on the max side (but you still need to clamp the value to be at least 0). There'll be a helper method for this, but it isn't available yet.
https://lore.kernel.org/rust-for-linux/[email protected]/

Quoted 4 lines

I will drop this patch and replace the code using this new method (which is just an optional probe-time self-test) by a polling loop on an atomic. That way we can wait for the helper method to be available and revisit this with the proper type.

OK, whatever works best. Just in case, I did stage a fixed-up version of the patch locally (in case I need to send a v6), as per Gary's recommended Delta approach. It looks like this:

rust/kernel/sync/completion.rs +31 -1
index 35ff049ff078..7756677cc6d7 100644
@@ -6,7 +6,15 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::{
+        Delta,
+        Jiffy, //
+    },
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +119,26 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }
+
+    /// Waits for the completion of a task, or until `timeout` elapses.
+    ///
+    /// This method is not interruptible. It clamps a negative `timeout` to zero.
+    ///
+    /// Returns the time remaining, at least one jiffy, if the task completes before `timeout`
+    /// elapses, and [`None`] otherwise.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Delta<Jiffy>) -> Option<Delta<Jiffy>> {
+        // CAST: the value is non-negative after `max`, so the cast to unsigned is lossless.
+        let timeout = timeout.as_jiffies().max(0) as c_ulong;
+
+        // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+        match unsafe { bindings::wait_for_completion_timeout(self.as_raw(), timeout) } {
+            0 => None,
+            // CAST: `remaining` is at most `timeout`, or `1` when `timeout` is zero, so it fits
+            // an `isize`.
+            remaining => Some(Delta::from_jiffies(remaining as isize)),
+        }
+    }
 }

thanks,

Signature · 1 line
John Hubbard
#11 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Quoted 68 lines · John Hubbard
Quoted 9 lines · Alexandre Courbot
Quoted 8 lines · Gary Guo
Quoted 7 lines · John Hubbard

From: Joel Fernandes <[email protected]>

A driver that runs an interrupt self-test during probe waits for the handler to fire. wait_for_completion() has no timeout, so a broken interrupt path stalls probe indefinitely. Add a timeout variant of wait_for_completion().

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
[jhubbard: return the remaining jiffies]

This email should have RfL list CC'ed.

Hi Gary! Apologies, my send-email scripts had drifted into a nova-centric direction. Will fix.

Quoted 47 lines
Quoted 46 lines
Quoted 27 lines
---
 rust/kernel/sync/completion.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
rust/kernel/sync/completion.rs +6 -1
index 35ff049ff078..7e8b3c1c880e 100644
@@ -6,7 +6,12 @@
 //!
 //! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
 
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+    bindings,
+    prelude::*,
+    time::Jiffies,
+    types::Opaque, //
+};
 
 /// Synchronization primitive to signal when a certain task has been completed.
 ///
@@ -111,4 +116,20 @@ pub fn wait_for_completion(&self) {
         // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
         unsafe { bindings::wait_for_completion(self.as_raw()) };
     }

rust/kernel/sync/completion.rs pub fn wait_for_completion(&self) { · line 128

Quoted 10 lines
rust/kernel/sync/completion.rs +10 -0
+
+    /// Wait for completion of a task, with a timeout.
+    ///
+    /// This method waits for the completion of a task, or until `timeout` elapses. It is not
+    /// interruptible. Returns the number of jiffies left when the task completed, or [`None`] if
+    /// `timeout` elapsed first.
+    ///
+    /// See also [`Completion::complete_all`].
+    #[inline]
+    pub fn wait_for_completion_timeout(&self, timeout: Jiffies) -> Option<Jiffies> {

NACK. Please use the typed Delta<Jiffy>, not this old type alias for c_ulong. I asked Miguel to take that late last cycle because I know there'll be new users that want to use Jiffies this cycle!

This would avoid you having to do clamping on the max side (but you still need to clamp the value to be at least 0). There'll be a helper method for this, but it isn't available yet.
https://lore.kernel.org/rust-for-linux/[email protected]/

Quoted 8 lines
Quoted 4 lines

I will drop this patch and replace the code using this new method (which is just an optional probe-time self-test) by a polling loop on an atomic. That way we can wait for the helper method to be available and revisit this with the proper type.

OK, whatever works best. Just in case, I did stage a fixed-up version of the patch locally (in case I need to send a v6), as per Gary's recommended Delta approach. It looks like this:

It's your series so your call, but since the elegant solution will involve using [1] which is still under review, I would suggest waiting for it to introduce wait_for_completion_timeout, moving forward with the rest of the series, and revisit using completions once the base code is ready to limit churn.

Otherwise we are at risk of missing the -rc6 cutoff for this series and that means an even busier next cycle. The polling code is also simpler overall than using completions, and for a probe-time selftest polling is really not a problem. Nova is not losing anything by not using completions for this.

[1] https://lore.kernel.org/rust-for-linux/[email protected]/

maintainer
#12 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

Quoted 18 lines · Alexandre Courbot
Quoted 10 lines · John Hubbard
Quoted 4 lines · Alexandre Courbot

I will drop this patch and replace the code using this new method (which is just an optional probe-time self-test) by a polling loop on an atomic. That way we can wait for the helper method to be available and revisit this with the proper type.

OK, whatever works best. Just in case, I did stage a fixed-up version of the patch locally (in case I need to send a v6), as per Gary's recommended Delta approach. It looks like this:

It's your series so your call, but since the elegant solution will involve using [1] which is still under review, I would suggest waiting for it to introduce wait_for_completion_timeout, moving forward with the rest of the series, and revisit using completions once the base code is ready to limit churn.

Quoted 5 lines

Otherwise we are at risk of missing the -rc6 cutoff for this series and that means an even busier next cycle. The polling code is also simpler overall than using completions, and for a probe-time selftest polling is really not a problem. Nova is not losing anything by not using completions for this.

I think putting it in without .as_jiffies_unsigned is fine. We can cleanup later.

But the code shouldn't be using the postfix .max() for clamping and should be using isize::max or core::cmp::max instead to avoid confusion, see
https://lore.kernel.org/rust-for-linux/[email protected]/.

Best,
Gary

maintainer
#13 lore raw

Re: [PATCH v5 02/15] rust: sync: completion: add wait_for_completion_timeout()

In reply to Alexandre Courbot
Quoted 4 lines · Alexandre Courbot

I will drop this patch and replace the code using this new method (which is just an optional probe-time self-test) by a polling loop on an atomic. That way we can wait for the helper method to be available and revisit this with the proper type.

Sounds good, let's do this.

#14 lore raw

PATCH v5 3/15 gpu: nova-core: add the GIN vector, leaf and subtree types

In reply to John Hubbard

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller. Each interrupt source has a GIN vector number, and the controller latches a pending vector in a two-level tree: one bit of a LEAF register, summarized two leaves at a time by one bit of the TOP register. A vector's number fixes its position in that tree:

leaf = vector / 32
bit = vector % 32
subtree = leaf / 2

A tree implements either 8 or 16 leaves, depending on the GPU family. The leaf count sets both the number of subtrees and the highest vector that the tree carries.

Without distinct types, a vector, a leaf index, a set of vectors within one leaf, a subtree and a set of subtrees are all plain integers. A caller can pass one where another belongs, and a register field can accept the wrong one.

Add a type for each of those, and for the leaf count. A vector converts to its own leaf, bit and subtree. Its constructor rejects, at build time, a number beyond the widest supported tree, and a validation method rejects, at run time, a number beyond the leaves that the current tree implements. A leaf count yields the set of subtrees that it implements.

The module has no user yet.

---
 drivers/gpu/nova-core/irq.rs                |  12 +
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 238 ++++++++++++++++++++
 drivers/gpu/nova-core/nova_core.rs          |   2 +
 3 files changed, 252 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq.rs
 create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs
drivers/gpu/nova-core/irq.rs new file +12 -0
new file mode 100644
index 000000000000..f1323f633a03
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! GPU interrupt support.
+//!
+//! GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller. It latches
+//! every interrupt source in a two-level register tree and delivers the tree to the CPU as a
+//! message-signaled PCI interrupt.
+//!
+//! See `Documentation/gpu/nova/core/interrupts.rst`.
+
+mod interrupt_tree;
drivers/gpu/nova-core/irq/interrupt_tree.rs new file +238 -0

238 lines not on this page · raw message

drivers/gpu/nova-core/nova_core.rs +2 -0
index 4f8ce4e4c187..8202c4982efa 100644
@@ -18,6 +18,8 @@
 mod fsp;
 mod gpu;
 mod gsp;
+#[expect(dead_code)]
+mod irq;
 mod mctp;
 mod mm;
 #[macro_use]
Signature · 1 line
2.55.0
#15 lore raw

PATCH v5 4/15 gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers

In reply to John Hubbard

GIN is the GPU's interrupt controller. It latches each interrupt source in a two-level tree of LEAF registers summarized by TOP, and raises the PCI interrupt when an enabled vector in an enabled subtree becomes pending. A message-signaled interrupt is delivered once per edge, and pre-Hopper MSI rearms delivery by writing the end-of-interrupt register in the BAR0 mirror of PCI configuration space.

Add the CPU tree registers that receiving GSP interrupts and running the software-triggered self-test need: the leaf pending and enable arrays, the TOP enables, and the leaf trigger. Add the end-of-interrupt register, NV_XVE_CYA_2, alongside them.

Use the NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_* names on every chipset. The pre-Hopper headers call the same tree NV_CTRL, but each function reaches its own tree through this aperture on both families. Declare the leaf arrays at 16 entries, the widest tree that any supported chipset implements.

Leave the read-only TOP summary undeclared. A vector that latched while disabled does not appear in TOP, so nova-core never descends from it and reads every implemented leaf instead.

Declare each leaf field as a set of vectors within one leaf and each TOP field as a set of subtrees, so that a set of subtrees cannot be written to a leaf register, nor the reverse. The trigger register's vector field is 12 bits wide, wider than any GIN vector, so a vector converts into it infallibly.

---
 drivers/gpu/nova-core/irq.rs                |  1 +
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 14 ++++
 drivers/gpu/nova-core/irq/regs.rs           | 88 +++++++++++++++++++++
 3 files changed, 103 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq/regs.rs
drivers/gpu/nova-core/irq.rs +1 -0
index f1323f633a03..1ec0bb055d3b 100644
@@ -10,3 +10,4 @@
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 mod interrupt_tree;
+mod regs;
drivers/gpu/nova-core/irq/interrupt_tree.rs +14 -0
index 206e6940ce77..279a43514b1f 100644
@@ -16,6 +16,8 @@
 
 use crate::num;
 
+use super::regs::*;
+
 /// Number of vectors one leaf register carries, one per bit.
 const VECTORS_PER_LEAF: u32 = u32::BITS;
 
@@ -31,6 +33,12 @@
 /// Number of bits needed to address every vector in the widest supported tree.
 const VECTOR_BITS: u32 = (MAX_NUM_LEAVES * VECTORS_PER_LEAF).ilog2();
 
+/// Width of the vector field in the leaf trigger register.
+const TRIGGER_VECTOR_BITS: u32 = {
+    let range = NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::VECTOR_RANGE;
+    num::u8_as_u32(*range.end() - *range.start() + 1)
+};
+
 /// Index of a leaf register within the widest supported tree. An 8-leaf tree implements only the
 /// lower half of the range.
 pub(super) type LeafIndex = Bounded<usize, { MAX_NUM_LEAVES.ilog2() }>;
@@ -236,3 +244,9 @@ pub(super) const fn validate(self, leaves: LeafCount) -> Result {
         }
     }
 }
+
+impl From<GinVector> for Bounded<u32, TRIGGER_VECTOR_BITS> {
+    fn from(vector: GinVector) -> Self {
+        vector.0.extend()
+    }
+}
drivers/gpu/nova-core/irq/regs.rs new file +88 -0

88 lines not on this page · raw message

Signature · 1 line
2.55.0
#16 lore raw

PATCH v5 5/15 gpu: nova-core: add the per-architecture GIN CPU interrupt HAL

In reply to John Hubbard

The GIN CPU interrupt tree differs by GPU family in two ways:

* The size of the tree. Turing, Ampere and Ada implement 8 leaves. Hopper and later families implement 16.

* The write that rearms delivery. A message-signaled interrupt is delivered once per edge, and the PCI side delivers nothing more until the CPU rearms it. Before Hopper, MSI rearms by writing the end-of-interrupt register in the BAR0 mirror of PCI configuration space. On Hopper and later, MSI rearms by clearing and then setting the TOP enables of every serviced subtree, which produces a new edge. MSI-X rearms the same way on every family, but for the handler's own subtree only, since each subtree has its own table entry.

Add an interrupt HAL that provides the leaf count and performs the rearm for each family. Name the interrupt type with two variants, MSI and MSI-X, since nova-core never allocates the level-triggered INTx that the kernel's PCI interrupt type also names.

The HAL has no user yet.

---
 drivers/gpu/nova-core/irq.rs           | 13 +++++
 drivers/gpu/nova-core/irq/hal.rs       | 67 ++++++++++++++++++++++++++
 drivers/gpu/nova-core/irq/hal/gh100.rs | 40 +++++++++++++++
 drivers/gpu/nova-core/irq/hal/tu102.rs | 43 +++++++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 drivers/gpu/nova-core/irq/hal.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs
drivers/gpu/nova-core/irq.rs +13 -0
index 1ec0bb055d3b..62c242b71dc8 100644
@@ -9,5 +9,18 @@
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
+mod hal;
 mod interrupt_tree;
 mod regs;
+
+/// The message-signaled interrupt type that Linux granted.
+///
+/// nova-core never requests INTx, so this has no variant for it, unlike [`kernel::pci::IrqType`].
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum MsiType {
+    /// A single message, which every subtree raises.
+    Msi,
+
+    /// One table entry per subtree.
+    MsiX,
+}
drivers/gpu/nova-core/irq/hal.rs new file +67 -0

67 lines not on this page · raw message

drivers/gpu/nova-core/irq/hal/gh100.rs new file +40 -0
new file mode 100644
index 000000000000..6e307c8d120b
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+use crate::driver::Bar0;
+
+use super::{
+    cycle_top_enables,
+    CpuInterruptHal,
+    LeafCount,
+    MsiType,
+    Subtree,
+    SubtreeSet, //
+};
+
+/// The CPU interrupt tree properties of Hopper and Blackwell.
+struct Gh100;
+
+impl CpuInterruptHal for Gh100 {
+    fn leaf_count(&self) -> LeafCount {
+        LeafCount::Sixteen
+    }
+
+    fn rearm_pci_irq(
+        &self,
+        bar: Bar0<'_>,
+        msi_type: MsiType,
+        serviced: SubtreeSet,
+        subtree: Subtree,
+    ) {
+        let subtrees = match msi_type {
+            MsiType::Msi => serviced,
+            MsiType::MsiX => subtree.into(),
+        };
+
+        cycle_top_enables(bar, subtrees);
+    }
+}
+
+const GH100: Gh100 = Gh100;
+pub(super) const GH100_HAL: &dyn CpuInterruptHal = &GH100;
drivers/gpu/nova-core/irq/hal/tu102.rs new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.55.0
#17 lore raw

PATCH v5 6/15 gpu: nova-core: add the GIN interrupt tree and allocate its vectors

In reply to John Hubbard

From: Joel Fernandes <[email protected]>

Servicing a GIN leaf has a required order: read its pending bits, then clear them. Clearing a leaf first discards every vector latched in it, and the hardware keeps no record of what was discarded. The interrupt never arrives, and no register shows that it was ever pending.

Every subtree enabled at TOP also needs an allocated PCI vector with a handler registered on it. Under MSI-X each subtree has its own table entry. Linux masks every entry until a driver requests its IRQ, and a masked entry sends no message. An enabled subtree whose entry was never requested raises interrupts that never reach a handler, while the leaf and TOP registers show them pending and enabled. Under MSI the whole tree raises a single message, so one entry serves every subtree.

Add the CPU interrupt tree of one PCIe function. The tree allocates the PCI vectors for the subtrees that it services and hands out the request for each, so that the chipset is given once and the vectors live as long as the tree. Reading a leaf yields the handle that clears it, so that the wrong order does not compile. Building a tree fails if it names a subtree that the GPU does not implement. A reset disables every vector, clears every pending bit, rearms delivery, and leaves the serviced subtrees disabled at TOP, so that a handler registered afterwards receives no interrupt left over from boot.

Request MSI-X entries 0 through the highest serviced subtree, since an MSI-X allocation cannot be sparse, and fall back to a single MSI message, never to INTx.

[jhubbard: reworked on top of the GIN vector types: a leaf read yields
the handle that clears it, enables are guarded, the leaf count and the rearm come from the interrupt HAL, the drain reads every implemented leaf rather than descending from TOP, the tree owns its PCI vectors, and a reset returns the tree to a known state before a handler is registered]

---
 drivers/gpu/nova-core/irq.rs                |   2 +-
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 298 +++++++++++++++++++-
 2 files changed, 293 insertions(+), 7 deletions(-)
drivers/gpu/nova-core/irq.rs +1 -1
index 62c242b71dc8..b2cfe73af114 100644
@@ -10,7 +10,7 @@
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
 mod hal;
-mod interrupt_tree;
+pub(crate) mod interrupt_tree;
 mod regs;
 
 /// The message-signaled interrupt type that Linux granted.
drivers/gpu/nova-core/irq/interrupt_tree.rs +292 -6

344 lines not on this page · raw message

Signature · 1 line
2.55.0
#18 lore raw

PATCH v5 7/15 gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor

In reply to John Hubbard

The GPU boots its own firmware, GFW, out of reset, and the driver must not program the GPU until GFW reports completion.

nova-core waited for GFW inside the Gpu constructor, which also boots the GSP. Code that has to run after GFW and before GSP boot, such as a probe-time hardware self-test, had nowhere to go.

Move the wait into probe, ahead of the Gpu constructor, and read the chipset there from a Spec that probe builds itself. Leave the DMA mask in the constructor, since it programs the host rather than the GPU.

Assisted-by LLM
---
 drivers/gpu/nova-core/driver.rs | 14 +++++++++++++-
 drivers/gpu/nova-core/gpu.rs    | 26 +++++++++++++++++++-------
 2 files changed, 32 insertions(+), 8 deletions(-)
drivers/gpu/nova-core/driver.rs +13 -1
index 291a047e4d86..fc321c6a10b0 100644
@@ -24,7 +24,11 @@
 
 use crate::{
     api::NovaCoreApi,
-    gpu::Gpu, //
+    gpu::{
+        self,
+        Gpu,
+        Spec, //
+    }, //
 };
 
 /// Counter for generating unique auxiliary device IDs.
@@ -113,6 +117,14 @@ fn probe<'bound>(
                 pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
             },
 
+            _: {
+                let spec = Spec::new(pdev.as_ref(), bar)?;
+
+                // We must wait for GFW_BOOT completion before doing any significant setup on
+                // the GPU.
+                gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+            },
+
             // TODO: Use self-referential pin-init syntax once available.
             gpu <- Gpu::new(
                 pdev,
drivers/gpu/nova-core/gpu.rs +19 -7
index fb6f8a86a503..65715f906030 100644
@@ -251,7 +251,7 @@ pub struct Spec {
 }
 
 impl Spec {
-    fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
+    pub(crate) fn new(dev: &device::Device, bar: Bar0<'_>) -> Result<Spec> {
         // Some brief notes about boot0 and boot42, in chronological order:
         //
         // NV04 through NV50:
@@ -397,10 +397,8 @@ pub(crate) fn new<'a>(
                 dev_info!(dev,"NVIDIA ({})\n", spec);
             })?,
 
-            // We must wait for GFW_BOOT completion before doing any significant setup on the GPU.
             _: {
-                let hal = hal::gpu_hal(spec.chipset);
-                let dma_mask = hal.dma_mask();
+                let dma_mask = hal::gpu_hal(spec.chipset).dma_mask();
 
                 // SAFETY: `Gpu` owns all DMA allocations for this device, and we are
                 // still constructing it, so no concurrent DMA allocations can exist.
@@ -412,9 +410,6 @@ pub(crate) fn new<'a>(
                 // SAFETY: `Gpu` owns all DMA allocations for this device, and we are
                 // still constructing it, so no concurrent DMA allocations can exist.
                 unsafe { pdev.dma_set_max_seg_size(u32::MAX) };
-
-                hal.wait_gfw_boot_completion(bar)
-                    .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))?;
             },
 
             // Initialize this early because `gsp_resources` depends on it.
@@ -533,6 +528,23 @@ pub(crate) fn run_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bou
     }
 }
 
+/// Waits for GFW, the GPU's boot firmware, to report completion.
+///
+/// The driver must not program the GPU before then.
+///
+/// # Errors
+///
+/// `ETIMEDOUT` if GFW does not report completion in time.
+pub(crate) fn wait_gfw_boot_completion(
+    dev: &device::Device<device::Bound>,
+    bar: Bar0<'_>,
+    chipset: Chipset,
+) -> Result {
+    hal::gpu_hal(chipset)
+        .wait_gfw_boot_completion(bar)
+        .inspect_err(|_| dev_err!(dev, "GFW boot did not complete\n"))
+}
+
 /// Reads the boot0 register and returns its raw value.
 pub(crate) fn boot_0_raw(bar: Bar0<'_>) -> u32 {
     bar.read(regs::NV_PMC_BOOT_0).into_raw()
Signature · 1 line
2.55.0
maintainer
#19 lore raw

Re: [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor

drivers/gpu/nova-core/driver.rs fn probe<'bound>( · line 126

Quoted 11 lines · John Hubbard
drivers/gpu/nova-core/driver.rs +7 -0
@@ -113,6 +117,14 @@ fn probe<'bound>(
                 pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
             },
 
+            _: {
+                let spec = Spec::new(pdev.as_ref(), bar)?;
+
+                // We must wait for GFW_BOOT completion before doing any significant setup on
+                // the GPU.
+                gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+            },

So, apparently this is done in order to be able to add this

	#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
	crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;

in a subsequent patch.

I prefer to make wait_gfw_boot_completion() an associated function of Gpu, keep it called from Gpu::new() and guard the whole block with #[cfg(CONFIG_NOVA_CORE_SELFTESTS)], such that eventually it becomes:

	#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
	_: {
	    let spec = Spec::new(pdev.as_ref(), bar)?;
	
	    Gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;

	    irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
	},

With this Gpu::new() remains self-contained and we don't build up corss initialization dependencies.

#20 lore raw

Re: [PATCH v5 07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor

Quoted 33 lines · Danilo Krummrich

drivers/gpu/nova-core/driver.rs fn probe<'bound>( · line 126

Quoted 11 lines · John Hubbard
drivers/gpu/nova-core/driver.rs +7 -0
@@ -113,6 +117,14 @@ fn probe<'bound>(
                 pdev.iomap_region(bar1_idx, c"nova-core/bar1")?
             },
 
+            _: {
+                let spec = Spec::new(pdev.as_ref(), bar)?;
+
+                // We must wait for GFW_BOOT completion before doing any significant setup on
+                // the GPU.
+                gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+            },

So, apparently this is done in order to be able to add this

	#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
	crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;

in a subsequent patch.

I prefer to make wait_gfw_boot_completion() an associated function of Gpu, keep it called from Gpu::new() and guard the whole block with #[cfg(CONFIG_NOVA_CORE_SELFTESTS)], such that eventually it becomes:

	#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
	_: {
	    let spec = Spec::new(pdev.as_ref(), bar)?;
	
	    Gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;

	    irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
	},
Quoted 2 lines

With this Gpu::new() remains self-contained and we don't build up corss initialization dependencies.

Indeed, this is architecturally more solid. I can apply the change upon pushing.

#21 lore raw

PATCH v5 8/15 gpu: nova-core: add an interrupt delivery self-test

In reply to John Hubbard

A GPU interrupt can be lost in the MSI or MSI-X allocation, in the GIN tree's enables, or in the rearm, and every one of those failures looks the same: no interrupt arrives, and no register or log line says which one broke.

Add a probe-time self-test, built under NOVA_CORE_SELFTESTS, that latches the CPU doorbell vector through the GIN software trigger and waits for a registered handler to service it. One delivery would pass with a broken rearm, because the first message-signaled interrupt arrives whether or not the driver rearms, so the test triggers twice and waits for the first handler to finish before the second trigger. It runs after GFW boot and before GSP boot, on a quiesced tree, and fails probe unless both deliveries arrive, each finds only the doorbell pending, and the leaf ends clear.

The doorbell has the same vector on every supported GPU, so the test names it without asking GSP-RM. It allocates the PCI vectors for the doorbell's subtree and releases them before returning, so under MSI-X the delivery also exercises that subtree's table entry.

---
 drivers/gpu/nova-core/Kconfig               |   5 +
 drivers/gpu/nova-core/driver.rs             |   5 +
 drivers/gpu/nova-core/irq.rs                |   2 +
 drivers/gpu/nova-core/irq/doorbell_test.rs  | 257 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs |  15 ++
 drivers/gpu/nova-core/nova_core.rs          |   2 +-
 6 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs
drivers/gpu/nova-core/Kconfig +5 -0
index 1934f17baa8b..2e11e46c99c7 100644
@@ -24,4 +24,9 @@ config NOVA_CORE_SELFTESTS
 	help
 	  Build the driver self-tests and run them when the GPU is probed.
 
+	  If the interrupt delivery test fails, the probe fails and the driver
+	  does not bind to the GPU. A broken interrupt path would otherwise
+	  show up later as a hang, far from its cause. Every other self-test
+	  logs its failure and lets the probe continue.
+
 	  If unsure, say N.
drivers/gpu/nova-core/driver.rs +5 -0
index fc321c6a10b0..6d45fec6d7cc 100644
@@ -123,6 +123,11 @@ fn probe<'bound>(
                 // We must wait for GFW_BOOT completion before doing any significant setup on
                 // the GPU.
                 gpu::wait_gfw_boot_completion(pdev.as_ref(), bar, spec.chipset)?;
+
+                // The self-test disables and drains the whole tree, so it has to run before
+                // `Gpu::new` boots the GSP.
+                #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+                crate::irq::doorbell_test::run_selftest(pdev, bar, spec.chipset)?;
             },
 
             // TODO: Use self-referential pin-init syntax once available.
drivers/gpu/nova-core/irq.rs +2 -0
index b2cfe73af114..ff8b00a82442 100644
@@ -9,6 +9,8 @@
 //!
 //! See `Documentation/gpu/nova/core/interrupts.rst`.
 
+#[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
+pub(crate) mod doorbell_test;
 mod hal;
 pub(crate) mod interrupt_tree;
 mod regs;
drivers/gpu/nova-core/irq/doorbell_test.rs new file +257 -0

257 lines not on this page · raw message

drivers/gpu/nova-core/irq/interrupt_tree.rs +15 -0
index 6e97828bbd42..4bb27cc8b6cd 100644
@@ -444,6 +444,21 @@ pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
         }
     }
 
+    /// Sets the pending bit of `vector` in its leaf, as if the vector's source had raised it.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if this tree does not implement `vector`.
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
+    pub(super) fn trigger(&self, vector: GinVector) -> Result {
+        vector.validate(self.hal.leaf_count())?;
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
+        );
+
+        Ok(())
+    }
+
     /// Disables every vector in every implemented leaf, including the subtrees that nova-core does
     /// not service. Call this only during probe.
     pub(super) fn disable_all_leaves(&self) {
drivers/gpu/nova-core/nova_core.rs +1 -1
index 8202c4982efa..8c0761dedda4 100644
@@ -18,7 +18,7 @@
 mod fsp;
 mod gpu;
 mod gsp;
-#[expect(dead_code)]
+#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
 mod irq;
 mod mctp;
 mod mm;
Signature · 1 line
2.55.0
#22 lore raw

PATCH v5 9/15 gpu: nova-core: log GSP events instead of discarding them

In reply to John Hubbard

The GSP posts unsolicited messages on the same queue that carries command replies: log records, OS error and robust-channel records, and lifecycle notices.

nova-core discarded every message that was not the reply that a caller was waiting for, and an unrecognized function code aborted the in-flight command. The GSP's error reports never reached the kernel log.

Log every non-reply message according to its function code, on the receive path that already reads it, and leave the in-flight command waiting for its reply. Event payloads, such as XID numbers and log contents, are not decoded.

Assisted-by LLM
---
 drivers/gpu/nova-core/gsp/cmdq.rs | 52 ++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 11 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +41 -11
index a5595da23407..4591f6c61ce6 100644
@@ -559,8 +559,7 @@ fn notify_gsp(bar: Bar0<'_>) {
 
     /// Sends `command` to the GSP and waits for the reply.
     ///
-    /// Messages with non-matching function codes are silently consumed until the expected reply
-    /// arrives.
+    /// Events that arrive before the reply are logged and consumed.
     ///
     /// The queue is locked for the entire send+receive cycle to ensure that no other command can
     /// be interleaved.
@@ -819,8 +818,8 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
 
     /// Receive a message from the GSP.
     ///
-    /// The expected message type is specified using the `M` generic parameter. If the pending
-    /// message has a different function code, `ERANGE` is returned and the message is consumed.
+    /// A message whose function code is `M::FUNCTION` is decoded and returned. Any other message
+    /// is logged as an event.
     ///
     /// The read pointer is always advanced past the message, regardless of whether it matched.
     ///
@@ -829,8 +828,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `EINVAL` if the function code of the message was not recognized.
-    /// - `ERANGE` if the message had a recognized but non-matching function code.
+    /// - `ERANGE` if the message was not the awaited reply.
     ///
     /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -839,11 +837,11 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         Error: From<M::InitError>,
     {
         let message = self.wait_for_msg(timeout)?;
-        let function = message.header.function().map_err(|_| EINVAL)?;
+        let function = message.header.function();
+        let seq = message.header.sequence();
 
-        // Extract the message. Store the result as we want to advance the read pointer even in
-        // case of failure.
-        let result = if function == M::FUNCTION {
+        // An early return here would leave the read pointer on this message.
+        let result = if matches!(function, Ok(f) if f == M::FUNCTION) {
             let (cmd, contents_1) = M::Message::from_bytes_prefix(message.contents.0).ok_or(EIO)?;
             let mut sbuffer = SBufferIter::new_reader([contents_1, message.contents.1]);
 
@@ -854,11 +852,13 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
                         dev_warn!(
                             &self.dev,
                             "GSP message {:?} has unprocessed data\n",
-                            function
+                            M::FUNCTION
                         );
                     }
                 })
         } else {
+            self.log_event(function, seq);
+
             Err(ERANGE)
         };
 
@@ -869,4 +869,34 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
 
         result
     }
+
+    /// Logs an event, meaning a message that no caller was waiting for.
+    ///
+    /// An OS error or robust-channel record is logged at error level and an unknown function code
+    /// at warning level. Every other event is recorded only by the receive trace in
+    /// [`Self::wait_for_msg`].
+    fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
+        match function {
+            Ok(MsgFunction::OsErrorLog) => {
+                dev_err!(&self.dev, "GSP reported an OS error (seq {})\n", seq);
+            }
+            Ok(MsgFunction::RcTriggered) => {
+                dev_err!(
+                    &self.dev,
+                    "GSP triggered robust-channel recovery (seq {})\n",
+                    seq
+                );
+            }
+            // Nothing to do for the remaining known function codes.
+            Ok(_) => {}
+            Err(raw) => {
+                dev_warn!(
+                    &self.dev,
+                    "unknown GSP message function {:#x} (seq {})\n",
+                    raw,
+                    seq
+                );
+            }
+        }
+    }
 }
Signature · 1 line
2.55.0
#23 lore raw

PATCH v5 10/15 gpu: nova-core: return ENOMSG for an unmatched GSP message

In reply to John Hubbard

The GSP posts unsolicited events on the same queue as command replies, so a receive that asks for one message type has to report that the message at the queue head was a different one.

That case returned ERANGE, which means a value outside a valid range and says nothing about a message.

Return ENOMSG, no message of the desired type, instead.

---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 6 +++---
 drivers/gpu/nova-core/gsp/commands.rs  | 2 +-
 drivers/gpu/nova-core/gsp/sequencer.rs | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +3 -3
index 4591f6c61ce6..ad2a14d08cb0 100644
@@ -585,7 +585,7 @@ pub(crate) fn send_command<M>(&self, command: M) -> Result<M::Reply>
         loop {
             match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
                 Ok(reply) => break Ok(reply),
-                Err(ERANGE) => continue,
+                Err(ENOMSG) => continue,
                 Err(e) => break Err(e),
             }
         }
@@ -828,7 +828,7 @@ fn wait_for_msg(&self, timeout: Delta) -> Result<GspMessage<'_>> {
     /// - `ETIMEDOUT` if `timeout` has elapsed before any message becomes available.
     /// - `EIO` if there was some inconsistency (e.g. message shorter than advertised) on the
     ///   message queue.
-    /// - `ERANGE` if the message was not the awaited reply.
+    /// - `ENOMSG` if the message was not the awaited reply.
     ///
     /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
     fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
@@ -859,7 +859,7 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         } else {
             self.log_event(function, seq);
 
-            Err(ERANGE)
+            Err(ENOMSG)
         };
 
         // Advance the read pointer past this message.
drivers/gpu/nova-core/gsp/commands.rs +1 -1
index 59d7d7fb15e8..0c787fd689bf 100644
@@ -191,7 +191,7 @@ pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
     loop {
         match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
             Ok(_) => break Ok(()),
-            Err(ERANGE) => continue,
+            Err(ENOMSG) => continue,
             Err(e) => break Err(e),
         }
     }
drivers/gpu/nova-core/gsp/sequencer.rs +1 -1
index dae34c11eb05..1782ed7d7ca6 100644
@@ -346,7 +346,7 @@ pub(crate) fn run(
         let seq_info = loop {
             match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
                 Ok(seq_info) => break seq_info,
-                Err(ERANGE) => continue,
+                Err(ENOMSG) => continue,
                 Err(e) => return Err(e),
             }
         };
Signature · 1 line
2.55.0
#24 lore raw

PATCH v5 11/15 gpu: nova-core: bound a GSP wait by a single deadline

In reply to John Hubbard

The GSP posts unsolicited events on the same queue as command replies, so a caller waiting for one message consumes whatever arrives first and reads again.

Every read started a fresh five-second timeout, so a steady stream of events extended the wait without bound. The two boot-time waits for an unsolicited event also released the queue mutex between reads, so a command sent from another thread could consume the event and leave the waiter to time out.

Compute one deadline when the wait begins and pass the time remaining to each read, and hold the queue mutex across the whole wait. Put the loop in one helper that the command reply wait and both boot-time event waits share.

Assisted-by LLM
---
 drivers/gpu/nova-core/gsp/cmdq.rs      | 72 ++++++++++++++++++++------
 drivers/gpu/nova-core/gsp/commands.rs  |  8 +--
 drivers/gpu/nova-core/gsp/sequencer.rs |  8 +--
 3 files changed, 59 insertions(+), 29 deletions(-)
drivers/gpu/nova-core/gsp/cmdq.rs +57 -15
index ad2a14d08cb0..ae3808de44e2 100644
@@ -28,7 +28,11 @@
         },
         Mutex, //
     },
-    time::Delta,
+    time::{
+        Delta,
+        Instant,
+        Monotonic, //
+    },
     transmute::{
         AsBytes,
         FromBytes, //
@@ -130,7 +134,9 @@ fn size(&self) -> usize {
 
 /// Trait representing messages received from the GSP.
 ///
-/// This trait tells [`Cmdq::receive_msg`] how it can receive a given type of message.
+/// A reply that [`Cmdq::send_command`] waits for, or an event that [`Cmdq::await_msg`] waits for.
+/// The receiver matches a message's function code against [`Self::FUNCTION`] and decodes the
+/// message with [`Self::read`].
 pub(crate) trait MessageFromGsp: Sized {
     /// Function identifying this message from the GSP.
     const FUNCTION: MsgFunction;
@@ -566,8 +572,9 @@ fn notify_gsp(bar: Bar0<'_>) {
     ///
     /// # Errors
     ///
-    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply is
-    ///   not received within the timeout.
+    /// - `ETIMEDOUT` if space does not become available to send the command, or if the reply does
+    ///   not arrive within [`Self::RECEIVE_TIMEOUT`] of the send, however many events arrive
+    ///   while waiting.
     /// - `EIO` if the variable payload requested by the command has not been entirely
     ///   written to by its [`CommandToGsp::init_variable_payload`] method.
     ///
@@ -582,13 +589,7 @@ pub(crate) fn send_command<M>(&self, command: M) -> Result<M::Reply>
         let mut inner = self.inner.lock();
         inner.send_command(command)?;
 
-        loop {
-            match inner.receive_msg::<M::Reply>(Self::RECEIVE_TIMEOUT) {
-                Ok(reply) => break Ok(reply),
-                Err(ENOMSG) => continue,
-                Err(e) => break Err(e),
-            }
-        }
+        inner.await_msg()
     }
 
     /// Sends `command` to the GSP without waiting for a reply.
@@ -608,15 +609,25 @@ pub(crate) fn send_command_no_wait<M>(&self, command: M) -> Result
         self.inner.lock().send_command(command)
     }
 
-    /// Receive a message from the GSP.
+    /// Waits for an unsolicited GSP event of type `M`. Events that arrive before it are logged and
+    /// consumed.
+    ///
+    /// The queue mutex is held for the whole wait, up to [`Self::RECEIVE_TIMEOUT`], so no other
+    /// caller can send a command or consume an event meanwhile.
     ///
-    /// See [`CmdqInner::receive_msg`] for details.
-    pub(crate) fn receive_msg<M: MessageFromGsp>(&self, timeout: Delta) -> Result<M>
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if the event does not arrive within [`Self::RECEIVE_TIMEOUT`] of the call,
+    ///   however many other events arrive while waiting.
+    /// - `EIO` if a message fails framing or checksum validation.
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+    pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
     where
         // This allows all error types, including `Infallible`, to be used for `M::InitError`.
         Error: From<M::InitError>,
     {
-        self.inner.lock().receive_msg(timeout)
+        self.inner.lock().await_msg()
     }
 }
 
@@ -870,6 +881,37 @@ fn receive_msg<M: MessageFromGsp>(&mut self, timeout: Delta) -> Result<M>
         result
     }
 
+    /// Receives a message of type `M`, waiting up to [`Cmdq::RECEIVE_TIMEOUT`] from the call.
+    ///
+    /// Any other message that arrives first is logged as an event and does not extend the
+    /// deadline.
+    ///
+    /// # Errors
+    ///
+    /// - `ETIMEDOUT` if no message of type `M` arrives before the deadline, however many other
+    ///   messages arrive while waiting.
+    /// - `EIO` if a message fails framing or checksum validation (see [`Self::wait_for_msg`]).
+    ///
+    /// Error codes returned by [`MessageFromGsp::read`] are propagated as-is.
+    fn await_msg<M: MessageFromGsp>(&mut self) -> Result<M>
+    where
+        // This allows all error types, including `Infallible`, to be used for `M::InitError`.
+        Error: From<M::InitError>,
+    {
+        let deadline = Instant::<Monotonic>::now() + Cmdq::RECEIVE_TIMEOUT;
+        loop {
+            let remaining = deadline - Instant::<Monotonic>::now();
+            if remaining.is_negative() {
+                break Err(ETIMEDOUT);
+            }
+            match self.receive_msg::<M>(remaining) {
+                Ok(msg) => break Ok(msg),
+                Err(ENOMSG) => continue,
+                Err(e) => break Err(e),
+            }
+        }
+    }
+
     /// Logs an event, meaning a message that no caller was waiting for.
     ///
     /// An OS error or robust-channel record is logged at error level and an unknown function code
drivers/gpu/nova-core/gsp/commands.rs +1 -7
index 0c787fd689bf..cdbb13674f08 100644
@@ -188,13 +188,7 @@ fn read(
 
 /// Waits for GSP initialization to complete.
 pub(crate) fn wait_gsp_init_done(cmdq: &Cmdq<'_>) -> Result {
-    loop {
-        match cmdq.receive_msg::<GspInitDone>(Cmdq::RECEIVE_TIMEOUT) {
-            Ok(_) => break Ok(()),
-            Err(ENOMSG) => continue,
-            Err(e) => break Err(e),
-        }
-    }
+    cmdq.await_msg::<GspInitDone>().map(|_| ())
 }
 
 /// The `GetGspStaticInfo` command.
drivers/gpu/nova-core/gsp/sequencer.rs +1 -7
index 1782ed7d7ca6..250adc9fe74f 100644
@@ -343,13 +343,7 @@ pub(crate) fn run(
         libos: &'a Coherent<'a, [LibosMemoryRegionInitArgument]>,
         bootloader_app_version: u32,
     ) -> Result {
-        let seq_info = loop {
-            match cmdq.receive_msg::<GspSequence>(Cmdq::RECEIVE_TIMEOUT) {
-                Ok(seq_info) => break seq_info,
-                Err(ENOMSG) => continue,
-                Err(e) => return Err(e),
-            }
-        };
+        let seq_info = cmdq.await_msg::<GspSequence>()?;
 
         let sequencer = GspSequencer {
             bar: ctx.bar,
Signature · 1 line
2.55.0
#25 lore raw

PATCH v5 12/15 gpu: nova-core: add the falcon interrupt registers and HAL methods

In reply to John Hubbard

A falcon latches each interrupt cause that is raised in its IRQSTAT register. On a RISC-V falcon, each cause is routed either to the host, meaning the CPU, or to the falcon's own core, and two routing registers hold that routing. The falcon signals the interrupt tree only when its set of host-routed causes goes from empty to non-empty, so a handler that clears the tree leaf while a cause is still latched has to write INTR_RETRIGGER, which makes the falcon re-emit its causes. Turing falcons have no INTR_RETRIGGER, and the routing registers have different offsets from GA102 on.

Add IRQSTAT, INTR_RETRIGGER, and the two routing registers, the last at both sets of offsets, and add two falcon HAL methods: one intersects an IRQSTAT value with the routing registers, and one retriggers the falcon, or does nothing on Turing. GA100 keeps the Turing HAL, as it does for boot, and the retrigger register is a property of that HAL's instance.

The two methods have no user yet.

Assisted-by LLM
---
 drivers/gpu/nova-core/falcon/hal.rs       | 21 ++++++-
 drivers/gpu/nova-core/falcon/hal/ga102.rs | 16 ++++++
 drivers/gpu/nova-core/falcon/hal/tu102.rs | 51 ++++++++++++++++-
 drivers/gpu/nova-core/regs.rs             | 69 +++++++++++++++++++++++
 4 files changed, 154 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/falcon/hal.rs +20 -1
index 7e532889a1f4..2f02da52334e 100644
@@ -12,6 +12,7 @@
         Architecture,
         Chipset, //
     },
+    regs,
 };
 
 mod ga102;
@@ -70,6 +71,24 @@ fn signature_reg_fuse_version(
     /// these. For anything above, the PIO registers appear to be masked to the CPU, so DMA is the
     /// only usable method.
     fn load_method(&self) -> LoadMethod;
+
+    /// Returns the causes in `latched` that are routed to the host, meaning the CPU, rather than
+    /// to the falcon's own RISC-V core.
+    ///
+    /// The causes routed to the core belong to the firmware running on it, and the host does not
+    /// service them.
+    #[expect(dead_code)]
+    fn host_routed_causes(
+        &self,
+        falcon: &Falcon<'_, E>,
+        latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT;
+
+    /// Retriggers the falcon, which then re-emits its host-routed causes into the interrupt tree.
+    ///
+    /// Turing falcons have no retrigger register, so on Turing this does nothing.
+    #[expect(dead_code)]
+    fn retrigger(&self, falcon: &Falcon<'_, E>);
 }
 
 /// Returns a boxed falcon HAL adequate for `chipset`.
@@ -86,7 +105,7 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
         }
         // GA100 boots like Turing so use Turing HAL
         Architecture::Ampere if chipset == Chipset::GA100 => {
-            KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
+            KBox::new(tu102::Tu102::<E>::ga100(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
         }
         Architecture::Ampere
         | Architecture::Ada
drivers/gpu/nova-core/falcon/hal/ga102.rs +16 -0
index f9a8444cf840..b0eb1901d213 100644
@@ -169,4 +169,20 @@ fn reset_eng(&self, falcon: &Falcon<'_, E>) -> Result {
     fn load_method(&self) -> LoadMethod {
         LoadMethod::Dma
     }
+
+    fn host_routed_causes(
+        &self,
+        falcon: &Falcon<'_, E>,
+        latched: regs::NV_PFALCON_FALCON_IRQSTAT,
+    ) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let pfalcon2 = falcon.pfalcon2;
+        let mask = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQMASK).value();
+        let dest = pfalcon2.read(regs::ga102::NV_PRISCV_RISCV_IRQDEST).value();
+
+        regs::NV_PFALCON_FALCON_IRQSTAT::from(latched.into_raw() & mask & dest)
+    }
+
+    fn retrigger(&self, falcon: &Falcon<'_, E>) {
+        super::tu102::retrigger_ga100(falcon);
+    }
 }
drivers/gpu/nova-core/falcon/hal/tu102.rs +49 -2

73 lines not on this page · raw message

drivers/gpu/nova-core/regs.rs +69 -0

98 lines not on this page · raw message

Signature · 1 line
2.55.0
#26 lore raw

PATCH v5 13/15 gpu: nova-core: service GSP events from the SWGEN0 interrupt

In reply to John Hubbard

The GSP posts each message for the CPU to the GSP-to-CPU queue and then raises SWGEN0, a software-generated interrupt cause of its falcon. GIN delivers SWGEN0 to the CPU on vector 155.

A thread waiting for a command reply was the only reader of the queue, so an event posted between commands stayed unread until the next command was sent.

Register a threaded handler on the GSP vector. Draining the queue takes the command-queue mutex, which can sleep, so the top half reads and writes only registers and wakes the IRQ thread to drain the queue. Every existing read of the queue asks for one message type and waits on a deadline, so add a drain that logs and consumes whatever the GSP has already posted and returns at once.

When a falcon cause other than SWGEN0 stays set after its latch is cleared, as the Blackwell fault-containment and ECC causes do, disable the GSP vector rather than retrigger the falcon. A retrigger would re-emit the cause at once, and the CPU would take the same interrupt again and again.

Assisted-by LLM
---
 drivers/gpu/nova-core/falcon/gsp.rs         |  47 ++++-
 drivers/gpu/nova-core/falcon/hal.rs         |   2 -
 drivers/gpu/nova-core/falcon/hal/tu102.rs   |   2 -
 drivers/gpu/nova-core/gpu.rs                |  96 +++++++++-
 drivers/gpu/nova-core/gsp/cmdq.rs           |  40 ++++
 drivers/gpu/nova-core/irq.rs                |   1 +
 drivers/gpu/nova-core/irq/gsp.rs            | 196 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/interrupt_tree.rs |   3 +
 drivers/gpu/nova-core/nova_core.rs          |   1 -
 9 files changed, 375 insertions(+), 13 deletions(-)
 create mode 100644 drivers/gpu/nova-core/irq/gsp.rs
drivers/gpu/nova-core/falcon/gsp.rs +45 -2
index 4c96ae325fda..1faefaede9fe 100644
@@ -47,13 +47,56 @@ fn pfalcon2(io: Bar0<'_>) -> Mmio<'_, super::PFalcon2Registers> {
 }
 
 impl<'a> Falcon<'a, Gsp> {
-    /// Clears the SWGEN0 bit in the Falcon's IRQ status clear register to
-    /// allow GSP to signal CPU for processing new messages in message queue.
+    /// Clears the SWGEN0 latch in the GSP falcon.
+    ///
+    /// While the latch is set, no later message signals the tree, so a caller that consumed a
+    /// notification by polling must clear it.
     pub(crate) fn clear_swgen0_intr(&self) {
         self.pfalcon
             .write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::zeroed().with_swgen0(true));
     }
 
+    /// Reads the GSP falcon causes that are routed to the host, without clearing any latch.
+    ///
+    /// Every one of them other than SWGEN0 reports a GSP fault.
+    pub(crate) fn read_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let latched = self.pfalcon.read(regs::NV_PFALCON_FALCON_IRQSTAT);
+
+        self.hal.host_routed_causes(self, latched)
+    }
+
+    /// Reads the host-routed causes and clears the SWGEN0 latch if it was set.
+    ///
+    /// Returns the causes as read, before the clear. No other latch changes.
+    pub(crate) fn take_host_intr(&self) -> regs::NV_PFALCON_FALCON_IRQSTAT {
+        let status = self.read_host_intr();
+
+        if status.swgen0() {
+            self.clear_swgen0_intr();
+        }
+
+        status
+    }
+
+    /// Clears the latch of every interrupt cause set in `status`.
+    ///
+    /// A cause driven from outside the falcon is still set on return, and
+    /// [`Self::read_host_intr`] reports the causes that remain.
+    pub(crate) fn clear_intr(&self, status: regs::NV_PFALCON_FALCON_IRQSTAT) {
+        self.pfalcon
+            .write_reg(regs::NV_PFALCON_FALCON_IRQSCLR::from(status.into_raw()));
+    }
+
+    /// Retriggers the GSP falcon, which then re-emits its host-routed causes into the tree.
+    ///
+    /// Call this only once every host cause is clear. A cause still set is re-emitted at once, and
+    /// its vector arrives again as soon as delivery is rearmed.
+    ///
+    /// Does nothing on Turing, whose falcons have no retrigger register.
+    pub(crate) fn retrigger_intr(&self) {
+        self.hal.retrigger(self);
+    }
+
     /// Checks if GSP reload/resume has completed during the boot process.
     pub(crate) fn check_reload_completed(&self, timeout: Delta) -> Result<bool> {
         read_poll_timeout(
drivers/gpu/nova-core/falcon/hal.rs +0 -2
index 2f02da52334e..2643a677caad 100644
@@ -77,7 +77,6 @@ fn signature_reg_fuse_version(
     ///
     /// The causes routed to the core belong to the firmware running on it, and the host does not
     /// service them.
-    #[expect(dead_code)]
     fn host_routed_causes(
         &self,
         falcon: &Falcon<'_, E>,
@@ -87,7 +86,6 @@ fn host_routed_causes(
     /// Retriggers the falcon, which then re-emits its host-routed causes into the interrupt tree.
     ///
     /// Turing falcons have no retrigger register, so on Turing this does nothing.
-    #[expect(dead_code)]
     fn retrigger(&self, falcon: &Falcon<'_, E>);
 }
 
drivers/gpu/nova-core/falcon/hal/tu102.rs +0 -2
index bd66b7759ea6..b25ac645a648 100644
@@ -28,7 +28,6 @@
 /// retrigger register.
 pub(super) struct Tu102<E: FalconEngine> {
     /// If `true`, the falcons have `NV_PFALCON_FALCON_INTR_RETRIGGER`.
-    #[expect(dead_code)]
     has_intr_retrigger: bool,
     _engine: PhantomData<E>,
 }
@@ -52,7 +51,6 @@ pub(super) fn ga100() -> Self {
 }
 
 /// Writes `NV_PFALCON_FALCON_INTR_RETRIGGER`.
-#[expect(dead_code)]
 pub(super) fn retrigger_ga100<E: FalconEngine>(falcon: &Falcon<'_, E>) {
     falcon.pfalcon.write(
         Array::at(0),
drivers/gpu/nova-core/gpu.rs +90 -6

140 lines not on this page · raw message

drivers/gpu/nova-core/gsp/cmdq.rs +40 -0
index ae3808de44e2..1b2f346a237f 100644
@@ -629,6 +629,20 @@ pub(crate) fn await_msg<M: MessageFromGsp>(&self) -> Result<M>
     {
         self.inner.lock().await_msg()
     }
+
+    /// Logs and consumes every message the GSP has already posted, and returns without waiting for
+    /// more.
+    ///
+    /// No caller is waiting for a reply while this holds the queue mutex, so every message is
+    /// logged as an event. See "Draining the GSP-to-CPU queue" in
+    /// `Documentation/gpu/nova/core/interrupts.rst`.
+    ///
+    /// # Errors
+    ///
+    /// `EIO` if a message fails framing or checksum validation.
+    pub(crate) fn drain(&self) -> Result {
+        self.inner.lock().drain()
+    }
 }
 
 /// Inner mutex protected state of [`Cmdq`].
@@ -941,4 +955,30 @@ fn log_event(&self, function: Result<MsgFunction, u32>, seq: u32) {
             }
         }
     }
+
+    /// Logs and consumes every message the queue holds.
+    ///
+    /// # Errors
+    ///
+    /// `EIO` if a message fails framing or checksum validation, or if a message's page count
+    /// overflows a `u32`.
+    fn drain(&mut self) -> Result {
+        while !self.gsp_mem.driver_read_area().0.is_empty() {
+            // A message is available, so this returns without waiting.
+            let msg = self.wait_for_msg(Delta::ZERO)?;
+
+            let pages =
+                u32::try_from(msg.header.length().div_ceil(GSP_PAGE_SIZE)).map_err(|_| {
+                    dev_err!(&self.dev, "GSP drain: message length overflow\n");
+                    EIO
+                })?;
+            let function = msg.header.function();
+            let seq = msg.header.sequence();
+
+            self.gsp_mem.advance_cpu_read_ptr(pages);
+            self.log_event(function, seq);
+        }
+
+        Ok(())
+    }
 }
drivers/gpu/nova-core/irq.rs +1 -0
index ff8b00a82442..02f9bd84dd19 100644
@@ -11,6 +11,7 @@
 
 #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
 pub(crate) mod doorbell_test;
+pub(crate) mod gsp;
 mod hal;
 pub(crate) mod interrupt_tree;
 mod regs;
drivers/gpu/nova-core/irq/gsp.rs new file +196 -0

196 lines not on this page · raw message

drivers/gpu/nova-core/irq/interrupt_tree.rs +3 -0
index 4bb27cc8b6cd..90d4d11e8ba3 100644
@@ -124,10 +124,12 @@ pub(super) const fn all() -> Self {
         Self(u32::MAX)
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn from_raw(raw: u32) -> Self {
         Self(raw)
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn into_raw(self) -> u32 {
         self.0
     }
@@ -240,6 +242,7 @@ pub(super) const fn new<const VECTOR: u32>() -> Self {
         Self(Bounded::<u32, VECTOR_BITS>::new::<VECTOR>())
     }
 
+    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
     pub(super) const fn into_raw(self) -> u32 {
         self.0.get()
     }
drivers/gpu/nova-core/nova_core.rs +0 -1
index 8c0761dedda4..a8f6dc22057b 100644
@@ -18,7 +18,6 @@
 mod fsp;
 mod gpu;
 mod gsp;
-#[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
 mod irq;
 mod mctp;
 mod mm;
Signature · 1 line
2.55.0
#27 lore raw

PATCH v5 14/15 gpu: nova-core: add KUnit tests for the interrupt tree

In reply to John Hubbard

The vector arithmetic and the leaf and subtree sets touch no hardware, so KUnit can cover them without a GPU. A wrong leaf, bit or subtree for a vector would otherwise show up only as a lost interrupt on hardware.

Add the nova_core_gin_tree suite. It checks that a leaf index stops at the widest supported tree, that a leaf count implies the right number of subtrees and vectors, that a leaf count enumerates every leaf in order, that a vector maps to the right leaf, bit and subtree, and that a vector beyond an 8-leaf tree is rejected there and accepted in a 16-leaf tree. It also exercises the subtree set operations and checks that every supported chipset implements the subtree that carries the GSP event.

Assisted-by LLM
---
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 128 +++++++++++++++++++-
 1 file changed, 127 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/irq/interrupt_tree.rs +127 -1
index 90d4d11e8ba3..590f921894b6 100644
@@ -129,7 +129,10 @@ pub(super) const fn from_raw(raw: u32) -> Self {
         Self(raw)
     }
 
-    #[cfg_attr(not(CONFIG_NOVA_CORE_SELFTESTS), expect(dead_code))]
+    #[cfg_attr(
+        not(any(CONFIG_NOVA_CORE_SELFTESTS, CONFIG_KUNIT = "y")),
+        expect(dead_code)
+    )]
     pub(super) const fn into_raw(self) -> u32 {
         self.0
     }
@@ -554,3 +557,126 @@ fn drop(&mut self) {
         clear_top_enables(self.bar, self.serviced);
     }
 }
+
+#[kunit_tests(nova_core_gin_tree)]
+mod tests {
+    use super::*;
+
+    /// A leaf index cannot name a leaf beyond the widest supported tree.
+    #[test]
+    fn leaf_index_bounds() {
+        assert!(LeafIndex::try_new(0).is_some());
+        assert!(LeafIndex::try_new(15).is_some());
+        assert!(LeafIndex::try_new(16).is_none());
+    }
+
+    /// The subtree count, the implemented-subtree set, and the vector count follow the leaf count.
+    #[test]
+    fn leaf_count_derives_subtrees_and_vectors() {
+        assert_eq!(LeafCount::Eight.subtree_count(), 4);
+        assert_eq!(
+            Bounded::<u32, 32>::from(LeafCount::Eight.subtree_set()).get(),
+            0x0f
+        );
+        assert_eq!(LeafCount::Eight.vector_count(), 256);
+
+        assert_eq!(LeafCount::Sixteen.subtree_count(), 8);
+        assert_eq!(
+            Bounded::<u32, 32>::from(LeafCount::Sixteen.subtree_set()).get(),
+            0xff
+        );
+        assert_eq!(LeafCount::Sixteen.vector_count(), 512);
+    }
+
+    /// A tree enumerates every leaf that it implements, in order, and no more.
+    #[test]
+    fn leaf_count_iter_covers_the_tree() {
+        for (count, expected) in [(LeafCount::Eight, 8usize), (LeafCount::Sixteen, 16)] {
+            let mut seen = 0;
+
+            for (index, leaf) in count.iter().enumerate() {
+                assert_eq!(leaf.get(), index);
+                seen += 1;
+            }
+
+            assert_eq!(seen, expected);
+        }
+    }
+
+    /// A vector maps to its leaf, its bit within that leaf, and its subtree. The doorbell (129)
+    /// and the GSP event (155) share a subtree.
+    #[test]
+    fn vector_maps_to_leaf_bit_and_subtree() {
+        let doorbell = GinVector::new::<129>();
+        let gsp = GinVector::new::<155>();
+
+        assert_eq!(doorbell.leaf_index().get(), 4);
+        assert_eq!(doorbell.leaf_mask().into_raw(), 1 << 1);
+        assert_eq!(doorbell.subtree().index(), 2);
+
+        assert_eq!(gsp.leaf_index().get(), 4);
+        assert_eq!(gsp.leaf_mask().into_raw(), 1 << 27);
+        assert_eq!(gsp.subtree().index(), 2);
+
+        assert_eq!(doorbell.subtree(), gsp.subtree());
+    }
+
+    /// Both fixed vectors are within the 8-leaf tree, so every supported chipset implements them.
+    #[test]
+    fn fixed_vectors_fit_the_narrowest_tree() {
+        assert!(GinVector::new::<129>().validate(LeafCount::Eight).is_ok());
+        assert!(GinVector::new::<155>().validate(LeafCount::Eight).is_ok());
+
+        // The first vector beyond an 8-leaf tree.
+        assert!(GinVector::new::<256>().validate(LeafCount::Eight).is_err());
+        assert!(GinVector::new::<256>().validate(LeafCount::Sixteen).is_ok());
+    }
+
+    /// A subtree set reports membership, intersection, and its span from subtree 0.
+    #[test]
+    fn subtree_set_operations() {
+        let gsp = GinVector::new::<155>().subtree();
+
+        assert!(LeafCount::Eight.subtree_set().contains(gsp));
+        assert!(!LeafCount::Eight.subtree_set().is_empty());
+
+        // The GSP needs no subtree above 2, so an MSI-X request covers entries 0 through 2.
+        assert_eq!(SubtreeSet::from(gsp).span(), 3);
+
+        // A 16-leaf tree implements every subtree that an 8-leaf tree does.
+        assert_eq!(
+            LeafCount::Sixteen
+                .subtree_set()
+                .intersection(LeafCount::Eight.subtree_set()),
+            LeafCount::Eight.subtree_set()
+        );
+    }
+
+    /// Iterating a subtree set yields each subtree once, lowest index first, and nothing for an
+    /// empty set.
+    #[test]
+    fn subtree_set_iterates_its_members() {
+        assert!(LeafCount::Eight
+            .subtree_set()
+            .iter()
+            .map(Subtree::index)
+            .eq([0u32, 1, 2, 3]));
+
+        let gsp = SubtreeSet::from(GinVector::new::<155>().subtree());
+        assert!(gsp.iter().map(Subtree::index).eq([2u32]));
+
+        let empty = SubtreeSet::from(Bounded::<u32, 32>::new::<0>());
+        assert_eq!(empty.iter().count(), 0);
+    }
+
+    /// Every supported chipset implements the subtree that carries the GSP event.
+    #[test]
+    fn gsp_subtree_is_implemented_everywhere() {
+        for &chipset in Chipset::ALL {
+            assert!(cpu_interrupt_hal(chipset)
+                .leaf_count()
+                .subtree_set()
+                .contains(crate::irq::gsp::GSP_SUBTREE));
+        }
+    }
+}
Signature · 1 line
2.55.0
#28 lore raw

PATCH v5 15/15 gpu: nova-core: document the GIN interrupt controller and GSP events

In reply to John Hubbard

The interrupt code rests on hardware behavior that the code cannot show on its own: how GIN, the GPU's interrupt controller, records and delivers interrupts, what edge-triggered delivery requires of a handler, and how the GSP signals the CPU. Some of those requirements come from OpenRM rather than from the hardware manuals.

Add a design document that records that behavior, the rules that nova-core follows because of it, and the terms that the code uses for it. The code comments cite the document by section rather than repeating it.

Assisted-by LLM
---
 Documentation/gpu/nova/core/interrupts.rst | 679 +++++++++++++++++++++
 Documentation/gpu/nova/index.rst           |   1 +
 2 files changed, 680 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/interrupts.rst
Documentation/gpu/nova/core/interrupts.rst new file +679 -0
new file mode 100644
index 000000000000..bd6b35aa8dcd
@@ -0,0 +1,679 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+=============================================
+GPU interrupt handling: GIN and the GSP event
+=============================================
+
+This document describes how nova-core receives interrupts from the GPU on Turing
+and later chipsets. It covers the GPU Interrupt and Notification unit (GIN),
+which is the GPU's interrupt controller, and the GSP event, the interrupt that
+nova-core services in normal operation.
+
+Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU
+also has on-chip processors that run their own firmware and receive their own
+interrupts. The GSP (GPU System Processor) is one of them.
+
+Register names are the names from the GPU hardware reference headers. The
+pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers
+call it ``NV_GIN``. This document calls it GIN throughout, because the tree that
+nova-core services is the same on every supported chipset. "Register naming"
+at the end says how the names map onto the headers. OpenRM, NVIDIA's
+open-source GPU kernel driver, is cited wherever nova-core follows it.
+
+Terminology
+===========
+
+The three levels of the controller, innermost first:
+
+leaf
+    One ``LEAF`` register. Each of its 32 bits is the pending bit of one
+    interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or
+    Blackwell tree has 16.
+
+subtree
+    Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the
+    unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery:
+    every interrupt from one subtree arrives on one MSI-X entry.
+
+tree
+    One ``TOP`` register and the leaves under it. Every PCIe function has its
+    own tree, and nova-core services the CPU tree of one function.
+
+The hardware headers, OpenRM, and the Linux PCI API all use the word "vector",
+each for a different number. This document gives each one its own name, and a
+bare "vector" always means a GIN vector.
+
+GIN vector
+    The GPU-internal interrupt source number. It addresses one bit of one leaf.
+    A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0
+    through 255. The CPU doorbell is vector 129 and the GSP event is vector
+    155.
+
+MSI-X entry
+    An index into the device's MSI-X table. One entry serves one subtree.
+
+PCI vector
+    One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X
+    entry, or the single MSI message.
+
+Linux IRQ number
+    What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI
+    vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as
+    well.
+
+The remaining terms, each named for the register or the specification that
+defines it:
+
+enable, disable a vector
+    Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``.
+
+enable, disable a subtree
+    Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``.
+
+serviced subtree
+    A subtree that nova-core enables and has a handler for.
+
+rearm
+    Restoring PCI interrupt delivery after servicing an interrupt. See
+    "Rearming PCI interrupt delivery".
+
+mask
+    Reserved for the two places where hardware and the PCI specification use
+    the word: the MSI-X per-entry Vector Control mask bit, which Linux
+    controls, and the falcon interrupt masks. It never names a GIN enable.
+
+latched, pending
+    Two names for one state, a set ``LEAF`` bit. The vector's source sets the
+    bit whether or not the vector is enabled.
+
+clear a vector
+    Write a 1 to the vector's bit in ``LEAF``. OpenRM calls the same operation
+    ``intrClearLeafVector_HAL``.
+
+pending bits
+    The plain 32-bit value read from a ``LEAF`` register.
+
+notification
+    An interrupt whose only content is that something happened, such as a
+    posted message. Servicing a notification means reading what it announces.
+    The unit that raised it needs no attention. The GSP event is one.
+
+unit
+    Any block that raises an interrupt. "Engine" is reserved for the blocks
+    that do user work: GR, CE, NVDEC, and the like.
+
+falcon
+    One of the GPU's microcontrollers (see
+    Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core
+    inside its falcon. A falcon latches each of its interrupt causes and routes
+    it either to the host, meaning the CPU, or to its own core.
+
+The GIN controller
+==================
+
+A GPU has many interrupt sources: the GSP, the copy engines, the graphics
+engine, video decode and encode, the MMU fault path, timers, and others. GIN
+records which of them are pending and raises the PCI interrupt to the CPU.
+
+Trees
+-----
+
+GIN keeps one tree for each destination it can deliver an interrupt to. The CPU
+has one tree per PCIe function, so the physical function and each virtual
+function have their own. The GSP has a tree, and so do the other on-chip
+processors that receive interrupts. Every tree has the same two-level layout,
+and a function reaches its own tree through the per-function register aperture.
+
+nova-core services the CPU tree of one function. A virtual function's tree
+belongs to that function's driver, and a processor's tree belongs to the
+firmware running on that processor.
+
+The two-level tree
+------------------
+
+A tree is a set of ``LEAF`` registers and one ``TOP`` register.
+
+* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors
+  ``32i`` through ``32i + 31``. A set bit is a pending vector.
+* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``,
+  which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled
+  vector is pending in either leaf.
+
+A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1.
+The other TOP bits read 0. The leaves and subtrees that a chipset has are its
+implemented leaves and subtrees, and "Per-architecture differences" gives the
+counts. For an 8-leaf tree::
+
+    TOP bit 0  ->  subtree 0  ->  LEAF(0), LEAF(1)   vectors   0..63
+    TOP bit 1  ->  subtree 1  ->  LEAF(2), LEAF(3)   vectors  64..127
+    TOP bit 2  ->  subtree 2  ->  LEAF(4), LEAF(5)   vectors 128..191
+    TOP bit 3  ->  subtree 3  ->  LEAF(6), LEAF(7)   vectors 192..255
+
+    LEAF(4), one bit per vector, holds vectors 128..159:
+
+      bit 1  = vector 129  (CPU doorbell)
+      bit 27 = vector 155  (GSP event)
+
+A vector's number fixes its place in the tree::
+
+    leaf    = vector / 32
+    bit     = vector % 32
+    subtree = leaf / 2
+
+Registers
+---------
+
+nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The
+leaf registers are arrays indexed by leaf number.
+
+* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit
+  clears that vector, and a 0 leaves the bit as it was.
+* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors
+  of leaf ``i``, one bit per vector.
+* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per
+  subtree.
+* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as
+  the vector's own source would. It is write-only. The self-test uses it.
+
+nova-core does not read ``TOP``. "Servicing the tree" says why.
+
+Every set and clear register acts per bit: a 1 performs the action for that
+bit, and a 0 leaves the bit alone. No register needs a read-modify-write.
+
+GIN delivers a vector to the CPU only when its leaf enable bit and its
+subtree's TOP enable bit are both set. The enables do not affect the latch. The
+source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show
+that bit, so reading the leaf is the only way to see it.
+
+How a unit interrupt reaches the CPU
+------------------------------------
+
+A unit does not write a ``LEAF`` register. Each unit has an interrupt control
+register that GSP firmware programs. The control register holds the unit's
+vector, the GFID that identifies the PCIe function whose tree receives the
+interrupt, and one enable bit per destination: the CPU, the GSP, and the other
+on-chip processors. When the unit has an event::
+
+    1. The unit sends GIN an interrupt message carrying the vector, the GFID,
+       and the destination enables from its control register.
+    2. In the tree of each destination that the message selects, GIN sets bit
+       (vector % 32) of LEAF(vector / 32).
+    3. If the vector and its subtree are enabled in the CPU tree, GIN raises
+       the PCI interrupt.
+
+Because firmware assigns the vectors, nova-core does not hardcode which vector
+belongs to which unit, with two exceptions. The hardware headers of every
+supported chipset define the GSP event as vector 155, and GSP firmware's own
+interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper
+hardware fixes that number, and GSP firmware keeps it on Hopper and later.
+nova-core names both by number. A driver can fetch the full unit-to-vector
+table from the GSP by RPC, and nouveau does. nova-core does not, because a
+fixed vector needs no lookup.
+
+Edge-triggered delivery
+-----------------------
+
+A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set
+until the CPU clears it. A source that stays high does not set the bit again.
+
+GIN raises the PCI interrupt for a subtree when the subtree's enabled pending
+state goes from low to high::
+
+    Per vector, in leaf i at bit b:
+        LEAF(i)[b] AND LEAF_EN(i)[b]
+
+    Per subtree N, across leaves 2N and 2N + 1:
+        OR of every enabled pending bit  ->  TOP[N]
+
+    Delivery for subtree N:
+        TOP[N] AND TOP_EN[N]  ->  rising edge  ->  PCI interrupt
+
+``TOP_EN`` applies after the summary, so disabling a subtree stops delivery
+without changing what ``TOP`` reports.
+
+Three consequences:
+
+* Code that must find every pending vector reads the leaves. A vector that
+  latched while disabled is not in ``TOP``.
+* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a
+  new edge. GIN delivers an interrupt for a pending bit left uncleared as soon
+  as its subtree is enabled again.
+* A source that holds its signal high produces no new edge after the CPU
+  clears the leaf bit. Such a source has to re-emit its interrupt. The falcons
+  do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon").
+
+Delivery over PCI
+=================
+
+GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux
+grants. nova-core requests MSI-X first and falls back to MSI, and never uses
+INTx.
+
+MSI has a single message, and every subtree raises that one message, so one
+Linux IRQ serves the whole tree.
+
+MSI-X gives each subtree its own table entry, at the index equal to the subtree
+number. Linux masks every entry until a driver requests its Linux IRQ number,
+and a masked entry sends no message: the GPU records the interrupt in the MSI-X
+pending bit array, where it stays until Linux unmasks the entry. An entry that
+the driver never requests is never unmasked. A driver that enables a subtree
+without requesting that subtree's entry loses every interrupt from that
+subtree, with nothing reported: the leaf and TOP registers show the vector
+pending and enabled while no handler runs.
+
+The serviced-subtree invariant
+------------------------------
+
+Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered
+handler.
+
+MSI satisfies this with its single message. MSI-X needs one allocated entry per
+serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests
+entries 0 through the highest serviced subtree::
+
+    MSI-X, with subtree 2 serviced:
+
+      subtree 0  ->  entry 0   allocated, no handler, stays masked
+      subtree 1  ->  entry 1   allocated, no handler, stays masked
+      subtree 2  ->  entry 2   handler here, and its rearm covers subtree 2
+
+    MSI, with any serviced set:
+
+      every serviced subtree  ->  the one allocated PCI vector, whose
+                                  handler's rearm covers the whole serviced set
+
+An allocated entry whose subtree nova-core does not service costs nothing. The
+entry stays masked, and a disabled subtree raises no interrupt.
+
+nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which
+is in subtree 2. OpenRM's headers place its UVM_SHARED interrupt category in
+subtree 2 on every chipset that nova-core supports. The self-test doorbell,
+vector 129, is in the same leaf, and the test allocates its own vectors for it
+(see "Self-test").
+
+Rearming PCI interrupt delivery
+-------------------------------
+
+A message-signaled interrupt is delivered once per edge, and the PCI side
+delivers no further interrupt until the CPU rearms it. The rearm operation
+depends on the GPU family and on the interrupt type that Linux granted:
+
+==================  =====  ===========================================
+Architecture        Type   Rearm operation
+==================  =====  ===========================================
+Turing through Ada  MSI    write the configuration-mirror EOI register
+Hopper and later    MSI    clear then set the serviced TOP enables
+Any                 MSI-X  clear then set the handler's own TOP enable
+==================  =====  ===========================================
+
+The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI
+configuration space, and the value written does not matter. The ``TOP_EN``
+cycle produces a new delivery edge. The MSI forms cover every serviced subtree,
+because one message serves all of them. The MSI-X form covers one subtree,
+because each serviced subtree has its own entry and its own handler.
+
+A handler rearms once per delivered interrupt, on every path, including the
+path where it finds its vector not pending. A handler that skips the rearm
+receives no further interrupts.
+
+OpenRM makes the same split. It writes the configuration-space EOI for MSI on
+pre-Hopper chipsets, and cycles the TOP enables of the subtrees that it
+services for Hopper-plus MSI and for MSI-X.
+
+Servicing the tree
+==================
+
+Servicing a leaf has a required order: read its pending bits, then clear them.
+Clearing a leaf before reading it discards every vector latched in it, and no
+register reports the loss. In nova-core, reading a leaf produces the handle that
+clears it, so that the wrong order does not compile. The handle clears exactly
+the bits that it read, so a vector that latched after the read stays pending.
+
+A handler clears its bit before it services the vector. Clearing afterwards
+would discard an interrupt that the source raised while the handler ran.
+
+nova-core services the tree in two ways.
+
+The notification path services one vector. It reads the vector's leaf, clears
+only the vector's bit, and rearms. The subtree stays enabled, and a vector
+pending beside it in the same leaf keeps its bit set for the code that services
+that vector. The GSP event handler takes this path, and so does the self-test
+handler.
+
+The startup drain walks the whole tree, because it must clear whatever is
+pending across every subtree rather than one known vector. It disables the
+serviced subtrees at ``TOP``, reads and clears every implemented leaf, and
+leaves the subtrees disabled for its caller to enable once the caller is ready
+for deliveries. The drain reads every leaf rather than descending from ``TOP``,
+because sources latch vectors during boot while those vectors are disabled, and
+``TOP`` does not show them. OpenRM's stall-interrupt path reads every leaf for
+the same reason.
+
+The two paths as register operations::
+
+    Startup drain, run once during probe:
+        write TOP_EN_CLEAR = serviced        stop new deliveries
+        for each implemented leaf i:
+            pending = read LEAF(i)
+            write LEAF(i) = pending          clear what was read
+        (returns with TOP_EN still clear)
+
+    Notification, the subtree stays enabled:
+        pending = read LEAF(leaf)            is the handler's bit set?
+        write LEAF(leaf) = bit               clear that one bit
+        rearm PCI interrupt delivery
+
+The drain clears every pending bit, including bits that nova-core never
+services. An uncleared bit holds its subtree in the pending state, and enabling
+that subtree again would deliver an interrupt for a vector that no handler
+services.
+
+The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through
+the configuration mirror, which the drain never writes. An interrupt delivered
+before probe had no handler to rearm it, so the tree reset rearms explicitly
+after the drain. A TOP-enable rearm leaves the serviced subtrees enabled, so
+the reset then disables them again.
+
+nova-core does not serialize access to the tree. The GSP event handler
+touches only its own leaf, and the drain runs during probe, before that handler
+is registered.
+
+Per-architecture differences
+============================
+
+The tree is the same on every supported GPU except for its size, which changes
+at Hopper:
+
+===================  ======  ========  ====================
+GPUs                 Leaves  Subtrees  Implemented subtrees
+===================  ======  ========  ====================
+Turing, Ampere, Ada  8       4         ``0x0f``
+Hopper, Blackwell    16      8         ``0xff``
+===================  ======  ========  ====================
+
+The interrupt HAL provides the leaf count, and the subtree count and the
+implemented-subtree set derive from it. A subtree that the chipset does not
+implement has no TOP bit, so building a tree that services one fails with
+``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported
+chipset has them.
+
+OpenRM's headers assign every interrupt category of a 16-leaf tree to leaves 0
+through 11. The drain reads all 16, because a vector can be latched in any
+implemented leaf.
+
+The HAL performs the rearm as well (see "Rearming PCI interrupt delivery"). Two
+falcon properties also differ by family, and the falcon HAL carries them:
+Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing registers
+moved at GA102 (see "Retriggering a falcon").
+
+The GSP event
+=============
+
+When the GSP has output for the CPU, it writes messages into the GSP-to-CPU
+queue in shared memory and raises SWGEN0, one of the software-generated
+interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector
+155, leaf 4 bit 27, in subtree 2.
+
+The queue carries notifications (log records, error records, lifecycle events)
+and command replies. A thread waiting for a reply reads the queue itself, so
+the interrupt is only the trigger to drain the queue (see "Draining the
+GSP-to-CPU queue").
+
+The falcon latches every cause that it raises, SWGEN0 among them, in its
+``IRQSTAT`` register. The handler services the host-routed causes and clears
+their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon re-emits
+any cause that latched in the meantime. "Retriggering a falcon" has the details.
+
+Draining the queue takes the command-queue mutex and walks shared memory, so it
+cannot run in hard interrupt context. nova-core registers a threaded handler,
+under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard
+interrupt context and reads and writes only registers, and it wakes the IRQ
+thread to drain the queue::
+
+    GSP writes messages into the GSP-to-CPU queue
+    GSP raises SWGEN0
+    GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending
+    PCI interrupt -> Linux IRQ -> top half, in hard interrupt context:
+        read LEAF(4), and if bit 27 is clear, rearm and return
+        clear bit 27 (the subtree stays enabled)
+        read the falcon causes routed to the host, clearing SWGEN0 if set
+        for every other host cause: log it, clear its latch, and read the
+            host causes back
+        if the clear ended all of them: retrigger the falcon
+        otherwise: disable vector 155 at its leaf and skip the retrigger
+        rearm PCI interrupt delivery
+        wake the IRQ thread if SWGEN0 was set
+    IRQ thread, which may sleep:
+        take the command-queue mutex and drain the GSP-to-CPU queue
+
+A halt and a posted message can be pending together, so the top half services
+every cause that the status reports.
+
+A drain fails when a message's framing or checksum is bad (see "Draining the
+GSP-to-CPU queue"). The message stays at the queue head, so every later drain
+would fail the same way, and the IRQ thread disables vector 155 and logs the
+failure, which leaves the queue unserviced until the device is reset.
+
+The handler, the self-test, and the rest of the driver read BAR0 through one
+shared mapping. nova-core unregisters an interrupt handler when the device
+unbinds, so a handler runs only while the mapping exists.
+
+Retriggering a falcon
+---------------------
+
+A falcon signals the tree when its set of host-routed causes goes from empty to
+non-empty. A cause left latched keeps the set non-empty, so no later cause
+signals the tree, and the vector is lost. For a cause that stays latched, the
+handler can clear the tree leaf first or the falcon latch first, and the loss
+is the same.
+
+``IRQSTAT`` latches every cause in the falcon, including the causes routed to
+the falcon's own RISC-V core and owned by the firmware running on it. A host
+handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and
+``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both
+before it reads or clears a cause. OpenRM computes the same intersection in
+``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing
+registers and GA102 moves them, so the offsets change at GA102 rather than at
+the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is
+read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on
+a RISC-V falcon.
+
+``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the
+tree, which supplies the transition that clearing the leaf lost. The handler
+writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it
+read, because a re-emitted cause that stays set arrives again at once and
+on every pass after that.
+
+``IRQSCLR`` ends a latch and does not end the source behind it, so a cause
+driven from outside the falcon stays set after the write. On Blackwell the
+fault-containment and ECC causes are driven that way: they appear in
+``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and
+``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler
+clears the latch of every host cause other than SWGEN0, reads the host causes
+back, and retriggers only when the read-back is empty. When a cause is still
+set, the handler disables vector 155 instead and reports that the device needs
+a reset. Disabling loses no notification: the cause that is still set holds the
+host-routed set non-empty, so the falcon would signal nothing further either
+way. OpenRM makes the same choice, and ``kgspService_TU102`` skips
+``kflcnIntrRetrigger`` once it has recorded a fatal error.
+
+The handler cannot distinguish a fault cause that arrives after the clear from
+one that the clear failed to end, so it disables the vector in that case too.
+Both mean that the GSP has faulted.
+
+Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create
+a transition that it has lost. It must leave no host cause latched: it reads the
+host-routed status once and takes every cause that the status reports, rather
+than stopping at the first one that it recognizes. One window stays open. A
+cause that arrives after the handler has read the status is not in the value
+that the handler clears, so it stays latched after the leaf has been cleared,
+and no later cause from that falcon signals the tree. OpenRM has the same window
+on Turing, where ``kflcnIntrRetrigger`` does nothing.
+
+Enabling the GSP event
+----------------------
+
+SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays
+set. nova-core's GSP boot code consumes the GSP's notifications by polling the
+queue, which leaves the latch set and leaves pending bits in the tree. The
+handoff from polling to interrupts has a required order::
+
+    disable every implemented vector    drop enables left by boot, or by a
+                                        driver that ran before this one
+    drain the tree                      clear stale pending bits
+    rearm PCI interrupt delivery        required under pre-Hopper MSI, which
+                                        no later step rearms
+    disable subtree 2 at TOP            a TOP-enable rearm enabled it again
+    clear the SWGEN0 latch              so the next message makes an edge
+    register the threaded handler       no delivery can reach it yet
+    enable vector 155 at LEAF(4)        the subtree is still disabled
+    enable subtree 2 at TOP             deliveries become possible here
+    drain the GSP-to-CPU queue          messages posted before the clear
+
+The first four steps are the tree reset. nova-core runs them, and clears the
+latch, before it registers the handler. Registering unmasks the PCI interrupt,
+and GIN would then deliver a vector that boot left enabled to a handler that
+services one vector and has no way to service any other. OpenRM clears every
+leaf enable at the same point for the same reason.
+
+nova-core clears the latch after the drain. If nova-core cleared the latch
+first, a message posted before the drain could set it again, along with bit 27
+of LEAF(4). The drain would then clear the leaf bit while the latch stays set,
+and no later message would signal the tree. Clearing after the drain can
+instead leave the leaf bit pending with the latch already clear. Enabling the
+subtree then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half
+clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue
+drain that follows reads the message.
+
+Clearing the latch makes the first interrupt possible. A message that the GSP
+posted before that clear produces no interrupt, so the sequence ends by
+draining the queue.
+
+The subtree is enabled at ``TOP`` only once the handler is registered. A
+TOP-enable rearm sets the enables that it cycles, so the reset disables the
+serviced subtrees again after its rearm, and the tree reaches the registration
+with subtree 2 disabled under every rearm method. On teardown nova-core disables
+the vector at its leaf, so that no vector in the subtree can be delivered, then
+calls ``free_irq()``, and disables the subtree last. Disabling the subtree
+earlier would let a handler still in flight enable it again through the
+``TOP_EN`` cycle of its rearm, which would leave the subtree enabled with no
+handler registered. The driver's ``Gpu`` object, which owns the tree, holds the
+subtree enable, and a handler's registration holds only the enable of its own
+vector, so that tearing down one handler does not disable a subtree that another
+handler shares. nova-core tears down the registration before it frees the queue
+that the handler drains and the falcon that the handler reads, and before it
+unloads the GSP.
+
+Draining the GSP-to-CPU queue
+-----------------------------
+
+The queue carries command replies and unsolicited events, and a message's
+function code says which it is.
+
+* A function code that matches the awaited reply: the message is decoded and
+  returned to the caller that sent the command.
+* Anything else is an event. An OS error record and a robust-channel record
+  are logged at error level, and an unrecognized function code at warning
+  level. The other known events (GSP logs, libos prints, assertion records,
+  lifecycle notices) need no action and get no line of their own, because the
+  receive trace at debug level already records every message's arrival with
+  its sequence number, function code, and length.
+
+The sequence number takes no part in the match, because the GSP does not echo
+the command's sequence number on every reply. On r570 the reply to
+``UnloadingGuestDriver`` carries sequence 0.
+
+The read pointer advances past every message that passes framing and checksum
+validation, whether it matched or was an event. A matched message that is too
+short to decode is the exception, and it stays at the queue head.
+
+A message's length is inside the region that the checksum covers, so once the
+framing or the checksum fails there is no trustworthy length with which to skip
+the message. Such a message also stays at the queue head, and every later
+receive fails on it with ``EIO`` until the device is reset.
+
+The polling path and the IRQ thread both read the queue under the command-queue
+mutex. Replies and events share one queue and one read pointer, so one lock is
+held across the whole drain. A thread waiting for a reply logs each event that
+arrives before the reply and keeps waiting. One deadline of 5 seconds applies to
+the whole wait, rather than a fresh timeout after each message, and the thread
+holds the mutex for the whole wait, so that no other caller consumes the message
+that it waits for.
+
+With one lock, a drain waits for an in-flight command's receive to finish or
+time out. For log and error records that delay does not matter.
+
+Self-test
+=========
+
+The self-test confirms that an interrupt injected at the GPU is delivered to a
+registered handler. It runs during probe, after the GPU's boot firmware, GFW,
+has completed, and before nova-core boots the GSP, because the test disables
+and drains the whole tree, including the GSP's leaf. The test is built only
+under ``CONFIG_NOVA_CORE_SELFTESTS``, like the other probe-time hardware tests.
+Those other tests log a failure and let probe continue. A failed delivery test
+fails probe, because an interrupt path that does not work leaves the driver
+unable to make progress later, in a place that says nothing about the cause.
+
+The test writes ``LEAF_TRIGGER`` with vector 129, the CPU doorbell, at leaf 4
+bit 1. The vector then takes the ordinary path to the CPU under the ordinary
+enables. The doorbell keeps the same number on every supported chipset, so the
+test names it without asking the GSP, which is not running yet.
+
+The test allocates PCI vectors for subtree 2 and resets the tree, which
+disables every vector, drains the tree, and rearms delivery. It checks that the
+doorbell bit starts out clear. It registers a non-threaded handler with a
+completion, under the name ``nova-core-selftest`` in ``/proc/interrupts``, and
+enables the vector and the subtree. It triggers the doorbell, waits up to 1000
+ms for the first delivery, triggers it again, and waits up to 1000 ms for the
+second. The handler takes the notification path: it clears only its own bit and
+rearms, and never walks the tree.
+
+The test triggers twice, and the second trigger waits for the first handler to
+finish. One delivery would prove nothing about the rearm, because the first
+message-signaled interrupt arrives whether the driver rearms or not, and two
+triggers in a row could coalesce into one delivery. A handler that walked the
+tree would prove nothing either: on every configuration except pre-Hopper MSI
+the rearm is a ``TOP_EN`` cycle, so a walk that enabled ``TOP`` again would
+rearm delivery whether the handler asked for it or not.
+
+The test passes only if both deliveries arrive, each finds the doorbell bit and
+nothing else pending in leaf 4, and the bit is clear once the vector is
+disabled. Anything else fails probe. Requiring the exact pending bits on the
+second delivery shows that the first handler's clear reached the hardware. No
+other vector in leaf 4 can be pending, because the test disabled every vector
+and runs before GSP boot.
+
+The test releases its PCI vectors and its handler before returning, so the
+driver's own allocation covers the GSP subtree and nothing else. Sharing the
+driver's allocation would have let the test pass only because the doorbell and
+the GSP event happen to be in the same subtree.
+
+The test exercises the path from the GPU to the handler without GSP firmware,
+which helps when bringing up PCI, MSI, MSI-X, and passthrough setups. Under
+MSI-X a pass also shows that the delivery arrived on the entry belonging to the
+serviced subtree.
+
+The vector arithmetic and the leaf and subtree sets depend on no hardware, and
+KUnit tests cover them instead.
+
+Register naming
+===============
+
+nova-core uses the ``NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_*`` names for the CPU
+tree on every supported chipset. That is the per-function aperture: each PCIe
+function reaches its own tree through it, at the same offsets. The controller
+also has a central aperture that exposes every function's tree. The pre-Hopper
+headers name it ``NV_CTRL_CPU_INTR_*`` and the Hopper-plus headers
+``NV_GIN_CPU_INTR_*``. nova-core does not use it. OpenRM's kernel-side code
+for the controller is the ``Intr`` object.
+
+Scope
+=====
+
+nova-core services the CPU tree of one function and nothing else. It implements
+no virtual-function tree management and no GFID routing, which belong to the
+physical function's driver or to firmware in a virtualized setup, and no MIG
+(multi-instance GPU) support. It services none of the subtrees that the
+hardware headers reserve for the stall interrupts of the host-driven engines.
Documentation/gpu/nova/index.rst +1 -0
index 59b206238498..224caef9ea42 100644
@@ -35,3 +35,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/falcon
    core/tlv
    core/pramin
+   core/interrupts
Signature · 1 line
2.55.0
#29 lore raw

Re: [PATCH v5 15/15] gpu: nova-core: document the GIN interrupt controller and GSP events

Quoted 11 lines · John Hubbard

The interrupt code rests on hardware behavior that the code cannot show on its own: how GIN, the GPU's interrupt controller, records and delivers interrupts, what edge-triggered delivery requires of a handler, and how the GSP signals the CPU. Some of those requirements come from OpenRM rather than from the hardware manuals.

Add a design document that records that behavior, the rules that nova-core follows because of it, and the terms that the code uses for it. The code comments cite the document by section rather than repeating it.

Quoted 6 lines
Assisted-by LLM
---
 Documentation/gpu/nova/core/interrupts.rst | 679 +++++++++++++++++++++
 Documentation/gpu/nova/index.rst           |   1 +
 2 files changed, 680 insertions(+)

I took the freedom to remove the last 3 paragraphs because they read like filler and didn't provide any useful information. The self-test in particular should be self-contained (and mostly is) and describing it at length here carries the risk that the code will eventually diverge from the doc.

But the rest was informative. Maybe we can give it another pass later to smoothen it further, but even as-is the document is helpful imho.

maintainer Acked-by on the v5 series
#30 lore raw
In reply to John Hubbard
Quoted 5 lines · John Hubbard

This series adds GPU interrupt support to nova-core, so that GSP events reach the driver as interrupts instead of only when the driver polls. v5 applies Alex's review of v4, and makes three of the design changes that he had set aside for after the merge. The series is 15 patches, down from 17. The two rust/ patches are unchanged.

With patch 7 and 8 adjusted accordingly,

#31 lore raw
In reply to John Hubbard
Quoted 19 lines · John Hubbard

This series adds GPU interrupt support to nova-core, so that GSP events reach the driver as interrupts instead of only when the driver polls. v5 applies Alex's review of v4, and makes three of the design changes that he had set aside for after the merge. The series is 15 patches, down from 17. The two rust/ patches are unchanged.

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller, and it latches pending interrupts in a two-level register tree. TOP, the tree's top-level enable register, holds one enable bit per subtree.

The design uses a threaded IRQ handler. The top half reads and writes only GPU registers, while the threaded bottom half drains the message queue.

Still TODO after this: retrieve the dynamic interrupt routing tables from GSP, and use those, instead of the simple, hard-coded approach that was done here for these first two interrupts.

Quoted 7 lines

The series is based on drm-rust-next at commit 658c5f0042e6 ("gpu:
nova-core: use a single try_pin_init!() block in probe()"), and it carries no prerequisite commits.

The patches, as applied to drm-rust-next, are on a git branch:

https://github.com/johnhubbard/linux/tree/nova-core-gin-interrupt-tree-v5/

This revision is a quite significant improvement, and although there are still a few things I want to fix before applying (notably on - guess what - the HAL :)), this is in the drive-by-upon-apply range. I'll leave a few review comments for the sake of completeness, but there should be no need to send a new revision, unless I missed something important on my first pass.

maintainer Applied
#32 lore raw
In reply to John Hubbard
Quoted 6 lines · John Hubbard

This series adds GPU interrupt support to nova-core, so that GSP events reach the driver as interrupts instead of only when the driver polls. v5 applies Alex's review of v4, and makes three of the design changes that he had set aside for after the merge. The series is 15 patches, down from 17. The two rust/ patches are unchanged.

Quoted 6 lines

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller, and it latches pending interrupts in a two-level register tree. TOP, the tree's top-level enable register, holds one enable bit per subtree.

[...]

Applied to drm-rust-next, thanks!

[01/15] rust: pci: declare IrqType and IrqTypes with impl_flags

        commit: 38c9764d330dfe1d59009179904e313b0923f7b4
[03/15] gpu: nova-core: add the GIN vector, leaf and subtree types
        commit: e6c579b496f2740c447d215ef868dc5f9cc7dc93

[04/15] gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers
[acourbot: keep item that doesn't need to be exported private.]
commit: 2ef55e142d29970641e9be04897457bb98a8eaca
[05/15] gpu: nova-core: add the per-architecture GIN CPU interrupt HAL

        commit: 4022bcaa34e2115e175f6a64022ec19a186b0d72
[06/15] gpu: nova-core: add the GIN interrupt tree and allocate its vectors
        commit: 939b40877cb883de1fa335e950ff1a3f3bf113ed
[07/15] gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
        [acourbot: move code to associated function, move relevant hunks to next
        patch.]
        commit: 94ddc4b4843da8b1858535b13318346cb950800d

[08/15] gpu: nova-core: add an interrupt delivery self-test
[acourbot: move relevant bits of previous patch into this one.] [acourbot: remove reference to self-test section of documentation.] [acourbot: replace completions with atomic polling code.]
commit: 9cb3f5d56e81fd97a82ef361b210bc2322daaaa7
[09/15] gpu: nova-core: log GSP events instead of discarding them

        commit: 0c551cdc465b9c99da33a21d6ee3b274cfda7006
[10/15] gpu: nova-core: return ENOMSG for an unmatched GSP message
        commit: d3cb3ed1a2ddd7cb77d9d270d82d3177ee744d1f
[11/15] gpu: nova-core: bound a GSP wait by a single deadline
        commit: 621b455c56376f5469d896b58c2316236a3f939d

[12/15] gpu: nova-core: add the falcon interrupt registers and HAL methods
[acourbot: remove implementation-specific comments on interface doc.] [acourbot: make constructor take boolean instead of having two methods.] [acourbot: add NVIDIA copyright headers to modified files.]
commit: e727d1c382a0cb2e10fd7aabef9217d1a37c80c7
[13/15] gpu: nova-core: service GSP events from the SWGEN0 interrupt
[acourbot: trim comment leaking unneeded implementation details.] [acourbot: make Subtree crate-visible here instead of in earlier patch.]
commit: 476abf9f669d87f993429ebb42668169464c907c
[14/15] gpu: nova-core: add KUnit tests for the interrupt tree

        commit: 306279ce974d396136544aa88027ee921ac09540

[15/15] gpu: nova-core: document the GIN interrupt controller and GSP events
[acourbot: cut the last 3 sections.]
commit: df718311a84188e26d499301c3921e4c392d557e