Skip to content

nova-core: GPU interrupt support and GSP event delivery

John Hubbard lkml nova-gpu 5 versions – Applied

Status from this message: Applied to drm-rust-next, thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest15 patches 16 repliesApplied
  2. v417 patches 10 repliesSuperseded
  3. v314 patches 18 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v215 patches22 repliesSuperseded
v117 patches21 repliesSuperseded

Changes in v5 from the cover letter · 16 lines

* Dropped the patch that poisoned the message queue after a framing or checksum failure. (Alex)

* The GSP interrupt handler borrows the GSP falcon and the command queue instead of holding a BAR and a chipset. (Alex)

* The per-chipset falcon interrupt operations are methods of the existing falcon HAL rather than of a separate falcon interrupt HAL. (Alex)

* The PCI vector allocation and the interrupt tree are one type, which the Gpu owns and the GSP handler borrows. (Alex)

* The tree reset is a method of the tree, and it disables the serviced subtrees at TOP last. (Alex)

* The Gpu holds the TOP enable of the GSP subtree, and the GSP handler's registration holds only the leaf enable of its own vector. (Alex)

* The interrupt tree's trigger and the message queue's drain are added by the patches that first call them, so the separate message queue drain patch is dropped. (Alex)

* The two KUnit suites that tested the HAL dispatchers are removed, and the tree suite's chipset check covers every chipset. (Alex)

* Smaller items from the same review (Alex):

* repr(u32) on the leaf count, and an if/else in the vector check.

* The interrupt HAL performs the rearm, and the tree holds the HAL instead of a leaf count and a rearm method.

* The GFW comment restored at the wait in probe, where GFW is the GPU's boot firmware, and one import group in probe.

* The self-test handler's clear and rearm grouped ahead of the count, and the superfluous dev local removed.

* A comment at the MSI fallback in the vector allocation.

* "chipset" for "part", and "OpenRM" for "Open RM".

* The falcon HAL patch's message no longer claims that a HAL call allocates, since there is no separate HAL left to justify.

Patches in v5 Applied

#Subject and filesSizeRepliesApplied
1/15rust: pci: declare IrqType and IrqTypes with impl_flagsrust/kernel/pci/irq.rs+21 -471 reply
2/15rust: sync: completion: add wait_for_completion_timeout()rust/kernel/sync/completion.rs+22 -19 replies
3/15gpu: nova-core: add the GIN vector, leaf and subtree typesdrivers/gpu/nova-core/irq.rs drivers/gpu/nova-core/irq/interrupt_tree.rs drivers/gpu/nova-core/nova_core.rs+252 -00 replies
4/15gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registersdrivers/gpu/nova-core/irq.rs drivers/gpu/nova-core/irq/interrupt_tree.rs drivers/gpu/nova-core/irq/regs.rs+103 -00 replies
5/15gpu: nova-core: add the per-architecture GIN CPU interrupt HALdrivers/gpu/nova-core/irq.rs drivers/gpu/nova-core/irq/hal.rs drivers/gpu/nova-core/irq/hal/gh100.rs +1 more+163 -00 replies
6/15gpu: nova-core: add the GIN interrupt tree and allocate its vectorsdrivers/gpu/nova-core/irq.rs drivers/gpu/nova-core/irq/interrupt_tree.rs+293 -70 replies
7/15gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructordrivers/gpu/nova-core/driver.rs drivers/gpu/nova-core/gpu.rs+32 -82 replies
8/15gpu: nova-core: add an interrupt delivery self-testdrivers/gpu/nova-core/Kconfig drivers/gpu/nova-core/driver.rs drivers/gpu/nova-core/irq.rs +3 more+285 -10 replies
9/15gpu: nova-core: log GSP events instead of discarding themdrivers/gpu/nova-core/gsp/cmdq.rs+41 -110 replies
10/15gpu: nova-core: return ENOMSG for an unmatched GSP messagedrivers/gpu/nova-core/gsp/cmdq.rs drivers/gpu/nova-core/gsp/commands.rs drivers/gpu/nova-core/gsp/sequencer.rs+5 -50 replies
11/15gpu: nova-core: bound a GSP wait by a single deadlinedrivers/gpu/nova-core/gsp/cmdq.rs drivers/gpu/nova-core/gsp/commands.rs drivers/gpu/nova-core/gsp/sequencer.rs+59 -290 replies
12/15gpu: nova-core: add the falcon interrupt registers and HAL methodsdrivers/gpu/nova-core/falcon/hal.rs drivers/gpu/nova-core/falcon/hal/ga102.rs drivers/gpu/nova-core/falcon/hal/tu102.rs +1 more+154 -30 replies
13/15gpu: nova-core: service GSP events from the SWGEN0 interruptdrivers/gpu/nova-core/falcon/gsp.rs drivers/gpu/nova-core/falcon/hal.rs drivers/gpu/nova-core/falcon/hal/tu102.rs +6 more+375 -130 replies
14/15gpu: nova-core: add KUnit tests for the interrupt treedrivers/gpu/nova-core/irq/interrupt_tree.rs+127 -10 replies
15/15gpu: nova-core: document the GIN interrupt controller and GSP eventsDocumentation/gpu/nova/core/interrupts.rst Documentation/gpu/nova/index.rst+680 -01 reply

Reviews trailers per patch

5 of 15 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchAlexandre CourbotWill PierceDanilo KrummrichAlice Ryhl
SeriesA-b
1/15R-b
2/15R-b A-b
3/15
4/15R-b
5/15R-b
6/15R-b
7/15
8/15
9/15
10/15
11/15
12/15
13/15
14/15
15/15

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v5 thread

Cover letter [PATCH v5 00/15] nova-core: GPU interrupt support and GSP event delivery

This series adds GPU interrupt support to nova-core, so that GSP events reach the driver as interrupts instead of only when the driver polls. v5 applies Alex's review of v4, and makes three of the design changes that he had set aside for after the merge. The series is 15 patches, down from 17. The two rust/ patches are unchanged.

GIN, the GPU Interrupt and Notification unit, is the GPU's interrupt controller, and it latches pending interrupts in a two-level register tree. TOP, the tree's top-level enable register, holds one enable bit per subtree.

The design uses a threaded IRQ handler. The top half reads and writes only GPU registers, while the threaded bottom half drains the message queue.

Still TODO after this: retrieve the dynamic interrupt routing tables from GSP, and use those, instead of the simple, hard-coded approach that was done here for these first two interrupts.

The series is based on drm-rust-next at commit 658c5f0042e6 ("gpu:
nova-core: use a single try_pin_init!() block in probe()"), and it carries no prerequisite commits.

The patches, as applied to drm-rust-next, are on a git branch:

https://github.com/johnhubbard/linux/tree/nova-core-gin-interrupt-tree-v5/

Changes in v5:

* Dropped the patch that poisoned the message queue after a framing or checksum failure. (Alex)

* The GSP interrupt handler borrows the GSP falcon and the command queue instead of holding a BAR and a chipset. (Alex)

* The per-chipset falcon interrupt operations are methods of the existing falcon HAL rather than of a separate falcon interrupt HAL. (Alex)

* The PCI vector allocation and the interrupt tree are one type, which the Gpu owns and the GSP handler borrows. (Alex)

* The tree reset is a method of the tree, and it disables the serviced subtrees at TOP last. (Alex)

* The Gpu holds the TOP enable of the GSP subtree, and the GSP handler's registration holds only the leaf enable of its own vector. (Alex)

* The interrupt tree's trigger and the message queue's drain are added by the patches that first call them, so the separate message queue drain patch is dropped. (Alex)

* The two KUnit suites that tested the HAL dispatchers are removed, and the tree suite's chipset check covers every chipset. (Alex)

* Smaller items from the same review (Alex):

* repr(u32) on the leaf count, and an if/else in the vector check.

* The interrupt HAL performs the rearm, and the tree holds the HAL instead of a leaf count and a rearm method.

* The GFW comment restored at the wait in probe, where GFW is the GPU's boot firmware, and one import group in probe.

* The self-test handler's clear and rearm grouped ahead of the count, and the superfluous dev local removed.

* A comment at the MSI fallback in the vector allocation.

* "chipset" for "part", and "OpenRM" for "Open RM".

* The falcon HAL patch's message no longer claims that a HAL call allocates, since there is no separate HAL left to justify.

TESTING: Tested on Turing (TU117), Ampere (GA104), and Blackwell (GB202), covering probe and chipset identification, the interrupt self-test, a GSP name query over the command queue, driver unload and reload, and the KUnit suites. No test produced a GSP fault on any of the three chips, so the fault path is untested.

Joel Fernandes (2):
rust: sync: completion: add wait_for_completion_timeout()
gpu: nova-core: add the GIN interrupt tree and allocate its vectors

John Hubbard (13):
rust: pci: declare IrqType and IrqTypes with impl_flags
gpu: nova-core: add the GIN vector, leaf and subtree types
gpu: nova-core: add the GIN CPU interrupt tree and MSI EOI registers
gpu: nova-core: add the per-architecture GIN CPU interrupt HAL
gpu: nova-core: wait for GFW boot in probe, not in the Gpu constructor
gpu: nova-core: add an interrupt delivery self-test
gpu: nova-core: log GSP events instead of discarding them
gpu: nova-core: return ENOMSG for an unmatched GSP message
gpu: nova-core: bound a GSP wait by a single deadline
gpu: nova-core: add the falcon interrupt registers and HAL methods
gpu: nova-core: service GSP events from the SWGEN0 interrupt
gpu: nova-core: add KUnit tests for the interrupt tree
gpu: nova-core: document the GIN interrupt controller and GSP events

 Documentation/gpu/nova/core/interrupts.rst  | 679 +++++++++++++++++++
 Documentation/gpu/nova/index.rst            |   1 +
 drivers/gpu/nova-core/Kconfig               |   5 +
 drivers/gpu/nova-core/driver.rs             |  19 +-
 drivers/gpu/nova-core/falcon/gsp.rs         |  47 +-
 drivers/gpu/nova-core/falcon/hal.rs         |  19 +-
 drivers/gpu/nova-core/falcon/hal/ga102.rs   |  16 +
 drivers/gpu/nova-core/falcon/hal/tu102.rs   |  49 +-
 drivers/gpu/nova-core/gpu.rs                | 122 +++-
 drivers/gpu/nova-core/gsp/cmdq.rs           | 166 ++++-
 drivers/gpu/nova-core/gsp/commands.rs       |   8 +-
 drivers/gpu/nova-core/gsp/sequencer.rs      |   8 +-
 drivers/gpu/nova-core/irq.rs                |  29 +
 drivers/gpu/nova-core/irq/doorbell_test.rs  | 257 ++++++++
 drivers/gpu/nova-core/irq/gsp.rs            | 196 ++++++
 drivers/gpu/nova-core/irq/hal.rs            |  67 ++
 drivers/gpu/nova-core/irq/hal/gh100.rs      |  40 ++
 drivers/gpu/nova-core/irq/hal/tu102.rs      |  43 ++
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 682 ++++++++++++++++++++
 drivers/gpu/nova-core/irq/regs.rs           |  88 +++
 drivers/gpu/nova-core/nova_core.rs          |   1 +
 drivers/gpu/nova-core/regs.rs               |  69 ++
 rust/kernel/pci/irq.rs                      |  68 +-
 rust/kernel/sync/completion.rs              |  23 +-
 24 files changed, 2594 insertions(+), 108 deletions(-)
 create mode 100644 Documentation/gpu/nova/core/interrupts.rst
 create mode 100644 drivers/gpu/nova-core/irq.rs
 create mode 100644 drivers/gpu/nova-core/irq/doorbell_test.rs
 create mode 100644 drivers/gpu/nova-core/irq/gsp.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/gh100.rs
 create mode 100644 drivers/gpu/nova-core/irq/hal/tu102.rs
 create mode 100644 drivers/gpu/nova-core/irq/interrupt_tree.rs
 create mode 100644 drivers/gpu/nova-core/irq/regs.rs

base-commit: 658c5f0042e6e897338dd04330a363aa7022c85d

Signature · 1 line
2.55.0