gpu: nova-core: add initial driver stub
Subsystems (8)
Danilo Krummrich
maintainerPATCH v2 1/2 gpu: nova-core: add initial driver stub
Add the initial nova-core driver stub.
nova-core is intended to serve as a common base for nova-drm (the corresponding DRM driver) and the vGPU manager VFIO driver, serving as a hard- and firmware abstraction layer for GSP-based NVIDIA GPUs.
The Nova project, including nova-core and nova-drm, in the long term, is intended to serve as the successor of Nouveau for all GSP-based GPUs.
The motivation for both, starting a successor project for Nouveau and doing so using the Rust programming language, is documented in detail through a previous post on the mailing list [1], an LWN article [2] and a talk from LPC '24.
In order to avoid the chicken and egg problem to require a user to upstream Rust abstractions, but at the same time require the Rust abstractions to implement the driver, nova-core kicks off as a driver stub and is subsequently developed upstream.
---
Changes in v2:
- Fix module name in Kconfig description. (John)
- Expand Kconfig description a bit. (John)
- Expand name for PCI BAR0 region.
- Do not store / print boot0 raw register value. (John)
- Rename CardType to Architecture, rename enum names to represent the architecture name and adjust enum values according to the register definition. (John)
- Add an abstraction for register accesses.
- Print chipset, architecture and revision.
- Load bootloader firmware. (Timur)
--- MAINTAINERS | 10 ++ drivers/gpu/Makefile | 1 + drivers/gpu/nova-core/Kconfig | 14 +++ drivers/gpu/nova-core/Makefile | 3 + drivers/gpu/nova-core/driver.rs | 47 ++++++++ drivers/gpu/nova-core/gpu.rs | 187 +++++++++++++++++++++++++++++ drivers/gpu/nova-core/nova_core.rs | 15 +++ drivers/gpu/nova-core/regs.rs | 60 +++++++++ drivers/video/Kconfig | 1 + 9 files changed, 338 insertions(+) create mode 100644 drivers/gpu/nova-core/Kconfig create mode 100644 drivers/gpu/nova-core/Makefile create mode 100644 drivers/gpu/nova-core/driver.rs create mode 100644 drivers/gpu/nova-core/gpu.rs create mode 100644 drivers/gpu/nova-core/nova_core.rs create mode 100644 drivers/gpu/nova-core/regs.rs
MAINTAINERS +10 -0
@@ -7446,6 +7446,16 @@ T: git https://gitlab.freedesktop.org/drm/nouveau.git F: drivers/gpu/drm/nouveau/ F: include/uapi/drm/nouveau_drm.h +CORE DRIVER FOR NVIDIA GPUS [RUST] +M: Danilo Krummrich <[email protected]> +L: [email protected] +S: Supported +Q: https://patchwork.freedesktop.org/project/nouveau/ +B: https://gitlab.freedesktop.org/drm/nova/-/issues +C: irc://irc.oftc.net/nouveau +T: git https://gitlab.freedesktop.org/drm/nova.git nova-next +F: drivers/gpu/nova-core/ + DRM DRIVER FOR OLIMEX LCD-OLINUXINO PANELS M: Stefan Mavrodiev <[email protected]> S: Maintained
drivers/gpu/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-y += host1x/ drm/ vga/ obj-$(CONFIG_IMX_IPUV3_CORE) += ipu-v3/ obj-$(CONFIG_TRACE_GPU_MEM) += trace/ +obj-$(CONFIG_NOVA_CORE) += nova-core/
drivers/gpu/nova-core/Kconfig new file +14 -0
@@ -0,0 +1,14 @@ +config NOVA_CORE + tristate "Nova Core GPU driver" + depends on PCI + depends on RUST + depends on RUST_FW_LOADER_ABSTRACTIONS + default n + help + Choose this if you want to build the Nova Core driver for Nvidia + GPUs based on the GPU System Processor (GSP). This is true for Turing + and later GPUs. + + This driver is work in progress and may not be functional. + + If M is selected, the module will be called nova_core.
drivers/gpu/nova-core/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_NOVA_CORE) += nova_core.o
drivers/gpu/nova-core/driver.rs new file +47 -0
47 lines not on this page · raw message
drivers/gpu/nova-core/gpu.rs new file +187 -0
187 lines not on this page · raw message
drivers/gpu/nova-core/nova_core.rs new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Nova Core GPU Driver + +mod driver; +mod gpu; +mod regs; + +kernel::module_pci_driver! { +type : driver::NovaCore , + name:"NovaCore" , + author:"Danilo Krummrich" , + description:"Nova Core GPU driver" , + license:"GPL v2" , +}
drivers/gpu/nova-core/regs.rs new file +60 -0
60 lines not on this page · raw message
drivers/video/Kconfig +1 -0
@@ -39,6 +39,7 @@ source "drivers/gpu/vga/Kconfig" source "drivers/gpu/host1x/Kconfig" source "drivers/gpu/ipu-v3/Kconfig" +source "drivers/gpu/nova-core/Kconfig" source "drivers/gpu/drm/Kconfig"
base-commit: 69b8923f5003664e3ffef102e73333edfa2abdcf
Signature · 1 line
2.48.1
Danilo Krummrich
maintainerPATCH v2 2/2 gpu: nova-core: add initial documentation
Add the initial documentation of the Nova project.
The initial project documentation consists out of a brief introduction of the project, as well as project guidelines both general and nova-core specific and a task list for nova-core specifically.
The task list is divided into tasks for general Rust infrastructure required by the project, tasks regarding GSP enablement and firmware abstraction, general GPU driver tasks as well as tasks related to external API design and test infrastructure.
---
- Add task "Generic register abstraction".
- Change complexity of "Debugfs abstractions".
--- Documentation/gpu/drivers.rst | 1 + Documentation/gpu/nova/core/guidelines.rst | 24 ++ Documentation/gpu/nova/core/todo.rst | 445 +++++++++++++++++++++ Documentation/gpu/nova/guidelines.rst | 73 ++++ Documentation/gpu/nova/index.rst | 30 ++ MAINTAINERS | 1 + 6 files changed, 574 insertions(+) create mode 100644 Documentation/gpu/nova/core/guidelines.rst create mode 100644 Documentation/gpu/nova/core/todo.rst create mode 100644 Documentation/gpu/nova/guidelines.rst create mode 100644 Documentation/gpu/nova/index.rst
Documentation/gpu/drivers.rst +1 -0
@@ -24,6 +24,7 @@ GPU Driver Documentation panfrost panthor zynqmp + nova/index .. only:: subproject and html
Documentation/gpu/nova/core/guidelines.rst new file +24 -0
@@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========== +Guidelines +========== + +This documents contains the guidelines for nova-core. Additionally, all common +guidelines of the Nova project do apply. + +Driver API +========== + +One main purpose of nova-core is to implement the abstraction around the +firmware interface of GSP and provide a firmware (version) independent API for +2nd level drivers, such as nova-drm or the vGPU manager VFIO driver. + +Therefore, it is not permitted to leak firmware (version) specifics, through the +driver API, to 2nd level drivers. + +Acceptance Criteria +=================== + +- To the extend possible, patches submitted to nova-core must be tested for + regressions with all 2nd level drivers.
Documentation/gpu/nova/core/todo.rst new file +445 -0
445 lines not on this page · raw message
Documentation/gpu/nova/guidelines.rst new file +73 -0
73 lines not on this page · raw message
Documentation/gpu/nova/index.rst new file +30 -0
@@ -0,0 +1,30 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +======================= +nova NVIDIA GPU drivers +======================= + +The nova driver project consists out of two separate drivers nova-core and +nova-drm and intends to supersede the nouveau driver for NVIDIA GPUs based on +the GPU System Processor (GSP). + +The following documents apply to both nova-core and nova-drm. + +.. toctree:: + :titlesonly: + + guidelines + +nova-core +========= + +The nova-core driver is the core driver for NVIDIA GPUs based on GSP. nova-core, +as the 1st level driver, provides an abstraction around the GPUs hard- and +firmware interfaces providing a common base for 2nd level drivers, such as the +vGPU manager VFIO driver and the nova-drm driver. + +.. toctree:: + :titlesonly: + + core/guidelines + core/todo
MAINTAINERS +1 -0
@@ -7454,6 +7454,7 @@ Q: https://patchwork.freedesktop.org/project/nouveau/ B: https://gitlab.freedesktop.org/drm/nova/-/issues C: irc://irc.oftc.net/nouveau T: git https://gitlab.freedesktop.org/drm/nova.git nova-next +F: Documentation/gpu/nova/ F: drivers/gpu/nova-core/ DRM DRIVER FOR OLIMEX LCD-OLINUXINO PANELS
Signature · 1 line
2.48.1
Re: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 82 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Add the initial documentation of the Nova project.
The initial project documentation consists out of a brief introduction of the project, as well as project guidelines both general and nova-core specific and a task list for nova-core specifically.
The task list is divided into tasks for general Rust infrastructure required by the project, tasks regarding GSP enablement and firmware abstraction, general GPU driver tasks as well as tasks related to external API design and test infrastructure.
Signed-off-by Danilo Krummrich <[email protected]>---
- Add task "Generic register abstraction".
- Change complexity of "Debugfs abstractions".--- Documentation/gpu/drivers.rst | 1 + Documentation/gpu/nova/core/guidelines.rst | 24 ++ Documentation/gpu/nova/core/todo.rst | 445 +++++++++++++++++++++ Documentation/gpu/nova/guidelines.rst | 73 ++++ Documentation/gpu/nova/index.rst | 30 ++ MAINTAINERS | 1 + 6 files changed, 574 insertions(+) create mode 100644 Documentation/gpu/nova/core/guidelines.rst create mode 100644 Documentation/gpu/nova/core/todo.rst create mode 100644 Documentation/gpu/nova/guidelines.rst create mode 100644 Documentation/gpu/nova/index.rstDocumentation/gpu/drivers.rst +1
-0@@ -24,6 +24,7 @@ GPU Driver Documentation panfrost panthor zynqmp + nova/index .. only:: subproject and htmlDocumentation/gpu/nova/core/guidelines.rst new file +24
-0@@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========== +Guidelines +========== + +This documents contains the guidelines for nova-core. Additionally, all common +guidelines of the Nova project do apply. + +Driver API +========== + +One main purpose of nova-core is to implement the abstraction around the +firmware interface of GSP and provide a firmware (version) independent API for +2nd level drivers, such as nova-drm or the vGPU manager VFIO driver. + +Therefore, it is not permitted to leak firmware (version) specifics, through the +driver API, to 2nd level drivers. + +Acceptance Criteria +=================== + +- To the extend possible, patches submitted to nova-core must be tested for + regressions with all 2nd level drivers.Documentation/gpu/nova/core/todo.rst new file +6
-0@@ -0,0 +1,445 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========= +Task List +========= +
...
Quoted 7 lines
Documentation/gpu/nova/core/todo.rst new file +7
-0+ +Generic register abstraction +---------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. +
Documentation/gpu/nova/core/todo.rst line 59
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO), + MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev. + ]) +
I think it is better not to tie this to pci::Bar and its operations. It would be better to have a intermediate container as the macro param. The container holds the register region vaddr pointer, size, read/write traits. The macro expands it from there, thus, we can also use this on firmware memory structures, e.g. GSP WPR2 info.
Probably we are looking for a even more generic solution/type for deferring a structure in the vaddr and generating the the accessing methods accordingly. It might also be useful later in GSP message queue manipulation, ELF header extraction, page table manipulation? (to avoid ambitious unsafe statements in the rust driver)
Quoted 49 lines
Documentation/gpu/nova/core/todo.rst new file +49
-0+This could expand to something like: + +.. code-block:: rust + + const BOOT0_OFFSET: usize = 0x00000000; + const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; + const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; + const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; + const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; + const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; + const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; + + struct Boot0(u32); + + impl Boot0 { + #[inline] + fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { + Self(bar.readl(BOOT0_OFFSET)) + } + + #[inline] + fn minor_revision(&self) -> u32 { + (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT + } + + #[inline] + fn major_revision(&self) -> u32 { + (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT + } + + #[inline] + fn revision(&self) -> u32 { + (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT + } + } + +Usage: + +.. code-block:: rust + + let bar = bar.try_access().ok_or(ENXIO)?; + + let boot0 = Boot0::read(&bar); + pr_info!("Revision: {}\n", boot0.revision()); + +| Complexity: Advanced + +Delay / Sleep abstractions +--------------------------
...
Quoted 4 lines
Documentation/gpu/nova/core/todo.rst new file +4
-0+ +VRAM memory allocator +--------------------- +
Documentation/gpu/nova/core/todo.rst line 283
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+Investigate options for a VRAM memory allocator. + +Some possible options: + - Rust abstractions for + - RB tree (interval tree) / drm_mm + - maple_tree + - native Rust collections + +| Complexity: Advanced +
I am leaning towards having the abstractions at a high level APIs, e.g. wrapping drm_mm and possibly the rust side can choose the backend type of drm_mm if it really needs a different type of data structure other than default supported by drm_mm. If we need more type of data structures, we can extend drm_mm in C side. That can save us some efforts.
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+Instance Memory +--------------- + +Implement support for instmem (bar2) used to store page tables. + +| Complexity: Intermediate +| Contact: Dave Airlie + +GPU System Processor (GSP) +==========================
...
Quoted 16 lines
Documentation/gpu/nova/core/todo.rst new file +16
-0+ +External APIs +============= + +nova-core base API +------------------ + +Work out the common pieces of the API to connect 2nd level drivers, i.e. vGPU +manager and nova-drm. + +| Complexity: Advanced + +vGPU manager API +---------------- + +Work out the API parts required by the vGPU manager, which are not covered by
Documentation/gpu/nova/core/todo.rst line 428
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+the base API. + +| Complexity: Advanced + +nova-core C API +--------------- + +Implement a C wrapper for the APIs required by the vGPU manager driver. + +| Complexity: Intermediate
Thanks for calling this out.
I believe the "vGPU manager API" is not a standalone task, as many of the required APIs will intersect with other components in nova-core.
As one of nova-core’s users, vGPU represents the simplest use case to get started with, offering significant value to both nova-core and its users in the near term.
I was thinking that if we could align with the folks on making vGPU + nova-core our initial short-term goal, it would be beneficial for nova-drm's development, since the APIs required for nova-drm are a superset of those needed for vGPU.
It would be valuable for us to be involved in key areas related to vGPU, including:
- Task review
- Design discussions
- Unit testing
Additionally, we are working on a vGPU requirements document that will outline the complete API needs for vGPU beyond those already covered in the RFC patches. Hope that will be published soon.
Quoted 23 lines
Documentation/gpu/nova/core/todo.rst new file +17
-0+ +Testing +======= + +CI pipeline +----------- + +Investigate option for continuous integration testing. + +This can go from as simple as running KUnit tests over running (graphics) CTS to +booting up (multiple) guest VMs to test VFIO use-cases. + +It might also be worth to consider the introduction of a new test suite directly +sitting on top of the uAPI for more targeted testing and debugging. There may be +options for collaboration / shared code with the Mesa project. + +| Complexity: AdvancedDocumentation/gpu/nova/guidelines.rst new file +0
-0
Documentation/gpu/nova/guidelines.rst line 2
Quoted 2 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0@@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) +
I think this will develop into a maintainer handbook in the future which reflects maintainer's requirements, thoughts, tips...maybe we can make it explicit? I think it is rules of book that we agree to follow.
A similar one can be found here.
https://lore.kernel.org/kvm/[email protected]/
Quoted 12 lines
Documentation/gpu/nova/guidelines.rst new file +12
-0+========== +Guidelines +========== + +This document describes the general project guidelines that apply to nova-core +and nova-drm. + +Language +======== + +The Nova project uses the Rust programming language. In this context, the +following rules apply.
Documentation/gpu/nova/guidelines.rst line 24
Quoted 10 lines
Documentation/gpu/nova/guidelines.rst new file +10
-0+ +- Unless technically necessary otherwise (e.g. uAPI), any driver code is written + in Rust. + +- Direct FFI calls to C kernel APIs must be avoided; instead C kernel APIs + should be accessed through shared Rust abstractions. + +- Unless technically necessary, unsafe Rust code must be avoided. In case of + technical necessity, unsafe code should be isolated in a separate component + providing a safe API for other driver code to use.
Also need to comment why the unsafe is the last possible approach to go. the last thing we want to see is "unsafe" flying here and there in a rust driver. :)
Documentation/gpu/nova/guidelines.rst line 37
Quoted 13 lines
Documentation/gpu/nova/guidelines.rst new file +13
-0+ +Style +----- + +All rules of the Rust for Linux project as documented in +:doc:`../../rust/coding-guidelines` apply. Additionally, the following rules +apply. + +- Code must be formatted with the ``rustfmt`` make target. + +- Code submitted for inclusion into the Nova driver project must pass the Rust + linter, which can be enabled with ``CLIPPY=1``. +
It would be also helpful to make the process explicit. E.g. sharing your command lines used to checking the patches. So folks can align with the expected outcome, e.g. command line parameters.
Z.
Quoted 84 lines
Documentation/gpu/nova/guidelines.rst new file +36
-0+Documentation +============= + +The availability of proper documentation is essential in terms of scalability, +accessability for new contributors and maintainability of a project in general, +but especially for a driver running as complex hardware as Nova is targeting. + +Hence, adding documentation of any kind is very much encouraged by the project. + +Besides that, there are some minimum requirements. + +- Every non-private structure needs at least a brief doc comment explaining the + semantical sense of the structure, as well as potential locking and lifetime + requirements. It is encouraged to have the same minimum documentation for + non-trivial private structures. + +- uAPIs must be fully documented with kernel-doc comments; additionally, the + semantical behavior must be explained including potential special or corner + cases. + +- The APIs connecting the 1st level driver (nova-core) with 2nd level drivers + must be fully documented. This includes doc comments, potential locking and + lifetime requirements, as well as example code if applicable. + +- Abbreviations must be explained when introduced; terminology must be uniquely + defined. + +- Register addresses, layouts, shift values and masks must be defined properly; + unless obvious, the semantical sense must be documented. This only applies if + the author is able to obtain the corresponding information. + +Acceptance Criteria +=================== + +- Patches must only be applied if reviewed by at least one other person on the + mailing list; this also applies for maintainers.Documentation/gpu/nova/index.rst new file +30
-0@@ -0,0 +1,30 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +======================= +nova NVIDIA GPU drivers +======================= + +The nova driver project consists out of two separate drivers nova-core and +nova-drm and intends to supersede the nouveau driver for NVIDIA GPUs based on +the GPU System Processor (GSP). + +The following documents apply to both nova-core and nova-drm. + +.. toctree:: + :titlesonly: + + guidelines + +nova-core +========= + +The nova-core driver is the core driver for NVIDIA GPUs based on GSP. nova-core, +as the 1st level driver, provides an abstraction around the GPUs hard- and +firmware interfaces providing a common base for 2nd level drivers, such as the +vGPU manager VFIO driver and the nova-drm driver. + +.. toctree:: + :titlesonly: + + core/guidelines + core/todoMAINTAINERS +1
-0@@ -7454,6 +7454,7 @@ Q: https://patchwork.freedesktop.org/project/nouveau/ B: https://gitlab.freedesktop.org/drm/nova/-/issues C: irc://irc.oftc.net/nouveau T: git https://gitlab.freedesktop.org/drm/nova.git nova-next +F: Documentation/gpu/nova/ F: drivers/gpu/nova-core/ DRM DRIVER FOR OLIMEX LCD-OLINUXINO PANELS
Re: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 4 lines · Zhi Wang
It would be also helpful to make the process explicit. E.g. sharing your command lines used to checking the patches. So folks can align with the expected outcome, e.g. command line parameters.
These two guidelines (and generally the few others above) are intended to apply to all Rust code in the kernel (i.e. not just rust/) -- their command lines are mentioned in Documentation/rust/. We could add a note to make that clearer if that helps. So I would suggest avoiding repetition here by referencing those.
We also mention it in our "Subsystem Profile document" at
https://rust-for-linux.com/contributing#submit-checklist-addendum.
Quoted 2 lines
Documentation/gpu/nova/guidelines.rst line 42
Quoted 2 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0+The availability of proper documentation is essential in terms of scalability, +accessability for new contributors and maintainability of a project in general,
Typo: accessibility?
Cheers,
Miguel
Re: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 6 lines · On Wed, 5 Feb 2025 15:13:12 +0100 Miguel Ojeda…
Quoted 4 lines · Zhi Wang
It would be also helpful to make the process explicit. E.g. sharing your command lines used to checking the patches. So folks can align with the expected outcome, e.g. command line parameters.
Quoted 8 lines
These two guidelines (and generally the few others above) are intended to apply to all Rust code in the kernel (i.e. not just
rust/) -- their command lines are mentioned inDocumentation/rust/. We could add a note to make that clearer if that helps. So I would suggest avoiding repetition here by referencing those.We also mention it in our "Subsystem Profile document" at
https://rust-for-linux.com/contributing#submit-checklist-addendum.
I think we can refer the links so that we don't need to explain the process in detail. I would prefer to have the exact command lines that maintainer are using in the doce. E.g. I was experiencing that folks using different params with checkpatch.pl, the outcome, .e.g. warnings are different. different spell-checks backend gives different errors.
It could be nice that we put the command lines explicitly so that folks would save some efforts on re-spin. It also saves maintainer's efforts.
Z.
Quoted 9 lines
Quoted 3 lines
Documentation/gpu/nova/guidelines.rst line 42
Quoted 3 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0+The availability of proper documentation is essential in terms of scalability, +accessability for new contributors and maintainability of a project in general,Typo: accessibility?
Cheers,
Miguel
Danilo Krummrich
maintainerRe: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 9 lines · Zhi Wang
Quoted 6 lines · On Wed, 5 Feb 2025 15:13:12 +0100 Miguel Ojeda…
Quoted 4 lines · Zhi Wang
It would be also helpful to make the process explicit. E.g. sharing your command lines used to checking the patches. So folks can align with the expected outcome, e.g. command line parameters.
Quoted 5 lines
Quoted 5 lines
These two guidelines (and generally the few others above) are intended to apply to all Rust code in the kernel (i.e. not just
rust/) -- their command lines are mentioned inDocumentation/rust/. We could add a note to make that clearer if that helps. So I would suggest avoiding repetition here by referencing those.
Regarding the two, for me they read more like suggestions. The few others are indeed pretty clearly documented in "general-information".
Gonna add references instead.
Quoted 4 lines
Quoted 3 lines
We also mention it in our "Subsystem Profile document" at
https://rust-for-linux.com/contributing#submit-checklist-addendum.
Quoted 8 lines
I think we can refer the links so that we don't need to explain the process in detail. I would prefer to have the exact command lines that maintainer are using in the doce. E.g. I was experiencing that folks using different params with checkpatch.pl, the outcome, .e.g. warnings are different. different spell-checks backend gives different errors.
It could be nice that we put the command lines explicitly so that folks would save some efforts on re-spin. It also saves maintainer's efforts.
Generally, I'm fine with adding the specific command that should be run before sending patches in a single place for convenience in this document.
But maybe it makes sense to consider whether this could go into the generic Rust documentation too?
Quoted 12 lines
Z.
Quoted 9 lines
Quoted 3 lines
Documentation/gpu/nova/guidelines.rst line 42
Quoted 3 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0+The availability of proper documentation is essential in terms of scalability, +accessability for new contributors and maintainability of a project in general,Typo: accessibility?
Cheers,
Miguel
Danilo Krummrich
maintainerRe: [PATCH v2 2/2] gpu: nova-core: add initial documentation
In reply to Zhi WangQuoted 16 lines · Zhi Wang
Quoted 13 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Documentation/gpu/nova/core/todo.rst new file +13
-0+ +Generic register abstraction +---------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. + +Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO),
Quoted 8 lines
Documentation/gpu/nova/core/todo.rst line 59
Quoted 4 lines
Documentation/gpu/nova/core/todo.rst new file +4
-0+ MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev. + ]) +I think it is better not to tie this to
pci::Barand its operations. It would be better to have a intermediate container as the macro param. The container holds the register region vaddr pointer, size, read/write traits.
I think for this particular example we want to tie it to the pci::Bar on way or the other.
But yes, the idea would be that any dyn Io works.
Quoted 8 lines
The macro expands it from there, thus, we can also use this on firmware memory structures, e.g. GSP WPR2 info.
Probably we are looking for a even more generic solution/type for deferring a structure in the vaddr and generating the the accessing methods accordingly. It might also be useful later in GSP message queue manipulation, ELF header extraction, page table manipulation? (to avoid ambitious unsafe statements in the rust driver)
I think it's hard to predict to what extend we can (or want to) generalize this. But I think those are good thoughts that should be considered working on this.
If you come up with a few lines to broaden the scope, I'm happy to add them to this section.
Quoted 65 lines
Quoted 50 lines
Documentation/gpu/nova/core/todo.rst new file +49
-0+This could expand to something like: + +.. code-block:: rust + + const BOOT0_OFFSET: usize = 0x00000000; + const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; + const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; + const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; + const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; + const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; + const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; + + struct Boot0(u32); + + impl Boot0 { + #[inline] + fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { + Self(bar.readl(BOOT0_OFFSET)) + } + + #[inline] + fn minor_revision(&self) -> u32 { + (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT + } + + #[inline] + fn major_revision(&self) -> u32 { + (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT + } + + #[inline] + fn revision(&self) -> u32 { + (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT + } + } + +Usage: + +.. code-block:: rust + + let bar = bar.try_access().ok_or(ENXIO)?; + + let boot0 = Boot0::read(&bar); + pr_info!("Revision: {}\n", boot0.revision()); + +| Complexity: Advanced + +Delay / Sleep abstractions +--------------------------...
Quoted 12 lines
Documentation/gpu/nova/core/todo.rst new file +12
-0+ +VRAM memory allocator +--------------------- + +Investigate options for a VRAM memory allocator. + +Some possible options: + - Rust abstractions for + - RB tree (interval tree) / drm_mm + - maple_tree + - native Rust collections +
Quoted 8 lines
Quoted 2 lines
+| Complexity: Advanced
+I am leaning towards having the abstractions at a high level APIs, e.g. wrapping
drm_mmand possibly the rust side can choose the backend type ofdrm_mmif it really needs a different type of data structure other than default supported bydrm_mm. If we need more type of data structures, we can extenddrm_mmin C side. That can save us some efforts.
I think we have to work out the advantages and disadvantages and see if there are any opportunities for synergies.
For instance, in case we end up with separate VMM management in nova-core and nova-drm, there might be synergies for the other options.
Quoted 46 lines
Quoted 11 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+Instance Memory +--------------- + +Implement support for instmem (bar2) used to store page tables. + +| Complexity: Intermediate +| Contact: Dave Airlie + +GPU System Processor (GSP) +==========================...
Quoted 16 lines
Documentation/gpu/nova/core/todo.rst new file +16
-0+ +External APIs +============= + +nova-core base API +------------------ + +Work out the common pieces of the API to connect 2nd level drivers, i.e. vGPU +manager and nova-drm. + +| Complexity: Advanced + +vGPU manager API +---------------- + +Work out the API parts required by the vGPU manager, which are not covered byDocumentation/gpu/nova/core/todo.rst line 428
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+the base API. + +| Complexity: Advanced + +nova-core C API +--------------- + +Implement a C wrapper for the APIs required by the vGPU manager driver. + +| Complexity: IntermediateThanks for calling this out.
I believe the "vGPU manager API" is not a standalone task, as many of the required APIs will intersect with other components in nova-core.
Quoted 8 lines
As one of nova-core’s users, vGPU represents the simplest use case to get started with, offering significant value to both nova-core and its users in the near term.
I was thinking that if we could align with the folks on making vGPU + nova-core our initial short-term goal, it would be beneficial for nova-drm's development, since the APIs required for nova-drm are a superset of those needed for vGPU.
I think it is common understanding to have nova-core + vGPU as a first milestone.
However, I think that we should also put some effort into the nova-drm side of things to not miss out on potential design contraints on both ends, nova-core <-> nova-drm as well as nova-drm <-> DRM core, since there are also other upcoming Rust DRM drivers.
Quoted 7 lines
It would be valuable for us to be involved in key areas related to vGPU, including:
- Task review
- Design discussions
- Unit testing
I'm a bit confused about who specifically is meant with "we", "the folks" and "us" in your statements above.
But of course everyone is welcome to be involved into the aspects mentioned above and the project in general. :-)
Quoted 4 lines
Additionally, we are working on a vGPU requirements document that will outline the complete API needs for vGPU beyond those already covered in the RFC patches. Hope that will be published soon.
I think this will be very useful for nova project!
Quoted 31 lines
Quoted 24 lines
Documentation/gpu/nova/core/todo.rst new file +17
-0+ +Testing +======= + +CI pipeline +----------- + +Investigate option for continuous integration testing. + +This can go from as simple as running KUnit tests over running (graphics) CTS to +booting up (multiple) guest VMs to test VFIO use-cases. + +It might also be worth to consider the introduction of a new test suite directly +sitting on top of the uAPI for more targeted testing and debugging. There may be +options for collaboration / shared code with the Mesa project. + +| Complexity: AdvancedDocumentation/gpu/nova/guidelines.rst new file +0
-0Documentation/gpu/nova/guidelines.rst line 2
Quoted 2 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0@@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) +I think this will develop into a maintainer handbook in the future which reflects maintainer's requirements, thoughts, tips...maybe we can make it explicit? I think it is rules of book that we agree to follow.
Quoted 2 lines
A similar one can be found here.
https://lore.kernel.org/kvm/[email protected]/
Yes, that's the idea. Is there anything additional you'd like to see added for this initial commit?
Quoted 19 lines
Quoted 19 lines
Documentation/gpu/nova/guidelines.rst new file +18
-0+========== +Guidelines +========== + +This document describes the general project guidelines that apply to nova-core +and nova-drm. + +Language +======== + +The Nova project uses the Rust programming language. In this context, the +following rules apply. + +- Unless technically necessary otherwise (e.g. uAPI), any driver code is written + in Rust. + +- Direct FFI calls to C kernel APIs must be avoided; instead C kernel APIs + should be accessed through shared Rust abstractions.
Quoted 8 lines
Documentation/gpu/nova/guidelines.rst line 24
Quoted 4 lines
Documentation/gpu/nova/guidelines.rst new file +4
-0+ +- Unless technically necessary, unsafe Rust code must be avoided. In case of + technical necessity, unsafe code should be isolated in a separate component + providing a safe API for other driver code to use.Also need to comment why the unsafe is the last possible approach to go. the last thing we want to see is "unsafe" flying here and there in a rust driver. :)
You mean adding an explanation as for why to avoid "unsafe"? I thought it's probably rather obvious.
But I'm also fine adding an explanation, suggestions?
Re: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 34 lines · Danilo Krummrich
Quoted 16 lines · Zhi Wang
Quoted 13 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Documentation/gpu/nova/core/todo.rst new file +13
-0+ +Generic register abstraction +---------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. + +Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO),Quoted 8 lines
Documentation/gpu/nova/core/todo.rst line 59
Quoted 4 lines
Documentation/gpu/nova/core/todo.rst new file +4
-0+ MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev. + ]) +I think it is better not to tie this to
pci::Barand its operations. It would be better to have a intermediate container as the macro param. The container holds the register region vaddr pointer, size, read/write traits.I think for this particular example we want to tie it to the
pci::Baron way or the other.But yes, the idea would be that any
dyn Ioworks.Quoted 3 lines
The macro expands it from there, thus, we can also use this on firmware memory structures, e.g. GSP WPR2 info.
Quoted 9 lines
Quoted 5 lines
Probably we are looking for a even more generic solution/type for deferring a structure in the vaddr and generating the the accessing methods accordingly. It might also be useful later in GSP message queue manipulation, ELF header extraction, page table manipulation? (to avoid ambitious unsafe statements in the rust driver)
I think it's hard to predict to what extend we can (or want to) generalize this. But I think those are good thoughts that should be considered working on this.
I am thinking of a macro doing this:
#[repr(C)]
#[derive(MemoryMappedStruct)] // Maybe a better name struct FirmwareStruct { // Firmware, headers definitions from the SDK
field1: u32,
field2: u16,
field3: u8,
}In the driver...
let my_struct = unsafe { FirmwareStruct::from_addr(vaddr) }; //
vaddr is the firmware VA, or put the vaddr in a common container generated by R4L Firmware abstraction, thus this unsafe can be opted-out as well.
// read
let v1 = my_struct.field1();
let v2 = my_struct.field2();
let v3 = my_struct.field3();
// write
my_struct.set_field1(0x42);
my_struct.set_field2(0x123);
my_struct.set_field3(0x9);...
I think it is doable and improves the readability a lot. It would be quite useful when extracting the headers and data structures, considering there are a lot of them coming.
Quoted 77 lines
If you come up with a few lines to broaden the scope, I'm happy to add them to this section.
Quoted 65 lines
Quoted 50 lines
Documentation/gpu/nova/core/todo.rst new file +49
-0+This could expand to something like: + +.. code-block:: rust + + const BOOT0_OFFSET: usize = 0x00000000; + const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; + const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; + const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; + const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; + const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; + const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; + + struct Boot0(u32); + + impl Boot0 { + #[inline] + fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { + Self(bar.readl(BOOT0_OFFSET)) + } + + #[inline] + fn minor_revision(&self) -> u32 { + (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT + } + + #[inline] + fn major_revision(&self) -> u32 { + (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT + } + + #[inline] + fn revision(&self) -> u32 { + (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT + } + } + +Usage: + +.. code-block:: rust + + let bar = bar.try_access().ok_or(ENXIO)?; + + let boot0 = Boot0::read(&bar); + pr_info!("Revision: {}\n", boot0.revision()); + +| Complexity: Advanced + +Delay / Sleep abstractions +--------------------------...
Quoted 12 lines
Documentation/gpu/nova/core/todo.rst new file +12
-0+ +VRAM memory allocator +--------------------- + +Investigate options for a VRAM memory allocator. + +Some possible options: + - Rust abstractions for + - RB tree (interval tree) / drm_mm + - maple_tree + - native Rust collections +Quoted 8 lines
Quoted 2 lines
+| Complexity: Advanced
+I am leaning towards having the abstractions at a high level APIs, e.g. wrapping
drm_mmand possibly the rust side can choose the backend type ofdrm_mmif it really needs a different type of data structure other than default supported bydrm_mm. If we need more type of data structures, we can extenddrm_mmin C side. That can save us some efforts.
Quoted 6 lines
I think we have to work out the advantages and disadvantages and see if there are any opportunities for synergies.
For instance, in case we end up with separate VMM management in nova-core and nova-drm, there might be synergies for the other options.
Is this for the userspace-controlled GPU VA? It would be interesting to know its different requirements from the current drm_mm. :)
Quoted 115 lines
Quoted 46 lines
Quoted 11 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+Instance Memory +--------------- + +Implement support for instmem (bar2) used to store page tables. + +| Complexity: Intermediate +| Contact: Dave Airlie + +GPU System Processor (GSP) +==========================...
Quoted 16 lines
Documentation/gpu/nova/core/todo.rst new file +16
-0+ +External APIs +============= + +nova-core base API +------------------ + +Work out the common pieces of the API to connect 2nd level drivers, i.e. vGPU +manager and nova-drm. + +| Complexity: Advanced + +vGPU manager API +---------------- + +Work out the API parts required by the vGPU manager, which are not covered byDocumentation/gpu/nova/core/todo.rst line 428
Quoted 10 lines
Documentation/gpu/nova/core/todo.rst new file +10
-0+the base API. + +| Complexity: Advanced + +nova-core C API +--------------- + +Implement a C wrapper for the APIs required by the vGPU manager driver. + +| Complexity: IntermediateThanks for calling this out.
I believe the "vGPU manager API" is not a standalone task, as many of the required APIs will intersect with other components in nova-core.
Quoted 8 lines
As one of nova-core’s users, vGPU represents the simplest use case to get started with, offering significant value to both nova-core and its users in the near term.
I was thinking that if we could align with the folks on making vGPU + nova-core our initial short-term goal, it would be beneficial for nova-drm's development, since the APIs required for nova-drm are a superset of those needed for vGPU.
I think it is common understanding to have nova-core + vGPU as a first milestone.
However, I think that we should also put some effort into the nova-drm side of things to not miss out on potential design contraints on both ends, nova-core <-> nova-drm as well as nova-drm <-> DRM core, since there are also other upcoming Rust DRM drivers.
Quoted 7 lines
It would be valuable for us to be involved in key areas related to vGPU, including:
- Task review
- Design discussions
- Unit testingI'm a bit confused about who specifically is meant with "we", "the folks" and "us" in your statements above.
But of course everyone is welcome to be involved into the aspects mentioned above and the project in general. :-)
Quoted 4 lines
Additionally, we are working on a vGPU requirements document that will outline the complete API needs for vGPU beyond those already covered in the RFC patches. Hope that will be published soon.
I think this will be very useful for nova project!
Quoted 31 lines
Quoted 24 lines
Documentation/gpu/nova/core/todo.rst new file +17
-0+ +Testing +======= + +CI pipeline +----------- + +Investigate option for continuous integration testing. + +This can go from as simple as running KUnit tests over running (graphics) CTS to +booting up (multiple) guest VMs to test VFIO use-cases. + +It might also be worth to consider the introduction of a new test suite directly +sitting on top of the uAPI for more targeted testing and debugging. There may be +options for collaboration / shared code with the Mesa project. + +| Complexity: AdvancedDocumentation/gpu/nova/guidelines.rst new file +0
-0Documentation/gpu/nova/guidelines.rst line 2
Quoted 2 lines
Documentation/gpu/nova/guidelines.rst new file +2
-0@@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) +I think this will develop into a maintainer handbook in the future which reflects maintainer's requirements, thoughts, tips...maybe we can make it explicit? I think it is rules of book that we agree to follow.
Quoted 6 lines
Quoted 2 lines
A similar one can be found here.
https://lore.kernel.org/kvm/[email protected]/Yes, that's the idea. Is there anything additional you'd like to see added for this initial commit?
I am mostly thinking of testing of vGPU, we can conclude that more clearly when we have basic stuff working. IMO, maybe some brief introduction about maintainer branches, requirements of patch comments, e.g. prefix of patch titles to different components, expectation of the patch comment body beside those already in submitting_patches.rst. I think those would be helpful.
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
Documentation/gpu/nova/guidelines.rst new file +18
-0+========== +Guidelines +========== + +This document describes the general project guidelines that apply to nova-core +and nova-drm. + +Language +======== + +The Nova project uses the Rust programming language. In this context, the +following rules apply. + +- Unless technically necessary otherwise (e.g. uAPI), any driver code is written + in Rust. + +- Direct FFI calls to C kernel APIs must be avoided; instead C kernel APIs + should be accessed through shared Rust abstractions.Quoted 8 lines
Documentation/gpu/nova/guidelines.rst line 24
Quoted 4 lines
Documentation/gpu/nova/guidelines.rst new file +4
-0+ +- Unless technically necessary, unsafe Rust code must be avoided. In case of + technical necessity, unsafe code should be isolated in a separate component + providing a safe API for other driver code to use.Also need to comment why the unsafe is the last possible approach to go. the last thing we want to see is "unsafe" flying here and there in a rust driver. :)
Quoted 4 lines
You mean adding an explanation as for why to avoid "unsafe"? I thought it's probably rather obvious.
But I'm also fine adding an explanation, suggestions?
It is always nice to make it clear. We can also refer this rules from R4L. :)
Z.
Danilo Krummrich
maintainerRe: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 24 lines · Zhi Wang
Quoted 16 lines · Danilo Krummrich
Quoted 15 lines · Zhi Wang
Documentation/gpu/nova/guidelines.rst line 2
Quoted 8 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Documentation/gpu/nova/guidelines.rst new file +2
-0@@ -0,0 +1,73 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) +I think this will develop into a maintainer handbook in the future which reflects maintainer's requirements, thoughts, tips...maybe we can make it explicit? I think it is rules of book that we agree to follow.
Quoted 6 lines
Quoted 2 lines
A similar one can be found here.
https://lore.kernel.org/kvm/[email protected]/Yes, that's the idea. Is there anything additional you'd like to see added for this initial commit?
Quoted 6 lines
I am mostly thinking of testing of vGPU, we can conclude that more clearly when we have basic stuff working. IMO, maybe some brief introduction about maintainer branches, requirements of patch comments, e.g. prefix of patch titles to different components, expectation of the patch comment body beside those already in
submitting_patches.rst. I think those would be helpful.
I agree, but I don't think we need those things right from the get-go. Let's add those things subsequently.
Alexandre Courbot
maintainerRe: [PATCH v2 2/2] gpu: nova-core: add initial documentation
In reply to Zhi WangQuoted 103 lines · Zhi Wang
Quoted 82 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Add the initial documentation of the Nova project.
The initial project documentation consists out of a brief introduction of the project, as well as project guidelines both general and nova-core specific and a task list for nova-core specifically.
The task list is divided into tasks for general Rust infrastructure required by the project, tasks regarding GSP enablement and firmware abstraction, general GPU driver tasks as well as tasks related to external API design and test infrastructure.
Signed-off-by Danilo Krummrich <[email protected]>---
- Add task "Generic register abstraction".
- Change complexity of "Debugfs abstractions".--- Documentation/gpu/drivers.rst | 1 + Documentation/gpu/nova/core/guidelines.rst | 24 ++ Documentation/gpu/nova/core/todo.rst | 445 +++++++++++++++++++++ Documentation/gpu/nova/guidelines.rst | 73 ++++ Documentation/gpu/nova/index.rst | 30 ++ MAINTAINERS | 1 + 6 files changed, 574 insertions(+) create mode 100644 Documentation/gpu/nova/core/guidelines.rst create mode 100644 Documentation/gpu/nova/core/todo.rst create mode 100644 Documentation/gpu/nova/guidelines.rst create mode 100644 Documentation/gpu/nova/index.rstDocumentation/gpu/drivers.rst +1
-0@@ -24,6 +24,7 @@ GPU Driver Documentation panfrost panthor zynqmp + nova/index .. only:: subproject and htmlDocumentation/gpu/nova/core/guidelines.rst new file +24
-0@@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========== +Guidelines +========== + +This documents contains the guidelines for nova-core. Additionally, all common +guidelines of the Nova project do apply. + +Driver API +========== + +One main purpose of nova-core is to implement the abstraction around the +firmware interface of GSP and provide a firmware (version) independent API for +2nd level drivers, such as nova-drm or the vGPU manager VFIO driver. + +Therefore, it is not permitted to leak firmware (version) specifics, through the +driver API, to 2nd level drivers. + +Acceptance Criteria +=================== + +- To the extend possible, patches submitted to nova-core must be tested for + regressions with all 2nd level drivers.Documentation/gpu/nova/core/todo.rst new file +6
-0@@ -0,0 +1,445 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========= +Task List +========= +...
Quoted 15 lines
Documentation/gpu/nova/core/todo.rst new file +15
-0+ +Generic register abstraction +---------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. + +Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO), + MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev.
Quoted 8 lines
Documentation/gpu/nova/core/todo.rst
Quoted 2 lines
Documentation/gpu/nova/core/todo.rst new file +2
-0+ ]) +I think it is better not to tie this to
pci::Barand its operations. It would be better to have a intermediate container as the macro param. The container holds the register region vaddr pointer, size, read/write traits. The macro expands it from there, thus, we can also use this on firmware memory structures, e.g. GSP WPR2 info.
Another reason for not tying this to a particular bus is that Tegra doesn't use PCI to reach the registers of its integrated GPU. Maybe we can remove that parameter from the register!() macro and have read() take a generic argument for its bar parameter instead, so that method gets automatically specialized for every type of bus we need to use?
Danilo Krummrich
maintainerRe: [PATCH v2 2/2] gpu: nova-core: add initial documentation
Quoted 43 lines · Alexandre Courbot
Quoted 33 lines · Zhi Wang
Quoted 13 lines · On Tue, 4 Feb 2025 20:03:12 +0100 Danilo Krummrich <[email protected]> wrote:
Documentation/gpu/nova/core/guidelines.rst new file +1
-0+ regressions with all 2nd level drivers.Documentation/gpu/nova/core/todo.rst new file +6
-0@@ -0,0 +1,445 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========= +Task List +========= +...
Quoted 15 lines
Documentation/gpu/nova/core/todo.rst new file +15
-0+ +Generic register abstraction +---------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. + +Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO), + MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev.Quoted 8 lines
Documentation/gpu/nova/core/todo.rst
Quoted 2 lines
Documentation/gpu/nova/core/todo.rst new file +2
-0+ ]) +I think it is better not to tie this to
pci::Barand its operations. It would be better to have a intermediate container as the macro param. The container holds the register region vaddr pointer, size, read/write traits. The macro expands it from there, thus, we can also use this on firmware memory structures, e.g. GSP WPR2 info.
Quoted 5 lines
Another reason for not tying this to a particular bus is that Tegra doesn't use PCI to reach the registers of its integrated GPU. Maybe we can remove that parameter from the register!() macro and have
read()take a generic argument for itsbarparameter instead, so that method gets automatically specialized for every type of bus we need to use?
This is just an example, I do not mean to tie this to the PCI bus. But rather make it generic, such that it can be tied to any I/O resource.
Being able to tie the generated code (but not the macro itself) to a specific resource though would be nice to have.
Alexandre Courbot
maintainerHi Danilo,
Here are few comments - or maybe I should say nits, as they are really minor. Note that all my experience in Rust is from user-space, so feel free to ignore anything that does not make sense in the context of the kernel or is too pedantic.
Quoted 36 lines · Danilo Krummrich
Add the initial nova-core driver stub.
nova-core is intended to serve as a common base for nova-drm (the corresponding DRM driver) and the vGPU manager VFIO driver, serving as a hard- and firmware abstraction layer for GSP-based NVIDIA GPUs.
The Nova project, including nova-core and nova-drm, in the long term, is intended to serve as the successor of Nouveau for all GSP-based GPUs.
The motivation for both, starting a successor project for Nouveau and doing so using the Rust programming language, is documented in detail through a previous post on the mailing list [1], an LWN article [2] and a talk from LPC '24.
In order to avoid the chicken and egg problem to require a user to upstream Rust abstractions, but at the same time require the Rust abstractions to implement the driver, nova-core kicks off as a driver stub and is subsequently developed upstream.
Link https://lwn.net/Articles/990736/ [2]Signed-off-by Danilo Krummrich <[email protected]>---
Changes in v2:
- Fix module name inKconfigdescription. (John)
- ExpandKconfigdescription a bit. (John)
- Expand name for PCI BAR0 region.
- Do not store / print boot0 raw register value. (John)
- Rename CardType to Architecture, rename enum names to represent the architecture name and adjust enum values according to the register definition. (John)
- Add an abstraction for register accesses.
- Print chipset, architecture and revision.
- Load bootloader firmware. (Timur)
---
<snip>
Quoted 8 lines
drivers/gpu/nova-core/gpu.rs new file +2
-0@@ -0,0 +1,187 @@ +// SPDX-License-Identifier: GPL-2.0 +
drivers/gpu/nova-core/gpu.rs line 12
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+use kernel::{ + device, devres::Devres , error::code::*, firmware, fmt, pci, prelude::*, str::CString , +}; + +use crate ::driver::Bar0 ; +use crate ::regs; +use core::fmt; + +/// Enum representation of the GPU chipset. +#[derive(fmt:: Debug )]
I suspect you will eventually want to also derive Copy and Clone, as well as PartialEq and Eq (so the assigned values can be used), but it's of course fine to postpone this until we actually need them.
Note that the usage made of Debug suggests that you actually want Display - but I understand that implementing Display would be more cumbersome.
Quoted 71 lines
drivers/gpu/nova-core/gpu.rs new file +71
-0+pub (crate )enum Chipset { +TU102 =0x162 , +TU104 =0x164 , +TU106 =0x166 , +TU117 =0x167 , +TU116 =0x168 , +GA102 =0x172 , +GA103 =0x173 , +GA104 =0x174 , +GA106 =0x176 , +GA107 =0x177 , +AD102 =0x192 , +AD103 =0x193 , +AD104 =0x194 , +AD106 =0x196 , +AD107 =0x197 , +} + +/// Enum representation of the GPU generation. +#[derive(fmt:: Debug )] +pub (crate )enum Architecture { +Turing =0x16 , +Ampere =0x17 , +Ada =0x19 , +} + +pub (crate )struct Revision { + major:u8 , + minor:u8 , +} + +/// Structure holding the metadata of the GPU. +pub (crate )struct Spec { + chipset:Chipset , + arch:Architecture , +/// The revision of the chipset. + revision:Revision , +} + +/// Structure encapsulating the firmware blobs required for the GPU to operate. +#[allow(dead_code)] +pub (crate )struct Firmware { + booter_load: firmware::Firmware , + booter_unload: firmware::Firmware , + bootloader: firmware::Firmware , + gsp: firmware::Firmware , +} + +/// Structure holding the resources required to operate the GPU. +#[allow(dead_code)] +#[pin_data] +pub (crate )struct Gpu { + spec:Spec , +/// MMIO mapping of PCI BAR 0 + bar:Devres <Bar0 >, + fw:Firmware , +} + +// TODO replace with something like derive(FromPrimitive) +impl Chipset { +fn from_u32 (value:u32 ) ->Option <Chipset > { +match value { +0x162 =>Some (Chipset ::TU102 ), +0x164 =>Some (Chipset ::TU104 ), +0x166 =>Some (Chipset ::TU106 ), +0x167 =>Some (Chipset ::TU117 ), +0x168 =>Some (Chipset ::TU116 ), +0x172 =>Some (Chipset ::GA102 ), +0x173 =>Some (Chipset ::GA103 ), +0x174 =>Some (Chipset ::GA104 ), +0x176 =>Some (Chipset ::GA106 ),
drivers/gpu/nova-core/gpu.rs line 93
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+ 0x177 => Some(Chipset::GA107), + 0x192 => Some(Chipset::AD102), + 0x193 => Some(Chipset::AD103), + 0x194 => Some(Chipset::AD104), + 0x196 => Some(Chipset::AD106), + 0x197 => Some(Chipset::AD107), + _ => None, + } + } +}
Shouldn't this be an implementation of TryFrom<u32>? By doing so you can return ENODEV as the error and simplify the caller code below.
Quoted 9 lines
drivers/gpu/nova-core/gpu.rs new file +9
-0+ +// TODO: +// - replace with something like derive(FromPrimitive) +// - consider to store within Chipset, if arbitrary_enum_discriminant becomes stable +impl Architecture { +fn from_u32 (value:u32 ) ->Option <Architecture > { +match value { +0x16 =>Some (Architecture ::Turing ), +0x17 =>Some (Architecture ::Ampere ),
drivers/gpu/nova-core/gpu.rs line 112
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+ 0x19 => Some(Architecture::Ada), + _ => None, + } + } +} + +impl Revision { +fn new (major:u8 , minor:u8 ) ->Self { +Self { major, minor } + }
Suggestion: add a version that takes a Boot0 as argument and call the right methods directly in the method instead of relying on the caller to do that for us, e.g:
fn from_boot0(boot0: ®s::Boot0) -> Self {
Self::new(boot0.major_rev(), boot0.minor_rev())
}Then new() can also be removed if Boot0 is the only sensible source of Revision.
(I'd argue that Boot0 should also implement Copy, that way this method can take it by value directly)
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+} + +impl fmt::Display for Revision { +fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result { +write! (f,"{:x}.{:x}" , self.major, self.minor) + } +} + +impl Spec { +fn new (bar: &Devres <Bar0 >) ->Result <Spec > {
drivers/gpu/nova-core/gpu.rs line 132
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+let bar = bar.try_access ().ok_or (ENXIO )?; +let boot0 = regs::Boot0 ::read (&bar); + +let Some (chipset) =Chipset ::from_u32 (boot0.chipset ())else { +return Err (ENODEV ); + }; + +let Some (arch) =Architecture ::from_u32 (boot0.arch ())else { +return Err (ENODEV ); + };
Technically the Architecture is already known if the Chipset has been built successfully, so there should be no need to build it again (and test for a failure that cannot happen at this point).
Since the architecture information is already embedded in Chipset, maybe we can have an arch() method there?
Something like:
impl Chipset {
pub(crate) fn arch(self) -> Architecture {
match self as u32 & !0xf {
0x160 => Architecture::Turing,
0x170 => Architecture::Ampere,
0x190 => Architecture::Ada,
_ => unreachable!(),
}
}
}This would also enable us to remove Architecture::from_u32() and Spec::arch, which is redundant with Spec::chipset anyway.
A better (but more verbose) implementation of Chipset::arch() might be to match every possible variant, so we get a build error if we forget to handle a new chipset instead of hitting the unreachable!() at runtime...
Quoted 26 lines
drivers/gpu/nova-core/gpu.rs new file +26
-0+ +let revision =Revision ::new (boot0.major_rev (), boot0.minor_rev ()); + +Ok (Self { + arch, + chipset, + revision, + }) + } +} + +impl Firmware { +fn new (dev: &device::Device , spec: &Spec , ver: &str ) ->Result <Firmware > { +let mut chip_name =CString ::try_from_fmt (fmt! ("{:?}" , spec.chipset))?; + chip_name.make_ascii_lowercase (); + +let fw_booter_load_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/booter_load-{}.bin" , &*chip_name, ver))?; +let fw_booter_unload_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/booter_unload-{}.bin" , &*chip_name, ver))?; +let fw_bootloader_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/bootloader-{}.bin" , &*chip_name, ver))?; +let fw_gsp_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/gsp-{}.bin" , &*chip_name, ver))?; + +let booter_load = firmware::Firmware ::request (&fw_booter_load_path, dev)?;
drivers/gpu/nova-core/gpu.rs line 168
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+let booter_unload = firmware::Firmware ::request (&fw_booter_unload_path, dev)?; +let bootloader = firmware::Firmware ::request (&fw_bootloader_path, dev)?; +let gsp = firmware::Firmware ::request (&fw_gsp_path, dev)?; + +Ok (Firmware { + booter_load, + booter_unload, + bootloader, + gsp, + })
This looks like a good opportunity to use a closure and avoid repeating the code:
let request_fw = |type_| {
CString::try_from_fmt(fmt!("nvidia/{}/gsp/{}-{}.bin", type_, &*chip_name, ver))
.and_then(|path| firmware::Firmware::request(&path, dev))
};It is also short enough that you can directly invoke it when building the Firmware object, without using temporary variables:
Ok(Firmware {
booter_load: request_fw("booter_load")?,
booter_unload: request_fw("booter_unload")?,
bootloader: request_fw("bootloader")?,
gsp: request_fw("gsp")?,
})IMHO this has the benefit of being more concise and keeping related operations closer.
Thanks!
Alex.
Danilo Krummrich
maintainerHi Alexandre,
Quoted 7 lines · Alexandre Courbot
drivers/gpu/nova-core/gpu.rs line 12
Quoted 3 lines
drivers/gpu/nova-core/gpu.rs new file +3
-0+ +/// Enum representation of the GPU chipset. +#[derive(fmt:: Debug )] I suspect you will eventually want to also derive Copy and Clone, as well as PartialEq and Eq (so the assigned values can be used), but it's of course fine to postpone this until we actually need them.
Indeed, the idea is to add it as needed.
Quoted 24 lines
Note that the usage made of Debug suggests that you actually want Display - but I understand that implementing Display would be more cumbersome.
Quoted 19 lines
drivers/gpu/nova-core/gpu.rs new file +19
-0+ +// TODO replace with something like derive(FromPrimitive) +impl Chipset { +fn from_u32 (value:u32 ) ->Option <Chipset > { +match value { +0x162 =>Some (Chipset ::TU102 ), +0x164 =>Some (Chipset ::TU104 ), +0x166 =>Some (Chipset ::TU106 ), +0x167 =>Some (Chipset ::TU117 ), +0x168 =>Some (Chipset ::TU116 ), +0x172 =>Some (Chipset ::GA102 ), +0x173 =>Some (Chipset ::GA103 ), +0x174 =>Some (Chipset ::GA104 ), +0x176 =>Some (Chipset ::GA106 ), +0x177 =>Some (Chipset ::GA107 ), +0x192 =>Some (Chipset ::AD102 ), +0x193 =>Some (Chipset ::AD103 ), +0x194 =>Some (Chipset ::AD104 ), +0x196 =>Some (Chipset ::AD106 ),
Quoted 8 lines
drivers/gpu/nova-core/gpu.rs line 93
Quoted 5 lines
drivers/gpu/nova-core/gpu.rs new file +5
-0+ 0x197 => Some(Chipset::AD107), + _ => None, + } + } +}Shouldn't this be an implementation of TryFrom<u32>? By doing so you can return ENODEV as the error and simplify the caller code below.
Yes, it should be. I wanted to change that, but forgot about it. Thanks for the reminder.
But ultimately, as the comment says, I'd like to have some kind of FromPrimitive implementation for that.
Quoted 25 lines
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +9
-0+ +// TODO: +// - replace with something like derive(FromPrimitive) +// - consider to store within Chipset, if arbitrary_enum_discriminant becomes stable +impl Architecture { +fn from_u32 (value:u32 ) ->Option <Architecture > { +match value { +0x16 =>Some (Architecture ::Turing ), +0x17 =>Some (Architecture ::Ampere ),drivers/gpu/nova-core/gpu.rs line 112
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+ 0x19 => Some(Architecture::Ada), + _ => None, + } + } +} + +impl Revision { +fn new (major:u8 , minor:u8 ) ->Self { +Self { major, minor } + }Suggestion: add a version that takes a Boot0 as argument and call the right methods directly in the method instead of relying on the caller to do that for us, e.g:
Quoted 7 lines
fn from_boot0(boot0: ®s::Boot0) -> Self { Self::new(boot0.major_rev(), boot0.minor_rev()) }Then
new()can also be removed if Boot0 is the only sensible source of Revision.
That's a good suggestion, I'll pick that up.
Quoted 34 lines
(I'd argue that Boot0 should also implement Copy, that way this method can take it by value directly)
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+} + +impl fmt::Display for Revision { +fn fmt (&self, f: &mut fmt::Formatter <'_>) -> fmt::Result { +write! (f,"{:x}.{:x}" , self.major, self.minor) + } +} + +impl Spec { +fn new (bar: &Devres <Bar0 >) ->Result <Spec > {drivers/gpu/nova-core/gpu.rs line 132
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+let bar = bar.try_access ().ok_or (ENXIO )?; +let boot0 = regs::Boot0 ::read (&bar); + +let Some (chipset) =Chipset ::from_u32 (boot0.chipset ())else { +return Err (ENODEV ); + }; + +let Some (arch) =Architecture ::from_u32 (boot0.arch ())else { +return Err (ENODEV ); + };Technically the Architecture is already known if the Chipset has been built successfully, so there should be no need to build it again (and test for a failure that cannot happen at this point).
Since the architecture information is already embedded in Chipset, maybe we can have an
arch()method there?Something like:
Quoted 10 lines
impl Chipset {
pub(crate) fn arch(self) -> Architecture {
match self as u32 & !0xf {
0x160 => Architecture::Turing,
0x170 => Architecture::Ampere,
0x190 => Architecture::Ada,
_ => unreachable!(),
}
}
}I thought about this, which is also why the comment above says: "consider to store within Chipset, if arbitrary_enum_discriminant becomes stable".
I did not go with what you suggest because it leaves us with either Chipset::arch() returning a Result, which is annoying, or with Chipset::arch() being able to panic the kernel, which I'd dislike even more.
There's also a third option, which would be to have some kind of unknown architecture, which we could catch later on, but that's just a worse variation of returning a Result.
Another reason was that I did not want to encode register specific masks into the Chipset type.
Quoted 8 lines
This would also enable us to remove
Architecture::from_u32()andSpec::arch, which is redundant withSpec::chipsetanyway.A better (but more verbose) implementation of
Chipset::arch()might be to match every possible variant, so we get a build error if we forget to handle a new chipset instead of hitting the unreachable!() at runtime...
I think that would indeed be a reasonable option.
Quoted 56 lines
Quoted 27 lines
drivers/gpu/nova-core/gpu.rs new file +26
-0+ +let revision =Revision ::new (boot0.major_rev (), boot0.minor_rev ()); + +Ok (Self { + arch, + chipset, + revision, + }) + } +} + +impl Firmware { +fn new (dev: &device::Device , spec: &Spec , ver: &str ) ->Result <Firmware > { +let mut chip_name =CString ::try_from_fmt (fmt! ("{:?}" , spec.chipset))?; + chip_name.make_ascii_lowercase (); + +let fw_booter_load_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/booter_load-{}.bin" , &*chip_name, ver))?; +let fw_booter_unload_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/booter_unload-{}.bin" , &*chip_name, ver))?; +let fw_bootloader_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/bootloader-{}.bin" , &*chip_name, ver))?; +let fw_gsp_path = +CString ::try_from_fmt (fmt! ("nvidia/{}/gsp/gsp-{}.bin" , &*chip_name, ver))?; + +let booter_load = firmware::Firmware ::request (&fw_booter_load_path, dev)?;drivers/gpu/nova-core/gpu.rs line 168
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs new file +10
-0+let booter_unload = firmware::Firmware ::request (&fw_booter_unload_path, dev)?; +let bootloader = firmware::Firmware ::request (&fw_bootloader_path, dev)?; +let gsp = firmware::Firmware ::request (&fw_gsp_path, dev)?; + +Ok (Firmware { + booter_load, + booter_unload, + bootloader, + gsp, + })This looks like a good opportunity to use a closure and avoid repeating the code:
let request_fw = |type_| { CString::try_from_fmt(fmt!("nvidia/{}/gsp/{}-{}.bin", type_, &*chip_name, ver)) .and_then(|path| firmware::Firmware::request(&path, dev)) };It is also short enough that you can directly invoke it when building the Firmware object, without using temporary variables:
Ok(Firmware { booter_load: request_fw("booter_load")?, booter_unload: request_fw("booter_unload")?, bootloader: request_fw("bootloader")?, gsp: request_fw("gsp")?, })
Quoted 2 lines
IMHO this has the benefit of being more concise and keeping related operations closer.
I agree, that's pretty clean.
Quoted 4 lines
Thanks!
Alex.
Alexandre Courbot
maintainerHi Danilo,
Quoted 40 lines · Danilo Krummrich
Quoted 22 lines
drivers/gpu/nova-core/gpu.rs line 132
Quoted 12 lines
drivers/gpu/nova-core/gpu.rs new file +12
-0+impl Spec { +fn new (bar: &Devres <Bar0 >) ->Result <Spec > { +let bar = bar.try_access ().ok_or (ENXIO )?; +let boot0 = regs::Boot0 ::read (&bar); + +let Some (chipset) =Chipset ::from_u32 (boot0.chipset ())else { +return Err (ENODEV ); + }; + +let Some (arch) =Architecture ::from_u32 (boot0.arch ())else { +return Err (ENODEV ); + };Technically the Architecture is already known if the Chipset has been built successfully, so there should be no need to build it again (and test for a failure that cannot happen at this point).
Since the architecture information is already embedded in Chipset, maybe we can have an
arch()method there?Something like:
Quoted 10 lines
impl Chipset { pub(crate) fn arch(self) -> Architecture { match self as u32 & !0xf { 0x160 => Architecture::Turing, 0x170 => Architecture::Ampere, 0x190 => Architecture::Ada, _ => unreachable!(), } } }I thought about this, which is also why the comment above says: "consider to store within Chipset, if
arbitrary_enum_discriminantbecomes stable".I did not go with what you suggest because it leaves us with either
Chipset::arch()returning a Result, which is annoying, or withChipset::arch()being able to panic the kernel, which I'd dislike even more.
Quoted 6 lines
There's also a third option, which would be to have some kind of unknown architecture, which we could catch later on, but that's just a worse variation of returning a Result.
Another reason was that I did not want to encode register specific masks into the Chipset type.
Agreed, none of these solutions are completely satisfying. From the point that we have successfully built a Chipset, we should be able to get its architecture without any runtime error or failure path. So I guess this leaves matching all the variants. It's a bit verbose, but hopefully later we can gather all the static information about chipsets in a single place, and generate these implementations (including a Display implementation - ideally the debug one would also print the hex representation of the chipset to be more useful for troubleshooting) using macros.
Cheers,
Alex.