Skip to contentSkip to conversation

gpu: nova: Export parameters from nova-core to nova-drm

RFC In review Alistair Popple 2 participants 7 messages Jul 3 – Jul 6

Subsystems (6)
DRM drivers DRM misc Rust DRM Rust +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover gpu: nova: Export parameters from nova-core to nova-drm

This patch series adds some basic GPU properties to the existing nova-drm GETPARAM ioctl. It builds on top of the "drm: Higher-Ranked Lifetime private data" series[1] to correctly manage lifetimes of registration data shared between DRM, auxbus and nova-core.

Properties are exported via a new AuxData type which implements methods to read data from the GPU. This has been implemented in a separate module to make the public API implementations obvious and to keep them in one place.

[1] - https://lore.kernel.org/rust-for-linux/[email protected]/

Cc Danilo Krummrich <[email protected]>
Cc Alice Ryhl <[email protected]>
Cc David Airlie <[email protected]>
Cc Alexandre Courbot <[email protected]>
Cc Benno Lossin <[email protected]>
Cc Gary Guo <[email protected]>
Cc Eliot Courtney <[email protected]>
Cc John Hubbard <[email protected]>

Alistair Popple (3):
gpu: nova-core: Add auxiliary bus registration data to nova-core
drm: nova: Add GETPARAM parameter to read the GPU chipset
drm: nova: Add a GETPARAM parameter to read usable VRAM size

 drivers/gpu/drm/nova/file.rs          |  6 ++++
 drivers/gpu/nova-core/auxdata.rs      | 25 +++++++++++++++
 drivers/gpu/nova-core/driver.rs       | 45 ++++++++++++++++++++-------
 drivers/gpu/nova-core/gpu.rs          | 16 +++++-----
 drivers/gpu/nova-core/gsp/commands.rs |  8 +++++
 drivers/gpu/nova-core/gsp/hal.rs      |  2 +-
 drivers/gpu/nova-core/nova_core.rs    |  1 +
 include/uapi/drm/nova_drm.h           | 14 +++++++++
 8 files changed, 95 insertions(+), 22 deletions(-)
 create mode 100644 drivers/gpu/nova-core/auxdata.rs
Signature · 1 line
2.54.0
#1 lore raw

RFC 1/3 gpu: nova-core: Add auxiliary bus registration data to nova-core

Nova core will be used to export core functionality to other drivers which will bind to it via auxiliary bus devices. To access methods on the nova-core GPU add an AuxData module and type which will implement the required methods and hold a reference to the associated GPU.

Create an instance of this type when registering the GPU on the auxbus for use as the auxbus registration data.

---
 drivers/gpu/nova-core/auxdata.rs   | 12 ++++++++
 drivers/gpu/nova-core/driver.rs    | 45 ++++++++++++++++++++++--------
 drivers/gpu/nova-core/gsp/hal.rs   |  2 +-
 drivers/gpu/nova-core/nova_core.rs |  1 +
 4 files changed, 47 insertions(+), 13 deletions(-)
 create mode 100644 drivers/gpu/nova-core/auxdata.rs
drivers/gpu/nova-core/auxdata.rs new file +12 -0
new file mode 100644
index 000000000000..266b5ce4ee89
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
+//! to interact with nova-core.
+
+use crate::gpu::Gpu;
+
+/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
+/// the GPU.
+pub struct AuxData<'bound> {
+    pub(crate) _gpu: &'bound Gpu<'bound>,
+}
drivers/gpu/nova-core/driver.rs +33 -12

72 lines not on this page · raw message

drivers/gpu/nova-core/gsp/hal.rs +1 -1
index d3e47ef206de..06647b2b7894 100644
@@ -36,7 +36,7 @@
 /// The GSP unload code might run in a situation where we cannot load firmware dynamically (e.g.
 /// because we are in shutdown and the file system is not accessible anymore). Thus, the firmware
 /// required for unloading is prepared at load time, and stored here until it needs to be run.
-pub(super) trait UnloadBundle: Send {
+pub(super) trait UnloadBundle: Send + Sync {
     /// Performs the steps required to properly reset the GSP after it has been stopped.
     fn run(
         &self,
drivers/gpu/nova-core/nova_core.rs +1 -0
index 735b8e17c6b6..e463f315635f 100644
@@ -13,6 +13,7 @@
 #[macro_use]
 mod bitfield;
 
+pub mod auxdata;
 mod driver;
 mod falcon;
 mod fb;
Signature · 1 line
2.54.0
maintainer
#2 lore raw

Re: [RFC 1/3] gpu: nova-core: Add auxiliary bus registration data to nova-core

drivers/gpu/nova-core/auxdata.rs

Quoted 5 lines · Alistair Popple
drivers/gpu/nova-core/auxdata.rs new file +0 -0
new file mode 100644
index 000000000000..266b5ce4ee89

I think it'd just call it something along the lines of api.rs, as this is going to be the file where the entry points into nova-core will live.

drivers/gpu/nova-core/auxdata.rs line 10

Quoted 11 lines
drivers/gpu/nova-core/auxdata.rs new file +10 -0
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
+//! to interact with nova-core.
+
+use crate::gpu::Gpu;
+
+/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
+/// the GPU.
+pub struct AuxData<'bound> {

Since this will also be the data type that will be exposed as an API entry point into nova-core, I'd rather call it e.g. NovaCoreApi.

We should provide an assoicated function NovaCoreApi::of(), which takes an &'a auxiliary::Device<Bound> as argument and returns &'a NovaCoreApi<'a> (at least as long as the type remains covariant).

This makes it a bit cleaner since it keeps the type assertion local to a single place and makes it transparent to nova-drm that this also is the auxiliary::Registration private data of nova-core.

(I'm already working on getting rid of the type ID check, but going through NovaCoreApi::of() is cleaner regardless.)

drivers/gpu/nova-core/auxdata.rs line 11

Quoted 1 line
drivers/gpu/nova-core/auxdata.rs new file +1 -0
+    pub(crate) _gpu: &'bound Gpu<'bound>,

I'd store this as Pin<&'bound Gpu<'bound>> as it better documents the pinning guarantee and allows calling Pin<&Self> methods on Gpu.

drivers/gpu/nova-core/auxdata.rs fn probe<'bound>( · line 85

Quoted 6 lines
drivers/gpu/nova-core/auxdata.rs new file +1 -1
@@ -78,7 +82,7 @@ fn probe<'bound>(
             pdev.enable_device_mem()?;
             pdev.set_master();
 
-            Ok(try_pin_init!(NovaCore {
+            Ok(try_pin_init!(&this in NovaCore {

This change shouldn't be needed.

Quoted 29 lines
drivers/gpu/nova-core/auxdata.rs new file +13 -9
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 // TODO: Use `&bar` self-referential pin-init syntax once available.
                 //
@@ -86,15 +90,32 @@ fn probe<'bound>(
                 // (`try_pin_init!()` initializes fields in declaration order), lives at a pinned
                 // stable address, and is dropped after `gpu` (struct field drop order).
                 gpu <- Gpu::new(pdev, unsafe { &*core::ptr::from_ref(bar) }),
-                _reg: auxiliary::Registration::new(
-                    pdev.as_ref(),
-                    c"nova-drm",
-                    // TODO[XARR]: Use XArray or perhaps IDA for proper ID allocation/recycling. For
-                    // now, use a simple atomic counter that never recycles IDs.
-                    AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
-                    crate::MODULE_NAME,
-                    (),
-                )?,
+
+                // SAFETY:
+                // - `NovaCore` is dropped when the device is unbound; i.e.
+                //   `mem::forget()` is never called on it.
+                // - `gpu` is initialized above, lives at a pinned stable
+                //   address, and is dropped after `_reg` (struct field drop
+                //   order).
+                _reg: unsafe {
+                    auxiliary::Registration::new_with_lt(
+                        pdev.as_ref(),
+                        c"nova-drm",
+                        // TODO[XARR]: Use XArray or perhaps IDA for proper ID allocation/recycling.
+                        // For now, use a simple atomic counter that never recycles IDs.

drivers/gpu/nova-core/auxdata.rs fn probe<'bound>( · line 115

Quoted 10 lines
drivers/gpu/nova-core/auxdata.rs new file +10 -0
+                        AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
+                        crate::MODULE_NAME,
+                        AuxData {
+                            // TODO: Use `&gpu` self-referential pin-init syntax once available.
+                            //
+                            // SAFETY: `this.gpu` is initialized before this expression is evaluated
+                            // (`try_pin_init!()` initializes fields in declaration order), lives at
+                            // a pinned stable address, and is dropped after `_reg` (struct field
+                            // drop order).
+                            _gpu: &*core::ptr::from_ref(&this.as_ref().gpu),

This is UB, we can't create a reference from 'this' before it is fully initialized.

	gpu: &*core::ptr::from_ref(gpu.as_ref().get_ref()),

should work instead and also gets us rid of this entirely. Or rather

	gpu: Pin::new_unchecked(
	    &*core::ptr::from_ref(gpu.as_ref().get_ref()),
	),

if stored as Pin<&'bound Gpu<'bound>>.

Quoted 6 lines
drivers/gpu/nova-core/auxdata.rs new file +3 -0
+                        },
+                    )?
+                },
             }))
         })
     }
maintainer
#3 lore raw

Re: [RFC 1/3] gpu: nova-core: Add auxiliary bus registration data to nova-core

Quoted 29 lines · Danilo Krummrich

drivers/gpu/nova-core/auxdata.rs

Quoted 5 lines · Alistair Popple
drivers/gpu/nova-core/auxdata.rs new file +0 -0
new file mode 100644
index 000000000000..266b5ce4ee89

I think it'd just call it something along the lines of api.rs, as this is going to be the file where the entry points into nova-core will live.

drivers/gpu/nova-core/auxdata.rs line 10

Quoted 11 lines
drivers/gpu/nova-core/auxdata.rs new file +10 -0
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
+//! to interact with nova-core.
+
+use crate::gpu::Gpu;
+
+/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
+/// the GPU.
+pub struct AuxData<'bound> {

Since this will also be the data type that will be exposed as an API entry point into nova-core, I'd rather call it e.g. NovaCoreApi.

We should provide an assoicated function NovaCoreApi::of(), which takes an &'a auxiliary::Device<Bound> as argument and returns &'a NovaCoreApi<'a> (at least as long as the type remains covariant).

Quoted 6 lines

This makes it a bit cleaner since it keeps the type assertion local to a single place and makes it transparent to nova-drm that this also is the auxiliary::Registration private data of nova-core.

(I'm already working on getting rid of the type ID check, but going through NovaCoreApi::of() is cleaner regardless.)

Thinking a bit more about this, I'm still not convinced about a design where the auxiliary::Device itself carries the type information. However, I think with the new lifetime design this isn't necessary in the first place.

We can just call NovaCoreApi::of() once from nova-drm's probe() function and subsequently store the struct NovaCoreApi<'a> in the bus device private data, DRM registration data, etc. Such that the only fallible path is in probe().

Please see [1] for reference, there's also a branch in [2].

Note that this only works for a covariant NovaCoreApi, once it becomes invariant we'll need something like this:

	/// Closure-based API handle for invariant registration data types.
	pub struct NovaCoreApiHandle<'a> {
	    adev: &'a auxiliary::Device<Bound>,
	}
	
	impl<'a> NovaCoreApiHandle<'a> {
	    fn of(adev: &'a auxiliary::Device<Bound>) -> Result<Self> {
	        adev.registration_data_with::<ForLt!(NovaCoreApi<'_>), ()>(|_| ())?;
	        Ok(Self { adev })
	    }
	
	    /// Access the [`NovaCoreApi`] through a closure.
	    pub fn with<R>(&self, f: impl for<'b> FnOnce(Pin<&'b NovaCoreApi<'b>>) -> R) -> R {
	        // PANIC: TypeId was validated in `of()`, cannot fail.
	        self.adev
	            .registration_data_with::<ForLt!(NovaCoreApi<'_>), R>(f)
	            .unwrap()
	    }
	}

We can still improve this a bit from the auxiliary API side, but in general this should be fine (added this to the branch in [2] as well).

With this the access patterns becomes

	// Once from probe(), calls NovaCoreApiHandle::of() internally.
	let handle = NovaCoreApi::handle(adev)?;

and from anywhere else

	handle.with(|api| api.chipset());

[1]

drivers/gpu/drm/nova/driver.rs +15 -2
index 739690bc2db5..89da0f6c3ba5 100644
@@ -1,5 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+use core::pin::Pin;
+
 use kernel::{
     auxiliary,
     device::{
@@ -17,6 +19,7 @@
 
 use crate::file::File;
 use crate::gem::NovaObject;
+use nova_core::api::NovaCoreApi;
 
 pub(crate) struct NovaDriver;
 
@@ -26,6 +29,11 @@ pub(crate) struct Nova<'bound> {
     _reg: drm::Registration<'bound, NovaDriver>,
 }
 
+/// DRM registration data, accessible from ioctl handlers via the registration guard.
+pub(crate) struct DrmRegData<'bound> {
+    pub(crate) api: Pin<&'bound NovaCoreApi<'bound>>,
+}
+
 /// Convienence type alias for the DRM device type for this driver
 pub(crate) type NovaDevice<Ctx = drm::Normal> = drm::Device<NovaDriver, Ctx>;
 
@@ -59,10 +67,15 @@ fn probe<'bound>(
         adev: &'bound auxiliary::Device<Core<'_>>,
         _info: &'bound Self::IdInfo,
     ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+        let api = NovaCoreApi::of(adev)?;
+
         let drm = drm::UnregisteredDevice::<Self>::new(adev, Ok(()))?;
+
+        let drm_data = DrmRegData { api };
+
         // SAFETY: `reg` is stored in `Nova` and dropped when the driver is unbound; it is
         // never forgotten.
-        let reg = unsafe { drm::Registration::new(adev.as_ref(), drm, (), 0)? };
+        let reg = unsafe { drm::Registration::new(adev.as_ref(), drm, drm_data, 0)? };
 
         Ok(Nova {
             drm: reg.device().into(),
@@ -74,7 +87,7 @@ fn probe<'bound>(
 #[vtable]
 impl drm::Driver for NovaDriver {
     type Data = ();
-    type RegistrationData<'a> = ();
+    type RegistrationData<'a> = DrmRegData<'a>;
     type File = File;
     type Object = gem::Object<NovaObject>;
     type ParentDevice<Ctx: DeviceContext> = auxiliary::Device<Ctx>;
drivers/gpu/drm/nova/file.rs +15 -12
index 57df54ba27bd..2488825c78cf 100644
@@ -1,7 +1,14 @@
 // SPDX-License-Identifier: GPL-2.0
 
-use crate::driver::{NovaDevice, NovaDriver};
-use crate::gem::NovaObject;
+use crate::{
+    driver::{
+        DrmRegData,
+        NovaDevice,
+        NovaDriver, //
+    },
+    gem::NovaObject,
+};
+
 use kernel::{
     alloc::flags::*,
     auxiliary,
@@ -16,9 +23,6 @@
     uapi,
 };
 
-use kernel::types::CovariantForLt;
-use nova_core::auxdata::AuxData;
-
 pub(crate) struct File;
 
 impl drm::file::DriverFile for File {
@@ -33,18 +37,17 @@ impl File {
     /// IOCTL: get_param: Query GPU / driver metadata.
     pub(crate) fn get_param(
         dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        reg_data: &DrmRegData<'_>,
         getparam: &mut uapi::drm_nova_getparam,
         _file: &drm::File<File>,
     ) -> Result<u32> {
         let adev: &auxiliary::Device<Bound> = dev.as_ref();
-        let pdev: &pci::Device<Bound> = adev.parent().try_into()?;
-        let data = adev.registration_data::<CovariantForLt!(AuxData<'_>)>()?;
+        let pdev: &pci::Device<_> = adev.parent().try_into()?;
 
         let value = match getparam.param as u32 {
             uapi::NOVA_GETPARAM_VRAM_BAR_SIZE => pdev.resource_len(1)?,
-            uapi::NOVA_GETPARAM_GPU_CHIPSET => data.chipset() as u64,
-            uapi::NOVA_GETPARAM_VRAM_SIZE => data.vram_size(),
+            uapi::NOVA_GETPARAM_GPU_CHIPSET => reg_data.api.chipset() as u64,
+            uapi::NOVA_GETPARAM_VRAM_SIZE => reg_data.api.vram_size(),
             _ => return Err(EINVAL),
         };
 
@@ -56,7 +59,7 @@ pub(crate) fn get_param(
     /// IOCTL: gem_create: Create a new DRM GEM object.
     pub(crate) fn gem_create(
         dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        _reg_data: &DrmRegData<'_>,
         req: &mut uapi::drm_nova_gem_create,
         file: &drm::File<File>,
     ) -> Result<u32> {
@@ -70,7 +73,7 @@ pub(crate) fn gem_create(
     /// IOCTL: gem_info: Query GEM metadata.
     pub(crate) fn gem_info(
         _dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        _reg_data: &DrmRegData<'_>,
         req: &mut uapi::drm_nova_gem_info,
         file: &drm::File<File>,
     ) -> Result<u32> {
drivers/gpu/nova-core/api.rs new file +39 -0
new file mode 100644
index 000000000000..ba99d5a27805
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova Core API for auxiliary bus child drivers.
+
+use core::pin::Pin;
+
+use kernel::{
+    auxiliary,
+    device::Bound,
+    prelude::*,
+    types::CovariantForLt, //
+};
+
+use crate::gpu::{
+    Chipset,
+    Gpu, //
+};
+
+/// API handle for auxiliary bus child drivers to interact with nova-core.
+pub struct NovaCoreApi<'bound> {
+    pub(crate) gpu: Pin<&'bound Gpu<'bound>>,
+}
+
+impl NovaCoreApi<'_> {
+    /// Obtain a [`NovaCoreApi`] handle from an auxiliary device registered by nova-core.
+    pub fn of(adev: &auxiliary::Device<Bound>) -> Result<Pin<&NovaCoreApi<'_>>> {
+        adev.registration_data::<CovariantForLt!(NovaCoreApi<'_>)>()
+    }
+
+    /// Returns the chipset of this GPU.
+    pub fn chipset(&self) -> Chipset {
+        self.gpu.spec.chipset
+    }
+
+    /// Returns the total usable VRAM size of this GPU in bytes.
+    pub fn vram_size(&self) -> u64 {
+        self.gpu.gsp_static_info.vram_size()
+    }
+}
drivers/gpu/nova-core/auxdata.rs deleted +0 -25
deleted file mode 100644
index 0a07a10da327..000000000000
@@ -1,25 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
-//! to interact with nova-core.
-
-use crate::gpu::Chipset;
-use crate::gpu::Gpu;
-
-/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
-/// the GPU.
-pub struct AuxData<'bound> {
-    pub(crate) gpu: &'bound Gpu<'bound>,
-}
-
-impl AuxData<'_> {
-    /// Returns the chipset of this GPU.
-    pub fn chipset(&self) -> Chipset {
-        self.gpu.spec.chipset
-    }
-
-    /// Returns the total usable VRAM size of this GPU in bytes.
-    pub fn vram_size(&self) -> u64 {
-        self.gpu.gsp_static_info.vram_size()
-    }
-}
drivers/gpu/nova-core/driver.rs +10 -6
index 8b0bebee0bdf..9cb2eecfae1c 100644
@@ -1,5 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+use core::pin::Pin;
+
 use kernel::{
     auxiliary,
     device::Core,
@@ -18,7 +20,7 @@
     types::CovariantForLt,
 };
 
-use crate::auxdata::AuxData;
+use crate::api::NovaCoreApi;
 use crate::gpu::Gpu;
 
 /// Counter for generating unique auxiliary device IDs.
@@ -30,7 +32,7 @@ pub(crate) struct NovaCore<'bound> {
     // device before dropping `gpu`, and drop `gpu` before `bar` because `Gpu`
     // borrows `bar`.
     #[allow(clippy::type_complexity)]
-    _reg: auxiliary::Registration<'bound, CovariantForLt!(AuxData<'_>)>,
+    _reg: auxiliary::Registration<'bound, CovariantForLt!(NovaCoreApi<'_>)>,
     #[pin]
     pub(crate) gpu: Gpu<'bound>,
     bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -82,7 +84,7 @@ fn probe<'bound>(
             pdev.enable_device_mem()?;
             pdev.set_master();
 
-            Ok(try_pin_init!(&this in NovaCore {
+            Ok(try_pin_init!(NovaCore {
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 // TODO: Use `&bar` self-referential pin-init syntax once available.
                 //
@@ -105,14 +107,16 @@ fn probe<'bound>(
                         // For now, use a simple atomic counter that never recycles IDs.
                         AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
                         crate::MODULE_NAME,
-                        AuxData {
+                        NovaCoreApi {
                             // TODO: Use `&gpu` self-referential pin-init syntax once available.
                             //
-                            // SAFETY: `this.gpu` is initialized before this expression is evaluated
+                            // SAFETY: `gpu` is initialized before this expression is evaluated
                             // (`try_pin_init!()` initializes fields in declaration order), lives at
                             // a pinned stable address, and is dropped after `_reg` (struct field
                             // drop order).
-                            gpu: &*core::ptr::from_ref(&this.as_ref().gpu),
+                            gpu: Pin::new_unchecked(
+                                &*core::ptr::from_ref(gpu.as_ref().get_ref()),
+                            ),
                         },
                     )?
                 },
drivers/gpu/nova-core/nova_core.rs +1 -1
index d00c8804269d..9463ae038e81 100644
@@ -10,7 +10,7 @@
     InPlaceModule, //
 };
 
-pub mod auxdata;
+pub mod api;
 mod driver;
 mod falcon;
 mod fb;

[2] https://git.kernel.org/pub/scm/linux/kernel/git/dakr/linux.git/log/?h=nova-api

#4 lore raw

Re: [RFC 1/3] gpu: nova-core: Add auxiliary bus registration data to nova-core

On 2026-07-06 at 03:09 +1000, Danilo Krummrich <[email protected]> wrote...

Quoted 11 lines · Danilo Krummrich
Quoted 10 lines · Alistair Popple

drivers/gpu/nova-core/auxdata.rs

Quoted 5 lines
drivers/gpu/nova-core/auxdata.rs new file +0 -0
new file mode 100644
index 000000000000..266b5ce4ee89

I think it'd just call it something along the lines of api.rs, as this is going to be the file where the entry points into nova-core will live.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/nova-core/auxdata.rs new file +5 -0
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
+//! to interact with nova-core.
+
Quoted 8 lines

drivers/gpu/nova-core/auxdata.rs line 10

Quoted 5 lines
drivers/gpu/nova-core/auxdata.rs new file +5 -0
+use crate::gpu::Gpu;
+
+/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
+/// the GPU.
+pub struct AuxData<'bound> {

Since this will also be the data type that will be exposed as an API entry point into nova-core, I'd rather call it e.g. NovaCoreApi.

Heh. Yeah, I didn't expect this name to survive :-) It was more a straw-man that I forgot to cleanup - NovaCoreApi works for me.

Quoted 11 lines
Quoted 10 lines

We should provide an assoicated function NovaCoreApi::of(), which takes an &'a auxiliary::Device<Bound> as argument and returns &'a NovaCoreApi<'a> (at least as long as the type remains covariant).

This makes it a bit cleaner since it keeps the type assertion local to a single place and makes it transparent to nova-drm that this also is the auxiliary::Registration private data of nova-core.

(I'm already working on getting rid of the type ID check, but going through NovaCoreApi::of() is cleaner regardless.)

Quoted 7 lines

Thinking a bit more about this, I'm still not convinced about a design where the auxiliary::Device itself carries the type information. However, I think with the new lifetime design this isn't necessary in the first place.

We can just call NovaCoreApi::of() once from nova-drm's probe() function and subsequently store the struct NovaCoreApi<'a> in the bus device private data, DRM registration data, etc. Such that the only fallible path is in probe().

I agree - and passing a handle to nova_core to access data was my original instinct and unposted implementation. Of course prior to the higher-ranked lifetime series that obviously involved some unsafe code (and possibly actual UB).

Now that we can safely store the handle in the DRM registration I think this looks cleaner - I just couldn't quite figure out the mechanics so thanks for the pointers. I will incorporate this into v2.

Quoted 1 line

Please see [1] for reference, there's also a branch in [2].

Thanks for doing both - I appreciate seeing the proposal in the mail thread.

Quoted 2 lines

Note that this only works for a covariant NovaCoreApi, once it becomes invariant we'll need something like this:

Yep. But for now at least it is covariant so I think we can avoid doing this until needed.

- Alistair

Quoted 314 lines
	/// Closure-based API handle for invariant registration data types.
	pub struct NovaCoreApiHandle<'a> {
	    adev: &'a auxiliary::Device<Bound>,
	}
	
	impl<'a> NovaCoreApiHandle<'a> {
	    fn of(adev: &'a auxiliary::Device<Bound>) -> Result<Self> {
	        adev.registration_data_with::<ForLt!(NovaCoreApi<'_>), ()>(|_| ())?;
	        Ok(Self { adev })
	    }
	
	    /// Access the [`NovaCoreApi`] through a closure.
	    pub fn with<R>(&self, f: impl for<'b> FnOnce(Pin<&'b NovaCoreApi<'b>>) -> R) -> R {
	        // PANIC: TypeId was validated in `of()`, cannot fail.
	        self.adev
	            .registration_data_with::<ForLt!(NovaCoreApi<'_>), R>(f)
	            .unwrap()
	    }
	}

We can still improve this a bit from the auxiliary API side, but in general this should be fine (added this to the branch in [2] as well).

With this the access patterns becomes

	// Once from probe(), calls NovaCoreApiHandle::of() internally.
	let handle = NovaCoreApi::handle(adev)?;

and from anywhere else

	handle.with(|api| api.chipset());

[1]

drivers/gpu/drm/nova/driver.rs +15 -2
index 739690bc2db5..89da0f6c3ba5 100644
@@ -1,5 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+use core::pin::Pin;
+
 use kernel::{
     auxiliary,
     device::{
@@ -17,6 +19,7 @@
 
 use crate::file::File;
 use crate::gem::NovaObject;
+use nova_core::api::NovaCoreApi;
 
 pub(crate) struct NovaDriver;
 
@@ -26,6 +29,11 @@ pub(crate) struct Nova<'bound> {
     _reg: drm::Registration<'bound, NovaDriver>,
 }
 
+/// DRM registration data, accessible from ioctl handlers via the registration guard.
+pub(crate) struct DrmRegData<'bound> {
+    pub(crate) api: Pin<&'bound NovaCoreApi<'bound>>,
+}
+
 /// Convienence type alias for the DRM device type for this driver
 pub(crate) type NovaDevice<Ctx = drm::Normal> = drm::Device<NovaDriver, Ctx>;
 
@@ -59,10 +67,15 @@ fn probe<'bound>(
         adev: &'bound auxiliary::Device<Core<'_>>,
         _info: &'bound Self::IdInfo,
     ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+        let api = NovaCoreApi::of(adev)?;
+
         let drm = drm::UnregisteredDevice::<Self>::new(adev, Ok(()))?;
+
+        let drm_data = DrmRegData { api };
+
         // SAFETY: `reg` is stored in `Nova` and dropped when the driver is unbound; it is
         // never forgotten.
-        let reg = unsafe { drm::Registration::new(adev.as_ref(), drm, (), 0)? };
+        let reg = unsafe { drm::Registration::new(adev.as_ref(), drm, drm_data, 0)? };
 
         Ok(Nova {
             drm: reg.device().into(),
@@ -74,7 +87,7 @@ fn probe<'bound>(
 #[vtable]
 impl drm::Driver for NovaDriver {
     type Data = ();
-    type RegistrationData<'a> = ();
+    type RegistrationData<'a> = DrmRegData<'a>;
     type File = File;
     type Object = gem::Object<NovaObject>;
     type ParentDevice<Ctx: DeviceContext> = auxiliary::Device<Ctx>;
drivers/gpu/drm/nova/file.rs +15 -12
index 57df54ba27bd..2488825c78cf 100644
@@ -1,7 +1,14 @@
 // SPDX-License-Identifier: GPL-2.0
 
-use crate::driver::{NovaDevice, NovaDriver};
-use crate::gem::NovaObject;
+use crate::{
+    driver::{
+        DrmRegData,
+        NovaDevice,
+        NovaDriver, //
+    },
+    gem::NovaObject,
+};
+
 use kernel::{
     alloc::flags::*,
     auxiliary,
@@ -16,9 +23,6 @@
     uapi,
 };
 
-use kernel::types::CovariantForLt;
-use nova_core::auxdata::AuxData;
-
 pub(crate) struct File;
 
 impl drm::file::DriverFile for File {
@@ -33,18 +37,17 @@ impl File {
     /// IOCTL: get_param: Query GPU / driver metadata.
     pub(crate) fn get_param(
         dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        reg_data: &DrmRegData<'_>,
         getparam: &mut uapi::drm_nova_getparam,
         _file: &drm::File<File>,
     ) -> Result<u32> {
         let adev: &auxiliary::Device<Bound> = dev.as_ref();
-        let pdev: &pci::Device<Bound> = adev.parent().try_into()?;
-        let data = adev.registration_data::<CovariantForLt!(AuxData<'_>)>()?;
+        let pdev: &pci::Device<_> = adev.parent().try_into()?;
 
         let value = match getparam.param as u32 {
             uapi::NOVA_GETPARAM_VRAM_BAR_SIZE => pdev.resource_len(1)?,
-            uapi::NOVA_GETPARAM_GPU_CHIPSET => data.chipset() as u64,
-            uapi::NOVA_GETPARAM_VRAM_SIZE => data.vram_size(),
+            uapi::NOVA_GETPARAM_GPU_CHIPSET => reg_data.api.chipset() as u64,
+            uapi::NOVA_GETPARAM_VRAM_SIZE => reg_data.api.vram_size(),
             _ => return Err(EINVAL),
         };
 
@@ -56,7 +59,7 @@ pub(crate) fn get_param(
     /// IOCTL: gem_create: Create a new DRM GEM object.
     pub(crate) fn gem_create(
         dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        _reg_data: &DrmRegData<'_>,
         req: &mut uapi::drm_nova_gem_create,
         file: &drm::File<File>,
     ) -> Result<u32> {
@@ -70,7 +73,7 @@ pub(crate) fn gem_create(
     /// IOCTL: gem_info: Query GEM metadata.
     pub(crate) fn gem_info(
         _dev: &NovaDevice<Registered>,
-        _reg_data: &(),
+        _reg_data: &DrmRegData<'_>,
         req: &mut uapi::drm_nova_gem_info,
         file: &drm::File<File>,
     ) -> Result<u32> {
drivers/gpu/nova-core/api.rs new file +39 -0
new file mode 100644
index 000000000000..ba99d5a27805
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Nova Core API for auxiliary bus child drivers.
+
+use core::pin::Pin;
+
+use kernel::{
+    auxiliary,
+    device::Bound,
+    prelude::*,
+    types::CovariantForLt, //
+};
+
+use crate::gpu::{
+    Chipset,
+    Gpu, //
+};
+
+/// API handle for auxiliary bus child drivers to interact with nova-core.
+pub struct NovaCoreApi<'bound> {
+    pub(crate) gpu: Pin<&'bound Gpu<'bound>>,
+}
+
+impl NovaCoreApi<'_> {
+    /// Obtain a [`NovaCoreApi`] handle from an auxiliary device registered by nova-core.
+    pub fn of(adev: &auxiliary::Device<Bound>) -> Result<Pin<&NovaCoreApi<'_>>> {
+        adev.registration_data::<CovariantForLt!(NovaCoreApi<'_>)>()
+    }
+
+    /// Returns the chipset of this GPU.
+    pub fn chipset(&self) -> Chipset {
+        self.gpu.spec.chipset
+    }
+
+    /// Returns the total usable VRAM size of this GPU in bytes.
+    pub fn vram_size(&self) -> u64 {
+        self.gpu.gsp_static_info.vram_size()
+    }
+}
drivers/gpu/nova-core/auxdata.rs deleted +0 -25
deleted file mode 100644
index 0a07a10da327..000000000000
@@ -1,25 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-//! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
-//! to interact with nova-core.
-
-use crate::gpu::Chipset;
-use crate::gpu::Gpu;
-
-/// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
-/// the GPU.
-pub struct AuxData<'bound> {
-    pub(crate) gpu: &'bound Gpu<'bound>,
-}
-
-impl AuxData<'_> {
-    /// Returns the chipset of this GPU.
-    pub fn chipset(&self) -> Chipset {
-        self.gpu.spec.chipset
-    }
-
-    /// Returns the total usable VRAM size of this GPU in bytes.
-    pub fn vram_size(&self) -> u64 {
-        self.gpu.gsp_static_info.vram_size()
-    }
-}
drivers/gpu/nova-core/driver.rs +10 -6
index 8b0bebee0bdf..9cb2eecfae1c 100644
@@ -1,5 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+use core::pin::Pin;
+
 use kernel::{
     auxiliary,
     device::Core,
@@ -18,7 +20,7 @@
     types::CovariantForLt,
 };
 
-use crate::auxdata::AuxData;
+use crate::api::NovaCoreApi;
 use crate::gpu::Gpu;
 
 /// Counter for generating unique auxiliary device IDs.
@@ -30,7 +32,7 @@ pub(crate) struct NovaCore<'bound> {
     // device before dropping `gpu`, and drop `gpu` before `bar` because `Gpu`
     // borrows `bar`.
     #[allow(clippy::type_complexity)]
-    _reg: auxiliary::Registration<'bound, CovariantForLt!(AuxData<'_>)>,
+    _reg: auxiliary::Registration<'bound, CovariantForLt!(NovaCoreApi<'_>)>,
     #[pin]
     pub(crate) gpu: Gpu<'bound>,
     bar: pci::Bar<'bound, BAR0_SIZE>,
@@ -82,7 +84,7 @@ fn probe<'bound>(
             pdev.enable_device_mem()?;
             pdev.set_master();
 
-            Ok(try_pin_init!(&this in NovaCore {
+            Ok(try_pin_init!(NovaCore {
                 bar: pdev.iomap_region_sized::<BAR0_SIZE>(0, c"nova-core/bar0")?,
                 // TODO: Use `&bar` self-referential pin-init syntax once available.
                 //
@@ -105,14 +107,16 @@ fn probe<'bound>(
                         // For now, use a simple atomic counter that never recycles IDs.
                         AUXILIARY_ID_COUNTER.fetch_add(1, Relaxed),
                         crate::MODULE_NAME,
-                        AuxData {
+                        NovaCoreApi {
                             // TODO: Use `&gpu` self-referential pin-init syntax once available.
                             //
-                            // SAFETY: `this.gpu` is initialized before this expression is evaluated
+                            // SAFETY: `gpu` is initialized before this expression is evaluated
                             // (`try_pin_init!()` initializes fields in declaration order), lives at
                             // a pinned stable address, and is dropped after `_reg` (struct field
                             // drop order).
-                            gpu: &*core::ptr::from_ref(&this.as_ref().gpu),
+                            gpu: Pin::new_unchecked(
+                                &*core::ptr::from_ref(gpu.as_ref().get_ref()),
+                            ),
                         },
                     )?
                 },
drivers/gpu/nova-core/nova_core.rs +1 -1
index d00c8804269d..9463ae038e81 100644
@@ -10,7 +10,7 @@
     InPlaceModule, //
 };
 
-pub mod auxdata;
+pub mod api;
 mod driver;
 mod falcon;
 mod fb;

[2] https://git.kernel.org/pub/scm/linux/kernel/git/dakr/linux.git/log/?h=nova-api

#5 lore raw

RFC 2/3 drm: nova: Add GETPARAM parameter to read the GPU chipset

In reply to Alistair Popple

One of the first things a user needs to know about a GPU is its chipset, so add an ioctl to return that.

---
 drivers/gpu/drm/nova/file.rs     |  5 +++++
 drivers/gpu/nova-core/auxdata.rs | 10 +++++++++-
 drivers/gpu/nova-core/driver.rs  |  2 +-
 drivers/gpu/nova-core/gpu.rs     | 10 +++++-----
 include/uapi/drm/nova_drm.h      |  7 +++++++
 5 files changed, 27 insertions(+), 7 deletions(-)
drivers/gpu/drm/nova/file.rs +5 -0
index 298c02bacb4b..8147dfb366d8 100644
@@ -16,6 +16,9 @@
     uapi,
 };
 
+use kernel::types::ForLt;
+use nova_core::auxdata::AuxData;
+
 pub(crate) struct File;
 
 impl drm::file::DriverFile for File {
@@ -36,9 +39,11 @@ pub(crate) fn get_param(
     ) -> Result<u32> {
         let adev: &auxiliary::Device<Bound> = dev.as_ref();
         let pdev: &pci::Device<Bound> = adev.parent().try_into()?;
+        let data = adev.registration_data::<ForLt!(AuxData<'_>)>()?;
 
         let value = match getparam.param as u32 {
             uapi::NOVA_GETPARAM_VRAM_BAR_SIZE => pdev.resource_len(1)?,
+            uapi::NOVA_GETPARAM_GPU_CHIPSET => data.chipset() as u64,
             _ => return Err(EINVAL),
         };
 
drivers/gpu/nova-core/auxdata.rs +9 -1
index 266b5ce4ee89..cd8a3511293b 100644
@@ -3,10 +3,18 @@
 //! Nova-core auxbus data. Contains all the methods used by the auxbus drivers
 //! to interact with nova-core.
 
+use crate::gpu::Chipset;
 use crate::gpu::Gpu;
 
 /// Auxiliary bus registration data. Used by the auxbus drivers to call methods on
 /// the GPU.
 pub struct AuxData<'bound> {
-    pub(crate) _gpu: &'bound Gpu<'bound>,
+    pub(crate) gpu: &'bound Gpu<'bound>,
+}
+
+impl AuxData<'_> {
+    /// Returns the chipset of this GPU.
+    pub fn chipset(&self) -> Chipset {
+        self.gpu.spec.chipset
+    }
 }
drivers/gpu/nova-core/driver.rs +1 -1
index ad232ca11833..193e87ac624e 100644
@@ -112,7 +112,7 @@ fn probe<'bound>(
                             // (`try_pin_init!()` initializes fields in declaration order), lives at
                             // a pinned stable address, and is dropped after `_reg` (struct field
                             // drop order).
-                            _gpu: &*core::ptr::from_ref(&this.as_ref().gpu),
+                            gpu: &*core::ptr::from_ref(&this.as_ref().gpu),
                         },
                     )?
                 },
drivers/gpu/nova-core/gpu.rs +5 -5
index 43c3f4f8df71..d4a71a1be6f1 100644
@@ -38,7 +38,7 @@ macro_rules! define_chipset {
     {
         /// Enum representation of the GPU chipset.
         #[derive(fmt::Debug, Copy, Clone, PartialOrd, Ord, PartialEq, Eq)]
-        pub(crate) enum Chipset {
+        pub enum Chipset {
             $($variant = $value),*,
         }
 
@@ -114,7 +114,7 @@ fn try_from(value: u32) -> Result<Self, Self::Error> {
 });
 
 impl Chipset {
-    pub(crate) const fn arch(self) -> Architecture {
+    pub const fn arch(self) -> Architecture {
         match self {
             Self::TU102 | Self::TU104 | Self::TU106 | Self::TU117 | Self::TU116 => {
                 Architecture::Turing
@@ -172,7 +172,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
 bounded_enum! {
     /// Enum representation of the GPU generation.
     #[derive(fmt::Debug, Copy, Clone)]
-    pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> {
+    pub enum Architecture with TryFrom<Bounded<u32, 6>> {
         Turing = 0x16,
         Ampere = 0x17,
         Hopper = 0x18,
@@ -206,7 +206,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
 /// Structure holding a basic description of the GPU: `Chipset` and `Revision`.
 #[derive(Clone, Copy)]
 pub(crate) struct Spec {
-    chipset: Chipset,
+    pub(crate) chipset: Chipset,
     revision: Revision,
 }
 
@@ -286,7 +286,7 @@ struct GspResources<'gpu> {
 /// Structure holding the resources required to operate the GPU.
 #[pin_data]
 pub(crate) struct Gpu<'gpu> {
-    spec: Spec,
+    pub(crate) spec: Spec,
     /// Static GPU information as provided by the GSP.
     gsp_static_info: GetGspStaticInfoReply,
     /// GSP and its resources.
include/uapi/drm/nova_drm.h +7 -0
index 3ca90ed9d2bb..63440f5dca9b 100644
@@ -25,6 +25,13 @@ extern "C" {
  */
 #define NOVA_GETPARAM_VRAM_BAR_SIZE	0x1
 
+/*
+ * NOVA_GETPARAM_GPU_CHIPSET
+ *
+ * Query the GPU chipset architecture/implementation.
+ */
+#define NOVA_GETPARAM_GPU_CHIPSET	0x2
+
 /**
  * struct drm_nova_getparam - query GPU and driver metadata
  */
Signature · 1 line
2.54.0
#6 lore raw

RFC 3/3 drm: nova: Add a GETPARAM parameter to read usable VRAM size

In reply to Alistair Popple

Add a GET_PARAM ioctl to return the total usable framebuffer size. The usable framebuffer excludes GSP carveouts and other protected regions so may differ in size from BAR size and total physical VRAM.

---
 drivers/gpu/drm/nova/file.rs          | 1 +
 drivers/gpu/nova-core/auxdata.rs      | 5 +++++
 drivers/gpu/nova-core/gpu.rs          | 6 ++----
 drivers/gpu/nova-core/gsp/commands.rs | 8 ++++++++
 include/uapi/drm/nova_drm.h           | 7 +++++++
 5 files changed, 23 insertions(+), 4 deletions(-)
drivers/gpu/drm/nova/file.rs +1 -0
index 8147dfb366d8..d6f7e21fbc48 100644
@@ -44,6 +44,7 @@ pub(crate) fn get_param(
         let value = match getparam.param as u32 {
             uapi::NOVA_GETPARAM_VRAM_BAR_SIZE => pdev.resource_len(1)?,
             uapi::NOVA_GETPARAM_GPU_CHIPSET => data.chipset() as u64,
+            uapi::NOVA_GETPARAM_VRAM_SIZE => data.vram_size(),
             _ => return Err(EINVAL),
         };
 
drivers/gpu/nova-core/auxdata.rs +5 -0
index cd8a3511293b..0a07a10da327 100644
@@ -17,4 +17,9 @@ impl AuxData<'_> {
     pub fn chipset(&self) -> Chipset {
         self.gpu.spec.chipset
     }
+
+    /// Returns the total usable VRAM size of this GPU in bytes.
+    pub fn vram_size(&self) -> u64 {
+        self.gpu.gsp_static_info.vram_size()
+    }
 }
drivers/gpu/nova-core/gpu.rs +2 -4
index d4a71a1be6f1..a089843f6d0a 100644
@@ -288,7 +288,7 @@ struct GspResources<'gpu> {
 pub(crate) struct Gpu<'gpu> {
     pub(crate) spec: Spec,
     /// Static GPU information as provided by the GSP.
-    gsp_static_info: GetGspStaticInfoReply,
+    pub(crate) gsp_static_info: GetGspStaticInfoReply,
     /// GSP and its resources.
     #[pin]
     gsp_resources: GspResources<'gpu>,
@@ -388,9 +388,7 @@ pub(crate) fn new(
                     dev_dbg!(
                         pdev,
                         "Total usable VRAM: {} MiB\n",
-                        info.usable_fb_regions.iter().fold(0u64, |res, region| res
-                            .saturating_add(region.end - region.start))
-                            / u64::SZ_1M
+                        info.vram_size() / u64::SZ_1M
                     );
                 }
 
drivers/gpu/nova-core/gsp/commands.rs +8 -0
index 86a3747cd31c..e41227fc015d 100644
@@ -242,6 +242,14 @@ pub(crate) fn gpu_name(&self) -> core::result::Result<&str, GpuNameError> {
             .to_str()
             .map_err(GpuNameError::InvalidUtf8)
     }
+
+    /// Returns the total usable VRAM size in bytes, i.e. the summed lengths of all usable FB
+    /// regions.
+    pub(crate) fn vram_size(&self) -> u64 {
+        self.usable_fb_regions
+            .iter()
+            .fold(0, |size, region| size.saturating_add(region.end - region.start))
+    }
 }
 
 pub(crate) use fw::commands::PowerStateLevel;
include/uapi/drm/nova_drm.h +7 -0
index 63440f5dca9b..eb98817906a8 100644
@@ -32,6 +32,13 @@ extern "C" {
  */
 #define NOVA_GETPARAM_GPU_CHIPSET	0x2
 
+/*
+ * NOVA_GETPARAM_VRAM_SIZE
+ *
+ * Query the total usable VRAM size in bytes.
+ */
+#define NOVA_GETPARAM_VRAM_SIZE	0x3
+
 /**
  * struct drm_nova_getparam - query GPU and driver metadata
  */
Signature · 1 line
2.54.0