gpu: nova-core: Add memory management support
Subsystems (8)
v9 cover gpu: nova-core: Add memory management support
This series adds support for nova-core memory management, including VRAM allocation, PRAMIN, VMM, page table walking, and BAR 1 read/writes. These are critical for channel management, vGPU, and all other memory management uses of nova-core.
It is based on linux -next and all patches, along with all the dependencies (such as buddy bindings, CList), can be found at:git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git (tag: nova-mm-v9-20260310)
Change log:
Changes from v8 to v9:
- Added fixes from Zhi Wang for bitfield position changes in virtual address and larger BAR1 size on some platforms. Tested and working for vGPU usecase!
- Refactored gsp: boot() to return only GspStaticInfo, removing FbLayout (Alex)
- bar1_pde_base and bar2_pde_base are now accessed via GspStaticInfo directly (Alex)
- Added new patch "gsp: Expose total physical VRAM end from FB region info" introducing total_fb_end() to expose VRAM extent. (Alex)
- Consolidated usable VRAM and BarUser setup; removed dedicated
"fb: Add usable_vram field to FbLayout", "mm: Use usable VRAM region for buddy allocator", and "mm: Add BarUser to struct Gpu and create at boot".
Changes from v7 to v8:
- Incorporated "Select GPU_BUDDY for VRAM allocation" patch from the dependency series (Alex).
- Significant patch reordering for better logical flow (GSP/FB patches moved earlier, page table patches, Vmm, Bar1, tests) (Alex).
- Replaced several 'as' usages with into_safe_cast() (Danilo, Alex).
- Updated BAR 1 test cases to include exercising the block size API (Eliot, Danilo).
Changes from v6 to v7:
- Addressed DMA fence signalling usecase per Danilo's feedback.
Pre v6:
- Simplified PRAMIN code (John Hubbard, Alex Courbot).
- Handled different MMU versions: ver2 versus ver3 (John Hubbard).
- Added BAR1 usecase so we have user of DRM Buddy / VMM (John Hubbard).
- Iterating over clist/buddy bindings.
Link to v8: https://lore.kernel.org/all/[email protected]/
Link to v7: https://lore.kernel.org/all/[email protected]/
Joel Fernandes (24):
gpu: nova-core: Select GPU_BUDDY for VRAM allocation
gpu: nova-core: Kconfig: Sort select statements alphabetically
gpu: nova-core: gsp: Return GspStaticInfo from boot()
gpu: nova-core: gsp: Extract usable FB region from GSP
gpu: nova-core: gsp: Expose total physical VRAM end from FB region
info
gpu: nova-core: mm: Add support to use PRAMIN windows to write to VRAM
docs: gpu: nova-core: Document the PRAMIN aperture mechanism
gpu: nova-core: mm: Add common memory management types
gpu: nova-core: mm: Add TLB flush support
gpu: nova-core: mm: Add GpuMm centralized memory manager
gpu: nova-core: mm: Add common types for all page table formats
gpu: nova-core: mm: Add MMU v2 page table types
gpu: nova-core: mm: Add MMU v3 page table types
gpu: nova-core: mm: Add unified page table entry wrapper enums
gpu: nova-core: mm: Add page table walker for MMU v2/v3
gpu: nova-core: mm: Add Virtual Memory Manager
gpu: nova-core: mm: Add virtual address range tracking to VMM
gpu: nova-core: mm: Add multi-page mapping API to VMM
gpu: nova-core: Add BAR1 aperture type and size constant
gpu: nova-core: mm: Add BAR1 user interface
gpu: nova-core: mm: Add BAR1 memory management self-tests
gpu: nova-core: mm: Add PRAMIN aperture self-tests
Zhi Wang (1):
gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
Documentation/gpu/nova/core/pramin.rst | 125 +++++ Documentation/gpu/nova/index.rst | 1 + MAINTAINERS | 14 +- drivers/gpu/nova-core/Kconfig | 13 +- drivers/gpu/nova-core/driver.rs | 3 + drivers/gpu/nova-core/gpu.rs | 90 ++- drivers/gpu/nova-core/gsp/boot.rs | 12 +- drivers/gpu/nova-core/gsp/commands.rs | 18 +- drivers/gpu/nova-core/gsp/fw/commands.rs | 59 ++ drivers/gpu/nova-core/mm.rs | 235 ++++++++ drivers/gpu/nova-core/mm/bar_user.rs | 412 ++++++++++++++ drivers/gpu/nova-core/mm/pagetable.rs | 481 ++++++++++++++++ drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 ++++++++ drivers/gpu/nova-core/mm/pagetable/ver3.rs | 337 ++++++++++++ drivers/gpu/nova-core/mm/pagetable/walk.rs | 218 ++++++++ drivers/gpu/nova-core/mm/pramin.rs | 502 +++++++++++++++++ drivers/gpu/nova-core/mm/tlb.rs | 90 +++ drivers/gpu/nova-core/mm/vmm.rs | 498 +++++++++++++++++ drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/regs.rs | 39 ++ 29 files changed, 4390 insertions(+), 10 deletions(-) create mode 100644 Documentation/gpu/nova/core/pramin.rst create mode 100644 drivers/gpu/nova-core/mm.rs create mode 100644 drivers/gpu/nova-core/mm/bar_user.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver3.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/walk.rs create mode 100644 drivers/gpu/nova-core/mm/pramin.rs create mode 100644 drivers/gpu/nova-core/mm/tlb.rs create mode 100644 drivers/gpu/nova-core/mm/vmm.rs
Signature · 1 line
2.34.1
PATCH v9 1/23 gpu: nova-core: Select GPU_BUDDY for VRAM allocation
nova-core will use the GPU buddy allocator for physical VRAM management. Enable it in Kconfig.
--- drivers/gpu/nova-core/Kconfig | 1 + 1 file changed, 1 insertion(+)
drivers/gpu/nova-core/Kconfig +1 -0
@@ -5,6 +5,7 @@ config NOVA_CORE depends on RUST select RUST_FW_LOADER_ABSTRACTIONS select AUXILIARY_BUS + select GPU_BUDDY default n help Choose this if you want to build the Nova Core driver for Nvidia
Signature · 1 line
2.34.1
Re: [PATCH v9 01/23] gpu: nova-core: Select GPU_BUDDY for VRAM allocation
Quoted 6 lines · Joel Fernandes
nova-core will use the GPU buddy allocator for physical VRAM management. Enable it in
Kconfig.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Alexandre Courbot
maintainerRe: [PATCH v9 01/23] gpu: nova-core: Select GPU_BUDDY for VRAM allocation
In reply to Joel FernandesQuoted 5 lines · Joel Fernandes
nova-core will use the GPU buddy allocator for physical VRAM management. Enable it in
Kconfig.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>
As I said in v8, let's squash this change with the first commit that actually makes use of GPU_BUDDY.
Re: [PATCH v9 01/23] gpu: nova-core: Select GPU_BUDDY for VRAM allocation
Quoted 2 lines · Alexandre Courbot
As I said in v8, let's squash this change with the first commit that actually makes use of
GPU_BUDDY.
Will do. The first commit actually using GPU_BUDDY is "Add GpuMm centralized memory manager", so I'll squash this Kconfig select into that commit in v10.
thanks,
--
Joel Fernandes
PATCH v9 2/23 gpu: nova-core: Kconfig: Sort select statements alphabetically
In reply to Joel FernandesReorder the select statements in NOVA_CORE Kconfig to be in alphabetical order.
--- drivers/gpu/nova-core/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/Kconfig +1 -1
@@ -3,9 +3,9 @@ config NOVA_CORE depends on 64BIT depends on PCI depends on RUST - select RUST_FW_LOADER_ABSTRACTIONS select AUXILIARY_BUS select GPU_BUDDY + select RUST_FW_LOADER_ABSTRACTIONS default n help Choose this if you want to build the Nova Core driver for Nvidia
Signature · 1 line
2.34.1
Re: [PATCH v9 02/23] gpu: nova-core: Kconfig: Sort select statements alphabetically
Quoted 6 lines · Joel Fernandes
Reorder the select statements in
NOVA_COREKconfigto be in alphabetical order.Suggested-by Danilo Krummrich <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Alexandre Courbot
maintainerRe: [PATCH v9 02/23] gpu: nova-core: Kconfig: Sort select statements alphabetically
In reply to Joel FernandesQuoted 5 lines · Joel Fernandes
Reorder the select statements in
NOVA_COREKconfigto be in alphabetical order.Suggested-by Danilo Krummrich <[email protected]>Signed-off-by Joel Fernandes <[email protected]>
This one is already in drm-rust-next.
Re: [PATCH v9 02/23] gpu: nova-core: Kconfig: Sort select statements alphabetically
Quoted 1 line · Alexandre Courbot
This one is already in drm-rust-next.
I'll rebase. thanks,
--
Joel Fernandes
PATCH v9 3/23 gpu: nova-core: gsp: Return GspStaticInfo from boot()
In reply to Joel FernandesRefactor the GSP boot function to return only the GspStaticInfo, removing the FbLayout from the return tuple.
This enables access required for memory management initialization to:
- bar1_pde_base: BAR1 page directory base.
- bar2_pde_base: BAR2 page directory base.
- usable memory regions in vidmem.
--- drivers/gpu/nova-core/gpu.rs | 9 +++++++-- drivers/gpu/nova-core/gsp/boot.rs | 12 ++++++++---- 2 files changed, 15 insertions(+), 6 deletions(-)
drivers/gpu/nova-core/gpu.rs +7 -2
@@ -18,7 +18,10 @@ }, fb::SysmemFlush, gfw, - gsp::Gsp, + gsp::{ + commands::GetGspStaticInfoReply, + Gsp, // + }, regs, };
@@ -252,6 +255,8 @@ pub(crate) struct Gpu {/// GSP runtime data. Temporarily an empty placeholder. #[pin] gsp: Gsp, +/// Static GPU information from GSP. + gsp_static_info: GetGspStaticInfoReply, }impl Gpu {
@@ -283,7 +288,7 @@ pub(crate) fn new<'a>( gsp <- Gsp::new(pdev), - _: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, + gsp_static_info: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, bar: devres_bar, })
drivers/gpu/nova-core/gsp/boot.rs +8 -4
@@ -32,7 +32,10 @@ }, gpu::Chipset, gsp::{ - commands, + commands::{ + self, + GetGspStaticInfoReply, // + }, sequencer::{GspSequencer ,GspSequencerParams ,//
@@ -126,7 +129,8 @@ fn run_fwsec_frts(/// user-space, patching them with signatures, and building firmware-specific intricate data /// structures that the GSP will use at runtime. /// -/// Upon return, the GSP is up and running, and its runtime object given as return value. +/// Upon return, the GSP is up and running, and static GPU information is returned. +/// pub (crate )fn boot (mut self:Pin <&mut Self >, pdev: &pci::Device<device::Bound>,
@@ -134,7 +138,7 @@ pub(crate) fn boot( chipset: Chipset, gsp_falcon: &Falcon<Gsp>, sec2_falcon: &Falcon<Sec2>, - ) -> Result { + ) -> Result<GetGspStaticInfoReply> {let dev = pdev.as_ref ();let bios =Vbios ::new (dev, bar)?;
@@ -225,6 +229,6 @@ pub(crate) fn boot( Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e), } -Ok (()) +Ok (info) } }
Signature · 1 line
2.34.1
Re: [PATCH v9 03/23] gpu: nova-core: gsp: Return GspStaticInfo from boot()
Quoted 2 lines · Joel Fernandes
Refactor the GSP boot function to return only the GspStaticInfo, removing the FbLayout from the return tuple.
I think the commit message may need updating - boot doesn't return FbLayout. And it returns GetGspStaticInfoReply, not GspStaticInfo.
Other than that,
Quoted 28 lines
drivers/gpu/nova-core/gsp/boot.rs +4
-3@@ -126,7 +129,8 @@ fn run_fwsec_frts(/// user-space, patching them with signatures, and building firmware-specific intricate data /// structures that the GSP will use at runtime. /// -/// Upon return, the GSP is up and running, and its runtime object given as return value. +/// Upon return, the GSP is up and running, and static GPU information is returned. +/// pub (crate )fn boot (mut self:Pin <&mut Self >, pdev: &pci::Device<device::Bound>,@@ -134,7 +138,7 @@ pub(crate) fn boot( chipset: Chipset, gsp_falcon: &Falcon<Gsp>, sec2_falcon: &Falcon<Sec2>, - ) -> Result { + ) -> Result<GetGspStaticInfoReply> {let dev = pdev.as_ref ();let bios =Vbios ::new (dev, bar)?;@@ -225,6 +229,6 @@ pub(crate) fn boot( Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e), } -Ok (()) +Ok (info) } }
PATCH v9 4/23 gpu: nova-core: gsp: Extract usable FB region from GSP
In reply to Joel FernandesAdd first_usable_fb_region() to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.
The extracted region is stored in GetGspStaticInfoReply and exposed via usable_fb_region() API for use by the memory subsystem.
--- drivers/gpu/nova-core/gsp/commands.rs | 11 ++++++-- drivers/gpu/nova-core/gsp/fw/commands.rs | 32 ++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 2 deletions(-)
drivers/gpu/nova-core/gsp/commands.rs +9 -2
@@ -4,6 +4,7 @@ array, convert::Infallible, ffi::FromBytesUntilNulError, + ops::Range, str::Utf8Error, // };
@@ -186,22 +187,28 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> { } } -/// The reply from the GSP to the [`GetGspInfo`] command. +/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// Usable FB (VRAM) region for driver memory allocation. +#[expect(dead_code)] +pub (crate ) usable_fb_region:Range <u64 >, }impl MessageFromGsp for GetGspStaticInfoReply {const FUNCTION :MsgFunction =MsgFunction ::GetGspStaticInfo ;type Message =GspStaticConfigInfo ; -type InitError =Infallible ; +type InitError =Error ;fn read ( msg: &Self ::Message , _sbuffer: &mut SBufferIter <array::IntoIter <&[u8 ],2 >>, ) ->Result <Self ,Self ::InitError > { +let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + usable_fb_region: base..base.saturating_add (size), }) } }
drivers/gpu/nova-core/gsp/fw/commands.rs +32 -0
@@ -5,6 +5,7 @@use kernel::{device, pci};use crate ::gsp::GSP_PAGE_SIZE ; +use crate ::num::IntoSafeCast ;use super ::bindings;
@@ -115,6 +116,37 @@ impl GspStaticConfigInfo {pub (crate )fn gpu_name_str (&self) -> [u8 ;64 ] { self.0 .gpuNameString } + +/// Extract the first usable FB region from GSP firmware data. +/// +/// Returns the first region suitable for driver memory allocation as a `(base, size)` tuple. +/// Usable regions are those that: +/// - Are not reserved for firmware internal use. +/// - Are not protected (hardware-enforced access restrictions). +/// - Support compression (can use GPU memory compression for bandwidth). +/// - Support ISO (isochronous memory for display requiring guaranteed bandwidth). +pub (crate )fn first_usable_fb_region (&self) ->Option <(u64 ,u64 )> { +let fb_info = &self.0 .fbRegionInfoParams; +for iin 0 ..fb_info.numFBRegions.into_safe_cast () { +if let Some (reg) = fb_info.fbRegion.get (i) { +// Skip malformed regions where limit < base. +if reg.limit < reg.base { +continue ; + } + +// Filter: not reserved, not protected, supports compression and ISO. +if reg.reserved ==0 + && reg.bProtected ==0 + && reg.supportCompressed !=0 + && reg.supportISO !=0 + { +let size = reg.limit - reg.base +1 ; +return Some ((reg.base, size)); + } + } + } +None + } }// SAFETY: Padding is explicit and will not contain uninitialized data.
Signature · 1 line
2.34.1
Re: [PATCH v9 04/23] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 44 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed via
usable_fb_region()API for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 11 ++++++-- drivers/gpu/nova-core/gsp/fw/commands.rs | 32 ++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 2 deletions(-)drivers/gpu/nova-core/gsp/commands.rs +6
-2@@ -4,6 +4,7 @@ array, convert::Infallible, ffi::FromBytesUntilNulError, + ops::Range, str::Utf8Error, // };@@ -186,22 +187,28 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> { } } -/// The reply from the GSP to the [`GetGspInfo`] command. +/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// Usable FB (VRAM) region for driver memory allocation. +#[expect(dead_code)] +pub (crate ) usable_fb_region:Range <u64 >, }impl MessageFromGsp for GetGspStaticInfoReply {const FUNCTION :MsgFunction =MsgFunction ::GetGspStaticInfo ;type Message =GspStaticConfigInfo ; -type InitError =Infallible ; +type InitError =Error ;
drivers/gpu/nova-core/gsp/commands.rs fn init(&self) -> impl Init<Self::Command, Self::InitError> { · line 211
Quoted 10 lines
drivers/gpu/nova-core/gsp/commands.rs +3
-0fn read ( msg: &Self ::Message , _sbuffer: &mut SBufferIter <array::IntoIter <&[u8 ],2 >>, ) ->Result <Self ,Self ::InitError > { +let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + usable_fb_region: base..base.saturating_add (size),
We already return a Result here, so why not use checked_add?:base..base.checked_add(size).ok_or(EOVERFLOW)?
Quoted 18 lines
drivers/gpu/nova-core/gsp/commands.rs +0
-0}) } }drivers/gpu/nova-core/gsp/fw/commands.rs +1
-0@@ -5,6 +5,7 @@use kernel::{device, pci};use crate ::gsp::GSP_PAGE_SIZE ; +use crate ::num::IntoSafeCast ;use super ::bindings;@@ -115,6 +116,37 @@ impl GspStaticConfigInfo {pub (crate )fn gpu_name_str (&self) -> [u8 ;64 ] { self.0 .gpuNameString
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 127
Quoted 10 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +9
-0} + +/// Extract the first usable FB region from GSP firmware data. +/// +/// Returns the first region suitable for driver memory allocation as a `(base, size)` tuple. +/// Usable regions are those that: +/// - Are not reserved for firmware internal use. +/// - Are not protected (hardware-enforced access restrictions). +/// - Support compression (can use GPU memory compression for bandwidth). +/// - Support ISO (isochronous memory for display requiring guaranteed bandwidth).
Are the above conditions all required (AND) or any required (OR)? Might be worth clarifying in the doc.
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 132
Quoted 5 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +5
-0+pub (crate )fn first_usable_fb_region (&self) ->Option <(u64 ,u64 )> { +let fb_info = &self.0 .fbRegionInfoParams; +for iin 0 ..fb_info.numFBRegions.into_safe_cast () { +if let Some (reg) = fb_info.fbRegion.get (i) { +// Skip malformed regions where limit < base.
Is it normal that it returns a bunch of broken regions?
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 144
Quoted 12 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +12
-0+if reg.limit < reg.base { +continue ; + } + +// Filter: not reserved, not protected, supports compression and ISO. +if reg.reserved ==0 + && reg.bProtected ==0 + && reg.supportCompressed !=0 + && reg.supportISO !=0 + { +let size = reg.limit - reg.base +1 ; +return Some ((reg.base, size));
This is identifying a range, so how about returning Option<Range<u64>> instead? It gets immediately converted into a range anyway.
Quoted 8 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +5
-0+ } + } + } +None + } }// SAFETY: Padding is explicit and will not contain uninitialized data.
Re: [PATCH v9 04/23] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 46 lines · Eliot Courtney
Quoted 45 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed via
usable_fb_region()API for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 11 ++++++-- drivers/gpu/nova-core/gsp/fw/commands.rs | 32 ++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 2 deletions(-)drivers/gpu/nova-core/gsp/commands.rs +6
-2@@ -4,6 +4,7 @@ array, convert::Infallible, ffi::FromBytesUntilNulError, + ops::Range, str::Utf8Error, // };@@ -186,22 +187,28 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> { } } -/// The reply from the GSP to the [`GetGspInfo`] command. +/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// Usable FB (VRAM) region for driver memory allocation. +#[expect(dead_code)] +pub (crate ) usable_fb_region:Range <u64 >, }impl MessageFromGsp for GetGspStaticInfoReply {const FUNCTION :MsgFunction =MsgFunction ::GetGspStaticInfo ;type Message =GspStaticConfigInfo ; -type InitError =Infallible ; +type InitError =Error ;
Quoted 12 lines
drivers/gpu/nova-core/gsp/commands.rs fn init(&self) -> impl Init<Self::Command, Self::InitError> { · line 211
Quoted 9 lines
drivers/gpu/nova-core/gsp/commands.rs +3
-0fn read ( msg: &Self ::Message , _sbuffer: &mut SBufferIter <array::IntoIter <&[u8 ],2 >>, ) ->Result <Self ,Self ::InitError > { +let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + usable_fb_region: base..base.saturating_add (size),We already return a Result here, so why not use
checked_add?:base..base.checked_add(size).ok_or(EOVERFLOW)?
Hmm, I think I was trying to play it safe and handle any situation of a corrupted size. But granted, it may be better to error out.
Quoted 16 lines
Quoted 16 lines
drivers/gpu/nova-core/gsp/commands.rs +0
-0}) } }drivers/gpu/nova-core/gsp/fw/commands.rs +1
-0@@ -5,6 +5,7 @@use kernel::{device, pci};use crate ::gsp::GSP_PAGE_SIZE ; +use crate ::num::IntoSafeCast ;use super ::bindings;
Quoted 16 lines
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 127
Quoted 13 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +9
-0@@ -115,6 +116,37 @@ impl GspStaticConfigInfo {pub (crate )fn gpu_name_str (&self) -> [u8 ;64 ] { self.0 .gpuNameString } + +/// Extract the first usable FB region from GSP firmware data. +/// +/// Returns the first region suitable for driver memory allocation as a `(base, size)` tuple. +/// Usable regions are those that: +/// - Are not reserved for firmware internal use. +/// - Are not protected (hardware-enforced access restrictions). +/// - Support compression (can use GPU memory compression for bandwidth). +/// - Support ISO (isochronous memory for display requiring guaranteed bandwidth). Are the above conditions all required (AND) or any required (OR)? Might be worth clarifying in the doc.
I am not sure if any clarification is needed there but I will reword to 'Usable regions are those that satisfy all of the following:'.
Quoted 8 lines
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 132
Quoted 6 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +5
-0+pub (crate )fn first_usable_fb_region (&self) ->Option <(u64 ,u64 )> { +let fb_info = &self.0 .fbRegionInfoParams; +for iin 0 ..fb_info.numFBRegions.into_safe_cast () { +if let Some (reg) = fb_info.fbRegion.get (i) { +// Skip malformed regions where limit < base. Is it normal that it returns a bunch of broken regions?
Not really. The aim was to make the code robust at a time when I was studying the FB regions. I will change the comment, and/or drop the check. Thanks for pointing it out.
Quoted 13 lines
drivers/gpu/nova-core/gsp/fw/commands.rs impl GspStaticConfigInfo { · line 144
Quoted 12 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +12
-0+if reg.limit < reg.base { +continue ; + } + +// Filter: not reserved, not protected, supports compression and ISO. +if reg.reserved ==0 + && reg.bProtected ==0 + && reg.supportCompressed !=0 + && reg.supportISO !=0 + { +let size = reg.limit - reg.base +1 ; +return Some ((reg.base, size));
Quoted 2 lines
This is identifying a range, so how about returning Option<Range<u64>> instead? It gets immediately converted into a range anyway.
Sure, that works.
--
Joel Fernandes
Alexandre Courbot
maintainerRe: [PATCH v9 04/23] gpu: nova-core: gsp: Extract usable FB region from GSP
In reply to Joel FernandesQuoted 9 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed via
usable_fb_region()API for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>
This doesn't take into account the feedback I gave in [1]. In particular, a TODO to handle the remaining regions looks important to me.
Re: [PATCH v9 04/23] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 3 lines · Alexandre Courbot
This doesn't take into account the feedback I gave in [1]. In particular, a TODO to handle the remaining regions looks important to me.
Sorry about missing the v8 feedback. Fixed in upcoming v10.
--
Joel Fernandes
PATCH v9 5/23 gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
In reply to Joel FernandesAdd total_fb_end() to GspStaticConfigInfo that computes the exclusive end address of the highest valid FB region covering both usable and GSP-reserved areas.
This allows callers to know the full physical VRAM extent, not just the allocatable portion.
--- drivers/gpu/nova-core/gsp/commands.rs | 6 ++++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+)
drivers/gpu/nova-core/gsp/commands.rs +6 -0
@@ -193,6 +193,9 @@ pub(crate) struct GetGspStaticInfoReply {/// Usable FB (VRAM) region for driver memory allocation. #[expect(dead_code)] pub(crate) usable_fb_region: Range<u64>, +/// End of VRAM. +#[expect(dead_code)] + pub(crate) total_fb_end: u64, }impl MessageFromGsp for GetGspStaticInfoReply {
@@ -206,9 +209,12 @@ fn read( ) -> Result<Self, Self::InitError> {let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +let total_fb_end = msg.total_fb_end ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), usable_fb_region: base..base.saturating_add (size), + total_fb_end, }) } }
drivers/gpu/nova-core/gsp/fw/commands.rs +19 -0
@@ -147,6 +147,25 @@ pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { }None } + +/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { +let fb_info = &self.0 .fbRegionInfoParams; +let mut max_end:Option <u64 > =None ; +for iin 0 ..fb_info.numFBRegions.into_safe_cast () { +if let Some (reg) = fb_info.fbRegion.get (i) { +if reg.limit < reg.base { +continue ; + } +let end = reg.limit.saturating_add (1 ); + max_end =Some (match max_end { +None => end, +Some (prev) => prev.max (end), + }); + } + } + max_end + } }// SAFETY: Padding is explicit and will not contain uninitialized data.
Signature · 1 line
2.34.1
Alexandre Courbot
maintainerRe: [PATCH v9 05/23] gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
Quoted 48 lines · Joel Fernandes
Add
total_fb_end()toGspStaticConfigInfothat computes the exclusive end address of the highest valid FB region covering both usable and GSP-reserved areas.This allows callers to know the full physical VRAM extent, not just the allocatable portion.
Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 6 ++++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+)drivers/gpu/nova-core/gsp/commands.rs +6
-0@@ -193,6 +193,9 @@ pub(crate) struct GetGspStaticInfoReply {/// Usable FB (VRAM) region for driver memory allocation. #[expect(dead_code)] pub(crate) usable_fb_region: Range<u64>, +/// End of VRAM. +#[expect(dead_code)] + pub(crate) total_fb_end: u64, }impl MessageFromGsp for GetGspStaticInfoReply {@@ -206,9 +209,12 @@ fn read( ) -> Result<Self, Self::InitError> {let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +let total_fb_end = msg.total_fb_end ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), usable_fb_region: base..base.saturating_add (size), + total_fb_end, }) } }drivers/gpu/nova-core/gsp/fw/commands.rs +0
-0@@ -147,6 +147,25 @@ pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { }None }
drivers/gpu/nova-core/gsp/fw/commands.rs pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { · line 159
Quoted 10 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +10
-0+ +/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { +let fb_info = &self.0 .fbRegionInfoParams; +let mut max_end:Option <u64 > =None ; +for iin 0 ..fb_info.numFBRegions.into_safe_cast () { +if let Some (reg) = fb_info.fbRegion.get (i) { +if reg.limit < reg.base { +continue ; + }
This is basically a repeat of the code of the previous patch. Let's implement an iterator over the FB memory regions (that filters out invalid regions) that we can leverage in both places so we don't need to repeat ourselves.
Re: [PATCH v9 05/23] gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
Quoted 4 lines · Alexandre Courbot
This is basically a repeat of the code of the previous patch. Let's implement an iterator over the FB memory regions (that filters out invalid regions) that we can leverage in both places so we don't need to repeat ourselves.
Good catch. Added fb_regions() to create an iterator which both this and previous patch use. Nice simplification:
/// Compute the end of physical VRAM from all FB regions.
pub(crate) fn total_fb_end(&self) -> Option<u64> {
self.fb_regions().map(|reg| reg.limit.saturating_add(1)).max()
}--
Joel Fernandes
PATCH v9 6/23 gpu: nova-core: mm: Add support to use PRAMIN windows to write to VRAM
In reply to Joel FernandesPRAMIN apertures are a crucial mechanism to direct read/write to VRAM. Add support for the same.
--- drivers/gpu/nova-core/mm.rs | 5 + drivers/gpu/nova-core/mm/pramin.rs | 293 +++++++++++++++++++++++++++++ drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/regs.rs | 6 + 4 files changed, 305 insertions(+) create mode 100644 drivers/gpu/nova-core/mm.rs create mode 100644 drivers/gpu/nova-core/mm/pramin.rs
drivers/gpu/nova-core/mm.rs new file +5 -0
@@ -0,0 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Memory management subsystems for nova-core. + +pub (crate )mod pramin;
drivers/gpu/nova-core/mm/pramin.rs new file +293 -0
293 lines not on this page · raw message
drivers/gpu/nova-core/nova_core.rs +1 -0
@@ -13,6 +13,7 @@mod gfw;mod gpu;mod gsp; +mod mm;mod num;mod regs;mod sbuffer;
drivers/gpu/nova-core/regs.rs +6 -0
@@ -32,6 +32,7 @@ Architecture, Chipset, // }, + mm::pramin::Bar0WindowTarget, num::FromSafeCast, };
@@ -102,6 +103,11 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result { 31:16 frts_err_code as u16; }); +register! (NV_PBUS_BAR0_WINDOW @0x00001700 ,"BAR0 window control for PRAMIN access" { +25 :24 targetas u8 ?=>Bar0WindowTarget ; +23 :0 window_baseas u32 ,"Window base address (bits 39:16 of FB addr)" ; +}); +// PFB // The following two registers together hold the physical system memory address that is used by the
Signature · 1 line
2.34.1
PATCH v9 7/23 docs: gpu: nova-core: Document the PRAMIN aperture mechanism
In reply to Joel FernandesAdd documentation for the PRAMIN aperture mechanism used by nova-core for direct VRAM access.
Nova only uses TARGET=VID_MEM for VRAM access. The SYS_MEM target values are documented for completeness but not used by the driver.
--- Documentation/gpu/nova/core/pramin.rst | 125 +++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 126 insertions(+) create mode 100644 Documentation/gpu/nova/core/pramin.rst
Documentation/gpu/nova/core/pramin.rst new file +125 -0
@@ -0,0 +1,125 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================= +PRAMIN aperture mechanism +========================= + +.. note:: + The following description is approximate and current as of the Ampere family. + It may change for future generations and is intended to assist in understanding + the driver code. + +Introduction +============ + +PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video RAM (VRAM) before +the GPU's Memory Management Unit (MMU) and page tables are initialized. This 1MB sliding window, +located at a fixed offset within BAR0, is essential for setting up page tables and other critical +GPU data structures without relying on the GPU's MMU. + +Architecture Overview +===================== + +The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe Bus Controller Unit) +and provides a CPU-accessible window into VRAM through the PCIe interface:: + + +-----------------+ PCIe +------------------------------+ + | CPU |<----------->| GPU | + +-----------------+ | | + | +----------------------+ | + | | PBUS | | + | | (Bus Controller) | | + | | | | + | | +--------------+<------------ (window starts at + | | | PRAMIN | | | BAR0 + 0x700000) + | | | Window | | | + | | | (1MB) | | | + | | +--------------+ | | + | | | | | + | +---------|------------+ | + | | | + | v | + | +----------------------+<------------ (Program PRAMIN to any + | | VRAM | | 64KB-aligned VRAM boundary) + | | (Several GBs) | | + | | | | + | | FB[0x000000000000] | | + | | ... | | + | | FB[0x7FFFFFFFFFF] | | + | +----------------------+ | + +------------------------------+ + +PBUS (PCIe Bus Controller) is responsible for, among other things, handling MMIO +accesses to the BAR registers. + +PRAMIN Window Operation +======================= + +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over +the entire VRAM address space using the ``NV_PBUS_BAR0_WINDOW`` register. + +Window Control Mechanism +------------------------- + +The window position is controlled via the PBUS ``BAR0_WINDOW`` register:: + + NV_PBUS_BAR0_WINDOW Register (0x1700): + +-------+--------+--------------------------------------+ + | 31:26 | 25:24 | 23:0 | + | RSVD | TARGET | BASE_ADDR | + | | | (bits 39:16 of VRAM address) | + +-------+--------+--------------------------------------+ + + BASE_ADDR field (bits 23:0): + - Contains bits [39:16] of the target VRAM address + - Provides 40-bit (1TB) address space coverage + - Must be programmed with 64KB-aligned addresses + + TARGET field (bits 25:24): + - 0x0: VRAM (Video Memory) + - 0x1: SYS_MEM_COH (Coherent System Memory) + - 0x2: SYS_MEM_NONCOH (Non-coherent System Memory) + - 0x3: Reserved + + .. note:: + Nova only uses TARGET=VRAM (0x0) for video memory access. The SYS_MEM + target values are documented here for hardware completeness but are + not used by the driver. + +64KB Alignment Requirement +--------------------------- + +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced +by the ``BASE_ADDR`` field representing bits [39:16] of the target address:: + + VRAM Address Calculation: + actual_vram_addr = (BASE_ADDR << 16) + pramin_offset + Where: + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] + - pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF] + + Example Window Positioning: + +---------------------------------------------------------+ + | VRAM Space | + | | + | 0x000000000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x0000FFFFF +-----------------+ | + | | + | | ^ | + | | | Window can slide | + | v | to any 64KB-aligned boundary | + | | + | 0x123400000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x1234FFFFF +-----------------+ | + | | + | ... | + | | + | 0x7FFFF0000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x7FFFFFFFF +-----------------+ | + +---------------------------------------------------------+
Documentation/gpu/nova/index.rst +1 -0
@@ -32,3 +32,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/devinit core/fwsec core/falcon + core/pramin
Signature · 1 line
2.34.1
PATCH v9 8/23 gpu: nova-core: mm: Add common memory management types
In reply to Joel FernandesAdd foundational types for GPU memory management. These types are used throughout the nova memory management subsystem for page table operations, address translation, and memory allocation.
--- drivers/gpu/nova-core/mm.rs | 160 ++++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+)
drivers/gpu/nova-core/mm.rs +160 -0
@@ -2,4 +2,164 @@//! Memory management subsystems for nova-core. +#![expect(dead_code)] +pub (crate )mod pramin; + +use kernel::sizes::SZ_4K ; + +use crate ::num::u64_as_usize; + +/// Page size in bytes (4 KiB). +pub (crate )const PAGE_SIZE :usize =SZ_4K ; + +bitfield! { +pub (crate )struct VramAddress (u64 ),"Physical VRAM address in GPU video memory" { +11 :0 offsetas u64 ,"Offset within 4KB page" ; +63 :12 frame_numberas u64 =>Pfn ,"Physical frame number" ; + } +} + +impl VramAddress { +/// Create a new VRAM address from a raw value. +pub (crate )const fn new (addr:u64 ) ->Self { +Self (addr) + } + +/// Get the raw address value as `usize` (useful for MMIO offsets). +pub (crate )const fn raw (&self) ->usize { +u64_as_usize (self.0 ) + } + +/// Get the raw address value as `u64`. +pub (crate )const fn raw_u64 (&self) ->u64 { + self.0 + } +} + +impl PartialEq for VramAddress { +fn eq (&self, other: &Self ) ->bool { + self.0 == other.0 + } +} + +impl Eq for VramAddress {} + +impl PartialOrd for VramAddress { +fn partial_cmp (&self, other: &Self ) ->Option <core::cmp::Ordering > { +Some (self.cmp (other)) + } +} + +impl Ord for VramAddress { +fn cmp (&self, other: &Self ) -> core::cmp::Ordering { + self.0 .cmp (&other.0 ) + } +} + +impl From <Pfn >for VramAddress { +fn from (pfn:Pfn ) ->Self { +Self ::default ().set_frame_number (pfn) + } +} + +// GPU virtual address. +bitfield! { +pub (crate )struct VirtualAddress (u64 ),"Virtual address in GPU address space" { +11 :0 offsetas u64 ,"Offset within 4KB page" ; +63 :12 frame_numberas u64 =>Vfn ,"Virtual frame number" ; + } +} + +impl VirtualAddress { +/// Create a new virtual address from a raw value. +#[expect(dead_code)] +pub (crate )const fn new (addr:u64 ) ->Self { +Self (addr) + } + +/// Get the raw address value as `u64`. +pub (crate )const fn raw_u64 (&self) ->u64 { + self.0 + } +} + +impl From <Vfn >for VirtualAddress { +fn from (vfn:Vfn ) ->Self { +Self ::default ().set_frame_number (vfn) + } +} + +/// Physical Frame Number. +/// +/// Represents a physical page in VRAM. +#[repr(transparent)] +#[derive( Debug , Clone , Copy , PartialEq , Eq , Default )] +pub (crate )struct Pfn (u64 ); + +impl Pfn { +/// Create a new PFN from a frame number. +pub (crate )const fn new (frame_number:u64 ) ->Self { +Self (frame_number) + } + +/// Get the raw frame number. +pub (crate )const fn raw (self) ->u64 { + self.0 + } +} + +impl From <VramAddress >for Pfn { +fn from (addr:VramAddress ) ->Self { + addr.frame_number () + } +} + +impl From <u64 >for Pfn { +fn from (val:u64 ) ->Self { +Self (val) + } +} + +impl From <Pfn >for u64 { +fn from (pfn:Pfn ) ->Self { + pfn.0 + } +} + +/// Virtual Frame Number. +/// +/// Represents a virtual page in GPU address space. +#[repr(transparent)] +#[derive( Debug , Clone , Copy , PartialEq , Eq , Default )] +pub (crate )struct Vfn (u64 ); + +impl Vfn { +/// Create a new VFN from a frame number. +pub (crate )const fn new (frame_number:u64 ) ->Self { +Self (frame_number) + } + +/// Get the raw frame number. +pub (crate )const fn raw (self) ->u64 { + self.0 + } +} + +impl From <VirtualAddress >for Vfn { +fn from (addr:VirtualAddress ) ->Self { + addr.frame_number () + } +} + +impl From <u64 >for Vfn { +fn from (val:u64 ) ->Self { +Self (val) + } +} + +impl From <Vfn >for u64 { +fn from (vfn:Vfn ) ->Self { + vfn.0 + } +}
Signature · 1 line
2.34.1
PATCH v9 9/23 gpu: nova-core: mm: Add TLB flush support
In reply to Joel FernandesAdd TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 90 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 33 ++++++++++++ 3 files changed, 124 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;
drivers/gpu/nova-core/mm/tlb.rs new file +90 -0
90 lines not on this page · raw message
drivers/gpu/nova-core/regs.rs +33 -0
@@ -455,3 +455,36 @@ pub(crate) mod ga100 { 0:0 display_disabled as bool; }); } + +// MMU TLB + +register! (NV_TLB_FLUSH_PDB_LO @0x00b830a0 ,"TLB flush register: PDB address bits [39:8]" { +31 :0 pdb_loas u32 ,"PDB address bits [39:8]" ; +}); + +impl NV_TLB_FLUSH_PDB_LO { +/// Create a register value from a PDB address. +/// +/// Extracts bits [39:8] of the address and shifts it right by 8 bits. +pub (crate )fn from_pdb_addr (addr:u64 ) ->Self { +Self ::default ().set_pdb_lo (((addr >>8 ) &0xFFFF_FFFF )as u32 ) + } +} + +register! (NV_TLB_FLUSH_PDB_HI @0x00b830a4 ,"TLB flush register: PDB address bits [47:40]" { +7 :0 pdb_hias u8 ,"PDB address bits [47:40]" ; +}); + +impl NV_TLB_FLUSH_PDB_HI { +/// Create a register value from a PDB address. +/// +/// Extracts bits [47:40] of the address and shifts it right by 40 bits. +pub (crate )fn from_pdb_addr (addr:u64 ) ->Self { +Self ::default ().set_pdb_hi (((addr >>40 ) &0xFF )as u8 ) + } +} + +register! (NV_TLB_FLUSH_CTRL @0x00b830b0 ,"TLB flush control register" { +0 :0 page_allas bool ,"Invalidate all pages" ; +31 :31 enableas bool ,"Enable/trigger flush (clears when flush completes)" ; +});
Signature · 1 line
2.34.1
PATCH v9 10/23 gpu: nova-core: mm: Add GpuMm centralized memory manager
In reply to Joel FernandesIntroduce GpuMm as the centralized GPU memory manager that owns:
- Buddy allocator for VRAM allocation.
- PRAMIN window for direct VRAM access.
- TLB manager for translation buffer operations.
This provides clean ownership model where GpuMm provides accessor methods for its components that can be used for memory management operations.
--- drivers/gpu/nova-core/gpu.rs | 32 +++++++++++- drivers/gpu/nova-core/gsp/commands.rs | 2 - drivers/gpu/nova-core/mm.rs | 70 ++++++++++++++++++++++++++- 3 files changed, 99 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/gpu.rs +31 -1
@@ -4,8 +4,10 @@ device, devres::Devres, fmt, + gpu::buddy::GpuBuddyParams, pci, prelude::*, + sizes::SZ_4K, sync::Arc, // };
@@ -22,6 +24,7 @@ commands::GetGspStaticInfoReply, Gsp, // }, + mm::GpuMm, regs, };
@@ -252,6 +255,9 @@ pub(crate) struct Gpu { gsp_falcon: Falcon<GspFalcon>, /// SEC2 falcon instance, used for GSP boot up and cleanup. sec2_falcon: Falcon<Sec2Falcon>, +/// GPU memory manager owning memory management resources. +#[pin] + mm: GpuMm,/// GSP runtime data. Temporarily an empty placeholder. #[pin] gsp: Gsp,
@@ -288,7 +294,31 @@ pub(crate) fn new<'a>( gsp <- Gsp::new(pdev), - gsp_static_info: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, + gsp_static_info: { + let info = gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)?; + +dev_info! ( + pdev.as_ref (), +"Using FB region: {:#x}..{:#x}\n" , + info.usable_fb_region.start, + info.usable_fb_region.end + ); + + info + }, + +// Create GPU memory manager owning memory management resources. + mm <- { +let usable_vram = &gsp_static_info.usable_fb_region; +// PRAMIN covers all physical VRAM (including GSP-reserved areas +// above the usable region, e.g. the BAR1 page directory). +let pramin_vram_region =0 ..gsp_static_info.total_fb_end; +GpuMm ::new (devres_bar.clone (),GpuBuddyParams { + base_offset: usable_vram.start, + physical_memory_size: usable_vram.end - usable_vram.start, + chunk_size:SZ_4K , + }, pramin_vram_region)? + }, bar: devres_bar, })
drivers/gpu/nova-core/gsp/commands.rs +0 -2
@@ -191,10 +191,8 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ],/// Usable FB (VRAM) region for driver memory allocation. -#[expect(dead_code)] pub (crate ) usable_fb_region:Range <u64 >,/// End of VRAM. -#[expect(dead_code)] pub (crate ) total_fb_end:u64 , }
drivers/gpu/nova-core/mm.rs +68 -2
77 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 11/23 gpu: nova-core: mm: Add common types for all page table formats
In reply to Joel FernandesAdd common page table types shared between MMU v2 and v3. These types are hardware-agnostic and used by both MMU versions.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/pagetable.rs | 155 ++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -4,6 +4,7 @@#![expect(dead_code)] +pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb;
drivers/gpu/nova-core/mm/pagetable.rs new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 12/23 gpu: nova-core: mm: Add MMU v2 page table types
In reply to Joel FernandesAdd page table entry and directory structures for MMU version 2 used by Turing/Ampere/Ada GPUs.
--- drivers/gpu/nova-core/mm/pagetable.rs | 2 + drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 +++++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rs
drivers/gpu/nova-core/mm/pagetable.rs +2 -0
@@ -8,6 +8,8 @@#![expect(dead_code)] +pub (crate )mod ver2; +use crate ::gpu::Architecture ;/// Extracts the page table index at a given level from a virtual address.
drivers/gpu/nova-core/mm/pagetable/ver2.rs new file +232 -0
232 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 13/23 gpu: nova-core: mm: Add MMU v3 page table types
In reply to Joel FernandesAdd page table entry and directory structures for MMU version 3 used by Hopper and later GPUs.
--- drivers/gpu/nova-core/mm/pagetable.rs | 1 + drivers/gpu/nova-core/mm/pagetable/ver3.rs | 337 +++++++++++++++++++++ 2 files changed, 338 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver3.rs
drivers/gpu/nova-core/mm/pagetable.rs +1 -0
@@ -9,6 +9,7 @@#![expect(dead_code)] pub (crate )mod ver2; +pub (crate )mod ver3;use crate ::gpu::Architecture ;
drivers/gpu/nova-core/mm/pagetable/ver3.rs new file +337 -0
337 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 14/23 gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesAdd unified Pte, Pde, and DualPde wrapper enums that abstract over MMU v2 and v3 page table entry formats. These enums allow the page table walker and VMM to work with both MMU versions.
Each unified type:
- Takes MmuVersion parameter in constructors
- Wraps both ver2 and ver3 variants
- Delegates method calls to the appropriate variant
This enables version-agnostic page table operations while keeping version-specific implementation details encapsulated in the ver2 and ver3 modules.
--- drivers/gpu/nova-core/mm/pagetable.rs | 322 ++++++++++++++++++++++++++ 1 file changed, 322 insertions(+)
drivers/gpu/nova-core/mm/pagetable.rs +322 -0
@@ -12,6 +12,13 @@pub (crate )mod ver3;use crate ::gpu::Architecture ; +use crate ::mm::{ + pramin, +Pfn , +VirtualAddress , +VramAddress ,// +}; +use kernel::prelude::*;/// Extracts the page table index at a given level from a virtual address. pub (crate )trait VaLevelIndex {
@@ -84,6 +91,96 @@ pub(crate) const fn as_index(&self) -> u64 { } } +impl MmuVersion { +/// Get the `PDE` levels (excluding PTE level) for page table walking. +pub (crate )fn pde_levels (&self) -> &'static [PageTableLevel ] { +match self { +Self ::V2 => ver2::PDE_LEVELS , +Self ::V3 => ver3::PDE_LEVELS , + } + } + +/// Get the PTE level for this MMU version. +pub (crate )fn pte_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::PTE_LEVEL , +Self ::V3 => ver3::PTE_LEVEL , + } + } + +/// Get the dual PDE level (128-bit entries) for this MMU version. +pub (crate )fn dual_pde_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::DUAL_PDE_LEVEL , +Self ::V3 => ver3::DUAL_PDE_LEVEL , + } + } + +/// Get the number of PDE levels for this MMU version. +pub (crate )fn pde_level_count (&self) ->usize { + self.pde_levels ().len () + } + +/// Get the entry size in bytes for a given level. +pub (crate )fn entry_size (&self, level:PageTableLevel ) ->usize { +if level == self.dual_pde_level () { +16 // 128-bit dual PDE + }else { +8 // 64-bit PDE/PTE + } + } + +/// Get the number of entries per page table page for a given level. +pub (crate )fn entries_per_page (&self, level:PageTableLevel ) ->usize { +match self { +Self ::V2 =>match level { +// TODO: Calculate these values from the bitfield dynamically +// instead of hardcoding them. +PageTableLevel ::Pdb =>4 ,// PD3 root: bits [48:47] = 2 bits +PageTableLevel ::L3 =>256 ,// PD0 dual: bits [28:21] = 8 bits + _ =>512 ,// PD2, PD1, PT: 9 bits each + }, +Self ::V3 =>match level { +PageTableLevel ::Pdb =>2 ,// PDE4 root: bit [56] = 1 bit, 2 entries +PageTableLevel ::L4 =>256 ,// PDE0 dual: bits [28:21] = 8 bits + _ =>512 ,// PDE3, PDE2, PDE1, PT: 9 bits each + }, + } + } + +/// Extract the page table index at `level` from `va` for this MMU version. +pub (crate )fn level_index (&self, va:VirtualAddress , level:u64 ) ->u64 { +match self { +Self ::V2 => ver2::VirtualAddressV2 ::new (va).level_index (level), +Self ::V3 => ver3::VirtualAddressV3 ::new (va).level_index (level), + } + } + +/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); + +// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +/// Memory aperture for Page Table Entries (`PTE`s). /// /// Determines which memory region the `PTE` points to.
@@ -156,3 +253,228 @@ fn from(val: AperturePde) -> Self { valas u8 } } + +/// Unified Page Table Entry wrapper for both MMU v2 and v3 `PTE` +/// types, allowing the walker to work with either format. +#[derive( Debug , Clone , Copy )] +pub (crate )enum Pte { +/// MMU v2 `PTE` (Turing/Ampere/Ada). +V2 (ver2::Pte ), +/// MMU v3 `PTE` (Hopper+). +V3 (ver3::Pte ), +} + +impl Pte { +/// Create a `PTE` from a raw `u64` value for the given MMU version. +pub (crate )fn new (version:MmuVersion , val:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::new (val)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::new (val)), + } + } + +/// Create an invalid `PTE` for the given MMU version. +pub (crate )fn invalid (version:MmuVersion ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::invalid ()), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::invalid ()), + } + } + +/// Create a valid `PTE` for video memory. +pub (crate )fn new_vram (version:MmuVersion , pfn:Pfn , writable:bool ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::new_vram (pfn, writable)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::new_vram (pfn, writable)), + } + } + +/// Check if this `PTE` is valid. +pub (crate )fn is_valid (&self) ->bool { +match self { +Self ::V2 (p) => p.valid (), +Self ::V3 (p) => p.valid (), + } + } + +/// Get the physical frame number. +pub (crate )fn frame_number (&self) ->Pfn { +match self { +Self ::V2 (p) => p.frame_number (), +Self ::V3 (p) => p.frame_number (), + } + } + +/// Get the raw `u64` value. +pub (crate )fn raw_u64 (&self) ->u64 { +match self { +Self ::V2 (p) => p.raw_u64 (), +Self ::V3 (p) => p.raw_u64 (), + } + } + +/// Read a `PTE` from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let val = window.try_read64 (addr.raw ())?; +Ok (Self ::new (mmu_version, val)) + } + +/// Write this `PTE` to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.raw_u64 ()) + } +} + +/// Unified Page Directory Entry wrapper for both MMU v2 and v3 `PDE`. +#[derive( Debug , Clone , Copy )] +pub (crate )enum Pde { +/// MMU v2 `PDE` (Turing/Ampere/Ada). +V2 (ver2::Pde ), +/// MMU v3 `PDE` (Hopper+). +V3 (ver3::Pde ), +} + +impl Pde { +/// Create a `PDE` from a raw `u64` value for the given MMU version. +pub (crate )fn new (version:MmuVersion , val:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::new (val)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::new (val)), + } + } + +/// Create a valid `PDE` pointing to a page table in video memory. +pub (crate )fn new_vram (version:MmuVersion , table_pfn:Pfn ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::new_vram (table_pfn)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::new_vram (table_pfn)), + } + } + +/// Create an invalid `PDE` for the given MMU version. +pub (crate )fn invalid (version:MmuVersion ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::invalid ()), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::invalid ()), + } + } + +/// Check if this `PDE` is valid. +pub (crate )fn is_valid (&self) ->bool { +match self { +Self ::V2 (p) => p.is_valid (), +Self ::V3 (p) => p.is_valid (), + } + } + +/// Get the VRAM address of the page table. +pub (crate )fn table_vram_address (&self) ->VramAddress { +match self { +Self ::V2 (p) => p.table_vram_address (), +Self ::V3 (p) => p.table_vram_address (), + } + } + +/// Get the raw `u64` value. +pub (crate )fn raw_u64 (&self) ->u64 { +match self { +Self ::V2 (p) => p.raw_u64 (), +Self ::V3 (p) => p.raw_u64 (), + } + } + +/// Read a `PDE` from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let val = window.try_read64 (addr.raw ())?; +Ok (Self ::new (mmu_version, val)) + } + +/// Write this `PDE` to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.raw_u64 ()) + } +} + +/// Unified Dual Page Directory Entry wrapper for both MMU v2 and v3 [`DualPde`]. +#[derive( Debug , Clone , Copy )] +pub (crate )enum DualPde { +/// MMU v2 [`DualPde`] (Turing/Ampere/Ada). +V2 (ver2::DualPde ), +/// MMU v3 [`DualPde`] (Hopper+). +V3 (ver3::DualPde ), +} + +impl DualPde { +/// Create a [`DualPde`] from raw 128-bit value (two `u64`s) for the given MMU version. +pub (crate )fn new (version:MmuVersion , big:u64 , small:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::DualPde ::new (big, small)), +MmuVersion ::V3 =>Self ::V3 (ver3::DualPde ::new (big, small)), + } + } + +/// Create a [`DualPde`] with only the small page table pointer set. +pub (crate )fn new_small (version:MmuVersion , table_pfn:Pfn ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::DualPde ::new_small (table_pfn)), +MmuVersion ::V3 =>Self ::V3 (ver3::DualPde ::new_small (table_pfn)), + } + } + +/// Check if the small page table pointer is valid. +pub (crate )fn has_small (&self) ->bool { +match self { +Self ::V2 (d) => d.has_small (), +Self ::V3 (d) => d.has_small (), + } + } + +/// Get the small page table VRAM address. +pub (crate )fn small_vram_address (&self) ->VramAddress { +match self { +Self ::V2 (d) => d.small.table_vram_address (), +Self ::V3 (d) => d.small.table_vram_address (), + } + } + +/// Get the raw `u64` value of the big PDE. +pub (crate )fn big_raw_u64 (&self) ->u64 { +match self { +Self ::V2 (d) => d.big.raw_u64 (), +Self ::V3 (d) => d.big.raw_u64 (), + } + } + +/// Get the raw `u64` value of the small PDE. +pub (crate )fn small_raw_u64 (&self) ->u64 { +match self { +Self ::V2 (d) => d.small.raw_u64 (), +Self ::V3 (d) => d.small.raw_u64 (), + } + } + +/// Read a dual PDE (128-bit) from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let lo = window.try_read64 (addr.raw ())?; +let hi = window.try_read64 (addr.raw () +8 )?; +Ok (Self ::new (mmu_version, lo, hi)) + } + +/// Write this dual PDE (128-bit) to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.big_raw_u64 ())?; + window.try_write64 (addr.raw () +8 , self.small_raw_u64 ()) + } +}
Signature · 1 line
2.34.1
PATCH v9 15/23 gpu: nova-core: mm: Add page table walker for MMU v2/v3
In reply to Joel FernandesAdd the page table walker implementation that traverses the page table hierarchy for both MMU v2 (5-level) and MMU v3 (6-level) to resolve virtual addresses to physical addresses or find PTE locations.
Currently only v2 has been tested (nova-core currently boots pre-hopper) with some initial prepatory work done for v3.
--- drivers/gpu/nova-core/mm/pagetable.rs | 1 + drivers/gpu/nova-core/mm/pagetable/walk.rs | 218 +++++++++++++++++++++ 2 files changed, 219 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/walk.rs
drivers/gpu/nova-core/mm/pagetable.rs +1 -0
@@ -10,6 +10,7 @@pub (crate )mod ver2;pub (crate )mod ver3; +pub (crate )mod walk;use crate ::gpu::Architecture ;use crate ::mm::{
drivers/gpu/nova-core/mm/pagetable/walk.rs new file +218 -0
218 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 16/23 gpu: nova-core: mm: Add Virtual Memory Manager
In reply to Joel FernandesAdd the Virtual Memory Manager (VMM) infrastructure for GPU address space management. Each Vmm instance manages a single address space identified by its Page Directory Base (PDB) address, used for Channel, BAR1 and BAR2 mappings.
Mapping APIs and virtual address range tracking are added in later commits.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/vmm.rs | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/vmm.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -7,6 +7,7 @@pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb; +pub (crate )mod vmm;use kernel::{ devres::Devres ,
drivers/gpu/nova-core/mm/vmm.rs new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 17/23 gpu: nova-core: mm: Add virtual address range tracking to VMM
In reply to Joel FernandesAdd virtual address range tracking to the VMM using a buddy allocator. This enables contiguous virtual address range allocation for mappings.
--- drivers/gpu/nova-core/mm/vmm.rs | 99 +++++++++++++++++++++++++++++---- 1 file changed, 88 insertions(+), 11 deletions(-)
drivers/gpu/nova-core/mm/vmm.rs +88 -11
@@ -7,19 +7,35 @@//! and handles TLB flushing after modifications. use kernel::{ - gpu::buddy::AllocatedBlocks , - prelude::*,// + gpu::buddy::{ +AllocatedBlocks , +GpuBuddy , +GpuBuddyAllocFlag , +GpuBuddyAllocMode , +GpuBuddyParams ,// + }, + prelude::*, + ptr::Alignment , + sizes::SZ_4K ,// }; -use crate ::mm::{ - pagetable::{ - walk::{PtWalk ,WalkResult }, -MmuVersion ,// +use core::ops::Range ; + +use crate ::{ + mm::{ + pagetable::{ + walk::{PtWalk ,WalkResult }, +MmuVersion ,// + }, +GpuMm , +Pfn , +Vfn , +VramAddress , +PAGE_SIZE ,// + }, + num::{ +IntoSafeCast ,// }, -GpuMm , -Pfn , -Vfn , -VramAddress ,// };/// Virtual Memory Manager for a GPU address space.
@@ -32,23 +48,84 @@ pub(crate) struct Vmm { pub(crate) mmu_version: MmuVersion,/// Page table allocations required for mappings. page_table_allocs: KVec<Pin<KBox<AllocatedBlocks>>>, +/// Buddy allocator for virtual address range tracking. + virt_buddy: GpuBuddy, }impl Vmm {/// Create a new [`Vmm`] for the given Page Directory Base address. -pub (crate )fn new (pdb_addr:VramAddress , mmu_version:MmuVersion ) ->Result <Self > { +/// +/// The [`Vmm`] will manage a virtual address space of `va_size` bytes. +pub (crate )fn new ( + pdb_addr:VramAddress , + mmu_version:MmuVersion , + va_size:u64 , + ) ->Result <Self > {// Only MMU v2 is supported for now. if mmu_version !=MmuVersion ::V2 {return Err (ENOTSUPP ); } +let virt_buddy =GpuBuddy ::new (GpuBuddyParams { + base_offset:0 , + physical_memory_size: va_size, + chunk_size:SZ_4K , + })?; +Ok (Self { pdb_addr, mmu_version, page_table_allocs:KVec ::new (), + virt_buddy, }) } +/// Allocate a contiguous virtual frame number range. +/// +/// # Arguments +/// +/// - `num_pages`: Number of pages to allocate. +/// - `va_range`: `None` = allocate anywhere, `Some(range)` = constrain allocation to the given +/// range. +pub (crate )fn alloc_vfn_range ( + &self, + num_pages:usize , + va_range:Option <Range <u64 >>, + ) ->Result <(Vfn ,Pin <KBox <AllocatedBlocks >>)> { +let size = num_pages.checked_mul (PAGE_SIZE ).ok_or (EOVERFLOW )?; + +let mode =match va_range { +Some (r) => { +let range_size = r.end.checked_sub (r.start).ok_or (EOVERFLOW )?; +if range_size != size.into_safe_cast () { +return Err (EINVAL ); + } +GpuBuddyAllocMode ::Range { + start: r.start, + end: r.end, + } + } +None =>GpuBuddyAllocMode ::Simple , + }; + +let alloc =KBox ::pin_init ( + self.virt_buddy.alloc_blocks ( + mode, + size, +Alignment ::new ::<SZ_4K >(), +GpuBuddyAllocFlag ::Contiguous , + ), +GFP_KERNEL , + )?; + +// Get the starting offset of the first block (only block as range is contiguous). +let offset = alloc.iter ().next ().ok_or (ENOMEM )?.offset (); +let page_size:u64 =PAGE_SIZE .into_safe_cast (); +let vfn =Vfn ::new (offset / page_size); + +Ok ((vfn, alloc)) + } +/// Read the [`Pfn`] for a mapped [`Vfn`] if one is mapped. pub (crate )fn read_mapping (&self, mm: &GpuMm , vfn:Vfn ) ->Result <Option <Pfn >> {let walker =PtWalk ::new (self.pdb_addr, self.mmu_version);
Signature · 1 line
2.34.1
PATCH v9 18/23 gpu: nova-core: mm: Add multi-page mapping API to VMM
In reply to Joel FernandesAdd the page table mapping and unmapping API to the Virtual Memory Manager, implementing a two-phase prepare/execute model suitable for use both inside and outside the DMA fence signalling critical path.
--- drivers/gpu/nova-core/mm/vmm.rs | 366 +++++++++++++++++++++++++++++++- 1 file changed, 363 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/mm/vmm.rs +363 -3
@@ -11,21 +11,34 @@ AllocatedBlocks, GpuBuddy, GpuBuddyAllocFlag, + GpuBuddyAllocFlags, GpuBuddyAllocMode, GpuBuddyParams, // }, prelude::*, ptr::Alignment, + rbtree::{RBTree, RBTreeNode}, sizes::SZ_4K, // }; -use core::ops::Range ; +use core::{ + cell::Cell , + ops::Range ,// +};use crate ::{ mm::{ pagetable::{ - walk::{PtWalk ,WalkResult }, -MmuVersion ,// + walk::{ +PtWalk , +WalkPdeResult , +WalkResult ,// + }, +DualPde , +MmuVersion , +PageTableLevel , +Pde , +Pte ,// },GpuMm ,Pfn ,
@@ -50,6 +63,74 @@ pub(crate) struct Vmm { page_table_allocs: KVec<Pin<KBox<AllocatedBlocks>>>,/// Buddy allocator for virtual address range tracking. virt_buddy: GpuBuddy, +/// Prepared PT pages pending PDE installation, keyed by `install_addr`. +/// +/// Populated by `Vmm` mapping prepare phase and drained in the execute phase. +/// Shared by all pending maps in the `Vmm`, thus preventing races where 2 +/// maps might be trying to install the same page table/directory entry pointer. + pt_pages: RBTree<VramAddress, PreparedPtPage>, +} + +/// A pre-allocated and zeroed page table page. +/// +/// Created during the mapping prepare phase and consumed during the mapping execute phase. +/// Stored in an [`RBTree`] keyed by the PDE slot address (`install_addr`). +struct PreparedPtPage { +/// The allocated and zeroed page table page. + alloc:Pin <KBox <AllocatedBlocks >>, +/// Page table level -- needed to determine if this PT page is for a dual PDE. + level:PageTableLevel , +} + +/// Multi-page prepared mapping -- VA range allocated, ready for execute. +/// +/// Produced by [`Vmm::prepare_map()`], consumed by [`Vmm::execute_map()`]. +/// The struct owns the VA space allocation between prepare and execute phases. +pub (crate )struct PreparedMapping { + vfn_start:Vfn , + num_pages:usize , + vfn_alloc:Pin <KBox <AllocatedBlocks >>, +} + +/// Result of a mapping operation -- tracks the active mapped range. +/// +/// Returned by [`Vmm::execute_map()`] and [`Vmm::map_pages()`]. +/// Owns the VA allocation; the VA range is freed when this is dropped. +/// Callers must call [`Vmm::unmap_pages()`] before dropping to invalidate +/// PTEs (dropping only frees the VA range, not the PTE entries). +pub (crate )struct MappedRange { +pub (crate ) vfn_start:Vfn , +pub (crate ) num_pages:usize , +/// VA allocation -- freed when [`MappedRange`] is dropped. + _vfn_alloc:Pin <KBox <AllocatedBlocks >>, +/// Logs a warning if dropped without unmapping. + _drop_guard:MustUnmapGuard , +} + +/// Guard that logs a warning once if a [`MappedRange`] is dropped without +/// calling [`Vmm::unmap_pages()`]. +struct MustUnmapGuard { + armed:Cell <bool >, +} + +impl MustUnmapGuard { +const fn new () ->Self { +Self { + armed:Cell ::new (true ), + } + } + +fn disarm (&self) { + self.armed.set (false ); + } +} + +impl Drop for MustUnmapGuard { +fn drop (&mut self) { +if self.armed.get () { + kernel::pr_warn!("MappedRange dropped without calling unmap_pages()\n" ); + } + } }impl Vmm {
@@ -77,6 +158,7 @@ pub(crate) fn new( mmu_version, page_table_allocs: KVec::new(), virt_buddy, + pt_pages: RBTree::new(), }) }
@@ -135,4 +217,282 @@ pub(crate) fn read_mapping(&self, mm: &GpuMm, vfn: Vfn) -> Result<Option<Pfn>> { WalkResult::Unmapped { .. } | WalkResult::PageTableMissing => Ok(None), } } + +/// Allocate and zero a physical page table page for a specific PDE slot. +/// Called during the map prepare phase. +fn alloc_and_zero_page_table ( + &mut self, + mm: &GpuMm , + level:PageTableLevel , + ) ->Result <PreparedPtPage > { +let blocks =KBox ::pin_init ( + mm.buddy ().alloc_blocks ( +GpuBuddyAllocMode ::Simple , +SZ_4K , +Alignment ::new ::<SZ_4K >(), +GpuBuddyAllocFlags ::default (), + ), +GFP_KERNEL , + )?; + +// Get page's VRAM address from the allocation. +let page_vram =VramAddress ::new (blocks.iter ().next ().ok_or (ENOMEM )?.offset ()); + +// Zero via PRAMIN. +let mut window = mm.pramin ().window ()?; +let base = page_vram.raw (); +for offin (0 ..PAGE_SIZE ).step_by (8 ) { + window.try_write64 (base + off,0 )?; + } + +Ok (PreparedPtPage { + alloc: blocks, + level, + }) + } + +/// Ensure all intermediate page table pages are prepared for a [`Vfn`]. Just +/// finds out which PDE pages are missing, allocates pages for them, and defers +/// installation to the execute phase. +/// +/// PRAMIN is released before each allocation and re-acquired after. Memory +/// allocations are done outside of holding this lock to prevent deadlocks with +/// the fence signalling critical path. +fn ensure_pte_path (&mut self, mm: &GpuMm , vfn:Vfn ) ->Result { +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let max_iter =2 * self.mmu_version.pde_level_count (); + +// Keep looping until all PDE levels are resolved. +for _in 0 ..max_iter { +let mut window = mm.pramin ().window ()?; + +// Walk PDE levels. The closure checks self.pt_pages for prepared-but-uninstalled +// pages, letting the walker continue through them as if they were installed in HW. +// The walker keeps calling the closure to get these "prepared but not installed" pages. +let result = walker.walk_pde_levels (&mut window, vfn, |install_addr| { + self.pt_pages + .get (&install_addr) + .and_then (|p|Some (VramAddress ::new (p.alloc.iter ().next ()?.offset ()))) + })?; + +match result { +WalkPdeResult ::Complete { .. } => { +// All PDE levels resolved. +return Ok (()); + } +WalkPdeResult ::Missing { + install_addr, + level, + } => { +// Drop PRAMIN before allocation. +drop (window); +let page = self.alloc_and_zero_page_table (mm, level)?; +let node =RBTreeNode ::new (install_addr, page,GFP_KERNEL )?; +let old = self.pt_pages.insert (node); +if old.is_some () { + kernel::pr_warn_once!( +"VMM: duplicate install_addr in pt_pages (internal consistency error)\n" + ); +return Err (EIO ); + } + +// Loop: re-acquire PRAMIN and re-walk from root. + } + } + } + + kernel::pr_warn!( +"VMM: ensure_pte_path: loop exhausted after {} iters (VFN {:?})\n" , + max_iter, + vfn + ); +Err (EIO ) + } + +/// Prepare resources for mapping `num_pages` pages. +/// +/// Allocates a contiguous VA range, then walks the hierarchy per-VFN to prepare pages +/// for all missing PDEs. Returns a [`PreparedMapping`] with the VA allocation. +/// +/// If `va_range` is not `None`, the VA range is constrained to the given range. Safe +/// to call outside the fence signalling critical path. +pub (crate )fn prepare_map ( + &mut self, + mm: &GpuMm , + num_pages:usize , + va_range:Option <Range <u64 >>, + ) ->Result <PreparedMapping > { +if num_pages ==0 { +return Err (EINVAL ); + } + +// Pre-reserve so execute_map() can use push_within_capacity (no alloc in +// fence signalling critical path). +// Upper bound on page table pages needed for the full tree (PTE pages + PDE +// pages at all levels). +let pt_upper_bound = self.mmu_version.pt_pages_upper_bound (num_pages); + self.page_table_allocs.reserve (pt_upper_bound,GFP_KERNEL )?; + +// Allocate contiguous VA range. +let (vfn_start, vfn_alloc) = self.alloc_vfn_range (num_pages, va_range)?; + +// Walk the hierarchy per-VFN to prepare pages for all missing PDEs. +for iin 0 ..num_pages { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (vfn_start.raw () + i_u64); + self.ensure_pte_path (mm, vfn)?; + } + +Ok (PreparedMapping { + vfn_start, + num_pages, + vfn_alloc, + }) + } + +/// Execute a prepared multi-page mapping. +/// +/// Drain prepared PT pages and install PDEs followed by single TLB flush. +pub (crate )fn execute_map ( + &mut self, + mm: &GpuMm , + prepared:PreparedMapping , + pfns: &[Pfn ], + writable:bool , + ) ->Result <MappedRange > { +if pfns.len () != prepared.num_pages { +return Err (EINVAL ); + } + +let PreparedMapping { + vfn_start, + num_pages, + vfn_alloc, + } = prepared; + +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let mut window = mm.pramin ().window ()?; + +// First, drain self.pt_pages, install all pending PDEs. +let mut cursor = self.pt_pages.cursor_front_mut (); +while let Some (c) = cursor { +let (next, node) = c.remove_current (); +let (install_addr, page) = node.to_key_value (); +let page_vram =VramAddress ::new (page.alloc.iter ().next ().ok_or (ENOMEM )?.offset ()); + +if page.level == self.mmu_version.dual_pde_level () { +let new_dpde =DualPde ::new_small (self.mmu_version,Pfn ::from (page_vram)); + new_dpde.write (&mut window, install_addr)?; + }else { +let new_pde =Pde ::new_vram (self.mmu_version,Pfn ::from (page_vram)); + new_pde.write (&mut window, install_addr)?; + } + +// Track the allocated pages in the `Vmm`. + self.page_table_allocs + .push_within_capacity (page.alloc) + .map_err (|_|ENOMEM )?; + + cursor = next; + } + +// Next, write PTEs (all PDEs now installed in HW). +for (i, &pfn)in pfns.iter ().enumerate () { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (vfn_start.raw () + i_u64); +let result = walker.walk_to_pte_lookup_with_window (&mut window, vfn)?; + +match result { +WalkResult ::Unmapped { pte_addr } |WalkResult ::Mapped { pte_addr, .. } => { +let pte =Pte ::new_vram (self.mmu_version, pfn, writable); + pte.write (&mut window, pte_addr)?; + } +WalkResult ::PageTableMissing => { + kernel::pr_warn_once!("VMM: page table missing for VFN {vfn:?}\n" ); +return Err (EIO ); + } + } + } + +drop (window); + +// Finally, flush the TLB. + mm.tlb ().flush (self.pdb_addr)?; + +Ok (MappedRange { + vfn_start, + num_pages, + _vfn_alloc: vfn_alloc, + _drop_guard:MustUnmapGuard ::new (), + }) + } + +/// Map pages doing prepare and execute in the same call. +/// +/// This is a convenience wrapper for callers outside the fence signalling critical +/// path (e.g., BAR mappings). For DRM usecases, [`Vmm::prepare_map()`] and +/// [`Vmm::execute_map()`] will be called separately. +pub (crate )fn map_pages ( + &mut self, + mm: &GpuMm , + pfns: &[Pfn ], + va_range:Option <Range <u64 >>, + writable:bool , + ) ->Result <MappedRange > { +if pfns.is_empty () { +return Err (EINVAL ); + } + +// Check if provided VA range is sufficient (if provided). +if let Some (ref range) = va_range { +let required:u64 = pfns + .len () + .checked_mul (PAGE_SIZE ) + .ok_or (EOVERFLOW )? + .into_safe_cast (); +let available = range.end.checked_sub (range.start).ok_or (EINVAL )?; +if available < required { +return Err (EINVAL ); + } + } + +let prepared = self.prepare_map (mm, pfns.len (), va_range)?; + self.execute_map (mm, prepared, pfns, writable) + } + +/// Unmap all pages in a [`MappedRange`] with a single TLB flush. +/// +/// Takes the range by value (consumes it), then invalidates PTEs for the range, +/// flushes the TLB, then drops the range (freeing the VA). PRAMIN lock is held. +pub (crate )fn unmap_pages (&mut self, mm: &GpuMm , range:MappedRange ) ->Result { +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let invalid_pte =Pte ::invalid (self.mmu_version); + +let mut window = mm.pramin ().window ()?; +for iin 0 ..range.num_pages { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (range.vfn_start.raw () + i_u64); +let result = walker.walk_to_pte_lookup_with_window (&mut window, vfn)?; + +match result { +WalkResult ::Mapped { pte_addr, .. } |WalkResult ::Unmapped { pte_addr } => { + invalid_pte.write (&mut window, pte_addr)?; + } +WalkResult ::PageTableMissing => { +continue ; + } + } + } +drop (window); + + mm.tlb ().flush (self.pdb_addr)?; + +// TODO: Internal page table pages (PDE, PTE pages) are still kept around. +// This is by design as repeated maps/unmaps will be fast. As a future TODO, +// we can add a reclaimer here to reclaim if VRAM is short. For now, the PT +// pages are dropped once the `Vmm` is dropped. + + range._drop_guard.disarm ();// Unmap complete, Ok to drop MappedRange. +Ok (()) + } }
Signature · 1 line
2.34.1
PATCH v9 19/23 gpu: nova-core: Add BAR1 aperture type and size constant
In reply to Joel FernandesAdd BAR1_SIZE constant and Bar1 type alias for the 256MB BAR1 aperture. These are prerequisites for BAR1 memory access functionality.
--- drivers/gpu/nova-core/driver.rs | 8 +++++++- drivers/gpu/nova-core/gsp/commands.rs | 4 ++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 8 ++++++++ 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/driver.rs +7 -1
@@ -13,7 +13,10 @@ Vendor, // }, prelude::*, - sizes::SZ_16M, + sizes::{ + SZ_16M, + SZ_256M, // + }, sync::{ atomic::{Atomic ,
@@ -37,6 +40,7 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; +pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. //
@@ -47,6 +51,8 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; +#[expect(dead_code)] +pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; kernel::pci_device_table!(PCI_TABLE ,
drivers/gpu/nova-core/gsp/commands.rs +4 -0
@@ -190,6 +190,9 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// BAR1 Page Directory Entry base address. +#[expect(dead_code)] +pub (crate ) bar1_pde_base:u64 ,/// Usable FB (VRAM) region for driver memory allocation. pub (crate ) usable_fb_region:Range <u64 >,/// End of VRAM.
@@ -211,6 +214,7 @@ fn read(Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + bar1_pde_base: msg.bar1_pde_base (), usable_fb_region: base..base.saturating_add (size), total_fb_end, })
drivers/gpu/nova-core/gsp/fw/commands.rs +8 -0
@@ -117,6 +117,14 @@ impl GspStaticConfigInfo { self.0 .gpuNameString } +/// Returns the BAR1 Page Directory Entry base address. +/// +/// This is the root page table address for BAR1 virtual memory, +/// set up by GSP-RM firmware. +pub (crate )fn bar1_pde_base (&self) ->u64 { + self.0 .bar1PdeBase + } +/// Extract the first usable FB region from GSP firmware data. /// /// Returns the first region suitable for driver memory allocation as a `(base, size)` tuple.
Signature · 1 line
2.34.1
PATCH v9 20/23 gpu: nova-core: mm: Add BAR1 user interface
In reply to Joel FernandesAdd the BAR1 user interface for CPU access to GPU video memory through the BAR1 aperture.
--- drivers/gpu/nova-core/driver.rs | 1 - drivers/gpu/nova-core/gpu.rs | 17 ++- drivers/gpu/nova-core/gsp/commands.rs | 1 - drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/bar_user.rs | 156 ++++++++++++++++++++++++++ 5 files changed, 173 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/nova-core/mm/bar_user.rs
drivers/gpu/nova-core/driver.rs +0 -1
@@ -51,7 +51,6 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -#[expect(dead_code)] pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; kernel::pci_device_table!(
drivers/gpu/nova-core/gpu.rs +16 -1
@@ -24,7 +24,12 @@ commands::GetGspStaticInfoReply, Gsp, // }, - mm::GpuMm, + mm::{ + bar_user::BarUser, + pagetable::MmuVersion, + GpuMm, + VramAddress, // + }, regs, };
@@ -263,6 +268,8 @@ pub(crate) struct Gpu { gsp: Gsp,/// Static GPU information from GSP. gsp_static_info: GetGspStaticInfoReply, +/// BAR1 user interface for CPU access to GPU virtual memory. + bar_user: BarUser, }impl Gpu {
@@ -320,6 +327,14 @@ pub(crate) fn new<'a>( }, pramin_vram_region)? }, +// Create BAR1 user interface for CPU access to GPU virtual memory. + bar_user: { + let pdb_addr = VramAddress::new(gsp_static_info.bar1_pde_base); +let mmu_version =MmuVersion ::from (spec.chipset.arch ()); +let bar1_size = pdev.resource_len (1 )?; +BarUser ::new (pdb_addr, mmu_version, bar1_size)? + }, + bar: devres_bar, }) }
drivers/gpu/nova-core/gsp/commands.rs +0 -1
@@ -191,7 +191,6 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ],/// BAR1 Page Directory Entry base address. -#[expect(dead_code)] pub (crate ) bar1_pde_base:u64 ,/// Usable FB (VRAM) region for driver memory allocation. pub (crate ) usable_fb_region:Range <u64 >,
drivers/gpu/nova-core/mm.rs +1 -0
@@ -4,6 +4,7 @@#![expect(dead_code)] +pub (crate )mod bar_user;pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb;
drivers/gpu/nova-core/mm/bar_user.rs new file +156 -0
156 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 21/23 gpu: nova-core: mm: Add BAR1 memory management self-tests
In reply to Joel FernandesAdd self-tests for BAR1 access during driver probe when CONFIG_NOVA_MM_SELFTESTS is enabled (default disabled). This results in testing the Vmm, GPU buddy allocator and BAR1 region all of which should function correctly for the tests to pass.
--- drivers/gpu/nova-core/Kconfig | 10 ++ drivers/gpu/nova-core/driver.rs | 2 + drivers/gpu/nova-core/gpu.rs | 38 ++++ drivers/gpu/nova-core/mm/bar_user.rs | 256 +++++++++++++++++++++++++++ 4 files changed, 306 insertions(+)
drivers/gpu/nova-core/Kconfig +10 -0
@@ -15,3 +15,13 @@ config NOVA_CORE This driver is work in progress and may not be functional. If M is selected, the module will be called nova_core. + +config NOVA_MM_SELFTESTS + bool "Memory management self-tests" + depends on NOVA_CORE + help + Enable self-tests for the memory management subsystem. When enabled, + tests are run during GPU probe to verify PRAMIN aperture access, + page table walking, and BAR1 virtual memory mapping functionality. + + This is a testing option and is default-disabled.
drivers/gpu/nova-core/driver.rs +2 -0
@@ -101,6 +101,8 @@ fn probe(pdev: &pci::Device<Core>, _info: &Self::IdInfo) -> impl PinInit<Self, E Ok(try_pin_init!(Self { gpu <- Gpu::new(pdev, bar.clone(), bar.access(pdev.as_ref())?), + // Run optional GPU selftests. + _: { gpu.run_selftests(pdev)? }, _reg <- auxiliary::Registration::new( pdev.as_ref(), c"nova-drm",
drivers/gpu/nova-core/gpu.rs +38 -0
@@ -349,4 +349,42 @@ pub(crate) fn unbind(&self, dev: &device::Device<device::Core>) { .inspect(|bar| self.sysmem_flush.unregister(bar)) .is_err()); } + +/// Run selftests on the constructed [`Gpu`]. +pub (crate )fn run_selftests ( +mut self:Pin <&mut Self >, + pdev: &pci::Device <device::Bound >, + ) ->Result { + self.as_mut ().run_mm_selftests (pdev)?; +Ok (()) + } + +#[cfg( CONFIG_NOVA_MM_SELFTESTS )] +fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { +use crate ::driver::BAR1_SIZE ; + +let mmu_version =MmuVersion ::from (self.spec.chipset.arch ()); + +// BAR1 self-tests. +let bar1 =Arc ::pin_init ( + pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), +GFP_KERNEL , + )?; +let bar1_access = bar1.access (pdev.as_ref ())?; + +crate ::mm::bar_user::run_self_test ( + pdev.as_ref (), + &self.mm, + bar1_access, + self.gsp_static_info.bar1_pde_base, + mmu_version, + )?; + +Ok (()) + } + +#[cfg(not( CONFIG_NOVA_MM_SELFTESTS ))] +fn run_mm_selftests (self:Pin <&mut Self >, _pdev: &pci::Device <device::Bound >) ->Result { +Ok (()) + } }
drivers/gpu/nova-core/mm/bar_user.rs +256 -0
259 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 22/23 gpu: nova-core: mm: Add PRAMIN aperture self-tests
In reply to Joel FernandesAdd self-tests for the PRAMIN aperture mechanism to verify correct operation during GPU probe. The tests validate various alignment requirements and corner cases.
The tests are default disabled and behind CONFIG_NOVA_MM_SELFTESTS. When enabled, tests run after GSP boot during probe.
--- drivers/gpu/nova-core/gpu.rs | 3 + drivers/gpu/nova-core/mm/pramin.rs | 209 +++++++++++++++++++++++++++++ 2 files changed, 212 insertions(+)
drivers/gpu/nova-core/gpu.rs +3 -0
@@ -365,6 +365,9 @@ fn run_mm_selftests(self: Pin<&mut Self>, pdev: &pci::Device<device::Bound>) ->let mmu_version =MmuVersion ::from (self.spec.chipset.arch ()); +// PRAMIN aperture self-tests. +crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?; +// BAR1 self-tests. let bar1 =Arc ::pin_init ( pdev.iomap_region_sized::<BAR1_SIZE>(1, c"nova-core/bar1"),
drivers/gpu/nova-core/mm/pramin.rs +209 -0
218 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v9 23/23 gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
In reply to Joel FernandesFrom: Zhi Wang <[email protected]>
Remove the hardcoded BAR1_SIZE = SZ_256M constant. On GPUs like L40 the BAR1 aperture is larger than 256MB; using a hardcoded size prevents large BAR1 from working and mapping it would fail.
--- drivers/gpu/nova-core/driver.rs | 8 ++------ drivers/gpu/nova-core/gpu.rs | 7 +------ 2 files changed, 3 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/driver.rs +2 -6
@@ -13,10 +13,7 @@ Vendor, // }, prelude::*, - sizes::{ - SZ_16M, - SZ_256M, // - }, + sizes::SZ_16M, sync::{ atomic::{Atomic ,
@@ -40,7 +37,6 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; -pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. //
@@ -51,7 +47,7 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; +pub (crate )type Bar1 = pci::Bar ; kernel::pci_device_table!(PCI_TABLE ,
drivers/gpu/nova-core/gpu.rs +1 -6
@@ -361,18 +361,13 @@ pub(crate) fn run_selftests(#[cfg( CONFIG_NOVA_MM_SELFTESTS )] fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { -use crate ::driver::BAR1_SIZE ; -let mmu_version =MmuVersion ::from (self.spec.chipset.arch ());// PRAMIN aperture self-tests. crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?;// BAR1 self-tests. -let bar1 =Arc ::pin_init ( - pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), -GFP_KERNEL , - )?; +let bar1 =Arc ::pin_init (pdev.iomap_region (1 ,c"nova-core/bar1" ),GFP_KERNEL )?;let bar1_access = bar1.access (pdev.as_ref ())?; crate::mm::bar_user::run_self_test(
Signature · 1 line
2.34.1
Patchwork
CI✗ Fi.CI.BUILD: failure for gpu: nova-core: Add memory management support (rev2)
In reply to Joel Fernandes== Series Details ==
== Series Details ==
Series: gpu: nova-core: Add memory management support (rev2)
URL : https://patchwork.freedesktop.org/series/162102/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/162102/revisions/2/mbox/ not applied
Applying: gpu: nova-core: Select GPU_BUDDY for VRAM allocation
Using index info to reconstruct a base tree...
M drivers/gpu/nova-core/Kconfig
Falling back to patching base and 3-way merge...
Auto-merging drivers/gpu/nova-core/Kconfig
CONFLICT (content): Merge conflict in drivers/gpu/nova-core/Kconfig
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 gpu: nova-core: Select GPU_BUDDY for VRAM allocation
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced
v10 cover gpu: nova-core: Add memory management support
In reply to Joel FernandesThis series adds support for nova-core memory management, including VRAM allocation, PRAMIN, VMM, page table walking, and BAR 1 read/writes. These are critical for channel management, vGPU, and all other memory management uses of nova-core.
The patches are based on drm-rust-next and work on Ampere, and should "just work on Blackwell" once John's Blackwell patches are merged, however it does not depend on those patches and can independently go in.
Change log:
Changes from v9 to v10:
- Rebased and dropped patches already merged in to drm-rust-next.
- GPU_BUDDY select folded into GpuMm patch.
- updated code with new register macro API.
- Refactored fb_regions() to use iterator (Alex Courbot).
- Renamed Pramin::window() to get_window() to make it more clear it is 'acquiring a resource'.
- Converted Bar0WindowTarget to bounded_enum! macro, replacing TryFrom. Allows to use with_* instead of try_with_*.
Changes from v8 to v9:
- Added fixes from Zhi Wang for bitfield position changes in virtual address and larger BAR1 size on some platforms. Tested and working for vGPU usecase!
- Refactored gsp: boot() to return only GspStaticInfo, removing FbLayout (Alex).
- bar1_pde_base and bar2_pde_base are now accessed via GspStaticInfo directly (Alex).
- Added new patch "gsp: Expose total physical VRAM end from FB region info" introducing total_fb_end() to expose VRAM extent (Alex).
- Consolidated usable VRAM and BarUser setup; removed dedicated
"fb: Add usable_vram field to FbLayout", "mm: Use usable VRAM region for buddy allocator", and "mm: Add BarUser to struct Gpu and create at boot".
Changes from v7 to v8:
- Incorporated "Select GPU_BUDDY for VRAM allocation" patch from the dependency series (Alex).
- Significant patch reordering for better logical flow (GSP/FB patches moved earlier, page table patches, Vmm, Bar1, tests) (Alex).
- Replaced several 'as' usages with into_safe_cast() (Danilo, Alex).
- Updated BAR 1 test cases to include exercising the block size API (Eliot, Danilo).
Changes from v6 to v7:
- Addressed DMA fence signalling usecase per Danilo's feedback.
Pre v6:
- Simplified PRAMIN code (John Hubbard, Alex Courbot).
- Handled different MMU versions: ver2 versus ver3 (John Hubbard).
- Added BAR1 usecase so we have user of DRM Buddy / VMM (John Hubbard).
- Iterating over clist/buddy bindings.
Link to v9: https://lore.kernel.org/all/[email protected]/
Link to v8: https://lore.kernel.org/all/[email protected]/
Link to v7: https://lore.kernel.org/all/[email protected]/
Joel Fernandes (20):
gpu: nova-core: gsp: Return GspStaticInfo from boot()
gpu: nova-core: gsp: Extract usable FB region from GSP
gpu: nova-core: gsp: Expose total physical VRAM end from FB region
info
gpu: nova-core: mm: Add support to use PRAMIN windows to write to VRAM
docs: gpu: nova-core: Document the PRAMIN aperture mechanism
gpu: nova-core: mm: Add common memory management types
gpu: nova-core: mm: Add TLB flush support
gpu: nova-core: mm: Add GpuMm centralized memory manager
gpu: nova-core: mm: Add common types for all page table formats
gpu: nova-core: mm: Add MMU v2 page table types
gpu: nova-core: mm: Add MMU v3 page table types
gpu: nova-core: mm: Add unified page table entry wrapper enums
gpu: nova-core: mm: Add page table walker for MMU v2/v3
gpu: nova-core: mm: Add Virtual Memory Manager
gpu: nova-core: mm: Add virtual address range tracking to VMM
gpu: nova-core: mm: Add multi-page mapping API to VMM
gpu: nova-core: Add BAR1 aperture type and size constant
gpu: nova-core: mm: Add BAR1 user interface
gpu: nova-core: mm: Add BAR1 memory management self-tests
gpu: nova-core: mm: Add PRAMIN aperture self-tests
Zhi Wang (1):
gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
Documentation/gpu/nova/core/pramin.rst | 123 +++++ Documentation/gpu/nova/index.rst | 1 + drivers/gpu/nova-core/Kconfig | 11 + drivers/gpu/nova-core/driver.rs | 3 + drivers/gpu/nova-core/gpu.rs | 94 +++- drivers/gpu/nova-core/gsp/boot.rs | 9 +- drivers/gpu/nova-core/gsp/commands.rs | 18 +- drivers/gpu/nova-core/gsp/fw/commands.rs | 59 ++- drivers/gpu/nova-core/mm.rs | 234 ++++++++++ drivers/gpu/nova-core/mm/bar_user.rs | 388 ++++++++++++++++ drivers/gpu/nova-core/mm/pagetable.rs | 489 ++++++++++++++++++++ drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 ++++++++++ drivers/gpu/nova-core/mm/pagetable/ver3.rs | 337 ++++++++++++++ drivers/gpu/nova-core/mm/pagetable/walk.rs | 218 +++++++++ drivers/gpu/nova-core/mm/pramin.rs | 489 ++++++++++++++++++++ drivers/gpu/nova-core/mm/tlb.rs | 95 ++++ drivers/gpu/nova-core/mm/vmm.rs | 499 +++++++++++++++++++++ drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/regs.rs | 52 +++ 19 files changed, 3344 insertions(+), 8 deletions(-) create mode 100644 Documentation/gpu/nova/core/pramin.rst create mode 100644 drivers/gpu/nova-core/mm.rs create mode 100644 drivers/gpu/nova-core/mm/bar_user.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver3.rs create mode 100644 drivers/gpu/nova-core/mm/pagetable/walk.rs create mode 100644 drivers/gpu/nova-core/mm/pramin.rs create mode 100644 drivers/gpu/nova-core/mm/tlb.rs create mode 100644 drivers/gpu/nova-core/mm/vmm.rs
base-commit: 76bce7ac51673640a4a46236ea723cf5543268d7
Signature · 1 line
2.34.1
PATCH v10 1/21 gpu: nova-core: gsp: Return GspStaticInfo from boot()
Refactor the GSP boot function to return only the GspStaticInfo, removing the FbLayout from the return tuple.
This enables access required for memory management initialization to:
- bar1_pde_base: BAR1 page directory base.
- bar2_pde_base: BAR2 page directory base.
- usable memory regions in vidmem.
--- drivers/gpu/nova-core/gpu.rs | 9 +++++++-- drivers/gpu/nova-core/gsp/boot.rs | 9 ++++++--- 2 files changed, 13 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/gpu.rs +7 -2
@@ -21,7 +21,10 @@ }, fb::SysmemFlush, gfw, - gsp::Gsp, + gsp::{ + commands::GetGspStaticInfoReply, + Gsp, // + }, regs, };
@@ -238,6 +241,8 @@ pub(crate) struct Gpu {/// GSP runtime data. Temporarily an empty placeholder. #[pin] gsp: Gsp, +/// Static GPU information from GSP. + gsp_static_info: GetGspStaticInfoReply, }impl Gpu {
@@ -269,7 +274,7 @@ pub(crate) fn new<'a>( gsp <- Gsp::new(pdev), - _: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, + gsp_static_info: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, bar: devres_bar, })
drivers/gpu/nova-core/gsp/boot.rs +6 -3
@@ -33,7 +33,10 @@ }, gpu::Chipset, gsp::{ - commands, + commands::{ + self, + GetGspStaticInfoReply, // + }, sequencer::{GspSequencer ,GspSequencerParams ,//
@@ -145,7 +148,7 @@ pub(crate) fn boot( chipset: Chipset, gsp_falcon: &Falcon<Gsp>, sec2_falcon: &Falcon<Sec2>, - ) -> Result { + ) -> Result<GetGspStaticInfoReply> {let dev = pdev.as_ref ();let bios =Vbios ::new (dev, bar)?;
@@ -235,6 +238,6 @@ pub(crate) fn boot( Err(e) => dev_warn!(pdev, "GPU name unavailable: {:?}\n", e), } -Ok (()) +Ok (info) } }
Signature · 1 line
2.34.1
Re: [PATCH v10 01/21] gpu: nova-core: gsp: Return GspStaticInfo from boot()
Quoted 11 lines · Joel Fernandes
Refactor the GSP boot function to return only the GspStaticInfo, removing the FbLayout from the return tuple.
This enables access required for memory management initialization to:
-bar1_pde_base: BAR1 page directory base.
-bar2_pde_base: BAR2 page directory base.
- usable memory regions in vidmem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Please see my feedback from v9[1] which still applies.
Alexandre Courbot
maintainerRe: [PATCH v10 01/21] gpu: nova-core: gsp: Return GspStaticInfo from boot()
In reply to Joel FernandesQuoted 2 lines · Joel Fernandes
Refactor the GSP boot function to return only the GspStaticInfo, removing the FbLayout from the return tuple.
We are not returning the FbLayout - that bit was introduced from an earlier revision of this series and is not in the original code.
PATCH v10 2/21 gpu: nova-core: gsp: Extract usable FB region from GSP
In reply to Joel FernandesAdd first_usable_fb_region() to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.
The extracted region is stored in GetGspStaticInfoReply and exposed as usable_fb_region field for use by the memory subsystem.
--- drivers/gpu/nova-core/gsp/commands.rs | 11 ++++-- drivers/gpu/nova-core/gsp/fw/commands.rs | 44 +++++++++++++++++++++++- 2 files changed, 52 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/gsp/commands.rs +9 -2
@@ -4,6 +4,7 @@ array, convert::Infallible, ffi::FromBytesUntilNulError, + ops::Range, str::Utf8Error, // };
@@ -189,22 +190,28 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> { } } -/// The reply from the GSP to the [`GetGspInfo`] command. +/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// Usable FB (VRAM) region for driver memory allocation. +#[expect(dead_code)] +pub (crate ) usable_fb_region:Range <u64 >, }impl MessageFromGsp for GetGspStaticInfoReply {const FUNCTION :MsgFunction =MsgFunction ::GetGspStaticInfo ;type Message =GspStaticConfigInfo ; -type InitError =Infallible ; +type InitError =Error ;fn read ( msg: &Self ::Message , _sbuffer: &mut SBufferIter <array::IntoIter <&[u8 ],2 >>, ) ->Result <Self ,Self ::InitError > { +let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + usable_fb_region: base..base.saturating_add (size), }) } }
drivers/gpu/nova-core/gsp/fw/commands.rs +43 -1
56 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v10 02/21] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 10 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed as
usable_fb_regionfield for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Please see my feedback from v9[1] which still applies.
[1]: https://lore.kernel.org/all/[email protected]/
thanks
Re: [PATCH v10 02/21] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 8 lines · Eliot Courtney
Quoted 7 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed as
usable_fb_regionfield for use by the memory subsystem.
Quoted 7 lines
Quoted 3 lines
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Please see my feedback from v9[1] which still applies.
Yeah, I am seeing it now. Amidst making the earlier 7.1 merge window for the DRM buddy and earlier patches in the series, I missed this. They seem to be simple nits and I will address them in the next revision. thanks,
--
Joel Fernandes
Re: [PATCH v10 02/21] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 17 lines · Joel Fernandes
Quoted 14 lines · Eliot Courtney
Quoted 10 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed as
usable_fb_regionfield for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Please see my feedback from v9[1] which still applies.
Quoted 8 lines
Quoted 1 line
Yeah, I am seeing it now. Amidst making the earlier 7.1 merge window for the DRM buddy and earlier patches in the series, I missed this. They seem to be simple nits and I will address them in the next revision. thanks,
--
Joel Fernandes
No worries. Sorry I have not gotten to more of the patches yet. Trying to get through some more now. Thanks!
Re: [PATCH v10 02/21] gpu: nova-core: gsp: Extract usable FB region from GSP
Quoted 24 lines · Eliot Courtney
Quoted 23 lines · Joel Fernandes
Quoted 8 lines · Eliot Courtney
Quoted 7 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed as
usable_fb_regionfield for use by the memory subsystem.Quoted 7 lines
Quoted 3 lines
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>---
Please see my feedback from v9[1] which still applies.
Yeah, I am seeing it now. Amidst making the earlier 7.1 merge window for the DRM buddy and earlier patches in the series, I missed this. They seem to be simple nits and I will address them in the next revision. thanks,
Quoted 5 lines
Quoted 2 lines
--
Joel FernandesNo worries. Sorry I have not gotten to more of the patches yet. Trying to get through some more now. Thanks!
So far all the comments have been good ones, so thanks. ;-)
--
Joel Fernandes
Alexandre Courbot
maintainerRe: [PATCH v10 02/21] gpu: nova-core: gsp: Extract usable FB region from GSP
In reply to Joel FernandesQuoted 27 lines · Joel Fernandes
Add
first_usable_fb_region()to GspStaticConfigInfo to extract the first usable FB region from GSP's fbRegionInfoParams. Usable regions are those that are not reserved or protected.The extracted region is stored in GetGspStaticInfoReply and exposed as
usable_fb_regionfield for use by the memory subsystem.Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 11 ++++-- drivers/gpu/nova-core/gsp/fw/commands.rs | 44 +++++++++++++++++++++++- 2 files changed, 52 insertions(+), 3 deletions(-)drivers/gpu/nova-core/gsp/commands.rs +1
-0@@ -4,6 +4,7 @@ array, convert::Infallible, ffi::FromBytesUntilNulError, + ops::Range, str::Utf8Error, // };
drivers/gpu/nova-core/gsp/commands.rs fn init(&self) -> impl Init<Self::Command, Self::InitError> { · line 198
Quoted 10 lines
drivers/gpu/nova-core/gsp/commands.rs +4
-1@@ -189,22 +190,28 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> { } } -/// The reply from the GSP to the [`GetGspInfo`] command. +/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// Usable FB (VRAM) region for driver memory allocation. +#[expect(dead_code)] +pub (crate ) usable_fb_region:Range <u64 >,
Let's print the region when creating the GPU (using dev_info or dev_dbg) - this can be useful to the user, and lets us remove this dead_code.
PATCH v10 3/21 gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
In reply to Joel FernandesAdd total_fb_end() to GspStaticConfigInfo that computes the exclusive end address of the highest valid FB region covering both usable and GSP-reserved areas.
This allows callers to know the full physical VRAM extent, not just the allocatable portion.
--- drivers/gpu/nova-core/gsp/commands.rs | 6 ++++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 7 +++++++ 2 files changed, 13 insertions(+)
drivers/gpu/nova-core/gsp/commands.rs +6 -0
@@ -196,6 +196,9 @@ pub(crate) struct GetGspStaticInfoReply {/// Usable FB (VRAM) region for driver memory allocation. #[expect(dead_code)] pub(crate) usable_fb_region: Range<u64>, +/// End of VRAM. +#[expect(dead_code)] + pub(crate) total_fb_end: u64, }impl MessageFromGsp for GetGspStaticInfoReply {
@@ -209,9 +212,12 @@ fn read( ) -> Result<Self, Self::InitError> {let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +let total_fb_end = msg.total_fb_end ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), usable_fb_region: base..base.saturating_add (size), + total_fb_end, }) } }
drivers/gpu/nova-core/gsp/fw/commands.rs +7 -0
@@ -163,6 +163,13 @@ pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { } }) } + +/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { + self.fb_regions () + .map (|reg| reg.limit.saturating_add (1 )) + .max () + } }// SAFETY: Padding is explicit and will not contain uninitialized data.
Signature · 1 line
2.34.1
Re: [PATCH v10 03/21] gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
Quoted 45 lines · Joel Fernandes
Add
total_fb_end()toGspStaticConfigInfothat computes the exclusive end address of the highest valid FB region covering both usable and GSP-reserved areas.This allows callers to know the full physical VRAM extent, not just the allocatable portion.
Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 6 ++++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 7 +++++++ 2 files changed, 13 insertions(+)drivers/gpu/nova-core/gsp/commands.rs +6
-0@@ -196,6 +196,9 @@ pub(crate) struct GetGspStaticInfoReply {/// Usable FB (VRAM) region for driver memory allocation. #[expect(dead_code)] pub(crate) usable_fb_region: Range<u64>, +/// End of VRAM. +#[expect(dead_code)] + pub(crate) total_fb_end: u64, }impl MessageFromGsp for GetGspStaticInfoReply {@@ -209,9 +212,12 @@ fn read( ) -> Result<Self, Self::InitError> {let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +let total_fb_end = msg.total_fb_end ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), usable_fb_region: base..base.saturating_add (size), + total_fb_end, }) } }drivers/gpu/nova-core/gsp/fw/commands.rs +0
-0
drivers/gpu/nova-core/gsp/fw/commands.rs pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { · line 170
Quoted 8 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +5
-0@@ -163,6 +163,13 @@ pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { } }) } + +/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { + self.fb_regions () + .map (|reg| reg.limit.saturating_add (1 ))
I think it would be better to used checked_add here.
Quoted 5 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +2
-0+ .max() + } }// SAFETY: Padding is explicit and will not contain uninitialized data.
Re: [PATCH v10 03/21] gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
Quoted 6 lines · Eliot Courtney
drivers/gpu/nova-core/gsp/fw/commands.rs +4
-0+/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { + self.fb_regions () + .map (|reg| reg.limit.saturating_add (1 ))I think it would be better to used
checked_addhere.
Fixed, thanks.
Signature · 1 line
Joel Fernandes
Re: [PATCH v10 03/21] gpu: nova-core: gsp: Expose total physical VRAM end from FB region info
In reply to Eliot CourtneyQuoted 48 lines · Eliot Courtney
Quoted 47 lines · Joel Fernandes
Add
total_fb_end()toGspStaticConfigInfothat computes the exclusive end address of the highest valid FB region covering both usable and GSP-reserved areas.This allows callers to know the full physical VRAM extent, not just the allocatable portion.
Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/gsp/commands.rs | 6 ++++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 7 +++++++ 2 files changed, 13 insertions(+)drivers/gpu/nova-core/gsp/commands.rs +6
-0@@ -196,6 +196,9 @@ pub(crate) struct GetGspStaticInfoReply {/// Usable FB (VRAM) region for driver memory allocation. #[expect(dead_code)] pub(crate) usable_fb_region: Range<u64>, +/// End of VRAM. +#[expect(dead_code)] + pub(crate) total_fb_end: u64, }impl MessageFromGsp for GetGspStaticInfoReply {@@ -209,9 +212,12 @@ fn read( ) -> Result<Self, Self::InitError> {let (base, size) = msg.first_usable_fb_region ().ok_or (ENODEV )?; +let total_fb_end = msg.total_fb_end ().ok_or (ENODEV )?; +Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), usable_fb_region: base..base.saturating_add (size), + total_fb_end, }) } }drivers/gpu/nova-core/gsp/fw/commands.rs +0
-0@@ -163,6 +163,13 @@ pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { } })
Quoted 9 lines
drivers/gpu/nova-core/gsp/fw/commands.rs pub(crate) fn first_usable_fb_region(&self) -> Option<(u64, u64)> { · line 170
Quoted 6 lines
drivers/gpu/nova-core/gsp/fw/commands.rs +5
-0} + +/// Compute the end of physical VRAM from all FB regions. +pub (crate )fn total_fb_end (&self) ->Option <u64 > { + self.fb_regions () + .map (|reg| reg.limit.saturating_add (1 ))I think it would be better to used
checked_addhere.
Done, thanks.
--
Joel Fernandes
PATCH v10 4/21 gpu: nova-core: mm: Add support to use PRAMIN windows to write to VRAM
In reply to Joel FernandesPRAMIN apertures are a crucial mechanism to direct read/write to VRAM. Add support for the same.
--- drivers/gpu/nova-core/mm.rs | 5 + drivers/gpu/nova-core/mm/pramin.rs | 280 +++++++++++++++++++++++++++++ drivers/gpu/nova-core/nova_core.rs | 1 + drivers/gpu/nova-core/regs.rs | 10 ++ 4 files changed, 296 insertions(+) create mode 100644 drivers/gpu/nova-core/mm.rs create mode 100644 drivers/gpu/nova-core/mm/pramin.rs
drivers/gpu/nova-core/mm.rs new file +5 -0
@@ -0,0 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! Memory management subsystems for nova-core. + +pub (crate )mod pramin;
drivers/gpu/nova-core/mm/pramin.rs new file +280 -0
280 lines not on this page · raw message
drivers/gpu/nova-core/nova_core.rs +1 -0
@@ -20,6 +20,7 @@mod gfw;mod gpu;mod gsp; +mod mm;#[macro_use] mod num;mod regs;
drivers/gpu/nova-core/regs.rs +10 -0
@@ -30,6 +30,7 @@ Architecture, Chipset, // }, + mm::pramin::Bar0WindowTarget, num::FromSafeCast, };
@@ -115,6 +116,15 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result { } } +register! { +/// BAR0 window control for PRAMIN access. +pub (crate )NV_PBUS_BAR0_WINDOW (u32 ) @0x00001700 { +25 :24 target ?=>Bar0WindowTarget ; +/// Window base address (bits 39:16 of FB addr). +23 :0 window_base; + } +} +// PFB register! {
Signature · 1 line
2.34.1
PATCH v10 5/21 docs: gpu: nova-core: Document the PRAMIN aperture mechanism
In reply to Joel FernandesAdd documentation for the PRAMIN aperture mechanism used by nova-core for direct VRAM access.
Nova only uses TARGET=VRAM for VRAM access. The SYS_MEM target values are documented for completeness but not used by the driver.
--- Documentation/gpu/nova/core/pramin.rst | 123 +++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | 1 + 2 files changed, 124 insertions(+) create mode 100644 Documentation/gpu/nova/core/pramin.rst
Documentation/gpu/nova/core/pramin.rst new file +123 -0
@@ -0,0 +1,123 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================= +PRAMIN aperture mechanism +========================= + +.. note:: + The following description is approximate and current as of the Ampere family. + It may change for future generations and is intended to assist in understanding + the driver code. + +Introduction +============ + +PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video RAM (VRAM) before +the GPU's Memory Management Unit (MMU) and page tables are initialized. This 1MB sliding window, +located at a fixed offset within BAR0, is essential for setting up page tables and other critical +GPU data structures without relying on the GPU's MMU. + +Architecture Overview +===================== + +The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe Bus Controller Unit) +and provides a CPU-accessible window into VRAM through the PCIe interface:: + + +-----------------+ PCIe +------------------------------+ + | CPU |<----------->| GPU | + +-----------------+ | | + | +----------------------+ | + | | PBUS | | + | | (Bus Controller) | | + | | | | + | | +--------------+<------------ (window starts at + | | | PRAMIN | | | BAR0 + 0x700000) + | | | Window | | | + | | | (1MB) | | | + | | +--------------+ | | + | | | | | + | +---------|------------+ | + | | | + | v | + | +----------------------+<------------ (Program PRAMIN to any + | | VRAM | | 64KB-aligned VRAM boundary) + | | (Several GBs) | | + | | | | + | | FB[0x000000000000] | | + | | ... | | + | | FB[0x7FFFFFFFFFF] | | + | +----------------------+ | + +------------------------------+ + +PBUS (PCIe Bus Controller) is responsible for, among other things, handling MMIO +accesses to the BAR registers. + +PRAMIN Window Operation +======================= + +The PRAMIN window provides a 1MB sliding aperture that can be repositioned over +the entire VRAM address space using the ``NV_PBUS_BAR0_WINDOW`` register. + +Window Control Mechanism +------------------------- + +:: + + NV_PBUS_BAR0_WINDOW Register (0x1700): + +-------+--------+--------------------------------------+ + | 31:26 | 25:24 | 23:0 | + | RSVD | TARGET | BASE_ADDR | + | | | (bits 39:16 of VRAM address) | + +-------+--------+--------------------------------------+ + + The 24-bit BASE_ADDR field encodes bits [39:16] of the target VRAM address, + providing 40-bit (1TB) address space coverage with 64KB alignment. + + TARGET field (bits 25:24): + - 0x0: VRAM (Video Memory) + - 0x1: SYS_MEM_COH (Coherent System Memory) + - 0x2: SYS_MEM_NONCOH (Non-coherent System Memory) + - 0x3: Reserved + + .. note:: + Nova only uses TARGET=VRAM (0x0) for video memory access. The SYS_MEM + target values are documented here for hardware completeness but are + not used by the driver. + +64KB Alignment Requirement +--------------------------- + +The PRAMIN window must be aligned to 64KB boundaries in VRAM. This is enforced +by the ``BASE_ADDR`` field representing bits [39:16] of the target address:: + + VRAM Address Calculation: + actual_vram_addr = (BASE_ADDR << 16) + pramin_offset + Where: + - BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0] + - pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF] + + Example Window Positioning: + +---------------------------------------------------------+ + | VRAM Space | + | | + | 0x000000000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x0000FFFFF +-----------------+ | + | | + | | ^ | + | | | Window can slide | + | v | to any 64KB-aligned boundary | + | | + | 0x123400000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x1234FFFFF +-----------------+ | + | | + | ... | + | | + | 0x7FFFF0000 +-----------------+ <-- 64KB aligned | + | | PRAMIN Window | | + | | (1MB) | | + | 0x7FFFFFFFF +-----------------+ | + +---------------------------------------------------------+
Documentation/gpu/nova/index.rst +1 -0
@@ -32,3 +32,4 @@ vGPU manager VFIO driver and the nova-drm driver. core/devinit core/fwsec core/falcon + core/pramin
Signature · 1 line
2.34.1
PATCH v10 6/21 gpu: nova-core: mm: Add common memory management types
In reply to Joel FernandesAdd foundational types for GPU memory management. These types are used throughout the nova memory management subsystem for page table operations, address translation, and memory allocation.
--- drivers/gpu/nova-core/mm.rs | 159 ++++++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+)
drivers/gpu/nova-core/mm.rs +159 -0
@@ -2,4 +2,163 @@//! Memory management subsystems for nova-core. +#![expect(dead_code)] +pub (crate )mod pramin; + +use kernel::sizes::SZ_4K ; + +use crate ::num::u64_as_usize; + +/// Page size in bytes (4 KiB). +pub (crate )const PAGE_SIZE :usize =SZ_4K ; + +bitfield! { +pub (crate )struct VramAddress (u64 ),"Physical VRAM address in GPU video memory" { +11 :0 offsetas u64 ,"Offset within 4KB page" ; +63 :12 frame_numberas u64 =>Pfn ,"Physical frame number" ; + } +} + +impl VramAddress { +/// Create a new VRAM address from a raw value. +pub (crate )const fn new (addr:u64 ) ->Self { +Self (addr) + } + +/// Get the raw address value as `usize` (useful for MMIO offsets). +pub (crate )const fn raw (&self) ->usize { +u64_as_usize (self.0 ) + } + +/// Get the raw address value as `u64`. +pub (crate )const fn raw_u64 (&self) ->u64 { + self.0 + } +} + +impl PartialEq for VramAddress { +fn eq (&self, other: &Self ) ->bool { + self.0 == other.0 + } +} + +impl Eq for VramAddress {} + +impl PartialOrd for VramAddress { +fn partial_cmp (&self, other: &Self ) ->Option <core::cmp::Ordering > { +Some (self.cmp (other)) + } +} + +impl Ord for VramAddress { +fn cmp (&self, other: &Self ) -> core::cmp::Ordering { + self.0 .cmp (&other.0 ) + } +} + +impl From <Pfn >for VramAddress { +fn from (pfn:Pfn ) ->Self { +Self ::default ().set_frame_number (pfn) + } +} + +bitfield! { +pub (crate )struct VirtualAddress (u64 ),"Virtual address in GPU address space" { +11 :0 offsetas u64 ,"Offset within 4KB page" ; +63 :12 frame_numberas u64 =>Vfn ,"Virtual frame number" ; + } +} + +impl VirtualAddress { +/// Create a new virtual address from a raw value. +#[expect(dead_code)] +pub (crate )const fn new (addr:u64 ) ->Self { +Self (addr) + } + +/// Get the raw address value as `u64`. +pub (crate )const fn raw_u64 (&self) ->u64 { + self.0 + } +} + +impl From <Vfn >for VirtualAddress { +fn from (vfn:Vfn ) ->Self { +Self ::default ().set_frame_number (vfn) + } +} + +/// Physical Frame Number. +/// +/// Represents a physical page in VRAM. +#[repr(transparent)] +#[derive( Debug , Clone , Copy , PartialEq , Eq , Default )] +pub (crate )struct Pfn (u64 ); + +impl Pfn { +/// Create a new PFN from a frame number. +pub (crate )const fn new (frame_number:u64 ) ->Self { +Self (frame_number) + } + +/// Get the raw frame number. +pub (crate )const fn raw (self) ->u64 { + self.0 + } +} + +impl From <VramAddress >for Pfn { +fn from (addr:VramAddress ) ->Self { + addr.frame_number () + } +} + +impl From <u64 >for Pfn { +fn from (val:u64 ) ->Self { +Self (val) + } +} + +impl From <Pfn >for u64 { +fn from (pfn:Pfn ) ->Self { + pfn.0 + } +} + +/// Virtual Frame Number. +/// +/// Represents a virtual page in GPU address space. +#[repr(transparent)] +#[derive( Debug , Clone , Copy , PartialEq , Eq , Default )] +pub (crate )struct Vfn (u64 ); + +impl Vfn { +/// Create a new VFN from a frame number. +pub (crate )const fn new (frame_number:u64 ) ->Self { +Self (frame_number) + } + +/// Get the raw frame number. +pub (crate )const fn raw (self) ->u64 { + self.0 + } +} + +impl From <VirtualAddress >for Vfn { +fn from (addr:VirtualAddress ) ->Self { + addr.frame_number () + } +} + +impl From <u64 >for Vfn { +fn from (val:u64 ) ->Self { +Self (val) + } +} + +impl From <Vfn >for u64 { +fn from (vfn:Vfn ) ->Self { + vfn.0 + } +}
Signature · 1 line
2.34.1
PATCH v10 7/21 gpu: nova-core: mm: Add TLB flush support
In reply to Joel FernandesAdd TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;
drivers/gpu/nova-core/mm/tlb.rs new file +95 -0
95 lines not on this page · raw message
drivers/gpu/nova-core/regs.rs +42 -0
45 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 78 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +41
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +};
drivers/gpu/nova-core/mm/tlb.rs line 51
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. + #[pin]
This comment says that the lock is acquired during the DMA fence signalling critical path, but IIUC we don't have anything like that right now. Is this based on future yet to be done work? Can we reword this in a way so it makes sense in the current state?
Quoted 7 lines
drivers/gpu/nova-core/mm/tlb.rs new file +7
-0+ lock: Mutex<()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self {
drivers/gpu/nova-core/mm/tlb.rs line 68
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ bar, + lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings.
If this must be called after every operation like that, I wonder if we can change the design to require a guard like pattern something to ensure flush is called. WDYT?
drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 1 line
drivers/gpu/nova-core/mm/tlb.rs new file +1
-0+pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {
Hopefully we don't need to be calling flush() from anywhere in the entire crate. Can you tighten the visibility here and in other places? Many things seem to be pub(crate) that don't need to be.
Quoted 12 lines
drivers/gpu/nova-core/mm/tlb.rs new file +12
-0+let _guard = self.lock.lock (); + +let bar = self.bar.try_access ().ok_or (ENODEV )?; + +// Write PDB address. + bar.write_reg (regs::NV_TLB_FLUSH_PDB_LO ::from_pdb_addr (pdb_addr.raw_u64 ())); + bar.write_reg (regs::NV_TLB_FLUSH_PDB_HI ::from_pdb_addr (pdb_addr.raw_u64 ())); + +// Trigger flush: invalidate all pages and enable. + bar.write_reg ( + regs::NV_TLB_FLUSH_CTRL ::zeroed () + .with_page_all(true)
drivers/gpu/nova-core/mm/tlb.rs line 91
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ .with_enable(true), + ); + +// Poll for completion - enable bit clears when flush is done. +read_poll_timeout ( + ||Ok (bar.read (regs::NV_TLB_FLUSH_CTRL )), + |ctrl: ®s::NV_TLB_FLUSH_CTRL | !ctrl.enable (), +Delta ::ZERO , +Delta ::from_secs (2 ), + )?;
This has zero delay on the read_poll_timeout - what about adding a small delay of a microsecond or so?
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 10 lines · Eliot Courtney
drivers/gpu/nova-core/mm/tlb.rs line 51
Quoted 5 lines
drivers/gpu/nova-core/mm/tlb.rs new file +5
-0+/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] This comment says that the lock is acquired during the DMA fence signalling critical path, but IIUC we don't have anything like that right now. Is this based on future yet to be done work? Can we reword this in a way so it makes sense in the current state?
Good point. Will reword. I'll still keep the references this design is specifically for that, but will refine to avoid making it look like a fence signalling usecase already exists. Thanks!
Quoted 7 lines
drivers/gpu/nova-core/mm/tlb.rs line 68
Quoted 3 lines
drivers/gpu/nova-core/mm/tlb.rs new file +3
-0+/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. If this must be called after every operation like that, I wonder if we can change the design to require a guard like pattern something to ensure flush is called. WDYT?
That's a good thought. I looked into this -- currently there are only 2 call sites (execute_map and unmap_pages in vmm.rs), and both follow the same pattern of dropping the PRAMIN window lock before flushing. A RAII guard would need careful lifetime management to maintain this lock ordering, and error propagation from drop is tricky. With just 2 call sites, the explicit flush calls are clearer and easier to audit. If more call sites emerge in the future, we can revisit adding a guard pattern then.
Quoted 5 lines
Quoted 1 line
+ pub(crate) fn
flush(&self, pdb_addr: VramAddress)-> Result {Hopefully we don't need to be calling
flush()from anywhere in the entire crate. Can you tighten the visibility here and in other places? Many things seem to be pub(crate) that don't need to be.
Agreed, will tighten flush() to pub(super) and do a broader visibility audit across the series. Thanks!
Quoted 9 lines
drivers/gpu/nova-core/mm/tlb.rs line 91
Quoted 6 lines
drivers/gpu/nova-core/mm/tlb.rs new file +6
-0+read_poll_timeout ( + ||Ok (bar.read (regs::NV_TLB_FLUSH_CTRL )), + |ctrl: ®s::NV_TLB_FLUSH_CTRL | !ctrl.enable (), +Delta ::ZERO , +Delta ::from_secs (2 ), + )?;This has zero delay on the
read_poll_timeout- what about adding a small delay of a microsecond or so?
I think Delta::ZERO is fine here -- it still calls cpu_relax() on each iteration (not a pure busy-spin). This matches what other DRM drivers do such polls, e.g. lima_mmu.c and panfrost_gpu.c both use read_poll_timeout with sleep_us=0 in some places. I also measured it on GA102 and its about ~1-1.5us which I think is well within busy-loop territory. From my experience with the scheduler, if we sleep, such short sleeps will be much longer than 1us due to scheduling latency. I also checked OpenRM and even there it is a sleepless spin-loop.
thanks,
Signature · 1 line
Joel Fernandes
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
In reply to Joel FernandesQuoted 96 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +59
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +}; + +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] + lock:Mutex <()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self { + bar,
drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. +pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {
This landed on my list randomly, so I took a look.
Wouldn’t you want to virtualize the invalidation based on your device?
For example, what if you need to register interface changes on future hardware?
Or, if you’re a VF, can you even do MMIO?
I’d relayer this.
Matt
Quoted 79 lines
drivers/gpu/nova-core/mm/tlb.rs new file +26
-0+let _guard = self.lock.lock (); + +let bar = self.bar.try_access ().ok_or (ENODEV )?; + +// Write PDB address. + bar.write_reg (regs::NV_TLB_FLUSH_PDB_LO ::from_pdb_addr (pdb_addr.raw_u64 ())); + bar.write_reg (regs::NV_TLB_FLUSH_PDB_HI ::from_pdb_addr (pdb_addr.raw_u64 ())); + +// Trigger flush: invalidate all pages and enable. + bar.write_reg ( + regs::NV_TLB_FLUSH_CTRL ::zeroed () + .with_page_all (true ) + .with_enable (true ), + ); + +// Poll for completion - enable bit clears when flush is done. +read_poll_timeout ( + ||Ok (bar.read (regs::NV_TLB_FLUSH_CTRL )), + |ctrl: ®s::NV_TLB_FLUSH_CTRL | !ctrl.enable (), +Delta ::ZERO , +Delta ::from_secs (2 ), + )?; + +Ok (()) + } +}drivers/gpu/nova-core/regs.rs +42
-0@@ -548,3 +548,45 @@ pub(crate) mod ga100 { } } } + +// MMU TLB + +register! { +/// TLB flush register: PDB address bits [39:8]. +pub (crate )NV_TLB_FLUSH_PDB_LO (u32 ) @0x00b830a0 { +/// PDB address bits [39:8]. +31 :0 pdb_lo =>u32 ; + } + +/// TLB flush register: PDB address bits [47:40]. +pub (crate )NV_TLB_FLUSH_PDB_HI (u32 ) @0x00b830a4 { +/// PDB address bits [47:40]. +7 :0 pdb_hi =>u8 ; + } + +/// TLB flush control register. +pub (crate )NV_TLB_FLUSH_CTRL (u32 ) @0x00b830b0 { +/// Invalidate all pages. +0 :0 page_all =>bool ; +/// Enable/trigger flush (clears when flush completes). +31 :31 enable =>bool ; + } +} + +impl NV_TLB_FLUSH_PDB_LO { +/// Create a register value from a PDB address. +/// +/// Extracts bits [39:8] of the address and shifts it right by 8 bits. +pub (crate )fn from_pdb_addr (addr:u64 ) ->Self { +Self ::zeroed ().with_pdb_lo (((addr >>8 ) &0xFFFF_FFFF )as u32 ) + } +} + +impl NV_TLB_FLUSH_PDB_HI { +/// Create a register value from a PDB address. +/// +/// Extracts bits [47:40] of the address and shifts it right by 40 bits. +pub (crate )fn from_pdb_addr (addr:u64 ) ->Self { +Self ::zeroed ().with_pdb_hi (((addr >>40 ) &0xFF )as u8 ) + } +}Signature · 1 line
2.34.1
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 110 lines · Matthew Brost
Quoted 96 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +59
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +}; + +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] + lock:Mutex <()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self { + bar,drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. +pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {This landed on my list randomly, so I took a look.
Quoted 2 lines
Wouldn’t you want to virtualize the invalidation based on your device? For example, what if you need to register interface changes on future hardware?
Good point, for future hardware it indeed makes sense. I will do that.
Quoted 1 line
Or, if you’re a VF, can you even do MMIO?
For VFs, TLB flush is typically done directly via VF's BAR0 MMIO. If there is VF-specific handling for any reason, that should probably be a different interface than a per-chip interface (more specifically, checking if Nova is booted on a VF).
thanks,
--
Joel Fernandes
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 113 lines · Joel Fernandes
Quoted 110 lines · Matthew Brost
Quoted 96 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +59
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +}; + +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] + lock:Mutex <()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self { + bar,drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. +pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {This landed on my list randomly, so I took a look.
Quoted 4 lines
Quoted 2 lines
Wouldn’t you want to virtualize the invalidation based on your device? For example, what if you need to register interface changes on future hardware?
Good point, for future hardware it indeed makes sense. I will do that.
Actually, at least in the future as far as I can see, the register definitions are the same for TLB invalidation are the same, so we are good and I will not be making any change in this regard.
But, thanks for raising the point and forcing me to double check!
--
Joel Fernandes
Re: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 119 lines · Joel Fernandes
Quoted 116 lines · Joel Fernandes
Quoted 112 lines · Matthew Brost
Quoted 96 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +59
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +}; + +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] + lock:Mutex <()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self { + bar,drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. +pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {This landed on my list randomly, so I took a look.
Wouldn’t you want to virtualize the invalidation based on your device? For example, what if you need to register interface changes on future hardware?
Quoted 7 lines
Quoted 1 line
Good point, for future hardware it indeed makes sense. I will do that.
Actually, at least in the future as far as I can see, the register definitions are the same for TLB invalidation are the same, so we are good and I will not be making any change in this regard.
But, thanks for raising the point and forcing me to double check!
Not my driver, but this looks like a classic “works now” change that may not hold up later, which is why I replied to something that isn’t really my business.
Again, not my area, but I’ve been through this before. Generally, getting the abstractions right up front pays off.
Matt
Quoted 3 lines
--
Joel Fernandes
Alexandre Courbot
maintainerRe: [PATCH v10 07/21] gpu: nova-core: mm: Add TLB flush support
Quoted 128 lines · Matthew Brost
Quoted 119 lines · Joel Fernandes
Quoted 116 lines · Joel Fernandes
Quoted 112 lines · Matthew Brost
Quoted 96 lines · Joel Fernandes
Add TLB (Translation Lookaside Buffer) flush support for GPU MMU.
After modifying page table entries, the GPU's TLB must be invalidated to ensure the new mappings take effect. The Tlb struct provides flush functionality through BAR0 registers.
The flush operation writes the page directory base address and triggers an invalidation, polling for completion with a 2 second timeout matching the Nouveau driver.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/tlb.rs | 95 +++++++++++++++++++++++++++++++++ drivers/gpu/nova-core/regs.rs | 42 +++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/tlb.rsdrivers/gpu/nova-core/mm.rs +1
-0@@ -5,6 +5,7 @@#![expect(dead_code)] pub (crate )mod pramin; +pub (crate )mod tlb;use kernel::sizes::SZ_4K ;drivers/gpu/nova-core/mm/tlb.rs new file +59
-0@@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 + +//! TLB (Translation Lookaside Buffer) flush support for GPU MMU. +//! +//! After modifying page table entries, the GPU's TLB must be flushed to +//! ensure the new mappings take effect. This module provides TLB flush +//! functionality for virtual memory managers. +//! +//! # Example +//! +//! ```ignore +//! use crate::mm::tlb::Tlb; +//! +//! fn page_table_update(tlb: &Tlb, pdb_addr: VramAddress) -> Result<()> { +//! // ... modify page tables ... +//! +//! // Flush TLB to make changes visible (polls for completion). +//! tlb.flush(pdb_addr)?; +//! +//! Ok(()) +//! } +//! ``` + +use kernel::{ + devres::Devres , + io::poll::read_poll_timeout, + io::Io , + new_mutex, + prelude::*, + sync::{ +Arc , +Mutex ,// + }, + time::Delta ,// +}; + +use crate ::{ + driver::Bar0 , + mm::VramAddress , + regs,// +}; + +/// TLB manager for GPU translation buffer operations. +#[pin_data] +pub (crate )struct Tlb { + bar:Arc <Devres <Bar0 >>, +/// TLB flush serialization lock: This lock is acquired during the +/// DMA fence signalling critical path. It must NEVER be held across any +/// reclaimable CPU memory allocations because the memory reclaim path can +/// call `dma_fence_wait()`, which would deadlock with this lock held. +#[pin] + lock:Mutex <()>, +} + +impl Tlb { +/// Create a new TLB manager. +pub (super )fn new (bar:Arc <Devres <Bar0 >>) ->impl PinInit <Self > { +pin_init! (Self { + bar,drivers/gpu/nova-core/mm/tlb.rs line 69
Quoted 10 lines
drivers/gpu/nova-core/mm/tlb.rs new file +10
-0+ lock <- new_mutex!((), "tlb_flush"), + }) + } + +/// Flush the GPU TLB for a specific page directory base. +/// +/// This invalidates all TLB entries associated with the given PDB address. +/// Must be called after modifying page table entries to ensure the GPU sees +/// the updated mappings. +pub (crate )fn flush (&self, pdb_addr:VramAddress ) ->Result {This landed on my list randomly, so I took a look.
Wouldn’t you want to virtualize the invalidation based on your device? For example, what if you need to register interface changes on future hardware?
Quoted 7 lines
Quoted 1 line
Good point, for future hardware it indeed makes sense. I will do that.
Actually, at least in the future as far as I can see, the register definitions are the same for TLB invalidation are the same, so we are good and I will not be making any change in this regard.
But, thanks for raising the point and forcing me to double check!
Quoted 6 lines
Not my driver, but this looks like a classic “works now” change that may not hold up later, which is why I replied to something that isn’t really my business.
Again, not my area, but I’ve been through this before. Generally, getting the abstractions right up front pays off.
Our policy so far has been to introduce abstractions when there is a justifiable hardware difference within the (reduced) set of GPUs that we support. Adding HALs imply using virtual calls, which are harder to trace, or monomorphization using generic types, which complicate the code (and are hard to justify when there is only one implementation).
Tlb is already an abstraction of sorts; the register accesses are well-contained within this leaf module, so if the need arises to support a different scheme this can be done transparently to callers, which is what matter imho.
PATCH v10 8/21 gpu: nova-core: mm: Add GpuMm centralized memory manager
In reply to Joel FernandesIntroduce GpuMm as the centralized GPU memory manager that owns:
- Buddy allocator for VRAM allocation.
- PRAMIN window for direct VRAM access.
- TLB manager for translation buffer operations.
This provides clean ownership model where GpuMm provides accessor methods for its components that can be used for memory management operations.
--- drivers/gpu/nova-core/Kconfig | 1 + drivers/gpu/nova-core/gpu.rs | 34 ++++++++++++- drivers/gpu/nova-core/gsp/commands.rs | 2 - drivers/gpu/nova-core/mm.rs | 70 ++++++++++++++++++++++++++- 4 files changed, 102 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/Kconfig +1 -0
@@ -4,6 +4,7 @@ config NOVA_CORE depends on PCI depends on RUST select AUXILIARY_BUS + select GPU_BUDDY select RUST_FW_LOADER_ABSTRACTIONS default n help
drivers/gpu/nova-core/gpu.rs +33 -1
@@ -4,10 +4,13 @@ device, devres::Devres, fmt, + gpu::buddy::GpuBuddyParams, io::Io, num::Bounded, pci, prelude::*, + ptr::Alignment, + sizes::SZ_4K, sync::Arc, // };
@@ -25,6 +28,7 @@ commands::GetGspStaticInfoReply, Gsp, // }, + mm::GpuMm, regs, };
@@ -238,6 +242,9 @@ pub(crate) struct Gpu { gsp_falcon: Falcon<GspFalcon>, /// SEC2 falcon instance, used for GSP boot up and cleanup. sec2_falcon: Falcon<Sec2Falcon>, +/// GPU memory manager owning memory management resources. +#[pin] + mm: GpuMm,/// GSP runtime data. Temporarily an empty placeholder. #[pin] gsp: Gsp,
@@ -274,7 +281,32 @@ pub(crate) fn new<'a>( gsp <- Gsp::new(pdev), - gsp_static_info: { gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)? }, + gsp_static_info: { + let info = gsp.boot(pdev, bar, spec.chipset, gsp_falcon, sec2_falcon)?; + +dev_info! ( + pdev.as_ref (), +"Using FB region: {:#x}..{:#x}\n" , + info.usable_fb_region.start, + info.usable_fb_region.end + ); + + info + }, + +// Create GPU memory manager owning memory management resources. + mm <- { +let usable_vram = &gsp_static_info.usable_fb_region; + +// PRAMIN covers all physical VRAM (including GSP-reserved areas +// above the usable region, e.g. the BAR1 page directory). +let pramin_vram_region =0 ..gsp_static_info.total_fb_end; +GpuMm ::new (devres_bar.clone (),GpuBuddyParams { + base_offset: usable_vram.start, + size: usable_vram.end - usable_vram.start, + chunk_size:Alignment ::new ::<SZ_4K >(), + }, pramin_vram_region)? + }, bar: devres_bar, })
drivers/gpu/nova-core/gsp/commands.rs +0 -2
@@ -194,10 +194,8 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ],/// Usable FB (VRAM) region for driver memory allocation. -#[expect(dead_code)] pub (crate ) usable_fb_region:Range <u64 >,/// End of VRAM. -#[expect(dead_code)] pub (crate ) total_fb_end:u64 , }
drivers/gpu/nova-core/mm.rs +68 -2
77 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 9/21 gpu: nova-core: mm: Add common types for all page table formats
In reply to Joel FernandesAdd common page table types shared between MMU v2 and v3. These types are hardware-agnostic and used by both MMU versions.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/pagetable.rs | 155 ++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -4,6 +4,7 @@#![expect(dead_code)] +pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb;
drivers/gpu/nova-core/mm/pagetable.rs new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 10/21 gpu: nova-core: mm: Add MMU v2 page table types
In reply to Joel FernandesAdd page table entry and directory structures for MMU version 2 used by Turing/Ampere/Ada GPUs.
--- drivers/gpu/nova-core/mm/pagetable.rs | 2 + drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 +++++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rs
drivers/gpu/nova-core/mm/pagetable.rs +2 -0
@@ -8,6 +8,8 @@#![expect(dead_code)] +pub (crate )mod ver2; +use crate ::gpu::Architecture ;/// Extracts the page table index at a given level from a virtual address.
drivers/gpu/nova-core/mm/pagetable/ver2.rs new file +232 -0
232 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v10 10/21] gpu: nova-core: mm: Add MMU v2 page table types
Quoted 16 lines · Joel Fernandes
Add page table entry and directory structures for MMU version 2 used by Turing/Ampere/Ada GPUs.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm/pagetable.rs | 2 + drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 +++++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rsdrivers/gpu/nova-core/mm/pagetable.rs +0
-0
drivers/gpu/nova-core/mm/pagetable.rs line 12
Quoted 5 lines
drivers/gpu/nova-core/mm/pagetable.rs +2
-0@@ -8,6 +8,8 @@#![expect(dead_code)] +pub (crate )mod ver2; +
This looks like it has more visibility than necessary. And it seems incorrect for anyone in the crate to care about MMU version details. This can probably be just 'mod ver2'. There are a lot of other types / functions in this series that could have tighter visibility. Could you go through and see if you can reduce a bunch to private or pub(super)?
thanks!
Re: [PATCH v10 10/21] gpu: nova-core: mm: Add MMU v2 page table types
Quoted 20 lines · Eliot Courtney
Quoted 19 lines · Joel Fernandes
Add page table entry and directory structures for MMU version 2 used by Turing/Ampere/Ada GPUs.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm/pagetable.rs | 2 + drivers/gpu/nova-core/mm/pagetable/ver2.rs | 232 +++++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver2.rsdrivers/gpu/nova-core/mm/pagetable.rs +0
-0@@ -8,6 +8,8 @@#![expect(dead_code)]
Quoted 9 lines
drivers/gpu/nova-core/mm/pagetable.rs line 12
Quoted 2 lines
drivers/gpu/nova-core/mm/pagetable.rs +2
-0+pub (crate )mod ver2; +This looks like it has more visibility than necessary. And it seems incorrect for anyone in the crate to care about MMU version details. This can probably be just 'mod ver2'. There are a lot of other types / functions in this series that could have tighter visibility. Could you go through and see if you can reduce a bunch to private or pub(super)?
Yes, indeed. I am tightening it now, there are several. Thanks!
--
Joel Fernandes
PATCH v10 11/21 gpu: nova-core: mm: Add MMU v3 page table types
In reply to Joel FernandesAdd page table entry and directory structures for MMU version 3 used by Hopper and later GPUs.
--- drivers/gpu/nova-core/mm/pagetable.rs | 1 + drivers/gpu/nova-core/mm/pagetable/ver3.rs | 337 +++++++++++++++++++++ 2 files changed, 338 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/ver3.rs
drivers/gpu/nova-core/mm/pagetable.rs +1 -0
@@ -9,6 +9,7 @@#![expect(dead_code)] pub (crate )mod ver2; +pub (crate )mod ver3;use crate ::gpu::Architecture ;
drivers/gpu/nova-core/mm/pagetable/ver3.rs new file +337 -0
337 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 12/21 gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesAdd unified Pte, Pde, and DualPde wrapper enums that abstract over MMU v2 and v3 page table entry formats. These enums allow the page table walker and VMM to work with both MMU versions.
Each unified type:
- Takes MmuVersion parameter in constructors
- Wraps both ver2 and ver3 variants
- Delegates method calls to the appropriate variant
This enables version-agnostic page table operations while keeping version-specific implementation details encapsulated in the ver2 and ver3 modules.
--- drivers/gpu/nova-core/mm/pagetable.rs | 330 ++++++++++++++++++++++++++ 1 file changed, 330 insertions(+)
drivers/gpu/nova-core/mm/pagetable.rs +330 -0
@@ -12,6 +12,13 @@pub (crate )mod ver3;use crate ::gpu::Architecture ; +use crate ::mm::{ + pramin, +Pfn , +VirtualAddress , +VramAddress ,// +}; +use kernel::prelude::*;/// Extracts the page table index at a given level from a virtual address. pub (crate )trait VaLevelIndex {
@@ -84,6 +91,96 @@ pub(crate) const fn as_index(&self) -> u64 { } } +impl MmuVersion { +/// Get the `PDE` levels (excluding PTE level) for page table walking. +pub (crate )fn pde_levels (&self) -> &'static [PageTableLevel ] { +match self { +Self ::V2 => ver2::PDE_LEVELS , +Self ::V3 => ver3::PDE_LEVELS , + } + } + +/// Get the PTE level for this MMU version. +pub (crate )fn pte_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::PTE_LEVEL , +Self ::V3 => ver3::PTE_LEVEL , + } + } + +/// Get the dual PDE level (128-bit entries) for this MMU version. +pub (crate )fn dual_pde_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::DUAL_PDE_LEVEL , +Self ::V3 => ver3::DUAL_PDE_LEVEL , + } + } + +/// Get the number of PDE levels for this MMU version. +pub (crate )fn pde_level_count (&self) ->usize { + self.pde_levels ().len () + } + +/// Get the entry size in bytes for a given level. +pub (crate )fn entry_size (&self, level:PageTableLevel ) ->usize { +if level == self.dual_pde_level () { +16 // 128-bit dual PDE + }else { +8 // 64-bit PDE/PTE + } + } + +/// Get the number of entries per page table page for a given level. +pub (crate )fn entries_per_page (&self, level:PageTableLevel ) ->usize { +match self { +Self ::V2 =>match level { +// TODO: Calculate these values from the bitfield dynamically +// instead of hardcoding them. +PageTableLevel ::Pdb =>4 ,// PD3 root: bits [48:47] = 2 bits +PageTableLevel ::L3 =>256 ,// PD0 dual: bits [28:21] = 8 bits + _ =>512 ,// PD2, PD1, PT: 9 bits each + }, +Self ::V3 =>match level { +PageTableLevel ::Pdb =>2 ,// PDE4 root: bit [56] = 1 bit, 2 entries +PageTableLevel ::L4 =>256 ,// PDE0 dual: bits [28:21] = 8 bits + _ =>512 ,// PDE3, PDE2, PDE1, PT: 9 bits each + }, + } + } + +/// Extract the page table index at `level` from `va` for this MMU version. +pub (crate )fn level_index (&self, va:VirtualAddress , level:u64 ) ->u64 { +match self { +Self ::V2 => ver2::VirtualAddressV2 ::new (va).level_index (level), +Self ::V3 => ver3::VirtualAddressV3 ::new (va).level_index (level), + } + } + +/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); + +// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +/// Memory aperture for Page Table Entries (`PTE`s). /// /// Determines which memory region the `PTE` points to.
@@ -156,3 +253,236 @@ fn from(val: AperturePde) -> Self { valas u8 } } + +/// Unified Page Table Entry wrapper for both MMU v2 and v3 `PTE` +/// types, allowing the walker to work with either format. +#[derive( Debug , Clone , Copy )] +pub (crate )enum Pte { +/// MMU v2 `PTE` (Turing/Ampere/Ada). +V2 (ver2::Pte ), +/// MMU v3 `PTE` (Hopper+). +V3 (ver3::Pte ), +} + +impl Pte { +/// Create a `PTE` from a raw `u64` value for the given MMU version. +pub (crate )fn new (version:MmuVersion , val:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::new (val)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::new (val)), + } + } + +/// Create an invalid `PTE` for the given MMU version. +pub (crate )fn invalid (version:MmuVersion ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::invalid ()), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::invalid ()), + } + } + +/// Create a valid `PTE` for video memory. +pub (crate )fn new_vram (version:MmuVersion , pfn:Pfn , writable:bool ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pte ::new_vram (pfn, writable)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pte ::new_vram (pfn, writable)), + } + } + +/// Check if this `PTE` is valid. +pub (crate )fn is_valid (&self) ->bool { +match self { +Self ::V2 (p) => p.valid (), +Self ::V3 (p) => p.valid (), + } + } + +/// Get the physical frame number. +pub (crate )fn frame_number (&self) ->Pfn { +match self { +Self ::V2 (p) => p.frame_number (), +Self ::V3 (p) => p.frame_number (), + } + } + +/// Get the raw `u64` value. +pub (crate )fn raw_u64 (&self) ->u64 { +match self { +Self ::V2 (p) => p.raw_u64 (), +Self ::V3 (p) => p.raw_u64 (), + } + } + +/// Read a `PTE` from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let val = window.try_read64 (addr.raw ())?; +Ok (Self ::new (mmu_version, val)) + } + +/// Write this `PTE` to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.raw_u64 ()) + } +} + +/// Unified Page Directory Entry wrapper for both MMU v2 and v3 `PDE`. +#[derive( Debug , Clone , Copy )] +pub (crate )enum Pde { +/// MMU v2 `PDE` (Turing/Ampere/Ada). +V2 (ver2::Pde ), +/// MMU v3 `PDE` (Hopper+). +V3 (ver3::Pde ), +} + +impl Pde { +/// Create a `PDE` from a raw `u64` value for the given MMU version. +pub (crate )fn new (version:MmuVersion , val:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::new (val)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::new (val)), + } + } + +/// Create a valid `PDE` pointing to a page table in video memory. +pub (crate )fn new_vram (version:MmuVersion , table_pfn:Pfn ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::new_vram (table_pfn)), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::new_vram (table_pfn)), + } + } + +/// Create an invalid `PDE` for the given MMU version. +pub (crate )fn invalid (version:MmuVersion ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::Pde ::invalid ()), +MmuVersion ::V3 =>Self ::V3 (ver3::Pde ::invalid ()), + } + } + +/// Check if this `PDE` is valid. +pub (crate )fn is_valid (&self) ->bool { +match self { +Self ::V2 (p) => p.is_valid (), +Self ::V3 (p) => p.is_valid (), + } + } + +/// Get the memory aperture of this `PDE`. +pub (crate )fn aperture (&self) ->AperturePde { +match self { +Self ::V2 (p) => p.aperture (), +Self ::V3 (p) => p.aperture (), + } + } + +/// Get the VRAM address of the page table. +pub (crate )fn table_vram_address (&self) ->VramAddress { +match self { +Self ::V2 (p) => p.table_vram_address (), +Self ::V3 (p) => p.table_vram_address (), + } + } + +/// Get the raw `u64` value. +pub (crate )fn raw_u64 (&self) ->u64 { +match self { +Self ::V2 (p) => p.raw_u64 (), +Self ::V3 (p) => p.raw_u64 (), + } + } + +/// Read a `PDE` from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let val = window.try_read64 (addr.raw ())?; +Ok (Self ::new (mmu_version, val)) + } + +/// Write this `PDE` to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.raw_u64 ()) + } +} + +/// Unified Dual Page Directory Entry wrapper for both MMU v2 and v3 [`DualPde`]. +#[derive( Debug , Clone , Copy )] +pub (crate )enum DualPde { +/// MMU v2 [`DualPde`] (Turing/Ampere/Ada). +V2 (ver2::DualPde ), +/// MMU v3 [`DualPde`] (Hopper+). +V3 (ver3::DualPde ), +} + +impl DualPde { +/// Create a [`DualPde`] from raw 128-bit value (two `u64`s) for the given MMU version. +pub (crate )fn new (version:MmuVersion , big:u64 , small:u64 ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::DualPde ::new (big, small)), +MmuVersion ::V3 =>Self ::V3 (ver3::DualPde ::new (big, small)), + } + } + +/// Create a [`DualPde`] with only the small page table pointer set. +pub (crate )fn new_small (version:MmuVersion , table_pfn:Pfn ) ->Self { +match version { +MmuVersion ::V2 =>Self ::V2 (ver2::DualPde ::new_small (table_pfn)), +MmuVersion ::V3 =>Self ::V3 (ver3::DualPde ::new_small (table_pfn)), + } + } + +/// Check if the small page table pointer is valid. +pub (crate )fn has_small (&self) ->bool { +match self { +Self ::V2 (d) => d.has_small (), +Self ::V3 (d) => d.has_small (), + } + } + +/// Get the small page table VRAM address. +pub (crate )fn small_vram_address (&self) ->VramAddress { +match self { +Self ::V2 (d) => d.small.table_vram_address (), +Self ::V3 (d) => d.small.table_vram_address (), + } + } + +/// Get the raw `u64` value of the big PDE. +pub (crate )fn big_raw_u64 (&self) ->u64 { +match self { +Self ::V2 (d) => d.big.raw_u64 (), +Self ::V3 (d) => d.big.raw_u64 (), + } + } + +/// Get the raw `u64` value of the small PDE. +pub (crate )fn small_raw_u64 (&self) ->u64 { +match self { +Self ::V2 (d) => d.small.raw_u64 (), +Self ::V3 (d) => d.small.raw_u64 (), + } + } + +/// Read a dual PDE (128-bit) from VRAM. +pub (crate )fn read ( + window: &mut pramin::PraminWindow <'_>, + addr:VramAddress , + mmu_version:MmuVersion , + ) ->Result <Self > { +let lo = window.try_read64 (addr.raw ())?; +let hi = window.try_read64 (addr.raw () +8 )?; +Ok (Self ::new (mmu_version, lo, hi)) + } + +/// Write this dual PDE (128-bit) to VRAM. +pub (crate )fn write (&self, window: &mut pramin::PraminWindow <'_>, addr:VramAddress ) ->Result { + window.try_write64 (addr.raw (), self.big_raw_u64 ())?; + window.try_write64 (addr.raw () +8 , self.small_raw_u64 ()) + } +}
Signature · 1 line
2.34.1
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 121 lines · Joel Fernandes
Add unified Pte, Pde, and DualPde wrapper enums that abstract over MMU v2 and v3 page table entry formats. These enums allow the page table walker and VMM to work with both MMU versions.
Each unified type:
- Takes MmuVersion parameter in constructors
- Wraps both ver2 and ver3 variants
- Delegates method calls to the appropriate variantThis enables version-agnostic page table operations while keeping version-specific implementation details encapsulated in the ver2 and ver3 modules.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm/pagetable.rs | 330 ++++++++++++++++++++++++++ 1 file changed, 330 insertions(+)drivers/gpu/nova-core/mm/pagetable.rs +87
-0@@ -12,6 +12,13 @@pub (crate )mod ver3;use crate ::gpu::Architecture ; +use crate ::mm::{ + pramin, +Pfn , +VirtualAddress , +VramAddress ,// +}; +use kernel::prelude::*;/// Extracts the page table index at a given level from a virtual address. pub (crate )trait VaLevelIndex {@@ -84,6 +91,96 @@ pub(crate) const fn as_index(&self) -> u64 { } } +impl MmuVersion { +/// Get the `PDE` levels (excluding PTE level) for page table walking. +pub (crate )fn pde_levels (&self) -> &'static [PageTableLevel ] { +match self { +Self ::V2 => ver2::PDE_LEVELS , +Self ::V3 => ver3::PDE_LEVELS , + } + } + +/// Get the PTE level for this MMU version. +pub (crate )fn pte_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::PTE_LEVEL , +Self ::V3 => ver3::PTE_LEVEL , + } + } + +/// Get the dual PDE level (128-bit entries) for this MMU version. +pub (crate )fn dual_pde_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::DUAL_PDE_LEVEL , +Self ::V3 => ver3::DUAL_PDE_LEVEL , + } + } + +/// Get the number of PDE levels for this MMU version. +pub (crate )fn pde_level_count (&self) ->usize { + self.pde_levels ().len () + } + +/// Get the entry size in bytes for a given level. +pub (crate )fn entry_size (&self, level:PageTableLevel ) ->usize { +if level == self.dual_pde_level () { +16 // 128-bit dual PDE + }else { +8 // 64-bit PDE/PTE + } + } + +/// Get the number of entries per page table page for a given level. +pub (crate )fn entries_per_page (&self, level:PageTableLevel ) ->usize { +match self { +Self ::V2 =>match level { +// TODO: Calculate these values from the bitfield dynamically +// instead of hardcoding them. +PageTableLevel ::Pdb =>4 ,// PD3 root: bits [48:47] = 2 bits +PageTableLevel ::L3 =>256 ,// PD0 dual: bits [28:21] = 8 bits + _ =>512 ,// PD2, PD1, PT: 9 bits each + }, +Self ::V3 =>match level { +PageTableLevel ::Pdb =>2 ,// PDE4 root: bit [56] = 1 bit, 2 entries +PageTableLevel ::L4 =>256 ,// PDE0 dual: bits [28:21] = 8 bits + _ =>512 ,// PDE3, PDE2, PDE1, PT: 9 bits each + }, + } + } + +/// Extract the page table index at `level` from `va` for this MMU version. +pub (crate )fn level_index (&self, va:VirtualAddress , level:u64 ) ->u64 { +match self { +Self ::V2 => ver2::VirtualAddressV2 ::new (va).level_index (level), +Self ::V3 => ver3::VirtualAddressV3 ::new (va).level_index (level), + } + } + +/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); +
drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 183
Quoted 10 lines
drivers/gpu/nova-core/mm/pagetable.rs +10
-0+// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +
We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too? Then you can parameterise Vmm/PtWalk on this type.
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 129 lines · Eliot Courtney
Quoted 128 lines · Joel Fernandes
Add unified Pte, Pde, and DualPde wrapper enums that abstract over MMU v2 and v3 page table entry formats. These enums allow the page table walker and VMM to work with both MMU versions.
Each unified type:
- Takes MmuVersion parameter in constructors
- Wraps both ver2 and ver3 variants
- Delegates method calls to the appropriate variantThis enables version-agnostic page table operations while keeping version-specific implementation details encapsulated in the ver2 and ver3 modules.
Cc Nikola Djukic <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/mm/pagetable.rs | 330 ++++++++++++++++++++++++++ 1 file changed, 330 insertions(+)drivers/gpu/nova-core/mm/pagetable.rs +94
-0@@ -12,6 +12,13 @@pub (crate )mod ver3;use crate ::gpu::Architecture ; +use crate ::mm::{ + pramin, +Pfn , +VirtualAddress , +VramAddress ,// +}; +use kernel::prelude::*;/// Extracts the page table index at a given level from a virtual address. pub (crate )trait VaLevelIndex {@@ -84,6 +91,96 @@ pub(crate) const fn as_index(&self) -> u64 { } } +impl MmuVersion { +/// Get the `PDE` levels (excluding PTE level) for page table walking. +pub (crate )fn pde_levels (&self) -> &'static [PageTableLevel ] { +match self { +Self ::V2 => ver2::PDE_LEVELS , +Self ::V3 => ver3::PDE_LEVELS , + } + } + +/// Get the PTE level for this MMU version. +pub (crate )fn pte_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::PTE_LEVEL , +Self ::V3 => ver3::PTE_LEVEL , + } + } + +/// Get the dual PDE level (128-bit entries) for this MMU version. +pub (crate )fn dual_pde_level (&self) ->PageTableLevel { +match self { +Self ::V2 => ver2::DUAL_PDE_LEVEL , +Self ::V3 => ver3::DUAL_PDE_LEVEL , + } + } + +/// Get the number of PDE levels for this MMU version. +pub (crate )fn pde_level_count (&self) ->usize { + self.pde_levels ().len () + } + +/// Get the entry size in bytes for a given level. +pub (crate )fn entry_size (&self, level:PageTableLevel ) ->usize { +if level == self.dual_pde_level () { +16 // 128-bit dual PDE + }else { +8 // 64-bit PDE/PTE + } + } + +/// Get the number of entries per page table page for a given level. +pub (crate )fn entries_per_page (&self, level:PageTableLevel ) ->usize { +match self { +Self ::V2 =>match level { +// TODO: Calculate these values from the bitfield dynamically +// instead of hardcoding them. +PageTableLevel ::Pdb =>4 ,// PD3 root: bits [48:47] = 2 bits +PageTableLevel ::L3 =>256 ,// PD0 dual: bits [28:21] = 8 bits + _ =>512 ,// PD2, PD1, PT: 9 bits each + }, +Self ::V3 =>match level { +PageTableLevel ::Pdb =>2 ,// PDE4 root: bit [56] = 1 bit, 2 entries +PageTableLevel ::L4 =>256 ,// PDE0 dual: bits [28:21] = 8 bits + _ =>512 ,// PDE3, PDE2, PDE1, PT: 9 bits each + }, + } + } + +/// Extract the page table index at `level` from `va` for this MMU version. +pub (crate )fn level_index (&self, va:VirtualAddress , level:u64 ) ->u64 { +match self { +Self ::V2 => ver2::VirtualAddressV2 ::new (va).level_index (level), +Self ::V3 => ver3::VirtualAddressV3 ::new (va).level_index (level), + } + } + +/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); + +// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total
Quoted 8 lines
drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 102
Quoted 3 lines
drivers/gpu/nova-core/mm/pagetable.rs +3
-0+ } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Quoted 1 line
Then you can parameterise Vmm/PtWalk on this type.
The match still to be done somewhere, so you end up matching on chipset to call the correct parametrized functions versus just passing in the parameter or chipset down, in some cases.
For now I am inclined to leave it as is. Also there's a Rust pitfall we all learnt during the turing and other patch reviews, sometimes doing a bunch of matches is good especially if the number of variants are expected to be fixed (in the mm case, version 2 and version 3). Traits have some disadvantages too, example dyn traits have to heap-allocated, parametrizing can increase code size (due to monomorphization) etc.
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 31 lines · Joel Fernandes
Quoted 22 lines
Quoted 22 lines
drivers/gpu/nova-core/mm/pagetable.rs +22
-0+/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); + +// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + totalQuoted 8 lines
drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 102
Quoted 3 lines
drivers/gpu/nova-core/mm/pagetable.rs +3
-0+ } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
Quoted 2 lines
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Thanks for looking into it. Sorry to be a bother, but would you have a branch around with the code? I'm curious what didn't look good about it.
Quoted 6 lines
Quoted 1 line
Then you can parameterise Vmm/PtWalk on this type.
The match still to be done somewhere, so you end up matching on chipset to call the correct parametrized functions versus just passing in the parameter or chipset down, in some cases.
Quoted 6 lines
For now I am inclined to leave it as is. Also there's a Rust pitfall we all learnt during the turing and other patch reviews, sometimes doing a bunch of matches is good especially if the number of variants are expected to be fixed (in the mm case, version 2 and version 3). Traits have some disadvantages too, example dyn traits have to heap-allocated, parametrizing can increase code size (due to monomorphization) etc.
Yeah, it's just this is a lot of matches in a lot of places. And we have ver2 / ver3 specific code leaking into the general pagetable.rs file. So it would be really nice if we could find a way to improve this specific aspect. We can reduce the match to happening in just one file. You can avoid heap allocation if you would like by making Vmm an enum, for example, and doing the match based dispatch there at the top of the API tree, rather than at the bottom where it fans out into a lot more locations.
Quoted 5 lines
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Hi Eliot,
Quoted 32 lines · Eliot Courtney
Quoted 31 lines · Joel Fernandes
Quoted 30 lines
Quoted 15 lines
drivers/gpu/nova-core/mm/pagetable.rs +15
-0+/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); +drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 183
Quoted 10 lines
drivers/gpu/nova-core/mm/pagetable.rs +10
-0+// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
Quoted 5 lines
Quoted 2 lines
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Thanks for looking into it. Sorry to be a bother, but would you have a branch around with the code? I'm curious what didn't look good about it.
Sorry but I already mentioned that above, the parameterizing of dozens of function call sites, 3-4 new traits (because each struct like Pte/Pde/DualPde etc each need their own trait which different MMU versions implement) etc. The code because hard to read and readability is the top critical criteria for me - I am personally strictly against "Lets use shiny features in language at the cost of making code unreadable". Because that translates into bugs and nightmare for maintainability.
I don't have the code at the moment, but if you still want to spend on time on this direction, feel free to share a tree. I am happy to take a look.
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
Then you can parameterise Vmm/PtWalk on this type.
The match still to be done somewhere, so you end up matching on chipset to call the correct parametrized functions versus just passing in the parameter or chipset down, in some cases.
Quoted 6 lines
For now I am inclined to leave it as is. Also there's a Rust pitfall we all learnt during the turing and other patch reviews, sometimes doing a bunch of matches is good especially if the number of variants are expected to be fixed (in the mm case, version 2 and version 3). Traits have some disadvantages too, example dyn traits have to heap-allocated, parametrizing can increase code size (due to monomorphization) etc.
Quoted 2 lines
Yeah, it's just this is a lot of matches in a lot of places. And we have ver2 / ver3 specific code leaking into the general
pagetable.rsfile. So
That's not a leak, that's by design. pagetable.rs is where the matches are centralized, most of the code changes here on out should happen outside of this file.
31 out of 42 matches in the mm code are in pagetable.rs, so it is already centralized.
Quoted 2 lines
it would be really nice if we could find a way to improve this specific aspect. We can reduce the match to happening in just one file.
Assuming we know what we're improving. ;-)
Quoted 4 lines
You can> avoid heap allocation if you would like by making Vmm an enum, for example, and doing the match based dispatch there at the top of the API tree, rather than at the bottom where it fans out into a lot more locations.
heap allocation is not always free, this code sensitive to dynamic allocations in the kernel, due to MM reclaim and locking. I would like to keep it simple.
thanks,
--
Joel Fernandes
Alexandre Courbot
maintainerRe: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 41 lines · Joel Fernandes
Hi Eliot,
Quoted 32 lines · Eliot Courtney
Quoted 31 lines · Joel Fernandes
Quoted 30 lines
Quoted 15 lines
drivers/gpu/nova-core/mm/pagetable.rs +15
-0+/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); +drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 183
Quoted 10 lines
drivers/gpu/nova-core/mm/pagetable.rs +10
-0+// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
Quoted 5 lines
Quoted 2 lines
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Thanks for looking into it. Sorry to be a bother, but would you have a branch around with the code? I'm curious what didn't look good about it.
Quoted 7 lines
Sorry but I already mentioned that above, the parameterizing of dozens of function call sites, 3-4 new traits (because each struct like Pte/Pde/DualPde etc each need their own trait which different MMU versions implement) etc. The code because hard to read and readability is the top critical criteria for me - I am personally strictly against "Lets use shiny features in language at the cost of making code unreadable". Because that translates into bugs and nightmare for maintainability.
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at PtWalk::read_pte_at_level, there are already at least 6 if version == 2 { } else { } branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.
If you use a trait here, and make PtWalk generic against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated Bounded type for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.
In order to fully benefit from it, we will need the bitfield macro from the kernel crate so the PDE/PTE fields can be Bounded, I will try to make it available quickly in a patch that you can depend on.
But long story short, and although I need to dive deeper into the code, this looks like a good candidate for using a trait and associated types.
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Hi Alex, Eliot, Danilo,
Thanks for taking a look. Let me respond to the specific points below.
Quoted 10 lines · Alexandre Courbot
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at
PtWalk::read_pte_at_level, there are already at least 6if version == 2 { } else { }branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.
I did some measurements and there is no notieceable difference in both approaches. I ran perf and loaded nova with self-tests running. The extra potential branching is lost in the noise. In both cases, loading nova and running the self-tests has ~119.7M branch instructions on my Ampere. The total instruction count is also identical (~615M).
I measured like this:
perf stat -e branches,branch-misses,cache-references,cache-misses,instructions,cycles -- modprobe nova_core
So I think the branching argument is not a strong one. I also did more measurements and the dominant time taken is MMIO. During the map prep and execute, page table walks are done. A TLB flush alone costs ~1.4 microseconds. And PRAMIN BAR0 writes to write the PTE is also about 1 microsecond. Considering this, I don't think the extra branching argument holds (even without branch prediction and speculation).
Also some branches cannot be eliminated even with parameterization:
if level == self.mmu_version.dual_pde_level() {
// 128-bit dual PDE read
} else {
// Regular 64-bit PDE read
}This isn't really a version branch -- it's a structural branch that distinguishes between 64-bit PDE and 128-bit dual PDE entries. Any MMU version with a dual PDE level would need this same distinction.
I also did code-generation size analysis (see diff of code used below):
Code generation analysis:
Module .ko size: Before: 511,792 bytes After: 524,464 bytes (+2.5%) .text section: Before: 112,620 bytes After: 116,628 bytes (+4,008 bytes)
The +4K .text growth is the monomorphization cost: every generic function is compiled twice (once for MmuV2, once for MmuV3).
Quoted 4 lines
If you use a trait here, and make
PtWalkgeneric against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.
Actually I was the one who suggested traits for Falcon ucode descriptor if you see this thread [1]. So basically you and Eliot are telling me to do what I suggested in [1]. :-) However, I disagree that it is the right choice for this code.
[1] https://lore.kernel.org/all/20251117231028.GA1095236@joelbox2/
I think the two cases are quite different in complexity:
The falcon ucode descriptor is essentially a set of flat field accessors and a few params (imem_sec_load_params, dmem_load_params).
The trait has ~10 simple getter methods. There's no multi-level hierarchy, no walker, and no generic propagation.
The MMU page table case is structurally different. Making PtWalk generic over an Mmu trait would require:
- PtWalk<M: Mmu> (the walker)
- Plus all the associated types: M::Pte, M::Pde, M::DualPde each needing their own trait bounds
And we would also need:
- Vmm<M: Mmu> (which creates PtWalk)
- BarUser<M: Mmu> (which creates Vmm)
I am also against making Vmm an enum as Eliot suggested:
enum Vmm {
V2(VmmInner<MmuV2>),
V3(VmmInner<MmuV3>),
}That moves the version complexity up to the reader. Code complexity IMO should decrease as we go up abstractions, making it easier for users (Vmm/Bar).
If you look at the the changes in vmm.rs to handle version dispatch there [2]:
Added: +109
Removed: -28
[2]
https://github.com/Edgeworth/linux/commit/3627af550b61256184d589e7ec666c1108971f0e
The main benefit of my approach is version-specific dispatch complexity is completely isolated inside MmuVersion thus making the code outside of pagetable.rs much more readable, without having to parametrize anything, and without code size increase. I think that is worth considering.
Quoted 7 lines
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated
Boundedtype for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.
I think Bounded validation is orthogonal to the dispatch model. We can add Bounded to the current design without restructuring into traits. For example:
// In ver2::Pte
pub fn new_vram(pfn: Bounded<Pfn, 25>, writable: bool) -> Self { ... }
// In ver3::Pte
pub fn new_vram(pfn: Bounded<Pfn, 40>, writable: bool) -> Self { ... }The unified Pte enum wrapper already dispatches to the correct version-specific constructor, which would enforce the correct Bounded constraint for that version.
Quoted 3 lines
In order to fully benefit from it, we will need the bitfield macro from the
kernelcrate so the PDE/PTE fields can beBounded, I will try to make it available quickly in a patch that you can depend on.
That would be great, and I'd be happy to integrate Bounded validation once the macro is available. I just don't think we need to restructure the dispatch model in order to benefit from it.
Quoted 2 lines
But long story short, and although I need to dive deeper into the code, this looks like a good candidate for using a trait and associated types.
The walker code (walk.rs) is already version-agnostic and reads cleanly. The version dispatch is encapsulated behind method calls, not exposed as inline if/else blocks.
Generic propagation (or version-specific dispatch at higher levels) adds more complexity at higher layers.
Enclosed below [3] is the diff I used for my testing with the data, I don't really see a net readability win there (IMO, it is a net-loss in readability).
thanks,
--
Joel Fernandes
Alexandre Courbot
maintainerRe: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 16 lines · Joel Fernandes
Hi Alex, Eliot, Danilo,
Thanks for taking a look. Let me respond to the specific points below.
Quoted 10 lines · Alexandre Courbot
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at
PtWalk::read_pte_at_level, there are already at least 6if version == 2 { } else { }branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.
Quoted 5 lines
I did some measurements and there is no notieceable difference in both approaches. I ran perf and loaded nova with self-tests running. The extra potential branching is lost in the noise. In both cases, loading nova and running the self-tests has ~119.7M branch instructions on my Ampere. The total instruction count is also identical (~615M).
That's expected - as I said, CPUs are fast - and that's also not my point. My issue is that we are doing countless tests that all resolve to the code path, a code path that is already known during probe time. That's a huge code smell.
When we create the GPU, we know whether we will be using v2 or v3 page tables. That we need to test that again 12 times per address resolution is a design issue, irrespective of performance. There are 24 version match sites in patch 12 alone.
And that's precisely a good justification for using monomorphization. v2 and v3 are technically two different page table implementations (they even have their own distinct module in your series), we just use generics to factorize the (source) code a bit.
Quoted 21 lines
I measured like this:
perf stat -e branches,branch-misses,cache-references,cache-misses,instructions,cycles -- modprobenova_coreSo I think the branching argument is not a strong one. I also did more measurements and the dominant time taken is MMIO. During the map prep and execute, page table walks are done. A TLB flush alone costs ~1.4 microseconds. And PRAMIN BAR0 writes to write the PTE is also about 1 microsecond. Considering this, I don't think the extra branching argument holds (even without branch prediction and speculation).
Also some branches cannot be eliminated even with parameterization:
if level == self.mmu_version.dual_pde_level() { // 128-bit dual PDE read } else { // Regular 64-bit PDE read }
Quoted 3 lines
This isn't really a version branch -- it's a structural branch that distinguishes between 64-bit PDE and 128-bit dual PDE entries. Any MMU version with a dual PDE level would need this same distinction.
The dual PDE level should be an associated constant - you still need to do the test, but note that you would also do it if there was only a single page table version. It's orthogonal to whether we use a trait or not here.
Quoted 10 lines
I also did code-generation size analysis (see diff of code used below):
Code generation analysis:
Module .ko size: Before: 511,792 bytes After: 524,464 bytes (+2.5%) .text section: Before: 112,620 bytes After: 116,628 bytes (+4,008 bytes)The +4K .text growth is the monomorphization cost: every generic function is compiled twice (once for MmuV2, once for MmuV3).
I would say this is working as intended then.
Quoted 6 lines
Quoted 5 lines
If you use a trait here, and make
PtWalkgeneric against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.
Quoted 7 lines
Actually I was the one who suggested traits for Falcon ucode descriptor if you see this thread [1]. So basically you and Eliot are telling me to do what I suggested in [1]. :-) However, I disagree that it is the right choice for this code.
[1] https://lore.kernel.org/all/20251117231028.GA1095236@joelbox2/
I think the two cases are quite different in complexity:
Exactly. The complexity is different (this one involves multiple traits and associated types) but the pattern is the same - and that's a pattern traits are designed to address. If we were supposed to stop applying it when things go beyond a certain level of complexity, the conceptors of Rust would not have bothered addings things like associated types.
These traits are nothing new, they simply formalize a reality that already exists in your code, which is that each version of the page table needs to implement a given set of methods. It's already there with the version doing dispatches, only it is not articulated clearly to the reader. So in that respect, having traits make the code *more* readable imho.
Quoted 56 lines
The falcon ucode descriptor is essentially a set of flat field accessors and a few params (
imem_sec_load_params,dmem_load_params).
The trait has ~10 simple getter methods. There's no multi-level hierarchy, no walker, and no generic propagation.The MMU page table case is structurally different. Making PtWalk generic over an Mmu trait would require:
- PtWalk<M: Mmu> (the walker)
- Plus all the associated types:M::Pte,M::Pde,M::DualPdeeach needing their own trait boundsAnd we would also need:
- Vmm<M: Mmu> (which creates PtWalk)
- BarUser<M: Mmu> (which creates Vmm)I am also against making Vmm an enum as Eliot suggested:
enum Vmm { V2(VmmInner<MmuV2>), V3(VmmInner<MmuV3>), }That moves the version complexity up to the reader. Code complexity IMO should decrease as we go up abstractions, making it easier for users (Vmm/Bar).
If you look at the the changes in
vmm.rsto handle version dispatch there [2]:
Added: +109
Removed: -28[2]
https://github.com/Edgeworth/linux/commit/3627af550b61256184d589e7ec666c1108971f0eThe main benefit of my approach is version-specific dispatch complexity is completely isolated inside MmuVersion thus making the code outside of
pagetable.rsmuch more readable, without having to parametrize anything, and without code size increase. I think that is worth considering.Quoted 7 lines
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated
Boundedtype for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.I think Bounded validation is orthogonal to the dispatch model. We can add Bounded to the current design without restructuring into traits. For example:
// In ver2::Pte pub fn new_vram(pfn: Bounded<Pfn, 25>, writable: bool) -> Self { ... } // In ver3::Pte pub fn new_vram(pfn: Bounded<Pfn, 40>, writable: bool) -> Self { ... }
Quoted 3 lines
The unified Pte enum wrapper already dispatches to the correct version-specific constructor, which would enforce the correct Bounded constraint for that version.
But then what type does the new_vram dispatch method take? Generic code lets us expose the expected Bounded type to the caller, which can do the proper validation. This is a small example, but I expect this pattern to come up in other parts of the code as well.
Quoted 8 lines
Quoted 4 lines
In order to fully benefit from it, we will need the bitfield macro from the
kernelcrate so the PDE/PTE fields can beBounded, I will try to make it available quickly in a patch that you can depend on.That would be great, and I'd be happy to integrate Bounded validation once the macro is available. I just don't think we need to restructure the dispatch model in order to benefit from it.
I'll finish the series and hopefully send it a bit later today. That's another significant rework for the series (sorry about that) but it should be worth the effort for the added correctness.
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Hi Alex,
Quoted 33 lines · Alexandre Courbot
Quoted 16 lines · Joel Fernandes
Hi Alex, Eliot, Danilo,
Thanks for taking a look. Let me respond to the specific points below.
Quoted 10 lines · Alexandre Courbot
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at
PtWalk::read_pte_at_level, there are already at least 6if version == 2 { } else { }branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.Quoted 5 lines
I did some measurements and there is no notieceable difference in both approaches. I ran perf and loaded nova with self-tests running. The extra potential branching is lost in the noise. In both cases, loading nova and running the self-tests has ~119.7M branch instructions on my Ampere. The total instruction count is also identical (~615M).
That's expected - as I said, CPUs are fast - and that's also not my point. My issue is that we are doing countless tests that all resolve to the code path, a code path that is already known during probe time. That's a huge code smell.
When we create the GPU, we know whether we will be using v2 or v3 page tables. That we need to test that again 12 times per address resolution is a design issue, irrespective of performance. There are 24 version match sites in patch 12 alone.
Quoted 4 lines
And that's precisely a good justification for using monomorphization. v2 and v3 are technically two different page table implementations (they even have their own distinct module in your series), we just use generics to factorize the (source) code a bit.
To repeat my point though, the extra tests don't complicate the code or change performance. The bottleneck is here is MMIO, not CPU cycles (and arguably testing shows no change in CPU cycles either). So I tend to prefer cleaner code (which in my view is the few matches in pagetable.rs versus templatizing lots of code). I do see your point of view though and I Ok with (trying out) the alternative as well, and it seems to be what mostly everyone wants. It is a little more boiler plate in higher layers which are not exposed to version matching, but not the end of the world :).
Quoted 20 lines
Quoted 20 lines
I measured like this:
perf stat -e branches,branch-misses,cache-references,cache-misses,instructions,cycles -- modprobenova_coreSo I think the branching argument is not a strong one. I also did more measurements and the dominant time taken is MMIO. During the map prep and execute, page table walks are done. A TLB flush alone costs ~1.4 microseconds. And PRAMIN BAR0 writes to write the PTE is also about 1 microsecond. Considering this, I don't think the extra branching argument holds (even without branch prediction and speculation).
Also some branches cannot be eliminated even with parameterization:
if level == self.mmu_version.dual_pde_level() { // 128-bit dual PDE read } else { // Regular 64-bit PDE read }
Quoted 8 lines
Quoted 3 lines
This isn't really a version branch -- it's a structural branch that distinguishes between 64-bit PDE and 128-bit dual PDE entries. Any MMU version with a dual PDE level would need this same distinction.
The dual PDE level should be an associated constant - you still need to do the test, but note that you would also do it if there was only a single page table version. It's orthogonal to whether we use a trait or not here.
Sure, fair enough.
Quoted 6 lines
Quoted 6 lines
I also did code-generation size analysis (see diff of code used below):
Code generation analysis:
Quoted 7 lines
Quoted 5 lines
Module .ko size: Before: 511,792 bytes After: 524,464 bytes (+2.5%) .text section: Before: 112,620 bytes After: 116,628 bytes (+4,008 bytes)The +4K .text growth is the monomorphization cost: every generic function is compiled twice (once for MmuV2, once for MmuV3).
I would say this is working as intended then.
2.5% is not much for 1 feature but overtime IMO it can add up and become significant, I think we should keep an eye on size.
Quoted 19 lines
Quoted 5 lines
Quoted 4 lines
If you use a trait here, and make
PtWalkgeneric against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.Quoted 7 lines
Actually I was the one who suggested traits for Falcon ucode descriptor if you see this thread [1]. So basically you and Eliot are telling me to do what I suggested in [1]. :-) However, I disagree that it is the right choice for this code.
[1] https://lore.kernel.org/all/20251117231028.GA1095236@joelbox2/
I think the two cases are quite different in complexity:
Exactly. The complexity is different (this one involves multiple traits and associated types) but the pattern is the same - and that's a pattern traits are designed to address. If we were supposed to stop applying it when things go beyond a certain level of complexity, the conceptors of Rust would not have bothered addings things like associated types.
Quoted 6 lines
These traits are nothing new, they simply formalize a reality that already exists in your code, which is that each version of the page table needs to implement a given set of methods. It's already there with the version doing dispatches, only it is not articulated clearly to the reader. So in that respect, having traits make the code *more* readable imho.
Makes the lower layers more readable and the higher layers more complex. Me, I'd have preferred to keep the version matching stuff to the lower layers / centralized. Maybe there is still a way to do that, while still using traits.
Quoted 56 lines
Quoted 56 lines
The falcon ucode descriptor is essentially a set of flat field accessors and a few params (
imem_sec_load_params,dmem_load_params).
The trait has ~10 simple getter methods. There's no multi-level hierarchy, no walker, and no generic propagation.The MMU page table case is structurally different. Making PtWalk generic over an Mmu trait would require:
- PtWalk<M: Mmu> (the walker)
- Plus all the associated types:M::Pte,M::Pde,M::DualPdeeach needing their own trait boundsAnd we would also need:
- Vmm<M: Mmu> (which creates PtWalk)
- BarUser<M: Mmu> (which creates Vmm)I am also against making Vmm an enum as Eliot suggested:
enum Vmm { V2(VmmInner<MmuV2>), V3(VmmInner<MmuV3>), }That moves the version complexity up to the reader. Code complexity IMO should decrease as we go up abstractions, making it easier for users (Vmm/Bar).
If you look at the the changes in
vmm.rsto handle version dispatch there [2]:
Added: +109
Removed: -28[2]
https://github.com/Edgeworth/linux/commit/3627af550b61256184d589e7ec666c1108971f0eThe main benefit of my approach is version-specific dispatch complexity is completely isolated inside MmuVersion thus making the code outside of
pagetable.rsmuch more readable, without having to parametrize anything, and without code size increase. I think that is worth considering.Quoted 7 lines
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated
Boundedtype for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.I think Bounded validation is orthogonal to the dispatch model. We can add Bounded to the current design without restructuring into traits. For example:
// In ver2::Pte pub fn new_vram(pfn: Bounded<Pfn, 25>, writable: bool) -> Self { ... } // In ver3::Pte pub fn new_vram(pfn: Bounded<Pfn, 40>, writable: bool) -> Self { ... }
Quoted 8 lines
Quoted 3 lines
The unified Pte enum wrapper already dispatches to the correct version-specific constructor, which would enforce the correct Bounded constraint for that version.
But then what type does the
new_vramdispatch method take? Generic code lets us expose the expectedBoundedtype to the caller, which can do the proper validation. This is a small example, but I expect this pattern to come up in other parts of the code as well.
Maybe I didn't follow your point, but the caller would do a version match and pass the correct bounded number of bits, but granted that's uglier if/once we switched to bounded types here.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In order to fully benefit from it, we will need the bitfield macro from the
kernelcrate so the PDE/PTE fields can beBounded, I will try to make it available quickly in a patch that you can depend on.
Quoted 7 lines
Quoted 3 lines
That would be great, and I'd be happy to integrate Bounded validation once the macro is available. I just don't think we need to restructure the dispatch model in order to benefit from it.
I'll finish the series and hopefully send it a bit later today. That's another significant rework for the series (sorry about that) but it should be worth the effort for the added correctness.
Sure!
thanks,
--
Joel Fernandes
Gary Guo
reviewerRe: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesQuoted 143 lines · Joel Fernandes
Hi Alex, Eliot, Danilo,
Thanks for taking a look. Let me respond to the specific points below.
Quoted 10 lines · Alexandre Courbot
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at
PtWalk::read_pte_at_level, there are already at least 6if version == 2 { } else { }branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.I did some measurements and there is no notieceable difference in both approaches. I ran perf and loaded nova with self-tests running. The extra potential branching is lost in the noise. In both cases, loading nova and running the self-tests has ~119.7M branch instructions on my Ampere. The total instruction count is also identical (~615M).
I measured like this:
perf stat -e branches,branch-misses,cache-references,cache-misses,instructions,cycles -- modprobenova_coreSo I think the branching argument is not a strong one. I also did more measurements and the dominant time taken is MMIO. During the map prep and execute, page table walks are done. A TLB flush alone costs ~1.4 microseconds. And PRAMIN BAR0 writes to write the PTE is also about 1 microsecond. Considering this, I don't think the extra branching argument holds (even without branch prediction and speculation).
Also some branches cannot be eliminated even with parameterization:
if level == self.mmu_version.dual_pde_level() { // 128-bit dual PDE read } else { // Regular 64-bit PDE read }This isn't really a version branch -- it's a structural branch that distinguishes between 64-bit PDE and 128-bit dual PDE entries. Any MMU version with a dual PDE level would need this same distinction.
I also did code-generation size analysis (see diff of code used below):
Code generation analysis:
Module .ko size: Before: 511,792 bytes After: 524,464 bytes (+2.5%) .text section: Before: 112,620 bytes After: 116,628 bytes (+4,008 bytes)The +4K .text growth is the monomorphization cost: every generic function is compiled twice (once for MmuV2, once for MmuV3).
Quoted 4 lines
If you use a trait here, and make
PtWalkgeneric against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.Actually I was the one who suggested traits for Falcon ucode descriptor if you see this thread [1]. So basically you and Eliot are telling me to do what I suggested in [1]. :-) However, I disagree that it is the right choice for this code.
[1] https://lore.kernel.org/all/20251117231028.GA1095236@joelbox2/
I think the two cases are quite different in complexity:
The falcon ucode descriptor is essentially a set of flat field accessors and a few params (
imem_sec_load_params,dmem_load_params).
The trait has ~10 simple getter methods. There's no multi-level hierarchy, no walker, and no generic propagation.The MMU page table case is structurally different. Making PtWalk generic over an Mmu trait would require:
- PtWalk<M: Mmu> (the walker)
- Plus all the associated types:M::Pte,M::Pde,M::DualPdeeach needing their own trait boundsAnd we would also need:
- Vmm<M: Mmu> (which creates PtWalk)
- BarUser<M: Mmu> (which creates Vmm)I am also against making Vmm an enum as Eliot suggested:
enum Vmm { V2(VmmInner<MmuV2>), V3(VmmInner<MmuV3>), }That moves the version complexity up to the reader. Code complexity IMO should decrease as we go up abstractions, making it easier for users (Vmm/Bar).
If you look at the the changes in
vmm.rsto handle version dispatch there [2]:
Added: +109
Removed: -28[2]
https://github.com/Edgeworth/linux/commit/3627af550b61256184d589e7ec666c1108971f0eThe main benefit of my approach is version-specific dispatch complexity is completely isolated inside MmuVersion thus making the code outside of
pagetable.rsmuch more readable, without having to parametrize anything, and without code size increase. I think that is worth considering.Quoted 7 lines
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated
Boundedtype for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.I think Bounded validation is orthogonal to the dispatch model. We can add Bounded to the current design without restructuring into traits. For example:
// In ver2::Pte pub fn new_vram(pfn: Bounded<Pfn, 25>, writable: bool) -> Self { ... } // In ver3::Pte pub fn new_vram(pfn: Bounded<Pfn, 40>, writable: bool) -> Self { ... }The unified Pte enum wrapper already dispatches to the correct version-specific constructor, which would enforce the correct Bounded constraint for that version.
Quoted 3 lines
In order to fully benefit from it, we will need the bitfield macro from the
kernelcrate so the PDE/PTE fields can beBounded, I will try to make it available quickly in a patch that you can depend on.That would be great, and I'd be happy to integrate Bounded validation once the macro is available. I just don't think we need to restructure the dispatch model in order to benefit from it.
Quoted 2 lines
But long story short, and although I need to dive deeper into the code, this looks like a good candidate for using a trait and associated types.
The walker code (
walk.rs) is already version-agnostic and reads cleanly. The version dispatch is encapsulated behind method calls, not exposed as inline if/else blocks.
Quoted 8 lines
Generic propagation (or version-specific dispatch at higher levels) adds more complexity at higher layers.
Enclosed below [3] is the diff I used for my testing with the data, I don't really see a net readability win there (IMO, it is a net-loss in readability).
IMO this diff is quite has got me quite in favour of trait approach.
I wanted about to purpose something similar (or maybe I had already?) trait approach some versions ago but didn't due to the eventual need of match like dispatch (like you had with vmm_dispatch), but your code made that looks not as bad as I thought it would be.
Best,
Gary
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 153 lines · Gary Guo
Quoted 143 lines · Joel Fernandes
Hi Alex, Eliot, Danilo,
Thanks for taking a look. Let me respond to the specific points below.
Quoted 10 lines · Alexandre Courbot
After a quick look I'd say that having a trait here would actually be *good* for correctness and maintainability.
The current design implies that every operation on a page table (most likely using the walker) goes through a branching point. Just looking at
PtWalk::read_pte_at_level, there are already at least 6if version == 2 { } else { }branches that all resolve to the same result. Include walking down the PDEs and you have at least a dozen of these just to resolve a virtual address. I know CPUs are fast, but this is still wasted cycles for no good reason.I did some measurements and there is no notieceable difference in both approaches. I ran perf and loaded nova with self-tests running. The extra potential branching is lost in the noise. In both cases, loading nova and running the self-tests has ~119.7M branch instructions on my Ampere. The total instruction count is also identical (~615M).
I measured like this:
perf stat -e branches,branch-misses,cache-references,cache-misses,instructions,cycles -- modprobenova_coreSo I think the branching argument is not a strong one. I also did more measurements and the dominant time taken is MMIO. During the map prep and execute, page table walks are done. A TLB flush alone costs ~1.4 microseconds. And PRAMIN BAR0 writes to write the PTE is also about 1 microsecond. Considering this, I don't think the extra branching argument holds (even without branch prediction and speculation).
Also some branches cannot be eliminated even with parameterization:
if level == self.mmu_version.dual_pde_level() { // 128-bit dual PDE read } else { // Regular 64-bit PDE read }This isn't really a version branch -- it's a structural branch that distinguishes between 64-bit PDE and 128-bit dual PDE entries. Any MMU version with a dual PDE level would need this same distinction.
I also did code-generation size analysis (see diff of code used below):
Code generation analysis:
Module .ko size: Before: 511,792 bytes After: 524,464 bytes (+2.5%) .text section: Before: 112,620 bytes After: 116,628 bytes (+4,008 bytes)The +4K .text growth is the monomorphization cost: every generic function is compiled twice (once for MmuV2, once for MmuV3).
Quoted 4 lines
If you use a trait here, and make
PtWalkgeneric against it, you can optimize this away. We had a similar situation when we introduced Turing support and the v2 ucode header, and tried both approaches: the trait-based one was slightly shorter, and arguably more readable.Actually I was the one who suggested traits for Falcon ucode descriptor if you see this thread [1]. So basically you and Eliot are telling me to do what I suggested in [1]. :-) However, I disagree that it is the right choice for this code.
[1] https://lore.kernel.org/all/20251117231028.GA1095236@joelbox2/
I think the two cases are quite different in complexity:
The falcon ucode descriptor is essentially a set of flat field accessors and a few params (
imem_sec_load_params,dmem_load_params).
The trait has ~10 simple getter methods. There's no multi-level hierarchy, no walker, and no generic propagation.The MMU page table case is structurally different. Making PtWalk generic over an Mmu trait would require:
- PtWalk<M: Mmu> (the walker)
- Plus all the associated types:M::Pte,M::Pde,M::DualPdeeach needing their own trait boundsAnd we would also need:
- Vmm<M: Mmu> (which creates PtWalk)
- BarUser<M: Mmu> (which creates Vmm)I am also against making Vmm an enum as Eliot suggested:
enum Vmm { V2(VmmInner<MmuV2>), V3(VmmInner<MmuV3>), }That moves the version complexity up to the reader. Code complexity IMO should decrease as we go up abstractions, making it easier for users (Vmm/Bar).
If you look at the the changes in
vmm.rsto handle version dispatch there [2]:
Added: +109
Removed: -28[2]
https://github.com/Edgeworth/linux/commit/3627af550b61256184d589e7ec666c1108971f0eThe main benefit of my approach is version-specific dispatch complexity is completely isolated inside MmuVersion thus making the code outside of
pagetable.rsmuch more readable, without having to parametrize anything, and without code size increase. I think that is worth considering.Quoted 7 lines
But the main argument to use a trait here IMO is that it enables associated types and constants. That's particularly critical since some equivalent fields have different lengths between v2 and v3. An associated
Boundedtype for these would force the caller to validate the length of these fields before calling a non-fallible operation, which is exactly the level of caution that we want when dealing with page tables.I think Bounded validation is orthogonal to the dispatch model. We can add Bounded to the current design without restructuring into traits. For example:
// In ver2::Pte pub fn new_vram(pfn: Bounded<Pfn, 25>, writable: bool) -> Self { ... } // In ver3::Pte pub fn new_vram(pfn: Bounded<Pfn, 40>, writable: bool) -> Self { ... }The unified Pte enum wrapper already dispatches to the correct version-specific constructor, which would enforce the correct Bounded constraint for that version.
Quoted 3 lines
In order to fully benefit from it, we will need the bitfield macro from the
kernelcrate so the PDE/PTE fields can beBounded, I will try to make it available quickly in a patch that you can depend on.That would be great, and I'd be happy to integrate Bounded validation once the macro is available. I just don't think we need to restructure the dispatch model in order to benefit from it.
Quoted 2 lines
But long story short, and although I need to dive deeper into the code, this looks like a good candidate for using a trait and associated types.
The walker code (
walk.rs) is already version-agnostic and reads cleanly. The version dispatch is encapsulated behind method calls, not exposed as inline if/else blocks.Quoted 8 lines
Generic propagation (or version-specific dispatch at higher levels) adds more complexity at higher layers.
Enclosed below [3] is the diff I used for my testing with the data, I don't really see a net readability win there (IMO, it is a net-loss in readability).
Quoted 7 lines
IMO this diff is quite has got me quite in favour of trait approach.
I wanted about to purpose something similar (or maybe I had already?) trait approach some versions ago but didn't due to the eventual need of
matchlike dispatch (like you had withvmm_dispatch), but your code made that looks not as bad as I thought it would be.
That's the drawback right, now vmm_dispatch has to deal with version difference where as before, the lower layers would. Maybe we can keep the vmm layer the way it is now, but do the dispatch itself at the lower layers, while still using traits like in the diff. I'll try that as well. :)
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesQuoted 49 lines · Joel Fernandes
Hi Eliot,
Quoted 32 lines · Eliot Courtney
Quoted 31 lines · Joel Fernandes
Quoted 30 lines
Quoted 15 lines
drivers/gpu/nova-core/mm/pagetable.rs +15
-0+/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); +drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 183
Quoted 10 lines
drivers/gpu/nova-core/mm/pagetable.rs +10
-0+// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
Quoted 5 lines
Quoted 2 lines
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Thanks for looking into it. Sorry to be a bother, but would you have a branch around with the code? I'm curious what didn't look good about it.
Sorry but I already mentioned that above, the parameterizing of dozens of function call sites, 3-4 new traits (because each struct like Pte/Pde/DualPde etc each need their own trait which different MMU versions implement) etc. The code because hard to read and readability is the top critical criteria for me - I am personally strictly against "Lets use shiny features in language at the cost of making code unreadable". Because that translates into bugs and nightmare for maintainability.
Quoted 2 lines
I don't have the code at the moment, but if you still want to spend on time on this direction, feel free to share a tree. I am happy to take a look.
I had a go at this, you can see the branch here [1] - it might not be perfect, but I think the shape is directionally good. It's structured so the HEAD commit has the diff from the current approach to the parametrised approach. The main decision is where to do the type erasure, I chose in Vmm since it looks like the main top level API for this code, but could do BarUser instead. I think it's overall better. I also think Alex's point about associated types making it easier to use the appropriate Bounded type is a good one.
[1]: https://github.com/Edgeworth/linux/commits/review/nova-mm-v10/
Quoted 28 lines
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
Then you can parameterise Vmm/PtWalk on this type.
The match still to be done somewhere, so you end up matching on chipset to call the correct parametrized functions versus just passing in the parameter or chipset down, in some cases.
Quoted 6 lines
For now I am inclined to leave it as is. Also there's a Rust pitfall we all learnt during the turing and other patch reviews, sometimes doing a bunch of matches is good especially if the number of variants are expected to be fixed (in the mm case, version 2 and version 3). Traits have some disadvantages too, example dyn traits have to heap-allocated, parametrizing can increase code size (due to monomorphization) etc.
Quoted 2 lines
Yeah, it's just this is a lot of matches in a lot of places. And we have ver2 / ver3 specific code leaking into the general
pagetable.rsfile. SoThat's not a leak, that's by design.
pagetable.rsis where the matches are centralized, most of the code changes here on out should happen outside of this file.31 out of 42 matches in the mm code are in
pagetable.rs, so it is already centralized.Quoted 2 lines
it would be really nice if we could find a way to improve this specific aspect. We can reduce the match to happening in just one file.
Assuming we know what we're improving. ;-)
Quoted 8 lines
Quoted 4 lines
You can> avoid heap allocation if you would like by making Vmm an enum, for example, and doing the match based dispatch there at the top of the API tree, rather than at the bottom where it fans out into a lot more locations.
heap allocation is not always free, this code sensitive to dynamic allocations in the kernel, due to MM reclaim and locking. I would like to keep it simple.
If you do the type erasure via enum in Vmm, you won't need to allocate it on the heap. The branch I posted above has an example on how to do this, although there might be a better way.
Quoted 5 lines
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Hi Eliot,
Quoted 62 lines · Eliot Courtney
Quoted 49 lines · Joel Fernandes
Hi Eliot,
Quoted 32 lines · Eliot Courtney
Quoted 31 lines · Joel Fernandes
Quoted 30 lines
Quoted 15 lines
drivers/gpu/nova-core/mm/pagetable.rs +15
-0+/// Compute upper bound on page table pages needed for `num_virt_pages`. +/// +/// Walks from PTE level up through PDE levels, accumulating the tree. +pub (crate )fn pt_pages_upper_bound (&self, num_virt_pages:usize ) ->usize { +let mut total =0 ; + +// PTE pages at the leaf level. +let pte_epp = self.entries_per_page (self.pte_level ()); +let mut pages_at_level = num_virt_pages.div_ceil (pte_epp); + total += pages_at_level; + +// Walk PDE levels bottom-up (reverse of pde_levels()). +for &levelin self.pde_levels ().iter ().rev () { +let epp = self.entries_per_page (level); +drivers/gpu/nova-core/mm/pagetable.rs pub(crate) const fn as_index(&self) -> u64 { · line 183
Quoted 10 lines
drivers/gpu/nova-core/mm/pagetable.rs +10
-0+// How many pages at this level do we need to point to +// the previous pages_at_level? + pages_at_level = pages_at_level.div_ceil (epp); + total += pages_at_level; + } + + total + } +} +We have a lot of matches on the MMU version here (and below in Pte, Pde, DualPde). What about making MmuVersion into a trait (e.g. Mmu) with associated types for Pte, Pde, DualPde which can implement traits defining their common operations too?
Quoted 5 lines
Quoted 2 lines
I coded this up and it did not look pretty, there's not much LOC savings and the code becomes harder to read because of parametrization of several functions. Also:
Thanks for looking into it. Sorry to be a bother, but would you have a branch around with the code? I'm curious what didn't look good about it.
Sorry but I already mentioned that above, the parameterizing of dozens of function call sites, 3-4 new traits (because each struct like Pte/Pde/DualPde etc each need their own trait which different MMU versions implement) etc. The code because hard to read and readability is the top critical criteria for me - I am personally strictly against "Lets use shiny features in language at the cost of making code unreadable". Because that translates into bugs and nightmare for maintainability.
Quoted 2 lines
I don't have the code at the moment, but if you still want to spend on time on this direction, feel free to share a tree. I am happy to take a look.
I had a go at this, you can see the branch here [1] - it might not be perfect, but I think the shape is directionally good. It's structured so the HEAD commit has the diff from the current approach to the parametrised approach. The main decision is where to do the type erasure, I chose in
Vmmsince it looks like the main top level API for this code, but could doBarUserinstead. I think it's overall better. I also think Alex's point about associated types making it easier to use the appropriate Bounded type is a good one.
Quoted 1 line
[1]: https://github.com/Edgeworth/linux/commits/review/nova-mm-v10/
First, thanks for the effort. I looked through this, its pretty much what I had before when I used traits. I don't think it is better to be honest. In fact your version is worse, it adds many new types and things like the following which I did not need before.
To put it mildly, the following suggestion should not be anywhere near my code:
/// Type-erased MMU-specific [Vmm] implementations.
enum VmmInner {
/// `Vmm` implementation for MMU v2.
V2(VmmImpl<MmuV2>),
/// `Vmm` implementation for MMU v3.
V3(VmmImpl<MmuV3>),
}/// MMU-specific [Vmm] implementation.
struct VmmImpl<M: Mmu> {
Seriously, I have to pass on this. :-)
And, you unfortunately seem to have ignored my point about requiring 4 NEW traits (Mmu, PteOps, PdeOps, DualPdeOps etc), which I did not need before. So you're making the code much much worse than before actually. We don't new traits and types pointlessly.
The only positive thing I could take away from your diff is the following (I thought I had already done that, but I'll double check).
- fn level_index(&self, level: u64) -> u64 {
+ fn level_index(&self, level: PageTableLevel) -> u64 {
Also you're parametrizing VirtualAddress as well which I did not have before:
- let va = VirtualAddress::from(vfn);
+ let va = M::va(VirtualAddress::from(vfn));
This is another step back.
Quoted 2 lines
I also think Alex's point about associated types making it easier to use the appropriate Bounded type is a good one.
I will reply to Alex thread, separately. I have some good data that should hopefully convince you and Alex that my approach in this patch is better (Version struct based dispatch than monomorphization). I would emphasize, as we all know, that we should make optimizations and changes based on real data and proper technical arguments so in the spirit of that, I have collected data with both approaches and I will reply to Alex's email with all that in there.
Also, the bounded types usage is orthogonal to version-parameterization. That can be done regardless, we already use bitfield macro in this code and can use bounded types within that if needed to restrict type creation. So I don't think we should mix the 2 concepts "bounded types" and "parameterization".
thanks,
--
Joel Fernandes
Danilo Krummrich
maintainerRe: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 2 lines · Joel Fernandes
So you're making the code much much worse than before actually. We don't new traits and types pointlessly.
I had a look at both approaches and yes, the traits can be considered boilerplate. But, they are not complex and they just list method signatures that each version's types already implement functionality wise and they get us rid of a lot of dispatch sites. The implementation turns out cleaner as there is less parameter threading throughout call chains, etc. Overall, it seems more scalable.
On the other hand, there are indeed more abstractions and type indirections to understand in Eliot's code. I.e. there are advantages and disadvantages to both approaches.
That said, please engage with Eliot's proposal, it is not as off as your reply implies and dismissing it right away is not what I'd like to see in this situation.
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 11 lines · Danilo Krummrich
Quoted 2 lines · Joel Fernandes
So you're making the code much much worse than before actually. We don't new traits and types pointlessly.
I had a look at both approaches and yes, the traits can be considered boilerplate. But, they are not complex and they just list method signatures that each version's types already implement functionality wise and they get us rid of a lot of dispatch sites. The implementation turns out cleaner as there is less parameter threading throughout call chains, etc. Overall, it seems more scalable.
Quoted 7 lines
On the other hand, there are indeed more abstractions and type indirections to understand in Eliot's code. I.e. there are advantages and disadvantages to both approaches.
That said, please engage with Eliot's proposal, it is not as off as your reply implies and dismissing it right away is not what I'd like to see in this situation.
No one is “dismissing it right away”. If you read all the threads, you will see that I already came up with the same approach and have spent time coding it up myself to learn its potential advantages and drawbacks. :-) I would never dismiss anyone’s suggestion just for the sake of it, without evaluating. Hence most of my morning went in collecting data with both approaches (more on that in another upcoming reply).
Also yes of course we have to look at all the pros and cons and carefully make a choice collectively, there are tradeoffs in both approaches.
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesQuoted 4 lines · Joel Fernandes
Quoted 3 lines · Eliot Courtney
Quoted 2 lines · Joel Fernandes
Quoted 1 line · Eliot Courtney
On Tue Apr 7, 2026 at 6:55 AM JST, Joel Fernandes wrote:
...>> [1]: https://github.com/Edgeworth/linux/commits/review/nova-mm-v10/
Quoted 4 lines
First, thanks for the effort. I looked through this, its pretty much what I had before when I used traits. I don't think it is better to be honest. In fact your version is worse, it adds many new types and things like the following which I did not need before.
Hi Joel and all,
I also looked through Eliot's above attempt carefully, and actually liked it a lot (sorry! haha):
* It cleans up the code. The initial working version was readable, but also had lots of noise on the screen: match statements and pairs of v2/v3 statements.
And interestingly, the mmu_version was, in effect, sporadically implementing a Trait-based approach. But because it is custom, readers don't benefit as much as they would with Traits, which tell you immediately how things are structured.
Joel, I am passionately in agreement with your principles: code must be readable on the screen.
In this case, though, Traits make considerably more readable, especially if one makes the very reasonable assumption that readers are thoroughly accustomed to dealing with Rust traits.
Quoted 3 lines
To put it mildly, the following suggestion should not be anywhere near my code:
lol I understand, believe me. But this is short and not too bad, really.
Quoted 1 line
/// Type-erased MMU-specific [
Vmm] implementations.
Type erasure remains a semi-exotic thing, IMHO. As such, another sentence to elaborate on this would be a nice touch.
Quoted 10 lines
enum VmmInner { /// `Vmm` implementation for MMU v2. V2(VmmImpl<MmuV2>), /// `Vmm` implementation for MMU v3. V3(VmmImpl<MmuV3>), }/// MMU-specific [
Vmm] implementation.
struct VmmImpl<M: Mmu> {
Quoted 6 lines
Seriously, I have to pass on this. :-)
And, you unfortunately seem to have ignored my point about requiring 4 NEW traits (Mmu, PteOps, PdeOps, DualPdeOps etc), which I did not need before. So you're making the code much much worse than before actually. We don't new traits and types pointlessly.
They are not pointless.
However! What I think would be nice is: do a new v11 with approximately this approach, and then we can beat it into being as readable as possible.
thanks,
Signature · 1 line
John Hubbard
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 62 lines · John Hubbard
Quoted 4 lines · Joel Fernandes
Quoted 4 lines · Eliot Courtney
Quoted 2 lines · Joel Fernandes
Quoted 1 line · Eliot Courtney
On Tue Apr 7, 2026 at 6:55 AM JST, Joel Fernandes wrote:
...>> [1]: https://github.com/Edgeworth/linux/commits/review/nova-mm-v10/
Quoted 4 lines
First, thanks for the effort. I looked through this, its pretty much what I had before when I used traits. I don't think it is better to be honest. In fact your version is worse, it adds many new types and things like the following which I did not need before.
Hi Joel and all,
I also looked through Eliot's above attempt carefully, and actually liked it a lot (sorry! haha):
* It cleans up the code. The initial working version was readable, but also had lots of noise on the screen: match statements and pairs of v2/v3 statements.
And interestingly, the
mmu_versionwas, in effect, sporadically implementing a Trait-based approach. But because it is custom, readers don't benefit as much as they would with Traits, which tell you immediately how things are structured.Joel, I am passionately in agreement with your principles: code must be readable on the screen.
In this case, though, Traits make considerably more readable, especially if one makes the very reasonable assumption that readers are thoroughly accustomed to dealing with Rust traits.
Quoted 3 lines
To put it mildly, the following suggestion should not be anywhere near my code:
lol I understand, believe me. But this is short and not too bad, really.
Quoted 1 line
/// Type-erased MMU-specific [
Vmm] implementations.Type erasure remains a semi-exotic thing, IMHO. As such, another sentence to elaborate on this would be a nice touch.
Quoted 10 lines
enum VmmInner { /// `Vmm` implementation for MMU v2. V2(VmmImpl<MmuV2>), /// `Vmm` implementation for MMU v3. V3(VmmImpl<MmuV3>), }/// MMU-specific [
Vmm] implementation.
struct VmmImpl<M: Mmu> {Quoted 6 lines
Seriously, I have to pass on this. :-)
And, you unfortunately seem to have ignored my point about requiring 4 NEW traits (Mmu, PteOps, PdeOps, DualPdeOps etc), which I did not need before. So you're making the code much much worse than before actually. We don't new traits and types pointlessly.
Quoted 5 lines
They are not pointless.
However! What I think would be nice is: do a new v11 with approximately this approach, and then we can beat it into being as readable as possible.
Since it is 3 against 1 here, I rest my case :-). I am still in disagreement since I do not see much benefit (that is why I said pointless above). Actually it is not even about readability, that is subjective (and I haven’t heard most people say parametrizing code for the sake of it makes it more readable anyway). It is that the code gen is worse, and the complexity is just moved to a higher level in the code, not removed. So what are we getting out of this really, other than more boiler plate in higher layers of the code that did not exist before? Not performance, not better generated code. Really nothing. See all the data points in my previous reply.
Note that if the mmu version threading bothers everyone so much, we can also pass down chipset instead and let the walker deal with determining versioning. Would that be better?
But otherwise and since you guys asked, here comes a parameterized v11... ;-).
(Coming next week since this week I’m working on IRQ handling).
thanks,
Signature · 1 line
Joel Fernandes
Danilo Krummrich
maintainerRe: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 1 line · Joel Fernandes
Since it is 3 against 1 here, I rest my case :-).
That's not how I'd view it. :)
Anyways, in case I'm included in "3", that's not my position. My point was to ensure we keep discussing advantages and disadvantages on their merits, as I think you both have good points.
Quoted 2 lines
I am still in disagreement since I do not see much benefit (that is why I said pointless above).
That is fair -- in this case please explain why the advantages pointed out by others are not worth it, propose something that picks up the best of both worlds, etc.
You can also turn it around and ask people whether they can tweak their counter proposal to get rid of specific parts you dislike for a reason.
IOW, keep the ball rolling, so we can come up with the best possible solution.
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Hi Danilo,
Quoted 8 lines · Danilo Krummrich
Quoted 1 line · Joel Fernandes
Since it is 3 against 1 here, I rest my case :-).
That's not how I'd view it. :)
Anyways, in case I'm included in "3", that's not my position. My point was to ensure we keep discussing advantages and disadvantages on their merits, as I think you both have good points.
Heh, yes I actually *did not* include you in the 3 since you sounded to be open to both. ;-)
Quoted 4 lines
Quoted 3 lines
I am still in disagreement since I do not see much benefit (that is why I said pointless above).
Quoted 8 lines
That is fair -- in this case please explain why the advantages pointed out by others are not worth it, propose something that picks up the best of both worlds, etc.
You can also turn it around and ask people whether they can tweak their counter proposal to get rid of specific parts you dislike for a reason.
IOW, keep the ball rolling, so we can come up with the best possible solution.
Good advice, thanks! I will try to come up with something that is acceptable to everyone and we can further debate pros/cons on v11.
There are some merits on the alternative proposal from Eliot/Alex that I'd like to explore while seeing if I can keep some of the merits in mine as well.
thanks,
Signature · 1 line
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 26 lines · Joel Fernandes
Hi Danilo,
Quoted 8 lines · Danilo Krummrich
Quoted 1 line · Joel Fernandes
Since it is 3 against 1 here, I rest my case :-).
That's not how I'd view it. :)
Anyways, in case I'm included in "3", that's not my position. My point was to ensure we keep discussing advantages and disadvantages on their merits, as I think you both have good points.
Heh, yes I actually *did not* include you in the 3 since you sounded to be open to both. ;-)
Quoted 11 lines
Quoted 3 lines
I am still in disagreement since I do not see much benefit (that is why I said pointless above).
That is fair -- in this case please explain why the advantages pointed out by others are not worth it, propose something that picks up the best of both worlds, etc.
You can also turn it around and ask people whether they can tweak their counter proposal to get rid of specific parts you dislike for a reason.
Quoted 7 lines
Quoted 1 line
IOW, keep the ball rolling, so we can come up with the best possible solution.
Good advice, thanks! I will try to come up with something that is acceptable to everyone and we can further debate pros/cons on v11.
There are some merits on the alternative proposal from Eliot/Alex that I'd like to explore while seeing if I can keep some of the merits in mine as well.
I think I found a nice approach. IMO the MMU version dispatch does not belong in Vmm/BarUser layers. Those are version-independent code. However I agree that doing version dispatch at every low-level page table operation is a bit heavy on matches (if we put the MMIO overhead counter-argument aside).
So how about the following approach?
PtWalk, PtMap and everything below it are monomorphized. Vmm and BarUser are not. Version dispatch is handled on PtWalk and PtMap entry points.
I think it makes it cleaner and splits the code up better too and the organizations makes sense because the version differences are related to page tables, not to generic concepts like Vmm and Bar.
Thoughts? Here is a preview:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/commit/?h=pt-traits-v2&id=ff22ba64f729f9f73258777231763a7b9804123b
thanks,
--
Joel Fernandes
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
Quoted 41 lines · Joel Fernandes
Quoted 26 lines · Joel Fernandes
Hi Danilo,
Quoted 8 lines · Danilo Krummrich
Quoted 1 line · Joel Fernandes
Since it is 3 against 1 here, I rest my case :-).
That's not how I'd view it. :)
Anyways, in case I'm included in "3", that's not my position. My point was to ensure we keep discussing advantages and disadvantages on their merits, as I think you both have good points.
Heh, yes I actually *did not* include you in the 3 since you sounded to be open to both. ;-)
Quoted 11 lines
Quoted 3 lines
I am still in disagreement since I do not see much benefit (that is why I said pointless above).
That is fair -- in this case please explain why the advantages pointed out by others are not worth it, propose something that picks up the best of both worlds, etc.
You can also turn it around and ask people whether they can tweak their counter proposal to get rid of specific parts you dislike for a reason.
Quoted 7 lines
Quoted 1 line
IOW, keep the ball rolling, so we can come up with the best possible solution.
Good advice, thanks! I will try to come up with something that is acceptable to everyone and we can further debate pros/cons on v11.
There are some merits on the alternative proposal from Eliot/Alex that I'd like to explore while seeing if I can keep some of the merits in mine as well.
I think I found a nice approach. IMO the MMU version dispatch does not belong in Vmm/BarUser layers. Those are version-independent code. However I agree that doing version dispatch at every low-level page table operation is a bit heavy on matches (if we put the MMIO overhead counter-argument aside).
Quoted 4 lines
So how about the following approach?
PtWalk, PtMap and everything below it are monomorphized. Vmm and BarUser are not. Version dispatch is handled on PtWalk and PtMap entry points.
Conceptually it sounds pretty good.
Quoted 8 lines
I think it makes it cleaner and splits the code up better too and the organizations makes sense because the version differences are related to page tables, not to generic concepts like Vmm and Bar.
Thoughts? Here is a preview:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/commit/?h=pt-traits-v2&id=ff22ba64f729f9f73258777231763a7b9804123b
Probably impractical to review that here, so let's do the review in a v11 posting, I think.
thanks,
Signature · 1 line
John Hubbard
Re: [PATCH v10 12/21] gpu: nova-core: mm: Add unified page table entry wrapper enums
In reply to Joel FernandesOn 4/9/26 3:33 AM, Joel Fernandes wrote:
...
Quoted 1 line
Since it is 3 against 1 here, I rest my case :-). I am still in
As Danilo points out, we want to foster healthy debate and land on whatever comes out of that. Not just by counting noses. :)
Quoted 9 lines
disagreement since I do not see much benefit (that is why I said pointless above). Actually it is not even about readability, that is subjective (and I haven’t heard most people say parametrizing code for the sake of it makes it more readable anyway). It is that the code gen is worse, and the complexity is just moved to a higher level in the code, not removed. So what are we getting out of this really, other than more boiler plate in higher layers of the code that did not exist before? Not performance, not better generated code. Really nothing. See all the data points in my previous reply.
Alex's latest response[1] does a *much* better job than mine, in explicitly highlighting what we get out of this. (It arrived a bit after your response here.) Please take a very close look at that response and see what you think.
The patterns in these page tables are not a new thing, and Rust has language features to help express them.
[1] https://lore.kernel.org/[email protected]
thanks,
Signature · 1 line
John Hubbard
PATCH v10 13/21 gpu: nova-core: mm: Add page table walker for MMU v2/v3
In reply to Joel FernandesAdd the page table walker implementation that traverses the page table hierarchy for both MMU v2 (5-level) and MMU v3 (6-level) to resolve virtual addresses to physical addresses or find PTE locations.
Currently only v2 has been tested (nova-core currently boots pre-hopper) with some initial preparatory work done for v3.
--- drivers/gpu/nova-core/mm/pagetable.rs | 1 + drivers/gpu/nova-core/mm/pagetable/walk.rs | 218 +++++++++++++++++++++ 2 files changed, 219 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/pagetable/walk.rs
drivers/gpu/nova-core/mm/pagetable.rs +1 -0
@@ -10,6 +10,7 @@pub (crate )mod ver2;pub (crate )mod ver3; +pub (crate )mod walk;use crate ::gpu::Architecture ;use crate ::mm::{
drivers/gpu/nova-core/mm/pagetable/walk.rs new file +218 -0
218 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 14/21 gpu: nova-core: mm: Add Virtual Memory Manager
In reply to Joel FernandesAdd the Virtual Memory Manager (VMM) infrastructure for GPU address space management. Each Vmm instance manages a single address space identified by its Page Directory Base (PDB) address, used for Channel, BAR1 and BAR2 mappings.
Mapping APIs and virtual address range tracking are added in later commits.
--- drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/vmm.rs | 63 +++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 drivers/gpu/nova-core/mm/vmm.rs
drivers/gpu/nova-core/mm.rs +1 -0
@@ -7,6 +7,7 @@pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb; +pub (crate )mod vmm;use kernel::{ devres::Devres ,
drivers/gpu/nova-core/mm/vmm.rs new file +63 -0
63 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 15/21 gpu: nova-core: mm: Add virtual address range tracking to VMM
In reply to Joel FernandesAdd virtual address range tracking to the VMM using a buddy allocator. This enables contiguous virtual address range allocation for mappings.
--- drivers/gpu/nova-core/mm/vmm.rs | 98 +++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 11 deletions(-)
drivers/gpu/nova-core/mm/vmm.rs +87 -11
@@ -7,19 +7,35 @@//! and handles TLB flushing after modifications. use kernel::{ - gpu::buddy::AllocatedBlocks , - prelude::*,// + gpu::buddy::{ +AllocatedBlocks , +GpuBuddy , +GpuBuddyAllocFlag , +GpuBuddyAllocMode , +GpuBuddyParams ,// + }, + prelude::*, + ptr::Alignment , + sizes::SZ_4K ,// }; -use crate ::mm::{ - pagetable::{ - walk::{PtWalk ,WalkResult }, -MmuVersion ,// +use core::ops::Range ; + +use crate ::{ + mm::{ + pagetable::{ + walk::{PtWalk ,WalkResult }, +MmuVersion ,// + }, +GpuMm , +Pfn , +Vfn , +VramAddress , +PAGE_SIZE ,// + }, + num::{ +IntoSafeCast ,// }, -GpuMm , -Pfn , -Vfn , -VramAddress ,// };/// Virtual Memory Manager for a GPU address space.
@@ -34,23 +50,83 @@ pub(crate) struct Vmm { pub(crate) mmu_version: MmuVersion,/// Page table allocations required for mappings. page_table_allocs: KVec<Pin<KBox<AllocatedBlocks>>>, +/// Buddy allocator for virtual address range tracking. + virt_buddy: GpuBuddy, }impl Vmm {/// Create a new [`Vmm`] for the given Page Directory Base address. -pub (crate )fn new (pdb_addr:VramAddress , mmu_version:MmuVersion ) ->Result <Self > { +/// +/// The [`Vmm`] will manage a virtual address space of `va_size` bytes. +pub (crate )fn new ( + pdb_addr:VramAddress , + mmu_version:MmuVersion , + va_size:u64 , + ) ->Result <Self > {// Only MMU v2 is supported for now. if mmu_version !=MmuVersion ::V2 {return Err (ENOTSUPP ); } +let virt_buddy =GpuBuddy ::new (GpuBuddyParams { + base_offset:0 , + size: va_size, + chunk_size:Alignment ::new ::<SZ_4K >(), + })?; +Ok (Self { pdb_addr, mmu_version, page_table_allocs:KVec ::new (), + virt_buddy, }) } +/// Allocate a contiguous virtual frame number range. +/// +/// # Arguments +/// +/// - `num_pages`: Number of pages to allocate. +/// - `va_range`: `None` = allocate anywhere, `Some(range)` = constrain allocation to the given +/// range. +pub (crate )fn alloc_vfn_range ( + &self, + num_pages:usize , + va_range:Option <Range <u64 >>, + ) ->Result <(Vfn ,Pin <KBox <AllocatedBlocks >>)> { +let size:u64 = (num_pagesas u64 ) + .checked_mul (PAGE_SIZE as u64 ) + .ok_or (EOVERFLOW )?; + +let mode =match va_range { +Some (r) => { +let range_size = r.end.checked_sub (r.start).ok_or (EOVERFLOW )?; +if range_size != size { +return Err (EINVAL ); + } +GpuBuddyAllocMode ::Range (r) + } +None =>GpuBuddyAllocMode ::Simple , + }; + +let alloc =KBox ::pin_init ( + self.virt_buddy.alloc_blocks ( + mode, + size, +Alignment ::new ::<SZ_4K >(), +GpuBuddyAllocFlag ::Contiguous , + ), +GFP_KERNEL , + )?; + +// Get the starting offset of the first block (only block as range is contiguous). +let offset = alloc.iter ().next ().ok_or (ENOMEM )?.offset (); +let page_size:u64 =PAGE_SIZE .into_safe_cast (); +let vfn =Vfn ::new (offset / page_size); + +Ok ((vfn, alloc)) + } +/// Read the [`Pfn`] for a mapped [`Vfn`] if one is mapped. pub (crate )fn read_mapping (&self, mm: &GpuMm , vfn:Vfn ) ->Result <Option <Pfn >> {let walker =PtWalk ::new (self.pdb_addr, self.mmu_version);
Signature · 1 line
2.34.1
PATCH v10 16/21 gpu: nova-core: mm: Add multi-page mapping API to VMM
In reply to Joel FernandesAdd the page table mapping and unmapping API to the Virtual Memory Manager, implementing a two-phase prepare/execute model suitable for use both inside and outside the DMA fence signalling critical path.
--- drivers/gpu/nova-core/mm/vmm.rs | 366 +++++++++++++++++++++++++++++++- 1 file changed, 363 insertions(+), 3 deletions(-)
drivers/gpu/nova-core/mm/vmm.rs +363 -3
@@ -11,21 +11,34 @@ AllocatedBlocks, GpuBuddy, GpuBuddyAllocFlag, + GpuBuddyAllocFlags, GpuBuddyAllocMode, GpuBuddyParams, // }, prelude::*, ptr::Alignment, + rbtree::{RBTree, RBTreeNode}, sizes::SZ_4K, // }; -use core::ops::Range ; +use core::{ + cell::Cell , + ops::Range ,// +};use crate ::{ mm::{ pagetable::{ - walk::{PtWalk ,WalkResult }, -MmuVersion ,// + walk::{ +PtWalk , +WalkPdeResult , +WalkResult ,// + }, +DualPde , +MmuVersion , +PageTableLevel , +Pde , +Pte ,// },GpuMm ,Pfn ,
@@ -52,6 +65,74 @@ pub(crate) struct Vmm { page_table_allocs: KVec<Pin<KBox<AllocatedBlocks>>>,/// Buddy allocator for virtual address range tracking. virt_buddy: GpuBuddy, +/// Prepared PT pages pending PDE installation, keyed by `install_addr`. +/// +/// Populated by `Vmm` mapping prepare phase and drained in the execute phase. +/// Shared by all pending maps in the `Vmm`, thus preventing races where 2 +/// maps might be trying to install the same page table/directory entry pointer. + pt_pages: RBTree<VramAddress, PreparedPtPage>, +} + +/// A pre-allocated and zeroed page table page. +/// +/// Created during the mapping prepare phase and consumed during the mapping execute phase. +/// Stored in an [`RBTree`] keyed by the PDE slot address (`install_addr`). +struct PreparedPtPage { +/// The allocated and zeroed page table page. + alloc:Pin <KBox <AllocatedBlocks >>, +/// Page table level -- needed to determine if this PT page is for a dual PDE. + level:PageTableLevel , +} + +/// Multi-page prepared mapping -- VA range allocated, ready for execute. +/// +/// Produced by [`Vmm::prepare_map()`], consumed by [`Vmm::execute_map()`]. +/// The struct owns the VA space allocation between prepare and execute phases. +pub (crate )struct PreparedMapping { + vfn_start:Vfn , + num_pages:usize , + vfn_alloc:Pin <KBox <AllocatedBlocks >>, +} + +/// Result of a mapping operation -- tracks the active mapped range. +/// +/// Returned by [`Vmm::execute_map()`] and [`Vmm::map_pages()`]. +/// Owns the VA allocation; the VA range is freed when this is dropped. +/// Callers must call [`Vmm::unmap_pages()`] before dropping to invalidate +/// PTEs (dropping only frees the VA range, not the PTE entries). +pub (crate )struct MappedRange { +pub (crate ) vfn_start:Vfn , +pub (crate ) num_pages:usize , +/// VA allocation -- freed when [`MappedRange`] is dropped. + _vfn_alloc:Pin <KBox <AllocatedBlocks >>, +/// Logs a warning if dropped without unmapping. + _drop_guard:MustUnmapGuard , +} + +/// Guard that logs a warning once if a [`MappedRange`] is dropped without +/// calling [`Vmm::unmap_pages()`]. +struct MustUnmapGuard { + armed:Cell <bool >, +} + +impl MustUnmapGuard { +const fn new () ->Self { +Self { + armed:Cell ::new (true ), + } + } + +fn disarm (&self) { + self.armed.set (false ); + } +} + +impl Drop for MustUnmapGuard { +fn drop (&mut self) { +if self.armed.get () { + kernel::pr_warn!("MappedRange dropped without calling unmap_pages()\n" ); + } + } }impl Vmm {
@@ -79,6 +160,7 @@ pub(crate) fn new( mmu_version, page_table_allocs: KVec::new(), virt_buddy, + pt_pages: RBTree::new(), }) }
@@ -136,4 +218,282 @@ pub(crate) fn read_mapping(&self, mm: &GpuMm, vfn: Vfn) -> Result<Option<Pfn>> { WalkResult::Unmapped { .. } | WalkResult::PageTableMissing => Ok(None), } } + +/// Allocate and zero a physical page table page for a specific PDE slot. +/// Called during the map prepare phase. +fn alloc_and_zero_page_table ( + &mut self, + mm: &GpuMm , + level:PageTableLevel , + ) ->Result <PreparedPtPage > { +let blocks =KBox ::pin_init ( + mm.buddy ().alloc_blocks ( +GpuBuddyAllocMode ::Simple , +SZ_4K .into_safe_cast (), +Alignment ::new ::<SZ_4K >(), +GpuBuddyAllocFlags ::default (), + ), +GFP_KERNEL , + )?; + +// Get page's VRAM address from the allocation. +let page_vram =VramAddress ::new (blocks.iter ().next ().ok_or (ENOMEM )?.offset ()); + +// Zero via PRAMIN. +let mut window = mm.pramin ().get_window ()?; +let base = page_vram.raw (); +for offin (0 ..PAGE_SIZE ).step_by (8 ) { + window.try_write64 (base + off,0 )?; + } + +Ok (PreparedPtPage { + alloc: blocks, + level, + }) + } + +/// Ensure all intermediate page table pages are prepared for a [`Vfn`]. Just +/// finds out which PDE pages are missing, allocates pages for them, and defers +/// installation to the execute phase. +/// +/// PRAMIN is released before each allocation and re-acquired after. Memory +/// allocations are done outside of holding this lock to prevent deadlocks with +/// the fence signalling critical path. +fn ensure_pte_path (&mut self, mm: &GpuMm , vfn:Vfn ) ->Result { +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let max_iter =2 * self.mmu_version.pde_level_count (); + +// Keep looping until all PDE levels are resolved. +for _in 0 ..max_iter { +let mut window = mm.pramin ().get_window ()?; + +// Walk PDE levels. The closure checks self.pt_pages for prepared-but-uninstalled +// pages, letting the walker continue through them as if they were installed in HW. +// The walker keeps calling the closure to get these "prepared but not installed" pages. +let result = walker.walk_pde_levels (&mut window, vfn, |install_addr| { + self.pt_pages + .get (&install_addr) + .and_then (|p|Some (VramAddress ::new (p.alloc.iter ().next ()?.offset ()))) + })?; + +match result { +WalkPdeResult ::Complete { .. } => { +// All PDE levels resolved. +return Ok (()); + } +WalkPdeResult ::Missing { + install_addr, + level, + } => { +// Drop PRAMIN before allocation. +drop (window); +let page = self.alloc_and_zero_page_table (mm, level)?; +let node =RBTreeNode ::new (install_addr, page,GFP_KERNEL )?; +let old = self.pt_pages.insert (node); +if old.is_some () { + kernel::pr_warn_once!( +"VMM: duplicate install_addr in pt_pages (internal consistency error)\n" + ); +return Err (EIO ); + } + +// Loop: re-acquire PRAMIN and re-walk from root. + } + } + } + + kernel::pr_warn!( +"VMM: ensure_pte_path: loop exhausted after {} iters (VFN {:?})\n" , + max_iter, + vfn + ); +Err (EIO ) + } + +/// Prepare resources for mapping `num_pages` pages. +/// +/// Allocates a contiguous VA range, then walks the hierarchy per-VFN to prepare pages +/// for all missing PDEs. Returns a [`PreparedMapping`] with the VA allocation. +/// +/// If `va_range` is not `None`, the VA range is constrained to the given range. Safe +/// to call outside the fence signalling critical path. +pub (crate )fn prepare_map ( + &mut self, + mm: &GpuMm , + num_pages:usize , + va_range:Option <Range <u64 >>, + ) ->Result <PreparedMapping > { +if num_pages ==0 { +return Err (EINVAL ); + } + +// Pre-reserve so execute_map() can use push_within_capacity (no alloc in +// fence signalling critical path). +// Upper bound on page table pages needed for the full tree (PTE pages + PDE +// pages at all levels). +let pt_upper_bound = self.mmu_version.pt_pages_upper_bound (num_pages); + self.page_table_allocs.reserve (pt_upper_bound,GFP_KERNEL )?; + +// Allocate contiguous VA range. +let (vfn_start, vfn_alloc) = self.alloc_vfn_range (num_pages, va_range)?; + +// Walk the hierarchy per-VFN to prepare pages for all missing PDEs. +for iin 0 ..num_pages { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (vfn_start.raw () + i_u64); + self.ensure_pte_path (mm, vfn)?; + } + +Ok (PreparedMapping { + vfn_start, + num_pages, + vfn_alloc, + }) + } + +/// Execute a prepared multi-page mapping. +/// +/// Drain prepared PT pages and install PDEs followed by single TLB flush. +pub (crate )fn execute_map ( + &mut self, + mm: &GpuMm , + prepared:PreparedMapping , + pfns: &[Pfn ], + writable:bool , + ) ->Result <MappedRange > { +if pfns.len () != prepared.num_pages { +return Err (EINVAL ); + } + +let PreparedMapping { + vfn_start, + num_pages, + vfn_alloc, + } = prepared; + +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let mut window = mm.pramin ().get_window ()?; + +// First, drain self.pt_pages, install all pending PDEs. +let mut cursor = self.pt_pages.cursor_front_mut (); +while let Some (c) = cursor { +let (next, node) = c.remove_current (); +let (install_addr, page) = node.to_key_value (); +let page_vram =VramAddress ::new (page.alloc.iter ().next ().ok_or (ENOMEM )?.offset ()); + +if page.level == self.mmu_version.dual_pde_level () { +let new_dpde =DualPde ::new_small (self.mmu_version,Pfn ::from (page_vram)); + new_dpde.write (&mut window, install_addr)?; + }else { +let new_pde =Pde ::new_vram (self.mmu_version,Pfn ::from (page_vram)); + new_pde.write (&mut window, install_addr)?; + } + +// Track the allocated pages in the `Vmm`. + self.page_table_allocs + .push_within_capacity (page.alloc) + .map_err (|_|ENOMEM )?; + + cursor = next; + } + +// Next, write PTEs (all PDEs now installed in HW). +for (i, &pfn)in pfns.iter ().enumerate () { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (vfn_start.raw () + i_u64); +let result = walker.walk_to_pte_lookup_with_window (&mut window, vfn)?; + +match result { +WalkResult ::Unmapped { pte_addr } |WalkResult ::Mapped { pte_addr, .. } => { +let pte =Pte ::new_vram (self.mmu_version, pfn, writable); + pte.write (&mut window, pte_addr)?; + } +WalkResult ::PageTableMissing => { + kernel::pr_warn_once!("VMM: page table missing for VFN {vfn:?}\n" ); +return Err (EIO ); + } + } + } + +drop (window); + +// Finally, flush the TLB. + mm.tlb ().flush (self.pdb_addr)?; + +Ok (MappedRange { + vfn_start, + num_pages, + _vfn_alloc: vfn_alloc, + _drop_guard:MustUnmapGuard ::new (), + }) + } + +/// Map pages doing prepare and execute in the same call. +/// +/// This is a convenience wrapper for callers outside the fence signalling critical +/// path (e.g., BAR mappings). For DRM usecases, [`Vmm::prepare_map()`] and +/// [`Vmm::execute_map()`] will be called separately. +pub (crate )fn map_pages ( + &mut self, + mm: &GpuMm , + pfns: &[Pfn ], + va_range:Option <Range <u64 >>, + writable:bool , + ) ->Result <MappedRange > { +if pfns.is_empty () { +return Err (EINVAL ); + } + +// Check if provided VA range is sufficient (if provided). +if let Some (ref range) = va_range { +let required:u64 = pfns + .len () + .checked_mul (PAGE_SIZE ) + .ok_or (EOVERFLOW )? + .into_safe_cast (); +let available = range.end.checked_sub (range.start).ok_or (EINVAL )?; +if available < required { +return Err (EINVAL ); + } + } + +let prepared = self.prepare_map (mm, pfns.len (), va_range)?; + self.execute_map (mm, prepared, pfns, writable) + } + +/// Unmap all pages in a [`MappedRange`] with a single TLB flush. +/// +/// Takes the range by value (consumes it), then invalidates PTEs for the range, +/// flushes the TLB, then drops the range (freeing the VA). PRAMIN lock is held. +pub (crate )fn unmap_pages (&mut self, mm: &GpuMm , range:MappedRange ) ->Result { +let walker =PtWalk ::new (self.pdb_addr, self.mmu_version); +let invalid_pte =Pte ::invalid (self.mmu_version); + +let mut window = mm.pramin ().get_window ()?; +for iin 0 ..range.num_pages { +let i_u64:u64 = i.into_safe_cast (); +let vfn =Vfn ::new (range.vfn_start.raw () + i_u64); +let result = walker.walk_to_pte_lookup_with_window (&mut window, vfn)?; + +match result { +WalkResult ::Mapped { pte_addr, .. } |WalkResult ::Unmapped { pte_addr } => { + invalid_pte.write (&mut window, pte_addr)?; + } +WalkResult ::PageTableMissing => { +continue ; + } + } + } +drop (window); + + mm.tlb ().flush (self.pdb_addr)?; + +// TODO: Internal page table pages (PDE, PTE pages) are still kept around. +// This is by design as repeated maps/unmaps will be fast. As a future TODO, +// we can add a reclaimer here to reclaim if VRAM is short. For now, the PT +// pages are dropped once the `Vmm` is dropped. + + range._drop_guard.disarm ();// Unmap complete, Ok to drop MappedRange. +Ok (()) + } }
Signature · 1 line
2.34.1
PATCH v10 17/21 gpu: nova-core: Add BAR1 aperture type and size constant
In reply to Joel FernandesAdd BAR1_SIZE constant and Bar1 type alias for the 256MB BAR1 aperture. These are prerequisites for BAR1 memory access functionality.
--- drivers/gpu/nova-core/driver.rs | 8 +++++++- drivers/gpu/nova-core/gsp/commands.rs | 4 ++++ drivers/gpu/nova-core/gsp/fw/commands.rs | 8 ++++++++ 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/driver.rs +7 -1
@@ -13,7 +13,10 @@ Vendor, // }, prelude::*, - sizes::SZ_16M, + sizes::{ + SZ_16M, + SZ_256M, // + }, sync::{ atomic::{Atomic ,
@@ -37,6 +40,7 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; +pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. //
@@ -47,6 +51,8 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; +#[expect(dead_code)] +pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; kernel::pci_device_table!(PCI_TABLE ,
drivers/gpu/nova-core/gsp/commands.rs +4 -0
@@ -193,6 +193,9 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {/// The reply from the GSP to the [`GetGspStaticInfo`] command. pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ], +/// BAR1 Page Directory Entry base address. +#[expect(dead_code)] +pub (crate ) bar1_pde_base:u64 ,/// Usable FB (VRAM) region for driver memory allocation. pub (crate ) usable_fb_region:Range <u64 >,/// End of VRAM.
@@ -214,6 +217,7 @@ fn read(Ok (GetGspStaticInfoReply { gpu_name: msg.gpu_name_str (), + bar1_pde_base: msg.bar1_pde_base (), usable_fb_region: base..base.saturating_add (size), total_fb_end, })
drivers/gpu/nova-core/gsp/fw/commands.rs +8 -0
@@ -125,6 +125,14 @@ impl GspStaticConfigInfo { self.0 .gpuNameString } +/// Returns the BAR1 Page Directory Entry base address. +/// +/// This is the root page table address for BAR1 virtual memory, +/// set up by GSP-RM firmware. +pub (crate )fn bar1_pde_base (&self) ->u64 { + self.0 .bar1PdeBase + } +/// Returns an iterator over valid FB regions from GSP firmware data. fn fb_regions ( &self,
Signature · 1 line
2.34.1
PATCH v10 18/21 gpu: nova-core: mm: Add BAR1 user interface
In reply to Joel FernandesAdd the BAR1 user interface for CPU access to GPU virtual memory through the BAR1 aperture.
--- drivers/gpu/nova-core/driver.rs | 1 - drivers/gpu/nova-core/gpu.rs | 21 +++- drivers/gpu/nova-core/gsp/commands.rs | 1 - drivers/gpu/nova-core/mm.rs | 1 + drivers/gpu/nova-core/mm/bar_user.rs | 152 ++++++++++++++++++++++++++ 5 files changed, 173 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/nova-core/mm/bar_user.rs
drivers/gpu/nova-core/driver.rs +0 -1
@@ -51,7 +51,6 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -#[expect(dead_code)] pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; kernel::pci_device_table!(
drivers/gpu/nova-core/gpu.rs +20 -1
@@ -28,7 +28,12 @@ commands::GetGspStaticInfoReply, Gsp, // }, - mm::GpuMm, + mm::{ + bar_user::BarUser, + pagetable::MmuVersion, + GpuMm, + VramAddress, // + }, regs, };
@@ -122,6 +127,11 @@ pub(crate) const fn arch(self) -> Architecture {pub (crate )const fn needs_fwsec_bootloader (self) ->bool {matches! (self.arch (),Architecture ::Turing ) ||matches! (self,Self ::GA100 ) } + +/// Returns the MMU version for this chipset. +pub (crate )fn mmu_version (self) ->MmuVersion { +MmuVersion ::from (self.arch ()) + } }// TODO
@@ -250,6 +260,8 @@ pub(crate) struct Gpu { gsp: Gsp,/// Static GPU information from GSP. gsp_static_info: GetGspStaticInfoReply, +/// BAR1 user interface for CPU access to GPU virtual memory. + bar_user: BarUser, }impl Gpu {
@@ -308,6 +320,13 @@ pub(crate) fn new<'a>( }, pramin_vram_region)? }, +// Create BAR1 user interface for CPU access to GPU virtual memory. + bar_user: { + let pdb_addr = VramAddress::new(gsp_static_info.bar1_pde_base); +let bar1_size = pdev.resource_len (1 )?; +BarUser ::new (pdb_addr, spec.chipset, bar1_size)? + }, + bar: devres_bar, }) }
drivers/gpu/nova-core/gsp/commands.rs +0 -1
@@ -194,7 +194,6 @@ fn init(&self) -> impl Init<Self::Command, Self::InitError> {pub (crate )struct GetGspStaticInfoReply { gpu_name: [u8 ;64 ],/// BAR1 Page Directory Entry base address. -#[expect(dead_code)] pub (crate ) bar1_pde_base:u64 ,/// Usable FB (VRAM) region for driver memory allocation. pub (crate ) usable_fb_region:Range <u64 >,
drivers/gpu/nova-core/mm.rs +1 -0
@@ -4,6 +4,7 @@#![expect(dead_code)] +pub (crate )mod bar_user;pub (crate )mod pagetable;pub (crate )mod pramin;pub (crate )mod tlb;
drivers/gpu/nova-core/mm/bar_user.rs new file +152 -0
152 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 19/21 gpu: nova-core: mm: Add BAR1 memory management self-tests
In reply to Joel FernandesAdd self-tests for BAR1 access during driver probe when CONFIG_NOVA_MM_SELFTESTS is enabled (default disabled). This results in testing the Vmm, GPU buddy allocator and BAR1 region all of which should function correctly for the tests to pass.
--- drivers/gpu/nova-core/Kconfig | 10 ++ drivers/gpu/nova-core/driver.rs | 2 + drivers/gpu/nova-core/gpu.rs | 36 ++++ drivers/gpu/nova-core/mm/bar_user.rs | 236 +++++++++++++++++++++++++++ 4 files changed, 284 insertions(+)
drivers/gpu/nova-core/Kconfig +10 -0
@@ -15,3 +15,13 @@ config NOVA_CORE This driver is work in progress and may not be functional. If M is selected, the module will be called nova_core. + +config NOVA_MM_SELFTESTS + bool "Memory management self-tests" + depends on NOVA_CORE + help + Enable self-tests for the memory management subsystem. When enabled, + tests are run during GPU probe to verify PRAMIN aperture access, + page table walking, and BAR1 virtual memory mapping functionality. + + This is a testing option and is default-disabled.
drivers/gpu/nova-core/driver.rs +2 -0
@@ -101,6 +101,8 @@ fn probe(pdev: &pci::Device<Core>, _info: &Self::IdInfo) -> impl PinInit<Self, E Ok(try_pin_init!(Self { gpu <- Gpu::new(pdev, bar.clone(), bar.access(pdev.as_ref())?), + // Run optional GPU selftests. + _: { gpu.run_selftests(pdev)? }, _reg <- auxiliary::Registration::new( pdev.as_ref(), c"nova-drm",
drivers/gpu/nova-core/gpu.rs +36 -0
@@ -341,4 +341,40 @@ pub(crate) fn unbind(&self, dev: &device::Device<device::Core>) { .inspect(|bar| self.sysmem_flush.unregister(bar)) .is_err()); } + +/// Run selftests on the constructed [`Gpu`]. +pub (crate )fn run_selftests ( +mut self:Pin <&mut Self >, + pdev: &pci::Device <device::Bound >, + ) ->Result { + self.as_mut ().run_mm_selftests (pdev)?; +Ok (()) + } + +#[cfg( CONFIG_NOVA_MM_SELFTESTS )] +fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { +use crate ::driver::BAR1_SIZE ; + +// BAR1 self-tests. +let bar1 =Arc ::pin_init ( + pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), +GFP_KERNEL , + )?; +let bar1_access = bar1.access (pdev.as_ref ())?; + +crate ::mm::bar_user::run_self_test ( + pdev.as_ref (), + &self.mm, + bar1_access, + self.gsp_static_info.bar1_pde_base, + self.spec.chipset, + )?; + +Ok (()) + } + +#[cfg(not( CONFIG_NOVA_MM_SELFTESTS ))] +fn run_mm_selftests (self:Pin <&mut Self >, _pdev: &pci::Device <device::Bound >) ->Result { +Ok (()) + } }
drivers/gpu/nova-core/mm/bar_user.rs +236 -0
239 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 20/21 gpu: nova-core: mm: Add PRAMIN aperture self-tests
In reply to Joel FernandesAdd self-tests for the PRAMIN aperture mechanism to verify correct operation during GPU probe. The tests validate various alignment requirements and corner cases.
The tests are default disabled and behind CONFIG_NOVA_MM_SELFTESTS. When enabled, tests run after GSP boot during probe.
--- drivers/gpu/nova-core/gpu.rs | 3 + drivers/gpu/nova-core/mm/pramin.rs | 209 +++++++++++++++++++++++++++++ 2 files changed, 212 insertions(+)
drivers/gpu/nova-core/gpu.rs +3 -0
@@ -355,6 +355,9 @@ pub(crate) fn run_selftests(fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result {use crate ::driver::BAR1_SIZE ; +// PRAMIN aperture self-tests. +crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?; +// BAR1 self-tests. let bar1 =Arc ::pin_init ( pdev.iomap_region_sized::<BAR1_SIZE>(1, c"nova-core/bar1"),
drivers/gpu/nova-core/mm/pramin.rs +209 -0
218 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v10 21/21 gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
In reply to Joel FernandesFrom: Zhi Wang <[email protected]>
Remove the hardcoded BAR1_SIZE = SZ_256M constant. On GPUs like L40 the BAR1 aperture is larger than 256MB; using a hardcoded size prevents large BAR1 from working and mapping it would fail.
--- drivers/gpu/nova-core/driver.rs | 8 ++------ drivers/gpu/nova-core/gpu.rs | 7 +------ 2 files changed, 3 insertions(+), 12 deletions(-)
drivers/gpu/nova-core/driver.rs +2 -6
@@ -13,10 +13,7 @@ Vendor, // }, prelude::*, - sizes::{ - SZ_16M, - SZ_256M, // - }, + sizes::SZ_16M, sync::{ atomic::{Atomic ,
@@ -40,7 +37,6 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; -pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. //
@@ -51,7 +47,7 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; +pub (crate )type Bar1 = pci::Bar ; kernel::pci_device_table!(PCI_TABLE ,
drivers/gpu/nova-core/gpu.rs +1 -6
@@ -353,16 +353,11 @@ pub(crate) fn run_selftests(#[cfg( CONFIG_NOVA_MM_SELFTESTS )] fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { -use crate ::driver::BAR1_SIZE ; -// PRAMIN aperture self-tests. crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?;// BAR1 self-tests. -let bar1 =Arc ::pin_init ( - pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), -GFP_KERNEL , - )?; +let bar1 =Arc ::pin_init (pdev.iomap_region (1 ,c"nova-core/bar1" ),GFP_KERNEL )?;let bar1_access = bar1.access (pdev.as_ref ())?; crate::mm::bar_user::run_self_test(
Signature · 1 line
2.34.1
Re: [PATCH v10 21/21] gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
Quoted 58 lines · Joel Fernandes
From: Zhi Wang <[email protected]>
Remove the hardcoded
BAR1_SIZE=SZ_256Mconstant. On GPUs like L40 the BAR1 aperture is larger than 256MB; using a hardcoded size prevents large BAR1 from working and mapping it would fail.Signed-off-by Zhi Wang <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/driver.rs | 8 ++------ drivers/gpu/nova-core/gpu.rs | 7 +------ 2 files changed, 3 insertions(+), 12 deletions(-)drivers/gpu/nova-core/driver.rs +2
-6@@ -13,10 +13,7 @@ Vendor, // }, prelude::*, - sizes::{ - SZ_16M, - SZ_256M, // - }, + sizes::SZ_16M, sync::{ atomic::{Atomic ,@@ -40,7 +37,6 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; -pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. // @@ -51,7 +47,7 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; +pub (crate )type Bar1 = pci::Bar ; kernel::pci_device_table!(PCI_TABLE ,drivers/gpu/nova-core/gpu.rs +0
-2@@ -353,16 +353,11 @@ pub(crate) fn run_selftests(#[cfg( CONFIG_NOVA_MM_SELFTESTS )] fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { -use crate ::driver::BAR1_SIZE ; -// PRAMIN aperture self-tests. crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?;
drivers/gpu/nova-core/gpu.rs pub(crate) fn run_selftests( · line 363
Quoted 10 lines
drivers/gpu/nova-core/gpu.rs +1
-4// BAR1 self-tests. -let bar1 =Arc ::pin_init ( - pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), -GFP_KERNEL , - )?; +let bar1 =Arc ::pin_init (pdev.iomap_region (1 ,c"nova-core/bar1" ),GFP_KERNEL )?;let bar1_access = bar1.access (pdev.as_ref ())?; crate::mm::bar_user::run_self_test(
Can we move this directly after patch 17 which adds the fixed bar1? Or alternatively fold it in while preserving Zhi's attribution (I am not sure what the conventional method for this is).
Re: [PATCH v10 21/21] gpu: nova-core: Use runtime BAR1 size instead of hardcoded 256MB
Quoted 68 lines · Eliot Courtney
Quoted 67 lines · Joel Fernandes
From: Zhi Wang <[email protected]>
Remove the hardcoded
BAR1_SIZE=SZ_256Mconstant. On GPUs like L40 the BAR1 aperture is larger than 256MB; using a hardcoded size prevents large BAR1 from working and mapping it would fail.Signed-off-by Zhi Wang <[email protected]>Signed-off-by Joel Fernandes <[email protected]>--- drivers/gpu/nova-core/driver.rs | 8 ++------ drivers/gpu/nova-core/gpu.rs | 7 +------ 2 files changed, 3 insertions(+), 12 deletions(-)drivers/gpu/nova-core/driver.rs +2
-6@@ -13,10 +13,7 @@ Vendor, // }, prelude::*, - sizes::{ - SZ_16M, - SZ_256M, // - }, + sizes::SZ_16M, sync::{ atomic::{Atomic ,@@ -40,7 +37,6 @@ pub(crate) struct NovaCore { }const BAR0_SIZE :usize =SZ_16M ; -pub (crate )const BAR1_SIZE :usize =SZ_256M ;// For now we only support Ampere which can use up to 47-bit DMA addresses. // @@ -51,7 +47,7 @@ pub(crate) struct NovaCore {const GPU_DMA_BITS :u32 =47 ;pub (crate )type Bar0 = pci::Bar <BAR0_SIZE >; -pub (crate )type Bar1 = pci::Bar <BAR1_SIZE >; +pub (crate )type Bar1 = pci::Bar ; kernel::pci_device_table!(PCI_TABLE ,drivers/gpu/nova-core/gpu.rs +1
-6@@ -353,16 +353,11 @@ pub(crate) fn run_selftests(#[cfg( CONFIG_NOVA_MM_SELFTESTS )] fn run_mm_selftests (self:Pin <&mut Self >, pdev: &pci::Device <device::Bound >) ->Result { -use crate ::driver::BAR1_SIZE ; -// PRAMIN aperture self-tests. crate ::mm::pramin::run_self_test (pdev.as_ref (), self.mm.pramin (), self.spec.chipset)?;// BAR1 self-tests. -let bar1 =Arc ::pin_init ( - pdev.iomap_region_sized ::<BAR1_SIZE >(1 ,c"nova-core/bar1" ), -GFP_KERNEL , - )?; +let bar1 =Arc ::pin_init (pdev.iomap_region (1 ,c"nova-core/bar1" ),GFP_KERNEL )?;let bar1_access = bar1.access (pdev.as_ref ())?;
Quoted 5 lines
drivers/gpu/nova-core/gpu.rs pub(crate) fn run_selftests( · line 363
Quoted 1 line
drivers/gpu/nova-core/gpu.rs +0
-0crate::mm::bar_user::run_self_test(Can we move this directly after patch 17 which adds the fixed bar1? Or alternatively fold it in while preserving Zhi's attribution (I am not sure what the conventional method for this is).
Generally, squashing and attribution works. I will do that with attribution. (Zhi did tell me he would be Ok with that as well in this instance).
thanks,
--
Joel Fernandes