gpu: nova-core: add GA100 support
Subsystems (4)
cover gpu: nova-core: add GA100 support
GA100 is an odd GPU. Architecturally, it's an Ampere. However, it uses the same GSP-RM boot process as a Turing. It's VBIOS, for whatever reason, has an IFR header that precedes the PCI ROM header and must be skipped. In addition, the FRTS window size must be set to 0 just because it's a GA100.
So add all the missing code that GA100 needs to be supported. There was some infrastructure already for GA100 (e.g. the FbHal), but it was incomplete.
Timur Tabi (6):
gpu: nova-core: use correct fwsignature for GA100
gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header
signature
gpu: nova-core: only boot FRTS if it actually exists
gpu: nova-core: add FbHal::frts_size() for GA100 support
gpu: nova-core: skip the IFR header if present
gpu: nova-core: enable GA100
drivers/gpu/nova-core/falcon/hal.rs | 3 +- drivers/gpu/nova-core/fb.rs | 6 ++-- drivers/gpu/nova-core/fb/hal.rs | 3 ++ drivers/gpu/nova-core/fb/hal/ga100.rs | 5 ++++ drivers/gpu/nova-core/fb/hal/ga102.rs | 8 +++++- drivers/gpu/nova-core/fb/hal/tu102.rs | 8 +++++- drivers/gpu/nova-core/firmware/gsp.rs | 2 +- drivers/gpu/nova-core/gsp/boot.rs | 5 +++- drivers/gpu/nova-core/vbios.rs | 41 +++++++++++++++++++++++++-- 9 files changed, 70 insertions(+), 11 deletions(-)
base-commit: a7a080bb4236ebe577b6776d940d1717912ff6dd
Signature · 1 line
2.53.0
PATCH 1/6 gpu: nova-core: use correct fwsignature for GA100
Although GA100 uses the same GSP-RM firmware as Turing, it has a different signature specifically for it.
--- drivers/gpu/nova-core/firmware/gsp.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/firmware/gsp.rs +1 -1
@@ -139,7 +139,7 @@ pub(crate) fn new<'a>( } Architecture::Turing => ".fwsignature_tu10x",// GA100 uses the same firmware as Turing - Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_tu10x", + Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_ga100", Architecture::Ampere => ".fwsignature_ga10x", Architecture::Ada => ".fwsignature_ad10x", };
Signature · 1 line
2.53.0
Re: [PATCH 1/6] gpu: nova-core: use correct fwsignature for GA100
Quoted 14 lines · Timur Tabi
Although GA100 uses the same GSP-RM firmware as Turing, it has a different signature specifically for it.
Fixes 121ea04cd9f2 ("gpu: nova-core: add support for Turing/GA100 fwsignature")Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/firmware/gsp.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/nova-core/firmware/gsp.rs +0
-0
drivers/gpu/nova-core/firmware/gsp.rs pub(crate) fn new<'a>( · line 145
Quoted 8 lines
drivers/gpu/nova-core/firmware/gsp.rs +1
-1@@ -139,7 +139,7 @@ pub(crate) fn new<'a>( } Architecture::Turing => ".fwsignature_tu10x",// GA100 uses the same firmware as Turing - Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_tu10x", + Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_ga100", Architecture::Ampere => ".fwsignature_ga10x", Architecture::Ada => ".fwsignature_ad10x", };
PATCH 2/6 gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header signature
In reply to Timur TabiNvidia GPUs have some PCI expansion ROM sections that have an Nvidia- specific signature instead of 0xAA55. Signature 0xBB77 is actually an internal-only value that has been deprecated for over a decade. Nova-core will never encounter a GPU with that signature, so don't look for it.
--- drivers/gpu/nova-core/vbios.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +1 -1
@@ -491,7 +491,7 @@ fn new(dev: &device::Device, data: &[u8]) -> Result<Self> {// Check for valid ROM signatures. match signature { -0xAA55 |0xBB77 |0x4E56 => {} +0xAA55 |0x4E56 => {} _ => {dev_err! (dev,"ROM signature unknown {:#x}\n" , signature);return Err (EINVAL );
Signature · 1 line
2.53.0
Re: [PATCH 2/6] gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header signature
Quoted 16 lines · Timur Tabi
Nvidia GPUs have some PCI expansion ROM sections that have an Nvidia- specific signature instead of
0xAA55. Signature0xBB77is actually an internal-only value that has been deprecated for over a decade. Nova-core will never encounter a GPU with that signature, so don't look for it.Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/vbios.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/nova-core/vbios.rs +0
-0
drivers/gpu/nova-core/vbios.rs fn new(dev: &device::Device, data: &[u8]) -> Result<Self> { · line 497
Quoted 8 lines
drivers/gpu/nova-core/vbios.rs +1
-1@@ -491,7 +491,7 @@ fn new(dev: &device::Device, data: &[u8]) -> Result<Self> {// Check for valid ROM signatures. match signature { -0xAA55 |0xBB77 |0x4E56 => {} +0xAA55 |0x4E56 => {} _ => {dev_err! (dev,"ROM signature unknown {:#x}\n" , signature);return Err (EINVAL );
PATCH 3/6 gpu: nova-core: only boot FRTS if it actually exists
In reply to Timur TabiOn some Nvidia GPUs (i.e. GA100), the FRTS region is not allocated because there is no display hardware. If the region does not exist, then FWSEC-FRTS should not be run.
--- drivers/gpu/nova-core/gsp/boot.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/gsp/boot.rs +4 -1
@@ -155,7 +155,10 @@ pub(crate) fn boot(let fb_layout =FbLayout ::new (chipset, bar, &gsp_fw)?;dev_dbg! (dev,"{:#x?}\n" , fb_layout); -Self ::run_fwsec_frts (dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; +// Only boot FRTS if it actually exists. +if !fb_layout.frts.is_empty () { +Self ::run_fwsec_frts (dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; + }let booter_loader =BooterFirmware ::new ( dev,
Signature · 1 line
2.53.0
Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists
Quoted 15 lines · Timur Tabi
On some Nvidia GPUs (i.e. GA100), the FRTS region is not allocated because there is no display hardware. If the region does not exist, then FWSEC-FRTS should not be run.
Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/gsp/boot.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/gpu/nova-core/gsp/boot.rs +0
-0@@ -155,7 +155,10 @@ pub(crate) fn boot(let fb_layout =FbLayout ::new (chipset, bar, &gsp_fw)?;
drivers/gpu/nova-core/gsp/boot.rs pub(crate) fn boot( · line 164
Quoted 10 lines
drivers/gpu/nova-core/gsp/boot.rs +4
-1dev_dbg! (dev,"{:#x?}\n" , fb_layout); -Self ::run_fwsec_frts (dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; +// Only boot FRTS if it actually exists. +if !fb_layout.frts.is_empty () { +Self ::run_fwsec_frts (dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; + }let booter_loader =BooterFirmware ::new ( dev,
If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.
Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists
Quoted 9 lines · Eliot Courtney
Quoted 8 lines
-
Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; + // Only boot FRTS if it actually exists.
+ if !fb_layout.frts.is_empty() {
+Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; + }
letbooter_loader= BooterFirmware::new(
dev,
Quoted 4 lines
If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the
frts_sizehal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.
Yeah, I like that.
Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists
Quoted 6 lines · Timur Tabi
Quoted 4 lines
If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the
frts_sizehal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.Yeah, I like that.
Actually, using Option<FbRange> won't work. The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but it would need to be FbRange<frtsStart..frtsStart>.
So I can update the HAL, but not the FbRange code.
Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists
Quoted 5 lines · Timur Tabi
Quoted 5 lines
Quoted 4 lines
If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the
frts_sizehal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.
Quoted 8 lines
Quoted 1 line
Yeah, I like that.
Actually, using Option<FbRange> won't work. The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but
it would need to be FbRange<frtsStart..frtsStart>.So I can update the HAL, but not the FbRange code.
So after working on this a bit, I don't think it's actually going to improve things. The problem is that even though I saw that "the FRTS region doesn't exist", it kinda does, it's just zero size. That's because GSP-RM actually cares (and uses) frtsStart (RM calls it frtsOffset) even if frtsSize is zero. So we need to keep the FbRange no matter what. Changing frts_size() to return an Option<NonZeroUsize> results in this code:
let frts_size = hal.frts_size().map_or(0, |n| n.get());
We all know what's actually going on internally, but this line just obfuscates it.
I will reword the commit messages to say that the window is zero size rather than being non- existent.
Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists
Quoted 21 lines · Timur Tabi
Quoted 5 lines · Timur Tabi
Quoted 5 lines
Quoted 4 lines
If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the
frts_sizehal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.Quoted 8 lines
Quoted 1 line
Yeah, I like that.
Actually, using Option<FbRange> won't work. The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but
it would need to be FbRange<frtsStart..frtsStart>.So I can update the HAL, but not the FbRange code.
So after working on this a bit, I don't think it's actually going to improve things. The problem is that even though I saw that "the FRTS region doesn't exist", it kinda does, it's just zero size. That's because GSP-RM actually cares (and uses) frtsStart (RM calls it frtsOffset) even if frtsSize is zero. So we need to keep the FbRange no matter what. Changing
frts_size()to return an Option<NonZeroUsize> results in this code:
Quoted 6 lines
let frts_size = hal.frts_size().map_or(0, |n| n.get());We all know what's actually going on internally, but this line just obfuscates it.
I will reword the commit messages to say that the window is zero size rather than being non- existent.
That seems fair enough, thank you for trying it out.
PATCH 4/6 gpu: nova-core: add FbHal::frts_size() for GA100 support
In reply to Timur TabiIntroduce FbHal method frts_size() to return the size of the FRTS window. GA100 is a special case in that there is no FRTS, and so the size must be set to 0.
Note that we cannot use supports_display() to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and so supports_display() returns False), but the FRTS window size still needs to be 1MB.
--- drivers/gpu/nova-core/fb.rs | 6 +++--- drivers/gpu/nova-core/fb/hal.rs | 3 +++ drivers/gpu/nova-core/fb/hal/ga100.rs | 5 +++++ drivers/gpu/nova-core/fb/hal/ga102.rs | 8 +++++++- drivers/gpu/nova-core/fb/hal/tu102.rs | 8 +++++++- 5 files changed, 25 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/fb.rs +3 -3
@@ -216,10 +216,10 @@ pub(crate) fn new(chipset: Chipset, bar: &Bar0, gsp_fw: &GspFirmware) -> Result<let frts = {const FRTS_DOWN_ALIGN :Alignment =Alignment ::new ::<SZ_128K >(); -const FRTS_SIZE :u64 =usize_as_u64 (SZ_1M ); -let frts_base = vga_workspace.start.align_down (FRTS_DOWN_ALIGN ) -FRTS_SIZE ; +let frts_size:u64 = hal.frts_size (); +let frts_base = vga_workspace.start.align_down (FRTS_DOWN_ALIGN ) - frts_size; -FbRange (frts_base..frts_base +FRTS_SIZE ) +FbRange (frts_base..frts_base + frts_size) };let boot = {
drivers/gpu/nova-core/fb/hal.rs +3 -0
@@ -25,6 +25,9 @@ pub(crate) trait FbHal {/// Returns the VRAM size, in bytes. fn vidmem_size (&self, bar: &Bar0 ) ->u64 ; + +/// Returns the FRTS size, in bytes. +fn frts_size (&self) ->u64 ; }/// Returns the HAL corresponding to `chipset`.
drivers/gpu/nova-core/fb/hal/ga100.rs +5 -0
@@ -66,6 +66,11 @@ fn supports_display(&self, bar: &Bar0) -> bool {fn vidmem_size (&self, bar: &Bar0 ) ->u64 {super ::tu102::vidmem_size_gp102 (bar) } + +// GA100 is a special case where it has no FRTS. +fn frts_size (&self) ->u64 { +0 + } }const GA100 :Ga100 =Ga100 ;
drivers/gpu/nova-core/fb/hal/ga102.rs +7 -1
@@ -2,12 +2,14 @@use kernel::{ io::Io , - prelude::*,// + prelude::*, + sizes::*,// };use crate ::{ driver::Bar0 , fb::hal::FbHal , + num::*, regs,// };
@@ -35,6 +37,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {fn vidmem_size (&self, bar: &Bar0 ) ->u64 {vidmem_size_ga102 (bar) } + +fn frts_size (&self) ->u64 { +usize_as_u64 (SZ_1M ) + } }const GA102 :Ga102 =Ga102 ;
drivers/gpu/nova-core/fb/hal/tu102.rs +7 -1
@@ -2,12 +2,14 @@use kernel::{ io::Io , - prelude::*,// + prelude::*, + sizes::*,// };use crate ::{ driver::Bar0 , fb::hal::FbHal , + num::*, regs,// };
@@ -56,6 +58,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {fn vidmem_size (&self, bar: &Bar0 ) ->u64 {vidmem_size_gp102 (bar) } + +fn frts_size (&self) ->u64 { +usize_as_u64 (SZ_1M ) + } }const TU102 :Tu102 =Tu102 ;
Signature · 1 line
2.53.0
Alexandre Courbot
maintainerRe: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support
Quoted 8 lines · Timur Tabi
Introduce FbHal method
frts_size()to return the size of the FRTS window. GA100 is a special case in that there is no FRTS, and so the size must be set to 0.Note that we cannot use
supports_display()to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and sosupports_display()returns False), but the FRTS window size still needs to be 1MB.
Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.
Quoted 79 lines
Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/fb.rs | 6 +++--- drivers/gpu/nova-core/fb/hal.rs | 3 +++ drivers/gpu/nova-core/fb/hal/ga100.rs | 5 +++++ drivers/gpu/nova-core/fb/hal/ga102.rs | 8 +++++++- drivers/gpu/nova-core/fb/hal/tu102.rs | 8 +++++++- 5 files changed, 25 insertions(+), 5 deletions(-)drivers/gpu/nova-core/fb.rs +3
-3@@ -216,10 +216,10 @@ pub(crate) fn new(chipset: Chipset, bar: &Bar0, gsp_fw: &GspFirmware) -> Result<let frts = {const FRTS_DOWN_ALIGN :Alignment =Alignment ::new ::<SZ_128K >(); -const FRTS_SIZE :u64 =usize_as_u64 (SZ_1M ); -let frts_base = vga_workspace.start.align_down (FRTS_DOWN_ALIGN ) -FRTS_SIZE ; +let frts_size:u64 = hal.frts_size (); +let frts_base = vga_workspace.start.align_down (FRTS_DOWN_ALIGN ) - frts_size; -FbRange (frts_base..frts_base +FRTS_SIZE ) +FbRange (frts_base..frts_base + frts_size) };let boot = {drivers/gpu/nova-core/fb/hal.rs +3
-0@@ -25,6 +25,9 @@ pub(crate) trait FbHal {/// Returns the VRAM size, in bytes. fn vidmem_size (&self, bar: &Bar0 ) ->u64 ; + +/// Returns the FRTS size, in bytes. +fn frts_size (&self) ->u64 ; }/// Returns the HAL corresponding to `chipset`. drivers/gpu/nova-core/fb/hal/ga100.rs +5
-0@@ -66,6 +66,11 @@ fn supports_display(&self, bar: &Bar0) -> bool {fn vidmem_size (&self, bar: &Bar0 ) ->u64 {super ::tu102::vidmem_size_gp102 (bar) } + +// GA100 is a special case where it has no FRTS. +fn frts_size (&self) ->u64 { +0 + } }const GA100 :Ga100 =Ga100 ;drivers/gpu/nova-core/fb/hal/ga102.rs +3
-1@@ -2,12 +2,14 @@use kernel::{ io::Io , - prelude::*,// + prelude::*, + sizes::*,// };use crate ::{ driver::Bar0 , fb::hal::FbHal , + num::*, regs,// };
drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43
Quoted 7 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4
-0@@ -35,6 +37,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {fn vidmem_size (&self, bar: &Bar0 ) ->u64 {vidmem_size_ga102 (bar) } + +fn frts_size (&self) ->u64 { +usize_as_u64 (SZ_1M ) + }
This implementation is identical to the one in tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance how read_sysmem_flush_page is handled between ga100 and ga102).
Re: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support
Quoted 8 lines · Alexandre Courbot
Quoted 4 lines
Note that we cannot use
supports_display()to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and sosupports_display()returns False), but the FRTS window size still needs to be 1MB.Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.
I put it in this order because we don't want to call run_fwsec_frts() if frts is empty. Otherwise, frts could be empty because of this patch, and we would call run_fwsec_frts() anyway.
Quoted 11 lines
drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43
Quoted 4 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4
-0+ +fn frts_size (&self) ->u64 { +usize_as_u64 (SZ_1M ) + }This implementation is identical to the one in
tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance howread_sysmem_flush_pageis handled between ga100 and ga102).
I figured since all it did is return a constant, that it would be simpler to just do that, but I will change it.
Alexandre Courbot
maintainerRe: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support
Quoted 6 lines · Timur Tabi
Quoted 5 lines · Alexandre Courbot
Quoted 4 lines
Note that we cannot use
supports_display()to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and sosupports_display()returns False), but the FRTS window size still needs to be 1MB.
Quoted 6 lines
Quoted 3 lines
Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.
I put it in this order because we don't want to call
run_fwsec_frts()if frts is empty. Otherwise, frts could be empty because of this patch, and we would callrun_fwsec_frts()anyway.
Good point, that makes sense - taking my suggestion back.
Quoted 13 lines
Quoted 6 lines
drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43
Quoted 5 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4
-0+ +fn frts_size (&self) ->u64 { +usize_as_u64 (SZ_1M ) + }Quoted 6 lines
This implementation is identical to the one in
tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance howread_sysmem_flush_pageis handled between ga100 and ga102).
Quoted 2 lines
I figured since all it did is return a constant, that it would be simpler to just do that, but I will change it.
Yeah I agree it doesn't change much, it's just for the sake of staying consistent through the codebase.
PATCH 5/6 gpu: nova-core: skip the IFR header if present
In reply to Timur TabiThe GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS). When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.
On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads. Therefore, the search for the PCI expansion must skip the IFR header, if found.
--- drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +38 -1
@@ -89,13 +89,50 @@ struct VbiosIterator<'a> { last_found: bool, } +/// IFR signature: ASCII "NVGI" as a little-endian u32. +const IFR_SIGNATURE :u32 =0x4947564E ; +/// ROM directory signature: ASCII "RFRD" as a little-endian u32. +const ROM_DIRECTORY_SIGNATURE :u32 =0x44524652 ; +impl <'a>VbiosIterator <'a> {fn new (dev: &'a device::Device , bar0: &'aBar0 ) ->Result <Self > { +let sig = bar0.try_read32 (ROM_OFFSET )?; + +let current_offset =if sig ==IFR_SIGNATURE { +let fixed1 = bar0.try_read32 (ROM_OFFSET +4 )?; +let version = ((fixed1 >>8 ) &0xff )as u8 ; + +match version { +// Note: We do not actually expect to see v1 or v2 on these GPUs +1 |2 => { +let fixed_data_size = ((fixed1 >>16 ) &0x7fff )as usize ; + bar0.try_read32 (ROM_OFFSET + fixed_data_size +4 )?as usize + } +3 => { +let fixed2 = bar0.try_read32 (ROM_OFFSET +8 )?; +let total_data_size = (fixed2 &0x000f_ffff )as usize ; +let dir_offset = bar0.try_read32 (ROM_OFFSET + total_data_size)?as usize +4096 ; +let dir_sig = bar0.try_read32 (ROM_OFFSET + dir_offset)?; +if dir_sig !=ROM_DIRECTORY_SIGNATURE { +dev_err! (dev,"could not find IFR ROM directory\n" ); +return Err (EINVAL ); + } + bar0.try_read32 (ROM_OFFSET + dir_offset +8 )?as usize + } + _ => { +dev_err! (dev,"unsupported IFR header version {}\n" , version); +return Err (EINVAL ); + } + } + }else { +0 + }; +Ok (Self { dev, bar0, data:KVec ::new (), - current_offset:0 , + current_offset, last_found:false , }) }
Signature · 1 line
2.53.0
Re: [PATCH 5/6] gpu: nova-core: skip the IFR header if present
Quoted 10 lines · Timur Tabi
The GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS). When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.
On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads. Therefore, the search for the PCI expansion must skip the IFR header, if found.
I like this commit message a lot because it made it really easy for me to understand why we are doing this. Could we concisely put a comment in the code as well explaining why it's necessary?
Quoted 11 lines
Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)drivers/gpu/nova-core/vbios.rs +0
-0
drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 95
Quoted 7 lines
drivers/gpu/nova-core/vbios.rs +4
-0@@ -89,13 +89,50 @@ struct VbiosIterator<'a> { last_found: bool, } +/// IFR signature: ASCII "NVGI" as a little-endian u32. +const IFR_SIGNATURE :u32 =0x4947564E ; +/// ROM directory signature: ASCII "RFRD" as a little-endian u32. +const ROM_DIRECTORY_SIGNATURE :u32 =0x44524652 ;
I wonder if we can use the bindings for some of these constants, including the fixed value offsets below. I found these values in OpenRM in dev_bus.h, for example (but it was in tu102 so even though the values are the same not sure if it's semantically correct):
``` #define NV_PBUS_IFR_FMT_FIXED0 0x00000000 /* */ #define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE 31:0 /* */ #define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE_VALUE 0x4947564E /* */ #define NV_PBUS_IFR_FMT_FIXED1 0x00000004 /* */ #define NV_PBUS_IFR_FMT_FIXED1_VERSIONSW 15:8 /* */ #define NV_PBUS_IFR_FMT_FIXED1_FIXED_DATA_SIZE 30:16 /* */ #define NV_PBUS_IFR_FMT_FIXED2 0x00000008 /* */ #define NV_PBUS_IFR_FMT_FIXED2_TOTAL_DATA_SIZE 19:0 /* */ ```
Unfortunately IIUC we can't use the bitfield range specifies via bindgen, but the FIXEDX offsets and some of the signature values might be doable via bindings. WDYT?
Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +2
-0+impl <'a>VbiosIterator <'a> {fn new (dev: &'a device::Device , bar0: &'aBar0 ) ->Result <Self > { +let sig = bar0.try_read32 (ROM_OFFSET )?;
drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 109
Quoted 10 lines
drivers/gpu/nova-core/vbios.rs +10
-0+ +let current_offset =if sig ==IFR_SIGNATURE { +let fixed1 = bar0.try_read32 (ROM_OFFSET +4 )?; +let version = ((fixed1 >>8 ) &0xff )as u8 ; + +match version { +// Note: We do not actually expect to see v1 or v2 on these GPUs +1 |2 => { +let fixed_data_size = ((fixed1 >>16 ) &0x7fff )as usize ; + bar0.try_read32 (ROM_OFFSET + fixed_data_size +4 )?as usize
Should this be an error instead if we don't expect to see it?
drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 113
Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +4
-0+ } + 3 => { +let fixed2 = bar0.try_read32 (ROM_OFFSET +8 )?; +let total_data_size = (fixed2 &0x000f_ffff )as usize ;
What about using the bitfield! macro to define bitfields for these things? e.g.
```
bitfield! {
struct IfrFixed1(u32) {
15:8 version as u8;
30:16 fixed_data_size as u16;
}
}
bitfield! {
struct IfrFixed2(u32) {
19:0 total_data_size as u32;
}
}
```drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 114
Quoted 1 line
drivers/gpu/nova-core/vbios.rs +1
-0+let dir_offset = bar0.try_read32 (ROM_OFFSET + total_data_size)?as usize +4096 ;
What about making an inline constant for 4096?
Quoted 25 lines
drivers/gpu/nova-core/vbios.rs +17
-1+let dir_sig = bar0.try_read32 (ROM_OFFSET + dir_offset)?; +if dir_sig !=ROM_DIRECTORY_SIGNATURE { +dev_err! (dev,"could not find IFR ROM directory\n" ); +return Err (EINVAL ); + } + bar0.try_read32 (ROM_OFFSET + dir_offset +8 )?as usize + } + _ => { +dev_err! (dev,"unsupported IFR header version {}\n" , version); +return Err (EINVAL ); + } + } + }else { +0 + }; +Ok (Self { dev, bar0, data:KVec ::new (), - current_offset:0 , + current_offset, last_found:false , }) }
Re: [PATCH 5/6] gpu: nova-core: skip the IFR header if present
Quoted 12 lines · Eliot Courtney
Quoted 10 lines · Timur Tabi
The GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS). When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.
On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads. Therefore, the search for the PCI expansion must skip the IFR header, if found.
Quoted 3 lines
I like this commit message a lot because it made it really easy for me to understand why we are doing this. Could we concisely put a comment in the code as well explaining why it's necessary?
Sure, I'll add one for v2.
Quoted 35 lines
Quoted 14 lines
Signed-off-by Timur Tabi <[email protected]>---
drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++-1 file changed, 38 insertions(+), 1 deletion(-)drivers/gpu/nova-core/vbios.rs +0
-0@@ -89,13 +89,50 @@ struct VbiosIterator<'a> { last_found: bool, }drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 95
Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +4
-0+/// IFR signature: ASCII "NVGI" as a little-endian u32. +const IFR_SIGNATURE :u32 =0x4947564E ; +/// ROM directory signature: ASCII "RFRD" as a little-endian u32. +const ROM_DIRECTORY_SIGNATURE :u32 =0x44524652 ;I wonder if we can use the bindings for some of these constants, including the fixed value offsets below. I found these values in OpenRM in
dev_bus.h, for example (but it was in tu102 so even though the values are the same not sure if it's semantically correct):``` #define NV_PBUS_IFR_FMT_FIXED0 0x00000000 /* */ #define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE 31:0 /* */ #define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE_VALUE 0x4947564E /* */ #define NV_PBUS_IFR_FMT_FIXED1 0x00000004 /* */ #define NV_PBUS_IFR_FMT_FIXED1_VERSIONSW 15:8 /* */ #define NV_PBUS_IFR_FMT_FIXED1_FIXED_DATA_SIZE 30:16 /* */ #define NV_PBUS_IFR_FMT_FIXED2 0x00000008 /* */ #define NV_PBUS_IFR_FMT_FIXED2_TOTAL_DATA_SIZE 19:0 /* */ ```
Quoted 3 lines
Unfortunately IIUC we can't use the bitfield range specifies via bindgen, but the FIXEDX offsets and some of the signature values might be doable via bindings. WDYT?
I can rename the two macros and add the rest as new constants, but adding them to bindgen.rs seems overkill.
Quoted 7 lines
Quoted 7 lines
impl<'a> VbiosIterator<'a> {
fnnew(dev: &'a device::Device, bar0: &'a Bar0)-> Result<Self> { + let sig =bar0.try_read32(ROM_OFFSET)?;
+
+ letcurrent_offset= if sig ==IFR_SIGNATURE{
+ let fixed1 =bar0.try_read32(ROM_OFFSET + 4)?; + let version = ((fixed1 >> 8) &0xff) as u8;
Quoted 8 lines
Quoted 6 lines
+
+ match version {
+ // Note: We do not actually expect to see v1 or v2 on these GPUs + 1 | 2 => {
+ letfixed_data_size= ((fixed1 >> 16) &0x7fff) as usize; +bar0.try_read32(ROM_OFFSET + fixed_data_size + 4)? as usizeShould this be an error instead if we don't expect to see it?
Well, I can't be certain that we'll never actually see it. For example, my corresponding patch for Nouveau does not have this comment because it's possible that Nouveau might encounter it since it supports much older GPUs that actually have an IFR header with those versions.
Quoted 8 lines
Quoted 4 lines
+ }
+ 3 => {
+ let fixed2 =bar0.try_read32(ROM_OFFSET + 8)?; + lettotal_data_size= (fixed2 & 0x000f_ffff) as usize;What about using the bitfield! macro to define bitfields for these things? e.g.
Quoted 14 lines
```
bitfield! {
struct IfrFixed1(u32) {
15:8 version as u8;
30:16 fixed_data_size as u16;
}
}
bitfield! {
struct IfrFixed2(u32) {
19:0 total_data_size as u32;
}
}
```Yes, that's a good idea.
Quoted 4 lines
Quoted 2 lines
+ let
dir_offset=bar0.try_read32(ROM_OFFSET + total_data_size)? as usize + 4096;What about making an inline constant for 4096?
Sure. I'd have to invent one since RM just hard-codes 4096.
Quoted 1 line
PATCH 6/6 gpu: nova-core: enable GA100
In reply to Timur TabiGA100 is a compute-only variant of GA102 that boots GSP-RM like a Turing, although it also has its own unique requirements.
Now that all the pieces are in place, we can enable GA100 support. Although architecturally like an Ampere, GA100 uses the same GSP-RM firmware files as Turing, and therefore must boot it like Turing does. However, as a compute-only part, GA100 has no display engine.
--- drivers/gpu/nova-core/falcon/hal.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/nova-core/falcon/hal.rs +1 -2
@@ -77,13 +77,12 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(use Chipset ::*;let hal =match chipset { -TU102 |TU104 |TU106 |TU116 |TU117 => { +TU102 |TU104 |TU106 |TU116 |TU117 |GA100 => {KBox ::new (tu102::Tu102 ::<E >::new (),GFP_KERNEL )?as KBox <dyn FalconHal <E >> }GA102 |GA103 |GA104 |GA106 |GA107 |AD102 |AD103 |AD104 |AD106 |AD107 => {KBox ::new (ga102::Ga102 ::<E >::new (),GFP_KERNEL )?as KBox <dyn FalconHal <E >> } - _ =>return Err (ENOTSUPP ), };Ok (hal)
Signature · 1 line
2.53.0
Re: [PATCH 6/6] gpu: nova-core: enable GA100
Quoted 22 lines · Timur Tabi
GA100 is a compute-only variant of GA102 that boots GSP-RM like a Turing, although it also has its own unique requirements.
Now that all the pieces are in place, we can enable GA100 support. Although architecturally like an Ampere, GA100 uses the same GSP-RM firmware files as Turing, and therefore must boot it like Turing does. However, as a compute-only part, GA100 has no display engine.
Signed-off-by Timur Tabi <[email protected]>--- drivers/gpu/nova-core/falcon/hal.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/gpu/nova-core/falcon/hal.rs +0
-1@@ -77,13 +77,12 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(use Chipset ::*;let hal =match chipset { -TU102 |TU104 |TU106 |TU116 |TU117 => {
drivers/gpu/nova-core/falcon/hal.rs pub(super) fn falcon_hal<E: FalconEngine + 'static>( · line 88
Quoted 10 lines
drivers/gpu/nova-core/falcon/hal.rs +1
-1+ TU102 | TU104 | TU106 | TU116 | TU117 | GA100 => {KBox ::new (tu102::Tu102 ::<E >::new (),GFP_KERNEL )?as KBox <dyn FalconHal <E >> } GA102 | GA103 | GA104 | GA106 | GA107 | AD102 | AD103 | AD104 | AD106 | AD107 => {KBox ::new (ga102::Ga102 ::<E >::new (),GFP_KERNEL )?as KBox <dyn FalconHal <E >> } - _ => return Err(ENOTSUPP), };Ok (hal)