Skip to contentSkip to conversation

gpu: nova-core: add GA100 support

PATCH Superseded Timur Tabi 3 participants 20 messages Apr 10 – Apr 15

Subsystems (4)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Timur Tabi

author
#0 lore raw

cover gpu: nova-core: add GA100 support

GA100 is an odd GPU. Architecturally, it's an Ampere. However, it uses the same GSP-RM boot process as a Turing. It's VBIOS, for whatever reason, has an IFR header that precedes the PCI ROM header and must be skipped. In addition, the FRTS window size must be set to 0 just because it's a GA100.

So add all the missing code that GA100 needs to be supported. There was some infrastructure already for GA100 (e.g. the FbHal), but it was incomplete.

Timur Tabi (6):
gpu: nova-core: use correct fwsignature for GA100
gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header
signature
gpu: nova-core: only boot FRTS if it actually exists
gpu: nova-core: add FbHal::frts_size() for GA100 support
gpu: nova-core: skip the IFR header if present
gpu: nova-core: enable GA100

 drivers/gpu/nova-core/falcon/hal.rs   |  3 +-
 drivers/gpu/nova-core/fb.rs           |  6 ++--
 drivers/gpu/nova-core/fb/hal.rs       |  3 ++
 drivers/gpu/nova-core/fb/hal/ga100.rs |  5 ++++
 drivers/gpu/nova-core/fb/hal/ga102.rs |  8 +++++-
 drivers/gpu/nova-core/fb/hal/tu102.rs |  8 +++++-
 drivers/gpu/nova-core/firmware/gsp.rs |  2 +-
 drivers/gpu/nova-core/gsp/boot.rs     |  5 +++-
 drivers/gpu/nova-core/vbios.rs        | 41 +++++++++++++++++++++++++--
 9 files changed, 70 insertions(+), 11 deletions(-)

base-commit: a7a080bb4236ebe577b6776d940d1717912ff6dd

Signature · 1 line
2.53.0
author
#1 lore raw

PATCH 1/6 gpu: nova-core: use correct fwsignature for GA100

Although GA100 uses the same GSP-RM firmware as Turing, it has a different signature specifically for it.

Fixes 121ea04cd9f2 ("gpu: nova-core: add support for Turing/GA100 fwsignature")
---
 drivers/gpu/nova-core/firmware/gsp.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/firmware/gsp.rs +1 -1
index 2fcc255c3bc8..63e464788c0b 100644
@@ -139,7 +139,7 @@ pub(crate) fn new<'a>(
                         }
                         Architecture::Turing => ".fwsignature_tu10x",
                         // GA100 uses the same firmware as Turing
-                        Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_tu10x",
+                        Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_ga100",
                         Architecture::Ampere => ".fwsignature_ga10x",
                         Architecture::Ada => ".fwsignature_ad10x",
                     };
Signature · 1 line
2.53.0
Reviewed-by on v1 1/6
#2 lore raw

Re: [PATCH 1/6] gpu: nova-core: use correct fwsignature for GA100

Quoted 14 lines · Timur Tabi

Although GA100 uses the same GSP-RM firmware as Turing, it has a different signature specifically for it.

Fixes 121ea04cd9f2 ("gpu: nova-core: add support for Turing/GA100 fwsignature")
---
 drivers/gpu/nova-core/firmware/gsp.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/firmware/gsp.rs +0 -0
index 2fcc255c3bc8..63e464788c0b 100644

drivers/gpu/nova-core/firmware/gsp.rs pub(crate) fn new<'a>( · line 145

Quoted 8 lines
drivers/gpu/nova-core/firmware/gsp.rs +1 -1
@@ -139,7 +139,7 @@ pub(crate) fn new<'a>(
                         }
                         Architecture::Turing => ".fwsignature_tu10x",
                         // GA100 uses the same firmware as Turing
-                        Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_tu10x",
+                        Architecture::Ampere if chipset == Chipset::GA100 => ".fwsignature_ga100",
                         Architecture::Ampere => ".fwsignature_ga10x",
                         Architecture::Ada => ".fwsignature_ad10x",
                     };
author
#3 lore raw

PATCH 2/6 gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header signature

In reply to Timur Tabi

Nvidia GPUs have some PCI expansion ROM sections that have an Nvidia- specific signature instead of 0xAA55. Signature 0xBB77 is actually an internal-only value that has been deprecated for over a decade. Nova-core will never encounter a GPU with that signature, so don't look for it.

---
 drivers/gpu/nova-core/vbios.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +1 -1
index ebda28e596c5..e726594eb130 100644
@@ -491,7 +491,7 @@ fn new(dev: &device::Device, data: &[u8]) -> Result<Self> {
 
         // Check for valid ROM signatures.
         match signature {
-            0xAA55 | 0xBB77 | 0x4E56 => {}
+            0xAA55 | 0x4E56 => {}
             _ => {
                 dev_err!(dev, "ROM signature unknown {:#x}\n", signature);
                 return Err(EINVAL);
Signature · 1 line
2.53.0
Reviewed-by on v1 2/6
#4 lore raw

Re: [PATCH 2/6] gpu: nova-core: do not consider 0xBB77 as a valid PCI ROM header signature

Quoted 16 lines · Timur Tabi

Nvidia GPUs have some PCI expansion ROM sections that have an Nvidia- specific signature instead of 0xAA55. Signature 0xBB77 is actually an internal-only value that has been deprecated for over a decade. Nova-core will never encounter a GPU with that signature, so don't look for it.

---
 drivers/gpu/nova-core/vbios.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +0 -0
index ebda28e596c5..e726594eb130 100644

drivers/gpu/nova-core/vbios.rs fn new(dev: &device::Device, data: &[u8]) -> Result<Self> { · line 497

Quoted 8 lines
drivers/gpu/nova-core/vbios.rs +1 -1
@@ -491,7 +491,7 @@ fn new(dev: &device::Device, data: &[u8]) -> Result<Self> {
 
         // Check for valid ROM signatures.
         match signature {
-            0xAA55 | 0xBB77 | 0x4E56 => {}
+            0xAA55 | 0x4E56 => {}
             _ => {
                 dev_err!(dev, "ROM signature unknown {:#x}\n", signature);
                 return Err(EINVAL);
author
#5 lore raw

PATCH 3/6 gpu: nova-core: only boot FRTS if it actually exists

In reply to Timur Tabi

On some Nvidia GPUs (i.e. GA100), the FRTS region is not allocated because there is no display hardware. If the region does not exist, then FWSEC-FRTS should not be run.

---
 drivers/gpu/nova-core/gsp/boot.rs | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/gsp/boot.rs +4 -1
index 18f356c9178e..d50dc9f41831 100644
@@ -155,7 +155,10 @@ pub(crate) fn boot(
         let fb_layout = FbLayout::new(chipset, bar, &gsp_fw)?;
         dev_dbg!(dev, "{:#x?}\n", fb_layout);
 
-        Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
+        // Only boot FRTS if it actually exists.
+        if !fb_layout.frts.is_empty() {
+            Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
+        }
 
         let booter_loader = BooterFirmware::new(
             dev,
Signature · 1 line
2.53.0
#6 lore raw

Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists

Quoted 15 lines · Timur Tabi

On some Nvidia GPUs (i.e. GA100), the FRTS region is not allocated because there is no display hardware. If the region does not exist, then FWSEC-FRTS should not be run.

---
 drivers/gpu/nova-core/gsp/boot.rs | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/gsp/boot.rs +0 -0
index 18f356c9178e..d50dc9f41831 100644
@@ -155,7 +155,10 @@ pub(crate) fn boot(
         let fb_layout = FbLayout::new(chipset, bar, &gsp_fw)?;

drivers/gpu/nova-core/gsp/boot.rs pub(crate) fn boot( · line 164

Quoted 10 lines
drivers/gpu/nova-core/gsp/boot.rs +4 -1
         dev_dbg!(dev, "{:#x?}\n", fb_layout);
 
-        Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
+        // Only boot FRTS if it actually exists.
+        if !fb_layout.frts.is_empty() {
+            Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?;
+        }
 
         let booter_loader = BooterFirmware::new(
             dev,

If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.

author
#7 lore raw

Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists

Quoted 9 lines · Eliot Courtney
Quoted 8 lines

-        Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; +        // Only boot FRTS if it actually exists.
+        if !fb_layout.frts.is_empty() {
+            Self::run_fwsec_frts(dev, chipset, gsp_falcon, bar, &bios, &fb_layout)?; +        }

          let booter_loader = BooterFirmware::new(
              dev,

Quoted 4 lines

If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.

Yeah, I like that.

author
#8 lore raw

Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists

Quoted 6 lines · Timur Tabi
Quoted 4 lines

If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.

Yeah, I like that.

Actually, using Option<FbRange> won't work. The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but it would need to be FbRange<frtsStart..frtsStart>.

So I can update the HAL, but not the FbRange code.

author
#9 lore raw

Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists

Quoted 5 lines · Timur Tabi
Quoted 5 lines
Quoted 4 lines

If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.

Quoted 8 lines
Quoted 1 line

Yeah, I like that.

Actually, using Option<FbRange> won't work.  The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but
it would need to be FbRange<frtsStart..frtsStart>.

So I can update the HAL, but not the FbRange code.

So after working on this a bit, I don't think it's actually going to improve things. The problem is that even though I saw that "the FRTS region doesn't exist", it kinda does, it's just zero size. That's because GSP-RM actually cares (and uses) frtsStart (RM calls it frtsOffset) even if frtsSize is zero. So we need to keep the FbRange no matter what. Changing frts_size() to return an Option<NonZeroUsize> results in this code:

            let frts_size = hal.frts_size().map_or(0, |n| n.get());

We all know what's actually going on internally, but this line just obfuscates it.

I will reword the commit messages to say that the window is zero size rather than being non- existent.

#10 lore raw

Re: [PATCH 3/6] gpu: nova-core: only boot FRTS if it actually exists

Quoted 21 lines · Timur Tabi
Quoted 5 lines · Timur Tabi
Quoted 5 lines
Quoted 4 lines

If FRTS is optional, WDYT about making fb_layout.frts an Option<FbRange> and having the frts_size hal method return Option<NonZeroUsize> for example? I think it will make it clearer in other locations that use FRTS that it possibly might not exist.

Quoted 8 lines
Quoted 1 line

Yeah, I like that.

Actually, using Option<FbRange> won't work.  The problem is that even if frtsSize is 0, frtsStart (aka frtsOffset) must still be a non-zero number, which means we still need to keep the FbRange, but
it would need to be FbRange<frtsStart..frtsStart>.

So I can update the HAL, but not the FbRange code.

So after working on this a bit, I don't think it's actually going to improve things. The problem is that even though I saw that "the FRTS region doesn't exist", it kinda does, it's just zero size. That's because GSP-RM actually cares (and uses) frtsStart (RM calls it frtsOffset) even if frtsSize is zero. So we need to keep the FbRange no matter what. Changing frts_size() to return an Option<NonZeroUsize> results in this code:

Quoted 6 lines
            let frts_size = hal.frts_size().map_or(0, |n| n.get());

We all know what's actually going on internally, but this line just obfuscates it.

I will reword the commit messages to say that the window is zero size rather than being non- existent.

That seems fair enough, thank you for trying it out.

author
#11 lore raw

PATCH 4/6 gpu: nova-core: add FbHal::frts_size() for GA100 support

In reply to Timur Tabi

Introduce FbHal method frts_size() to return the size of the FRTS window. GA100 is a special case in that there is no FRTS, and so the size must be set to 0.

Note that we cannot use supports_display() to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and so supports_display() returns False), but the FRTS window size still needs to be 1MB.

---
 drivers/gpu/nova-core/fb.rs           | 6 +++---
 drivers/gpu/nova-core/fb/hal.rs       | 3 +++
 drivers/gpu/nova-core/fb/hal/ga100.rs | 5 +++++
 drivers/gpu/nova-core/fb/hal/ga102.rs | 8 +++++++-
 drivers/gpu/nova-core/fb/hal/tu102.rs | 8 +++++++-
 5 files changed, 25 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/fb.rs +3 -3
index bdd5eed760e1..abffc21427a1 100644
@@ -216,10 +216,10 @@ pub(crate) fn new(chipset: Chipset, bar: &Bar0, gsp_fw: &GspFirmware) -> Result<
 
         let frts = {
             const FRTS_DOWN_ALIGN: Alignment = Alignment::new::<SZ_128K>();
-            const FRTS_SIZE: u64 = usize_as_u64(SZ_1M);
-            let frts_base = vga_workspace.start.align_down(FRTS_DOWN_ALIGN) - FRTS_SIZE;
+            let frts_size: u64 = hal.frts_size();
+            let frts_base = vga_workspace.start.align_down(FRTS_DOWN_ALIGN) - frts_size;
 
-            FbRange(frts_base..frts_base + FRTS_SIZE)
+            FbRange(frts_base..frts_base + frts_size)
         };
 
         let boot = {
drivers/gpu/nova-core/fb/hal.rs +3 -0
index aba0abd8ee00..1c01a6cbed65 100644
@@ -25,6 +25,9 @@ pub(crate) trait FbHal {
 
     /// Returns the VRAM size, in bytes.
     fn vidmem_size(&self, bar: &Bar0) -> u64;
+
+    /// Returns the FRTS size, in bytes.
+    fn frts_size(&self) -> u64;
 }
 
 /// Returns the HAL corresponding to `chipset`.
drivers/gpu/nova-core/fb/hal/ga100.rs +5 -0
index 1c03783cddef..0e7d91fbd130 100644
@@ -66,6 +66,11 @@ fn supports_display(&self, bar: &Bar0) -> bool {
     fn vidmem_size(&self, bar: &Bar0) -> u64 {
         super::tu102::vidmem_size_gp102(bar)
     }
+
+    // GA100 is a special case where it has no FRTS.
+    fn frts_size(&self) -> u64 {
+        0
+    }
 }
 
 const GA100: Ga100 = Ga100;
drivers/gpu/nova-core/fb/hal/ga102.rs +7 -1
index 4b9f0f74d0e7..167d38e34370 100644
@@ -2,12 +2,14 @@
 
 use kernel::{
     io::Io,
-    prelude::*, //
+    prelude::*,
+    sizes::*, //
 };
 
 use crate::{
     driver::Bar0,
     fb::hal::FbHal,
+    num::*,
     regs, //
 };
 
@@ -35,6 +37,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {
     fn vidmem_size(&self, bar: &Bar0) -> u64 {
         vidmem_size_ga102(bar)
     }
+
+    fn frts_size(&self) -> u64 {
+        usize_as_u64(SZ_1M)
+    }
 }
 
 const GA102: Ga102 = Ga102;
drivers/gpu/nova-core/fb/hal/tu102.rs +7 -1
index 281bb796e198..3482469f9cf1 100644
@@ -2,12 +2,14 @@
 
 use kernel::{
     io::Io,
-    prelude::*, //
+    prelude::*,
+    sizes::*, //
 };
 
 use crate::{
     driver::Bar0,
     fb::hal::FbHal,
+    num::*,
     regs, //
 };
 
@@ -56,6 +58,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {
     fn vidmem_size(&self, bar: &Bar0) -> u64 {
         vidmem_size_gp102(bar)
     }
+
+    fn frts_size(&self) -> u64 {
+        usize_as_u64(SZ_1M)
+    }
 }
 
 const TU102: Tu102 = Tu102;
Signature · 1 line
2.53.0
#12 lore raw

Re: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support

Quoted 8 lines · Timur Tabi

Introduce FbHal method frts_size() to return the size of the FRTS window. GA100 is a special case in that there is no FRTS, and so the size must be set to 0.

Note that we cannot use supports_display() to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and so supports_display() returns False), but the FRTS window size still needs to be 1MB.

Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.

Quoted 79 lines
---
 drivers/gpu/nova-core/fb.rs           | 6 +++---
 drivers/gpu/nova-core/fb/hal.rs       | 3 +++
 drivers/gpu/nova-core/fb/hal/ga100.rs | 5 +++++
 drivers/gpu/nova-core/fb/hal/ga102.rs | 8 +++++++-
 drivers/gpu/nova-core/fb/hal/tu102.rs | 8 +++++++-
 5 files changed, 25 insertions(+), 5 deletions(-)
drivers/gpu/nova-core/fb.rs +3 -3
index bdd5eed760e1..abffc21427a1 100644
@@ -216,10 +216,10 @@ pub(crate) fn new(chipset: Chipset, bar: &Bar0, gsp_fw: &GspFirmware) -> Result<
 
         let frts = {
             const FRTS_DOWN_ALIGN: Alignment = Alignment::new::<SZ_128K>();
-            const FRTS_SIZE: u64 = usize_as_u64(SZ_1M);
-            let frts_base = vga_workspace.start.align_down(FRTS_DOWN_ALIGN) - FRTS_SIZE;
+            let frts_size: u64 = hal.frts_size();
+            let frts_base = vga_workspace.start.align_down(FRTS_DOWN_ALIGN) - frts_size;
 
-            FbRange(frts_base..frts_base + FRTS_SIZE)
+            FbRange(frts_base..frts_base + frts_size)
         };
 
         let boot = {
drivers/gpu/nova-core/fb/hal.rs +3 -0
index aba0abd8ee00..1c01a6cbed65 100644
@@ -25,6 +25,9 @@ pub(crate) trait FbHal {
 
     /// Returns the VRAM size, in bytes.
     fn vidmem_size(&self, bar: &Bar0) -> u64;
+
+    /// Returns the FRTS size, in bytes.
+    fn frts_size(&self) -> u64;
 }
 
 /// Returns the HAL corresponding to `chipset`.
drivers/gpu/nova-core/fb/hal/ga100.rs +5 -0
index 1c03783cddef..0e7d91fbd130 100644
@@ -66,6 +66,11 @@ fn supports_display(&self, bar: &Bar0) -> bool {
     fn vidmem_size(&self, bar: &Bar0) -> u64 {
         super::tu102::vidmem_size_gp102(bar)
     }
+
+    // GA100 is a special case where it has no FRTS.
+    fn frts_size(&self) -> u64 {
+        0
+    }
 }
 
 const GA100: Ga100 = Ga100;
drivers/gpu/nova-core/fb/hal/ga102.rs +3 -1
index 4b9f0f74d0e7..167d38e34370 100644
@@ -2,12 +2,14 @@
 
 use kernel::{
     io::Io,
-    prelude::*, //
+    prelude::*,
+    sizes::*, //
 };
 
 use crate::{
     driver::Bar0,
     fb::hal::FbHal,
+    num::*,
     regs, //
 };
 

drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43

Quoted 7 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4 -0
@@ -35,6 +37,10 @@ fn supports_display(&self, bar: &Bar0) -> bool {
     fn vidmem_size(&self, bar: &Bar0) -> u64 {
         vidmem_size_ga102(bar)
     }
+
+    fn frts_size(&self) -> u64 {
+        usize_as_u64(SZ_1M)
+    }

This implementation is identical to the one in tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance how read_sysmem_flush_page is handled between ga100 and ga102).

author
#13 lore raw

Re: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support

Quoted 8 lines · Alexandre Courbot
Quoted 4 lines

Note that we cannot use supports_display() to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and so supports_display() returns False), but the FRTS window size still needs to be 1MB.

Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.

I put it in this order because we don't want to call run_fwsec_frts() if frts is empty. Otherwise, frts could be empty because of this patch, and we would call run_fwsec_frts() anyway.

Quoted 11 lines

drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43

Quoted 4 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4 -0
+
+    fn frts_size(&self) -> u64 {
+        usize_as_u64(SZ_1M)
+    }

This implementation is identical to the one in tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance how read_sysmem_flush_page is handled between ga100 and ga102).

I figured since all it did is return a constant, that it would be simpler to just do that, but I will change it.

#14 lore raw

Re: [PATCH 4/6] gpu: nova-core: add FbHal::frts_size() for GA100 support

Quoted 6 lines · Timur Tabi
Quoted 5 lines · Alexandre Courbot
Quoted 4 lines

Note that we cannot use supports_display() to determine the FRTS size because there are other GPUs (e.g. GA102GL) that have display disabled (and so supports_display() returns False), but the FRTS window size still needs to be 1MB.

Quoted 6 lines
Quoted 3 lines

Order-wise this should probably come before patch 3 - I was a bit confused about the fact that we are checking the size of the frts region since it is defined as a constant until this patch.

I put it in this order because we don't want to call run_fwsec_frts() if frts is empty. Otherwise, frts could be empty because of this patch, and we would call run_fwsec_frts() anyway.

Good point, that makes sense - taking my suggestion back.

Quoted 13 lines
Quoted 6 lines

drivers/gpu/nova-core/fb/hal/ga102.rs fn supports_display(&self, bar: &Bar0) -> bool { · line 43

Quoted 5 lines
drivers/gpu/nova-core/fb/hal/ga102.rs +4 -0
+
+    fn frts_size(&self) -> u64 {
+        usize_as_u64(SZ_1M)
+    }
Quoted 6 lines

This implementation is identical to the one in tu102.rs. As a pattern, when different HALs use different implementations, we define it into a function that both HALs can call instead of repeating the code (see for instance how read_sysmem_flush_page is handled between ga100 and ga102).

Quoted 2 lines

I figured since all it did is return a constant, that it would be simpler to just do that, but I will change it.

Yeah I agree it doesn't change much, it's just for the sake of staying consistent through the codebase.

author
#15 lore raw

PATCH 5/6 gpu: nova-core: skip the IFR header if present

In reply to Timur Tabi

The GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS). When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.

On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads. Therefore, the search for the PCI expansion must skip the IFR header, if found.

---
 drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +38 -1
index e726594eb130..a7ec68448a4a 100644
@@ -89,13 +89,50 @@ struct VbiosIterator<'a> {
     last_found: bool,
 }
 
+/// IFR signature: ASCII "NVGI" as a little-endian u32.
+const IFR_SIGNATURE: u32 = 0x4947564E;
+/// ROM directory signature: ASCII "RFRD" as a little-endian u32.
+const ROM_DIRECTORY_SIGNATURE: u32 = 0x44524652;
+
 impl<'a> VbiosIterator<'a> {
     fn new(dev: &'a device::Device, bar0: &'a Bar0) -> Result<Self> {
+        let sig = bar0.try_read32(ROM_OFFSET)?;
+
+        let current_offset = if sig == IFR_SIGNATURE {
+            let fixed1 = bar0.try_read32(ROM_OFFSET + 4)?;
+            let version = ((fixed1 >> 8) & 0xff) as u8;
+
+            match version {
+                // Note: We do not actually expect to see v1 or v2 on these GPUs
+                1 | 2 => {
+                    let fixed_data_size = ((fixed1 >> 16) & 0x7fff) as usize;
+                    bar0.try_read32(ROM_OFFSET + fixed_data_size + 4)? as usize
+                }
+                3 => {
+                    let fixed2 = bar0.try_read32(ROM_OFFSET + 8)?;
+                    let total_data_size = (fixed2 & 0x000f_ffff) as usize;
+                    let dir_offset = bar0.try_read32(ROM_OFFSET + total_data_size)? as usize + 4096;
+                    let dir_sig = bar0.try_read32(ROM_OFFSET + dir_offset)?;
+                    if dir_sig != ROM_DIRECTORY_SIGNATURE {
+                        dev_err!(dev, "could not find IFR ROM directory\n");
+                        return Err(EINVAL);
+                    }
+                    bar0.try_read32(ROM_OFFSET + dir_offset + 8)? as usize
+                }
+                _ => {
+                    dev_err!(dev, "unsupported IFR header version {}\n", version);
+                    return Err(EINVAL);
+                }
+            }
+        } else {
+            0
+        };
+
         Ok(Self {
             dev,
             bar0,
             data: KVec::new(),
-            current_offset: 0,
+            current_offset,
             last_found: false,
         })
     }
Signature · 1 line
2.53.0
#16 lore raw

Re: [PATCH 5/6] gpu: nova-core: skip the IFR header if present

Quoted 10 lines · Timur Tabi

The GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS). When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.

On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads. Therefore, the search for the PCI expansion must skip the IFR header, if found.

I like this commit message a lot because it made it really easy for me to understand why we are doing this. Could we concisely put a comment in the code as well explaining why it's necessary?

Quoted 11 lines
---
 drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +0 -0
index e726594eb130..a7ec68448a4a 100644

drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 95

Quoted 7 lines
drivers/gpu/nova-core/vbios.rs +4 -0
@@ -89,13 +89,50 @@ struct VbiosIterator<'a> {
     last_found: bool,
 }
 
+/// IFR signature: ASCII "NVGI" as a little-endian u32.
+const IFR_SIGNATURE: u32 = 0x4947564E;
+/// ROM directory signature: ASCII "RFRD" as a little-endian u32.
+const ROM_DIRECTORY_SIGNATURE: u32 = 0x44524652;

I wonder if we can use the bindings for some of these constants, including the fixed value offsets below. I found these values in OpenRM in dev_bus.h, for example (but it was in tu102 so even though the values are the same not sure if it's semantically correct):

```
#define NV_PBUS_IFR_FMT_FIXED0                        0x00000000 /*       */
#define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE                    31:0 /*       */
#define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE_VALUE        0x4947564E /*       */
#define NV_PBUS_IFR_FMT_FIXED1                        0x00000004 /*       */
#define NV_PBUS_IFR_FMT_FIXED1_VERSIONSW                    15:8 /*       */
#define NV_PBUS_IFR_FMT_FIXED1_FIXED_DATA_SIZE             30:16 /*       */
#define NV_PBUS_IFR_FMT_FIXED2                        0x00000008 /*       */
#define NV_PBUS_IFR_FMT_FIXED2_TOTAL_DATA_SIZE              19:0 /*       */
```

Unfortunately IIUC we can't use the bitfield range specifies via bindgen, but the FIXEDX offsets and some of the signature values might be doable via bindings. WDYT?

Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +2 -0
+
 impl<'a> VbiosIterator<'a> {
     fn new(dev: &'a device::Device, bar0: &'a Bar0) -> Result<Self> {
+        let sig = bar0.try_read32(ROM_OFFSET)?;

drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 109

Quoted 10 lines
drivers/gpu/nova-core/vbios.rs +10 -0
+
+        let current_offset = if sig == IFR_SIGNATURE {
+            let fixed1 = bar0.try_read32(ROM_OFFSET + 4)?;
+            let version = ((fixed1 >> 8) & 0xff) as u8;
+
+            match version {
+                // Note: We do not actually expect to see v1 or v2 on these GPUs
+                1 | 2 => {
+                    let fixed_data_size = ((fixed1 >> 16) & 0x7fff) as usize;
+                    bar0.try_read32(ROM_OFFSET + fixed_data_size + 4)? as usize

Should this be an error instead if we don't expect to see it?

drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 113

Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +4 -0
+                }
+                3 => {
+                    let fixed2 = bar0.try_read32(ROM_OFFSET + 8)?;
+                    let total_data_size = (fixed2 & 0x000f_ffff) as usize;

What about using the bitfield! macro to define bitfields for these things? e.g.

```
bitfield! {
    struct IfrFixed1(u32) {
        15:8 version as u8;
        30:16 fixed_data_size as u16;
    }
}

bitfield! {
    struct IfrFixed2(u32) {
        19:0 total_data_size as u32;
    }
}
```

drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 114

Quoted 1 line
drivers/gpu/nova-core/vbios.rs +1 -0
+                    let dir_offset = bar0.try_read32(ROM_OFFSET + total_data_size)? as usize + 4096;

What about making an inline constant for 4096?

Quoted 25 lines
drivers/gpu/nova-core/vbios.rs +17 -1
+                    let dir_sig = bar0.try_read32(ROM_OFFSET + dir_offset)?;
+                    if dir_sig != ROM_DIRECTORY_SIGNATURE {
+                        dev_err!(dev, "could not find IFR ROM directory\n");
+                        return Err(EINVAL);
+                    }
+                    bar0.try_read32(ROM_OFFSET + dir_offset + 8)? as usize
+                }
+                _ => {
+                    dev_err!(dev, "unsupported IFR header version {}\n", version);
+                    return Err(EINVAL);
+                }
+            }
+        } else {
+            0
+        };
+
         Ok(Self {
             dev,
             bar0,
             data: KVec::new(),
-            current_offset: 0,
+            current_offset,
             last_found: false,
         })
     }
author
#17 lore raw

Re: [PATCH 5/6] gpu: nova-core: skip the IFR header if present

Quoted 12 lines · Eliot Courtney
Quoted 10 lines · Timur Tabi

The GPU's ROM may begin with an Init-from-ROM (IFR) header that precedes the PCI Expansion ROM images (VBIOS).  When present, the PROM shadow method must parse this header to determine the offset where the PCI ROM images actually begin, and adjust all subsequent reads accordingly.

On most GPUs this is not needed because the IFR microcode has already applied the ROM offset so that PROM reads transparently skip the header. On GA100, for whatever reason, the IFR offset is not applied to PROM reads.  Therefore, the search for the PCI expansion must skip the IFR header, if found.

Quoted 3 lines

I like this commit message a lot because it made it really easy for me to understand why we are doing this. Could we concisely put a comment in the code as well explaining why it's necessary?

Sure, I'll add one for v2.

Quoted 35 lines
Quoted 14 lines

---
 drivers/gpu/nova-core/vbios.rs | 39 +++++++++++++++++++++++++++++++++-

 1 file changed, 38 insertions(+), 1 deletion(-)
drivers/gpu/nova-core/vbios.rs +0 -0
index e726594eb130..a7ec68448a4a 100644
@@ -89,13 +89,50 @@ struct VbiosIterator<'a> {
      last_found: bool,
  }

drivers/gpu/nova-core/vbios.rs struct VbiosIterator<'a> { · line 95

Quoted 4 lines
drivers/gpu/nova-core/vbios.rs +4 -0
+/// IFR signature: ASCII "NVGI" as a little-endian u32.
+const IFR_SIGNATURE: u32 = 0x4947564E;
+/// ROM directory signature: ASCII "RFRD" as a little-endian u32.
+const ROM_DIRECTORY_SIGNATURE: u32 = 0x44524652;

I wonder if we can use the bindings for some of these constants, including the fixed value offsets below. I found these values in OpenRM in dev_bus.h, for example (but it was in tu102 so even though the values are the same not sure if it's semantically correct):

```
#define NV_PBUS_IFR_FMT_FIXED0                        0x00000000 /*       */
#define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE                    31:0 /*       */
#define NV_PBUS_IFR_FMT_FIXED0_SIGNATURE_VALUE        0x4947564E /*       */
#define NV_PBUS_IFR_FMT_FIXED1                        0x00000004 /*       */
#define NV_PBUS_IFR_FMT_FIXED1_VERSIONSW                    15:8 /*       */
#define NV_PBUS_IFR_FMT_FIXED1_FIXED_DATA_SIZE             30:16 /*       */
#define NV_PBUS_IFR_FMT_FIXED2                        0x00000008 /*       */
#define NV_PBUS_IFR_FMT_FIXED2_TOTAL_DATA_SIZE              19:0 /*       */
```
Quoted 3 lines

Unfortunately IIUC we can't use the bitfield range specifies via bindgen, but the FIXEDX offsets and some of the signature values might be doable via bindings. WDYT?

I can rename the two macros and add the rest as new constants, but adding them to bindgen.rs seems overkill.

Quoted 7 lines
Quoted 7 lines

 impl<'a> VbiosIterator<'a> {
     fn new(dev: &'a device::Device, bar0: &'a Bar0) -> Result<Self> { +        let sig = bar0.try_read32(ROM_OFFSET)?;
+
+        let current_offset = if sig == IFR_SIGNATURE {
+            let fixed1 = bar0.try_read32(ROM_OFFSET + 4)?; +            let version = ((fixed1 >> 8) & 0xff) as u8;

Quoted 8 lines
Quoted 6 lines

+
+            match version {
+                // Note: We do not actually expect to see v1 or v2 on these GPUs +                1 | 2 => {
+                    let fixed_data_size = ((fixed1 >> 16) & 0x7fff) as usize; +                    bar0.try_read32(ROM_OFFSET + fixed_data_size + 4)? as usize

Should this be an error instead if we don't expect to see it?

Well, I can't be certain that we'll never actually see it. For example, my corresponding patch for Nouveau does not have this comment because it's possible that Nouveau might encounter it since it supports much older GPUs that actually have an IFR header with those versions.

Quoted 8 lines
Quoted 4 lines

+                }
+                3 => {
+                    let fixed2 = bar0.try_read32(ROM_OFFSET + 8)?; +                    let total_data_size = (fixed2 & 0x000f_ffff) as usize;

What about using the bitfield! macro to define bitfields for these things? e.g.

Quoted 14 lines
```
bitfield! {
    struct IfrFixed1(u32) {
        15:8 version as u8;
        30:16 fixed_data_size as u16;
    }
}

bitfield! {
    struct IfrFixed2(u32) {
        19:0 total_data_size as u32;
    }
}
```

Yes, that's a good idea.

Quoted 4 lines
Quoted 2 lines

+                    let dir_offset = bar0.try_read32(ROM_OFFSET + total_data_size)? as usize + 4096;

What about making an inline constant for 4096?

Sure. I'd have to invent one since RM just hard-codes 4096.

Quoted 1 line
author
#18 lore raw

PATCH 6/6 gpu: nova-core: enable GA100

In reply to Timur Tabi

GA100 is a compute-only variant of GA102 that boots GSP-RM like a Turing, although it also has its own unique requirements.

Now that all the pieces are in place, we can enable GA100 support. Although architecturally like an Ampere, GA100 uses the same GSP-RM firmware files as Turing, and therefore must boot it like Turing does. However, as a compute-only part, GA100 has no display engine.

---
 drivers/gpu/nova-core/falcon/hal.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/nova-core/falcon/hal.rs +1 -2
index a7e5ea8d0272..5c2f92629eec 100644
@@ -77,13 +77,12 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
     use Chipset::*;
 
     let hal = match chipset {
-        TU102 | TU104 | TU106 | TU116 | TU117 => {
+        TU102 | TU104 | TU106 | TU116 | TU117 | GA100 => {
             KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
         }
         GA102 | GA103 | GA104 | GA106 | GA107 | AD102 | AD103 | AD104 | AD106 | AD107 => {
             KBox::new(ga102::Ga102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
         }
-        _ => return Err(ENOTSUPP),
     };
 
     Ok(hal)
Signature · 1 line
2.53.0
Reviewed-by on v1 6/6
#19 lore raw

Re: [PATCH 6/6] gpu: nova-core: enable GA100

Quoted 22 lines · Timur Tabi

GA100 is a compute-only variant of GA102 that boots GSP-RM like a Turing, although it also has its own unique requirements.

Now that all the pieces are in place, we can enable GA100 support. Although architecturally like an Ampere, GA100 uses the same GSP-RM firmware files as Turing, and therefore must boot it like Turing does. However, as a compute-only part, GA100 has no display engine.

---
 drivers/gpu/nova-core/falcon/hal.rs | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/nova-core/falcon/hal.rs +0 -1
index a7e5ea8d0272..5c2f92629eec 100644
@@ -77,13 +77,12 @@ pub(super) fn falcon_hal<E: FalconEngine + 'static>(
     use Chipset::*;
 
     let hal = match chipset {
-        TU102 | TU104 | TU106 | TU116 | TU117 => {

drivers/gpu/nova-core/falcon/hal.rs pub(super) fn falcon_hal<E: FalconEngine + 'static>( · line 88

Quoted 10 lines
drivers/gpu/nova-core/falcon/hal.rs +1 -1
+        TU102 | TU104 | TU106 | TU116 | TU117 | GA100 => {
             KBox::new(tu102::Tu102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
         }
         GA102 | GA103 | GA104 | GA106 | GA107 | AD102 | AD103 | AD104 | AD106 | AD107 => {
             KBox::new(ga102::Ga102::<E>::new(), GFP_KERNEL)? as KBox<dyn FalconHal<E>>
         }
-        _ => return Err(ENOTSUPP),
     };
 
     Ok(hal)