Skip to contentSkip to conversation

BUG: SiFive Unmatched: BAR0 on video card is never initialized

DISCUSSION Yuri Zaporozhets 2 participants 4 messages Nov 19, 2024 – Jul 1

lore
Share & tools
lore
#0 lore raw

Hi Heinrich; hi all owners of SiFive Unmatched + PCIe video card,

Default U-Boot configuration for Unmatched always fails to initialize BAR0 of my video card (NVidia GT710), because the PCI driver selects the MEMORY region at 0x70000000 for bus_start/phys_start ("region 2", according to decode_regions() output), which has the size of 0x1000000. Naturally, the video card's BAR0 doesn't fit into this small region, because its size is also 0x1000000:

pci_auto_config_devices: start
PCI Autoconfig: Bus Memory region: [70000000-70ffffff],

                  Physical Memory [70000000-70ffffff]
  PCI Autoconfig: Bus Prefetchable Mem region: [2000000000-3fffffffff],
                  Physical Memory [2000000000-3fffffffff]
  PCI Autoconfig: Bus I/O region: [60080000-6008ffff],
                  Physical Memory [60080000-6008ffff]

.......
pci_auto_config_devices: device pci_7:0.0
PCI Autoconfig: BAR 0, Mem, size=0x1000000, bus_lower=0x70400000, bus_start=0x70000000, phys_start=0x70000000 addr = 0x71000000
No room in resource, avail start=70400000 / size=1000000, need=1000000
PCI: Failed autoconfig bar 10

The situation changes drastically if CONFIG_PCI_REGION_MULTI_ENTRY is set to "yes". After this, PCI starts to use "region 1" for MEMORY, which has the size of 0xFF70000, and is, obviously, suitable for GPU's BAR0:

pci_uclass_pre_probe, bus=-1/pcie@e00000000, parent=soc
decode_regions: len=28, cells_per_record=7
decode_regions: region 0, pci_addr=60080000, addr=60080000, size=10000, space_code=1
- type=1, pos=0
decode_regions: region 1, pci_addr=60090000, addr=60090000, size=ff70000, space_code=2
- type=0, pos=1
decode_regions: region 2, pci_addr=70000000, addr=70000000, size=1000000, space_code=2
- type=0, pos=2
decode_regions: region 3, pci_addr=2000000000, addr=2000000000, size=2000000000, space_code=3
- type=8, pos=3
....
PCIE-0: Link up (Gen1-x8, Bus0)
pci_uclass_post_probe: probing bus 0
pci_bind_bus_devices: bus 0/pcie@e00000000: found device 0, function 0: find ret=-19
pci_find_and_bind_driver: Searching for driver: vendor=f15e, device=0
pci_find_and_bind_driver: No match found: bound generic driver instead
pci_auto_config_devices: start
PCI Autoconfig: Bus Memory region: [60090000-6fffffff],

                Physical Memory [60090000-6fffffff]
PCI Autoconfig: Bus Prefetchable Mem region: [2000000000-3fffffffff],
                Physical Memory [2000000000-3fffffffff]
PCI Autoconfig: Bus I/O region: [60080000-6008ffff],
                Physical Memory [60080000-6008ffff]

This picture is in full accord with Linux kernel, which also uses 0x69000000 ("region 1") for bus_start and phys_start.

With this setting, all BARs on all devices are initialized correctly, and, moreover, I can access memory-mapped VGA ports of the card (see [1] for details), which is essential for initializing the card in text mode with bios_emulator:

PCI Autoconfig: BAR 0, Mem, size=0x1000000, bus_lower=0x60500000, bus_start=0x60090000, phys_start=0x60090000 address=0x61000000 bus_lower=0x62000000

PCI Autoconfig: BAR 1, Prf64, size=0x8000000, bus_lower=0x2000000000, bus_start=0x2000000000, phys_start=0x2000000000 address=0x2000000000 bus_lower=0x2008000000

PCI Autoconfig: BAR 2, Prf64, size=0x2000000, bus_lower=0x2008000000, bus_start=0x2000000000, phys_start=0x2000000000 address=0x2008000000 bus_lower=0x200a000000

PCI Autoconfig: BAR 3, I/O, size=0x80, bus_lower=0x60080000, bus_start=0x60080000, phys_start=0x60080000 address=0x60080000 bus_lower=0x60080080

PCI Autoconfig: ROM, size=0x80000, bus_lower=0x62000000, bus_start=0x60090000, phys_start=0x60090000 address=0x62000000 bus_lower=0x62080000

Notes:

[1]: https://envytools.readthedocs.io/en/latest/hw/mmio.html#gf100-mmio-map

Signature · 2 lines
Regards,
Yuri
#1 lore raw

Am 19. November 2024 21:45:39 MEZ schrieb Yuri Zaporozhets <[email protected]>:

Quoted 71 lines

Hi Heinrich; hi all owners of SiFive Unmatched + PCIe video card,

Default U-Boot configuration for Unmatched always fails to initialize BAR0 of my video card (NVidia GT710), because the PCI driver selects the MEMORY region at 0x70000000 for bus_start/phys_start ("region 2", according to decode_regions() output), which has the size of 0x1000000. Naturally, the video card's BAR0 doesn't fit into this small region, because its size is also 0x1000000:

pci_auto_config_devices: start
PCI Autoconfig: Bus Memory region: [70000000-70ffffff],

                 Physical Memory [70000000-70ffffff]
 PCI Autoconfig: Bus Prefetchable Mem region: [2000000000-3fffffffff],
                 Physical Memory [2000000000-3fffffffff]
 PCI Autoconfig: Bus I/O region: [60080000-6008ffff],
                 Physical Memory [60080000-6008ffff]

.......
pci_auto_config_devices: device pci_7:0.0
PCI Autoconfig: BAR 0, Mem, size=0x1000000, bus_lower=0x70400000, bus_start=0x70000000, phys_start=0x70000000 addr = 0x71000000
No room in resource, avail start=70400000 / size=1000000, need=1000000
PCI: Failed autoconfig bar 10

The situation changes drastically if CONFIG_PCI_REGION_MULTI_ENTRY is set to "yes". After this, PCI starts to use "region 1" for MEMORY, which has the size of 0xFF70000, and is, obviously, suitable for GPU's BAR0:

pci_uclass_pre_probe, bus=-1/pcie@e00000000, parent=soc
decode_regions: len=28, cells_per_record=7
decode_regions: region 0, pci_addr=60080000, addr=60080000, size=10000, space_code=1
- type=1, pos=0
decode_regions: region 1, pci_addr=60090000, addr=60090000, size=ff70000, space_code=2
- type=0, pos=1
decode_regions: region 2, pci_addr=70000000, addr=70000000, size=1000000, space_code=2
- type=0, pos=2
decode_regions: region 3, pci_addr=2000000000, addr=2000000000, size=2000000000, space_code=3
- type=8, pos=3
....
PCIE-0: Link up (Gen1-x8, Bus0)
pci_uclass_post_probe: probing bus 0
pci_bind_bus_devices: bus 0/pcie@e00000000: found device 0, function 0: find ret=-19
pci_find_and_bind_driver: Searching for driver: vendor=f15e, device=0
pci_find_and_bind_driver: No match found: bound generic driver instead
pci_auto_config_devices: start
PCI Autoconfig: Bus Memory region: [60090000-6fffffff],

               Physical Memory [60090000-6fffffff]
PCI Autoconfig: Bus Prefetchable Mem region: [2000000000-3fffffffff],
               Physical Memory [2000000000-3fffffffff]
PCI Autoconfig: Bus I/O region: [60080000-6008ffff],
               Physical Memory [60080000-6008ffff]

This picture is in full accord with Linux kernel, which also uses 0x69000000 ("region 1") for bus_start and phys_start.

With this setting, all BARs on all devices are initialized correctly, and, moreover, I can access memory-mapped VGA ports of the card (see [1] for details), which is essential for initializing the card in text mode with bios_emulator:

PCI Autoconfig: BAR 0, Mem, size=0x1000000, bus_lower=0x60500000, bus_start=0x60090000, phys_start=0x60090000 address=0x61000000 bus_lower=0x62000000

PCI Autoconfig: BAR 1, Prf64, size=0x8000000, bus_lower=0x2000000000, bus_start=0x2000000000, phys_start=0x2000000000 address=0x2000000000 bus_lower=0x2008000000

PCI Autoconfig: BAR 2, Prf64, size=0x2000000, bus_lower=0x2008000000, bus_start=0x2000000000, phys_start=0x2000000000 address=0x2008000000 bus_lower=0x200a000000

PCI Autoconfig: BAR 3, I/O, size=0x80, bus_lower=0x60080000, bus_start=0x60080000, phys_start=0x60080000 address=0x60080000 bus_lower=0x60080080

Quoted 7 lines

PCI Autoconfig: ROM, size=0x80000, bus_lower=0x62000000, bus_start=0x60090000, phys_start=0x60090000 address=0x62000000 bus_lower=0x62080000

Notes:

[1]: https://envytools.readthedocs.io/en/latest/hw/mmio.html#gf100-mmio-map

Hello Yuri,

Thank you for sharing your investigation results. Could you, please, send a patch?

Were you able to show output on HDMI?

Best regards

Heinrich

#2 lore raw

Hi Heinrich,

Quoted 4 lines · Heinrich Schuchardt

Hello Yuri,

Thank you for sharing your investigation results. Could you, please, send a patch?

Sure. Already did.

Quoted 1 line

Were you able to show output on HDMI?

Oh, not that fast -- I'm working on it :-) . Currently, I have my 1992 Hitachi CRT monitor hooked to GT710's VGA output, and I'm very much determined to get the text mode ("VGA mode 3") working on it -- it's my favorite mode of OS kernel development. After that, perhaps, I'll try HDMI too ;-)

Signature · 2 lines
Regards,
Yuri
#3 lore raw

RISC-V (Unmatched): NVidia GK-208 card successfully initialized by U-Boot

In reply to Heinrich Schuchardt

Hi Heinrich, hi U-Boot RISC-V community,

Quoted 5 lines · Heinrich Schuchardt

Hello Yuri,

[...]

Were you able to show output on HDMI?

Yes :-)

(a slightly longer answer: but I could not get the x86 bios_emulator working with my card, despite all the efforts. So I went a completely different way)

I'm creating a blog series (8 posts) about my findings and about the end result. Can be seen at: https://r-tty.blogspot.com . Today I will also record a demonstration video.

To all SiFive Unmatched users (and also the users of PolarFire Icicle Kit!), who use the board with the PCIe video controller: please write me if you're interested. I'll be opening the sources of my "VideoBIOS" program soon (GPL2-licensed), after I finish some architectural work about QSOE's video console. The project will be available at: https://gitlab.com/bios-riscv

Signature · 2 lines
Regards,
Yuri