SPARC64: getting "no compatible bridge window" errors :/
Hello, very very sorry about the late reply. Life has been hectic. Also, not sure if this is how I reply to one of these, sorry if I screwed it up :)
Quoted 1 line
This is great, thanks a lot for your report! Is this a regression?
Believe it or not, I am a brand new SPARC user :). So I can't say right now. Should I try a few old kernel releases to check?
Quoted 1 line
Any chance you could collect a dmesg log with "ofpci_debug=1"?
https://gitlab.freedesktop.org/drm/amd/uploads/0ed3c92921d7f88b06654b5f46e9756d/dmesg
Quoted 3 lines
Do the devices we complain about (NICs and storage HBAs 09:00.0, 09:00.1, 0d:00.0, 0d:00.1, 0e:00.0, 0f:00.0, 0001:03:00.0, 0001:03:00.1, 0001:0:00.0, 0001:0a:00.1) work?
Well, I don't have any fiber optic equipment: these just came with the server. Also it has wayy too many NICs. I can't quite say. However... for the HBAs, that's where my root is :O. This is mildly concerning :D.
Quoted 6 lines · Richard Rogalski
Hello, very very sorry about the late reply. Life has been hectic. Also, not sure if this is how I reply to one of these, sorry if I screwed it up :)
Quoted 1 line
This is great, thanks a lot for your report! Is this a regression?
Believe it or not, I am a brand new SPARC user :). So I can't say right now. Should I try a few old kernel releases to check?
I wouldn't bother trying older kernels. In fact, I just noticed that you're running a 5.15 kernel, which is about a year old. It would be much more interesting to try to reproduce the problem on a current kernel, e.g., v6.0.
At https://packages.gentoo.org/packages/sys-kernel/gentoo-kernel, it doesn't look like sparc gets much attention ;)
Quoted 4 lines
Quoted 1 line
Any chance you could collect a dmesg log with "ofpci_debug=1"?
https://gitlab.freedesktop.org/drm/amd/uploads/0ed3c92921d7f88b06654b5f46e9756d/dmesg
Quoted 8 lines
Quoted 3 lines
Do the devices we complain about (NICs and storage HBAs 09:00.0, 09:00.1, 0d:00.0, 0d:00.1, 0e:00.0, 0f:00.0, 0001:03:00.0, 0001:03:00.1, 0001:0:00.0, 0001:0a:00.1) work?
Well, I don't have any fiber optic equipment: these just came with the server. Also it has wayy too many NICs. I can't quite say. However... for the HBAs, that's where my root is :O. This is mildly concerning :D.
I spent way too long looking at these PCI resource weirdnesses. Bottom line: ignore them.
From your ofpci_debug dmesg log (annotated with logging the PCI core would do if it were doing this instead of the sparc OF code):
pci@400: PCI MEM [mem 0x84000100000-0x8407f7fffff] offset 84000000000
pci@400: PCI MEM64 [mem 0x84100000000-0x84dffffffff] offset 80000000000
pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff]) pci_bus 0000:00: root bus resource [mem 0x84100000000-0x84dffffffff] (bus address [0x4100000000-0x4dffffffff])
pci 0000:04:00.0: can't claim VGA legacy [mem 0x000a0000-0x000bffff]: no compatible bridge window
This one happens because according to OF, there is no bridge
aperture to the PCI bus 0xa0000-0xbffff region. The only accessible
PCI bus regions are [0x00100000-0x7f7fffff] and
[0x4100000000-0x4dffffffff]. Probably an OF defect. pci 0000:02:0c.0: PCI bridge to [bus 09]
pci 0000:02:0c.0: Using flags[0010220c] start[0000004120000000] size[0000000010000000]
pci 0000:02:0c.0: bridge window [mem 0x84120000000-0x8412fffffff 64bit pref]
pci 0000:09:00.0: can't claim BAR 0 [mem 0x84120000000-0x8412007ffff 64bit]: no compatible bridge window
These and similar warnings happen because OF says the upstream bridge window is prefetchable, but this is a non-prefetchable BAR. These likely work fine because in most cases prefetching will not occur on PCIe, even though the bridge window allows it.
So the warnings above are mostly harmless. If you were to hot-add something, there could be issues because we aren't keeping track of the space these devices use.
lspci on sparc is unusual: it shows PCI bus addresses, not CPU physical addresses like other arches [1], which means we see things like this in dmesg, which shows the CPU physical address:
pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff]) pci 0000:04:00.0: reg 0x10: [mem 0x84000800000-0x84000ffffff]
and this in lspci, which is the PCI bus address:
0000:04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10) (prog-if 00 [VGA controller])
Region 0: Memory at 00800000 (32-bit, non-prefetchable) [size=8M]
Annoying but harmless.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/?id=v5.18#n1