Resizable BARs vs bridges with BARs
RFC 1/1 Resizable BARs vs bridges with BARs
Hi,
I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR:
pci_bus 0000:5d: root bus resource [io 0x8000-0x9fff window] pci_bus 0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window] pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<< pci_bus 0000:5d: root bus resource [bus 5d-7f]
But resizing fails with -ENOSPC. The topology looks like this:
+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380]
\-04.0-[61]----00.0 Intel Corporation Device 4f92The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: b9000000-ba0fffff [size=17M]
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]
Kernel driver in use: pcieportThe trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
Quoted 1 line
Quoted 1 line
Quoted 1 line
Region 0: Memory at
b010000000(64-bit, prefetchable)
Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: b9000000-ba0fffff [size=17M]
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]
Kernel driver in use: pcieportNote region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M]
Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M]
Expansion ROM at <ignored> [disabled]
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GBIt's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device. In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a pci_request_region() or equivalent by looking for the IORESOURCE_BUSY flag, but I also suspect this is potentially racy.
The patch below works for me, allowing the new resourceN_resize sysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks,
Alex
drivers/pci/setup-bus.c +7 -1
@@ -2137,13 +2137,19 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type) next = bridge; do { bridge = next; - for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; + for (i = PCI_STD_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; i++) { struct resource *res = &bridge->resource[i]; if ((res->flags ^ type) & PCI_RES_TYPE_MASK) continue; + if (i < PCI_STD_NUM_BARS) { + if (!(res->flags & IORESOURCE_BUSY)) + pci_release_resource(bridge, i); + continue; + } + /* Ignore BARs which are still in use */ if (res->child) continue;
Hi Alex,
Am 19.11.22 um 00:09 schrieb Alex Williamson:
Quoted 45 lines
Hi,
I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR:
pci_bus0000:5d: root bus resource [io 0x8000-0x9fff window]pci_bus0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]pci_bus0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<<pci_bus0000:5d: root bus resource [bus 5d-7f]But resizing fails with
-ENOSPC. The topology looks like this:+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380] \-04.0-[61]----00.0 Intel Corporation Device 4f92The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M] Kernel driver in use: pcieportThe trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
Quoted 1 line
Quoted 1 line
Quoted 1 line
Region 0: Memory at
b010000000(64-bit, prefetchable)Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Kernel driver in use: pcieportNote region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M] Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M] Expansion ROM at <ignored> [disabled] Capabilities: [420 v1] Physical Resizable BAR BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
Quoted 3 lines
It's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device.
This is expected. Bridges always have a 32bit non prefetchable and a 64bit prefetchable BAR. This is part of the PCI(e) spec.
Quoted 4 lines
In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a
pci_request_region()or equivalent by looking for theIORESOURCE_BUSYflag, but I also suspect this is potentially racy.
That sounds like we have a misunderstanding here how those bridges work. The upstream bridges should include all the resources of the downstream devices/bridges in their BARs.
Quoted 4 lines
The patch below works for me, allowing the new
resourceN_resizesysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks,
The correct approach is to remove all the drivers (EFI, vesafb etc...) which are using the PCI(e) devices under the bridge in question. Then release the resources and puzzle everything back together.
See amdgpu_device_resize_fb_bar() how to do this correctly.
Regards,
Christian.
Quoted 29 lines
Alex
drivers/pci/setup-bus.c +7
-1@@ -2137,13 +2137,19 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type) next = bridge; do { bridge = next; - for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; + for (i = PCI_STD_RESOURCES; i < PCI_BRIDGE_RESOURCE_END; i++) { struct resource *res = &bridge->resource[i]; if ((res->flags ^ type) & PCI_RES_TYPE_MASK) continue; + if (i < PCI_STD_NUM_BARS) { + if (!(res->flags & IORESOURCE_BUSY)) + pci_release_resource(bridge, i); + continue; + } + /* Ignore BARs which are still in use */ if (res->child) continue;
Hi Christian,
Quoted 48 lines · On Sat, 19 Nov 2022 12:02:55 +0100 Christian König <[email protected]>…
Hi Alex,
Am 19.11.22 um 00:09 schrieb Alex Williamson:
Quoted 45 lines
Hi,
I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR:
pci_bus0000:5d: root bus resource [io 0x8000-0x9fff window]pci_bus0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]pci_bus0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<<pci_bus0000:5d: root bus resource [bus 5d-7f]But resizing fails with
-ENOSPC. The topology looks like this:+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380] \-04.0-[61]----00.0 Intel Corporation Device 4f92The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M] Kernel driver in use: pcieportThe trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
Quoted 1 line
Quoted 1 line
Quoted 1 line
Region 0: Memory at
b010000000(64-bit, prefetchable)Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Kernel driver in use: pcieportNote region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M] Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M] Expansion ROM at <ignored> [disabled] Capabilities: [420 v1] Physical Resizable BAR BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
Quoted 6 lines
Quoted 3 lines
It's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device.
This is expected. Bridges always have a 32bit non prefetchable and a 64bit prefetchable BAR. This is part of the PCI(e) spec.
To be clear, the issue is a bridge implementing a 64-bit, prefetchable BAR at config offset 0x10 & 0x14, not the limit/base registers that define the bridge windows for prefetchable and non-prefetchable downstream resources.
Quoted 8 lines
Quoted 4 lines
In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a
pci_request_region()or equivalent by looking for theIORESOURCE_BUSYflag, but I also suspect this is potentially racy.That sounds like we have a misunderstanding here how those bridges work. The upstream bridges should include all the resources of the downstream devices/bridges in their BARs.
Correct, and the issue is that the bridge at 5e:00.0 _consumes_ a portion of the window we need to resize at the root port.
Root port:
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]
Upstream switch port:
Region 0: Memory at b010000000 (64-bit, prefetchable)
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]
It's that Region 0 resource that prevents resizing.
Quoted 10 lines
Quoted 4 lines
The patch below works for me, allowing the new
resourceN_resizesysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks,The correct approach is to remove all the drivers (EFI, vesafb etc...) which are using the PCI(e) devices under the bridge in question. Then release the resources and puzzle everything back together.
See
amdgpu_device_resize_fb_bar()how to do this correctly.
Resource resizing in pci-sysfs is largely modeled after the amdgpu code, but I don't see any special provisions for handling conflicting resources consumed on intermediate devices. The driver attached to the upstream switch port is pcieport and removing it doesn't resolve the problem. The necessary resource on the root port still reports a child.
Is amdgppu resizing known to work in cases where the GPU is downstream of a PCIe switch that consumes resources of the same type and the root port aperture needs to be resized? I suspect it does not. Thanks,
Alex
Hi Alex,
Am 19.11.22 um 15:07 schrieb Alex Williamson:
Quoted 56 lines
Hi Christian,
Quoted 51 lines · On Sat, 19 Nov 2022 12:02:55 +0100 Christian König <[email protected]>…
Hi Alex,
Am 19.11.22 um 00:09 schrieb Alex Williamson:
Quoted 48 lines
Hi,
I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR:
pci_bus0000:5d: root bus resource [io 0x8000-0x9fff window]pci_bus0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]pci_bus0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<<pci_bus0000:5d: root bus resource [bus 5d-7f]But resizing fails with
-ENOSPC. The topology looks like this:+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380] \-04.0-[61]----00.0 Intel Corporation Device 4f92The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M] Kernel driver in use: pcieportThe trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
>>> Region 0: Memory at b010000000 (64-bit, prefetchable) Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Kernel driver in use: pcieportNote region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M] Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M] Expansion ROM at <ignored> [disabled] Capabilities: [420 v1] Physical Resizable BAR BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GBIt's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device.
Quoted 6 lines
Quoted 2 lines
This is expected. Bridges always have a 32bit non prefetchable and a 64bit prefetchable BAR. This is part of the PCI(e) spec.
To be clear, the issue is a bridge implementing a 64-bit, prefetchable BAR at config offset
0x10&0x14, not the limit/base registers that define the bridge windows for prefetchable and non-prefetchable downstream resources.
WHAT? I've never heard of a bridge with this configuration. I don't fully remember the spec, but I'm pretty sure that this isn't something standard.
Can you give me the output of "sudo lspci -vvvv -s $busID" for this device.
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a
pci_request_region()or equivalent by looking for theIORESOURCE_BUSYflag, but I also suspect this is potentially racy.That sounds like we have a misunderstanding here how those bridges work. The upstream bridges should include all the resources of the downstream devices/bridges in their BARs.
Correct, and the issue is that the bridge at 5e:00.0 _consumes_ a portion of the window we need to resize at the root port.
Quoted 8 lines
Root port:
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]Upstream switch port:
Region 0: Memory atb010000000(64-bit, prefetchable)
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]It's that Region 0 resource that prevents resizing.
Could it be that some of the ACPI tables are broken and because of this we add a fixed resource to this device?
Otherwise I have a hard time coming up with a way for a bridge to have a BAR in the config space.
Quoted 16 lines
Quoted 9 lines
Quoted 4 lines
The patch below works for me, allowing the new
resourceN_resizesysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks,The correct approach is to remove all the drivers (EFI, vesafb etc...) which are using the PCI(e) devices under the bridge in question. Then release the resources and puzzle everything back together.
See
amdgpu_device_resize_fb_bar()how to do this correctly.Resource resizing in pci-sysfs is largely modeled after the amdgpu code, but I don't see any special provisions for handling conflicting resources consumed on intermediate devices. The driver attached to the upstream switch port is pcieport and removing it doesn't resolve the problem. The necessary resource on the root port still reports a child.
Quoted 3 lines
Is amdgppu resizing known to work in cases where the GPU is downstream of a PCIe switch that consumes resources of the same type and the root port aperture needs to be resized? I suspect it does not. Thanks,
Well we have the possibility to add extra space to bridges on the kernel command line for this.
This is used for things like hotplug behind bridges with limited address space.
Quite a while ago there was also a patch set which dynamically binds/unbinds drivers from resources to resize the BARs. But that never got far because of locking problems.
Regards,
Christian.
Quoted 3 lines
Alex
Hi Christian,
Quoted 52 lines · On Sat, 19 Nov 2022 20:14:15 +0100 Christian König <[email protected]>…
Am 19.11.22 um 15:07 schrieb Alex Williamson:
Quoted 51 lines
Quoted 49 lines · On Sat, 19 Nov 2022 12:02:55 +0100 Christian König <[email protected]>…
Am 19.11.22 um 00:09 schrieb Alex Williamson:
Quoted 43 lines
I'm trying to get resizable BARs working in a configuration where my root bus resources provide plenty of aperture for the BAR:
pci_bus0000:5d: root bus resource [io 0x8000-0x9fff window]pci_bus0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]pci_bus0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<<pci_bus0000:5d: root bus resource [bus 5d-7f]But resizing fails with
-ENOSPC. The topology looks like this:+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380] \-04.0-[61]----00.0 Intel Corporation Device 4f92The BIOS is not fluent in resizable BARs and only programs the root port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M] Kernel driver in use: pcieportThe trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
>>> Region 0: Memory at b010000000 (64-bit, prefetchable) Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b9000000-ba0fffff [size=17M] Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Kernel driver in use: pcieportNote region 0 of this bridge, which is 64-bit, prefetchable and therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M] Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M] Expansion ROM at <ignored> [disabled] Capabilities: [420 v1] Physical Resizable BAR BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GBQuoted 3 lines
It's a shame that the hardware designers didn't mark the upstream port BAR as non-prefetchable to avoid it living in the same resource aperture as the resizable BAR on the downstream device.
This is expected. Bridges always have a 32bit non prefetchable and a 64bit prefetchable BAR. This is part of the PCI(e) spec.
Quoted 8 lines
Quoted 4 lines
To be clear, the issue is a bridge implementing a 64-bit, prefetchable BAR at config offset
0x10&0x14, not the limit/base registers that define the bridge windows for prefetchable and non-prefetchable downstream resources.WHAT? I've never heard of a bridge with this configuration. I don't fully remember the spec, but I'm pretty sure that this isn't something standard.
Type1 config space allows for two standard BARs.
Quoted 1 line
Can you give me the output of "sudo lspci -vvvv -s $busID" for this device.
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 42
NUMA node: 0
IOMMU group: 1
Region 0: Memory at bff0000000 (64-bit, prefetchable) [size=8M]
Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: b9000000-ba0fffff [size=17M]
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP+ LTR+
10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS+
AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: EgressBlck+
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS+
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: Upstream Port
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Power Budgeting <?>
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [1a0 v1] Lane Margining at the Receiver <?>
Capabilities: [1d4 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1dc v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=14us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [1f8 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2f8 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [330 v1] Data Link Feature <?>
Kernel driver in use: pcieport
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 4 lines
In any case, it's my understanding that our bridge drivers don't generally make use of bridge BARs. I think we can test whether a driver has done a
pci_request_region()or equivalent by looking for theIORESOURCE_BUSYflag, but I also suspect this is potentially racy.That sounds like we have a misunderstanding here how those bridges work. The upstream bridges should include all the resources of the downstream devices/bridges in their BARs.
Correct, and the issue is that the bridge at 5e:00.0 _consumes_ a portion of the window we need to resize at the root port.
Root port:
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]
Quoted 8 lines
Quoted 5 lines
Upstream switch port:
Region 0: Memory atb010000000(64-bit, prefetchable)
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]It's that Region 0 resource that prevents resizing.
Could it be that some of the ACPI tables are broken and because of this we add a fixed resource to this device?
The switch is part of a plug-in card, I'd not expect ACPI to be involved. It's just a standard BAR:
# setpci -s 5e:00.0 BASE_ADDRESS_0 f000000c # setpci -s 5e:00.0 BASE_ADDRESS_1 000000bf # setpci -s 5e:00.0 BASE_ADDRESS_0=ffffffff # setpci -s 5e:00.0 BASE_ADDRESS_1=ffffffff # setpci -s 5e:00.0 BASE_ADDRESS_0 ff80000c # setpci -s 5e:00.0 BASE_ADDRESS_1 ffffffff
All this would have transparently worked if they would have chosen to implement a non-prefetchable BAR.
Quoted 2 lines
Otherwise I have a hard time coming up with a way for a bridge to have a BAR in the config space.
It's a standard part of the Type1 config header.
Quoted 20 lines
Quoted 16 lines
Quoted 9 lines
Quoted 4 lines
The patch below works for me, allowing the new
resourceN_resizesysfs attribute to resize the root port window within the provided bus window. Is this the right answer? How can we make it feel less sketchy? Thanks,The correct approach is to remove all the drivers (EFI, vesafb etc...) which are using the PCI(e) devices under the bridge in question. Then release the resources and puzzle everything back together.
See
amdgpu_device_resize_fb_bar()how to do this correctly.Resource resizing in pci-sysfs is largely modeled after the amdgpu code, but I don't see any special provisions for handling conflicting resources consumed on intermediate devices. The driver attached to the upstream switch port is pcieport and removing it doesn't resolve the problem. The necessary resource on the root port still reports a child.
Quoted 3 lines
Is amdgppu resizing known to work in cases where the GPU is downstream of a PCIe switch that consumes resources of the same type and the root port aperture needs to be resized? I suspect it does not. Thanks,
Quoted 5 lines
Well we have the possibility to add extra space to bridges on the kernel command line for this.
This is used for things like hotplug behind bridges with limited address space.
AFAIK, this is only for hotplug slots, my root port is HotPlug-.
I'd also like to make pci=realloc aware of resizable BARs, but it hits the same problem. Thanks,
Alex