PCI rescan : can't assign mem
I have a board with an PCI Express-to-PCI Bridge (Texas Instruments XIO2001).
Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB memory map each.
The PCI rescan "echo 1 > /sys/bus/pci/rescan" memory assign fails.
dmesg messages:
[ 150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000 [ 150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000 [ 150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000- 0x000fffff 64bit pref] to [bus 01-02] add_size 200000 [ 150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000- 0x05ffffff] to [bus 01-02] add_size 2000000 [ 150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000- 0x000fffff 64bit pref] to [bus 04] add_size 200000 [ 150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment [ 150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff] get_res_add_size add_size 2000000 [ 150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 0x6000000) [ 150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000) [ 150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000) [ 150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 0x4000000) [ 150.979253] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000) [ 150.979260] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000) [ 150.979266] pci 0000:01:00.0: BAR 14: can't assign mem (size 0x4000000) [ 150.979272] pci 0000:02:03.0: BAR 0: can't assign mem (size 0x2000000) [ 150.979277] pci 0000:02:04.0: BAR 0: can't assign mem (size 0x2000000) [ 150.979281] pci 0000:01:00.0: PCI bridge to [bus 02]
lspci:
00:00.0 Host bridge: Intel Corporation ValleyView SSA-CUnit (rev 11) 00:02.0 VGA compatible controller: Intel Corporation ValleyView Gen7 (rev
11)
00:1a.0 Encryption controller: Intel Corporation ValleyView SEC (rev 11) 00:1b.0 Audio device: Intel Corporation ValleyView High Definition Audio Controller (rev 11)
00:1c.0 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1c.1 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1c.2 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1d.0 USB controller: Intel Corporation ValleyView USB Enhanced Host Controller (rev 11)
00:1f.0 ISA bridge: Intel Corporation ValleyView Power Control Unit (rev 11) 00:1f.3 SMBus: Intel Corporation ValleyView SMBus Controller (rev 11) 01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge 02:03.0 Bridge: ONA Electroerosion PCI32TLITE FILSTRUP1 PCI to VME Bridge Controller (rev 04)
02:04.0 Communication controller: ONA Electroerosion Device 000a 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 05) 04:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6101/6102 single- port PATA133 interface (rev c0)
lspci -vt
-[0000:00]-+-00.0 Intel Corporation ValleyView SSA-CUnit
+-02.0 Intel Corporation ValleyView Gen7
+-1a.0 Intel Corporation ValleyView SEC
+-1b.0 Intel Corporation ValleyView High Definition Audio
Controller
+-1c.0-[01-02]----00.0-[02]--+-03.0 ONA Electroerosion
PCI32TLITE FILSTRUP1 PCI to VME Bridge Controller
| \-04.0 ONA Electroerosion Device
000a
+-1c.1-[03]----00.0 Realtek Semiconductor Co., Ltd.
RTL8101E/RTL8102E PCI Express Fast Ethernet controller
+-1c.2-[04]----00.0 Marvell Technology Group Ltd. 88SE6101/6102
single-port PATA133 interface
+-1d.0 Intel Corporation ValleyView USB Enhanced Host Controller
+-1f.0 Intel Corporation ValleyView Power Control Unit
\-1f.3 Intel Corporation ValleyView SMBus ControllerLinux: Ubuntu 14.04
Kernel: 3.13.0-62-generic
Is possible to fix it?
Quoted 32 lines · Peio
I have a board with an PCI Express-to-PCI Bridge (Texas Instruments XIO2001).
Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB memory map each.
The PCI rescan "echo 1 >/sys/bus/pci/rescan" memory assign fails.dmesg messages:
[ 150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000 [ 150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000 [ 150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000- 0x000fffff 64bit pref] to [bus 01-02] add_size 200000 [ 150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000- 0x05ffffff] to [bus 01-02] add_size 2000000 [ 150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000- 0x000fffff 64bit pref] to [bus 04] add_size 200000 [ 150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has bogus alignment [ 150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff] get_res_add_size add_size 2000000 [ 150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 0x6000000) [ 150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000) [ 150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000) [ 150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size
Quoted 4 lines
0x4000000)Linux: Ubuntu 14.04
Kernel: 3.13.0-62-generic
how about current upsteam kernel?
Quoted 1 line
Is possible to fix it?
Please whole post boot log with "debug ignore_loglevel".
Need _CRS from BIOS provide enough MMIO range for 00:1c.0
Yinghai
Quoted 10 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
how about current upsteam kernel?
Quoted 1 line
Is possible to fix it?
Please whole post boot log with "debug
ignore_loglevel".Need
_CRSfrom BIOS provide enough MMIO range for 00:1c.0Yinghai
I prefer the stable kernel used by ubuntu 14.04.
My BIOS don't have MMIO range configuration options.
Log with "debug ignore_loglevel" sentto Yinghai (too long for post here).
Quoted 7 lines · Peio
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
how about current upsteam kernel?
Quoted 1 line
Is possible to fix it?
Quoted 7 lines
Quoted 4 lines
Please whole post boot log with "debug
ignore_loglevel".Need
_CRSfrom BIOS provide enough MMIO range for 00:1c.0I prefer the stable kernel used by ubuntu 14.04.
My BIOS don't have MMIO range configuration options.
Log with "debugignore_loglevel" sentto Yinghai (too long for post here).
root bus resource:
[ 0.277955] PCI host bridge to bus 0000:00 [ 0.277966] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277975] pci_bus 0000:00: root bus resource [io 0x0000-0x006f] [ 0.277983] pci_bus 0000:00: root bus resource [io 0x0078-0x0cf7] [ 0.277991] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff] [ 0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000fffff] [ 0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-0x909ffffe]
and 00:02.0 use almost all
[ 0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff] [ 0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff pref]
so we have no chance to allocate MMIO to 00:1c.0
[ 0.359994] pci 0000:00:1c.0: bridge window [io 0x1000-0x0fff] to [bus 01-02] add_size 1000 [ 0.360009] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01-02] add_size 200000 [ 0.360022] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01-02] add_size 200000 [ 0.360045] pci 0000:00:1c.2: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 [ 0.360061] pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000 [ 0.360072] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 0.360084] pci 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000 [ 0.360095] pci 0000:00:1c.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 [ 0.360111] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000) [ 0.360123] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000) [ 0.360134] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000) [ 0.360146] pci 0000:00:1c.0: BAR 13: assigned [io 0x4000-0x4fff] [ 0.360158] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000) [ 0.360169] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000) [ 0.360180] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000) [ 0.360189] pci 0000:01:00.0: PCI bridge to [bus 02] [ 0.360214] pci 0000:00:1c.0: PCI bridge to [bus 01-02] [ 0.360223] pci 0000:00:1c.0: bridge window [io 0x4000-0x4fff]
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
Quoted 1 line · On Tue, Sep 1, 2015 at 2:01 AM, Peio <peio <at> azkarate.eu> wrote:
Yinghai Lu <yinghai <at> kernel.org> writes:
Quoted 8 lines
root bus resource:
[ 0.277955] PCI host bridge to bus 0000:00 [ 0.277966] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277975] pci_bus 0000:00: root bus resource [io 0x0000-0x006f] [ 0.277983] pci_bus 0000:00: root bus resource [io 0x0078-0x0cf7] [ 0.277991] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-
0x000bffff]
Quoted 1 line
[ 0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-
0x000dffff]
Quoted 1 line
[ 0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-
0x000fffff]
Quoted 1 line
[ 0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-
0x909ffffe]
Quoted 5 lines
and 00:02.0 use almost all
[ 0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff] [ 0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff
pref]
Quoted 3 lines
so we have no chance to allocate MMIO to 00:1c.0
Any workaround?.
Is the BIOS responsible for allocating memory resources in Linux?. Perhaps changing some BIOS parameter ACPI related?.
If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.
Quoted 11 lines · Peio
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
and 00:02.0 use almost all
[ 0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff] [ 0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffffpref]
Quoted 3 lines
so we have no chance to allocate MMIO to 00:1c.0
Any workaround?.
boot with pci=nocrs ?
Quoted 2 lines
Is the BIOS responsible for allocating memory resources in Linux?.
The resource range for root bus. As BIOS may need to set other registers to make them effective.
Quoted 1 line
Perhaps changing some BIOS parameter ACPI related?.
Try to get BIOS update.
Quoted 2 lines
If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.
should not.
Quoted 8 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
boot with pci=nocrs ?
Try to get BIOS update.
Quoted 2 lines
If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.
should not.
I'll try pci=nocrs and BIOS update.
Thank you Yinghai.
Re: PCI rescan : can't assign mem [SOLVED]
Quoted 8 lines · Peio <peio <at> azkarate.eu> writes:
Quoted 5 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
boot with pci=nocrs ?
Try to get BIOS update.
Quoted 1 line
If I change BIOS parameters I'll need a fresh install or the actual
kernel
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
reconigzes changes?.
should not.
I'll try pci=nocrs and BIOS update.
Thank you Yinghai.
I solved the issue.
1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci root bus).
2.- Remove upstream bridge :
sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove
3.- Rescan PCI bus
sudo echo 1 > /sys/bus/pci/rescan
Bjorn Helgaas
maintainerRe: PCI rescan : can't assign mem [SOLVED]
Quoted 21 lines · Peio
Quoted 8 lines · Peio <peio <at> azkarate.eu> writes:
Quoted 5 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
boot with pci=nocrs ?
Try to get BIOS update.
Quoted 1 line
If I change BIOS parameters I'll need a fresh install or the actual
kernel
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
reconigzes changes?.
should not.
I'll try pci=nocrs and BIOS update.
Thank you Yinghai.I solved the issue.
Quoted 8 lines
1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci root bus).
2.- Remove upstream bridge :
sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove3.- Rescan PCI bus
sudo echo 1 >/sys/bus/pci/rescan
This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.
I haven't looked at the problem in detail, but if the BIOS gives us a _CRS with host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.
Bjorn
Re: PCI rescan : can't assign mem [SOLVED]
Quoted 11 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:
This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.
I haven't looked at the problem in detail, but if the BIOS gives us a
_CRSwith host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.Bjorn
I'm sure itsn't a platform limitation, a BIOS issue assigning resources instead.
Anyway, It's a closed system, without hot-add or not own devices.
I suspect this workaround is fine to me.
Bjorn Helgaas
maintainerRe: PCI rescan : can't assign mem [SOLVED]
Quoted 11 lines · Peio
Quoted 9 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:
This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.
I haven't looked at the problem in detail, but if the BIOS gives us a
_CRSwith host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.
Quoted 4 lines
Quoted 2 lines
Bjorn
I'm sure itsn't a platform limitation, a BIOS issue assigning resources instead.
Yep. I include the BIOS in "the platform" because it's something we have no control over.
Quoted 3 lines
Anyway, It's a closed system, without hot-add or not own devices.
I suspect this workaround is fine to me.
OK. I didn't phrase the risk very well. When we use "pci=nocrs", we're ignoring the host bridge apertures the BIOS tells us about. That means Linux may assign resources outside of those apertures to PCI devices. They *might* work, but there's no guarantee that they will. This can affect devices present at boot as well as hot-added ones.
If we were really smart, we might emit a warning when we assigned resources outside of the apertures, but that's probably more work than it's worth.
If the workaround works for you, that's great and you should keep using it. I just don't want to propagate the idea that "pci=nocrs" is
the general solution for anything. Ideally, a user should not have
to use kernel parameters except for debugging or to work around hardware or firmware problems.
Bjorn
Re: PCI rescan : can't assign mem [SOLVED]
Quoted 6 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:
Quoted 2 lines · On Fri, Sep 4, 2015 at 12:32 AM, Peio <peio <at> azkarate.eu> wrote:
Bjorn Helgaas <bhelgaas <at> google.com> writes:
-
Quoted 8 lines
If the workaround works for you, that's great and you should keep using it. I just don't want to propagate the idea that "pci=nocrs" is
the general solution for anything. Ideally, a user should not have
to use kernel parameters except for debugging or to work around hardware or firmware problems.Bjorn
At the moment I'll use the workaround, but a better solution will be, update the BIOS to a version assigning more memory resources to pcie. Thank you for your warning Bjorn.
Peio