Skip to contentSkip to conversation

PCI rescan : can't assign mem

DISCUSSION Peio 3 participants 12 messages Aug 31, 2015 – Sep 7, 2015 PCI

lore
Share & tools
lore

Peio

author
#0 lore raw

I have a board with an PCI Express-to-PCI Bridge (Texas Instruments XIO2001).
Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB memory map each.
The PCI rescan "echo 1 > /sys/bus/pci/rescan" memory assign fails.

dmesg messages:

[  150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000
[  150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000
[  150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 01-02] add_size 200000
[  150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000-
0x05ffffff] to [bus 01-02] add_size 2000000
[  150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 04] add_size 200000
[  150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has 
bogus alignment
[  150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff] 
get_res_add_size add_size 2000000
[  150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 
64bit pref] get_res_add_size add_size 200000
[  150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 
64bit pref] get_res_add_size add_size 200000
[  150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 
0x6000000)
[  150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 
0x4000000)
[  150.979253] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979260] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979266] pci 0000:01:00.0: BAR 14: can't assign mem (size 0x4000000)
[  150.979272] pci 0000:02:03.0: BAR 0: can't assign mem (size 0x2000000)
[  150.979277] pci 0000:02:04.0: BAR 0: can't assign mem (size 0x2000000)
[  150.979281] pci 0000:01:00.0: PCI bridge to [bus 02]

lspci:
00:00.0 Host bridge: Intel Corporation ValleyView SSA-CUnit (rev 11) 00:02.0 VGA compatible controller: Intel Corporation ValleyView Gen7 (rev
11)
00:1a.0 Encryption controller: Intel Corporation ValleyView SEC (rev 11) 00:1b.0 Audio device: Intel Corporation ValleyView High Definition Audio Controller (rev 11)
00:1c.0 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1c.1 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1c.2 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev
11)
00:1d.0 USB controller: Intel Corporation ValleyView USB Enhanced Host Controller (rev 11)
00:1f.0 ISA bridge: Intel Corporation ValleyView Power Control Unit (rev 11) 00:1f.3 SMBus: Intel Corporation ValleyView SMBus Controller (rev 11) 01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge 02:03.0 Bridge: ONA Electroerosion PCI32TLITE FILSTRUP1 PCI to VME Bridge Controller (rev 04)
02:04.0 Communication controller: ONA Electroerosion Device 000a 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 05) 04:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6101/6102 single- port PATA133 interface (rev c0)

lspci -vt

-[0000:00]-+-00.0  Intel Corporation ValleyView SSA-CUnit
           +-02.0  Intel Corporation ValleyView Gen7
           +-1a.0  Intel Corporation ValleyView SEC
           +-1b.0  Intel Corporation ValleyView High Definition Audio 
Controller
           +-1c.0-[01-02]----00.0-[02]--+-03.0  ONA Electroerosion 
PCI32TLITE FILSTRUP1 PCI to VME Bridge Controller
           |                            \-04.0  ONA Electroerosion Device 
000a
           +-1c.1-[03]----00.0  Realtek Semiconductor Co., Ltd. 
RTL8101E/RTL8102E PCI Express Fast Ethernet controller
           +-1c.2-[04]----00.0  Marvell Technology Group Ltd. 88SE6101/6102 
single-port PATA133 interface
           +-1d.0  Intel Corporation ValleyView USB Enhanced Host Controller
           +-1f.0  Intel Corporation ValleyView Power Control Unit
           \-1f.3  Intel Corporation ValleyView SMBus Controller

Linux: Ubuntu 14.04
Kernel: 3.13.0-62-generic

Is possible to fix it?

#1 lore raw
Quoted 32 lines · Peio

I have a board with an PCI Express-to-PCI Bridge (Texas Instruments XIO2001).
Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB memory map each.
The PCI rescan "echo 1 > /sys/bus/pci/rescan" memory assign fails.

dmesg messages:

[  150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000
[  150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000
[  150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 01-02] add_size 200000
[  150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000-
0x05ffffff] to [bus 01-02] add_size 2000000
[  150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 04] add_size 200000
[  150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has
bogus alignment
[  150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff]
get_res_add_size add_size 2000000
[  150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[  150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[  150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size
0x6000000)
[  150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size
0x200000)
[  150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size
0x200000)
[  150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size
Quoted 4 lines

0x4000000)

Linux: Ubuntu 14.04
Kernel: 3.13.0-62-generic

how about current upsteam kernel?

Quoted 1 line

Is possible to fix it?

Please whole post boot log with "debug ignore_loglevel".

Need _CRS from BIOS provide enough MMIO range for 00:1c.0

Yinghai

author
#2 lore raw
Quoted 10 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

how about current upsteam kernel?

Quoted 1 line

Is possible to fix it?

Please whole post boot log with "debug ignore_loglevel".

Need _CRS from BIOS provide enough MMIO range for 00:1c.0

Yinghai

I prefer the stable kernel used by ubuntu 14.04.
My BIOS don't have MMIO range configuration options.
Log with "debug ignore_loglevel" sentto Yinghai (too long for post here).

#3 lore raw
Quoted 7 lines · Peio
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

how about current upsteam kernel?

Quoted 1 line

Is possible to fix it?

Quoted 7 lines
Quoted 4 lines

Please whole post boot log with "debug ignore_loglevel".

Need _CRS from BIOS provide enough MMIO range for 00:1c.0

I prefer the stable kernel used by ubuntu 14.04.
My BIOS don't have MMIO range configuration options.
Log with "debug ignore_loglevel" sentto Yinghai (too long for post here).

root bus resource:

[    0.277955] PCI host bridge to bus 0000:00
[    0.277966] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.277975] pci_bus 0000:00: root bus resource [io  0x0000-0x006f]
[    0.277983] pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7]
[    0.277991] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff]
[    0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000fffff]
[    0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-0x909ffffe]

and 00:02.0 use almost all

[    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
[    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff pref]

so we have no chance to allocate MMIO to 00:1c.0

[    0.359994] pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to
[bus 01-02] add_size 1000
[    0.360009] pci 0000:00:1c.0: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 01-02] add_size 200000
[    0.360022] pci 0000:00:1c.0: bridge window [mem
0x00100000-0x000fffff] to [bus 01-02] add_size 200000
[    0.360045] pci 0000:00:1c.2: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000
[    0.360061] pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x000fffff]
get_res_add_size add_size 200000
[    0.360072] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[    0.360084] pci 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[    0.360095] pci 0000:00:1c.0: res[13]=[io  0x1000-0x0fff]
get_res_add_size add_size 1000
[    0.360111] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000)
[    0.360123] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000)
[    0.360134] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000)
[    0.360146] pci 0000:00:1c.0: BAR 13: assigned [io  0x4000-0x4fff]
[    0.360158] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000)
[    0.360169] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000)
[    0.360180] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000)
[    0.360189] pci 0000:01:00.0: PCI bridge to [bus 02]
[    0.360214] pci 0000:00:1c.0: PCI bridge to [bus 01-02]
[    0.360223] pci 0000:00:1c.0:   bridge window [io  0x4000-0x4fff]
author
#4 lore raw
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:
Quoted 1 line · On Tue, Sep 1, 2015 at 2:01 AM, Peio <peio <at> azkarate.eu> wrote:

Yinghai Lu <yinghai <at> kernel.org> writes:

Quoted 8 lines

root bus resource:

[    0.277955] PCI host bridge to bus 0000:00
[    0.277966] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.277975] pci_bus 0000:00: root bus resource [io  0x0000-0x006f]
[    0.277983] pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7]
[    0.277991] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-

0x000bffff]

Quoted 1 line
[    0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-

0x000dffff]

Quoted 1 line
[    0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-

0x000fffff]

Quoted 1 line
[    0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-

0x909ffffe]

Quoted 5 lines

and 00:02.0 use almost all

[    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
[    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff 

pref]

Quoted 3 lines

so we have no chance to allocate MMIO to 00:1c.0

Any workaround?.

Is the BIOS responsible for allocating memory resources in Linux?. Perhaps changing some BIOS parameter ACPI related?.
If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.

#5 lore raw
Quoted 11 lines · Peio
Quoted 4 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

and 00:02.0 use almost all

[    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
[    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff

pref]

Quoted 3 lines

so we have no chance to allocate MMIO to 00:1c.0

Any workaround?.

boot with pci=nocrs ?

Quoted 2 lines

Is the BIOS responsible for allocating memory resources in Linux?.

The resource range for root bus. As BIOS may need to set other registers to make them effective.

Quoted 1 line

Perhaps changing some BIOS parameter ACPI related?.

Try to get BIOS update.

Quoted 2 lines

If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.

should not.

author
#6 lore raw
Quoted 8 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

boot with pci=nocrs ?

Try to get BIOS update.

Quoted 2 lines

If I change BIOS parameters I'll need a fresh install or the actual kernel reconigzes changes?.

should not.

I'll try pci=nocrs and BIOS update.
Thank you Yinghai.

author
#7 lore raw

Re: PCI rescan : can't assign mem [SOLVED]

Quoted 8 lines · Peio <peio <at> azkarate.eu> writes:
Quoted 5 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

boot with pci=nocrs ?

Try to get BIOS update.

Quoted 1 line

If I change BIOS parameters I'll need a fresh install or the actual

kernel

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

reconigzes changes?.

should not.

I'll try pci=nocrs and BIOS update.
Thank you Yinghai.

I solved the issue.

1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci root bus).

2.- Remove upstream bridge :
sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove

3.- Rescan PCI bus
sudo echo 1 > /sys/bus/pci/rescan

maintainer
#8 lore raw

Re: PCI rescan : can't assign mem [SOLVED]

Quoted 21 lines · Peio
Quoted 8 lines · Peio <peio <at> azkarate.eu> writes:
Quoted 5 lines · Yinghai Lu <yinghai <at> kernel.org> writes:

boot with pci=nocrs ?

Try to get BIOS update.

Quoted 1 line

If I change BIOS parameters I'll need a fresh install or the actual

kernel

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

reconigzes changes?.

should not.

I'll try pci=nocrs and BIOS update.
Thank you Yinghai.

I solved the issue.

Quoted 8 lines

1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci root bus).

2.- Remove upstream bridge :
sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove

3.- Rescan PCI bus
sudo echo 1 > /sys/bus/pci/rescan

This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.

I haven't looked at the problem in detail, but if the BIOS gives us a _CRS with host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.

Bjorn

author
#9 lore raw

Re: PCI rescan : can't assign mem [SOLVED]

Quoted 11 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:

This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.

I haven't looked at the problem in detail, but if the BIOS gives us a _CRS with host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.

Bjorn

I'm sure itsn't a platform limitation, a BIOS issue assigning resources instead.

Anyway, It's a closed system, without hot-add or not own devices.

I suspect this workaround is fine to me.

maintainer
#10 lore raw

Re: PCI rescan : can't assign mem [SOLVED]

Quoted 11 lines · Peio
Quoted 9 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:

This is a workaround, not a solution. It is not safe to use "pci=nocrs" in general because it can break things related to resource assignment, including hot-add of new devices.

I haven't looked at the problem in detail, but if the BIOS gives us a _CRS with host bridge windows that are too small to accommodate all your devices, there's really nothing the kernel can do about it. You just have to regard it as a limitation of that platform.

Quoted 4 lines
Quoted 2 lines

Bjorn

I'm sure itsn't a platform limitation, a BIOS issue assigning resources instead.

Yep. I include the BIOS in "the platform" because it's something we have no control over.

Quoted 3 lines

Anyway, It's a closed system, without hot-add or not own devices.

I suspect this workaround is fine to me.

OK. I didn't phrase the risk very well. When we use "pci=nocrs", we're ignoring the host bridge apertures the BIOS tells us about. That means Linux may assign resources outside of those apertures to PCI devices. They *might* work, but there's no guarantee that they will. This can affect devices present at boot as well as hot-added ones.

If we were really smart, we might emit a warning when we assigned resources outside of the apertures, but that's probably more work than it's worth.

If the workaround works for you, that's great and you should keep using it. I just don't want to propagate the idea that "pci=nocrs" is
the general solution for anything. Ideally, a user should not have
to use kernel parameters except for debugging or to work around hardware or firmware problems.

Bjorn

author
#11 lore raw

Re: PCI rescan : can't assign mem [SOLVED]

Quoted 6 lines · Bjorn Helgaas <bhelgaas <at> google.com> writes:
Quoted 2 lines · On Fri, Sep 4, 2015 at 12:32 AM, Peio <peio <at> azkarate.eu> wrote:

Bjorn Helgaas <bhelgaas <at> google.com> writes:
-

Quoted 8 lines

If the workaround works for you, that's great and you should keep using it. I just don't want to propagate the idea that "pci=nocrs" is
the general solution for anything. Ideally, a user should not have
to use kernel parameters except for debugging or to work around hardware or firmware problems.

Bjorn

At the moment I'll use the workaround, but a better solution will be, update the BIOS to a version assigning more memory resources to pcie. Thank you for your warning Bjorn.

Peio