Skip to contentSkip to conversation

IO Space Allocation Question on ARM64

DISCUSSION Sinan Kaya 2 participants 3 messages May 22, 2017 PCI

lore
Share & tools
lore

Sinan Kaya

author
#0 lore raw

We are seeing IO space allocation failure on QDF2400 with multiple switch hierarchy below. This card has 20 PCI bridges with endpoints attached to each one to increase the number of endpoints supported.

QDF2400 supports a maximum of 64k IO space aka 16 bit IO on the root port.

-+-[0003:00]---00.0-[01-16]----00.0-[02-16]--+-00.0-[03]----00.0

 |                                           +-01.0-[04]----00.0
 |                                           +-02.0-[05]----00.0
 |                                           +-03.0-[06]----00.0
 |                                           +-04.0-[07]----00.0
 |                                           +-05.0-[08]----00.0
 |                                           +-06.0-[09]----00.0
 |                                           +-07.0-[0a]----00.0
 |                                           +-08.0-[0b]--
 |                                           +-09.0-[0c]--
 |                                           +-0a.0-[0d]--
 |                                           +-0b.0-[0e]--
 |                                           +-0c.0-[0f]--
 |                                           +-0d.0-[10]--
 |                                           +-0e.0-[11]--
 |                                           +-0f.0-[12]--
 |                                           +-10.0-[13]--
 |                                           +-11.0-[14]--
 |                                           +-12.0-[15]--
 |                                           \-13.0-[16]--

What we so far is that Linux is assigning an IO resource for every single bridge that it finds even though there is nothing connected behind on some of the bridges. Do we know the reason for this behavior?

[    4.288269] pci 0003:00:00.0: BAR 13: failed to assign [io  size 0x15000]
[    4.317175] pci 0003:01:00.0: BAR 13: failed to assign [io  size 0x14000]
[    4.644518] pci 0003:02:00.0: BAR 13: failed to assign [io  size 0x1000]
[    4.657539] pci 0003:02:01.0: BAR 13: failed to assign [io  size 0x1000]
[    4.670560] pci 0003:02:02.0: BAR 13: failed to assign [io  size 0x1000]
[    4.683581] pci 0003:02:03.0: BAR 13: failed to assign [io  size 0x1000]
[    4.696602] pci 0003:02:04.0: BAR 13: failed to assign [io  size 0x1000]
[    4.709623] pci 0003:02:05.0: BAR 13: failed to assign [io  size 0x1000]
[    4.722644] pci 0003:02:06.0: BAR 13: failed to assign [io  size 0x1000]
[    4.735664] pci 0003:02:07.0: BAR 13: failed to assign [io  size 0x1000]
[    4.748685] pci 0003:02:08.0: BAR 13: failed to assign [io  size 0x1000]
[    4.761706] pci 0003:02:09.0: BAR 13: failed to assign [io  size 0x1000]
[    4.774727] pci 0003:02:0a.0: BAR 13: failed to assign [io  size 0x1000]
[    4.787748] pci 0003:02:0b.0: BAR 13: failed to assign [io  size 0x1000]
[    4.800768] pci 0003:02:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    4.813789] pci 0003:02:0d.0: BAR 13: failed to assign [io  size 0x1000]
[    4.826810] pci 0003:02:0e.0: BAR 13: failed to assign [io  size 0x1000]
[    4.839831] pci 0003:02:0f.0: BAR 13: failed to assign [io  size 0x1000]
[    4.852852] pci 0003:02:10.0: BAR 13: failed to assign [io  size 0x1000]
[    4.865873] pci 0003:02:11.0: BAR 13: failed to assign [io  size 0x1000]
[    4.878893] pci 0003:02:12.0: BAR 13: failed to assign [io  size 0x1000]
[    4.891914] pci 0003:02:13.0: BAR 13: failed to assign [io  size 0x1000]

UEFI seems to be assigning an IO resource to bridge only if finds an endpoint behind it with an actual IO BAR space.

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
#1 lore raw
Quoted 54 lines · On Mon, 22 May 2017 13:50:31 -0400 Sinan Kaya <[email protected]> wrote:

We are seeing IO space allocation failure on QDF2400 with multiple switch hierarchy below. This card has 20 PCI bridges with endpoints attached to each one to increase the number of endpoints supported.

QDF2400 supports a maximum of 64k IO space aka 16 bit IO on the root port.

-+-[0003:00]---00.0-[01-16]----00.0-[02-16]--+-00.0-[03]----00.0

 |                                           +-01.0-[04]----00.0
 |                                           +-02.0-[05]----00.0
 |                                           +-03.0-[06]----00.0
 |                                           +-04.0-[07]----00.0
 |                                           +-05.0-[08]----00.0
 |                                           +-06.0-[09]----00.0
 |                                           +-07.0-[0a]----00.0
 |                                           +-08.0-[0b]--
 |                                           +-09.0-[0c]--
 |                                           +-0a.0-[0d]--
 |                                           +-0b.0-[0e]--
 |                                           +-0c.0-[0f]--
 |                                           +-0d.0-[10]--
 |                                           +-0e.0-[11]--
 |                                           +-0f.0-[12]--
 |                                           +-10.0-[13]--
 |                                           +-11.0-[14]--
 |                                           +-12.0-[15]--
 |                                           \-13.0-[16]--

What we so far is that Linux is assigning an IO resource for every single bridge that it finds even though there is nothing connected behind on some of the bridges. Do we know the reason for this behavior?

[    4.288269] pci 0003:00:00.0: BAR 13: failed to assign [io  size 0x15000]
[    4.317175] pci 0003:01:00.0: BAR 13: failed to assign [io  size 0x14000]
[    4.644518] pci 0003:02:00.0: BAR 13: failed to assign [io  size 0x1000]
[    4.657539] pci 0003:02:01.0: BAR 13: failed to assign [io  size 0x1000]
[    4.670560] pci 0003:02:02.0: BAR 13: failed to assign [io  size 0x1000]
[    4.683581] pci 0003:02:03.0: BAR 13: failed to assign [io  size 0x1000]
[    4.696602] pci 0003:02:04.0: BAR 13: failed to assign [io  size 0x1000]
[    4.709623] pci 0003:02:05.0: BAR 13: failed to assign [io  size 0x1000]
[    4.722644] pci 0003:02:06.0: BAR 13: failed to assign [io  size 0x1000]
[    4.735664] pci 0003:02:07.0: BAR 13: failed to assign [io  size 0x1000]
[    4.748685] pci 0003:02:08.0: BAR 13: failed to assign [io  size 0x1000]
[    4.761706] pci 0003:02:09.0: BAR 13: failed to assign [io  size 0x1000]
[    4.774727] pci 0003:02:0a.0: BAR 13: failed to assign [io  size 0x1000]
[    4.787748] pci 0003:02:0b.0: BAR 13: failed to assign [io  size 0x1000]
[    4.800768] pci 0003:02:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    4.813789] pci 0003:02:0d.0: BAR 13: failed to assign [io  size 0x1000]
[    4.826810] pci 0003:02:0e.0: BAR 13: failed to assign [io  size 0x1000]
[    4.839831] pci 0003:02:0f.0: BAR 13: failed to assign [io  size 0x1000]
[    4.852852] pci 0003:02:10.0: BAR 13: failed to assign [io  size 0x1000]
[    4.865873] pci 0003:02:11.0: BAR 13: failed to assign [io  size 0x1000]
[    4.878893] pci 0003:02:12.0: BAR 13: failed to assign [io  size 0x1000]
[    4.891914] pci 0003:02:13.0: BAR 13: failed to assign [io  size 0x1000]
Quoted 2 lines

UEFI seems to be assigning an IO resource to bridge only if finds an endpoint behind it with an actual IO BAR space.

If it's anything like we saw with similar issues on QEMU, it might be because the slots report themselves as hotplug capable and Linux tries to allocate minimum apertures for any hotplug slot. See hpiosize and hpmemsize options for pci=. Thanks,

Alex

author
#2 lore raw

Hi Alex,

Quoted 6 lines · Alex Williamson
Quoted 2 lines

UEFI seems to be assigning an IO resource to bridge only if finds an endpoint behind it with an actual IO BAR space.

If it's anything like we saw with similar issues on QEMU, it might be because the slots report themselves as hotplug capable and Linux tries to allocate minimum apertures for any hotplug slot. See hpiosize and hpmemsize options for pci=. Thanks,

That was it. I was suspicious of this but didn't realize that we were allocating 0x1000 as a minimum due to hotplug.

Thanks for the suggestion. I didn't know the hpmemsize and hpiosize arguments.

Sinan

Signature · 3 lines
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.