Skip to contentSkip to conversation

pci_bus_distribute_available_resources() is wrong?

DISCUSSION Alexander Motin 2 participants 6 messages Dec 12, 2022 – Dec 13, 2022 2 messages not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): CAErSpo7WrAg5D4xyv0SycoDc1etSspU_TL6XMAK4STYrXDrGNQ@mail.gmail.com

#2 lore raw
Quoted 29 lines · Bjorn Helgaas
Quoted 20 lines · Alexander Motin

Hi,

I am writing to you three as the authors of Linux drivers/pci/setup-bus.c pci_bus_distribute_available_resources() function. Trying to debug PCI hot-plug issue on passive side of AMD NTB I hit this function, behavior of which I looks very suspicious to me, which I believe cause resource allocation problems we observe.

As I see, this function distributes extra size of parent memory window of hot-plug PCI bridge between memory windows of child bridges. It probably makes some sense, but I see a problem in the fact that the function only looks on children bridge memory windows, but not any other resources (of bridges or other devices that may be there).

In my AMD NTB case PCI topology looks this way:

+-[0000:80]-+-00.0
|           +-01.1-[81-83]----00.0-[82-83]----00.0-[83]--+-00.0 Dummy
|           |                                            \-00.1 NTB
Quoted 7 lines

80:01.1 is the root bridge where the hot-plug happens. The 81:00.0 bridge in addition to memory windows has small 16KB BAR. But since it is the only bridge on the bus, the function passes all available resources down to its children. As result, that BAR fails to allocate.
And while that BAR seems not really needed, in some cases the
allocation error makes whole memory window to be disabled, that ends up in NTB device driver attach failure.

Quoted 5 lines

Mika is working on what sounds like the same problem. His current patch series is at
https://lore.kernel.org/linux-pci/[email protected]/

We would appreciate your comments and testing as that series is developed.

Thank you, Bjorn. This definitely looks related, but as you've already noted in your review there, present patch does not handle BARs of the bridge itself, that I have in my case. I'd be happy to test the updated patch. Please keep me in a loop.

I also agree with your comment that the same should be done in case of multiple bridges. I am generally not sure the cases of single bridge or not having hot-plug on this level should be any specific.

Quoted 18 lines
Quoted 18 lines

It may be rare to see PCI bridges with BARs, but I know that at least some PLX bridges also have BARs for configuration purposes. Also I suppose the same problem would happen if there are other device on the bus aside of the bridge.

I've tried to disable pci_bus_distribute_available_resources(), and it fixed the problem. But I suppose it may cause problems in cases for which this function was developed (hot-plug of JBOD supporting hot-plug?). Am I right?

I would appreciate your feedback on this issue. I am new to this code area of Linux and not sure how to better fix this, but the way the code is written now looks very wrong to me.

Thanks.

--
Alexander Motin

Signature · 1 line
Alexander Motin
#3 lore raw

Hi,

Quoted 10 lines · Alexander Motin
Quoted 9 lines · Bjorn Helgaas
Quoted 8 lines · Alexander Motin

Hi,

I am writing to you three as the authors of Linux drivers/pci/setup-bus.c pci_bus_distribute_available_resources() function. Trying to debug PCI hot-plug issue on passive side of AMD NTB I hit this function, behavior of which I looks very suspicious to me, which I believe cause resource allocation problems we observe.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

As I see, this function distributes extra size of parent memory window of hot-plug PCI bridge between memory windows of child bridges. It probably makes some sense, but I see a problem in the fact that the function only looks on children bridge memory windows, but not any other resources (of bridges or other devices that may be there).

Right the idea was that we allocate the spare resources for the possible hotplug downstream ports so that it is possible to extend that topology without running out of resources. This is mostly used with Thunderbolt/USB4 PCIe tunneling.

However, like many have noticed, it does not handle the more generic PCI case well. Sorry about that.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In my AMD NTB case PCI topology looks this way:

+-[0000:80]-+-00.0
|           +-01.1-[81-83]----00.0-[82-83]----00.0-[83]--+-00.0 Dummy
|           |                                            \-00.1 NTB
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

80:01.1 is the root bridge where the hot-plug happens. The 81:00.0 bridge in addition to memory windows has small 16KB BAR. But since it is the only bridge on the bus, the function passes all available resources down to its children. As result, that BAR fails to allocate.
And while that BAR seems not really needed, in some cases the
allocation error makes whole memory window to be disabled, that ends up in NTB device driver attach failure.

Just out of the curiosity, is this PCIe or PCI topology?

Quoted 6 lines
Quoted 5 lines

Mika is working on what sounds like the same problem. His current patch series is at
https://lore.kernel.org/linux-pci/[email protected]/

We would appreciate your comments and testing as that series is developed.

Quoted 8 lines

Thank you, Bjorn. This definitely looks related, but as you've already noted in your review there, present patch does not handle BARs of the bridge itself, that I have in my case. I'd be happy to test the updated patch. Please keep me in a loop.

I also agree with your comment that the same should be done in case of multiple bridges. I am generally not sure the cases of single bridge or not having hot-plug on this level should be any specific.

Yeah, I'm working on a new version of the patch series that should take these into consideration. The challenge is that the code has been used with the Thunderbolt/USB4 PCIe tunneling for some time already and we don't want to break that either.

I'm also more than happy to test any patches regarding this if someone else wants to work on it ;-)

#4 lore raw
Quoted 17 lines · Mika Westerberg
Quoted 8 lines · Alexander Motin
Quoted 7 lines · Bjorn Helgaas
Quoted 6 lines · Alexander Motin

In my AMD NTB case PCI topology looks this way:

+-[0000:80]-+-00.0
|           +-01.1-[81-83]----00.0-[82-83]----00.0-[83]--+-00.0 Dummy
|           |                                            \-00.1 NTB
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

80:01.1 is the root bridge where the hot-plug happens. The 81:00.0 bridge in addition to memory windows has small 16KB BAR. But since it is the only bridge on the bus, the function passes all available resources down to its children. As result, that BAR fails to allocate.
And while that BAR seems not really needed, in some cases the
allocation error makes whole memory window to be disabled, that ends up in NTB device driver attach failure.

Quoted 1 line

Just out of the curiosity, is this PCIe or PCI topology?

All PCIe: hot-plug root <-> upstream <-> downstream <-> two endpoints.

Quoted 15 lines
Quoted 6 lines
Quoted 5 lines

Mika is working on what sounds like the same problem. His current patch series is at
https://lore.kernel.org/linux-pci/[email protected]/

We would appreciate your comments and testing as that series is developed.

Quoted 8 lines

Thank you, Bjorn. This definitely looks related, but as you've already noted in your review there, present patch does not handle BARs of the bridge itself, that I have in my case. I'd be happy to test the updated patch. Please keep me in a loop.

I also agree with your comment that the same should be done in case of multiple bridges. I am generally not sure the cases of single bridge or not having hot-plug on this level should be any specific.

Quoted 7 lines

Yeah, I'm working on a new version of the patch series that should take these into consideration. The challenge is that the code has been used with the Thunderbolt/USB4 PCIe tunneling for some time already and we don't want to break that either.

I'm also more than happy to test any patches regarding this if someone else wants to work on it ;-)

I was kind of ready to dive in, I hate hacks and tunables to workaround bugs. But as I have told, I see this code first time, so my solutions may appear not right. But I'll help as I can, if needed.

Signature · 1 line
Alexander Motin
#5 lore raw

Hi,

Quoted 6 lines · Alexander Motin
Quoted 2 lines

I'm also more than happy to test any patches regarding this if someone else wants to work on it ;-)

I was kind of ready to dive in, I hate hacks and tunables to workaround bugs. But as I have told, I see this code first time, so my solutions may appear not right. But I'll help as I can, if needed.

That's good to hear :) Okay feel free to use any of my previous patches as base if needed. I can test the Thunderbolt/USB4 and the QEMU PCI/PCIe topologies so please keep me in the loop when submitting.

#6 lore raw
Quoted 11 lines · Mika Westerberg
Quoted 6 lines · Alexander Motin
Quoted 2 lines

I'm also more than happy to test any patches regarding this if someone else wants to work on it ;-)

I was kind of ready to dive in, I hate hacks and tunables to workaround bugs. But as I have told, I see this code first time, so my solutions may appear not right. But I'll help as I can, if needed.

That's good to hear :) Okay feel free to use any of my previous patches as base if needed. I can test the Thunderbolt/USB4 and the QEMU PCI/PCIe topologies so please keep me in the loop when submitting.

It is quite a quick change of roles, don't you find? It is you created the problem. It is your email includes "@linux.intel.com" here. Don't you feel some responsibility? ;)

Where are the current result of "I'm working on a new version of the patch series that should take these into consideration"?

Signature · 1 line
Alexander Motin
#7 lore raw

Hi,

Quoted 9 lines · Alexander Motin
Quoted 8 lines · Mika Westerberg
Quoted 6 lines · Alexander Motin
Quoted 2 lines

I'm also more than happy to test any patches regarding this if someone else wants to work on it ;-)

I was kind of ready to dive in, I hate hacks and tunables to workaround bugs. But as I have told, I see this code first time, so my solutions may appear not right. But I'll help as I can, if needed.

Quoted 7 lines
Quoted 3 lines

That's good to hear :) Okay feel free to use any of my previous patches as base if needed. I can test the Thunderbolt/USB4 and the QEMU PCI/PCIe topologies so please keep me in the loop when submitting.

It is quite a quick change of roles, don't you find? It is you created the problem. It is your email includes "@linux.intel.com" here. Don't you feel some responsibility? ;)

Sorry I must have misunderstood what you meant then. I was under impression that you were ready to go and fix the issue. Okay no problem I will work on this myself then and keep you guys updated.

Quoted 2 lines

Where are the current result of "I'm working on a new version of the patch series that should take these into consideration"?

Still early "draft" I will submit it once it is in better shape (and I have validated it works in the known cases). Anyway it will happen after the merge window is closed.