Debugging incorrect mps settings detected
Hello,
Hot adding a pci device results in improper MPS settings detected, reading 128 even though the upstream port is 256. This causes a warning print to be logged suggesting I should report a bug. :)
Here's the dmesg of the hot plug event:
[ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27) [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802 [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 2107.014986] pcieport 0000:05:07.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff pref] [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff 64bit] [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09] [ 2107.015037] pcieport 0000:05:07.0: bridge window [mem 0xdb600000-0xdb7fffff] [ 2107.015042] pcieport 0000:05:07.0: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
The commit message that added the warning indicates it may be a BIOS issue that incorrectly configures this, but BIOS isn't involved with setting the MPS on a device hot add, am I right? Or is that wrong? This is a Dell R620 server, if that matters, though we see this on other platforms as well.
Using pci=pcie_bus_perf works around the issue, but that seems a bit odd to suggest to users. Is setting MPS on hot-add something the PCI driver layer ought to be doing, or should I be looking into the platform BIOS instead?
Thanks!
Keith
Bjorn Helgaas
maintainer[+cc Yijing, Jon]
Quoted 32 lines · Keith Busch
Hello,
Hot adding a pci device results in improper MPS settings detected, reading 128 even though the upstream port is 256. This causes a warning print to be logged suggesting I should report a bug. :)
Here's the dmesg of the hot plug event:
[ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27) [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802 [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 2107.014986] pcieport 0000:05:07.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff pref] [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff 64bit] [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09] [ 2107.015037] pcieport 0000:05:07.0: bridge window [mem 0xdb600000-0xdb7fffff] [ 2107.015042] pcieport 0000:05:07.0: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
Quoted 3 lines
The commit message that added the warning indicates it may be a BIOS issue that incorrectly configures this, but BIOS isn't involved with setting the MPS on a device hot add, am I right? Or is that wrong?
The BIOS is not involved in setting MPS on a device hot-add.
Quoted 7 lines
This is a Dell R620 server, if that matters, though we see this on other platforms as well.
Using pci=pcie_bus_perf works around the issue, but that seems a bit odd to suggest to users. Is setting MPS on hot-add something the PCI driver layer ought to be doing, or should I be looking into the platform BIOS instead?
The PCI core should be doing something intelligent here. If we can't configure MPS safely, we should prevent the device from being enabled.
But this is stuff the PCI core doesn't implement yet.
It's hard to reconfigure MPS after boot because it affects other devices in the system, and I don't think we can always change the settings of other devices while they're active.
I'm not sure that warning is correct. It seems like it should suggest pcie_bus_peer2peer, which is documented as guaranteeing that hot-added devices will work. But this is always confusing to me, and I could be wrong about this.
Bjorn
accurate>> 0xd5200000-0xd52fffff 64bit pref]
Quoted 14 lines
Quoted 8 lines
[ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platformThe commit message that added the warning indicates it may be a BIOS issue that incorrectly configures this, but BIOS isn't involved with setting the MPS on a device hot add, am I right? Or is that wrong?
The BIOS is not involved in setting MPS on a device hot-add.
Quoted 3 lines
This is a Dell R620 server, if that matters, though we see this on other platforms as well.
Quoted 8 lines
Quoted 4 lines
Using pci=pcie_bus_perf works around the issue, but that seems a bit odd to suggest to users. Is setting MPS on hot-add something the PCI driver layer ought to be doing, or should I be looking into the platform BIOS instead?
The PCI core should be doing something intelligent here. If we can't configure MPS safely, we should prevent the device from being enabled.
But this is stuff the PCI core doesn't implement yet.
Hi Bjorn,
We had a long discussion about fix this issue.
Now more BIOS configure device mps larger than 128B, So I think
we should consider to fix it before we have a perfect solution about mps setting.
Quoted 4 lines
It's hard to reconfigure MPS after boot because it affects other devices in the system, and I don't think we can always change the settings of other devices while they're active.
Yes, what we can only change is the device not active after hotplug.
Quoted 5 lines
I'm not sure that warning is correct. It seems like it should suggest
pcie_bus_peer2peer, which is documented as guaranteeing that hot-added devices will work. But this is always confusing to me, and I could be wrong about this.
I think warning is proper, pcie_bus_safe is designed to fix the hotlplug issue. pcie_bus_peer2perr guarantee that hotplug device work fine, because all mps setting 128B. Maybe we should change document a little, make it more clear.
Quoted 4 lines
Bjorn
Signature · 2 lines
Thanks! Yijing
Quoted 44 lines · Bjorn Helgaas
[+cc Yijing, Jon]
Quoted 32 lines · Keith Busch
Hello,
Hot adding a pci device results in improper MPS settings detected, reading 128 even though the upstream port is 256. This causes a warning print to be logged suggesting I should report a bug. :)
Here's the dmesg of the hot plug event:
[ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27) [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802 [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 2107.014986] pcieport 0000:05:07.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff pref] [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff 64bit] [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09] [ 2107.015037] pcieport 0000:05:07.0: bridge window [mem 0xdb600000-0xdb7fffff] [ 2107.015042] pcieport 0000:05:07.0: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platformQuoted 3 lines
The commit message that added the warning indicates it may be a BIOS issue that incorrectly configures this, but BIOS isn't involved with setting the MPS on a device hot add, am I right? Or is that wrong?
The BIOS is not involved in setting MPS on a device hot-add.
Quoted 3 lines
This is a Dell R620 server, if that matters, though we see this on other platforms as well.
Quoted 8 lines
Quoted 4 lines
Using pci=pcie_bus_perf works around the issue, but that seems a bit odd to suggest to users. Is setting MPS on hot-add something the PCI driver layer ought to be doing, or should I be looking into the platform BIOS instead?
The PCI core should be doing something intelligent here. If we can't configure MPS safely, we should prevent the device from being enabled.
But this is stuff the PCI core doesn't implement yet.
The tuning of the MPS is disabled by default, which is why the warning is there. Now, we can attempt to tune it regardless of whether tuning is enabled (assuming this device is the only one under the root port), we can fail to all the hotplug, or we can enable "safe" tuning by default. The first option seems a bit hacky. The second will fail too often, since the MPSS of the hotplugged device would have to match the MPS setting of slot/fabric. The third seems like the best option.
Quoted 3 lines
It's hard to reconfigure MPS after boot because it affects other devices in the system, and I don't think we can always change the settings of other devices while they're active.
Unless the slot is under its own root port, otherwise the MPS will have to be floored. This is the behavior in the "safe" tuning.
I suggest that the "safe" tuning get made the default behavior and we let it soak for a little while to shake out any issues that there may be.
Thanks,
Jon
Quoted 6 lines
I'm not sure that warning is correct. It seems like it should suggest
pcie_bus_peer2peer, which is documented as guaranteeing that hot-added devices will work. But this is always confusing to me, and I could be wrong about this.Bjorn
Quoted 18 lines · Jon Mason
Quoted 4 lines · Bjorn Helgaas
[+cc Yijing, Jon]
The PCI core should be doing something intelligent here. If we can't configure MPS safely, we should prevent the device from being enabled.
But this is stuff the PCI core doesn't implement yet.The tuning of the MPS is disabled by default, which is why the warning is there. Now, we can attempt to tune it regardless of whether tuning is enabled (assuming this device is the only one under the root port), we can fail to all the hotplug, or we can enable "safe" tuning by default. The first option seems a bit hacky. The second will fail too often, since the MPSS of the hotplugged device would have to match the MPS setting of slot/fabric. The third seems like the best option.
Quoted 3 lines
It's hard to reconfigure MPS after boot because it affects other devices in the system, and I don't think we can always change the settings of other devices while they're active.
Quoted 6 lines
Unless the slot is under its own root port, otherwise the MPS will have to be floored. This is the behavior in the "safe" tuning.
I suggest that the "safe" tuning get made the default behavior and we let it soak for a little while to shake out any issues that there may be.
Thanks for all the info. I've tried different tunings on several different platforms that were all failing before. The 'safe' tuning was the only one successful on all platforms, so just providing a data point that this appears to be a good direction to go.
We are still seeing issues too with the maxpayload default setting being incorrect for hotplug PCIE systems on Linux.
The default setting doesn't work if any root port is configured with MPS>128. Adding in a hotplug device will default to MPS=128 and will cause PCIE error. Clamping to pci_bus_safe causes the whole system to run inefficiently at MPS=128
Windows and VMWare are already using the equivalent of pcie_bus_perf and not seeing problems.
--jordan hargrave
Dell Enterprise Linux Engineering
________________________________________
From: [email protected] [[email protected]] On Behalf Of Bjorn Helgaas [[email protected]]
Sent: Tuesday, June 03, 2014 4:01 PM
To: Keith Busch
Cc: [email protected]; Yijing Wang; Jon Mason
Subject: Re: Debugging incorrect mps settings detected
[+cc Yijing, Jon]
Quoted 32 lines · Keith Busch
Hello,
Hot adding a pci device results in improper MPS settings detected, reading 128 even though the upstream port is 256. This causes a warning print to be logged suggesting I should report a bug. :)
Here's the dmesg of the hot plug event:
[ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27) [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802 [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 2107.014986] pcieport 0000:05:07.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000) [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff pref] [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff 64bit] [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09] [ 2107.015037] pcieport 0000:05:07.0: bridge window [mem 0xdb600000-0xdb7fffff] [ 2107.015042] pcieport 0000:05:07.0: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
Quoted 3 lines
The commit message that added the warning indicates it may be a BIOS issue that incorrectly configures this, but BIOS isn't involved with setting the MPS on a device hot add, am I right? Or is that wrong?
The BIOS is not involved in setting MPS on a device hot-add.
Quoted 7 lines
This is a Dell R620 server, if that matters, though we see this on other platforms as well.
Using pci=pcie_bus_perf works around the issue, but that seems a bit odd to suggest to users. Is setting MPS on hot-add something the PCI driver layer ought to be doing, or should I be looking into the platform BIOS instead?
The PCI core should be doing something intelligent here. If we can't configure MPS safely, we should prevent the device from being enabled.
But this is stuff the PCI core doesn't implement yet.
It's hard to reconfigure MPS after boot because it affects other devices in the system, and I don't think we can always change the settings of other devices while they're active.
I'm not sure that warning is correct. It seems like it should suggest pcie_bus_peer2peer, which is documented as guaranteeing that hot-added devices will work. But this is always confusing to me, and I could be wrong about this.
Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html