Does dwc/pci-layerscape.c support AER?
Subsystems (2)
I see "aer" mentioned in layerscape DT 'interrupt-names':
$ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer"; arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; ...
But I don't know whether or how these are connected to the AER driver (drivers/pci/pcie/aer.c).
Does the AER driver actually work on these platforms?
Is there some magic that connects the 'interrupt-names'/'interrupts' DT properties to the pcie_device.irq that aer_probe() requests and hooks up with the aer_irq() handler?
The pcie_device.irq for AER was assigned by portdrv in this path:
pcie_portdrv_probe
pcie_port_device_register(pci_dev *dev)
int irqs[PCIE_PORT_DEVICE_MAXSERVICES]
pcie_init_service_irqs(dev, irqs, ...)
# try MSI/MSI-X first:
pcie_port_enable_irq_vec(dev, irqs, ...)
pci_alloc_irq_vectors(PCI_IRQ_MSIX | PCI_IRQ_MSI)
pcie_message_numbers(dev, mask, &pme, &aer, &dpc)
irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer)
# fall back to INTx if no MSI/MSI-X
pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_INTX);
for (i = 0; ...; i++)
pcie_device_init(pdev, irqs[i])
pcie = kzalloc() # struct pcie_device
pcie->irq = irq # <-- pcie_device.irqbut I only see attempts to use MSI/MSI-X/INTx, which we discover and configure based on the MSI or MSI-X Capability or the INTx pin advertised at PCI_INTERRUPT_PIN.
I don't see anything related to DT or platform IRQs that I can connect with the DT 'interrupts' property.
Bjorn
Hi Bjorn,
I'm not so familiar with layerscape, but I'm familiar with parts of this:
Quoted 8 lines · Bjorn Helgaas
I see "aer" mentioned in layerscape DT 'interrupt-names':
$ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer"; arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; ...
NB: in fsl-ls1012a.dtsi at least, the &pcie1 node has 'msi-parent = <&msi>' that points at a 'fsl,ls1012a-msi'-compatible irqchip. I expect that's how MSIs are getting resolved, and so that's likely how portdrv gets its PME IRQ to enable AER among other things.
I think the other SoCs you point at above do something similar, or with GIC ITS.
Quoted 9 lines
But I don't know whether or how these are connected to the AER driver (
drivers/pci/pcie/aer.c).Does the AER driver actually work on these platforms?
Is there some magic that connects the 'interrupt-names'/'interrupts' DT properties to the pcie_device.irq that
aer_probe()requests and hooks up with theaer_irq()handler?
I think you're right to notice the disconnect. I believe a typical DWC implementation does not actually route the root port's PME/AER through its MSI line, so many SoCs will have a dedicated platform IRQ for them instead -- but I also don't know of any that actually hook those platform IRQs up to the AER driver properly [1]. If they want AER, they do something like commit ("9c4cd0aef259 arm64: dts: qcom: x1e80100:
enable GICv3 ITS for PCIe").
Brian
[1] Although I've seen some try to abuse interrupt-map to pretend that their platform-AER interrupt is actually an INTx source... But that breaks actually INTx support.
Frank Li
maintainerQuoted 36 lines · Bjorn Helgaas
I see "aer" mentioned in layerscape DT 'interrupt-names':
$ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer"; arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer"; ...But I don't know whether or how these are connected to the AER driver (
drivers/pci/pcie/aer.c).Does the AER driver actually work on these platforms?
Is there some magic that connects the 'interrupt-names'/'interrupts' DT properties to the pcie_device.irq that
aer_probe()requests and hooks up with theaer_irq()handler?The pcie_device.irq for AER was assigned by portdrv in this path:
pcie_portdrv_probe pcie_port_device_register(pci_dev *dev) int irqs[PCIE_PORT_DEVICE_MAXSERVICES] pcie_init_service_irqs(dev, irqs, ...) # try MSI/MSI-X first: pcie_port_enable_irq_vec(dev, irqs, ...) pci_alloc_irq_vectors(PCI_IRQ_MSIX | PCI_IRQ_MSI) pcie_message_numbers(dev, mask, &pme, &aer, &dpc) irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer) # fall back to INTx if no MSI/MSI-X pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_INTX); for (i = 0; ...; i++) pcie_device_init(pdev, irqs[i]) pcie = kzalloc() # struct pcie_device pcie->irq = irq # <-- pcie_device.irq
Quoted 6 lines
but I only see attempts to use MSI/MSI-X/INTx, which we discover and configure based on the MSI or MSI-X Capability or the INTx pin advertised at
PCI_INTERRUPT_PIN.I don't see anything related to DT or platform IRQs that I can connect with the DT 'interrupts' property.
There are several attempts to upstream customer Aer irq support in past years.
For example:
https://www.mail-archive.com/[email protected]/msg1161848.html
some change port drivers.
If you think it is valuable to support customer AER IRQ support, I can restart this work.
Frank
Quoted 2 lines
Bjorn
Hi Frank,
Quoted 1 line · Frank Li
Quoted 1 line
Does the AER driver actually work on these platforms?
...
Quoted 9 lines
There are several attempts to upstream customer Aer irq support in past years.
For example:
https://www.mail-archive.com/[email protected]/msg1161848.htmlsome change port drivers.
If you think it is valuable to support customer AER IRQ support, I can restart this work.
Interesting thread. I read through it, but I'm still not convinced about one detail:
Are you sure that AER can't possibly work over MSI? Even today, the Synopsys manuals say that their integrated MSI receiver "terminate[s] inbound MSI requests (received on the RX wire)" and after terminating, "an interrupt is signaled locally through the msi_ctrl_int output."
That means that their msi_ctrl_int signal only handles MSI requests from downstream functions, and it implies that the default drivers/pci/controller/dwc/pcie-designware-host.c dw_pcie_msi_domain_info implementation will not actually see MSIs from the root port (such as PME and AER). So yes, it *appears* that AER does not work over MSI.
But crucially, it does *not* mean that the port will not generate valid MSI requests, if you have some kind of logic that will receive it. So for instance, I pointed out in another reply that some SoCs choose to hook up GIC ITS:
commit 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe")
"""
Note that using the GIC ITS on x1e80100 will cause Advanced Error Reporting (AER) interrupts to be received on errors unlike when using the internal MSI controller. Consequently, notifications about (correctable) errors may now be logged for errors that previously went unnoticed.
"""
And in fact, your arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi seems to be doing the same. I'd be surprised if these port MSIs still don't work after that.
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...
Brian
Frank Li
maintainerQuoted 46 lines · Brian Norris
Hi Frank,
Quoted 1 line · Frank Li
Quoted 1 line
Does the AER driver actually work on these platforms?
...
Quoted 9 lines
There are several attempts to upstream customer Aer irq support in past years.
For example:
https://www.mail-archive.com/[email protected]/msg1161848.htmlsome change port drivers.
If you think it is valuable to support customer AER IRQ support, I can restart this work.
Interesting thread. I read through it, but I'm still not convinced about one detail:
Are you sure that AER can't possibly work over MSI? Even today, the Synopsys manuals say that their integrated MSI receiver "terminate[s] inbound MSI requests (received on the RX wire)" and after terminating, "an interrupt is signaled locally through the
msi_ctrl_intoutput."That means that their
msi_ctrl_intsignal only handles MSI requests from downstream functions, and it implies that the defaultdrivers/pci/controller/dwc/pcie-designware-host.cdw_pcie_msi_domain_infoimplementation will not actually see MSIs from the root port (such as PME and AER). So yes, it *appears* that AER does not work over MSI.But crucially, it does *not* mean that the port will not generate valid MSI requests, if you have some kind of logic that will receive it. So for instance, I pointed out in another reply that some SoCs choose to hook up GIC ITS:
commit
9c4cd0aef259("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe")"""
Note that using the GIC ITS on x1e80100 will cause Advanced Error Reporting (AER) interrupts to be received on errors unlike when using the internal MSI controller. Consequently, notifications about (correctable) errors may now be logged for errors that previously went unnoticed.
"""
Quoted 7 lines
And in fact, your
arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsiseems to be doing the same. I'd be surprised if these port MSIs still don't work after that.OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...
I saw AER and PME irq registed. But I have not seen irq increased. I am not sure how to inject an error to test it.
I suppose it go through AER irq line.
Frank
Quoted 2 lines
Brian
Hi Frank,
Thanks for the response.
Quoted 2 lines · Frank Li
I saw AER and PME irq registed. But I have not seen irq increased. I am not sure how to inject an error to test it.
I've tested AER-like conditions via one of two ways:
1. force asserting PERST#, and then try to read a config register. This should generate Complection Timeouts at least, and possibly other errors. This method may not necessarily yield AER logs, as it may also reset the error reporting registers that the Linux AER driver would expect to read. But it probably should still trigger an interrupt.
This depends on having access to PERST#; many SoCs provide this as a GPIO which you could potentially control, although I don't see this in the layerscape driver at the moment.
2. asserting HOT RESET in the DWC controller. This is especially implementation specific, as it depends on how (if at all) the hot reset signal is connected into your SoC.
Not sure if any of that helps you for testing. And maybe you want to wire up your platform IRQs anyway.
Brian
On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
Sorry for jumping late into this thread. I missed it completely as I was not CCed.
Quoted 46 lines
Hi Frank,
Quoted 1 line · Frank Li
Quoted 1 line
Does the AER driver actually work on these platforms?
...
Quoted 9 lines
There are several attempts to upstream customer Aer irq support in past years.
For example:
https://www.mail-archive.com/[email protected]/msg1161848.htmlsome change port drivers.
If you think it is valuable to support customer AER IRQ support, I can restart this work.
Interesting thread. I read through it, but I'm still not convinced about one detail:
Are you sure that AER can't possibly work over MSI? Even today, the Synopsys manuals say that their integrated MSI receiver "terminate[s] inbound MSI requests (received on the RX wire)" and after terminating, "an interrupt is signaled locally through the
msi_ctrl_intoutput."That means that their
msi_ctrl_intsignal only handles MSI requests from downstream functions, and it implies that the defaultdrivers/pci/controller/dwc/pcie-designware-host.cdw_pcie_msi_domain_infoimplementation will not actually see MSIs from the root port (such as PME and AER). So yes, it *appears* that AER does not work over MSI.But crucially, it does *not* mean that the port will not generate valid MSI requests, if you have some kind of logic that will receive it. So for instance, I pointed out in another reply that some SoCs choose to hook up GIC ITS:
commit
9c4cd0aef259("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe")"""
Note that using the GIC ITS on x1e80100 will cause Advanced Error Reporting (AER) interrupts to be received on errors unlike when using the internal MSI controller. Consequently, notifications about (correctable) errors may now be logged for errors that previously went unnoticed.
"""
Quoted 8 lines
And in fact, your
arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsiseems to be doing the same. I'd be surprised if these port MSIs still don't work after that.OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...
Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
So I think there is an incentive in allowing the AER driver to work with vendor specific IRQs.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 7 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...
Quoted 6 lines
Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Do Qcom SoCs route INTx?
Quoted 2 lines
So I think there is an incentive in allowing the AER driver to work with vendor specific IRQs.
Yeah, I suppose even if my SoC (and Qcom, depending on the above answer) might work with INTx, it really does seem like an arbitrary decision about what SoC makers connected which DWC signals, so I suspect this is true.
Brian
Quoted 15 lines · Brian Norris
Quoted 7 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...Quoted 6 lines
Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Quoted 6 lines
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Do Qcom SoCs route INTx?
Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi cmdline parameter.
Quoted 8 lines
Quoted 2 lines
So I think there is an incentive in allowing the AER driver to work with vendor specific IRQs.
Yeah, I suppose even if my SoC (and Qcom, depending on the above answer) might work with INTx, it really does seem like an arbitrary decision about what SoC makers connected which DWC signals, so I suspect this is true.
Maybe we should be able to extend the dmi quirk in portdrv.c to allow Root Ports or host bridge to use INT-X instead of forcing them to use cmdline params.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 20 lines · Manivannan Sadhasivam
Quoted 19 lines · Brian Norris
Quoted 7 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...Quoted 6 lines
Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Quoted 5 lines
Quoted 2 lines
Do Qcom SoCs route INTx?
Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi cmdline parameter.
Cool, so it sounds like you might be in a better spot than layerscape, based on the explanations I've seen from them. They seem to require multiplexing multiple platform-specific interrupts, which isn't as easy to pretend is a proper INTx line.
Quoted 11 lines
Quoted 8 lines
Quoted 2 lines
So I think there is an incentive in allowing the AER driver to work with vendor specific IRQs.
Yeah, I suppose even if my SoC (and Qcom, depending on the above answer) might work with INTx, it really does seem like an arbitrary decision about what SoC makers connected which DWC signals, so I suspect this is true.
Maybe we should be able to extend the dmi quirk in
portdrv.cto allow Root Ports or host bridge to use INT-X instead of forcing them to use cmdline params.
Yeah, it sounds like that would be sufficient for non-ITS Qualcomm PCIe, and also for my SoCs. It's also necessary (but maybe not sufficient) for Layerscape too.
One complexity: this is not exclusively a "host bridge" quirk. It's also a quirk of the MSI provider. So for example, looking at arch/arm64/boot/dts/qcom/x1e80100.dtsi, you have 2 controllers without GIC ITS, and 2 with GIC ITS, all using the "same" type of bridge. IIUC, only the former 2 would need to fall back to INTx.
Brian
Quoted 19 lines · Brian Norris
Quoted 18 lines · Manivannan Sadhasivam
Quoted 17 lines · Brian Norris
Quoted 15 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Do Qcom SoCs route INTx?
Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi cmdline parameter.
Ugh. I think these controllers might be out of spec (or maybe we're not configuring MSI/MSI-X correctly for them). Per PCIe r7.0, sec 6.1.4.3:
While enabled for MSI or MSI-X operation, a Function is prohibited from using INTx interrupts (if implemented) to request service (MSI, MSI-X, and INTx are mutually exclusive).
If the controller advertises MSI or MSI-X, I think we will enable it and expect AER, PME, hotplug, etc. to use it.
Bjorn
Frank Li
maintainerQuoted 33 lines · Bjorn Helgaas
Quoted 19 lines · Brian Norris
Quoted 18 lines · Manivannan Sadhasivam
Quoted 17 lines · Brian Norris
Quoted 15 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Do Qcom SoCs route INTx?
Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi cmdline parameter.
Ugh. I think these controllers might be out of spec (or maybe we're not configuring MSI/MSI-X correctly for them). Per PCIe r7.0, sec 6.1.4.3:
Quoted 6 lines
While enabled for MSI or MSI-X operation, a Function is prohibited from using INTx interrupts (if implemented) to request service (MSI, MSI-X, and INTx are mutually exclusive).
If the controller advertises MSI or MSI-X, I think we will enable it and expect AER, PME, hotplug, etc. to use it.
I think it is controller implement problem, which route AER irq to INTx or other irq line in SOC. Consider many users have similar issues, can you add workaround or quirk for this type implementation.
Frank
Quoted 2 lines
Bjorn
Quoted 38 lines · Frank Li
Quoted 37 lines · Bjorn Helgaas
Quoted 19 lines · Brian Norris
Quoted 18 lines · Manivannan Sadhasivam
Quoted 17 lines · Brian Norris
Quoted 15 lines · Manivannan Sadhasivam
Quoted 5 lines · Brian Norris
On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
OTOH, I do also believe there are SoCs where DWC PCIe is available, but there is no external MSI controller, and so that same problem still may exist. I may even have such SoCs available...Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation. And the same should be true for other vendors also.
Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which is only handled by the controller driver. This is similar to the 'aer' SPI interrupt in layerscape platforms.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Yeah, I have some SoCs like this as well. But I also believe that I have INTx available, and that even when MSI doesn't work for AER/PME, INTx might.
Do Qcom SoCs route INTx?
Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi cmdline parameter.
Ugh. I think these controllers might be out of spec (or maybe we're not configuring MSI/MSI-X correctly for them). Per PCIe r7.0, sec 6.1.4.3:
While enabled for MSI or MSI-X operation, a Function is prohibited from using INTx interrupts (if implemented) to request service (MSI, MSI-X, and INTx are mutually exclusive).
Quoted 7 lines
Quoted 2 lines
If the controller advertises MSI or MSI-X, I think we will enable it and expect AER, PME, hotplug, etc. to use it.
I think it is controller implement problem, which route AER irq to INTx or other irq line in SOC. Consider many users have similar issues, can you add workaround or quirk for this type implementation.
I think the spec allows the AER interrupt to be routed to INTx, but only if the Root Port doesn't advertise MSI or MSI-X support or the OS decides not to enable MSI/MSI-X.
Since there's no generic way to say "MSI/MSI-X works on this Root Port, but only for certain interrupts, not including AER," there's no way for Linux to figure out that it shouldn't enable MSI/MSI-X.
*Somebody* can add some kind of quirk for this; not me because I don't have the hardware. It needs to be generic so the same solution works for all these devices.
Bjorn