Fully enable AER
Subsystems (2)
v4 cover Fully enable AER
While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.
Please note that these changes are quite invasive, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
Stefan Roese (3):
PCI/AER: Call pcie_set_ecrc_checking() for each PCIe device PCI/portdrv: Don't disable AER reporting in
get_port_device_capability()
PCI/AER: Enable AER on all PCIe devices supporting it
drivers/pci/pcie/aer.c | 10 +++++++--- drivers/pci/pcie/portdrv_core.c | 9 +-------- 2 files changed, 8 insertions(+), 11 deletions(-)
Signature · 1 line
2.35.0
PATCH v4 1/3 PCI/AER: Call pcie_set_ecrc_checking() for each PCIe device
Make sure that pcie_set_ecrc_checking() is called for each PCIe device, also when it's hot-plugged. This is done by moving pcie_set_ecrc_checking() to pci_aer_init().
---
v4:
- New patch
drivers/pci/pcie/aer.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/pcie/aer.c +3 -3
@@ -387,6 +387,9 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);pci_aer_clear_status (dev); + +/* Enable ECRC checking if enabled and configured */ +pcie_set_ecrc_checking (dev); }void pci_aer_exit (struct pci_dev *dev)
@@ -1223,9 +1226,6 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)pci_disable_pcie_error_reporting (dev); } -if (enable) -pcie_set_ecrc_checking (dev); -return 0 ; }
Signature · 1 line
2.35.0
Re: [PATCH v4 1/3] PCI/AER: Call pcie_set_ecrc_checking() for each PCIe device
Quoted 11 lines · On Tuesday 25 January 2022 08:18:18 Stefan Roese wrote:
Make sure that
pcie_set_ecrc_checking()is called for each PCIe device, also when it's hot-plugged. This is done by movingpcie_set_ecrc_checking()topci_aer_init().Signed-off-by Stefan Roese <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>
Quoted 34 lines
---
v4:
- New patchdrivers/pci/pcie/aer.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)drivers/pci/pcie/aer.c +3
-3@@ -387,6 +387,9 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);pci_aer_clear_status (dev); + +/* Enable ECRC checking if enabled and configured */ +pcie_set_ecrc_checking (dev); }void pci_aer_exit (struct pci_dev *dev)@@ -1223,9 +1226,6 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)pci_disable_pcie_error_reporting (dev); } -if (enable) -pcie_set_ecrc_checking (dev); -return 0 ; }Signature · 1 line
2.35.0
PATCH v4 2/3 PCI/portdrv: Don't disable AER reporting in get_port_device_capability()
In reply to Stefan RoeseTesting has shown, that AER reporting is currently disabled in the DevCtl registers of all non Root Port PCIe devices on systems using pcie_ports_native || host->native_aer. Practically disabling AER completely in such systems. This is due to the fact that with commit 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization"), a call to pci_disable_pcie_error_reporting() was added *after* the PCIe AER setup was completed for the PCIe device tree.
Here a longer analysis about the currect status of AER enaling / disabling upon bootup provided by Bjorn:
pcie_portdrv_probe
pcie_port_device_register
get_port_device_capability
pci_disable_pcie_error_reporting
clear CERE NFERE FERE URRE # <-- disable for RP USP DSP
pcie_device_init
device_register # new AER service device
aer_probe
aer_enable_rootport # RP only
set_downstream_devices_error_reporting
set_device_error_reporting # self (RP)
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for RP
pci_walk_bus
set_device_error_reporting
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for USP DSP
In a typical Root Port -> Endpoint hierarchy, the above:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port.
In a deeper Root Port -> Upstream Switch Port -> Downstream Switch Port -> Endpoint hierarchy:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Enables Error Reporting for both Switch Ports,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port,
- Disables Error Reporting for the Switch Ports when pcie_portdrv_probe() claims them. AER does not re-enable it because these are not Root Ports.
This patch now removes this call to pci_disable_pcie_error_reporting() from get_port_device_capability(), leaving the already enabled AER configuration intact. With this change, AER is enabled in the Root Port and the PCIe switch upstream and downstream ports. Only the PCIe Endpoints don't have AER enabled yet. A follow-up patch will take care of this Endpoint enabling.
---
v4:
- No change
v3:
- Added RB tag from Pali
v2:
- Enhance commit message as suggested by Bjorn
drivers/pci/pcie/portdrv_core.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/pci/pcie/portdrv_core.c +1 -8
@@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)#ifdef CONFIG_PCIEAER if (dev->aer_cap &&pci_aer_available () && - (pcie_ports_native || host->native_aer)) { + (pcie_ports_native || host->native_aer)) services |=PCIE_PORT_SERVICE_AER ; - -/* -* Disable AER on this port in case it's been enabled by the -* BIOS (the AER service driver will enable it when necessary). -*/ -pci_disable_pcie_error_reporting (dev); - }#endif /* Root Ports and Root Complex Event Collectors may generate PMEs */
Signature · 1 line
2.35.0
PATCH v4 3/3 PCI/AER: Enable AER on all PCIe devices supporting it
In reply to Stefan RoeseWith this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.
Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
When "pci=noaer" is selected, AER stays disabled of course.
---
v4:
- No change
v3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done in pci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.
drivers/pci/pcie/aer.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/pci/pcie/aer.c +4 -0
@@ -388,6 +388,10 @@ void pci_aer_init(struct pci_dev *dev)pci_aer_clear_status (dev); +/* Enable AER if requested */ +if (pci_aer_available ()) +pci_enable_pcie_error_reporting (dev); +/* Enable ECRC checking if enabled and configured */ pcie_set_ecrc_checking (dev); }
Signature · 1 line
2.35.0
Re: [PATCH v4 3/3] PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 9 lines · On Tuesday 25 January 2022 08:18:20 Stefan Roese wrote:
With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.
Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
When "pci=noaer" is selected, AER stays disabled of course.
Quoted 7 lines
Signed-off-by Stefan Roese <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>
Quoted 31 lines
---
v4:
- No changev3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done inpci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.drivers/pci/pcie/aer.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/pci/pcie/aer.c +4
-0@@ -388,6 +388,10 @@ void pci_aer_init(struct pci_dev *dev)pci_aer_clear_status (dev); +/* Enable AER if requested */ +if (pci_aer_available ()) +pci_enable_pcie_error_reporting (dev); +/* Enable ECRC checking if enabled and configured */ pcie_set_ecrc_checking (dev); }Signature · 1 line
2.35.0
Re: [PATCH v4 3/3] PCI/AER: Enable AER on all PCIe devices supporting it
In reply to Stefan RoeseQuoted 34 lines · Stefan Roese
With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.
Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
When "pci=noaer" is selected, AER stays disabled of course.
Signed-off-by Stefan Roese <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>---
v4:
- No changev3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done inpci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.drivers/pci/pcie/aer.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/pci/pcie/aer.c +0
-0
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 393
Quoted 6 lines
drivers/pci/pcie/aer.c +3
-0@@ -388,6 +388,10 @@ void pci_aer_init(struct pci_dev *dev)pci_aer_clear_status (dev); +/* Enable AER if requested */ +if (pci_aer_available ()) +pci_enable_pcie_error_reporting (dev);
There are a lot of devices that do this explicitly [1] May suggest a cleanup patch to follow-up?
[1] https://elixir.bootlin.com/linux/v5.18-rc1/A/ident/pci_enable_pcie_error_reporting
... Also a quirk list in the future for broken devices
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-0+/* Enable ECRC checking if enabled and configured */ pcie_set_ecrc_checking (dev); }
Re: [PATCH v4 3/3] PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 32 lines · Jonathan Derrick
Quoted 29 lines · Stefan Roese
With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.
Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
When "pci=noaer" is selected, AER stays disabled of course.
Signed-off-by Stefan Roese <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>---
v4:
- No changev3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done inpci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.
drivers/pci/pcie/aer.c| 4 ++++1 file changed, 4 insertions(+)
Quoted 11 lines
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 391
Quoted 9 lines
drivers/pci/pcie/aer.c +3
-0@@ -388,6 +388,10 @@ void pci_aer_init(struct pci_dev *dev) pci_aer_clear_status(dev); + /* Enable AER if requested */ + if (pci_aer_available()) + pci_enable_pcie_error_reporting(dev);There are a lot of devices that do this explicitly [1] May suggest a cleanup patch to follow-up?
Yes, good idea. I can try to work on this once this patchset is merged.
Quoted 4 lines
[1] https://elixir.bootlin.com/linux/v5.18-rc1/A/ident/pci_enable_pcie_error_reporting
... Also a quirk list in the future for broken devices
IMHO this should only be done, when such devices are detected.
Thanks,
Stefan
Re: [PATCH v4 3/3] PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 45 lines · Stefan Roese
Quoted 32 lines · Jonathan Derrick
Quoted 29 lines · Stefan Roese
With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.
Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
When "pci=noaer" is selected, AER stays disabled of course.
Signed-off-by Stefan Roese <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>---
v4:
- No changev3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done inpci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.
drivers/pci/pcie/aer.c| 4 ++++1 file changed, 4 insertions(+)Quoted 11 lines
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 391
Quoted 9 lines
drivers/pci/pcie/aer.c +3
-0@@ -388,6 +388,10 @@ void pci_aer_init(struct pci_dev *dev) pci_aer_clear_status(dev); + /* Enable AER if requested */ + if (pci_aer_available()) + pci_enable_pcie_error_reporting(dev);There are a lot of devices that do this explicitly [1] May suggest a cleanup patch to follow-up?
Quoted 8 lines
Yes, good idea. I can try to work on this once this patchset is merged.
Quoted 4 lines
[1] https://elixir.bootlin.com/linux/v5.18-rc1/A/ident/pci_enable_pcie_error_reporting
... Also a quirk list in the future for broken devices
IMHO this should only be done, when such devices are detected.
Yep; I'm just anticipating
Quoted 3 lines
Thanks,
Stefan
Quoted 27 lines · Stefan Roese
While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.
Please note that these changes are quite invasive, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
Cc Rafael J. Wysocki <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>Stefan Roese (3):
PCI/AER: Callpcie_set_ecrc_checking()for each PCIe device PCI/portdrv: Don't disable AER reporting in
get_port_device_capability()
PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 4 lines
drivers/pci/pcie/aer.c | 10 +++++++--- drivers/pci/pcie/portdrv_core.c | 9 +-------- 2 files changed, 8 insertions(+), 11 deletions(-)
Bjorn, what's the status of this patchset? I know this is sensible stuff, to fully enable PCIe AER. How should be handle this? Do you plan to pull this soon'ish? Please let me know if something is missing.
Thanks,
Stefan
Quoted 28 lines · Stefan Roese
Quoted 27 lines · Stefan Roese
While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.
Please note that these changes are quite invasive, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
Cc Rafael J. Wysocki <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>Stefan Roese (3):
PCI/AER: Callpcie_set_ecrc_checking()for each PCIe device PCI/portdrv: Don't disable AER reporting inget_port_device_capability()
PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 8 lines
Quoted 4 lines
drivers/pci/pcie/aer.c | 10 +++++++---
drivers/pci/pcie/portdrv_core.c| 9 +--------2 files changed, 8 insertions(+), 11 deletions(-)Bjorn, what's the status of this patchset? I know this is sensible stuff, to fully enable PCIe AER. How should be handle this? Do you plan to pull this soon'ish? Please let me know if something is missing.
Hi Bjorn,
maybe it helps, if I send the patchwork links for these patches as well. Here we go:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
All 3 are delegated to you.
Thanks,
Stefan
Bjorn Helgaas
AppliedQuoted 27 lines · Stefan Roese
While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.
Please note that these changes are quite invasive, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.
Cc Rafael J. Wysocki <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Pali Rohár <[email protected]>Cc Bharat Kumar Gogada <[email protected]>Cc Michal Simek <[email protected]>Cc Yao Hongbo <[email protected]>Cc Naveen Naidu <[email protected]>Stefan Roese (3):
PCI/AER: Callpcie_set_ecrc_checking()for each PCIe device PCI/portdrv: Don't disable AER reporting in
get_port_device_capability()
PCI/AER: Enable AER on all PCIe devices supporting it
Quoted 3 lines
drivers/pci/pcie/aer.c | 10 +++++++--- drivers/pci/pcie/portdrv_core.c | 9 +-------- 2 files changed, 8 insertions(+), 11 deletions(-)
Applied to pci/err for v5.20, thanks and sorry for the delay.