Skip to contentSkip to conversation

Fully enable AER

PATCH Superseded Stefan Roese 2 participants 6 messages Jan 17, 2022

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#1 lore raw

PATCH v2 1/2 PCI/portdrv: Don't disable AER reporting in get_port_device_capability()

In reply to Stefan Roese

Testing has shown, that AER reporting is currently disabled in the DevCtl registers of all non Root Port PCIe devices on systems using pcie_ports_native || host->native_aer. Practically disabling AER completely in such systems. This is due to the fact that with commit 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization"), a call to pci_disable_pcie_error_reporting() was added *after* the PCIe AER setup was completed for the PCIe device tree.

Here a longer analysis about the currect status of AER enaling / disabling upon bootup provided by Bjorn:

pcie_portdrv_probe
pcie_port_device_register
get_port_device_capability
pci_disable_pcie_error_reporting
clear CERE NFERE FERE URRE # <-- disable for RP USP DSP
pcie_device_init
device_register # new AER service device
aer_probe
aer_enable_rootport # RP only
set_downstream_devices_error_reporting
set_device_error_reporting # self (RP)
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for RP
pci_walk_bus
set_device_error_reporting
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for USP DSP

In a typical Root Port -> Endpoint hierarchy, the above:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port.

In a deeper Root Port -> Upstream Switch Port -> Downstream Switch Port -> Endpoint hierarchy:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Enables Error Reporting for both Switch Ports,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port,
- Disables Error Reporting for the Switch Ports when pcie_portdrv_probe() claims them. AER does not re-enable it because these are not Root Ports.

This patch now removes this call to pci_disable_pcie_error_reporting() from get_port_device_capability(), leaving the already enabled AER configuration intact. With this change, AER is enabled in the Root Port and the PCIe switch upstream and downstream ports. Only the PCIe Endpoints don't have AER enabled yet. A follow-up patch will take care of this Endpoint enabling.

Fixes 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
Cc Rafael J. Wysocki <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

---
v2:
- Enhance commit message as suggested by Bjorn

 drivers/pci/pcie/portdrv_core.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/pci/pcie/portdrv_core.c +1 -8
index f81c7be4d7d8..27b990cedb4c 100644
@@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 #ifdef CONFIG_PCIEAER
 	if (dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer)) {
+	    (pcie_ports_native || host->native_aer))
 		services |= PCIE_PORT_SERVICE_AER;
-
-		/*
-		 * Disable AER on this port in case it's been enabled by the
-		 * BIOS (the AER service driver will enable it when necessary).
-		 */
-		pci_disable_pcie_error_reporting(dev);
-	}
 #endif
 
 	/* Root Ports and Root Complex Event Collectors may generate PMEs */
Signature · 1 line
2.34.1
#3 lore raw

PATCH v2 2/2 PCI/AER: Enable AER on Endpoints as well

In reply to Stefan Roese

Currently, the PCIe AER subsystem does not enable AER in the PCIe Endpoints via the Device Control register. It's only done for the Root Port and all PCIe Ports in between the Root Port and the Endpoint(s). Some device drivers enable AER in their PCIe device by directly calling pci_enable_pcie_error_reporting(). But in most cases, AER is currently disabled in the PCIe Endpoints.

This patch enables AER on PCIe Endpoints now as well in set_device_error_reporting(). This will make the ad-hoc calls to pci_enable_pcie_error_reporting() superfluous.

Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

---
v2:
- New patch

 drivers/pci/pcie/aer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +2 -1
index 9fa1f97e5b27..385e2033d7b5 100644
@@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
 	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
 	    (type == PCI_EXP_TYPE_RC_EC) ||
 	    (type == PCI_EXP_TYPE_UPSTREAM) ||
-	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
+	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
+	    (type == PCI_EXP_TYPE_ENDPOINT)) {
 		if (enable)
 			pci_enable_pcie_error_reporting(dev);
 		else
Signature · 1 line
2.34.1
#0 lore raw

v2 cover Fully enable AER

While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.

Please note that these changes are quite invasie, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

Stefan Roese (2):
PCI/portdrv: Don't disable AER reporting in
get_port_device_capability()
PCI/AER: Enable AER on Endpoints as well

 drivers/pci/pcie/aer.c          | 3 ++-
 drivers/pci/pcie/portdrv_core.c | 9 +--------
 2 files changed, 3 insertions(+), 9 deletions(-)
Signature · 1 line
2.34.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] PCI/portdrv: Don't disable AER reporting in get_port_device_capability()

In reply to Stefan Roese
Quoted 56 lines · On Monday 17 January 2022 09:03:47 Stefan Roese wrote:

Testing has shown, that AER reporting is currently disabled in the DevCtl registers of all non Root Port PCIe devices on systems using pcie_ports_native || host->native_aer. Practically disabling AER completely in such systems. This is due to the fact that with commit 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization"), a call to pci_disable_pcie_error_reporting() was added *after* the PCIe AER setup was completed for the PCIe device tree.

Here a longer analysis about the currect status of AER enaling / disabling upon bootup provided by Bjorn:

pcie_portdrv_probe
pcie_port_device_register
get_port_device_capability
pci_disable_pcie_error_reporting
clear CERE NFERE FERE URRE # <-- disable for RP USP DSP
pcie_device_init
device_register # new AER service device
aer_probe
aer_enable_rootport # RP only
set_downstream_devices_error_reporting
set_device_error_reporting # self (RP)
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for RP
pci_walk_bus
set_device_error_reporting
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for USP DSP

In a typical Root Port -> Endpoint hierarchy, the above:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port.

In a deeper Root Port -> Upstream Switch Port -> Downstream Switch Port -> Endpoint hierarchy:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Enables Error Reporting for both Switch Ports,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port,
- Disables Error Reporting for the Switch Ports when pcie_portdrv_probe() claims them. AER does not re-enable it because these are not Root Ports.

This patch now removes this call to pci_disable_pcie_error_reporting() from get_port_device_capability(), leaving the already enabled AER configuration intact. With this change, AER is enabled in the Root Port and the PCIe switch upstream and downstream ports. Only the PCIe Endpoints don't have AER enabled yet. A follow-up patch will take care of this Endpoint enabling.

Quoted 9 lines
Fixes 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
Cc Rafael J. Wysocki <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>
Quoted 31 lines

---
v2:
- Enhance commit message as suggested by Bjorn

 drivers/pci/pcie/portdrv_core.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/pci/pcie/portdrv_core.c +1 -8
index f81c7be4d7d8..27b990cedb4c 100644
@@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 #ifdef CONFIG_PCIEAER
 	if (dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer)) {
+	    (pcie_ports_native || host->native_aer))
 		services |= PCIE_PORT_SERVICE_AER;
-
-		/*
-		 * Disable AER on this port in case it's been enabled by the
-		 * BIOS (the AER service driver will enable it when necessary).
-		 */
-		pci_disable_pcie_error_reporting(dev);
-	}
 #endif
 
 	/* Root Ports and Root Complex Event Collectors may generate PMEs */
Signature · 1 line
2.34.1
Reviewed-by on v2 2/2
#4 lore raw

Re: [PATCH v2 2/2] PCI/AER: Enable AER on Endpoints as well

In reply to Stefan Roese
Quoted 11 lines · On Monday 17 January 2022 09:03:48 Stefan Roese wrote:

Currently, the PCIe AER subsystem does not enable AER in the PCIe Endpoints via the Device Control register. It's only done for the Root Port and all PCIe Ports in between the Root Port and the Endpoint(s). Some device drivers enable AER in their PCIe device by directly calling pci_enable_pcie_error_reporting(). But in most cases, AER is currently disabled in the PCIe Endpoints.

This patch enables AER on PCIe Endpoints now as well in set_device_error_reporting(). This will make the ad-hoc calls to pci_enable_pcie_error_reporting() superfluous.

Quoted 7 lines
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>
Quoted 12 lines

---
v2:
- New patch

 drivers/pci/pcie/aer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +0 -0
index 9fa1f97e5b27..385e2033d7b5 100644

drivers/pci/pcie/aer.c static int set_device_error_reporting(struct pci_dev *dev, void *data) · line 1220

Quoted 6 lines
drivers/pci/pcie/aer.c +2 -1
@@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
 	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
 	    (type == PCI_EXP_TYPE_RC_EC) ||
 	    (type == PCI_EXP_TYPE_UPSTREAM) ||
-	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
+	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
+	    (type == PCI_EXP_TYPE_ENDPOINT)) {

Hm... maybe another question to discussion: Why enabling of AER is limited just to above PCIe port types? Why we do not want to enable it for _all_ PCIe devices? Currently in the above list are missing Legacy endpoints (which probably do not support AER and so do not have AER capability in config space), Root Complex Integrated Endpoints (these should provide AER supports too, right?), PCIe to PCI/X Bridges (these may generate its own AER errors) and PCI to PCIe Bridges (these are maybe complicated as subtree behind such bridges are regular PCIe devices and so could fully support AER but on legacy PCI bus there is probably no access to extended config space where is AER). But in all of these cases, are there any issues with enabling AER via function pci_enable_pcie_error_reporting()? For me it looks like that in the worst case dev just does not have AER capability in config space or extended config space is not accessible (which is same as no AER capability).

Quoted 6 lines
drivers/pci/pcie/aer.c +0 -0
 		if (enable)
 			pci_enable_pcie_error_reporting(dev);
 		else
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v2 2/2] PCI/AER: Enable AER on Endpoints as well

Quoted 48 lines · Pali Rohár
Quoted 11 lines · On Monday 17 January 2022 09:03:48 Stefan Roese wrote:

Currently, the PCIe AER subsystem does not enable AER in the PCIe Endpoints via the Device Control register. It's only done for the Root Port and all PCIe Ports in between the Root Port and the Endpoint(s). Some device drivers enable AER in their PCIe device by directly calling pci_enable_pcie_error_reporting(). But in most cases, AER is currently disabled in the PCIe Endpoints.

This patch enables AER on PCIe Endpoints now as well in set_device_error_reporting(). This will make the ad-hoc calls to pci_enable_pcie_error_reporting() superfluous.

Quoted 7 lines
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>
Quoted 12 lines

---
v2:
- New patch

  drivers/pci/pcie/aer.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +0 -0
index 9fa1f97e5b27..385e2033d7b5 100644

drivers/pci/pcie/aer.c static int set_device_error_reporting(struct pci_dev *dev, void *data) · line 1220

Quoted 6 lines
drivers/pci/pcie/aer.c +2 -1
@@ -1216,7 +1216,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)
  	if ((type == PCI_EXP_TYPE_ROOT_PORT) ||
  	    (type == PCI_EXP_TYPE_RC_EC) ||
  	    (type == PCI_EXP_TYPE_UPSTREAM) ||
-	    (type == PCI_EXP_TYPE_DOWNSTREAM)) {
+	    (type == PCI_EXP_TYPE_DOWNSTREAM) ||
+	    (type == PCI_EXP_TYPE_ENDPOINT)) {

Hm... maybe another question to discussion: Why enabling of AER is limited just to above PCIe port types? Why we do not want to enable it for _all_ PCIe devices? Currently in the above list are missing Legacy endpoints (which probably do not support AER and so do not have AER capability in config space), Root Complex Integrated Endpoints (these should provide AER supports too, right?), PCIe to PCI/X Bridges (these may generate its own AER errors) and PCI to PCIe Bridges (these are

Quoted 8 lines

maybe complicated as subtree behind such bridges are regular PCIe devices and so could fully support AER but on legacy PCI bus there is probably no access to extended config space where is AER). But in all of these cases, are there any issues with enabling AER via function pci_enable_pcie_error_reporting()? For me it looks like that in the worst case dev just does not have AER capability in config space or extended config space is not accessible (which is same as no AER capability).

I also had similar thoughts on this and was a bit unsure here. Perhaps Bjorn can also comment. But...

... I noticed that AER is still disabled for hot-plugged PCIe devices. As this code patch will not get called in this HP case. Right now I'm testing with this patch here, which could be used instead of this one from this mail:

PCI/AER: Enable AER on all PCIe devices supporting it

With this change, AER is now also enabled for hot-plugged PCIe devices as pci_aer_init() is also called upon hot-plugging of a PCIe device. When "pci=noaer" is selected, AER stays disabled of course.

drivers/pci/pcie/aer.c +0 -0
index 385e2033d7b5..94107309ef2d 100644
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
        pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, 

sizeof(u32) * n);

        pci_aer_clear_status(dev);

+
+ /* Enable AER if requested */
+ if (pci_aer_available())
+ pci_enable_pcie_error_reporting(dev);
}

This has the same effect of enabling AER in each PCIe Endpoint and also is available for hot-plugged Endpoints. If nobody objects, I'll probably use this patch as patch 2/2 in the next patchset version.

Comments welcome as always.

Thanks,
Stefan