Skip to contentSkip to conversation

Fully enable AER

PATCH Superseded Stefan Roese 4 participants 13 messages Jan 19, 2022 – Jan 20, 2022

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#1 lore raw

PATCH v3 1/2 PCI/portdrv: Don't disable AER reporting in get_port_device_capability()

In reply to Stefan Roese

Testing has shown, that AER reporting is currently disabled in the DevCtl registers of all non Root Port PCIe devices on systems using pcie_ports_native || host->native_aer. Practically disabling AER completely in such systems. This is due to the fact that with commit 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization"), a call to pci_disable_pcie_error_reporting() was added *after* the PCIe AER setup was completed for the PCIe device tree.

Here a longer analysis about the currect status of AER enaling / disabling upon bootup provided by Bjorn:

pcie_portdrv_probe
pcie_port_device_register
get_port_device_capability
pci_disable_pcie_error_reporting
clear CERE NFERE FERE URRE # <-- disable for RP USP DSP
pcie_device_init
device_register # new AER service device
aer_probe
aer_enable_rootport # RP only
set_downstream_devices_error_reporting
set_device_error_reporting # self (RP)
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for RP
pci_walk_bus
set_device_error_reporting
if (RP || USP || DSP)
pci_enable_pcie_error_reporting
set CERE NFERE FERE URRE # <-- enable for USP DSP

In a typical Root Port -> Endpoint hierarchy, the above:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port.

In a deeper Root Port -> Upstream Switch Port -> Downstream Switch Port -> Endpoint hierarchy:
- Disables Error Reporting for the Root Port,
- Enables Error Reporting for the Root Port,
- Enables Error Reporting for both Switch Ports,
- Does NOT enable Error Reporting for the Endpoint because it is not a Root Port or Switch Port,
- Disables Error Reporting for the Switch Ports when pcie_portdrv_probe() claims them. AER does not re-enable it because these are not Root Ports.

This patch now removes this call to pci_disable_pcie_error_reporting() from get_port_device_capability(), leaving the already enabled AER configuration intact. With this change, AER is enabled in the Root Port and the PCIe switch upstream and downstream ports. Only the PCIe Endpoints don't have AER enabled yet. A follow-up patch will take care of this Endpoint enabling.

Fixes 2bd50dd800b5 ("PCI: PCIe: Disable PCIe port services during port initialization")
Cc Rafael J. Wysocki <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

---
v3:
- Added RB tag from Pali

v2:
- Enhance commit message as suggested by Bjorn

 drivers/pci/pcie/portdrv_core.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/pci/pcie/portdrv_core.c +1 -8
index f81c7be4d7d8..27b990cedb4c 100644
@@ -244,15 +244,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 #ifdef CONFIG_PCIEAER
 	if (dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer)) {
+	    (pcie_ports_native || host->native_aer))
 		services |= PCIE_PORT_SERVICE_AER;
-
-		/*
-		 * Disable AER on this port in case it's been enabled by the
-		 * BIOS (the AER service driver will enable it when necessary).
-		 */
-		pci_disable_pcie_error_reporting(dev);
-	}
 #endif
 
 	/* Root Ports and Root Complex Event Collectors may generate PMEs */
Signature · 1 line
2.34.1
#0 lore raw

v3 cover Fully enable AER

While working on AER support on a ZynqMP based system, which has some PCIe Device connected via a PCIe switch, problems with AER enabling in the Device Control registers of all PCIe devices but the Root Port. In fact, only the Root Port has AER enabled right now. This patch set now fixes this problem by first fixing the AER enabing in the interconnected PCIe switches between the Root Port and the PCIe devices and in a 2nd patch, also enabling AER in the PCIe Endpoints.

Please note that these changes are quite invasive, as with these patches applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

Cc Rafael J. Wysocki <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

Stefan Roese (2):
PCI/portdrv: Don't disable AER reporting in
get_port_device_capability()
PCI/AER: Enable AER on all PCIe devices supporting it

 drivers/pci/pcie/aer.c          | 4 ++++
 drivers/pci/pcie/portdrv_core.c | 9 +--------
 2 files changed, 5 insertions(+), 8 deletions(-)
Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/2 PCI/AER: Enable AER on all PCIe devices supporting it

With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.

Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

When "pci=noaer" is selected, AER stays disabled of course.

Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

---
v3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done in pci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.

 drivers/pci/pcie/aer.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/pci/pcie/aer.c +4 -0
index 9fa1f97e5b27..01a25e4a5168 100644
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
 	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
 	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
 }
 
 void pci_aer_exit(struct pci_dev *dev)
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

Quoted 8 lines · On Wednesday 19 January 2022 10:22:00 Stefan Roese wrote:

With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.

Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

When "pci=noaer" is selected, AER stays disabled of course.

Hello Stefan! I was thinking more about this change and I'm not sure what happens if AER-capable PCIe device is hotplugged into some PCIe switch connected in the PCIe hierarchy where Root Port is not AER-capable (e.g. current linux implementation of pci-aardvark.c and pci-mvebu.c). My feeling is that in this case AER should not be enabled as there is nobody who can deliver AER interrupt to the OS. But I really do not know what is supposed from kernel AER driver, so lets wait for Bjorn reply.

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Quoted 35 lines
Cc Bjorn Helgaas <[email protected]>
Cc Pali Rohár <[email protected]>
Cc Bharat Kumar Gogada <[email protected]>
Cc Michal Simek <[email protected]>
Cc Yao Hongbo <[email protected]>
Cc Naveen Naidu <[email protected]>

---
v3:
- New patch, replacing the "old" 2/2 patch
Now enabling of AER for each PCIe device is done in pci_aer_init(), which also makes sure that AER is enabled in each PCIe device even when it's hot-plugged.

 drivers/pci/pcie/aer.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/pci/pcie/aer.c +4 -0
index 9fa1f97e5b27..01a25e4a5168 100644
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
 	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
 	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
 }
 
 void pci_aer_exit(struct pci_dev *dev)
Signature · 1 line
2.34.1
#9 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

In reply to Stefan Roese

drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 394

Quoted 9 lines · Stefan Roese
drivers/pci/pcie/aer.c +4 -0
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
 	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
 	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
 }

Hasn't it always been the device specific driver's responsibility to call this function?

#10 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

Quoted 4 lines · Keith Busch
Quoted 3 lines · Stefan Roese
drivers/pci/pcie/aer.c +0 -0
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
 	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
Quoted 9 lines

drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 394

Quoted 6 lines
drivers/pci/pcie/aer.c +4 -0
 	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
 }

Hasn't it always been the device specific driver's responsibility to call this function?

So far it has been done by the driver, because the PCI core doesn't do it. But is there a reason it should be done by the driver? It doesn't seem necessarily device-specific.

Bjorn

#11 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

Quoted 15 lines · Bjorn Helgaas
Quoted 4 lines · Keith Busch
Quoted 3 lines · Stefan Roese
drivers/pci/pcie/aer.c +0 -0
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
 	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
Quoted 9 lines

drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 394

Quoted 6 lines
drivers/pci/pcie/aer.c +4 -0
 	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
 }

Hasn't it always been the device specific driver's responsibility to call this function?

Quoted 3 lines

So far it has been done by the driver, because the PCI core doesn't do it. But is there a reason it should be done by the driver? It doesn't seem necessarily device-specific.

I was thinking the device driver knows if it provides .err_handler callbacks in order to respond to AER handling, so it would know if it is ready for its device to enable error reporting. But I guess it doesn't really matter if the driver provides callbacks anyway.

#4 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

In reply to Pali Rohár
Quoted 10 lines · Pali Rohár
Quoted 8 lines · On Wednesday 19 January 2022 10:22:00 Stefan Roese wrote:

With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.

Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

When "pci=noaer" is selected, AER stays disabled of course.

Quoted 8 lines

Hello Stefan! I was thinking more about this change and I'm not sure what happens if AER-capable PCIe device is hotplugged into some PCIe switch connected in the PCIe hierarchy where Root Port is not AER-capable (e.g. current linux implementation of pci-aardvark.c and pci-mvebu.c). My feeling is that in this case AER should not be enabled as there is nobody who can deliver AER interrupt to the OS. But I really do not know what is supposed from kernel AER driver, so lets wait for Bjorn reply.

But what happens right now, when a device driver like the NVMe driver calls pci_enable_pcie_error_reporting() ? There is also no checking, if the connected Root Port or some switch / bridge in-between supports AER or not. IIUTC, this is identical to what this patch here does. Enable AER in the device and if the upstream infrastructure does not support AER, then the AER event will just not be received by the Kernel. Which is most likely not worse than not enabling AER at all on this device. Or am I missing something?

Quoted 5 lines

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Agreed, this is another area to look into. Not sure if it's okay to address this, once this patch-set has been accepted (if it will be).

Thanks,
Stefan

#12 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

In reply to Keith Busch
Quoted 16 lines · Keith Busch
Quoted 15 lines · Bjorn Helgaas
Quoted 13 lines · Keith Busch

drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 394

Quoted 9 lines · Stefan Roese
drivers/pci/pcie/aer.c +4 -0
@@ -387,6 +387,10 @@ void pci_aer_init(struct pci_dev *dev)
  	pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) * n);
 
  	pci_aer_clear_status(dev);
+
+	/* Enable AER if requested */
+	if (pci_aer_available())
+		pci_enable_pcie_error_reporting(dev);
  }

Hasn't it always been the device specific driver's responsibility to call this function?

Quoted 8 lines
Quoted 3 lines

So far it has been done by the driver, because the PCI core doesn't do it. But is there a reason it should be done by the driver? It doesn't seem necessarily device-specific.

I was thinking the device driver knows if it provides .err_handler callbacks in order to respond to AER handling, so it would know if it is ready for its device to enable error reporting. But I guess it doesn't really matter if the driver provides callbacks anyway.

That's my understanding as well.

Thanks,
Stefan

#5 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

In reply to Stefan Roese
Quoted 20 lines · On Thursday 20 January 2022 08:31:31 Stefan Roese wrote:
Quoted 10 lines · Pali Rohár
Quoted 8 lines · On Wednesday 19 January 2022 10:22:00 Stefan Roese wrote:

With this change, AER is now enabled on all PCIe devices, also when the PCIe device is hot-plugged.

Please note that this change is quite invasive, as with this patch applied, AER now will be enabled in the Device Control registers of all available PCIe Endpoints, which currently is not the case.

When "pci=noaer" is selected, AER stays disabled of course.

Quoted 8 lines

Hello Stefan! I was thinking more about this change and I'm not sure what happens if AER-capable PCIe device is hotplugged into some PCIe switch connected in the PCIe hierarchy where Root Port is not AER-capable (e.g. current linux implementation of pci-aardvark.c and pci-mvebu.c). My feeling is that in this case AER should not be enabled as there is nobody who can deliver AER interrupt to the OS. But I really do not know what is supposed from kernel AER driver, so lets wait for Bjorn reply.

Quoted 8 lines

But what happens right now, when a device driver like the NVMe driver calls pci_enable_pcie_error_reporting() ? There is also no checking, if the connected Root Port or some switch / bridge in-between supports AER or not. IIUTC, this is identical to what this patch here does. Enable AER in the device and if the upstream infrastructure does not support AER, then the AER event will just not be received by the Kernel. Which is most likely not worse than not enabling AER at all on this device. Or am I missing something?

You are right!

Seems that AER code has lot of candidates for followup fixes/cleanups...

Quoted 11 lines
Quoted 5 lines

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Agreed, this is another area to look into. Not sure if it's okay to address this, once this patch-set has been accepted (if it will be).

Thanks,
Stefan

#6 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

In reply to Stefan Roese
Quoted 10 lines · Stefan Roese
Quoted 5 lines · Pali Rohár

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Agreed, this is another area to look into. Not sure if it's okay to address this, once this patch-set has been accepted (if it will be).

ECRC might be something that could be peeled off first to reduce the complexity of AER itself.

The ECRC capability and enable bits are in the AER Capability, so I think it should be moved to pci_aer_init() so it happens for every device as we enumerate it.

As far as I can tell, there is no requirement that every device in the path support ECRC, so it can be enabled independently for each device. I think devices that don't support ECRC checking must handle TLPs with ECRC without error.

Per Table 6-5, ECRC check failures result in a device logging the prefix/header of the TLP and sending ERR_NONFATAL or ERR_COR. I think this is useful regardless of whether AER interrupts are enabled because error information is logged where the ECRC failure was detected.

Bjorn

#7 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Stefan Roese
Quoted 5 lines · Pali Rohár

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Agreed, this is another area to look into. Not sure if it's okay to address this, once this patch-set has been accepted (if it will be).

Quoted 6 lines

ECRC might be something that could be peeled off first to reduce the complexity of AER itself.

The ECRC capability and enable bits are in the AER Capability, so I think it should be moved to pci_aer_init() so it happens for every device as we enumerate it.

Just that there is no misunderstanding: You are thinking about something like this:

drivers/pci/pcie/aer.c +0 -0
index 9fa1f97e5b27..5585fefc4d0e 100644
@@ -387,6 +387,9 @@ void pci_aer_init(struct pci_dev *dev)
        pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, 

sizeof(u32) * n);

        pci_aer_clear_status(dev);

+
+ /* Enable ECRC checking if enabled and configured */
+ pcie_set_ecrc_checking(dev);
}

void pci_aer_exit(struct pci_dev *dev)
@@ -1223,9 +1226,6 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)

                        pci_disable_pcie_error_reporting(dev);
        }

- if (enable)
- pcie_set_ecrc_checking(dev);
-

        return 0;

}

Perhaps as patch 1/3 in this patch series? Or as some completely separate patch?

Thanks,
Stefan

Quoted 5 lines

As far as I can tell, there is no requirement that every device in the path support ECRC, so it can be enabled independently for each device. I think devices that don't support ECRC checking must handle TLPs with ECRC without error.

Quoted 8 lines

Per Table 6-5, ECRC check failures result in a device logging the prefix/header of the TLP and sending ERR_NONFATAL or ERR_COR. I think this is useful regardless of whether AER interrupts are enabled because error information is logged where the ECRC failure was detected.

Bjorn

Viele Grüße,
Stefan Roese

Signature · 3 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-51 Fax: (+49)-8142-66989-80 Email: [email protected]
#8 lore raw

Re: [PATCH v3 2/2] PCI/AER: Enable AER on all PCIe devices supporting it

Quoted 49 lines · Stefan Roese
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Stefan Roese
Quoted 5 lines · Pali Rohár

And when you opened this issue with hotplugging, another thing for followup changes in future is calling pcie_set_ecrc_checking() function to align ECRC state of newly hotplugged device with "pci=ecrc=..." cmdline option. As currently it is done only at that function set_device_error_reporting().

Agreed, this is another area to look into. Not sure if it's okay to address this, once this patch-set has been accepted (if it will be).

Quoted 6 lines

ECRC might be something that could be peeled off first to reduce the complexity of AER itself.

The ECRC capability and enable bits are in the AER Capability, so I think it should be moved to pci_aer_init() so it happens for every device as we enumerate it.

Just that there is no misunderstanding: You are thinking about something like this:

drivers/pci/pcie/aer.c +3 -0
index 9fa1f97e5b27..5585fefc4d0e 100644
@@ -387,6 +387,9 @@ void pci_aer_init(struct pci_dev *dev)
        pci_add_ext_cap_save_buffer(dev, PCI_EXT_CAP_ID_ERR, sizeof(u32) *
 n);
 
        pci_aer_clear_status(dev);
+
+       /* Enable ECRC checking if enabled and configured */
+       pcie_set_ecrc_checking(dev);
 }
 

void pci_aer_exit(struct pci_dev *dev)

drivers/pci/pcie/aer.c +0 -3
@@ -1223,9 +1226,6 @@ static int set_device_error_reporting(struct pci_dev
 *dev, void *data)
                        pci_disable_pcie_error_reporting(dev);
        }
 
-       if (enable)
-               pcie_set_ecrc_checking(dev);
-
        return 0;
 }
Quoted 2 lines

Perhaps as patch 1/3 in this patch series? Or as some completely separate patch?

Yes. Probably as 1/3, since subsequent patches may depend on this one, or at least may not apply cleanly without this one.

Bjorn