Add RCEC handling to PCI/AER
Subsystems (2)
cover Add RCEC handling to PCI/AER
Root Complex Event Collectors (RCEC) provide support for terminating error and PME messages from Root Complex Integrated Endpoints (RCiEPs). An RCEC resides on a Bus in the Root Complex. Multiple RCECs can in fact reside on a single bus. An RCEC will explicitly declare supported RCiEPs through the Root Complex Endpoint Association Extended Capability.
(See PCIe 5.0-1, sections 1.3.2.3 (RCiEP), and 7.9.10 (RCEC Ext. Cap.))
The kernel lacks handling for these RCECs and the error messages received from their respective associated RCiEPs. More recently, a new CPU interconnect, Compute eXpress Link (CXL) depends on RCEC capabilities for purposes of error messaging from CXL 1.1 supported RCiEP devices.
DocLink: https://www.computeexpresslink.org/
This use case is not limited to CXL. Existing hardware today includes support for RCECs, such as the Denverton microserver product family. Future hardware will be forthcoming.
(See Intel Document, Order number: 33061-003US)
So services such as AER or PME could be associated with an RCEC driver. In the case of CXL, if an RCiEP (i.e., CXL 1.1 device) is associated with a platform's RCEC it shall signal PME and AER error conditions through that RCEC.
Towards the above use cases, add the missing RCEC class and extend the PCIe Root Port and service drivers to allow association of RCiEPs to their respective parent RCEC and facilitate handling of terminating error and PME messages.
TESTING:
Results:
1) Show RCiEPs which are associated with RCECs:
Run dmesg | grep "RCiEP"
Log:
[ 8.981698] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e8:01.0
[ 8.988830] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e8:02.0
[ 8.995956] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e9:00.0
[ 9.023034] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:01.0
[ 9.030159] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:02.0
[ 9.037282] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ee:00.0
[ 9.064294] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:01.0
[ 9.071409] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:02.0
[ 9.078526] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f3:00.0
[ 9.105535] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:01.0
[ 9.112652] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:02.0
[ 9.119774] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f8:00.0
2) Inject a correctable error to the RCiEP 0000:e9:00.0
Run ./aer_inject <a parameter file as below>:
AER
PCI_ID 0000:e9:00.0
COR_STATUS BAD_TLP
HEADER_LOG 0 1 2 3
Log:
[ 253.248362] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000040/00000000 into device 0000:e9:00.0
[ 253.260656] pcieport 0000:e8:00.4: AER: Corrected error received: 0000:e9:00.0
[ 253.269919] pci 0000:e9:00.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 253.282549] pci 0000:e9:00.0: AER: device [8086:4940] error status/mask=00000040/00002000
[ 253.293937] pci 0000:e9:00.0: AER: [ 6] BadTLP
3) Inject a non-fatal error to the RCiEP 0000:e8:01.0
Run ./aer_inject <a parameter file as below>:
AER
PCI_ID 0000:e8:01.0
UNCOR_STATUS COMP_ABORT
HEADER_LOG 0 1 2 3
Log:
[ 288.405326] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000000/00008000 into device 0000:e8:01.0
[ 288.416881] pcieport 0000:e8:00.4: AER: Uncorrected (Non-Fatal) error received: 0000:e8:01.0
[ 288.427487] igen6_edac 0000:e8:01.0: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Completer ID)
[ 288.442098] igen6_edac 0000:e8:01.0: AER: device [8086:0b25] error status/mask=00008000/00100000
[ 288.452869] igen6_edac 0000:e8:01.0: AER: [15] CmpltAbrt
[ 288.461118] igen6_edac 0000:e8:01.0: AER: TLP Header: 00000000 00000001 00000002 00000003
[ 288.471192] igen6_edac 0000:e8:01.0: AER: device recovery successful
4) Inject a fatal error to the RCiEP 0000:ed:01.0
Run ./aer_inject <a parameter file as below>:
AER
PCI_ID 0000:ed:01.0
UNCOR_STATUS MALF_TLP
HEADER_LOG 0 1 2 3
Log:
[ 535.537281] pcieport 0000:ed:00.4: aer_inject: Injecting errors 00000000/00040000 into device 0000:ed:01.0
[ 535.551911] pcieport 0000:ed:00.4: AER: Uncorrected (Fatal) error received: 0000:ed:01.0
[ 535.561556] igen6_edac 0000:ed:01.0: AER: PCIe Bus Error: severity=Uncorrected (Fatal), type=Inaccessible, (Unregistered Agent ID)
[ 535.684964] igen6_edac 0000:ed:01.0: AER: device recovery successfulJonathan Cameron (1):
PCI/AER: Extend AER error handling to RCECs
Qiuxu Zhuo (6):
pci_ids: Add class code and extended capability for RCEC
PCI: Extend Root Port Driver to support RCEC
PCI/portdrv: Add pcie_walk_rcec() to walk RCiEPs associated with RCEC PCI/AER: Apply function level reset to RCiEP on fatal error
PCI: Add 'rcec' field to pci_dev for associated RCiEPs PCI/AER: Add RCEC AER error injection support
Sean V Kelley (2):
PCI/AER: Add RCEC AER handling
PCI/PME: Add RCEC PME handling
drivers/pci/pcie/aer.c | 43 ++++++++++++----- drivers/pci/pcie/aer_inject.c | 5 +- drivers/pci/pcie/err.c | 85 +++++++++++++++++++++++++++------ drivers/pci/pcie/pme.c | 15 ++++-- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++ drivers/pci/pcie/portdrv_pci.c | 20 +++++++- include/linux/pci.h | 3 ++ include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++ 10 files changed, 232 insertions(+), 31 deletions(-)
--
2.27.0
RFC PATCH 1/9 pci_ids: Add class code and extended capability for RCEC
From: Qiuxu Zhuo <[email protected]>
A PCIe Root Complex Event Collector(RCEC) has the base class 0x08, sub-class 0x07, and programming interface 0x00. Add the class code 0x0807 to identify RCEC devices and add the defines for the RCEC Endpoint Association Extended Capability.
See PCI Express Base Specification, version 5.0-1, section "1.3.4 Root Complex Event Collector" and section "7.9.10 Root Complex Event Collector Endpoint Association Extended Capability"
--- include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++++++ 2 files changed, 8 insertions(+)
include/linux/pci_ids.h +1 -0
@@ -81,6 +81,7 @@#define PCI_CLASS_SYSTEM_RTC 0x0803#define PCI_CLASS_SYSTEM_PCI_HOTPLUG 0x0804#define PCI_CLASS_SYSTEM_SDHCI 0x0805 +#define PCI_CLASS_SYSTEM_RCEC 0x0807#define PCI_CLASS_SYSTEM_OTHER 0x0880#define PCI_BASE_CLASS_INPUT 0x09
include/uapi/linux/pci_regs.h +7 -0
@@ -828,6 +828,13 @@#define PCI_PWR_CAP_BUDGET (x) ((x) & 1)/* Included in system budget */ #define PCI_EXT_CAP_PWR_SIZEOF 16 +/* Root Complex Event Collector Endpoint Association */ +#define PCI_RCEC_RCIEP_BITMAP 4/* Associated Bitmap for RCiEPs */ +#define PCI_RCEC_BUSN 8/* RCEC Associated Bus Numbers */ +#define PCI_RCEC_BUSN_REG_VER 0x02/* Least capability version that BUSN present */ +#define PCI_RCEC_BUSN_NEXT (x) (((x) >> 8) & 0xff) +#define PCI_RCEC_BUSN_LAST (x) (((x) >> 16) & 0xff) +/* Vendor-Specific (VSEC, PCI_EXT_CAP_ID_VNDR) */ #define PCI_VNDR_HEADER 4/* Vendor-Specific Header */ #define PCI_VNDR_HEADER_ID (x) ((x) & 0xffff)
Signature · 1 line
2.27.0
Re: [RFC PATCH 1/9] pci_ids: Add class code and extended capability for RCEC
Quoted 10 lines · On Fri, 24 Jul 2020 10:22:15 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
A PCIe Root Complex Event Collector(RCEC) has the base class
0x08, sub-class0x07, and programming interface0x00. Add the class code0x0807to identify RCEC devices and add the defines for the RCEC Endpoint Association Extended Capability.See PCI Express Base Specification, version 5.0-1, section "1.3.4 Root Complex Event Collector" and section "7.9.10 Root Complex Event Collector Endpoint Association Extended Capability"
Add a reference to the document
"PCI Code and ID Assignment Specification"
for the class number.
From the change log on latest version seems like it's been there since version 1.4.
There is a worrying note (bottom of page 16 of 1.12 version of that docs) in there that says some older specs used 0x0806 for RCECs and that we should use the port type field to actually check if we have one.
Hopefully we won't encounter any of those in the wild.
Otherwise, it's exactly what the spec says.
We could bike shed on naming choices, but the ones you have seem clear enough to me.
FWIW
Jonathan
Quoted 37 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++++++ 2 files changed, 8 insertions(+)include/linux/pci_ids.h +1
-0@@ -81,6 +81,7 @@#define PCI_CLASS_SYSTEM_RTC 0x0803#define PCI_CLASS_SYSTEM_PCI_HOTPLUG 0x0804#define PCI_CLASS_SYSTEM_SDHCI 0x0805 +#define PCI_CLASS_SYSTEM_RCEC 0x0807#define PCI_CLASS_SYSTEM_OTHER 0x0880#define PCI_BASE_CLASS_INPUT 0x09include/uapi/linux/pci_regs.h +7
-0@@ -828,6 +828,13 @@#define PCI_PWR_CAP_BUDGET (x) ((x) & 1)/* Included in system budget */ #define PCI_EXT_CAP_PWR_SIZEOF 16 +/* Root Complex Event Collector Endpoint Association */ +#define PCI_RCEC_RCIEP_BITMAP 4/* Associated Bitmap for RCiEPs */ +#define PCI_RCEC_BUSN 8/* RCEC Associated Bus Numbers */ +#define PCI_RCEC_BUSN_REG_VER 0x02/* Least capability version that BUSN present */ +#define PCI_RCEC_BUSN_NEXT (x) (((x) >> 8) & 0xff) +#define PCI_RCEC_BUSN_LAST (x) (((x) >> 16) & 0xff) +/* Vendor-Specific (VSEC, PCI_EXT_CAP_ID_VNDR) */ #define PCI_VNDR_HEADER 4/* Vendor-Specific Header */ #define PCI_VNDR_HEADER_ID (x) ((x) & 0xffff)
Re: [RFC PATCH 1/9] pci_ids: Add class code and extended capability for RCEC
Quoted 10 lines · On Mon, 27 Jul 2020 11:00:10 +0100 Jonathan Cameron…
Quoted 7 lines · On Fri, 24 Jul 2020 10:22:15 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
A PCIe Root Complex Event Collector(RCEC) has the base class
0x08, sub-class0x07, and programming interface0x00. Add the class code0x0807to identify RCEC devices and add the defines for the RCEC Endpoint Association Extended Capability.
Quoted 7 lines
Quoted 3 lines
See PCI Express Base Specification, version 5.0-1, section "1.3.4 Root Complex Event Collector" and section "7.9.10 Root Complex Event Collector Endpoint Association Extended Capability"
Add a reference to the document
"PCI Code and ID Assignment Specification"
for the class number.
Actually probably no need. I'd somehow managed to fail to notice the class code is also given in section 1.3.4 of the main spec.
Quoted 57 lines
From the change log on latest version seems like it's been there since version 1.4.
There is a worrying note (bottom of page 16 of 1.12 version of that docs) in there that says some older specs used
0x0806for RCECs and that we should use the port type field to actually check if we have one.Hopefully we won't encounter any of those in the wild.
Otherwise, it's exactly what the spec says.
We could bike shed on naming choices, but the ones you have seem clear enough to me.FWIW
Reviewed-by Jonathan Cameron <[email protected]>Jonathan
Quoted 37 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++++++ 2 files changed, 8 insertions(+)include/linux/pci_ids.h +1
-0@@ -81,6 +81,7 @@#define PCI_CLASS_SYSTEM_RTC 0x0803#define PCI_CLASS_SYSTEM_PCI_HOTPLUG 0x0804#define PCI_CLASS_SYSTEM_SDHCI 0x0805 +#define PCI_CLASS_SYSTEM_RCEC 0x0807#define PCI_CLASS_SYSTEM_OTHER 0x0880#define PCI_BASE_CLASS_INPUT 0x09include/uapi/linux/pci_regs.h +7
-0@@ -828,6 +828,13 @@#define PCI_PWR_CAP_BUDGET (x) ((x) & 1)/* Included in system budget */ #define PCI_EXT_CAP_PWR_SIZEOF 16 +/* Root Complex Event Collector Endpoint Association */ +#define PCI_RCEC_RCIEP_BITMAP 4/* Associated Bitmap for RCiEPs */ +#define PCI_RCEC_BUSN 8/* RCEC Associated Bus Numbers */ +#define PCI_RCEC_BUSN_REG_VER 0x02/* Least capability version that BUSN present */ +#define PCI_RCEC_BUSN_NEXT (x) (((x) >> 8) & 0xff) +#define PCI_RCEC_BUSN_LAST (x) (((x) >> 16) & 0xff) +/* Vendor-Specific (VSEC, PCI_EXT_CAP_ID_VNDR) */ #define PCI_VNDR_HEADER 4/* Vendor-Specific Header */ #define PCI_VNDR_HEADER_ID (x) ((x) & 0xffff)
Re: [RFC PATCH 1/9] pci_ids: Add class code and extended capability for RCEC
Quoted 17 lines · Jonathan Cameron
Quoted 14 lines · On Mon, 27 Jul 2020 11:00:10 +0100 Jonathan Cameron…
Quoted 10 lines · On Fri, 24 Jul 2020 10:22:15 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
A PCIe Root Complex Event Collector(RCEC) has the base class
0x08, sub-class0x07, and programming interface0x00. Add the class code0x0807to identify RCEC devices and add the defines for the RCEC Endpoint Association Extended Capability.See PCI Express Base Specification, version 5.0-1, section "1.3.4 Root Complex Event Collector" and section "7.9.10 Root Complex Event Collector Endpoint Association Extended Capability"
Quoted 6 lines
Quoted 3 lines
Add a reference to the document
"PCI Code and ID Assignment Specification"
for the class number.Actually probably no need. I'd somehow managed to fail to notice the class code is also given in section 1.3.4 of the main spec.
Okay, will leave as is.
Thanks,
Sean
Quoted 60 lines
Quoted 60 lines
From the change log on latest version seems like it's been there since version 1.4.
There is a worrying note (bottom of page 16 of 1.12 version of that docs) in there that says some older specs used
0x0806for RCECs and that we should use the port type field to actually check if we have one.Hopefully we won't encounter any of those in the wild.
Otherwise, it's exactly what the spec says.
We could bike shed on naming choices, but the ones you have seem clear enough to me.FWIW
Reviewed-by Jonathan Cameron <[email protected]>Jonathan
Quoted 39 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++++++ 2 files changed, 8 insertions(+)include/linux/pci_ids.h +1
-0@@ -81,6 +81,7 @@#define PCI_CLASS_SYSTEM_RTC 0x0803#define PCI_CLASS_SYSTEM_PCI_HOTPLUG 0x0804#define PCI_CLASS_SYSTEM_SDHCI 0x0805 +#define PCI_CLASS_SYSTEM_RCEC 0x0807#define PCI_CLASS_SYSTEM_OTHER 0x0880#define PCI_BASE_CLASS_INPUT 0x09include/uapi/linux/pci_regs.h +0
-0@@ -828,6 +828,13 @@#define PCI_PWR_CAP_BUDGET (x) ((x) & 1)/* Included in system budget */
#definePCI_EXT_CAP_PWR_SIZEOF16include/uapi/linux/pci_regs.h +4
-0+/* Root Complex Event Collector Endpoint Association */ +#define PCI_RCEC_RCIEP_BITMAP 4/* Associated Bitmap for RCiEPs */ +#define PCI_RCEC_BUSN 8/* RCEC Associated Bus Numbers */ +#define PCI_RCEC_BUSN_REG_VER 0x02/* Least capability version that BUSN present */
include/uapi/linux/pci_regs.h +3
-0+#define PCI_RCEC_BUSN_NEXT (x) (((x) >> 8) & 0xff) +#define PCI_RCEC_BUSN_LAST (x) (((x) >> 16) & 0xff) +/* Vendor-Specific (VSEC, PCI_EXT_CAP_ID_VNDR) */ #define PCI_VNDR_HEADER 4/* Vendor-Specific Header */ #define PCI_VNDR_HEADER_ID (x) ((x) & 0xffff)
RFC PATCH 2/9 PCI: Extend Root Port Driver to support RCEC
In reply to Sean V KelleyFrom: Qiuxu Zhuo <[email protected]>
If a Root Complex Integrated Endpoint (RCiEP) is implemented, errors may optionally be sent to a corresponding Root Complex Event Collector (RCEC). Each RCiEP must be associated with no more than one RCEC. Interface errors are reported to the OS by RCECs.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
Given the commonality with Root Ports and the need to also support AER and PME services for RCECs, extend the Root Port driver to support RCEC devices through the addition of the RCEC Class ID to the driver structure.
--- drivers/pci/pcie/portdrv_pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/pci/pcie/portdrv_pci.c +4 -1
@@ -106,7 +106,8 @@ static int pcie_portdrv_probe(struct pci_dev *dev,if (!pci_is_pcie (dev) || ((pci_pcie_type (dev) !=PCI_EXP_TYPE_ROOT_PORT ) && (pci_pcie_type (dev) !=PCI_EXP_TYPE_UPSTREAM ) && - (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ))) + (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ) && + (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC )))return -ENODEV ; status =pcie_port_device_register (dev);
@@ -195,6 +196,8 @@ static const struct pci_device_id port_pci_ids[] = { { PCI_DEVICE_CLASS(((PCI_CLASS_BRIDGE_PCI << 8) | 0x00), ~0) },/* subtractive decode PCI-to-PCI bridge, class type is 060401h */ { PCI_DEVICE_CLASS(((PCI_CLASS_BRIDGE_PCI << 8) | 0x01), ~0) }, +/* handle any Root Complex Event Collector */ + { PCI_DEVICE_CLASS(((PCI_CLASS_SYSTEM_RCEC << 8) | 0x00), ~0) }, { }, };
Signature · 1 line
2.27.0
Re: [RFC PATCH 2/9] PCI: Extend Root Port Driver to support RCEC
Quoted 14 lines · On Fri, 24 Jul 2020 10:22:16 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
If a Root Complex Integrated Endpoint (RCiEP) is implemented, errors may optionally be sent to a corresponding Root Complex Event Collector (RCEC). Each RCiEP must be associated with no more than one RCEC. Interface errors are reported to the OS by RCECs.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
Quoted 5 lines
Given the commonality with Root Ports and the need to also support AER and PME services for RCECs, extend the Root Port driver to support RCEC devices through the addition of the RCEC Class ID to the driver structure.
Hi.
I'm surprised it ended up this simple :) Seems we are a bit lucky that the existing code is rather flexible on what is there and what isn't and that there is never any reason to directly touch the various type1 specific registers (given as I read the spec, an RCEC uses a type0 config space header unlike the ports).
Given you mention PME, it's probably worth noting (I think) you aren't actually enabling the service yet as there is a check in that path on whether we have a root port or not.
https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/pci/pcie/portdrv_core.c#L241
Thanks,
Jonathan
Quoted 30 lines
Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>--- drivers/pci/pcie/portdrv_pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/pci/pcie/portdrv_pci.c +4
-1@@ -106,7 +106,8 @@ static int pcie_portdrv_probe(struct pci_dev *dev,if (!pci_is_pcie (dev) || ((pci_pcie_type (dev) !=PCI_EXP_TYPE_ROOT_PORT ) && (pci_pcie_type (dev) !=PCI_EXP_TYPE_UPSTREAM ) && - (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ))) + (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ) && + (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC )))return -ENODEV ; status =pcie_port_device_register (dev);@@ -195,6 +196,8 @@ static const struct pci_device_id port_pci_ids[] = { { PCI_DEVICE_CLASS(((PCI_CLASS_BRIDGE_PCI << 8) | 0x00), ~0) },/* subtractive decode PCI-to-PCI bridge, class type is 060401h */ { PCI_DEVICE_CLASS(((PCI_CLASS_BRIDGE_PCI << 8) | 0x01), ~0) }, +/* handle any Root Complex Event Collector */ + { PCI_DEVICE_CLASS(((PCI_CLASS_SYSTEM_RCEC << 8) | 0x00), ~0) }, { }, };
Re: [RFC PATCH 2/9] PCI: Extend Root Port Driver to support RCEC
Quoted 22 lines · Jonathan Cameron
Quoted 14 lines · On Fri, 24 Jul 2020 10:22:16 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
If a Root Complex Integrated Endpoint (RCiEP) is implemented, errors may optionally be sent to a corresponding Root Complex Event Collector (RCEC). Each RCiEP must be associated with no more than one RCEC. Interface errors are reported to the OS by RCECs.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
Quoted 5 lines
Given the commonality with Root Ports and the need to also support AER and PME services for RCECs, extend the Root Port driver to support RCEC devices through the addition of the RCEC Class ID to the driver structure.
Quoted 7 lines
Hi.
I'm surprised it ended up this simple :) Seems we are a bit lucky that the existing code is rather flexible on what is there and what isn't and that there is never any reason to directly touch the various type1 specific registers (given as I read the spec, an RCEC uses a type0 config space header unlike the ports).
Which is part of the reason why I chose to refer to it as an RFC, because it seemed simpler and I was unsure if we were missing anything, and it turns out we were. Unfortunately, to avoid churn, I’ve left quite a bit of comments/naming intact with “root”/“port” terms.
Quoted 5 lines
Given you mention PME, it's probably worth noting (I think) you aren't actually enabling the service yet as there is a check in that path on whether we have a root port or not.
https://elixir.bootlin.com/linux/v5.8-rc4/source/drivers/pci/pcie/portdrv_core.c#L241
Good catch, testing has been only done at this point with AER injection. Will correct.
Thanks,
Sean
Quoted 38 lines
Thanks,
Jonathan
Quoted 32 lines
Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>--- drivers/pci/pcie/portdrv_pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/pci/pcie/portdrv_pci.c +2
-1@@ -106,7 +106,8 @@ static int pcie_portdrv_probe(struct pci_dev *dev, if (!pci_is_pcie(dev) || ((pci_pcie_type (dev) !=PCI_EXP_TYPE_ROOT_PORT ) && (pci_pcie_type (dev) !=PCI_EXP_TYPE_UPSTREAM ) && - (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ))) + (pci_pcie_type (dev) !=PCI_EXP_TYPE_DOWNSTREAM ) && + (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ))) return -ENODEV;drivers/pci/pcie/portdrv_pci.c +2
-0status =pcie_port_device_register (dev);@@ -195,6 +196,8 @@ static const struct pci_device_id port_pci_ids[] = { {PCI_DEVICE_CLASS (((PCI_CLASS_BRIDGE_PCI <<8 ) |0x00 ), ~0 ) },/* subtractive decode PCI-to-PCI bridge, class type is 060401h */ {PCI_DEVICE_CLASS (((PCI_CLASS_BRIDGE_PCI <<8 ) |0x01 ), ~0 ) }, +/* handle any Root Complex Event Collector */ + {PCI_DEVICE_CLASS (((PCI_CLASS_SYSTEM_RCEC <<8 ) |0x00 ), ~0 ) }, { }, };
RFC PATCH 3/9 PCI/portdrv: Add pcie_walk_rcec() to walk RCiEPs associated with RCEC
In reply to Sean V KelleyFrom: Qiuxu Zhuo <[email protected]>
When an RCEC device signals error(s) to a CPU core, the CPU core needs to walk all the RCiEPs associated with that RCEC to check errors. So add the function pcie_walk_rcec() to walk all RCiEPs associated with the RCEC device.
--- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+)
drivers/pci/pcie/portdrv.h +2 -0
@@ -116,6 +116,8 @@ void pcie_port_service_unregister(struct pcie_port_service_driver *new);extern struct bus_type pcie_port_bus_type;int pcie_port_device_register (struct pci_dev *dev); +void pcie_walk_rcec (struct pci_dev *rcec,int (*cb)(struct pci_dev *,void *), +void *userdata);#ifdef CONFIG_PM int pcie_port_device_suspend (struct device *dev);int pcie_port_device_resume_noirq (struct device *dev);
drivers/pci/pcie/portdrv_core.c +82 -0
94 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [RFC PATCH 3/9] PCI/portdrv: Add pcie_walk_rcec() to walk RCiEPs associated with RCEC
Quoted 10 lines · On Fri, 24 Jul 2020 10:22:17 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When an RCEC device signals error(s) to a CPU core, the CPU core needs to walk all the RCiEPs associated with that RCEC to check errors. So add the function
pcie_walk_rcec()to walk all RCiEPs associated with the RCEC device.Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>
A few trivial points inline. With those tidied up.
Quoted 48 lines
--- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+)drivers/pci/pcie/portdrv.h +2
-0@@ -116,6 +116,8 @@ void pcie_port_service_unregister(struct pcie_port_service_driver *new);extern struct bus_type pcie_port_bus_type;int pcie_port_device_register (struct pci_dev *dev); +void pcie_walk_rcec (struct pci_dev *rcec,int (*cb)(struct pci_dev *,void *), +void *userdata);#ifdef CONFIG_PM int pcie_port_device_suspend (struct device *dev);int pcie_port_device_resume_noirq (struct device *dev);drivers/pci/pcie/portdrv_core.c +15
-0@@ -14,6 +14,7 @@#include <linux/pm_runtime.h> #include <linux/string.h> #include <linux/slab.h> +#include <linux/bitops.h> #include <linux/aer.h> #include "../pci.h" @@ -365,6 +366,87 @@ int pcie_port_device_register(struct pci_dev *dev)return status; } +static int pcie_walk_rciep_devfn (struct pci_bus *pbus,int (*cb)(struct pci_dev *,void *), +void *userdata,unsigned long bitmap) +{ +unsigned int dev, fn; +struct pci_dev *pdev; +int retval; + +for_each_set_bit (dev, &bitmap,32 ) { +for (fn =0 ; fn <8 ; fn++) { + pdev =pci_get_slot (pbus,PCI_DEVFN (dev, fn)); + +if (!pdev ||pci_pcie_type (pdev) !=PCI_EXP_TYPE_RC_END ) +continue ; +
drivers/pci/pcie/portdrv_core.c int pcie_port_device_register(struct pci_dev *dev) · line 392
Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +10
-0+ retval =cb (pdev, userdata); +if (retval) +return retval; + } + } + +return 0 ; +} + +/** pcie_walk_rcec - Walk RCiEP devices associating with RCEC and call callback.
/**
* pcie...
Quoted 37 lines
drivers/pci/pcie/portdrv_core.c +37
-0+* @rcec RCEC whose RCiEP devices should be walked. +* @cb Callback to be called for each RCiEP device found. +* @userdata Arbitrary pointer to be passed to callback. +* +* Walk the given RCEC. Call the provided callback on each RCiEP device found. +* +* We check the return of @cb each time. If it returns anything +* other than 0, we break out. +* +*/ +void pcie_walk_rcec (struct pci_dev *rcec,int (*cb)(struct pci_dev *,void *), +void *userdata) +{ +u32 pos, bitmap, hdr, busn; +u8 ver, nextbusn, lastbusn; +struct pci_bus *pbus; +unsigned int bnr; + + pos =pci_find_ext_capability (rcec,PCI_EXT_CAP_ID_RCEC ); +if (!pos) +return ; + + pbus =pci_find_bus (pci_domain_nr (rcec->bus), rcec->bus->number); +if (!pbus) +return ; + +pci_read_config_dword (rcec, pos +PCI_RCEC_RCIEP_BITMAP , &bitmap); + +/* Find RCiEP devices on the same bus as the RCEC */ +if (pcie_walk_rciep_devfn (pbus, cb, userdata, (unsigned long )bitmap)) +return ; + +/* Check whether RCEC BUSN register is present */ +pci_read_config_dword (rcec, pos, &hdr); + ver =PCI_EXT_CAP_VER (hdr); +if (ver <PCI_RCEC_BUSN_REG_VER ) +return ;
drivers/pci/pcie/portdrv_core.c int pcie_port_device_register(struct pci_dev *dev) · line 439
Quoted 10 lines
drivers/pci/pcie/portdrv_core.c +10
-0+ +pci_read_config_dword (rcec, pos +PCI_RCEC_BUSN , &busn); + nextbusn =PCI_RCEC_BUSN_NEXT (busn); + lastbusn =PCI_RCEC_BUSN_LAST (busn); + +/* All RCiEP devices are on the same bus as the RCEC */ +if (nextbusn ==0xff && lastbusn ==0x00 ) +return ; + +for (bnr = nextbusn; bnr < (lastbusn +1 ); bnr++) {
Why not bnr <= lastbusn? Seems more intuitive way of making it clear the range is inclusive.
Quoted 13 lines
drivers/pci/pcie/portdrv_core.c +10
-0+ pbus =pci_find_bus (pci_domain_nr (rcec->bus), bnr); +if (!pbus) +continue ; + +/* Find RCiEP devices on the given bus */ +if (pcie_walk_rciep_devfn (pbus, cb, userdata,0xffffffff )) +return ; + } +} +#ifdef CONFIG_PM typedef int (*pcie_pm_callback_t )(struct pcie_device *);
Re: [RFC PATCH 3/9] PCI/portdrv: Add pcie_walk_rcec() to walk RCiEPs associated with RCEC
Hi,
Comments below:
Quoted 77 lines · Jonathan Cameron
Quoted 10 lines · On Fri, 24 Jul 2020 10:22:17 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When an RCEC device signals error(s) to a CPU core, the CPU core needs to walk all the RCiEPs associated with that RCEC to check errors. So add the function
pcie_walk_rcec()to walk all RCiEPs associated with the RCEC device.Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>A few trivial points inline. With those tidied up.
Reviewed-by Jonathan Cameron <[email protected]>Quoted 59 lines
--- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+)drivers/pci/pcie/portdrv.h +2
-0@@ -116,6 +116,8 @@ void pcie_port_service_unregister(struct pcie_port_service_driver *new);extern struct bus_type pcie_port_bus_type;int pcie_port_device_register (struct pci_dev *dev); +void pcie_walk_rcec (struct pci_dev *rcec,int (*cb)(struct pci_dev *,void *), +void *userdata);#ifdef CONFIG_PM int
pcie_port_device_suspend(struct device *dev); intpcie_port_device_resume_noirq(struct device *dev);drivers/pci/pcie/portdrv_core.c +21
-0@@ -14,6 +14,7 @@#include <linux/pm_runtime.h> #include <linux/string.h> #include <linux/slab.h> +#include <linux/bitops.h> #include <linux/aer.h> #include "../pci.h" @@ -365,6 +366,87 @@ int pcie_port_device_register(struct pci_dev *dev)return status; } +static int pcie_walk_rciep_devfn (struct pci_bus *pbus,int (*cb)(struct pci_dev *,void *), +void *userdata,unsigned long bitmap) +{ +unsigned int dev, fn; +struct pci_dev *pdev; +int retval; + +for_each_set_bit (dev, &bitmap,32 ) { +for (fn =0 ; fn <8 ; fn++) { + pdev =pci_get_slot (pbus,PCI_DEVFN (dev, fn)); + +if (!pdev ||pci_pcie_type (pdev) !=PCI_EXP_TYPE_RC_END ) +continue ; + + retval =cb (pdev, userdata); +if (retval) +return retval; + } + } +
Quoted 8 lines
Quoted 5 lines
drivers/pci/pcie/portdrv_core.c +4
-0+return 0 ; +} + +/** pcie_walk_rcec - Walk RCiEP devices associating with RCEC and call callback.
/**
* pcie...
Will correct.
Quoted 46 lines
Quoted 46 lines
+ * @rcec RCEC whose RCiEP devices should be walked. + * @cb Callback to be called for each RCiEP device found. + * @userdata Arbitrary pointer to be passed to callback.+ *
+ * Walk the given RCEC. Call the provided callback on each RCiEP device found.
+ *
+ * We check the return of @cb each time. If it returns anything
+ * other than 0, we break out.
+ *
+ */
+void pcie_walk_rcec(structpci_dev*rcec, int (*cb)(structpci_dev*, void *),drivers/pci/pcie/portdrv_core.c +19
-0+ void *userdata) +{ +u32 pos, bitmap, hdr, busn; +u8 ver, nextbusn, lastbusn; +struct pci_bus *pbus; +unsigned int bnr; + + pos =pci_find_ext_capability (rcec,PCI_EXT_CAP_ID_RCEC ); +if (!pos) +return ; + + pbus =pci_find_bus (pci_domain_nr (rcec->bus), rcec->bus->number); +if (!pbus) +return ; + +pci_read_config_dword (rcec, pos +PCI_RCEC_RCIEP_BITMAP , &bitmap); + +/* Find RCiEP devices on the same bus as the RCEC */ + if (pcie_walk_rciep_devfn(pbus, cb, userdata, (unsignedlong)bitmap))
drivers/pci/pcie/portdrv_core.c +12
-0+return ; + +/* Check whether RCEC BUSN register is present */ +pci_read_config_dword (rcec, pos, &hdr); + ver =PCI_EXT_CAP_VER (hdr); +if (ver <PCI_RCEC_BUSN_REG_VER ) +return ; + +pci_read_config_dword (rcec, pos +PCI_RCEC_BUSN , &busn); + nextbusn =PCI_RCEC_BUSN_NEXT (busn); + lastbusn =PCI_RCEC_BUSN_LAST (busn); +
Quoted 8 lines
drivers/pci/pcie/portdrv_core.c int pcie_port_device_register(struct pci_dev *dev) · line 439
Quoted 5 lines
drivers/pci/pcie/portdrv_core.c +5
-0+/* All RCiEP devices are on the same bus as the RCEC */ +if (nextbusn ==0xff && lastbusn ==0x00 ) +return ; + +for (bnr = nextbusn; bnr < (lastbusn +1 ); bnr++) {Why not bnr <= lastbusn? Seems more intuitive way of making it clear the range is inclusive.
Good suggestion. Will change.
Thanks,
Sean
Quoted 15 lines
Quoted 15 lines
drivers/pci/pcie/portdrv_core.c +10
-0+ pbus =pci_find_bus (pci_domain_nr (rcec->bus), bnr); +if (!pbus) +continue ; + +/* Find RCiEP devices on the given bus */ +if (pcie_walk_rciep_devfn (pbus, cb, userdata,0xffffffff )) +return ; + } +} +#ifdef CONFIG_PM typedef int (*pcie_pm_callback_t )(struct pcie_device *);
RFC PATCH 4/9 PCI/AER: Extend AER error handling to RCECs
In reply to Sean V KelleyFrom: Jonathan Cameron <[email protected]>
Currently the kernel does not handle AER errors for Root Complex integrated End Points (RCiEPs)[0]. These devices sit on a root bus within the Root Complex (RC). AER handling is performed by a Root Complex Event Collector (RCEC) [1] which is a effectively a type of RCiEP on the same root bus.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
In addition to the defined OS level handling of the reset flow for the associated RCiEPs of an RCEC, it is possible to also have a firmware first model. In that case there is no need to take any actions on the RCEC because the firmware is responsible for them. This is true where APEI [2] is used to report the AER errors via a GHES[v2] HEST entry [3] and relevant AER CPER record [4] and Firmware First handling is in use.
We effectively end up with two different types of discovery for purposes of handling AER errors:
1) Normal bus walk - we pass the downstream port above a bus to which the device is attached and it walks everything below that point.
2) An RCiEP with no visible association with an RCEC as there is no need to walk devices. In that case, the flow is to just call the callbacks for the actual device.
A new walk function, similar to pci_bus_walk is provided that takes a pci_dev instead of a bus. If that dev corresponds to a downstream port it will walk the subordinate bus of that downstream port. If the dev does not then it will call the function on that device alone.
[0] ACPI PCI Express Base Specification 5.0-1 1.3.2.3 Root Complex Integrated Endpoint Rules.
[1] ACPI PCI Express Base Specification 5.0-1 6.2 Error Signalling and Logging
[2] ACPI Specification 6.3 Chapter 18 ACPI Platform Error Interface (APEI)
[3] ACPI Specification 6.3 18.2.3.7 Generic Hardware Error Source
[4] UEFI Specification 2.8, N.2.7 PCI Express Error Section
---
Changes since v2 [1]:
- Renamed to pci_walk_dev_affected() to reflect the aer affected devices
- Localized to err.c and made static
- Added check for RCEC to reflect
- Tightened up commit log from earlier inquiry focused RFC
[1] https://lore.kernel.org/linux-pci/[email protected]/
--- drivers/pci/pcie/err.c | 55 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 10 deletions(-)
drivers/pci/pcie/err.c +45 -10
@@ -146,38 +146,69 @@ static int report_resume(struct pci_dev *dev, void *data)return 0 ; } +/** pci_walk_dev_affected - walk devices potentially AER affected +* @dev device which may be an RCEC with associated RCiEPs, +* an RCiEP associated with an RCEC, or a Port. +* @cb callback to be called for each device found +* @userdata arbitrary pointer to be passed to callback. +* +* If the device provided is a port, walk the subordinate bus, +* including any bridged devices on buses under this bus. +* Call the provided callback on each device found. +* +* If the device provided has no subordinate bus, call the provided +* callback on the device itself. +* +*/ +static void pci_walk_dev_affected (struct pci_dev *dev,int (*cb)(struct pci_dev *,void *), +void *userdata) +{ +if (dev->subordinate) { +pci_walk_bus (dev->subordinate, cb, userdata); + }else { +cb (dev, userdata); + } +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev)) {pci_ers_result_t status =PCI_ERS_RESULT_CAN_RECOVER ; -struct pci_bus *bus;/* * Error recovery runs on all subordinates of the first downstream port. * If the downstream port detected the error, it is cleared at the end. +* For RCiEPs we should reset just the RCiEP itself. */ if (!(pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || -pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM )) +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) dev = dev->bus->self; - bus = dev->subordinate;pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { -pci_walk_bus (bus, report_frozen_detected, &status); +pci_walk_dev_affected (dev, report_frozen_detected, &status); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { +pci_warn (dev,"link reset not possible for RCiEP\n" ); + status =PCI_ERS_RESULT_NONE ; + goto failed; + } + status =reset_link (dev);if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); goto failed; } }else { -pci_walk_bus (bus, report_normal_detected, &status); +pci_walk_dev_affected (dev, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) { status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast mmio_enabled message\n" ); -pci_walk_bus (bus, report_mmio_enabled, &status); +pci_walk_dev_affected (dev, report_mmio_enabled, &status); }if (status ==PCI_ERS_RESULT_NEED_RESET ) {
@@ -188,17 +219,21 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,*/ status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast slot_reset message\n" ); -pci_walk_bus (bus, report_slot_reset, &status); +pci_walk_dev_affected (dev, report_slot_reset, &status); }if (status !=PCI_ERS_RESULT_RECOVERED ) goto failed;pci_dbg (dev,"broadcast resume message\n" ); -pci_walk_bus (bus, report_resume, &status); +pci_walk_dev_affected (dev, report_resume, &status); -pci_aer_clear_device_status (dev); -pci_aer_clear_nonfatal_status (dev); +if ((pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) { +pci_aer_clear_device_status (dev); +pci_aer_clear_nonfatal_status (dev); + }pci_info (dev,"device recovery successful\n" );return status;
Signature · 1 line
2.27.0
Re: [RFC PATCH 4/9] PCI/AER: Extend AER error handling to RCECs
Quoted 43 lines · On Fri, 24 Jul 2020 10:22:18 -0700 Sean V Kelley <[email protected]>…
From: Jonathan Cameron <[email protected]>
Currently the kernel does not handle AER errors for Root Complex integrated End Points (RCiEPs)[0]. These devices sit on a root bus within the Root Complex (RC). AER handling is performed by a Root Complex Event Collector (RCEC) [1] which is a effectively a type of RCiEP on the same root bus.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
In addition to the defined OS level handling of the reset flow for the associated RCiEPs of an RCEC, it is possible to also have a firmware first model. In that case there is no need to take any actions on the RCEC because the firmware is responsible for them. This is true where APEI [2] is used to report the AER errors via a GHES[v2] HEST entry [3] and relevant AER CPER record [4] and Firmware First handling is in use.
We effectively end up with two different types of discovery for purposes of handling AER errors:
1) Normal bus walk - we pass the downstream port above a bus to which the device is attached and it walks everything below that point.
2) An RCiEP with no visible association with an RCEC as there is no need to walk devices. In that case, the flow is to just call the callbacks for the actual device.
A new walk function, similar to
pci_bus_walkis provided that takes apci_devinstead of a bus. If that dev corresponds to a downstream port it will walk the subordinate bus of that downstream port. If the dev does not then it will call the function on that device alone.[0] ACPI PCI Express Base Specification 5.0-1 1.3.2.3 Root Complex Integrated Endpoint Rules.
[1] ACPI PCI Express Base Specification 5.0-1 6.2 Error Signalling and Logging
[2] ACPI Specification 6.3 Chapter 18 ACPI Platform Error Interface (APEI)
[3] ACPI Specification 6.3 18.2.3.7 Generic Hardware Error Source
[4] UEFI Specification 2.8, N.2.7 PCI Express Error Section
Quoted 6 lines
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Sean V Kelley <[email protected]>---
Changes since v2 [1]:- Renamed to
pci_walk_dev_affected()to reflect the aer affected devices
Make sense.
Quoted 1 line
- Localized to
err.cand made static
Makes sense.
Quoted 1 line
- Added check for RCEC to reflect
That comment probably needs a bit more...
Quoted 1 line
- Tightened up commit log from earlier inquiry focused RFC
Cool.
Looks good to me and I like the new naming.
A few really trivial tidy ups suggested for things that were less than neat in my patch.
Jonathan
Quoted 11 lines
[1] https://lore.kernel.org/linux-pci/[email protected]/
--- drivers/pci/pcie/err.c | 55 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 10 deletions(-)drivers/pci/pcie/err.c +0
-0
drivers/pci/pcie/err.c static int report_resume(struct pci_dev *dev, void *data) · line 149
Quoted 4 lines
drivers/pci/pcie/err.c +1
-0@@ -146,38 +146,69 @@ static int report_resume(struct pci_dev *dev, void *data)return 0 ; } +/** pci_walk_dev_affected - walk devices potentially AER affected
/**
* pci_walk_dev_affected
There is a bit of a mixture in pci files between the two styles, but I'm fairly sure kernel-doc is supposed to be as I'm suggesting (I had this wrong due to cut and paste in earlier version!)
drivers/pci/pcie/err.c static int report_resume(struct pci_dev *dev, void *data) · line 161
Quoted 12 lines
drivers/pci/pcie/err.c +12
-0+* @dev device which may be an RCEC with associated RCiEPs, +* an RCiEP associated with an RCEC, or a Port. +* @cb callback to be called for each device found +* @userdata arbitrary pointer to be passed to callback. +* +* If the device provided is a port, walk the subordinate bus, +* including any bridged devices on buses under this bus. +* Call the provided callback on each device found. +* +* If the device provided has no subordinate bus, call the provided +* callback on the device itself. +*
I also had an ugly pointless newline here. oops :)
Quoted 85 lines
drivers/pci/pcie/err.c +32
-10+*/ +static void pci_walk_dev_affected (struct pci_dev *dev,int (*cb)(struct pci_dev *,void *), +void *userdata) +{ +if (dev->subordinate) { +pci_walk_bus (dev->subordinate, cb, userdata); + }else { +cb (dev, userdata); + } +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev)) {pci_ers_result_t status =PCI_ERS_RESULT_CAN_RECOVER ; -struct pci_bus *bus;/* * Error recovery runs on all subordinates of the first downstream port. * If the downstream port detected the error, it is cleared at the end. +* For RCiEPs we should reset just the RCiEP itself. */ if (!(pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || -pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM )) +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) dev = dev->bus->self; - bus = dev->subordinate;pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { -pci_walk_bus (bus, report_frozen_detected, &status); +pci_walk_dev_affected (dev, report_frozen_detected, &status); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { +pci_warn (dev,"link reset not possible for RCiEP\n" ); + status =PCI_ERS_RESULT_NONE ; + goto failed; + } + status =reset_link (dev);if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); goto failed; } }else { -pci_walk_bus (bus, report_normal_detected, &status); +pci_walk_dev_affected (dev, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) { status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast mmio_enabled message\n" ); -pci_walk_bus (bus, report_mmio_enabled, &status); +pci_walk_dev_affected (dev, report_mmio_enabled, &status); }if (status ==PCI_ERS_RESULT_NEED_RESET ) {@@ -188,17 +219,21 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,*/ status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast slot_reset message\n" ); -pci_walk_bus (bus, report_slot_reset, &status); +pci_walk_dev_affected (dev, report_slot_reset, &status); }if (status !=PCI_ERS_RESULT_RECOVERED ) goto failed;pci_dbg (dev,"broadcast resume message\n" ); -pci_walk_bus (bus, report_resume, &status); +pci_walk_dev_affected (dev, report_resume, &status); -pci_aer_clear_device_status (dev); -pci_aer_clear_nonfatal_status (dev); +if ((pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) { +pci_aer_clear_device_status (dev); +pci_aer_clear_nonfatal_status (dev); + }pci_info (dev,"device recovery successful\n" );return status;
Re: [RFC PATCH 4/9] PCI/AER: Extend AER error handling to RCECs
Quoted 53 lines · Jonathan Cameron
Quoted 43 lines · On Fri, 24 Jul 2020 10:22:18 -0700 Sean V Kelley <[email protected]>…
From: Jonathan Cameron <[email protected]>
Currently the kernel does not handle AER errors for Root Complex integrated End Points (RCiEPs)[0]. These devices sit on a root bus within the Root Complex (RC). AER handling is performed by a Root Complex Event Collector (RCEC) [1] which is a effectively a type of RCiEP on the same root bus.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
In addition to the defined OS level handling of the reset flow for the associated RCiEPs of an RCEC, it is possible to also have a firmware first model. In that case there is no need to take any actions on the RCEC because the firmware is responsible for them. This is true where APEI [2] is used to report the AER errors via a GHES[v2] HEST entry [3] and relevant AER CPER record [4] and Firmware First handling is in use.
We effectively end up with two different types of discovery for purposes of handling AER errors:
1) Normal bus walk - we pass the downstream port above a bus to which the device is attached and it walks everything below that point.
2) An RCiEP with no visible association with an RCEC as there is no need to walk devices. In that case, the flow is to just call the callbacks for the actual device.
A new walk function, similar to
pci_bus_walkis provided that takes apci_devinstead of a bus. If that dev corresponds to a downstream port it will walk the subordinate bus of that downstream port. If the dev does not then it will call the function on that device alone.[0] ACPI PCI Express Base Specification 5.0-1 1.3.2.3 Root Complex Integrated Endpoint Rules.
[1] ACPI PCI Express Base Specification 5.0-1 6.2 Error Signalling and Logging
[2] ACPI Specification 6.3 Chapter 18 ACPI Platform Error Interface (APEI)
[3] ACPI Specification 6.3 18.2.3.7 Generic Hardware Error Source
[4] UEFI Specification 2.8, N.2.7 PCI Express Error SectionQuoted 7 lines
Signed-off-by Jonathan Cameron <[email protected]>Signed-off-by Sean V Kelley <[email protected]>---
Changes since v2 [1]:- Renamed to
pci_walk_dev_affected()to reflect the aer affected devices
Quoted 8 lines
Make sense.
Quoted 1 line
- Localized to
err.cand made staticMakes sense.
Quoted 1 line
- Added check for RCEC to reflect
That comment probably needs a bit more...
Will add to the details.
Quoted 27 lines
Quoted 2 lines
- Tightened up commit log from earlier inquiry focused RFC
Cool.
Looks good to me and I like the new naming.
A few really trivial tidy ups suggested for things that were less than neat in my patch.
Jonathan
Quoted 16 lines
[1] https://lore.kernel.org/linux-pci/[email protected]/
--- drivers/pci/pcie/err.c | 55 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 10 deletions(-)drivers/pci/pcie/err.c +0
-0@@ -146,38 +146,69 @@ static int report_resume(struct pci_dev *dev, void *data)return 0 ; }
Quoted 7 lines
drivers/pci/pcie/err.c static int report_resume(struct pci_dev *dev, void *data) · line 149
Quoted 1 line
drivers/pci/pcie/err.c +1
-0+/** pci_walk_dev_affected - walk devices potentially AER affected /**
*pci_walk_dev_affectedThere is a bit of a mixture in pci files between the two styles, but I'm fairly sure kernel-doc is supposed to be as I'm suggesting (I had this wrong due to cut and paste in earlier version!)
Will fix.
Quoted 14 lines
drivers/pci/pcie/err.c static int report_resume(struct pci_dev *dev, void *data) · line 161
Quoted 13 lines
drivers/pci/pcie/err.c +12
-0+* @dev device which may be an RCEC with associated RCiEPs, +* an RCiEP associated with an RCEC, or a Port. +* @cb callback to be called for each device found +* @userdata arbitrary pointer to be passed to callback. +* +* If the device provided is a port, walk the subordinate bus, +* including any bridged devices on buses under this bus. +* Call the provided callback on each device found. +* +* If the device provided has no subordinate bus, call the provided +* callback on the device itself. +*
Quoted 1 line
I also had an ugly pointless newline here. oops :)
Will fix.
Thanks,
Sean
Quoted 90 lines
Quoted 90 lines
drivers/pci/pcie/err.c +11
-1+*/ +static void pci_walk_dev_affected (struct pci_dev *dev,int (*cb)(struct pci_dev *,void *), +void *userdata) +{ +if (dev->subordinate) { +pci_walk_bus (dev->subordinate, cb, userdata); + }else { +cb (dev, userdata); + } +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev)) {pci_ers_result_t status =PCI_ERS_RESULT_CAN_RECOVER ; -struct pci_bus *bus;/* * Error recovery runs on all subordinates of the first downstream port.
* If the downstream port detected the error, it is cleared at the
end.drivers/pci/pcie/err.c +21
-9+* For RCiEPs we should reset just the RCiEP itself. */ if (!(pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || -pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM )) +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) dev = dev->bus->self; - bus = dev->subordinate;pci_dbg (dev,"broadcast error_detected message\n" );if (state == pci_channel_io_frozen) { -pci_walk_bus (bus, report_frozen_detected, &status); +pci_walk_dev_affected (dev, report_frozen_detected, &status); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { +pci_warn (dev,"link reset not possible for RCiEP\n" ); + status =PCI_ERS_RESULT_NONE ; + goto failed; + } + status =reset_link (dev);if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"link reset failed\n" ); goto failed; } }else { -pci_walk_bus (bus, report_normal_detected, &status); +pci_walk_dev_affected (dev, report_normal_detected, &status); }if (status ==PCI_ERS_RESULT_CAN_RECOVER ) { status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast mmio_enabled message\n" ); -pci_walk_bus (bus, report_mmio_enabled, &status); +pci_walk_dev_affected (dev, report_mmio_enabled, &status); }if (status ==PCI_ERS_RESULT_NEED_RESET ) {@@ -188,17 +219,21 @@ pci_ers_result_t pcie_do_recovery(structpci_dev *dev, */ status =PCI_ERS_RESULT_RECOVERED ;pci_dbg (dev,"broadcast slot_reset message\n" ); -pci_walk_bus (bus, report_slot_reset, &status); +pci_walk_dev_affected (dev, report_slot_reset, &status); }if (status !=PCI_ERS_RESULT_RECOVERED ) goto failed;pci_dbg (dev,"broadcast resume message\n" ); -pci_walk_bus (bus, report_resume, &status); +pci_walk_dev_affected (dev, report_resume, &status); -pci_aer_clear_device_status (dev); -pci_aer_clear_nonfatal_status (dev); +if ((pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) { +pci_aer_clear_device_status (dev); +pci_aer_clear_nonfatal_status (dev); + }pci_info (dev,"device recovery successful\n" );return status;
Re: [RFC PATCH 4/9] PCI/AER: Extend AER error handling to RCECs
In reply to Sean V KelleyQuoted 46 lines · On Fri, 24 Jul 2020 10:22:18 -0700 Sean V Kelley <[email protected]>…
From: Jonathan Cameron <[email protected]>
Currently the kernel does not handle AER errors for Root Complex integrated End Points (RCiEPs)[0]. These devices sit on a root bus within the Root Complex (RC). AER handling is performed by a Root Complex Event Collector (RCEC) [1] which is a effectively a type of RCiEP on the same root bus.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
In addition to the defined OS level handling of the reset flow for the associated RCiEPs of an RCEC, it is possible to also have a firmware first model. In that case there is no need to take any actions on the RCEC because the firmware is responsible for them. This is true where APEI [2] is used to report the AER errors via a GHES[v2] HEST entry [3] and relevant AER CPER record [4] and Firmware First handling is in use.
We effectively end up with two different types of discovery for purposes of handling AER errors:
1) Normal bus walk - we pass the downstream port above a bus to which the device is attached and it walks everything below that point.
2) An RCiEP with no visible association with an RCEC as there is no need to walk devices. In that case, the flow is to just call the callbacks for the actual device.
A new walk function, similar to
pci_bus_walkis provided that takes apci_devinstead of a bus. If that dev corresponds to a downstream port it will walk the subordinate bus of that downstream port. If the dev does not then it will call the function on that device alone.[0] ACPI PCI Express Base Specification 5.0-1 1.3.2.3 Root Complex Integrated Endpoint Rules.
[1] ACPI PCI Express Base Specification 5.0-1 6.2 Error Signalling and Logging
[2] ACPI Specification 6.3 Chapter 18 ACPI Platform Error Interface (APEI)
[3] ACPI Specification 6.3 18.2.3.7 Generic Hardware Error Source
[4] UEFI Specification 2.8, N.2.7 PCI Express Error SectionSigned-off-by Jonathan Cameron <[email protected]>Signed-off-by Sean V Kelley <[email protected]>---
...
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 201
Quoted 6 lines
drivers/pci/pcie/err.c +1
-3pci_dbg (dev,"broadcast resume message\n" ); -pci_walk_bus (bus, report_resume, &status); +pci_walk_dev_affected (dev, report_resume, &status); -pci_aer_clear_device_status (dev); -pci_aer_clear_nonfatal_status (dev);
This code had changed a little in Bjorn's pci/next branch so do a rebase on that before v2.
Quoted 9 lines
drivers/pci/pcie/err.c +6
-0+if ((pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) { +pci_aer_clear_device_status (dev); +pci_aer_clear_nonfatal_status (dev); + }pci_info (dev,"device recovery successful\n" );return status;
Re: [RFC PATCH 4/9] PCI/AER: Extend AER error handling to RCECs
Quoted 56 lines · Jonathan Cameron
Quoted 46 lines · On Fri, 24 Jul 2020 10:22:18 -0700 Sean V Kelley <[email protected]>…
From: Jonathan Cameron <[email protected]>
Currently the kernel does not handle AER errors for Root Complex integrated End Points (RCiEPs)[0]. These devices sit on a root bus within the Root Complex (RC). AER handling is performed by a Root Complex Event Collector (RCEC) [1] which is a effectively a type of RCiEP on the same root bus.
For an RCEC (technically not a Bridge), error messages "received" from associated RCiEPs must be enabled for "transmission" in order to cause a System Error via the Root Control register or (when the Advanced Error Reporting Capability is present) reporting via the Root Error Command register and logging in the Root Error Status register and Error Source Identification register.
In addition to the defined OS level handling of the reset flow for the associated RCiEPs of an RCEC, it is possible to also have a firmware first model. In that case there is no need to take any actions on the RCEC because the firmware is responsible for them. This is true where APEI [2] is used to report the AER errors via a GHES[v2] HEST entry [3] and relevant AER CPER record [4] and Firmware First handling is in use.
We effectively end up with two different types of discovery for purposes of handling AER errors:
1) Normal bus walk - we pass the downstream port above a bus to which the device is attached and it walks everything below that point.
2) An RCiEP with no visible association with an RCEC as there is no need to walk devices. In that case, the flow is to just call the callbacks for the actual device.
A new walk function, similar to
pci_bus_walkis provided that takes apci_devinstead of a bus. If that dev corresponds to a downstream port it will walk the subordinate bus of that downstream port. If the dev does not then it will call the function on that device alone.[0] ACPI PCI Express Base Specification 5.0-1 1.3.2.3 Root Complex Integrated Endpoint Rules.
[1] ACPI PCI Express Base Specification 5.0-1 6.2 Error Signalling and Logging
[2] ACPI Specification 6.3 Chapter 18 ACPI Platform Error Interface (APEI)
[3] ACPI Specification 6.3 18.2.3.7 Generic Hardware Error Source
[4] UEFI Specification 2.8, N.2.7 PCI Express Error SectionSigned-off-by Jonathan Cameron <[email protected]>Signed-off-by Sean V Kelley <[email protected]>---
...
Quoted 4 lines
drivers/pci/pcie/err.c +1
-1pci_dbg (dev,"broadcast resume message\n" ); -pci_walk_bus (bus, report_resume, &status); +pci_walk_dev_affected (dev, report_resume, &status);
Quoted 5 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 201
Quoted 2 lines
drivers/pci/pcie/err.c +0
-2-pci_aer_clear_device_status (dev); -pci_aer_clear_nonfatal_status (dev);This code had changed a little in Bjorn's pci/next branch so do a rebase on that before v2.
Will ensure rebase includes pci/next.
Thanks,
Sean
Quoted 10 lines
Quoted 10 lines
drivers/pci/pcie/err.c +6
-0+if ((pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC )) { +pci_aer_clear_device_status (dev); +pci_aer_clear_nonfatal_status (dev); + }pci_info (dev,"device recovery successful\n" );return status;
RFC PATCH 5/9 PCI/AER: Apply function level reset to RCiEP on fatal error
In reply to Sean V KelleyFrom: Qiuxu Zhuo <[email protected]>
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)
drivers/pci/pcie/err.c +22 -9
@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev } } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) +{ +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ; + +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))
@@ -191,15 +202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,if (state == pci_channel_io_frozen) {pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { -pci_warn (dev,"link reset not possible for RCiEP\n" ); - status =PCI_ERS_RESULT_NONE ; - goto failed; - } - - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); - goto failed; + status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); + goto failed; + } + }else { + status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); + goto failed; + } } }else {pci_walk_dev_affected (dev, report_normal_detected, &status);
Signature · 1 line
2.27.0
kernel test robot
build botRe: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Sean,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on v5.8-rc6]
[cannot apply to pci/next next-20200724]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Sean-V-Kelley/Add-RCEC-handling-to-PCI-AER/20200725-012544 base: ba47d845d715a010f7b51f6f89bae32845e6acb7 config: alpha-randconfig-s031-20200725 (attached as .config) compiler: alpha-linux-gcc (GCC) 9.3.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# apt-get install sparse
# sparse version: v0.6.2-93-g4c6cbe55-dirty
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=alpha
If you fix the issue, kindly add following tag as appropriate
sparse warnings: (new ones prefixed by >>)
>> drivers/pci/pcie/err.c:176:24: sparse: sparse: incorrect type in return expression (different base types) @@ expected restricted pci_channel_state_t enum pci_channel_state @@ got restricted pci_ers_result_t @@ >> drivers/pci/pcie/err.c:176:24: sparse: expected restricted pci_channel_state_t enum pci_channel_state >> drivers/pci/pcie/err.c:176:24: sparse: got restricted pci_ers_result_t drivers/pci/pcie/err.c:179:24: sparse: sparse: incorrect type in return expression (different base types) @@ expected restricted pci_channel_state_t enum pci_channel_state @@ got restricted pci_ers_result_t @@ drivers/pci/pcie/err.c:179:24: sparse: expected restricted pci_channel_state_t enum pci_channel_state drivers/pci/pcie/err.c:179:24: sparse: got restricted pci_ers_result_t drivers/pci/pcie/err.c:181:16: sparse: sparse: incorrect type in return expression (different base types) @@ expected restricted pci_channel_state_t enum pci_channel_state @@ got restricted pci_ers_result_t @@ drivers/pci/pcie/err.c:181:16: sparse: expected restricted pci_channel_state_t enum pci_channel_state drivers/pci/pcie/err.c:181:16: sparse: got restricted pci_ers_result_t >> drivers/pci/pcie/err.c:205:32: sparse: sparse: incorrect type in assignment (different base types) @@ expected restricted pci_ers_result_t [addressable] [usertype] status @@ got restricted pci_channel_state_t enum pci_channel_state @@ >> drivers/pci/pcie/err.c:205:32: sparse: expected restricted pci_ers_result_t [addressable] [usertype] status >> drivers/pci/pcie/err.c:205:32: sparse: got restricted pci_channel_state_t enum pci_channel_state
vim +176 drivers/pci/pcie/err.c
172
173 static enum pci_channel_state flr_on_rciep(struct pci_dev *dev)
174 {
175 if (!pcie_has_flr(dev))
> 176 return PCI_ERS_RESULT_NONE;
177
178 if (pcie_flr(dev))
179 return PCI_ERS_RESULT_DISCONNECT;
180
181 return PCI_ERS_RESULT_RECOVERED;
182 }
183
184 pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
185 enum pci_channel_state state,
186 pci_ers_result_t (*reset_link)(struct pci_dev *pdev))
187 {
188 pci_ers_result_t status = PCI_ERS_RESULT_CAN_RECOVER;
189
190 /*
191 * Error recovery runs on all subordinates of the first downstream port.
192 * If the downstream port detected the error, it is cleared at the end.
193 * For RCiEPs we should reset just the RCiEP itself.
194 */
195 if (!(pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
196 pci_pcie_type(dev) == PCI_EXP_TYPE_DOWNSTREAM ||
197 pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END ||
198 pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC))
199 dev = dev->bus->self;
200
201 pci_dbg(dev, "broadcast error_detected message\n");
202 if (state == pci_channel_io_frozen) {
203 pci_walk_dev_affected(dev, report_frozen_detected, &status);
204 if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) {
> 205 status = flr_on_rciep(dev);---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
In reply to Sean V KelleyQuoted 4 lines · On Fri, 24 Jul 2020 10:22:19 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
I'd like to understand more on your reasoning for flr here. Is it simply that it is all we can do, or is there some basis in a spec somewhere?
Quoted 55 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)drivers/pci/pcie/err.c +22
-9@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev } } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) +{ +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ; + +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))@@ -191,15 +202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,if (state == pci_channel_io_frozen) {pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { -pci_warn (dev,"link reset not possible for RCiEP\n" ); - status =PCI_ERS_RESULT_NONE ; - goto failed; - } - - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); - goto failed; + status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); + goto failed; + } + }else { + status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); + goto failed; + } } }else {pci_walk_dev_affected (dev, report_normal_detected, &status);
RE: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
Quoted 13 lines
From: Jonathan Cameron <[email protected]>
Sent: Monday, July 27, 2020 7:17 PM
To: Kelley, Sean V <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]; Zhuo, Qiuxu <[email protected]>
Subject: Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal errorOn Fri, 24 Jul 2020 10:22:19 -0700
Sean V Kelley <[email protected]> wrote:
Quoted 9 lines
Quoted 4 lines
From: Qiuxu Zhuo <[email protected]>
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
I'd like to understand more on your reasoning for flr here. Is it simply that it is all we can do, or is there some basis in a spec somewhere?
Yes. Though there isn't the link reset for the RCiEP here, I think we should still be able to reset the RCiEP via FLR on fatal error, if the RCiEP supports FLR.
-Qiuxu
Quoted 59 lines
drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct
Quoted 11 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)drivers/pci/pcie/err.c index renamed from drivers/pci/pcie/err.c +0
-0
044df004f20b..9b3ec94bdf1d100644drivers/pci/pcie/err.c +0
-0@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct
pci_dev*dev, int (*cb)(structpci_devQuoted 6 lines
drivers/pci/pcie/err.c +3
-0} } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) { +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ;drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev · line 186
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+ +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))@@ -191,15
drivers/pci/pcie/err.c
Quoted 3 lines
drivers/pci/pcie/err.c +1
-0+202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,drivers/pci/pcie/err.c +0
-0if (state == pci_channel_io_frozen) { pci_walk_dev_affected(dev, report_frozen_detected,&status);
Quoted 14 lines
drivers/pci/pcie/err.c +4
-9if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { -pci_warn (dev,"link reset not possible for RCiEP\n" ); - status =PCI_ERS_RESULT_NONE ; - goto failed; - } - - status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); - goto failed; + status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); + goto failed;drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+ } + }else { + status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); + goto failed; + } } }else { pci_walk_dev_affected(dev, report_normal_detected,&status);
Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
Quoted 18 lines · Qiuxu
Quoted 18 lines
From: Jonathan Cameron <[email protected]>
Sent: Monday, July 27, 2020 7:17 PM
To: Kelley, Sean V <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Luck, Tony <[email protected]>; [email protected]; [email protected]; [email protected]; Zhuo, Qiuxu <[email protected]>
Subject: Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal errorQuoted 4 lines · On Fri, 24 Jul 2020 10:22:19 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
Quoted 9 lines
Quoted 4 lines
I'd like to understand more on your reasoning for flr here. Is it simply that it is all we can do, or is there some basis in a spec somewhere?
Yes. Though there isn't the link reset for the RCiEP here, I think we should still be able to reset the RCiEP via FLR on fatal error, if the RCiEP supports FLR.
-Qiuxu
Also see PCIe 5.0-1, Sec. 6.6.2 Function Level Reset (FLR)
Implementation of FLR is optional (not required), but is strongly recommended. For an example use case consider CXL. Function 0 DVSEC instances control for the CXL functionality of the entire CXL device. FLR may succeed in recovering from CXL.io domain errors.
Thanks,
Sean
Quoted 59 lines
Quoted 59 lines
drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct
Quoted 11 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)drivers/pci/pcie/err.c index renamed from drivers/pci/pcie/err.c +0
-0
044df004f20b..9b3ec94bdf1d100644drivers/pci/pcie/err.c +0
-0@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct
pci_dev*dev, int (*cb)(structpci_devQuoted 6 lines
drivers/pci/pcie/err.c +3
-0} } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) { +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ;drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev · line 186
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+ +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))@@ -191,15
Quoted 3 lines
drivers/pci/pcie/err.c +1
-0+202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,if (state ==
pci_channel_io_frozen) { pci_walk_dev_affected(dev,report_frozen_detected,&status);
Quoted 14 lines
if (
pci_pcie_type(dev)==PCI_EXP_TYPE_RC_END) {drivers/pci/pcie/err.c +4
-9-pci_warn (dev,"link reset not possible for RCiEP\n" ); -status =PCI_ERS_RESULT_NONE ; -goto failed; -} - -status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); -goto failed; +status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); +goto failed;drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+} +}else { +status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); +goto failed; +} } }else { pci_walk_dev_affected(dev, report_normal_detected,&status);
Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
Quoted 36 lines · On Tue, 28 Jul 2020 09:14:11 -0700 Sean V Kelley <[email protected]>…
Quoted 27 lines · Qiuxu
Quoted 18 lines
From: Jonathan Cameron <[email protected]>
Sent: Monday, July 27, 2020 7:17 PM
To: Kelley, Sean V <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Luck,
Tony <[email protected]>; [email protected]; [email protected];
[email protected]; Zhuo, Qiuxu <[email protected]>
Subject: Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP
on fatal errorQuoted 2 lines · On Fri, 24 Jul 2020 10:22:19 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
Quoted 8 lines
Quoted 2 lines
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
I'd like to understand more on your reasoning for flr here. Is it simply that it is all we can do, or is there some basis in a spec
somewhere?Quoted 6 lines
Yes. Though there isn't the link reset for the RCiEP here, I think we should still be able to reset the RCiEP via FLR on fatal error, if the RCiEP supports FLR.
-Qiuxu
Quoted 6 lines
Also see PCIe 5.0-1, Sec. 6.6.2 Function Level Reset (FLR)
Implementation of FLR is optional (not required), but is strongly recommended. For an example use case consider CXL. Function 0 DVSEC instances control for the CXL functionality of the entire CXL device. FLR may succeed in recovering from CXL.io domain errors.
That feels a little bit of a weak argument in favour. PCI spec lists examples of use only for FLR and I can't see this matching any of them, but then they are only examples, so we could argue it doesn't exclude this use. It's not allowed to affect the link state, but I guess it 'might' recover from some other type of error?
I'd have read the statement in the CXL spec you are referring to as matching with the first example in the PCIe spec which is about recovering from software errors. For example, unexpected VM tear down.
@Bjorn / All. What's your view on using FLR as a reset to do when you don't have any other hammers to use?
Personally I don't have a particular problem with this, it just doesn't fit with my mental model of what FLR is for (which may well need adjusting :)
Jonathan
Quoted 64 lines
Thanks,
Sean
Quoted 59 lines
Quoted 59 lines
drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct
Quoted 11 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)drivers/pci/pcie/err.c index renamed from drivers/pci/pcie/err.c +0
-0
044df004f20b..9b3ec94bdf1d100644drivers/pci/pcie/err.c +0
-0@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct
pci_dev*dev, int (*cb)(structpci_devQuoted 6 lines
drivers/pci/pcie/err.c +3
-0} } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) { +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ;drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev · line 186
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+ +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))@@ -191,15
Quoted 3 lines
drivers/pci/pcie/err.c +1
-0+202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,if (state ==
pci_channel_io_frozen) { pci_walk_dev_affected(dev,report_frozen_detected,&status);
Quoted 14 lines
if (
pci_pcie_type(dev)==PCI_EXP_TYPE_RC_END) {drivers/pci/pcie/err.c +4
-9-pci_warn (dev,"link reset not possible for RCiEP\n" ); -status =PCI_ERS_RESULT_NONE ; -goto failed; -} - -status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); -goto failed; +status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); +goto failed;drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+} +}else { +status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); +goto failed; +} } }else { pci_walk_dev_affected(dev, report_normal_detected,&status);
Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP on fatal error
Quoted 52 lines · Jonathan Cameron
Quoted 36 lines · On Tue, 28 Jul 2020 09:14:11 -0700 Sean V Kelley <[email protected]>…
Quoted 27 lines · Qiuxu
Quoted 18 lines
From: Jonathan Cameron <[email protected]>
Sent: Monday, July 27, 2020 7:17 PM
To: Kelley, Sean V <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Luck,
Tony <[email protected]>; [email protected]; [email protected];
[email protected]; Zhuo, Qiuxu <[email protected]>
Subject: Re: [RFC PATCH 5/9] PCI/AER: Apply function level reset to RCiEP
on fatal errorQuoted 2 lines · On Fri, 24 Jul 2020 10:22:19 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
Quoted 8 lines
Quoted 2 lines
Attempt to do function level reset for an RCiEP associated with an RCEC device on fatal error.
I'd like to understand more on your reasoning for flr here. Is it simply that it is all we can do, or is there some basis in a spec
somewhere?Quoted 6 lines
Yes. Though there isn't the link reset for the RCiEP here, I think we should still be able to reset the RCiEP via FLR on fatal error, if the RCiEP supports FLR.
-Qiuxu
Quoted 6 lines
Also see PCIe 5.0-1, Sec. 6.6.2 Function Level Reset (FLR)
Implementation of FLR is optional (not required), but is strongly recommended. For an example use case consider CXL. Function 0 DVSEC instances control for the CXL functionality of the entire CXL device. FLR may succeed in recovering from CXL.io domain errors.
That feels a little bit of a weak argument in favour. PCI spec lists examples of use only for FLR and I can't see this matching any of them, but then they are only examples, so we could argue it doesn't exclude this use. It's not allowed to affect the link state, but I guess it 'might' recover from some other type of error?
Quoted 3 lines
I'd have read the statement in the CXL spec you are referring to as matching with the first example in the PCIe spec which is about recovering from software errors. For example, unexpected VM tear down.
From my perspective, it can add value as the point is to address device functions and their associated software states. As the section in the spec goes on to state:
“The FLR mechanism enables software to quiesce and reset Endpoint hardware with Function-level granularity. Three example usage models illustrate the benefits of this feature:…”
Later changes in CXL 2.0 Section 9.8 (as of 0.9 draft) further look to extend FLR with an eFLR or now referred to as CXL Reset.
“All Functions in a CXL 2.0 (Single Logical Device) SLD that participate in CXL.cache or CXL.mem are required to support either FLR or CXL Reset. MLDs (Multiple Logical Devices), on the other hand, are required to support CXL Reset.”
In my mind the question is whether this change is too limited in scope with this patch series (RCiEP) and whether FLR should be considered in a broader, i.e., EP, as a ‘hammer’ so to speak.
Thanks,
Sean
Quoted 75 lines
@Bjorn / All. What's your view on using FLR as a reset to do when you don't
have any other hammers to use?Personally I don't have a particular problem with this, it just doesn't fit with my mental model of what FLR is for (which may well need adjusting :)
Jonathan
Quoted 64 lines
Thanks,
Sean
Quoted 59 lines
Quoted 59 lines
drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct
Quoted 11 lines
Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/err.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-)drivers/pci/pcie/err.c index renamed from drivers/pci/pcie/err.c +0
-0
044df004f20b..9b3ec94bdf1d100644drivers/pci/pcie/err.c +0
-0@@ -170,6 +170,17 @@ static void pci_walk_dev_affected(struct
pci_dev*dev, int (*cb)(structpci_devQuoted 6 lines
drivers/pci/pcie/err.c +3
-0} } +static enum pci_channel_state flr_on_rciep (struct pci_dev *dev) { +if (!pcie_has_flr (dev)) +return PCI_ERS_RESULT_NONE ;drivers/pci/pcie/err.c static void pci_walk_dev_affected(struct pci_dev *dev, int (*cb)(struct pci_dev · line 186
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+ +if (pcie_flr (dev)) +return PCI_ERS_RESULT_DISCONNECT ; + +return PCI_ERS_RESULT_RECOVERED ; +} +pci_ers_result_t pcie_do_recovery (struct pci_dev *dev,enum pci_channel_state state,pci_ers_result_t (*reset_link)(struct pci_dev *pdev))@@ -191,15
Quoted 3 lines
drivers/pci/pcie/err.c +1
-0+202,17 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,if (state ==
pci_channel_io_frozen) { pci_walk_dev_affected(dev,report_frozen_detected,&status);
Quoted 14 lines
if (
pci_pcie_type(dev)==PCI_EXP_TYPE_RC_END) {drivers/pci/pcie/err.c +4
-9-pci_warn (dev,"link reset not possible for RCiEP\n" ); -status =PCI_ERS_RESULT_NONE ; -goto failed; -} - -status =reset_link (dev); -if (status !=PCI_ERS_RESULT_RECOVERED ) { -pci_warn (dev,"link reset failed\n" ); -goto failed; +status =flr_on_rciep (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"function level reset failed\n" ); +goto failed;drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
Quoted 10 lines
drivers/pci/pcie/err.c +7
-0+} +}else { +status =reset_link (dev); +if (status !=PCI_ERS_RESULT_RECOVERED ) { +pci_warn (dev,"link reset failed\n" ); +goto failed; +} } }else { pci_walk_dev_affected(dev, report_normal_detected,&status);
RFC PATCH 6/9 PCI: Add 'rcec' field to pci_dev for associated RCiEPs
In reply to Sean V KelleyFrom: Qiuxu Zhuo <[email protected]>
When attempting error recovery for an RCiEP associated with an RCEC device, there needs to be a way to update the Root Error Status, the Uncorrectable Error Status and the Uncorrectable Error Severity of the parent RCEC. So add the 'rcec' field to the pci_dev structure and provide a hook for the Root Port Driver to associate RCiEPs with their respective parent RCEC.
--- drivers/pci/pcie/aer.c | 9 +++++---- drivers/pci/pcie/err.c | 9 +++++++++ drivers/pci/pcie/portdrv_pci.c | 15 +++++++++++++++ include/linux/pci.h | 3 +++ 4 files changed, 32 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer.c +5 -4
@@ -1335,17 +1335,18 @@ static int aer_probe(struct pcie_device *dev)static pci_ers_result_t aer_root_reset (struct pci_dev *dev) {int aer = dev->aer_cap; +int rc =0 ;u32 reg32; -int rc; -/* Disable Root's interrupt in response to error messages */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , ®32); reg32 &= ~ROOT_PORT_INTR_ON_MESG_MASK ;pci_write_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , reg32); - rc =pci_bus_error_reset (dev); -pci_info (dev,"Root Port link has been reset\n" ); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) { + rc =pci_bus_error_reset (dev); +pci_info (dev,"Root Port link has been reset\n" ); + }/* Clear Root Error Status */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , ®32);
drivers/pci/pcie/err.c +9 -0
@@ -203,6 +203,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { status =flr_on_rciep (dev); +/* +* The callback only clears the Root Error Status +* of the RCEC (see aer.c). +*/ +reset_link (dev->rcec);if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"function level reset failed\n" ); goto failed;
@@ -246,7 +251,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC)) {pci_aer_clear_device_status (dev);pci_aer_clear_nonfatal_status (dev); + } else if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) { +pci_aer_clear_device_status (dev->rcec); +pci_aer_clear_nonfatal_status (dev->rcec); } +pci_info (dev,"device recovery successful\n" );return status;
drivers/pci/pcie/portdrv_pci.c +15 -0
@@ -90,6 +90,18 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {#define PCIE_PORTDRV_PM_OPS NULL#endif /* !PM */ +static int pcie_hook_rcec (struct pci_dev *pdev,void *data) +{ +struct pci_dev *rcec = (struct pci_dev *)data; + + pdev->rcec = rcec; +pci_info (rcec,"RCiEP(under an RCEC) %04x:%02x:%02x.%d\n" , +pci_domain_nr (pdev->bus), pdev->bus->number, +PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn)); + +return 0 ; +} +/* * pcie_portdrv_probe - Probe PCI-Express port devices * @dev: PCI-Express port device being probed
@@ -110,6 +122,9 @@ static int pcie_portdrv_probe(struct pci_dev *dev, (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)))return -ENODEV ; +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, pcie_hook_rcec, dev); + status =pcie_port_device_register (dev);if (status)return status;
include/linux/pci.h +3 -0
@@ -326,6 +326,9 @@ struct pci_dev {#ifdef CONFIG_PCIEAER u16 aer_cap;/* AER capability offset */ struct aer_stats *aer_stats;/* AER stats for this device */ +#endif +#ifdef CONFIG_PCIEPORTBUS +struct pci_dev *rcec;/* Associated RCEC device */ #endif u8 pcie_cap;/* PCIe capability offset */ u8 msi_cap;/* MSI capability offset */
Signature · 1 line
2.27.0
Re: [RFC PATCH 6/9] PCI: Add 'rcec' field to pci_dev for associated RCiEPs
Quoted 11 lines · On Fri, 24 Jul 2020 10:22:20 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When attempting error recovery for an RCiEP associated with an RCEC device, there needs to be a way to update the Root Error Status, the Uncorrectable Error Status and the Uncorrectable Error Severity of the parent RCEC. So add the 'rcec' field to the
pci_devstructure and provide a hook for the Root Port Driver to associate RCiEPs with their respective parent RCEC.Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>
I haven't tested yet, but I think there is one path in here that breaks my case (no OS visible rcec / all done in firmware GHESv2 / APEI)
Jonathan
Quoted 39 lines
--- drivers/pci/pcie/aer.c | 9 +++++---- drivers/pci/pcie/err.c | 9 +++++++++ drivers/pci/pcie/portdrv_pci.c | 15 +++++++++++++++ include/linux/pci.h | 3 +++ 4 files changed, 32 insertions(+), 4 deletions(-)drivers/pci/pcie/aer.c +5
-4@@ -1335,17 +1335,18 @@ static int aer_probe(struct pcie_device *dev)static pci_ers_result_t aer_root_reset (struct pci_dev *dev) {int aer = dev->aer_cap; +int rc =0 ;u32 reg32; -int rc; -/* Disable Root's interrupt in response to error messages */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , ®32); reg32 &= ~ROOT_PORT_INTR_ON_MESG_MASK ;pci_write_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , reg32); - rc =pci_bus_error_reset (dev); -pci_info (dev,"Root Port link has been reset\n" ); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) { + rc =pci_bus_error_reset (dev); +pci_info (dev,"Root Port link has been reset\n" ); + }/* Clear Root Error Status */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , ®32);drivers/pci/pcie/err.c +0
-0
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 210
Quoted 8 lines
drivers/pci/pcie/err.c +5
-0@@ -203,6 +203,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { status =flr_on_rciep (dev); +/* +* The callback only clears the Root Error Status +* of the RCEC (see aer.c). +*/ +reset_link (dev->rcec);
This looks dangerous for my case where APEI / GHESv2 is used. In that case
we don't expose an RCEC at all. I don't think the reset_link callback
is safe to a null pointer here. Fix may be as simple as
if (dev->rcec)
reset_link(dev->rcec);
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 256
Quoted 10 lines
drivers/pci/pcie/err.c +3
-0if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"function level reset failed\n" ); goto failed;@@ -246,7 +251,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC)) {pci_aer_clear_device_status (dev);pci_aer_clear_nonfatal_status (dev); + } else if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) { +pci_aer_clear_device_status (dev->rcec); +pci_aer_clear_nonfatal_status (dev->rcec);
These may be safe as in my both now have protections for !pcie_aer_is_native.
Quoted 11 lines
drivers/pci/pcie/err.c +1
-0} +pci_info (dev,"device recovery successful\n" );return status;drivers/pci/pcie/portdrv_pci.c +0
-0@@ -90,6 +90,18 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {#define PCIE_PORTDRV_PM_OPS NULL
drivers/pci/pcie/portdrv_pci.c static const struct dev_pm_ops pcie_portdrv_pm_ops = { · line 100
Quoted 10 lines
drivers/pci/pcie/portdrv_pci.c +8
-0#endif /* !PM */ +static int pcie_hook_rcec (struct pci_dev *pdev,void *data) +{ +struct pci_dev *rcec = (struct pci_dev *)data; + + pdev->rcec = rcec; +pci_info (rcec,"RCiEP(under an RCEC) %04x:%02x:%02x.%d\n" , +pci_domain_nr (pdev->bus), pdev->bus->number, +PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn));
We may want to make this debug info at somepoint if we have a way
of discovering it from userspace. The PCI boot up is extremely
verbose already!
Quoted 31 lines
drivers/pci/pcie/portdrv_pci.c +7
-0+ +return 0 ; +} +/* * pcie_portdrv_probe - Probe PCI-Express port devices * @dev: PCI-Express port device being probed @@ -110,6 +122,9 @@ static int pcie_portdrv_probe(struct pci_dev *dev, (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)))return -ENODEV ; +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, pcie_hook_rcec, dev); + status =pcie_port_device_register (dev);if (status)return status;include/linux/pci.h +3
-0@@ -326,6 +326,9 @@ struct pci_dev {#ifdef CONFIG_PCIEAER u16 aer_cap;/* AER capability offset */ struct aer_stats *aer_stats;/* AER stats for this device */ +#endif +#ifdef CONFIG_PCIEPORTBUS +struct pci_dev *rcec;/* Associated RCEC device */ #endif u8 pcie_cap;/* PCIe capability offset */ u8 msi_cap;/* MSI capability offset */
Re: [RFC PATCH 6/9] PCI: Add 'rcec' field to pci_dev for associated RCiEPs
Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Fri, 24 Jul 2020 10:22:20 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When attempting error recovery for an RCiEP associated with an RCEC device, there needs to be a way to update the Root Error Status, the Uncorrectable Error Status and the Uncorrectable Error Severity of the parent RCEC. So add the 'rcec' field to the
pci_devstructure and provide a hook for the Root Port Driver to associate RCiEPs with their respective parent RCEC.
Quoted 8 lines
Quoted 3 lines
Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>I haven't tested yet, but I think there is one path in here that breaks my case (no OS visible rcec / all done in firmware GHESv2 / APEI)
Jonathan
Okay, keep me up to date and if there is a way I can reproduce or better test for your use case let me know and I will do so.
Quoted 54 lines
Quoted 40 lines
--- drivers/pci/pcie/aer.c | 9 +++++---- drivers/pci/pcie/err.c | 9 +++++++++ drivers/pci/pcie/portdrv_pci.c | 15 +++++++++++++++ include/linux/pci.h | 3 +++ 4 files changed, 32 insertions(+), 4 deletions(-)drivers/pci/pcie/aer.c +5
-4@@ -1335,17 +1335,18 @@ static int aer_probe(struct pcie_device *dev)static pci_ers_result_t aer_root_reset (struct pci_dev *dev) {int aer = dev->aer_cap; +int rc =0 ;u32 reg32; -int rc; -/* Disable Root's interrupt in response to error messages */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , ®32); reg32 &= ~ROOT_PORT_INTR_ON_MESG_MASK ;pci_write_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , reg32); - rc =pci_bus_error_reset (dev); -pci_info (dev,"Root Port link has been reset\n" ); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) { + rc =pci_bus_error_reset (dev); +pci_info (dev,"Root Port link has been reset\n" ); + }/* Clear Root Error Status */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , ®32);drivers/pci/pcie/err.c +0
-0drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 211
Quoted 9 lines
drivers/pci/pcie/err.c +5
-0@@ -203,6 +203,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { status =flr_on_rciep (dev); +/* +* The callback only clears the Root Error Status +* of the RCEC (see aer.c). +*/ +reset_link (dev->rcec);This looks dangerous for my case where APEI / GHESv2 is used. In that case
we don't expose an RCEC at all. I don't think thereset_linkcallback
is safe to a null pointer here. Fix may be as simple as
if (dev->rcec)
Quoted 1 line
reset_link(dev->rcec);
Good catch. Will look to NULL case as well as walk through code where dev->rcec is further referenced (below and elsewhere) to ensure nothing falls through. Glad you are pointing these out.
Quoted 16 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev · line 257
Quoted 13 lines
drivers/pci/pcie/err.c +0
-0if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"function level reset failed\n" );drivers/pci/pcie/err.c +3
-0goto failed;@@ -246,7 +251,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC)) {pci_aer_clear_device_status (dev);pci_aer_clear_nonfatal_status (dev); + } else if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) { +pci_aer_clear_device_status (dev->rcec); +pci_aer_clear_nonfatal_status (dev->rcec);These may be safe as in my both now have protections for !pcie_aer_is_native.
I’ll have a look again for the NULL case
Quoted 14 lines
Quoted 14 lines
drivers/pci/pcie/err.c +1
-0} +pci_info (dev,"device recovery successful\n" );return status;drivers/pci/pcie/portdrv_pci.c +0
-0@@ -90,6 +90,18 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {#define PCIE_PORTDRV_PM_OPS NULL#endif /* !PM */
Quoted 12 lines
drivers/pci/pcie/portdrv_pci.c static const struct dev_pm_ops pcie_portdrv_pm_ops = { · line 100
Quoted 8 lines
drivers/pci/pcie/portdrv_pci.c +8
-0+static int pcie_hook_rcec (struct pci_dev *pdev,void *data) +{ +struct pci_dev *rcec = (struct pci_dev *)data; + + pdev->rcec = rcec; +pci_info (rcec,"RCiEP(under an RCEC) %04x:%02x:%02x.%d\n" , +pci_domain_nr (pdev->bus), pdev->bus->number, +PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn));We may want to make this debug info at somepoint if we have a way
of discovering it from userspace. The PCI boot up is extremely
verbose already!
I’ve submitted patches to pciutils, which expose this very detail:
https://lore.kernel.org/linux-pci/[email protected]/
Flexible either way, could remove or make it debug info.
Thanks,
Sean
Quoted 33 lines
Quoted 33 lines
drivers/pci/pcie/portdrv_pci.c +4
-0+ +return 0 ; +} +/* * pcie_portdrv_probe - Probe PCI-Express port devices * @dev: PCI-Express port device being probed
drivers/pci/pcie/portdrv_pci.c +3
-0@@ -110,6 +122,9 @@ static int pcie_portdrv_probe(struct pci_dev *dev, (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)))return -ENODEV ; +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, pcie_hook_rcec, dev); + status =pcie_port_device_register (dev);if (status)drivers/pci/pcie/portdrv_pci.c +0
-0return status;include/linux/pci.h +3
-0@@ -326,6 +326,9 @@ struct pci_dev {#ifdef CONFIG_PCIEAER u16 aer_cap;/* AER capability offset */ struct aer_stats *aer_stats;/* AER stats for this device */ +#endif +#ifdef CONFIG_PCIEPORTBUS +struct pci_dev *rcec;/* Associated RCEC device */ #endif u8 pcie_cap;/* PCIe capability offset */ u8 msi_cap;/* MSI capability offset */
Re: [RFC PATCH 6/9] PCI: Add 'rcec' field to pci_dev for associated RCiEPs
In reply to Jonathan CameronQuoted 75 lines · On Mon, 27 Jul 2020 12:23:58 +0100 Jonathan Cameron…
Quoted 11 lines · On Fri, 24 Jul 2020 10:22:20 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When attempting error recovery for an RCiEP associated with an RCEC device, there needs to be a way to update the Root Error Status, the Uncorrectable Error Status and the Uncorrectable Error Severity of the parent RCEC. So add the 'rcec' field to the
pci_devstructure and provide a hook for the Root Port Driver to associate RCiEPs with their respective parent RCEC.Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>I haven't tested yet, but I think there is one path in here that breaks my case (no OS visible rcec / all done in firmware GHESv2 / APEI)
Jonathan
Quoted 39 lines
--- drivers/pci/pcie/aer.c | 9 +++++---- drivers/pci/pcie/err.c | 9 +++++++++ drivers/pci/pcie/portdrv_pci.c | 15 +++++++++++++++ include/linux/pci.h | 3 +++ 4 files changed, 32 insertions(+), 4 deletions(-)drivers/pci/pcie/aer.c +5
-4@@ -1335,17 +1335,18 @@ static int aer_probe(struct pcie_device *dev)static pci_ers_result_t aer_root_reset (struct pci_dev *dev) {int aer = dev->aer_cap; +int rc =0 ;u32 reg32; -int rc; -/* Disable Root's interrupt in response to error messages */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , ®32); reg32 &= ~ROOT_PORT_INTR_ON_MESG_MASK ;pci_write_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , reg32); - rc =pci_bus_error_reset (dev); -pci_info (dev,"Root Port link has been reset\n" ); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) { + rc =pci_bus_error_reset (dev); +pci_info (dev,"Root Port link has been reset\n" ); + }/* Clear Root Error Status */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , ®32);drivers/pci/pcie/err.c +0
-0drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 210
Quoted 8 lines
drivers/pci/pcie/err.c +5
-0@@ -203,6 +203,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { status =flr_on_rciep (dev); +/* +* The callback only clears the Root Error Status +* of the RCEC (see aer.c). +*/ +reset_link (dev->rcec);This looks dangerous for my case where APEI / GHESv2 is used. In that case
we don't expose an RCEC at all. I don't think thereset_linkcallback
is safe to a null pointer here. Fix may be as simple as
if (dev->rcec)reset_link(dev->rcec);
Quoted 9 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 255
Quoted 9 lines
drivers/pci/pcie/err.c +2
-0if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"function level reset failed\n" ); goto failed;@@ -246,7 +251,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC)) {pci_aer_clear_device_status (dev);pci_aer_clear_nonfatal_status (dev); + } else if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) { +pci_aer_clear_device_status (dev->rcec);
This needs updating as well to match current approach in Bjorn's tree
(reworked version of my fix for this in th !is_native case)
Quoted 3 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 256
Quoted 1 line
drivers/pci/pcie/err.c +1
-0+pci_aer_clear_nonfatal_status (dev->rcec);These may be safe as in my both now have protections for !pcie_aer_is_native.
Except I'm wrong. That function performs a local check based on the pci_dev passed in, so blows up anyway.
So this whole block needs an if(dev->rcec), or potentially protect it with a call to pci_aer_is_native(dev) on the basis we shouldn't be able to get here unless we are not doing native aer or we have an rcec.
Jonathan
Quoted 59 lines
Quoted 12 lines
drivers/pci/pcie/err.c +1
-0} +pci_info (dev,"device recovery successful\n" );return status;drivers/pci/pcie/portdrv_pci.c +0
-0@@ -90,6 +90,18 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {#define PCIE_PORTDRV_PM_OPS NULLdrivers/pci/pcie/portdrv_pci.c static const struct dev_pm_ops pcie_portdrv_pm_ops = { · line 100
Quoted 10 lines
drivers/pci/pcie/portdrv_pci.c +8
-0#endif /* !PM */ +static int pcie_hook_rcec (struct pci_dev *pdev,void *data) +{ +struct pci_dev *rcec = (struct pci_dev *)data; + + pdev->rcec = rcec; +pci_info (rcec,"RCiEP(under an RCEC) %04x:%02x:%02x.%d\n" , +pci_domain_nr (pdev->bus), pdev->bus->number, +PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn));We may want to make this debug info at somepoint if we have a way
of discovering it from userspace. The PCI boot up is extremely
verbose already!Quoted 31 lines
drivers/pci/pcie/portdrv_pci.c +7
-0+ +return 0 ; +} +/* * pcie_portdrv_probe - Probe PCI-Express port devices * @dev: PCI-Express port device being probed @@ -110,6 +122,9 @@ static int pcie_portdrv_probe(struct pci_dev *dev, (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)))return -ENODEV ; +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, pcie_hook_rcec, dev); + status =pcie_port_device_register (dev);if (status)return status;include/linux/pci.h +3
-0@@ -326,6 +326,9 @@ struct pci_dev {#ifdef CONFIG_PCIEAER u16 aer_cap;/* AER capability offset */ struct aer_stats *aer_stats;/* AER stats for this device */ +#endif +#ifdef CONFIG_PCIEPORTBUS +struct pci_dev *rcec;/* Associated RCEC device */ #endif u8 pcie_cap;/* PCIe capability offset */ u8 msi_cap;/* MSI capability offset */
Re: [RFC PATCH 6/9] PCI: Add 'rcec' field to pci_dev for associated RCiEPs
Quoted 92 lines · Jonathan Cameron
Quoted 77 lines · On Mon, 27 Jul 2020 12:23:58 +0100 Jonathan Cameron…
Quoted 11 lines · On Fri, 24 Jul 2020 10:22:20 -0700 Sean V Kelley <[email protected]>…
From: Qiuxu Zhuo <[email protected]>
When attempting error recovery for an RCiEP associated with an RCEC device, there needs to be a way to update the Root Error Status, the Uncorrectable Error Status and the Uncorrectable Error Severity of the parent RCEC. So add the 'rcec' field to the
pci_devstructure and provide a hook for the Root Port Driver to associate RCiEPs with their respective parent RCEC.Co-developed-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>I haven't tested yet, but I think there is one path in here that breaks my case (no OS visible rcec / all done in firmware GHESv2 / APEI)
Jonathan
Quoted 40 lines
--- drivers/pci/pcie/aer.c | 9 +++++---- drivers/pci/pcie/err.c | 9 +++++++++ drivers/pci/pcie/portdrv_pci.c | 15 +++++++++++++++ include/linux/pci.h | 3 +++ 4 files changed, 32 insertions(+), 4 deletions(-)drivers/pci/pcie/aer.c +5
-4@@ -1335,17 +1335,18 @@ static int aer_probe(struct pcie_device *dev)static pci_ers_result_t aer_root_reset (struct pci_dev *dev) {int aer = dev->aer_cap; +int rc =0 ;u32 reg32; -int rc; -/* Disable Root's interrupt in response to error messages */ pci_read_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , ®32); reg32 &= ~ROOT_PORT_INTR_ON_MESG_MASK ;pci_write_config_dword (dev, aer +PCI_ERR_ROOT_COMMAND , reg32); - rc =pci_bus_error_reset (dev); -pci_info (dev,"Root Port link has been reset\n" ); +if (pci_pcie_type (dev) !=PCI_EXP_TYPE_RC_EC ) { + rc =pci_bus_error_reset (dev); +pci_info (dev,"Root Port link has been reset\n" ); + }/* Clear Root Error Status */ drivers/pci/pcie/aer.c +0
-0pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , ®32);drivers/pci/pcie/err.c +0
-0drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct · line 211
Quoted 9 lines
drivers/pci/pcie/err.c +5
-0@@ -203,6 +203,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,pci_walk_dev_affected (dev, report_frozen_detected, &status);if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_END ) { status =flr_on_rciep (dev); +/* +* The callback only clears the Root Error Status +* of the RCEC (see aer.c). +*/ +reset_link (dev->rcec);This looks dangerous for my case where APEI / GHESv2 is used. In that case
we don't expose an RCEC at all. I don't think thereset_linkcallback
is safe to a null pointer here. Fix may be as simple as
if (dev->rcec)reset_link(dev->rcec);Quoted 10 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct · line 256
Quoted 10 lines
drivers/pci/pcie/err.c +0
-0if (status !=PCI_ERS_RESULT_RECOVERED ) {pci_warn (dev,"function level reset failed\n" );drivers/pci/pcie/err.c +2
-0goto failed;@@ -246,7 +251,11 @@ pci_ers_result_t pcie_do_recovery(structpci_dev *dev, pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC)) {pci_aer_clear_device_status (dev );pci_aer_clear_nonfatal_status (dev); + } else if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) { +pci_aer_clear_device_status (dev->rcec);This needs updating as well to match current approach in Bjorn's tree
Quoted 1 line
(reworked version of my fix for this in th !is_native case)
Will rework.
Quoted 4 lines
Quoted 4 lines
drivers/pci/pcie/err.c pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, · line 256
Quoted 2 lines
drivers/pci/pcie/err.c +1
-0+pci_aer_clear_nonfatal_status (dev->rcec);These may be safe as in my both now have protections for !pcie_aer_is_native.
Quoted 8 lines
Except I'm wrong. That function performs a local check based on the
pci_devpassed in, so blows up anyway.So this whole block needs an
if(dev->rcec), or potentially protect it with a call topci_aer_is_native(dev)on the basis we shouldn't be able to get here unless we are not doing native aer or we have an rcec.Jonathan
Thanks Jonathan. Will rework that flow and checks on dev->rcec based on Bjorn’s current tree.
Sean
Quoted 62 lines
Quoted 62 lines
Quoted 14 lines
drivers/pci/pcie/err.c +1
-0} +pci_info (dev,"device recovery successful\n" );return status;drivers/pci/pcie/portdrv_pci.c +0
-0@@ -90,6 +90,18 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {#define PCIE_PORTDRV_PM_OPS NULLdrivers/pci/pcie/portdrv_pci.c static const struct dev_pm_ops · line 101
Quoted 10 lines
drivers/pci/pcie/portdrv_pci.c +8
-0#endif /* !PM */ +static int pcie_hook_rcec (struct pci_dev *pdev,void *data) +{ +struct pci_dev *rcec = (struct pci_dev *)data; + + pdev->rcec = rcec; +pci_info (rcec,"RCiEP(under an RCEC) %04x:%02x:%02x.%d\n" , +pci_domain_nr (pdev->bus), pdev->bus->number, +PCI_SLOT (pdev->devfn),PCI_FUNC (pdev->devfn));We may want to make this debug info at somepoint if we have a way
of discovering it from userspace. The PCI boot up is extremely
verbose already!Quoted 32 lines
drivers/pci/pcie/portdrv_pci.c +4
-0+ +return 0 ; +} +/* * pcie_portdrv_probe - Probe PCI-Express port devices * @dev: PCI-Express port device being probed
drivers/pci/pcie/portdrv_pci.c +3
-0@@ -110,6 +122,9 @@ static int pcie_portdrv_probe(struct pci_dev *dev, (pci_pcie_type(dev) != PCI_EXP_TYPE_RC_EC)))return -ENODEV ; +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, pcie_hook_rcec, dev); + status =pcie_port_device_register (dev);if (status)drivers/pci/pcie/portdrv_pci.c +0
-0return status;include/linux/pci.h +3
-0@@ -326,6 +326,9 @@ struct pci_dev {#ifdef CONFIG_PCIEAER u16 aer_cap;/* AER capability offset */ struct aer_stats *aer_stats;/* AER stats for this device */ +#endif +#ifdef CONFIG_PCIEPORTBUS +struct pci_dev *rcec;/* Associated RCEC device */ #endif u8 pcie_cap;/* PCIe capability offset */ u8 msi_cap;/* MSI capability offset */
RFC PATCH 7/9 PCI/AER: Add RCEC AER handling
In reply to Sean V KelleyThe Root Complex Event Collectors(RCEC) appear as peers to Root Ports and also have the AER capability. So add RCEC support to the current AER service driver and attach the AER service driver to the RCEC device.
--- drivers/pci/pcie/aer.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
drivers/pci/pcie/aer.c +27 -7
@@ -303,7 +303,7 @@ int pci_aer_raw_clear_status(struct pci_dev *dev)return -EIO ; port_type =pci_pcie_type (dev); -if (port_type ==PCI_EXP_TYPE_ROOT_PORT ) { +if (port_type ==PCI_EXP_TYPE_ROOT_PORT || port_type ==PCI_EXP_TYPE_RC_EC ) {pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , &status);pci_write_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , status); }
@@ -389,6 +389,12 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);pci_aer_clear_status (dev); + +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) { +if (!pci_find_ext_capability (dev,PCI_EXT_CAP_ID_RCEC )) +return ; +pci_info (dev,"AER: RCEC CAP FOUND and cap_has_rtctl = %d\n" , n); + } }void pci_aer_exit (struct pci_dev *dev)
@@ -577,7 +583,8 @@ static umode_t aer_stats_attrs_are_visible(struct kobject *kobj,if ((a == &dev_attr_aer_rootport_total_err_cor.attr || a == &dev_attr_aer_rootport_total_err_fatal.attr || a == &dev_attr_aer_rootport_total_err_nonfatal.attr) && -pci_pcie_type (pdev) !=PCI_EXP_TYPE_ROOT_PORT ) + ((pci_pcie_type (pdev) !=PCI_EXP_TYPE_ROOT_PORT ) && + (pci_pcie_type (pdev) !=PCI_EXP_TYPE_RC_EC )))return 0 ;return a->mode;
@@ -894,7 +901,10 @@ static bool find_source_device(struct pci_dev *parent,if (result)return true; -pci_walk_bus (parent->subordinate, find_device_iter, e_info); +if (pci_pcie_type (parent) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (parent, find_device_iter, e_info); +else +pci_walk_bus (parent->subordinate, find_device_iter, e_info);if (!e_info->error_dev_num) {pci_info (parent,"can't find device of ID%04x\n" , e_info->id);
@@ -1030,6 +1040,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)if (!(info->status & ~info->mask))return 0 ; }else if (pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ||pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || info->severity ==AER_NONFATAL ) {
@@ -1182,6 +1193,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)int type =pci_pcie_type (dev);if ((type ==PCI_EXP_TYPE_ROOT_PORT ) || + (type ==PCI_EXP_TYPE_RC_EC ) || + (type ==PCI_EXP_TYPE_RC_END ) || (type ==PCI_EXP_TYPE_UPSTREAM ) || (type ==PCI_EXP_TYPE_DOWNSTREAM )) {if (enable)
@@ -1206,9 +1219,11 @@ static void set_downstream_devices_error_reporting(struct pci_dev *dev, {set_device_error_reporting (dev, &enable); -if (!dev->subordinate) -return ; -pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, set_device_error_reporting, &enable); +else if (dev->subordinate) +pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); + }/**
@@ -1306,6 +1321,11 @@ static int aer_probe(struct pcie_device *dev)struct device *device = &dev->device;struct pci_dev *port = dev->port; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + rpc =devm_kzalloc (device,sizeof (struct aer_rpc ),GFP_KERNEL );if (!rpc)return -ENOMEM ;
@@ -1362,7 +1382,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)static struct pcie_port_service_driver aerdriver = { .name ="aer" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT , .service =PCIE_PORT_SERVICE_AER , .probe = aer_probe,
Signature · 1 line
2.27.0
Re: [RFC PATCH 7/9] PCI/AER: Add RCEC AER handling
Quoted 7 lines · On Fri, 24 Jul 2020 10:22:21 -0700 Sean V Kelley <[email protected]>…
The Root Complex Event Collectors(RCEC) appear as peers to Root Ports and also have the AER capability. So add RCEC support to the current AER service driver and attach the AER service driver to the RCEC device.
Co-developed-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>
A few questions and comments for this patch.
See inline.
Jonathan
Quoted 18 lines
--- drivers/pci/pcie/aer.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)drivers/pci/pcie/aer.c +1
-1@@ -303,7 +303,7 @@ int pci_aer_raw_clear_status(struct pci_dev *dev)return -EIO ; port_type =pci_pcie_type (dev); -if (port_type ==PCI_EXP_TYPE_ROOT_PORT ) { +if (port_type ==PCI_EXP_TYPE_ROOT_PORT || port_type ==PCI_EXP_TYPE_RC_EC ) {pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , &status);pci_write_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , status); }
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 396
Quoted 8 lines
drivers/pci/pcie/aer.c +5
-0@@ -389,6 +389,12 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);pci_aer_clear_status (dev); + +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) { +if (!pci_find_ext_capability (dev,PCI_EXT_CAP_ID_RCEC )) +return ; +pci_info (dev,"AER: RCEC CAP FOUND and cap_has_rtctl = %d\n" , n);
It feels like failing to find an RC_EC extended cap in a RCEC deserved a nice strong error message. Perhaps this isn't the right place to do it though. For that matter, why are we checking for it here?
drivers/pci/pcie/aer.c static umode_t aer_stats_attrs_are_visible(struct kobject *kobj, · line 585
Quoted 8 lines
drivers/pci/pcie/aer.c +1
-0+ } }void pci_aer_exit (struct pci_dev *dev)@@ -577,7 +583,8 @@ static umode_t aer_stats_attrs_are_visible(struct kobject *kobj, if ((a == &dev_attr_aer_rootport_total_err_cor.attr || a == &dev_attr_aer_rootport_total_err_fatal.attr || a == &dev_attr_aer_rootport_total_err_nonfatal.attr) &&
It is a bit ugly to have these called rootport_total_err etc for the rcec. Perhaps we should just add additional attributes to reflect we are looking at an RCEC?
Quoted 27 lines
drivers/pci/pcie/aer.c +7
-2- pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) + ((pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) && + (pci_pcie_type(pdev) != PCI_EXP_TYPE_RC_EC)))return 0 ;return a->mode;@@ -894,7 +901,10 @@ static bool find_source_device(struct pci_dev *parent,if (result)return true; -pci_walk_bus (parent->subordinate, find_device_iter, e_info); +if (pci_pcie_type (parent) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (parent, find_device_iter, e_info); +else +pci_walk_bus (parent->subordinate, find_device_iter, e_info);if (!e_info->error_dev_num) {pci_info (parent,"can't find device of ID%04x\n" , e_info->id);@@ -1030,6 +1040,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)if (!(info->status & ~info->mask))return 0 ; }else if (pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ||pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || info->severity ==AER_NONFATAL ) {
drivers/pci/pcie/aer.c static int set_device_error_reporting(struct pci_dev *dev, void *data) · line 1197
Quoted 5 lines
drivers/pci/pcie/aer.c +2
-0@@ -1182,6 +1193,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)int type =pci_pcie_type (dev); if ((type == PCI_EXP_TYPE_ROOT_PORT) || + (type == PCI_EXP_TYPE_RC_EC) || + (type == PCI_EXP_TYPE_RC_END) ||
Why add RC_END here?
Quoted 31 lines
drivers/pci/pcie/aer.c +10
-3(type == PCI_EXP_TYPE_UPSTREAM) || (type == PCI_EXP_TYPE_DOWNSTREAM)) {if (enable)@@ -1206,9 +1219,11 @@ static void set_downstream_devices_error_reporting(struct pci_dev *dev, {set_device_error_reporting (dev, &enable); -if (!dev->subordinate) -return ; -pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, set_device_error_reporting, &enable); +else if (dev->subordinate) +pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); + }/** @@ -1306,6 +1321,11 @@ static int aer_probe(struct pcie_device *dev)struct device *device = &dev->device;struct pci_dev *port = dev->port; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + rpc =devm_kzalloc (device,sizeof (struct aer_rpc ),GFP_KERNEL );if (!rpc)return -ENOMEM ;
drivers/pci/pcie/aer.c static pci_ers_result_t aer_root_reset(struct pci_dev *dev) · line 1385
Quoted 5 lines
drivers/pci/pcie/aer.c +1
-1@@ -1362,7 +1382,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)static struct pcie_port_service_driver aerdriver = { .name ="aer" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT ,
Why this particular change? Seems that is a lot wider than simply adding RCEC. Obviously we'll then drop out in the aer_probe but it is still rather inelegant.
Quoted 3 lines
drivers/pci/pcie/aer.c +0
-0.service = PCIE_PORT_SERVICE_AER, .probe = aer_probe,
Re: [RFC PATCH 7/9] PCI/AER: Add RCEC AER handling
Quoted 40 lines · Jonathan Cameron
Quoted 7 lines · On Fri, 24 Jul 2020 10:22:21 -0700 Sean V Kelley <[email protected]>…
The Root Complex Event Collectors(RCEC) appear as peers to Root Ports and also have the AER capability. So add RCEC support to the current AER service driver and attach the AER service driver to the RCEC device.
Co-developed-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>A few questions and comments for this patch.
See inline.
Jonathan
Quoted 22 lines
--- drivers/pci/pcie/aer.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)drivers/pci/pcie/aer.c +1
-1@@ -303,7 +303,7 @@ int pci_aer_raw_clear_status(struct pci_dev *dev)return -EIO ; port_type =pci_pcie_type (dev); -if (port_type ==PCI_EXP_TYPE_ROOT_PORT ) { +if (port_type ==PCI_EXP_TYPE_ROOT_PORT || port_type ==PCI_EXP_TYPE_RC_EC ) {pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , &status);pci_write_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , status);drivers/pci/pcie/aer.c +0
-0}@@ -389,6 +389,12 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);
Quoted 10 lines
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 396
Quoted 6 lines
drivers/pci/pcie/aer.c +5
-0pci_aer_clear_status (dev); + +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) { +if (!pci_find_ext_capability (dev,PCI_EXT_CAP_ID_RCEC )) +return ; +pci_info (dev,"AER: RCEC CAP FOUND and cap_has_rtctl = %d\n" , n);It feels like failing to find an
RC_ECextended cap in a RCEC deserved a nice strong error message. Perhaps this isn't the right place to do it though. For that matter, why are we checking for it here?
Sorry, I’ve left an in-development output in the code. Will replace with a check with more meaningful output elsewhere.
Quoted 4 lines
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-0+ } }
Quoted 10 lines
drivers/pci/pcie/aer.c static umode_t aer_stats_attrs_are_visible(struct · line 586
Quoted 6 lines
void
pci_aer_exit(struct pci_dev *dev)drivers/pci/pcie/aer.c +0
-0@@ -577,7 +583,8 @@ static umode_t aer_stats_attrs_are_visible(struct kobject *kobj, if ((a == &dev_attr_aer_rootport_total_err_cor.attr || a == &dev_attr_aer_rootport_total_err_fatal.attr || a == &dev_attr_aer_rootport_total_err_nonfatal.attr) &&It is a bit ugly to have these called
rootport_total_erretc for the rcec. Perhaps we should just add additional attributes to reflect we are looking at an RCEC?
I was trying to avoid any renaming to reduce churn as I did with my first patch for ACPI / CLX_OSC support. Will take a look.
Quoted 33 lines
Quoted 33 lines
drivers/pci/pcie/aer.c +2
-1- pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) + ((pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) && + (pci_pcie_type(pdev) != PCI_EXP_TYPE_RC_EC)))return 0 ;drivers/pci/pcie/aer.c +4
-1return a->mode;@@ -894,7 +901,10 @@ static bool find_source_device(struct pci_dev *parent, if (result)return true; -pci_walk_bus (parent->subordinate, find_device_iter, e_info); +if (pci_pcie_type (parent) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (parent, find_device_iter, e_info); +else +pci_walk_bus (parent->subordinate, find_device_iter, e_info);if (!e_info->error_dev_num) {drivers/pci/pcie/aer.c +1
-0pci_info (parent,"can't find device of ID%04x\n" , e_info->id);@@ -1030,6 +1040,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)if (!(info->status & ~info->mask))return 0 ; }else if (pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ||pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || info->severity ==AER_NONFATAL ) {@@ -1182,6 +1193,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)int type =pci_pcie_type (dev);
Quoted 5 lines
drivers/pci/pcie/aer.c static int set_device_error_reporting(struct pci_dev *dev, void *data) · line 1197
Quoted 3 lines
drivers/pci/pcie/aer.c +2
-0if ((type == PCI_EXP_TYPE_ROOT_PORT) || + (type == PCI_EXP_TYPE_RC_EC) || + (type == PCI_EXP_TYPE_RC_END) ||Why add
RC_ENDhere?
I’m not clear on your question. Errors can come from RCEC or RCiEPs. We still need to enable reporting by the RCiEPs.
Quoted 37 lines
Quoted 37 lines
drivers/pci/pcie/aer.c +0
-0(type == PCI_EXP_TYPE_UPSTREAM) || (type == PCI_EXP_TYPE_DOWNSTREAM)) {drivers/pci/pcie/aer.c +5
-3if (enable)@@ -1206,9 +1219,11 @@ static void set_downstream_devices_error_reporting(struct pci_dev *dev, { set_device_error_reporting(dev, &enable); -if (!dev->subordinate) -return ; -pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, set_device_error_reporting, &enable); +else if (dev->subordinate) +pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +}
/**
drivers/pci/pcie/aer.c +5
-0@@ -1306,6 +1321,11 @@ static int aer_probe(struct pcie_device *dev)struct device *device = &dev->device;struct pci_dev *port = dev->port; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + rpc =devm_kzalloc (device,sizeof (struct aer_rpc ),GFP_KERNEL );if (!rpc)return -ENOMEM ;@@ -1362,7 +1382,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
Quoted 8 lines
drivers/pci/pcie/aer.c static pci_ers_result_t aer_root_reset(struct pci_dev *dev) · line 1385
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-1static struct pcie_port_service_driver aerdriver = { .name ="aer" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT ,Why this particular change? Seems that is a lot wider than simply adding RCEC. Obviously we'll then drop out in the
aer_probebut it is still rather inelegant.
In order to extend the service drivers to non-root-port devices (i.e., RCEC), the simple path appeared to only require setting the type to ANY_PORT and catching the needed types arriving in the probe. Would you prefer extending to a type2? I’m not sure how one is more elegant than another but open to that approach. However, this seems to require less code perhaps and seems consistent with most ‘drop-out’ conditional patterns in the kernel. The same applies to pme.
Thanks,
Sean
Quoted 4 lines
Quoted 4 lines
drivers/pci/pcie/aer.c +0
-0.service = PCIE_PORT_SERVICE_AER,.probe =
aer_probe,
Re: [RFC PATCH 7/9] PCI/AER: Add RCEC AER handling
Quoted 112 lines · On Mon, 27 Jul 2020 08:19:39 -0700 Sean V Kelley <[email protected]>…
Quoted 41 lines · Jonathan Cameron
Quoted 8 lines · On Fri, 24 Jul 2020 10:22:21 -0700 Sean V Kelley <[email protected]>…
The Root Complex Event Collectors(RCEC) appear as peers to Root Ports and also have the AER capability. So add RCEC support to the current AER
service driver and attach the AER service driver to the RCEC device.Co-developed-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>A few questions and comments for this patch.
See inline.
Jonathan
Quoted 22 lines
--- drivers/pci/pcie/aer.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)drivers/pci/pcie/aer.c +1
-1@@ -303,7 +303,7 @@ int pci_aer_raw_clear_status(struct pci_dev *dev)return -EIO ; port_type =pci_pcie_type (dev); -if (port_type ==PCI_EXP_TYPE_ROOT_PORT ) { +if (port_type ==PCI_EXP_TYPE_ROOT_PORT || port_type ==PCI_EXP_TYPE_RC_EC ) {pci_read_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , &status);pci_write_config_dword (dev, aer +PCI_ERR_ROOT_STATUS , status);drivers/pci/pcie/aer.c +0
-0}@@ -389,6 +389,12 @@ void pci_aer_init(struct pci_dev *dev)pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ERR ,sizeof (u32) * n);Quoted 11 lines
drivers/pci/pcie/aer.c void pci_aer_init(struct pci_dev *dev) · line 396
Quoted 6 lines
drivers/pci/pcie/aer.c +5
-0pci_aer_clear_status (dev); + +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) { +if (!pci_find_ext_capability (dev,PCI_EXT_CAP_ID_RCEC )) +return ; +pci_info (dev,"AER: RCEC CAP FOUND and cap_has_rtctl = %d\n" , n);It feels like failing to find an
RC_ECextended cap in a RCEC deserved a nice strong error message. Perhaps this isn't the right place to do it
though. For that matter, why are we checking for it here?Sorry, I’ve left an in-development output in the code. Will replace with a check with more meaningful output elsewhere.
Quoted 4 lines
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-0+ } }Quoted 12 lines
drivers/pci/pcie/aer.c static umode_t aer_stats_attrs_are_visible(struct · line 586
Quoted 6 lines
void
pci_aer_exit(struct pci_dev *dev)drivers/pci/pcie/aer.c +0
-0@@ -577,7 +583,8 @@ static umode_t aer_stats_attrs_are_visible(struct kobject *kobj, if ((a == &dev_attr_aer_rootport_total_err_cor.attr || a == &dev_attr_aer_rootport_total_err_fatal.attr || a == &dev_attr_aer_rootport_total_err_nonfatal.attr) &&It is a bit ugly to have these called
rootport_total_erretc for the rcec.
Perhaps we should just add additional attributes to reflect we are looking at
an RCEC?I was trying to avoid any renaming to reduce churn as I did with my first patch for ACPI /
CLX_OSCsupport.
Will take a look.Quoted 33 lines
Quoted 33 lines
drivers/pci/pcie/aer.c +2
-1- pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) + ((pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT) && + (pci_pcie_type(pdev) != PCI_EXP_TYPE_RC_EC)))return 0 ;drivers/pci/pcie/aer.c +4
-1return a->mode;@@ -894,7 +901,10 @@ static bool find_source_device(struct pci_dev *parent, if (result)return true; -pci_walk_bus (parent->subordinate, find_device_iter, e_info); +if (pci_pcie_type (parent) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (parent, find_device_iter, e_info); +else +pci_walk_bus (parent->subordinate, find_device_iter, e_info);if (!e_info->error_dev_num) {drivers/pci/pcie/aer.c +1
-0pci_info (parent,"can't find device of ID%04x\n" , e_info->id);@@ -1030,6 +1040,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)if (!(info->status & ~info->mask))return 0 ; }else if (pci_pcie_type (dev) ==PCI_EXP_TYPE_ROOT_PORT || +pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ||pci_pcie_type (dev) ==PCI_EXP_TYPE_DOWNSTREAM || info->severity ==AER_NONFATAL ) {@@ -1182,6 +1193,8 @@ static int set_device_error_reporting(struct pci_dev *dev, void *data)int type =pci_pcie_type (dev);
Quoted 8 lines
Quoted 5 lines
drivers/pci/pcie/aer.c static int set_device_error_reporting(struct pci_dev *dev, void *data) · line 1197
Quoted 3 lines
drivers/pci/pcie/aer.c +2
-0if ((type == PCI_EXP_TYPE_ROOT_PORT) || + (type == PCI_EXP_TYPE_RC_EC) || + (type == PCI_EXP_TYPE_RC_END) ||Why add
RC_ENDhere?I’m not clear on your question. Errors can come from RCEC or RCiEPs. We still need to enable reporting by the RCiEPs.
I was curious to see that we need it in this code path for an RCiEP but not for a normal EP. From a quick glance it looks like that is often done in the drivers for the EPs themselves rather than here.
Quoted 47 lines
Quoted 38 lines
Quoted 38 lines
drivers/pci/pcie/aer.c +0
-0(type == PCI_EXP_TYPE_UPSTREAM) || (type == PCI_EXP_TYPE_DOWNSTREAM)) {drivers/pci/pcie/aer.c +5
-3if (enable)@@ -1206,9 +1219,11 @@ static void set_downstream_devices_error_reporting(struct pci_dev *dev, { set_device_error_reporting(dev, &enable); -if (!dev->subordinate) -return ; -pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +if (pci_pcie_type (dev) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (dev, set_device_error_reporting, &enable); +else if (dev->subordinate) +pci_walk_bus (dev->subordinate, set_device_error_reporting, &enable); +}
/**
drivers/pci/pcie/aer.c +5
-0@@ -1306,6 +1321,11 @@ static int aer_probe(struct pcie_device *dev)struct device *device = &dev->device;struct pci_dev *port = dev->port; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + rpc =devm_kzalloc (device,sizeof (struct aer_rpc ),GFP_KERNEL );if (!rpc)return -ENOMEM ;@@ -1362,7 +1382,7 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)Quoted 8 lines
drivers/pci/pcie/aer.c static pci_ers_result_t aer_root_reset(struct pci_dev *dev) · line 1385
Quoted 4 lines
drivers/pci/pcie/aer.c +1
-1static struct pcie_port_service_driver aerdriver = { .name ="aer" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT ,Why this particular change? Seems that is a lot wider than simply adding RCEC. Obviously we'll then drop out in the
aer_probebut it is still rather inelegant.
Quoted 7 lines
In order to extend the service drivers to non-root-port devices (i.e., RCEC), the simple path appeared to only require setting the type to
ANY_PORTand catching the needed types arriving in the probe. Would you prefer extending to a type2? I’m not sure how one is more elegant than another but open to that approach. However, this seems to require less code perhaps and seems consistent with most ‘drop-out’ conditional patterns in the kernel. The same applies to pme.
I'd miss understood this bit. It's fine as you have it here.
Jonathan
Quoted 10 lines
Thanks,
Sean
Quoted 4 lines
Quoted 4 lines
drivers/pci/pcie/aer.c +0
-0.service = PCIE_PORT_SERVICE_AER,.probe =
aer_probe,
RFC PATCH 8/9 PCI/PME: Add RCEC PME handling
In reply to Sean V KelleyThe Root Complex Event Collectors(RCEC) appear as peers of Root Ports and also have the PME capability. So add RCEC support to the current PME service driver and attach the PME service driver to the RCEC device.
--- drivers/pci/pcie/pme.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/pci/pcie/pme.c +11 -4
@@ -310,7 +310,10 @@ static int pcie_pme_can_wakeup(struct pci_dev *dev, void *ign)static void pcie_pme_mark_devices (struct pci_dev *port) {pcie_pme_can_wakeup (port,NULL ); -if (port->subordinate) + +if (pci_pcie_type (port) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (port, pcie_pme_can_wakeup,NULL ); +else if (port->subordinate)pci_walk_bus (port->subordinate, pcie_pme_can_wakeup,NULL ); }
@@ -320,10 +323,15 @@ static void pcie_pme_mark_devices(struct pci_dev *port)*/ static int pcie_pme_probe (struct pcie_device *srv) { -struct pci_dev *port; +struct pci_dev *port = srv->port;struct pcie_pme_service_data *data;int ret; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)return -ENOMEM ;
@@ -333,7 +341,6 @@ static int pcie_pme_probe(struct pcie_device *srv) data->srv = srv;set_service_data (srv, data); - port = srv->port;pcie_pme_interrupt_enable (port, false);pcie_clear_root_pme_status (port);
@@ -445,7 +452,7 @@ static void pcie_pme_remove(struct pcie_device *srv)static struct pcie_port_service_driver pcie_pme_driver = { .name ="pcie_pme" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT , .service =PCIE_PORT_SERVICE_PME , .probe = pcie_pme_probe,
Signature · 1 line
2.27.0
Re: [RFC PATCH 8/9] PCI/PME: Add RCEC PME handling
Quoted 53 lines · Sean V Kelley
The Root Complex Event Collectors(RCEC) appear as peers of Root Ports and also have the PME capability. So add RCEC support to the current PME service driver and attach the PME service driver to the RCEC device.
Co-developed-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/pme.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)drivers/pci/pcie/pme.c +10
-3@@ -310,7 +310,10 @@ static int pcie_pme_can_wakeup(struct pci_dev *dev, void *ign)static void pcie_pme_mark_devices (struct pci_dev *port) {pcie_pme_can_wakeup (port,NULL ); -if (port->subordinate) + +if (pci_pcie_type (port) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (port, pcie_pme_can_wakeup,NULL ); +else if (port->subordinate)pci_walk_bus (port->subordinate, pcie_pme_can_wakeup,NULL ); }@@ -320,10 +323,15 @@ static void pcie_pme_mark_devices(struct pci_dev *port)*/ static int pcie_pme_probe (struct pcie_device *srv) { -struct pci_dev *port; +struct pci_dev *port = srv->port;struct pcie_pme_service_data *data;int ret; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)return -ENOMEM ;@@ -333,7 +341,6 @@ static int pcie_pme_probe(struct pcie_device *srv) data->srv = srv;set_service_data (srv, data); - port = srv->port;pcie_pme_interrupt_enable (port, false);pcie_clear_root_pme_status (port);
drivers/pci/pcie/pme.c static void pcie_pme_remove(struct pcie_device *srv) · line 455
Quoted 5 lines
drivers/pci/pcie/pme.c +1
-1@@ -445,7 +452,7 @@ static void pcie_pme_remove(struct pcie_device *srv)static struct pcie_port_service_driver pcie_pme_driver = { .name ="pcie_pme" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT ,
Maybe we can use port_type for driver matching. There is no need to check the type of port in pcie_pme_probe function.
Jay
Quoted 4 lines
drivers/pci/pcie/pme.c +0
-0.service = PCIE_PORT_SERVICE_PME, .probe = pcie_pme_probe,
Re: [RFC PATCH 8/9] PCI/PME: Add RCEC PME handling
Quoted 53 lines · On Tue, 4 Aug 2020 16:35:59 +0800 Jay Fang <[email protected]> wrote:
Quoted 52 lines · Sean V Kelley
The Root Complex Event Collectors(RCEC) appear as peers of Root Ports and also have the PME capability. So add RCEC support to the current PME service driver and attach the PME service driver to the RCEC device.
Co-developed-by Qiuxu Zhuo <[email protected]>Signed-off-by Sean V Kelley <[email protected]>Signed-off-by Qiuxu Zhuo <[email protected]>--- drivers/pci/pcie/pme.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)drivers/pci/pcie/pme.c +10
-3@@ -310,7 +310,10 @@ static int pcie_pme_can_wakeup(struct pci_dev *dev, void *ign)static void pcie_pme_mark_devices (struct pci_dev *port) {pcie_pme_can_wakeup (port,NULL ); -if (port->subordinate) + +if (pci_pcie_type (port) ==PCI_EXP_TYPE_RC_EC ) +pcie_walk_rcec (port, pcie_pme_can_wakeup,NULL ); +else if (port->subordinate)pci_walk_bus (port->subordinate, pcie_pme_can_wakeup,NULL ); }@@ -320,10 +323,15 @@ static void pcie_pme_mark_devices(struct pci_dev *port)*/ static int pcie_pme_probe (struct pcie_device *srv) { -struct pci_dev *port; +struct pci_dev *port = srv->port;struct pcie_pme_service_data *data;int ret; +/* Limit to Root Ports or Root Complex Event Collectors */ +if ((pci_pcie_type (port) !=PCI_EXP_TYPE_RC_EC ) && + (pci_pcie_type (port) !=PCI_EXP_TYPE_ROOT_PORT )) +return -ENODEV ; + data =kzalloc (sizeof (*data),GFP_KERNEL );if (!data)return -ENOMEM ;@@ -333,7 +341,6 @@ static int pcie_pme_probe(struct pcie_device *srv) data->srv = srv;set_service_data (srv, data); - port = srv->port;pcie_pme_interrupt_enable (port, false);pcie_clear_root_pme_status (port);
Quoted 9 lines
drivers/pci/pcie/pme.c static void pcie_pme_remove(struct pcie_device *srv) · line 455
Quoted 6 lines
drivers/pci/pcie/pme.c +1
-1@@ -445,7 +452,7 @@ static void pcie_pme_remove(struct pcie_device *srv)static struct pcie_port_service_driver pcie_pme_driver = { .name ="pcie_pme" , - .port_type =PCI_EXP_TYPE_ROOT_PORT , + .port_type =PCIE_ANY_PORT ,Maybe we can use
port_typefor driver matching. There is no need to check the type of port inpcie_pme_probefunction.
I walked into the same hole for the AER case.port_type is effectively an enum so there is no way of specifying several types unless you want to register different instances of pcie_port_service_driver and that isn't currently possible.
The PCIE_ANY_PORT is a special case value.
https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/pci_regs.h#L477
https://elixir.bootlin.com/linux/latest/source/drivers/pci/pci-driver.c#L1651
So odd corner case, but I think this is the right solution. Anything better would require a lot more code to change.
Jonathan
Quoted 7 lines
Jay
Quoted 4 lines
drivers/pci/pcie/pme.c +0
-0.service = PCIE_PORT_SERVICE_PME, .probe = pcie_pme_probe,
RFC PATCH 9/9 PCI/AER: Add RCEC AER error injection support
In reply to Sean V KelleyFrom: Qiuxu Zhuo <[email protected]>
The Root Complex Event Collectors(RCEC) appear as peers to Root Ports and also have the AER capability. So add RCEC support to the current AER error injection driver.
--- drivers/pci/pcie/aer_inject.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer_inject.c +4 -1
@@ -333,8 +333,11 @@ static int aer_inject(struct aer_error_inj *einj)if (!dev)return -ENODEV ; rpdev =pcie_find_root_port (dev); +/* If Root port not found, try to find an RCEC */ +if (!rpdev) + rpdev = dev->rcec;if (!rpdev) { -pci_err (dev,"Root port not found\n" ); +pci_err (dev,"Root port or RCEC not found\n" ); ret = -ENODEV ; goto out_put; }
Signature · 1 line
2.27.0
Quoted 26 lines · On Fri, 24 Jul 2020 10:22:14 -0700 Sean V Kelley <[email protected]>…
Root Complex Event Collectors (RCEC) provide support for terminating error and PME messages from Root Complex Integrated Endpoints (RCiEPs). An RCEC resides on a Bus in the Root Complex. Multiple RCECs can in fact reside on a single bus. An RCEC will explicitly declare supported RCiEPs through the Root Complex Endpoint Association Extended Capability.
(See PCIe 5.0-1, sections 1.3.2.3 (RCiEP), and 7.9.10 (RCEC Ext. Cap.))
The kernel lacks handling for these RCECs and the error messages received from their respective associated RCiEPs. More recently, a new CPU interconnect, Compute eXpress Link (CXL) depends on RCEC capabilities for purposes of error messaging from CXL 1.1 supported RCiEP devices.
DocLink: https://www.computeexpresslink.org/
This use case is not limited to CXL. Existing hardware today includes support for RCECs, such as the Denverton microserver product family. Future hardware will be forthcoming.
(See Intel Document, Order number: 33061-003US)
So services such as AER or PME could be associated with an RCEC driver. In the case of CXL, if an RCiEP (i.e., CXL 1.1 device) is associated with a platform's RCEC it shall signal PME and AER error conditions through that RCEC.
Quoted 4 lines
Towards the above use cases, add the missing RCEC class and extend the PCIe Root Port and service drivers to allow association of RCiEPs to their respective parent RCEC and facilitate handling of terminating error and PME messages.
Silly question number 1. Why an RFC? I always find it helps to highlight which bits you are unsure on / want particular input on.
Otherwise, I've left the PME and error injection patches as I don't really know anything about those two paths.
I'll fire up my APEI etc test VMs in the nex day or so and report back if there any problems with that case (fairly sure there is one in patch 6, highlighted in review but it is possible I've missed others.
It all seems to have come together rather simpler than I was expecting which is great!
Thanks,
Jonathan
Quoted 94 lines
TESTING:
Results: 1) Show RCiEPs which are associated with RCECs: Run dmesg | grep "RCiEP" Log: [ 8.981698] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e8:01.0 [ 8.988830] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e8:02.0 [ 8.995956] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e9:00.0 [ 9.023034] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:01.0 [ 9.030159] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:02.0 [ 9.037282] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ee:00.0 [ 9.064294] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:01.0 [ 9.071409] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:02.0 [ 9.078526] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f3:00.0 [ 9.105535] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:01.0 [ 9.112652] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:02.0 [ 9.119774] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f8:00.0 2) Inject a correctable error to the RCiEP 0000:e9:00.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:e9:00.0 COR_STATUS BAD_TLP HEADER_LOG 0 1 2 3 Log: [ 253.248362] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000040/00000000 into device 0000:e9:00.0 [ 253.260656] pcieport 0000:e8:00.4: AER: Corrected error received: 0000:e9:00.0 [ 253.269919] pci 0000:e9:00.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID) [ 253.282549] pci 0000:e9:00.0: AER: device [8086:4940] error status/mask=00000040/00002000 [ 253.293937] pci 0000:e9:00.0: AER: [ 6] BadTLP 3) Inject a non-fatal error to the RCiEP 0000:e8:01.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:e8:01.0 UNCOR_STATUS COMP_ABORT HEADER_LOG 0 1 2 3 Log: [ 288.405326] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000000/00008000 into device 0000:e8:01.0 [ 288.416881] pcieport 0000:e8:00.4: AER: Uncorrected (Non-Fatal) error received: 0000:e8:01.0 [ 288.427487] igen6_edac 0000:e8:01.0: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Completer ID) [ 288.442098] igen6_edac 0000:e8:01.0: AER: device [8086:0b25] error status/mask=00008000/00100000 [ 288.452869] igen6_edac 0000:e8:01.0: AER: [15] CmpltAbrt [ 288.461118] igen6_edac 0000:e8:01.0: AER: TLP Header: 00000000 00000001 00000002 00000003 [ 288.471192] igen6_edac 0000:e8:01.0: AER: device recovery successful 4) Inject a fatal error to the RCiEP 0000:ed:01.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:ed:01.0 UNCOR_STATUS MALF_TLP HEADER_LOG 0 1 2 3 Log: [ 535.537281] pcieport 0000:ed:00.4: aer_inject: Injecting errors 00000000/00040000 into device 0000:ed:01.0 [ 535.551911] pcieport 0000:ed:00.4: AER: Uncorrected (Fatal) error received: 0000:ed:01.0 [ 535.561556] igen6_edac 0000:ed:01.0: AER: PCIe Bus Error: severity=Uncorrected (Fatal), type=Inaccessible, (Unregistered Agent ID) [ 535.684964] igen6_edac 0000:ed:01.0: AER: device recovery successfulJonathan Cameron (1):
PCI/AER: Extend AER error handling to RCECsQiuxu Zhuo (6):
pci_ids: Add class code and extended capability for RCEC
PCI: Extend Root Port Driver to support RCEC
PCI/portdrv: Addpcie_walk_rcec()to walk RCiEPs associated with RCEC PCI/AER: Apply function level reset to RCiEP on fatal error
PCI: Add 'rcec' field topci_devfor associated RCiEPs PCI/AER: Add RCEC AER error injection supportSean V Kelley (2):
PCI/AER: Add RCEC AER handling
PCI/PME: Add RCEC PME handlingdrivers/pci/pcie/aer.c | 43 ++++++++++++----- drivers/pci/pcie/aer_inject.c | 5 +- drivers/pci/pcie/err.c | 85 +++++++++++++++++++++++++++------ drivers/pci/pcie/pme.c | 15 ++++-- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++ drivers/pci/pcie/portdrv_pci.c | 20 +++++++- include/linux/pci.h | 3 ++ include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++ 10 files changed, 232 insertions(+), 31 deletions(-)--
2.27.0
Hi Jonathan,
Quoted 29 lines · Jonathan Cameron
Quoted 26 lines · On Fri, 24 Jul 2020 10:22:14 -0700 Sean V Kelley <[email protected]>…
Root Complex Event Collectors (RCEC) provide support for terminating error and PME messages from Root Complex Integrated Endpoints (RCiEPs). An RCEC resides on a Bus in the Root Complex. Multiple RCECs can in fact reside on a single bus. An RCEC will explicitly declare supported RCiEPs through the Root Complex Endpoint Association Extended Capability.
(See PCIe 5.0-1, sections 1.3.2.3 (RCiEP), and 7.9.10 (RCEC Ext. Cap.))
The kernel lacks handling for these RCECs and the error messages received from their respective associated RCiEPs. More recently, a new CPU interconnect, Compute eXpress Link (CXL) depends on RCEC capabilities for purposes of error messaging from CXL 1.1 supported RCiEP devices.
DocLink: https://www.computeexpresslink.org/
This use case is not limited to CXL. Existing hardware today includes support for RCECs, such as the Denverton microserver product family. Future hardware will be forthcoming.
(See Intel Document, Order number: 33061-003US)
So services such as AER or PME could be associated with an RCEC driver. In the case of CXL, if an RCiEP (i.e., CXL 1.1 device) is associated with a platform's RCEC it shall signal PME and AER error conditions through that RCEC.
Quoted 7 lines
Quoted 4 lines
Towards the above use cases, add the missing RCEC class and extend the PCIe Root Port and service drivers to allow association of RCiEPs to their respective parent RCEC and facilitate handling of terminating error and PME messages.
Silly question number 1. Why an RFC? I always find it helps to highlight which bits you are unsure on / want particular input on.
I suppose it was because we were continuing the conversation from discussion on the mailing list. And I was not fully sure about the impact to your use case. No worries, I will remove it.
Quoted 10 lines
Otherwise, I've left the PME and error injection patches as I don't really know anything about those two paths.
I'll fire up my APEI etc test VMs in the nex day or so and report back if there any problems with that case (fairly sure there is one in patch 6, highlighted in review but it is possible I've missed others.
It all seems to have come together rather simpler than I was expecting which is great!
Sounds good, looking forward to your feedback.
Sean
Quoted 128 lines
Thanks,
Jonathan
Quoted 122 lines
TESTING:
Results:
1) Show RCiEPs which are associated with RCECs:
Run dmesg | grep "RCiEP"
Log:
[ 8.981698] pcieport 0000:e8:00.4: RCiEP(under an RCEC)
0000:e8:01.0[ 8.988830] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e8:02.0 [ 8.995956] pcieport 0000:e8:00.4: RCiEP(under an RCEC) 0000:e9:00.0 [ 9.023034] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:01.0 [ 9.030159] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ed:02.0 [ 9.037282] pcieport 0000:ed:00.4: RCiEP(under an RCEC) 0000:ee:00.0 [ 9.064294] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:01.0 [ 9.071409] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f2:02.0 [ 9.078526] pcieport 0000:f2:00.4: RCiEP(under an RCEC) 0000:f3:00.0 [ 9.105535] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:01.0 [ 9.112652] pcieport 0000:f7:00.4: RCiEP(under an RCEC) 0000:f7:02.0 [ 9.119774] pcieport 0000:f7:00.4: RCiEP(under an RCEC)0000:f8:00.0
2) Inject a correctable error to the RCiEP 0000:e9:00.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:e9:00.0 COR_STATUS BAD_TLP HEADER_LOG 0 1 2 3 Log: [ 253.248362] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000040/00000000 into device 0000:e9:00.0 [ 253.260656] pcieport 0000:e8:00.4: AER: Corrected error received: 0000:e9:00.0 [ 253.269919] pci 0000:e9:00.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID) [ 253.282549] pci 0000:e9:00.0: AER: device [8086:4940] error status/mask=00000040/00002000 [ 253.293937] pci 0000:e9:00.0: AER: [ 6] BadTLP3) Inject a non-fatal error to the RCiEP 0000:e8:01.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:e8:01.0 UNCOR_STATUS COMP_ABORT HEADER_LOG 0 1 2 3 Log: [ 288.405326] pcieport 0000:e8:00.4: aer_inject: Injecting errors 00000000/00008000 into device 0000:e8:01.0 [ 288.416881] pcieport 0000:e8:00.4: AER: Uncorrected (Non-Fatal) error received: 0000:e8:01.0 [ 288.427487] igen6_edac 0000:e8:01.0: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Completer ID) [ 288.442098] igen6_edac 0000:e8:01.0: AER: device [8086:0b25] error status/mask=00008000/00100000 [ 288.452869] igen6_edac 0000:e8:01.0: AER: [15] CmpltAbrt [ 288.461118] igen6_edac 0000:e8:01.0: AER: TLP Header: 00000000 00000001 00000002 00000003 [ 288.471192] igen6_edac 0000:e8:01.0: AER: device recoverysuccessful
4) Inject a fatal error to the RCiEP 0000:ed:01.0 Run ./aer_inject <a parameter file as below>: AER PCI_ID 0000:ed:01.0 UNCOR_STATUS MALF_TLP HEADER_LOG 0 1 2 3 Log: [ 535.537281] pcieport 0000:ed:00.4: aer_inject: Injecting errors 00000000/00040000 into device 0000:ed:01.0 [ 535.551911] pcieport 0000:ed:00.4: AER: Uncorrected (Fatal) error received: 0000:ed:01.0 [ 535.561556] igen6_edac 0000:ed:01.0: AER: PCIe Bus Error: severity=Uncorrected (Fatal), type=Inaccessible, (Unregistered Agent ID) [ 535.684964] igen6_edac 0000:ed:01.0: AER: device recoverysuccessful
Jonathan Cameron (1):
PCI/AER: Extend AER error handling to RCECsQiuxu Zhuo (6):
pci_ids: Add class code and extended capability for RCEC
PCI: Extend Root Port Driver to support RCEC PCI/portdrv: Addpcie_walk_rcec()to walk RCiEPs associated with
RCEC
PCI/AER: Apply function level reset to RCiEP on fatal error
PCI: Add 'rcec' field topci_devfor associated RCiEPs PCI/AER: Add RCEC AER error injection supportSean V Kelley (2):
PCI/AER: Add RCEC AER handling
PCI/PME: Add RCEC PME handlingdrivers/pci/pcie/aer.c | 43 ++++++++++++----- drivers/pci/pcie/aer_inject.c | 5 +- drivers/pci/pcie/err.c | 85 +++++++++++++++++++++++++++------ drivers/pci/pcie/pme.c | 15 ++++-- drivers/pci/pcie/portdrv.h | 2 + drivers/pci/pcie/portdrv_core.c | 82 +++++++++++++++++++++++++++++++ drivers/pci/pcie/portdrv_pci.c | 20 +++++++- include/linux/pci.h | 3 ++ include/linux/pci_ids.h | 1 + include/uapi/linux/pci_regs.h | 7 +++ 10 files changed, 232 insertions(+), 31 deletions(-)--
2.27.0