Skip to contentSkip to conversation

pci/aer: Handle Advisory Non-Fatal properly

PATCH Superseded Wang, Qingshun 3 participants 14 messages Jan 11, 2024 – Jan 16, 2024

Subsystems (6)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover pci/aer: Handle Advisory Non-Fatal properly

According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal ERR_COR instead of ERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.

For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:

[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           

If the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:

[  170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  170.552420] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00041000/00180020
[  170.561904] pcieport 0000:b7:02.0:    [12] TLP                    (First)
[  170.569656] pcieport 0000:b7:02.0:    [18] MalfTLP       

To handle this case properly, this patch set adds additional fields in aer_err_info to track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data to aer_event tracepoint, which would help to better understand ANFE and other errors for external observation.

In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:

[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           
[  148.459868] pcieport 0000:b7:02.0:   Uncorrectable errors that may cause Advisory Non-Fatal:
[  148.459868] pcieport 0000:b7:02.0:    [18] TLP

Wang, Qingshun (4):
pci/aer: Store more information in aer_err_info
pci/aer: Handle Advisory Non-Fatal properly
pci/aer: Fetch information for FTrace
ras: Trace more information in aer_event

 drivers/acpi/apei/ghes.c      |  16 ++-
 drivers/cxl/core/pci.c        |  15 ++-
 drivers/pci/pci.h             |  12 ++-
 drivers/pci/pcie/aer.c        | 188 +++++++++++++++++++++++++++-------
 include/linux/aer.h           |   6 +-
 include/linux/pci.h           |  27 +++++
 include/ras/ras_event.h       |  48 ++++++++-
 include/uapi/linux/pci_regs.h |   1 +
 8 files changed, 266 insertions(+), 47 deletions(-)

base-commit: 610a9b8f49fbcf1100716370d3b5f6f884a2835a

Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/4 pci/aer: Store more information in aer_err_info

Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.

Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

---
 drivers/acpi/apei/ghes.c | 10 ++++-
 drivers/cxl/core/pci.c   |  6 ++-
 drivers/pci/pci.h        |  8 +++-
 drivers/pci/pcie/aer.c   | 93 ++++++++++++++++++++++++++--------------
 include/linux/aer.h      |  4 +-
 include/linux/pci.h      | 27 ++++++++++++
 6 files changed, 111 insertions(+), 37 deletions(-)
drivers/acpi/apei/ghes.c +9 -1
index 63ad0541db38..0e7ae84a5d3f 100644
@@ -576,6 +576,8 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
 
 	if (pcie_err->validation_bits & CPER_PCIE_VALID_DEVICE_ID &&
 	    pcie_err->validation_bits & CPER_PCIE_VALID_AER_INFO) {
+		struct pcie_capability_regs *pcie_caps;
+		u16 device_status = 0;
 		unsigned int devfn;
 		int aer_severity;
 		u8 *aer_info;
@@ -598,11 +600,17 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
 			return;
 		memcpy(aer_info, pcie_err->aer_info, sizeof(struct aer_capability_regs));
 
+		if (pcie_err->validation_bits & CPER_PCIE_VALID_CAPABILITY) {
+			pcie_caps = (struct pcie_capability_regs *) &pcie_err->capability;
+			device_status = pcie_caps->device_status;
+		}
+
 		aer_recover_queue(pcie_err->device_id.segment,
 				  pcie_err->device_id.bus,
 				  devfn, aer_severity,
 				  (struct aer_capability_regs *)
-				  aer_info);
+				  aer_info,
+				  device_status);
 	}
 #endif
 }
drivers/cxl/core/pci.c +5 -1
index 37e1652afbc7..2d099c6463b4 100644
@@ -902,6 +902,7 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
 	struct aer_capability_regs aer_regs;
 	struct cxl_dport *dport;
 	struct cxl_port *port;
+	u16 device_status;
 	int severity;
 
 	port = cxl_pci_find_port(pdev, &dport);
@@ -916,7 +917,10 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
 	if (!cxl_rch_get_aer_severity(&aer_regs, &severity))
 		return;
 
-	pci_print_aer(pdev, severity, &aer_regs);
+	if (pcie_capability_read_word(pdev, PCI_EXP_DEVSTA, &device_status))
+		return;
+
+	pci_print_aer(pdev, severity, &aer_regs, device_status);
 
 	if (severity == AER_CORRECTABLE)
 		cxl_handle_rdport_cor_ras(cxlds, dport);
drivers/pci/pci.h +6 -2
index 5ecbcf041179..1e28adfafc06 100644
@@ -405,8 +405,12 @@ struct aer_err_info {
 	unsigned int __pad2:2;
 	unsigned int tlp_header_valid:1;
 
-	unsigned int status;		/* COR/UNCOR Error Status */
-	unsigned int mask;		/* COR/UNCOR Error Mask */
+	u32 cor_mask;		/* COR Error Mask */
+	u32 cor_status;		/* COR Error Status */
+	u32 uncor_mask;		/* UNCOR Error Mask */
+	u32 uncor_status;	/* UNCOR Error Status */
+	u32 uncor_severity;	/* UNCOR Error Severity */
+	u16 device_status;
 	struct aer_header_log_regs tlp;	/* TLP Header */
 };
 
drivers/pci/pcie/aer.c +62 -31

227 lines not on this page · raw message

include/linux/aer.h +2 -2
index f6ea2f57d808..8332731b7212 100644
@@ -52,9 +52,9 @@ static inline int pcie_aer_is_native(struct pci_dev *dev) { return 0; }
 #endif
 
 void pci_print_aer(struct pci_dev *dev, int aer_severity,
-		    struct aer_capability_regs *aer);
+		    struct aer_capability_regs *aer, u16 device_status);
 int cper_severity_to_aer(int cper_severity);
 void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
-		       int severity, struct aer_capability_regs *aer_regs);
+		       int severity, struct aer_capability_regs *aer_regs, u16 device_status);
 #endif //_AER_H_
 
include/linux/pci.h +27 -0
index dea043bc1e38..0a4f679bca90 100644
@@ -318,6 +318,33 @@ struct pci_sriov;
 struct pci_p2pdma;
 struct rcec_ea;
 
+struct pcie_capability_regs {
+	u8 pcie_cap_id;
+	u8 next_cap_ptr;
+	u16 pcie_caps;
+	u32 device_caps;
+	u16 device_control;
+	u16 device_status;
+	u32 link_caps;
+	u16 link_control;
+	u16 link_status;
+	u32 slot_caps;
+	u16 slot_control;
+	u16 slot_status;
+	u16 root_control;
+	u16 root_caps;
+	u32 root_status;
+	u32 device_caps_2;
+	u16 device_control_2;
+	u16 device_status_2;
+	u32 link_caps_2;
+	u16 link_control_2;
+	u16 link_status_2;
+	u32 slot_caps_2;
+	u16 slot_control_2;
+	u16 slot_status_2;
+};
+
 /* The pci_dev structure describes PCI devices */
 struct pci_dev {
 	struct list_head bus_list;	/* Node in per-bus list */
Signature · 1 line
2.42.0
#2 lore raw

Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info

Quoted 11 lines · Qingshun

Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.

Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

Hi,

I don't recall giving my tag for this. You are not allowed to make them up unless the person explicitly gives that Reviewed-by line to you in one of the replies. That is, please don't infer the Reviewed-by tag from just replying.

Signature · 1 line
 i.
#3 lore raw

Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info

Quoted 14 lines · Ilpo Järvinen
Quoted 11 lines · Qingshun

Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.

Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

Quoted 9 lines

Hi,

I don't recall giving my tag for this. You are not allowed to make them up unless the person explicitly gives that Reviewed-by line to you in one of the replies. That is, please don't infer the Reviewed-by tag from just replying.

Signature · 1 line
 i.

Hi,
I'm sorry about that, and apologies to anyone else whose tags were added by mistake, I'll correct it.

Best regards,
Wang, Qingshun

#4 lore raw

Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info

In reply to Wang, Qingshun

Please update subject lines of all these patches to match the capitalization of drivers/pci/ history.

Quoted 7 lines · Qingshun

Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.

Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

Can you say something here about the benefit of doing this? This text essentially describes the code in English, but we can read the code. What we need here in the commit log is the *reason* for making this change.

Bjorn

#5 lore raw

Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info

Quoted 2 lines · Bjorn Helgaas

Please update subject lines of all these patches to match the capitalization of drivers/pci/ history.

Sure, will do.

Quoted 7 lines · Qingshun
Quoted 5 lines

Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.

Quoted 7 lines
Quoted 2 lines

Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

Can you say something here about the benefit of doing this? This text essentially describes the code in English, but we can read the code. What we need here in the commit log is the *reason* for making this change.

Thanks for the advice, I will update the commit message like this:

When Advisory Non-Fatal errors are raised, both correctable and uncorrectable error statuses will be set. The current kernel code cannot store both statuses at the same time, thus failing to handle ANFE properly. In addition, to avoid clearing UEs that are not ANFE by accident, UE severity and Device Status also need to be recorded: any fatal UE cannot be ANFE, and if Fatal/Non-Fatal Error Detected is set in Device Status, do not take any assumption and let UE handler to clear UE status.

Store status and mask of both correctable and uncorrectable errors in aer_err_info. The severity of UEs and the values of the Device Status register are also recorded, which will be used to determine UEs that should be handled by the ANFE handler. Refactor the rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.

Quoted 2 lines

Bjorn

#6 lore raw

PATCH 2/4 pci/aer: Handle Advisory Non-Fatal properly

In reply to Wang, Qingshun

If we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.

Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.

---
 drivers/pci/pcie/aer.c | 58 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +57 -1
index 9311323a2391..86e7cfd71f23 100644
@@ -107,6 +107,12 @@ struct aer_stats {
 					PCI_ERR_ROOT_MULTI_COR_RCV |	\
 					PCI_ERR_ROOT_MULTI_UNCOR_RCV)
 
+#define AER_ERR_ANFE_UNC_MASK		(PCI_ERR_UNC_POISON_TLP |	\
+					PCI_ERR_UNC_COMP_TIME |		\
+					PCI_ERR_UNC_COMP_ABORT |	\
+					PCI_ERR_UNC_UNX_COMP |		\
+					PCI_ERR_UNC_UNSUP)
+
 static int pcie_aer_disable;
 static pci_ers_result_t aer_root_reset(struct pci_dev *dev);
 
@@ -612,6 +618,29 @@ const struct attribute_group aer_stats_attr_group = {
 	.is_visible = aer_stats_attrs_are_visible,
 };
 
+static int anfe_get_related_err(struct aer_err_info *info)
+{
+	/*
+	 * Take the most conservative route here. If there are
+	 * Non-Fatal/Fatal errors detected, do not assume any
+	 * bit in uncor_status is set by ANFE.
+	 */
+	if (info->device_status & (PCI_EXP_DEVSTA_NFED | PCI_EXP_DEVSTA_FED))
+		return 0;
+	/*
+	 * An UNCOR error may cause Advisory Non-Fatal error if:
+	 *	a. The severity of the error is Non-Fatal.
+	 *	b. The error is one of the following:
+	 *		1. Poisoned TLP
+	 *		2. Completion Timeout
+	 *		3. Completer Abort
+	 *		4. Unexpected Completion
+	 *		5. Unsupported Request
+	 */
+	return info->uncor_status & ~info->uncor_mask
+		& AER_ERR_ANFE_UNC_MASK & ~info->severity;
+}
+
 static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 				   struct aer_err_info *info)
 {
@@ -678,6 +707,7 @@ static void __aer_print_error(struct pci_dev *dev,
 			      struct aer_err_info *info)
 {
 	unsigned long status;
+	unsigned long anfe_status;
 	const char **strings;
 	const char *level, *errmsg;
 	int i;
@@ -700,6 +730,21 @@ static void __aer_print_error(struct pci_dev *dev,
 		pci_printk(level, dev, "   [%2d] %-22s%s\n", i, errmsg,
 				info->first_error == i ? " (First)" : "");
 	}
+
+	if (info->severity == AER_CORRECTABLE && (status & PCI_ERR_COR_ADV_NFAT)) {
+		anfe_status = anfe_get_related_err(info);
+		if (anfe_status) {
+			pci_printk(level, dev, "Uncorrectable errors that may cause Advisory Non-Fatal:");
+			for_each_set_bit(i, &anfe_status, 32) {
+				errmsg = aer_uncorrectable_error_string[i];
+				if (!errmsg)
+					errmsg = "Unknown Error Bit";
+
+				pci_printk(level, dev, "   [%2d] %-22s\n", i, errmsg);
+			}
+		}
+	}
+
 	pci_dev_aer_stats_incr(dev, info);
 }
 
@@ -1092,6 +1137,14 @@ static inline void cxl_rch_handle_error(struct pci_dev *dev,
 					struct aer_err_info *info) { }
 #endif
 
+static void handle_advisory_nonfatal(struct pci_dev *dev, struct aer_err_info *info)
+{
+	int aer = dev->aer_cap;
+
+	pci_write_config_dword(dev, aer + PCI_ERR_UNCOR_STATUS,
+			       anfe_get_related_err(info));
+}
+
 /**
  * pci_aer_handle_error - handle logging error into an event log
  * @dev: pointer to pci_dev data structure of error source device
@@ -1108,9 +1161,12 @@ static void pci_aer_handle_error(struct pci_dev *dev, struct aer_err_info *info)
 		 * Correctable error does not need software intervention.
 		 * No need to go through error recovery process.
 		 */
-		if (aer)
+		if (aer) {
 			pci_write_config_dword(dev, aer + PCI_ERR_COR_STATUS,
 					info->cor_status);
+			if (info->cor_status & PCI_ERR_COR_ADV_NFAT)
+				handle_advisory_nonfatal(dev, info);
+		}
 		if (pcie_aer_is_native(dev)) {
 			struct pci_driver *pdrv = dev->driver;
 
Signature · 1 line
2.42.0
#7 lore raw
Quoted 16 lines · Qingshun

If we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.

Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.

drivers/pci/pcie/aer.c +8 -0
+static int anfe_get_related_err(struct aer_err_info *info)
+{
+	/*
+	 * Take the most conservative route here. If there are
+	 * Non-Fatal/Fatal errors detected, do not assume any
+	 * bit in uncor_status is set by ANFE.
+	 */
+	if (info->device_status & (PCI_EXP_DEVSTA_NFED | PCI_EXP_DEVSTA_FED))

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 638

Quoted 10 lines
drivers/pci/pcie/aer.c +10 -0
+		return 0;
+	/*
+	 * An UNCOR error may cause Advisory Non-Fatal error if:
+	 *	a. The severity of the error is Non-Fatal.
+	 *	b. The error is one of the following:
+	 *		1. Poisoned TLP
+	 *		2. Completion Timeout
+	 *		3. Completer Abort
+	 *		4. Unexpected Completion
+	 *		5. Unsupported Request

This could benefit from a reference to the spec that outlines these conditions.

Bjorn

#8 lore raw
Quoted 22 lines · Bjorn Helgaas
Quoted 21 lines · Qingshun

If we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.

Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.

drivers/pci/pcie/aer.c +13 -0
+static int anfe_get_related_err(struct aer_err_info *info)
+{
+	/*
+	 * Take the most conservative route here. If there are
+	 * Non-Fatal/Fatal errors detected, do not assume any
+	 * bit in uncor_status is set by ANFE.
+	 */
+	if (info->device_status & (PCI_EXP_DEVSTA_NFED | PCI_EXP_DEVSTA_FED))
+		return 0;
+	/*
+	 * An UNCOR error may cause Advisory Non-Fatal error if:
+	 *	a. The severity of the error is Non-Fatal.
+	 *	b. The error is one of the following:
Quoted 8 lines

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 638

Quoted 5 lines
drivers/pci/pcie/aer.c +5 -0
+	 *		1. Poisoned TLP
+	 *		2. Completion Timeout
+	 *		3. Completer Abort
+	 *		4. Unexpected Completion
+	 *		5. Unsupported Request

This could benefit from a reference to the spec that outlines these conditions.

Thanks for suggestion. Will add a reference to latest spec.

Quoted 2 lines

Bjorn

Best regards
Wang, Qingshun

#9 lore raw

PATCH 3/4 pci/aer: Fetch information for FTrace

In reply to Wang, Qingshun

Fetch and store the data of 3 more registers: Link Status, Device Control 2 and Advanced Error Capabilities and Control. This data will be traced in FTrace in the following patch.

---
 drivers/acpi/apei/ghes.c |  8 +++++++-
 drivers/cxl/core/pci.c   | 11 ++++++++++-
 drivers/pci/pci.h        |  4 ++++
 drivers/pci/pcie/aer.c   | 26 ++++++++++++++++++++------
 include/linux/aer.h      |  6 ++++--
 5 files changed, 45 insertions(+), 10 deletions(-)
drivers/acpi/apei/ghes.c +7 -1
index 0e7ae84a5d3f..a15bb93c6904 100644
@@ -577,7 +577,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
 	if (pcie_err->validation_bits & CPER_PCIE_VALID_DEVICE_ID &&
 	    pcie_err->validation_bits & CPER_PCIE_VALID_AER_INFO) {
 		struct pcie_capability_regs *pcie_caps;
+		u16 device_control_2 = 0;
 		u16 device_status = 0;
+		u16 link_status = 0;
 		unsigned int devfn;
 		int aer_severity;
 		u8 *aer_info;
@@ -602,7 +604,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
 
 		if (pcie_err->validation_bits & CPER_PCIE_VALID_CAPABILITY) {
 			pcie_caps = (struct pcie_capability_regs *) &pcie_err->capability;
+			device_control_2 = pcie_caps->device_control_2;
 			device_status = pcie_caps->device_status;
+			link_status = pcie_caps->link_status;
 		}
 
 		aer_recover_queue(pcie_err->device_id.segment,
@@ -610,7 +614,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
 				  devfn, aer_severity,
 				  (struct aer_capability_regs *)
 				  aer_info,
-				  device_status);
+				  device_status,
+				  link_status,
+				  device_control_2);
 	}
 #endif
 }
drivers/cxl/core/pci.c +10 -1
index 2d099c6463b4..c1ea77057341 100644
@@ -902,7 +902,9 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
 	struct aer_capability_regs aer_regs;
 	struct cxl_dport *dport;
 	struct cxl_port *port;
+	u16 device_control_2;
 	u16 device_status;
+	u16 link_status;
 	int severity;
 
 	port = cxl_pci_find_port(pdev, &dport);
@@ -917,10 +919,17 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
 	if (!cxl_rch_get_aer_severity(&aer_regs, &severity))
 		return;
 
+	if (pcie_capability_read_word(pdev, PCI_EXP_DEVCTL2, &device_control_2))
+		return;
+
 	if (pcie_capability_read_word(pdev, PCI_EXP_DEVSTA, &device_status))
 		return;
 
-	pci_print_aer(pdev, severity, &aer_regs, device_status);
+	if (pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &link_status))
+		return;
+
+	pci_print_aer(pdev, severity, &aer_regs, device_status,
+		      link_status, device_control_2);
 
 	if (severity == AER_CORRECTABLE)
 		cxl_handle_rdport_cor_ras(cxlds, dport);
drivers/pci/pci.h +4 -0
index 1e28adfafc06..333f2194a360 100644
@@ -410,7 +410,11 @@ struct aer_err_info {
 	u32 uncor_mask;		/* UNCOR Error Mask */
 	u32 uncor_status;	/* UNCOR Error Status */
 	u32 uncor_severity;	/* UNCOR Error Severity */
+
+	u16 link_status;
+	u32 aer_cap_ctrl;	/* AER Capabilities and Control */
 	u16 device_status;
+	u16 device_control_2;
 	struct aer_header_log_regs tlp;	/* TLP Header */
 };
 
drivers/pci/pcie/aer.c +20 -6
index 86e7cfd71f23..b5d4ea8e5f8d 100644
@@ -822,7 +822,8 @@ EXPORT_SYMBOL_GPL(cper_severity_to_aer);
 #endif
 
 void pci_print_aer(struct pci_dev *dev, int aer_severity,
-		   struct aer_capability_regs *aer, u16 device_status)
+		   struct aer_capability_regs *aer, u16 device_status,
+		   u16 link_status, u16 device_control_2)
 {
 	int layer, agent, tlp_header_valid = 0;
 	u32 status, mask;
@@ -847,7 +848,10 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	info.uncor_status = aer->uncor_status;
 	info.uncor_severity = aer->uncor_severity;
 	info.uncor_mask = aer->uncor_mask;
+	info.link_status = link_status;
+	info.aer_cap_ctrl = aer->cap_control;
 	info.device_status = device_status;
+	info.device_control_2 = device_control_2;
 	info.first_error = PCI_ERR_CAP_FEP(aer->cap_control);
 
 	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
@@ -1197,7 +1201,9 @@ struct aer_recover_entry {
 	u8	devfn;
 	u16	domain;
 	int	severity;
+	u16	link_status;
 	u16	device_status;
+	u16	device_control_2;
 	struct aer_capability_regs *regs;
 };
 
@@ -1218,7 +1224,8 @@ static void aer_recover_work_func(struct work_struct *work)
 			       PCI_SLOT(entry.devfn), PCI_FUNC(entry.devfn));
 			continue;
 		}
-		pci_print_aer(pdev, entry.severity, entry.regs, entry.device_status);
+		pci_print_aer(pdev, entry.severity, entry.regs, entry.device_status,
+			      entry.link_status, entry.device_control_2);
 		/*
 		 * Memory for aer_capability_regs(entry.regs) is being allocated from the
 		 * ghes_estatus_pool to protect it from overwriting when multiple sections
@@ -1247,7 +1254,8 @@ static DEFINE_SPINLOCK(aer_recover_ring_lock);
 static DECLARE_WORK(aer_recover_work, aer_recover_work_func);
 
 void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
-		       int severity, struct aer_capability_regs *aer_regs, u16 device_status)
+		       int severity, struct aer_capability_regs *aer_regs, u16 device_status,
+		       u16 link_status, u16 device_control_2)
 {
 	struct aer_recover_entry entry = {
 		.bus		= bus,
@@ -1255,7 +1263,9 @@ void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
 		.domain		= domain,
 		.severity	= severity,
 		.regs		= aer_regs,
+		.link_status	= link_status,
 		.device_status	= device_status,
+		.device_control_2 = device_control_2,
 	};
 
 	if (kfifo_in_spinlocked(&aer_recover_ring, &entry, 1,
@@ -1281,7 +1291,6 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)
 {
 	int type = pci_pcie_type(dev);
 	int aer = dev->aer_cap;
-	int temp;
 
 	/* Must reset in this function */
 	info->cor_status = 0;
@@ -1309,8 +1318,14 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)
 			&info->uncor_severity);
 		pci_read_config_dword(dev, aer + PCI_ERR_UNCOR_MASK,
 			&info->uncor_mask);
+		pci_read_config_dword(dev, aer + PCI_ERR_CAP,
+			&info->aer_cap_ctrl);
+		pcie_capability_read_word(dev, PCI_EXP_LNKSTA,
+			&info->link_status);
 		pcie_capability_read_word(dev, PCI_EXP_DEVSTA,
 			&info->device_status);
+		pcie_capability_read_word(dev, PCI_EXP_DEVCTL2,
+			&info->device_control_2);
 	} else {
 		return 1;
 	}
@@ -1323,8 +1338,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)
 			return 0;
 
 		/* Get First Error Pointer */
-		pci_read_config_dword(dev, aer + PCI_ERR_CAP, &temp);
-		info->first_error = PCI_ERR_CAP_FEP(temp);
+		info->first_error = PCI_ERR_CAP_FEP(info->aer_cap_ctrl);
 
 		if (info->uncor_status & AER_LOG_TLP_MASKS) {
 			info->tlp_header_valid = 1;
include/linux/aer.h +4 -2
index 8332731b7212..333b72c19c60 100644
@@ -52,9 +52,11 @@ static inline int pcie_aer_is_native(struct pci_dev *dev) { return 0; }
 #endif
 
 void pci_print_aer(struct pci_dev *dev, int aer_severity,
-		    struct aer_capability_regs *aer, u16 device_status);
+		    struct aer_capability_regs *aer, u16 device_status,
+		    u16 link_status, u16 device_control_2);
 int cper_severity_to_aer(int cper_severity);
 void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
-		       int severity, struct aer_capability_regs *aer_regs, u16 device_status);
+		       int severity, struct aer_capability_regs *aer_regs, u16 device_status,
+		       u16 link_status, u16 device_control_2);
 #endif //_AER_H_
 
Signature · 1 line
2.42.0
#10 lore raw

PATCH 4/4 ras: Trace more information in aer_event

In reply to Wang, Qingshun

Add following fields in aer_event to better understand Advisory Non-Fatal and other errors for external observation:

- cor_status (Correctable Error Status)
- cor_mask (Correctable Error Mask)
- uncor_status (Uncorrectable Error Status)
- uncor_severity (Uncorrectable Error Severity)
- uncor_mask (Uncorrectable Error Mask)
- aer_cap_ctrl (AER Capabilities and Control)
- link_status (Link Status)
- device_status (Device Status)
- device_control_2 (Device Control 2)

In addition to the raw register value, value of following fields are extracted and logged for better observability:

- "First Error Pointer" and "Completion Timeout Prefix/Header Log Capable" from "AER Capabilities and Control"
- "Completion Timeout Value" and "Completion Timeout Disable" from "Device Control 2"

---
 drivers/pci/pcie/aer.c        | 17 +++++++++++--
 include/ras/ras_event.h       | 48 ++++++++++++++++++++++++++++++++---
 include/uapi/linux/pci_regs.h |  1 +
 3 files changed, 60 insertions(+), 6 deletions(-)
drivers/pci/pcie/aer.c +15 -2
index b5d4ea8e5f8d..52ca9bf26e84 100644
@@ -754,6 +754,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 	int layer, agent;
 	int id = pci_dev_id(dev);
 	const char *level;
+	struct aer_capability_regs aer_caps;
 
 	if (info->severity == AER_CORRECTABLE) {
 		status = info->cor_status;
@@ -790,8 +791,18 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 	if (info->id && info->error_dev_num > 1 && info->id == id)
 		pci_err(dev, "  Error of this Agent is reported first\n");
 
+	aer_caps = (struct aer_capability_regs) {
+	  .cor_status = info->cor_status,
+	  .cor_mask = info->cor_mask,
+	  .uncor_status = info->uncor_status,
+	  .uncor_severity = info->uncor_severity,
+	  .uncor_mask = info->uncor_mask,
+	  .cap_control = info->aer_cap_ctrl
+	};
 	trace_aer_event(dev_name(&dev->dev), (status & ~mask),
-			info->severity, info->tlp_header_valid, &info->tlp);
+			info->severity, info->tlp_header_valid, &info->tlp,
+			&aer_caps, info->link_status,
+			info->device_status, info->device_control_2);
 }
 
 static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
@@ -867,7 +878,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 		__print_tlp_header(dev, &aer->header_log);
 
 	trace_aer_event(dev_name(&dev->dev), (status & ~mask),
-			aer_severity, tlp_header_valid, &aer->header_log);
+			aer_severity, tlp_header_valid, &aer->header_log,
+			aer, info.link_status,
+			info.device_status, info.device_control_2);
 }
 EXPORT_SYMBOL_NS_GPL(pci_print_aer, CXL);
 
include/ras/ras_event.h +44 -4

79 lines not on this page · raw message

include/uapi/linux/pci_regs.h +1 -0
index a39193213ff2..54160ed2a8c9 100644
@@ -787,6 +787,7 @@
 #define  PCI_ERR_CAP_ECRC_GENE	0x00000040	/* ECRC Generation Enable */
 #define  PCI_ERR_CAP_ECRC_CHKC	0x00000080	/* ECRC Check Capable */
 #define  PCI_ERR_CAP_ECRC_CHKE	0x00000100	/* ECRC Check Enable */
+#define  PCI_ERR_CAP_CTO_LOGC	0x00001000	/* Completion Timeout Prefix/Header Log Capable */
 #define PCI_ERR_HEADER_LOG	0x1c	/* Header Log Register (16 bytes) */
 #define PCI_ERR_ROOT_COMMAND	0x2c	/* Root Error Command */
 #define  PCI_ERR_ROOT_CMD_COR_EN	0x00000001 /* Correctable Err Reporting Enable */
Signature · 1 line
2.42.0
#12 lore raw
In reply to Wang, Qingshun
Quoted 37 lines · Qingshun

According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal ERR_COR instead of ERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.

For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:

[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           

If the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:

[  170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  170.552420] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00041000/00180020
[  170.561904] pcieport 0000:b7:02.0:    [12] TLP                    (First)
[  170.569656] pcieport 0000:b7:02.0:    [18] MalfTLP       

To handle this case properly, this patch set adds additional fields in aer_err_info to track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data to aer_event tracepoint, which would help to better understand ANFE and other errors for external observation.

In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:

Quoted 6 lines
[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           
[  148.459868] pcieport 0000:b7:02.0:   Uncorrectable errors that may cause Advisory Non-Fatal:
[  148.459868] pcieport 0000:b7:02.0:    [18] TLP

Thanks for the overview here. It would be good to put some of these details in the commit logs of the patches that implement this, because this cover letter is not preserved when the series is merged.

If/when you do, remove the timestamps because they're not relevant and are merely distracting. Indent quoted material a couple spaces.

Update the citations to a current spec revision (PCIe r6.0, or maybe PCIe r6.1). The section numbers are probably the same, but there's no point in citing a revision that's 6.5 years old when newer ones are available.

Bjorn

#13 lore raw
Quoted 45 lines · Bjorn Helgaas
Quoted 37 lines · Qingshun

According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal ERR_COR instead of ERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.

For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:

[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           

If the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:

[  170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  170.552420] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00041000/00180020
[  170.561904] pcieport 0000:b7:02.0:    [12] TLP                    (First)
[  170.569656] pcieport 0000:b7:02.0:    [18] MalfTLP       

To handle this case properly, this patch set adds additional fields in aer_err_info to track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data to aer_event tracepoint, which would help to better understand ANFE and other errors for external observation.

In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:

Quoted 6 lines
[  148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0
[  148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
[  148.459863] pcieport 0000:b7:02.0:   device [8086:0db0] error status/mask=00002000/00000000
[  148.459868] pcieport 0000:b7:02.0:    [13] NonFatalErr           
[  148.459868] pcieport 0000:b7:02.0:   Uncorrectable errors that may cause Advisory Non-Fatal:
[  148.459868] pcieport 0000:b7:02.0:    [18] TLP
Quoted 3 lines

Thanks for the overview here. It would be good to put some of these details in the commit logs of the patches that implement this, because this cover letter is not preserved when the series is merged.

Thanks for your advice, will put some of these details in commit logs, mainly in PATCH 2.

Quoted 3 lines

If/when you do, remove the timestamps because they're not relevant and are merely distracting. Indent quoted material a couple spaces.

Agreed. Thanks.

Quoted 5 lines

Update the citations to a current spec revision (PCIe r6.0, or maybe PCIe r6.1). The section numbers are probably the same, but there's no point in citing a revision that's 6.5 years old when newer ones are available.

Makes sense, thanks!

Quoted 2 lines

Bjorn