pci/aer: Handle Advisory Non-Fatal properly
Subsystems (6)
cover pci/aer: Handle Advisory Non-Fatal properly
According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal ERR_COR instead of ERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.
For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErr
If the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:
[ 170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID) [ 170.552420] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00041000/00180020 [ 170.561904] pcieport 0000:b7:02.0: [12] TLP (First) [ 170.569656] pcieport 0000:b7:02.0: [18] MalfTLP
To handle this case properly, this patch set adds additional fields in aer_err_info to track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data to aer_event tracepoint, which would help to better understand ANFE and other errors for external observation.
In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErr [ 148.459868] pcieport 0000:b7:02.0: Uncorrectable errors that may cause Advisory Non-Fatal: [ 148.459868] pcieport 0000:b7:02.0: [18] TLP
Wang, Qingshun (4):
pci/aer: Store more information in aer_err_info
pci/aer: Handle Advisory Non-Fatal properly
pci/aer: Fetch information for FTrace
ras: Trace more information in aer_event
drivers/acpi/apei/ghes.c | 16 ++- drivers/cxl/core/pci.c | 15 ++- drivers/pci/pci.h | 12 ++- drivers/pci/pcie/aer.c | 188 +++++++++++++++++++++++++++------- include/linux/aer.h | 6 +- include/linux/pci.h | 27 +++++ include/ras/ras_event.h | 48 ++++++++- include/uapi/linux/pci_regs.h | 1 + 8 files changed, 266 insertions(+), 47 deletions(-)
base-commit: 610a9b8f49fbcf1100716370d3b5f6f884a2835a
Signature · 1 line
2.42.0
PATCH 1/4 pci/aer: Store more information in aer_err_info
Store status and mask of both correctable and uncorrectable errors in aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.
Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
--- drivers/acpi/apei/ghes.c | 10 ++++- drivers/cxl/core/pci.c | 6 ++- drivers/pci/pci.h | 8 +++- drivers/pci/pcie/aer.c | 93 ++++++++++++++++++++++++++-------------- include/linux/aer.h | 4 +- include/linux/pci.h | 27 ++++++++++++ 6 files changed, 111 insertions(+), 37 deletions(-)
drivers/acpi/apei/ghes.c +9 -1
@@ -576,6 +576,8 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)if (pcie_err->validation_bits &CPER_PCIE_VALID_DEVICE_ID && pcie_err->validation_bits &CPER_PCIE_VALID_AER_INFO ) { +struct pcie_capability_regs *pcie_caps; +u16 device_status =0 ;unsigned int devfn;int aer_severity;u8 *aer_info;
@@ -598,11 +600,17 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)return ;memcpy (aer_info, pcie_err->aer_info,sizeof (struct aer_capability_regs )); +if (pcie_err->validation_bits &CPER_PCIE_VALID_CAPABILITY ) { + pcie_caps = (struct pcie_capability_regs *) &pcie_err->capability; + device_status = pcie_caps->device_status; + } +aer_recover_queue (pcie_err->device_id.segment, pcie_err->device_id.bus, devfn, aer_severity, (struct aer_capability_regs *) - aer_info); + aer_info, + device_status); }#endif }
drivers/cxl/core/pci.c +5 -1
@@ -902,6 +902,7 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)struct aer_capability_regs aer_regs;struct cxl_dport *dport;struct cxl_port *port; +u16 device_status;int severity; port =cxl_pci_find_port (pdev, &dport);
@@ -916,7 +917,10 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)if (!cxl_rch_get_aer_severity (&aer_regs, &severity))return ; -pci_print_aer (pdev, severity, &aer_regs); +if (pcie_capability_read_word (pdev,PCI_EXP_DEVSTA , &device_status)) +return ; + +pci_print_aer (pdev, severity, &aer_regs, device_status);if (severity ==AER_CORRECTABLE )cxl_handle_rdport_cor_ras (cxlds, dport);
drivers/pci/pci.h +6 -2
@@ -405,8 +405,12 @@ struct aer_err_info { unsigned int __pad2:2; unsigned int tlp_header_valid:1; -unsigned int status;/* COR/UNCOR Error Status */ -unsigned int mask;/* COR/UNCOR Error Mask */ +u32 cor_mask;/* COR Error Mask */ +u32 cor_status;/* COR Error Status */ +u32 uncor_mask;/* UNCOR Error Mask */ +u32 uncor_status;/* UNCOR Error Status */ +u32 uncor_severity;/* UNCOR Error Severity */ +u16 device_status;struct aer_header_log_regs tlp;/* TLP Header */ };
drivers/pci/pcie/aer.c +62 -31
227 lines not on this page · raw message
include/linux/aer.h +2 -2
@@ -52,9 +52,9 @@ static inline int pcie_aer_is_native(struct pci_dev *dev) { return 0; }#endif void pci_print_aer (struct pci_dev *dev,int aer_severity, -struct aer_capability_regs *aer); +struct aer_capability_regs *aer,u16 device_status);int cper_severity_to_aer (int cper_severity);void aer_recover_queue (int domain,unsigned int bus,unsigned int devfn, -int severity,struct aer_capability_regs *aer_regs); +int severity,struct aer_capability_regs *aer_regs,u16 device_status);#endif //_AER_H_
include/linux/pci.h +27 -0
@@ -318,6 +318,33 @@ struct pci_sriov;struct pci_p2pdma ;struct rcec_ea ; +struct pcie_capability_regs { +u8 pcie_cap_id; +u8 next_cap_ptr; +u16 pcie_caps; +u32 device_caps; +u16 device_control; +u16 device_status; +u32 link_caps; +u16 link_control; +u16 link_status; +u32 slot_caps; +u16 slot_control; +u16 slot_status; +u16 root_control; +u16 root_caps; +u32 root_status; +u32 device_caps_2; +u16 device_control_2; +u16 device_status_2; +u32 link_caps_2; +u16 link_control_2; +u16 link_status_2; +u32 slot_caps_2; +u16 slot_control_2; +u16 slot_status_2; +}; +/* The pci_dev structure describes PCI devices */ struct pci_dev {struct list_head bus_list;/* Node in per-bus list */
Signature · 1 line
2.42.0
Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info
Quoted 11 lines · Qingshun
Store status and mask of both correctable and uncorrectable errors in
aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
Reviewed-by "Andy Shevchenko" <[email protected]>Reviewed-by "Luck, Tony" <[email protected]>Reviewed-by "Ilpo Järvinen" <[email protected]>
Hi,
I don't recall giving my tag for this. You are not allowed to make them up unless the person explicitly gives that Reviewed-by line to you in one of the replies. That is, please don't infer the Reviewed-by tag from just replying.
Signature · 1 line
i.
Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info
Quoted 14 lines · Ilpo Järvinen
Quoted 11 lines · Qingshun
Store status and mask of both correctable and uncorrectable errors in
aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
Reviewed-by "Andy Shevchenko" <[email protected]>Reviewed-by "Luck, Tony" <[email protected]>Reviewed-by "Ilpo Järvinen" <[email protected]>
Quoted 9 lines
Hi,
I don't recall giving my tag for this. You are not allowed to make them up unless the person explicitly gives that Reviewed-by line to you in one of the replies. That is, please don't infer the Reviewed-by tag from just replying.
Signature · 1 line
i.
Hi,
I'm sorry about that, and apologies to anyone else whose tags were added by mistake, I'll correct it.
Best regards,
Wang, Qingshun
Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info
In reply to Wang, QingshunPlease update subject lines of all these patches to match the capitalization of drivers/pci/ history.
Quoted 7 lines · Qingshun
Store status and mask of both correctable and uncorrectable errors in
aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
Can you say something here about the benefit of doing this? This text essentially describes the code in English, but we can read the code. What we need here in the commit log is the *reason* for making this change.
Bjorn
Re: [PATCH 1/4] pci/aer: Store more information in aer_err_info
Quoted 2 lines · Bjorn Helgaas
Please update subject lines of all these patches to match the capitalization of
drivers/pci/history.
Sure, will do.
Quoted 7 lines · Qingshun
Quoted 5 lines
Store status and mask of both correctable and uncorrectable errors in
aer_err_info. Severity of uncorrectable errors and the values of Device Status register is also recorded in order to filter out errors that cannot cause Advisory Non-Fatal error.
Quoted 7 lines
Quoted 2 lines
Refactor rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
Can you say something here about the benefit of doing this? This text essentially describes the code in English, but we can read the code. What we need here in the commit log is the *reason* for making this change.
Thanks for the advice, I will update the commit message like this:
When Advisory Non-Fatal errors are raised, both correctable and uncorrectable error statuses will be set. The current kernel code cannot store both statuses at the same time, thus failing to handle ANFE properly. In addition, to avoid clearing UEs that are not ANFE by accident, UE severity and Device Status also need to be recorded: any fatal UE cannot be ANFE, and if Fatal/Non-Fatal Error Detected is set in Device Status, do not take any assumption and let UE handler to clear UE status.
Store status and mask of both correctable and uncorrectable errors in aer_err_info. The severity of UEs and the values of the Device Status register are also recorded, which will be used to determine UEs that should be handled by the ANFE handler. Refactor the rest of the code to use cor/uncor_status and cor/uncor_mask fields instead of status and mask fields.
Quoted 2 lines
Bjorn
PATCH 2/4 pci/aer: Handle Advisory Non-Fatal properly
In reply to Wang, QingshunIf we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.
Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.
--- drivers/pci/pcie/aer.c | 58 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +57 -1
@@ -107,6 +107,12 @@ struct aer_stats { PCI_ERR_ROOT_MULTI_COR_RCV | \ PCI_ERR_ROOT_MULTI_UNCOR_RCV) +#define AER_ERR_ANFE_UNC_MASK (PCI_ERR_UNC_POISON_TLP | \ + PCI_ERR_UNC_COMP_TIME | \ + PCI_ERR_UNC_COMP_ABORT | \ + PCI_ERR_UNC_UNX_COMP | \ + PCI_ERR_UNC_UNSUP) +static int pcie_aer_disable;static pci_ers_result_t aer_root_reset (struct pci_dev *dev);
@@ -612,6 +618,29 @@ const struct attribute_group aer_stats_attr_group = { .is_visible = aer_stats_attrs_are_visible, }; +static int anfe_get_related_err (struct aer_err_info *info) +{ +/* +* Take the most conservative route here. If there are +* Non-Fatal/Fatal errors detected, do not assume any +* bit in uncor_status is set by ANFE. +*/ +if (info->device_status & (PCI_EXP_DEVSTA_NFED |PCI_EXP_DEVSTA_FED )) +return 0 ; +/* +* An UNCOR error may cause Advisory Non-Fatal error if: +* a. The severity of the error is Non-Fatal. +* b. The error is one of the following: +* 1. Poisoned TLP +* 2. Completion Timeout +* 3. Completer Abort +* 4. Unexpected Completion +* 5. Unsupported Request +*/ +return info->uncor_status & ~info->uncor_mask + &AER_ERR_ANFE_UNC_MASK & ~info->severity; +} +static void pci_dev_aer_stats_incr (struct pci_dev *pdev,struct aer_err_info *info) {
@@ -678,6 +707,7 @@ static void __aer_print_error(struct pci_dev *dev, struct aer_err_info *info) {unsigned long status; +unsigned long anfe_status;const char **strings;const char *level, *errmsg;int i;
@@ -700,6 +730,21 @@ static void __aer_print_error(struct pci_dev *dev,pci_printk (level, dev," [%2d] %-22s%s\n" , i, errmsg, info->first_error == i ?" (First)" :"" ); } + +if (info->severity ==AER_CORRECTABLE && (status &PCI_ERR_COR_ADV_NFAT )) { + anfe_status =anfe_get_related_err (info); +if (anfe_status) { +pci_printk (level, dev,"Uncorrectable errors that may cause Advisory Non-Fatal:" ); +for_each_set_bit (i, &anfe_status,32 ) { + errmsg = aer_uncorrectable_error_string[i]; +if (!errmsg) + errmsg ="Unknown Error Bit" ; + +pci_printk (level, dev," [%2d] %-22s\n" , i, errmsg); + } + } + } +pci_dev_aer_stats_incr (dev, info); }
@@ -1092,6 +1137,14 @@ static inline void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info) { }#endif +static void handle_advisory_nonfatal (struct pci_dev *dev,struct aer_err_info *info) +{ +int aer = dev->aer_cap; + +pci_write_config_dword (dev, aer +PCI_ERR_UNCOR_STATUS , +anfe_get_related_err (info)); +} +/** * pci_aer_handle_error - handle logging error into an event log * @dev: pointer to pci_dev data structure of error source device
@@ -1108,9 +1161,12 @@ static void pci_aer_handle_error(struct pci_dev *dev, struct aer_err_info *info)* Correctable error does not need software intervention. * No need to go through error recovery process. */ -if (aer) +if (aer) {pci_write_config_dword (dev, aer +PCI_ERR_COR_STATUS , info->cor_status); +if (info->cor_status &PCI_ERR_COR_ADV_NFAT ) +handle_advisory_nonfatal (dev, info); + }if (pcie_aer_is_native (dev)) {struct pci_driver *pdrv = dev->driver;
Signature · 1 line
2.42.0
Quoted 16 lines · Qingshun
If we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.
Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.
drivers/pci/pcie/aer.c +8
-0+static int anfe_get_related_err (struct aer_err_info *info) +{ +/* +* Take the most conservative route here. If there are +* Non-Fatal/Fatal errors detected, do not assume any +* bit in uncor_status is set by ANFE. +*/ +if (info->device_status & (PCI_EXP_DEVSTA_NFED |PCI_EXP_DEVSTA_FED ))
drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 638
Quoted 10 lines
drivers/pci/pcie/aer.c +10
-0+return 0 ; +/* +* An UNCOR error may cause Advisory Non-Fatal error if: +* a. The severity of the error is Non-Fatal. +* b. The error is one of the following: +* 1. Poisoned TLP +* 2. Completion Timeout +* 3. Completer Abort +* 4. Unexpected Completion +* 5. Unsupported Request
This could benefit from a reference to the spec that outlines these conditions.
Bjorn
Quoted 22 lines · Bjorn Helgaas
Quoted 21 lines · Qingshun
If we are processing an Advisory Non-Fatal Error, first check the Device Status. If any of Fatal/Non-Fatal Error Detected bits is set, leave it to uncorrectable error handler to clear the UE status bit, which should be executed right after the CE handler in this case.
Otherwise, filter out uncorrectable errors that is not possible to trigger an Advisory Non-Fatal Error, then clear all the rest status bits.
drivers/pci/pcie/aer.c +13
-0+static int anfe_get_related_err (struct aer_err_info *info) +{ +/* +* Take the most conservative route here. If there are +* Non-Fatal/Fatal errors detected, do not assume any +* bit in uncor_status is set by ANFE. +*/ +if (info->device_status & (PCI_EXP_DEVSTA_NFED |PCI_EXP_DEVSTA_FED )) +return 0 ; +/* +* An UNCOR error may cause Advisory Non-Fatal error if: +* a. The severity of the error is Non-Fatal. +* b. The error is one of the following:
Quoted 8 lines
drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 638
Quoted 5 lines
drivers/pci/pcie/aer.c +5
-0+* 1. Poisoned TLP +* 2. Completion Timeout +* 3. Completer Abort +* 4. Unexpected Completion +* 5. Unsupported Request This could benefit from a reference to the spec that outlines these conditions.
Thanks for suggestion. Will add a reference to latest spec.
Quoted 2 lines
Bjorn
Best regards
Wang, Qingshun
PATCH 3/4 pci/aer: Fetch information for FTrace
In reply to Wang, QingshunFetch and store the data of 3 more registers: Link Status, Device Control 2 and Advanced Error Capabilities and Control. This data will be traced in FTrace in the following patch.
--- drivers/acpi/apei/ghes.c | 8 +++++++- drivers/cxl/core/pci.c | 11 ++++++++++- drivers/pci/pci.h | 4 ++++ drivers/pci/pcie/aer.c | 26 ++++++++++++++++++++------ include/linux/aer.h | 6 ++++-- 5 files changed, 45 insertions(+), 10 deletions(-)
drivers/acpi/apei/ghes.c +7 -1
@@ -577,7 +577,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)if (pcie_err->validation_bits &CPER_PCIE_VALID_DEVICE_ID && pcie_err->validation_bits &CPER_PCIE_VALID_AER_INFO ) {struct pcie_capability_regs *pcie_caps; +u16 device_control_2 =0 ;u16 device_status =0 ; +u16 link_status =0 ;unsigned int devfn;int aer_severity;u8 *aer_info;
@@ -602,7 +604,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)if (pcie_err->validation_bits &CPER_PCIE_VALID_CAPABILITY ) { pcie_caps = (struct pcie_capability_regs *) &pcie_err->capability; + device_control_2 = pcie_caps->device_control_2; device_status = pcie_caps->device_status; + link_status = pcie_caps->link_status; } aer_recover_queue(pcie_err->device_id.segment,
@@ -610,7 +614,9 @@ static void ghes_handle_aer(struct acpi_hest_generic_data *gdata) devfn, aer_severity, (struct aer_capability_regs *) aer_info, - device_status); + device_status, + link_status, + device_control_2); }#endif }
drivers/cxl/core/pci.c +10 -1
@@ -902,7 +902,9 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)struct aer_capability_regs aer_regs;struct cxl_dport *dport;struct cxl_port *port; +u16 device_control_2;u16 device_status; +u16 link_status;int severity; port =cxl_pci_find_port (pdev, &dport);
@@ -917,10 +919,17 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)if (!cxl_rch_get_aer_severity (&aer_regs, &severity))return ; +if (pcie_capability_read_word (pdev,PCI_EXP_DEVCTL2 , &device_control_2)) +return ; +if (pcie_capability_read_word (pdev,PCI_EXP_DEVSTA , &device_status))return ; -pci_print_aer (pdev, severity, &aer_regs, device_status); +if (pcie_capability_read_word (pdev,PCI_EXP_LNKSTA , &link_status)) +return ; + +pci_print_aer (pdev, severity, &aer_regs, device_status, + link_status, device_control_2);if (severity ==AER_CORRECTABLE )cxl_handle_rdport_cor_ras (cxlds, dport);
drivers/pci/pci.h +4 -0
@@ -410,7 +410,11 @@ struct aer_err_info {u32 uncor_mask;/* UNCOR Error Mask */ u32 uncor_status;/* UNCOR Error Status */ u32 uncor_severity;/* UNCOR Error Severity */ + +u16 link_status; +u32 aer_cap_ctrl;/* AER Capabilities and Control */ u16 device_status; +u16 device_control_2;struct aer_header_log_regs tlp;/* TLP Header */ };
drivers/pci/pcie/aer.c +20 -6
@@ -822,7 +822,8 @@ EXPORT_SYMBOL_GPL(cper_severity_to_aer);#endif void pci_print_aer (struct pci_dev *dev,int aer_severity, -struct aer_capability_regs *aer,u16 device_status) +struct aer_capability_regs *aer,u16 device_status, +u16 link_status,u16 device_control_2) {int layer, agent, tlp_header_valid =0 ;u32 status, mask;
@@ -847,7 +848,10 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity, info.uncor_status = aer->uncor_status; info.uncor_severity = aer->uncor_severity; info.uncor_mask = aer->uncor_mask; + info.link_status = link_status; + info.aer_cap_ctrl = aer->cap_control; info.device_status = device_status; + info.device_control_2 = device_control_2; info.first_error =PCI_ERR_CAP_FEP (aer->cap_control);pci_err (dev,"aer_status: 0x%08x, aer_mask: 0x%08x\n" , status, mask);
@@ -1197,7 +1201,9 @@ struct aer_recover_entry {u8 devfn;u16 domain;int severity; +u16 link_status;u16 device_status; +u16 device_control_2;struct aer_capability_regs *regs; };
@@ -1218,7 +1224,8 @@ static void aer_recover_work_func(struct work_struct *work) PCI_SLOT(entry.devfn), PCI_FUNC(entry.devfn));continue ; } -pci_print_aer (pdev, entry.severity, entry.regs, entry.device_status); +pci_print_aer (pdev, entry.severity, entry.regs, entry.device_status, + entry.link_status, entry.device_control_2);/* * Memory for aer_capability_regs(entry.regs) is being allocated from the * ghes_estatus_pool to protect it from overwriting when multiple sections
@@ -1247,7 +1254,8 @@ static DEFINE_SPINLOCK(aer_recover_ring_lock); static DECLARE_WORK(aer_recover_work, aer_recover_work_func);void aer_recover_queue (int domain,unsigned int bus,unsigned int devfn, -int severity,struct aer_capability_regs *aer_regs,u16 device_status) +int severity,struct aer_capability_regs *aer_regs,u16 device_status, +u16 link_status,u16 device_control_2) {struct aer_recover_entry entry = { .bus = bus,
@@ -1255,7 +1263,9 @@ void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn, .domain = domain, .severity = severity, .regs = aer_regs, + .link_status = link_status, .device_status = device_status, + .device_control_2 = device_control_2, }; if (kfifo_in_spinlocked(&aer_recover_ring, &entry, 1,
@@ -1281,7 +1291,6 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info) {int type =pci_pcie_type (dev);int aer = dev->aer_cap; -int temp;/* Must reset in this function */ info->cor_status =0 ;
@@ -1309,8 +1318,14 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info) &info->uncor_severity);pci_read_config_dword (dev, aer +PCI_ERR_UNCOR_MASK , &info->uncor_mask); +pci_read_config_dword (dev, aer +PCI_ERR_CAP , + &info->aer_cap_ctrl); +pcie_capability_read_word (dev,PCI_EXP_LNKSTA , + &info->link_status);pcie_capability_read_word (dev,PCI_EXP_DEVSTA , &info->device_status); +pcie_capability_read_word (dev,PCI_EXP_DEVCTL2 , + &info->device_control_2); }else {return 1 ; }
@@ -1323,8 +1338,7 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)return 0 ;/* Get First Error Pointer */ -pci_read_config_dword (dev, aer +PCI_ERR_CAP , &temp); - info->first_error =PCI_ERR_CAP_FEP (temp); + info->first_error =PCI_ERR_CAP_FEP (info->aer_cap_ctrl);if (info->uncor_status &AER_LOG_TLP_MASKS ) { info->tlp_header_valid =1 ;
include/linux/aer.h +4 -2
@@ -52,9 +52,11 @@ static inline int pcie_aer_is_native(struct pci_dev *dev) { return 0; }#endif void pci_print_aer (struct pci_dev *dev,int aer_severity, -struct aer_capability_regs *aer,u16 device_status); +struct aer_capability_regs *aer,u16 device_status, +u16 link_status,u16 device_control_2);int cper_severity_to_aer (int cper_severity);void aer_recover_queue (int domain,unsigned int bus,unsigned int devfn, -int severity,struct aer_capability_regs *aer_regs,u16 device_status); +int severity,struct aer_capability_regs *aer_regs,u16 device_status, +u16 link_status,u16 device_control_2);#endif //_AER_H_
Signature · 1 line
2.42.0
PATCH 4/4 ras: Trace more information in aer_event
In reply to Wang, QingshunAdd following fields in aer_event to better understand Advisory Non-Fatal and other errors for external observation:
- cor_status (Correctable Error Status)
- cor_mask (Correctable Error Mask)
- uncor_status (Uncorrectable Error Status)
- uncor_severity (Uncorrectable Error Severity)
- uncor_mask (Uncorrectable Error Mask)
- aer_cap_ctrl (AER Capabilities and Control)
- link_status (Link Status)
- device_status (Device Status)
- device_control_2 (Device Control 2)
In addition to the raw register value, value of following fields are extracted and logged for better observability:
- "First Error Pointer" and "Completion Timeout Prefix/Header Log Capable" from "AER Capabilities and Control"
- "Completion Timeout Value" and "Completion Timeout Disable" from "Device Control 2"
--- drivers/pci/pcie/aer.c | 17 +++++++++++-- include/ras/ras_event.h | 48 ++++++++++++++++++++++++++++++++--- include/uapi/linux/pci_regs.h | 1 + 3 files changed, 60 insertions(+), 6 deletions(-)
drivers/pci/pcie/aer.c +15 -2
@@ -754,6 +754,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)int layer, agent;int id =pci_dev_id (dev);const char *level; +struct aer_capability_regs aer_caps;if (info->severity ==AER_CORRECTABLE ) { status = info->cor_status;
@@ -790,8 +791,18 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)if (info->id && info->error_dev_num >1 && info->id == id)pci_err (dev," Error of this Agent is reported first\n" ); + aer_caps = (struct aer_capability_regs ) { + .cor_status = info->cor_status, + .cor_mask = info->cor_mask, + .uncor_status = info->uncor_status, + .uncor_severity = info->uncor_severity, + .uncor_mask = info->uncor_mask, + .cap_control = info->aer_cap_ctrl + };trace_aer_event (dev_name (&dev->dev), (status & ~mask), - info->severity, info->tlp_header_valid, &info->tlp); + info->severity, info->tlp_header_valid, &info->tlp, + &aer_caps, info->link_status, + info->device_status, info->device_control_2); }static void aer_print_port_info (struct pci_dev *dev,struct aer_err_info *info)
@@ -867,7 +878,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,__print_tlp_header (dev, &aer->header_log);trace_aer_event (dev_name (&dev->dev), (status & ~mask), - aer_severity, tlp_header_valid, &aer->header_log); + aer_severity, tlp_header_valid, &aer->header_log, + aer, info.link_status, + info.device_status, info.device_control_2); }EXPORT_SYMBOL_NS_GPL (pci_print_aer,CXL );
include/ras/ras_event.h +44 -4
79 lines not on this page · raw message
include/uapi/linux/pci_regs.h +1 -0
@@ -787,6 +787,7 @@#define PCI_ERR_CAP_ECRC_GENE 0x00000040/* ECRC Generation Enable */ #define PCI_ERR_CAP_ECRC_CHKC 0x00000080/* ECRC Check Capable */ #define PCI_ERR_CAP_ECRC_CHKE 0x00000100/* ECRC Check Enable */ +#define PCI_ERR_CAP_CTO_LOGC 0x00001000/* Completion Timeout Prefix/Header Log Capable */ #define PCI_ERR_HEADER_LOG 0x1c/* Header Log Register (16 bytes) */ #define PCI_ERR_ROOT_COMMAND 0x2c/* Root Error Command */ #define PCI_ERR_ROOT_CMD_COR_EN 0x00000001/* Correctable Err Reporting Enable */
Signature · 1 line
2.42.0
Hi,
Please ignore the Reivewed-by tags in this patch series, they were added by mistake.
Best regards,
Wang, Qingshun
Quoted 37 lines · Qingshun
According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal
ERR_CORinstead ofERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErrIf the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:
[ 170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID) [ 170.552420] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00041000/00180020 [ 170.561904] pcieport 0000:b7:02.0: [12] TLP (First) [ 170.569656] pcieport 0000:b7:02.0: [18] MalfTLPTo handle this case properly, this patch set adds additional fields in
aer_err_infoto track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data toaer_eventtracepoint, which would help to better understand ANFE and other errors for external observation.In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:
Quoted 6 lines
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErr [ 148.459868] pcieport 0000:b7:02.0: Uncorrectable errors that may cause Advisory Non-Fatal: [ 148.459868] pcieport 0000:b7:02.0: [18] TLP
Thanks for the overview here. It would be good to put some of these details in the commit logs of the patches that implement this, because this cover letter is not preserved when the series is merged.
If/when you do, remove the timestamps because they're not relevant and are merely distracting. Indent quoted material a couple spaces.
Update the citations to a current spec revision (PCIe r6.0, or maybe PCIe r6.1). The section numbers are probably the same, but there's no point in citing a revision that's 6.5 years old when newer ones are available.
Bjorn
Quoted 45 lines · Bjorn Helgaas
Quoted 37 lines · Qingshun
According to PCIe specification 4.0 sections 6.2.3.2.4 and 6.2.4.3, certain uncorrectable errors will signal
ERR_CORinstead ofERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in both Correctable Error Status register and Uncorrectable Error Status register. Currently, when handling AER event the kernel will only look at CE status or UE status, but never both. In the Advisory Non-Fatal Error case, bits set in UE status register will not be reported and cleared until the next Fatal/Non-Fatal error arrives.For instance, before this patch series, once kernel receives an ANFE with Poisoned TLP in OS native AER mode, only status of CE will be reported and cleared:
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErrIf the kernel receives a Malformed TLP after that, two UE will be reported, which is unexpected. Malformed TLP Header was lost since the previous ANF gated the TLP header logs:
[ 170.540192] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID) [ 170.552420] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00041000/00180020 [ 170.561904] pcieport 0000:b7:02.0: [12] TLP (First) [ 170.569656] pcieport 0000:b7:02.0: [18] MalfTLPTo handle this case properly, this patch set adds additional fields in
aer_err_infoto track both CE and UE status/mask and UE severity. This information will later be used to determine the register bits that need to be cleared. Additionally, adds more data toaer_eventtracepoint, which would help to better understand ANFE and other errors for external observation.In the previous scenario, after this patch series, both CE status and related UE status will be reported and cleared after ANFE:
Quoted 6 lines
[ 148.459816] pcieport 0000:b7:02.0: AER: Corrected error received: 0000:b7:02.0 [ 148.459858] pcieport 0000:b7:02.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID) [ 148.459863] pcieport 0000:b7:02.0: device [8086:0db0] error status/mask=00002000/00000000 [ 148.459868] pcieport 0000:b7:02.0: [13] NonFatalErr [ 148.459868] pcieport 0000:b7:02.0: Uncorrectable errors that may cause Advisory Non-Fatal: [ 148.459868] pcieport 0000:b7:02.0: [18] TLP
Quoted 3 lines
Thanks for the overview here. It would be good to put some of these details in the commit logs of the patches that implement this, because this cover letter is not preserved when the series is merged.
Thanks for your advice, will put some of these details in commit logs, mainly in PATCH 2.
Quoted 3 lines
If/when you do, remove the timestamps because they're not relevant and are merely distracting. Indent quoted material a couple spaces.
Agreed. Thanks.
Quoted 5 lines
Update the citations to a current spec revision (PCIe r6.0, or maybe PCIe r6.1). The section numbers are probably the same, but there's no point in citing a revision that's 6.5 years old when newer ones are available.
Makes sense, thanks!
Quoted 2 lines
Bjorn