Fix long standing AER Error Handling Issues
Subsystems (2)
v5 cover Fix long standing AER Error Handling Issues
This patch series aims at fixing some of the AER error handling issues we have.
Currently we have the following issues:
1. Confusing message in aer_print_error()
2. aer_err_info not being initialized completely in DPC path before we print the AER logs
3. A bug [1] in clearing of AER registers in the native AER path
[1] https://lore.kernel.org/linux-pci/20151229155822.GA17321@localhost/
The patch series fixes the above things.
PATCH 1:
- Fixes the first issue
- This patch is independent of other patches and can be applied seperately
PATCH 2 - 3:
- Fixes the second issue
- Patch 3 is depended on Patch 2 in the series
PATCH 4
- Fixes the bug in clearing of AER registers which leades to AER message spew [1]
PATCH 5:
- Adds extra information (devctl register) in AER error logs.
- Patch 5 depends on Patch 4 of the series
Thanks,
Naveen Naidu
Changelog
=========
v5:
- Edit the commit message of Patch 1 and Patch 5 to include how to test the AER messages using aer-inject.
- Edit Patch 3 to initialize info.id depending on the trigger reason.
- Drop few patches (v4 4/8, 5/8 7/8) since they were wrong.
v4:
- Fix logical error in 6/8, in the previous version of the patch set there was a bug, in how I added the devices to the queue.
v3:
- Edit the commit messages to be in imperative style and split the commits to be more atomic.
v2:
- Add [PATCH 7] which includes the device control register information in AER error logs.
Naveen Naidu (5):
[PATCH v5 1/5] PCI/AER: Remove ID from aer_agent_string[] [PATCH v5 2/5] PCI: Cleanup struct aer_err_info
[PATCH v5 3/5] PCI/DPC: Initialize info.id in dpc_process_error() [PATCH v5 4/5] PCI/AER: Clear error device AER registers in aer_irq() [PATCH v5 5/5] PCI/AER: Include DEVCTL in aer_print_error()
drivers/pci/pci.h | 23 +++- drivers/pci/pcie/aer.c | 269 ++++++++++++++++++++++++++++------------- drivers/pci/pcie/dpc.c | 16 ++- 3 files changed, 214 insertions(+), 94 deletions(-)
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
PATCH v5 1/5 PCI/AER: Remove ID from aer_agent_string[]
Currently, we do not print the "id" field in the AER error logs. Yet the aer_agent_string[] has the word "id" in it. The AER error log looks like:
pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Without the "id" field in the error log, The aer_agent_string[] (eg: "Receiver ID") does not make sense. A user reading the aer_agent_string[] in the log, might inadvertently look for an "id" field and not finding it might lead to confusion.
Remove the "ID" from the aer_agent_string[].
It is easy to reproduce this by using aer-inject:
$ aer-inject -s 00:03:0 corr-err-file
The content of the corr-err-file file is as below:
AER
COR_STATUS BAD_TLP
HEADER_LOG 0 1 2 3
The following are sample dummy errors inject via aer-inject.
Before
=======
In 010caed4ccb6 ("PCI/AER: Decode Error Source Requester ID"), the "id" field was removed from the AER error logs, so currently AER logs look like:
pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03:0
pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID) <--- no id field
pcieport 0000:00:03.0: device [1b36:000c] error status/mask=00000040/0000e000
pcieport 0000:00:03.0: [ 6] BadTLP
After
======
pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver)
pcieport 0000:00:03.0: device [1b36:000c] error status/mask=00000040/0000e000
pcieport 0000:00:03.0: [ 6] BadTLP
--- drivers/pci/pcie/aer.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/pci/pcie/aer.c +5 -5
@@ -516,10 +516,10 @@ static const char *aer_uncorrectable_error_string[] = { };static const char *aer_agent_string[] = { -"Receiver ID" , -"Requester ID" , -"Completer ID" , -"Transmitter ID" +"Receiver" , +"Requester" , +"Completer" , +"Transmitter" }; #define aer_stats_dev_attr(name, stats_array, strings_array, \
@@ -703,7 +703,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)const char *level;if (!info->status) { -pci_err (dev,"PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n" , +pci_err (dev,"PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent)\n" , aer_error_severity_string[info->severity]); goto out; }
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
PATCH v5 2/5 PCI: Cleanup struct aer_err_info
In reply to Naveen NaiduThe id, status and the mask fields of the struct aer_err_info comes directly from the registers, hence their sizes should be explicit.
The length of these registers are:
- id: 16 bits - Represents the Error Source Requester ID
- status: 32 bits - COR/UNCOR Error Status
- mask: 32 bits - COR/UNCOR Error Mask
Since the length of the above registers are even, use u16 and u32 to represent their values.
Also remove the __pad fields.
"pahole" was run on the modified struct aer_err_info and the size remains unchanged.
--- drivers/pci/pci.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/pci/pci.h +3 -5
@@ -427,18 +427,16 @@ struct aer_err_info {struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES ];int error_dev_num; - unsigned int id:16; +u16 id; unsigned int severity:2; /* 0:NONFATAL | 1:FATAL | 2:COR */ - unsigned int __pad1:5; unsigned int multi_error_valid:1; unsigned int first_error:5; - unsigned int __pad2:2; unsigned int tlp_header_valid:1; -unsigned int status;/* COR/UNCOR Error Status */ -unsigned int mask;/* COR/UNCOR Error Mask */ +u32 status;/* COR/UNCOR Error Status */ +u32 mask;/* COR/UNCOR Error Mask */ struct aer_header_log_regs tlp;/* TLP Header */ };
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
PATCH v5 3/5 PCI/DPC: Initialize info.id in dpc_process_error()
In reply to Naveen NaiduIn the dpc_process_error() path, info.id isn't initialized before being passed to aer_print_error(). In the corresponding AER path, it is initialized in aer_isr_one_error().
The error message shown during Coverity Scan is:
Coverity #1461602
CID 1461602 (#1 of 1): Uninitialized scalar variable (UNINIT)
8. uninit_use_in_call: Using uninitialized value info.id when calling aer_print_error.
Also Per PCIe r5.0, sec 7.9.15.5, the Source ID is defined only when the Trigger Reason indicates ERR_NONFATAL or ERR_FATAL. Initialize the "info.id" based on the trigger reason before passing it to aer_print_error()
--- drivers/pci/pcie/dpc.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
drivers/pci/pcie/dpc.c +12 -4
@@ -262,16 +262,24 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,void dpc_process_error (struct pci_dev *pdev) { -u16 cap = pdev->dpc_cap, status, source, reason, ext_reason; +u16 cap = pdev->dpc_cap, status, reason, ext_reason;struct aer_err_info info;pci_read_config_word (pdev, cap +PCI_EXP_DPC_STATUS , &status); -pci_read_config_word (pdev, cap +PCI_EXP_DPC_SOURCE_ID , &source); + reason = (status &PCI_EXP_DPC_STATUS_TRIGGER_RSN ) >>1 ; + +/* +* Per PCIe r5.0, sec 7.9.15.5, the Source ID is defined only when the +* Trigger Reason indicates ERR_NONFATAL or ERR_FATAL. +*/ +if (reason ==1 || reason ==2 ) +pci_read_config_word (pdev, cap +PCI_EXP_DPC_SOURCE_ID , &info.id); +else + info.id =0 ;pci_info (pdev,"containment event, status:%#06x source:%#06x\n" , - status, source); + status, info.id); - reason = (status &PCI_EXP_DPC_STATUS_TRIGGER_RSN ) >>1 ; ext_reason = (status &PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT ) >>5 ; pci_warn(pdev, "%s detected\n", (reason == 0) ? "unmasked uncorrectable error" :
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
PATCH v5 4/5 PCI/AER: Clear error device AER registers in aer_irq()
In reply to Naveen NaiduConverge the APEI path and native AER path of clearing the AER registers of the error device.
In APEI path, the system firmware clears the AER registers before handing off the record to OS. But in "native AER" path, the execution path of clearing the AER register is as follows:
aer_isr_one_error
aer_print_port_info
if (find_source_device())
aer_process_err_devices
handle_error_source
pci_write_config_dword(dev, PCI_ERR_COR_STATUS, ...)
The above path has a bug, if the find_source_device() fails, AER registers are not cleared from the error device. This means, the error device will keep reporting the error again and again and would lead to message spew.
Related Bug Report:
https://lore.kernel.org/linux-pci/20151229155822.GA17321@localhost/
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1521173
The above bug could be avoided, if the AER registers are cleared during the AER IRQ handler aer_irq(), which would provide guarantee that the AER error registers are always cleared. This is similar to how APEI handles these errors.
The main aim is that:
When an interrupt handler deals with a interrupt, it must *always* clear the source of the interrupt.
--- drivers/pci/pci.h | 13 ++- drivers/pci/pcie/aer.c | 249 ++++++++++++++++++++++++++++------------- 2 files changed, 184 insertions(+), 78 deletions(-)
drivers/pci/pci.h +12 -1
@@ -424,7 +424,6 @@ static inline bool pci_dev_is_added(const struct pci_dev *dev)#define AER_MAX_MULTI_ERR_DEVICES 5/* Not likely to have more */ struct aer_err_info { -struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES ];int error_dev_num;u16 id;
@@ -440,6 +439,18 @@ struct aer_err_info { struct aer_header_log_regs tlp; /* TLP Header */ }; +/* Preliminary AER error information processed from Root port */ +struct aer_devices_err_info { +struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES ]; +struct aer_err_info err_info; +}; + +/* AER information associated with each error device */ +struct aer_dev_err_info { +struct pci_dev *dev; +struct aer_err_info err_info; +}; +int aer_get_device_error_info (struct pci_dev *dev,struct aer_err_info *info);void aer_print_error (struct pci_dev *dev,struct aer_err_info *info);#endif /* CONFIG_PCIEAER */
drivers/pci/pcie/aer.c +172 -77
371 lines not on this page · raw message
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
PATCH v5 5/5 PCI/AER: Include DEVCTL in aer_print_error()
In reply to Naveen NaiduPrint the contents of Device Control Register of the device which detected the error. This might help in faster error diagnosis.
It is easy to test this by using aer-inject:
$ aer-inject -s 00:03:0 corr-err-file
The content of the corr-err-file is as below:
AER
COR_STATUS BAD_TLP
HEADER_LOG 0 1 2 3
Sample output from dummy error injected by aer-inject:
pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver)
pcieport 0000:00:03.0: device [1b36:000c] error status/mask=00000040/0000e000, devctl=0x000f <-- devctl added to the error log
pcieport 0000:00:03.0: [ 6] BadTLP
--- drivers/pci/pci.h | 2 ++ drivers/pci/pcie/aer.c | 10 ++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
drivers/pci/pci.h +2 -0
@@ -437,6 +437,8 @@ struct aer_err_info {u32 status;/* COR/UNCOR Error Status */ u32 mask;/* COR/UNCOR Error Mask */ struct aer_header_log_regs tlp;/* TLP Header */ + +u16 devctl; };/* Preliminary AER error information processed from Root port */
drivers/pci/pcie/aer.c +8 -2
@@ -729,8 +729,8 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info) aer_error_severity_string[info->severity], aer_error_layer[layer], aer_agent_string[agent]); -pci_printk (level, dev," device [%04x:%04x] error status/mask=%08x/%08x\n" , - dev->vendor, dev->device, info->status, info->mask); +pci_printk (level, dev," device [%04x:%04x] error status/mask=%08x/%08x, devctl=%#06x\n" , + dev->vendor, dev->device, info->status, info->mask, info->devctl);__aer_print_error (dev, info);
@@ -1083,6 +1083,12 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)if (!aer)return 0 ; +/* +* Cache the value of Device Control Register now, because later the +* device might not be available +*/ +pcie_capability_read_word (dev,PCI_EXP_DEVCTL , &info->devctl); +if (info->severity ==AER_CORRECTABLE ) {pci_read_config_dword (dev, aer +PCI_ERR_COR_STATUS , &info->status);
Signature · 6 lines
2.25.1 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees