Skip to contentSkip to conversation

Rate limit AER logs

PATCH Superseded Bjorn Helgaas 4 participants 57 messages May 20, 2025 – May 22, 2025

Subsystems (3)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v7 cover Rate limit AER logs

From: Bjorn Helgaas <[email protected]>

This work is mostly due to Jon Pan-Doh and Karolina Stolarek. I rebased this to v6.15-rc1, factored out some of the trace and statistics updates, and added some minor cleanups.

I'm sorry to post a v7 so soon after v6, but I really want to get this in v6.16 so it needs to get into pci/next soonish. I pushed this to pci/aer at https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/log/?h=aer
(head cbde036e5615 ("PCI/AER: Add sysfs attributes for log ratelimits")) and appended the interdiff from v6 to v7 below.

Proposal
========

When using native AER, spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device per-error-severity ratelimits for more robust error logging. Allow userspace to configure ratelimits via sysfs knobs.

Motivation
==========

Inconsistent PCIe error handling, exacerbated at datacenter scale (myriad of devices), affects repairabilitiy flows for fleet operators.

Exposing PCIe errors/debug info in-band for a userspace daemon (e.g. rasdaemon) to collect/pass on to repairability services will allow for more predictable repair flows and decrease machine downtime.

Background
==========

AER error spam has been observed many times, both publicly (e.g. [1], [2], [3]) and privately. While it usually occurs with correctable errors, it can happen with uncorrectable errors (e.g. during new HW bringup).

There have been previous attempts to add ratelimits to AER logs ([4], [5]). The most recent attempt[5] has many similarities with the proposed approach.

v7:
- Update sysfs doc target kernel version & date (Ilpo)
- Fix sysfs doc "AER ratelimiting" typo (Ilpo)
- Ratelimit Correctable and Non-Fatal but not Fatal errors (Sathy)
- Rename "struct aer_report" to "aer_info" (Sathy)
- Expand comments about combining ratelimit for multiple devices (Ilpo)
- Rework Error Source logging ratelimiting (Sathy)
- Factor out aer_isr_one_error_type() to reduce code duplication
- Log DPC errors, which are all Fatal, at KERN_ERR (Sathy)
- Improve dpc_process_error() structure (Ilpo)
v6: https://lore.kernel.org/r/[email protected]
- Rebase to v6.15-rc1
- Initialize struct aer_err_info completely before using it
- Log DPC Error Source ID only when it's valid
- Consolidate AER Error Source ID logging to one place
- Tidy Error Source ID bus/dev/fn decoding using macros
- Rename aer_print_port_info() to aer_print_source()
- Consolidate trace events and statistic updates to one non-ratelimited place
- Save log level in struct aer_err_info instead of passing as parameter
v5: https://lore.kernel.org/r/[email protected]
- Handle multi-error AER by evaluating ratelimits once and storing result
- Reword/rename commit messages/functions/variable
v4: https://lore.kernel.org/r/[email protected]
- Fix bug where trace not emitted with malformed aer_err_info
- Extend ratelimit to malformed aer_err_info
- Update commit messages with patch motivation
- Squash AER sysfs filename change (Patch 8)
v3: https://lore.kernel.org/r/[email protected]
- Ratelimit aer_print_port_info() (drop Patch 1)
- Add ratelimit enable toggle
- Move trace outside of ratelimit
- Split log level (Patch 2) into two
- More descriptive documentation/sysfs naming
v2: https://lore.kernel.org/r/[email protected]
- Rebased on top of pci/aer (6.14.rc-1)
- Split series into log and IRQ ratelimits (defer patch 5)
- Dropped patch 8 (Move AER sysfs)
- Added log level cleanup patch[7] from Karolina's series
- Fixed bug where dpc errors didn't increment counters
- "X callbacks suppressed" message on ratelimit release -> immediately
- Separate documentation into own patch
v1: https://lore.kernel.org/r/[email protected]

[1] https://bugzilla.kernel.org/show_bug.cgi?id=215027
[2] https://bugzilla.kernel.org/show_bug.cgi?id=201517
[3] https://bugzilla.kernel.org/show_bug.cgi?id=196183
[4] https://lore.kernel.org/linux-pci/[email protected]/
[5] https://lore.kernel.org/linux-pci/[email protected]/
[6]
https://lore.kernel.org/linux-pci/8bcb8c9a7b38ce3bdaca5a64fe76f08b0b337511.1742202797.git.k
[email protected]/
[7]
https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.k
[email protected]/

Bjorn Helgaas (10):
PCI/DPC: Initialize aer_err_info before using it
PCI/DPC: Log Error Source ID only when valid
PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error() PCI/AER: Consolidate Error Source ID logging in
aer_isr_one_error_type()
PCI/AER: Extract bus/dev/fn in aer_print_port_info() with

    PCI_BUS_NUM(), etc

PCI/AER: Move aer_print_source() earlier in file PCI/AER: Initialize aer_err_info before using it PCI/AER: Simplify pci_print_aer()
PCI/AER: Update statistics early in logging PCI/AER: Combine trace_aer_event() with statistics updates

Jon Pan-Doh (4):
PCI/AER: Rename aer_print_port_info() to aer_print_source() PCI/AER: Ratelimit correctable and non-fatal error logging PCI/AER: Add ratelimits to PCI AER Documentation PCI/AER: Add sysfs attributes for log ratelimits

Karolina Stolarek (3):
PCI/AER: Check log level once and remember it
PCI/AER: Make all pci_print_aer() log levels depend on error type PCI/AER: Rename struct aer_stats to aer_info

 ...es-aer_stats => sysfs-bus-pci-devices-aer} |  34 ++
 Documentation/PCI/pcieaer-howto.rst           |  17 +-
 drivers/pci/pci-sysfs.c                       |   1 +
 drivers/pci/pci.h                             |   5 +-
 drivers/pci/pcie/aer.c                        | 387 +++++++++++++-----
 drivers/pci/pcie/dpc.c                        |  68 +--
 include/linux/pci.h                           |   2 +-
 7 files changed, 370 insertions(+), 144 deletions(-)
 rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)
Signature · 1 line
2.43.0
Documentation/ABI/testing/sysfs-bus-pci-devices-aer +12 -12
index 771204197b71..01bb577bfee8 100644
@@ -127,16 +127,16 @@ They represent configurable ratelimits of logs per error type.
 See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
 
 What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
-Date:		March 2025
-KernelVersion:	6.15.0
+Date:		May 2025
+KernelVersion:	6.16.0
 Contact:	[email protected], [email protected]
 Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
-		gets whether or not AER is currently enabled. Enabled by
-		default.
+		gets whether or not AER ratelimiting is currently enabled.
+		Enabled by default.
 
 What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
-Date:		March 2025
-KernelVersion:	6.15.0
+Date:		May 2025
+KernelVersion:	6.16.0
 Contact:	[email protected], [email protected]
 Description:	Ratelimit burst for correctable error logs. Writing a value
 		changes the number of errors (burst) allowed per interval
@@ -144,10 +144,10 @@ Description:	Ratelimit burst for correctable error logs. Writing a value
 		current ratelimit burst.
 
 What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
-Date:		March 2025
-KernelVersion:	6.15.0
+Date:		May 2025
+KernelVersion:	6.16.0
 Contact:	[email protected], [email protected]
-Description:	Ratelimit burst for uncorrectable error logs. Writing a
-		value changes the number of errors (burst) allowed per
-		interval (5 second window) before ratelimiting. Reading
-		gets the current ratelimit burst.
+Description:	Ratelimit burst for non-fatal uncorrectable error logs.
+		Writing a value changes the number of errors (burst)
+		allowed per interval (5 second window) before ratelimiting.
+		Reading gets the current ratelimit burst.
Documentation/PCI/pcieaer-howto.rst +2 -1
index 043cdb3194be..4b71e2f43ca7 100644
@@ -91,7 +91,8 @@ AER Ratelimits
 Since error messages can be generated for each transaction, we may see
 large volumes of errors reported. To prevent spammy devices from flooding
 the console/stalling execution, messages are throttled by device and error
-type (correctable vs. uncorrectable).
+type (correctable vs. non-fatal uncorrectable).  Fatal errors, including
+DPC errors, are not ratelimited.
 
 AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
 DEFAULT_RATELIMIT_INTERVAL (5 seconds).
drivers/pci/pcie/aer.c +86 -63

322 lines not on this page · raw message

drivers/pci/pcie/dpc.c +14 -17
index 597df7790f36..530c5e2cf7e8 100644
@@ -252,14 +252,14 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 	else
 		info->severity = AER_NONFATAL;
 
-	info->level = KERN_WARNING;
+	info->level = KERN_ERR;
 	return 1;
 }
 
 void dpc_process_error(struct pci_dev *pdev)
 {
 	u16 cap = pdev->dpc_cap, status, source, reason, ext_reason;
-	struct aer_err_info info = { 0 };
+	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
 
@@ -269,6 +269,13 @@ void dpc_process_error(struct pci_dev *pdev)
 	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR:
 		pci_warn(pdev, "containment event, status:%#06x: unmasked uncorrectable error detected\n",
 			 status);
+		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
+		    aer_get_device_error_info(pdev, &info)) {
+			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
+			aer_print_error(pdev, &info);
+			pci_aer_clear_nonfatal_status(pdev);
+			pci_aer_clear_fatal_status(pdev);
+		}
 		break;
 	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE:
 	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE:
@@ -280,7 +287,7 @@ void dpc_process_error(struct pci_dev *pdev)
 				"ERR_FATAL" : "ERR_NONFATAL",
 			 pci_domain_nr(pdev->bus), PCI_BUS_NUM(source),
 			 PCI_SLOT(source), PCI_FUNC(source));
-		return;
+		break;
 	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT:
 		ext_reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT;
 		pci_warn(pdev, "containment event, status:%#06x: %s detected\n",
@@ -290,22 +297,12 @@ void dpc_process_error(struct pci_dev *pdev)
 			 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_SW_TRIGGER) ?
 			 "software trigger" :
 			 "reserved error");
+		/* show RP PIO error detail information */
+		if (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO &&
+		    pdev->dpc_rp_extensions)
+			dpc_process_rp_pio_error(pdev);
 		break;
 	}
-
-	/* show RP PIO error detail information */
-	if (pdev->dpc_rp_extensions &&
-	    reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT &&
-	    ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO)
-		dpc_process_rp_pio_error(pdev);
-	else if (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR &&
-		 dpc_get_aer_uncorrect_severity(pdev, &info) &&
-		 aer_get_device_error_info(pdev, &info)) {
-		info.ratelimit = 1;	/* no ratelimiting */
-		aer_print_error(pdev, &info);
-		pci_aer_clear_nonfatal_status(pdev);
-		pci_aer_clear_fatal_status(pdev);
-	}
 }
 
 static void pci_clear_surpdn_errors(struct pci_dev *pdev)
include/linux/pci.h +1 -1
index 4b11a90107cb..81a81dbfc873 100644
@@ -346,7 +346,7 @@ struct pci_dev {
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
 #ifdef CONFIG_PCIEAER
 	u16		aer_cap;	/* AER capability offset */
-	struct aer_report *aer_report;	/* AER report for this device */
+	struct aer_info	*aer_info;	/* AER info for this device */
 #endif
 #ifdef CONFIG_PCIEPORTBUS
 	struct rcec_ea	*rcec_ea;	/* RCEC cached endpoint association */
#1 lore raw

PATCH v7 1/17 PCI/DPC: Initialize aer_err_info before using it

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Initialize "info" at declaration so it starts as all zeros.

---
 drivers/pci/pcie/dpc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pcie/dpc.c +1 -1
index df42f15c9829..3daaf61c79c9 100644
@@ -258,7 +258,7 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 void dpc_process_error(struct pci_dev *pdev)
 {
 	u16 cap = pdev->dpc_cap, status, source, reason, ext_reason;
-	struct aer_err_info info;
+	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
Signature · 1 line
2.43.0
Reviewed-by on v7 1/17
#2 lore raw

Re: [PATCH v7 01/17] PCI/DPC: Initialize aer_err_info before using it

Quoted 11 lines · On Tue, 20 May 2025 16:50:18 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Initialize "info" at declaration so it starts as all zeros.

I chased this through a bit and looks like at least some unset fields would result in garbage prints. So maybe needs a fixes tag? info->tlp_header_valid is an easy one to follow as only set in some paths.

Otherwise absolutely makes sense.

Quoted 17 lines
---
 drivers/pci/pcie/dpc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pcie/dpc.c +1 -1
index df42f15c9829..3daaf61c79c9 100644
@@ -258,7 +258,7 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 void dpc_process_error(struct pci_dev *pdev)
 {
 	u16 cap = pdev->dpc_cap, status, source, reason, ext_reason;
-	struct aer_err_info info;
+	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
#3 lore raw

Re: [PATCH v7 01/17] PCI/DPC: Initialize aer_err_info before using it

Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Tue, 20 May 2025 16:50:18 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Initialize "info" at declaration so it starts as all zeros.

Quoted 7 lines
Quoted 3 lines

I chased this through a bit and looks like at least some unset fields would result in garbage prints. So maybe needs a fixes tag? info->tlp_header_valid is an easy one to follow as only set in some paths.

I don't see a current issue related to info->tlp_header_valid because it's always cleared in the path from DPC:

  dpc_process_error
    dpc_get_aer_uncorrect_severity
      info->severity = AER_FATAL or AER_NONFATAL
    aer_get_device_error_info
      info->status = 0;
      info->tlp_header_valid = 0;    # unconditional
      pci_read_config_dword(PCI_ERR_UNCOR_STATUS, &info->status);
      pci_read_config_dword(PCI_ERR_UNCOR_MASK, &info->mask);
      info->first_error = PCI_ERR_CAP_FEP(...);
    aer_print_error

However, only info->{severity,status,tlp_header_valid,mask, first_error} are initialized before dpc_process_error() calls aer_print_error(), and aer_print_error() uses info->{id, error_dev_num}, which haven't been initialized.

AFAICT, this has been the case since
https://git.kernel.org/linus/8aefa9b0d910 ("PCI/DPC: Print AER status in DPC event handling")

Here's aer_get_device_error_info() at that time:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pcie/aer.c?id=8aefa9b0d910#n867

and aer_print_error():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pcie/aer.c?id=8aefa9b0d910#n530

At the time of 8aefa9b0d910, info->severity was also used without initialization; this was fixed by 9f08a5d896ce ("PCI/DPC: Fix print AER status in DPC event handling").

But I think the info->{id,error_dev_num} has been there since.

Anyway, I added:

Fixes 8aefa9b0d910 ("PCI/DPC: Print AER status in DPC event handling")
Quoted 20 lines

Otherwise absolutely makes sense.

Quoted 17 lines
---
 drivers/pci/pcie/dpc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pcie/dpc.c +1 -1
index df42f15c9829..3daaf61c79c9 100644
@@ -258,7 +258,7 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 void dpc_process_error(struct pci_dev *pdev)
 {
 	u16 cap = pdev->dpc_cap, status, source, reason, ext_reason;
-	struct aer_err_info info;
+	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
Reviewed-by on v7 1/17
#4 lore raw

Re: [PATCH v7 01/17] PCI/DPC: Initialize aer_err_info before using it

In reply to Bjorn Helgaas
Quoted 21 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Initialize "info" at declaration so it starts as all zeros.

---
 drivers/pci/pcie/dpc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/pcie/dpc.c +0 -0
index df42f15c9829..3daaf61c79c9 100644

drivers/pci/pcie/dpc.c static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev, · line 264

Quoted 8 lines
drivers/pci/pcie/dpc.c +1 -1
@@ -258,7 +258,7 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 void dpc_process_error(struct pci_dev *pdev)
 {
 	u16 cap = pdev->dpc_cap, status, source, reason, ext_reason;
-	struct aer_err_info info;
+	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
Signature · 1 line
 i.
#5 lore raw

PATCH v7 2/17 PCI/DPC: Log Error Source ID only when valid

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

DPC Error Source ID is only valid when the DPC Trigger Reason indicates that DPC was triggered due to reception of an ERR_NONFATAL or ERR_FATAL Message (PCIe r6.0, sec 7.9.14.5).

When DPC was triggered by ERR_NONFATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) or ERR_FATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) from a downstream device, log the Error Source ID (decoded into domain/bus/device/function). Don't print the source otherwise, since it's not valid.

For DPC trigger due to reception of ERR_NONFATAL or ERR_FATAL, the dmesg logging changes:

- pci 0000:00:01.0: DPC: containment event, status:0x000d source:0x0200
- pci 0000:00:01.0: DPC: ERR_FATAL detected
+ pci 0000:00:01.0: DPC: containment event, status:0x000d, ERR_FATAL received from 0000:02:00.0

and when DPC triggered for other reasons, where DPC Error Source ID is undefined, e.g., unmasked uncorrectable error:

- pci 0000:00:01.0: DPC: containment event, status:0x0009 source:0x0200
- pci 0000:00:01.0: DPC: unmasked uncorrectable error detected
+ pci 0000:00:01.0: DPC: containment event, status:0x0009: unmasked uncorrectable error detected

Previously the "containment event" message was at KERN_INFO and the "%s detected" message was at KERN_WARNING. Now the single message is at KERN_WARNING.

---
 drivers/pci/pcie/dpc.c | 64 ++++++++++++++++++++++++------------------
 1 file changed, 36 insertions(+), 28 deletions(-)
drivers/pci/pcie/dpc.c +36 -28
index 3daaf61c79c9..9d85f1b3b761 100644
@@ -261,37 +261,45 @@ void dpc_process_error(struct pci_dev *pdev)
 	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
-	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
-
-	pci_info(pdev, "containment event, status:%#06x source:%#06x\n",
-		 status, source);
 
 	reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN;
-	ext_reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT;
-	pci_warn(pdev, "%s detected\n",
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR) ?
-		 "unmasked uncorrectable error" :
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) ?
-		 "ERR_NONFATAL" :
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) ?
-		 "ERR_FATAL" :
-		 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO) ?
-		 "RP PIO error" :
-		 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_SW_TRIGGER) ?
-		 "software trigger" :
-		 "reserved error");
 
-	/* show RP PIO error detail information */
-	if (pdev->dpc_rp_extensions &&
-	    reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT &&
-	    ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO)
-		dpc_process_rp_pio_error(pdev);
-	else if (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR &&
-		 dpc_get_aer_uncorrect_severity(pdev, &info) &&
-		 aer_get_device_error_info(pdev, &info)) {
-		aer_print_error(pdev, &info);
-		pci_aer_clear_nonfatal_status(pdev);
-		pci_aer_clear_fatal_status(pdev);
+	switch (reason) {
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR:
+		pci_warn(pdev, "containment event, status:%#06x: unmasked uncorrectable error detected\n",
+			 status);
+		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
+		    aer_get_device_error_info(pdev, &info)) {
+			aer_print_error(pdev, &info);
+			pci_aer_clear_nonfatal_status(pdev);
+			pci_aer_clear_fatal_status(pdev);
+		}
+		break;
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE:
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE:
+		pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID,
+				     &source);
+		pci_warn(pdev, "containment event, status:%#06x, %s received from %04x:%02x:%02x.%d\n",
+			 status,
+			 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) ?
+				"ERR_FATAL" : "ERR_NONFATAL",
+			 pci_domain_nr(pdev->bus), PCI_BUS_NUM(source),
+			 PCI_SLOT(source), PCI_FUNC(source));
+		break;
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT:
+		ext_reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT;
+		pci_warn(pdev, "containment event, status:%#06x: %s detected\n",
+			 status,
+			 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO) ?
+			 "RP PIO error" :
+			 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_SW_TRIGGER) ?
+			 "software trigger" :
+			 "reserved error");
+		/* show RP PIO error detail information */
+		if (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO &&
+		    pdev->dpc_rp_extensions)
+			dpc_process_rp_pio_error(pdev);
+		break;
 	}
 }
 
Signature · 1 line
2.43.0
Reviewed-by on v7 2/17
#6 lore raw

Re: [PATCH v7 02/17] PCI/DPC: Log Error Source ID only when valid

Quoted 25 lines · On Tue, 20 May 2025 16:50:19 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

DPC Error Source ID is only valid when the DPC Trigger Reason indicates that DPC was triggered due to reception of an ERR_NONFATAL or ERR_FATAL Message (PCIe r6.0, sec 7.9.14.5).

When DPC was triggered by ERR_NONFATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) or ERR_FATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) from a downstream device, log the Error Source ID (decoded into domain/bus/device/function). Don't print the source otherwise, since it's not valid.

For DPC trigger due to reception of ERR_NONFATAL or ERR_FATAL, the dmesg logging changes:

- pci 0000:00:01.0: DPC: containment event, status:0x000d source:0x0200
- pci 0000:00:01.0: DPC: ERR_FATAL detected
+ pci 0000:00:01.0: DPC: containment event, status:0x000d, ERR_FATAL received from 0000:02:00.0

and when DPC triggered for other reasons, where DPC Error Source ID is undefined, e.g., unmasked uncorrectable error:

- pci 0000:00:01.0: DPC: containment event, status:0x0009 source:0x0200
- pci 0000:00:01.0: DPC: unmasked uncorrectable error detected
+ pci 0000:00:01.0: DPC: containment event, status:0x0009: unmasked uncorrectable error detected

Quoted 7 lines

Previously the "containment event" message was at KERN_INFO and the "%s detected" message was at KERN_WARNING. Now the single message is at KERN_WARNING.

Matches the spec conditions as far as I can tell.

I guess interesting debate on whether providing extra garbage info is a bug or not. Maybe a fixes tag for this one as well?

I briefly wondered if it makes sense to have a prefix string initialized outside the switch with "containment event, status:%#06x:" made sense but it's probably not worth the effort and maybe makes it harder to grep for the error messages. So in the end
I think your code here is the best option.

Jonathan

#7 lore raw

Re: [PATCH v7 02/17] PCI/DPC: Log Error Source ID only when valid

Quoted 32 lines · Jonathan Cameron
Quoted 29 lines · On Tue, 20 May 2025 16:50:19 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

DPC Error Source ID is only valid when the DPC Trigger Reason indicates that DPC was triggered due to reception of an ERR_NONFATAL or ERR_FATAL Message (PCIe r6.0, sec 7.9.14.5).

When DPC was triggered by ERR_NONFATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) or ERR_FATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) from a downstream device, log the Error Source ID (decoded into domain/bus/device/function). Don't print the source otherwise, since it's not valid.

For DPC trigger due to reception of ERR_NONFATAL or ERR_FATAL, the dmesg logging changes:

- pci 0000:00:01.0: DPC: containment event, status:0x000d source:0x0200
- pci 0000:00:01.0: DPC: ERR_FATAL detected
+ pci 0000:00:01.0: DPC: containment event, status:0x000d, ERR_FATAL received from 0000:02:00.0

and when DPC triggered for other reasons, where DPC Error Source ID is undefined, e.g., unmasked uncorrectable error:

- pci 0000:00:01.0: DPC: containment event, status:0x0009 source:0x0200
- pci 0000:00:01.0: DPC: unmasked uncorrectable error detected
+ pci 0000:00:01.0: DPC: containment event, status:0x0009: unmasked uncorrectable error detected

Previously the "containment event" message was at KERN_INFO and the "%s detected" message was at KERN_WARNING. Now the single message is at KERN_WARNING.

Quoted 7 lines
Quoted 3 lines

Matches the spec conditions as far as I can tell.

I guess interesting debate on whether providing extra garbage info is a bug or not. Maybe a fixes tag for this one as well?

I added:

Fixes 26e515713342 ("PCI: Add Downstream Port Containment driver")

since it looks like we've printed the source unconditionally since the addition of DPC:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pcie/pcie-dpc.c?id=26e515713342#n69

Quoted 9 lines

I briefly wondered if it makes sense to have a prefix string initialized outside the switch with "containment event, status:%#06x:" made sense but it's probably not worth the effort and maybe makes it harder to grep for the error messages. So in the end
I think your code here is the best option.

Jonathan

Reviewed-by on v7 2/17
#8 lore raw

Re: [PATCH v7 02/17] PCI/DPC: Log Error Source ID only when valid

In reply to Bjorn Helgaas
Quoted 105 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

DPC Error Source ID is only valid when the DPC Trigger Reason indicates that DPC was triggered due to reception of an ERR_NONFATAL or ERR_FATAL Message (PCIe r6.0, sec 7.9.14.5).

When DPC was triggered by ERR_NONFATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) or ERR_FATAL (PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) from a downstream device, log the Error Source ID (decoded into domain/bus/device/function). Don't print the source otherwise, since it's not valid.

For DPC trigger due to reception of ERR_NONFATAL or ERR_FATAL, the dmesg logging changes:

- pci 0000:00:01.0: DPC: containment event, status:0x000d source:0x0200
- pci 0000:00:01.0: DPC: ERR_FATAL detected
+ pci 0000:00:01.0: DPC: containment event, status:0x000d, ERR_FATAL received from 0000:02:00.0

and when DPC triggered for other reasons, where DPC Error Source ID is undefined, e.g., unmasked uncorrectable error:

- pci 0000:00:01.0: DPC: containment event, status:0x0009 source:0x0200
- pci 0000:00:01.0: DPC: unmasked uncorrectable error detected
+ pci 0000:00:01.0: DPC: containment event, status:0x0009: unmasked uncorrectable error detected

Previously the "containment event" message was at KERN_INFO and the "%s detected" message was at KERN_WARNING. Now the single message is at KERN_WARNING.

---
 drivers/pci/pcie/dpc.c | 64 ++++++++++++++++++++++++------------------
 1 file changed, 36 insertions(+), 28 deletions(-)
drivers/pci/pcie/dpc.c +29 -28
index 3daaf61c79c9..9d85f1b3b761 100644
@@ -261,37 +261,45 @@ void dpc_process_error(struct pci_dev *pdev)
 	struct aer_err_info info = {};
 
 	pci_read_config_word(pdev, cap + PCI_EXP_DPC_STATUS, &status);
-	pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID, &source);
-
-	pci_info(pdev, "containment event, status:%#06x source:%#06x\n",
-		 status, source);
 
 	reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN;
-	ext_reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT;
-	pci_warn(pdev, "%s detected\n",
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR) ?
-		 "unmasked uncorrectable error" :
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE) ?
-		 "ERR_NONFATAL" :
-		 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) ?
-		 "ERR_FATAL" :
-		 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO) ?
-		 "RP PIO error" :
-		 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_SW_TRIGGER) ?
-		 "software trigger" :
-		 "reserved error");
 
-	/* show RP PIO error detail information */
-	if (pdev->dpc_rp_extensions &&
-	    reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT &&
-	    ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO)
-		dpc_process_rp_pio_error(pdev);
-	else if (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR &&
-		 dpc_get_aer_uncorrect_severity(pdev, &info) &&
-		 aer_get_device_error_info(pdev, &info)) {
-		aer_print_error(pdev, &info);
-		pci_aer_clear_nonfatal_status(pdev);
-		pci_aer_clear_fatal_status(pdev);
+	switch (reason) {
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_UNCOR:
+		pci_warn(pdev, "containment event, status:%#06x: unmasked uncorrectable error detected\n",
+			 status);
+		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
+		    aer_get_device_error_info(pdev, &info)) {
+			aer_print_error(pdev, &info);
+			pci_aer_clear_nonfatal_status(pdev);
+			pci_aer_clear_fatal_status(pdev);
+		}
+		break;
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_NFE:
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE:
+		pci_read_config_word(pdev, cap + PCI_EXP_DPC_SOURCE_ID,
+				     &source);
+		pci_warn(pdev, "containment event, status:%#06x, %s received from %04x:%02x:%02x.%d\n",
+			 status,
+			 (reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_FE) ?
+				"ERR_FATAL" : "ERR_NONFATAL",
+			 pci_domain_nr(pdev->bus), PCI_BUS_NUM(source),
+			 PCI_SLOT(source), PCI_FUNC(source));
+		break;
+	case PCI_EXP_DPC_STATUS_TRIGGER_RSN_IN_EXT:
+		ext_reason = status & PCI_EXP_DPC_STATUS_TRIGGER_RSN_EXT;
+		pci_warn(pdev, "containment event, status:%#06x: %s detected\n",
+			 status,
+			 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO) ?
+			 "RP PIO error" :
+			 (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_SW_TRIGGER) ?

drivers/pci/pcie/dpc.c void dpc_process_error(struct pci_dev *pdev) · line 305

Quoted 10 lines
drivers/pci/pcie/dpc.c +7 -0
+			 "software trigger" :
+			 "reserved error");
+		/* show RP PIO error detail information */
+		if (ext_reason == PCI_EXP_DPC_STATUS_TRIGGER_RSN_RP_PIO &&
+		    pdev->dpc_rp_extensions)
+			dpc_process_rp_pio_error(pdev);
+		break;
 	}
 }
 
Signature · 1 line
 i.
#9 lore raw

PATCH v7 3/17 PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error()

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

aer_isr_one_error() duplicates the Error Source ID logging and AER error processing for Correctable Errors and Uncorrectable Errors. Factor out the duplicated code to aer_isr_one_error_type().

aer_isr_one_error() doesn't need the struct aer_rpc pointer, so pass it the Root Port or RCEC pci_dev pointer instead.

---
 drivers/pci/pcie/aer.c | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +23 -13
index a1cf8c7ef628..568229288ca3 100644
@@ -1273,17 +1273,32 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 }
 
 /**
- * aer_isr_one_error - consume an error detected by Root Port
- * @rpc: pointer to the Root Port which holds an error
+ * aer_isr_one_error_type - consume a Correctable or Uncorrectable Error
+ *			    detected by Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
+ * @info: pointer to AER error info
+ */
+static void aer_isr_one_error_type(struct pci_dev *root,
+				   struct aer_err_info *info)
+{
+	aer_print_port_info(root, info);
+
+	if (find_source_device(root, info))
+		aer_process_err_devices(info);
+}
+
+/**
+ * aer_isr_one_error - consume error(s) signaled by an AER interrupt from
+ *		       Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
  * @e_src: pointer to an error source
  */
-static void aer_isr_one_error(struct aer_rpc *rpc,
+static void aer_isr_one_error(struct pci_dev *root,
 		struct aer_err_source *e_src)
 {
-	struct pci_dev *pdev = rpc->rpd;
 	struct aer_err_info e_info;
 
-	pci_rootport_aer_stats_incr(pdev, e_src);
+	pci_rootport_aer_stats_incr(root, e_src);
 
 	/*
 	 * There is a possibility that both correctable error and
@@ -1297,10 +1312,8 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 			e_info.multi_error_valid = 1;
 		else
 			e_info.multi_error_valid = 0;
-		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 
 	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
@@ -1316,10 +1329,7 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 		else
 			e_info.multi_error_valid = 0;
 
-		aer_print_port_info(pdev, &e_info);
-
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 }
 
@@ -1340,7 +1350,7 @@ static irqreturn_t aer_isr(int irq, void *context)
 		return IRQ_NONE;
 
 	while (kfifo_get(&rpc->aer_fifo, &e_src))
-		aer_isr_one_error(rpc, &e_src);
+		aer_isr_one_error(rpc->rpd, &e_src);
 	return IRQ_HANDLED;
 }
 
Signature · 1 line
2.43.0
Reviewed-by on v7 3/17
#10 lore raw

Re: [PATCH v7 03/17] PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error()

Quoted 11 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

aer_isr_one_error() duplicates the Error Source ID logging and AER error processing for Correctable Errors and Uncorrectable Errors. Factor out the duplicated code to aer_isr_one_error_type().

aer_isr_one_error() doesn't need the struct aer_rpc pointer, so pass it the Root Port or RCEC pci_dev pointer instead.

---

Quoted 8 lines
  drivers/pci/pcie/aer.c | 36 +++++++++++++++++++++++-------------
  1 file changed, 23 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +0 -0
index a1cf8c7ef628..568229288ca3 100644

drivers/pci/pcie/aer.c static inline void aer_process_err_devices(struct aer_err_info *e_info) · line 1276

Quoted 6 lines
drivers/pci/pcie/aer.c +1 -2
@@ -1273,17 +1273,32 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
  }
 
  /**
- * aer_isr_one_error - consume an error detected by Root Port
- * @rpc: pointer to the Root Port which holds an error
+ * aer_isr_one_error_type - consume a Correctable or Uncorrectable Error

/s/cosume/Consume/

Quoted 64 lines
drivers/pci/pcie/aer.c +22 -11
+ *			    detected by Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
+ * @info: pointer to AER error info
+ */
+static void aer_isr_one_error_type(struct pci_dev *root,
+				   struct aer_err_info *info)
+{
+	aer_print_port_info(root, info);
+
+	if (find_source_device(root, info))
+		aer_process_err_devices(info);
+}
+
+/**
+ * aer_isr_one_error - consume error(s) signaled by an AER interrupt from
+ *		       Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
   * @e_src: pointer to an error source
   */
-static void aer_isr_one_error(struct aer_rpc *rpc,
+static void aer_isr_one_error(struct pci_dev *root,
  		struct aer_err_source *e_src)
  {
-	struct pci_dev *pdev = rpc->rpd;
  	struct aer_err_info e_info;
 
-	pci_rootport_aer_stats_incr(pdev, e_src);
+	pci_rootport_aer_stats_incr(root, e_src);
 
  	/*
  	 * There is a possibility that both correctable error and
@@ -1297,10 +1312,8 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
  			e_info.multi_error_valid = 1;
  		else
  			e_info.multi_error_valid = 0;
-		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
  	}
 
  	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
@@ -1316,10 +1329,7 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
  		else
  			e_info.multi_error_valid = 0;
 
-		aer_print_port_info(pdev, &e_info);
-
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
  	}
  }
 
@@ -1340,7 +1350,7 @@ static irqreturn_t aer_isr(int irq, void *context)
  		return IRQ_NONE;
 
  	while (kfifo_get(&rpc->aer_fifo, &e_src))
-		aer_isr_one_error(rpc, &e_src);
+		aer_isr_one_error(rpc->rpd, &e_src);
  	return IRQ_HANDLED;
  }
 
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Reviewed-by on v7 3/17
#11 lore raw

Re: [PATCH v7 03/17] PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error()

In reply to Bjorn Helgaas
Quoted 10 lines · On Tue, 20 May 2025 16:50:20 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

aer_isr_one_error() duplicates the Error Source ID logging and AER error processing for Correctable Errors and Uncorrectable Errors. Factor out the duplicated code to aer_isr_one_error_type().

aer_isr_one_error() doesn't need the struct aer_rpc pointer, so pass it the Root Port or RCEC pci_dev pointer instead.

One passing comment inside (on neighbouring code) Otherwise it is a sensible bit of cleanup.

Quoted 30 lines
---
 drivers/pci/pcie/aer.c | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +16 -2
index a1cf8c7ef628..568229288ca3 100644
@@ -1273,17 +1273,32 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 }
 
 /**
- * aer_isr_one_error - consume an error detected by Root Port
- * @rpc: pointer to the Root Port which holds an error
+ * aer_isr_one_error_type - consume a Correctable or Uncorrectable Error
+ *			    detected by Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
+ * @info: pointer to AER error info
+ */
+static void aer_isr_one_error_type(struct pci_dev *root,
+				   struct aer_err_info *info)
+{
+	aer_print_port_info(root, info);
+
+	if (find_source_device(root, info))
+		aer_process_err_devices(info);
+}
+
+/**
+ * aer_isr_one_error - consume error(s) signaled by an AER interrupt from

drivers/pci/pcie/aer.c static inline void aer_process_err_devices(struct aer_err_info *e_info) · line 1299

Quoted 10 lines
drivers/pci/pcie/aer.c +3 -2
+ *		       Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
  * @e_src: pointer to an error source
  */
-static void aer_isr_one_error(struct aer_rpc *rpc,
+static void aer_isr_one_error(struct pci_dev *root,
 		struct aer_err_source *e_src)
 {
-	struct pci_dev *pdev = rpc->rpd;
 	struct aer_err_info e_info;

I wonder if, in the interests of readability this should be initialized. That would allows some conditions to set only the valid case (ones) rather than explicit zeros.

Quoted 39 lines
drivers/pci/pcie/aer.c +4 -9
-	pci_rootport_aer_stats_incr(pdev, e_src);
+	pci_rootport_aer_stats_incr(root, e_src);
 
 	/*
 	 * There is a possibility that both correctable error and
@@ -1297,10 +1312,8 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 			e_info.multi_error_valid = 1;
 		else
 			e_info.multi_error_valid = 0;
-		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 
 	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
@@ -1316,10 +1329,7 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 		else
 			e_info.multi_error_valid = 0;
 
-		aer_print_port_info(pdev, &e_info);
-
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 }
 
@@ -1340,7 +1350,7 @@ static irqreturn_t aer_isr(int irq, void *context)
 		return IRQ_NONE;
 
 	while (kfifo_get(&rpc->aer_fifo, &e_src))
-		aer_isr_one_error(rpc, &e_src);
+		aer_isr_one_error(rpc->rpd, &e_src);
 	return IRQ_HANDLED;
 }
 
Reviewed-by on v7 3/17
#12 lore raw

Re: [PATCH v7 03/17] PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error()

In reply to Bjorn Helgaas
Quoted 82 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

aer_isr_one_error() duplicates the Error Source ID logging and AER error processing for Correctable Errors and Uncorrectable Errors. Factor out the duplicated code to aer_isr_one_error_type().

aer_isr_one_error() doesn't need the struct aer_rpc pointer, so pass it the Root Port or RCEC pci_dev pointer instead.

---
 drivers/pci/pcie/aer.c | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +22 -12
index a1cf8c7ef628..568229288ca3 100644
@@ -1273,17 +1273,32 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 }
 
 /**
- * aer_isr_one_error - consume an error detected by Root Port
- * @rpc: pointer to the Root Port which holds an error
+ * aer_isr_one_error_type - consume a Correctable or Uncorrectable Error
+ *			    detected by Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
+ * @info: pointer to AER error info
+ */
+static void aer_isr_one_error_type(struct pci_dev *root,
+				   struct aer_err_info *info)
+{
+	aer_print_port_info(root, info);
+
+	if (find_source_device(root, info))
+		aer_process_err_devices(info);
+}
+
+/**
+ * aer_isr_one_error - consume error(s) signaled by an AER interrupt from
+ *		       Root Port or RCEC
+ * @root: pointer to Root Port or RCEC that signaled AER interrupt
  * @e_src: pointer to an error source
  */
-static void aer_isr_one_error(struct aer_rpc *rpc,
+static void aer_isr_one_error(struct pci_dev *root,
 		struct aer_err_source *e_src)
 {
-	struct pci_dev *pdev = rpc->rpd;
 	struct aer_err_info e_info;
 
-	pci_rootport_aer_stats_incr(pdev, e_src);
+	pci_rootport_aer_stats_incr(root, e_src);
 
 	/*
 	 * There is a possibility that both correctable error and
@@ -1297,10 +1312,8 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 			e_info.multi_error_valid = 1;
 		else
 			e_info.multi_error_valid = 0;
-		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 
 	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
@@ -1316,10 +1329,7 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 		else
 			e_info.multi_error_valid = 0;
 
-		aer_print_port_info(pdev, &e_info);
-
-		if (find_source_device(pdev, &e_info))
-			aer_process_err_devices(&e_info);
+		aer_isr_one_error_type(root, &e_info);
 	}
 }
 

drivers/pci/pcie/aer.c static irqreturn_t aer_isr(int irq, void *context) · line 1356

Quoted 8 lines
drivers/pci/pcie/aer.c +1 -1
@@ -1340,7 +1350,7 @@ static irqreturn_t aer_isr(int irq, void *context)
 		return IRQ_NONE;
 
 	while (kfifo_get(&rpc->aer_fifo, &e_src))
-		aer_isr_one_error(rpc, &e_src);
+		aer_isr_one_error(rpc->rpd, &e_src);
 	return IRQ_HANDLED;
 }
 
Signature · 1 line
 i.
#13 lore raw

PATCH v7 4/17 PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Previously we decoded the AER Error Source ID in aer_isr_one_error_type(), then again in find_source_device() if we didn't find any devices with errors logged in their AER Capabilities.

Consolidate this so we only decode and log the Error Source ID once in aer_isr_one_error_type(). Add a "details" parameter so we can add a note when we didn't find any downstream devices with errors logged in their AER Capability.

This changes the dmesg logging when we found no devices with errors logged:

- pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0
- pci 0000:00:01.0: AER: found no error details for 0000:02:00.0
+ pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0 (no details found)

---
 drivers/pci/pcie/aer.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +9 -13
index 568229288ca3..488a6408c7a8 100644
@@ -733,16 +733,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
+static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
+				const char *details)
 {
 	u8 bus = info->id >> 8;
 	u8 devfn = info->id & 0xff;
 
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d\n",
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
 		 info->multi_error_valid ? "Multiple " : "",
 		 aer_error_severity_string[info->severity],
 		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn));
+		 PCI_FUNC(devfn), details);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -926,15 +927,8 @@ static bool find_source_device(struct pci_dev *parent,
 	else
 		pci_walk_bus(parent->subordinate, find_device_iter, e_info);
 
-	if (!e_info->error_dev_num) {
-		u8 bus = e_info->id >> 8;
-		u8 devfn = e_info->id & 0xff;
-
-		pci_info(parent, "found no error details for %04x:%02x:%02x.%d\n",
-			 pci_domain_nr(parent->bus), bus, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn));
+	if (!e_info->error_dev_num)
 		return false;
-	}
 	return true;
 }
 
@@ -1281,9 +1275,11 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 static void aer_isr_one_error_type(struct pci_dev *root,
 				   struct aer_err_info *info)
 {
-	aer_print_port_info(root, info);
+	bool found;
 
-	if (find_source_device(root, info))
+	found = find_source_device(root, info);
+	aer_print_port_info(root, info, found ? "" : " (no details found");
+	if (found)
 		aer_process_err_devices(info);
 }
 
Signature · 1 line
2.43.0
Reviewed-by on v7 4/17
#14 lore raw

Re: [PATCH v7 04/17] PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()

Quoted 11 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Previously we decoded the AER Error Source ID in aer_isr_one_error_type(), then again in find_source_device() if we didn't find any devices with errors logged in their AER Capabilities.

Consolidate this so we only decode and log the Error Source ID once in aer_isr_one_error_type(). Add a "details" parameter so we can add a note when we didn't find any downstream devices with errors logged in their AER Capability.

Quoted 8 lines

This changes the dmesg logging when we found no devices with errors logged:

- pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0
- pci 0000:00:01.0: AER: found no error details for 0000:02:00.0
+ pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0 (no details found)

---

Quoted 59 lines
  drivers/pci/pcie/aer.c | 22 +++++++++-------------
  1 file changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +9 -13
index 568229288ca3..488a6408c7a8 100644
@@ -733,16 +733,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
  			info->severity, info->tlp_header_valid, &info->tlp);
  }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
+static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
+				const char *details)
  {
  	u8 bus = info->id >> 8;
  	u8 devfn = info->id & 0xff;
 
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d\n",
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
  		 info->multi_error_valid ? "Multiple " : "",
  		 aer_error_severity_string[info->severity],
  		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn));
+		 PCI_FUNC(devfn), details);
  }
 
  #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -926,15 +927,8 @@ static bool find_source_device(struct pci_dev *parent,
  	else
  		pci_walk_bus(parent->subordinate, find_device_iter, e_info);
 
-	if (!e_info->error_dev_num) {
-		u8 bus = e_info->id >> 8;
-		u8 devfn = e_info->id & 0xff;
-
-		pci_info(parent, "found no error details for %04x:%02x:%02x.%d\n",
-			 pci_domain_nr(parent->bus), bus, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn));
+	if (!e_info->error_dev_num)
  		return false;
-	}
  	return true;
  }
 
@@ -1281,9 +1275,11 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
  static void aer_isr_one_error_type(struct pci_dev *root,
  				   struct aer_err_info *info)
  {
-	aer_print_port_info(root, info);
+	bool found;
 
-	if (find_source_device(root, info))
+	found = find_source_device(root, info);
+	aer_print_port_info(root, info, found ? "" : " (no details found");
+	if (found)
  		aer_process_err_devices(info);
  }
 
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Reviewed-by on v7 4/17
#15 lore raw

Re: [PATCH v7 04/17] PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()

In reply to Bjorn Helgaas
Quoted 11 lines · On Tue, 20 May 2025 16:50:21 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Previously we decoded the AER Error Source ID in aer_isr_one_error_type(), then again in find_source_device() if we didn't find any devices with errors logged in their AER Capabilities.

Consolidate this so we only decode and log the Error Source ID once in aer_isr_one_error_type(). Add a "details" parameter so we can add a note when we didn't find any downstream devices with errors logged in their AER Capability.

Quoted 7 lines

This changes the dmesg logging when we found no devices with errors logged:

- pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0
- pci 0000:00:01.0: AER: found no error details for 0000:02:00.0
+ pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0 (no details found)

Nice little improvement. I'll assume you reuse details later as otherwise passing a bool and creating the (no details found) in aer_print_port_info() would have been simpler to my eyes as it would have put all the string generation in one place.

Quoted 60 lines
---
 drivers/pci/pcie/aer.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +9 -13
index 568229288ca3..488a6408c7a8 100644
@@ -733,16 +733,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
+static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
+				const char *details)
 {
 	u8 bus = info->id >> 8;
 	u8 devfn = info->id & 0xff;
 
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d\n",
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
 		 info->multi_error_valid ? "Multiple " : "",
 		 aer_error_severity_string[info->severity],
 		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn));
+		 PCI_FUNC(devfn), details);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -926,15 +927,8 @@ static bool find_source_device(struct pci_dev *parent,
 	else
 		pci_walk_bus(parent->subordinate, find_device_iter, e_info);
 
-	if (!e_info->error_dev_num) {
-		u8 bus = e_info->id >> 8;
-		u8 devfn = e_info->id & 0xff;
-
-		pci_info(parent, "found no error details for %04x:%02x:%02x.%d\n",
-			 pci_domain_nr(parent->bus), bus, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn));
+	if (!e_info->error_dev_num)
 		return false;
-	}
 	return true;
 }
 
@@ -1281,9 +1275,11 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 static void aer_isr_one_error_type(struct pci_dev *root,
 				   struct aer_err_info *info)
 {
-	aer_print_port_info(root, info);
+	bool found;
 
-	if (find_source_device(root, info))
+	found = find_source_device(root, info);
+	aer_print_port_info(root, info, found ? "" : " (no details found");
+	if (found)
 		aer_process_err_devices(info);
 }
 
#16 lore raw

Re: [PATCH v7 04/17] PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()

Quoted 20 lines · Jonathan Cameron
Quoted 17 lines · On Tue, 20 May 2025 16:50:21 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Previously we decoded the AER Error Source ID in aer_isr_one_error_type(), then again in find_source_device() if we didn't find any devices with errors logged in their AER Capabilities.

Consolidate this so we only decode and log the Error Source ID once in aer_isr_one_error_type(). Add a "details" parameter so we can add a note when we didn't find any downstream devices with errors logged in their AER Capability.

This changes the dmesg logging when we found no devices with errors logged:

- pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0
- pci 0000:00:01.0: AER: found no error details for 0000:02:00.0
+ pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0 (no details found)

Quoted 7 lines
Quoted 1 line

Nice little improvement. I'll assume you reuse details later as otherwise passing a bool and creating the (no details found) in aer_print_port_info() would have been simpler to my eyes as it would have put all the string generation in one place.

Great idea! Since there's only one caller, I think passing a bool is much nicer.

Quoted 63 lines
Quoted 60 lines
---
 drivers/pci/pcie/aer.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +9 -13
index 568229288ca3..488a6408c7a8 100644
@@ -733,16 +733,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
+static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
+				const char *details)
 {
 	u8 bus = info->id >> 8;
 	u8 devfn = info->id & 0xff;
 
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d\n",
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
 		 info->multi_error_valid ? "Multiple " : "",
 		 aer_error_severity_string[info->severity],
 		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn));
+		 PCI_FUNC(devfn), details);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -926,15 +927,8 @@ static bool find_source_device(struct pci_dev *parent,
 	else
 		pci_walk_bus(parent->subordinate, find_device_iter, e_info);
 
-	if (!e_info->error_dev_num) {
-		u8 bus = e_info->id >> 8;
-		u8 devfn = e_info->id & 0xff;
-
-		pci_info(parent, "found no error details for %04x:%02x:%02x.%d\n",
-			 pci_domain_nr(parent->bus), bus, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn));
+	if (!e_info->error_dev_num)
 		return false;
-	}
 	return true;
 }
 
@@ -1281,9 +1275,11 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 static void aer_isr_one_error_type(struct pci_dev *root,
 				   struct aer_err_info *info)
 {
-	aer_print_port_info(root, info);
+	bool found;
 
-	if (find_source_device(root, info))
+	found = find_source_device(root, info);
+	aer_print_port_info(root, info, found ? "" : " (no details found");
+	if (found)
 		aer_process_err_devices(info);
 }
 
Reviewed-by on v7 4/17
#17 lore raw

Re: [PATCH v7 04/17] PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()

In reply to Bjorn Helgaas
Quoted 69 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Previously we decoded the AER Error Source ID in aer_isr_one_error_type(), then again in find_source_device() if we didn't find any devices with errors logged in their AER Capabilities.

Consolidate this so we only decode and log the Error Source ID once in aer_isr_one_error_type(). Add a "details" parameter so we can add a note when we didn't find any downstream devices with errors logged in their AER Capability.

This changes the dmesg logging when we found no devices with errors logged:

- pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0
- pci 0000:00:01.0: AER: found no error details for 0000:02:00.0
+ pci 0000:00:01.0: AER: Correctable error message received from 0000:02:00.0 (no details found)

---
 drivers/pci/pcie/aer.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)
drivers/pci/pcie/aer.c +5 -11
index 568229288ca3..488a6408c7a8 100644
@@ -733,16 +733,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info)
+static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
+				const char *details)
 {
 	u8 bus = info->id >> 8;
 	u8 devfn = info->id & 0xff;
 
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d\n",
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
 		 info->multi_error_valid ? "Multiple " : "",
 		 aer_error_severity_string[info->severity],
 		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn));
+		 PCI_FUNC(devfn), details);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -926,15 +927,8 @@ static bool find_source_device(struct pci_dev *parent,
 	else
 		pci_walk_bus(parent->subordinate, find_device_iter, e_info);
 
-	if (!e_info->error_dev_num) {
-		u8 bus = e_info->id >> 8;
-		u8 devfn = e_info->id & 0xff;
-
-		pci_info(parent, "found no error details for %04x:%02x:%02x.%d\n",
-			 pci_domain_nr(parent->bus), bus, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn));
+	if (!e_info->error_dev_num)
 		return false;
-	}
 	return true;
 }
 
@@ -1281,9 +1275,11 @@ static inline void aer_process_err_devices(struct aer_err_info *e_info)
 static void aer_isr_one_error_type(struct pci_dev *root,
 				   struct aer_err_info *info)
 {

drivers/pci/pcie/aer.c static inline void aer_process_err_devices(struct aer_err_info *e_info) · line 1285

Quoted 10 lines
drivers/pci/pcie/aer.c +4 -2
-	aer_print_port_info(root, info);
+	bool found;
 
-	if (find_source_device(root, info))
+	found = find_source_device(root, info);
+	aer_print_port_info(root, info, found ? "" : " (no details found");
+	if (found)
 		aer_process_err_devices(info);
 }
 
Signature · 1 line
 i.
#18 lore raw

PATCH v7 5/17 PCI/AER: Extract bus/dev/fn in aer_print_port_info() with PCI_BUS_NUM(), etc

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Use PCI_BUS_NUM(), PCI_SLOT(), PCI_FUNC() to extract the bus number, device, and function number directly from the Error Source ID. There's no need to shift and mask it explicitly.

---
 drivers/pci/pcie/aer.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer.c +3 -4
index 488a6408c7a8..65f9277c822c 100644
@@ -736,14 +736,13 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
 				const char *details)
 {
-	u8 bus = info->id >> 8;
-	u8 devfn = info->id & 0xff;
+	u16 source = info->id;
 
 	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
 		 info->multi_error_valid ? "Multiple " : "",
 		 aer_error_severity_string[info->severity],
-		 pci_domain_nr(dev->bus), bus, PCI_SLOT(devfn),
-		 PCI_FUNC(devfn), details);
+		 pci_domain_nr(dev->bus), PCI_BUS_NUM(source),
+		 PCI_SLOT(source), PCI_FUNC(source), details);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
Signature · 1 line
2.43.0
Reviewed-by on v7 5/17
#19 lore raw

Re: [PATCH v7 05/17] PCI/AER: Extract bus/dev/fn in aer_print_port_info() with PCI_BUS_NUM(), etc

Quoted 10 lines · On Tue, 20 May 2025 16:50:22 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Use PCI_BUS_NUM(), PCI_SLOT(), PCI_FUNC() to extract the bus number, device, and function number directly from the Error Source ID. There's no need to shift and mask it explicitly.

#20 lore raw

PATCH v7 6/17 PCI/AER: Rename aer_print_port_info() to aer_print_source()

In reply to Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Rename aer_print_port_info() to aer_print_source() to be more descriptive. This logs the Error Source ID logged by a Root Port or Root Complex Event Collector when it receives an ERR_COR, ERR_NONFATAL, or ERR_FATAL Message.

---
 drivers/pci/pcie/aer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/pcie/aer.c +3 -3
index 65f9277c822c..4f2527d9a365 100644
@@ -733,8 +733,8 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_port_info(struct pci_dev *dev, struct aer_err_info *info,
-				const char *details)
+static void aer_print_source(struct pci_dev *dev, struct aer_err_info *info,
+			     const char *details)
 {
 	u16 source = info->id;
 
@@ -1277,7 +1277,7 @@ static void aer_isr_one_error_type(struct pci_dev *root,
 	bool found;
 
 	found = find_source_device(root, info);
-	aer_print_port_info(root, info, found ? "" : " (no details found");
+	aer_print_source(root, info, found ? "" : " (no details found");
 	if (found)
 		aer_process_err_devices(info);
 }
Signature · 1 line
2.43.0
Reviewed-by on v7 6/17
#21 lore raw

Re: [PATCH v7 06/17] PCI/AER: Rename aer_print_port_info() to aer_print_source()

Quoted 6 lines · On Tue, 20 May 2025 16:50:23 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Rename aer_print_port_info() to aer_print_source() to be more descriptive. This logs the Error Source ID logged by a Root Port or Root Complex Event Collector when it receives an ERR_COR, ERR_NONFATAL, or ERR_FATAL Message.

Quoted 6 lines

Makes sense.

#22 lore raw

PATCH v7 7/17 PCI/AER: Move aer_print_source() earlier in file

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Move aer_print_source() earlier in the file so a future change can use it from aer_print_error(), where it's easier to rate limit it.

---
 drivers/pci/pcie/aer.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
drivers/pci/pcie/aer.c +12 -12
index 4f2527d9a365..520c21ed4ba9 100644
@@ -696,6 +696,18 @@ static void __aer_print_error(struct pci_dev *dev,
 	pci_dev_aer_stats_incr(dev, info);
 }
 
+static void aer_print_source(struct pci_dev *dev, struct aer_err_info *info,
+			     const char *details)
+{
+	u16 source = info->id;
+
+	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
+		 info->multi_error_valid ? "Multiple " : "",
+		 aer_error_severity_string[info->severity],
+		 pci_domain_nr(dev->bus), PCI_BUS_NUM(source),
+		 PCI_SLOT(source), PCI_FUNC(source), details);
+}
+
 void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	int layer, agent;
@@ -733,18 +745,6 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
-static void aer_print_source(struct pci_dev *dev, struct aer_err_info *info,
-			     const char *details)
-{
-	u16 source = info->id;
-
-	pci_info(dev, "%s%s error message received from %04x:%02x:%02x.%d%s\n",
-		 info->multi_error_valid ? "Multiple " : "",
-		 aer_error_severity_string[info->severity],
-		 pci_domain_nr(dev->bus), PCI_BUS_NUM(source),
-		 PCI_SLOT(source), PCI_FUNC(source), details);
-}
-
 #ifdef CONFIG_ACPI_APEI_PCIEAER
 int cper_severity_to_aer(int cper_severity)
 {
Signature · 1 line
2.43.0
Reviewed-by on v7 7/17
#23 lore raw

Re: [PATCH v7 07/17] PCI/AER: Move aer_print_source() earlier in file

Quoted 9 lines · On Tue, 20 May 2025 16:50:24 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Move aer_print_source() earlier in the file so a future change can use it from aer_print_error(), where it's easier to rate limit it.

#24 lore raw

PATCH v7 8/17 PCI/AER: Initialize aer_err_info before using it

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "e_info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Initialize "e_info" at declaration with a designated initializer list, which initializes the other members to zero.

---
 drivers/pci/pcie/aer.c | 39 +++++++++++++++++----------------------
 1 file changed, 17 insertions(+), 22 deletions(-)
drivers/pci/pcie/aer.c +17 -22
index 520c21ed4ba9..e6693f910a23 100644
@@ -1289,9 +1289,9 @@ static void aer_isr_one_error_type(struct pci_dev *root,
  * @e_src: pointer to an error source
  */
 static void aer_isr_one_error(struct pci_dev *root,
-		struct aer_err_source *e_src)
+			      struct aer_err_source *e_src)
 {
-	struct aer_err_info e_info;
+	u32 status = e_src->status;
 
 	pci_rootport_aer_stats_incr(root, e_src);
 
@@ -1299,30 +1299,25 @@ static void aer_isr_one_error(struct pci_dev *root,
 	 * There is a possibility that both correctable error and
 	 * uncorrectable error being logged. Report correctable error first.
 	 */
-	if (e_src->status & PCI_ERR_ROOT_COR_RCV) {
-		e_info.id = ERR_COR_ID(e_src->id);
-		e_info.severity = AER_CORRECTABLE;
-
-		if (e_src->status & PCI_ERR_ROOT_MULTI_COR_RCV)
-			e_info.multi_error_valid = 1;
-		else
-			e_info.multi_error_valid = 0;
+	if (status & PCI_ERR_ROOT_COR_RCV) {
+		int multi = status & PCI_ERR_ROOT_MULTI_COR_RCV;
+		struct aer_err_info e_info = {
+			.id = ERR_COR_ID(e_src->id),
+			.severity = AER_CORRECTABLE,
+			.multi_error_valid = multi ? 1 : 0,
+		};
 
 		aer_isr_one_error_type(root, &e_info);
 	}
 
-	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
-		e_info.id = ERR_UNCOR_ID(e_src->id);
-
-		if (e_src->status & PCI_ERR_ROOT_FATAL_RCV)
-			e_info.severity = AER_FATAL;
-		else
-			e_info.severity = AER_NONFATAL;
-
-		if (e_src->status & PCI_ERR_ROOT_MULTI_UNCOR_RCV)
-			e_info.multi_error_valid = 1;
-		else
-			e_info.multi_error_valid = 0;
+	if (status & PCI_ERR_ROOT_UNCOR_RCV) {
+		int fatal = status & PCI_ERR_ROOT_FATAL_RCV;
+		int multi = status & PCI_ERR_ROOT_MULTI_UNCOR_RCV;
+		struct aer_err_info e_info = {
+			.id = ERR_UNCOR_ID(e_src->id),
+			.severity = fatal ? AER_FATAL : AER_NONFATAL,
+			.multi_error_valid = multi ? 1 : 0,
+		};
 
 		aer_isr_one_error_type(root, &e_info);
 	}
Signature · 1 line
2.43.0
Reviewed-by on v7 8/17
#25 lore raw

Re: [PATCH v7 08/17] PCI/AER: Initialize aer_err_info before using it

Quoted 6 lines · On Tue, 20 May 2025 16:50:25 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Previously the struct aer_err_info "e_info" was allocated on the stack without being initialized, so it contained junk except for the fields we explicitly set later.

Quoted 7 lines

Initialize "e_info" at declaration with a designated initializer list, which initializes the other members to zero.

Ah. Here it is :) Ignore earlier comment.

Quoted 65 lines
 drivers/pci/pcie/aer.c | 39 +++++++++++++++++----------------------
 1 file changed, 17 insertions(+), 22 deletions(-)
drivers/pci/pcie/aer.c +17 -22
index 520c21ed4ba9..e6693f910a23 100644
@@ -1289,9 +1289,9 @@ static void aer_isr_one_error_type(struct pci_dev *root,
  * @e_src: pointer to an error source
  */
 static void aer_isr_one_error(struct pci_dev *root,
-		struct aer_err_source *e_src)
+			      struct aer_err_source *e_src)
 {
-	struct aer_err_info e_info;
+	u32 status = e_src->status;
 
 	pci_rootport_aer_stats_incr(root, e_src);
 
@@ -1299,30 +1299,25 @@ static void aer_isr_one_error(struct pci_dev *root,
 	 * There is a possibility that both correctable error and
 	 * uncorrectable error being logged. Report correctable error first.
 	 */
-	if (e_src->status & PCI_ERR_ROOT_COR_RCV) {
-		e_info.id = ERR_COR_ID(e_src->id);
-		e_info.severity = AER_CORRECTABLE;
-
-		if (e_src->status & PCI_ERR_ROOT_MULTI_COR_RCV)
-			e_info.multi_error_valid = 1;
-		else
-			e_info.multi_error_valid = 0;
+	if (status & PCI_ERR_ROOT_COR_RCV) {
+		int multi = status & PCI_ERR_ROOT_MULTI_COR_RCV;
+		struct aer_err_info e_info = {
+			.id = ERR_COR_ID(e_src->id),
+			.severity = AER_CORRECTABLE,
+			.multi_error_valid = multi ? 1 : 0,
+		};
 
 		aer_isr_one_error_type(root, &e_info);
 	}
 
-	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
-		e_info.id = ERR_UNCOR_ID(e_src->id);
-
-		if (e_src->status & PCI_ERR_ROOT_FATAL_RCV)
-			e_info.severity = AER_FATAL;
-		else
-			e_info.severity = AER_NONFATAL;
-
-		if (e_src->status & PCI_ERR_ROOT_MULTI_UNCOR_RCV)
-			e_info.multi_error_valid = 1;
-		else
-			e_info.multi_error_valid = 0;
+	if (status & PCI_ERR_ROOT_UNCOR_RCV) {
+		int fatal = status & PCI_ERR_ROOT_FATAL_RCV;
+		int multi = status & PCI_ERR_ROOT_MULTI_UNCOR_RCV;
+		struct aer_err_info e_info = {
+			.id = ERR_UNCOR_ID(e_src->id),
+			.severity = fatal ? AER_FATAL : AER_NONFATAL,
+			.multi_error_valid = multi ? 1 : 0,
+		};
 
 		aer_isr_one_error_type(root, &e_info);
 	}
#26 lore raw

PATCH v7 9/17 PCI/AER: Simplify pci_print_aer()

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Simplify pci_print_aer() by initializing the struct aer_err_info "info" with a designated initializer list (it was previously initialized with memset()) and using pci_name().

---
 drivers/pci/pcie/aer.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/pci/pcie/aer.c +8 -8
index e6693f910a23..d845079429f0 100644
@@ -765,7 +765,10 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 {
 	int layer, agent, tlp_header_valid = 0;
 	u32 status, mask;
-	struct aer_err_info info;
+	struct aer_err_info info = {
+		.severity = aer_severity,
+		.first_error = PCI_ERR_CAP_FEP(aer->cap_control),
+	};
 
 	if (aer_severity == AER_CORRECTABLE) {
 		status = aer->cor_status;
@@ -776,14 +779,11 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 		tlp_header_valid = status & AER_LOG_TLP_MASKS;
 	}
 
-	layer = AER_GET_LAYER_ERROR(aer_severity, status);
-	agent = AER_GET_AGENT(aer_severity, status);
-
-	memset(&info, 0, sizeof(info));
-	info.severity = aer_severity;
 	info.status = status;
 	info.mask = mask;
-	info.first_error = PCI_ERR_CAP_FEP(aer->cap_control);
+
+	layer = AER_GET_LAYER_ERROR(aer_severity, status);
+	agent = AER_GET_AGENT(aer_severity, status);
 
 	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
 	__aer_print_error(dev, &info);
@@ -797,7 +797,7 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
 
-	trace_aer_event(dev_name(&dev->dev), (status & ~mask),
+	trace_aer_event(pci_name(dev), (status & ~mask),
 			aer_severity, tlp_header_valid, &aer->header_log);
 }
 EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
Signature · 1 line
2.43.0
Reviewed-by on v7 9/17
#27 lore raw

Re: [PATCH v7 09/17] PCI/AER: Simplify pci_print_aer()

Quoted 10 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Simplify pci_print_aer() by initializing the struct aer_err_info "info" with a designated initializer list (it was previously initialized with memset()) and using pci_name().

---

Quoted 46 lines
  drivers/pci/pcie/aer.c | 16 ++++++++--------
  1 file changed, 8 insertions(+), 8 deletions(-)
drivers/pci/pcie/aer.c +8 -8
index e6693f910a23..d845079429f0 100644
@@ -765,7 +765,10 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
  {
  	int layer, agent, tlp_header_valid = 0;
  	u32 status, mask;
-	struct aer_err_info info;
+	struct aer_err_info info = {
+		.severity = aer_severity,
+		.first_error = PCI_ERR_CAP_FEP(aer->cap_control),
+	};
 
  	if (aer_severity == AER_CORRECTABLE) {
  		status = aer->cor_status;
@@ -776,14 +779,11 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
  		tlp_header_valid = status & AER_LOG_TLP_MASKS;
  	}
 
-	layer = AER_GET_LAYER_ERROR(aer_severity, status);
-	agent = AER_GET_AGENT(aer_severity, status);
-
-	memset(&info, 0, sizeof(info));
-	info.severity = aer_severity;
  	info.status = status;
  	info.mask = mask;
-	info.first_error = PCI_ERR_CAP_FEP(aer->cap_control);
+
+	layer = AER_GET_LAYER_ERROR(aer_severity, status);
+	agent = AER_GET_AGENT(aer_severity, status);
 
  	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
  	__aer_print_error(dev, &info);
@@ -797,7 +797,7 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
  	if (tlp_header_valid)
  		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
 
-	trace_aer_event(dev_name(&dev->dev), (status & ~mask),
+	trace_aer_event(pci_name(dev), (status & ~mask),
  			aer_severity, tlp_header_valid, &aer->header_log);
  }
  EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Reviewed-by on v7 9/17
#28 lore raw

Re: [PATCH v7 09/17] PCI/AER: Simplify pci_print_aer()

In reply to Bjorn Helgaas
Quoted 9 lines · On Tue, 20 May 2025 16:50:26 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

Simplify pci_print_aer() by initializing the struct aer_err_info "info" with a designated initializer list (it was previously initialized with memset()) and using pci_name().

Another nice cleanup.

#29 lore raw

PATCH v7 10/17 PCI/AER: Update statistics early in logging

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

There are two AER logging entry points:

- aer_print_error() is used by DPC (dpc_process_error()) and native AER handling (aer_process_err_devices()).

- pci_print_aer() is used by GHES (aer_recover_work_func()) and CXL (cxl_handle_rdport_errors())

Both use __aer_print_error() to print the AER error bits. Previously __aer_print_error() also incremented the AER statistics via pci_dev_aer_stats_incr().

Call pci_dev_aer_stats_incr() early in the entry points instead of in __aer_print_error() so we update the statistics even if the actual printing of error bits is rate limited by a future change.

---
 drivers/pci/pcie/aer.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +4 -1
index d845079429f0..53b7559564a9 100644
@@ -693,7 +693,6 @@ static void __aer_print_error(struct pci_dev *dev,
 		aer_printk(level, dev, "   [%2d] %-22s%s\n", i, errmsg,
 				info->first_error == i ? " (First)" : "");
 	}
-	pci_dev_aer_stats_incr(dev, info);
 }
 
 static void aer_print_source(struct pci_dev *dev, struct aer_err_info *info,
@@ -714,6 +713,8 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 	int id = pci_dev_id(dev);
 	const char *level;
 
+	pci_dev_aer_stats_incr(dev, info);
+
 	if (!info->status) {
 		pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
 			aer_error_severity_string[info->severity]);
@@ -782,6 +783,8 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	info.status = status;
 	info.mask = mask;
 
+	pci_dev_aer_stats_incr(dev, &info);
+
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
Signature · 1 line
2.43.0
Reviewed-by on v7 10/17
#30 lore raw

Re: [PATCH v7 10/17] PCI/AER: Update statistics early in logging

Quoted 14 lines · On Tue, 20 May 2025 16:50:27 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

There are two AER logging entry points:

- aer_print_error() is used by DPC (dpc_process_error()) and native AER handling (aer_process_err_devices()).

- pci_print_aer() is used by GHES (aer_recover_work_func()) and CXL (cxl_handle_rdport_errors())

Both use __aer_print_error() to print the AER error bits. Previously __aer_print_error() also incremented the AER statistics via pci_dev_aer_stats_incr().

Quoted 8 lines

Call pci_dev_aer_stats_incr() early in the entry points instead of in __aer_print_error() so we update the statistics even if the actual printing of error bits is rate limited by a future change.

Always felt odd that a stat got updated in a _print_ function so even without the other reasoning this is a good change.

Quoted 34 lines
---
 drivers/pci/pcie/aer.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/pci/pcie/aer.c +4 -1
index d845079429f0..53b7559564a9 100644
@@ -693,7 +693,6 @@ static void __aer_print_error(struct pci_dev *dev,
 		aer_printk(level, dev, "   [%2d] %-22s%s\n", i, errmsg,
 				info->first_error == i ? " (First)" : "");
 	}
-	pci_dev_aer_stats_incr(dev, info);
 }
 
 static void aer_print_source(struct pci_dev *dev, struct aer_err_info *info,
@@ -714,6 +713,8 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 	int id = pci_dev_id(dev);
 	const char *level;
 
+	pci_dev_aer_stats_incr(dev, info);
+
 	if (!info->status) {
 		pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
 			aer_error_severity_string[info->severity]);
@@ -782,6 +783,8 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	info.status = status;
 	info.mask = mask;
 
+	pci_dev_aer_stats_incr(dev, &info);
+
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
#31 lore raw

PATCH v7 11/17 PCI/AER: Combine trace_aer_event() with statistics updates

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

As with the AER statistics, we always want to emit trace events, even if the actual dmesg logging is rate limited.

Call trace_aer_event() directly from pci_dev_aer_stats_incr(), where we update the statistics.

---
 drivers/pci/pcie/aer.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/pci/pcie/aer.c +6 -6
index 53b7559564a9..ec63825a808e 100644
@@ -625,6 +625,9 @@ static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 	u64 *counter = NULL;
 	struct aer_stats *aer_stats = pdev->aer_stats;
 
+	trace_aer_event(pci_name(pdev), (info->status & ~info->mask),
+			info->severity, info->tlp_header_valid, &info->tlp);
+
 	if (!aer_stats)
 		return;
 
@@ -741,9 +744,6 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 out:
 	if (info->id && info->error_dev_num > 1 && info->id == id)
 		pci_err(dev, "  Error of this Agent is reported first\n");
-
-	trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
-			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -782,6 +782,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	info.status = status;
 	info.mask = mask;
+	info.tlp_header_valid = tlp_header_valid;
+	if (tlp_header_valid)
+		info.tlp = aer->header_log;
 
 	pci_dev_aer_stats_incr(dev, &info);
 
@@ -799,9 +802,6 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
-
-	trace_aer_event(pci_name(dev), (status & ~mask),
-			aer_severity, tlp_header_valid, &aer->header_log);
 }
 EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
 
Signature · 1 line
2.43.0
#32 lore raw

Re: [PATCH v7 11/17] PCI/AER: Combine trace_aer_event() with statistics updates

Quoted 5 lines · On Tue, 20 May 2025 16:50:28 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

As with the AER statistics, we always want to emit trace events, even if the actual dmesg logging is rate limited.

Quoted 7 lines

Call trace_aer_event() directly from pci_dev_aer_stats_incr(), where we update the statistics.

Hmm. This runs a bit counter to what I liked in previous patch. Whilst convenient to issue trace points in the stats update function it's not obvious behavior given naming. Maybe just duplicate the call and call it immediately after the pci_dev_aer_stats_incr() calls?

Quoted 48 lines
---
 drivers/pci/pcie/aer.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/pci/pcie/aer.c +6 -6
index 53b7559564a9..ec63825a808e 100644
@@ -625,6 +625,9 @@ static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 	u64 *counter = NULL;
 	struct aer_stats *aer_stats = pdev->aer_stats;
 
+	trace_aer_event(pci_name(pdev), (info->status & ~info->mask),
+			info->severity, info->tlp_header_valid, &info->tlp);
+
 	if (!aer_stats)
 		return;
 
@@ -741,9 +744,6 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 out:
 	if (info->id && info->error_dev_num > 1 && info->id == id)
 		pci_err(dev, "  Error of this Agent is reported first\n");
-
-	trace_aer_event(dev_name(&dev->dev), (info->status & ~info->mask),
-			info->severity, info->tlp_header_valid, &info->tlp);
 }
 
 #ifdef CONFIG_ACPI_APEI_PCIEAER
@@ -782,6 +782,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	info.status = status;
 	info.mask = mask;
+	info.tlp_header_valid = tlp_header_valid;
+	if (tlp_header_valid)
+		info.tlp = aer->header_log;
 
 	pci_dev_aer_stats_incr(dev, &info);
 
@@ -799,9 +802,6 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
-
-	trace_aer_event(pci_name(dev), (status & ~mask),
-			aer_severity, tlp_header_valid, &aer->header_log);
 }
 EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
 
#33 lore raw

Re: [PATCH v7 11/17] PCI/AER: Combine trace_aer_event() with statistics updates

Quoted 16 lines · Jonathan Cameron
Quoted 5 lines · On Tue, 20 May 2025 16:50:28 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Bjorn Helgaas <[email protected]>

As with the AER statistics, we always want to emit trace events, even if the actual dmesg logging is rate limited.

Quoted 7 lines

Call trace_aer_event() directly from pci_dev_aer_stats_incr(), where we update the statistics.

Quoted 4 lines

Hmm. This runs a bit counter to what I liked in previous patch. Whilst convenient to issue trace points in the stats update function it's not obvious behavior given naming. Maybe just duplicate the call and call it immediately after the pci_dev_aer_stats_incr() calls?

Good point, thanks. I made this change, which also means we don't need to copy the header log into "info" because we can just use aer->header_log as before:

Quoted 21 lines
Quoted 20 lines
drivers/pci/pcie/aer.c +3 -3
@@ -782,6 +782,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	info.status = status;
 	info.mask = mask;
+	info.tlp_header_valid = tlp_header_valid;
+	if (tlp_header_valid)
+		info.tlp = aer->header_log;
 
 	pci_dev_aer_stats_incr(dev, &info);
 
@@ -799,9 +802,6 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
-
-	trace_aer_event(pci_name(dev), (status & ~mask),
-			aer_severity, tlp_header_valid, &aer->header_log);
 }
 EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
 
#34 lore raw

PATCH v7 12/17 PCI/AER: Check log level once and remember it

In reply to Bjorn Helgaas

From: Karolina Stolarek <[email protected]>

When reporting an AER error, we check its type multiple times to determine the log level for each message. Do this check only in the top-level functions (aer_isr_one_error(), pci_print_aer()) and save the level in struct aer_err_info.

[bhelgaas: save log level in struct aer_err_info instead of passing it as a parameter]

---
 drivers/pci/pci.h      |  1 +
 drivers/pci/pcie/aer.c | 21 ++++++++++-----------
 drivers/pci/pcie/dpc.c |  1 +
 3 files changed, 12 insertions(+), 11 deletions(-)
drivers/pci/pci.h +1 -0
index b81e99cd4b62..705f9ef58acc 100644
@@ -588,6 +588,7 @@ static inline bool pci_dev_test_and_set_removed(struct pci_dev *dev)
 struct aer_err_info {
 	struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES];
 	int error_dev_num;
+	const char *level;		/* printk level */
 
 	unsigned int id:16;
 
drivers/pci/pcie/aer.c +10 -11
index ec63825a808e..f5e9961d2c63 100644
@@ -672,21 +672,18 @@ static void pci_rootport_aer_stats_incr(struct pci_dev *pdev,
 	}
 }
 
-static void __aer_print_error(struct pci_dev *dev,
-			      struct aer_err_info *info)
+static void __aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	const char **strings;
 	unsigned long status = info->status & ~info->mask;
-	const char *level, *errmsg;
+	const char *level = info->level;
+	const char *errmsg;
 	int i;
 
-	if (info->severity == AER_CORRECTABLE) {
+	if (info->severity == AER_CORRECTABLE)
 		strings = aer_correctable_error_string;
-		level = KERN_WARNING;
-	} else {
+	else
 		strings = aer_uncorrectable_error_string;
-		level = KERN_ERR;
-	}
 
 	for_each_set_bit(i, &status, 32) {
 		errmsg = strings[i];
@@ -714,7 +711,7 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	int layer, agent;
 	int id = pci_dev_id(dev);
-	const char *level;
+	const char *level = info->level;
 
 	pci_dev_aer_stats_incr(dev, info);
 
@@ -727,8 +724,6 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 	layer = AER_GET_LAYER_ERROR(info->severity, info->status);
 	agent = AER_GET_AGENT(info->severity, info->status);
 
-	level = (info->severity == AER_CORRECTABLE) ? KERN_WARNING : KERN_ERR;
-
 	aer_printk(level, dev, "PCIe Bus Error: severity=%s, type=%s, (%s)\n",
 		   aer_error_severity_string[info->severity],
 		   aer_error_layer[layer], aer_agent_string[agent]);
@@ -774,9 +769,11 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	if (aer_severity == AER_CORRECTABLE) {
 		status = aer->cor_status;
 		mask = aer->cor_mask;
+		info.level = KERN_WARNING;
 	} else {
 		status = aer->uncor_status;
 		mask = aer->uncor_mask;
+		info.level = KERN_ERR;
 		tlp_header_valid = status & AER_LOG_TLP_MASKS;
 	}
 
@@ -1307,6 +1304,7 @@ static void aer_isr_one_error(struct pci_dev *root,
 		struct aer_err_info e_info = {
 			.id = ERR_COR_ID(e_src->id),
 			.severity = AER_CORRECTABLE,
+			.level = KERN_WARNING,
 			.multi_error_valid = multi ? 1 : 0,
 		};
 
@@ -1319,6 +1317,7 @@ static void aer_isr_one_error(struct pci_dev *root,
 		struct aer_err_info e_info = {
 			.id = ERR_UNCOR_ID(e_src->id),
 			.severity = fatal ? AER_FATAL : AER_NONFATAL,
+			.level = KERN_ERR,
 			.multi_error_valid = multi ? 1 : 0,
 		};
 
drivers/pci/pcie/dpc.c +1 -0
index 9d85f1b3b761..6c98fabdba57 100644
@@ -252,6 +252,7 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 	else
 		info->severity = AER_NONFATAL;
 
+	info->level = KERN_ERR;
 	return 1;
 }
 
Signature · 1 line
2.43.0
Reviewed-by on v7 12/17
#35 lore raw

Re: [PATCH v7 12/17] PCI/AER: Check log level once and remember it

Quoted 7 lines · On Tue, 20 May 2025 16:50:29 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Karolina Stolarek <[email protected]>

When reporting an AER error, we check its type multiple times to determine the log level for each message. Do this check only in the top-level functions (aer_isr_one_error(), pci_print_aer()) and save the level in struct aer_err_info.

Quoted 7 lines

[bhelgaas: save log level in struct aer_err_info instead of passing it as a parameter]

#36 lore raw

PATCH v7 13/17 PCI/AER: Make all pci_print_aer() log levels depend on error type

In reply to Bjorn Helgaas

From: Karolina Stolarek <[email protected]>

Some existing logs in pci_print_aer() log with error severity by default. Convert them to depend on error type (consistent with rest of AER logging).

---
 drivers/pci/pcie/aer.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/pci/pcie/aer.c +11 -5
index f5e9961d2c63..4cdcf0ebd86d 100644
@@ -788,15 +788,21 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
-	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
+	aer_printk(info.level, dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n",
+		   status, mask);
 	__aer_print_error(dev, &info);
-	pci_err(dev, "aer_layer=%s, aer_agent=%s\n",
-		aer_error_layer[layer], aer_agent_string[agent]);
+	aer_printk(info.level, dev, "aer_layer=%s, aer_agent=%s\n",
+		   aer_error_layer[layer], aer_agent_string[agent]);
 
 	if (aer_severity != AER_CORRECTABLE)
-		pci_err(dev, "aer_uncor_severity: 0x%08x\n",
-			aer->uncor_severity);
+		aer_printk(info.level, dev, "aer_uncor_severity: 0x%08x\n",
+			   aer->uncor_severity);
 
+	/*
+	 * pcie_print_tlp_log() uses KERN_ERR, but we only call it when
+	 * tlp_header_valid is set, and info.level is always KERN_ERR in
+	 * that case.
+	 */
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
 }
Signature · 1 line
2.43.0
Reviewed-by on v7 13/17
#37 lore raw

Re: [PATCH v7 13/17] PCI/AER: Make all pci_print_aer() log levels depend on error type

Quoted 10 lines · On Tue, 20 May 2025 16:50:30 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Karolina Stolarek <[email protected]>

Some existing logs in pci_print_aer() log with error severity by default. Convert them to depend on error type (consistent with rest of AER logging).

One comment inline. It is painful to have a non trivial comment that says we didn't pass a parameter for 'reason X' when maybe it would be simpler to just pass it and not care that it always takes the same value?

Either way this is fine.

Quoted 21 lines
---
 drivers/pci/pcie/aer.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/pci/pcie/aer.c +4 -3
index f5e9961d2c63..4cdcf0ebd86d 100644
@@ -788,15 +788,21 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
-	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
+	aer_printk(info.level, dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n",
+		   status, mask);
 	__aer_print_error(dev, &info);
-	pci_err(dev, "aer_layer=%s, aer_agent=%s\n",
-		aer_error_layer[layer], aer_agent_string[agent]);
+	aer_printk(info.level, dev, "aer_layer=%s, aer_agent=%s\n",
+		   aer_error_layer[layer], aer_agent_string[agent]);
 

drivers/pci/pcie/aer.c void pci_print_aer(struct pci_dev *dev, int aer_severity, · line 804

Quoted 10 lines
drivers/pci/pcie/aer.c +6 -2
 	if (aer_severity != AER_CORRECTABLE)
-		pci_err(dev, "aer_uncor_severity: 0x%08x\n",
-			aer->uncor_severity);
+		aer_printk(info.level, dev, "aer_uncor_severity: 0x%08x\n",
+			   aer->uncor_severity);
 
+	/*
+	 * pcie_print_tlp_log() uses KERN_ERR, but we only call it when
+	 * tlp_header_valid is set, and info.level is always KERN_ERR in
+	 * that case.

I wonder if it's easier to just pass the level in than have the comment?

Quoted 4 lines
drivers/pci/pcie/aer.c +1 -0
+	 */
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
 }
#38 lore raw

Re: [PATCH v7 13/17] PCI/AER: Make all pci_print_aer() log levels depend on error type

Quoted 14 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 20 May 2025 16:50:30 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Karolina Stolarek <[email protected]>

Some existing logs in pci_print_aer() log with error severity by default. Convert them to depend on error type (consistent with rest of AER logging).

Quoted 4 lines

One comment inline. It is painful to have a non trivial comment that says we didn't pass a parameter for 'reason X' when maybe it would be simpler to just pass it and not care that it always takes the same value?

I agree. I added a patch to pass the level to pcie_print_tlp_log().

Quoted 43 lines

Either way this is fine.

Quoted 21 lines
---
 drivers/pci/pcie/aer.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/pci/pcie/aer.c +4 -3
index f5e9961d2c63..4cdcf0ebd86d 100644
@@ -788,15 +788,21 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
-	pci_err(dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n", status, mask);
+	aer_printk(info.level, dev, "aer_status: 0x%08x, aer_mask: 0x%08x\n",
+		   status, mask);
 	__aer_print_error(dev, &info);
-	pci_err(dev, "aer_layer=%s, aer_agent=%s\n",
-		aer_error_layer[layer], aer_agent_string[agent]);
+	aer_printk(info.level, dev, "aer_layer=%s, aer_agent=%s\n",
+		   aer_error_layer[layer], aer_agent_string[agent]);
 

drivers/pci/pcie/aer.c void pci_print_aer(struct pci_dev *dev, int aer_severity, · line 804

Quoted 10 lines
drivers/pci/pcie/aer.c +6 -2
 	if (aer_severity != AER_CORRECTABLE)
-		pci_err(dev, "aer_uncor_severity: 0x%08x\n",
-			aer->uncor_severity);
+		aer_printk(info.level, dev, "aer_uncor_severity: 0x%08x\n",
+			   aer->uncor_severity);
 
+	/*
+	 * pcie_print_tlp_log() uses KERN_ERR, but we only call it when
+	 * tlp_header_valid is set, and info.level is always KERN_ERR in
+	 * that case.

I wonder if it's easier to just pass the level in than have the comment?

Quoted 4 lines
drivers/pci/pcie/aer.c +1 -0
+	 */
 	if (tlp_header_valid)
 		pcie_print_tlp_log(dev, &aer->header_log, dev_fmt("  "));
 }
#39 lore raw

PATCH v7 14/17 PCI/AER: Rename struct aer_stats to aer_info

In reply to Bjorn Helgaas

From: Karolina Stolarek <[email protected]>

Update name to reflect the broader definition of structs/variables that are stored (e.g. ratelimits). This is a preparatory patch for adding rate limit support.

---
 drivers/pci/pcie/aer.c | 46 +++++++++++++++++++++---------------------
 include/linux/pci.h    |  2 +-
 2 files changed, 24 insertions(+), 24 deletions(-)
drivers/pci/pcie/aer.c +23 -23
index 4cdcf0ebd86d..4f1bff0f000f 100644
@@ -54,8 +54,8 @@ struct aer_rpc {
 	DECLARE_KFIFO(aer_fifo, struct aer_err_source, AER_ERROR_SOURCES_MAX);
 };
 
-/* AER stats for the device */
-struct aer_stats {
+/* AER info for the device */
+struct aer_info {
 
 	/*
 	 * Fields for all AER capable devices. They indicate the errors
@@ -377,7 +377,7 @@ void pci_aer_init(struct pci_dev *dev)
 	if (!dev->aer_cap)
 		return;
 
-	dev->aer_stats = kzalloc(sizeof(struct aer_stats), GFP_KERNEL);
+	dev->aer_info = kzalloc(sizeof(*dev->aer_info), GFP_KERNEL);
 
 	/*
 	 * We save/restore PCI_ERR_UNCOR_MASK, PCI_ERR_UNCOR_SEVER,
@@ -398,8 +398,8 @@ void pci_aer_init(struct pci_dev *dev)
 
 void pci_aer_exit(struct pci_dev *dev)
 {
-	kfree(dev->aer_stats);
-	dev->aer_stats = NULL;
+	kfree(dev->aer_info);
+	dev->aer_info = NULL;
 }
 
 #define AER_AGENT_RECEIVER		0
@@ -537,10 +537,10 @@ static const char *aer_agent_string[] = {
 {									\
 	unsigned int i;							\
 	struct pci_dev *pdev = to_pci_dev(dev);				\
-	u64 *stats = pdev->aer_stats->stats_array;			\
+	u64 *stats = pdev->aer_info->stats_array;			\
 	size_t len = 0;							\
 									\
-	for (i = 0; i < ARRAY_SIZE(pdev->aer_stats->stats_array); i++) {\
+	for (i = 0; i < ARRAY_SIZE(pdev->aer_info->stats_array); i++) {	\
 		if (strings_array[i])					\
 			len += sysfs_emit_at(buf, len, "%s %llu\n",	\
 					     strings_array[i],		\
@@ -551,7 +551,7 @@ static const char *aer_agent_string[] = {
 					     i, stats[i]);		\
 	}								\
 	len += sysfs_emit_at(buf, len, "TOTAL_%s %llu\n", total_string,	\
-			     pdev->aer_stats->total_field);		\
+			     pdev->aer_info->total_field);		\
 	return len;							\
 }									\
 static DEVICE_ATTR_RO(name)
@@ -572,7 +572,7 @@ aer_stats_dev_attr(aer_dev_nonfatal, dev_nonfatal_errs,
 		     char *buf)						\
 {									\
 	struct pci_dev *pdev = to_pci_dev(dev);				\
-	return sysfs_emit(buf, "%llu\n", pdev->aer_stats->field);	\
+	return sysfs_emit(buf, "%llu\n", pdev->aer_info->field);	\
 }									\
 static DEVICE_ATTR_RO(name)
 
@@ -599,7 +599,7 @@ static umode_t aer_stats_attrs_are_visible(struct kobject *kobj,
 	struct device *dev = kobj_to_dev(kobj);
 	struct pci_dev *pdev = to_pci_dev(dev);
 
-	if (!pdev->aer_stats)
+	if (!pdev->aer_info)
 		return 0;
 
 	if ((a == &dev_attr_aer_rootport_total_err_cor.attr ||
@@ -623,28 +623,28 @@ static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 	unsigned long status = info->status & ~info->mask;
 	int i, max = -1;
 	u64 *counter = NULL;
-	struct aer_stats *aer_stats = pdev->aer_stats;
+	struct aer_info *aer_info = pdev->aer_info;
 
 	trace_aer_event(pci_name(pdev), (info->status & ~info->mask),
 			info->severity, info->tlp_header_valid, &info->tlp);
 
-	if (!aer_stats)
+	if (!aer_info)
 		return;
 
 	switch (info->severity) {
 	case AER_CORRECTABLE:
-		aer_stats->dev_total_cor_errs++;
-		counter = &aer_stats->dev_cor_errs[0];
+		aer_info->dev_total_cor_errs++;
+		counter = &aer_info->dev_cor_errs[0];
 		max = AER_MAX_TYPEOF_COR_ERRS;
 		break;
 	case AER_NONFATAL:
-		aer_stats->dev_total_nonfatal_errs++;
-		counter = &aer_stats->dev_nonfatal_errs[0];
+		aer_info->dev_total_nonfatal_errs++;
+		counter = &aer_info->dev_nonfatal_errs[0];
 		max = AER_MAX_TYPEOF_UNCOR_ERRS;
 		break;
 	case AER_FATAL:
-		aer_stats->dev_total_fatal_errs++;
-		counter = &aer_stats->dev_fatal_errs[0];
+		aer_info->dev_total_fatal_errs++;
+		counter = &aer_info->dev_fatal_errs[0];
 		max = AER_MAX_TYPEOF_UNCOR_ERRS;
 		break;
 	}
@@ -656,19 +656,19 @@ static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 static void pci_rootport_aer_stats_incr(struct pci_dev *pdev,
 				 struct aer_err_source *e_src)
 {
-	struct aer_stats *aer_stats = pdev->aer_stats;
+	struct aer_info *aer_info = pdev->aer_info;
 
-	if (!aer_stats)
+	if (!aer_info)
 		return;
 
 	if (e_src->status & PCI_ERR_ROOT_COR_RCV)
-		aer_stats->rootport_total_cor_errs++;
+		aer_info->rootport_total_cor_errs++;
 
 	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
 		if (e_src->status & PCI_ERR_ROOT_FATAL_RCV)
-			aer_stats->rootport_total_fatal_errs++;
+			aer_info->rootport_total_fatal_errs++;
 		else
-			aer_stats->rootport_total_nonfatal_errs++;
+			aer_info->rootport_total_nonfatal_errs++;
 	}
 }
 
include/linux/pci.h +1 -1
index 0e8e3fd77e96..81a81dbfc873 100644
@@ -346,7 +346,7 @@ struct pci_dev {
 	u8		hdr_type;	/* PCI header type (`multi' flag masked out) */
 #ifdef CONFIG_PCIEAER
 	u16		aer_cap;	/* AER capability offset */
-	struct aer_stats *aer_stats;	/* AER stats for this device */
+	struct aer_info	*aer_info;	/* AER info for this device */
 #endif
 #ifdef CONFIG_PCIEPORTBUS
 	struct rcec_ea	*rcec_ea;	/* RCEC cached endpoint association */
Signature · 1 line
2.43.0
Reviewed-by on v7 14/17
#40 lore raw

Re: [PATCH v7 14/17] PCI/AER: Rename struct aer_stats to aer_info

Quoted 6 lines · On Tue, 20 May 2025 16:50:31 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Karolina Stolarek <[email protected]>

Update name to reflect the broader definition of structs/variables that are stored (e.g. ratelimits). This is a preparatory patch for adding rate limit support.

Quoted 6 lines
#41 lore raw

PATCH v7 15/17 PCI/AER: Ratelimit correctable and non-fatal error logging

In reply to Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

There are two AER logging entry points:

- aer_print_error() is used by DPC and native AER

- pci_print_aer() is used by GHES and CXL

The native AER aer_print_error() case includes a loop that may log details from multiple devices. This is ratelimited such that we log all the details we find if any of the devices has not hit the ratelimit. If no such device details are found, we still log the Error Source from the ERR_* Message, ratelimited by the Root Port or RCEC that received it.

The DPC aer_print_error() case is not ratelimited, since this only happens for fatal errors.

The CXL pci_print_aer() case is ratelimited by the Error Source device.

The GHES pci_print_aer() case is via aer_recover_work_func(), which searches for the Error Source device. If the device is not found, there's no per-device ratelimit, so we use a system-wide ratelimit that covers all error types (correctable, non-fatal, and fatal).

Sargun at Meta reported internally that a flood of AER errors causes RCU CPU stall warnings and CSD-lock warnings.

Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show true count of 11.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: commit log, factor out trace_aer_event() and aer_print_rp_info() changes to previous patches, collect single aer_err_info.ratelimit as union of ratelimits of all error source devices, don't ratelimit fatal errors, "aer_report" -> "aer_info"]

---
 drivers/pci/pci.h      |  3 +-
 drivers/pci/pcie/aer.c | 66 ++++++++++++++++++++++++++++++++++++++----
 drivers/pci/pcie/dpc.c |  1 +
 3 files changed, 64 insertions(+), 6 deletions(-)
drivers/pci/pci.h +2 -1
index 705f9ef58acc..65c466279ade 100644
@@ -593,7 +593,8 @@ struct aer_err_info {
 	unsigned int id:16;
 
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */
+	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
 	unsigned int first_error:5;
drivers/pci/pcie/aer.c +61 -5

140 lines not on this page · raw message

drivers/pci/pcie/dpc.c +1 -0
index 6c98fabdba57..530c5e2cf7e8 100644
@@ -271,6 +271,7 @@ void dpc_process_error(struct pci_dev *pdev)
 			 status);
 		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
 		    aer_get_device_error_info(pdev, &info)) {
+			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
 			aer_print_error(pdev, &info);
 			pci_aer_clear_nonfatal_status(pdev);
 			pci_aer_clear_fatal_status(pdev);
Signature · 1 line
2.43.0
Reviewed-by on v7 15/17
#42 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

Quoted 38 lines · Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

There are two AER logging entry points:

- aer_print_error() is used by DPC and native AER

- pci_print_aer() is used by GHES and CXL

The native AER aer_print_error() case includes a loop that may log details from multiple devices. This is ratelimited such that we log all the details we find if any of the devices has not hit the ratelimit. If no such device details are found, we still log the Error Source from the ERR_* Message, ratelimited by the Root Port or RCEC that received it.

The DPC aer_print_error() case is not ratelimited, since this only happens for fatal errors.

The CXL pci_print_aer() case is ratelimited by the Error Source device.

The GHES pci_print_aer() case is via aer_recover_work_func(), which searches for the Error Source device. If the device is not found, there's no per-device ratelimit, so we use a system-wide ratelimit that covers all error types (correctable, non-fatal, and fatal).

Sargun at Meta reported internally that a flood of AER errors causes RCU CPU stall warnings and CSD-lock warnings.

Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show true count of 11.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

Quoted 8 lines

[bhelgaas: commit log, factor out trace_aer_event() and aer_print_rp_info() changes to previous patches, collect single aer_err_info.ratelimit as union of ratelimits of all error source devices, don't ratelimit fatal errors, "aer_report" -> "aer_info"]

---

Quoted 32 lines
  drivers/pci/pci.h      |  3 +-
  drivers/pci/pcie/aer.c | 66 ++++++++++++++++++++++++++++++++++++++----
  drivers/pci/pcie/dpc.c |  1 +
  3 files changed, 64 insertions(+), 6 deletions(-)
drivers/pci/pci.h +2 -1
index 705f9ef58acc..65c466279ade 100644
@@ -593,7 +593,8 @@ struct aer_err_info {
  	unsigned int id:16;
 
  	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */
+	unsigned int __pad1:4;
  	unsigned int multi_error_valid:1;
 
  	unsigned int first_error:5;
drivers/pci/pcie/aer.c +1 -0
index 4f1bff0f000f..f9e684ac7878 100644
@@ -28,6 +28,7 @@
  #include <linux/interrupt.h>
  #include <linux/delay.h>
  #include <linux/kfifo.h>
+#include <linux/ratelimit.h>
  #include <linux/slab.h>
  #include <acpi/apei.h>
  #include <acpi/ghes.h>

drivers/pci/pcie/aer.c struct aer_info { · line 95

Quoted 7 lines
drivers/pci/pcie/aer.c +4 -0
@@ -88,6 +89,10 @@ struct aer_info {
  	u64 rootport_total_cor_errs;
  	u64 rootport_total_fatal_errs;
  	u64 rootport_total_nonfatal_errs;
+
+	/* Ratelimits for errors */
+	struct ratelimit_state cor_log_ratelimit;
+	struct ratelimit_state uncor_log_ratelimit;

Nit: Do you think we should name it as nonfatal_log_ratelimit?

Quoted 147 lines
drivers/pci/pcie/aer.c +56 -5
  };
 
  #define AER_LOG_TLP_MASKS		(PCI_ERR_UNC_POISON_TLP|	\
@@ -379,6 +384,11 @@ void pci_aer_init(struct pci_dev *dev)
 
  	dev->aer_info = kzalloc(sizeof(*dev->aer_info), GFP_KERNEL);
 
+	ratelimit_state_init(&dev->aer_info->cor_log_ratelimit,
+			     DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST);
+	ratelimit_state_init(&dev->aer_info->uncor_log_ratelimit,
+			     DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST);
+
  	/*
  	 * We save/restore PCI_ERR_UNCOR_MASK, PCI_ERR_UNCOR_SEVER,
  	 * PCI_ERR_COR_MASK, and PCI_ERR_CAP.  Root and Root Complex Event
@@ -672,6 +682,18 @@ static void pci_rootport_aer_stats_incr(struct pci_dev *pdev,
  	}
  }
 
+static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)
+{
+	struct ratelimit_state *ratelimit;
+
+	if (severity == AER_CORRECTABLE)
+		ratelimit = &dev->aer_info->cor_log_ratelimit;
+	else
+		ratelimit = &dev->aer_info->uncor_log_ratelimit;
+
+	return __ratelimit(ratelimit);
+}
+
  static void __aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
  {
  	const char **strings;
@@ -715,6 +737,9 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 
  	pci_dev_aer_stats_incr(dev, info);
 
+	if (!info->ratelimit)
+		return;
+
  	if (!info->status) {
  		pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
  			aer_error_severity_string[info->severity]);
@@ -785,6 +810,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
  	pci_dev_aer_stats_incr(dev, &info);
 
+	if (!aer_ratelimit(dev, info.severity))
+		return;
+
  	layer = AER_GET_LAYER_ERROR(aer_severity, status);
  	agent = AER_GET_AGENT(aer_severity, status);
 
@@ -815,8 +843,19 @@ EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
   */
  static int add_error_device(struct aer_err_info *e_info, struct pci_dev *dev)
  {
+	/*
+	 * Ratelimit AER log messages.  "dev" is either the source
+	 * identified by the root's Error Source ID or it has an unmasked
+	 * error logged in its own AER Capability.  If any of these devices
+	 * has not reached its ratelimit, log messages for all of them.
+	 * Messages are emitted when "e_info->ratelimit" is non-zero.
+	 *
+	 * Note that "e_info->ratelimit" was already initialized to 1 for the
+	 * ERR_FATAL case.
+	 */
  	if (e_info->error_dev_num < AER_MAX_MULTI_ERR_DEVICES) {
  		e_info->dev[e_info->error_dev_num] = pci_dev_get(dev);
+		e_info->ratelimit |= aer_ratelimit(dev, e_info->severity);
  		e_info->error_dev_num++;
  		return 0;
  	}
@@ -914,7 +953,7 @@ static int find_device_iter(struct pci_dev *dev, void *data)
   * e_info->error_dev_num and e_info->dev[], based on the given information.
   */
  static bool find_source_device(struct pci_dev *parent,
-		struct aer_err_info *e_info)
+			       struct aer_err_info *e_info)
  {
  	struct pci_dev *dev = parent;
  	int result;
@@ -1140,9 +1179,10 @@ static void aer_recover_work_func(struct work_struct *work)
  		pdev = pci_get_domain_bus_and_slot(entry.domain, entry.bus,
  						   entry.devfn);
  		if (!pdev) {
-			pr_err("no pci_dev for %04x:%02x:%02x.%x\n",
-			       entry.domain, entry.bus,
-			       PCI_SLOT(entry.devfn), PCI_FUNC(entry.devfn));
+			pr_err_ratelimited("%04x:%02x:%02x.%x: no pci_dev found\n",
+					   entry.domain, entry.bus,
+					   PCI_SLOT(entry.devfn),
+					   PCI_FUNC(entry.devfn));
  			continue;
  		}
  		pci_print_aer(pdev, entry.severity, entry.regs);
@@ -1283,7 +1323,21 @@ static void aer_isr_one_error_type(struct pci_dev *root,
  	bool found;
 
  	found = find_source_device(root, info);
-	aer_print_source(root, info, found ? "" : " (no details found");
+
+	/*
+	 * If we're going to log error messages, we've already set
+	 * "info->ratelimit" to non-zero (which enables printing) because
+	 * this is either an ERR_FATAL or we found a device with an error
+	 * logged in its AER Capability.
+	 *
+	 * If we didn't find the Error Source device, at least log the
+	 * Requester ID from the ERR_* Message received by the Root Port or
+	 * RCEC, ratelimited by the RP or RCEC.
+	 */
+	if (info->ratelimit ||
+	    (!found && aer_ratelimit(root, info->severity)))
+		aer_print_source(root, info, found ? "" : " (no details found");
+
  	if (found)
  		aer_process_err_devices(info);
  }
@@ -1317,12 +1371,14 @@ static void aer_isr_one_error(struct pci_dev *root,
  		aer_isr_one_error_type(root, &e_info);
  	}
 
+	/* Note that messages for ERR_FATAL are never ratelimited */
  	if (status & PCI_ERR_ROOT_UNCOR_RCV) {
  		int fatal = status & PCI_ERR_ROOT_FATAL_RCV;
  		int multi = status & PCI_ERR_ROOT_MULTI_UNCOR_RCV;
  		struct aer_err_info e_info = {
  			.id = ERR_UNCOR_ID(e_src->id),
  			.severity = fatal ? AER_FATAL : AER_NONFATAL,
+			.ratelimit = fatal ? 1 : 0,
  			.level = KERN_ERR,
  			.multi_error_valid = multi ? 1 : 0,
  		};
drivers/pci/pcie/dpc.c +1 -0
index 6c98fabdba57..530c5e2cf7e8 100644
@@ -271,6 +271,7 @@ void dpc_process_error(struct pci_dev *pdev)
  			 status);
  		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
  		    aer_get_device_error_info(pdev, &info)) {
+			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
  			aer_print_error(pdev, &info);
  			pci_aer_clear_nonfatal_status(pdev);
  			pci_aer_clear_fatal_status(pdev);
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#43 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

Quoted 8 lines · Sathyanarayanan Kuppuswamy
Quoted 6 lines · Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

Quoted 5 lines

drivers/pci/pcie/aer.c struct aer_info { · line 95

Quoted 3 lines
drivers/pci/pcie/aer.c +3 -0
+	/* Ratelimits for errors */
+	struct ratelimit_state cor_log_ratelimit;
+	struct ratelimit_state uncor_log_ratelimit;

Nit: Do you think we should name it as nonfatal_log_ratelimit?

Maybe so. We can always change this internal name, so I guess the important part is the sysfs filename ("/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log").

"ratelimit_burst_nonfatal_log" is not quite parallel with "ratelimit_burst_cor_log" the way "ratelimit_burst_uncor_log" is.

But it's definitely true that the underlying PCIe Messages are ERR_COR, ERR_NONFATAL, and ERR_FATAL.

So I think this is more than a nit, and you're right that we should use "cor" and "nonfatal" somehow.

I'll work on that tomorrow.

Reviewed-by on v7 15/17
#44 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

In reply to Bjorn Helgaas
Quoted 38 lines · Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

There are two AER logging entry points:

- aer_print_error() is used by DPC and native AER

- pci_print_aer() is used by GHES and CXL

The native AER aer_print_error() case includes a loop that may log details from multiple devices. This is ratelimited such that we log all the details we find if any of the devices has not hit the ratelimit. If no such device details are found, we still log the Error Source from the ERR_* Message, ratelimited by the Root Port or RCEC that received it.

The DPC aer_print_error() case is not ratelimited, since this only happens for fatal errors.

The CXL pci_print_aer() case is ratelimited by the Error Source device.

The GHES pci_print_aer() case is via aer_recover_work_func(), which searches for the Error Source device. If the device is not found, there's no per-device ratelimit, so we use a system-wide ratelimit that covers all error types (correctable, non-fatal, and fatal).

Sargun at Meta reported internally that a flood of AER errors causes RCU CPU stall warnings and CSD-lock warnings.

Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show true count of 11.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

Quoted 7 lines

[bhelgaas: commit log, factor out trace_aer_event() and aer_print_rp_info() changes to previous patches, collect single aer_err_info.ratelimit as union of ratelimits of all error source devices, don't ratelimit fatal errors, "aer_report" -> "aer_info"]

Signature · 1 line
 i.
Quoted 188 lines
---
 drivers/pci/pci.h      |  3 +-
 drivers/pci/pcie/aer.c | 66 ++++++++++++++++++++++++++++++++++++++----
 drivers/pci/pcie/dpc.c |  1 +
 3 files changed, 64 insertions(+), 6 deletions(-)
drivers/pci/pci.h +2 -1
index 705f9ef58acc..65c466279ade 100644
@@ -593,7 +593,8 @@ struct aer_err_info {
 	unsigned int id:16;
 
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */
+	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
 	unsigned int first_error:5;
drivers/pci/pcie/aer.c +61 -5
index 4f1bff0f000f..f9e684ac7878 100644
@@ -28,6 +28,7 @@
 #include <linux/interrupt.h>
 #include <linux/delay.h>
 #include <linux/kfifo.h>
+#include <linux/ratelimit.h>
 #include <linux/slab.h>
 #include <acpi/apei.h>
 #include <acpi/ghes.h>
@@ -88,6 +89,10 @@ struct aer_info {
 	u64 rootport_total_cor_errs;
 	u64 rootport_total_fatal_errs;
 	u64 rootport_total_nonfatal_errs;
+
+	/* Ratelimits for errors */
+	struct ratelimit_state cor_log_ratelimit;
+	struct ratelimit_state uncor_log_ratelimit;
 };
 
 #define AER_LOG_TLP_MASKS		(PCI_ERR_UNC_POISON_TLP|	\
@@ -379,6 +384,11 @@ void pci_aer_init(struct pci_dev *dev)
 
 	dev->aer_info = kzalloc(sizeof(*dev->aer_info), GFP_KERNEL);
 
+	ratelimit_state_init(&dev->aer_info->cor_log_ratelimit,
+			     DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST);
+	ratelimit_state_init(&dev->aer_info->uncor_log_ratelimit,
+			     DEFAULT_RATELIMIT_INTERVAL, DEFAULT_RATELIMIT_BURST);
+
 	/*
 	 * We save/restore PCI_ERR_UNCOR_MASK, PCI_ERR_UNCOR_SEVER,
 	 * PCI_ERR_COR_MASK, and PCI_ERR_CAP.  Root and Root Complex Event
@@ -672,6 +682,18 @@ static void pci_rootport_aer_stats_incr(struct pci_dev *pdev,
 	}
 }
 
+static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)
+{
+	struct ratelimit_state *ratelimit;
+
+	if (severity == AER_CORRECTABLE)
+		ratelimit = &dev->aer_info->cor_log_ratelimit;
+	else
+		ratelimit = &dev->aer_info->uncor_log_ratelimit;
+
+	return __ratelimit(ratelimit);
+}
+
 static void __aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	const char **strings;
@@ -715,6 +737,9 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 
 	pci_dev_aer_stats_incr(dev, info);
 
+	if (!info->ratelimit)
+		return;
+
 	if (!info->status) {
 		pci_err(dev, "PCIe Bus Error: severity=%s, type=Inaccessible, (Unregistered Agent ID)\n",
 			aer_error_severity_string[info->severity]);
@@ -785,6 +810,9 @@ void pci_print_aer(struct pci_dev *dev, int aer_severity,
 
 	pci_dev_aer_stats_incr(dev, &info);
 
+	if (!aer_ratelimit(dev, info.severity))
+		return;
+
 	layer = AER_GET_LAYER_ERROR(aer_severity, status);
 	agent = AER_GET_AGENT(aer_severity, status);
 
@@ -815,8 +843,19 @@ EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
  */
 static int add_error_device(struct aer_err_info *e_info, struct pci_dev *dev)
 {
+	/*
+	 * Ratelimit AER log messages.  "dev" is either the source
+	 * identified by the root's Error Source ID or it has an unmasked
+	 * error logged in its own AER Capability.  If any of these devices
+	 * has not reached its ratelimit, log messages for all of them.
+	 * Messages are emitted when "e_info->ratelimit" is non-zero.
+	 *
+	 * Note that "e_info->ratelimit" was already initialized to 1 for the
+	 * ERR_FATAL case.
+	 */
 	if (e_info->error_dev_num < AER_MAX_MULTI_ERR_DEVICES) {
 		e_info->dev[e_info->error_dev_num] = pci_dev_get(dev);
+		e_info->ratelimit |= aer_ratelimit(dev, e_info->severity);
 		e_info->error_dev_num++;
 		return 0;
 	}
@@ -914,7 +953,7 @@ static int find_device_iter(struct pci_dev *dev, void *data)
  * e_info->error_dev_num and e_info->dev[], based on the given information.
  */
 static bool find_source_device(struct pci_dev *parent,
-		struct aer_err_info *e_info)
+			       struct aer_err_info *e_info)
 {
 	struct pci_dev *dev = parent;
 	int result;
@@ -1140,9 +1179,10 @@ static void aer_recover_work_func(struct work_struct *work)
 		pdev = pci_get_domain_bus_and_slot(entry.domain, entry.bus,
 						   entry.devfn);
 		if (!pdev) {
-			pr_err("no pci_dev for %04x:%02x:%02x.%x\n",
-			       entry.domain, entry.bus,
-			       PCI_SLOT(entry.devfn), PCI_FUNC(entry.devfn));
+			pr_err_ratelimited("%04x:%02x:%02x.%x: no pci_dev found\n",
+					   entry.domain, entry.bus,
+					   PCI_SLOT(entry.devfn),
+					   PCI_FUNC(entry.devfn));
 			continue;
 		}
 		pci_print_aer(pdev, entry.severity, entry.regs);
@@ -1283,7 +1323,21 @@ static void aer_isr_one_error_type(struct pci_dev *root,
 	bool found;
 
 	found = find_source_device(root, info);
-	aer_print_source(root, info, found ? "" : " (no details found");
+
+	/*
+	 * If we're going to log error messages, we've already set
+	 * "info->ratelimit" to non-zero (which enables printing) because
+	 * this is either an ERR_FATAL or we found a device with an error
+	 * logged in its AER Capability.
+	 *
+	 * If we didn't find the Error Source device, at least log the
+	 * Requester ID from the ERR_* Message received by the Root Port or
+	 * RCEC, ratelimited by the RP or RCEC.
+	 */
+	if (info->ratelimit ||
+	    (!found && aer_ratelimit(root, info->severity)))
+		aer_print_source(root, info, found ? "" : " (no details found");
+
 	if (found)
 		aer_process_err_devices(info);
 }
@@ -1317,12 +1371,14 @@ static void aer_isr_one_error(struct pci_dev *root,
 		aer_isr_one_error_type(root, &e_info);
 	}
 
+	/* Note that messages for ERR_FATAL are never ratelimited */
 	if (status & PCI_ERR_ROOT_UNCOR_RCV) {
 		int fatal = status & PCI_ERR_ROOT_FATAL_RCV;
 		int multi = status & PCI_ERR_ROOT_MULTI_UNCOR_RCV;
 		struct aer_err_info e_info = {
 			.id = ERR_UNCOR_ID(e_src->id),
 			.severity = fatal ? AER_FATAL : AER_NONFATAL,
+			.ratelimit = fatal ? 1 : 0,
 			.level = KERN_ERR,
 			.multi_error_valid = multi ? 1 : 0,
 		};
drivers/pci/pcie/dpc.c +1 -0
index 6c98fabdba57..530c5e2cf7e8 100644
@@ -271,6 +271,7 @@ void dpc_process_error(struct pci_dev *pdev)
 			 status);
 		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
 		    aer_get_device_error_info(pdev, &info)) {
+			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
 			aer_print_error(pdev, &info);
 			pci_aer_clear_nonfatal_status(pdev);
 			pci_aer_clear_fatal_status(pdev);
#45 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

In reply to Bjorn Helgaas
Quoted 6 lines · On Tue, 20 May 2025 16:50:32 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

See below. I'm not sure that logging of fatal error should affect the rate for non fatal errors + the rate limit infrastructure kind of assumes that you only call it if you are planning to respect it's decision.

Given overall aim is to restrict rates, maybe we don't care if we sometimes throttle earlier that we might expect with a simpler separation of what is being limited.

I don't mind strongly either way.

Quoted 50 lines

There are two AER logging entry points:

- aer_print_error() is used by DPC and native AER

- pci_print_aer() is used by GHES and CXL

The native AER aer_print_error() case includes a loop that may log details from multiple devices. This is ratelimited such that we log all the details we find if any of the devices has not hit the ratelimit. If no such device details are found, we still log the Error Source from the ERR_* Message, ratelimited by the Root Port or RCEC that received it.

The DPC aer_print_error() case is not ratelimited, since this only happens for fatal errors.

The CXL pci_print_aer() case is ratelimited by the Error Source device.

The GHES pci_print_aer() case is via aer_recover_work_func(), which searches for the Error Source device. If the device is not found, there's no per-device ratelimit, so we use a system-wide ratelimit that covers all error types (correctable, non-fatal, and fatal).

Sargun at Meta reported internally that a flood of AER errors causes RCU CPU stall warnings and CSD-lock warnings.

Tested using aer-inject[1]. Sent 11 AER errors. Observed 10 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) show true count of 11.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: commit log, factor out trace_aer_event() and aer_print_rp_info() changes to previous patches, collect single aer_err_info.ratelimit as union of ratelimits of all error source devices, don't ratelimit fatal errors, "aer_report" -> "aer_info"]

---
 drivers/pci/pci.h      |  3 +-
 drivers/pci/pcie/aer.c | 66 ++++++++++++++++++++++++++++++++++++++----
 drivers/pci/pcie/dpc.c |  1 +
 3 files changed, 64 insertions(+), 6 deletions(-)
drivers/pci/pci.h +0 -0
index 705f9ef58acc..65c466279ade 100644

drivers/pci/pci.h struct aer_err_info { · line 596

Quoted 5 lines
drivers/pci/pci.h +1 -1
@@ -593,7 +593,8 @@ struct aer_err_info {
 	unsigned int id:16;
 
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */

That naming is less than intuitive. Maybe expand it to ratelimit_print or something like that.

Quoted 16 lines
drivers/pci/pci.h +1 -0
+	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
 	unsigned int first_error:5;
drivers/pci/pcie/aer.c +3 -0
index 4f1bff0f000f..f9e684ac7878 100644
@@ -815,8 +843,19 @@ EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
  */
 static int add_error_device(struct aer_err_info *e_info, struct pci_dev *dev)
 {
+	/*
+	 * Ratelimit AER log messages.  "dev" is either the source
+	 * identified by the root's Error Source ID or it has an unmasked

drivers/pci/pcie/aer.c EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL"); · line 858

Quoted 10 lines
drivers/pci/pcie/aer.c +8 -0
+	 * error logged in its own AER Capability.  If any of these devices
+	 * has not reached its ratelimit, log messages for all of them.
+	 * Messages are emitted when "e_info->ratelimit" is non-zero.
+	 *
+	 * Note that "e_info->ratelimit" was already initialized to 1 for the
+	 * ERR_FATAL case.
+	 */
 	if (e_info->error_dev_num < AER_MAX_MULTI_ERR_DEVICES) {
 		e_info->dev[e_info->error_dev_num] = pci_dev_get(dev);
+		e_info->ratelimit |= aer_ratelimit(dev, e_info->severity);

So this is a little odd. I think it works but there is code inside __ratelimit that I think we should not be calling for that ERROR_FATAL case (whether we should call lots of times for each device isn't obvious either
but maybe that is more valid).

In the event of it already being 1 due to ERROR_FATAL you will falsely trigger a potential print from inside __ratelimit() if we were rate limited and no longer are but only skipped FATAL prints. My concern is that function is kind of assuming it's only called in cases where a rate limit decision is being made and the implementation may change in future).

https://elixir.bootlin.com/linux/v6.14.7/source/lib/ratelimit.c#L56

Maybe,

		if (!info->ratelimit)
			e_info->ratelimit = aer_ratelimit(dev, e_info->severity);

is an alternative option.
That allows a multiplication factor on the rate as all device count for 1.

drivers/pci/pcie/aer.c static int find_device_iter(struct pci_dev *dev, void *data) · line 956

Quoted 9 lines
drivers/pci/pcie/aer.c +1 -1
 		e_info->error_dev_num++;
 		return 0;
 	}
@@ -914,7 +953,7 @@ static int find_device_iter(struct pci_dev *dev, void *data)
  * e_info->error_dev_num and e_info->dev[], based on the given information.
  */
 static bool find_source_device(struct pci_dev *parent,
-		struct aer_err_info *e_info)
+			       struct aer_err_info *e_info)

Unrelated change

Quoted 5 lines
drivers/pci/pcie/aer.c +0 -0
 {
 	struct pci_dev *dev = parent;
 	int result;

pci_aer_clear_fatal_status(pdev);

#46 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

Quoted 23 lines · Jonathan Cameron
Quoted 6 lines · On Tue, 20 May 2025 16:50:32 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

See below. I'm not sure that logging of fatal error should affect the rate for non fatal errors + the rate limit infrastructure kind of assumes that you only call it if you are planning to respect it's decision.

Given overall aim is to restrict rates, maybe we don't care if we sometimes throttle earlier that we might expect with a simpler separation of what is being limited.

I don't mind strongly either way.

Quoted 3 lines
drivers/pci/pci.h +0 -0
@@ -593,7 +593,8 @@ struct aer_err_info {
 	unsigned int id:16;
 
Quoted 6 lines

drivers/pci/pci.h struct aer_err_info { · line 596

Quoted 3 lines
drivers/pci/pci.h +1 -1
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */

That naming is less than intuitive. Maybe expand it to ratelimit_print or something like that.

True, although it does match uses like "if (aer_ratelimit(...))"

I'll try ratelimit_print and see how you like it :)

Quoted 32 lines
Quoted 16 lines
drivers/pci/pci.h +1 -0
+	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
 	unsigned int first_error:5;
drivers/pci/pcie/aer.c +3 -0
index 4f1bff0f000f..f9e684ac7878 100644
@@ -815,8 +843,19 @@ EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
  */
 static int add_error_device(struct aer_err_info *e_info, struct pci_dev *dev)
 {
+	/*
+	 * Ratelimit AER log messages.  "dev" is either the source
+	 * identified by the root's Error Source ID or it has an unmasked

drivers/pci/pcie/aer.c EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL"); · line 858

Quoted 10 lines
drivers/pci/pcie/aer.c +8 -0
+	 * error logged in its own AER Capability.  If any of these devices
+	 * has not reached its ratelimit, log messages for all of them.
+	 * Messages are emitted when "e_info->ratelimit" is non-zero.
+	 *
+	 * Note that "e_info->ratelimit" was already initialized to 1 for the
+	 * ERR_FATAL case.
+	 */
 	if (e_info->error_dev_num < AER_MAX_MULTI_ERR_DEVICES) {
 		e_info->dev[e_info->error_dev_num] = pci_dev_get(dev);
+		e_info->ratelimit |= aer_ratelimit(dev, e_info->severity);

So this is a little odd. I think it works but there is code inside __ratelimit that I think we should not be calling for that ERROR_FATAL case (whether we should call lots of times for each device isn't obvious either but maybe that is more valid).

Quoted 6 lines

In the event of it already being 1 due to ERROR_FATAL you will falsely trigger a potential print from inside __ratelimit() if we were rate limited and no longer are but only skipped FATAL prints. My concern is that function is kind of assuming it's only called in cases where a rate limit decision is being made and the implementation may change in future).

Hmmm. That's pretty subtle, thanks for catching this.

In the light of day, ".ratelimit = fatal ? 1 : 0" looks a bit sketchy. If we want to avoid ratelimiting AER_FATAL, maybe aer_ratelimit() should just return 1 ("print") unconditionally in that case, without calling __ratelimit():

static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)

  {
    struct ratelimit_state *ratelimit;

    if (severity == AER_FATAL)
      return 1;       /* AER_FATAL not ratelimited */

    if (severity == AER_CORRECTABLE)
      ratelimit = &dev->aer_info->cor_log_ratelimit;
    else
      ratelimit = &dev->aer_info->uncor_log_ratelimit;

    return __ratelimit(ratelimit);

}

That still leaves this question of how to deal with info->dev[] when there's more than one entry, which is kind of an annoying case that only happens for the native AER path.

I think it's because for a single AER interrupt from an RP/RCEC, we collect the root info in one struct aer_err_info and scrape all the downstream devices for anything interesting. We visit each downstream device and is_error_source() reads its status register, but we only keep the pci_dev pointer, so aer_get_device_error_info() has to read the status registers *again*. This all seems kind of obtuse.

The point of the OR above in add_error_device() was to try to match up RP/RCEC logging with downstream device logging so they're ratelimited the same. If we ratelimit the Error Source ID based on the RP/RCEC and the details based on the downstream devices individually, they'll get out of sync, so sometimes we'll print an Error Source ID and elide the details and vice versa.

I wanted to make it so that if we log downstream details, we also log the Error Source ID. But maybe we should ratelimit downstream devices individually (instead of doing this weird union) and make the RP/RCEC part more explicit, e.g.,

  add_error_device(...)
  {
    int i = e_info->error_dev_num;
    e_info->dev[i] = pci_dev_get(dev);
    e_info->error_dev_num++;

    if (aer_ratelimit(dev, e_info->severity)) {
      e_info->root_ratelimit_print = 1;
      e_info->ratelimit_print[i] = 1;
    }

}

Quoted 7 lines

https://elixir.bootlin.com/linux/v6.14.7/source/lib/ratelimit.c#L56

Maybe,

		if (!info->ratelimit)
			e_info->ratelimit = aer_ratelimit(dev, e_info->severity);

is an alternative option.
That allows a multiplication factor on the rate as all device count for 1.

#47 lore raw

Re: [PATCH v7 15/17] PCI/AER: Ratelimit correctable and non-fatal error logging

Quoted 94 lines · On Wed, 21 May 2025 17:54:30 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 23 lines · Jonathan Cameron
Quoted 6 lines · On Tue, 20 May 2025 16:50:32 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device ratelimits for AER correctable and non-fatal uncorrectable errors that use the kernel defaults (10 per 5s). Logging of fatal errors is not ratelimited.

See below. I'm not sure that logging of fatal error should affect the rate for non fatal errors + the rate limit infrastructure kind of assumes that you only call it if you are planning to respect it's decision.

Given overall aim is to restrict rates, maybe we don't care if we sometimes throttle earlier that we might expect with a simpler separation of what is being limited.

I don't mind strongly either way.

Quoted 3 lines
drivers/pci/pci.h +0 -0
@@ -593,7 +593,8 @@ struct aer_err_info {
 	unsigned int id:16;
 
Quoted 6 lines

drivers/pci/pci.h struct aer_err_info { · line 596

Quoted 3 lines
drivers/pci/pci.h +1 -1
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int __pad1:5;
+	unsigned int ratelimit:1;	/* 0=skip, 1=print */  

That naming is less than intuitive. Maybe expand it to ratelimit_print or something like that.

True, although it does match uses like "if (aer_ratelimit(...))"

I'll try ratelimit_print and see how you like it :)

Quoted 32 lines
Quoted 16 lines
drivers/pci/pci.h +1 -0
+	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
 	unsigned int first_error:5;
drivers/pci/pcie/aer.c +3 -0
index 4f1bff0f000f..f9e684ac7878 100644
@@ -815,8 +843,19 @@ EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL");
  */
 static int add_error_device(struct aer_err_info *e_info, struct pci_dev *dev)
 {
+	/*
+	 * Ratelimit AER log messages.  "dev" is either the source
+	 * identified by the root's Error Source ID or it has an unmasked

drivers/pci/pcie/aer.c EXPORT_SYMBOL_NS_GPL(pci_print_aer, "CXL"); · line 858

Quoted 10 lines
drivers/pci/pcie/aer.c +8 -0
+	 * error logged in its own AER Capability.  If any of these devices
+	 * has not reached its ratelimit, log messages for all of them.
+	 * Messages are emitted when "e_info->ratelimit" is non-zero.
+	 *
+	 * Note that "e_info->ratelimit" was already initialized to 1 for the
+	 * ERR_FATAL case.
+	 */
 	if (e_info->error_dev_num < AER_MAX_MULTI_ERR_DEVICES) {
 		e_info->dev[e_info->error_dev_num] = pci_dev_get(dev);
+		e_info->ratelimit |= aer_ratelimit(dev, e_info->severity);  

So this is a little odd. I think it works but there is code inside __ratelimit that I think we should not be calling for that ERROR_FATAL case (whether we should call lots of times for each device isn't obvious either but maybe that is more valid).

Quoted 6 lines

In the event of it already being 1 due to ERROR_FATAL you will falsely trigger a potential print from inside __ratelimit() if we were rate limited and no longer are but only skipped FATAL prints. My concern is that function is kind of assuming it's only called in cases where a rate limit decision is being made and the implementation may change in future).

Hmmm. That's pretty subtle, thanks for catching this.

In the light of day, ".ratelimit = fatal ? 1 : 0" looks a bit sketchy. If we want to avoid ratelimiting AER_FATAL, maybe aer_ratelimit() should just return 1 ("print") unconditionally in that case, without calling __ratelimit():

static int aer_ratelimit(struct pci_dev *dev, unsigned int severity)

  {
    struct ratelimit_state *ratelimit;

    if (severity == AER_FATAL)
      return 1;       /* AER_FATAL not ratelimited */

    if (severity == AER_CORRECTABLE)
      ratelimit = &dev->aer_info->cor_log_ratelimit;
    else
      ratelimit = &dev->aer_info->uncor_log_ratelimit;

    return __ratelimit(ratelimit);
Quoted 1 line

}

Neat solution so go with that.

Quoted 28 lines

That still leaves this question of how to deal with info->dev[] when there's more than one entry, which is kind of an annoying case that only happens for the native AER path.

I think it's because for a single AER interrupt from an RP/RCEC, we collect the root info in one struct aer_err_info and scrape all the downstream devices for anything interesting. We visit each downstream device and is_error_source() reads its status register, but we only keep the pci_dev pointer, so aer_get_device_error_info() has to read the status registers *again*. This all seems kind of obtuse.

The point of the OR above in add_error_device() was to try to match up RP/RCEC logging with downstream device logging so they're ratelimited the same. If we ratelimit the Error Source ID based on the RP/RCEC and the details based on the downstream devices individually, they'll get out of sync, so sometimes we'll print an Error Source ID and elide the details and vice versa.

I wanted to make it so that if we log downstream details, we also log the Error Source ID. But maybe we should ratelimit downstream devices individually (instead of doing this weird union) and make the RP/RCEC part more explicit, e.g.,

  add_error_device(...)
  {
    int i = e_info->error_dev_num;
Quoted 8 lines
    e_info->dev[i] = pci_dev_get(dev);
    e_info->error_dev_num++;

    if (aer_ratelimit(dev, e_info->severity)) {
      e_info->root_ratelimit_print = 1;
      e_info->ratelimit_print[i] = 1;
    }

}

As it's a weird corner case, I don't really mind how you handle it. I'm not sure I grasp this last suggestion fully but can look at the full code if you do go with something like that.

Jonathan

Quoted 8 lines
Quoted 8 lines

https://elixir.bootlin.com/linux/v6.14.7/source/lib/ratelimit.c#L56

Maybe,

		if (!info->ratelimit)
			e_info->ratelimit = aer_ratelimit(dev, e_info->severity);

is an alternative option.
That allows a multiplication factor on the rate as all device count for 1.

#48 lore raw

PATCH v7 16/17 PCI/AER: Add ratelimits to PCI AER Documentation

In reply to Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Add ratelimits section for rationale and defaults.

---
 Documentation/PCI/pcieaer-howto.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)
Documentation/PCI/pcieaer-howto.rst +12 -0
index f013f3b27c82..6fb31516fff1 100644
@@ -85,6 +85,18 @@ In the example, 'Requester ID' means the ID of the device that sent
 the error message to the Root Port. Please refer to PCIe specs for other
 fields.
 
+AER Ratelimits
+--------------
+
+Since error messages can be generated for each transaction, we may see
+large volumes of errors reported. To prevent spammy devices from flooding
+the console/stalling execution, messages are throttled by device and error
+type (correctable vs. non-fatal uncorrectable).  Fatal errors, including
+DPC errors, are not ratelimited.
+
+AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
+DEFAULT_RATELIMIT_INTERVAL (5 seconds).
+
 AER Statistics / Counters
 -------------------------
 
Signature · 1 line
2.43.0
#49 lore raw

PATCH v7 17/17 PCI/AER: Add sysfs attributes for log ratelimits

In reply to Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

Update AER sysfs ABI filename to reflect the broader scope of AER sysfs attributes (e.g. stats and ratelimits).

Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats ->
sysfs-bus-pci-devices-aer

Tested using aer-inject[1]. Configured correctable log ratelimit to 5. Sent 6 AER errors. Observed 5 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) shows 6.

Disabled ratelimiting and sent 6 more AER errors. Observed all 6 errors logged and accounted in AER stats (12 total errors).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: note fatal errors are not ratelimited, "aer_report" -> "aer_info"]
---
 ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++++++
 Documentation/PCI/pcieaer-howto.rst           |  5 +-
 drivers/pci/pci-sysfs.c                       |  1 +
 drivers/pci/pci.h                             |  1 +
 drivers/pci/pcie/aer.c                        | 99 +++++++++++++++++++
 5 files changed, 139 insertions(+), 1 deletion(-)
 rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)
Documentation/ABI/testing/sysfs-bus-pci-devices-aer renamed from Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats +34 -0
similarity index 77%
rename from Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
rename to Documentation/ABI/testing/sysfs-bus-pci-devices-aer
index d1f67bb81d5d..01bb577bfee8 100644
@@ -117,3 +117,37 @@ Date:		July 2018
 KernelVersion:	4.19.0
 Contact:	[email protected], [email protected]
 Description:	Total number of ERR_NONFATAL messages reported to rootport.
+
+PCIe AER ratelimits
+-------------------
+
+These attributes show up under all the devices that are AER capable.
+They represent configurable ratelimits of logs per error type.
+
+See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
+		gets whether or not AER ratelimiting is currently enabled.
+		Enabled by default.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for correctable error logs. Writing a value
+		changes the number of errors (burst) allowed per interval
+		(5 second window) before ratelimiting. Reading gets the
+		current ratelimit burst.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for non-fatal uncorrectable error logs.
+		Writing a value changes the number of errors (burst)
+		allowed per interval (5 second window) before ratelimiting.
+		Reading gets the current ratelimit burst.
Documentation/PCI/pcieaer-howto.rst +4 -1
index 6fb31516fff1..4b71e2f43ca7 100644
@@ -97,12 +97,15 @@ DPC errors, are not ratelimited.
 AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
 DEFAULT_RATELIMIT_INTERVAL (5 seconds).
 
+Ratelimits are exposed in the form of sysfs attributes and configurable.
+See Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
+
 AER Statistics / Counters
 -------------------------
 
 When PCIe AER errors are captured, the counters / statistics are also exposed
 in the form of sysfs attributes which are documented at
-Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
+Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
 
 Developer Guide
 ===============
drivers/pci/pci-sysfs.c +1 -0
index c6cda56ca52c..278de99b00ce 100644
@@ -1805,6 +1805,7 @@ const struct attribute_group *pci_dev_attr_groups[] = {
 	&pcie_dev_attr_group,
 #ifdef CONFIG_PCIEAER
 	&aer_stats_attr_group,
+	&aer_attr_group,
 #endif
 #ifdef CONFIG_PCIEASPM
 	&aspm_ctrl_attr_group,
drivers/pci/pci.h +1 -0
index 65c466279ade..a3261e842d6d 100644
@@ -963,6 +963,7 @@ void pci_no_aer(void);
 void pci_aer_init(struct pci_dev *dev);
 void pci_aer_exit(struct pci_dev *dev);
 extern const struct attribute_group aer_stats_attr_group;
+extern const struct attribute_group aer_attr_group;
 void pci_aer_clear_fatal_status(struct pci_dev *dev);
 int pci_aer_clear_status(struct pci_dev *dev);
 int pci_aer_raw_clear_status(struct pci_dev *dev);
drivers/pci/pcie/aer.c +99 -0

105 lines not on this page · raw message

Signature · 1 line
2.43.0
Reviewed-by on v7 17/17
#50 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

Quoted 19 lines · Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

Update AER sysfs ABI filename to reflect the broader scope of AER sysfs attributes (e.g. stats and ratelimits).

   Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats ->
     sysfs-bus-pci-devices-aer

Tested using aer-inject[1]. Configured correctable log ratelimit to 5. Sent 6 AER errors. Observed 5 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) shows 6.

Disabled ratelimiting and sent 6 more AER errors. Observed all 6 errors logged and accounted in AER stats (12 total errors).

Quoted 8 lines

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: note fatal errors are not ratelimited, "aer_report" -> "aer_info"]

---

Quoted 208 lines
  ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++++++
  Documentation/PCI/pcieaer-howto.rst           |  5 +-
  drivers/pci/pci-sysfs.c                       |  1 +
  drivers/pci/pci.h                             |  1 +
  drivers/pci/pcie/aer.c                        | 99 +++++++++++++++++++
  5 files changed, 139 insertions(+), 1 deletion(-)

rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)

Documentation/ABI/testing/sysfs-bus-pci-devices-aer renamed from Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats +34 -0
similarity index 77%
rename from Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
rename to Documentation/ABI/testing/sysfs-bus-pci-devices-aer
index d1f67bb81d5d..01bb577bfee8 100644
@@ -117,3 +117,37 @@ Date:		July 2018
  KernelVersion:	4.19.0
  Contact:	[email protected], [email protected]
  Description:	Total number of ERR_NONFATAL messages reported to rootport.
+
+PCIe AER ratelimits
+-------------------
+
+These attributes show up under all the devices that are AER capable.
+They represent configurable ratelimits of logs per error type.
+
+See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
+		gets whether or not AER ratelimiting is currently enabled.
+		Enabled by default.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for correctable error logs. Writing a value
+		changes the number of errors (burst) allowed per interval
+		(5 second window) before ratelimiting. Reading gets the
+		current ratelimit burst.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for non-fatal uncorrectable error logs.
+		Writing a value changes the number of errors (burst)
+		allowed per interval (5 second window) before ratelimiting.
+		Reading gets the current ratelimit burst.
Documentation/PCI/pcieaer-howto.rst +4 -1
index 6fb31516fff1..4b71e2f43ca7 100644
@@ -97,12 +97,15 @@ DPC errors, are not ratelimited.
  AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
  DEFAULT_RATELIMIT_INTERVAL (5 seconds).
 
+Ratelimits are exposed in the form of sysfs attributes and configurable.
+See Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
+
  AER Statistics / Counters
  -------------------------
 
  When PCIe AER errors are captured, the counters / statistics are also exposed
  in the form of sysfs attributes which are documented at
-Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
+Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
 
  Developer Guide
  ===============
drivers/pci/pci-sysfs.c +1 -0
index c6cda56ca52c..278de99b00ce 100644
@@ -1805,6 +1805,7 @@ const struct attribute_group *pci_dev_attr_groups[] = {
  	&pcie_dev_attr_group,
  #ifdef CONFIG_PCIEAER
  	&aer_stats_attr_group,
+	&aer_attr_group,
  #endif
  #ifdef CONFIG_PCIEASPM
  	&aspm_ctrl_attr_group,
drivers/pci/pci.h +1 -0
index 65c466279ade..a3261e842d6d 100644
@@ -963,6 +963,7 @@ void pci_no_aer(void);
  void pci_aer_init(struct pci_dev *dev);
  void pci_aer_exit(struct pci_dev *dev);
  extern const struct attribute_group aer_stats_attr_group;
+extern const struct attribute_group aer_attr_group;
  void pci_aer_clear_fatal_status(struct pci_dev *dev);
  int pci_aer_clear_status(struct pci_dev *dev);
  int pci_aer_raw_clear_status(struct pci_dev *dev);
drivers/pci/pcie/aer.c +99 -0
index f9e684ac7878..9b8dea317a79 100644
@@ -627,6 +627,105 @@ const struct attribute_group aer_stats_attr_group = {
  	.is_visible = aer_stats_attrs_are_visible,
  };
 
+/*
+ * Ratelimit enable toggle
+ * 0: disabled with ratelimit.interval = 0
+ * 1: enabled with ratelimit.interval = nonzero
+ */
+static ssize_t ratelimit_log_enable_show(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	bool enabled = pdev->aer_info->cor_log_ratelimit.interval != 0;
+
+	return sysfs_emit(buf, "%d\n", enabled);
+}
+
+static ssize_t ratelimit_log_enable_store(struct device *dev,
+					  struct device_attribute *attr,
+					  const char *buf, size_t count)
+{
+	struct pci_dev *pdev = to_pci_dev(dev);
+	bool enable;
+	int interval;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	if (kstrtobool(buf, &enable) < 0)
+		return -EINVAL;
+
+	if (enable)
+		interval = DEFAULT_RATELIMIT_INTERVAL;
+	else
+		interval = 0;
+
+	pdev->aer_info->cor_log_ratelimit.interval = interval;
+	pdev->aer_info->uncor_log_ratelimit.interval = interval;
+
+	return count;
+}
+static DEVICE_ATTR_RW(ratelimit_log_enable);
+
+#define aer_ratelimit_burst_attr(name, ratelimit)			\
+	static ssize_t							\
+	name##_show(struct device *dev, struct device_attribute *attr,	\
+		    char *buf)						\
+{									\
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+									\
+	return sysfs_emit(buf, "%d\n",					\
+			  pdev->aer_info->ratelimit.burst);		\
+}									\
+									\
+	static ssize_t							\
+	name##_store(struct device *dev, struct device_attribute *attr,	\
+		     const char *buf, size_t count)			\
+{									\
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+	int burst;							\
+									\
+	if (!capable(CAP_SYS_ADMIN))					\
+		return -EPERM;						\
+									\
+	if (kstrtoint(buf, 0, &burst) < 0)				\
+		return -EINVAL;						\
+									\
+	pdev->aer_info->ratelimit.burst = burst;			\
+									\
+	return count;							\
+}									\
+static DEVICE_ATTR_RW(name)
+
+aer_ratelimit_burst_attr(ratelimit_burst_cor_log, cor_log_ratelimit);
+aer_ratelimit_burst_attr(ratelimit_burst_uncor_log, uncor_log_ratelimit);
+
+static struct attribute *aer_attrs[] = {
+	&dev_attr_ratelimit_log_enable.attr,
+	&dev_attr_ratelimit_burst_cor_log.attr,
+	&dev_attr_ratelimit_burst_uncor_log.attr,
+	NULL
+};
+
+static umode_t aer_attrs_are_visible(struct kobject *kobj,
+				     struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	if (!pdev->aer_info)
+		return 0;
+
+	return a->mode;
+}
+
+const struct attribute_group aer_attr_group = {
+	.name = "aer",
+	.attrs = aer_attrs,
+	.is_visible = aer_attrs_are_visible,
+};
+
  static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
  				   struct aer_err_info *info)
  {
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#51 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

In reply to Bjorn Helgaas
Quoted 19 lines · On Tue, 20 May 2025 16:50:34 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

Update AER sysfs ABI filename to reflect the broader scope of AER sysfs attributes (e.g. stats and ratelimits).

Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats ->
sysfs-bus-pci-devices-aer

Tested using aer-inject[1]. Configured correctable log ratelimit to 5. Sent 6 AER errors. Observed 5 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) shows 6.

Disabled ratelimiting and sent 6 more AER errors. Observed all 6 errors logged and accounted in AER stats (12 total errors).

Quoted 7 lines

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: note fatal errors are not ratelimited, "aer_report" -> "aer_info"]

There is some relatively new SYSFS infra that I think will help make this slightly nicer by getting rid of the extra directory when there is nothing to be done with it.

Quoted 15 lines
---
 ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++++++
 Documentation/PCI/pcieaer-howto.rst           |  5 +-
 drivers/pci/pci-sysfs.c                       |  1 +
 drivers/pci/pci.h                             |  1 +
 drivers/pci/pcie/aer.c                        | 99 +++++++++++++++++++
 5 files changed, 139 insertions(+), 1 deletion(-)
 rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)
drivers/pci/pcie/aer.c +0 -0
index f9e684ac7878..9b8dea317a79 100644

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 634

Quoted 8 lines
drivers/pci/pcie/aer.c +5 -0
@@ -627,6 +627,105 @@ const struct attribute_group aer_stats_attr_group = {
 	.is_visible = aer_stats_attrs_are_visible,
 };
 
+#define aer_ratelimit_burst_attr(name, ratelimit)			\
+	static ssize_t							\
+	name##_show(struct device *dev, struct device_attribute *attr,	\
+		    char *buf)						\
+{									\

A little odd looking to indent this less than the line above.

Quoted 42 lines
drivers/pci/pcie/aer.c +42 -0
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+									\
+	return sysfs_emit(buf, "%d\n",					\
+			  pdev->aer_info->ratelimit.burst);		\
+}									\
+									\
+	static ssize_t							\
+	name##_store(struct device *dev, struct device_attribute *attr,	\
+		     const char *buf, size_t count)			\
+{									\
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+	int burst;							\
+									\
+	if (!capable(CAP_SYS_ADMIN))					\
+		return -EPERM;						\
+									\
+	if (kstrtoint(buf, 0, &burst) < 0)				\
+		return -EINVAL;						\
+									\
+	pdev->aer_info->ratelimit.burst = burst;			\
+									\
+	return count;							\
+}									\
+static DEVICE_ATTR_RW(name)
+
+aer_ratelimit_burst_attr(ratelimit_burst_cor_log, cor_log_ratelimit);
+aer_ratelimit_burst_attr(ratelimit_burst_uncor_log, uncor_log_ratelimit);
+
+static struct attribute *aer_attrs[] = {
+	&dev_attr_ratelimit_log_enable.attr,
+	&dev_attr_ratelimit_burst_cor_log.attr,
+	&dev_attr_ratelimit_burst_uncor_log.attr,
+	NULL
+};
+
+static umode_t aer_attrs_are_visible(struct kobject *kobj,
+				     struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	if (!pdev->aer_info)

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 727

Quoted 10 lines
drivers/pci/pcie/aer.c +10 -0
+		return 0;
+
+	return a->mode;
+}
+
+const struct attribute_group aer_attr_group = {
+	.name = "aer",
+	.attrs = aer_attrs,
+	.is_visible = aer_attrs_are_visible,
+};

There are a bunch of macros to simplify cases where a whole group is either enabled or not and make the group itself go away if there is nothing to be shown.

DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE() combined with SYSFS_GROUP_VISIBLE() around the assignment does what we want here I think.

Whilst we can't retrofit that stuff onto existing ABI as someone may be assuming directory presence, we can make sysfs less cluttered for new stuff.

Maybe I'm missing why that doesn't work here though!

J

Quoted 4 lines
drivers/pci/pcie/aer.c +1 -0
+
 static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 				   struct aer_err_info *info)
 {
#52 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

Quoted 117 lines · Jonathan Cameron
Quoted 19 lines · On Tue, 20 May 2025 16:50:34 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

Update AER sysfs ABI filename to reflect the broader scope of AER sysfs attributes (e.g. stats and ratelimits).

Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats ->
sysfs-bus-pci-devices-aer

Tested using aer-inject[1]. Configured correctable log ratelimit to 5. Sent 6 AER errors. Observed 5 errors logged while AER stats (cat /sys/bus/pci/devices/<dev>/aer_dev_correctable) shows 6.

Disabled ratelimiting and sent 6 more AER errors. Observed all 6 errors logged and accounted in AER stats (12 total errors).

Quoted 7 lines

[1] https://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git

[bhelgaas: note fatal errors are not ratelimited, "aer_report" -> "aer_info"]

There is some relatively new SYSFS infra that I think will help make this slightly nicer by getting rid of the extra directory when there is nothing to be done with it.

Quoted 15 lines
---
 ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++++++
 Documentation/PCI/pcieaer-howto.rst           |  5 +-
 drivers/pci/pci-sysfs.c                       |  1 +
 drivers/pci/pci.h                             |  1 +
 drivers/pci/pcie/aer.c                        | 99 +++++++++++++++++++
 5 files changed, 139 insertions(+), 1 deletion(-)
 rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)
drivers/pci/pcie/aer.c +0 -0
index f9e684ac7878..9b8dea317a79 100644

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 634

Quoted 8 lines
drivers/pci/pcie/aer.c +5 -0
@@ -627,6 +627,105 @@ const struct attribute_group aer_stats_attr_group = {
 	.is_visible = aer_stats_attrs_are_visible,
 };
 
+#define aer_ratelimit_burst_attr(name, ratelimit)			\
+	static ssize_t							\
+	name##_show(struct device *dev, struct device_attribute *attr,	\
+		    char *buf)						\
+{									\

A little odd looking to indent this less than the line above.

Quoted 42 lines
drivers/pci/pcie/aer.c +42 -0
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+									\
+	return sysfs_emit(buf, "%d\n",					\
+			  pdev->aer_info->ratelimit.burst);		\
+}									\
+									\
+	static ssize_t							\
+	name##_store(struct device *dev, struct device_attribute *attr,	\
+		     const char *buf, size_t count)			\
+{									\
+	struct pci_dev *pdev = to_pci_dev(dev);				\
+	int burst;							\
+									\
+	if (!capable(CAP_SYS_ADMIN))					\
+		return -EPERM;						\
+									\
+	if (kstrtoint(buf, 0, &burst) < 0)				\
+		return -EINVAL;						\
+									\
+	pdev->aer_info->ratelimit.burst = burst;			\
+									\
+	return count;							\
+}									\
+static DEVICE_ATTR_RW(name)
+
+aer_ratelimit_burst_attr(ratelimit_burst_cor_log, cor_log_ratelimit);
+aer_ratelimit_burst_attr(ratelimit_burst_uncor_log, uncor_log_ratelimit);
+
+static struct attribute *aer_attrs[] = {
+	&dev_attr_ratelimit_log_enable.attr,
+	&dev_attr_ratelimit_burst_cor_log.attr,
+	&dev_attr_ratelimit_burst_uncor_log.attr,
+	NULL
+};
+
+static umode_t aer_attrs_are_visible(struct kobject *kobj,
+				     struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	if (!pdev->aer_info)

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 727

Quoted 10 lines
drivers/pci/pcie/aer.c +10 -0
+		return 0;
+
+	return a->mode;
+}
+
+const struct attribute_group aer_attr_group = {
+	.name = "aer",
+	.attrs = aer_attrs,
+	.is_visible = aer_attrs_are_visible,
+};

There are a bunch of macros to simplify cases where a whole group is either enabled or not and make the group itself go away if there is nothing to be shown.

Quoted 6 lines

DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE() combined with SYSFS_GROUP_VISIBLE() around the assignment does what we want here I think.

Whilst we can't retrofit that stuff onto existing ABI as someone may be assuming directory presence,

Are you sure about this? That empty directories are part of ABI as well? Are any of these directories listed under Documentation/ABI ?

I can see somebody could in theory rely on the existance of empty directories but it's not like it contains any real substance without a file with the actual content of interest so it seems somewhat strange to check for directory and not the file of interest itself.

Quoted 11 lines

we can make sysfs less cluttered for new stuff.

Maybe I'm missing why that doesn't work here though!

J

Quoted 4 lines
drivers/pci/pcie/aer.c +1 -0
+
 static void pci_dev_aer_stats_incr(struct pci_dev *pdev,
 				   struct aer_err_info *info)
 {
Signature · 1 line
 i.
#53 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

In reply to Jonathan Cameron
Quoted 14 lines · Jonathan Cameron
Quoted 5 lines · On Tue, 20 May 2025 16:50:34 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

...

There is some relatively new SYSFS infra that I think will help make this slightly nicer by getting rid of the extra directory when there is nothing to be done with it.

Quoted 7 lines

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 675

Quoted 5 lines
drivers/pci/pcie/aer.c +5 -0
+#define aer_ratelimit_burst_attr(name, ratelimit)			\
+	static ssize_t							\
+	name##_show(struct device *dev, struct device_attribute *attr,	\
+		    char *buf)						\
+{									\

A little odd looking to indent this less than the line above.

Yep, fixed.

Quoted 14 lines

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 727

Quoted 5 lines
drivers/pci/pcie/aer.c +5 -0
+const struct attribute_group aer_attr_group = {
+	.name = "aer",
+	.attrs = aer_attrs,
+	.is_visible = aer_attrs_are_visible,
+};

There are a bunch of macros to simplify cases where a whole group is either enabled or not and make the group itself go away if there is nothing to be shown.

DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE() combined with SYSFS_GROUP_VISIBLE() around the assignment does what we want here I think.

Quoted 5 lines

Whilst we can't retrofit that stuff onto existing ABI as someone may be assuming directory presence, we can make sysfs less cluttered for new stuff.

Maybe I'm missing why that doesn't work here though!

Is this something we can fix later, or are we locking ourselves into user-visible ABI that's hard to change? I'm kind of against the wall relative to the v6.16 merge window and haven't had time to dig into this part.

Reviewed-by
#54 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

Quoted 43 lines · On Wed, 21 May 2025 17:59:42 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 14 lines · Jonathan Cameron
Quoted 5 lines · On Tue, 20 May 2025 16:50:34 -0500 Bjorn Helgaas <[email protected]> wrote:

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

...

There is some relatively new SYSFS infra that I think will help make this slightly nicer by getting rid of the extra directory when there is nothing to be done with it.

Quoted 7 lines

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 675

Quoted 5 lines
drivers/pci/pcie/aer.c +5 -0
+#define aer_ratelimit_burst_attr(name, ratelimit)			\
+	static ssize_t							\
+	name##_show(struct device *dev, struct device_attribute *attr,	\
+		    char *buf)						\
+{									\  

A little odd looking to indent this less than the line above.

Yep, fixed.

Quoted 18 lines

drivers/pci/pcie/aer.c const struct attribute_group aer_stats_attr_group = { · line 727

Quoted 5 lines
drivers/pci/pcie/aer.c +5 -0
+const struct attribute_group aer_attr_group = {
+	.name = "aer",
+	.attrs = aer_attrs,
+	.is_visible = aer_attrs_are_visible,
+};  

There are a bunch of macros to simplify cases where a whole group is either enabled or not and make the group itself go away if there is nothing to be shown.

DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE() combined with SYSFS_GROUP_VISIBLE() around the assignment does what we want here I think.

Whilst we can't retrofit that stuff onto existing ABI as someone may be assuming directory presence, we can make sysfs less cluttered for new stuff.

Quoted 6 lines
Quoted 1 line

Maybe I'm missing why that doesn't work here though!

Is this something we can fix later, or are we locking ourselves into user-visible ABI that's hard to change? I'm kind of against the wall relative to the v6.16 merge window and haven't had time to dig into this part.

That comes down to Ilpo's question of whether empty directories are ABI (specifically if anyone notices us removing them). It seems unlikely anyone will code against requirement for an empty dir, but you never know.

Given we probably have a bunch of these in PCI anyway that predate that magic, one more isn't a problem even if we decide we can't tidy it up later.

So I'm fine with not bothering to hide the dir for now (and maybe for ever).

Jonathan

Quoted 1 line
#55 lore raw

Re: [PATCH v7 17/17] PCI/AER: Add sysfs attributes for log ratelimits

In reply to Bjorn Helgaas
Quoted 29 lines · Bjorn Helgaas

From: Jon Pan-Doh <[email protected]>

Allow userspace to read/write log ratelimits per device (including enable/disable). Create aer/ sysfs directory to store them and any future aer configs.

(quoted diff) +23 -0
+PCIe AER ratelimits
+-------------------
+
+These attributes show up under all the devices that are AER capable.
+They represent configurable ratelimits of logs per error type.
+
+See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
+		gets whether or not AER ratelimiting is currently enabled.
+		Enabled by default.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for correctable error logs. Writing a value
+		changes the number of errors (burst) allowed per interval
+		(5 second window) before ratelimiting. Reading gets the

Review

Quoted 10 lines
(quoted diff) +10 -0
+		current ratelimit burst.
+
+What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected], [email protected]
+Description:	Ratelimit burst for non-fatal uncorrectable error logs.
+		Writing a value changes the number of errors (burst)
+		allowed per interval (5 second window) before ratelimiting.
+		Reading gets the current ratelimit burst.

I propose that we rename these to:

aer/correctable_ratelimit_burst
aer/nonfatal_ratelimit_burst

I think "log" is probably unnecessary, and the error type part should correspond with the existing entries (unfortunately not in the "aer" directory):

aer_dev_correctable
aer_dev_nonfatal

and making "ratelimit_burst" a suffix will make it correspond to these:

  fs/ext4/sysfs.c:        ATTR_LIST(err_ratelimit_burst),
  fs/ext4/sysfs.c:        ATTR_LIST(warning_ratelimit_burst),
  fs/ext4/sysfs.c:        ATTR_LIST(msg_ratelimit_burst),

lib/fault-inject.c:FAULT_CONFIGFS_ATTR_NAMED(ratelimit_burst, "verbose_ratelimit_burst",

Also, I think we should split up "aer/ratelimit_log_enable" so we can do it separately for correctable and nonfatal errors and name as:

aer/correctable_ratelimit_interval_ms
aer/nonfatal_ratelimit_interval_ms

so we can set the interval as well as the burst and to match these:

  fs/ext4/sysfs.c:        ATTR_LIST(err_ratelimit_interval_ms),
  fs/ext4/sysfs.c:        ATTR_LIST(warning_ratelimit_interval_ms),
  fs/ext4/sysfs.c:        ATTR_LIST(msg_ratelimit_interval_ms),

lib/fault-inject.c:FAULT_CONFIGFS_ATTR_NAMED(ratelimit_interval, "verbose_ratelimit_interval_ms",

I'll post the series as a whole again, but here's the interdiff related to this particular change:

Documentation/ABI/testing/sysfs-bus-pci-devices-aer +23 -13
index 01bb577bfee8..5ed284523956 100644
@@ -126,28 +126,38 @@ They represent configurable ratelimits of logs per error type.
 
 See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
+What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_interval_ms
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
-Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
-		gets whether or not AER ratelimiting is currently enabled.
-		Enabled by default.
+Contact:	[email protected]
+Description:	Writing 0 disables AER correctable error log ratelimiting.
+		Writing a positive value sets the ratelimit interval in ms.
+		Default is DEFAULT_RATELIMIT_INTERVAL (5000 ms).
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
+What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_burst
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
+Contact:	[email protected]
 Description:	Ratelimit burst for correctable error logs. Writing a value
 		changes the number of errors (burst) allowed per interval
-		(5 second window) before ratelimiting. Reading gets the
-		current ratelimit burst.
+		before ratelimiting. Reading gets the current ratelimit
+		burst. Default is DEFAULT_RATELIMIT_BURST (10).
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
+What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_interval_ms
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
+Contact:	[email protected]
+Description:	Writing 0 disables AER non-fatal uncorrectable error log
+		ratelimiting. Writing a positive value sets the ratelimit
+		interval in ms. Default is DEFAULT_RATELIMIT_INTERVAL
+		(5000 ms).
+
+What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_burst
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected]
 Description:	Ratelimit burst for non-fatal uncorrectable error logs.
 		Writing a value changes the number of errors (burst)
-		allowed per interval (5 second window) before ratelimiting.
-		Reading gets the current ratelimit burst.
+		allowed per interval before ratelimiting. Reading gets the
+		current ratelimit burst. Default is DEFAULT_RATELIMIT_BURST
+		(10).
drivers/pci/pcie/aer.c +56 -50

142 lines not on this page · raw message

#56 lore raw
In reply to Bjorn Helgaas
Quoted 11 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

This work is mostly due to Jon Pan-Doh and Karolina Stolarek. I rebased this to v6.15-rc1, factored out some of the trace and statistics updates, and added some minor cleanups.

I'm sorry to post a v7 so soon after v6, but I really want to get this in v6.16 so it needs to get into pci/next soonish. I pushed this to pci/aer at https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/log/?h=aer
(head cbde036e5615 ("PCI/AER: Add sysfs attributes for log ratelimits")) and appended the interdiff from v6 to v7 below.

Thank you ever so much for all your reviewing time. I addressed most of the comments (except Sathy's comments about the sysfs cor/uncor/nonfatal names), but I hate to deluge you all with another full posting.

So for now I updated the "aer" branch above (current head d41e0decb7d7 ("PCI/AER: Add sysfs attributes for log ratelimits")) and attached the interdiff between v7 and d41e0decb7d7 below.

drivers/pci/pci.h +6 -4
index a3261e842d6d..eca2812cfd25 100644
@@ -587,13 +587,14 @@ static inline bool pci_dev_test_and_set_removed(struct pci_dev *dev)
 
 struct aer_err_info {
 	struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES];
+	int ratelimit_print[AER_MAX_MULTI_ERR_DEVICES];
 	int error_dev_num;
 	const char *level;		/* printk level */
 
 	unsigned int id:16;
 
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int ratelimit:1;	/* 0=skip, 1=print */
+	unsigned int root_ratelimit_print:1;	/* 0=skip, 1=print */
 	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
@@ -606,15 +607,16 @@ struct aer_err_info {
 	struct pcie_tlp_log tlp;	/* TLP Header */
 };
 
-int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info);
-void aer_print_error(struct pci_dev *dev, struct aer_err_info *info);
+int aer_get_device_error_info(struct aer_err_info *info, int i);
+void aer_print_error(struct aer_err_info *info, int i);
 
 int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2,
 		      unsigned int tlp_len, bool flit,
 		      struct pcie_tlp_log *log);
 unsigned int aer_tlp_log_len(struct pci_dev *dev, u32 aercc);
 void pcie_print_tlp_log(const struct pci_dev *dev,
-			const struct pcie_tlp_log *log, const char *pfx);
+			const struct pcie_tlp_log *log, const char *level,
+			const char *pfx);
 #endif	/* CONFIG_PCIEAER */
 
 #ifdef CONFIG_PCIEPORTBUS
drivers/pci/pcie/aer.c +76 -62

262 lines not on this page · raw message

drivers/pci/pcie/dpc.c +7 -4
index 530c5e2cf7e8..fc18349614d7 100644
@@ -222,7 +222,7 @@ static void dpc_process_rp_pio_error(struct pci_dev *pdev)
 			  dpc_tlp_log_len(pdev),
 			  pdev->subordinate->flit_mode,
 			  &tlp_log);
-	pcie_print_tlp_log(pdev, &tlp_log, dev_fmt(""));
+	pcie_print_tlp_log(pdev, &tlp_log, KERN_ERR, dev_fmt(""));
 
 	if (pdev->dpc_rp_log_size < PCIE_STD_NUM_TLP_HEADERLOG + 1)
 		goto clear_status;
@@ -253,6 +253,10 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 		info->severity = AER_NONFATAL;
 
 	info->level = KERN_ERR;
+
+	info->dev[0] = dev;
+	info->error_dev_num = 1;
+
 	return 1;
 }
 
@@ -270,9 +274,8 @@ void dpc_process_error(struct pci_dev *pdev)
 		pci_warn(pdev, "containment event, status:%#06x: unmasked uncorrectable error detected\n",
 			 status);
 		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
-		    aer_get_device_error_info(pdev, &info)) {
-			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
-			aer_print_error(pdev, &info);
+		    aer_get_device_error_info(&info, 0)) {
+			aer_print_error(&info, 0);
 			pci_aer_clear_nonfatal_status(pdev);
 			pci_aer_clear_fatal_status(pdev);
 		}
drivers/pci/pcie/tlp.c +4 -2
index 890d5391d7f5..71f8fc9ea2ed 100644
@@ -98,12 +98,14 @@ int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2,
  * pcie_print_tlp_log - Print TLP Header / Prefix Log contents
  * @dev: PCIe device
  * @log: TLP Log structure
+ * @level: Printk log level
  * @pfx: String prefix
  *
  * Prints TLP Header and Prefix Log information held by @log.
  */
 void pcie_print_tlp_log(const struct pci_dev *dev,
-			const struct pcie_tlp_log *log, const char *pfx)
+			const struct pcie_tlp_log *log, const char *level,
+			const char *pfx)
 {
 	/* EE_PREFIX_STR fits the extended DW space needed for the Flit mode */
 	char buf[11 * PCIE_STD_MAX_TLP_HEADERLOG + 1];
@@ -130,6 +132,6 @@ void pcie_print_tlp_log(const struct pci_dev *dev,
 		}
 	}
 
-	pci_err(dev, "%sTLP Header%s: %s\n", pfx,
+	dev_printk(level, &dev->dev, "%sTLP Header%s: %s\n", pfx,
 		log->flit ? " (Flit)" : "", buf);
 }