Skip to content

Rate limit AER logs

Bjorn Helgaas linux-pci linuxppc-dev lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest20 patches 17 repliesIn review
  2. v717 patches 39 repliesSuperseded
  3. v616 patches 49 repliesSuperseded

Changes in v8 from the cover letter

- Rename sysfs ratelimit burst files:

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/20PCI/DPC: Initialize aer_err_info before using itdrivers/pci/pcie/dpc.c+1 -10 replies
2/20PCI/DPC: Log Error Source ID only when validdrivers/pci/pcie/dpc.c+36 -280 replies
3/20PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error()drivers/pci/pcie/aer.c+23 -130 replies
4/20PCI/AER: Consolidate Error Source ID logging in aer_isr_one_error_type()drivers/pci/pcie/aer.c+9 -130 replies
5/20PCI/AER: Extract bus/dev/fn in aer_print_port_info() with PCI_BUS_NUM(), etcdrivers/pci/pcie/aer.c+4 -40 replies
6/20PCI/AER: Rename aer_print_port_info() to aer_print_source()drivers/pci/pcie/aer.c+3 -30 replies
7/20PCI/AER: Move aer_print_source() earlier in filedrivers/pci/pcie/aer.c+13 -130 replies
8/20PCI/AER: Initialize aer_err_info before using itdrivers/pci/pcie/aer.c+17 -220 replies
9/20PCI/AER: Simplify pci_print_aer()drivers/pci/pcie/aer.c+8 -80 replies
10/20PCI/AER: Update statistics before ratelimitingdrivers/pci/pcie/aer.c+4 -10 replies
11/20PCI/AER: Trace error event before ratelimitingdrivers/pci/pcie/aer.c+4 -60 replies
12/20PCI/AER: Check log level once and remember itdrivers/pci/pci.h drivers/pci/pcie/aer.c drivers/pci/pcie/dpc.c+12 -110 replies
13/20PCI/ERR: Add printk level to pcie_print_tlp_log()drivers/pci/pci.h drivers/pci/pcie/aer.c drivers/pci/pcie/dpc.c +1 more+10 -64 replies
14/20PCI/AER: Reduce pci_print_aer() correctable error level to KERN_WARNINGdrivers/pci/pcie/aer.c+6 -50 replies
15/20PCI/AER: Rename struct aer_stats to aer_infodrivers/pci/pcie/aer.c include/linux/pci.h+24 -240 replies
16/20PCI/AER: Convert aer_get_device_error_info(), aer_print_error() to indexdrivers/pci/pci.h drivers/pci/pcie/aer.c drivers/pci/pcie/dpc.c+31 -143 replies
17/20PCI/AER: Simplify add_error_device()drivers/pci/pcie/aer.c+9 -62 replies
18/20PCI/AER: Ratelimit correctable and non-fatal error loggingdrivers/pci/pci.h drivers/pci/pcie/aer.c+67 -66 replies
19/20PCI/AER: Add ratelimits to PCI AER DocumentationDocumentation/PCI/pcieaer-howto.rst+12 -00 replies
20/20PCI/AER: Add sysfs attributes for log ratelimitsDocumentation/ABI/testing/sysfs-bus-pci-devices-aer Documentation/PCI/pcieaer-howto.rst drivers/pci/pci-sysfs.c +2 more+155 -11 reply

Reviews trailers per patch

All 20 patches have a Reviewed-by or Acked-by.

PatchKrzysztof WilczyńskiKuppuswamy SathyanarayananIlpo JärvinenJonathan CameronPaul E. McKenney
1/20T-b R-b R-b R-b
2/20T-b R-b R-b R-b
3/20R-b R-b R-b
4/20R-b R-b R-b
5/20T-b R-b R-b R-b
6/20T-b R-b R-b R-b
7/20T-b R-b R-b R-b
8/20T-b R-b R-b R-b
9/20T-b R-b R-b R-b
10/20T-b R-b R-b R-b
11/20T-b R-b R-b
12/20T-b R-b R-b R-b
13/20R-b R-b
14/20T-b R-b R-b R-b
15/20T-b R-b R-b R-b
16/20R-b R-b
17/20R-b R-b
18/20R-b
19/20T-b R-b A-b
20/20R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v8 thread

Cover letter [PATCH v8 00/20] Rate limit AER logs

From: Bjorn Helgaas <[email protected]>

This work is mostly due to Jon Pan-Doh and Karolina Stolarek. I rebased this to v6.15-rc1, factored out some of the trace and statistics updates, and added some minor cleanups.

I pushed this to pci/aer at
https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git/log/?h=aer
(head a524e63307cf ("PCI/AER: Add sysfs attributes for log ratelimits")) and appended the interdiff from v7 to v8 below.

Proposal
========

When using native AER, spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device per-error-severity ratelimits for more robust error logging. Allow userspace to configure ratelimits via sysfs knobs.

Motivation
==========

Inconsistent PCIe error handling, exacerbated at datacenter scale (myriad of devices), affects repairabilitiy flows for fleet operators.

Exposing PCIe errors/debug info in-band for a userspace daemon (e.g. rasdaemon) to collect/pass on to repairability services will allow for more predictable repair flows and decrease machine downtime.

Background
==========

AER error spam has been observed many times, both publicly (e.g. [1], [2], [3]) and privately. While it usually occurs with correctable errors, it can happen with uncorrectable errors (e.g. during new HW bringup).

There have been previous attempts to add ratelimits to AER logs ([4], [5]). The most recent attempt[5] has many similarities with the proposed approach.

v8:
- Rename sysfs ratelimit burst files:

    ratelimit_burst_cor_log -> correctable_ratelimit_burst (Sathy)
    ratelimit_burst_uncor_log -> nonfatal_ratelimit_burst
- Split sysfs ratelimit_log_enable for correctable and nonfatal and make it
  an interval instead of a toggle:
    ratelimit_log_enable -> correctable_ratelimit_interval_ms
                         -> nonfatal_ratelimit_interval_ms

- Rework aer_get_device_error_info() and aer_print_error() to take an index instead of pci_dev pointer
- Move trace_aer_event() out of pci_dev_aer_stats_incr() (Jonathan)
- Move AER_FATAL checking to aer_ratelimit() to avoid calling __ratelimit(nonfatal_ratelimit) when we know we don't want to ratelimit fatal errors (Jonathan)
- Move all Error Source ID string building into aer_print_source() instead of putting part in caller (Jonathan)
- Rename struct aer_err_info.ratelimit -> ratelimit_print[] (Jonathan)
- Pass printk level into pcie_print_tlp_log() (Jonathan)
- Rework Error Source ratelimiting vs detail ratelimiting (Jonathan)
v7: https://lore.kernel.org/r/[email protected]
- Update sysfs doc target kernel version & date (Ilpo)
- Fix sysfs doc "AER ratelimiting" typo (Ilpo)
- Ratelimit Correctable and Non-Fatal but not Fatal errors (Sathy)
- Rename "struct aer_report" to "aer_info" (Sathy)
- Expand comments about combining ratelimit for multiple devices (Ilpo)
- Rework Error Source logging ratelimiting (Sathy)
- Factor out aer_isr_one_error_type() to reduce code duplication
- Log DPC errors, which are all Fatal, at KERN_ERR (Sathy)
- Improve dpc_process_error() structure (Ilpo)
v6: https://lore.kernel.org/r/[email protected]
- Rebase to v6.15-rc1
- Initialize struct aer_err_info completely before using it
- Log DPC Error Source ID only when it's valid
- Consolidate AER Error Source ID logging to one place
- Tidy Error Source ID bus/dev/fn decoding using macros
- Rename aer_print_port_info() to aer_print_source()
- Consolidate trace events and statistic updates to one non-ratelimited place
- Save log level in struct aer_err_info instead of passing as parameter
v5: https://lore.kernel.org/r/[email protected]
- Handle multi-error AER by evaluating ratelimits once and storing result
- Reword/rename commit messages/functions/variable
v4: https://lore.kernel.org/r/[email protected]
- Fix bug where trace not emitted with malformed aer_err_info
- Extend ratelimit to malformed aer_err_info
- Update commit messages with patch motivation
- Squash AER sysfs filename change (Patch 8)
v3: https://lore.kernel.org/r/[email protected]
- Ratelimit aer_print_port_info() (drop Patch 1)
- Add ratelimit enable toggle
- Move trace outside of ratelimit
- Split log level (Patch 2) into two
- More descriptive documentation/sysfs naming
v2: https://lore.kernel.org/r/[email protected]
- Rebased on top of pci/aer (6.14.rc-1)
- Split series into log and IRQ ratelimits (defer patch 5)
- Dropped patch 8 (Move AER sysfs)
- Added log level cleanup patch[7] from Karolina's series
- Fixed bug where dpc errors didn't increment counters
- "X callbacks suppressed" message on ratelimit release -> immediately
- Separate documentation into own patch
v1: https://lore.kernel.org/r/[email protected]

[1] https://bugzilla.kernel.org/show_bug.cgi?id=215027
[2] https://bugzilla.kernel.org/show_bug.cgi?id=201517
[3] https://bugzilla.kernel.org/show_bug.cgi?id=196183
[4] https://lore.kernel.org/linux-pci/[email protected]/
[5] https://lore.kernel.org/linux-pci/[email protected]/
[6]
https://lore.kernel.org/linux-pci/8bcb8c9a7b38ce3bdaca5a64fe76f08b0b337511.1742202797.git.k
[email protected]/
[7]
https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.k
[email protected]/

Bjorn Helgaas (13):
PCI/DPC: Initialize aer_err_info before using it
PCI/DPC: Log Error Source ID only when valid
PCI/AER: Factor COR/UNCOR error handling out from aer_isr_one_error() PCI/AER: Consolidate Error Source ID logging in
aer_isr_one_error_type()
PCI/AER: Extract bus/dev/fn in aer_print_port_info() with

    PCI_BUS_NUM(), etc

PCI/AER: Move aer_print_source() earlier in file PCI/AER: Initialize aer_err_info before using it PCI/AER: Simplify pci_print_aer()
PCI/AER: Update statistics before ratelimiting
PCI/AER: Trace error event before ratelimiting
PCI/ERR: Add printk level to pcie_print_tlp_log() PCI/AER: Convert aer_get_device_error_info(), aer_print_error() to
index
PCI/AER: Simplify add_error_device()

Jon Pan-Doh (4):
PCI/AER: Rename aer_print_port_info() to aer_print_source() PCI/AER: Ratelimit correctable and non-fatal error logging PCI/AER: Add ratelimits to PCI AER Documentation PCI/AER: Add sysfs attributes for log ratelimits

Karolina Stolarek (3):
PCI/AER: Check log level once and remember it PCI/AER: Reduce pci_print_aer() correctable error level to
KERN_WARNING
PCI/AER: Rename struct aer_stats to aer_info

 ...es-aer_stats => sysfs-bus-pci-devices-aer} |  44 ++
 Documentation/PCI/pcieaer-howto.rst           |  17 +-
 drivers/pci/pci-sysfs.c                       |   1 +
 drivers/pci/pci.h                             |  13 +-
 drivers/pci/pcie/aer.c                        | 441 +++++++++++++-----
 drivers/pci/pcie/dpc.c                        |  73 +--
 drivers/pci/pcie/tlp.c                        |   6 +-
 include/linux/pci.h                           |   2 +-
 8 files changed, 430 insertions(+), 167 deletions(-)
 rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (72%)
Signature · 1 line
2.43.0
Documentation/ABI/testing/sysfs-bus-pci-devices-aer +23 -13
index 01bb577bfee8..5ed284523956 100644
@@ -126,28 +126,38 @@ They represent configurable ratelimits of logs per error type.
 
 See Documentation/PCI/pcieaer-howto.rst for more info on ratelimits.
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_log_enable
+What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_interval_ms
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
-Description:	Writing 1/0 enables/disables AER log ratelimiting. Reading
-		gets whether or not AER ratelimiting is currently enabled.
-		Enabled by default.
+Contact:	[email protected]
+Description:	Writing 0 disables AER correctable error log ratelimiting.
+		Writing a positive value sets the ratelimit interval in ms.
+		Default is DEFAULT_RATELIMIT_INTERVAL (5000 ms).
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_cor_log
+What:		/sys/bus/pci/devices/<dev>/aer/correctable_ratelimit_burst
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
+Contact:	[email protected]
 Description:	Ratelimit burst for correctable error logs. Writing a value
 		changes the number of errors (burst) allowed per interval
-		(5 second window) before ratelimiting. Reading gets the
-		current ratelimit burst.
+		before ratelimiting. Reading gets the current ratelimit
+		burst. Default is DEFAULT_RATELIMIT_BURST (10).
 
-What:		/sys/bus/pci/devices/<dev>/aer/ratelimit_burst_uncor_log
+What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_interval_ms
 Date:		May 2025
 KernelVersion:	6.16.0
-Contact:	[email protected], [email protected]
+Contact:	[email protected]
+Description:	Writing 0 disables AER non-fatal uncorrectable error log
+		ratelimiting. Writing a positive value sets the ratelimit
+		interval in ms. Default is DEFAULT_RATELIMIT_INTERVAL
+		(5000 ms).
+
+What:		/sys/bus/pci/devices/<dev>/aer/nonfatal_ratelimit_burst
+Date:		May 2025
+KernelVersion:	6.16.0
+Contact:	[email protected]
 Description:	Ratelimit burst for non-fatal uncorrectable error logs.
 		Writing a value changes the number of errors (burst)
-		allowed per interval (5 second window) before ratelimiting.
-		Reading gets the current ratelimit burst.
+		allowed per interval before ratelimiting. Reading gets the
+		current ratelimit burst. Default is DEFAULT_RATELIMIT_BURST
+		(10).
drivers/pci/pci.h +6 -4
index a3261e842d6d..eca2812cfd25 100644
@@ -587,13 +587,14 @@ static inline bool pci_dev_test_and_set_removed(struct pci_dev *dev)
 
 struct aer_err_info {
 	struct pci_dev *dev[AER_MAX_MULTI_ERR_DEVICES];
+	int ratelimit_print[AER_MAX_MULTI_ERR_DEVICES];
 	int error_dev_num;
 	const char *level;		/* printk level */
 
 	unsigned int id:16;
 
 	unsigned int severity:2;	/* 0:NONFATAL | 1:FATAL | 2:COR */
-	unsigned int ratelimit:1;	/* 0=skip, 1=print */
+	unsigned int root_ratelimit_print:1;	/* 0=skip, 1=print */
 	unsigned int __pad1:4;
 	unsigned int multi_error_valid:1;
 
@@ -606,15 +607,16 @@ struct aer_err_info {
 	struct pcie_tlp_log tlp;	/* TLP Header */
 };
 
-int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info);
-void aer_print_error(struct pci_dev *dev, struct aer_err_info *info);
+int aer_get_device_error_info(struct aer_err_info *info, int i);
+void aer_print_error(struct aer_err_info *info, int i);
 
 int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2,
 		      unsigned int tlp_len, bool flit,
 		      struct pcie_tlp_log *log);
 unsigned int aer_tlp_log_len(struct pci_dev *dev, u32 aercc);
 void pcie_print_tlp_log(const struct pci_dev *dev,
-			const struct pcie_tlp_log *log, const char *pfx);
+			const struct pcie_tlp_log *log, const char *level,
+			const char *pfx);
 #endif	/* CONFIG_PCIEAER */
 
 #ifdef CONFIG_PCIEPORTBUS
drivers/pci/pcie/aer.c +131 -111

390 lines not on this page · raw message

drivers/pci/pcie/dpc.c +7 -4
index 530c5e2cf7e8..fc18349614d7 100644
@@ -222,7 +222,7 @@ static void dpc_process_rp_pio_error(struct pci_dev *pdev)
 			  dpc_tlp_log_len(pdev),
 			  pdev->subordinate->flit_mode,
 			  &tlp_log);
-	pcie_print_tlp_log(pdev, &tlp_log, dev_fmt(""));
+	pcie_print_tlp_log(pdev, &tlp_log, KERN_ERR, dev_fmt(""));
 
 	if (pdev->dpc_rp_log_size < PCIE_STD_NUM_TLP_HEADERLOG + 1)
 		goto clear_status;
@@ -253,6 +253,10 @@ static int dpc_get_aer_uncorrect_severity(struct pci_dev *dev,
 		info->severity = AER_NONFATAL;
 
 	info->level = KERN_ERR;
+
+	info->dev[0] = dev;
+	info->error_dev_num = 1;
+
 	return 1;
 }
 
@@ -270,9 +274,8 @@ void dpc_process_error(struct pci_dev *pdev)
 		pci_warn(pdev, "containment event, status:%#06x: unmasked uncorrectable error detected\n",
 			 status);
 		if (dpc_get_aer_uncorrect_severity(pdev, &info) &&
-		    aer_get_device_error_info(pdev, &info)) {
-			info.ratelimit = 1;	/* ERR_FATAL; no ratelimit */
-			aer_print_error(pdev, &info);
+		    aer_get_device_error_info(&info, 0)) {
+			aer_print_error(&info, 0);
 			pci_aer_clear_nonfatal_status(pdev);
 			pci_aer_clear_fatal_status(pdev);
 		}
drivers/pci/pcie/tlp.c +4 -2
index 890d5391d7f5..71f8fc9ea2ed 100644
@@ -98,12 +98,14 @@ int pcie_read_tlp_log(struct pci_dev *dev, int where, int where2,
  * pcie_print_tlp_log - Print TLP Header / Prefix Log contents
  * @dev: PCIe device
  * @log: TLP Log structure
+ * @level: Printk log level
  * @pfx: String prefix
  *
  * Prints TLP Header and Prefix Log information held by @log.
  */
 void pcie_print_tlp_log(const struct pci_dev *dev,
-			const struct pcie_tlp_log *log, const char *pfx)
+			const struct pcie_tlp_log *log, const char *level,
+			const char *pfx)
 {
 	/* EE_PREFIX_STR fits the extended DW space needed for the Flit mode */
 	char buf[11 * PCIE_STD_MAX_TLP_HEADERLOG + 1];
@@ -130,6 +132,6 @@ void pcie_print_tlp_log(const struct pci_dev *dev,
 		}
 	}
 
-	pci_err(dev, "%sTLP Header%s: %s\n", pfx,
+	dev_printk(level, &dev->dev, "%sTLP Header%s: %s\n", pfx,
 		log->flit ? " (Flit)" : "", buf);
 }