Skip to contentSkip to conversation

suppress excessive AER output

RFC In review Dave Jones 4 participants 9 messages Aug 3, 2011 – Aug 5, 2011 PCI

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

RFC 1/1 suppress excessive AER output

I have a machine that has developed some kind of problem with its onboard ethernet. It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Dave

drivers/pci/pcie/aer/aerdrv_errprint.c +11 -0
index 3ea5173..4ec88c6 100644
@@ -153,6 +153,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	int id = ((dev->bus->number << 8) | dev->devfn);
 	char prefix[44];
+	static unsigned long aer_printk_limit = 0;
+
+	aer_printk_limit++;
+
+	if (aer_printk_limit > 100)
+		return;
+
+	if (aer_printk_limit == 100) {
+		printk(KERN_ERR "Reached limit of 100 AER errors. Further AER output suppressed.\n");
+		return;
+	}
 
 	snprintf(prefix, sizeof(prefix), "%s%s %s: ",
 		 (info->severity == AER_CORRECTABLE) ? KERN_WARNING : KERN_ERR,
#1 lore raw

Dave,

How about adding a new module parameter aer_printk_limit, so user space could reset it any time?

Yanmin

-----Original Message-----
From: Dave Jones [mailto:[email protected]]
Sent: Thursday, August 04, 2011 6:34 AM
To: Linux Kernel
Cc: Nguyen, Tom L; Zhang, Yanmin
Subject: [rfc] suppress excessive AER output

I have a machine that has developed some kind of problem with its onboard ethernet. It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Dave

drivers/pci/pcie/aer/aerdrv_errprint.c +11 -0
index 3ea5173..4ec88c6 100644
@@ -153,6 +153,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
 {
 	int id = ((dev->bus->number << 8) | dev->devfn);
 	char prefix[44];
+	static unsigned long aer_printk_limit = 0;
+
+	aer_printk_limit++;
+
+	if (aer_printk_limit > 100)
+		return;
+
+	if (aer_printk_limit == 100) {
+		printk(KERN_ERR "Reached limit of 100 AER errors. Further AER output suppressed.\n");
+		return;
+	}
 
 	snprintf(prefix, sizeof(prefix), "%s%s %s: ",
 		 (info->severity == AER_CORRECTABLE) ? KERN_WARNING : KERN_ERR,
author
#2 lore raw
Quoted 3 lines · Yanmin

Dave,

How about adding a new module parameter aer_printk_limit, so user space could reset it any time?

Sure. Though for the built-in case, how would that work ?

Dave

#3 lore raw

Dave,

My idea is application has opportunity to reset it. Consider the critical mission environment, admin could hot-unplug any failed devices without rebooting system. Then, admin doesn't want to lose AER monitoring.

With the module parameter, admin could change it under /sys/module/aer_drv/parameters.

Yanmin

-----Original Message-----
From: Dave Jones [mailto:[email protected]]
Sent: Friday, August 05, 2011 9:50 AM
To: Zhang, Yanmin
Cc: Linux Kernel; Nguyen, Tom L; Huang, Ying
Subject: Re: [rfc] suppress excessive AER output

Quoted 3 lines · Yanmin

Dave,

How about adding a new module parameter aer_printk_limit, so user space could reset it any time?

Sure. Though for the built-in case, how would that work ?

Dave

author
#4 lore raw
Quoted 5 lines · Yanmin

Dave,

My idea is application has opportunity to reset it. Consider the critical mission environment, admin could hot-unplug any failed devices without rebooting system. Then, admin doesn't want to lose AER monitoring.

With the module parameter, admin could change it under /sys/module/aer_drv/parameters.

ah, I see. Yes, that sounds sensible.

Dave.

#5 lore raw
In reply to Dave Jones
Quoted 5 lines · Dave Jones

I have a machine that has developed some kind of problem with its onboard ethernet.  It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Quoted 8 lines

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Why not use __ratelimit to implement this feature?

Best Regards,
Huang Ying

author
#6 lore raw
Quoted 10 lines · huang ying
Quoted 9 lines · Dave Jones

I have a machine that has developed some kind of problem with its onboard ethernet.  It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

Quoted 6 lines
Quoted 4 lines

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Why not use __ratelimit to implement this feature?

that would be better than the current situation probably, but my gut feeling is that there's still going to be a lot of spew.

Dave

#7 lore raw
In reply to Dave Jones

Hi,

Quoted 5 lines · Dave Jones

I have a machine that has developed some kind of problem with its onboard ethernet.  It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Quoted 9 lines

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Please do not reinvent the wheel and use printk_ratelimited().

Thanks,
- Arnaud

Quoted 30 lines

       Dave

drivers/pci/pcie/aer/aerdrv_errprint.c +11 -1
index 3ea5173..4ec88c6 100644
@@ -153,6 +153,17 @@ void aer_print_error(struct pci_dev *dev, struct aer_err_info *info)
  {
        int id = ((dev->bus->number << 8) | dev->devfn);
        char prefix[44];
+       static unsigned long aer_printk_limit = 0;
+
+       aer_printk_limit++;
+
+       if (aer_printk_limit > 100)
+               return;
+
+       if (aer_printk_limit == 100) {
+               printk(KERN_ERR "Reached limit of 100 AER errors. Further AER output suppressed.\n");
+               return;
+       }
 
        snprintf(prefix, sizeof(prefix), "%s%s %s: ",
                 (info->severity == AER_CORRECTABLE) ? KERN_WARNING : KERN_ERR,
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

author
#8 lore raw
Quoted 12 lines · Arnaud Lacombe

Hi,

Quoted 9 lines · Dave Jones

I have a machine that has developed some kind of problem with its onboard ethernet.  It still boots, but spewed almost 1.5G of text (2381585 instances of the warning below) before we realised what was going on, and blacklisted the igb driver.

Is it worth logging every single error when we're flooding like this ? It seems unlikely that we'll find useful information in amongst that much data that wasn't already in the first 100 instances.

Quoted 6 lines
Quoted 5 lines

I picked 100 in the (untested) example patch below arbitarily, but the exact value could be smaller, or slightly bigger..

could we do something like this maybe ?

Please do not reinvent the wheel and use printk_ratelimited().

It's a different wheel.

printk_ratelimit slows down the output, but would still cause a lot of messages.

my diff turns it off completely after a threshold (apart from at the ulong wrap, which I overlooked).

Dave