EDAC: update edac printk wrappers to use printk_ratelimited.
PATCH 1/1 EDAC: update edac printk wrappers to use printk_ratelimited.
Update printk to the ratelimited version, so that in some corner cases when vast of CE errors show up, the kernel logging can be suppressed.
--- drivers/edac/edac_mc.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/edac/edac_mc.h +5 -5
@@ -46,19 +46,19 @@#endif #define edac_printk(level, prefix, fmt, arg...) \ - printk(level "EDAC " prefix ": " fmt, ##arg) + printk_ratelimited(level "EDAC " prefix ": " fmt, ##arg) #define edac_mc_printk(mci, level, fmt, arg...) \ - printk(level "EDAC MC%d: " fmt, mci->mc_idx, ##arg) + printk_ratelimited(level "EDAC MC%d: " fmt, mci->mc_idx, ##arg) #define edac_mc_chipset_printk(mci, level, prefix, fmt, arg...) \ - printk(level "EDAC " prefix " MC%d: " fmt, mci->mc_idx, ##arg) + printk_ratelimited(level "EDAC " prefix " MC%d: " fmt, mci->mc_idx, ##arg) #define edac_device_printk(ctl, level, fmt, arg...) \ - printk(level "EDAC DEVICE%d: " fmt, ctl->dev_idx, ##arg) + printk_ratelimited(level "EDAC DEVICE%d: " fmt, ctl->dev_idx, ##arg) #define edac_pci_printk(ctl, level, fmt, arg...) \ - printk(level "EDAC PCI%d: " fmt, ctl->pci_idx, ##arg) + printk_ratelimited(level "EDAC PCI%d: " fmt, ctl->pci_idx, ##arg)/* prefixes for edac_printk() and edac_mc_printk() */ #define EDAC_MC "MC"
Signature · 1 line
2.17.1
Borislav Petkov
maintainerQuoted 2 lines · Lei Wang
Update printk to the ratelimited version, so that in some corner cases when vast of CE errors show up, the kernel logging can be suppressed.
Err, why?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteHi Boris,
We found a corner case in production environment that there are ~500 CE errors per second. The SoC otherwise functions just fine. Making printk ratelimited reduced CE error logging to < 20 per second. Though this is just one case so far, we think moving to printk_ratelimited could benefit broader use as well, by helping control the amount of kernel logging. In most running condition, the error rate is way below the limit. And in an error case like this one, vast error logging would not provide much valuable details, rather it's storming the kernel logging.
Thanks,
-Lei
-----Original Message-----
From: Borislav Petkov <[email protected]>
Sent: Wednesday, May 5, 2021 11:01 AM
To: wangglei <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Lei Wang (DPLAT) <[email protected]>; Hang Li <[email protected]>; [email protected]; Brandon Waller <[email protected]>
Subject: [EXTERNAL] Re: [PATCH] EDAC: update edac printk wrappers to use printk_ratelimited.
Quoted 2 lines · Lei Wang
Update printk to the ratelimited version, so that in some corner cases when vast of CE errors show up, the kernel logging can be suppressed.
Err, why?
--
Regards/Gruss,
Boris.
Borislav Petkov
maintainerHi Lei,
Quoted 1 line · Lei Wang (DPLAT)
Hi Boris,
first of all, please do not top-post.
Quoted 3 lines
We found a corner case in production environment that there are ~500 CE errors per second. The SoC otherwise functions just fine. Making printk ratelimited reduced CE error logging to < 20 per second.
If you want to avoid CE logs flooding dmesg, there's a couple of things you can do:
1. Use drivers/ras/cec.c
2. Do not load EDAC drivers at all since you don't care about the error reports, apparently.
3. Fix the CE source: replace the DIMMs, etc.
Quoted 3 lines
Though this is just one case so far, we think moving to
printk_ratelimitedcould benefit broader use as well, by helping control the amount of kernel logging.
No, this will make EDAC driver loading output incomplete when some of the messages are omitted due to the ratelimiting. And no, this is not going to happen.
HTH.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 11 lines · Borislav Petkov
Hi Lei,
Quoted 1 line · Lei Wang (DPLAT)
Hi Boris,
first of all, please do not top-post.
Quoted 3 lines
We found a corner case in production environment that there are ~500 CE errors per second. The SoC otherwise functions just fine. Making printk ratelimited reduced CE error logging to < 20 per second.
Quoted 7 lines
If you want to avoid CE logs flooding dmesg, there's a couple of things you can do:
1. Use
drivers/ras/cec.c2. Do not load EDAC drivers at all since you don't care about the error reports, apparently.
Lei, if you don't care about the CE error messages at all, there's also an edac_mc_log_ce module parameter that can be used to quiet the message emitted from edac_ce_error():
https://www.kernel.org/doc/html/latest/admin-guide/ras.html#module-parameters
Quoted 9 lines
3. Fix the CE source: replace the DIMMs, etc.
Quoted 3 lines
Though this is just one case so far, we think moving to
printk_ratelimitedcould benefit broader use as well, by helping control the amount of kernel logging.No, this will make EDAC driver loading output incomplete when some of the messages are omitted due to the ratelimiting. And no, this is not going to happen.
Boris, I agree that a more surgical approach is needed than this if Lei still needs some traces of the CE error messages in the logs. Would it be any more acceptable to add an edac_mc_printk_ratelimited() macro, which uses printk_ratelimited(), and then call that new macro from edac_ce_error()?
If you still don't want those CE errors ratelimited by default, perhaps a new, non-default mode (2) could be added to the edac_mc_log_ce module parameter that uses the ratelimited variant?
Tyler
Quoted 9 lines
HTH.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Borislav Petkov
maintainerQuoted 3 lines · Tyler Hicks
Would it be any more acceptable to add an
edac_mc_printk_ratelimited()macro, which usesprintk_ratelimited(), and then call that new macro fromedac_ce_error()?
You guys are way off here: the intent of EDAC drivers is to accurately report errors for purposes of counting them and doing analysis on that collected data as to whether components are going wrong - not to ratelimit them as some nuisance output.
With breaking the EDAC reporting, you're barking up the wrong tree - if you don't want to see those errors, do not load the drivers. It is that simple.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 5 lines · Borislav Petkov
Quoted 3 lines · Tyler Hicks
Would it be any more acceptable to add an
edac_mc_printk_ratelimited()macro, which usesprintk_ratelimited(), and then call that new macro fromedac_ce_error()?
Quoted 8 lines
You guys are way off here: the intent of EDAC drivers is to accurately report errors for purposes of counting them and doing analysis on that collected data as to whether components are going wrong - not to ratelimit them as some nuisance output.
With breaking the EDAC reporting, you're barking up the wrong tree - if you don't want to see those errors, do not load the drivers. It is that simple.
As I understand it, the idea here wasn't to treat the log messages as a nuisance that should be completely squelched. The counters are monitored and provide the definitive way to detect large scale problems but the CE log messages can be an easier-to-discover way for humans to identify potential problems when, for example, centralized log aggregation and indexing is in place.
The thought was that the full stream of log messages isn't necessary to notice that there's a problem when they are being emitted at such a high rate (500 per second). They're just filling up disk space and/or wasting networking bandwidth at that point. Of course, the best course of action here is to service the machine but there's still a period of time between the CE errors popping up and the machine being serviced.
Tyler
Borislav Petkov
maintainerQuoted 4 lines · Tyler Hicks
The thought was that the full stream of log messages isn't necessary to notice that there's a problem when they are being emitted at such a high rate (500 per second). They're just filling up disk space and/or wasting networking bandwidth at that point.
I already asked about this but lemme point it out again: have you guys looked at drivers/ras/cec.c ?
With that there won't be *any* error reports in dmesg and it will even poison and offline pages which generate excessive errors so that ...
Quoted 3 lines
Of course, the best course of action here is to service the machine but there's still a period of time between the CE errors popping up and the machine being serviced.
... you'll have ample time to service the machine.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 8 lines · Borislav Petkov
Quoted 4 lines · Tyler Hicks
The thought was that the full stream of log messages isn't necessary to notice that there's a problem when they are being emitted at such a high rate (500 per second). They're just filling up disk space and/or wasting networking bandwidth at that point.
I already asked about this but lemme point it out again: have you guys looked at
drivers/ras/cec.c?
We'll have a closer look. Thanks for the pointer!
Tyler
Quoted 16 lines
With that there won't be *any* error reports in dmesg and it will even poison and offline pages which generate excessive errors so that ...
Quoted 3 lines
Of course, the best course of action here is to service the machine but there's still a period of time between the CE errors popping up and the machine being serviced.
... you'll have ample time to service the machine.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Quoted 11 lines · Tyler Hicks
Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Tyler Hicks
The thought was that the full stream of log messages isn't necessary to notice that there's a problem when they are being emitted at such a high rate (500 per second). They're just filling up disk space and/or wasting networking bandwidth at that point.
I already asked about this but lemme point it out again: have you guys looked at
drivers/ras/cec.c?We'll have a closer look. Thanks for the pointer!
This is x86-specific and not applicable in our situation.
Tyler
Quoted 19 lines
Tyler
Quoted 16 lines
With that there won't be *any* error reports in dmesg and it will even poison and offline pages which generate excessive errors so that ...
Quoted 3 lines
Of course, the best course of action here is to service the machine but there's still a period of time between the CE errors popping up and the machine being serviced.
... you'll have ample time to service the machine.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Borislav Petkov
maintainerQuoted 1 line · Tyler Hicks
This is x86-specific
That's because it is used by x86 currently. It shouldn't be hard to use it on another arch though as the machinery is pretty generic.
Quoted 1 line
and not applicable in our situation.
What is your situation? ARM?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 9 lines · Borislav Petkov
Quoted 1 line · Tyler Hicks
This is x86-specific
That's because it is used by x86 currently. It shouldn't be hard to use it on another arch though as the machinery is pretty generic.
Quoted 1 line
and not applicable in our situation.
What is your situation? ARM?
Yes, though I'm not sure if those additional features are important/useful enough for us to generalize that driver. The main motivation here was just to prevent storage/network from being flooded by obviously-bad nodes that haven't been offlined yet. :)
Lei and others on cc will need to evaluate porting cec.c and what it will gain them. Thanks again.
Tyler
Quoted 7 lines
Signature · 5 lines
-- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Luck, Tony
maintainerQuoted 9 lines
Quoted 1 line
What is your situation? ARM?
Yes, though I'm not sure if those additional features are important/useful enough for us to generalize that driver. The main motivation here was just to prevent storage/network from being flooded by obviously-bad nodes that haven't been offlined yet. :)
Lei and others on cc will need to evaluate porting
cec.cand what it will gain them. Thanks again.
Tyler,
You might also look at the x86 "storm" detection code (tl;dr version "If error interrupts are coming too fast, turn off the interrupts and poll").
-Tony
Em Wed, 5 May 2021 18:01:52 -0500
Tyler Hicks <[email protected]> escreveu:
Quoted 7 lines · Borislav Petkov
Quoted 6 lines · Tyler Hicks
Quoted 1 line
This is x86-specific
That's because it is used by x86 currently. It shouldn't be hard to use it on another arch though as the machinery is pretty generic.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
and not applicable in our situation.
What is your situation? ARM?
Yes, though I'm not sure if those additional features are important/useful enough for us to generalize that driver. The main motivation here was just to prevent storage/network from being flooded by obviously-bad nodes that haven't been offlined yet. :)
Well, if a machine starts to produce 500+ errors per second, then it should be offlined as soon as possible, as otherwise bad results will be produced ;-)
See, the CE error reporting mechanism is meant to be used together with some error correction code algorithm like the ones used on ECC memories. Such algorithms are designed to detect a single errored bit with a change usually at the ~10⁻4 to 10^-7 order (the precision depends on how many bits are used and what algorithm is used), but if there are two wrong bits at the same word, the chance to detect is a lot lower.
So, keeping the server enabled up to the point that it would consume enough resources at the storage/network to bother someone sounds a terrible idea, as sooner or later it will miss an error or produce an uncorrected event ;-)
Besides that, if you're running rasdaemon to capture the hardware errors, the storage will also be flooded by something like that, even if you disable them from going to syslog via sys/module/edac_core/parameters/edac_mc_log_ce.
Now, the question is: are those 500+ errors per second a real hardware problem, or is it due to some broken error report mechanism?
In the latter case, the driver or the hardware that it is producing invalid errors should be fixed.
Quoted 3 lines
Lei and others on cc will need to evaluate porting
cec.cand what it will gain them. Thanks again.
Regards,
Mauro