x86, MCE: Drop the default decoding notifier

From: Borislav Petkov <[email protected]>

Signed-off-by Prarit Bhargava <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Quoted 3 lines · Borislav Petkov

From: Borislav Petkov <[email protected]>

Signed-off-by Prarit Bhargava <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Hm, that From/SOB combination does not look right.

Review

Quoted 3 lines · Borislav Petkov

(quoted diff) +3 -0

+	if (ret != NOTIFY_STOP)
+		printk_ratelimited(KERN_EMERG HW_ERR
+				   "Run the above through 'mcelog --ascii'\n");

pr_emerg_ratelimited() would allow you to keep that broken line happy?

Quoted 8 lines · Ingo Molnar

Review

Quoted 3 lines · Borislav Petkov

(quoted diff) +3 -0

+	if (ret != NOTIFY_STOP)
+		printk_ratelimited(KERN_EMERG HW_ERR
+				   "Run the above through 'mcelog --ascii'\n");

pr_emerg_ratelimited() would allow you to keep that broken line happy?

Not entirely. I had to do the exit-early-to-save-an-indentation-level trick also :)

--
From: Borislav Petkov <[email protected]>
Date: Wed, 13 Apr 2011 14:32:06 +0200
Subject: [PATCH -v2.1 2/2] x86, MCE: Drop the default decoding notifier

The default notifier doesn't make a lot of sense to call in the correctable errors case. Drop it and emit the mcelog decoding hint only in the uncorrectable errors case and when no notifier is registered. Also, limit issuing the "mcelog --ascii" message in the rare case when we dump unreported CEs before panicking.

While at it, remove unused old x86_mce_decode_callback from the header.

Quoted 4 lines

From: Borislav Petkov <[email protected]>
Date: Wed, 13 Apr 2011 14:32:06 +0200
Subject: [PATCH -v2.1 2/2] x86, MCE: Drop the default decoding notifier

Quoted 8 lines

The default notifier doesn't make a lot of sense to call in the correctable errors case. Drop it and emit the mcelog decoding hint only in the uncorrectable errors case and when no notifier is registered. Also, limit issuing the "mcelog --ascii" message in the rare case when we dump unreported CEs before panicking.

While at it, remove unused old x86_mce_decode_callback from the header.

Can we please print something if we please log something in the case of a correctable error, when we only report it via mcelog?

I have a stupid recent intel cpu here that hits that case and without the default x86_mce_decode_callback I wouldn't have even known that I am getting something like 50 correctable errors an hour on one of my machines. In particular I am it hits so often I am seeing:
"mce_notify_irq: 2 callbacks suppressed". I need to get those dimms replaced soon because in a new product I simply can't imagine that many correctable errors.

Quoted 16 lines · Eric W. Biederman

Quoted 4 lines

From: Borislav Petkov <[email protected]>
Date: Wed, 13 Apr 2011 14:32:06 +0200
Subject: [PATCH -v2.1 2/2] x86, MCE: Drop the default decoding notifier

Quoted 8 lines

The default notifier doesn't make a lot of sense to call in the correctable errors case. Drop it and emit the mcelog decoding hint only in the uncorrectable errors case and when no notifier is registered. Also, limit issuing the "mcelog --ascii" message in the rare case when we dump unreported CEs before panicking.

While at it, remove unused old x86_mce_decode_callback from the header.

Can we please print something if we please log something in the case of a correctable error, when we only report it via mcelog?

Quoted 7 lines

I have a stupid recent intel cpu here that hits that case and without the default x86_mce_decode_callback I wouldn't have even known that I am getting something like 50 correctable errors an hour on one of my machines. In particular I am it hits so often I am seeing:
"mce_notify_irq: 2 callbacks suppressed". I need to get those dimms replaced soon because in a new product I simply can't imagine that many correctable errors.

Isn't there a mcelog daemon or something that polls /dev/mcelog and tells you about those DRAM ECCs in some log file where you're supposed to look? :)

Quoted 25 lines · Borislav Petkov

Quoted 23 lines · Eric W. Biederman

Quoted 4 lines

From: Borislav Petkov <[email protected]>
Date: Wed, 13 Apr 2011 14:32:06 +0200
Subject: [PATCH -v2.1 2/2] x86, MCE: Drop the default decoding notifier

Quoted 8 lines

The default notifier doesn't make a lot of sense to call in the correctable errors case. Drop it and emit the mcelog decoding hint only in the uncorrectable errors case and when no notifier is registered. Also, limit issuing the "mcelog --ascii" message in the rare case when we dump unreported CEs before panicking.

While at it, remove unused old x86_mce_decode_callback from the header.

Can we please print something if we please log something in the case of a correctable error, when we only report it via mcelog?

I have a stupid recent intel cpu here that hits that case and without the default x86_mce_decode_callback I wouldn't have even known that I am getting something like 50 correctable errors an hour on one of my machines. In particular I am it hits so often I am seeing:
"mce_notify_irq: 2 callbacks suppressed". I need to get those dimms replaced soon because in a new product I simply can't imagine that many correctable errors.

Quoted 3 lines

Isn't there a mcelog daemon or something that polls /dev/mcelog and tells you about those DRAM ECCs in some log file where you're supposed to look? :)

On fedora 14 there is a cron job that writes to /var/log/mcelog, and does not go through syslog. But you have to be proactive and look there. If the people who work on this code can't even remember where to look I can't imagine how anyone else can remember.

Which is why I object to the removal of the one printk that told me something was broken on my machine.

So far from what I have seen /dev/mcelog and the userspace mcelog is over complicated and near useless. It seems to focused around the notion that "This is not a software problem, please do not bug Andi Kleen about it"

On Tue, Apr 26, 2011 at 05:06:39PM -0400, Eric W. Biederman wrote:

Ha!

I'm working exactly in the opposite direction actually - drop mcelog and make RAS much more user friendly. As a first step, this is why we have all that MCE decoding code for AMD hw and when you get an error, you can't miss it:

Apr 20 21:08:24 kepek kernel: [  300.816122] [Hardware Error]: MC4_STATUS[Over|CE|MiscV|-|AddrV|CECC]: 0xdc00c000c6080a13
Apr 20 21:08:24 kepek kernel: [  300.825156] [Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected on the NB.
Apr 20 21:08:24 kepek kernel: [  300.825167] EDAC amd64 MC0: CE ERROR_ADDRESS= 0x4171fe380
Apr 20 21:08:24 kepek kernel: [  300.825257] EDAC MC0: CE page 0x4171fe, offset 0x380, grain 0, syndrome 0xc601, row 3, channel 0, label "": amd64_

edac

or this:

Apr 15 16:54:17 kepek kernel: [72187.027059] [Hardware Error]: MC0_STATUS[-|UE|-|-|AddrV|UECC]: 0xb400210000010016
Apr 15 16:54:17 kepek kernel: [72187.027059] [Hardware Error]: Data Cache Error: L2 TLB multimatch.
Apr 15 16:54:17 kepek kernel: [72187.027059] [Hardware Error]: cache level: L2, tx: DATA

There's also this RAS daemon I'm hacking on which uses perf to carry error information to userspace and do more than reporting it. For example, server farm guys don't want to scan syslog for every CECC error but rather have it collected somewhere on one machine, maybe over the network, etc, etc.

So now is the time to speak up and let me know how you would like to get the error reported? In general, what should be done differently in Linux wrt to RAS.

Quoted 2 lines

Which is why I object to the removal of the one printk that told me something was broken on my machine.

I dunno, maybe it's time we moved the MCE decoding functionality which is shared by most of x86 into core code. Ingo, Peter, Thomas, what do you guys think?

This'll at least put something in the logs that is sensible instead of useless strings which tell the users what to do next. Also, we can ratelimit it so that DIMMs generating too many CECCs don't flood them too much. Hmm...

Sure. Although any DIMM that is generating so many correctable errors that you need to rate limit it in the kernel, won't likely to confine itself to correctable errors.

Still it can happen that things are so bad that you do need to rate limit it in the kernel. Still with those you start wondering "How did this machine boot?" So printk_ratelimit sounds like a fine idea.

Quoted 7 lines

Sure. Although any DIMM that is generating so many correctable errors that you need to rate limit it in the kernel, won't likely to confine itself to correctable errors.

Still it can happen that things are so bad that you do need to rate limit it in the kernel. Still with those you start wondering "How did this machine boot?" So printk_ratelimit sounds like a fine idea.

Perhaps we really want thresholds rather than rate limits (for corrected errors). One corrected error shouldn't cause any but the most paranoid to worry. A couple of errors from the same DIMM close together might be some cause for concern, but could just be happenstance. Enough errors that rate limiting looks useful, and you are into "something needs to be done" territory.

Quoted 25 lines · Borislav Petkov

Quoted 23 lines · Eric W. Biederman

Quoted 4 lines

From: Borislav Petkov <[email protected]>
Date: Wed, 13 Apr 2011 14:32:06 +0200
Subject: [PATCH -v2.1 2/2] x86, MCE: Drop the default decoding notifier

Quoted 8 lines

The default notifier doesn't make a lot of sense to call in the correctable errors case. Drop it and emit the mcelog decoding hint only in the uncorrectable errors case and when no notifier is registered. Also, limit issuing the "mcelog --ascii" message in the rare case when we dump unreported CEs before panicking.

While at it, remove unused old x86_mce_decode_callback from the header.

Can we please print something if we please log something in the case of a correctable error, when we only report it via mcelog?

I have a stupid recent intel cpu here that hits that case and without the default x86_mce_decode_callback I wouldn't have even known that I am getting something like 50 correctable errors an hour on one of my machines. In particular I am it hits so often I am seeing:
"mce_notify_irq: 2 callbacks suppressed". I need to get those dimms replaced soon because in a new product I simply can't imagine that many correctable errors.

Quoted 3 lines

Isn't there a mcelog daemon or something that polls /dev/mcelog and tells you about those DRAM ECCs in some log file where you're supposed to look? :)

On fedora 14 there is a cron job that writes to /var/log/mcelog, and does not go through syslog.

But you have to be proactive and look
there. If the people who work on this code can't even remember where to look I can't imagine how anyone else can remember. Which is why I object to the removal of the one printk that told me something was broken on my machine.

So far from what I have seen /dev/mcelog and the userspace mcelog is over complicated and near useless.

It seems to focused around the
notion that "This is not a software problem, please do not bug Andi Kleen about it"

Well it is a hardware problem so I do need to RMA that hardware. Sigh.

Eric