Skip to contentSkip to conversation

x86/mce: Initialize "bank" when we find a fatal error in mce_no_way_out()

PATCH In review Tony Luck 3 participants 5 messages Feb 1, 2019 – Feb 3, 2019

Subsystems (2)
lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

PATCH 1/1 x86/mce: Initialize "bank" when we find a fatal error in mce_no_way_out()

Internal injection testing crashed with a console log that said:

mce: [Hardware Error]: CPU 7: Machine Check Exception: f Bank 0: bd80000000100134

This caused a lot of head scratching because the MCACOD (bits 15:0) of that status is a signature from an L1 data cache error. But Linux says that it found it in "Bank 0", which on this model CPU only reports L1 instruction cache errors.

The answer was that Linux doesn't initialize "m->bank" in the case that it finds a fatal error in the mce_no_way_out() pre-scan of banks. If this was a local machine check, then we pass this partially initialized "struct mce" to mce_panic().

Fix is simple. Just initialize m->bank in the case that we found a fatal error.

Fixes 40c36e2741d7 ("x86/mce: Fix incorrect "Machine check from unknown source" message")
Cc [email protected] # v4.18 Note pre-v5.0 arch/x86/kernel/cpu/mce/core.c was called arch/x86/kernel/cpu/mcheck/mce.c
---
 arch/x86/kernel/cpu/mce/core.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kernel/cpu/mce/core.c +1 -0
index 672c7225cb1b..6ce290c506d9 100644
@@ -784,6 +784,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,
 			quirk_no_way_out(i, m, regs);
 
 		if (mce_severity(m, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) {
+			m->bank = i;
 			mce_read_aux(m, i);
 			*msg = tmp;
 			return 1;
Signature · 1 line
2.19.1
maintainer
#1 lore raw
Quoted 26 lines · Tony Luck

Internal injection testing crashed with a console log that said:

mce: [Hardware Error]: CPU 7: Machine Check Exception: f Bank 0: bd80000000100134

This caused a lot of head scratching because the MCACOD (bits 15:0) of that status is a signature from an L1 data cache error. But Linux says that it found it in "Bank 0", which on this model CPU only reports L1 instruction cache errors.

The answer was that Linux doesn't initialize "m->bank" in the case that it finds a fatal error in the mce_no_way_out() pre-scan of banks. If this was a local machine check, then we pass this partially initialized "struct mce" to mce_panic().

Fix is simple. Just initialize m->bank in the case that we found a fatal error.

Fixes 40c36e2741d7 ("x86/mce: Fix incorrect "Machine check from unknown source" message")
Cc [email protected] # v4.18 Note pre-v5.0 arch/x86/kernel/cpu/mce/core.c was called arch/x86/kernel/cpu/mcheck/mce.c
---
 arch/x86/kernel/cpu/mce/core.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kernel/cpu/mce/core.c +0 -0
index 672c7225cb1b..6ce290c506d9 100644

arch/x86/kernel/cpu/mce/core.c static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp, · line 787

Quoted 4 lines
arch/x86/kernel/cpu/mce/core.c +1 -0
@@ -784,6 +784,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,
 			quirk_no_way_out(i, m, regs);
 
 		if (mce_severity(m, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) {
+			m->bank = i;

So conceptually this write belongs in...

arch/x86/kernel/cpu/mce/core.c static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp, · line 788

Quoted 1 line
arch/x86/kernel/cpu/mce/core.c +0 -0
 			mce_read_aux(m, i);

... this function, i.e., in mce_read_aux() because it gets the bank number passed in already. And our calling pattern when populating struct mce is:

	mce_gather_info()
	mce_read_aux()

so it'll be more robust if we moved it there.

Also, that argument "i" of mce_read_aux() is not very telling and it should be "bank" but that would complicate the stable backporting so if you feel like it, you could do a second, cleanup patch ontop to fix that too.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
author maintainer
#2 lore raw
Quoted 7 lines · Borislav Petkov

arch/x86/kernel/cpu/mce/core.c static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp, · line 787

Quoted 2 lines · Tony Luck
arch/x86/kernel/cpu/mce/core.c +1 -0
 		if (mce_severity(m, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) {
+			m->bank = i;

So conceptually this write belongs in...

Quoted 1 line

mce_read_aux(m, i);

Really? That function starts with the comment

	/*
	 * Read ADDR and MISC registers.
	 */

and the name is pretty descriptive that it is "reading auxiliary" information.

Quoted 7 lines

... this function, i.e., in mce_read_aux() because it gets the bank number passed in already. And our calling pattern when populating struct mce is:

	mce_gather_info()
	mce_read_aux()

Only two exising uses:

1) in machine_check_poll()

	mce_gather_info(&m, NULL);
	for (each bank) {
		m.bank = i;
		mce_read_aux(&m, i);
		...
	}

2) in __mc_scan_banks

	// mce_gather_info() already done in do_machine_check
	for (each bank) {
		m.bank = i;
		mce_read_aux(m, i);
		...
	}
Quoted 2 lines

so it'll be more robust if we moved it there.

It would be redundant to move it there for both
existing uses.

Quoted 4 lines

Also, that argument "i" of mce_read_aux() is not very telling and it should be "bank" but that would complicate the stable backporting so if you feel like it, you could do a second, cleanup patch ontop to fix that too.

Agreed that "bank" is clearer than "i". But also agreed that should be a separate patch for another day.

-Tony

maintainer
#3 lore raw
Quoted 4 lines · Tony
Quoted 1 line

so it'll be more robust if we moved it there.

It would be redundant to move it there for both
existing uses.

Maybe. But if the bank write happens there, it won't be "forgotten" again.

I don't care what the functions are called. If they need to do something more, like *fully* populating struct mce so that a proper record gets logged further down the line, then we need to make them do so. That's the point I'm trying to make.

Sure, the stable fix should be simple for easier backporting but in order to avoid this thing happening again in the future, we should probably look at unifying and making those paths easier to use.

Right now we have

1. mce_setup() - initial population of struct mce, called from mce_gather_info(), a.o.
2. mce_gather_info() collects global regs
3. mce_read_aux() reading aux regs

and at the very end, we hand it into mce_log().

And in-between we poke (or don't poke) in some fields.

IOW, if the struct mce finishing population is concentrated in a single function, it will be much harder to forget setting such fields in the future.

In this case, the devil is in the detail so I'll have a look at this when more time.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#4 lore raw

[tip:x86/urgent] x86/MCE: Initialize mce.bank in the case of a fatal error in mce_no_way_out()

In reply to Tony Luck
x86/MCE: Initialize mce.bank in the case of a fatal error in mce_no_way_out()
Commit-ID:  d28af26faa0b1daf3c692603d46bc4687c16f19e
Gitweb:     https://git.kernel.org/tip/d28af26faa0b1daf3c692603d46bc4687c16f19e
Author:     Tony Luck <[email protected]>
AuthorDate: Thu, 31 Jan 2019 16:33:41 -0800
Committer:  Borislav Petkov <[email protected]>
CommitDate: Sun, 3 Feb 2019 13:24:24 +0100

x86/MCE: Initialize mce.bank in the case of a fatal error in mce_no_way_out()

Internal injection testing crashed with a console log that said:

mce: [Hardware Error]: CPU 7: Machine Check Exception: f Bank 0: bd80000000100134

This caused a lot of head scratching because the MCACOD (bits 15:0) of that status is a signature from an L1 data cache error. But Linux says that it found it in "Bank 0", which on this model CPU only reports L1 instruction cache errors.

The answer was that Linux doesn't initialize "m->bank" in the case that it finds a fatal error in the mce_no_way_out() pre-scan of banks. If this was a local machine check, then this partially initialized struct mce is being passed to mce_panic().

Fix is simple: just initialize m->bank in the case of a fatal error.

Fixes 40c36e2741d7 ("x86/mce: Fix incorrect "Machine check from unknown source" message")
Cc "H. Peter Anvin" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vishal Verma <[email protected]>
Cc x86-ml <[email protected]>
Cc [email protected] # v4.18 Note pre-v5.0 arch/x86/kernel/cpu/mce/core.c was called arch/x86/kernel/cpu/mcheck/mce.c
---
 arch/x86/kernel/cpu/mce/core.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kernel/cpu/mce/core.c +1 -0
index 672c7225cb1b..6ce290c506d9 100644
@@ -784,6 +784,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,
 			quirk_no_way_out(i, m, regs);
 
 		if (mce_severity(m, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) {
+			m->bank = i;
 			mce_read_aux(m, i);
 			*msg = tmp;
 			return 1;