x86/MCE: Get microcode revision from cpu_data instead of boot_cpu_data
Subsystems (3)
PATCH 1/1 x86/MCE: Get microcode revision from cpu_data instead of boot_cpu_data
Commit fa94d0c6e0f3 ("x86/MCE: Save microcode revision in machine check records") extended MCE entries to report the microcode revision taken from boot_cpu_data. Unfortunately, boot_cpu_data isn't updated on late microcode loading, thus making MCE entries slightly incorrect. Use cpu_data instead, which is updated on late microcode loading. This also fixes the corner case in which the microcode revision isn't coherent across CPUs (which may happen on late microcode loading failure).
--- arch/x86/kernel/cpu/mce/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/core.c +1 -1
@@ -127,7 +127,7 @@ void mce_setup(struct mce *m) m->apicid =cpu_data (m->extcpu).topo.initial_apicid; m->mcgcap =__rdmsr (MSR_IA32_MCG_CAP ); m->ppin =cpu_data (m->extcpu).ppin; - m->microcode = boot_cpu_data.microcode; + m->microcode =cpu_data (m->extcpu).microcode; } DEFINE_PER_CPU(struct mce, injectm);
Signature · 12 lines
2.33.0 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879
Tony Luck
maintainerQuoted 4 lines · Filippo Sironi
Commit
fa94d0c6e0f3("x86/MCE: Save microcode revision in machine check records") extended MCE entries to report the microcode revision taken fromboot_cpu_data. Unfortunately,boot_cpu_dataisn't updated on late microcode loading, thus making MCE entries slightly incorrect. Use
This code in intel.c:apply_microcode_late() looks like it tries to update boot_cpu_data:
466 cpu_data(cpu).microcode = uci->cpu_sig.rev;
467 if (!cpu)
468 boot_cpu_data.microcode = uci->cpu_sig.rev;
Is that not working for some reason?
Quoted 3 lines
cpu_datainstead, which is updated on late microcode loading. This also fixes the corner case in which the microcode revision isn't coherent across CPUs (which may happen on late microcode loading failure).
But this does seem a worthwhile change to help diagnose things if late load is somehow only applied to some subset of CPUs.
Quoted 21 lines
Signed-off-by Filippo Sironi <[email protected]>--- arch/x86/kernel/cpu/mce/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/mce/core.c +1
-1@@ -127,7 +127,7 @@ void mce_setup(struct mce *m) m->apicid =cpu_data (m->extcpu).topo.initial_apicid; m->mcgcap =__rdmsr (MSR_IA32_MCG_CAP ); m->ppin =cpu_data (m->extcpu).ppin; - m->microcode = boot_cpu_data.microcode; + m->microcode =cpu_data (m->extcpu).microcode; } DEFINE_PER_CPU(struct mce, injectm);Signature · 1 line
2.33.0
-Tony
Borislav Petkov
maintainerQuoted 2 lines · Tony Luck
But this does seem a worthwhile change to help diagnose things if late load is somehow only applied to some subset of CPUs.
We already do that, see load_late_stop_cpus().
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 10 lines · Filippo Sironi
Quoted 4 lines
Commit
fa94d0c6e0f3("x86/MCE: Save microcode revision in machine check records") extended MCE entries to report the microcode revision taken fromboot_cpu_data. Unfortunately,boot_cpu_dataisn't updated on late microcode loading, thus making MCE entries slightly incorrect. UseThis code in intel.c:apply_microcode_late() looks like it tries to update
boot_cpu_data:
Quoted 6 lines
466 cpu_data(cpu).microcode =
uci->cpu_sig.rev;
467 if (!cpu)
468 boot_cpu_data.microcode =uci->cpu_sig.rev;Is that not working for some reason?
It is... I had this change in our tree for a long long while and just realized that the issue of boot_cpu_data not being updated was addressed with commit 370a132bb222 ("x86/microcode: Make sure boot_cpu_data.microcode is up-to-date").
Quoted 7 lines
Quoted 3 lines
cpu_datainstead, which is updated on late microcode loading. This also fixes the corner case in which the microcode revision isn't coherent across CPUs (which may happen on late microcode loading failure).But this does seem a worthwhile change to help diagnose things if late load is somehow only applied to some subset of CPUs.
Yes, but, as Boris points out:
Quoted 4 lines
Quoted 2 lines
But this does seem a worthwhile change to help diagnose things if late load is somehow only applied to some subset of CPUs.
We already do that, see
load_late_stop_cpus().
Boris, I just took a quick look and I might be missing something. If cores fail to load the microcode or timeout, we taint the kernel, print an error message, and then bubble up an error to userspace via:
load_late_stop_cpusload_late_lockedreload_store
Right?
We would take servers that fail out of production; however, for others it might be interesting to have the correct information. The patch - with a reworked commit message - might still be useful to a few.
Filippo
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Borislav Petkov
maintainerQuoted 9 lines · Filippo
Boris, I just took a quick look and I might be missing something. If cores fail to load the microcode or timeout, we taint the kernel, print an error message, and then bubble up an error to userspace via:
load_late_stop_cpusload_late_lockedreload_storeRight?
Yap.
Quoted 1 line
We would take servers that fail out of production;
And I'd like to hear about such issues. We added this failure checking only recently because something might go wrong and we want to warn. But it all updates fine here so kinda hard to test.
My expectation is that if microcode fails loading on a subset of machines, the machine would more or less freeze. Depending, ofc, on what the microcode is updating...
Quoted 3 lines
however, for others it might be interesting to have the correct information. The patch - with a reworked commit message - might still be useful to a few.
https://lore.kernel.org/r/[email protected]
:)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 10 lines
Quoted 9 lines
Boris, I just took a quick look and I might be missing something. If cores fail to load the microcode or timeout, we taint the kernel, print an error message, and then bubble up an error to userspace via:
load_late_stop_cpusload_late_lockedreload_storeRight?
Quoted 7 lines
Yap.
Quoted 1 line
We would take servers that fail out of production;
And I'd like to hear about such issues. We added this failure checking only recently because something might go wrong and we want to warn. But it all updates fine here so kinda hard to test.
In a very large fleet, let's say that we have a handful of DPMs when considering the entire processor, which means that in terms of cores, the defect rate is much much lower.
What we've seen in these cases is that early loading - through the BIOS, I actually never tried via the hypervisor - is successful while late loading consistently fails. When it fails, we've seen two cases: 1/ the core still reports the old microcode version or 2/ the core reports a bogus microcode version (0xfffffffe is quite common, at least on Intel).
Quoted 3 lines
My expectation is that if microcode fails loading on a subset of machines, the machine would more or less freeze. Depending, ofc, on what the microcode is updating...
It's bi-modal. We've seen servers that move along till we take them out of production as well as servers that fail with an MCE of some sort likely leading to a CATERR/IERR.
Quoted 9 lines
Quoted 3 lines
however, for others it might be interesting to have the correct information. The patch - with a reworked commit message - might still be useful to a few.
https://lore.kernel.org/r/[email protected] <mailto:[email protected]>
:)
:looking:
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Borislav Petkov
maintainerQuoted 1 line · Filippo Sironi
Unfortunately,
boot_cpu_dataisn't updated on late microcode loading,
Looks good to me here:
[ 50.881231] microcode: Unsafe microcode update: Microcode header does not specify a required min version [ 50.892577] microcode: Late microcode loading without minimal revision check. [ 50.900099] microcode: You should switch to early loading, if possible. [ 50.958919] microcode: load: updated on 6 primary CPUs with 0 siblings [ 50.965856] microcode: revision: 0xbe -> 0xf0
That comes from:
pr_info("revision: 0x%x -> 0x%x\n", old_rev, boot_cpu_data.microcode);Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette