AMD VM crashing on deferred memory error injection
Subsystems (3)
v3 cover AMD VM crashing on deferred memory error injection
From: William Roche <[email protected]>
v3 changes:
- Commit title and message changed to put the emphasis on SMCA access correctness - Borislav Petkov feedback.
v2 changes:
- Commit title changed to:
x86/mce/amd: Fix VM crash during deferred error handling
- Commit message with capitalized QEMU and KVM as well as the imperative statement suggested by Yazen
- "CC stable" tag placed after "Signed-off-by"
(The documentation asks for "the sign-off area" without more details)
- blank line added to separate SMCA code block and the update of MCA_STATUS.
--
After the integration of the following commit:
7cb735d7c0cb x86/mce: Unify AMD DFR handler with MCA Polling
A problem was found with AMD Qemu VM - it started to crash when dealing with deferred memory error injection with a stack trace like:
mce: MSR access error: WRMSR to 0xc0002098 (tried to write 0x0000000000000000) at rIP: 0xffffffff8229894d (mce_wrmsrq+0x1d/0x60)
amd_clear_bank+0x6e/0x70 machine_check_poll+0x228/0x2e0 ? __pfx_mce_timer_fn+0x10/0x10 mce_timer_fn+0xb1/0x130 ? __pfx_mce_timer_fn+0x10/0x10 call_timer_fn+0x26/0x120 __run_timers+0x202/0x290 run_timer_softirq+0x49/0x100 handle_softirqs+0xeb/0x2c0 __irq_exit_rcu+0xda/0x100 sysvec_apic_timer_interrupt+0x71/0x90 [...] Kernel panic - not syncing: MCA architectural violation!
See the discussion at:
https://lore.kernel.org/all/[email protected]/
We identified a problem with SMCA specific registers access from non-SMCA platforms like a QEMU/KVM machine.
This patch is checkpatch.pl clean.
Unit test of memory error injection works fine with it.
William Roche (1):
x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
arch/x86/kernel/cpu/mce/amd.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.3
PATCH v3 1/1 x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
From: William Roche <[email protected]>
Access to SMCA specific registers like MCA_DESTAT should only be done after having checked the smca bit. Avoiding a non-SMCA machine (like AMD QEMU/KVM VMs) crash during deferred error handling.
--- arch/x86/kernel/cpu/mce/amd.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/mce/amd.c +11 -6
@@ -875,13 +875,18 @@ void amd_clear_bank(struct mce *m) {amd_reset_thr_limit (m->bank); -/* Clear MCA_DESTAT for all deferred errors even those logged in MCA_STATUS. */ -if (m->status &MCI_STATUS_DEFERRED ) -mce_wrmsrq (MSR_AMD64_SMCA_MCx_DESTAT (m->bank),0 ); +if (mce_flags.smca) { +/* +* Clear MCA_DESTAT for all deferred errors even those +* logged in MCA_STATUS. +*/ +if (m->status &MCI_STATUS_DEFERRED ) +mce_wrmsrq (MSR_AMD64_SMCA_MCx_DESTAT (m->bank),0 ); -/* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ -if (m->kflags &MCE_CHECK_DFR_REGS ) -return ; +/* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ +if (m->kflags &MCE_CHECK_DFR_REGS ) +return ; + }mce_wrmsrq (mca_msr_reg (m->bank,MCA_STATUS ),0 ); }
Signature · 1 line
2.47.3
Re: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 5 lines · “William Roche
From: William Roche <[email protected]>
Access to SMCA specific registers like
MCA_DESTATshould only be done after having checked the smca bit. Avoiding a non-SMCA machine (like AMD QEMU/KVM VMs) crash during deferred error handling.
Not good enough. I rewrote it to:
Author: William Roche <[email protected]> Date: Tue Mar 17 10:38:10 2026 +0000
x86/mce/amd: Check SMCA feature bit before accessing SMCA MSRs
People do effort to inject MCEs into guests in order to simulate/test handling of real hardware errors. These efforts are of a questionable nature because, for one, a guest cannot really make any assumptions about the underlying machine and especially which MSR accesses the hypervisor filters and which it doesn't. See Link tag for the whole background.
However, regardless of virtualization or not, access to SMCA-specific registers like MCA_DESTAT should only be done after having checked the smca feature bit. And there are AMD machines like Bulldozer (the one before Zen1) which do support deferred errors but are not SMCA machines.
Therefore, properly check the feature bit before accessing related MSRs.
[ bp: Rewrite commit message. ]
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 27 lines · Borislav Petkov
Quoted 5 lines · “William Roche
From: William Roche <[email protected]>
Access to SMCA specific registers like
MCA_DESTATshould only be done after having checked the smca bit. Avoiding a non-SMCA machine (like AMD QEMU/KVM VMs) crash during deferred error handling.Not good enough. I rewrote it to:
Author: William Roche <[email protected]> Date: Tue Mar 17 10:38:10 2026 +0000x86/mce/amd: Check SMCA feature bit before accessing SMCA MSRs
People do effort to inject MCEs into guests in order to simulate/test handling of real hardware errors. These efforts are of a questionable nature because, for one, a guest cannot really make any assumptions about the underlying machine and especially which MSR accesses the hypervisor filters and which it doesn't. See Link tag for the whole background.
However, regardless of virtualization or not, access to SMCA-specific registers like
MCA_DESTATshould only be done after having checked the smca feature bit. And there are AMD machines like Bulldozer (the one before Zen1) which do support deferred errors but are not SMCA machines.Therefore, properly check the feature bit before accessing related MSRs.
Quoted 8 lines
[ bp: Rewrite commit message. ]
Fixes 7cb735d7c0cb ("x86/mce: Unify AMD DFR handler with MCA Polling")Signed-off-by William Roche <[email protected]>Signed-off-by Borislav Petkov (AMD) <[email protected]>Reviewed-by Yazen Ghannam <[email protected]>
Thank you.
William.
Borislav Petkov
maintainerRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 31 lines · William Roche
Quoted 22 lines · Borislav Petkov
Quoted 5 lines · “William Roche
From: William Roche <[email protected]>
Access to SMCA specific registers like
MCA_DESTATshould only be done after having checked the smca bit. Avoiding a non-SMCA machine (like AMD QEMU/KVM VMs) crash during deferred error handling.Not good enough. I rewrote it to:
Author: William Roche <[email protected]> Date: Tue Mar 17 10:38:10 2026 +0000x86/mce/amd: Check SMCA feature bit before accessing SMCA MSRs People do effort to inject MCEs into guests in order to simulate/test handling of real hardware errors. These efforts are of a questionable nature because, for one, a guest cannot really make any assumptions about the underlying machine and especially which MSR accesses the hypervisor filters and which it doesn't. See Link tag for the whole background.
However, regardless of virtualization or not, access to SMCA-specific registers likeMCA_DESTATshould only be done after having checked the smca feature bit. And there are AMD machines like Bulldozer (the one before Zen1) which do support deferred errors but are not SMCA machines. Therefore, properly check the feature bit before accessing related MSRs.Quoted 7 lines
[ bp: Rewrite commit message. ]Fixes 7cb735d7c0cb ("x86/mce: Unify AMD DFR handler with MCA Polling")Signed-off-by William Roche <[email protected]>Signed-off-by Borislav Petkov (AMD) <[email protected]>Reviewed-by Yazen Ghannam <[email protected]>
Quoted 1 line
Thank you.
Rewrote it again after talking to Yazen. A patch needs to have the proper justification why it exists!
x86/mce/amd: Check SMCA feature bit before accessing SMCA MSRs
People do effort to inject MCEs into guests in order to simulate/test handling of hardware errors. The real use case behind it is testing the handling of SIGBUS which the memory failure code sends to the process.
If that process is QEMU, instead of killing the whole guest, the MCE can be injected into the guest kernel so that latter can attempt proper handling and kill the user *process* in the guest, instead, which caused the MCE. The assumption being here that the whole injection flow can supply enough information that the guest kernel can poinpoint the right process. But that's a different topic...
Regardless of virtualization or not, access to SMCA-specific registers like MCA_DESTAT should only be done after having checked the smca feature bit. And there are AMD machines like Bulldozer (the one before Zen1) which do support deferred errors but are not SMCA machines.
Therefore, properly check the feature bit before accessing related MSRs.
[ bp: Rewrite commit message. ]
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
I just wanted to give a small precision about the VM error relay mechanism:
Quoted 9 lines · Borislav Petkov
Rewrote it again after talking to Yazen. A patch needs to have the proper justification why it exists!
x86/mce/amd: Check SMCA feature bit before accessing SMCA MSRs
People do effort to inject MCEs into guests in order to simulate/test handling of hardware errors. The real use case behind it is testing the handling of SIGBUS which the memory failure code sends to the process.
Quoted 6 lines
If that process is QEMU, instead of killing the whole guest, the MCE can be injected into the guest kernel so that latter can attempt proper handling and kill the user *process* in the guest, instead, which caused the MCE. The assumption being here that the whole injection flow can supply enough information that the guest kernel can poinpoint the right process. But that's a different topic...
Relaying the error to the guest doesn't only have a value to target a VM process but also deal with free memory or clean file cache memory impacted etc... Cases where a memory error may not crash the kernel can benefit to the VM too (Kernel RAS features that are, as you said, a different topic :) )
There is also a small typo in "pinpoint"
Quoted 9 lines
Regardless of virtualization or not, access to SMCA-specific registers like
MCA_DESTATshould only be done after having checked the smca feature bit. And there are AMD machines like Bulldozer (the one before Zen1) which do support deferred errors but are not SMCA machines.Therefore, properly check the feature bit before accessing related MSRs.
[ bp: Rewrite commit message. ]
Thank you very much for your feedback !
William.
Borislav Petkov
maintainerRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 4 lines · William Roche
Relaying the error to the guest doesn't only have a value to target a VM process but also deal with free memory or clean file cache memory impacted etc... Cases where a memory error may not crash the kernel can benefit to the VM too
I don't understand - what do you mean with "free memory or clean file cache memory"?
Quoted 1 line
There is also a small typo in "pinpoint"
Ack, fixed.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 8 lines · Borislav Petkov
Quoted 4 lines · William Roche
Relaying the error to the guest doesn't only have a value to target a VM process but also deal with free memory or clean file cache memory impacted etc... Cases where a memory error may not crash the kernel can benefit to the VM too
I don't understand - what do you mean with "free memory or clean file cache memory"?
The physical address of an uncorrected memory error (if/when it can be identified) can give a chance to a kernel reaction depending on the state (and type) of the impacted memory -- as implemented in mm/memory-failure.c with error_states[], me_pagecache_clean() or try_memory_failure()...
The Kernel can try to "deal" with the error. The process case (with its SIGBUS) is probably the most common one, but a few kernel memory pages impacted by a memory error could be isolated (poisoned) without requiring a kernel crash. Free memory pages or clean page cache pages could be an example of that, they are poisoned and should not be used by the system after that. The kernel can also return EIO error on poisoned page cache failed access attempt, etc...
These mechanisms are implemented for the bare-metal running kernel, but what is really interesting when relaying the error to a VM is that its kernel can, in some cases, also benefit from these mechanisms. And having a chance (even small) to avoid a VM crash is a significant gain for virtualized workload.
Just giving my point of view on why we care about VM relayed memory errors :)
William.
Borislav Petkov
maintainerRe: [PATCH v3 1/1] x86/mce/amd: Guard SMCA DESTAT access on non-SMCA machines
Quoted 13 lines · William Roche
The physical address of an uncorrected memory error (if/when it can be identified) can give a chance to a kernel reaction depending on the state (and type) of the impacted memory -- as implemented in
mm/memory-failure.cwith error_states[],me_pagecache_clean()ortry_memory_failure()...The Kernel can try to "deal" with the error. The process case (with its SIGBUS) is probably the most common one, but a few kernel memory pages impacted by a memory error could be isolated (poisoned) without requiring a kernel crash. Free memory pages or clean page cache pages could be an example of that, they are poisoned and should not be used by the system after that. The kernel can also return EIO error on poisoned page cache failed access attempt, etc...
Quoted 8 lines
These mechanisms are implemented for the bare-metal running kernel, but what is really interesting when relaying the error to a VM is that its kernel can, in some cases, also benefit from these mechanisms. And having a chance (even small) to avoid a VM crash is a significant gain for virtualized workload.
Just giving my point of view on why we care about VM relayed memory errors :)
Ah, you want to be able to handle an error belonging to a guest, regardless of which part it hits. As in, the guest memory hit could be pagecache, free memory, etc... anything that would prevent the guest from dying unnecessary death.
Ack, makes sense.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette