Fix MCE handling on AMD hosts
cover Fix MCE handling on AMD hosts
In the event that a guest process attempts to access memory that has been poisoned in response to a deferred uncorrected MCE, an AMD system will currently generate a SIGBUS error which will result in the entire guest being shutdown. Ideally, we only want to kill the guest process that accessed poisoned memory in this case.
This support has been included in qemu for Intel hosts for a long time, but there are a couple of changes needed for AMD hosts. First, we will need to expose the SUCCOR cpuid bit to guests. Second, we need to modify the MCE injection code to avoid Intel specific behavior when we are running on an AMD host.
John Allen (2):
i386: Add support for SUCCOR feature
i386: Fix MCE support for AMD hosts
target/i386/cpu.c | 2 +- target/i386/cpu.h | 4 ++++ target/i386/helper.c | 4 ++++ target/i386/kvm/kvm.c | 17 +++++++++++------ 4 files changed, 20 insertions(+), 7 deletions(-)
Signature · 1 line
2.39.3
PATCH 1/2 i386: Add support for SUCCOR feature
Add cpuid bit definition for the SUCCOR feature. This cpuid bit is required to be exposed to guests to allow them to handle machine check exceptions on AMD hosts.
--- target/i386/cpu.c | 2 +- target/i386/cpu.h | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-)
target/i386/cpu.c +1 -1
@@ -5874,7 +5874,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,break ;case 0x80000007 : *eax =0 ; - *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ; *ecx =0 ; *edx = env->features[FEAT_8000_0007_EDX ];break ;
target/i386/cpu.h +4 -0
@@ -555,6 +555,7 @@ typedef enum FeatureWord {FEAT_7_1_EAX ,/* CPUID[EAX=7,ECX=1].EAX */ FEAT_8000_0001_EDX ,/* CPUID[8000_0001].EDX */ FEAT_8000_0001_ECX ,/* CPUID[8000_0001].ECX */ +FEAT_8000_0007_EBX ,/* CPUID[8000_0007].EBX */ FEAT_8000_0007_EDX ,/* CPUID[8000_0007].EDX */ FEAT_8000_0008_EBX ,/* CPUID[8000_0008].EBX */ FEAT_C000_0001_EDX, /* CPUID[C000_0001].EDX */
@@ -856,6 +857,9 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];/* Packets which contain IP payload have LIP values */ #define CPUID_14_0_ECX_LIP (1U << 31) +/* RAS Features */ +#define CPUID_8000_0007_EBX_SUCCOR (1U << 1) +/* CLZERO instruction */ #define CPUID_8000_0008_EBX_CLZERO (1U << 0)/* Always save/restore FP error pointers */
Signature · 1 line
2.39.3
Re: [PATCH 1/2] i386: Add support for SUCCOR feature
Hi John,
Thanks for the patches. Few comments below.
Quoted 16 lines · John Allen
Add cpuid bit definition for the SUCCOR feature. This cpuid bit is required to be exposed to guests to allow them to handle machine check exceptions on AMD hosts.
Reported-by William Roche <[email protected]>Signed-off-by John Allen <[email protected]>--- target/i386/cpu.c | 2 +- target/i386/cpu.h | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-)target/i386/cpu.c +0
-0
target/i386/cpu.c void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, · line 5877
Quoted 5 lines
target/i386/cpu.c +1
-1@@ -5874,7 +5874,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,break ;case 0x80000007 : *eax =0 ; - *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ;
This is adding this feature unconditionally which does not seem right. Couple of things.
1. Add the feature word for SUCCOR. Users can add this feature using the feature word "+succor".
2. Also define CPUID_8000_0007_EBX_SUCCOR : In this case, we can add this feature as part of the EPYC Model update.
Thanks
Babu
Re: [PATCH 1/2] i386: Add support for SUCCOR feature
+x86 qemu folks
Quoted 15 lines · Babu
Hi John,
Thanks for the patches. Few comments below.Quoted 11 lines · John Allen
Add cpuid bit definition for the SUCCOR feature. This cpuid bit is required to be exposed to guests to allow them to handle machine check exceptions on AMD hosts.
Reported-by William Roche <[email protected]>Signed-off-by John Allen <[email protected]>--- target/i386/cpu.c | 2 +- target/i386/cpu.h | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-)
Quoted 16 lines
target/i386/cpu.c void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, · line 5877
Quoted 10 lines
target/i386/cpu.c +1
-1@@ -5874,7 +5874,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,break ;case 0x80000007 : *eax =0 ; - *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ;This is adding this feature unconditionally which does not seem right. Couple of things.
1. Add the feature word for SUCCOR. Users can add this feature using the feature word "+succor".
The feature requires no specific SVM/KVM support, it's really just a bit enumerated to the guest that's used to tell whether AMD MCE handling is supported, instead of blantly crashing outright. So I think the thinking here is that GET_SUPPORTED_CPUID won't return SUCCOR and the feature gets filtered out if it's declared as a regular named feature like you do in the CPU models. Making it relevant for any hypervisor kernel.
Quoted 3 lines
2. Also define
CPUID_8000_0007_EBX_SUCCOR: In this case, we can add this feature as part of the EPYC Model update.
how would you set CPUID regardless of the hypervisor enumerates should it not require active emulation by the host? I am sort of interested on what is the right way to do this, as it would be better to do this regardless of the gets advertised by ioctls alike.
Quoted 3 lines
Thanks
Babu
Re: [PATCH 1/2] i386: Add support for SUCCOR feature
In reply to John Allentarget/i386/cpu.c void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, · line 5880
Quoted 7 lines · John Allen
case 0x80000007: *eax = 0;target/i386/cpu.c +1
-1- *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ; *ecx =0 ; *edx = env->features[FEAT_8000_0007_EDX ];break ;
I agree that it needs no hypervisor support, but Babu is right that you cannot add it unconditionally (especially not on Intel processors).
You can special case CPUID_8000_0007_EBX_SUCCOR in kvm_arch_get_supported_cpuid() so that it is added even on old kernels. There are already several such cases. Adding it to KVM is nice to have anyway, so please send a patch for that.
Also, the patch does not compile (probably you missed a prerequisite) as it lacks all the rigamarole that is needed to add FEAT_8000_0007_EBX.
Paolo
Re: [PATCH 1/2] i386: Add support for SUCCOR feature
Quoted 9 lines · Paolo Bonzini
target/i386/cpu.c void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, · line 5880
Quoted 7 lines · John Allen
case 0x80000007: *eax = 0;target/i386/cpu.c +1
-1- *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ; *ecx =0 ; *edx = env->features[FEAT_8000_0007_EDX ];break ;
Quoted 7 lines
I agree that it needs no hypervisor support, but Babu is right that you cannot add it unconditionally (especially not on Intel processors).
You can special case
CPUID_8000_0007_EBX_SUCCORinkvm_arch_get_supported_cpuid()so that it is added even on old kernels. There are already several such cases. Adding it to KVM is nice to have anyway, so please send a patch for that.
By adding it to KVM do you mean adding a patch to the kernel to expose the cpuid bit? Or do you mean just adding the special case to kvm_arch_get_supported_cpuid?
For the kvm_arch_get_supported_cpuid case, I don't understand how this would be different from unconditionally exposing the bit as done above. Can you help me understand what you have in mind for this?
I might add a case like below:
...
} else if (function == 0x80000007 && reg == R_EBX) {
ret |= CPUID_8000_0007_EBX_SUCCOR;...
If we wanted to only expose the bit for AMD cpus, we would then need to call IS_AMD_CPU with the CPUX86State as a paramter which would mean that kvm_arch_get_supported_cpuid and all of its callers would need to take the CPUX86State as a parameter. Is there another way to differentiate between AMD and Intel cpus in this case?
Quoted 3 lines
Also, the patch does not compile (probably you missed a prerequisite) as it lacks all the rigamarole that is needed to add
FEAT_8000_0007_EBX.
I'm not encountering any compilation issues. What are the errors that you are seeing?
Thanks,
John
Re: [PATCH 1/2] i386: Add support for SUCCOR feature
Quoted 26 lines · John Allen
Quoted 9 lines · Paolo Bonzini
target/i386/cpu.c void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, · line 5880
Quoted 7 lines · John Allen
case 0x80000007: *eax = 0;target/i386/cpu.c +1
-1- *ebx =0 ; + *ebx = env->features[FEAT_8000_0007_EBX ] |CPUID_8000_0007_EBX_SUCCOR ; *ecx =0 ; *edx = env->features[FEAT_8000_0007_EDX ];break ;Quoted 7 lines
I agree that it needs no hypervisor support, but Babu is right that you cannot add it unconditionally (especially not on Intel processors).
You can special case
CPUID_8000_0007_EBX_SUCCORinkvm_arch_get_supported_cpuid()so that it is added even on old kernels. There are already several such cases. Adding it to KVM is nice to have anyway, so please send a patch for that.By adding it to KVM do you mean adding a patch to the kernel to expose the cpuid bit? Or do you mean just adding the special case to
kvm_arch_get_supported_cpuid?For the
kvm_arch_get_supported_cpuidcase, I don't understand how this would be different from unconditionally exposing the bit as done above. Can you help me understand what you have in mind for this?
Quoted 6 lines
I might add a case like below:
...} else if (function == 0x80000007 && reg == R_EBX) { ret |= CPUID_8000_0007_EBX_SUCCOR;...
IIUC the thinking here is to cover a hypervisor that doesn't advertise SUCCOR via supported cpuid from the kernel, by adding that elif statement above. So when a CPU model is configured, the filtering of cpuid leafs logic takes place, it doesn't "dissappear" when the values from the kernel isn't set.
Otherwise, if it's advertised by the kernel, everything goes as normal in this filtering logic and everything works.
This presumes that you set "succor" feature string in the CPUID model definition instead of unilaterally defining it like this patch. Such that only AMD CPU models advertise the feature instead of everybody.
Quoted 6 lines
If we wanted to only expose the bit for AMD cpus, we would then need to call
IS_AMD_CPUwith the CPUX86State as a paramter which would mean thatkvm_arch_get_supported_cpuidand all of its callers would need to take the CPUX86State as a parameter. Is there another way to differentiate between AMD and Intel cpus in this case?
There might be an implicit convetion here where the caller is the one selecting all the bits -- so kvm_arch_get_supported_cpuid won't care to validate which vendor supports a given bit. Usually steered down via CPU model names (-cpu Genoa), or named CPU features (-cpu Genoa,-feature-name) e.g. Nothing stops you from trying to use a Intel CPUID on a AMD host. So the IS_AMD_CPU inside kvm_arch_get_supported_cpuid doesn't seem to matter
Joao
PATCH 2/2 i386: Fix MCE support for AMD hosts
In reply to John AllenFor the most part, AMD hosts can use the same MCE injection code as Intel but, there are instances where the qemu implementation is Intel specific. First, MCE deliviery works differently on AMD and does not support broadcast. Second, kvm_mce_inject generates MCEs that include a number of Intel specific status bits. Modify kvm_mce_inject to properly generate MCEs on AMD platforms.
--- target/i386/helper.c | 4 ++++ target/i386/kvm/kvm.c | 17 +++++++++++------ 2 files changed, 15 insertions(+), 6 deletions(-)
target/i386/helper.c +4 -0
@@ -76,6 +76,10 @@ int cpu_x86_support_mca_broadcast(CPUX86State *env)int family =0 ;int model =0 ; +if (IS_AMD_CPU (env)) { +return 0 ; + } +cpu_x86_version (env, &family, &model);if ((family ==6 && model >=14 ) || family >6 ) {return 1 ;
target/i386/kvm/kvm.c +11 -6
@@ -530,16 +530,21 @@ static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code)CPUState *cs =CPU (cpu);CPUX86State *env = &cpu->env;uint64_t status =MCI_STATUS_VAL |MCI_STATUS_UC |MCI_STATUS_EN | -MCI_STATUS_MISCV |MCI_STATUS_ADDRV |MCI_STATUS_S ; +MCI_STATUS_MISCV |MCI_STATUS_ADDRV ;uint64_t mcg_status =MCG_STATUS_MCIP ;int flags =0 ; -if (code ==BUS_MCEERR_AR ) { - status |=MCI_STATUS_AR |0x134 ; - mcg_status |=MCG_STATUS_EIPV ; +if (!IS_AMD_CPU (env)) { + status |=MCI_STATUS_S ; +if (code ==BUS_MCEERR_AR ) { + status |=MCI_STATUS_AR |0x134 ; + mcg_status |=MCG_STATUS_EIPV ; + }else { + status |=0xc0 ; + mcg_status |=MCG_STATUS_RIPV ; + } }else { - status |=0xc0 ; - mcg_status |=MCG_STATUS_RIPV ; + mcg_status |=MCG_STATUS_EIPV |MCG_STATUS_RIPV ; } flags =cpu_x86_support_mca_broadcast (env) ?MCE_INJECT_BROADCAST :0 ;
Signature · 1 line
2.39.3
Re: [PATCH 2/2] i386: Fix MCE support for AMD hosts
+x86 qemu folks
Quoted 49 lines · John Allen
For the most part, AMD hosts can use the same MCE injection code as Intel but, there are instances where the qemu implementation is Intel specific. First, MCE deliviery works differently on AMD and does not support broadcast. Second,
kvm_mce_injectgenerates MCEs that include a number of Intel specific status bits. Modifykvm_mce_injectto properly generate MCEs on AMD platforms.Reported-by William Roche <[email protected]>Signed-off-by John Allen <[email protected]>--- target/i386/helper.c | 4 ++++ target/i386/kvm/kvm.c | 17 +++++++++++------ 2 files changed, 15 insertions(+), 6 deletions(-)target/i386/helper.c +4
-0@@ -76,6 +76,10 @@ int cpu_x86_support_mca_broadcast(CPUX86State *env)int family =0 ;int model =0 ; +if (IS_AMD_CPU (env)) { +return 0 ; + } +cpu_x86_version (env, &family, &model);if ((family ==6 && model >=14 ) || family >6 ) {return 1 ;target/i386/kvm/kvm.c +6
-4@@ -530,16 +530,21 @@ static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code)CPUState *cs =CPU (cpu);CPUX86State *env = &cpu->env;uint64_t status =MCI_STATUS_VAL |MCI_STATUS_UC |MCI_STATUS_EN | -MCI_STATUS_MISCV |MCI_STATUS_ADDRV |MCI_STATUS_S ; +MCI_STATUS_MISCV |MCI_STATUS_ADDRV ;uint64_t mcg_status =MCG_STATUS_MCIP ;int flags =0 ; -if (code ==BUS_MCEERR_AR ) { - status |=MCI_STATUS_AR |0x134 ; - mcg_status |=MCG_STATUS_EIPV ; +if (!IS_AMD_CPU (env)) { + status |=MCI_STATUS_S ; +if (code ==BUS_MCEERR_AR ) { + status |=MCI_STATUS_AR |0x134 ; + mcg_status |=MCG_STATUS_EIPV ;
target/i386/kvm/kvm.c static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code) · line 549
Quoted 10 lines
target/i386/kvm/kvm.c +5
-2+ }else { + status |=0xc0 ; + mcg_status |=MCG_STATUS_RIPV ; + } }else { - status |=0xc0 ; - mcg_status |=MCG_STATUS_RIPV ; + mcg_status |=MCG_STATUS_EIPV |MCG_STATUS_RIPV ; }
I was gonna say that we should only handle BUS_MCEERR_AR for AMD, but the way you came up with, does seem to work from quick testing. And it's better to log an error that silently ignore it obviously.
Quoted 1 line
target/i386/kvm/kvm.c +0
-0flags =cpu_x86_support_mca_broadcast (env) ?MCE_INJECT_BROADCAST :0 ;