KVM: EFER.LMSLE cleanup
v2 cover KVM: EFER.LMSLE cleanup
KVM has never properly virtualized EFER.LMSLE. However, when the "nested" module parameter is set, KVM lets the guest set EFER.LMSLE. Ostensibly, this is so that SLES11 Xen 4.0 will boot as a nested hypervisor.
KVM passes EFER.LMSLE to the hardware through the VMCB, so the setting works most of the time, but the KVM instruction emulator completely ignores the bit, so incorrect guest behavior is almost certainly assured.
With Zen3, AMD has abandoned EFER.LMSLE. KVM still allows it, though, as long as "nested" is set. However, since the hardware doesn't support it, the next VMRUN after the emulated WRMSR will fail with "invalid VMCB."
To clean things up, revert the hack that allowed a KVM guest to set EFER.LMSLE, and enumerate CPUID.80000008H:EDX.EferLmsleUnsupported[bit 20] in KVM_GET_SUPPORTED_CPUID on SVM hosts.
Jim Mattson (3):
Revert "KVM: SVM: Allow EFER.LMSLE to be set with nested svm" x86/cpufeatures: Introduce X86_FEATURE_NO_LMSLE
KVM: SVM: Unconditionally enumerate EferLmsleUnsupported
arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 2 -- arch/x86/kvm/svm/svm.c | 3 ++- 3 files changed, 3 insertions(+), 3 deletions(-)
v1 -> v2: Make no attempt to preserve existing behavior [Sean, Borislav]
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
PATCH v2 1/3 Revert "KVM: SVM: Allow EFER.LMSLE to be set with nested svm"
Revert the hack that allowed a guest to set EFER.LMSLE, since no attempt was ever made to properly virtualize the feature.
Now that AMD has deprecated the feature, the ROI for properly virtualizing it is vanishingly small.
This reverts commit eec4b140c924b4c650e9a89e01d223266490e325.
--- arch/x86/include/asm/msr-index.h | 2 -- arch/x86/kvm/svm/svm.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-)
arch/x86/include/asm/msr-index.h +0 -2
@@ -28,7 +28,6 @@#define _EFER_LMA 10/* Long mode active (read-only) */ #define _EFER_NX 11/* No execute enable */ #define _EFER_SVME 12/* Enable virtualization */ -#define _EFER_LMSLE 13/* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14/* Enable Fast FXSAVE/FXRSTOR */ #define EFER_SCE (1<<_EFER_SCE)
@@ -36,7 +35,6 @@#define EFER_LMA (1<<_EFER_LMA)#define EFER_NX (1<<_EFER_NX)#define EFER_SVME (1<<_EFER_SVME) -#define EFER_LMSLE (1<<_EFER_LMSLE)#define EFER_FFXSR (1<<_EFER_FFXSR)/* Intel MSRs. Some also available on other CPUs */
arch/x86/kvm/svm/svm.c +1 -1
@@ -5012,7 +5012,7 @@ static __init int svm_hardware_setup(void)if (nested) {printk (KERN_INFO "kvm: Nested Virtualization enabled\n" ); -kvm_enable_efer_bits (EFER_SVME |EFER_LMSLE ); +kvm_enable_efer_bits (EFER_SVME ); }/*
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
PATCH v2 2/3 x86/cpufeatures: Introduce X86_FEATURE_NO_LMSLE
In reply to Jim MattsonWhen AMD introduced "Long Mode Segment Limit Enable" (a.k.a. "VMware mode"), the feature was not enumerated by a CPUID bit. Now that VMware has abandoned binary translation, AMD has deprecated EFER.LMSLE.
The absence of the feature *is* now enumerated by a CPUID bit (a la Intel's X86_FEATURE_ZERO_FCS_DCS and X86_FEATURE_FDP_EXCPTN_ONLY).
This defeature bit is already present in feature word 13, but it was previously anonymous. Name it X86_FEATURE_NO_LMSLE, so that KVM can reference it when constructing a supported guest CPUID table.
Since this bit indicates the absence of a feature, don't enumerate it in /proc/cpuinfo.
--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -319,6 +319,7 @@#define X86_FEATURE_AMD_IBRS (13*32+14)/* "" Indirect Branch Restricted Speculation */ #define X86_FEATURE_AMD_STIBP (13*32+15)/* "" Single Thread Indirect Branch Predictors */ #define X86_FEATURE_AMD_STIBP_ALWAYS_ON (13*32+17)/* "" Single Thread Indirect Branch Predictors always-on preferred */ +#define X86_FEATURE_NO_LMSLE (13*32+20)/* "" EFER_LMSLE is unsupported */ #define X86_FEATURE_AMD_PPIN (13*32+23)/* Protected Processor Inventory Number */ #define X86_FEATURE_AMD_SSBD (13*32+24)/* "" Speculative Store Bypass Disable */ #define X86_FEATURE_VIRT_SSBD (13*32+25)/* Virtualized Speculative Store Bypass Disable */
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Re: [PATCH v2 2/3] x86/cpufeatures: Introduce X86_FEATURE_NO_LMSLE
Quoted 25 lines · Jim Mattson
When AMD introduced "Long Mode Segment Limit Enable" (a.k.a. "VMware mode"), the feature was not enumerated by a CPUID bit. Now that VMware has abandoned binary translation, AMD has deprecated EFER.LMSLE.
The absence of the feature *is* now enumerated by a CPUID bit (a la Intel's
X86_FEATURE_ZERO_FCS_DCSandX86_FEATURE_FDP_EXCPTN_ONLY).This defeature bit is already present in feature word 13, but it was previously anonymous. Name it
X86_FEATURE_NO_LMSLE, so that KVM can reference it when constructing a supported guest CPUID table.Since this bit indicates the absence of a feature, don't enumerate it in
/proc/cpuinfo.Signed-off-by Jim Mattson <[email protected]>--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)arch/x86/include/asm/cpufeatures.h +0
-0
arch/x86/include/asm/cpufeatures.h line 325
Quoted 7 lines
arch/x86/include/asm/cpufeatures.h +1
-0@@ -319,6 +319,7 @@#define X86_FEATURE_AMD_IBRS (13*32+14)/* "" Indirect Branch Restricted Speculation */ #define X86_FEATURE_AMD_STIBP (13*32+15)/* "" Single Thread Indirect Branch Predictors */ #define X86_FEATURE_AMD_STIBP_ALWAYS_ON (13*32+17)/* "" Single Thread Indirect Branch Predictors always-on preferred */ +#define X86_FEATURE_NO_LMSLE (13*32+20)/* "" EFER_LMSLE is unsupported */ #define X86_FEATURE_AMD_PPIN (13*32+23)/* Protected Processor Inventory Number */ #define X86_FEATURE_AMD_SSBD (13*32+24)/* "" Speculative Store Bypass Disable */ #define X86_FEATURE_VIRT_SSBD (13*32+25)/* Virtualized Speculative Store Bypass Disable */
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v2 3/3 KVM: SVM: Unconditionally enumerate EferLmsleUnsupported
In reply to Jim MattsonCPUID.80000008H:EDX.EferLmsleUnsupported[bit 20] indicates that IA32_EFER.LMSLE[bit 13] is unsupported and must be zero.
KVM doesn't support "Long Mode Segment Limit Enable," even if the underlying physical processor does, so set that bit in the guest CPUID table returned by KVM_GET_SUPPORTED_CPUID.
--- arch/x86/kvm/svm/svm.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kvm/svm/svm.c +1 -0
@@ -4945,6 +4945,7 @@ static __init void svm_set_cpu_caps(void)if (boot_cpu_has (X86_FEATURE_LS_CFG_SSBD ) ||boot_cpu_has (X86_FEATURE_AMD_SSBD ))kvm_cpu_cap_set (X86_FEATURE_VIRT_SSBD ); +kvm_cpu_cap_set (X86_FEATURE_NO_LMSLE );/* AMD PMU PERFCTR_CORE CPUID */ if (enable_pmu &&boot_cpu_has (X86_FEATURE_PERFCTR_CORE ))
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Sean Christopherson
maintainerRe: [PATCH v2 3/3] KVM: SVM: Unconditionally enumerate EferLmsleUnsupported
Quoted 17 lines · Jim Mattson
CPUID.80000008H:EDX.EferLmsleUnsupported[bit 20] indicates that IA32_EFER.LMSLE[bit 13] is unsupported and must be zero.
KVM doesn't support "Long Mode Segment Limit Enable," even if the underlying physical processor does, so set that bit in the guest CPUID table returned by
KVM_GET_SUPPORTED_CPUID.Signed-off-by Jim Mattson <[email protected]>--- arch/x86/kvm/svm/svm.c | 1 + 1 file changed, 1 insertion(+)arch/x86/kvm/svm/svm.c +0
-0
arch/x86/kvm/svm/svm.c static __init void svm_set_cpu_caps(void) · line 4948
Quoted 4 lines
arch/x86/kvm/svm/svm.c +1
-0@@ -4945,6 +4945,7 @@ static __init void svm_set_cpu_caps(void)if (boot_cpu_has (X86_FEATURE_LS_CFG_SSBD ) ||boot_cpu_has (X86_FEATURE_AMD_SSBD ))kvm_cpu_cap_set (X86_FEATURE_VIRT_SSBD ); +kvm_cpu_cap_set (X86_FEATURE_NO_LMSLE );
This can go in common x86 code, e.g. if someone wants to run an AMD VM on Intel hardware.
Side topic, in the context of this series, the below diff highlights how silly it is for PSFD to be banished from cpufeatures.h. While we have Boris's attention (and ACK!), can you tack on a patch to move drop KVM_X86_FEATURE_PSFD and move the bit to cpufeatures.h where it belongs?
arch/x86/kvm/cpuid.c +1 -0
@@ -696,6 +696,7 @@ void kvm_set_cpu_caps(void)F (AMD_SSB_NO ) |F (AMD_STIBP ) |F (AMD_STIBP_ALWAYS_ON ) |__feature_bit (KVM_X86_FEATURE_PSFD ) ); +kvm_cpu_cap_set (X86_FEATURE_NO_LMSLE );/* * AMD has separate bits for each SPEC_CTRL bit.
Borislav Petkov
maintainerQuoted 4 lines · Jim Mattson
Jim Mattson (3):
Revert "KVM: SVM: Allow EFER.LMSLE to be set with nested svm" x86/cpufeatures: IntroduceX86_FEATURE_NO_LMSLE
KVM: SVM: Unconditionally enumerate EferLmsleUnsupported
Why do you need those two if you revert the hack? After the revert, anything that tries to set LMSLE should get a #GP anyway, no?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteSean Christopherson
maintainerQuoted 8 lines · Borislav Petkov
Quoted 4 lines · Jim Mattson
Jim Mattson (3):
Revert "KVM: SVM: Allow EFER.LMSLE to be set with nested svm" x86/cpufeatures: IntroduceX86_FEATURE_NO_LMSLE
KVM: SVM: Unconditionally enumerate EferLmsleUnsupportedWhy do you need those two if you revert the hack? After the revert, anything that tries to set LMSLE should get a #GP anyway, no?
Yes, but ideally KVM would explicitly tell the guest "you don't have LMSLE". Probably a moot point, but at the same time I don't see a reason not to be explicit.
Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Jim Mattson
Jim Mattson (3):
Revert "KVM: SVM: Allow EFER.LMSLE to be set with nested svm" x86/cpufeatures: IntroduceX86_FEATURE_NO_LMSLE
KVM: SVM: Unconditionally enumerate EferLmsleUnsupportedWhy do you need those two if you revert the hack? After the revert, anything that tries to set LMSLE should get a #GP anyway, no?
Reporting that CPUID bit gives us the right to raise #GP. AMD CPUs (going way back) that don't report EferLmsleUnsupported do not raise #GP.
Borislav Petkov
maintainerQuoted 3 lines · Sean Christopherson
Yes, but ideally KVM would explicitly tell the guest "you don't have LMSLE". Probably a moot point, but at the same time I don't see a reason not to be explicit.
Yes but...
Quoted 3 lines · Jim Mattson
Reporting that CPUID bit gives us the right to raise #GP. AMD CPUs (going way back) that don't report EferLmsleUnsupported do not raise #GP.
... what does "gives us the right" mean exactly?
I'm pretty sure I'm missing something about how KVM works but wouldn't it raise a guest #GP when the guest tries to set an unsupported EFER bit? I.e., why do you need to explicitly do
kvm_cpu_cap_set(X86_FEATURE_NO_LMSLE);
and not handle this like any other EFER reserved bit?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 15 lines · Borislav Petkov
Quoted 3 lines · Sean Christopherson
Yes, but ideally KVM would explicitly tell the guest "you don't have LMSLE". Probably a moot point, but at the same time I don't see a reason not to be explicit.
Yes but...
Quoted 3 lines · Jim Mattson
Reporting that CPUID bit gives us the right to raise #GP. AMD CPUs (going way back) that don't report EferLmsleUnsupported do not raise #GP.
... what does "gives us the right" mean exactly?
Quoted 7 lines
I'm pretty sure I'm missing something about how KVM works but wouldn't it raise a guest #GP when the guest tries to set an unsupported EFER bit? I.e., why do you need to explicitly do
kvm_cpu_cap_set(X86_FEATURE_NO_LMSLE);and not handle this like any other EFER reserved bit?
EFER.LMLSE is not a reserved bit on AMD64 CPUs, unless CPUID.80000008:EBX[20] is set (or you're running very, very old hardware).
We really shouldn't just decide on a whim to treat EFER.LMSLE as reserved under KVM. The guest CPUID information represents our detailed contract with the guest software. By setting CPUID.80000008:EBX[20], we are telling the guest that if it tries to set EFER.LMSLE, we will raise a #GP.
If we don't set that bit in the guest CPUID information and we raise #GP on an attempt to set EFER.LMSLE, the virtual hardware is defective. We could document this behavior as an erratum, but since a mechanism exists to declare that the guest can expect EFER.LMSLE to #GP, doesn't it make sense to use it?
Borislav Petkov
maintainerQuoted 9 lines · Jim Mattson
EFER.LMLSE is not a reserved bit on AMD64 CPUs, unless CPUID.80000008:EBX[20] is set (or you're running very, very old hardware).
We really shouldn't just decide on a whim to treat EFER.LMSLE as reserved under KVM. The guest CPUID information represents our detailed contract with the guest software. By setting CPUID.80000008:EBX[20], we are telling the guest that if it tries to set EFER.LMSLE, we will raise a #GP.
I understand all that. What I'm asking is, what happens in KVM *after* your patch 1/3 is applied when a guest tries to set EFER.LMSLE? Does it #GP or does it allow the WRMSR to succeed? I.e., does KVM check when reserved bits in that MSR are being set?
By looking at it, there's kvm_enable_efer_bits() so it looks like KVM does control which bits are allowed to set and which not...?
Quoted 3 lines
If we don't set that bit in the guest CPUID information and we raise #GP on an attempt to set EFER.LMSLE, the virtual hardware is defective.
See, this is what I don't get - why is it defective? After the revert, that bit to KVM is reserved.
Quoted 3 lines
We could document this behavior as an erratum, but since a mechanism exists to declare that the guest can expect EFER.LMSLE to #GP, doesn't it make sense to use it?
I don't mind all that and the X86_FEATURE bit and so on - I'm just trying to ask you guys: what is KVM's behavior when the guest tries to set a reserved EFER bit.
Maybe I'm not expressing myself precisely enough...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 12 lines · Borislav Petkov
Quoted 9 lines · Jim Mattson
EFER.LMLSE is not a reserved bit on AMD64 CPUs, unless CPUID.80000008:EBX[20] is set (or you're running very, very old hardware).
We really shouldn't just decide on a whim to treat EFER.LMSLE as reserved under KVM. The guest CPUID information represents our detailed contract with the guest software. By setting CPUID.80000008:EBX[20], we are telling the guest that if it tries to set EFER.LMSLE, we will raise a #GP.
Quoted 7 lines
I understand all that. What I'm asking is, what happens in KVM *after* your patch 1/3 is applied when a guest tries to set EFER.LMSLE? Does it #GP or does it allow the WRMSR to succeed? I.e., does KVM check when reserved bits in that MSR are being set?
By looking at it, there's
kvm_enable_efer_bits()so it looks like KVM does control which bits are allowed to set and which not...?
Yes, after the revert, KVM will treat the bit as reserved, and it will synthesize a #GP, *in violation of the architectural specification.* As I said, we could document this behavior as a KVM erratum.
Quoted 6 lines
Quoted 3 lines
If we don't set that bit in the guest CPUID information and we raise #GP on an attempt to set EFER.LMSLE, the virtual hardware is defective.
See, this is what I don't get - why is it defective? After the revert, that bit to KVM is reserved.
KVM can't just decide willy nilly to reserve arbitrary bits. If it is in violation of AMD's architectural specification, the virtual CPU is defective.
Quoted 9 lines
Quoted 3 lines
We could document this behavior as an erratum, but since a mechanism exists to declare that the guest can expect EFER.LMSLE to #GP, doesn't it make sense to use it?
I don't mind all that and the
X86_FEATUREbit and so on - I'm just trying to ask you guys: what is KVM's behavior when the guest tries to set a reserved EFER bit.Maybe I'm not expressing myself precisely enough...
I feel the same way. :-(
The two patches after the revert are to amend the contract with the guest (as expressed by the guest CPUID table) so that the KVM virtual CPU can raise a #GP on EFER.LMSLE and still conform to the architectural specification.
From the APM, volume 2, 4.12.2 Data Limit Checks in 64-bit Mode:
Quoted 1 line
Data segment limit checking in 64-bit mode is not supported by all processor implementations and has been deprecated. If CPUID Fn8000_0008_EBX[EferLmlseUnsupported](bit 20) = 1, 64-bit mode segment limit checking is not supported and attempting to enable this feature by setting EFER.LMSLE =1 will result in a #GP exception.
Borislav Petkov
maintainerQuoted 2 lines · Jim Mattson
Yes, after the revert, KVM will treat the bit as reserved, and it will synthesize a #GP, *in violation of the architectural specification.*
Architectural, schmarchitectural... Intel hasn't implemented it so meh.
Quoted 3 lines
KVM can't just decide willy nilly to reserve arbitrary bits. If it is in violation of AMD's architectural specification, the virtual CPU is defective.
Grrr, after your revert that this bit was *only* reserved and nothing else to KVM. Like every other reserved bit in EFER. Yeah, yeah, AMD specified it as architectural but Intel didn't implement it so there's this thing on paper and there's reality...
Anyway, enough about this - we're on the same page.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 7 lines · Borislav Petkov
Quoted 2 lines · Jim Mattson
Yes, after the revert, KVM will treat the bit as reserved, and it will synthesize a #GP, *in violation of the architectural specification.*
Architectural, schmarchitectural... Intel hasn't implemented it so meh.
Quoted 8 lines
Quoted 3 lines
KVM can't just decide willy nilly to reserve arbitrary bits. If it is in violation of AMD's architectural specification, the virtual CPU is defective.
Grrr, after your revert that this bit was *only* reserved and nothing else to KVM. Like every other reserved bit in EFER. Yeah, yeah, AMD specified it as architectural but Intel didn't implement it so there's this thing on paper and there's reality...
AMD defined the 64-bit x86 extensions while Intel was distracted with their VLIW science fair project. In this space, Intel produces AMD64 compatible CPUs. The definitive specification comes from AMD (which is sad, because AMD's documentation is abysmal).
Borislav Petkov
maintainerQuoted 3 lines · Jim Mattson
AMD defined the 64-bit x86 extensions while Intel was distracted with their VLIW science fair project. In this space, Intel produces AMD64 compatible CPUs.
Almost-compatible. And maybe, just maybe, because Intel were probably and practically forced to implement AMD64 but then thought, oh well, we'll do some things differently.
Quoted 2 lines
The definitive specification comes from AMD (which is sad, because AMD's documentation is abysmal).
Just don't tell me the SDM is better...
But you and I are really talking past each other: there's nothing definitive about a spec if, while implementing it, the other vendor is doing some subtle, but very software visible things differently.
I.e., the theory vs reality point I'm trying to get across.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 13 lines · Borislav Petkov
Quoted 3 lines · Jim Mattson
AMD defined the 64-bit x86 extensions while Intel was distracted with their VLIW science fair project. In this space, Intel produces AMD64 compatible CPUs.
Almost-compatible. And maybe, just maybe, because Intel were probably and practically forced to implement AMD64 but then thought, oh well, we'll do some things differently.
Quoted 2 lines
The definitive specification comes from AMD (which is sad, because AMD's documentation is abysmal).
Quoted 7 lines
Just don't tell me the SDM is better...
But you and I are really talking past each other: there's nothing definitive about a spec if, while implementing it, the other vendor is doing some subtle, but very software visible things differently.
I.e., the theory vs reality point I'm trying to get across.
I get it. In reality, all of the reverse polarity CPUID feature bits are essentially useless.
The only software that actually uses LMSLE is defunct. That software predated the definition of CPUID.80000008:EBX.EferLmlseUnsupported and is no longer being updated, so it isn't going to check the feature bit. It's just going to fail with an unexpected #GP.
Maybe you think I'm being overly pedantic, but the code to do the right thing is trivial, so why not do it?
This way, if anyone files a bug against KVM because an old VMware hypervisor dies with an unexpected #GP, I can point to the spec and say that it's user error.