x86/kvm/hyper-v: fix enlightened VMCS & QEMU4.2
cover x86/kvm/hyper-v: fix enlightened VMCS & QEMU4.2
With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails. While the regression is in QEMU, it may still be preferable to fix this in KVM.
It would be great if we could just omit the VMX feature filtering in KVM and make this guest's responsibility: if it switches to using enlightened vmcs it should be aware that not all hardware features are going to be supported. Genuine Hyper-V, however, fails the test. It, for example, enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES and without 'apic_access_addr' field in eVMCS there's not much we can do in KVM. Microsoft confirms the bug but it is unclear if it will ever get fixed in the existing Hyper-V versions as genuine Hyper-V never exposes these unsupported controls to L1.
Changes since 'RFC':
- Go with the bare minimum [Paolo]
KVM RFC:
https://lore.kernel.org/kvm/[email protected]/
QEMU RFC@:
https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg00123.html
Vitaly Kuznetsov (3):
x86/kvm/hyper-v: remove stale evmcs_already_enabled check from
nested_enable_evmcs()
x86/kvm/hyper-v: move VMX controls sanitization out of
nested_enable_evmcs()
x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for
nested guests
arch/x86/kvm/vmx/evmcs.c | 90 ++++++++++++++++++++++++++++++++++----- arch/x86/kvm/vmx/evmcs.h | 3 ++ arch/x86/kvm/vmx/nested.c | 3 ++ arch/x86/kvm/vmx/vmx.c | 16 ++++++- 4 files changed, 99 insertions(+), 13 deletions(-)
Signature · 1 line
2.24.1
PATCH 1/3 x86/kvm/hyper-v: remove stale evmcs_already_enabled check from nested_enable_evmcs()
In nested_enable_evmcs() evmcs_already_enabled check doesn't really do
anything: controls are already sanitized and we return '0' regardless. Just drop the check.
--- arch/x86/kvm/vmx/evmcs.c | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/kvm/vmx/evmcs.c +0 -5
@@ -350,17 +350,12 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version) {struct vcpu_vmx *vmx =to_vmx (vcpu); -bool evmcs_already_enabled = vmx->nested.enlightened_vmcs_enabled; vmx->nested.enlightened_vmcs_enabled = true;if (vmcs_version) *vmcs_version =nested_get_evmcs_version (vcpu); -/* We don't support disabling the feature for simplicity. */ -if (evmcs_already_enabled) -return 0 ; - vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL ; vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL ; vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL ;
Signature · 1 line
2.24.1
PATCH 2/3 x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()
In reply to Vitaly KuznetsovWith fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.
It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there are some problematic controls:SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES VM_{ENTRY,EXIT}_LOAD_IA32_PERF_GLOBAL_CTR
which Hyper-V enables. As there are no corresponding fields in eVMCS, we can't handle this properly in KVM. This is a Hyper-V issue.
Move VMX controls sanitization from nested_enable_evmcs() to vmx_get_msr(), and do the bare minimum (controls, which are known to cause issues). This allows userspace to keep setting controls it wants and at the same time hides them from the guest.
--- arch/x86/kvm/vmx/evmcs.c | 32 ++++++++++++++++++++++++++------ arch/x86/kvm/vmx/evmcs.h | 1 + arch/x86/kvm/vmx/vmx.c | 16 ++++++++++++++-- 3 files changed, 41 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/evmcs.c +26 -6
@@ -346,6 +346,32 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)return 0 ; } +void nested_evmcs_filter_control_msr (u32 msr_index,u64 *pdata) +{ +u32 ctl_low = (u32)*pdata; +u32 ctl_high = (u32)(*pdata >>32 ); + +/* +* Hyper-V 2016 and 2019 try using these features even when eVMCS +* is enabled but there are no corresponding fields. +*/ +switch (msr_index) { +case MSR_IA32_VMX_EXIT_CTLS : +case MSR_IA32_VMX_TRUE_EXIT_CTLS : + ctl_high &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL ; +break ; +case MSR_IA32_VMX_ENTRY_CTLS : +case MSR_IA32_VMX_TRUE_ENTRY_CTLS : + ctl_high &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL ; +break ; +case MSR_IA32_VMX_PROCBASED_CTLS2 : + ctl_high &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES ; +break ; + } + + *pdata = ctl_low | ((u64 )ctl_high <<32 ); +} +int nested_enable_evmcs (struct kvm_vcpu *vcpu,uint16_t *vmcs_version) {
@@ -356,11 +382,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,if (vmcs_version) *vmcs_version =nested_get_evmcs_version (vcpu); - vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL ; - vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL ; - vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL ; - vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC ; - vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC ; -return 0 ; }
arch/x86/kvm/vmx/evmcs.h +1 -0
@@ -201,5 +201,6 @@ bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa);uint16_t nested_get_evmcs_version (struct kvm_vcpu *vcpu);int nested_enable_evmcs (struct kvm_vcpu *vcpu,uint16_t *vmcs_version); +void nested_evmcs_filter_control_msr (u32 msr_index,u64 *pdata);#endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/vmx.c +14 -2
@@ -1849,8 +1849,20 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:if (!nested_vmx_allowed (vcpu))return 1 ; -return vmx_get_vmx_msr (&vmx->nested.msrs, msr_info->index, - &msr_info->data); +if (vmx_get_vmx_msr (&vmx->nested.msrs, msr_info->index, + &msr_info->data)) +return 1 ; +/* +* Enlightened VMCS v1 doesn't have certain fields, but buggy +* Hyper-V versions are still trying to use corresponding +* features when they are exposed. Filter out the essential +* minimum. +*/ +if (!msr_info->host_initiated && + vmx->nested.enlightened_vmcs_enabled) +nested_evmcs_filter_control_msr (msr_info->index, + &msr_info->data); +break ;case MSR_IA32_RTIT_CTL :if (pt_mode !=PT_MODE_HOST_GUEST )return 1 ;
Signature · 1 line
2.24.1
PATCH 3/3 x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests
In reply to Vitaly KuznetsovSane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.
It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.
--- arch/x86/kvm/vmx/evmcs.c | 53 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/evmcs.h | 2 ++ arch/x86/kvm/vmx/nested.c | 3 +++ 3 files changed, 58 insertions(+)
arch/x86/kvm/vmx/evmcs.c +53 -0
@@ -7,6 +7,7 @@#include "evmcs.h" #include "vmcs.h" #include "vmx.h" +#include "trace.h" DEFINE_STATIC_KEY_FALSE (enable_evmcs);
@@ -372,6 +373,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) *pdata = ctl_low | ((u64 )ctl_high <<32 ); } +int nested_evmcs_check_controls (struct vmcs12 *vmcs12) +{ +int ret =0 ; +u32 unsupp_ctl; + + unsupp_ctl = vmcs12->pin_based_vm_exec_control & +EVMCS1_UNSUPPORTED_PINCTRL ; +if (unsupp_ctl) { +trace_kvm_nested_vmenter_failed ( +"eVMCS: unsupported pin-based VM-execution controls" , + unsupp_ctl); + ret = -EINVAL ; + } + + unsupp_ctl = vmcs12->secondary_vm_exec_control & +EVMCS1_UNSUPPORTED_2NDEXEC ; +if (unsupp_ctl) { +trace_kvm_nested_vmenter_failed ( +"eVMCS: unsupported secondary VM-execution controls" , + unsupp_ctl); + ret = -EINVAL ; + } + + unsupp_ctl = vmcs12->vm_exit_controls & +EVMCS1_UNSUPPORTED_VMEXIT_CTRL ; +if (unsupp_ctl) { +trace_kvm_nested_vmenter_failed ( +"eVMCS: unsupported VM-exit controls" , + unsupp_ctl); + ret = -EINVAL ; + } + + unsupp_ctl = vmcs12->vm_entry_controls & +EVMCS1_UNSUPPORTED_VMENTRY_CTRL ; +if (unsupp_ctl) { +trace_kvm_nested_vmenter_failed ( +"eVMCS: unsupported VM-entry controls" , + unsupp_ctl); + ret = -EINVAL ; + } + + unsupp_ctl = vmcs12->vm_function_control &EVMCS1_UNSUPPORTED_VMFUNC ; +if (unsupp_ctl) { +trace_kvm_nested_vmenter_failed ( +"eVMCS: unsupported VM-function controls" , + unsupp_ctl); + ret = -EINVAL ; + } + +return ret; +} +int nested_enable_evmcs (struct kvm_vcpu *vcpu,uint16_t *vmcs_version) {
arch/x86/kvm/vmx/evmcs.h +2 -0
@@ -10,6 +10,7 @@#include "capabilities.h" #include "vmcs.h" +#include "vmcs12.h" struct vmcs_config ;
@@ -202,5 +203,6 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);int nested_enable_evmcs (struct kvm_vcpu *vcpu,uint16_t *vmcs_version);void nested_evmcs_filter_control_msr (u32 msr_index,u64 *pdata); +int nested_evmcs_check_controls (struct vmcs12 *vmcs12);#endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/nested.c +3 -0
@@ -2767,6 +2767,9 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu, nested_check_vm_entry_controls(vcpu, vmcs12)) return -EINVAL; +if (to_vmx (vcpu)->nested.enlightened_vmcs_enabled) +return nested_evmcs_check_controls (vmcs12); +return 0 ; }
Signature · 1 line
2.24.1
Quoted 32 lines · Vitaly Kuznetsov
With fine grained VMX feature enablement QEMU>=4.2 tries to do
KVM_SET_MSRSwith default (matching CPU model) values and in case eVMCS is also enabled, fails. While the regression is in QEMU, it may still be preferable to fix this in KVM.It would be great if we could just omit the VMX feature filtering in KVM and make this guest's responsibility: if it switches to using enlightened vmcs it should be aware that not all hardware features are going to be supported. Genuine Hyper-V, however, fails the test. It, for example, enables
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSESand without 'apic_access_addr' field in eVMCS there's not much we can do in KVM. Microsoft confirms the bug but it is unclear if it will ever get fixed in the existing Hyper-V versions as genuine Hyper-V never exposes these unsupported controls to L1.Changes since 'RFC':
- Go with the bare minimum [Paolo]KVM RFC:
https://lore.kernel.org/kvm/[email protected]/QEMU RFC@:
https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg00123.htmlVitaly Kuznetsov (3):
x86/kvm/hyper-v: remove staleevmcs_already_enabledcheck from
nested_enable_evmcs()
x86/kvm/hyper-v: move VMX controls sanitization out of
nested_enable_evmcs()
x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for
nested guests
Quoted 6 lines
arch/x86/kvm/vmx/evmcs.c | 90 ++++++++++++++++++++++++++++++++++----- arch/x86/kvm/vmx/evmcs.h | 3 ++ arch/x86/kvm/vmx/nested.c | 3 ++ arch/x86/kvm/vmx/vmx.c | 16 ++++++- 4 files changed, 99 insertions(+), 13 deletions(-)
Queued, thanks.
Paolo