Skip to contentSkip to conversation

x86/kvm/hyper-v: fix enlightened VMCS & QEMU4.2

RFC PATCH In review Vitaly Kuznetsov 4 participants 34 messages Jan 15, 2020 – Feb 3, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover x86/kvm/hyper-v: fix enlightened VMCS & QEMU4.2

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails. While the regression is in QEMU, it may still be preferable to fix this in KVM.

It would be great if we could just omit the VMX feature filtering in KVM and make this guest's responsibility: if it switches to using enlightened vmcs it should be aware that not all hardware features are going to be supported. Genuine Hyper-V, however, fails the test. In particular, it enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES and without 'apic_access_addr' field in eVMCS there's not much we can do in KVM.

The suggested approach in this patch series is: move VMX feature filtering to vmx_get_msr() so only guest doesn't see them when eVMCS is enabled (PATCH2) and add a check that it doesn't enable them (PATCH3).

I can't say that I'm a great fan of this workaround myself, thus RFC.

My initial RFC sent to qemu-devel@:
https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg00123.html

Vitaly Kuznetsov (3):
x86/kvm/hyper-v: remove stale evmcs_already_enabled check from
nested_enable_evmcs()
x86/kvm/hyper-v: move VMX controls sanitization out of
nested_enable_evmcs()
x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for
nested guests

 arch/x86/kvm/vmx/evmcs.c  | 99 ++++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/vmx/evmcs.h  |  2 +
 arch/x86/kvm/vmx/nested.c |  3 ++
 arch/x86/kvm/vmx/vmx.c    | 10 +++-
 4 files changed, 100 insertions(+), 14 deletions(-)
Signature · 1 line
2.24.1
#1 lore raw

RFC PATCH 1/3 x86/kvm/hyper-v: remove stale evmcs_already_enabled check from nested_enable_evmcs()

In nested_enable_evmcs() evmcs_already_enabled check doesn't really do
anything: controls are already sanitized and we return '0' regardless. Just drop the check.

---
 arch/x86/kvm/vmx/evmcs.c | 5 -----
 1 file changed, 5 deletions(-)
arch/x86/kvm/vmx/evmcs.c +0 -5
index 72359709cdc1..89c3e0caf39f 100644
@@ -350,17 +350,12 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
-	bool evmcs_already_enabled = vmx->nested.enlightened_vmcs_enabled;
 
 	vmx->nested.enlightened_vmcs_enabled = true;
 
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	/* We don't support disabling the feature for simplicity. */
-	if (evmcs_already_enabled)
-		return 0;
-
 	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
 	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
 	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
Signature · 1 line
2.24.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH RFC 1/3] x86/kvm/hyper-v: remove stale evmcs_already_enabled check from nested_enable_evmcs()

Quoted 9 lines · Vitaly Kuznetsov

In nested_enable_evmcs() evmcs_already_enabled check doesn't really do
anything: controls are already sanitized and we return '0' regardless. Just drop the check.

-Liran

Quoted 29 lines

---
arch/x86/kvm/vmx/evmcs.c | 5 -----

1 file changed, 5 deletions(-)
arch/x86/kvm/vmx/evmcs.c +0 -5
index 72359709cdc1..89c3e0caf39f 100644
@@ -350,17 +350,12 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
-	bool evmcs_already_enabled = vmx->nested.enlightened_vmcs_enabled;
 
 	vmx->nested.enlightened_vmcs_enabled = true;
 
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	/* We don't support disabling the feature for simplicity. */
-	if (evmcs_already_enabled)
-		return 0;
-
 	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
 	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
 	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
Signature · 1 line
2.24.1
#3 lore raw

RFC PATCH 2/3 x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Move VMX controls sanitization from nested_enable_evmcs() to vmx_get_msr(), this allows userspace to keep setting controls it wants and at the same time hides them from the guest.

---
 arch/x86/kvm/vmx/evmcs.c | 38 ++++++++++++++++++++++++++++++++------
 arch/x86/kvm/vmx/evmcs.h |  1 +
 arch/x86/kvm/vmx/vmx.c   | 10 ++++++++--
 3 files changed, 41 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/evmcs.c +32 -6
index 89c3e0caf39f..b5d6582ba589 100644
@@ -346,6 +346,38 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
        return 0;
 }
 
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
+{
+	u32 ctl_low = (u32)*pdata, ctl_high = (u32)(*pdata >> 32);
+	/*
+	 * Enlightened VMCS doesn't have certain fields, make sure we don't
+	 * expose unsupported controls to L1.
+	 */
+
+	switch (msr_index) {
+	case MSR_IA32_VMX_PINBASED_CTLS:
+	case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
+		break;
+	case MSR_IA32_VMX_EXIT_CTLS:
+	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
+		break;
+	case MSR_IA32_VMX_ENTRY_CTLS:
+	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
+		break;
+	case MSR_IA32_VMX_PROCBASED_CTLS2:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
+		break;
+	case MSR_IA32_VMX_VMFUNC:
+		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
+		break;
+	}
+
+	*pdata = ctl_low | ((u64)ctl_high << 32);
+}
+
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
@@ -356,11 +388,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
-	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
-	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
-	vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
-	vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC;
-
 	return 0;
 }
arch/x86/kvm/vmx/evmcs.h +1 -0
index 07ebf6882a45..b88d9807a796 100644
@@ -201,5 +201,6 @@ bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa);
 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata);
 
 #endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/vmx.c +8 -2
index e3394c839dea..8eb74618b8d8 100644
@@ -1849,8 +1849,14 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:
 		if (!nested_vmx_allowed(vcpu))
 			return 1;
-		return vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
-				       &msr_info->data);
+		if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
+				    &msr_info->data))
+			return 1;
+		if (!msr_info->host_initiated &&
+		    vmx->nested.enlightened_vmcs_enabled)
+			nested_evmcs_filter_control_msr(msr_info->index,
+							&msr_info->data);
+		break;
 	case MSR_IA32_RTIT_CTL:
 		if (pt_mode != PT_MODE_HOST_GUEST)
 			return 1;
Signature · 1 line
2.24.1
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 8 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 9 lines

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Have you tried contacted someone at Hyper-V team about this?

Quoted 17 lines

Move VMX controls sanitization from nested_enable_evmcs() to vmx_get_msr(), this allows userspace to keep setting controls it wants and at the same time hides them from the guest.

---

arch/x86/kvm/vmx/evmcs.c | 38 ++++++++++++++++++++++++++++++++------
arch/x86/kvm/vmx/evmcs.h |  1 +
arch/x86/kvm/vmx/vmx.c   | 10 ++++++++--
3 files changed, 41 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/evmcs.c +0 -0
index 89c3e0caf39f..b5d6582ba589 100644

arch/x86/kvm/vmx/evmcs.c uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu) · line 351

Quoted 6 lines
arch/x86/kvm/vmx/evmcs.c +3 -0
@@ -346,6 +346,38 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
       return 0;
 }
 
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
+{
+	u32 ctl_low = (u32)*pdata, ctl_high = (u32)(*pdata >> 32);

Nit: I dislike defining & initialising multiple local vars on same line.

Quoted 63 lines
arch/x86/kvm/vmx/evmcs.c +29 -6
+	/*
+	 * Enlightened VMCS doesn't have certain fields, make sure we don't
+	 * expose unsupported controls to L1.
+	 */
+
+	switch (msr_index) {
+	case MSR_IA32_VMX_PINBASED_CTLS:
+	case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
+		break;
+	case MSR_IA32_VMX_EXIT_CTLS:
+	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
+		break;
+	case MSR_IA32_VMX_ENTRY_CTLS:
+	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
+		break;
+	case MSR_IA32_VMX_PROCBASED_CTLS2:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
+		break;
+	case MSR_IA32_VMX_VMFUNC:
+		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
+		break;
+	}
+
+	*pdata = ctl_low | ((u64)ctl_high << 32);
+}
+
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
@@ -356,11 +388,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
-	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
-	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
-	vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
-	vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC;
-
 	return 0;
 }
arch/x86/kvm/vmx/evmcs.h +1 -0
index 07ebf6882a45..b88d9807a796 100644
@@ -201,5 +201,6 @@ bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa);
 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata);
 
 #endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/vmx.c +0 -0
index e3394c839dea..8eb74618b8d8 100644
@@ -1849,8 +1849,14 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:
 		if (!nested_vmx_allowed(vcpu))
 			return 1;

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 1859

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +8 -2
-		return vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
-				       &msr_info->data);
+		if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
+				    &msr_info->data))
+			return 1;
+		if (!msr_info->host_initiated &&
+		    vmx->nested.enlightened_vmcs_enabled)
+			nested_evmcs_filter_control_msr(msr_info->index,
+							&msr_info->data);
+		break;

Nit: It seems more elegant to me to put the call to nested_evmcs_filter_control_msr() inside vmx_get_vmx_msr().

The patch itself makes sense to me and looks correct.

-Liran

#5 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 16 lines · Liran Alon
Quoted 6 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 9 lines

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Quoted 2 lines

Have you tried contacted someone at Hyper-V team about this?

Yes, I have.

Quoted 93 lines
Quoted 17 lines

Move VMX controls sanitization from nested_enable_evmcs() to vmx_get_msr(), this allows userspace to keep setting controls it wants and at the same time hides them from the guest.

---

arch/x86/kvm/vmx/evmcs.c | 38 ++++++++++++++++++++++++++++++++------
arch/x86/kvm/vmx/evmcs.h |  1 +
arch/x86/kvm/vmx/vmx.c   | 10 ++++++++--
3 files changed, 41 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/evmcs.c +0 -0
index 89c3e0caf39f..b5d6582ba589 100644

arch/x86/kvm/vmx/evmcs.c uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu) · line 351

Quoted 6 lines
arch/x86/kvm/vmx/evmcs.c +3 -0
@@ -346,6 +346,38 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
       return 0;
 }
 
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
+{
+	u32 ctl_low = (u32)*pdata, ctl_high = (u32)(*pdata >> 32);

Nit: I dislike defining & initialising multiple local vars on same line.

Quoted 67 lines
arch/x86/kvm/vmx/evmcs.c +29 -6
+	/*
+	 * Enlightened VMCS doesn't have certain fields, make sure we don't
+	 * expose unsupported controls to L1.
+	 */
+
+	switch (msr_index) {
+	case MSR_IA32_VMX_PINBASED_CTLS:
+	case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
+		break;
+	case MSR_IA32_VMX_EXIT_CTLS:
+	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
+		break;
+	case MSR_IA32_VMX_ENTRY_CTLS:
+	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
+		break;
+	case MSR_IA32_VMX_PROCBASED_CTLS2:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
+		break;
+	case MSR_IA32_VMX_VMFUNC:
+		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
+		break;
+	}
+
+	*pdata = ctl_low | ((u64)ctl_high << 32);
+}
+
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
@@ -356,11 +388,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
-	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
-	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
-	vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
-	vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC;
-
 	return 0;
 }
arch/x86/kvm/vmx/evmcs.h +1 -0
index 07ebf6882a45..b88d9807a796 100644
@@ -201,5 +201,6 @@ bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa);
 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata);
 
 #endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/vmx.c +2 -2
index e3394c839dea..8eb74618b8d8 100644
@@ -1849,8 +1849,14 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:
 		if (!nested_vmx_allowed(vcpu))
 			return 1;
-		return vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
-				       &msr_info->data);
+		if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
+				    &msr_info->data))
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 1859

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +6 -0
+			return 1;
+		if (!msr_info->host_initiated &&
+		    vmx->nested.enlightened_vmcs_enabled)
+			nested_evmcs_filter_control_msr(msr_info->index,
+							&msr_info->data);
+		break;

Nit: It seems more elegant to me to put the call to nested_evmcs_filter_control_msr() inside vmx_get_vmx_msr().

Sure, will move it there (in case we actually decide to merge this)

Quoted 2 lines

The patch itself makes sense to me and looks correct.

Thanks!

Signature · 1 line
Vitaly
#6 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 18 lines · Vitaly Kuznetsov
Quoted 16 lines · Liran Alon
Quoted 15 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Quoted 4 lines
Quoted 2 lines

Have you tried contacted someone at Hyper-V team about this?

Yes, I have.

I heard back from MS, they admited the bug and suggested us ... to hide the control from L1 when eVMCS is enabled. No surprises here. They also said the bug is unlikely to get fixed in the existing Hyper-V versions (2016 and 2019) so it seems we're stuck with the hack for awhile :-(

...

Quoted 24 lines
Quoted 14 lines
Quoted 14 lines

#endif /* __KVM_X86_VMX_EVMCS_H */

arch/x86/kvm/vmx/vmx.c +2 -2
index e3394c839dea..8eb74618b8d8 100644
@@ -1849,8 +1849,14 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:
 		if (!nested_vmx_allowed(vcpu))
 			return 1;
-		return vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
-				       &msr_info->data);
+		if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
+				    &msr_info->data))
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 1859

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +6 -0
+			return 1;
+		if (!msr_info->host_initiated &&
+		    vmx->nested.enlightened_vmcs_enabled)
+			nested_evmcs_filter_control_msr(msr_info->index,
+							&msr_info->data);
+		break;

Nit: It seems more elegant to me to put the call to nested_evmcs_filter_control_msr() inside vmx_get_vmx_msr().

Quoted 2 lines

Sure, will move it there (in case we actually decide to merge this)

Thinking more about it, we can't check host_initiated in vmx_get_vmx_msr() as it's not passed there, this can probably get changed but I don't see a big difference so I'll probably keep the hack where it is now.

Signature · 1 line
Vitaly
#7 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Vitaly Kuznetsov
Quoted 3 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

As in, Qemu is blindly throwing values at KVM and complains on failure? That seems like a Qemu bug, especially since Qemu needs to explicitly do KVM_CAP_HYPERV_ENLIGHTENED_VMCS to enable eVMCS.

#8 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 12 lines · Sean Christopherson
Quoted 3 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

As in, Qemu is blindly throwing values at KVM and complains on failure? That seems like a Qemu bug, especially since Qemu needs to explicitly do KVM_CAP_HYPERV_ENLIGHTENED_VMCS to enable eVMCS.

See: https://patchwork.kernel.org/patch/11316021/
For more context.

-Liran

#9 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 7 lines · Liran Alon
Quoted 7 lines · Sean Christopherson
Quoted 3 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 6 lines
Quoted 3 lines

As in, Qemu is blindly throwing values at KVM and complains on failure? That seems like a Qemu bug, especially since Qemu needs to explicitly do KVM_CAP_HYPERV_ENLIGHTENED_VMCS to enable eVMCS.

See: https://patchwork.kernel.org/patch/11316021/
For more context.

Ya,

while it would certainly be possible to require that userspace takes into account KVM_CAP_HYPERV_ENLIGHTENED_VMCS (which is an opt-in) when doing KVM_SET_MSRS there doesn't seem to be an existing (easy) way to figure out which VMX controls were filtered out after enabling KVM_CAP_HYPERV_ENLIGHTENED_VMCS: KVM_GET_MSRS returns global &vmcs_config.nested values for VMX MSRs (vmx_get_msr_feature()).

Signature · 1 line
Vitaly
#10 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 16 lines · Vitaly Kuznetsov
Quoted 7 lines · Liran Alon
Quoted 7 lines · Sean Christopherson
Quoted 3 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 6 lines
Quoted 3 lines

As in, Qemu is blindly throwing values at KVM and complains on failure? That seems like a Qemu bug, especially since Qemu needs to explicitly do KVM_CAP_HYPERV_ENLIGHTENED_VMCS to enable eVMCS.

See: https://patchwork.kernel.org/patch/11316021/
For more context.

Quoted 8 lines

Ya,

while it would certainly be possible to require that userspace takes into account KVM_CAP_HYPERV_ENLIGHTENED_VMCS (which is an opt-in) when doing KVM_SET_MSRS there doesn't seem to be an existing (easy) way to figure out which VMX controls were filtered out after enabling KVM_CAP_HYPERV_ENLIGHTENED_VMCS: KVM_GET_MSRS returns global &vmcs_config.nested values for VMX MSRs (vmx_get_msr_feature()).

Ah, I was looking at the call to vmx_get_vmx_msr(&vmx->nested.msrs, ...) in vmx_get_msr().

Why not just do this in Qemu? IMO that's not a major ask, e.g. Qemu is doing a decent amount of manual adjustment anyways. And Qemu isn't even using the result of KVM_GET_MSRS so I don't think it's fair to say this is solely KVM's fault.

target/i386/kvm.c +17 -0
index 1d10046a6c..6545bb323e 100644
@@ -2623,6 +2623,23 @@ static void kvm_msr_entry_add_vmx(X86CPU *cpu, FeatureWordArray f)
              MSR_VMX_EPT_UC | MSR_VMX_EPT_WB : 0);
     uint64_t fixed_vmx_ept_vpid = kvm_vmx_ept_vpid & fixed_vmx_ept_mask;
 
+    /* Hyper-V's eVMCS does't support certain features, adjust accordingly. */
+    if (cpu->hyperv_evmcs) {
+        f[FEAT_VMX_PINBASED_CTLS] &= ~(VMX_PIN_BASED_VMX_PREEMPTION_TIMER |
+                                       VMX_PIN_BASED_POSTED_INTR);
+        f[FEAT_VMX_EXIT_CTLS] &= ~VMX_VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
+        f[FEAT_VMX_ENTRY_CTLS] &= ~VMX_VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
+        f[FEAT_VMX_SECONDARY_CTLS] &= ~(VMX_SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
+                                        VMX_SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
+                                        VMX_SECONDARY_EXEC_APIC_REGISTER_VIRT |
+                                        VMX_SECONDARY_EXEC_ENABLE_PML |
+                                        VMX_SECONDARY_EXEC_ENABLE_VMFUNC |
+                                        VMX_SECONDARY_EXEC_SHADOW_VMCS |
+                                        /* VMX_SECONDARY_EXEC_TSC_SCALING | */
+                                        VMX_SECONDARY_EXEC_PAUSE_LOOP_EXITING);
+        f[FEAT_VMX_VMFUNC]         &= ~MSR_VMX_VMFUNC_EPT_SWITCHING;
+    }
+
     kvm_msr_entry_add(cpu, MSR_IA32_VMX_TRUE_PROCBASED_CTLS,
                       make_vmx_msr_value(MSR_IA32_VMX_TRUE_PROCBASED_CTLS,
                                          f[FEAT_VMX_PROCBASED_CTLS]));
#11 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 53 lines · Sean Christopherson
Quoted 16 lines · Vitaly Kuznetsov
Quoted 7 lines · Liran Alon
Quoted 7 lines · Sean Christopherson
Quoted 3 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 6 lines
Quoted 3 lines

As in, Qemu is blindly throwing values at KVM and complains on failure? That seems like a Qemu bug, especially since Qemu needs to explicitly do KVM_CAP_HYPERV_ENLIGHTENED_VMCS to enable eVMCS.

See: https://patchwork.kernel.org/patch/11316021/
For more context.

Quoted 8 lines

Ya,

while it would certainly be possible to require that userspace takes into account KVM_CAP_HYPERV_ENLIGHTENED_VMCS (which is an opt-in) when doing KVM_SET_MSRS there doesn't seem to be an existing (easy) way to figure out which VMX controls were filtered out after enabling KVM_CAP_HYPERV_ENLIGHTENED_VMCS: KVM_GET_MSRS returns global &vmcs_config.nested values for VMX MSRs (vmx_get_msr_feature()).

Ah, I was looking at the call to vmx_get_vmx_msr(&vmx->nested.msrs, ...) in vmx_get_msr().

Why not just do this in Qemu? IMO that's not a major ask, e.g. Qemu is doing a decent amount of manual adjustment anyways. And Qemu isn't even using the result of KVM_GET_MSRS so I don't think it's fair to say this is solely KVM's fault.

target/i386/kvm.c +10 -0
index 1d10046a6c..6545bb323e 100644
@@ -2623,6 +2623,23 @@ static void kvm_msr_entry_add_vmx(X86CPU *cpu, FeatureWordArray f)
              MSR_VMX_EPT_UC | MSR_VMX_EPT_WB : 0);
     uint64_t fixed_vmx_ept_vpid = kvm_vmx_ept_vpid & fixed_vmx_ept_mask;
 
+    /* Hyper-V's eVMCS does't support certain features, adjust accordingly. */
+    if (cpu->hyperv_evmcs) {
+        f[FEAT_VMX_PINBASED_CTLS] &= ~(VMX_PIN_BASED_VMX_PREEMPTION_TIMER |
+                                       VMX_PIN_BASED_POSTED_INTR);
+        f[FEAT_VMX_EXIT_CTLS] &= ~VMX_VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
+        f[FEAT_VMX_ENTRY_CTLS] &= ~VMX_VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
+        f[FEAT_VMX_SECONDARY_CTLS] &= ~(VMX_SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
+                                        VMX_SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
+                                        VMX_SECONDARY_EXEC_APIC_REGISTER_VIRT |
+                                        VMX_SECONDARY_EXEC_ENABLE_PML |

target/i386/kvm.c static void kvm_msr_entry_add_vmx(X86CPU *cpu, FeatureWordArray f) · line 2645

Quoted 10 lines
target/i386/kvm.c +7 -0
+                                        VMX_SECONDARY_EXEC_ENABLE_VMFUNC |
+                                        VMX_SECONDARY_EXEC_SHADOW_VMCS |
+                                        /* VMX_SECONDARY_EXEC_TSC_SCALING | */
+                                        VMX_SECONDARY_EXEC_PAUSE_LOOP_EXITING);
+        f[FEAT_VMX_VMFUNC]         &= ~MSR_VMX_VMFUNC_EPT_SWITCHING;
+    }
+
     kvm_msr_entry_add(cpu, MSR_IA32_VMX_TRUE_PROCBASED_CTLS,
                       make_vmx_msr_value(MSR_IA32_VMX_TRUE_PROCBASED_CTLS,
                                          f[FEAT_VMX_PROCBASED_CTLS]));

I accuse you of not reading my PATCH0 :-)

https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg00123.html
does exactly this :-)

P.S. I expect Paolo to comment on which hack he hates less :-)

Signature · 1 line
Vitaly
maintainer
#13 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Sean Christopherson
Quoted 4 lines · Sean Christopherson

Why not just do this in Qemu? IMO that's not a major ask, e.g. Qemu is doing a decent amount of manual adjustment anyways. And Qemu isn't even using the result of KVM_GET_MSRS so I don't think it's fair to say this is solely KVM's fault.

IMHO the features should stay available in case the guest chooses not to use eVMCS. A guest that uses eVMCS should know which features it cannot use and not enable them.

Paolo

maintainer
#14 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Vitaly Kuznetsov
Quoted 4 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 9 lines

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Unless there are objections, I plan to apply these patches.

Paolo

Quoted 99 lines

Move VMX controls sanitization from nested_enable_evmcs() to vmx_get_msr(), this allows userspace to keep setting controls it wants and at the same time hides them from the guest.

---
 arch/x86/kvm/vmx/evmcs.c | 38 ++++++++++++++++++++++++++++++++------
 arch/x86/kvm/vmx/evmcs.h |  1 +
 arch/x86/kvm/vmx/vmx.c   | 10 ++++++++--
 3 files changed, 41 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/evmcs.c +32 -6
index 89c3e0caf39f..b5d6582ba589 100644
@@ -346,6 +346,38 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
        return 0;
 }
 
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
+{
+	u32 ctl_low = (u32)*pdata, ctl_high = (u32)(*pdata >> 32);
+	/*
+	 * Enlightened VMCS doesn't have certain fields, make sure we don't
+	 * expose unsupported controls to L1.
+	 */
+
+	switch (msr_index) {
+	case MSR_IA32_VMX_PINBASED_CTLS:
+	case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
+		break;
+	case MSR_IA32_VMX_EXIT_CTLS:
+	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
+		break;
+	case MSR_IA32_VMX_ENTRY_CTLS:
+	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
+		break;
+	case MSR_IA32_VMX_PROCBASED_CTLS2:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
+		break;
+	case MSR_IA32_VMX_VMFUNC:
+		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
+		break;
+	}
+
+	*pdata = ctl_low | ((u64)ctl_high << 32);
+}
+
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
@@ -356,11 +388,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 	if (vmcs_version)
 		*vmcs_version = nested_get_evmcs_version(vcpu);
 
-	vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
-	vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
-	vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
-	vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
-	vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC;
-
 	return 0;
 }
arch/x86/kvm/vmx/evmcs.h +1 -0
index 07ebf6882a45..b88d9807a796 100644
@@ -201,5 +201,6 @@ bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa);
 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
+void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata);
 
 #endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/vmx.c +8 -2
index e3394c839dea..8eb74618b8d8 100644
@@ -1849,8 +1849,14 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:
 		if (!nested_vmx_allowed(vcpu))
 			return 1;
-		return vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
-				       &msr_info->data);
+		if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
+				    &msr_info->data))
+			return 1;
+		if (!msr_info->host_initiated &&
+		    vmx->nested.enlightened_vmcs_enabled)
+			nested_evmcs_filter_control_msr(msr_info->index,
+							&msr_info->data);
+		break;
 	case MSR_IA32_RTIT_CTL:
 		if (pt_mode != PT_MODE_HOST_GUEST)
 			return 1;
#15 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 19 lines · Paolo Bonzini
Quoted 4 lines · Vitaly Kuznetsov

With fine grained VMX feature enablement QEMU>=4.2 tries to do KVM_SET_MSRS with default (matching CPU model) values and in case eVMCS is also enabled, fails.

Quoted 9 lines

It would be possible to drop VMX feature filtering completely and make this a guest's responsibility: if it decides to use eVMCS it should know which fields are available and which are not. Hyper-V mostly complies to this, however, there is at least one problematic control:
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
which Hyper-V enables. As there is no 'apic_addr_field' in eVMCS, we fail to handle this properly in KVM. It is unclear how this is supposed to work, genuine Hyper-V doesn't expose the control so it is possible that this is just a bug (in Hyper-V).

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

Quoted 6 lines

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Unless there are objections, I plan to apply these patches.

Doesn't applying this patch contradict your earlier opinion? This patch would still hide the affected controls from the guest because the host controls enlightened_vmcs_enabled.

Quoted 3 lines · Paolo Bonzini

IMHO the features should stay available in case the guest chooses not to use eVMCS. A guest that uses eVMCS should know which features it cannot use and not enable them.

Makes sense, wasn't thinking about the scenario where the guest doesn't support eVMCS or doesn't want to use it for whatever reason.

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS? Or alternatively, squashing all the disallowed bits.

#16 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 5 lines · Sean Christopherson
Quoted 3 lines · Paolo Bonzini

IMHO the features should stay available in case the guest chooses not to use eVMCS. A guest that uses eVMCS should know which features it cannot use and not enable them.

Quoted 7 lines

Makes sense, wasn't thinking about the scenario where the guest doesn't support eVMCS or doesn't want to use it for whatever reason.

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS?

Unfortunately, this doesn't work because ... Windows. Not only Hyper-V enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES, it actually expects it to work (somehow) so when I do

arch/x86/kvm/vmx/evmcs.c +0 -6
index 72359709cdc1..e6c30eec2817 100644
@@ -361,11 +361,5 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
        if (evmcs_already_enabled)
                return 0;
 
-       vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
-       vmx->nested.msrs.entry_ctls_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
-       vmx->nested.msrs.exit_ctls_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
-       vmx->nested.msrs.secondary_ctls_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
-       vmx->nested.msrs.vmfunc_controls &= ~EVMCS1_UNSUPPORTED_VMFUNC;
-
        return 0;
 }
arch/x86/kvm/vmx/nested.c +1 -1
index bb8afe0c5e7f..cd1f5a1c884b 100644
@@ -1590,7 +1590,7 @@ static int copy_enlightened_to_vmcs12(struct vcpu_vmx *vmx)
                        evmcs->pin_based_vm_exec_control;
                vmcs12->vm_exit_controls = evmcs->vm_exit_controls;
                vmcs12->secondary_vm_exec_control =
-                       evmcs->secondary_vm_exec_control;
+                       evmcs->secondary_vm_exec_control & ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
        }
 
        if (unlikely(!(evmcs->hv_clean_fields &

Hyper-V 2016 with > 1 vCPU fails to boot :-(

Signature · 1 line
Vitaly
maintainer
#17 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson
Quoted 9 lines

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Quoted 4 lines
Quoted 1 line

Unless there are objections, I plan to apply these patches.

Doesn't applying this patch contradict your earlier opinion? This patch would still hide the affected controls from the guest because the host controls enlightened_vmcs_enabled.

It does. Unfortunately the key sentence is "we're stuck with it for existing Windows versions". :(

Quoted 5 lines

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS? Or alternatively, squashing all the disallowed bits.

Hmm, that is also a possibility. It's a very hacky one, but I guess adding APIC virtualization to eVMCS would require bumping the version to
2. Vitaly, what do you think?

Paolo

#18 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 24 lines · Paolo Bonzini
Quoted 9 lines · Sean Christopherson
Quoted 9 lines

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Quoted 4 lines
Quoted 1 line

Unless there are objections, I plan to apply these patches.

Doesn't applying this patch contradict your earlier opinion? This patch would still hide the affected controls from the guest because the host controls enlightened_vmcs_enabled.

It does. Unfortunately the key sentence is "we're stuck with it for existing Windows versions". :(

Quoted 5 lines

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS? Or alternatively, squashing all the disallowed bits.

Quoted 3 lines

Hmm, that is also a possibility. It's a very hacky one, but I guess adding APIC virtualization to eVMCS would require bumping the version to
2. Vitaly, what do you think?

As I already replied to Sean I like the idea to filter out unsupported controls from eVMCS but unfortunately it doesn't work: Hyper-V actually expects APIC virtualization to work when it enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES (I have no idea how without apic_access_addr field but). I checked and at least Hyper-V 2016 doesn't boot (when >1 vCPU).

Signature · 1 line
Vitaly
#19 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 12 lines · Vitaly Kuznetsov
Quoted 10 lines · Paolo Bonzini
Quoted 9 lines · Sean Christopherson
Quoted 9 lines

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Unless there are objections, I plan to apply these patches.

Doesn't applying this patch contradict your earlier opinion? This patch would still hide the affected controls from the guest because the host controls enlightened_vmcs_enabled.

It does. Unfortunately the key sentence is "we're stuck with it for existing Windows versions". :(

Ah, I didn't understand what "it" referred to :-)

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS? Or alternatively, squashing all the disallowed bits.

Hmm, that is also a possibility. It's a very hacky one, but I guess adding APIC virtualization to eVMCS would require bumping the version to
2. Vitaly, what do you think?

Quoted 6 lines

As I already replied to Sean I like the idea to filter out unsupported controls from eVMCS but unfortunately it doesn't work: Hyper-V actually expects APIC virtualization to work when it enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES (I have no idea how without apic_access_addr field but). I checked and at least Hyper-V 2016 doesn't boot (when >1 vCPU).

Nice.

I still don't see what we gain from applying this patch. Once eVMCS is enabled by userspace, which presumably happens before the guest is launched, the guest will see the eVMCS-unfriendly controls as being unsupported, both for eVMCS and regular VMCS. AFAICT, we're adding a fairly ugly hack to KVM just so that KVM can lie to userspace about what controls will be exposed to the guest.

Can we extend the API to use cap->args[1] to control whether or not the unsupported controls are removed from vmx->nested.msrs? Userspace could pass '1' to leave the controls untouched and then surgically hide the controls that the guest is too dumb to know it shouldn't use by writing the appropriate MSRs. Assuming existing userspace is expected/required to zero out args[1..3], this would be fully backwards compatible.

arch/x86/kvm/vmx/evmcs.c +3 -3
index 72359709cdc1..241a769be738 100644
@@ -346,8 +346,8 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
        return 0;
 }
 
-int nested_enable_evmcs(struct kvm_vcpu *vcpu,
-                       uint16_t *vmcs_version)
+int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version,
+                       bool allow_unsupported_controls)
 {
        struct vcpu_vmx *vmx = to_vmx(vcpu);
        bool evmcs_already_enabled = vmx->nested.enlightened_vmcs_enabled;
@@ -358,7 +358,7 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
                *vmcs_version = nested_get_evmcs_version(vcpu);
 
        /* We don't support disabling the feature for simplicity. */
-       if (evmcs_already_enabled)
+       if (evmcs_already_enabled || allow_unsupported_controls)
                return 0;
 
        vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
arch/x86/kvm/x86.c +2 -1
index 0cccc52e2d0a..5e1b8d51277b 100644
@@ -4005,7 +4005,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
        case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
                if (!kvm_x86_ops->nested_enable_evmcs)
                        return -ENOTTY;
-               r = kvm_x86_ops->nested_enable_evmcs(vcpu, &vmcs_version);
+               r = kvm_x86_ops->nested_enable_evmcs(vcpu, &vmcs_version,
+                                                    cap->args[1]);
                if (!r) {
                        user_ptr = (void __user *)(uintptr_t)cap->args[0];
                        if (copy_to_user(user_ptr, &vmcs_version,
#20 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 49 lines · Sean Christopherson
Quoted 12 lines · Vitaly Kuznetsov
Quoted 10 lines · Paolo Bonzini
Quoted 9 lines · Sean Christopherson
Quoted 9 lines

Yes, it most likely is and it would be nice if Microsoft fixed it, but I guess we're stuck with it for existing Windows versions. Well, for one we found a bug in Hyper-V and not the converse. :)

There is a problem with this approach, in that we're stuck with it forever due to live migration. But I guess if in the future eVMCS v2 adds an apic_address field we can limit the hack to eVMCS v1. Another possibility is to use the quirks mechanism but it's overkill for now.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Unless there are objections, I plan to apply these patches.

Doesn't applying this patch contradict your earlier opinion? This patch would still hide the affected controls from the guest because the host controls enlightened_vmcs_enabled.

It does. Unfortunately the key sentence is "we're stuck with it for existing Windows versions". :(

Ah, I didn't understand what "it" referred to :-)

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Rather than update vmx->nested.msrs or filter vmx_get_msr(), what about manually adding eVMCS consistency checks on the disallowed bits and handle SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES as a one-off case by simply clearing it from the eVMCS? Or alternatively, squashing all the disallowed bits.

Hmm, that is also a possibility. It's a very hacky one, but I guess adding APIC virtualization to eVMCS would require bumping the version to
2. Vitaly, what do you think?

Quoted 6 lines

As I already replied to Sean I like the idea to filter out unsupported controls from eVMCS but unfortunately it doesn't work: Hyper-V actually expects APIC virtualization to work when it enables SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES (I have no idea how without apic_access_addr field but). I checked and at least Hyper-V 2016 doesn't boot (when >1 vCPU).

Nice.

I still don't see what we gain from applying this patch. Once eVMCS is enabled by userspace, which presumably happens before the guest is launched, the guest will see the eVMCS-unfriendly controls as being unsupported, both for eVMCS and regular VMCS. AFAICT, we're adding a fairly ugly hack to KVM just so that KVM can lie to userspace about what controls will be exposed to the guest.

Quoted 6 lines

Can we extend the API to use cap->args[1] to control whether or not the unsupported controls are removed from vmx->nested.msrs? Userspace could pass '1' to leave the controls untouched and then surgically hide the controls that the guest is too dumb to know it shouldn't use by writing the appropriate MSRs. Assuming existing userspace is expected/required to zero out args[1..3], this would be fully backwards compatible.

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Quoted 7 lines
arch/x86/kvm/vmx/evmcs.c +0 -0
index 72359709cdc1..241a769be738 100644

arch/x86/kvm/vmx/evmcs.c uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu) · line 350

Quoted 7 lines
arch/x86/kvm/vmx/evmcs.c +2 -2
@@ -346,8 +346,8 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu)
        return 0;
 }
 
-int nested_enable_evmcs(struct kvm_vcpu *vcpu,
-                       uint16_t *vmcs_version)
+int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version,
+                       bool allow_unsupported_controls)

Personally, I'd call it 'keep_unsupported_controls'.

Quoted 27 lines
arch/x86/kvm/vmx/evmcs.c +1 -1
 {
        struct vcpu_vmx *vmx = to_vmx(vcpu);
        bool evmcs_already_enabled = vmx->nested.enlightened_vmcs_enabled;
@@ -358,7 +358,7 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,
                *vmcs_version = nested_get_evmcs_version(vcpu);
 
        /* We don't support disabling the feature for simplicity. */
-       if (evmcs_already_enabled)
+       if (evmcs_already_enabled || allow_unsupported_controls)
                return 0;
 
        vmx->nested.msrs.pinbased_ctls_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
arch/x86/kvm/x86.c +2 -1
index 0cccc52e2d0a..5e1b8d51277b 100644
@@ -4005,7 +4005,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
        case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
                if (!kvm_x86_ops->nested_enable_evmcs)
                        return -ENOTTY;
-               r = kvm_x86_ops->nested_enable_evmcs(vcpu, &vmcs_version);
+               r = kvm_x86_ops->nested_enable_evmcs(vcpu, &vmcs_version,
+                                                    cap->args[1]);
                if (!r) {
                        user_ptr = (void __user *)(uintptr_t)cap->args[0];
                        if (copy_to_user(user_ptr, &vmcs_version,
Signature · 1 line
Vitaly
maintainer
#21 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 9 lines · Vitaly Kuznetsov

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Yes, that's the point. It *is* a hack in KVM, but it is generally preferrable to have an easier API for userspace, if there's only one way to do it.

Though we could be a bit more "surgical" and only remove SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on non-eVMCS guests. Vitaly, can you prepare a v2 that does that and adds a huge "hack alert" comment that explains the discussion?

Paolo

#22 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 11 lines · Paolo Bonzini
Quoted 9 lines · Vitaly Kuznetsov

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Quoted 8 lines

Yes, that's the point. It *is* a hack in KVM, but it is generally preferrable to have an easier API for userspace, if there's only one way to do it.

Though we could be a bit more "surgical" and only remove SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on non-eVMCS guests. Vitaly, can you prepare a v2 that does that and adds a huge "hack alert" comment that explains the discussion?

Yes, sure. I'd like to do more testing to make sure filtering out SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES is enough for other Hyper-V versions too (who knows how many bugs are there :-)

Signature · 1 line
Vitaly
#23 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 17 lines · Vitaly Kuznetsov
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · Vitaly Kuznetsov

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Yes, that's the point. It *is* a hack in KVM, but it is generally preferrable to have an easier API for userspace, if there's only one way to do it.

Quoted 8 lines
Quoted 4 lines

Though we could be a bit more "surgical" and only remove SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on non-eVMCS guests. Vitaly, can you prepare a v2 that does that and adds a huge "hack alert" comment that explains the discussion?

Yes, sure. I'd like to do more testing to make sure filtering out SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES is enough for other Hyper-V versions too (who knows how many bugs are there :-)

... and the answer is -- more than one :-)

I've tested Hyper-V 2016/2019 BIOS and UEFI-booted and the minimal viable set seems to be:

MSR_IA32_VMX_PROCBASED_CTLS2:
~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES

MSR_IA32_VMX_ENTRY_CTLS/MSR_IA32_VMX_TRUE_ENTRY_CTLS:
~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL

MSR_IA32_VMX_EXIT_CTLS/MSR_IA32_VMX_TRUE_EXIT_CTLS:

	~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL

with these filtered out all 4 versions are at least able to boot with >1 vCPU and run a nested guest (different from Windows management partition).

This still feels a bit fragile as who knows under which circumstances Hyper-V might want to enable additional (missing) controls.

If there are no objections and if we still think it would be beneficial to minimize the list of controls we filter out (and not go with the full set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).

Signature · 1 line
Vitaly
#24 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 40 lines · Vitaly Kuznetsov
Quoted 17 lines · Vitaly Kuznetsov
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · Vitaly Kuznetsov

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Yes, that's the point. It *is* a hack in KVM, but it is generally preferrable to have an easier API for userspace, if there's only one way to do it.

Quoted 8 lines
Quoted 4 lines

Though we could be a bit more "surgical" and only remove SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on non-eVMCS guests. Vitaly, can you prepare a v2 that does that and adds a huge "hack alert" comment that explains the discussion?

Yes, sure. I'd like to do more testing to make sure filtering out SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES is enough for other Hyper-V versions too (who knows how many bugs are there :-)

... and the answer is -- more than one :-)

I've tested Hyper-V 2016/2019 BIOS and UEFI-booted and the minimal viable set seems to be:

MSR_IA32_VMX_PROCBASED_CTLS2:
~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES

MSR_IA32_VMX_ENTRY_CTLS/MSR_IA32_VMX_TRUE_ENTRY_CTLS:
~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL

MSR_IA32_VMX_EXIT_CTLS/MSR_IA32_VMX_TRUE_EXIT_CTLS:

Quoted 8 lines
	~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL

with these filtered out all 4 versions are at least able to boot with >1 vCPU and run a nested guest (different from Windows management partition).

This still feels a bit fragile as who knows under which circumstances Hyper-V might want to enable additional (missing) controls.

No strong opinion, I'm good either way.

Quoted 3 lines

If there are no objections and if we still think it would be beneficial to minimize the list of controls we filter out (and not go with the full set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).

One last idea, can we keep the MSR filtering as is and add the hack in vmx_restore_control_msr()? That way the (userspace) host and guest see the same values when reading the affected MSRs, and eVMCS wouldn't need it's own hook to do consistency checks.

@@ -1181,28 +1181,38 @@ static int
vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)

 {
        u64 supported;

- u32 *lowp, *highp;
+ u32 *lowp, *highp, evmcs_unsupported;

        switch (msr_index) {
        case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
                lowp = &vmx->nested.msrs.pinbased_ctls_low;
                highp = &vmx->nested.msrs.pinbased_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_PINCTRL;
                break;
        case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
                lowp = &vmx->nested.msrs.procbased_ctls_low;
                highp = &vmx->nested.msrs.procbased_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = 0;
                break;
        case MSR_IA32_VMX_TRUE_EXIT_CTLS:
                lowp = &vmx->nested.msrs.exit_ctls_low;
                highp = &vmx->nested.msrs.exit_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
                break;
        case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
                lowp = &vmx->nested.msrs.entry_ctls_low;
                highp = &vmx->nested.msrs.entry_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
                break;
        case MSR_IA32_VMX_PROCBASED_CTLS2:
                lowp = &vmx->nested.msrs.secondary_ctls_low;
                highp = &vmx->nested.msrs.secondary_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_2NDEXEC;
                break;
        default:
                BUG();

@@ -1210,6 +1220,9 @@ vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)

        supported = vmx_control_msr(*lowp, *highp);

+ /* HACK! */
+ data &= ~(u64)evmcs_unsupported << 32;
+

        /* Check must-be-1 bits are still 1. */
        if (!is_bitwise_subset(data, supported, GENMASK_ULL(31, 0)))
#25 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 52 lines · Sean Christopherson
Quoted 40 lines · Vitaly Kuznetsov
Quoted 17 lines · Vitaly Kuznetsov
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · Vitaly Kuznetsov

Yes, in case we're back to the idea to filter things out in QEMU we can do this. What I don't like is that every other userspace which decides to enable eVMCS will have to perform the exact same surgery as in case it sets allow_unsupported_controls=0 it'll have to know (hardcode) the filtering (or KVM_SET_MSRS will fail) and in case it opts for allow_unsupported_controls=1 Windows guests just won't boot without the filtering.

It seems to be 1:1, eVMCSv1 requires the filter.

Yes, that's the point. It *is* a hack in KVM, but it is generally preferrable to have an easier API for userspace, if there's only one way to do it.

Quoted 8 lines
Quoted 4 lines

Though we could be a bit more "surgical" and only remove SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on non-eVMCS guests. Vitaly, can you prepare a v2 that does that and adds a huge "hack alert" comment that explains the discussion?

Yes, sure. I'd like to do more testing to make sure filtering out SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES is enough for other Hyper-V versions too (who knows how many bugs are there :-)

... and the answer is -- more than one :-)

I've tested Hyper-V 2016/2019 BIOS and UEFI-booted and the minimal viable set seems to be:

MSR_IA32_VMX_PROCBASED_CTLS2:
~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES

MSR_IA32_VMX_ENTRY_CTLS/MSR_IA32_VMX_TRUE_ENTRY_CTLS:
~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL

MSR_IA32_VMX_EXIT_CTLS/MSR_IA32_VMX_TRUE_EXIT_CTLS:

Quoted 8 lines
	~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL

with these filtered out all 4 versions are at least able to boot with >1 vCPU and run a nested guest (different from Windows management partition).

This still feels a bit fragile as who knows under which circumstances Hyper-V might want to enable additional (missing) controls.

No strong opinion, I'm good either way.

Quoted 8 lines
Quoted 3 lines

If there are no objections and if we still think it would be beneficial to minimize the list of controls we filter out (and not go with the full set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).

One last idea, can we keep the MSR filtering as is and add the hack in vmx_restore_control_msr()? That way the (userspace) host and guest see the same values when reading the affected MSRs, and eVMCS wouldn't need it's own hook to do consistency checks.

Yes but (if I'm not mistaken) we'll have then to keep the filtering we currently do in nested_enable_evmcs(): if userspace doesn't do KVM_SET_MSR for VMX MSRs (QEMU<4.2) then the filtering in vmx_restore_control_msr() won't happen and the guest will see the unfiltered set of controls...

Quoted 51 lines
(quoted diff) +14 -1
@@ -1181,28 +1181,38 @@ static int
 vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)
 {
        u64 supported;
-       u32 *lowp, *highp;
+       u32 *lowp, *highp, evmcs_unsupported;
 
        switch (msr_index) {
        case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
                lowp = &vmx->nested.msrs.pinbased_ctls_low;
                highp = &vmx->nested.msrs.pinbased_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_PINCTRL;
                break;
        case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
                lowp = &vmx->nested.msrs.procbased_ctls_low;
                highp = &vmx->nested.msrs.procbased_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = 0;
                break;
        case MSR_IA32_VMX_TRUE_EXIT_CTLS:
                lowp = &vmx->nested.msrs.exit_ctls_low;
                highp = &vmx->nested.msrs.exit_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
                break;
        case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
                lowp = &vmx->nested.msrs.entry_ctls_low;
                highp = &vmx->nested.msrs.entry_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
                break;
        case MSR_IA32_VMX_PROCBASED_CTLS2:
                lowp = &vmx->nested.msrs.secondary_ctls_low;
                highp = &vmx->nested.msrs.secondary_ctls_high;
+               if (vmx->nested.enlightened_vmcs_enabled)
+                       evmcs_unsupported = EVMCS1_UNSUPPORTED_2NDEXEC;
                break;
        default:
                BUG();
@@ -1210,6 +1220,9 @@ vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)
 
        supported = vmx_control_msr(*lowp, *highp);
 
+       /* HACK! */
+       data &= ~(u64)evmcs_unsupported << 32;
+
        /* Check must-be-1 bits are still 1. */
        if (!is_bitwise_subset(data, supported, GENMASK_ULL(31, 0)))
 
Signature · 1 line
Vitaly
maintainer
#26 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 7 lines · Vitaly Kuznetsov
Quoted 7 lines
Quoted 3 lines

If there are no objections and if we still think it would be beneficial to minimize the list of controls we filter out (and not go with the full set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).

One last idea, can we keep the MSR filtering as is and add the hack in vmx_restore_control_msr()? That way the (userspace) host and guest see the same values when reading the affected MSRs, and eVMCS wouldn't need it's own hook to do consistency checks.

Quoted 6 lines

Yes but (if I'm not mistaken) we'll have then to keep the filtering we currently do in nested_enable_evmcs(): if userspace doesn't do KVM_SET_MSR for VMX MSRs (QEMU<4.2) then the filtering in vmx_restore_control_msr() won't happen and the guest will see the unfiltered set of controls...

Indeed. The place you used in the RFC is the best we can do, I am afraid.

Paolo

#27 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

Quoted 8 lines · Paolo Bonzini
Quoted 7 lines · Vitaly Kuznetsov
Quoted 7 lines
Quoted 3 lines

If there are no objections and if we still think it would be beneficial to minimize the list of controls we filter out (and not go with the full set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).

One last idea, can we keep the MSR filtering as is and add the hack in vmx_restore_control_msr()? That way the (userspace) host and guest see the same values when reading the affected MSRs, and eVMCS wouldn't need it's own hook to do consistency checks.

Quoted 9 lines
Quoted 6 lines

Yes but (if I'm not mistaken) we'll have then to keep the filtering we currently do in nested_enable_evmcs(): if userspace doesn't do KVM_SET_MSR for VMX MSRs (QEMU<4.2) then the filtering in vmx_restore_control_msr() won't happen and the guest will see the unfiltered set of controls...

Indeed. The place you used in the RFC is the best we can do, I am afraid.

In case we decide to filter out the full set of unsupported stuff there's basically nothing to change, feel free to just treat the RFC as non-RFC :-) (and personally, I'd prefer to keep the 'full set' in the filter as it is less fragile; the 'short list' I came up with is the result of my experiments on one hardware host only and I'm not sure what may make Hyper-V behave differently).

I can re-submit, of course, if needed.

Signature · 1 line
Vitaly
#28 lore raw

Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

In reply to Vitaly Kuznetsov
Quoted 11 lines · Vitaly Kuznetsov
Quoted 4 lines · Sean Christopherson

One last idea, can we keep the MSR filtering as is and add the hack in vmx_restore_control_msr()? That way the (userspace) host and guest see the same values when reading the affected MSRs, and eVMCS wouldn't need it's own hook to do consistency checks.

Yes but (if I'm not mistaken) we'll have then to keep the filtering we currently do in nested_enable_evmcs(): if userspace doesn't do KVM_SET_MSR for VMX MSRs (QEMU<4.2) then the filtering in vmx_restore_control_msr() won't happen and the guest will see the unfiltered set of controls...

Ya, my thought was to add this on top of the nested_enable_evmcs() code.

#29 lore raw

RFC PATCH 3/3 x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests

In reply to Vitaly Kuznetsov

Sane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.

It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.

---
 arch/x86/kvm/vmx/evmcs.c  | 56 ++++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/evmcs.h  |  1 +
 arch/x86/kvm/vmx/nested.c |  3 +++
 3 files changed, 59 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.c +55 -1
index b5d6582ba589..88f462866396 100644
@@ -4,9 +4,11 @@
 #include <linux/smp.h>
 
 #include "../hyperv.h"
-#include "evmcs.h"
 #include "vmcs.h"
+#include "vmcs12.h"
+#include "evmcs.h"
 #include "vmx.h"
+#include "trace.h"
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
@@ -378,6 +380,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	*pdata = ctl_low | ((u64)ctl_high << 32);
 }
 
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
+{
+	int ret = 0;
+	u32 unsupp_ctl;
+
+	unsupp_ctl = vmcs12->pin_based_vm_exec_control &
+		EVMCS1_UNSUPPORTED_PINCTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported pin-based VM-execution controls",
+			unsupp_ctl);
+		ret = -EINVAL;
+	}
+
+	unsupp_ctl = vmcs12->secondary_vm_exec_control &
+		EVMCS1_UNSUPPORTED_2NDEXEC;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported secondary VM-execution controls",
+			unsupp_ctl);
+		ret = -EINVAL;
+	}
+
+	unsupp_ctl = vmcs12->vm_exit_controls &
+		EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported VM-exit controls",
+			unsupp_ctl);
+		ret = -EINVAL;
+	}
+
+	unsupp_ctl = vmcs12->vm_entry_controls &
+		EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported VM-entry controls",
+			unsupp_ctl);
+		ret = -EINVAL;
+	}
+
+	unsupp_ctl = vmcs12->vm_function_control & EVMCS1_UNSUPPORTED_VMFUNC;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported VM-function controls",
+			unsupp_ctl);
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version)
 {
arch/x86/kvm/vmx/evmcs.h +1 -0
index b88d9807a796..cb7517a5a41c 100644
@@ -202,5 +202,6 @@ uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
 void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata);
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12);
 
 #endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/nested.c +3 -0
index 4aea7d304beb..7c720b095663 100644
@@ -2767,6 +2767,9 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,
 	    nested_check_vm_entry_controls(vcpu, vmcs12))
 		return -EINVAL;
 
+	if (to_vmx(vcpu)->nested.enlightened_vmcs_enabled)
+		return nested_evmcs_check_controls(vmcs12);
+
 	return 0;
 }
 
Signature · 1 line
2.24.1
#30 lore raw

Re: [PATCH RFC 3/3] x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests

Quoted 44 lines · Vitaly Kuznetsov

Sane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.

It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.

---

arch/x86/kvm/vmx/evmcs.c  | 56 ++++++++++++++++++++++++++++++++++++++-
arch/x86/kvm/vmx/evmcs.h  |  1 +
arch/x86/kvm/vmx/nested.c |  3 +++
3 files changed, 59 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.c +4 -1
index b5d6582ba589..88f462866396 100644
@@ -4,9 +4,11 @@
 #include <linux/smp.h>
 
 #include "../hyperv.h"
-#include "evmcs.h"
 #include "vmcs.h"
+#include "vmcs12.h"
+#include "evmcs.h"
 #include "vmx.h"
+#include "trace.h"
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
@@ -378,6 +380,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	*pdata = ctl_low | ((u64)ctl_high << 32);
 }
 
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12)

arch/x86/kvm/vmx/evmcs.c void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) · line 393

Quoted 10 lines
arch/x86/kvm/vmx/evmcs.c +10 -0
+{
+	int ret = 0;
+	u32 unsupp_ctl;
+
+	unsupp_ctl = vmcs12->pin_based_vm_exec_control &
+		EVMCS1_UNSUPPORTED_PINCTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported pin-based VM-execution controls",
+			unsupp_ctl);

Why not move "CC” macro from nested.c to nested.h and use it here as-well instead of replicating it’s logic?

-Liran

#31 lore raw

Re: [PATCH RFC 3/3] x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests

Quoted 41 lines · Liran Alon
Quoted 41 lines · Vitaly Kuznetsov

Sane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.

It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.

---

arch/x86/kvm/vmx/evmcs.c  | 56 ++++++++++++++++++++++++++++++++++++++-
arch/x86/kvm/vmx/evmcs.h  |  1 +
arch/x86/kvm/vmx/nested.c |  3 +++
3 files changed, 59 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.c +3 -1
index b5d6582ba589..88f462866396 100644
@@ -4,9 +4,11 @@
 #include <linux/smp.h>
 
 #include "../hyperv.h"
-#include "evmcs.h"
 #include "vmcs.h"
+#include "vmcs12.h"
+#include "evmcs.h"
 #include "vmx.h"
+#include "trace.h"
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
@@ -378,6 +380,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	*pdata = ctl_low | ((u64)ctl_high << 32);
 }
 
Quoted 14 lines

arch/x86/kvm/vmx/evmcs.c void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) · line 393

Quoted 11 lines
arch/x86/kvm/vmx/evmcs.c +11 -0
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
+{
+	int ret = 0;
+	u32 unsupp_ctl;
+
+	unsupp_ctl = vmcs12->pin_based_vm_exec_control &
+		EVMCS1_UNSUPPORTED_PINCTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported pin-based VM-execution controls",
+			unsupp_ctl);

Why not move "CC” macro from nested.c to nested.h and use it here as-well instead of replicating it’s logic?

Because error messages I add are both human readable and conform to SDM! :-)

On a more serious not yes, we should probably do that. We may even want to use it in non-nesting (and non VMX) code in KVM.

Thanks,

Signature · 1 line
Vitaly
#32 lore raw

Re: [PATCH RFC 3/3] x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests

Quoted 58 lines · Vitaly Kuznetsov
Quoted 41 lines · Liran Alon
Quoted 41 lines · Vitaly Kuznetsov

Sane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.

It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.

---

arch/x86/kvm/vmx/evmcs.c  | 56 ++++++++++++++++++++++++++++++++++++++-
arch/x86/kvm/vmx/evmcs.h  |  1 +
arch/x86/kvm/vmx/nested.c |  3 +++
3 files changed, 59 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.c +3 -1
index b5d6582ba589..88f462866396 100644
@@ -4,9 +4,11 @@
 #include <linux/smp.h>
 
 #include "../hyperv.h"
-#include "evmcs.h"
 #include "vmcs.h"
+#include "vmcs12.h"
+#include "evmcs.h"
 #include "vmx.h"
+#include "trace.h"
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
@@ -378,6 +380,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	*pdata = ctl_low | ((u64)ctl_high << 32);
 }
 
Quoted 14 lines

arch/x86/kvm/vmx/evmcs.c void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) · line 393

Quoted 11 lines
arch/x86/kvm/vmx/evmcs.c +11 -0
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
+{
+	int ret = 0;
+	u32 unsupp_ctl;
+
+	unsupp_ctl = vmcs12->pin_based_vm_exec_control &
+		EVMCS1_UNSUPPORTED_PINCTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported pin-based VM-execution controls",
+			unsupp_ctl);

Why not move "CC” macro from nested.c to nested.h and use it here as-well instead of replicating it’s logic?

Quoted 5 lines

Because error messages I add are both human readable and conform to SDM! :-)

On a more serious not yes, we should probably do that. We may even want to use it in non-nesting (and non VMX) code in KVM.

No, the CC() macro is short for Consistency Check, i.e. specific to nVMX. Even if KVM ends up adding nested_evmcs_check_controls(), which I'm hoping can be avoided, I'd still be hesitant to expose CC() in nested.h.

maintainer
#33 lore raw

Re: [PATCH RFC 3/3] x86/kvm/hyper-v: don't allow to turn on unsupported VMX controls for nested guests

Quoted 62 lines · Sean Christopherson
Quoted 61 lines · Vitaly Kuznetsov
Quoted 41 lines · Liran Alon
Quoted 41 lines · Vitaly Kuznetsov

Sane L1 hypervisors are not supposed to turn any of the unsupported VMX controls on for its guests and nested_vmx_check_controls() checks for that. This is, however, not the case for the controls which are supported on the host but are missing in enlightened VMCS and when eVMCS is in use.

It would certainly be possible to add these missing checks to nested_check_vm_execution_controls()/_vm_exit_controls()/.. but it seems preferable to keep eVMCS-specific stuff in eVMCS and reduce the impact on non-eVMCS guests by doing less unrelated checks. Create a separate nested_evmcs_check_controls() for this purpose.

---

arch/x86/kvm/vmx/evmcs.c  | 56 ++++++++++++++++++++++++++++++++++++++-
arch/x86/kvm/vmx/evmcs.h  |  1 +
arch/x86/kvm/vmx/nested.c |  3 +++
3 files changed, 59 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.c +3 -1
index b5d6582ba589..88f462866396 100644
@@ -4,9 +4,11 @@
 #include <linux/smp.h>
 
 #include "../hyperv.h"
-#include "evmcs.h"
 #include "vmcs.h"
+#include "vmcs12.h"
+#include "evmcs.h"
 #include "vmx.h"
+#include "trace.h"
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
@@ -378,6 +380,58 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	*pdata = ctl_low | ((u64)ctl_high << 32);
 }
 
Quoted 14 lines

arch/x86/kvm/vmx/evmcs.c void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) · line 393

Quoted 11 lines
arch/x86/kvm/vmx/evmcs.c +11 -0
+int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
+{
+	int ret = 0;
+	u32 unsupp_ctl;
+
+	unsupp_ctl = vmcs12->pin_based_vm_exec_control &
+		EVMCS1_UNSUPPORTED_PINCTRL;
+	if (unsupp_ctl) {
+		trace_kvm_nested_vmenter_failed(
+			"eVMCS: unsupported pin-based VM-execution controls",
+			unsupp_ctl);

Why not move "CC” macro from nested.c to nested.h and use it here as-well instead of replicating it’s logic?

Because error messages I add are both human readable and conform to SDM! :-)

Quoted 7 lines
Quoted 2 lines

On a more serious not yes, we should probably do that. We may even want to use it in non-nesting (and non VMX) code in KVM.

No, the CC() macro is short for Consistency Check, i.e. specific to nVMX. Even if KVM ends up adding nested_evmcs_check_controls(), which I'm hoping can be avoided, I'd still be hesitant to expose CC() in nested.h.

For now let's keep Vitaly's patch as is. It's definitely a good one as it would catch Hyper-V's issue immediately even without patch 2 (which is the only really contentious change).

Paolo