Skip to contentSkip to conversation

KVM: nVMX: Fix Windows 11 + WSL2 + Enlightened VMCS

PATCH In review Vitaly Kuznetsov 2 participants 12 messages Jan 12, 2022 – Jan 28, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover KVM: nVMX: Fix Windows 11 + WSL2 + Enlightened VMCS

Changes since v2 [Sean]:
- Tweak a comment in PATCH5.
- Add Reviewed-by: tags to PATCHes 3 and 5.

Original description:

Windows 11 with enabled Hyper-V role doesn't boot on KVM when Enlightened VMCS interface is provided to it. The observed behavior doesn't conform to Hyper-V TLFS. In particular, I'm observing 'VMREAD' instructions trying to access field 0x4404 ("VM-exit interruption information"). TLFS, however, is very clear this should not be happening:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior."

Microsoft confirms this is a bug in Hyper-V which is supposed to get fixed eventually. For the time being, implement a workaround in KVM allowing VMREAD instructions to read from the currently loaded Enlightened VMCS.

Patches 1-2 are unrelated fixes to VMX feature MSR filtering when eVMCS is enabled. Patches 3 and 4 are preparatory changes, patch 5 implements the workaround.

Vitaly Kuznetsov (5):
KVM: nVMX: Also filter MSR_IA32_VMX_TRUE_PINBASED_CTLS when eVMCS
KVM: nVMX: eVMCS: Filter out VM_EXIT_SAVE_VMX_PREEMPTION_TIMER
KVM: nVMX: Rename vmcs_to_field_offset{,_table}
KVM: nVMX: Implement evmcs_field_offset() suitable for handle_vmread()
KVM: nVMX: Allow VMREAD when Enlightened VMCS is in use

 arch/x86/kvm/vmx/evmcs.c  |  4 +--
 arch/x86/kvm/vmx/evmcs.h  | 48 +++++++++++++++++++++++++------
 arch/x86/kvm/vmx/nested.c | 59 +++++++++++++++++++++++++++------------
 arch/x86/kvm/vmx/vmcs12.c |  4 +--
 arch/x86/kvm/vmx/vmcs12.h |  6 ++--
 5 files changed, 87 insertions(+), 34 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/5 KVM: nVMX: Also filter MSR_IA32_VMX_TRUE_PINBASED_CTLS when eVMCS

Similar to MSR_IA32_VMX_EXIT_CTLS/MSR_IA32_VMX_TRUE_EXIT_CTLS, MSR_IA32_VMX_ENTRY_CTLS/MSR_IA32_VMX_TRUE_ENTRY_CTLS pair, MSR_IA32_VMX_TRUE_PINBASED_CTLS needs to be filtered the same way MSR_IA32_VMX_PINBASED_CTLS is currently filtered as guests may solely rely on 'true' MSR data.

Note, none of the currently existing Windows/Hyper-V versions are known to stumble upon the unfiltered MSR_IA32_VMX_TRUE_PINBASED_CTLS, the change is aimed at making the filtering future proof.

---
 arch/x86/kvm/vmx/evmcs.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/kvm/vmx/evmcs.c +1 -0
index ba6f99f584ac..a7ed30d5647a 100644
@@ -362,6 +362,7 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	case MSR_IA32_VMX_PROCBASED_CTLS2:
 		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
 		break;
+	case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
 	case MSR_IA32_VMX_PINBASED_CTLS:
 		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
 		break;
Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/5 KVM: nVMX: eVMCS: Filter out VM_EXIT_SAVE_VMX_PREEMPTION_TIMER

In reply to Vitaly Kuznetsov

Enlightened VMCS v1 doesn't have VMX_PREEMPTION_TIMER_VALUE field, PIN_BASED_VMX_PREEMPTION_TIMER is also filtered out already so it makes sense to filter out VM_EXIT_SAVE_VMX_PREEMPTION_TIMER too.

Note, none of the currently existing Windows/Hyper-V versions are known to enable 'save VMX-preemption timer value' when eVMCS is in use, the change is aimed at making the filtering future proof.

---
 arch/x86/kvm/vmx/evmcs.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/evmcs.h +3 -1
index 16731d2cf231..3a461a32128b 100644
@@ -59,7 +59,9 @@ DECLARE_STATIC_KEY_FALSE(enable_evmcs);
 	 SECONDARY_EXEC_SHADOW_VMCS |					\
 	 SECONDARY_EXEC_TSC_SCALING |					\
 	 SECONDARY_EXEC_PAUSE_LOOP_EXITING)
-#define EVMCS1_UNSUPPORTED_VMEXIT_CTRL (VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL)
+#define EVMCS1_UNSUPPORTED_VMEXIT_CTRL					\
+	(VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL |				\
+	 VM_EXIT_SAVE_VMX_PREEMPTION_TIMER)
 #define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL)
 #define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING)
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH v3 3/5 KVM: nVMX: Rename vmcs_to_field_offset{,_table}

In reply to Vitaly Kuznetsov

vmcs_to_field_offset{,_table} may sound misleading as VMCS is an opaque blob which is not supposed to be accessed directly. In fact, vmcs_to_field_offset{,_table} are related to KVM defined VMCS12 structure.

Rename vmcs_field_to_offset() to get_vmcs12_field_offset() for clarity.

No functional change intended.

---
 arch/x86/kvm/vmx/nested.c | 6 +++---
 arch/x86/kvm/vmx/vmcs12.c | 4 ++--
 arch/x86/kvm/vmx/vmcs12.h | 6 +++---
 3 files changed, 8 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/nested.c +3 -3
index f235f77cbc03..5941ba05b509 100644
@@ -5111,7 +5111,7 @@ static int handle_vmread(struct kvm_vcpu *vcpu)
 	/* Decode instruction info and find the field to read */
 	field = kvm_register_read(vcpu, (((instr_info) >> 28) & 0xf));
 
-	offset = vmcs_field_to_offset(field);
+	offset = get_vmcs12_field_offset(field);
 	if (offset < 0)
 		return nested_vmx_fail(vcpu, VMXERR_UNSUPPORTED_VMCS_COMPONENT);
 
@@ -5214,7 +5214,7 @@ static int handle_vmwrite(struct kvm_vcpu *vcpu)
 
 	field = kvm_register_read(vcpu, (((instr_info) >> 28) & 0xf));
 
-	offset = vmcs_field_to_offset(field);
+	offset = get_vmcs12_field_offset(field);
 	if (offset < 0)
 		return nested_vmx_fail(vcpu, VMXERR_UNSUPPORTED_VMCS_COMPONENT);
 
@@ -6462,7 +6462,7 @@ static u64 nested_vmx_calc_vmcs_enum_msr(void)
 	max_idx = 0;
 	for (i = 0; i < nr_vmcs12_fields; i++) {
 		/* The vmcs12 table is very, very sparsely populated. */
-		if (!vmcs_field_to_offset_table[i])
+		if (!vmcs12_field_offsets[i])
 			continue;
 
 		idx = vmcs_field_index(VMCS12_IDX_TO_ENC(i));
arch/x86/kvm/vmx/vmcs12.c +2 -2
index cab6ba7a5005..2251b60920f8 100644
@@ -8,7 +8,7 @@
 	FIELD(number, name),						\
 	[ROL16(number##_HIGH, 6)] = VMCS12_OFFSET(name) + sizeof(u32)
 
-const unsigned short vmcs_field_to_offset_table[] = {
+const unsigned short vmcs12_field_offsets[] = {
 	FIELD(VIRTUAL_PROCESSOR_ID, virtual_processor_id),
 	FIELD(POSTED_INTR_NV, posted_intr_nv),
 	FIELD(GUEST_ES_SELECTOR, guest_es_selector),
@@ -151,4 +151,4 @@ const unsigned short vmcs_field_to_offset_table[] = {
 	FIELD(HOST_RSP, host_rsp),
 	FIELD(HOST_RIP, host_rip),
 };
-const unsigned int nr_vmcs12_fields = ARRAY_SIZE(vmcs_field_to_offset_table);
+const unsigned int nr_vmcs12_fields = ARRAY_SIZE(vmcs12_field_offsets);
arch/x86/kvm/vmx/vmcs12.h +3 -3
index 2a45f026ee11..746129ddd5ae 100644
@@ -361,10 +361,10 @@ static inline void vmx_check_vmcs12_offsets(void)
 	CHECK_OFFSET(guest_pml_index, 996);
 }
 
-extern const unsigned short vmcs_field_to_offset_table[];
+extern const unsigned short vmcs12_field_offsets[];
 extern const unsigned int nr_vmcs12_fields;
 
-static inline short vmcs_field_to_offset(unsigned long field)
+static inline short get_vmcs12_field_offset(unsigned long field)
 {
 	unsigned short offset;
 	unsigned int index;
@@ -377,7 +377,7 @@ static inline short vmcs_field_to_offset(unsigned long field)
 		return -ENOENT;
 
 	index = array_index_nospec(index, nr_vmcs12_fields);
-	offset = vmcs_field_to_offset_table[index];
+	offset = vmcs12_field_offsets[index];
 	if (offset == 0)
 		return -ENOENT;
 	return offset;
Signature · 1 line
2.34.1
#4 lore raw

PATCH v3 4/5 KVM: nVMX: Implement evmcs_field_offset() suitable for handle_vmread()

In reply to Vitaly Kuznetsov

In preparation to allowing reads from Enlightened VMCS from handle_vmread(), implement evmcs_field_offset() to get the correct read offset. get_evmcs_offset(), which is being used by KVM-on-Hyper-V, is almost what's needed but a few things need to be adjusted. First, WARN_ON() is unacceptable for handle_vmread() as any field can (in theory) be supplied by the guest and not all fields are defined in eVMCS v1. Second, we need to handle 'holes' in eVMCS (missing fields). It also sounds like a good idea to WARN_ON() if such fields are ever accessed by KVM-on-Hyper-V.

Implement dedicated evmcs_field_offset() helper.

No functional change intended.

---
 arch/x86/kvm/vmx/evmcs.c |  3 +--
 arch/x86/kvm/vmx/evmcs.h | 32 ++++++++++++++++++++++++--------
 2 files changed, 25 insertions(+), 10 deletions(-)
arch/x86/kvm/vmx/evmcs.c +1 -2
index a7ed30d5647a..87e3dc10edf4 100644
@@ -12,8 +12,6 @@
 
 DEFINE_STATIC_KEY_FALSE(enable_evmcs);
 
-#if IS_ENABLED(CONFIG_HYPERV)
-
 #define EVMCS1_OFFSET(x) offsetof(struct hv_enlightened_vmcs, x)
 #define EVMCS1_FIELD(number, name, clean_field)[ROL16(number, 6)] = \
 		{EVMCS1_OFFSET(name), clean_field}
@@ -296,6 +294,7 @@ const struct evmcs_field vmcs_field_to_evmcs_1[] = {
 };
 const unsigned int nr_evmcs_1_fields = ARRAY_SIZE(vmcs_field_to_evmcs_1);
 
+#if IS_ENABLED(CONFIG_HYPERV)
 __init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf)
 {
 	vmcs_conf->pin_based_exec_ctrl &= ~EVMCS1_UNSUPPORTED_PINCTRL;
arch/x86/kvm/vmx/evmcs.h +24 -8
index 3a461a32128b..9bc2521b159e 100644
@@ -65,8 +65,6 @@ DECLARE_STATIC_KEY_FALSE(enable_evmcs);
 #define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL)
 #define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING)
 
-#if IS_ENABLED(CONFIG_HYPERV)
-
 struct evmcs_field {
 	u16 offset;
 	u16 clean_field;
@@ -75,26 +73,44 @@ struct evmcs_field {
 extern const struct evmcs_field vmcs_field_to_evmcs_1[];
 extern const unsigned int nr_evmcs_1_fields;
 
-static __always_inline int get_evmcs_offset(unsigned long field,
-					    u16 *clean_field)
+static __always_inline int evmcs_field_offset(unsigned long field,
+					      u16 *clean_field)
 {
 	unsigned int index = ROL16(field, 6);
 	const struct evmcs_field *evmcs_field;
 
-	if (unlikely(index >= nr_evmcs_1_fields)) {
-		WARN_ONCE(1, "KVM: accessing unsupported EVMCS field %lx\n",
-			  field);
+	if (unlikely(index >= nr_evmcs_1_fields))
 		return -ENOENT;
-	}
 
 	evmcs_field = &vmcs_field_to_evmcs_1[index];
 
+	/*
+	 * Use offset=0 to detect holes in eVMCS. This offset belongs to
+	 * 'revision_id' but this field has no encoding and is supposed to
+	 * be accessed directly.
+	 */
+	if (unlikely(!evmcs_field->offset))
+		return -ENOENT;
+
 	if (clean_field)
 		*clean_field = evmcs_field->clean_field;
 
 	return evmcs_field->offset;
 }
 
+#if IS_ENABLED(CONFIG_HYPERV)
+
+static __always_inline int get_evmcs_offset(unsigned long field,
+					    u16 *clean_field)
+{
+	int offset = evmcs_field_offset(field, clean_field);
+
+	WARN_ONCE(offset < 0, "KVM: accessing unsupported EVMCS field %lx\n",
+		  field);
+
+	return offset;
+}
+
 static __always_inline void evmcs_write64(unsigned long field, u64 value)
 {
 	u16 clean_field;
Signature · 1 line
2.34.1
#5 lore raw

PATCH v3 5/5 KVM: nVMX: Allow VMREAD when Enlightened VMCS is in use

In reply to Vitaly Kuznetsov

Hyper-V TLFS explicitly forbids VMREAD and VMWRITE instructions when Enlightened VMCS interface is in use:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior.""

Windows 11 + WSL2 seems to ignore this, attempts to VMREAD VMCS field 0x4404 ("VM-exit interruption information") are observed. Failing these attempts with nested_vmx_failInvalid() makes such guests unbootable.

Microsoft confirms this is a Hyper-V bug and claims that it'll get fixed eventually but for the time being we need a workaround. (Temporary) allow VMREAD to get data from the currently loaded Enlightened VMCS.

Note: VMWRITE instructions remain forbidden, it is not clear how to handle them properly and hopefully won't ever be needed.

---
 arch/x86/kvm/vmx/evmcs.h  | 12 +++++++++
 arch/x86/kvm/vmx/nested.c | 55 +++++++++++++++++++++++++++------------
 2 files changed, 51 insertions(+), 16 deletions(-)
arch/x86/kvm/vmx/evmcs.h +12 -0
index 9bc2521b159e..8d70f9aea94b 100644
@@ -98,6 +98,18 @@ static __always_inline int evmcs_field_offset(unsigned long field,
 	return evmcs_field->offset;
 }
 
+static inline u64 evmcs_read_any(struct hv_enlightened_vmcs *evmcs,
+				 unsigned long field, u16 offset)
+{
+	/*
+	 * vmcs12_read_any() doesn't care whether the supplied structure
+	 * is 'struct vmcs12' or 'struct hv_enlightened_vmcs' as it takes
+	 * the exact offset of the required field, use it for convenience
+	 * here.
+	 */
+	return vmcs12_read_any((void *)evmcs, field, offset);
+}
+
 #if IS_ENABLED(CONFIG_HYPERV)
 
 static __always_inline int get_evmcs_offset(unsigned long field,
arch/x86/kvm/vmx/nested.c +39 -16

72 lines not on this page · raw message

Signature · 1 line
2.34.1
#6 lore raw
In reply to Vitaly Kuznetsov
Quoted 15 lines · Vitaly Kuznetsov

Changes since v2 [Sean]:
- Tweak a comment in PATCH5.
- Add Reviewed-by: tags to PATCHes 3 and 5.

Original description:

Windows 11 with enabled Hyper-V role doesn't boot on KVM when Enlightened VMCS interface is provided to it. The observed behavior doesn't conform to Hyper-V TLFS. In particular, I'm observing 'VMREAD' instructions trying to access field 0x4404 ("VM-exit interruption information"). TLFS, however, is very clear this should not be happening:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior."

Quoted 8 lines

Microsoft confirms this is a bug in Hyper-V which is supposed to get fixed eventually. For the time being, implement a workaround in KVM allowing VMREAD instructions to read from the currently loaded Enlightened VMCS.

Patches 1-2 are unrelated fixes to VMX feature MSR filtering when eVMCS is enabled. Patches 3 and 4 are preparatory changes, patch 5 implements the workaround.

Paolo,

would it be possible to pick this up for 5.17? Technically, this is a "fix", even if the bug itself is not in KVM)

Signature · 1 line
Vitaly
#7 lore raw
Quoted 26 lines · Vitaly Kuznetsov
Quoted 15 lines · Vitaly Kuznetsov

Changes since v2 [Sean]:
- Tweak a comment in PATCH5.
- Add Reviewed-by: tags to PATCHes 3 and 5.

Original description:

Windows 11 with enabled Hyper-V role doesn't boot on KVM when Enlightened VMCS interface is provided to it. The observed behavior doesn't conform to Hyper-V TLFS. In particular, I'm observing 'VMREAD' instructions trying to access field 0x4404 ("VM-exit interruption information"). TLFS, however, is very clear this should not be happening:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior."

Quoted 8 lines

Microsoft confirms this is a bug in Hyper-V which is supposed to get fixed eventually. For the time being, implement a workaround in KVM allowing VMREAD instructions to read from the currently loaded Enlightened VMCS.

Patches 1-2 are unrelated fixes to VMX feature MSR filtering when eVMCS is enabled. Patches 3 and 4 are preparatory changes, patch 5 implements the workaround.

Quoted 4 lines

Paolo,

would it be possible to pick this up for 5.17? Technically, this is a "fix", even if the bug itself is not in KVM)

Ping)

Signature · 1 line
Vitaly
maintainer
#8 lore raw
In reply to Vitaly Kuznetsov
Quoted 26 lines · Vitaly Kuznetsov
Quoted 15 lines · Vitaly Kuznetsov

Changes since v2 [Sean]:
- Tweak a comment in PATCH5.
- Add Reviewed-by: tags to PATCHes 3 and 5.

Original description:

Windows 11 with enabled Hyper-V role doesn't boot on KVM when Enlightened VMCS interface is provided to it. The observed behavior doesn't conform to Hyper-V TLFS. In particular, I'm observing 'VMREAD' instructions trying to access field 0x4404 ("VM-exit interruption information"). TLFS, however, is very clear this should not be happening:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior."

Quoted 8 lines

Microsoft confirms this is a bug in Hyper-V which is supposed to get fixed eventually. For the time being, implement a workaround in KVM allowing VMREAD instructions to read from the currently loaded Enlightened VMCS.

Patches 1-2 are unrelated fixes to VMX feature MSR filtering when eVMCS is enabled. Patches 3 and 4 are preparatory changes, patch 5 implements the workaround.

Quoted 4 lines

Paolo,

would it be possible to pick this up for 5.17? Technically, this is a "fix", even if the bug itself is not in KVM)

Yes, it is. I have queued the patch, and feel free to send a 5.16 backport too.

Paolo

#9 lore raw
Quoted 27 lines · Paolo Bonzini
Quoted 26 lines · Vitaly Kuznetsov
Quoted 23 lines · Vitaly Kuznetsov

Changes since v2 [Sean]:
- Tweak a comment in PATCH5.
- Add Reviewed-by: tags to PATCHes 3 and 5.

Original description:

Windows 11 with enabled Hyper-V role doesn't boot on KVM when Enlightened VMCS interface is provided to it. The observed behavior doesn't conform to Hyper-V TLFS. In particular, I'm observing 'VMREAD' instructions trying to access field 0x4404 ("VM-exit interruption information"). TLFS, however, is very clear this should not be happening:

"Any VMREAD or VMWRITE instructions while an enlightened VMCS is active is unsupported and can result in unexpected behavior."

Microsoft confirms this is a bug in Hyper-V which is supposed to get fixed eventually. For the time being, implement a workaround in KVM allowing VMREAD instructions to read from the currently loaded Enlightened VMCS.

Patches 1-2 are unrelated fixes to VMX feature MSR filtering when eVMCS is enabled. Patches 3 and 4 are preparatory changes, patch 5 implements the workaround.

Quoted 6 lines
Quoted 4 lines

Paolo,

would it be possible to pick this up for 5.17? Technically, this is a "fix", even if the bug itself is not in KVM)

Yes, it is. I have queued the patch

Thanks!

Quoted 1 line

and feel free to send a 5.16 backport too.

I see your pull request to Linus, will send the backport when it lands. In fact, all 5 patches apply to 5.16 without issues but I guess stable@ tooling won't pick them up automatically.

Signature · 1 line
Vitaly
maintainer
#10 lore raw
Quoted 3 lines · Vitaly Kuznetsov

I see your pull request to Linus, will send the backport when it lands. In fact, all 5 patches apply to 5.16 without issues but I guess stable@ tooling won't pick them up automatically.

I noticed, but I wasn't sure what the dependencies were among them and whether you'd prefer to only have patches 3-5; so I didn't just add "Cc: stable" myself.

Paolo

#11 lore raw
Quoted 8 lines · Paolo Bonzini
Quoted 3 lines · Vitaly Kuznetsov

I see your pull request to Linus, will send the backport when it lands. In fact, all 5 patches apply to 5.16 without issues but I guess stable@ tooling won't pick them up automatically.

I noticed, but I wasn't sure what the dependencies were among them and whether you'd prefer to only have patches 3-5; so I didn't just add "Cc:
stable" myself.

Actually yes, patches 1/2 don't fix any "real" issue (afair) so we can be a bit more conservative with 5.16.

Signature · 1 line
Vitaly