Skip to contentSkip to conversation

KVM: nVMX: Fixes for VMX capability MSR invariance

PATCH In review Oliver Upton 3 participants 14 messages Feb 2, 2022 – Feb 3, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover KVM: nVMX: Fixes for VMX capability MSR invariance

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Such ordering requirements should be entirely avoided. This series stops KVM from modifying VMX VM-{Entry,Exit} control capability MSRs when the CPUID changes. With this series applied, MSR writes from userspace before and after KVM_SET_CPUID2 are preserved.

This series cleanly applies to 5.17-rc2. Confirmed the bug with the included selftest, and also verified the fix. Tested with KVM selftests on a Skylake box.

Oliver Upton (4):
KVM: nVMX: Don't change VM-{Entry,Exit} ctrl MSRs on PMU CPUID update
KVM: nVMX: Don't change VM-{Entry,Exit} ctrl MSRs on MPX CPUID update
selftests: KVM: Add test for "load IA32_PERF_GLOBAL_CTRL" invariance
selftests: KVM: Add test case for "{load/clear} IA32_BNDCFGS"
invariance

 arch/x86/kvm/vmx/nested.c                     |  21 ----
 arch/x86/kvm/vmx/nested.h                     |   1 -
 arch/x86/kvm/vmx/pmu_intel.c                  |   2 -
 arch/x86/kvm/vmx/vmx.c                        |  21 +---
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/include/x86_64/vmx.h        |   2 +
 .../kvm/x86_64/vmx_capability_msrs_test.c     | 119 ++++++++++++++++++
 8 files changed, 124 insertions(+), 44 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_capability_msrs_test.c
Signature · 1 line
2.35.0.rc2.247.g8bbb082509-goog
#1 lore raw

PATCH 1/4 KVM: nVMX: Don't change VM-{Entry,Exit} ctrl MSRs on PMU CPUID update

Ultimately, it is up to userspace to decide what capabilities should be advertised to the guest, including the VMX capability MSRs. Furthermore, there shouldn't be any ordering requirements around the KVM_SET_MSR and KVM_SET_CPUID2 ioctls. However, KVM has not respected the values written by userspace for IA32_VMX_TRUE_{ENTRY,EXIT}_CTLS. Instead, KVM updates the value of these MSRs when CPUID is changed. If userspace advertises a PMU to the guest that supports IA32_PERF_GLOBAL_CTRL (CPUID.0AH:EAX[7:0] > 1), KVM will adjust the values for MSR_IA32_VMX_TRUE_{ENTRY,EXIT}_CTLS to unconditionally advertise the "load IA32_PERF_GLOBAL_CTRL" control bit (or not, if the PMU doesn't support the MSR).

The end result of these shenanigans is that VMMs that clear the "load PERF_GLOBAL_CTRL" bits before setting CPUID will see the bits re-enabled if it provides a supporting vPMU to the guest. Only if the MSR writes happen after setting CPUID will userspace's intentions be upheld.

Since there are no ordering expectations around these ioctls, fix the issue by simply not touching the VMX capability MSRs during an update to guest CPUID. Note that the "load IA32_PERF_GLOBAL_CTRL" bits are already set by default in the respective VMX capability MSRs, if supported by hardware.

Fixes 03a8871add95 ("KVM: nVMX: Expose load IA32_PERF_GLOBAL_CTRL VM-{Entry,Exit} control")
---
 arch/x86/kvm/vmx/nested.c    | 21 ---------------------
 arch/x86/kvm/vmx/nested.h    |  1 -
 arch/x86/kvm/vmx/pmu_intel.c |  2 --
 3 files changed, 24 deletions(-)
arch/x86/kvm/vmx/nested.c +0 -21
index ba34e94049c7..4ed2409c2bd7 100644
@@ -4797,27 +4797,6 @@ int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification,
 	return 0;
 }
 
-void nested_vmx_pmu_entry_exit_ctls_update(struct kvm_vcpu *vcpu)
-{
-	struct vcpu_vmx *vmx;
-
-	if (!nested_vmx_allowed(vcpu))
-		return;
-
-	vmx = to_vmx(vcpu);
-	if (kvm_x86_ops.pmu_ops->is_valid_msr(vcpu, MSR_CORE_PERF_GLOBAL_CTRL)) {
-		vmx->nested.msrs.entry_ctls_high |=
-				VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
-		vmx->nested.msrs.exit_ctls_high |=
-				VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
-	} else {
-		vmx->nested.msrs.entry_ctls_high &=
-				~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
-		vmx->nested.msrs.exit_ctls_high &=
-				~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
-	}
-}
-
 static int nested_vmx_get_vmptr(struct kvm_vcpu *vcpu, gpa_t *vmpointer,
 				int *ret)
 {
arch/x86/kvm/vmx/nested.h +0 -1
index b69a80f43b37..14ad756aac46 100644
@@ -32,7 +32,6 @@ int vmx_set_vmx_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data);
 int vmx_get_vmx_msr(struct nested_vmx_msrs *msrs, u32 msr_index, u64 *pdata);
 int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification,
 			u32 vmx_instruction_info, bool wr, int len, gva_t *ret);
-void nested_vmx_pmu_entry_exit_ctls_update(struct kvm_vcpu *vcpu);
 void nested_mark_vmcs12_pages_dirty(struct kvm_vcpu *vcpu);
 bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcpu, unsigned int port,
 				 int size);
arch/x86/kvm/vmx/pmu_intel.c +0 -2
index 466d18fc0c5d..ad1adbaa7d9e 100644
@@ -541,8 +541,6 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu)
 	bitmap_set(pmu->all_valid_pmc_idx,
 		INTEL_PMC_MAX_GENERIC, pmu->nr_arch_fixed_counters);
 
-	nested_vmx_pmu_entry_exit_ctls_update(vcpu);
-
 	if (intel_pmu_lbr_is_compatible(vcpu))
 		x86_perf_get_lbr(&lbr_desc->records);
 	else
Signature · 1 line
2.35.0.rc2.247.g8bbb082509-goog
#2 lore raw

PATCH 2/4 KVM: nVMX: Don't change VM-{Entry,Exit} ctrl MSRs on MPX CPUID update

In reply to Oliver Upton

KVM does not respect userspace's configuration of the "{load,clear} IA32_BNDCFGS" bit in the IA32_VMX_TRUE_{ENTRY,EXIT}_CTLS MSRs, depending on the order in which it sets the MSR and configures the guest's CPUID. When the guest's CPUID is set, KVM will expose the "load IA32_BNDCFGS" bits if MPX is exposed in the guest CPUID. In order to clear the bit, userspace would need to write the VMX capability MSRs after updating CPUID.

There are no ordering requirements on these ioctls. Fix the issue by simply not updating the "{load,clear} IA32_BNDCFGS" bits on CPUID update. Note that these bits are already exposed by default in the respective VMX capability MSRs, if supported by hardware.

Fixes 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled")
---
 arch/x86/kvm/vmx/vmx.c | 21 +--------------------
 1 file changed, 1 insertion(+), 20 deletions(-)
arch/x86/kvm/vmx/vmx.c +1 -20
index aca3ae2a02f3..8cf58ba60b01 100644
@@ -7227,23 +7227,6 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)
 #undef cr4_fixed1_update
 }
 
-static void nested_vmx_entry_exit_ctls_update(struct kvm_vcpu *vcpu)
-{
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
-
-	if (kvm_mpx_supported()) {
-		bool mpx_enabled = guest_cpuid_has(vcpu, X86_FEATURE_MPX);
-
-		if (mpx_enabled) {
-			vmx->nested.msrs.entry_ctls_high |= VM_ENTRY_LOAD_BNDCFGS;
-			vmx->nested.msrs.exit_ctls_high |= VM_EXIT_CLEAR_BNDCFGS;
-		} else {
-			vmx->nested.msrs.entry_ctls_high &= ~VM_ENTRY_LOAD_BNDCFGS;
-			vmx->nested.msrs.exit_ctls_high &= ~VM_EXIT_CLEAR_BNDCFGS;
-		}
-	}
-}
-
 static void update_intel_pt_cfg(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -7335,10 +7318,8 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 			~(FEAT_CTL_VMX_ENABLED_INSIDE_SMX |
 			  FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX);
 
-	if (nested_vmx_allowed(vcpu)) {
+	if (nested_vmx_allowed(vcpu))
 		nested_vmx_cr_fixed1_bits_update(vcpu);
-		nested_vmx_entry_exit_ctls_update(vcpu);
-	}
 
 	if (boot_cpu_has(X86_FEATURE_INTEL_PT) &&
 			guest_cpuid_has(vcpu, X86_FEATURE_INTEL_PT))
Signature · 1 line
2.35.0.rc2.247.g8bbb082509-goog
#3 lore raw

PATCH 3/4 selftests: KVM: Add test for "load IA32_PERF_GLOBAL_CTRL" invariance

In reply to Oliver Upton

Assert that clearing the "load IA32_PERF_GLOBAL_CTRL" VM-{Entry,Exit} capability bits is preserved across KVM_SET_CPUID2.

---
 tools/testing/selftests/kvm/.gitignore        |  1 +
 tools/testing/selftests/kvm/Makefile          |  1 +
 .../kvm/x86_64/vmx_capability_msrs_test.c     | 82 +++++++++++++++++++
 3 files changed, 84 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86_64/vmx_capability_msrs_test.c
tools/testing/selftests/kvm/.gitignore +1 -0
index dce7de7755e6..d1e6757aed2f 100644
@@ -36,6 +36,7 @@
 /x86_64/userspace_io_test
 /x86_64/userspace_msr_exit_test
 /x86_64/vmx_apic_access_test
+/x86_64/vmx_capability_msrs_test
 /x86_64/vmx_close_while_nested_test
 /x86_64/vmx_dirty_log_test
 /x86_64/vmx_exception_with_invalid_guest_state
tools/testing/selftests/kvm/Makefile +1 -0
index 0e4926bc9a58..083c437a852f 100644
@@ -68,6 +68,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/sync_regs_test
 TEST_GEN_PROGS_x86_64 += x86_64/userspace_io_test
 TEST_GEN_PROGS_x86_64 += x86_64/userspace_msr_exit_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_apic_access_test
+TEST_GEN_PROGS_x86_64 += x86_64/vmx_capability_msrs_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_close_while_nested_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_dirty_log_test
 TEST_GEN_PROGS_x86_64 += x86_64/vmx_exception_with_invalid_guest_state
tools/testing/selftests/kvm/x86_64/vmx_capability_msrs_test.c new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.35.0.rc2.247.g8bbb082509-goog
#4 lore raw

PATCH 4/4 selftests: KVM: Add test case for "{load/clear} IA32_BNDCFGS" invariance

In reply to Oliver Upton

Assert that clearing the "{load/clear IA32_BNDCFGS" bits is preserved across KVM_SET_CPUID2.

---
 .../selftests/kvm/include/x86_64/vmx.h        |  2 +
 .../kvm/x86_64/vmx_capability_msrs_test.c     | 37 +++++++++++++++++++
 2 files changed, 39 insertions(+)
tools/testing/selftests/kvm/include/x86_64/vmx.h +2 -0
index 583ceb0d1457..811c66d9be74 100644
@@ -80,6 +80,7 @@
 #define VM_EXIT_SAVE_IA32_EFER			0x00100000
 #define VM_EXIT_LOAD_IA32_EFER			0x00200000
 #define VM_EXIT_SAVE_VMX_PREEMPTION_TIMER	0x00400000
+#define VM_EXIT_CLEAR_BNDCFGS			0x00800000
 
 #define VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR	0x00036dff
 
@@ -90,6 +91,7 @@
 #define VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL	0x00002000
 #define VM_ENTRY_LOAD_IA32_PAT			0x00004000
 #define VM_ENTRY_LOAD_IA32_EFER			0x00008000
+#define VM_ENTRY_LOAD_BNDCFGS			0x00010000
 
 #define VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR	0x000011ff
 
tools/testing/selftests/kvm/x86_64/vmx_capability_msrs_test.c +37 -0
index 8a1a545e658b..a0851b1224aa 100644
@@ -67,6 +67,42 @@ static void load_perf_global_ctrl_test(struct kvm_vm *vm)
 		    "\"load IA32_PERF_GLOBAL_CTRL\" VM-Exit bit set");
 }
 
+/*
+ * Test to assert that clearing the "load IA32_BNDCFGS" and "clear IA32_BNDCFGS"
+ * control capability bits is preserved across a KVM_SET_CPUID2.
+ */
+static void bndcfgs_ctrl_test(struct kvm_vm *vm)
+{
+	uint32_t entry_low, entry_high, exit_low, exit_high;
+	struct kvm_cpuid2 *cpuid;
+
+	get_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_ENTRY_CTLS, &entry_low, &entry_high);
+	get_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_EXIT_CTLS, &exit_low, &exit_high);
+
+	if (!(entry_high & VM_ENTRY_LOAD_BNDCFGS) || !(exit_high & VM_EXIT_CLEAR_BNDCFGS)) {
+		print_skip("\"{load,clear} IA32_BNDCFGS\" controls not supported");
+		return;
+	}
+
+	entry_high &= ~VM_ENTRY_LOAD_BNDCFGS;
+	exit_high &= ~VM_EXIT_CLEAR_BNDCFGS;
+
+	set_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_ENTRY_CTLS, entry_low, entry_high);
+	set_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_EXIT_CTLS, exit_low, exit_high);
+
+	cpuid = kvm_get_supported_cpuid();
+	vcpu_set_cpuid(vm, VCPU_ID, cpuid);
+
+	get_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_ENTRY_CTLS, &entry_low, &entry_high);
+	get_vmx_capability_msr(vm, MSR_IA32_VMX_TRUE_EXIT_CTLS, &exit_low, &exit_high);
+
+	TEST_ASSERT(!(entry_high & VM_ENTRY_LOAD_BNDCFGS),
+		    "\"load IA32_BNDCFGS\" VM-Entry bit set");
+	TEST_ASSERT(!(exit_high & VM_EXIT_CLEAR_BNDCFGS),
+		    "\"clear IA32_BNDCFGS\" VM-Exit bit set");
+}
+
+
 int main(void)
 {
 	struct kvm_vm *vm;
@@ -77,6 +113,7 @@ int main(void)
 	vm = vm_create_default(VCPU_ID, 0, NULL);
 
 	load_perf_global_ctrl_test(vm);
+	bndcfgs_ctrl_test(vm);
 
 	kvm_vm_free(vm);
 }
Signature · 1 line
2.35.0.rc2.247.g8bbb082509-goog
#5 lore raw
In reply to Oliver Upton
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Do you mean *before*?

#6 lore raw
Quoted 11 lines · Jim Mattson
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Quoted 1 line

Do you mean *before*?

No, after, otherwise the CPUID updates will override the MSR updates.

MSR_IA32_FEAT_CTL has this same issue. But that mess also highlights an issue with this series: if userspace relies on KVM to do the updates, it will break the existing ABI, e.g. I'm pretty sure older versions of QEMU rely on KVM to adjust the MSRs.

I agree that KVM should keep its nose out of this stuff, especially since most VMX controls are technically not architecturally tied to CPUID. But we probably need an opt-in from userspace to stop mucking with the MSRs.

#7 lore raw
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Jim Mattson
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Quoted 3 lines
Quoted 1 line

Do you mean *before*?

No, after, otherwise the CPUID updates will override the MSR updates.

Wasn't that the intention behind this code in the first place (to override KVM_SET_MSR based on CPUID bits)? If not, what was the intention behind this code?

#8 lore raw
Quoted 15 lines · Jim Mattson
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Jim Mattson
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Do you mean *before*?

No, after, otherwise the CPUID updates will override the MSR updates.

Wasn't that the intention behind this code in the first place (to override KVM_SET_MSR based on CPUID bits)? If not, what was the intention behind this code?

Suppose a VMM desperately wants to hide the "load IA32_PERF_GLOBAL_CTRL" bits, in spite of providing a supporting vPMU. The only way to do so at the moment is to write the control MSR after the CPUID write.

#9 lore raw
In reply to Jim Mattson
Quoted 14 lines · Jim Mattson
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Jim Mattson
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Do you mean *before*?

No, after, otherwise the CPUID updates will override the MSR updates.

Wasn't that the intention behind this code in the first place (to override KVM_SET_MSR based on CPUID bits)? If not, what was the intention behind this code?

The MPX side is from commit 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled"), which was a miguided "fix" to workaround an L1 KVM bug in L0. And also to workaround an L0 userspace bug (failure to set VMX MSRs).

The PMU bug looks to be a similar snafu, it appears to workaround a userspace bug (again, failure to set VMX MSRs) in KVM.

But once userspace started taking ownership of VMX MSRs, KVM's hack-a-fixes just got in the way :-/

#10 lore raw
In reply to Sean Christopherson
Quoted 13 lines · Sean Christopherson
Quoted 11 lines · Jim Mattson
Quoted 9 lines · Oliver Upton

Ultimately, it is the responsibility of userspace to configure an appropriate MSR value for the CPUID it provides its guest. However, there are a few bits in VMX capability MSRs where KVM intervenes. The "load IA32_PERF_GLOBAL_CTRL", "load IA32_BNDCFGS", and "clear IA32_BNDCFGS" bits in the VMX VM-{Entry,Exit} control capability MSRs are updated every time userspace sets the guest's CPUID. In so doing, there is an imposed ordering between ioctls, that userspace must set MSR values *after* setting the guest's CPUID.

Quoted 8 lines
Quoted 1 line

Do you mean *before*?

No, after, otherwise the CPUID updates will override the MSR updates.

MSR_IA32_FEAT_CTL has this same issue. But that mess also highlights an issue with this series: if userspace relies on KVM to do the updates, it will break the existing ABI, e.g. I'm pretty sure older versions of QEMU rely on KVM to adjust the MSRs.

I realize I failed to add a note about exactly this in the cover letter. It seems, based on the commit 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled") we opted to handle the VMX capability MSR in-kernel rather than expecting userspace to pick a sane value that matches the set CPUID. So what really has become ABI here? It seems as though one could broadly state that KVM owns VMX VM-{Entry,Exit} control MSRs without opt-in, or narrowly assert that only the bits in this series are in fact ABI.

Regardless, since we must uphold this misbehavior as ABI, we have a regression since KVM doesn't override the MSR write if it comes after the CPUID write.

Quoted 3 lines

I agree that KVM should keep its nose out of this stuff, especially since most VMX controls are technically not architecturally tied to CPUID. But we probably need an opt-in from userspace to stop mucking with the MSRs.

Bleh, I wanted to avoid the age-old problem of naming, but alas...

--
Thanks,
Oliver

#11 lore raw
Quoted 6 lines · Oliver Upton
Quoted 4 lines · Sean Christopherson

MSR_IA32_FEAT_CTL has this same issue. But that mess also highlights an issue with this series: if userspace relies on KVM to do the updates, it will break the existing ABI, e.g. I'm pretty sure older versions of QEMU rely on KVM to adjust the MSRs.

Quoted 8 lines

I realize I failed to add a note about exactly this in the cover letter. It seems, based on the commit 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled") we opted to handle the VMX capability MSR in-kernel rather than expecting userspace to pick a sane value that matches the set CPUID. So what really has become ABI here? It seems as though one could broadly state that KVM owns VMX VM-{Entry,Exit} control MSRs without opt-in, or narrowly assert that only the bits in this series are in fact ABI.

I don't know Paolo's position, but personally I feel quite strongly that KVM should not manipulate the guest vCPU model. KVM should reject changes that put the kernel at risk, but otherwise userspace should have full control.

Quoted 9 lines

Regardless, since we must uphold this misbehavior as ABI, we have a regression since KVM doesn't override the MSR write if it comes after the CPUID write.

Quoted 3 lines

I agree that KVM should keep its nose out of this stuff, especially since most VMX controls are technically not architecturally tied to CPUID. But we probably need an opt-in from userspace to stop mucking with the MSRs.

Bleh, I wanted to avoid the age-old problem of naming, but alas...

I think a single quirk would suffice, e.g. KVM_X86_QUIRK_KVM_DOESNT_LIKE_TO_SHARE.

#12 lore raw
Quoted 17 lines · Sean Christopherson
Quoted 6 lines · Oliver Upton
Quoted 4 lines · Sean Christopherson

MSR_IA32_FEAT_CTL has this same issue. But that mess also highlights an issue with this series: if userspace relies on KVM to do the updates, it will break the existing ABI, e.g. I'm pretty sure older versions of QEMU rely on KVM to adjust the MSRs.

Quoted 8 lines

I realize I failed to add a note about exactly this in the cover letter. It seems, based on the commit 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled") we opted to handle the VMX capability MSR in-kernel rather than expecting userspace to pick a sane value that matches the set CPUID. So what really has become ABI here? It seems as though one could broadly state that KVM owns VMX VM-{Entry,Exit} control MSRs without opt-in, or narrowly assert that only the bits in this series are in fact ABI.

Quoted 3 lines

I don't know Paolo's position, but personally I feel quite strongly that KVM should not manipulate the guest vCPU model. KVM should reject changes that put the kernel at risk, but otherwise userspace should have full control.

Ditto, just want to make sure there's agreement that such a quirk only applies to these bits and not the whole pile.

#13 lore raw
In reply to Sean Christopherson
Quoted 17 lines · Sean Christopherson
Quoted 6 lines · Oliver Upton
Quoted 4 lines · Sean Christopherson

MSR_IA32_FEAT_CTL has this same issue. But that mess also highlights an issue with this series: if userspace relies on KVM to do the updates, it will break the existing ABI, e.g. I'm pretty sure older versions of QEMU rely on KVM to adjust the MSRs.

Quoted 8 lines

I realize I failed to add a note about exactly this in the cover letter. It seems, based on the commit 5f76f6f5ff96 ("KVM: nVMX: Do not expose MPX VMX controls when guest MPX disabled") we opted to handle the VMX capability MSR in-kernel rather than expecting userspace to pick a sane value that matches the set CPUID. So what really has become ABI here? It seems as though one could broadly state that KVM owns VMX VM-{Entry,Exit} control MSRs without opt-in, or narrowly assert that only the bits in this series are in fact ABI.

Quoted 3 lines

I don't know Paolo's position, but personally I feel quite strongly that KVM should not manipulate the guest vCPU model. KVM should reject changes that put the kernel at risk, but otherwise userspace should have full control.

I agree wholeheartedly. Userspace should not have to do KVM_GET_CPUID2 after KVM_SET_CPUID2 to find out what the guest's actual CPUID table looks like. However, today, it does.

Similarly, for read-only MSRs, userspace should be able to assume that KVM_GET_MSRS will always return the same value that was passed to KVM_SET_MSRS. However, that is not the case today, either.