Skip to contentSkip to conversation

Fix issue with not starting nesting guests on my system

PATCH Applied Maxim Levitsky 4 participants 13 messages May 23, 2020 – May 27, 2020

Subsystems (3)

v1 of 3 Applied Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Fix issue with not starting nesting guests on my system

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Best regards,
Maxim Levitsky

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities

Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.

Note that vmx_waitpkg_supported checks that WAITPKG
_can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.

Fixes e69e72faa3a0 KVM: x86: Add support for user wait instructions
---
 arch/x86/kvm/vmx/vmx.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/vmx/vmx.c +3 -0
index 55712dd86bafa..fca493d4517c5 100644
@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)
 	/* CPUID 0x80000001 */
 	if (!cpu_has_vmx_rdtscp())
 		kvm_cpu_cap_clear(X86_FEATURE_RDTSCP);
+
+	if (vmx_waitpkg_supported())
+		kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
 }
 
 static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu)
Signature · 1 line
2.26.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities

Quoted 7 lines · Maxim Levitsky

Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.

Note that vmx_waitpkg_supported checks that WAITPKG
_can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.

These line wraps are quite weird and inconsistent.

Quoted 2 lines
Fixes e69e72faa3a0 KVM: x86: Add support for user wait instructions

Checkpatch doesn't complain, but the preferred Fixes format is

Fixes e69e72faa3a07 ("KVM: x86: Add support for user wait instructions")

e.g.

  git show -s --pretty='tformat:%h ("%s")'

For the code itself:

Quoted 23 lines
---
 arch/x86/kvm/vmx/vmx.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/vmx/vmx.c +3 -0
index 55712dd86bafa..fca493d4517c5 100644
@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)
 	/* CPUID 0x80000001 */
 	if (!cpu_has_vmx_rdtscp())
 		kvm_cpu_cap_clear(X86_FEATURE_RDTSCP);
+
+	if (vmx_waitpkg_supported())
+		kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
 }
 
 static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu)
Signature · 1 line
2.26.2
#3 lore raw

Re: [PATCH 1/2] kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities

Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Maxim Levitsky

Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.

Note that vmx_waitpkg_supported checks that WAITPKG _can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.

These line wraps are quite weird and inconsistent.

Known issue for me, I usually don't have line wrapping enabled, and I wrap the lines a bit earlier that 72 character limit. I'll re-formatted the commit message to be on 72 line format and I will try now to pay much more attention to that.

Quoted 6 lines
Quoted 3 lines
Fixes e69e72faa3a0 KVM: x86: Add support for user wait

instructions

Checkpatch doesn't complain, but the preferred Fixes format is

Quoted 8 lines
Fixes e69e72faa3a07 ("KVM: x86: Add support for user wait

instructions")

e.g.

  git show -s --pretty='tformat:%h ("%s")'

Got it, and added to git aliases :-)

Quoted 4 lines

For the code itself:

Thank you!

Quoted 24 lines
Quoted 24 lines
---
 arch/x86/kvm/vmx/vmx.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/vmx/vmx.c +3 -0
index 55712dd86bafa..fca493d4517c5 100644
@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)
 	/* CPUID 0x80000001 */
 	if (!cpu_has_vmx_rdtscp())
 		kvm_cpu_cap_clear(X86_FEATURE_RDTSCP);
+
+	if (vmx_waitpkg_supported())
+		kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
 }
 
 static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu)
Signature · 1 line
2.26.2
#4 lore raw

PATCH 2/2 kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

In reply to Maxim Levitsky

This msr is only available when the host supports WAITPKG feature.

This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.

Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROL
---
 arch/x86/kvm/x86.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/kvm/x86.c +4 -0
index b226fb8abe41b..4752293312947 100644
@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void)
 			    min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp))
 				continue;
 			break;
+		case MSR_IA32_UMWAIT_CONTROL:
+			if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
+				continue;
+			break;
 		default:
 			break;
 		}
Signature · 1 line
2.26.2
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Quoted 8 lines · Maxim Levitsky

This msr is only available when the host supports WAITPKG feature.

This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.

Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROL

Same comments on the line wraps and Fixes tag.

For the code:

Quoted 23 lines
---
 arch/x86/kvm/x86.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/kvm/x86.c +4 -0
index b226fb8abe41b..4752293312947 100644
@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void)
 			    min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp))
 				continue;
 			break;
+		case MSR_IA32_UMWAIT_CONTROL:
+			if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
+				continue;
+			break;
 		default:
 			break;
 		}
Signature · 1 line
2.26.2
#6 lore raw

Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Maxim Levitsky

This msr is only available when the host supports WAITPKG feature.

This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.

Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROL

Same comments on the line wraps and Fixes tag.

I rewrote the commit message and I hope that the new version is better. I fixed the 'fixes' message as well.

Quoted 4 lines

For the code:

Thank you!

Best regards,
Maxim Levitsky

Quoted 25 lines
Quoted 25 lines
---
 arch/x86/kvm/x86.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/kvm/x86.c +4 -0
index b226fb8abe41b..4752293312947 100644
@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void)
 			    min(INTEL_PMC_MAX_GENERIC,
 x86_pmu.num_counters_gp))
 				continue;
 			break;
+		case MSR_IA32_UMWAIT_CONTROL:
+			if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
+				continue;
+			break;
 		default:
 			break;

}

Signature · 1 line
2.26.2
Reviewed-by on the v1 series
#7 lore raw
In reply to Maxim Levitsky
Quoted 22 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Best regards,
Maxim Levitsky

Quoted 8 lines

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

  arch/x86/kvm/vmx/vmx.c | 3 +++
  arch/x86/kvm/x86.c     | 4 ++++
  2 files changed, 7 insertions(+)

Nit: The added 'break' statement in patch# 2 is not required.

#8 lore raw
Quoted 28 lines · Krish Sadhukhan
Quoted 26 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Best regards,
Maxim Levitsky

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Quoted 5 lines
Quoted 4 lines
 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)

Nit: The added 'break' statement in patch# 2 is not required.

It is unless you want to add a fallthrough annotation.

#9 lore raw
In reply to Maxim Levitsky
Quoted 19 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Quoted 6 lines

Best regards,
Maxim Levitsky

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Standard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".

Quoted 9 lines
 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)
Signature · 1 line
2.26.2
#10 lore raw
Quoted 25 lines · Sean Christopherson
Quoted 24 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support
at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without
checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety
check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in
kvm/queue

Quoted 8 lines
Quoted 6 lines

Best regards,
Maxim Levitsky

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Standard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".

This another thing I usually mess up in the commit messages. Fixed and noted for futher patches

Quoted 9 lines
Quoted 9 lines
 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)
Signature · 1 line
2.26.2
#11 lore raw
In reply to Sean Christopherson
Quoted 24 lines · Sean Christopherson
Quoted 23 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support
at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without
checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety
check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Quoted 8 lines
Quoted 6 lines

Best regards,
Maxim Levitsky

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

Standard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".

Noted and I will use it from now on.
Thanks!

Best regards,
Maxim Levitsky

Quoted 9 lines
Quoted 9 lines
 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)
Signature · 1 line
2.26.2
maintainer Applied
#12 lore raw
In reply to Maxim Levitsky
Quoted 22 lines · Maxim Levitsky

On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.

I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.

When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.

V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue

Best regards,
Maxim Levitsky

Quoted 8 lines

Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally

 arch/x86/kvm/vmx/vmx.c | 3 +++
 arch/x86/kvm/x86.c     | 4 ++++
 2 files changed, 7 insertions(+)

Queued for 5.7, thanks (with cosmetic touches to the commit message, and moving the "case" earlier to avoid conflicts).

Paolo