Fix issue with not starting nesting guests on my system
cover Fix issue with not starting nesting guests on my system
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr MSR_IA32_UMWAIT_CONTROL which my system doesn't support at all anyway.
I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in KVM_GET_MSR_INDEX_LIST without checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.
When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.
V2: * added a patch to setup correctly the X86_FEATURE_WAITPKG kvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue
Best regards,
Maxim Levitsky
Maxim Levitsky (2):
kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)
Signature · 1 line
2.26.2
PATCH 1/2 kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities
Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.
Note that vmx_waitpkg_supported checks that WAITPKG
_can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.
--- arch/x86/kvm/vmx/vmx.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kvm/vmx/vmx.c +3 -0
@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)/* CPUID 0x80000001 */ if (!cpu_has_vmx_rdtscp ())kvm_cpu_cap_clear (X86_FEATURE_RDTSCP ); + +if (vmx_waitpkg_supported ()) +kvm_cpu_cap_check_and_set (X86_FEATURE_WAITPKG ); }static void vmx_request_immediate_exit (struct kvm_vcpu *vcpu)
Signature · 1 line
2.26.2
Re: [PATCH 1/2] kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities
Quoted 7 lines · Maxim Levitsky
Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.
Note that
vmx_waitpkg_supportedchecks that WAITPKG
_can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.
These line wraps are quite weird and inconsistent.
Quoted 2 lines
Fixes e69e72faa3a0 KVM: x86: Add support for user wait instructions
Checkpatch doesn't complain, but the preferred Fixes format is
e.g.
git show -s --pretty='tformat:%h ("%s")'For the code itself:
Quoted 23 lines
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kvm/vmx/vmx.c +3
-0@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)/* CPUID 0x80000001 */ if (!cpu_has_vmx_rdtscp ())kvm_cpu_cap_clear (X86_FEATURE_RDTSCP ); + +if (vmx_waitpkg_supported ()) +kvm_cpu_cap_check_and_set (X86_FEATURE_WAITPKG ); }static void vmx_request_immediate_exit (struct kvm_vcpu *vcpu)Signature · 1 line
2.26.2
Re: [PATCH 1/2] kvm/x86/vmx: enable X86_FEATURE_WAITPKG in KVM capabilities
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Maxim Levitsky
Even though we might not allow the guest to use WAITPKG's new instructions, we should tell KVM that the feature is supported by the host CPU.
Note that
vmx_waitpkg_supportedchecks that WAITPKG _can_ be set in secondary execution controls as specified by VMX capability MSR, rather that we actually enable it for a guest.These line wraps are quite weird and inconsistent.
Known issue for me, I usually don't have line wrapping enabled, and I wrap the lines a bit earlier that 72 character limit. I'll re-formatted the commit message to be on 72 line format and I will try now to pay much more attention to that.
Quoted 6 lines
Quoted 3 lines
Fixes e69e72faa3a0 KVM: x86: Add support for user waitinstructions
Checkpatch doesn't complain, but the preferred Fixes format is
Quoted 8 lines
Fixes e69e72faa3a07 ("KVM: x86: Add support for user waitinstructions")
e.g.
git show -s --pretty='tformat:%h ("%s")'
Got it, and added to git aliases :-)
Quoted 4 lines
For the code itself:
Reviewed-by Sean Christopherson <[email protected]>
Thank you!
Quoted 24 lines
Quoted 24 lines
Suggested-by Paolo Bonzini <[email protected]>Signed-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kvm/vmx/vmx.c +3
-0@@ -7298,6 +7298,9 @@ static __init void vmx_set_cpu_caps(void)/* CPUID 0x80000001 */ if (!cpu_has_vmx_rdtscp ())kvm_cpu_cap_clear (X86_FEATURE_RDTSCP ); + +if (vmx_waitpkg_supported ()) +kvm_cpu_cap_check_and_set (X86_FEATURE_WAITPKG ); }static void vmx_request_immediate_exit (struct kvm_vcpu *vcpu)Signature · 1 line
2.26.2
PATCH 2/2 kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Maxim LevitskyThis msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kvm/x86.c +4 -0
@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void) min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp)) continue;break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ; +break ;default :break ; }
Signature · 1 line
2.26.2
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 8 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROL
Same comments on the line wraps and Fixes tag.
For the code:
Quoted 23 lines
Signed-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +4
-0@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void) min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp)) continue;break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ; +break ;default :break ; }Signature · 1 line
2.26.2
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSame comments on the line wraps and Fixes tag.
I rewrote the commit message and I hope that the new version is better. I fixed the 'fixes' message as well.
Quoted 4 lines
For the code:
Reviewed-by Sean Christopherson <[email protected]>
Thank you!
Best regards,
Maxim Levitsky
Quoted 25 lines
Quoted 25 lines
Signed-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +4
-0@@ -5316,6 +5316,10 @@ static void kvm_init_msr_list(void)min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp)) continue;break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ; +break ;default :break ;}
Signature · 1 line
2.26.2
Quoted 22 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.
V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queueBest regards,
Maxim Levitsky
Quoted 8 lines
Maxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionallyarch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)
Nit: The added 'break' statement in patch# 2 is not required.
Quoted 28 lines · Krish Sadhukhan
Quoted 26 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.
V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queueBest regards,
Maxim LevitskyMaxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionally
Quoted 5 lines
Quoted 4 lines
arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)Nit: The added 'break' statement in patch# 2 is not required.
It is unless you want to add a fallthrough annotation.
Quoted 19 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.
V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue
Quoted 6 lines
Best regards,
Maxim LevitskyMaxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionally
Standard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)Signature · 1 line
2.26.2
Quoted 25 lines · Sean Christopherson
Quoted 24 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support
at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout
checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety
check doesn't work anymore.V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in
kvm/queue
Quoted 8 lines
Quoted 6 lines
Best regards,
Maxim LevitskyMaxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionallyStandard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".
This another thing I usually mess up in the commit messages. Fixed and noted for futher patches
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)Signature · 1 line
2.26.2
Quoted 24 lines · Sean Christopherson
Quoted 23 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support
at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout
checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety
check doesn't work anymore.V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queue
Quoted 8 lines
Quoted 6 lines
Best regards,
Maxim LevitskyMaxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionallyStandard scoping in the shortlog is "KVM: VMX:" and "KVM: x86:".
Noted and I will use it from now on.
Thanks!
Best regards,
Maxim Levitsky
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)Signature · 1 line
2.26.2
Quoted 22 lines · Maxim Levitsky
On my AMD machine I noticed that I can't start any nested guests, because nested KVM (everything from master git branches) complains that it can't find msr
MSR_IA32_UMWAIT_CONTROLwhich my system doesn't support at all anyway.I traced it to the recently added UMWAIT support to qemu and kvm. The kvm portion exposed the new MSR in
KVM_GET_MSR_INDEX_LISTwithout checking that it the underlying feature is supported in CPUID. It happened to work when non nested because as a precation kvm, tries to read each MSR on host before adding it to that list, and when read gets a #GP it ignores it.When running nested, the L1 hypervisor can be set to ignore unknown msr read/writes (I need this for some other guests), thus this safety check doesn't work anymore.
V2: * added a patch to setup correctly the
X86_FEATURE_WAITPKGkvm capability
* dropped the cosmetic fix patch as it is now fixed in kvm/queueBest regards,
Maxim Levitsky
Quoted 8 lines
Maxim Levitsky (2):
kvm/x86/vmx: enableX86_FEATURE_WAITPKGin KVM capabilities kvm/x86: don't exposeMSR_IA32_UMWAIT_CONTROLunconditionallyarch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 2 files changed, 7 insertions(+)
Queued for 5.7, thanks (with cosmetic touches to the commit message, and moving the "case" earlier to avoid conflicts).
Paolo