Fix breakage from adding MSR_IA32_UMWAIT_CONTROL
cover Fix breakage from adding MSR_IA32_UMWAIT_CONTROL
Currently code in kvm_get_supported_msrs always returns this msr as supported by KVM, however it is not supported at all on AMD and it is only supported on few select Intel systems.
This happened to work in native virtualization case, because KVM's code also tries to read these msrs, and on an exception drops them from the supported msr list.
However when running nested, and outer hypervisor set to ignore unknown msrs, the read from this msr doesn't get an excption, and thus KVM thinks that this msr should be on supported msr list.
I don't think we should rely on exception to check if a feature is supported since that msr can be even in theory assigned to something else on AMD for example.
Also I included a cosmetic fix for an issue I found in the same function.
Best regards,
Maxim Levitsky
Maxim Levitsky (2):
kvm: cosmetic: remove wrong braces in kvm_init_msr_list switch kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
arch/x86/kvm/x86.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/2 kvm: cosmetic: remove wrong braces in kvm_init_msr_list switch
I think these were added accidentally.
--- arch/x86/kvm/x86.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/x86.c +1 -2
@@ -5299,7 +5299,7 @@ static void kvm_init_msr_list(void) !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) continue;break ; - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: { + case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:if (!kvm_cpu_cap_has (X86_FEATURE_INTEL_PT ) || msrs_to_save_all[i] -MSR_IA32_RTIT_ADDR0_A >=intel_pt_validate_hw_cap (PT_CAP_num_address_ranges) *2 )
@@ -5314,7 +5314,6 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; - }default :break ; }
Signature · 1 line
2.26.2
Re: [PATCH 1/2] kvm: cosmetic: remove wrong braces in kvm_init_msr_list switch
Quoted 21 lines · Maxim Levitsky
I think these were added accidentally.
Signed-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/kvm/x86.c +1
-1@@ -5299,7 +5299,7 @@ static void kvm_init_msr_list(void) !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) continue;break ; - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: { + case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:if (!kvm_cpu_cap_has (X86_FEATURE_INTEL_PT ) || msrs_to_save_all[i] -MSR_IA32_RTIT_ADDR0_A >=intel_pt_validate_hw_cap (PT_CAP_num_address_ranges) *2 )
arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5319
Quoted 7 lines
arch/x86/kvm/x86.c +0
-1@@ -5314,7 +5314,6 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; - }default :break ; }
Signature · 1 line
Vitaly
PATCH 2/2 kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Maxim LevitskyThis msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kvm/x86.c +4 -0
@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;default :break ; }
Signature · 1 line
2.26.2
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5320
Quoted 7 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;
I'm probably missing something but (if I understand correctly) the only effect of dropping MSR_IA32_UMWAIT_CONTROL from msrs_to_save would be that KVM userspace won't see it in e.g. KVM_GET_MSR_INDEX_LIST. But why is this causing an issue? I see both vmx_get_msr()/vmx_set_msr() have 'host_initiated' check:
case MSR_IA32_UMWAIT_CONTROL:
if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx))
return 1;so KVM userspace should be able to read/write this MSR even when there's no hardware support for it. Or who's trying to read/write it?
Also, kvm_cpu_cap_has() check is not equal to vmx_has_waitpkg() which checks secondary execution controls.
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0default :break ; }
Signature · 1 line
Vitaly
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 35 lines · Vitaly Kuznetsov
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5320
Quoted 7 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;I'm probably missing something but (if I understand correctly) the only effect of dropping
MSR_IA32_UMWAIT_CONTROLfrommsrs_to_savewould be that KVM userspace won't see it in e.g.KVM_GET_MSR_INDEX_LIST. But why is this causing an issue? I see both vmx_get_msr()/vmx_set_msr() have 'host_initiated' check:
Quoted 3 lines
case MSR_IA32_UMWAIT_CONTROL:
if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx))
return 1;Here it fails like that:
1. KVM_GET_MSR_INDEX_LIST returns this msrs, and qemu notes that it is supported in 'has_msr_umwait' global var
2. Qemu does kvm_arch_get/put_registers->kvm_get/put_msrs->ioctl(KVM_GET_MSRS) and while doing this it adds MSR_IA32_UMWAIT_CONTROL to that msr list. That reaches 'svm_get_msr', and this one knows nothing about MSR_IA32_UMWAIT_CONTROL.
So the difference here is that vmx_get_msr not called at all.
I can add this msr to svm_get_msr instead but that feels wrong since this feature is not yet supported on AMD.
When AMD adds support for this feature, then the VMX specific code can be moved to kvm_get_msr_common I guess.
Quoted 6 lines
so KVM userspace should be able to read/write this MSR even when there's no hardware support for it. Or who's trying to read/write it?
Also,
kvm_cpu_cap_has()check is not equal tovmx_has_waitpkg()which checks secondary execution controls.
I was afraid that something like that will happen, but in this particular case we can only check CPUID support and if supported, the then it means we are dealing with intel system and thus vmx_get_msr will be called and ignore that msr.
Calling vmx_has_waitpkg from the common code doesn't seem right, and besides, it checks the secondary controls which are set by the host and can change, at least in theory during runtime (I don't know if KVM does this).
Note that if I now understand correctly, the 'host_initiated' means that MSR read/write is done by the host itself and not on behalf of the guest.
Best regards,
Maxim Levitsky
Quoted 4 lines
Quoted 4 lines
arch/x86/kvm/x86.c +0
-0default :break ; }
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 49 lines · Maxim Levitsky
Quoted 35 lines · Vitaly Kuznetsov
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5320
Quoted 7 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;I'm probably missing something but (if I understand correctly) the only effect of dropping
MSR_IA32_UMWAIT_CONTROLfrommsrs_to_savewould be that KVM userspace won't see it in e.g.KVM_GET_MSR_INDEX_LIST. But why is this causing an issue? I see both vmx_get_msr()/vmx_set_msr() have 'host_initiated' check:Quoted 3 lines
case MSR_IA32_UMWAIT_CONTROL: if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx)) return 1;Here it fails like that:
1.
KVM_GET_MSR_INDEX_LISTreturns this msrs, and qemu notes that it is supported in 'has_msr_umwait' global var2. Qemu does kvm_arch_get/put_registers->kvm_get/put_msrs->ioctl(KVM_GET_MSRS) and while doing this it adds
MSR_IA32_UMWAIT_CONTROLto that msr list. That reaches 'svm_get_msr', and this one knows nothing aboutMSR_IA32_UMWAIT_CONTROL.
Quoted 7 lines
So the difference here is that
vmx_get_msrnot called at all.
I can add this msr tosvm_get_msrinstead but that feels wrong since this feature is not yet supported on AMD.
When AMD adds support for this feature, then the VMX specific code can be moved tokvm_get_msr_commonI guess.
Oh, SVM, I missed that completely)
Quoted 13 lines
Quoted 7 lines
so KVM userspace should be able to read/write this MSR even when there's no hardware support for it. Or who's trying to read/write it?
Also,
kvm_cpu_cap_has()check is not equal tovmx_has_waitpkg()which checks secondary execution controls.I was afraid that something like that will happen, but in this particular case we can only check CPUID support and if supported, the then it means we are dealing with intel system and thus
vmx_get_msrwill be called and ignore that msr.
Quoted 6 lines
Calling
vmx_has_waitpkgfrom the common code doesn't seem right, and besides, it checks the secondary controls which are set by the host and can change, at least in theory during runtime (I don't know if KVM does this).Note that if I now understand correctly, the '
host_initiated' means that MSR read/write is done by the host itself and not on behalf of the guest.
Yes, it does that.
We have kvm_x86_ops.has_emulated_msr() mechanism, can we use it here? E.g. completely untested
arch/x86/kvm/svm/svm.c +2 -0
@@ -3471,6 +3471,8 @@ static bool svm_has_emulated_msr(int index)case MSR_IA32_MCG_EXT_CTL : case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:return false; +case MSR_IA32_UMWAIT_CONTROL : +return false;default :break ; }
arch/x86/kvm/x86.c +1 -1
@@ -1183,7 +1183,6 @@ static const u32 msrs_to_save_all[] = {MSR_IA32_RTIT_ADDR1_A ,MSR_IA32_RTIT_ADDR1_B ,MSR_IA32_RTIT_ADDR2_A ,MSR_IA32_RTIT_ADDR2_B ,MSR_IA32_RTIT_ADDR3_A ,MSR_IA32_RTIT_ADDR3_B , -MSR_IA32_UMWAIT_CONTROL ,MSR_ARCH_PERFMON_FIXED_CTR0 ,MSR_ARCH_PERFMON_FIXED_CTR1 , MSR_ARCH_PERFMON_FIXED_CTR0 + 2, MSR_ARCH_PERFMON_FIXED_CTR0 + 3,
@@ -1266,6 +1265,7 @@ static const u32 emulated_msrs_all[] = {MSR_IA32_VMX_PROCBASED_CTLS2 ,MSR_IA32_VMX_EPT_VPID_CAP ,MSR_IA32_VMX_VMFUNC , +MSR_IA32_UMWAIT_CONTROL ,MSR_K7_HWCR , MSR_KVM_POLL_CONTROL,
Signature · 1 line
Vitaly
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 111 lines · Vitaly Kuznetsov
Quoted 49 lines · Maxim Levitsky
Quoted 35 lines · Vitaly Kuznetsov
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5320
Quoted 7 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;I'm probably missing something but (if I understand correctly) the only effect of dropping
MSR_IA32_UMWAIT_CONTROLfrommsrs_to_savewould be that KVM userspace won't see it in e.g.KVM_GET_MSR_INDEX_LIST. But why is this causing an issue? I see both vmx_get_msr()/vmx_set_msr() have 'host_initiated' check:Quoted 3 lines
case MSR_IA32_UMWAIT_CONTROL: if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx)) return 1;Here it fails like that:
1.
KVM_GET_MSR_INDEX_LISTreturns this msrs, and qemu notes that it is supported in 'has_msr_umwait' global var2. Qemu does kvm_arch_get/put_registers->kvm_get/put_msrs->ioctl(KVM_GET_MSRS) and while doing this it adds
MSR_IA32_UMWAIT_CONTROLto that msr list. That reaches 'svm_get_msr', and this one knows nothing aboutMSR_IA32_UMWAIT_CONTROL.Quoted 7 lines
So the difference here is that
vmx_get_msrnot called at all.
I can add this msr tosvm_get_msrinstead but that feels wrong since this feature is not yet supported on AMD.
When AMD adds support for this feature, then the VMX specific code can be moved tokvm_get_msr_commonI guess.Oh, SVM, I missed that completely)
Quoted 11 lines
Quoted 5 lines
so KVM userspace should be able to read/write this MSR even when there's no hardware support for it. Or who's trying to read/write it?
Also,
kvm_cpu_cap_has()check is not equal tovmx_has_waitpkg()which checks secondary execution controls.I was afraid that something like that will happen, but in this particular case we can only check CPUID support and if supported, the then it means we are dealing with intel system and thus
vmx_get_msrwill be called and ignore that msr.Quoted 6 lines
Calling
vmx_has_waitpkgfrom the common code doesn't seem right, and besides, it checks the secondary controls which are set by the host and can change, at least in theory during runtime (I don't know if KVM does this).Note that if I now understand correctly, the '
host_initiated' means that MSR read/write is done by the host itself and not on behalf of the guest.Yes, it does that.
We have
kvm_x86_ops.has_emulated_msr()mechanism, can we use it here? E.g. completely untestedarch/x86/kvm/svm/svm.c +2
-0@@ -3471,6 +3471,8 @@ static bool svm_has_emulated_msr(int index)case MSR_IA32_MCG_EXT_CTL : case MSR_IA32_VMX_BASIC ... MSR_IA32_VMX_VMFUNC:return false; +case MSR_IA32_UMWAIT_CONTROL : +return false;default :break ; }arch/x86/kvm/x86.c +0
-1@@ -1183,7 +1183,6 @@ static const u32 msrs_to_save_all[] = {MSR_IA32_RTIT_ADDR1_A ,MSR_IA32_RTIT_ADDR1_B ,MSR_IA32_RTIT_ADDR2_A ,MSR_IA32_RTIT_ADDR2_B ,MSR_IA32_RTIT_ADDR3_A ,MSR_IA32_RTIT_ADDR3_B , -MSR_IA32_UMWAIT_CONTROL ,MSR_ARCH_PERFMON_FIXED_CTR0 ,MSR_ARCH_PERFMON_FIXED_CTR1 , MSR_ARCH_PERFMON_FIXED_CTR0 + 2, MSR_ARCH_PERFMON_FIXED_CTR0 + 3,
arch/x86/kvm/x86.c static const u32 emulated_msrs_all[] = { · line 1271
Quoted 7 lines
arch/x86/kvm/x86.c +1
-0@@ -1266,6 +1265,7 @@ static const u32 emulated_msrs_all[] = {MSR_IA32_VMX_PROCBASED_CTLS2 ,MSR_IA32_VMX_EPT_VPID_CAP ,MSR_IA32_VMX_VMFUNC , +MSR_IA32_UMWAIT_CONTROL ,MSR_K7_HWCR , MSR_KVM_POLL_CONTROL,
I don't see any reason why the above won't work, and to be honest
I also took a look at this but to me it wasn't clear what the purpose of the emulated msrs is, this is why I took the approach in the patch I had sent.
It 'seems' (although this is not enforced anywhere) that emulated msr list is intended for MSRs that are emulated by KVM, which means that KVM traps these msrs, and give guest arbitrary values it thinks that the guest should see.
However MSR_IA32_UMWAIT_CONTROL appears to be exposed directly to the guest without any traps, with the virtualization done by cpu, and the only intervention we do is to set a value to be load when guest mode is entered and value to be loaded when guest mode is done (using VMX msr entry/exit msr lists), I see that done by atomic_switch_umwait_control_msr.
So I am not sure if we should add it to emulated_msrs_all list.
Paulo, what do you think about this? I personally don't mind how to fix this as long as it works and everyone agrees on the patch.
Best regards,
Maxim Levitsky
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Maxim LevitskyQuoted 36 lines · Maxim Levitsky
Quoted 35 lines · Vitaly Kuznetsov
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5320
Quoted 7 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;I'm probably missing something but (if I understand correctly) the only effect of dropping
MSR_IA32_UMWAIT_CONTROLfrommsrs_to_savewould be that KVM userspace won't see it in e.g.KVM_GET_MSR_INDEX_LIST. But why is this causing an issue? I see both vmx_get_msr()/vmx_set_msr() have 'host_initiated' check:
Quoted 8 lines
Quoted 3 lines
case MSR_IA32_UMWAIT_CONTROL: if (!msr_info->host_initiated && !vmx_has_waitpkg(vmx)) return 1;Here it fails like that:
1.
KVM_GET_MSR_INDEX_LISTreturns this msrs, and qemu notes that it is supported in 'has_msr_umwait' global var
In general, KVM_GET_MSR_INDEX_LIST won't return MSR_IA32_UMWAIT_CONTROL if KVM cannot read this MSR, see kvm_init_msr_list().
You hit issue because you used "ignore_msrs".
Paolo Bonzini
maintainerRe: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Maxim LevitskyQuoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5323
Quoted 10 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;default :break ; }
The patch is correct, and matches what is done for the other entries of msrs_to_save_all. However, while looking at it I noticed that X86_FEATURE_WAITPKG is actually never added, and that is because it was also not added to the supported CPUID in commit e69e72faa3a0 ("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before the kvm_cpu_cap mechanism was added.
So while at it you should also fix that. The right way to do that is to add a
if (vmx_waitpkg_supported())
kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);in vmx_set_cpu_caps.
Thanks,
Paolo
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 41 lines · Paolo Bonzini
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5323
Quoted 10 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;default :break ; }The patch is correct, and matches what is done for the other entries of
msrs_to_save_all. However, while looking at it I noticed thatX86_FEATURE_WAITPKGis actually never added, and that is because it was also not added to the supported CPUID in commite69e72faa3a0("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before thekvm_cpu_capmechanism was added.So while at it you should also fix that. The right way to do that is to add a
Quoted 6 lines
if (vmx_waitpkg_supported()) kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);in
vmx_set_cpu_caps.Thanks,
Thank you very much for finding this. I didn't expect this to be broken. I will send a new version with this fix as well tomorrow.
Best regards,
Maxim Levitsky
Quoted 3 lines
Paolo
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Paolo BonziniQuoted 38 lines · Paolo Bonzini
Quoted 19 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>--- arch/x86/kvm/x86.c | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c static void kvm_init_msr_list(void) · line 5323
Quoted 10 lines
arch/x86/kvm/x86.c +4
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -MSR_ARCH_PERFMON_EVENTSEL0 >=min (INTEL_PMC_MAX_GENERIC , x86_pmu.num_counters_gp))continue ; +break ; +case MSR_IA32_UMWAIT_CONTROL : +if (!kvm_cpu_cap_has (X86_FEATURE_WAITPKG )) +continue ;default :break ; }The patch is correct, and matches what is done for the other entries of
msrs_to_save_all. However, while looking at it I noticed thatX86_FEATURE_WAITPKGis actually never added, and that is because it was also not added to the supported CPUID in commite69e72faa3a0("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before thekvm_cpu_capmechanism was added.
Quoted 5 lines
So while at it you should also fix that. The right way to do that is to add a
if (vmx_waitpkg_supported()) kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
+ Tao
I remember there is certainly some reason why we don't expose WAITPKG to guest by default.
Tao, please help clarify it.
Thanks,
-Xiaoyao
Quoted 7 lines
in
vmx_set_cpu_caps.Thanks,
Paolo
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 45 lines · Xiaoyao Li
Quoted 38 lines · Paolo Bonzini
Quoted 21 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>---
arch/x86/kvm/x86.c| 4 ++++1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -
MSR_ARCH_PERFMON_EVENTSEL0>=min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp))Quoted 8 lines
continue;
+ break;
+ caseMSR_IA32_UMWAIT_CONTROL:
+ if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) + continue;
default:
break;
}The patch is correct, and matches what is done for the other entries of msrs_to_save_all. However, while looking at it I noticed that
X86_FEATURE_WAITPKGis actually never added, and that is because it was also not added to the supported CPUID in commite69e72faa3a0("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before thekvm_cpu_capmechanism was added.Quoted 5 lines
So while at it you should also fix that. The right way to do that is to add a
if (
vmx_waitpkg_supported())kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);
Quoted 9 lines
+ Tao
I remember there is certainly some reason why we don't expose WAITPKG to guest by default.
Tao, please help clarify it.
Thanks,
-Xiaoyao
Because in VM, umwait and tpause can put a (psysical) CPU into a power saving state. So from host view, this cpu will be 100% usage by VM. Although umwait and tpause just cause short wait(maybe 100 microseconds), we still want to unconditionally expose WAITPKG in VM.
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 52 lines · Tao Xu
Quoted 49 lines · Xiaoyao Li
Quoted 38 lines · Paolo Bonzini
Quoted 21 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>---
arch/x86/kvm/x86.c| 4 ++++1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -
MSR_ARCH_PERFMON_EVENTSEL0>=min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp))Quoted 8 lines
continue;
+ break;
+ caseMSR_IA32_UMWAIT_CONTROL:
+ if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) + continue;
default:
break;
}The patch is correct, and matches what is done for the other entries of msrs_to_save_all. However, while looking at it I noticed that
X86_FEATURE_WAITPKGis actually never added, and that is because it was also not added to the supported CPUID in commite69e72faa3a0("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before thekvm_cpu_capmechanism was added.Quoted 5 lines
So while at it you should also fix that. The right way to do that is to add a
if (
vmx_waitpkg_supported())kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);+ Tao
I remember there is certainly some reason why we don't expose WAITPKG to guest by default.
Quoted 7 lines
Quoted 5 lines
Tao, please help clarify it.
Thanks,
-XiaoyaoBecause in VM, umwait and tpause can put a (psysical) CPU into a power saving state. So from host view, this cpu will be 100% usage by VM. Although umwait and tpause just cause short wait(maybe 100 microseconds), we still want to unconditionally expose WAITPKG in VM.
I guess you typed "unconditionally" by mistake that you meant to say "conditionally" in fact?
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 55 lines · Xiaoyao Li
Quoted 54 lines · Tao Xu
Quoted 51 lines · Xiaoyao Li
Quoted 38 lines · Paolo Bonzini
Quoted 21 lines · Maxim Levitsky
This msr is only available when the host supports WAITPKG feature.
This breaks a nested guest, if the L1 hypervisor is set to ignore unknown msrs, because the only other safety check that the kernel does is that it attempts to read the msr and rejects it if it gets an exception.
Fixes 6e3ba4abce KVM: vmx: Emulate MSR IA32_UMWAIT_CONTROLSigned-off-by Maxim Levitsky <[email protected]>---
arch/x86/kvm/x86.c| 4 ++++1 file changed, 4 insertions(+)arch/x86/kvm/x86.c +0
-0@@ -5314,6 +5314,10 @@ static void kvm_init_msr_list(void)if (msrs_to_save_all[i] -
MSR_ARCH_PERFMON_EVENTSEL0>=min(INTEL_PMC_MAX_GENERIC, x86_pmu.num_counters_gp))Quoted 8 lines
continue;
+ break;
+ caseMSR_IA32_UMWAIT_CONTROL:
+ if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) + continue;
default:
break;
}The patch is correct, and matches what is done for the other entries of msrs_to_save_all. However, while looking at it I noticed that
X86_FEATURE_WAITPKGis actually never added, and that is because it was also not added to the supported CPUID in commite69e72faa3a0("KVM: x86:
Add support for user wait instructions", 2019-09-24), which was before thekvm_cpu_capmechanism was added.Quoted 5 lines
So while at it you should also fix that. The right way to do that is to add a
if (
vmx_waitpkg_supported())kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG);+ Tao
I remember there is certainly some reason why we don't expose WAITPKG to guest by default.
Tao, please help clarify it.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Thanks,
-XiaoyaoBecause in VM, umwait and tpause can put a (psysical) CPU into a power saving state. So from host view, this cpu will be 100% usage by VM. Although umwait and tpause just cause short wait(maybe 100 microseconds), we still want to unconditionally expose WAITPKG in VM.
I guess you typed "unconditionally" by mistake that you meant to say "conditionally" in fact?
I am sorry, I mean:
By default, we don't expose WAITPKG to guest. For QEMU, we can use "-overcommit cpu-pm=on" to use WAITPKG.
Paolo Bonzini
maintainerRe: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 4 lines · Tao Xu
I am sorry, I mean:
By default, we don't expose WAITPKG to guest. For QEMU, we can use "-overcommit cpu-pm=on" to use WAITPKG.
But UMONITOR, UMWAIT and TPAUSE are not NOPs on older processors (which I should have checked before committing your patch, I admit). So you have broken "-cpu host -overcommit cpu-pm=on" on any processor that doesn't have WAITPKG. I'll send a patch.
Paolo
Re: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
Quoted 5 lines · Paolo Bonzini
Quoted 3 lines · Tao Xu
I am sorry, I mean:
By default, we don't expose WAITPKG to guest. For QEMU, we can use "-overcommit cpu-pm=on" to use WAITPKG.
Quoted 7 lines
But UMONITOR, UMWAIT and TPAUSE are not NOPs on older processors (which I should have checked before committing your patch, I admit). So you have broken "-cpu host -overcommit cpu-pm=on" on any processor that doesn't have WAITPKG. I'll send a patch.
Paolo
Any update on that?
I accidently hit this today while updating my guest's kernel. Turns out I had '-overcommit cpu-pm=on' enabled and -cpu host, and waitpkg (which my AMD cpu doesn't have by any means) was silently exposed to the guest but it didn't use it, but the mainline kernel started using it and so it crashes.
Took me some time to realize that I am hitting this issue.
The CPUID_7_0_ECX_WAITPKG is indeed cleared in KVM_GET_SUPPORTED_CPUID, and code in qemu sets/clears it depending on 'cpu-pm' value.
This patch (copy-pasted so probably not to apply) works for me regardless if we want to fix the KVM_GET_SUPPORTED_CPUID returned value (which I think we should).
It basically detects the presence of the UMWAIT by presense of MSR_IA32_UMWAIT_CONTROL
in the global KVM_GET_MSR_INDEX_LIST, which I recently fixed too, to not return this
msr if the host CPUID doesn't support it.
So this works but is a bit ugly IMHO.
target/i386/kvm.c +1 -1
@@ -412,7 +412,7 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, uint32_t function, ret &= ~(CPUID_7_0_EBX_RTM |CPUID_7_0_EBX_HLE ); } }else if (function ==7 && index ==0 && reg ==R_ECX ) { -if (enable_cpu_pm) { +if (enable_cpu_pm && has_msr_umwait) { ret |=CPUID_7_0_ECX_WAITPKG ; }else { ret &= ~CPUID_7_0_ECX_WAITPKG ;
Signature · 4 lines
Should I send this patch officially? Best regards, Maxim Levitsky
Paolo Bonzini
maintainerRe: [PATCH 2/2] kvm/x86: don't expose MSR_IA32_UMWAIT_CONTROL unconditionally
In reply to Xiaoyao LiQuoted 2 lines · Xiaoyao Li
I remember there is certainly some reason why we don't expose WAITPKG to guest by default.
That's a userspace policy decision. KVM_GET_SUPPORTED_CPUID should still tell userspace that it's supported.
Paolo
Quoted 1 line
Tao, please help clarify it.