kvm: x86: emulate MSR_PLATFORM_INFO msr bits
PATCH 1/1 kvm: x86: emulate MSR_PLATFORM_INFO msr bits
For intel platform, The BzyMhz field of Turbostat shows zero due to the missing of part msr bits of MSR_PLATFORM_INFO.
Acquire necessary msr bits, and expose following msr info to guest, to make sure guest can get correct turbo frequency info.
MSR_PLATFORM_INFO bits
bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)
--- arch/x86/include/asm/msr-index.h | 4 ++++ arch/x86/kvm/x86.c | 25 ++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-)
arch/x86/include/asm/msr-index.h +4 -0
@@ -68,6 +68,10 @@#define MSR_PLATFORM_INFO 0x000000ce#define MSR_PLATFORM_INFO_CPUID_FAULT_BIT 31#define MSR_PLATFORM_INFO_CPUID_FAULT BIT_ULL(MSR_PLATFORM_INFO_CPUID_FAULT_BIT) +/* MSR_PLATFORM_INFO bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) */ +#define MSR_PLATFORM_INFO_MAX_NON_TURBO_LIM_RATIO 0x00000000ff00 +/* MSR_PLATFORM_INFO bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO) */ +#define MSR_PLATFORM_INFO_MAX_EFFICIENCY_RATIO 0xff0000000000#define MSR_IA32_UMWAIT_CONTROL 0xe1#define MSR_IA32_UMWAIT_CONTROL_C02_DISABLE BIT(0)
arch/x86/kvm/x86.c +24 -1
@@ -1679,6 +1679,29 @@ static u64 kvm_get_arch_capabilities(void)return data; } + +static u64 kvm_get_msr_platform_info (void ) +{ +u64 msr_platform_info =0 ; + +rdmsrl_safe (MSR_PLATFORM_INFO , &msr_platform_info); +/* +* MSR_PLATFORM_INFO bits: +* bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) +* bit 31, CPUID Faulting Enabled (CPUID_FAULTING_EN) +* bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO) +* +* Emulate part msr bits, expose above msr info to guest, +* to make sure guest can get correct turbo frequency info. +*/ + + msr_platform_info &= (MSR_PLATFORM_INFO_MAX_NON_TURBO_LIM_RATIO | +MSR_PLATFORM_INFO_MAX_EFFICIENCY_RATIO ); + msr_platform_info |=MSR_PLATFORM_INFO_CPUID_FAULT ; + +return msr_platform_info; +} +static int kvm_get_msr_feature (struct kvm_msr_entry *msr) {switch (msr->index) {
@@ -11919,7 +11942,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) goto free_guest_fpu; vcpu->arch.arch_capabilities =kvm_get_arch_capabilities (); - vcpu->arch.msr_platform_info =MSR_PLATFORM_INFO_CPUID_FAULT ; + vcpu->arch.msr_platform_info =kvm_get_msr_platform_info ();kvm_xen_init_vcpu (vcpu);kvm_vcpu_mtrr_init (vcpu);vcpu_load (vcpu);
Signature · 1 line
1.8.3.1
Quoted 5 lines · Hao Xiang
For intel platform, The BzyMhz field of Turbostat shows zero due to the missing of part msr bits of
MSR_PLATFORM_INFO.Acquire necessary msr bits, and expose following msr info to guest, to make sure guest can get correct turbo frequency info.
Userspace VMM (e.g., QEMU) can configure this MSR for guests. Please refer to tools/testing/selftests/kvm/x86_64/platform_info_test.c.
The question is why KVM needs this patch given KVM already provides interfaces for QEMU to configure the MSR.
For reason that,
The turbo frequency info depends on specific machine type. And the msr value of MSR_PLATFORM_INFO may be diferent on diffrent generation machine.
Get following msr bits (needed by turbostat on intel platform) by rdmsr MSR_PLATFORM_INFO directly in KVM is more reasonable. And set these msr bits as vcpu->arch.msr_platform_info default value.
-bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO)
-bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)
Quoted 12 lines · Chao Gao
Quoted 5 lines · Hao Xiang
For intel platform, The BzyMhz field of Turbostat shows zero due to the missing of part msr bits of
MSR_PLATFORM_INFO.Acquire necessary msr bits, and expose following msr info to guest, to make sure guest can get correct turbo frequency info.
Userspace VMM (e.g., QEMU) can configure this MSR for guests. Please refer to
tools/testing/selftests/kvm/x86_64/platform_info_test.c.The question is why KVM needs this patch given KVM already provides interfaces for QEMU to configure the MSR.
Review
Quoted 10 lines · Hao Xiang
For reason that,
The turbo frequency info depends on specific machine type. And the msr value of
MSR_PLATFORM_INFOmay be diferent on diffrent generation machine.Get following msr bits (needed by turbostat on intel platform) by rdmsr
MSR_PLATFORM_INFOdirectly in KVM is more reasonable. And set these msr bits asvcpu->arch.msr_platform_infodefault value.(quoted diff) +0
-2-bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) -bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)
I don't get why QEMU cannot do this with the existing interface, e.g., KVM_SET_MSRS.
will the MSR value be migrated during VM migration?
looks we are in a dilemma. on one side, if the value is migrated, the value can become inconsisntent with hardware value. On the other side, changing the ratio bits at runtime isn't the architectural behavior.
And the MSR is per-socket. In theory, a system can have two sockets with different values of the MSR. what if a vCPU is created on a socket and then later runs on the other socket?
Quoted 14 lines · Chao Gao
Quoted 12 lines · Hao Xiang
Quoted 5 lines
For intel platform, The BzyMhz field of Turbostat shows zero due to the missing of part msr bits of
MSR_PLATFORM_INFO.Acquire necessary msr bits, and expose following msr info to guest, to make sure guest can get correct turbo frequency info.
Userspace VMM (e.g., QEMU) can configure this MSR for guests. Please refer to
tools/testing/selftests/kvm/x86_64/platform_info_test.c.The question is why KVM needs this patch given KVM already provides interfaces for QEMU to configure the MSR.
Quoted 17 lines · Chao Gao
Review
Quoted 10 lines · Hao Xiang
For reason that,
The turbo frequency info depends on specific machine type. And the msr value of
MSR_PLATFORM_INFOmay be diferent on diffrent generation machine.Get following msr bits (needed by turbostat on intel platform) by rdmsr
MSR_PLATFORM_INFOdirectly in KVM is more reasonable. And set these msr bits asvcpu->arch.msr_platform_infodefault value.(quoted diff) +0
-2-bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) -bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)I don't get why QEMU cannot do this with the existing interface, e.g.,
KVM_SET_MSRS.will the MSR value be migrated during VM migration?
Quoted 8 lines
looks we are in a dilemma. on one side, if the value is migrated, the value can become inconsisntent with hardware value. On the other side, changing the ratio bits at runtime isn't the architectural behavior.
And the MSR is per-socket. In theory, a system can have two sockets with different values of the MSR. what if a vCPU is created on a socket and then later runs on the other socket?
Set these msr bits (needed by turbostat on intel platform) in KVM by default. Of cource, QEMU can also set MSR value by need. It does not conflict.
Quoted 14 lines
Quoted 14 lines · Chao Gao
Quoted 12 lines · Hao Xiang
Quoted 5 lines
For intel platform, The BzyMhz field of Turbostat shows zero due to the missing of part msr bits of
MSR_PLATFORM_INFO.Acquire necessary msr bits, and expose following msr info to guest, to make sure guest can get correct turbo frequency info.
Userspace VMM (e.g., QEMU) can configure this MSR for guests. Please refer to
tools/testing/selftests/kvm/x86_64/platform_info_test.c.The question is why KVM needs this patch given KVM already provides interfaces for QEMU to configure the MSR.
Sean Christopherson
maintainer+Aaron
When resending a patch, e.g. to change To: or Cc:, tag it RESEND. I got three copies of this...
Quoted 24 lines · Hao Xiang
Quoted 21 lines · Chao Gao
Review
Quoted 10 lines · Hao Xiang
For reason that,
The turbo frequency info depends on specific machine type. And the msr value of
MSR_PLATFORM_INFOmay be diferent on diffrent generation machine.Get following msr bits (needed by turbostat on intel platform) by rdmsr
MSR_PLATFORM_INFOdirectly in KVM is more reasonable. And set these msr bits asvcpu->arch.msr_platform_infodefault value.(quoted diff) +0
-2-bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) -bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)I don't get why QEMU cannot do this with the existing interface, e.g.,
KVM_SET_MSRS.will the MSR value be migrated during VM migration?
looks we are in a dilemma. on one side, if the value is migrated, the value can become inconsisntent with hardware value. On the other side, changing the ratio bits at runtime isn't the architectural behavior.
Quoted 8 lines
Quoted 4 lines
And the MSR is per-socket. In theory, a system can have two sockets with different values of the MSR. what if a vCPU is created on a socket and then later runs on the other socket?
Set these msr bits (needed by turbostat on intel platform) in KVM by default.
Of cource, QEMU can also set MSR value by need. It does not conflict.
It doesn't conflict per se, but it's still problematic. By stuffing a default value, KVM _forces_ userspace to override the MSR to align with the topology and CPUID defined by userspace. And if userspace uses KVM's "default" CPUID, or lack thereof, using the underlying values from hardware are all but guaranteed to be wrong.
The existing code that sets MSR_PLATFORM_INFO_CPUID_FAULT really should not exist, i.e. KVM shouldn't shouldn't assume userspace wants to expose CPUID faulting to the guest. That particular one probably isn't worth trying to retroactively fix.
Ditto for setting MSR_IA32_ARCH_CAPABILITIES; KVM is overstepping, but doing so likely doesn't cause problems.
MSR_IA32_PERF_CAPABILITIES is a different story. Setting a non-zero default value is blatantly wrong, as KVM will advertise vPMU features even if userspace doesn't advertise. Aaron is planning on sending a patch for this one (I'm hoping we can get away with retroactively dropping the code without having to add a quirk).
*If* we need KVM to expose the ratios to userspace, then the correct way to do so is handle turbo and efficiency ratio information is to by implementing support in kvm_get_msr_feature(), i.e. KVM_GET_MSRS on /dev/kvm. Emphasis on "if", because I would prefer to do nothing in KVM if that information is already surfaced to userspace through other mechanisms in the kernel.
Quoted 35 lines · Sean Christopherson
+Aaron
When resending a patch, e.g. to change To: or Cc:, tag it RESEND. I got three copies of this...
Quoted 29 lines · Hao Xiang
Quoted 25 lines · Chao Gao
Review
Quoted 10 lines · Hao Xiang
For reason that,
The turbo frequency info depends on specific machine type. And the msr value of
MSR_PLATFORM_INFOmay be diferent on diffrent generation machine.Get following msr bits (needed by turbostat on intel platform) by rdmsr
MSR_PLATFORM_INFOdirectly in KVM is more reasonable. And set these msr bits asvcpu->arch.msr_platform_infodefault value.(quoted diff) +0
-2-bit 15:8, Maximum Non-Turbo Ratio (MAX_NON_TURBO_LIM_RATIO) -bit 47:40, Maximum Efficiency Ratio (MAX_EFFICIENCY_RATIO)I don't get why QEMU cannot do this with the existing interface, e.g.,
KVM_SET_MSRS.will the MSR value be migrated during VM migration?
looks we are in a dilemma. on one side, if the value is migrated, the value can become inconsisntent with hardware value. On the other side, changing the ratio bits at runtime isn't the architectural behavior.
And the MSR is per-socket. In theory, a system can have two sockets with different values of the MSR. what if a vCPU is created on a socket and then later runs on the other socket?
Quoted 7 lines
Quoted 3 lines
Set these msr bits (needed by turbostat on intel platform) in KVM by default.
Of cource, QEMU can also set MSR value by need. It does not conflict.It doesn't conflict per se, but it's still problematic. By stuffing a default value, KVM _forces_ userspace to override the MSR to align with the topology and CPUID defined by userspace.
I don't understand how this MSR is related to topology and CPUID?
Quoted 3 lines
And if userspace uses KVM's "default" CPUID, or lack
thereof, using the underlying values from hardware are all but guaranteed to be wrong.
Could you please elaborate?
Quoted 17 lines
The existing code that sets
MSR_PLATFORM_INFO_CPUID_FAULTreally should not exist, i.e. KVM shouldn't shouldn't assume userspace wants to expose CPUID faulting to the guest. That particular one probably isn't worth trying to retroactively fix.Ditto for setting
MSR_IA32_ARCH_CAPABILITIES; KVM is overstepping, but doing so likely doesn't cause problems.
MSR_IA32_PERF_CAPABILITIESis a different story. Setting a non-zero default value is blatantly wrong, as KVM will advertise vPMU features even if userspace doesn't advertise. Aaron is planning on sending a patch for this one (I'm hoping we can get away with retroactively dropping the code without having to add a quirk).*If* we need KVM to expose the ratios to userspace, then the correct way to do so is handle turbo and efficiency ratio information is to by implementing support in
kvm_get_msr_feature(), i.e.KVM_GET_MSRSon/dev/kvm. Emphasis on "if", because I would prefer to do nothing in KVM if that information is already surfaced to userspace through other mechanisms in the kernel.
Sean Christopherson
maintainerQuoted 10 lines · Xiaoyao Li
Quoted 7 lines · Sean Christopherson
Quoted 3 lines
Set these msr bits (needed by turbostat on intel platform) in KVM by default. Of cource, QEMU can also set MSR value by need. It does not conflict.
It doesn't conflict per se, but it's still problematic. By stuffing a default value, KVM _forces_ userspace to override the MSR to align with the topology and CPUID defined by userspace.
I don't understand how this MSR is related to topology and CPUID?
Heh, looked at the SDM to double check myself, and the first hit when searching for MSR_PLATFORM_INFO says:
When TSC scaling is enabled for a guest using Intel PT, the VMM should ensure that the value of Maximum Non-Turbo Ratio[15:8] in MSR_PLATFORM_INFO (MSR 0CEH) and the TSC/”core crystal clock” ratio (EBX/EAX) in CPUID leaf 15H are set in a manner consistent with the resulting TSC rate that will be visible to the VM.
As Chao pointed out, the MSR is technically per package, so a weird setup could have sockets with different frequencies, or enumerate a virtual topology to the guest with such a configuration. I doubt/hope no one actually does something like that, but it's theoretically possible, and one of the many reasons why KVM needs to stay out of the way and let userspace define the vCPU model.
Quoted 4 lines
Quoted 2 lines
And if userspace uses KVM's "default" CPUID, or lack thereof, using the underlying values from hardware are all but guaranteed to be wrong.
Could you please elaborate?
I guess an empty CPUID would probably be ok? If there's no CPUID.0x15, it can't be wrong. It's largely a moot point though, I highly doubt anyone runs a "real" VM without populating _something_ in guest CPUID.
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Sean Christopherson
Quoted 3 lines
Set these msr bits (needed by turbostat on intel platform) in KVM by default. Of cource, QEMU can also set MSR value by need. It does not conflict.
It doesn't conflict per se, but it's still problematic. By stuffing a default value, KVM _forces_ userspace to override the MSR to align with the topology and CPUID defined by userspace.
I don't understand how this MSR is related to topology and CPUID?
Quoted 7 lines
Heh, looked at the SDM to double check myself, and the first hit when searching for
MSR_PLATFORM_INFOsays:When TSC scaling is enabled for a guest using Intel PT, the VMM should ensure that the value of Maximum Non-Turbo Ratio[15:8] in
MSR_PLATFORM_INFO(MSR 0CEH) and the TSC/”core crystal clock” ratio (EBX/EAX) in CPUID leaf 15H are set in a manner consistent with the resulting TSC rate that will be visible to the VM.
I see.
Quoted 3 lines
As Chao pointed out, the MSR is technically per package, so a weird setup could have sockets with different frequencies, or enumerate a virtual topology to the guest with such a configuration.
Every feature might get into trouble if not consistent across packages, no matter per-thread/per-core/per-package.
Quoted 3 lines
I doubt/hope no one actually does something
like that, but it's theoretically possible, and one of the many reasons why KVM needs to stay out of the way and let userspace define the vCPU model.
For this specific case, the max non-turbo frequency needs to be consistent with TSC frequency. Because KVM has default TSC frequency as host's tsc_khz, for correctness, it should have a default value to match with KVM's default TSC when userspace provide no explicit configuration.
But it's not the problem this patch targets. I'm OK to keep returning 0 as-is until some bug reported due to the inconsistent between max non-turbo frequency and TSC frequency.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
And if userspace uses KVM's "default" CPUID, or lack thereof, using the underlying values from hardware are all but guaranteed to be wrong.
Could you please elaborate?
I guess an empty CPUID would probably be ok? If there's no CPUID.0x15, it can't be wrong. It's largely a moot point though, I highly doubt anyone runs a "real" VM without populating _something_ in guest CPUID.
current QEMU doesn't configure CPUID leaf 0x15, nor does it configure MSR_PLATFORM_INFO[15:8]. I need to take time to dig how Linux gets the TSC frequency.