KVM: x86: Give host userspace control for MSR_RAPL_POWER_UNIT and MSR_PKG_POWER_STATUS
RFC 1/1 KVM: x86: Give host userspace control for MSR_RAPL_POWER_UNIT and MSR_PKG_POWER_STATUS
Allow userspace to update the MSR_RAPL_POWER_UNIT and MSR_PKG_POWER_STATUS powercap registers. By default, these MSRs still return 0.
This enables VMMs running on top of KVM with access to energy metrics like /sys/devices/virtual/powercap/*/*/energy_uj to compute VMs power values in proportion with other metrics (e.g. CPU %guest, steal time, etc.) and periodically update the MSRs with ioctl KVM_SET_MSRS so that the guest OS can consume them using power metering tools.
---
Notes:
The main goal of this patch is to bring a first step to give energy awareness to VMs.
As of today, KVM always report 0 in these MSRs since the entire host power consumption needs to be hidden from the guests. However, there is no fallback mechanism for VMs to measure their power usage.
The idea is to let the VMMs running on top of KVM periodically update those MSRs with representative values of the VM's power consumption.
If this solution is accepted, VMMs like QEMU will need to be patched to set proper values in these registers and enable power metering in guests.
I am submitting this as an RFC to get input/feedback from a broader audience who may be aware of potential side effects of such a mechanism.
Regards,
Anthony
"If you can’t measure it, you can’t improve it." – Lord Kelvin
arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/x86.c | 18 ++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -1006,6 +1006,10 @@ struct kvm_vcpu_arch {*/ bool pdptrs_from_userspace; +/* Powercap related MSRs */ +u64 msr_rapl_power_unit; +u64 msr_pkg_energy_status; +#if IS_ENABLED (CONFIG_HYPERV )hpa_t hv_root_tdp;#endif
arch/x86/kvm/x86.c +16 -2
@@ -1528,6 +1528,10 @@ static const u32 emulated_msrs_all[] = {MSR_K7_HWCR ,MSR_KVM_POLL_CONTROL , + +/* The following MSRs can be updated by the userspace */ +MSR_RAPL_POWER_UNIT , + MSR_PKG_ENERGY_STATUS, };static u32 emulated_msrs[ARRAY_SIZE (emulated_msrs_all)];
@@ -3888,6 +3892,12 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* as to-be-saved, even if an MSRs isn't fully supported. */ return !msr_info->host_initiated || data; +case MSR_RAPL_POWER_UNIT : + vcpu->arch.msr_rapl_power_unit = data; +break ; +case MSR_PKG_ENERGY_STATUS : + vcpu->arch.msr_pkg_energy_status = data; +break ;default :if (kvm_pmu_is_valid_msr (vcpu, msr))return kvm_pmu_set_msr (vcpu, msr_info);
@@ -3973,13 +3983,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* data here. Do not conditionalize this on CPUID, as KVM does not do * so for existing CPU-specific MSRs. */ -case MSR_RAPL_POWER_UNIT :case MSR_PP0_ENERGY_STATUS :/* Power plane 0 (core) */ case MSR_PP1_ENERGY_STATUS :/* Power plane 1 (graphics uncore) */ -case MSR_PKG_ENERGY_STATUS :/* Total package */ case MSR_DRAM_ENERGY_STATUS :/* DRAM controller */ msr_info->data =0 ;break ; +case MSR_RAPL_POWER_UNIT : + msr_info->data = vcpu->arch.msr_rapl_power_unit; +break ; +case MSR_PKG_ENERGY_STATUS :/* Total package */ + msr_info->data = vcpu->arch.msr_pkg_energy_status; +break ;case MSR_IA32_PEBS_ENABLE :case MSR_IA32_DS_AREA :case MSR_PEBS_DATA_CFG :
Signature · 1 line
2.39.0
Quoted 26 lines · Anthony Harivel
Allow userspace to update the
MSR_RAPL_POWER_UNITandMSR_PKG_POWER_STATUSpowercap registers. By default, these MSRs still return 0.This enables VMMs running on top of KVM with access to energy metrics like
/sys/devices/virtual/powercap/*/*/energy_ujto compute VMs power values in proportion with other metrics (e.g. CPU %guest, steal time, etc.) and periodically update the MSRs with ioctlKVM_SET_MSRSso that the guest OS can consume them using power metering tools.Cc Paolo Bonzini <[email protected]>Cc Christophe Fontaine <[email protected]>Signed-off-by Anthony Harivel <[email protected]>---
Notes:
The main goal of this patch is to bring a first step to give energy awareness to VMs.As of today, KVM always report 0 in these MSRs since the entire host power consumption needs to be hidden from the guests. However, there is no fallback mechanism for VMs to measure their power usage.
The idea is to let the VMMs running on top of KVM periodically update those MSRs with representative values of the VM's power consumption.
Quoted 6 lines
If this solution is accepted, VMMs like QEMU will need to be patched to set proper values in these registers and enable power metering in guests.
I am submitting this as an RFC to get input/feedback from a broader audience who may be aware of potential side effects of such a mechanism.
Set aside how user space VMM emulate these 2 MSRs correctly, it can request the MSR READ to exit to user space via KVM_X86_SET_MSR_FILTER. So user space VMM can just enable the read filter of these 2 MSRs and provide the emulation itself.
Quoted 4 lines · Xiaoyao Li
Set aside how user space VMM emulate these 2 MSRs correctly, it can request the MSR READ to exit to user space via
KVM_X86_SET_MSR_FILTER. So user space VMM can just enable the read filter of these 2 MSRs and provide the emulation itself.
Thanks for your feedback. I totally miss out on this possibility. On Qemu, only one msr is using it (MSR_CORE_THREAD_COUNT). I did a test yesterday in Qemu adding a filter with kvm_filter_msr() and a callback to setup the msr with a dummy value and it's perfectly working.
Paolo Bonzini
maintainerQuoted 10 lines · Anthony Harivel
Allow userspace to update the
MSR_RAPL_POWER_UNITandMSR_PKG_POWER_STATUSpowercap registers. By default, these MSRs still return 0.This enables VMMs running on top of KVM with access to energy metrics like
/sys/devices/virtual/powercap/*/*/energy_ujto compute VMs power values in proportion with other metrics (e.g. CPU %guest, steal time, etc.) and periodically update the MSRs with ioctlKVM_SET_MSRSso that the guest OS can consume them using power metering tools.
Do you have a QEMU prototype patch? My gut feeling is that these MSRs should be handled entirely within KVM using the sched_in and sched_out notifiers. This would also allow exposing the values to the host using the statistics subsystem.
Paolo
Quoted 98 lines
Cc Paolo Bonzini <[email protected]>Cc Christophe Fontaine <[email protected]>Signed-off-by Anthony Harivel <[email protected]>---
Notes:
The main goal of this patch is to bring a first step to give energy awareness to VMs.As of today, KVM always report 0 in these MSRs since the entire host power consumption needs to be hidden from the guests. However, there is no fallback mechanism for VMs to measure their power usage.
The idea is to let the VMMs running on top of KVM periodically update those MSRs with representative values of the VM's power consumption.
If this solution is accepted, VMMs like QEMU will need to be patched to set proper values in these registers and enable power metering in guests.
I am submitting this as an RFC to get input/feedback from a broader audience who may be aware of potential side effects of such a mechanism.
Regards,
Anthony"If you can’t measure it, you can’t improve it." – Lord Kelvin
arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/x86.c | 18 ++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +4
-0@@ -1006,6 +1006,10 @@ struct kvm_vcpu_arch {*/ bool pdptrs_from_userspace; +/* Powercap related MSRs */ +u64 msr_rapl_power_unit; +u64 msr_pkg_energy_status; +#if IS_ENABLED (CONFIG_HYPERV )hpa_t hv_root_tdp;#endif arch/x86/kvm/x86.c +16
-3@@ -1528,6 +1528,10 @@ static const u32 emulated_msrs_all[] = {MSR_K7_HWCR ,MSR_KVM_POLL_CONTROL , + +/* The following MSRs can be updated by the userspace */ +MSR_RAPL_POWER_UNIT , + MSR_PKG_ENERGY_STATUS, };static u32 emulated_msrs[ARRAY_SIZE (emulated_msrs_all)];@@ -3888,6 +3892,12 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* as to-be-saved, even if an MSRs isn't fully supported. */ return !msr_info->host_initiated || data; +case MSR_RAPL_POWER_UNIT : + vcpu->arch.msr_rapl_power_unit = data; +break ; +case MSR_PKG_ENERGY_STATUS : + vcpu->arch.msr_pkg_energy_status = data; +break ;default :if (kvm_pmu_is_valid_msr (vcpu, msr))return kvm_pmu_set_msr (vcpu, msr_info);@@ -3973,13 +3983,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* data here. Do not conditionalize this on CPUID, as KVM does not do * so for existing CPU-specific MSRs. */ -case MSR_RAPL_POWER_UNIT :case MSR_PP0_ENERGY_STATUS :/* Power plane 0 (core) */ case MSR_PP1_ENERGY_STATUS :/* Power plane 1 (graphics uncore) */ -case MSR_PKG_ENERGY_STATUS :/* Total package */ case MSR_DRAM_ENERGY_STATUS :/* DRAM controller */ msr_info->data =0 ;break ; +case MSR_RAPL_POWER_UNIT : + msr_info->data = vcpu->arch.msr_rapl_power_unit; +break ; +case MSR_PKG_ENERGY_STATUS :/* Total package */ + msr_info->data = vcpu->arch.msr_pkg_energy_status; +break ;case MSR_IA32_PEBS_ENABLE :case MSR_IA32_DS_AREA :case MSR_PEBS_DATA_CFG : --2.39.0
After a second though and further tests with ftrace backend activated, if we update those MSRs in VMM (Qemu in my context) with
a KVM_X86_SET_MSR_FILTER, each time we will read the value we will go through the following KVM exit:
kvm_userspace_exit: reason KVM_EXIT_X86_RDMSR (29)
Because some various MSRs (i.e MSR_{PKG,PP0,PP1}_ENERGY_STATUS) being counters (of uJoules), their values must be update regularly to make sens for the Power tools.
So I'm wondering if the contexts switching (KVM->userpace->KVM) to update all MSRs will cause performance issues?
Quoted 7 lines
Do you have a QEMU prototype patch? My gut feeling is that these MSRs should be handled entirely within KVM using the
sched_inandsched_outnotifiers. This would also allow exposing the values to the host using the statistics subsystem.Paolo
I did some Qemu hack with Qtimer to update the values regularly but I'm definitely not satisfied with the implementation.
What I'm pretty sure is that updating the values should be done separately from the callback that consume the value. This would ensure the consistency of the values.
In the hypothesis those MSRs are handled within KVM, we can read MSRs with rdmsrl_safe() but how can we get the percentage of CPU used by Qemu to get a proportional value of the counter?
Regards,
Anthony
Quoted 98 lines
Quoted 98 lines
Cc Paolo Bonzini <[email protected]>Cc Christophe Fontaine <[email protected]>Signed-off-by Anthony Harivel <[email protected]>---
Notes:
The main goal of this patch is to bring a first step to give energy awareness to VMs.As of today, KVM always report 0 in these MSRs since the entire host power consumption needs to be hidden from the guests. However, there is no fallback mechanism for VMs to measure their power usage.
The idea is to let the VMMs running on top of KVM periodically update those MSRs with representative values of the VM's power consumption.
If this solution is accepted, VMMs like QEMU will need to be patched to set proper values in these registers and enable power metering in guests.
I am submitting this as an RFC to get input/feedback from a broader audience who may be aware of potential side effects of such a mechanism.
Regards,
Anthony"If you can’t measure it, you can’t improve it." – Lord Kelvin
arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/x86.c | 18 ++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +4
-0@@ -1006,6 +1006,10 @@ struct kvm_vcpu_arch {*/ bool pdptrs_from_userspace; +/* Powercap related MSRs */ +u64 msr_rapl_power_unit; +u64 msr_pkg_energy_status; +#if IS_ENABLED (CONFIG_HYPERV )hpa_t hv_root_tdp;#endif arch/x86/kvm/x86.c +16
-3@@ -1528,6 +1528,10 @@ static const u32 emulated_msrs_all[] = {MSR_K7_HWCR ,MSR_KVM_POLL_CONTROL , + +/* The following MSRs can be updated by the userspace */ +MSR_RAPL_POWER_UNIT , + MSR_PKG_ENERGY_STATUS, };static u32 emulated_msrs[ARRAY_SIZE (emulated_msrs_all)];@@ -3888,6 +3892,12 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* as to-be-saved, even if an MSRs isn't fully supported. */ return !msr_info->host_initiated || data; +case MSR_RAPL_POWER_UNIT : + vcpu->arch.msr_rapl_power_unit = data; +break ; +case MSR_PKG_ENERGY_STATUS : + vcpu->arch.msr_pkg_energy_status = data; +break ;default :if (kvm_pmu_is_valid_msr (vcpu, msr))return kvm_pmu_set_msr (vcpu, msr_info);@@ -3973,13 +3983,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* data here. Do not conditionalize this on CPUID, as KVM does not do * so for existing CPU-specific MSRs. */ -case MSR_RAPL_POWER_UNIT :case MSR_PP0_ENERGY_STATUS :/* Power plane 0 (core) */ case MSR_PP1_ENERGY_STATUS :/* Power plane 1 (graphics uncore) */ -case MSR_PKG_ENERGY_STATUS :/* Total package */ case MSR_DRAM_ENERGY_STATUS :/* DRAM controller */ msr_info->data =0 ;break ; +case MSR_RAPL_POWER_UNIT : + msr_info->data = vcpu->arch.msr_rapl_power_unit; +break ; +case MSR_PKG_ENERGY_STATUS :/* Total package */ + msr_info->data = vcpu->arch.msr_pkg_energy_status; +break ;case MSR_IA32_PEBS_ENABLE :case MSR_IA32_DS_AREA :case MSR_PEBS_DATA_CFG : --2.39.0
Paolo Bonzini
maintainerQuoted 2 lines · Anthony Harivel
So I'm wondering if the contexts switching (
KVM->userpace->KVM) to update all MSRs will cause performance issues?
How often do you anticipate them to be read by the guest?
Quoted 7 lines
What I'm pretty sure is that updating the values should be done separately from the callback that consume the value. This would ensure the consistency of the values.
In the hypothesis those MSRs are handled within KVM, we can read MSRs with
rdmsrl_safe()but how can we get the percentage of CPU used by Qemu to get a proportional value of the counter?
If you are okay with only counting the time spent in the guest, i.e. not in QEMU, you can snapshot the energy value when the vCPU starts (kvm_arch_vcpu_load, kvm_sched_in) and update it when the vCPU stops (kvm_sched_out, kvm_get_msr and post_kvm_run_save).
Paolo
Quoted 6 lines · Paolo Bonzini
Quoted 2 lines · Anthony Harivel
So I'm wondering if the contexts switching (
KVM->userpace->KVM) to update all MSRs will cause performance issues?How often do you anticipate them to be read by the guest?
The more often we can update the register, the more accurate we can be. Not accurate in terms of consumption but in terms of time. Imagine a burst of data that needs to be processed by a vCPU every 100ms, if we update the registers every seconds we will never target which burst is consuming the most. It's a total made up example but it gives an idea. But we have to keep in mind that those energy counters are updated at the average rate of ~1ms.
Quoted 8 lines
Quoted 7 lines
What I'm pretty sure is that updating the values should be done separately from the callback that consume the value. This would ensure the consistency of the values.
In the hypothesis those MSRs are handled within KVM, we can read MSRs with
rdmsrl_safe()but how can we get the percentage of CPU used by Qemu to get a proportional value of the counter?
Quoted 6 lines
If you are okay with only counting the time spent in the guest, i.e. not in QEMU, you can snapshot the energy value when the vCPU starts (
kvm_arch_vcpu_load,kvm_sched_in) and update it when the vCPU stops (kvm_sched_out,kvm_get_msrandpost_kvm_run_save).Paolo
I think this is great idea and would have worked perfectly if we were able to get the individual core power consumption. However we are down to, in the best case, to the power consumption of all Cores per package (PP0/Core power plane). And in the server CPU Family (i.e Xeon), we only have the Package power plane which include consumption of all cores in the package plus the uncore power consumption (i.e. last level of caches and memory controller; consumption of DRAM memory not included). So depending on the workload of all the cores and the uses of the uncore part, the energy counter of the package is increasing more or less quicker between samples of the same period.
Anthony