KVM: x86: disable halt polling when powersave governor is used
PATCH 1/1 KVM: x86: disable halt polling when powersave governor is used
Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.
disabling halt polling in powersave governor can save the precious power in power critical case.
FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).
On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf UDP_RR case runs from two VMs locate on two different physical machines.
FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%
Netperf Base Disable-halt-polling Delta%UDP_RR 509.8 508.5 -0.3%
--- arch/x86/kvm/x86.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +16 -1
@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)bool kvm_arch_no_poll (struct kvm_vcpu *vcpu) { -return (vcpu->arch.msr_kvm_poll_control &1 ) ==0 ; +struct cpufreq_policy *policy =cpufreq_cpu_get (vcpu->cpu); +bool powersave = false; + +/* +* Halt polling could consume much CPU power, if CPU frequency +* governor is set to "powersave", disable halt polling. +*/ +if (policy) { +if ((policy->policy ==CPUFREQ_POLICY_POWERSAVE ) || + (policy->governor && + !strncmp (policy->governor->name,"powersave" , +CPUFREQ_NAME_LEN ))) + powersave = true; +cpufreq_cpu_put (policy); + } +return ((vcpu->arch.msr_kvm_poll_control &1 ) ==0 ) || powersave; }EXPORT_SYMBOL_GPL (kvm_arch_no_poll);
Signature · 1 line
2.34.1
Sean Christopherson
maintainerQuoted 34 lines · Dapeng Mi
Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.
disabling halt polling in powersave governor can save the precious power in power critical case.
FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).
On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf
UDP_RRcase runs from two VMs locate on two different physical machines.FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%Netperf Base Disable-halt-polling Delta%
UDP_RR509.8 508.5 -0.3%Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/kvm/x86.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13018
Quoted 5 lines
arch/x86/kvm/x86.c +1
-1@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)bool kvm_arch_no_poll (struct kvm_vcpu *vcpu) { -return (vcpu->arch.msr_kvm_poll_control &1 ) ==0 ; +struct cpufreq_policy *policy =cpufreq_cpu_get (vcpu->cpu);
Preemption is not disabled at this point, which means that using vcpu->cpu is potentially unsafe. Given that cpufreq is refcounting the returned object, I gotta imaging get migrated to a different pCPU would be problematic.
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13019
Quoted 1 line
arch/x86/kvm/x86.c +1
-0+bool powersave = false;
I don't see anything in here that's x86 specific. Unless I'm missing something, this belongs in common KVM.
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13027
Quoted 8 lines
arch/x86/kvm/x86.c +8
-0+ +/* +* Halt polling could consume much CPU power, if CPU frequency +* governor is set to "powersave", disable halt polling. +*/ +if (policy) { + if ((policy->policy == CPUFREQ_POLICY_POWERSAVE) || + (policy->governor &&
Indentation is messed up.
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13028
Quoted 1 line
arch/x86/kvm/x86.c +1
-0+ !strncmp(policy->governor->name, "powersave",
KVM should not be comparing magic strings. If the cpufreq subsystem can't get policy->policy right, then that needs to be fixed.
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13033
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+ CPUFREQ_NAME_LEN))) + powersave = true; +cpufreq_cpu_put (policy); + } +return ((vcpu->arch.msr_kvm_poll_control &1 ) ==0 ) || powersave;
Doing all of the above work if polling is disabled is silly.
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13035
Quoted 2 lines
arch/x86/kvm/x86.c +0
-0}EXPORT_SYMBOL_GPL (kvm_arch_no_poll);
All in all, _if_ we want to do this automatically and not let userspace decide how to manage powersave vs. halt-poll, I think this should be more like:
--- virt/kvm/kvm_main.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c +20 -1
@@ -29,6 +29,7 @@#include <linux/file.h> #include <linux/syscore_ops.h> #include <linux/cpu.h> +#include <linux/cpufreq.h> #include <linux/sched/signal.h> #include <linux/sched/mm.h> #include <linux/sched/stat.h>
@@ -3483,6 +3484,23 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, ktime_t start, } } +static bool kvm_cpufreq_no_halt_poll (struct kvm_vcpu *vcpu) +{ +struct cpufreq_policy *policy; +bool powersave = false; + +preempt_disable (); + + policy =cpufreq_cpu_get (vcpu->cpu); +if (policy) { + powersave = (policy->policy ==CPUFREQ_POLICY_POWERSAVE ); +cpufreq_cpu_put (policy); + } + +preempt_enable (); +return powersave; +} +/* * Emulate a vCPU halt condition, e.g. HLT on x86, WFI on arm, etc... If halt * polling is enabled, busy wait for a short time before blocking to avoid the
@@ -3491,7 +3509,8 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, ktime_t start,*/ void kvm_vcpu_halt (struct kvm_vcpu *vcpu) { -bool halt_poll_allowed = !kvm_arch_no_poll (vcpu); +const bool halt_poll_allowed = !kvm_arch_no_poll (vcpu) && + !kvm_cpufreq_no_halt_poll (vcpu);bool do_halt_poll = halt_poll_allowed && vcpu->halt_poll_ns;ktime_t start, cur, poll_end;bool waited = false;
base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354
Quoted 45 lines
From: Sean Christopherson <[email protected]>
Sent: Saturday, October 8, 2022 1:52 AM
To: Mi, Dapeng1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH] KVM: x86: disable halt polling when powersave governor is usedQuoted 35 lines · Dapeng Mi
Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.
disabling halt polling in powersave governor can save the precious power in power critical case.
FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).
On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf
UDP_RRcase runs from two VMs locate on two different physical machines.FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%Netperf Base Disable-halt-polling Delta%
UDP_RR509.8 508.5 -0.3%Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/kvm/x86.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/x86/kvm/x86.c index renamed from arch/x86/kvm/x86.c +0
-0
d7374d768296..c0eb6574cbbb100644arch/x86/kvm/x86.c +0
-0@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)
Quoted 7 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13017
Quoted 3 lines
arch/x86/kvm/x86.c +1
-1bool kvm_arch_no_poll (struct kvm_vcpu *vcpu) { -return (vcpu->arch.msr_kvm_poll_control &1 ) ==0 ; +struct cpufreq_policy *policy =cpufreq_cpu_get (vcpu->cpu);Preemption is not disabled at this point, which means that using
vcpu->cpuis potentially unsafe. Given that cpufreq is refcounting the returned object, I gotta imaging get migrated to a different pCPU would be problematic.
Thanks for pointing this. Per my learning, even vCPU migrates to a different pCPU in the progress of getting the cpufreq policy, the only consequence is to get an outdated (maybe inaccurate) policy value. Halt polling mechanism still can get the updated and correct cpufreq policy in next time. And even we disable preemption in the process of obtaining cpufreq policy, the vCPU is still possible to be migrated a different pCPU after enabling preemption and before calling the halt polling judging logic.
Quoted 6 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13018
Quoted 2 lines
arch/x86/kvm/x86.c +1
-0+bool powersave = false;I don't see anything in here that's x86 specific. Unless I'm missing something, this belongs in common KVM.
Yes, this is generic.
Quoted 10 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13026
Quoted 8 lines
arch/x86/kvm/x86.c +8
-0+ +/* +* Halt polling could consume much CPU power, if CPU frequency +* governor is set to "powersave", disable halt polling. +*/ +if (policy) { + if ((policy->policy == CPUFREQ_POLICY_POWERSAVE) || + (policy->governor &&Indentation is messed up.
Sure. Would change.
Quoted 6 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13027
Quoted 2 lines
arch/x86/kvm/x86.c +1
-0+ !strncmp(policy->governor->name,"powersave",
KVM should not be comparing magic strings. If the cpufreq subsystem can't get
policy->policyright, then that needs to be fixed.
Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like intel_pstate driver. For this kind of driver, the cpufreq governor is saved in the policy->policy field. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type is saved in the governor->name field. For the second kind of cpufreq driver, the policy->policy field is meaningless and we have to read the governor name.
Quoted 8 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13032
Quoted 6 lines
arch/x86/kvm/x86.c +5
-0+ CPUFREQ_NAME_LEN))) + powersave = true; +cpufreq_cpu_put (policy); + } +return ((vcpu->arch.msr_kvm_poll_control &1 ) ==0 ) || powersave;Doing all of the above work if polling is disabled is silly.
Correct. Would change.
Quoted 6 lines
arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13034
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0}EXPORT_SYMBOL_GPL (kvm_arch_no_poll);All in all, _if_ we want to do this automatically and not let userspace decide how to manage powersave vs. halt-poll, I think this should be more like:
Thanks for your sample. Would change in V2 patch.
Quoted 54 lines
--- virt/kvm/kvm_main.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)virt/kvm/kvm_main.c index renamed from virt/kvm/kvm_main.c +0
-0
e30f1b4ecfa5..01116859cb31100644virt/kvm/kvm_main.c +19
-1@@ -29,6 +29,7 @@#include <linux/file.h> #include <linux/syscore_ops.h> #include <linux/cpu.h> +#include <linux/cpufreq.h> #include <linux/sched/signal.h> #include <linux/sched/mm.h> #include <linux/sched/stat.h> @@ -3483,6 +3484,23 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, ktime_t start, } } +static bool kvm_cpufreq_no_halt_poll (struct kvm_vcpu *vcpu) { +struct cpufreq_policy *policy; +bool powersave = false; + +preempt_disable (); + + policy =cpufreq_cpu_get (vcpu->cpu); +if (policy) { + powersave = (policy->policy ==CPUFREQ_POLICY_POWERSAVE ); +cpufreq_cpu_put (policy); + } + +preempt_enable (); +return powersave; +} +/* * Emulate a vCPU halt condition, e.g. HLT on x86, WFI on arm, etc... If halt * polling is enabled, busy wait for a short time before blocking to avoid the @@ -3491,7 +3509,8 @@ static inline void update_halt_poll_stats(structkvm_vcpu *vcpu, ktime_t start, */ void kvm_vcpu_halt (struct kvm_vcpu *vcpu) { -bool halt_poll_allowed = !kvm_arch_no_poll (vcpu); +const bool halt_poll_allowed = !kvm_arch_no_poll (vcpu) && + !kvm_cpufreq_no_halt_poll (vcpu);bool do_halt_poll = halt_poll_allowed && vcpu->halt_poll_ns;ktime_t start, cur, poll_end;bool waited = false;base-commit:
e18d6152ff0f41b7f01f9817372022df04e0d354--
Sean Christopherson
maintainerQuoted 6 lines · On Sat, Oct 08, 2022, Mi, Dapeng1 wrote:
Quoted 5 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ !strncmp(policy->governor->name,"powersave",
KVM should not be comparing magic strings. If the cpufreq subsystem can't get
policy->policyright, then that needs to be fixed.
Quoted 8 lines
Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like
intel_pstatedriver. For this kind of driver, the cpufreq governor is saved in thepolicy->policyfield. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type is saved in thegovernor->namefield. For the second kind of cpufreq driver, thepolicy->policyfield is meaningless and we have to read the governor name.
That doesn't mean it's ok to bleed those internal details into KVM. I would much rather cpufreq provide a helper to get the effective policy, e.g.
unsigned int cpufreq_cpu_get_policy(unsigned int cpu)
{
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
unsigned int pol;
if (!policy)
return CPUFREQ_POLICY_UNKNOWN;
pol = policy->policy
if (pol == CPUFREQ_POLICY_UNKNOWN && policy->governor)
pol = cpufreq_parse_policy(policy->governor->name);
cpufreq_cpu_put(policy);}
Quoted 44 lines
From: Sean Christopherson <[email protected]>
Sent: Wednesday, October 26, 2022 1:07 AM
To: Mi, Dapeng1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH] KVM: x86: disable halt polling when powersave governor is usedQuoted 10 lines · On Sat, Oct 08, 2022, Mi, Dapeng1 wrote:
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ !strncmp(policy->governor->name,"powersave",
KVM should not be comparing magic strings. If the cpufreq subsystem can't get
policy->policyright, then that needs to be fixed.Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like
intel_pstatedriver.
Quoted 3 lines
For this kind of driver, the cpufreq governor is saved in the
policy->policyfield. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type issaved in the
governor->namefield.Quoted 2 lines
For the second kind of cpufreq driver, the
policy->policyfield is meaningless and we have to read the governor name.That doesn't mean it's ok to bleed those internal details into KVM. I would much rather cpufreq provide a helper to get the effective policy, e.g.
unsigned int
cpufreq_cpu_get_policy(unsigned int cpu){ struct cpufreq_policy *policy = cpufreq_cpu_get(cpu); unsigned int pol; if (!policy) return CPUFREQ_POLICY_UNKNOWN; pol = policy->policy if (pol == CPUFREQ_POLICY_UNKNOWN && policy->governor) pol = cpufreq_parse_policy(policy->governor->name); cpufreq_cpu_put(policy);
Quoted 1 line
}
Thanks Sean for reviewing. Would do in next version.