Skip to contentSkip to conversation

KVM: x86: disable halt polling when powersave governor is used

PATCH In review Dapeng Mi 2 participants 5 messages Sep 15, 2022 – Nov 15, 2022

Subsystems (3)
lore
Share & tools
lore

Dapeng Mi

author
#0 lore raw

PATCH 1/1 KVM: x86: disable halt polling when powersave governor is used

Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.

disabling halt polling in powersave governor can save the precious power in power critical case.

FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).

On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf UDP_RR case runs from two VMs locate on two different physical machines.

FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%

Netperf Base Disable-halt-polling Delta%
UDP_RR 509.8 508.5 -0.3%

---
 arch/x86/kvm/x86.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +16 -1
index d7374d768296..c0eb6574cbbb 100644
@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)
 
 bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
 {
-	return (vcpu->arch.msr_kvm_poll_control & 1) == 0;
+	struct cpufreq_policy *policy = cpufreq_cpu_get(vcpu->cpu);
+	bool powersave = false;
+
+	/*
+	 * Halt polling could consume much CPU power, if CPU frequency
+	 * governor is set to "powersave", disable halt polling.
+	 */
+	if (policy) {
+		if ((policy->policy == CPUFREQ_POLICY_POWERSAVE) ||
+			(policy->governor &&
+				!strncmp(policy->governor->name, "powersave",
+					CPUFREQ_NAME_LEN)))
+			powersave = true;
+		cpufreq_cpu_put(policy);
+	}
+	return ((vcpu->arch.msr_kvm_poll_control & 1) == 0) || powersave;
 }
 EXPORT_SYMBOL_GPL(kvm_arch_no_poll);
 
Signature · 1 line
2.34.1
#1 lore raw
Quoted 34 lines · Dapeng Mi

Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.

disabling halt polling in powersave governor can save the precious power in power critical case.

FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).

On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf UDP_RR case runs from two VMs locate on two different physical machines.

FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%

Netperf Base Disable-halt-polling Delta%
UDP_RR 509.8 508.5 -0.3%

---
 arch/x86/kvm/x86.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +0 -0
index d7374d768296..c0eb6574cbbb 100644

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13018

Quoted 5 lines
arch/x86/kvm/x86.c +1 -1
@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)
 
 bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
 {
-	return (vcpu->arch.msr_kvm_poll_control & 1) == 0;
+	struct cpufreq_policy *policy = cpufreq_cpu_get(vcpu->cpu);

Preemption is not disabled at this point, which means that using vcpu->cpu is potentially unsafe. Given that cpufreq is refcounting the returned object, I gotta imaging get migrated to a different pCPU would be problematic.

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13019

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+	bool powersave = false;

I don't see anything in here that's x86 specific. Unless I'm missing something, this belongs in common KVM.

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13027

Quoted 8 lines
arch/x86/kvm/x86.c +8 -0
+
+	/*
+	 * Halt polling could consume much CPU power, if CPU frequency
+	 * governor is set to "powersave", disable halt polling.
+	 */
+	if (policy) {
+		if ((policy->policy == CPUFREQ_POLICY_POWERSAVE) ||
+			(policy->governor &&

Indentation is messed up.

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13028

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+				!strncmp(policy->governor->name, "powersave",

KVM should not be comparing magic strings. If the cpufreq subsystem can't get policy->policy right, then that needs to be fixed.

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13033

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+					CPUFREQ_NAME_LEN)))
+			powersave = true;
+		cpufreq_cpu_put(policy);
+	}
+	return ((vcpu->arch.msr_kvm_poll_control & 1) == 0) || powersave;

Doing all of the above work if polling is disabled is silly.

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13035

Quoted 2 lines
arch/x86/kvm/x86.c +0 -0
 }
 EXPORT_SYMBOL_GPL(kvm_arch_no_poll);

All in all, _if_ we want to do this automatically and not let userspace decide how to manage powersave vs. halt-poll, I think this should be more like:

---
 virt/kvm/kvm_main.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c +20 -1
index e30f1b4ecfa5..01116859cb31 100644
@@ -29,6 +29,7 @@
 #include <linux/file.h>
 #include <linux/syscore_ops.h>
 #include <linux/cpu.h>
+#include <linux/cpufreq.h>
 #include <linux/sched/signal.h>
 #include <linux/sched/mm.h>
 #include <linux/sched/stat.h>
@@ -3483,6 +3484,23 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, ktime_t start,
 	}
 }
 
+static bool kvm_cpufreq_no_halt_poll(struct kvm_vcpu *vcpu)
+{
+	struct cpufreq_policy *policy;
+	bool powersave = false;
+
+	preempt_disable();
+
+	policy = cpufreq_cpu_get(vcpu->cpu);
+	if (policy) {
+		powersave = (policy->policy == CPUFREQ_POLICY_POWERSAVE);
+		cpufreq_cpu_put(policy);
+	}
+
+	preempt_enable();
+	return powersave;
+}
+
 /*
  * Emulate a vCPU halt condition, e.g. HLT on x86, WFI on arm, etc...  If halt
  * polling is enabled, busy wait for a short time before blocking to avoid the
@@ -3491,7 +3509,8 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, ktime_t start,
  */
 void kvm_vcpu_halt(struct kvm_vcpu *vcpu)
 {
-	bool halt_poll_allowed = !kvm_arch_no_poll(vcpu);
+	const bool halt_poll_allowed = !kvm_arch_no_poll(vcpu) &&
+				       !kvm_cpufreq_no_halt_poll(vcpu);
 	bool do_halt_poll = halt_poll_allowed && vcpu->halt_poll_ns;
 	ktime_t start, cur, poll_end;
 	bool waited = false;

base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354

#2 lore raw
Quoted 45 lines

From: Sean Christopherson <[email protected]>
Sent: Saturday, October 8, 2022 1:52 AM
To: Mi, Dapeng1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH] KVM: x86: disable halt polling when powersave governor is used

Quoted 35 lines · Dapeng Mi

Halt polling is enabled by default even through the CPU frequency governor is configured to powersave. Generally halt polling would consume extra power and this's not identical with the intent of powersave governor.

disabling halt polling in powersave governor can save the precious power in power critical case.

FIO random read test on Alder Lake platform shows halt polling occupies ~17% CPU utilization and consume 7% extra CPU power. After disabling halt polling, CPU has more chance to enter deeper C-states (C1E%: 25.3% -> 33.4%, C10%: 4.4% -> 17.4%).

On Alder Lake platform, we don't find there are obvious performance downgrade after disabling halt polling on FIO and Netperf cases. Netperf UDP_RR case runs from two VMs locate on two different physical machines.

FIO(MB/s) Base Disable-halt-polling Delta%
Rand-read 432.6 436.3 0.8%

Netperf Base Disable-halt-polling Delta%
UDP_RR 509.8 508.5 -0.3%

---
 arch/x86/kvm/x86.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c index renamed from arch/x86/kvm/x86.c +0 -0

d7374d768296..c0eb6574cbbb 100644

arch/x86/kvm/x86.c +0 -0
@@ -13015,7 +13015,22 @@ bool kvm_vector_hashing_enabled(void)
 
Quoted 7 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13017

Quoted 3 lines
arch/x86/kvm/x86.c +1 -1
 bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)  {
-	return (vcpu->arch.msr_kvm_poll_control & 1) == 0;
+	struct cpufreq_policy *policy = cpufreq_cpu_get(vcpu->cpu);

Preemption is not disabled at this point, which means that using vcpu->cpu is potentially unsafe. Given that cpufreq is refcounting the returned object, I gotta imaging get migrated to a different pCPU would be problematic.

Thanks for pointing this. Per my learning, even vCPU migrates to a different pCPU in the progress of getting the cpufreq policy, the only consequence is to get an outdated (maybe inaccurate) policy value. Halt polling mechanism still can get the updated and correct cpufreq policy in next time. And even we disable preemption in the process of obtaining cpufreq policy, the vCPU is still possible to be migrated a different pCPU after enabling preemption and before calling the halt polling judging logic.

Quoted 6 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13018

Quoted 2 lines
arch/x86/kvm/x86.c +1 -0
+	bool powersave = false;

I don't see anything in here that's x86 specific. Unless I'm missing something, this belongs in common KVM.

Yes, this is generic.

Quoted 10 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13026

Quoted 8 lines
arch/x86/kvm/x86.c +8 -0
+
+	/*
+	 * Halt polling could consume much CPU power, if CPU frequency
+	 * governor is set to "powersave", disable halt polling.
+	 */
+	if (policy) {
+		if ((policy->policy == CPUFREQ_POLICY_POWERSAVE) ||
+			(policy->governor &&

Indentation is messed up.

Sure. Would change.

Quoted 6 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13027

Quoted 2 lines
arch/x86/kvm/x86.c +1 -0
+				!strncmp(policy->governor->name,

"powersave",

KVM should not be comparing magic strings. If the cpufreq subsystem can't get policy->policy right, then that needs to be fixed.

Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like intel_pstate driver. For this kind of driver, the cpufreq governor is saved in the policy->policy field. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type is saved in the governor->name field. For the second kind of cpufreq driver, the policy->policy field is meaningless and we have to read the governor name.

Quoted 8 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13032

Quoted 6 lines
arch/x86/kvm/x86.c +5 -0
+					CPUFREQ_NAME_LEN)))
+			powersave = true;
+		cpufreq_cpu_put(policy);
+	}
+	return ((vcpu->arch.msr_kvm_poll_control & 1) == 0) || powersave;

Doing all of the above work if polling is disabled is silly.

Correct. Would change.

Quoted 6 lines

arch/x86/kvm/x86.c bool kvm_vector_hashing_enabled(void) · line 13034

Quoted 3 lines
arch/x86/kvm/x86.c +0 -0
 }
 EXPORT_SYMBOL_GPL(kvm_arch_no_poll);

All in all, _if_ we want to do this automatically and not let userspace decide how to manage powersave vs. halt-poll, I think this should be more like:

Thanks for your sample. Would change in V2 patch.

Quoted 54 lines
---
 virt/kvm/kvm_main.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c index renamed from virt/kvm/kvm_main.c +0 -0

e30f1b4ecfa5..01116859cb31 100644

virt/kvm/kvm_main.c +19 -1
@@ -29,6 +29,7 @@
 #include <linux/file.h>
 #include <linux/syscore_ops.h>
 #include <linux/cpu.h>
+#include <linux/cpufreq.h>
 #include <linux/sched/signal.h>
 #include <linux/sched/mm.h>
 #include <linux/sched/stat.h>
@@ -3483,6 +3484,23 @@ static inline void update_halt_poll_stats(struct
 kvm_vcpu *vcpu, ktime_t start,
 	}
 }
 
+static bool kvm_cpufreq_no_halt_poll(struct kvm_vcpu *vcpu) {
+	struct cpufreq_policy *policy;
+	bool powersave = false;
+
+	preempt_disable();
+
+	policy = cpufreq_cpu_get(vcpu->cpu);
+	if (policy) {
+		powersave = (policy->policy == CPUFREQ_POLICY_POWERSAVE);
+		cpufreq_cpu_put(policy);
+	}
+
+	preempt_enable();
+	return powersave;
+}
+
 /*
  * Emulate a vCPU halt condition, e.g. HLT on x86, WFI on arm, etc...  If halt
  * polling is enabled, busy wait for a short time before blocking to avoid the
@@ -3491,7 +3509,8 @@ static inline void update_halt_poll_stats(struct
 kvm_vcpu *vcpu, ktime_t start,
  */
 void kvm_vcpu_halt(struct kvm_vcpu *vcpu)  {
-	bool halt_poll_allowed = !kvm_arch_no_poll(vcpu);
+	const bool halt_poll_allowed = !kvm_arch_no_poll(vcpu) &&
+				       !kvm_cpufreq_no_halt_poll(vcpu);
 	bool do_halt_poll = halt_poll_allowed && vcpu->halt_poll_ns;
 	ktime_t start, cur, poll_end;
 	bool waited = false;

base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354 --

#3 lore raw
Quoted 6 lines · On Sat, Oct 08, 2022, Mi, Dapeng1 wrote:
Quoted 5 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+				!strncmp(policy->governor->name,

"powersave",

KVM should not be comparing magic strings. If the cpufreq subsystem can't get policy->policy right, then that needs to be fixed.

Quoted 8 lines

Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like intel_pstate driver. For this kind of driver, the cpufreq governor is saved in the policy->policy field. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type is saved in the governor->name field. For the second kind of cpufreq driver, the policy->policy field is meaningless and we have to read the governor name.

That doesn't mean it's ok to bleed those internal details into KVM. I would much rather cpufreq provide a helper to get the effective policy, e.g.

unsigned int cpufreq_cpu_get_policy(unsigned int cpu)

  {
	struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
	unsigned int pol;

	if (!policy)
		return CPUFREQ_POLICY_UNKNOWN;

	pol = policy->policy
	if (pol == CPUFREQ_POLICY_UNKNOWN && policy->governor)
		pol = cpufreq_parse_policy(policy->governor->name);

	cpufreq_cpu_put(policy);

}

#4 lore raw
Quoted 44 lines

From: Sean Christopherson <[email protected]>
Sent: Wednesday, October 26, 2022 1:07 AM
To: Mi, Dapeng1 <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: [PATCH] KVM: x86: disable halt polling when powersave governor is used

Quoted 10 lines · On Sat, Oct 08, 2022, Mi, Dapeng1 wrote:
Quoted 6 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+				!strncmp(policy->governor->name,

"powersave",

KVM should not be comparing magic strings. If the cpufreq subsystem can't get
policy->policy right, then that needs to be fixed.

Yeah, using magic strings looks a little bit strange, but this is what is cpufreq doing. Currently cpufreq mechanism supports two kinds of drivers, one is the driver which has the built-in governor, like intel_pstate

driver.

Quoted 3 lines

For this kind of driver, the cpufreq governor is saved in the policy->policy field. The other is the traditional driver which is independent with cpufreq governor and the cpufreq governor type is

saved in the governor->name field.

Quoted 2 lines

For the second kind of cpufreq driver, the policy->policy field is meaningless and we have to read the governor name.

That doesn't mean it's ok to bleed those internal details into KVM. I would much rather cpufreq provide a helper to get the effective policy, e.g.

unsigned int cpufreq_cpu_get_policy(unsigned int cpu)

  {
	struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
	unsigned int pol;

	if (!policy)
		return CPUFREQ_POLICY_UNKNOWN;

	pol = policy->policy
	if (pol == CPUFREQ_POLICY_UNKNOWN && policy->governor)
		pol = cpufreq_parse_policy(policy->governor->name);

	cpufreq_cpu_put(policy);
Quoted 1 line

}

Thanks Sean for reviewing. Would do in next version.