Skip to contentSkip to conversation

KVM: X86: Emulate APERF/MPERF to report actual VCPU frequency

PATCH In review Like Xu 4 participants 6 messages Jun 23, 2020 – Dec 22, 2021

Subsystems (3)
lore
Share & tools
lore

Like Xu

author
#0 lore raw

PATCH 1/1 KVM: X86: Emulate APERF/MPERF to report actual VCPU frequency

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---
 arch/x86/include/asm/kvm_host.h | 12 ++++++
 arch/x86/kvm/cpuid.c            |  8 +++-
 arch/x86/kvm/x86.c              | 76 ++++++++++++++++++++++++++++++++-
 3 files changed, 94 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +12 -0
index f852ee350beb..c48b9a0a086e 100644
@@ -539,6 +539,16 @@ struct kvm_vcpu_hv_stimer {
 	bool msg_pending;
 };
 
+/* vCPU thermal and power context */
+struct kvm_vcpu_hwp {
+	/* Hardware Coordination Feedback Capability (Presence of APERF/MPERF) */
+	bool hw_coord_fb_cap;
+	/* MPERF increases with a fixed frequency */
+	u64 mperf;
+	/* APERF increases with the current/actual frequency */
+	u64 aperf;
+};
+
 /* Hyper-V synthetic interrupt controller (SynIC)*/
 struct kvm_vcpu_hv_synic {
 	u64 version;
@@ -829,6 +839,8 @@ struct kvm_vcpu_arch {
 
 	/* AMD MSRC001_0015 Hardware Configuration */
 	u64 msr_hwcr;
+
+	struct kvm_vcpu_hwp hwp;
 };
 
 struct kvm_lpage_info {
arch/x86/kvm/cpuid.c +7 -1
index 8a294f9747aa..7057809e7cfd 100644
@@ -78,6 +78,11 @@ int kvm_update_cpuid(struct kvm_vcpu *vcpu)
 			apic->lapic_timer.timer_mode_mask = 1 << 17;
 	}
 
+	best = kvm_find_cpuid_entry(vcpu, 0x6, 0);
+	if (best && best->function == 0x6 &&
+	    boot_cpu_has(X86_FEATURE_APERFMPERF) && (best->ecx & 0x1))
+		vcpu->arch.hwp.hw_coord_fb_cap = true;
+
 	best = kvm_find_cpuid_entry(vcpu, 7, 0);
 	if (best && boot_cpu_has(X86_FEATURE_PKU) && best->function == 0x7)
 		cpuid_entry_change(best, X86_FEATURE_OSPKE,
@@ -561,7 +566,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 	case 6: /* Thermal management */
 		entry->eax = 0x4; /* allow ARAT */
 		entry->ebx = 0;
-		entry->ecx = 0;
+		/* allow aperf/mperf to report the true VCPU frequency. */
+		entry->ecx = boot_cpu_has(X86_FEATURE_APERFMPERF) ? 0x1 : 0;
 		entry->edx = 0;
 		break;
 	/* function 7 has additional index. */
arch/x86/kvm/x86.c +75 -1

114 lines not on this page · raw message

Signature · 1 line
2.21.3
#1 lore raw
Quoted 13 lines · Like Xu

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---

...

Quoted 14 lines
arch/x86/kvm/x86.c +2 -1
@@ -8312,7 +8376,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		dm_request_for_irq_injection(vcpu) &&
 		kvm_cpu_accept_dm_intr(vcpu);
 	fastpath_t exit_fastpath;
-
+	u64 enter_mperf = 0, enter_aperf = 0, exit_mperf = 0, exit_aperf = 0;
 	bool req_immediate_exit = false;
 
 	if (kvm_request_pending(vcpu)) {
@@ -8516,8 +8580,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
 	}
 
+	if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap))

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8593

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
+		get_host_amperf(&enter_mperf, &enter_aperf);
+
 	exit_fastpath = kvm_x86_ops.run(vcpu);
 
+	if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap)) {
+		get_host_amperf(&exit_mperf, &exit_aperf);
+		vcpu_update_amperf(vcpu, get_amperf_delta(enter_aperf, exit_aperf),
+			get_amperf_delta(enter_mperf, exit_mperf));
+	}
+

Is there an alternative approach that doesn't require 4 RDMSRs on every VMX round trip? That's literally more expensive than VM-Enter + VM-Exit combined.

E.g. what about adding KVM_X86_DISABLE_EXITS_APERF_MPERF and exposing the MSRs for read when that capability is enabled?

#2 lore raw
Quoted 47 lines · Sean Christopherson
Quoted 13 lines · Like Xu

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---

...

Quoted 14 lines
arch/x86/kvm/x86.c +2 -1
@@ -8312,7 +8376,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              dm_request_for_irq_injection(vcpu) &&
              kvm_cpu_accept_dm_intr(vcpu);
      fastpath_t exit_fastpath;
-
+     u64 enter_mperf = 0, enter_aperf = 0, exit_mperf = 0, exit_aperf = 0;
      bool req_immediate_exit = false;
 
      if (kvm_request_pending(vcpu)) {
@@ -8516,8 +8580,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
      }
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap))

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8593

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
+             get_host_amperf(&enter_mperf, &enter_aperf);
+
      exit_fastpath = kvm_x86_ops.run(vcpu);
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap)) {
+             get_host_amperf(&exit_mperf, &exit_aperf);
+             vcpu_update_amperf(vcpu, get_amperf_delta(enter_aperf, exit_aperf),
+                     get_amperf_delta(enter_mperf, exit_mperf));
+     }
+

Is there an alternative approach that doesn't require 4 RDMSRs on every VMX round trip? That's literally more expensive than VM-Enter + VM-Exit combined.

Quoted 2 lines

E.g. what about adding KVM_X86_DISABLE_EXITS_APERF_MPERF and exposing the MSRs for read when that capability is enabled?

When would you load the hardware MSRs with the guest/host values?

#3 lore raw
Quoted 49 lines · Jim Mattson
Quoted 47 lines · Sean Christopherson
Quoted 13 lines · Like Xu

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---

...

Quoted 14 lines
arch/x86/kvm/x86.c +2 -1
@@ -8312,7 +8376,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              dm_request_for_irq_injection(vcpu) &&
              kvm_cpu_accept_dm_intr(vcpu);
      fastpath_t exit_fastpath;
-
+     u64 enter_mperf = 0, enter_aperf = 0, exit_mperf = 0, exit_aperf = 0;
      bool req_immediate_exit = false;
 
      if (kvm_request_pending(vcpu)) {
@@ -8516,8 +8580,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
      }
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap))

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8593

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
+             get_host_amperf(&enter_mperf, &enter_aperf);
+
      exit_fastpath = kvm_x86_ops.run(vcpu);
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap)) {
+             get_host_amperf(&exit_mperf, &exit_aperf);
+             vcpu_update_amperf(vcpu, get_amperf_delta(enter_aperf, exit_aperf),
+                     get_amperf_delta(enter_mperf, exit_mperf));
+     }
+

Is there an alternative approach that doesn't require 4 RDMSRs on every VMX round trip? That's literally more expensive than VM-Enter + VM-Exit combined.

Quoted 4 lines
Quoted 2 lines

E.g. what about adding KVM_X86_DISABLE_EXITS_APERF_MPERF and exposing the MSRs for read when that capability is enabled?

When would you load the hardware MSRs with the guest/host values?

Ugh, I was thinking the MSRs were read-only.

Doesn't this also interact with TSC scaling?

#4 lore raw
Quoted 51 lines · Sean Christopherson
Quoted 49 lines · Jim Mattson
Quoted 47 lines · Sean Christopherson
Quoted 13 lines · Like Xu

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---

...

Quoted 14 lines
arch/x86/kvm/x86.c +2 -1
@@ -8312,7 +8376,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              dm_request_for_irq_injection(vcpu) &&
              kvm_cpu_accept_dm_intr(vcpu);
      fastpath_t exit_fastpath;
-
+     u64 enter_mperf = 0, enter_aperf = 0, exit_mperf = 0, exit_aperf = 0;
      bool req_immediate_exit = false;
 
      if (kvm_request_pending(vcpu)) {
@@ -8516,8 +8580,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
              vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
      }
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap))

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8593

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
+             get_host_amperf(&enter_mperf, &enter_aperf);
+
      exit_fastpath = kvm_x86_ops.run(vcpu);
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap)) {
+             get_host_amperf(&exit_mperf, &exit_aperf);
+             vcpu_update_amperf(vcpu, get_amperf_delta(enter_aperf, exit_aperf),
+                     get_amperf_delta(enter_mperf, exit_mperf));
+     }
+

Is there an alternative approach that doesn't require 4 RDMSRs on every VMX round trip? That's literally more expensive than VM-Enter + VM-Exit combined.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

E.g. what about adding KVM_X86_DISABLE_EXITS_APERF_MPERF and exposing the MSRs for read when that capability is enabled?

When would you load the hardware MSRs with the guest/host values?

Ugh, I was thinking the MSRs were read-only.

EVen if they were read-only, they should power on to zero, and they will most likely not be zero when a guest powers on.

Quoted 1 line

Doesn't this also interact with TSC scaling?

Yes, it should!

#5 lore raw
Quoted 42 lines · Jim Mattson
Quoted 40 lines · Sean Christopherson
Quoted 38 lines · Jim Mattson
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Like Xu

The aperf/mperf are used to report current CPU frequency after 7d5905dc14a "x86 / CPU: Always show current CPU frequency in /proc/cpuinfo". But guest kernel always reports a fixed VCPU frequency in the /proc/cpuinfo, which may confuse users especially when turbo is enabled on the host.

Emulate guest APERF/MPERF capability based their values on the host.

---

...

Quoted 18 lines
arch/x86/kvm/x86.c +4 -1
@@ -8312,7 +8376,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
               dm_request_for_irq_injection(vcpu) &&
               kvm_cpu_accept_dm_intr(vcpu);
       fastpath_t exit_fastpath;
-
+     u64 enter_mperf = 0, enter_aperf = 0, exit_mperf = 0, exit_aperf = 0;
       bool req_immediate_exit = false;
 
       if (kvm_request_pending(vcpu)) {
@@ -8516,8 +8580,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
               vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
       }
 
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap))
+             get_host_amperf(&enter_mperf, &enter_aperf);
+
       exit_fastpath = kvm_x86_ops.run(vcpu);
 
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 8593

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+     if (unlikely(vcpu->arch.hwp.hw_coord_fb_cap)) {
+             get_host_amperf(&exit_mperf, &exit_aperf);
+             vcpu_update_amperf(vcpu, get_amperf_delta(enter_aperf, exit_aperf),
+                     get_amperf_delta(enter_mperf, exit_mperf));
+     }
+

Is there an alternative approach that doesn't require 4 RDMSRs on every VMX round trip? That's literally more expensive than VM-Enter + VM-Exit combined.

It looks like we have quite a few users who are expecting this feature in different scenarios.

I will add a fast path for RO usage and a slow path if the guest tries to change the AMPERF values.

Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

E.g. what about adding KVM_X86_DISABLE_EXITS_APERF_MPERF and exposing the MSRs for read when that capability is enabled?

When would you load the hardware MSRs with the guest/host values?

Ugh, I was thinking the MSRs were read-only.

EVen if they were read-only, they should power on to zero, and they will most likely not be zero when a guest powers on.

Can we assume that "not zero when the guest is on" will not harm any guests ?

Quoted 4 lines
Quoted 2 lines

Doesn't this also interact with TSC scaling?

Yes, it should!

We have too much of a historical burden on TSC emulations.

For practical reasons, what if we only expose the AMPERF cap if the host/guest has both CONSTANT_TSC and NONSTOP_TSC ?

One more design concern, I wonder if it is *safe* for the guest to read amperf on pCPU[x] the first time and on pCPU[y] the next time.

Any input ?

Thanks,
Like Xu