target/i386/kvm/pmu: PMU Enhancement, Bugfix and Cleanup

Although AMD PERFCORE and PerfMonV2 are removed when "-pmu" is configured, there is no way to fully disable KVM AMD PMU virtualization. Neither "-cpu host,-pmu" nor "-cpu EPYC" achieves this.

As a result, the following message still appears in the VM dmesg:

[    0.263615] Performance Events: AMD PMU driver.

However, the expected output should be:

[    0.596381] Performance Events: PMU not available due to virtualization, using software events only.
[    0.600972] NMI watchdog: Perf NMI watchdog permanently disabled

This occurs because AMD does not use any CPUID bit to indicate PMU availability.

To address this, KVM_CAP_PMU_CAPABILITY is used to set KVM_PMU_CAP_DISABLE when "-pmu" is configured.

Signed-off-by Dongli Zhang <[email protected]>

Reviewed-by Xiaoyao Li <[email protected]>

Reviewed-by Zhao Liu <[email protected]>

Reviewed-by Dapeng Mi <[email protected]>

  target/i386/kvm/kvm.c | 31 +++++++++++++++++++++++++++++++
  1 file changed, 31 insertions(+)

target/i386/kvm/kvm.c +31 -0

index 7b9b740a8e..c98832f423 100644
@@ -179,6 +179,8 @@ static int has_triple_fault_event;
 
  static bool has_msr_mcg_ext_ctl;
 
+static int pmu_cap;
+
  static struct kvm_cpuid2 *cpuid_cache;
  static struct kvm_cpuid2 *hv_cpuid_cache;
  static struct kvm_msr_list *kvm_feature_msrs;

@@ -2080,6 +2082,33 @@ full:
 
  int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
  {
+    static bool first = true;
+    int ret;
+
+    if (first) {
+        first = false;
+
+        /*
+         * Since Linux v5.18, KVM provides a VM-level capability to easily
+         * disable PMUs; however, QEMU has been providing PMU property per
+         * CPU since v1.6. In order to accommodate both, have to configure
+         * the VM-level capability here.
+         *
+         * KVM_PMU_CAP_DISABLE doesn't change the PMU
+         * behavior on Intel platform because current "pmu" property works
+         * as expected.
+         */
+        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {
+            ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                    KVM_PMU_CAP_DISABLE);
+            if (ret < 0) {
+                error_setg_errno(errp, -ret,
+                                 "Failed to set KVM_PMU_CAP_DISABLE");
+                return ret;
+            }
+        }
+    }
+
      if (is_tdx_vm()) {
          return tdx_pre_create_vcpu(cpu, errp);
      }

@@ -3391,6 +3420,8 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
          }
      }
 
+    pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
+
      return 0;
  }
 

Quoted 16 lines · Dongli Zhang

Although AMD PERFCORE and PerfMonV2 are removed when "-pmu" is configured, there is no way to fully disable KVM AMD PMU virtualization. Neither "-cpu host,-pmu" nor "-cpu EPYC" achieves this.

As a result, the following message still appears in the VM dmesg:

[    0.263615] Performance Events: AMD PMU driver.

However, the expected output should be:

[    0.596381] Performance Events: PMU not available due to virtualization, using software events only. [    0.600972] NMI watchdog: Perf NMI watchdog permanently disabled

This occurs because AMD does not use any CPUID bit to indicate PMU availability.

Quoted 7 lines

To address this, KVM_CAP_PMU_CAPABILITY is used to set KVM_PMU_CAP_DISABLE when "-pmu" is configured.

Signed-off-by Dongli Zhang <[email protected]>

Reviewed-by Xiaoyao Li <[email protected]>

Reviewed-by Zhao Liu <[email protected]>

Reviewed-by Dapeng Mi <[email protected]>

This is commit 4f0dd09e61b080c27589fe35b575503783c5a256 (v10.2.0-1114-g4f0dd09e61b) in the master branch now.

This change seem to broke booting windows 10/11 on an AMD host, in paticular with -cpu EPYC-Rome-v4.  It were initially reported in debian qemu bug tracker, and I verified and bisected the issue on my AMD desktop machine (which is AMD Ryzen 7 5700G, Zen 3.

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Before this patch, it boots ok.

There's no other fancy options, just

 qemu-system-x86_64 -m 2G -enable-kvm -cpu EPYC-Rome-v4 -smp 2

(I'm replying to the original email message with all original recipients, hope it's okay).

Can someone please take a look?

On 01-10-2026 14:02, Michael Tokarev wrote:

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 3 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Quoted 1 line

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

Quoted 9 lines · Michael Tokarev

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 3 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Quoted 5 lines

Quoted 1 line

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

A guest OS cannot expect a functioning PMU if the hypervisor has disabled it. Linux runs a basic sanity test to determine this. See:
https://github.com/torvalds/linux/blob/master/arch/x86/events/core.c#L268

Quoted 15 lines · Sandipan Das

Quoted 14 lines · Michael Tokarev

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 6 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

Quoted 3 lines

A guest OS cannot expect a functioning PMU if the hypervisor has disabled it. Linux runs a basic sanity test to determine this. See:
https://github.com/torvalds/linux/blob/master/arch/x86/events/core.c#L268

So.. what do we do with all this?  :)

I kinda see the intention of the original change, but it results in non-bootable systems (linux too, as I think you demonstrate above).

Quoted 22 lines · Michael Tokarev

Quoted 10 lines · Sandipan Das

Quoted 9 lines · Michael Tokarev

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 3 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

A guest OS cannot expect a functioning PMU if the hypervisor has disabled it. Linux runs a basic sanity test to determine this. See:
https://github.com/torvalds/linux/blob/master/arch/x86/events/core.c#L268

So.. what do we do with all this?  :)

Quoted 3 lines

I kinda see the intention of the original change, but it results in non-bootable systems (linux too, as I think you demonstrate above).

Because of the sanity test, Linux will boot but not register the "cpu" PMU. Perhaps others should also run a similar sanity test on PMU hardware rather than making assumptions and failing to boot.

On 01-10-2026 16: 42, Sandipan Das wrote: > On 01-10-2026 16: 35, Michael Tokarev wrote: >> On 10/1/26 13: 43, Sandipan Das wrote: >>> On 01-10-2026 15:
30, Michael Tokarev wrote: >>>> On 10/1/26 11: 54, Sandipan Das

Quoted 12 lines · Sandipan Das

Quoted 11 lines · Michael Tokarev

Quoted 10 lines · Sandipan Das

Quoted 9 lines · Michael Tokarev

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 3 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

A guest OS cannot expect a functioning PMU if the hypervisor has disabled it. Linux runs a basic sanity test to determine this. See:
https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/master/arch/

x86/events/core.c*L268__;Iw!!ACWV5N9M2RV99hQ!
PdJ7r8w0UUuD5iOlCxu7931wevMkDprX9yCV5mkfktbFkpOobtIWrkMt0Gmb9Ztnj3iZHwTK4UQHh85raOPEmgo$ <https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/master/arch/x86/events/core.c*L268__;Iw!!ACWV5N9M2RV99hQ!PdJ7r8w0UUuD5iOlCxu7931wevMkDprX9yCV5mkfktbFkpOobtIWrkMt0Gmb9Ztnj3iZHwTK4UQHh85raOPEmgo$>

Quoted 10 lines

Quoted 6 lines

So.. what do we do with all this? :)

I kinda see the intention of the original change, but it results in non-bootable systems (linux too, as I think you demonstrate above).

Because of the sanity test, Linux will boot but not register the "cpu" PMU. Perhaps others should also run a similar sanity test on PMU hardware rather than making assumptions and failing to boot.

After looking at kvm_msr and kvm_cpuid traces, I see that Windows is indeed getting #GPs upon accessing MSRs 0xc0010200 and 0xc0010201 but it is not at fault here since it expects these MSRs to be available as guest CPUID 0x80000001[ECX].PerfCtrExtCore is set. This feature bit should have been cleared by QEMU when "pmu=off".

It boots up when "pmu=off" is combined with "perfctr-core=off" and "perfmon-v2=off". The missing pieces here are the following:

https://lore.kernel.org/qemu-devel/[email protected]/
https://lore.kernel.org/qemu-devel/[email protected]/

Dongli, Zhao,

I see a discussion about compat array at
https://lore.kernel.org/qemu-devel/[email protected]/

Is this ready for the two patches above to be pulled in?

Quoted 50 lines · Sandipan Das

On 01-10-2026 16: 42, Sandipan Das wrote: > On 01-10-2026 16: 35, Michael Tokarev wrote: >> On 10/1/26 13: 43, Sandipan Das wrote: >>> On 01-10-2026 15:
30, Michael Tokarev wrote: >>>> On 10/1/26 11: 54, Sandipan Das

Quoted 12 lines · Sandipan Das

Quoted 11 lines · Michael Tokarev

Quoted 10 lines · Sandipan Das

Quoted 9 lines · Michael Tokarev

Quoted 1 line · Sandipan Das

On 01-10-2026 14:02, Michael Tokarev wrote:

...

Quoted 3 lines

Quoted 3 lines

When booting windows11 VM with -cpu EPYC-Rome-v4 -enable-kvm, the guest enters tight CPU loop while displaying a boot screen, before it starts showing up the rotating circle (which shows boot progress).

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

What happens if you use "-cpu EPYC-Rome-v4,+pmu"?

With that, it boots fine, just like before this patch. I checked current master too - same, +pmu fixes it.

A guest OS cannot expect a functioning PMU if the hypervisor has disabled it. Linux runs a basic sanity test to determine this. See:
https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/master/arch/

x86/events/core.c*L268__;Iw!!ACWV5N9M2RV99hQ!
PdJ7r8w0UUuD5iOlCxu7931wevMkDprX9yCV5mkfktbFkpOobtIWrkMt0Gmb9Ztnj3iZHwTK4UQHh85raOPEmgo$ <https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/master/arch/x86/events/core.c*L268__;Iw!!ACWV5N9M2RV99hQ!PdJ7r8w0UUuD5iOlCxu7931wevMkDprX9yCV5mkfktbFkpOobtIWrkMt0Gmb9Ztnj3iZHwTK4UQHh85raOPEmgo$>

Quoted 10 lines

Quoted 6 lines

So.. what do we do with all this? :)

I kinda see the intention of the original change, but it results in non-bootable systems (linux too, as I think you demonstrate above).

Because of the sanity test, Linux will boot but not register the "cpu" PMU. Perhaps others should also run a similar sanity test on PMU hardware rather than making assumptions and failing to boot.

After looking at kvm_msr and kvm_cpuid traces, I see that Windows is indeed getting #GPs upon accessing MSRs 0xc0010200 and 0xc0010201 but it is not at fault here since it expects these MSRs to be available as guest CPUID 0x80000001[ECX].PerfCtrExtCore is set. This feature bit should have been cleared by QEMU when "pmu=off".

It boots up when "pmu=off" is combined with "perfctr-core=off" and "perfmon-v2=off". The missing pieces here are the following:

https://lore.kernel.org/qemu-devel/[email protected]/
https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 7 lines

Dongli, Zhao,

I see a discussion about compat array at
https://lore.kernel.org/qemu-devel/[email protected]/

Is this ready for the two patches above to be pulled in?

Thanks Sandipan for analyzing the issue!

Hi Zhao,

I noticed that the machine compat arrays are now available. Would you be able to send updated versions of patches 1 and 2 with the compatibility property you suggested?

When PMU is enabled in QEMU, there is a chance that PMU virtualization is completely disabled by the KVM module parameter kvm.enable_pmu=N.

The kvm.enable_pmu parameter is introduced since Linux v5.17. Its permission is 0444. It does not change until a reload of the KVM module.

Read the kvm.enable_pmu value from the module sysfs to give a chance to
provide more information about vPMU enablement.> Signed-off-by: Dongli Zhang <[email protected]>

Reviewed-by Zhao Liu <[email protected]>

Reviewed-by Dapeng Mi <[email protected]>

---
Changed since v2:
- Rework the code flow following Zhao's suggestion.
- Return error when:

    (*kvm_enable_pmu == 'N' && X86_CPU(cpu)->enable_pmu)

Changed since v3:
- Re-split the cases into enable_pmu and !enable_pmu, following Zhao's suggestion.
- Rework the commit messages.
- Bring back global static variable 'kvm_pmu_disabled' from v2.
Changed since v4:
- Add Reviewed-by from Zhao.
Changed since v5:
- Rebase on top of most recent QEMU.
Changed since v6:
- Add Reviewed-by from Dapeng Mi.

 target/i386/kvm/kvm.c | 61 +++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 17 deletions(-)

target/i386/kvm/kvm.c +4 -11

index 3b803c662d..338b9558e4 100644
@@ -187,6 +187,10 @@ static int has_triple_fault_event;
 static bool has_msr_mcg_ext_ctl;
 
 static int pmu_cap;
+/*
+ * Read from /sys/module/kvm/parameters/enable_pmu.
+ */
+static bool kvm_pmu_disabled;
 
 static struct kvm_cpuid2 *cpuid_cache;
 static struct kvm_cpuid2 *hv_cpuid_cache;

@@ -2068,23 +2072,30 @@ int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
     if (first) {
         first = false;
 
-        /*
-         * Since Linux v5.18, KVM provides a VM-level capability to easily
-         * disable PMUs; however, QEMU has been providing PMU property per
-         * CPU since v1.6. In order to accommodate both, have to configure
-         * the VM-level capability here.
-         *
-         * KVM_PMU_CAP_DISABLE doesn't change the PMU
-         * behavior on Intel platform because current "pmu" property works
-         * as expected.
-         */
-        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {

target/i386/kvm/kvm.c +23 -0

+            }
+        } else {
+            /*
+             * Since Linux v5.18, KVM provides a VM-level capability to easily
+             * disable PMUs; however, QEMU has been providing PMU property per
+             * CPU since v1.6. In order to accommodate both, have to configure
+             * the VM-level capability here.
+             *
+             * KVM_PMU_CAP_DISABLE doesn't change the PMU
+             * behavior on Intel platform because current "pmu" property works
+             * as expected.
+             */
+            if (pmu_cap & KVM_PMU_CAP_DISABLE) {
+                ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                        KVM_PMU_CAP_DISABLE);
+                if (ret < 0) {
+                    error_setg_errno(errp, -ret,
+                                     "Failed to set KVM_PMU_CAP_DISABLE");
+                    return ret;
+                }
             }
         }
     }

@@ -3302,6 +3313,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     int ret;
     struct utsname utsname;
     Error *local_err = NULL;
+    g_autofree char *kvm_enable_pmu;
 
     /*
      * Initialize confidential guest (SEV/TDX) context, if required

@@ -3437,6 +3449,21 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
 
+    /*
+     * The enable_pmu parameter is introduced since Linux v5.17,

target/i386/kvm/kvm.c +3 -0

+        }
+    }
+
     return 0;
 }
 

On 12/29/2025 11:42 PM, Dongli Zhang wrote:

Quoted 18 lines

static struct kvm_cpuid2 *cpuid_cache;
static struct kvm_cpuid2 *hv_cpuid_cache;

target/i386/kvm/kvm.c +0 -11

@@ -2068,23 +2072,30 @@ int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
     if (first) {
         first = false;
 
-        /*
-         * Since Linux v5.18, KVM provides a VM-level capability to easily
-         * disable PMUs; however, QEMU has been providing PMU property per
-         * CPU since v1.6. In order to accommodate both, have to configure
-         * the VM-level capability here.
-         *
-         * KVM_PMU_CAP_DISABLE doesn't change the PMU
-         * behavior on Intel platform because current "pmu" property works
-         * as expected.
-         */
-        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {

target/i386/kvm/kvm.c int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp) · line 2078

Quoted 10 lines

target/i386/kvm/kvm.c +4 -6

-            ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
-                                    KVM_PMU_CAP_DISABLE);
-            if (ret < 0) {
-                error_setg_errno(errp, -ret,
-                                 "Failed to set KVM_PMU_CAP_DISABLE");
-                return ret;
+        if (X86_CPU(cpu)->enable_pmu) {
+            if (kvm_pmu_disabled) {
+                warn_report("Failed to enable PMU since "
+                            "KVM's enable_pmu parameter is disabled");

I'm wondering about the intended value of this patch?

If enable_pmu is true in QEMU but the corresponding KVM parameter is false, then KVM_GET_SUPPORTED_CPUID or KVM_GET_MSRS should be able to tell that the PMU feature is not supported by host.

The logic implemented in this patch seems somewhat redundant.

Additionally, in this scenario — where the user intends to enable a feature but the host cannot support it — normally no warning is emitted by QEMU.

Quoted 38 lines

target/i386/kvm/kvm.c +23 -0

+            }
+        } else {
+            /*
+             * Since Linux v5.18, KVM provides a VM-level capability to easily
+             * disable PMUs; however, QEMU has been providing PMU property per
+             * CPU since v1.6. In order to accommodate both, have to configure
+             * the VM-level capability here.
+             *
+             * KVM_PMU_CAP_DISABLE doesn't change the PMU
+             * behavior on Intel platform because current "pmu" property works
+             * as expected.
+             */
+            if (pmu_cap & KVM_PMU_CAP_DISABLE) {
+                ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                        KVM_PMU_CAP_DISABLE);
+                if (ret < 0) {
+                    error_setg_errno(errp, -ret,
+                                     "Failed to set KVM_PMU_CAP_DISABLE");
+                    return ret;
+                }
             }
         }
     }

@@ -3302,6 +3313,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     int ret;
     struct utsname utsname;
     Error *local_err = NULL;
+    g_autofree char *kvm_enable_pmu;
 
     /*
      * Initialize confidential guest (SEV/TDX) context, if required

@@ -3437,6 +3449,21 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
 
+    /*
+     * The enable_pmu parameter is introduced since Linux v5.17,

target/i386/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 3463

Quoted 10 lines

target/i386/kvm/kvm.c +10 -0

+     * give a chance to provide more information about vPMU
+     * enablement.
+     *
+     * The kvm.enable_pmu's permission is 0444. It does not change
+     * until a reload of the KVM module.
+     */
+    if (g_file_get_contents("/sys/module/kvm/parameters/enable_pmu",
+                            &kvm_enable_pmu, NULL, NULL)) {
+        if (*kvm_enable_pmu == 'N') {
+            kvm_pmu_disabled = true;

It’s generally better not to rely on KVM’s internal implementation unless really necessary.

For example, in the new mediated vPMU framework, even if the KVM module parameter enable_pmu is set, the per-guest kvm->arch.enable_pmu could still be cleared.

In such a case, the logic here might not be correct.

Hi Zide,

Quoted 3 lines · Zide

On 12/29/2025 11:42 PM, Dongli Zhang wrote:

[snip]

Quoted 31 lines

Quoted 18 lines

static struct kvm_cpuid2 *cpuid_cache;
static struct kvm_cpuid2 *hv_cpuid_cache;

target/i386/kvm/kvm.c +0 -11

@@ -2068,23 +2072,30 @@ int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
     if (first) {
         first = false;
 
-        /*
-         * Since Linux v5.18, KVM provides a VM-level capability to easily
-         * disable PMUs; however, QEMU has been providing PMU property per
-         * CPU since v1.6. In order to accommodate both, have to configure
-         * the VM-level capability here.
-         *
-         * KVM_PMU_CAP_DISABLE doesn't change the PMU
-         * behavior on Intel platform because current "pmu" property works
-         * as expected.
-         */
-        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {

target/i386/kvm/kvm.c int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp) · line 2078

Quoted 10 lines

target/i386/kvm/kvm.c +4 -6

-            ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
-                                    KVM_PMU_CAP_DISABLE);
-            if (ret < 0) {
-                error_setg_errno(errp, -ret,
-                                 "Failed to set KVM_PMU_CAP_DISABLE");
-                return ret;
+        if (X86_CPU(cpu)->enable_pmu) {
+            if (kvm_pmu_disabled) {
+                warn_report("Failed to enable PMU since "
+                            "KVM's enable_pmu parameter is disabled");

I'm wondering about the intended value of this patch?

Quoted 5 lines

If enable_pmu is true in QEMU but the corresponding KVM parameter is false, then KVM_GET_SUPPORTED_CPUID or KVM_GET_MSRS should be able to tell that the PMU feature is not supported by host.

The logic implemented in this patch seems somewhat redundant.

For Intel, the QEMU userspace can determine if the vPMU is disabled by KVM through the use of KVM_GET_SUPPORTED_CPUID.

However, this approach does not apply to AMD. Unlike Intel, AMD does not rely on CPUID to detect whether PMU is supported. By default, we can assume that PMU is always available, except for the recent PerfMonV2 feature.

The main objective of this PATCH 4/7 is to introduce the variable 'kvm_pmu_disabled', which will be reused in PATCH 5/7 to skip any PMU initialization if the parameter is set to 'N'.

The 'kvm_pmu_disabled' variable is used to differentiate between the following two scenarios on AMD:

(1) A newer KVM with KVM_PMU_CAP_DISABLE support, but explicitly disabled via the KVM parameter ('N').

(2) An older KVM without KVM_CAP_PMU_CAPABILITY support.

In both cases, the call to KVM_CAP_PMU_CAPABILITY extension support check may return 0.

By reading the file "/sys/module/kvm/parameters/enable_pmu", we can distinguish between these two scenarios.

As you mentioned, another approach would be to use KVM_GET_MSRS to specifically probe for AMD during QEMU initialization. In this case, we can set 'kvm_pmu_disabled' to true if reading the AMD PMU MSR registers fails.

To implement this, we may need to:

1. Turn this patch to be AMD specific by probing the AMD PMU registers during initialization. We may need go create a new function in QEMU to use KVM_GET_MSRS for probing only, or we may re-use kvm_arch_get_supported_msr_feature() or kvm_get_one_msr(). I may change in the next version.

2. Limit the usage of 'kvm_pmu_disabled' to be AMD specific in PATCH 5/7.

Quoted 4 lines

Additionally, in this scenario — where the user intends to enable a feature but the host cannot support it — normally no warning is emitted by QEMU.

According to the usage of QEMU, may I assume QEMU already prints warning logs for unsupported features? The below is an example.

QEMU 10.2.50 monitor - type 'help' for more information
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.rtm [bit 11]

Quoted 52 lines

Quoted 38 lines

target/i386/kvm/kvm.c +23 -0

+            }
+        } else {
+            /*
+             * Since Linux v5.18, KVM provides a VM-level capability to easily
+             * disable PMUs; however, QEMU has been providing PMU property per
+             * CPU since v1.6. In order to accommodate both, have to configure
+             * the VM-level capability here.
+             *
+             * KVM_PMU_CAP_DISABLE doesn't change the PMU
+             * behavior on Intel platform because current "pmu" property works
+             * as expected.
+             */
+            if (pmu_cap & KVM_PMU_CAP_DISABLE) {
+                ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                        KVM_PMU_CAP_DISABLE);
+                if (ret < 0) {
+                    error_setg_errno(errp, -ret,
+                                     "Failed to set KVM_PMU_CAP_DISABLE");
+                    return ret;
+                }
             }
         }
     }

@@ -3302,6 +3313,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     int ret;
     struct utsname utsname;
     Error *local_err = NULL;
+    g_autofree char *kvm_enable_pmu;
 
     /*
      * Initialize confidential guest (SEV/TDX) context, if required

@@ -3437,6 +3449,21 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
 
+    /*
+     * The enable_pmu parameter is introduced since Linux v5.17,

target/i386/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 3463

Quoted 10 lines

target/i386/kvm/kvm.c +10 -0

+     * give a chance to provide more information about vPMU
+     * enablement.
+     *
+     * The kvm.enable_pmu's permission is 0444. It does not change
+     * until a reload of the KVM module.
+     */
+    if (g_file_get_contents("/sys/module/kvm/parameters/enable_pmu",
+                            &kvm_enable_pmu, NULL, NULL)) {
+        if (*kvm_enable_pmu == 'N') {
+            kvm_pmu_disabled = true;

It’s generally better not to rely on KVM’s internal implementation unless really necessary.

Quoted 5 lines

For example, in the new mediated vPMU framework, even if the KVM module parameter enable_pmu is set, the per-guest kvm->arch.enable_pmu could still be cleared.

In such a case, the logic here might not be correct.

Would the Mediated vPMU set KVM_PMU_CAP_DISABLE to clear per-VM enable_pmu even when the global KVM parameter enable_pmu=N is set?

In this scenario, we plan to rely on KVM_PMU_CAP_DISABLE only when the value of "/sys/module/kvm/parameters/enable_pmu" is not equal to N.

Can I assume that this will work with Mediated vPMU?

Is there any possibility to follow the current approach before Mediated vPMU is finalized for mainline, and later introduce an incremental change using KVM_GET_MSRS probing? The current approach is straightforward and can work with existing Linux kernel source code.

For quite some time, QEMU has lacked support for disabling or resetting AMD PMU registers. If we could add this feature before Mediated vPMU is finalized, it would benefit many existing kernel versions. This patchset solves production bugs.

Feel free to let me know your thought, while I would starting working on next version now.

Thank you very much!

Dongli Zhang

Quoted 57 lines · Dongli Zhang

Hi Zide,

Quoted 3 lines · Zide

On 12/29/2025 11:42 PM, Dongli Zhang wrote:

[snip]

Quoted 31 lines

Quoted 18 lines

static struct kvm_cpuid2 *cpuid_cache;
static struct kvm_cpuid2 *hv_cpuid_cache;

target/i386/kvm/kvm.c +0 -11

@@ -2068,23 +2072,30 @@ int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
     if (first) {
         first = false;
 
-        /*
-         * Since Linux v5.18, KVM provides a VM-level capability to easily
-         * disable PMUs; however, QEMU has been providing PMU property per
-         * CPU since v1.6. In order to accommodate both, have to configure
-         * the VM-level capability here.
-         *
-         * KVM_PMU_CAP_DISABLE doesn't change the PMU
-         * behavior on Intel platform because current "pmu" property works
-         * as expected.
-         */
-        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {

target/i386/kvm/kvm.c int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp) · line 2078

Quoted 10 lines

target/i386/kvm/kvm.c +4 -6

-            ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
-                                    KVM_PMU_CAP_DISABLE);
-            if (ret < 0) {
-                error_setg_errno(errp, -ret,
-                                 "Failed to set KVM_PMU_CAP_DISABLE");
-                return ret;
+        if (X86_CPU(cpu)->enable_pmu) {
+            if (kvm_pmu_disabled) {
+                warn_report("Failed to enable PMU since "
+                            "KVM's enable_pmu parameter is disabled");

I'm wondering about the intended value of this patch?

Quoted 5 lines

If enable_pmu is true in QEMU but the corresponding KVM parameter is false, then KVM_GET_SUPPORTED_CPUID or KVM_GET_MSRS should be able to tell that the PMU feature is not supported by host.

The logic implemented in this patch seems somewhat redundant.

For Intel, the QEMU userspace can determine if the vPMU is disabled by KVM through the use of KVM_GET_SUPPORTED_CPUID.

However, this approach does not apply to AMD. Unlike Intel, AMD does not rely on CPUID to detect whether PMU is supported. By default, we can assume that PMU is always available, except for the recent PerfMonV2 feature.

The main objective of this PATCH 4/7 is to introduce the variable 'kvm_pmu_disabled', which will be reused in PATCH 5/7 to skip any PMU initialization if the parameter is set to 'N'.

Review

Quoted 10 lines

(quoted diff) +10 -0

+static void kvm_init_pmu_info(struct kvm_cpuid2 *cpuid, X86CPU *cpu)
+{
+    CPUX86State *env = &cpu->env;
+
+    /*
+     * The PMU virtualization is disabled by kvm.enable_pmu=N.
+     */
+    if (kvm_pmu_disabled) {
+        return;
+    }

Thanks for explanation.

Quoted 6 lines

The 'kvm_pmu_disabled' variable is used to differentiate between the following two scenarios on AMD:

(1) A newer KVM with KVM_PMU_CAP_DISABLE support, but explicitly disabled via the KVM parameter ('N').

Quoted 7 lines

(2) An older KVM without KVM_CAP_PMU_CAPABILITY support.

In both cases, the call to KVM_CAP_PMU_CAPABILITY extension support check may return 0.

By reading the file "/sys/module/kvm/parameters/enable_pmu", we can distinguish between these two scenarios.

As described in PATCH 1/7, without issuing KVM_PMU_CAP_DISABLE, KVM has no way to know that userspace does not intend to enable vPMU in AMD platforms, and therefore does not fault guest accesses to PMU MSRs.

My understanding is that the issue being addressed here is basically the
opposite: QEMU does not know that vPMU is disabled by KVM.

IIUC, one difference between Intel and AMD is that AMD lacks a CPUID leaf to indicate the availability of PMU version 1. But Intel potentially could be in the same situation that KVM advertises PMU availability but it's not actually supported. (e.g. kvm->arch.enable_pmu is false while modules parameter enable_pmu is true).

From the guest’s point of view, it probes PMU MSRs to determine whether PMU support is present and it's fine in this situation.

In userspace, QEMU may issue KVM_SET_MSRS / KVM_GET_MSRS to KVM without knowing that vPMU has been disabled by KVM. I think these IOCTLs should not fail, since KVM states that “Userspace is allowed to read MSRs, and write ‘0’ to MSRs, that KVM advertises to userspace, even if an MSR isn’t fully supported.”

My current understanding is that AMD should be fine even without kvm_pmu_disabled, but I may be missing some context here.

The bottom line is this patch doesn't handle the cases that KVM still could disable vPMU support even if enable_pmu is true.

Quoted 6 lines

As you mentioned, another approach would be to use KVM_GET_MSRS to specifically probe for AMD during QEMU initialization. In this case, we can set 'kvm_pmu_disabled' to true if reading the AMD PMU MSR registers fails.

Quoted 8 lines

To implement this, we may need to:

1. Turn this patch to be AMD specific by probing the AMD PMU registers during initialization. We may need go create a new function in QEMU to use KVM_GET_MSRS for probing only, or we may re-use kvm_arch_get_supported_msr_feature() or kvm_get_one_msr(). I may change in the next version.

2. Limit the usage of 'kvm_pmu_disabled' to be AMD specific in PATCH 5/7.

I guess this might make things more complicated.

Quoted 78 lines

Quoted 4 lines

Additionally, in this scenario — where the user intends to enable a feature but the host cannot support it — normally no warning is emitted by QEMU.

According to the usage of QEMU, may I assume QEMU already prints warning logs for unsupported features? The below is an example.

QEMU 10.2.50 monitor - type 'help' for more information
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.rtm [bit 11]

Quoted 52 lines

Quoted 38 lines

target/i386/kvm/kvm.c +23 -0

+            }
+        } else {
+            /*
+             * Since Linux v5.18, KVM provides a VM-level capability to easily
+             * disable PMUs; however, QEMU has been providing PMU property per
+             * CPU since v1.6. In order to accommodate both, have to configure
+             * the VM-level capability here.
+             *
+             * KVM_PMU_CAP_DISABLE doesn't change the PMU
+             * behavior on Intel platform because current "pmu" property works
+             * as expected.
+             */
+            if (pmu_cap & KVM_PMU_CAP_DISABLE) {
+                ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                        KVM_PMU_CAP_DISABLE);
+                if (ret < 0) {
+                    error_setg_errno(errp, -ret,
+                                     "Failed to set KVM_PMU_CAP_DISABLE");
+                    return ret;
+                }
             }
         }
     }

@@ -3302,6 +3313,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     int ret;
     struct utsname utsname;
     Error *local_err = NULL;
+    g_autofree char *kvm_enable_pmu;
 
     /*
      * Initialize confidential guest (SEV/TDX) context, if required

@@ -3437,6 +3449,21 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
 
+    /*
+     * The enable_pmu parameter is introduced since Linux v5.17,

target/i386/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 3463

Quoted 10 lines

target/i386/kvm/kvm.c +10 -0

+     * give a chance to provide more information about vPMU
+     * enablement.
+     *
+     * The kvm.enable_pmu's permission is 0444. It does not change
+     * until a reload of the KVM module.
+     */
+    if (g_file_get_contents("/sys/module/kvm/parameters/enable_pmu",
+                            &kvm_enable_pmu, NULL, NULL)) {
+        if (*kvm_enable_pmu == 'N') {
+            kvm_pmu_disabled = true;

It’s generally better not to rely on KVM’s internal implementation unless really necessary.

Quoted 5 lines

For example, in the new mediated vPMU framework, even if the KVM module parameter enable_pmu is set, the per-guest kvm->arch.enable_pmu could still be cleared.

In such a case, the logic here might not be correct.

Would the Mediated vPMU set KVM_PMU_CAP_DISABLE to clear per-VM enable_pmu even when the global KVM parameter enable_pmu=N is set?

In this scenario, we plan to rely on KVM_PMU_CAP_DISABLE only when the value of "/sys/module/kvm/parameters/enable_pmu" is not equal to N.

Quoted 7 lines

Can I assume that this will work with Mediated vPMU?

Is there any possibility to follow the current approach before Mediated vPMU is finalized for mainline, and later introduce an incremental change using KVM_GET_MSRS probing? The current approach is straightforward and can work with existing Linux kernel source code.

Apologies for the incorrect statement I made earlier regarding mediated vPMU.

According to the mediated vPMU v6, the only behavior specific to mediated vPMU is that kvm->arch.enable_pmu may be cleared when irqchip_in_kernel() is not true:
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!IS8XQG3Zx84utP2QScNlxp-0H5JAgr89lBb1j2oGVJJop3WMyK6X2I5mlerMPA06wkJy8VFd1x6XGEHc6kWn$

However, this does not imply that mediated vPMU requires any special handling here. In theory, KVM could clear kvm->arch.enable_pmu in the future for other reasons.

Hi Zide,

Quoted 87 lines · Zide

Quoted 57 lines · Dongli Zhang

Hi Zide,

Quoted 3 lines · Zide

On 12/29/2025 11:42 PM, Dongli Zhang wrote:

[snip]

Quoted 31 lines

Quoted 18 lines

static struct kvm_cpuid2 *cpuid_cache;
static struct kvm_cpuid2 *hv_cpuid_cache;

target/i386/kvm/kvm.c +0 -11

@@ -2068,23 +2072,30 @@ int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp)
     if (first) {
         first = false;
 
-        /*
-         * Since Linux v5.18, KVM provides a VM-level capability to easily
-         * disable PMUs; however, QEMU has been providing PMU property per
-         * CPU since v1.6. In order to accommodate both, have to configure
-         * the VM-level capability here.
-         *
-         * KVM_PMU_CAP_DISABLE doesn't change the PMU
-         * behavior on Intel platform because current "pmu" property works
-         * as expected.
-         */
-        if ((pmu_cap & KVM_PMU_CAP_DISABLE) && !X86_CPU(cpu)->enable_pmu) {

target/i386/kvm/kvm.c int kvm_arch_pre_create_vcpu(CPUState *cpu, Error **errp) · line 2078

Quoted 10 lines

target/i386/kvm/kvm.c +4 -6

-            ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
-                                    KVM_PMU_CAP_DISABLE);
-            if (ret < 0) {
-                error_setg_errno(errp, -ret,
-                                 "Failed to set KVM_PMU_CAP_DISABLE");
-                return ret;
+        if (X86_CPU(cpu)->enable_pmu) {
+            if (kvm_pmu_disabled) {
+                warn_report("Failed to enable PMU since "
+                            "KVM's enable_pmu parameter is disabled");

I'm wondering about the intended value of this patch?

Quoted 5 lines

If enable_pmu is true in QEMU but the corresponding KVM parameter is false, then KVM_GET_SUPPORTED_CPUID or KVM_GET_MSRS should be able to tell that the PMU feature is not supported by host.

The logic implemented in this patch seems somewhat redundant.

For Intel, the QEMU userspace can determine if the vPMU is disabled by KVM through the use of KVM_GET_SUPPORTED_CPUID.

However, this approach does not apply to AMD. Unlike Intel, AMD does not rely on CPUID to detect whether PMU is supported. By default, we can assume that PMU is always available, except for the recent PerfMonV2 feature.

The main objective of this PATCH 4/7 is to introduce the variable 'kvm_pmu_disabled', which will be reused in PATCH 5/7 to skip any PMU initialization if the parameter is set to 'N'.

Review

Quoted 10 lines

(quoted diff) +10 -0

+static void kvm_init_pmu_info(struct kvm_cpuid2 *cpuid, X86CPU *cpu)
+{
+    CPUX86State *env = &cpu->env;
+
+    /*
+     * The PMU virtualization is disabled by kvm.enable_pmu=N.
+     */
+    if (kvm_pmu_disabled) {
+        return;
+    }

Thanks for explanation.

Quoted 6 lines

The 'kvm_pmu_disabled' variable is used to differentiate between the following two scenarios on AMD:

(1) A newer KVM with KVM_PMU_CAP_DISABLE support, but explicitly disabled via the KVM parameter ('N').

Quoted 7 lines

(2) An older KVM without KVM_CAP_PMU_CAPABILITY support.

In both cases, the call to KVM_CAP_PMU_CAPABILITY extension support check may return 0.

By reading the file "/sys/module/kvm/parameters/enable_pmu", we can distinguish between these two scenarios.

Quoted 6 lines

As described in PATCH 1/7, without issuing KVM_PMU_CAP_DISABLE, KVM has no way to know that userspace does not intend to enable vPMU in AMD platforms, and therefore does not fault guest accesses to PMU MSRs.

My understanding is that the issue being addressed here is basically the
opposite: QEMU does not know that vPMU is disabled by KVM.

Exactly.

Otherwise, QEMU issues unwanted MSR writes for every vCPU during QEMU reset.

Quoted 16 lines

IIUC, one difference between Intel and AMD is that AMD lacks a CPUID leaf to indicate the availability of PMU version 1. But Intel potentially could be in the same situation that KVM advertises PMU availability but it's not actually supported. (e.g. kvm->arch.enable_pmu is false while modules parameter enable_pmu is true).

From the guest’s point of view, it probes PMU MSRs to determine whether PMU support is present and it's fine in this situation.

In userspace, QEMU may issue KVM_SET_MSRS / KVM_GET_MSRS to KVM without knowing that vPMU has been disabled by KVM. I think these IOCTLs should not fail, since KVM states that “Userspace is allowed to read MSRs, and write ‘0’ to MSRs, that KVM advertises to userspace, even if an MSR isn’t fully supported.”

Quoted 5 lines

My current understanding is that AMD should be fine even without kvm_pmu_disabled, but I may be missing some context here.

The bottom line is this patch doesn't handle the cases that KVM still could disable vPMU support even if enable_pmu is true.

Yes. There are still unwanted PMU MSR writes from QEMU. This just seems odd.

The concern with unwanted MSR writes was initially raised by Maksim Davydov:

https://lore.kernel.org/qemu-devel/[email protected]/

As shown below on the v6.0 KVM hypervisor (AMD), while there are no errors from QEMU, numerous annoying warnings are generated. (If I recall correctly, this can also be triggered from the VM itself.)

However, here the logs are not only due to vcpu0, but indeed every vcpu.

[  280.802976] kvm_set_msr_common: 1910 callbacks suppressed
[  280.802981] kvm [18411]: vcpu0, guest rIP: 0xffffffffa4c97844 disabled
perfctr wrmsr: 0xc0010007 data 0xffff
[  295.345747] kvm [18411]: vcpu0, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010004 data 0x0
[  295.355379] kvm [18411]: vcpu0, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010005 data 0x0
[  295.364997] kvm [18411]: vcpu0, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010006 data 0x0
[  295.374618] kvm [18411]: vcpu0, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010007 data 0x0
[  295.385048] kvm [18411]: vcpu1, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010004 data 0x0
[  295.394694] kvm [18411]: vcpu1, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010005 data 0x0
[  295.404317] kvm [18411]: vcpu1, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010006 data 0x0
[  295.413928] kvm [18411]: vcpu1, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010007 data 0x0
[  295.424319] kvm [18411]: vcpu2, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010004 data 0x0
[  295.433963] kvm [18411]: vcpu2, guest rIP: 0xfff0 disabled perfctr wrmsr:
0xc0010005 data 0x0
[  301.966571] kvm_set_msr_common: 1910 callbacks suppressed
[  301.966577] kvm [18411]: vcpu0, guest rIP: 0xffffffff8ac97844 disabled

perfctr wrmsr: 0xc0010007 data 0xffff

Quoted 106 lines

Quoted 6 lines

As you mentioned, another approach would be to use KVM_GET_MSRS to specifically probe for AMD during QEMU initialization. In this case, we can set 'kvm_pmu_disabled' to true if reading the AMD PMU MSR registers fails.

Quoted 8 lines

To implement this, we may need to:

1. Turn this patch to be AMD specific by probing the AMD PMU registers during initialization. We may need go create a new function in QEMU to use KVM_GET_MSRS for probing only, or we may re-use kvm_arch_get_supported_msr_feature() or kvm_get_one_msr(). I may change in the next version.

2. Limit the usage of 'kvm_pmu_disabled' to be AMD specific in PATCH 5/7.

I guess this might make things more complicated.

Quoted 78 lines

Quoted 4 lines

Additionally, in this scenario — where the user intends to enable a feature but the host cannot support it — normally no warning is emitted by QEMU.

According to the usage of QEMU, may I assume QEMU already prints warning logs for unsupported features? The below is an example.

QEMU 10.2.50 monitor - type 'help' for more information
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID[eax=07h,ecx=00h].EBX.rtm [bit 11]

Quoted 52 lines

Quoted 38 lines

target/i386/kvm/kvm.c +23 -0

+            }
+        } else {
+            /*
+             * Since Linux v5.18, KVM provides a VM-level capability to easily
+             * disable PMUs; however, QEMU has been providing PMU property per
+             * CPU since v1.6. In order to accommodate both, have to configure
+             * the VM-level capability here.
+             *
+             * KVM_PMU_CAP_DISABLE doesn't change the PMU
+             * behavior on Intel platform because current "pmu" property works
+             * as expected.
+             */
+            if (pmu_cap & KVM_PMU_CAP_DISABLE) {
+                ret = kvm_vm_enable_cap(kvm_state, KVM_CAP_PMU_CAPABILITY, 0,
+                                        KVM_PMU_CAP_DISABLE);
+                if (ret < 0) {
+                    error_setg_errno(errp, -ret,
+                                     "Failed to set KVM_PMU_CAP_DISABLE");
+                    return ret;
+                }
             }
         }
     }

@@ -3302,6 +3313,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     int ret;
     struct utsname utsname;
     Error *local_err = NULL;
+    g_autofree char *kvm_enable_pmu;
 
     /*
      * Initialize confidential guest (SEV/TDX) context, if required

@@ -3437,6 +3449,21 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     pmu_cap = kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY);
 
+    /*
+     * The enable_pmu parameter is introduced since Linux v5.17,

target/i386/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 3463

Quoted 10 lines

target/i386/kvm/kvm.c +10 -0

+     * give a chance to provide more information about vPMU
+     * enablement.
+     *
+     * The kvm.enable_pmu's permission is 0444. It does not change
+     * until a reload of the KVM module.
+     */
+    if (g_file_get_contents("/sys/module/kvm/parameters/enable_pmu",
+                            &kvm_enable_pmu, NULL, NULL)) {
+        if (*kvm_enable_pmu == 'N') {
+            kvm_pmu_disabled = true;

It’s generally better not to rely on KVM’s internal implementation unless really necessary.

Quoted 5 lines

For example, in the new mediated vPMU framework, even if the KVM module parameter enable_pmu is set, the per-guest kvm->arch.enable_pmu could still be cleared.

In such a case, the logic here might not be correct.

Would the Mediated vPMU set KVM_PMU_CAP_DISABLE to clear per-VM enable_pmu even when the global KVM parameter enable_pmu=N is set?

In this scenario, we plan to rely on KVM_PMU_CAP_DISABLE only when the value of "/sys/module/kvm/parameters/enable_pmu" is not equal to N.

Quoted 7 lines

Can I assume that this will work with Mediated vPMU?

Is there any possibility to follow the current approach before Mediated vPMU is finalized for mainline, and later introduce an incremental change using KVM_GET_MSRS probing? The current approach is straightforward and can work with existing Linux kernel source code.

Apologies for the incorrect statement I made earlier regarding mediated vPMU.

Quoted 9 lines

According to the mediated vPMU v6, the only behavior specific to mediated vPMU is that kvm->arch.enable_pmu may be cleared when irqchip_in_kernel() is not true:
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!IS8XQG3Zx84utP2QScNlxp-0H5JAgr89lBb1j2oGVJJop3WMyK6X2I5mlerMPA06wkJy8VFd1x6XGEHc6kWn$

However, this does not imply that mediated vPMU requires any special handling here. In theory, KVM could clear kvm->arch.enable_pmu in the future for other reasons.

While "KVM could clear kvm->arch.enable_pmu in the future," I don't think KVM may set kvm->arch.enable_pmu if the global enable_pmu is set to 'N'.

Taking Intel VMX EPT as an example, once "/sys/module/kvm_intel/parameters/ept" is globally disabled, there's no way within KVM software to enable it for any guest VM **after** 'ept' is set to N.

Similarly, "/sys/module/kvm/parameters/enable_pmu=N" indicates that this KVM host will not support PMU virtualization in any way. Therefore, there should be no way to enable vPMU for any guest VM if the global parameter is set to 'N'. Here we read from ths parameter only during QEMU initialization.

That's why I believe it's reliable to trust the setting when "/sys/module/kvm/parameters/enable_pmu=N".

In this way, we can avoid many unnecessary MSR writes, especially in cases where a VM has 300+ vCPUs, even though these may be equivalent to NOPs with optimizations in more recent KVM versions.

The objective isn't to improve performance. Minimizing the number of unwanted MSR writes from QEMU reduces the chances of failure (e.g., due to any QEMU software bug). We can simply avoid those unwanted MSR/NOPs by reading from a KVM parameter.

From a user's perspective, this just seems odd. "/sys/module/kvm/parameters/enable_pmu=N" is a reliable setting. If there's a configuration mismatch between QEMU and KVM, a warning could alert the user.

I can remove this patch, along with the 'kvm_pmu_disabled' variable.

Thank you very much!

Dongli Zhang

PMU MSRs are set by QEMU only at levels >= KVM_PUT_RESET_STATE, excluding runtime. Therefore, updating these MSRs without stopping events should be acceptable.

In addition, KVM creates kernel perf events with host mode excluded (exclude_host = 1). While the events remain active, they don't increment the counter during QEMU vCPU userspace mode.

Finally, The kvm_put_msrs() sets the MSRs using KVM_SET_MSRS. The x86 KVM processes these MSRs one by one in a loop, only saving the config and triggering the KVM_REQ_PMU request. This approach does not immediately stop the event before updating PMC. This approach is true since Linux kernel commit 68fb4757e867 ("KVM: x86/pmu: Defer reprogram_counter() to kvm_pmu_handle_event"), that is, v6.2.

No Fixed tag is going to be added for the commit 0d89436786b0 ("kvm:
migrate vPMU state"), because this isn't a bugfix.

Signed-off-by Dongli Zhang <[email protected]>

Reviewed-by Zhao Liu <[email protected]>

Reviewed-by Dapeng Mi <[email protected]>

---
Changed since v3:
- Re-order reasons in commit messages.
- Mention KVM's commit 68fb4757e867 (v6.2).
- Keep Zhao's review as there isn't code change.
Changed since v6:
- Add Reviewed-by from Dapeng Mi.

 target/i386/kvm/kvm.c | 9 ---------
 1 file changed, 9 deletions(-)

target/i386/kvm/kvm.c +0 -9

index 99837048b8..742dc6ac0d 100644
@@ -4222,13 +4222,6 @@ static int kvm_put_msrs(X86CPU *cpu, KvmPutState level)
         }
 
         if ((IS_INTEL_CPU(env) || IS_ZHAOXIN_CPU(env)) && pmu_version > 0) {
-            if (pmu_version > 1) {
-                /* Stop the counter.  */
-                kvm_msr_entry_add(cpu, MSR_CORE_PERF_FIXED_CTR_CTRL, 0);
-                kvm_msr_entry_add(cpu, MSR_CORE_PERF_GLOBAL_CTRL, 0);
-            }
-
-            /* Set the counter values.  */
             for (i = 0; i < num_pmu_fixed_counters; i++) {
                 kvm_msr_entry_add(cpu, MSR_CORE_PERF_FIXED_CTR0 + i,
                                   env->msr_fixed_counters[i]);

@@ -4244,8 +4237,6 @@ static int kvm_put_msrs(X86CPU *cpu, KvmPutState level)
                                   env->msr_global_status);
                 kvm_msr_entry_add(cpu, MSR_CORE_PERF_GLOBAL_OVF_CTRL,
                                   env->msr_global_ovf_ctrl);
-
-                /* Now start the PMU.  */
                 kvm_msr_entry_add(cpu, MSR_CORE_PERF_FIXED_CTR_CTRL,
                                   env->msr_fixed_ctr_ctrl);
                 kvm_msr_entry_add(cpu, MSR_CORE_PERF_GLOBAL_CTRL,

Quoted 3 lines · Dongli Zhang

PMU MSRs are set by QEMU only at levels >= KVM_PUT_RESET_STATE, excluding runtime. Therefore, updating these MSRs without stopping events should be acceptable.

It seems preferable to keep the existing logic. The sequence of disabling -> setting new counters -> re-enabling is complete and reasonable. Re-enabling the PMU implicitly tell KVM to do whatever actions are needed to make the new counters take effect.

If the purpose of this patch to improve performance, given that this is a non-critical path, trading this clear and robust logic for a minor performance gain does not seem necessary.

Quoted 10 lines

In addition, KVM creates kernel perf events with host mode excluded (exclude_host = 1). While the events remain active, they don't increment the counter during QEMU vCPU userspace mode.

Finally, The kvm_put_msrs() sets the MSRs using KVM_SET_MSRS. The x86 KVM processes these MSRs one by one in a loop, only saving the config and triggering the KVM_REQ_PMU request. This approach does not immediately stop the event before updating PMC. This approach is true since Linux kernel commit 68fb4757e867 ("KVM: x86/pmu: Defer reprogram_counter() to kvm_pmu_handle_event"), that is, v6.2.

This seems to assume KVM's internal behavior. While that is true today (and possibly in the future), it’s not necessary for QEMU to make such assumptions, as that could unnecessarily limit KVM’s flexibility to change its behavior later.

Hi Zide,

Quoted 28 lines · Zide

Quoted 3 lines · Dongli Zhang

PMU MSRs are set by QEMU only at levels >= KVM_PUT_RESET_STATE, excluding runtime. Therefore, updating these MSRs without stopping events should be acceptable.

It seems preferable to keep the existing logic. The sequence of disabling -> setting new counters -> re-enabling is complete and reasonable. Re-enabling the PMU implicitly tell KVM to do whatever actions are needed to make the new counters take effect.

If the purpose of this patch to improve performance, given that this is a non-critical path, trading this clear and robust logic for a minor performance gain does not seem necessary.

Quoted 10 lines

In addition, KVM creates kernel perf events with host mode excluded (exclude_host = 1). While the events remain active, they don't increment the counter during QEMU vCPU userspace mode.

Finally, The kvm_put_msrs() sets the MSRs using KVM_SET_MSRS. The x86 KVM processes these MSRs one by one in a loop, only saving the config and triggering the KVM_REQ_PMU request. This approach does not immediately stop the event before updating PMC. This approach is true since Linux kernel commit 68fb4757e867 ("KVM: x86/pmu: Defer reprogram_counter() to kvm_pmu_handle_event"), that is, v6.2.

Quoted 5 lines

This seems to assume KVM's internal behavior. While that is true today (and possibly in the future), it’s not necessary for QEMU to make such assumptions, as that could unnecessarily limit KVM’s flexibility to change its behavior later.

To "assume KVM's internal behavior" is only one of the two reasons. The first reason is that QEMU controls the state of the vCPU to ensure this action only occurs when "levels >= KVM_PUT_RESET_STATE."

Thanks to "(level >= KVM_PUT_RESET_STATE)", QEMU is able to avoid unnecessary updates to many MSR registers during runtime.

The main objective is to sync the implementation for Intel and AMD.

Both MSR_CORE_PERF_FIXED_CTR_CTRL and MSR_CORE_PERF_GLOBAL_CTRL are reset to zero only in the case where "has_pmu_version > 1." Otherwise, we may need to reset the MSR_P6_PERFCTR_N registers before writing to the counter registers. Without PATCH 7/7, an additional patch will be required to fix the workflow for handling PMU registers to reset control registers before counter registers.

If the plan is to maintain the current logic, we may need to adjust the logic for the AMD registers as well. In PATCH 6/7, we never reset global registers before writing to control and counter registers.

Would you mine sharing your thoughts on it?

Thank you very much!

Dongli Zhang

Quoted 47 lines · Dongli Zhang

Hi Zide,

Quoted 28 lines · Zide

Quoted 3 lines · Dongli Zhang

PMU MSRs are set by QEMU only at levels >= KVM_PUT_RESET_STATE, excluding runtime. Therefore, updating these MSRs without stopping events should be acceptable.

It seems preferable to keep the existing logic. The sequence of disabling -> setting new counters -> re-enabling is complete and reasonable. Re-enabling the PMU implicitly tell KVM to do whatever actions are needed to make the new counters take effect.

If the purpose of this patch to improve performance, given that this is a non-critical path, trading this clear and robust logic for a minor performance gain does not seem necessary.

Quoted 10 lines

In addition, KVM creates kernel perf events with host mode excluded (exclude_host = 1). While the events remain active, they don't increment the counter during QEMU vCPU userspace mode.

Finally, The kvm_put_msrs() sets the MSRs using KVM_SET_MSRS. The x86 KVM processes these MSRs one by one in a loop, only saving the config and triggering the KVM_REQ_PMU request. This approach does not immediately stop the event before updating PMC. This approach is true since Linux kernel commit 68fb4757e867 ("KVM: x86/pmu: Defer reprogram_counter() to kvm_pmu_handle_event"), that is, v6.2.

Quoted 5 lines

This seems to assume KVM's internal behavior. While that is true today (and possibly in the future), it’s not necessary for QEMU to make such assumptions, as that could unnecessarily limit KVM’s flexibility to change its behavior later.

To "assume KVM's internal behavior" is only one of the two reasons. The first reason is that QEMU controls the state of the vCPU to ensure this action only occurs when "levels >= KVM_PUT_RESET_STATE."

Thanks to "(level >= KVM_PUT_RESET_STATE)", QEMU is able to avoid unnecessary updates to many MSR registers during runtime.

The main objective is to sync the implementation for Intel and AMD.

Quoted 5 lines

Both MSR_CORE_PERF_FIXED_CTR_CTRL and MSR_CORE_PERF_GLOBAL_CTRL are reset to zero only in the case where "has_pmu_version > 1." Otherwise, we may need to reset the MSR_P6_PERFCTR_N registers before writing to the counter registers. Without PATCH 7/7, an additional patch will be required to fix the workflow for handling PMU registers to reset control registers before counter registers.

I might be missing something here, but since this is not for runtime, I don’t quite understand the need to reset the control registers.

Quoted 6 lines

If the plan is to maintain the current logic, we may need to adjust the logic for the AMD registers as well. In PATCH 6/7, we never reset global registers before writing to control and counter registers.

Would you mine sharing your thoughts on it?

Personally, I would lean towards keeping the current logic and instead adjusting patch 6/7 to reset the global registers. This is just my view, and please don’t feel obligated to follow it.