REGRESSION on linux-next (next-20250919)
Hello Sean,
Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.
This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.
Since the version next-20250919 [2], we are seeing the following regression
````````````````````````````````````````````````````````````````````````````````` <4>[ 10.973827] ------------[ cut here ]------------ <4>[ 10.973841] WARNING: arch/x86/events/core.c:3089 at perf_get_x86_pmu_capability+0xd/0xc0, CPU#15: (udev-worker)/386 ... <4>[ 10.974028] Call Trace: <4>[ 10.974030] <TASK> <4>[ 10.974033] ? kvm_init_pmu_capability+0x2b/0x190 [kvm] <4>[ 10.974154] kvm_x86_vendor_init+0x1b0/0x1a40 [kvm] <4>[ 10.974248] vmx_init+0xdb/0x260 [kvm_intel] <4>[ 10.974278] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974296] vt_init+0x12/0x9d0 [kvm_intel] <4>[ 10.974309] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974322] do_one_initcall+0x60/0x3f0 <4>[ 10.974335] do_init_module+0x97/0x2b0 <4>[ 10.974345] load_module+0x2d08/0x2e30 <4>[ 10.974349] ? __kernel_read+0x158/0x2f0 <4>[ 10.974370] ? kernel_read_file+0x2b1/0x320 <4>[ 10.974381] init_module_from_file+0x96/0xe0 <4>[ 10.974384] ? init_module_from_file+0x96/0xe0 <4>[ 10.974399] idempotent_init_module+0x117/0x330 <4>[ 10.974415] __x64_sys_finit_module+0x73/0xe0 ... `````````````````````````````````````````````````````````````````````````````````
Details log can be found in [3].
After bisecting the tree, the following patch [4] seems to be the first "bad" commit
````````````````````````````````````````````````````````````````````````````````````````````````````````` From 51f34b1e650fc5843530266cea4341750bd1ae37 Mon Sep 17 00:00:00 2001 From: Sean Christopherson <[email protected]> Date: Wed, 6 Aug 2025 12:56:39 -0700 Subject: KVM: x86/pmu: Snapshot host (i.e. perf's) reported PMU capabilities Take a snapshot of the unadulterated PMU capabilities provided by perf so that KVM can compare guest vPMU capabilities against hardware capabilities when determining whether or not to intercept PMU MSRs (and RDPMC). `````````````````````````````````````````````````````````````````````````````````````````````````````````
We also verified that if we revert the patch the issue is not seen.
Could you please check why the patch causes this regression and provide a fix if necessary?
Thank you.
Regards
Chaitanya
[1]
https://intel-gfx-ci.01.org/tree/linux-next/combined-alt.html?
[2] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250919
[3]
https://intel-gfx-ci.01.org/tree/linux-next/next-20250919/bat-arlh-2/boot0.txt
[4] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250919&id=51f34b1e650fc5843530266cea4341750bd1ae37
Quoted 53 lines · Chaitanya Kumar
Hello Sean,
Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.
This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.
Since the version next-20250919 [2], we are seeing the following regression
````````````````````````````````````````````````````````````````````````````````` <4>[ 10.973827] ------------[ cut here ]------------ <4>[ 10.973841] WARNING: arch/x86/events/core.c:3089 at perf_get_x86_pmu_capability+0xd/0xc0, CPU#15: (udev-worker)/386 ... <4>[ 10.974028] Call Trace: <4>[ 10.974030] <TASK> <4>[ 10.974033] ? kvm_init_pmu_capability+0x2b/0x190 [kvm] <4>[ 10.974154] kvm_x86_vendor_init+0x1b0/0x1a40 [kvm] <4>[ 10.974248] vmx_init+0xdb/0x260 [kvm_intel] <4>[ 10.974278] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974296] vt_init+0x12/0x9d0 [kvm_intel] <4>[ 10.974309] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974322] do_one_initcall+0x60/0x3f0 <4>[ 10.974335] do_init_module+0x97/0x2b0 <4>[ 10.974345] load_module+0x2d08/0x2e30 <4>[ 10.974349] ? __kernel_read+0x158/0x2f0 <4>[ 10.974370] ? kernel_read_file+0x2b1/0x320 <4>[ 10.974381] init_module_from_file+0x96/0xe0 <4>[ 10.974384] ? init_module_from_file+0x96/0xe0 <4>[ 10.974399] idempotent_init_module+0x117/0x330 <4>[ 10.974415] __x64_sys_finit_module+0x73/0xe0 ... `````````````````````````````````````````````````````````````````````````````````Details log can be found in [3].
After bisecting the tree, the following patch [4] seems to be the first "bad" commit
````````````````````````````````````````````````````````````````````````````````````````````````````````` From 51f34b1e650fc5843530266cea4341750bd1ae37 Mon Sep 17 00:00:00 2001 From: Sean Christopherson <[email protected]> Date: Wed, 6 Aug 2025 12:56:39 -0700 Subject: KVM: x86/pmu: Snapshot host (i.e. perf's) reported PMU capabilities Take a snapshot of the unadulterated PMU capabilities provided by perf so that KVM can compare guest vPMU capabilities against hardware capabilities when determining whether or not to intercept PMU MSRs (and RDPMC). `````````````````````````````````````````````````````````````````````````````````````````````````````````
Quoted 4 lines
We also verified that if we revert the patch the issue is not seen.
Could you please check why the patch causes this regression and provide a fix if necessary?
Hi Chaitanya,
I suppose you found this warning on a hybrid client platform, right? It looks the warning is triggered by the below WARN_ON_ONCE() in perf_get_x86_pmu_capability() function.
if (WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) || !x86_pmu_initialized()) {
memset(cap, 0, sizeof(*cap));
return;
}
The below change should fix it (just building, not test it). I would run a full scope vPMU test after I come back from China national day's holiday. Thanks.
arch/x86/kvm/pmu.c +0 -0
@@ -108,8 +108,6 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops)
bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS;
- perf_get_x86_pmu_capability(&kvm_host_pmu);
-
/*
* Hybrid PMUs don't play nice with virtualization without careful
* configuration by userspace, and KVM's APIs for reporting supported @@ -120,6 +118,8 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) enable_pmu = false;
if (enable_pmu) {
+ perf_get_x86_pmu_capability(&kvm_host_pmu);
+
/*
* WARN if perf did NOT disable hardware PMU if the number of
* architecturally required GP counters aren't present, i.e. if
Quoted 16 lines
Thank you.
Regards
Chaitanya
[1]
https://intel-gfx-ci.01.org/tree/linux-next/combined-alt.html?
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250919
[3]
https://intel-gfx-ci.01.org/tree/linux-next/next-20250919/bat-arlh-2/boot0.txt
[4]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20250919&id=51f34b1e650fc5843530266cea4341750bd1ae37
Sean Christopherson
maintainerQuoted 66 lines · Dapeng Mi
Quoted 53 lines · Chaitanya Kumar
Hello Sean,
Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.
This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.
Since the version next-20250919 [2], we are seeing the following regression
````````````````````````````````````````````````````````````````````````````````` <4>[ 10.973827] ------------[ cut here ]------------ <4>[ 10.973841] WARNING: arch/x86/events/core.c:3089 at perf_get_x86_pmu_capability+0xd/0xc0, CPU#15: (udev-worker)/386 ... <4>[ 10.974028] Call Trace: <4>[ 10.974030] <TASK> <4>[ 10.974033] ? kvm_init_pmu_capability+0x2b/0x190 [kvm] <4>[ 10.974154] kvm_x86_vendor_init+0x1b0/0x1a40 [kvm] <4>[ 10.974248] vmx_init+0xdb/0x260 [kvm_intel] <4>[ 10.974278] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974296] vt_init+0x12/0x9d0 [kvm_intel] <4>[ 10.974309] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974322] do_one_initcall+0x60/0x3f0 <4>[ 10.974335] do_init_module+0x97/0x2b0 <4>[ 10.974345] load_module+0x2d08/0x2e30 <4>[ 10.974349] ? __kernel_read+0x158/0x2f0 <4>[ 10.974370] ? kernel_read_file+0x2b1/0x320 <4>[ 10.974381] init_module_from_file+0x96/0xe0 <4>[ 10.974384] ? init_module_from_file+0x96/0xe0 <4>[ 10.974399] idempotent_init_module+0x117/0x330 <4>[ 10.974415] __x64_sys_finit_module+0x73/0xe0 ... `````````````````````````````````````````````````````````````````````````````````Details log can be found in [3].
After bisecting the tree, the following patch [4] seems to be the first "bad" commit
````````````````````````````````````````````````````````````````````````````````````````````````````````` From 51f34b1e650fc5843530266cea4341750bd1ae37 Mon Sep 17 00:00:00 2001 From: Sean Christopherson <[email protected]> Date: Wed, 6 Aug 2025 12:56:39 -0700 Subject: KVM: x86/pmu: Snapshot host (i.e. perf's) reported PMU capabilities Take a snapshot of the unadulterated PMU capabilities provided by perf so that KVM can compare guest vPMU capabilities against hardware capabilities when determining whether or not to intercept PMU MSRs (and RDPMC). `````````````````````````````````````````````````````````````````````````````````````````````````````````Quoted 4 lines
We also verified that if we revert the patch the issue is not seen.
Could you please check why the patch causes this regression and provide a fix if necessary?
Hi Chaitanya,
I suppose you found this warning on a hybrid client platform, right? It looks the warning is triggered by the below WARN_ON_ONCE() in
perf_get_x86_pmu_capability()function.
Quoted 8 lines
if (WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) || !x86_pmu_initialized()) {
memset(cap, 0, sizeof(*cap));
return;
}The below change should fix it (just building, not test it). I would run a full scope vPMU test after I come back from China national day's holiday.
I have access to a hybrid system, I'll also double check there (though I'm 99.9% certain you've got it right).
Quoted 16 lines
Thanks.
arch/x86/kvm/pmu.c +0
-2@@ -108,8 +108,6 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported
arch/x86/kvm/pmu.c void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) · line 125
Quoted 8 lines
arch/x86/kvm/pmu.c +2
-0@@ -120,6 +118,8 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) enable_pmu = false; if (enable_pmu) { + perf_get_x86_pmu_capability(&kvm_host_pmu); + /** WARN if perf did NOT disable hardware PMU if the number of * architecturally required GP counters aren't present, i.e. if
If we go this route, then the !enable_pmu path should explicitly zero kvm_host_pmu so that the behavior is consistent userspace loads kvm.ko with enable_pmu=0, versus enable_pmu being cleared because of lack of support.
if (!enable_pmu) {
memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu));
memset(&kvm_pmu_cap, 0, sizeof(kvm_pmu_cap));
return;
}The alternative would be keep kvm_host_pmu valid at all times for !HYBRID, which is what I intended with the bad patch, but that too would lead to inconsistent behavior. So I think it makes sense to go with Dapeng's approach; we can always revisit this if some future thing in KVM _needs_ kvm_host_pmu even with enable_pmu=0.
if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu = false;
memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu));
} else {
perf_get_x86_pmu_capability(&kvm_host_pmu);
}Quoted 119 lines · Sean Christopherson
Quoted 66 lines · Dapeng Mi
Quoted 53 lines · Chaitanya Kumar
Hello Sean,
Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.
This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.
Since the version next-20250919 [2], we are seeing the following regression
````````````````````````````````````````````````````````````````````````````````` <4>[ 10.973827] ------------[ cut here ]------------ <4>[ 10.973841] WARNING: arch/x86/events/core.c:3089 at perf_get_x86_pmu_capability+0xd/0xc0, CPU#15: (udev-worker)/386 ... <4>[ 10.974028] Call Trace: <4>[ 10.974030] <TASK> <4>[ 10.974033] ? kvm_init_pmu_capability+0x2b/0x190 [kvm] <4>[ 10.974154] kvm_x86_vendor_init+0x1b0/0x1a40 [kvm] <4>[ 10.974248] vmx_init+0xdb/0x260 [kvm_intel] <4>[ 10.974278] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974296] vt_init+0x12/0x9d0 [kvm_intel] <4>[ 10.974309] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974322] do_one_initcall+0x60/0x3f0 <4>[ 10.974335] do_init_module+0x97/0x2b0 <4>[ 10.974345] load_module+0x2d08/0x2e30 <4>[ 10.974349] ? __kernel_read+0x158/0x2f0 <4>[ 10.974370] ? kernel_read_file+0x2b1/0x320 <4>[ 10.974381] init_module_from_file+0x96/0xe0 <4>[ 10.974384] ? init_module_from_file+0x96/0xe0 <4>[ 10.974399] idempotent_init_module+0x117/0x330 <4>[ 10.974415] __x64_sys_finit_module+0x73/0xe0 ... `````````````````````````````````````````````````````````````````````````````````Details log can be found in [3].
After bisecting the tree, the following patch [4] seems to be the first "bad" commit
````````````````````````````````````````````````````````````````````````````````````````````````````````` From 51f34b1e650fc5843530266cea4341750bd1ae37 Mon Sep 17 00:00:00 2001 From: Sean Christopherson <[email protected]> Date: Wed, 6 Aug 2025 12:56:39 -0700 Subject: KVM: x86/pmu: Snapshot host (i.e. perf's) reported PMU capabilities Take a snapshot of the unadulterated PMU capabilities provided by perf so that KVM can compare guest vPMU capabilities against hardware capabilities when determining whether or not to intercept PMU MSRs (and RDPMC). `````````````````````````````````````````````````````````````````````````````````````````````````````````Quoted 4 lines
We also verified that if we revert the patch the issue is not seen.
Could you please check why the patch causes this regression and provide a fix if necessary?
Hi Chaitanya,
I suppose you found this warning on a hybrid client platform, right? It looks the warning is triggered by the below WARN_ON_ONCE() in
perf_get_x86_pmu_capability()function.Quoted 8 lines
if (WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) || !x86_pmu_initialized()) {
memset(cap, 0, sizeof(*cap));
return;
}The below change should fix it (just building, not test it). I would run a full scope vPMU test after I come back from China national day's holiday.
I have access to a hybrid system, I'll also double check there (though I'm 99.9% certain you've got it right).
Quoted 16 lines
Thanks.
arch/x86/kvm/pmu.c +0
-2@@ -108,8 +108,6 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported arch/x86/kvm/pmu.c void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) · line 125
Quoted 8 lines
arch/x86/kvm/pmu.c +2
-0@@ -120,6 +118,8 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) enable_pmu = false; if (enable_pmu) { + perf_get_x86_pmu_capability(&kvm_host_pmu); + /** WARN if perf did NOT disable hardware PMU if the number of * architecturally required GP counters aren't present, i.e. if If we go this route, then the !enable_pmu path should explicitly zero
kvm_host_pmuso that the behavior is consistent userspace loads kvm.ko with enable_pmu=0, versusenable_pmubeing cleared because of lack of support.if (!enable_pmu) { memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); memset(&kvm_pmu_cap, 0, sizeof(kvm_pmu_cap)); return; }The alternative would be keep
kvm_host_pmuvalid at all times for !HYBRID, which is what I intended with the bad patch, but that too would lead to inconsistent behavior. So I think it makes sense to go with Dapeng's approach; we can always revisit this if some future thing in KVM _needs_kvm_host_pmueven with enable_pmu=0.
Quoted 6 lines
if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu = false;
memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu));
} else {
perf_get_x86_pmu_capability(&kvm_host_pmu);
}Thank you for your responses.
Following change fixes the issue for us.
arch/x86/kvm/pmu.c +0 -2
@@ -108,16 +108,18 @@ void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops)bool is_intel = boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ;int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; -perf_get_x86_pmu_capability (&kvm_host_pmu); -/* * Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting
supported
* vPMU features do not account for hybrid PMUs. Disable vPMU
support
* for hybrid PMUs until KVM gains a way to let userspace opt-in. */
- if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU))
+ if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu = false;+ memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu));
+ } else {
+ perf_get_x86_pmu_capability(&kvm_host_pmu);
+ }
Regards
Chaitanya
Hi,
Quoted 23 lines · Chaitanya Kumar
Thank you for your responses.
Following change fixes the issue for us.
arch/x86/kvm/pmu.c +0
-2@@ -108,16 +108,18 @@ void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported
* vPMU features do not account for hybrid PMUs. Disable vPMU
support
* for hybrid PMUs until KVM gains a way to let userspace opt-in.
Quoted 8 lines
*/
- if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) + if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu= false;
+memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); + } else {
+perf_get_x86_pmu_capability(&kvm_host_pmu); + }
Can we expect a formal patch soon?
Regards
Chaitanya
Quoted 27 lines · Chaitanya Kumar
Hi,
Quoted 24 lines · Chaitanya Kumar
Thank you for your responses.
Following change fixes the issue for us.
arch/x86/kvm/pmu.c +0
-2@@ -108,16 +108,18 @@ void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported
* vPMU features do not account for hybrid PMUs. Disable vPMU
support
* for hybrid PMUs until KVM gains a way to let userspace opt-in. */
Quoted 8 lines
Quoted 7 lines
- if (
cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) + if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu= false;
+memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); + } else {
+perf_get_x86_pmu_capability(&kvm_host_pmu); + }Can we expect a formal patch soon?
I'd like to post a patch to fix this tomorrow if Sean has no bandwidth on this. Thanks.
Quoted 4 lines
Regards
Chaitanya
Sean Christopherson
maintainerQuoted 38 lines · Dapeng Mi
Quoted 27 lines · Chaitanya Kumar
Hi,
Quoted 24 lines · Chaitanya Kumar
Thank you for your responses.
Following change fixes the issue for us.
arch/x86/kvm/pmu.c +0
-2@@ -108,16 +108,18 @@ void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported
* vPMU features do not account for hybrid PMUs. Disable vPMU
support
* for hybrid PMUs until KVM gains a way to let userspace opt-in. */Quoted 8 lines
Quoted 7 lines
- if (
cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) + if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu= false;
+memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); + } else {
+perf_get_x86_pmu_capability(&kvm_host_pmu); + }Can we expect a formal patch soon?
Quoted 2 lines
I'd like to post a patch to fix this tomorrow if Sean has no bandwidth on this. Thanks.
Sorry, my bad, I was waiting for you to post a patch, but that wasn't at all clear. So yeah, go ahead and post one :-)
Quoted 37 lines · Sean Christopherson
Quoted 36 lines · Dapeng Mi
Quoted 35 lines · Chaitanya Kumar
Hi,
Quoted 23 lines · Chaitanya Kumar
Thank you for your responses.
Following change fixes the issue for us.
arch/x86/kvm/pmu.c +0
-2@@ -108,16 +108,18 @@ void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported
* vPMU features do not account for hybrid PMUs. Disable vPMU
support
* for hybrid PMUs until KVM gains a way to let userspace opt-in.Quoted 8 lines
*/
- if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) + if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu= false;
+memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); + } else {
+perf_get_x86_pmu_capability(&kvm_host_pmu); + }Can we expect a formal patch soon?
Quoted 4 lines
Quoted 2 lines
I'd like to post a patch to fix this tomorrow if Sean has no bandwidth on this. Thanks.
Sorry, my bad, I was waiting for you to post a patch, but that wasn't at all clear. So yeah, go ahead and post one :-)
Sure. Would post it now.
Quoted 115 lines · Sean Christopherson
Quoted 64 lines · Dapeng Mi
Quoted 53 lines · Chaitanya Kumar
Hello Sean,
Hope you are doing well. I am Chaitanya from the linux graphics team in Intel.
This mail is regarding a regression we are seeing in our CI runs[1] on linux-next repository.
Since the version next-20250919 [2], we are seeing the following regression
````````````````````````````````````````````````````````````````````````````````` <4>[ 10.973827] ------------[ cut here ]------------ <4>[ 10.973841] WARNING: arch/x86/events/core.c:3089 at perf_get_x86_pmu_capability+0xd/0xc0, CPU#15: (udev-worker)/386 ... <4>[ 10.974028] Call Trace: <4>[ 10.974030] <TASK> <4>[ 10.974033] ? kvm_init_pmu_capability+0x2b/0x190 [kvm] <4>[ 10.974154] kvm_x86_vendor_init+0x1b0/0x1a40 [kvm] <4>[ 10.974248] vmx_init+0xdb/0x260 [kvm_intel] <4>[ 10.974278] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974296] vt_init+0x12/0x9d0 [kvm_intel] <4>[ 10.974309] ? __pfx_vt_init+0x10/0x10 [kvm_intel] <4>[ 10.974322] do_one_initcall+0x60/0x3f0 <4>[ 10.974335] do_init_module+0x97/0x2b0 <4>[ 10.974345] load_module+0x2d08/0x2e30 <4>[ 10.974349] ? __kernel_read+0x158/0x2f0 <4>[ 10.974370] ? kernel_read_file+0x2b1/0x320 <4>[ 10.974381] init_module_from_file+0x96/0xe0 <4>[ 10.974384] ? init_module_from_file+0x96/0xe0 <4>[ 10.974399] idempotent_init_module+0x117/0x330 <4>[ 10.974415] __x64_sys_finit_module+0x73/0xe0 ... `````````````````````````````````````````````````````````````````````````````````Details log can be found in [3].
After bisecting the tree, the following patch [4] seems to be the first "bad" commit
````````````````````````````````````````````````````````````````````````````````````````````````````````` From 51f34b1e650fc5843530266cea4341750bd1ae37 Mon Sep 17 00:00:00 2001 From: Sean Christopherson <[email protected]> Date: Wed, 6 Aug 2025 12:56:39 -0700 Subject: KVM: x86/pmu: Snapshot host (i.e. perf's) reported PMU capabilities Take a snapshot of the unadulterated PMU capabilities provided by perf so that KVM can compare guest vPMU capabilities against hardware capabilities when determining whether or not to intercept PMU MSRs (and RDPMC). `````````````````````````````````````````````````````````````````````````````````````````````````````````Quoted 4 lines
We also verified that if we revert the patch the issue is not seen.
Could you please check why the patch causes this regression and provide a fix if necessary?
Hi Chaitanya,
I suppose you found this warning on a hybrid client platform, right? It looks the warning is triggered by the below WARN_ON_ONCE() in
perf_get_x86_pmu_capability()function.Quoted 8 lines
if (WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) || !x86_pmu_initialized()) {
memset(cap, 0, sizeof(*cap));
return;
}The below change should fix it (just building, not test it). I would run a full scope vPMU test after I come back from China national day's holiday.
I have access to a hybrid system, I'll also double check there (though I'm 99.9% certain you've got it right).
Quoted 16 lines
Thanks.
arch/x86/kvm/pmu.c +0
-2@@ -108,8 +108,6 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL; int min_nr_gp_ctrs = pmu_ops->MIN_NR_GP_COUNTERS; - perf_get_x86_pmu_capability(&kvm_host_pmu); - /** Hybrid PMUs don't play nice with virtualization without careful * configuration by userspace, and KVM's APIs for reporting supported arch/x86/kvm/pmu.c void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) · line 125
Quoted 8 lines
arch/x86/kvm/pmu.c +2
-0@@ -120,6 +118,8 @@ void kvm_init_pmu_capability(struct kvm_pmu_ops *pmu_ops) enable_pmu = false; if (enable_pmu) { + perf_get_x86_pmu_capability(&kvm_host_pmu); + /** WARN if perf did NOT disable hardware PMU if the number of * architecturally required GP counters aren't present, i.e. if If we go this route, then the !enable_pmu path should explicitly zero
kvm_host_pmuso that the behavior is consistent userspace loads kvm.ko with enable_pmu=0, versusenable_pmubeing cleared because of lack of support.if (!enable_pmu) { memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu)); memset(&kvm_pmu_cap, 0, sizeof(kvm_pmu_cap)); return; }The alternative would be keep
kvm_host_pmuvalid at all times for !HYBRID, which is what I intended with the bad patch, but that too would lead to inconsistent behavior. So I think it makes sense to go with Dapeng's approach; we can always revisit this if some future thing in KVM _needs_kvm_host_pmueven with enable_pmu=0.
Quoted 6 lines
if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) {
enable_pmu = false;
memset(&kvm_host_pmu, 0, sizeof(kvm_host_pmu));
} else {
perf_get_x86_pmu_capability(&kvm_host_pmu);
}Yeah, it looks better. We should decouple "enable_pmu" and "kvm_host_pmu" as the initial design. Thanks.
Quoted 1 line
Patchwork
CI✗ Fi.CI.BUILD: failure for REGRESSION on linux-next (next-20250919)
In reply to Borah, Chaitanya Kumar== Series Details ==
== Series Details ==
Series: REGRESSION on linux-next (next-20250919)
URL : https://patchwork.freedesktop.org/series/155228/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/155228/revisions/1/mbox/ not applied
Applying: REGRESSION on linux-next (next-20250919)
error: corrupt patch at line 6
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 REGRESSION on linux-next (next-20250919)
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Build failed, no error log produced