perf/x86: Miscellaneous PMU bug fixes and optimizations
Subsystems (2)
v3 cover perf/x86: Miscellaneous PMU bug fixes and optimizations
Changes since v2:
- Patch 6/8: Fix the typo in comments (Thomas).
- Patch 8/8: New patch which prevents drain_pebs() reentry.
Changes since v1:
- Patch 3/7: Ensure cpuc->pmu is not the static pmu before calling hybrid_pmu(cpuc->pmu) in intel_pmu_cpu_dead() (Sashiko).
This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
The patch 7/8 optimizes ACR handling in match_prev_assignment() and mitigate the performance overhead.
The patch 8/8 enhances intel_pmu_drain_pebs_buffer() to prevent drain_pebs() reentry.
Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test
$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156
History:
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (8):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access perf/x86/intel: Fix intel_cap handling on hybrid PMUs perf/x86: Optimize ACR handling in match_prev_assignment() perf/x86/intel: Prevent drain_pebs() reentry
arch/x86/events/core.c | 45 ++++++++++++---- arch/x86/events/intel/core.c | 101 ++++++++++++++++++++++++++--------- arch/x86/events/intel/ds.c | 7 --- arch/x86/events/perf_event.h | 12 ++--- 4 files changed, 113 insertions(+), 52 deletions(-)
base-commit: edda9051e267b7390c7ce24b1b71434414ad156e
Signature · 1 line
2.34.1
PATCH v3 1/8 perf/x86: Unregister PMI handler on PMU init failure
Fix an NMI handler leak in init_hw_perf_events(). When PMU initialization fails after register_nmi_handler(), the error path exits without calling unregister_nmi_handler(), leaving a stale NMI_LOCAL "PMI" handler registered. Add the missing call before clearing x86_pmu state.
--- arch/x86/events/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/events/core.c +3 -1
@@ -2219,7 +2219,7 @@ static int __init init_hw_perf_events(void) err =cpuhp_setup_state (CPUHP_PERF_X86_PREPARE ,"perf/x86:prepare" , x86_pmu_prepare_cpu, x86_pmu_dead_cpu);if (err) -return err; + goto pmi_unregister; err = cpuhp_setup_state(CPUHP_AP_PERF_X86_STARTING, "perf/x86:starting", x86_pmu_starting_cpu,
@@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)cpuhp_remove_state (CPUHP_AP_PERF_X86_STARTING ); out:cpuhp_remove_state (CPUHP_PERF_X86_PREPARE ); +pmi_unregister: +unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu:memset (&x86_pmu,0 ,sizeof (x86_pmu));return err;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Pre-existing Use-After-Free window for cpuc->pmu if perf_pmu_register() fails on hybrid systems. --
--- Patch [1]: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure --- commit 841a0ed2adb8a95774faf3b5508e90fdd97d9135
Author: Dapeng Mi <[email protected]>
perf/x86: Unregister PMI handler on PMU init failure
This commit fixes an NMI handler leak in init_hw_perf_events() by unregistering the NMI handler when PMU initialization fails before clearing the x86_pmu state.
Quoted 4 lines
arch/x86/events/core.c +0
-0
[ ... ]
arch/x86/events/core.c static int __init init_hw_perf_events(void) · line 2278
Quoted 7 lines
arch/x86/events/core.c +2
-0@@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)cpuhp_remove_state (CPUHP_AP_PERF_X86_STARTING ); out:cpuhp_remove_state (CPUHP_PERF_X86_PREPARE ); +pmi_unregister: +unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu:
[Severity: High]
This is a pre-existing issue, but does this error path leave a dangling pointer that could lead to a Use-After-Free?
On hybrid systems, if perf_pmu_register() fails, the error path frees the x86_pmu.hybrid_pmu structures. However, CPUHP_AP_PERF_X86_STARTING was already set up, so cpuc->pmu on all online CPUs was initialized to point to these now-freed structures.
The CPU hotplug state rollback doesn't set cpuc->pmu back to NULL. Since x86_pmu_static_call_update() has already routed x86_pmu_guest_get_msrs to intel_guest_get_msrs(), and the perf_is_hybrid static key remains enabled, could a subsequent KVM guest launch trigger a UAF?
When KVM calls perf_guest_get_msrs(), it jumps to intel_guest_get_msrs(), which unconditionally dereferences the dangling cpuc->pmu pointer:
arch/x86/events/intel/core.c:intel_guest_get_msrs() {
...
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
...
}Quoted 2 lines
arch/x86/events/core.c +0
-0memset (&x86_pmu,0 ,sizeof (x86_pmu));return err;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure
Quoted 41 lines · On 7/17/2026 4:33 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Pre-existing Use-After-Free window forcpuc->pmuifperf_pmu_register()fails on hybrid systems. ----- Patch [1]: [Patch v3 1/8] perf/x86: Unregister PMI handler on PMU init failure --- commit
841a0ed2adb8a95774faf3b5508e90fdd97d9135
Author: Dapeng Mi <[email protected]>perf/x86: Unregister PMI handler on PMU init failure
This commit fixes an NMI handler leak in
init_hw_perf_events()by unregistering the NMI handler when PMU initialization fails before clearing thex86_pmustate.Quoted 4 lines
arch/x86/events/core.c +0
-0[ ... ]
arch/x86/events/core.c static int __init init_hw_perf_events(void) · line 2278
Quoted 7 lines
arch/x86/events/core.c +2
-0@@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)cpuhp_remove_state (CPUHP_AP_PERF_X86_STARTING ); out:cpuhp_remove_state (CPUHP_PERF_X86_PREPARE ); +pmi_unregister: +unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu:[Severity: High]
This is a pre-existing issue, but does this error path leave a dangling pointer that could lead to a Use-After-Free?On hybrid systems, if
perf_pmu_register()fails, the error path frees the x86_pmu.hybrid_pmu structures. However,CPUHP_AP_PERF_X86_STARTINGwas already set up, socpuc->pmuon all online CPUs was initialized to point to these now-freed structures.The CPU hotplug state rollback doesn't set
cpuc->pmuback to NULL. Sincex86_pmu_static_call_update()has already routedx86_pmu_guest_get_msrstointel_guest_get_msrs(), and theperf_is_hybridstatic key remains enabled, could a subsequent KVM guest launch trigger a UAF?
Quoted 8 lines
When KVM calls
perf_guest_get_msrs(), it jumps tointel_guest_get_msrs(), which unconditionally dereferences the danglingcpuc->pmupointer:arch/x86/events/intel/core.c:intel_guest_get_msrs() { ... u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl); ... }
This issue would be fixed in the subsequent patch 2/8 which would reset "perf_is_hybrid" to false. So hybrid() would never touch these freed structures. Thanks.
Quoted 3 lines
Quoted 3 lines
arch/x86/events/core.c +0
-0memset (&x86_pmu,0 ,sizeof (x86_pmu));return err;
[tip: perf/core] perf/x86: Unregister PMI handler on PMU init failure
In reply to Dapeng MiThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 91787047b1dc52b1fee68b07f79af70aae5fce47 Gitweb: https://git.kernel.org/tip/91787047b1dc52b1fee68b07f79af70aae5fce47 Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:35 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:47 +02:00
perf/x86: Unregister PMI handler on PMU init failure
Fix an NMI handler leak in init_hw_perf_events(). When PMU initialization fails after register_nmi_handler(), the error path exits without calling unregister_nmi_handler(), leaving a stale NMI_LOCAL "PMI" handler registered. Add the missing call before clearing x86_pmu state.
--- arch/x86/events/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/events/core.c +3 -1
@@ -2219,7 +2219,7 @@ static int __init init_hw_perf_events(void) err =cpuhp_setup_state (CPUHP_PERF_X86_PREPARE ,"perf/x86:prepare" , x86_pmu_prepare_cpu, x86_pmu_dead_cpu);if (err) -return err; + goto pmi_unregister; err = cpuhp_setup_state(CPUHP_AP_PERF_X86_STARTING, "perf/x86:starting", x86_pmu_starting_cpu,
@@ -2273,6 +2273,8 @@ out1:cpuhp_remove_state (CPUHP_AP_PERF_X86_STARTING ); out:cpuhp_remove_state (CPUHP_PERF_X86_PREPARE ); +pmi_unregister: +unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu:memset (&x86_pmu,0 ,sizeof (x86_pmu));return err;
PATCH v3 2/8 perf/x86: Free hybrid state on PMU init failure
In reply to Dapeng MiIf PMU initialization fails, for example in check_hw_exists(), hybrid state can be left partially initialized: x86_pmu.hybrid_pmu is not freed and perf_is_hybrid remains set. This can leak memory and leave stale hybrid state reachable after a failed init path.
Add x86_pmu_free_hybrid() and use it on PMU init failure paths so all hybrid-related state is consistently reset.
--- arch/x86/events/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/events/core.c +12 -3
@@ -2130,6 +2130,17 @@ void x86_pmu_show_pmu_cap(struct pmu *pmu)pr_info ("... global_ctrl mask: %016llx\n" ,hybrid (pmu, intel_ctrl)); } +static void x86_pmu_free_hybrid (void ) +{ +if (!x86_pmu.hybrid_pmu) +return ; + +static_branch_disable (&perf_is_hybrid); +kfree (x86_pmu.hybrid_pmu); + x86_pmu.hybrid_pmu =NULL ; + x86_pmu.num_hybrid_pmus =0 ; +} +static int __init init_hw_perf_events (void ) {struct x86_pmu_quirk *quirk;
@@ -2258,9 +2269,6 @@ static int __init init_hw_perf_events(void)for (j =0 ; j < i; j++)perf_pmu_unregister (&x86_pmu.hybrid_pmu[j].pmu);pr_warn ("Failed to register hybrid PMUs\n" ); -kfree (x86_pmu.hybrid_pmu); - x86_pmu.hybrid_pmu =NULL ; - x86_pmu.num_hybrid_pmus =0 ; goto out2; } }
@@ -2276,6 +2284,7 @@ static int __init init_hw_perf_events(void) pmi_unregister:unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu: +x86_pmu_free_hybrid ();memset (&x86_pmu,0 ,sizeof (x86_pmu));return err; }
Signature · 1 line
2.34.1
[tip: perf/core] perf/x86: Free hybrid state on PMU init failure
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 96746e731e1626545e343974ca4673739dceb31b Gitweb: https://git.kernel.org/tip/96746e731e1626545e343974ca4673739dceb31b Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:36 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:47 +02:00
perf/x86: Free hybrid state on PMU init failure
If PMU initialization fails, for example in check_hw_exists(), hybrid state can be left partially initialized: x86_pmu.hybrid_pmu is not freed and perf_is_hybrid remains set. This can leak memory and leave stale hybrid state reachable after a failed init path.
Add x86_pmu_free_hybrid() and use it on PMU init failure paths so all hybrid-related state is consistently reset.
--- arch/x86/events/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/events/core.c +12 -3
@@ -2130,6 +2130,17 @@ void x86_pmu_show_pmu_cap(struct pmu *pmu)pr_info ("... global_ctrl mask: %016llx\n" ,hybrid (pmu, intel_ctrl)); } +static void x86_pmu_free_hybrid (void ) +{ +if (!x86_pmu.hybrid_pmu) +return ; + +static_branch_disable (&perf_is_hybrid); +kfree (x86_pmu.hybrid_pmu); + x86_pmu.hybrid_pmu =NULL ; + x86_pmu.num_hybrid_pmus =0 ; +} +static int __init init_hw_perf_events (void ) {struct x86_pmu_quirk *quirk;
@@ -2258,9 +2269,6 @@ static int __init init_hw_perf_events(void)for (j =0 ; j < i; j++)perf_pmu_unregister (&x86_pmu.hybrid_pmu[j].pmu);pr_warn ("Failed to register hybrid PMUs\n" ); -kfree (x86_pmu.hybrid_pmu); - x86_pmu.hybrid_pmu =NULL ; - x86_pmu.num_hybrid_pmus =0 ; goto out2; } }
@@ -2276,6 +2284,7 @@ out: pmi_unregister:unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu: +x86_pmu_free_hybrid ();memset (&x86_pmu,0 ,sizeof (x86_pmu));return err; }
PATCH v3 3/8 perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
In reply to Dapeng MiIn failure paths, cpuc->pmu can still point to the global static pmu instead of an embedded x86_hybrid_pmu::pmu. Calling hybrid_pmu() on that pointer causes an invalid container conversion and may lead to out-of-bounds access.
This can happen in at least two cases:
- init_hybrid_pmu() fails check_hw_exists() and leaves cpuc->pmu as-is.
- CPU hotplug fails between CPUHP_PERF_X86_PREPARE and CPUHP_AP_PERF_X86_STARTING, and rollback invokes intel_pmu_cpu_dead().
Fix both paths by:
- Clear cpuc->pmu to NULL when check_hw_exists() fails.
- Validat that cpuc->pmu is not the global static pmu before calling hybrid_pmu() in intel_pmu_cpu_dead().
A new helper x86_get_static_pmu() is added to get the global static pmu.
--- arch/x86/events/core.c | 5 +++++ arch/x86/events/intel/core.c | 7 +++++-- arch/x86/events/perf_event.h | 1 + 3 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/core.c +5 -0
@@ -790,6 +790,11 @@ int is_x86_event(struct perf_event *event)return false; } +inline struct pmu *x86_get_static_pmu (void ) +{ +return &pmu; +} +struct pmu *x86_get_pmu (unsigned int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);
arch/x86/events/intel/core.c +5 -2
@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;return false; + }pr_info ("%s PMU driver: " , pmu->name);
@@ -6475,11 +6477,12 @@ void intel_cpuc_finish(struct cpu_hw_events *cpuc)static void intel_pmu_cpu_dead (int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu); +struct pmu *pmu =x86_get_static_pmu ();release_arch_pebs_buf_on_cpu (cpu);intel_cpuc_finish (cpuc); -if (is_hybrid () && cpuc->pmu) +if (is_hybrid () && cpuc->pmu && cpuc->pmu != pmu)cpumask_clear_cpu (cpu, &hybrid_pmu (cpuc->pmu)->supported_cpus); }
arch/x86/events/perf_event.h +1 -0
@@ -1161,6 +1161,7 @@ static struct perf_pmu_format_hybrid_attr format_attr_hybrid_##_name = {\ .pmu_type = _pmu, \ } +struct pmu *x86_get_static_pmu (void );struct pmu *x86_get_pmu (unsigned int cpu);extern struct x86_pmu x86_pmu__read_mostly ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Setting cpuc->pmu = NULL on initialization failure triggers a user-triggerable WARN_ON_ONCE in x86_get_pmu().
Pre-existing issues:
- [High] Unchecked MSR writes in NMI handler cause a #GP fault on asymmetric PMU configurations. --
--- Patch [3]: [Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts --- commit 2ef12c59a0f0eb0a83a45dd11cf187ca55beecdf
Author: Dapeng Mi <[email protected]>
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
Fix invalid container conversions in failure paths by clearing cpuc->pmu to NULL when check_hw_exists() fails, and validating cpuc->pmu in intel_pmu_cpu_dead(). A new helper x86_get_static_pmu() is added.
arch/x86/events/intel/core.c static bool init_hybrid_pmu(int cpu) · line 6333
Quoted 11 lines
arch/x86/events/intel/core.c +2
-1@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;
[Severity: High]
Does setting cpuc->pmu to NULL here trigger a warning in x86_get_pmu()?
If check_hw_exists() fails (e.g., due to a hypervisor restricting PMU MSRs), we now set cpuc->pmu to NULL but leave the CPU in the possible_cpus mask.
If a user later opens a perf event utilizing Architectural LBRs (like perf record -b), x86_reserve_hardware() calls reserve_lbr_buffers() which iterates over all possible CPUs:
arch/x86/events/intel/lbr.c:reserve_lbr_buffers() {
...
for_each_possible_cpu(cpu) {
kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
...
}Because the failed CPU's cpuc->pmu is NULL, x86_get_pmu() hits its strict WARN_ON_ONCE(!cpuc->pmu) assertion. If panic_on_warn=1 is set, could this upgrade a graceful PMU initialization failure into a local panic?
Since intel_pmu_cpu_dead() now safely guards against the invalid cast with cpuc->pmu != pmu, is this assignment necessary?
[Severity: High]
This is a pre-existing issue, but does failing check_hw_exists() leave the CPU vulnerable to a #GP fault during a spurious NMI?
If a CPU is brought online but its PMU MSRs are disabled by a hypervisor, initialization for that CPU aborts here. However, perf_event_nmi_handler() is registered globally if at least one CPU succeeds.
If an active perf event runs on a healthy CPU (so active_events > 0), and a spurious or broadcast NMI strikes the broken CPU, it will execute:
intel_pmu_handle_irq()
__intel_pmu_disable_all(true)
wrmsrq(MSR_CORE_PERF_GLOBAL_CTRL, 0);Could accessing this non-existent MSR on the failed CPU trigger an unchecked #GP fault and kernel panic?
Quoted 4 lines
arch/x86/events/intel/core.c +1
-0return false; + }pr_info ("%s PMU driver: " , pmu->name);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [Patch v3 3/8] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
Quoted 47 lines · On 7/17/2026 5:31 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Settingcpuc->pmu = NULLon initialization failure triggers a user-triggerableWARN_ON_ONCEinx86_get_pmu().Pre-existing issues:
- [High] Unchecked MSR writes in NMI handler cause a #GP fault on asymmetric PMU configurations. ----- Patch [3]: [Patch v3 3/8] perf/x86: Guard
intel_pmu_cpu_dead()against invalid hybrid PMU casts --- commit2ef12c59a0f0eb0a83a45dd11cf187ca55beecdf
Author: Dapeng Mi <[email protected]>perf/x86: Guard
intel_pmu_cpu_dead()against invalid hybrid PMU castsFix invalid container conversions in failure paths by clearing
cpuc->pmuto NULL whencheck_hw_exists()fails, and validatingcpuc->pmuinintel_pmu_cpu_dead(). A new helperx86_get_static_pmu()is added.arch/x86/events/intel/core.c static bool init_hybrid_pmu(int cpu) · line 6333
Quoted 11 lines
arch/x86/events/intel/core.c +2
-1@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;[Severity: High]
Does settingcpuc->pmuto NULL here trigger a warning inx86_get_pmu()?If
check_hw_exists()fails (e.g., due to a hypervisor restricting PMU MSRs), we now setcpuc->pmuto NULL but leave the CPU in thepossible_cpusmask.If a user later opens a perf event utilizing Architectural LBRs (like perf record -b),
x86_reserve_hardware()callsreserve_lbr_buffers()which iterates over all possible CPUs:arch/x86/events/intel/lbr.c:reserve_lbr_buffers() { ... for_each_possible_cpu(cpu) { kmem_cache = x86_get_pmu(cpu)->task_ctx_cache; ... }
Quoted 6 lines
Because the failed CPU's
cpuc->pmuis NULL,x86_get_pmu()hits its strictWARN_ON_ONCE(!cpuc->pmu)assertion. If panic_on_warn=1 is set, could this upgrade a graceful PMU initialization failure into a local panic?Since
intel_pmu_cpu_dead()now safely guards against the invalid cast withcpuc->pmu!= pmu, is this assignment necessary?
This is expected. If the PMU initialization fails on a cpu, the cpuc->pmu has to set NULL, which is the simplest (maybe best) way to indicate the initialization failure.
Strictly speaking, it's unnecessary to check if cpuc->pmu != NULL, but we still want to keep it since it obviously indicates the cpuc->pmu could be reset to NULL.
Quoted 16 lines
[Severity: High]
This is a pre-existing issue, but does failingcheck_hw_exists()leave the CPU vulnerable to a #GP fault during a spurious NMI?If a CPU is brought online but its PMU MSRs are disabled by a hypervisor, initialization for that CPU aborts here. However,
perf_event_nmi_handler()is registered globally if at least one CPU succeeds.If an active perf event runs on a healthy CPU (so
active_events> 0), and a spurious or broadcast NMI strikes the broken CPU, it will execute:
intel_pmu_handle_irq()__intel_pmu_disable_all(true) wrmsrq(MSR_CORE_PERF_GLOBAL_CTRL, 0);
Quoted 7 lines
Could accessing this non-existent MSR on the failed CPU trigger an unchecked #GP fault and kernel panic?
arch/x86/events/intel/core.c static bool init_hybrid_pmu(int cpu) · line 6337
Quoted 4 lines
arch/x86/events/intel/core.c +1
-0return false; + }pr_info ("%s PMU driver: " , pmu->name);
Yeah, it's possible but it's hard to fix in current hotplug implementation. The best way to fix this issue is to let hotplug rollback and offline the CPU, but it seems currently the failure of CPUHP_AP_PERF_X86_STARTING won't really cause the rollback.
[tip: perf/core] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
In reply to Dapeng MiThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 278a3731c9d08bc6ea489c1987c6b7a7020f5d2b Gitweb: https://git.kernel.org/tip/278a3731c9d08bc6ea489c1987c6b7a7020f5d2b Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:37 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:47 +02:00
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
In failure paths, cpuc->pmu can still point to the global static pmu instead of an embedded x86_hybrid_pmu::pmu. Calling hybrid_pmu() on that pointer causes an invalid container conversion and may lead to out-of-bounds access.
This can happen in at least two cases:
- init_hybrid_pmu() fails check_hw_exists() and leaves cpuc->pmu as-is.
- CPU hotplug fails between CPUHP_PERF_X86_PREPARE and CPUHP_AP_PERF_X86_STARTING, and rollback invokes intel_pmu_cpu_dead().
Fix both paths by:
- Clear cpuc->pmu to NULL when check_hw_exists() fails.
- Validat that cpuc->pmu is not the global static pmu before calling hybrid_pmu() in intel_pmu_cpu_dead().
A new helper x86_get_static_pmu() is added to get the global static pmu.
--- arch/x86/events/core.c | 5 +++++ arch/x86/events/intel/core.c | 7 +++++-- arch/x86/events/perf_event.h | 1 + 3 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/core.c +5 -0
@@ -790,6 +790,11 @@ int is_x86_event(struct perf_event *event)return false; } +inline struct pmu *x86_get_static_pmu (void ) +{ +return &pmu; +} +struct pmu *x86_get_pmu (unsigned int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);
arch/x86/events/intel/core.c +5 -2
@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;return false; + }pr_info ("%s PMU driver: " , pmu->name);
@@ -6475,11 +6477,12 @@ void intel_cpuc_finish(struct cpu_hw_events *cpuc)static void intel_pmu_cpu_dead (int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu); +struct pmu *pmu =x86_get_static_pmu ();release_arch_pebs_buf_on_cpu (cpu);intel_cpuc_finish (cpuc); -if (is_hybrid () && cpuc->pmu) +if (is_hybrid () && cpuc->pmu && cpuc->pmu != pmu)cpumask_clear_cpu (cpu, &hybrid_pmu (cpuc->pmu)->supported_cpus); }
arch/x86/events/perf_event.h +1 -0
@@ -1161,6 +1161,7 @@ static struct perf_pmu_format_hybrid_attr format_attr_hybrid_##_name = {\ .pmu_type = _pmu, \ } +struct pmu *x86_get_static_pmu (void );struct pmu *x86_get_pmu (unsigned int cpu);extern struct x86_pmu x86_pmu__read_mostly ;
PATCH v3 4/8 perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
In reply to Dapeng Miintel_pmu_cpu_prepare() allocates per-CPU perf state first and then sets up the arch PEBS buffer. If alloc_arch_pebs_buf_on_cpu() fails, the previously allocated cpuc resources are left behind.
Make the failure path call intel_cpuc_finish(cpuc) to release the per-CPU state allocated by intel_cpuc_prepare().
--- arch/x86/events/intel/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +9 -2
@@ -5924,13 +5924,20 @@ int intel_cpuc_prepare(struct cpu_hw_events *cpuc, int cpu)static int intel_pmu_cpu_prepare (int cpu) { +struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);int ret; - ret =intel_cpuc_prepare (&per_cpu (cpu_hw_events, cpu), cpu); + ret =intel_cpuc_prepare (cpuc, cpu);if (ret)return ret; -return alloc_arch_pebs_buf_on_cpu (cpu); + ret =alloc_arch_pebs_buf_on_cpu (cpu); +if (ret) { +intel_cpuc_finish (cpuc); +return ret; + } + +return 0 ; }static void flip_smm_bit (void *data)
Signature · 1 line
2.34.1
[tip: perf/core] perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 85182f902afda28ca7ae3408b72f31640a5cfc73 Gitweb: https://git.kernel.org/tip/85182f902afda28ca7ae3408b72f31640a5cfc73 Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:38 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:47 +02:00
perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
intel_pmu_cpu_prepare() allocates per-CPU perf state first and then sets up the arch PEBS buffer. If alloc_arch_pebs_buf_on_cpu() fails, the previously allocated cpuc resources are left behind.
Make the failure path call intel_cpuc_finish(cpuc) to release the per-CPU state allocated by intel_cpuc_prepare().
--- arch/x86/events/intel/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +9 -2
@@ -5924,13 +5924,20 @@ err:static int intel_pmu_cpu_prepare (int cpu) { +struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);int ret; - ret =intel_cpuc_prepare (&per_cpu (cpu_hw_events, cpu), cpu); + ret =intel_cpuc_prepare (cpuc, cpu);if (ret)return ret; -return alloc_arch_pebs_buf_on_cpu (cpu); + ret =alloc_arch_pebs_buf_on_cpu (cpu); +if (ret) { +intel_cpuc_finish (cpuc); +return ret; + } + +return 0 ; }static void flip_smm_bit (void *data)
PATCH v3 5/8 perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
In reply to Dapeng MiOn hybrid systems, init_hw_perf_events() can call check_hw_exists() with the global PMU pointer after perf_is_hybrid is set. In that case, fixed_counter_disabled() uses hybrid() on a non-hybrid PMU object, so the intel_ctrl access is taken from the wrong layout and can read out of bounds.
fixed_counter_disabled() was added in commit 32451614da2a ("perf/x86/intel: Support CPUID 10.ECX to disable fixed counters"), when fixed counters were tracked via num_fixed_counters. Today fixed counters are represented by fixed_cntr_mask, so this helper is obsolete.
Remove fixed_counter_disabled() and its callers, and rely directly on the fixed-counter bitmask. With the helper gone, check_hw_exists() no longer needs a PMU argument, so drop that parameter as well. This removes the invalid hybrid access and closes the out-of-bounds read risk.
--- arch/x86/events/core.c | 8 ++------ arch/x86/events/intel/core.c | 4 +--- arch/x86/events/perf_event.h | 9 +-------- 3 files changed, 4 insertions(+), 17 deletions(-)
arch/x86/events/core.c +2 -6
@@ -265,7 +265,7 @@ static void release_pmc_hardware(void) {}#endif -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask) {u64 val, val_fail =-1 , val_new= ~0 ;
@@ -297,8 +297,6 @@ bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,if (ret) goto msr_fail;for_each_set_bit (i, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (i, pmu)) -continue ;if (val & (0x03ULL << i*4 )) { bios_fail =1 ; val_fail = val;
@@ -1618,8 +1616,6 @@ void perf_event_print_debug(void) cpu, idx, prev_left); }for_each_set_bit (idx, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;rdmsrq (x86_pmu_fixed_ctr_addr (idx), pmc_count); pr_info("CPU#%d: fixed-PMC%d count: %016llx\n",
@@ -2180,7 +2176,7 @@ static int __init init_hw_perf_events(void)pmu_check_apic ();/* sanity check that the hardware exists or is emulated */ -if (!check_hw_exists (&pmu, x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) +if (!check_hw_exists (x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) goto out_bad_pmu;pr_cont ("%s PMU driver.\n" , x86_pmu.name);
arch/x86/events/intel/core.c +1 -3
@@ -3713,8 +3713,6 @@ static void intel_pmu_reset(void)wrmsrq_safe (x86_pmu_event_addr (idx),0ull ); }for_each_set_bit (idx, fixed_cntr_mask,INTEL_PMC_MAX_FIXED ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;wrmsrq_safe (x86_pmu_fixed_ctr_addr (idx),0ull ); }
@@ -6336,7 +6334,7 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { +if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) { cpuc->pmu =NULL ;return false; }
arch/x86/events/perf_event.h +1 -8
@@ -1243,7 +1243,7 @@ static inline int x86_pmu_rdpmc_index(int index)return x86_pmu.rdpmc_index ? x86_pmu.rdpmc_index (index) : index; } -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask);int x86_add_exclusive (unsigned int what);
@@ -1456,13 +1456,6 @@ ssize_t events_hybrid_sysfs_show(struct device *dev,struct device_attribute *attr, char *page); -static inline bool fixed_counter_disabled (int i,struct pmu *pmu) -{ -u64 intel_ctrl =hybrid (pmu, intel_ctrl); - -return !(intel_ctrl >> (i +INTEL_PMC_IDX_FIXED )); -} -#ifdef CONFIG_CPU_SUP_AMD int amd_pmu_init (void );
Signature · 1 line
2.34.1
[tip: perf/core] perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 3ba87c5bf11bea80e56abe17730085bc962ecfbe Gitweb: https://git.kernel.org/tip/3ba87c5bf11bea80e56abe17730085bc962ecfbe Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:39 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:48 +02:00
perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
On hybrid systems, init_hw_perf_events() can call check_hw_exists() with the global PMU pointer after perf_is_hybrid is set. In that case, fixed_counter_disabled() uses hybrid() on a non-hybrid PMU object, so the intel_ctrl access is taken from the wrong layout and can read out of bounds.
fixed_counter_disabled() was added in commit 32451614da2a ("perf/x86/intel: Support CPUID 10.ECX to disable fixed counters"), when fixed counters were tracked via num_fixed_counters. Today fixed counters are represented by fixed_cntr_mask, so this helper is obsolete.
Remove fixed_counter_disabled() and its callers, and rely directly on the fixed-counter bitmask. With the helper gone, check_hw_exists() no longer needs a PMU argument, so drop that parameter as well. This removes the invalid hybrid access and closes the out-of-bounds read risk.
--- arch/x86/events/core.c | 8 ++------ arch/x86/events/intel/core.c | 4 +--- arch/x86/events/perf_event.h | 9 +-------- 3 files changed, 4 insertions(+), 17 deletions(-)
arch/x86/events/core.c +2 -6
@@ -265,7 +265,7 @@ static void release_pmc_hardware(void) {}#endif -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask) {u64 val, val_fail =-1 , val_new= ~0 ;
@@ -297,8 +297,6 @@ bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,if (ret) goto msr_fail;for_each_set_bit (i, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (i, pmu)) -continue ;if (val & (0x03ULL << i*4 )) { bios_fail =1 ; val_fail = val;
@@ -1618,8 +1616,6 @@ void perf_event_print_debug(void) cpu, idx, prev_left); }for_each_set_bit (idx, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;rdmsrq (x86_pmu_fixed_ctr_addr (idx), pmc_count); pr_info("CPU#%d: fixed-PMC%d count: %016llx\n",
@@ -2180,7 +2176,7 @@ static int __init init_hw_perf_events(void)pmu_check_apic ();/* sanity check that the hardware exists or is emulated */ -if (!check_hw_exists (&pmu, x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) +if (!check_hw_exists (x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) goto out_bad_pmu;pr_cont ("%s PMU driver.\n" , x86_pmu.name);
arch/x86/events/intel/core.c +1 -3
@@ -3713,8 +3713,6 @@ static void intel_pmu_reset(void)wrmsrq_safe (x86_pmu_event_addr (idx),0ull ); }for_each_set_bit (idx, fixed_cntr_mask,INTEL_PMC_MAX_FIXED ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;wrmsrq_safe (x86_pmu_fixed_ctr_addr (idx),0ull ); }
@@ -6336,7 +6334,7 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { +if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) { cpuc->pmu =NULL ;return false; }
arch/x86/events/perf_event.h +1 -8
@@ -1243,7 +1243,7 @@ static inline int x86_pmu_rdpmc_index(int index)return x86_pmu.rdpmc_index ? x86_pmu.rdpmc_index (index) : index; } -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask);int x86_add_exclusive (unsigned int what);
@@ -1456,13 +1456,6 @@ ssize_t events_hybrid_sysfs_show(struct device *dev,struct device_attribute *attr, char *page); -static inline bool fixed_counter_disabled (int i,struct pmu *pmu) -{ -u64 intel_ctrl =hybrid (pmu, intel_ctrl); - -return !(intel_ctrl >> (i +INTEL_PMC_IDX_FIXED )); -} -#ifdef CONFIG_CPU_SUP_AMD int amd_pmu_init (void );
PATCH v3 6/8 perf/x86/intel: Fix intel_cap handling on hybrid PMUs
In reply to Dapeng Miintel_cap (IA32_PERF_CAPABILITIES) updates are currently tied to X86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths. As a result, hybrid PMU capability state can be updated under the wrong condition.
Also, intel_pmu_broken_perf_cap() is too narrow. Per RPL018, the missing PERF_METRICS_AVAILABLE bit affects both Raptor Lake and Meteor Lake parts, not only the currently covered subset.
Move intel_cap updates out of the ARCH_PERFMON_EXT-gated path, extend intel_pmu_broken_perf_cap() coverage to both RPL and MTL families, and introduce intel_update_pmu_caps() to centralize PMU capability updates.
--- arch/x86/events/intel/core.c | 41 ++++++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 11 deletions(-)
arch/x86/events/intel/core.c +30 -11
@@ -6147,8 +6147,15 @@ static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs);static inline bool intel_pmu_broken_perf_cap (void ) { -/* The Perf Metric (Bit 15) is always cleared */ -if (boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || +/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* RPL and MTL. Details can be found in RPL018 erratum. +* https://edc.intel.com/content/www/us/en/design/products/platforms/details/raptor-lake-s/13th-generation-core-processor-specification-update/errata-details/ +*/ +if (boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_P || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_S || + boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || boot_cpu_data.x86_vfm ==INTEL_METEORLAKE_L )return true;
@@ -6183,7 +6190,7 @@ static inline void __intel_update_large_pebs_flags(struct pmu *pmu)#define counter_mask (_gp, _fixed) ((_gp) | ((u64)(_fixed) << INTEL_PMC_IDX_FIXED)) -static void update_pmu_cap (struct pmu *pmu) +static void update_pmu_cap_from_perfmonext (struct pmu *pmu) {unsigned int eax, ebx, ecx, edx;union cpuid35_eax eax_0;
@@ -6241,10 +6248,24 @@ static void update_pmu_cap(struct pmu *pmu)WARN_ON (x86_pmu.arch_pebs ==1 ); x86_pmu.arch_pebs =0 ; } +} + +static void intel_update_pmu_caps (struct pmu *pmu) +{ +if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) +update_pmu_cap_from_perfmonext (pmu); -if (!intel_pmu_broken_perf_cap ()) { -/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */ -rdmsrq (MSR_IA32_PERF_CAPABILITIES ,hybrid (pmu, intel_cap).capabilities); +if (is_hybrid () &&this_cpu_has (X86_FEATURE_PDCM )) { +rdmsrq (MSR_IA32_PERF_CAPABILITIES , +hybrid (pmu, intel_cap).capabilities); + +/* +* Restore perf_metrics on platforms with broken +* perf_capablities. +*/ +if (intel_pmu_broken_perf_cap () && +hybrid_pmu (pmu)->pmu_type == hybrid_big) +hybrid (pmu, intel_cap).perf_metrics =1 ; } }
@@ -6329,9 +6350,7 @@ static bool init_hybrid_pmu(int cpu)if (!cpumask_empty (&pmu->supported_cpus)) goto end; -if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (&pmu->pmu); - +intel_update_pmu_caps (&pmu->pmu);intel_pmu_check_hybrid_pmus (pmu);if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) {
@@ -8828,8 +8847,8 @@ __init int intel_pmu_init(void)* from the leaf 0xa. The core specific update will be done later * when a new type is online. */ -if (!is_hybrid () &&boot_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (NULL ); +if (!is_hybrid ()) +intel_update_pmu_caps (NULL );if (x86_pmu.arch_pebs) { static_call_update(intel_pmu_disable_event_ext,
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerRe: [Patch v3 6/8] perf/x86/intel: Fix intel_cap handling on hybrid PMUs
arch/x86/events/intel/core.c static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs); · line 6154
Quoted 5 lines · Dapeng Mi
arch/x86/events/intel/core.c +5
-0+/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* RPL and MTL. Details can be found in RPL018 erratum. +* https://edc.intel.com/content/www/us/en/design/products/platforms/details/raptor-lake-s/13th-generation-core-processor-specification-update/errata-details/ +*/
Typically intel.com URLs are unstable and not worth the effort of including anywhere. I'll strip this out, its probably easy enough to google the erratum number.
Re: [Patch v3 6/8] perf/x86/intel: Fix intel_cap handling on hybrid PMUs
Quoted 9 lines · Peter Zijlstra
arch/x86/events/intel/core.c static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs); · line 6154
Quoted 5 lines · Dapeng Mi
arch/x86/events/intel/core.c +5
-0+/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* RPL and MTL. Details can be found in RPL018 erratum. +* https://edc.intel.com/content/www/us/en/design/products/platforms/details/raptor-lake-s/13th-generation-core-processor-specification-update/errata-details/ +*/ Typically intel.com URLs are unstable and not worth the effort of including anywhere. I'll strip this out, its probably easy enough to google the erratum number.
Sure. Thanks.
[tip: perf/core] perf/x86/intel: Fix intel_cap handling on hybrid PMUs
In reply to Dapeng MiThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: dd384d661f06e3725aad3a932a53e0ff81fb8805 Gitweb: https://git.kernel.org/tip/dd384d661f06e3725aad3a932a53e0ff81fb8805 Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:40 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:48 +02:00
perf/x86/intel: Fix intel_cap handling on hybrid PMUs
intel_cap (IA32_PERF_CAPABILITIES) updates are currently tied to X86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths. As a result, hybrid PMU capability state can be updated under the wrong condition.
Also, intel_pmu_broken_perf_cap() is too narrow. Per RPL018, the missing PERF_METRICS_AVAILABLE bit affects both Raptor Lake and Meteor Lake parts, not only the currently covered subset.
Move intel_cap updates out of the ARCH_PERFMON_EXT-gated path, extend intel_pmu_broken_perf_cap() coverage to both RPL and MTL families, and introduce intel_update_pmu_caps() to centralize PMU capability updates.
--- arch/x86/events/intel/core.c | 40 +++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 11 deletions(-)
arch/x86/events/intel/core.c +29 -11
@@ -6147,8 +6147,14 @@ static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs);static inline bool intel_pmu_broken_perf_cap (void ) { -/* The Perf Metric (Bit 15) is always cleared */ -if (boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || +/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* RPL and MTL. Details can be found in RPL018 erratum. +*/ +if (boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_P || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_S || + boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || boot_cpu_data.x86_vfm ==INTEL_METEORLAKE_L )return true;
@@ -6183,7 +6189,7 @@ static inline void __intel_update_large_pebs_flags(struct pmu *pmu)#define counter_mask (_gp, _fixed) ((_gp) | ((u64)(_fixed) << INTEL_PMC_IDX_FIXED)) -static void update_pmu_cap (struct pmu *pmu) +static void update_pmu_cap_from_perfmonext (struct pmu *pmu) {unsigned int eax, ebx, ecx, edx;union cpuid35_eax eax_0;
@@ -6241,10 +6247,24 @@ static void update_pmu_cap(struct pmu *pmu)WARN_ON (x86_pmu.arch_pebs ==1 ); x86_pmu.arch_pebs =0 ; } +} + +static void intel_update_pmu_caps (struct pmu *pmu) +{ +if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) +update_pmu_cap_from_perfmonext (pmu); -if (!intel_pmu_broken_perf_cap ()) { -/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */ -rdmsrq (MSR_IA32_PERF_CAPABILITIES ,hybrid (pmu, intel_cap).capabilities); +if (is_hybrid () &&this_cpu_has (X86_FEATURE_PDCM )) { +rdmsrq (MSR_IA32_PERF_CAPABILITIES , +hybrid (pmu, intel_cap).capabilities); + +/* +* Restore perf_metrics on platforms with broken +* perf_capablities. +*/ +if (intel_pmu_broken_perf_cap () && +hybrid_pmu (pmu)->pmu_type == hybrid_big) +hybrid (pmu, intel_cap).perf_metrics =1 ; } }
@@ -6329,9 +6349,7 @@ static bool init_hybrid_pmu(int cpu)if (!cpumask_empty (&pmu->supported_cpus)) goto end; -if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (&pmu->pmu); - +intel_update_pmu_caps (&pmu->pmu);intel_pmu_check_hybrid_pmus (pmu);if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) {
@@ -8828,8 +8846,8 @@ __init int intel_pmu_init(void)* from the leaf 0xa. The core specific update will be done later * when a new type is online. */ -if (!is_hybrid () &&boot_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (NULL ); +if (!is_hybrid ()) +intel_update_pmu_caps (NULL );if (x86_pmu.arch_pebs) { static_call_update(intel_pmu_disable_event_ext,
PATCH v3 7/8 perf/x86: Optimize ACR handling in match_prev_assignment()
In reply to Dapeng Mimatch_prev_assignment() currently forces a mismatch for ACR events, so ACR counter indices are reprogrammed on every scheduling pass. That causes avoidable overhead because disable and enable paths must touch multiple MSRs.
The previous ACR assignment is already cached in acr_cfg_b[]. Use that state to compare the newly computed ACR counter indices in hwc->config1 against the cached value in acr_cfg_b[hwc->idx]. If they match, skip unnecessary disable and enable work.
Also tighten is_acr_self_reload_event() so it first verifies the event is an ACR event before testing for the self-reload case.
--- arch/x86/events/core.c | 13 ++++++++++++- arch/x86/events/perf_event.h | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/events/core.c +12 -1
@@ -1297,6 +1297,17 @@ int x86_perf_rdpmc_index(struct perf_event *event)return event->hw.event_base_rdpmc; } +static inline bool acr_match_prev_indices (struct perf_event *event, +struct cpu_hw_events *cpuc) +{ +struct hw_perf_event *hwc = &event->hw; + +if (!is_acr_event_group (event)) +return true; +/* ACR counter indices don't change. */ +return hwc->config1 == cpuc->acr_cfg_b[hwc->idx]; +} +static inline int match_prev_assignment (struct perf_event *event,struct cpu_hw_events *cpuc,int i)
@@ -1306,7 +1317,7 @@ static inline int match_prev_assignment(struct perf_event *event,return hwc->idx == cpuc->assign[i] && hwc->last_cpu ==smp_processor_id () && hwc->last_tag == cpuc->tags[i] && - !is_acr_event_group (event); +acr_match_prev_indices (event, cpuc); }static void x86_pmu_start (struct perf_event *event,int flags);
arch/x86/events/perf_event.h +1 -1
@@ -141,7 +141,7 @@ static inline bool is_acr_self_reload_event(struct perf_event *event) {struct hw_perf_event *hwc = &event->hw; -if (hwc->idx <0 ) +if (hwc->idx <0 || !is_acr_event_group (event))return false;return test_bit (hwc->idx, (unsigned long *)&hwc->config1);
Signature · 1 line
2.34.1
[tip: perf/core] perf/x86: Optimize ACR handling in match_prev_assignment()
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 917d558b151cad5b05991e5eaee22efab33525ca Gitweb: https://git.kernel.org/tip/917d558b151cad5b05991e5eaee22efab33525ca Author: Dapeng Mi <[email protected]> AuthorDate: Fri, 17 Jul 2026 16:03:41 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 10 Aug 2026 15:05:48 +02:00
perf/x86: Optimize ACR handling in match_prev_assignment()
match_prev_assignment() currently forces a mismatch for ACR events, so ACR counter indices are reprogrammed on every scheduling pass. That causes avoidable overhead because disable and enable paths must touch multiple MSRs.
The previous ACR assignment is already cached in acr_cfg_b[]. Use that state to compare the newly computed ACR counter indices in hwc->config1 against the cached value in acr_cfg_b[hwc->idx]. If they match, skip unnecessary disable and enable work.
Also tighten is_acr_self_reload_event() so it first verifies the event is an ACR event before testing for the self-reload case.
--- arch/x86/events/core.c | 13 ++++++++++++- arch/x86/events/perf_event.h | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/events/core.c +12 -1
@@ -1297,6 +1297,17 @@ int x86_perf_rdpmc_index(struct perf_event *event)return event->hw.event_base_rdpmc; } +static inline bool acr_match_prev_indices (struct perf_event *event, +struct cpu_hw_events *cpuc) +{ +struct hw_perf_event *hwc = &event->hw; + +if (!is_acr_event_group (event)) +return true; +/* ACR counter indices don't change. */ +return hwc->config1 == cpuc->acr_cfg_b[hwc->idx]; +} +static inline int match_prev_assignment (struct perf_event *event,struct cpu_hw_events *cpuc,int i)
@@ -1306,7 +1317,7 @@ static inline int match_prev_assignment(struct perf_event *event,return hwc->idx == cpuc->assign[i] && hwc->last_cpu ==smp_processor_id () && hwc->last_tag == cpuc->tags[i] && - !is_acr_event_group (event); +acr_match_prev_indices (event, cpuc); }static void x86_pmu_start (struct perf_event *event,int flags);
arch/x86/events/perf_event.h +1 -1
@@ -141,7 +141,7 @@ static inline bool is_acr_self_reload_event(struct perf_event *event) {struct hw_perf_event *hwc = &event->hw; -if (hwc->idx <0 ) +if (hwc->idx <0 || !is_acr_event_group (event))return false;return test_bit (hwc->idx, (unsigned long *)&hwc->config1);
PATCH v3 8/8 perf/x86/intel: Prevent drain_pebs() reentry
In reply to Dapeng MiThe PEBS buffer is shared by all events on a CPU, so drain_pebs() must not run concurrently. If it is reentered, one instance may observe stale buffer state and potentially access out-of-bound memory.
Most invocations happen in NMI context, which naturally prevents reentry. However, drain_pebs() is also reachable from process context via intel_pmu_drain_pebs_buffer().
In those paths, the PMU is often already disabled, but not guaranteed. For example, __intel_pmu_pebs_disable() only disables the target counter, so other active counters can still raise a PMI and interrupt an in-flight drain_pebs().
Introduce __intel_pmu_quiesce() and __intel_pmu_resume() helpers and use them in intel_pmu_drain_pebs_buffer() to disable the full PMU around the drain_pebs() call, preventing reentry.
--- arch/x86/events/intel/core.c | 40 ++++++++++++++++++++++++++++-------- arch/x86/events/intel/ds.c | 7 ------- 2 files changed, 32 insertions(+), 15 deletions(-)
arch/x86/events/intel/core.c +32 -8
@@ -3125,6 +3125,36 @@ static void intel_pmu_del_event(struct perf_event *event)this_cpu_ptr (&cpu_hw_events)->n_late_setup--; } +static inline void __intel_pmu_quiesce (bool pmu_enabled) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->enabled =0 ; +if (pmu_enabled) +intel_pmu_disable_all (); +} + +static inline void __intel_pmu_resume (bool pmu_enabled) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->enabled = pmu_enabled; +if (pmu_enabled) +intel_pmu_enable_all (0 ); +} + +void intel_pmu_drain_pebs_buffer (void ) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +bool pmu_enabled = cpuc->enabled; +struct perf_sample_data data; + +/* Disable PMU so no new PMI can interrupt and re-enter drain_pebs(). */ +__intel_pmu_quiesce (pmu_enabled); +static_call (x86_pmu_drain_pebs)(NULL , &data); +__intel_pmu_resume (pmu_enabled); +} +static int icl_set_topdown_event_period (struct perf_event *event) {struct hw_perf_event *hwc = &event->hw;
@@ -3322,10 +3352,7 @@ static void intel_pmu_read_event(struct perf_event *event)if (is_metric_event (event) && (cpuc->txn_flags &PERF_PMU_TXN_READ ))return ; - cpuc->enabled =0 ; -if (pmu_enabled) -intel_pmu_disable_all (); - +__intel_pmu_quiesce (pmu_enabled);/* * If the PEBS counters snapshotting is enabled, * the topdown event is available in PEBS records.
@@ -3334,10 +3361,7 @@ static void intel_pmu_read_event(struct perf_event *event)static_call (intel_pmu_update_topdown_event)(event,NULL ); elseintel_pmu_drain_pebs_buffer (); - - cpuc->enabled = pmu_enabled; -if (pmu_enabled) -intel_pmu_enable_all (0 ); +__intel_pmu_resume (pmu_enabled);return ; }
arch/x86/events/intel/ds.c +0 -7
@@ -1240,13 +1240,6 @@ int intel_pmu_drain_bts_buffer(void)return 1 ; } -void intel_pmu_drain_pebs_buffer (void ) -{ -struct perf_sample_data data; - -static_call (x86_pmu_drain_pebs)(NULL , &data); -} -/* * PEBS */
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] TOCTOU race in intel_pmu_drain_bts_buffer leading to duplicate BTS records. --
--- Patch [8]: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry --- commit 77ebceadaa828dfb1bed75e560551cf29b1ae9f5
Author: Dapeng Mi <[email protected]>
perf/x86/intel: Prevent drain_pebs() reentry
The PEBS buffer is shared by all events on a CPU, so drain_pebs() must not run concurrently. This commit introduces __intel_pmu_quiesce() and __intel_pmu_resume() helpers to disable the full PMU around the drain_pebs() call, preventing reentry.
arch/x86/events/intel/ds.c int intel_pmu_drain_bts_buffer(void) · line 1241
Quoted 7 lines
arch/x86/events/intel/ds.c +0
-0@@ -1240,13 +1240,6 @@ int intel_pmu_drain_bts_buffer(void)return 1 ; }
[Severity: High]
This is a pre-existing issue, but since you are fixing the drain reentry race for PEBS, does intel_pmu_drain_bts_buffer() need similar protection?
If a user configures a BTS event alongside another active counter and disables the BTS event, the stop IPI can call this sequence without globally disabling the PMU:
__perf_event_stop()
x86_pmu_stop()
intel_pmu_disable_event()
intel_pmu_drain_bts_buffer()
If a PMI interrupts execution precisely between reading the top pointer and resetting the index:
arch/x86/events/intel/ds.c:intel_pmu_drain_bts_buffer() {
...
base = (struct bts_record *)(unsigned long)ds->bts_buffer_base;
top = (struct bts_record *)(unsigned long)ds->bts_index;
if (top <= base)
return 0;
memset(®s, 0, sizeof(regs));
<-- PMI FIRES HERE -->
ds->bts_index = ds->bts_buffer_base;
...
}Could the NMI handler unconditionally call intel_pmu_drain_bts_buffer(), draining the records and resetting the index, only for the interrupted context to resume and push the exact same records again using its stale top pointer?
Would it make sense to wrap the BTS draining logic with the newly introduced __intel_pmu_quiesce() and __intel_pmu_resume() helpers as well?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 56 lines · On 7/17/2026 6:38 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] TOCTOU race inintel_pmu_drain_bts_bufferleading to duplicate BTS records. ----- Patch [8]: [Patch v3 8/8] perf/x86/intel: Prevent
drain_pebs()reentry --- commit77ebceadaa828dfb1bed75e560551cf29b1ae9f5
Author: Dapeng Mi <[email protected]>perf/x86/intel: Prevent
drain_pebs()reentryThe PEBS buffer is shared by all events on a CPU, so
drain_pebs()must not run concurrently. This commit introduces__intel_pmu_quiesce()and__intel_pmu_resume()helpers to disable the full PMU around thedrain_pebs()call, preventing reentry.arch/x86/events/intel/ds.c int intel_pmu_drain_bts_buffer(void) · line 1241
Quoted 7 lines
arch/x86/events/intel/ds.c +0
-0@@ -1240,13 +1240,6 @@ int intel_pmu_drain_bts_buffer(void)return 1 ; }[Severity: High]
This is a pre-existing issue, but since you are fixing the drain reentry race for PEBS, doesintel_pmu_drain_bts_buffer()need similar protection?If a user configures a BTS event alongside another active counter and disables the BTS event, the stop IPI can call this sequence without globally disabling the PMU:
__perf_event_stop()
x86_pmu_stop()
intel_pmu_disable_event()
intel_pmu_drain_bts_buffer()If a PMI interrupts execution precisely between reading the top pointer and resetting the index:
arch/x86/events/intel/ds.c:intel_pmu_drain_bts_buffer() { ... base = (struct bts_record *)(unsigned long)ds->bts_buffer_base; top = (struct bts_record *)(unsigned long)ds->bts_index; if (top <= base) return 0; memset(®s, 0, sizeof(regs)); <-- PMI FIRES HERE --> ds->bts_index = ds->bts_buffer_base; ... }
Quoted 7 lines
Could the NMI handler unconditionally call
intel_pmu_drain_bts_buffer(), draining the records and resetting the index, only for the interrupted context to resume and push the exact same records again using its stale top pointer?Would it make sense to wrap the BTS draining logic with the newly introduced
__intel_pmu_quiesce()and__intel_pmu_resume()helpers as well?
Yeah, that looks a similar race. Considering BTS would not be used broadly, would fix it later in an independent patch.
Thanks.
Quoted 1 line
Peter Zijlstra
maintainerRe: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
In reply to Dapeng MiQuoted 8 lines · Dapeng Mi
The PEBS buffer is shared by all events on a CPU, so
drain_pebs()must not run concurrently. If it is reentered, one instance may observe stale buffer state and potentially access out-of-bound memory.Most invocations happen in NMI context, which naturally prevents reentry. However,
drain_pebs()is also reachable from process context viaintel_pmu_drain_pebs_buffer().
Quoted 9 lines
In those paths, the PMU is often already disabled, but not guaranteed. For example,
__intel_pmu_pebs_disable()only disables the target counter, so other active counters can still raise a PMI and interrupt an in-flightdrain_pebs().Introduce
__intel_pmu_quiesce()and__intel_pmu_resume()helpers and use them inintel_pmu_drain_pebs_buffer()to disable the full PMU around thedrain_pebs()call, preventing reentry.
It is not at all clear to me where the exact recursion happens. (The word you're looking for was recursion, not concurrent).
Re: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Dapeng Mi
The PEBS buffer is shared by all events on a CPU, so
drain_pebs()must not run concurrently. If it is reentered, one instance may observe stale buffer state and potentially access out-of-bound memory.Most invocations happen in NMI context, which naturally prevents reentry. However,
drain_pebs()is also reachable from process context viaintel_pmu_drain_pebs_buffer().In those paths, the PMU is often already disabled, but not guaranteed. For example,
__intel_pmu_pebs_disable()only disables the target counter, so other active counters can still raise a PMI and interrupt an in-flightdrain_pebs().
Quoted 6 lines
Quoted 4 lines
Introduce
__intel_pmu_quiesce()and__intel_pmu_resume()helpers and use them inintel_pmu_drain_pebs_buffer()to disable the full PMU around thedrain_pebs()call, preventing reentry.It is not at all clear to me where the exact recursion happens. (The word you're looking for was recursion, not concurrent).
Yes, the word "concurrently" is not accurate, reentry is the more accurate word.
Currently drain_pebs() would be called in two places, one is the in the PMI handler, like handle_pmi_common(). The other place is intel_pmu_drain_pebs_buffer() which is from process context.
So when intel_pmu_drain_pebs_buffer() is calling drain_pebs(), if there is an active PEBS event triggering PMI, it would interrupt current in-flight drain_pebs() and lead to drain_pebs() reentry.
The good news is the global pmu has been disabled in most places before calling intel_pmu_drain_pebs_buffer(), so no new PMI can be triggered to interrupt current running drain_pebs() helper, but not all places does so, like __intel_pmu_pebs_disable() where only the target counter has been disabled instead of the whole PMU. So it's still possible tjat another active PEBS event triggers PMI and interrupts current running drain_pebs().
Take the intel_pmu_drain_arch_pebs() as an example,
base = cpuc->pebs_vaddr;
top = cpuc->pebs_vaddr + (index.wr << ARCH_PEBS_INDEX_WR_SHIFT); ------> interrupted here ...
index.wr = 0;
index.full = 0;
index.en = 1;
if (cpuc->n_pebs == cpuc->n_large_pebs)
index.thresh = ARCH_PEBS_THRESH_MULTI;
else
index.thresh = ARCH_PEBS_THRESH_SINGLE;
wrmsrq(MSR_IA32_PEBS_INDEX, index.whole);
Assume the drain_pebs() is interrupted just after reading the top value by a new PEBS PMI and the PMI handler would drain all the PEBS buffer. When the PMI returns and the original drian_pebs() continues to execute but it doesn't know the PEBS buffer has been cleared and may access some stale data and lead to some unexpected errors.
Peter Zijlstra
maintainerRe: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 26 lines · Dapeng
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Dapeng Mi
The PEBS buffer is shared by all events on a CPU, so
drain_pebs()must not run concurrently. If it is reentered, one instance may observe stale buffer state and potentially access out-of-bound memory.Most invocations happen in NMI context, which naturally prevents reentry. However,
drain_pebs()is also reachable from process context viaintel_pmu_drain_pebs_buffer().In those paths, the PMU is often already disabled, but not guaranteed. For example,
__intel_pmu_pebs_disable()only disables the target counter, so other active counters can still raise a PMI and interrupt an in-flightdrain_pebs().Quoted 6 lines
Quoted 4 lines
Introduce
__intel_pmu_quiesce()and__intel_pmu_resume()helpers and use them inintel_pmu_drain_pebs_buffer()to disable the full PMU around thedrain_pebs()call, preventing reentry.It is not at all clear to me where the exact recursion happens. (The word you're looking for was recursion, not concurrent).
Yes, the word "concurrently" is not accurate, reentry is the more accurate word.
Quoted 7 lines
Currently
drain_pebs()would be called in two places, one is the in the PMI handler, likehandle_pmi_common(). The other place isintel_pmu_drain_pebs_buffer()which is from process context.So when
intel_pmu_drain_pebs_buffer()is callingdrain_pebs(), if there is an active PEBS event triggering PMI, it would interrupt current in-flightdrain_pebs()and lead todrain_pebs()reentry.
What is the actual callchain here?
Because the one I'm thinking off is:
perf_ctx_disable();
perf_ctx_sched_task_cb()
x86_pmu_sched_task()
intel_pmu_sched_task()
intel_pmu_pebs_sched_task()
intel_pmu_drain_pebs_buffer(); perf_ctx_enable();
And that one has the full pmu disabled, because of context switch etc. And then there is one in perf_read():
pmu->read()
intel_pmu_read_event()
pmu_enabled = cpuc->enabled;
cpuc->enabled = 0;
if (pmu_enabled)
intel_pmu_disable_all();
intel_pmu_drain_pebs_buffer();
cpuc->enabled = pmu_enabled;
if (pmu_enabled)
intel_pmu_enable_all();So what specific callchain is going sideways?
Re: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 58 lines · Peter Zijlstra
Quoted 24 lines · Dapeng
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Dapeng Mi
The PEBS buffer is shared by all events on a CPU, so
drain_pebs()must not run concurrently. If it is reentered, one instance may observe stale buffer state and potentially access out-of-bound memory.Most invocations happen in NMI context, which naturally prevents reentry. However,
drain_pebs()is also reachable from process context viaintel_pmu_drain_pebs_buffer().In those paths, the PMU is often already disabled, but not guaranteed. For example,
__intel_pmu_pebs_disable()only disables the target counter, so other active counters can still raise a PMI and interrupt an in-flightdrain_pebs().Quoted 6 lines
Quoted 4 lines
Introduce
__intel_pmu_quiesce()and__intel_pmu_resume()helpers and use them inintel_pmu_drain_pebs_buffer()to disable the full PMU around thedrain_pebs()call, preventing reentry.It is not at all clear to me where the exact recursion happens. (The word you're looking for was recursion, not concurrent).
Yes, the word "concurrently" is not accurate, reentry is the more accurate word.
Quoted 7 lines
Currently
drain_pebs()would be called in two places, one is the in the PMI handler, likehandle_pmi_common(). The other place isintel_pmu_drain_pebs_buffer()which is from process context.So when
intel_pmu_drain_pebs_buffer()is callingdrain_pebs(), if there is an active PEBS event triggering PMI, it would interrupt current in-flightdrain_pebs()and lead todrain_pebs()reentry.What is the actual callchain here?
Because the one I'm thinking off is:
perf_ctx_disable();
perf_ctx_sched_task_cb()x86_pmu_sched_task() intel_pmu_sched_task() intel_pmu_pebs_sched_task() intel_pmu_drain_pebs_buffer();
perf_ctx_enable();And that one has the full pmu disabled, because of context switch etc. And then there is one in
perf_read():pmu->read() intel_pmu_read_event() pmu_enabled = cpuc->enabled; cpuc->enabled = 0; if (pmu_enabled) intel_pmu_disable_all(); intel_pmu_drain_pebs_buffer(); cpuc->enabled = pmu_enabled; if (pmu_enabled) intel_pmu_enable_all();
Quoted 1 line
So what specific callchain is going sideways?
Yes, the whole PMU would be disabled for these 2 call-chains.
But for the helper __intel_pmu_pebs_disable(), it seems the whole PMU is not disabled, only the target counter has been stopped. Here is one call-chain.
__perf_addr_filters_adjust()
perf_event_stop()
__perf_event_stop()
x86_pmu_stop() (event->pmu->stop)
intel_pmu_disable_event()
intel_pmu_pebs_disable()
__intel_pmu_pebs_disable()
intel_pmu_drain_large_pebs()
intel_pmu_drain_pebs_buffer()
Quoted 1 line
Peter Zijlstra
maintainerRe: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 5 lines · Dapeng
Quoted 1 line
So what specific callchain is going sideways?
But for the helper
__intel_pmu_pebs_disable(), it seems the whole PMU is not disabled, only the target counter has been stopped. Here is one call-chain.
Quoted 10 lines
__perf_addr_filters_adjust()
perf_event_stop()
__perf_event_stop()
x86_pmu_stop()(event->pmu->stop)
intel_pmu_disable_event()
intel_pmu_pebs_disable()
__intel_pmu_pebs_disable()
intel_pmu_drain_large_pebs()
intel_pmu_drain_pebs_buffer()
Right, so that needs to be included in the changelog. Also, lets target that more specifically.
How about something like so?
---
arch/x86/events/intel/core.c +24 -9
@@ -3125,6 +3125,27 @@ static void intel_pmu_del_event(struct perf_event *event)this_cpu_ptr (&cpu_hw_events)->n_late_setup--; } +int __intel_pmu_quiesce (void ) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +int pmu_enabled = cpuc->enabled; + + cpuc->enabled =0 ; +if (pmu_enabled) +intel_pmu_disable_all (); + +return pmu_enabled; +} + +void __intel_pmu_resume (bool pmu_enabled) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->enabled = pmu_enabled; +if (pmu_enabled) +intel_pmu_enable_all (0 ); +} +static int icl_set_topdown_event_period (struct perf_event *event) {struct hw_perf_event *hwc = &event->hw;
@@ -3316,16 +3337,13 @@ static void intel_pmu_read_event(struct perf_event *event)if (event->hw.flags & (PERF_X86_EVENT_AUTO_RELOAD |PERF_X86_EVENT_TOPDOWN ) ||is_pebs_counter_event_group (event)) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -bool pmu_enabled = cpuc->enabled; +int pmu_enabled;/* Only need to call update_topdown_event() once for group read. */ if (is_metric_event (event) && (cpuc->txn_flags &PERF_PMU_TXN_READ ))return ; - cpuc->enabled =0 ; -if (pmu_enabled) -intel_pmu_disable_all (); - + pmu_enabled =__intel_pmu_quiesce ();/* * If the PEBS counters snapshotting is enabled, * the topdown event is available in PEBS records.
@@ -3334,10 +3352,7 @@ static void intel_pmu_read_event(struct perf_event *event)static_call (intel_pmu_update_topdown_event)(event,NULL ); elseintel_pmu_drain_pebs_buffer (); - - cpuc->enabled = pmu_enabled; -if (pmu_enabled) -intel_pmu_enable_all (0 ); +__intel_pmu_resume (pmu_enabled);return ; }
arch/x86/events/intel/ds.c +7 -1
@@ -1242,8 +1242,11 @@ int intel_pmu_drain_bts_buffer(void)void intel_pmu_drain_pebs_buffer (void ) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_sample_data data; +WARN_ON_ONCE (cpuc->enabled); +static_call (x86_pmu_drain_pebs)(NULL , &data); }
@@ -1864,8 +1867,11 @@ static void intel_pmu_pebs_via_pt_enable(struct perf_event *event)static inline void intel_pmu_drain_large_pebs (struct cpu_hw_events *cpuc) {if (cpuc->n_pebs == cpuc->n_large_pebs && - cpuc->n_pebs != cpuc->n_pebs_via_pt) + cpuc->n_pebs != cpuc->n_pebs_via_pt) { +int enabled =__intel_pmu_quiesce ();intel_pmu_drain_pebs_buffer (); +__intel_pmu_resume (enabled); + } }static void __intel_pmu_pebs_enable (struct perf_event *event)
arch/x86/events/perf_event.h +3 -0
@@ -1644,6 +1644,9 @@ static __always_inline void __intel_pmu_lbr_disable(void)wrmsrq (MSR_IA32_DEBUGCTLMSR , debugctl); } +extern int __intel_pmu_quiesce (void ); +extern void __intel_pmu_resume (bool pmu_enabled); +int intel_pmu_save_and_restart (struct perf_event *event); struct event_constraint *
Re: [Patch v3 8/8] perf/x86/intel: Prevent drain_pebs() reentry
Quoted 16 lines · Peter Zijlstra
Quoted 4 lines · Dapeng
Quoted 1 line
So what specific callchain is going sideways?
But for the helper
__intel_pmu_pebs_disable(), it seems the whole PMU is not disabled, only the target counter has been stopped. Here is one call-chain.Quoted 10 lines
__perf_addr_filters_adjust()
perf_event_stop()
__perf_event_stop()
x86_pmu_stop()(event->pmu->stop)
intel_pmu_disable_event()
intel_pmu_pebs_disable()
__intel_pmu_pebs_disable()
intel_pmu_drain_large_pebs()
intel_pmu_drain_pebs_buffer()
Quoted 2 lines
Right, so that needs to be included in the changelog. Also, lets target that more specifically.
Sure.
Quoted 2 lines
How about something like so?
It looks good to me. Would post a new version. Thanks.
Quoted 109 lines
---
arch/x86/events/intel/core.c +24
-9@@ -3125,6 +3125,27 @@ static void intel_pmu_del_event(struct perf_event *event)this_cpu_ptr (&cpu_hw_events)->n_late_setup--; } +int __intel_pmu_quiesce (void ) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +int pmu_enabled = cpuc->enabled; + + cpuc->enabled =0 ; +if (pmu_enabled) +intel_pmu_disable_all (); + +return pmu_enabled; +} + +void __intel_pmu_resume (bool pmu_enabled) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->enabled = pmu_enabled; +if (pmu_enabled) +intel_pmu_enable_all (0 ); +} +static int icl_set_topdown_event_period (struct perf_event *event) {struct hw_perf_event *hwc = &event->hw;@@ -3316,16 +3337,13 @@ static void intel_pmu_read_event(struct perf_event *event)if (event->hw.flags & (PERF_X86_EVENT_AUTO_RELOAD |PERF_X86_EVENT_TOPDOWN ) ||is_pebs_counter_event_group (event)) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -bool pmu_enabled = cpuc->enabled; +int pmu_enabled;/* Only need to call update_topdown_event() once for group read. */ if (is_metric_event (event) && (cpuc->txn_flags &PERF_PMU_TXN_READ ))return ; - cpuc->enabled =0 ; -if (pmu_enabled) -intel_pmu_disable_all (); - + pmu_enabled =__intel_pmu_quiesce ();/* * If the PEBS counters snapshotting is enabled, * the topdown event is available in PEBS records. @@ -3334,10 +3352,7 @@ static void intel_pmu_read_event(struct perf_event *event)static_call (intel_pmu_update_topdown_event)(event,NULL ); elseintel_pmu_drain_pebs_buffer (); - - cpuc->enabled = pmu_enabled; -if (pmu_enabled) -intel_pmu_enable_all (0 ); +__intel_pmu_resume (pmu_enabled);return ; }arch/x86/events/intel/ds.c +7
-1@@ -1242,8 +1242,11 @@ int intel_pmu_drain_bts_buffer(void)void intel_pmu_drain_pebs_buffer (void ) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_sample_data data; +WARN_ON_ONCE (cpuc->enabled); +static_call (x86_pmu_drain_pebs)(NULL , &data); }@@ -1864,8 +1867,11 @@ static void intel_pmu_pebs_via_pt_enable(struct perf_event *event)static inline void intel_pmu_drain_large_pebs (struct cpu_hw_events *cpuc) {if (cpuc->n_pebs == cpuc->n_large_pebs && - cpuc->n_pebs != cpuc->n_pebs_via_pt) + cpuc->n_pebs != cpuc->n_pebs_via_pt) { +int enabled =__intel_pmu_quiesce ();intel_pmu_drain_pebs_buffer (); +__intel_pmu_resume (enabled); + } }static void __intel_pmu_pebs_enable (struct perf_event *event)arch/x86/events/perf_event.h +3
-0@@ -1644,6 +1644,9 @@ static __always_inline void __intel_pmu_lbr_disable(void)wrmsrq (MSR_IA32_DEBUGCTLMSR , debugctl); } +extern int __intel_pmu_quiesce (void ); +extern void __intel_pmu_resume (bool pmu_enabled); +int intel_pmu_save_and_restart (struct perf_event *event); struct event_constraint *
For the whole series:
Quoted 51 lines · Dapeng Mi
Changes since v2:
- Patch 6/8: Fix the typo in comments (Thomas).
- Patch 8/8: New patch which preventsdrain_pebs()reentry.Changes since v1:
- Patch 3/7: Ensurecpuc->pmuis not the static pmu before callinghybrid_pmu(cpuc->pmu)inintel_pmu_cpu_dead()(Sashiko).This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/The patch 7/8 optimizes ACR handling in
match_prev_assignment()and mitigate the performance overhead.The patch 8/8 enhances
intel_pmu_drain_pebs_buffer()to preventdrain_pebs()reentry.Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156History:
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Dapeng Mi (8):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure
perf/x86: Guardintel_pmu_cpu_dead()against invalid hybrid PMU casts perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access perf/x86/intel: Fixintel_caphandling on hybrid PMUs perf/x86: Optimize ACR handling inmatch_prev_assignment()perf/x86/intel: Preventdrain_pebs()reentryarch/x86/events/core.c | 45 ++++++++++++---- arch/x86/events/intel/core.c | 101 ++++++++++++++++++++++++++--------- arch/x86/events/intel/ds.c | 7 --- arch/x86/events/perf_event.h | 12 ++--- 4 files changed, 113 insertions(+), 52 deletions(-)base-commit:
edda9051e267b7390c7ce24b1b71434414ad156e
Peter,
Kindly ping. Could you please review this bug fixing patch series? The changes in this series suppose to be low risky.
This series is a preceding patchset of the "Support SIMD/eGPRs/SSP registers sampling for perf" patch series.
https://lore.kernel.org/all/[email protected]/
Thanks.
Quoted 51 lines · Dapeng Mi
Changes since v2:
- Patch 6/8: Fix the typo in comments (Thomas).
- Patch 8/8: New patch which preventsdrain_pebs()reentry.Changes since v1:
- Patch 3/7: Ensurecpuc->pmuis not the static pmu before callinghybrid_pmu(cpuc->pmu)inintel_pmu_cpu_dead()(Sashiko).This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/The patch 7/8 optimizes ACR handling in
match_prev_assignment()and mitigate the performance overhead.The patch 8/8 enhances
intel_pmu_drain_pebs_buffer()to preventdrain_pebs()reentry.Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156History:
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Dapeng Mi (8):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure
perf/x86: Guardintel_pmu_cpu_dead()against invalid hybrid PMU casts perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access perf/x86/intel: Fixintel_caphandling on hybrid PMUs perf/x86: Optimize ACR handling inmatch_prev_assignment()perf/x86/intel: Preventdrain_pebs()reentryarch/x86/events/core.c | 45 ++++++++++++---- arch/x86/events/intel/core.c | 101 ++++++++++++++++++++++++++--------- arch/x86/events/intel/ds.c | 7 --- arch/x86/events/perf_event.h | 12 ++--- 4 files changed, 113 insertions(+), 52 deletions(-)base-commit:
edda9051e267b7390c7ce24b1b71434414ad156e