perf/x86: Miscellaneous PMU bug fixes and optimizations
Subsystems (2)
v2 cover perf/x86: Miscellaneous PMU bug fixes and optimizations
Changes since v1:
- Patch 3/7: Ensure cpuc->pmu is not the static pmu before calling hybrid_pmu(cpuc->pmu) in intel_pmu_cpu_dead() (Sashiko).
This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
The last patch 7/7 optimizes ACR handling in match_prev_assignment() and mitigate the performance overhead.
Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test
$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156
History:
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (7):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access perf/x86/intel: Fix intel_cap handling on hybrid PMUs perf/x86: Optimize ACR handling in match_prev_assignment()
arch/x86/events/core.c | 45 +++++++++++++++++++------- arch/x86/events/intel/core.c | 61 ++++++++++++++++++++++++++---------- arch/x86/events/perf_event.h | 12 ++----- 3 files changed, 81 insertions(+), 37 deletions(-)
base-commit: edda9051e267b7390c7ce24b1b71434414ad156e
Signature · 1 line
2.34.1
PATCH v2 1/7 perf/x86: Unregister PMI handler on PMU init failure
Fix an NMI handler leak in init_hw_perf_events(). When PMU initialization fails after register_nmi_handler(), the error path exits without calling unregister_nmi_handler(), leaving a stale NMI_LOCAL "PMI" handler registered. Add the missing call before clearing x86_pmu state.
--- arch/x86/events/core.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/events/core.c +3 -1
@@ -2219,7 +2219,7 @@ static int __init init_hw_perf_events(void) err =cpuhp_setup_state (CPUHP_PERF_X86_PREPARE ,"perf/x86:prepare" , x86_pmu_prepare_cpu, x86_pmu_dead_cpu);if (err) -return err; + goto pmi_unregister; err = cpuhp_setup_state(CPUHP_AP_PERF_X86_STARTING, "perf/x86:starting", x86_pmu_starting_cpu,
@@ -2273,6 +2273,8 @@ static int __init init_hw_perf_events(void)cpuhp_remove_state (CPUHP_AP_PERF_X86_STARTING ); out:cpuhp_remove_state (CPUHP_PERF_X86_PREPARE ); +pmi_unregister: +unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu:memset (&x86_pmu,0 ,sizeof (x86_pmu));return err;
Signature · 1 line
2.34.1
PATCH v2 2/7 perf/x86: Free hybrid state on PMU init failure
In reply to Dapeng MiIf PMU initialization fails, for example in check_hw_exists(), hybrid state can be left partially initialized: x86_pmu.hybrid_pmu is not freed and perf_is_hybrid remains set. This can leak memory and leave stale hybrid state reachable after a failed init path.
Add x86_pmu_free_hybrid() and use it on PMU init failure paths so all hybrid-related state is consistently reset.
--- arch/x86/events/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/events/core.c +12 -3
@@ -2130,6 +2130,17 @@ void x86_pmu_show_pmu_cap(struct pmu *pmu)pr_info ("... global_ctrl mask: %016llx\n" ,hybrid (pmu, intel_ctrl)); } +static void x86_pmu_free_hybrid (void ) +{ +if (!x86_pmu.hybrid_pmu) +return ; + +static_branch_disable (&perf_is_hybrid); +kfree (x86_pmu.hybrid_pmu); + x86_pmu.hybrid_pmu =NULL ; + x86_pmu.num_hybrid_pmus =0 ; +} +static int __init init_hw_perf_events (void ) {struct x86_pmu_quirk *quirk;
@@ -2258,9 +2269,6 @@ static int __init init_hw_perf_events(void)for (j =0 ; j < i; j++)perf_pmu_unregister (&x86_pmu.hybrid_pmu[j].pmu);pr_warn ("Failed to register hybrid PMUs\n" ); -kfree (x86_pmu.hybrid_pmu); - x86_pmu.hybrid_pmu =NULL ; - x86_pmu.num_hybrid_pmus =0 ; goto out2; } }
@@ -2276,6 +2284,7 @@ static int __init init_hw_perf_events(void) pmi_unregister:unregister_nmi_handler (NMI_LOCAL ,"PMI" ); out_bad_pmu: +x86_pmu_free_hybrid ();memset (&x86_pmu,0 ,sizeof (x86_pmu));return err; }
Signature · 1 line
2.34.1
PATCH v2 3/7 perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
In reply to Dapeng MiIn failure paths, cpuc->pmu can still point to the global static pmu instead of an embedded x86_hybrid_pmu::pmu. Calling hybrid_pmu() on that pointer causes an invalid container conversion and may lead to out-of-bounds access.
This can happen in at least two cases:
- init_hybrid_pmu() fails check_hw_exists() and leaves cpuc->pmu as-is.
- CPU hotplug fails between CPUHP_PERF_X86_PREPARE and CPUHP_AP_PERF_X86_STARTING, and rollback invokes intel_pmu_cpu_dead().
Fix both paths by:
- Clear cpuc->pmu to NULL when check_hw_exists() fails.
- Validat that cpuc->pmu is not the global static pmu before calling hybrid_pmu() in intel_pmu_cpu_dead().
A new helper x86_get_static_pmu() is added to get the global static pmu.
--- arch/x86/events/core.c | 5 +++++ arch/x86/events/intel/core.c | 7 +++++-- arch/x86/events/perf_event.h | 1 + 3 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/core.c +5 -0
@@ -790,6 +790,11 @@ int is_x86_event(struct perf_event *event)return false; } +inline struct pmu *x86_get_static_pmu (void ) +{ +return &pmu; +} +struct pmu *x86_get_pmu (unsigned int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);
arch/x86/events/intel/core.c +5 -2
@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;return false; + }pr_info ("%s PMU driver: " , pmu->name);
@@ -6475,11 +6477,12 @@ void intel_cpuc_finish(struct cpu_hw_events *cpuc)static void intel_pmu_cpu_dead (int cpu) {struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu); +struct pmu *pmu =x86_get_static_pmu ();release_arch_pebs_buf_on_cpu (cpu);intel_cpuc_finish (cpuc); -if (is_hybrid () && cpuc->pmu) +if (is_hybrid () && cpuc->pmu && cpuc->pmu != pmu)cpumask_clear_cpu (cpu, &hybrid_pmu (cpuc->pmu)->supported_cpus); }
arch/x86/events/perf_event.h +1 -0
@@ -1161,6 +1161,7 @@ static struct perf_pmu_format_hybrid_attr format_attr_hybrid_##_name = {\ .pmu_type = _pmu, \ } +struct pmu *x86_get_static_pmu (void );struct pmu *x86_get_pmu (unsigned int cpu);extern struct x86_pmu x86_pmu__read_mostly ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v2 3/7] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Race condition in NMI handler causing out-of-bounds access during hybrid CPU bring-up.
- [High] Unhandled #GP exceptions (kernel panic) in NMI and hotplug paths if hybrid PMU initialization fails. --
--- Patch [3]: [Patch v2 3/7] perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts --- commit 419f89d32dfc7abe942483376114f8e4845bc1a8
Author: Dapeng Mi <[email protected]>
perf/x86: Guard intel_pmu_cpu_dead() against invalid hybrid PMU casts
In failure paths, cpuc->pmu can point to the global static pmu instead of an embedded x86_hybrid_pmu::pmu. This commit clears cpuc->pmu to NULL when check_hw_exists() fails and validates cpuc->pmu before hybrid container conversion during CPU rollback paths.
Quoted 4 lines
arch/x86/events/intel/core.c +0
-0
[ ... ]
arch/x86/events/intel/core.c static bool init_hybrid_pmu(int cpu) · line 6335
Quoted 9 lines
arch/x86/events/intel/core.c +3
-1@@ -6329,8 +6329,10 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) +if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { + cpuc->pmu =NULL ;return false; + }
[Severity: High]
This is a pre-existing issue, but is it possible for an NMI to fire before this initialization completes during hybrid CPU bring-up?
If a secondary CPU unmasks NMIs early and an NMI fires before the CPUHP_AP_PERF_X86_STARTING callback executes init_hybrid_pmu(), perf_event_nmi_handler() will run.
If active_events is greater than zero globally, intel_pmu_handle_irq() executes:
arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
...
bool late_ack = hybrid_bit(cpuc->pmu, late_ack);
...
}Since cpuc->pmu is statically initialized to the global &pmu at this stage, could this cause hybrid_pmu() to perform an invalid container_of cast and lead to an out-of-bounds access?
[Severity: High]
This is a pre-existing issue, but if check_hw_exists() fails and we return false here, leaving the CPU online without a functioning PMU, could this lead to unhandled #GP exceptions?
If an NMI fires while active_events is greater than zero globally, intel_pmu_handle_irq() unconditionally reads PMU MSRs:
arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
...
status = intel_pmu_get_status();
...
}Since check_hw_exists() failed, the MSRs are likely inaccessible. Would executing rdmsrl() trigger a #GP exception and a kernel panic?
Similarly, during CPU teardown, does intel_pmu_cpu_dying() blindly write to MSRs without checking if the PMU was successfully initialized?
arch/x86/events/intel/ds.c:fini_arch_pebs_on_cpu() {
...
wrmsrq_on_cpu(cpu, MSR_IA32_PEBS_BASE, 0);
...
}Could this wrmsrq_on_cpu() without fault protection also cause a #GP panic if PMU initialization had failed?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v2 4/7 perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails
In reply to Dapeng Miintel_pmu_cpu_prepare() allocates per-CPU perf state first and then sets up the arch PEBS buffer. If alloc_arch_pebs_buf_on_cpu() fails, the previously allocated cpuc resources are left behind.
Make the failure path call intel_cpuc_finish(cpuc) to release the per-CPU state allocated by intel_cpuc_prepare().
--- arch/x86/events/intel/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +9 -2
@@ -5924,13 +5924,20 @@ int intel_cpuc_prepare(struct cpu_hw_events *cpuc, int cpu)static int intel_pmu_cpu_prepare (int cpu) { +struct cpu_hw_events *cpuc = &per_cpu (cpu_hw_events, cpu);int ret; - ret =intel_cpuc_prepare (&per_cpu (cpu_hw_events, cpu), cpu); + ret =intel_cpuc_prepare (cpuc, cpu);if (ret)return ret; -return alloc_arch_pebs_buf_on_cpu (cpu); + ret =alloc_arch_pebs_buf_on_cpu (cpu); +if (ret) { +intel_cpuc_finish (cpuc); +return ret; + } + +return 0 ; }static void flip_smm_bit (void *data)
Signature · 1 line
2.34.1
PATCH v2 5/7 perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
In reply to Dapeng MiOn hybrid systems, init_hw_perf_events() can call check_hw_exists() with the global PMU pointer after perf_is_hybrid is set. In that case, fixed_counter_disabled() uses hybrid() on a non-hybrid PMU object, so the intel_ctrl access is taken from the wrong layout and can read out of bounds.
fixed_counter_disabled() was added in commit 32451614da2a ("perf/x86/intel: Support CPUID 10.ECX to disable fixed counters"), when fixed counters were tracked via num_fixed_counters. Today fixed counters are represented by fixed_cntr_mask, so this helper is obsolete.
Remove fixed_counter_disabled() and its callers, and rely directly on the fixed-counter bitmask. With the helper gone, check_hw_exists() no longer needs a PMU argument, so drop that parameter as well. This removes the invalid hybrid access and closes the out-of-bounds read risk.
--- arch/x86/events/core.c | 8 ++------ arch/x86/events/intel/core.c | 4 +--- arch/x86/events/perf_event.h | 9 +-------- 3 files changed, 4 insertions(+), 17 deletions(-)
arch/x86/events/core.c +2 -6
@@ -265,7 +265,7 @@ static void release_pmc_hardware(void) {}#endif -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask) {u64 val, val_fail =-1 , val_new= ~0 ;
@@ -297,8 +297,6 @@ bool check_hw_exists(struct pmu *pmu, unsigned long *cntr_mask,if (ret) goto msr_fail;for_each_set_bit (i, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (i, pmu)) -continue ;if (val & (0x03ULL << i*4 )) { bios_fail =1 ; val_fail = val;
@@ -1618,8 +1616,6 @@ void perf_event_print_debug(void) cpu, idx, prev_left); }for_each_set_bit (idx, fixed_cntr_mask,X86_PMC_IDX_MAX ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;rdmsrq (x86_pmu_fixed_ctr_addr (idx), pmc_count); pr_info("CPU#%d: fixed-PMC%d count: %016llx\n",
@@ -2180,7 +2176,7 @@ static int __init init_hw_perf_events(void)pmu_check_apic ();/* sanity check that the hardware exists or is emulated */ -if (!check_hw_exists (&pmu, x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) +if (!check_hw_exists (x86_pmu.cntr_mask, x86_pmu.fixed_cntr_mask)) goto out_bad_pmu;pr_cont ("%s PMU driver.\n" , x86_pmu.name);
arch/x86/events/intel/core.c +1 -3
@@ -3713,8 +3713,6 @@ static void intel_pmu_reset(void)wrmsrq_safe (x86_pmu_event_addr (idx),0ull ); }for_each_set_bit (idx, fixed_cntr_mask,INTEL_PMC_MAX_FIXED ) { -if (fixed_counter_disabled (idx, cpuc->pmu)) -continue ;wrmsrq_safe (x86_pmu_fixed_ctr_addr (idx),0ull ); }
@@ -6336,7 +6334,7 @@ static bool init_hybrid_pmu(int cpu)intel_pmu_check_hybrid_pmus (pmu); -if (!check_hw_exists (&pmu->pmu, pmu->cntr_mask, pmu->fixed_cntr_mask)) { +if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) { cpuc->pmu =NULL ;return false; }
arch/x86/events/perf_event.h +1 -8
@@ -1243,7 +1243,7 @@ static inline int x86_pmu_rdpmc_index(int index)return x86_pmu.rdpmc_index ? x86_pmu.rdpmc_index (index) : index; } -bool check_hw_exists (struct pmu *pmu,unsigned long *cntr_mask, +bool check_hw_exists (unsigned long *cntr_mask,unsigned long *fixed_cntr_mask);int x86_add_exclusive (unsigned int what);
@@ -1456,13 +1456,6 @@ ssize_t events_hybrid_sysfs_show(struct device *dev,struct device_attribute *attr, char *page); -static inline bool fixed_counter_disabled (int i,struct pmu *pmu) -{ -u64 intel_ctrl =hybrid (pmu, intel_ctrl); - -return !(intel_ctrl >> (i +INTEL_PMC_IDX_FIXED )); -} -#ifdef CONFIG_CPU_SUP_AMD int amd_pmu_init (void );
Signature · 1 line
2.34.1
PATCH v2 6/7 perf/x86/intel: Fix intel_cap handling on hybrid PMUs
In reply to Dapeng Miintel_cap (IA32_PERF_CAPABILITIES) updates are currently tied to X86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths. As a result, hybrid PMU capability state can be updated under the wrong condition.
Also, intel_pmu_broken_perf_cap() is too narrow. Per RPL018, the missing PERF_METRICS_AVAILABLE bit affects both Raptor Lake and Meteor Lake parts, not only the currently covered subset.
Move intel_cap updates out of the ARCH_PERFMON_EXT-gated path, extend intel_pmu_broken_perf_cap() coverage to both RPL and MTL families, and introduce intel_update_pmu_caps() to centralize PMU capability updates.
--- arch/x86/events/intel/core.c | 41 ++++++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 11 deletions(-)
arch/x86/events/intel/core.c +30 -11
@@ -6147,8 +6147,15 @@ static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs);static inline bool intel_pmu_broken_perf_cap (void ) { -/* The Perf Metric (Bit 15) is always cleared */ -if (boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || +/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* PRL and MTL. Details can be found in RPL018 Errata Details. +* https://edc.intel.com/content/www/us/en/design/products/platforms/details/raptor-lake-s/13th-generation-core-processor-specification-update/errata-details/ +*/ +if (boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_P || + boot_cpu_data.x86_vfm ==INTEL_RAPTORLAKE_S || + boot_cpu_data.x86_vfm ==INTEL_METEORLAKE || boot_cpu_data.x86_vfm ==INTEL_METEORLAKE_L )return true;
@@ -6183,7 +6190,7 @@ static inline void __intel_update_large_pebs_flags(struct pmu *pmu)#define counter_mask (_gp, _fixed) ((_gp) | ((u64)(_fixed) << INTEL_PMC_IDX_FIXED)) -static void update_pmu_cap (struct pmu *pmu) +static void update_pmu_cap_from_perfmonext (struct pmu *pmu) {unsigned int eax, ebx, ecx, edx;union cpuid35_eax eax_0;
@@ -6241,10 +6248,24 @@ static void update_pmu_cap(struct pmu *pmu)WARN_ON (x86_pmu.arch_pebs ==1 ); x86_pmu.arch_pebs =0 ; } +} + +static void intel_update_pmu_caps (struct pmu *pmu) +{ +if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) +update_pmu_cap_from_perfmonext (pmu); -if (!intel_pmu_broken_perf_cap ()) { -/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */ -rdmsrq (MSR_IA32_PERF_CAPABILITIES ,hybrid (pmu, intel_cap).capabilities); +if (is_hybrid () &&this_cpu_has (X86_FEATURE_PDCM )) { +rdmsrq (MSR_IA32_PERF_CAPABILITIES , +hybrid (pmu, intel_cap).capabilities); + +/* +* Restore perf_metrics on platforms with broken +* perf_capablities. +*/ +if (intel_pmu_broken_perf_cap () && +hybrid_pmu (pmu)->pmu_type == hybrid_big) +hybrid (pmu, intel_cap).perf_metrics =1 ; } }
@@ -6329,9 +6350,7 @@ static bool init_hybrid_pmu(int cpu)if (!cpumask_empty (&pmu->supported_cpus)) goto end; -if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (&pmu->pmu); - +intel_update_pmu_caps (&pmu->pmu);intel_pmu_check_hybrid_pmus (pmu);if (!check_hw_exists (pmu->cntr_mask, pmu->fixed_cntr_mask)) {
@@ -8828,8 +8847,8 @@ __init int intel_pmu_init(void)* from the leaf 0xa. The core specific update will be done later * when a new type is online. */ -if (!is_hybrid () &&boot_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (NULL ); +if (!is_hybrid ()) +intel_update_pmu_caps (NULL );if (x86_pmu.arch_pebs) { static_call_update(intel_pmu_disable_event_ext,
Signature · 1 line
2.34.1
Re: [Patch v2 6/7] perf/x86/intel: Fix intel_cap handling on hybrid PMUs
Quoted 39 lines · Dapeng Mi
intel_cap(IA32_PERF_CAPABILITIES) updates are currently tied toX86_FEATURE_ARCH_PERFMON_EXT, but these are independent feature paths.
As a result, hybrid PMU capability state can be updated under the wrong
condition.Also,
intel_pmu_broken_perf_cap()is too narrow. Per RPL018, the missingPERF_METRICS_AVAILABLEbit affects both Raptor Lake and Meteor Lake parts, not only the currently covered subset.Move
intel_capupdates out of the ARCH_PERFMON_EXT-gated path, extendintel_pmu_broken_perf_cap()coverage to both RPL and MTL families, and
introduceintel_update_pmu_caps()to centralize PMU capability updates.Signed-off-by Dapeng Mi <[email protected]>---
arch/x86/events/intel/core.c | 41 ++++++++++++++++++++++++++-------- --1 file changed, 30 insertions(+), 11 deletions(-)arch/x86/events/intel/core.c +0
-0@@ -6147,8 +6147,15 @@ static void intel_pmu_check_extra_regs(struct extra_reg *extra_regs);
static inline boolintel_pmu_broken_perf_cap(void){arch/x86/events/intel/core.c +3
-2-/* The Perf Metric (Bit 15) is always cleared */ - if (boot_cpu_data.x86_vfm == INTEL_METEORLAKE || +/* +* The Perf Metric (Bit 15) is always cleared on P-core of +* PRL and MTL. Details can be found in RPL018 Errata
Quoted 1 line
Details.
Typo here. Should be RPL.
Thanks,
Tom
Quoted 76 lines
arch/x86/events/intel/core.c +1
-0+* arch/x86/events/intel/core.c +5
-0+*/ + if (boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE || + boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE_P || + boot_cpu_data.x86_vfm == INTEL_RAPTORLAKE_S || + boot_cpu_data.x86_vfm == INTEL_METEORLAKE ||boot_cpu_data.x86_vfm ==
INTEL_METEORLAKE_L) return true;arch/x86/events/intel/core.c +0
-0@@ -6183,7 +6190,7 @@ static inline void __intel_update_large_pebs_flags(struct pmu *pmu)
#definecounter_mask(_gp, _fixed)((_gp) | ((u64)(_fixed) <<INTEL_PMC_IDX_FIXED))arch/x86/events/intel/core.c +1
-1-static void update_pmu_cap (struct pmu *pmu) +static void update_pmu_cap_from_perfmonext (struct pmu *pmu){
unsigned int eax, ebx, ecx, edx;
unioncpuid35_eaxeax_0;arch/x86/events/intel/core.c +6
-0@@ -6241,10 +6248,24 @@ static void update_pmu_cap(struct pmu *pmu) WARN_ON(x86_pmu.arch_pebs == 1); x86_pmu.arch_pebs = 0; } +} + +static void intel_update_pmu_caps (struct pmu *pmu) +{ +if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) +update_pmu_cap_from_perfmonext (pmu);arch/x86/events/intel/core.c +0
-2-if (!intel_pmu_broken_perf_cap ()) { -/* Perf Metric (Bit 15) and PEBS via PT (Bit 16) are hybrid enumeration */
arch/x86/events/intel/core.c +0
-1- rdmsrq(MSR_IA32_PERF_CAPABILITIES, hybrid(pmu,intel_cap).capabilities);
arch/x86/events/intel/core.c +11
-0+if (is_hybrid () &&this_cpu_has (X86_FEATURE_PDCM )) { +rdmsrq (MSR_IA32_PERF_CAPABILITIES , + hybrid(pmu, intel_cap).capabilities); + +/* +* Restore perf_metrics on platforms with broken +* perf_capablities. +*/ +if (intel_pmu_broken_perf_cap () && + hybrid_pmu(pmu)->pmu_type == hybrid_big) +hybrid (pmu, intel_cap).perf_metrics =1 ;}
}arch/x86/events/intel/core.c +0
-0@@ -6329,9 +6350,7 @@ static bool init_hybrid_pmu(int cpu) if (!cpumask_empty(&pmu->supported_cpus)) goto end;arch/x86/events/intel/core.c +1
-3-if (this_cpu_has (X86_FEATURE_ARCH_PERFMON_EXT )) -update_pmu_cap (&pmu->pmu); - +intel_update_pmu_caps (&pmu->pmu);
intel_pmu_check_hybrid_pmus(pmu);
if (!check_hw_exists(pmu->cntr_mask,pmu->fixed_cntr_mask)) {arch/x86/events/intel/core.c +0
-0@@ -8828,8 +8847,8 @@ __init int intel_pmu_init(void) * from the leaf 0xa. The core specific update will be donelater
* when a new type is online.
*/arch/x86/events/intel/core.c +0
-1- if (!is_hybrid() &&
boot_cpu_has(X86_FEATURE_ARCH_PERFMON_EXT))arch/x86/events/intel/core.c +2
-1-update_pmu_cap (NULL ); +if (!is_hybrid ()) +intel_update_pmu_caps (NULL );
if (x86_pmu.arch_pebs) { static_call_update(intel_pmu_disable_event_ext,
PATCH v2 7/7 perf/x86: Optimize ACR handling in match_prev_assignment()
In reply to Dapeng Mimatch_prev_assignment() currently forces a mismatch for ACR events, so ACR counter indices are reprogrammed on every scheduling pass. That causes avoidable overhead because disable and enable paths must touch multiple MSRs.
The previous ACR assignment is already cached in acr_cfg_b[]. Use that state to compare the newly computed ACR counter indices in hwc->config1 against the cached value in acr_cfg_b[hwc->idx]. If they match, skip unnecessary disable and enable work.
Also tighten is_acr_self_reload_event() so it first verifies the event is an ACR event before testing for the self-reload case.
--- arch/x86/events/core.c | 13 ++++++++++++- arch/x86/events/perf_event.h | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/events/core.c +12 -1
@@ -1297,6 +1297,17 @@ int x86_perf_rdpmc_index(struct perf_event *event)return event->hw.event_base_rdpmc; } +static inline bool acr_match_prev_indices (struct perf_event *event, +struct cpu_hw_events *cpuc) +{ +struct hw_perf_event *hwc = &event->hw; + +if (!is_acr_event_group (event)) +return true; +/* ACR counter indices don't change. */ +return hwc->config1 == cpuc->acr_cfg_b[hwc->idx]; +} +static inline int match_prev_assignment (struct perf_event *event,struct cpu_hw_events *cpuc,int i)
@@ -1306,7 +1317,7 @@ static inline int match_prev_assignment(struct perf_event *event,return hwc->idx == cpuc->assign[i] && hwc->last_cpu ==smp_processor_id () && hwc->last_tag == cpuc->tags[i] && - !is_acr_event_group (event); +acr_match_prev_indices (event, cpuc); }static void x86_pmu_start (struct perf_event *event,int flags);
arch/x86/events/perf_event.h +1 -1
@@ -141,7 +141,7 @@ static inline bool is_acr_self_reload_event(struct perf_event *event) {struct hw_perf_event *hwc = &event->hw; -if (hwc->idx <0 ) +if (hwc->idx <0 || !is_acr_event_group (event))return false;return test_bit (hwc->idx, (unsigned long *)&hwc->config1);
Signature · 1 line
2.34.1
Hi Peter,
Not sure if you have bandwidth to review the patch-set? This patchset fixes some recently found issues by Sashiko. Most of issues are error handling related. Thanks.
Quoted 41 lines · Dapeng Mi
Changes since v1:
- Patch 3/7: Ensurecpuc->pmuis not the static pmu before callinghybrid_pmu(cpuc->pmu)inintel_pmu_cpu_dead()(Sashiko).This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/The last patch 7/7 optimizes ACR handling in
match_prev_assignment()and mitigate the performance overhead.Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156History:
v1: https://lore.kernel.org/all/[email protected]/Dapeng Mi (7):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure
perf/x86: Guardintel_pmu_cpu_dead()against invalid hybrid PMU casts perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access perf/x86/intel: Fixintel_caphandling on hybrid PMUs perf/x86: Optimize ACR handling inmatch_prev_assignment()arch/x86/events/core.c | 45 +++++++++++++++++++------- arch/x86/events/intel/core.c | 61 ++++++++++++++++++++++++++---------- arch/x86/events/perf_event.h | 12 ++----- 3 files changed, 81 insertions(+), 37 deletions(-)base-commit:
edda9051e267b7390c7ce24b1b71434414ad156e
Quoted 10 lines · Dapeng Mi
Changes since v1:
- Patch 3/7: Ensurecpuc->pmuis not the static pmu before callinghybrid_pmu(cpuc->pmu)inintel_pmu_cpu_dead()(Sashiko).This series fixes recently found x86 core PMU bugs. Most of bugs are found by Sashiko in reviewing the "Support SIMD/eGPRs/SSP registers sampling for perf" patch-set, e.g.,
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
The last patch 7/7 optimizes ACR handling in
match_prev_assignment()and
mitigate the performance overhead.
Other than the typo in a comment in patch 6/7, LGTM.
Quoted 34 lines
Tests:
Below test cases are run on Diamonds and Novalake. No issues are found.
- Perf stat test$ perf test 119 - Perf record tests $ perf test 155 - Perf record LBR tests $ perf test 156History:
v1:
https://lore.kernel.org/all/[email protected]/Dapeng Mi (7):
perf/x86: Unregister PMI handler on PMU init failure perf/x86: Free hybrid state on PMU init failure perf/x86: Guardintel_pmu_cpu_dead()against invalid hybrid PMU casts
perf/x86/intel: Unwind cpuc state if PEBS buffer setup fails perf/x86: Remove stale fixed counter helper and fix hybrid PMU access
perf/x86/intel: Fixintel_caphandling on hybrid PMUs perf/x86: Optimize ACR handling inmatch_prev_assignment()arch/x86/events/core.c | 45 +++++++++++++++++++-------
arch/x86/events/intel/core.c| 61 ++++++++++++++++++++++++++-------- --
arch/x86/events/perf_event.h | 12 ++-----3 files changed, 81 insertions(+), 37 deletions(-)base-commit:
edda9051e267b7390c7ce24b1b71434414ad156e