perf/x86: Fix multiple LBR sampling issues
Subsystems (3)
v5 cover perf/x86: Fix multiple LBR sampling issues
This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.
The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.
Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceed MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue in perf_snapshot_branch_stack() for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10-11: Add a dedicated LBR stack for BPF LBR snapshotting to avoid overwriting the default LBR stack of other LBR events.
- Patch 12: Ensure perf_snapshot_branch_stack() snapshots the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 13: Prevent PEBS LBR data from overwriting non-PEBS LBR data when both overflow in the same PMI.
- Patch 14: Add the missing lbr_pebs_users accounting for arch-PEBS.
- Patch 15-16: Ensure LBR counters are not dropped when multiple LBR counter events overflow in the same PMI.
- Patch 17: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid LBR stack out-of-bounds access.
- Patch 18: Fix the shared LBR stack depth truncation issue.
Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.
Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).
Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/
History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (17):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel/lbr: Limit branch snapshots to valid entries perf/x86: Add lbr_enabled to track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling perf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed perf/x86/intel: Account arch-PEBS in lbr_pebs_users perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler perf/x86/intel: Pass LBR stack size into store_lbr() perf/core: Don't overwrite shared LBR stack depth
Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries
arch/x86/events/amd/core.c | 14 +++- arch/x86/events/amd/lbr.c | 14 +++- arch/x86/events/intel/core.c | 135 ++++++++++++++++++++++++++----- arch/x86/events/intel/ds.c | 8 +- arch/x86/events/intel/lbr.c | 153 ++++++++++++++++++++++------------- arch/x86/events/perf_event.h | 37 +++++++-- include/linux/perf_event.h | 13 ++- kernel/events/core.c | 8 +- 8 files changed, 285 insertions(+), 97 deletions(-)
base-commit: f5a78ecccd98a04c47cea1cd63c2a2fdce8a4b48
Signature · 1 line
2.34.1
PATCH v5 1/18 perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Add a sanity check for x86_pmu.lbr_nr. If the hardware reports more LBR entries than MAX_LBR_ENTRIES, disable LBR to avoid out-of-bounds access in the cached LBR stack handling path.
--- arch/x86/events/amd/lbr.c | 8 +++++++- arch/x86/events/intel/core.c | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/events/amd/lbr.c +7 -1
@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }
arch/x86/events/intel/core.c +7 -0
@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ; + +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +for (i =0 ; i < x86_pmu.lbr_nr; i++) {if (!(check_msr (x86_pmu.lbr_from + i,0xffffUL ) &&check_msr (x86_pmu.lbr_to + i,0xffffUL )))
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 01/18] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf/x86/intel: incomplete Arch LBR disablement in intel_pmu_init() - [Medium] perf/x86/amd…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf/x86/intel: incomplete Arch LBR disablement in intel_pmu_init()
- [Medium] perf/x86/amd: amd_pmu_lbr_init() returns 0 when LBR is disabled --
commit 03c28823478ca7f089145cd93c93be2824eab310
Author: Dapeng Mi <[email protected]>
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
This commit adds a sanity check on x86_pmu.lbr_nr during AMD and Intel PMU initialization to disable LBR if the hardware reports more entries than MAX_LBR_ENTRIES.
Quoted 9 lines
arch/x86/events/amd/lbr.c +0
-1@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr);
arch/x86/events/amd/lbr.c __init int amd_pmu_lbr_init(void) · line 445
Quoted 10 lines
arch/x86/events/amd/lbr.c +7
-0+if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }
[Severity: Medium]
Should amd_pmu_lbr_init() return an error code such as -EOPNOTSUPP when x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES?
Because amd_pmu_lbr_init() still returns 0 after setting x86_pmu.lbr_nr = 0, its caller in arch/x86/events/amd/core.c treats LBR initialization as successful and installs the LBR callbacks:
arch/x86/events/amd/core.c:amd_core_pmu_init() {
...
/* LBR and BRS are mutually exclusive features */
if (!amd_pmu_lbr_init()) {
/* LBR requires flushing on context switch */
x86_pmu.sched_task = amd_pmu_lbr_sched_task;
static_call_update(amd_pmu_branch_hw_config, amd_pmu_lbr_hw_config);
static_call_update(amd_pmu_branch_reset, amd_pmu_lbr_reset);
static_call_update(amd_pmu_branch_add, amd_pmu_lbr_add);
static_call_update(amd_pmu_branch_del, amd_pmu_lbr_del);
/* Only support branch_stack snapshot on perfmon v2 */
if (x86_pmu.handle_irq == amd_pmu_v2_handle_irq)
static_call_update(perf_snapshot_branch_stack, amd_pmu_v2_snapshot_branch_stack);
}
...
}If perf_snapshot_branch_stack is later invoked (for example via bpf_get_branch_snapshot()), amd_pmu_v2_snapshot_branch_stack() runs without checking x86_pmu.lbr_nr and touches MSR_AMD64_PERF_CNTR_GLOBAL_CTL, MSR_AMD_DBG_EXTN_CFG, and MSR_IA32_DEBUGCTLMSR:
arch/x86/events/amd/core.c:amd_pmu_v2_snapshot_branch_stack() {
...
local_irq_save(flags);
amd_pmu_core_disable_all();
__amd_pmu_lbr_disable();
cpuc = this_cpu_ptr(&cpu_hw_events);
amd_pmu_lbr_read();
cnt = min(cnt, x86_pmu.lbr_nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
amd_pmu_v2_enable_all(0);
local_irq_restore(flags);
...
}Quoted 5 lines
arch/x86/events/intel/core.c +0
-0
arch/x86/events/intel/core.c __init int intel_pmu_init(void) · line 9031
Quoted 9 lines
arch/x86/events/intel/core.c +6
-0@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ; + +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }
[Severity: Medium]
When X86_FEATURE_ARCH_LBR is enabled and CPUID leaf 28 enumerates a depth greater than MAX_LBR_ENTRIES, does zeroing x86_pmu.lbr_nr here leave Architectural LBR partially initialized and leak x86_pmu.task_ctx_cache and per-CPU cpuc->lbr_xsave buffers?
In intel_pmu_init(), intel_pmu_arch_lbr_init() and intel_pmu_check_event_constraints_all() are called before this check:
arch/x86/events/intel/core.c:intel_pmu_init() {
...
if (boot_cpu_has(X86_FEATURE_ARCH_LBR))
intel_pmu_arch_lbr_init();
intel_pmu_check_event_constraints_all(NULL);
...
}Inside arch/x86/events/intel/lbr.c:intel_pmu_arch_lbr_init(), lbr_nr is computed as fls(eax.split.lbr_depth_mask) * 8. Because it does not bail out to clear_arch_lbr when lbr_nr > MAX_LBR_ENTRIES, it already:
- writes lbr_nr to MSR_ARCH_LBR_DEPTH
- allocates x86_pmu.task_ctx_cache
- enables the x86_lbr_mispred, x86_lbr_cycles, and x86_lbr_type static branches
- sets x86_pmu.lbr_ctl_map (which keeps x86_pmu_has_lbr_callstack() true)
- sets PMU_FL_BR_CNTR and PMU_FL_DYN_CONSTRAINT in x86_pmu.flags
- leaves X86_FEATURE_ARCH_LBR set
As a result, when any hardware perf event is later opened, reserve_lbr_buffers() still allocates cpuc->lbr_xsave across all possible CPUs from task_ctx_cache:
arch/x86/events/intel/lbr.c:reserve_lbr_buffers() {
...
if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR))
return;
for_each_possible_cpu(cpu) {
cpuc = per_cpu_ptr(&cpu_hw_events, cpu);
kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
if (!kmem_cache || cpuc->lbr_xsave)
continue;
cpuc->lbr_xsave = kmem_cache_alloc_node(kmem_cache,
GFP_KERNEL | __GFP_ZERO,
cpu_to_node(cpu));
}
}In addition, lbr_is_visible() in arch/x86/events/intel/core.c still exposes branch_counter_nr and branch_counter_width in sysfs because PMU_FL_BR_CNTR remains set, and attach_perf_ctx_data() can still allocate task->perf_ctx_data for events with PERF_SAMPLE_READ and PERF_SAMPLE_BRANCH_CALL_STACK.
Should intel_pmu_arch_lbr_init() check lbr_nr > MAX_LBR_ENTRIES before writing MSR_ARCH_LBR_DEPTH and jump to clear_arch_lbr so X86_FEATURE_ARCH_LBR is cleared cleanly?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Peter Zijlstra
maintainerRe: [Patch v5 01/18] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
In reply to Dapeng MiQuoted 3 lines · Dapeng Mi
Add a sanity check for x86_pmu.lbr_nr. If the hardware reports more LBR entries than
MAX_LBR_ENTRIES, disable LBR to avoid out-of-bounds access in the cached LBR stack handling path.
Cute, can this actually happen?
Quoted 33 lines
Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/amd/lbr.c | 8 +++++++- arch/x86/events/intel/core.c | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/x86/events/amd/lbr.c +7
-1@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }arch/x86/events/intel/core.c +0
-0@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ;
arch/x86/events/intel/core.c __init int intel_pmu_init(void) · line 9035
Quoted 10 lines
arch/x86/events/intel/core.c +7
-0+ +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +for (i =0 ; i < x86_pmu.lbr_nr; i++) {if (!(check_msr (x86_pmu.lbr_from + i,0xffffUL ) &&check_msr (x86_pmu.lbr_to + i,0xffffUL )))
Would it make sense to have but one copy of this check in init_hw_perf_events(), after calling into the micro-arch code?
Re: [Patch v5 01/18] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Dapeng Mi
Add a sanity check for x86_pmu.lbr_nr. If the hardware reports more LBR entries than
MAX_LBR_ENTRIES, disable LBR to avoid out-of-bounds access in the cached LBR stack handling path.Cute, can this actually happen?
Currently it won't happen on real hardware since the maximum LBR stack depth what hardware supports is 32. But it's better have this sanity check assume the LBR stack depth could be extended to 64 or larger.
Quoted 38 lines
Quoted 38 lines
Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/amd/lbr.c | 8 +++++++- arch/x86/events/intel/core.c | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/x86/events/amd/lbr.c +7
-1@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }arch/x86/events/intel/core.c +4
-0@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ; + +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES );
Quoted 8 lines
arch/x86/events/intel/core.c __init int intel_pmu_init(void) · line 9035
Quoted 6 lines
arch/x86/events/intel/core.c +3
-0+ x86_pmu.lbr_nr =0 ; + } +for (i =0 ; i < x86_pmu.lbr_nr; i++) {if (!(check_msr (x86_pmu.lbr_from + i,0xffffUL ) &&check_msr (x86_pmu.lbr_to + i,0xffffUL )))Would it make sense to have but one copy of this check in
init_hw_perf_events(), after calling into the micro-arch code?
Yes, that's what the previous version does, but Sashiko complains it's too late.
"
Does zeroing x86_pmu.lbr_nr here in init_hw_perf_events() happen too late, after intel_pmu_init() and amd_pmu_init() have already configured hardware and driver state using the unvalidated x86_pmu.lbr_nr value?
"
https://lore.kernel.org/all/[email protected]/
It makes sense in some ways, so move the sanity check into the vendor init in this version.
Thanks.
Peter Zijlstra
maintainerRe: [Patch v5 01/18] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Quoted 69 lines · Dapeng
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Dapeng Mi
Add a sanity check for x86_pmu.lbr_nr. If the hardware reports more LBR entries than
MAX_LBR_ENTRIES, disable LBR to avoid out-of-bounds access in the cached LBR stack handling path.Cute, can this actually happen?
Currently it won't happen on real hardware since the maximum LBR stack depth what hardware supports is 32. But it's better have this sanity check assume the LBR stack depth could be extended to 64 or larger.
Quoted 38 lines
Quoted 38 lines
Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/amd/lbr.c | 8 +++++++- arch/x86/events/intel/core.c | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/x86/events/amd/lbr.c +7
-1@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }arch/x86/events/intel/core.c +4
-0@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ; + +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES );Quoted 8 lines
arch/x86/events/intel/core.c __init int intel_pmu_init(void) · line 9035
Quoted 6 lines
arch/x86/events/intel/core.c +3
-0+ x86_pmu.lbr_nr =0 ; + } +for (i =0 ; i < x86_pmu.lbr_nr; i++) {if (!(check_msr (x86_pmu.lbr_from + i,0xffffUL ) &&check_msr (x86_pmu.lbr_to + i,0xffffUL )))Would it make sense to have but one copy of this check in
init_hw_perf_events(), after calling into the micro-arch code?Yes, that's what the previous version does, but Sashiko complains it's too late.
"
Does zeroing x86_pmu.lbr_nr here in
init_hw_perf_events()happen too late, afterintel_pmu_init()andamd_pmu_init()have already configured hardware and driver state using the unvalidated x86_pmu.lbr_nr value?
Quoted 6 lines
"
https://lore.kernel.org/all/[email protected]/
It makes sense in some ways, so move the sanity check into the vendor init in this version.
Urgh. But there are no events yet that early, none of that can actually happen.
If we're really worried about this, change the micro-arch init to spawn in disabled state and we call x86_pmu_enable_all() at the end.
Also, this is before x86_pmu_static_call_update(), nothing really 'works' yet anyway.
Re: [Patch v5 01/18] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Quoted 72 lines · Peter Zijlstra
Quoted 65 lines · Dapeng
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Dapeng Mi
Add a sanity check for x86_pmu.lbr_nr. If the hardware reports more LBR entries than
MAX_LBR_ENTRIES, disable LBR to avoid out-of-bounds access in the cached LBR stack handling path.Cute, can this actually happen?
Currently it won't happen on real hardware since the maximum LBR stack depth what hardware supports is 32. But it's better have this sanity check assume the LBR stack depth could be extended to 64 or larger.
Quoted 37 lines
Quoted 37 lines
Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/amd/lbr.c | 8 +++++++- arch/x86/events/intel/core.c | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/x86/events/amd/lbr.c +7
-1@@ -433,7 +433,13 @@ __init int amd_pmu_lbr_init(void) ebx.full =cpuid_ebx (EXT_PERFMON_DEBUG_FEATURES ); x86_pmu.lbr_nr = ebx.split.lbr_v2_stack_sz; -pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + }else { +pr_cont ("%d-deep LBR, " , x86_pmu.lbr_nr); + }return 0 ; }arch/x86/events/intel/core.c +4
-0@@ -9023,6 +9023,13 @@ __init int intel_pmu_init(void)*/ if (x86_pmu.lbr_tos && !check_msr (x86_pmu.lbr_tos,0x3UL )) x86_pmu.lbr_nr =0 ; + +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES );Quoted 8 lines
arch/x86/events/intel/core.c __init int intel_pmu_init(void) · line 9035
Quoted 6 lines
arch/x86/events/intel/core.c +3
-0+ x86_pmu.lbr_nr =0 ; + } +for (i =0 ; i < x86_pmu.lbr_nr; i++) {if (!(check_msr (x86_pmu.lbr_from + i,0xffffUL ) &&check_msr (x86_pmu.lbr_to + i,0xffffUL )))Would it make sense to have but one copy of this check in
init_hw_perf_events(), after calling into the micro-arch code?Yes, that's what the previous version does, but Sashiko complains it's too late.
"
Does zeroing x86_pmu.lbr_nr here in
init_hw_perf_events()happen too late, afterintel_pmu_init()andamd_pmu_init()have already configured hardware and driver state using the unvalidated x86_pmu.lbr_nr value?Quoted 6 lines
"
https://lore.kernel.org/all/[email protected]/
It makes sense in some ways, so move the sanity check into the vendor init in this version.
Quoted 8 lines
Urgh. But there are no events yet that early, none of that can actually happen.
If we're really worried about this, change the micro-arch init to spawn in disabled state and we call
x86_pmu_enable_all()at the end.Also, this is before
x86_pmu_static_call_update(), nothing really 'works' yet anyway.
Yeah, I agree. I don't think it would cause any real functionality issue by deferring the LBR disabling into init_hw_perf_events(). The only flaw is that the vendor pmu init function has already printed the LBR stack depth which indicates LBR works, and then init_hw_perf_events() prints LBR is disabled. It may cause some confusion for uses. But anyway, I don't think it's a big deal.
PATCH v5 2/18 perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
In reply to Dapeng Miintel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.
Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.
--- arch/x86/events/intel/lbr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/events/intel/lbr.c +9 -1
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + cpuc->lbr_stack.nr =0 ; +if (!x86_pmu.lbr_nr)return ;
@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)intel_pmu_store_lbr (cpuc,NULL );return ; } + xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); + }else { +/* Clear the LBR stack to avoid leaking stale entries. */ + cpuc->lbr_stack.nr =0 ; + } }void intel_pmu_lbr_read (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 02/18] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
intel_pmu_arch_lbr_read_xsave()doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.
Quoted 8 lines
Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when
intel_pmu_lbr_reset()is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Peter Zijlstra
maintainerRe: [Patch v5 02/18] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
In reply to Dapeng MiQuoted 21 lines · Dapeng Mi
intel_pmu_arch_lbr_read_xsave()doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when
intel_pmu_lbr_reset()is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/lbr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/events/intel/lbr.c +0
-0
arch/x86/events/intel/lbr.c void intel_pmu_lbr_reset(void) · line 200
Quoted 8 lines
arch/x86/events/intel/lbr.c +2
-0@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + cpuc->lbr_stack.nr =0 ; +if (!x86_pmu.lbr_nr)return ;
This also clears the stack if there is no LBR; was that on purpose? The Changelog is silent on this point.
Quoted 20 lines
arch/x86/events/intel/lbr.c +7
-1@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)intel_pmu_store_lbr (cpuc,NULL );return ; } + xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); + }else { +/* Clear the LBR stack to avoid leaking stale entries. */ + cpuc->lbr_stack.nr =0 ; + } }void intel_pmu_lbr_read (void )Signature · 1 line
2.34.1
Re: [Patch v5 02/18] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Dapeng Mi
intel_pmu_arch_lbr_read_xsave()doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when
intel_pmu_lbr_reset()is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/lbr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/events/intel/lbr.c +0
-0@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
Quoted 7 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_reset(void) · line 200
Quoted 5 lines
arch/x86/events/intel/lbr.c +2
-0+ cpuc->lbr_stack.nr =0 ; +if (!x86_pmu.lbr_nr)return ;This also clears the stack if there is no LBR; was that on purpose? The Changelog is silent on this point.
Yes, it's on purpose. The intent is to clear the cached software LBR stack as well when resetting the hardware LBR stack. This is to avoid leak the LBR stack information in some unexpected cases. In theory, it can be placed after the !x86_pmu.lbr_nr, I hope the software LBR stack can always be cleared regardless of there is LBR user or not.
My fault, I missed to describe this change in the change log. I would add it in next version.
Thanks.
Quoted 21 lines
Quoted 21 lines
arch/x86/events/intel/lbr.c +7
-1@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)intel_pmu_store_lbr (cpuc,NULL );return ; } + xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); + }else { +/* Clear the LBR stack to avoid leaking stale entries. */ + cpuc->lbr_stack.nr =0 ; + } }void intel_pmu_lbr_read (void )Signature · 1 line
2.34.1
PATCH v5 3/18 perf/x86/amd/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiFrom: Sandipan Das <[email protected]>
The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.
--- arch/x86/events/amd/core.c | 2 +- arch/x86/events/amd/lbr.c | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/events/amd/core.c +1 -1
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u cpuc =this_cpu_ptr (&cpu_hw_events);amd_pmu_lbr_read (); - cnt =min (cnt, x86_pmu.lbr_nr); + cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );
arch/x86/events/amd/lbr.c +3 -1
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)struct branch_entry entry;int out =0 , idx, i; -if (!cpuc->lbr_users) +if (!cpuc->lbr_users) { + cpuc->lbr_stack.nr =0 ;return ; + }for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 03/18] perf/x86/amd/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
From: Sandipan Das <[email protected]>
The branch snapshot callback used by
bpf_get_branch_snapshot()copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read()skips reading the registers, so that a stale count is never reused.
Quoted 6 lines
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Reported-by Sashiko <[email protected]>Signed-off-by Sandipan Das <[email protected]>Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v5 4/18 perf/x86/intel/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiThe branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
--- arch/x86/events/intel/core.c | 2 +- arch/x86/events/intel/lbr.c | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +1 -1
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr); + cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);intel_pmu_enable_all (0 );
arch/x86/events/intel/lbr.c +7 -2
@@ -1027,14 +1027,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +/* * Don't read when all LBRs users are using adaptive PEBS. * * This could be smarter and actually check the event, * but this simple approach seems to work for now. */ -if (!cpuc->lbr_users ||vlbr_exclude_host () || - cpuc->lbr_users == cpuc->lbr_pebs_users) +if (cpuc->lbr_users == cpuc->lbr_pebs_users)return ; x86_pmu.lbr_read (cpuc);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 04/18] perf/x86/intel/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
The branch snapshot callback used by
bpf_get_branch_snapshot()always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
Quoted 6 lines
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Co-developed-by Sandipan Das <[email protected]>Signed-off-by Sandipan Das <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v5 5/18 perf/x86: Add lbr_enabled to track LBR state
In reply to Dapeng MiAdd a lbr_enabled flag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR in perf_snapshot_branch_stack() without checking whether LBR has already been disabled.
--- arch/x86/events/amd/lbr.c | 2 ++ arch/x86/events/intel/core.c | 5 +++++ arch/x86/events/intel/lbr.c | 2 ++ arch/x86/events/perf_event.h | 9 +++++++++ 4 files changed, 18 insertions(+)
arch/x86/events/amd/lbr.c +2 -0
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)if (!cpuc->lbr_users || !x86_pmu.lbr_nr)return ; + cpuc->lbr_enabled =1 ; +/* Set hardware branch filter */ if (cpuc->lbr_select) { lbr_select = cpuc->lbr_sel->config &LBR_SELECT_MASK ;
arch/x86/events/intel/core.c +5 -0
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* It needs to be restored when leaving the handler. */ pmu_enabled = cpuc->enabled; + +/* If pmu version < 4, LBR freeze disables LBR. */ +if (x86_pmu.version <4 ) + cpuc->lbr_enabled =0 ; +/* * In general, the early ACK is only applied for old platforms. * For the big core starts from Haswell, the late ACK should be
arch/x86/events/intel/lbr.c +2 -0
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)if (pmi && x86_pmu.version >=4 )return ; + cpuc->lbr_enabled =1 ; +/* * No need to reprogram LBR_SELECT in a PMI, as it * did not change.
arch/x86/events/perf_event.h +9 -0
@@ -473,6 +473,7 @@ struct cpu_hw_events {unsigned long active_mask[BITS_TO_LONGS (X86_PMC_IDX_MAX )];unsigned long dirty[BITS_TO_LONGS (X86_PMC_IDX_MAX )];int enabled; +int lbr_enabled;int n_events;/* the # of events in the below arrays */ int n_added;/* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event); static __always_inline void __amd_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 dbg_ctl, dbg_extn_cfg; + cpuc->lbr_enabled =0 ; +rdmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg);wrmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN );
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void) static __always_inline void __intel_pmu_arch_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->lbr_enabled =0 ;wrmsrq (MSR_ARCH_LBR_CTL ,0 ); } static __always_inline void __intel_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 debugctl; + cpuc->lbr_enabled =0 ;rdmsrq (MSR_IA32_DEBUGCTLMSR , debugctl); debugctl &= ~(DEBUGCTLMSR_LBR |DEBUGCTLMSR_FREEZE_LBRS_ON_PMI );wrmsrq (MSR_IA32_DEBUGCTLMSR , debugctl);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 05/18] perf/x86: Add lbr_enabled to track LBR state
Sashiko has reviewed this patch and found no issues.
Quoted 8 lines
Add a
lbr_enabledflag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR inperf_snapshot_branch_stack()without checking whether LBR has already been disabled.Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v5 6/18 perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Miamd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/amd/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/events/amd/core.c +9 -3
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events); -amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (lbr_enabled) +amd_pmu_lbr_enable_all (); +if (pmu_enabled) +amd_pmu_core_enable_all ();local_irq_restore (flags);return cnt;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 06/18] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set while PMU is disabled --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set while PMU is disabled --
commit cb298524c7ea53df65b77e5a09ad097e570397f6
Author: Dapeng Mi <[email protected]>
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates amd_pmu_v2_snapshot_branch_stack() to save the current PMU and LBR enabled state and only re-enable them after taking the LBR snapshot if they were originally enabled. This avoids unconditionally re-enabling the PMU and LBR when called from a PMI context where they were already disabled.
Quoted 23 lines
arch/x86/events/amd/core.c +5
-1@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 956
Quoted 10 lines
arch/x86/events/amd/core.c +4
-2-amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (lbr_enabled) +amd_pmu_lbr_enable_all (); +if (pmu_enabled) +amd_pmu_core_enable_all ();
[Severity: Medium]
Should amd_pmu_v2_snapshot_branch_stack() in arch/x86/events/amd/core.c also clear cpuc->enabled = 0 while the core PMU is disabled, and restore cpuc->enabled = pmu_enabled before re-enabling it?
While __amd_pmu_lbr_disable() sets cpuc->lbr_enabled = 0 and amd_pmu_lbr_enable_all() sets cpuc->lbr_enabled = 1, amd_pmu_core_disable_all() only clears the hardware MSR and leaves cpuc->enabled set to 1.
If a BPF program running in non-NMI context (such as a kprobe, tracepoint, or fentry/fexit hook) calls bpf_get_branch_snapshot() -> amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save() masks normal interrupts but does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack() is in amd_pmu_lbr_read() or memcpy(), perf_event_nmi_handler() -> amd_pmu_v2_handle_irq() will observe cpuc->enabled == 1:
arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
pmu_enabled = cpuc->enabled;
cpuc->enabled = 0;
/* Stop counting but do not disable LBR */
amd_pmu_core_disable_all();
...
done:
cpuc->enabled = pmu_enabled;
/* Resume counting only if PMU is active */
if (pmu_enabled)
amd_pmu_core_enable_all();
}Similarly, if a BPF program triggered inside an NMI calls amd_pmu_v2_snapshot_branch_stack() re-entrantly during that window, it will also see cpuc->enabled == 1.
In both cases, wouldn't the NMI path call amd_pmu_core_enable_all() on exit and prematurely re-enable hardware PMU counting before the outer amd_pmu_v2_snapshot_branch_stack() has finished?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v5 7/18 perf/x86/intel: Add helper to enable PMU without LBR
In reply to Dapeng MiCurrently, __intel_pmu_enable_all() also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper, ___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.
Also remove the unused added argument from __intel_pmu_enable_all().
No functional change intended.
--- arch/x86/events/intel/core.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +9 -5
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)intel_pmu_lbr_disable_all (); } -static void __intel_pmu_enable_all (int added,bool pmi) +static void ___intel_pmu_enable_all (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 intel_ctrl =hybrid (cpuc->pmu, intel_ctrl); -intel_pmu_lbr_enable_all (pmi); -if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {wrmsrq (MSR_ARCH_PERFMON_FIXED_CTR_CTRL , cpuc->fixed_ctrl_val); cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi) } } +static void __intel_pmu_enable_all (bool pmi) +{ +intel_pmu_lbr_enable_all (pmi); +___intel_pmu_enable_all (); +} +static void intel_pmu_enable_all (int added) {static_call_cond (x86_pmu_pebs_enable_all)(); -__intel_pmu_enable_all (added, false); +__intel_pmu_enable_all (false); }static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled;if (pmu_enabled) -__intel_pmu_enable_all (0 , true); +__intel_pmu_enable_all (true);intel_bts_enable_local ();/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 07/18] perf/x86/intel: Add helper to enable PMU without LBR
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
Currently,
__intel_pmu_enable_all()also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper,___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.Also remove the unused added argument from
__intel_pmu_enable_all().No functional change intended.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
PATCH v5 8/18 perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/intel/core.c | 38 ++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +34 -4
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added) static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries, -unsigned int cnt,unsigned long flags) +unsigned int cnt,unsigned long flags, +int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -intel_pmu_enable_all (0 ); + +/* +* Do not blindly call intel_pmu_enable_all() to re-enable LBR. +* This path may be entered from the PMI handler through +* bpf_overflow_handler(). +* +* If PMU version < 4, LBR is disabled by the LBR freeze mechanism, +* LBR can't be re-enabled here. +* If PMU version >= 4, LBR is only frozen instead of disabled, +* LBR should be re-enabled here. +*/ +if (lbr_enabled) +intel_pmu_lbr_enable_all (false); +if (pmu_enabled) +___intel_pmu_enable_all (); +local_irq_restore (flags);return cnt; }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,static int intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }static int intel_pmu_snapshot_arch_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_arch_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 08/18] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
Sashiko has reviewed this patch and found no issues.
Quoted 5 lines
__intel_pmu_snapshot_branch_stack()currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Quoted 7 lines
Only restore the PMU and LBR on the BPF path if they were originally enabled.
Reported-by Sashiko <[email protected]>Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v5 9/18 perf/x86/intel: Keep LBR frozen throughout PMI handling
In reply to Dapeng MiFor the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
--- arch/x86/events/intel/core.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +11 -2
@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again:intel_pmu_lbr_read (); -intel_pmu_ack_status (status); + +/* +* Do not clear the LBR frozen status here; keep LBR frozen +* throughout PMI handling. +*/ +intel_pmu_ack_status (status & ~GLOBAL_STATUS_LBRS_FROZEN ); +if (++loops >100 ) {static bool warned;
@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* Repeat if there is more work to be done: */ status =intel_pmu_get_status (); -if (status) +if (status & ~GLOBAL_STATUS_LBRS_FROZEN ) goto again; done: +if (status &GLOBAL_STATUS_LBRS_FROZEN ) +intel_pmu_ack_status (status &GLOBAL_STATUS_LBRS_FROZEN ); +if (mid_ack)apic_write (APIC_LVTPC ,APIC_DM_NMI );/* Only restore PMU state when it's active. See x86_pmu_disable(). */
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 09/18] perf/x86/intel: Keep LBR frozen throughout PMI handling
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when
bpf_get_branch_snapshot()is invoked frombpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Quoted 8 lines
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Peter Zijlstra
maintainerRe: [Patch v5 09/18] perf/x86/intel: Keep LBR frozen throughout PMI handling
In reply to Dapeng MiQuoted 47 lines · Dapeng Mi
For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when
bpf_get_branch_snapshot()is invoked frombpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)arch/x86/events/intel/core.c +7
-2@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again:intel_pmu_lbr_read (); -intel_pmu_ack_status (status); + +/* +* Do not clear the LBR frozen status here; keep LBR frozen +* throughout PMI handling. +*/ +intel_pmu_ack_status (status & ~GLOBAL_STATUS_LBRS_FROZEN ); +if (++loops >100 ) {static bool warned;@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* Repeat if there is more work to be done: */ status =intel_pmu_get_status (); -if (status)
arch/x86/events/intel/core.c static int intel_pmu_handle_irq(struct pt_regs *regs) · line 4099
Quoted 10 lines
arch/x86/events/intel/core.c +4
-0+if (status & ~GLOBAL_STATUS_LBRS_FROZEN ) goto again; done: +if (status &GLOBAL_STATUS_LBRS_FROZEN ) +intel_pmu_ack_status (status &GLOBAL_STATUS_LBRS_FROZEN ); +if (mid_ack)apic_write (APIC_LVTPC ,APIC_DM_NMI );/* Only restore PMU state when it's active. See x86_pmu_disable(). */
Given the whole PMU is disabled per __intel_pmu_disable_all(), all that can really happen is for more bits to be set, right? I mean, any counter cleared on the first ack_status, will never be re-set, it already fired and its disabled. All that can happen is that the disable is 'delayed' and a counter can maybe fire a little bit after the disable; getting missed on the first go.
Can we really get more than 1 extra loop there?
I'm sure this code has changed lots since we added that loop thing there.
Anyway, if we can never get more than 1 extra loop, we can probably move the ack to the end, without having to do a second.
I suppose this isn't something you have to do now, but keep it on the todo list or something?
Re: [Patch v5 09/18] perf/x86/intel: Keep LBR frozen throughout PMI handling
Quoted 67 lines · Peter Zijlstra
Quoted 47 lines · Dapeng Mi
For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when
bpf_get_branch_snapshot()is invoked frombpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)arch/x86/events/intel/core.c +7
-2@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again:intel_pmu_lbr_read (); -intel_pmu_ack_status (status); + +/* +* Do not clear the LBR frozen status here; keep LBR frozen +* throughout PMI handling. +*/ +intel_pmu_ack_status (status & ~GLOBAL_STATUS_LBRS_FROZEN ); +if (++loops >100 ) {static bool warned;@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* Repeat if there is more work to be done: */ status =intel_pmu_get_status (); -if (status)arch/x86/events/intel/core.c static int intel_pmu_handle_irq(struct pt_regs *regs) · line 4099
Quoted 10 lines
arch/x86/events/intel/core.c +4
-0+if (status & ~GLOBAL_STATUS_LBRS_FROZEN ) goto again; done: +if (status &GLOBAL_STATUS_LBRS_FROZEN ) +intel_pmu_ack_status (status &GLOBAL_STATUS_LBRS_FROZEN ); +if (mid_ack)apic_write (APIC_LVTPC ,APIC_DM_NMI );/* Only restore PMU state when it's active. See x86_pmu_disable(). */ Given the whole PMU is disabled per
__intel_pmu_disable_all(), all that can really happen is for more bits to be set, right? I mean, any counter cleared on the firstack_status, will never be re-set, it already fired and its disabled. All that can happen is that the disable is 'delayed' and a counter can maybe fire a little bit after the disable; getting missed on the first go.Can we really get more than 1 extra loop there?
Quoted 8 lines
I'm sure this code has changed lots since we added that loop thing there.
Anyway, if we can never get more than 1 extra loop, we can probably move the ack to the end, without having to do a second.
I suppose this isn't something you have to do now, but keep it on the todo list or something?
Sure. I would investigate the history and check if it can be removed.
PATCH v5 10/18 perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
In reply to Dapeng MiThe LBR read helpers currently assume the default stack, cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the *_lbr_read() helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Update the related callers, including intel_pmu_store_lbr() and intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.
No function change intended.
--- arch/x86/events/intel/core.c | 9 ++- arch/x86/events/intel/lbr.c | 113 ++++++++++++++++++++--------------- arch/x86/events/perf_event.h | 16 +++-- 3 files changed, 84 insertions(+), 54 deletions(-)
arch/x86/events/intel/core.c +6 -3
@@ -2826,10 +2826,12 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -intel_pmu_lbr_read (); +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); +memcpy (entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cnt);/* * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
@@ -4061,7 +4063,8 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (); +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters);/* * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +66 -47
251 lines not on this page · raw message
arch/x86/events/perf_event.h +12 -4
@@ -1192,7 +1192,9 @@ struct x86_pmu { unsigned int lbr_counters:4;void (*lbr_reset)(void ); -void (*lbr_read)(struct cpu_hw_events *cpuc); +void (*lbr_read)(struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries);void (*lbr_save)(void *ctx);void (*lbr_restore)(void *ctx);
@@ -1992,11 +1994,17 @@ void intel_pmu_lbr_enable_all(bool pmi);void intel_pmu_lbr_disable_all (void ); -void intel_pmu_lbr_read (void ); +void intel_pmu_lbr_read (struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries, +u64 *lbr_counters); -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries); -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries);void intel_pmu_lbr_save (void *ctx);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 10/18] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Sashiko has reviewed this patch and found no issues.
Quoted 6 lines
The LBR read helpers currently assume the default stack,
cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the*_lbr_read()helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Quoted 7 lines
Update the related callers, including
intel_pmu_store_lbr()andintel_pmu_lbr_filter(), to pass the appropriate stack explicitly.No function change intended.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Peter Zijlstra
maintainerRe: [Patch v5 10/18] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
In reply to Dapeng MiQuoted 37 lines · Dapeng Mi
arch/x86/events/intel/lbr.c +9
-4@@ -105,7 +105,10 @@#define ARCH_LBR_CTL_MASK 0x7f000e -static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc); +static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries, +u64 *lbr_counters); static __always_inline bool is_lbr_call_stack_bit_set(u64 config) {@@ -741,10 +744,12 @@ void intel_pmu_lbr_disable_all(void) } } -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;@@ -775,11 +780,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)* is the same as the linear address, allowing us to merge the LIP and EIP * LBR formats. */ -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc) · line 792
Quoted 10 lines
arch/x86/events/intel/lbr.c +3
-1+struct perf_branch_stack *lbr_stack, + struct perf_branch_entry *lbr_entries) { bool need_info = false, call_stack = false;unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;int out =0 ;
This makes no sense; why do you keep passing lbr_entries around when we have lbr_stack.entries[] ?
Re: [Patch v5 10/18] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Quoted 42 lines · Peter Zijlstra
Quoted 41 lines · Dapeng Mi
arch/x86/events/intel/lbr.c +11
-4@@ -105,7 +105,10 @@#define ARCH_LBR_CTL_MASK 0x7f000e -static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc); +static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries, +u64 *lbr_counters); static __always_inline bool is_lbr_call_stack_bit_set(u64 config) {@@ -741,10 +744,12 @@ void intel_pmu_lbr_disable_all(void) } } -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;@@ -775,11 +780,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)* is the same as the linear address, allowing us to merge the LIP and EIP * LBR formats. */ -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {bool need_info = false, call_stack = false;
Quoted 8 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc) · line 792
Quoted 6 lines
arch/x86/events/intel/lbr.c +1
-1unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;int out =0 ;This makes no sense; why do you keep passing
lbr_entriesaround when we have lbr_stack.entries[] ?
Yeah, that's actually what I did at the beginning. But sashiko raised the warning below. :(
"
Does this trigger a CONFIG_UBSAN_BOUNDS panic when intel_pmu_store_lbr() is inlined?
The lbr_stack pointer points to a structperf_branch_stack embedded directly in struct cpu_hw_events. Because it is an embedded value-type struct, the compiler considers its size to be fixed (16 bytes), treating the trailing flexible array member entries[] as having 0 elements.
Accessing lbr_stack->entries[i] where i >= 0 can trigger strict array bounds violations under -fstrict-flex-arrays=3 and CONFIG_UBSAN_BOUNDS.
"
Peter Zijlstra
maintainerRe: [Patch v5 10/18] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Quoted 68 lines · Dapeng
Quoted 42 lines · Peter Zijlstra
Quoted 41 lines · Dapeng Mi
arch/x86/events/intel/lbr.c +11
-4@@ -105,7 +105,10 @@#define ARCH_LBR_CTL_MASK 0x7f000e -static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc); +static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries, +u64 *lbr_counters); static __always_inline bool is_lbr_call_stack_bit_set(u64 config) {@@ -741,10 +744,12 @@ void intel_pmu_lbr_disable_all(void) } } -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;@@ -775,11 +780,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)* is the same as the linear address, allowing us to merge the LIP and EIP * LBR formats. */ -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {bool need_info = false, call_stack = false;Quoted 8 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc) · line 792
Quoted 6 lines
arch/x86/events/intel/lbr.c +1
-1unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;int out =0 ;This makes no sense; why do you keep passing
lbr_entriesaround when we have lbr_stack.entries[] ?Yeah, that's actually what I did at the beginning. But sashiko raised the warning below. :(
"
Does this trigger a
CONFIG_UBSAN_BOUNDSpanic whenintel_pmu_store_lbr()is inlined?
Thelbr_stackpointer points to a structperf_branch_stackembedded directly in structcpu_hw_events. Because it is an embedded value-type struct, the compiler considers its size to be fixed (16 bytes), treating the trailing flexible array member entries[] as having 0 elements.
Accessing lbr_stack->entries[i] where i >= 0 can trigger strict array bounds violations under -fstrict-flex-arrays=3 andCONFIG_UBSAN_BOUNDS.
Quoted 1 line
"
So I'm generally in favour bounds checking; but:
- it really should get removed from UBSAN;
- this flex array nonsense is absolutely insane.
There is this TRAILING_OVERLAP thing, which is complete and utter batshit insane crazy and I will refuse to use it.
So one, we already violate this, and two, if we want to fix it something like so will probably do.
---
arch/x86/events/amd/brs.c +2 -3
@@ -282,7 +282,6 @@ void amd_brs_drain(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_event *event = cpuc->events[0 ]; -struct perf_branch_entry *br = cpuc->lbr_entries;union amd_debug_extn_cfg cfg;u32 i, nr =0 , num, tos, start;u32 shift =64 - boot_cpu_data.x86_virt_bits;
@@ -343,7 +342,7 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; - br[nr] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[nr] = (struct perf_branch_entry ){ .from = from, .to = to, };
@@ -351,7 +350,7 @@ void amd_brs_drain(void) } empty:/* Record number of sampled branches */ - cpuc->lbr_stack.nr = nr; +cpu_lbr_stack (cpuc)->nr = nr; }/*
arch/x86/events/amd/core.c +2 -2
@@ -945,7 +945,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, uamd_pmu_lbr_read (); cnt =min (cnt, x86_pmu.lbr_nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); +memcpy (entries,cpu_lbr_stack (cpuc)->entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );local_irq_restore (flags);
@@ -1014,7 +1014,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)if (!x86_perf_event_set_period (event))continue ; -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/amd/lbr.c +16 -17
@@ -110,9 +110,9 @@ static void amd_pmu_lbr_filter(void) ((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE)) fused_only = true; -for (i =0 ; i < cpuc->lbr_stack.nr; i++) { - from = cpuc->lbr_entries[i].from; - to = cpuc->lbr_entries[i].to; +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; i++) { + from =cpu_lbr_stack (cpuc)->entries[i].from; + to =cpu_lbr_stack (cpuc)->entries[i].to; type =branch_type_fused (from, to,0 , &offset);/*
@@ -121,33 +121,33 @@ static void amd_pmu_lbr_filter(void)* actual branch */ if (offset) { - cpuc->lbr_entries[i].from += offset; +cpu_lbr_stack (cpuc)->entries[i].from += offset;if (fused_only)continue ; }/* If type does not correspond, then discard */ if (type ==X86_BR_NONE || (br_sel & type) != type || - (!(br_sel &X86_BR_KERNEL ) &&kernel_ip (cpuc->lbr_entries[i].from))) { - cpuc->lbr_entries[i].from =0 ;/* mark invalid */ + (!(br_sel &X86_BR_KERNEL ) &&kernel_ip (cpu_lbr_stack (cpuc)->entries[i].from))) { +cpu_lbr_stack (cpuc)->entries[i].from =0 ;/* mark invalid */ compress = true; }if ((br_sel &X86_BR_TYPE_SAVE ) ==X86_BR_TYPE_SAVE ) - cpuc->lbr_entries[i].type =common_branch_type (type); +cpu_lbr_stack (cpuc)->entries[i].type =common_branch_type (type); }if (!compress)return ;/* Remove all invalid entries */ -for (i =0 ; i < cpuc->lbr_stack.nr; ) { -if (!cpuc->lbr_entries[i].from) { +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; ) { +if (!cpu_lbr_stack (cpuc)->entries[i].from) { j = i; -while (++j < cpuc->lbr_stack.nr) - cpuc->lbr_entries[j -1 ] = cpuc->lbr_entries[j]; - cpuc->lbr_stack.nr--; -if (!cpuc->lbr_entries[i].from) +while (++j <cpu_lbr_stack (cpuc)->nr) +cpu_lbr_stack (cpuc)->entries[j -1 ] =cpu_lbr_stack (cpuc)->entries[j]; +cpu_lbr_stack (cpuc)->nr--; +if (!cpu_lbr_stack (cpuc)->entries[i].from)continue ; } i++;
@@ -164,7 +164,6 @@ static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {void amd_pmu_lbr_read (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -struct perf_branch_entry *br = cpuc->lbr_entries;struct branch_entry entry;int out =0 , idx, i;
@@ -202,7 +201,7 @@ void amd_pmu_lbr_read(void)*/ idx = (entry.to.split.valid <<1 ) | entry.to.split.spec; - br[out] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[out] = (struct perf_branch_entry ){ .from =sign_ext_branch_ip (entry.from.split.ip), .to =sign_ext_branch_ip (entry.to.split.ip), .mispred = entry.from.split.mispredict,
@@ -212,13 +211,13 @@ void amd_pmu_lbr_read(void) out++; } - cpuc->lbr_stack.nr = out; +cpu_lbr_stack (cpuc)->nr = out;/* * Internal register renaming always ensures that LBR From[0] and * LBR To[0] always represent the TOS */ - cpuc->lbr_stack.hw_idx =0 ; +cpu_lbr_stack (cpuc)->hw_idx =0 ;/* Perform further software filtering */ amd_pmu_lbr_filter ();
arch/x86/events/core.c +1 -1
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)perf_sample_data_init (&data,0 , last_period); -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/intel/ds.c +4 -4
@@ -2010,8 +2010,8 @@ void intel_pmu_pebs_disable_all(void)static int intel_pmu_pebs_fixup_ip (struct pt_regs *regs) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -unsigned long from = cpuc->lbr_entries[0 ].from; -unsigned long old_to, to = cpuc->lbr_entries[0 ].to; +unsigned long from =cpu_lbr_stack (cpuc)->entries[0 ].from; +unsigned long old_to, to =cpu_lbr_stack (cpuc)->entries[0 ].to;unsigned long ip = regs->ip;int is_64bit =0 ;void *kaddr;
@@ -2026,7 +2026,7 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)/* * No LBR entry, no basic block, no rewinding */ -if (!cpuc->lbr_stack.nr || !from || !to) +if (!cpu_lbr_stack (cpuc)->nr || !from || !to)return 0 ;/*
@@ -2318,7 +2318,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,if (x86_pmu.intel_cap.pebs_format >=3 )setup_pebs_time (event, data, pebs->tsc); -perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); } static void adaptive_pebs_save_regs(struct pt_regs *regs,
arch/x86/events/intel/lbr.c +31 -35
194 lines not on this page · raw message
arch/x86/events/perf_event.h +8 -2
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {struct x86_perf_task_context ;#define MAX_LBR_ENTRIES 32 +#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES) +enum {LBR_FORMAT_32 =0x00 ,LBR_FORMAT_LIP =0x01 ,
@@ -524,8 +526,7 @@ struct cpu_hw_events {*/ int lbr_users;int lbr_pebs_users; -struct perf_branch_stack lbr_stack; -struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ]; +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 lbr_counters[MAX_LBR_ENTRIES ];/* branch stack extra */ union {struct er_account *lbr_sel;
@@ -587,6 +588,11 @@ struct cpu_hw_events {struct pmu *pmu; }; +static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__lbr_stack; +} +#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \ { .idxmsk64 = (n) }, \ .code = (c), \
include/linux/perf_event.h +1 -1
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)struct perf_branch_stack {u64 nr;u64 hw_idx; -struct perf_branch_entry entries[]; + struct perf_branch_entry entries[] __counted_by(nr); };struct task_struct ;
Re: [Patch v5 10/18] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Quoted 526 lines · Peter Zijlstra
Quoted 66 lines · Dapeng
Quoted 42 lines · Peter Zijlstra
Quoted 41 lines · Dapeng Mi
arch/x86/events/intel/lbr.c +11
-4@@ -105,7 +105,10 @@#define ARCH_LBR_CTL_MASK 0x7f000e -static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc); +static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries, +u64 *lbr_counters); static __always_inline bool is_lbr_call_stack_bit_set(u64 config) {@@ -741,10 +744,12 @@ void intel_pmu_lbr_disable_all(void) } } -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;@@ -775,11 +780,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)* is the same as the linear address, allowing us to merge the LIP and EIP * LBR formats. */ -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc) +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) {bool need_info = false, call_stack = false;Quoted 8 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) · line 778
Quoted 6 lines
arch/x86/events/intel/lbr.c +1
-1unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries; +struct perf_branch_entry *br = lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;int out =0 ;This makes no sense; why do you keep passing
lbr_entriesaround when we have lbr_stack.entries[] ?Yeah, that's actually what I did at the beginning. But sashiko raised the warning below. :(
"
Does this trigger a
CONFIG_UBSAN_BOUNDSpanic whenintel_pmu_store_lbr()is inlined?
Thelbr_stackpointer points to a structperf_branch_stackembedded directly in structcpu_hw_events. Because it is an embedded value-type struct, the compiler considers its size to be fixed (16 bytes), treating the trailing flexible array member entries[] as having 0 elements.
Accessing lbr_stack->entries[i] where i >= 0 can trigger strict array bounds violations under -fstrict-flex-arrays=3 andCONFIG_UBSAN_BOUNDS.Quoted 1 line
"
So I'm generally in favour bounds checking; but:
- it really should get removed from UBSAN;
- this flex array nonsense is absolutely insane.There is this
TRAILING_OVERLAPthing, which is complete and utter batshit insane crazy and I will refuse to use it.So one, we already violate this, and two, if we want to fix it something like so will probably do.
---
arch/x86/events/amd/brs.c +2
-3@@ -282,7 +282,6 @@ void amd_brs_drain(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_event *event = cpuc->events[0 ]; -struct perf_branch_entry *br = cpuc->lbr_entries;union amd_debug_extn_cfg cfg;u32 i, nr =0 , num, tos, start;u32 shift =64 - boot_cpu_data.x86_virt_bits;@@ -343,7 +342,7 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; - br[nr] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[nr] = (struct perf_branch_entry ){ .from = from, .to = to, };@@ -351,7 +350,7 @@ void amd_brs_drain(void) } empty:/* Record number of sampled branches */ - cpuc->lbr_stack.nr = nr; +cpu_lbr_stack (cpuc)->nr = nr; }/* arch/x86/events/amd/core.c +2
-2@@ -945,7 +945,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, uamd_pmu_lbr_read (); cnt =min (cnt, x86_pmu.lbr_nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); +memcpy (entries,cpu_lbr_stack (cpuc)->entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );local_irq_restore (flags);@@ -1014,7 +1014,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)if (!x86_perf_event_set_period (event))continue ; -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }arch/x86/events/amd/lbr.c +16
-17@@ -110,9 +110,9 @@ static void amd_pmu_lbr_filter(void) ((br_sel & X86_BR_TYPE_SAVE) != X86_BR_TYPE_SAVE)) fused_only = true; -for (i =0 ; i < cpuc->lbr_stack.nr; i++) { - from = cpuc->lbr_entries[i].from; - to = cpuc->lbr_entries[i].to; +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; i++) { + from =cpu_lbr_stack (cpuc)->entries[i].from; + to =cpu_lbr_stack (cpuc)->entries[i].to; type =branch_type_fused (from, to,0 , &offset);/* @@ -121,33 +121,33 @@ static void amd_pmu_lbr_filter(void)* actual branch */ if (offset) { - cpuc->lbr_entries[i].from += offset; +cpu_lbr_stack (cpuc)->entries[i].from += offset;if (fused_only)continue ; }/* If type does not correspond, then discard */ if (type ==X86_BR_NONE || (br_sel & type) != type || - (!(br_sel &X86_BR_KERNEL ) &&kernel_ip (cpuc->lbr_entries[i].from))) { - cpuc->lbr_entries[i].from =0 ;/* mark invalid */ + (!(br_sel &X86_BR_KERNEL ) &&kernel_ip (cpu_lbr_stack (cpuc)->entries[i].from))) { +cpu_lbr_stack (cpuc)->entries[i].from =0 ;/* mark invalid */ compress = true; }if ((br_sel &X86_BR_TYPE_SAVE ) ==X86_BR_TYPE_SAVE ) - cpuc->lbr_entries[i].type =common_branch_type (type); +cpu_lbr_stack (cpuc)->entries[i].type =common_branch_type (type); }if (!compress)return ;/* Remove all invalid entries */ -for (i =0 ; i < cpuc->lbr_stack.nr; ) { -if (!cpuc->lbr_entries[i].from) { +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; ) { +if (!cpu_lbr_stack (cpuc)->entries[i].from) { j = i; -while (++j < cpuc->lbr_stack.nr) - cpuc->lbr_entries[j -1 ] = cpuc->lbr_entries[j]; - cpuc->lbr_stack.nr--; -if (!cpuc->lbr_entries[i].from) +while (++j <cpu_lbr_stack (cpuc)->nr) +cpu_lbr_stack (cpuc)->entries[j -1 ] =cpu_lbr_stack (cpuc)->entries[j]; +cpu_lbr_stack (cpuc)->nr--; +if (!cpu_lbr_stack (cpuc)->entries[i].from)continue ; } i++;@@ -164,7 +164,6 @@ static const int lbr_spec_map[PERF_BR_SPEC_MAX] = {void amd_pmu_lbr_read (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -struct perf_branch_entry *br = cpuc->lbr_entries;struct branch_entry entry;int out =0 , idx, i;@@ -202,7 +201,7 @@ void amd_pmu_lbr_read(void)*/ idx = (entry.to.split.valid <<1 ) | entry.to.split.spec; - br[out] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[out] = (struct perf_branch_entry ){ .from =sign_ext_branch_ip (entry.from.split.ip), .to =sign_ext_branch_ip (entry.to.split.ip), .mispred = entry.from.split.mispredict,@@ -212,13 +211,13 @@ void amd_pmu_lbr_read(void) out++; } - cpuc->lbr_stack.nr = out; +cpu_lbr_stack (cpuc)->nr = out;/* * Internal register renaming always ensures that LBR From[0] and * LBR To[0] always represent the TOS */ - cpuc->lbr_stack.hw_idx =0 ; +cpu_lbr_stack (cpuc)->hw_idx =0 ;/* Perform further software filtering */ amd_pmu_lbr_filter ();arch/x86/events/core.c +1
-1@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)perf_sample_data_init (&data,0 , last_period); -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }arch/x86/events/intel/ds.c +4
-4@@ -2010,8 +2010,8 @@ void intel_pmu_pebs_disable_all(void)static int intel_pmu_pebs_fixup_ip (struct pt_regs *regs) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -unsigned long from = cpuc->lbr_entries[0 ].from; -unsigned long old_to, to = cpuc->lbr_entries[0 ].to; +unsigned long from =cpu_lbr_stack (cpuc)->entries[0 ].from; +unsigned long old_to, to =cpu_lbr_stack (cpuc)->entries[0 ].to;unsigned long ip = regs->ip;int is_64bit =0 ;void *kaddr;@@ -2026,7 +2026,7 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)/* * No LBR entry, no basic block, no rewinding */ -if (!cpuc->lbr_stack.nr || !from || !to) +if (!cpu_lbr_stack (cpuc)->nr || !from || !to)return 0 ;/* @@ -2318,7 +2318,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,if (x86_pmu.intel_cap.pebs_format >=3 )setup_pebs_time (event, data, pebs->tsc); -perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); } static void adaptive_pebs_save_regs(struct pt_regs *regs,arch/x86/events/intel/lbr.c +31
-35@@ -740,7 +740,6 @@ void intel_pmu_lbr_disable_all(void)void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc) {unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;@@ -756,14 +755,13 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)rdmsrq (x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr); - *br = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[i] = (struct perf_branch_entry ){ .from = msr_lastbranch.from, .to = msr_lastbranch.to, }; - br++; } - cpuc->lbr_stack.nr = i; - cpuc->lbr_stack.hw_idx = tos; +cpu_lbr_stack (cpuc)->nr = i; +cpu_lbr_stack (cpuc)->hw_idx = tos; }/* @@ -775,7 +773,6 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) {bool need_info = false, call_stack = false;unsigned long mask = x86_pmu.lbr_nr -1 ; -struct perf_branch_entry *br = cpuc->lbr_entries;u64 tos =intel_pmu_lbr_tos ();int i;int out =0 ;@@ -847,7 +844,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)if (abort && x86_pmu.lbr_double_abort && out >0 ) out--; - br[out] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[out] = (struct perf_branch_entry ){ .from = from, .to = to, .mispred = mis,@@ -858,8 +855,8 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) }; out++; } - cpuc->lbr_stack.nr = out; - cpuc->lbr_stack.hw_idx = tos; +cpu_lbr_stack (cpuc)->nr = out; +cpu_lbr_stack (cpuc)->hw_idx = tos; }static DEFINE_STATIC_KEY_FALSE (x86_lbr_mispred);@@ -903,7 +900,6 @@ static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBstatic void intel_pmu_store_lbr (struct cpu_hw_events *cpuc,struct lbr_entry *entries) { -struct perf_branch_entry *e;struct lbr_entry *lbr;u64 from, to, info;bool mispred;@@ -911,7 +907,6 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,for (i =0 ; i < x86_pmu.lbr_nr; i++) { lbr = entries ? &entries[i] :NULL ; - e = &cpuc->lbr_entries[i]; from =rdlbr_from (i, lbr);/* @@ -925,7 +920,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc, mispred =get_lbr_mispred (info); - *e = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[i] = (struct perf_branch_entry ){ .from = from, .to = to, .mispred = mispred,@@ -946,7 +941,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc, }; } - cpuc->lbr_stack.nr = i; +cpu_lbr_stack (cpuc)->nr = i; }/* @@ -972,15 +967,15 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,WARN_ON_ONCE (!pos); -for (i =0 ; i < cpuc->lbr_stack.nr; i++) { - src = cpuc->lbr_entries[i].reserved; +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; i++) { + src =cpu_lbr_stack (cpuc)->entries[i].reserved; dst =0 ;for (j =0 ; j < pos; j++) { cnt = (src >> (order[j] *LBR_INFO_BR_CNTR_BITS )) &LBR_INFO_BR_CNTR_MASK ; dst |= cnt << j *LBR_INFO_BR_CNTR_BITS ; } cpuc->lbr_counters[i] = dst; - cpuc->lbr_entries[i].reserved =0 ; +cpu_lbr_stack (cpuc)->entries[i].reserved =0 ; } }@@ -990,11 +985,11 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data, {if (is_branch_counters_group (event)) {intel_pmu_lbr_counters_reorder (cpuc, event); -perf_sample_save_brstack (data, event, &cpuc->lbr_stack, cpuc->lbr_counters); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc), cpuc->lbr_counters);return ; } -perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); }static void intel_pmu_arch_lbr_read (struct cpu_hw_events *cpuc)@@ -1216,21 +1211,22 @@ static const int arch_lbr_br_type_map[ARCH_LBR_BR_TYPE_MAP_MAX] = {static void intel_pmu_lbr_filter (struct cpu_hw_events *cpuc) { -u64 from, to;int br_sel = cpuc->br_sel;int i, j, type, from_plm, to_plm;bool compress = false; +u64 from, to;/* if sampling all branches, then nothing to filter */ if (((br_sel &X86_BR_ALL ) ==X86_BR_ALL ) && ((br_sel &X86_BR_TYPE_SAVE ) !=X86_BR_TYPE_SAVE ))return ; -for (i =0 ; i < cpuc->lbr_stack.nr; i++) { +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; i++) { +struct perf_branch_entry *be = &cpu_lbr_stack (cpuc)->entries[i]; - from = cpuc->lbr_entries[i].from; - to = cpuc->lbr_entries[i].to; - type = cpuc->lbr_entries[i].type; + from = be->from; + to = be->to; + type = be->type;/* * Parse the branch type recorded in LBR_x_INFO MSR. @@ -1242,9 +1238,9 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc) to_plm =kernel_ip (to) ?X86_BR_KERNEL :X86_BR_USER ; type = arch_lbr_br_type_map[type] | to_plm; }else - type =branch_type (from, to, cpuc->lbr_entries[i].abort); + type =branch_type (from, to, be->abort);if (type !=X86_BR_NONE && (br_sel &X86_BR_ANYTX )) { -if (cpuc->lbr_entries[i].in_tx) +if (be->in_tx) type |=X86_BR_IN_TX ;else type |=X86_BR_NO_TX ;@@ -1258,27 +1254,27 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc)*/ if (type ==X86_BR_NONE || (br_sel & type) != type || (!(br_sel &X86_BR_KERNEL ) && (from_plm &X86_BR_KERNEL ))) { - cpuc->lbr_entries[i].from =0 ; + be->from =0 ; compress = true; }if ((br_sel &X86_BR_TYPE_SAVE ) ==X86_BR_TYPE_SAVE ) - cpuc->lbr_entries[i].type =common_branch_type (type); + be->type =common_branch_type (type); }if (!compress)return ;/* remove all entries with from=0 */ -for (i =0 ; i < cpuc->lbr_stack.nr; ) { -if (!cpuc->lbr_entries[i].from) { +for (i =0 ; i <cpu_lbr_stack (cpuc)->nr; ) { +if (!cpu_lbr_stack (cpuc)->entries[i].from) { j = i; -while (++j < cpuc->lbr_stack.nr) { - cpuc->lbr_entries[j-1 ] = cpuc->lbr_entries[j]; +while (++j <cpu_lbr_stack (cpuc)->nr) { +cpu_lbr_stack (cpuc)->entries[j-1 ] =cpu_lbr_stack (cpuc)->entries[j]; cpuc->lbr_counters[j-1 ] = cpuc->lbr_counters[j]; } - cpuc->lbr_stack.nr--; -if (!cpuc->lbr_entries[i].from) +cpu_lbr_stack (cpuc)->nr--; +if (!cpu_lbr_stack (cpuc)->entries[i].from)continue ; } i++;@@ -1292,9 +1288,9 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)/* Cannot get TOS for large PEBS and Arch LBR */ if (cpu_feature_enabled (X86_FEATURE_ARCH_LBR ) || (cpuc->n_pebs == cpuc->n_large_pebs)) - cpuc->lbr_stack.hw_idx =-1ULL ; +cpu_lbr_stack (cpuc)->hw_idx =-1ULL ;else - cpuc->lbr_stack.hw_idx =intel_pmu_lbr_tos (); +cpu_lbr_stack (cpuc)->hw_idx =intel_pmu_lbr_tos ();intel_pmu_store_lbr (cpuc, lbr);intel_pmu_lbr_filter (cpuc);arch/x86/events/perf_event.h +8
-2@@ -447,6 +447,8 @@ struct intel_excl_cntrs {struct x86_perf_task_context ;#define MAX_LBR_ENTRIES 32 +#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES) +enum {LBR_FORMAT_32 =0x00 ,LBR_FORMAT_LIP =0x01 ,@@ -524,8 +526,7 @@ struct cpu_hw_events {*/ int lbr_users;int lbr_pebs_users; -struct perf_branch_stack lbr_stack; -struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ]; +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 lbr_counters[MAX_LBR_ENTRIES ];/* branch stack extra */ union {struct er_account *lbr_sel;@@ -587,6 +588,11 @@ struct cpu_hw_events {struct pmu *pmu; }; +static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__lbr_stack; +} +#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \ { .idxmsk64 = (n) }, \ .code = (c), \include/linux/perf_event.h +0
-0
include/linux/perf_event.h static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag) · line 121
Quoted 8 lines
include/linux/perf_event.h +1
-1@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)struct perf_branch_stack {u64 nr;u64 hw_idx; -struct perf_branch_entry entries[]; + struct perf_branch_entry entries[] __counted_by(nr); };struct task_struct ;
This looks great. I would adopt this in next version. Thanks.
PATCH v5 11/18 perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
In reply to Dapeng MiBPF LBR snapshotting currently reuses the default LBR stack (cpuc->lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context via bpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
--- arch/x86/events/intel/core.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +11 -5
@@ -2819,18 +2819,24 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags,int pmu_enabled,int lbr_enabled) { -struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot); -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); - cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); +intel_pmu_lbr_read (&bpf->stack, bpf->entries, bpf->counters); + cnt = min_t(unsigned int, cnt, bpf->stack.nr); -memcpy (entries, cpuc->lbr_entries, +memcpy (entries, bpf->entries,sizeof (struct perf_branch_entry ) * cnt);/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 11/18] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
Sashiko has reviewed this patch and found no issues.
Quoted 8 lines
BPF LBR snapshotting currently reuses the default LBR stack (
cpuc->lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context viabpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Quoted 6 lines
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Peter Zijlstra
maintainerRe: [Patch v5 11/18] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
In reply to Dapeng MiQuoted 8 lines · Dapeng Mi
BPF LBR snapshotting currently reuses the default LBR stack (
cpuc->lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context viabpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Quoted 6 lines
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>
AMD has the same problem, no?
Quoted 41 lines
--- arch/x86/events/intel/core.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)arch/x86/events/intel/core.c +11
-5@@ -2819,18 +2819,24 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags,int pmu_enabled,int lbr_enabled) { -struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot); -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); - cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); +intel_pmu_lbr_read (&bpf->stack, bpf->entries, bpf->counters); + cnt = min_t(unsigned int, cnt, bpf->stack.nr); -memcpy (entries, cpuc->lbr_entries, +memcpy (entries, bpf->entries,sizeof (struct perf_branch_entry ) * cnt);/* Signature · 1 line
2.34.1
Re: [Patch v5 11/18] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Dapeng Mi
BPF LBR snapshotting currently reuses the default LBR stack (
cpuc->lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context viabpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Quoted 7 lines
Quoted 6 lines
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>AMD has the same problem, no?
No. According what I know, currently AMD doesn't support branch sampling in IBS and the LBR is frozen in the whole PMI handling, BPF would see same LBR stack here with the PMI handler at the PMI entry. So although BPF path indeed overwrites the default cpuc->lbr_stack, it won't break anything since the snapshotted LBR stacks are same for PMI and BPF.
But this indeed causes the deviation on the coding level. A generic way is to add BPF dedicated LBR stack in cpuc structure, then use the dedicated BPF stack to store BPF snapshot for both AMD and Intel. But it could waste some memory on AMD platforms.
arch/x86/events/perf_event.h +0 -0
@@ -530,6 +530,9 @@ struct cpu_hw_events {
struct perf_branch_stack lbr_stack;
struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES]; u64 lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
+ struct perf_branch_stack bpf_lbr_stack;
+ struct perf_branch_entry bpf_lbr_entries[MAX_LBR_ENTRIES]; + u64 bpf_lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
union {
struct er_account *lbr_sel; struct er_account *lbr_ctl;
What do you prefer?
Thanks.
Quoted 42 lines
Quoted 42 lines
--- arch/x86/events/intel/core.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)arch/x86/events/intel/core.c +11
-5@@ -2819,18 +2819,24 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags,int pmu_enabled,int lbr_enabled) { -struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot); -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); - cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); +intel_pmu_lbr_read (&bpf->stack, bpf->entries, bpf->counters); + cnt = min_t(unsigned int, cnt, bpf->stack.nr); -memcpy (entries, cpuc->lbr_entries, +memcpy (entries, bpf->entries,sizeof (struct perf_branch_entry ) * cnt);/* Signature · 1 line
2.34.1
PATCH v5 12/18 perf/x86/intel: Always snapshot live hardware LBR state for BPF
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
--- arch/x86/events/intel/core.c | 10 ++++++++-- arch/x86/events/intel/lbr.c | 9 --------- 2 files changed, 8 insertions(+), 11 deletions(-)
arch/x86/events/intel/core.c +8 -2
@@ -4069,8 +4069,14 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + }/* * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +0 -9
@@ -1050,15 +1050,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc, lbr_stack, lbr_entries);intel_pmu_lbr_filter (cpuc, lbr_stack, lbr_entries, lbr_counters);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 12/18] perf/x86/intel: Always snapshot live hardware LBR state for BPF
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
__intel_pmu_snapshot_branch_stack()is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Peter Zijlstra
maintainerRe: [Patch v5 12/18] perf/x86/intel: Always snapshot live hardware LBR state for BPF
In reply to Dapeng MiQuoted 46 lines · Dapeng Mi
__intel_pmu_snapshot_branch_stack()is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 10 ++++++++-- arch/x86/events/intel/lbr.c | 9 --------- 2 files changed, 8 insertions(+), 11 deletions(-)arch/x86/events/intel/core.c +8
-2@@ -4069,8 +4069,14 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + }/* * Do not clear the LBR frozen status here; keep LBR frozen arch/x86/events/intel/lbr.c +0
-2@@ -1050,15 +1050,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS.
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack, · line 1055
Quoted 10 lines
arch/x86/events/intel/lbr.c +0
-7-* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc, lbr_stack, lbr_entries);intel_pmu_lbr_filter (cpuc, lbr_stack, lbr_entries, lbr_counters);
AFAICT AMD could be similarly modified, it currently unconditionally reads the LBR stack, irrespective of lbr_users.
Re: [Patch v5 12/18] perf/x86/intel: Always snapshot live hardware LBR state for BPF
Quoted 56 lines · Peter Zijlstra
Quoted 55 lines · Dapeng Mi
__intel_pmu_snapshot_branch_stack()is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 10 ++++++++-- arch/x86/events/intel/lbr.c | 9 --------- 2 files changed, 8 insertions(+), 11 deletions(-)arch/x86/events/intel/core.c +8
-2@@ -4069,8 +4069,14 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (&cpuc->lbr_stack, - cpuc->lbr_entries, cpuc->lbr_counters); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + }/* * Do not clear the LBR frozen status here; keep LBR frozen arch/x86/events/intel/lbr.c +0
-9@@ -1050,15 +1050,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc, lbr_stack, lbr_entries);
Quoted 3 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack, · line 1055
Quoted 1 line
arch/x86/events/intel/lbr.c +0
-0intel_pmu_lbr_filter (cpuc, lbr_stack, lbr_entries, lbr_counters);AFAICT AMD could be similarly modified, it currently unconditionally reads the LBR stack, irrespective of
lbr_users.
Yes, LBR stack is only needed to snapshot when lbr_users > 0.
Thanks.
PATCH v5 13/18 perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
In reply to Dapeng MiPEBS and non-PEBS events currently share the same LBR stack, cpuc->lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.
When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, and userspace may observe the wrong branch stack instead of the live LBR state at PMI entry.
Add a lbr_need_update flag to indicate when the LBR state must be refreshed. If PEBS events and non-PEBS LBR events overflow in the same PMI, set this flag so the non-PEBS LBR event refreshes the live LBR state. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once even when multiple non-PEBS LBR events overflow in the same PMI.
--- arch/x86/events/intel/core.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +20 -3
@@ -3863,8 +3863,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -int bit; +bool lbr_need_update = false;int handled =0 ; +int bit;inc_perf_irq_stat ();
@@ -3929,6 +3930,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/*
@@ -3944,6 +3947,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/*
@@ -4002,8 +4007,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) {static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_need_update as true. +*/ + lbr_need_update = true; + } last_period = event->hw.last_period;
@@ -4012,8 +4023,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (lbr_need_update) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + lbr_need_update = false; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 13/18] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
Sashiko has reviewed this patch and found no issues.
Quoted 17 lines
PEBS and non-PEBS events currently share the same LBR stack,
cpuc->lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, and userspace may observe the wrong branch stack instead of the live LBR state at PMI entry.
Add a
lbr_need_updateflag to indicate when the LBR state must be refreshed. If PEBS events and non-PEBS LBR events overflow in the same PMI, set this flag so the non-PEBS LBR event refreshes the live LBR state. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once even when multiple non-PEBS LBR events overflow in the same PMI.
Quoted 2 lines
Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Peter Zijlstra
maintainerRe: [Patch v5 13/18] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
In reply to Dapeng MiQuoted 78 lines · Dapeng Mi
PEBS and non-PEBS events currently share the same LBR stack,
cpuc->lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, and userspace may observe the wrong branch stack instead of the live LBR state at PMI entry.
Add a
lbr_need_updateflag to indicate when the LBR state must be refreshed. If PEBS events and non-PEBS LBR events overflow in the same PMI, set this flag so the non-PEBS LBR event refreshes the live LBR state. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once even when multiple non-PEBS LBR events overflow in the same PMI.Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)arch/x86/events/intel/core.c +14
-3@@ -3863,8 +3863,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -int bit; +bool lbr_need_update = false;int handled =0 ; +int bit;inc_perf_irq_stat ();@@ -3929,6 +3930,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/* @@ -3944,6 +3947,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/* @@ -4002,8 +4007,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) {static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_need_update as true. +*/ + lbr_need_update = true; + } last_period = event->hw.last_period;@@ -4012,8 +4023,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) {
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4036
Quoted 10 lines
arch/x86/events/intel/core.c +6
-0+if (lbr_need_update) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + lbr_need_update = false; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);
Or... you move intel_pmu_lbr_read() from handle_irq() into pmi_common() after we done the PEBS things?
Re: [Patch v5 13/18] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
Quoted 87 lines · Peter Zijlstra
Quoted 86 lines · Dapeng Mi
PEBS and non-PEBS events currently share the same LBR stack,
cpuc->lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, and userspace may observe the wrong branch stack instead of the live LBR state at PMI entry.
Add a
lbr_need_updateflag to indicate when the LBR state must be refreshed. If PEBS events and non-PEBS LBR events overflow in the same PMI, set this flag so the non-PEBS LBR event refreshes the live LBR state. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once even when multiple non-PEBS LBR events overflow in the same PMI.Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)arch/x86/events/intel/core.c +20
-3@@ -3863,8 +3863,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -int bit; +bool lbr_need_update = false;int handled =0 ; +int bit;inc_perf_irq_stat ();@@ -3929,6 +3930,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/* @@ -3944,6 +3947,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + + lbr_need_update = true; }/* @@ -4002,8 +4007,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) {static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_need_update as true. +*/ + lbr_need_update = true; + } last_period = event->hw.last_period;@@ -4012,8 +4023,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (lbr_need_update) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + lbr_need_update = false; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }
Quoted 4 lines
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4036
Quoted 2 lines
arch/x86/events/intel/core.c +0
-0x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);Or... you move
intel_pmu_lbr_read()fromhandle_irq()intopmi_common()after we done the PEBS things?
Yes, I ever tried to do that, but I found it would break the PEBS LBR sampling on the platforms prior to ICE LAKE. The PEBS doesn't support snapshot LBR stack on these old platforms, so PMI handler has to snapshot the LBR stack before draining the PEBS buffer.
See below code in setup_pebs_fixed_sample_data().
/*
* v3 supplies an accurate time stamp, so we use that
* for the time stamp.
*
* We can only do this for the default trace clock. */
if (x86_pmu.intel_cap.pebs_format >= 3) setup_pebs_time(event, data, pebs->tsc);
perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
Thanks.
Peter Zijlstra
maintainerRe: [Patch v5 13/18] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
Quoted 20 lines · Dapeng
Quoted 14 lines
Quoted 14 lines
arch/x86/events/intel/core.c +7
-1@@ -4012,8 +4023,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (lbr_need_update) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + lbr_need_update = false; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }Quoted 5 lines
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4036
Quoted 2 lines
arch/x86/events/intel/core.c +0
-0x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);Or... you move
intel_pmu_lbr_read()fromhandle_irq()intopmi_common()after we done the PEBS things?
Quoted 4 lines
Yes, I ever tried to do that, but I found it would break the PEBS LBR sampling on the platforms prior to ICE LAKE. The PEBS doesn't support snapshot LBR stack on these old platforms, so PMI handler has to snapshot the LBR stack before draining the PEBS buffer.
Argh; we have so many variations :-(
Still, re-reading those MSRs seems wasteful, esp. if most modern chips will be affected.
Can't we have a toggle between early and late LBR read or something?
Re: [Patch v5 13/18] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events if needed
Quoted 24 lines · Peter Zijlstra
Quoted 14 lines · Dapeng
Quoted 14 lines
Quoted 14 lines
arch/x86/events/intel/core.c +7
-1@@ -4012,8 +4023,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (lbr_need_update) { +intel_pmu_lbr_read (&cpuc->lbr_stack, + cpuc->lbr_entries, cpuc->lbr_counters); + lbr_need_update = false; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }Quoted 8 lines
Quoted 4 lines
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4036
Quoted 2 lines
arch/x86/events/intel/core.c +0
-0x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);Or... you move
intel_pmu_lbr_read()fromhandle_irq()intopmi_common()after we done the PEBS things?Yes, I ever tried to do that, but I found it would break the PEBS LBR sampling on the platforms prior to ICE LAKE. The PEBS doesn't support snapshot LBR stack on these old platforms, so PMI handler has to snapshot the LBR stack before draining the PEBS buffer.
Quoted 6 lines
Argh; we have so many variations :-(
Still, re-reading those MSRs seems wasteful, esp. if most modern chips will be affected.
Can't we have a toggle between early and late LBR read or something?
Hmm, yes. I would refactor the code and eliminate the duplicated LBR reading. Thanks.
PATCH v5 14/18 perf/x86/intel: Account arch-PEBS in lbr_pebs_users
In reply to Dapeng Milbr_pebs_users tracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.
Add the missing lbr_pebs_users accounting for arch-PEBS as well.
--- arch/x86/events/intel/lbr.c | 4 ++-- arch/x86/events/perf_event.h | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
arch/x86/events/intel/lbr.c +2 -2
@@ -618,7 +618,7 @@ void intel_pmu_lbr_add(struct perf_event *event)* be 'new'. Conversely, a new event can get installed through the * context switch path for the first time. */ -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++;perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)
@@ -689,7 +689,7 @@ void intel_pmu_lbr_del(struct perf_event *event)if (event->hw.flags &PERF_X86_EVENT_LBR_SELECT ) cpuc->lbr_select =0 ; -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users--; cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 );
arch/x86/events/perf_event.h +7 -0
@@ -2085,6 +2085,13 @@ static inline bool intel_pmu_has_pebs(void) return x86_pmu.ds_pebs || x86_pmu.arch_pebs; } +static inline bool intel_pmu_has_pebs_lbr (struct pmu *pmu) +{ +return x86_pmu.arch_pebs ? + !!(hybrid (pmu, arch_pebs_cap).caps &ARCH_PEBS_LBR ) : + !!x86_pmu.intel_cap.pebs_baseline; +} +#else /* CONFIG_CPU_SUP_INTEL */ static inline void reserve_ds_buffers (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 14/18] perf/x86/intel: Account arch-PEBS in lbr_pebs_users
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
lbr_pebs_userstracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.Add the missing
lbr_pebs_usersaccounting for arch-PEBS as well.Reported-by Sashiko <[email protected]>Fixes d243d0bb64af ("perf/x86/intel: Initialize architectural PEBS")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
PATCH v5 15/18 perf/x86/intel: Clear raw LBR counters in the LBR stack
In reply to Dapeng MiRaw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.
If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.
--- arch/x86/events/intel/core.c | 1 + arch/x86/events/intel/lbr.c | 14 ++++++++++++++ arch/x86/events/perf_event.h | 3 +++ 3 files changed, 18 insertions(+)
arch/x86/events/intel/core.c +1 -0
@@ -2834,6 +2834,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot);intel_pmu_lbr_read (&bpf->stack, bpf->entries, bpf->counters); +intel_pmu_lbr_clear_counters (&bpf->stack, bpf->entries); cnt = min_t(unsigned int, cnt, bpf->stack.nr); memcpy(entries, bpf->entries,
arch/x86/events/intel/lbr.c +14 -0
@@ -996,6 +996,19 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc, } } +void intel_pmu_lbr_clear_counters (struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries) +{ +int i; + +if (!(x86_pmu.flags &PMU_FL_BR_CNTR )) +return ; + +/* reserved field stores raw branch counters, clear it. */ +for (i =0 ; i < lbr_stack->nr; i++) + lbr_entries[i].reserved =0 ; +} +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event)
@@ -1006,6 +1019,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,return ; } +intel_pmu_lbr_clear_counters (&cpuc->lbr_stack, cpuc->lbr_entries);perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); }
arch/x86/events/perf_event.h +3 -0
@@ -1971,6 +1971,9 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);void intel_pebs_init (void ); +void intel_pmu_lbr_clear_counters (struct perf_branch_stack *lbr_stack, +struct perf_branch_entry *lbr_entries); +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 15/18] perf/x86/intel: Clear raw LBR counters in the LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
Raw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.
If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.
Reported-by Sashiko <[email protected]>Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
PATCH v5 16/18 perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
In reply to Dapeng MiWhen handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
This situation is rare, the performance impact from extra refill should be negligible.
--- arch/x86/events/intel/core.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/events/intel/core.c +10 -1
@@ -3864,6 +3864,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +bool lbr_counter_cleared = false;bool lbr_need_update = false;int handled =0 ;int bit;
@@ -4025,12 +4026,20 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period);if (has_branch_stack (event)) { -if (lbr_need_update) { +/* +* The raw LBR counters stored in the reserved field of +* the LBR entries were cleared during processing of the +* previous LBR event, so refresh the LBR stack again if +* this event needs the LBR counters. +*/ +if (lbr_need_update || + (lbr_counter_cleared &&is_branch_counters_group (event))) {intel_pmu_lbr_read (&cpuc->lbr_stack, cpuc->lbr_entries, cpuc->lbr_counters); lbr_need_update = false; }intel_pmu_lbr_save_brstack (&data, cpuc, event); + lbr_counter_cleared = true; }x86_pmu_clear_perf_regs (regs);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 16/18] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
When handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
Quoted 6 lines
This situation is rare, the performance impact from extra refill should be negligible.
Reported-by Sashiko <[email protected]>Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
PATCH v5 17/18 perf/x86/intel: Pass LBR stack size into store_lbr()
In reply to Dapeng Miintel_pmu_store_lbr() assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.
Add a lbr_nr parameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.
--- arch/x86/events/intel/ds.c | 8 ++++---- arch/x86/events/intel/lbr.c | 16 +++++++++------- arch/x86/events/perf_event.h | 2 +- 3 files changed, 14 insertions(+), 12 deletions(-)
arch/x86/events/intel/ds.c +4 -4
@@ -2591,12 +2591,12 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,if (format_group &PEBS_DATACFG_LBRS ) {struct lbr_entry *lbr = next_record; -int num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) +unsigned int num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) &0xff ) +1 ; next_record = next_record + num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
@@ -2777,7 +2777,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,if (header->lbr) {struct arch_pebs_lbr_header *lbr_header = next_record;struct lbr_entry *lbr; -int num_lbr; +unsigned int num_lbr; next_record = lbr_header +1 ; lbr = next_record;
@@ -2788,7 +2788,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, next_record += num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
arch/x86/events/intel/lbr.c +9 -7
@@ -911,7 +911,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)static_assert ((64 -PERF_BRANCH_ENTRY_INFO_BITS_MAX ) >LBR_INFO_BR_CNTR_NUM *LBR_INFO_BR_CNTR_BITS ); -static void intel_pmu_store_lbr (struct lbr_entry *entries, +static void intel_pmu_store_lbr (struct lbr_entry *entries,int lbr_nr,struct perf_branch_stack *lbr_stack,struct perf_branch_entry *lbr_entries) {
@@ -921,7 +921,7 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,bool mispred;int i; -for (i =0 ; i < x86_pmu.lbr_nr; i++) { +for (i =0 ; i < lbr_nr; i++) { lbr = entries ? &entries[i] :NULL ; e = &lbr_entries[i];
@@ -1027,7 +1027,7 @@ static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,struct perf_branch_stack *lbr_stack, struct perf_branch_entry *lbr_entries) { -intel_pmu_store_lbr (NULL ,lbr_stack ,lbr_entries ); + intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack, lbr_entries); } static void
@@ -1038,14 +1038,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;if (!xsave) { -intel_pmu_store_lbr (NULL , lbr_stack, lbr_entries); +intel_pmu_store_lbr (NULL , x86_pmu.lbr_nr, lbr_stack, lbr_entries);return ; } xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR );if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { -intel_pmu_store_lbr (xsave->lbr.entries, lbr_stack, lbr_entries); +intel_pmu_store_lbr (xsave->lbr.entries, x86_pmu.lbr_nr, + lbr_stack, lbr_entries); }else {/* Clear the LBR stack to avoid leaking stale entries. */ lbr_stack->nr =0 ;
@@ -1323,7 +1324,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc, } } -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr) +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1334,7 +1335,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)else cpuc->lbr_stack.hw_idx =intel_pmu_lbr_tos (); -intel_pmu_store_lbr (lbr, &cpuc->lbr_stack, cpuc->lbr_entries); +intel_pmu_store_lbr (lbr,min (lbr_nr, x86_pmu.lbr_nr), + &cpuc->lbr_stack, cpuc->lbr_entries);intel_pmu_lbr_filter (cpuc, &cpuc->lbr_stack, cpuc->lbr_entries, cpuc->lbr_counters); }
arch/x86/events/perf_event.h +1 -1
@@ -1967,7 +1967,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);void intel_pmu_drain_pebs_buffer (void ); -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr); +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr);void intel_pebs_init (void );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 17/18] perf/x86/intel: Pass LBR stack size into store_lbr()
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
intel_pmu_store_lbr()assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.Add a
lbr_nrparameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.Reported-by Sashiko <[email protected]>Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
PATCH v5 18/18 perf/core: Don't overwrite shared LBR stack depth
In reply to Dapeng Miperf_sample_save_brstack() currently clamps the shared LBR stack depth to min(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a small sample_max_stack from one event can shrink the depth for all others in the same PMI.
As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.
--- include/linux/perf_event.h | 13 ++++++++++--- kernel/events/core.c | 8 ++++---- 2 files changed, 14 insertions(+), 7 deletions(-)
include/linux/perf_event.h +10 -3
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline u64 max_lbr_stack_depth (struct perf_event *event, +struct perf_branch_stack *brs) +{ +return min_t (u16, event->attr.sample_max_stack, brs->nr); +} +static inline void perf_sample_save_brstack (struct perf_sample_data *data,struct perf_event *event,struct perf_branch_stack *brs,u64 *brs_cntr) {int size =sizeof (u64);/* nr */ +u64 lbr_nr;if (!has_branch_stack (event))return ;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,if (branch_sample_hw_index (event)) size +=sizeof (u64); - brs->nr =min_t (u16, event->attr.sample_max_stack, brs->nr); + lbr_nr =max_lbr_stack_depth (event, brs); - size += brs->nr *sizeof (struct perf_branch_entry ); + size += lbr_nr *sizeof (struct perf_branch_entry );/* * The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,* of events of each branch. */ if (brs_cntr) - size += brs->nr *sizeof (u64); + size += lbr_nr *sizeof (u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr;
kernel/events/core.c +4 -4
@@ -8442,12 +8442,12 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {if (data->br_stack) { +u64 lbr_nr =max_lbr_stack_depth (event, data->br_stack);size_t size; - size = data->br_stack->nr - *sizeof (struct perf_branch_entry ); + size = lbr_nr *sizeof (struct perf_branch_entry ); -perf_output_put (handle, data->br_stack->nr); +perf_output_put (handle, lbr_nr);if (branch_sample_hw_index (event))perf_output_put (handle, data->br_stack->hw_idx);perf_output_copy (handle, data->br_stack->entries, size);
@@ -8456,7 +8456,7 @@ void perf_output_sample(struct perf_output_handle *handle,* right after the struct perf_branch_stack. */ if (data->br_stack_cntr) { - size = data->br_stack->nr *sizeof (u64); + size = lbr_nr *sizeof (u64);perf_output_copy (handle, data->br_stack_cntr, size); } }else {
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v5 18/18] perf/core: Don't overwrite shared LBR stack depth
Sashiko has reviewed this patch and found no issues.
Quoted 12 lines
perf_sample_save_brstack()currently clamps the shared LBR stack depth tomin(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a smallsample_max_stackfrom one event can shrink the depth for all others in the same PMI.As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in
perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.
Quoted 3 lines
Reported-by Sashiko <[email protected]>Fixes c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18