perf/x86: Fix multiple LBR sampling issues
Subsystems (8)
v7 cover perf/x86: Fix multiple LBR sampling issues
This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.
The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.
Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack depth never exceeds MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware has actually written LBR data into the XSAVE area before parsing it.
- Patch 03-04: Fix a potential LBR stack out-of-bounds access on AMD and Intel, respectively.
- Patch 05-08: Fix incorrect PMU and LBR re-enabling in perf_snapshot_branch_stack() on AMD and Intel, respectively.
- Patch 09: Keep the LBR frozen throughout PMI processing so that all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add bounds tracking for the number of entries counted in struct perf_branch_stack.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshots to avoid clobbering the default LBR stack used by other LBR events.
- Patch 13: Make perf_snapshot_branch_stack() snapshot the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14-16: Account lbr_users/lbr_pebs_users for arch-PEBS and pure branch counter events.
- Patch 17-19: Prevent the PEBS LBR stack from overwriting the non-PEBS LBR stack in several corner cases.
- Patch 20-21: Ensure LBR counters are not lost when multiple LBR counter events overflow in the same PMI.
- Patch 22: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid an LBR stack out-of-bounds access.
- Patch 23: Fix the shared LBR stack depth truncation issue.
- Patch 24: Fix incorrect clearing of the PERF_METRICS overflow bit.
Changes in v7:
- Patch 06/08: Set/clear the cpuc->enabled flag to track the PMU enabling state in the *_snapshot_branch_stack() helpers. (Sashiko)
- Patch 10: Rename __lbr_stack[] to __bhrb_stack[] and the related helpers accordingly, to stay consistent with the original powerpc code.
- Patch 14-16: New patches. Fix lbr_users/lbr_pebs_users not being accounted correctly in several corner cases. (Sashiko)
- Patch 17-19: New patches. Fix the non-PEBS LBR stack being overwritten by the PEBS LBR stack in two corner cases. (Sashiko)
- Patch 24: New patch. Fix the PERF_METRICS overflow bit not being cleared correctly.
Changes in v6:
- Patch 01: Move LBR stack depth validation into common init_hw_perf_events(). (Peter)
- Patch 10: New patch, add __counted_by() tracking for flexible array entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD. (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading. (Peter)
- Patch 16: New patch, remove dead lbr_pebs_users.
Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.
Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).
Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/
History:
v6: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (23):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel: Limit branch snapshots to valid entries perf/x86: Add lbr_enabled to track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling
perf: Add counted-bounds tracking for perf_branch_stack perf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/intel: Account lbr_pebs_users for arch-PEBS events perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events perf/x86/intel: Account LBR user counts for pure branch counter events perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI
handler
perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR
users
perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refresh LBR stack for LBR counter events in PMI
handler
perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr() perf/core: Don't mutate shared branch stack depth perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing
Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries
arch/powerpc/perf/core-book3s.c | 47 ++++-- arch/x86/events/amd/brs.c | 7 +- arch/x86/events/amd/core.c | 25 +++- arch/x86/events/amd/lbr.c | 48 ++++--- arch/x86/events/core.c | 8 +- arch/x86/events/intel/core.c | 231 +++++++++++++++++++++++++----- arch/x86/events/intel/ds.c | 31 ++-- arch/x86/events/intel/lbr.c | 169 +++++++++++++--------- arch/x86/events/perf_event.h | 52 ++++++- arch/x86/include/asm/perf_event.h | 1 + drivers/perf/arm_brbe.c | 5 + include/linux/perf_event.h | 15 +- kernel/events/core.c | 8 +- kernel/trace/bpf_trace.c | 6 +- 14 files changed, 477 insertions(+), 176 deletions(-)
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
Signature · 1 line
2.34.1
PATCH v7 1/24 perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.
--- arch/x86/events/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/events/core.c +6 -0
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)if (!x86_pmu.config_mask) x86_pmu.config_mask =X86_RAW_EVENT_MASK ; +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR depth %d exceeds max %d, disabling LBR\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +perf_events_lapic_init ();register_nmi_handler (NMI_LOCAL , perf_event_nmi_handler,0 ,"PMI" );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 01/24] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
Disable LBR if x86_pmu.lbr_nr is larger than
MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v7 2/24 perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
In reply to Dapeng Miintel_pmu_arch_lbr_read_xsave() did not verify whether LBR data was actually present in the XSAVE area. When the LBR state was in the INIT state, the hardware might not have written valid data to the XSAVE buffer, allowing stale or incomplete data to be cached.
Add an explicit presence check for the LBR XSAVE state before updating the cached LBR stack. Also clear the cached LBR stack in intel_pmu_lbr_reset() to prevent stale entries from being reused across context switches and similar transitions.
--- arch/x86/events/intel/lbr.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/x86/events/intel/lbr.c +8 -1
@@ -198,6 +198,7 @@ void intel_pmu_lbr_reset(void) x86_pmu.lbr_reset (); + cpuc->lbr_stack.nr =0 ; cpuc->last_task_ctx =NULL ; cpuc->last_log_id =0 ;if (!cpu_feature_enabled (X86_FEATURE_ARCH_LBR ) && cpuc->lbr_select)
@@ -1010,9 +1011,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)intel_pmu_store_lbr (cpuc,NULL );return ; } + xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); + }else { +/* Clear the LBR stack to avoid leaking stale entries. */ + cpuc->lbr_stack.nr =0 ; + } }void intel_pmu_lbr_read (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 02/24] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 5 lines
intel_pmu_arch_lbr_read_xsave()did not verify whether LBR data was actually present in the XSAVE area. When the LBR state was in the INIT state, the hardware might not have written valid data to the XSAVE buffer, allowing stale or incomplete data to be cached.
Quoted 7 lines
Add an explicit presence check for the LBR XSAVE state before updating the cached LBR stack. Also clear the cached LBR stack in
intel_pmu_lbr_reset()to prevent stale entries from being reused across context switches and similar transitions.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v7 3/24 perf/x86/amd/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiFrom: Sandipan Das <[email protected]>
The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.
--- arch/x86/events/amd/core.c | 2 +- arch/x86/events/amd/lbr.c | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/events/amd/core.c +1 -1
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u cpuc =this_cpu_ptr (&cpu_hw_events);amd_pmu_lbr_read (); - cnt =min (cnt, x86_pmu.lbr_nr); + cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );
arch/x86/events/amd/lbr.c +3 -1
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)struct branch_entry entry;int out =0 , idx, i; -if (!cpuc->lbr_users) +if (!cpuc->lbr_users) { + cpuc->lbr_stack.nr =0 ;return ; + }for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 03/24] perf/x86/amd/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
From: Sandipan Das <[email protected]>
The branch snapshot callback used by
bpf_get_branch_snapshot()copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read()skips reading the registers, so that a stale count is never reused.
Quoted 6 lines
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Reported-by Sashiko <[email protected]>Signed-off-by Sandipan Das <[email protected]>Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v7 4/24 perf/x86/intel: Limit branch snapshots to valid entries
In reply to Dapeng MiThe branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries, and reset lbr_stack.nr when there are no active LBR events or the guest owns the LBR hardware, so that stale or guest LBR data can't leak.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed by making the BPF snapshot always read the live hardware LBR state.
--- arch/x86/events/intel/core.c | 2 +- arch/x86/events/intel/lbr.c | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +1 -1
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr); + cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);intel_pmu_enable_all (0 );
arch/x86/events/intel/lbr.c +7 -2
@@ -1026,14 +1026,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +/* * Don't read when all LBRs users are using adaptive PEBS. * * This could be smarter and actually check the event, * but this simple approach seems to work for now. */ -if (!cpuc->lbr_users ||vlbr_exclude_host () || - cpuc->lbr_users == cpuc->lbr_pebs_users) +if (cpuc->lbr_users == cpuc->lbr_pebs_users)return ; x86_pmu.lbr_read (cpuc);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 04/24] perf/x86/intel: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 15 lines
The branch snapshot callback used by
bpf_get_branch_snapshot()always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.Limit copying to lbr_stack.nr entries, and reset lbr_stack.nr when there are no active LBR events or the guest owns the LBR hardware, so that stale or guest LBR data can't leak.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed by making the BPF snapshot always read the live hardware LBR state.
Quoted 6 lines
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Co-developed-by Sandipan Das <[email protected]>Signed-off-by Sandipan Das <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [Patch v7 04/24] perf/x86/intel: Limit branch snapshots to valid entries
In reply to Dapeng MiTL;DR: Resetting the shared lbr_stack.nr in intel_pmu_lbr_read() may corrupt an in-flight host adaptive-PEBS branch-stack sample when KVM vLBR is active and a BPF program calls…
Quoted 1 line
perf/x86/intel: Limit branch snapshots to valid entries
TL;DR: Resetting the shared lbr_stack.nr in intel_pmu_lbr_read() may corrupt an in-flight host adaptive-PEBS branch-stack sample when KVM vLBR is active and a BPF program calls bpf_get_branch_snapshot(); a later patch fixes it.
Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0
[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(void) · line 1034
Quoted 10 lines
arch/x86/events/intel/lbr.c +6
-0@@ -1026,14 +1026,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +
Can this reset of cpuc->lbr_stack.nr corrupt a host adaptive-PEBS sample that is still in flight?
Before this patch the vlbr_exclude_host() case returned without writing anything. Now the BPF snapshot path writes to the same per-CPU struct that a PMI-time sample may still point at:
intel_pmu_handle_irq()
handle_pmi_common() -> x86_pmu_drain_pebs
__intel_pmu_pebs_last_event()
setup_pebs_adaptive_sample_data()
intel_pmu_store_pebs_lbrs(lbr) /* cpuc->lbr_stack.nr = N */
intel_pmu_lbr_save_brstack()
perf_sample_save_brstack() /* data->dyn_size += N entries */
/* data->br_stack = &cpuc->lbr_stack */
perf_event_overflow()
__perf_event_overflow()
bpf_overflow_handler()
BPF prog -> bpf_get_branch_snapshot()
__intel_pmu_snapshot_branch_stack()
intel_pmu_lbr_read() /* vlbr_exclude_host(): nr = 0 */
overflow_handler() -> perf_output_sample()
perf_output_put(handle, data->br_stack->nr); /* now 0 */
perf_output_copy(handle, data->br_stack->entries, ...);The record header was sized for N branch entries, but nr is now 0 and no entries are written, so the PEBS LBR data for that sample is lost and the tail of the record is never written.
This looks reachable when KVM's vLBR event is active (it sets INTEL_PMC_IDX_FIXED_VLBR in intel_ctrl_exclude_host_mask) alongside a host PEBS event using PERF_SAMPLE_BRANCH_STACK, with a perf_event BPF program calling bpf_get_branch_snapshot(). Before this patch the same sequence produced a consistent sample.
The !cpuc->lbr_users half of the condition does not look affected, since any event whose sample references cpuc->lbr_stack is counted in lbr_users.
The series appears to fix this later in 61909a79de63 ("perf/x86/intel:
Use a dedicated LBR stack for BPF snapshotting"), where intel_pmu_lbr_read() takes the stack to fill as a parameter. Is it worth avoiding the shared-stack write in this patch, so that the intermediate commit does not have this window?
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(void) · line 1037
Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0/* * Don't read when all LBRs users are using adaptive PEBS. *
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
Re: [Patch v7 04/24] perf/x86/intel: Limit branch snapshots to valid entries
Quoted 55 lines · On 11/18/2026 12:10 PM, [email protected] wrote:
Quoted 1 line
perf/x86/intel: Limit branch snapshots to valid entries
TL;DR: Resetting the shared lbr_stack.nr in
intel_pmu_lbr_read()may corrupt an in-flight host adaptive-PEBS branch-stack sample when KVM vLBR is active and a BPF program callsbpf_get_branch_snapshot(); a later patch fixes it.Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(void) · line 1034
Quoted 10 lines
arch/x86/events/intel/lbr.c +6
-0@@ -1026,14 +1026,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +Can this reset of
cpuc->lbr_stack.nrcorrupt a host adaptive-PEBS sample that is still in flight?Before this patch the
vlbr_exclude_host()case returned without writing anything. Now the BPF snapshot path writes to the same per-CPU struct that a PMI-time sample may still point at:
intel_pmu_handle_irq()
handle_pmi_common()->x86_pmu_drain_pebs__intel_pmu_pebs_last_event() setup_pebs_adaptive_sample_data() intel_pmu_store_pebs_lbrs(lbr) /* cpuc->lbr_stack.nr = N */ intel_pmu_lbr_save_brstack() perf_sample_save_brstack() /* data->dyn_size += N entries */ /* data->br_stack = &cpuc->lbr_stack */ perf_event_overflow() __perf_event_overflow() bpf_overflow_handler() BPF prog -> bpf_get_branch_snapshot() __intel_pmu_snapshot_branch_stack() intel_pmu_lbr_read() /* vlbr_exclude_host(): nr = 0 */ overflow_handler() -> perf_output_sample() perf_output_put(handle, data->br_stack->nr); /* now 0 */ perf_output_copy(handle, data->br_stack->entries, ...);The record header was sized for N branch entries, but nr is now 0 and no entries are written, so the PEBS LBR data for that sample is lost and the tail of the record is never written.
This looks reachable when KVM's vLBR event is active (it sets
INTEL_PMC_IDX_FIXED_VLBRinintel_ctrl_exclude_host_mask) alongside a host PEBS event usingPERF_SAMPLE_BRANCH_STACK, with aperf_eventBPF program callingbpf_get_branch_snapshot(). Before this patch the same sequence produced a consistent sample.
Quoted 8 lines
The !cpuc->lbr_users half of the condition does not look affected, since any event whose sample references
cpuc->lbr_stackis counted inlbr_users.The series appears to fix this later in
61909a79de63("perf/x86/intel:
Use a dedicated LBR stack for BPF snapshotting"), whereintel_pmu_lbr_read()takes the stack to fill as a parameter. Is it worth avoiding the shared-stack write in this patch, so that the intermediate commit does not have this window?
Yeah, the patch order can be refined. But I'm not sure if it's worthy to post a new version just for the patch sequence. Anyway, I would refine this patch order if there is a new version. Thanks.
Quoted 9 lines
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read(void) · line 1037
Quoted 4 lines
arch/x86/events/intel/lbr.c +0
-0/* * Don't read when all LBRs users are using adaptive PEBS. * ---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
PATCH v7 5/24 perf/x86: Add lbr_enabled to track LBR state
In reply to Dapeng MiAdd a lbr_enabled flag to track whether LBR is enabled. This allows the branch snapshot callbacks to avoid blindly re-enabling LBR in perf_snapshot_branch_stack() without checking whether LBR has already been disabled.
--- arch/x86/events/amd/lbr.c | 2 ++ arch/x86/events/intel/core.c | 5 +++++ arch/x86/events/intel/lbr.c | 2 ++ arch/x86/events/perf_event.h | 9 +++++++++ 4 files changed, 18 insertions(+)
arch/x86/events/amd/lbr.c +2 -0
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)if (!cpuc->lbr_users || !x86_pmu.lbr_nr)return ; + cpuc->lbr_enabled =1 ; +/* Set hardware branch filter */ if (cpuc->lbr_select) { lbr_select = cpuc->lbr_sel->config &LBR_SELECT_MASK ;
arch/x86/events/intel/core.c +5 -0
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* It needs to be restored when leaving the handler. */ pmu_enabled = cpuc->enabled; + +/* If PMU version < 4, LBR freeze disables LBR. */ +if (x86_pmu.version <4 ) + cpuc->lbr_enabled =0 ; +/* * In general, the early ACK is only applied for old platforms. * For the big core starts from Haswell, the late ACK should be
arch/x86/events/intel/lbr.c +2 -0
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)if (pmi && x86_pmu.version >=4 )return ; + cpuc->lbr_enabled =1 ; +/* * No need to reprogram LBR_SELECT in a PMI, as it * did not change.
arch/x86/events/perf_event.h +9 -0
@@ -473,6 +473,7 @@ struct cpu_hw_events {unsigned long active_mask[BITS_TO_LONGS (X86_PMC_IDX_MAX )];unsigned long dirty[BITS_TO_LONGS (X86_PMC_IDX_MAX )];int enabled; +int lbr_enabled;int n_events;/* the # of events in the below arrays */ int n_added;/* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event); static __always_inline void __amd_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 dbg_ctl, dbg_extn_cfg; + cpuc->lbr_enabled =0 ; +rdmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg);wrmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN );
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void) static __always_inline void __intel_pmu_arch_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->lbr_enabled =0 ;wrmsrq (MSR_ARCH_LBR_CTL ,0 ); } static __always_inline void __intel_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 debugctl; + cpuc->lbr_enabled =0 ;rdmsrq (MSR_IA32_DEBUGCTLMSR , debugctl); debugctl &= ~(DEBUGCTLMSR_LBR |DEBUGCTLMSR_FREEZE_LBRS_ON_PMI );wrmsrq (MSR_IA32_DEBUGCTLMSR , debugctl);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 05/24] perf/x86: Add lbr_enabled to track LBR state
Sashiko has reviewed this patch and found no issues.
Quoted 6 lines
Add a
lbr_enabledflag to track whether LBR is enabled. This allows the branch snapshot callbacks to avoid blindly re-enabling LBR inperf_snapshot_branch_stack()without checking whether LBR has already been disabled.Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v7 6/24 perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Miamd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/amd/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/events/amd/core.c +12 -3
@@ -932,22 +932,31 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled; + cpuc->enabled =0 ;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events); -amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (pmu_enabled) { + cpuc->enabled =1 ; +amd_pmu_core_enable_all (); + } +if (lbr_enabled) +amd_pmu_lbr_enable_all ();local_irq_restore (flags);return cnt;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 06/24] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] perf/x86/amd: PMU enabled before LBR in amd_pmu_v2_snapshot_branch_stack() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] perf/x86/amd: PMU enabled before LBR in amd_pmu_v2_snapshot_branch_stack()
--
commit 496e4315c6fe26a4c320a0d5de1185d312c47c38
Author: Dapeng Mi <[email protected]>
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit saves the enabled states of the core PMU and LBR in amd_pmu_v2_snapshot_branch_stack() and only restores them if they were enabled before the snapshot was taken.
Quoted 26 lines
arch/x86/events/amd/core.c +6
-2@@ -932,22 +932,31 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled; + cpuc->enabled =0 ;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events); -amd_pmu_lbr_read ();
arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 959
Quoted 10 lines
arch/x86/events/amd/core.c +6
-1cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (pmu_enabled) { + cpuc->enabled =1 ; +amd_pmu_core_enable_all (); + } +if (lbr_enabled) +amd_pmu_lbr_enable_all ();
[Severity: Low]
Should amd_pmu_lbr_enable_all() run before amd_pmu_core_enable_all(), matching amd_pmu_v2_enable_all()?
Calling amd_pmu_core_enable_all() first restarts the hardware PMCs while amd_pmu_lbr_enable_all() is still reprogramming the LBR MSRs, so the PMCs count the MSR accesses in amd_pmu_lbr_enable_all() and can overflow before DEBUGCTLMSR_FREEZE_LBRS_ON_PMI and DBG_EXTN_CFG_LBRV2EN are restored.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v7 7/24 perf/x86/intel: Add helper to enable PMU without LBR
In reply to Dapeng MiCurrently, __intel_pmu_enable_all() also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper, ___intel_pmu_enable_all(), so the PMU can be enabled independently. This allows the branch snapshot path to re-enable the PMU without touching LBR.
Also remove the unused added argument from __intel_pmu_enable_all().
No functional change intended.
--- arch/x86/events/intel/core.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +9 -5
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)intel_pmu_lbr_disable_all (); } -static void __intel_pmu_enable_all (int added,bool pmi) +static void ___intel_pmu_enable_all (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 intel_ctrl =hybrid (cpuc->pmu, intel_ctrl); -intel_pmu_lbr_enable_all (pmi); -if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {wrmsrq (MSR_ARCH_PERFMON_FIXED_CTR_CTRL , cpuc->fixed_ctrl_val); cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi) } } +static void __intel_pmu_enable_all (bool pmi) +{ +intel_pmu_lbr_enable_all (pmi); +___intel_pmu_enable_all (); +} +static void intel_pmu_enable_all (int added) {static_call_cond (x86_pmu_pebs_enable_all)(); -__intel_pmu_enable_all (added, false); +__intel_pmu_enable_all (false); }static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled;if (pmu_enabled) -__intel_pmu_enable_all (0 , true); +__intel_pmu_enable_all (true);intel_bts_enable_local ();/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 07/24] perf/x86/intel: Add helper to enable PMU without LBR
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
Currently,
__intel_pmu_enable_all()also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper,___intel_pmu_enable_all(), so the PMU can be enabled independently. This allows the branch snapshot path to re-enable the PMU without touching LBR.Also remove the unused added argument from
__intel_pmu_enable_all().No functional change intended.
Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
PATCH v7 8/24 perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/intel/core.c | 39 ++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +35 -4
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added) static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries, -unsigned int cnt,unsigned long flags) +unsigned int cnt,unsigned long flags, +int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -2829,7 +2830,21 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -intel_pmu_enable_all (0 ); + +/* +* This path may be entered from the PMI handler through +* bpf_overflow_handler(), so only re-enable what was enabled +* on entry. On PMU version < 4 the LBR freeze disables LBR, +* so it must stay disabled. On version 4 and later, LBR is +* only frozen and can be re-enabled. +*/ +if (pmu_enabled) { + cpuc->enabled =1 ; +___intel_pmu_enable_all (); + } +if (lbr_enabled) +intel_pmu_lbr_enable_all (false); +local_irq_restore (flags);return cnt; }
@@ -2837,27 +2852,43 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,static int intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled; + cpuc->enabled =0 ;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }static int intel_pmu_snapshot_arch_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled; + cpuc->enabled =0 ;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_arch_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 08/24] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
Sashiko has reviewed this patch and found no issues.
Quoted 5 lines
__intel_pmu_snapshot_branch_stack()currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Quoted 7 lines
Only restore the PMU and LBR on the BPF path if they were originally enabled.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v7 9/24 perf/x86/intel: Keep LBR frozen throughout PMI handling
In reply to Dapeng MiOn PMU version 4 and later, the streamlined freeze sets PERF_GLOBAL_STATUS.LBR_FRZ to freeze LBR when a counter overflows. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
--- arch/x86/events/intel/core.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +11 -2
@@ -4063,7 +4063,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again:intel_pmu_lbr_read (); -intel_pmu_ack_status (status); + +/* +* Keep the LBR state frozen throughout PMI handling so that the PMI +* processing itself does not pollute the LBR state. +*/ +intel_pmu_ack_status (status & ~GLOBAL_STATUS_LBRS_FROZEN ); +if (++loops >100 ) {static bool warned;
@@ -4082,10 +4088,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* Repeat if there is more work to be done: */ status =intel_pmu_get_status (); -if (status) +if (status & ~GLOBAL_STATUS_LBRS_FROZEN ) goto again; done: +if (status &GLOBAL_STATUS_LBRS_FROZEN ) +intel_pmu_ack_status (status &GLOBAL_STATUS_LBRS_FROZEN ); +if (mid_ack)apic_write (APIC_LVTPC ,APIC_DM_NMI );/* Only restore PMU state when it's active. See x86_pmu_disable(). */
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 09/24] perf/x86/intel: Keep LBR frozen throughout PMI handling
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
On PMU version 4 and later, the streamlined freeze sets PERF_GLOBAL_STATUS.LBR_FRZ to freeze LBR when a counter overflows. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when
bpf_get_branch_snapshot()is invoked frombpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Quoted 8 lines
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
PATCH v7 10/24 perf: Add counted-bounds tracking for perf_branch_stack
In reply to Dapeng Mistruct perf_branch_stack uses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member in cpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.
Annotate entries[] with __counted_by(nr) so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR/BHRB stacks to raw backing arrays and add accessors that return the struct perf_branch_stack view.
Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
--- arch/powerpc/perf/core-book3s.c | 47 +++++++++++------ arch/x86/events/amd/brs.c | 7 +-- arch/x86/events/amd/core.c | 10 ++-- arch/x86/events/amd/lbr.c | 44 +++++++++------- arch/x86/events/core.c | 2 +- arch/x86/events/intel/core.c | 5 +- arch/x86/events/intel/ds.c | 12 +++-- arch/x86/events/intel/lbr.c | 89 +++++++++++++++++---------------- arch/x86/events/perf_event.h | 19 ++++++- drivers/perf/arm_brbe.c | 5 ++ include/linux/perf_event.h | 2 +- 11 files changed, 149 insertions(+), 93 deletions(-)
arch/powerpc/perf/core-book3s.c +32 -15
@@ -30,6 +30,8 @@#define BHRB_PREDICTION 0x0000000000000001#define BHRB_EA 0xFFFFFFFFFFFFFFFCUL +#define BHRB_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES) +struct cpu_hw_events {int n_events;int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {u64 bhrb_filter;/* BHRB HW branch filter */ unsigned int bhrb_users;void *bhrb_context; -struct perf_branch_stack bhrb_stack; -struct perf_branch_entry bhrb_entries[BHRB_MAX_ENTRIES ]; +u64 __bhrb_stack[BHRB_STACK_SIZE /sizeof (u64)];u64 ic_init;/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)return ((TRAP (regs) ==INTERRUPT_PERFMON ) && regs->result); } +static __always_inline struct perf_branch_stack *cpu_bhrb_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__bhrb_stack; +} + +static __always_inline void cpu_init_bhrb_stack(struct cpu_hw_events *cpuc, int nr) +{ +/* +* Initialize the branch-stack count before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ +cpu_bhrb_stack (cpuc)->nr = nr; +} +/* * Things that are specific to 64-bit implementations. */
@@ -487,12 +502,15 @@ static __u64 power_pmu_bhrb_to(u64 addr)/* Processing BHRB entries */ static void power_pmu_bhrb_read (struct perf_event *event,struct cpu_hw_events *cpuhw) { +struct perf_branch_stack *bhrb =cpu_bhrb_stack (cpuhw); +int r_index, u_index, pred;u64 val;u64 addr; -int r_index, u_index, pred; r_index =0 ; u_index =0 ; + +cpu_init_bhrb_stack (cpuhw, ppmu->bhrb_nr);while (r_index < ppmu->bhrb_nr) {/* Assembly read function */ val =read_bhrb (r_index++);
@@ -541,9 +559,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events */* Target branches use two entries * (ie. computed gotos/XL form) */ - cpuhw->bhrb_entries[u_index].to = addr; - cpuhw->bhrb_entries[u_index].mispred = pred; - cpuhw->bhrb_entries[u_index].predicted = ~pred; + bhrb->entries[u_index].to = addr; + bhrb->entries[u_index].mispred = pred; + bhrb->entries[u_index].predicted = ~pred;/* Get from address in next entry */ val =read_bhrb (r_index++);
@@ -554,22 +572,21 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events * r_index--; addr =0 ; } - cpuhw->bhrb_entries[u_index].from = addr; + bhrb->entries[u_index].from = addr; }else {/* Branches to immediate field (ie I or B form) */ - cpuhw->bhrb_entries[u_index].from = addr; - cpuhw->bhrb_entries[u_index].to = -power_pmu_bhrb_to (addr); - cpuhw->bhrb_entries[u_index].mispred = pred; - cpuhw->bhrb_entries[u_index].predicted = ~pred; + bhrb->entries[u_index].from = addr; + bhrb->entries[u_index].to =power_pmu_bhrb_to (addr); + bhrb->entries[u_index].mispred = pred; + bhrb->entries[u_index].predicted = ~pred; } u_index++; } } - cpuhw->bhrb_stack.nr = u_index; - cpuhw->bhrb_stack.hw_idx =-1ULL ; + bhrb->nr = u_index; + bhrb->hw_idx =-1ULL ;return ; }
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,struct cpu_hw_events *cpuhw; cpuhw =this_cpu_ptr (&cpu_hw_events);power_pmu_bhrb_read (event, cpuhw); -perf_sample_save_brstack (&data, event, &cpuhw->bhrb_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_bhrb_stack (cpuhw),NULL ); } if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
arch/x86/events/amd/brs.c +4 -3
@@ -282,7 +282,6 @@ void amd_brs_drain(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_event *event = cpuc->events[0 ]; -struct perf_branch_entry *br = cpuc->lbr_entries;union amd_debug_extn_cfg cfg;u32 i, nr =0 , num, tos, start;u32 shift =64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void) num = tos - start +1 ; +cpu_init_lbr_stack (cpuc, num); +/* * BRS is only one pass (saturation) from MSROFF to depth-1 * MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; - br[nr] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[nr] = (struct perf_branch_entry ){ .from = from, .to = to, };
@@ -351,7 +352,7 @@ void amd_brs_drain(void) } empty:/* Record number of sampled branches */ - cpuc->lbr_stack.nr = nr; +cpu_lbr_stack (cpuc)->nr = nr; }/*
arch/x86/events/amd/core.c +6 -4
@@ -928,7 +928,8 @@ static int amd_pmu_handle_irq(struct pt_regs *regs)* details. It's up to caller to provide enough space in *entries* to fit all * LBR records, otherwise returned result will be truncated to *cnt* entries. */ -static int amd_pmu_v2_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) +static int amd_pmu_v2_snapshot_branch_stack (struct perf_branch_entry *entries, +unsigned int cnt) {struct cpu_hw_events *cpuc;unsigned long flags;
@@ -948,8 +949,9 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u__amd_pmu_lbr_disable ();amd_pmu_lbr_read (); - cnt =min (cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); + cnt =min (cnt,cpu_lbr_stack (cpuc)->nr); +memcpy (entries,cpu_lbr_stack (cpuc)->entries, +sizeof (struct perf_branch_entry ) * cnt);if (pmu_enabled) { cpuc->enabled =1 ;
@@ -1023,7 +1025,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)if (!x86_perf_event_set_period (event))continue ; -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/amd/lbr.c +26 -18
113 lines not on this page · raw message
arch/x86/events/core.c +1 -1
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)perf_sample_data_init (&data,0 , last_period); -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/intel/core.c +3 -2
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); + cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); +memcpy (entries,cpu_lbr_stack (cpuc)->entries, +sizeof (struct perf_branch_entry ) * cnt);/* * This path may be entered from the PMI handler through
arch/x86/events/intel/ds.c +8 -4
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)static int intel_pmu_pebs_fixup_ip (struct pt_regs *regs) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -unsigned long from = cpuc->lbr_entries[0 ].from; -unsigned long old_to, to = cpuc->lbr_entries[0 ].to; +unsigned long from, old_to, to;unsigned long ip = regs->ip;int is_64bit =0 ;void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)/* * No LBR entry, no basic block, no rewinding */ -if (!cpuc->lbr_stack.nr || !from || !to) +if (!cpu_lbr_stack (cpuc)->nr) +return 0 ; + + from =cpu_lbr_stack (cpuc)->entries[0 ].from; + to =cpu_lbr_stack (cpuc)->entries[0 ].to; +if (!from || !to)return 0 ;/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,if (x86_pmu.intel_cap.pebs_format >=3 )setup_pebs_time (event, data, pebs->tsc); -perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); } static void adaptive_pebs_save_regs(struct pt_regs *regs,
arch/x86/events/intel/lbr.c +46 -43
255 lines not on this page · raw message
arch/x86/events/perf_event.h +17 -2
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {struct x86_perf_task_context ;#define MAX_LBR_ENTRIES 32 +#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES) +enum {LBR_FORMAT_32 =0x00 ,LBR_FORMAT_LIP =0x01 ,
@@ -525,8 +527,7 @@ struct cpu_hw_events {*/ int lbr_users;int lbr_pebs_users; -struct perf_branch_stack lbr_stack; -struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ]; +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 lbr_counters[MAX_LBR_ENTRIES ];/* branch stack extra */ union {struct er_account *lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {struct pmu *pmu; }; +static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__lbr_stack; +} + +static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr) +{ +/* +* Initialize the branch-stack count before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ +cpu_lbr_stack (cpuc)->nr = nr; +} +#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \ { .idxmsk64 = (n) }, \ .code = (c), \
drivers/perf/arm_brbe.c +5 -0
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,prepare_event_branch_type_mask (branch_sample_type, event_type_mask); +/* +* Initialize the branch_stack->nr before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ + branch_stack->nr = nr_hw;for (int bank =0 ; bank < nr_banks; bank++) {int nr_remaining = nr_hw - (bank *BRBE_BANK_MAX_ENTRIES );int nr_this_bank =min (nr_remaining,BRBE_BANK_MAX_ENTRIES );
include/linux/perf_event.h +1 -1
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)struct perf_branch_stack {u64 nr;u64 hw_idx; -struct perf_branch_entry entries[]; + struct perf_branch_entry entries[] __counted_by(nr); };struct task_struct ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 10/24] perf: Add counted-bounds tracking for perf_branch_stack
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
struct
perf_branch_stackuses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member incpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.Annotate entries[] with
__counted_by(nr)so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR/BHRB stacks to raw backing arrays and add accessors that return the structperf_branch_stackview.
Quoted 7 lines
Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
PATCH v7 11/24 perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
In reply to Dapeng MiThe LBR read helpers currently assume the default stack, cpuc->__lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the *_lbr_read() helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Do the same for intel_pmu_store_lbr() and intel_pmu_lbr_filter(), and pass the LBR counters array explicitly as well, since it is filtered together with the stack.
No functional change intended.
--- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/core.c | 4 +- arch/x86/events/intel/lbr.c | 89 ++++++++++++++++++++---------------- arch/x86/events/perf_event.h | 16 ++++--- 5 files changed, 64 insertions(+), 49 deletions(-)
arch/x86/events/amd/brs.c +1 -1
@@ -316,7 +316,7 @@ void amd_brs_drain(void) num = tos - start +1 ; -cpu_init_lbr_stack (cpuc, num); +cpu_init_lbr_stack (cpu_lbr_stack (cpuc), num);/* * BRS is only one pass (saturation) from MSROFF to depth-1
arch/x86/events/amd/lbr.c +1 -1
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)return ; } -cpu_init_lbr_stack (cpuc, x86_pmu.lbr_nr); +cpu_init_lbr_stack (cpu_lbr_stack (cpuc), x86_pmu.lbr_nr);for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
arch/x86/events/intel/core.c +2 -2
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -intel_pmu_lbr_read (); +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4063,7 +4063,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (); +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters);/* * Keep the LBR state frozen throughout PMI handling so that the PMI
arch/x86/events/intel/lbr.c +50 -39
234 lines not on this page · raw message
arch/x86/events/perf_event.h +10 -6
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_evereturn (struct perf_branch_stack *)cpuc->__lbr_stack; } -static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr) +static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr) {/* * Initialize the branch-stack count before populating entries[] * to avoid UBSAN complaints from the __counted_by(nr) flexible array. */ -cpu_lbr_stack (cpuc)->nr = nr; + lbr_stack->nr = nr; }#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \
@@ -1207,7 +1207,8 @@ struct x86_pmu { unsigned int lbr_counters:4;void (*lbr_reset)(void ); -void (*lbr_read)(struct cpu_hw_events *cpuc); +void (*lbr_read)(struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack);void (*lbr_save)(void *ctx);void (*lbr_restore)(void *ctx);
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);void intel_pmu_lbr_disable_all (void ); -void intel_pmu_lbr_read (void ); +void intel_pmu_lbr_read (struct perf_branch_stack *lbr_stack, +u64 *lbr_counters); -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack); -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack);void intel_pmu_lbr_save (void *ctx);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 11/24] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Sashiko has reviewed this patch and found no issues.
Quoted 6 lines
The LBR read helpers currently assume the default stack,
cpuc->__lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the*_lbr_read()helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Quoted 7 lines
Do the same for
intel_pmu_store_lbr()andintel_pmu_lbr_filter(), and pass the LBR counters array explicitly as well, since it is filtered together with the stack.No functional change intended.
Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
PATCH v7 12/24 perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
In reply to Dapeng MiBPF LBR snapshotting currently reuses the default LBR stack (cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context via bpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
--- arch/x86/events/intel/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +12 -3
@@ -2819,17 +2819,26 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (false); } +struct lbr_snapshot { +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags,int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot); +struct perf_branch_stack *lbr_stack; -intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); - cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); + lbr_stack = (struct perf_branch_stack *)bpf->__lbr_stack; +intel_pmu_lbr_read (lbr_stack, bpf->counters); + cnt = min_t(unsigned int, cnt, lbr_stack->nr); -memcpy (entries,cpu_lbr_stack (cpuc)->entries, +memcpy (entries, lbr_stack->entries,sizeof (struct perf_branch_entry ) * cnt);/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 12/24] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
Sashiko has reviewed this patch and found no issues.
Quoted 8 lines
BPF LBR snapshotting currently reuses the default LBR stack (
cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context viabpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Quoted 6 lines
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
PATCH v7 13/24 perf/x86/intel: Always snapshot live hardware LBR state for BPF
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. Move the lbr_pebs_users check into the PMI handler, which is the only caller that relies on it, so the BPF path always snapshots the live hardware LBR state.
--- arch/x86/events/intel/core.c | 7 ++++++- arch/x86/events/intel/lbr.c | 9 --------- 2 files changed, 6 insertions(+), 10 deletions(-)
arch/x86/events/intel/core.c +6 -1
@@ -4072,7 +4072,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters);/* * Keep the LBR state frozen throughout PMI handling so that the PMI
arch/x86/events/intel/lbr.c +0 -9
@@ -1042,15 +1042,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc, lbr_stack);intel_pmu_lbr_filter (cpuc, lbr_stack, lbr_counters);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 13/24] perf/x86/intel: Always snapshot live hardware LBR state for BPF
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
__intel_pmu_snapshot_branch_stack()is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.
Quoted 8 lines
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. Move the
lbr_pebs_userscheck into the PMI handler, which is the only caller that relies on it, so the BPF path always snapshots the live hardware LBR state.Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
PATCH v7 14/24 perf/x86/intel: Account lbr_pebs_users for arch-PEBS events
In reply to Dapeng Milbr_pebs_users tracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.
Add the missing lbr_pebs_users accounting for arch-PEBS as well.
--- arch/x86/events/intel/lbr.c | 4 ++-- arch/x86/events/perf_event.h | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-)
arch/x86/events/intel/lbr.c +2 -2
@@ -616,7 +616,7 @@ void intel_pmu_lbr_add(struct perf_event *event)* be 'new'. Conversely, a new event can get installed through the * context switch path for the first time. */ -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++;perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)
@@ -687,7 +687,7 @@ void intel_pmu_lbr_del(struct perf_event *event)if (event->hw.flags &PERF_X86_EVENT_LBR_SELECT ) cpuc->lbr_select =0 ; -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users--; cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 );
arch/x86/events/perf_event.h +7 -0
@@ -2096,6 +2096,13 @@ static inline bool intel_pmu_has_pebs(void) return x86_pmu.ds_pebs || x86_pmu.arch_pebs; } +static inline bool intel_pmu_has_pebs_lbr (struct pmu *pmu) +{ +if (x86_pmu.arch_pebs) +return hybrid (pmu, arch_pebs_cap).caps &ARCH_PEBS_LBR ; +return x86_pmu.intel_cap.pebs_baseline; +} +#else /* CONFIG_CPU_SUP_INTEL */ static inline void reserve_ds_buffers (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
lbr_pebs_userstracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.Add the missing
lbr_pebs_usersaccounting for arch-PEBS as well.Fixes d243d0bb64af ("perf/x86/intel: Initialize architectural PEBS")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
[email protected]
AI reviewRe: [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events
In reply to Dapeng MiTL;DR: - On arch-PEBS, a non-PEBS branch-counter sibling event may report a stale branch stack with zero counters until a later patch fixes it; consider reordering - Fixes tag may point to the wrong…
Quoted 1 line
perf/x86/intel: Account
lbr_pebs_usersfor arch-PEBS events
TL;DR:
- On arch-PEBS, a non-PEBS branch-counter sibling event may report a stale branch stack with zero counters until a later patch fixes it; consider reordering
- Fixes tag may point to the wrong commit
Quoted 11 lines
perf/x86/intel: Account
lbr_pebs_usersfor arch-PEBS events
lbr_pebs_userstracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.Add the missing
lbr_pebs_usersaccounting for arch-PEBS as well.Fixes d243d0bb64af ("perf/x86/intel: Initialize architectural PEBS")[ ... ]Signed-off-by Dapeng Mi <[email protected]>
d243d0bb64af only parses the arch-PEBS CPUID leaves and adds the enable/disable callbacks, it does not make arch-PEBS use LBR. The ARCH_PEBS_LBR handling and PERF_SAMPLE_BRANCH_STACK in large_pebs_flags came with 52448a0a7390, which predates this series.
Should the tag point there instead:
Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0
[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_add(struct perf_event *event) · line 622
Quoted 9 lines
arch/x86/events/intel/lbr.c +1
-1@@ -616,7 +616,7 @@ void intel_pmu_lbr_add(struct perf_event *event)* be 'new'. Conversely, a new event can get installed through the * context switch path for the first time. */ -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++;perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)
Can this make a non-PEBS branch counter event report a stale branch stack on arch-PEBS platforms?
With lbr_pebs_users now non-zero on arch-PEBS, the existing shortcut in intel_pmu_handle_irq() is taken:
if (cpuc->lbr_users != cpuc->lbr_pebs_users)
intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);That shortcut assumes every consumer of the LBR stack is counted in lbr_users. An event whose branch_sample_type is only PERF_SAMPLE_BRANCH_COUNTERS plus PLM bits has PERF_X86_EVENT_NEEDS_BRANCH_STACK cleared in intel_pmu_hw_config(), so intel_pmu_add_event() never calls intel_pmu_lbr_add() for it and it is not counted in lbr_users. It still has has_branch_stack() set and samples through the non-PEBS path in handle_pmi_common():
if (has_branch_stack(event))
intel_pmu_lbr_save_brstack(&data, cpuc, event);Consider a group on an arch-PEBS CPU with ARCH_PEBS_LBR and PMU_FL_BR_CNTR, with a precise leader using branch_type=any,counter and a non-precise sampling sibling using only branch_type=counter. After this patch lbr_users == 1 and lbr_pebs_users == 1. If only the sibling overflows, intel_pmu_lbr_read() is skipped, and intel_pmu_lbr_save_brstack() -> intel_pmu_lbr_counters_reorder() works on whatever cpu_lbr_stack(cpuc) last held. That is either nr == 0 after intel_pmu_lbr_reset(), or the leader's last PEBS-record snapshot from intel_pmu_store_pebs_lbrs(), whose raw counters in entries[i].reserved were already zeroed when the leader's sample was saved.
So the sibling would report a stale branch stack with all branch counters zero, where before this patch it got the live LBR state. The same problem already exists for legacy adaptive PEBS (pebs_baseline), and this patch extends it to arch-PEBS.
This looks to be addressed later in the series by 0479605ddd39 ("perf/x86/intel: Account LBR user counts for pure branch counter events"), and 9edb70d448eb ("perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed") replaces the lbr_users != lbr_pebs_users check at PMI entry with an on-demand re-read. Would it make sense to order 0479605ddd39 before this patch so the intermediate state does not regress?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
Re: [Patch v7 14/24] perf/x86/intel: Account lbr_pebs_users for arch-PEBS events
Quoted 18 lines · On 11/18/2026 12:10 PM, [email protected] wrote:
Quoted 1 line
perf/x86/intel: Account
lbr_pebs_usersfor arch-PEBS eventsTL;DR:
- On arch-PEBS, a non-PEBS branch-counter sibling event may report a stale branch stack with zero counters until a later patch fixes it; consider reordering
- Fixes tag may point to the wrong commitQuoted 11 lines
perf/x86/intel: Account
lbr_pebs_usersfor arch-PEBS events
lbr_pebs_userstracks how many PEBS events are currently using LBR. It is updated for legacy PEBS events, but arch-PEBS events were missing from that accounting.Add the missing
lbr_pebs_usersaccounting for arch-PEBS as well.Fixes d243d0bb64af ("perf/x86/intel: Initialize architectural PEBS")[ ... ]Signed-off-by Dapeng Mi <[email protected]>
Quoted 8 lines
d243d0bb64afonly parses the arch-PEBS CPUID leaves and adds the enable/disable callbacks, it does not make arch-PEBS use LBR. TheARCH_PEBS_LBRhandling andPERF_SAMPLE_BRANCH_STACKinlarge_pebs_flagscame with52448a0a7390, which predates this series.Should the tag point there instead:
Fixes 52448a0a7390 ("perf/x86/intel: Setup PEBS data configuration and enable legacy groups")
Not exactly. The arch-PEBS feature including the arch-PEBS assisted LBR snapshotting is supported by a bunch of patches instead of a single patch, e.g., the commit d243d0bb64af enumerates if the arch-PEBS has the capability to support LBR snapshot, it's a more appropriate and accurate commit comparing with 52448a0a7390.
Quoted 51 lines
Quoted 4 lines
arch/x86/events/intel/lbr.c +0
-0[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_add(struct perf_event *event) · line 622
Quoted 9 lines
arch/x86/events/intel/lbr.c +1
-1@@ -616,7 +616,7 @@ void intel_pmu_lbr_add(struct perf_event *event)* be 'new'. Conversely, a new event can get installed through the * context switch path for the first time. */ -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip >0 ) cpuc->lbr_pebs_users++;perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)Can this make a non-PEBS branch counter event report a stale branch stack on arch-PEBS platforms?
With
lbr_pebs_usersnow non-zero on arch-PEBS, the existing shortcut inintel_pmu_handle_irq()is taken:if (cpuc->lbr_users != cpuc->lbr_pebs_users) intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);That shortcut assumes every consumer of the LBR stack is counted in
lbr_users. An event whosebranch_sample_typeis onlyPERF_SAMPLE_BRANCH_COUNTERSplus PLM bits hasPERF_X86_EVENT_NEEDS_BRANCH_STACKcleared inintel_pmu_hw_config(), sointel_pmu_add_event()never callsintel_pmu_lbr_add()for it and it is not counted inlbr_users. It still hashas_branch_stack()set and samples through the non-PEBS path inhandle_pmi_common():if (has_branch_stack(event)) intel_pmu_lbr_save_brstack(&data, cpuc, event);Consider a group on an arch-PEBS CPU with
ARCH_PEBS_LBRandPMU_FL_BR_CNTR, with a precise leader using branch_type=any,counter and a non-precise sampling sibling using only branch_type=counter. After this patchlbr_users== 1 andlbr_pebs_users== 1. If only the sibling overflows,intel_pmu_lbr_read()is skipped, andintel_pmu_lbr_save_brstack()->intel_pmu_lbr_counters_reorder()works on whatevercpu_lbr_stack(cpuc)last held. That is either nr == 0 afterintel_pmu_lbr_reset(), or the leader's last PEBS-record snapshot fromintel_pmu_store_pebs_lbrs(), whose raw counters in entries[i].reserved were already zeroed when the leader's sample was saved.So the sibling would report a stale branch stack with all branch counters zero, where before this patch it got the live LBR state. The same problem already exists for legacy adaptive PEBS (
pebs_baseline), and this patch extends it to arch-PEBS.
Quoted 7 lines
This looks to be addressed later in the series by
0479605ddd39("perf/x86/intel: Account LBR user counts for pure branch counter events"), and9edb70d448eb("perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed") replaces thelbr_users!=lbr_pebs_userscheck at PMI entry with an on-demand re-read. Would it make sense to order0479605ddd39before this patch so the intermediate state does not regress?
Hmm, I don't think the patch order matters. If we put the commit 9edb70d448eb ahead before this patch, the whole arch-PEBS assisted LBR snapshot won't be counted without this patch regardless of it's a pure branch counter event. It seems worse.
Thanks.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
PATCH v7 15/24 perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
In reply to Dapeng MiA pure branch counter event (i.e. an event which only applies PERF_SAMPLE_BRANCH_COUNTERS without requiring the branch stack itself) doesn't set up the LBR on its own. It relies on being grouped with a normal branch event, which configures the LBR for it. Such an event cannot work standalone.
Set the PERF_EV_CAP_SIBLING flag for pure branch counter events, so that the generic perf core schedules them out once the group leader is removed from the group.
Opportunistically refine the code style.
--- arch/x86/events/intel/core.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +11 -4
@@ -5130,10 +5130,17 @@ static int intel_pmu_hw_config(struct perf_event *event)* require any branch stack setup. * Clear the bit to avoid unnecessary branch stack setup. */ -if (0 == (event->attr.branch_sample_type & - ~(PERF_SAMPLE_BRANCH_PLM_ALL | -PERF_SAMPLE_BRANCH_COUNTERS ))) - event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +if (!(event->attr.branch_sample_type & + ~(PERF_SAMPLE_BRANCH_PLM_ALL |PERF_SAMPLE_BRANCH_COUNTERS ))) { + event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +/* +* A pure branch counter event doesn't set up the LBR +* itself. It must stay grouped with a normal branch +* event which configures the LBR for it. Otherwise, +* the LBR cannot be configured correctly. +*/ + event->event_caps |=PERF_EV_CAP_SIBLING ; + }/* * Force the leader to be a LBR event. So LBRs can be reset
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] perf/x86/intel: incomplete group detach handling in intel_pmu_hw_config() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: incomplete group detach handling in intel_pmu_hw_config()
--
commit 2f6a2a62d0ff0bf2794f702072c776c06e629ff3
Author: Dapeng Mi <[email protected]>
perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
This commit sets PERF_EV_CAP_SIBLING on pure branch counter events in intel_pmu_hw_config() so that the perf core schedules them out if the group leader is detached.
Quoted 16 lines
arch/x86/events/intel/core.c +4
-4@@ -5130,10 +5130,17 @@ static int intel_pmu_hw_config(struct perf_event *event)* require any branch stack setup. * Clear the bit to avoid unnecessary branch stack setup. */ -if (0 == (event->attr.branch_sample_type & - ~(PERF_SAMPLE_BRANCH_PLM_ALL | -PERF_SAMPLE_BRANCH_COUNTERS ))) - event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +if (!(event->attr.branch_sample_type & + ~(PERF_SAMPLE_BRANCH_PLM_ALL |PERF_SAMPLE_BRANCH_COUNTERS ))) { + event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +/*
arch/x86/events/intel/core.c static int intel_pmu_hw_config(struct perf_event *event) · line 5146
Quoted 10 lines
arch/x86/events/intel/core.c +7
-0+* A pure branch counter event doesn't set up the LBR +* itself. It must stay grouped with a normal branch +* event which configures the LBR for it. Otherwise, +* the LBR cannot be configured correctly. +*/ + event->event_caps |=PERF_EV_CAP_SIBLING ; + }/* * Force the leader to be a LBR event. So LBRs can be reset
[Severity: Medium]
What happens on group detach when a branch counter sibling has other branch_sample_type filter bits set alongside PERF_SAMPLE_BRANCH_COUNTERS?
First, a counting sibling in PERF_SAMPLE_READ mode with a branch filter (such as PERF_SAMPLE_BRANCH_ANY | PERF_SAMPLE_BRANCH_COUNTERS, which "perf record -e '{e1,e2}:S' -j any,counter" configures) skips setting PERF_X86_EVENT_NEEDS_BRANCH_STACK earlier in intel_pmu_hw_config():
arch/x86/events/intel/core.c:intel_pmu_hw_config() {
if (needs_branch_stack(event)) {
/* Avoid branch stack setup for counting events in SAMPLE READ */
if (is_sampling_event(event) ||
!(event->attr.sample_type & PERF_SAMPLE_READ))
event->hw.flags |= PERF_X86_EVENT_NEEDS_BRANCH_STACK;
}
}Because branch_sample_type contains PERF_SAMPLE_BRANCH_ANY, the condition here is false and PERF_EV_CAP_SIBLING is not set either. If the group leader is closed, perf_group_detach() promotes this sibling to a standalone leader that stays scheduled with ARCH_PERFMON_EVENTSEL_BR_CNTR enabled in hardware without LBR configured.
Second, for a sampling sibling that retains PERF_X86_EVENT_NEEDS_BRANCH_STACK and has PERF_SAMPLE_BRANCH_COUNTERS set, intel_pmu_hw_config() only sets PERF_X86_EVENT_BRANCH_COUNTERS on leader->hw.flags and not on event->hw.flags. When the original leader is closed and perf_promote_sibling_to_leader() sets sibling->group_leader = sibling, is_branch_counters_group(sibling) becomes false, so intel_pmu_lbr_save_brstack() skips intel_pmu_lbr_counters_reorder() and intel_pmu_lbr_del() skips intel_pmu_lbr_reset().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Re: [Patch v7 15/24] perf/x86/intel: Set PERF_EV_CAP_SIBLING for pure branch counter events
Quoted 59 lines · On 11/18/2026 12:05 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: incomplete group detach handling inintel_pmu_hw_config()
--commit
2f6a2a62d0ff0bf2794f702072c776c06e629ff3
Author: Dapeng Mi <[email protected]>perf/x86/intel: Set
PERF_EV_CAP_SIBLINGfor pure branch counter eventsThis commit sets
PERF_EV_CAP_SIBLINGon pure branch counter events inintel_pmu_hw_config()so that the perf core schedules them out if the group leader is detached.Quoted 16 lines
arch/x86/events/intel/core.c +4
-4@@ -5130,10 +5130,17 @@ static int intel_pmu_hw_config(struct perf_event *event)* require any branch stack setup. * Clear the bit to avoid unnecessary branch stack setup. */ -if (0 == (event->attr.branch_sample_type & - ~(PERF_SAMPLE_BRANCH_PLM_ALL | -PERF_SAMPLE_BRANCH_COUNTERS ))) - event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +if (!(event->attr.branch_sample_type & + ~(PERF_SAMPLE_BRANCH_PLM_ALL |PERF_SAMPLE_BRANCH_COUNTERS ))) { + event->hw.flags &= ~PERF_X86_EVENT_NEEDS_BRANCH_STACK ; +/* arch/x86/events/intel/core.c static int intel_pmu_hw_config(struct perf_event *event) · line 5146
Quoted 10 lines
arch/x86/events/intel/core.c +7
-0+* A pure branch counter event doesn't set up the LBR +* itself. It must stay grouped with a normal branch +* event which configures the LBR for it. Otherwise, +* the LBR cannot be configured correctly. +*/ + event->event_caps |=PERF_EV_CAP_SIBLING ; + }/* * Force the leader to be a LBR event. So LBRs can be reset [Severity: Medium]
What happens on group detach when a branch counter sibling has otherbranch_sample_typefilter bits set alongsidePERF_SAMPLE_BRANCH_COUNTERS?First, a counting sibling in
PERF_SAMPLE_READmode with a branch filter (such asPERF_SAMPLE_BRANCH_ANY|PERF_SAMPLE_BRANCH_COUNTERS, which "perf record -e '{e1,e2}:S' -j any,counter" configures) skips settingPERF_X86_EVENT_NEEDS_BRANCH_STACKearlier inintel_pmu_hw_config():arch/x86/events/intel/core.c:intel_pmu_hw_config() { if (needs_branch_stack(event)) { /* Avoid branch stack setup for counting events in SAMPLE READ */ if (is_sampling_event(event) || !(event->attr.sample_type & PERF_SAMPLE_READ)) event->hw.flags |= PERF_X86_EVENT_NEEDS_BRANCH_STACK; } }
Quoted 5 lines
Because
branch_sample_typecontainsPERF_SAMPLE_BRANCH_ANY, the condition here is false andPERF_EV_CAP_SIBLINGis not set either. If the group leader is closed,perf_group_detach()promotes this sibling to a standalone leader that stays scheduled withARCH_PERFMON_EVENTSEL_BR_CNTRenabled in hardware without LBR configured.
Yeah, it's an issue, but not introduced by this patchset. It needs a separate fix.
Quoted 9 lines
Second, for a sampling sibling that retains
PERF_X86_EVENT_NEEDS_BRANCH_STACKand hasPERF_SAMPLE_BRANCH_COUNTERSset,intel_pmu_hw_config()only setsPERF_X86_EVENT_BRANCH_COUNTERSonleader->hw.flagsand not onevent->hw.flags. When the original leader is closed andperf_promote_sibling_to_leader()setssibling->group_leader= sibling,is_branch_counters_group(sibling)becomes false, sointel_pmu_lbr_save_brstack()skipsintel_pmu_lbr_counters_reorder()andintel_pmu_lbr_del()skipsintel_pmu_lbr_reset().
Ditto.
Quoted 1 line
PATCH v7 16/24 perf/x86/intel: Account LBR user counts for pure branch counter events
In reply to Dapeng Miintel_pmu_hw_config() clears PERF_X86_EVENT_NEEDS_BRANCH_STACK for pure branch counter events. As a result, intel_pmu_add_event() and intel_pmu_del_event() do not account lbr_users and lbr_pebs_users correctly for these events.
Add intel_pmu_account_pure_branch_counter() to adjust the LBR user counts when pure branch counter events are added or removed.
--- arch/x86/events/intel/core.c | 39 ++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/x86/events/intel/core.c +39 -0
@@ -3168,10 +3168,47 @@ static __always_inline bool intel_pmu_needs_branch_stack(struct perf_event *evenreturn event->hw.flags &PERF_X86_EVENT_NEEDS_BRANCH_STACK ; } +static inline bool is_pure_branch_counter (struct perf_event *event) +{ +return is_sampling_event (event) && +has_branch_stack (event) && +branch_sample_counters (event) && + !intel_pmu_needs_branch_stack (event); +} + +/* +* A pure branch counter event clears PERF_X86_EVENT_NEEDS_BRANCH_STACK, +* so intel_pmu_lbr_add()/intel_pmu_lbr_del() do not update lbr_users +* and lbr_pebs_users. Handle the accounting explicitly here instead. +* See intel_pmu_hw_config() for details. +*/ +static void +intel_pmu_account_pure_branch_counter (struct perf_event *event,bool add) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + +if (!x86_pmu.lbr_nr) +return ; + +if (add) { + cpuc->lbr_users++; +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip) + cpuc->lbr_pebs_users++; + }else { + cpuc->lbr_users--; +if (intel_pmu_has_pebs_lbr (event->pmu) && event->attr.precise_ip) + cpuc->lbr_pebs_users--; +WARN_ON_ONCE (cpuc->lbr_users <0 ); +WARN_ON_ONCE (cpuc->lbr_pebs_users <0 ); + } +} +static void intel_pmu_del_event (struct perf_event *event) {if (intel_pmu_needs_branch_stack (event))intel_pmu_lbr_del (event); +if (is_pure_branch_counter (event)) +intel_pmu_account_pure_branch_counter (event, false);if (event->attr.precise_ip)intel_pmu_pebs_del (event); if (is_pebs_counter_event_group(event) ||
@@ -3734,6 +3771,8 @@ static void intel_pmu_add_event(struct perf_event *event)intel_pmu_pebs_add (event);if (intel_pmu_needs_branch_stack (event))intel_pmu_lbr_add (event); +if (is_pure_branch_counter (event)) +intel_pmu_account_pure_branch_counter (event, true);if (is_pebs_counter_event_group (event) ||is_acr_event_group (event))this_cpu_ptr (&cpu_hw_events)->n_late_setup++;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 16/24] perf/x86/intel: Account LBR user counts for pure branch counter events
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
intel_pmu_hw_config()clearsPERF_X86_EVENT_NEEDS_BRANCH_STACKfor pure branch counter events. As a result,intel_pmu_add_event()andintel_pmu_del_event()do not accountlbr_usersandlbr_pebs_userscorrectly for these events.Add
intel_pmu_account_pure_branch_counter()to adjust the LBR user counts when pure branch counter events are added or removed.Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
PATCH v7 17/24 perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
In reply to Dapeng MiPEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.
When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, so userspace can see the wrong branch stack instead of the live LBR state at PMI entry.
Track whether the LBR stack still contains the live state from PMI entry with a cpuc->lbr_stack_live flag. The flag starts out clear on each pass of the PMI handler and is set once the live LBR state has been read. Draining PEBS records with LBR data clears it again, so a non-PEBS LBR event re-reads the stack only when it is needed. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once for multiple non-PEBS LBR events in the same PMI.
--- arch/x86/events/intel/core.c | 18 ++++++++++++++---- arch/x86/events/intel/lbr.c | 1 + arch/x86/events/perf_event.h | 1 + 3 files changed, 16 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +14 -4
@@ -4054,8 +4054,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (!cpuc->lbr_stack_live) { +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), + cpuc->lbr_counters); + cpuc->lbr_stack_live = true; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);
@@ -4111,12 +4117,16 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: + cpuc->lbr_stack_live = false;/* -* No need to read the LBR stack when all LBR users are PEBS events, -* since the PEBS handler updates it. +* If PEBS can't record LBRs, draining PEBS can't overwrite the +* LBR stack, so read it here at PMI entry. Otherwise, it is read +* after PEBS draining when needed. */ -if (cpuc->lbr_users != cpuc->lbr_pebs_users) +if (cpuc->lbr_users >0 && !intel_pmu_has_pebs_lbr (cpuc->pmu)) {intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); + cpuc->lbr_stack_live = true; + }/* * Keep the LBR state frozen throughout PMI handling so that the PMI
arch/x86/events/intel/lbr.c +1 -0
@@ -1317,6 +1317,7 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)intel_pmu_store_lbr (lbr,cpu_lbr_stack (cpuc));intel_pmu_lbr_filter (cpuc,cpu_lbr_stack (cpuc), cpuc->lbr_counters); + cpuc->lbr_stack_live = false; }/*
arch/x86/events/perf_event.h +1 -0
@@ -525,6 +525,7 @@ struct cpu_hw_events {/* * Intel LBR bits */ +bool lbr_stack_live;/* If LBR stack holds the live LBR state */ int lbr_users;int lbr_pebs_users;u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 17/24] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
Sashiko has reviewed this patch and found no issues.
Quoted 18 lines
PEBS and non-PEBS events share the same LBR stack in
cpuc->__lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, so userspace can see the wrong branch stack instead of the live LBR state at PMI entry.
Track whether the LBR stack still contains the live state from PMI entry with a
cpuc->lbr_stack_liveflag. The flag starts out clear on each pass of the PMI handler and is set once the live LBR state has been read. Draining PEBS records with LBR data clears it again, so a non-PEBS LBR event re-reads the stack only when it is needed. Since the LBR state is frozen for the duration of PMI handling, it only needs to be read once for multiple non-PEBS LBR events in the same PMI.
Quoted 2 lines
Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
PATCH v7 18/24 perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler
In reply to Dapeng MiWhen a regular counter which is read by a PEBS counter-event group overflows, handle_pmi_common() drains the pending PEBS records to ensure the stale counter snapshots are processed before handling the counter's overflow. Currently this is done by invoking the x86_pmu_drain_pebs() static call with a non-NULL regs argument.
This is incorrect. At this point the PEBS buffer has not reached its interrupt threshold yet (a real PEBS buffer overflow would have already been handled earlier), and the PMI being serviced belongs to the regular counter, not to the PEBS events. With a non-NULL regs, the drained records are processed with perf_event_overflow(), which charges a PMI to the PEBS events via __perf_event_account_interrupt(). These spurious interrupt counts may falsely trigger the PMI throttling and unexpectedly disable the PEBS events.
Call intel_pmu_drain_pebs_buffer() instead. It passes a NULL regs to x86_pmu_drain_pebs(), so the records are emitted with perf_event_output(), which skips the overflow handling and throttle accounting.
--- arch/x86/events/intel/core.c | 2 +- arch/x86/events/intel/ds.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
@@ -4045,7 +4045,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* The PEBS buffer has to be drained before handling the A-PMI */ if (is_pebs_counter_event_group (event)) -static_call (x86_pmu_drain_pebs)(regs, &data); +intel_pmu_drain_pebs_buffer (); last_period = event->hw.last_period;
arch/x86/events/intel/ds.c +8 -0
@@ -1262,6 +1262,14 @@ int intel_pmu_drain_bts_buffer(void)return 1 ; } +/* +* Drain the pending PEBS buffer for the current CPU. +* This helper should only be used to drain a PEBS buffer which +* hasn't overflowed yet, e.g. on a context switch with large PEBS, +* because the NULL regs argument passed to x86_pmu_drain_pebs() +* makes the records be processed with perf_event_output() rather +* than perf_event_overflow(). +*/ void intel_pmu_drain_pebs_buffer (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 18/24] perf/x86/intel: Fix draining of un-overflowed PEBS buffer in PMI handler
Sashiko has reviewed this patch and found no issues.
Quoted 15 lines
When a regular counter which is read by a PEBS counter-event group overflows,
handle_pmi_common()drains the pending PEBS records to ensure the stale counter snapshots are processed before handling the counter's overflow. Currently this is done by invoking thex86_pmu_drain_pebs()static call with a non-NULL regs argument.This is incorrect. At this point the PEBS buffer has not reached its interrupt threshold yet (a real PEBS buffer overflow would have already been handled earlier), and the PMI being serviced belongs to the regular counter, not to the PEBS events. With a non-NULL regs, the drained records are processed with
perf_event_overflow(), which charges a PMI to the PEBS events via__perf_event_account_interrupt(). These spurious interrupt counts may falsely trigger the PMI throttling and unexpectedly disable the PEBS events.
Quoted 8 lines
Call
intel_pmu_drain_pebs_buffer()instead. It passes a NULL regs tox86_pmu_drain_pebs(), so the records are emitted withperf_event_output(), which skips the overflow handling and throttle accounting.Fixes e02e9b0374c3 ("perf/x86/intel: Support PEBS counters snapshotting")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
PATCH v7 19/24 perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users
In reply to Dapeng MiIf an event needs a branch stack while PEBS LBR users are active, the PEBS buffer may get drained before the event's sample is output. The drain overwrites the live LBR snapshot read by the PMI handler, so the stale PEBS LBR stack is reported to userspace instead.
This can happen, for example, when:
- The event belongs to a SAMPLE_READ group that includes a PEBS event snapshotting the LBR stack, and a later perf_event_overflow() calls perf_output_read_group(), which drains the PEBS buffer via intel_pmu_read_event();
- The overflow triggers group throttling, disabling a PEBS LBR event and causing intel_pmu_disable_event() to drain the PEBS buffer before the live LBR snapshot is output.
Thus, drain the PEBS buffer whenever the overflowed event needs a branch stack and PEBS LBR users are active.
--- arch/x86/events/intel/core.c | 54 ++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 15 deletions(-)
arch/x86/events/intel/core.c +39 -15
@@ -4027,24 +4027,48 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)continue ;/* -* There may be unprocessed PEBS records in the PEBS buffer, -* which still stores the previous values. -* Process those records first before handling the latest value. -* For example, -* A is a regular counter -* B is a PEBS event which reads A -* C is a PEBS event +* Drain any pending PEBS records before handling the overflow +* for the event in two cases: * -* The following can happen: -* B-assist A=1 -* C A=2 -* B-assist A=3 -* A-overflow-PMI A=4 -* C-assist-PMI (PEBS buffer) A=5 +* 1) The event belongs to a PEBS counter event group. Other +* PEBS events may record the previous count for this event. +* If those buffered PEBS records are processed only after +* intel_pmu_save_and_restart() updates the count, the counter +* value will be corrupted. * -* The PEBS buffer has to be drained before handling the A-PMI +* Example: +* A = regular counter +* B = PEBS event that reads A +* C = PEBS event +* +* Sequence: +* B-assist A=1 +* C A=2 +* B-assist A=3 +* A-overflow-PMI A=4 +* C-assist-PMI (PEBS) A=5 +* +* Therefore, the PEBS buffer must be drained before handling +* the A overflow. +* +* 2) The event needs a branch stack, but active PEBS LBR users +* may drain the PEBS buffer and overwrite the live LBR +* snapshot updated by the PMI handler. This can happen, +* for example, when: +* +* - the event belongs to a SAMPLE_READ group that includes +* a PEBS event which snapshots the LBR stack; a later +* perf_event_overflow() may call perf_output_read_group(), +* which drains the PEBS buffer via intel_pmu_read_event(); +* - the overflow triggers group throttling, disabling a PEBS +* LBR event and causing intel_pmu_disable_event() to drain +* the PEBS buffer before the live LBR snapshot is output. +* +* Therefore, drain the PEBS buffer whenever the event needs +* a branch stack and PEBS LBR users are active. */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event) || + (has_branch_stack (event) && cpuc->lbr_pebs_users >0 ))intel_pmu_drain_pebs_buffer (); last_period = event->hw.last_period;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 19/24] perf/x86/intel: Drain PEBS buffer for branch events with PEBS LBR users
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
If an event needs a branch stack while PEBS LBR users are active, the PEBS buffer may get drained before the event's sample is output. The drain overwrites the live LBR snapshot read by the PMI handler, so the stale PEBS LBR stack is reported to userspace instead.
This can happen, for example, when:
- The event belongs to aSAMPLE_READgroup that includes a PEBS event snapshotting the LBR stack, and a laterperf_event_overflow()callsperf_output_read_group(), which drains the PEBS buffer viaintel_pmu_read_event();
- The overflow triggers group throttling, disabling a PEBS LBR event and causingintel_pmu_disable_event()to drain the PEBS buffer before the live LBR snapshot is output.
Quoted 6 lines
Thus, drain the PEBS buffer whenever the overflowed event needs a branch stack and PEBS LBR users are active.
Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
PATCH v7 20/24 perf/x86/intel: Clear raw LBR counters in the LBR stack
In reply to Dapeng MiThe raw branch counters are stored in the reserved field of each LBR entry. The branch counters path clears that field after reordering the counters, but the BPF snapshot path and the regular branch stack path leave the raw values in place, so they leak to userspace.
Clear the raw counters on those paths too.
--- arch/x86/events/intel/core.c | 1 + arch/x86/events/intel/lbr.c | 13 +++++++++++++ arch/x86/events/perf_event.h | 2 ++ 3 files changed, 16 insertions(+)
arch/x86/events/intel/core.c +1 -0
@@ -2836,6 +2836,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, lbr_stack = (struct perf_branch_stack *)bpf->__lbr_stack;intel_pmu_lbr_read (lbr_stack, bpf->counters); +intel_pmu_lbr_clear_raw_counters (lbr_stack); cnt = min_t(unsigned int, cnt, lbr_stack->nr); memcpy(entries, lbr_stack->entries,
arch/x86/events/intel/lbr.c +13 -0
@@ -991,6 +991,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc, } } +void intel_pmu_lbr_clear_raw_counters (struct perf_branch_stack *lbr_stack) +{ +int i; + +if (!(x86_pmu.flags &PMU_FL_BR_CNTR )) +return ; + +/* The reserved field holds the raw branch counters. */ +for (i =0 ; i < lbr_stack->nr; i++) + lbr_stack->entries[i].reserved =0 ; +} +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event)
@@ -1001,6 +1013,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,return ; } +intel_pmu_lbr_clear_raw_counters (cpu_lbr_stack (cpuc));perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); }
arch/x86/events/perf_event.h +2 -0
@@ -1986,6 +1986,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);void intel_pebs_init (void ); +void intel_pmu_lbr_clear_raw_counters (struct perf_branch_stack *lbr_stack); +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
The raw branch counters are stored in the reserved field of each LBR entry. The branch counters path clears that field after reordering the counters, but the BPF snapshot path and the regular branch stack path leave the raw values in place, so they leak to userspace.
Clear the raw counters on those paths too.
Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
[email protected]
AI reviewRe: [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
In reply to Dapeng MiTL;DR: Clearing raw LBR counters in the shared stack may zero them before a later branch-counter event in the same PMI reads them, giving all-zero counter samples until a later commit fixes it.
Quoted 1 line
perf/x86/intel: Clear raw LBR counters in the LBR stack
TL;DR: Clearing raw LBR counters in the shared stack may zero them before a later branch-counter event in the same PMI reads them, giving all-zero counter samples until a later commit fixes it.
Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0
[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_save_brstack(struct perf_sample_data *data, · line 1018
Quoted 7 lines
arch/x86/events/intel/lbr.c +1
-0@@ -1001,6 +1013,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,return ; } +intel_pmu_lbr_clear_raw_counters (cpu_lbr_stack (cpuc));perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); }
Can this clear the counters before a later event in the same PMI has consumed them?
handle_pmi_common() reads the LBR stack only once per PMI and reuses it for every overflowing branch event:
if (has_branch_stack(event)) {
if (!cpuc->lbr_stack_live) {
intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
cpuc->lbr_stack_live = true;
}
intel_pmu_lbr_save_brstack(&data, cpuc, event);
}Suppose event A is an ordinary LBR event on counter 0 and event B is the sampling event of a branch counters group on counter 1, and both overflow in the same PMI. A is handled first, and intel_pmu_lbr_save_brstack() calls intel_pmu_lbr_clear_raw_counters(), which zeroes entries[i].reserved in the shared cpu_lbr_stack().
When B is handled, lbr_stack_live is still true so the stack is not re-read, and intel_pmu_lbr_counters_reorder() reads reserved == 0 for every entry. B's sample then reports all-zero branch counters.
Before this patch the non-counters path left reserved alone, so B got the correct counters in this ordering.
The later commit 57fc5222dbd3 ("perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler") addresses this case by re-reading the LBR stack before a branch counters group event once an earlier LBR event has cleared it. Until that commit, a bisect landing on this patch would see wrong branch counter samples in this scenario.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
Re: [Patch v7 20/24] perf/x86/intel: Clear raw LBR counters in the LBR stack
Quoted 41 lines · On 11/18/2026 11:54 AM, [email protected] wrote:
Quoted 1 line
perf/x86/intel: Clear raw LBR counters in the LBR stack
TL;DR: Clearing raw LBR counters in the shared stack may zero them before a later branch-counter event in the same PMI reads them, giving all-zero counter samples until a later commit fixes it.
Quoted 3 lines
arch/x86/events/intel/lbr.c +0
-0[ ... ]
arch/x86/events/intel/lbr.c void intel_pmu_lbr_save_brstack(struct perf_sample_data *data, · line 1018
Quoted 7 lines
arch/x86/events/intel/lbr.c +1
-0@@ -1001,6 +1013,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,return ; } +intel_pmu_lbr_clear_raw_counters (cpu_lbr_stack (cpuc));perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); }Can this clear the counters before a later event in the same PMI has consumed them?
handle_pmi_common()reads the LBR stack only once per PMI and reuses it for every overflowing branch event:if (has_branch_stack(event)) { if (!cpuc->lbr_stack_live) { intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters); cpuc->lbr_stack_live = true; } intel_pmu_lbr_save_brstack(&data, cpuc, event); }Suppose event A is an ordinary LBR event on counter 0 and event B is the sampling event of a branch counters group on counter 1, and both overflow in the same PMI. A is handled first, and
intel_pmu_lbr_save_brstack()callsintel_pmu_lbr_clear_raw_counters(), which zeroes entries[i].reserved in the sharedcpu_lbr_stack().When B is handled,
lbr_stack_liveis still true so the stack is not re-read, andintel_pmu_lbr_counters_reorder()reads reserved == 0 for every entry. B's sample then reports all-zero branch counters.
Quoted 8 lines
Before this patch the non-counters path left reserved alone, so B got the correct counters in this ordering.
The later commit
57fc5222dbd3("perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler") addresses this case by re-reading the LBR stack before a branch counters group event once an earlier LBR event has cleared it. Until that commit, a bisect landing on this patch would see wrong branch counter samples in this scenario.
Per my understanding, we'd better keep current patch order. Without this patch, the LBR stack needs to be refreshed only when there are two and more LBR counter events, so the patch 57fc5222dbd3 needs to refresh LBR stack only for the cases that there are 2 and more LBR counter events, and we have to add a new patch to relax the condition, the LBR stack should be refreshed as long as there are 1 LBR counter event and other LBR events.
It introduce unnecessary complexity. Thanks.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/44231525404
PATCH v7 21/24 perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler
In reply to Dapeng MiWhen handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refresh the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
This situation is rare, so the extra refresh should have negligible performance impact.
--- arch/x86/events/intel/core.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +10 -2
@@ -3906,8 +3906,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -int bit; +bool lbr_counter_cleared = false;int handled =0 ; +int bit;inc_perf_irq_stat ();
@@ -4080,12 +4081,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period);if (has_branch_stack (event)) { -if (!cpuc->lbr_stack_live) { +/* +* Re-read the LBR stack if it no longer holds the live state, +* or if an earlier LBR event already cleared the raw counters +* that this event needs. +*/ +if (!cpuc->lbr_stack_live || + (lbr_counter_cleared &&is_branch_counters_group (event))) {intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); cpuc->lbr_stack_live = true; }intel_pmu_lbr_save_brstack (&data, cpuc, event); + lbr_counter_cleared = true; }x86_pmu_clear_perf_regs (regs);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 21/24] perf/x86/intel: Refresh LBR stack for LBR counter events in PMI handler
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
When handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refresh the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
Quoted 6 lines
This situation is rare, so the extra refresh should have negligible performance impact.
Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
PATCH v7 22/24 perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr()
In reply to Dapeng Miintel_pmu_store_lbr() assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.
Add a lbr_nr parameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.
--- arch/x86/events/intel/ds.c | 11 ++++++----- arch/x86/events/intel/lbr.c | 18 ++++++++++-------- arch/x86/events/perf_event.h | 2 +- 3 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/events/intel/ds.c +6 -5
@@ -2603,12 +2603,13 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,if (format_group &PEBS_DATACFG_LBRS ) {struct lbr_entry *lbr = next_record; -int num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) - &0xff ) +1 ; +unsigned int num_lbr; + + num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) &0xff ) +1 ; next_record = next_record + num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
@@ -2789,7 +2790,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,if (header->lbr) {struct arch_pebs_lbr_header *lbr_header = next_record;struct lbr_entry *lbr; -int num_lbr; +unsigned int num_lbr; next_record = lbr_header +1 ; lbr = next_record;
@@ -2800,7 +2801,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, next_record += num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
arch/x86/events/intel/lbr.c +10 -8
@@ -906,7 +906,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)static_assert ((64 -PERF_BRANCH_ENTRY_INFO_BITS_MAX ) >LBR_INFO_BR_CNTR_NUM *LBR_INFO_BR_CNTR_BITS ); -static void intel_pmu_store_lbr (struct lbr_entry *entries, +static void intel_pmu_store_lbr (struct lbr_entry *entries,unsigned int lbr_nr,struct perf_branch_stack *lbr_stack) {struct perf_branch_entry *e;
@@ -915,8 +915,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,bool mispred;int i; -cpu_init_lbr_stack (lbr_stack, x86_pmu.lbr_nr); -for (i =0 ; i < x86_pmu.lbr_nr; i++) { +cpu_init_lbr_stack (lbr_stack, lbr_nr); +for (i =0 ; i < lbr_nr; i++) { lbr = entries ? &entries[i] :NULL ; e = &lbr_stack->entries[i];
@@ -1020,7 +1020,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,static void intel_pmu_arch_lbr_read (struct cpu_hw_events *cpuc,struct perf_branch_stack *lbr_stack) { -intel_pmu_store_lbr (NULL , lbr_stack); +intel_pmu_store_lbr (NULL , x86_pmu.lbr_nr, lbr_stack); } static void
@@ -1030,14 +1030,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;if (!xsave) { -intel_pmu_store_lbr (NULL , lbr_stack); +intel_pmu_store_lbr (NULL , x86_pmu.lbr_nr, lbr_stack);return ; } xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR );if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { -intel_pmu_store_lbr (xsave->lbr.entries, lbr_stack); +intel_pmu_store_lbr (xsave->lbr.entries, + x86_pmu.lbr_nr, lbr_stack); }else {/* Clear the LBR stack to avoid leaking stale entries. */ lbr_stack->nr =0 ;
@@ -1317,7 +1318,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc, lbr_stack->nr = lbr_nr; } -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr) +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1328,7 +1329,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)else cpu_lbr_stack (cpuc)->hw_idx =intel_pmu_lbr_tos (); -intel_pmu_store_lbr (lbr,cpu_lbr_stack (cpuc)); +intel_pmu_store_lbr (lbr,min (lbr_nr, x86_pmu.lbr_nr), +cpu_lbr_stack (cpuc));intel_pmu_lbr_filter (cpuc,cpu_lbr_stack (cpuc), cpuc->lbr_counters); cpuc->lbr_stack_live = false; }
arch/x86/events/perf_event.h +1 -1
@@ -1982,7 +1982,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);void intel_pmu_drain_pebs_buffer (void ); -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr); +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr);void intel_pebs_init (void );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 22/24] perf/x86/intel: Pass LBR stack size into intel_pmu_store_lbr()
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
intel_pmu_store_lbr()assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.Add a
lbr_nrparameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=22
PATCH v7 23/24 perf/core: Don't mutate shared branch stack depth
In reply to Dapeng Miperf_sample_save_brstack() clamps the branch stack depth by directly overwriting brs->nr with min(event->attr.sample_max_stack, brs->nr). This is unsafe because the branch stack buffer is shared by all branch events in the same PMI: once one event with a small sample_max_stack truncates brs->nr, every subsequent event processed in that PMI sees the shrunken depth and silently loses branch records it asked for.
Add perf_event_brstack_depth(), which computes the per-event depth without modifying the shared branch stack. Use it in the relevant callers so each event is limited only by its own sample_max_stack, without clobbering the shared LBR state.
bpf_read_branch_records() also relied on the clamped brs->nr, so switch it to perf_event_brstack_depth() to preserve the same per-event limit for records returned to BPF programs.
--- include/linux/perf_event.h | 13 ++++++++++--- kernel/events/core.c | 8 ++++---- kernel/trace/bpf_trace.c | 6 ++++-- 3 files changed, 18 insertions(+), 9 deletions(-)
include/linux/perf_event.h +10 -3
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline u64 perf_event_brstack_depth (struct perf_event *event, +struct perf_branch_stack *brs) +{ +return min_t (u16, event->attr.sample_max_stack, brs->nr); +} +static inline void perf_sample_save_brstack (struct perf_sample_data *data,struct perf_event *event,struct perf_branch_stack *brs,u64 *brs_cntr) {int size =sizeof (u64);/* nr */ +u64 nr;if (!has_branch_stack (event))return ;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,if (branch_sample_hw_index (event)) size +=sizeof (u64); - brs->nr =min_t (u16, event->attr.sample_max_stack, brs->nr); + nr =perf_event_brstack_depth (event, brs); - size += brs->nr *sizeof (struct perf_branch_entry ); + size += nr *sizeof (struct perf_branch_entry );/* * The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,* of events of each branch. */ if (brs_cntr) - size += brs->nr *sizeof (u64); + size += nr *sizeof (u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr;
kernel/events/core.c +4 -4
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {if (data->br_stack) { +u64 nr =perf_event_brstack_depth (event, data->br_stack);size_t size; - size = data->br_stack->nr - *sizeof (struct perf_branch_entry ); + size = nr *sizeof (struct perf_branch_entry ); -perf_output_put (handle, data->br_stack->nr); +perf_output_put (handle, nr);if (branch_sample_hw_index (event))perf_output_put (handle, data->br_stack->hw_idx);perf_output_copy (handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,* right after the struct perf_branch_stack. */ if (data->br_stack_cntr) { - size = data->br_stack->nr *sizeof (u64); + size = nr *sizeof (u64);perf_output_copy (handle, data->br_stack_cntr, size); } }else {
kernel/trace/bpf_trace.c +4 -2
@@ -1549,6 +1549,7 @@ BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,static const u32 br_entry_size =sizeof (struct perf_branch_entry );struct perf_branch_stack *br_stack = ctx->data->br_stack;u32 to_copy; +u64 nr;if (unlikely (flags & ~BPF_F_GET_BRANCH_RECORDS_SIZE ))return -EINVAL ;
@@ -1559,13 +1560,14 @@ BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,if (unlikely (!br_stack))return -ENOENT ; + nr =perf_event_brstack_depth (ctx->event, br_stack);if (flags &BPF_F_GET_BRANCH_RECORDS_SIZE ) -return br_stack->nr * br_entry_size; +return nr * br_entry_size;if (!buf || (size % br_entry_size !=0 ))return -EINVAL ; - to_copy =min_t (u32, br_stack->nr * br_entry_size, size); + to_copy =min_t (u32, nr * br_entry_size, size);memcpy (buf, br_stack->entries, to_copy);return to_copy;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 23/24] perf/core: Don't mutate shared branch stack depth
Sashiko has reviewed this patch and found no issues.
Quoted 12 lines
perf_sample_save_brstack()clamps the branch stack depth by directly overwritingbrs->nrwithmin(event->attr.sample_max_stack, brs->nr). This is unsafe because the branch stack buffer is shared by all branch events in the same PMI: once one event with a smallsample_max_stacktruncatesbrs->nr, every subsequent event processed in that PMI sees the shrunken depth and silently loses branch records it asked for.Add
perf_event_brstack_depth(), which computes the per-event depth without modifying the shared branch stack. Use it in the relevant callers so each event is limited only by its ownsample_max_stack, without clobbering the shared LBR state.
Quoted 7 lines
bpf_read_branch_records()also relied on the clampedbrs->nr, so switch it toperf_event_brstack_depth()to preserve the same per-event limit for records returned to BPF programs.Fixes c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")Reported-by Sashiko <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=23
PATCH v7 24/24 perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing
In reply to Dapeng MiThe overflow status bit for PERF_METRICS is cleared incorrectly in handle_pmi_common(). The code clears the raw bit index value instead of the corresponding BIT_ULL(...) mask, which leaves the overflow flag in an incorrect state.
Define GLOBAL_STATUS_PERF_METRICS_OVF as the proper bit mask and use it when clearing the PERF_METRICS overflow bit in handle_pmi_common().
--- arch/x86/events/intel/core.c | 4 ++-- arch/x86/include/asm/perf_event.h | 1 + 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +2 -2
@@ -3972,7 +3972,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)*/ if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) - status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + status &= ~GLOBAL_STATUS_PERF_METRICS_OVF ; }/*
@@ -3987,7 +3987,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ] &&is_pebs_counter_event_group (cpuc->events[INTEL_PMC_IDX_FIXED_SLOTS ])) - status &= ~GLOBAL_STATUS_PERF_METRICS_OVF_BIT ; + status &= ~GLOBAL_STATUS_PERF_METRICS_OVF ; }/*
arch/x86/include/asm/perf_event.h +1 -0
@@ -452,6 +452,7 @@ static inline bool is_topdown_idx(int idx)#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT 54#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD BIT_ULL(GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT)#define GLOBAL_STATUS_PERF_METRICS_OVF_BIT 48 +#define GLOBAL_STATUS_PERF_METRICS_OVF BIT_ULL(GLOBAL_STATUS_PERF_METRICS_OVF_BIT)#define GLOBAL_CTRL_EN_PERF_METRICS BIT_ULL(48)/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v7 24/24] perf/x86/intel: Fix incorrect PERF_METRICS overflow bit clearing
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
The overflow status bit for
PERF_METRICSis cleared incorrectly inhandle_pmi_common(). The code clears the raw bit index value instead of the correspondingBIT_ULL(...)mask, which leaves the overflow flag in an incorrect state.Define
GLOBAL_STATUS_PERF_METRICS_OVFas the proper bit mask and use it when clearing thePERF_METRICSoverflow bit inhandle_pmi_common().Fixes a5f5e1238f4f ("perf/x86/intel: Don't clear perf metrics overflow bit unconditionally")Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24