perf/x86: Fix multiple LBR sampling issues
Subsystems (5)
v6 cover perf/x86: Fix multiple LBR sampling issues
This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.
The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.
Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceed MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue in perf_snapshot_branch_stack() for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add counted bounds tracking for perf_branch_stack{}.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid overwriting the default LBR stack of other LBR events.
- Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14: Refresh LBR stack in PMI only when LBR users are active for AMD.
- Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when both overflow in the same PMI.
- Patch 16: Remove dead lbr_pebs_users and relative code.
- Patch 17-18: Ensure LBR counters are not dropped when multiple LBR counter events overflow in the same PMI.
- Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid LBR stack out-of-bounds access.
- Patch 20: Fix the shared LBR stack depth truncation issue.
Changes in v6:
- Patch 01: Move LBR stack depth validation into common init_hw_perf_events(). (Peter)
- Patch 10: New patch, add __counted_by() tracking for flexible array entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD. (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading. (Peter)
- Patch 16: New patch, remove dead lbr_pebs_users.
Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.
Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).
Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/
History:
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (19):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel/lbr: Limit branch snapshots to valid entries perf/x86: Add lbr_enabled to track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling
perf: Add counted-bounds tracking for perf_branch_stack perf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed perf/x86/intel: Remove dead lbr_pebs_users
perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler perf/x86/intel: Pass LBR stack size into store_lbr() perf/core: Don't overwrite shared LBR stack depth
Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries
arch/powerpc/perf/core-book3s.c | 43 +++++--- arch/x86/events/amd/brs.c | 7 +- arch/x86/events/amd/core.c | 21 ++-- arch/x86/events/amd/lbr.c | 48 +++++---- arch/x86/events/core.c | 8 +- arch/x86/events/intel/core.c | 130 ++++++++++++++++++++---- arch/x86/events/intel/ds.c | 20 ++-- arch/x86/events/intel/lbr.c | 170 +++++++++++++++++++------------- arch/x86/events/perf_event.h | 45 +++++++-- drivers/perf/arm_brbe.c | 5 + include/linux/perf_event.h | 15 ++- kernel/events/core.c | 8 +- 12 files changed, 365 insertions(+), 155 deletions(-)
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
Signature · 1 line
2.34.1
PATCH v6 1/20 perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.
--- arch/x86/events/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/events/core.c +6 -0
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)if (!x86_pmu.config_mask) x86_pmu.config_mask =X86_RAW_EVENT_MASK ; +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +perf_events_lapic_init ();register_nmi_handler (NMI_LOCAL , perf_event_nmi_handler,0 ,"PMI" );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
Disable LBR if x86_pmu.lbr_nr is larger than
MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v6 2/20 perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
In reply to Dapeng Miintel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.
Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.
--- arch/x86/events/intel/lbr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/events/intel/lbr.c +9 -1
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + cpuc->lbr_stack.nr =0 ; +if (!x86_pmu.lbr_nr)return ;
@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)intel_pmu_store_lbr (cpuc,NULL );return ; } + xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); + }else { +/* Clear the LBR stack to avoid leaking stale entries. */ + cpuc->lbr_stack.nr =0 ; + } }void intel_pmu_lbr_read (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
intel_pmu_arch_lbr_read_xsave()doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.
Quoted 8 lines
Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when
intel_pmu_lbr_reset()is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v6 3/20 perf/x86/amd/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiFrom: Sandipan Das <[email protected]>
The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.
--- arch/x86/events/amd/core.c | 2 +- arch/x86/events/amd/lbr.c | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/events/amd/core.c +1 -1
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u cpuc =this_cpu_ptr (&cpu_hw_events);amd_pmu_lbr_read (); - cnt =min (cnt, x86_pmu.lbr_nr); + cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );
arch/x86/events/amd/lbr.c +3 -1
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)struct branch_entry entry;int out =0 , idx, i; -if (!cpuc->lbr_users) +if (!cpuc->lbr_users) { + cpuc->lbr_stack.nr =0 ;return ; + }for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
From: Sandipan Das <[email protected]>
The branch snapshot callback used by
bpf_get_branch_snapshot()copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read()skips reading the registers, so that a stale count is never reused.
Quoted 6 lines
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Reported-by Sashiko <[email protected]>Signed-off-by Sandipan Das <[email protected]>Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v6 4/20 perf/x86/intel/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiThe branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
--- arch/x86/events/intel/core.c | 2 +- arch/x86/events/intel/lbr.c | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +1 -1
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr); + cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);intel_pmu_enable_all (0 );
arch/x86/events/intel/lbr.c +7 -2
@@ -1027,14 +1027,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +/* * Don't read when all LBRs users are using adaptive PEBS. * * This could be smarter and actually check the event, * but this simple approach seems to work for now. */ -if (!cpuc->lbr_users ||vlbr_exclude_host () || - cpuc->lbr_users == cpuc->lbr_pebs_users) +if (cpuc->lbr_users == cpuc->lbr_pebs_users)return ; x86_pmu.lbr_read (cpuc);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 04/20] perf/x86/intel/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
The branch snapshot callback used by
bpf_get_branch_snapshot()always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
Quoted 6 lines
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Co-developed-by Sandipan Das <[email protected]>Signed-off-by Sandipan Das <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v6 5/20 perf/x86: Add lbr_enabled to track LBR state
In reply to Dapeng MiAdd a lbr_enabled flag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR in perf_snapshot_branch_stack() without checking whether LBR has already been disabled.
--- arch/x86/events/amd/lbr.c | 2 ++ arch/x86/events/intel/core.c | 5 +++++ arch/x86/events/intel/lbr.c | 2 ++ arch/x86/events/perf_event.h | 9 +++++++++ 4 files changed, 18 insertions(+)
arch/x86/events/amd/lbr.c +2 -0
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)if (!cpuc->lbr_users || !x86_pmu.lbr_nr)return ; + cpuc->lbr_enabled =1 ; +/* Set hardware branch filter */ if (cpuc->lbr_select) { lbr_select = cpuc->lbr_sel->config &LBR_SELECT_MASK ;
arch/x86/events/intel/core.c +5 -0
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* It needs to be restored when leaving the handler. */ pmu_enabled = cpuc->enabled; + +/* If pmu version < 4, LBR freeze disables LBR. */ +if (x86_pmu.version <4 ) + cpuc->lbr_enabled =0 ; +/* * In general, the early ACK is only applied for old platforms. * For the big core starts from Haswell, the late ACK should be
arch/x86/events/intel/lbr.c +2 -0
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)if (pmi && x86_pmu.version >=4 )return ; + cpuc->lbr_enabled =1 ; +/* * No need to reprogram LBR_SELECT in a PMI, as it * did not change.
arch/x86/events/perf_event.h +9 -0
@@ -473,6 +473,7 @@ struct cpu_hw_events {unsigned long active_mask[BITS_TO_LONGS (X86_PMC_IDX_MAX )];unsigned long dirty[BITS_TO_LONGS (X86_PMC_IDX_MAX )];int enabled; +int lbr_enabled;int n_events;/* the # of events in the below arrays */ int n_added;/* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event); static __always_inline void __amd_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 dbg_ctl, dbg_extn_cfg; + cpuc->lbr_enabled =0 ; +rdmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg);wrmsrq (MSR_AMD_DBG_EXTN_CFG , dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN );
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void) static __always_inline void __intel_pmu_arch_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->lbr_enabled =0 ;wrmsrq (MSR_ARCH_LBR_CTL ,0 ); } static __always_inline void __intel_pmu_lbr_disable(void) { +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 debugctl; + cpuc->lbr_enabled =0 ;rdmsrq (MSR_IA32_DEBUGCTLMSR , debugctl); debugctl &= ~(DEBUGCTLMSR_LBR |DEBUGCTLMSR_FREEZE_LBRS_ON_PMI );wrmsrq (MSR_IA32_DEBUGCTLMSR , debugctl);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state
Sashiko has reviewed this patch and found no issues.
Quoted 8 lines
Add a
lbr_enabledflag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR inperf_snapshot_branch_stack()without checking whether LBR has already been disabled.Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v6 6/20 perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Miamd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/amd/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/events/amd/core.c +9 -3
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events); -amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (lbr_enabled) +amd_pmu_lbr_enable_all (); +if (pmu_enabled) +amd_pmu_core_enable_all ();local_irq_restore (flags);return cnt;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set --
commit dbf85e8b6a5abeff651d80166ad0c339896c5b42
Author: Dapeng Mi <[email protected]>
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates amd_pmu_v2_snapshot_branch_stack() to save cpuc->enabled and cpuc->lbr_enabled before disabling the PMU and LBR, and only re-enable them if they were previously enabled.
Quoted 23 lines
arch/x86/events/amd/core.c +5
-1@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events);
arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 956
Quoted 10 lines
arch/x86/events/amd/core.c +4
-2-amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (lbr_enabled) +amd_pmu_lbr_enable_all (); +if (pmu_enabled) +amd_pmu_core_enable_all ();
[Severity: Medium]
As reported by Sashiko on v5 [1], while __amd_pmu_lbr_disable() clears cpuc->lbr_enabled = 0, amd_pmu_core_disable_all() only clears the hardware global control MSR and leaves cpuc->enabled set to 1.
If a BPF program in non-NMI context calls bpf_get_branch_snapshot() -> amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save() does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack() is reading LBR MSRs or copying entries, perf_event_nmi_handler() -> amd_pmu_v2_handle_irq() will see cpuc->enabled == 1 and re-enable the core PMU counters on exit:
arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
pmu_enabled = cpuc->enabled;
cpuc->enabled = 0;
/* Stop counting but do not disable LBR */
amd_pmu_core_disable_all();
...
done:
cpuc->enabled = pmu_enabled;
/* Resume counting only if PMU is active */
if (pmu_enabled)
amd_pmu_core_enable_all();
}Similarly, if an NMI triggers a nested BPF call to amd_pmu_v2_snapshot_branch_stack(), the nested call will also observe cpuc->enabled == 1 and call amd_pmu_core_enable_all() before returning to the outer snapshot.
Should amd_pmu_v2_snapshot_branch_stack() clear cpuc->enabled = 0 after saving pmu_enabled and restore cpuc->enabled = pmu_enabled before re-enabling the core PMU, matching amd_pmu_v2_handle_irq()?
[1] https://lore.kernel.org/r/[email protected]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
Quoted 51 lines · On 11/6/2026 1:01 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd:amd_pmu_v2_snapshot_branch_stack()leavescpuc->enabledset --commit
dbf85e8b6a5abeff651d80166ad0c339896c5b42
Author: Dapeng Mi <[email protected]>perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates
amd_pmu_v2_snapshot_branch_stack()to savecpuc->enabledandcpuc->lbr_enabledbefore disabling the PMU and LBR, and only re-enable them if they were previously enabled.Quoted 23 lines
arch/x86/events/amd/core.c +5
-1@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u {struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* * The sequence of steps to freeze LBR should be completely inlined * and contain no branches to minimize contamination of LBR snapshot */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;amd_pmu_core_disable_all ();__amd_pmu_lbr_disable (); - cpuc =this_cpu_ptr (&cpu_hw_events);arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 956
Quoted 10 lines
arch/x86/events/amd/core.c +4
-2-amd_pmu_lbr_read (); cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -amd_pmu_v2_enable_all (0 ); +if (lbr_enabled) +amd_pmu_lbr_enable_all (); +if (pmu_enabled) +amd_pmu_core_enable_all ();[Severity: Medium]
As reported by Sashiko on v5 [1], while__amd_pmu_lbr_disable()clearscpuc->lbr_enabled= 0,amd_pmu_core_disable_all()only clears the hardware global control MSR and leavescpuc->enabledset to 1.
Quoted 6 lines
If a BPF program in non-NMI context calls
bpf_get_branch_snapshot()->amd_pmu_v2_snapshot_branch_stack()whilecpuc->enabledis 1,local_irq_save()does not mask NMIs. If an NMI arrives whileamd_pmu_v2_snapshot_branch_stack()is reading LBR MSRs or copying entries,perf_event_nmi_handler()->amd_pmu_v2_handle_irq()will seecpuc->enabled== 1 and re-enable the core PMU counters on exit:
This issue should never happen in reality since amd_pmu_core_disable_all() would disable PMU, so no PMI can be interrupt the BPF LBR snapshot.
Quoted 27 lines
arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() { pmu_enabled = cpuc->enabled; cpuc->enabled = 0; /* Stop counting but do not disable LBR */ amd_pmu_core_disable_all(); ... done: cpuc->enabled = pmu_enabled; /* Resume counting only if PMU is active */ if (pmu_enabled) amd_pmu_core_enable_all(); }Similarly, if an NMI triggers a nested BPF call to
amd_pmu_v2_snapshot_branch_stack(), the nested call will also observecpuc->enabled== 1 and callamd_pmu_core_enable_all()before returning to the outer snapshot.Should
amd_pmu_v2_snapshot_branch_stack()clearcpuc->enabled= 0 after savingpmu_enabledand restorecpuc->enabled=pmu_enabledbefore re-enabling the core PMU, matchingamd_pmu_v2_handle_irq()?
PATCH v6 7/20 perf/x86/intel: Add helper to enable PMU without LBR
In reply to Dapeng MiCurrently, __intel_pmu_enable_all() also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper, ___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.
Also remove the unused added argument from __intel_pmu_enable_all().
No functional change intended.
--- arch/x86/events/intel/core.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +9 -5
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)intel_pmu_lbr_disable_all (); } -static void __intel_pmu_enable_all (int added,bool pmi) +static void ___intel_pmu_enable_all (void ) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);u64 intel_ctrl =hybrid (cpuc->pmu, intel_ctrl); -intel_pmu_lbr_enable_all (pmi); -if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {wrmsrq (MSR_ARCH_PERFMON_FIXED_CTR_CTRL , cpuc->fixed_ctrl_val); cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi) } } +static void __intel_pmu_enable_all (bool pmi) +{ +intel_pmu_lbr_enable_all (pmi); +___intel_pmu_enable_all (); +} +static void intel_pmu_enable_all (int added) {static_call_cond (x86_pmu_pebs_enable_all)(); -__intel_pmu_enable_all (added, false); +__intel_pmu_enable_all (false); }static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled;if (pmu_enabled) -__intel_pmu_enable_all (0 , true); +__intel_pmu_enable_all (true);intel_bts_enable_local ();/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
Currently,
__intel_pmu_enable_all()also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper,___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.Also remove the unused added argument from
__intel_pmu_enable_all().No functional change intended.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
PATCH v6 8/20 perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.
Only restore the PMU and LBR on the BPF path if they were originally enabled.
--- arch/x86/events/intel/core.c | 38 ++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +34 -4
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added) static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries, -unsigned int cnt,unsigned long flags) +unsigned int cnt,unsigned long flags, +int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -intel_pmu_enable_all (0 ); + +/* +* Do not blindly call intel_pmu_enable_all() to re-enable LBR. +* This path may be entered from the PMI handler through +* bpf_overflow_handler(). +* +* If PMU version < 4, LBR is disabled by the LBR freeze mechanism, +* LBR can't be re-enabled here. +* If PMU version >= 4, LBR is only frozen instead of disabled, +* LBR should be re-enabled here. +*/ +if (lbr_enabled) +intel_pmu_lbr_enable_all (false); +if (pmu_enabled) +___intel_pmu_enable_all (); +local_irq_restore (flags);return cnt; }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,static int intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }static int intel_pmu_snapshot_arch_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_arch_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack() --
commit f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
Author: Dapeng Mi <[email protected]>
perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates __intel_pmu_snapshot_branch_stack() and its callers to save the PMU and LBR enabled state and only re-enable them after taking a branch snapshot if they were originally enabled.
Quoted 70 lines
arch/x86/events/intel/core.c +29
-3@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added) static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries, -unsigned int cnt,unsigned long flags) +unsigned int cnt,unsigned long flags, +int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -intel_pmu_enable_all (0 ); + +/* +* Do not blindly call intel_pmu_enable_all() to re-enable LBR. +* This path may be entered from the PMI handler through +* bpf_overflow_handler(). +* +* If PMU version < 4, LBR is disabled by the LBR freeze mechanism, +* LBR can't be re-enabled here. +* If PMU version >= 4, LBR is only frozen instead of disabled, +* LBR should be re-enabled here. +*/ +if (lbr_enabled) +intel_pmu_lbr_enable_all (false); +if (pmu_enabled) +___intel_pmu_enable_all (); +local_irq_restore (flags);return cnt; }@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,static int intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }static int intel_pmu_snapshot_arch_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags);
arch/x86/events/intel/core.c __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, · line 2891
Quoted 10 lines
arch/x86/events/intel/core.c +5
-1+ cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_arch_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }
[Severity: Medium]
Should intel_pmu_snapshot_branch_stack() and intel_pmu_snapshot_arch_branch_stack() also clear cpuc->enabled = 0 before calling __intel_pmu_disable_all(false), and restore cpuc->enabled = pmu_enabled in __intel_pmu_snapshot_branch_stack()?
While __intel_pmu_lbr_disable() and __intel_pmu_arch_lbr_disable() clear cpuc->lbr_enabled = 0, __intel_pmu_disable_all(false) only writes 0 to MSR_CORE_PERF_GLOBAL_CTRL without updating cpuc->enabled.
When bpf_get_branch_snapshot() is invoked outside PMI context (where cpuc->enabled is 1), local_irq_save() does not mask NMIs. If an in-flight PMI or another NMI arrives while __intel_pmu_snapshot_branch_stack() is reading the LBRs, perf_event_nmi_handler() -> intel_pmu_handle_irq() observes cpuc->enabled == 1 and re-enables the PMU (and LBR on PMU version < 4) before returning:
arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
pmu_enabled = cpuc->enabled;
...
done:
if (mid_ack)
apic_write(APIC_LVTPC, APIC_DM_NMI);
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
cpuc->enabled = pmu_enabled;
if (pmu_enabled)
__intel_pmu_enable_all(true);
}Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program during __intel_pmu_snapshot_branch_stack() would see cpuc->enabled == 1 and prematurely call ___intel_pmu_enable_all() before the outer snapshot finishes.
Setting cpuc->enabled = 0 is a plain store with no branches, so could it be cleared alongside saving pmu_enabled and restored in __intel_pmu_snapshot_branch_stack() to match x86_pmu_disable(), __intel_pmu_quiesce(), and intel_pmu_handle_irq()?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
Quoted 103 lines · On 11/6/2026 1:06 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/intel:cpuc->enablednot cleared inintel_pmu_snapshot_branch_stack()--commit
f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
Author: Dapeng Mi <[email protected]>perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled
This commit updates
__intel_pmu_snapshot_branch_stack()and its callers to save the PMU and LBR enabled state and only re-enable them after taking a branch snapshot if they were originally enabled.Quoted 70 lines
arch/x86/events/intel/core.c +29
-3@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added) static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries, -unsigned int cnt,unsigned long flags) +unsigned int cnt,unsigned long flags, +int pmu_enabled,int lbr_enabled) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); -intel_pmu_enable_all (0 ); + +/* +* Do not blindly call intel_pmu_enable_all() to re-enable LBR. +* This path may be entered from the PMI handler through +* bpf_overflow_handler(). +* +* If PMU version < 4, LBR is disabled by the LBR freeze mechanism, +* LBR can't be re-enabled here. +* If PMU version >= 4, LBR is only frozen instead of disabled, +* LBR should be re-enabled here. +*/ +if (lbr_enabled) +intel_pmu_lbr_enable_all (false); +if (pmu_enabled) +___intel_pmu_enable_all (); +local_irq_restore (flags);return cnt; }@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,static int intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags); + cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }static int intel_pmu_snapshot_arch_branch_stack (struct perf_branch_entry *entries,unsigned int cnt) { +struct cpu_hw_events *cpuc;unsigned long flags; +int pmu_enabled; +int lbr_enabled;/* must not have branches... */ local_irq_save (flags);arch/x86/events/intel/core.c __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, · line 2891
Quoted 10 lines
arch/x86/events/intel/core.c +5
-1+ cpuc =this_cpu_ptr (&cpu_hw_events); + pmu_enabled = cpuc->enabled; + lbr_enabled = cpuc->lbr_enabled;__intel_pmu_disable_all (false);/* we don't care about BTS */ __intel_pmu_arch_lbr_disable ();/* ... until here */ -return __intel_pmu_snapshot_branch_stack (entries, cnt, flags); +return __intel_pmu_snapshot_branch_stack (entries, cnt, flags, + pmu_enabled, lbr_enabled); }[Severity: Medium]
Shouldintel_pmu_snapshot_branch_stack()andintel_pmu_snapshot_arch_branch_stack()also clearcpuc->enabled= 0 before calling__intel_pmu_disable_all(false), and restorecpuc->enabled=pmu_enabledin__intel_pmu_snapshot_branch_stack()?While
__intel_pmu_lbr_disable()and__intel_pmu_arch_lbr_disable()clearcpuc->lbr_enabled= 0,__intel_pmu_disable_all(false)only writes 0 toMSR_CORE_PERF_GLOBAL_CTRLwithout updatingcpuc->enabled.
Quoted 6 lines
When
bpf_get_branch_snapshot()is invoked outside PMI context (wherecpuc->enabledis 1),local_irq_save()does not mask NMIs. If an in-flight PMI or another NMI arrives while__intel_pmu_snapshot_branch_stack()is reading the LBRs,perf_event_nmi_handler()->intel_pmu_handle_irq()observescpuc->enabled== 1 and re-enables the PMU (and LBR on PMU version < 4) before returning:
Same with AMD. This issue should never happen in reality since __intel_pmu_disable_all() would disable PMU, so no PMI can be interrupt the BPF LBR snapshot.
Quoted 23 lines
arch/x86/events/intel/core.c:intel_pmu_handle_irq() { pmu_enabled = cpuc->enabled; ... done: if (mid_ack) apic_write(APIC_LVTPC, APIC_DM_NMI); /* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled; if (pmu_enabled) __intel_pmu_enable_all(true); }Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program during
__intel_pmu_snapshot_branch_stack()would seecpuc->enabled== 1 and prematurely call___intel_pmu_enable_all()before the outer snapshot finishes.Setting
cpuc->enabled= 0 is a plain store with no branches, so could it be cleared alongside savingpmu_enabledand restored in__intel_pmu_snapshot_branch_stack()to matchx86_pmu_disable(),__intel_pmu_quiesce(), andintel_pmu_handle_irq()?
PATCH v6 9/20 perf/x86/intel: Keep LBR frozen throughout PMI handling
In reply to Dapeng MiFor the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
--- arch/x86/events/intel/core.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +11 -2
@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again:intel_pmu_lbr_read (); -intel_pmu_ack_status (status); + +/* +* Do not clear the LBR frozen status here; keep LBR frozen +* throughout PMI handling. +*/ +intel_pmu_ack_status (status & ~GLOBAL_STATUS_LBRS_FROZEN ); +if (++loops >100 ) {static bool warned;
@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)* Repeat if there is more work to be done: */ status =intel_pmu_get_status (); -if (status) +if (status & ~GLOBAL_STATUS_LBRS_FROZEN ) goto again; done: +if (status &GLOBAL_STATUS_LBRS_FROZEN ) +intel_pmu_ack_status (status &GLOBAL_STATUS_LBRS_FROZEN ); +if (mid_ack)apic_write (APIC_LVTPC ,APIC_DM_NMI );/* Only restore PMU state when it's active. See x86_pmu_disable(). */
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.
This can pollute or overwrite the intended branch snapshot, especially when
bpf_get_branch_snapshot()is invoked frombpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.
Quoted 8 lines
Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
PATCH v6 10/20 perf: Add counted-bounds tracking for perf_branch_stack
In reply to Dapeng Mistruct perf_branch_stack uses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member in cpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.
Annotate entries[] with __counted_by(nr) so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR stack backing storage to a complete __lbr_stack[] array and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack* view for x86 and powerpc.
Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
--- arch/powerpc/perf/core-book3s.c | 43 +++++++++++----- arch/x86/events/amd/brs.c | 7 +-- arch/x86/events/amd/core.c | 6 +-- arch/x86/events/amd/lbr.c | 44 +++++++++------- arch/x86/events/core.c | 2 +- arch/x86/events/intel/core.c | 5 +- arch/x86/events/intel/ds.c | 12 +++-- arch/x86/events/intel/lbr.c | 89 +++++++++++++++++---------------- arch/x86/events/perf_event.h | 19 ++++++- drivers/perf/arm_brbe.c | 5 ++ include/linux/perf_event.h | 2 +- 11 files changed, 144 insertions(+), 90 deletions(-)
arch/powerpc/perf/core-book3s.c +30 -13
@@ -30,6 +30,8 @@#define BHRB_PREDICTION 0x0000000000000001#define BHRB_EA 0xFFFFFFFFFFFFFFFCUL +#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES) +struct cpu_hw_events {int n_events;int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {u64 bhrb_filter;/* BHRB HW branch filter */ unsigned int bhrb_users;void *bhrb_context; -struct perf_branch_stack bhrb_stack; -struct perf_branch_entry bhrb_entries[BHRB_MAX_ENTRIES ]; +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 ic_init;/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)return ((TRAP (regs) ==INTERRUPT_PERFMON ) && regs->result); } +static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__lbr_stack; +} + +static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr) +{ +/* +* Initialize the branch-stack count before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ +cpu_lbr_stack (cpuc)->nr = nr; +} +/* * Things that are specific to 64-bit implementations. */
@@ -493,6 +508,8 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events * r_index =0 ; u_index =0 ; + +cpu_init_lbr_stack (cpuhw, ppmu->bhrb_nr);while (r_index < ppmu->bhrb_nr) {/* Assembly read function */ val =read_bhrb (r_index++);
@@ -541,9 +558,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events */* Target branches use two entries * (ie. computed gotos/XL form) */ - cpuhw->bhrb_entries[u_index].to = addr; - cpuhw->bhrb_entries[u_index].mispred = pred; - cpuhw->bhrb_entries[u_index].predicted = ~pred; +cpu_lbr_stack (cpuhw)->entries[u_index].to = addr; +cpu_lbr_stack (cpuhw)->entries[u_index].mispred = pred; +cpu_lbr_stack (cpuhw)->entries[u_index].predicted = ~pred;/* Get from address in next entry */ val =read_bhrb (r_index++);
@@ -554,22 +571,22 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events * r_index--; addr =0 ; } - cpuhw->bhrb_entries[u_index].from = addr; +cpu_lbr_stack (cpuhw)->entries[u_index].from = addr; }else {/* Branches to immediate field (ie I or B form) */ - cpuhw->bhrb_entries[u_index].from = addr; - cpuhw->bhrb_entries[u_index].to = +cpu_lbr_stack (cpuhw)->entries[u_index].from = addr; +cpu_lbr_stack (cpuhw)->entries[u_index].to =power_pmu_bhrb_to (addr); - cpuhw->bhrb_entries[u_index].mispred = pred; - cpuhw->bhrb_entries[u_index].predicted = ~pred; +cpu_lbr_stack (cpuhw)->entries[u_index].mispred = pred; +cpu_lbr_stack (cpuhw)->entries[u_index].predicted = ~pred; } u_index++; } } - cpuhw->bhrb_stack.nr = u_index; - cpuhw->bhrb_stack.hw_idx =-1ULL ; +cpu_lbr_stack (cpuhw)->nr = u_index; +cpu_lbr_stack (cpuhw)->hw_idx =-1ULL ;return ; }
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,struct cpu_hw_events *cpuhw; cpuhw =this_cpu_ptr (&cpu_hw_events);power_pmu_bhrb_read (event, cpuhw); -perf_sample_save_brstack (&data, event, &cpuhw->bhrb_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuhw),NULL ); } if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
arch/x86/events/amd/brs.c +4 -3
@@ -282,7 +282,6 @@ void amd_brs_drain(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);struct perf_event *event = cpuc->events[0 ]; -struct perf_branch_entry *br = cpuc->lbr_entries;union amd_debug_extn_cfg cfg;u32 i, nr =0 , num, tos, start;u32 shift =64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void) num = tos - start +1 ; +cpu_init_lbr_stack (cpuc, num); +/* * BRS is only one pass (saturation) from MSROFF to depth-1 * MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; - br[nr] = (struct perf_branch_entry ){ +cpu_lbr_stack (cpuc)->entries[nr] = (struct perf_branch_entry ){ .from = from, .to = to, };
@@ -351,7 +352,7 @@ void amd_brs_drain(void) } empty:/* Record number of sampled branches */ - cpuc->lbr_stack.nr = nr; +cpu_lbr_stack (cpuc)->nr = nr; }/*
arch/x86/events/amd/core.c +3 -3
@@ -947,8 +947,8 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u__amd_pmu_lbr_disable ();amd_pmu_lbr_read (); - cnt =min (cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); + cnt =min (cnt,cpu_lbr_stack (cpuc)->nr); +memcpy (entries,cpu_lbr_stack (cpuc)->entries,sizeof (struct perf_branch_entry ) * cnt);if (lbr_enabled)amd_pmu_lbr_enable_all ();
@@ -1020,7 +1020,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)if (!x86_perf_event_set_period (event))continue ; -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/amd/lbr.c +26 -18
113 lines not on this page · raw message
arch/x86/events/core.c +1 -1
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)perf_sample_data_init (&data,0 , last_period); -perf_sample_save_brstack (&data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (&data, event,cpu_lbr_stack (cpuc),NULL );perf_event_overflow (event, &data, regs); }
arch/x86/events/intel/core.c +3 -2
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); + cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); +memcpy (entries,cpu_lbr_stack (cpuc)->entries, +sizeof (struct perf_branch_entry ) * cnt);/* * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
arch/x86/events/intel/ds.c +8 -4
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)static int intel_pmu_pebs_fixup_ip (struct pt_regs *regs) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -unsigned long from = cpuc->lbr_entries[0 ].from; -unsigned long old_to, to = cpuc->lbr_entries[0 ].to; +unsigned long from, old_to, to;unsigned long ip = regs->ip;int is_64bit =0 ;void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)/* * No LBR entry, no basic block, no rewinding */ -if (!cpuc->lbr_stack.nr || !from || !to) +if (!cpu_lbr_stack (cpuc)->nr) +return 0 ; + + from =cpu_lbr_stack (cpuc)->entries[0 ].from; + to =cpu_lbr_stack (cpuc)->entries[0 ].to; +if (!from || !to)return 0 ;/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,if (x86_pmu.intel_cap.pebs_format >=3 )setup_pebs_time (event, data, pebs->tsc); -perf_sample_save_brstack (data, event, &cpuc->lbr_stack,NULL ); +perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); } static void adaptive_pebs_save_regs(struct pt_regs *regs,
arch/x86/events/intel/lbr.c +46 -43
255 lines not on this page · raw message
arch/x86/events/perf_event.h +17 -2
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {struct x86_perf_task_context ;#define MAX_LBR_ENTRIES 32 +#define LBR_STACK_SIZE struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES) +enum {LBR_FORMAT_32 =0x00 ,LBR_FORMAT_LIP =0x01 ,
@@ -525,8 +527,7 @@ struct cpu_hw_events {*/ int lbr_users;int lbr_pebs_users; -struct perf_branch_stack lbr_stack; -struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ]; +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 lbr_counters[MAX_LBR_ENTRIES ];/* branch stack extra */ union {struct er_account *lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {struct pmu *pmu; }; +static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc) +{ +return (struct perf_branch_stack *)cpuc->__lbr_stack; +} + +static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr) +{ +/* +* Initialize the branch-stack count before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ +cpu_lbr_stack (cpuc)->nr = nr; +} +#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \ { .idxmsk64 = (n) }, \ .code = (c), \
drivers/perf/arm_brbe.c +5 -0
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,prepare_event_branch_type_mask (branch_sample_type, event_type_mask); +/* +* Initialize the branch_stack->nr before populating entries[] +* to avoid UBSAN complaints from the __counted_by(nr) flexible array. +*/ + branch_stack->nr = nr_hw;for (int bank =0 ; bank < nr_banks; bank++) {int nr_remaining = nr_hw - (bank *BRBE_BANK_MAX_ENTRIES );int nr_this_bank =min (nr_remaining,BRBE_BANK_MAX_ENTRIES );
include/linux/perf_event.h +1 -1
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)struct perf_branch_stack {u64 nr;u64 hw_idx; -struct perf_branch_entry entries[]; + struct perf_branch_entry entries[] __counted_by(nr); };struct task_struct ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
struct
perf_branch_stackuses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member incpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.Annotate entries[] with
__counted_by(nr)so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR stack backing storage to a complete __lbr_stack[] array and add a helpercpu_lbr_stack()to recover the struct perf_branch_stack* view for x86 and powerpc.Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.
No functional change intended.
Quoted 10 lines
Suggested-by Peter Zijlstra <[email protected]>Cc Sandipan Das <[email protected]>Cc Madhavan Srinivasan <[email protected]>Cc Michael Ellerman <[email protected]>Cc Nicholas Piggin <[email protected]>Cc "Christophe Leroy (CS GROUP)" <[email protected]>Cc Will Deacon <[email protected]>Cc Mark Rutland <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
PATCH v6 11/20 perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
In reply to Dapeng MiThe LBR read helpers currently assume the default stack, cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the *_lbr_read() helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Update the related callers, including intel_pmu_store_lbr() and intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.
No function change intended.
--- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/core.c | 4 +- arch/x86/events/intel/lbr.c | 89 ++++++++++++++++++++---------------- arch/x86/events/perf_event.h | 16 ++++--- 5 files changed, 64 insertions(+), 49 deletions(-)
arch/x86/events/amd/brs.c +1 -1
@@ -316,7 +316,7 @@ void amd_brs_drain(void) num = tos - start +1 ; -cpu_init_lbr_stack (cpuc, num); +cpu_init_lbr_stack (cpu_lbr_stack (cpuc), num);/* * BRS is only one pass (saturation) from MSROFF to depth-1
arch/x86/events/amd/lbr.c +1 -1
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)return ; } -cpu_init_lbr_stack (cpuc, x86_pmu.lbr_nr); +cpu_init_lbr_stack (cpu_lbr_stack (cpuc), x86_pmu.lbr_nr);for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
arch/x86/events/intel/core.c +2 -2
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -intel_pmu_lbr_read (); +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4062,7 +4062,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (); +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters);/* * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +50 -39
234 lines not on this page · raw message
arch/x86/events/perf_event.h +10 -6
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_evereturn (struct perf_branch_stack *)cpuc->__lbr_stack; } -static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr) +static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr) {/* * Initialize the branch-stack count before populating entries[] * to avoid UBSAN complaints from the __counted_by(nr) flexible array. */ -cpu_lbr_stack (cpuc)->nr = nr; + lbr_stack->nr = nr; }#define __EVENT_CONSTRAINT_RANGE (c, e, n, m, w, o, f) { \
@@ -1207,7 +1207,8 @@ struct x86_pmu { unsigned int lbr_counters:4;void (*lbr_reset)(void ); -void (*lbr_read)(struct cpu_hw_events *cpuc); +void (*lbr_read)(struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack);void (*lbr_save)(void *ctx);void (*lbr_restore)(void *ctx);
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);void intel_pmu_lbr_disable_all (void ); -void intel_pmu_lbr_read (void ); +void intel_pmu_lbr_read (struct perf_branch_stack *lbr_stack, +u64 *lbr_counters); -void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_32 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack); -void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc); +void intel_pmu_lbr_read_64 (struct cpu_hw_events *cpuc, +struct perf_branch_stack *lbr_stack);void intel_pmu_lbr_save (void *ctx);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks
Sashiko has reviewed this patch and found no issues.
Quoted 6 lines
The LBR read helpers currently assume the default stack,
cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the*_lbr_read()helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.
Quoted 7 lines
Update the related callers, including
intel_pmu_store_lbr()andintel_pmu_lbr_filter(), to pass the appropriate stack explicitly.No function change intended.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
PATCH v6 12/20 perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
In reply to Dapeng MiBPF LBR snapshotting currently reuses the default LBR stack (cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context via bpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
--- arch/x86/events/intel/core.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +12 -4
@@ -2819,17 +2819,25 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (false); } +struct lbr_snapshot { +u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags,int pmu_enabled,int lbr_enabled) { -struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *bpf =this_cpu_ptr (&lbr_snapshot); +struct perf_branch_stack *lbr_stack = + (struct perf_branch_stack *)bpf->__lbr_stack; -intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); - cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr); +intel_pmu_lbr_read (lbr_stack, bpf->counters); + cnt = min_t(unsigned int, cnt, lbr_stack->nr); -memcpy (entries,cpu_lbr_stack (cpuc)->entries, +memcpy (entries, lbr_stack->entries,sizeof (struct perf_branch_entry ) * cnt);/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting
Sashiko has reviewed this patch and found no issues.
Quoted 8 lines
BPF LBR snapshotting currently reuses the default LBR stack (
cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context viabpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.
Quoted 6 lines
Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.
Reported-by Sashiko <[email protected]>Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
PATCH v6 13/20 perf/x86/intel: Always snapshot live hardware LBR state for BPF
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.
BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
--- arch/x86/events/intel/core.c | 7 ++++++- arch/x86/events/intel/lbr.c | 9 --------- 2 files changed, 6 insertions(+), 10 deletions(-)
arch/x86/events/intel/core.c +6 -1
@@ -4070,7 +4070,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters);/* * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +0 -9
@@ -1043,15 +1043,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc, lbr_stack);intel_pmu_lbr_filter (cpuc, lbr_stack, lbr_counters);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
__intel_pmu_snapshot_branch_stack()is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
PATCH v6 14/20 perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
In reply to Dapeng MiRefreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.
Limit the LBR stack refresh to the PMI path only when LBR users are present.
--- arch/x86/events/amd/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/events/amd/core.c +2 -1
@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)/* Read branch records */ if (x86_pmu.lbr_nr) { -amd_pmu_lbr_read (); +if (cpuc->lbr_users >0 ) +amd_pmu_lbr_read (); status &= ~GLOBAL_STATUS_LBRS_FROZEN ; }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
Sashiko has reviewed this patch and found no issues.
Quoted 9 lines
Refreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.
Limit the LBR stack refresh to the PMI path only when LBR users are present.
Suggested-by Peter Zijlstra <[email protected]>Cc "Sandipan Das <[email protected]>"Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Re: [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active
In reply to Dapeng MiQuoted 18 lines · Dapeng Mi
Refreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.
Limit the LBR stack refresh to the PMI path only when LBR users are present.
Suggested-by Peter Zijlstra <[email protected]>Cc "Sandipan Das <[email protected]>"Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/amd/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/x86/events/amd/core.c +0
-0
arch/x86/events/amd/core.c static int amd_pmu_v2_handle_irq(struct pt_regs *regs) · line 995
Quoted 9 lines
arch/x86/events/amd/core.c +2
-1@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)/* Read branch records */ if (x86_pmu.lbr_nr) { -amd_pmu_lbr_read (); +if (cpuc->lbr_users >0 ) +amd_pmu_lbr_read (); status &= ~GLOBAL_STATUS_LBRS_FROZEN ; }
This patch is actually unnecessary since amd_pmu_lbr_read() internally checks cpuc->lbr_users and would early return if cpuc->lbr_users is 0.
Would drop this patch in next version. Thanks.
PATCH v6 15/20 perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
In reply to Dapeng MiPEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.
When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, so userspace can see the wrong branch stack instead of the live LBR state at PMI entry.
Track whether the LBR state has already been refreshed with a lbr_refreshed flag. If PEBS events overwrite the shared LBR stack before the non-PEBS LBR event is processed, clear the flag so the live LBR state is refreshed again. Remove the unconditional LBR refresh at PMI entry and initialize the flag as false so the stack is refreshed only when an active non-PEBS LBR event requires it. Since the LBR state is frozen while the PMI is being handled, it only needs to be read once for multiple non-PEBS LBR events in the same interrupt.
--- arch/x86/events/intel/core.c | 40 +++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-)
arch/x86/events/intel/core.c +30 -10
@@ -3864,8 +3864,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) {struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -int bit; +bool lbr_refreshed = false;int handled =0 ; +int bit;inc_perf_irq_stat ();
@@ -3906,6 +3907,15 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)if (__test_and_clear_bit (GLOBAL_STATUS_BUFFER_OVF_BIT , (unsigned long *)&status)) {u64 pebs_enabled = cpuc->pebs_enabled; +/* +* Prior to adaptive PEBS, PEBS does not support LBR snapshotting. +* Refresh the LBR stack explicitly when LBR users are present. +*/ +if (cpuc->lbr_users >0 && !x86_pmu.intel_cap.pebs_baseline) { +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); + lbr_refreshed = true; + } + handled +=x86_pmu_handle_guest_pebs (regs, &data); handled +=static_call (x86_pmu_drain_pebs)(regs, &data);/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) { +/* +* PEBS counter sampling is only supported on adaptive PEBS +* and arch-PEBS, both of which snapshot the LBR stack. +* No explicit LBR refresh is needed here. +*/ static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_refreshed as false. +*/ + lbr_refreshed = false; + } last_period = event->hw.last_period;
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) { +if (!lbr_refreshed) { +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), + cpuc->lbr_counters); + lbr_refreshed = true; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);
@@ -4070,13 +4097,6 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -/* -* No need to read the LBR stack when all LBR users are PEBS events, -* since the PEBS handler updates it. -*/ -if (cpuc->lbr_users != cpuc->lbr_pebs_users) -intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); -/* * Do not clear the LBR frozen status here; keep LBR frozen * throughout PMI handling.
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain
--
commit f13e61130af8b47007c91f048b386359760f05ad
Author: Dapeng Mi <[email protected]>
perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
This commit tracks whether the shared LBR stack has been refreshed during PMI handling with a lbr_refreshed flag and removes the unconditional LBR read at PMI entry so the live LBR state is read on demand for non-PEBS LBR events and re-read if PEBS draining overwrites it.
Quoted 4 lines
arch/x86/events/intel/core.c +0
-0
[ ... ]
Quoted 27 lines
arch/x86/events/intel/core.c +13
-2@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) { +/* +* PEBS counter sampling is only supported on adaptive PEBS +* and arch-PEBS, both of which snapshot the LBR stack. +* No explicit LBR refresh is needed here. +*/ static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_refreshed as false. +*/ + lbr_refreshed = false; + } last_period = event->hw.last_period;@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) {
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4047
Quoted 10 lines
arch/x86/events/intel/core.c +6
-0+if (!lbr_refreshed) { +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), + cpuc->lbr_counters); + lbr_refreshed = true; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);
[Severity: Medium]
Can perf_event_overflow() still drain the PEBS buffer and clobber cpu_lbr_stack(cpuc) when is_pebs_counter_event_group(event) is false?
When a non-PEBS event using PERF_SAMPLE_READ | PERF_FORMAT_GROUP is grouped with a PERF_X86_EVENT_AUTO_RELOAD PEBS LBR sibling (without PERF_X86_EVENT_PEBS_CNTR set on the leader), perf_event_overflow() -> perf_output_sample() -> perf_output_read_group() -> intel_pmu_read_event() calls intel_pmu_drain_pebs_buffer() before PERF_SAMPLE_BRANCH_STACK is written. Draining pending PEBS records calls intel_pmu_store_pebs_lbrs(), which overwrites cpu_lbr_stack(cpuc) before the current non-PEBS sample copies data.br_stack and leaves lbr_refreshed set to true for any subsequent non-PEBS LBR events in the same PMI.
In addition, because the unconditional intel_pmu_lbr_read() at PMI entry is removed, an indirect PEBS drain on pre-adaptive PEBS (!x86_pmu.intel_cap.pebs_baseline) when GLOBAL_STATUS_BUFFER_OVF_BIT is clear will run intel_pmu_pebs_fixup_ip() against an unrefreshed cpu_lbr_stack(cpuc).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Re: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
Quoted 73 lines · On 11/6/2026 1:58 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain
--commit
f13e61130af8b47007c91f048b386359760f05ad
Author: Dapeng Mi <[email protected]>perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed
This commit tracks whether the shared LBR stack has been refreshed during PMI handling with a
lbr_refreshedflag and removes the unconditional LBR read at PMI entry so the live LBR state is read on demand for non-PEBS LBR events and re-read if PEBS draining overwrites it.Quoted 4 lines
arch/x86/events/intel/core.c +0
-0[ ... ]
Quoted 27 lines
arch/x86/events/intel/core.c +13
-2@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)* * The PEBS buffer has to be drained before handling the A-PMI */ -if (is_pebs_counter_event_group (event)) +if (is_pebs_counter_event_group (event)) { +/* +* PEBS counter sampling is only supported on adaptive PEBS +* and arch-PEBS, both of which snapshot the LBR stack. +* No explicit LBR refresh is needed here. +*/ static_call (x86_pmu_drain_pebs)(regs, &data); +/* +* Draining the PEBS buffer can overwrite the shared +* LBR stack, so mark lbr_refreshed as false. +*/ + lbr_refreshed = false; + } last_period = event->hw.last_period;@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period); -if (has_branch_stack (event)) +if (has_branch_stack (event)) {arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4047
Quoted 10 lines
arch/x86/events/intel/core.c +6
-0+if (!lbr_refreshed) { +intel_pmu_lbr_read (cpu_lbr_stack (cpuc), + cpuc->lbr_counters); + lbr_refreshed = true; + }intel_pmu_lbr_save_brstack (&data, cpuc, event); + }x86_pmu_clear_perf_regs (regs);x86_pmu_update_perf_regs (event, &data, regs, false);[Severity: Medium]
Canperf_event_overflow()still drain the PEBS buffer and clobbercpu_lbr_stack(cpuc)whenis_pebs_counter_event_group(event)is false?When a non-PEBS event using
PERF_SAMPLE_READ|PERF_FORMAT_GROUPis grouped with aPERF_X86_EVENT_AUTO_RELOADPEBS LBR sibling (withoutPERF_X86_EVENT_PEBS_CNTRset on the leader),perf_event_overflow()->perf_output_sample()->perf_output_read_group()->intel_pmu_read_event()callsintel_pmu_drain_pebs_buffer()beforePERF_SAMPLE_BRANCH_STACKis written. Draining pending PEBS records callsintel_pmu_store_pebs_lbrs(), which overwritescpu_lbr_stack(cpuc)before the current non-PEBS sample copies data.br_stack and leaveslbr_refreshedset to true for any subsequent non-PEBS LBR events in the same PMI.
Quoted 5 lines
In addition, because the unconditional
intel_pmu_lbr_read()at PMI entry is removed, an indirect PEBS drain on pre-adaptive PEBS (!x86_pmu.intel_cap.pebs_baseline) whenGLOBAL_STATUS_BUFFER_OVF_BITis clear will runintel_pmu_pebs_fixup_ip()against an unrefreshedcpu_lbr_stack(cpuc).
Sigh, Sashiko can always give surprise. :)
It's a real issue. It looks a new version has to be posted to fix this issue.
Quoted 1 line
PATCH v6 16/20 perf/x86/intel: Remove dead lbr_pebs_users
In reply to Dapeng MiThe lbr_pebs_users accounting is no longer used, so remove the dead field and the related code paths that maintain it.
--- arch/x86/events/intel/lbr.c | 5 ----- arch/x86/events/perf_event.h | 1 - 2 files changed, 6 deletions(-)
arch/x86/events/intel/lbr.c +0 -5
@@ -617,8 +617,6 @@ void intel_pmu_lbr_add(struct perf_event *event)* be 'new'. Conversely, a new event can get installed through the * context switch path for the first time. */ -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) - cpuc->lbr_pebs_users++;perf_sched_cb_inc (event->pmu);if (!cpuc->lbr_users++ && !event->total_time_running)intel_pmu_lbr_reset ();
@@ -688,11 +686,8 @@ void intel_pmu_lbr_del(struct perf_event *event)if (event->hw.flags &PERF_X86_EVENT_LBR_SELECT ) cpuc->lbr_select =0 ; -if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip >0 ) - cpuc->lbr_pebs_users--; cpuc->lbr_users--;WARN_ON_ONCE (cpuc->lbr_users <0 ); -WARN_ON_ONCE (cpuc->lbr_pebs_users <0 );perf_sched_cb_dec (event->pmu);/*
arch/x86/events/perf_event.h +0 -1
@@ -526,7 +526,6 @@ struct cpu_hw_events {* Intel LBR bits */ int lbr_users; -int lbr_pebs_users;u64 __lbr_stack[LBR_STACK_SIZE /sizeof (u64)];u64 lbr_counters[MAX_LBR_ENTRIES ];/* branch stack extra */ union {
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users
Sashiko has reviewed this patch and found no issues.
Quoted 4 lines
The
lbr_pebs_usersaccounting is no longer used, so remove the dead field and the related code paths that maintain it.Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
PATCH v6 17/20 perf/x86/intel: Clear raw LBR counters in the LBR stack
In reply to Dapeng MiRaw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.
If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.
--- arch/x86/events/intel/core.c | 1 + arch/x86/events/intel/lbr.c | 13 +++++++++++++ arch/x86/events/perf_event.h | 2 ++ 3 files changed, 16 insertions(+)
arch/x86/events/intel/core.c +1 -0
@@ -2835,6 +2835,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, (struct perf_branch_stack *)bpf->__lbr_stack;intel_pmu_lbr_read (lbr_stack, bpf->counters); +intel_pmu_lbr_clear_counters (lbr_stack); cnt = min_t(unsigned int, cnt, lbr_stack->nr); memcpy(entries, lbr_stack->entries,
arch/x86/events/intel/lbr.c +13 -0
@@ -987,6 +987,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc, } } +void intel_pmu_lbr_clear_counters (struct perf_branch_stack *lbr_stack) +{ +int i; + +if (!(x86_pmu.flags &PMU_FL_BR_CNTR )) +return ; + +/* reserved field stores raw branch counters, clear it. */ +for (i =0 ; i < lbr_stack->nr; i++) + lbr_stack->entries[i].reserved =0 ; +} +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event)
@@ -997,6 +1009,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,return ; } +intel_pmu_lbr_clear_counters (cpu_lbr_stack (cpuc));perf_sample_save_brstack (data, event,cpu_lbr_stack (cpuc),NULL ); }
arch/x86/events/perf_event.h +2 -0
@@ -1984,6 +1984,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);void intel_pebs_init (void ); +void intel_pmu_lbr_clear_counters (struct perf_branch_stack *lbr_stack); +void intel_pmu_lbr_save_brstack (struct perf_sample_data *data,struct cpu_hw_events *cpuc,struct perf_event *event);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
Raw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.
If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.
Reported-by Sashiko <[email protected]>Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
PATCH v6 18/20 perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
In reply to Dapeng MiWhen handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
This situation is rare, the performance impact from extra refill should be negligible.
--- arch/x86/events/intel/core.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/events/intel/core.c +10 -1
@@ -3866,6 +3866,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)struct perf_sample_data data;struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);bool lbr_refreshed = false; +bool lbr_counter_cleared = false;int handled =0 ;int bit;
@@ -4036,12 +4037,20 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , last_period);if (has_branch_stack (event)) { -if (!lbr_refreshed) { +/* +* The raw LBR counters stored in the reserved field of +* the LBR entries were cleared during processing of the +* previous LBR event, so refresh the LBR stack again if +* this event needs the LBR counters. +*/ +if (!lbr_refreshed || + (lbr_counter_cleared &&is_branch_counters_group (event))) {intel_pmu_lbr_read (cpu_lbr_stack (cpuc), cpuc->lbr_counters); lbr_refreshed = true; }intel_pmu_lbr_save_brstack (&data, cpuc, event); + lbr_counter_cleared = true; }x86_pmu_clear_perf_regs (regs);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
When handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.
However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.
Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.
Quoted 6 lines
This situation is rare, the performance impact from extra refill should be negligible.
Reported-by Sashiko <[email protected]>Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
PATCH v6 19/20 perf/x86/intel: Pass LBR stack size into store_lbr()
In reply to Dapeng Miintel_pmu_store_lbr() assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.
Add a lbr_nr parameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.
--- arch/x86/events/intel/ds.c | 8 ++++---- arch/x86/events/intel/lbr.c | 18 ++++++++++-------- arch/x86/events/perf_event.h | 2 +- 3 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/events/intel/ds.c +4 -4
@@ -2595,12 +2595,12 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,if (format_group &PEBS_DATACFG_LBRS ) {struct lbr_entry *lbr = next_record; -int num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) +unsigned int num_lbr = ((format_group >>PEBS_DATACFG_LBR_SHIFT ) &0xff ) +1 ; next_record = next_record + num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
@@ -2781,7 +2781,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,if (header->lbr) {struct arch_pebs_lbr_header *lbr_header = next_record;struct lbr_entry *lbr; -int num_lbr; +unsigned int num_lbr; next_record = lbr_header +1 ; lbr = next_record;
@@ -2792,7 +2792,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event, next_record += num_lbr *sizeof (struct lbr_entry );if (has_branch_stack (event)) { -intel_pmu_store_pebs_lbrs (lbr); +intel_pmu_store_pebs_lbrs (lbr, num_lbr);intel_pmu_lbr_save_brstack (data, cpuc, event); } }
arch/x86/events/intel/lbr.c +10 -8
@@ -902,7 +902,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)static_assert ((64 -PERF_BRANCH_ENTRY_INFO_BITS_MAX ) >LBR_INFO_BR_CNTR_NUM *LBR_INFO_BR_CNTR_BITS ); -static void intel_pmu_store_lbr (struct lbr_entry *entries, +static void intel_pmu_store_lbr (struct lbr_entry *entries,int lbr_nr,struct perf_branch_stack *lbr_stack) {struct perf_branch_entry *e;
@@ -911,8 +911,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,bool mispred;int i; -cpu_init_lbr_stack (lbr_stack, x86_pmu.lbr_nr); -for (i =0 ; i < x86_pmu.lbr_nr; i++) { +cpu_init_lbr_stack (lbr_stack, lbr_nr); +for (i =0 ; i < lbr_nr; i++) { lbr = entries ? &entries[i] :NULL ; e = &lbr_stack->entries[i];
@@ -1016,7 +1016,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,static void intel_pmu_arch_lbr_read (struct cpu_hw_events *cpuc,struct perf_branch_stack *lbr_stack) { -intel_pmu_store_lbr (NULL , lbr_stack); +intel_pmu_store_lbr (NULL , x86_pmu.lbr_nr, lbr_stack); } static void
@@ -1026,14 +1026,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;if (!xsave) { -intel_pmu_store_lbr (NULL , lbr_stack); +intel_pmu_store_lbr (NULL , x86_pmu.lbr_nr, lbr_stack);return ; } xsave->header.xfeatures =0 ;xsaves (&xsave->xsave,XFEATURE_MASK_LBR );if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) { -intel_pmu_store_lbr (xsave->lbr.entries, lbr_stack); +intel_pmu_store_lbr (xsave->lbr.entries, + x86_pmu.lbr_nr, lbr_stack); }else {/* Clear the LBR stack to avoid leaking stale entries. */ lbr_stack->nr =0 ;
@@ -1313,7 +1314,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc, lbr_stack->nr = lbr_nr; } -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr) +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
@@ -1324,7 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)else cpu_lbr_stack (cpuc)->hw_idx =intel_pmu_lbr_tos (); -intel_pmu_store_lbr (lbr,cpu_lbr_stack (cpuc)); +intel_pmu_store_lbr (lbr,min (lbr_nr, x86_pmu.lbr_nr), +cpu_lbr_stack (cpuc));intel_pmu_lbr_filter (cpuc,cpu_lbr_stack (cpuc), cpuc->lbr_counters); }
arch/x86/events/perf_event.h +1 -1
@@ -1980,7 +1980,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);void intel_pmu_drain_pebs_buffer (void ); -void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr); +void intel_pmu_store_pebs_lbrs (struct lbr_entry *lbr,unsigned int lbr_nr);void intel_pebs_init (void );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr()
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
intel_pmu_store_lbr()assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.Add a
lbr_nrparameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.Reported-by Sashiko <[email protected]>Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
PATCH v6 20/20 perf/core: Don't overwrite shared LBR stack depth
In reply to Dapeng Miperf_sample_save_brstack() currently clamps the shared LBR stack depth to min(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a small sample_max_stack from one event can shrink the depth for all others in the same PMI.
As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.
--- include/linux/perf_event.h | 13 ++++++++++--- kernel/events/core.c | 8 ++++---- 2 files changed, 14 insertions(+), 7 deletions(-)
include/linux/perf_event.h +10 -3
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline u64 max_lbr_stack_depth (struct perf_event *event, +struct perf_branch_stack *brs) +{ +return min_t (u16, event->attr.sample_max_stack, brs->nr); +} +static inline void perf_sample_save_brstack (struct perf_sample_data *data,struct perf_event *event,struct perf_branch_stack *brs,u64 *brs_cntr) {int size =sizeof (u64);/* nr */ +u64 lbr_nr;if (!has_branch_stack (event))return ;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,if (branch_sample_hw_index (event)) size +=sizeof (u64); - brs->nr =min_t (u16, event->attr.sample_max_stack, brs->nr); + lbr_nr =max_lbr_stack_depth (event, brs); - size += brs->nr *sizeof (struct perf_branch_entry ); + size += lbr_nr *sizeof (struct perf_branch_entry );/* * The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,* of events of each branch. */ if (brs_cntr) - size += brs->nr *sizeof (u64); + size += lbr_nr *sizeof (u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr;
kernel/events/core.c +4 -4
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {if (data->br_stack) { +u64 lbr_nr =max_lbr_stack_depth (event, data->br_stack);size_t size; - size = data->br_stack->nr - *sizeof (struct perf_branch_entry ); + size = lbr_nr *sizeof (struct perf_branch_entry ); -perf_output_put (handle, data->br_stack->nr); +perf_output_put (handle, lbr_nr);if (branch_sample_hw_index (event))perf_output_put (handle, data->br_stack->hw_idx);perf_output_copy (handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,* right after the struct perf_branch_stack. */ if (data->br_stack_cntr) { - size = data->br_stack->nr *sizeof (u64); + size = lbr_nr *sizeof (u64);perf_output_copy (handle, data->br_stack_cntr, size); } }else {
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth
Sashiko has reviewed this patch and found no issues.
Quoted 12 lines
perf_sample_save_brstack()currently clamps the shared LBR stack depth tomin(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a smallsample_max_stackfrom one event can shrink the depth for all others in the same PMI.As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in
perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.
Quoted 3 lines
Reported-by Sashiko <[email protected]>Fixes c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
Sashiko reported an issue on the patch 15/20 "perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed",
https://lore.kernel.org/all/[email protected]/.
It looks a new version is needed to fix the issue. Thanks.
Quoted 130 lines · Dapeng Mi
This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.
The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceedMAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue inperf_snapshot_branch_stack()for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add counted bounds tracking for perf_branch_stack{}.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid overwriting the default LBR stack of other LBR events.
- Patch 13: Ensureperf_snapshot_branch_stack()snapshots the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14: Refresh LBR stack in PMI only when LBR users are active for AMD.
- Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when both overflow in the same PMI.
- Patch 16: Remove deadlbr_pebs_usersand relative code.
- Patch 17-18: Ensure LBR counters are not dropped when multiple LBR counter events overflow in the same PMI.
- Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid LBR stack out-of-bounds access.
- Patch 20: Fix the shared LBR stack depth truncation issue.Changes in v6:
- Patch 01: Move LBR stack depth validation into commoninit_hw_perf_events(). (Peter)
- Patch 10: New patch, add__counted_by()tracking for flexible array entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD. (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading. (Peter)
- Patch 16: New patch, remove deadlbr_pebs_users.Changes in v5:
- Patch 02: Clearxsave->header.xfeaturesbefore calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceedsMAX_LBR_ENTRIES(Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to excludeMAX_LBR_ENTRIES(Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/History:
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Dapeng Mi (19):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceedsMAX_LBR_ENTRIESperf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel/lbr: Limit branch snapshots to valid entries perf/x86: Addlbr_enabledto track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling
perf: Add counted-bounds tracking forperf_branch_stackperf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed perf/x86/intel: Remove deadlbr_pebs_users
perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler perf/x86/intel: Pass LBR stack size intostore_lbr()perf/core: Don't overwrite shared LBR stack depthSandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entriesarch/powerpc/perf/core-book3s.c | 43 +++++--- arch/x86/events/amd/brs.c | 7 +- arch/x86/events/amd/core.c | 21 ++-- arch/x86/events/amd/lbr.c | 48 +++++---- arch/x86/events/core.c | 8 +- arch/x86/events/intel/core.c | 130 ++++++++++++++++++++---- arch/x86/events/intel/ds.c | 20 ++-- arch/x86/events/intel/lbr.c | 170 +++++++++++++++++++------------- arch/x86/events/perf_event.h | 45 +++++++-- drivers/perf/arm_brbe.c | 5 + include/linux/perf_event.h | 15 ++- kernel/events/core.c | 8 +- 12 files changed, 365 insertions(+), 155 deletions(-)base-commit:
6628e7d4c6b25c8d18f1345e3a20d21829aae7e4