Skip to contentSkip to conversation

perf/x86: Fix multiple LBR sampling issues

PATCH Superseded Dapeng Mi 2 participants 46 messages Nov 6 – Nov 12

Subsystems (5)
x86 PowerPC BPF perf events +1 more subsystems

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Dapeng Mi

author
#0 lore raw

v6 cover perf/x86: Fix multiple LBR sampling issues

This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.

The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.

Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceed MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue in perf_snapshot_branch_stack() for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add counted bounds tracking for perf_branch_stack{}.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid overwriting the default LBR stack of other LBR events.
- Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14: Refresh LBR stack in PMI only when LBR users are active for AMD.
- Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when both overflow in the same PMI.
- Patch 16: Remove dead lbr_pebs_users and relative code.
- Patch 17-18: Ensure LBR counters are not dropped when multiple LBR counter events overflow in the same PMI.
- Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid LBR stack out-of-bounds access.
- Patch 20: Fix the shared LBR stack depth truncation issue.

Changes in v6:
- Patch 01: Move LBR stack depth validation into common init_hw_perf_events(). (Peter)
- Patch 10: New patch, add __counted_by() tracking for flexible array entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD. (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading. (Peter)
- Patch 16: New patch, remove dead lbr_pebs_users.

Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.

Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).

Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).

Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/

History:
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Dapeng Mi (19):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel/lbr: Limit branch snapshots to valid entries perf/x86: Add lbr_enabled to track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling
perf: Add counted-bounds tracking for perf_branch_stack perf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed perf/x86/intel: Remove dead lbr_pebs_users
perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler perf/x86/intel: Pass LBR stack size into store_lbr() perf/core: Don't overwrite shared LBR stack depth

Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries

 arch/powerpc/perf/core-book3s.c |  43 +++++---
 arch/x86/events/amd/brs.c       |   7 +-
 arch/x86/events/amd/core.c      |  21 ++--
 arch/x86/events/amd/lbr.c       |  48 +++++----
 arch/x86/events/core.c          |   8 +-
 arch/x86/events/intel/core.c    | 130 ++++++++++++++++++++----
 arch/x86/events/intel/ds.c      |  20 ++--
 arch/x86/events/intel/lbr.c     | 170 +++++++++++++++++++-------------
 arch/x86/events/perf_event.h    |  45 +++++++--
 drivers/perf/arm_brbe.c         |   5 +
 include/linux/perf_event.h      |  15 ++-
 kernel/events/core.c            |   8 +-
 12 files changed, 365 insertions(+), 155 deletions(-)

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v6 1/20 perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES

Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.

---
 arch/x86/events/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/core.c +6 -0
index 4e344fb1ae3e..82d9b0e5ad1c 100644
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)
 	if (!x86_pmu.config_mask)
 		x86_pmu.config_mask = X86_RAW_EVENT_MASK;
 
+	if (x86_pmu.lbr_nr > MAX_LBR_ENTRIES) {
+		pr_warn("LBR entries %d exceeds max %d, disable LBR.\n",
+			x86_pmu.lbr_nr, MAX_LBR_ENTRIES);
+		x86_pmu.lbr_nr = 0;
+	}
+
 	perf_events_lapic_init();
 	register_nmi_handler(NMI_LOCAL, perf_event_nmi_handler, 0, "PMI");
 
Signature · 1 line
2.34.1
#2 lore raw

Re: [Patch v6 01/20] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES

Sashiko has reviewed this patch and found no issues.
Quoted 4 lines

Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#3 lore raw

PATCH v6 2/20 perf/x86/intel: Validate LBR XSAVE state before updating LBR stack

In reply to Dapeng Mi

intel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.

Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.

Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")
---
 arch/x86/events/intel/lbr.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/events/intel/lbr.c +9 -1
index 0b446d8912e4..a2e4ba7e1a4f 100644
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
+	cpuc->lbr_stack.nr = 0;
+
 	if (!x86_pmu.lbr_nr)
 		return;
 
@@ -1010,9 +1012,15 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc)
 		intel_pmu_store_lbr(cpuc, NULL);
 		return;
 	}
+	xsave->header.xfeatures = 0;
 	xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
 
-	intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+	if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
+		intel_pmu_store_lbr(cpuc, xsave->lbr.entries);
+	} else {
+		/* Clear the LBR stack to avoid leaking stale entries. */
+		cpuc->lbr_stack.nr = 0;
+	}
 }
 
 void intel_pmu_lbr_read(void)
Signature · 1 line
2.34.1
#4 lore raw

Re: [Patch v6 02/20] perf/x86/intel: Validate LBR XSAVE state before updating LBR stack

Sashiko has reviewed this patch and found no issues.
Quoted 4 lines

intel_pmu_arch_lbr_read_xsave() doesn't check if the LBR data is present in the xsave area. If LBR data is in the INIT state, the LBR data may not be really written into the xsave area.

Quoted 8 lines

Add an explicit LBR XSAVE state presence check before updating the cached LBR stack. Also clear the cached LBR stack when intel_pmu_lbr_reset() is called, in addition to resetting the hardware LBR register file, to prevent stale LBR data from being reused after a context switch or similar transition.

Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#5 lore raw

PATCH v6 3/20 perf/x86/amd/lbr: Limit branch snapshots to valid entries

In reply to Dapeng Mi

From: Sandipan Das <[email protected]>

The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.

Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
---
 arch/x86/events/amd/core.c | 2 +-
 arch/x86/events/amd/lbr.c  | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/events/amd/core.c +1 -1
index 49b6b8fce566..bf631f60757c 100644
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 	cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	amd_pmu_lbr_read();
-	cnt = min(cnt, x86_pmu.lbr_nr);
+	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
 	amd_pmu_v2_enable_all(0);
arch/x86/events/amd/lbr.c +3 -1
index a55646fcb846..6c93b8042c73 100644
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)
 	struct branch_entry entry;
 	int out = 0, idx, i;
 
-	if (!cpuc->lbr_users)
+	if (!cpuc->lbr_users) {
+		cpuc->lbr_stack.nr = 0;
 		return;
+	}
 
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		entry.from.full	= amd_pmu_lbr_get_from(i);
Signature · 1 line
2.34.1
#6 lore raw

Re: [Patch v6 03/20] perf/x86/amd/lbr: Limit branch snapshots to valid entries

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

From: Sandipan Das <[email protected]>

The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.

Quoted 6 lines
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#7 lore raw

PATCH v6 4/20 perf/x86/intel/lbr: Limit branch snapshots to valid entries

In reply to Dapeng Mi

The branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.

For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
---
 arch/x86/events/intel/core.c | 2 +-
 arch/x86/events/intel/lbr.c  | 9 +++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +1 -1
index 0a34d674df59..5dfb9e9156e8 100644
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	intel_pmu_lbr_read();
-	cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
+	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 	intel_pmu_enable_all(0);
arch/x86/events/intel/lbr.c +7 -2
index a2e4ba7e1a4f..1ae853f7ac1a 100644
@@ -1027,14 +1027,19 @@ void intel_pmu_lbr_read(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
+	if (!cpuc->lbr_users || vlbr_exclude_host()) {
+		/* Reset lbr_stack.nr to prevent stale data leakage. */
+		cpuc->lbr_stack.nr = 0;
+		return;
+	}
+
 	/*
 	 * Don't read when all LBRs users are using adaptive PEBS.
 	 *
 	 * This could be smarter and actually check the event,
 	 * but this simple approach seems to work for now.
 	 */
-	if (!cpuc->lbr_users || vlbr_exclude_host() ||
-	    cpuc->lbr_users == cpuc->lbr_pebs_users)
+	if (cpuc->lbr_users == cpuc->lbr_pebs_users)
 		return;
 
 	x86_pmu.lbr_read(cpuc);
Signature · 1 line
2.34.1
#8 lore raw

Re: [Patch v6 04/20] perf/x86/intel/lbr: Limit branch snapshots to valid entries

Sashiko has reviewed this patch and found no issues.
Quoted 14 lines

The branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.

Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.

For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.

Quoted 6 lines
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author
#9 lore raw

PATCH v6 5/20 perf/x86: Add lbr_enabled to track LBR state

In reply to Dapeng Mi

Add a lbr_enabled flag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR in perf_snapshot_branch_stack() without checking whether LBR has already been disabled.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
---
 arch/x86/events/amd/lbr.c    | 2 ++
 arch/x86/events/intel/core.c | 5 +++++
 arch/x86/events/intel/lbr.c  | 2 ++
 arch/x86/events/perf_event.h | 9 +++++++++
 4 files changed, 18 insertions(+)
arch/x86/events/amd/lbr.c +2 -0
index 6c93b8042c73..400c94ecded0 100644
@@ -399,6 +399,8 @@ void amd_pmu_lbr_enable_all(void)
 	if (!cpuc->lbr_users || !x86_pmu.lbr_nr)
 		return;
 
+	cpuc->lbr_enabled = 1;
+
 	/* Set hardware branch filter */
 	if (cpuc->lbr_select) {
 		lbr_select = cpuc->lbr_sel->config & LBR_SELECT_MASK;
arch/x86/events/intel/core.c +5 -0
index 5dfb9e9156e8..e99b311c7b8d 100644
@@ -4001,6 +4001,11 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	 * It needs to be restored when leaving the handler.
 	 */
 	pmu_enabled = cpuc->enabled;
+
+	/* If pmu version < 4, LBR freeze disables LBR. */
+	if (x86_pmu.version < 4)
+		cpuc->lbr_enabled = 0;
+
 	/*
 	 * In general, the early ACK is only applied for old platforms.
 	 * For the big core starts from Haswell, the late ACK should be
arch/x86/events/intel/lbr.c +2 -0
index 1ae853f7ac1a..ee724b28d90c 100644
@@ -132,6 +132,8 @@ static void __intel_pmu_lbr_enable(bool pmi)
 	if (pmi && x86_pmu.version >= 4)
 		return;
 
+	cpuc->lbr_enabled = 1;
+
 	/*
 	 * No need to reprogram LBR_SELECT in a PMI, as it
 	 * did not change.
arch/x86/events/perf_event.h +9 -0
index e274802ef062..a4aab8130e92 100644
@@ -473,6 +473,7 @@ struct cpu_hw_events {
 	unsigned long		active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	unsigned long		dirty[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	int			enabled;
+	int			lbr_enabled;
 
 	int			n_events; /* the # of events in the below arrays */
 	int			n_added;  /* the # last events in the below arrays;
@@ -1676,8 +1677,11 @@ int amd_pmu_lbr_hw_config(struct perf_event *event);
 
 static __always_inline void __amd_pmu_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 dbg_ctl, dbg_extn_cfg;
 
+	cpuc->lbr_enabled = 0;
+
 	rdmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg);
 	wrmsrq(MSR_AMD_DBG_EXTN_CFG, dbg_extn_cfg & ~DBG_EXTN_CFG_LBRV2EN);
 
@@ -1829,13 +1833,18 @@ static __always_inline void __intel_pmu_pebs_disable_all(void)
 
 static __always_inline void __intel_pmu_arch_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+	cpuc->lbr_enabled = 0;
 	wrmsrq(MSR_ARCH_LBR_CTL, 0);
 }
 
 static __always_inline void __intel_pmu_lbr_disable(void)
 {
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 debugctl;
 
+	cpuc->lbr_enabled = 0;
 	rdmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
 	debugctl &= ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
 	wrmsrq(MSR_IA32_DEBUGCTLMSR, debugctl);
Signature · 1 line
2.34.1
#10 lore raw

Re: [Patch v6 05/20] perf/x86: Add lbr_enabled to track LBR state

Sashiko has reviewed this patch and found no issues.
Quoted 8 lines

Add a lbr_enabled flag to track whether LBR is enabled. This will be used by a subsequent change to avoid blindly re-enabling LBR in perf_snapshot_branch_stack() without checking whether LBR has already been disabled.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author
#11 lore raw

PATCH v6 6/20 perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

In reply to Dapeng Mi

amd_pmu_v2_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.

Only restore the PMU and LBR on the BPF path if they were originally enabled.

Cc Sandipan Das <[email protected]>
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")
---
 arch/x86/events/amd/core.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/events/amd/core.c +9 -3
index bf631f60757c..851d8a938f52 100644
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 {
 	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/*
 	 * The sequence of steps to freeze LBR should be completely inlined
 	 * and contain no branches to minimize contamination of LBR snapshot
 	 */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	amd_pmu_core_disable_all();
 	__amd_pmu_lbr_disable();
 
-	cpuc = this_cpu_ptr(&cpu_hw_events);
-
 	amd_pmu_lbr_read();
 	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
-	amd_pmu_v2_enable_all(0);
+	if (lbr_enabled)
+		amd_pmu_lbr_enable_all();
+	if (pmu_enabled)
+		amd_pmu_core_enable_all();
 	local_irq_restore(flags);
 
 	return cnt;
Signature · 1 line
2.34.1
#12 lore raw

Re: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set --

commit dbf85e8b6a5abeff651d80166ad0c339896c5b42
Author: Dapeng Mi <[email protected]>

perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

This commit updates amd_pmu_v2_snapshot_branch_stack() to save cpuc->enabled and cpuc->lbr_enabled before disabling the PMU and LBR, and only re-enable them if they were previously enabled.

Quoted 23 lines
arch/x86/events/amd/core.c +5 -1
index bf631f60757cc..851d8a938f522 100644
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 {
 	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/*
 	 * The sequence of steps to freeze LBR should be completely inlined
 	 * and contain no branches to minimize contamination of LBR snapshot
 	 */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	amd_pmu_core_disable_all();
 	__amd_pmu_lbr_disable();
 
-	cpuc = this_cpu_ptr(&cpu_hw_events);

arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 956

Quoted 10 lines
arch/x86/events/amd/core.c +4 -2
-
 	amd_pmu_lbr_read();
 	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
-	amd_pmu_v2_enable_all(0);
+	if (lbr_enabled)
+		amd_pmu_lbr_enable_all();
+	if (pmu_enabled)
+		amd_pmu_core_enable_all();

[Severity: Medium]
As reported by Sashiko on v5 [1], while __amd_pmu_lbr_disable() clears cpuc->lbr_enabled = 0, amd_pmu_core_disable_all() only clears the hardware global control MSR and leaves cpuc->enabled set to 1.

If a BPF program in non-NMI context calls bpf_get_branch_snapshot() -> amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save() does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack() is reading LBR MSRs or copying entries, perf_event_nmi_handler() -> amd_pmu_v2_handle_irq() will see cpuc->enabled == 1 and re-enable the core PMU counters on exit:

arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
	pmu_enabled = cpuc->enabled;
	cpuc->enabled = 0;

	/* Stop counting but do not disable LBR */
	amd_pmu_core_disable_all();
	...
done:
	cpuc->enabled = pmu_enabled;

	/* Resume counting only if PMU is active */
	if (pmu_enabled)
		amd_pmu_core_enable_all();
}

Similarly, if an NMI triggers a nested BPF call to amd_pmu_v2_snapshot_branch_stack(), the nested call will also observe cpuc->enabled == 1 and call amd_pmu_core_enable_all() before returning to the outer snapshot.

Should amd_pmu_v2_snapshot_branch_stack() clear cpuc->enabled = 0 after saving pmu_enabled and restore cpuc->enabled = pmu_enabled before re-enabling the core PMU, matching amd_pmu_v2_handle_irq()?

[1] https://lore.kernel.org/r/[email protected]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author
#13 lore raw

Re: [Patch v6 06/20] perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

Quoted 51 lines · On 11/6/2026 1:01 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/amd: amd_pmu_v2_snapshot_branch_stack() leaves cpuc->enabled set --

commit dbf85e8b6a5abeff651d80166ad0c339896c5b42
Author: Dapeng Mi <[email protected]>

perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled

This commit updates amd_pmu_v2_snapshot_branch_stack() to save cpuc->enabled and cpuc->lbr_enabled before disabling the PMU and LBR, and only re-enable them if they were previously enabled.

Quoted 23 lines
arch/x86/events/amd/core.c +5 -1
index bf631f60757cc..851d8a938f522 100644
@@ -932,22 +932,28 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 {
 	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/*
 	 * The sequence of steps to freeze LBR should be completely inlined
 	 * and contain no branches to minimize contamination of LBR snapshot
 	 */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	amd_pmu_core_disable_all();
 	__amd_pmu_lbr_disable();
 
-	cpuc = this_cpu_ptr(&cpu_hw_events);

arch/x86/events/amd/core.c static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u · line 956

Quoted 10 lines
arch/x86/events/amd/core.c +4 -2
-
 	amd_pmu_lbr_read();
 	cnt = min(cnt, cpuc->lbr_stack.nr);
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
 
-	amd_pmu_v2_enable_all(0);
+	if (lbr_enabled)
+		amd_pmu_lbr_enable_all();
+	if (pmu_enabled)
+		amd_pmu_core_enable_all();

[Severity: Medium]
As reported by Sashiko on v5 [1], while __amd_pmu_lbr_disable() clears cpuc->lbr_enabled = 0, amd_pmu_core_disable_all() only clears the hardware global control MSR and leaves cpuc->enabled set to 1.

Quoted 6 lines

If a BPF program in non-NMI context calls bpf_get_branch_snapshot() -> amd_pmu_v2_snapshot_branch_stack() while cpuc->enabled is 1, local_irq_save() does not mask NMIs. If an NMI arrives while amd_pmu_v2_snapshot_branch_stack() is reading LBR MSRs or copying entries, perf_event_nmi_handler() -> amd_pmu_v2_handle_irq() will see cpuc->enabled == 1 and re-enable the core PMU counters on exit:

This issue should never happen in reality since amd_pmu_core_disable_all() would disable PMU, so no PMI can be interrupt the BPF LBR snapshot.

Quoted 27 lines
arch/x86/events/amd/core.c:amd_pmu_v2_handle_irq() {
	pmu_enabled = cpuc->enabled;
	cpuc->enabled = 0;

	/* Stop counting but do not disable LBR */
	amd_pmu_core_disable_all();
	...
done:
	cpuc->enabled = pmu_enabled;

	/* Resume counting only if PMU is active */
	if (pmu_enabled)
		amd_pmu_core_enable_all();
}

Similarly, if an NMI triggers a nested BPF call to amd_pmu_v2_snapshot_branch_stack(), the nested call will also observe cpuc->enabled == 1 and call amd_pmu_core_enable_all() before returning to the outer snapshot.

Should amd_pmu_v2_snapshot_branch_stack() clear cpuc->enabled = 0 after saving pmu_enabled and restore cpuc->enabled = pmu_enabled before re-enabling the core PMU, matching amd_pmu_v2_handle_irq()?

[1] https://lore.kernel.org/r/[email protected]

author
#14 lore raw

PATCH v6 7/20 perf/x86/intel: Add helper to enable PMU without LBR

In reply to Dapeng Mi

Currently, __intel_pmu_enable_all() also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper, ___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.

Also remove the unused added argument from __intel_pmu_enable_all().

No functional change intended.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
---
 arch/x86/events/intel/core.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/events/intel/core.c +9 -5
index e99b311c7b8d..9d8398da1989 100644
@@ -2783,13 +2783,11 @@ static __always_inline void intel_pmu_disable_all(void)
 	intel_pmu_lbr_disable_all();
 }
 
-static void __intel_pmu_enable_all(int added, bool pmi)
+static void ___intel_pmu_enable_all(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
 
-	intel_pmu_lbr_enable_all(pmi);
-
 	if (cpuc->fixed_ctrl_val != cpuc->active_fixed_ctrl_val) {
 		wrmsrq(MSR_ARCH_PERFMON_FIXED_CTR_CTRL, cpuc->fixed_ctrl_val);
 		cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val;
@@ -2809,10 +2807,16 @@ static void __intel_pmu_enable_all(int added, bool pmi)
 	}
 }
 
+static void __intel_pmu_enable_all(bool pmi)
+{
+	intel_pmu_lbr_enable_all(pmi);
+	___intel_pmu_enable_all();
+}
+
 static void intel_pmu_enable_all(int added)
 {
 	static_call_cond(x86_pmu_pebs_enable_all)();
-	__intel_pmu_enable_all(added, false);
+	__intel_pmu_enable_all(false);
 }
 
 static noinline int
@@ -4056,7 +4060,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
 	cpuc->enabled = pmu_enabled;
 	if (pmu_enabled)
-		__intel_pmu_enable_all(0, true);
+		__intel_pmu_enable_all(true);
 	intel_bts_enable_local();
 
 	/*
Signature · 1 line
2.34.1
#15 lore raw

Re: [Patch v6 07/20] perf/x86/intel: Add helper to enable PMU without LBR

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

Currently, __intel_pmu_enable_all() also re-enables LBR in addition to the PMU. Split out the PMU-only path into a new helper, ___intel_pmu_enable_all(), so the PMU can be enabled independently. This helper will be used by a subsequent patch.

Also remove the unused added argument from __intel_pmu_enable_all().

No functional change intended.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author
#16 lore raw

PATCH v6 8/20 perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

In reply to Dapeng Mi

__intel_pmu_snapshot_branch_stack() currently re-enables the PMU and LBR unconditionally even when they were already disabled in PMI context. This can cause events such as cycles and instructions to be counted while the PMI handler is running, which is not intended.

Only restore the PMU and LBR on the BPF path if they were originally enabled.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
---
 arch/x86/events/intel/core.c | 38 ++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +34 -4
index 9d8398da1989..739c454827ac 100644
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
 
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
-				  unsigned int cnt, unsigned long flags)
+				  unsigned int cnt, unsigned long flags,
+				  int pmu_enabled, int lbr_enabled)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
-	intel_pmu_enable_all(0);
+
+	/*
+	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
+	 * This path may be entered from the PMI handler through
+	 * bpf_overflow_handler().
+	 *
+	 * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
+	 * LBR can't be re-enabled here.
+	 * If PMU version >= 4, LBR is only frozen instead of disabled,
+	 * LBR should be re-enabled here.
+	 */
+	if (lbr_enabled)
+		intel_pmu_lbr_enable_all(false);
+	if (pmu_enabled)
+		___intel_pmu_enable_all();
+
 	local_irq_restore(flags);
 	return cnt;
 }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 static int
 intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 static int
 intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_arch_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 /*
Signature · 1 line
2.34.1
#17 lore raw

Re: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack() --

commit f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
Author: Dapeng Mi <[email protected]>

perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

This commit updates __intel_pmu_snapshot_branch_stack() and its callers to save the PMU and LBR enabled state and only re-enable them after taking a branch snapshot if they were originally enabled.

Quoted 70 lines
arch/x86/events/intel/core.c +29 -3
index 9d8398da1989f..739c454827ac5 100644
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
 
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
-				  unsigned int cnt, unsigned long flags)
+				  unsigned int cnt, unsigned long flags,
+				  int pmu_enabled, int lbr_enabled)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
-	intel_pmu_enable_all(0);
+
+	/*
+	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
+	 * This path may be entered from the PMI handler through
+	 * bpf_overflow_handler().
+	 *
+	 * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
+	 * LBR can't be re-enabled here.
+	 * If PMU version >= 4, LBR is only frozen instead of disabled,
+	 * LBR should be re-enabled here.
+	 */
+	if (lbr_enabled)
+		intel_pmu_lbr_enable_all(false);
+	if (pmu_enabled)
+		___intel_pmu_enable_all();
+
 	local_irq_restore(flags);
 	return cnt;
 }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 static int
 intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 static int
 intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);

arch/x86/events/intel/core.c __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, · line 2891

Quoted 10 lines
arch/x86/events/intel/core.c +5 -1
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_arch_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }

[Severity: Medium]
Should intel_pmu_snapshot_branch_stack() and intel_pmu_snapshot_arch_branch_stack() also clear cpuc->enabled = 0 before calling __intel_pmu_disable_all(false), and restore cpuc->enabled = pmu_enabled in __intel_pmu_snapshot_branch_stack()?

While __intel_pmu_lbr_disable() and __intel_pmu_arch_lbr_disable() clear cpuc->lbr_enabled = 0, __intel_pmu_disable_all(false) only writes 0 to MSR_CORE_PERF_GLOBAL_CTRL without updating cpuc->enabled.

When bpf_get_branch_snapshot() is invoked outside PMI context (where cpuc->enabled is 1), local_irq_save() does not mask NMIs. If an in-flight PMI or another NMI arrives while __intel_pmu_snapshot_branch_stack() is reading the LBRs, perf_event_nmi_handler() -> intel_pmu_handle_irq() observes cpuc->enabled == 1 and re-enables the PMU (and LBR on PMU version < 4) before returning:

arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
	pmu_enabled = cpuc->enabled;
	...
done:
	if (mid_ack)
		apic_write(APIC_LVTPC, APIC_DM_NMI);
	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
	cpuc->enabled = pmu_enabled;
	if (pmu_enabled)
		__intel_pmu_enable_all(true);
}

Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program during __intel_pmu_snapshot_branch_stack() would see cpuc->enabled == 1 and prematurely call ___intel_pmu_enable_all() before the outer snapshot finishes.

Setting cpuc->enabled = 0 is a plain store with no branches, so could it be cleared alongside saving pmu_enabled and restored in __intel_pmu_snapshot_branch_stack() to match x86_pmu_disable(), __intel_pmu_quiesce(), and intel_pmu_handle_irq()?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author
#18 lore raw

Re: [Patch v6 08/20] perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

Quoted 103 lines · On 11/6/2026 1:06 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86/intel: cpuc->enabled not cleared in intel_pmu_snapshot_branch_stack() --

commit f8f32e2d56da9a93e561d91b6c3fb89e787c5d9e
Author: Dapeng Mi <[email protected]>

perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled

This commit updates __intel_pmu_snapshot_branch_stack() and its callers to save the PMU and LBR enabled state and only re-enable them after taking a branch snapshot if they were originally enabled.

Quoted 70 lines
arch/x86/events/intel/core.c +29 -3
index 9d8398da1989f..739c454827ac5 100644
@@ -2821,7 +2821,8 @@ static void intel_pmu_enable_all(int added)
 
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
-				  unsigned int cnt, unsigned long flags)
+				  unsigned int cnt, unsigned long flags,
+				  int pmu_enabled, int lbr_enabled)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -2829,7 +2830,22 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
 
 	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
-	intel_pmu_enable_all(0);
+
+	/*
+	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
+	 * This path may be entered from the PMI handler through
+	 * bpf_overflow_handler().
+	 *
+	 * If PMU version < 4, LBR is disabled by the LBR freeze mechanism,
+	 * LBR can't be re-enabled here.
+	 * If PMU version >= 4, LBR is only frozen instead of disabled,
+	 * LBR should be re-enabled here.
+	 */
+	if (lbr_enabled)
+		intel_pmu_lbr_enable_all(false);
+	if (pmu_enabled)
+		___intel_pmu_enable_all();
+
 	local_irq_restore(flags);
 	return cnt;
 }
@@ -2837,27 +2853,41 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 static int
 intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }
 
 static int
 intel_pmu_snapshot_arch_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
 {
+	struct cpu_hw_events *cpuc;
 	unsigned long flags;
+	int pmu_enabled;
+	int lbr_enabled;
 
 	/* must not have branches... */
 	local_irq_save(flags);

arch/x86/events/intel/core.c __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries, · line 2891

Quoted 10 lines
arch/x86/events/intel/core.c +5 -1
+	cpuc = this_cpu_ptr(&cpu_hw_events);
+	pmu_enabled = cpuc->enabled;
+	lbr_enabled = cpuc->lbr_enabled;
 	__intel_pmu_disable_all(false); /* we don't care about BTS */
 	__intel_pmu_arch_lbr_disable();
 	/*            ... until here */
-	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
+	return __intel_pmu_snapshot_branch_stack(entries, cnt, flags,
+						 pmu_enabled, lbr_enabled);
 }

[Severity: Medium]
Should intel_pmu_snapshot_branch_stack() and intel_pmu_snapshot_arch_branch_stack() also clear cpuc->enabled = 0 before calling __intel_pmu_disable_all(false), and restore cpuc->enabled = pmu_enabled in __intel_pmu_snapshot_branch_stack()?

While __intel_pmu_lbr_disable() and __intel_pmu_arch_lbr_disable() clear cpuc->lbr_enabled = 0, __intel_pmu_disable_all(false) only writes 0 to MSR_CORE_PERF_GLOBAL_CTRL without updating cpuc->enabled.

Quoted 6 lines

When bpf_get_branch_snapshot() is invoked outside PMI context (where cpuc->enabled is 1), local_irq_save() does not mask NMIs. If an in-flight PMI or another NMI arrives while __intel_pmu_snapshot_branch_stack() is reading the LBRs, perf_event_nmi_handler() -> intel_pmu_handle_irq() observes cpuc->enabled == 1 and re-enables the PMU (and LBR on PMU version < 4) before returning:

Same with AMD. This issue should never happen in reality since __intel_pmu_disable_all() would disable PMU, so no PMI can be interrupt the BPF LBR snapshot.

Quoted 23 lines
arch/x86/events/intel/core.c:intel_pmu_handle_irq() {
	pmu_enabled = cpuc->enabled;
	...
done:
	if (mid_ack)
		apic_write(APIC_LVTPC, APIC_DM_NMI);
	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
	cpuc->enabled = pmu_enabled;
	if (pmu_enabled)
		__intel_pmu_enable_all(true);
}

Similarly, a nested snapshot from an NMI or fentry/kprobe BPF program during __intel_pmu_snapshot_branch_stack() would see cpuc->enabled == 1 and prematurely call ___intel_pmu_enable_all() before the outer snapshot finishes.

Setting cpuc->enabled = 0 is a plain store with no branches, so could it be cleared alongside saving pmu_enabled and restored in __intel_pmu_snapshot_branch_stack() to match x86_pmu_disable(), __intel_pmu_quiesce(), and intel_pmu_handle_irq()?

author
#19 lore raw

PATCH v6 9/20 perf/x86/intel: Keep LBR frozen throughout PMI handling

In reply to Dapeng Mi

For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.

This can pollute or overwrite the intended branch snapshot, especially when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.

Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.

Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
---
 arch/x86/events/intel/core.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +11 -2
index 739c454827ac..ab6261ab54b6 100644
@@ -4062,7 +4062,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	loops = 0;
 again:
 	intel_pmu_lbr_read();
-	intel_pmu_ack_status(status);
+
+	/*
+	 * Do not clear the LBR frozen status here; keep LBR frozen
+	 * throughout PMI handling.
+	 */
+	intel_pmu_ack_status(status & ~GLOBAL_STATUS_LBRS_FROZEN);
+
 	if (++loops > 100) {
 		static bool warned;
 
@@ -4081,10 +4087,13 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	 * Repeat if there is more work to be done:
 	 */
 	status = intel_pmu_get_status();
-	if (status)
+	if (status & ~GLOBAL_STATUS_LBRS_FROZEN)
 		goto again;
 
 done:
+	if (status & GLOBAL_STATUS_LBRS_FROZEN)
+		intel_pmu_ack_status(status & GLOBAL_STATUS_LBRS_FROZEN);
+
 	if (mid_ack)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
 	/* Only restore PMU state when it's active. See x86_pmu_disable(). */
Signature · 1 line
2.34.1
#20 lore raw

Re: [Patch v6 09/20] perf/x86/intel: Keep LBR frozen throughout PMI handling

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

For the platforms which has 4+ PMU version, streamline LBR freeze is supported. The PERF_GLOBAL_STATUS.LBR_FRZ bit is set to freeze LBR once a PMU counter overflow. However, the current PMI handler clears the LBR freeze bit too early, allowing branch operations that occur during PMI handling to be recorded into the LBR stack.

This can pollute or overwrite the intended branch snapshot, especially when bpf_get_branch_snapshot() is invoked from bpf_overflow_handler(). In that case, branches recorded while processing the PMI are mixed into the original LBR data.

Quoted 8 lines

Defer unfreezing until after PMI handling completes so the LBR remains frozen for the entire PMI path and no branch activity from the handler is recorded.

Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author
#21 lore raw

PATCH v6 10/20 perf: Add counted-bounds tracking for perf_branch_stack

In reply to Dapeng Mi

struct perf_branch_stack uses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member in cpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.

Annotate entries[] with __counted_by(nr) so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR stack backing storage to a complete __lbr_stack[] array and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack* view for x86 and powerpc.

Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.

No functional change intended.

Cc Sandipan Das <[email protected]>
Cc Madhavan Srinivasan <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc "Christophe Leroy (CS GROUP)" <[email protected]>
Cc Will Deacon <[email protected]>
Cc Mark Rutland <[email protected]>
---
 arch/powerpc/perf/core-book3s.c | 43 +++++++++++-----
 arch/x86/events/amd/brs.c       |  7 +--
 arch/x86/events/amd/core.c      |  6 +--
 arch/x86/events/amd/lbr.c       | 44 +++++++++-------
 arch/x86/events/core.c          |  2 +-
 arch/x86/events/intel/core.c    |  5 +-
 arch/x86/events/intel/ds.c      | 12 +++--
 arch/x86/events/intel/lbr.c     | 89 +++++++++++++++++----------------
 arch/x86/events/perf_event.h    | 19 ++++++-
 drivers/perf/arm_brbe.c         |  5 ++
 include/linux/perf_event.h      |  2 +-
 11 files changed, 144 insertions(+), 90 deletions(-)
arch/powerpc/perf/core-book3s.c +30 -13
index 10ded865583c..978fd39b586b 100644
@@ -30,6 +30,8 @@
 #define BHRB_PREDICTION		0x0000000000000001
 #define BHRB_EA			0xFFFFFFFFFFFFFFFCUL
 
+#define LBR_STACK_SIZE		struct_size_t(struct perf_branch_stack, entries, BHRB_MAX_ENTRIES)
+
 struct cpu_hw_events {
 	int n_events;
 	int n_percpu;
@@ -54,8 +56,7 @@ struct cpu_hw_events {
 	u64				bhrb_filter;	/* BHRB HW branch filter */
 	unsigned int			bhrb_users;
 	void				*bhrb_context;
-	struct	perf_branch_stack	bhrb_stack;
-	struct	perf_branch_entry	bhrb_entries[BHRB_MAX_ENTRIES];
+	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				ic_init;
 
 	/* Store the PMC values */
@@ -177,6 +178,20 @@ static bool regs_use_siar(struct pt_regs *regs)
 	return ((TRAP(regs) == INTERRUPT_PERFMON) && regs->result);
 }
 
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+	return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+	/*
+	 * Initialize the branch-stack count before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	cpu_lbr_stack(cpuc)->nr = nr;
+}
+
 /*
  * Things that are specific to 64-bit implementations.
  */
@@ -493,6 +508,8 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 
 	r_index = 0;
 	u_index = 0;
+
+	cpu_init_lbr_stack(cpuhw, ppmu->bhrb_nr);
 	while (r_index < ppmu->bhrb_nr) {
 		/* Assembly read function */
 		val = read_bhrb(r_index++);
@@ -541,9 +558,9 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 				/* Target branches use two entries
 				 * (ie. computed gotos/XL form)
 				 */
-				cpuhw->bhrb_entries[u_index].to = addr;
-				cpuhw->bhrb_entries[u_index].mispred = pred;
-				cpuhw->bhrb_entries[u_index].predicted = ~pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].to = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
 
 				/* Get from address in next entry */
 				val = read_bhrb(r_index++);
@@ -554,22 +571,22 @@ static void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *
 					r_index--;
 					addr = 0;
 				}
-				cpuhw->bhrb_entries[u_index].from = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
 			} else {
 				/* Branches to immediate field 
 				   (ie I or B form) */
-				cpuhw->bhrb_entries[u_index].from = addr;
-				cpuhw->bhrb_entries[u_index].to =
+				cpu_lbr_stack(cpuhw)->entries[u_index].from = addr;
+				cpu_lbr_stack(cpuhw)->entries[u_index].to =
 					power_pmu_bhrb_to(addr);
-				cpuhw->bhrb_entries[u_index].mispred = pred;
-				cpuhw->bhrb_entries[u_index].predicted = ~pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].mispred = pred;
+				cpu_lbr_stack(cpuhw)->entries[u_index].predicted = ~pred;
 			}
 			u_index++;
 
 		}
 	}
-	cpuhw->bhrb_stack.nr = u_index;
-	cpuhw->bhrb_stack.hw_idx = -1ULL;
+	cpu_lbr_stack(cpuhw)->nr = u_index;
+	cpu_lbr_stack(cpuhw)->hw_idx = -1ULL;
 	return;
 }
 
@@ -2332,7 +2349,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
 			struct cpu_hw_events *cpuhw;
 			cpuhw = this_cpu_ptr(&cpu_hw_events);
 			power_pmu_bhrb_read(event, cpuhw);
-			perf_sample_save_brstack(&data, event, &cpuhw->bhrb_stack, NULL);
+			perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuhw), NULL);
 		}
 
 		if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
arch/x86/events/amd/brs.c +4 -3
index 54b13faba116..1906d1257646 100644
@@ -282,7 +282,6 @@ void amd_brs_drain(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	struct perf_event *event = cpuc->events[0];
-	struct perf_branch_entry *br = cpuc->lbr_entries;
 	union amd_debug_extn_cfg cfg;
 	u32 i, nr = 0, num, tos, start;
 	u32 shift = 64 - boot_cpu_data.x86_virt_bits;
@@ -317,6 +316,8 @@ void amd_brs_drain(void)
 
 	num = tos - start + 1;
 
+	cpu_init_lbr_stack(cpuc, num);
+
 	/*
 	 * BRS is only one pass (saturation) from MSROFF to depth-1
 	 * MSROFF wraps to zero when buffer is full
@@ -343,7 +344,7 @@ void amd_brs_drain(void)
 		if (!amd_brs_match_plm(event, from, to))
 			continue;
 
-		br[nr] = (struct perf_branch_entry){
+		cpu_lbr_stack(cpuc)->entries[nr] = (struct perf_branch_entry){
 			.from	= from,
 			.to	= to,
 		};
@@ -351,7 +352,7 @@ void amd_brs_drain(void)
 	}
 empty:
 	/* Record number of sampled branches */
-	cpuc->lbr_stack.nr = nr;
+	cpu_lbr_stack(cpuc)->nr = nr;
 }
 
 /*
arch/x86/events/amd/core.c +3 -3
index 851d8a938f52..160102a46b18 100644
@@ -947,8 +947,8 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u
 	__amd_pmu_lbr_disable();
 
 	amd_pmu_lbr_read();
-	cnt = min(cnt, cpuc->lbr_stack.nr);
-	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+	cnt = min(cnt, cpu_lbr_stack(cpuc)->nr);
+	memcpy(entries, cpu_lbr_stack(cpuc)->entries, sizeof(struct perf_branch_entry) * cnt);
 
 	if (lbr_enabled)
 		amd_pmu_lbr_enable_all();
@@ -1020,7 +1020,7 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
 		if (!x86_perf_event_set_period(event))
 			continue;
 
-		perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+		perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
 
 		perf_event_overflow(event, &data, regs);
 	}
arch/x86/events/amd/lbr.c +26 -18

113 lines not on this page · raw message

arch/x86/events/core.c +1 -1
index 82d9b0e5ad1c..a7bdc6f370ec 100644
@@ -2197,7 +2197,7 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		perf_sample_save_brstack(&data, event, &cpuc->lbr_stack, NULL);
+		perf_sample_save_brstack(&data, event, cpu_lbr_stack(cpuc), NULL);
 
 		perf_event_overflow(event, &data, regs);
 	}
arch/x86/events/intel/core.c +3 -2
index ab6261ab54b6..42d4d525a729 100644
@@ -2827,9 +2827,10 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
 	intel_pmu_lbr_read();
-	cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);
+	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
 
-	memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
+	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+	       sizeof(struct perf_branch_entry) * cnt);
 
 	/*
 	 * Do not blindly call intel_pmu_enable_all() to re-enable LBR.
arch/x86/events/intel/ds.c +8 -4
index 8444670cee4a..332754e1da4f 100644
@@ -2010,8 +2010,7 @@ void intel_pmu_pebs_disable_all(void)
 static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-	unsigned long from = cpuc->lbr_entries[0].from;
-	unsigned long old_to, to = cpuc->lbr_entries[0].to;
+	unsigned long from, old_to, to;
 	unsigned long ip = regs->ip;
 	int is_64bit = 0;
 	void *kaddr;
@@ -2026,7 +2025,12 @@ static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
 	/*
 	 * No LBR entry, no basic block, no rewinding
 	 */
-	if (!cpuc->lbr_stack.nr || !from || !to)
+	if (!cpu_lbr_stack(cpuc)->nr)
+		return 0;
+
+	from = cpu_lbr_stack(cpuc)->entries[0].from;
+	to = cpu_lbr_stack(cpuc)->entries[0].to;
+	if (!from || !to)
 		return 0;
 
 	/*
@@ -2318,7 +2322,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
 	if (x86_pmu.intel_cap.pebs_format >= 3)
 		setup_pebs_time(event, data, pebs->tsc);
 
-	perf_sample_save_brstack(data, event, &cpuc->lbr_stack, NULL);
+	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
 static void adaptive_pebs_save_regs(struct pt_regs *regs,
arch/x86/events/intel/lbr.c +46 -43

255 lines not on this page · raw message

arch/x86/events/perf_event.h +17 -2
index a4aab8130e92..e3d625a6d7eb 100644
@@ -447,6 +447,8 @@ struct intel_excl_cntrs {
 struct x86_perf_task_context;
 #define MAX_LBR_ENTRIES		32
 
+#define LBR_STACK_SIZE		struct_size_t(struct perf_branch_stack, entries, MAX_LBR_ENTRIES)
+
 enum {
 	LBR_FORMAT_32		= 0x00,
 	LBR_FORMAT_LIP		= 0x01,
@@ -525,8 +527,7 @@ struct cpu_hw_events {
 	 */
 	int				lbr_users;
 	int				lbr_pebs_users;
-	struct perf_branch_stack	lbr_stack;
-	struct perf_branch_entry	lbr_entries[MAX_LBR_ENTRIES];
+	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
 	union {
 		struct er_account		*lbr_sel;
@@ -588,6 +589,20 @@ struct cpu_hw_events {
 	struct pmu			*pmu;
 };
 
+static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_events *cpuc)
+{
+	return (struct perf_branch_stack *)cpuc->__lbr_stack;
+}
+
+static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+{
+	/*
+	 * Initialize the branch-stack count before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	cpu_lbr_stack(cpuc)->nr = nr;
+}
+
 #define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) {	\
 	{ .idxmsk64 = (n) },		\
 	.code = (c),			\
drivers/perf/arm_brbe.c +5 -0
index 073d14584741..e2b32b239de1 100644
@@ -763,6 +763,11 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 
 	prepare_event_branch_type_mask(branch_sample_type, event_type_mask);
 
+	/*
+	 * Initialize the branch_stack->nr before populating entries[]
+	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
+	 */
+	branch_stack->nr = nr_hw;
 	for (int bank = 0; bank < nr_banks; bank++) {
 		int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
 		int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
include/linux/perf_event.h +1 -1
index eb3f7bab498c..19de3f1cbd6e 100644
@@ -115,7 +115,7 @@ static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag)
 struct perf_branch_stack {
 	u64				nr;
 	u64				hw_idx;
-	struct perf_branch_entry	entries[];
+	struct perf_branch_entry	entries[] __counted_by(nr);
 };
 
 struct task_struct;
Signature · 1 line
2.34.1
#22 lore raw

Re: [Patch v6 10/20] perf: Add counted-bounds tracking for perf_branch_stack

Sashiko has reviewed this patch and found no issues.
Quoted 19 lines

struct perf_branch_stack uses a trailing flexible array member, entries[], whose valid length is tracked by nr. When this structure is embedded as a value member in cpu_hw_events, the compiler can treat the trailing flexible array as having a fixed zero-length layout under -fstrict-flex-arrays=3 and UBSan, which can lead to strict array-bounds diagnostics when entries[] is indexed.

Annotate entries[] with __counted_by(nr) so the compiler and sanitizer track the array length from the nr field. Because a structure containing a counted flexible array cannot be embedded as a value member, switch the per-CPU LBR stack backing storage to a complete __lbr_stack[] array and add a helper cpu_lbr_stack() to recover the struct perf_branch_stack* view for x86 and powerpc.

Also initialize the branch-stack count before populating entries[] to avoid false-positive bounds checks while the stack is being filled.

No functional change intended.

Quoted 10 lines
Cc Sandipan Das <[email protected]>
Cc Madhavan Srinivasan <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc "Christophe Leroy (CS GROUP)" <[email protected]>
Cc Will Deacon <[email protected]>
Cc Mark Rutland <[email protected]>

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author
#23 lore raw

PATCH v6 11/20 perf/x86/intel: Allow LBR read helpers to operate on explicit stacks

In reply to Dapeng Mi

The LBR read helpers currently assume the default stack, cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the *_lbr_read() helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.

Update the related callers, including intel_pmu_store_lbr() and intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.

No function change intended.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
---
 arch/x86/events/amd/brs.c    |  2 +-
 arch/x86/events/amd/lbr.c    |  2 +-
 arch/x86/events/intel/core.c |  4 +-
 arch/x86/events/intel/lbr.c  | 89 ++++++++++++++++++++----------------
 arch/x86/events/perf_event.h | 16 ++++---
 5 files changed, 64 insertions(+), 49 deletions(-)
arch/x86/events/amd/brs.c +1 -1
index 1906d1257646..475e3699d3ca 100644
@@ -316,7 +316,7 @@ void amd_brs_drain(void)
 
 	num = tos - start + 1;
 
-	cpu_init_lbr_stack(cpuc, num);
+	cpu_init_lbr_stack(cpu_lbr_stack(cpuc), num);
 
 	/*
 	 * BRS is only one pass (saturation) from MSROFF to depth-1
arch/x86/events/amd/lbr.c +1 -1
index 3e4e5e984649..95898a1d9901 100644
@@ -179,7 +179,7 @@ void amd_pmu_lbr_read(void)
 		return;
 	}
 
-	cpu_init_lbr_stack(cpuc, x86_pmu.lbr_nr);
+	cpu_init_lbr_stack(cpu_lbr_stack(cpuc), x86_pmu.lbr_nr);
 
 	for (i = 0; i < x86_pmu.lbr_nr; i++) {
 		entry.from.full	= amd_pmu_lbr_get_from(i);
arch/x86/events/intel/core.c +2 -2
index 42d4d525a729..b7045c05b736 100644
@@ -2826,7 +2826,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
-	intel_pmu_lbr_read();
+	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
 
 	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
@@ -4062,7 +4062,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	intel_pmu_lbr_read();
+	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +50 -39

234 lines not on this page · raw message

arch/x86/events/perf_event.h +10 -6
index e3d625a6d7eb..fa0af528cd1e 100644
@@ -594,13 +594,13 @@ static __always_inline struct perf_branch_stack *cpu_lbr_stack(struct cpu_hw_eve
 	return (struct perf_branch_stack *)cpuc->__lbr_stack;
 }
 
-static __always_inline void cpu_init_lbr_stack(struct cpu_hw_events *cpuc, int nr)
+static __always_inline void cpu_init_lbr_stack(struct perf_branch_stack *lbr_stack, int nr)
 {
 	/*
 	 * Initialize the branch-stack count before populating entries[]
 	 * to avoid UBSAN complaints from the __counted_by(nr) flexible array.
 	 */
-	cpu_lbr_stack(cpuc)->nr = nr;
+	lbr_stack->nr = nr;
 }
 
 #define __EVENT_CONSTRAINT_RANGE(c, e, n, m, w, o, f) {	\
@@ -1207,7 +1207,8 @@ struct x86_pmu {
 	unsigned int	lbr_counters:4;
 
 	void		(*lbr_reset)(void);
-	void		(*lbr_read)(struct cpu_hw_events *cpuc);
+	void		(*lbr_read)(struct cpu_hw_events *cpuc,
+				    struct perf_branch_stack *lbr_stack);
 	void		(*lbr_save)(void *ctx);
 	void		(*lbr_restore)(void *ctx);
 
@@ -2007,11 +2008,14 @@ void intel_pmu_lbr_enable_all(bool pmi);
 
 void intel_pmu_lbr_disable_all(void);
 
-void intel_pmu_lbr_read(void);
+void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
+			u64 *lbr_counters);
 
-void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack);
 
-void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc);
+void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc,
+			   struct perf_branch_stack *lbr_stack);
 
 void intel_pmu_lbr_save(void *ctx);
 
Signature · 1 line
2.34.1
#24 lore raw

Re: [Patch v6 11/20] perf/x86/intel: Allow LBR read helpers to operate on explicit stacks

Sashiko has reviewed this patch and found no issues.
Quoted 6 lines

The LBR read helpers currently assume the default stack, cpuc->lbr_stack, and therefore cannot be reused for other LBR stacks that may be in use. Extend the *_lbr_read() helpers to take an explicit LBR stack argument, so they can operate on any LBR stack instead of only the default one.

Quoted 7 lines

Update the related callers, including intel_pmu_store_lbr() and intel_pmu_lbr_filter(), to pass the appropriate stack explicitly.

No function change intended.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author
#25 lore raw

PATCH v6 12/20 perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting

In reply to Dapeng Mi

BPF LBR snapshotting currently reuses the default LBR stack (cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context via bpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.

Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.

Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")
---
 arch/x86/events/intel/core.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
arch/x86/events/intel/core.c +12 -4
index b7045c05b736..a3aaa98108cd 100644
@@ -2819,17 +2819,25 @@ static void intel_pmu_enable_all(int added)
 	__intel_pmu_enable_all(false);
 }
 
+struct lbr_snapshot {
+	u64 __lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
+	u64 counters[MAX_LBR_ENTRIES];
+};
+static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);
+
 static noinline int
 __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 				  unsigned int cnt, unsigned long flags,
 				  int pmu_enabled, int lbr_enabled)
 {
-	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	struct lbr_snapshot *bpf = this_cpu_ptr(&lbr_snapshot);
+	struct perf_branch_stack *lbr_stack =
+		(struct perf_branch_stack *)bpf->__lbr_stack;
 
-	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
-	cnt = min_t(unsigned int, cnt, cpu_lbr_stack(cpuc)->nr);
+	intel_pmu_lbr_read(lbr_stack, bpf->counters);
+	cnt = min_t(unsigned int, cnt, lbr_stack->nr);
 
-	memcpy(entries, cpu_lbr_stack(cpuc)->entries,
+	memcpy(entries, lbr_stack->entries,
 	       sizeof(struct perf_branch_entry) * cnt);
 
 	/*
Signature · 1 line
2.34.1
#26 lore raw

Re: [Patch v6 12/20] perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting

Sashiko has reviewed this patch and found no issues.
Quoted 8 lines

BPF LBR snapshotting currently reuses the default LBR stack (cpuc->__lbr_stack) when storing sampled data. This is unsafe when the BPF helper runs in PMI context via bpf_overflow_handler(): it can overwrite the cached LBR state for active PEBS LBR events, causing incorrect LBR data to be reported to userspace. When the PEBS and BPF stacks have different depths, this can also lead to out-of-bounds reads when the LBR stack is later exported.

Quoted 6 lines

Use a dedicated LBR stack for BPF snapshotting so the BPF path no longer clobbers the shared LBR state used by other perf events.

Fixes 425507fa5f32 ("perf/x86/intel/lbr: Optimize v4 LBR unfreezing")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author
#27 lore raw

PATCH v6 13/20 perf/x86/intel: Always snapshot live hardware LBR state for BPF

In reply to Dapeng Mi

__intel_pmu_snapshot_branch_stack() is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.

BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")
---
 arch/x86/events/intel/core.c | 7 ++++++-
 arch/x86/events/intel/lbr.c  | 9 ---------
 2 files changed, 6 insertions(+), 10 deletions(-)
arch/x86/events/intel/core.c +6 -1
index a3aaa98108cd..fbde874af618 100644
@@ -4070,7 +4070,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+	/*
+	 * No need to read the LBR stack when all LBR users are PEBS events,
+	 * since the PEBS handler updates it.
+	 */
+	if (cpuc->lbr_users != cpuc->lbr_pebs_users)
+		intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
arch/x86/events/intel/lbr.c +0 -9
index 979b094ae228..27e4be839926 100644
@@ -1043,15 +1043,6 @@ void intel_pmu_lbr_read(struct perf_branch_stack *lbr_stack,
 		return;
 	}
 
-	/*
-	 * Don't read when all LBRs users are using adaptive PEBS.
-	 *
-	 * This could be smarter and actually check the event,
-	 * but this simple approach seems to work for now.
-	 */
-	if (cpuc->lbr_users == cpuc->lbr_pebs_users)
-		return;
-
 	x86_pmu.lbr_read(cpuc, lbr_stack);
 
 	intel_pmu_lbr_filter(cpuc, lbr_stack, lbr_counters);
Signature · 1 line
2.34.1
#28 lore raw

Re: [Patch v6 13/20] perf/x86/intel: Always snapshot live hardware LBR state for BPF

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

__intel_pmu_snapshot_branch_stack() is intended to capture the current live hardware LBR state for the BPF path. However, that snapshot is currently skipped when all LBR users are PEBS events.

BPF LBR snapshotting now uses its own dedicated LBR stack, so there is no longer a risk of clobbering the shared default LBR stack. As a result, always snapshot the live hardware LBR state for the BPF path.

Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
author
#29 lore raw

PATCH v6 14/20 perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active

In reply to Dapeng Mi

Refreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.

Limit the LBR stack refresh to the PMI path only when LBR users are present.

---
 arch/x86/events/amd/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/events/amd/core.c +2 -1
index 160102a46b18..f0425e369837 100644
@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
 
 	/* Read branch records */
 	if (x86_pmu.lbr_nr) {
-		amd_pmu_lbr_read();
+		if (cpuc->lbr_users > 0)
+			amd_pmu_lbr_read();
 		status &= ~GLOBAL_STATUS_LBRS_FROZEN;
 	}
 
Signature · 1 line
2.34.1
#30 lore raw

Re: [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

Refreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.

Limit the LBR stack refresh to the PMI path only when LBR users are present.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
author
#31 lore raw

Re: [Patch v6 14/20] perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active

In reply to Dapeng Mi
Quoted 18 lines · Dapeng Mi

Refreshing the LBR stack in the PMI handler is unnecessary when no LBR users are active. This adds avoidable overhead and hurts performance.

Limit the LBR stack refresh to the PMI path only when LBR users are present.

---
 arch/x86/events/amd/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/events/amd/core.c +0 -0
index 160102a46b18..f0425e369837 100644

arch/x86/events/amd/core.c static int amd_pmu_v2_handle_irq(struct pt_regs *regs) · line 995

Quoted 9 lines
arch/x86/events/amd/core.c +2 -1
@@ -988,7 +988,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
 
 	/* Read branch records */
 	if (x86_pmu.lbr_nr) {
-		amd_pmu_lbr_read();
+		if (cpuc->lbr_users > 0)
+			amd_pmu_lbr_read();
 		status &= ~GLOBAL_STATUS_LBRS_FROZEN;
 	}
 

This patch is actually unnecessary since amd_pmu_lbr_read() internally checks cpuc->lbr_users and would early return if cpuc->lbr_users is 0.

Would drop this patch in next version. Thanks.

author
#32 lore raw

PATCH v6 15/20 perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

In reply to Dapeng Mi

PEBS and non-PEBS events share the same LBR stack in cpuc->__lbr_stack. In the PMI handler, the non-PEBS path refreshes the LBR state before the PEBS buffer is drained.

When PEBS and non-PEBS LBR events overflow in the same PMI, draining the PEBS buffer can overwrite the shared LBR stack with the PEBS snapshot. This leaves stale LBR data from the PEBS record in place, so userspace can see the wrong branch stack instead of the live LBR state at PMI entry.

Track whether the LBR state has already been refreshed with a lbr_refreshed flag. If PEBS events overwrite the shared LBR stack before the non-PEBS LBR event is processed, clear the flag so the live LBR state is refreshed again. Remove the unconditional LBR refresh at PMI entry and initialize the flag as false so the stack is refreshed only when an active non-PEBS LBR event requires it. Since the LBR state is frozen while the PMI is being handled, it only needs to be read once for multiple non-PEBS LBR events in the same interrupt.

Fixes c22497f5838c ("perf/x86/intel: Support adaptive PEBS v4")
---
 arch/x86/events/intel/core.c | 40 +++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 10 deletions(-)
arch/x86/events/intel/core.c +30 -10
index fbde874af618..b267c68f9467 100644
@@ -3864,8 +3864,9 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 {
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-	int bit;
+	bool lbr_refreshed = false;
 	int handled = 0;
+	int bit;
 
 	inc_perf_irq_stat();
 
@@ -3906,6 +3907,15 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 	if (__test_and_clear_bit(GLOBAL_STATUS_BUFFER_OVF_BIT, (unsigned long *)&status)) {
 		u64 pebs_enabled = cpuc->pebs_enabled;
 
+		/*
+		 * Prior to adaptive PEBS, PEBS does not support LBR snapshotting.
+		 * Refresh the LBR stack explicitly when LBR users are present.
+		 */
+		if (cpuc->lbr_users > 0 && !x86_pmu.intel_cap.pebs_baseline) {
+			intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
+			lbr_refreshed = true;
+		}
+
 		handled += x86_pmu_handle_guest_pebs(regs, &data);
 		handled += static_call(x86_pmu_drain_pebs)(regs, &data);
 		/* Ensure no "suspicious NMI" warning for empty PEBS buffer. */
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		 *
 		 * The PEBS buffer has to be drained before handling the A-PMI
 		 */
-		if (is_pebs_counter_event_group(event))
+		if (is_pebs_counter_event_group(event)) {
+			/*
+			 * PEBS counter sampling is only supported on adaptive PEBS
+			 * and arch-PEBS, both of which snapshot the LBR stack.
+			 * No explicit LBR refresh is needed here.
+			 */
 			static_call(x86_pmu_drain_pebs)(regs, &data);
+			/*
+			 * Draining the PEBS buffer can overwrite the shared
+			 * LBR stack, so mark lbr_refreshed as false.
+			 */
+			lbr_refreshed = false;
+		}
 
 		last_period = event->hw.last_period;
 
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		if (has_branch_stack(event))
+		if (has_branch_stack(event)) {
+			if (!lbr_refreshed) {
+				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+						   cpuc->lbr_counters);
+				lbr_refreshed = true;
+			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+		}
 
 		x86_pmu_clear_perf_regs(regs);
 		x86_pmu_update_perf_regs(event, &data, regs, false);
@@ -4070,13 +4097,6 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
-	/*
-	 * No need to read the LBR stack when all LBR users are PEBS events,
-	 * since the PEBS handler updates it.
-	 */
-	if (cpuc->lbr_users != cpuc->lbr_pebs_users)
-		intel_pmu_lbr_read(cpu_lbr_stack(cpuc), cpuc->lbr_counters);
-
 	/*
 	 * Do not clear the LBR frozen status here; keep LBR frozen
 	 * throughout PMI handling.
Signature · 1 line
2.34.1
#33 lore raw

Re: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain
--

commit f13e61130af8b47007c91f048b386359760f05ad
Author: Dapeng Mi <[email protected]>

perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

This commit tracks whether the shared LBR stack has been refreshed during PMI handling with a lbr_refreshed flag and removes the unconditional LBR read at PMI entry so the live LBR state is read on demand for non-PEBS LBR events and re-read if PEBS draining overwrites it.

Quoted 4 lines
arch/x86/events/intel/core.c +0 -0
index fbde874af6182..b267c68f94676 100644

[ ... ]

Quoted 27 lines
arch/x86/events/intel/core.c +13 -2
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		 *
 		 * The PEBS buffer has to be drained before handling the A-PMI
 		 */
-		if (is_pebs_counter_event_group(event))
+		if (is_pebs_counter_event_group(event)) {
+			/*
+			 * PEBS counter sampling is only supported on adaptive PEBS
+			 * and arch-PEBS, both of which snapshot the LBR stack.
+			 * No explicit LBR refresh is needed here.
+			 */
 			static_call(x86_pmu_drain_pebs)(regs, &data);
+			/*
+			 * Draining the PEBS buffer can overwrite the shared
+			 * LBR stack, so mark lbr_refreshed as false.
+			 */
+			lbr_refreshed = false;
+		}
 
 		last_period = event->hw.last_period;
 
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		if (has_branch_stack(event))
+		if (has_branch_stack(event)) {

arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4047

Quoted 10 lines
arch/x86/events/intel/core.c +6 -0
+			if (!lbr_refreshed) {
+				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+						   cpuc->lbr_counters);
+				lbr_refreshed = true;
+			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+		}
 
 		x86_pmu_clear_perf_regs(regs);
 		x86_pmu_update_perf_regs(event, &data, regs, false);

[Severity: Medium]
Can perf_event_overflow() still drain the PEBS buffer and clobber cpu_lbr_stack(cpuc) when is_pebs_counter_event_group(event) is false?

When a non-PEBS event using PERF_SAMPLE_READ | PERF_FORMAT_GROUP is grouped with a PERF_X86_EVENT_AUTO_RELOAD PEBS LBR sibling (without PERF_X86_EVENT_PEBS_CNTR set on the leader), perf_event_overflow() -> perf_output_sample() -> perf_output_read_group() -> intel_pmu_read_event() calls intel_pmu_drain_pebs_buffer() before PERF_SAMPLE_BRANCH_STACK is written. Draining pending PEBS records calls intel_pmu_store_pebs_lbrs(), which overwrites cpu_lbr_stack(cpuc) before the current non-PEBS sample copies data.br_stack and leaves lbr_refreshed set to true for any subsequent non-PEBS LBR events in the same PMI.

In addition, because the unconditional intel_pmu_lbr_read() at PMI entry is removed, an indirect PEBS drain on pre-adaptive PEBS (!x86_pmu.intel_cap.pebs_baseline) when GLOBAL_STATUS_BUFFER_OVF_BIT is clear will run intel_pmu_pebs_fixup_ip() against an unrefreshed cpu_lbr_stack(cpuc).

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author
#34 lore raw

Re: [Patch v6 15/20] perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

Quoted 73 lines · On 11/6/2026 1:58 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/x86/intel: unrefreshed or clobbered LBR stack on indirect PEBS drain
--

commit f13e61130af8b47007c91f048b386359760f05ad
Author: Dapeng Mi <[email protected]>

perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed

This commit tracks whether the shared LBR stack has been refreshed during PMI handling with a lbr_refreshed flag and removes the unconditional LBR read at PMI entry so the live LBR state is read on demand for non-PEBS LBR events and re-read if PEBS draining overwrites it.

Quoted 4 lines
arch/x86/events/intel/core.c +0 -0
index fbde874af6182..b267c68f94676 100644

[ ... ]

Quoted 27 lines
arch/x86/events/intel/core.c +13 -2
@@ -4003,8 +4013,19 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		 *
 		 * The PEBS buffer has to be drained before handling the A-PMI
 		 */
-		if (is_pebs_counter_event_group(event))
+		if (is_pebs_counter_event_group(event)) {
+			/*
+			 * PEBS counter sampling is only supported on adaptive PEBS
+			 * and arch-PEBS, both of which snapshot the LBR stack.
+			 * No explicit LBR refresh is needed here.
+			 */
 			static_call(x86_pmu_drain_pebs)(regs, &data);
+			/*
+			 * Draining the PEBS buffer can overwrite the shared
+			 * LBR stack, so mark lbr_refreshed as false.
+			 */
+			lbr_refreshed = false;
+		}
 
 		last_period = event->hw.last_period;
 
@@ -4013,8 +4034,14 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 
 		perf_sample_data_init(&data, 0, last_period);
 
-		if (has_branch_stack(event))
+		if (has_branch_stack(event)) {

arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 4047

Quoted 10 lines
arch/x86/events/intel/core.c +6 -0
+			if (!lbr_refreshed) {
+				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
+						   cpuc->lbr_counters);
+				lbr_refreshed = true;
+			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+		}
 
 		x86_pmu_clear_perf_regs(regs);
 		x86_pmu_update_perf_regs(event, &data, regs, false);

[Severity: Medium]
Can perf_event_overflow() still drain the PEBS buffer and clobber cpu_lbr_stack(cpuc) when is_pebs_counter_event_group(event) is false?

When a non-PEBS event using PERF_SAMPLE_READ | PERF_FORMAT_GROUP is grouped with a PERF_X86_EVENT_AUTO_RELOAD PEBS LBR sibling (without PERF_X86_EVENT_PEBS_CNTR set on the leader), perf_event_overflow() -> perf_output_sample() -> perf_output_read_group() -> intel_pmu_read_event() calls intel_pmu_drain_pebs_buffer() before PERF_SAMPLE_BRANCH_STACK is written. Draining pending PEBS records calls intel_pmu_store_pebs_lbrs(), which overwrites cpu_lbr_stack(cpuc) before the current non-PEBS sample copies data.br_stack and leaves lbr_refreshed set to true for any subsequent non-PEBS LBR events in the same PMI.

Quoted 5 lines

In addition, because the unconditional intel_pmu_lbr_read() at PMI entry is removed, an indirect PEBS drain on pre-adaptive PEBS (!x86_pmu.intel_cap.pebs_baseline) when GLOBAL_STATUS_BUFFER_OVF_BIT is clear will run intel_pmu_pebs_fixup_ip() against an unrefreshed cpu_lbr_stack(cpuc).

Sigh, Sashiko can always give surprise. :)

It's a real issue. It looks a new version has to be posted to fix this issue.

Quoted 1 line
author
#35 lore raw

PATCH v6 16/20 perf/x86/intel: Remove dead lbr_pebs_users

In reply to Dapeng Mi

The lbr_pebs_users accounting is no longer used, so remove the dead field and the related code paths that maintain it.

---
 arch/x86/events/intel/lbr.c  | 5 -----
 arch/x86/events/perf_event.h | 1 -
 2 files changed, 6 deletions(-)
arch/x86/events/intel/lbr.c +0 -5
index 27e4be839926..3500cf3518f4 100644
@@ -617,8 +617,6 @@ void intel_pmu_lbr_add(struct perf_event *event)
 	 * be 'new'. Conversely, a new event can get installed through the
 	 * context switch path for the first time.
 	 */
-	if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
-		cpuc->lbr_pebs_users++;
 	perf_sched_cb_inc(event->pmu);
 	if (!cpuc->lbr_users++ && !event->total_time_running)
 		intel_pmu_lbr_reset();
@@ -688,11 +686,8 @@ void intel_pmu_lbr_del(struct perf_event *event)
 	if (event->hw.flags & PERF_X86_EVENT_LBR_SELECT)
 		cpuc->lbr_select = 0;
 
-	if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0)
-		cpuc->lbr_pebs_users--;
 	cpuc->lbr_users--;
 	WARN_ON_ONCE(cpuc->lbr_users < 0);
-	WARN_ON_ONCE(cpuc->lbr_pebs_users < 0);
 	perf_sched_cb_dec(event->pmu);
 
 	/*
arch/x86/events/perf_event.h +0 -1
index fa0af528cd1e..0213e2a8efff 100644
@@ -526,7 +526,6 @@ struct cpu_hw_events {
 	 * Intel LBR bits
 	 */
 	int				lbr_users;
-	int				lbr_pebs_users;
 	u64				__lbr_stack[LBR_STACK_SIZE / sizeof(u64)];
 	u64				lbr_counters[MAX_LBR_ENTRIES]; /* branch stack extra */
 	union {
Signature · 1 line
2.34.1
#36 lore raw

Re: [Patch v6 16/20] perf/x86/intel: Remove dead lbr_pebs_users

Sashiko has reviewed this patch and found no issues.
Quoted 4 lines

The lbr_pebs_users accounting is no longer used, so remove the dead field and the related code paths that maintain it.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
author
#37 lore raw

PATCH v6 17/20 perf/x86/intel: Clear raw LBR counters in the LBR stack

In reply to Dapeng Mi

Raw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.

If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.

Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")
---
 arch/x86/events/intel/core.c |  1 +
 arch/x86/events/intel/lbr.c  | 13 +++++++++++++
 arch/x86/events/perf_event.h |  2 ++
 3 files changed, 16 insertions(+)
arch/x86/events/intel/core.c +1 -0
index b267c68f9467..707bff285ed7 100644
@@ -2835,6 +2835,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,
 		(struct perf_branch_stack *)bpf->__lbr_stack;
 
 	intel_pmu_lbr_read(lbr_stack, bpf->counters);
+	intel_pmu_lbr_clear_counters(lbr_stack);
 	cnt = min_t(unsigned int, cnt, lbr_stack->nr);
 
 	memcpy(entries, lbr_stack->entries,
arch/x86/events/intel/lbr.c +13 -0
index 3500cf3518f4..fbc053ebbeaf 100644
@@ -987,6 +987,18 @@ static void intel_pmu_lbr_counters_reorder(struct cpu_hw_events *cpuc,
 	}
 }
 
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack)
+{
+	int i;
+
+	if (!(x86_pmu.flags & PMU_FL_BR_CNTR))
+		return;
+
+	/* reserved field stores raw branch counters, clear it. */
+	for (i = 0; i < lbr_stack->nr; i++)
+		lbr_stack->entries[i].reserved = 0;
+}
+
 void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 				struct cpu_hw_events *cpuc,
 				struct perf_event *event)
@@ -997,6 +1009,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 		return;
 	}
 
+	intel_pmu_lbr_clear_counters(cpu_lbr_stack(cpuc));
 	perf_sample_save_brstack(data, event, cpu_lbr_stack(cpuc), NULL);
 }
 
arch/x86/events/perf_event.h +2 -0
index 0213e2a8efff..a4e53c99eaf5 100644
@@ -1984,6 +1984,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
 
 void intel_pebs_init(void);
 
+void intel_pmu_lbr_clear_counters(struct perf_branch_stack *lbr_stack);
+
 void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 				struct cpu_hw_events *cpuc,
 				struct perf_event *event);
Signature · 1 line
2.34.1
#38 lore raw

Re: [Patch v6 17/20] perf/x86/intel: Clear raw LBR counters in the LBR stack

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

Raw LBR counters are currently stored in the reserved fields of the LBR stack even when no LBR event needs them.

If an LBR counter event is present, it clears the raw LBR counters while processing the stack, but those counters can be leaked to userspace when no LBR event requires them. Clear the raw LBR counters when no LBR event uses them.

Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author
#39 lore raw

PATCH v6 18/20 perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler

In reply to Dapeng Mi

When handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.

However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.

Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.

This situation is rare, the performance impact from extra refill should be negligible.

Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")
---
 arch/x86/events/intel/core.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/events/intel/core.c +10 -1
index 707bff285ed7..f4d4613fb9f7 100644
@@ -3866,6 +3866,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	bool lbr_refreshed = false;
+	bool lbr_counter_cleared = false;
 	int handled = 0;
 	int bit;
 
@@ -4036,12 +4037,20 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)
 		perf_sample_data_init(&data, 0, last_period);
 
 		if (has_branch_stack(event)) {
-			if (!lbr_refreshed) {
+			/*
+			 * The raw LBR counters stored in the reserved field of
+			 * the LBR entries were cleared during processing of the
+			 * previous LBR event, so refresh the LBR stack again if
+			 * this event needs the LBR counters.
+			 */
+			if (!lbr_refreshed ||
+			    (lbr_counter_cleared && is_branch_counters_group(event))) {
 				intel_pmu_lbr_read(cpu_lbr_stack(cpuc),
 						   cpuc->lbr_counters);
 				lbr_refreshed = true;
 			}
 			intel_pmu_lbr_save_brstack(&data, cpuc, event);
+			lbr_counter_cleared = true;
 		}
 
 		x86_pmu_clear_perf_regs(regs);
Signature · 1 line
2.34.1
#40 lore raw

Re: [Patch v6 18/20] perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler

Sashiko has reviewed this patch and found no issues.
Quoted 13 lines

When handling an LBR counter event, the raw LBR counters in the reserved field of each LBR entry are cleared after the stack is processed and reported to userspace. This prevents raw counter data from leaking.

However, if multiple events overflow in the same PMI, an LBR counter event may not be the first one processed. In that case, the raw LBR counters may already have been cleared by the previous LBR event, so the later LBR counter event cannot read the correct values.

Refill the LBR stack before processing an LBR counter event when multiple events are handled in a single PMI. This restores the correct raw LBR counter data for the event in this corner case.

Quoted 6 lines

This situation is rare, the performance impact from extra refill should be negligible.

Fixes 33744916196b ("perf/x86/intel: Support branch counters logging")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
author
#41 lore raw

PATCH v6 19/20 perf/x86/intel: Pass LBR stack size into store_lbr()

In reply to Dapeng Mi

intel_pmu_store_lbr() assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.

Add a lbr_nr parameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.

Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")
---
 arch/x86/events/intel/ds.c   |  8 ++++----
 arch/x86/events/intel/lbr.c  | 18 ++++++++++--------
 arch/x86/events/perf_event.h |  2 +-
 3 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/events/intel/ds.c +4 -4
index 332754e1da4f..95a2166f5a47 100644
@@ -2595,12 +2595,12 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
 
 	if (format_group & PEBS_DATACFG_LBRS) {
 		struct lbr_entry *lbr = next_record;
-		int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
+		unsigned int num_lbr = ((format_group >> PEBS_DATACFG_LBR_SHIFT)
 					& 0xff) + 1;
 		next_record = next_record + num_lbr * sizeof(struct lbr_entry);
 
 		if (has_branch_stack(event)) {
-			intel_pmu_store_pebs_lbrs(lbr);
+			intel_pmu_store_pebs_lbrs(lbr, num_lbr);
 			intel_pmu_lbr_save_brstack(data, cpuc, event);
 		}
 	}
@@ -2781,7 +2781,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
 	if (header->lbr) {
 		struct arch_pebs_lbr_header *lbr_header = next_record;
 		struct lbr_entry *lbr;
-		int num_lbr;
+		unsigned int num_lbr;
 
 		next_record = lbr_header + 1;
 		lbr = next_record;
@@ -2792,7 +2792,7 @@ static void setup_arch_pebs_sample_data(struct perf_event *event,
 		next_record += num_lbr * sizeof(struct lbr_entry);
 
 		if (has_branch_stack(event)) {
-			intel_pmu_store_pebs_lbrs(lbr);
+			intel_pmu_store_pebs_lbrs(lbr, num_lbr);
 			intel_pmu_lbr_save_brstack(data, cpuc, event);
 		}
 	}
arch/x86/events/intel/lbr.c +10 -8
index fbc053ebbeaf..ce6aa35e20fb 100644
@@ -902,7 +902,7 @@ static __always_inline u16 get_lbr_cycles(u64 info)
 
 static_assert((64 - PERF_BRANCH_ENTRY_INFO_BITS_MAX) > LBR_INFO_BR_CNTR_NUM * LBR_INFO_BR_CNTR_BITS);
 
-static void intel_pmu_store_lbr(struct lbr_entry *entries,
+static void intel_pmu_store_lbr(struct lbr_entry *entries, int lbr_nr,
 				struct perf_branch_stack *lbr_stack)
 {
 	struct perf_branch_entry *e;
@@ -911,8 +911,8 @@ static void intel_pmu_store_lbr(struct lbr_entry *entries,
 	bool mispred;
 	int i;
 
-	cpu_init_lbr_stack(lbr_stack, x86_pmu.lbr_nr);
-	for (i = 0; i < x86_pmu.lbr_nr; i++) {
+	cpu_init_lbr_stack(lbr_stack, lbr_nr);
+	for (i = 0; i < lbr_nr; i++) {
 		lbr = entries ? &entries[i] : NULL;
 		e = &lbr_stack->entries[i];
 
@@ -1016,7 +1016,7 @@ void intel_pmu_lbr_save_brstack(struct perf_sample_data *data,
 static void intel_pmu_arch_lbr_read(struct cpu_hw_events *cpuc,
 				    struct perf_branch_stack *lbr_stack)
 {
-	intel_pmu_store_lbr(NULL, lbr_stack);
+	intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
 }
 
 static void
@@ -1026,14 +1026,15 @@ intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc,
 	struct x86_perf_task_context_arch_lbr_xsave *xsave = cpuc->lbr_xsave;
 
 	if (!xsave) {
-		intel_pmu_store_lbr(NULL, lbr_stack);
+		intel_pmu_store_lbr(NULL, x86_pmu.lbr_nr, lbr_stack);
 		return;
 	}
 	xsave->header.xfeatures = 0;
 	xsaves(&xsave->xsave, XFEATURE_MASK_LBR);
 
 	if (xsave->header.xfeatures & XFEATURE_MASK_LBR) {
-		intel_pmu_store_lbr(xsave->lbr.entries, lbr_stack);
+		intel_pmu_store_lbr(xsave->lbr.entries,
+				    x86_pmu.lbr_nr, lbr_stack);
 	} else {
 		/* Clear the LBR stack to avoid leaking stale entries. */
 		lbr_stack->nr = 0;
@@ -1313,7 +1314,7 @@ intel_pmu_lbr_filter(struct cpu_hw_events *cpuc,
 	lbr_stack->nr = lbr_nr;
 }
 
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 
@@ -1324,7 +1325,8 @@ void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr)
 	else
 		cpu_lbr_stack(cpuc)->hw_idx = intel_pmu_lbr_tos();
 
-	intel_pmu_store_lbr(lbr, cpu_lbr_stack(cpuc));
+	intel_pmu_store_lbr(lbr, min(lbr_nr, x86_pmu.lbr_nr),
+			    cpu_lbr_stack(cpuc));
 	intel_pmu_lbr_filter(cpuc, cpu_lbr_stack(cpuc), cpuc->lbr_counters);
 }
 
arch/x86/events/perf_event.h +1 -1
index a4e53c99eaf5..1b3fa997d89d 100644
@@ -1980,7 +1980,7 @@ void intel_pmu_pebs_late_setup(struct cpu_hw_events *cpuc);
 
 void intel_pmu_drain_pebs_buffer(void);
 
-void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr);
+void intel_pmu_store_pebs_lbrs(struct lbr_entry *lbr, unsigned int lbr_nr);
 
 void intel_pebs_init(void);
 
Signature · 1 line
2.34.1
#42 lore raw

Re: [Patch v6 19/20] perf/x86/intel: Pass LBR stack size into store_lbr()

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

intel_pmu_store_lbr() assumes the LBR stack always contains x86_pmu.lbr_nr entries, but arch-PEBS can use a smaller configured depth. This can result in out-of-bounds reads when the stack is shallower than the hardware maximum.

Add a lbr_nr parameter to the helper so the code only processes the available entries and avoids overrunning the LBR buffer.

Fixes d21954c8a0ff ("perf/x86/intel: Process arch-PEBS records or record fragments")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
author
#43 lore raw

PATCH v6 20/20 perf/core: Don't overwrite shared LBR stack depth

In reply to Dapeng Mi

perf_sample_save_brstack() currently clamps the shared LBR stack depth to min(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a small sample_max_stack from one event can shrink the depth for all others in the same PMI.

As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.

Fixes c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")
---
 include/linux/perf_event.h | 13 ++++++++++---
 kernel/events/core.c       |  8 ++++----
 2 files changed, 14 insertions(+), 7 deletions(-)
include/linux/perf_event.h +10 -3
index 19de3f1cbd6e..b9a4a821c7c6 100644
@@ -1424,12 +1424,19 @@ static inline bool has_branch_stack(struct perf_event *event)
 	return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
 }
 
+static inline u64 max_lbr_stack_depth(struct perf_event *event,
+				      struct perf_branch_stack *brs)
+{
+	return min_t(u16, event->attr.sample_max_stack, brs->nr);
+}
+
 static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 					    struct perf_event *event,
 					    struct perf_branch_stack *brs,
 					    u64 *brs_cntr)
 {
 	int size = sizeof(u64); /* nr */
+	u64 lbr_nr;
 
 	if (!has_branch_stack(event))
 		return;
@@ -1439,9 +1446,9 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 	if (branch_sample_hw_index(event))
 		size += sizeof(u64);
 
-	brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr);
+	lbr_nr = max_lbr_stack_depth(event, brs);
 
-	size += brs->nr * sizeof(struct perf_branch_entry);
+	size += lbr_nr * sizeof(struct perf_branch_entry);
 
 	/*
 	 * The extension space for counters is appended after the
@@ -1449,7 +1456,7 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data,
 	 * of events of each branch.
 	 */
 	if (brs_cntr)
-		size += brs->nr * sizeof(u64);
+		size += lbr_nr * sizeof(u64);
 
 	data->br_stack = brs;
 	data->br_stack_cntr = brs_cntr;
kernel/events/core.c +4 -4
index 457d75ea7142..db2613c2e1a2 100644
@@ -8462,12 +8462,12 @@ void perf_output_sample(struct perf_output_handle *handle,
 
 	if (sample_type & PERF_SAMPLE_BRANCH_STACK) {
 		if (data->br_stack) {
+			u64 lbr_nr = max_lbr_stack_depth(event, data->br_stack);
 			size_t size;
 
-			size = data->br_stack->nr
-			     * sizeof(struct perf_branch_entry);
+			size = lbr_nr * sizeof(struct perf_branch_entry);
 
-			perf_output_put(handle, data->br_stack->nr);
+			perf_output_put(handle, lbr_nr);
 			if (branch_sample_hw_index(event))
 				perf_output_put(handle, data->br_stack->hw_idx);
 			perf_output_copy(handle, data->br_stack->entries, size);
@@ -8476,7 +8476,7 @@ void perf_output_sample(struct perf_output_handle *handle,
 			 * right after the struct perf_branch_stack.
 			 */
 			if (data->br_stack_cntr) {
-				size = data->br_stack->nr * sizeof(u64);
+				size = lbr_nr * sizeof(u64);
 				perf_output_copy(handle, data->br_stack_cntr, size);
 			}
 		} else {
Signature · 1 line
2.34.1
#44 lore raw

Re: [Patch v6 20/20] perf/core: Don't overwrite shared LBR stack depth

Sashiko has reviewed this patch and found no issues.
Quoted 12 lines

perf_sample_save_brstack() currently clamps the shared LBR stack depth to min(event->attr.sample_max_stack, brs->nr). This is unsafe because the global LBR stack is shared across all LBR events, and a small sample_max_stack from one event can shrink the depth for all others in the same PMI.

As a result, later events may output a shorter LBR stack than requested even when they do not require a small sample depth. Do not overwrite the shared LBR stack depth in perf_sample_save_brstack(); instead, keep the per-event limit in a local variable so each event is constrained independently without mutating global state.

Quoted 3 lines
Fixes c53e14f1ea4a ("perf: Extend per event callchain limit to branch stack")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
author
#45 lore raw
In reply to Dapeng Mi

Sashiko reported an issue on the patch 15/20 "perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed",
https://lore.kernel.org/all/[email protected]/.

It looks a new version is needed to fix the issue. Thanks.

Quoted 130 lines · Dapeng Mi

This patch series fixes several x86 LBR sampling issues discovered during review and debugging. Most of them are related to BPF LBR snapshotting support.

The series originally targeted the BPF stack out-of-bounds access issue, but additional problems were found during debugging and code review, including:
- PMU and LBR being re-enabled unconditionally at BPF snapshotting exit, even when they had already been disabled before BPF snapshotting on both AMD and Intel.
- LBR not being fully frozen in the PMI handler on Intel, which allows branches in the PMI path to pollute the BPF LBR snapshot.
- BPF snapshotting overwriting the cached LBR stack of PEBS LBR events on Intel.
- PEBS LBR events overwriting the LBR stack of non-PEBS LBR events on Intel when both overflow in the same PMI.
- LBR counter events reporting zero counters on Intel when multiple LBR counter events overflow in the same PMI.

Patch layout:
- Patch 01: Add a sanity check to ensure the LBR stack does not exceed MAX_LBR_ENTRIES.
- Patch 02: Validate that hardware LBR data has actually been stored in the XSAVE area before parsing it.
- Patch 03-04: Fix the potential LBR stack out-of-bounds access issue on AMD and Intel separately.
- Patch 05-08: Fix the incorrect PMU and LBR re-enabling issue in perf_snapshot_branch_stack() for AMD and Intel separately.
- Patch 09: Freeze LBR throughout the PMI processing period and ensure all LBR events observe a consistent LBR stack in the PMI handler.
- Patch 10: Add counted bounds tracking for perf_branch_stack{}.
- Patch 11-12: Add a dedicated LBR stack for BPF LBR snapshotting to avoid overwriting the default LBR stack of other LBR events.
- Patch 13: Ensure perf_snapshot_branch_stack() snapshots the live hardware LBR data instead of the cached PEBS LBR stack.
- Patch 14: Refresh LBR stack in PMI only when LBR users are active for AMD.
- Patch 15: Prevent PEBS LBR data from overwriting non-PEBS LBR data when both overflow in the same PMI.
- Patch 16: Remove dead lbr_pebs_users and relative code.
- Patch 17-18: Ensure LBR counters are not dropped when multiple LBR counter events overflow in the same PMI.
- Patch 19: Clamp the parsed PEBS LBR stack depth to x86_pmu.lbr_nr to avoid LBR stack out-of-bounds access.
- Patch 20: Fix the shared LBR stack depth truncation issue.

Changes in v6:
- Patch 01: Move LBR stack depth validation into common init_hw_perf_events(). (Peter)
- Patch 10: New patch, add __counted_by() tracking for flexible array entries[] of perf_branch_stack{}. (Peter)
- Patch 14: Refresh LBR stack only when LBR users are active for AMD. (Peter)
- Patch 15: Refactor code to ensure no duplicated LBR stack reading. (Peter)
- Patch 16: New patch, remove dead lbr_pebs_users.

Changes in v5:
- Patch 02: Clear xsave->header.xfeatures before calling the xsaves instructuion to read LBR states.
- Patch 05-18: New patches to fix newly discovered issues described above.

Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).

Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).

Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/

History:
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Dapeng Mi (19):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel: Validate LBR XSAVE state before updating LBR stack perf/x86/intel/lbr: Limit branch snapshots to valid entries perf/x86: Add lbr_enabled to track LBR state
perf/x86/amd: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Add helper to enable PMU without LBR perf/x86/intel: Avoid re-enabling PMU and LBR for BPF when disabled perf/x86/intel: Keep LBR frozen throughout PMI handling
perf: Add counted-bounds tracking for perf_branch_stack perf/x86/intel: Allow LBR read helpers to operate on explicit stacks perf/x86/intel: Use a dedicated LBR stack for BPF snapshotting perf/x86/intel: Always snapshot live hardware LBR state for BPF perf/x86/amd: Refresh LBR stack in PMI only when LBR users are active perf/x86/intel: Refresh LBR stack for non-PEBS LBR events when needed perf/x86/intel: Remove dead lbr_pebs_users
perf/x86/intel: Clear raw LBR counters in the LBR stack perf/x86/intel: Refill LBR stack for LBR counter events in PMI handler perf/x86/intel: Pass LBR stack size into store_lbr() perf/core: Don't overwrite shared LBR stack depth

Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries

 arch/powerpc/perf/core-book3s.c |  43 +++++---
 arch/x86/events/amd/brs.c       |   7 +-
 arch/x86/events/amd/core.c      |  21 ++--
 arch/x86/events/amd/lbr.c       |  48 +++++----
 arch/x86/events/core.c          |   8 +-
 arch/x86/events/intel/core.c    | 130 ++++++++++++++++++++----
 arch/x86/events/intel/ds.c      |  20 ++--
 arch/x86/events/intel/lbr.c     | 170 +++++++++++++++++++-------------
 arch/x86/events/perf_event.h    |  45 +++++++--
 drivers/perf/arm_brbe.c         |   5 +
 include/linux/perf_event.h      |  15 ++-
 kernel/events/core.c            |   8 +-
 12 files changed, 365 insertions(+), 155 deletions(-)

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4