perf/x86: Limit branch snapshots to valid entries
Subsystems (3)
v4 cover perf/x86: Limit branch snapshots to valid entries
Sashiko reported that the vendor-specific branch snapshot callbacks used by bpf_get_branch_snapshot() copy up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], regardless of how many valid entries the last LBR read produced. This makes BPF programs read stale or invalid branch records. Patch 1 addresses AMD and patch 2 addresses Intel.
Changes in v4:
- Patch 01: Disable LBR if parsed x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES (Sashiko).
- Patch 05: New patch, which always snapshots live hardware LBR stack for BPF path (Sashiko).
Changes in v3:
- Patch 01: A new patch to ensure x86_pmu.lbr_nr not to exclude MAX_LBR_ENTRIES (Internal Sashiko).
- Patch 03: Clean cached LBR stack when reseting hardware LBR registers, which prevents stale LBR stack leakage.
- Patch 04: Don't reset cached LBR stack if all LBR events are PEBS events. This ensures PEBS captured LBR stack isn't cleared (Sashiko).
Changes in v2:
- Include a fix for Intel LBR.
- Add Tested-by from Dapeng to the Intel fix.
https://lore.kernel.org/all/[email protected]/
History:
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (4):
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES perf/x86/intel/lbr: Reset the cached LBR stack in
intel_pmu_lbr_reset()
perf/x86/intel/lbr: Limit branch snapshots to valid entries pef/x86/intel: Always snapshot live hardware LBR state for BPF path
Sandipan Das (1):
perf/x86/amd/lbr: Limit branch snapshots to valid entries
arch/x86/events/amd/core.c | 2 +- arch/x86/events/amd/lbr.c | 4 +++- arch/x86/events/core.c | 6 +++++ arch/x86/events/intel/core.c | 45 +++++++++++++++++++++++++++++++++--- arch/x86/events/intel/lbr.c | 19 +++++++-------- 5 files changed, 62 insertions(+), 14 deletions(-)
base-commit: f5a78ecccd98a04c47cea1cd63c2a2fdce8a4b48
Signature · 1 line
2.34.1
PATCH v4 1/5 perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Disable LBR if x86_pmu.lbr_nr is larger than MAX_LBR_ENTRIES, preventing out-of-bounds access in the cached LBR stack handling path.
--- arch/x86/events/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/events/core.c +6 -0
@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)if (!x86_pmu.config_mask) x86_pmu.config_mask =X86_RAW_EVENT_MASK ; +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +perf_events_lapic_init ();register_nmi_handler (NMI_LOCAL , perf_event_nmi_handler,0 ,"PMI" );
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v4 1/5] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/x86: init_hw_perf_events() disables LBR too late after vendor PMU init --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86: init_hw_perf_events() disables LBR too late after vendor PMU init --
commit c6c1bbb1f4feb929dd247ddd507177ce58d871e1
Author: Dapeng Mi <[email protected]>
perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
This commit disables last branch record support in init_hw_perf_events() if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES to prevent out-of-bounds accesses in the cached branch stack handling path.
Quoted 6 lines
arch/x86/events/core.c +0
-0@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)if (!x86_pmu.config_mask)
arch/x86/events/core.c static int __init init_hw_perf_events(void) · line 2655
Quoted 10 lines
arch/x86/events/core.c +6
-0x86_pmu.config_mask =X86_RAW_EVENT_MASK ; +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +perf_events_lapic_init ();register_nmi_handler (NMI_LOCAL , perf_event_nmi_handler,0 ,"PMI" );
[Severity: Medium]
Does zeroing x86_pmu.lbr_nr here in init_hw_perf_events() happen too late, after intel_pmu_init() and amd_pmu_init() have already configured hardware and driver state using the unvalidated x86_pmu.lbr_nr value?
In arch/x86/events/intel/lbr.c:intel_pmu_arch_lbr_init(), if lbr_nr exceeds MAX_LBR_ENTRIES, initialization proceeds instead of taking the clear_arch_lbr path:
lbr_nr = fls(eax.split.lbr_depth_mask) * 8; if (!lbr_nr) goto clear_arch_lbr; /* Apply the max depth of Arch LBR */ if (wrmsrq_safe(MSR_ARCH_LBR_DEPTH, lbr_nr)) goto clear_arch_lbr; ... x86_pmu.lbr_counters = ecx.split.lbr_counters; x86_pmu.lbr_nr = lbr_nr; if (!!x86_pmu.lbr_counters) x86_pmu.flags |= PMU_FL_BR_CNTR | PMU_FL_DYN_CONSTRAINT;
Because clear_arch_lbr is skipped, X86_FEATURE_ARCH_LBR remains enabled, MSR_ARCH_LBR_DEPTH is written, pmu->task_ctx_cache is allocated via create_lbr_kmem_cache(), and x86_pmu.lbr_ctl_map and static branches are configured.
Subsequently, arch/x86/events/intel/lbr.c:reserve_lbr_buffers() checks the feature bit and task_ctx_cache rather than x86_pmu.lbr_nr:
if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR))
return;
for_each_possible_cpu(cpu) {
cpuc = per_cpu_ptr(&cpu_hw_events, cpu);
kmem_cache = x86_get_pmu(cpu)->task_ctx_cache;
if (!kmem_cache || cpuc->lbr_xsave)
continue;
cpuc->lbr_xsave = kmem_cache_alloc_node(kmem_cache,
GFP_KERNEL | __GFP_ZERO,
cpu_to_node(cpu));
}Wouldn't this still allocate per-cpu cpuc->lbr_xsave buffers on hardware reservation even after x86_pmu.lbr_nr is set to 0?
Similarly, in arch/x86/events/intel/core.c:lbr_is_visible(), leaving PMU_FL_BR_CNTR set in x86_pmu.flags keeps branch_counter_nr and branch_counter_width visible in sysfs when x86_pmu.lbr_nr is 0:
/* branches */ if (i == 0) return x86_pmu.lbr_nr ? attr->mode : 0; return (x86_pmu.flags & PMU_FL_BR_CNTR) ? attr->mode : 0;
In addition, both arch/x86/events/intel/core.c:intel_pmu_init() and arch/x86/events/amd/core.c:amd_core_pmu_init() check x86_pmu.lbr_nr before init_hw_perf_events() clears it:
if (x86_pmu.lbr_nr) {
intel_pmu_lbr_init();
pr_cont("%d-deep LBR, ", x86_pmu.lbr_nr);
/* only support branch_stack snapshot for perfmon >= v2 */
if (x86_pmu.disable_all == intel_pmu_disable_all) {
if (boot_cpu_has(X86_FEATURE_ARCH_LBR)) {
static_call_update(perf_snapshot_branch_stack,
intel_pmu_snapshot_arch_branch_stack);
} else {
static_call_update(perf_snapshot_branch_stack,
intel_pmu_snapshot_branch_stack);
}
}
}And in amd_core_pmu_init():
if (!amd_pmu_lbr_init()) {
/* LBR requires flushing on context switch */
x86_pmu.sched_task = amd_pmu_lbr_sched_task;
static_call_update(amd_pmu_branch_hw_config, amd_pmu_lbr_hw_config);
static_call_update(amd_pmu_branch_reset, amd_pmu_lbr_reset);
static_call_update(amd_pmu_branch_add, amd_pmu_lbr_add);
static_call_update(amd_pmu_branch_del, amd_pmu_lbr_del);
/* Only support branch_stack snapshot on perfmon v2 */
if (x86_pmu.handle_irq == amd_pmu_v2_handle_irq)
static_call_update(perf_snapshot_branch_stack, amd_pmu_v2_snapshot_branch_stack);
} else if (!amd_brs_init()) {If bpf_get_branch_snapshot() is later called when x86_pmu.lbr_nr is 0, won't the registered perf_snapshot_branch_stack callback still run, manipulate branch and global control registers, and unconditionally call intel_pmu_enable_all(0) or amd_pmu_v2_enable_all(0)?
Should the MAX_LBR_ENTRIES check be performed before or during the vendor-specific initialization routines (such as intel_pmu_lbr_init(), intel_pmu_arch_lbr_init(), and amd_pmu_lbr_init()) so that None of the hardware state, slab caches, flags, or static calls are enabled when lbr_nr is out of range?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [Patch v4 1/5] perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds MAX_LBR_ENTRIES
Quoted 128 lines · On 10/21/2026 1:42 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/x86:init_hw_perf_events()disables LBR too late after vendor PMU init --commit
c6c1bbb1f4feb929dd247ddd507177ce58d871e1
Author: Dapeng Mi <[email protected]>perf/x86: Disable LBR if x86_pmu.lbr_nr exceeds
MAX_LBR_ENTRIESThis commit disables last branch record support in
init_hw_perf_events()if x86_pmu.lbr_nr exceedsMAX_LBR_ENTRIESto prevent out-of-bounds accesses in the cached branch stack handling path.Quoted 6 lines
arch/x86/events/core.c +0
-0@@ -2645,6 +2645,12 @@ static int __init init_hw_perf_events(void)if (!x86_pmu.config_mask)arch/x86/events/core.c static int __init init_hw_perf_events(void) · line 2655
Quoted 10 lines
arch/x86/events/core.c +6
-0x86_pmu.config_mask =X86_RAW_EVENT_MASK ; +if (x86_pmu.lbr_nr >MAX_LBR_ENTRIES ) { +pr_warn ("LBR entries %d exceeds max %d, disable LBR.\n" , + x86_pmu.lbr_nr,MAX_LBR_ENTRIES ); + x86_pmu.lbr_nr =0 ; + } +perf_events_lapic_init ();register_nmi_handler (NMI_LOCAL , perf_event_nmi_handler,0 ,"PMI" );[Severity: Medium]
Does zeroing x86_pmu.lbr_nr here ininit_hw_perf_events()happen too late, afterintel_pmu_init()andamd_pmu_init()have already configured hardware and driver state using the unvalidated x86_pmu.lbr_nr value?In arch/x86/events/intel/lbr.c:intel_pmu_arch_lbr_init(), if
lbr_nrexceedsMAX_LBR_ENTRIES, initialization proceeds instead of taking theclear_arch_lbrpath:lbr_nr = fls(eax.split.lbr_depth_mask) * 8; if (!lbr_nr) goto clear_arch_lbr; /* Apply the max depth of Arch LBR */ if (wrmsrq_safe(MSR_ARCH_LBR_DEPTH, lbr_nr)) goto clear_arch_lbr; ... x86_pmu.lbr_counters = ecx.split.lbr_counters; x86_pmu.lbr_nr = lbr_nr; if (!!x86_pmu.lbr_counters) x86_pmu.flags |= PMU_FL_BR_CNTR | PMU_FL_DYN_CONSTRAINT;Because
clear_arch_lbris skipped,X86_FEATURE_ARCH_LBRremains enabled,MSR_ARCH_LBR_DEPTHis written,pmu->task_ctx_cacheis allocated viacreate_lbr_kmem_cache(), and x86_pmu.lbr_ctl_map and static branches are configured.Subsequently, arch/x86/events/intel/lbr.c:reserve_lbr_buffers() checks the feature bit and
task_ctx_cacherather than x86_pmu.lbr_nr:if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR)) return; for_each_possible_cpu(cpu) { cpuc = per_cpu_ptr(&cpu_hw_events, cpu); kmem_cache = x86_get_pmu(cpu)->task_ctx_cache; if (!kmem_cache || cpuc->lbr_xsave) continue; cpuc->lbr_xsave = kmem_cache_alloc_node(kmem_cache, GFP_KERNEL | __GFP_ZERO, cpu_to_node(cpu)); }Wouldn't this still allocate per-cpu
cpuc->lbr_xsavebuffers on hardware reservation even after x86_pmu.lbr_nr is set to 0?Similarly, in arch/x86/events/intel/core.c:lbr_is_visible(), leaving
PMU_FL_BR_CNTRset in x86_pmu.flags keepsbranch_counter_nrandbranch_counter_widthvisible in sysfs when x86_pmu.lbr_nr is 0:/* branches */ if (i == 0) return x86_pmu.lbr_nr ? attr->mode : 0; return (x86_pmu.flags & PMU_FL_BR_CNTR) ? attr->mode : 0;In addition, both arch/x86/events/intel/core.c:intel_pmu_init() and arch/x86/events/amd/core.c:amd_core_pmu_init() check x86_pmu.lbr_nr before
init_hw_perf_events()clears it:if (x86_pmu.lbr_nr) { intel_pmu_lbr_init(); pr_cont("%d-deep LBR, ", x86_pmu.lbr_nr); /* only support branch_stack snapshot for perfmon >= v2 */ if (x86_pmu.disable_all == intel_pmu_disable_all) { if (boot_cpu_has(X86_FEATURE_ARCH_LBR)) { static_call_update(perf_snapshot_branch_stack, intel_pmu_snapshot_arch_branch_stack); } else { static_call_update(perf_snapshot_branch_stack, intel_pmu_snapshot_branch_stack); } } }And in
amd_core_pmu_init():if (!amd_pmu_lbr_init()) { /* LBR requires flushing on context switch */ x86_pmu.sched_task = amd_pmu_lbr_sched_task; static_call_update(amd_pmu_branch_hw_config, amd_pmu_lbr_hw_config); static_call_update(amd_pmu_branch_reset, amd_pmu_lbr_reset); static_call_update(amd_pmu_branch_add, amd_pmu_lbr_add); static_call_update(amd_pmu_branch_del, amd_pmu_lbr_del); /* Only support branch_stack snapshot on perfmon v2 */ if (x86_pmu.handle_irq == amd_pmu_v2_handle_irq) static_call_update(perf_snapshot_branch_stack, amd_pmu_v2_snapshot_branch_stack); } else if (!amd_brs_init()) {If
bpf_get_branch_snapshot()is later called when x86_pmu.lbr_nr is 0, won't the registeredperf_snapshot_branch_stackcallback still run, manipulate branch and global control registers, and unconditionally callintel_pmu_enable_all(0)oramd_pmu_v2_enable_all(0)?
Quoted 5 lines
Should the
MAX_LBR_ENTRIEScheck be performed before or during the vendor-specific initialization routines (such asintel_pmu_lbr_init(),intel_pmu_arch_lbr_init(), andamd_pmu_lbr_init()) so that None of the hardware state, slab caches, flags, or static calls are enabled whenlbr_nris out of range?
The change is good enough. The branch sampling perf event creation is guarded with x86_pmu.lbr_nr regardless of on Intel or AMD platforms. If x86_pmu.lbr_nr is 0, then no events with branch sampling can be created successfully, and if no active LBR event is present, the LBR hardware is not really enabled.
Since it's a common validation for all x86 vendor platforms, it should be placed into the x86 common PMU initialization path.
Thanks.
Quoted 1 line
PATCH v4 2/5 perf/x86/amd/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiFrom: Sandipan Das <[email protected]>
The branch snapshot callback used by bpf_get_branch_snapshot() copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when amd_pmu_lbr_read() skips reading the registers, so that a stale count is never reused.
--- arch/x86/events/amd/core.c | 2 +- arch/x86/events/amd/lbr.c | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/events/amd/core.c +1 -1
@@ -944,7 +944,7 @@ static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, u cpuc =this_cpu_ptr (&cpu_hw_events);amd_pmu_lbr_read (); - cnt =min (cnt, x86_pmu.lbr_nr); + cnt =min (cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);amd_pmu_v2_enable_all (0 );
arch/x86/events/amd/lbr.c +3 -1
@@ -168,8 +168,10 @@ void amd_pmu_lbr_read(void)struct branch_entry entry;int out =0 , idx, i; -if (!cpuc->lbr_users) +if (!cpuc->lbr_users) { + cpuc->lbr_stack.nr =0 ;return ; + }for (i =0 ; i < x86_pmu.lbr_nr; i++) { entry.from.full =amd_pmu_lbr_get_from (i);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v4 2/5] perf/x86/amd/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
From: Sandipan Das <[email protected]>
The branch snapshot callback used by
bpf_get_branch_snapshot()copies up to x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr are invalid, but they end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when
amd_pmu_lbr_read()skips reading the registers, so that a stale count is never reused.
Quoted 6 lines
Fixes a4d18112e531 ("perf/x86/amd: Support capturing LBR from software events")Reported-by Sashiko <[email protected]>Signed-off-by Sandipan Das <[email protected]>Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v4 3/5 perf/x86/intel/lbr: Reset the cached LBR stack in intel_pmu_lbr_reset()
In reply to Dapeng MiReset the cached LBR stack in addition to the hardware LBR register file when intel_pmu_lbr_reset() is called. This prevents stale LBR data from being reused after a context switch or similar transition.
Also adjust intel_pmu_arch_lbr_read_xsave() to update the cached LBR stack only when new hardware LBR data is present.
--- arch/x86/events/intel/lbr.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/events/intel/lbr.c +6 -1
@@ -193,6 +193,8 @@ void intel_pmu_lbr_reset(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + cpuc->lbr_stack.nr =0 ; +if (!x86_pmu.lbr_nr)return ;
@@ -1012,7 +1014,10 @@ static void intel_pmu_arch_lbr_read_xsave(struct cpu_hw_events *cpuc) }xsaves (&xsave->xsave,XFEATURE_MASK_LBR ); -intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +if (xsave->header.xfeatures &XFEATURE_MASK_LBR ) +intel_pmu_store_lbr (cpuc, xsave->lbr.entries); +else + cpuc->lbr_stack.nr =0 ; }void intel_pmu_lbr_read (void )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v4 3/5] perf/x86/intel/lbr: Reset the cached LBR stack in intel_pmu_lbr_reset()
Sashiko has reviewed this patch and found no issues.
Quoted 9 lines
Reset the cached LBR stack in addition to the hardware LBR register file when
intel_pmu_lbr_reset()is called. This prevents stale LBR data from being reused after a context switch or similar transition.Also adjust
intel_pmu_arch_lbr_read_xsave()to update the cached LBR stack only when new hardware LBR data is present.Fixes c085fb877467 ("perf/x86/intel/lbr: Support XSAVES for arch LBR read")Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v4 4/5 perf/x86/intel/lbr: Limit branch snapshots to valid entries
In reply to Dapeng MiThe branch snapshot callback used by bpf_get_branch_snapshot() always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.
Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
--- arch/x86/events/intel/core.c | 2 +- arch/x86/events/intel/lbr.c | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +1 -1
@@ -2822,7 +2822,7 @@ __intel_pmu_snapshot_branch_stack(struct perf_branch_entry *entries,struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);intel_pmu_lbr_read (); - cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr); + cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr);memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);intel_pmu_enable_all (0 );
arch/x86/events/intel/lbr.c +7 -2
@@ -1024,14 +1024,19 @@ void intel_pmu_lbr_read(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +if (!cpuc->lbr_users ||vlbr_exclude_host ()) { +/* Reset lbr_stack.nr to prevent stale data leakage. */ + cpuc->lbr_stack.nr =0 ; +return ; + } +/* * Don't read when all LBRs users are using adaptive PEBS. * * This could be smarter and actually check the event, * but this simple approach seems to work for now. */ -if (!cpuc->lbr_users ||vlbr_exclude_host () || - cpuc->lbr_users == cpuc->lbr_pebs_users) +if (cpuc->lbr_users == cpuc->lbr_pebs_users)return ; x86_pmu.lbr_read (cpuc);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v4 4/5] perf/x86/intel/lbr: Limit branch snapshots to valid entries
Sashiko has reviewed this patch and found no issues.
Quoted 14 lines
The branch snapshot callback used by
bpf_get_branch_snapshot()always copies x86_pmu.lbr_nr entries from cpuc->lbr_entries[], even when the last LBR read kept fewer branches. The entries past lbr_stack.nr hold invalid branches which end up reaching BPF programs.Limit copying to lbr_stack.nr entries and also reset lbr_stack.nr when no active LBR events or LBR hardware is owned by guest. So the stale or guest LBR stack data can't be leaked.
For the case where all LBR events are PEBS events, hardware LBR reads are currently skipped; that can leave stale LBR stack data from previous PEBS sampling in place and may be inconsistent with the intended semantics of
bpf_get_branch_snapshot(). This will be addressed in a follow-up patch.
Quoted 6 lines
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Reported-by Sashiko <[email protected]>Co-developed-by Sandipan Das <[email protected]>Signed-off-by Sandipan Das <[email protected]>Signed-off-by Dapeng Mi <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v4 5/5 pef/x86/intel: Always snapshot live hardware LBR state for BPF path
In reply to Dapeng Mi__intel_pmu_snapshot_branch_stack() is meant to snapshot the current runtime LBR state from hardware for the BPF path. However, stale cached PEBS LBR data can be snapshotted instead when a PEBS LBR user is active.
Fix this by always snapshotting the live hardware LBR state, regardless of whether a PEBS LBR user is active. Reading the runtime LBR registers can overwrite the cached PEBS LBR stack, so save and restore the cached PEBS LBR state around the snapshot. This preserves the PEBS LBR data for later output to userspace in the normal PEBS record path.
--- arch/x86/events/intel/core.c | 43 ++++++++++++++++++++++++++++++++++-- arch/x86/events/intel/lbr.c | 9 -------- 2 files changed, 41 insertions(+), 11 deletions(-)
arch/x86/events/intel/core.c +41 -2
@@ -2815,16 +2815,50 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (added, false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *snapshot =this_cpu_ptr (&lbr_snapshot); + +/* +* This may be called from bpf_event_handler() in a PMI context. If +* a PEBS LBR user is active, the PEBS sample record has already updated +* cpuc->lbr_stack. Reading the LBR registers here would overwrite that +* cached PEBS LBR stack, so save it first and restore it afterward. This +* keeps the cached LBR data available for later output to userspace in +* the normal PEBS record path. +*/ +if (cpuc->lbr_pebs_users) { +memcpy (&snapshot->stack, &cpuc->lbr_stack, +sizeof (struct perf_branch_stack )); +memcpy (snapshot->entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (snapshot->counters, cpuc->lbr_counters, +sizeof (u64) * cpuc->lbr_stack.nr); + }intel_pmu_lbr_read (); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); + +if (cpuc->lbr_pebs_users) { +memcpy (&cpuc->lbr_stack, &snapshot->stack, +sizeof (struct perf_branch_stack )); +memcpy (cpuc->lbr_entries, snapshot->entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (cpuc->lbr_counters, snapshot->counters, +sizeof (u64) * cpuc->lbr_stack.nr); + } +intel_pmu_enable_all (0 );local_irq_restore (flags);return cnt;
@@ -4022,7 +4056,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read ();intel_pmu_ack_status (status);if (++loops >100 ) {static bool warned;
arch/x86/events/intel/lbr.c +0 -9
@@ -1030,15 +1030,6 @@ void intel_pmu_lbr_read(void)return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc);intel_pmu_lbr_filter (cpuc);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [Patch v4 5/5] pef/x86/intel: Always snapshot live hardware LBR state for BPF path
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [Critical] perf/x86/intel: LBR state corruption in __intel_pmu_snapshot_branch_stack() - [High]…
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] perf/x86/intel: LBR state corruption in __intel_pmu_snapshot_branch_stack()
- [High] perf/x86/intel: __intel_pmu_snapshot_branch_stack() enables PMU in PMI
- [Medium] perf/x86/intel: unfrozen LBRs and PMU re-enable in PMI branch snapshot
- [Medium] perf/x86: NMI race on lbr_snapshot in __intel_pmu_snapshot_branch_stack()
- [Medium] perf/x86/intel: unfiltered LBR branch counter bits in perf_branch_entry.reserved --
commit 4d2b360d61a8a8f6825b3cd919182ee99dbe2d76
Author: Dapeng Mi <[email protected]>
pef/x86/intel: Always snapshot live hardware LBR state for BPF path
This commit updates __intel_pmu_snapshot_branch_stack() to always read live hardware LBR registers for the BPF snapshot path even when PEBS LBR users are active. To avoid clobbering cached PEBS LBR data during a PMI, it saves and restores cpuc->lbr_stack, cpuc->lbr_entries, and cpuc->lbr_counters around intel_pmu_lbr_read() using a per-CPU buffer, and moves the cpuc->lbr_users == cpuc->lbr_pebs_users check into intel_pmu_handle_irq().
Quoted 5 lines
arch/x86/events/intel/core.c +0
-0
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2823
Quoted 9 lines
arch/x86/events/intel/core.c +6
-0@@ -2815,16 +2815,50 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (added, false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);
[Severity: Medium]
Can an NMI race on the per-CPU lbr_snapshot buffer when __intel_pmu_snapshot_branch_stack() is called in non-NMI context?
In arch/x86/events/intel/core.c, intel_pmu_snapshot_branch_stack() and intel_pmu_snapshot_arch_branch_stack() call local_irq_save(flags) and __intel_pmu_disable_all(false) before invoking __intel_pmu_snapshot_branch_stack(). However, local_irq_save() only masks normal interrupts, and an in-flight PMI or another NMI can still arrive after __intel_pmu_disable_all() due to NMI delivery latency.
When cpuc->lbr_pebs_users > 0 and a non-NMI BPF program (such as an fentry/fexit trampoline via __bpf_prog_enter_recur() or a raw_tracepoint, where bpf_prog_active is not incremented) calls:
bpf_get_branch_snapshot()
-> intel_pmu_snapshot_branch_stack()
-> __intel_pmu_snapshot_branch_stack()an arriving PMI can still invoke bpf_overflow_handler() -> bpf_get_branch_snapshot() -> __intel_pmu_snapshot_branch_stack() on the same CPU, overwriting this_cpu_ptr(&lbr_snapshot) while the outer call is between its save and restore steps. Even if the PMI does not call bpf_get_branch_snapshot(), intel_pmu_handle_irq() updates cpuc->lbr_stack and cpuc->lbr_entries while the interrupted snapshot is saving or restoring them.
Since cpuc->lbr_stack only holds live sample data awaiting output during a PMI, should the save and restore of cpuc->lbr_stack be restricted to NMI context or otherwise protected against NMI reentrancy?
Quoted 14 lines
arch/x86/events/intel/core.c +9
-0+ static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *snapshot =this_cpu_ptr (&lbr_snapshot); + +/* +* This may be called from bpf_event_handler() in a PMI context. If +* a PEBS LBR user is active, the PEBS sample record has already updated +* cpuc->lbr_stack. Reading the LBR registers here would overwrite that +* cached PEBS LBR stack, so save it first and restore it afterward. This +* keeps the cached LBR data available for later output to userspace in
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2847
Quoted 10 lines
arch/x86/events/intel/core.c +10
-0+* the normal PEBS record path. +*/ +if (cpuc->lbr_pebs_users) { +memcpy (&snapshot->stack, &cpuc->lbr_stack, +sizeof (struct perf_branch_stack )); +memcpy (snapshot->entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (snapshot->counters, cpuc->lbr_counters, +sizeof (u64) * cpuc->lbr_stack.nr); + }
[Severity: Critical]
What happens in __intel_pmu_snapshot_branch_stack() when cpuc->lbr_users is non-zero, but cpuc->lbr_pebs_users is 0?
In arch/x86/events/intel/lbr.c:intel_pmu_lbr_add(), cpuc->lbr_pebs_users is only incremented when x86_pmu.intel_cap.pebs_baseline is set and event->attr.precise_ip > 0:
if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0) cpuc->lbr_pebs_users++;
When non-PEBS hardware LBR events (precise_ip == 0), legacy PEBS LBR events (pre-Ice Lake where pebs_baseline == 0), or Architectural PEBS LBR events (arch/x86/events/intel/ds.c:setup_arch_pebs_sample_data(), where pebs_baseline == 0) are active, cpuc->lbr_users > 0 while cpuc->lbr_pebs_users == 0.
In those paths, handle_pmi_common(), setup_pebs_fixed_sample_data(), and setup_arch_pebs_sample_data() call intel_pmu_lbr_save_brstack() before perf_event_overflow() -> __perf_event_overflow() -> bpf_overflow_handler():
if (has_branch_stack(event)) intel_pmu_lbr_save_brstack(&data, cpuc, event); ... perf_event_overflow(event, &data, regs);
Inside include/linux/perf_event.h:perf_sample_save_brstack(), brs (&cpuc->lbr_stack) is clamped in place and used to precompute data->dyn_size:
brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr); size += brs->nr * sizeof(struct perf_branch_entry); if (brs_cntr) size += brs->nr * sizeof(u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr; data->dyn_size += size; data->sample_flags |= PERF_SAMPLE_BRANCH_STACK;
If bpf_overflow_handler() then calls bpf_get_branch_snapshot() -> __intel_pmu_snapshot_branch_stack() with cpuc->lbr_pebs_users == 0, the save and restore of cpuc->lbr_stack, cpuc->lbr_entries, and cpuc->lbr_counters are skipped, and intel_pmu_lbr_read() overwrites them in place:
1. In kernel/events/core.c:perf_output_sample(), data->br_stack still points to &cpuc->lbr_stack. Because intel_pmu_lbr_read() resets cpuc->lbr_stack.nr up to x86_pmu.lbr_nr, undoing the sample_max_stack clamp after data->dyn_size was already computed, perf_output_sample() can copy more perf_branch_entry (and counter) elements than perf_output_begin() reserved in the ring buffer.
2. The frozen LBR entries captured at PMI entry (or from the Arch PEBS record) in cpuc->lbr_entries are overwritten by unfrozen post-ack LBR registers containing PMI handler branches.
3. For branch counter groups, intel_pmu_lbr_counters_reorder() has already extracted counter bits from cpuc->lbr_entries[i].reserved into cpuc->lbr_counters[i] and cleared .reserved; calling intel_pmu_lbr_read() -> intel_pmu_store_lbr() writes raw counter bits back into cpuc->lbr_entries[i].reserved and desynchronizes cpuc->lbr_counters[].
Should the save and restore in __intel_pmu_snapshot_branch_stack() be gated on cpuc->lbr_users instead of cpuc->lbr_pebs_users (and save/restore based on the pre-clamp entry count or full array when needed)?
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2851
Quoted 5 lines
arch/x86/events/intel/core.c +0
-1intel_pmu_lbr_read (); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);
[Severity: Medium]
On platforms with Architectural LBR and branch counter logging, does copying cpuc->lbr_entries directly after intel_pmu_lbr_read() leak raw branch counter bits in entries[i].reserved to BPF programs?
When intel_pmu_snapshot_arch_branch_stack() calls __intel_pmu_snapshot_branch_stack(), intel_pmu_lbr_read() invokes intel_pmu_arch_lbr_read() or intel_pmu_arch_lbr_read_xsave(), which calls arch/x86/events/intel/lbr.c:intel_pmu_store_lbr():
/*
* Leverage the reserved field of
* cpuc->lbr_entries[i] to temporarily store the
* branch counters information. The later code will
* decide what content can be disclosed to the perf
* tool. Pleae see intel_pmu_lbr_counters_reorder(). */
.reserved = (info >> LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,
Normally intel_pmu_lbr_counters_reorder() filters those bits and clears cpuc->lbr_entries[i].reserved to 0 before the entries are exposed. Since __intel_pmu_snapshot_branch_stack() copies cpuc->lbr_entries directly into entries without calling intel_pmu_lbr_counters_reorder(), won't the raw hardware counter bits remain in the UAPI perf_branch_entry.reserved field returned by bpf_get_branch_snapshot()?
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2862
Quoted 11 lines
arch/x86/events/intel/core.c +10
-0+ +if (cpuc->lbr_pebs_users) { +memcpy (&cpuc->lbr_stack, &snapshot->stack, +sizeof (struct perf_branch_stack )); +memcpy (cpuc->lbr_entries, snapshot->entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (cpuc->lbr_counters, snapshot->counters, +sizeof (u64) * cpuc->lbr_stack.nr); + } +intel_pmu_enable_all (0 );
[Severity: High]
Is it safe for __intel_pmu_snapshot_branch_stack() to unconditionally call intel_pmu_enable_all(0) when invoked from bpf_overflow_handler() in a PMI?
In arch/x86/events/intel/core.c:intel_pmu_handle_irq(), the PMI handler saves cpuc->enabled, sets cpuc->enabled = 0, and disables the PMU in hardware:
pmu_enabled = cpuc->enabled; ... cpuc->enabled = 0; __intel_pmu_disable_all(true);
The PMI handler expects the PMU, PEBS, BTS, and LBRs to remain disabled in hardware while handle_pmi_common() and __intel_pmu_pebs_last_event() drain PEBS records and reprogram counter periods, only restoring the hardware state at the done label if pmu_enabled was set:
/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled; if (pmu_enabled) __intel_pmu_enable_all(0, true);
When a BPF program runs via perf_event_overflow() -> bpf_overflow_handler() during a PMI and calls bpf_get_branch_snapshot() -> __intel_pmu_snapshot_branch_stack(), calling intel_pmu_enable_all(0) invokes x86_pmu_pebs_enable_all() and __intel_pmu_enable_all(0, false) while cpuc->enabled is 0. This re-enables MSR_IA32_PEBS_ENABLE, LBRs, BTS, and MSR_CORE_PERF_GLOBAL_CTRL before __intel_pmu_pebs_last_event() and handle_pmi_common() finish reprogramming counter periods, and leaves the hardware PMU enabled after the PMI exits if the PMI interrupted an x86_pmu_disable() section where pmu_enabled was 0.
Should __intel_pmu_snapshot_branch_stack() avoid calling intel_pmu_enable_all(0) when cpuc->enabled is 0?
Quoted 4 lines
arch/x86/events/intel/core.c +0
-0local_irq_restore (flags);return cnt;@@ -4022,7 +4056,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
arch/x86/events/intel/core.c static int intel_pmu_handle_irq(struct pt_regs *regs) · line 4065
Quoted 10 lines
arch/x86/events/intel/core.c +6
-1loops =0 ; again: -intel_pmu_lbr_read (); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read ();intel_pmu_ack_status (status);
[Severity: Medium]
When cpuc->lbr_users == cpuc->lbr_pebs_users and a BPF program calls bpf_get_branch_snapshot() from bpf_overflow_handler() during a PMI, haven't the hardware LBR registers already been unfrozen by intel_pmu_ack_status(status)?
In arch/x86/events/intel/core.c:intel_pmu_handle_irq(), __intel_pmu_disable_all(true) clears MSR_CORE_PERF_GLOBAL_CTRL but does not clear LBR enable bits in MSR_IA32_DEBUGCTLMSR or MSR_ARCH_LBR_CTL, relying on GLOBAL_STATUS_LBRS_FROZEN to keep the LBR stack frozen.
As noted in arch/x86/events/intel/lbr.c:__intel_pmu_lbr_enable():
/* * No need to unfreeze manually, as v4 can do that as part * of the GLOBAL_STATUS ack. */ if (pmi && x86_pmu.version >= 4) return;
On PMU v4+ platforms (which include all pebs_baseline platforms), calling intel_pmu_ack_status(status) clears GLOBAL_STATUS_LBRS_FROZEN and immediately unfreezes the hardware LBRs. By the time handle_pmi_common() -> x86_pmu_drain_pebs() -> __intel_pmu_pebs_last_event() sets up the PEBS sample and invokes bpf_overflow_handler() -> __intel_pmu_snapshot_branch_stack(), the unfrozen hardware LBR MSRs have been recording branches inside the PMI handler itself (and __intel_pmu_snapshot_branch_stack() then also calls intel_pmu_enable_all(0) before the PMI finishes).
Can __intel_pmu_snapshot_branch_stack() still obtain an unpolluted hardware LBR snapshot in this PMI path after intel_pmu_ack_status(status) has unfrozen the LBRs?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [Patch v4 5/5] pef/x86/intel: Always snapshot live hardware LBR state for BPF path
Quoted 61 lines · On 10/21/2026 2:05 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] perf/x86/intel: LBR state corruption in__intel_pmu_snapshot_branch_stack()
- [High] perf/x86/intel:__intel_pmu_snapshot_branch_stack()enables PMU in PMI
- [Medium] perf/x86/intel: unfrozen LBRs and PMU re-enable in PMI branch snapshot
- [Medium] perf/x86: NMI race onlbr_snapshotin__intel_pmu_snapshot_branch_stack()
- [Medium] perf/x86/intel: unfiltered LBR branch counter bits in perf_branch_entry.reserved --commit
4d2b360d61a8a8f6825b3cd919182ee99dbe2d76
Author: Dapeng Mi <[email protected]>pef/x86/intel: Always snapshot live hardware LBR state for BPF path
This commit updates
__intel_pmu_snapshot_branch_stack()to always read live hardware LBR registers for the BPF snapshot path even when PEBS LBR users are active. To avoid clobbering cached PEBS LBR data during a PMI, it saves and restorescpuc->lbr_stack,cpuc->lbr_entries, andcpuc->lbr_countersaroundintel_pmu_lbr_read()using a per-CPU buffer, and moves thecpuc->lbr_users==cpuc->lbr_pebs_userscheck intointel_pmu_handle_irq().Quoted 5 lines
arch/x86/events/intel/core.c +0
-0arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2823
Quoted 9 lines
arch/x86/events/intel/core.c +6
-0@@ -2815,16 +2815,50 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (added, false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);[Severity: Medium]
Can an NMI race on the per-CPUlbr_snapshotbuffer when__intel_pmu_snapshot_branch_stack()is called in non-NMI context?In
arch/x86/events/intel/core.c,intel_pmu_snapshot_branch_stack()andintel_pmu_snapshot_arch_branch_stack()calllocal_irq_save(flags)and__intel_pmu_disable_all(false)before invoking__intel_pmu_snapshot_branch_stack(). However,local_irq_save()only masks normal interrupts, and an in-flight PMI or another NMI can still arrive after__intel_pmu_disable_all()due to NMI delivery latency.When
cpuc->lbr_pebs_users> 0 and a non-NMI BPF program (such as an fentry/fexit trampoline via__bpf_prog_enter_recur()or araw_tracepoint, wherebpf_prog_activeis not incremented) calls:
bpf_get_branch_snapshot()-> intel_pmu_snapshot_branch_stack() -> __intel_pmu_snapshot_branch_stack()an arriving PMI can still invoke
bpf_overflow_handler()->bpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack()on the same CPU, overwritingthis_cpu_ptr(&lbr_snapshot)while the outer call is between its save and restore steps. Even if the PMI does not callbpf_get_branch_snapshot(),intel_pmu_handle_irq()updatescpuc->lbr_stackandcpuc->lbr_entrieswhile the interrupted snapshot is saving or restoring them.
Quoted 3 lines
Since
cpuc->lbr_stackonly holds live sample data awaiting output during a PMI, should the save and restore ofcpuc->lbr_stackbe restricted to NMI context or otherwise protected against NMI reentrancy?
This can't happen in reality. The whole PMU including LBR has been disabled before calling __intel_pmu_snapshot_branch_stack(). Since PMU hardware has been disabled, then no PMI can be triggered at the period of __intel_pmu_snapshot_branch_stack() running. So there is no reentry for the __intel_pmu_snapshot_branch_stack() helper, and cpuc->lbr_entries won't be corrupted.
Quoted 89 lines
Quoted 15 lines
arch/x86/events/intel/core.c +9
-0+ static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *snapshot =this_cpu_ptr (&lbr_snapshot); + +/* +* This may be called from bpf_event_handler() in a PMI context. If +* a PEBS LBR user is active, the PEBS sample record has already updated +* cpuc->lbr_stack. Reading the LBR registers here would overwrite that +* cached PEBS LBR stack, so save it first and restore it afterward. This +* keeps the cached LBR data available for later output to userspace in arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2847
Quoted 10 lines
arch/x86/events/intel/core.c +10
-0+* the normal PEBS record path. +*/ +if (cpuc->lbr_pebs_users) { +memcpy (&snapshot->stack, &cpuc->lbr_stack, +sizeof (struct perf_branch_stack )); +memcpy (snapshot->entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (snapshot->counters, cpuc->lbr_counters, +sizeof (u64) * cpuc->lbr_stack.nr); + }[Severity: Critical]
What happens in__intel_pmu_snapshot_branch_stack()whencpuc->lbr_usersis non-zero, butcpuc->lbr_pebs_usersis 0?In arch/x86/events/intel/lbr.c:intel_pmu_lbr_add(),
cpuc->lbr_pebs_usersis only incremented when x86_pmu.intel_cap.pebs_baseline is set andevent->attr.precise_ip> 0:if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0) cpuc->lbr_pebs_users++;When non-PEBS hardware LBR events (
precise_ip== 0), legacy PEBS LBR events (pre-Ice Lake wherepebs_baseline== 0), or Architectural PEBS LBR events (arch/x86/events/intel/ds.c:setup_arch_pebs_sample_data(), wherepebs_baseline== 0) are active,cpuc->lbr_users> 0 whilecpuc->lbr_pebs_users== 0.In those paths,
handle_pmi_common(),setup_pebs_fixed_sample_data(), andsetup_arch_pebs_sample_data()callintel_pmu_lbr_save_brstack()beforeperf_event_overflow()->__perf_event_overflow()->bpf_overflow_handler():if (has_branch_stack(event)) intel_pmu_lbr_save_brstack(&data, cpuc, event); ... perf_event_overflow(event, &data, regs);Inside include/linux/perf_event.h:perf_sample_save_brstack(), brs (
&cpuc->lbr_stack) is clamped in place and used to precomputedata->dyn_size:brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr); size += brs->nr * sizeof(struct perf_branch_entry); if (brs_cntr) size += brs->nr * sizeof(u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr; data->dyn_size += size; data->sample_flags |= PERF_SAMPLE_BRANCH_STACK;If
bpf_overflow_handler()then callsbpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack()withcpuc->lbr_pebs_users== 0, the save and restore ofcpuc->lbr_stack,cpuc->lbr_entries, andcpuc->lbr_countersare skipped, andintel_pmu_lbr_read()overwrites them in place:1. In kernel/events/core.c:perf_output_sample(),
data->br_stackstill points to&cpuc->lbr_stack. Becauseintel_pmu_lbr_read()resetscpuc->lbr_stack.nrup to x86_pmu.lbr_nr, undoing thesample_max_stackclamp afterdata->dyn_sizewas already computed,perf_output_sample()can copy moreperf_branch_entry(and counter) elements thanperf_output_begin()reserved in the ring buffer.2. The frozen LBR entries captured at PMI entry (or from the Arch PEBS record) in
cpuc->lbr_entriesare overwritten by unfrozen post-ack LBR registers containing PMI handler branches.3. For branch counter groups,
intel_pmu_lbr_counters_reorder()has already extracted counter bits from cpuc->lbr_entries[i].reserved into cpuc->lbr_counters[i] and cleared .reserved; callingintel_pmu_lbr_read()->intel_pmu_store_lbr()writes raw counter bits back into cpuc->lbr_entries[i].reserved and desynchronizes cpuc->lbr_counters[].
Quoted 3 lines
Should the save and restore in
__intel_pmu_snapshot_branch_stack()be gated oncpuc->lbr_usersinstead ofcpuc->lbr_pebs_users(and save/restore based on the pre-clamp entry count or full array when needed)?
This is a real issue. For arch-PEBS, the lbr_pebs_users is missed to account. Would add a new patch to address this issue.
Quoted 26 lines
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2851
Quoted 6 lines
arch/x86/events/intel/core.c +0
-1intel_pmu_lbr_read (); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);[Severity: Medium]
On platforms with Architectural LBR and branch counter logging, does copyingcpuc->lbr_entriesdirectly afterintel_pmu_lbr_read()leak raw branch counter bits in entries[i].reserved to BPF programs?When
intel_pmu_snapshot_arch_branch_stack()calls__intel_pmu_snapshot_branch_stack(),intel_pmu_lbr_read()invokesintel_pmu_arch_lbr_read()orintel_pmu_arch_lbr_read_xsave(), which calls arch/x86/events/intel/lbr.c:intel_pmu_store_lbr():/*
* Leverage the reserved field of
* cpuc->lbr_entries[i] to temporarily store the
* branch counters information. The later code will
* decide what content can be disclosed to the perf
* tool. Pleae seeintel_pmu_lbr_counters_reorder(). */
.reserved = (info >>LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,
Quoted 6 lines
Normally
intel_pmu_lbr_counters_reorder()filters those bits and clears cpuc->lbr_entries[i].reserved to 0 before the entries are exposed. Since__intel_pmu_snapshot_branch_stack()copiescpuc->lbr_entriesdirectly into entries without callingintel_pmu_lbr_counters_reorder(), won't the raw hardware counter bits remain in the UAPI perf_branch_entry.reserved field returned bybpf_get_branch_snapshot()?
Currently the intel_pmu_lbr_filter() has already filtered the LBR entries which contain the sensitive information, like kernel address if only user space sampling is required. After the filtering, there should be no any sensitive data including the LBR counters. I suppose it can be exposed to user space.
Quoted 45 lines
arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2862
Quoted 12 lines
arch/x86/events/intel/core.c +10
-0+ +if (cpuc->lbr_pebs_users) { +memcpy (&cpuc->lbr_stack, &snapshot->stack, +sizeof (struct perf_branch_stack )); +memcpy (cpuc->lbr_entries, snapshot->entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (cpuc->lbr_counters, snapshot->counters, +sizeof (u64) * cpuc->lbr_stack.nr); + } +intel_pmu_enable_all (0 );[Severity: High]
Is it safe for__intel_pmu_snapshot_branch_stack()to unconditionally callintel_pmu_enable_all(0)when invoked frombpf_overflow_handler()in a PMI?In arch/x86/events/intel/core.c:intel_pmu_handle_irq(), the PMI handler saves
cpuc->enabled, setscpuc->enabled= 0, and disables the PMU in hardware:pmu_enabled = cpuc->enabled; ... cpuc->enabled = 0; __intel_pmu_disable_all(true);The PMI handler expects the PMU, PEBS, BTS, and LBRs to remain disabled in hardware while
handle_pmi_common()and__intel_pmu_pebs_last_event()drain PEBS records and reprogram counter periods, only restoring the hardware state at the done label ifpmu_enabledwas set:/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled; if (pmu_enabled) __intel_pmu_enable_all(0, true);When a BPF program runs via
perf_event_overflow()->bpf_overflow_handler()during a PMI and callsbpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack(), callingintel_pmu_enable_all(0)invokesx86_pmu_pebs_enable_all()and__intel_pmu_enable_all(0, false)whilecpuc->enabledis 0. This re-enablesMSR_IA32_PEBS_ENABLE, LBRs, BTS, andMSR_CORE_PERF_GLOBAL_CTRLbefore__intel_pmu_pebs_last_event()andhandle_pmi_common()finish reprogramming counter periods, and leaves the hardware PMU enabled after the PMI exits if the PMI interrupted anx86_pmu_disable()section wherepmu_enabledwas 0.
Quoted 2 lines
Should
__intel_pmu_snapshot_branch_stack()avoid callingintel_pmu_enable_all(0)whencpuc->enabledis 0?
Yeah, that's a real issue, would fix it in next version.
Quoted 45 lines
Quoted 5 lines
arch/x86/events/intel/core.c +0
-0local_irq_restore (flags);return cnt;@@ -4022,7 +4056,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)arch/x86/events/intel/core.c static int intel_pmu_handle_irq(struct pt_regs *regs) · line 4065
Quoted 10 lines
arch/x86/events/intel/core.c +6
-1loops =0 ; again: -intel_pmu_lbr_read (); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read ();intel_pmu_ack_status (status);[Severity: Medium]
Whencpuc->lbr_users==cpuc->lbr_pebs_usersand a BPF program callsbpf_get_branch_snapshot()frombpf_overflow_handler()during a PMI, haven't the hardware LBR registers already been unfrozen byintel_pmu_ack_status(status)?In arch/x86/events/intel/core.c:intel_pmu_handle_irq(),
__intel_pmu_disable_all(true)clearsMSR_CORE_PERF_GLOBAL_CTRLbut does not clear LBR enable bits inMSR_IA32_DEBUGCTLMSRorMSR_ARCH_LBR_CTL, relying onGLOBAL_STATUS_LBRS_FROZENto keep the LBR stack frozen.As noted in arch/x86/events/intel/lbr.c:__intel_pmu_lbr_enable():
/* * No need to unfreeze manually, as v4 can do that as part * of the GLOBAL_STATUS ack. */ if (pmi && x86_pmu.version >= 4) return;On PMU v4+ platforms (which include all
pebs_baselineplatforms), callingintel_pmu_ack_status(status)clearsGLOBAL_STATUS_LBRS_FROZENand immediately unfreezes the hardware LBRs. By the timehandle_pmi_common()->x86_pmu_drain_pebs()->__intel_pmu_pebs_last_event()sets up the PEBS sample and invokesbpf_overflow_handler()->__intel_pmu_snapshot_branch_stack(), the unfrozen hardware LBR MSRs have been recording branches inside the PMI handler itself (and__intel_pmu_snapshot_branch_stack()then also callsintel_pmu_enable_all(0)before the PMI finishes).
Quoted 3 lines
Can
__intel_pmu_snapshot_branch_stack()still obtain an unpolluted hardware LBR snapshot in this PMI path afterintel_pmu_ack_status(status)has unfrozen the LBRs?
Ah, it's indeed an issue. Would fix it in next version.
Thanks.
Quoted 3 lines
[ ... ]
Re: [Patch v4 5/5] pef/x86/intel: Always snapshot live hardware LBR state for BPF path
In reply to [email protected]Quoted 233 lines · On 10/21/2026 2:05 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] perf/x86/intel: LBR state corruption in__intel_pmu_snapshot_branch_stack()
- [High] perf/x86/intel:__intel_pmu_snapshot_branch_stack()enables PMU in PMI
- [Medium] perf/x86/intel: unfrozen LBRs and PMU re-enable in PMI branch snapshot
- [Medium] perf/x86: NMI race onlbr_snapshotin__intel_pmu_snapshot_branch_stack()
- [Medium] perf/x86/intel: unfiltered LBR branch counter bits in perf_branch_entry.reserved --commit
4d2b360d61a8a8f6825b3cd919182ee99dbe2d76
Author: Dapeng Mi <[email protected]>pef/x86/intel: Always snapshot live hardware LBR state for BPF path
This commit updates
__intel_pmu_snapshot_branch_stack()to always read live hardware LBR registers for the BPF snapshot path even when PEBS LBR users are active. To avoid clobbering cached PEBS LBR data during a PMI, it saves and restorescpuc->lbr_stack,cpuc->lbr_entries, andcpuc->lbr_countersaroundintel_pmu_lbr_read()using a per-CPU buffer, and moves thecpuc->lbr_users==cpuc->lbr_pebs_userscheck intointel_pmu_handle_irq().Quoted 5 lines
arch/x86/events/intel/core.c +0
-0arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2823
Quoted 9 lines
arch/x86/events/intel/core.c +6
-0@@ -2815,16 +2815,50 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (added, false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot);[Severity: Medium]
Can an NMI race on the per-CPUlbr_snapshotbuffer when__intel_pmu_snapshot_branch_stack()is called in non-NMI context?In
arch/x86/events/intel/core.c,intel_pmu_snapshot_branch_stack()andintel_pmu_snapshot_arch_branch_stack()calllocal_irq_save(flags)and__intel_pmu_disable_all(false)before invoking__intel_pmu_snapshot_branch_stack(). However,local_irq_save()only masks normal interrupts, and an in-flight PMI or another NMI can still arrive after__intel_pmu_disable_all()due to NMI delivery latency.When
cpuc->lbr_pebs_users> 0 and a non-NMI BPF program (such as an fentry/fexit trampoline via__bpf_prog_enter_recur()or araw_tracepoint, wherebpf_prog_activeis not incremented) calls:
bpf_get_branch_snapshot()-> intel_pmu_snapshot_branch_stack() -> __intel_pmu_snapshot_branch_stack()an arriving PMI can still invoke
bpf_overflow_handler()->bpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack()on the same CPU, overwritingthis_cpu_ptr(&lbr_snapshot)while the outer call is between its save and restore steps. Even if the PMI does not callbpf_get_branch_snapshot(),intel_pmu_handle_irq()updatescpuc->lbr_stackandcpuc->lbr_entrieswhile the interrupted snapshot is saving or restoring them.Since
cpuc->lbr_stackonly holds live sample data awaiting output during a PMI, should the save and restore ofcpuc->lbr_stackbe restricted to NMI context or otherwise protected against NMI reentrancy?Quoted 14 lines
arch/x86/events/intel/core.c +9
-0+ static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *snapshot =this_cpu_ptr (&lbr_snapshot); + +/* +* This may be called from bpf_event_handler() in a PMI context. If +* a PEBS LBR user is active, the PEBS sample record has already updated +* cpuc->lbr_stack. Reading the LBR registers here would overwrite that +* cached PEBS LBR stack, so save it first and restore it afterward. This +* keeps the cached LBR data available for later output to userspace in arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2847
Quoted 10 lines
arch/x86/events/intel/core.c +10
-0+* the normal PEBS record path. +*/ +if (cpuc->lbr_pebs_users) { +memcpy (&snapshot->stack, &cpuc->lbr_stack, +sizeof (struct perf_branch_stack )); +memcpy (snapshot->entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (snapshot->counters, cpuc->lbr_counters, +sizeof (u64) * cpuc->lbr_stack.nr); + }[Severity: Critical]
What happens in__intel_pmu_snapshot_branch_stack()whencpuc->lbr_usersis non-zero, butcpuc->lbr_pebs_usersis 0?In arch/x86/events/intel/lbr.c:intel_pmu_lbr_add(),
cpuc->lbr_pebs_usersis only incremented when x86_pmu.intel_cap.pebs_baseline is set andevent->attr.precise_ip> 0:if (x86_pmu.intel_cap.pebs_baseline && event->attr.precise_ip > 0) cpuc->lbr_pebs_users++;When non-PEBS hardware LBR events (
precise_ip== 0), legacy PEBS LBR events (pre-Ice Lake wherepebs_baseline== 0), or Architectural PEBS LBR events (arch/x86/events/intel/ds.c:setup_arch_pebs_sample_data(), wherepebs_baseline== 0) are active,cpuc->lbr_users> 0 whilecpuc->lbr_pebs_users== 0.In those paths,
handle_pmi_common(),setup_pebs_fixed_sample_data(), andsetup_arch_pebs_sample_data()callintel_pmu_lbr_save_brstack()beforeperf_event_overflow()->__perf_event_overflow()->bpf_overflow_handler():if (has_branch_stack(event)) intel_pmu_lbr_save_brstack(&data, cpuc, event); ... perf_event_overflow(event, &data, regs);Inside include/linux/perf_event.h:perf_sample_save_brstack(), brs (
&cpuc->lbr_stack) is clamped in place and used to precomputedata->dyn_size:brs->nr = min_t(u16, event->attr.sample_max_stack, brs->nr); size += brs->nr * sizeof(struct perf_branch_entry); if (brs_cntr) size += brs->nr * sizeof(u64); data->br_stack = brs; data->br_stack_cntr = brs_cntr; data->dyn_size += size; data->sample_flags |= PERF_SAMPLE_BRANCH_STACK;If
bpf_overflow_handler()then callsbpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack()withcpuc->lbr_pebs_users== 0, the save and restore ofcpuc->lbr_stack,cpuc->lbr_entries, andcpuc->lbr_countersare skipped, andintel_pmu_lbr_read()overwrites them in place:1. In kernel/events/core.c:perf_output_sample(),
data->br_stackstill points to&cpuc->lbr_stack. Becauseintel_pmu_lbr_read()resetscpuc->lbr_stack.nrup to x86_pmu.lbr_nr, undoing thesample_max_stackclamp afterdata->dyn_sizewas already computed,perf_output_sample()can copy moreperf_branch_entry(and counter) elements thanperf_output_begin()reserved in the ring buffer.2. The frozen LBR entries captured at PMI entry (or from the Arch PEBS record) in
cpuc->lbr_entriesare overwritten by unfrozen post-ack LBR registers containing PMI handler branches.3. For branch counter groups,
intel_pmu_lbr_counters_reorder()has already extracted counter bits from cpuc->lbr_entries[i].reserved into cpuc->lbr_counters[i] and cleared .reserved; callingintel_pmu_lbr_read()->intel_pmu_store_lbr()writes raw counter bits back into cpuc->lbr_entries[i].reserved and desynchronizes cpuc->lbr_counters[].Should the save and restore in
__intel_pmu_snapshot_branch_stack()be gated oncpuc->lbr_usersinstead ofcpuc->lbr_pebs_users(and save/restore based on the pre-clamp entry count or full array when needed)?arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2850
Quoted 5 lines
arch/x86/events/intel/core.c +0
-1intel_pmu_lbr_read (); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt);[Severity: Medium]
On platforms with Architectural LBR and branch counter logging, does copyingcpuc->lbr_entriesdirectly afterintel_pmu_lbr_read()leak raw branch counter bits in entries[i].reserved to BPF programs?When
intel_pmu_snapshot_arch_branch_stack()calls__intel_pmu_snapshot_branch_stack(),intel_pmu_lbr_read()invokesintel_pmu_arch_lbr_read()orintel_pmu_arch_lbr_read_xsave(), which calls arch/x86/events/intel/lbr.c:intel_pmu_store_lbr():/*
* Leverage the reserved field of
* cpuc->lbr_entries[i] to temporarily store the
* branch counters information. The later code will
* decide what content can be disclosed to the perf
* tool. Pleae seeintel_pmu_lbr_counters_reorder(). */
.reserved = (info >>LBR_INFO_BR_CNTR_OFFSET) &
LBR_INFO_BR_CNTR_FULL_MASK,Normally
intel_pmu_lbr_counters_reorder()filters those bits and clears cpuc->lbr_entries[i].reserved to 0 before the entries are exposed. Since__intel_pmu_snapshot_branch_stack()copiescpuc->lbr_entriesdirectly into entries without callingintel_pmu_lbr_counters_reorder(), won't the raw hardware counter bits remain in the UAPI perf_branch_entry.reserved field returned bybpf_get_branch_snapshot()?arch/x86/events/intel/core.c static void intel_pmu_enable_all(int added) · line 2861
Quoted 11 lines
arch/x86/events/intel/core.c +10
-0+ +if (cpuc->lbr_pebs_users) { +memcpy (&cpuc->lbr_stack, &snapshot->stack, +sizeof (struct perf_branch_stack )); +memcpy (cpuc->lbr_entries, snapshot->entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (cpuc->lbr_counters, snapshot->counters, +sizeof (u64) * cpuc->lbr_stack.nr); + } +intel_pmu_enable_all (0 );[Severity: High]
Is it safe for__intel_pmu_snapshot_branch_stack()to unconditionally callintel_pmu_enable_all(0)when invoked frombpf_overflow_handler()in a PMI?In arch/x86/events/intel/core.c:intel_pmu_handle_irq(), the PMI handler saves
cpuc->enabled, setscpuc->enabled= 0, and disables the PMU in hardware:pmu_enabled = cpuc->enabled; ... cpuc->enabled = 0; __intel_pmu_disable_all(true);The PMI handler expects the PMU, PEBS, BTS, and LBRs to remain disabled in hardware while
handle_pmi_common()and__intel_pmu_pebs_last_event()drain PEBS records and reprogram counter periods, only restoring the hardware state at the done label ifpmu_enabledwas set:/* Only restore PMU state when it's active. See x86_pmu_disable(). */ cpuc->enabled = pmu_enabled; if (pmu_enabled) __intel_pmu_enable_all(0, true);When a BPF program runs via
perf_event_overflow()->bpf_overflow_handler()during a PMI and callsbpf_get_branch_snapshot()->__intel_pmu_snapshot_branch_stack(), callingintel_pmu_enable_all(0)invokesx86_pmu_pebs_enable_all()and__intel_pmu_enable_all(0, false)whilecpuc->enabledis 0. This re-enablesMSR_IA32_PEBS_ENABLE, LBRs, BTS, andMSR_CORE_PERF_GLOBAL_CTRLbefore__intel_pmu_pebs_last_event()andhandle_pmi_common()finish reprogramming counter periods, and leaves the hardware PMU enabled after the PMI exits if the PMI interrupted anx86_pmu_disable()section wherepmu_enabledwas 0.
Quoted 2 lines
Should
__intel_pmu_snapshot_branch_stack()avoid callingintel_pmu_enable_all(0)whencpuc->enabledis 0?
Hi Sandipan,
Although this issue is reported for Intel platform, it looks AMD platform has similar issue. To ensure AMD and Intel platform has consistent behavior, the below changes are made for AMD platform as well. Could you please double check if it's correct for AMD platform? Thanks.
arch/x86/events/amd/core.c +0 -0
@@ -930,24 +930,33 @@ static int amd_pmu_handle_irq(struct pt_regs *regs)
*/
static int amd_pmu_v2_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
{
- struct cpu_hw_events *cpuc;
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); unsigned long flags;
+ int pmu_enabled;
/*
* The sequence of steps to freeze LBR should be completely inlined
* and contain no branches to minimize contamination of LBR snapshot */
local_irq_save(flags);
+ pmu_enabled = cpuc->enabled;
amd_pmu_core_disable_all();
__amd_pmu_lbr_disable();
- cpuc = this_cpu_ptr(&cpu_hw_events);
-
amd_pmu_lbr_read();
cnt = min(cnt, cpuc->lbr_stack.nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry)
* cnt);
- amd_pmu_v2_enable_all(0);
+ if (pmu_enabled) {
+ amd_pmu_v2_enable_all(0);
+ } else {
+ /*
+ * When pmu_enabled is false, we are probably in the PMI handler.
+ * LBR is frozen in PMI context, and there is no later explicit + * LBR enable before the PMI returns, so re-enable it here. + */
+ amd_pmu_lbr_enable_all();
+ }
local_irq_restore(flags);
return cnt;
Quoted 51 lines
Quoted 5 lines
arch/x86/events/intel/core.c +0
-0local_irq_restore (flags);return cnt;@@ -4022,7 +4056,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)arch/x86/events/intel/core.c static int intel_pmu_handle_irq(struct pt_regs *regs) · line 4065
Quoted 10 lines
arch/x86/events/intel/core.c +6
-1loops =0 ; again: -intel_pmu_lbr_read (); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read ();intel_pmu_ack_status (status);[Severity: Medium]
Whencpuc->lbr_users==cpuc->lbr_pebs_usersand a BPF program callsbpf_get_branch_snapshot()frombpf_overflow_handler()during a PMI, haven't the hardware LBR registers already been unfrozen byintel_pmu_ack_status(status)?In arch/x86/events/intel/core.c:intel_pmu_handle_irq(),
__intel_pmu_disable_all(true)clearsMSR_CORE_PERF_GLOBAL_CTRLbut does not clear LBR enable bits inMSR_IA32_DEBUGCTLMSRorMSR_ARCH_LBR_CTL, relying onGLOBAL_STATUS_LBRS_FROZENto keep the LBR stack frozen.As noted in arch/x86/events/intel/lbr.c:__intel_pmu_lbr_enable():
/* * No need to unfreeze manually, as v4 can do that as part * of the GLOBAL_STATUS ack. */ if (pmi && x86_pmu.version >= 4) return;On PMU v4+ platforms (which include all
pebs_baselineplatforms), callingintel_pmu_ack_status(status)clearsGLOBAL_STATUS_LBRS_FROZENand immediately unfreezes the hardware LBRs. By the timehandle_pmi_common()->x86_pmu_drain_pebs()->__intel_pmu_pebs_last_event()sets up the PEBS sample and invokesbpf_overflow_handler()->__intel_pmu_snapshot_branch_stack(), the unfrozen hardware LBR MSRs have been recording branches inside the PMI handler itself (and__intel_pmu_snapshot_branch_stack()then also callsintel_pmu_enable_all(0)before the PMI finishes).Can
__intel_pmu_snapshot_branch_stack()still obtain an unpolluted hardware LBR snapshot in this PMI path afterintel_pmu_ack_status(status)has unfrozen the LBRs?[ ... ]
Re: [Patch v4 5/5] pef/x86/intel: Always snapshot live hardware LBR state for BPF path
In reply to Dapeng MiSashiko found several new long-standing LBR issues (not caused by the patch series), https://lore.kernel.org/all/[email protected]/
Would post new patch series to fix the newly found issues. Thanks.
Quoted 107 lines · Dapeng Mi
__intel_pmu_snapshot_branch_stack()is meant to snapshot the current runtime LBR state from hardware for the BPF path. However, stale cached PEBS LBR data can be snapshotted instead when a PEBS LBR user is active.Fix this by always snapshotting the live hardware LBR state, regardless of whether a PEBS LBR user is active. Reading the runtime LBR registers can overwrite the cached PEBS LBR stack, so save and restore the cached PEBS LBR state around the snapshot. This preserves the PEBS LBR data for later output to userspace in the normal PEBS record path.
Fixes c22ac2a3d4bd ("perf: Enable branch record for software events")Signed-off-by Dapeng Mi <[email protected]>--- arch/x86/events/intel/core.c | 43 ++++++++++++++++++++++++++++++++++-- arch/x86/events/intel/lbr.c | 9 -------- 2 files changed, 41 insertions(+), 11 deletions(-)arch/x86/events/intel/core.c +41
-2@@ -2815,16 +2815,50 @@ static void intel_pmu_enable_all(int added)__intel_pmu_enable_all (added, false); } +struct lbr_snapshot { +struct perf_branch_stack stack; +struct perf_branch_entry entries[MAX_LBR_ENTRIES ]; +u64 counters[MAX_LBR_ENTRIES ]; +}; +static DEFINE_PER_CPU(struct lbr_snapshot, lbr_snapshot); + static noinline int__intel_pmu_snapshot_branch_stack (struct perf_branch_entry *entries,unsigned int cnt,unsigned long flags) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +struct lbr_snapshot *snapshot =this_cpu_ptr (&lbr_snapshot); + +/* +* This may be called from bpf_event_handler() in a PMI context. If +* a PEBS LBR user is active, the PEBS sample record has already updated +* cpuc->lbr_stack. Reading the LBR registers here would overwrite that +* cached PEBS LBR stack, so save it first and restore it afterward. This +* keeps the cached LBR data available for later output to userspace in +* the normal PEBS record path. +*/ +if (cpuc->lbr_pebs_users) { +memcpy (&snapshot->stack, &cpuc->lbr_stack, +sizeof (struct perf_branch_stack )); +memcpy (snapshot->entries, cpuc->lbr_entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (snapshot->counters, cpuc->lbr_counters, +sizeof (u64) * cpuc->lbr_stack.nr); + }intel_pmu_lbr_read (); cnt = min_t(unsigned int, cnt, cpuc->lbr_stack.nr); -memcpy (entries, cpuc->lbr_entries,sizeof (struct perf_branch_entry ) * cnt); + +if (cpuc->lbr_pebs_users) { +memcpy (&cpuc->lbr_stack, &snapshot->stack, +sizeof (struct perf_branch_stack )); +memcpy (cpuc->lbr_entries, snapshot->entries, +sizeof (struct perf_branch_entry ) * cpuc->lbr_stack.nr); +memcpy (cpuc->lbr_counters, snapshot->counters, +sizeof (u64) * cpuc->lbr_stack.nr); + } +intel_pmu_enable_all (0 );local_irq_restore (flags);return cnt;@@ -4022,7 +4056,12 @@ static int intel_pmu_handle_irq(struct pt_regs *regs) loops =0 ; again: -intel_pmu_lbr_read (); +/* +* No need to read the LBR stack when all LBR users are PEBS events, +* since the PEBS handler updates it. +*/ +if (cpuc->lbr_users != cpuc->lbr_pebs_users) +intel_pmu_lbr_read ();intel_pmu_ack_status (status);if (++loops >100 ) {static bool warned;arch/x86/events/intel/lbr.c +0
-9@@ -1030,15 +1030,6 @@ void intel_pmu_lbr_read(void)return ; } -/* -* Don't read when all LBRs users are using adaptive PEBS. -* -* This could be smarter and actually check the event, -* but this simple approach seems to work for now. -*/ -if (cpuc->lbr_users == cpuc->lbr_pebs_users) -return ; - x86_pmu.lbr_read (cpuc);intel_pmu_lbr_filter (cpuc);