perf/core: sched_task() dispatch and branch entry fixes
Subsystems (5)
v7 cover perf/core: sched_task() dispatch and branch entry fixes
These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.
Patch 1 stops __perf_pmu_sched_task() passing a NULL pmu_ctx to pmu->sched_task(). armv8pmu_sched_task() is the only implementation that dereferences the argument, so the oops needs BRBE.
Patch 2 makes perf_pmu_sched_task() visit PMUs whose events are all CPU-wide. They are skipped today on every switch to a task that has a perf context but no event on that PMU, so branch records leak across the task boundary with perf record -b -a. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally, so x86 LBR is affected the same way.
Patch 3 has each caller fill struct perf_branch_entry in one assignment rather than clearing the bitfields first, and drops perf_clear_branch_entry_bitfields(). The helper had drifted from the
struct: new_type and priv were never cleared, and arm_pmuv3.c allocates the per-CPU branch stack with kmalloc().
Tested on a 128 CPU arm64 machine with BRBE. A WARN_ON_ONCE() at the gate patch 2 adds to perf_ctx_sched_task_cb() fires within seconds of running perf record -b -a alongside a task-bound event pinned to a different CPU. Also built for x86, which patch 3 touches.
Changes in v7:
- Patch 1: pass &cpc->epc unconditionally. cpc->task_epc is NULL there both before and after patch 2, so the conditional was dead.
- Patch 2: gate perf_pmu_sched_task() on cpc->task_epc alone. It is never set without a task context scheduled in, so the cpuctx->task_ctx test was redundant and the two gates are now inverses.
- Patch 3: fill the entry at each site instead of renaming the helper, as suggested by Peter.
- Dropped the v6 review tags, all three patches changed.
Changes in v6:
- Split the sched_task() fix into patches 1 and 2; the NULL dereference and the missed dispatch are separate bugs with different reachability.
- Gate perf_ctx_sched_task_cb() on cpc->task_epc. v5 removed the early return in perf_pmu_sched_task() without it, so both paths ran for a task whose event for that PMU is pinned to another CPU. Caught by the WARN_ON_ONCE() described above.
- Tag patches 1 and 2 for stable.
- Send separately from the BRBE series, rebased onto tip perf/core.
[1] https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/bpf/[email protected]/
Puranjay Mohan (3):
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() perf/core: Run sched_task() for PMUs with only CPU-wide events perf/core: Fill branch entries with a single assignment
arch/x86/events/amd/brs.c | 9 +++-- arch/x86/events/amd/lbr.c | 16 ++++----- arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++----------------- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 17 ---------- kernel/events/core.c | 15 ++++++--- 6 files changed, 58 insertions(+), 66 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH v7 1/3 perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. __perf_pmu_sched_task() therefore always passes NULL:
Unable to handle kernel NULL pointer dereference at virtual address 00 pc : armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0
Pass &cpc->epc instead, the CPU-wide context for this PMU, which the function already dereferences a few lines up to find pmu.
armv8pmu_sched_task() is the only in-tree implementation that dereferences the argument, and it only reads ->pmu, so the oops needs BRBE, added in v6.17.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (&cpc->epc, task, sched_in);perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Per-CPU PMUs missing sched_task callbacks during context switches if a task context exists. --
--- Patch [1]: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() --- commit 333df3e02c2d6b54e5bd528f91779820677d118e
Author: Puranjay Mohan <[email protected]>
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. This patch passes &cpc->epc instead, which is the CPU-wide context for this PMU, avoiding a NULL pointer dereference.
kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3910
Quoted 10 lines
kernel/events/core.c +1
-1@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (&cpc->epc, task, sched_in);
[Severity: High]
This isn't a bug introduced by this patch, but does the early return in perf_pmu_sched_task() skip callbacks for CPU-wide PMUs?
If a system-wide PMU event is active (sched_cb_usage > 0), and a context switch occurs to or from a task with an active perf event (meaning cpuctx->task_ctx is set), perf_pmu_sched_task() returns early:
kernel/events/core.c:perf_pmu_sched_task() {
...
/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
if (prev == next || cpuctx->task_ctx)
return;
...
}It seems to rely on perf_event_context_sched_out/in() to handle the callbacks. However, perf_ctx_sched_task_cb() only iterates over the task's pmu_ctx_list:
kernel/events/core.c:perf_ctx_sched_task_cb() {
...
list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
cpc = this_cpc(pmu_ctx->pmu);
if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
}
...
}If the task's perf event is for a different PMU (like a software event), the CPU-wide PMU won't be in ctx->pmu_ctx_list. Does this mean the CPU-wide PMU's sched_task() callback is completely skipped during the context switch, potentially corrupting hardware tracing or leaking branch records across tasks?
Quoted 3 lines
kernel/events/core.c +0
-0perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
Quoted 60 lines · On Mon, Aug 10, 2026 at 3:03 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Per-CPU PMUs missingsched_taskcallbacks during context switches if a task context exists. ----- Patch [1]: [PATCH v7 1/3] perf/core: Fix NULL
pmu_ctxpassed topmu->sched_task()--- commit333df3e02c2d6b54e5bd528f91779820677d118e
Author: Puranjay Mohan <[email protected]>perf/core: Fix NULL
pmu_ctxpassed topmu->sched_task()
perf_pmu_sched_task()returns early whencpuctx->task_ctxis set, andcpc->task_epcis only non-NULL while a task context is scheduled in on this CPU. This patch passes&cpc->epcinstead, which is the CPU-wide context for this PMU, avoiding a NULL pointer dereference.kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3910
Quoted 10 lines
kernel/events/core.c +1
-1@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (&cpc->epc, task, sched_in);[Severity: High]
This isn't a bug introduced by this patch, but does the early return inperf_pmu_sched_task()skip callbacks for CPU-wide PMUs?If a system-wide PMU event is active (
sched_cb_usage> 0), and a context switch occurs to or from a task with an active perf event (meaningcpuctx->task_ctxis set),perf_pmu_sched_task()returns early:kernel/events/core.c:perf_pmu_sched_task() { ... /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ if (prev == next || cpuctx->task_ctx) return; ... }It seems to rely on perf_event_context_sched_out/in() to handle the callbacks. However,
perf_ctx_sched_task_cb()only iterates over the task'spmu_ctx_list:kernel/events/core.c:perf_ctx_sched_task_cb() { ... list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc = this_cpc(pmu_ctx->pmu); if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in); } ... }
Quoted 8 lines
If the task's perf event is for a different PMU (like a software event), the CPU-wide PMU won't be in
ctx->pmu_ctx_list. Does this mean the CPU-wide PMU'ssched_task()callback is completely skipped during the context switch, potentially corrupting hardware tracing or leaking branch records across tasks?Quoted 3 lines
kernel/events/core.c +0
-0perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
Next patch fixes this exact issue.
[tip: perf/urgent] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
In reply to Puranjay MohanThe following commit has been merged into the perf/urgent branch of tip:
The following commit has been merged into the perf/urgent branch of tip:
Commit-ID: 36bb85cf36cab15fb611cb44b78a5df06e4e69a2 Gitweb: https://git.kernel.org/tip/36bb85cf36cab15fb611cb44b78a5df06e4e69a2 Author: Puranjay Mohan <[email protected]> AuthorDate: Mon, 10 Aug 2026 06:35:34 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 23 Sep 2026 11:48:35 +02:00
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. __perf_pmu_sched_task() therefore always passes NULL:
Unable to handle kernel NULL pointer dereference at virtual address 00 pc : armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0
Pass &cpc->epc instead, the CPU-wide context for this PMU, which the function already dereferences a few lines up to find pmu.
armv8pmu_sched_task() is the only in-tree implementation that dereferences the argument, and it only reads ->pmu, so the oops needs BRBE, added in v6.17.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -3914,7 +3914,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (&cpc->epc, task, sched_in);perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
PATCH v7 2/3 perf/core: Run sched_task() for PMUs with only CPU-wide events
In reply to Puranjay Mohanperf_pmu_sched_task() returns early when cpuctx->task_ctx is set and leaves the work to perf_ctx_sched_task_cb(), which only walks ctx->pmu_ctx_list. A PMU whose events are all CPU-wide is not on that list, so nothing calls its sched_task(). With
perf record -b -e cycles -a -- ls
armv8pmu_sched_task() is skipped on every switch to a task that has a perf context but no event on that PMU, and BRBE records leak across the task boundary. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally too, so LBR records leak the same way on x86.
Drop the early return and skip only the CPCs that perf_ctx_sched_task_cb() handles. That one needs a gate of its own to make the split exact: it tests cpc->sched_cb_usage, which perf_sched_cb_inc() sets per CPU for every branch stack user, so a task with an event for that PMU pinned to another CPU would be handled twice. On x86 the second __intel_pmu_lbr_restore() finds lbr_stack_state == LBR_NONE and calls intel_pmu_lbr_reset(), throwing away the callstack the first one restored.
cpc->task_epc is set only while a task context is scheduled in, and there is one epc per PMU on ctx->pmu_ctx_list, so the two gates are inverses.
For the CPCs perf_pmu_sched_task() picks up, the callback now runs outside the perf_ctx_disable() and perf_ctx_enable() pair in perf_event_context_sched_in(). __perf_pmu_sched_task() disables the PMU around the call itself.
--- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
@@ -3757,6 +3757,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc =this_cpc (pmu_ctx->pmu); +if (cpc->task_epc != pmu_ctx) +continue ; +if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, task, sched_in); }
@@ -3917,15 +3920,17 @@ static void perf_pmu_sched_task(struct task_struct *prev, struct task_struct *next, bool sched_in) { -struct perf_cpu_context *cpuctx =this_cpu_ptr (&perf_cpu_context);struct perf_cpu_pmu_context *cpc; -/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ -if (prev == next || cpuctx->task_ctx) +if (prev == next)return ; -list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) +list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) { +if (cpc->task_epc) +continue ; +__perf_pmu_sched_task (cpc, sched_in ? next : prev, sched_in); + } } static void perf_event_switch(struct task_struct *task,
Signature · 1 line
2.53.0-Meta
[tip: perf/urgent] perf/core: Run sched_task() for PMUs with only CPU-wide events
The following commit has been merged into the perf/urgent branch of tip:
The following commit has been merged into the perf/urgent branch of tip:
Commit-ID: 3d8d74100954a3b17e5c5e37adfe14e16b1db103 Gitweb: https://git.kernel.org/tip/3d8d74100954a3b17e5c5e37adfe14e16b1db103 Author: Puranjay Mohan <[email protected]> AuthorDate: Mon, 10 Aug 2026 06:35:35 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 23 Sep 2026 11:48:35 +02:00
perf/core: Run sched_task() for PMUs with only CPU-wide events
perf_pmu_sched_task() returns early when cpuctx->task_ctx is set and leaves the work to perf_ctx_sched_task_cb(), which only walks ctx->pmu_ctx_list. A PMU whose events are all CPU-wide is not on that list, so nothing calls its sched_task(). With
perf record -b -e cycles -a -- ls
armv8pmu_sched_task() is skipped on every switch to a task that has a perf context but no event on that PMU, and BRBE records leak across the task boundary. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally too, so LBR records leak the same way on x86.
Drop the early return and skip only the CPCs that perf_ctx_sched_task_cb() handles. That one needs a gate of its own to make the split exact: it tests cpc->sched_cb_usage, which perf_sched_cb_inc() sets per CPU for every branch stack user, so a task with an event for that PMU pinned to another CPU would be handled twice. On x86 the second __intel_pmu_lbr_restore() finds lbr_stack_state == LBR_NONE and calls intel_pmu_lbr_reset(), throwing away the callstack the first one restored.
cpc->task_epc is set only while a task context is scheduled in, and there is one epc per PMU on ctx->pmu_ctx_list, so the two gates are inverses.
For the CPCs perf_pmu_sched_task() picks up, the callback now runs outside the perf_ctx_disable() and perf_ctx_enable() pair in perf_event_context_sched_in(). __perf_pmu_sched_task() disables the PMU around the call itself.
--- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
@@ -3764,6 +3764,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc =this_cpc (pmu_ctx->pmu); +if (cpc->task_epc != pmu_ctx) +continue ; +if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, task, sched_in); }
@@ -3924,15 +3927,17 @@ static void perf_pmu_sched_task(struct task_struct *prev, struct task_struct *next, bool sched_in) { -struct perf_cpu_context *cpuctx =this_cpu_ptr (&perf_cpu_context);struct perf_cpu_pmu_context *cpc, *cpc2; -/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ -if (prev == next || cpuctx->task_ctx) +if (prev == next)return ; -list_for_each_entry_safe (cpc, cpc2,this_cpu_ptr (&sched_cb_list), sched_cb_entry) +list_for_each_entry_safe (cpc, cpc2,this_cpu_ptr (&sched_cb_list), sched_cb_entry) { +if (cpc->task_epc) +continue ; +__perf_pmu_sched_task (cpc, sched_in ? next : prev, sched_in); + } } static void perf_event_switch(struct task_struct *task,
PATCH v7 3/3 perf/core: Fill branch entries with a single assignment
In reply to Puranjay Mohanperf_clear_branch_entry_bitfields() clears the bitfields of struct perf_branch_entry one by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind: new_type and priv were added to perf_branch_entry and never added here.
Only BRBE writes those two, and neither for every record. brbe_set_perf_entry_type() leaves new_type alone for a branch type it does not recognise, and priv is not set for source-only records. arm_pmuv3.c allocates the per-CPU branch stack with kmalloc(), so such a record reaches userspace with whatever the slot held: uninitialised kmalloc() data on the first pass over the buffer, the previous record's values after that. Nothing under arch/x86/events/ writes either field, so only arm64 is affected.
Assign the whole entry at each site instead. Everything not named is then zero, and there is no list to keep in sync. The bitfields add up to exactly 64 bits, so the struct has no padding to leave undefined.
perf_clear_branch_entry_bitfields() has no callers left, so remove it. perf_entry_from_brbe_regset() assigns an empty literal instead, since it fills from/to conditionally. PERF_BR_SPEC_NA is 0, so dropping the explicit spec assignment changes nothing.
--- arch/x86/events/amd/brs.c | 9 +++-- arch/x86/events/amd/lbr.c | 16 ++++----- arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++----------------- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 17 ---------- 5 files changed, 48 insertions(+), 61 deletions(-)
arch/x86/events/amd/brs.c +4 -5
@@ -343,11 +343,10 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; -perf_clear_branch_entry_bitfields (br+nr); - - br[nr].from = from; - br[nr].to = to; - + br[nr] = (struct perf_branch_entry ){ + .from = from, + .to = to, + }; nr++; } empty:
arch/x86/events/amd/lbr.c +8 -8
@@ -184,13 +184,6 @@ void amd_pmu_lbr_read(void) entry.to.split.reserved)continue ; -perf_clear_branch_entry_bitfields (br + out); - - br[out].from =sign_ext_branch_ip (entry.from.split.ip); - br[out].to =sign_ext_branch_ip (entry.to.split.ip); - br[out].mispred = entry.from.split.mispredict; - br[out].predicted = !br[out].mispred; -/* * Set branch speculation information using the status of * the valid and spec bits.
@@ -208,7 +201,14 @@ void amd_pmu_lbr_read(void)* speculative and took the correct path */ idx = (entry.to.split.valid <<1 ) | entry.to.split.spec; - br[out].spec = lbr_spec_map[idx]; + + br[out] = (struct perf_branch_entry ){ + .from =sign_ext_branch_ip (entry.from.split.ip), + .to =sign_ext_branch_ip (entry.to.split.ip), + .mispred = entry.from.split.mispredict, + .predicted = !entry.from.split.mispredict, + .spec = lbr_spec_map[idx], + }; out++; }
arch/x86/events/intel/lbr.c +35 -30
89 lines not on this page · raw message
drivers/perf/arm_brbe.c +1 -1
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *entreturn false; brbinf = bregs.brbinf; -perf_clear_branch_entry_bitfields (entry); + *entry = (struct perf_branch_entry ){ };if (brbe_record_is_complete (brbinf)) { entry->from = bregs.brbsrc; entry->to = bregs.brbtgt;
include/linux/perf_event.h +0 -17
@@ -1467,23 +1467,6 @@ static inline u32 perf_sample_data_size(struct perf_sample_data *data,return size; } -/* -* Clear all bitfields in the perf_branch_entry. -* The to and from fields are not cleared because they are -* systematically modified by caller. -*/ -static inline void perf_clear_branch_entry_bitfields (struct perf_branch_entry *br) -{ - br->mispred =0 ; - br->predicted =0 ; - br->in_tx =0 ; - br->abort =0 ; - br->cycles =0 ; - br->type =0 ; - br->spec =PERF_BR_SPEC_NA ; - br->reserved =0 ; -} - extern void perf_output_sample(struct perf_output_handle *handle, struct perf_event_header *header, struct perf_sample_data *data,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v7 3/3] perf/core: Fill branch entries with a single assignment
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The BPF branch snapshot helpers (__intel_pmu_snapshot_branch_stack and amd_pmu_v2_snapshot_branch_stack) unconditionally copy stale branch records (LBRs) to BPF buffers, leading to cross-context information leaks. --
--- Patch [3]: [PATCH v7 3/3] perf/core: Fill branch entries with a single assignment --- commit f67f9ecd08216c6081d3d24e81906c232c6917fe
Author: Puranjay Mohan <[email protected]>
perf/core: Fill branch entries with a single assignment
This commit replaces the field-by-field initialization of perf_branch_entry with a single structure assignment. This automatically zeroes out any unnamed fields, removing the need to manually keep perf_clear_branch_entry_bitfields in sync with the structure definition.
Quoted 4 lines
arch/x86/events/intel/lbr.c +0
-0
[ ... ]
Quoted 14 lines
arch/x86/events/intel/lbr.c +2
-8@@ -847,14 +847,15 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)if (abort && x86_pmu.lbr_double_abort && out >0 ) out--; -perf_clear_branch_entry_bitfields (br+out); - br[out].from = from; - br[out].to = to; - br[out].mispred = mis; - br[out].predicted = pred; - br[out].in_tx = in_tx; - br[out].abort = abort; - br[out].cycles = cycles; + br[out] = (struct perf_branch_entry ){ + .from = from,
arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) · line 861
Quoted 10 lines
arch/x86/events/intel/lbr.c +7
-0+ .to = to, + .mispred = mis, + .predicted = pred, + .in_tx = in_tx, + .abort = abort, + .cycles = cycles, + }; out++; } cpuc->lbr_stack.nr = out;
[Severity: High]
This is a pre-existing issue, but while reviewing how these branch entries are populated and tracked by cpuc->lbr_stack.nr, I noticed a potential information leak in the BPF branch snapshot helpers.
Does __intel_pmu_snapshot_branch_stack() unconditionally copy stale branch records to the BPF buffer?
arch/x86/events/intel/core.c:__intel_pmu_snapshot_branch_stack() {
...
cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
...
}When the number of valid LBR entries (cpuc->lbr_stack.nr) is less than the hardware capacity (x86_pmu.lbr_nr), this memcpy appears to copy the trailing unwritten entries from cpuc->lbr_entries. This condition can happen routinely in call-stack mode with shallow stacks, or when Intel's double-abort erratum skips an entry (as seen just above in this diff).
The AMD implementation seems to have the exact same logic error:
arch/x86/events/amd/core.c:amd_pmu_v2_snapshot_branch_stack() {
...
cnt = min(cnt, x86_pmu.lbr_nr);
memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
...
}Could this allow a BPF tracing program to read stale hardware branch records left over from previously executed contexts, leaking execution history from other tasks or the kernel?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[tip: perf/urgent] perf/core: Fill branch entries with a single assignment
In reply to Puranjay MohanThe following commit has been merged into the perf/urgent branch of tip:
The following commit has been merged into the perf/urgent branch of tip:
Commit-ID: 24b620729e53d978b3e425f55bc66efd3bab1f59 Gitweb: https://git.kernel.org/tip/24b620729e53d978b3e425f55bc66efd3bab1f59 Author: Puranjay Mohan <[email protected]> AuthorDate: Mon, 10 Aug 2026 06:35:36 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 23 Sep 2026 11:48:36 +02:00
perf/core: Fill branch entries with a single assignment
perf_clear_branch_entry_bitfields() clears the bitfields of struct perf_branch_entry one by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind: new_type and priv were added to perf_branch_entry and never added here.
Only BRBE writes those two, and neither for every record. brbe_set_perf_entry_type() leaves new_type alone for a branch type it does not recognise, and priv is not set for source-only records. arm_pmuv3.c allocates the per-CPU branch stack with kmalloc(), so such a record reaches userspace with whatever the slot held: uninitialised kmalloc() data on the first pass over the buffer, the previous record's values after that. Nothing under arch/x86/events/ writes either field, so only arm64 is affected.
Assign the whole entry at each site instead. Everything not named is then zero, and there is no list to keep in sync. The bitfields add up to exactly 64 bits, so the struct has no padding to leave undefined.
perf_clear_branch_entry_bitfields() has no callers left, so remove it. perf_entry_from_brbe_regset() assigns an empty literal instead, since it fills from/to conditionally. PERF_BR_SPEC_NA is 0, so dropping the explicit spec assignment changes nothing.
--- arch/x86/events/amd/brs.c | 9 ++--- arch/x86/events/amd/lbr.c | 16 ++++----- arch/x86/events/intel/lbr.c | 65 +++++++++++++++++++----------------- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 17 +--------- 5 files changed, 48 insertions(+), 61 deletions(-)
arch/x86/events/amd/brs.c +4 -5
@@ -343,11 +343,10 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; -perf_clear_branch_entry_bitfields (br+nr); - - br[nr].from = from; - br[nr].to = to; - + br[nr] = (struct perf_branch_entry ){ + .from = from, + .to = to, + }; nr++; } empty:
arch/x86/events/amd/lbr.c +8 -8
@@ -184,13 +184,6 @@ void amd_pmu_lbr_read(void) entry.to.split.reserved)continue ; -perf_clear_branch_entry_bitfields (br + out); - - br[out].from =sign_ext_branch_ip (entry.from.split.ip); - br[out].to =sign_ext_branch_ip (entry.to.split.ip); - br[out].mispred = entry.from.split.mispredict; - br[out].predicted = !br[out].mispred; -/* * Set branch speculation information using the status of * the valid and spec bits.
@@ -208,7 +201,14 @@ void amd_pmu_lbr_read(void)* speculative and took the correct path */ idx = (entry.to.split.valid <<1 ) | entry.to.split.spec; - br[out].spec = lbr_spec_map[idx]; + + br[out] = (struct perf_branch_entry ){ + .from =sign_ext_branch_ip (entry.from.split.ip), + .to =sign_ext_branch_ip (entry.to.split.ip), + .mispred = entry.from.split.mispredict, + .predicted = !entry.from.split.mispredict, + .spec = lbr_spec_map[idx], + }; out++; }
arch/x86/events/intel/lbr.c +35 -30
89 lines not on this page · raw message
drivers/perf/arm_brbe.c +1 -1
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *entreturn false; brbinf = bregs.brbinf; -perf_clear_branch_entry_bitfields (entry); + *entry = (struct perf_branch_entry ){ };if (brbe_record_is_complete (brbinf)) { entry->from = bregs.brbsrc; entry->to = bregs.brbtgt;
include/linux/perf_event.h +0 -17
@@ -1467,23 +1467,6 @@ static inline u32 perf_sample_data_size(struct perf_sample_data *data,return size; } -/* -* Clear all bitfields in the perf_branch_entry. -* The to and from fields are not cleared because they are -* systematically modified by caller. -*/ -static inline void perf_clear_branch_entry_bitfields (struct perf_branch_entry *br) -{ - br->mispred =0 ; - br->predicted =0 ; - br->in_tx =0 ; - br->abort =0 ; - br->cycles =0 ; - br->type =0 ; - br->spec =PERF_BR_SPEC_NA ; - br->reserved =0 ; -} - extern void perf_output_sample(struct perf_output_handle *handle, struct perf_event_header *header, struct perf_sample_data *data,
Hi Puranjay,
Quoted 5 lines · Puranjay Mohan
These three fixes were found while adding BRBE support for
bpf_get_branch_snapshot()on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.
perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in armv8pmu_sched_task().
Commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") changed armv8pmu_sched_task() from reading the per-CPU cpu_armpmu variable to dereferencing pmu_ctx->pmu via to_arm_pmu(pmu_ctx->pmu). Meanwhile, commit bd2756811766 ("perf:
Rewrite core context handling") made __perf_pmu_sched_task() pass cpc->task_epc to pmu->sched_task(). cpc->task_epc is only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, so cpc->task_epc stays NULL. The NULL argument is dereferenced and crashes.
With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any WARN_ON.
Ian Rogers
reviewerQuoted 22 lines · wuyifan
Hi Puranjay,
Quoted 5 lines · Puranjay Mohan
These three fixes were found while adding BRBE support for
bpf_get_branch_snapshot()on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in
armv8pmu_sched_task().Commit
fa9d27773873("perf:arm_pmu: Kill last use of per-CPUcpu_armpmupointer") changedarmv8pmu_sched_task()from reading the per-CPUcpu_armpmuvariable to dereferencingpmu_ctx->pmuviato_arm_pmu(pmu_ctx->pmu). Meanwhile, commitbd2756811766("perf:
Rewrite core context handling") made__perf_pmu_sched_task()passcpc->task_epctopmu->sched_task().cpc->task_epcis only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, socpc->task_epcstays NULL. The NULL argument is dereferenced and crashes.
Quoted 5 lines
With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any
WARN_ON.Tested-by Yifan Wu <[email protected]>
Hi,
Is there anything that needs doing to land this series? Avoiding a perf crash seems worthwhile. I believe Peter and Ingo are best placed to review the changes to kernel/events/core.c.
Thanks,
Ian
Quoted 28 lines · Ian Rogers
Quoted 26 lines · wuyifan
Hi Puranjay,
Quoted 5 lines · Puranjay Mohan
These three fixes were found while adding BRBE support for
bpf_get_branch_snapshot()on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in
armv8pmu_sched_task().Commit
fa9d27773873("perf:arm_pmu: Kill last use of per-CPUcpu_armpmupointer") changedarmv8pmu_sched_task()from reading the per-CPUcpu_armpmuvariable to dereferencingpmu_ctx->pmuviato_arm_pmu(pmu_ctx->pmu). Meanwhile, commitbd2756811766("perf:
Rewrite core context handling") made__perf_pmu_sched_task()passcpc->task_epctopmu->sched_task().cpc->task_epcis only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, socpc->task_epcstays NULL. The NULL argument is dereferenced and crashes.With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any
WARN_ON.
Quoted 8 lines
Quoted 1 line
Tested-by Yifan Wu <[email protected]>Hi,
Is there anything that needs doing to land this series? Avoiding a perf crash seems worthwhile. I believe Peter and Ingo are best placed to review the changes to
kernel/events/core.c.
another ping... seems like worthwhile problems to fix
Quoted 3 lines
Thanks,
Ian
Peter Zijlstra
maintainerQuoted 5 lines · Puranjay Mohan
Puranjay Mohan (3):
perf/core: Fix NULLpmu_ctxpassed topmu->sched_task()perf/core: Runsched_task()for PMUs with only CPU-wide events perf/core: Fill branch entries with a single assignment
Quoted 8 lines
arch/x86/events/amd/brs.c | 9 +++-- arch/x86/events/amd/lbr.c | 16 ++++----- arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++----------------- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 17 ---------- kernel/events/core.c | 15 ++++++--- 6 files changed, 58 insertions(+), 66 deletions(-)
Found this series ... I'll feed it to the robots and barring them screaming I'll push it out to -tip.