Skip to contentSkip to conversation

perf/core: sched_task() dispatch and branch entry fixes

PATCH Applied Puranjay Mohan 8 participants 14 messages Aug 10 – Sep 23

Subsystems (5)
Stable branch x86 BPF perf events +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v7 cover perf/core: sched_task() dispatch and branch entry fixes

These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.

Patch 1 stops __perf_pmu_sched_task() passing a NULL pmu_ctx to pmu->sched_task(). armv8pmu_sched_task() is the only implementation that dereferences the argument, so the oops needs BRBE.

Patch 2 makes perf_pmu_sched_task() visit PMUs whose events are all CPU-wide. They are skipped today on every switch to a task that has a perf context but no event on that PMU, so branch records leak across the task boundary with perf record -b -a. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally, so x86 LBR is affected the same way.

Patch 3 has each caller fill struct perf_branch_entry in one assignment rather than clearing the bitfields first, and drops perf_clear_branch_entry_bitfields(). The helper had drifted from the
struct: new_type and priv were never cleared, and arm_pmuv3.c allocates the per-CPU branch stack with kmalloc().

Tested on a 128 CPU arm64 machine with BRBE. A WARN_ON_ONCE() at the gate patch 2 adds to perf_ctx_sched_task_cb() fires within seconds of running perf record -b -a alongside a task-bound event pinned to a different CPU. Also built for x86, which patch 3 touches.

Changes in v7:
- Patch 1: pass &cpc->epc unconditionally. cpc->task_epc is NULL there both before and after patch 2, so the conditional was dead.
- Patch 2: gate perf_pmu_sched_task() on cpc->task_epc alone. It is never set without a task context scheduled in, so the cpuctx->task_ctx test was redundant and the two gates are now inverses.
- Patch 3: fill the entry at each site instead of renaming the helper, as suggested by Peter.
- Dropped the v6 review tags, all three patches changed.

Changes in v6:
- Split the sched_task() fix into patches 1 and 2; the NULL dereference and the missed dispatch are separate bugs with different reachability.
- Gate perf_ctx_sched_task_cb() on cpc->task_epc. v5 removed the early return in perf_pmu_sched_task() without it, so both paths ran for a task whose event for that PMU is pinned to another CPU. Caught by the WARN_ON_ONCE() described above.
- Tag patches 1 and 2 for stable.
- Send separately from the BRBE series, rebased onto tip perf/core.

[1] https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/bpf/[email protected]/

Puranjay Mohan (3):
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() perf/core: Run sched_task() for PMUs with only CPU-wide events perf/core: Fill branch entries with a single assignment

 arch/x86/events/amd/brs.c   |  9 +++--
 arch/x86/events/amd/lbr.c   | 16 ++++-----
 arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++-----------------
 drivers/perf/arm_brbe.c     |  2 +-
 include/linux/perf_event.h  | 17 ----------
 kernel/events/core.c        | 15 ++++++---
 6 files changed, 58 insertions(+), 66 deletions(-)
Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH v7 1/3 perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. __perf_pmu_sched_task() therefore always passes NULL:

  Unable to handle kernel NULL pointer dereference at virtual address 00
  pc : armv8pmu_sched_task+0x14/0x50
  Call trace:
   armv8pmu_sched_task+0x14/0x50 (P)
   perf_pmu_sched_task+0xac/0x108
   __perf_event_task_sched_out+0x6c/0xe0

Pass &cpc->epc instead, the CPU-wide context for this PMU, which the function already dereferences a few lines up to find pmu.

armv8pmu_sched_task() is the only in-tree implementation that dereferences the argument, and it only reads ->pmu, so the oops needs BRBE, added in v6.17.

Fixes bd2756811766 ("perf: Rewrite core context handling")
---
 kernel/events/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
index 4638544205f28..05635217696c2 100644
@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
 	perf_ctx_lock(cpuctx, cpuctx->task_ctx);
 	perf_pmu_disable(pmu);
 
-	pmu->sched_task(cpc->task_epc, task, sched_in);
+	pmu->sched_task(&cpc->epc, task, sched_in);
 
 	perf_pmu_enable(pmu);
 	perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Per-CPU PMUs missing sched_task callbacks during context switches if a task context exists. --

--- Patch [1]: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() --- commit 333df3e02c2d6b54e5bd528f91779820677d118e
Author: Puranjay Mohan <[email protected]>

perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. This patch passes &cpc->epc instead, which is the CPU-wide context for this PMU, avoiding a NULL pointer dereference.

kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3910

Quoted 10 lines
kernel/events/core.c +1 -1
index ba5bd6a78fe7b..c372213ffc883 100644
@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
 	perf_ctx_lock(cpuctx, cpuctx->task_ctx);
 	perf_pmu_disable(pmu);
 
-	pmu->sched_task(cpc->task_epc, task, sched_in);
+	pmu->sched_task(&cpc->epc, task, sched_in);

[Severity: High]
This isn't a bug introduced by this patch, but does the early return in perf_pmu_sched_task() skip callbacks for CPU-wide PMUs?

If a system-wide PMU event is active (sched_cb_usage > 0), and a context switch occurs to or from a task with an active perf event (meaning cpuctx->task_ctx is set), perf_pmu_sched_task() returns early:

kernel/events/core.c:perf_pmu_sched_task() {
	...
	/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
	if (prev == next || cpuctx->task_ctx)
		return;
	...
}

It seems to rely on perf_event_context_sched_out/in() to handle the callbacks. However, perf_ctx_sched_task_cb() only iterates over the task's pmu_ctx_list:

kernel/events/core.c:perf_ctx_sched_task_cb() {
	...
	list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
		cpc = this_cpc(pmu_ctx->pmu);

		if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
			pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
	}
	...
}

If the task's perf event is for a different PMU (like a software event), the CPU-wide PMU won't be in ctx->pmu_ctx_list. Does this mean the CPU-wide PMU's sched_task() callback is completely skipped during the context switch, potentially corrupting hardware tracing or leaking branch records across tasks?

Quoted 3 lines
kernel/events/core.c +0 -0
 	perf_pmu_enable(pmu);
 	perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

Quoted 60 lines · On Mon, Aug 10, 2026 at 3:03 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Per-CPU PMUs missing sched_task callbacks during context switches if a task context exists. --

--- Patch [1]: [PATCH v7 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() --- commit 333df3e02c2d6b54e5bd528f91779820677d118e
Author: Puranjay Mohan <[email protected]>

perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. This patch passes &cpc->epc instead, which is the CPU-wide context for this PMU, avoiding a NULL pointer dereference.

kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3910

Quoted 10 lines
kernel/events/core.c +1 -1
index ba5bd6a78fe7b..c372213ffc883 100644
@@ -3907,7 +3907,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
      perf_ctx_lock(cpuctx, cpuctx->task_ctx);
      perf_pmu_disable(pmu);
 
-     pmu->sched_task(cpc->task_epc, task, sched_in);
+     pmu->sched_task(&cpc->epc, task, sched_in);

[Severity: High]
This isn't a bug introduced by this patch, but does the early return in perf_pmu_sched_task() skip callbacks for CPU-wide PMUs?

If a system-wide PMU event is active (sched_cb_usage > 0), and a context switch occurs to or from a task with an active perf event (meaning cpuctx->task_ctx is set), perf_pmu_sched_task() returns early:

kernel/events/core.c:perf_pmu_sched_task() {
        ...
        /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
        if (prev == next || cpuctx->task_ctx)
                return;
        ...
}

It seems to rely on perf_event_context_sched_out/in() to handle the callbacks. However, perf_ctx_sched_task_cb() only iterates over the task's pmu_ctx_list:

kernel/events/core.c:perf_ctx_sched_task_cb() {
        ...
        list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
                cpc = this_cpc(pmu_ctx->pmu);

                if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
                        pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
        }
        ...
}
Quoted 8 lines

If the task's perf event is for a different PMU (like a software event), the CPU-wide PMU won't be in ctx->pmu_ctx_list. Does this mean the CPU-wide PMU's sched_task() callback is completely skipped during the context switch, potentially corrupting hardware tracing or leaking branch records across tasks?

Quoted 3 lines
kernel/events/core.c +0 -0
      perf_pmu_enable(pmu);
      perf_ctx_unlock(cpuctx, cpuctx->task_ctx);

Next patch fixes this exact issue.

merge bot Applied
#4 lore raw

[tip: perf/urgent] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

In reply to Puranjay Mohan
The following commit has been merged into the perf/urgent branch of tip:

The following commit has been merged into the perf/urgent branch of tip:

Commit-ID:     36bb85cf36cab15fb611cb44b78a5df06e4e69a2
Gitweb:        https://git.kernel.org/tip/36bb85cf36cab15fb611cb44b78a5df06e4e69a2
Author:        Puranjay Mohan <[email protected]>
AuthorDate:    Mon, 10 Aug 2026 06:35:34 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 23 Sep 2026 11:48:35 +02:00

perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. __perf_pmu_sched_task() therefore always passes NULL:

  Unable to handle kernel NULL pointer dereference at virtual address 00
  pc : armv8pmu_sched_task+0x14/0x50
  Call trace:
   armv8pmu_sched_task+0x14/0x50 (P)
   perf_pmu_sched_task+0xac/0x108
   __perf_event_task_sched_out+0x6c/0xe0

Pass &cpc->epc instead, the CPU-wide context for this PMU, which the function already dereferences a few lines up to find pmu.

armv8pmu_sched_task() is the only in-tree implementation that dereferences the argument, and it only reads ->pmu, so the oops needs BRBE, added in v6.17.

---
 kernel/events/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
index e180134..7ce72f3 100644
@@ -3914,7 +3914,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
 	perf_ctx_lock(cpuctx, cpuctx->task_ctx);
 	perf_pmu_disable(pmu);
 
-	pmu->sched_task(cpc->task_epc, task, sched_in);
+	pmu->sched_task(&cpc->epc, task, sched_in);
 
 	perf_pmu_enable(pmu);
 	perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
#5 lore raw

PATCH v7 2/3 perf/core: Run sched_task() for PMUs with only CPU-wide events

In reply to Puranjay Mohan

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set and leaves the work to perf_ctx_sched_task_cb(), which only walks ctx->pmu_ctx_list. A PMU whose events are all CPU-wide is not on that list, so nothing calls its sched_task(). With

perf record -b -e cycles -a -- ls

armv8pmu_sched_task() is skipped on every switch to a task that has a perf context but no event on that PMU, and BRBE records leak across the task boundary. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally too, so LBR records leak the same way on x86.

Drop the early return and skip only the CPCs that perf_ctx_sched_task_cb() handles. That one needs a gate of its own to make the split exact: it tests cpc->sched_cb_usage, which perf_sched_cb_inc() sets per CPU for every branch stack user, so a task with an event for that PMU pinned to another CPU would be handled twice. On x86 the second __intel_pmu_lbr_restore() finds lbr_stack_state == LBR_NONE and calls intel_pmu_lbr_reset(), throwing away the callstack the first one restored.

cpc->task_epc is set only while a task context is scheduled in, and there is one epc per PMU on ctx->pmu_ctx_list, so the two gates are inverses.

For the CPCs perf_pmu_sched_task() picks up, the callback now runs outside the perf_ctx_disable() and perf_ctx_enable() pair in perf_event_context_sched_in(). __perf_pmu_sched_task() disables the PMU around the call itself.

Fixes bd2756811766 ("perf: Rewrite core context handling")
---
 kernel/events/core.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
index 05635217696c2..34eb05e9d74d0 100644
@@ -3757,6 +3757,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,
 	list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
 		cpc = this_cpc(pmu_ctx->pmu);
 
+		if (cpc->task_epc != pmu_ctx)
+			continue;
+
 		if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
 			pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
 	}
@@ -3917,15 +3920,17 @@ static void perf_pmu_sched_task(struct task_struct *prev,
 				struct task_struct *next,
 				bool sched_in)
 {
-	struct perf_cpu_context *cpuctx = this_cpu_ptr(&perf_cpu_context);
 	struct perf_cpu_pmu_context *cpc;
 
-	/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
-	if (prev == next || cpuctx->task_ctx)
+	if (prev == next)
 		return;
 
-	list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry)
+	list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) {
+		if (cpc->task_epc)
+			continue;
+
 		__perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in);
+	}
 }
 
 static void perf_event_switch(struct task_struct *task,
Signature · 1 line
2.53.0-Meta
merge bot Applied
#6 lore raw

[tip: perf/urgent] perf/core: Run sched_task() for PMUs with only CPU-wide events

The following commit has been merged into the perf/urgent branch of tip:

The following commit has been merged into the perf/urgent branch of tip:

Commit-ID:     3d8d74100954a3b17e5c5e37adfe14e16b1db103
Gitweb:        https://git.kernel.org/tip/3d8d74100954a3b17e5c5e37adfe14e16b1db103
Author:        Puranjay Mohan <[email protected]>
AuthorDate:    Mon, 10 Aug 2026 06:35:35 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 23 Sep 2026 11:48:35 +02:00

perf/core: Run sched_task() for PMUs with only CPU-wide events

perf_pmu_sched_task() returns early when cpuctx->task_ctx is set and leaves the work to perf_ctx_sched_task_cb(), which only walks ctx->pmu_ctx_list. A PMU whose events are all CPU-wide is not on that list, so nothing calls its sched_task(). With

perf record -b -e cycles -a -- ls

armv8pmu_sched_task() is skipped on every switch to a task that has a perf context but no event on that PMU, and BRBE records leak across the task boundary. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally too, so LBR records leak the same way on x86.

Drop the early return and skip only the CPCs that perf_ctx_sched_task_cb() handles. That one needs a gate of its own to make the split exact: it tests cpc->sched_cb_usage, which perf_sched_cb_inc() sets per CPU for every branch stack user, so a task with an event for that PMU pinned to another CPU would be handled twice. On x86 the second __intel_pmu_lbr_restore() finds lbr_stack_state == LBR_NONE and calls intel_pmu_lbr_reset(), throwing away the callstack the first one restored.

cpc->task_epc is set only while a task context is scheduled in, and there is one epc per PMU on ctx->pmu_ctx_list, so the two gates are inverses.

For the CPCs perf_pmu_sched_task() picks up, the callback now runs outside the perf_ctx_disable() and perf_ctx_enable() pair in perf_event_context_sched_in(). __perf_pmu_sched_task() disables the PMU around the call itself.

---
 kernel/events/core.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
index 7ce72f3..634d2cc 100644
@@ -3764,6 +3764,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,
 	list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
 		cpc = this_cpc(pmu_ctx->pmu);
 
+		if (cpc->task_epc != pmu_ctx)
+			continue;
+
 		if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
 			pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
 	}
@@ -3924,15 +3927,17 @@ static void perf_pmu_sched_task(struct task_struct *prev,
 				struct task_struct *next,
 				bool sched_in)
 {
-	struct perf_cpu_context *cpuctx = this_cpu_ptr(&perf_cpu_context);
 	struct perf_cpu_pmu_context *cpc, *cpc2;
 
-	/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
-	if (prev == next || cpuctx->task_ctx)
+	if (prev == next)
 		return;
 
-	list_for_each_entry_safe(cpc, cpc2, this_cpu_ptr(&sched_cb_list), sched_cb_entry)
+	list_for_each_entry_safe(cpc, cpc2, this_cpu_ptr(&sched_cb_list), sched_cb_entry) {
+		if (cpc->task_epc)
+			continue;
+
 		__perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in);
+	}
 }
 
 static void perf_event_switch(struct task_struct *task,
#7 lore raw

PATCH v7 3/3 perf/core: Fill branch entries with a single assignment

In reply to Puranjay Mohan

perf_clear_branch_entry_bitfields() clears the bitfields of struct perf_branch_entry one by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind: new_type and priv were added to perf_branch_entry and never added here.

Only BRBE writes those two, and neither for every record. brbe_set_perf_entry_type() leaves new_type alone for a branch type it does not recognise, and priv is not set for source-only records. arm_pmuv3.c allocates the per-CPU branch stack with kmalloc(), so such a record reaches userspace with whatever the slot held: uninitialised kmalloc() data on the first pass over the buffer, the previous record's values after that. Nothing under arch/x86/events/ writes either field, so only arm64 is affected.

Assign the whole entry at each site instead. Everything not named is then zero, and there is no list to keep in sync. The bitfields add up to exactly 64 bits, so the struct has no padding to leave undefined.

perf_clear_branch_entry_bitfields() has no callers left, so remove it. perf_entry_from_brbe_regset() assigns an empty literal instead, since it fills from/to conditionally. PERF_BR_SPEC_NA is 0, so dropping the explicit spec assignment changes nothing.

Fixes b190bc4ac9e6 ("perf: Extend branch type classification")
Fixes 5402d25aa571 ("perf: Capture branch privilege information")
---
 arch/x86/events/amd/brs.c   |  9 +++--
 arch/x86/events/amd/lbr.c   | 16 ++++-----
 arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++-----------------
 drivers/perf/arm_brbe.c     |  2 +-
 include/linux/perf_event.h  | 17 ----------
 5 files changed, 48 insertions(+), 61 deletions(-)
arch/x86/events/amd/brs.c +4 -5
index dc564688f3d73..54b13faba116c 100644
@@ -343,11 +343,10 @@ void amd_brs_drain(void)
 		if (!amd_brs_match_plm(event, from, to))
 			continue;
 
-		perf_clear_branch_entry_bitfields(br+nr);
-
-		br[nr].from = from;
-		br[nr].to   = to;
-
+		br[nr] = (struct perf_branch_entry){
+			.from	= from,
+			.to	= to,
+		};
 		nr++;
 	}
 empty:
arch/x86/events/amd/lbr.c +8 -8
index 9d9c961989d51..a55646fcb8465 100644
@@ -184,13 +184,6 @@ void amd_pmu_lbr_read(void)
 		    entry.to.split.reserved)
 			continue;
 
-		perf_clear_branch_entry_bitfields(br + out);
-
-		br[out].from	= sign_ext_branch_ip(entry.from.split.ip);
-		br[out].to	= sign_ext_branch_ip(entry.to.split.ip);
-		br[out].mispred	= entry.from.split.mispredict;
-		br[out].predicted = !br[out].mispred;
-
 		/*
 		 * Set branch speculation information using the status of
 		 * the valid and spec bits.
@@ -208,7 +201,14 @@ void amd_pmu_lbr_read(void)
 		 * speculative and took the correct path
 		 */
 		idx = (entry.to.split.valid << 1) | entry.to.split.spec;
-		br[out].spec = lbr_spec_map[idx];
+
+		br[out] = (struct perf_branch_entry){
+			.from		= sign_ext_branch_ip(entry.from.split.ip),
+			.to		= sign_ext_branch_ip(entry.to.split.ip),
+			.mispred	= entry.from.split.mispredict,
+			.predicted	= !entry.from.split.mispredict,
+			.spec		= lbr_spec_map[idx],
+		};
 		out++;
 	}
 
arch/x86/events/intel/lbr.c +35 -30

89 lines not on this page · raw message

drivers/perf/arm_brbe.c +1 -1
index ba554e0c846c4..254be4da8ae29 100644
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 		return false;
 
 	brbinf = bregs.brbinf;
-	perf_clear_branch_entry_bitfields(entry);
+	*entry = (struct perf_branch_entry){ };
 	if (brbe_record_is_complete(brbinf)) {
 		entry->from = bregs.brbsrc;
 		entry->to = bregs.brbtgt;
include/linux/perf_event.h +0 -17
index 48d851fbd8ea5..310681cccb50a 100644
@@ -1467,23 +1467,6 @@ static inline u32 perf_sample_data_size(struct perf_sample_data *data,
 	return size;
 }
 
-/*
- * Clear all bitfields in the perf_branch_entry.
- * The to and from fields are not cleared because they are
- * systematically modified by caller.
- */
-static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *br)
-{
-	br->mispred	= 0;
-	br->predicted	= 0;
-	br->in_tx	= 0;
-	br->abort	= 0;
-	br->cycles	= 0;
-	br->type	= 0;
-	br->spec	= PERF_BR_SPEC_NA;
-	br->reserved	= 0;
-}
-
 extern void perf_output_sample(struct perf_output_handle *handle,
 			       struct perf_event_header *header,
 			       struct perf_sample_data *data,
Signature · 1 line
2.53.0-Meta
#8 lore raw

Re: [PATCH v7 3/3] perf/core: Fill branch entries with a single assignment

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The BPF branch snapshot helpers (__intel_pmu_snapshot_branch_stack and amd_pmu_v2_snapshot_branch_stack) unconditionally copy stale branch records (LBRs) to BPF buffers, leading to cross-context information leaks. --

--- Patch [3]: [PATCH v7 3/3] perf/core: Fill branch entries with a single assignment --- commit f67f9ecd08216c6081d3d24e81906c232c6917fe
Author: Puranjay Mohan <[email protected]>

perf/core: Fill branch entries with a single assignment

This commit replaces the field-by-field initialization of perf_branch_entry with a single structure assignment. This automatically zeroes out any unnamed fields, removing the need to manually keep perf_clear_branch_entry_bitfields in sync with the structure definition.

Quoted 4 lines
arch/x86/events/intel/lbr.c +0 -0
index cae2e02fe6cc4..539e9feffa356 100644

[ ... ]

Quoted 14 lines
arch/x86/events/intel/lbr.c +2 -8
@@ -847,14 +847,15 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		if (abort && x86_pmu.lbr_double_abort && out > 0)
 			out--;
 
-		perf_clear_branch_entry_bitfields(br+out);
-		br[out].from	 = from;
-		br[out].to	 = to;
-		br[out].mispred	 = mis;
-		br[out].predicted = pred;
-		br[out].in_tx	 = in_tx;
-		br[out].abort	 = abort;
-		br[out].cycles	 = cycles;
+		br[out] = (struct perf_branch_entry){
+			.from		= from,

arch/x86/events/intel/lbr.c void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) · line 861

Quoted 10 lines
arch/x86/events/intel/lbr.c +7 -0
+			.to		= to,
+			.mispred	= mis,
+			.predicted	= pred,
+			.in_tx		= in_tx,
+			.abort		= abort,
+			.cycles		= cycles,
+		};
 		out++;
 	}
 	cpuc->lbr_stack.nr = out;

[Severity: High]
This is a pre-existing issue, but while reviewing how these branch entries are populated and tracked by cpuc->lbr_stack.nr, I noticed a potential information leak in the BPF branch snapshot helpers.

Does __intel_pmu_snapshot_branch_stack() unconditionally copy stale branch records to the BPF buffer?

arch/x86/events/intel/core.c:__intel_pmu_snapshot_branch_stack() {
    ...
    cnt = min_t(unsigned int, cnt, x86_pmu.lbr_nr);
    memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
    ...
}

When the number of valid LBR entries (cpuc->lbr_stack.nr) is less than the hardware capacity (x86_pmu.lbr_nr), this memcpy appears to copy the trailing unwritten entries from cpuc->lbr_entries. This condition can happen routinely in call-stack mode with shallow stacks, or when Intel's double-abort erratum skips an entry (as seen just above in this diff).

The AMD implementation seems to have the exact same logic error:

arch/x86/events/amd/core.c:amd_pmu_v2_snapshot_branch_stack() {
    ...
    cnt = min(cnt, x86_pmu.lbr_nr);
    memcpy(entries, cpuc->lbr_entries, sizeof(struct perf_branch_entry) * cnt);
    ...
}

Could this allow a BPF tracing program to read stale hardware branch records left over from previously executed contexts, leaking execution history from other tasks or the kernel?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
merge bot Applied
#9 lore raw

[tip: perf/urgent] perf/core: Fill branch entries with a single assignment

In reply to Puranjay Mohan
The following commit has been merged into the perf/urgent branch of tip:

The following commit has been merged into the perf/urgent branch of tip:

Commit-ID:     24b620729e53d978b3e425f55bc66efd3bab1f59
Gitweb:        https://git.kernel.org/tip/24b620729e53d978b3e425f55bc66efd3bab1f59
Author:        Puranjay Mohan <[email protected]>
AuthorDate:    Mon, 10 Aug 2026 06:35:36 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 23 Sep 2026 11:48:36 +02:00

perf/core: Fill branch entries with a single assignment

perf_clear_branch_entry_bitfields() clears the bitfields of struct perf_branch_entry one by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind: new_type and priv were added to perf_branch_entry and never added here.

Only BRBE writes those two, and neither for every record. brbe_set_perf_entry_type() leaves new_type alone for a branch type it does not recognise, and priv is not set for source-only records. arm_pmuv3.c allocates the per-CPU branch stack with kmalloc(), so such a record reaches userspace with whatever the slot held: uninitialised kmalloc() data on the first pass over the buffer, the previous record's values after that. Nothing under arch/x86/events/ writes either field, so only arm64 is affected.

Assign the whole entry at each site instead. Everything not named is then zero, and there is no list to keep in sync. The bitfields add up to exactly 64 bits, so the struct has no padding to leave undefined.

perf_clear_branch_entry_bitfields() has no callers left, so remove it. perf_entry_from_brbe_regset() assigns an empty literal instead, since it fills from/to conditionally. PERF_BR_SPEC_NA is 0, so dropping the explicit spec assignment changes nothing.

Fixes b190bc4ac9e6 ("perf: Extend branch type classification")
Fixes 5402d25aa571 ("perf: Capture branch privilege information")
---
 arch/x86/events/amd/brs.c   |  9 ++---
 arch/x86/events/amd/lbr.c   | 16 ++++-----
 arch/x86/events/intel/lbr.c | 65 +++++++++++++++++++-----------------
 drivers/perf/arm_brbe.c     |  2 +-
 include/linux/perf_event.h  | 17 +---------
 5 files changed, 48 insertions(+), 61 deletions(-)
arch/x86/events/amd/brs.c +4 -5
index dc56468..54b13fa 100644
@@ -343,11 +343,10 @@ void amd_brs_drain(void)
 		if (!amd_brs_match_plm(event, from, to))
 			continue;
 
-		perf_clear_branch_entry_bitfields(br+nr);
-
-		br[nr].from = from;
-		br[nr].to   = to;
-
+		br[nr] = (struct perf_branch_entry){
+			.from	= from,
+			.to	= to,
+		};
 		nr++;
 	}
 empty:
arch/x86/events/amd/lbr.c +8 -8
index 9d9c961..a55646f 100644
@@ -184,13 +184,6 @@ void amd_pmu_lbr_read(void)
 		    entry.to.split.reserved)
 			continue;
 
-		perf_clear_branch_entry_bitfields(br + out);
-
-		br[out].from	= sign_ext_branch_ip(entry.from.split.ip);
-		br[out].to	= sign_ext_branch_ip(entry.to.split.ip);
-		br[out].mispred	= entry.from.split.mispredict;
-		br[out].predicted = !br[out].mispred;
-
 		/*
 		 * Set branch speculation information using the status of
 		 * the valid and spec bits.
@@ -208,7 +201,14 @@ void amd_pmu_lbr_read(void)
 		 * speculative and took the correct path
 		 */
 		idx = (entry.to.split.valid << 1) | entry.to.split.spec;
-		br[out].spec = lbr_spec_map[idx];
+
+		br[out] = (struct perf_branch_entry){
+			.from		= sign_ext_branch_ip(entry.from.split.ip),
+			.to		= sign_ext_branch_ip(entry.to.split.ip),
+			.mispred	= entry.from.split.mispredict,
+			.predicted	= !entry.from.split.mispredict,
+			.spec		= lbr_spec_map[idx],
+		};
 		out++;
 	}
 
arch/x86/events/intel/lbr.c +35 -30

89 lines not on this page · raw message

drivers/perf/arm_brbe.c +1 -1
index ba554e0..254be4d 100644
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 		return false;
 
 	brbinf = bregs.brbinf;
-	perf_clear_branch_entry_bitfields(entry);
+	*entry = (struct perf_branch_entry){ };
 	if (brbe_record_is_complete(brbinf)) {
 		entry->from = bregs.brbsrc;
 		entry->to = bregs.brbtgt;
include/linux/perf_event.h +0 -17
index 5842552..915c6fd 100644
@@ -1467,23 +1467,6 @@ static inline u32 perf_sample_data_size(struct perf_sample_data *data,
 	return size;
 }
 
-/*
- * Clear all bitfields in the perf_branch_entry.
- * The to and from fields are not cleared because they are
- * systematically modified by caller.
- */
-static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *br)
-{
-	br->mispred	= 0;
-	br->predicted	= 0;
-	br->in_tx	= 0;
-	br->abort	= 0;
-	br->cycles	= 0;
-	br->type	= 0;
-	br->spec	= PERF_BR_SPEC_NA;
-	br->reserved	= 0;
-}
-
 extern void perf_output_sample(struct perf_output_handle *handle,
 			       struct perf_event_header *header,
 			       struct perf_sample_data *data,
Tested-by on the v7 series
#10 lore raw
In reply to Puranjay Mohan

Hi Puranjay,

Quoted 5 lines · Puranjay Mohan

These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.

perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in armv8pmu_sched_task().

Commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") changed armv8pmu_sched_task() from reading the per-CPU cpu_armpmu variable to dereferencing pmu_ctx->pmu via to_arm_pmu(pmu_ctx->pmu). Meanwhile, commit bd2756811766 ("perf:
Rewrite core context handling") made __perf_pmu_sched_task() pass cpc->task_epc to pmu->sched_task(). cpc->task_epc is only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, so cpc->task_epc stays NULL. The NULL argument is dereferenced and crashes.

With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any WARN_ON.

reviewer
#11 lore raw
Quoted 22 lines · wuyifan

Hi Puranjay,

Quoted 5 lines · Puranjay Mohan

These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.

perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in armv8pmu_sched_task().

Commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") changed armv8pmu_sched_task() from reading the per-CPU cpu_armpmu variable to dereferencing pmu_ctx->pmu via to_arm_pmu(pmu_ctx->pmu). Meanwhile, commit bd2756811766 ("perf:
Rewrite core context handling") made __perf_pmu_sched_task() pass cpc->task_epc to pmu->sched_task(). cpc->task_epc is only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, so cpc->task_epc stays NULL. The NULL argument is dereferenced and crashes.

Quoted 5 lines

With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any WARN_ON.

Hi,

Is there anything that needs doing to land this series? Avoiding a perf crash seems worthwhile. I believe Peter and Ingo are best placed to review the changes to kernel/events/core.c.

Thanks,
Ian

#12 lore raw
Quoted 28 lines · Ian Rogers
Quoted 26 lines · wuyifan

Hi Puranjay,

Quoted 5 lines · Puranjay Mohan

These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and were carried in that series until v6 [1]. They do not depend on it, so they go on their own; the version number continues from that series to avoid two numbering schemes for the same patches.

perf record -C <cpu> -b [<command>] triggers a NULL pointer dereference in armv8pmu_sched_task().

Commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") changed armv8pmu_sched_task() from reading the per-CPU cpu_armpmu variable to dereferencing pmu_ctx->pmu via to_arm_pmu(pmu_ctx->pmu). Meanwhile, commit bd2756811766 ("perf:
Rewrite core context handling") made __perf_pmu_sched_task() pass cpc->task_epc to pmu->sched_task(). cpc->task_epc is only non-NULL while a task context is scheduled in; CPU-bound events do not schedule a task context, so cpc->task_epc stays NULL. The NULL argument is dereferenced and crashes.

With the patches applied, perf record -C <cpu> -b no longer crashes, and system-wide perf record -b -e <event> -a alongside a task-bound event on a different CPU runs without any WARN_ON.

Quoted 8 lines
Quoted 1 line

Hi,

Is there anything that needs doing to land this series? Avoiding a perf crash seems worthwhile. I believe Peter and Ingo are best placed to review the changes to kernel/events/core.c.

another ping... seems like worthwhile problems to fix

Quoted 3 lines

Thanks,
Ian

maintainer
#13 lore raw
In reply to Puranjay Mohan
Quoted 5 lines · Puranjay Mohan

Puranjay Mohan (3):
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() perf/core: Run sched_task() for PMUs with only CPU-wide events perf/core: Fill branch entries with a single assignment

Quoted 8 lines
 arch/x86/events/amd/brs.c   |  9 +++--
 arch/x86/events/amd/lbr.c   | 16 ++++-----
 arch/x86/events/intel/lbr.c | 65 ++++++++++++++++++++-----------------
 drivers/perf/arm_brbe.c     |  2 +-
 include/linux/perf_event.h  | 17 ----------
 kernel/events/core.c        | 15 ++++++---
 6 files changed, 58 insertions(+), 66 deletions(-)

Found this series ... I'll feed it to the robots and barring them screaming I'll push it out to -tip.