perf/core: sched_task() dispatch and branch entry fixes
Subsystems (5)
v6 cover perf/core: sched_task() dispatch and branch entry fixes
These three fixes were found while adding BRBE support for bpf_get_branch_snapshot() on arm64 and have been carried in that series since v1 [1]. They do not depend on it, so they go on their own from here; the version number continues from that series to avoid two numbering schemes for the same patches.
Patch 1 stops __perf_pmu_sched_task() passing a NULL pmu_ctx to pmu->sched_task(). armv8pmu_sched_task() is the only implementation that dereferences the argument, so the oops needs BRBE.
Patch 2 makes perf_pmu_sched_task() visit PMUs whose events are all CPU-wide. They are skipped today whenever the scheduled task has a perf event of its own, so branch records leak across task boundaries with perf record -b -a. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally, so x86 LBR is affected the same way.
Dropping the early return alone leaves both dispatch paths running for one case, so perf_ctx_sched_task_cb() gains a matching gate. The two could instead be collapsed into perf_pmu_sched_task() alone, since __perf_pmu_sched_task() already passes the same epc, but that would move the callback out of the perf_ctx_disable() window for every PMU rather than just that one case, which seemed like too much for a fix tagged for stable.
Patch 3 clears struct perf_branch_entry with a single struct assignment. perf_clear_branch_entry_bitfields() had drifted from the struct: new_type and priv were never cleared, and arm_pmuv3.c allocates the per-CPU branch stack with kmalloc().
Tested on a 128 CPU arm64 machine with BRBE. A WARN_ON_ONCE() at the gate patch 2 adds to perf_ctx_sched_task_cb() fires within seconds of running perf record -b -a alongside a task-bound event pinned to a different CPU.
Changes in v6:
- Split the sched_task() fix into patches 1 and 2; the NULL dereference and the missed dispatch are separate bugs with different reachability.
- Gate perf_ctx_sched_task_cb() on cpc->task_epc. v5 removed the early return in perf_pmu_sched_task() without it, so both paths ran for a task whose event for that PMU is pinned to another CPU. Caught by the WARN_ON_ONCE() described above.
- Tag patches 1 and 2 for stable.
- Send separately from the BRBE series, rebased onto tip perf/core.
[1] https://lore.kernel.org/all/[email protected]/
Based on tip perf/core (f4dfab174244).
Puranjay Mohan (3):
perf/core: Fix NULL pmu_ctx passed to pmu->sched_task() perf/core: Run sched_task() for PMUs with only CPU-wide events perf/core: Clear the whole branch entry in perf_clear_branch_entry()
arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/lbr.c | 6 +++--- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 16 ++-------------- kernel/events/core.c | 15 +++++++++++---- 6 files changed, 19 insertions(+), 24 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH v6 1/3 perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
perf_pmu_sched_task() returns early when cpuctx->task_ctx is set, and cpc->task_epc is only non-NULL while a task context is scheduled in on this CPU. __perf_pmu_sched_task() therefore always passes NULL:
Unable to handle kernel NULL pointer dereference at virtual address 00 pc : armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0
Pass &cpc->epc instead. __perf_init_event_pmu_context() sets its ->pmu when the PMU is registered; ->ctx stays NULL until a CPU-wide event attaches. That is enough here because armv8pmu_sched_task() is the only in-tree implementation that dereferences the argument at all, and it only reads ->pmu. The oops therefore needs BRBE, which arrived in v6.17.
--- kernel/events/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/events/core.c +2 -1
@@ -3907,7 +3907,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (cpc->task_epc ? cpc->task_epc : &cpc->epc, + task, sched_in);perf_pmu_enable (pmu);perf_ctx_unlock (cpuctx, cpuctx->task_ctx);
Signature · 1 line
2.53.0-Meta
Peter Zijlstra
maintainerRe: [PATCH v6 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
Quoted 30 lines · Puranjay Mohan
perf_pmu_sched_task()returns early whencpuctx->task_ctxis set, andcpc->task_epcis only non-NULL while a task context is scheduled in on this CPU.__perf_pmu_sched_task()therefore always passes NULL:Unable to handle kernel NULL pointer dereference at virtual address 00 pc : armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0Pass
&cpc->epcinstead.__perf_init_event_pmu_context()sets its ->pmu when the PMU is registered; ->ctx stays NULL until a CPU-wide event attaches. That is enough here becausearmv8pmu_sched_task()is the only in-tree implementation that dereferences the argument at all, and it only reads ->pmu. The oops therefore needs BRBE, which arrived in v6.17.Fixes bd2756811766 ("perf: Rewrite core context handling")Acked-by Usama Arif <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- kernel/events/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/events/core.c +0
-0
kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3911
Quoted 6 lines
kernel/events/core.c +2
-1@@ -3907,7 +3907,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu); - pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (cpc->task_epc ? cpc->task_epc : &cpc->epc, + task, sched_in);
This can be written like: cpc->task_epc ?: &cpc->epc.
However, if cpc-task_epc is always NULL, then this all seems like an overly complicated way of writing: &cpc->epc. What gives?
Re: [PATCH v6 1/3] perf/core: Fix NULL pmu_ctx passed to pmu->sched_task()
Quoted 35 lines · Peter Zijlstra
Quoted 33 lines · Puranjay Mohan
perf_pmu_sched_task()returns early whencpuctx->task_ctxis set, andcpc->task_epcis only non-NULL while a task context is scheduled in on this CPU.__perf_pmu_sched_task()therefore always passes NULL:Unable to handle kernel NULL pointer dereference at virtual address 00 pc : armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0Pass
&cpc->epcinstead.__perf_init_event_pmu_context()sets its ->pmu when the PMU is registered; ->ctx stays NULL until a CPU-wide event attaches. That is enough here becausearmv8pmu_sched_task()is the only in-tree implementation that dereferences the argument at all, and it only reads ->pmu. The oops therefore needs BRBE, which arrived in v6.17.Fixes bd2756811766 ("perf: Rewrite core context handling")Acked-by Usama Arif <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- kernel/events/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/events/core.c +0
-0@@ -3907,7 +3907,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,perf_ctx_lock (cpuctx, cpuctx->task_ctx);perf_pmu_disable (pmu);
Quoted 8 lines
kernel/events/core.c static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, · line 3911
Quoted 3 lines
kernel/events/core.c +2
-1- pmu->sched_task (cpc->task_epc, task, sched_in); + pmu->sched_task (cpc->task_epc ? cpc->task_epc : &cpc->epc, + task, sched_in);This can be written like:
cpc->task_epc?:&cpc->epc.However, if cpc-task_epc is always NULL, then this all seems like an overly complicated way of writing:
&cpc->epc. What gives?
Nothing, you're right, it should just be &cpc->epc.
I wrote the ternary expecting patch 2 to make cpc->task_epc reachable here, but it does not.
Will fix it in v7.
Thanks,
Puranjay
PATCH v6 2/3 perf/core: Run sched_task() for PMUs with only CPU-wide events
In reply to Puranjay Mohanperf_pmu_sched_task() returns early when cpuctx->task_ctx is set and leaves the work to perf_ctx_sched_task_cb(). That one only walks ctx->pmu_ctx_list, so a PMU whose events are all CPU-wide is never visited and its sched_task() callback does not run. With
perf record -b -e cycles -a -- ls
armv8pmu_sched_task() is skipped on every switch to a task that has a perf event of its own, and BRBE records leak across the task boundary. intel_pmu_lbr_add() calls perf_sched_cb_inc() unconditionally as well, so LBR records leak the same way on x86.
Drop the early return and instead skip the individual CPCs that perf_ctx_sched_task_cb() already handles.
That requires the two to agree on which CPC belongs to which path, and they do not. perf_ctx_sched_task_cb() gates on cpc->sched_cb_usage, which perf_sched_cb_inc() sets per CPU for every branch stack user, while the new gate uses cpc->task_epc, which __link_epc() sets only on the CPU the task context is scheduled in on. A task with an event for that PMU pinned to another CPU has an epc on ctx->pmu_ctx_list while cpc->task_epc is NULL, so both paths would run and sched_task() would be called twice per context switch. On x86 the second __intel_pmu_lbr_restore() finds lbr_stack_state == LBR_NONE and calls intel_pmu_lbr_reset(), throwing away the callstack the first one restored. So gate perf_ctx_sched_task_cb() on cpc->task_epc too.
For the CPCs that perf_pmu_sched_task() now handles, the callback no longer runs inside the perf_ctx_disable() and perf_ctx_enable() pair in perf_event_context_sched_in(). __perf_pmu_sched_task() disables the PMU around the call itself, so the callback still runs with it disabled.
--- kernel/events/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/events/core.c +9 -3
@@ -3757,6 +3757,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc =this_cpc (pmu_ctx->pmu); +if (cpc->task_epc != pmu_ctx) +continue ; +if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, task, sched_in); }
@@ -3921,12 +3924,15 @@ static void perf_pmu_sched_task(struct task_struct *prev,struct perf_cpu_context *cpuctx =this_cpu_ptr (&perf_cpu_context);struct perf_cpu_pmu_context *cpc; -/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ -if (prev == next || cpuctx->task_ctx) +if (prev == next)return ; -list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) +list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) { +if (cpuctx->task_ctx && cpc->task_epc) +continue ; +__perf_pmu_sched_task (cpc, sched_in ? next : prev, sched_in); + } } static void perf_event_switch(struct task_struct *task,
Signature · 1 line
2.53.0-Meta
Peter Zijlstra
maintainerRe: [PATCH v6 2/3] perf/core: Run sched_task() for PMUs with only CPU-wide events
Quoted 45 lines · Puranjay Mohan
perf_pmu_sched_task()returns early whencpuctx->task_ctxis set and leaves the work toperf_ctx_sched_task_cb(). That one only walksctx->pmu_ctx_list, so a PMU whose events are all CPU-wide is never visited and itssched_task()callback does not run. Withperf record -b -e cycles -a -- ls
armv8pmu_sched_task()is skipped on every switch to a task that has a perf event of its own, and BRBE records leak across the task boundary.intel_pmu_lbr_add()callsperf_sched_cb_inc()unconditionally as well, so LBR records leak the same way on x86.Drop the early return and instead skip the individual CPCs that
perf_ctx_sched_task_cb()already handles.That requires the two to agree on which CPC belongs to which path, and they do not.
perf_ctx_sched_task_cb()gates oncpc->sched_cb_usage, whichperf_sched_cb_inc()sets per CPU for every branch stack user, while the new gate usescpc->task_epc, which__link_epc()sets only on the CPU the task context is scheduled in on. A task with an event for that PMU pinned to another CPU has an epc onctx->pmu_ctx_listwhilecpc->task_epcis NULL, so both paths would run andsched_task()would be called twice per context switch. On x86 the second__intel_pmu_lbr_restore()findslbr_stack_state==LBR_NONEand callsintel_pmu_lbr_reset(), throwing away the callstack the first one restored. So gateperf_ctx_sched_task_cb()oncpc->task_epctoo.For the CPCs that
perf_pmu_sched_task()now handles, the callback no longer runs inside theperf_ctx_disable()andperf_ctx_enable()pair inperf_event_context_sched_in().__perf_pmu_sched_task()disables the PMU around the call itself, so the callback still runs with it disabled.Fixes bd2756811766 ("perf: Rewrite core context handling")Acked-by Usama Arif <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- kernel/events/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/events/core.c +0
-0
kernel/events/core.c static void perf_ctx_sched_task_cb(struct perf_event_context *ctx, · line 3761
Quoted 5 lines
kernel/events/core.c +2
-0@@ -3757,6 +3757,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc =this_cpc (pmu_ctx->pmu); +if (cpc->task_epc != pmu_ctx) +continue ;
Why isn't this the inverse condition of the below? That is, we should call either this or the other, right?
Quoted 6 lines
kernel/events/core.c +1
-0+if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, task, sched_in); }@@ -3921,12 +3924,15 @@ static void perf_pmu_sched_task(struct task_struct *prev,struct perf_cpu_context *cpuctx =this_cpu_ptr (&perf_cpu_context);
kernel/events/core.c static void perf_pmu_sched_task(struct task_struct *prev, · line 3931
Quoted 10 lines
kernel/events/core.c +3
-3struct perf_cpu_pmu_context *cpc; -/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ -if (prev == next || cpuctx->task_ctx) +if (prev == next)return ; -list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) +list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) { +if (cpuctx->task_ctx && cpc->task_epc)
I'm not sure I see the need for the first part of that clause; why isn't: 'cpuc->task_epc' sufficient?
Quoted 10 lines
kernel/events/core.c +3
-0+continue ; +__perf_pmu_sched_task (cpc, sched_in ? next : prev, sched_in); + } } static void perf_event_switch(struct task_struct *task,Signature · 1 line
2.53.0-Meta
Re: [PATCH v6 2/3] perf/core: Run sched_task() for PMUs with only CPU-wide events
Quoted 50 lines · Peter Zijlstra
Quoted 48 lines · Puranjay Mohan
perf_pmu_sched_task()returns early whencpuctx->task_ctxis set and leaves the work toperf_ctx_sched_task_cb(). That one only walksctx->pmu_ctx_list, so a PMU whose events are all CPU-wide is never visited and itssched_task()callback does not run. Withperf record -b -e cycles -a -- ls
armv8pmu_sched_task()is skipped on every switch to a task that has a perf event of its own, and BRBE records leak across the task boundary.intel_pmu_lbr_add()callsperf_sched_cb_inc()unconditionally as well, so LBR records leak the same way on x86.Drop the early return and instead skip the individual CPCs that
perf_ctx_sched_task_cb()already handles.That requires the two to agree on which CPC belongs to which path, and they do not.
perf_ctx_sched_task_cb()gates oncpc->sched_cb_usage, whichperf_sched_cb_inc()sets per CPU for every branch stack user, while the new gate usescpc->task_epc, which__link_epc()sets only on the CPU the task context is scheduled in on. A task with an event for that PMU pinned to another CPU has an epc onctx->pmu_ctx_listwhilecpc->task_epcis NULL, so both paths would run andsched_task()would be called twice per context switch. On x86 the second__intel_pmu_lbr_restore()findslbr_stack_state==LBR_NONEand callsintel_pmu_lbr_reset(), throwing away the callstack the first one restored. So gateperf_ctx_sched_task_cb()oncpc->task_epctoo.For the CPCs that
perf_pmu_sched_task()now handles, the callback no longer runs inside theperf_ctx_disable()andperf_ctx_enable()pair inperf_event_context_sched_in().__perf_pmu_sched_task()disables the PMU around the call itself, so the callback still runs with it disabled.Fixes bd2756811766 ("perf: Rewrite core context handling")Acked-by Usama Arif <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- kernel/events/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/events/core.c +0
-0@@ -3757,6 +3757,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc =this_cpc (pmu_ctx->pmu);
Quoted 5 lines
kernel/events/core.c static void perf_ctx_sched_task_cb(struct perf_event_context *ctx, · line 3761
Quoted 2 lines
kernel/events/core.c +2
-0+if (cpc->task_epc != pmu_ctx) +continue ;Why isn't this the inverse condition of the below? That is, we should call either this or the other, right?
It should be. There's only one epc per PMU on ctx->pmu_ctx_list, so != pmu_ctx here is really just "task_epc isn't set", and once the extra term below goes the two are inverses.
Quoted 14 lines
Quoted 14 lines
kernel/events/core.c +2
-2+if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, task, sched_in); }@@ -3921,12 +3924,15 @@ static void perf_pmu_sched_task(struct task_struct *prev,struct perf_cpu_context *cpuctx =this_cpu_ptr (&perf_cpu_context);struct perf_cpu_pmu_context *cpc; -/* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ -if (prev == next || cpuctx->task_ctx) +if (prev == next)return ;
Quoted 6 lines
kernel/events/core.c static void perf_pmu_sched_task(struct task_struct *prev, · line 3931
Quoted 3 lines
kernel/events/core.c +2
-1-list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) +list_for_each_entry (cpc,this_cpu_ptr (&sched_cb_list), sched_cb_entry) { +if (cpuctx->task_ctx && cpc->task_epc)I'm not sure I see the need for the first part of that clause; why isn't: '
cpuc->task_epc' sufficient?
It is, task_epc can't be set without task_ctx. ctx_sched_in() sets task_ctx first, ctx_sched_out() clears both, and the window in between is under ctx->lock with IRQs off so the scheduler never lands in it. That includes ctx_resched() scheduling out a single pmu, which leaves it open the longest.
Will fix both in v7.
Thanks,
Puranjay
PATCH v6 3/3 perf/core: Clear the whole branch entry in perf_clear_branch_entry()
In reply to Puranjay Mohanperf_clear_branch_entry_bitfields() clears the bitfields of struct perf_branch_entry one by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind: new_type and priv were added to perf_branch_entry and never added here.
Only BRBE writes those two, and neither is written for every record. brbe_set_perf_entry_type() leaves new_type alone for a branch type it does not recognise, and priv is not set for source-only records. arm_pmuv3.c allocates the per-CPU branch stack with kmalloc(), so such a record carries whatever the slot held: uninitialised kmalloc() data on the first pass over the buffer, the previous record's values after that. Both reach userspace through the branch stack. Nothing under arch/x86/events/ writes either field, so x86 is unaffected.
Clear the entry with a single struct assignment instead:
*br = (struct perf_branch_entry){ };The bitfields add up to exactly 64 bits, so there is no padding, and every caller assigns from/to immediately afterwards, so zeroing those as well changes nothing. PERF_BR_SPEC_NA is 0, so dropping the explicit spec assignment leaves the behaviour unchanged. Nothing needs keeping in sync when a field is added.
The helper no longer touches only bitfields, so rename it to perf_clear_branch_entry().
--- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/lbr.c | 6 +++--- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 16 ++-------------- 5 files changed, 8 insertions(+), 20 deletions(-)
arch/x86/events/amd/brs.c +1 -1
@@ -343,7 +343,7 @@ void amd_brs_drain(void)rdmsrq (brs_from (brs_idx), from); -perf_clear_branch_entry_bitfields (br+nr); +perf_clear_branch_entry (br + nr); br[nr].from = from; br[nr].to = to;
arch/x86/events/amd/lbr.c +1 -1
@@ -183,7 +183,7 @@ void amd_pmu_lbr_read(void) entry.to.split.reserved)continue ; -perf_clear_branch_entry_bitfields (br + out); +perf_clear_branch_entry (br + out); br[out].from =sign_ext_branch_ip (entry.from.split.ip); br[out].to =sign_ext_branch_ip (entry.to.split.ip);
arch/x86/events/intel/lbr.c +3 -3
@@ -756,7 +756,7 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)rdmsrq (x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr); -perf_clear_branch_entry_bitfields (br); +perf_clear_branch_entry (br); br->from = msr_lastbranch.from; br->to = msr_lastbranch.to;
@@ -847,7 +847,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)if (abort && x86_pmu.lbr_double_abort && out >0 ) out--; -perf_clear_branch_entry_bitfields (br+out); +perf_clear_branch_entry (br + out); br[out].from = from; br[out].to = to; br[out].mispred = mis;
@@ -921,7 +921,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc, to =rdlbr_to (i, lbr); info =rdlbr_info (i, lbr); -perf_clear_branch_entry_bitfields (e); +perf_clear_branch_entry (e); e->from = from; e->to = to;
drivers/perf/arm_brbe.c +1 -1
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *entreturn false; brbinf = bregs.brbinf; -perf_clear_branch_entry_bitfields (entry); +perf_clear_branch_entry (entry);if (brbe_record_is_complete (brbinf)) { entry->from = bregs.brbsrc; entry->to = bregs.brbtgt;
include/linux/perf_event.h +2 -14
@@ -1467,21 +1467,9 @@ static inline u32 perf_sample_data_size(struct perf_sample_data *data,return size; } -/* -* Clear all bitfields in the perf_branch_entry. -* The to and from fields are not cleared because they are -* systematically modified by caller. -*/ -static inline void perf_clear_branch_entry_bitfields (struct perf_branch_entry *br) +static inline void perf_clear_branch_entry (struct perf_branch_entry *br) { - br->mispred =0 ; - br->predicted =0 ; - br->in_tx =0 ; - br->abort =0 ; - br->cycles =0 ; - br->type =0 ; - br->spec =PERF_BR_SPEC_NA ; - br->reserved =0 ; + *br = (struct perf_branch_entry ){ }; } extern void perf_output_sample(struct perf_output_handle *handle,
Signature · 1 line
2.53.0-Meta
Peter Zijlstra
maintainerRe: [PATCH v6 3/3] perf/core: Clear the whole branch entry in perf_clear_branch_entry()
Quoted 19 lines · Puranjay Mohan
perf_clear_branch_entry_bitfields()clears the bitfields of structperf_branch_entryone by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind:new_typeand priv were added toperf_branch_entryand never added here.Only BRBE writes those two, and neither is written for every record.
brbe_set_perf_entry_type()leavesnew_typealone for a branch type it does not recognise, and priv is not set for source-only records.arm_pmuv3.callocates the per-CPU branch stack withkmalloc(), so such a record carries whatever the slot held: uninitialisedkmalloc()data on the first pass over the buffer, the previous record's values after that. Both reach userspace through the branch stack. Nothing underarch/x86/events/writes either field, so x86 is unaffected.Clear the entry with a single struct assignment instead:
*br = (struct perf_branch_entry){ };
Quoted 8 lines
The bitfields add up to exactly 64 bits, so there is no padding, and every caller assigns from/to immediately afterwards, so zeroing those as well changes nothing.
PERF_BR_SPEC_NAis 0, so dropping the explicit spec assignment leaves the behaviour unchanged. Nothing needs keeping in sync when a field is added.The helper no longer touches only bitfields, so rename it to
perf_clear_branch_entry().
Fair enough I suppose, but then why not write it like so?
---
arch/x86/events/amd/brs.c +1 -5
@@ -343,11 +343,7 @@ void amd_brs_drain(void)if (!amd_brs_match_plm (event, from, to))continue ; -perf_clear_branch_entry_bitfields (br+nr); - - br[nr].from = from; - br[nr].to = to; - + br[nr] = (struct perf_branch_entry ){ from, to }; nr++; } empty:
arch/x86/events/amd/lbr.c +7 -7
@@ -184,12 +184,6 @@ void amd_pmu_lbr_read(void) entry.to.split.reserved)continue ; -perf_clear_branch_entry_bitfields (br + out); - - br[out].from =sign_ext_branch_ip (entry.from.split.ip); - br[out].to =sign_ext_branch_ip (entry.to.split.ip); - br[out].mispred = entry.from.split.mispredict; - br[out].predicted = !br[out].mispred;/* * Set branch speculation information using the status of
@@ -208,7 +202,13 @@ void amd_pmu_lbr_read(void)* speculative and took the correct path */ idx = (entry.to.split.valid <<1 ) | entry.to.split.spec; - br[out].spec = lbr_spec_map[idx]; + br[out] = (struct perf_branch_entry ) { + .from =sign_ext_branch_ip (entry.from.split.ip), + .to =sign_ext_branch_ip (entry.to.split.ip), + .mispred = entry.from.split.mispredict, + .predicted = !entry.from.split.mispredict, + .spec = lbr_spec_map[idx], + }; out++; }
arch/x86/events/intel/lbr.c +32 -30
80 lines not on this page · raw message
drivers/perf/arm_brbe.c +1 -1
@@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(return false; brbinf = bregs.brbinf; -perf_clear_branch_entry_bitfields (entry); + *entry = (struct perf_branch_entry ) { };if (brbe_record_is_complete (brbinf)) { entry->from = bregs.brbsrc; entry->to = bregs.brbtgt;
include/linux/perf_event.h +0 -17
@@ -1467,23 +1467,6 @@ static inline u32 perf_sample_data_size(return size; } -/* -* Clear all bitfields in the perf_branch_entry. -* The to and from fields are not cleared because they are -* systematically modified by caller. -*/ -static inline void perf_clear_branch_entry_bitfields (struct perf_branch_entry *br) -{ - br->mispred =0 ; - br->predicted =0 ; - br->in_tx =0 ; - br->abort =0 ; - br->cycles =0 ; - br->type =0 ; - br->spec =PERF_BR_SPEC_NA ; - br->reserved =0 ; -} - extern void perf_output_sample(struct perf_output_handle *handle, struct perf_event_header *header, struct perf_sample_data *data,
Re: [PATCH v6 3/3] perf/core: Clear the whole branch entry in perf_clear_branch_entry()
Quoted 27 lines · Peter Zijlstra
Quoted 25 lines · Puranjay Mohan
perf_clear_branch_entry_bitfields()clears the bitfields of structperf_branch_entryone by one and leaves from/to alone, since callers overwrite those straight away. The list has to be kept in sync with the struct by hand and has already fallen behind:new_typeand priv were added toperf_branch_entryand never added here.Only BRBE writes those two, and neither is written for every record.
brbe_set_perf_entry_type()leavesnew_typealone for a branch type it does not recognise, and priv is not set for source-only records.arm_pmuv3.callocates the per-CPU branch stack withkmalloc(), so such a record carries whatever the slot held: uninitialisedkmalloc()data on the first pass over the buffer, the previous record's values after that. Both reach userspace through the branch stack. Nothing underarch/x86/events/writes either field, so x86 is unaffected.Clear the entry with a single struct assignment instead:
*br = (struct perf_branch_entry){ };The bitfields add up to exactly 64 bits, so there is no padding, and every caller assigns from/to immediately afterwards, so zeroing those as well changes nothing.
PERF_BR_SPEC_NAis 0, so dropping the explicit spec assignment leaves the behaviour unchanged. Nothing needs keeping in sync when a field is added.
Quoted 4 lines
Quoted 2 lines
The helper no longer touches only bitfields, so rename it to
perf_clear_branch_entry().Fair enough I suppose, but then why not write it like so?
Yes, that is better. I will use your version in v7.
Thanks,
Puranjay
Quoted 52 lines · Puranjay Mohan
These three fixes were found while adding BRBE support for
bpf_get_branch_snapshot()on arm64 and have been carried in that series since v1 [1]. They do not depend on it, so they go on their own from here; the version number continues from that series to avoid two numbering schemes for the same patches.Patch 1 stops
__perf_pmu_sched_task()passing a NULLpmu_ctxtopmu->sched_task().armv8pmu_sched_task()is the only implementation that dereferences the argument, so the oops needs BRBE.Patch 2 makes
perf_pmu_sched_task()visit PMUs whose events are all CPU-wide. They are skipped today whenever the scheduled task has a perf event of its own, so branch records leak across task boundaries with perf record -b -a.intel_pmu_lbr_add()callsperf_sched_cb_inc()unconditionally, so x86 LBR is affected the same way.Dropping the early return alone leaves both dispatch paths running for one case, so
perf_ctx_sched_task_cb()gains a matching gate. The two could instead be collapsed intoperf_pmu_sched_task()alone, since__perf_pmu_sched_task()already passes the same epc, but that would move the callback out of theperf_ctx_disable()window for every PMU rather than just that one case, which seemed like too much for a fix tagged for stable.Patch 3 clears struct
perf_branch_entrywith a single struct assignment.perf_clear_branch_entry_bitfields()had drifted from the struct:new_typeand priv were never cleared, andarm_pmuv3.callocates the per-CPU branch stack withkmalloc().Tested on a 128 CPU arm64 machine with BRBE. A
WARN_ON_ONCE()at the gate patch 2 adds toperf_ctx_sched_task_cb()fires within seconds of running perf record -b -a alongside a task-bound event pinned to a different CPU.Changes in v6:
- Split thesched_task()fix into patches 1 and 2; the NULL dereference and the missed dispatch are separate bugs with different reachability.
- Gateperf_ctx_sched_task_cb()oncpc->task_epc. v5 removed the early return inperf_pmu_sched_task()without it, so both paths ran for a task whose event for that PMU is pinned to another CPU. Caught by theWARN_ON_ONCE()described above.
- Tag patches 1 and 2 for stable.
- Send separately from the BRBE series, rebased onto tip perf/core.[1] https://lore.kernel.org/all/[email protected]/
Based on tip perf/core (
f4dfab174244).Puranjay Mohan (3):
perf/core: Fix NULLpmu_ctxpassed topmu->sched_task()perf/core: Runsched_task()for PMUs with only CPU-wide events perf/core: Clear the whole branch entry inperf_clear_branch_entry()
Quoted 8 lines
arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/lbr.c | 6 +++--- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 16 ++-------------- kernel/events/core.c | 15 +++++++++++---- 6 files changed, 19 insertions(+), 24 deletions(-)