perf/core: Fix sampling period inconsistency across CPU migration
PATCH v3 1/1 perf/core: Fix sampling period inconsistency across CPU migration
When per-task software events are sampled, period_left is not managed consistently when task migration happens. The perf_event may observe a different hw_perf_event::period_left on the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a stale period_left after migration.
Introduce struct perf_task_context as a per-task container to preserve period_left across CPU migrations. A separate structure is used rather than adding fields to hw_perf_event, because hw_perf_event is a general-purpose structure shared by all event types (hardware, software, tracepoint, breakpoint, etc.) and embedding per-task sampling state there would bloat it for the majority of events that do not need it. perf_task_context is only allocated for per-task software sampling events.
Multiple per-CPU perf_event instances originating from the same perf_event_open caller share a single perf_task_context via refcounting. The perf_event owner field is used to distinguish events from different perf_event_open callers, preventing unrelated sampling sessions from interfering with each other. For inherited events (where owner is NULL), the inherit flag relaxes the owner check so that child events properly share perf_task_context. The allocation condition for inherited events checks that the parent event actually has a perf_task_ctxp, ensuring only genuine software events propagate the context. The existing perf_task_context lookup uses perf_lock_task_context() to safely access the task's event context under proper RCU and IRQ protection.
perf_task_context serves purely as a transport for period_left across CPU migrations. On event removal (swevent_del for non-clock events, cancel_hrtimer for clock events), hw_perf_event::period_left is backed up to perf_task_context::period_left. On event addition (swevent_add for non-clock events, start_hrtimer for clock events), perf_task_context::period_left is restored to hw_perf_event::period_left. During normal operation between migrations, hw_perf_event::period_left remains the sole working copy, keeping existing code paths unaffected.
To reproduce, force CPU migration during task-clock sampling:
$ sysbench cpu --threads=1 --time=60 run & $ sleep 0.1 $ TID=$(ls /proc/$!/task/ | grep -v "^$!$") $ perf record -e task-clock -c 1000000000 -t $TID &
# Force migration across CPUs every 1.2 seconds
$ while kill -0 $TID 2>/dev/null; do
taskset -p -c 0 $TID; sleep 1.2
taskset -p -c 1 $TID; sleep 1.2
taskset -p -c 2 $TID; sleep 1.2
done # Check sample intervals (expected: ~1.000s each)
$ perf script -F time | \
awk 'NR==1 {prev=$1; next} {print $1-prev; prev=$1}'Without this patch, sample intervals show significant deviation from the expected 1-second period after each migration. With this patch, intervals remain consistent.
---
Changes in v3:
- Move struct perf_task_context and perf_event_equal_task_ctx macro inside #ifdef CONFIG_PERF_EVENTS guard to fix build error on configs where CONFIG_PERF_EVENTS is disabled (local64_t undefined)
Changes in v2:
- Use perf_lock_task_context() to safely access the task's event context, avoiding a potential use-after-free and IRQ inversion deadlock
- Tighten allocation condition for inherited events by checking parent_event->perf_task_ctxp instead of just parent_event
include/linux/perf_event.h | 18 +++++++++++ kernel/events/core.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+)
include/linux/perf_event.h +18 -0
@@ -829,6 +829,9 @@ struct perf_event {u16 read_size;struct hw_perf_event hw; +/* Per-task sampling state for sw events, survives CPU migration */ +struct perf_task_context *perf_task_ctxp; +struct perf_event_context *ctx;/* * event->pmu_ctx points to perf_event_pmu_context in which the event
@@ -1207,6 +1210,21 @@ perf_cgroup_from_task(struct task_struct *task, struct perf_event_context *ctx)#ifdef CONFIG_PERF_EVENTS +#define perf_event_equal_task_ctx (a1, a2) \ + ((a1)->config == (a2)->config && \ + (a1)->sample_period == (a2)->sample_period) + +/** +* struct perf_task_context - per-task software event context +* +* Shared across per-CPU perf_event instances of the same task to +* preserve period_left across CPU migrations. +*/ +struct perf_task_context { +refcount_t refcount; +local64_t period_left; +}; +extern struct perf_event_context *perf_cpu_task_ctx (void ); extern void *perf_aux_output_begin(struct perf_output_handle *handle,
kernel/events/core.c +77 -0
143 lines not on this page · raw message
Signature · 1 line
2.49.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Missing put_ctx() in perf_get_task_ctxp() causes a permanent reference leak of perf_event_context…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Missing put_ctx() in perf_get_task_ctxp() causes a permanent reference leak of perf_event_context and task_struct.
- [High] Erroneous perf_task_ctxp sharing clobbers states of independent profiling sessions.
--
commit 8823a0e265a5483fceb1f344697e122c9270c810
Author: Minwoo Ahn <[email protected]>
perf/core: Fix sampling period inconsistency across CPU migration
This commit introduces struct perf_task_context to preserve period_left across CPU migrations for per-task software sampling events. It adds allocation and refcounting logic to share this context among related events on the same task.
Quoted 4 lines
kernel/events/core.c +0
-0
[ ... ]
Quoted 9 lines
kernel/events/core.c +5
-0@@ -13259,6 +13291,41 @@ static void account_event(struct perf_event *event)account_pmu_sb_event (event); } +static struct perf_task_context * +perf_get_task_ctxp (struct perf_event *event,struct task_struct *task, +bool inherit) +{ +struct perf_task_context *ctxp =NULL ;
kernel/events/core.c static void account_event(struct perf_event *event) · line 13308
Quoted 10 lines
kernel/events/core.c +10
-0+struct perf_event_context *ctx; +struct perf_event *iter; +unsigned long flags; + + ctx =perf_lock_task_context (task, &flags); +if (ctx) { +list_for_each_entry (iter, &ctx->event_list, event_entry) { + if (iter->perf_task_ctxp && + (iter->owner == current || + (inherit && !iter->owner)) &&
For inherited child events, both iter->owner and event->owner will be NULL. If a parent task is being profiled by two separate and independent perf sessions, will this (inherit && !iter->owner) condition inadvertently merge both child events into a single perf_task_context?
Would it be safer to verify parent lineage, for example by checking if iter->parent->perf_task_ctxp equals event->parent->perf_task_ctxp, to ensure the inherited events actually belong to the same profiling session?
kernel/events/core.c static void account_event(struct perf_event *event) · line 13310
Quoted 2 lines
kernel/events/core.c +2
-0+ perf_event_equal_task_ctx(&iter->attr, + &event->attr)) {
Could this logic mistakenly merge the sampling states of completely independent perf events in the same task?
If a process opens multiple distinct sampling events for the same task with identical configurations and sample periods, they appear to pass this equality check. Since they share the same perf_task_context, won't they corrupt each other's period_left states when the task schedules out?
kernel/events/core.c static void account_event(struct perf_event *event) · line 13316
Quoted 6 lines
kernel/events/core.c +6
-0+ ctxp = iter->perf_task_ctxp; +refcount_inc (&ctxp->refcount); +break ; + } + } +raw_spin_unlock_irqrestore (&ctx->lock, flags);
The perf_lock_task_context() helper takes a reference on the returned ctx using refcount_inc_not_zero().
Does this path leak the perf_event_context (and consequently the pinned task_struct) by returning without calling put_ctx(ctx) to drop that reference?
Quoted 11 lines
kernel/events/core.c +11
-0+ } + +if (!ctxp) { + ctxp = kzalloc_obj(struct perf_task_context); +if (!ctxp) +return NULL ; +refcount_set (&ctxp->refcount,1 ); + } + +return ctxp; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1