perf/core: Fix sampling period inconsistency across CPU migration
PATCH v2 1/1 perf/core: Fix sampling period inconsistency across CPU migration
When per-task software events are sampled, period_left is not managed consistently when task migration happens. The perf_event may observe a different hw_perf_event::period_left on the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a stale period_left after migration.
Introduce struct perf_task_context as a per-task container to preserve period_left across CPU migrations. A separate structure is used rather than adding fields to hw_perf_event, because hw_perf_event is a general-purpose structure shared by all event types (hardware, software, tracepoint, breakpoint, etc.) and embedding per-task sampling state there would bloat it for the majority of events that do not need it. perf_task_context is only allocated for per-task software sampling events.
Multiple per-CPU perf_event instances originating from the same perf_event_open caller share a single perf_task_context via refcounting. The perf_event owner field is used to distinguish events from different perf_event_open callers, preventing unrelated sampling sessions from interfering with each other. For inherited events (where owner is NULL), the inherit flag relaxes the owner check so that child events properly share perf_task_context. The allocation condition for inherited events checks that the parent event actually has a perf_task_ctxp, ensuring only genuine software events propagate the context. The existing perf_task_context lookup uses perf_lock_task_context() to safely access the task's event context under proper RCU and IRQ protection.
perf_task_context serves purely as a transport for period_left across CPU migrations. On event removal (swevent_del for non-clock events, cancel_hrtimer for clock events), hw_perf_event::period_left is backed up to perf_task_context::period_left. On event addition (swevent_add for non-clock events, start_hrtimer for clock events), perf_task_context::period_left is restored to hw_perf_event::period_left. During normal operation between migrations, hw_perf_event::period_left remains the sole working copy, keeping existing code paths unaffected.
To reproduce, force CPU migration during task-clock sampling:
$ sysbench cpu --threads=1 --time=60 run & $ sleep 0.1 $ TID=$(ls /proc/$!/task/ | grep -v "^$!$") $ perf record -e task-clock -c 1000000000 -t $TID &
# Force migration across CPUs every 1.2 seconds
$ while kill -0 $TID 2>/dev/null; do
taskset -p -c 0 $TID; sleep 1.2
taskset -p -c 1 $TID; sleep 1.2
taskset -p -c 2 $TID; sleep 1.2
done # Check sample intervals (expected: ~1.000s each)
$ perf script -F time | \
awk 'NR==1 {prev=$1; next} {print $1-prev; prev=$1}'Without this patch, sample intervals show significant deviation from the expected 1-second period after each migration. With this patch, intervals remain consistent.
---
Changes in v2:
- Use perf_lock_task_context() to safely access the task's event context, avoiding a potential use-after-free and IRQ inversion deadlock
- Tighten allocation condition for inherited events by checking parent_event->perf_task_ctxp instead of just parent_event
include/linux/perf_event.h | 18 +++++++++++ kernel/events/core.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+)
include/linux/perf_event.h +18 -0
@@ -829,6 +829,9 @@ struct perf_event {u16 read_size;struct hw_perf_event hw; +/* Per-task sampling state for sw events, survives CPU migration */ +struct perf_task_context *perf_task_ctxp; +struct perf_event_context *ctx;/* * event->pmu_ctx points to perf_event_pmu_context in which the event
@@ -1148,6 +1151,21 @@ struct perf_cpu_context {struct perf_event *heap_default[2 ]; }; +#define perf_event_equal_task_ctx (a1, a2) \ + ((a1)->config == (a2)->config && \ + (a1)->sample_period == (a2)->sample_period) + +/** +* struct perf_task_context - per-task software event context +* +* Shared across per-CPU perf_event instances of the same task to +* preserve period_left across CPU migrations. +*/ +struct perf_task_context { +refcount_t refcount; +local64_t period_left; +}; +struct perf_output_handle {struct perf_event *event;struct perf_buffer *rb;
kernel/events/core.c +77 -0
143 lines not on this page · raw message
Signature · 1 line
2.49.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Memory leak of struct perf_event_context in perf_get_task_ctxp() - [High] Cross-session…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Memory leak of struct perf_event_context in perf_get_task_ctxp()
- [High] Cross-session interference due to flawed inheritance matching in perf_get_task_ctxp() --
commit 1659d24a7199e1068822505af2a6e5f2de5e9e1a
Author: Minwoo Ahn <[email protected]>
perf/core: Fix sampling period inconsistency across CPU migration
This commit introduces struct perf_task_context to preserve the sampling state of per-task software events when task migration occurs. It allocates a shared context across per-CPU perf_event instances to transport the period_left value between CPUs.
[ ... ]
Quoted 2 lines
kernel/events/core.c +0
-0
[ ... ]
Quoted 9 lines
kernel/events/core.c +5
-0@@ -13259,6 +13291,41 @@ static void account_event(struct perf_event *event)account_pmu_sb_event (event); } +static struct perf_task_context * +perf_get_task_ctxp (struct perf_event *event,struct task_struct *task, +bool inherit) +{ +struct perf_task_context *ctxp =NULL ;
kernel/events/core.c static void account_event(struct perf_event *event) · line 13308
Quoted 10 lines
kernel/events/core.c +10
-0+struct perf_event_context *ctx; +struct perf_event *iter; +unsigned long flags; + + ctx =perf_lock_task_context (task, &flags); +if (ctx) { +list_for_each_entry (iter, &ctx->event_list, event_entry) { + if (iter->perf_task_ctxp && + (iter->owner == current || + (inherit && !iter->owner)) &&
Could this cause independent tracing sessions to interfere with each other?
If two separate sessions monitor the same process with identical configurations, their inherited child events will both have an owner of NULL.
Since perf_event_equal_task_ctx() only compares the config and sample_period, the second session's child event might incorrectly match the first session's event.
Would this lead to independent sessions sharing the same perf_task_ctxp and clobbering each other's period_left state during task migrations?
kernel/events/core.c static void account_event(struct perf_event *event) · line 13316
Quoted 8 lines
kernel/events/core.c +8
-0+ perf_event_equal_task_ctx(&iter->attr, + &event->attr)) { + ctxp = iter->perf_task_ctxp; +refcount_inc (&ctxp->refcount); +break ; + } + } +raw_spin_unlock_irqrestore (&ctx->lock, flags);
Does this leak the perf_event_context?
perf_lock_task_context() returns the context with its reference count incremented, so the caller is typically responsible for dropping the reference by calling put_ctx() after unlocking it.
Because put_ctx() is missing here, it seems this could leak the context and the task_struct it pins each time a per-task software sampling event is successfully created.
Quoted 11 lines
kernel/events/core.c +11
-0+ } + +if (!ctxp) { + ctxp = kzalloc_obj(struct perf_task_context); +if (!ctxp) +return NULL ; +refcount_set (&ctxp->refcount,1 ); + } + +return ctxp; +}
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1