perf/core: Fix sampling period inconsistency across CPU migration
PATCH v4 1/1 perf/core: Fix sampling period inconsistency across CPU migration
When per-task software events are sampled, period_left is not managed consistently when task migration happens. The perf_event may observe a different hw_perf_event::period_left on the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a stale period_left after migration.
Introduce struct perf_task_context as a per-task container to preserve period_left across CPU migrations. A separate structure is used rather than adding fields to hw_perf_event, because hw_perf_event is a general-purpose structure shared by all event types (hardware, software, tracepoint, breakpoint, etc.) and embedding per-task sampling state there would bloat it for the majority of events that do not need it. perf_task_context is only allocated for per-task software sampling events.
Multiple per-CPU perf_event instances originating from the same perf_event_open caller share a single perf_task_context via refcounting. The perf_event owner field is used to distinguish events from different perf_event_open callers, preventing unrelated sampling sessions from interfering with each other. For inherited events (where owner is NULL), child events are matched by verifying that their parent events share the same perf_task_ctxp, ensuring only events from the same profiling session share context. The allocation condition for inherited events checks that the parent event actually has a perf_task_ctxp, ensuring only genuine software events propagate the context. The perf_task_context lookup uses perf_lock_task_context() to safely access the task's event context under proper RCU and IRQ protection.
perf_task_context serves purely as a transport for period_left across CPU migrations. On event removal (swevent_del for non-clock events, cancel_hrtimer for clock events), hw_perf_event::period_left is backed up to perf_task_context::period_left. On event addition (swevent_add for non-clock events, start_hrtimer for clock events), perf_task_context::period_left is restored to hw_perf_event::period_left. During normal operation between migrations, hw_perf_event::period_left remains the sole working copy, keeping existing code paths unaffected.
To reproduce, force CPU migration during task-clock sampling:
$ sysbench cpu --threads=1 --time=60 run & $ sleep 0.1 $ TID=$(ls /proc/$!/task/ | grep -v "^$!$") $ perf record -e task-clock -c 1000000000 -t $TID &
# Force migration across CPUs every 1.2 seconds
$ while kill -0 $TID 2>/dev/null; do
taskset -p -c 0 $TID; sleep 1.2
taskset -p -c 1 $TID; sleep 1.2
taskset -p -c 2 $TID; sleep 1.2
done # Check sample intervals (expected: ~1.000s each)
$ perf script -F time | \
awk 'NR==1 {prev=$1; next} {print $1-prev; prev=$1}'Without this patch, sample intervals show significant deviation from the expected 1-second period after each migration. With this patch, intervals remain consistent.
---
Changes in v4:
- Fix perf_event_context reference leak by adding put_ctx() after perf_lock_task_context()
- Use parent event lineage (parent->perf_task_ctxp) to match inherited events, preventing cross-session interference when multiple profiling sessions monitor the same task
Changes in v3:
- Move struct perf_task_context and perf_event_equal_task_ctx macro inside #ifdef CONFIG_PERF_EVENTS guard to fix build error on configs where CONFIG_PERF_EVENTS is disabled (local64_t undefined)
Changes in v2:
- Use perf_lock_task_context() to safely access the task's event context, avoiding a potential use-after-free and IRQ inversion deadlock
- Tighten allocation condition for inherited events by checking parent_event->perf_task_ctxp instead of just parent_event
include/linux/perf_event.h | 18 +++++++++++ kernel/events/core.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+)
include/linux/perf_event.h +18 -0
@@ -829,6 +829,9 @@ struct perf_event {u16 read_size;struct hw_perf_event hw; +/* Per-task sampling state for sw events, survives CPU migration */ +struct perf_task_context *perf_task_ctxp; +struct perf_event_context *ctx;/* * event->pmu_ctx points to perf_event_pmu_context in which the event
@@ -1207,6 +1210,21 @@ perf_cgroup_from_task(struct task_struct *task, struct perf_event_context *ctx)#ifdef CONFIG_PERF_EVENTS +#define perf_event_equal_task_ctx (a1, a2) \ + ((a1)->config == (a2)->config && \ + (a1)->sample_period == (a2)->sample_period) + +/** +* struct perf_task_context - per-task software event context +* +* Shared across per-CPU perf_event instances of the same task to +* preserve period_left across CPU migrations. +*/ +struct perf_task_context { +refcount_t refcount; +local64_t period_left; +}; +extern struct perf_event_context *perf_cpu_task_ctx (void ); extern void *perf_aux_output_begin(struct perf_output_handle *handle,
kernel/events/core.c +81 -0
147 lines not on this page · raw message
Signature · 1 line
2.49.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Flawed event matching heuristic falsely groups distinct sampling sessions, causing them to corrupt…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Flawed event matching heuristic falsely groups distinct sampling sessions, causing them to corrupt each other's sampling period.
- [Medium] TOCTOU race in perf_get_task_ctxp prevents context sharing for concurrently created per-CPU events.
- [Medium] Plain lockless read of iter->owner risks data races with concurrent task exits.
--
commit fcbb40982fea3f4e611342ab82ef8efe806efcc9
Author: Minwoo Ahn <[email protected]>
perf/core: Fix sampling period inconsistency across CPU migration
This commit introduces struct perf_task_context to preserve period_left across CPU migrations for per-task software sampling events. A shared context is used to prevent period resets when a task migrates across CPUs.
Quoted 4 lines
kernel/events/core.c +0
-0
[ ... ]
Quoted 8 lines
kernel/events/core.c +4
-0@@ -13259,6 +13291,45 @@ static void account_event(struct perf_event *event)account_pmu_sb_event (event); } +static struct perf_task_context * +perf_get_task_ctxp (struct perf_event *event,struct task_struct *task, +struct perf_event *parent_event) +{
kernel/events/core.c static void account_event(struct perf_event *event) · line 13307
Quoted 10 lines
kernel/events/core.c +10
-0+struct perf_task_context *ctxp =NULL ; +struct perf_event_context *ctx; +struct perf_event *iter; +unsigned long flags; + + ctx =perf_lock_task_context (task, &flags); +if (ctx) { +list_for_each_entry (iter, &ctx->event_list, event_entry) { + if (iter->perf_task_ctxp && + (iter->owner == current ||
The event owner field is cleared asynchronously when the owner task exits or releases the event descriptor, such as via smp_store_release() in perf_event_release_kernel().
Could this plain read of iter->owner race with a concurrent task exit? Should this be accessed using READ_ONCE() to avoid data races?
kernel/events/core.c static void account_event(struct perf_event *event) · line 13313
Quoted 6 lines
kernel/events/core.c +6
-0+ (parent_event && !iter->owner && + iter->parent && + iter->parent->perf_task_ctxp == + parent_event->perf_task_ctxp)) && +perf_event_equal_task_ctx (&iter->attr, + &event->attr)) {
The perf_event_equal_task_ctx() macro only compares the config and sample_period attributes. If a user opens multiple independent software events for the same task with identical configs and periods (for example, perf record -e task-clock,task-clock), or if the events differ in unchecked attributes like exclude_user or exclude_kernel, will they incorrectly share the same perf_task_context?
If these independent events share the same context, when the task is scheduled out, both events would sequentially write their hw.period_left to the shared ctxp->period_left, clobbering the first write. Upon schedule in, both events would restore from this clobbered value.
Could this entangle and corrupt their respective sampling periodicities?
kernel/events/core.c static void account_event(struct perf_event *event) · line 13324
Quoted 11 lines
kernel/events/core.c +11
-0+ ctxp = iter->perf_task_ctxp; +refcount_inc (&ctxp->refcount); +break ; + } + } +raw_spin_unlock_irqrestore (&ctx->lock, flags); +put_ctx (ctx); + } + +if (!ctxp) { + ctxp = kzalloc_obj(struct perf_task_context);
If a userspace profiling application concurrently calls sys_perf_event_open() from multiple threads to create per-CPU events for the same task, both threads could perform the locked search above before either event reaches perf_install_in_context().
Since neither event is in ctx->event_list yet, will both threads fail to find a match, independently allocate new perf_task_context structures here, and fail to share the context?
Does this race condition leave the migration period inconsistency unfixed for concurrently created events?
Quoted 7 lines
kernel/events/core.c +7
-0+if (!ctxp) +return NULL ; +refcount_set (&ctxp->refcount,1 ); + } + +return ctxp; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Peter Zijlstra
maintainerQuoted 6 lines · Minwoo Ahn
When per-task software events are sampled,
period_leftis not managed consistently when task migration happens. Theperf_eventmay observe a differenthw_perf_event::period_lefton the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a staleperiod_leftafter migration.
How? This is just vague words, not actually saying anything of substance.
Quoted 1 line
Introduce struct
perf_task_contextas a per-task container to
How can you propose a solution to a non-defined problem?
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Minwoo Ahn
When per-task software events are sampled,
period_leftis not managed consistently when task migration happens. Theperf_eventmay observe a differenthw_perf_event::period_lefton the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a staleperiod_leftafter migration.
Quoted 6 lines
How? This is just vague words, not actually saying anything of substance.
Quoted 1 line
Introduce struct
perf_task_contextas a per-task container toHow can you propose a solution to a non-defined problem?
Let me describe the problem more concretely.
A common per-task sampling invocation such asperf record -e task-clock -c 1000000 ./a.out orperf record -t TID opens, unless the user constrains it with -C, one perf_event for each online CPU on the system. So a single target task ends up with N distinct perf_event objects, each carrying its own hw_perf_event::period_left that only counts down while the task is running on the CPU that object is bound to.
When the task migrates from CPU_X to CPU_Y, the event on CPU_X is scheduled out and the event on CPU_Y is scheduled in. They are distinct objects, so the partial period accumulated on CPU_X stays in that object and is dropped, and CPU_Y's event resumes from whatever period_left it last held -- typically a full sample_period left over from when its hrtimer was last armed.
Timeline (sample_period = 1.0s, migrate at t=0.6s):
t=0.0s on CPU0; event_CPU0->period_left = 1.0s
t=0.6s migrate CPU0 -> CPU1
event_CPU0 sched_out, period_left = 0.4s
event_CPU1 sched_in, period_left = 1.0s (never decremented)
t=1.6s first sample fires on CPU1 (task expected one at t=1.0s)
So sample intervals seen by the task no longer correspond to sample_period.
The example above uses task-clock for clarity (sample_period is in nanoseconds and period_left counts down with on-CPU time), but the same mechanism applies to any per-task software sampling event: each per-CPU object holds its own period_left and the partial progress on the source CPU is not transferred to the destination CPU on migration. What is dropped on migration is the unit each event counts in (occurrences for non-clock sw events, time for clock events).
perf record --per-thread avoids this by opening a single perf_event with cpu=-1, but it also disables inheritance, so it cannot sample threads spawned after the run starts. The default per-CPU mode is the one that supports inheritance, and it is the mode where the inconsistency above is observed.
The per-task period_left should remain consistent from the task's point of view across migrations, and that is the motivation behind this patch.
Thanks,
Minwoo