perf/core: Fix sampling period inconsistency across CPU migration
PATCH 1/1 perf/core: Fix sampling period inconsistency across CPU migration
When per-task software events are sampled, period_left is not managed consistently when task migration happens. The perf_event may observe a different hw_perf_event::period_left on the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a stale period_left after migration.
Introduce struct perf_task_context as a per-task container to preserve period_left across CPU migrations. A separate structure is used rather than adding fields to hw_perf_event, because hw_perf_event is a general-purpose structure shared by all event types (hardware, software, tracepoint, breakpoint, etc.) and embedding per-task sampling state there would bloat it for the majority of events that do not need it. perf_task_context is only allocated for per-task software sampling events.
Multiple per-CPU perf_event instances originating from the same perf_event_open caller share a single perf_task_context via refcounting. The perf_event owner field is used to distinguish events from different perf_event_open callers, preventing unrelated sampling sessions from interfering with each other. For inherited events (where owner is NULL), the inherit flag relaxes the owner check so that child events properly share perf_task_context. The allocation condition also accounts for inherited events whose attr.type has been remapped from PERF_TYPE_SOFTWARE to a dynamic PMU type during initialization.
perf_task_context serves purely as a transport for period_left across CPU migrations. On event removal (swevent_del for non-clock events, cancel_hrtimer for clock events), hw_perf_event::period_left is backed up to perf_task_context::period_left. On event addition (swevent_add for non-clock events, start_hrtimer for clock events), perf_task_context::period_left is restored to hw_perf_event::period_left. During normal operation between migrations, hw_perf_event::period_left remains the sole working copy, keeping existing code paths unaffected.
To reproduce, force CPU migration during task-clock sampling:
$ sysbench cpu --threads=1 --time=60 run & $ sleep 0.1 $ TID=$(ls /proc/$!/task/ | grep -v "^$!$") $ perf record -e task-clock -c 1000000000 -t $TID &
# Force migration across CPUs every 1.2 seconds
$ while kill -0 $TID 2>/dev/null; do
taskset -p -c 0 $TID; sleep 1.2
taskset -p -c 1 $TID; sleep 1.2
taskset -p -c 2 $TID; sleep 1.2
done # Check sample intervals (expected: ~1.000s each)
$ perf script -F time | \
awk 'NR==1 {prev=$1; next} {print $1-prev; prev=$1}'Without this patch, sample intervals show significant deviation from the expected 1-second period after each migration. With this patch, intervals remain consistent.
--- include/linux/perf_event.h | 18 +++++++++++ kernel/events/core.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+)
include/linux/perf_event.h +18 -0
@@ -829,6 +829,9 @@ struct perf_event {u16 read_size;struct hw_perf_event hw; +/* Per-task sampling state for sw events, survives CPU migration */ +struct perf_task_context *perf_task_ctxp; +struct perf_event_context *ctx;/* * event->pmu_ctx points to perf_event_pmu_context in which the event
@@ -1148,6 +1151,21 @@ struct perf_cpu_context {struct perf_event *heap_default[2 ]; }; +#define perf_event_equal_task_ctx (a1, a2) \ + ((a1)->config == (a2)->config && \ + (a1)->sample_period == (a2)->sample_period) + +/** +* struct perf_task_context - per-task software event context +* +* Shared across per-CPU perf_event instances of the same task to +* preserve period_left across CPU migrations. +*/ +struct perf_task_context { +refcount_t refcount; +local64_t period_left; +}; +struct perf_output_handle {struct perf_event *event;struct perf_buffer *rb;
kernel/events/core.c +75 -0
141 lines not on this page · raw message
Signature · 1 line
2.49.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] Use-After-Free of perf_event_context in perf_get_task_ctxp() due to lockless…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Use-After-Free of perf_event_context in perf_get_task_ctxp() due to lockless task->perf_event_ctxp access.
- [High] IRQ inversion deadlock in perf_get_task_ctxp() due to raw_spin_lock() without disabling interrupts.
- [High] State corruption and logic flaw due to missing event type validation in perf_event_equal_task_ctx(). --
commit 9f612ffa1ab8888251bf71b52ebf7bdaadacbb11
Author: Minwoo Ahn <[email protected]>
perf/core: Fix sampling period inconsistency across CPU migration
This commit introduces a per-task container to preserve period_left across CPU migrations for software sampling events. The context is shared across multiple per-CPU perf_event instances originating from the same caller via refcounting.
Quoted 4 lines
include/linux/perf_event.h +0
-0
[ ... ]
include/linux/perf_event.h struct perf_cpu_context { · line 1156
Quoted 7 lines
include/linux/perf_event.h +3
-0@@ -1148,6 +1151,21 @@ struct perf_cpu_context {struct perf_event *heap_default[2 ]; }; +#define perf_event_equal_task_ctx (a1, a2) \ + ((a1)->config == (a2)->config && \ + (a1)->sample_period == (a2)->sample_period)
Should this macro also compare attr.type? Since it only compares config and sample_period, unrelated events with small config IDs (such as an inherited hardware event and a software CPU-clock event that both have a config of 0) could erroneously match. This would cause independent sampling sessions to share state and overwrite each other's period_left.
Quoted 11 lines
include/linux/perf_event.h +11
-0+ +/** +* struct perf_task_context - per-task software event context +* +* Shared across per-CPU perf_event instances of the same task to +* preserve period_left across CPU migrations. +*/ +struct perf_task_context { +refcount_t refcount; +local64_t period_left; +};
[ ... ]
Quoted 4 lines
kernel/events/core.c +0
-0
[ ... ]
kernel/events/core.c static void account_event(struct perf_event *event) · line 13299
Quoted 10 lines
kernel/events/core.c +6
-0@@ -13259,6 +13291,40 @@ static void account_event(struct perf_event *event)account_pmu_sb_event (event); } +static struct perf_task_context * +perf_get_task_ctxp (struct perf_event *event,struct task_struct *task, +bool inherit) +{ +struct perf_task_context *ctxp =NULL ; +struct perf_event_context *ctx = task->perf_event_ctxp;
Does this code risk a use-after-free of the perf_event_context?
The task->perf_event_ctxp pointer is read directly into the local ctx variable without holding rcu_read_lock() or using the safe perf_lock_task_context() helper.
Although the target task_struct is pinned by get_task_struct(), the perf_event_context it points to is dynamically managed. If the target task exits concurrently, perf_event_exit_task() could set task->perf_event_ctxp to NULL and schedule the context for destruction.
Because this accesses the context outside an RCU read-side critical section, the allocating thread could be preempted while the RCU grace period elapses, allowing the context memory to be freed before the lock is acquired below.
kernel/events/core.c static void account_event(struct perf_event *event) · line 13303
Quoted 4 lines
kernel/events/core.c +4
-0+struct perf_event *iter; + +if (ctx) { +raw_spin_lock (&ctx->lock);
Could this cause an IRQ inversion deadlock?
perf_get_task_ctxp() is called from perf_event_alloc() during the sys_perf_event_open syscall, which executes in a sleepable process context with hardware interrupts enabled.
ctx->lock is an IRQ-safe raw spinlock heavily used in hardirq contexts. If a timer or PMU interrupt fires on the local CPU while this lock is held, the interrupt handler could attempt to acquire the same ctx->lock, resulting in an immediate IRQ inversion deadlock that hangs the CPU.
Quoted 6 lines
kernel/events/core.c +6
-0+list_for_each_entry (iter, &ctx->event_list, event_entry) { +if (iter->perf_task_ctxp && + (iter->owner == current || + (inherit && !iter->owner)) && +perf_event_equal_task_ctx (&iter->attr, + &event->attr)) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Minwoo,
kernel test robot noticed the following build errors:
[auto build test ERROR on perf-tools-next/perf-tools-next]
[also build test ERROR on tip/perf/core perf-tools/perf-tools linus/master v7.1-rc1 next-20260428] [cannot apply to acme/perf/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Minwoo-Ahn/perf-core-Fix-sampling-period-inconsistency-across-CPU-migration/20260429-115721 base: https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git perf-tools-next patch link: https://lore.kernel.org/r/20260428115317.22839-1-mwahn402%40gmail.com patch subject: [PATCH] perf/core: Fix sampling period inconsistency across CPU migration config: arc-allnoconfig (https://download.01.org/0day-ci/archive/20260429/[email protected]/config) compiler: arc-linux-gcc (GCC) 15.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260429/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from include/linux/trace_events.h:10,
from include/trace/syscall.h:7, from include/linux/syscalls.h:95, from include/linux/syscalls_api.h:1, from kernel/sched/sched.h:64, from kernel/sched/rq-offsets.c:5:
>> include/linux/perf_event.h:1166:9: error: unknown type name 'local64_t'; did you mean 'local_t'?
1166 | local64_t period_left;
| ^~~~~~~~~
| local_t make[3]: *** [scripts/Makefile.build:184: kernel/sched/rq-offsets.s] Error 1 make[3]: Target 'prepare' not remade because of errors. make[2]: *** [Makefile:1337: prepare0] Error 2
make[2]: Target 'prepare' not remade because of errors. make[1]: *** [Makefile:248: __sub-make] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +1166 include/linux/perf_event.h
1153
1154 #define perf_event_equal_task_ctx(a1, a2) \
1155 ((a1)->config == (a2)->config && \
1156 (a1)->sample_period == (a2)->sample_period)
1157
1158 /**
1159 * struct perf_task_context - per-task software event context
1160 *
1161 * Shared across per-CPU perf_event instances of the same task to
1162 * preserve period_left across CPU migrations.
1163 */
1164 struct perf_task_context {
1165 refcount_t refcount;
> 1166 local64_t period_left;
1167 };
1168
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki