Skip to contentSkip to conversation

perf/core: Fix sampling period inconsistency across CPU migration

PATCH Superseded Minwoo Ahn 3 participants 3 messages Apr 28 – Apr 29 perf events

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Minwoo Ahn

author
#0 lore raw

PATCH 1/1 perf/core: Fix sampling period inconsistency across CPU migration

When per-task software events are sampled, period_left is not managed consistently when task migration happens. The perf_event may observe a different hw_perf_event::period_left on the new CPU, breaking the sampling periodicity. Even if a task was near its sampling point, it would use a stale period_left after migration.

Introduce struct perf_task_context as a per-task container to preserve period_left across CPU migrations. A separate structure is used rather than adding fields to hw_perf_event, because hw_perf_event is a general-purpose structure shared by all event types (hardware, software, tracepoint, breakpoint, etc.) and embedding per-task sampling state there would bloat it for the majority of events that do not need it. perf_task_context is only allocated for per-task software sampling events.

Multiple per-CPU perf_event instances originating from the same perf_event_open caller share a single perf_task_context via refcounting. The perf_event owner field is used to distinguish events from different perf_event_open callers, preventing unrelated sampling sessions from interfering with each other. For inherited events (where owner is NULL), the inherit flag relaxes the owner check so that child events properly share perf_task_context. The allocation condition also accounts for inherited events whose attr.type has been remapped from PERF_TYPE_SOFTWARE to a dynamic PMU type during initialization.

perf_task_context serves purely as a transport for period_left across CPU migrations. On event removal (swevent_del for non-clock events, cancel_hrtimer for clock events), hw_perf_event::period_left is backed up to perf_task_context::period_left. On event addition (swevent_add for non-clock events, start_hrtimer for clock events), perf_task_context::period_left is restored to hw_perf_event::period_left. During normal operation between migrations, hw_perf_event::period_left remains the sole working copy, keeping existing code paths unaffected.

To reproduce, force CPU migration during task-clock sampling:

  $ sysbench cpu --threads=1 --time=60 run &
  $ sleep 0.1
  $ TID=$(ls /proc/$!/task/ | grep -v "^$!$")
  $ perf record -e task-clock -c 1000000000 -t $TID &

# Force migration across CPUs every 1.2 seconds

  $ while kill -0 $TID 2>/dev/null; do
        taskset -p -c 0 $TID; sleep 1.2
        taskset -p -c 1 $TID; sleep 1.2
        taskset -p -c 2 $TID; sleep 1.2
    done
  # Check sample intervals (expected: ~1.000s each)
  $ perf script -F time | \
        awk 'NR==1 {prev=$1; next} {print $1-prev; prev=$1}'

Without this patch, sample intervals show significant deviation from the expected 1-second period after each migration. With this patch, intervals remain consistent.

---
 include/linux/perf_event.h | 18 +++++++++++
 kernel/events/core.c       | 75 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 93 insertions(+)
include/linux/perf_event.h +18 -0
index 48d851fbd8ea..84827f81cc9c 100644
@@ -829,6 +829,9 @@ struct perf_event {
 	u16				read_size;
 	struct hw_perf_event		hw;
 
+	/* Per-task sampling state for sw events, survives CPU migration */
+	struct perf_task_context	*perf_task_ctxp;
+
 	struct perf_event_context	*ctx;
 	/*
 	 * event->pmu_ctx points to perf_event_pmu_context in which the event
@@ -1148,6 +1151,21 @@ struct perf_cpu_context {
 	struct perf_event		*heap_default[2];
 };
 
+#define perf_event_equal_task_ctx(a1, a2)	\
+	((a1)->config == (a2)->config &&	\
+	 (a1)->sample_period == (a2)->sample_period)
+
+/**
+ * struct perf_task_context - per-task software event context
+ *
+ * Shared across per-CPU perf_event instances of the same task to
+ * preserve period_left across CPU migrations.
+ */
+struct perf_task_context {
+	refcount_t			refcount;
+	local64_t			period_left;
+};
+
 struct perf_output_handle {
 	struct perf_event		*event;
 	struct perf_buffer		*rb;
kernel/events/core.c +75 -0

141 lines not on this page · raw message

Signature · 1 line
2.49.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] Use-After-Free of perf_event_context in perf_get_task_ctxp() due to lockless…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Use-After-Free of perf_event_context in perf_get_task_ctxp() due to lockless task->perf_event_ctxp access.
- [High] IRQ inversion deadlock in perf_get_task_ctxp() due to raw_spin_lock() without disabling interrupts.
- [High] State corruption and logic flaw due to missing event type validation in perf_event_equal_task_ctx(). --

commit 9f612ffa1ab8888251bf71b52ebf7bdaadacbb11
Author: Minwoo Ahn <[email protected]>

perf/core: Fix sampling period inconsistency across CPU migration

This commit introduces a per-task container to preserve period_left across CPU migrations for software sampling events. The context is shared across multiple per-CPU perf_event instances originating from the same caller via refcounting.

Quoted 4 lines
include/linux/perf_event.h +0 -0
index 48d851fbd8ea..84827f81cc9c 100644

[ ... ]

include/linux/perf_event.h struct perf_cpu_context { · line 1156

Quoted 7 lines
include/linux/perf_event.h +3 -0
@@ -1148,6 +1151,21 @@ struct perf_cpu_context {
 	struct perf_event		*heap_default[2];
 };
 
+#define perf_event_equal_task_ctx(a1, a2)	\
+	((a1)->config == (a2)->config &&	\
+	 (a1)->sample_period == (a2)->sample_period)

Should this macro also compare attr.type? Since it only compares config and sample_period, unrelated events with small config IDs (such as an inherited hardware event and a software CPU-clock event that both have a config of 0) could erroneously match. This would cause independent sampling sessions to share state and overwrite each other's period_left.

Quoted 11 lines
include/linux/perf_event.h +11 -0
+
+/**
+ * struct perf_task_context - per-task software event context
+ *
+ * Shared across per-CPU perf_event instances of the same task to
+ * preserve period_left across CPU migrations.
+ */
+struct perf_task_context {
+	refcount_t			refcount;
+	local64_t			period_left;
+};

[ ... ]

Quoted 4 lines
kernel/events/core.c +0 -0
index 6d1f8bad7e1c..bd106e0b854a 100644

[ ... ]

kernel/events/core.c static void account_event(struct perf_event *event) · line 13299

Quoted 10 lines
kernel/events/core.c +6 -0
@@ -13259,6 +13291,40 @@ static void account_event(struct perf_event *event)
 	account_pmu_sb_event(event);
 }
 
+static struct perf_task_context *
+perf_get_task_ctxp(struct perf_event *event, struct task_struct *task,
+		   bool inherit)
+{
+	struct perf_task_context *ctxp = NULL;
+	struct perf_event_context *ctx = task->perf_event_ctxp;

Does this code risk a use-after-free of the perf_event_context?

The task->perf_event_ctxp pointer is read directly into the local ctx variable without holding rcu_read_lock() or using the safe perf_lock_task_context() helper.

Although the target task_struct is pinned by get_task_struct(), the perf_event_context it points to is dynamically managed. If the target task exits concurrently, perf_event_exit_task() could set task->perf_event_ctxp to NULL and schedule the context for destruction.

Because this accesses the context outside an RCU read-side critical section, the allocating thread could be preempted while the RCU grace period elapses, allowing the context memory to be freed before the lock is acquired below.

kernel/events/core.c static void account_event(struct perf_event *event) · line 13303

Quoted 4 lines
kernel/events/core.c +4 -0
+	struct perf_event *iter;
+
+	if (ctx) {
+		raw_spin_lock(&ctx->lock);

Could this cause an IRQ inversion deadlock?

perf_get_task_ctxp() is called from perf_event_alloc() during the sys_perf_event_open syscall, which executes in a sleepable process context with hardware interrupts enabled.

ctx->lock is an IRQ-safe raw spinlock heavily used in hardirq contexts. If a timer or PMU interrupt fires on the local CPU while this lock is held, the interrupt handler could attempt to acquire the same ctx->lock, resulting in an immediate IRQ inversion deadlock that hangs the CPU.

Quoted 6 lines
kernel/events/core.c +6 -0
+		list_for_each_entry(iter, &ctx->event_list, event_entry) {
+			if (iter->perf_task_ctxp &&
+			    (iter->owner == current ||
+			     (inherit && !iter->owner)) &&
+			    perf_event_equal_task_ctx(&iter->attr,
+						     &event->attr)) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Minwoo Ahn
kernel test robot noticed the following build errors:

Hi Minwoo,

kernel test robot noticed the following build errors:

[auto build test ERROR on perf-tools-next/perf-tools-next]
[also build test ERROR on tip/perf/core perf-tools/perf-tools linus/master v7.1-rc1 next-20260428] [cannot apply to acme/perf/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Minwoo-Ahn/perf-core-Fix-sampling-period-inconsistency-across-CPU-migration/20260429-115721
base:   https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git perf-tools-next
patch link:    https://lore.kernel.org/r/20260428115317.22839-1-mwahn402%40gmail.com
patch subject: [PATCH] perf/core: Fix sampling period inconsistency across CPU migration
config: arc-allnoconfig (https://download.01.org/0day-ci/archive/20260429/[email protected]/config)
compiler: arc-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260429/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from include/linux/trace_events.h:10,
from include/trace/syscall.h:7, from include/linux/syscalls.h:95, from include/linux/syscalls_api.h:1, from kernel/sched/sched.h:64, from kernel/sched/rq-offsets.c:5:

>> include/linux/perf_event.h:1166:9: error: unknown type name 'local64_t'; did you mean 'local_t'?
    1166 |         local64_t                       period_left;
         |         ^~~~~~~~~
         |         local_t

make[3]: *** [scripts/Makefile.build:184: kernel/sched/rq-offsets.s] Error 1 make[3]: Target 'prepare' not remade because of errors. make[2]: *** [Makefile:1337: prepare0] Error 2
make[2]: Target 'prepare' not remade because of errors. make[1]: *** [Makefile:248: __sub-make] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.

vim +1166 include/linux/perf_event.h

1153
1154 #define perf_event_equal_task_ctx(a1, a2) \
1155 ((a1)->config == (a2)->config && \
1156 (a1)->sample_period == (a2)->sample_period)
1157
1158 /**
1159 * struct perf_task_context - per-task software event context
1160 *
1161 * Shared across per-CPU perf_event instances of the same task to
1162 * preserve period_left across CPU migrations.
1163 */
1164 struct perf_task_context {
1165 refcount_t refcount;

> 1166		local64_t			period_left;

1167 };
1168

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki