Skip to contentSkip to conversation

allow trace events to be accessed from any CPU

PATCH In review Junxiao Chang 3 participants 4 messages May 27 – Sep 15 perf events

lore
Share & tools
lore
#0 lore raw

cover allow trace events to be accessed from any CPU

The issue was reproduced while debugging real-time scheduling performance. An extra SMP IPI is triggered when using perf stat to collect trace event statistics.

Trace events should be accessible from any CPU without requiring an IPI when the current CPU differs from the event CPU.

Set the trace event PMU scope to PERF_PMU_SCOPE_SYS_WIDE so that trace events can be accessed system-wide. With this change, no extra SMP IPI is triggered even when the current CPU differs from the event CPU.

Junxiao Chang (1):
perf/core: allow trace events to be accessed from any CPU

 kernel/events/core.c | 1 +
 1 file changed, 1 insertion(+)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/1 perf/core: allow trace events to be accessed from any CPU

Using perf to collect trace event statistics may introduce unnecessary SMP IPI interrupts. For example:

perf stat -e i915:i915_context_create -I 1000

In perf_event_read(), an SMP IPI is triggered when the event CPU differs from the local CPU.

Allow trace events to be accessed from any CPU so the event CPU can be set to the local CPU, eliminating the extra IPI overhead.

---
 kernel/events/core.c | 1 +
 1 file changed, 1 insertion(+)
kernel/events/core.c +1 -0
index acdb1f02924da..b3a817c0eb3a2 100644
@@ -11376,6 +11376,7 @@ static struct pmu perf_tracepoint = {
 	.start		= perf_swevent_start,
 	.stop		= perf_swevent_stop,
 	.read		= perf_swevent_read,
+	.scope		= PERF_PMU_SCOPE_SYS_WIDE,
 };
 
 static int perf_tp_filter_match(struct perf_event *event,
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/1] perf/core: allow trace events to be accessed from any CPU

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Critical] Cross-CPU invocation of __perf_event_read() corrupts the target CPU context's time due to…

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Cross-CPU invocation of __perf_event_read() corrupts the target CPU context's time due to non-monotonic local_clock() drift, causing integer underflows.
- [High] Setting PERF_PMU_SCOPE_SYS_WIDE incorrectly limits system-wide tracepoint event collection to a single CPU, breaking system-wide monitoring across the kernel.
- [High] Uncore PMU context migration during CPU hotplug is incorrectly triggered for tracepoint events, leading to per-CPU semantic violations and double-counting.
- [Medium] Per-task tracepoint events silently fail to update their time attributes due to an incorrect CPU context check when read cross-CPU. --

commit 115effb2b7801e60a78fb41cfb479a744b13b4de
Author: Junxiao Chang <[email protected]>

perf/core: allow trace events to be accessed from any CPU

This commit attempts to reduce SMP IPI overhead when reading trace events by setting the perf_tracepoint PMU scope to PERF_PMU_SCOPE_SYS_WIDE, allowing access from any CPU.

kernel/events/core.c static struct pmu perf_tracepoint = { · line 11327

Quoted 9 lines
kernel/events/core.c +1 -0
index bcaf175f3ded2..da9c41392130e 100644
@@ -11324,6 +11324,7 @@ static struct pmu perf_tracepoint = {
 	.start		= perf_swevent_start,
 	.stop		= perf_swevent_stop,
 	.read		= perf_swevent_read,
+	.scope		= PERF_PMU_SCOPE_SYS_WIDE,

[Severity: High]
Will this cause system-wide tracepoint event collection to be limited to a single CPU?

Setting perf_tracepoint.scope to PERF_PMU_SCOPE_SYS_WIDE causes the PMU framework to dynamically expose a cpumask sysfs file:

/sys/bus/event_source/devices/tracepoint/cpumask

This returns perf_online_sys_mask, which typically contains only a single CPU.

Userspace tools like perf record -a parse this sysfs attribute, assume the PMU is an uncore or system PMU, and only open tracepoint events on the single CPU specified in the mask.

Does this regression break system-wide software tracing by silently ignoring tracepoint activity on all other CPUs?

[Severity: High]
Could this trigger incorrect PMU context migration during CPU hotplug?

When a CPU is offlined, perf_event_clear_cpumask() iterates over all PMUs with a defined scope. For perf_tracepoint, it invokes perf_pmu_migrate_context() to migrate all tracepoint events from the offlining CPU to another online CPU.

Since tracepoint events are inherently per-CPU, migrating an event pinned to CPU A over to CPU B means CPU B will now host multiple identical tracepoint events.

Will this regression lead to duplicated records and double-counting when the tracepoint triggers on CPU B?

Also, when CPU A comes back online, its events are not migrated back, potentially leaving its tracepoint monitoring permanently broken.

[Severity: Critical]
Does bypassing the SMP IPI and calling __perf_event_read() cross-CPU risk corrupting the target CPU context's time?

If __perf_event_read() executes on the local CPU (CPU B) for an event active on a remote CPU (CPU A), it calls ctx_time_update_event(), which executes update_context_time(ctx).

This function calculates "now - ctx->timestamp" using perf_clock(), which wraps local_clock(). This returns the hardware clock of the current CPU (CPU B).

Since local_clock() is not strictly synchronized across CPUs, CPU B's clock can be slightly behind CPU A's last recorded timestamp.

Could this negative delta underflow the unsigned 64-bit integer, yielding a massive positive value that is added to ctx->time, permanently corrupting the timekeeping baseline for all events hosted on CPU A's context?

[Severity: Medium]
Will per-task tracepoint events fail to update their time attributes during cross-CPU reads?

When __perf_event_read() executes on the local CPU (CPU B) instead of the active CPU (CPU A), it checks if the task is currently active:

if (ctx->task && cpuctx->task_ctx != ctx)

    return;

Because cpuctx is retrieved via this_cpu_ptr(&perf_cpu_context) on CPU B, its task_ctx reflects CPU B's active task.

Since this naturally mismatches the event's context from CPU A, won't this regression cause the condition to incorrectly evaluate to true, silently returning and skipping perf_event_update_time()?

Quoted 1 line
kernel/events/core.c +0 -0
 };
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#3 lore raw

Re: [PATCH 1/1] perf/core: allow trace events to be accessed from any CPU

In reply to Junxiao Chang
Quoted 6 lines · Junxiao Chang

Using perf to collect trace event statistics may introduce unnecessary SMP IPI interrupts. For example:

perf stat -e i915:i915_context_create -I 1000

Quoted 7 lines

In perf_event_read(), an SMP IPI is triggered when the event CPU differs from the local CPU.

Allow trace events to be accessed from any CPU so the event CPU can be set to the local CPU, eliminating the extra IPI overhead.

Hi Junxiao,

Reducing IPIs is great, but the Sashiko review pointed at a bunch of issues that seemed like unintended consequences from the change. What is your plan to address those issues?

Thanks,
Ian

Quoted 19 lines
---
 kernel/events/core.c | 1 +
 1 file changed, 1 insertion(+)
kernel/events/core.c +1 -1
index acdb1f02924da..b3a817c0eb3a2 100644
@@ -11376,6 +11376,7 @@ static struct pmu perf_tracepoint = {
        .start          = perf_swevent_start,
        .stop           = perf_swevent_stop,
        .read           = perf_swevent_read,
+       .scope          = PERF_PMU_SCOPE_SYS_WIDE,
 };
 
 static int perf_tp_filter_match(struct perf_event *event,
--

2.43.0