perf/core: Fix sigtrap task_work races
v4 cover perf/core: Fix sigtrap task_work races
Two fixes for sigtrap task_work handling since the switch to always enqueue SIGTRAP via task_work.
The warning splats and PoCs for both patches are too long to inline in the commit messages, so they are provided as separate replies at the following links:
Patch 1: https://lore.kernel.org/all/[email protected]/
Patch 2: https://lore.kernel.org/all/[email protected]/
Patch 1 fixes a re-entrancy race in __perf_event_overflow(): pending_work was set only after task_work_add(), so an NMI could re-enter and enqueue the same pending_task twice, creating a cycle in the task work list and unbalancing the event refcount. Claim pending_work before enqueuing, and clear it if the enqueue fails.
Patch 2 fixes a WARN_ON_ONCE() in perf_sigtrap() triggered by the context fast-swap path swapping ctx->task while a sigtrap event has pending task work. SIGTRAP is targeted at ctx->task, so exclude sigtrap events from fast-swap alongside inherit+sample-read events.
Changelog
=========
v3 -> v4:
- v3: https://lore.kernel.org/all/[email protected]/
- Patch 1: Use acquire/release pairs for pending_work updates (Sebastian)
- Patch 2: Remove redundant nr_no_switch_fast updates (Sebastian)
v2 -> v3:
- v2: https://lore.kernel.org/all/[email protected]/
- Added Patch 2 to fix WARN_ON_ONCE() in perf_sigtrap()
v1 -> v2:
- v1: https://lore.kernel.org/all/[email protected]/
- Drop the redundant event->pending_work assignment. (Sashiko)
Tengda Wu (2):
perf/core: Prevent duplicate sigtrap task work enqueue perf/core: Disable context fast-swap for sigtrap events
include/linux/perf_event.h | 8 +++---- kernel/events/core.c | 48 +++++++++++++++++++++++++------------- 2 files changed, 35 insertions(+), 21 deletions(-)
Signature · 1 line
2.34.1
PATCH v4 1/2 perf/core: Prevent duplicate sigtrap task work enqueue
The sigtrap handling in __perf_event_overflow() checks event->pending_work before calling task_work_add(), but only sets pending_work after the task work has been queued.
This leaves a window where an NMI can interrupt task_work_add() and re-enter __perf_event_overflow() for the same event. Since pending_work is still zero, the nested invocation can enqueue the same pending_task again.
The re-entrant call path looks like this:
__perf_event_overflow()
task_work_add()
kasan_record_aux_stack()
<NMI>
__perf_event_overflow()
task_work_add()The nested invocation may enqueue pending_task before the outer invocation does. When the outer invocation resumes, it inserts the same task work entry again, potentially creating a cycle in the task work list.
As a result, perf_pending_task() may be executed repeatedly, leading to unbalanced event reference counting.
Fix this by atomically claiming pending_work with cmpxchg_acquire() before calling task_work_add(). This ensures that only one invocation can enqueue the task work, even when another overflow occurs re-entrantly from an NMI.
If task_work_add() fails, clear the claim with smp_store_release() to allow a subsequent overflow to retry.
In perf_pending_task(), defer clearing pending_work until after perf_sigtrap() has completed. Use smp_store_release() to ensure that a subsequent overflow cannot re-queue the task work before the previous invocation has finished handling it.
Also use smp_load_acquire() when checking pending_work in the exclude_kernel warning path to ensure the read is properly ordered.
--- kernel/events/core.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)
kernel/events/core.c +26 -13
@@ -7775,10 +7775,14 @@ static void perf_pending_task(struct callback_head *head)*/ rctx =perf_swevent_get_recursion_context (); -if (event->pending_work) { - event->pending_work =0 ; +if (smp_load_acquire (&event->pending_work)) {perf_sigtrap (event);local_dec (&event->ctx->nr_no_switch_fast); +/* +* Ensure the pending work is fully handled before allowing +* another overflow to queue it. See __perf_event_overflow(). +*/ +smp_store_release (&event->pending_work,0 ); }put_event (event);
@@ -10950,16 +10954,25 @@ static int __perf_event_overflow(struct perf_event *event, notify_mode =in_nmi () ?TWA_NMI_CURRENT :TWA_RESUME ; -if (!event->pending_work && - !task_work_add (current, &event->pending_task, notify_mode)) { - event->pending_work = pending_id; -local_inc (&event->ctx->nr_no_switch_fast); -WARN_ON_ONCE (!atomic_long_inc_not_zero (&event->refcount)); - - event->pending_addr =0 ; -if (valid_sample && (data->sample_flags &PERF_SAMPLE_ADDR )) - event->pending_addr = data->addr; - +/* +* Claim pending_work before queueing the task work to prevent +* another overflow from queueing it concurrently. +*/ +if (cmpxchg_acquire (&event->pending_work,0 , pending_id) ==0 ) { +if (!task_work_add (current, &event->pending_task, notify_mode)) { +local_inc (&event->ctx->nr_no_switch_fast); +WARN_ON_ONCE (!atomic_long_inc_not_zero (&event->refcount)); + + event->pending_addr =0 ; +if (valid_sample && (data->sample_flags &PERF_SAMPLE_ADDR )) + event->pending_addr = data->addr; + }else { +/* +* Publish the failed queueing attempt so +* a subsequent overflow can retry. +*/ +smp_store_release (&event->pending_work,0 ); + } }else if (event->attr.exclude_kernel && valid_sample) {/* * Should not be able to return to user space without
@@ -10973,7 +10986,7 @@ static int __perf_event_overflow(struct perf_event *event,* To approximate progress (with false negatives), * check 32-bit hash of the current IP. */ -WARN_ON_ONCE (event->pending_work != pending_id); +WARN_ON_ONCE (smp_load_acquire (&event->pending_work) != pending_id); } }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v4 1/2] perf/core: Prevent duplicate sigtrap task work enqueue
Sashiko has reviewed this patch and found no issues.
Quoted 15 lines
The sigtrap handling in
__perf_event_overflow()checksevent->pending_workbefore callingtask_work_add(), but only setspending_workafter the task work has been queued.This leaves a window where an NMI can interrupt
task_work_add()and re-enter__perf_event_overflow()for the same event. Sincepending_workis still zero, the nested invocation can enqueue the samepending_taskagain.The re-entrant call path looks like this:
__perf_event_overflow()task_work_add() kasan_record_aux_stack() <NMI>
Quoted 6 lines
[ ... ]
theexclude_kernelwarning path to ensure the read is properly ordered.Fixes c5d93d23a260 ("perf: Enqueue SIGTRAP always via task_work.")Signed-off-by Tengda Wu <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v4 2/2 perf/core: Disable context fast-swap for sigtrap events
In reply to Tengda Wuperf_sigtrap() contains:
WARN_ON_ONCE(event->ctx->task != current)
The warning assumes that an event's ctx->task always matches the task that ends up running the pending task_work. That assumption can be broken by the context fast-swap path.
The failure sequence is:
1. A -> B switch, perf_event_context_sched_out() takes the fast path and does WRITE_ONCE(ctx->task, next) / WRITE_ONCE(next_ctx->task, task). An NMI then hits while the context is swapped, and __perf_event_overflow() queues pending task_work on the *current* task (A), even though the event now belongs to B's swapped-in context view.
2. B -> A switch. Because A now has pending task_work, nr_no_switch_fast is non-zero, so the slow path is taken and ctx->task is *not* swapped back.
3. A returns to user mode, task_work runs perf_pending_task() -> perf_sigtrap(), and event->ctx->task no longer equals current, tripping the WARN_ON_ONCE().
The context fast-swap path was already excluded for events that need stable task association (inherit + sample read). sigtrap events have the same requirement: the SIGTRAP is targeted at ctx->task and must not be delivered after ctx->task has been swapped to another task.
Fix this by also excluding events with attr->sigtrap from the fast-swap path, i.e. bump nr_no_switch_fast for sigtrap events in list_add_event()/list_del_event(), and update the comment describing the invariant accordingly.
--- include/linux/perf_event.h | 8 +++----- kernel/events/core.c | 11 +++++++---- 2 files changed, 10 insertions(+), 9 deletions(-)
include/linux/perf_event.h +3 -5
@@ -1066,11 +1066,9 @@ struct perf_event_context {* The count of events for which using the switch-out fast path * should be avoided. * -* Sum (event->pending_work + events with -* (attr->inherit && (attr->sample_type & PERF_SAMPLE_READ))) -* -* The SIGTRAP is targeted at ctx->task, as such it won't do changing -* that until the signal is delivered. +* Sum (events with +* (attr->inherit && (attr->sample_type & PERF_SAMPLE_READ)) + +* events with attr->sigtrap) */ local_t nr_no_switch_fast; };
kernel/events/core.c +7 -4
@@ -2009,7 +2009,7 @@ list_add_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user++;if (event->attr.inherit_stat) ctx->nr_stat++; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_inc (&ctx->nr_no_switch_fast);if (event->state >PERF_EVENT_STATE_OFF )
@@ -2235,7 +2235,7 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user--;if (event->attr.inherit_stat) ctx->nr_stat--; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_dec (&ctx->nr_no_switch_fast);list_del_rcu (&event->event_entry);
@@ -3841,6 +3841,11 @@ perf_event_context_sched_out(struct task_struct *task, struct task_struct *next)* SAMPLE_READ events they should be switched * out using the slow path so that they are * treated as if they were distinct contexts. +* +* A context holding a sigtrap event must also +* take the slow path, even when nothing is +* pending, since the sigtrap delivery relies on +* the ctx->task relation as well. */ raw_spin_unlock (&next_ctx->lock);rcu_read_unlock ();
@@ -7777,7 +7782,6 @@ static void perf_pending_task(struct callback_head *head)if (smp_load_acquire (&event->pending_work)) {perf_sigtrap (event); -local_dec (&event->ctx->nr_no_switch_fast);/* * Ensure the pending work is fully handled before allowing * another overflow to queue it. See __perf_event_overflow().
@@ -10960,7 +10964,6 @@ static int __perf_event_overflow(struct perf_event *event,*/ if (cmpxchg_acquire (&event->pending_work,0 , pending_id) ==0 ) {if (!task_work_add (current, &event->pending_task, notify_mode)) { -local_inc (&event->ctx->nr_no_switch_fast);WARN_ON_ONCE (!atomic_long_inc_not_zero (&event->refcount)); event->pending_addr =0 ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v4 2/2] perf/core: Disable context fast-swap for sigtrap events
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] perf/core: list_del_event() drops nr_no_switch_fast before perf_pending_task() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/core: list_del_event() drops nr_no_switch_fast before perf_pending_task()
--
commit a5820a5e4b514a31e08f42157caca3b1bf8abed1
Author: Tengda Wu <[email protected]>
perf/core: Disable context fast-swap for sigtrap events
This commit disables the context switch fast-swap path for contexts containing sigtrap events by accounting attr->sigtrap in nr_no_switch_fast during list_add_event() and list_del_event() instead of __perf_event_overflow() and perf_pending_task().
Quoted 22 lines
kernel/events/core.c +2
-2@@ -1999,7 +1999,7 @@ list_add_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user++;if (event->attr.inherit_stat) ctx->nr_stat++; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_inc (&ctx->nr_no_switch_fast);if (event->state >PERF_EVENT_STATE_OFF )@@ -2225,7 +2225,7 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user--;if (event->attr.inherit_stat) ctx->nr_stat--; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_dec (&ctx->nr_no_switch_fast);list_del_rcu (&event->event_entry);
[ ... ]
Quoted 9 lines
kernel/events/core.c +0
-1@@ -7758,7 +7763,6 @@ static void perf_pending_task(struct callback_head *head)if (smp_load_acquire (&event->pending_work)) {perf_sigtrap (event); -local_dec (&event->ctx->nr_no_switch_fast);/* * Ensure the pending work is fully handled before allowing * another overflow to queue it. See __perf_event_overflow().
kernel/events/core.c static int __perf_event_overflow(struct perf_event *event, · line 10944
Quoted 7 lines
kernel/events/core.c +0
-1@@ -10935,7 +10939,6 @@ static int __perf_event_overflow(struct perf_event *event,*/ if (cmpxchg_acquire (&event->pending_work,0 , pending_id) ==0 ) {if (!task_work_add (current, &event->pending_task, notify_mode)) { -local_inc (&event->ctx->nr_no_switch_fast);WARN_ON_ONCE (!atomic_long_inc_not_zero (&event->refcount)); event->pending_addr =0 ;
[Severity: Medium]
Can ctx->nr_no_switch_fast drop to zero while event->pending_task is still queued if the sigtrap event is detached before perf_pending_task() runs?
Detaching an event does not cancel or flush event->pending_task; instead, pending_task holds a reference to the event until the task returns to user space. When a parent event with attr.inherit and attr.sigtrap is closed while a child event has pending_task queued, perf_event_release_kernel() detaches the child events from their cloned contexts without uncloning them:
CPU0 (task A) CPU1__perf_event_overflow(child_A)
task_work_add(A, &child_A->pending_task)
perf_event_release_kernel(parent)
perf_remove_from_context(child_A, ...)
list_del_event(child_A, ctx_A)
local_dec(&ctx_A->nr_no_switch_fast)
perf_remove_from_context(child_B, ...)
list_del_event(child_B, ctx_B)
local_dec(&ctx_B->nr_no_switch_fast)
schedule() (A -> B)
perf_event_context_sched_out(A, B)
/* context_equiv(ctx_A, ctx_B) && nr_no_switch_fast == 0 */
WRITE_ONCE(ctx_A->task, B)
...
perf_pending_task(&child_A->pending_task)
perf_sigtrap(child_A)
WARN_ON_ONCE(event->ctx->task != current) <-- trips WARN, drops SIGTRAPKeeping the increment in __perf_event_overflow() and decrement in perf_pending_task() alongside list_add_event() and list_del_event() would keep nr_no_switch_fast elevated until pending_task completes. Note that because list_add_event() and list_del_event() can run on a remote CPU under ctx->lock while __perf_event_overflow() or perf_pending_task() runs on ctx->task's CPU without ctx->lock, nr_no_switch_fast would also need to be atomic_t rather than CPU-local local_t.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v4 2/2] perf/core: Disable context fast-swap for sigtrap events
Quoted 87 lines · On 2026/11/17 19:53, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] perf/core:list_del_event()dropsnr_no_switch_fastbeforeperf_pending_task()
--commit
a5820a5e4b514a31e08f42157caca3b1bf8abed1
Author: Tengda Wu <[email protected]>perf/core: Disable context fast-swap for sigtrap events
This commit disables the context switch fast-swap path for contexts containing sigtrap events by accounting
attr->sigtrapinnr_no_switch_fastduringlist_add_event()andlist_del_event()instead of__perf_event_overflow()andperf_pending_task().Quoted 22 lines
kernel/events/core.c +2
-2@@ -1999,7 +1999,7 @@ list_add_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user++;if (event->attr.inherit_stat) ctx->nr_stat++; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_inc (&ctx->nr_no_switch_fast);if (event->state >PERF_EVENT_STATE_OFF )@@ -2225,7 +2225,7 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx) ctx->nr_user--;if (event->attr.inherit_stat) ctx->nr_stat--; -if (has_inherit_and_sample_read (&event->attr)) +if (has_inherit_and_sample_read (&event->attr) || event->attr.sigtrap)local_dec (&ctx->nr_no_switch_fast);list_del_rcu (&event->event_entry);[ ... ]
Quoted 9 lines
kernel/events/core.c +0
-1@@ -7758,7 +7763,6 @@ static void perf_pending_task(struct callback_head *head)if (smp_load_acquire (&event->pending_work)) {perf_sigtrap (event); -local_dec (&event->ctx->nr_no_switch_fast);/* * Ensure the pending work is fully handled before allowing * another overflow to queue it. See __perf_event_overflow(). kernel/events/core.c static int __perf_event_overflow(struct perf_event *event, · line 10944
Quoted 7 lines
kernel/events/core.c +0
-1@@ -10935,7 +10939,6 @@ static int __perf_event_overflow(struct perf_event *event,*/ if (cmpxchg_acquire (&event->pending_work,0 , pending_id) ==0 ) {if (!task_work_add (current, &event->pending_task, notify_mode)) { -local_inc (&event->ctx->nr_no_switch_fast);WARN_ON_ONCE (!atomic_long_inc_not_zero (&event->refcount)); event->pending_addr =0 ;[Severity: Medium]
Canctx->nr_no_switch_fastdrop to zero whileevent->pending_taskis still queued if the sigtrap event is detached beforeperf_pending_task()runs?Detaching an event does not cancel or flush
event->pending_task; instead,pending_taskholds a reference to the event until the task returns to user space. When a parent event with attr.inherit and attr.sigtrap is closed while a child event haspending_taskqueued,perf_event_release_kernel()detaches the child events from their cloned contexts without uncloning them:CPU0 (task A) CPU1
__perf_event_overflow(child_A)
task_work_add(A, &child_A->pending_task)perf_event_release_kernel(parent) perf_remove_from_context(child_A, ...) list_del_event(child_A, ctx_A) local_dec(&ctx_A->nr_no_switch_fast) perf_remove_from_context(child_B, ...) list_del_event(child_B, ctx_B) local_dec(&ctx_B->nr_no_switch_fast) schedule() (A -> B) perf_event_context_sched_out(A, B) /* context_equiv(ctx_A, ctx_B) && nr_no_switch_fast == 0 */ WRITE_ONCE(ctx_A->task, B) ... perf_pending_task(&child_A->pending_task) perf_sigtrap(child_A) WARN_ON_ONCE(event->ctx->task != current) <-- trips WARN, drops SIGTRAP
Quoted 3 lines
Keeping the increment in
__perf_event_overflow()and decrement inperf_pending_task()alongsidelist_add_event()andlist_del_event()would keepnr_no_switch_fastelevated untilpending_taskcompletes.
It does appear that such a scenario exists.
The nr_no_switch_fast refcount distinguishes two types of states:
1. Whether the SIGTRAP event still has a pending_task that has not yet been executed. (Pre-existing)
2. Whether the SIGTRAP event is still attached to the context. (Newly added)
Among them:
Type 1 covers the scenario where nr_no_switch_fast goes from 1 to 0 at the end. Type 2 covers the scenario where nr_no_switch_fast goes from 0 to 1 at the beginning.
Both have windows for taking the fast path, and an nr_no_switch_fast guard is needed. Deleting 1 in this patch does indeed seem incorrect.
Hi Sebastian, what do you think?
If there's no problem, I will roll back the deleted counter (i.e., the v3 version of this patch).
Thanks,
Tengda
Quoted 6 lines
Note that because
list_add_event()andlist_del_event()can run on a remote CPU underctx->lockwhile__perf_event_overflow()orperf_pending_task()runs on ctx->task's CPU withoutctx->lock,nr_no_switch_fastwould also need to beatomic_trather than CPU-locallocal_t.