perf/core: Strengthen userpage update ordering
PATCH 1/1 perf/core: Strengthen userpage update ordering
The perf event mmap userpage uses a sequence counter to let userspace obtain a consistent snapshot of its data. The existing compiler barriers reflect the original self-monitoring use case, where the counter was updated and consumed on the same CPU.
Some consumers also use the userpage only for time conversion and may read it from a CPU other than the one updating the event. For example, perf_read_tsc_conversion() reads the time conversion fields without constraining the caller to the event's CPU.
Make the publication ordering explicit for such readers by replacing the compiler barriers around the userpage payload update with smp_wmb(). Document that cross-CPU readers of the time conversion fields must use read memory barriers and reject odd or changed sequence values.
This does not change the UAPI layout or the values exposed to userspace. It strengthens the ordering guarantee for cross-CPU readers on weakly ordered architectures.
--- include/uapi/linux/perf_event.h | 6 ++++-- kernel/events/core.c | 6 ++++-- tools/include/uapi/linux/perf_event.h | 6 ++++-- tools/perf/design.txt | 6 ++++-- 4 files changed, 16 insertions(+), 8 deletions(-)
include/uapi/linux/perf_event.h +4 -2
@@ -629,8 +629,10 @@ struct perf_event_mmap_page {* barrier(); * } while (pc->lock != seq); * -* NOTE: for obvious reason this only works on self-monitoring -* processes. +* NOTE: Reading the hardware counter as shown above only works for +* self-monitoring processes. A reader on another CPU may snapshot +* the time conversion fields, but must use rmb() around the field +* reads and retry if lock is odd or changes. */ __u32 lock;/* seqlock for synchronization */ __u32 index;/* hardware event identifier */
kernel/events/core.c +4 -2
@@ -6852,7 +6852,8 @@ void perf_event_update_userpage(struct perf_event *event) userpg = rb->user_page; ++userpg->lock; -barrier (); +/* Publish the odd lock value before updating the payload. */ +smp_wmb (); userpg->index =perf_event_index (event); userpg->offset =perf_event_count (event, false);if (userpg->index)
@@ -6866,7 +6867,8 @@ void perf_event_update_userpage(struct perf_event *event)arch_perf_update_userpage (event, userpg, now); -barrier (); +/* Publish the payload before the final lock update. */ +smp_wmb (); ++userpg->lock;preempt_enable (); unlock:
tools/include/uapi/linux/perf_event.h +4 -2
@@ -629,8 +629,10 @@ struct perf_event_mmap_page {* barrier(); * } while (pc->lock != seq); * -* NOTE: for obvious reason this only works on self-monitoring -* processes. +* NOTE: Reading the hardware counter as shown above only works for +* self-monitoring processes. A reader on another CPU may snapshot +* the time conversion fields, but must use rmb() around the field +* reads and retry if lock is odd or changes. */ __u32 lock;/* seqlock for synchronization */ __u32 index;/* hardware event identifier */
tools/perf/design.txt +4 -2
@@ -316,8 +316,10 @@ struct perf_event_mmap_page { * barrier(); * } while (pc->lock != seq); * - * NOTE: for obvious reason this only works on self-monitoring - * processes. + * NOTE: Reading the hardware counter as shown above only works for + * self-monitoring processes. A reader on another CPU may snapshot + * the time conversion fields, but must use rmb() around the field + * reads and retry if lock is odd or changes. */ __u32 lock; /* seqlock for synchronization */ __u32 index; /* hardware counter identifier */
Signature · 1 line
2.55.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Preexisting seqlock writer re-entrancy bug in perf_event_update_userpage leading to corrupted data for userspace readers. --
commit 019372a4e61c18f0c5a22ddb487c7acef14ffc7e
Author: Thaumy Cheng <[email protected]>
perf/core: Strengthen userpage update ordering
This commit replaces compiler barriers with smp_wmb() around the userpage payload update in perf_event_update_userpage(). It aims to strengthen the publication ordering for cross-CPU readers of time conversion fields on weakly ordered architectures.
kernel/events/core.c void perf_event_update_userpage(struct perf_event *event) · line 6862
Quoted 11 lines
kernel/events/core.c +2
-1@@ -6858,7 +6858,8 @@ void perf_event_update_userpage(struct perf_event *event) userpg = rb->user_page; ++userpg->lock; -barrier (); +/* Publish the odd lock value before updating the payload. */ +smp_wmb ();
[Severity: High]
This is a pre-existing issue, but does perf_event_update_userpage() have a seqlock writer re-entrancy bug here that defeats the synchronization?
Since this function disables preemption (via preempt_disable() earlier in the function) but leaves hardware interrupts enabled when called from task context (such as from perf_mmap_rb()), what happens if a PMU hardware interrupt like an NMI triggers during this window?
If the NMI handler also calls perf_event_update_userpage() for the same event, it would increment userpg->lock again, advancing it to an even value. A concurrent userspace reader on another CPU would then see an even lock value and could read actively-torn payload data (like time_mult, time_shift, or time_offset), leading to bogus timestamps.
Should hardware interrupts be disabled before incrementing the seqlock to prevent this writer re-entrancy?
Quoted 2 lines
kernel/events/core.c +0
-0userpg->index =perf_event_index (event); userpg->offset =perf_event_count (event, false);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1