Skip to content

perf/core: Fix software event sample data, then stop zeroing it

Usama Arif bpf linux-perf-users lkml stable 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest3 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3perf/core: Fix perf_sample_data reuse across software eventskernel/events/core.c+7 -100 replies
2/3perf/core: Don't let BPF programs read an uninitialized sample addresskernel/events/core.c+3 -00 replies
3/3perf/core: Don't zero perf_sample_data in do_perf_sw_event()kernel/events/core.c+1 -12 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it

do_perf_sw_event() builds samples for software events (context switches, page faults, ...) in an on-stack struct perf_sample_data. This series fixes two bugs around that struct and then stops zero-initializing it, a 256-byte memset that every context switch pays once a context-switch counter is open anywhere.

Patch 1 initializes the struct for each matching event. Today a sampling event inherits the sample ID, timestamp, pid/tid and callchain prepared for the event before it, and sometimes a record size smaller than what is then written, overrunning the records that follow in its ring buffer. Two concurrent "perf record -e page-faults,context-switches -c 1 -a" sessions make the older one fail. An unprivileged user's page-fault event can do this to root's CPU-wide session. Broken since v6.3.

Patch 2 stops BPF programs from reading an unset data->addr through bpf_perf_event_data::addr. On INIT_STACK_NONE kernels this returned leftover stack, mostly kernel pointers, on almost all context-switch and cpu-clock samples in my tests. INIT_STACK_ALL_ZERO mostly hides it. It affects all event types, needs CAP_BPF and CAP_PERFMON, and has been there since v6.1.

Patch 3 marks the struct __uninitialized. On every path from do_perf_sw_event(), each field is written before it is read, counting events never read the struct, and it is never copied as a whole, so its padding never leaves the kernel. A privileged BPF program using bpf_cast_to_kern_ctx() can still read unprepared fields, and would see stack contents instead of 0, as it already does on INIT_STACK_NONE kernels.

Patches 1-2 are for perf/urgent and stable, patch 3 for perf/core.

Performance: with sched_yield() ping-pong on one CPU, software counters add on average 54 ns to a ~627 ns context switch, and patch 3 brings that to 44 ns (10 rounds, spec_rstack_overflow=off).

Usama Arif (3):
perf/core: Fix perf_sample_data reuse across software events perf/core: Don't let BPF programs read an uninitialized sample address perf/core: Don't zero perf_sample_data in do_perf_sw_event()

 kernel/events/core.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
Signature · 1 line
2.53.1-Meta