Revert "tracing: perf: Fix stale head for perf syscall tracing"
Subsystems (3)
PATCH 1/1 Revert "tracing: perf: Fix stale head for perf syscall tracing"
This reverts commit be94a3a77e7eb99c53418de7d4a01bc7eb3f634e.
The upstream commit was applied after commit 57918341dd19 ("bpf: Add sleepable support for classic tracepoint programs"), which moved BPF program execution before the per-CPU perf event list is checked.
The 7.1.y implementation has not undergone that restructuring. It handles BPF and perf consumers together, and already reloads the current CPU's perf event list after syscall_get_data().
As a result, the backport adds an unconditional hlist_empty() return before bpf_prog_array_valid() is consulted. A syscall trace event can have a valid BPF program array while the current CPU's perf event list is empty. In that case the early return prevents the BPF program from running.
This breaks BPF consumers of faultable syscall-enter tracepoints such as sys_enter_execve. One observed consequence is that Proton VPN's app-based split tunneling no longer receives exec argument events and therefore cannot match newly started processes.
Revert the stable backport. The existing head assignment below the removed block preserves its intended stale-head protection while retaining the previous BPF-aware handling.
--- kernel/trace/trace_syscalls.c | 5 ----- 1 file changed, 5 deletions(-)
kernel/trace/trace_syscalls.c +0 -5
@@ -1436,11 +1436,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)if (syscall_get_data (sys_data, args, &user_ptr, &size, user_sizes, &uargs, buf_size) <0 )return ; - -/* The above may have caused a migration */ - head =this_cpu_ptr (sys_data->enter_event->perf_events); -if (hlist_empty (head)) -return ; } head =this_cpu_ptr (sys_data->enter_event->perf_events);
Signature · 1 line
2.51.0
Steven Rostedt
maintainerQuoted 15 lines · On Sun, 16 Aug 2026 15:31:01 +0200 Peter Jung <[email protected]> wrote:
This reverts commit
be94a3a77e7eb99c53418de7d4a01bc7eb3f634e.The upstream commit was applied after commit
57918341dd19("bpf: Add sleepable support for classic tracepoint programs"), which moved BPF program execution before the per-CPU perf event list is checked.The 7.1.y implementation has not undergone that restructuring. It handles BPF and perf consumers together, and already reloads the current CPU's perf event list after
syscall_get_data().As a result, the backport adds an unconditional
hlist_empty()return beforebpf_prog_array_valid()is consulted. A syscall trace event can have a valid BPF program array while the current CPU's perf event list is empty. In that case the early return prevents the BPF program from running.
Quoted 8 lines
This breaks BPF consumers of faultable syscall-enter tracepoints such as
sys_enter_execve. One observed consequence is that Proton VPN's app-based split tunneling no longer receives exec argument events and therefore cannot match newly started processes.Revert the stable backport. The existing head assignment below the removed block preserves its intended stale-head protection while retaining the previous BPF-aware handling.
OK, so the Fixes on the original commit be94a3a77e7e should have had:
Instead?
-- Steve
Quoted 25 lines
Fixes be94a3a77e7e ("tracing: perf: Fix stale head for perf syscall tracing")Assisted-by Codex:gpt-5.6-solSigned-off-by Peter Jung <[email protected]>--- kernel/trace/trace_syscalls.c | 5 ----- 1 file changed, 5 deletions(-)kernel/trace/trace_syscalls.c +0
-5@@ -1436,11 +1436,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)if (syscall_get_data (sys_data, args, &user_ptr, &size, user_sizes, &uargs, buf_size) <0 )return ; - -/* The above may have caused a migration */ - head =this_cpu_ptr (sys_data->enter_event->perf_events); -if (hlist_empty (head)) -return ; } head =this_cpu_ptr (sys_data->enter_event->perf_events);
Quoted 27 lines · Steven Rostedt
Quoted 15 lines · On Sun, 16 Aug 2026 15:31:01 +0200 Peter Jung<[email protected]> wrote:
This reverts commit
be94a3a77e7eb99c53418de7d4a01bc7eb3f634e.The upstream commit was applied after commit
57918341dd19("bpf: Add sleepable support for classic tracepoint programs"), which moved BPF program execution before the per-CPU perf event list is checked.The 7.1.y implementation has not undergone that restructuring. It handles BPF and perf consumers together, and already reloads the current CPU's perf event list after
syscall_get_data().As a result, the backport adds an unconditional
hlist_empty()return beforebpf_prog_array_valid()is consulted. A syscall trace event can have a valid BPF program array while the current CPU's perf event list is empty. In that case the early return prevents the BPF program from running.Quoted 8 lines
This breaks BPF consumers of faultable syscall-enter tracepoints such as
sys_enter_execve. One observed consequence is that Proton VPN's app-based split tunneling no longer receives exec argument events and therefore cannot match newly started processes.Revert the stable backport. The existing head assignment below the removed block preserves its intended stale-head protection while retaining the previous BPF-aware handling.
Quoted 8 lines
OK, so the Fixes on the original commit
be94a3a77e7eshould have had:Fixes 57918341dd19 ("bpf: Add sleepable support for classic tracepoint programs")Instead?
-- Steve
Yes, this should fix it too, but has been not verified yet. I will run a test tomorrow and let you know.
Best,
Peter
Quoted 26 lines
Quoted 25 lines
Fixes be94a3a77e7e ("tracing: perf: Fix stale head for perf syscall tracing")Assisted-by Codex:gpt-5.6-solSigned-off-by Peter Jung<[email protected]>--- kernel/trace/trace_syscalls.c | 5 ----- 1 file changed, 5 deletions(-)kernel/trace/trace_syscalls.c +0
-5@@ -1436,11 +1436,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)if (syscall_get_data (sys_data, args, &user_ptr, &size, user_sizes, &uargs, buf_size) <0 )return ; - -/* The above may have caused a migration */ - head =this_cpu_ptr (sys_data->enter_event->perf_events); -if (hlist_empty (head)) -return ; } head =this_cpu_ptr (sys_data->enter_event->perf_events);
Sasha Levin
maintainerQuoted 3 lines
The 7.1.y implementation has not undergone that restructuring. It handles BPF and perf consumers together, and already reloads the current CPU's perf event list after
syscall_get_data().
The analysis checks out on 7.1.y: 57918341dd19 ("bpf: Add sleepable support for classic tracepoint programs") is not there, so the added hlist_empty() early return runs before bpf_prog_array_valid() is ever consulted and a BPF-only consumer stops firing.
Since this reverts a commit that already shipped in 7.1.6, I would like your test result before queueing it - you mentioned you would have one today. Ping the thread once you do and I will pick it up.
Signature · 2 lines
Thanks, Sasha
Alban Crequy
Tested-byHi Sasha, Peter,
Another data point: Ubuntu picked up the same backport in its 7.0 kernel (7.0.0-38-generic), and it breaks Inspektor Gadget and bpftrace in the same way. A BPF program on sys_enter_execve or sys_enter_openat only runs on CPU 0; sys_exit_* and sys_enter_close keep working on all CPUs. Reproducer:
bpftrace -e 'tracepoint:syscalls:sys_enter_openat /comm == "cat"/ { @[cpu] = count(); }' &
for c in $(seq 0 7); do taskset -c $c cat /etc/hostname >/dev/null; doneOnly @[0] shows up.
I also tested v7.1.13 under QEMU with 4 CPUs, using a small cilium/ebpf test that counts per-CPU hits of a tracepoint program while running 10 syscalls pinned to each CPU:
v7.1.13: sys_enter_openat [10 0 0 0], sys_enter_close [10 10 10 10], sys_exit_openat [10 10 10 10]
v7.1.13 + revert: sys_enter_openat [10 10 10 10], sys_enter_close [10 10 10 10], sys_exit_openat [10 10 10 10]
The revert looks right to me for 7.1.y, and for any other stable branch without 57918341dd19.
Alban