arm64: ftrace: enable single ftrace_ops for direct calls
cover arm64: ftrace: enable single ftrace_ops for direct calls
Beyond the DIRECT_CALLS support on arm64 [1], select HAVE_SINGLE_FTRACE_DIRECT_OPS for arm64, which enables the BPF tracing_multi link on arm64 [2].
All tracing_multi link selftests are passed on arm64.
cd tools/testing/selftests/bpf
./test_progs -a 'tracing_multi_*'
#545/1 tracing_multi_attach_rollback/put:OK #545/2 tracing_multi_attach_rollback/unlink:OK #545 tracing_multi_attach_rollback:OK #546 tracing_multi_bench_attach:OK #547/1 tracing_multi_test/skel_api:OK #547/2 tracing_multi_test/link_api_pattern:OK #547/3 tracing_multi_test/link_api_ids:OK #547/4 tracing_multi_test/module_skel_api:OK #547/5 tracing_multi_test/module_link_api_pattern:OK #547/6 tracing_multi_test/module_link_api_ids:OK #547/7 tracing_multi_test/intersect:OK #547/8 tracing_multi_test/cookies:OK #547/9 tracing_multi_test/session:OK #547/10 tracing_multi_test/attach_api_fails:OK #547/11 tracing_multi_test/fentry_direct_access:OK #547/12 tracing_multi_test/fexit_direct_access:OK #547/13 tracing_multi_test/fsession_direct_access:OK #547/14 tracing_multi_test/fentry_after_multi:OK #547 tracing_multi_test:OK
Summary: 3/16 PASSED, 0 SKIPPED, 0 FAILED
./test_progs -a 'fill_link_info/tracing_multi_*'
#132/11 fill_link_info/tracing_multi_link_info:OK #132/12 fill_link_info/tracing_multi_invalid_ubuff:OK #132 fill_link_info:OK
Summary: 1/2 PASSED, 0 SKIPPED, 0 FAILED
Links:
[1] https://lore.kernel.org/all/[email protected]/
[2] https://github.com/kernel-patches/bpf/pull/12986
Leon Hwang (2):
arm64: ftrace: enable single ftrace_ops for direct calls selftests/bpf: Enable tracing_multi tests on arm64
arch/arm64/Kconfig | 2 ++ arch/arm64/kernel/ftrace.c | 3 +-- tools/testing/selftests/bpf/prog_tests/tracing_multi.c | 9 +++++++-- tools/testing/selftests/bpf/test_progs.h | 6 ++++++ 4 files changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0
PATCH 1/2 arm64: ftrace: enable single ftrace_ops for direct calls
The BPF tracing multi link updates several direct-call sites through one ftrace_ops. Its implementation is therefore gated by HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not use ops->direct_call.
--- arch/arm64/Kconfig | 2 ++ arch/arm64/kernel/ftrace.c | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/arm64/Kconfig +2 -0
@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \ (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
arch/arm64/kernel/ftrace.c +1 -2
@@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,/* * If a custom trampoline is unreachable, rely on the ftrace_caller -* trampoline which knows how to indirectly reach that trampoline -* through ops->direct_call. +* trampoline which knows how to indirectly reach that trampoline. */ if (*addr !=FTRACE_ADDR && !reachable_by_bl (*addr, pc)) *addr =FTRACE_ADDR ;
Signature · 1 line
2.55.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing RCU read barrier (rcu_dereference()) when accessing direct_functions. --
--- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls --- commit 738274783fa707248af104af5e45da296f7e4d84
Author: Leon Hwang <[email protected]>
arm64: ftrace: enable single ftrace_ops for direct calls
The BPF tracing multi link updates several direct-call sites through one ftrace_ops. This commit selects HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64 when DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled. It also generalizes the unreachable-trampoline comment.
arch/arm64/Kconfig config ARM64 · line 192
Quoted 10 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS
[Severity: High]
This is a pre-existing issue, but by enabling HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64, does this expose the architecture to a missing RCU read barrier in the ftrace code?
When HAVE_SINGLE_FTRACE_DIRECT_OPS is selected, call_direct_funcs() switches from using READ_ONCE(ops->direct_call) to using ftrace_find_rec_direct():
kernel/trace/ftrace.c:call_direct_funcs() {
...
#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS addr = ftrace_find_rec_direct(ip);
#else
...
}
Looking at ftrace_find_rec_direct(), it accesses the __rcu pointer direct_functions directly without an rcu_dereference():
kernel/trace/ftrace.c:ftrace_find_rec_direct() {
struct ftrace_func_entry *entry;
entry = __ftrace_lookup_ip(direct_functions, ip);...
}
Without the READ_ONCE() provided by rcu_dereference(), the compiler might reload the pointer between computing the hash key and accessing the bucket array inside __ftrace_lookup_ip().
If a concurrent writer substitutes a new hash table with a different size between these reloads, could the computed key exceed the new bucket array bounds and lead to an out-of-bounds dereference?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 54 lines · On 27/7/26 22:52, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing RCU read barrier (rcu_dereference()) when accessingdirect_functions. ----- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single
ftrace_opsfor direct calls --- commit738274783fa707248af104af5e45da296f7e4d84
Author: Leon Hwang <[email protected]>arm64: ftrace: enable single
ftrace_opsfor direct callsThe BPF tracing multi link updates several direct-call sites through one
ftrace_ops. This commit selectsHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64 whenDYNAMIC_FTRACE_WITH_DIRECT_CALLSis enabled. It also generalizes the unreachable-trampoline comment.arch/arm64/Kconfig config ARM64 · line 192
Quoted 10 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS[Severity: High]
This is a pre-existing issue, but by enablingHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64, does this expose the architecture to a missing RCU read barrier in the ftrace code?When
HAVE_SINGLE_FTRACE_DIRECT_OPSis selected,call_direct_funcs()switches from usingREAD_ONCE(ops->direct_call)to usingftrace_find_rec_direct():kernel/trace/ftrace.c:call_direct_funcs() {
...#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS addr = ftrace_find_rec_direct(ip);#else
...
}Looking at
ftrace_find_rec_direct(), it accesses the__rcupointerdirect_functionsdirectly without anrcu_dereference():kernel/trace/ftrace.c:ftrace_find_rec_direct() { struct ftrace_func_entry *entry; entry = __ftrace_lookup_ip(direct_functions, ip);...
}
Quoted 8 lines
Without the
READ_ONCE()provided byrcu_dereference(), the compiler might reload the pointer between computing the hash key and accessing the bucket array inside__ftrace_lookup_ip().If a concurrent writer substitutes a new hash table with a different size between these reloads, could the computed key exceed the new bucket array bounds and lead to an out-of-bounds dereference?
Hi Steve, Masami,
Could you take a look at this issue? The issue looks legitimate.
Thanks,
Leon
Steven Rostedt
maintainerQuoted 65 lines · On Tue, 28 Jul 2026 10:19:22 +0800 Leon Hwang <[email protected]> wrote:
Quoted 54 lines · On 27/7/26 22:52, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing RCU read barrier (rcu_dereference()) when accessingdirect_functions. ----- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single
ftrace_opsfor direct calls --- commit738274783fa707248af104af5e45da296f7e4d84
Author: Leon Hwang <[email protected]>arm64: ftrace: enable single
ftrace_opsfor direct callsThe BPF tracing multi link updates several direct-call sites through one
ftrace_ops. This commit selectsHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64 whenDYNAMIC_FTRACE_WITH_DIRECT_CALLSis enabled. It also generalizes the unreachable-trampoline comment.arch/arm64/Kconfig config ARM64 · line 192
Quoted 10 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS[Severity: High]
This is a pre-existing issue, but by enablingHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64, does this expose the architecture to a missing RCU read barrier in the ftrace code?When
HAVE_SINGLE_FTRACE_DIRECT_OPSis selected,call_direct_funcs()switches from usingREAD_ONCE(ops->direct_call)to usingftrace_find_rec_direct():kernel/trace/ftrace.c:call_direct_funcs() {
...#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS addr = ftrace_find_rec_direct(ip);#else
...
}Looking at
ftrace_find_rec_direct(), it accesses the__rcupointerdirect_functionsdirectly without anrcu_dereference():kernel/trace/ftrace.c:ftrace_find_rec_direct() { struct ftrace_func_entry *entry; entry = __ftrace_lookup_ip(direct_functions, ip);...
}Quoted 8 lines
Without the
READ_ONCE()provided byrcu_dereference(), the compiler might reload the pointer between computing the hash key and accessing the bucket array inside__ftrace_lookup_ip().If a concurrent writer substitutes a new hash table with a different size between these reloads, could the computed key exceed the new bucket array bounds and lead to an out-of-bounds dereference?
Quoted 3 lines
Hi Steve, Masami,
Could you take a look at this issue? The issue looks legitimate.
Yeah, it's possible (but unlikely) for the compiler to do that. But there's nothing harming the code to do:
( Untested and not even compiled )
kernel/trace/ftrace.c +2 -1
@@ -2639,7 +2639,8 @@ unsigned long ftrace_find_rec_direct(unsigned long ip) {struct ftrace_func_entry *entry; - entry =__ftrace_lookup_ip (direct_functions, ip); +guard (preempt_notrace)(); + entry =__ftrace_lookup_ip (rcu_dereference_sched (direct_functions), ip);if (!entry)return 0 ;
-- Steve
Quoted 68 lines · Steven Rostedt
Quoted 65 lines · On Tue, 28 Jul 2026 10:19:22 +0800 Leon Hwang <[email protected]> wrote:
Quoted 62 lines · On 27/7/26 22:52, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing RCU read barrier (rcu_dereference()) when accessingdirect_functions. ----- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single
ftrace_opsfor direct calls --- commit738274783fa707248af104af5e45da296f7e4d84
Author: Leon Hwang <[email protected]>arm64: ftrace: enable single
ftrace_opsfor direct callsThe BPF tracing multi link updates several direct-call sites through one
ftrace_ops. This commit selectsHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64 whenDYNAMIC_FTRACE_WITH_DIRECT_CALLSis enabled. It also generalizes the unreachable-trampoline comment.arch/arm64/Kconfig config ARM64 · line 192
Quoted 10 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS[Severity: High]
This is a pre-existing issue, but by enablingHAVE_SINGLE_FTRACE_DIRECT_OPSon arm64, does this expose the architecture to a missing RCU read barrier in the ftrace code?When
HAVE_SINGLE_FTRACE_DIRECT_OPSis selected,call_direct_funcs()switches from usingREAD_ONCE(ops->direct_call)to usingftrace_find_rec_direct():kernel/trace/ftrace.c:call_direct_funcs() {
...#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS addr = ftrace_find_rec_direct(ip);#else
...
}Looking at
ftrace_find_rec_direct(), it accesses the__rcupointerdirect_functionsdirectly without anrcu_dereference():kernel/trace/ftrace.c:ftrace_find_rec_direct() { struct ftrace_func_entry *entry; entry = __ftrace_lookup_ip(direct_functions, ip);...
}Without the
READ_ONCE()provided byrcu_dereference(), the compiler might reload the pointer between computing the hash key and accessing the bucket array inside__ftrace_lookup_ip().If a concurrent writer substitutes a new hash table with a different size between these reloads, could the computed key exceed the new bucket array bounds and lead to an out-of-bounds dereference?
Quoted 6 lines
Quoted 3 lines
Hi Steve, Masami,
Could you take a look at this issue? The issue looks legitimate.
Yeah, it's possible (but unlikely) for the compiler to do that. But there's nothing harming the code to do:
Good to know it's unlikely in practice.
If you don't mind, I'll send a separate patch for the fix based on the diff below. It compiles and works fine in my testing.
Thanks,
Leon
Quoted 20 lines
( Untested and not even compiled )
kernel/trace/ftrace.c +2
-2@@ -2639,7 +2639,8 @@ unsigned long ftrace_find_rec_direct(unsigned long ip) {struct ftrace_func_entry *entry; - entry =__ftrace_lookup_ip (direct_functions, ip); +guard (preempt_notrace)(); + entry =__ftrace_lookup_ip (rcu_dereference_sched (direct_functions), ip);if (!entry)return 0 ; -- Steve
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.
hi,
iirc arm needs ops->direct_call for trampolines to work properly, that's the reason we enabled single ftrace_ops on x86 only:
424f6a361096 bpf,x86: Use single ftrace_ops for direct calls
I don't understand the arm usage of op->direct_call, but it looks like (codex thinks) that the fast path won't always work without op->direct_call being set
from arch/arm64/kernel/entry-ftrace.S:
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * If the op has a direct call, handle it immediately without * saving/restoring registers. */ ldr x17, [x11, #FTRACE_OPS_DIRECT_CALL] // op->direct_call cbnz x17, ftrace_caller_direct #endif #endif
jirka
Quoted 15 lines · Jiri Olsa
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.hi,
iirc arm needsops->direct_callfor trampolines to work properly, that's the reason we enabled singleftrace_opson x86 only:
Quoted 5 lines
424f6a361096bpf,x86: Use singleftrace_opsfor direct callsI don't understand the arm usage of
op->direct_call, but it looks like (codex thinks) that the fast path won't always work withoutop->direct_callbeing set
In theory, op->direct_call is the fallback for the case:
HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF direct ops. This is what Codex thinks about.
However, after searching the call sites of register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would set ops->direct_call, the actual callers are in bpf/trampoline.c, which have been gated with HAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y, op->direct_call is unused.
If someone uses register_ftrace_direct() or modify_ftrace_direct/_nolock() outside BPF in the future, op->direct_call will be used.
So, I'd like to drop the comment change in the next revision.
Thanks,
Leon
Quoted 14 lines
from
arch/arm64/kernel/entry-ftrace.S:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * If the op has a direct call, handle it immediately without * saving/restoring registers. */ ldr x17, [x11, #FTRACE_OPS_DIRECT_CALL] // op->direct_call cbnz x17, ftrace_caller_direct #endif #endifjirka
Quoted 33 lines · Leon Hwang
Quoted 15 lines · Jiri Olsa
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.hi,
iirc arm needsops->direct_callfor trampolines to work properly, that's the reason we enabled singleftrace_opson x86 only:Quoted 5 lines
424f6a361096bpf,x86: Use singleftrace_opsfor direct callsI don't understand the arm usage of
op->direct_call, but it looks like (codex thinks) that the fast path won't always work withoutop->direct_callbeing setIn theory,
op->direct_callis the fallback for the case:
HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF direct ops. This is what Codex thinks about.However, after searching the call sites of register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would set
ops->direct_call, the actual callers are inbpf/trampoline.c, which have been gated withHAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y,op->direct_callis unused.
Quoted 3 lines
If someone uses
register_ftrace_direct()or modify_ftrace_direct/_nolock() outside BPF in the future,op->direct_callwill be used.
so any 2nd user on the same function triggers the slow path, and that's fine, the single_ops handles slow path correctly
but IIUC the out-of-range target will now always go through slow path, which in current code it doesn't, right?
jirka
Quoted 38 lines · Jiri Olsa
Quoted 33 lines · Leon Hwang
Quoted 15 lines · Jiri Olsa
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.hi,
iirc arm needsops->direct_callfor trampolines to work properly, that's the reason we enabled singleftrace_opson x86 only:Quoted 5 lines
424f6a361096bpf,x86: Use singleftrace_opsfor direct callsI don't understand the arm usage of
op->direct_call, but it looks like (codex thinks) that the fast path won't always work withoutop->direct_callbeing setIn theory,
op->direct_callis the fallback for the case:
HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF direct ops. This is what Codex thinks about.However, after searching the call sites of register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would set
ops->direct_call, the actual callers are inbpf/trampoline.c, which have been gated withHAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y,op->direct_callis unused.Quoted 3 lines
If someone uses
register_ftrace_direct()or modify_ftrace_direct/_nolock() outside BPF in the future,op->direct_callwill be used.
Quoted 6 lines
so any 2nd user on the same function triggers the slow path, and that's fine, the
single_opshandles slow path correctlybut IIUC the out-of-range target will now always go through slow path, which in current code it doesn't, right?
Right.
Pls see the sibling thread with Ihor. With in-range BPF trampoline allocation, the slow-path regression should mostly disappear.
Let's continue the discussion in that thread.
Thanks,
Leon
Quoted 46 lines · Leon Hwang
Quoted 38 lines · Jiri Olsa
Quoted 33 lines · Leon Hwang
Quoted 15 lines · Jiri Olsa
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.hi,
iirc arm needsops->direct_callfor trampolines to work properly, that's the reason we enabled singleftrace_opson x86 only:Quoted 5 lines
424f6a361096bpf,x86: Use singleftrace_opsfor direct callsI don't understand the arm usage of
op->direct_call, but it looks like (codex thinks) that the fast path won't always work withoutop->direct_callbeing setIn theory,
op->direct_callis the fallback for the case:
HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF direct ops. This is what Codex thinks about.However, after searching the call sites of register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would set
ops->direct_call, the actual callers are inbpf/trampoline.c, which have been gated withHAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y,op->direct_callis unused.Quoted 3 lines
If someone uses
register_ftrace_direct()or modify_ftrace_direct/_nolock() outside BPF in the future,op->direct_callwill be used.Quoted 6 lines
so any 2nd user on the same function triggers the slow path, and that's fine, the
single_opshandles slow path correctlybut IIUC the out-of-range target will now always go through slow path, which in current code it doesn't, right?
Quoted 6 lines
Right.
Pls see the sibling thread with Ihor. With in-range BPF trampoline allocation, the slow-path regression should mostly disappear.
Let's continue the discussion in that thread.
please paste the link/subject for that thread
thanks,
jirka
Quoted 2 lines · Jiri Olsa
Quoted 1 line · Leon Hwang
On 30/7/26 18:58, Jiri Olsa wrote:
[...]
Quoted 10 lines
Quoted 7 lines
Right.
Pls see the sibling thread with Ihor. With in-range BPF trampoline allocation, the slow-path regression should mostly disappear.
Let's continue the discussion in that thread.
please paste the link/subject for that thread
The link is
https://lore.kernel.org/all/[email protected]/.
Thanks,
Leon
Quoted 12 lines · Leon Hwang
Quoted 2 lines · Jiri Olsa
Quoted 1 line · Leon Hwang
On 30/7/26 18:58, Jiri Olsa wrote:
[...]
Quoted 6 lines
Quoted 6 lines
Right.
Pls see the sibling thread with Ihor. With in-range BPF trampoline allocation, the slow-path regression should mostly disappear.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Let's continue the discussion in that thread.
please paste the link/subject for that thread
The link is
https://lore.kernel.org/all/[email protected]/.
ugh I see, somehow I did not receive Ihor's email.. :\
thanks,
jirka
Ihor Solodrai
reviewerQuoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.
Hi Leon,
I don't think this change can land as is yet. The series doesn't even apply cleanly to bpf-next, but that's minor.
More importantly, it depends on Jose's series [1], which is not in the mainline yet. And there Mark has raised performance concerns [2] and the discussion still seems to be open.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/amjnf5gz0xP5PTSB@J2N7QTR9R3/
Quoted 13 lines
Assisted-by Codex:gpt-5.6-solSigned-off-by Leon Hwang <[email protected]>--- arch/arm64/Kconfig | 2 ++ arch/arm64/kernel/ftrace.c | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)arch/arm64/Kconfig +0
-0
arch/arm64/Kconfig config ARM64 · line 192
Quoted 5 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS\
The select is only conditional on DYNAMIC_FTRACE_WITH_DIRECT_CALLS, so it can be set along with HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS. And AFAIU this would make the fast path effectively dead: every BPF direct call routed through ftrace_caller now goes through the slow path.
As Mark noted in the other thread, on arm64 trampolines come from EXECMEM_BPF, so they always land out of BL range (chance of landing in range is 256M/terabytes).
I vibe-slop-coded a benchmark and ran it on a Neoverse V2 machine, and toggling your config change seems to be causing a 1.3x regression in the tracing overhead:
do-nothing fentry (r0=0; exit) on __arm64_sys_getpid, 20 M calls, min-of-N, several boots. Results:
untraced (base) traced overhead Kernel A ~128 ns/call ~145.5 ns ~17.6 ns (fast path: br x17) Kernel B ~127 ns/call ~150.4 ns ~23.4 ns (slow path: save regs + call_direct_funcs + hash)
This confirms Jiri's suspicion.
However my understanding is the regression should mostly disappear in case some version of in-range trampoline allocation on arm64 lands.
So, I think the landing sequence should be something like follows:
* in-range BPF-trampoline allocation that Jose proposed [3]
* then HAVE_SINGLE_FTRACE_DIRECT_OPS selection
After all of that reaches mainline, then a selftest patch can go through the bpf-next.
Quoted 17 lines
arch/arm64/Kconfig +0
-0select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \ (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))arch/arm64/kernel/ftrace.c +1
-2@@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,/* * If a custom trampoline is unreachable, rely on the ftrace_caller -* trampoline which knows how to indirectly reach that trampoline -* through ops->direct_call. +* trampoline which knows how to indirectly reach that trampoline. */ if (*addr !=FTRACE_ADDR && !reachable_by_bl (*addr, pc)) *addr =FTRACE_ADDR ;
Quoted 55 lines · Ihor Solodrai
Quoted 9 lines · Leon Hwang
The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated byHAVE_SINGLE_FTRACE_DIRECT_OPSin addition toDYNAMIC_FTRACE_WITH_DIRECT_CALLS.Select
HAVE_SINGLE_FTRACE_DIRECT_OPSwhenever arm64 enables dynamic ftrace direct calls. This enables BPF tracing multi links on arm64. Also generalize the unreachable-trampoline comment because the single-ops path does not useops->direct_call.Hi Leon,
I don't think this change can land as is yet. The series doesn't even apply cleanly to bpf-next, but that's minor.
More importantly, it depends on Jose's series [1], which is not in the mainline yet. And there Mark has raised performance concerns [2] and the discussion still seems to be open.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/amjnf5gz0xP5PTSB@J2N7QTR9R3/Quoted 13 lines
Assisted-by Codex:gpt-5.6-solSigned-off-by Leon Hwang <[email protected]>--- arch/arm64/Kconfig | 2 ++ arch/arm64/kernel/ftrace.c | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)arch/arm64/Kconfig +0
-0arch/arm64/Kconfig config ARM64 · line 192
Quoted 5 lines
arch/arm64/Kconfig +2
-0@@ -188,6 +188,8 @@ config ARM64 CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SINGLE_FTRACE_DIRECT_OPS \ + if DYNAMIC_FTRACE_WITH_DIRECT_CALLS\The select is only conditional on
DYNAMIC_FTRACE_WITH_DIRECT_CALLS, so it can be set along withHAVE_DYNAMIC_FTRACE_WITH_CALL_OPS. And AFAIU this would make the fast path effectively dead: every BPF direct call routed throughftrace_callernow goes through the slow path.As Mark noted in the other thread, on arm64 trampolines come from
EXECMEM_BPF, so they always land out of BL range (chance of landing in range is 256M/terabytes).I vibe-slop-coded a benchmark and ran it on a Neoverse V2 machine, and toggling your config change seems to be causing a 1.3x regression in the tracing overhead:
Quoted 5 lines
do-nothing fentry (r0=0; exit) on
__arm64_sys_getpid, 20 M calls, min-of-N, several boots. Results:untraced (base) traced overhead Kernel A ~128 ns/call ~145.5 ns ~17.6 ns (fast path: br x17) Kernel B ~127 ns/call ~150.4 ns ~23.4 ns (slow path: save regs + call_direct_funcs + hash)
Thanks for your testing.
Quoted 2 lines
This confirms Jiri's suspicion.
True.
Quoted 10 lines
However my understanding is the regression should mostly disappear in case some version of in-range trampoline allocation on arm64 lands.
So, I think the landing sequence should be something like follows:
* in-range BPF-trampoline allocation that Jose proposed [3]
* thenHAVE_SINGLE_FTRACE_DIRECT_OPSselectionAfter all of that reaches mainline, then a selftest patch can go through the bpf-next.
Sounds reasonable.
This series is based on Jose's series and is intended for the arm64 tree, rather than bpf-next. Like Jose's series, this series aims to enable BPF tracing_multi link on arm64. And yes, with in-range BPF-trampoline allocation, the regression should mostly disappear.
I'll follow the sequence and repost the patches afterward.
Thanks,
Leon
Quoted 21 lines
[3] https://lore.kernel.org/all/[email protected]/
Quoted 17 lines
arch/arm64/Kconfig +0
-0select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \ (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))arch/arm64/kernel/ftrace.c +1
-2@@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,/* * If a custom trampoline is unreachable, rely on the ftrace_caller -* trampoline which knows how to indirectly reach that trampoline -* through ops->direct_call. +* trampoline which knows how to indirectly reach that trampoline. */ if (*addr !=FTRACE_ADDR && !reachable_by_bl (*addr, pc)) *addr =FTRACE_ADDR ;
PATCH 2/2 selftests/bpf: Enable tracing_multi tests on arm64
In reply to Leon Hwangarm64 now supports dynamic ftrace direct calls backed by a single ftrace_ops, which enables BPF tracing multi links.
Define ARCH_SUPPORTS_TRACING_MULTI_LINK for x86-64 and arm64, and use the shared capability to gate the tracing_multi functional, rollback, and attach benchmark tests.
This allows the tests to run on arm64 while continuing to skip architectures without tracing multi link support. It also prevents the rollback test from treating an unsupported-link error as its expected attachment failure.
--- tools/testing/selftests/bpf/prog_tests/tracing_multi.c | 9 +++++++-- tools/testing/selftests/bpf/test_progs.h | 6 ++++++ 2 files changed, 13 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/tracing_multi.c +7 -2
@@ -612,7 +612,7 @@ void serial_test_tracing_multi_bench_attach(void)struct btf *btf;int err; -#ifndef __x86_64__ +#if !ARCH_SUPPORTS_TRACING_MULTI_LINK test__skip ();return ;#endif
@@ -923,6 +923,11 @@ static void test_rollback_unlink(void)void serial_test_tracing_multi_attach_rollback (void ) { +#if !ARCH_SUPPORTS_TRACING_MULTI_LINK +test__skip (); +return ; +#endif +if (test__start_subtest ("put" ))test_rollback_put ();if (test__start_subtest ("unlink" ))
@@ -931,7 +936,7 @@ void serial_test_tracing_multi_attach_rollback(void)void test_tracing_multi_test (void ) { -#ifndef __x86_64__ +#if !ARCH_SUPPORTS_TRACING_MULTI_LINK test__skip ();return ;#endif
tools/testing/selftests/bpf/test_progs.h +6 -0
@@ -567,4 +567,10 @@ void validate_msgs(const char *log_buf, struct expected_msgs *msgs,void free_msgs (struct expected_msgs *msgs);void verify_test_stderr (struct bpf_object *obj,struct bpf_program *prog); +#if defined(__x86_64__) || defined(__aarch64__) +#define ARCH_SUPPORTS_TRACING_MULTI_LINK 1 +#else +#define ARCH_SUPPORTS_TRACING_MULTI_LINK 0 +#endif +#endif /* __TEST_PROGS_H */
Signature · 1 line
2.55.0