bpf, x86: Use global buffer for trampoline size generation
Subsystems (3)
Jiri Olsa
reviewerPATCH v2 1/1 bpf, x86: Use global buffer for trampoline size generation
Currently arch_bpf_trampoline_size allocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while __arch_prepare_bpf_trampoline() calculates the required size, and the generated trampoline is discarded.
Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves tracing_multi attachment time.
With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256s
With the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258s
---
v1: https://lore.kernel.org/bpf/[email protected]/
v2 changes:
- added comment to init_trampoline_size_image why we use execmem allocation from EXECMEM_MODULE_DATA [Alexei,Mike]
arch/x86/net/bpf_jit_comp.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/net/bpf_jit_comp.c +21 -15
@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h>
@@ -35,6 +36,21 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ +/* +* The generated trampoline contains calls and jumps with 32bit relative +* offsets, so the scratch image must be in the execmem range. +* On x86, module data and executable memory share the same address range, +* so using EXECMEM_MODULE_DATA to get writable memory. +*/ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); +return trampoline_size_image ?0 : -ENOMEM ; +} +late_initcall (init_trampoline_size_image); +static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {if (len ==1 )
@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) -* that returns writable memory in the module address space. -*/ - image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); -if (!image) +if (!trampoline_size_image)return -ENOMEM ; - ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -execmem_free (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }static int emit_bpf_dispatcher (u8 **pprog,int a,int b,s64 *progs,u8 *image,u8 *buf)
Signature · 1 line
2.54.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Concurrent calls to arch_bpf_trampoline_size() write to the globally shared, unsynchronized…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent calls to arch_bpf_trampoline_size() write to the globally shared, unsynchronized trampoline_size_image buffer, resulting in a data race (Undefined Behavior) and cache-line contention. --
commit 4c6aeb9264cf2877d767fb9a021802485dd9d3a7
Author: Jiri Olsa <[email protected]>
bpf, x86: Use global buffer for trampoline size generation
This commit changes the BPF trampoline size calculation to use a single, globally allocated executable memory buffer instead of allocating a temporary one per invocation. This change aims to improve attachment time during tracing by eliminating the overhead of repeated memory allocations and frees.
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +1
-0@@ -35,6 +36,21 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static void *trampoline_size_image;
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 49
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ +static int __init init_trampoline_size_image (void ) +{ +/* +* The generated trampoline contains calls and jumps with 32bit relative +* offsets, so the scratch image must be in the execmem range. +* On x86, module data and executable memory share the same address range, +* so using EXECMEM_MODULE_DATA to get writable memory. +*/ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );
[Severity: Medium]
Is this global allocation safe against concurrent access? Multiple threads could compute trampoline sizes concurrently, for example when attaching BPF programs to kernel functions that hash to different buckets in trampoline_locks, or when concurrently updating different struct_ops maps.
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +3
-0+return trampoline_size_image ?0 : -ENOMEM ; +} +late_initcall (init_trampoline_size_image);
[ ... ]
Quoted 18 lines
arch/x86/net/bpf_jit_comp.c +1
-11@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) -* that returns writable memory in the module address space. -*/ - image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); -if (!image) +if (!trampoline_size_image)return -ENOMEM ;
arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 4027
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +4
-4- ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -execmem_free (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }
[Severity: Medium]
When __arch_prepare_bpf_trampoline() writes to trampoline_size_image, could this lead to a data race if called concurrently?
Since trampoline_size_image is globally shared and no lock protects it here, writers in emit_code() might overwrite each other's data:
arch_bpf_trampoline_size()
__arch_prepare_bpf_trampoline()
emit_code()
*(u32 *)ptr = bytes;While the memory contents aren't executed, concurrent writes will cause undefined behavior and cache-line ping-ponging, which could trigger KCSAN warnings and crash systems with panic_on_warn enabled. Should this use a per-CPU buffer or have explicit synchronization?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 34 lines · On Thu, Sep 10, 2026 at 09:55:56AM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent calls toarch_bpf_trampoline_size()write to the globally shared, unsynchronizedtrampoline_size_imagebuffer, resulting in a data race (Undefined Behavior) and cache-line contention. --commit
4c6aeb9264cf2877d767fb9a021802485dd9d3a7
Author: Jiri Olsa <[email protected]>bpf, x86: Use global buffer for trampoline size generation
This commit changes the BPF trampoline size calculation to use a single, globally allocated executable memory buffer instead of allocating a temporary one per invocation. This change aims to improve attachment time during tracing by eliminating the overhead of repeated memory allocations and frees.
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +9
-0@@ -35,6 +36,21 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ +/* +* The generated trampoline contains calls and jumps with 32bit relative +* offsets, so the scratch image must be in the execmem range. +* On x86, module data and executable memory share the same address range, +* so using EXECMEM_MODULE_DATA to get writable memory.
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 49
Quoted 2 lines
arch/x86/net/bpf_jit_comp.c +2
-0+*/ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );[Severity: Medium]
Is this global allocation safe against concurrent access? Multiple threads could compute trampoline sizes concurrently, for example when attaching BPF programs to kernel functions that hash to different buckets intrampoline_locks, or when concurrently updating differentstruct_opsmaps.
we can have multiple instances running without the lock, the trampoline does not depend on already generated code and we do not use the image we need just the size
Quoted 43 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +3
-0+return trampoline_size_image ?0 : -ENOMEM ; +} +late_initcall (init_trampoline_size_image);[ ... ]
Quoted 18 lines
arch/x86/net/bpf_jit_comp.c +1
-11@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) -* that returns writable memory in the module address space. -*/ - image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); -if (!image) +if (!trampoline_size_image)return -ENOMEM ;arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 4027
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +4
-4- ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -execmem_free (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }[Severity: Medium]
When__arch_prepare_bpf_trampoline()writes totrampoline_size_image, could this lead to a data race if called concurrently?Since
trampoline_size_imageis globally shared and no lock protects it here, writers inemit_code()might overwrite each other's data:
arch_bpf_trampoline_size()
__arch_prepare_bpf_trampoline()
Quoted 7 lines
emit_code() *(u32 *)ptr = bytes;While the memory contents aren't executed, concurrent writes will cause undefined behavior and cache-line ping-ponging, which could trigger KCSAN warnings and crash systems with
panic_on_warnenabled. Should this use a per-CPU buffer or have explicit synchronization?
the image for arch_bpf_trampoline_size is just a scratch pad that's never used/executed, we take just the size, so no problem here
jirka
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Thu, 10 Sep 2026 11:43:53 +0200 you wrote:
Currently
arch_bpf_trampoline_sizeallocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while__arch_prepare_bpf_trampoline()calculates the required size, and the generated trampoline is discarded.Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves
tracing_multiattachment time.[...]
Here is the summary with links:
- [PATCHv2,bpf-next] bpf, x86: Use global buffer for trampoline size generation
https://git.kernel.org/bpf/bpf-next/c/29fcf9ce4bbd
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html