bpf, x86: Use global buffer for trampoline size generation
Subsystems (3)
Jiri Olsa
reviewerPATCH 1/1 bpf, x86: Use global buffer for trampoline size generation
Currently arch_bpf_trampoline_size allocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while __arch_prepare_bpf_trampoline() calculates the required size, and the generated trampoline is discarded.
Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves tracing_multi attachment time.
With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256s
With the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258s
---
was "bpf, x86: Add support for jit dry run",
- doing this by having single scratch page instead as suggested by Alexei
arch/x86/net/bpf_jit_comp.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/net/bpf_jit_comp.c +15 -15
@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h>
@@ -35,6 +36,15 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); +return trampoline_size_image ?0 : -ENOMEM ; +} +late_initcall (init_trampoline_size_image); +static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {if (len ==1 )
@@ -4000,24 +4010,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) -* that returns writable memory in the module address space. -*/ - image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); -if (!image) +if (!trampoline_size_image)return -ENOMEM ; - ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -execmem_free (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }static int emit_bpf_dispatcher (u8 **pprog,int a,int b,s64 *progs,u8 *image,u8 *buf)
Signature · 1 line
2.54.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Concurrent BPF trampoline size calculations cause data races and severe cacheline bouncing on…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent BPF trampoline size calculations cause data races and severe cacheline bouncing on the globally shared trampoline_size_image buffer. --
commit d4cd8458939c8e8adafba04b2627a4ef1338ac21
Author: Jiri Olsa <[email protected]>
Subject: bpf, x86: Use global buffer for trampoline size generation
This commit replaces the per-invocation allocation of a temporary trampoline buffer in arch_bpf_trampoline_size() with a globally shared buffer allocated during initialization. This reduces the time it takes to calculate the required size by reusing the writable scratch page.
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 18 lines
arch/x86/net/bpf_jit_comp.c +1
-11@@ -4000,24 +4010,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) -* that returns writable memory in the module address space. -*/ - image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE ); -if (!image) +if (!trampoline_size_image)return -ENOMEM ;
arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 4021
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +4
-4- ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -execmem_free (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }
[Severity: Medium]
Can this lead to concurrent modifications of the shared scratch buffer?
If multiple threads attach trampolines or load BPF struct_ops programs simultaneously, they might both invoke arch_bpf_trampoline_size() and execute lockless writes into trampoline_size_image (such as inside emit_code()).
For example, bpf_struct_ops_map_update_elem() calls into the trampoline size calculation while holding only a per-map lock, allowing parallel updates across different maps.
While the generated bytes are ultimately discarded and the calculation depends on local pointer arithmetic rather than the buffer's contents, do these concurrent lockless writes risk severe cacheline bouncing? This could degrade concurrent attachment performance and will likely trigger KCSAN data race warnings.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 46 lines · Jiri Olsa
Currently
arch_bpf_trampoline_sizeallocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while__arch_prepare_bpf_trampoline()calculates the required size, and the generated trampoline is discarded.Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves
tracing_multiattachment time.With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256sWith the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258sSigned-off-by Jiri Olsa <[email protected]>---
was "bpf, x86: Add support for jit dry run",
- doing this by having single scratch page instead as suggested by Alexeiarch/x86/net/bpf_jit_comp.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h>
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +5
-0@@ -35,6 +36,15 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );
I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
If so then alloc it and free it every time. No need to keep one page in reserve.
pw-bot: cr
Quoted 50 lines · Alexei Starovoitov
Quoted 49 lines · Jiri Olsa
Currently
arch_bpf_trampoline_sizeallocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while__arch_prepare_bpf_trampoline()calculates the required size, and the generated trampoline is discarded.Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves
tracing_multiattachment time.With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256sWith the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258sSigned-off-by Jiri Olsa <[email protected]>---
was "bpf, x86: Add support for jit dry run",
- doing this by having single scratch page instead as suggested by Alexeiarch/x86/net/bpf_jit_comp.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h> @@ -35,6 +36,15 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true};
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
ugh sorry I forgot.. the page/image needs to be in execmem range for emitting call/jmp otherwise the delta won't fit in 4 bytes and it fails on emit_patch is_simm32 check
Quoted 2 lines
If so then alloc it and free it every time. No need to keep one page in reserve.
hum, you mean drop the change then?
jirka
Quoted 61 lines · Jiri Olsa
Quoted 50 lines · Alexei Starovoitov
Quoted 49 lines · Jiri Olsa
Currently
arch_bpf_trampoline_sizeallocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while__arch_prepare_bpf_trampoline()calculates the required size, and the generated trampoline is discarded.Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves
tracing_multiattachment time.With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256sWith the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258sSigned-off-by Jiri Olsa <[email protected]>---
was "bpf, x86: Add support for jit dry run",
- doing this by having single scratch page instead as suggested by Alexeiarch/x86/net/bpf_jit_comp.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h> @@ -35,6 +36,15 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true};Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
Quoted 3 lines
ugh sorry I forgot.. the page/image needs to be in execmem range for emitting call/jmp otherwise the delta won't fit in 4 bytes and it fails on
emit_patchis_simm32check
ahh. I see. Please mention it in the comment and
also use bpf_jit_alloc_exec(PAGE_SIZE).
execmem_alloc(EXECMEM_MODULE_DATA...) was working by accident. The VM ranges could have been different.
Quoted 4 lines
Quoted 2 lines
If so then alloc it and free it every time. No need to keep one page in reserve.
hum, you mean drop the change then?
Since it needs to be execmem preallocating one page for this is fine.
Quoted 66 lines · Alexei Starovoitov
Quoted 61 lines · Jiri Olsa
Quoted 50 lines · Alexei Starovoitov
Quoted 49 lines · Jiri Olsa
Currently
arch_bpf_trampoline_sizeallocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while__arch_prepare_bpf_trampoline()calculates the required size, and the generated trampoline is discarded.Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves
tracing_multiattachment time.With current code:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55227 functions serial_test_tracing_multi_bench_attach: attached in 1.563s serial_test_tracing_multi_bench_attach: detached in 0.256sWith the fix:
# ./test_progs -t tracing_multi_bench_attach -v ... serial_test_tracing_multi_bench_attach: found 55235 functions serial_test_tracing_multi_bench_attach: attached in 0.798s serial_test_tracing_multi_bench_attach: detached in 0.258sSigned-off-by Jiri Olsa <[email protected]>---
was "bpf, x86: Add support for jit dry run",
- doing this by having single scratch page instead as suggested by Alexeiarch/x86/net/bpf_jit_comp.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h> @@ -35,6 +36,15 @@ void __asan_store8(void *p);static bool all_callee_regs_used[4 ] = {true, true, true, true};Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
Quoted 3 lines
ugh sorry I forgot.. the page/image needs to be in execmem range for emitting call/jmp otherwise the delta won't fit in 4 bytes and it fails on
emit_patchis_simm32check
Quoted 5 lines
ahh. I see. Please mention it in the comment and
also usebpf_jit_alloc_exec(PAGE_SIZE).
execmem_alloc(EXECMEM_MODULE_DATA...)was working by accident. The VM ranges could have been different.
bpf_jit_alloc_exec returns read only page so I'd use bpf_jit_alloc_exec_rw, but it was recently removed in:
c7a2a3618290 x86/bpf: Make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA
so I ended up with:
trampoline_size_image = execmem_alloc_rw(EXECMEM_BPF, PAGE_SIZE);
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
If so then alloc it and free it every time. No need to keep one page in reserve.
hum, you mean drop the change then?
Since it needs to be execmem preallocating one page for this is fine.
ok, great
jirka
Quoted 15 lines · Jiri Olsa
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
ugh sorry I forgot.. the page/image needs to be in execmem range for emitting call/jmp otherwise the delta won't fit in 4 bytes and it fails on
emit_patchis_simm32check
Quoted 5 lines
Quoted 5 lines
ahh. I see. Please mention it in the comment and
also usebpf_jit_alloc_exec(PAGE_SIZE).
execmem_alloc(EXECMEM_MODULE_DATA...)was working by accident. The VM ranges could have been different.
On x86 it's not an accident ;-)
But would be a surprise if this is copied to another architecture.
Quoted 8 lines
bpf_jit_alloc_execreturns read only page so I'd usebpf_jit_alloc_exec_rw, but it was recently removed in:
c7a2a3618290x86/bpf: Makearch_bpf_trampoline_sizeallocate fromEXECMEM_MODULE_DATAso I ended up with:
trampoline_size_image = execmem_alloc_rw(EXECMEM_BPF, PAGE_SIZE);
This will work, but it will permanently damage 2M mapping in the module address space.
execmem_alloc_rw() takes a page out of the ROX cache and resets that page permissions to RW. This splits the large mapping of the 2M cache page and since this is a permanent allocation, that chunk will be forever mapped with 4k pages.
execmem_alloc(EXECMEM_MODULE_DATA...) does not use ROX cache, so it won't take a page out of ROX cache and reset it's permissions.
Since this is anyway x86 code and on x86 the ranges for data and code are *always* the same, I'd keep execmem_alloc(EXECMEM_MODULE_DATA...) and add a comment.
Quoted 1 line
jirka
Signature · 2 lines
Sincerely yours, Mike.
Quoted 42 lines · Mike Rapoport
Quoted 15 lines · Jiri Olsa
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 43
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );I think I asked it earlier... why does it have to be execmem ? Can it be normal page?
ugh sorry I forgot.. the page/image needs to be in execmem range for emitting call/jmp otherwise the delta won't fit in 4 bytes and it fails on
emit_patchis_simm32checkQuoted 5 lines
Quoted 5 lines
ahh. I see. Please mention it in the comment and
also usebpf_jit_alloc_exec(PAGE_SIZE).
execmem_alloc(EXECMEM_MODULE_DATA...)was working by accident. The VM ranges could have been different.On x86 it's not an accident ;-)
But would be a surprise if this is copied to another architecture.Quoted 8 lines
bpf_jit_alloc_execreturns read only page so I'd usebpf_jit_alloc_exec_rw, but it was recently removed in:
c7a2a3618290x86/bpf: Makearch_bpf_trampoline_sizeallocate fromEXECMEM_MODULE_DATAso I ended up with:
trampoline_size_image = execmem_alloc_rw(EXECMEM_BPF, PAGE_SIZE);This will work, but it will permanently damage 2M mapping in the module address space.
execmem_alloc_rw()takes a page out of the ROX cache and resets that page permissions to RW. This splits the large mapping of the 2M cache page and since this is a permanent allocation, that chunk will be forever mapped with 4k pages.
Quoted 6 lines
execmem_alloc(EXECMEM_MODULE_DATA...)does not use ROX cache, so it won't take a page out of ROX cache and reset it's permissions.Since this is anyway x86 code and on x86 the ranges for data and code are *always* the same, I'd keep
execmem_alloc(EXECMEM_MODULE_DATA...)and add a comment.
sgtm, if there's no objection I'll add comment to init_trampoline_size_image and send new version
thanks,
jirka