Skip to contentSkip to conversation

bpf, x86: Use global buffer for trampoline size generation

PATCH Applied Jiri Olsa 4 participants 4 messages Sep 10 – Sep 14

Subsystems (3)
lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

PATCH v2 1/1 bpf, x86: Use global buffer for trampoline size generation

Currently arch_bpf_trampoline_size allocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while __arch_prepare_bpf_trampoline() calculates the required size, and the generated trampoline is discarded.

Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves tracing_multi attachment time.

With current code:

  # ./test_progs -t tracing_multi_bench_attach -v
  ...
  serial_test_tracing_multi_bench_attach: found 55227 functions
  serial_test_tracing_multi_bench_attach: attached in   1.563s
  serial_test_tracing_multi_bench_attach: detached in   0.256s

With the fix:

  # ./test_progs -t tracing_multi_bench_attach -v
  ...
  serial_test_tracing_multi_bench_attach: found 55235 functions
  serial_test_tracing_multi_bench_attach: attached in   0.798s
  serial_test_tracing_multi_bench_attach: detached in   0.258s

---
v1: https://lore.kernel.org/bpf/[email protected]/

v2 changes:
- added comment to init_trampoline_size_image why we use execmem allocation from EXECMEM_MODULE_DATA [Alexei,Mike]

 arch/x86/net/bpf_jit_comp.c | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/net/bpf_jit_comp.c +21 -15
index bba351944202..9f9b1456dab0 100644
@@ -9,6 +9,7 @@
 #include <linux/filter.h>
 #include <linux/if_vlan.h>
 #include <linux/bitfield.h>
+#include <linux/init.h>
 #include <linux/bpf.h>
 #include <linux/bpf_verifier.h>
 #include <linux/memory.h>
@@ -35,6 +36,21 @@ void __asan_store8(void *p);
 
 static bool all_callee_regs_used[4] = {true, true, true, true};
 
+static void *trampoline_size_image;
+
+static int __init init_trampoline_size_image(void)
+{
+	/*
+	 * The generated trampoline contains calls and jumps with 32bit relative
+	 * offsets, so the scratch image must be in the execmem range.
+	 * On x86, module data and executable memory share the same address range,
+	 * so using EXECMEM_MODULE_DATA to get writable memory.
+	 */
+	trampoline_size_image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
+	return trampoline_size_image ? 0 : -ENOMEM;
+}
+late_initcall(init_trampoline_size_image);
+
 static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
 {
 	if (len == 1)
@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 			     struct bpf_tramp_nodes *tnodes, void *func_addr)
 {
 	struct bpf_tramp_image im;
-	void *image;
-	int ret;
 
-	/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
-	 *
-	 * We cannot use kvmalloc here, because we need image to be in
-	 * module memory range.
-	 * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA)
-	 * that returns writable memory in the module address space.
-	 */
-	image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
-	if (!image)
+	if (!trampoline_size_image)
 		return -ENOMEM;
 
-	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
-					    m, flags, tnodes, func_addr);
-	execmem_free(image);
-	return ret;
+	return __arch_prepare_bpf_trampoline(&im, trampoline_size_image,
+					     trampoline_size_image + PAGE_SIZE,
+					     trampoline_size_image, m, flags,
+					     tnodes, func_addr);
 }
 
 static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image, u8 *buf)
Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Concurrent calls to arch_bpf_trampoline_size() write to the globally shared, unsynchronized…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent calls to arch_bpf_trampoline_size() write to the globally shared, unsynchronized trampoline_size_image buffer, resulting in a data race (Undefined Behavior) and cache-line contention. --

commit 4c6aeb9264cf2877d767fb9a021802485dd9d3a7
Author: Jiri Olsa <[email protected]>

bpf, x86: Use global buffer for trampoline size generation

This commit changes the BPF trampoline size calculation to use a single, globally allocated executable memory buffer instead of allocating a temporary one per invocation. This change aims to improve attachment time during tracing by eliminating the overhead of repeated memory allocations and frees.

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index bba351944202c..9f9b1456dab0d 100644

[ ... ]

Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +1 -0
@@ -35,6 +36,21 @@ void __asan_store8(void *p);
 
 static bool all_callee_regs_used[4] = {true, true, true, true};
 
+static void *trampoline_size_image;

arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 49

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+
+static int __init init_trampoline_size_image(void)
+{
+	/*
+	 * The generated trampoline contains calls and jumps with 32bit relative
+	 * offsets, so the scratch image must be in the execmem range.
+	 * On x86, module data and executable memory share the same address range,
+	 * so using EXECMEM_MODULE_DATA to get writable memory.
+	 */
+	trampoline_size_image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);

[Severity: Medium]
Is this global allocation safe against concurrent access? Multiple threads could compute trampoline sizes concurrently, for example when attaching BPF programs to kernel functions that hash to different buckets in trampoline_locks, or when concurrently updating different struct_ops maps.

Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +3 -0
+	return trampoline_size_image ? 0 : -ENOMEM;
+}
+late_initcall(init_trampoline_size_image);

[ ... ]

Quoted 18 lines
arch/x86/net/bpf_jit_comp.c +1 -11
@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 			     struct bpf_tramp_nodes *tnodes, void *func_addr)
 {
 	struct bpf_tramp_image im;
-	void *image;
-	int ret;
 
-	/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
-	 *
-	 * We cannot use kvmalloc here, because we need image to be in
-	 * module memory range.
-	 * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA)
-	 * that returns writable memory in the module address space.
-	 */
-	image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
-	if (!image)
+	if (!trampoline_size_image)
 		return -ENOMEM;

arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 4027

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +4 -4
 
-	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
-					    m, flags, tnodes, func_addr);
-	execmem_free(image);
-	return ret;
+	return __arch_prepare_bpf_trampoline(&im, trampoline_size_image,
+					     trampoline_size_image + PAGE_SIZE,
+					     trampoline_size_image, m, flags,
+					     tnodes, func_addr);
 }

[Severity: Medium]
When __arch_prepare_bpf_trampoline() writes to trampoline_size_image, could this lead to a data race if called concurrently?

Since trampoline_size_image is globally shared and no lock protects it here, writers in emit_code() might overwrite each other's data:

arch_bpf_trampoline_size()
__arch_prepare_bpf_trampoline()

    emit_code()
      *(u32 *)ptr = bytes;

While the memory contents aren't executed, concurrent writes will cause undefined behavior and cache-line ping-ponging, which could trigger KCSAN warnings and crash systems with panic_on_warn enabled. Should this use a per-CPU buffer or have explicit synchronization?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
Quoted 34 lines · On Thu, Sep 10, 2026 at 09:55:56AM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent calls to arch_bpf_trampoline_size() write to the globally shared, unsynchronized trampoline_size_image buffer, resulting in a data race (Undefined Behavior) and cache-line contention. --

commit 4c6aeb9264cf2877d767fb9a021802485dd9d3a7
Author: Jiri Olsa <[email protected]>

bpf, x86: Use global buffer for trampoline size generation

This commit changes the BPF trampoline size calculation to use a single, globally allocated executable memory buffer instead of allocating a temporary one per invocation. This change aims to improve attachment time during tracing by eliminating the overhead of repeated memory allocations and frees.

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index bba351944202c..9f9b1456dab0d 100644

[ ... ]

Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +9 -0
@@ -35,6 +36,21 @@ void __asan_store8(void *p);
 
 static bool all_callee_regs_used[4] = {true, true, true, true};
 
+static void *trampoline_size_image;
+
+static int __init init_trampoline_size_image(void)
+{
+	/*
+	 * The generated trampoline contains calls and jumps with 32bit relative
+	 * offsets, so the scratch image must be in the execmem range.
+	 * On x86, module data and executable memory share the same address range,
+	 * so using EXECMEM_MODULE_DATA to get writable memory.
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c void __asan_store8(void *p); · line 49

Quoted 2 lines
arch/x86/net/bpf_jit_comp.c +2 -0
+	 */
+	trampoline_size_image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);

[Severity: Medium]
Is this global allocation safe against concurrent access? Multiple threads could compute trampoline sizes concurrently, for example when attaching BPF programs to kernel functions that hash to different buckets in trampoline_locks, or when concurrently updating different struct_ops maps.

we can have multiple instances running without the lock, the trampoline does not depend on already generated code and we do not use the image we need just the size

Quoted 43 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +3 -0
+	return trampoline_size_image ? 0 : -ENOMEM;
+}
+late_initcall(init_trampoline_size_image);

[ ... ]

Quoted 18 lines
arch/x86/net/bpf_jit_comp.c +1 -11
@@ -4000,24 +4016,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 			     struct bpf_tramp_nodes *tnodes, void *func_addr)
 {
 	struct bpf_tramp_image im;
-	void *image;
-	int ret;
 
-	/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
-	 *
-	 * We cannot use kvmalloc here, because we need image to be in
-	 * module memory range.
-	 * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA)
-	 * that returns writable memory in the module address space.
-	 */
-	image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
-	if (!image)
+	if (!trampoline_size_image)
 		return -ENOMEM;

arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 4027

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +4 -4
 
-	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
-					    m, flags, tnodes, func_addr);
-	execmem_free(image);
-	return ret;
+	return __arch_prepare_bpf_trampoline(&im, trampoline_size_image,
+					     trampoline_size_image + PAGE_SIZE,
+					     trampoline_size_image, m, flags,
+					     tnodes, func_addr);
 }

[Severity: Medium]
When __arch_prepare_bpf_trampoline() writes to trampoline_size_image, could this lead to a data race if called concurrently?

Since trampoline_size_image is globally shared and no lock protects it here, writers in emit_code() might overwrite each other's data:

arch_bpf_trampoline_size()
__arch_prepare_bpf_trampoline()

Quoted 7 lines
    emit_code()
      *(u32 *)ptr = bytes;

While the memory contents aren't executed, concurrent writes will cause undefined behavior and cache-line ping-ponging, which could trigger KCSAN warnings and crash systems with panic_on_warn enabled. Should this use a per-CPU buffer or have explicit synchronization?

the image for arch_bpf_trampoline_size is just a scratch pad that's never used/executed, we take just the size, so no problem here

jirka

#3 lore raw
In reply to Jiri Olsa
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 10 lines · On Thu, 10 Sep 2026 11:43:53 +0200 you wrote:

Currently arch_bpf_trampoline_size allocates and frees a temporary trampoline buffer on every invocation. The buffer is only used as a scratch space while __arch_prepare_bpf_trampoline() calculates the required size, and the generated trampoline is discarded.

Allocating a writable scratch page during kernel initialization and reusing it for all size calculations. This improves tracing_multi attachment time.

[...]

Here is the summary with links:
- [PATCHv2,bpf-next] bpf, x86: Use global buffer for trampoline size generation

    https://git.kernel.org/bpf/bpf-next/c/29fcf9ce4bbd

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html