Skip to contentSkip to conversation

x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

PATCH Applied Mike Rapoport 5 participants 6 messages Aug 18 – Aug 20

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

From: "Mike Rapoport (Microsoft)" <[email protected]>

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)

Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.

Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")
---
 arch/x86/net/bpf_jit_comp.c | 8 +++++---
 include/linux/filter.h      | 1 -
 kernel/bpf/core.c           | 5 -----
 3 files changed, 5 insertions(+), 9 deletions(-)
arch/x86/net/bpf_jit_comp.c +5 -3
index b2feec81e231..7d064d3e2788 100644
@@ -13,6 +13,7 @@
 #include <linux/bpf_verifier.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
+#include <linux/execmem.h>
 #include <asm/extable.h>
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
@@ -3706,15 +3707,16 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 	 *
 	 * We cannot use kvmalloc here, because we need image to be in
 	 * module memory range.
-	 * Since it must be writable use bpf_jit_alloc_exec_rw().
+	 * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA)
+	 * that returns writable memory in the module address space.
 	 */
-	image = bpf_jit_alloc_exec_rw(PAGE_SIZE);
+	image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
 	if (!image)
 		return -ENOMEM;
 
 	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
 					    m, flags, tnodes, func_addr);
-	bpf_jit_free_exec(image);
+	execmem_free(image);
 	return ret;
 }
 
include/linux/filter.h +0 -1
index 32d5297c557e..14acb2455746 100644
@@ -1333,7 +1333,6 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
 void bpf_jit_binary_free(struct bpf_binary_header *hdr);
 u64 bpf_jit_alloc_exec_limit(void);
 void *bpf_jit_alloc_exec(unsigned long size);
-void *bpf_jit_alloc_exec_rw(unsigned long size);
 void bpf_jit_free_exec(void *addr);
 void bpf_jit_free(struct bpf_prog *fp);
 struct bpf_binary_header *
kernel/bpf/core.c +0 -5
index e2076667b245..1b89c18cf246 100644
@@ -1128,11 +1128,6 @@ void *bpf_jit_alloc_exec(unsigned long size)
 	return execmem_alloc(EXECMEM_BPF, size);
 }
 
-void *bpf_jit_alloc_exec_rw(unsigned long size)
-{
-	return execmem_alloc_rw(EXECMEM_BPF, size);
-}
-
 void bpf_jit_free_exec(void *addr)
 {
 	execmem_free(addr);

base-commit: 07edbd8fe6416f45e9ba6b0edcd3e40770601ebc

Signature · 1 line
2.53.0
#1 lore raw
The Fixes tag references commit f0334294a428, which does not exist in this repository. Should this be:
Quoted 19 lines

x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)

Quoted 7 lines

Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.

Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")
             ^^^^^^^^^^^^

The Fixes tag references commit f0334294a428, which does not exist in this repository. Should this be:

Fixes 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable")

That commit introduced bpf_jit_alloc_exec_rw() and the performance issue being addressed here.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32142507111

#2 lore raw
Quoted 26 lines · On Tue, Aug 18, 2026 at 02:07:23PM +0000, [email protected] wrote:
Quoted 19 lines

x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)

Quoted 7 lines

Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.

Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")
Quoted 6 lines
             ^^^^^^^^^^^^

The Fixes tag references commit f0334294a428, which does not exist in this repository. Should this be:

Fixes 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable")

Argh, checked the log on a wrong branch.

Can you folks fix it up when applying or you'd need v3?

Quoted 2 lines

That commit introduced bpf_jit_alloc_exec_rw() and the performance issue being addressed here.

Signature · 2 lines
Sincerely yours,
Mike.
maintainer
#3 lore raw
Quoted 29 lines · Mike Rapoport
Quoted 28 lines · On Tue, Aug 18, 2026 at 02:07:23PM +0000, [email protected] wrote:
Quoted 19 lines

x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)

Quoted 7 lines

Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.

Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")
              ^^^^^^^^^^^^
Quoted 8 lines
Quoted 4 lines

The Fixes tag references commit f0334294a428, which does not exist in this repository. Should this be:

Fixes 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable")

Argh, checked the log on a wrong branch.

Can you folks fix it up when applying or you'd need v3?

Yeap, we'll do, np.

Quoted 3 lines
Quoted 2 lines

That commit introduced bpf_jit_alloc_exec_rw() and the performance issue being addressed here.

Tested-by on v1 1/1
#4 lore raw
In reply to Mike Rapoport
Quoted 19 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.

The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.

On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)

Quoted 8 lines

Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.

Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")

thanks,
jirak

Quoted 71 lines
---
 arch/x86/net/bpf_jit_comp.c | 8 +++++---
 include/linux/filter.h      | 1 -
 kernel/bpf/core.c           | 5 -----
 3 files changed, 5 insertions(+), 9 deletions(-)
arch/x86/net/bpf_jit_comp.c +5 -3
index b2feec81e231..7d064d3e2788 100644
@@ -13,6 +13,7 @@
 #include <linux/bpf_verifier.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
+#include <linux/execmem.h>
 #include <asm/extable.h>
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
@@ -3706,15 +3707,16 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 	 *
 	 * We cannot use kvmalloc here, because we need image to be in
 	 * module memory range.
-	 * Since it must be writable use bpf_jit_alloc_exec_rw().
+	 * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA)
+	 * that returns writable memory in the module address space.
 	 */
-	image = bpf_jit_alloc_exec_rw(PAGE_SIZE);
+	image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE);
 	if (!image)
 		return -ENOMEM;
 
 	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
 					    m, flags, tnodes, func_addr);
-	bpf_jit_free_exec(image);
+	execmem_free(image);
 	return ret;
 }
 
include/linux/filter.h +0 -1
index 32d5297c557e..14acb2455746 100644
@@ -1333,7 +1333,6 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
 void bpf_jit_binary_free(struct bpf_binary_header *hdr);
 u64 bpf_jit_alloc_exec_limit(void);
 void *bpf_jit_alloc_exec(unsigned long size);
-void *bpf_jit_alloc_exec_rw(unsigned long size);
 void bpf_jit_free_exec(void *addr);
 void bpf_jit_free(struct bpf_prog *fp);
 struct bpf_binary_header *
kernel/bpf/core.c +0 -5
index e2076667b245..1b89c18cf246 100644
@@ -1128,11 +1128,6 @@ void *bpf_jit_alloc_exec(unsigned long size)
 	return execmem_alloc(EXECMEM_BPF, size);
 }
 
-void *bpf_jit_alloc_exec_rw(unsigned long size)
-{
-	return execmem_alloc_rw(EXECMEM_BPF, size);
-}
-
 void bpf_jit_free_exec(void *addr)
 {
 	execmem_free(addr);

base-commit: 07edbd8fe6416f45e9ba6b0edcd3e40770601ebc

Signature · 1 line
2.53.0
#5 lore raw
In reply to Mike Rapoport
This patch was applied to bpf/bpf.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This patch was applied to bpf/bpf.git (master)
by Daniel Borkmann <[email protected]>:

Quoted 9 lines · On Tue, 18 Aug 2026 16:05:10 +0300 you wrote:

From: "Mike Rapoport (Microsoft)" <[email protected]>

Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].

The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().

[...]

Here is the summary with links:
- x86/bpf: make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA

    https://git.kernel.org/bpf/bpf/c/c7a2a3618290

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html