bpf, x86: enable EXECMEM_ROX_CACHE for BPF allocations
v3 cover bpf, x86: enable EXECMEM_ROX_CACHE for BPF allocations
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call bpf_jit_alloc_exec() immediately make it ROX with set_memory_rox().
The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with text_poke().
These patches do several small adjustments to how BPF allocates executable memory and enable EXECMEM_ROX_CACHE for BPF allocations on x86.
---
v3 changes:
* replace vmalloc() with vzalloc() in bpf_dispatcher_change_prog()
* rebase on the current bpf-next
v2: https://patch.msgid.link/[email protected]
* rebase on the current bpf-next
v1: https://patch.msgid.link/[email protected]
---
---
Mike Rapoport (Microsoft) (5):
bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
bpf: drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec()
bpf: alloc_prog_pack(): skip ROX management for already ROX memory bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable x86/bpf: enable EXECMEM_ROX_CACHE for BPF allocations
arch/x86/mm/init.c | 4 ++-- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 30 +++++++++++++++++++++--------- kernel/bpf/dispatcher.c | 5 ++++- 5 files changed, 30 insertions(+), 15 deletions(-)
---
base-commit: d1f4b56417a3dc1a0600f960b14f46bd25eda89d
change-id: 20260626-execmem-x86-rox-bpf-v0-b4241ade80df
--
Sincerely yours,
Mike.
PATCH v3 1/5 bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
bpf_dispatcher->rw_image is a temporary writable buffer that arch_prepare_bpf_dispatcher() fills and then copies into bpf_dispatcher->image using bpf_arch_text_copy().
The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against ->image, so ->rw_image does not need to live in the module address range.
Allocate ->rw_image with vzalloc() to avoid permissions dance when EXECMEM_BPF will be backed by ROX caches.
Using vzalloc() rather than vmalloc() ensures that the memory that bpf_dispatcher_update() unconditionally copies into the executable buffer is zeroed, which is not ideal but still better than random memory returned by the existing bpf_jit_alloc_exec() or plain vmalloc().
Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of space in the more scarce module address space.
--- kernel/bpf/dispatcher.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/bpf/dispatcher.c +4 -1
@@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, d->image =bpf_prog_pack_alloc (PAGE_SIZE , bpf_jit_fill_hole_with_zero, false);if (!d->image) goto out; - d->rw_image =bpf_jit_alloc_exec (PAGE_SIZE ); +/* d->rw_image doesn't need to be in module memory range, so we +* can use vzalloc. +*/ + d->rw_image =vzalloc (PAGE_SIZE );if (!d->rw_image) {bpf_prog_pack_free (d->image,PAGE_SIZE ); d->image =NULL ;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 1/5] bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
This isn't a bug, but for multi-line comments under kernel/bpf/ the opening /* is expected to sit on its own line, so would this read better as:
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 154
Quoted 13 lines
kernel/bpf/dispatcher.c +4
-1@@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, d->image =bpf_prog_pack_alloc (PAGE_SIZE , bpf_jit_fill_hole_with_zero, false);if (!d->image) goto out; - d->rw_image =bpf_jit_alloc_exec (PAGE_SIZE ); +/* d->rw_image doesn't need to be in module memory range, so we +* can use vzalloc. +*/ + d->rw_image =vzalloc (PAGE_SIZE );
This isn't a bug, but for multi-line comments under kernel/bpf/ the opening /* is expected to sit on its own line, so would this read better as:
/* * d->rw_image doesn't need to be in module memory range, so we * can use vzalloc. */
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 157
Quoted 3 lines
kernel/bpf/dispatcher.c +0
-0if (!d->rw_image) {bpf_prog_pack_free (d->image,PAGE_SIZE ); d->image =NULL ;
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/29482648692
Re: [PATCH bpf-next v3 1/5] bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
Quoted 17 lines · On Thu, Jul 16, 2026 at 08:52:25AM +0000, [email protected] wrote:
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 154
Quoted 13 lines
kernel/bpf/dispatcher.c +4
-1@@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, d->image =bpf_prog_pack_alloc (PAGE_SIZE , bpf_jit_fill_hole_with_zero, false);if (!d->image) goto out; - d->rw_image =bpf_jit_alloc_exec (PAGE_SIZE ); +/* d->rw_image doesn't need to be in module memory range, so we +* can use vzalloc. +*/ + d->rw_image =vzalloc (PAGE_SIZE );This isn't a bug, but for multi-line comments under
kernel/bpf/the opening /* is expected to sit on its own line, so would this read better as:
Quoted 4 lines
/* * d->rw_image doesn't need to be in module memory range, so we * can use vzalloc. */
This really contradicts what I see in the code :/
Quoted 10 lines
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 157
Quoted 3 lines
kernel/bpf/dispatcher.c +0
-0if (!d->rw_image) {bpf_prog_pack_free (d->image,PAGE_SIZE ); d->image =NULL ;---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/29482648692
Signature · 2 lines
Sincerely yours, Mike.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/5] bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
Quoted 18 lines · Mike Rapoport
Quoted 17 lines · On Thu, Jul 16, 2026 at 08:52:25AM +0000, [email protected] wrote:
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 154
Quoted 13 lines
kernel/bpf/dispatcher.c +4
-1@@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, d->image =bpf_prog_pack_alloc (PAGE_SIZE , bpf_jit_fill_hole_with_zero, false);if (!d->image) goto out; - d->rw_image =bpf_jit_alloc_exec (PAGE_SIZE ); +/* d->rw_image doesn't need to be in module memory range, so we +* can use vzalloc. +*/ + d->rw_image =vzalloc (PAGE_SIZE );This isn't a bug, but for multi-line comments under
kernel/bpf/the opening /* is expected to sit on its own line, so would this read better as:
Quoted 7 lines
Quoted 4 lines
/* * d->rw_image doesn't need to be in module memory range, so we * can use vzalloc. */This really contradicts what I see in the code :/
We ended up changing to the multi-line convention, but you can let it be. People are out on vacation, hence things are moving slow, but I plan to get to this series soon. Sorry about the delay.
Quoted 10 lines
Quoted 10 lines
kernel/bpf/dispatcher.c void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, · line 157
Quoted 3 lines
kernel/bpf/dispatcher.c +0
-0if (!d->rw_image) {bpf_prog_pack_free (d->image,PAGE_SIZE ); d->image =NULL ;---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/29482648692
PATCH v3 2/5 bpf: drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec()
In reply to Mike Rapoport (Microsoft)bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the corresponding execmem APIs.
Architectures define the properties of the memory range needed by BPF in their initialization of execmem and don't need to override neither of them.
Drop the __weak qualifier from bpf_jit_alloc_exec() and bpf_jit_free_exec().
--- kernel/bpf/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/core.c +2 -2
@@ -1116,12 +1116,12 @@ void bpf_jit_uncharge_modmem(u32 size)atomic_long_sub (size, &bpf_jit_current); } -void *__weak bpf_jit_alloc_exec (unsigned long size) +void *bpf_jit_alloc_exec (unsigned long size) {return execmem_alloc (EXECMEM_BPF , size); } -void __weak bpf_jit_free_exec (void *addr) +void bpf_jit_free_exec (void *addr) {execmem_free (addr); }
Signature · 1 line
2.53.0
PATCH v3 3/5 bpf: alloc_prog_pack(): skip ROX management for already ROX memory
In reply to Mike Rapoport (Microsoft)execmem_alloc() can return ROX memory that is already filled with architecture defined trapping instructions.
In preparation for enabling this mode for BPF on x86, make sure that there is no redundant management of the ROX memory.
There is no need to fill allocated memory with trapping instructions, to request permissions reset on free and to set ROX permissions as this all is handled by execmem_alloc().
Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if execmem_alloc() returns ROX memory and skip the redundant steps in that case.
--- kernel/bpf/core.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
kernel/bpf/core.c +14 -7
@@ -916,6 +916,11 @@ static LIST_HEAD(pack_list);#define BPF_PROG_CHUNK_COUNT (BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE) +static bool bpf_jit_mem_is_rox (void ) +{ +return execmem_is_rox (EXECMEM_BPF ); +} +static struct bpf_prog_pack *alloc_new_pack (bpf_jit_fill_hole_t bpf_fill_ill_insns) {struct bpf_prog_pack *pack;
@@ -927,16 +932,18 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins pack->ptr =bpf_jit_alloc_exec (BPF_PROG_PACK_SIZE );if (!pack->ptr) goto out; -bpf_fill_ill_insns (pack->ptr,BPF_PROG_PACK_SIZE );bitmap_zero (pack->bitmap,BPF_PROG_PACK_SIZE /BPF_PROG_CHUNK_SIZE );if (static_branch_unlikely (&bpf_pred_flush_enabled)) pack->arch_flush_needed = true; -set_vm_flush_reset_perms (pack->ptr); - err =set_memory_rox ((unsigned long )pack->ptr, -BPF_PROG_PACK_SIZE /PAGE_SIZE ); -if (err) - goto out; +if (!bpf_jit_mem_is_rox ()) { +bpf_fill_ill_insns (pack->ptr,BPF_PROG_PACK_SIZE ); +set_vm_flush_reset_perms (pack->ptr); + err =set_memory_rox ((unsigned long )pack->ptr, +BPF_PROG_PACK_SIZE /PAGE_SIZE ); +if (err) + goto out; + }list_add_tail (&pack->list, &pack_list);return pack;
@@ -965,7 +972,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, boolpr_warn_once ("BPF: Predictors not flushed for allocations greater than BPF_PROG_PACK_SIZE\n" ); size =round_up (size,PAGE_SIZE ); ptr =bpf_jit_alloc_exec (size); -if (ptr) { +if (ptr && !bpf_jit_mem_is_rox ()) {int err;bpf_fill_ill_insns (ptr, size);
Signature · 1 line
2.53.0
PATCH v3 4/5 bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
In reply to Mike Rapoport (Microsoft)arch_bpf_trampoline_size() allocates a buffer to get actual size required for a trampoline.
This buffer must be in the module address space because __arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for buffer allocation in arch_bpf_trampoline_size().
--- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +2 -3
@@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,int ret;/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* This will NOT cause fragmentation in direct map, as we do not -* call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. +* Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image =bpf_jit_alloc_exec (PAGE_SIZE ); + image =bpf_jit_alloc_exec_rw (PAGE_SIZE );if (!image)return -ENOMEM ;
include/linux/filter.h +1 -0
@@ -1333,6 +1333,7 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,void bpf_jit_binary_free (struct bpf_binary_header *hdr);u64 bpf_jit_alloc_exec_limit (void );void *bpf_jit_alloc_exec (unsigned long size); +void *bpf_jit_alloc_exec_rw (unsigned long size);void bpf_jit_free_exec (void *addr);void bpf_jit_free (struct bpf_prog *fp); struct bpf_binary_header *
kernel/bpf/core.c +5 -0
@@ -1128,6 +1128,11 @@ void *bpf_jit_alloc_exec(unsigned long size)return execmem_alloc (EXECMEM_BPF , size); } +void *bpf_jit_alloc_exec_rw (unsigned long size) +{ +return execmem_alloc_rw (EXECMEM_BPF , size); +} +void bpf_jit_free_exec (void *addr) {execmem_free (addr);
Signature · 1 line
2.53.0
Re: [PATCH bpf-next v3 4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
Quoted 26 lines · Mike Rapoport (Microsoft)
arch_bpf_trampoline_size()allocates a buffer to get actual size required for a trampoline.This buffer must be in the module address space because
__arch_prepare_bpf_trampoline()calculates rel32 offsets relatively to that buffer.In preparation for enabling ROX mode for
EXECMEM_BPFmake sure that the allocated memory is writable.Add
bpf_jit_alloc_exec_rw()wrapper forexecmem_alloc_rw()and use it for > buffer allocation inarch_bpf_trampoline_size().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +0
-0@@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,int ret;
arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 3711
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2
-3/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* This will NOT cause fragmentation in direct map, as we do not -* call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. +* Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image =bpf_jit_alloc_exec (PAGE_SIZE ); + image =bpf_jit_alloc_exec_rw (PAGE_SIZE );
hi,
this change (this particular patch plus possibly others in this set) is causing tracing_multi attachment bench slowdown
the benchmark allocates huge number of trampolines and I'm seeing extra arch_bpf_trampoline_size code paths in the attached perf profile
I'm not that familiar with the allocator, but following hack makes the benchmark ok again:
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6a94370a2448..bbff3c9c6681 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1130,7 +1130,7 @@ void *bpf_jit_alloc_exec(unsigned long size)
void *bpf_jit_alloc_exec_rw(unsigned long size)
{
- return execmem_alloc_rw(EXECMEM_BPF, size);
+ return execmem_alloc(EXECMEM_MODULE_DATA, size);
}
void bpf_jit_free_exec(void *addr)I still need to do more checks, but I'm wondering if we could actually fix this by not allocating image data in arch_bpf_trampoline_size at all.. and just teach __arch_prepare_bpf_trampoline to survive NULL image data and just return the size in such case
thanks,
jirka
---
11.11% test_progs [kernel.kallsyms] [k] decay_va_pool_node
|
---decay_va_pool_node
__purge_vmap_area_lazy
_vm_unmap_aliases
change_page_attr_set_clr
|
|--4.58%--set_memory_rw
| |
| |--2.33%--execmem_alloc_rw
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --2.25%--__execmem_cache_free
| execmem_free
| arch_bpf_trampoline_size
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_multi_attach
| bpf_tracing_multi_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| sys_bpf_fd
| bpf_link_create
| bpf_program__attach_tracing_multi
| serial_test_tracing_multi_bench_attach
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
|--4.52%--set_memory_nx
| |
| |--2.32%--execmem_alloc_rw
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --2.20%--__execmem_cache_free
| execmem_free
| arch_bpf_trampoline_size
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_multi_attach
| bpf_tracing_multi_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| sys_bpf_fd
| bpf_link_create
| bpf_program__attach_tracing_multi
| serial_test_tracing_multi_bench_attach
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--2.01%--set_memory_rox
|
--2.01%--__execmem_cache_free
execmem_free arch_bpf_trampoline_size bpf_trampoline_update __bpf_trampoline_link_prog bpf_trampoline_multi_attach bpf_tracing_multi_attach __sys_bpf
__x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall
sys_bpf_fd
bpf_link_create bpf_program__attach_tracing_multi serial_test_tracing_multi_bench_attach run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start
6.15% test_progs [kernel.kallsyms] [k] smp_call_function_many_cond
|
---smp_call_function_many_cond
on_each_cpu_cond_mask
|
|--4.80%--cpa_flush
| change_page_attr_set_clr
| |
| |--2.37%--set_memory_rox
| | __execmem_cache_free
| | execmem_free
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| |--1.21%--set_memory_nx
| | execmem_alloc_rw
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --1.21%--set_memory_rw
| execmem_alloc_rw
| |
| --1.21%--arch_bpf_trampoline_size
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_multi_attach
| bpf_tracing_multi_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| sys_bpf_fd
| bpf_link_create
| bpf_program__attach_tracing_multi
| serial_test_tracing_multi_bench_attach
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--1.21%--__change_page_attr_set_clr
change_page_attr_set_clr set_memory_nx
execmem_alloc_rw
|
--1.20%--arch_bpf_trampoline_size
bpf_trampoline_update __bpf_trampoline_link_prog bpf_trampoline_multi_attach bpf_tracing_multi_attach __sys_bpf
__x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall
sys_bpf_fd
bpf_link_create bpf_program__attach_tracing_multi serial_test_tracing_multi_bench_attach run_one_test
main __libc_start_call_main __libc_start_main@@GLIBC_2.34 _start
5.01% test_progs [kernel.kallsyms] [k] __raw_callee_save___pv_queued_spin_unlock
|
---__raw_callee_save___pv_queued_spin_unlock
do_raw_spin_unlock
|
|--4.20%--_raw_spin_unlock
| |
| --4.10%--decay_va_pool_node
| __purge_vmap_area_lazy
| _vm_unmap_aliases
| change_page_attr_set_clr
| |
| |--1.73%--set_memory_nx
| | |
| | |--0.95%--__execmem_cache_free
| | | execmem_free
| | | arch_bpf_trampoline_size
| | | bpf_trampoline_update
| | | __bpf_trampoline_link_prog
| | | bpf_trampoline_multi_attach
| | | bpf_tracing_multi_attach
| | | __sys_bpf
| | | __x64_sys_bpf
| | | do_syscall_64
| | | entry_SYSCALL_64_after_hwframe
| | | syscall
| | | sys_bpf_fd
| | | bpf_link_create
| | | bpf_program__attach_tracing_multi
| | | serial_test_tracing_multi_bench_attach
| | | run_one_test
| | | main
| | | __libc_start_call_main
| | | __libc_start_main@@GLIBC_2.34
| | | _start
| | |
| | --0.78%--execmem_alloc_rw
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| |--1.51%--set_memory_rw
| | |
| | |--0.76%--__execmem_cache_free
| | | execmem_free
| | | arch_bpf_trampoline_size
| | | bpf_trampoline_update
| | | __bpf_trampoline_link_prog
| | | bpf_trampoline_multi_attach
| | | bpf_tracing_multi_attach
| | | __sys_bpf
| | | __x64_sys_bpf
| | | do_syscall_64
| | | entry_SYSCALL_64_after_hwframe
| | | syscall
| | | sys_bpf_fd
| | | bpf_link_create
| | | bpf_program__attach_tracing_multi
| | | serial_test_tracing_multi_bench_attach
| | | run_one_test
| | | main
| | | __libc_start_call_main
| | | __libc_start_main@@GLIBC_2.34
| | | _start
| | |
| | --0.75%--execmem_alloc_rw
| | arch_bpf_trampoline_size
| | bpf_trampoline_update
| | __bpf_trampoline_link_prog
| | bpf_trampoline_multi_attach
| | bpf_tracing_multi_attach
| | __sys_bpf
| | __x64_sys_bpf
| | do_syscall_64
| | entry_SYSCALL_64_after_hwframe
| | syscall
| | sys_bpf_fd
| | bpf_link_create
| | bpf_program__attach_tracing_multi
| | serial_test_tracing_multi_bench_attach
| | run_one_test
| | main
| | __libc_start_call_main
| | __libc_start_main@@GLIBC_2.34
| | _start
| |
| --0.86%--set_memory_rox
| __execmem_cache_free
| execmem_free
| arch_bpf_trampoline_size
| bpf_trampoline_update
| __bpf_trampoline_link_prog
| bpf_trampoline_multi_attach
| bpf_tracing_multi_attach
| __sys_bpf
| __x64_sys_bpf
| do_syscall_64
| entry_SYSCALL_64_after_hwframe
| syscall
| sys_bpf_fd
| bpf_link_create
| bpf_program__attach_tracing_multi
| serial_test_tracing_multi_bench_attach
| run_one_test
| main
| __libc_start_call_main
| __libc_start_main@@GLIBC_2.34
| _start
|
--0.80%--_raw_spin_unlock_irqrestore
4.88% test_progs [kernel.kallsyms] [k] get_symbol_offset
Re: [PATCH bpf-next v3 4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
Hi,
Quoted 48 lines · Jiri Olsa
Quoted 26 lines · Mike Rapoport (Microsoft)
arch_bpf_trampoline_size()allocates a buffer to get actual size required for a trampoline.This buffer must be in the module address space because
__arch_prepare_bpf_trampoline()calculates rel32 offsets relatively to that buffer.In preparation for enabling ROX mode for
EXECMEM_BPFmake sure that the allocated memory is writable.Add
bpf_jit_alloc_exec_rw()wrapper forexecmem_alloc_rw()and use it for > buffer allocation inarch_bpf_trampoline_size().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +0
-0@@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,int ret;arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 3711
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2
-3/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* This will NOT cause fragmentation in direct map, as we do not -* call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. +* Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image =bpf_jit_alloc_exec (PAGE_SIZE ); + image =bpf_jit_alloc_exec_rw (PAGE_SIZE );hi,
this change (this particular patch plus possibly others in this set) is causingtracing_multiattachment bench slowdownthe benchmark allocates huge number of trampolines and I'm seeing extra
arch_bpf_trampoline_sizecode paths in the attached perf profileI'm not that familiar with the allocator, but following hack makes the benchmark ok again:
Quoted 10 lines
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6a94370a2448..bbff3c9c6681 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1130,7 +1130,7 @@ void *bpf_jit_alloc_exec(unsigned long size)
void *bpf_jit_alloc_exec_rw(unsigned long size)
{
- return execmem_alloc_rw(EXECMEM_BPF, size);
+ return execmem_alloc(EXECMEM_MODULE_DATA, size);This is fine on x86 because all execmem types live in the same 2G range. Other architectures may have _BPF and MODULE_DATA in different ranges, e.g arm64 and some configurations of powerpc.
Quoted 9 lines
} void bpf_jit_free_exec(void *addr)I still need to do more checks, but I'm wondering if we could actually fix this by not allocating image data in
arch_bpf_trampoline_sizeat all.. and just teach__arch_prepare_bpf_trampolineto survive NULL image data and just return the size in such case
Since bpf_jit_alloc_exec_rw() is only used by x86::arch_bpf_trampoline_size() I think we can directly allocate from _MODULE_DATA there and drop bpf_jit_alloc_exec_rw(), like (build tested only) patch below does.
If/when other architectures would implement arch_bpf_trampoline_size(), they'll probably need to deal with this differently.
From 14dff78529a9204408a5c14036954df5f2f76431 Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (Microsoft)" <[email protected]>
Date: Sun, 16 Aug 2026 11:27:32 +0300
Subject: [PATCH] x86/bpf: make arch_bpf_trampoline_size allocate from
EXECMEM_MODULE_DATA
Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1].
The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free().
With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX.
The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space.
On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA)
Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper.
--- arch/x86/net/bpf_jit_comp.c | 8 +++++--- include/linux/filter.h | 1 - kernel/bpf/core.c | 5 ----- 3 files changed, 5 insertions(+), 9 deletions(-)
arch/x86/net/bpf_jit_comp.c +5 -3
@@ -13,6 +13,7 @@#include <linux/bpf_verifier.h> #include <linux/memory.h> #include <linux/sort.h> +#include <linux/execmem.h> #include <asm/extable.h> #include <asm/ftrace.h> #include <asm/set_memory.h>
@@ -3706,15 +3707,16 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,* * We cannot use kvmalloc here, because we need image to be in * module memory range. -* Since it must be writable use bpf_jit_alloc_exec_rw(). +* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) +* that returns writable memory in the module address space. */ - image =bpf_jit_alloc_exec_rw (PAGE_SIZE ); + image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );if (!image)return -ENOMEM ; ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, m, flags, tnodes, func_addr); -bpf_jit_free_exec (image); +execmem_free (image);return ret; }
include/linux/filter.h +0 -1
@@ -1333,7 +1333,6 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,void bpf_jit_binary_free (struct bpf_binary_header *hdr);u64 bpf_jit_alloc_exec_limit (void );void *bpf_jit_alloc_exec (unsigned long size); -void *bpf_jit_alloc_exec_rw (unsigned long size);void bpf_jit_free_exec (void *addr);void bpf_jit_free (struct bpf_prog *fp); struct bpf_binary_header *
kernel/bpf/core.c +0 -5
@@ -1128,11 +1128,6 @@ void *bpf_jit_alloc_exec(unsigned long size)return execmem_alloc (EXECMEM_BPF , size); } -void *bpf_jit_alloc_exec_rw (unsigned long size) -{ -return execmem_alloc_rw (EXECMEM_BPF , size); -} -void bpf_jit_free_exec (void *addr) {execmem_free (addr);
Signature · 1 line
2.53.0
Quoted 2 lines
thanks,
jirka
Signature · 2 lines
Sincerely yours, Mike.
Jiri Olsa
Tested-byRe: [PATCH bpf-next v3 4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
On Sun, Aug 16, 2026 at 11:57:35AM +0300, Mike Rapoport wrote:
SNIP
Quoted 23 lines
From
14dff78529a9204408a5c14036954df5f2f76431Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (Microsoft)" <[email protected]>
Date: Sun, 16 Aug 2026 11:27:32 +0300
Subject: [PATCH] x86/bpf: makearch_bpf_trampoline_sizeallocate from
EXECMEM_MODULE_DATAJiri Olsa reports slowdown of
tracing_multibenchmark that allocates huge number of trampolines [1].The slowdown caused by extra protection changes in
execmem_alloc_rw()andexecmem_free().With ROX caches enabled, all execmem allocations except
EXECMEM_MODULE_DATAare ROX after the allocation.execmem_alloc_rw()temporarily sets them to W+NX andexecmem_free()resets them back to ROX.The only user of
bpf_jit_alloc_exec_rw()isx86::arch_bpf_trampoline_size()that only needs a temporary writable buffer in the modules address space.On x86 executable memory and module data are constrained to the same address range, so
x86::arch_bpf_trampoline_size()can directly useexecmem_alloc(EXECMEM_MODULE_DATA)
Quoted 8 lines
Replace the call to
bpf_jit_alloc_exec_rw()with a call toexecmem_alloc(EXECMEM_MODULE_DATA)inx86::arch_bpf_trampoline_size()and dropbpf_jit_alloc_exec_rw()helper.Reported-by Jiri Olsa <[email protected]>Fixes f0334294a428 ("bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable")Signed-off-by Mike Rapoport (Microsoft) <[email protected]>
looks good, thanks
serial_test_tracing_multi_bench_attach: found 55077 functionsserial_test_tracing_multi_bench_attach: attached in 1.470sserial_test_tracing_multi_bench_attach: detached in 0.249s
jirka
Quoted 76 lines
--- arch/x86/net/bpf_jit_comp.c | 8 +++++--- include/linux/filter.h | 1 - kernel/bpf/core.c | 5 ----- 3 files changed, 5 insertions(+), 9 deletions(-)arch/x86/net/bpf_jit_comp.c +5
-3@@ -13,6 +13,7 @@#include <linux/bpf_verifier.h> #include <linux/memory.h> #include <linux/sort.h> +#include <linux/execmem.h> #include <asm/extable.h> #include <asm/ftrace.h> #include <asm/set_memory.h> @@ -3706,15 +3707,16 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,* * We cannot use kvmalloc here, because we need image to be in * module memory range. -* Since it must be writable use bpf_jit_alloc_exec_rw(). +* Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) +* that returns writable memory in the module address space. */ - image =bpf_jit_alloc_exec_rw (PAGE_SIZE ); + image =execmem_alloc (EXECMEM_MODULE_DATA ,PAGE_SIZE );if (!image)return -ENOMEM ; ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, m, flags, tnodes, func_addr); -bpf_jit_free_exec (image); +execmem_free (image);return ret; }include/linux/filter.h +0
-1@@ -1333,7 +1333,6 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,void bpf_jit_binary_free (struct bpf_binary_header *hdr);u64 bpf_jit_alloc_exec_limit (void );void *bpf_jit_alloc_exec (unsigned long size); -void *bpf_jit_alloc_exec_rw (unsigned long size);void bpf_jit_free_exec (void *addr);void bpf_jit_free (struct bpf_prog *fp); struct bpf_binary_header *kernel/bpf/core.c +0
-5@@ -1128,11 +1128,6 @@ void *bpf_jit_alloc_exec(unsigned long size)return execmem_alloc (EXECMEM_BPF , size); } -void *bpf_jit_alloc_exec_rw (unsigned long size) -{ -return execmem_alloc_rw (EXECMEM_BPF , size); -} -void bpf_jit_free_exec (void *addr) {execmem_free (addr);Signature · 1 line
2.53.0Quoted 2 lines
thanks,
jirkaSignature · 2 lines
Sincerely yours, Mike.
Daniel Borkmann
maintainerRe: [PATCH bpf-next v3 4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
In reply to Mike RapoportQuoted 74 lines · Mike Rapoport
Quoted 48 lines · Jiri Olsa
Quoted 26 lines · Mike Rapoport (Microsoft)
arch_bpf_trampoline_size()allocates a buffer to get actual size required for a trampoline.This buffer must be in the module address space because
__arch_prepare_bpf_trampoline()calculates rel32 offsets relatively to that buffer.In preparation for enabling ROX mode for
EXECMEM_BPFmake sure that the allocated memory is writable.Add
bpf_jit_alloc_exec_rw()wrapper forexecmem_alloc_rw()and use it for > buffer allocation inarch_bpf_trampoline_size().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +0
-0@@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,int ret;arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 3711
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2
-3/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* This will NOT cause fragmentation in direct map, as we do not -* call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. +* Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image =bpf_jit_alloc_exec (PAGE_SIZE ); + image =bpf_jit_alloc_exec_rw (PAGE_SIZE );hi,
this change (this particular patch plus possibly others in this set) is causingtracing_multiattachment bench slowdownthe benchmark allocates huge number of trampolines and I'm seeing extra
arch_bpf_trampoline_sizecode paths in the attached perf profileI'm not that familiar with the allocator, but following hack makes the benchmark ok again:
Quoted 10 lines
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6a94370a2448..bbff3c9c6681 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1130,7 +1130,7 @@ void *bpf_jit_alloc_exec(unsigned long size) void *bpf_jit_alloc_exec_rw(unsigned long size) { - return execmem_alloc_rw(EXECMEM_BPF, size); + return execmem_alloc(EXECMEM_MODULE_DATA, size);This is fine on x86 because all execmem types live in the same 2G range. Other architectures may have
_BPFandMODULE_DATAin different ranges, e.g arm64 and some configurations of powerpc.Quoted 9 lines
} void bpf_jit_free_exec(void *addr)I still need to do more checks, but I'm wondering if we could actually fix this by not allocating image data in
arch_bpf_trampoline_sizeat all.. and just teach__arch_prepare_bpf_trampolineto survive NULL image data and just return the size in such case
Quoted 7 lines
Since
bpf_jit_alloc_exec_rw()is only used byx86::arch_bpf_trampoline_size()I think we can directly allocate from_MODULE_DATAthere and dropbpf_jit_alloc_exec_rw(), like (build tested only) patch below does.If/when other architectures would implement
arch_bpf_trampoline_size(), they'll probably need to deal with this differently.
Mike, could you send this as a regular patch to bpf list, so it can go through BPF CI?
Thanks,
Daniel
Re: [PATCH bpf-next v3 4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
In reply to Jiri OlsaQuoted 63 lines · Jiri Olsa
Quoted 26 lines · Mike Rapoport (Microsoft)
arch_bpf_trampoline_size()allocates a buffer to get actual size required for a trampoline.This buffer must be in the module address space because
__arch_prepare_bpf_trampoline()calculates rel32 offsets relatively to that buffer.In preparation for enabling ROX mode for
EXECMEM_BPFmake sure that the allocated memory is writable.Add
bpf_jit_alloc_exec_rw()wrapper forexecmem_alloc_rw()and use it for > buffer allocation inarch_bpf_trampoline_size().Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +0
-0@@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,int ret;arch/x86/net/bpf_jit_comp.c int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · line 3711
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2
-3/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* This will NOT cause fragmentation in direct map, as we do not -* call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. +* Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image =bpf_jit_alloc_exec (PAGE_SIZE ); + image =bpf_jit_alloc_exec_rw (PAGE_SIZE );hi,
this change (this particular patch plus possibly others in this set) is causingtracing_multiattachment bench slowdownthe benchmark allocates huge number of trampolines and I'm seeing extra
arch_bpf_trampoline_sizecode paths in the attached perf profileI'm not that familiar with the allocator, but following hack makes the benchmark ok again:
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6a94370a2448..bbff3c9c6681 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1130,7 +1130,7 @@ void *bpf_jit_alloc_exec(unsigned long size) void *bpf_jit_alloc_exec_rw(unsigned long size) { - return execmem_alloc_rw(EXECMEM_BPF, size); + return execmem_alloc(EXECMEM_MODULE_DATA, size); } void bpf_jit_free_exec(void *addr)
Quoted 4 lines
I still need to do more checks, but I'm wondering if we could actually fix this by not allocating image data in
arch_bpf_trampoline_sizeat all.. and just teach__arch_prepare_bpf_trampolineto survive NULL image data and just return the size in such case
fyi doing ^^^ is not so straigh forward as I hoped for, but if we just allocate dummy page at init and used that in arch_bpf_trampoline_size we get some speedup
I'll rebase that on top of your fix and send it out later
with your fix:
serial_test_tracing_multi_bench_attach: found 55077 functions
serial_test_tracing_multi_bench_attach: attached in 1.470s
serial_test_tracing_multi_bench_attach: detached in 0.249s
with change below:
serial_test_tracing_multi_bench_attach: found 55079 functions
serial_test_tracing_multi_bench_attach: attached in 0.754s
serial_test_tracing_multi_bench_attach: detached in 0.247s
thanks,
jirka
---
arch/x86/net/bpf_jit_comp.c +18 -16
@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bitfield.h> +#include <linux/init.h> #include <linux/bpf.h> #include <linux/bpf_verifier.h> #include <linux/memory.h>
@@ -23,6 +24,19 @@static bool all_callee_regs_used[4 ] = {true, true, true, true}; +/* +* Reuse a writable image in the BPF execmem range for size calculation. +* Its contents do not affect size calculation. +*/ +static void *trampoline_size_image; + +static int __init init_trampoline_size_image (void ) +{ + trampoline_size_image =bpf_jit_alloc_exec_rw (PAGE_SIZE ); +return trampoline_size_image ?0 : -ENOMEM ; +} +late_initcall (init_trampoline_size_image); +static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {if (len ==1 )
@@ -3811,23 +3825,11 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) { struct bpf_tramp_image im; -void *image; -int ret; - -/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). -* -* We cannot use kvmalloc here, because we need image to be in -* module memory range. -* Since it must be writable use bpf_jit_alloc_exec_rw(). -*/ - image =bpf_jit_alloc_exec_rw (PAGE_SIZE ); -if (!image) -return -ENOMEM ; - ret =__arch_prepare_bpf_trampoline (&im, image, image +PAGE_SIZE , image, - m, flags, tnodes, func_addr); -bpf_jit_free_exec (image); -return ret; +return __arch_prepare_bpf_trampoline (&im, trampoline_size_image, + trampoline_size_image +PAGE_SIZE , + trampoline_size_image, m, flags, + tnodes, func_addr); }static int emit_bpf_dispatcher (u8 **pprog,int a,int b,s64 *progs,u8 *image,u8 *buf)
PATCH v3 5/5 x86/bpf: enable EXECMEM_ROX_CACHE for BPF allocations
In reply to Mike Rapoport (Microsoft)BPF core and x86 JIT use text poking and temporary writable buffers and thus can handle ROX memory.
Enable ROX cache for EXECMEM_BPF when configuration and CPU features allow that.
--- arch/x86/mm/init.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/init.c +2 -2
@@ -1107,10 +1107,10 @@ struct execmem_info __init *execmem_arch_setup(void) .alignment = MODULE_ALIGN, }, [EXECMEM_BPF] = { - .flags = EXECMEM_KASAN_SHADOW, + .flags = flags, .start = start, .end = MODULES_END, - .pgprot = PAGE_KERNEL, + .pgprot = pgprot, .alignment = MODULE_ALIGN, }, [EXECMEM_MODULE_DATA] = {
Signature · 1 line
2.53.0
Song Liu
reviewerQuoted 7 lines · Mike Rapoport (Microsoft)
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call
bpf_jit_alloc_exec()immediately make it ROX withset_memory_rox().
Quoted 5 lines
The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with
text_poke().These patches do several small adjustments to how BPF allocates executable memory and enable
EXECMEM_ROX_CACHEfor BPF allocations on x86.
After this set, we are still using bpf_prog_pack_alloc() from x86 code. I think the goal is to eventually remove bpf_prog_pack_alloc(). What's our plan for the next steps (toward removing bpf_prog_pack_alloc)?
Thanks,
Song
Quoted 12 lines · Song Liu
Quoted 10 lines · Mike Rapoport (Microsoft)
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call
bpf_jit_alloc_exec()immediately make it ROX withset_memory_rox().The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with
text_poke().
Quoted 6 lines
Quoted 2 lines
These patches do several small adjustments to how BPF allocates executable memory and enable
EXECMEM_ROX_CACHEfor BPF allocations on x86.After this set, we are still using
bpf_prog_pack_alloc()from x86 code. I think the goal is to eventually removebpf_prog_pack_alloc(). What's our plan for the next steps (toward removingbpf_prog_pack_alloc)?
"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Since BPF is the only user the easiest would be just to move prog_pack logic from BPF to execmem and call it a day.
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid BPF_PROG_CHUNK_SIZE.
Maybe it would be also possible to teach SLUB to use execmem_alloc() instead of alloc_pages() but that's surely the most far fetched one :)
And since we are talking about bpf_prog_pack_alloc(), why BPF_PROG_PACK_SIZE accounts for num_possible_nodes():
#define BPF_PROG_PACK_SIZE (SZ_2M * num_possible_nodes())
Is it an elaborate choice or it was picked to work around older vmalloc_huge() limitations?
Quoted 2 lines
Thanks,
Song
Signature · 2 lines
Sincerely yours, Mike.
Song Liu
reviewerQuoted 17 lines · Mike Rapoport
Quoted 15 lines · Song Liu
Quoted 12 lines · Mike Rapoport (Microsoft)
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call
bpf_jit_alloc_exec()immediately make it ROX withset_memory_rox().The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with
text_poke().These patches do several small adjustments to how BPF allocates executable memory and enable
EXECMEM_ROX_CACHEfor BPF allocations on x86.
Quoted 7 lines
Quoted 3 lines
After this set, we are still using
bpf_prog_pack_alloc()from x86 code. I think the goal is to eventually removebpf_prog_pack_alloc(). What's our plan for the next steps (toward removingbpf_prog_pack_alloc)?"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Sub-page allocation is not a hard requirement here. Using 4kB for each small BPF program isn't too bad. We added bpf_prog_pack to avoid fragmentation of direct map page table entry (caused by W^X requirement). If execmem allocator reserves large enough ROX memory (with PMD page table entries) and reuses them properly, we shouldn't see page table fragmentation getting worse over time. Then, we can use 4kB granularity allocation for BPF programs. (I am not sure about 64kB pages..).
Quoted 3 lines
Since BPF is the only user the easiest would be just to move
prog_packlogic from BPF to execmem and call it a day.
If we move to bigger page sizes, say 64kB, there will be other users that would benefit from sub page allocation, right?
Quoted 6 lines
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid
BPF_PROG_CHUNK_SIZE.Maybe it would be also possible to teach SLUB to use
execmem_alloc()instead ofalloc_pages()but that's surely the most far fetched one :)
I was thinking some rb-tree algorithm might be useful here, something similar to vmap.
Quoted 7 lines
And since we are talking about
bpf_prog_pack_alloc(), whyBPF_PROG_PACK_SIZEaccounts fornum_possible_nodes():#define
BPF_PROG_PACK_SIZE(SZ_2M*num_possible_nodes())Is it an elaborate choice or it was picked to work around older
vmalloc_huge()limitations?
It is a bit complicated. The goal is to get PMDs for prog_pack. We can adjust this if vmalloc_huge() changes after that.
Thanks,
Song
Quoted 25 lines · Song Liu
Quoted 16 lines · Mike Rapoport
Quoted 15 lines · Song Liu
Quoted 12 lines · Mike Rapoport (Microsoft)
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call
bpf_jit_alloc_exec()immediately make it ROX withset_memory_rox().The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with
text_poke().These patches do several small adjustments to how BPF allocates executable memory and enable
EXECMEM_ROX_CACHEfor BPF allocations on x86.Quoted 7 lines
Quoted 3 lines
After this set, we are still using
bpf_prog_pack_alloc()from x86 code. I think the goal is to eventually removebpf_prog_pack_alloc(). What's our plan for the next steps (toward removingbpf_prog_pack_alloc)?"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Quoted 8 lines
Sub-page allocation is not a hard requirement here. Using 4kB for each small BPF program isn't too bad. We added
bpf_prog_packto avoid fragmentation of direct map page table entry (caused by W^X requirement). If execmem allocator reserves large enough ROX memory (with PMD page table entries) and reuses them properly, we shouldn't see page table fragmentation getting worse over time. Then, we can use 4kB granularity allocation for BPF programs. (I am not sure about 64kB pages..).
execmem uses PMD_SIZE pages for ROX cache. Since right now it's only supported on x86, the issue with oversized large page with 64k base pages didn't come up yet.
Quoted 5 lines
Quoted 2 lines
Since BPF is the only user the easiest would be just to move
prog_packlogic from BPF to execmem and call it a day.If we move to bigger page sizes, say 64kB, there will be other users that would benefit from sub page allocation, right?
Maybe modules could, don't know TBH.
There is another caveat with sub-page allocations: they must preserve ROX and no subsystem except BPF can deal with ROX-only allocations and writable copies.
Quoted 9 lines
Quoted 6 lines
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid
BPF_PROG_CHUNK_SIZE.Maybe it would be also possible to teach SLUB to use
execmem_alloc()instead ofalloc_pages()but that's surely the most far fetched one :)I was thinking some rb-tree algorithm might be useful here, something similar to vmap.
Could be that or maple_tree especially considering there's an active discussion about converting vmalloc to maple_tree as well:
https://lore.kernel.org/linux-mm/[email protected]/
But the choice of allocation algorithm is anyway secondary to the decision whether execmem should have the sub-page allocator.
If you say that a page per BPF program is fine, than maybe we can just rip off bpf_prog_pack_alloc().
If BPF still needs to maintain the ability to allocate in smaller chunks, then maybe the best way is to keep bpf_prog_pack_alloc() as is because I don't foresee other users for small page allocations any time soon.
Quoted 10 lines
Quoted 7 lines
And since we are talking about
bpf_prog_pack_alloc(), whyBPF_PROG_PACK_SIZEaccounts fornum_possible_nodes():#define
BPF_PROG_PACK_SIZE(SZ_2M*num_possible_nodes())Is it an elaborate choice or it was picked to work around older
vmalloc_huge()limitations?It is a bit complicated. The goal is to get PMDs for
prog_pack. We can adjust this ifvmalloc_huge()changes after that.
Since commit c82be0be9576 ("mm: vmalloc: don't account for number of nodes for HUGE_VMAP allocations") vmalloc_huge() is fine with PMD_SIZE regardless of number of nodes.
With 64k pages it's still a lot though.
Quoted 2 lines
Thanks,
Song
Signature · 2 lines
Sincerely yours, Mike.
Song Liu
reviewerOn Fri, Jul 17, 2026 at 2:29 AM Mike Rapoport <[email protected]> wrote:
[...]
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Quoted 8 lines
Sub-page allocation is not a hard requirement here. Using 4kB for each small BPF program isn't too bad. We added
bpf_prog_packto avoid fragmentation of direct map page table entry (caused by W^X requirement). If execmem allocator reserves large enough ROX memory (with PMD page table entries) and reuses them properly, we shouldn't see page table fragmentation getting worse over time. Then, we can use 4kB granularity allocation for BPF programs. (I am not sure about 64kB pages..).
Quoted 3 lines
execmem uses
PMD_SIZEpages for ROX cache. Since right now it's only supported on x86, the issue with oversized large page with 64k base pages didn't come up yet.
With 64kB page size, PMD_SIZE is probably too big for execmem.
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
Since BPF is the only user the easiest would be just to move
prog_packlogic from BPF to execmem and call it a day.If we move to bigger page sizes, say 64kB, there will be other users that would benefit from sub page allocation, right?
Maybe modules could, don't know TBH.
There is another caveat with sub-page allocations: they must preserve ROX and no subsystem except BPF can deal with ROX-only allocations and writable copies.
I think ftrace and kprobe could be good users?
Quoted 32 lines
Quoted 9 lines
Quoted 6 lines
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid
BPF_PROG_CHUNK_SIZE.Maybe it would be also possible to teach SLUB to use
execmem_alloc()instead ofalloc_pages()but that's surely the most far fetched one :)I was thinking some rb-tree algorithm might be useful here, something similar to vmap.
Could be that or
maple_treeespecially considering there's an active discussion about converting vmalloc tomaple_treeas well:https://lore.kernel.org/linux-mm/[email protected]/
But the choice of allocation algorithm is anyway secondary to the decision whether execmem should have the sub-page allocator.
If you say that a page per BPF program is fine, than maybe we can just rip off
bpf_prog_pack_alloc().
If BPF still needs to maintain the ability to allocate in smaller chunks, then maybe the best way is to keepbpf_prog_pack_alloc()as is because I don't foresee other users for small page allocations any time soon.Quoted 8 lines
Quoted 7 lines
And since we are talking about
bpf_prog_pack_alloc(), whyBPF_PROG_PACK_SIZEaccounts fornum_possible_nodes():#define
BPF_PROG_PACK_SIZE(SZ_2M*num_possible_nodes())Is it an elaborate choice or it was picked to work around older
vmalloc_huge()limitations?
Quoted 6 lines
Quoted 2 lines
It is a bit complicated. The goal is to get PMDs for
prog_pack. We can adjust this ifvmalloc_huge()changes after that.Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.
Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let bpf_prog_pack handle sub page allocations. Then, BPF_PROG_PACK_SIZE will be PAGE_SIZE. This should be a net win for x86_64. Other archs will be the same. WDYT?
Thanks,
Song
Quoted 16 lines · Song Liu
On Fri, Jul 17, 2026 at 2:29 AM Mike Rapoport <[email protected]> wrote:
[...]Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Sub-page allocation is not a hard requirement here. Using 4kB for each small BPF program isn't too bad. We added
bpf_prog_packto avoid fragmentation of direct map page table entry (caused by W^X requirement). If execmem allocator reserves large enough ROX memory (with PMD page table entries) and reuses them properly, we shouldn't see page table fragmentation getting worse over time. Then, we can use 4kB granularity allocation for BPF programs. (I am not sure about 64kB pages..).
Quoted 5 lines
Quoted 3 lines
execmem uses
PMD_SIZEpages for ROX cache. Since right now it's only supported on x86, the issue with oversized large page with 64k base pages didn't come up yet.With 64kB page size,
PMD_SIZEis probably too big for execmem.
Yeah, I think so. But maybe on arm64 with 64kB pages we could use CONT_PTE.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Since BPF is the only user the easiest would be just to move
prog_packlogic from BPF to execmem and call it a day.If we move to bigger page sizes, say 64kB, there will be other users that would benefit from sub page allocation, right?
Quoted 6 lines
Quoted 4 lines
Maybe modules could, don't know TBH.
There is another caveat with sub-page allocations: they must preserve ROX and no subsystem except BPF can deal with ROX-only allocations and writable copies.I think ftrace and kprobe could be good users?
Don't know about ftrace, but kprobes essentially has page-sized caches and uses them internally.
Quoted 39 lines
Quoted 32 lines
Quoted 9 lines
Quoted 6 lines
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid
BPF_PROG_CHUNK_SIZE.Maybe it would be also possible to teach SLUB to use
execmem_alloc()instead ofalloc_pages()but that's surely the most far fetched one :)I was thinking some rb-tree algorithm might be useful here, something similar to vmap.
Could be that or
maple_treeespecially considering there's an active discussion about converting vmalloc tomaple_treeas well:https://lore.kernel.org/linux-mm/[email protected]/
But the choice of allocation algorithm is anyway secondary to the decision whether execmem should have the sub-page allocator.
If you say that a page per BPF program is fine, than maybe we can just rip off
bpf_prog_pack_alloc().
If BPF still needs to maintain the ability to allocate in smaller chunks, then maybe the best way is to keepbpf_prog_pack_alloc()as is because I don't foresee other users for small page allocations any time soon.Quoted 8 lines
Quoted 7 lines
And since we are talking about
bpf_prog_pack_alloc(), whyBPF_PROG_PACK_SIZEaccounts fornum_possible_nodes():#define
BPF_PROG_PACK_SIZE(SZ_2M*num_possible_nodes())Is it an elaborate choice or it was picked to work around older
vmalloc_huge()limitations?Quoted 6 lines
Quoted 2 lines
It is a bit complicated. The goal is to get PMDs for
prog_pack. We can adjust this ifvmalloc_huge()changes after that.Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.
Quoted 5 lines
Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let
bpf_prog_packhandle sub page allocations. Then,BPF_PROG_PACK_SIZEwill bePAGE_SIZE. This should be a net win forx86_64. Other archs will be the same. WDYT?
Makes perfect sense to me for x86.
On architectures that support large mappings in the direct map and do not support EXECMEM_CACHE_ROX this will increase direct map fragmentation.
Quoted 2 lines
Thanks,
Song
Signature · 2 lines
Sincerely yours, Mike.
Song Liu
reviewerQuoted 81 lines · Mike Rapoport
Quoted 16 lines · Song Liu
On Fri, Jul 17, 2026 at 2:29 AM Mike Rapoport <[email protected]> wrote:
[...]Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
"It works, don't touch"? ;-)
We can add another layer for sub-page allocations to execmem.
Sub-page allocation is not a hard requirement here. Using 4kB for each small BPF program isn't too bad. We added
bpf_prog_packto avoid fragmentation of direct map page table entry (caused by W^X requirement). If execmem allocator reserves large enough ROX memory (with PMD page table entries) and reuses them properly, we shouldn't see page table fragmentation getting worse over time. Then, we can use 4kB granularity allocation for BPF programs. (I am not sure about 64kB pages..).Quoted 5 lines
Quoted 3 lines
execmem uses
PMD_SIZEpages for ROX cache. Since right now it's only supported on x86, the issue with oversized large page with 64k base pages didn't come up yet.With 64kB page size,
PMD_SIZEis probably too big for execmem.Yeah, I think so. But maybe on arm64 with 64kB pages we could use
CONT_PTE.Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Since BPF is the only user the easiest would be just to move
prog_packlogic from BPF to execmem and call it a day.If we move to bigger page sizes, say 64kB, there will be other users that would benefit from sub page allocation, right?
Quoted 6 lines
Quoted 4 lines
Maybe modules could, don't know TBH.
There is another caveat with sub-page allocations: they must preserve ROX and no subsystem except BPF can deal with ROX-only allocations and writable copies.I think ftrace and kprobe could be good users?
Don't know about ftrace, but kprobes essentially has page-sized caches and uses them internally.
Quoted 39 lines
Quoted 38 lines
Quoted 9 lines
Quoted 6 lines
Another option is to add a slab-like layer for sub-page allocations to execmem. This is more complex but it would allow to get rid of the rigid
BPF_PROG_CHUNK_SIZE.Maybe it would be also possible to teach SLUB to use
execmem_alloc()instead ofalloc_pages()but that's surely the most far fetched one :)I was thinking some rb-tree algorithm might be useful here, something similar to vmap.
Could be that or
maple_treeespecially considering there's an active discussion about converting vmalloc tomaple_treeas well:https://lore.kernel.org/linux-mm/[email protected]/
But the choice of allocation algorithm is anyway secondary to the decision whether execmem should have the sub-page allocator.
If you say that a page per BPF program is fine, than maybe we can just rip off
bpf_prog_pack_alloc().
If BPF still needs to maintain the ability to allocate in smaller chunks, then maybe the best way is to keepbpf_prog_pack_alloc()as is because I don't foresee other users for small page allocations any time soon.Quoted 10 lines
Quoted 7 lines
And since we are talking about
bpf_prog_pack_alloc(), whyBPF_PROG_PACK_SIZEaccounts fornum_possible_nodes():#define
BPF_PROG_PACK_SIZE(SZ_2M*num_possible_nodes())Is it an elaborate choice or it was picked to work around older
vmalloc_huge()limitations?It is a bit complicated. The goal is to get PMDs for
prog_pack. We can adjust this ifvmalloc_huge()changes after that.Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.
Quoted 7 lines
Quoted 5 lines
Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let
bpf_prog_packhandle sub page allocations. Then,BPF_PROG_PACK_SIZEwill bePAGE_SIZE. This should be a net win forx86_64. Other archs will be the same. WDYT?Makes perfect sense to me for x86.
Can we include this change to this patch set? This will make this change a meaningful step towards migrating to EXECMEM.
Quoted 2 lines
On architectures that support large mappings in the direct map and do not support
EXECMEM_CACHE_ROXthis will increase direct map fragmentation.
Architectures that support bpf_prog_pack can benefit from EXECMEM_CACHE_ROX. What does it take to also enable EXECMEM_CACHE_ROX for them?
Thanks,
Song
Quoted 12 lines · Song Liu
Quoted 11 lines · Mike Rapoport
Quoted 10 lines
Quoted 4 lines
Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let
bpf_prog_packhandle sub page allocations. Then,BPF_PROG_PACK_SIZEwill bePAGE_SIZE. This should be a net win forx86_64. Other archs will be the same. WDYT?
Quoted 4 lines
Quoted 1 line
Makes perfect sense to me for x86.
Can we include this change to this patch set? This will make this change a meaningful step towards migrating to EXECMEM.
I'd prefer to leave this as a separate followup.
The decision whether to use EXECMEM_ROX_CACHE is made at runtime, which means that BPF_PROG_PACK_SIZE should become a variable, that variable needs initialization at some point during boot and that in turn requires a new helper in execmem that will say if EXECMEM_CACHE_ROX is enabled.
Quoted 6 lines
Quoted 2 lines
On architectures that support large mappings in the direct map and do not support
EXECMEM_CACHE_ROXthis will increase direct map fragmentation.Architectures that support
bpf_prog_packcan benefit fromEXECMEM_CACHE_ROX. What does it take to also enableEXECMEM_CACHE_ROXfor them?
For EXECMEM_ROX_CACHE to help in reducing the fragmentation and TLB pressure an architecture should support collapsing of the split mappings at least in vmalloc/modules space.
There is an RFC for arm64 to add support for EXECMEM_ROX_CACHE there:
https://lore.kernel.org/all/[email protected]
and it looks like it'll take a couple of iterations :)
I don't how much work it would be for other architectures.
Quoted 2 lines
Thanks,
Song
Signature · 2 lines
Sincerely yours, Mike.
Quoted 19 lines · Mike Rapoport
Quoted 12 lines · Song Liu
Quoted 11 lines · Mike Rapoport
Quoted 10 lines
Quoted 4 lines
Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let
bpf_prog_packhandle sub page allocations. Then,BPF_PROG_PACK_SIZEwill bePAGE_SIZE. This should be a net win forx86_64. Other archs will be the same. WDYT?Quoted 4 lines
Quoted 1 line
Makes perfect sense to me for x86.
Can we include this change to this patch set? This will make this change a meaningful step towards migrating to EXECMEM.
Quoted 6 lines
I'd prefer to leave this as a separate followup.
The decision whether to use
EXECMEM_ROX_CACHEis made at runtime, which means thatBPF_PROG_PACK_SIZEshould become a variable, that variable needs initialization at some point during boot and that in turn requires a new helper in execmem that will say ifEXECMEM_CACHE_ROXis enabled.
Hmm... deciding whether to use EXECMEM_ROX_CACHE at runtime seems weird. Can we enable it all the time and decide how much memory it reserves at runtime?
Given this constraint, we can probably ship this now and do the rest in a follow up set.
For the set
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
On architectures that support large mappings in the direct map and do not support
EXECMEM_CACHE_ROXthis will increase direct map fragmentation.Architectures that support
bpf_prog_packcan benefit fromEXECMEM_CACHE_ROX. What does it take to also enableEXECMEM_CACHE_ROXfor them?
[...]
Quoted 22 lines · Song Liu
Quoted 21 lines · Mike Rapoport
Quoted 12 lines · Song Liu
Quoted 11 lines · Mike Rapoport
Quoted 10 lines
Quoted 4 lines
Since commit
c82be0be9576("mm: vmalloc: don't account for number of nodes forHUGE_VMAPallocations")vmalloc_huge()is fine withPMD_SIZEregardless of number of nodes.Before execmem can handle sub page allocations, how about we send allocations that are bigger than page size directly to execmem, and let
bpf_prog_packhandle sub page allocations. Then,BPF_PROG_PACK_SIZEwill bePAGE_SIZE. This should be a net win forx86_64. Other archs will be the same. WDYT?Quoted 4 lines
Quoted 1 line
Makes perfect sense to me for x86.
Can we include this change to this patch set? This will make this change a meaningful step towards migrating to EXECMEM.
I'd prefer to leave this as a separate followup.
Quoted 8 lines
Quoted 4 lines
The decision whether to use
EXECMEM_ROX_CACHEis made at runtime, which means thatBPF_PROG_PACK_SIZEshould become a variable, that variable needs initialization at some point during boot and that in turn requires a new helper in execmem that will say ifEXECMEM_CACHE_ROXis enabled.Hmm... deciding whether to use
EXECMEM_ROX_CACHEat runtime seems weird. Can we enable it all the time and decide how much memory it reserves at runtime?
Enabling ROX cache without using PMD_SIZE mappings is pure overhead without advantage of using larger mappings for executable memory.
Quoted 2 lines
Given this constraint, we can probably ship this now and do the rest in a follow up set.
Yeah, this is something to think more about :)
Quoted 2 lines
For the set
Acked-by Song Liu <[email protected]>
Thanks!
Signature · 2 lines
Sincerely yours, Mike.
This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:
Quoted 10 lines · On Thu, 16 Jul 2026 10:51:34 +0300 you wrote:
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most paths that call
bpf_jit_alloc_exec()immediately make it ROX withset_memory_rox().The code generation, at least on x86, uses separately allocated writable buffers and then updates the actual text memory with
text_poke().[...]
Here is the summary with links:
- [bpf-next,v3,1/5] bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
https://git.kernel.org/bpf/bpf-next/c/810a273919df - [bpf-next,v3,2/5] bpf: drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec() https://git.kernel.org/bpf/bpf-next/c/4946eb5d37cb - [bpf-next,v3,3/5] bpf: alloc_prog_pack(): skip ROX management for already ROX memory https://git.kernel.org/bpf/bpf-next/c/7516714947da - [bpf-next,v3,4/5] bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable https://git.kernel.org/bpf/bpf-next/c/5bf02dbf39fa - [bpf-next,v3,5/5] x86/bpf: enable EXECMEM_ROX_CACHE for BPF allocations https://git.kernel.org/bpf/bpf-next/c/6b88aaec81db
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html