Make sleepable arena paths use sleepable alloc_pages
Subsystems (4)
Emil Tsalapatis
reviewerv4 cover Make sleepable arena paths use sleepable alloc_pages
The arena_alloc_pages() call takes a sleepable argument based on whether its caller is a sleepable BPF function. This flag, along with the context the kfunc is called in, decides whether the call will try to fulfill the allocation using the regular or the _nolock variant of the alloc_pages API, by means of bpf_map_alloc_pages().
However, the arena_alloc_pages() call currently only makes allocations inside an IRQ-disabled critical section. This forces all allocations to use the _nolock() API, which may eagerly fail where the regular variant would eventually succeed. There have been reports of this happening for sched-ext schedulers.
Restructure arena_alloc_pages() to use the _nolock() page allocation API only when necessary. This requires moving allocations outside of the spinlock critical section for sleepable calls, which in turn requires slightly different logic in the allocation path. Replace the page list allocation with logic that reuses pcp_llist to chain allocated pages together, allowing us to merge the code paths for both sleepable and nonsleepable arena allocations.
Also fix kfunc specialization to not unnecessarily force the nonsleepable version of bpf_arena_alloc_pages() for call sites that do not need it. This requires making specialization per-call site instead of overwriting the descriptor during fixups.
v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/)
- Keep the sleepable and non-sleepable allocation paths within arena_alloc_pages (Alexei)
- Incorporate bot feedback on selftests (bot-ci)
v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/)
- Remove the intermediate page array allocation in bpf_arena_alloc_pages() and use the pcp_llist pointer instead (Alexei)
- Fix function specialization to only specialize to the nonsleepable version when necessary
v3 -> v4 (https://lore.kernel.org/bpf/[email protected]/)
- Remove in-flight pages tracking (Alexei)
- Remove stale split sleepable/nonsleepable error handling (Alexei)
Emil Tsalapatis (7):
bpf: Use an llist for page allocations
bpf: Add sleepable argument to bpf_alloc_pages()
bpf: Add sleepable arena page allocation path selftests/bpf: Test large allocations for both sleepable/nonsleepable
arena users
bpf: Support call-site kfunc specialization for near calls
bpf: Support call-site kfunc specialization for far calls selftests/bpf: Test per-call site function specialization
include/linux/bpf.h | 10 +- include/linux/bpf_verifier.h | 15 +- kernel/bpf/arena.c | 158 +++++++++--------- kernel/bpf/fixups.c | 18 +- kernel/bpf/syscall.c | 43 +++-- kernel/bpf/verifier.c | 82 ++++++++- .../selftests/bpf/prog_tests/file_reader.c | 15 ++ .../testing/selftests/bpf/progs/file_reader.c | 129 ++++++++++++++ .../bpf/progs/verifier_arena_large.c | 64 +++++-- 9 files changed, 405 insertions(+), 129 deletions(-)
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v4 1/7 bpf: Use an llist for page allocations
bpf_map_alloc_pages() does not use its map argument. Storing allocated pages in an array also forces arena callers to allocate a separate pointer array.
Expose the single-page allocator as bpf_alloc_page(), rename the bulk helper to bpf_alloc_pages(), and return bulk allocations through an llist using page->pcp_llist. Add bpf_free_pages() to safely release all pages remaining on such a list.
--- include/linux/bpf.h | 6 ++- kernel/bpf/arena.c | 95 +++++++++++++++++++------------------------- kernel/bpf/syscall.c | 39 ++++++++++-------- 3 files changed, 67 insertions(+), 73 deletions(-)
include/linux/bpf.h +4 -2
@@ -2884,8 +2884,10 @@ struct bpf_map *bpf_map_get_curr_or_next(u32 *id);struct bpf_prog *bpf_prog_get_curr_or_next (u32 *id); -int bpf_map_alloc_pages (const struct bpf_map *map,int nid, -unsigned long nr_pages,struct page **page_array); +struct page *bpf_alloc_page (int nid); +int bpf_alloc_pages (int nid,unsigned long nr_pages, +struct llist_head *pages); +void bpf_free_pages (struct llist_head *pages);#ifdef CONFIG_MEMCG void bpf_map_memcg_enter (const struct bpf_map *map,struct mem_cgroup **old_memcg,struct mem_cgroup **new_memcg);
kernel/bpf/arena.c +40 -55
181 lines not on this page · raw message
kernel/bpf/syscall.c +23 -16
@@ -602,7 +602,7 @@ static bool can_alloc_pages(void) !IS_ENABLED (CONFIG_PREEMPT_RT ); } -static struct page *__bpf_alloc_page (int nid) +struct page *bpf_alloc_page (int nid) {if (!can_alloc_pages ())return alloc_pages_nolock (__GFP_ACCOUNT, nid,0 );
@@ -613,27 +613,34 @@ static struct page *__bpf_alloc_page(int nid) 0); } -int bpf_map_alloc_pages (const struct bpf_map *map,int nid, -unsigned long nr_pages,struct page **pages) +void bpf_free_pages (struct llist_head *pages) { -unsigned long i, j; +struct llist_node *node; +struct page *page, *tmp; + + node =llist_del_all (pages); +llist_for_each_entry_safe (page, tmp, node, pcp_llist) +free_pages_nolock (page,0 ); +} + +int bpf_alloc_pages (int nid,unsigned long nr_pages, +struct llist_head *pages) +{ +unsigned long i;struct page *pg; -int ret =0 ;for (i =0 ; i < nr_pages; i++) { - pg =__bpf_alloc_page (nid); - -if (pg) { - pages[i] = pg; -continue ; - } -for (j =0 ; j < i; j++) -free_pages_nolock (pages[j],0 ); - ret = -ENOMEM ; -break ; + pg =bpf_alloc_page (nid); +if (!pg) + goto free_pages; +llist_add (&pg->pcp_llist, pages); } -return ret; +return 0 ; + +free_pages: +bpf_free_pages (pages); +return -ENOMEM ; }static int btf_field_cmp (const void *a,const void *b)
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v4 2/7 bpf: Add sleepable argument to bpf_alloc_pages()
In reply to Emil Tsalapatisbpf_alloc_pages() currently decides whether it may use the blocking page allocator from the current execution context alone. Let callers further restrict that choice by passing whether their context is sleepable.
Use the blocking allocator only when both the caller and runtime context allow sleeping. Add __GFP_RETRY_MAYFAIL so this path can reclaim without invoking the OOM killer when the allocation is charged to another memcg. Existing non-sleepable callers retain the no-lock allocation behavior.
--- include/linux/bpf.h | 4 ++-- kernel/bpf/arena.c | 6 +++--- kernel/bpf/syscall.c | 10 +++++----- 3 files changed, 10 insertions(+), 10 deletions(-)
include/linux/bpf.h +2 -2
@@ -2884,9 +2884,9 @@ struct bpf_map *bpf_map_get_curr_or_next(u32 *id);struct bpf_prog *bpf_prog_get_curr_or_next (u32 *id); -struct page *bpf_alloc_page (int nid); +struct page *bpf_alloc_page (int nid,bool sleepable);int bpf_alloc_pages (int nid,unsigned long nr_pages, -struct llist_head *pages); +struct llist_head *pages,bool sleepable);void bpf_free_pages (struct llist_head *pages);#ifdef CONFIG_MEMCG void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
kernel/bpf/arena.c +3 -3
@@ -317,7 +317,7 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)INIT_WORK (&arena->free_work, arena_free_worker);bpf_map_init_from_attr (&arena->map, attr); - arena->scratch_page =bpf_alloc_page (NUMA_NO_NODE ); + arena->scratch_page =bpf_alloc_page (NUMA_NO_NODE , true);if (!arena->scratch_page) goto err_free_arena;
@@ -550,7 +550,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)* The probed page was freed meanwhile or preallocation failed; * try the non-blocking allocator, we cannot sleep here. */ - new_page =bpf_alloc_page (map->numa_node); + new_page =bpf_alloc_page (map->numa_node, false);if (!new_page) { fault_ret =VM_FAULT_SIGBUS ; goto out_err_locked_memcg;
@@ -771,7 +771,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); - ret =bpf_alloc_pages (node_id, page_cnt, &pages); + ret =bpf_alloc_pages (node_id, page_cnt, &pages, false);if (ret) goto out;
kernel/bpf/syscall.c +5 -5
@@ -602,14 +602,14 @@ static bool can_alloc_pages(void) !IS_ENABLED (CONFIG_PREEMPT_RT ); } -struct page *bpf_alloc_page (int nid) +struct page *bpf_alloc_page (int nid,bool sleepable) { -if (!can_alloc_pages ()) +if (!sleepable || !can_alloc_pages ())return alloc_pages_nolock (__GFP_ACCOUNT, nid,0 );return alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT - | __GFP_NOWARN, + | __GFP_NOWARN | __GFP_RETRY_MAYFAIL,0 ); }
@@ -624,13 +624,13 @@ void bpf_free_pages(struct llist_head *pages) }int bpf_alloc_pages (int nid,unsigned long nr_pages, -struct llist_head *pages) +struct llist_head *pages,bool sleepable) {unsigned long i;struct page *pg;for (i =0 ; i < nr_pages; i++) { - pg =bpf_alloc_page (nid); + pg =bpf_alloc_page (nid, sleepable);if (!pg) goto free_pages;llist_add (&pg->pcp_llist, pages);
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v4 3/7 bpf: Add sleepable arena page allocation path
In reply to Emil TsalapatisThe bpf_arena_alloc_pages() function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use of alloc_pages_nolock() in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.
Allocate the arena pages before taking the critical section and pass whether the caller can sleep to bpf_alloc_pages(). This lets sleepable callers use the blocking allocator while non-sleepable callers retain the no-lock allocation behavior.
--- kernel/bpf/arena.c | 73 +++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 27 deletions(-)
kernel/bpf/arena.c +46 -27
@@ -713,6 +713,27 @@ static u64 clear_lo32(u64 val)return val & ~(u64 )~0U ; } +static int arena_adjust_tree (struct bpf_arena *arena,long uaddr,long page_cnt,long *pgoff) +{ +int ret; + +/* Special case where user is requesting specific range. */ +if (uaddr) { + ret =is_range_tree_set (&arena->rt, *pgoff, page_cnt); +if (ret) +return ret; +return range_tree_clear (&arena->rt, *pgoff, page_cnt); + } + + ret =range_tree_find (&arena->rt, page_cnt); +if (ret <0 ) +return ret; + + *pgoff = ret; + +return range_tree_clear (&arena->rt, *pgoff, page_cnt); +} +/* * Allocate pages and vmap them into kernel vmalloc area. * Later the pages will be mmaped into user space vma.
@@ -730,6 +751,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntunsigned long flags;long pgoff =0 ;u32 uaddr32; +long addr =0 ;int ret; if (node_id != NUMA_NO_NODE &&
@@ -747,8 +769,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* requested address will be outside of user VMA */ return 0 ; } -bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + + ret =bpf_alloc_pages (node_id, page_cnt, &pages, sleepable); +if (ret) + goto out_memcg; + data.arena = arena; data.pages = &pages; data.i =0 ;
@@ -756,25 +782,14 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntif (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) goto out_free_pages; -if (uaddr) { - ret =is_range_tree_set (&arena->rt, pgoff, page_cnt); -if (ret) - goto out_unlock_free_pages; - ret =range_tree_clear (&arena->rt, pgoff, page_cnt); - }else { - ret = pgoff =range_tree_find (&arena->rt, page_cnt); -if (pgoff >=0 ) - ret =range_tree_clear (&arena->rt, pgoff, page_cnt); + ret =arena_adjust_tree (arena, uaddr, page_cnt, &pgoff); +if (ret) { +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + goto out_free_pages; } -if (ret) - goto out_unlock_free_pages; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); - ret =bpf_alloc_pages (node_id, page_cnt, &pages, false); -if (ret) - goto out; -/* * Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 * will not overflow 32-bit. Lower 32-bit need to represent
@@ -787,26 +802,30 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt page_cnt << PAGE_SHIFT, apply_range_set_cb, &data); mapped = data.i;if (ret) - goto out; + goto out_unmap;flush_vmap_cache (kern_vm_start + uaddr32, mapped <<PAGE_SHIFT );raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); -bpf_map_memcg_exit (old_memcg, new_memcg); -return clear_lo32 (arena->user_vm_start) + uaddr32; -out: + + addr =clear_lo32 (arena->user_vm_start) + uaddr32; + goto out_memcg; + +out_unmap: +/* Error handling: Undo partial mappings. */ +flush_vmap_cache (kern_vm_start + uaddr32, mapped <<PAGE_SHIFT );range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped);raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); -if (mapped) { -flush_vmap_cache (kern_vm_start + uaddr32, mapped <<PAGE_SHIFT ); +if (mapped)arena_free_pages (arena, uaddr32, mapped, sleepable); - } - goto out_free_pages; -out_unlock_free_pages: -raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + out_free_pages: +/* Error handling: Free back any unmapped pages. */ bpf_free_pages (&pages); + +out_memcg:bpf_map_memcg_exit (old_memcg, new_memcg); -return 0 ; + +return addr; }/*
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 3/7] bpf: Add sleepable arena page allocation path
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: CPU lockups and memory pressure due to massive page allocations before range validation --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: CPU lockups and memory pressure due to massive page allocations before range validation --
commit 0ff0c13da92aa5b627b22373d01f9a6a0085c43f
Author: Emil Tsalapatis <[email protected]>
bpf: Add sleepable arena page allocation path
This commit modifies the BPF arena allocator to allocate physical pages outside the spinlock. This allows sleepable BPF programs to use the blocking allocator slow path when under memory pressure.
Quoted 3 lines
kernel/bpf/arena.c +0
-0
[ ... ]
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 777
Quoted 11 lines
kernel/bpf/arena.c +5
-1@@ -747,8 +769,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* requested address will be outside of user VMA */ return 0 ; } -bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + + ret =bpf_alloc_pages (node_id, page_cnt, &pages, sleepable); +if (ret) + goto out_memcg; +
[Severity: High]
Could this early allocation trigger system instability or CPU lockups under pressure?
By moving bpf_alloc_pages() before the arena_adjust_tree() check, a BPF program can request a huge page_cnt (up to the entire arena size). Because the physical pages are allocated before we verify if the requested virtual address range is actually available in the range tree, this could result in massive physical memory allocations and blocking reclaims.
If the requested virtual range is already occupied, the allocation will just immediately free everything when the range check fails under the lock. Could this be used by a BPF program to induce excessive memory pressure?
Quoted 3 lines
kernel/bpf/arena.c +0
-0data.arena = arena; data.pages = &pages; data.i =0 ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Emil Tsalapatis
reviewerRe: [PATCH bpf-next v4 3/7] bpf: Add sleepable arena page allocation path
Quoted 42 lines · On Fri, Sep 25, 2026 at 11:46 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: CPU lockups and memory pressure due to massive page allocations before range validation --commit
0ff0c13da92aa5b627b22373d01f9a6a0085c43f
Author: Emil Tsalapatis <[email protected]>bpf: Add sleepable arena page allocation path
This commit modifies the BPF arena allocator to allocate physical pages outside the spinlock. This allows sleepable BPF programs to use the blocking allocator slow path when under memory pressure.
Quoted 3 lines
kernel/bpf/arena.c +0
-0[ ... ]
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 777
Quoted 11 lines
kernel/bpf/arena.c +5
-1@@ -747,8 +769,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* requested address will be outside of user VMA */ return 0 ; } -bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + + ret =bpf_alloc_pages (node_id, page_cnt, &pages, sleepable); +if (ret) + goto out_memcg; +[Severity: High]
Could this early allocation trigger system instability or CPU lockups under pressure?By moving
bpf_alloc_pages()before thearena_adjust_tree()check, a BPF program can request a hugepage_cnt(up to the entire arena size). Because the physical pages are allocated before we verify if the requested virtual address range is actually available in the range tree, this could result in massive physical memory allocations and blocking reclaims.
Quoted 8 lines
If the requested virtual range is already occupied, the allocation will just immediately free everything when the range check fails under the lock. Could this be used by a BPF program to induce excessive memory pressure?
Quoted 3 lines
kernel/bpf/arena.c +0
-0data.arena = arena; data.pages = &pages; data.i =0 ;
This scenario is handled well enough as we discussed in the previous version.
As an aside, the set has a merge conflict because it is being routed to the bpf tree but the patch series is based off of bpf-next. I think it may be better to target bpf-next instead, otherwise we're going to have a conflict merging the two later (bpf-next is further ahead arena-wise so to speak).
Quoted 2 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Emil Tsalapatis
reviewerPATCH v4 4/7 selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
In reply to Emil TsalapatisWe have relaxed the limitation of 1024-page batches for both sleepable and nonsleepable arena allocations by removing the intermediate page array. Add a test to confirm that large allocations succeed for both.
--- .../bpf/progs/verifier_arena_large.c | 64 +++++++++++++++---- 1 file changed, 50 insertions(+), 14 deletions(-)
tools/testing/selftests/bpf/progs/verifier_arena_large.c +50 -14
@@ -10,6 +10,9 @@#include <bpf_arena_common.h> #define ARENA_SIZE (1ull << 32) +#define LARGE_PAGE_CNT 1025 + +volatile int zero =0 ;struct { __uint(type, BPF_MAP_TYPE_ARENA);
@@ -284,6 +287,7 @@ int big_alloc2(void *ctx)return 0 ; } +/* Nonsleepable because it binds to a socket program. */ SEC("socket") __success __retval(0) int big_alloc3(void *ctx)
@@ -291,24 +295,56 @@ int big_alloc3(void *ctx)#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) char __arena *pages;u64 i; +int err =0 ; -/* -* Allocate 2051 pages in one go to check how kmalloc_nolock() handles large requests. -* Since kmalloc_nolock() can allocate up to 1024 struct page * at a time, this call should -* result in three batches: two batches of 1024 pages each, followed by a final batch of 3 -* pages. -*/ - pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 ); +/* Verify that a nonsleepable allocation larger than 1024 pages succeeds. */ + pages =bpf_arena_alloc_pages (&arena,NULL ,LARGE_PAGE_CNT ,NUMA_NO_NODE ,0 );if (!pages) -return 0 ; +return 1 ; + +for (i = zero; i <LARGE_PAGE_CNT && can_loop; i++) + pages[i *PAGE_SIZE ] =123 ; + +for (i = zero; i <LARGE_PAGE_CNT && can_loop; i++) { +if (pages[i *PAGE_SIZE ] ==123 ) +continue ; + err =2 ; +break ; + } + +bpf_arena_free_pages (&arena, pages,LARGE_PAGE_CNT ); +return err; +#endif +return 0 ; +} -bpf_for (i,0 ,2051 ) - pages[i * PAGE_SIZE] = 123; - bpf_for(i, 0, 2051) -if (pages[i *PAGE_SIZE ] !=123 ) -return i; +/* SYSCALL programs are always sleepable. */ +SEC("syscall") +__success __retval(0) +int big_alloc4(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + char __arena *pages; +u64 i; +int err =0 ; + +/* Verify that a sleepable allocation larger than 1024 pages succeeds. */ + pages =bpf_arena_alloc_pages (&arena,NULL ,LARGE_PAGE_CNT ,NUMA_NO_NODE ,0 ); +if (!pages) +return 1 ; + +for (i = zero; i <LARGE_PAGE_CNT && can_loop; i++) + pages[i *PAGE_SIZE ] =123 ; + +for (i = zero; i <LARGE_PAGE_CNT && can_loop; i++) { +if (pages[i *PAGE_SIZE ] ==123 ) +continue ; + err =2 ; +break ; + } -bpf_arena_free_pages (&arena, pages,2051 ); +bpf_arena_free_pages (&arena, pages,LARGE_PAGE_CNT ); +return err;#endif return 0 ; }
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v4 5/7 bpf: Support call-site kfunc specialization for near calls
In reply to Emil Tsalapatisspecialize_kfunc() currently updates the canonical kfunc descriptor in place. It is not currently possible to swtich different specializations of a kfunc per call site in the same program. In fact, specializations are order-dependent: Once a function is specialized, all subsequent call sites are specialized even if they wouldn't trigger specialization themselves. This is especially an issue for bpf_arena_alloc_pages() that is specialized into its non-sleepable for all call sites after a single non-sleepable one.
Allow per-call site kfunc specialization for JITs that use near calls. Implement this by keeping two versions of the kfunc table, one with just the initial kfuncs and one with all valid specializations for the program. We currently assume 2 concurrent specializations for each kfunc. This is a conservative estimate, since most of them do not specialize at all.
--- include/linux/bpf_verifier.h | 13 ++++--- kernel/bpf/verifier.c | 67 +++++++++++++++++++++++++++++++++--- 2 files changed, 71 insertions(+), 9 deletions(-)
include/linux/bpf_verifier.h +9 -4
@@ -1714,6 +1714,8 @@ enum bpf_reg_arg_type { };#define MAX_KFUNC_DESCS 256 +/* Each kfunc can have its canonical and one specialized call target. */ +#define MAX_KFUNC_CALL_DESCS (MAX_KFUNC_DESCS * 2)struct bpf_kfunc_desc {struct btf_func_model func_model;
@@ -1726,12 +1728,15 @@ struct bpf_kfunc_desc {struct bpf_kfunc_desc_tab {u32 nr_descs; +u32 nr_base_descs;/* Sorted by func_id (BTF ID) and offset (fd_array offset) during -* verification. JITs do lookups by bpf_insn, where func_id may not be -* available, therefore at the end of verification do_misc_fixups() -* sorts this by imm and offset. +* verification. The first nr_base_descs entries are the canonical +* descriptors used for verifier lookups. Call specialization may append +* immutable descriptors for additional targets. Near-call JITs look up +* descriptors by imm and offset after do_misc_fixups() sorts the table. * -* Grown one entry at a time by bpf_add_kfunc_call(). +* Grown one entry at a time by bpf_add_kfunc_call() and during +* call specialization. */ struct bpf_kfunc_desc descs[]; };
kernel/bpf/verifier.c +62 -5
111 lines not on this page · raw message
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v4 6/7 bpf: Support call-site kfunc specialization for far calls
In reply to Emil TsalapatisFar-call JITs keep the kfunc BTF ID in the call immediate, so it cannot distinguish multiple specialized targets for the same kfunc and BTF object.
Store the finalized target descriptor index in the call offset. Use that index for address and function-model lookup, and keep the descriptor table in verification order so the indices remain stable.
--- include/linux/bpf.h | 4 ++-- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/fixups.c | 18 ++++++++++++++---- kernel/bpf/verifier.c | 35 ++++++++++++++++++++++------------- 4 files changed, 40 insertions(+), 19 deletions(-)
include/linux/bpf.h +2 -2
@@ -3283,7 +3283,7 @@ const struct btf_func_model * bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn);int bpf_get_kfunc_addr (const struct bpf_prog *prog,u32 func_id, -u16 btf_fd_idx,u8 **func_addr); +u16 desc_idx,u8 **func_addr);struct bpf_core_ctx {struct bpf_verifier_log *log;
@@ -3626,7 +3626,7 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,static inline int bpf_get_kfunc_addr (const struct bpf_prog *prog,u32 func_id, -u16 btf_fd_idx,u8 **func_addr) +u16 desc_idx,u8 **func_addr) {return -ENOTSUPP ; }
include/linux/bpf_verifier.h +2 -0
@@ -1734,6 +1734,8 @@ struct bpf_kfunc_desc_tab {* descriptors used for verifier lookups. Call specialization may append * immutable descriptors for additional targets. Near-call JITs look up * descriptors by imm and offset after do_misc_fixups() sorts the table. +* Far-call JITs use the descriptor index stored in the finalized call's +* off field, so their table remains in verification order. * * Grown one entry at a time by bpf_add_kfunc_call() and during * call specialization.
kernel/bpf/fixups.c +14 -4
@@ -140,6 +140,13 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,struct bpf_kfunc_desc_tab *tab; tab = prog->aux->kfunc_tab; +if (bpf_jit_supports_far_kfunc_call ()) { +if (insn->off <0 || insn->off >= tab->nr_descs) +return NULL ; + res = &tab->descs[insn->off]; +return res->func_id == insn->imm ? &res->func_model :NULL ; + } + res =bsearch (&desc, tab->descs, tab->nr_descs,sizeof (tab->descs[0 ]), kfunc_desc_cmp_by_imm_off);
@@ -2517,11 +2524,14 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) } } - ret =sort_kfunc_descs_by_imm_off (env); -if (ret) -return ret; +/* +* Do not change kfunc desc position into the table for far JIT. +* because we use the indices in the instructions. +*/ +if (bpf_jit_supports_far_kfunc_call ()) +return 0 ; -return 0 ; +return sort_kfunc_descs_by_imm_off (env); } static struct bpf_prog *inline_bpf_loop(struct bpf_verifier_env *env,
kernel/bpf/verifier.c +22 -13
@@ -2575,12 +2575,16 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) }int bpf_get_kfunc_addr (const struct bpf_prog *prog,u32 func_id, -u16 btf_fd_idx,u8 **func_addr) +u16 desc_idx,u8 **func_addr) { +struct bpf_kfunc_desc_tab *tab;const struct bpf_kfunc_desc *desc; - desc =find_kfunc_desc (prog, func_id, btf_fd_idx); -if (!desc) + tab = prog->aux->kfunc_tab; +if (desc_idx >= tab->nr_descs) +return -EFAULT ; + desc = &tab->descs[desc_idx]; +if (desc->func_id != func_id)return -EFAULT ; *func_addr = (u8 *)desc->addr;
@@ -21303,7 +21307,8 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc }static int add_kfunc_desc_target (struct bpf_verifier_env *env, -const struct bpf_kfunc_desc *target_desc) +const struct bpf_kfunc_desc *target_desc, +u16 *desc_idx) {struct bpf_kfunc_desc desc = *target_desc;struct bpf_kfunc_desc_tab *new_tab;
@@ -21316,8 +21321,10 @@ static int add_kfunc_desc_target(struct bpf_verifier_env *env,for (i =0 ; i < tab->nr_descs; i++) {if (tab->descs[i].func_id == desc.func_id && tab->descs[i].offset == desc.offset && - tab->descs[i].addr == desc.addr) + tab->descs[i].addr == desc.addr) { + *desc_idx = i;return 0 ; + } }if (tab->nr_descs ==MAX_KFUNC_CALL_DESCS ) {
@@ -21332,6 +21339,7 @@ static int add_kfunc_desc_target(struct bpf_verifier_env *env, tab = new_tab; prog_aux->kfunc_tab = tab; + *desc_idx = tab->nr_descs; tab->descs[tab->nr_descs++] = desc;return 0 ; }
@@ -21359,6 +21367,7 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,struct bpf_kfunc_desc desc_copy;struct bpf_kfunc_desc *desc;unsigned long call_imm; +u16 desc_idx;bool near_call;int err;
@@ -21381,10 +21390,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } near_call = !bpf_jit_supports_far_kfunc_call (); -if (near_call) { - desc_copy = *desc; - desc = &desc_copy; - } + desc_copy = *desc; + desc = &desc_copy; err =specialize_kfunc (env, desc, insn_idx);if (err)
@@ -21398,12 +21405,14 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EINVAL ; } insn->imm = call_imm; - - err =add_kfunc_desc_target (env, desc); -if (err) -return err; } + err =add_kfunc_desc_target (env, desc, &desc_idx); +if (err) +return err; +if (!near_call) + insn->off = desc_idx; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) };
Signature · 1 line
2.52.0
Re: [PATCH bpf-next v4 6/7] bpf: Support call-site kfunc specialization for far calls
kernel/bpf/fixups.c int bpf_do_misc_fixups(struct bpf_verifier_env *env) · line 2534
Quoted 12 lines · Emil Tsalapatis
kernel/bpf/fixups.c +7
-4- ret =sort_kfunc_descs_by_imm_off (env); -if (ret) -return ret; +/* +* Do not change kfunc desc position into the table for far JIT. +* because we use the indices in the instructions. +*/ +if (bpf_jit_supports_far_kfunc_call ()) +return 0 ; -return 0 ; +return sort_kfunc_descs_by_imm_off (env);
can we store the index in insn->off for near calls too ? Near call JITs take the address from insn->imm and need only func_model from the desc.
Then all JITs do the same lookup in bpf_jit_find_kfunc_model() and no need for sort_kfunc_descs_by_imm_off(), desc->imm and bsearch.
as you mentioned in the other thread pls target bpf-next. We're too far into -rc to do such large changes in bpf tree.
pw-bot: cr
Emil Tsalapatis
reviewerPATCH v4 7/7 selftests/bpf: Test per-call site function specialization
In reply to Emil TsalapatisAdd a test to ensure function call specialization is done per-call site. Use bpf_dynptr_from_file that has observably different behavior between its sleepable and nonsleepable versions. The sleepable path fails in the sleepable __kernel_read() call with -EIO, while the nonsleepable fails in the page-cache lookup path with -EFAULT. Test that whatever the order the nonsleepable and sleepable calls are made in the program, both call sites use the correct specialized kfunc.
--- .../selftests/bpf/prog_tests/file_reader.c | 15 ++ .../testing/selftests/bpf/progs/file_reader.c | 129 ++++++++++++++++++ 2 files changed, 144 insertions(+)
tools/testing/selftests/bpf/prog_tests/file_reader.c +15 -0
@@ -7,10 +7,12 @@#include "file_reader_fail.skel.h" #include <dlfcn.h> #include <sys/mman.h> +#include <sys/stat.h> const char *user_ptr ="hello world" ;char file_contents[256000 ];void *addr; +__u64 beyond_eof_offset;void *get_executable_base_addr (void ) {
@@ -26,12 +28,18 @@ void *get_executable_base_addr(void)static int initialize_file_contents (void ) { +struct stat st;int fd, page_sz =sysconf (_SC_PAGESIZE);ssize_t n =0 , cur; fd =open ("/proc/self/exe" ,O_RDONLY );if (!ASSERT_OK_FD (fd,"Open /proc/self/exe\n" ))return 1 ; +if (!ASSERT_OK (fstat (fd, &st),"fstat /proc/self/exe" )) { +close (fd); +return 1 ; + } + beyond_eof_offset = st.st_size + (1ULL <<30 ); do { cur = read(fd, file_contents + n, sizeof(file_contents) - n);
@@ -75,6 +83,7 @@ static void run_test(const char *prog_name)memcpy (skel->bss->user_buf, file_contents,sizeof (file_contents)); skel->bss->pid =getpid (); + skel->bss->beyond_eof_offset = beyond_eof_offset; err =file_reader__load (skel);if (!ASSERT_OK (err,"file_reader__load" ))
@@ -110,6 +119,12 @@ void test_file_reader(void)if (test__start_subtest ("on_open_validate_file_read" ))run_test ("on_open_validate_file_read" ); +if (test__start_subtest ("on_open_non_sleepable_first" )) +run_test ("on_open_non_sleepable_first" ); + +if (test__start_subtest ("on_open_sleepable_first" )) +run_test ("on_open_sleepable_first" ); +if (test__start_subtest ("negative" ))RUN_TESTS (file_reader_fail); }
tools/testing/selftests/bpf/progs/file_reader.c +129 -0
150 lines not on this page · raw message
Signature · 1 line
2.52.0