Skip to contentSkip to conversation

Make sleepable arena paths use sleepable alloc_pages

PATCH RESEND Applied Emil Tsalapatis 4 participants 11 messages Oct 2

Subsystems (4)
lore
Share & tools
lore

Emil Tsalapatis

author reviewer
#0 lore raw

v6 cover Make sleepable arena paths use sleepable alloc_pages

The arena_alloc_pages() call takes a sleepable argument based on whether its caller is a sleepable BPF function. This flag, along with the context the kfunc is called in, decides whether the call will try to fulfill the allocation using the regular or the _nolock variant of the alloc_pages API, by means of bpf_map_alloc_pages().

However, the arena_alloc_pages() call currently only makes allocations inside an IRQ-disabled critical section. This forces all allocations to use the _nolock() API, which may eagerly fail where the regular variant would eventually succeed. There have been reports of this happening for sched-ext schedulers.

Restructure arena_alloc_pages() to use the _nolock() page allocation API only when necessary. This requires moving allocations outside of the spinlock critical section for sleepable calls, which in turn requires slightly different logic in the allocation path. Replace the page list allocation with logic that reuses pcp_llist to chain allocated pages together, allowing us to merge the code paths for both sleepable and nonsleepable arena allocations.

Also fix kfunc specialization to not unnecessarily force the nonsleepable version of bpf_arena_alloc_pages() for call sites that do not need it. This requires making specialization per-call site instead of overwriting the descriptor during fixups.

v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/)

- Keep the sleepable and non-sleepable allocation paths within arena_alloc_pages (Alexei)
- Incorporate bot feedback on selftests (bot-ci)

v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/)

- Remove the intermediate page array allocation in bpf_arena_alloc_pages() and use the pcp_llist pointer instead (Alexei)
- Fix function specialization to only specialize to the nonsleepable version when necessary

v3 -> v4 (https://lore.kernel.org/bpf/[email protected]/)

- Remove in-flight pages tracking (Alexei)
- Remove stale split sleepable/nonsleepable error handling (Alexei)

v4 -> v5 (https://lore.kernel.org/bpf/[email protected]/)

- Remove unnecessary imm-based sorting for kfunc desc table (Alexei)
- Skip all nonspecialized kfunc descs during the linear scan done by function specialization

v5 -> v6 (https://lore.kernel.org/bpf/[email protected]/)

- Use verifier_bug() for diagnostics (bot)

Emil Tsalapatis (7):
bpf: Use an llist for page allocations
bpf: Add sleepable argument to bpf_alloc_pages()
bpf: Add sleepable arena page allocation path selftests/bpf: Test large allocations for both sleepable/nonsleepable
arena users
bpf: Directly store kfunc desc index in instruction off field
bpf: Support per-call-site kfunc specialization selftests/bpf: Test per-call site function specialization

 include/linux/bpf.h                           |  10 +-
 include/linux/bpf_verifier.h                  |  15 +-
 kernel/bpf/arena.c                            | 158 +++++++++---------
 kernel/bpf/fixups.c                           |  70 +-------
 kernel/bpf/syscall.c                          |  43 +++--
 kernel/bpf/verifier.c                         | 107 +++++++++++-
 .../selftests/bpf/prog_tests/file_reader.c    |  15 ++
 .../testing/selftests/bpf/progs/file_reader.c | 129 ++++++++++++++
 .../bpf/progs/verifier_arena_large.c          |  64 +++++--
 9 files changed, 423 insertions(+), 188 deletions(-)
Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH RESEND v6 1/7 bpf: Use an llist for page allocations

bpf_map_alloc_pages() does not use its map argument. Storing allocated pages in an array also forces arena callers to allocate a separate pointer array.

Expose the single-page allocator as bpf_alloc_page(), rename the bulk helper to bpf_alloc_pages(), and return bulk allocations through an llist using page->pcp_llist. Add bpf_free_pages() to safely release all pages remaining on such a list.

---
 include/linux/bpf.h  |  6 ++-
 kernel/bpf/arena.c   | 95 +++++++++++++++++++-------------------------
 kernel/bpf/syscall.c | 39 ++++++++++--------
 3 files changed, 67 insertions(+), 73 deletions(-)
include/linux/bpf.h +4 -2
index 4bae3796c42f..904b539810c7 100644
@@ -2902,8 +2902,10 @@ struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
 struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
 
 
-int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
-			unsigned long nr_pages, struct page **page_array);
+struct page *bpf_alloc_page(int nid);
+int bpf_alloc_pages(int nid, unsigned long nr_pages,
+		    struct llist_head *pages);
+void bpf_free_pages(struct llist_head *pages);
 #ifdef CONFIG_MEMCG
 void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
 			 struct mem_cgroup **new_memcg);
kernel/bpf/arena.c +40 -55

181 lines not on this page · raw message

kernel/bpf/syscall.c +23 -16
index ac52f4ae414c..a5df15a6cd51 100644
@@ -602,7 +602,7 @@ static bool can_alloc_pages(void)
 		!IS_ENABLED(CONFIG_PREEMPT_RT);
 }
 
-static struct page *__bpf_alloc_page(int nid)
+struct page *bpf_alloc_page(int nid)
 {
 	if (!can_alloc_pages())
 		return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0);
@@ -613,27 +613,34 @@ static struct page *__bpf_alloc_page(int nid)
 				0);
 }
 
-int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
-			unsigned long nr_pages, struct page **pages)
+void bpf_free_pages(struct llist_head *pages)
 {
-	unsigned long i, j;
+	struct llist_node *node;
+	struct page *page, *tmp;
+
+	node = llist_del_all(pages);
+	llist_for_each_entry_safe(page, tmp, node, pcp_llist)
+		free_pages_nolock(page, 0);
+}
+
+int bpf_alloc_pages(int nid, unsigned long nr_pages,
+		    struct llist_head *pages)
+{
+	unsigned long i;
 	struct page *pg;
-	int ret = 0;
 
 	for (i = 0; i < nr_pages; i++) {
-		pg = __bpf_alloc_page(nid);
-
-		if (pg) {
-			pages[i] = pg;
-			continue;
-		}
-		for (j = 0; j < i; j++)
-			free_pages_nolock(pages[j], 0);
-		ret = -ENOMEM;
-		break;
+		pg = bpf_alloc_page(nid);
+		if (!pg)
+			goto free_pages;
+		llist_add(&pg->pcp_llist, pages);
 	}
 
-	return ret;
+	return 0;
+
+free_pages:
+	bpf_free_pages(pages);
+	return -ENOMEM;
 }
 
 static int btf_field_cmp(const void *a, const void *b)
Signature · 1 line
2.52.0
author reviewer
#2 lore raw

PATCH RESEND v6 2/7 bpf: Add sleepable argument to bpf_alloc_pages()

In reply to Emil Tsalapatis

bpf_alloc_pages() currently decides whether it may use the blocking page allocator from the current execution context alone. Let callers further restrict that choice by passing whether their context is sleepable.

Use the blocking allocator only when both the caller and runtime context allow sleeping. Add __GFP_RETRY_MAYFAIL so this path can reclaim without invoking the OOM killer when the allocation is charged to another memcg. Existing non-sleepable callers retain the no-lock allocation behavior.

---
 include/linux/bpf.h  |  4 ++--
 kernel/bpf/arena.c   |  6 +++---
 kernel/bpf/syscall.c | 10 +++++-----
 3 files changed, 10 insertions(+), 10 deletions(-)
include/linux/bpf.h +2 -2
index 904b539810c7..670eb9f3f20a 100644
@@ -2902,9 +2902,9 @@ struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
 struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
 
 
-struct page *bpf_alloc_page(int nid);
+struct page *bpf_alloc_page(int nid, bool sleepable);
 int bpf_alloc_pages(int nid, unsigned long nr_pages,
-		    struct llist_head *pages);
+		    struct llist_head *pages, bool sleepable);
 void bpf_free_pages(struct llist_head *pages);
 #ifdef CONFIG_MEMCG
 void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
kernel/bpf/arena.c +3 -3
index de4f7c7f68f5..c556df7730c4 100644
@@ -317,7 +317,7 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 	INIT_WORK(&arena->free_work, arena_free_worker);
 	bpf_map_init_from_attr(&arena->map, attr);
 
-	arena->scratch_page = bpf_alloc_page(NUMA_NO_NODE);
+	arena->scratch_page = bpf_alloc_page(NUMA_NO_NODE, true);
 	if (!arena->scratch_page)
 		goto err_free_arena;
 
@@ -550,7 +550,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 		 * The probed page was freed meanwhile or preallocation failed;
 		 * try the non-blocking allocator, we cannot sleep here.
 		 */
-		new_page = bpf_alloc_page(map->numa_node);
+		new_page = bpf_alloc_page(map->numa_node, false);
 		if (!new_page) {
 			fault_ret = VM_FAULT_SIGBUS;
 			goto out_err_locked_memcg;
@@ -771,7 +771,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 
 	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
 
-	ret = bpf_alloc_pages(node_id, page_cnt, &pages);
+	ret = bpf_alloc_pages(node_id, page_cnt, &pages, false);
 	if (ret)
 		goto out;
 
kernel/bpf/syscall.c +5 -5
index a5df15a6cd51..ef8fb2f6e6e3 100644
@@ -602,14 +602,14 @@ static bool can_alloc_pages(void)
 		!IS_ENABLED(CONFIG_PREEMPT_RT);
 }
 
-struct page *bpf_alloc_page(int nid)
+struct page *bpf_alloc_page(int nid, bool sleepable)
 {
-	if (!can_alloc_pages())
+	if (!sleepable || !can_alloc_pages())
 		return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0);
 
 	return alloc_pages_node(nid,
 				GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT
-				| __GFP_NOWARN,
+				| __GFP_NOWARN | __GFP_RETRY_MAYFAIL,
 				0);
 }
 
@@ -624,13 +624,13 @@ void bpf_free_pages(struct llist_head *pages)
 }
 
 int bpf_alloc_pages(int nid, unsigned long nr_pages,
-		    struct llist_head *pages)
+		    struct llist_head *pages, bool sleepable)
 {
 	unsigned long i;
 	struct page *pg;
 
 	for (i = 0; i < nr_pages; i++) {
-		pg = bpf_alloc_page(nid);
+		pg = bpf_alloc_page(nid, sleepable);
 		if (!pg)
 			goto free_pages;
 		llist_add(&pg->pcp_llist, pages);
Signature · 1 line
2.52.0
author reviewer
#3 lore raw

PATCH RESEND v6 3/7 bpf: Add sleepable arena page allocation path

In reply to Emil Tsalapatis

The bpf_arena_alloc_pages() function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use of alloc_pages_nolock() in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.

Allocate the arena pages before taking the critical section and pass whether the caller can sleep to bpf_alloc_pages(). This lets sleepable callers use the blocking allocator while non-sleepable callers retain the no-lock allocation behavior.

Fixes b8467290edab ("bpf: arena: make arena kfuncs any context safe")
---
 kernel/bpf/arena.c | 73 +++++++++++++++++++++++++++++-----------------
 1 file changed, 46 insertions(+), 27 deletions(-)
kernel/bpf/arena.c +46 -27
index c556df7730c4..0ff707707da0 100644
@@ -713,6 +713,27 @@ static u64 clear_lo32(u64 val)
 	return val & ~(u64)~0U;
 }
 
+static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, long *pgoff)
+{
+	int ret;
+
+	/* Special case where user is requesting specific range. */
+	if (uaddr) {
+		ret = is_range_tree_set(&arena->rt, *pgoff, page_cnt);
+		if (ret)
+			return ret;
+		return range_tree_clear(&arena->rt, *pgoff, page_cnt);
+	}
+
+	ret = range_tree_find(&arena->rt, page_cnt);
+	if (ret < 0)
+		return ret;
+
+	*pgoff = ret;
+
+	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
+}
+
 /*
  * Allocate pages and vmap them into kernel vmalloc area.
  * Later the pages will be mmaped into user space vma.
@@ -730,6 +751,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	unsigned long flags;
 	long pgoff = 0;
 	u32 uaddr32;
+	long addr = 0;
 	int ret;
 
 	if (node_id != NUMA_NO_NODE &&
@@ -747,8 +769,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			/* requested address will be outside of user VMA */
 			return 0;
 	}
-
 	bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
+
+	ret = bpf_alloc_pages(node_id, page_cnt, &pages, sleepable);
+	if (ret)
+		goto out_memcg;
+
 	data.arena = arena;
 	data.pages = &pages;
 	data.i = 0;
@@ -756,25 +782,14 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
 		goto out_free_pages;
 
-	if (uaddr) {
-		ret = is_range_tree_set(&arena->rt, pgoff, page_cnt);
-		if (ret)
-			goto out_unlock_free_pages;
-		ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
-	} else {
-		ret = pgoff = range_tree_find(&arena->rt, page_cnt);
-		if (pgoff >= 0)
-			ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
+	ret = arena_adjust_tree(arena, uaddr, page_cnt, &pgoff);
+	if (ret) {
+		raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+		goto out_free_pages;
 	}
-	if (ret)
-		goto out_unlock_free_pages;
 
 	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
 
-	ret = bpf_alloc_pages(node_id, page_cnt, &pages, false);
-	if (ret)
-		goto out;
-
 	/*
 	 * Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
 	 * will not overflow 32-bit. Lower 32-bit need to represent
@@ -787,26 +802,30 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 				  page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
 	mapped = data.i;
 	if (ret)
-		goto out;
+		goto out_unmap;
 
 	flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
 	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
-	bpf_map_memcg_exit(old_memcg, new_memcg);
-	return clear_lo32(arena->user_vm_start) + uaddr32;
-out:
+
+	addr = clear_lo32(arena->user_vm_start) + uaddr32;
+	goto out_memcg;
+
+out_unmap:
+	/* Error handling: Undo partial mappings. */
+	flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
 	range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped);
 	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
-	if (mapped) {
-		flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
+	if (mapped)
 		arena_free_pages(arena, uaddr32, mapped, sleepable);
-	}
-	goto out_free_pages;
-out_unlock_free_pages:
-	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+
 out_free_pages:
+	/* Error handling: Free back any unmapped pages. */
 	bpf_free_pages(&pages);
+
+out_memcg:
 	bpf_map_memcg_exit(old_memcg, new_memcg);
-	return 0;
+
+	return addr;
 }
 
 /*
Signature · 1 line
2.52.0
author reviewer
#4 lore raw

PATCH RESEND v6 4/7 selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users

In reply to Emil Tsalapatis

We have relaxed the limitation of 1024-page batches for both sleepable and nonsleepable arena allocations by removing the intermediate page array. Add a test to confirm that large allocations succeed for both.

---
 .../bpf/progs/verifier_arena_large.c          | 64 +++++++++++++++----
 1 file changed, 50 insertions(+), 14 deletions(-)
tools/testing/selftests/bpf/progs/verifier_arena_large.c +50 -14
index 6ab8730d4878..dbdea14ca76f 100644
@@ -10,6 +10,9 @@
 #include <bpf_arena_common.h>
 
 #define ARENA_SIZE (1ull << 32)
+#define LARGE_PAGE_CNT 1025
+
+volatile int zero = 0;
 
 struct {
 	__uint(type, BPF_MAP_TYPE_ARENA);
@@ -284,6 +287,7 @@ int big_alloc2(void *ctx)
 	return 0;
 }
 
+/* Nonsleepable because it binds to a socket program. */
 SEC("socket")
 __success __retval(0)
 int big_alloc3(void *ctx)
@@ -291,24 +295,56 @@ int big_alloc3(void *ctx)
 #if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
 	char __arena *pages;
 	u64 i;
+	int err = 0;
 
-	/*
-	 * Allocate 2051 pages in one go to check how kmalloc_nolock() handles large requests.
-	 * Since kmalloc_nolock() can allocate up to 1024 struct page * at a time, this call should
-	 * result in three batches: two batches of 1024 pages each, followed by a final batch of 3
-	 * pages.
-	 */
-	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);
+	/* Verify that a nonsleepable allocation larger than 1024 pages succeeds. */
+	pages = bpf_arena_alloc_pages(&arena, NULL, LARGE_PAGE_CNT, NUMA_NO_NODE, 0);
 	if (!pages)
-		return 0;
+		return 1;
+
+	for (i = zero; i < LARGE_PAGE_CNT && can_loop; i++)
+		pages[i * PAGE_SIZE] = 123;
+
+	for (i = zero; i < LARGE_PAGE_CNT && can_loop; i++) {
+		if (pages[i * PAGE_SIZE] == 123)
+			continue;
+		err = 2;
+		break;
+	}
+
+	bpf_arena_free_pages(&arena, pages, LARGE_PAGE_CNT);
+	return err;
+#endif
+	return 0;
+}
 
-	bpf_for(i, 0, 2051)
-			pages[i * PAGE_SIZE] = 123;
-	bpf_for(i, 0, 2051)
-			if (pages[i * PAGE_SIZE] != 123)
-				return i;
+/* SYSCALL programs are always sleepable. */
+SEC("syscall")
+__success __retval(0)
+int big_alloc4(void *ctx)
+{
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+	char __arena *pages;
+	u64 i;
+	int err = 0;
+
+	/* Verify that a sleepable allocation larger than 1024 pages succeeds. */
+	pages = bpf_arena_alloc_pages(&arena, NULL, LARGE_PAGE_CNT, NUMA_NO_NODE, 0);
+	if (!pages)
+		return 1;
+
+	for (i = zero; i < LARGE_PAGE_CNT && can_loop; i++)
+		pages[i * PAGE_SIZE] = 123;
+
+	for (i = zero; i < LARGE_PAGE_CNT && can_loop; i++) {
+		if (pages[i * PAGE_SIZE] == 123)
+			continue;
+		err = 2;
+		break;
+	}
 
-	bpf_arena_free_pages(&arena, pages, 2051);
+	bpf_arena_free_pages(&arena, pages, LARGE_PAGE_CNT);
+	return err;
 #endif
 	return 0;
 }
Signature · 1 line
2.52.0
author reviewer
#5 lore raw

PATCH RESEND v6 5/7 bpf: Directly store kfunc desc index in instruction off field

In reply to Emil Tsalapatis

Currently, the verifier sort the kfunc desc table twice: Once during kfunc collection by function ID, useful during verification, and once by immediate address, useful during JIT bytecode lowering time. The latter sort can be removed by storing in the instruction the kfunc desc array index the desc is in and using that instead. Modify the JITs to follow the same convention.

This change simplifies the subsequent patch that adds per-call site function specialization.

---
 include/linux/bpf.h          |  4 +--
 include/linux/bpf_verifier.h |  7 ++--
 kernel/bpf/fixups.c          | 70 +++++-------------------------------
 kernel/bpf/verifier.c        | 25 ++++++++++---
 4 files changed, 33 insertions(+), 73 deletions(-)
include/linux/bpf.h +2 -2
index 670eb9f3f20a..eed7f8f1e417 100644
@@ -3301,7 +3301,7 @@ const struct btf_func_model *
 bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
 			 const struct bpf_insn *insn);
 int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
-		       u16 btf_fd_idx, u8 **func_addr);
+		       u16 desc_idx, u8 **func_addr);
 
 struct bpf_core_ctx {
 	struct bpf_verifier_log *log;
@@ -3644,7 +3644,7 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
 
 static inline int
 bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
-		   u16 btf_fd_idx, u8 **func_addr)
+		   u16 desc_idx, u8 **func_addr)
 {
 	return -ENOTSUPP;
 }
include/linux/bpf_verifier.h +3 -4
index c775bd757706..5cbae5d05fe7 100644
@@ -1784,12 +1784,12 @@ enum bpf_reg_arg_type {
 };
 
 #define MAX_KFUNC_DESCS 256
+static_assert(MAX_KFUNC_DESCS <= S16_MAX + 1);
 
 struct bpf_kfunc_desc {
 	struct btf_func_model func_model;
 	struct bpf_func_proto proto;
 	u32 func_id;
-	s32 imm;
 	u16 offset;
 	unsigned long addr;
 };
@@ -1797,9 +1797,8 @@ struct bpf_kfunc_desc {
 struct bpf_kfunc_desc_tab {
 	u32 nr_descs;
 	/* Sorted by func_id (BTF ID) and offset (fd_array offset) during
-	 * verification. JITs do lookups by bpf_insn, where func_id may not be
-	 * available, therefore at the end of verification do_misc_fixups()
-	 * sorts this by imm and offset.
+	 * verification. JITs use the descriptor index stored in the finalized
+	 * call's off field.
 	 *
 	 * Grown one entry at a time by bpf_add_kfunc_call().
 	 */
kernel/bpf/fixups.c +8 -62

103 lines not on this page · raw message

kernel/bpf/verifier.c +20 -5
index 03dbc0e00398..8183a8f22ed5 100644
@@ -2584,12 +2584,16 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)
 }
 
 int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
-		       u16 btf_fd_idx, u8 **func_addr)
+		       u16 desc_idx, u8 **func_addr)
 {
+	struct bpf_kfunc_desc_tab *tab;
 	const struct bpf_kfunc_desc *desc;
 
-	desc = find_kfunc_desc(prog, func_id, btf_fd_idx);
-	if (!desc)
+	tab = prog->aux->kfunc_tab;
+	if (desc_idx >= tab->nr_descs)
+		return -EFAULT;
+	desc = &tab->descs[desc_idx];
+	if (desc->func_id != func_id)
 		return -EFAULT;
 
 	*func_addr = (u8 *)desc->addr;
@@ -22079,6 +22083,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
 	struct bpf_kfunc_desc *desc;
+	unsigned long call_imm;
+	u16 desc_idx;
 	int err;
 
 	if (!insn->imm) {
@@ -22098,13 +22104,22 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			     insn->imm);
 		return -EFAULT;
 	}
+	desc_idx = desc - env->prog->aux->kfunc_tab->descs;
 
 	err = specialize_kfunc(env, desc, insn_idx);
 	if (err)
 		return err;
 
-	if (!bpf_jit_supports_far_kfunc_call())
-		insn->imm = BPF_CALL_IMM(desc->addr);
+	if (!bpf_jit_supports_far_kfunc_call()) {
+		call_imm = BPF_CALL_IMM(desc->addr);
+		if ((unsigned long)(s32)call_imm != call_imm) {
+			verbose(env, "address of kernel func_id %u is out of range\n",
+				desc->func_id);
+			return -EINVAL;
+		}
+		insn->imm = call_imm;
+	}
+	insn->off = desc_idx;
 
 	if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
 		struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
Signature · 1 line
2.52.0
#6 lore raw

Re: [RESEND PATCH bpf-next v6 5/7] bpf: Directly store kfunc desc index in instruction off field

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpf: JIT compilation fails for kfuncs emitted in prologues --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: JIT compilation fails for kfuncs emitted in prologues
--

commit ba562be891ad9f52f9742001cfab2a791d6aefb8
Author: Emil Tsalapatis <[email protected]>

bpf: Directly store kfunc desc index in instruction off field

This patch modifies the verifier to store the kfunc descriptor index directly in the instruction's off field. It removes the need to sort the descriptor table twice, simplifying subsequent function specialization patches.

Quoted 4 lines
kernel/bpf/fixups.c +0 -0
index 37cf130ebb57b..cb7219ff68bd1 100644

[ ... ]

Quoted 4 lines

const struct btf_func_model * bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn)
{

[ ... ]

Quoted 6 lines
kernel/bpf/fixups.c +0 -5
 	tab = prog->aux->kfunc_tab;
-	res = bsearch(&desc, tab->descs, tab->nr_descs,
-		      sizeof(tab->descs[0]), kfunc_desc_cmp_by_imm_off);
-
-	return res ? &res->func_model : NULL;
-}

[ ... ]

kernel/bpf/fixups.c int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn) · line 132

Quoted 7 lines
kernel/bpf/fixups.c +5 -0
+	if (insn->off < 0 || insn->off >= tab->nr_descs)
+		return NULL;
 
+	res = &tab->descs[insn->off];
 	if (bpf_jit_supports_far_kfunc_call()) {
+		if (res->func_id != insn->imm)
+			return NULL;

[Severity: Medium]
Will this lead to a JIT failure for kfuncs emitted in prologues or epilogues?

When bpf_convert_ctx_accesses() calls gen_prologue(), it can emit a kfunc call where insn->off is 0. This is then processed by add_kfunc_in_insns(), which calls bpf_add_kfunc_call() to register the kfunc:

kernel/bpf/fixups.c:add_kfunc_in_insns() {
	...
	for (i = 0; i < cnt; i++, insn++) {
		if (bpf_pseudo_kfunc_call(insn)) {
			ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
			if (ret < 0)
				return ret;
		}
	}
	...
}

However, add_kfunc_in_insns() does not update insn->off to the descriptor index returned by bpf_add_kfunc_call().

Later, when the JIT compiler calls bpf_jit_find_kfunc_model() and uses insn->off (which is still 0) to fetch the descriptor, could it fetch the wrong descriptor and abort the compilation due to a func_id mismatch?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#7 lore raw

Re: [RESEND PATCH bpf-next v6 5/7] bpf: Directly store kfunc desc index in instruction off field

In reply to Emil Tsalapatis
TL;DR: Commit message only: it says the JITs are modified, but no arch JIT changes; only the generic lookup helpers use the new insn->off index. Possibly a leftover sentence; suggests rewording.
Quoted 1 line

bpf: Directly store kfunc desc index in instruction off field

TL;DR: Commit message only: it says the JITs are modified, but no arch JIT changes; only the generic lookup helpers use the new insn->off index. Possibly a leftover sentence; suggests rewording.

Quoted 9 lines

bpf: Directly store kfunc desc index in instruction off field

Currently, the verifier sort the kfunc desc table twice: Once during kfunc collection by function ID, useful during verification, and once by immediate address, useful during JIT bytecode lowering time. The latter sort can be removed by storing in the instruction the kfunc desc array index the desc is in and using that instead. Modify the JITs to follow the same convention.

Quoted 4 lines

This change simplifies the subsequent patch that adds per-call site function specialization.

The changelog says "Modify the JITs to follow the same convention.", but the patch only touches include/linux/bpf.h, include/linux/bpf_verifier.h, kernel/bpf/fixups.c and kernel/bpf/verifier.c. No arch JIT is modified.

The arch JITs (x86 do_jit, x86_32 emit_kfunc_call, arm64 build_insn, riscv bpf_jit_emit_insn, s390 bpf_jit_insn, loongarch, powerpc prepare_for_kfunc_call) are unchanged. They pick up the new insn->off meaning only through the generic helpers bpf_jit_find_kfunc_model() and bpf_jit_get_func_addr() -> bpf_get_kfunc_addr(), which this patch rewrites to index tab->descs[insn->off].

Is that sentence left over from an earlier version of the patch? Could it be reworded to say that the generic JIT lookup helpers (bpf_jit_find_kfunc_model() and bpf_get_kfunc_addr()) now use the stored index?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36999828295

author reviewer
#8 lore raw

PATCH RESEND v6 6/7 bpf: Support per-call-site kfunc specialization

In reply to Emil Tsalapatis

specialize_kfunc() currently updates the canonical kfunc descriptor in place. Different call sites therefore cannot select different specializations of the same kfunc, and the selected target depends on verification order.

Keep canonical descriptors unchanged for verifier lookups. Specialize a copy for each call site, then reuse or append an immutable target descriptor and store its index in the finalized call instruction.

Allow space for one canonical and one specialized target per kfunc. This is conservative because most kfuncs do not specialize. Adding specialized kfunc versions does not require resorting the array because lookups are only used for deduplication and func_model lookup. Deduplication for specialized kfuncs is handled by the specialization process itself, while all specializations of a function share the same proto and func_model.

---
 include/linux/bpf_verifier.h | 12 +++--
 kernel/bpf/verifier.c        | 88 +++++++++++++++++++++++++++++++++---
 2 files changed, 91 insertions(+), 9 deletions(-)
include/linux/bpf_verifier.h +9 -3
index 5cbae5d05fe7..7bdbda7764c7 100644
@@ -1784,7 +1784,9 @@ enum bpf_reg_arg_type {
 };
 
 #define MAX_KFUNC_DESCS 256
-static_assert(MAX_KFUNC_DESCS <= S16_MAX + 1);
+/* Each kfunc can have its canonical and one specialized call target. */
+#define MAX_KFUNC_CALL_DESCS (MAX_KFUNC_DESCS * 2)
+static_assert(MAX_KFUNC_CALL_DESCS <= S16_MAX + 1);
 
 struct bpf_kfunc_desc {
 	struct btf_func_model func_model;
@@ -1796,11 +1798,15 @@ struct bpf_kfunc_desc {
 
 struct bpf_kfunc_desc_tab {
 	u32 nr_descs;
+	u32 nr_base_descs;
 	/* Sorted by func_id (BTF ID) and offset (fd_array offset) during
 	 * verification. JITs use the descriptor index stored in the finalized
-	 * call's off field.
+	 * call's off field. The first nr_base_descs entries are the canonical
+	 * descriptors used for verifier lookups. Call specialization may append
+	 * immutable descriptors for additional targets.
 	 *
-	 * Grown one entry at a time by bpf_add_kfunc_call().
+	 * Grown one entry at a time by bpf_add_kfunc_call() and during
+	 * call specialization.
 	 */
 	struct bpf_kfunc_desc descs[];
 };
kernel/bpf/verifier.c +82 -6

140 lines not on this page · raw message

Signature · 1 line
2.52.0
author reviewer
#9 lore raw

PATCH RESEND v6 7/7 selftests/bpf: Test per-call site function specialization

In reply to Emil Tsalapatis

Add a test to ensure function call specialization is done per-call site. Use bpf_dynptr_from_file that has observably different behavior between its sleepable and nonsleepable versions. The sleepable path fails in the sleepable __kernel_read() call with -EIO, while the nonsleepable fails in the page-cache lookup path with -EFAULT. Test that whatever the order the nonsleepable and sleepable calls are made in the program, both call sites use the correct specialized kfunc.

---
 .../selftests/bpf/prog_tests/file_reader.c    |  15 ++
 .../testing/selftests/bpf/progs/file_reader.c | 129 ++++++++++++++++++
 2 files changed, 144 insertions(+)
tools/testing/selftests/bpf/prog_tests/file_reader.c +15 -0
index 48aae7ea0e4b..e59c6c87e9d5 100644
@@ -7,10 +7,12 @@
 #include "file_reader_fail.skel.h"
 #include <dlfcn.h>
 #include <sys/mman.h>
+#include <sys/stat.h>
 
 const char *user_ptr = "hello world";
 char file_contents[256000];
 void *addr;
+__u64 beyond_eof_offset;
 
 void *get_executable_base_addr(void)
 {
@@ -26,12 +28,18 @@ void *get_executable_base_addr(void)
 
 static int initialize_file_contents(void)
 {
+	struct stat st;
 	int fd, page_sz = sysconf(_SC_PAGESIZE);
 	ssize_t n = 0, cur;
 
 	fd = open("/proc/self/exe", O_RDONLY);
 	if (!ASSERT_OK_FD(fd, "Open /proc/self/exe\n"))
 		return 1;
+	if (!ASSERT_OK(fstat(fd, &st), "fstat /proc/self/exe")) {
+		close(fd);
+		return 1;
+	}
+	beyond_eof_offset = st.st_size + (1ULL << 30);
 
 	do {
 		cur = read(fd, file_contents + n, sizeof(file_contents) - n);
@@ -75,6 +83,7 @@ static void run_test(const char *prog_name)
 
 	memcpy(skel->bss->user_buf, file_contents, sizeof(file_contents));
 	skel->bss->pid = getpid();
+	skel->bss->beyond_eof_offset = beyond_eof_offset;
 
 	err = file_reader__load(skel);
 	if (!ASSERT_OK(err, "file_reader__load"))
@@ -110,6 +119,12 @@ void test_file_reader(void)
 	if (test__start_subtest("on_open_validate_file_read"))
 		run_test("on_open_validate_file_read");
 
+	if (test__start_subtest("on_open_non_sleepable_first"))
+		run_test("on_open_non_sleepable_first");
+
+	if (test__start_subtest("on_open_sleepable_first"))
+		run_test("on_open_sleepable_first");
+
 	if (test__start_subtest("negative"))
 		RUN_TESTS(file_reader_fail);
 }
tools/testing/selftests/bpf/progs/file_reader.c +129 -0

150 lines not on this page · raw message

Signature · 1 line
2.52.0
#10 lore raw
In reply to Emil Tsalapatis
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 6 lines · On Fri, 2 Oct 2026 10:52:11 +0000 you wrote:

The arena_alloc_pages() call takes a sleepable argument based on whether its caller is a sleepable BPF function. This flag, along with the context the kfunc is called in, decides whether the call will try to fulfill the allocation using the regular or the _nolock variant of the alloc_pages API, by means of bpf_map_alloc_pages().

Quoted 7 lines

However, the arena_alloc_pages() call currently only makes allocations inside an IRQ-disabled critical section. This forces all allocations to use the _nolock() API, which may eagerly fail where the regular variant would eventually succeed. There have been reports of this happening for sched-ext schedulers.

[...]

Here is the summary with links:
- [RESEND,bpf-next,v6,1/7] bpf: Use an llist for page allocations

    https://git.kernel.org/bpf/bpf-next/c/11de29f38a34
  - [RESEND,bpf-next,v6,2/7] bpf: Add sleepable argument to bpf_alloc_pages()
    https://git.kernel.org/bpf/bpf-next/c/cbe7d2bf7355
  - [RESEND,bpf-next,v6,3/7] bpf: Add sleepable arena page allocation path
    https://git.kernel.org/bpf/bpf-next/c/5316ede1f5d0
  - [RESEND,bpf-next,v6,4/7] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
    https://git.kernel.org/bpf/bpf-next/c/0f6512272538
  - [RESEND,bpf-next,v6,5/7] bpf: Directly store kfunc desc index in instruction off field
    https://git.kernel.org/bpf/bpf-next/c/baf4e66f038f
  - [RESEND,bpf-next,v6,6/7] bpf: Support per-call-site kfunc specialization
    https://git.kernel.org/bpf/bpf-next/c/23290213e1ab
  - [RESEND,bpf-next,v6,7/7] selftests/bpf: Test per-call site function specialization
    https://git.kernel.org/bpf/bpf-next/c/fc38f07781ca

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html