Skip to contentSkip to conversation

Remove KF_SLEEPABLE from arena kfuncs

PATCH Superseded Puranjay Mohan 4 participants 13 messages Nov 11, 2025 – Nov 13, 2025

Subsystems (4)

v1 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

cover Remove KF_SLEEPABLE from arena kfuncs

This set allows arena kfuncs to be called from non-sleepable contexts. It is acheived by the following changes:

The range_tree is now protected with a rqspinlock and not a mutex, this change is enough to make bpf_arena_reserve_pages() any context safe.

bpf_arena_alloc_pages() had four points where it could sleep:

1. Mutex to protect range_tree: now replaced with rqspinlock

2. kvcalloc() for allocations: now replaced with kmalloc_nolock()

3. Allocating pages with bpf_map_alloc_pages(): this already calls alloc_pages_nolock() in non-sleepable contexts and therefore is safe.

4. Setting up kernel page tables with vm_area_map_pages():
vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. Now, at arena creation time populate all page table levels except the last level when new pages need to be inserted call apply_to_page_range() again which will only set_pte_at() those pages and will not allocate memory.

The above four changes make bpf_arena_alloc_pages() any context safe.

bpf_arena_free_pages() has to do the following steps:

1. Update the range_tree
2. vm_area_unmap_pages(): to unmap pages from kernel vm_area
3. flush the tlb: done by 2, already.
4. zap_pages(): to unmap pages from user page tables
5. free pages.

The third patch in this set makes bpf_arena_free_pages() polymorphic using the specialize_kfunc() mechanism. When called from a sleepable context, arena_free_pages() remains mostly unchanged except the following:
1. rqspinlock is taken now instead of the mutex for the range tree
2. Instead of using vm_area_unmap_pages() that can free intermediate page table levels, apply_to_existing_page_range() with a callback is used that only does pte_clear() on the last level and leaves the intermediate page table levels intact. This is needed to make sure that bpf_arena_alloc_pages() can safely do set_pte_at() without allocating intermediate page tables.

When arena_free_pages() is called from a non-sleepable context or it fails to acquire the rqspinlock in the sleepable case, a lock-less list of struct arena_free_span is used to queue the uaddr and page cnt. kmalloc_nolock() is used to allocate this arena_free_span, this can fail but we need to make this trade-off for frees done from non-sleepable context.

arena_free_pages() then raises an irq_work whose handler in turn schedules work that iterate this list and clears ptes, flushes tlbs, zap pages, and frees pages for the queued uaddr and page cnts.

apply_range_clear_cb() with apply_to_existing_page_range() is used to clear PTEs and collect pages to be freed, struct llist_node pcp_llist; in the struct page is used to do this.

The arena selftest fails to load on s390x, this is due to an unrelated bug in the verifier that is being exposed by the selftest that I here. I have already sent a patch[1] to fix this.

[1] https://lore.kernel.org/all/[email protected]/

Puranjay Mohan (4):
bpf: arena: populate vm_area without allocating memory
bpf: arena: use kmalloc_nolock() in place of kvcalloc()
bpf: arena: make arena kfuncs any context safe
selftests: bpf: test non-sleepable arena allocations

 include/linux/bpf.h                           |   2 +
 kernel/bpf/arena.c                            | 290 +++++++++++++++---
 kernel/bpf/verifier.c                         |   5 +
 .../selftests/bpf/prog_tests/arena_list.c     |  20 +-
 .../testing/selftests/bpf/progs/arena_list.c  |  11 +
 .../selftests/bpf/progs/verifier_arena.c      | 185 +++++++++++
 6 files changed, 472 insertions(+), 41 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/4 bpf: arena: populate vm_area without allocating memory

vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. In order to make bpf_arena_alloc_pages() kfunc non-sleepable change bpf arena to populate pages without allocating memory:
- at arena creation time populate all page table levels except the last level
- when new pages need to be inserted call apply_to_page_range() again with apply_range_set_cb() which will only set_pte_at() those pages and will not allocate memory.
- when freeing pages call apply_to_existing_page_range with apply_range_clear_cb() to clear the pte for the page to be removed. This doesn't free intermediate page table levels.

---
 kernel/bpf/arena.c | 74 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 68 insertions(+), 6 deletions(-)
kernel/bpf/arena.c +68 -6
index 1074ac4459f2..dd5100a2f93c 100644
@@ -92,6 +92,63 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)
 	return (u32)(uaddr - (u32)arena->user_vm_start) >> PAGE_SHIFT;
 }
 
+struct apply_range_data {
+	struct page **pages;
+	int i;
+};
+
+static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
+{
+	struct apply_range_data *d = data;
+	struct page *page;
+
+	if (!data)
+		return 0;
+	/* sanity check */
+	if (unlikely(!pte_none(ptep_get(pte))))
+		return -EBUSY;
+
+	page = d->pages[d->i++];
+	/* paranoia, similar to vmap_pages_pte_range() */
+	if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
+		return -EINVAL;
+
+	set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));
+	return 0;
+}
+
+static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
+{
+	struct mm_struct *mm = &init_mm;
+	pte_t old_pte;
+	struct page *page;
+
+	/* sanity check */
+	old_pte = ptep_get(pte);
+	if (pte_none(old_pte) || !pte_present(old_pte))
+		return 0; /* nothing to do */
+
+	/* get page and free it */
+	page = pte_page(old_pte);
+	if (WARN_ON_ONCE(!page))
+		return -EINVAL;
+
+	pte_clear(mm, addr, pte);
+
+	/* ensure no stale TLB entries */
+	flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
+
+	__free_page(page);
+
+	return 0;
+}
+
+static int populate_pgtable_except_pte(struct bpf_arena *arena)
+{
+	return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
+				   KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);
+}
+
 static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 {
 	struct vm_struct *kern_vm;
@@ -144,6 +201,9 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 		goto err;
 	}
 	mutex_init(&arena->lock);
+	err = populate_pgtable_except_pte(arena);
+	if (err)
+		goto err;
 
 	return &arena->map;
 err:
@@ -286,6 +346,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 	if (ret)
 		return VM_FAULT_SIGSEGV;
 
+	struct apply_range_data data = { .pages = &page, .i = 0 };
 	/* Account into memcg of the process that created bpf_arena */
 	ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
 	if (ret) {
@@ -293,7 +354,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 		return VM_FAULT_SIGSEGV;
 	}
 
-	ret = vm_area_map_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE, &page);
+	ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
 	if (ret) {
 		range_tree_set(&arena->rt, vmf->pgoff, 1);
 		__free_page(page);
@@ -428,7 +489,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	/* user_vm_end/start are fixed before bpf prog runs */
 	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;
 	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
-	struct page **pages;
+	struct page **pages = NULL;
 	long pgoff = 0;
 	u32 uaddr32;
 	int ret, i;
@@ -465,6 +526,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	if (ret)
 		goto out_free_pages;
 
+	struct apply_range_data data = { .pages = pages, .i = 0 };
 	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);
 	if (ret)
 		goto out;
@@ -477,8 +539,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	 * kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow
 	 * lower 32-bit and it's ok.
 	 */
-	ret = vm_area_map_pages(arena->kern_vm, kern_vm_start + uaddr32,
-				kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE, pages);
+	ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32,
+				  page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
 	if (ret) {
 		for (i = 0; i < page_cnt; i++)
 			__free_page(pages[i]);
@@ -545,8 +607,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
 			 * page_cnt is big it's faster to do the batched zap.
 			 */
 			zap_pages(arena, full_uaddr, 1);
-		vm_area_unmap_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE);
-		__free_page(page);
+		apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
+					     apply_range_clear_cb, NULL);
 	}
 }
 
Signature · 1 line
2.47.3
#2 lore raw

Re: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory

Does this leak the arena and range_tree on error? The error path at the range_tree_set() failure (above these lines) calls bpf_map_area_free(arena) before goto err, but this new error path doesn't…
Quoted 4 lines
kernel/bpf/arena.c +0 -0
index 1074ac445..dd5100a2f 100644

[ ... ]

Quoted 64 lines
kernel/bpf/arena.c +57 -0
@@ -92,6 +92,63 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)
 	return (u32)(uaddr - (u32)arena->user_vm_start) >> PAGE_SHIFT;
 }
 
+struct apply_range_data {
+	struct page **pages;
+	int i;
+};
+
+static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
+{
+	struct apply_range_data *d = data;
+	struct page *page;
+
+	if (!data)
+		return 0;
+	/* sanity check */
+	if (unlikely(!pte_none(ptep_get(pte))))
+		return -EBUSY;
+
+	page = d->pages[d->i++];
+	/* paranoia, similar to vmap_pages_pte_range() */
+	if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
+		return -EINVAL;
+
+	set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));
+	return 0;
+}
+
+static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
+{
+	struct mm_struct *mm = &init_mm;
+	pte_t old_pte;
+	struct page *page;
+
+	/* sanity check */
+	old_pte = ptep_get(pte);
+	if (pte_none(old_pte) || !pte_present(old_pte))
+		return 0; /* nothing to do */
+
+	/* get page and free it */
+	page = pte_page(old_pte);
+	if (WARN_ON_ONCE(!page))
+		return -EINVAL;
+
+	pte_clear(mm, addr, pte);
+
+	/* ensure no stale TLB entries */
+	flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
+
+	__free_page(page);
+
+	return 0;
+}
+
+static int populate_pgtable_except_pte(struct bpf_arena *arena)
+{
+	return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
+				   KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);
+}
+
 static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 {
 	struct vm_struct *kern_vm;

[ ... ]

kernel/bpf/arena.c static struct bpf_map *arena_map_alloc(union bpf_attr *attr) · line 206

Quoted 7 lines
kernel/bpf/arena.c +3 -0
@@ -144,6 +201,9 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
 		goto err;
 	}
 	mutex_init(&arena->lock);
+	err = populate_pgtable_except_pte(arena);
+	if (err)
+		goto err;

Does this leak the arena and range_tree on error? The error path at the range_tree_set() failure (above these lines) calls bpf_map_area_free(arena) before goto err, but this new error path doesn't clean up arena or call range_tree_destroy() before jumping to err.

Quoted 3 lines
kernel/bpf/arena.c +0 -0
 	return &arena->map;
 err:

[ ... ]

Quoted 17 lines
kernel/bpf/arena.c +1 -0
@@ -286,6 +346,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 	if (ret)
 		return VM_FAULT_SIGSEGV;
 
+	struct apply_range_data data = { .pages = &page, .i = 0 };
 	/* Account into memcg of the process that created bpf_arena */
 	ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
 	if (ret) {
kernel/bpf/arena.c +1 -1
 		range_tree_set(&arena->rt, vmf->pgoff, 1);
 		return VM_FAULT_SIGSEGV;
 	}
 
-	ret = vm_area_map_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE, &page);
+	ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
 	if (ret) {
 		range_tree_set(&arena->rt, vmf->pgoff, 1);
 		__free_page(page);

[ ... ]

Quoted 9 lines
kernel/bpf/arena.c +1 -1
@@ -428,7 +489,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	/* user_vm_end/start are fixed before bpf prog runs */
 	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;
 	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
-	struct page **pages;
+	struct page **pages = NULL;
 	long pgoff = 0;
 	u32 uaddr32;
 	int ret, i;

[ ... ]

Quoted 19 lines
kernel/bpf/arena.c +3 -2
@@ -465,6 +526,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	if (ret)
 		goto out_free_pages;
 
+	struct apply_range_data data = { .pages = pages, .i = 0 };
 	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);
 	if (ret)
 		goto out;
@@ -477,8 +539,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 	 * kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow
 	 * lower 32-bit and it's ok.
 	 */
-	ret = vm_area_map_pages(arena->kern_vm, kern_vm_start + uaddr32,
-				kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE, pages);
+	ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32,
+				  page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
 	if (ret) {
 		for (i = 0; i < page_cnt; i++)
 			__free_page(pages[i]);

[ ... ]

kernel/bpf/arena.c static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) · line 611

Quoted 8 lines
kernel/bpf/arena.c +2 -2
@@ -545,8 +607,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
 			 * page_cnt is big it's faster to do the batched zap.
 			 */
 			zap_pages(arena, full_uaddr, 1);
-		vm_area_unmap_pages(arena->kern_vm, kaddr, kaddr + PAGE_SIZE);
-		__free_page(page);
+		apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
+					     apply_range_clear_cb, NULL);

Can this free pages multiple times? This is inside a loop that iterates page_cnt times, with kaddr incrementing by PAGE_SIZE each iteration. But apply_to_existing_page_range() is called with size page_cnt << PAGE_SHIFT, which processes page_cnt pages starting from kaddr.

On the first iteration (i=0), it processes pages [0..page_cnt-1]. On the second iteration (i=1, kaddr+=PAGE_SIZE), it processes pages [1..page_cnt], overlapping with the first iteration.

Since apply_range_clear_cb() calls __free_page() for each page, page 0 gets freed page_cnt times, page 1 gets freed page_cnt-1 times, etc.

The original code called vm_area_unmap_pages() with size PAGE_SIZE, processing one page per iteration. Should the new code use PAGE_SIZE instead of page_cnt << PAGE_SHIFT?

kernel/bpf/arena.c static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) · line 614

Quoted 3 lines
kernel/bpf/arena.c +0 -0
 	}
 }
 

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19272481461

#3 lore raw

Re: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory

In reply to Puranjay Mohan
kernel test robot noticed the following build errors:

Hi Puranjay,

kernel test robot noticed the following build errors:

[auto build test ERROR on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Puranjay-Mohan/bpf-arena-populate-vm_area-without-allocating-memory/20251112-004253
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20251111163424.16471-2-puranjay%40kernel.org
patch subject: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory
config: loongarch-defconfig (https://download.01.org/0day-ci/archive/20251112/[email protected]/config)
compiler: clang version 19.1.7 (https://github.com/llvm/llvm-project cd708029e0b2869e80abe31ddb175f7c35361f90)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251112/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> kernel/bpf/arena.c:139:2: error: call to undeclared function 'flush_tlb_kernel_range'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     139 |         flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
         |         ^

1 error generated.

vim +/flush_tlb_kernel_range +139 kernel/bpf/arena.c

119
120 static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
121 {
122 struct mm_struct *mm = &init_mm;
123 pte_t old_pte;
124 struct page *page;
125
126 /* sanity check */
127 old_pte = ptep_get(pte);
128 if (pte_none(old_pte) || !pte_present(old_pte))
129 return 0; /* nothing to do */
130
131 /* get page and free it */
132 page = pte_page(old_pte);
133 if (WARN_ON_ONCE(!page))
134 return -EINVAL;
135
136 pte_clear(mm, addr, pte);
137
138 /* ensure no stale TLB entries */

 > 139		flush_tlb_kernel_range(addr, addr + PAGE_SIZE);

140
141 __free_page(page);
142
143 return 0;
144 }
145

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#4 lore raw

Re: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory

In reply to Puranjay Mohan
kernel test robot noticed the following build warnings:

Hi Puranjay,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Puranjay-Mohan/bpf-arena-populate-vm_area-without-allocating-memory/20251112-004253
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20251111163424.16471-2-puranjay%40kernel.org
patch subject: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory
config: arm64-randconfig-003-20251112 (https://download.01.org/0day-ci/archive/20251113/[email protected]/config)
compiler: aarch64-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251113/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

kernel/bpf/arena.c: In function 'apply_range_clear_cb':

>> kernel/bpf/arena.c:122:20: warning: unused variable 'mm' [-Wunused-variable]
     struct mm_struct *mm = &init_mm;
                       ^~

vim +/mm +122 kernel/bpf/arena.c

119
120 static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
121 {

 > 122		struct mm_struct *mm = &init_mm;

123 pte_t old_pte;
124 struct page *page;
125
126 /* sanity check */
127 old_pte = ptep_get(pte);
128 if (pte_none(old_pte) || !pte_present(old_pte))
129 return 0; /* nothing to do */
130
131 /* get page and free it */
132 page = pte_page(old_pte);
133 if (WARN_ON_ONCE(!page))
134 return -EINVAL;
135
136 pte_clear(mm, addr, pte);
137
138 /* ensure no stale TLB entries */
139 flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
140
141 __free_page(page);
142
143 return 0;
144 }
145

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#5 lore raw

Re: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory

In reply to Puranjay Mohan
kernel test robot noticed the following build errors:

Hi Puranjay,

kernel test robot noticed the following build errors:

[auto build test ERROR on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Puranjay-Mohan/bpf-arena-populate-vm_area-without-allocating-memory/20251112-004253
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20251111163424.16471-2-puranjay%40kernel.org
patch subject: [PATCH bpf-next 1/4] bpf: arena: populate vm_area without allocating memory
config: loongarch-randconfig-002-20251112 (https://download.01.org/0day-ci/archive/20251112/[email protected]/config)
compiler: loongarch64-linux-gcc (GCC) 13.4.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251112/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

kernel/bpf/arena.c: In function 'apply_range_clear_cb':

>> kernel/bpf/arena.c:139:9: error: implicit declaration of function 'flush_tlb_kernel_range' [-Werror=implicit-function-declaration]
     139 |         flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
         |         ^~~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/flush_tlb_kernel_range +139 kernel/bpf/arena.c

119
120 static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
121 {
122 struct mm_struct *mm = &init_mm;
123 pte_t old_pte;
124 struct page *page;
125
126 /* sanity check */
127 old_pte = ptep_get(pte);
128 if (pte_none(old_pte) || !pte_present(old_pte))
129 return 0; /* nothing to do */
130
131 /* get page and free it */
132 page = pte_page(old_pte);
133 if (WARN_ON_ONCE(!page))
134 return -EINVAL;
135
136 pte_clear(mm, addr, pte);
137
138 /* ensure no stale TLB entries */

 > 139		flush_tlb_kernel_range(addr, addr + PAGE_SIZE);

140
141 __free_page(page);
142
143 return 0;
144 }
145

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#6 lore raw

PATCH 2/4 bpf: arena: use kmalloc_nolock() in place of kvcalloc()

In reply to Puranjay Mohan

To make arena_alloc_pages() safe to be called from any context, replace kvcalloc() with kmalloc_nolock() so as it doesn't sleep or take any locks.

---
 kernel/bpf/arena.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/bpf/arena.c +3 -4
index dd5100a2f93c..9d8a8eb447fe 100644
@@ -506,8 +506,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			return 0;
 	}
 
-	/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */
-	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL);
+	pages = kmalloc_nolock(page_cnt * sizeof(struct page *), __GFP_ZERO, -1);
 	if (!pages)
 		return 0;
 
@@ -546,12 +545,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			__free_page(pages[i]);
 		goto out;
 	}
-	kvfree(pages);
+	kfree_nolock(pages);
 	return clear_lo32(arena->user_vm_start) + uaddr32;
 out:
 	range_tree_set(&arena->rt, pgoff, page_cnt);
 out_free_pages:
-	kvfree(pages);
+	kfree_nolock(pages);
 	return 0;
 }
 
Signature · 1 line
2.47.3
#7 lore raw

Re: [PATCH bpf-next 2/4] bpf: arena: use kmalloc_nolock() in place of kvcalloc()

Can kmalloc_nolock() handle the same allocation sizes as kvcalloc()?

kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 509

Quoted 11 lines
kernel/bpf/arena.c +1 -2
index dd5100a2f..9d8a8eb44 100644
@@ -506,8 +506,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			return 0;
 	}
 
-	/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */
-	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL);
+	pages = kmalloc_nolock(page_cnt * sizeof(struct page *), __GFP_ZERO, -1);

Can kmalloc_nolock() handle the same allocation sizes as kvcalloc()?

Looking at arena_alloc_pages(), page_cnt can be up to the full arena size (4GB / 4KB = 1M pages). This means the allocation size can reach 1M * 8 = 8MB for the pages array.

In mm/slub.c:5686, kmalloc_nolock() returns NULL for allocations larger than KMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE * 2) = 8KB on systems with 4KB pages.

kvcalloc() could handle larger allocations by falling back to vmalloc. With kmalloc_nolock(), arena allocations requiring more than 1024 pages will fail where they previously succeeded.

For example, arena_alloc_pages(arena, 0, 1025, node_id) allocates 1025 * 8 = 8200 bytes, which exceeds the 8KB limit and will return NULL.

Quoted 3 lines
kernel/bpf/arena.c +0 -0
 	if (!pages)
 		return 0;
 

[ ... ]

Quoted 4 lines
kernel/bpf/arena.c +0 -0
@@ -546,12 +545,12 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			__free_page(pages[i]);
 		goto out;
 	}

kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 555

Quoted 10 lines
kernel/bpf/arena.c +2 -2
-	kvfree(pages);
+	kfree_nolock(pages);
 	return clear_lo32(arena->user_vm_start) + uaddr32;
 out:
 	range_tree_set(&arena->rt, pgoff, page_cnt);
 out_free_pages:
-	kvfree(pages);
+	kfree_nolock(pages);
 	return 0;
 }

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19272481461

#8 lore raw

Re: [PATCH bpf-next 2/4] bpf: arena: use kmalloc_nolock() in place of kvcalloc()

Quoted 19 lines · On Tue, Nov 11, 2025 at 9:01 AM <[email protected]> wrote:

kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 509

Quoted 12 lines
kernel/bpf/arena.c +1 -2
index dd5100a2f..9d8a8eb44 100644
@@ -506,8 +506,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
                      return 0;
      }
 
-     /* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */
-     pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL);
+     pages = kmalloc_nolock(page_cnt * sizeof(struct page *), __GFP_ZERO, -1);

Can kmalloc_nolock() handle the same allocation sizes as kvcalloc()?

Looking at arena_alloc_pages(), page_cnt can be up to the full arena size (4GB / 4KB = 1M pages). This means the allocation size can reach 1M * 8 = 8MB for the pages array.

Quoted 7 lines

In mm/slub.c:5686, kmalloc_nolock() returns NULL for allocations larger than KMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE * 2) = 8KB on systems with 4KB pages.

kvcalloc() could handle larger allocations by falling back to vmalloc. With kmalloc_nolock(), arena allocations requiring more than 1024 pages will fail where they previously succeeded.

sched-ext as a main user of arena allocates 512 pages at a time, but AI has a good point. 1024 is too limiting.
Let's round down this kmalloc_nolock() to 1024 * 8 and add an explicit loop to reuse this array while allocating all of page_cnt pages. 1024 pages at a time.
Error path will be a bit more complex, since we would
need to call arena_free_pages(), since some pages might be already populated and active.

pw-bot: cr

#9 lore raw

PATCH 3/4 bpf: arena: make arena kfuncs any context safe

In reply to Puranjay Mohan

Make arena related kfuncs any context safe by the following changes:

bpf_arena_alloc_pages() and bpf_arena_reserve_pages():
Replace the usage of the mutex with a rqspinlock for range tree and use kmalloc_nolock() wherever needed. Use free_pages_nolock() to free pages from any context.
apply_range_set/clear_cb() with apply_to_page_range() has already made populating the vm_area in bpf_arena_alloc_pages() any context safe.

bpf_arena_free_pages(): defer the main logic to a workqueue if it is called from a non-sleepable context.

specialize_kfunc() is used to replace the sleepable arena_free_pages() with bpf_arena_free_pages_non_sleepable() when the verifier detects the call is from a non-sleepable context.

In the non-sleepable case, arena_free_pages() queues the address and the page count to be freed to a lock-less list of struct arena_free_spans and raises an irq_work. The irq_work handler calls schedules_work() as it is safe to be called from irq context. arena_free_worker() (the work queue handler) iterates these spans and clears ptes, flushes tlb, zaps pages, and calls __free_page().

---
 include/linux/bpf.h   |   2 +
 kernel/bpf/arena.c    | 227 +++++++++++++++++++++++++++++++++++-------
 kernel/bpf/verifier.c |   5 +
 3 files changed, 199 insertions(+), 35 deletions(-)
include/linux/bpf.h +2 -0
index 09d5dc541d1c..5279212694b4 100644
@@ -673,6 +673,8 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
 int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
 				   struct bpf_dynptr *ptr__uninit);
 
+void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
+
 extern const struct bpf_map_ops bpf_map_offload_ops;
 
 /* bpf_type_flag contains a set of flags that are applicable to the values of
kernel/bpf/arena.c +192 -35

413 lines not on this page · raw message

kernel/bpf/verifier.c +5 -0
index 1268fa075d4c..407f75daa1cb 100644
@@ -12319,6 +12319,7 @@ enum special_kfunc_type {
 	KF___bpf_trap,
 	KF_bpf_task_work_schedule_signal,
 	KF_bpf_task_work_schedule_resume,
+	KF_bpf_arena_free_pages,
 };
 
 BTF_ID_LIST(special_kfunc_list)
@@ -12393,6 +12394,7 @@ BTF_ID(func, bpf_dynptr_file_discard)
 BTF_ID(func, __bpf_trap)
 BTF_ID(func, bpf_task_work_schedule_signal)
 BTF_ID(func, bpf_task_work_schedule_resume)
+BTF_ID(func, bpf_arena_free_pages)
 
 static bool is_task_work_add_kfunc(u32 func_id)
 {
@@ -22350,6 +22352,9 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc
 	} else if (func_id == special_kfunc_list[KF_bpf_dynptr_from_file]) {
 		if (!env->insn_aux_data[insn_idx].non_sleepable)
 			addr = (unsigned long)bpf_dynptr_from_file_sleepable;
+	} else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) {
+		if (env->insn_aux_data[insn_idx].non_sleepable)
+			addr = (unsigned long)bpf_arena_free_pages_non_sleepable;
 	}
 
 set_imm:
Signature · 1 line
2.47.3
#10 lore raw

Re: [PATCH bpf-next 3/4] bpf: arena: make arena kfuncs any context safe

Can the page reference race with arena_free_pages()? The sequence:
Quoted 4 lines
include/linux/bpf.h +0 -0
index 09d5dc541..527921269 100644

[ ... ]

Quoted 4 lines
kernel/bpf/arena.c +0 -0
index 9d8a8eb44..f330b51de 100644

[ ... ]

Quoted 44 lines
kernel/bpf/arena.c +12 -5
@@ -327,12 +351,19 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
 	struct page *page;
 	long kbase, kaddr;
+	unsigned long flags;
 	int ret;
 
 	kbase = bpf_arena_get_kern_vm_start(arena);
 	kaddr = kbase + (u32)(vmf->address);
 
-	guard(mutex)(&arena->lock);
+	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
+		/*
+		 * This is an impossible case and would only trigger if res_spin_lock is buggy or
+		 * due to another kernel bug.
+		 */
+		return VM_FAULT_RETRY;
+
 	page = vmalloc_to_page((void *)kaddr);
 	if (page)
 		/* already have a page vmap-ed */
@@ -344,26 +375,30 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 
 	ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
 	if (ret)
-		return VM_FAULT_SIGSEGV;
+		goto out_unlock_sigsegv;
 
 	struct apply_range_data data = { .pages = &page, .i = 0 };
 	/* Account into memcg of the process that created bpf_arena */
 	ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
 	if (ret) {
 		range_tree_set(&arena->rt, vmf->pgoff, 1);
-		return VM_FAULT_SIGSEGV;
+		goto out_unlock_sigsegv;
 	}
 
 	ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
 	if (ret) {
 		range_tree_set(&arena->rt, vmf->pgoff, 1);
-		__free_page(page);
-		return VM_FAULT_SIGSEGV;
+		free_pages_nolock(page, 0);
+		goto out_unlock_sigsegv;

kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 402

Quoted 10 lines
kernel/bpf/arena.c +4 -0
 	}
 out:
+	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
 	page_ref_add(page, 1);
 	vmf->page = page;
 	return 0;
+out_unlock_sigsegv:
+	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+	return VM_FAULT_SIGSEGV;
 }

Can the page reference race with arena_free_pages()? The sequence:

Thread A (arena_vm_fault):

    bpf_map_alloc_pages() -> page refcount=1
    apply_to_page_range() -> maps PTE
    raw_res_spin_unlock_irqrestore()
    page_ref_add(page, 1) -> tries to increment to 2

Thread B (arena_free_pages):

    raw_res_spin_lock_irqsave()
    apply_to_existing_page_range()->apply_range_clear_cb() -> clears PTE
    raw_res_spin_unlock_irqrestore()
    __free_page(page) -> refcount 1->0, page freed

Between Thread A's unlock and page_ref_add(), Thread B can acquire the lock, clear the PTE, and free the page. Then Thread A's page_ref_add() operates on freed memory.

Should page_ref_add() happen before releasing the spinlock, or should the initial allocation start with refcount=2?

[ ... ]

kernel/bpf/verifier.c

Quoted 1 line
kernel/bpf/verifier.c +0 -0

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19272481461

#11 lore raw

Re: [PATCH bpf-next 3/4] bpf: arena: make arena kfuncs any context safe

Quoted 84 lines · On Tue, Nov 11, 2025 at 9:01 AM <[email protected]> wrote:
Quoted 5 lines
include/linux/bpf.h +0 -0
index 09d5dc541..527921269 100644

[ ... ]

Quoted 4 lines
kernel/bpf/arena.c +0 -0
index 9d8a8eb44..f330b51de 100644

[ ... ]

Quoted 44 lines
kernel/bpf/arena.c +12 -5
@@ -327,12 +351,19 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
      struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
      struct page *page;
      long kbase, kaddr;
+     unsigned long flags;
      int ret;
 
      kbase = bpf_arena_get_kern_vm_start(arena);
      kaddr = kbase + (u32)(vmf->address);
 
-     guard(mutex)(&arena->lock);
+     if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
+             /*
+              * This is an impossible case and would only trigger if res_spin_lock is buggy or
+              * due to another kernel bug.
+              */
+             return VM_FAULT_RETRY;
+
      page = vmalloc_to_page((void *)kaddr);
      if (page)
              /* already have a page vmap-ed */
@@ -344,26 +375,30 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
 
      ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
      if (ret)
-             return VM_FAULT_SIGSEGV;
+             goto out_unlock_sigsegv;
 
      struct apply_range_data data = { .pages = &page, .i = 0 };
      /* Account into memcg of the process that created bpf_arena */
      ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
      if (ret) {
              range_tree_set(&arena->rt, vmf->pgoff, 1);
-             return VM_FAULT_SIGSEGV;
+             goto out_unlock_sigsegv;
      }
 
      ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
      if (ret) {
              range_tree_set(&arena->rt, vmf->pgoff, 1);
-             __free_page(page);
-             return VM_FAULT_SIGSEGV;
+             free_pages_nolock(page, 0);
+             goto out_unlock_sigsegv;

kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 402

Quoted 10 lines
kernel/bpf/arena.c +4 -0
      }
 out:
+     raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
      page_ref_add(page, 1);
      vmf->page = page;
      return 0;
+out_unlock_sigsegv:
+     raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+     return VM_FAULT_SIGSEGV;
 }

Can the page reference race with arena_free_pages()? The sequence:

Thread A (arena_vm_fault):

    bpf_map_alloc_pages() -> page refcount=1
    apply_to_page_range() -> maps PTE
    raw_res_spin_unlock_irqrestore()
    page_ref_add(page, 1) -> tries to increment to 2

Thread B (arena_free_pages):

    raw_res_spin_lock_irqsave()
    apply_to_existing_page_range()->apply_range_clear_cb() -> clears PTE
    raw_res_spin_unlock_irqrestore()
    __free_page(page) -> refcount 1->0, page freed
Quoted 5 lines

Between Thread A's unlock and page_ref_add(), Thread B can acquire the lock, clear the PTE, and free the page. Then Thread A's page_ref_add() operates on freed memory.

Should page_ref_add() happen before releasing the spinlock,

AI has a point. page_ref should be under lock.

#12 lore raw

PATCH 4/4 selftests: bpf: test non-sleepable arena allocations

In reply to Puranjay Mohan

As arena kfuncs can now be called from non-sleepable contexts, test this by adding non-sleepable copies of tests in verifier_arena, this is done by using a socket program instead of syscall.

Augment the arena_list selftest to also run in non-sleepable context by taking rcu_read_lock.

---
 .../selftests/bpf/prog_tests/arena_list.c     |  20 +-
 .../testing/selftests/bpf/progs/arena_list.c  |  11 ++
 .../selftests/bpf/progs/verifier_arena.c      | 185 ++++++++++++++++++
 3 files changed, 211 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/prog_tests/arena_list.c +15 -5
index d15867cddde0..4f2866a615ce 100644
@@ -27,17 +27,23 @@ static int list_sum(struct arena_list_head *head)
 	return sum;
 }
 
-static void test_arena_list_add_del(int cnt)
+static void test_arena_list_add_del(int cnt, bool nonsleepable)
 {
 	LIBBPF_OPTS(bpf_test_run_opts, opts);
 	struct arena_list *skel;
 	int expected_sum = (u64)cnt * (cnt - 1) / 2;
 	int ret, sum;
 
-	skel = arena_list__open_and_load();
-	if (!ASSERT_OK_PTR(skel, "arena_list__open_and_load"))
+	skel = arena_list__open();
+	if (!ASSERT_OK_PTR(skel, "arena_list__open"))
 		return;
 
+	skel->rodata->nonsleepable = nonsleepable;
+
+	ret = arena_list__load(skel);
+	if (!ASSERT_OK(ret, "arena_list__load"))
+		goto out;
+
 	skel->bss->cnt = cnt;
 	ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts);
 	ASSERT_OK(ret, "ret_add");
@@ -65,7 +71,11 @@ static void test_arena_list_add_del(int cnt)
 void test_arena_list(void)
 {
 	if (test__start_subtest("arena_list_1"))
-		test_arena_list_add_del(1);
+		test_arena_list_add_del(1, false);
 	if (test__start_subtest("arena_list_1000"))
-		test_arena_list_add_del(1000);
+		test_arena_list_add_del(1000, false);
+	if (test__start_subtest("arena_list_1_nonsleepable"))
+		test_arena_list_add_del(1, true);
+	if (test__start_subtest("arena_list_1000_nonsleepable"))
+		test_arena_list_add_del(1000, true);
 }
tools/testing/selftests/bpf/progs/arena_list.c +11 -0
index 3a2ddcacbea6..235d8cc95bdd 100644
@@ -30,6 +30,7 @@ struct arena_list_head __arena *list_head;
 int list_sum;
 int cnt;
 bool skip = false;
+const volatile bool nonsleepable = false;
 
 #ifdef __BPF_FEATURE_ADDR_SPACE_CAST
 long __arena arena_sum;
@@ -42,6 +43,9 @@ int test_val SEC(".addr_space.1");
 
 int zero;
 
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
 SEC("syscall")
 int arena_list_add(void *ctx)
 {
@@ -71,6 +75,10 @@ int arena_list_del(void *ctx)
 	struct elem __arena *n;
 	int sum = 0;
 
+	/* Take rcu_read_lock to test non-sleepable context */
+	if (nonsleepable)
+		bpf_rcu_read_lock();
+
 	arena_sum = 0;
 	list_for_each_entry(n, list_head, node) {
 		sum += n->value;
@@ -79,6 +87,9 @@ int arena_list_del(void *ctx)
 		bpf_free(n);
 	}
 	list_sum = sum;
+
+	if (nonsleepable)
+		bpf_rcu_read_unlock();
 #else
 	skip = true;
 #endif
tools/testing/selftests/bpf/progs/verifier_arena.c +185 -0

227 lines not on this page · raw message

Signature · 1 line
2.47.3