bpf: Make sleepable arena paths use sleepable alloc_pages
Subsystems (4)
Emil Tsalapatis
reviewercover bpf: Make sleepable arena paths use sleepable alloc_pages
The arena_alloc_pages() call takes a sleepable argument based on whether its caller is a sleepable BPF function. This flag, along with the context the kfunc is called in, decides whether the call will try to fulfill the allocation using the regular or the _nolock variant of the alloc_pages API, by means of bpf_map_alloc_pages().
However, the arena_alloc_pages() call currently only makes allocations inside an IRQ-disabled critical section. This forces all allocations to use the _nolock() API, which may eagerly fail where the regular variant would eventually succeed. There have been reports of this happening for sched-ext schedulers.
Restructure arena_alloc_pages() to use the _nolock() page allocation API only when necessary. This requires moving allocations outside of the spinlock critical section for sleepable calls, which in turn requires slightly different logic in the allocation path. Make a separate sleepable code path that implements this logic.
Emil Tsalapatis (3):
bpf: Factor out nonsleepable arena allocation logic
bpf: Add sleepable arena page allocation path selftests/bpf: Test large allocations for both sleepable/nonsleepable
arena users
include/linux/bpf.h | 6 + kernel/bpf/arena.c | 188 +++++++++++++----- kernel/bpf/syscall.c | 8 +- .../bpf/progs/verifier_arena_large.c | 27 ++- 4 files changed, 175 insertions(+), 54 deletions(-)
Signature · 1 line
2.54.0
Emil Tsalapatis
reviewerPATCH 1/3 bpf: Factor out nonsleepable arena allocation logic
The bpf_arena_alloc_pages() is supposed to transparently choose whether to use _nolock() allocation variants or not depending on the caller's context. This is implemented internally by testing the context a __bpf_alloc_page call is made in. However, the implementation of bpf_arena_alloc_pages() forces the use of the _nolock() variants by calling __bpf_alloc_page after disabling IRQs by taking a spinlock. This in turn causes spurious allocation failures for some workloads like scx schedulers due to ZONE_NORMAL falling below the minimum watermark, even though the caller is sleepable could wait until there is memory available.
To fix this we need to call __bpf_alloc_page outside of the spinlock critical section when the allocation is called from a sleepable BPF function. As a first step, refactor the existing logic to simplify adding the path in the next commit.
No functional changes in this patch. The followup will add a proper sleepable path.
--- kernel/bpf/arena.c | 126 +++++++++++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 49 deletions(-)
kernel/bpf/arena.c +77 -49
@@ -661,68 +661,57 @@ static u64 clear_lo32(u64 val)return val & ~(u64 )~0U ; } -/* -* Allocate pages and vmap them into kernel vmalloc area. -* Later the pages will be mmaped into user space vma. -*/ -static long arena_alloc_pages (struct bpf_arena *arena,long uaddr,long page_cnt,int node_id, -bool sleepable) + +static int arena_adjust_tree (struct bpf_arena *arena,long uaddr,long page_cnt,long *pgoff) +{ +int ret; + +/* Special case where user is requesting specific range. */ +if (uaddr) { + ret =is_range_tree_set (&arena->rt, *pgoff, page_cnt); +if (ret) +return ret; +return range_tree_clear (&arena->rt, *pgoff, page_cnt); + } + + ret =range_tree_find (&arena->rt, page_cnt); +if (ret <0 ) +return ret; + + *pgoff = ret; + +return range_tree_clear (&arena->rt, *pgoff, page_cnt); +} + +static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id,bool sleepable) { -/* user_vm_end/start are fixed before bpf prog runs */ -long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); -struct mem_cgroup *new_memcg, *old_memcg;struct apply_range_data data;struct page **pages =NULL ;long remaining, mapped =0 ;long alloc_pages;unsigned long flags; -long pgoff =0 ;u32 uaddr32;int ret, i; -if (node_id !=NUMA_NO_NODE && - ((unsigned int )node_id >= nr_node_ids || !node_online (node_id))) -return 0 ; - -if (page_cnt > page_cnt_max) -return 0 ; - -if (uaddr) { -if (uaddr & ~PAGE_MASK ) -return 0 ; - pgoff =compute_pgoff (arena, uaddr); -if (pgoff > page_cnt_max - page_cnt) -/* requested address will be outside of user VMA */ -return 0 ; - } - -bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg);/* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. */ alloc_pages =min (page_cnt,KMALLOC_MAX_CACHE_SIZE /sizeof (struct page *)); pages =kmalloc_nolock (alloc_pages *sizeof (struct page *), __GFP_ACCOUNT,NUMA_NO_NODE ); -if (!pages) { -bpf_map_memcg_exit (old_memcg, new_memcg); +if (!pages)return 0 ; - } + data.arena = arena; data.pages = pages;if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) goto out_free_pages; -if (uaddr) { - ret =is_range_tree_set (&arena->rt, pgoff, page_cnt); -if (ret) - goto out_unlock_free_pages; - ret =range_tree_clear (&arena->rt, pgoff, page_cnt); - }else { - ret = pgoff =range_tree_find (&arena->rt, page_cnt); -if (pgoff >=0 ) - ret =range_tree_clear (&arena->rt, pgoff, page_cnt); + ret =arena_adjust_tree (arena, uaddr, page_cnt, &pgoff); +if (ret) { +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + goto out_free_pages; } -if (ret) - goto out_unlock_free_pages; remaining = page_cnt; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE );
@@ -735,7 +724,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt ret =bpf_map_alloc_pages (&arena->map, node_id, this_batch, pages);if (ret) - goto out; + goto out_unmap;/* * Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
@@ -754,31 +743,70 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt mapped += data.i;for (i = data.i; i < this_batch; i++)free_pages_nolock (pages[i],0 ); - goto out; + goto out_unmap; } mapped += this_batch; remaining -= this_batch; } +flush_vmap_cache (kern_vm_start + uaddr32, mapped <<PAGE_SHIFT );raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +kfree_nolock (pages); -bpf_map_memcg_exit (old_memcg, new_memcg); +return clear_lo32 (arena->user_vm_start) + uaddr32; -out: + +out_unmap:range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped);raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);if (mapped) {flush_vmap_cache (kern_vm_start + uaddr32, mapped <<PAGE_SHIFT );arena_free_pages (arena, uaddr32, mapped, sleepable); } - goto out_free_pages; -out_unlock_free_pages: -raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + out_free_pages:kfree_nolock (pages); -bpf_map_memcg_exit (old_memcg, new_memcg);return 0 ; + +} + +/* +* Allocate pages and vmap them into kernel vmalloc area. +* Later the pages will be mmaped into user space vma. +*/ +static long arena_alloc_pages (struct bpf_arena *arena,long uaddr,long page_cnt,int node_id, +bool sleepable) +{ +/* user_vm_end/start are fixed before bpf prog runs */ +long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ; +struct mem_cgroup *new_memcg, *old_memcg; +long addr; +long pgoff =0 ; + +if (node_id !=NUMA_NO_NODE && + ((unsigned int )node_id >= nr_node_ids || !node_online (node_id))) +return 0 ; + +if (page_cnt > page_cnt_max) +return 0 ; + +if (uaddr) { +if (uaddr & ~PAGE_MASK ) +return 0 ; + pgoff =compute_pgoff (arena, uaddr); +if (pgoff > page_cnt_max - page_cnt) +/* requested address will be outside of user VMA */ +return 0 ; + } + +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + + addr =arena_alloc_pages_internal (arena, page_cnt, uaddr, pgoff, node_id, sleepable); + +bpf_map_memcg_exit (old_memcg, new_memcg); + +return addr; }/*
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 1/3] bpf: Factor out nonsleepable arena allocation logic
This isn't a bug, but could this sentence be reworded (e.g. "even though a sleepable caller could wait until memory becomes available") so the motivation reads cleanly?
Quoted 6 lines
commit
434fc482ce1027a7af536690a218094b882f2ad2Author: Emil Tsalapatis <[email protected]> Date: Mon Aug 24 04:25:28 2026 -0400bpf: Factor out nonsleepable arena allocation logic
Quoted 10 lines
The
bpf_arena_alloc_pages()is supposed to transparently choose whether to use_nolock()allocation variants or not depending on the caller's context. This is implemented internally by testing the context a__bpf_alloc_pagecall is made in. However, the implementation ofbpf_arena_alloc_pages()forces the use of the_nolock()variants by calling__bpf_alloc_pageafter disabling IRQs by taking a spinlock. This in turn causes spurious allocation failures for some workloads like scx schedulers due toZONE_NORMALfalling below the minimum watermark, even though the caller is sleepable could wait until there is memory available.
This isn't a bug, but could this sentence be reworded (e.g. "even though a sleepable caller could wait until memory becomes available") so the motivation reads cleanly?
Quoted 28 lines
kernel/bpf/arena.c +14
-6@@ -661,68 +661,57 @@ static u64 clear_lo32(u64 val)return val & ~(u64 )~0U ; } -/* -* Allocate pages and vmap them into kernel vmalloc area. -* Later the pages will be mmaped into user space vma. -*/ -static long arena_alloc_pages (struct bpf_arena *arena,long uaddr,long page_cnt,int node_id, -bool sleepable) + +static int arena_adjust_tree (struct bpf_arena *arena,long uaddr,long page_cnt,long *pgoff) +{ +int ret; + +/* Special case where user is requesting specific range. */ +if (uaddr) { + ret =is_range_tree_set (&arena->rt, *pgoff, page_cnt); +if (ret) +return ret; +return range_tree_clear (&arena->rt, *pgoff, page_cnt); + } + + ret =range_tree_find (&arena->rt, page_cnt);
kernel/bpf/arena.c static u64 clear_lo32(u64 val) · line 687
Quoted 10 lines
kernel/bpf/arena.c +10
-0+if (ret <0 ) +return ret; + + *pgoff = ret; + +return range_tree_clear (&arena->rt, *pgoff, page_cnt); +} + +static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id,bool sleepable)
This isn't a bug, but would it read better to keep the existing (uaddr, page_cnt) argument order in arena_alloc_pages_internal() so all three functions agree, given both parameters are plain long?
arena_adjust_tree() uses (uaddr, page_cnt) and arena_alloc_pages() also uses (uaddr, page_cnt), but arena_alloc_pages_internal() uses (page_cnt, uaddr).
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315
Emil Tsalapatis
reviewerPATCH 2/3 bpf: Add sleepable arena page allocation path
In reply to Emil TsalapatisThe bpf_arena_alloc_pages() function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use of alloc_pages_nolock() in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.
Add a separate arena page allocation path just for sleepable callers. The path preallocates the arena memory to be added to the tree before taking the critical section.
--- include/linux/bpf.h | 6 ++++ kernel/bpf/arena.c | 76 +++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 8 +---- 3 files changed, 79 insertions(+), 11 deletions(-)
include/linux/bpf.h +6 -0
@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +static inline bool is_bpf_alloc_nonsleepable (void ) +{ +return preempt_count () >0 ||irqs_disabled () || +IS_ENABLED (CONFIG_PREEMPT_RT ); +} +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT )void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);
kernel/bpf/arena.c +72 -4
94 lines not on this page · raw message
kernel/bpf/syscall.c +1 -7
@@ -596,15 +596,9 @@ static void bpf_map_release_memcg(struct bpf_map *map) }#endif -static bool can_alloc_pages (void ) -{ -return preempt_count () ==0 && !irqs_disabled () && - !IS_ENABLED (CONFIG_PREEMPT_RT ); -} -static struct page *__bpf_alloc_page (int nid) { -if (!can_alloc_pages ()) +if (is_bpf_alloc_nonsleepable ())return alloc_pages_nolock (__GFP_ACCOUNT, nid,0 ); return alloc_pages_node(nid,
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 2/3] bpf: Add sleepable arena page allocation path
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Hard lockup via integer overflow: `data.i << PAGE_SHIFT` passes an astronomically large negative…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Hard lockup via integer overflow: data.i << PAGE_SHIFT passes an astronomically large negative size to flush_vmap_cache on mapping failure. --
--- Patch [2]: [PATCH 2/3] bpf: Add sleepable arena page allocation path --- commit a93c0f542ad1ecf8f00cde8cf86699ce952035a7
Author: Emil Tsalapatis <[email protected]>
bpf: Add sleepable arena page allocation path
This commit adds a separate sleepable allocation path for BPF arena memory. It preallocates memory before taking the critical section to prevent frequent allocation failures from sleepable BPF contexts.
Quoted 4 lines
kernel/bpf/arena.c +0
-0
[ ... ]
Quoted 3 lines
kernel/bpf/arena.c +3
-0+static long arena_alloc_pages_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) +{
[ ... ]
kernel/bpf/arena.c static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, · line 736
Quoted 3 lines
kernel/bpf/arena.c +3
-0+out_unmap: +/* data.i pages were mapped, undo only those who failed to map */ +flush_vmap_cache (kern_vm_start + uaddr32, data.i <<PAGE_SHIFT );
[Severity: High]
Can this shift operation overflow?
If apply_to_page_range() maps a large number of pages (e.g. over 524,287) before failing, data.i << PAGE_SHIFT will overflow since data.i is a signed 32-bit int.
Because flush_vmap_cache() expects an unsigned long size, a negative value here would sign-extend to a massive 64-bit size, potentially causing a CPU stall or lockup in the flush routine.
Would casting data.i to unsigned long before the shift prevent this?
Quoted 3 lines
kernel/bpf/arena.c +3
-0+range_tree_set (&arena->rt, pgoff + data.i, page_cnt - data.i); + +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH 2/3] bpf: Add sleepable arena page allocation path
In reply to Emil TsalapatisThis isn't a bug, but does the comment describe the right thing?
Quoted 73 lines
include/linux/bpf.h +6
-0@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +static inline bool is_bpf_alloc_nonsleepable (void ) +{ +return preempt_count () >0 ||irqs_disabled () || +IS_ENABLED (CONFIG_PREEMPT_RT ); +} +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT )void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);kernel/bpf/arena.c +46
-2@@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,return range_tree_clear (&arena->rt, *pgoff, page_cnt); } -static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, -long uaddr,long pgoff,int node_id,bool sleepable) +static long arena_alloc_pages_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) +{ +u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data; +struct page **pages =NULL ; +unsigned long flags; +u32 uaddr32; +int ret, i; + + pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL_ACCOUNT ); +if (!pages) +return 0 ; + + ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); +if (ret) { +kvfree (pages); +return 0 ; + } + + data.i =0 ; + data.pages = pages; + data.arena = arena; + +if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) + goto out_free_pages; + + ret =arena_adjust_tree (arena, uaddr, page_cnt, &pgoff); +if (ret) { +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + goto out_free_pages; + } + + uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); + + ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, + page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data); + +if (ret) + goto out_unmap; + +flush_vmap_cache (kern_vm_start + uaddr32, page_cnt <<PAGE_SHIFT ); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + +kvfree (pages); +
kernel/bpf/arena.c static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, · line 741
Quoted 10 lines
kernel/bpf/arena.c +10
-0+return clear_lo32 (arena->user_vm_start) + uaddr32; + +out_unmap: +/* data.i pages were mapped, undo only those who failed to map */ +flush_vmap_cache (kern_vm_start + uaddr32, data.i <<PAGE_SHIFT ); +range_tree_set (&arena->rt, pgoff + data.i, page_cnt - data.i); + +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); + +arena_free_pages (arena, uaddr32, data.i, true);
This isn't a bug, but does the comment describe the right thing?
The comment says "undo only those who failed to map", but the code below unmaps the pages that succeeded:
arena_free_pages(arena, uaddr32, data.i, true);
This unmaps exactly data.i pages starting at uaddr32, which are the ones that apply_range_set_cb() installed (data.i is incremented once per successful PTE). The pages that failed to map get freed in the out_free_pages loop below:
for (i = data.i; i < page_cnt; i++)
__free_page(pages[i]);Should the comment say something like "data.i pages were mapped, undo those" instead?
Quoted 14 lines
kernel/bpf/arena.c +11
-0+ +out_free_pages: +for (i = data.i; i < page_cnt; i++) +__free_page (pages[i]); + +kvfree (pages); +return 0 ; +} + +static long arena_alloc_pages_non_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) {u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena);struct apply_range_data data;
[ ... ]
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 873
Quoted 9 lines
kernel/bpf/arena.c +4
-1@@ -802,7 +867,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntbpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); - addr =arena_alloc_pages_internal (arena, page_cnt, uaddr, pgoff, node_id, sleepable); +if (sleepable && !is_bpf_alloc_nonsleepable ()) + addr =arena_alloc_pages_sleepable (arena, page_cnt, uaddr, pgoff, node_id); +else + addr =arena_alloc_pages_non_sleepable (arena, page_cnt, uaddr, pgoff, node_id);
Can this dispatch block while in an RCU read-side critical section?
The sleepable parameter is hardcoded to true by the kfunc entry point in kernel/bpf/arena.c:
void *bpf_arena_alloc_pages(struct bpf_map *map, void *addr__ign,
u32 page_cnt, int node_id) {
return (void *)arena_alloc_pages(arena, (long)addr__ign,
page_cnt, node_id, true);}
and that kfunc is registered without KF_SLEEPABLE:
BTF_ID_FLAGS(func, bpf_arena_alloc_pages,
KF_ARENA_RET | KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
The verifier checks in kernel/bpf/verifier.c specifically prevent non-sleepable programs from calling sleepable kfuncs:
sleepable = bpf_is_kfunc_sleepable(&meta);
if (sleepable && !in_sleepable(env)) {
verbose(env, "program must be sleepable to call sleepable "
"kfunc %s\n", func_name);
return -EACCES;
}but that check never triggers because bpf_arena_alloc_pages() is not tagged KF_SLEEPABLE. So non-sleepable programs can call it.
is_bpf_alloc_nonsleepable() checks for:
preempt_count() > 0 || irqs_disabled() || IS_ENABLED(CONFIG_PREEMPT_RT)
That detects hardirq/softirq/NMI/preempt-disabled/IRQ-disabled/RT contexts, but it does not detect the RCU read-side critical section that every non-sleepable BPF program runs inside.
Non-sleepable trampoline programs (fentry/fexit/fmod_ret/raw_tp/ struct_ops) enter via __bpf_prog_enter() in kernel/bpf/trampoline.c, which calls rcu_read_lock_dont_migrate():
rcu_read_lock_dont_migrate() {
if (IS_ENABLED(CONFIG_PREEMPT_RCU))
migrate_disable();
rcu_read_lock();
}On CONFIG_PREEMPT_RCU=y kernels (the default for CONFIG_PREEMPT_DYNAMIC distro kernels), rcu_read_lock() is __rcu_read_lock() which only increments current->rcu_read_lock_nesting, and migrate_disable() uses a scoped guard(preempt) so it does not leave preempt_count() elevated.
So with CONFIG_PREEMPT_RT=n, is_bpf_alloc_nonsleepable() returns false and arena_alloc_pages_sleepable() is selected.
arena_alloc_pages_sleepable() then calls:
kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL_ACCOUNT) bpf_map_alloc_pages() -> alloc_pages_node(..., GFP_KERNEL, ...) arena_free_pages() -> mutex_lock() and flush_tlb_kernel_range()
All of these can sleep. Documentation/RCU/whatisRCU.rst states "It is illegal to block while in an RCU read-side critical section" even under CONFIG_PREEMPT_RCU. Being preempted is permitted, but voluntarily blocking is not.
The blocked reader is queued on rnp->blkd_tasks, so every concurrent synchronize_rcu() on the system waits for the duration of the direct reclaim. Under memory pressure (which the commit message says triggers this path for "heavy BPF arena users like scx"), this yields unbounded RCU grace-period extension, RCU CPU stall warnings, and delayed freeing kernel-wide. The task also blocks with current->migration_disabled set and rq->nr_pinned elevated, which stalls CPU hotplug for the same duration.
The in-tree selftests confirm that non-sleepable programs are accepted:tools/testing/selftests/bpf/progs/verifier_arena.c has several SEC("socket") __success programs calling bpf_arena_alloc_pages(), and tools/testing/selftests/bpf/progs/struct_ops_arena.c shows non-sleepable SEC("struct_ops/...") callbacks bound to an arena map.
Should the sleepability decision be based on the verifier-known program context rather than this runtime heuristic? For example, tagging bpf_arena_alloc_pages() with KF_SLEEPABLE and adding a separate non-sleepable kfunc, or plumbing the prog's sleepable flag down to arena_alloc_pages()?
Quoted 11 lines
kernel/bpf/syscall.c +0
-3@@ -596,15 +596,9 @@ static void bpf_map_release_memcg(struct bpf_map *map) }#endif -static bool can_alloc_pages (void ) -{ - return preempt_count() == 0 && !irqs_disabled() &&
kernel/bpf/syscall.c static void bpf_map_release_memcg(struct bpf_map *map) · line 604
Quoted 10 lines
kernel/bpf/syscall.c +1
-4- !IS_ENABLED (CONFIG_PREEMPT_RT ); -} -static struct page *__bpf_alloc_page (int nid) { -if (!can_alloc_pages ()) +if (is_bpf_alloc_nonsleepable ())return alloc_pages_nolock (__GFP_ACCOUNT, nid,0 ); return alloc_pages_node(nid,
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315
Re: [PATCH 2/3] bpf: Add sleepable arena page allocation path
In reply to Emil TsalapatisQuoted 60 lines · Emil Tsalapatis
The
bpf_arena_alloc_pages()function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use ofalloc_pages_nolock()in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.Add a separate arena page allocation path just for sleepable callers. The path preallocates the arena memory to be added to the tree before taking the critical section.
Signed-off-by Emil Tsalapatis <[email protected]>--- include/linux/bpf.h | 6 ++++ kernel/bpf/arena.c | 76 +++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 8 +---- 3 files changed, 79 insertions(+), 11 deletions(-)include/linux/bpf.h +6
-0@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +static inline bool is_bpf_alloc_nonsleepable (void ) +{ +return preempt_count () >0 ||irqs_disabled () || +IS_ENABLED (CONFIG_PREEMPT_RT ); +} +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT )void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);kernel/bpf/arena.c +14
-2@@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,return range_tree_clear (&arena->rt, *pgoff, page_cnt); } -static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, -long uaddr,long pgoff,int node_id,bool sleepable) +static long arena_alloc_pages_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) +{ +u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data; +struct page **pages =NULL ; +unsigned long flags; +u32 uaddr32; +int ret, i; + + pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL_ACCOUNT ); +if (!pages) +return 0 ; +
kernel/bpf/arena.c static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, · line 709
Quoted 10 lines
kernel/bpf/arena.c +10
-0+ ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); +if (ret) { +kvfree (pages); +return 0 ; + } + + data.i =0 ; + data.pages = pages; + data.arena = arena; +
Why have this split?
Always use an approach of allocating pages outside of the lock?
Considering severity of the issue bpf tree is probably appropriate.
pw-bot: cr
Emil Tsalapatis
reviewerRe: [PATCH 2/3] bpf: Add sleepable arena page allocation path
Quoted 67 lines · Alexei Starovoitov
Quoted 65 lines · Emil Tsalapatis
The
bpf_arena_alloc_pages()function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use ofalloc_pages_nolock()in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.Add a separate arena page allocation path just for sleepable callers. The path preallocates the arena memory to be added to the tree before taking the critical section.
Signed-off-by Emil Tsalapatis <[email protected]>--- include/linux/bpf.h | 6 ++++ kernel/bpf/arena.c | 76 +++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 8 +---- 3 files changed, 79 insertions(+), 11 deletions(-)include/linux/bpf.h +6
-0@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +static inline bool is_bpf_alloc_nonsleepable (void ) +{ +return preempt_count () >0 ||irqs_disabled () || +IS_ENABLED (CONFIG_PREEMPT_RT ); +} +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT )void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);kernel/bpf/arena.c +19
-2@@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,return range_tree_clear (&arena->rt, *pgoff, page_cnt); } -static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, -long uaddr,long pgoff,int node_id,bool sleepable) +static long arena_alloc_pages_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) +{ +u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data; +struct page **pages =NULL ; +unsigned long flags; +u32 uaddr32; +int ret, i; + + pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL_ACCOUNT ); +if (!pages) +return 0 ; + + ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); +if (ret) { +kvfree (pages); +return 0 ; + }
Quoted 9 lines
kernel/bpf/arena.c static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, · line 709
Quoted 5 lines
kernel/bpf/arena.c +5
-0+ + data.i =0 ; + data.pages = pages; + data.arena = arena; +Why have this split?
Always use an approach of allocating pages outside of the lock?
Mainly to avoid multiple kmalloc_nolock() allocations outside the lock for the sleepable
path. That should matter for larger allocations. The two separate paths also stay simpler
than if we had a single path (at the cost of a bit of extra duplication). I can change to a single
code path during respin if you feel these are not worth having two internal calls.
Quoted 1 line
Considering severity of the issue bpf tree is probably appropriate.
Will do.
Quoted 2 lines
pw-bot: cr
Re: [PATCH 2/3] bpf: Add sleepable arena page allocation path
Quoted 79 lines · Emil Tsalapatis
Quoted 67 lines · Alexei Starovoitov
Quoted 65 lines · Emil Tsalapatis
The
bpf_arena_alloc_pages()function currently only allocates pages inside a spinlock critical section with IRQs off. This forces the use ofalloc_pages_nolock()in the BPF allocator, even when the caller is a sleepable BPF function. This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed. This can be triggered consistently by heavy BPF arena users like scx.Add a separate arena page allocation path just for sleepable callers. The path preallocates the arena memory to be added to the tree before taking the critical section.
Signed-off-by Emil Tsalapatis <[email protected]>--- include/linux/bpf.h | 6 ++++ kernel/bpf/arena.c | 76 +++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 8 +---- 3 files changed, 79 insertions(+), 11 deletions(-)include/linux/bpf.h +6
-0@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +static inline bool is_bpf_alloc_nonsleepable (void ) +{ +return preempt_count () >0 ||irqs_disabled () || +IS_ENABLED (CONFIG_PREEMPT_RT ); +} +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT )void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);kernel/bpf/arena.c +19
-2@@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,return range_tree_clear (&arena->rt, *pgoff, page_cnt); } -static long arena_alloc_pages_internal (struct bpf_arena *arena,long page_cnt, -long uaddr,long pgoff,int node_id,bool sleepable) +static long arena_alloc_pages_sleepable (struct bpf_arena *arena,long page_cnt, +long uaddr,long pgoff,int node_id) +{ +u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data; +struct page **pages =NULL ; +unsigned long flags; +u32 uaddr32; +int ret, i; + + pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL_ACCOUNT ); +if (!pages) +return 0 ; + + ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); +if (ret) { +kvfree (pages); +return 0 ; + }Quoted 9 lines
kernel/bpf/arena.c static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, · line 709
Quoted 5 lines
kernel/bpf/arena.c +5
-0+ + data.i =0 ; + data.pages = pages; + data.arena = arena; +Why have this split?
Always use an approach of allocating pages outside of the lock?
Quoted 8 lines
Mainly to avoid multiple
kmalloc_nolock()allocations outside the lock for the sleepable
path. That should matter for larger allocations. The two separate paths also stay simpler
than if we had a single path (at the cost of a bit of extra duplication). I can change to a single
code path during respin if you feel these are not worth having two internal calls.
Hmm. I would think one way to do things is easier. Less branches to worry about. Give it a shot? If it looks worse, then keep it as-is.
Emil Tsalapatis
reviewerPATCH 3/3 selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
In reply to Emil TsalapatisWe now use different code paths in the internal allocator when allocating arena memory, depending on whether the caller is sleepable or not. These paths mostly differ functionally for large allocations, so add extra testing for that case.
--- .../bpf/progs/verifier_arena_large.c | 27 ++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/verifier_arena_large.c +26 -1
@@ -284,6 +284,7 @@ int big_alloc2(void *ctx)return 0 ; } +/* Nonsleepable because it binds to a socket program. */ SEC("socket") __success __retval(0) int big_alloc3(void *ctx)
@@ -300,7 +301,31 @@ int big_alloc3(void *ctx)*/ pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 );if (!pages) -return 0 ; +return 1 ; + +bpf_for (i,0 ,2051 ) + pages[i * PAGE_SIZE] = 123; + bpf_for(i, 0, 2051) +if (pages[i *PAGE_SIZE ] !=123 ) +return i; + +bpf_arena_free_pages (&arena, pages,2051 ); +#endif +return 0 ; +} + +/* SYSCALL programs are always sleepable. */ +SEC("syscall") +__success __retval(0) +int big_alloc4(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + char __arena *pages; +u64 i; + + pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 ); +if (!pages) +return 1 ;bpf_for (i,0 ,2051 ) pages[i * PAGE_SIZE] = 123;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Medium] The validation loop's error path incorrectly returns the loop index i, meaning a memory validation failure on the first page (i == 0) will falsely report as a successful test.
Pre-existing issues:
- [High] Hard lockup (watchdog timeout) in arena_alloc_pages_non_sleepable due to a massive page allocation loop executing with local interrupts disabled. --
--- Patch [3]: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users --- commit 97b45503d1d3323d8eb083b751d71f1bec8430fe
Author: Emil Tsalapatis <[email protected]>
selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
We now use different code paths in the internal allocator when allocating arena memory, depending on whether the caller is sleepable or not. These paths mostly differ functionally for large allocations, so add extra testing for that case.
Quoted 3 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/verifier_arena_large.c int big_alloc3(void *ctx) · line 302
Quoted 3 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +0
-0@@ -300,7 +301,31 @@ int big_alloc3(void *ctx)*/ pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 );
[Severity: High]
This is a pre-existing issue, but does this allocation of 2051 pages trigger a hard lockup in the underlying allocator?
Looking at arena_alloc_pages_non_sleepable() in kernel/bpf/arena.c, the code disables interrupts and runs a batch allocation loop:
kernel/bpf/arena.c:arena_alloc_pages_non_sleepable() {
...
while (remaining) {
long this_batch = min(remaining, alloc_pages);
memset(pages, 0, this_batch * sizeof(struct page *));
ret = bpf_map_alloc_pages(&arena->map, node_id, this_batch, pages);
...
ret = apply_to_page_range(&init_mm, ...);
}
...
}Since this path is executed with a raw spinlock held and local interrupts disabled, could a large page_cnt like 2051 cause a watchdog timeout?
tools/testing/selftests/bpf/progs/verifier_arena_large.c int big_alloc3(void *ctx) · line 310
Quoted 9 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +7
-1if (!pages) -return 0 ; +return 1 ; + +bpf_for (i,0 ,2051 ) + pages[i * PAGE_SIZE] = 123; + bpf_for(i, 0, 2051) +if (pages[i *PAGE_SIZE ] !=123 ) +return i;
[Severity: Medium]
Does this error path mask validation failures on the first page?
Since the program expects 0 for success (__success __retval(0)), if the validation fails on the first page (i == 0), the loop returns 0. This perfectly aliases the expected success code and completely masks the failure.
Additionally, failing on the second page (i == 1) returns 1, which aliases the allocation failure code above.
Could this return a distinct error code instead of the loop index?
Quoted 14 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +14
-0+ +bpf_arena_free_pages (&arena, pages,2051 ); +#endif +return 0 ; +} + +/* SYSCALL programs are always sleepable. */ +SEC("syscall") +__success __retval(0) +int big_alloc4(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + char __arena *pages; +u64 i;
tools/testing/selftests/bpf/progs/verifier_arena_large.c int big_alloc3(void *ctx)
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +7
-0+ + pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 ); +if (!pages) +return 1 ;bpf_for (i,0 ,2051 ) pages[i * PAGE_SIZE] = 123; + bpf_for(i, 0, 2051) +if (pages[i *PAGE_SIZE ] !=123 ) +return i;
[Severity: Medium]
Similarly, does returning the loop index here falsely report success if the first page fails validation?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users
In reply to Emil TsalapatisIs it safe to make big_alloc3 fail the test when allocation fails?
Quoted 4 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +0
-0
[ ... ]
Quoted 9 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +1
-0@@ -284,6 +284,7 @@ int big_alloc2(void *ctx)return 0 ; } +/* Nonsleepable because it binds to a socket program. */ SEC("socket") __success __retval(0) int big_alloc3(void *ctx)
tools/testing/selftests/bpf/progs/verifier_arena_large.c int big_alloc3(void *ctx) · line 304
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_arena_large.c +1
-1@@ -300,7 +301,31 @@ int big_alloc3(void *ctx)*/ pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 );if (!pages) -return 0 ; +return 1 ;
Is it safe to make big_alloc3 fail the test when allocation fails?
The commit message says "add extra testing for that case," but the pre-existing big_alloc3 deliberately tolerated allocation failure with 'return 0'. This matters because big_alloc3 is SEC("socket"), and BPF_PROG_TEST_RUN executes socket programs with local_bh_disable():
net/bpf/test_run.c:bpf_prog_test_run_skb()
local_bh_disable();
...
ret = bpf_test_run(skb, repeat, &retval, &duration, false);
...
local_bh_enable();That makes preempt_count() > 0, so is_bpf_alloc_nonsleepable() returns true and bpf_arena_alloc_pages() selects arena_alloc_pages_non_sleepable() (as intended, matching the commit message "different code paths ... depending on whether the caller is sleepable or not").
The non-sleepable path can legitimately return NULL for reasons that aren't kernel bugs. From d1f68a079731 in this same series:
"This forces the use of alloc_pages_nolock() in the BPF allocator ...
This in turn causes allocation failures even in cases where falling into the allocator slow path and possibly sleeping would eventually succeed."
So a 2051-page nolock allocation can fail when ZONE_NORMAL is below the min watermark, or memory is fragmented, or the test runs on a busy CI VM. With 'return 1', big_alloc3 would report a false regression in those cases, even though the allocator is working as designed.
Would it make sense to keep the tolerant 'return 0' for big_alloc3 (the non-sleepable variant that uses alloc_pages_nolock), and reserve the strict 'return 1' for the new big_alloc4 (the sleepable variant that can use the slow path)?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315