bpf: arena: handle memory.max on fault-in with reclaim/OOM
Subsystems (5)
v4 cover bpf: arena: handle memory.max on fault-in with reclaim/OOM
Since commit e66fe1bc6d25 ("bpf: arena: Reintroduce memcg accounting"), arena pages are charged to the memcg of the process that created the arena. That accounting exposes two problems in the arena user page fault path.
1. The fault-in allocation runs under arena->spinlock, so it can only use the non-blocking allocator, which never reclaims. Once memory.current is at memory.max the allocation simply fails. Reaching memory.max is completely normal for a healthy application - e.g. reading a large file fills memory.current with page cache - so the process ends up killed for no real reason.
2. That failure is turned into VM_FAULT_SIGSEGV, which is misleading: the faulting address is a perfectly valid arena address. When we know it is an out-of-memory condition we can return VM_FAULT_OOM and let the memcg OOM path handle it properly.
Preallocate the page outside the lock (patch 2), the way do_anonymous_page() does, so the allocation can sleep and go through reclaim and the OOM path. This needs a sleepable allocator (patch 1), because can_alloc_pages() is a conservative guess for BPF program context and always forces the non-blocking allocator under PREEMPT_RT. patch 3&4 adds a selftest that faults an arena in under a memory.max limit: without the fix the child gets SIGSEGV on a valid address, with it the child is killed by the memcg OOM killer.
v3 -> v4:
- rebase bpf-next and fix conflict
- add Reviewed-by tag from Emil Tsalapatis
v2 -> v3:
- selftest: check the memcg OOM via memory.events "oom_kill" instead of the exit signal; it only aims to pass on the fixed kernel, since the unfixed SIGSEGV is racy.
v1 -> v2:
- Rebase on the separate deadlock fix (found by the Sashiko AI review), now applied to bpf-next.
- Honor the map's NUMA node on fault-in.
- Return VM_FAULT_SIGBUS for the non-recoverable faults (lock, range-tree and page-table failures); a scratch-page hole stays VM_FAULT_SIGSEGV only under BPF_F_SEGV_ON_FAULT. (Kumar Kartikeya Dwivedi)
- Add read_cgroup_file() to cgroup_helpers instead of open-coding the /mnt/... path in the test. (Emil Tsalapatis)
- Dump the cgroup memory stats on test failure to ease debugging.
v2:
https://lore.kernel.org/bpf/[email protected]/
v1:
https://lore.kernel.org/bpf/[email protected]/
Jiayuan Chen (4):
bpf: Add a sleepable page allocator for map memory
bpf: arena: allocate the fault-in page outside the lock selftests/bpf: Add read_cgroup_file() to cgroup_helpers selftests/bpf: Add a test for arena fault-in under memory.max
include/linux/bpf.h | 1 + kernel/bpf/arena.c | 92 ++++++++--- kernel/bpf/syscall.c | 21 ++- tools/testing/selftests/bpf/cgroup_helpers.c | 67 ++++++++ tools/testing/selftests/bpf/cgroup_helpers.h | 4 + .../selftests/bpf/prog_tests/arena_memcg.c | 156 ++++++++++++++++++ .../testing/selftests/bpf/progs/arena_memcg.c | 24 +++ 7 files changed, 340 insertions(+), 25 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_memcg.c create mode 100644 tools/testing/selftests/bpf/progs/arena_memcg.c
Signature · 1 line
2.43.0
PATCH v4 1/4 bpf: Add a sleepable page allocator for map memory
bpf_map_alloc_pages() picks the allocator via can_alloc_pages(), a conservative guess for BPF program context that is always false under PREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.
Add bpf_map_alloc_page_sleepable() for callers that know they are sleepable. Like the other bpf map allocators it places the page on the map's numa_node and does not follow the faulting task's NUMA mempolicy; arena memory is shared, so the map's node is the right placement policy. The next patch uses it from the arena page fault handler.
---
To sashiko:
- The alloc_pages_nolock() fallback still zeroes: it forces __GFP_ZERO internally and only accepts __GFP_ACCOUNT.
- __GFP_ZERO is unchanged from the existing __bpf_alloc_page(), and no arena-capable arch has D-cache aliasing, so there is no dcache concern.
--- include/linux/bpf.h | 1 + kernel/bpf/syscall.c | 21 +++++++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
@@ -2784,6 +2784,7 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);int bpf_map_alloc_pages (const struct bpf_map *map,int nid,unsigned long nr_pages,struct page **page_array); +struct page *bpf_map_alloc_page_sleepable (const struct bpf_map *map);#ifdef CONFIG_MEMCG void bpf_map_memcg_enter (const struct bpf_map *map,struct mem_cgroup **old_memcg,struct mem_cgroup **new_memcg);
kernel/bpf/syscall.c +17 -4
@@ -602,15 +602,14 @@ static bool can_alloc_pages(void) !IS_ENABLED (CONFIG_PREEMPT_RT ); } +#define BPF_PAGE_GFP (GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT | __GFP_NOWARN) +static struct page *__bpf_alloc_page (int nid) {if (!can_alloc_pages ())return alloc_pages_nolock (__GFP_ACCOUNT, nid,0 ); -return alloc_pages_node (nid, -GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT - | __GFP_NOWARN, -0 ); +return alloc_pages_node (nid,BPF_PAGE_GFP ,0 ); } int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
@@ -636,6 +635,20 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,return ret; } +/* +* For callers that know they run in a sleepable context, e.g. a user page +* fault handler. can_alloc_pages() is a conservative guess made for BPF +* program context - notably it is always false on PREEMPT_RT - so going +* through bpf_map_alloc_pages() there would needlessly pick the +* non-blocking allocator, which never reclaims and never engages the OOM +* machinery. +*/ +struct page *bpf_map_alloc_page_sleepable (const struct bpf_map *map) +{ +might_sleep (); +return alloc_pages_node (map->numa_node,BPF_PAGE_GFP ,0 ); +} +static int btf_field_cmp (const void *a,const void *b) {const struct btf_field *f1 = a, *f2 = b;
Signature · 1 line
2.43.0
PATCH v4 2/4 bpf: arena: allocate the fault-in page outside the lock
arena_vm_fault() allocated the page while holding arena->spinlock, so it could only use the non-blocking allocator. Once the memcg is at memory.max that allocation just fails, the fault turns into VM_FAULT_SIGSEGV, and the process gets a SIGSEGV on a perfectly valid arena address. Hitting memory.max is routine (e.g. page cache from reading a big file), so this kills innocent processes.
Rework the fault handler:
- Preallocate the page before taking the lock, like do_anonymous_page() does, so it can sleep, reclaim and go through the OOM path, and return VM_FAULT_OOM on failure so the memcg OOM handler runs instead of a fake segfault.
- A lockless probe skips that preallocation when a page is already mapped (e.g. allocated by the bpf program), so the common case wastes no allocation. The rare race where such a page is freed before we take the lock falls back to the non-blocking allocator under the lock.
- Return VM_FAULT_SIGBUS for the non-recoverable errors (lock failure, range-tree and page-table failures) instead of VM_FAULT_SIGSEGV; only BPF_F_SEGV_ON_FAULT, and a scratch-page hole under that flag, is a real user addressing error and keeps VM_FAULT_SIGSEGV.
- Tidy up the error labels.
--- kernel/bpf/arena.c | 92 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 71 insertions(+), 21 deletions(-)
kernel/bpf/arena.c +71 -21
@@ -481,7 +481,8 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)struct bpf_map *map = vmf->vma->vm_file->private_data; struct bpf_arena *arena = container_of(map, struct bpf_arena, map);struct mem_cgroup *new_memcg, *old_memcg; -struct page *page; +struct page *page, *new_page =NULL ; +vm_fault_t fault_ret;long kbase, kaddr;unsigned long flags;int ret;
@@ -489,59 +490,108 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) kbase =bpf_arena_get_kern_vm_start (arena); kaddr = kbase + (u32)(vmf->address); -if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) + page =vmalloc_to_page ((void *)kaddr); +if (!page && !(arena->map.map_flags &BPF_F_SEGV_ON_FAULT )) { +/* +* We run in process context here, so preallocate the page +* outside the lock with an explicitly sleepable allocator. It +* can then go through reclaim (both memcg and global) and the +* OOM path, the way do_anonymous_page() does; under +* arena->spinlock only the non-blocking allocator is available, +* which never reclaims. That also decides the return value: +* VM_FAULT_OOM below is only meaningful if the OOM machinery was +* actually engaged, which the non-blocking allocator never does. +*/ +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + new_page =bpf_map_alloc_page_sleepable (map); +bpf_map_memcg_exit (old_memcg, new_memcg); +if (!new_page) +return VM_FAULT_OOM ; + } + +if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) {/* * A failed lock means a possible deadlock was detected. Don't * return VM_FAULT_RETRY: this handler never took mmap_lock, but * the fault path would re-take it on retry and deadlock. Fail. */ +if (new_page) +free_pages_nolock (new_page,0 );return VM_FAULT_SIGBUS ; + } page =vmalloc_to_page ((void *)kaddr);if (page) { -if (page == arena->scratch_page) -/* BPF triggered scratch here; don't lazy-alloc over it */ - goto out_sigsegv; +if (page == arena->scratch_page) { +/* +* A scratch page marks a hole. Segfault only if the user +* asked for it; otherwise we could lazy-allocate but +* choose not to over a hole, so report a bus error. +*/ + fault_ret = (arena->map.map_flags &BPF_F_SEGV_ON_FAULT ) ? +VM_FAULT_SIGSEGV :VM_FAULT_SIGBUS ; + goto out_err_locked; + }/* already have a page vmap-ed */ goto out; } +if (arena->map.map_flags &BPF_F_SEGV_ON_FAULT ) { +/* User space requested to segfault when page is not allocated by bpf prog */ + fault_ret =VM_FAULT_SIGSEGV ; + goto out_err_locked; + } +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); -if (arena->map.map_flags &BPF_F_SEGV_ON_FAULT ) -/* User space requested to segfault when page is not allocated by bpf prog */ - goto out_sigsegv_memcg; +if (!new_page) { +/* +* Very rare race: the bpf program had allocated a page here, so +* the lockless probe saw it and we skipped preallocation, but it +* freed the page before we took the lock. Now we do need one; +* sleeping is not allowed here, so fall back to the non-blocking +* allocator and give up if it fails. +*/ + ret =bpf_map_alloc_pages (map, map->numa_node,1 , &new_page); +if (ret) { + fault_ret =VM_FAULT_SIGBUS ; + goto out_err_locked_memcg; + } + } ret =range_tree_clear (&arena->rt, vmf->pgoff,1 ); -if (ret) - goto out_sigsegv_memcg; - -struct apply_range_data data = { .arena = arena, .pages = &page, .i =0 }; -/* Account into memcg of the process that created bpf_arena */ - ret =bpf_map_alloc_pages (map,NUMA_NO_NODE ,1 , &page);if (ret) { -range_tree_set (&arena->rt, vmf->pgoff,1 ); - goto out_sigsegv_memcg; + fault_ret =VM_FAULT_SIGBUS ; + goto out_err_locked_memcg; } +struct apply_range_data data = { .arena = arena, .pages = &new_page, .i =0 }; ret =apply_to_page_range (&init_mm, kaddr,PAGE_SIZE , apply_range_set_cb, &data);if (ret) {range_tree_set (&arena->rt, vmf->pgoff,1 ); -free_pages_nolock (page,0 ); - goto out_sigsegv_memcg; + fault_ret =VM_FAULT_SIGBUS ; + goto out_err_locked_memcg; }flush_vmap_cache (kaddr,PAGE_SIZE );bpf_map_memcg_exit (old_memcg, new_memcg); +/* new_page was consumed */ + page = new_page; + new_page =NULL ; out:page_ref_add (page,1 );raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +if (new_page) +free_pages_nolock (new_page,0 ); vmf->page = page;return 0 ; -out_sigsegv_memcg: + +out_err_locked_memcg:bpf_map_memcg_exit (old_memcg, new_memcg); -out_sigsegv: +out_err_locked:raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); -return VM_FAULT_SIGSEGV ; +if (new_page) +free_pages_nolock (new_page,0 ); +return fault_ret; }static const struct vm_operations_struct arena_vm_ops = {
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 2/4] bpf: arena: allocate the fault-in page outside the lock
Can this return VM_FAULT_OOM when the allocation is charged to a foreign memcg?
Quoted 4 lines
kernel/bpf/arena.c +0
-0
[ ... ]
Quoted 12 lines
kernel/bpf/arena.c +8
-0@@ -489,59 +490,108 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) kbase =bpf_arena_get_kern_vm_start (arena); kaddr = kbase + (u32)(vmf->address); + page =vmalloc_to_page ((void *)kaddr); +if (!page && !(arena->map.map_flags &BPF_F_SEGV_ON_FAULT )) { +/* +* We run in process context here, so preallocate the page +* outside the lock with an explicitly sleepable allocator. It +* can then go through reclaim (both memcg and global) and the +* OOM path, the way do_anonymous_page() does; under +* arena->spinlock only the non-blocking allocator is available,
kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 510
Quoted 10 lines
kernel/bpf/arena.c +10
-0+* which never reclaims. That also decides the return value: +* VM_FAULT_OOM below is only meaningful if the OOM machinery was +* actually engaged, which the non-blocking allocator never does. +*/ +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + new_page =bpf_map_alloc_page_sleepable (map); +bpf_map_memcg_exit (old_memcg, new_memcg); +if (!new_page) +return VM_FAULT_OOM ; + }
Can this return VM_FAULT_OOM when the allocation is charged to a foreign memcg?
The preallocation charges the arena creator's memcg, not current's:
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
/* set_active_memcg(bpf_map_get_memcg(map)) */
new_page = bpf_map_alloc_page_sleepable(map);
/* GFP_KERNEL|__GFP_ZERO|__GFP_ACCOUNT|__GFP_NOWARN */For an order-0 GFP_KERNEL allocation the only realistic NULL is a memcg charge rejection. mem_cgroup_oom() calls out_of_memory() with oom_control pointing at the arena owner's memcg, so the OOM killer can only select victims inside that cgroup.
VM_FAULT_OOM reaches pagefault_out_of_memory() (mm/oom_kill.c:1180-1193), which only resolves current->memcg_in_oom; otherwise it prints the rate-limited "Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF" message and returns. Because the over-limit memcg is foreign, killing tasks there never makes fatal_signal_pending(current) true, so try_charge_memcg()'s force-charge escape hatch never fires.
If the owner memcg's charge is dominated by memory its tasks cannot free (for example the arena's own pages, which are only freed by the bpf program or by map destruction and are unreclaimable), or if that cgroup has no OOM-eligible tasks left, mem_cgroup_oom() returns false, try_charge_memcg() returns -ENOMEM, and the instruction is retried in a loop.
This creates an unbounded fault/retry livelock burning CPU in the faulting task plus dmesg spam, where the pre-image returned a bounded VM_FAULT_SIGSEGV. Before this patch the in-lock allocator never engaged reclaim or the OOM killer, so a user touching an arena page could not kill anything. Now any task that can mmap the arena (the fd can be passed via SCM_RIGHTS or a bpffs pin, so the faulting task need not be in the owner's cgroup at all) can drive the memcg OOM killer in the owner's cgroup one task at a time simply by faulting in pages.
VM_FAULT_OOM promises the caller that the OOM machinery was engaged on a domain where retrying can succeed. The in-code comment acknowledges the general point ("VM_FAULT_OOM below is only meaningful if the OOM machinery was actually engaged") but not the foreign-memcg case. Would it make sense to use a bounded status for the charge-failure case (VM_FAULT_SIGBUS, or VM_FAULT_OOM only when the charged memcg == current's memcg)? That would keep the fix's benefit (reclaim now runs, which is the routine memory.max case the changelog cares about) without the unbounded retry.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32450158690
Re: [PATCH bpf-next v4 2/4] bpf: arena: allocate the fault-in page outside the lock
Quoted 57 lines · On 8/21/26 1:59 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/arena.c +0
-0[ ... ]
Quoted 12 lines
kernel/bpf/arena.c +8
-0@@ -489,59 +490,108 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) kbase =bpf_arena_get_kern_vm_start (arena); kaddr = kbase + (u32)(vmf->address); + page =vmalloc_to_page ((void *)kaddr); +if (!page && !(arena->map.map_flags &BPF_F_SEGV_ON_FAULT )) { +/* +* We run in process context here, so preallocate the page +* outside the lock with an explicitly sleepable allocator. It +* can then go through reclaim (both memcg and global) and the +* OOM path, the way do_anonymous_page() does; under +* arena->spinlock only the non-blocking allocator is available, kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 510
Quoted 10 lines
kernel/bpf/arena.c +10
-0+* which never reclaims. That also decides the return value: +* VM_FAULT_OOM below is only meaningful if the OOM machinery was +* actually engaged, which the non-blocking allocator never does. +*/ +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + new_page =bpf_map_alloc_page_sleepable (map); +bpf_map_memcg_exit (old_memcg, new_memcg); +if (!new_page) +return VM_FAULT_OOM ; + }Can this return
VM_FAULT_OOMwhen the allocation is charged to a foreign memcg?The preallocation charges the arena creator's memcg, not current's:
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); /* set_active_memcg(bpf_map_get_memcg(map)) */ new_page = bpf_map_alloc_page_sleepable(map); /* GFP_KERNEL|__GFP_ZERO|__GFP_ACCOUNT|__GFP_NOWARN */For an order-0
GFP_KERNELallocation the only realistic NULL is a memcg charge rejection.mem_cgroup_oom()callsout_of_memory()withoom_controlpointing at the arena owner's memcg, so the OOM killer can only select victims inside that cgroup.
VM_FAULT_OOMreachespagefault_out_of_memory()(mm/oom_kill.c:1180-1193), which only resolvescurrent->memcg_in_oom; otherwise it prints the rate-limited "HuhVM_FAULT_OOMleaked out to the #PF handler. Retrying PF" message and returns. Because the over-limit memcg is foreign, killing tasks there never makesfatal_signal_pending(current)true, so try_charge_memcg()'s force-charge escape hatch never fires.If the owner memcg's charge is dominated by memory its tasks cannot free (for example the arena's own pages, which are only freed by the bpf program or by map destruction and are unreclaimable), or if that cgroup has no OOM-eligible tasks left,
mem_cgroup_oom()returns false,try_charge_memcg()returns-ENOMEM, and the instruction is retried in a loop.
Quoted 8 lines
This creates an unbounded fault/retry livelock burning CPU in the faulting task plus dmesg spam, where the pre-image returned a bounded
VM_FAULT_SIGSEGV. Before this patch the in-lock allocator never engaged reclaim or the OOM killer, so a user touching an arena page could not kill anything. Now any task that can mmap the arena (the fd can be passed viaSCM_RIGHTSor a bpffs pin, so the faulting task need not be in the owner's cgroup at all) can drive the memcg OOM killer in the owner's cgroup one task at a time simply by faulting in pages.
Thanks, you're right — this is a real bug, and I could reproduce it.
The root cause is simply that we return VM_FAULT_OOM.
That return value does nothing useful for arena: all reclaim and OOM handling (both memcg and global) already happens inside
bpf_map_alloc_page_sleepable().
The fix is to return VM_FAULT_SIGBUS instead of VM_FAULT_OOM.
PATCH v4 3/4 selftests/bpf: Add read_cgroup_file() to cgroup_helpers
In reply to Jiayuan Chencgroup_helpers has write_cgroup_file()/write_cgroup_file_parent() but no read counterpart. Add read_cgroup_file() and read_cgroup_file_parent() so a forked child can read a cgroup file (e.g. memory.current) from the work dir owned by the parent that set the environment up, without hand-building the /mnt/... path.
--- tools/testing/selftests/bpf/cgroup_helpers.c | 67 ++++++++++++++++++++ tools/testing/selftests/bpf/cgroup_helpers.h | 4 ++ 2 files changed, 71 insertions(+)
tools/testing/selftests/bpf/cgroup_helpers.c +67 -0
@@ -188,6 +188,73 @@ int write_cgroup_file_parent(const char *relative_path, const char *file,return __write_cgroup_file (cgroup_path, file, buf); } +static int __read_cgroup_file (const char *cgroup_path,const char *file, +char *buf,size_t len) +{ +char file_path[PATH_MAX +1 ]; +ssize_t got; +int fd; + +snprintf (file_path,sizeof (file_path),"%s/%s" , cgroup_path, file); + fd =open (file_path,O_RDONLY ); +if (fd <0 ) { +log_err ("Opening %s" , file_path); +return 1 ; + } + + got =read (fd, buf, len -1 ); +if (got <0 ) { +log_err ("Reading %s" , file_path); +close (fd); +return 1 ; + } + buf[got] ='\0' ; +close (fd); +return 0 ; +} + +/** +* read_cgroup_file() - Read from a cgroup file +* @relative_path: The cgroup path, relative to the workdir +* @file: The name of the file in cgroupfs to read from +* @buf: Buffer to read into, NUL-terminated on success +* @len: Size of @buf +* +* Read from a file in the given cgroup's directory. +* +* If successful, 0 is returned. +*/ +int read_cgroup_file (const char *relative_path,const char *file, +char *buf,size_t len) +{ +char cgroup_path[PATH_MAX -24 ]; + +format_cgroup_path (cgroup_path, relative_path); +return __read_cgroup_file (cgroup_path, file, buf, len); +} + +/** +* read_cgroup_file_parent() - Read from a cgroup file in the parent process +* workdir +* @relative_path: The cgroup path, relative to the parent process workdir +* @file: The name of the file in cgroupfs to read from +* @buf: Buffer to read into, NUL-terminated on success +* @len: Size of @buf +* +* Read from a file in the given cgroup's directory under the parent process +* workdir. +* +* If successful, 0 is returned. +*/ +int read_cgroup_file_parent (const char *relative_path,const char *file, +char *buf,size_t len) +{ +char cgroup_path[PATH_MAX -24 ]; + +format_parent_cgroup_path (cgroup_path, relative_path); +return __read_cgroup_file (cgroup_path, file, buf, len); +} +/** * setup_cgroup_environment() - Setup the cgroup environment *
tools/testing/selftests/bpf/cgroup_helpers.h +4 -0
@@ -15,6 +15,10 @@ int write_cgroup_file(const char *relative_path, const char *file, const char *buf);int write_cgroup_file_parent (const char *relative_path,const char *file,const char *buf); +int read_cgroup_file (const char *relative_path,const char *file, +char *buf,size_t len); +int read_cgroup_file_parent (const char *relative_path,const char *file, +char *buf,size_t len);int cgroup_setup_and_join (const char *relative_path);int get_root_cgroup (void );int create_and_get_cgroup (const char *relative_path);
Signature · 1 line
2.43.0
PATCH v4 4/4 selftests/bpf: Add a test for arena fault-in under memory.max
In reply to Jiayuan ChenA child joins a memcg capped 64M above its post-load usage and faults an arena in until it runs out of that budget.
With the fix the arena page comes from the sleepable allocator, so hitting memory.max goes through the memcg OOM path and the child is OOM-killed, which the test checks via memory.events "oom_kill".
Without the fix the test may still pass, because a concurrent blocking allocation in the child (e.g. a COW fault on an inherited page) can hit memory.max and OOM-kill it first. The goal is only that the fixed kernel passes reliably.
# test_progs -v -t arena_memcg serial_test_arena_memcg:PASS:child killed by signal serial_test_arena_memcg:PASS:memcg oom_kill #5 arena_memcg:OK
# dmesg (the OOM comes from the arena sleepable allocation) test_progs invoked oom-killer: gfp_mask=GFP_KERNEL_ACCOUNT|__GFP_ZERO arena_vm_fault+0x4bc/0xad0 Memory cgroup out of memory: Killed process 473 (test_progs)
--- .../selftests/bpf/prog_tests/arena_memcg.c | 156 ++++++++++++++++++ .../testing/selftests/bpf/progs/arena_memcg.c | 24 +++ 2 files changed, 180 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_memcg.c create mode 100644 tools/testing/selftests/bpf/progs/arena_memcg.c
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +156 -0
@@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <test_progs.h> +#include <fcntl.h> +#include <signal.h> +#include <sys/mman.h> +#include <sys/wait.h> +#include <unistd.h> +#include <sys/user.h> +#ifndef PAGE_SIZE /* on some archs it comes in sys/user.h */ +#include <unistd.h> +#define PAGE_SIZE getpagesize() +#endif + +#include "cgroup_helpers.h" +#include "arena_memcg.skel.h" + +#define CG_PATH "/arena_memcg" + +/* Budget the arena gets on top of whatever is already charged after load. */ +#define ARENA_BUDGET (64 * 1024 * 1024) + +static void dump_memcg (int (*rd)(const char *,const char *,char *,size_t )) +{ +char buf[512 ]; + +/* +* memory.current reads 0 once the child has left the cgroup, so it only +* carries information when dumped from the live child; memory.peak and +* memory.events survive the child and tell the story either way. +*/ +if (!rd (CG_PATH ,"memory.current" , buf,sizeof (buf))) +fprintf (stderr,"memory.current: %s" , buf); +if (!rd (CG_PATH ,"memory.max" , buf,sizeof (buf))) +fprintf (stderr,"memory.max: %s" , buf); +if (!rd (CG_PATH ,"memory.peak" , buf,sizeof (buf))) +fprintf (stderr,"memory.peak: %s" , buf); +if (!rd (CG_PATH ,"memory.events" , buf,sizeof (buf))) +fprintf (stderr,"memory.events:\n%s" , buf); +fflush (NULL );/* _exit() in the child would not flush stdio otherwise */ +} + +/* Read one key from a flat keyed cgroup file, e.g. "oom_kill" in memory.events. */ +static long cg_read_key (const char *cg,const char *file,const char *key) +{ +char buf[512 ], *p; + +if (read_cgroup_file (cg, file, buf,sizeof (buf))) +return -1 ; + p =strstr (buf, key); +if (!p) +return -1 ; +return strtol (p +strlen (key),NULL ,10 ); +} + +void serial_test_arena_memcg (void ) +{ +int cgroup_fd =-1 , status; +const long ps =PAGE_SIZE ; +char buf[64 ]; +pid_t pid; + +if (setup_cgroup_environment ()) +return ; + + cgroup_fd =create_and_get_cgroup (CG_PATH ); +if (!ASSERT_OK_FD (cgroup_fd,"create_and_get_cgroup" )) + goto out; + +/* No memory controller -> nothing to test. */ +if (read_cgroup_file (CG_PATH ,"memory.current" , buf,sizeof (buf))) { +test__skip (); + goto out; + } + + pid =fork (); +if (!ASSERT_GE (pid,0 ,"fork" )) + goto out; +if (pid ==0 ) { +struct arena_memcg *cskel; +__u32 i, npages; +char *base; +size_t sz; +long cur; + +/* +* Do everything from the child: the arena vma is VM_DONTCOPY so +* it would not survive fork(), only the child should be under the +* limit so that a memcg OOM cannot pick test_progs, and a map is +* charged to the memcg of the task that creates it - so join +* before load. The cgroup work dir belongs to the parent that set +* the environment up, so reach it with the _parent() helpers. +* Errors are reported to the parent through the exit code, since +* ASSERT_* in a forked child does not reach it. +*/ +snprintf (buf,sizeof (buf),"%d" ,getpid ()); +if (write_cgroup_file_parent (CG_PATH ,"cgroup.procs" , buf)) +_exit (2 ); + + cskel =arena_memcg__open_and_load (); +if (!cskel) +_exit (3 ); + + base =bpf_map__initial_value (cskel->maps.arena, &sz); +if (!base) +_exit (4 ); + npages =bpf_map__max_entries (cskel->maps.arena); + +/* +* Cap only now, after load: everything but the fault-in is +* charged, so the arena gets a fixed budget regardless of what +* the load itself cost, and the load can never hit the limit. +*/ +if (read_cgroup_file_parent (CG_PATH ,"memory.current" , buf,sizeof (buf))) +_exit (5 ); + cur =strtol (buf,NULL ,10 ); +snprintf (buf,sizeof (buf),"%ld" , cur +ARENA_BUDGET ); +if (write_cgroup_file_parent (CG_PATH ,"memory.max" , buf)) +_exit (6 ); + +for (i =0 ; i < npages; i++) + base[(size_t )i * ps] =1 ; +/* Faulted everything without dying: no pressure built, dump why. */ +dump_memcg (read_cgroup_file_parent); +_exit (0 ); + } + +if (!ASSERT_EQ (waitpid (pid, &status,0 ), pid,"waitpid" )) + goto out; + +/* A non-zero exit means the child failed to set up; the code says where. */ +if (WIFEXITED (status) &&WEXITSTATUS (status)) { +ASSERT_OK (WEXITSTATUS (status),"child setup" ); + goto out; + } + +/* +* Faulting a valid arena address until memory.max is hit must not look +* like an invalid access. Without the fix the fault path allocated with +* the non-blocking allocator, turned its -ENOMEM into VM_FAULT_SIGSEGV, +* and the child died with SIGSEGV on a valid address; now it is handled +* by the memcg OOM path instead. A SIGKILL alone would not prove the +* memcg OOM killer did it (a global OOM or an unrelated crash could also +* kill the child), so check memory.events.oom_kill, which records the +* memcg OOM and survives the child. +*/ +if (!ASSERT_TRUE (WIFSIGNALED (status),"child killed by signal" )) + goto out; +if (!ASSERT_GE (cg_read_key (CG_PATH ,"memory.events" ,"oom_kill" ),1 , +"memcg oom_kill" )) +dump_memcg (read_cgroup_file); +out: +if (cgroup_fd >=0 ) +close (cgroup_fd); +cleanup_cgroup_environment (); +}
tools/testing/selftests/bpf/progs/arena_memcg.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_arena_common.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 50000); /* number of pages */ +#ifdef __TARGET_ARCH_arm64 + __ulong(map_extra, 0x1ull << 32); /* start of mmap() region */ +#else + __ulong(map_extra, 0x1ull << 44); /* start of mmap() region */ +#endif +} arena SEC(".maps"); + +SEC("syscall") +int noop (void *ctx) +{ +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 4/4] selftests/bpf: Add a test for arena fault-in under memory.max
Looking at fflush(NULL) - does this need to flush every open stream? Under -v with output redirected, any test_progs output still buffered in stdout at fork() time would be emitted a second time by…
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +0
-0@@ -0,0 +1,156 @@
[ ... ]
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +9
-0+static void dump_memcg (int (*rd)(const char *,const char *,char *,size_t )) +{ +char buf[512 ]; + +/* +* memory.current reads 0 once the child has left the cgroup, so it only +* carries information when dumped from the live child; memory.peak and +* memory.events survive the child and tell the story either way. +*/
tools/testing/selftests/bpf/prog_tests/arena_memcg.c line 41
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +10
-0+if (!rd (CG_PATH ,"memory.current" , buf,sizeof (buf))) +fprintf (stderr,"memory.current: %s" , buf); +if (!rd (CG_PATH ,"memory.max" , buf,sizeof (buf))) +fprintf (stderr,"memory.max: %s" , buf); +if (!rd (CG_PATH ,"memory.peak" , buf,sizeof (buf))) +fprintf (stderr,"memory.peak: %s" , buf); +if (!rd (CG_PATH ,"memory.events" , buf,sizeof (buf))) +fprintf (stderr,"memory.events:\n%s" , buf); +fflush (NULL );/* _exit() in the child would not flush stdio otherwise */ +}
Looking at fflush(NULL) - does this need to flush every open stream? Under -v with output redirected, any test_progs output still buffered in stdout at fork() time would be emitted a second time by the child. Would fflush(stderr) be sufficient here?
Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +13
-0+ +/* Read one key from a flat keyed cgroup file, e.g. "oom_kill" in memory.events. */ +static long cg_read_key (const char *cg,const char *file,const char *key) +{ +char buf[512 ], *p; + +if (read_cgroup_file (cg, file, buf,sizeof (buf))) +return -1 ; + p =strstr (buf, key); +if (!p) +return -1 ; +return strtol (p +strlen (key),NULL ,10 ); +}
tools/testing/selftests/bpf/prog_tests/arena_memcg.c line 64
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +10
-0+ +void serial_test_arena_memcg (void ) +{ +int cgroup_fd =-1 , status; +const long ps =PAGE_SIZE ; +char buf[64 ]; +pid_t pid; + +if (setup_cgroup_environment ()) +return ;
Can setup_cgroup_environment() failing here result in a silent pass? setup_cgroup_environment() can fail at several points - mkdir, unshare, mount operations - meaning on a host without the right capabilities or where cgroup2 cannot be mounted, the test returns before any ASSERT_* runs and test_progs records arena_memcg:OK even though nothing was tested. Since the arena/memcg interaction is the entire point of the test, would wrapping this in ASSERT_OK() or adding an explicit skip be clearer?
tools/testing/selftests/bpf/prog_tests/arena_memcg.c line 74
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +10
-0+ + cgroup_fd =create_and_get_cgroup (CG_PATH ); +if (!ASSERT_OK_FD (cgroup_fd,"create_and_get_cgroup" )) + goto out; + +/* No memory controller -> nothing to test. */ +if (read_cgroup_file (CG_PATH ,"memory.current" , buf,sizeof (buf))) { +test__skip (); + goto out; + }
A subsystem pattern flags this as potentially concerning: when the memory controller is not available, test__skip() increments the skip counter but prints no reason - a #N arena_memcg:SKIP line in a CI log would give no indication of why. The comment records it in the source, but nothing emits it at runtime. Every other arena test in this directory prints an explicit reason with its skip:
prog_tests/arena_htab.c:
printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__); prog_tests/arena_list.c, arena_spin_lock.c, arena_atomics.c:
[similar patterns]
Would adding printf("%s:SKIP:no memory controller\n", __func__); before test__skip() match the neighbours?
Quoted 21 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +21
-0+ + pid =fork (); +if (!ASSERT_GE (pid,0 ,"fork" )) + goto out; +if (pid ==0 ) { +struct arena_memcg *cskel; +__u32 i, npages; +char *base; +size_t sz; +long cur; + +/* +* Do everything from the child: the arena vma is VM_DONTCOPY so +* it would not survive fork(), only the child should be under the +* limit so that a memcg OOM cannot pick test_progs, and a map is +* charged to the memcg of the task that creates it - so join +* before load. The cgroup work dir belongs to the parent that set +* the environment up, so reach it with the _parent() helpers. +* Errors are reported to the parent through the exit code, since +* ASSERT_* in a forked child does not reach it. +*/
[ ... ]
tools/testing/selftests/bpf/prog_tests/arena_memcg.c line 125
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +5
-0+for (i =0 ; i < npages; i++) + base[(size_t )i * ps] =1 ; +/* Faulted everything without dying: no pressure built, dump why. */ +dump_memcg (read_cgroup_file_parent); +_exit (0 );
Can the child's dump_memcg() output reach the parent's log? stdio_hijack_init() redirects stderr to an in-memory stream in non-verbose runs, and after fork() the child has a private copy-on-write copy of that FILE. The fprintf() calls in dump_memcg() and fflush(NULL) write only into the child's address space, then _exit(0) throws them away. So in a default test_progs run - exactly where diagnostics are needed - the parent then fails ASSERT_TRUE(WIFSIGNALED(status)) with no memcg state attached. The sample output in the commit message was captured with -v, the one mode where this path does work. The second dump_memcg() call at the end of the function runs in the parent and is unaffected. Would writing the numbers over a pipe, or having the parent dump memory.peak/memory.events on the !WIFSIGNALED path, get them into the log?
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +1
-0+ }
[ ... ]
tools/testing/selftests/bpf/prog_tests/arena_memcg.c line 151
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/arena_memcg.c new file +5
-0+if (!ASSERT_TRUE (WIFSIGNALED (status),"child killed by signal" )) + goto out; +if (!ASSERT_GE (cg_read_key (CG_PATH ,"memory.events" ,"oom_kill" ),1 , +"memcg oom_kill" )) +dump_memcg (read_cgroup_file);
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32450158690
Re: [PATCH bpf-next v4 1/4] bpf: Add a sleepable page allocator for map memory
In reply to Jiayuan ChenQuoted 5 lines · Jiayuan Chen
bpf_map_alloc_pages()picks the allocator viacan_alloc_pages(), a conservative guess for BPF program context that is always false underPREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.
I thought one of the main motivations was reentrancy. As you cannot really assume the context bpf_map_alloc_pages is called from there is an extra care needed so that this doesn't re-enter the allocator from bpf program called from allocator path and deadlock.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH bpf-next v4 1/4] bpf: Add a sleepable page allocator for map memory
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Jiayuan Chen
bpf_map_alloc_pages()picks the allocator viacan_alloc_pages(), a conservative guess for BPF program context that is always false underPREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.I thought one of the main motivations was reentrancy. As you cannot really assume the context
bpf_map_alloc_pagesis called from there is an extra care needed so that this doesn't re-enter the allocator from bpf program called from allocator path and deadlock.
Agreed, bpf_map_alloc_pages() is designed to be safe to run in any context.
The commit message should be precise.
Re: [PATCH bpf-next v4 1/4] bpf: Add a sleepable page allocator for map memory
Quoted 14 lines · Jiayuan Chen
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Jiayuan Chen
bpf_map_alloc_pages()picks the allocator viacan_alloc_pages(), a conservative guess for BPF program context that is always false underPREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.I thought one of the main motivations was reentrancy. As you cannot really assume the context
bpf_map_alloc_pagesis called from there is an extra care needed so that this doesn't re-enter the allocator from bpf program called from allocator path and deadlock.
Quoted 3 lines
Agreed,
bpf_map_alloc_pages()is designed to be safe to run in any context.The commit message should be precise.
How do you achive any level of safety for the _sleepable version? Vast majority of the kernel is sleepable but that doesn't mean this is safe from the mm reentrancy POV.
Signature · 2 lines
Michal Hocko SUSE Labs
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v4 1/4] bpf: Add a sleepable page allocator for map memory
Quoted 15 lines · Michal Hocko
Quoted 14 lines · Jiayuan Chen
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Jiayuan Chen
bpf_map_alloc_pages()picks the allocator viacan_alloc_pages(), a conservative guess for BPF program context that is always false underPREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.I thought one of the main motivations was reentrancy. As you cannot really assume the context
bpf_map_alloc_pagesis called from there is an extra care needed so that this doesn't re-enter the allocator from bpf program called from allocator path and deadlock.
Quoted 7 lines
Quoted 3 lines
Agreed,
bpf_map_alloc_pages()is designed to be safe to run in any context.The commit message should be precise.
How do you achive any level of safety for the
_sleepableversion? Vast majority of the kernel is sleepable but that doesn't mean this is safe from the mm reentrancy POV.
I think it will only be invoked directly in the arena map fault handler, which runs in task context. If an interrupt occurs and tries to allocate pages, can_alloc_pages() will return false and it should pick the _nolock() variant which is safe against reentrancy.
Re: [PATCH bpf-next v4 1/4] bpf: Add a sleepable page allocator for map memory
Quoted 20 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Jiayuan Chen
Quoted 10 lines · Michal Hocko
Quoted 5 lines · Jiayuan Chen
bpf_map_alloc_pages()picks the allocator viacan_alloc_pages(), a conservative guess for BPF program context that is always false underPREEMPT_RT. So even a caller that really is sleepable gets the non-blocking allocator, which never reclaims and never engages the OOM machinery.I thought one of the main motivations was reentrancy. As you cannot really assume the context
bpf_map_alloc_pagesis called from there is an extra care needed so that this doesn't re-enter the allocator from bpf program called from allocator path and deadlock.Agreed,
bpf_map_alloc_pages()is designed to be safe to run in any context.The commit message should be precise.
Quoted 8 lines
Quoted 3 lines
How do you achive any level of safety for the
_sleepableversion? Vast majority of the kernel is sleepable but that doesn't mean this is safe from the mm reentrancy POV.I think it will only be invoked directly in the arena map fault handler, which runs in task context. If an interrupt occurs and tries to allocate pages,
can_alloc_pages()will return false and it should pick the_nolock()variant which is safe against reentrancy.
So you rely on callers to know what they are doing. If that is the case and generally acceptable by the BPF community (no real saying from me in that matter) then make sure all that is properly documented. Because sleepable context is not merely enough.
Signature · 2 lines
Michal Hocko SUSE Labs