bpf: Fix arena memory incoherence
Subsystems (3)
Emil Tsalapatis
reviewerv6 cover bpf: Fix arena memory incoherence
Arena memory is tracked in two places: the range tree, which records which addresses are allocated, and the kernel and user page tables. The two are updated under different locks, and that leaves two races:
- Since b8467290edab ("bpf: arena: make arena kfuncs any context safe"), a free releases its range before the kernel TLB flush and the user zap complete. A concurrent allocation can then write through stale translations into pages that are about to be freed.
- A user fault takes a reference to the page and installs the user PTE after dropping the arena lock. A free that runs in between leaves user space mapping a page that the arena has already released.
In both cases BPF programs and user space can see different pages at the same arena address.
- A free keeps its range allocated until the TLB flush and the user zap are complete, and only then releases it with range_tree_set(). User faults on the range get SIGBUS until then. Frees serialize on zap_mutex.
- A user fault pins the page, locks its folio, and revalidates the kernel mapping before installing the user PTE, the way filemap_fault() synchronizes with truncate. The free waits for the folio lock of every page it removed before it zaps user mappings.
Patches 1 and 2 fix the two races. Patch 3 adds selftests.
v5 -> v6 (https://lore.kernel.org/bpf/[email protected]/)
- Return the locked page from arena_vm_fault() with VM_FAULT_LOCKED instead of installing the user PTE there with finish_fault(). The fault checks PageHWPoison() before it returns the page. (Alexei)
- Give faults on a range that is being freed the same SIGBUS as faults on a reserved range. Drop the record of the range being freed and the fault-side wait. (Alexei)
- Drop the trylock and deferral scheme. Frees serialize on zap_mutex and zap user mappings with zap_pages(), which takes mmap_read_lock() of each mapping process as in mainline. Keep the single-page zap shortcut. Sleepable frees are deferred only when the arena spinlock cannot be taken, and the worker retries such a free in place instead of requeueing itself. (Alexei)
- Drop v5 patch 1. The worker no longer requeues itself, so the flush_work() in arena_map_free() waits for all of its work. (Alexei)
- Drop v5 patch 4, which retried faults when the arena lock timed out. (Alexei)
- Drop the immediate release of ranges without kernel PTEs; they go through the regular free path.
- Selftests: map the page into user space before the paused free, and let the fault_free_realloc reader skip reads that raise SIGBUS.
v4 -> v5 (https://lore.kernel.org/bpf/[email protected]/)
- Drop the third range-tree state and the is_range_tree_set() change (v4 patches 4 and 5). A free keeps its range allocated until the TLB flush and the user zap complete, then releases it with range_tree_set(). range_tree_clear() only allocates when the whole range is free, and then there is nothing to free. (Alexei)
- Drop the per-arena mapping, page->mapping, the mapping-tree zap and the new bpf_map_ops callback (v4 patches 1 and 2). (Alexei)
- Replace opref (v4 patch 7) with the folio lock. A fault rechecks the kernel PTE with the folio locked and installs the user PTE itself with finish_fault(), and a free waits for the folio lock of each page it removed before zapping (new patch 2). (Alexei)
- Give faults on a reserved range the scratch_page error. Faults on the range being freed sleep until the free releases it, instead of spinning on VM_FAULT_RETRY. (Alexei)
- Trylock arena->lock and the mmap read lock of every process that maps the arena before retiring a range, and defer the free if one is contended. The worker waits for the contended lock instead of requeueing itself. The deferred span, allocated with __GFP_RETRY_MAYFAIL when the caller can sleep, is the only allocation before the ptes are cleared. (Alexei)
- Release a range without kernel ptes, such as a reserved one, right away without arena->lock or mmap_lock, so freeing and reallocating a reserved range is never deferred in sleepable context.
- Retry faults when the arena lock times out, keeping SIGBUS for -EDEADLK (new patch 4).
- Selftests: rename the subtests, and pause frees in zap_vma_range(), which arm64 can trace, instead of flush_tlb_kernel_range().
v3 -> v4 (https://lore.kernel.org/bpf/[email protected]/)
- Add per-arena anonymous mappings and associate pages with them
- Return VM_FAULT_LOCKED and let generic MM install user PTEs; wait for locked fault pages before TLB flushing and zapping (Alexei).
- Add mutual exclusion between arena free and fault operations to prevent incoherency.
- Return VM_FAULT_NOPAGE for non-retryable GUP faults when progress is impossible.
- Stop and drain self-requeueing deferred frees before map teardown.
- Track ownership of unavailable ranges independently of free progress and retry worker reservations on -ENOMEM without starting teardown.
- Make arena free span state machine explicit.
v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/)
- Rebase on bpf-next and resolve conflicts
- Fold in the atomic mark_available range tree operation into initial fix patches
v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/)
- Use __range_iter_next() in is_range_tree_set() (bot-ci)
- Move -EAGAIN handling to the patch where the errno is introduced (bot-ci)
- Add refactoring patch for (Puranjay)
- Avoid leaking unavailable range tree nodes with atomic mark_available operation()
- Avoid retries on res spinlock locking for -EDEADLK (Puranjay)
Emil Tsalapatis (3):
bpf: Revalidate faulted arena pages before installing user mappings
bpf: Fix arena race between page free and alloc leading to incoherency selftests/bpf: Add arena allocation race tests
kernel/bpf/arena.c | 177 +++++----- .../selftests/bpf/prog_tests/arena_race.c | 320 ++++++++++++++++++ .../testing/selftests/bpf/progs/arena_race.c | 166 +++++++++ 3 files changed, 575 insertions(+), 88 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_race.c create mode 100644 tools/testing/selftests/bpf/progs/arena_race.c
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v6 1/3 bpf: Revalidate faulted arena pages before installing user mappings
A user fault on an arena pins the page that the kernel arena maps, drops the arena spinlock and returns the page to the core fault code, which then locks the folio and installs the user PTE. A concurrent free can clear the kernel mapping in that window. The page reference prevents a use-after-free, but the fault can still install a user PTE for a page the arena has already released. User space and BPF programs then see different memory at the same arena offset.
Close the window from the fault side. Lock the page, recheck that the kernel mapping still points to it, and return it with VM_FAULT_LOCKED so that it stays locked until the core fault code has installed the user PTE. If a free removed the page, drop it and return VM_FAULT_NOPAGE so that the fault starts over. Return VM_FAULT_HWPOISON for a poisoned page.
This change alone does not close the race, because the free path does not yet wait for the page lock. The following change makes frees wait for in-flight faults before zapping user mappings.
--- kernel/bpf/arena.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
kernel/bpf/arena.c +16 -2
@@ -584,8 +584,22 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);if (new_page)free_pages_nolock (new_page,0 ); - vmf->page = page; -return 0 ; + +/* +* A free clears the kernel pte and then takes the page lock before it +* zaps user vma-s. Either it waits for this fault and zaps its pte, or +* the page is gone from here. +*/ +lock_page (page); +if (vmalloc_to_page ((void *)kaddr) == page && !PageHWPoison (page)) { + vmf->page = page; +return VM_FAULT_LOCKED ; + } +/* The page was freed meanwhile. Let the fault start over. */ + fault_ret =PageHWPoison (page) ?VM_FAULT_HWPOISON :VM_FAULT_NOPAGE ; +unlock_page (page); +put_page (page); +return fault_ret; out_err_locked_memcg:bpf_map_memcg_exit (old_memcg, new_memcg);
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v6 1/3] bpf: Revalidate faulted arena pages before installing user mappings
TL;DR: Possible NULL pointer dereference in arena_vm_fault(): returning VM_FAULT_LOCKED for mapping-less arena pages may crash __do_fault() if a memory failure poisons a mapped page right after the…
Quoted 1 line
bpf: Revalidate faulted arena pages before installing user mappings
TL;DR: Possible NULL pointer dereference in arena_vm_fault(): returning VM_FAULT_LOCKED for mapping-less arena pages may crash __do_fault() if a memory failure poisons a mapped page right after the check.
Quoted 4 lines
kernel/bpf/arena.c +0
-0
[ ... ]
Quoted 7 lines
kernel/bpf/arena.c +1
-2@@ -584,8 +584,22 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);if (new_page)free_pages_nolock (new_page,0 ); - vmf->page = page; -return 0 ; +
kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 597
Quoted 10 lines
kernel/bpf/arena.c +10
-0+/* +* A free clears the kernel pte and then takes the page lock before it +* zaps user vma-s. Either it waits for this fault and zaps its pte, or +* the page is gone from here. +*/ +lock_page (page); +if (vmalloc_to_page ((void *)kaddr) == page && !PageHWPoison (page)) { + vmf->page = page; +return VM_FAULT_LOCKED ; + }
^^^^
Can this return path lead to a NULL pointer dereference when the page is poisoned after this check?
Once arena_vm_fault() returns VM_FAULT_LOCKED, __do_fault() takes the locked branch of its hwpoison handling, which assumes a page-cache folio with a valid folio->mapping. Arena pages come from alloc_pages_node()/alloc_pages_nolock() and never get a mapping, so folio->mapping is NULL.
The !PageHWPoison(page) test keeps already-poisoned pages out of that branch, but it is a check-then-act sequence. memory_failure() runs TestSetPageHWPoison(p) before it takes a reference or the folio lock, so holding the page lock here does not stop it from setting the flag. __do_fault() then reads the flag again:
ret = vma->vm_ops->fault(vmf); /* arena: VM_FAULT_LOCKED */
...
if (unlikely(PageHWPoison(vmf->page))) {
vm_fault_t poisonret = VM_FAULT_HWPOISON;
if (ret & VM_FAULT_LOCKED) {
if (folio_mapped(folio))
unmap_mapping_folio(folio);and unmap_mapping_folio() does:
struct address_space *mapping = folio->mapping; /* NULL */
...
i_mmap_lock_read(mapping); /* down_read(&mapping->i_mmap_rwsem) */A possible sequence:
1. Processes A and B mmap the same arena. This is allowed, since arena_map_mmap() lets a second process use the same address. Page P is already mapped by a user PTE in A, so folio_mapped(P) is true via folio_add_file_rmap_ptes() in set_pte_range().
2. B touches the same offset. arena_vm_fault() locks P, sees vmalloc_to_page(kaddr) == P and !PageHWPoison(P), and returns VM_FAULT_LOCKED.
3. Before B's __do_fault() reads PageHWPoison(vmf->page), memory_failure() runs for pfn(P) on another CPU (MCE/GHES, madvise(MADV_HWPOISON) from A, or hwpoison_inject) and sets PG_hwpoison.
4. B's __do_fault() sees PageHWPoison, VM_FAULT_LOCKED and folio_mapped(), and calls unmap_mapping_folio() with a NULL mapping, which dereferences NULL in i_mmap_lock_read().
Before this patch the handler returned 0, so the poisoned branch in __do_fault() only did folio_put() and returned VM_FAULT_HWPOISON without touching folio->mapping. The window is short, but this opens a crash path during memory-error recovery.
tcmu_vma_fault() also returns VM_FAULT_LOCKED for mapping-less pages, which suggests the mm core gap is latent elsewhere, but that does not make the arena case safe.
Would it make sense to give arena pages a mapping the way fb_deferred_io does, or to make the mm core skip unmap_mapping_folio() when folio->mapping is NULL?
The following commit in the series (e49bf3a39cc3 "bpf: Fix arena race between page free and alloc leading to incoherency") does not touch this path.
kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 605
Quoted 8 lines
kernel/bpf/arena.c +5
-0+/* The page was freed meanwhile. Let the fault start over. */ + fault_ret =PageHWPoison (page) ?VM_FAULT_HWPOISON :VM_FAULT_NOPAGE ; +unlock_page (page); +put_page (page); +return fault_ret; out_err_locked_memcg:bpf_map_memcg_exit (old_memcg, new_memcg);
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39318700728
Emil Tsalapatis
reviewerPATCH v6 2/3 bpf: Fix arena race between page free and alloc leading to incoherency
In reply to Emil TsalapatisExisting arena kfunc code has an underlying race condition that can lead to writes being lost from the BPF program's point of view:
a) A memory range gets freed by operation (1), and its range is added back to the arena range tree.
b) A concurrent allocation (2) reallocates the range, and does writes to it. Writes from that CPU may follow the stale TLB entries into the pages that are about to be freed.
c) (1) invalidates the TLB. The old pages, and any writes done to them, are now inaccessible. zap_pages() similarly removes the mappings for userspace threads.
This can be triggered by particularly demanding BPF arena data structures that constantly allocate and deallocate memory, like hash table allocations.
Solve this ABA problem by keeping the range allocated until TLB invalidation and unmapping are complete. Under the arena spinlock, claim any free pages in the range and clear its kernel PTEs. Then drop the spinlock, lock and unlock each cleared page to wait for faults that are still installing a user PTE, flush the kernel TLB and zap the user page tables. Finally, retake the spinlock to release the range. Frees serialize on zap_mutex, so an overlapping free cannot release the range early. Claiming the range only allocates a range tree node when the whole range is free, and then there is nothing to free.
User faults on a range that is allocated without a page now raise SIGBUS instead of populating it. That covers a range reserved with bpf_arena_reserve_pages() and a range whose kernel PTEs a free has already cleared. Touching a page while a BPF program frees it is a user space bug, and in a BPF_F_SEGV_ON_FAULT arena it already raises SIGSEGV.
Sleepable frees run synchronously and are deferred only when the arena spinlock cannot be taken. The deferred free worker retries such a free in place instead of requeueing itself, so the flush_work() in arena_map_free() waits for all of its work.
--- kernel/bpf/arena.c | 159 +++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 86 deletions(-)
kernel/bpf/arena.c +73 -86
@@ -543,6 +543,12 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) goto out_err_locked; } +if (is_range_tree_set (&arena->rt, vmf->pgoff,1 )) { +/* Allocated without a page: reserved or being freed */ + fault_ret =VM_FAULT_SIGBUS ; + goto out_err_locked; + } +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg);if (!new_page) {
@@ -868,7 +874,7 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)* otherwise call B's mark would make call A skip a vml that A has * not yet zapped for A's uaddr range. */ -mutex_lock (&arena->zap_mutex); +lockdep_assert_held (&arena->zap_mutex);mutex_lock (&arena->lock); my_gen = ++arena->zap_gen;for (;;) {
@@ -902,49 +908,41 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)mutex_lock (&arena->lock); }mutex_unlock (&arena->lock); -mutex_unlock (&arena->zap_mutex); } -static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt,bool sleepable) +/* +* The range stays allocated in the range tree until the pages are unmapped +* from vmalloc area and from all user space vma-s, so that nobody can +* populate it while stale TLB entries or user ptes are still around. +*/ +static int arena_free_range (struct bpf_arena *arena,long uaddr,long page_cnt) { -struct mem_cgroup *new_memcg, *old_memcg; -u64 full_uaddr, uaddr_end; -long kaddr, pgoff; -struct page *page; -struct llist_head free_pages; -struct llist_node *pos, *t; -struct arena_free_span *s; +u64 full_uaddr =clear_lo32 (arena->user_vm_start) + uaddr; +long kaddr =bpf_arena_get_kern_vm_start (arena) + uaddr; +long pgoff =compute_pgoff (arena, uaddr);struct clear_range_data cdata; +struct llist_node *pos, *t; +LLIST_HEAD (free_pages);unsigned long flags; -int ret =0 ; - -/* only aligned lower 32-bit are relevant */ - uaddr = (u32 )uaddr; - uaddr &=PAGE_MASK ; - kaddr =bpf_arena_get_kern_vm_start (arena) + uaddr; - full_uaddr =clear_lo32 (arena->user_vm_start) + uaddr; -if (full_uaddr < arena->user_vm_start) -return ; - uaddr_end =min (arena->user_vm_end, full_uaddr + (page_cnt <<PAGE_SHIFT )); -if (full_uaddr >= uaddr_end) -return ; - - page_cnt = (uaddr_end - full_uaddr) >>PAGE_SHIFT ; - pgoff =compute_pgoff (arena, uaddr); -bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); +struct page *page; +int ret; -if (!sleepable) - goto defer; +/* An overlapping free must not release the range before this one is done */ +guard (mutex)(&arena->zap_mutex); ret =raw_res_spin_lock_irqsave (&arena->spinlock, flags); - -/* Can't proceed without holding the spinlock so defer the free */ if (ret) - goto defer; +return ret; -range_tree_set (&arena->rt, pgoff, page_cnt); +/* +* Take the free pages of the range too. It allocates, and can fail, +* only when the whole range is free. Then there is nothing to do. +*/ +if (range_tree_clear (&arena->rt, pgoff, page_cnt)) { +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +return 0 ; + } -init_llist_head (&free_pages); cdata.arena = arena; cdata.free_pages = &free_pages;/* clear ptes and collect struct pages */
@@ -954,6 +952,12 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,/* drop the lock to do the tlb flush and zap pages */ raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +/* wait for arena_vm_fault() that found the page before its pte was cleared */ +llist_for_each_entry (page, free_pages.first, pcp_llist) { +lock_page (page); +unlock_page (page); + } +/* ensure no stale TLB entries */ flush_tlb_kernel_range (kaddr, kaddr + (page_cnt *PAGE_SIZE ));
@@ -972,8 +976,39 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,zap_pages (arena, full_uaddr,1 );__free_page (page); } -bpf_map_memcg_exit (old_memcg, new_memcg); +/* The callers hold no other rqspinlock, so this one can only time out */ +while (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) +cond_resched (); +range_tree_set (&arena->rt, pgoff, page_cnt); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +return 0 ; +} + +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt,bool sleepable) +{ +struct mem_cgroup *new_memcg, *old_memcg; +u64 full_uaddr, uaddr_end; +struct arena_free_span *s; + +/* only aligned lower 32-bit are relevant */ + uaddr = (u32 )uaddr; + uaddr &=PAGE_MASK ; + full_uaddr =clear_lo32 (arena->user_vm_start) + uaddr; +if (full_uaddr < arena->user_vm_start) +return ; + uaddr_end =min (arena->user_vm_end, full_uaddr + (page_cnt <<PAGE_SHIFT )); +if (full_uaddr >= uaddr_end) +return ; + + page_cnt = (uaddr_end - full_uaddr) >>PAGE_SHIFT ; +bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); + +/* Can't proceed without holding the spinlock so defer the free */ +if (!sleepable ||arena_free_range (arena, uaddr, page_cnt)) + goto defer; + +bpf_map_memcg_exit (old_memcg, new_memcg);return ; defer:
@@ -1035,66 +1070,18 @@ static void arena_free_worker(struct work_struct *work) { struct bpf_arena *arena = container_of(work, struct bpf_arena, free_work);struct mem_cgroup *new_memcg, *old_memcg; -struct llist_node *list, *pos, *t; +struct llist_node *pos, *t;struct arena_free_span *s; -u64 arena_vm_start, user_vm_start; -struct llist_head free_pages; -struct clear_range_data cdata; -struct page *page; -unsigned long full_uaddr; -long kaddr, page_cnt, pgoff; -unsigned long flags; - -if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) { -schedule_work (work); -return ; - }bpf_map_memcg_enter (&arena->map, &old_memcg, &new_memcg); -init_llist_head (&free_pages); - cdata.arena = arena; - cdata.free_pages = &free_pages; - arena_vm_start =bpf_arena_get_kern_vm_start (arena); - user_vm_start =bpf_arena_get_user_vm_start (arena); - - list =llist_del_all (&arena->free_spans); -llist_for_each (pos, list) { - s = llist_entry(pos, struct arena_free_span, node); - page_cnt = s->page_cnt; - kaddr = arena_vm_start + s->uaddr; - pgoff =compute_pgoff (arena, s->uaddr); - -/* clear ptes and collect pages in free_pages llist */ -apply_to_existing_page_range (&init_mm, kaddr, page_cnt <<PAGE_SHIFT , - apply_range_clear_cb, &cdata); - -range_tree_set (&arena->rt, pgoff, page_cnt); - } -raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); - -/* Iterate the list again without holding spinlock to do the tlb flush and zap_pages */ -llist_for_each_safe (pos, t, list) { +llist_for_each_safe (pos, t,llist_del_all (&arena->free_spans)) { s = llist_entry(pos, struct arena_free_span, node); - page_cnt = s->page_cnt; - full_uaddr =clear_lo32 (user_vm_start) + s->uaddr; - kaddr = arena_vm_start + s->uaddr; - -/* ensure no stale TLB entries */ -flush_tlb_kernel_range (kaddr, kaddr + (page_cnt *PAGE_SIZE )); - -/* remove pages from user vmas */ -zap_pages (arena, full_uaddr, page_cnt); - +while (arena_free_range (arena, s->uaddr, s->page_cnt)) +cond_resched ();kfree_nolock (s); } -/* free all pages collected by apply_to_existing_page_range() in the first loop */ -llist_for_each_safe (pos, t,__llist_del_all (&free_pages)) { - page = llist_entry(pos, struct page, pcp_llist); -__free_page (page); - } -bpf_map_memcg_exit (old_memcg, new_memcg); }
Signature · 1 line
2.52.0
Emil Tsalapatis
reviewerPATCH v6 3/3 selftests/bpf: Add arena allocation race tests
In reply to Emil TsalapatisAdd selftests for races between arena frees, reallocation and user faults:
- realloc_during_free, realloc_during_deferred_free: a synchronous or deferred free keeps its range allocated until it completes.
- fault_free_realloc: user and BPF views of a page stay coherent while it is repeatedly faulted, freed and reallocated.
Pause frees in zap_vma_range(), after the range's kernel mappings are gone. Unlike flush_tlb_kernel_range(), which is inline on arm64, it is generic MM code that fentry can attach to. Map the page into user space first, so that the free has to zap it. Print why a subtest is skipped when a tracing target is missing. The reader in fault_free_realloc skips reads that raise SIGBUS, which a fault on a page that is being freed does.
--- .../selftests/bpf/prog_tests/arena_race.c | 320 ++++++++++++++++++ .../testing/selftests/bpf/progs/arena_race.c | 166 +++++++++ 2 files changed, 486 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_race.c create mode 100644 tools/testing/selftests/bpf/progs/arena_race.c
tools/testing/selftests/bpf/prog_tests/arena_race.c new file +320 -0
@@ -0,0 +1,320 @@ +// SPDX-License-Identifier: GPL-2.0 + +#define _GNU_SOURCE +#include <sched.h> +#include <setjmp.h> +#include <signal.h> +#include <sys/syscall.h> +#include <test_progs.h> + +#include "arena_race.skel.h" + +struct free_thread_ctx { +struct arena_race *skel; +int err; +__u32 retval; +}; + +struct fault_thread_ctx { +__u64 *addr; +int stop; +}; + +static int run_prog (struct bpf_program *prog,const char *name) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +int err; + + err =bpf_prog_test_run_opts (bpf_program__fd (prog), &opts); +return ASSERT_OK (err, name) &&ASSERT_OK (opts.retval, name) ?0 :-1 ; +} + +/* Trigger the sleepable free page path. */ +static void *run_free_thread (void *arg) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +struct free_thread_ctx *ctx = arg; + + ctx->skel->bss->target_tid =sys_gettid (); + ctx->err =bpf_prog_test_run_opts ( +bpf_program__fd (ctx->skel->progs.free_page), &opts); + ctx->retval = opts.retval; +return NULL ; +} + +static __threadsigjmp_buf reader_env; +static __threadbool reader_armed; + +/* Only the reader expects SIGBUS; anywhere else it keeps its default action. */ +static void reader_sigbus (int sig) +{ +if (!reader_armed) { +signal (sig,SIG_DFL ); +raise (sig); +return ; + } +siglongjmp (reader_env,1 ); +} + +/* Continuously fault in the address. */ +static void *fault_reader_thread (void *arg) +{ +struct fault_thread_ctx *ctx = arg; +__u64 *addr; + +/* Touching a page while BPF frees it raises SIGBUS; skip that read. */ +sigsetjmp (reader_env,1 ); + reader_armed = true; +while (!READ_ONCE (ctx->stop)) { + addr =__atomic_load_n (&ctx->addr, __ATOMIC_ACQUIRE); + (void )READ_ONCE (*addr); + } + reader_armed = false; +return NULL ; +} + +static int wait_for (int *p) +{ +__u64 deadline =get_time_ns () +5ULL *1000 *1000 *1000 ; + +while (!READ_ONCE (*p)) { +if (get_time_ns () > deadline) +return -ETIMEDOUT ; + } +return 0 ; +} + +/* Skip the subtest, saying why, if @func cannot be traced with fentry. */ +static bool require_fentry_target (const char *test,const char *func) +{ +if (libbpf_find_vmlinux_btf_id (func,BPF_TRACE_FENTRY ) >0 ) +return true; +printf ("%s:SKIP: %s is not an fentry target\n" , test, func); +test__skip (); +return false; +} + +static struct arena_race *setup_arena (__u64 **addr,bool trace_zap) +{ +struct arena_race *skel; +size_t arena_sz; +char *base; +int err; + + skel =arena_race__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return NULL ; + err =bpf_program__set_autoload (skel->progs.trace_zap, trace_zap); +if (!ASSERT_OK (err,"trace_zap_autoload" )) + goto err_out; + + err =arena_race__load (skel); +if (!ASSERT_OK (err,"load" )) + goto err_out; + err =arena_race__attach (skel); +if (!ASSERT_OK (err,"attach" )) + goto err_out; + +if (run_prog (skel->progs.alloc_old,"alloc_old" )) + goto err_out; +if (skel->bss->skip) { +printf ("%s:SKIP: compiler doesn't support arena_cast\n" , __func__); +test__skip (); + goto err_out; + } + + base =bpf_map__initial_value (skel->maps.arena, &arena_sz); +if (!ASSERT_OK_PTR (base,"arena_base" )) + goto err_out; + *addr = (__u64 *)(base +getpagesize ()); +if (!ASSERT_EQ ((unsigned long )skel->bss->ptr, (unsigned long )*addr, +"arena_ptr" )) + goto err_out; +return skel; + +err_out: +arena_race__destroy (skel); +return NULL ; +} + +static void test_realloc_during_free (bool deferred) +{ +struct free_thread_ctx ctx = {}; +struct arena_race *skel; +pthread_t thread; +__u64 *addr; +bool thread_created = false, zap_seen = false, completed = false; +int err; + +if (!require_fentry_target (__func__,"zap_vma_range" )) +return ; + + skel =setup_arena (&addr, true); +if (!skel) +return ; + +/* Map the page into user space, so that the free has to zap it. */ +if (!ASSERT_EQ (READ_ONCE (*addr), skel->rodata->old_marker,"old_marker" )) + goto out; + +/* Pause the free after its TLB flush, before it releases the range. */ + skel->bss->pause_on_zap =1 ; + +if (deferred) { +/* +* Test the nonsleepable free path that gets +* deferred to a worker in the kernel. We do +* so by triggering the arena operation from +* a nonsleepable tracepoint context. +*/ + skel->bss->trigger_pid_tgid = + ((__u64 )getpid () <<32 ) | (__u32 )sys_gettid (); + skel->bss->trigger_syscall = SYS_getpgid; + skel->bss->deferred_free =1 ; +if (!ASSERT_GE (syscall (SYS_getpgid,0 ),0 ,"deferred_free" )) + goto release; + }else { +/* +* Test the sleepable arena free path through a +* syscall test prog. +*/ + ctx.skel = skel; + err =pthread_create (&thread,NULL , run_free_thread, &ctx); +if (!ASSERT_OK (err,"pthread_create" )) { + skel->bss->release =1 ; + goto out; + } + thread_created = true; + } + +/* Wait until the free reaches the user zap. */ + err =wait_for (&skel->bss->zap_entered); +if (!ASSERT_OK (err,"zap_entered" )) + goto release; + + zap_seen = true; + +/* The range must stay allocated until the free completes. */ +run_prog (skel->progs.try_realloc,"realloc_during_free" ); +ASSERT_NULL (skel->bss->realloc_ptr,"realloc_during_free" ); + +release: + skel->bss->release =1 ; +if (thread_created) { +ASSERT_OK (pthread_join (thread,NULL ),"pthread_join" ); + thread_created = false; + completed =ASSERT_OK (ctx.err,"free_run" ) && +ASSERT_OK (ctx.retval,"free_retval" ); + }else if (skel->bss->target_tid) { + err =wait_for (&skel->bss->worker_exited); + completed =ASSERT_OK (err,"worker_exited" ); + } +ASSERT_FALSE (skel->bss->timed_out,"zap_timed_out" ); + +if (zap_seen && completed && + !run_prog (skel->progs.try_realloc,"realloc_after_free" )) { +ASSERT_EQ ((unsigned long )skel->bss->realloc_ptr, + (unsigned long )addr,"realloc_after_free" ); +ASSERT_EQ (*addr, skel->rodata->new_marker,"new_marker" ); + } +out: +arena_race__destroy (skel); +} + +/* +* Force a race between a faulting thread in userspace and a +* free operation on the arena. +*/ +static void test_fault_free_realloc (void ) +{ +struct sigaction sigbus = { .sa_handler = reader_sigbus }, old_sigbus; +struct fault_thread_ctx fault = {}; +struct arena_race *skel; +pthread_t fault_thread; +bool fault_created = false; +__u64 *addr; +__u64 expected, value; +int err, i; + + skel =setup_arena (&addr, false); +if (!skel) +return ; + + skel->bss->realloc_after_free =1 ; + +if (!ASSERT_OK (sigaction (SIGBUS , &sigbus, &old_sigbus),"sigaction_sigbus" )) + goto out_destroy; + fault.addr = addr; + err =pthread_create (&fault_thread,NULL , fault_reader_thread, &fault); +if (!ASSERT_OK (err,"pthread_create_fault" )) + goto out; + fault_created = true; + +for (i =0 ; i <1000 ; i++) { +LIBBPF_OPTS (bpf_test_run_opts, opts); + +/* Keep deferred frees from earlier iterations on separate slots. */ +if (i) { + skel->bss->page_idx = i; +if (run_prog (skel->progs.alloc_old,"alloc_next" )) + goto out; + addr = (__u64 *)(unsigned long )skel->bss->ptr; +__atomic_store_n (&fault.addr, addr, __ATOMIC_RELEASE); + } + + err =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.free_page), + &opts); +if (err) { +ASSERT_OK (err,"free_realloc" ); +break ; + } +if (opts.retval) { +ASSERT_OK (opts.retval,"free_realloc" ); +break ; + } +/* The reader may populate the slot first, or free may defer. */ +if (!skel->bss->realloc_ptr) +continue ; + value = *addr; + expected = skel->bss->current_marker; +if (value != expected) { +ASSERT_EQ (value, expected,"marker_after_realloc" ); +break ; + } + } + +WRITE_ONCE (fault.stop,1 ); +if (fault_created) { +ASSERT_OK (pthread_join (fault_thread,NULL ),"pthread_join_fault" ); + fault_created = false; + } +out: +WRITE_ONCE (fault.stop,1 ); +if (fault_created) +pthread_join (fault_thread,NULL ); +sigaction (SIGBUS , &old_sigbus,NULL ); +out_destroy: +arena_race__destroy (skel); +} + +void serial_test_arena_race (void ) +{ +cpu_set_t cpuset; +int err; + + err =sched_getaffinity (0 ,sizeof (cpuset), &cpuset); +if (!ASSERT_OK (err,"sched_getaffinity" )) +return ; +if (CPU_COUNT (&cpuset) <2 ) { +printf ("%s:SKIP: at least two runnable CPUs are required\n" , __func__); +test__skip (); +return ; + } + +if (test__start_subtest ("realloc_during_free" )) +test_realloc_during_free (false); +if (test__start_subtest ("realloc_during_deferred_free" )) +test_realloc_during_free (true); +if (test__start_subtest ("fault_free_realloc" )) +test_fault_free_realloc (); +}
tools/testing/selftests/bpf/progs/arena_race.c new file +166 -0
166 lines not on this page · raw message
Signature · 1 line
2.52.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Mon, 19 Oct 2026 20:11:05 +0000 you wrote:
Arena memory is tracked in two places: the range tree, which records which addresses are allocated, and the kernel and user page tables. The two are updated under different locks, and that leaves two races:
- Since
b8467290edab("bpf: arena: make arena kfuncs any context safe"), a free releases its range before the kernel TLB flush and the user zap complete. A concurrent allocation can then write through stale translations into pages that are about to be freed.[...]
Here is the summary with links:
- [bpf-next,v6,1/3] bpf: Revalidate faulted arena pages before installing user mappings
https://git.kernel.org/bpf/bpf-next/c/05bee0638bac - [bpf-next,v6,2/3] bpf: Fix arena race between page free and alloc leading to incoherency https://git.kernel.org/bpf/bpf-next/c/7244a2e58d21 - [bpf-next,v6,3/3] selftests/bpf: Add arena allocation race tests https://git.kernel.org/bpf/bpf-next/c/6c372031078c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html