Skip to content

bpf: Fix arena memory incoherence

Emil Tsalapatis bpf 6 versions – Applied

Status from this message: This series was applied to bpf/bpf-next.git (master)

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v6 Latest3 patches 0 repliesApplied 3/3
  2. v55 patches 2 repliesSuperseded
  3. v48 patches 14 repliesSuperseded
Earlier versions · 3
VersionPostedPatchesRepliesStatus
v36 patches3 repliesSuperseded
v27 patches0 repliesSuperseded
v15 patches2 repliesSuperseded

Patches in v6 All 3 applied

#Subject and filesSizeRepliesApplied
1/3bpf: Revalidate faulted arena pages before installing user mappingskernel/bpf/arena.c+16 -20 repliesApplied
2/3bpf: Fix arena race between page free and alloc leading to incoherencykernel/bpf/arena.c+73 -860 repliesApplied
3/3selftests/bpf: Add arena allocation race teststools/testing/selftests/bpf/prog_tests/arena_race.c tools/testing/selftests/bpf/progs/arena_race.c+486 -00 repliesApplied

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v6 yet.

Read the v6 thread

Cover letter [PATCH bpf-next v6 0/3] bpf: Fix arena memory incoherence

Arena memory is tracked in two places: the range tree, which records which addresses are allocated, and the kernel and user page tables. The two are updated under different locks, and that leaves two races:

- Since b8467290edab ("bpf: arena: make arena kfuncs any context safe"), a free releases its range before the kernel TLB flush and the user zap complete. A concurrent allocation can then write through stale translations into pages that are about to be freed.

- A user fault takes a reference to the page and installs the user PTE after dropping the arena lock. A free that runs in between leaves user space mapping a page that the arena has already released.

In both cases BPF programs and user space can see different pages at the same arena address.

- A free keeps its range allocated until the TLB flush and the user zap are complete, and only then releases it with range_tree_set(). User faults on the range get SIGBUS until then. Frees serialize on zap_mutex.

- A user fault pins the page, locks its folio, and revalidates the kernel mapping before installing the user PTE, the way filemap_fault() synchronizes with truncate. The free waits for the folio lock of every page it removed before it zaps user mappings.

Patches 1 and 2 fix the two races. Patch 3 adds selftests.

v5 -> v6 (https://lore.kernel.org/bpf/[email protected]/)

- Return the locked page from arena_vm_fault() with VM_FAULT_LOCKED instead of installing the user PTE there with finish_fault(). The fault checks PageHWPoison() before it returns the page. (Alexei)
- Give faults on a range that is being freed the same SIGBUS as faults on a reserved range. Drop the record of the range being freed and the fault-side wait. (Alexei)
- Drop the trylock and deferral scheme. Frees serialize on zap_mutex and zap user mappings with zap_pages(), which takes mmap_read_lock() of each mapping process as in mainline. Keep the single-page zap shortcut. Sleepable frees are deferred only when the arena spinlock cannot be taken, and the worker retries such a free in place instead of requeueing itself. (Alexei)
- Drop v5 patch 1. The worker no longer requeues itself, so the flush_work() in arena_map_free() waits for all of its work. (Alexei)
- Drop v5 patch 4, which retried faults when the arena lock timed out. (Alexei)
- Drop the immediate release of ranges without kernel PTEs; they go through the regular free path.
- Selftests: map the page into user space before the paused free, and let the fault_free_realloc reader skip reads that raise SIGBUS.

v4 -> v5 (https://lore.kernel.org/bpf/[email protected]/)

- Drop the third range-tree state and the is_range_tree_set() change (v4 patches 4 and 5). A free keeps its range allocated until the TLB flush and the user zap complete, then releases it with range_tree_set(). range_tree_clear() only allocates when the whole range is free, and then there is nothing to free. (Alexei)
- Drop the per-arena mapping, page->mapping, the mapping-tree zap and the new bpf_map_ops callback (v4 patches 1 and 2). (Alexei)
- Replace opref (v4 patch 7) with the folio lock. A fault rechecks the kernel PTE with the folio locked and installs the user PTE itself with finish_fault(), and a free waits for the folio lock of each page it removed before zapping (new patch 2). (Alexei)
- Give faults on a reserved range the scratch_page error. Faults on the range being freed sleep until the free releases it, instead of spinning on VM_FAULT_RETRY. (Alexei)
- Trylock arena->lock and the mmap read lock of every process that maps the arena before retiring a range, and defer the free if one is contended. The worker waits for the contended lock instead of requeueing itself. The deferred span, allocated with __GFP_RETRY_MAYFAIL when the caller can sleep, is the only allocation before the ptes are cleared. (Alexei)
- Release a range without kernel ptes, such as a reserved one, right away without arena->lock or mmap_lock, so freeing and reallocating a reserved range is never deferred in sleepable context.
- Retry faults when the arena lock times out, keeping SIGBUS for -EDEADLK (new patch 4).
- Selftests: rename the subtests, and pause frees in zap_vma_range(), which arm64 can trace, instead of flush_tlb_kernel_range().

v3 -> v4 (https://lore.kernel.org/bpf/[email protected]/)

- Add per-arena anonymous mappings and associate pages with them
- Return VM_FAULT_LOCKED and let generic MM install user PTEs; wait for locked fault pages before TLB flushing and zapping (Alexei).
- Add mutual exclusion between arena free and fault operations to prevent incoherency.
- Return VM_FAULT_NOPAGE for non-retryable GUP faults when progress is impossible.
- Stop and drain self-requeueing deferred frees before map teardown.
- Track ownership of unavailable ranges independently of free progress and retry worker reservations on -ENOMEM without starting teardown.
- Make arena free span state machine explicit.

v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/)

- Rebase on bpf-next and resolve conflicts
- Fold in the atomic mark_available range tree operation into initial fix patches

v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/)

- Use __range_iter_next() in is_range_tree_set() (bot-ci)
- Move -EAGAIN handling to the patch where the errno is introduced (bot-ci)
- Add refactoring patch for (Puranjay)
- Avoid leaking unavailable range tree nodes with atomic mark_available operation()
- Avoid retries on res spinlock locking for -EDEADLK (Puranjay)

Emil Tsalapatis (3):
bpf: Revalidate faulted arena pages before installing user mappings
bpf: Fix arena race between page free and alloc leading to incoherency selftests/bpf: Add arena allocation race tests

 kernel/bpf/arena.c                            | 177 +++++-----
 .../selftests/bpf/prog_tests/arena_race.c     | 320 ++++++++++++++++++
 .../testing/selftests/bpf/progs/arena_race.c  | 166 +++++++++
 3 files changed, 575 insertions(+), 88 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_race.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_race.c
Signature · 1 line
2.52.0