bpf: Overwrite scratch PTE when allocating arena pages
Subsystems (2)
Tejun Heo
reviewerPATCH 1/1 bpf: Overwrite scratch PTE when allocating arena pages
apply_range_set_cb() maps the pages for a new arena allocation. It returned -EBUSY when the target PTE was already populated.
Kernel-fault recovery can leave the per-arena scratch page in an otherwise unallocated arena PTE: a BPF program that touched an unallocated address gets the scratch page installed there. A later bpf_arena_alloc_pages() covering that page then finds the PTE populated, returns -EBUSY, and leaves the scratch page in place. Every subsequent allocation of that page fails the same way.
Drop the must-be-empty check so set_pte_at() installs the real page unconditionally, overwriting a scratch (or empty) PTE.
--- kernel/bpf/arena.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/bpf/arena.c +1 -3
@@ -131,15 +131,13 @@ static int apply_range_set_cb(pte_t *pteif (!data)return 0 ; -/* sanity check */ -if (unlikely (!pte_none (ptep_get (pte)))) -return -EBUSY ; page = d->pages[d->i];/* paranoia, similar to vmap_pages_pte_range() */ if (WARN_ON_ONCE (!pfn_valid (page_to_pfn (page))))return -EINVAL ; +/* May be none or the scratch page, overwrite either way */ set_pte_at (&init_mm, addr, pte,mk_pte (page,PAGE_KERNEL )); d->i++;return 0 ;
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Overwriting a present PTE (the scratch page) with a new page via set_pte_at() without first…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Overwriting a present PTE (the scratch page) with a new page via set_pte_at() without first clearing it and flushing the TLB violates Break-Before-Make (BBM) architecture rules and causes silent data corruption via stale TLB translations. --
commit f66bc91b18a1403f8777f0486173da0584f46ca3
Author: Tejun Heo <[email protected]>
bpf: Overwrite scratch PTE when allocating arena pages
apply_range_set_cb() previously returned -EBUSY when encountering a populated PTE. This patch removes that check to allow unconditionally overwriting the scratch page left behind by kernel-fault recovery during new arena allocations.
Quoted 9 lines
kernel/bpf/arena.c +0
-1@@ -157,15 +157,13 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)if (!data)return 0 ; -/* sanity check */
kernel/bpf/arena.c static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) · line 167
Quoted 10 lines
kernel/bpf/arena.c +1
-2-if (unlikely (!pte_none (ptep_get (pte)))) -return -EBUSY ; page = d->pages[d->i];/* paranoia, similar to vmap_pages_pte_range() */ if (WARN_ON_ONCE (!pfn_valid (page_to_pfn (page))))return -EINVAL ; +/* May be none or the scratch page, overwrite either way */ set_pte_at (&init_mm, addr, pte,mk_pte (page,PAGE_KERNEL ));
[Severity: Critical]
Does unconditionally overwriting a present PTE with set_pte_at() without first clearing it violate Break-Before-Make architecture rules?
On architectures like arm64, overwriting a valid, present PTE without a sequence like ptep_get_and_clear() can trigger hardware TLB conflict aborts or panics.
Additionally, since this changes the PFN of a present PTE (the scratch page), doesn't this require a TLB flush?
Looking at the caller arena_alloc_pages(), it appears to only invoke flush_vmap_cache() after the apply_to_page_range() loop:
ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32 + (mapped << PAGE_SHIFT), this_batch << PAGE_SHIFT, apply_range_set_cb, &data); ... flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
Since flush_vmap_cache() flushes data caches and not the TLB, could CPUs retain stale TLB mappings to the scratch page? If so, subsequent accesses might hit the shared scratch page instead of the newly allocated page, bypassing memory isolation.
Quoted 3 lines
kernel/bpf/arena.c +0
-0d->i++;return 0 ;}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewWhen set_pte_at() overwrites a present PTE (the scratch page) with a different PFN (the new real page), does the arena allocation path flush the TLB?
Quoted 9 lines
kernel/bpf/arena.c +0
-1@@ -157,15 +157,13 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)if (!data)return 0 ; -/* sanity check */
kernel/bpf/arena.c static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) · line 167
Quoted 10 lines
kernel/bpf/arena.c +1
-2-if (unlikely (!pte_none (ptep_get (pte)))) -return -EBUSY ; page = d->pages[d->i];/* paranoia, similar to vmap_pages_pte_range() */ if (WARN_ON_ONCE (!pfn_valid (page_to_pfn (page))))return -EINVAL ; +/* May be none or the scratch page, overwrite either way */ set_pte_at (&init_mm, addr, pte,mk_pte (page,PAGE_KERNEL ));
^^^^^^^^^^
When set_pte_at() overwrites a present PTE (the scratch page) with a different PFN (the new real page), does the arena allocation path flush the TLB?
The caller arena_alloc_pages() only calls flush_vmap_cache() after apply_to_page_range(), which expands to flush_cache_vmap() (a no-op on x86). This flushes data caches but not TLB entries.
Before this patch, apply_range_set_cb() only transitioned not-present to present, which requires no TLB flush. But now it can transition present(scratch) to present(real), changing the underlying PFN.
When a BPF program faults on an unallocated arena address, the kernel-fault recovery path installs the scratch page into that PTE:
bpf_arena_handle_page_fault()
-> apply_range_set_scratch_cb()
-> ptep_try_set()
The faulting CPU then retries the access, loading a TLB entry that maps the arena kern_vm address to the scratch_page PFN. This entry becomes stale when a later arena_alloc_pages() overwrites the PTE to point to a real page.
The arena kern_vm region is mapped in init_mm and shared across all CPUs. Without a TLB flush, can a CPU holding a stale entry continue reading/writing the shared scratch page instead of the new real page?
The free path in arena_free_pages() and arena_free_worker() already calls flush_tlb_kernel_range() after clearing arena PTEs:
arena_free_pages():
ptep_get_and_clear(...)
flush_tlb_kernel_range(start, end);arena_free_worker():
apply_to_page_range(...)
flush_tlb_kernel_range(...)Does the alloc path need a similar flush when overwriting a present PTE?
kernel/bpf/arena.c static int apply_range_set_cb(pte_t *pte · line 143
Quoted 2 lines
kernel/bpf/arena.c +0
-0d->i++;return 0 ;
---
Quoted 15 lines
commit
066af9e73183b42ad5ee27805163ce25053ed9bf
Author: Tejun Heo <[email protected]>bpf: Overwrite scratch PTE when allocating arena pages
apply_range_set_cb()maps the pages for a new arena allocation. It returned-EBUSYwhen the target PTE was already populated.Kernel-fault recovery can leave the per-arena scratch page in an otherwise unallocated arena PTE: a BPF program that touched an unallocated address gets the scratch page installed there. A later
bpf_arena_alloc_pages()covering that page then finds the PTE populated, returns-EBUSY, and leaves the scratch page in place. Every subsequent allocation of that page fails the same way.
Quoted 6 lines
Drop the must-be-empty check so
set_pte_at()installs the real page unconditionally, overwriting a scratch (or empty) PTE.Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>Cc David Hildenbrand <[email protected]>
This looks like a bug fix for the scratch page allocation failure. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26718992256