bpf/arena: Direct kernel-side access
Subsystems (10)
Tejun Heo
reviewerPATCH v4 1/1 bpf/arena: Direct kernel-side access
Hello,
This makes BPF arena memory directly dereferenceable from kernel code (struct_ops callbacks, kfuncs). Each arena gets a per-arena scratch page that an arch fault hook installs into empty PTEs on kernel-side faults, after KFENCE. The faulting instruction retries and the violation is reported through the program's BPF stream.
v4:
- Patch 1: note that the strict-zero cmpxchg is narrower than pte_none() in inline comments on both x86 and arm64. (Andrea)
- Patch 2: stub bpf_arena_handle_page_fault() for !CONFIG_BPF_SYSCALL via a new include/linux/bpf_defs.h. (lkp)
- Patch 7: scx_arena_alloc() retries via a loop instead of a single retry on pool growth. (Andrea)
- Picked up Reviewed-by tags from Emil and Andrea.
v3: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v1 (RFC): https://lore.kernel.org/r/[email protected]
Motivation
----------
sched_ext's ops_cid.set_cmask() hands the BPF scheduler a struct scx_cmask *. The kernel translates a kernel cpumask to a cmask, but it had no way to write into the arena, so the cmask lived in kernel memory and was passed as a trusted pointer. BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.
The shape isn't unique to set_cmask. Sub-scheduler support is on the way and more sched_ext callbacks will want to pass structured data to BPF. Anywhere a kfunc or struct_ops callback wants to hand a struct to a BPF program, arena residence is the natural answer.
Approach
--------
Each arena gets a per-arena scratch page. Arenas stay sparsely mapped as today - PTEs are populated only for allocated pages. A new arch fault hook (bpf_arena_handle_page_fault) is wired into x86 page_fault_oops() and arm64 __do_kernel_fault(), after KFENCE. When a kernel-side access faults inside an arena's kern_vm range, the helper walks the stack to find the BPF program responsible, range-checks the fault address against prog->aux->arena, and atomically installs the scratch page into the empty PTE via the new ptep_try_set() wrapper. The kernel instruction retries and reads/writes the scratch page. Free paths and map destruction treat scratch as non-owned. Real allocation refuses to overwrite scratch (apply_range_set_cb returns -EBUSY). A scratched address stays dead until map destroy, since its presence means the BPF program has already malfunctioned.
The mechanism is default behavior - no UAPI flag.
What this preserves
-------------------
All the debugging properties of today's sparse-PTE design are preserved:
* BPF programs still fault on unmapped arena accesses. The fault semantics (instruction retry with rdst = 0) and the violation report through bpf_streams are unchanged for prog-side accesses.
* The first kernel-side touch of an unmapped address is reported via bpf_streams the same way as a prog-side fault, with the stack walk attributing it to the originating prog.
* User-side fault on a never-scratched address still lazy-allocates a real page (or returns SIGSEGV under BPF_F_SEGV_ON_FAULT). User-side fault on a scratched address SIGSEGVs.
What changes for the kernel-side caller is just that an unmapped deref no longer oopses - it retries through the scratch page and emits a violation report. The same shape today's BPF instruction faults have.
Patches 1-2 (atomic PTE install + arena scratch-page recovery)
--------------------------------------------------------------
mm: Add ptep_try_set() for lockless empty-slot installs
bpf: Recover arena kernel faults with scratch page
Patches 3-5 (helpers used by struct_ops registration)
-----------------------------------------------------
bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers
bpf: Add bpf_struct_ops_for_each_prog()
bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena()
Patches 6-8 (sched_ext: arena auto-discovery, allocator, set_cmask)
-------------------------------------------------------------------
sched_ext: Require an arena for cid-form schedulers
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Convert ops.set_cmask() to arena-resident cmask
Patch 6 reads each member prog's prog->aux->arena via bpf_prog_arena() and requires the cid-form struct_ops to reference exactly one arena. Patch 7 builds a gen_pool sub-allocator inside that arena. Patch 8 converts set_cmask() to write into arena memory. BPF dereferences via __arena like any other arena struct, no probe-reads.
Base
----
sched_ext/for-7.2 (f31e89a8f583)
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git arena-direct-v4
Documentation/bpf/kfuncs.rst | 14 +++ arch/arm64/include/asm/pgtable.h | 12 ++ arch/arm64/mm/fault.c | 10 +- arch/x86/include/asm/pgtable.h | 12 ++ arch/x86/mm/fault.c | 12 +- include/linux/bpf.h | 14 +++ include/linux/bpf_defs.h | 19 +++ include/linux/pgtable.h | 25 ++++ kernel/bpf/arena.c | 216 +++++++++++++++++++++++++++------- kernel/bpf/bpf_struct_ops.c | 36 ++++++ kernel/bpf/core.c | 5 + kernel/sched/build_policy.c | 4 + kernel/sched/ext.c | 135 ++++++++++++++++++++- kernel/sched/ext_arena.c | 126 ++++++++++++++++++++ kernel/sched/ext_arena.h | 18 +++ kernel/sched/ext_cid.c | 20 +--- kernel/sched/ext_internal.h | 23 +++- tools/sched_ext/include/scx/cid.bpf.h | 52 -------- tools/sched_ext/scx_qmap.bpf.c | 5 +- 19 files changed, 630 insertions(+), 128 deletions(-)
Thanks.
--
tejun
Tejun Heo
reviewerPATCH 1/8 mm: Add ptep_try_set() for lockless empty-slot installs
Add ptep_try_set(ptep, new_pte): atomically set *ptep to new_pte iff it is currently pte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.
The intended caller is the upcoming bpf_arena kernel-side fault recovery path. The install runs from a page fault that can be nested under locks held by the faulting kernel caller (e.g. a BPF program holding raw_res_spin_lock_irqsave on its arena's spinlock), so trylock-and-retry would A-A deadlock. Lock-free cmpxchg is the only viable option, which constrains this helper to special kernel page tables where concurrent writers cooperate via atomic accessors.
The generic version in <linux/pgtable.h> returns false. x86 and arm64 override with try_cmpxchg-based implementations on the underlying pteval. Other architectures get the false stub - the callers there already fall through to oops.
v2: Rename to ptep_try_set(). Tighten kerneldoc. (David, Alexei)
v3: Note that strict-zero cmpxchg is narrower than pte_none(). (Andrea)
--- arch/arm64/include/asm/pgtable.h | 12 ++++++++++++ arch/x86/include/asm/pgtable.h | 12 ++++++++++++ include/linux/pgtable.h | 25 +++++++++++++++++++++++++ 3 files changed, 49 insertions(+)
arch/arm64/include/asm/pgtable.h +12 -0
@@ -1830,6 +1830,18 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,return __ptep_get_and_clear (mm, addr, ptep); } +/* +* Note: strictly-zero compare is narrower than pte_none(), but the gap is +* harmless: a fresh kernel PTE has no software bits set. +*/ +static inline bool ptep_try_set (pte_t *ptep,pte_t new_pte) +{ +pteval_t old =0 ; + +return try_cmpxchg (&pte_val (*ptep), &old,pte_val (new_pte)); +} +#define ptep_try_set ptep_try_set +#define test_and_clear_young_ptes test_and_clear_young_ptesstatic inline bool test_and_clear_young_ptes (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,unsigned int nr)
arch/x86/include/asm/pgtable.h +12 -0
@@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, }while (!try_cmpxchg ((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte)); } +/* +* Note: strictly-zero compare is narrower than pte_none(), but the gap is +* harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs. +*/ +static inline bool ptep_try_set (pte_t *ptep,pte_t new_pte) +{ +pte_t old_pte =__pte (0 ); + +return try_cmpxchg ((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte); +} +#define ptep_try_set ptep_try_set +#define flush_tlb_fix_spurious_fault (vma, address, ptep) do { } while (0)#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS
include/linux/pgtable.h +25 -0
@@ -1036,6 +1036,31 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres }#endif +#ifndef ptep_try_set +/** +* ptep_try_set - atomically set an empty kernel PTE +* @ptep: page table entry +* @new_pte: value to install +* +* Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on +* success, false if the slot was already populated or the arch has no +* implementation. +* +* For special kernel page tables only - never user page tables. The caller must +* prevent concurrent teardown of @ptep and must accept that other writers may +* race. Concurrent clearers must use ptep_get_and_clear() so racing accesses +* agree on the outcome. +* +* Architectures opt in by providing a cmpxchg-based override and defining +* ptep_try_set as an identity macro. The generic stub returns false, which is +* correct for callers that fall through to oops on failure. +*/ +static inline bool ptep_try_set (pte_t *ptep,pte_t new_pte) +{ +return false; +} +#endif +#ifndef wrprotect_ptes/** * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same
Signature · 1 line
2.54.0
Re: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Quoted 12 lines
include/linux/pgtable.h +10
-0+#ifndef ptep_try_set +/** +* ptep_try_set - atomically set an empty kernel PTE +* @ptep: page table entry +* @new_pte: value to install +* +* Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on +* success, false if the slot was already populated or the arch has no +* implementation. +*
include/linux/pgtable.h static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · line 1058
Quoted 10 lines
include/linux/pgtable.h +10
-0+* For special kernel page tables only - never user page tables. The caller must +* prevent concurrent teardown of @ptep and must accept that other writers may +* race. Concurrent clearers must use ptep_get_and_clear() so racing accesses +* agree on the outcome. +* +* Architectures opt in by providing a cmpxchg-based override and defining +* ptep_try_set as an identity macro. The generic stub returns false, which is +* correct for callers that fall through to oops on failure. +*/ +static inline bool ptep_try_set (pte_t *ptep,pte_t new_pte)
There are only a handful of valid use cases for this (and the "fall through to oops on failure") makes this very clear.
So I'm fine with this, assuming there won't really be a lot of other users besides ebpf.
Signature · 3 lines
Cheers, David
Re: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
In reply to Tejun HeoThis series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 12 lines · On Fri, 22 May 2026 07:22:12 -1000 you wrote:
Add
ptep_try_set(ptep, new_pte): atomically set *ptep tonew_pteiff it is currentlypte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.The intended caller is the upcoming
bpf_arenakernel-side fault recovery path. The install runs from a page fault that can be nested under locks held by the faulting kernel caller (e.g. a BPF program holdingraw_res_spin_lock_irqsaveon its arena's spinlock), so trylock-and-retry would A-A deadlock. Lock-free cmpxchg is the only viable option, which constrains this helper to special kernel page tables where concurrent writers cooperate via atomic accessors.
Quoted 1 line
[...]
Here is the summary with links:
- [1/8] mm: Add ptep_try_set() for lockless empty-slot installs
https://git.kernel.org/bpf/bpf-next/c/258df8fce42f - [2/8] bpf: Recover arena kernel faults with scratch page (no matching commit) - [3/8] bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers https://git.kernel.org/bpf/bpf-next/c/f211c81ddc36 - [4/8] bpf: Add bpf_struct_ops_for_each_prog() https://git.kernel.org/bpf/bpf-next/c/7c48a28c1bbe - [5/8] bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena() https://git.kernel.org/bpf/bpf-next/c/53cc12a2dc88
- [6/8] sched_ext: Require an arena for cid-form schedulers (no matching commit)
- [7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages (no matching commit)
- [8/8] sched_ext: Convert ops.set_cmask() to arena-resident cmask (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Will Deacon
maintainerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
In reply to Tejun HeoQuoted 36 lines · Tejun Heo
Add
ptep_try_set(ptep, new_pte): atomically set *ptep tonew_pteiff it is currentlypte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.The intended caller is the upcoming
bpf_arenakernel-side fault recovery path. The install runs from a page fault that can be nested under locks held by the faulting kernel caller (e.g. a BPF program holdingraw_res_spin_lock_irqsaveon its arena's spinlock), so trylock-and-retry would A-A deadlock. Lock-free cmpxchg is the only viable option, which constrains this helper to special kernel page tables where concurrent writers cooperate via atomic accessors.The generic version in <
linux/pgtable.h> returns false. x86 and arm64 override with try_cmpxchg-based implementations on the underlying pteval. Other architectures get the false stub - the callers there already fall through to oops.v2: Rename to
ptep_try_set(). Tighten kerneldoc. (David, Alexei)
v3: Note that strict-zero cmpxchg is narrower thanpte_none(). (Andrea)Suggested-by Kumar Kartikeya Dwivedi <[email protected]>Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>Reviewed-by Andrea Righi <[email protected]>Cc David Hildenbrand <[email protected]>--- arch/arm64/include/asm/pgtable.h | 12 ++++++++++++ arch/x86/include/asm/pgtable.h | 12 ++++++++++++ include/linux/pgtable.h | 25 +++++++++++++++++++++++++ 3 files changed, 49 insertions(+)arch/arm64/include/asm/pgtable.h +0
-0
arch/arm64/include/asm/pgtable.h static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · line 1836
Quoted 7 lines
arch/arm64/include/asm/pgtable.h +4
-0@@ -1830,6 +1830,18 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,return __ptep_get_and_clear (mm, addr, ptep); } +/* +* Note: strictly-zero compare is narrower than pte_none(), but the gap is +* harmless: a fresh kernel PTE has no software bits set. +*/
This comment really confused me :/
What is a "fresh" kernel PTE and why do you specifically call out "software bits" if the CAS requires all 64 bits to be 0? Why is that narrower than pte_none() given that pte_none() for arm64 is:
#define pte_none(pte) (!pte_val(pte))
Will
Tejun Heo
reviewerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Hello,
Quoted 5 lines · Will Deacon
Quoted 4 lines
+/*
+ * Note: strictly-zero compare is narrower thanpte_none(), but the gap is
+ * harmless: a fresh kernel PTE has no software bits set.
+ */
Quoted 7 lines
This comment really confused me :/
What is a "fresh" kernel PTE and why do you specifically call out "software bits" if the CAS requires all 64 bits to be 0? Why is that narrower than
pte_none()given thatpte_none()for arm64 is:#define
pte_none(pte)(!pte_val(pte))
Yeah, that's complete non-sense for arm. The comment is about x86's pte_none() excluding DIRTY and ACCESSED due to an erratum when testing none and how that doesn't matter here. This shouldn't have been copied to arm. I'll send a patch to remove that.
Thanks.
Signature · 1 line
tejun
David Hildenbrand (Arm)
maintainerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Quoted 14 lines · Tejun Heo
Hello,
Quoted 11 lines · Will Deacon
Quoted 4 lines
+/*
+ * Note: strictly-zero compare is narrower thanpte_none(), but the gap is
+ * harmless: a fresh kernel PTE has no software bits set.
+ */This comment really confused me :/
What is a "fresh" kernel PTE and why do you specifically call out "software bits" if the CAS requires all 64 bits to be 0? Why is that narrower than
pte_none()given thatpte_none()for arm64 is:
Quoted 6 lines
Quoted 1 line
#define
pte_none(pte)(!pte_val(pte))Yeah, that's complete non-sense for arm. The comment is about x86's
pte_none()excluding DIRTY and ACCESSED due to an erratum when testing none and how that doesn't matter here. This shouldn't have been copied to arm. I'll send a patch to remove that.
Is BPF maybe picking up patches from other subsystems up too early without waiting for acks?
Signature · 3 lines
Cheers, David
Tejun Heo
reviewerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Hello, David.
Quoted 2 lines
Is BPF maybe picking up patches from other subsystems up too early without waiting for acks?
Do you mean the comment cleanup I just sent, or the original ptep_try_set() merge?
The original went through several rounds, and you acked the patch that introduced both problems (258df8fce42f) before it landed.
The barriers fix came from Catalin, he raised the issue and reviewed the fix. The comment cleanup got picked up quickly. Maybe that's the bpf folks wanting it in ahead of the merge window.
Thanks.
--
tejun
David Hildenbrand (Arm)
maintainerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Quoted 7 lines · Tejun Heo
Hello, David.
Quoted 2 lines
Is BPF maybe picking up patches from other subsystems up too early without waiting for acks?
Do you mean the comment cleanup I just sent, or the original
ptep_try_set()merge?
I am talking about any patch that touches arch code without an ACK from the maintainer.
Maybe arm64 and x86 people are fine with that, but I heard other voices recently, thus my question.
Quoted 3 lines
The original went through several rounds, and you acked the patch that introduced both problems (
258df8fce42f) before it landed.
I am not an arm64 maintainer although working for the company :)
Signature · 3 lines
Cheers, David
Tejun Heo
reviewerRe: [PATCH 1/8] mm: Add ptep_try_set() for lockless empty-slot installs
Hello,
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 5 lines · Tejun Heo
Hello, David.
Quoted 2 lines
Is BPF maybe picking up patches from other subsystems up too early without waiting for acks?
Quoted 8 lines
Quoted 2 lines
Do you mean the comment cleanup I just sent, or the original
ptep_try_set()merge?I am talking about any patch that touches arch code without an ACK from the maintainer.
Maybe arm64 and x86 people are fine with that, but I heard other voices recently, thus my question.
I see. I don't know the dynamics here. I'll leave it for bpf folks.
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerPATCH 2/8 bpf: Recover arena kernel faults with scratch page
In reply to Tejun HeoFrom: Kumar Kartikeya Dwivedi <[email protected]>
BPF arena usage is becoming more prevalent, but kernel <-> BPF communication over arena memory is awkward today. Data has to be staged through a trusted kernel pointer with extra code and copying on the BPF side. While reads through arena pointers can use a fault-safe helper, writes don't have a good solution. The in-line alternative would need instruction emulation or asm fixup labels.
Enable direct kernel-side reads and writes within GUARD_SZ / 2 of any handed-in arena pointer, without bounds checking. A per-arena scratch page is installed by the arch fault path into empty arena kernel PTEs - x86 from page_fault_oops() for not-present faults, arm64 from __do_kernel_fault() for translation faults, both after the existing exception-table and KFENCE handling. The faulting instruction retries and the access is also reported through the program's BPF stream, preserving error reporting.
bpf_prog_find_from_stack() resolves the current BPF program (and its arena) from the kernel stack - no new bpf_run_ctx state is added. Recovery covers the 4 GiB arena plus the upper half-guard (GUARD_SZ / 2). The lower half-guard is excluded because well-behaved kfuncs only access forward from arena pointers. The kfunc-author contract - access at most GUARD_SZ / 2 past a handed-in pointer - is documented in Documentation/bpf/kfuncs.rst.
The install is lock-free via ptep_try_set(). On race-loss the winning installer's PTE is already valid, so the access retry succeeds. The arena clear path uses ptep_get_and_clear() so installer and clearer race through atomic accessors. No flush_tlb_kernel_range() afterwards. Stale "not mapped" entries just cause one extra re-fault, cheaper than a global IPI on every install.
Scratch exists only to keep the kernel from oopsing on an in-line arena access. Its presence at a PTE means the BPF program has already malfunctioned, and the violation is reported through the program's BPF stream. The only requirement for behavior on a scratched PTE is that the kernel doesn't crash. In particular, any user-side access through such a PTE may segfault. The shared scratch page is freed once during map destruction.
BPF instruction faults continue to use the existing JIT exception-table path. This patch changes only the kernel-text fault path. No UAPI flag is added. The new behavior is the default.
v2: Use ptep_get_and_clear() in apply_range_clear_cb(). (David)
v3: Stub bpf_arena_handle_page_fault() for !CONFIG_BPF_SYSCALL. (lkp)
--- Documentation/bpf/kfuncs.rst | 14 +++ arch/arm64/mm/fault.c | 10 +- arch/x86/mm/fault.c | 12 ++- include/linux/bpf.h | 1 + include/linux/bpf_defs.h | 19 ++++ kernel/bpf/arena.c | 177 +++++++++++++++++++++++++++-------- kernel/bpf/core.c | 5 + 7 files changed, 191 insertions(+), 47 deletions(-) create mode 100644 include/linux/bpf_defs.h
Documentation/bpf/kfuncs.rst +14 -0
@@ -462,6 +462,20 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +2.8 Accessing arena memory through kfunc arguments +-------------------------------------------------- + +A read or write at any address inside an arena does not oops the kernel. +Unallocated arena pages are lazily backed by a scratch page and the +access is reported through the program's BPF stream as an error. Only +the BPF program's correctness is affected; the kernel itself remains +intact. + +The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that +is also covered by this recovery. A kfunc handed an arena pointer may +therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking +against the arena. Larger accesses must verify the range explicitly. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations
arch/arm64/mm/fault.c +7 -3
@@ -9,6 +9,7 @@#include <linux/acpi.h> #include <linux/bitfield.h> +#include <linux/bpf_defs.h> #include <linux/extable.h> #include <linux/kfence.h> #include <linux/signal.h>
@@ -416,9 +417,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, }else if (addr <PAGE_SIZE ) { msg ="NULL pointer dereference" ; }else { -if (esr_fsc_is_translation_fault (esr) && -kfence_handle_page_fault (addr, esr & ESR_ELx_WNR, regs)) -return ; +if (esr_fsc_is_translation_fault (esr)) { +if (kfence_handle_page_fault (addr, esr & ESR_ELx_WNR, regs)) +return ; +if (bpf_arena_handle_page_fault (addr, esr & ESR_ELx_WNR, regs->pc)) +return ; + } msg ="paging request" ; }
arch/x86/mm/fault.c +8 -4
@@ -8,6 +8,7 @@#include <linux/sched/task_stack.h> /* task_stack_*(), ... */ #include <linux/kdebug.h> /* oops_begin/end, ... */ #include <linux/memblock.h> /* max_low_pfn */ +#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */ #include <linux/kfence.h> /* kfence_handle_page_fault */ #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ #include <linux/mmiotrace.h> /* kmmio_handler, ... */
@@ -688,10 +689,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code,if (IS_ENABLED (CONFIG_EFI ))efi_crash_gracefully_on_page_fault (address); -/* Only not-present faults should be handled by KFENCE. */ -if (!(error_code &X86_PF_PROT ) && -kfence_handle_page_fault (address, error_code &X86_PF_WRITE , regs)) -return ; +/* Only not-present faults should be handled by KFENCE or BPF arena. */ +if (!(error_code &X86_PF_PROT )) { +if (kfence_handle_page_fault (address, error_code &X86_PF_WRITE , regs)) +return ; +if (bpf_arena_handle_page_fault (address, error_code &X86_PF_WRITE , regs->ip)) +return ; + } oops:/*
include/linux/bpf.h +1 -0
@@ -6,6 +6,7 @@#include <uapi/linux/bpf.h> #include <uapi/linux/filter.h> +#include <linux/bpf_defs.h> #include <crypto/sha2.h> #include <linux/workqueue.h>
include/linux/bpf_defs.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Subset of bpf.h declarations, split out so files that need only these +* declarations can avoid bpf.h's full include cost. +*/ +#ifndef _LINUX_BPF_DEFS_H +#define _LINUX_BPF_DEFS_H + +#ifdef CONFIG_BPF_SYSCALL +bool bpf_arena_handle_page_fault (unsigned long addr,bool is_write,unsigned long fault_ip); +#else +static inline bool bpf_arena_handle_page_fault (unsigned long addr,bool is_write, +unsigned long fault_ip) +{ +return false; +} +#endif + +#endif /* _LINUX_BPF_DEFS_H */
kernel/bpf/arena.c +137 -40
326 lines not on this page · raw message
kernel/bpf/core.c +5 -0
@@ -3290,6 +3290,11 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) {return 0 ; } +__weak bool bpf_arena_handle_page_fault (unsigned long addr,bool is_write, +unsigned long fault_ip) +{ +return false; +}#ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init (void )
Signature · 1 line
2.54.0
David Hildenbrand (Arm)
maintainerRe: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Quoted 206 lines · Tejun Heo
From: Kumar Kartikeya Dwivedi <[email protected]>
BPF arena usage is becoming more prevalent, but kernel <-> BPF communication over arena memory is awkward today. Data has to be staged through a trusted kernel pointer with extra code and copying on the BPF side. While reads through arena pointers can use a fault-safe helper, writes don't have a good solution. The in-line alternative would need instruction emulation or asm fixup labels.
Enable direct kernel-side reads and writes within
GUARD_SZ/ 2 of any handed-in arena pointer, without bounds checking. A per-arena scratch page is installed by the arch fault path into empty arena kernel PTEs - x86 frompage_fault_oops()for not-present faults, arm64 from__do_kernel_fault()for translation faults, both after the existing exception-table and KFENCE handling. The faulting instruction retries and the access is also reported through the program's BPF stream, preserving error reporting.
bpf_prog_find_from_stack()resolves the current BPF program (and its arena) from the kernel stack - no newbpf_run_ctxstate is added. Recovery covers the 4 GiB arena plus the upper half-guard (GUARD_SZ/ 2). The lower half-guard is excluded because well-behaved kfuncs only access forward from arena pointers. The kfunc-author contract - access at mostGUARD_SZ/ 2 past a handed-in pointer - is documented inDocumentation/bpf/kfuncs.rst.The install is lock-free via
ptep_try_set(). On race-loss the winning installer's PTE is already valid, so the access retry succeeds. The arena clear path usesptep_get_and_clear()so installer and clearer race through atomic accessors. Noflush_tlb_kernel_range()afterwards. Stale "not mapped" entries just cause one extra re-fault, cheaper than a global IPI on every install.Scratch exists only to keep the kernel from oopsing on an in-line arena access. Its presence at a PTE means the BPF program has already malfunctioned, and the violation is reported through the program's BPF stream. The only requirement for behavior on a scratched PTE is that the kernel doesn't crash. In particular, any user-side access through such a PTE may segfault. The shared scratch page is freed once during map destruction.
BPF instruction faults continue to use the existing JIT exception-table path. This patch changes only the kernel-text fault path. No UAPI flag is added. The new behavior is the default.
v2: Use
ptep_get_and_clear()inapply_range_clear_cb(). (David)
v3: Stubbpf_arena_handle_page_fault()for !CONFIG_BPF_SYSCALL. (lkp)Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Tejun Heo <[email protected]>Reviewed-by Emil Tsalapatis <[email protected]>Cc David Hildenbrand <[email protected]>--- Documentation/bpf/kfuncs.rst | 14 +++ arch/arm64/mm/fault.c | 10 +- arch/x86/mm/fault.c | 12 ++- include/linux/bpf.h | 1 + include/linux/bpf_defs.h | 19 ++++ kernel/bpf/arena.c | 177 +++++++++++++++++++++++++++-------- kernel/bpf/core.c | 5 + 7 files changed, 191 insertions(+), 47 deletions(-) create mode 100644 include/linux/bpf_defs.hDocumentation/bpf/kfuncs.rst +14
-0@@ -462,6 +462,20 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +2.8 Accessing arena memory through kfunc arguments +-------------------------------------------------- + +A read or write at any address inside an arena does not oops the kernel. +Unallocated arena pages are lazily backed by a scratch page and the +access is reported through the program's BPF stream as an error. Only +the BPF program's correctness is affected; the kernel itself remains +intact. + +The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that +is also covered by this recovery. A kfunc handed an arena pointer may +therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking +against the arena. Larger accesses must verify the range explicitly. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectationsarch/arm64/mm/fault.c +7
-3@@ -9,6 +9,7 @@#include <linux/acpi.h> #include <linux/bitfield.h> +#include <linux/bpf_defs.h> #include <linux/extable.h> #include <linux/kfence.h> #include <linux/signal.h> @@ -416,9 +417,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, }else if (addr <PAGE_SIZE ) { msg ="NULL pointer dereference" ; }else { -if (esr_fsc_is_translation_fault (esr) && -kfence_handle_page_fault (addr, esr & ESR_ELx_WNR, regs)) -return ; +if (esr_fsc_is_translation_fault (esr)) { +if (kfence_handle_page_fault (addr, esr & ESR_ELx_WNR, regs)) +return ; +if (bpf_arena_handle_page_fault (addr, esr & ESR_ELx_WNR, regs->pc)) +return ; + } msg ="paging request" ; }arch/x86/mm/fault.c +8
-4@@ -8,6 +8,7 @@#include <linux/sched/task_stack.h> /* task_stack_*(), ... */ #include <linux/kdebug.h> /* oops_begin/end, ... */ #include <linux/memblock.h> /* max_low_pfn */ +#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */ #include <linux/kfence.h> /* kfence_handle_page_fault */ #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ #include <linux/mmiotrace.h> /* kmmio_handler, ... */ @@ -688,10 +689,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code,if (IS_ENABLED (CONFIG_EFI ))efi_crash_gracefully_on_page_fault (address); -/* Only not-present faults should be handled by KFENCE. */ -if (!(error_code &X86_PF_PROT ) && -kfence_handle_page_fault (address, error_code &X86_PF_WRITE , regs)) -return ; +/* Only not-present faults should be handled by KFENCE or BPF arena. */ +if (!(error_code &X86_PF_PROT )) { +if (kfence_handle_page_fault (address, error_code &X86_PF_WRITE , regs)) +return ; +if (bpf_arena_handle_page_fault (address, error_code &X86_PF_WRITE , regs->ip)) +return ; + } oops:/* include/linux/bpf.h +1
-0@@ -6,6 +6,7 @@#include <uapi/linux/bpf.h> #include <uapi/linux/filter.h> +#include <linux/bpf_defs.h> #include <crypto/sha2.h> #include <linux/workqueue.h> include/linux/bpf_defs.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Subset of bpf.h declarations, split out so files that need only these +* declarations can avoid bpf.h's full include cost. +*/ +#ifndef _LINUX_BPF_DEFS_H +#define _LINUX_BPF_DEFS_H + +#ifdef CONFIG_BPF_SYSCALL +bool bpf_arena_handle_page_fault (unsigned long addr,bool is_write,unsigned long fault_ip); +#else +static inline bool bpf_arena_handle_page_fault (unsigned long addr,bool is_write, +unsigned long fault_ip) +{ +return false; +} +#endif + +#endif /* _LINUX_BPF_DEFS_H */ kernel/bpf/arena.c +6
-0@@ -53,6 +53,7 @@ struct bpf_arena {u64 user_vm_start;u64 user_vm_end;struct vm_struct *kern_vm; +struct page *scratch_page;struct range_tree rt;/* protects rt */ rqspinlock_t spinlock;@@ -118,6 +119,11 @@ struct apply_range_data {int i; }; +struct clear_range_data { +struct llist_head *free_pages; +struct page *scratch_page; +}; +static int apply_range_set_cb (pte_t *pte,unsigned long addr,void *data) {struct apply_range_data *d = data;
kernel/bpf/arena.c static void flush_vmap_cache(unsigned long start, unsigned long size) · line 151
Quoted 2 lines
kernel/bpf/arena.c +0
-0@@ -144,33 +150,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size)flush_cache_vmap (start, start + size); }
There is still the chance that apply_range_set_cb() could race with scratch insertion, right?
Shouldn't we also be using ptep_try_set() there?
The nasty thing is handling whether ptep_try_set() actually works.
Something like the following on top, maybe?
kernel/bpf/arena.c +0 -0
@@ -122,19 +122,27 @@ static int apply_range_set_cb(pte_t *pte, unsigned long
addr, void *data)
{
struct apply_range_data *d = data;
struct page *page;+ pte_t pteval;
if (!data)
return 0;- /* sanity check */
- if (unlikely(!pte_none(ptep_get(pte))))
- return -EBUSY;
page = d->pages[d->i];
/* paranoia, similar to vmap_pages_pte_range() */
if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
return -EINVAL;- set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));
+ pteval = mk_pte(page, PAGE_KERNEL);
+#ifdef ptep_try_set
+ if (unlikely(!ptep_try_set(pte, pteval)))
+ return -EBUSY;
+#else
+ if (unlikely(!pte_none(ptep_get(pte))))
+ return -EBUSY;
+
+ set_pte_at(&init_mm, addr, pte, pteval);
+#endif
d->i++;
return 0;}
Signature · 3 lines
Cheers, David
Re: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Quoted 3 lines · David Hildenbrand (Arm)
There is still the chance that
apply_range_set_cb()could race with scratch insertion, right?
yes, it can race, but the fix is to remove only:
- /* sanity check */
- if (unlikely(!pte_none(ptep_get(pte))))
- return -EBUSY;
It was sanity check, now it's simply in the way.
It should do set_pte_at() unconditionally.
It's totally fine to set it to proper page instead of scratch or empty.
Tejun,
do you mind sending a patch to remove these 3 lines or should I do it?
Changing topic. re: zap_vma_range() issue.
It's not forgotten, just complicated, since we can't just take mmap_read_lock under arena->lock, since arena_vm_close() is called with mmap_write_lock held and it takes arena->lock. Will figure something out.
Quoted 43 lines
Shouldn't we also be using
ptep_try_set()there?The nasty thing is handling whether
ptep_try_set()actually works.Something like the following on top, maybe?
kernel/bpf/arena.c +11
-4@@ -122,19 +122,27 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) {struct apply_range_data *d = data;struct page *page; +pte_t pteval;if (!data)return 0 ; -/* sanity check */ -if (unlikely (!pte_none (ptep_get (pte)))) -return -EBUSY ; page = d->pages[d->i];/* paranoia, similar to vmap_pages_pte_range() */ if (WARN_ON_ONCE (!pfn_valid (page_to_pfn (page))))return -EINVAL ; -set_pte_at (&init_mm, addr, pte,mk_pte (page,PAGE_KERNEL )); + pteval =mk_pte (page,PAGE_KERNEL ); +#ifdef ptep_try_set +if (unlikely (!ptep_try_set (pte, pteval))) +return -EBUSY ; +#else +if (unlikely (!pte_none (ptep_get (pte)))) +return -EBUSY ; + +set_pte_at (&init_mm, addr, pte, pteval); +#endif d->i++;return 0 ; }
Tejun Heo
reviewerRe: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Hello,
Quoted 1 line
It should do
set_pte_at()unconditionally.
I think two concerns are tangled here.
1. The racing write. set_pte_at() and the scratch installer's ptep_try_set() hit the same PTE with no common lock. On x86-64 and arm64 set_pte_at() is a single atomic store, so it can't tear against the cmpxchg, but a plain store racing a cmpxchg isn't atomic in general. David, is that the worry - an arch where set_pte_at() is split and could tear - or something else?
2. The SEGV. It's a BPF program failure propagating out as a SEGV. Maybe not ideal, but as long as we surface the BPF error properly, it doesn't necessarily seem broken to me.
Thanks.
--
tejun
Re: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Quoted 14 lines · Tejun Heo
Hello,
Quoted 1 line
It should do
set_pte_at()unconditionally.I think two concerns are tangled here.
1. The racing write.
set_pte_at()and the scratch installer'sptep_try_set()hit the same PTE with no common lock. On x86-64 and arm64set_pte_at()is a single atomic store, so it can't tear against the cmpxchg, but a plain store racing a cmpxchg isn't atomic in general. David, is that the worry - an arch whereset_pte_at()is split and could tear - or something else?
Quoted 3 lines
2. The SEGV. It's a BPF program failure propagating out as a SEGV. Maybe not ideal, but as long as we surface the BPF error properly, it doesn't necessarily seem broken to me.
returning EBUSY because apply_range_set_cb() hit scratch page
is SEGV out of arena_vm_fault() and arguably ok-ish,
but bpf_arena_alloc_pages() returning NULL because scratch page was in the range just sucks.
Earlier bpf prog passed the wrong arena addr to kfunc and triggered that scratch page. It broke the contract and kept the pieces,
so ok-ish too, but overwriting scratch page with proper page during bpf_arena_alloc_pages() is imo much better behavior.
That scratch page will cause all future bpf_arena_alloc_pages() fail as well. Hence I prefer that check removed.
Tejun Heo
reviewerRe: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Hello,
On Fri, May 29, 2026 at 11:38:21AM -0700, Alexei Starovoitov wrote:
...
Quoted 11 lines
Quoted 10 lines
1. The racing write.
set_pte_at()and the scratch installer'sptep_try_set()hit the same PTE with no common lock. On x86-64 and arm64set_pte_at()is a single atomic store, so it can't tear against the cmpxchg, but a plain store racing a cmpxchg isn't atomic in general. David, is that the worry - an arch whereset_pte_at()is split and could tear - or something else?2. The SEGV. It's a BPF program failure propagating out as a SEGV. Maybe not ideal, but as long as we surface the BPF error properly, it doesn't necessarily seem broken to me.
Quoted 10 lines
returning EBUSY because
apply_range_set_cb()hit scratch page
is SEGV out ofarena_vm_fault()and arguably ok-ish,
butbpf_arena_alloc_pages()returning NULL because scratch page was in the range just sucks.
Earlier bpf prog passed the wrong arena addr to kfunc and triggered that scratch page. It broke the contract and kept the pieces,
so ok-ish too, but overwriting scratch page with proper page duringbpf_arena_alloc_pages()is imo much better behavior.
That scratch page will cause all futurebpf_arena_alloc_pages()fail as well. Hence I prefer that check removed.
Yeah, let's do that. David, would that be enough? Or are you still concerned about set_pte_at() competing with ptep_try_set()?
Thanks.
Signature · 1 line
tejun
Tejun Heo
reviewerRe: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Hello,
I posted the check removal [1], and Sashiko's review flagged a break-before-make problem with it [2] that I think is real.
The scratch page is a present PAGE_KERNEL mapping, so having apply_range_set_cb() overwrite it via set_pte_at() during bpf_arena_alloc_pages() is a valid->valid PFN change. I'm not familiar with arm at all. David, my understanding is that's a break-before-make violation on arm64, and that on any arch the stale TLB entry keeps resolving to the shared scratch page until it's flushed, so a later access can hit scratch instead of the new page. Is that what you were worried about?
So instead of just dropping the check, the install should route through an invalid entry rather than overwrite in place:
while (!ptep_try_set(pte, mk_pte(page, PAGE_KERNEL))) {
old = ptep_get(pte);
if (pte_none(old))
continue;
if (WARN_ON_ONCE(pte_page(old) != arena->scratch_page))
return -EBUSY;
ptep_get_and_clear(&init_mm, addr, pte);
broke_scratch = true;
}ptep_try_set() only fills a none slot, so the slot goes scratch->none->page and never valid->valid, and the loop copes with a concurrent fault re-scratching it. This also closes the set_pte_at()-vs-ptep_try_set() race I raised earlier, since both sides are now cmpxchg. A broken scratch entry was live, so the caller flush_tlb_kernel_range()s those pages when broke_scratch is set, like arena_free_pages() already does after clearing.
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
Thanks.
--
tejun
David Hildenbrand (Arm)
maintainerRe: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
Quoted 4 lines · Tejun Heo
Hello,
I posted the check removal [1], and Sashiko's review flagged a break-before-make problem with it [2] that I think is real.
Yeah, and as I raised previously, this is very questionable locking design :)
Either everybody works with atomics or nobody.
Quoted 12 lines
The scratch page is a present
PAGE_KERNELmapping, so havingapply_range_set_cb()overwrite it viaset_pte_at()duringbpf_arena_alloc_pages()is avalid->validPFN change. I'm not familiar with arm at all. David, my understanding is that's a break-before-make violation on arm64, and that on any arch the stale TLB entry keeps resolving to the shared scratch page until it's flushed, so a later access can hit scratch instead of the new page. Is that what you were worried about?So instead of just dropping the check, the install should route through an invalid entry rather than overwrite in place:
Quoted 9 lines
while (!ptep_try_set(pte, mk_pte(page, PAGE_KERNEL))) {
old = ptep_get(pte);
if (pte_none(old))
continue;
if (WARN_ON_ONCE(pte_page(old) != arena->scratch_page))
return -EBUSY;
ptep_get_and_clear(&init_mm, addr, pte);
broke_scratch = true;
}We have to handle architectures where ptep_try_set() is not implemented (as I tried with my variant).
Signature · 3 lines
Cheers, David
Re: [PATCH 2/8] bpf: Recover arena kernel faults with scratch page
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
Hello,
I posted the check removal [1], and Sashiko's review flagged a break-before-make problem with it [2] that I think is real.
ohh. interesting.
Quoted 13 lines
So instead of just dropping the check, the install should route through an invalid entry rather than overwrite in place:
while (!ptep_try_set(pte, mk_pte(page, PAGE_KERNEL))) { old = ptep_get(pte); if (pte_none(old)) continue; if (WARN_ON_ONCE(pte_page(old) != arena->scratch_page)) return -EBUSY; ptep_get_and_clear(&init_mm, addr, pte); broke_scratch = true; }
Quoted 6 lines
ptep_try_set()only fills a none slot, so the slot goesscratch->none->pageand nevervalid->valid, and the loop copes with a concurrent fault re-scratching it. This also closes the set_pte_at()-vs-ptep_try_set() race I raised earlier, since both sides are now cmpxchg. A broken scratch entry was live, so the caller flush_tlb_kernel_range()s those pages whenbroke_scratchis set, likearena_free_pages()already does after clearing.
Makes sense to me.
Tejun Heo
reviewerPATCH 3/8 bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers
In reply to Tejun HeoThe existing kernel-side export of bpf_arena_alloc_pages is _non_sleepable only - it's used by the verifier to inline the kfunc when the call site is non-sleepable. There is no sleepable equivalent for kernel callers. The kfunc bpf_arena_alloc_pages itself is BPF-only.
sched_ext needs sleepable kernel-side allocs for its arena pool init/grow paths. Add bpf_arena_alloc_pages_sleepable() mirroring the _non_sleepable wrapper but passing sleepable=true to arena_alloc_pages().
--- include/linux/bpf.h | 8 ++++++++ kernel/bpf/arena.c | 13 +++++++++++++ 2 files changed, 21 insertions(+)
include/linux/bpf.h +8 -0
@@ -679,6 +679,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt); +void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id, +u64 flags);#else static inline void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags)
@@ -689,6 +691,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addrstatic inline void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt) { } + +static inline void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +return NULL ; +}#endif extern const struct bpf_map_ops bpf_map_offload_ops;
kernel/bpf/arena.c +13 -0
@@ -934,6 +934,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pagreturn (void *)arena_alloc_pages (arena, (long )addr__ign, page_cnt, node_id, false); } + +void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + +if (map->map_type !=BPF_MAP_TYPE_ARENA || flags || !page_cnt) +return NULL ; + +return (void *)arena_alloc_pages (arena, (long )addr__ign, page_cnt, node_id, true); +} + __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) {struct bpf_map *map = p__map;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 4/8 bpf: Add bpf_struct_ops_for_each_prog()
In reply to Tejun HeoAdd a helper that walks the member progs of the struct_ops map containing a given @kdata vmtable. struct_ops ->reg() callbacks (and similar) sometimes need to inspect the loaded BPF programs, e.g. to discover maps they reference via prog->aux->used_maps.
The implementation mirrors bpf_struct_ops_id(): container_of @kdata to recover the bpf_struct_ops_map, then iterate st_map->links[i]->prog for i in [0, funcs_cnt). Same access pattern, no new locking - by the time ->reg() fires st_map is fully populated and stable.
A sched_ext follow-up walks the member progs of a cid-form scheduler's struct_ops map, reads prog->aux->arena directly, and requires all member progs to reference exactly one arena, without requiring the BPF program to call a registration kfunc.
--- include/linux/bpf.h | 3 +++ kernel/bpf/bpf_struct_ops.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/linux/bpf.h +3 -0
@@ -2129,6 +2129,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);void bpf_prog_disassoc_struct_ops (struct bpf_prog *prog);void *bpf_prog_get_assoc_struct_ops (const struct bpf_prog_aux *aux);u32 bpf_struct_ops_id (const void *kdata); +int bpf_struct_ops_for_each_prog (const void *kdata, +int (*cb)(struct bpf_prog *prog,void *data), +void *data);#ifdef CONFIG_NET /* Define it here to avoid the use of forward declaration */
kernel/bpf/bpf_struct_ops.c +36 -0
@@ -1203,6 +1203,42 @@ u32 bpf_struct_ops_id(const void *kdata) }EXPORT_SYMBOL_GPL (bpf_struct_ops_id); +/** +* bpf_struct_ops_for_each_prog - Invoke @cb for each member prog +* @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg) +* @cb: callback invoked once per member prog; non-zero return stops iteration +* @data: opaque argument passed to @cb +* +* Walks the struct_ops member progs registered on the map containing @kdata. +* Intended for use from struct_ops ->reg() callbacks (and similar) that need to +* inspect the loaded BPF programs (for example to discover maps they reference +* via @prog->aux->used_maps). +* +* Return 0 if iteration completed, otherwise the first non-zero @cb return. +*/ +int bpf_struct_ops_for_each_prog (const void *kdata, +int (*cb)(struct bpf_prog *prog,void *data), +void *data) +{ +struct bpf_struct_ops_value *kvalue; +struct bpf_struct_ops_map *st_map; +u32 i; +int ret; + + kvalue = container_of(kdata, struct bpf_struct_ops_value, data); + st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue); + +for (i =0 ; i < st_map->funcs_cnt; i++) { +if (!st_map->links[i]) +continue ; + ret =cb (st_map->links[i]->prog, data); +if (ret) +return ret; + } +return 0 ; +} +EXPORT_SYMBOL_GPL (bpf_struct_ops_for_each_prog); +static bool bpf_struct_ops_valid_to_reg (struct bpf_map *map) {struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 5/8 bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena()
In reply to Tejun Heostruct bpf_arena is opaque to callers outside arena.c. Add two helpers for struct_ops subsystems that need to reach into an arena:
bpf_arena_map_kern_vm_start(struct bpf_map *map)
returns @map's kern_vm_start. A sched_ext follow-up needs this to translate kern_va <-> uaddr.
bpf_prog_arena(struct bpf_prog *prog)
returns the bpf_map of the arena referenced by @prog (NULL if @prog references no arena). The verifier enforces at most one arena per program. Used by struct_ops callers that auto-discover an arena from a member prog and need to take a map reference.
--- include/linux/bpf.h | 2 ++ kernel/bpf/arena.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
include/linux/bpf.h +2 -0
@@ -618,6 +618,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, struct bpf_spin_lock *spin_lock);u64 bpf_arena_get_kern_vm_start (struct bpf_arena *arena);u64 bpf_arena_get_user_vm_start (struct bpf_arena *arena); +u64 bpf_arena_map_kern_vm_start (struct bpf_map *map); +struct bpf_map *bpf_prog_arena (struct bpf_prog *prog);int bpf_obj_name_cpy (char *dst,const char *src,unsigned int size);struct bpf_offload_dev ;
kernel/bpf/arena.c +26 -0
@@ -84,6 +84,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena)return arena ? arena->user_vm_start :0 ; } +/** +* bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map * +* @map: a BPF_MAP_TYPE_ARENA map +* +* Return @map's kern_vm_start. +*/ +u64 bpf_arena_map_kern_vm_start (struct bpf_map *map) +{ + return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map)); +} + +/** +* bpf_prog_arena - return the bpf_map of the arena referenced by @prog +* @prog: a loaded BPF program +* +* The verifier enforces at most one arena per program and stores it in +* prog->aux->arena. Return that arena's underlying bpf_map, or NULL if +* @prog does not reference an arena. +*/ +struct bpf_map *bpf_prog_arena (struct bpf_prog *prog) +{ +struct bpf_arena *arena = prog->aux->arena; + +return arena ? &arena->map :NULL ; +} +static long arena_map_peek_elem (struct bpf_map *map,void *value) {return -EOPNOTSUPP ;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 6/8 sched_ext: Require an arena for cid-form schedulers
In reply to Tejun HeoUpcoming patches will let the kernel place arena-resident scratch shared with the BPF program (e.g. per-CPU set_cmask cmask) so the BPF side can dereference it directly via __arena pointers, replacing the current cmask_copy_from_kernel() probe-read loop. That requires each cid-form scheduler to expose its arena to the kernel. Kernel- side accesses are recovered by the per-arena scratch-page mechanism.
bpf_scx_reg_cid() walks the struct_ops member progs via bpf_struct_ops_for_each_prog() and reads each prog's arena via bpf_prog_arena(). The verifier enforces one arena per program, so each member prog contributes at most one arena. All non-NULL contributions must match and at least one member prog must use an arena. The map ref is held on scx_sched and dropped on sched destroy. cpu-form schedulers (bpf_scx_reg) are unchanged - no arena requirement.
--- kernel/sched/ext.c | 56 ++++++++++++++++++++++++++++++++++++- kernel/sched/ext_internal.h | 8 ++++++ 2 files changed, 63 insertions(+), 1 deletion(-)
kernel/sched/ext.c +55 -1
@@ -5003,6 +5003,8 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +if (sch->arena_map) +bpf_map_put (sch->arena_map);kfree (sch); }
@@ -6746,6 +6748,7 @@ struct scx_enable_cmd {struct sched_ext_ops_cid *ops_cid; };bool is_cid_type; +struct bpf_map *arena_map;/* arena ref to transfer to sch */ int ret; };
@@ -6913,6 +6916,15 @@ static struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, return ERR_PTR(ret); }#endif /* CONFIG_EXT_SUB_SCHED */ + +/* +* Consume the arena_map ref bpf_scx_reg_cid() took. Defer to here so +* earlier failure paths leave cmd->arena_map set and bpf_scx_reg_cid +* drops the ref. After this point, sch owns the ref and any cleanup +* runs through scx_sched_free_rcu_work() which puts it. +*/ + sch->arena_map = cmd->arena_map; + cmd->arena_map =NULL ;return sch;#ifdef CONFIG_EXT_SUB_SCHED
@@ -7898,11 +7910,53 @@ static int bpf_scx_reg(void *kdata, struct bpf_link *link)return scx_enable (&cmd, link); } +struct scx_arena_scan { +struct bpf_map *arena; +int err; +}; + +/* +* The verifier enforces one arena per BPF program, so each struct_ops +* member prog contributes at most one arena via bpf_prog_arena(). +* Require all non-NULL contributions to match. +*/ +static int scx_arena_scan_prog (struct bpf_prog *prog,void *data) +{ +struct scx_arena_scan *s = data; +struct bpf_map *arena =bpf_prog_arena (prog); + +if (!arena) +return 0 ; +if (s->arena && s->arena != arena) { + s->err = -EINVAL ; +return 1 ; + } + s->arena = arena; +return 0 ; +} +static int bpf_scx_reg_cid (void *kdata,struct bpf_link *link) {struct scx_enable_cmd cmd = { .ops_cid = kdata, .is_cid_type = true }; +struct scx_arena_scan scan = {}; +int ret; -return scx_enable (&cmd, link); +bpf_struct_ops_for_each_prog (kdata, scx_arena_scan_prog, &scan); +if (scan.err) { +pr_err ("sched_ext: cid-form scheduler uses multiple arena maps\n" ); +return scan.err; + } +if (!scan.arena) { +pr_err ("sched_ext: cid-form scheduler must use a BPF arena map\n" ); +return -EINVAL ; + } + +bpf_map_inc (scan.arena); + cmd.arena_map = scan.arena; + ret =scx_enable (&cmd, link); +if (cmd.arena_map)/* not consumed by scx_alloc_and_add_sched() */ +bpf_map_put (cmd.arena_map); +return ret; }static void bpf_scx_unreg (void *kdata,struct bpf_link *link)
kernel/sched/ext_internal.h +8 -0
@@ -1111,6 +1111,14 @@ struct scx_sched {struct sched_ext_ops_cid ops_cid; };bool is_cid_type;/* true if registered via bpf_sched_ext_ops_cid */ + +/* +* Arena map auto-discovered from member progs at struct_ops attach. +* cid-form schedulers must use exactly one arena across all member +* progs. NULL on cpu-form. +*/ +struct bpf_map *arena_map; +DECLARE_BITMAP (has_op,SCX_OPI_END );/*
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 7/8 sched_ext: Sub-allocator over kernel-claimed BPF arena pages
In reply to Tejun HeoBuild a per-scheduler sub-allocator on top of pages claimed from the BPF arena registered in the previous patch. Subsequent kernel-managed arena-resident structures (e.g. per-CPU set_cmask cmask) carve their storage from this pool.
scx_arena_pool_init() creates a gen_pool. scx_arena_alloc() returns the kernel VA. On exhaustion, the pool grows by claiming more pages via bpf_arena_alloc_pages_sleepable(). Chunks are added at the kernel-side mapping address. Callers translate to the BPF-arena form themselves if needed.
Allocations sleep (GFP_KERNEL) - they may grow the pool through vzalloc and arena page allocation. All current consumers run from the enable path (after ops.init() and the kernel-side arena auto-discovery, before validate_ops()), where sleeping is fine.
scx_arena_pool_destroy() walks each chunk, returns outstanding ranges to the gen_pool with gen_pool_free() and then calls gen_pool_destroy(). The underlying arena pages are released when the arena map itself is torn down, so the pool destroy doesn't free them explicitly.
v2: Switch scx_arena_alloc() to a loop. (Andrea)
--- kernel/sched/build_policy.c | 4 ++ kernel/sched/ext.c | 11 ++++ kernel/sched/ext_arena.c | 126 ++++++++++++++++++++++++++++++++++++ kernel/sched/ext_arena.h | 18 ++++++ kernel/sched/ext_internal.h | 5 ++ 5 files changed, 164 insertions(+) create mode 100644 kernel/sched/ext_arena.c create mode 100644 kernel/sched/ext_arena.h
kernel/sched/build_policy.c +4 -0
@@ -59,12 +59,16 @@#ifdef CONFIG_SCHED_CLASS_EXT # include <linux/btf_ids.h> +# include <linux/find.h> +# include <linux/genalloc.h> # include "ext_types.h" # include "ext_internal.h" # include "ext_cid.h" +# include "ext_arena.h" # include "ext_idle.h" # include "ext.c" # include "ext_cid.c" +# include "ext_arena.c" # include "ext_idle.c" #endif
kernel/sched/ext.c +11 -0
@@ -5003,6 +5003,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +scx_arena_pool_destroy (sch);if (sch->arena_map)bpf_map_put (sch->arena_map);kfree (sch);
@@ -7155,6 +7156,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; } + ret =scx_arena_pool_init (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)if (((void (**)(void ))ops)[i])set_bit (i, sch->has_op);
@@ -7473,6 +7480,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work) sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; } + ret =scx_arena_pool_init (sch); +if (ret) + goto err_disable; +if (validate_ops (sch, ops)) goto err_disable;
kernel/sched/ext_arena.c new file +126 -0
126 lines not on this page · raw message
kernel/sched/ext_arena.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst +* +* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2025 Tejun Heo <[email protected]> +*/ +#ifndef _KERNEL_SCHED_EXT_ARENA_H +#define _KERNEL_SCHED_EXT_ARENA_H + +struct scx_sched ; + +s32 scx_arena_pool_init (struct scx_sched *sch); +void scx_arena_pool_destroy (struct scx_sched *sch); +void *scx_arena_alloc (struct scx_sched *sch,size_t size); +void scx_arena_free (struct scx_sched *sch,void *kern_va,size_t size); + +#endif /* _KERNEL_SCHED_EXT_ARENA_H */
kernel/sched/ext_internal.h +5 -0
@@ -1116,8 +1116,13 @@ struct scx_sched {* Arena map auto-discovered from member progs at struct_ops attach. * cid-form schedulers must use exactly one arena across all member * progs. NULL on cpu-form. +* +* @arena_pool sub-allocates @arena_map. Each gen_pool chunk is added +* at the kernel-side mapping address. Grows on demand and pages are +* not released until sched destroy. */ struct bpf_map *arena_map; +struct gen_pool *arena_pool;DECLARE_BITMAP (has_op,SCX_OPI_END );
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 8/8 sched_ext: Convert ops.set_cmask() to arena-resident cmask
In reply to Tejun Heoops_cid.set_cmask() expects a cmask. The kernel couldn't write into the arena, so it translated cpumask -> cmask in kernel memory and passed the result as a trusted pointer. The BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.
With direct kernel-side arena access now in place, build the cmask in the arena. The kernel writes to it through the kern_va side of the dual mapping. BPF directly dereferences it via an __arena pointer like any other arena struct.
--- kernel/sched/ext.c | 68 +++++++++++++++++++++++++-- kernel/sched/ext_cid.c | 20 +------- kernel/sched/ext_internal.h | 10 +++- tools/sched_ext/include/scx/cid.bpf.h | 52 -------------------- tools/sched_ext/scx_qmap.bpf.c | 5 +- 5 files changed, 75 insertions(+), 80 deletions(-)
kernel/sched/ext.c +63 -5
@@ -621,11 +621,16 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,update_locked_rq (rq);if (scx_is_cid_type ()) { -struct scx_cmask *cmask =this_cpu_ptr (scx_set_cmask_scratch); - -lockdep_assert_irqs_disabled (); -scx_cpumask_to_cmask (cpumask, cmask); - sch->ops_cid.set_cmask (task, cmask); +struct scx_cmask *kern_va = *this_cpu_ptr (sch->set_cmask_scratch); +unsigned long uaddr = (unsigned long )kern_va - +bpf_arena_map_kern_vm_start (sch->arena_map); +/* +* Build the per-CPU arena cmask and hand BPF the uaddr. Caller +* holds the rq lock with IRQs disabled, which makes us the sole +* user of the scratch area. +*/ +scx_cpumask_to_cmask (cpumask, kern_va); + sch->ops_cid.set_cmask (task, (struct scx_cmask *)uaddr); }else { sch->ops.set_cpumask (task, cpumask); }
@@ -4949,6 +4954,48 @@ static const struct attribute_group scx_global_attr_group = {static void free_pnode (struct scx_sched_pnode *pnode);static void free_exit_info (struct scx_exit_info *ei); +static s32 scx_set_cmask_scratch_alloc (struct scx_sched *sch) +{ + size_t size = struct_size_t(struct scx_cmask, bits, +SCX_CMASK_NR_WORDS (num_possible_cpus ())); +int cpu; + +if (!sch->is_cid_type || !sch->arena_pool) +return 0 ; + + sch->set_cmask_scratch = alloc_percpu(struct scx_cmask *); +if (!sch->set_cmask_scratch) +return -ENOMEM ; + +for_each_possible_cpu (cpu) { +struct scx_cmask **slot =per_cpu_ptr (sch->set_cmask_scratch, cpu); + + *slot =scx_arena_alloc (sch, size); +if (!*slot) +return -ENOMEM ; +scx_cmask_init (*slot,0 ,num_possible_cpus ()); + } +return 0 ; +} + +static void scx_set_cmask_scratch_free (struct scx_sched *sch) +{ + size_t size = struct_size_t(struct scx_cmask, bits, +SCX_CMASK_NR_WORDS (num_possible_cpus ())); +int cpu; + +if (!sch->set_cmask_scratch) +return ; + +for_each_possible_cpu (cpu) { +struct scx_cmask **slot =per_cpu_ptr (sch->set_cmask_scratch, cpu); + +scx_arena_free (sch, *slot, size); + } +free_percpu (sch->set_cmask_scratch); + sch->set_cmask_scratch =NULL ; +} +static void scx_sched_free_rcu_work (struct work_struct *work) {struct rcu_work *rcu_work =to_rcu_work (work);
@@ -5003,6 +5050,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +scx_set_cmask_scratch_free (sch);scx_arena_pool_destroy (sch);if (sch->arena_map)bpf_map_put (sch->arena_map);
@@ -7162,6 +7210,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) goto err_disable; } + ret =scx_set_cmask_scratch_alloc (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)if (((void (**)(void ))ops)[i])set_bit (i, sch->has_op);
@@ -7484,6 +7538,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work)if (ret) goto err_disable; + ret =scx_set_cmask_scratch_alloc (sch); +if (ret) + goto err_disable; +if (validate_ops (sch, ops)) goto err_disable;
kernel/sched/ext_cid.c +1 -19
@@ -7,14 +7,6 @@*/ #include <linux/cacheinfo.h> -/* -* Per-cpu scratch cmask used by scx_call_op_set_cpumask() to synthesize a -* cmask from a cpumask. Allocated alongside the cid arrays on first enable -* and never freed. Sized to the full cid space. Caller holds rq lock so -* this_cpu_ptr is safe. -*/ -struct scx_cmask __percpu *scx_set_cmask_scratch; -/* * cid tables. *
@@ -54,8 +46,6 @@ static s32 scx_cid_arrays_alloc(void)u32 npossible =num_possible_cpus ();s16 *cid_to_cpu, *cpu_to_cid;struct scx_cid_topo *cid_topo; -struct scx_cmask __percpu *set_cmask_scratch; -s32 cpu;if (scx_cid_to_cpu_tbl)return 0 ;
@@ -63,25 +53,17 @@ static s32 scx_cid_arrays_alloc(void) cid_to_cpu =kzalloc_objs (*scx_cid_to_cpu_tbl, npossible,GFP_KERNEL ); cpu_to_cid =kzalloc_objs (*scx_cpu_to_cid_tbl, nr_cpu_ids,GFP_KERNEL ); cid_topo =kmalloc_objs (*scx_cid_topo, npossible,GFP_KERNEL ); - set_cmask_scratch =__alloc_percpu (struct_size (set_cmask_scratch, bits, -SCX_CMASK_NR_WORDS (npossible)), -sizeof (u64)); -if (!cid_to_cpu || !cpu_to_cid || !cid_topo || !set_cmask_scratch) { +if (!cid_to_cpu || !cpu_to_cid || !cid_topo) {kfree (cid_to_cpu);kfree (cpu_to_cid);kfree (cid_topo); -free_percpu (set_cmask_scratch);return -ENOMEM ; }WRITE_ONCE (scx_cid_to_cpu_tbl, cid_to_cpu);WRITE_ONCE (scx_cpu_to_cid_tbl, cpu_to_cid);WRITE_ONCE (scx_cid_topo, cid_topo); -for_each_possible_cpu (cpu) -scx_cmask_init (per_cpu_ptr (set_cmask_scratch, cpu), -0 , npossible); -WRITE_ONCE (scx_set_cmask_scratch, set_cmask_scratch);return 0 ; }
kernel/sched/ext_internal.h +8 -2
@@ -1124,6 +1124,14 @@ struct scx_sched {struct bpf_map *arena_map;struct gen_pool *arena_pool; +/* +* Per-CPU arena cmask used by scx_call_op_set_cpumask() to hand a cmask +* to ops_cid.set_cmask(). The kernel writes through the stored kern_va; +* the BPF-arena uaddr handed to BPF is recovered by subtracting the +* arena's kern_vm_start. +*/ +struct scx_cmask *__percpu *set_cmask_scratch; +DECLARE_BITMAP (has_op,SCX_OPI_END );/*
@@ -1480,8 +1488,6 @@ enum scx_ops_state {extern struct scx_sched __rcu *scx_root; DECLARE_PER_CPU(struct rq *, scx_locked_rq_state); -extern struct scx_cmask __percpu *scx_set_cmask_scratch; -/* * True when the currently loaded scheduler hierarchy is cid-form. All scheds * in a hierarchy share one form, so this single key tells callsites which
tools/sched_ext/include/scx/cid.bpf.h +0 -52
56 lines not on this page · raw message
tools/sched_ext/scx_qmap.bpf.c +3 -2
@@ -919,14 +919,15 @@ void BPF_STRUCT_OPS(qmap_update_idle, s32 cid, bool idle) }void BPF_STRUCT_OPS (qmap_set_cmask ,struct task_struct *p, -const struct scx_cmask *cmask) +const struct scx_cmask *cmask_in) { + struct scx_cmask __arena *cmask = (struct scx_cmask __arena *)(long)cmask_in;task_ctx_t *taskc; taskc =lookup_task_ctx (p);if (!taskc)return ; -cmask_copy_from_kernel (&taskc->cpus_allowed, cmask); +cmask_copy (&taskc->cpus_allowed, cmask); }struct monitor_timer {
Signature · 1 line
2.54.0
Quoted 27 lines · Tejun Heo
Patches 1-2 (atomic PTE install + arena scratch-page recovery)
--------------------------------------------------------------mm: Add
ptep_try_set()for lockless empty-slot installs
bpf: Recover arena kernel faults with scratch pagePatches 3-5 (helpers used by
struct_opsregistration)
-----------------------------------------------------bpf: Add sleepable variant of
bpf_arena_alloc_pagesfor kernel callers
bpf: Addbpf_struct_ops_for_each_prog()
bpf/arena: Addbpf_arena_map_kern_vm_start()andbpf_prog_arena()Patches 6-8 (
sched_ext: arena auto-discovery, allocator,set_cmask)
-------------------------------------------------------------------
sched_ext: Require an arena for cid-form schedulers
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Convertops.set_cmask()to arena-resident cmaskPatch 6 reads each member prog's
prog->aux->arenaviabpf_prog_arena()and requires the cid-formstruct_opsto reference exactly one arena. Patch 7 builds agen_poolsub-allocator inside that arena. Patch 8 convertsset_cmask()to write into arena memory. BPF dereferences via__arenalike any other arena struct, no probe-reads.
Quoted 6 lines
Base
----sched_ext/for-7.2 (
f31e89a8f583)Git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.gitarena-direct-v4
I created a stable branch bpf-next/arena_direct_access with the first 5 patches and merged it into bpf-next/master.
Tejun,
please pull it into sched-ext tree as well.
Let's do any follow ups on top.
Thanks everyone!
Tejun Heo
reviewerHello,
Patches 1-5 went through bpf-next and are pulled into sched_ext/for-7.2. Applied 6-8 to for-7.2.
Thanks.
Signature · 1 line
tejun