Skip to contentSkip to conversation

bpf/arena: Direct kernel-side access

PATCH Superseded Tejun Heo 5 participants 28 messages May 17 – May 25

Subsystems (10)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Tejun Heo

author reviewer
#0 lore raw

PATCH v2 1/1 bpf/arena: Direct kernel-side access

Hello,

This makes BPF arena memory directly dereferenceable from kernel code (struct_ops callbacks, kfuncs). Each arena gets a per-arena scratch page that an arch fault hook installs into empty PTEs on kernel-side faults, after KFENCE. The faulting instruction retries and the violation is reported through the program's BPF stream.

v2 changes since RFC v1 ([email protected]):

Switched from pre-filling the arena recovery range with a fallback page to leaving PTEs sparse and installing the scratch page from an arch fault hook on demand. Pre-fill erased the fault and lost error attribution; on-demand keeps the fault visible and reports the violation through the program's bpf_stream. Plus misc fixes.

Motivation
----------

sched_ext's ops_cid.set_cmask() hands the BPF scheduler a struct scx_cmask *. The kernel translates a kernel cpumask to a cmask, but it had no way to write into the arena, so the cmask lived in kernel memory and was passed as a trusted pointer. BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.

The shape isn't unique to set_cmask. Sub-scheduler support is on the way and more sched_ext callbacks will want to pass structured data to BPF. Anywhere a kfunc or struct_ops callback wants to hand a struct to a BPF program, arena residence is the natural answer.

Approach
--------

Each arena gets a per-arena scratch page. Arenas stay sparsely mapped as today - PTEs are populated only for allocated pages. A new arch fault hook (bpf_arena_handle_page_fault) is wired into x86 page_fault_oops() and arm64 __do_kernel_fault(), after KFENCE. When a kernel-side access faults inside an arena's kern_vm range, the helper walks the stack to find the BPF program responsible, range- checks the fault address against prog->aux->arena, and atomically installs the scratch page into the empty PTE via the new ptep_try_install() wrapper. The kernel instruction retries and reads/writes the scratch page. Free paths and map destruction treat scratch as non-owned. Real allocation refuses to overwrite scratch (apply_range_set_cb returns -EBUSY); a scratched address stays dead until map destroy, since its presence means the BPF program has already malfunctioned.

The mechanism is default behavior - no UAPI flag.

What this preserves
-------------------

All the debugging properties of today's sparse-PTE design are preserved:

* BPF programs still fault on unmapped arena accesses. The fault semantics (instruction retry with rdst = 0) and the violation report through bpf_streams are unchanged for prog-side accesses.

* The first kernel-side touch of an unmapped address is reported via bpf_streams the same way as a prog-side fault, with the stack walk attributing it to the originating prog.

* User-side fault on a never-scratched address still lazy-allocates a real page (or returns SIGSEGV under BPF_F_SEGV_ON_FAULT). User- side fault on a scratched address SIGSEGVs.

What changes for the kernel-side caller is just that an unmapped deref no longer oopses - it retries through the scratch page and emits a violation report. The same shape today's BPF instruction faults have.

Patches 1-2 (atomic PTE install + arena scratch-page recovery)
--------------------------------------------------------------

mm: Add ptep_try_install() for lockless empty-slot installs
bpf: Recover arena kernel faults with scratch page

Patches 3-5 (helpers used by struct_ops registration)
-----------------------------------------------------

bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers
bpf: Add bpf_struct_ops_for_each_prog()
bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena()

Patches 6-8 (sched_ext: arena auto-discovery, allocator, set_cmask)
-------------------------------------------------------------------

sched_ext: Require an arena for cid-form schedulers
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Convert ops.set_cmask() to arena-resident cmask

Patch 6 reads each member prog's prog->aux->arena via bpf_prog_arena() and requires the cid-form struct_ops to reference exactly one arena. Patch 7 builds a gen_pool sub-allocator inside that arena. Patch 8 converts set_cmask() to write into arena memory; BPF dereferences via __arena like any other arena struct, no probe-reads.

Base
----

sched_ext/for-7.2 (c9017d335aab) with cmask-prep-v2.1 applied:
https://lore.kernel.org/r/[email protected]

Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git arena-direct-v2

 Documentation/bpf/kfuncs.rst          |  14 +++
 arch/arm64/include/asm/pgtable.h      |   8 ++
 arch/arm64/mm/fault.c                 |  10 +-
 arch/x86/include/asm/pgtable.h        |   8 ++
 arch/x86/mm/fault.c                   |  12 +-
 include/linux/bpf.h                   |  14 +++
 include/linux/bpf_defs.h              |  11 ++
 include/linux/pgtable.h               |  16 +++
 kernel/bpf/arena.c                    | 209 ++++++++++++++++++++++++++++------
 kernel/bpf/bpf_struct_ops.c           |  36 ++++++
 kernel/bpf/core.c                     |   5 +
 kernel/sched/build_policy.c           |   4 +
 kernel/sched/ext.c                    | 135 +++++++++++++++++++++-
 kernel/sched/ext_arena.c              | 127 +++++++++++++++++++++
 kernel/sched/ext_arena.h              |  18 +++
 kernel/sched/ext_cid.c                |  20 +---
 kernel/sched/ext_internal.h           |  23 +++-
 tools/sched_ext/include/scx/cid.bpf.h |  52 ---------
 tools/sched_ext/scx_qmap.bpf.c        |   5 +-
 19 files changed, 604 insertions(+), 123 deletions(-)

Thanks.

--
tejun

author reviewer
#1 lore raw

PATCH 1/8 mm: Add ptep_try_install() for lockless empty-slot installs

Add ptep_try_install(ptep, new_pte): atomically set *ptep to new_pte iff it is currently pte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.

The intended caller is the upcoming bpf_arena kernel-side fault recovery path. The install runs from a page fault and may have to contend with locks already held by the faulting kernel caller, so keeping it lock-free via cmpxchg is the safe choice.

The generic version in <linux/pgtable.h> returns false. x86 and arm64 override with try_cmpxchg-based implementations on the underlying pteval. Other architectures get the false stub - the callers there already fall through to oops.

---
 arch/arm64/include/asm/pgtable.h |  8 ++++++++
 arch/x86/include/asm/pgtable.h   |  8 ++++++++
 include/linux/pgtable.h          | 16 ++++++++++++++++
 3 files changed, 32 insertions(+)
arch/arm64/include/asm/pgtable.h +8 -0
index 9029b81ccbe8..eb1dd59aaea3 100644
@@ -1830,6 +1830,14 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 	return __ptep_get_and_clear(mm, addr, ptep);
 }
 
+#define __HAVE_ARCH_PTEP_TRY_INSTALL
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	pteval_t old = 0;
+
+	return try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte));
+}
+
 #define test_and_clear_young_ptes test_and_clear_young_ptes
 static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma,
 		unsigned long addr, pte_t *ptep, unsigned int nr)
arch/x86/include/asm/pgtable.h +8 -0
index 13e3e9a054cb..afff54a77ffd 100644
@@ -1284,6 +1284,14 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
 	} while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte));
 }
 
+#define __HAVE_ARCH_PTEP_TRY_INSTALL
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	pte_t old_pte = __pte(0);
+
+	return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte);
+}
+
 #define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0)
 
 #define  __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS
include/linux/pgtable.h +16 -0
index cdd68ed3ae1a..60a4afdc9131 100644
@@ -1036,6 +1036,22 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres
 }
 #endif
 
+#ifndef __HAVE_ARCH_PTEP_TRY_INSTALL
+/**
+ * ptep_try_install - atomically install an empty PTE
+ * @ptep: page table entry
+ * @new_pte: value to install
+ *
+ * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return
+ * true on success. Architectures opt in by providing a cmpxchg-based
+ * override. The generic stub returns false.
+ */
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	return false;
+}
+#endif
+
 #ifndef wrprotect_ptes
 /**
  * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same
Signature · 1 line
2.54.0
#2 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 18 lines · Tejun Heo

Add ptep_try_install(ptep, new_pte): atomically set *ptep to new_pte iff it is currently pte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.

The intended caller is the upcoming bpf_arena kernel-side fault recovery path. The install runs from a page fault and may have to contend with locks already held by the faulting kernel caller, so keeping it lock-free via cmpxchg is the safe choice.

The generic version in <linux/pgtable.h> returns false. x86 and arm64 override with try_cmpxchg-based implementations on the underlying pteval. Other architectures get the false stub - the callers there already fall through to oops.

---

[...]

Quoted 6 lines
include/linux/pgtable.h +5 -0
+#ifndef __HAVE_ARCH_PTEP_TRY_INSTALL
+/**
+ * ptep_try_install - atomically install an empty PTE
+ * @ptep: page table entry
+ * @new_pte: value to install

include/linux/pgtable.h static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · line 1053

Quoted 10 lines
include/linux/pgtable.h +10 -0
+ *
+ * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return
+ * true on success. Architectures opt in by providing a cmpxchg-based
+ * override. The generic stub returns false.
+ */
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	return false;
+}
+#endif

Ehm, what?

This is a very, very, very bad generic idea/interface.

On which ptes is this supposed to be used? User ptes or kernel ptes?

Surely we don't want this on user ptes.

Signature · 3 lines
Cheers,

David
author reviewer
#3 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Hello,

Quoted 9 lines · David Hildenbrand (Arm)

include/linux/pgtable.h static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · line 1053

Quoted 5 lines · Tejun Heo
include/linux/pgtable.h +5 -0
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	return false;
+}
+#endif

Ehm, what?

Quoted 5 lines

This is a very, very, very bad generic idea/interface.

On which ptes is this supposed to be used? User ptes or kernel ptes?

Surely we don't want this on user ptes.

Yeah, this is only for the BPF arena PTEs which are already managed in their own way. I'd be happy to place / gate it however appropriate.

Thanks.

Signature · 1 line
tejun
#4 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 12 lines · Tejun Heo

Hello,

Quoted 9 lines · David Hildenbrand (Arm)

include/linux/pgtable.h static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · line 1053

Quoted 5 lines · Tejun Heo
include/linux/pgtable.h +5 -0
+static inline bool ptep_try_install(pte_t *ptep, pte_t new_pte)
+{
+	return false;
+}
+#endif

Ehm, what?

Quoted 8 lines
Quoted 5 lines

This is a very, very, very bad generic idea/interface.

On which ptes is this supposed to be used? User ptes or kernel ptes?

Surely we don't want this on user ptes.

Yeah, this is only for the BPF arena PTEs which are already managed in their own way. I'd be happy to place / gate it however appropriate.

So, we only use it within apply_to_page_range() with init_mm, where we don't use page table locks.

Usually we seem to grab the &arena->spinlock to protect the kernel page tables.

And you're saying that we might get called from a page fault handler where that lock can already be held.

Is that really possible? I'd much rather prefer to trylock and retry, unless that can really result in deadlocks. But I have the feeling that such deadlocks should be impossible here.

Can you elaborate on the deadlock situation and how that can happen in valid scenarios?

The thing is that ptep_try_install() relies on other page table walking and insertion code to do the right thing. Shaky.

For example, staring at apply_range_set_cb(), what prevents:

(1) apply_range_set_cb() finding pte_none(ptep_get(pte)
(2) apply_range_set_scratch_cb() succeeding ptep_try_install()
(3) apply_range_set_cb() overwriting the pte with set_pte_at()

Between (2) and (3) CPUs could access the scratch PTE.

Signature · 3 lines
Cheers,

David
author reviewer
#5 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Hello, David.

Quoted 3 lines · David Hildenbrand (Arm)

Is that really possible? I'd much rather prefer to trylock and retry, unless that can really result in deadlocks. But I have the feeling that such deadlocks should be impossible here.

I'm not well versed in either mm or BPF, so the BPF folks will have a better take. But here's a scenario that seemed plausible to me:

1. A bpf prog calls bpf_arena_alloc_pages() on its arena. The kernel takes arena->spinlock via raw_res_spin_lock_irqsave().
2. Under the lock, the alloc path goes through bpf_map_alloc_pages() -> alloc_pages_node(), which fires trace_mm_page_alloc().
3. A BPF tracepoint program on mm_page_alloc that shares the arena starts running with the lock still held.
4. The tracepoint program calls a kfunc, passing an arena pointer one entry past the array it meant to touch.
5. The kfunc dereferences. The kernel-side address is unbacked, so the CPU faults.

trylock + retry at 5 would A-A deadlock.

Quoted 7 lines

For example, staring at apply_range_set_cb(), what prevents:

(1) apply_range_set_cb() finding pte_none(ptep_get(pte)
(2) apply_range_set_scratch_cb() succeeding ptep_try_install()
(3) apply_range_set_cb() overwriting the pte with set_pte_at()

Between (2) and (3) CPUs could access the scratch PTE.

Scratch only gets installed when BPF passes an unallocated arena address to the kernel side, which is itself the violation, reported through the program's BPF stream. Behavior at that addr is then undefined. For scx, the scheduler should be aborted and torn down.

The only requirements are that the kernel doesn't oops and the violation gets caught. Beyond that, behavior at the address is unspecified, and which installer wins the race doesn't matter as long as kernel integrity holds.

Thanks.

--
tejun

#6 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 21 lines · Tejun Heo

Hello, David.

Quoted 3 lines · David Hildenbrand (Arm)

Is that really possible? I'd much rather prefer to trylock and retry, unless that can really result in deadlocks. But I have the feeling that such deadlocks should be impossible here.

I'm not well versed in either mm or BPF, so the BPF folks will have a better take. But here's a scenario that seemed plausible to me:

1. A bpf prog calls bpf_arena_alloc_pages() on its arena. The kernel takes arena->spinlock via raw_res_spin_lock_irqsave().
2. Under the lock, the alloc path goes through bpf_map_alloc_pages() -> alloc_pages_node(), which fires trace_mm_page_alloc().
3. A BPF tracepoint program on mm_page_alloc that shares the arena starts running with the lock still held.
4. The tracepoint program calls a kfunc, passing an arena pointer one entry past the array it meant to touch.
5. The kfunc dereferences. The kernel-side address is unbacked, so the CPU faults.

Quoted 1 line

trylock + retry at 5 would A-A deadlock.

Okay, so removing that specific tracepoint (or rather, any tracpoints under the lock) would solve the problem, right?

Quoted 14 lines
Quoted 8 lines

For example, staring at apply_range_set_cb(), what prevents:

(1) apply_range_set_cb() finding pte_none(ptep_get(pte)
(2) apply_range_set_scratch_cb() succeeding ptep_try_install()
(3) apply_range_set_cb() overwriting the pte with set_pte_at()

Between (2) and (3) CPUs could access the scratch PTE.

Scratch only gets installed when BPF passes an unallocated arena address to the kernel side, which is itself the violation, reported through the program's BPF stream. Behavior at that addr is then undefined. For scx, the scheduler should be aborted and torn down.

Quoted 4 lines

The only requirements are that the kernel doesn't oops and the violation gets caught. Beyond that, behavior at the address is unspecified, and which installer wins the race doesn't matter as long as kernel integrity holds.

You'll have inconsistent TLB state.

I really don't like that approach.

We should really try to just take the lock, and remove any code under the lock that could trigger such unpleasant deadlocks.

Is that feasible?

Signature · 3 lines
Cheers,

David
#7 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 46 lines · David Hildenbrand (Arm)
Quoted 21 lines · Tejun Heo

Hello, David.

Quoted 3 lines · David Hildenbrand (Arm)

Is that really possible? I'd much rather prefer to trylock and retry, unless that can really result in deadlocks. But I have the feeling that such deadlocks should be impossible here.

I'm not well versed in either mm or BPF, so the BPF folks will have a better take. But here's a scenario that seemed plausible to me:

1. A bpf prog calls bpf_arena_alloc_pages() on its arena. The kernel takes arena->spinlock via raw_res_spin_lock_irqsave().
2. Under the lock, the alloc path goes through bpf_map_alloc_pages() -> alloc_pages_node(), which fires trace_mm_page_alloc().
3. A BPF tracepoint program on mm_page_alloc that shares the arena starts running with the lock still held.
4. The tracepoint program calls a kfunc, passing an arena pointer one entry past the array it meant to touch.
5. The kfunc dereferences. The kernel-side address is unbacked, so the CPU faults.

Quoted 1 line

trylock + retry at 5 would A-A deadlock.

Okay, so removing that specific tracepoint (or rather, any tracpoints under the lock) would solve the problem, right?

Quoted 14 lines
Quoted 8 lines

For example, staring at apply_range_set_cb(), what prevents:

(1) apply_range_set_cb() finding pte_none(ptep_get(pte)
(2) apply_range_set_scratch_cb() succeeding ptep_try_install()
(3) apply_range_set_cb() overwriting the pte with set_pte_at()

Between (2) and (3) CPUs could access the scratch PTE.

Scratch only gets installed when BPF passes an unallocated arena address to the kernel side, which is itself the violation, reported through the program's BPF stream. Behavior at that addr is then undefined. For scx, the scheduler should be aborted and torn down.

Quoted 4 lines

The only requirements are that the kernel doesn't oops and the violation gets caught. Beyond that, behavior at the address is unspecified, and which installer wins the race doesn't matter as long as kernel integrity holds.

Quoted 9 lines

You'll have inconsistent TLB state.

I really don't like that approach.

We should really try to just take the lock, and remove any code under the lock that could trigger such unpleasant deadlocks.

Is that feasible?

... or can we run into similar problems with kprobes? (I am obviously no bpf expert ...)

Signature · 3 lines
Cheers,

David
author reviewer
#8 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 1 line · David Hildenbrand (Arm)

On 5/19/26 11:05, David Hildenbrand (Arm) wrote:

...

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

The only requirements are that the kernel doesn't oops and the violation gets caught. Beyond that, behavior at the address is unspecified, and which installer wins the race doesn't matter as long as kernel integrity holds.

You'll have inconsistent TLB state.

Wouldn't it still be either or with both cases being okay?

Quoted 9 lines
Quoted 6 lines

I really don't like that approach.

We should really try to just take the lock, and remove any code under the lock that could trigger such unpleasant deadlocks.

Is that feasible?

... or can we run into similar problems with kprobes? (I am obviously no bpf expert ...)

Yeah, I mean, that was just the first TP I found scanning the code. Any kprobes or other TPs in the path would behave the same.

When this fault triggers, the BPF program has already malfunctioned, so it's not going to be a high frequency path and performance isn't a primary consideration. So, anything that can ensure that the kernel doesn't crash or lock up would be fine. Any better ideas?

Thanks.

Signature · 1 line
tejun
#9 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 23 lines · Tejun Heo
Quoted 1 line · David Hildenbrand (Arm)

On 5/19/26 11:05, David Hildenbrand (Arm) wrote:

...

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

The only requirements are that the kernel doesn't oops and the violation gets caught. Beyond that, behavior at the address is unspecified, and which installer wins the race doesn't matter as long as kernel integrity holds.

You'll have inconsistent TLB state.

Wouldn't it still be either or with both cases being okay?

Quoted 9 lines
Quoted 6 lines

I really don't like that approach.

We should really try to just take the lock, and remove any code under the lock that could trigger such unpleasant deadlocks.

Is that feasible?

... or can we run into similar problems with kprobes? (I am obviously no bpf expert ...)

Quoted 7 lines

Yeah, I mean, that was just the first TP I found scanning the code. Any kprobes or other TPs in the path would behave the same.

When this fault triggers, the BPF program has already malfunctioned, so it's not going to be a high frequency path and performance isn't a primary consideration. So, anything that can ensure that the kernel doesn't crash or lock up would be fine. Any better ideas?

As you guys already figured out the trylock is not an option. The fault handler has to install _some_ page and let kernel continue. Scratch page or arena page doesn't matter. Potentially different CPUs will see different page. It's not a concern at all.
bpf prog is buggy, but the kernel will continue to work without a glitch. bpf runtime will disable and unload misbehaving prog.

#10 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 14 lines · Alexei Starovoitov
Quoted 13 lines · Tejun Heo

On Tue, May 19, 2026 at 11:40:48AM +0200, David Hildenbrand (Arm) wrote:
...

Wouldn't it still be either or with both cases being okay?

Quoted 3 lines

... or can we run into similar problems with kprobes? (I am obviously no bpf expert ...)

Yeah, I mean, that was just the first TP I found scanning the code. Any kprobes or other TPs in the path would behave the same.

Quoted 6 lines
Quoted 4 lines

When this fault triggers, the BPF program has already malfunctioned, so it's not going to be a high frequency path and performance isn't a primary consideration. So, anything that can ensure that the kernel doesn't crash or lock up would be fine. Any better ideas?

As you guys already figured out the trylock is not an option.

And that should be carefully documented.

Quoted 5 lines

The fault handler has to install _some_ page and let kernel continue. Scratch page or arena page doesn't matter. Potentially different CPUs will see different page. It's not a concern at all.
bpf prog is buggy, but the kernel will continue to work without a glitch. bpf runtime will disable and unload misbehaving prog.

Having one page table walker overwrite a scratch page on race is just rather ... questionable locking design, that just screams for problems long-term, really.

At least in apply_range_clear_cb() one could similarly switch to ptep_try_install() to at least have both these paths handle races in a reasonable way. (having to handle when ptep_try_install() is not really implemented)

Anyhow, the documentation of ptep_try_install() must clearly spell out that this must be used very carefully, and only in special kernel page tables, never user page tables. There are likely other scenarios we should document (caller must prevent concurrent page table teardown somehow, and must be prepared to handle races if other code is not using atomics).

To highlight that, we should likely consider adding a "kernel" in the name, like "ptep_try_install_kernel()".

I am also not sure if "install" is the right terminology and whether it should instead be "ptep_try_set()". (set_pte_at is the non-atomic interface right now)

Further note that last time I talked to Linus about arch helpers, he preferred

#define ptep_try_install ptep_try_install

over __HAVE_ARCH_PTEP_TRY_INSTALL

Signature · 3 lines
Cheers,

David
#11 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 16 lines · David Hildenbrand (Arm)
Quoted 14 lines · Alexei Starovoitov
Quoted 13 lines · Tejun Heo

On Tue, May 19, 2026 at 11:40:48AM +0200, David Hildenbrand (Arm) wrote:
...

Wouldn't it still be either or with both cases being okay?

Quoted 3 lines

... or can we run into similar problems with kprobes? (I am obviously no bpf expert ...)

Yeah, I mean, that was just the first TP I found scanning the code. Any kprobes or other TPs in the path would behave the same.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

When this fault triggers, the BPF program has already malfunctioned, so it's not going to be a high frequency path and performance isn't a primary consideration. So, anything that can ensure that the kernel doesn't crash or lock up would be fine. Any better ideas?

As you guys already figured out the trylock is not an option.

And that should be carefully documented.

+1

Quoted 4 lines

At least in apply_range_clear_cb() one could similarly switch to ptep_try_install() to at least have both these paths handle races in a reasonable way. (having to handle when ptep_try_install() is not really implemented)

You mean to use ptep_get_and_clear() ?
Makes sense to me.

Also noticed bogus defensive check there that should be removed:
+ if (unlikely(!d))
+ return 0;

Quoted 11 lines

Anyhow, the documentation of ptep_try_install() must clearly spell out that this must be used very carefully, and only in special kernel page tables, never user page tables. There are likely other scenarios we should document (caller must prevent concurrent page table teardown somehow, and must be prepared to handle races if other code is not using atomics).

To highlight that, we should likely consider adding a "kernel" in the name, like "ptep_try_install_kernel()".

I am also not sure if "install" is the right terminology and whether it should instead be "ptep_try_set()". (set_pte_at is the non-atomic interface right now)

I suggested using the ptep_try_set() name too :)

Quoted 5 lines

Further note that last time I talked to Linus about arch helpers, he preferred

#define ptep_try_install ptep_try_install

over __HAVE_ARCH_PTEP_TRY_INSTALL

ok.
I guess __HAVE_ARCH_PTEP_GET_AND_CLEAR is legacy ?

#12 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 7 lines
Quoted 4 lines

At least in apply_range_clear_cb() one could similarly switch to ptep_try_install() to at least have both these paths handle races in a reasonable way. (having to handle when ptep_try_install() is not really implemented)

You mean to use ptep_get_and_clear() ?
Makes sense to me.

Yes, using also an atomic to replace it.

I recall that ptep_get_and_clear() might not be atomic, but I guess it is on the architectures where ptep_try_install() is currently implemented.

Quoted 15 lines
Quoted 7 lines

Anyhow, the documentation of ptep_try_install() must clearly spell out that this must be used very carefully, and only in special kernel page tables, never user page tables. There are likely other scenarios we should document (caller must prevent concurrent page table teardown somehow, and must be prepared to handle races if other code is not using atomics).

Quoted 5 lines

To highlight that, we should likely consider adding a "kernel" in the name, like "ptep_try_install_kernel()".

I am also not sure if "install" is the right terminology and whether it should instead be "ptep_try_set()". (set_pte_at is the non-atomic interface right now)

I suggested using the ptep_try_set() name too :)

Quoted 8 lines
Quoted 5 lines

Further note that last time I talked to Linus about arch helpers, he preferred

#define ptep_try_install ptep_try_install

over __HAVE_ARCH_PTEP_TRY_INSTALL

ok.
I guess __HAVE_ARCH_PTEP_GET_AND_CLEAR is legacy ?

Yeah ... and we have plenty of legacy around :)

Signature · 3 lines
Cheers,

David
author reviewer
#13 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

In reply to David Hildenbrand (Arm)
Quoted 1 line · David Hildenbrand (Arm)

And that should be carefully documented.

Will do.

Quoted 6 lines
Quoted 5 lines

The fault handler has to install _some_ page and let kernel continue. Scratch page or arena page doesn't matter. Potentially different CPUs will see different page. It's not a concern at all.
bpf prog is buggy, but the kernel will continue to work without a glitch. bpf runtime will disable and unload misbehaving prog.

Quoted 6 lines

Having one page table walker overwrite a scratch page on race is just rather ... questionable locking design, that just screams for problems long-term, really.

At least in apply_range_clear_cb() one could similarly switch to ptep_try_install() to at least have both these paths handle races in a reasonable way. (having to handle when ptep_try_install() is not really implemented)

Hmm... wouldn't that be more confusing on apply_range_clear_cb() side? Whether it maintains the current behavior (if collide with scratch, try again with scratch as the original value) or flip the behavior (fail if scratch), that extra logic is spurious, and those tend to confuse people as they force asking why it has to be that specific way. If the goal is documenting the subtlety, wouldn't a detailed comment serve the purpose better?

Quoted 11 lines

Anyhow, the documentation of ptep_try_install() must clearly spell out that this must be used very carefully, and only in special kernel page tables, never user page tables. There are likely other scenarios we should document (caller must prevent concurrent page table teardown somehow, and must be prepared to handle races if other code is not using atomics).

To highlight that, we should likely consider adding a "kernel" in the name, like "ptep_try_install_kernel()".

I am also not sure if "install" is the right terminology and whether it should instead be "ptep_try_set()". (set_pte_at is the non-atomic interface right now)

So, ptep_try_set_kernel()?

Quoted 5 lines

Further note that last time I talked to Linus about arch helpers, he preferred

#define ptep_try_install ptep_try_install

over __HAVE_ARCH_PTEP_TRY_INSTALL

Will do.

Thanks.

Signature · 1 line
tejun
#14 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

Quoted 18 lines · Tejun Heo
Quoted 1 line · David Hildenbrand (Arm)

And that should be carefully documented.

Will do.

Quoted 6 lines
Quoted 5 lines

The fault handler has to install _some_ page and let kernel continue. Scratch page or arena page doesn't matter. Potentially different CPUs will see different page. It's not a concern at all.
bpf prog is buggy, but the kernel will continue to work without a glitch. bpf runtime will disable and unload misbehaving prog.

Quoted 6 lines

Having one page table walker overwrite a scratch page on race is just rather ... questionable locking design, that just screams for problems long-term, really.

At least in apply_range_clear_cb() one could similarly switch to ptep_try_install() to at least have both these paths handle races in a reasonable way. (having to handle when ptep_try_install() is not really implemented)

Quoted 7 lines

Hmm... wouldn't that be more confusing on apply_range_clear_cb() side? Whether it maintains the current behavior (if collide with scratch, try again with scratch as the original value) or flip the behavior (fail if scratch), that extra logic is spurious, and those tend to confuse people as they force asking why it has to be that specific way. If the goal is documenting the subtlety, wouldn't a detailed comment serve the purpose better?

I would let all operations that can happen concurrently work with atomics, not just a single one.

Alexei correctly concluded that ptep_get_and_clear() might be one option that should do on x86 and arm64 from a quick glimpse.

Quoted 7 lines
Quoted 7 lines

Anyhow, the documentation of ptep_try_install() must clearly spell out that this must be used very carefully, and only in special kernel page tables, never user page tables. There are likely other scenarios we should document (caller must prevent concurrent page table teardown somehow, and must be prepared to handle races if other code is not using atomics).

Quoted 7 lines
Quoted 5 lines

To highlight that, we should likely consider adding a "kernel" in the name, like "ptep_try_install_kernel()".

I am also not sure if "install" is the right terminology and whether it should instead be "ptep_try_set()". (set_pte_at is the non-atomic interface right now)

So, ptep_try_set_kernel()?

I guess we could do that, but we don't have a lot of existing functions that are specific to kernel page tables ... only pte_offset_kernel() and pte_alloc_one_kernel()/pte_free_kernel().

The most important part is getting the documentation right.

We should probably document that code concurrently clearing PTEs should be using ptep_get_and_clear().

Signature · 3 lines
Cheers,

David
#15 lore raw

Re: [PATCH 1/8] mm: Add ptep_try_install() for lockless empty-slot installs

In reply to Tejun Heo
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 10 lines · On Sun, 17 May 2026 11:12:25 -1000 you wrote:

Add ptep_try_install(ptep, new_pte): atomically set *ptep to new_pte iff it is currently pte_none(). Returns true on success, false if the slot was already populated or the arch has no implementation.

The intended caller is the upcoming bpf_arena kernel-side fault recovery path. The install runs from a page fault and may have to contend with locks already held by the faulting kernel caller, so keeping it lock-free via cmpxchg is the safe choice.

[...]

Here is the summary with links:
- [1/8] mm: Add ptep_try_install() for lockless empty-slot installs (no matching commit)
- [2/8] bpf: Recover arena kernel faults with scratch page
(no matching commit)
- [3/8] bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers

    https://git.kernel.org/bpf/bpf-next/c/f211c81ddc36
  - [4/8] bpf: Add bpf_struct_ops_for_each_prog()
    https://git.kernel.org/bpf/bpf-next/c/7c48a28c1bbe
  - [5/8] bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena()
    https://git.kernel.org/bpf/bpf-next/c/53cc12a2dc88

- [6/8] sched_ext: Require an arena for cid-form schedulers (no matching commit)
- [7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages (no matching commit)
- [8/8] sched_ext: Convert ops.set_cmask() to arena-resident cmask (no matching commit)

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
author reviewer
#16 lore raw

PATCH 2/8 bpf: Recover arena kernel faults with scratch page

In reply to Tejun Heo

From: Kumar Kartikeya Dwivedi <[email protected]>

BPF arena usage is becoming more prevalent, but kernel <-> BPF communication over arena memory is awkward today. Data has to be staged through a trusted kernel pointer with extra code and copying on the BPF side. While reads through arena pointers can use a fault-safe helper, writes don't have a good solution; the in-line alternative would need instruction emulation or asm fixup labels.

Enable direct kernel-side reads and writes within GUARD_SZ / 2 of any handed-in arena pointer, without bounds checking. A per-arena scratch page is installed by the arch fault path into empty arena kernel PTEs - x86 from page_fault_oops() for not-present faults, arm64 from __do_kernel_fault() for translation faults, both after the existing exception-table and KFENCE handling. The faulting instruction retries and the access is also reported through the program's BPF stream, preserving error reporting.

bpf_prog_find_from_stack() resolves the current BPF program (and its arena) from the kernel stack - no new bpf_run_ctx state is added. Recovery covers the 4 GiB arena plus the upper half-guard (GUARD_SZ / 2); the lower half-guard is excluded because well-behaved kfuncs only access forward from arena pointers. The kfunc-author contract - access at most GUARD_SZ / 2 past a handed-in pointer - is documented in Documentation/bpf/kfuncs.rst.

The install is lock-free via ptep_try_install(): on race-loss the winning installer's PTE is already valid, so the access retry succeeds. No flush_tlb_kernel_range() afterwards - stale "not mapped" entries just cause one extra re-fault, cheaper than a global IPI on every install.

Scratch exists only to keep the kernel from oopsing on an in-line arena access. Its presence at a PTE means the BPF program has already malfunctioned, and the violation is reported through the program's BPF stream. The only requirement for behavior on a scratched PTE is that the kernel doesn't crash; in particular, any user-side access through such a PTE may segfault. The shared scratch page is freed once during map destruction.

BPF instruction faults continue to use the existing JIT exception-table path; this patch changes only the kernel-text fault path. No UAPI flag is added; the new behavior is the default.

---
 Documentation/bpf/kfuncs.rst |  14 +++
 arch/arm64/mm/fault.c        |  10 ++-
 arch/x86/mm/fault.c          |  12 ++-
 include/linux/bpf.h          |   1 +
 include/linux/bpf_defs.h     |  11 +++
 kernel/bpf/arena.c           | 170 +++++++++++++++++++++++++++--------
 kernel/bpf/core.c            |   5 ++
 7 files changed, 181 insertions(+), 42 deletions(-)
 create mode 100644 include/linux/bpf_defs.h
Documentation/bpf/kfuncs.rst +14 -0
index 75e6c078e0e7..6d497e720998 100644
@@ -462,6 +462,20 @@ In order to accommodate such requirements, the verifier will enforce strict
 PTR_TO_BTF_ID type matching if two types have the exact same name, with one
 being suffixed with ``___init``.
 
+2.8 Accessing arena memory through kfunc arguments
+--------------------------------------------------
+
+A read or write at any address inside an arena does not oops the kernel.
+Unallocated arena pages are lazily backed by a scratch page and the
+access is reported through the program's BPF stream as an error. Only
+the BPF program's correctness is affected; the kernel itself remains
+intact.
+
+The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that
+is also covered by this recovery. A kfunc handed an arena pointer may
+therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
+against the arena. Larger accesses must verify the range explicitly.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
arch/arm64/mm/fault.c +7 -3
index 920a8b244d59..0d58d667fcd8 100644
@@ -9,6 +9,7 @@
 
 #include <linux/acpi.h>
 #include <linux/bitfield.h>
+#include <linux/bpf_defs.h>
 #include <linux/extable.h>
 #include <linux/kfence.h>
 #include <linux/signal.h>
@@ -416,9 +417,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr,
 	} else if (addr < PAGE_SIZE) {
 		msg = "NULL pointer dereference";
 	} else {
-		if (esr_fsc_is_translation_fault(esr) &&
-		    kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
-			return;
+		if (esr_fsc_is_translation_fault(esr)) {
+			if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
+				return;
+			if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc))
+				return;
+		}
 
 		msg = "paging request";
 	}
arch/x86/mm/fault.c +8 -4
index f0e77e084482..b0f103ddbd23 100644
@@ -8,6 +8,7 @@
 #include <linux/sched/task_stack.h>	/* task_stack_*(), ...		*/
 #include <linux/kdebug.h>		/* oops_begin/end, ...		*/
 #include <linux/memblock.h>		/* max_low_pfn			*/
+#include <linux/bpf_defs.h>		/* bpf_arena_handle_page_fault	*/
 #include <linux/kfence.h>		/* kfence_handle_page_fault	*/
 #include <linux/kprobes.h>		/* NOKPROBE_SYMBOL, ...		*/
 #include <linux/mmiotrace.h>		/* kmmio_handler, ...		*/
@@ -688,10 +689,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code,
 	if (IS_ENABLED(CONFIG_EFI))
 		efi_crash_gracefully_on_page_fault(address);
 
-	/* Only not-present faults should be handled by KFENCE. */
-	if (!(error_code & X86_PF_PROT) &&
-	    kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
-		return;
+	/* Only not-present faults should be handled by KFENCE or BPF arena. */
+	if (!(error_code & X86_PF_PROT)) {
+		if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
+			return;
+		if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip))
+			return;
+	}
 
 oops:
 	/*
include/linux/bpf.h +1 -0
index 0136a108d083..831996c411cf 100644
@@ -6,6 +6,7 @@
 
 #include <uapi/linux/bpf.h>
 #include <uapi/linux/filter.h>
+#include <linux/bpf_defs.h>
 
 #include <crypto/sha2.h>
 #include <linux/workqueue.h>
include/linux/bpf_defs.h new file +11 -0
new file mode 100644
index 000000000000..d98e033b8c0b
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Subset of bpf.h declarations, split out so files that need only these
+ * declarations can avoid bpf.h's full include cost.
+ */
+#ifndef _LINUX_BPF_DEFS_H
+#define _LINUX_BPF_DEFS_H
+
+bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip);
+
+#endif /* _LINUX_BPF_DEFS_H */
kernel/bpf/arena.c +135 -35

316 lines not on this page · raw message

kernel/bpf/core.c +5 -0
index 066b86e7233c..fa368d8920d9 100644
@@ -3290,6 +3290,11 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)
 {
 	return 0;
 }
+__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
+					unsigned long fault_ip)
+{
+	return false;
+}
 
 #ifdef CONFIG_BPF_SYSCALL
 static int __init bpf_global_ma_init(void)
Signature · 1 line
2.54.0
author reviewer
#17 lore raw

PATCH 3/8 bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers

In reply to Tejun Heo

The existing kernel-side export of bpf_arena_alloc_pages is _non_sleepable only - it's used by the verifier to inline the kfunc when the call site is non-sleepable. There is no sleepable equivalent for kernel callers; the kfunc bpf_arena_alloc_pages itself is BPF-only.

sched_ext needs sleepable kernel-side allocs for its arena pool init/grow paths. Add bpf_arena_alloc_pages_sleepable() mirroring the _non_sleepable wrapper but passing sleepable=true to arena_alloc_pages().

---
 include/linux/bpf.h |  8 ++++++++
 kernel/bpf/arena.c  | 13 +++++++++++++
 2 files changed, 21 insertions(+)
include/linux/bpf.h +8 -0
index 831996c411cf..64968ca6db51 100644
@@ -679,6 +679,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
 void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
 					  u64 flags);
 void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
+void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
+				      u64 flags);
 #else
 static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
 							int node_id, u64 flags)
@@ -689,6 +691,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr
 static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
 {
 }
+
+static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
+						    int node_id, u64 flags)
+{
+	return NULL;
+}
 #endif
 
 extern const struct bpf_map_ops bpf_map_offload_ops;
kernel/bpf/arena.c +13 -0
index bc696bef7104..8eb7b95f4999 100644
@@ -937,6 +937,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag
 
 	return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);
 }
+
+void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
+				      int node_id, u64 flags)
+{
+	struct bpf_map *map = p__map;
+	struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
+
+	if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)
+		return NULL;
+
+	return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true);
+}
+
 __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)
 {
 	struct bpf_map *map = p__map;
Signature · 1 line
2.54.0
author reviewer
#18 lore raw

PATCH 4/8 bpf: Add bpf_struct_ops_for_each_prog()

In reply to Tejun Heo

Add a helper that walks the member progs of the struct_ops map containing a given @kdata vmtable. struct_ops ->reg() callbacks (and similar) sometimes need to inspect the loaded BPF programs, e.g. to discover maps they reference via prog->aux->used_maps.

The implementation mirrors bpf_struct_ops_id(): container_of @kdata to recover the bpf_struct_ops_map, then iterate st_map->links[i]->prog for i in [0, funcs_cnt). Same access pattern, no new locking - by the time ->reg() fires st_map is fully populated and stable.

A sched_ext follow-up walks the member progs of a cid-form scheduler's struct_ops map, reads prog->aux->arena directly, and requires all member progs to reference exactly one arena, without requiring the BPF program to call a registration kfunc.

---
 include/linux/bpf.h         |  3 +++
 kernel/bpf/bpf_struct_ops.c | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/linux/bpf.h +3 -0
index 64968ca6db51..5b99d786e98c 100644
@@ -2129,6 +2129,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);
 void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
 void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
 u32 bpf_struct_ops_id(const void *kdata);
+int bpf_struct_ops_for_each_prog(const void *kdata,
+				 int (*cb)(struct bpf_prog *prog, void *data),
+				 void *data);
 
 #ifdef CONFIG_NET
 /* Define it here to avoid the use of forward declaration */
kernel/bpf/bpf_struct_ops.c +36 -0
index 05b366b821c3..16aec18ed31b 100644
@@ -1203,6 +1203,42 @@ u32 bpf_struct_ops_id(const void *kdata)
 }
 EXPORT_SYMBOL_GPL(bpf_struct_ops_id);
 
+/**
+ * bpf_struct_ops_for_each_prog - Invoke @cb for each member prog
+ * @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg)
+ * @cb: callback invoked once per member prog; non-zero return stops iteration
+ * @data: opaque argument passed to @cb
+ *
+ * Walks the struct_ops member progs registered on the map containing @kdata.
+ * Intended for use from struct_ops ->reg() callbacks (and similar) that need to
+ * inspect the loaded BPF programs (for example to discover maps they reference
+ * via @prog->aux->used_maps).
+ *
+ * Return 0 if iteration completed, otherwise the first non-zero @cb return.
+ */
+int bpf_struct_ops_for_each_prog(const void *kdata,
+				 int (*cb)(struct bpf_prog *prog, void *data),
+				 void *data)
+{
+	struct bpf_struct_ops_value *kvalue;
+	struct bpf_struct_ops_map *st_map;
+	u32 i;
+	int ret;
+
+	kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
+	st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
+
+	for (i = 0; i < st_map->funcs_cnt; i++) {
+		if (!st_map->links[i])
+			continue;
+		ret = cb(st_map->links[i]->prog, data);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog);
+
 static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
 {
 	struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
Signature · 1 line
2.54.0
author reviewer
#19 lore raw

PATCH 5/8 bpf/arena: Add bpf_arena_map_kern_vm_start() and bpf_prog_arena()

In reply to Tejun Heo

struct bpf_arena is opaque to callers outside arena.c. Add two helpers for struct_ops subsystems that need to reach into an arena:

bpf_arena_map_kern_vm_start(struct bpf_map *map)
returns @map's kern_vm_start. A sched_ext follow-up needs this to translate kern_va <-> uaddr.

bpf_prog_arena(struct bpf_prog *prog)
returns the bpf_map of the arena referenced by @prog (NULL if @prog references no arena). The verifier enforces at most one arena per program. Used by struct_ops callers that auto-discover an arena from a member prog and need to take a map reference.

---
 include/linux/bpf.h |  2 ++
 kernel/bpf/arena.c  | 26 ++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
include/linux/bpf.h +2 -0
index 5b99d786e98c..e1ba57c10aaa 100644
@@ -618,6 +618,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
 		      struct bpf_spin_lock *spin_lock);
 u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
 u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
+u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
+struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
 int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
 
 struct bpf_offload_dev;
kernel/bpf/arena.c +26 -0
index 8eb7b95f4999..5a82d39e1916 100644
@@ -84,6 +84,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena)
 	return arena ? arena->user_vm_start : 0;
 }
 
+/**
+ * bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map *
+ * @map: a BPF_MAP_TYPE_ARENA map
+ *
+ * Return @map's kern_vm_start.
+ */
+u64 bpf_arena_map_kern_vm_start(struct bpf_map *map)
+{
+	return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map));
+}
+
+/**
+ * bpf_prog_arena - return the bpf_map of the arena referenced by @prog
+ * @prog: a loaded BPF program
+ *
+ * The verifier enforces at most one arena per program and stores it in
+ * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if
+ * @prog does not reference an arena.
+ */
+struct bpf_map *bpf_prog_arena(struct bpf_prog *prog)
+{
+	struct bpf_arena *arena = prog->aux->arena;
+
+	return arena ? &arena->map : NULL;
+}
+
 static long arena_map_peek_elem(struct bpf_map *map, void *value)
 {
 	return -EOPNOTSUPP;
Signature · 1 line
2.54.0
author reviewer
#20 lore raw

PATCH 6/8 sched_ext: Require an arena for cid-form schedulers

In reply to Tejun Heo

Upcoming patches will let the kernel place arena-resident scratch shared with the BPF program (e.g. per-CPU set_cmask cmask) so the BPF side can dereference it directly via __arena pointers, replacing the current cmask_copy_from_kernel() probe-read loop. That requires each cid-form scheduler to expose its arena to the kernel. Kernel- side accesses are recovered by the per-arena scratch-page mechanism.

bpf_scx_reg_cid() walks the struct_ops member progs via bpf_struct_ops_for_each_prog() and reads each prog's arena via bpf_prog_arena(). The verifier enforces one arena per program, so each member prog contributes at most one arena. All non-NULL contributions must match and at least one member prog must use an arena. The map ref is held on scx_sched and dropped on sched destroy. cpu-form schedulers (bpf_scx_reg) are unchanged - no arena requirement.

---
 kernel/sched/ext.c          | 56 ++++++++++++++++++++++++++++++++++++-
 kernel/sched/ext_internal.h |  8 ++++++
 2 files changed, 63 insertions(+), 1 deletion(-)
kernel/sched/ext.c +55 -1
index 64f8a096f133..94aab7037329 100644
@@ -5009,6 +5009,8 @@ static void scx_sched_free_rcu_work(struct work_struct *work)
 
 	rhashtable_free_and_destroy(&sch->dsq_hash, NULL, NULL);
 	free_exit_info(sch->exit_info);
+	if (sch->arena_map)
+		bpf_map_put(sch->arena_map);
 	kfree(sch);
 }
 
@@ -6732,6 +6734,7 @@ struct scx_enable_cmd {
 		struct sched_ext_ops_cid	*ops_cid;
 	};
 	bool			is_cid_type;
+	struct bpf_map		*arena_map;	/* arena ref to transfer to sch */
 	int			ret;
 };
 
@@ -6898,6 +6901,15 @@ static struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,
 		return ERR_PTR(ret);
 	}
 #endif	/* CONFIG_EXT_SUB_SCHED */
+
+	/*
+	 * Consume the arena_map ref bpf_scx_reg_cid() took. Defer to here so
+	 * earlier failure paths leave cmd->arena_map set and bpf_scx_reg_cid
+	 * drops the ref. After this point, sch owns the ref and any cleanup
+	 * runs through scx_sched_free_rcu_work() which puts it.
+	 */
+	sch->arena_map = cmd->arena_map;
+	cmd->arena_map = NULL;
 	return sch;
 
 #ifdef CONFIG_EXT_SUB_SCHED
@@ -7884,11 +7896,53 @@ static int bpf_scx_reg(void *kdata, struct bpf_link *link)
 	return scx_enable(&cmd, link);
 }
 
+struct scx_arena_scan {
+	struct bpf_map	*arena;
+	int		err;
+};
+
+/*
+ * The verifier enforces one arena per BPF program, so each struct_ops
+ * member prog contributes at most one arena via bpf_prog_arena().
+ * Require all non-NULL contributions to match.
+ */
+static int scx_arena_scan_prog(struct bpf_prog *prog, void *data)
+{
+	struct scx_arena_scan *s = data;
+	struct bpf_map *arena = bpf_prog_arena(prog);
+
+	if (!arena)
+		return 0;
+	if (s->arena && s->arena != arena) {
+		s->err = -EINVAL;
+		return 1;
+	}
+	s->arena = arena;
+	return 0;
+}
+
 static int bpf_scx_reg_cid(void *kdata, struct bpf_link *link)
 {
 	struct scx_enable_cmd cmd = { .ops_cid = kdata, .is_cid_type = true };
+	struct scx_arena_scan scan = {};
+	int ret;
 
-	return scx_enable(&cmd, link);
+	bpf_struct_ops_for_each_prog(kdata, scx_arena_scan_prog, &scan);
+	if (scan.err) {
+		pr_err("sched_ext: cid-form scheduler uses multiple arena maps\n");
+		return scan.err;
+	}
+	if (!scan.arena) {
+		pr_err("sched_ext: cid-form scheduler must use a BPF arena map\n");
+		return -EINVAL;
+	}
+
+	bpf_map_inc(scan.arena);
+	cmd.arena_map = scan.arena;
+	ret = scx_enable(&cmd, link);
+	if (cmd.arena_map)		/* not consumed by scx_alloc_and_add_sched() */
+		bpf_map_put(cmd.arena_map);
+	return ret;
 }
 
 static void bpf_scx_unreg(void *kdata, struct bpf_link *link)
kernel/sched/ext_internal.h +8 -0
index 7258aea94b9f..d40cfd29ddaa 100644
@@ -1111,6 +1111,14 @@ struct scx_sched {
 		struct sched_ext_ops_cid	ops_cid;
 	};
 	bool			is_cid_type;	/* true if registered via bpf_sched_ext_ops_cid */
+
+	/*
+	 * Arena map auto-discovered from member progs at struct_ops attach.
+	 * cid-form schedulers must use exactly one arena across all member
+	 * progs. NULL on cpu-form.
+	 */
+	struct bpf_map		*arena_map;
+
 	DECLARE_BITMAP(has_op, SCX_OPI_END);
 
 	/*
Signature · 1 line
2.54.0
author reviewer
#21 lore raw

PATCH 7/8 sched_ext: Sub-allocator over kernel-claimed BPF arena pages

In reply to Tejun Heo

Build a per-scheduler sub-allocator on top of pages claimed from the BPF arena registered in the previous patch. Subsequent kernel-managed arena-resident structures (e.g. per-CPU set_cmask cmask) carve their storage from this pool.

scx_arena_pool_init() creates a gen_pool. scx_arena_alloc() returns the kernel VA. On exhaustion, the pool grows by claiming more pages via bpf_arena_alloc_pages_sleepable(). Chunks are added at the kernel-side mapping address; callers translate to the BPF-arena form themselves if needed.

Allocations sleep (GFP_KERNEL) - they may grow the pool through vzalloc and arena page allocation. All current consumers run from the enable path (after ops.init() and the kernel-side arena auto-discovery, before validate_ops()), where sleeping is fine.

scx_arena_pool_destroy() walks each chunk, returns outstanding ranges to the gen_pool with gen_pool_free() and then calls gen_pool_destroy(). The underlying arena pages are released when the arena map itself is torn down, so the pool destroy doesn't free them explicitly.

---
 kernel/sched/build_policy.c |   4 ++
 kernel/sched/ext.c          |  11 ++++
 kernel/sched/ext_arena.c    | 127 ++++++++++++++++++++++++++++++++++++
 kernel/sched/ext_arena.h    |  18 +++++
 kernel/sched/ext_internal.h |   5 ++
 5 files changed, 165 insertions(+)
 create mode 100644 kernel/sched/ext_arena.c
 create mode 100644 kernel/sched/ext_arena.h
kernel/sched/build_policy.c +4 -0
index 5e76c9177d54..067979a7b69e 100644
@@ -59,12 +59,16 @@
 
 #ifdef CONFIG_SCHED_CLASS_EXT
 # include <linux/btf_ids.h>
+# include <linux/find.h>
+# include <linux/genalloc.h>
 # include "ext_types.h"
 # include "ext_internal.h"
 # include "ext_cid.h"
+# include "ext_arena.h"
 # include "ext_idle.h"
 # include "ext.c"
 # include "ext_cid.c"
+# include "ext_arena.c"
 # include "ext_idle.c"
 #endif
 
kernel/sched/ext.c +11 -0
index 94aab7037329..3025fbe198d3 100644
@@ -5009,6 +5009,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)
 
 	rhashtable_free_and_destroy(&sch->dsq_hash, NULL, NULL);
 	free_exit_info(sch->exit_info);
+	scx_arena_pool_destroy(sch);
 	if (sch->arena_map)
 		bpf_map_put(sch->arena_map);
 	kfree(sch);
@@ -7140,6 +7141,12 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		sch->exit_info->flags |= SCX_EFLAG_INITIALIZED;
 	}
 
+	ret = scx_arena_pool_init(sch);
+	if (ret) {
+		cpus_read_unlock();
+		goto err_disable;
+	}
+
 	for (i = SCX_OPI_CPU_HOTPLUG_BEGIN; i < SCX_OPI_CPU_HOTPLUG_END; i++)
 		if (((void (**)(void))ops)[i])
 			set_bit(i, sch->has_op);
@@ -7458,6 +7465,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work)
 		sch->exit_info->flags |= SCX_EFLAG_INITIALIZED;
 	}
 
+	ret = scx_arena_pool_init(sch);
+	if (ret)
+		goto err_disable;
+
 	if (validate_ops(sch, ops))
 		goto err_disable;
 
kernel/sched/ext_arena.c new file +127 -0

127 lines not on this page · raw message

kernel/sched/ext_arena.h new file +18 -0
new file mode 100644
index 000000000000..4f3610160102
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst
+ *
+ * Copyright (c) 2025 Meta Platforms, Inc. and affiliates.
+ * Copyright (c) 2025 Tejun Heo <[email protected]>
+ */
+#ifndef _KERNEL_SCHED_EXT_ARENA_H
+#define _KERNEL_SCHED_EXT_ARENA_H
+
+struct scx_sched;
+
+s32 scx_arena_pool_init(struct scx_sched *sch);
+void scx_arena_pool_destroy(struct scx_sched *sch);
+void *scx_arena_alloc(struct scx_sched *sch, size_t size);
+void scx_arena_free(struct scx_sched *sch, void *kern_va, size_t size);
+
+#endif /* _KERNEL_SCHED_EXT_ARENA_H */
kernel/sched/ext_internal.h +5 -0
index d40cfd29ddaa..ff7e882bd67a 100644
@@ -1116,8 +1116,13 @@ struct scx_sched {
 	 * Arena map auto-discovered from member progs at struct_ops attach.
 	 * cid-form schedulers must use exactly one arena across all member
 	 * progs. NULL on cpu-form.
+	 *
+	 * @arena_pool sub-allocates @arena_map. Each gen_pool chunk is added
+	 * at the kernel-side mapping address. Grows on demand and pages are
+	 * not released until sched destroy.
 	 */
 	struct bpf_map		*arena_map;
+	struct gen_pool		*arena_pool;
 
 	DECLARE_BITMAP(has_op, SCX_OPI_END);
 
Signature · 1 line
2.54.0
maintainer
#22 lore raw

Re: [PATCH 7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages

Quoted 16 lines · Tejun Heo

Build a per-scheduler sub-allocator on top of pages claimed from the BPF arena registered in the previous patch. Subsequent kernel-managed arena-resident structures (e.g. per-CPU set_cmask cmask) carve their storage from this pool.

scx_arena_pool_init() creates a gen_pool. scx_arena_alloc() returns the kernel VA. On exhaustion, the pool grows by claiming more pages via bpf_arena_alloc_pages_sleepable(). Chunks are added at the kernel-side mapping address; callers translate to the BPF-arena form themselves if needed.

Allocations sleep (GFP_KERNEL) - they may grow the pool through vzalloc and arena page allocation. All current consumers run from the enable path (after ops.init() and the kernel-side arena auto-discovery, before validate_ops()), where sleeping is fine.

Quoted 4 lines

scx_arena_pool_destroy() walks each chunk, returns outstanding ranges to the gen_pool with gen_pool_free() and then calls gen_pool_destroy(). The underlying arena pages are released when the arena map itself is torn down, so the pool destroy doesn't free them explicitly.

Should this really be part of scx rather than be part of the bpf-arena thing proper?

author reviewer
#23 lore raw

Re: [PATCH 7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages

Hello,

On Mon, May 18, 2026 at 09:20:42AM +0200, Peter Zijlstra wrote:
...

Quoted 2 lines

Should this really be part of scx rather than be part of the bpf-arena thing proper?

It's just a layer on top of arena. If bpf folks are okay with it, I don't see why it can't be a common utility thing on the bpf side.

Thanks.

Signature · 1 line
tejun
#24 lore raw

Re: [PATCH 7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages

Quoted 9 lines · Tejun Heo

Hello,

On Mon, May 18, 2026 at 09:20:42AM +0200, Peter Zijlstra wrote:
...

Quoted 2 lines

Should this really be part of scx rather than be part of the bpf-arena thing proper?

It's just a layer on top of arena. If bpf folks are okay with it, I don't see why it can't be a common utility thing on the bpf side.

Well, this gen_pool based allocator of arena memory is a temporary hack. It's ok for rare allocation like in this at scx init time, but not suitable for active arena management. We don't need to expose it beyond scx. Having said that the fast and generic allocator for arena is definitely needed. This break through with scratch page and bpf_arena_handle_page_fault() cannot be overstated. It is a huge improvement for kernel <-> bpf interaction. Not only kfuncs can now read arena without ugly __get_kernel_nofault(), but we can reuse mm/slub.c to manage arena memory!
The key idea is simply this:
get_freepointer() {
if (s->flags & SLAB_BPF_ARENA)

    return (void *)(s->arena_kern_vm_start | (u32)(unsigned long)ptr);
}

I'm sloping a prototype.

author reviewer
#25 lore raw

Re: [PATCH 7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages

Hello,

Quoted 3 lines · Alexei Starovoitov

Well, this gen_pool based allocator of arena memory is a temporary hack. It's ok for rare allocation like in this at scx init time, but not suitable for active arena management. We don't need to expose it beyond scx.

I see. Peter, as Alexei is already prototyping a slab-based arena allocator, how about keeping the gen_pool layer scx-local for now? Once the proper allocator lands, scx can switch to it and the custom piece goes away.

Thanks.

--
tejun

maintainer
#26 lore raw

Re: [PATCH 7/8] sched_ext: Sub-allocator over kernel-claimed BPF arena pages

Quoted 11 lines · Tejun Heo

Hello,

Quoted 3 lines · Alexei Starovoitov

Well, this gen_pool based allocator of arena memory is a temporary hack. It's ok for rare allocation like in this at scx init time, but not suitable for active arena management. We don't need to expose it beyond scx.

I see. Peter, as Alexei is already prototyping a slab-based arena allocator, how about keeping the gen_pool layer scx-local for now? Once the proper allocator lands, scx can switch to it and the custom piece goes away.

OK.

author reviewer
#27 lore raw

PATCH 8/8 sched_ext: Convert ops.set_cmask() to arena-resident cmask

In reply to Tejun Heo

ops_cid.set_cmask() expects a cmask. The kernel couldn't write into the arena, so it translated cpumask -> cmask in kernel memory and passed the result as a trusted pointer. The BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.

With direct kernel-side arena access now in place, build the cmask in the arena. The kernel writes to it through the kern_va side of the dual mapping; BPF directly dereferences it via an __arena pointer like any other arena struct.

---
 kernel/sched/ext.c                    | 68 +++++++++++++++++++++++++--
 kernel/sched/ext_cid.c                | 20 +-------
 kernel/sched/ext_internal.h           | 10 +++-
 tools/sched_ext/include/scx/cid.bpf.h | 52 --------------------
 tools/sched_ext/scx_qmap.bpf.c        |  5 +-
 5 files changed, 75 insertions(+), 80 deletions(-)
kernel/sched/ext.c +63 -5
index 3025fbe198d3..1369dc7e4b4e 100644
@@ -621,11 +621,16 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,
 		update_locked_rq(rq);
 
 	if (scx_is_cid_type()) {
-		struct scx_cmask *cmask = this_cpu_ptr(scx_set_cmask_scratch);
-
-		lockdep_assert_irqs_disabled();
-		scx_cpumask_to_cmask(cpumask, cmask);
-		sch->ops_cid.set_cmask(task, cmask);
+		struct scx_cmask *kern_va = *this_cpu_ptr(sch->set_cmask_scratch);
+		unsigned long uaddr = (unsigned long)kern_va -
+			bpf_arena_map_kern_vm_start(sch->arena_map);
+		/*
+		 * Build the per-CPU arena cmask and hand BPF the uaddr. Caller
+		 * holds the rq lock with IRQs disabled, which makes us the sole
+		 * user of the scratch area.
+		 */
+		scx_cpumask_to_cmask(cpumask, kern_va);
+		sch->ops_cid.set_cmask(task, (struct scx_cmask *)uaddr);
 	} else {
 		sch->ops.set_cpumask(task, cpumask);
 	}
@@ -4957,6 +4962,48 @@ static const struct attribute_group scx_global_attr_group = {
 static void free_pnode(struct scx_sched_pnode *pnode);
 static void free_exit_info(struct scx_exit_info *ei);
 
+static s32 scx_set_cmask_scratch_alloc(struct scx_sched *sch)
+{
+	size_t size = struct_size_t(struct scx_cmask, bits,
+				    SCX_CMASK_NR_WORDS(num_possible_cpus()));
+	int cpu;
+
+	if (!sch->is_cid_type || !sch->arena_pool)
+		return 0;
+
+	sch->set_cmask_scratch = alloc_percpu(struct scx_cmask *);
+	if (!sch->set_cmask_scratch)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu) {
+		struct scx_cmask **slot = per_cpu_ptr(sch->set_cmask_scratch, cpu);
+
+		*slot = scx_arena_alloc(sch, size);
+		if (!*slot)
+			return -ENOMEM;
+		scx_cmask_init(*slot, 0, num_possible_cpus());
+	}
+	return 0;
+}
+
+static void scx_set_cmask_scratch_free(struct scx_sched *sch)
+{
+	size_t size = struct_size_t(struct scx_cmask, bits,
+				    SCX_CMASK_NR_WORDS(num_possible_cpus()));
+	int cpu;
+
+	if (!sch->set_cmask_scratch)
+		return;
+
+	for_each_possible_cpu(cpu) {
+		struct scx_cmask **slot = per_cpu_ptr(sch->set_cmask_scratch, cpu);
+
+		scx_arena_free(sch, *slot, size);
+	}
+	free_percpu(sch->set_cmask_scratch);
+	sch->set_cmask_scratch = NULL;
+}
+
 static void scx_sched_free_rcu_work(struct work_struct *work)
 {
 	struct rcu_work *rcu_work = to_rcu_work(work);
@@ -5009,6 +5056,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)
 
 	rhashtable_free_and_destroy(&sch->dsq_hash, NULL, NULL);
 	free_exit_info(sch->exit_info);
+	scx_set_cmask_scratch_free(sch);
 	scx_arena_pool_destroy(sch);
 	if (sch->arena_map)
 		bpf_map_put(sch->arena_map);
@@ -7147,6 +7195,12 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		goto err_disable;
 	}
 
+	ret = scx_set_cmask_scratch_alloc(sch);
+	if (ret) {
+		cpus_read_unlock();
+		goto err_disable;
+	}
+
 	for (i = SCX_OPI_CPU_HOTPLUG_BEGIN; i < SCX_OPI_CPU_HOTPLUG_END; i++)
 		if (((void (**)(void))ops)[i])
 			set_bit(i, sch->has_op);
@@ -7469,6 +7523,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work)
 	if (ret)
 		goto err_disable;
 
+	ret = scx_set_cmask_scratch_alloc(sch);
+	if (ret)
+		goto err_disable;
+
 	if (validate_ops(sch, ops))
 		goto err_disable;
 
kernel/sched/ext_cid.c +1 -19
index 5cd14143f88f..245a39e2e5eb 100644
@@ -7,14 +7,6 @@
  */
 #include <linux/cacheinfo.h>
 
-/*
- * Per-cpu scratch cmask used by scx_call_op_set_cpumask() to synthesize a
- * cmask from a cpumask. Allocated alongside the cid arrays on first enable
- * and never freed. Sized to the full cid space. Caller holds rq lock so
- * this_cpu_ptr is safe.
- */
-struct scx_cmask __percpu *scx_set_cmask_scratch;
-
 /*
  * cid tables.
  *
@@ -54,8 +46,6 @@ static s32 scx_cid_arrays_alloc(void)
 	u32 npossible = num_possible_cpus();
 	s16 *cid_to_cpu, *cpu_to_cid;
 	struct scx_cid_topo *cid_topo;
-	struct scx_cmask __percpu *set_cmask_scratch;
-	s32 cpu;
 
 	if (scx_cid_to_cpu_tbl)
 		return 0;
@@ -63,25 +53,17 @@ static s32 scx_cid_arrays_alloc(void)
 	cid_to_cpu = kzalloc_objs(*scx_cid_to_cpu_tbl, npossible, GFP_KERNEL);
 	cpu_to_cid = kzalloc_objs(*scx_cpu_to_cid_tbl, nr_cpu_ids, GFP_KERNEL);
 	cid_topo = kmalloc_objs(*scx_cid_topo, npossible, GFP_KERNEL);
-	set_cmask_scratch = __alloc_percpu(struct_size(set_cmask_scratch, bits,
-						       SCX_CMASK_NR_WORDS(npossible)),
-					   sizeof(u64));
 
-	if (!cid_to_cpu || !cpu_to_cid || !cid_topo || !set_cmask_scratch) {
+	if (!cid_to_cpu || !cpu_to_cid || !cid_topo) {
 		kfree(cid_to_cpu);
 		kfree(cpu_to_cid);
 		kfree(cid_topo);
-		free_percpu(set_cmask_scratch);
 		return -ENOMEM;
 	}
 
 	WRITE_ONCE(scx_cid_to_cpu_tbl, cid_to_cpu);
 	WRITE_ONCE(scx_cpu_to_cid_tbl, cpu_to_cid);
 	WRITE_ONCE(scx_cid_topo, cid_topo);
-	for_each_possible_cpu(cpu)
-		scx_cmask_init(per_cpu_ptr(set_cmask_scratch, cpu),
-			       0, npossible);
-	WRITE_ONCE(scx_set_cmask_scratch, set_cmask_scratch);
 	return 0;
 }
 
kernel/sched/ext_internal.h +8 -2
index ff7e882bd67a..9bb65367f510 100644
@@ -1124,6 +1124,14 @@ struct scx_sched {
 	struct bpf_map		*arena_map;
 	struct gen_pool		*arena_pool;
 
+	/*
+	 * Per-CPU arena cmask used by scx_call_op_set_cpumask() to hand a cmask
+	 * to ops_cid.set_cmask(). The kernel writes through the stored kern_va;
+	 * the BPF-arena uaddr handed to BPF is recovered by subtracting the
+	 * arena's kern_vm_start.
+	 */
+	struct scx_cmask * __percpu *set_cmask_scratch;
+
 	DECLARE_BITMAP(has_op, SCX_OPI_END);
 
 	/*
@@ -1480,8 +1488,6 @@ enum scx_ops_state {
 extern struct scx_sched __rcu *scx_root;
 DECLARE_PER_CPU(struct rq *, scx_locked_rq_state);
 
-extern struct scx_cmask __percpu *scx_set_cmask_scratch;
-
 /*
  * True when the currently loaded scheduler hierarchy is cid-form. All scheds
  * in a hierarchy share one form, so this single key tells callsites which
tools/sched_ext/include/scx/cid.bpf.h +0 -52

56 lines not on this page · raw message

tools/sched_ext/scx_qmap.bpf.c +3 -2
index 7e77f22674ea..8a2d6a8ebd8e 100644
@@ -919,14 +919,15 @@ void BPF_STRUCT_OPS(qmap_update_idle, s32 cid, bool idle)
 }
 
 void BPF_STRUCT_OPS(qmap_set_cmask, struct task_struct *p,
-		    const struct scx_cmask *cmask)
+		    const struct scx_cmask *cmask_in)
 {
+	struct scx_cmask __arena *cmask = (struct scx_cmask __arena *)(long)cmask_in;
 	task_ctx_t *taskc;
 
 	taskc = lookup_task_ctx(p);
 	if (!taskc)
 		return;
-	cmask_copy_from_kernel(&taskc->cpus_allowed, cmask);
+	cmask_copy(&taskc->cpus_allowed, cmask);
 }
 
 struct monitor_timer {
Signature · 1 line
2.54.0