bpf/arena: Direct kernel-side access
Subsystems (4)
Tejun Heo
reviewercover bpf/arena: Direct kernel-side access
Hello,
This RFC is for the BPF folks. I don't know BPF code, so please take patches 1-6 with a big grain of salt - they make sense to me, but I don't know whether this is the right shape, or whether the whole direction is wrong.
Motivation
~~~~~~~~~~
sched_ext's ops_cid.set_cmask() hands the BPF scheduler a struct scx_cmask *. The kernel translates a kernel cpumask to a cmask, but it had no way to write into the arena, so the cmask lived in kernel memory and was passed as a trusted pointer. BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.
The shape isn't unique to set_cmask. Sub-scheduler support is on the way and more sched_ext callbacks will want to pass structured data to BPF. Anywhere a kfunc or struct_ops callback wants to hand a struct to a BPF program, arena residence is the natural answer.
Approach
~~~~~~~~
Add BPF_F_ARENA_MAP_ALWAYS. Arenas created with the flag pre-populate every kern_vm PTE with a per-arena "garbage" page, so kernel-side accesses anywhere in the 4G range never fault. arena_alloc_pages() replaces the garbage PTE with a real page; arena_free_pages() restores it. User-side fault semantics are unchanged. (The garbage-page mapping was suggested by Alexei)
What MAP_ALWAYS gives up is the sparse-PTE fault protection. That protection is already shallow: it only catches accesses outside the allocated set. Within the allocated set, a buggy BPF program can still corrupt arena memory in countless ways, and that much larger class of bugs goes undetected with or without sparse PTEs. The general answer to catching arena memory bugs is arena ASAN, which addresses both classes uniformly. Given that, trading the narrow sparse-PTE protection for clean direct kernel-side access seems like a fair deal as long as it stays opt-in. Arenas without the flag behave exactly as before.
Patches 1-3 (BPF: direct-access mode)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bpf/arena: Plumb struct bpf_arena * through PTE callbacks bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers
Patches 4-7 (arena auto-discovery in struct_ops registration)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bpf: Add bpf_struct_ops_for_each_prog()
bpf: Add bpf_prog_for_each_used_map()
bpf/arena: Add bpf_arena_map_kern_vm_start()
sched_ext: Require MAP_ALWAYS arena for cid-form schedulers
4-6 are helpers; 7 is the first user. From the cid-form struct_ops .reg() callback, sched_ext walks all member progs' aux->used_maps[] and requires exactly one BPF_MAP_TYPE_ARENA, with MAP_ALWAYS set, to be referenced across the whole struct_ops. Reaching into prog->aux from .reg() and expressing the requirement entirely in code (not in the struct_ops type) is the part I'm least sure about - see the open questions.
Patches 8-9 (sched_ext: arena allocator and set_cmask conversion)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
sched_ext: Sub-allocator over kernel-claimed BPF arena pages
sched_ext: Convert ops.set_cmask() to arena-resident cmask
The kernel sub-allocates inside the arena via gen_pool over claimed pages and uses that storage to back the per-CPU set_cmask cmask. BPF dereferences via __arena like any other arena struct - no helpers, no probe-reads, no error paths.
Open questions
~~~~~~~~~~~~~~
I'd appreciate input on any of these, including "this is the wrong direction":
* Does the overall shape - BPF_F_ARENA_MAP_ALWAYS, the struct_ops / used_maps walk helpers, the kern_vm_start lookup - look reasonable? Or would you structure something fundamentally differently?
* Patch 7's enforcement - "this struct_ops registration must reference exactly one MAP_ALWAYS arena, walked out of prog->aux->used_maps[] from inside .reg()" - is purely runtime code in sched_ext. Is there a cleaner mechanism?
* In patch 9 the BPF side receives an arena pointer dressed as a regular kernel pointer and casts it back:
void BPF_STRUCT_OPS(qmap_set_cmask, struct task_struct *p,
const struct scx_cmask *cmask_in)
{
struct scx_cmask __arena *cmask =
(struct scx_cmask __arena *)(long)cmask_in;
...
} I'd much rather declare the struct_ops callback so the BPF side receives a struct scx_cmask __arena * directly and the cast goes away.
This stack is based on sched_ext/cid-cmask (064d11edd78b) - for-7.2 + the already-posted ext_*.c include reorg [1] + 17 cid/cmask patches.
[1] https://lore.kernel.org/r/[email protected]
Git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git cid-arena-rfc
include/linux/bpf.h | 15 ++++ include/uapi/linux/bpf.h | 7 ++ kernel/bpf/arena.c | 127 +++++++++++++++++++++++++++---- kernel/bpf/bpf_struct_ops.c | 36 +++++++++ kernel/bpf/core.c | 29 +++++++ kernel/sched/build_policy.c | 4 + kernel/sched/ext.c | 137 ++++++++++++++++++++++++++++++++-- kernel/sched/ext_arena.c | 128 +++++++++++++++++++++++++++++++ kernel/sched/ext_arena.h | 18 +++++ kernel/sched/ext_cid.c | 16 +--- kernel/sched/ext_internal.h | 25 ++++++- kernel/sched/ext_types.h | 10 +++ tools/sched_ext/include/scx/cid.bpf.h | 44 ----------- tools/sched_ext/scx_qmap.bpf.c | 8 +- 14 files changed, 520 insertions(+), 84 deletions(-)
Thanks.
--
tejun
Tejun Heo
reviewerRFC PATCH 1/9 bpf/arena: Plumb struct bpf_arena * through PTE callbacks
A subsequent change needs the PTE callbacks in bpf_arena to consult per-arena state. Make struct bpf_arena * reachable from each:
- apply_range_set_cb: add an arena field to apply_range_data. The data arg can no longer be NULL (it now carries arena), so the "skip PTE install" sentinel used by populate_pgtable_except_pte() shifts from data == NULL to data->pages == NULL.
- apply_range_clear_cb: introduce struct apply_range_clear_data { arena, free_pages } in place of the bare struct llist_head * arg.
- existing_page_cb: arena_map_free() passes arena instead of NULL. The callback doesn't read it yet.
No behavior change.
--- kernel/bpf/arena.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-)
kernel/bpf/arena.c +29 -10
@@ -114,16 +114,22 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) }struct apply_range_data { -struct page **pages; +struct bpf_arena *arena; +struct page **pages;/* NULL: skip PTE install */ int i; }; +struct apply_range_clear_data { +struct bpf_arena *arena; +struct llist_head *free_pages; +}; +static int apply_range_set_cb (pte_t *pte,unsigned long addr,void *data) {struct apply_range_data *d = data;struct page *page; -if (!data) +if (!d->pages)return 0 ;/* sanity check */ if (unlikely (!pte_none (ptep_get (pte))))
@@ -144,8 +150,9 @@ static void flush_vmap_cache(unsigned long start, unsigned long size)flush_cache_vmap (start, start + size); } -static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *free_pages) +static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *data) { +struct apply_range_clear_data *d = data;pte_t old_pte;struct page *page;
@@ -161,16 +168,18 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pagespte_clear (&init_mm, addr, pte);/* Add page to the list so it is freed later */ -if (free_pages) -__llist_add (&page->pcp_llist, free_pages); +if (d->free_pages) +__llist_add (&page->pcp_llist, d->free_pages);return 0 ; }static int populate_pgtable_except_pte (struct bpf_arena *arena) { +struct apply_range_data data = { .arena = arena }; +return apply_to_page_range (&init_mm,bpf_arena_get_kern_vm_start (arena), -KERN_VM_SZ -GUARD_SZ , apply_range_set_cb,NULL ); +KERN_VM_SZ -GUARD_SZ , apply_range_set_cb, &data); }static struct bpf_map *arena_map_alloc (union bpf_attr *attr)
@@ -286,7 +295,7 @@ static void arena_map_free(struct bpf_map *map)* free those pages. */ apply_to_existing_page_range (&init_mm,bpf_arena_get_kern_vm_start (arena), -KERN_VM_SZ -GUARD_SZ , existing_page_cb,NULL ); +KERN_VM_SZ -GUARD_SZ , existing_page_cb, arena);free_vm_area (arena->kern_vm);range_tree_destroy (&arena->rt);bpf_map_area_free (arena);
@@ -388,7 +397,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)if (ret) goto out_unlock_sigsegv; -struct apply_range_data data = { .pages = &page, .i =0 }; +struct apply_range_data data = { .arena = arena, .pages = &page, .i =0 };/* Account into memcg of the process that created bpf_arena */ ret =bpf_map_alloc_pages (map,NUMA_NO_NODE ,1 , &page);if (ret) {
@@ -569,6 +578,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntbpf_map_memcg_exit (old_memcg, new_memcg);return 0 ; } + data.arena = arena; data.pages = pages;if (raw_res_spin_lock_irqsave (&arena->spinlock, flags))
@@ -696,9 +706,13 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,range_tree_set (&arena->rt, pgoff, page_cnt);init_llist_head (&free_pages); +struct apply_range_clear_data clear_data = { + .arena = arena, + .free_pages = &free_pages, + };/* clear ptes and collect struct pages */ apply_to_existing_page_range (&init_mm, kaddr, page_cnt <<PAGE_SHIFT , - apply_range_clear_cb, &free_pages); + apply_range_clear_cb, &clear_data);/* drop the lock to do the tlb flush and zap pages */ raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);
@@ -804,6 +818,11 @@ static void arena_free_worker(struct work_struct *work) arena_vm_start =bpf_arena_get_kern_vm_start (arena); user_vm_start =bpf_arena_get_user_vm_start (arena); +struct apply_range_clear_data clear_data = { + .arena = arena, + .free_pages = &free_pages, + }; + list =llist_del_all (&arena->free_spans);llist_for_each (pos, list) { s = llist_entry(pos, struct arena_free_span, node);
@@ -813,7 +832,7 @@ static void arena_free_worker(struct work_struct *work)/* clear ptes and collect pages in free_pages llist */ apply_to_existing_page_range (&init_mm, kaddr, page_cnt <<PAGE_SHIFT , - apply_range_clear_cb, &free_pages); + apply_range_clear_cb, &clear_data);range_tree_set (&arena->rt, pgoff, page_cnt); }
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 2/9 bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
In reply to Tejun Heobpf_arena's kern_vm range is selectively populated: only allocated pages have PTEs. This catches a narrow class of buggy BPF programs that dereference unmapped arena addresses, but the protection is shallow - within the allocated set there are countless ways for a buggy program to corrupt arena memory.
It does, however, impose cost on the kernel side accesses. A kfunc or struct_ops callback that wants to consume an arena pointer cannot simply load through it; the page may have been freed underneath, so the access has to go through copy_from_kernel_nofault(). Out-parameter writes currently have no equivalent.
Arena is becoming the primary memory model for BPF programs, and more kfunc / struct_ops surfaces will want to read and write arena memory directly. The actual answer for catching arena memory bugs is arena ASAN, which addresses all memory access bugs meaningfully. Given that, it's worth offering an opt-in mode that drops the partial fault protection in exchange for cheap direct kernel-side access.
Add BPF_F_ARENA_MAP_ALWAYS. Arenas created with this flag allocate a per-arena "garbage" page and pre-populate every PTE in the kern_vm range to point at it. arena_alloc_pages() replaces the garbage PTE with a real page; arena_free_pages() restores the garbage PTE instead of clearing. arena_vm_fault() ignores the garbage page so user-side fault semantics are unchanged.
Stores into garbage-backed addresses are silently absorbed; loads return indeterminate bytes. Userspace mappings are unaffected. The flag is opt-in - arenas without it behave exactly as before.
--- include/uapi/linux/bpf.h | 7 +++++ kernel/bpf/arena.c | 62 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 64 insertions(+), 5 deletions(-)
include/uapi/linux/bpf.h +7 -0
@@ -1456,6 +1456,13 @@ enum {/* Enable BPF ringbuf overwrite mode */ BPF_F_RB_OVERWRITE = (1U <<19 ), + +/* Keep every kernel-side PTE in a BPF_MAP_TYPE_ARENA backed by a per-arena +* "garbage" page so that kernel-side accesses anywhere in the arena's 4G range +* never fault. Loads from unallocated or freed regions return indeterminate +* bytes; stores are silently absorbed. Userspace mappings are unaffected. +*/ + BPF_F_ARENA_MAP_ALWAYS = (1U << 20), };/* Flags for BPF_PROG_QUERY. */
kernel/bpf/arena.c +57 -5
134 lines not on this page · raw message
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 3/9 bpf: Add sleepable variant of bpf_arena_alloc_pages for kernel callers
In reply to Tejun HeoThe existing kernel-side export of bpf_arena_alloc_pages is _non_sleepable only - it's used by the verifier to inline the kfunc when the call site is non-sleepable. There is no sleepable equivalent for kernel callers; the kfunc bpf_arena_alloc_pages itself is BPF-only.
sched_ext needs sleepable kernel-side allocs for its arena pool init/grow paths. Add bpf_arena_alloc_pages_sleepable() mirroring the _non_sleepable wrapper but passing sleepable=true to arena_alloc_pages().
--- include/linux/bpf.h | 8 ++++++++ kernel/bpf/arena.c | 13 +++++++++++++ 2 files changed, 21 insertions(+)
include/linux/bpf.h +8 -0
@@ -678,6 +678,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags);void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt); +void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id, +u64 flags);#else static inline void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id,u64 flags)
@@ -688,6 +690,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addrstatic inline void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt) { } + +static inline void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +return NULL ; +}#endif extern const struct bpf_map_ops bpf_map_offload_ops;
kernel/bpf/arena.c +13 -0
@@ -947,6 +947,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pagreturn (void *)arena_alloc_pages (arena, (long )addr__ign, page_cnt, node_id, false); } + +void *bpf_arena_alloc_pages_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + +if (map->map_type !=BPF_MAP_TYPE_ARENA || flags || !page_cnt) +return NULL ; + +return (void *)arena_alloc_pages (arena, (long )addr__ign, page_cnt, node_id, true); +} + __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) {struct bpf_map *map = p__map;
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 4/9 bpf: Add bpf_struct_ops_for_each_prog()
In reply to Tejun HeoAdd a helper that walks the member progs of the struct_ops map containing a given @kdata vmtable. struct_ops ->reg() callbacks (and similar) sometimes need to inspect the loaded BPF programs, e.g. to discover maps they reference via prog->aux->used_maps.
The implementation mirrors bpf_struct_ops_id(): container_of @kdata to recover the bpf_struct_ops_map, then iterate st_map->links[i]->prog for i in [0, funcs_cnt). Same access pattern, no new locking - by the time ->reg() fires st_map is fully populated and stable.
A sched_ext follow-up uses this to require cid-form schedulers to use exactly one BPF_F_ARENA_MAP_ALWAYS arena across their member progs, without requiring the BPF program to call a registration kfunc.
--- include/linux/bpf.h | 3 +++ kernel/bpf/bpf_struct_ops.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/linux/bpf.h +3 -0
@@ -2128,6 +2128,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);void bpf_prog_disassoc_struct_ops (struct bpf_prog *prog);void *bpf_prog_get_assoc_struct_ops (const struct bpf_prog_aux *aux);u32 bpf_struct_ops_id (const void *kdata); +int bpf_struct_ops_for_each_prog (const void *kdata, +int (*cb)(struct bpf_prog *prog,void *data), +void *data);#ifdef CONFIG_NET /* Define it here to avoid the use of forward declaration */
kernel/bpf/bpf_struct_ops.c +36 -0
@@ -1203,6 +1203,42 @@ u32 bpf_struct_ops_id(const void *kdata) }EXPORT_SYMBOL_GPL (bpf_struct_ops_id); +/** +* bpf_struct_ops_for_each_prog - Invoke @cb for each member prog +* @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg) +* @cb: callback invoked once per member prog; non-zero return stops iteration +* @data: opaque argument passed to @cb +* +* Walks the struct_ops member progs registered on the map containing @kdata. +* Intended for use from struct_ops ->reg() callbacks (and similar) that need to +* inspect the loaded BPF programs (for example to discover maps they reference +* via @prog->aux->used_maps). +* +* Return 0 if iteration completed, otherwise the first non-zero @cb return. +*/ +int bpf_struct_ops_for_each_prog (const void *kdata, +int (*cb)(struct bpf_prog *prog,void *data), +void *data) +{ +struct bpf_struct_ops_value *kvalue; +struct bpf_struct_ops_map *st_map; +u32 i; +int ret; + + kvalue = container_of(kdata, struct bpf_struct_ops_value, data); + st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue); + +for (i =0 ; i < st_map->funcs_cnt; i++) { +if (!st_map->links[i]) +continue ; + ret =cb (st_map->links[i]->prog, data); +if (ret) +return ret; + } +return 0 ; +} +EXPORT_SYMBOL_GPL (bpf_struct_ops_for_each_prog); +static bool bpf_struct_ops_valid_to_reg (struct bpf_map *map) {struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 5/9 bpf: Add bpf_prog_for_each_used_map()
In reply to Tejun HeoWrap the prog->aux->used_maps[] walk and its used_maps_mutex behind a helper. Existing in-tree callers open-code the same lock + iterate pattern (e.g. bpf_check_tail_call in core.c, the verifier and syscall paths); a sched_ext follow-up needs the same loop and would otherwise reach into bpf_prog_aux directly.
--- include/linux/bpf.h | 3 +++ kernel/bpf/core.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
include/linux/bpf.h +3 -0
@@ -2338,6 +2338,9 @@ static inline bool map_type_contains_progs(struct bpf_map *map)bool bpf_prog_map_compatible (struct bpf_map *map,const struct bpf_prog *fp);int bpf_prog_calc_tag (struct bpf_prog *fp); +int bpf_prog_for_each_used_map (struct bpf_prog *prog, +int (*cb)(struct bpf_map *map,void *data), +void *data);const struct bpf_func_proto *bpf_get_trace_printk_proto (void );const struct bpf_func_proto *bpf_get_trace_vprintk_proto (void );
kernel/bpf/core.c +29 -0
@@ -2510,6 +2510,35 @@ static int bpf_check_tail_call(const struct bpf_prog *fp)return ret; } +/** +* bpf_prog_for_each_used_map - Invoke @cb for each map @prog references +* @prog: BPF program whose used_maps to walk +* @cb: callback invoked once per map; non-zero return stops iteration +* @data: opaque argument passed to @cb +* +* Holds prog->aux->used_maps_mutex across the walk. +* +* Return 0 if iteration completed, otherwise the first non-zero @cb return. +*/ +int bpf_prog_for_each_used_map (struct bpf_prog *prog, +int (*cb)(struct bpf_map *map,void *data), +void *data) +{ +struct bpf_prog_aux *aux = prog->aux; +int ret =0 ; +u32 i; + +mutex_lock (&aux->used_maps_mutex); +for (i =0 ; i < aux->used_map_cnt; i++) { + ret =cb (aux->used_maps[i], data); +if (ret) +break ; + } +mutex_unlock (&aux->used_maps_mutex); +return ret; +} +EXPORT_SYMBOL_GPL (bpf_prog_for_each_used_map); +static bool bpf_prog_select_interpreter (struct bpf_prog *fp) {bool select_interpreter = false;
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 6/9 bpf/arena: Add bpf_arena_map_kern_vm_start()
In reply to Tejun Heobpf_arena_get_kern_vm_start() takes a struct bpf_arena *. The struct itself isn't exposed, so callers outside arena.c that hold only a struct bpf_map * (e.g. struct_ops subsystems handling an arena map fd) can't reach it.
Add bpf_arena_map_kern_vm_start() which takes struct bpf_map * and container_of()s to the arena, with a type check. A sched_ext follow-up needs this to translate kern_va <-> uaddr for arena pages it claims.
--- include/linux/bpf.h | 1 + kernel/bpf/arena.c | 13 +++++++++++++ 2 files changed, 14 insertions(+)
include/linux/bpf.h +1 -0
@@ -617,6 +617,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, struct bpf_spin_lock *spin_lock);u64 bpf_arena_get_kern_vm_start (struct bpf_arena *arena);u64 bpf_arena_get_user_vm_start (struct bpf_arena *arena); +u64 bpf_arena_map_kern_vm_start (struct bpf_map *map);int bpf_obj_name_cpy (char *dst,const char *src,unsigned int size);struct bpf_offload_dev ;
kernel/bpf/arena.c +13 -0
@@ -85,6 +85,19 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena)return arena ? arena->user_vm_start :0 ; } +/** +* bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map * +* @map: a BPF_MAP_TYPE_ARENA map +* +* Return @map's kern_vm_start, or 0 (with WARN) if @map isn't an arena. +*/ +u64 bpf_arena_map_kern_vm_start (struct bpf_map *map) +{ +if (WARN_ON_ONCE (!map || map->map_type !=BPF_MAP_TYPE_ARENA )) +return 0 ; + return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map)); +} +static long arena_map_peek_elem (struct bpf_map *map,void *value) {return -EOPNOTSUPP ;
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 7/9 sched_ext: Require MAP_ALWAYS arena for cid-form schedulers
In reply to Tejun HeoUpcoming patches will let the kernel place arena-resident scratch shared with the BPF program (e.g. per-CPU set_cmask cmask) so the BPF side can dereference it directly via __arena pointers, replacing the current cmask_copy_from_kernel() probe-read loop. That requires each cid-form scheduler to expose its arena to the kernel and to opt into BPF_F_ARENA_MAP_ALWAYS so kernel-side stores never fault.
bpf_scx_reg_cid() walks the struct_ops member progs via the new bpf_struct_ops_for_each_prog() helper and discovers the arena from prog->aux->used_maps. It requires exactly one BPF_MAP_TYPE_ARENA across all member progs and rejects if BPF_F_ARENA_MAP_ALWAYS is not set. The map ref is held on scx_sched and dropped on sched destroy. cpu-form schedulers (bpf_scx_reg) are unchanged - no arena requirement.
scx_qmap adds BPF_F_ARENA_MAP_ALWAYS to its arena map definition.
v2: Defer sch->arena_map = cmd->arena_map consumption past
scx_alloc_and_add_sched() failure points so an early kzalloc/kstrdup failure leaves cmd->arena_map set; bpf_scx_reg_cid() then drops the ref via the existing cmd.arena_map cleanup.
--- kernel/sched/ext.c | 59 +++++++++++++++++++++++++++++++++- kernel/sched/ext_internal.h | 9 ++++++ tools/sched_ext/scx_qmap.bpf.c | 2 +- 3 files changed, 68 insertions(+), 2 deletions(-)
kernel/sched/ext.c +58 -1
@@ -4916,6 +4916,8 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +if (sch->arena_map) +bpf_map_put (sch->arena_map);kfree (sch); }
@@ -6588,6 +6590,7 @@ struct scx_enable_cmd {struct sched_ext_ops_cid *ops_cid; };bool is_cid_type; +struct bpf_map *arena_map;/* arena ref to transfer to sch */ int ret; };
@@ -6751,6 +6754,15 @@ static struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, return ERR_PTR(ret); }#endif /* CONFIG_EXT_SUB_SCHED */ + +/* +* Consume the arena_map ref bpf_scx_reg_cid() took. Defer to here so +* earlier failure paths leave cmd->arena_map set and bpf_scx_reg_cid +* drops the ref. After this point, sch owns the ref and any cleanup +* runs through scx_sched_free_rcu_work() which puts it. +*/ + sch->arena_map = cmd->arena_map; + cmd->arena_map =NULL ;return sch; err_free_lb_resched:
@@ -7676,11 +7688,56 @@ static int bpf_scx_reg(void *kdata, struct bpf_link *link)return scx_enable (&cmd, link); } +struct scx_arena_scan { +struct bpf_map *arena; +int err; +}; + +static int scx_arena_scan_map (struct bpf_map *m,void *data) +{ +struct scx_arena_scan *s = data; + +if (m->map_type !=BPF_MAP_TYPE_ARENA ) +return 0 ; +if (s->arena && s->arena != m) { + s->err = -EINVAL ; +return 1 ; + } + s->arena = m; +return 0 ; +} + +static int scx_arena_scan_prog (struct bpf_prog *prog,void *data) +{ +return bpf_prog_for_each_used_map (prog, scx_arena_scan_map, data); +} +static int bpf_scx_reg_cid (void *kdata,struct bpf_link *link) {struct scx_enable_cmd cmd = { .ops_cid = kdata, .is_cid_type = true }; +struct scx_arena_scan scan = {}; +int ret; -return scx_enable (&cmd, link); +bpf_struct_ops_for_each_prog (kdata, scx_arena_scan_prog, &scan); +if (scan.err) { +pr_err ("sched_ext: cid-form scheduler uses multiple arena maps\n" ); +return scan.err; + } +if (!scan.arena) { +pr_err ("sched_ext: cid-form scheduler must use a BPF arena map\n" ); +return -EINVAL ; + } +if (!(scan.arena->map_flags &BPF_F_ARENA_MAP_ALWAYS )) { +pr_err ("sched_ext: arena map requires BPF_F_ARENA_MAP_ALWAYS for cid-form\n" ); +return -EINVAL ; + } + +bpf_map_inc (scan.arena); + cmd.arena_map = scan.arena; + ret =scx_enable (&cmd, link); +if (cmd.arena_map)/* not consumed by scx_alloc_and_add_sched() */ +bpf_map_put (cmd.arena_map); +return ret; }static void bpf_scx_unreg (void *kdata,struct bpf_link *link)
kernel/sched/ext_internal.h +9 -0
@@ -1102,6 +1102,15 @@ struct scx_sched {struct sched_ext_ops_cid ops_cid; };bool is_cid_type;/* true if registered via bpf_sched_ext_ops_cid */ + +/* +* Arena map auto-discovered from member progs at struct_ops attach. +* cid-form schedulers must use exactly one arena with +* BPF_F_ARENA_MAP_ALWAYS to enable direct arena access from kernel +* side. NULL on cpu-form. +*/ +struct bpf_map *arena_map; +DECLARE_BITMAP (has_op,SCX_OPI_END );/*
tools/sched_ext/scx_qmap.bpf.c +1 -1
@@ -83,7 +83,7 @@ UEI_DEFINE(uei);*/ struct { __uint(type, BPF_MAP_TYPE_ARENA); - __uint(map_flags, BPF_F_MMAPABLE); + __uint(map_flags, BPF_F_MMAPABLE | BPF_F_ARENA_MAP_ALWAYS); __uint(max_entries, 1 << 16); /* upper bound in pages */#if defined(__TARGET_ARCH_arm64) || defined(__aarch64__) __ulong(map_extra, 0x1ull << 32); /* user/BPF mmap base */
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 8/9 sched_ext: Sub-allocator over kernel-claimed BPF arena pages
In reply to Tejun HeoBuild a per-scheduler sub-allocator on top of pages claimed from the BPF arena registered in the previous patch. Subsequent kernel-managed arena-resident structures (e.g. per-CPU set_cmask cmask) carve their storage from this pool.
scx_arena_pool_init() creates a gen_pool. scx_arena_alloc() returns a kernel VA and the matching BPF-arena uaddr in *@uaddr_out. On exhaustion, the pool grows by claiming more pages via bpf_arena_alloc_pages_sleepable(). Each chunk is added to the gen_pool with kern_va as the "virt" and uaddr as the "phys", so gen_pool_virt_to_phys() recovers the uaddr for handing to BPF.
Allocations sleep (GFP_KERNEL) - they may grow the pool through vzalloc and arena page allocation. All current consumers run from the enable path (after ops.init() and the kernel-side arena auto-discovery, before validate_ops()), where sleeping is fine.
scx_arena_pool_destroy() walks each chunk, returns outstanding ranges to the gen_pool with gen_pool_free() and then calls gen_pool_destroy(). The underlying arena pages are released when the arena map itself is torn down, so the pool destroy doesn't free them explicitly.
--- kernel/sched/build_policy.c | 4 ++ kernel/sched/ext.c | 11 ++++ kernel/sched/ext_arena.c | 128 ++++++++++++++++++++++++++++++++++++ kernel/sched/ext_arena.h | 18 +++++ kernel/sched/ext_internal.h | 6 ++ 5 files changed, 167 insertions(+) create mode 100644 kernel/sched/ext_arena.c create mode 100644 kernel/sched/ext_arena.h
kernel/sched/build_policy.c +4 -0
@@ -59,12 +59,16 @@#ifdef CONFIG_SCHED_CLASS_EXT # include <linux/btf_ids.h> +# include <linux/find.h> +# include <linux/genalloc.h> # include "ext_types.h" # include "ext_internal.h" # include "ext_cid.h" +# include "ext_arena.h" # include "ext_idle.h" # include "ext.c" # include "ext_cid.c" +# include "ext_arena.c" # include "ext_idle.c" #endif
kernel/sched/ext.c +11 -0
@@ -4916,6 +4916,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +scx_arena_pool_destroy (sch);if (sch->arena_map)bpf_map_put (sch->arena_map);kfree (sch);
@@ -6975,6 +6976,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; } + ret =scx_arena_pool_init (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)if (((void (**)(void ))ops)[i])set_bit (i, sch->has_op);
@@ -7264,6 +7271,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work) sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; } + ret =scx_arena_pool_init (sch); +if (ret) + goto err_disable; +if (validate_ops (sch, ops)) goto err_disable;
kernel/sched/ext_arena.c new file +128 -0
128 lines not on this page · raw message
kernel/sched/ext_arena.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst +* +* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2025 Tejun Heo <[email protected]> +*/ +#ifndef _KERNEL_SCHED_EXT_ARENA_H +#define _KERNEL_SCHED_EXT_ARENA_H + +struct scx_sched ; + +s32 scx_arena_pool_init (struct scx_sched *sch); +void scx_arena_pool_destroy (struct scx_sched *sch); +void *scx_arena_alloc (struct scx_sched *sch,size_t size,u32 *uaddr_out); +void scx_arena_free (struct scx_sched *sch,void *kern_va,size_t size); + +#endif /* _KERNEL_SCHED_EXT_ARENA_H */
kernel/sched/ext_internal.h +6 -0
@@ -1108,8 +1108,14 @@ struct scx_sched {* cid-form schedulers must use exactly one arena with * BPF_F_ARENA_MAP_ALWAYS to enable direct arena access from kernel * side. NULL on cpu-form. +* +* @arena_pool sub-allocates @arena_map. Each gen_pool chunk is added +* with kern_va as the "virt" address and the matching BPF uaddr as the +* "phys", so gen_pool_virt_to_phys() recovers the uaddr for handing to +* BPF. Grows on demand and pages are not released until sched destroy. */ struct bpf_map *arena_map; +struct gen_pool *arena_pool;DECLARE_BITMAP (has_op,SCX_OPI_END );
Signature · 1 line
2.53.0
Tejun Heo
reviewerRFC PATCH 9/9 sched_ext: Convert ops.set_cmask() to arena-resident cmask
In reply to Tejun Heoops_cid.set_cmask() expects a cmask. The kernel couldn't write into the arena, so it translated cpumask -> cmask in kernel memory and passed the result as a trusted pointer. The BPF cmask helpers all operate on arena cmasks though, so the BPF side had to word-by-word probe-read the kernel cmask into an arena cmask via cmask_copy_from_kernel() before any helper could touch it. It works, but is clumsy.
With direct kernel-side arena access now in place, build the cmask in the arena. The kernel writes to it through the kern_va side of the dual mapping; BPF directly dereferences it via an __arena pointer like any other arena struct.
--- kernel/sched/ext.c | 67 +++++++++++++++++++++++++-- kernel/sched/ext_cid.c | 16 +------ kernel/sched/ext_internal.h | 10 +++- kernel/sched/ext_types.h | 10 ++++ tools/sched_ext/include/scx/cid.bpf.h | 44 ------------------ tools/sched_ext/scx_qmap.bpf.c | 6 ++- 6 files changed, 86 insertions(+), 67 deletions(-)
kernel/sched/ext.c +63 -4
@@ -622,11 +622,15 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,update_locked_rq (rq);if (scx_is_cid_type ()) { -struct scx_cmask *cmask =this_cpu_ptr (scx_set_cmask_scratch); +struct scx_cmask_scratch *s =this_cpu_ptr (sch->set_cmask_scratch); -lockdep_assert_irqs_disabled (); -scx_cpumask_to_cmask (cpumask, cmask); - sch->ops_cid.set_cmask (task, cmask); +/* +* Build the per-CPU arena cmask and hand BPF the uaddr. Caller +* holds the rq lock with IRQs disabled, which makes us the sole +* user of the scratch area. +*/ +scx_cpumask_to_cmask (cpumask, s->kern_va); + sch->ops_cid.set_cmask (task, (struct scx_cmask *)(unsigned long )s->uaddr); }else { sch->ops.set_cpumask (task, cpumask); }
@@ -4864,6 +4868,47 @@ static const struct attribute_group scx_global_attr_group = {static void free_pnode (struct scx_sched_pnode *pnode);static void free_exit_info (struct scx_exit_info *ei); +/* Byte size of a struct scx_cmask covering num_possible_cpus(). Set at boot. */ +static size_t scx_possible_cmask_size; + +static s32 scx_set_cmask_scratch_alloc (struct scx_sched *sch) +{ +int cpu; + +if (!sch->is_cid_type || !sch->arena_pool) +return 0 ; + + sch->set_cmask_scratch = alloc_percpu(struct scx_cmask_scratch); +if (!sch->set_cmask_scratch) +return -ENOMEM ; + +for_each_possible_cpu (cpu) { +struct scx_cmask_scratch *s =per_cpu_ptr (sch->set_cmask_scratch, cpu); + + s->kern_va =scx_arena_alloc (sch, scx_possible_cmask_size, &s->uaddr); +if (!s->kern_va) +return -ENOMEM ; +scx_cmask_init (s->kern_va,0 ,num_possible_cpus ()); + } +return 0 ; +} + +static void scx_set_cmask_scratch_free (struct scx_sched *sch) +{ +int cpu; + +if (!sch->set_cmask_scratch) +return ; + +for_each_possible_cpu (cpu) { +struct scx_cmask_scratch *s =per_cpu_ptr (sch->set_cmask_scratch, cpu); + +scx_arena_free (sch, s->kern_va, scx_possible_cmask_size); + } +free_percpu (sch->set_cmask_scratch); + sch->set_cmask_scratch =NULL ; +} +static void scx_sched_free_rcu_work (struct work_struct *work) {struct rcu_work *rcu_work =to_rcu_work (work);
@@ -4916,6 +4961,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL );free_exit_info (sch->exit_info); +scx_set_cmask_scratch_free (sch);scx_arena_pool_destroy (sch);if (sch->arena_map)bpf_map_put (sch->arena_map);
@@ -6982,6 +7028,12 @@ static void scx_root_enable_workfn(struct kthread_work *work) goto err_disable; } + ret =scx_set_cmask_scratch_alloc (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)if (((void (**)(void ))ops)[i])set_bit (i, sch->has_op);
@@ -7275,6 +7327,10 @@ static void scx_sub_enable_workfn(struct kthread_work *work)if (ret) goto err_disable; + ret =scx_set_cmask_scratch_alloc (sch); +if (ret) + goto err_disable; +if (validate_ops (sch, ops)) goto err_disable;
@@ -8202,6 +8258,9 @@ void __init init_sched_ext_class(void)WRITE_ONCE (v,SCX_ENQ_WAKEUP |SCX_DEQ_SLEEP |SCX_KICK_PREEMPT |SCX_TG_ONLINE ); + scx_possible_cmask_size = struct_size_t(struct scx_cmask, bits, +SCX_CMASK_NR_WORDS (num_possible_cpus ())); +scx_idle_init_masks ();for_each_possible_cpu (cpu) {
kernel/sched/ext_cid.c +1 -15
@@ -7,14 +7,6 @@*/ #include <linux/cacheinfo.h> -/* -* Per-cpu scratch cmask used by scx_call_op_set_cpumask() to synthesize a -* cmask from a cpumask. Allocated alongside the cid arrays on first enable -* and never freed. Sized to the full cid space. Caller holds rq lock so -* this_cpu_ptr is safe. -*/ -struct scx_cmask __percpu *scx_set_cmask_scratch; -/* * cid tables. *
@@ -54,7 +46,6 @@ static s32 scx_cid_arrays_alloc(void)u32 npossible =num_possible_cpus ();s16 *cid_to_cpu, *cpu_to_cid;struct scx_cid_topo *cid_topo; -struct scx_cmask __percpu *set_cmask_scratch;if (scx_cid_to_cpu_tbl)return 0 ;
@@ -62,22 +53,17 @@ static s32 scx_cid_arrays_alloc(void) cid_to_cpu =kzalloc_objs (*scx_cid_to_cpu_tbl, npossible,GFP_KERNEL ); cpu_to_cid =kzalloc_objs (*scx_cpu_to_cid_tbl, nr_cpu_ids,GFP_KERNEL ); cid_topo =kmalloc_objs (*scx_cid_topo, npossible,GFP_KERNEL ); - set_cmask_scratch =__alloc_percpu (struct_size (set_cmask_scratch, bits, -SCX_CMASK_NR_WORDS (npossible)), -sizeof (u64)); -if (!cid_to_cpu || !cpu_to_cid || !cid_topo || !set_cmask_scratch) { +if (!cid_to_cpu || !cpu_to_cid || !cid_topo) {kfree (cid_to_cpu);kfree (cpu_to_cid);kfree (cid_topo); -free_percpu (set_cmask_scratch);return -ENOMEM ; }WRITE_ONCE (scx_cid_to_cpu_tbl, cid_to_cpu);WRITE_ONCE (scx_cpu_to_cid_tbl, cpu_to_cid);WRITE_ONCE (scx_cid_topo, cid_topo); -WRITE_ONCE (scx_set_cmask_scratch, set_cmask_scratch);return 0 ; }
kernel/sched/ext_internal.h +8 -2
@@ -1117,6 +1117,14 @@ struct scx_sched {struct bpf_map *arena_map;struct gen_pool *arena_pool; +/* +* Per-CPU arena cmask used by scx_call_op_set_cpumask() to hand a cmask +* to ops_cid.set_cmask(). Each entry stashes both the kernel VA (for +* the kernel to write into) and the BPF-arena uaddr (passed to BPF as +* the cmask pointer). +*/ +struct scx_cmask_scratch __percpu *set_cmask_scratch; +DECLARE_BITMAP (has_op,SCX_OPI_END );/*
@@ -1473,8 +1481,6 @@ enum scx_ops_state {extern struct scx_sched __rcu *scx_root; DECLARE_PER_CPU(struct rq *, scx_locked_rq_state); -extern struct scx_cmask __percpu *scx_set_cmask_scratch; -/* * True when the currently loaded scheduler hierarchy is cid-form. All scheds * in a hierarchy share one form, so this single key tells callsites which
kernel/sched/ext_types.h +10 -0
@@ -101,4 +101,14 @@ struct scx_cmask {#define SCX_CMASK_DEFINE (name, cap_bits) \ DEFINE_RAW_FLEX(struct scx_cmask, name, bits, SCX_CMASK_NR_WORDS(cap_bits)) +/* +* Stash for one arena-resident cmask. @kern_va points into the kernel's +* view of the BPF arena; @uaddr is the matching BPF-arena address to +* hand to BPF (cast to struct scx_cmask *). +*/ +struct scx_cmask_scratch { +struct scx_cmask *kern_va; +u32 uaddr; +}; +#endif /* _KERNEL_SCHED_EXT_TYPES_H */
tools/sched_ext/include/scx/cid.bpf.h +0 -44
48 lines not on this page · raw message
tools/sched_ext/scx_qmap.bpf.c +4 -2
@@ -922,14 +922,16 @@ void BPF_STRUCT_OPS(qmap_update_idle, s32 cid, bool idle) }void BPF_STRUCT_OPS (qmap_set_cmask ,struct task_struct *p, -const struct scx_cmask *cmask) +const struct scx_cmask *cmask_in) { + struct scx_cmask __arena *cmask = + (struct scx_cmask __arena *)(long)cmask_in;task_ctx_t *taskc; taskc =lookup_task_ctx (p);if (!taskc)return ; -cmask_copy_from_kernel (&taskc->cpus_allowed, cmask); +cmask_copy (&taskc->cpus_allowed, cmask); }struct monitor_timer {
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH 5/9] bpf: Add bpf_prog_for_each_used_map()
In reply to Tejun HeoQuoted 54 lines · Tejun Heo
Wrap the prog->aux->used_maps[] walk and its
used_maps_mutexbehind a helper. Existing in-tree callers open-code the same lock + iterate pattern (e.g.bpf_check_tail_callincore.c, the verifier and syscall paths); asched_extfollow-up needs the same loop and would otherwise reach intobpf_prog_auxdirectly.Signed-off-by Tejun Heo <[email protected]>--- include/linux/bpf.h | 3 +++ kernel/bpf/core.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)include/linux/bpf.h +3
-0@@ -2338,6 +2338,9 @@ static inline bool map_type_contains_progs(struct bpf_map *map)bool bpf_prog_map_compatible (struct bpf_map *map,const struct bpf_prog *fp);int bpf_prog_calc_tag (struct bpf_prog *fp); +int bpf_prog_for_each_used_map (struct bpf_prog *prog, +int (*cb)(struct bpf_map *map,void *data), +void *data);const struct bpf_func_proto *bpf_get_trace_printk_proto (void );const struct bpf_func_proto *bpf_get_trace_vprintk_proto (void );kernel/bpf/core.c +19
-0@@ -2510,6 +2510,35 @@ static int bpf_check_tail_call(const struct bpf_prog *fp)return ret; } +/** +* bpf_prog_for_each_used_map - Invoke @cb for each map @prog references +* @prog: BPF program whose used_maps to walk +* @cb: callback invoked once per map; non-zero return stops iteration +* @data: opaque argument passed to @cb +* +* Holds prog->aux->used_maps_mutex across the walk. +* +* Return 0 if iteration completed, otherwise the first non-zero @cb return. +*/ +int bpf_prog_for_each_used_map (struct bpf_prog *prog, +int (*cb)(struct bpf_map *map,void *data), +void *data) +{ +struct bpf_prog_aux *aux = prog->aux; +int ret =0 ; +u32 i; + +mutex_lock (&aux->used_maps_mutex);
kernel/bpf/core.c static int bpf_check_tail_call(const struct bpf_prog *fp) · line 2541
Quoted 10 lines
kernel/bpf/core.c +10
-0+for (i =0 ; i < aux->used_map_cnt; i++) { + ret =cb (aux->used_maps[i], data); +if (ret) +break ; + } +mutex_unlock (&aux->used_maps_mutex); +return ret; +} +EXPORT_SYMBOL_GPL (bpf_prog_for_each_used_map); +
Since each program only has one arena, and you use this to determine whether the program's arena has a flag, why not just add a bpf_prog_arena() accessor and check the result's flag directly? You can do prog->aux->arena inside it to return the bpf_map pointer.
Quoted 6 lines
kernel/bpf/core.c +0
-1static bool bpf_prog_select_interpreter (struct bpf_prog *fp) {bool select_interpreter = false; --2.53.0
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
In reply to Tejun HeoQuoted 27 lines · Tejun Heo
bpf_arena's
kern_vmrange is selectively populated: only allocated pages have PTEs. This catches a narrow class of buggy BPF programs that dereference unmapped arena addresses, but the protection is shallow - within the allocated set there are countless ways for a buggy program to corrupt arena memory.It does, however, impose cost on the kernel side accesses. A kfunc or
struct_opscallback that wants to consume an arena pointer cannot simply load through it; the page may have been freed underneath, so the access has to go throughcopy_from_kernel_nofault(). Out-parameter writes currently have no equivalent.Arena is becoming the primary memory model for BPF programs, and more kfunc /
struct_opssurfaces will want to read and write arena memory directly. The actual answer for catching arena memory bugs is arena ASAN, which addresses all memory access bugs meaningfully. Given that, it's worth offering an opt-in mode that drops the partial fault protection in exchange for cheap direct kernel-side access.Add
BPF_F_ARENA_MAP_ALWAYS. Arenas created with this flag allocate a per-arena "garbage" page and pre-populate every PTE in thekern_vmrange to point at it.arena_alloc_pages()replaces the garbage PTE with a real page;arena_free_pages()restores the garbage PTE instead of clearing.arena_vm_fault()ignores the garbage page so user-side fault semantics are unchanged.
Quoted 7 lines
Stores into garbage-backed addresses are silently absorbed; loads return indeterminate bytes. Userspace mappings are unaffected. The flag is opt-in - arenas without it behave exactly as before.
Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>---
If we go down this route, we should probably make this flag the default behavior. Otherwise, we cannot universally enable passing arena memory into kfuncs. Every subsystem will have to check the flag, we'll have to gate being able to pass memory based on the flag's presence, etc., which just adds complexity everywhere. It will eliminate a few patches in this set too. From the programmer's perspective, program behavior isn't changing much, so we can use zeroed page (to guarantee faulting loads return 0) instead of setting the PTE to NULL. While at it we should drop bpf_prog_report_arena_violation, and its various users.
Summarizing past discussions on all this, with more details on various pros/cons:
Currently, the semantics for a fault dictate that the program simply continues, and the destination register becomes 0. One could argue the ideal form should have been to abort the program on fault, but that wasn't possible at the time of implementation. We added fault reporting to the program's streams to improve debuggability. Now since we have an ASAN implementation, you can likely run that to catch memory safety problems. An argument against this is that it doesn't help surface a class of issues for production programs. We don't have data on whether stray faults or memory corruption within present pages is the more common occurrence of bugs in the small set of programs using arenas, so it hard to pass any clear judgement. One thing we do lose is faults on NULL-derefs, which are likely common, but Emil had some ideas on that.
Another thing we lose is the ability to build something like GWP-Asan
[0] that we can run in production programs without paying much of the performance cost by sampling allocations we want to detect bugs for. But between ASAN and Rust-BPF plans, I am not sure how compelling it will be going forward. So while it's sort of sad to lose the ability to fault feedback, it is also non-trivial to enable direct access to arena memory for the kernel while preserving faults (I won't go into the details here) without using fault-safe memcpy to move data from/to arena on the kernel side.
[0]: https://llvm.org/docs/GwpAsan.html
Quoted 1 line
[...]
Emil Tsalapatis
reviewerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 64 lines · Kumar Kartikeya Dwivedi
Quoted 27 lines · Tejun Heo
bpf_arena's
kern_vmrange is selectively populated: only allocated pages have PTEs. This catches a narrow class of buggy BPF programs that dereference unmapped arena addresses, but the protection is shallow - within the allocated set there are countless ways for a buggy program to corrupt arena memory.It does, however, impose cost on the kernel side accesses. A kfunc or
struct_opscallback that wants to consume an arena pointer cannot simply load through it; the page may have been freed underneath, so the access has to go throughcopy_from_kernel_nofault(). Out-parameter writes currently have no equivalent.Arena is becoming the primary memory model for BPF programs, and more kfunc /
struct_opssurfaces will want to read and write arena memory directly. The actual answer for catching arena memory bugs is arena ASAN, which addresses all memory access bugs meaningfully. Given that, it's worth offering an opt-in mode that drops the partial fault protection in exchange for cheap direct kernel-side access.Add
BPF_F_ARENA_MAP_ALWAYS. Arenas created with this flag allocate a per-arena "garbage" page and pre-populate every PTE in thekern_vmrange to point at it.arena_alloc_pages()replaces the garbage PTE with a real page;arena_free_pages()restores the garbage PTE instead of clearing.arena_vm_fault()ignores the garbage page so user-side fault semantics are unchanged.Quoted 7 lines
Stores into garbage-backed addresses are silently absorbed; loads return indeterminate bytes. Userspace mappings are unaffected. The flag is opt-in - arenas without it behave exactly as before.
Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>---
If we go down this route, we should probably make this flag the default behavior. Otherwise, we cannot universally enable passing arena memory into kfuncs. Every subsystem will have to check the flag, we'll have to gate being able to pass memory based on the flag's presence, etc., which just adds complexity everywhere. It will eliminate a few patches in this set too. From the programmer's perspective, program behavior isn't changing much, so we can use zeroed page (to guarantee faulting loads return 0) instead of setting the PTE to NULL. While at it we should drop
bpf_prog_report_arena_violation, and its various users.Summarizing past discussions on all this, with more details on various pros/cons:
Currently, the semantics for a fault dictate that the program simply continues, and the destination register becomes 0. One could argue the ideal form should have been to abort the program on fault, but that wasn't possible at the time of implementation. We added fault reporting to the program's streams to improve debuggability. Now since we have an ASAN implementation, you can likely run that to catch memory safety problems. An argument against this is that it doesn't help surface a class of issues for production programs. We don't have data on whether stray faults or memory corruption within present pages is the more common occurrence of bugs in the small set of programs using arenas, so it hard to pass any clear judgement. One thing we do lose is faults on NULL-derefs, which are likely common, but Emil had some ideas on that.
Quoted 9 lines
Another thing we lose is the ability to build something like GWP-Asan
[0] that we can run in production programs without paying much of the performance cost by sampling allocations we want to detect bugs for. But between ASAN and Rust-BPF plans, I am not sure how compelling it will be going forward. So while it's sort of sad to lose the ability to fault feedback, it is also non-trivial to enable direct access to arena memory for the kernel while preserving faults (I won't go into the details here) without using fault-safe memcpy to move data from/to arena on the kernel side.
I completely agree with the discussion points, though imo we do not need to make this flag the default if we support it. The complexity is mostly checking whether a kfunc that takes arena arguments accepts the burden of validating them, or if it depends on the new flag to prevent faults. Any new kfuncs should have clear semantics on that, and we can validate proper behavior with selftests.
Whatever we choose, I am strongly in favor of keeping some kind of error reporting when touching the first page in the arena. This has been by far the biggest indicator of bugs, and if we only keep ASAN then we lose our strongest signal for most use cases. This is made even worse by the fact the new flag is incompatible with GWP-Asan, making it too costly to run sanitization at scale.
For the flag, the solution would be to move reserving the low addresses of arenas from libarena to the arena itself. The arena would have a low watermark below which it would retain the existing faulting behavior. The kfunc would bounds check check the arguments to ensure they're not below the low watermark, and fail if they are.
It's not ideal - it adds the burden of bounds checking into the kfunc - but it's reasonable that arena-related kfuncs should take into account the arena's semantics.
Quoted 4 lines
[0]: https://llvm.org/docs/GwpAsan.html
Quoted 1 line
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 76 lines · Emil Tsalapatis
Quoted 64 lines · Kumar Kartikeya Dwivedi
Quoted 27 lines · Tejun Heo
bpf_arena's
kern_vmrange is selectively populated: only allocated pages have PTEs. This catches a narrow class of buggy BPF programs that dereference unmapped arena addresses, but the protection is shallow - within the allocated set there are countless ways for a buggy program to corrupt arena memory.It does, however, impose cost on the kernel side accesses. A kfunc or
struct_opscallback that wants to consume an arena pointer cannot simply load through it; the page may have been freed underneath, so the access has to go throughcopy_from_kernel_nofault(). Out-parameter writes currently have no equivalent.Arena is becoming the primary memory model for BPF programs, and more kfunc /
struct_opssurfaces will want to read and write arena memory directly. The actual answer for catching arena memory bugs is arena ASAN, which addresses all memory access bugs meaningfully. Given that, it's worth offering an opt-in mode that drops the partial fault protection in exchange for cheap direct kernel-side access.Add
BPF_F_ARENA_MAP_ALWAYS. Arenas created with this flag allocate a per-arena "garbage" page and pre-populate every PTE in thekern_vmrange to point at it.arena_alloc_pages()replaces the garbage PTE with a real page;arena_free_pages()restores the garbage PTE instead of clearing.arena_vm_fault()ignores the garbage page so user-side fault semantics are unchanged.Quoted 7 lines
Stores into garbage-backed addresses are silently absorbed; loads return indeterminate bytes. Userspace mappings are unaffected. The flag is opt-in - arenas without it behave exactly as before.
Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>---
If we go down this route, we should probably make this flag the default behavior. Otherwise, we cannot universally enable passing arena memory into kfuncs. Every subsystem will have to check the flag, we'll have to gate being able to pass memory based on the flag's presence, etc., which just adds complexity everywhere. It will eliminate a few patches in this set too. From the programmer's perspective, program behavior isn't changing much, so we can use zeroed page (to guarantee faulting loads return 0) instead of setting the PTE to NULL. While at it we should drop
bpf_prog_report_arena_violation, and its various users.Summarizing past discussions on all this, with more details on various pros/cons:
Currently, the semantics for a fault dictate that the program simply continues, and the destination register becomes 0. One could argue the ideal form should have been to abort the program on fault, but that wasn't possible at the time of implementation. We added fault reporting to the program's streams to improve debuggability. Now since we have an ASAN implementation, you can likely run that to catch memory safety problems. An argument against this is that it doesn't help surface a class of issues for production programs. We don't have data on whether stray faults or memory corruption within present pages is the more common occurrence of bugs in the small set of programs using arenas, so it hard to pass any clear judgement. One thing we do lose is faults on NULL-derefs, which are likely common, but Emil had some ideas on that.
Quoted 9 lines
Another thing we lose is the ability to build something like GWP-Asan
[0] that we can run in production programs without paying much of the performance cost by sampling allocations we want to detect bugs for. But between ASAN and Rust-BPF plans, I am not sure how compelling it will be going forward. So while it's sort of sad to lose the ability to fault feedback, it is also non-trivial to enable direct access to arena memory for the kernel while preserving faults (I won't go into the details here) without using fault-safe memcpy to move data from/to arena on the kernel side.
Quoted 6 lines
I completely agree with the discussion points, though imo we do not need to make this flag the default if we support it. The complexity is mostly checking whether a kfunc that takes arena arguments accepts the burden of validating them, or if it depends on the new flag to prevent faults. Any new kfuncs should have clear semantics on that, and we can validate proper behavior with selftests.
The main problem is accessing the arena or arena flags etc. to decide whether we can read / write the address. It needs to be passed around or retrieved at runtime from within the kfunc. It also makes it conditional on the flag, so depending on whether a flag is set my program will load or not load, since verifier prevents me from passing arena memory as argument to kfunc. In practice, once sched-ext requires it for its programs it will defacto be the default since that's where arenas are used (for now at least). At that point, why bother with the flag.
Quoted 14 lines
Whatever we choose, I am strongly in favor of keeping some kind of error reporting when touching the first page in the arena. This has been by far the biggest indicator of bugs, and if we only keep ASAN then we lose our strongest signal for most use cases. This is made even worse by the fact the new flag is incompatible with GWP-Asan, making it too costly to run sanitization at scale.
For the flag, the solution would be to move reserving the low addresses of arenas from libarena to the arena itself. The arena would have a low watermark below which it would retain the existing faulting behavior. The kfunc would bounds check check the arguments to ensure they're not below the low watermark, and fail if they are.
Quoted 3 lines
It's not ideal - it adds the burden of bounds checking into the kfunc - but it's reasonable that arena-related kfuncs should take into account the arena's semantics.
Another data point to consider is that if we omit this initial faultable region for catching NULL-derefs, we gain the ability to allow passing arena memory into any kfunc where memory arguments are taken, which might be pretty useful. We won't be able to do it if we have the initial region as faultable since we can't rely on kernel writes hitting a page without any checks on the memory region. You must treat arena arguments specially and cannot mix them with other memory arguments.
The other way to keep the faultable region in the beginning would be to emit some assertion/runtime check in the verifier and abort the program if the arena memory being passed into the helper is accessible for the size parameter used for the helper call, or fix it up to some page that is likely to be present.
In practice, if most users set the flag then I think you likely lose the benefit of the default behavior, or cannot rely on it anyway. When it becomes a dependency for passing arena memory in the kernel to helpers, most users will blindly set it.
So in the end, it boils down to whether we think retaining faults (e.g., conditionally for the NULL case) is critical, and whether we have some convincing evidence for it.
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 6 lines
Quoted 5 lines
[0]: https://llvm.org/docs/GwpAsan.html
Quoted 1 line
[...]
Re: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported via bpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it the page_fault_oops() can have a callback into bpf/arena helper similar to kfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.
Emil Tsalapatis
reviewerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
In reply to Kumar Kartikeya DwivediQuoted 84 lines · Kumar Kartikeya Dwivedi
Quoted 76 lines · Emil Tsalapatis
Quoted 64 lines · Kumar Kartikeya Dwivedi
Quoted 27 lines · Tejun Heo
bpf_arena's
kern_vmrange is selectively populated: only allocated pages have PTEs. This catches a narrow class of buggy BPF programs that dereference unmapped arena addresses, but the protection is shallow - within the allocated set there are countless ways for a buggy program to corrupt arena memory.It does, however, impose cost on the kernel side accesses. A kfunc or
struct_opscallback that wants to consume an arena pointer cannot simply load through it; the page may have been freed underneath, so the access has to go throughcopy_from_kernel_nofault(). Out-parameter writes currently have no equivalent.Arena is becoming the primary memory model for BPF programs, and more kfunc /
struct_opssurfaces will want to read and write arena memory directly. The actual answer for catching arena memory bugs is arena ASAN, which addresses all memory access bugs meaningfully. Given that, it's worth offering an opt-in mode that drops the partial fault protection in exchange for cheap direct kernel-side access.Add
BPF_F_ARENA_MAP_ALWAYS. Arenas created with this flag allocate a per-arena "garbage" page and pre-populate every PTE in thekern_vmrange to point at it.arena_alloc_pages()replaces the garbage PTE with a real page;arena_free_pages()restores the garbage PTE instead of clearing.arena_vm_fault()ignores the garbage page so user-side fault semantics are unchanged.Quoted 7 lines
Stores into garbage-backed addresses are silently absorbed; loads return indeterminate bytes. Userspace mappings are unaffected. The flag is opt-in - arenas without it behave exactly as before.
Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Tejun Heo <[email protected]>---
If we go down this route, we should probably make this flag the default behavior. Otherwise, we cannot universally enable passing arena memory into kfuncs. Every subsystem will have to check the flag, we'll have to gate being able to pass memory based on the flag's presence, etc., which just adds complexity everywhere. It will eliminate a few patches in this set too. From the programmer's perspective, program behavior isn't changing much, so we can use zeroed page (to guarantee faulting loads return 0) instead of setting the PTE to NULL. While at it we should drop
bpf_prog_report_arena_violation, and its various users.Summarizing past discussions on all this, with more details on various pros/cons:
Currently, the semantics for a fault dictate that the program simply continues, and the destination register becomes 0. One could argue the ideal form should have been to abort the program on fault, but that wasn't possible at the time of implementation. We added fault reporting to the program's streams to improve debuggability. Now since we have an ASAN implementation, you can likely run that to catch memory safety problems. An argument against this is that it doesn't help surface a class of issues for production programs. We don't have data on whether stray faults or memory corruption within present pages is the more common occurrence of bugs in the small set of programs using arenas, so it hard to pass any clear judgement. One thing we do lose is faults on NULL-derefs, which are likely common, but Emil had some ideas on that.
Quoted 9 lines
Another thing we lose is the ability to build something like GWP-Asan
[0] that we can run in production programs without paying much of the performance cost by sampling allocations we want to detect bugs for. But between ASAN and Rust-BPF plans, I am not sure how compelling it will be going forward. So while it's sort of sad to lose the ability to fault feedback, it is also non-trivial to enable direct access to arena memory for the kernel while preserving faults (I won't go into the details here) without using fault-safe memcpy to move data from/to arena on the kernel side.Quoted 6 lines
I completely agree with the discussion points, though imo we do not need to make this flag the default if we support it. The complexity is mostly checking whether a kfunc that takes arena arguments accepts the burden of validating them, or if it depends on the new flag to prevent faults. Any new kfuncs should have clear semantics on that, and we can validate proper behavior with selftests.
Quoted 10 lines
The main problem is accessing the arena or arena flags etc. to decide whether we can read / write the address. It needs to be passed around or retrieved at runtime from within the kfunc. It also makes it conditional on the flag, so depending on whether a flag is set my program will load or not load, since verifier prevents me from passing arena memory as argument to kfunc. In practice, once sched-ext requires it for its programs it will defacto be the default since that's where arenas are used (for now at least). At that point, why bother with the flag.
While sched_ext is currently the main user of arenas, there are other potential users - the *_ext's being developed in MM, for example If we change the default behavior, we risk making arenas less useful for them until Rust-BPF or an equivalent solution prevents memory access errors further up the BPF software stack. I think ASAN can only partly help in terms of reporting since it won't be on by default.
As an aside, whether Rust-BPF would solve the problem depends on whether we allow/require unsafe Rust to be compilable down to BPF, and how much users end up writing and deploying unsafe Rust. Anecdotally, I've seen arena-based data structure implementations in Rust that are full of unsafe blocks.
Quoted 33 lines
Quoted 14 lines
Whatever we choose, I am strongly in favor of keeping some kind of error reporting when touching the first page in the arena. This has been by far the biggest indicator of bugs, and if we only keep ASAN then we lose our strongest signal for most use cases. This is made even worse by the fact the new flag is incompatible with GWP-Asan, making it too costly to run sanitization at scale.
For the flag, the solution would be to move reserving the low addresses of arenas from libarena to the arena itself. The arena would have a low watermark below which it would retain the existing faulting behavior. The kfunc would bounds check check the arguments to ensure they're not below the low watermark, and fail if they are.
Quoted 3 lines
It's not ideal - it adds the burden of bounds checking into the kfunc - but it's reasonable that arena-related kfuncs should take into account the arena's semantics.
Another data point to consider is that if we omit this initial faultable region for catching NULL-derefs, we gain the ability to allow passing arena memory into any kfunc where memory arguments are taken, which might be pretty useful. We won't be able to do it if we have the initial region as faultable since we can't rely on kernel writes hitting a page without any checks on the memory region. You must treat arena arguments specially and cannot mix them with other memory arguments.
The other way to keep the faultable region in the beginning would be to emit some assertion/runtime check in the verifier and abort the program if the arena memory being passed into the helper is accessible for the size parameter used for the helper call, or fix it up to some page that is likely to be present.
Quoted 8 lines
In practice, if most users set the flag then I think you likely lose the benefit of the default behavior, or cannot rely on it anyway. When it becomes a dependency for passing arena memory in the kernel to helpers, most users will blindly set it.
So in the end, it boils down to whether we think retaining faults (e.g., conditionally for the NULL case) is critical, and whether we have some convincing evidence for it.
Fair enough. While anecdotal, IME it makes a big difference to be able to track NULL dereferences. Explicit checks within the program do help, but at that point we are depending on implementing perfect error handling for the program.
Quoted 12 lines
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 6 lines
Quoted 5 lines
[0]: https://llvm.org/docs/GwpAsan.html
Quoted 1 line
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
In reply to Alexei StarovoitovQuoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 11 lines
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported viabpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it thepage_fault_oops()can have a callback into bpf/arena helper similar tokfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.
Interesting idea. So I guess this page remains mapped once kernel faults on it. I guess we can still reset it to NULL if we alloc and free a page at the same address, so it's just a drop-in to prevent further faults inside the kernel, since emulating instructions is ugly and we're not using asm wrappers that have fixup labels etc. If we end up allocating and freeing something at the same address it will likely get reset to NULL (that would be ideal). But even if this happens in parallel we may fault again and then will just fix up the NULL pte with scratch page again. We can likely also preserve fault reporting into streams when such scratch pages are brought in.
Re: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 23 lines · Kumar Kartikeya Dwivedi
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 11 lines
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported viabpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it thepage_fault_oops()can have a callback into bpf/arena helper similar tokfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.
Quoted 10 lines
Interesting idea. So I guess this page remains mapped once kernel faults on it. I guess we can still reset it to NULL if we alloc and free a page at the same address, so it's just a drop-in to prevent further faults inside the kernel, since emulating instructions is ugly and we're not using asm wrappers that have fixup labels etc. If we end up allocating and freeing something at the same address it will likely get reset to NULL (that would be ideal). But even if this happens in parallel we may fault again and then will just fix up the NULL pte with scratch page again. We can likely also preserve fault reporting into streams when such scratch pages are brought in.
Yep. All makes sense.
The hope is that faults from kfuncs should be rare compared to faults from regular arena bugs.
So the stuck scratch page shouldn't happen often and faults on unmapped will still be seen most of the time.
Emil Tsalapatis
reviewerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 36 lines · Alexei Starovoitov
Quoted 23 lines · Kumar Kartikeya Dwivedi
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 11 lines
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported viabpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it thepage_fault_oops()can have a callback into bpf/arena helper similar tokfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.Quoted 10 lines
Interesting idea. So I guess this page remains mapped once kernel faults on it. I guess we can still reset it to NULL if we alloc and free a page at the same address, so it's just a drop-in to prevent further faults inside the kernel, since emulating instructions is ugly and we're not using asm wrappers that have fixup labels etc. If we end up allocating and freeing something at the same address it will likely get reset to NULL (that would be ideal). But even if this happens in parallel we may fault again and then will just fix up the NULL pte with scratch page again. We can likely also preserve fault reporting into streams when such scratch pages are brought in.
Quoted 5 lines
Yep. All makes sense.
The hope is that faults from kfuncs should be rare compared to faults from regular arena bugs.
So the stuck scratch page shouldn't happen often and faults on unmapped will still be seen most of the time.
This sounds great, it pretty much retains all arena behavior that we care about. The most important part is that it reliably reports the first memory access error, which even now is the only one that is meaningful. The delta with current behavior is that subsequent accesses are not caught, but we don't care about those because they are very likely caused by reading zeros during the initial buggy access.
Would the scratch page be actually mapped into the arena radix tree, or just the pte? Because if it doesn't then I think we don't even need to worry about resetting it from the arena side. Just allocating it at a later time will overwrite the scratch page PTE with new valid page, Until then the page is accessing the scratch page, but again we only care about the first buggy access.
Small nit: Maybe default page instead of scratch page? Scratch page sounds a bit like scratch space but we don't actually use the page to store any data.
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 50 lines · Emil Tsalapatis
Quoted 36 lines · Alexei Starovoitov
Quoted 23 lines · Kumar Kartikeya Dwivedi
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 11 lines
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported viabpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it thepage_fault_oops()can have a callback into bpf/arena helper similar tokfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.Quoted 10 lines
Interesting idea. So I guess this page remains mapped once kernel faults on it. I guess we can still reset it to NULL if we alloc and free a page at the same address, so it's just a drop-in to prevent further faults inside the kernel, since emulating instructions is ugly and we're not using asm wrappers that have fixup labels etc. If we end up allocating and freeing something at the same address it will likely get reset to NULL (that would be ideal). But even if this happens in parallel we may fault again and then will just fix up the NULL pte with scratch page again. We can likely also preserve fault reporting into streams when such scratch pages are brought in.
Quoted 5 lines
Yep. All makes sense.
The hope is that faults from kfuncs should be rare compared to faults from regular arena bugs.
So the stuck scratch page shouldn't happen often and faults on unmapped will still be seen most of the time.This sounds great, it pretty much retains all arena behavior that we care about. The most important part is that it reliably reports the first memory access error, which even now is the only one that is meaningful. The delta with current behavior is that subsequent accesses are not caught, but we don't care about those because they are very likely caused by reading zeros during the initial buggy access.
Quoted 2 lines
Would the scratch page be actually mapped into the arena radix tree, or just the pte? Because if it doesn't then I think we don't even need to
Just the PTE.
Quoted 2 lines
worry about resetting it from the arena side. Just allocating it at a later time will overwrite the scratch page PTE with new valid page,
Which is fine IMO, and how it should be. Alloc and free cycle sets it to NULL, so be it. Users can also do it in parallel, that case will just cause a fault in the kernel again and we'll reset the PTE to the scratch page again.
Quoted 2 lines
Until then the page is accessing the scratch page, but again we only care about the first buggy access.
Right.
Quoted 4 lines
Small nit: Maybe default page instead of scratch page? Scratch page sounds a bit like scratch space but we don't actually use the page to store any data.
It likely should also be zeroed out, to preserve the idea that reading 'faulting' regions returns zeroes. Let's just go with scratch page term.
I think the main idea is we install a page fault handler after the KCSAN one, from the fault handler, use bpf_prog_find_from_stack() to obtain the first program in the stack trace, which will be the one originating the fault inside the kernel. Then make sure the faulting address lies in the prog->aux->arena, (likely including guard pages in its range), and just install the PTE for the zeroed out scratch page at that point and continue.
I thought about various races, to me it seems it should be ok. If parallel installation wins over us, it either installed a valid page replacing scratch PTE, at which point we just let the kernel retry, or installed a scratch page. If it races and replaces existing scratch or valid page with NULL after we checked, we fault again and retry. In any case, either the kernel continues or it ends up faulting again, at which point we can handle the fault again and attempt to fix it up.
We likely need to make sure the existing thing is pte_none() only install if pte_none(), otherwise leave things as is. If racy attempts unmap and set scratch or valid page to none, we will fault again and reinstall. If racy attempts install scratch page or valid page, we let it be as is. More importantly we shouldn't install scratch page over a valid page, I think.
Our PTE installation likely takes the form try_cmpxchg(pte, NULL, scratch_page).
One corner case is that we may have cached scratch page TLB translations for a range we are trying to alloc pages over. Typically the way to eliminate stale TLBs would be to just do flush_tlb_kernel_range(). In this case I wonder whether we just skip it to avoid the cost and let the stale TLB stay, since it likely came due to program passing faultable memory into kernel.
That said, a cheaper fix would be to install PTEs under the lock not with WRITE_ONCE() but xchg() so that we can inspect if we overwrote an entry that had scratch page and only do the extra TLB flush in that case. I would be fine with either option (leaving it as is, or the above), as long as we document it somewhere (either in the commit log or a comment in the code), just so we don't forget.
The main question is, what are the next steps? Do you want to take a stab at implementing this?
Emil Tsalapatis
reviewerRe: [RFC PATCH 2/9] bpf/arena: Add BPF_F_ARENA_MAP_ALWAYS for direct kernel access
Quoted 54 lines · Kumar Kartikeya Dwivedi
Quoted 50 lines · Emil Tsalapatis
Quoted 36 lines · Alexei Starovoitov
Quoted 23 lines · Kumar Kartikeya Dwivedi
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
If not, the best course to me seems to be to make the flag behavior default, and just rely on ASan (and Rust in the future) to prevent any memory safety issues, and drop the stream based feedback on fault, etc.
Quoted 11 lines
Agree that this needs to be new default without new uapi flags. How about we tweak the idea further.
Let all arena pages be unmapped initially. bpf progs will fault on them and will be reported viabpf_streams.
But we also prepare one "scratch page". Let's use this name, since "garbage page" reads too dirty.
When kernel faults we populate pte with that scratch page and let the kernel code retry.
To implement it thepage_fault_oops()can have a callback into bpf/arena helper similar tokfence_handle_page_fault. If fault address is in arena, do kfence_unprotect()-like.Quoted 10 lines
Interesting idea. So I guess this page remains mapped once kernel faults on it. I guess we can still reset it to NULL if we alloc and free a page at the same address, so it's just a drop-in to prevent further faults inside the kernel, since emulating instructions is ugly and we're not using asm wrappers that have fixup labels etc. If we end up allocating and freeing something at the same address it will likely get reset to NULL (that would be ideal). But even if this happens in parallel we may fault again and then will just fix up the NULL pte with scratch page again. We can likely also preserve fault reporting into streams when such scratch pages are brought in.
Quoted 5 lines
Yep. All makes sense.
The hope is that faults from kfuncs should be rare compared to faults from regular arena bugs.
So the stuck scratch page shouldn't happen often and faults on unmapped will still be seen most of the time.This sounds great, it pretty much retains all arena behavior that we care about. The most important part is that it reliably reports the first memory access error, which even now is the only one that is meaningful. The delta with current behavior is that subsequent accesses are not caught, but we don't care about those because they are very likely caused by reading zeros during the initial buggy access.
Quoted 2 lines
Would the scratch page be actually mapped into the arena radix tree, or just the pte? Because if it doesn't then I think we don't even need to
Quoted 8 lines
Just the PTE.
Quoted 2 lines
worry about resetting it from the arena side. Just allocating it at a later time will overwrite the scratch page PTE with new valid page,
Which is fine IMO, and how it should be. Alloc and free cycle sets it to NULL, so be it. Users can also do it in parallel, that case will just cause a fault in the kernel again and we'll reset the PTE to the scratch page again.
Yeah this is why this solution does not interfere with any BPF arena code. The allocator does not need to know about the scratch PTEs at all, it can just allocate over them and automatically turns the address valid.
Quoted 43 lines
Quoted 3 lines
Until then the page is accessing the scratch page, but again we only care about the first buggy access.
Right.
Quoted 4 lines
Small nit: Maybe default page instead of scratch page? Scratch page sounds a bit like scratch space but we don't actually use the page to store any data.
It likely should also be zeroed out, to preserve the idea that reading 'faulting' regions returns zeroes. Let's just go with scratch page term.
I think the main idea is we install a page fault handler after the KCSAN one, from the fault handler, use
bpf_prog_find_from_stack()to obtain the first program in the stack trace, which will be the one originating the fault inside the kernel. Then make sure the faulting address lies in theprog->aux->arena, (likely including guard pages in its range), and just install the PTE for the zeroed out scratch page at that point and continue.I thought about various races, to me it seems it should be ok. If parallel installation wins over us, it either installed a valid page replacing scratch PTE, at which point we just let the kernel retry, or installed a scratch page. If it races and replaces existing scratch or valid page with NULL after we checked, we fault again and retry. In any case, either the kernel continues or it ends up faulting again, at which point we can handle the fault again and attempt to fix it up.
We likely need to make sure the existing thing is
pte_none()only install ifpte_none(), otherwise leave things as is. If racy attempts unmap and set scratch or valid page to none, we will fault again and reinstall. If racy attempts install scratch page or valid page, we let it be as is. More importantly we shouldn't install scratch page over a valid page, I think.Our PTE installation likely takes the form
try_cmpxchg(pte, NULL, scratch_page).One corner case is that we may have cached scratch page TLB translations for a range we are trying to alloc pages over. Typically the way to eliminate stale TLBs would be to just do
flush_tlb_kernel_range(). In this case I wonder whether we just skip it to avoid the cost and let the stale TLB stay, since it likely came due to program passing faultable memory into kernel.
Quoted 7 lines
That said, a cheaper fix would be to install PTEs under the lock not with
WRITE_ONCE()butxchg()so that we can inspect if we overwrote an entry that had scratch page and only do the extra TLB flush in that case. I would be fine with either option (leaving it as is, or the above), as long as we document it somewhere (either in the commit log or a comment in the code), just so we don't forget.
Let's skip the flush. When we hit races like that during the kfunc we should care more about completing the call than about the result since the program is already buggy.
Quoted 2 lines
The main question is, what are the next steps? Do you want to take a stab at implementing this?
Can do, I will send a patch.