bpf: Scope callback arguments to their frame
Subsystems (4)
Ihor Solodrai
reviewerv4 cover bpf: Scope callback arguments to their frame
A callback-calling helper can pass its callback a pointer that is only valid for the callback invocation, and the verifier has no way to express that lifetime.
Two helpers need fixing:
* bpf_user_ringbuf_drain() passes a CONST_PTR_TO_DYNPTR over a sample it releases as soon as the callback returns. Saving this pointer in callback_ctx leaves it pointing into reused kernel stack and allows arbitrary kernel read and write since v7.3.
* bpf_for_each_map_elem() over an array or per-CPU array map passes the address of a u32 held in bpf_for_each_array_elem()'s own frame, leaking four bytes of kernel stack.
Give the callback frame an id and make it both the id and the parent_id of such an argument, so that slices and clones descend from it. Invalidate the id and its descendants when the frame is popped.
---
v3->v4:
* Rebase on bpf-next and include the callback setter cleanups [1] as patches 1-2 (Alexei)
* Pair frame_id in join_func(), and add a selftest for it
* Drop the Fixes: tags and Cc: stable
v3: https://lore.kernel.org/r/[email protected]
v2->v3:
* Replace REF_TYPE_FRAME with a frame_id in bpf_func_state and drop the preparation and refactoring patches (Alexei)
* Rebase on bpf, and add Fixes: tags
v2: https://lore.kernel.org/r/[email protected]
v1->v2:
* Add the refactoring patches: release frame references in one scan, and fold diagnostics into the REF_TYPE_FRAME introduction
* Replace temporary argument mask and deferred register scan with reference acquisition directly in the owning verifier state during callback setup (Eduard)
* Reuse callback_park_dynptr() for the nested drain test
v1: https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/[email protected]
---
Ihor Solodrai (6):
bpf: Share map-backed callback register setup
bpf: Rely on callback frame initialization defaults
bpf: Scope the bpf_user_ringbuf_drain() dynptr to its callback frame selftests/bpf: Cover the user ringbuf callback dynptr lifetime
bpf: Scope the bpf_for_each_map_elem() array key to the callback frame selftests/bpf: Cover callback-frame map key lifetime
include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 2 + kernel/bpf/arraymap.c | 19 ++- kernel/bpf/join.c | 3 + kernel/bpf/states.c | 3 + kernel/bpf/verifier.c | 128 ++++---------- .../selftests/bpf/progs/user_ringbuf_fail.c | 161 ++++++++++++++++++ .../bpf/progs/verifier_iterating_callbacks.c | 91 ++++++++++ 8 files changed, 316 insertions(+), 93 deletions(-)
base-commit: 0021f17128de5dc0b19f1a523f448df2440d8088
Signature · 1 line
2.56.0
Ihor Solodrai
reviewerPATCH v4 1/6 bpf: Share map-backed callback register setup
Map iteration, timer, task-work and RCU callbacks repeat the same register type, offset and map-metadata setup for their map, key and value arguments.
Use one initializer for map-backed callback registers. Leave argument selection, value ids and callback-specific policy in each setter.
No functional change.
--- kernel/bpf/verifier.c | 81 +++++++++++-------------------------------- 1 file changed, 21 insertions(+), 60 deletions(-)
kernel/bpf/verifier.c +21 -60
@@ -12215,6 +12215,16 @@ static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,return check_func_callx_one (env, reg->subprogno, insn_idx); } +static void mark_map_callback_reg (struct bpf_reg_state *reg, +const struct bpf_reg_state *map_reg, +enum bpf_reg_type type) +{ + reg->type = type; +__mark_reg_known_zero (reg); + reg->map_ptr = map_reg->map_ptr; + reg->map_uid = map_reg->map_uid; +} +int map_set_for_each_callback_args (struct bpf_verifier_env *env,struct bpf_func_state *caller,struct bpf_func_state *callee)
@@ -12226,15 +12236,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,*/ callee->regs[BPF_REG_1 ] = caller->regs[BPF_REG_1 ]; - callee->regs[BPF_REG_2 ].type =PTR_TO_MAP_KEY ; -__mark_reg_known_zero (&callee->regs[BPF_REG_2 ]); - callee->regs[BPF_REG_2 ].map_ptr = caller->regs[BPF_REG_1 ].map_ptr; - callee->regs[BPF_REG_2 ].map_uid = caller->regs[BPF_REG_1 ].map_uid; - - callee->regs[BPF_REG_3 ].type =PTR_TO_MAP_VALUE ; -__mark_reg_known_zero (&callee->regs[BPF_REG_3 ]); - callee->regs[BPF_REG_3 ].map_ptr = caller->regs[BPF_REG_1 ].map_ptr; - callee->regs[BPF_REG_3 ].map_uid = caller->regs[BPF_REG_1 ].map_uid; +mark_map_callback_reg (&callee->regs[BPF_REG_2 ], &caller->regs[BPF_REG_1 ],PTR_TO_MAP_KEY ); +mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_1 ],PTR_TO_MAP_VALUE ); callee->regs[BPF_REG_3 ].id = ++env->id_gen;/* pointer to stack or null */
@@ -12325,26 +12328,12 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,struct bpf_func_state *callee, int insn_idx) { -struct bpf_map *map_ptr = caller->regs[BPF_REG_1 ].map_ptr; -u32 map_uid = caller->regs[BPF_REG_1 ].map_uid; -/* bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn); * callback_fn(struct bpf_map *map, void *key, void *value); */ - callee->regs[BPF_REG_1 ].type =CONST_PTR_TO_MAP ; -__mark_reg_known_zero (&callee->regs[BPF_REG_1 ]); - callee->regs[BPF_REG_1 ].map_ptr = map_ptr; - callee->regs[BPF_REG_1 ].map_uid = map_uid; - - callee->regs[BPF_REG_2 ].type =PTR_TO_MAP_KEY ; -__mark_reg_known_zero (&callee->regs[BPF_REG_2 ]); - callee->regs[BPF_REG_2 ].map_ptr = map_ptr; - callee->regs[BPF_REG_2 ].map_uid = map_uid; - - callee->regs[BPF_REG_3 ].type =PTR_TO_MAP_VALUE ; -__mark_reg_known_zero (&callee->regs[BPF_REG_3 ]); - callee->regs[BPF_REG_3 ].map_ptr = map_ptr; - callee->regs[BPF_REG_3 ].map_uid = map_uid; + mark_map_callback_reg(&callee->regs[BPF_REG_1], &caller->regs[BPF_REG_1], CONST_PTR_TO_MAP); +mark_map_callback_reg (&callee->regs[BPF_REG_2 ], &caller->regs[BPF_REG_1 ],PTR_TO_MAP_KEY ); +mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_1 ],PTR_TO_MAP_VALUE ); callee->regs[BPF_REG_3 ].id = ++env->id_gen;/* unused */
@@ -12446,26 +12435,12 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env,struct bpf_func_state *callee, int insn_idx) { -struct bpf_map *map_ptr = caller->regs[BPF_REG_3 ].map_ptr; -u32 map_uid = caller->regs[BPF_REG_3 ].map_uid; -/* * callback_fn(struct bpf_map *map, void *key, void *value); */ - callee->regs[BPF_REG_1 ].type =CONST_PTR_TO_MAP ; -__mark_reg_known_zero (&callee->regs[BPF_REG_1 ]); - callee->regs[BPF_REG_1 ].map_ptr = map_ptr; - callee->regs[BPF_REG_1 ].map_uid = map_uid; - - callee->regs[BPF_REG_2 ].type =PTR_TO_MAP_KEY ; -__mark_reg_known_zero (&callee->regs[BPF_REG_2 ]); - callee->regs[BPF_REG_2 ].map_ptr = map_ptr; - callee->regs[BPF_REG_2 ].map_uid = map_uid; - - callee->regs[BPF_REG_3 ].type =PTR_TO_MAP_VALUE ; -__mark_reg_known_zero (&callee->regs[BPF_REG_3 ]); - callee->regs[BPF_REG_3 ].map_ptr = map_ptr; - callee->regs[BPF_REG_3 ].map_uid = map_uid; + mark_map_callback_reg(&callee->regs[BPF_REG_1], &caller->regs[BPF_REG_3], CONST_PTR_TO_MAP); +mark_map_callback_reg (&callee->regs[BPF_REG_2 ], &caller->regs[BPF_REG_3 ],PTR_TO_MAP_KEY ); +mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_3 ],PTR_TO_MAP_VALUE ); callee->regs[BPF_REG_3 ].id = ++env->id_gen;/* unused */
@@ -12481,26 +12456,12 @@ static int set_rcu_callback_state(struct bpf_verifier_env *env,struct bpf_func_state *callee, int insn_idx) { -struct bpf_map *map_ptr = caller->regs[BPF_REG_2 ].map_ptr; -u32 map_uid = caller->regs[BPF_REG_2 ].map_uid; -/* * callback_fn(struct bpf_map *map, void *key, void *value); */ - callee->regs[BPF_REG_1 ].type =CONST_PTR_TO_MAP ; -__mark_reg_known_zero (&callee->regs[BPF_REG_1 ]); - callee->regs[BPF_REG_1 ].map_ptr = map_ptr; - callee->regs[BPF_REG_1 ].map_uid = map_uid; - - callee->regs[BPF_REG_2 ].type =PTR_TO_MAP_KEY ; -__mark_reg_known_zero (&callee->regs[BPF_REG_2 ]); - callee->regs[BPF_REG_2 ].map_ptr = map_ptr; - callee->regs[BPF_REG_2 ].map_uid = map_uid; - - callee->regs[BPF_REG_3 ].type =PTR_TO_MAP_VALUE ; -__mark_reg_known_zero (&callee->regs[BPF_REG_3 ]); - callee->regs[BPF_REG_3 ].map_ptr = map_ptr; - callee->regs[BPF_REG_3 ].map_uid = map_uid; + mark_map_callback_reg(&callee->regs[BPF_REG_1], &caller->regs[BPF_REG_2], CONST_PTR_TO_MAP); +mark_map_callback_reg (&callee->regs[BPF_REG_2 ], &caller->regs[BPF_REG_2 ],PTR_TO_MAP_KEY ); +mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_2 ],PTR_TO_MAP_VALUE );/* unused */ bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]);
Signature · 1 line
2.56.0
Ihor Solodrai
reviewerPATCH v4 2/6 bpf: Rely on callback frame initialization defaults
In reply to Ihor SolodraiBoth callback entry paths call init_func_state(), which initializes all registers to NOT_INIT and the callback return range to zero.
Callback setters repeat resets for unused registers, and the timer setter repeats the default return range. Leave those defaults to frame initialization and set only arguments and callback-specific state in the setters.
No functional change.
--- kernel/bpf/verifier.c | 31 ------------------------------- 1 file changed, 31 deletions(-)
kernel/bpf/verifier.c +0 -31
@@ -12242,9 +12242,6 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,/* pointer to stack or null */ callee->regs[BPF_REG_4 ] = caller->regs[BPF_REG_3 ]; - -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]);return 0 ; }
@@ -12313,11 +12310,6 @@ static int set_loop_callback_state(struct bpf_verifier_env *env, callee->regs[BPF_REG_1 ].type =SCALAR_VALUE ; callee->regs[BPF_REG_2 ] = caller->regs[BPF_REG_3 ]; -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_3 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); - callee->in_callback_fn = true; callee->callback_ret_range =retval_range (0 ,1 );return 0 ;
@@ -12336,11 +12328,7 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_1 ],PTR_TO_MAP_VALUE ); callee->regs[BPF_REG_3 ].id = ++env->id_gen; -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); callee->in_async_callback_fn = true; - callee->callback_ret_range =retval_range (0 ,0 );return 0 ; }
@@ -12363,10 +12351,6 @@ static int set_find_vma_callback_state(struct bpf_verifier_env *env,/* pointer to stack or null */ callee->regs[BPF_REG_3 ] = caller->regs[BPF_REG_4 ]; - -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); callee->in_callback_fn = true;/* bpf_find_vma() runs the callback under mmap_lock and RCU. */ callee->in_rcu_callback_fn = true;
@@ -12383,15 +12367,9 @@ static int set_user_ringbuf_callback_state(struct bpf_verifier_env *env,* callback_ctx, u64 flags); * callback_fn(const struct bpf_dynptr_t* dynptr, void *callback_ctx); */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_0 ]);mark_dynptr_cb_reg (env, &callee->regs[BPF_REG_1 ],BPF_DYNPTR_TYPE_LOCAL ); callee->regs[BPF_REG_2 ] = caller->regs[BPF_REG_3 ]; -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_3 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); - callee->in_callback_fn = true; callee->callback_ret_range =retval_range (0 ,1 );return 0 ;
@@ -12422,9 +12400,6 @@ static int set_rbtree_add_callback_state(struct bpf_verifier_env *env,mark_reg_graph_node (callee->regs,BPF_REG_2 , &field->graph_root);ref_set_non_owning (env, &callee->regs[BPF_REG_2 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_3 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); callee->in_callback_fn = true; callee->callback_ret_range =retval_range (0 ,1 );return 0 ;
@@ -12443,9 +12418,6 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env,mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_3 ],PTR_TO_MAP_VALUE ); callee->regs[BPF_REG_3 ].id = ++env->id_gen; -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); callee->in_async_callback_fn = true; callee->callback_ret_range =retval_range (S32_MIN ,S32_MAX );return 0 ;
@@ -12463,9 +12435,6 @@ static int set_rcu_callback_state(struct bpf_verifier_env *env,mark_map_callback_reg (&callee->regs[BPF_REG_2 ], &caller->regs[BPF_REG_2 ],PTR_TO_MAP_KEY );mark_map_callback_reg (&callee->regs[BPF_REG_3 ], &caller->regs[BPF_REG_2 ],PTR_TO_MAP_VALUE ); -/* unused */ -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_4 ]); -bpf_mark_reg_not_init (env, &callee->regs[BPF_REG_5 ]); callee->in_async_callback_fn = true; callee->callback_ret_range =retval_range (S32_MIN ,S32_MAX );return 0 ;
Signature · 1 line
2.56.0
Ihor Solodrai
reviewerPATCH v4 3/6 bpf: Scope the bpf_user_ringbuf_drain() dynptr to its callback frame
In reply to Ihor SolodraiThe dynptr argument to a bpf_user_ringbuf_drain() callback points to a descriptor on the helper's stack. The descriptor and the sample it describes are valid only during that callback invocation.
The callback can save the dynptr pointer, a slice, or a clone in the caller's frame and use it after the drain returns. An escaped descriptor pointer can allow arbitrary kernel read/write once the helper's stack is reused.
The verifier assigns the dynptr a type and an id, but does not tie it or its descendants to the callback's lifetime.
Give the callback frame an id and make it both the id and the parent_id of R1, then invalidate that id and everything derived from it when the frame is popped. R1 has to be its own parent: slices take the dynptr's id as their parent_id, and bpf_dynptr_data() clobbers R1, so with a separate dynptr id nothing would connect a slice to the frame.
Pair the frame id in join_func() like any other id, so joining states from different callback frames cannot link a value to the wrong frame.
--- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/join.c | 3 +++ kernel/bpf/states.c | 3 +++ kernel/bpf/verifier.c | 16 ++++++++++++++++ 4 files changed, 24 insertions(+)
include/linux/bpf_verifier.h +2 -0
@@ -477,6 +477,8 @@ struct bpf_func_state {* reused after returns, while this ID is preserved across state clones. */ u32 diag_frame_id; +/* parent_id of the args that are gone when this frame returns */ +u32 frame_id;/* subprog number == index within subprog_info * zero == main subprog */
kernel/bpf/join.c +3 -0
@@ -424,6 +424,9 @@ static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,return false;if (!join_loop_stack (fa, fb))return false; +jid_note (js, fa->frame_id, fb->frame_id); + }else if (!jid_strict_ok (js, fa->frame_id, fb->frame_id)) { +return false; }for (i =0 ; i <MAX_BPF_REG ; i++) {
kernel/bpf/states.c +3 -0
@@ -628,6 +628,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat old->entry_preempt_locks != cur->entry_preempt_locks)return false; +if (!check_ids (old->frame_id, cur->frame_id, &env->idmap_scratch)) +return false; +for (i =0 ; i <MAX_BPF_REG ; i++)if (((1 << i) & live_regs) && !regsafe (env, &old->regs[i], &cur->regs[i], &env->idmap_scratch))
kernel/bpf/verifier.c +16 -0
@@ -12245,6 +12245,16 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,return 0 ; } +/* The arg points into the stack frame of the helper that calls the callback */ +static void mark_frame_scoped_arg (struct bpf_verifier_env *env, +struct bpf_func_state *callee,u32 regno) +{ +if (!callee->frame_id) + callee->frame_id = ++env->id_gen; + callee->regs[regno].id = callee->frame_id; + callee->regs[regno].parent_id = callee->frame_id; +} +static int set_callee_state (struct bpf_verifier_env *env,struct bpf_func_state *caller,struct bpf_func_state *callee,int insn_idx)
@@ -12368,6 +12378,7 @@ static int set_user_ringbuf_callback_state(struct bpf_verifier_env *env,* callback_fn(const struct bpf_dynptr_t* dynptr, void *callback_ctx); */ mark_dynptr_cb_reg (env, &callee->regs[BPF_REG_1 ],BPF_DYNPTR_TYPE_LOCAL ); +mark_frame_scoped_arg (env, callee,BPF_REG_1 ); callee->regs[BPF_REG_2 ] = caller->regs[BPF_REG_3 ]; callee->in_callback_fn = true;
@@ -12606,6 +12617,11 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)if (err)return err;invalidate_stack_slices (env, callee->frameno,INT_MIN ,0 ); +if (callee->frame_id) { + err =invalidate_reference (env, callee->frame_id); +if (err) +return err; + }/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite, * there function call logic would reschedule callback visit.
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 3/6] bpf: Scope the bpf_user_ringbuf_drain() dynptr to its callback frame
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] bpf: missing Fixes trailer for commit 205715673844 in commit message --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] bpf: missing Fixes trailer for commit 205715673844 in commit message
--
commit 3259f9f888525f719a6f09fc51c44281ce232398
Author: Ihor Solodrai <[email protected]>
Quoted 20 lines
bpf: Scope the
bpf_user_ringbuf_drain()dynptr to its callback frameThe dynptr argument to a
bpf_user_ringbuf_drain()callback points to a descriptor on the helper's stack. The descriptor and the sample it describes are valid only during that callback invocation.The callback can save the dynptr pointer, a slice, or a clone in the caller's frame and use it after the drain returns. An escaped descriptor pointer can allow arbitrary kernel read/write once the helper's stack is reused.
The verifier assigns the dynptr a type and an id, but does not tie it or its descendants to the callback's lifetime.
Give the callback frame an id and make it both the id and the
parent_idof R1, then invalidate that id and everything derived from it when the frame is popped. R1 has to be its own parent: slices take the dynptr's id as theirparent_id, andbpf_dynptr_data()clobbers R1, so with a separate dynptr id nothing would connect a slice to the frame.
Quoted 6 lines
Pair the frame id in
join_func()like any other id, so joining states from different callback frames cannot link a value to the wrong frame.Reported-by Nicholas Carlini <[email protected]>Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>
[Severity: Low]
As reported by [email protected] on v1 [1], this fixes a lifetime bug in set_user_ringbuf_callback_state(). Should the commit message include a Fixes tag?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ihor Solodrai
reviewerPATCH v4 4/6 selftests/bpf: Cover the user ringbuf callback dynptr lifetime
In reply to Ihor SolodraiThe dynptr passed to a bpf_user_ringbuf_drain() callback, and values derived from it, must not be used after that callback returns.
Exercise escapes through the callback context, slices returned by bpf_dynptr_data() and bpf_dynptr_slice(), and clones stored in the caller's frame. Also cover an inner callback's dynptr escaping into an outer callback.
A program that leaves either the outer or the inner dynptr in the same slot and only tests it for NULL must still load. This covers the frame id pairing in join_func().
--- .../selftests/bpf/progs/user_ringbuf_fail.c | 161 ++++++++++++++++++ 1 file changed, 161 insertions(+)
tools/testing/selftests/bpf/progs/user_ringbuf_fail.c +161 -0
@@ -1,9 +1,11 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ +#include <stdbool.h> #include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include "bpf_misc.h" +#include "bpf_kfuncs.h" char _license[] SEC("license") = "GPL";
@@ -243,3 +245,162 @@ int user_ringbuf_callback_const_ptr_to_dynptr_reg_off(void *ctx) callback_adjust_bpf_dynptr_reg_off, NULL, 0);return 0 ; } + +/* The sample goes back to the producer as soon as the callback returns. */ +struct dynptr_ctx { +struct bpf_dynptr *saved; +}; + +static long callback_park_dynptr (struct bpf_dynptr *dynptr,void *context) +{ +struct dynptr_ctx *c = context; + + c->saved = dynptr; +return 0 ; +} + +SEC("?raw_tp") +__failure __msg("R3 type=scalar expected=fp, dynptr_ptr") +int user_ringbuf_callback_park_dynptr(void *ctx) +{ +struct dynptr_ctx c = {}; +char buf[8 ] = {}; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_dynptr, &c,0 ); +if (c.saved) +bpf_dynptr_read (buf,sizeof (buf), c.saved,0 ,0 ); +return buf[0 ]; +} + +struct slice_ctx { +char *p; +}; + +static long callback_park_data_slice (struct bpf_dynptr *dynptr,void *context) +{ +struct slice_ctx *c = context; + + c->p =bpf_dynptr_data (dynptr,0 ,8 ); +return 0 ; +} + +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") +int user_ringbuf_callback_park_data_slice(void *ctx) +{ +struct slice_ctx c = {}; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_data_slice, &c,0 ); +if (c.p) +return c.p[0 ]; +return 0 ; +} + +static long callback_park_kfunc_slice (struct bpf_dynptr *dynptr,void *context) +{ +struct slice_ctx *c = context; + + c->p =bpf_dynptr_slice (dynptr,0 ,NULL ,8 ); +return 0 ; +} + +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") +int user_ringbuf_callback_park_kfunc_slice(void *ctx) +{ +struct slice_ctx c = {}; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_kfunc_slice, &c,0 ); +if (c.p) +return c.p[0 ]; +return 0 ; +} + +struct clone_ctx { +struct bpf_dynptr clone; +__u64 armed; +}; + +static long callback_park_clone (struct bpf_dynptr *dynptr,void *context) +{ +struct clone_ctx *c = context; + +bpf_dynptr_clone (dynptr, &c->clone); + c->armed =1 ; +return 0 ; +} + +SEC("?raw_tp") +__failure __msg("Expected an initialized dynptr as R3") +int user_ringbuf_callback_park_clone(void *ctx) +{ +struct clone_ctx c = {}; +char buf[8 ] = {}; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_clone, &c,0 ); +if (c.armed) +bpf_dynptr_read (buf,sizeof (buf), &c.clone,0 ,0 ); +return buf[0 ]; +} + +SEC("?raw_tp") +__failure __msg("Expected an initialized dynptr as R1") +int user_ringbuf_callback_park_clone_then_slice(void *ctx) +{ +struct clone_ctx c = {}; +char *p; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_clone, &c,0 ); +if (c.armed) { + p =bpf_dynptr_data (&c.clone,0 ,8 ); +if (p) +return p[0 ]; + } +return 0 ; +} + +/* An inner drain's dynptr must not escape into the outer callback either. */ +static long callback_park_outer (struct bpf_dynptr *dynptr,void *context) +{ +struct dynptr_ctx inner = {}; +char buf[8 ] = {}; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_dynptr, &inner,0 ); +if (inner.saved) +bpf_dynptr_read (buf,sizeof (buf), inner.saved,0 ,0 ); +return buf[0 ] ?1 :0 ; +} + +SEC("?raw_tp") +__failure __msg("R3 type=scalar expected=fp, dynptr_ptr") +int user_ringbuf_callback_nested_park_inner(void *ctx) +{ +bpf_user_ringbuf_drain (&user_ringbuf, callback_park_outer,NULL ,0 ); +return 0 ; +} + +static long callback_maybe_park (struct bpf_dynptr *dynptr,void *context) +{ +struct dynptr_ctx *c = context; + +if (bpf_get_prandom_u32 () &1 ) + c->saved = dynptr; +return 0 ; +} + +/* inner.saved ends up holding either the outer or the inner dynptr. */ +static long callback_maybe_park_outer (struct bpf_dynptr *dynptr,void *context) +{ +struct dynptr_ctx inner = { .saved = dynptr }; + +bpf_user_ringbuf_drain (&user_ringbuf, callback_maybe_park, &inner,0 ); +return inner.saved ?1 :0 ; +} + +SEC("?raw_tp") +__success +int user_ringbuf_callback_nested_maybe_park (void *ctx) +{ +bpf_user_ringbuf_drain (&user_ringbuf, callback_maybe_park_outer,NULL ,0 ); +return 0 ; +}
Signature · 1 line
2.56.0
Ihor Solodrai
reviewerPATCH v4 5/6 bpf: Scope the bpf_for_each_map_elem() array key to the callback frame
In reply to Ihor Solodraibpf_for_each_map_elem() passes array and per-CPU array callbacks the address of a key on the helper's stack.
A callback can save that pointer in its context and dereference it after the helper returns. Loads through PTR_TO_MAP_KEY are not fault-protected, allowing a four-byte read-only leak of reused kernel stack.
The verifier tracks the pointer's type and key size, but does not tie its lifetime to the callback invocation.
Scope R2 to the callback frame in the array map callback setup so the key and its copies are invalidated on callback return. Leave hash keys and map values unchanged, since their storage is not callback-local.
--- include/linux/bpf.h | 2 ++ kernel/bpf/arraymap.c | 19 +++++++++++++++++-- kernel/bpf/verifier.c | 4 ++-- 3 files changed, 21 insertions(+), 4 deletions(-)
include/linux/bpf.h +2 -0
@@ -3291,6 +3291,8 @@ int bpf_iter_map_check_value_write(const struct bpf_prog *prog,int map_set_for_each_callback_args (struct bpf_verifier_env *env,struct bpf_func_state *caller,struct bpf_func_state *callee); +void mark_frame_scoped_arg (struct bpf_verifier_env *env, +struct bpf_func_state *callee,u32 regno);int bpf_percpu_hash_copy (struct bpf_map *map,void *key,void *value,u64 flags);int bpf_percpu_array_copy (struct bpf_map *map,void *key,void *value,u64 flags);
kernel/bpf/arraymap.c +17 -2
@@ -855,6 +855,21 @@ static u64 array_map_mem_usage(const struct bpf_map *map)return usage; } +static int array_map_set_for_each_callback_args (struct bpf_verifier_env *env, +struct bpf_func_state *caller, +struct bpf_func_state *callee) +{ +int err; + + err =map_set_for_each_callback_args (env, caller, callee); +if (err) +return err; + +/* the key is on the stack of bpf_for_each_array_elem() */ +mark_frame_scoped_arg (env, callee,BPF_REG_2 ); +return 0 ; +} + BTF_ID_LIST_SINGLE(array_map_btf_ids, struct, bpf_array)const struct bpf_map_ops array_map_ops = { .map_meta_equal = array_map_meta_equal,
@@ -875,7 +890,7 @@ const struct bpf_map_ops array_map_ops = { .map_check_btf = array_map_check_btf, .map_lookup_batch = generic_map_lookup_batch, .map_update_batch = generic_map_update_batch, - .map_set_for_each_callback_args = map_set_for_each_callback_args, + .map_set_for_each_callback_args = array_map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_array_elem, .map_mem_usage = array_map_mem_usage, .map_btf_id = &array_map_btf_ids[0],
@@ -900,7 +915,7 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_check_btf = array_map_check_btf, .map_lookup_batch = generic_map_lookup_batch, .map_update_batch = generic_map_update_batch, - .map_set_for_each_callback_args = map_set_for_each_callback_args, + .map_set_for_each_callback_args = array_map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_array_elem, .map_mem_usage = array_map_mem_usage, .map_btf_id = &array_map_btf_ids[0],
kernel/bpf/verifier.c +2 -2
@@ -12246,8 +12246,8 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, }/* The arg points into the stack frame of the helper that calls the callback */ -static void mark_frame_scoped_arg (struct bpf_verifier_env *env, -struct bpf_func_state *callee,u32 regno) +void mark_frame_scoped_arg (struct bpf_verifier_env *env, +struct bpf_func_state *callee,u32 regno) {if (!callee->frame_id) callee->frame_id = ++env->id_gen;
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 5/6] bpf: Scope the bpf_for_each_map_elem() array key to the callback frame
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] bpf: missing Fixes trailer for bpf_for_each_map_elem() array key fix --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] bpf: missing Fixes trailer for bpf_for_each_map_elem() array key fix
--
commit 96799b25d62e44bb557604554c52bc9deffd87cf
Author: Ihor Solodrai <[email protected]>
Quoted 13 lines
bpf: Scope the
bpf_for_each_map_elem()array key to the callback frame
bpf_for_each_map_elem()passes array and per-CPU array callbacks the address of a key on the helper's stack.A callback can save that pointer in its context and dereference it after the helper returns. Loads through
PTR_TO_MAP_KEYare not fault-protected, allowing a four-byte read-only leak of reused kernel stack.The verifier tracks the pointer's type and key size, but does not tie its lifetime to the callback invocation.
Quoted 7 lines
Scope R2 to the callback frame in the array map callback setup so the key and its copies are invalidated on callback return. Leave hash keys and map values unchanged, since their storage is not callback-local.
Reported-by Nicholas Carlini <[email protected]>Suggested-by Alexei Starovoitov <[email protected]>Signed-off-by Ihor Solodrai <[email protected]>
[Severity: Low]
Should this include a Fixes tag for the commit that added array map support to bpf_for_each_map_elem()?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Ihor Solodrai
reviewerPATCH v4 6/6 selftests/bpf: Cover callback-frame map key lifetime
In reply to Ihor SolodraiArray and per-CPU array callbacks to bpf_for_each_map_elem() receive a key on the helper's stack.
Check that saving this key in the callback context and dereferencing it after iteration is rejected.
Also require the same pattern to keep verifying for a hash key and an array value, whose storage outlives the callback. These controls guard the array-only and key-only scope of the lifetime restriction.
--- .../bpf/progs/verifier_iterating_callbacks.c | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+)
tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c +91 -0
@@ -9,6 +9,20 @@ struct {__type (value, __u64); } map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 8); + __type(key, __u32); + __type(value, __u64); +} percpu_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 8); + __type(key, __u32); + __type(value, __u64); +} hash_map SEC(".maps"); +struct { __uint(type, BPF_MAP_TYPE_USER_RINGBUF); __uint(max_entries, 8);
@@ -860,4 +874,81 @@ __naked void check_add_const_regsafe_off(void) : __clobber_common); } +struct key_ctx { +__u32 *key; +}; + +static long park_key_cb (struct bpf_map *map,__u32 *key,__u64 *value, +void *context) +{ +struct key_ctx *c = context; + + c->key = key; +return 0 ; +} + +/* bpf_for_each_array_elem() passes a key from its own stack frame. */ +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") +int array_park_map_key(void *ctx) +{ +struct key_ctx c = {}; + +bpf_for_each_map_elem (&map, park_key_cb, &c,0 ); +if (c.key) +return *c.key; +return 0 ; +} + +SEC("?raw_tp") +__failure __msg("invalid mem access 'scalar'") +int percpu_array_park_map_key(void *ctx) +{ +struct key_ctx c = {}; + +bpf_for_each_map_elem (&percpu_map, park_key_cb, &c,0 ); +if (c.key) +return *c.key; +return 0 ; +} + +/* A hash key points into the element, which outlives the callback. */ +SEC("?raw_tp") +__success +int hash_park_map_key (void *ctx) +{ +struct key_ctx c = {}; + +bpf_for_each_map_elem (&hash_map, park_key_cb, &c,0 ); +if (c.key) +return *c.key; +return 0 ; +} + +struct value_ctx { +__u64 *value; +}; + +static long park_value_cb (struct bpf_map *map,__u32 *key,__u64 *value, +void *context) +{ +struct value_ctx *c = context; + + c->value = value; +return 0 ; +} + +/* Only the key is frame-scoped; the element lives until map teardown. */ +SEC("?raw_tp") +__success +int array_park_map_value (void *ctx) +{ +struct value_ctx c = {}; + +bpf_for_each_map_elem (&map, park_value_cb, &c,0 ); +if (c.value) +return *c.value; +return 0 ; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.56.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 4 lines · On Fri, 13 Nov 2026 14:39:15 -0700 you wrote:
A callback-calling helper can pass its callback a pointer that is only valid for the callback invocation, and the verifier has no way to express that lifetime.
Quoted 8 lines
Two helpers need fixing:
*
bpf_user_ringbuf_drain()passes aCONST_PTR_TO_DYNPTRover a sample it releases as soon as the callback returns. Saving this pointer incallback_ctxleaves it pointing into reused kernel stack and allows arbitrary kernel read and write since v7.3.[...]
Here is the summary with links:
- [bpf-next,v4,1/6] bpf: Share map-backed callback register setup
https://git.kernel.org/bpf/bpf-next/c/acd250fab540 - [bpf-next,v4,2/6] bpf: Rely on callback frame initialization defaults https://git.kernel.org/bpf/bpf-next/c/9c8582c81b7a - [bpf-next,v4,3/6] bpf: Scope the bpf_user_ringbuf_drain() dynptr to its callback frame https://git.kernel.org/bpf/bpf-next/c/e559d46c34cd - [bpf-next,v4,4/6] selftests/bpf: Cover the user ringbuf callback dynptr lifetime https://git.kernel.org/bpf/bpf-next/c/f4e3977ba0f7 - [bpf-next,v4,5/6] bpf: Scope the bpf_for_each_map_elem() array key to the callback frame https://git.kernel.org/bpf/bpf-next/c/3be66bbbaa7d - [bpf-next,v4,6/6] selftests/bpf: Cover callback-frame map key lifetime https://git.kernel.org/bpf/bpf-next/c/9f2c5a659f0b
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html