Exceptions - 1/2
Subsystems (8)
Kumar Kartikeya Dwivedi
maintainerv2 cover Exceptions - 1/2
This series implements the _first_ part of the runtime and verifier support needed to enable BPF exceptions. Exceptions thrown from programs are processed as an immediate exit from the program, which unwinds all the active stack frames until the main stack frame, and returns to the BPF program's caller. The ability to perform this unwinding safely allows the program to test conditions that are always true at runtime but which the verifier has no visibility into.
Thus, it also reduces verification effort by safely terminating redundant paths that can be taken within a program.
The patches to perform runtime resource cleanup during the frame-by-frame unwinding will be posted as a follow-up to this set.
It must be noted that exceptions are not an error handling mechanism for unlikely runtime conditions, but a way to safely terminate the execution of a program in presence of conditions that should never occur at runtime. They are meant to serve higher-level primitives such as program assertions.
The following kfuncs and macros are introduced:
Assertion macros are also introduced, please see patch 13 for their documentation.
/* Description
* Throw a BPF exception from the program, immediately terminating its
* execution and unwinding the stack. The supplied 'cookie' parameter
* will be the return value of the program when an exception is thrown,
* and the default exception callback is used. Otherwise, if an exception
* callback is set using the '__exception_cb(callback)' declaration tag
* on the main program, the 'cookie' parameter will be the callback's only
* input argument.
*
* Thus, in case of default exception callback, 'cookie' is subjected to
* constraints on the program's return value (as with R0 on exit).
* Otherwise, the return value of the marked exception callback will be
* subjected to the same checks.
*
* Note that throwing an exception with lingering resources (locks,
* references, etc.) will lead to a verification error.
*
* Note that callbacks *cannot* call this helper.
* Returns
* Never.
* Throws
* An exception with the specified 'cookie' value.
*/
extern void bpf_throw(u64 cookie) __ksym;
/* This macro must be used to mark the exception callback corresponding to the
* main program. For example:
*
* int exception_cb(u64 cookie) {
* return cookie;
* }
*
* SEC("tc")
* __exception_cb(exception_cb)
* int main_prog(struct __sk_buff *ctx) {
* ...
* return TC_ACT_OK;
* }
*
* Here, exception callback for the main program will be 'exception_cb'. Note
* that this attribute can only be used once, and multiple exception callbacks
* specified for the main program will lead to verification error.
*/
\#define __exception_cb(name) __attribute__((btf_decl_tag("exception_callback:" #name)))
As such, a program can only install an exception handler once for the lifetime of a BPF program, and this handler cannot be changed at runtime. The purpose of the handler is to simply interpret the cookie value supplied by the bpf_throw call, and execute user-defined logic corresponding to it. The primary purpose of allowing a handler is to control the return value of the program. The default handler returns the cookie value passed to bpf_throw when an exception is thrown.
Fixing the handler for the lifetime of the program eliminates tricky and expensive handling in case of runtime changes of the handler callback when programs begin to nest, where it becomes more complex to save and restore the active handler at runtime.
This version of offline unwinding based BPF exceptions is truly zero overhead, with the exception of generation of a default callback which contains a few instructions to return a default return value (0) when no exception callback is supplied by the user.
Callbacks are disallowed from throwing BPF exceptions for now, since such exceptions need to cross the callback helper boundary (and therefore must care about unwinding kernel state), however it is possible to lift this restriction in the future follow-up.
Exceptions terminate propogating at program boundaries, hence both BPF_PROG_TYPE_EXT and tail call targets return to their caller context the return value of the exception callback, in the event that they throw an exception. Thus, exceptions do not cross extension or tail call boundary.
However, this is mostly an implementation choice, and can be changed to suit more user-friendly semantics.
Known issues
------------
* Just asm volatile ("call bpf_throw" :::) does not emit DATASEC .ksyms for bpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Address all comments from Alexei.
* Fix a few bugs and corner cases in the implementations found during testing. Also add new selftests for these cases.
* Reinstate patch to consider ksym.end part of the program (but reworked to cover other corner cases).
* Implement new style of tagging exception callbacks, add libbpf support for the new declaration tag.
* Limit support to 64-bit integer types for assertion macros. The compiler ends up performing shifts or bitwise and operations when finally making use of the value, which defeats the purpose of the macro. On noalu32 mode, the shifts may also happen before use, hurting reliability.
* Comprehensively test assertion macros and their side effects on the verifier state, register bounds, etc.
* Fix a KASAN false positive warning.
RFC v1 -> v1
RFC v1: https://lore.kernel.org/bpf/[email protected]
* Completely rework the unwinding infrastructure to use offline unwinding support.
* Remove the runtime exception state and program rewriting code.
* Make bpf_set_exception_callback idempotent to avoid vexing synchronization and state clobbering issues in presence of program nesting.
* Disable bpf_throw within callback functions, for now.
* Allow bpf_throw in tail call programs and extension programs, removing limitations of rewrite based unwinding.
* Expand selftests.
Kumar Kartikeya Dwivedi (14):
arch/x86: Implement arch_bpf_stack_walk
bpf: Implement support for adding hidden subprogs
bpf: Implement BPF exceptions
bpf: Refactor check_btf_func and split into two phases
bpf: Add support for custom exception callbacks
bpf: Perform CFG walk for exception callback
bpf: Treat first argument as return value for bpf_throw
bpf: Prevent KASAN false positive with bpf_throw
bpf: Detect IP == ksym.end as part of BPF program
bpf: Disallow extensions to exception callbacks
bpf: Fix kfunc callback register type handling
libbpf: Add support for custom exception callbacks selftests/bpf: Add BPF assertion macros selftests/bpf: Add tests for BPF exceptions
arch/x86/net/bpf_jit_comp.c | 118 ++++- include/linux/bpf.h | 8 +- include/linux/bpf_verifier.h | 8 +- include/linux/filter.h | 8 + include/linux/kasan.h | 2 + kernel/bpf/btf.c | 29 +- kernel/bpf/core.c | 29 +- kernel/bpf/helpers.c | 45 ++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 455 +++++++++++++++--- tools/lib/bpf/libbpf.c | 166 ++++++- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../testing/selftests/bpf/bpf_experimental.h | 287 +++++++++++ .../selftests/bpf/prog_tests/exceptions.c | 324 +++++++++++++ .../testing/selftests/bpf/progs/exceptions.c | 368 ++++++++++++++ .../selftests/bpf/progs/exceptions_assert.c | 135 ++++++ .../selftests/bpf/progs/exceptions_ext.c | 59 +++ .../selftests/bpf/progs/exceptions_fail.c | 347 +++++++++++++ 19 files changed, 2271 insertions(+), 121 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_assert.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ext.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_fail.c
base-commit: 2adbb7637fd1fcec93f4680ddb5ddbbd1a91aefb
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/14 arch/x86: Implement arch_bpf_stack_walk
The plumbing for offline unwinding when we throw an exception in programs would require walking the stack, hence introduce a new arch_bpf_stack_walk function. This is provided when the JIT supports exceptions, i.e. bpf_jit_supports_exceptions is true. The arch-specific code is really minimal, hence it should be straightforward to extend this support to other architectures as well, as it reuses the logic of arch_stack_walk, but allowing access to unwind_state data.
Once the stack pointer and frame pointer are known for the main subprog during the unwinding, we know the stack layout and location of any callee-saved registers which must be restored before we return back to the kernel. This handling will be added in the subsequent patches.
Note that while we primarily unwind through BPF frames, which are effectively CONFIG_UNWINDER_FRAME_POINTER, we still need one of this or CONFIG_UNWINDER_ORC to be able to unwind through the bpf_throw frame from which we begin walking the stack. We also require both sp and bp (stack and frame pointers) from the unwind_state structure, which are only available when one of these two options are enabled.
--- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++++++++++++ include/linux/filter.h | 2 ++ kernel/bpf/core.c | 9 +++++++++ 3 files changed, 39 insertions(+)
arch/x86/net/bpf_jit_comp.c +28 -0
@@ -16,6 +16,7 @@#include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/text-patching.h> +#include <asm/unwind.h> static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {
@@ -2913,3 +2914,30 @@ void bpf_jit_free(struct bpf_prog *prog)bpf_prog_unlock_free (prog); } + +bool bpf_jit_supports_exceptions (void ) +{ +/* We unwind through both kernel frames (starting from within bpf_throw +* call) and BPF frames. Therefore we require one of ORC or FP unwinder +* to be enabled to walk kernel frames and reach BPF frames in the stack +* trace. +*/ +return IS_ENABLED (CONFIG_UNWINDER_ORC ) ||IS_ENABLED (CONFIG_UNWINDER_FRAME_POINTER ); +} + +void arch_bpf_stack_walk (bool (*consume_fn)(void *cookie,u64 ip,u64 sp,u64 bp),void *cookie) +{ +#if defined(CONFIG_UNWINDER_ORC ) || defined(CONFIG_UNWINDER_FRAME_POINTER ) +struct unwind_state state; +unsigned long addr; + +for (unwind_start (&state, current,NULL ,NULL ); !unwind_done (&state); +unwind_next_frame (&state)) { + addr =unwind_get_return_address (&state); +if (!addr || !consume_fn (cookie, (u64 )addr, (u64 )state.sp, (u64 )state.bp)) +break ; + } +return ; +#endif +WARN (1 ,"verification of programs using bpf_throw should have failed\n" ); +}
include/linux/filter.h +2 -0
@@ -912,6 +912,8 @@ bool bpf_jit_needs_zext(void);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_far_kfunc_call (void ); +bool bpf_jit_supports_exceptions (void ); +void arch_bpf_stack_walk (bool (*consume_fn)(void *cookie,u64 ip,u64 sp,u64 bp),void *cookie);bool bpf_helper_changes_pkt_data (void *func);static inline bool bpf_dump_raw_ok (const struct cred *cred)
kernel/bpf/core.c +9 -0
@@ -2914,6 +2914,15 @@ int __weak bpf_arch_text_invalidate(void *dst, size_t len)return -ENOTSUPP ; } +bool __weak bpf_jit_supports_exceptions (void ) +{ +return false; +} + +void __weak arch_bpf_stack_walk (bool (*consume_fn)(void *cookie,u64 ip,u64 sp,u64 bp),void *cookie) +{ +} +#ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init (void ) {
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/14 bpf: Implement support for adding hidden subprogs
In reply to Kumar Kartikeya DwivediIntroduce support in the verifier for generating a subprogram and include it as part of a BPF program dynamically after the do_check phase is complete. The first user will be the next patch which generates default exception callbacks if none are set for the program. The phase of invocation will be do_misc_fixups. Note that this is an internal verifier function, and should be used with instruction blocks which uphold the invariants stated in check_subprogs.
Since these subprogs are always appended to the end of the instruction sequence of the program, it becomes relatively inexpensive to do the related adjustments to the subprog_info of the program. Only the fake exit subprogram is shifted forward, making room for our new subprog.
This is useful to insert a new subprogram, get it JITed, and obtain its function pointer. The next patch will use this functionality to insert a default exception callback which will be invoked after unwinding the stack.
Note that these added subprograms are invisible to userspace, and never reported in BPF_OBJ_GET_INFO_BY_ID etc. For now, only a single subprogram is supported, but more can be easily supported in the future.
To this end, two function counts are introduced now, the existing func_cnt, and real_func_cnt, the latter including hidden programs. This allows us to conver the JIT code to use the real_func_cnt for management of resources while syscall path continues working with existing func_cnt.
--- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 3 ++- kernel/bpf/core.c | 12 ++++++------ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 36 +++++++++++++++++++++++++++++++++--- 5 files changed, 43 insertions(+), 11 deletions(-)
include/linux/bpf.h +1 -0
@@ -1377,6 +1377,7 @@ struct bpf_prog_aux {u32 stack_depth;u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ +u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */ u32 func_idx;/* 0 for non-func prog, the index in func array for func prog */ u32 attach_btf_id;/* in-kernel BTF type id to attach to */ u32 ctx_arg_info_size;
include/linux/bpf_verifier.h +2 -1
@@ -587,6 +587,7 @@ struct bpf_verifier_env {u32 used_map_cnt;/* number of used maps */ u32 used_btf_cnt;/* number of used BTF objects */ u32 id_gen;/* used to generate unique reg IDs */ +u32 hidden_subprog_cnt;/* number of hidden subprogs */ bool explore_alu_limits;bool allow_ptr_leaks;bool allow_uninit_stack;
@@ -597,7 +598,7 @@ struct bpf_verifier_env {struct bpf_insn_aux_data *insn_aux_data;/* array of per-insn state */ const struct bpf_line_info *prev_linfo;struct bpf_verifier_log log; -struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS +1 ]; +struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS +2 ];/* max + 2 for the fake and exception subprogs */ union {struct bpf_idmap idmap_scratch;struct bpf_idset idset_scratch;
kernel/bpf/core.c +6 -6
@@ -212,7 +212,7 @@ void bpf_prog_fill_jited_linfo(struct bpf_prog *prog,const struct bpf_line_info *linfo;void **jited_linfo; -if (!prog->aux->jited_linfo) +if (!prog->aux->jited_linfo || prog->aux->func_idx > prog->aux->func_cnt)/* Userspace did not provide linfo */ return ;
@@ -539,7 +539,7 @@ static void bpf_prog_kallsyms_del_subprogs(struct bpf_prog *fp) {int i; -for (i =0 ; i < fp->aux->func_cnt; i++) +for (i =0 ; i < fp->aux->real_func_cnt; i++)bpf_prog_kallsyms_del (fp->aux->func[i]); }
@@ -589,7 +589,7 @@ bpf_prog_ksym_set_name(struct bpf_prog *prog) sym =bin2hex (sym, prog->tag,sizeof (prog->tag));/* prog->aux->name will be ignored if full btf name is available */ -if (prog->aux->func_info_cnt) { +if (prog->aux->func_info_cnt && prog->aux->func_idx < prog->aux->func_info_cnt) { type =btf_type_by_id (prog->aux->btf, prog->aux->func_info[prog->aux->func_idx].type_id); func_name =btf_name_by_offset (prog->aux->btf, type->name_off);
@@ -1208,7 +1208,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,if (!extra_pass) addr =NULL ;else if (prog->aux->func && - off >=0 && off < prog->aux->func_cnt) + off >=0 && off < prog->aux->real_func_cnt) addr = (u8 *)prog->aux->func[off]->bpf_func;else return -EINVAL ;
@@ -2721,7 +2721,7 @@ static void bpf_prog_free_deferred(struct work_struct *work)#endif if (aux->dst_trampoline)bpf_trampoline_put (aux->dst_trampoline); -for (i =0 ; i < aux->func_cnt; i++) { +for (i =0 ; i < aux->real_func_cnt; i++) {/* We can just unlink the subprog poke descriptor table as * it was originally linked to the main program and is also * released along with it.
@@ -2729,7 +2729,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) aux->func[i]->aux->poke_tab =NULL ;bpf_jit_free (aux->func[i]); } -if (aux->func_cnt) { +if (aux->real_func_cnt) {kfree (aux->func);bpf_prog_unlock_free (aux->prog); }else {
kernel/bpf/syscall.c +1 -1
@@ -2746,7 +2746,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)* period before we can tear down JIT memory since symbols * are already exposed under kallsyms. */ -__bpf_prog_put_noref (prog, prog->aux->func_cnt); +__bpf_prog_put_noref (prog, prog->aux->real_func_cnt);return err; free_prog_sec:free_uid (prog->aux->user);
kernel/bpf/verifier.c +33 -3
@@ -15108,7 +15108,8 @@ static void adjust_btf_func(struct bpf_verifier_env *env)if (!aux->func_info)return ; -for (i =0 ; i < env->subprog_cnt; i++) +/* func_info is not available for hidden subprogs */ +for (i =0 ; i < env->subprog_cnt - env->hidden_subprog_cnt; i++) aux->func_info[i].insn_off = env->subprog_info[i].start; }
@@ -18053,7 +18054,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)* the call instruction, as an index for this list */ func[i]->aux->func = func; - func[i]->aux->func_cnt = env->subprog_cnt; + func[i]->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + func[i]->aux->real_func_cnt = env->subprog_cnt; }for (i =0 ; i < env->subprog_cnt; i++) { old_bpf_func = func[i]->bpf_func;
@@ -18099,7 +18101,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->extable = func[0 ]->aux->extable; prog->aux->num_exentries = func[0 ]->aux->num_exentries; prog->aux->func = func; - prog->aux->func_cnt = env->subprog_cnt; + prog->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + prog->aux->real_func_cnt = env->subprog_cnt;bpf_prog_jit_attempt_done (prog);return 0 ; out_free:
@@ -18307,6 +18310,33 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return 0 ; } +/* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ +static __maybe_unused int add_hidden_subprog (struct bpf_verifier_env *env,struct bpf_insn *patch,int len) +{ +struct bpf_subprog_info *info = env->subprog_info; +int cnt = env->subprog_cnt; +struct bpf_prog *prog; + +/* We only reserve one slot for hidden subprogs in subprog_info. */ +if (env->hidden_subprog_cnt) { +verbose (env,"verifier internal error: only one hidden subprog supported\n" ); +return -EFAULT ; + } +/* We're not patching any existing instruction, just appending the new +* ones for the hidden subprog. Hence all of the adjustment operations +* in bpf_patch_insn_data are no-ops. +*/ + prog =bpf_patch_insn_data (env, env->prog->len -1 , patch, len); +if (!prog) +return -ENOMEM ; + env->prog = prog; + info[cnt +1 ].start = info[cnt].start; + info[cnt].start = prog->len - len +1 ; + env->subprog_cnt++; + env->hidden_subprog_cnt++; +return 0 ; +} +/* Do various post-verification rewrites in a single program pass. * These rewrites simplify JIT and interpreter implementations. */
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 3/14 bpf: Implement BPF exceptions
In reply to Kumar Kartikeya DwivediThis patch implements BPF exceptions, and introduces a bpf_throw kfunc to allow programs to throw exceptions during their execution at runtime. A bpf_throw invocation is treated as an immediate termination of the program, returning back to its caller within the kernel, unwinding all stack frames.
This allows the program to simplify its implementation, by testing for runtime conditions which the verifier has no visibility into, and assert that they are true. In case they are not, the program can simply throw an exception from the other branch.
BPF exceptions are explicitly *NOT* an unlikely slowpath error handling primitive, and this objective has guided design choices of the implementation of the them within the kernel (with the bulk of the cost for unwinding the stack offloaded to the bpf_throw kfunc).
The implementation of this mechanism requires use of add_hidden_subprog mechanism introduced in the previous patch, which generates a couple of instructions to move R1 to R0 and exit. The JIT then rewrites the prologue of this subprog to take the stack pointer and frame pointer as inputs and reset the stack frame, popping all callee-saved registers saved by the main subprog. The bpf_throw function then walks the stack at runtime, and invokes this exception subprog with the stack and frame pointers as parameters.
Reviewers must take note that currently the main program is made to save all callee-saved registers on x86_64 during entry into the program. This is because we must do an equivalent of a lightweight context switch when unwinding the stack, therefore we need the callee-saved registers of the caller of the BPF program to be able to return with a sane state.
Note that we have to additionally handle r12, even though it is not used by the program, because when throwing the exception the program makes an entry into the kernel which could clobber r12 after saving it on the stack. To be able to preserve the value we received on program entry, we push r12 and restore it from the generated subprogram when unwinding the stack.
For now, bpf_throw invocation fails when lingering resources or locks exist in that path of the program. In a future followup, bpf_throw will be extended to perform frame-by-frame unwinding to release lingering resources for each stack frame, removing this limitation.
--- arch/x86/net/bpf_jit_comp.c | 90 ++++++++++++-- include/linux/bpf.h | 3 + include/linux/bpf_verifier.h | 4 + include/linux/filter.h | 6 + kernel/bpf/core.c | 2 +- kernel/bpf/helpers.c | 38 ++++++ kernel/bpf/verifier.c | 116 +++++++++++++++--- .../testing/selftests/bpf/bpf_experimental.h | 16 +++ 8 files changed, 248 insertions(+), 27 deletions(-)
arch/x86/net/bpf_jit_comp.c +78 -12
@@ -18,6 +18,8 @@#include <asm/text-patching.h> #include <asm/unwind.h> +static bool all_callee_regs_used[4 ] = {true, true, true, true}; +static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {if (len ==1 )
@@ -256,6 +258,14 @@ struct jit_context {/* Number of bytes that will be skipped on tailcall */ #define X86_TAIL_CALL_OFFSET (11 + ENDBR_INSN_SIZE) +static void push_r12 (u8 **pprog) +{ +u8 *prog = *pprog; + +EMIT2 (0x41 ,0x54 );/* push r12 */ + *pprog = prog; +} +static void push_callee_regs (u8 **pprog,bool *callee_regs_used) {u8 *prog = *pprog;
@@ -271,6 +281,14 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static void pop_r12 (u8 **pprog) +{ +u8 *prog = *pprog; + +EMIT2 (0x41 ,0x5C );/* pop r12 */ + *pprog = prog; +} +static void pop_callee_regs (u8 **pprog,bool *callee_regs_used) {u8 *prog = *pprog;
@@ -292,7 +310,8 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)* while jumping to another program */ static void emit_prologue (u8 **pprog,u32 stack_depth,bool ebpf_from_cbpf, -bool tail_call_reachable,bool is_subprog) +bool tail_call_reachable,bool is_subprog, +bool is_exception_cb) {u8 *prog = *pprog;
@@ -308,8 +327,22 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf,else EMIT2 (0x66 ,0x90 );/* nop2 */ } -EMIT1 (0x55 );/* push rbp */ -EMIT3 (0x48 ,0x89 ,0xE5 );/* mov rbp, rsp */ +/* Exception callback receives FP as second parameter */ +if (is_exception_cb) { +EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ +EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +/* The main frame must have exception_boundary as true, so we +* first restore those callee-saved regs from stack, before +* reusing the stack frame. +*/ +pop_callee_regs (&prog, all_callee_regs_used); +pop_r12 (&prog); +/* Reset the stack frame. */ +EMIT3 (0x48 ,0x89 ,0xEC );/* mov rsp, rbp */ + }else { +EMIT1 (0x55 );/* push rbp */ +EMIT3 (0x48 ,0x89 ,0xE5 );/* mov rbp, rsp */ + }/* X86_TAIL_CALL_OFFSET is here */ EMIT_ENDBR ();
@@ -468,7 +501,8 @@ static void emit_return(u8 **pprog, u8 *ip)* goto *(prog->bpf_func + prologue_size); * out: */ -static void emit_bpf_tail_call_indirect (u8 **pprog,bool *callee_regs_used, +static void emit_bpf_tail_call_indirect (struct bpf_prog *bpf_prog, +u8 **pprog,bool *callee_regs_used,u32 stack_depth,u8 *ip,struct jit_context *ctx) {
@@ -518,7 +552,12 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, offset = ctx->tail_call_indirect_label - (prog +2 - start);EMIT2 (X86_JE , offset);/* je out */ -pop_callee_regs (&prog, callee_regs_used); +if (bpf_prog->aux->exception_boundary) { +pop_callee_regs (&prog, all_callee_regs_used); +pop_r12 (&prog); + }else { +pop_callee_regs (&prog, callee_regs_used); + }EMIT1 (0x58 );/* pop rax */ if (stack_depth)
@@ -542,7 +581,8 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, *pprog = prog; } -static void emit_bpf_tail_call_direct (struct bpf_jit_poke_descriptor *poke, +static void emit_bpf_tail_call_direct (struct bpf_prog *bpf_prog, +struct bpf_jit_poke_descriptor *poke,u8 **pprog,u8 *ip,bool *callee_regs_used,u32 stack_depth,struct jit_context *ctx)
@@ -571,7 +611,13 @@ static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke,emit_jump (&prog, (u8 *)poke->tailcall_target +X86_PATCH_SIZE , poke->tailcall_bypass); -pop_callee_regs (&prog, callee_regs_used); +if (bpf_prog->aux->exception_boundary) { +pop_callee_regs (&prog, all_callee_regs_used); +pop_r12 (&prog); + }else { +pop_callee_regs (&prog, callee_regs_used); + } +EMIT1 (0x58 );/* pop rax */ if (stack_depth)EMIT3_off32 (0x48 ,0x81 ,0xC4 ,round_up (stack_depth,8 ));
@@ -1042,8 +1088,21 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageemit_prologue (&prog, bpf_prog->aux->stack_depth,bpf_prog_was_classic (bpf_prog), tail_call_reachable, - bpf_prog->aux->func_idx !=0 ); -push_callee_regs (&prog, callee_regs_used); + bpf_prog->aux->func_idx !=0 , bpf_prog->aux->exception_cb); +/* Exception callback will clobber callee regs for its own use, and +* restore the original callee regs from main prog's stack frame. +*/ +if (bpf_prog->aux->exception_boundary) { +/* We also need to save r12, which is not mapped to any BPF +* register, as we throw after entry into the kernel, which may +* overwrite r12. +*/ +push_r12 (&prog); +push_callee_regs (&prog, all_callee_regs_used); + }else { +push_callee_regs (&prog, callee_regs_used); + } + ilen = prog - temp;if (rw_image)
@@ -1642,13 +1701,15 @@ st: if (is_imm8(insn->off))case BPF_JMP |BPF_TAIL_CALL :if (imm32) -emit_bpf_tail_call_direct (&bpf_prog->aux->poke_tab[imm32 -1 ], +emit_bpf_tail_call_direct (bpf_prog, + &bpf_prog->aux->poke_tab[imm32 -1 ], &prog, image + addrs[i -1 ], callee_regs_used, bpf_prog->aux->stack_depth, ctx);else - emit_bpf_tail_call_indirect(&prog, + emit_bpf_tail_call_indirect(bpf_prog, + &prog, callee_regs_used, bpf_prog->aux->stack_depth, image + addrs[i - 1],
@@ -1901,7 +1962,12 @@ st: if (is_imm8(insn->off)) seen_exit = true;/* Update cleanup_addr */ ctx->cleanup_addr = proglen; -pop_callee_regs (&prog, callee_regs_used); +if (bpf_prog->aux->exception_boundary) { +pop_callee_regs (&prog, all_callee_regs_used); +pop_r12 (&prog); + }else { +pop_callee_regs (&prog, callee_regs_used); + }EMIT1 (0xC9 );/* leave */ emit_return (&prog, image + addrs[i -1 ] + (prog - temp));break ;
include/linux/bpf.h +3 -0
@@ -1398,6 +1398,8 @@ struct bpf_prog_aux {bool sleepable;bool tail_call_reachable;bool xdp_has_frags; +bool exception_cb; +bool exception_boundary;/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto;/* function name for valid attach_btf_id */
@@ -1420,6 +1422,7 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; +unsigned int (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp);#ifdef CONFIG_SECURITY void *security;#endif
include/linux/bpf_verifier.h +4 -0
@@ -540,7 +540,9 @@ struct bpf_subprog_info {bool has_tail_call;bool tail_call_reachable;bool has_ld_abs; +bool is_cb;bool is_async_cb; +bool is_exception_cb; };struct bpf_verifier_env ;
@@ -588,6 +590,7 @@ struct bpf_verifier_env {u32 used_btf_cnt;/* number of used BTF objects */ u32 id_gen;/* used to generate unique reg IDs */ u32 hidden_subprog_cnt;/* number of hidden subprogs */ +int exception_callback_subprog;bool explore_alu_limits;bool allow_ptr_leaks;bool allow_uninit_stack;
@@ -595,6 +598,7 @@ struct bpf_verifier_env {bool bypass_spec_v1;bool bypass_spec_v4;bool seen_direct_write; +bool seen_exception;struct bpf_insn_aux_data *insn_aux_data;/* array of per-insn state */ const struct bpf_line_info *prev_linfo;struct bpf_verifier_log log;
include/linux/filter.h +6 -0
@@ -1129,6 +1129,7 @@ const char *__bpf_address_lookup(unsigned long addr, unsigned long *size,bool is_bpf_text_address (unsigned long addr);int bpf_get_kallsym (unsigned int symnum,unsigned long *value,char *type,char *sym); +struct bpf_prog *bpf_prog_ksym_find (unsigned long addr);static inline const char * bpf_address_lookup(unsigned long addr, unsigned long *size,
@@ -1196,6 +1197,11 @@ static inline int bpf_get_kallsym(unsigned int symnum, unsigned long *value,return -ERANGE ; } +static inline struct bpf_prog *bpf_prog_ksym_find (unsigned long addr) +{ +return NULL ; +} +static inline const char *bpf_address_lookup (unsigned long addr,unsigned long *size,unsigned long *off,char **modname,char *sym)
kernel/bpf/core.c +1 -1
@@ -733,7 +733,7 @@ bool is_bpf_text_address(unsigned long addr)return ret; } -static struct bpf_prog *bpf_prog_ksym_find (unsigned long addr) +struct bpf_prog *bpf_prog_ksym_find (unsigned long addr) {struct bpf_ksym *ksym =bpf_ksym_find (addr);
kernel/bpf/helpers.c +38 -0
@@ -2429,6 +2429,43 @@ __bpf_kfunc void bpf_rcu_read_unlock(void)rcu_read_unlock (); } +struct bpf_throw_ctx { +struct bpf_prog_aux *aux; +u64 sp; +u64 bp; +int cnt; +}; + +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) +{ +struct bpf_throw_ctx *ctx = cookie; +struct bpf_prog *prog; + +if (!is_bpf_text_address (ip)) +return !ctx->cnt; + prog =bpf_prog_ksym_find (ip); + ctx->cnt++; +if (!prog->aux->id) +return true; + ctx->aux = prog->aux; + ctx->sp = sp; + ctx->bp = bp; +return false; +} + +__bpf_kfunc void bpf_throw(u64 cookie) +{ +struct bpf_throw_ctx ctx = {}; + +arch_bpf_stack_walk (bpf_stack_walker, &ctx); +WARN_ON_ONCE (!ctx.aux); +if (ctx.aux) +WARN_ON_ONCE (!ctx.aux->exception_boundary); +WARN_ON_ONCE (!ctx.bp); +WARN_ON_ONCE (!ctx.cnt); + ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); +} +__diag_pop ();BTF_SET8_START (generic_btf_ids)
@@ -2456,6 +2493,7 @@ BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)#endif BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_throw)BTF_SET8_END (generic_btf_ids) static const struct btf_kfunc_id_set generic_kfunc_set = {
kernel/bpf/verifier.c +102 -14
262 lines not on this page · raw message
tools/testing/selftests/bpf/bpf_experimental.h +16 -0
@@ -131,4 +131,20 @@ extern int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *nod*/ extern struct bpf_rb_node *bpf_rbtree_first (struct bpf_rb_root *root) __ksym; +/* Description +* Throw a BPF exception from the program, immediately terminating its +* execution and unwinding the stack. The supplied 'cookie' parameter +* will be the return value of the program when an exception is thrown. +* +* Note that throwing an exception with lingering resources (locks, +* references, etc.) will lead to a verification error. +* +* Note that callbacks *cannot* call this helper. +* Returns +* Never. +* Throws +* An exception with the specified 'cookie' value. +*/ +extern void bpf_throw(u64 cookie) __ksym; +#endif
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 03/14] bpf: Implement BPF exceptions
Quoted 7 lines · Kumar Kartikeya Dwivedi
kernel/bpf/helpers.c +7
-0+ +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) +{ +struct bpf_throw_ctx *ctx = cookie; +struct bpf_prog *prog; + +if (!is_bpf_text_address (ip))
kernel/bpf/helpers.c __bpf_kfunc void bpf_rcu_read_unlock(void) · line 2454
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+return !ctx->cnt; + prog =bpf_prog_ksym_find (ip); + ctx->cnt++; +if (!prog->aux->id) +return true; + ctx->aux = prog->aux; + ctx->sp = sp; + ctx->bp = bp; +return false; +}
Took me some time to understand what !prog->aux->id is doing. Let's add a helper: is_subprog() and check:prog->aux->func_idx != 0
since that's what arm64, x64, s390 JITs are using.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 03/14] bpf: Implement BPF exceptions
Quoted 16 lines · Alexei Starovoitov
Quoted 14 lines · Kumar Kartikeya Dwivedi
kernel/bpf/helpers.c +14
-0+ +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) +{ +struct bpf_throw_ctx *ctx = cookie; +struct bpf_prog *prog; + +if (!is_bpf_text_address (ip)) +return !ctx->cnt; + prog =bpf_prog_ksym_find (ip); + ctx->cnt++; +if (!prog->aux->id) +return true; + ctx->aux = prog->aux; + ctx->sp = sp;
Quoted 8 lines
kernel/bpf/helpers.c __bpf_kfunc void bpf_rcu_read_unlock(void) · line 2454
Quoted 3 lines
kernel/bpf/helpers.c +3
-0+ ctx->bp = bp; +return false; +}Took me some time to understand what !prog->aux->id is doing. Let's add a helper:
is_subprog()and check:prog->aux->func_idx!= 0
since that's what arm64, x64, s390 JITs are using.
Ack, I will fix it.
Kumar Kartikeya Dwivedi
maintainerPATCH v2 4/14 bpf: Refactor check_btf_func and split into two phases
In reply to Kumar Kartikeya DwivediThis patch splits the check_btf_info's check_btf_func check into two separate phases. The first phase sets up the BTF and prepares func_info, but does not perform any validation of required invariants for subprogs just yet. This is left to the second phase, which happens where check_btf_info executes currently, and performs the line_info and CO-RE relocation.
The reason to perform this split is to obtain the userspace supplied func_info information before we perform the add_subprog call, where we would now require finding and adding subprogs that may not have a bpf_pseudo_call or bpf_pseudo_func instruction in the program.
We require this as we want to enable userspace to supply exception callbacks that can override the default hidden subprogram generated by the verifier (which performs a hardcoded action). In such a case, the exception callback may never be referenced in an instruction, but will still be suitably annotated (by way of BTF declaration tags). For finding this exception callback, we would require the program's BTF information, and the supplied func_info information which maps BTF type IDs to subprograms.
Since the exception callback won't actually be referenced through instructions, later checks in check_cfg and do_check_subprogs will not verify the subprog. This means that add_subprog needs to add them in the add_subprog_and_kfunc phase before we move forward, which is why the BTF and func_info are required at that point.
--- kernel/bpf/verifier.c | 128 +++++++++++++++++++++++++++++++++--------- 1 file changed, 100 insertions(+), 28 deletions(-)
kernel/bpf/verifier.c +100 -28
@@ -15013,20 +15013,18 @@ static int check_abnormal_return(struct bpf_verifier_env *env)#define MIN_BPF_FUNCINFO_SIZE 8#define MAX_FUNCINFO_REC_SIZE 252 -static int check_btf_func (struct bpf_verifier_env *env, -const union bpf_attr *attr, -bpfptr_t uattr) +static int check_btf_func_early (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) { -const struct btf_type *type, *func_proto, *ret_type; -u32 i, nfuncs, urec_size, min_size;u32 krec_size =sizeof (struct bpf_func_info ); +const struct btf_type *type, *func_proto; +u32 i, nfuncs, urec_size, min_size;struct bpf_func_info *krecord; -struct bpf_func_info_aux *info_aux =NULL ;struct bpf_prog *prog;const struct btf *btf; -bpfptr_t urecord;u32 prev_offset =0 ; -bool scalar_return; +bpfptr_t urecord;int ret = -ENOMEM ; nfuncs = attr->func_info_cnt;
@@ -15036,11 +15034,6 @@ static int check_btf_func(struct bpf_verifier_env *env,return 0 ; } -if (nfuncs != env->subprog_cnt) { -verbose (env,"number of funcs in func_info doesn't match number of subprogs\n" ); -return -EINVAL ; - } - urec_size = attr->func_info_rec_size; if (urec_size < MIN_BPF_FUNCINFO_SIZE || urec_size > MAX_FUNCINFO_REC_SIZE ||
@@ -15058,9 +15051,6 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord =kvcalloc (nfuncs, krec_size,GFP_KERNEL | __GFP_NOWARN);if (!krecord)return -ENOMEM ; - info_aux =kcalloc (nfuncs,sizeof (*info_aux),GFP_KERNEL | __GFP_NOWARN); -if (!info_aux) - goto err_free;for (i =0 ; i < nfuncs; i++) { ret =bpf_check_uarg_tail_zero (urecord, krec_size, urec_size);
@@ -15099,11 +15089,6 @@ static int check_btf_func(struct bpf_verifier_env *env, goto err_free; } -if (env->subprog_info[i].start != krecord[i].insn_off) { -verbose (env,"func_info BTF section doesn't match subprog layout in BPF program\n" ); - goto err_free; - } -/* check type_id */ type =btf_type_by_id (btf, krecord[i].type_id);if (!type || !btf_type_is_func (type)) {
@@ -15111,12 +15096,80 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord[i].type_id); goto err_free; } - info_aux[i].linkage =BTF_INFO_VLEN (type->info); func_proto =btf_type_by_id (btf, type->type);if (unlikely (!func_proto || !btf_type_is_func_proto (func_proto)))/* btf_func_check() already verified it during BTF load */ goto err_free; + + prev_offset = krecord[i].insn_off; +bpfptr_add (&urecord, urec_size); + } + + prog->aux->func_info = krecord; + prog->aux->func_info_cnt = nfuncs; +return 0 ; + +err_free: +kvfree (krecord); +return ret; +} + +static int check_btf_func (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) +{ +const struct btf_type *type, *func_proto, *ret_type; +u32 i, nfuncs, urec_size, min_size; +u32 krec_size =sizeof (struct bpf_func_info ); +struct bpf_func_info *krecord; +struct bpf_func_info_aux *info_aux =NULL ; +struct bpf_prog *prog; +const struct btf *btf; +bpfptr_t urecord; +u32 prev_offset =0 ; +bool scalar_return; +int ret = -ENOMEM ; + + nfuncs = attr->func_info_cnt; +if (!nfuncs) { +if (check_abnormal_return (env)) +return -EINVAL ; +return 0 ; + } +if (nfuncs != env->subprog_cnt) { +verbose (env,"number of funcs in func_info doesn't match number of subprogs\n" ); +return -EINVAL ; + } + + urec_size = attr->func_info_rec_size; + + prog = env->prog; + btf = prog->aux->btf; + + urecord =make_bpfptr (attr->func_info, uattr.is_kernel); + min_size =min_t (u32, krec_size, urec_size); + + krecord = prog->aux->func_info; + info_aux =kcalloc (nfuncs,sizeof (*info_aux),GFP_KERNEL | __GFP_NOWARN); +if (!info_aux) +return -ENOMEM ; + +for (i =0 ; i < nfuncs; i++) { +/* check insn_off */ + ret = -EINVAL ; + +if (env->subprog_info[i].start != krecord[i].insn_off) { +verbose (env,"func_info BTF section doesn't match subprog layout in BPF program\n" ); + goto err_free; + } + +/* Already checked type_id */ + type =btf_type_by_id (btf, krecord[i].type_id); + info_aux[i].linkage =BTF_INFO_VLEN (type->info); +/* Already checked func_proto */ + func_proto =btf_type_by_id (btf, type->type); + ret_type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); scalar_return =btf_type_is_small_int (ret_type) ||btf_is_any_enum (ret_type);
@@ -15133,13 +15186,10 @@ static int check_btf_func(struct bpf_verifier_env *env,bpfptr_add (&urecord, urec_size); } - prog->aux->func_info = krecord; - prog->aux->func_info_cnt = nfuncs; prog->aux->func_info_aux = info_aux;return 0 ; err_free: -kvfree (krecord);kfree (info_aux);return ret; }
@@ -15357,9 +15407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,return err; } -static int check_btf_info (struct bpf_verifier_env *env, -const union bpf_attr *attr, -bpfptr_t uattr) +static int check_btf_info_early (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) {struct btf *btf;int err;
@@ -15379,6 +15429,24 @@ static int check_btf_info(struct bpf_verifier_env *env, } env->prog->aux->btf = btf; + err =check_btf_func_early (env, attr, uattr); +if (err) +return err; +return 0 ; +} + +static int check_btf_info (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) +{ +int err; + +if (!attr->func_info_cnt && !attr->line_info_cnt) { +if (check_abnormal_return (env)) +return -EINVAL ; +return 0 ; + } + err =check_btf_func (env, attr, uattr);if (err)return err;
@@ -19842,6 +19910,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3if (!env->explored_states) goto skip_full_check; + ret =check_btf_info_early (env, attr, uattr); +if (ret <0 ) + goto skip_full_check; + ret =add_subprog_and_kfunc (env);if (ret <0 ) goto skip_full_check;
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 5/14 bpf: Add support for custom exception callbacks
In reply to Kumar Kartikeya DwivediBy default, the subprog generated by the verifier to handle a thrown exception hardcodes a return value of 0. To allow user-defined logic and modification of the return value when an exception is thrown, introduce the 'exception_callback:' declaration tag, which marks a callback as the default exception handler for the program.
The format of the declaration tag is 'exception_callback:<value>', where <value> is the name of the exception callback. Each main program can be tagged using this BTF declaratiion tag to associate it with an exception callback. In case the tag is absent, the default callback is used.
As such, the exception callback cannot be modified at runtime, only set during verification.
Allowing modification of the callback for the current program execution at runtime leads to issues when the programs begin to nest, as any per-CPU state maintaing this information will have to be saved and restored. We don't want it to stay in bpf_prog_aux as this takes a global effect for all programs. An alternative solution is spilling the callback pointer at a known location on the program stack on entry, and then passing this location to bpf_throw as a parameter.
However, since exceptions are geared more towards a use case where they are ideally never invoked, optimizing for this use case and adding to the complexity has diminishing returns.
--- include/linux/bpf.h | 4 +- include/linux/bpf_verifier.h | 1 + kernel/bpf/btf.c | 29 +++-- kernel/bpf/verifier.c | 118 ++++++++++++++++-- .../testing/selftests/bpf/bpf_experimental.h | 31 ++++- 5 files changed, 165 insertions(+), 18 deletions(-)
include/linux/bpf.h +3 -1
@@ -2364,9 +2364,11 @@ int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog,int btf_check_subprog_call (struct bpf_verifier_env *env,int subprog,struct bpf_reg_state *regs);int btf_prepare_func_args (struct bpf_verifier_env *env,int subprog, -struct bpf_reg_state *reg); +struct bpf_reg_state *reg,bool is_ex_cb);int btf_check_type_match (struct bpf_verifier_log *log,const struct bpf_prog *prog,struct btf *btf,const struct btf_type *t); +const char *btf_find_decl_tag_value (const struct btf *btf,const struct btf_type *pt, +int comp_idx,const char *tag_key);struct bpf_prog *bpf_prog_by_id (u32 id);struct bpf_link *bpf_link_by_id (u32 id);
include/linux/bpf_verifier.h +1 -0
@@ -300,6 +300,7 @@ struct bpf_func_state {bool in_callback_fn;struct tnum callback_ret_range;bool in_async_callback_fn; +bool in_exception_callback_fn;/* The following fields should be last. See copy_func_state() */ int acquired_refs;
kernel/bpf/btf.c +21 -8
@@ -3308,10 +3308,10 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,return BTF_FIELD_FOUND ; } -static const char *btf_find_decl_tag_value (const struct btf *btf, -const struct btf_type *pt, -int comp_idx,const char *tag_key) +const char *btf_find_decl_tag_value (const struct btf *btf,const struct btf_type *pt, +int comp_idx,const char *tag_key) { +const char *value =NULL ;int i;for (i =1 ; i <btf_nr_types (btf); i++) {
@@ -3325,9 +3325,14 @@ static const char *btf_find_decl_tag_value(const struct btf *btf,continue ;if (strncmp (__btf_name_by_offset (btf, t->name_off), tag_key, len))continue ; -return __btf_name_by_offset (btf, t->name_off) + len; +/* Prevent duplicate entries for same type */ +if (value) +return ERR_PTR (-EEXIST ); + value =__btf_name_by_offset (btf, t->name_off) + len; } -return NULL ; +if (!value) +return ERR_PTR (-ENOENT ); +return value; } static int
@@ -3345,7 +3350,7 @@ btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,if (t->size != sz)return BTF_FIELD_IGNORE ; value_type =btf_find_decl_tag_value (btf, pt, comp_idx,"contains:" ); -if (!value_type) +if (IS_ERR (value_type))return -EINVAL ; node_field_name =strstr (value_type,":" );if (!node_field_name)
@@ -6949,7 +6954,7 @@ int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog,* (either PTR_TO_CTX or SCALAR_VALUE). */ int btf_prepare_func_args (struct bpf_verifier_env *env,int subprog, -struct bpf_reg_state *regs) +struct bpf_reg_state *regs,bool is_ex_cb) {struct bpf_verifier_log *log = &env->log;struct bpf_prog *prog = env->prog;
@@ -7006,7 +7011,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, tname, nargs, MAX_BPF_FUNC_REG_ARGS);return -EINVAL ; } -/* check that function returns int */ +/* check that function returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t)) t =btf_type_by_id (btf, t->type);
@@ -7055,6 +7060,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, i, btf_type_str(t), tname);return -EINVAL ; } +/* We have already ensured that the callback returns an integer, just +* like all global subprogs. We need to determine it only has a single +* scalar argument. +*/ +if (is_ex_cb && (nargs !=1 || regs[BPF_REG_1 ].type !=SCALAR_VALUE )) { +bpf_log (log,"exception cb only supports single integer argument\n" ); +return -EINVAL ; + }return 0 ; }
kernel/bpf/verifier.c +110 -8
192 lines not on this page · raw message
tools/testing/selftests/bpf/bpf_experimental.h +30 -1
@@ -134,7 +134,16 @@ extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;/* Description * Throw a BPF exception from the program, immediately terminating its * execution and unwinding the stack. The supplied 'cookie' parameter -* will be the return value of the program when an exception is thrown. +* will be the return value of the program when an exception is thrown, +* and the default exception callback is used. Otherwise, if an exception +* callback is set using the '__exception_cb(callback)' declaration tag +* on the main program, the 'cookie' parameter will be the callback's only +* input argument. +* +* Thus, in case of default exception callback, 'cookie' is subjected to +* constraints on the program's return value (as with R0 on exit). +* Otherwise, the return value of the marked exception callback will be +* subjected to the same checks. * * Note that throwing an exception with lingering resources (locks, * references, etc.) will lead to a verification error.
@@ -147,4 +156,24 @@ extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;*/ extern void bpf_throw(u64 cookie) __ksym; +/* This macro must be used to mark the exception callback corresponding to the +* main program. For example: +* +* int exception_cb(u64 cookie) { +* return cookie; +* } +* +* SEC("tc") +* __exception_cb(exception_cb) +* int main_prog(struct __sk_buff *ctx) { +* ... +* return TC_ACT_OK; +* } +* +* Here, exception callback for the main program will be 'exception_cb'. Note +* that this attribute can only be used once, and multiple exception callbacks +* specified for the main program will lead to verification error. +*/ +#define __exception_cb (name) __attribute__((btf_decl_tag("exception_callback:" #name))) +#endif
Signature · 1 line
2.41.0
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v2 05/14] bpf: Add support for custom exception callbacks
Quoted 36 lines · Kumar Kartikeya Dwivedi
kernel/bpf/verifier.c +28
-0@@ -2457,6 +2457,73 @@ static int add_subprog(struct bpf_verifier_env *env, int off)return env->subprog_cnt -1 ; } +static int bpf_find_exception_callback_insn_off (struct bpf_verifier_env *env) +{ +struct bpf_prog_aux *aux = env->prog->aux; +struct btf *btf = aux->btf; +const struct btf_type *t; +const char *name; +u32 main_btf_id; +int ret, i, j; + +/* Non-zero func_info_cnt implies valid btf */ +if (!aux->func_info_cnt) +return 0 ; + main_btf_id = aux->func_info[0 ].type_id; + + t =btf_type_by_id (btf, main_btf_id); +if (!t) { +verbose (env,"invalid btf id for main subprog in func_info\n" ); +return -EINVAL ; + } + + name =btf_find_decl_tag_value (btf, t,-1 ,"exception_callback:" ); +if (IS_ERR (name)) { + ret =PTR_ERR (name); +/* If there is no tag present, there is no exception callback */ +if (ret == -ENOENT ) + ret =0 ; +else if (ret == -EEXIST ) +verbose (env,"multiple exception callback tags for main subprog\n" );
kernel/bpf/verifier.c static int add_subprog(struct bpf_verifier_env *env, int off) · line 2497
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return ret; + } + + ret = -ENOENT ; +for (i =0 ; i <btf_nr_types (btf); i++) { + t =btf_type_by_id (btf, i); +if (!btf_type_is_func (t)) +continue ; +if (strcmp (name,btf_name_by_offset (btf, t->name_off))) +continue ;
nit. btf_find_by_name_kind() could be used here.
Quoted 27 lines
kernel/bpf/verifier.c +27
-0+if (btf_func_linkage (t) !=BTF_FUNC_GLOBAL ) { +verbose (env,"exception callback '%s' must have global linkage\n" , name); +return -EINVAL ; + } > + + ret =0 ; +for (j =0 ; j < aux->func_info_cnt; j++) { +if (aux->func_info[j].type_id != i) +continue ; + ret = aux->func_info[j].insn_off; +/* Further func_info and subprog checks will also happen +* later, so assume this is the right insn_off for now. +*/ +if (!ret) { +verbose (env,"invalid exception callback insn_off in func_info: 0\n" ); + ret = -EINVAL ; + } + } +if (!ret) { +verbose (env,"exception callback type id not found in func_info\n" ); + ret = -EINVAL ; + } +break ; + } +if (ret == -ENOENT ) +verbose (env,"exception callback '%s' could not be found in BTF\n" , name); +return ret; +}
Kumar Kartikeya Dwivedi
maintainerPATCH v2 6/14 bpf: Perform CFG walk for exception callback
In reply to Kumar Kartikeya DwivediSince exception callbacks are not referenced using bpf_pseudo_func and bpf_pseudo_call instructions, check_cfg traversal will never explore instructions of the exception callback. Even after adding the subprog, the program will then fail with a 'unreachable insn' error.
We thus need to begin walking from the start of the exception callback again in check_cfg after a complete CFG traversal finishes, so as to explore the CFG rooted at the exception callback.
--- kernel/bpf/verifier.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +13 -2
@@ -15029,8 +15029,8 @@ static int check_cfg(struct bpf_verifier_env *env) {int insn_cnt = env->prog->len;int *insn_stack, *insn_state; -int ret =0 ; -int i; +int ex_insn_beg, i, ret =0 ; +bool ex_done = false; insn_state = env->cfg.insn_state =kvcalloc (insn_cnt,sizeof (int ),GFP_KERNEL );if (!insn_state)
@@ -15046,6 +15046,7 @@ static int check_cfg(struct bpf_verifier_env *env) insn_stack[0 ] =0 ;/* 0 is the first instruction */ env->cfg.cur_stack =1 ; +walk_cfg:while (env->cfg.cur_stack >0 ) {int t = insn_stack[env->cfg.cur_stack -1 ];
@@ -15072,6 +15073,16 @@ static int check_cfg(struct bpf_verifier_env *env) goto err_free; } +if (env->exception_callback_subprog && !ex_done) { + ex_insn_beg = env->subprog_info[env->exception_callback_subprog].start; + + insn_state[ex_insn_beg] =DISCOVERED ; + insn_stack[0 ] = ex_insn_beg; + env->cfg.cur_stack =1 ; + ex_done = true; + goto walk_cfg; + } +for (i =0 ; i < insn_cnt; i++) {if (insn_state[i] !=EXPLORED ) {verbose (env,"unreachable insn %d\n" , i);
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 7/14 bpf: Treat first argument as return value for bpf_throw
In reply to Kumar Kartikeya DwivediIn case of the default exception callback, change the behavior of bpf_throw, where the passed cookie value is no longer ignored, but is instead the return value of the default exception callback. As such, we need to place restrictions on the value being passed into bpf_throw in such a case, only allowing those permitted by the check_return_code function.
Thus, bpf_throw can now control the return value of the program from each call site without having the user install a custom exception callback just to override the return value when an exception is thrown.
We also modify the hidden subprog instructions to now move BPF_REG_1 to BPF_REG_0, so as to set the return value before exit in the default callback.
--- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
kernel/bpf/verifier.c +24 -13
@@ -11415,6 +11415,8 @@ static int fetch_kfunc_meta(struct bpf_verifier_env *env,return 0 ; } +static int check_return_code (struct bpf_verifier_env *env,int regno); +static int check_kfunc_call (struct bpf_verifier_env *env,struct bpf_insn *insn,int *insn_idx_p) {
@@ -11538,6 +11540,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -ENOTSUPP ; } env->seen_exception = true; + +/* In the case of the default callback, the cookie value passed +* to bpf_throw becomes the return value of the program. +*/ +if (!env->exception_callback_subprog) { + err =check_return_code (env,BPF_REG_1 ); +if (err <0 ) +return err; + } }for (i =0 ; i <CALLER_SAVED_REGS ; i++)
@@ -14612,7 +14623,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)return 0 ; } -static int check_return_code (struct bpf_verifier_env *env) +static int check_return_code (struct bpf_verifier_env *env,int regno) {struct tnum enforce_attach_type_range = tnum_unknown;const struct bpf_prog *prog = env->prog;
@@ -14646,22 +14657,22 @@ static int check_return_code(struct bpf_verifier_env *env)* of bpf_exit, which means that program wrote * something into it earlier */ - err =check_reg_arg (env,BPF_REG_0 ,SRC_OP ); + err =check_reg_arg (env, regno,SRC_OP );if (err)return err; -if (is_pointer_value (env,BPF_REG_0 )) { -verbose (env,"R0 leaks addr as return value\n" ); +if (is_pointer_value (env, regno)) { +verbose (env,"R%d leaks addr as return value\n" , regno);return -EACCES ; } - reg =cur_regs (env) +BPF_REG_0 ; + reg =cur_regs (env) + regno;if (frame->in_async_callback_fn) {/* enforce return zero from async callbacks like timer */ if (reg->type !=SCALAR_VALUE ) { -verbose (env,"In async callback the register R0 is not a known value (%s)\n" , -reg_type_str (env, reg->type)); +verbose (env,"In async callback the register R%d is not a known value (%s)\n" , + regno,reg_type_str (env, reg->type));return -EINVAL ; }
@@ -14674,8 +14685,8 @@ static int check_return_code(struct bpf_verifier_env *env)if (is_subprog && !frame->in_exception_callback_fn) {if (reg->type !=SCALAR_VALUE ) { -verbose (env,"At subprogram exit the register R0 is not a scalar value (%s)\n" , -reg_type_str (env, reg->type)); +verbose (env,"At subprogram exit the register R%d is not a scalar value (%s)\n" , + regno,reg_type_str (env, reg->type));return -EINVAL ; }return 0 ;
@@ -14757,8 +14768,8 @@ static int check_return_code(struct bpf_verifier_env *env) }if (reg->type !=SCALAR_VALUE ) { -verbose (env,"At program exit the register R0 is not a known value (%s)\n" , -reg_type_str (env, reg->type)); +verbose (env,"At program exit the register R%d is not a known value (%s)\n" , + regno,reg_type_str (env, reg->type));return -EINVAL ; }
@@ -16955,7 +16966,7 @@ static int do_check(struct bpf_verifier_env *env)continue ; } - err =check_return_code (env); + err =check_return_code (env,BPF_REG_0 );if (err)return err; process_bpf_exit:
@@ -18601,7 +18612,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)if (env->seen_exception && !env->exception_callback_subprog) {struct bpf_insn patch[] = { env->prog->insnsi[insn_cnt -1 ], -BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ),BPF_EXIT_INSN (), };
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 8/14 bpf: Prevent KASAN false positive with bpf_throw
In reply to Kumar Kartikeya DwivediThe KASAN stack instrumentation when CONFIG_KASAN_STACK is true poisons the stack of a function when it is entered and unpoisons it when leaving. However, in the case of bpf_throw, we will never return as we switch our stack frame to the BPF exception callback. Later, this discrepancy will lead to confusing KASAN splats when kernel resumes execution on return from the BPF program.
Fix this by unpoisoning everything below the stack pointer of the BPF program, which should cover the range that would not be unpoisoned. An example splat is below:
BUG: KASAN: stack-out-of-bounds in stack_trace_consume_entry+0x14e/0x170 Write of size 8 at addr ffffc900013af958 by task test_progs/227 CPU: 0 PID: 227 Comm: test_progs Not tainted 6.5.0-rc2-g43f1c6c9052a-dirty #26 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-2.fc39 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4a/0x80 print_report+0xcf/0x670 ? arch_stack_walk+0x79/0x100 kasan_report+0xda/0x110 ? stack_trace_consume_entry+0x14e/0x170 ? stack_trace_consume_entry+0x14e/0x170 ? __pfx_stack_trace_consume_entry+0x10/0x10 stack_trace_consume_entry+0x14e/0x170 ? __sys_bpf+0xf2e/0x41b0 arch_stack_walk+0x8b/0x100 ? __sys_bpf+0xf2e/0x41b0 ? bpf_prog_test_run_skb+0x341/0x1c70 ? bpf_prog_test_run_skb+0x341/0x1c70 stack_trace_save+0x9b/0xd0 ? __pfx_stack_trace_save+0x10/0x10 ? __kasan_slab_free+0x109/0x180 ? bpf_prog_test_run_skb+0x341/0x1c70 ? __sys_bpf+0xf2e/0x41b0 ? __x64_sys_bpf+0x78/0xc0 ? do_syscall_64+0x3c/0x90 ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 kasan_save_stack+0x33/0x60 ? kasan_save_stack+0x33/0x60 ? kasan_set_track+0x25/0x30 ? kasan_save_free_info+0x2b/0x50 ? __kasan_slab_free+0x109/0x180 ? kmem_cache_free+0x191/0x460 ? bpf_prog_test_run_skb+0x341/0x1c70 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x109/0x180 kmem_cache_free+0x191/0x460 bpf_prog_test_run_skb+0x341/0x1c70 ? __pfx_bpf_prog_test_run_skb+0x10/0x10 ? __fget_light+0x51/0x220 __sys_bpf+0xf2e/0x41b0 ? __might_fault+0xa2/0x170 ? __pfx___sys_bpf+0x10/0x10 ? lock_release+0x1de/0x620 ? __might_fault+0xcd/0x170 ? __pfx_lock_release+0x10/0x10 ? __pfx_blkcg_maybe_throttle_current+0x10/0x10 __x64_sys_bpf+0x78/0xc0 ? syscall_enter_from_user_mode+0x20/0x50 do_syscall_64+0x3c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f0fbb38880d Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d
89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f3 45 12 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe13907de8 EFLAGS: 00000206 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007ffe13908708 RCX: 00007f0fbb38880d
RDX: 0000000000000050 RSI: 00007ffe13907e20 RDI: 000000000000000a
RBP: 00007ffe13907e00 R08: 0000000000000000 R09: 00007ffe13907e20
R10: 0000000000000064 R11: 0000000000000206 R12: 0000000000000003
R13: 0000000000000000 R14: 00007f0fbb532000 R15: 0000000000cfbd90
</TASK> The buggy address belongs to stack of task test_progs/227
KASAN internal error: frame info validation failed; invalid marker: 0
The buggy address belongs to the virtual mapping at
[ffffc900013a8000, ffffc900013b1000) created by:
kernel_clone+0xcd/0x600
The buggy address belongs to the physical page:
page:00000000b70f4332 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x11418f
flags: 0x2fffe0000000000(node=0|zone=2|lastcpupid=0x7fff)
page_type: 0xffffffff()
raw: 02fffe0000000000 0000000000000000 dead000000000122 0000000000000000
raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffffc900013af800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffffc900013af880: 00 00 00 f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3 00
>ffffc900013af900: 00 00 00 00 00 00 00 00 00 00 00 f1 00 00 00 00
^
ffffc900013af980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffffc900013afa00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================Disabling lock debugging due to kernel taint
--- include/linux/kasan.h | 2 ++ kernel/bpf/helpers.c | 6 ++++++ 2 files changed, 8 insertions(+)
include/linux/kasan.h +2 -0
@@ -283,8 +283,10 @@ static inline bool kasan_check_byte(const void *address)#if defined(CONFIG_KASAN ) && defined(CONFIG_KASAN_STACK )void kasan_unpoison_task_stack (struct task_struct *task); +asmlinkage void kasan_unpoison_task_stack_below(const void *watermark);#else static inline void kasan_unpoison_task_stack (struct task_struct *task) {} +static inline void kasan_unpoison_task_stack_below (const void *watermark) {}#endif #ifdef CONFIG_KASAN_GENERIC
kernel/bpf/helpers.c +6 -0
@@ -22,6 +22,7 @@#include <linux/security.h> #include <linux/btf_ids.h> #include <linux/bpf_mem_alloc.h> +#include <linux/kasan.h> #include "../../lib/kstrtox.h"
@@ -2463,6 +2464,11 @@ __bpf_kfunc void bpf_throw(u64 cookie)WARN_ON_ONCE (!ctx.aux->exception_boundary);WARN_ON_ONCE (!ctx.bp);WARN_ON_ONCE (!ctx.cnt); +/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning +* deeper stack depths than ctx.sp as we do not return from bpf_throw, +* which skips compiler generated instrumentation to do the same. +*/ +kasan_unpoison_task_stack_below ((void *)ctx.sp); ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); }
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 08/14] bpf: Prevent KASAN false positive with bpf_throw
Andrey, Dmitry,
Please help review this patch.
Quoted 155 lines · Kumar Kartikeya Dwivedi
The KASAN stack instrumentation when
CONFIG_KASAN_STACKis true poisons the stack of a function when it is entered and unpoisons it when leaving. However, in the case ofbpf_throw, we will never return as we switch our stack frame to the BPF exception callback. Later, this discrepancy will lead to confusing KASAN splats when kernel resumes execution on return from the BPF program.Fix this by unpoisoning everything below the stack pointer of the BPF program, which should cover the range that would not be unpoisoned. An example splat is below:
BUG: KASAN: stack-out-of-bounds in stack_trace_consume_entry+0x14e/0x170 Write of size 8 at addr ffffc900013af958 by task test_progs/227 CPU: 0 PID: 227 Comm: test_progs Not tainted 6.5.0-rc2-g43f1c6c9052a-dirty #26 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-2.fc39 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4a/0x80 print_report+0xcf/0x670 ? arch_stack_walk+0x79/0x100 kasan_report+0xda/0x110 ? stack_trace_consume_entry+0x14e/0x170 ? stack_trace_consume_entry+0x14e/0x170 ? __pfx_stack_trace_consume_entry+0x10/0x10 stack_trace_consume_entry+0x14e/0x170 ? __sys_bpf+0xf2e/0x41b0 arch_stack_walk+0x8b/0x100 ? __sys_bpf+0xf2e/0x41b0 ? bpf_prog_test_run_skb+0x341/0x1c70 ? bpf_prog_test_run_skb+0x341/0x1c70 stack_trace_save+0x9b/0xd0 ? __pfx_stack_trace_save+0x10/0x10 ? __kasan_slab_free+0x109/0x180 ? bpf_prog_test_run_skb+0x341/0x1c70 ? __sys_bpf+0xf2e/0x41b0 ? __x64_sys_bpf+0x78/0xc0 ? do_syscall_64+0x3c/0x90 ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 kasan_save_stack+0x33/0x60 ? kasan_save_stack+0x33/0x60 ? kasan_set_track+0x25/0x30 ? kasan_save_free_info+0x2b/0x50 ? __kasan_slab_free+0x109/0x180 ? kmem_cache_free+0x191/0x460 ? bpf_prog_test_run_skb+0x341/0x1c70 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x109/0x180 kmem_cache_free+0x191/0x460 bpf_prog_test_run_skb+0x341/0x1c70 ? __pfx_bpf_prog_test_run_skb+0x10/0x10 ? __fget_light+0x51/0x220 __sys_bpf+0xf2e/0x41b0 ? __might_fault+0xa2/0x170 ? __pfx___sys_bpf+0x10/0x10 ? lock_release+0x1de/0x620 ? __might_fault+0xcd/0x170 ? __pfx_lock_release+0x10/0x10 ? __pfx_blkcg_maybe_throttle_current+0x10/0x10 __x64_sys_bpf+0x78/0xc0 ? syscall_enter_from_user_mode+0x20/0x50 do_syscall_64+0x3c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f0fbb38880d Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f3 45 12 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe13907de8 EFLAGS: 00000206ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX:00007ffe13908708RCX:00007f0fbb38880d
RDX: 0000000000000050 RSI:00007ffe13907e20RDI:000000000000000a
RBP:00007ffe13907e00R08: 0000000000000000 R09:00007ffe13907e20
R10: 0000000000000064 R11: 0000000000000206 R12: 0000000000000003
R13: 0000000000000000 R14:00007f0fbb532000R15:0000000000cfbd90</TASK> The buggy address belongs to stack of task test_progs/227KASAN internal error: frame info validation failed; invalid marker: 0
The buggy address belongs to the virtual mapping at [ffffc900013a8000, ffffc900013b1000) created by: kernel_clone+0xcd/0x600 The buggy address belongs to the physical page: page:00000000b70f4332 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x11418f flags: 0x2fffe0000000000(node=0|zone=2|lastcpupid=0x7fff) page_type: 0xffffffff() raw: 02fffe0000000000 0000000000000000 dead000000000122 0000000000000000 raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffffc900013af800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013af880: 00 00 00 f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3 00 >ffffc900013af900: 00 00 00 00 00 00 00 00 00 00 00 f1 00 00 00 00 ^ ffffc900013af980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013afa00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ==================================================================Disabling lock debugging due to kernel taint
Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Vincenzo Frascino <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/kasan.h | 2 ++ kernel/bpf/helpers.c | 6 ++++++ 2 files changed, 8 insertions(+)include/linux/kasan.h +2
-0@@ -283,8 +283,10 @@ static inline bool kasan_check_byte(const void *address)#if defined(CONFIG_KASAN ) && defined(CONFIG_KASAN_STACK )void kasan_unpoison_task_stack (struct task_struct *task); +asmlinkage void kasan_unpoison_task_stack_below(const void *watermark);#else static inline void kasan_unpoison_task_stack (struct task_struct *task) {} +static inline void kasan_unpoison_task_stack_below (const void *watermark) {}#endif #ifdef CONFIG_KASAN_GENERIC kernel/bpf/helpers.c +6
-1@@ -22,6 +22,7 @@#include <linux/security.h> #include <linux/btf_ids.h> #include <linux/bpf_mem_alloc.h> +#include <linux/kasan.h> #include "../../lib/kstrtox.h" @@ -2463,6 +2464,11 @@ __bpf_kfunc void bpf_throw(u64 cookie)WARN_ON_ONCE (!ctx.aux->exception_boundary);WARN_ON_ONCE (!ctx.bp);WARN_ON_ONCE (!ctx.cnt); +/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning +* deeper stack depths than ctx.sp as we do not return from bpf_throw, +* which skips compiler generated instrumentation to do the same. +*/ +kasan_unpoison_task_stack_below ((void *)ctx.sp); ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); } --2.41.0
Andrey Konovalov
reviewerRe: [PATCH bpf-next v2 08/14] bpf: Prevent KASAN false positive with bpf_throw
In reply to Kumar Kartikeya DwivediQuoted 1 line · Kumar Kartikeya Dwivedi
Hi Kumar,
include/linux/kasan.h static inline bool kasan_check_byte(const void *address) · line 290
Quoted 9 lines
include/linux/kasan.h +2
-0@@ -283,8 +283,10 @@ static inline bool kasan_check_byte(const void *address)#if defined(CONFIG_KASAN ) && defined(CONFIG_KASAN_STACK )void kasan_unpoison_task_stack (struct task_struct *task); +asmlinkage void kasan_unpoison_task_stack_below(const void *watermark);#else static inline void kasan_unpoison_task_stack (struct task_struct *task) {} +static inline void kasan_unpoison_task_stack_below (const void *watermark) {}#endif
Please also drop the kasan_unpoison_task_stack_below declaration from mm/kasan/kasan.h.
Also, could you please split this change into 2: one that exposes kasan_unpoison_task_stack_below and another that changes the bpf code. This will greatly simplify backporting KASAN changes for older kernels.
Thank you!
Kumar Kartikeya Dwivedi
maintainerPATCH v2 9/14 bpf: Detect IP == ksym.end as part of BPF program
In reply to Kumar Kartikeya DwivediNow that bpf_throw kfunc is the first such call instruction that has noreturn semantics within the verifier, this also kicks in dead code elimination in unprecedented ways. For one, any instruction following a bpf_throw call will never be marked as seen. Moreover, if a callchain ends up throwing, any instructions after the call instruction to the eventually throwing subprog in callers will also never be marked as seen.
The tempting way to fix this would be to emit extra 'int3' instructions which bump the jited_len of a program, and ensure that during runtime when a program throws, we can discover its boundaries even if the call instruction to bpf_throw (or to subprogs that always throw) is emitted as the final instruction in the program.
An example of such a program would be this:
do_something():
... r0 = 0 exit
foo():
r1 = 0 call bpf_throw r0 = 0 exit
bar(cond):
if r1 != 0 goto pc+2 call do_something exit call foo r0 = 0 // Never seen by verifier exit //
main(ctx):
r1 = ... call bar r0 = 0 exit
Here, if we do end up throwing, the stacktrace would be the following:
bpf_throw
foo
bar
main
In bar, the final instruction emitted will be the call to foo, as such, the return address will be the subsequent instruction (which the JIT emits as int3 on x86). This will end up lying outside the jited_len of the program, thus, when unwinding, we will fail to discover the return address as belonging to any program and end up in a panic due to the unreliable stack unwinding of BPF programs that we never expect.
To remedy this case, make bpf_prog_ksym_find treat IP == ksym.end as part of the BPF program, so that is_bpf_text_address returns true when such a case occurs, and we are able to unwind reliably when the final instruction ends up being a call instruction.
--- kernel/bpf/core.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/bpf/core.c +5 -1
@@ -623,7 +623,11 @@ static __always_inline int bpf_tree_comp(void *key, struct latch_tree_node *n)if (val < ksym->start)return -1 ; -if (val >= ksym->end) +/* Ensure that we detect return addresses as part of the program, when +* the final instruction is a call for a program part of the stack +* trace. Therefore, do val > ksym->end instead of val >= ksym->end. +*/ +if (val > ksym->end)return 1 ;return 0 ;
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 10/14 bpf: Disallow extensions to exception callbacks
In reply to Kumar Kartikeya DwivediDuring testing, it was discovered that extensions to exception callbacks had no checks, upon running a testcase, the kernel ended up running off the end of a program having final call as bpf_throw, and hitting int3 instructions.
The reason is that while the default exception callback would have reset the stack frame to return back to the main program's caller, the replacing extension program will simply return back to bpf_throw, which will instead return back to the program and the program will continue execution, now in an undefined state where anything could happen.
The way to support extensions to an exception callback would be to mark the BPF_PROG_TYPE_EXT main subprog as an exception_cb, and prevent it from calling bpf_throw. This would make the JIT produce a prologue that restores saved registers and reset the stack frame. But let's not do that until there is a concrete use case for this, and simply disallow this for now.
One key point here to note is that currently X86_TAIL_CALL_OFFSET didn't require any modifications, even though we emit instructions before the corresponding endbr64 instruction. This is because we ensure that a main subprog never serves as an exception callback, and therefore the exception callback (which will be a global subprog) can never serve as the tail call target, eliminating any discrepancies. However, once we support a BPF_PROG_TYPE_EXT to also act as an exception callback, it will end up requiring change to the tail call offset to account for the extra instructions. For simplicitly, tail calls could be disabled for such targets.
Noting the above, it appears better to wait for a concrete use case before choosing to permit extension programs to replace exception callbacks.
--- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 5 +++++ 2 files changed, 6 insertions(+)
kernel/bpf/helpers.c +1 -0
@@ -2470,6 +2470,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)*/ kasan_unpoison_task_stack_below ((void *)ctx.sp); ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); +WARN (1 ,"A call to BPF exception callback should never return\n" ); }__diag_pop ();
kernel/bpf/verifier.c +5 -0
@@ -19622,6 +19622,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, "Extension programs should be JITed\n");return -EINVAL ; } +if (aux->func && aux->func[subprog]->aux->exception_cb) { +bpf_log (log, +"Extension programs cannot replace exception callback\n" ); +return -EINVAL ; + } }if (!tgt_prog->jited) {bpf_log (log,"Can attach to only JITed progs\n" );
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 10/14] bpf: Disallow extensions to exception callbacks
Quoted 56 lines · Kumar Kartikeya Dwivedi
During testing, it was discovered that extensions to exception callbacks had no checks, upon running a testcase, the kernel ended up running off the end of a program having final call as
bpf_throw, and hitting int3 instructions.The reason is that while the default exception callback would have reset the stack frame to return back to the main program's caller, the replacing extension program will simply return back to
bpf_throw, which will instead return back to the program and the program will continue execution, now in an undefined state where anything could happen.The way to support extensions to an exception callback would be to mark the
BPF_PROG_TYPE_EXTmain subprog as anexception_cb, and prevent it from callingbpf_throw. This would make the JIT produce a prologue that restores saved registers and reset the stack frame. But let's not do that until there is a concrete use case for this, and simply disallow this for now.One key point here to note is that currently
X86_TAIL_CALL_OFFSETdidn't require any modifications, even though we emit instructions before the corresponding endbr64 instruction. This is because we ensure that a main subprog never serves as an exception callback, and therefore the exception callback (which will be a global subprog) can never serve as the tail call target, eliminating any discrepancies. However, once we support aBPF_PROG_TYPE_EXTto also act as an exception callback, it will end up requiring change to the tail call offset to account for the extra instructions. For simplicitly, tail calls could be disabled for such targets.Noting the above, it appears better to wait for a concrete use case before choosing to permit extension programs to replace exception callbacks.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 5 +++++ 2 files changed, 6 insertions(+)kernel/bpf/helpers.c +1
-0@@ -2470,6 +2470,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)*/ kasan_unpoison_task_stack_below ((void *)ctx.sp); ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); +WARN (1 ,"A call to BPF exception callback should never return\n" ); }__diag_pop ();kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c int bpf_check_attach_target(struct bpf_verifier_log *log, · line 19628
Quoted 7 lines
kernel/bpf/verifier.c +4
-0@@ -19622,6 +19622,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, "Extension programs should be JITed\n");return -EINVAL ; } +if (aux->func && aux->func[subprog]->aux->exception_cb) { +bpf_log (log, +"Extension programs cannot replace exception callback\n" ); +return -EINVAL ;
Should we disallow fentry/fexit to exception cb as well? Probably things will go wrong for similar reasons as freplace.
And also disallow fentry/fexit for main prog that is exception_boundary ? since bpf trampoline doesn't know that it needs to save r12.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 10/14] bpf: Disallow extensions to exception callbacks
Quoted 52 lines · Alexei Starovoitov
Quoted 50 lines · Kumar Kartikeya Dwivedi
During testing, it was discovered that extensions to exception callbacks had no checks, upon running a testcase, the kernel ended up running off the end of a program having final call as
bpf_throw, and hitting int3 instructions.The reason is that while the default exception callback would have reset the stack frame to return back to the main program's caller, the replacing extension program will simply return back to
bpf_throw, which will instead return back to the program and the program will continue execution, now in an undefined state where anything could happen.The way to support extensions to an exception callback would be to mark the
BPF_PROG_TYPE_EXTmain subprog as anexception_cb, and prevent it from callingbpf_throw. This would make the JIT produce a prologue that restores saved registers and reset the stack frame. But let's not do that until there is a concrete use case for this, and simply disallow this for now.One key point here to note is that currently
X86_TAIL_CALL_OFFSETdidn't require any modifications, even though we emit instructions before the corresponding endbr64 instruction. This is because we ensure that a main subprog never serves as an exception callback, and therefore the exception callback (which will be a global subprog) can never serve as the tail call target, eliminating any discrepancies. However, once we support aBPF_PROG_TYPE_EXTto also act as an exception callback, it will end up requiring change to the tail call offset to account for the extra instructions. For simplicitly, tail calls could be disabled for such targets.Noting the above, it appears better to wait for a concrete use case before choosing to permit extension programs to replace exception callbacks.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 5 +++++ 2 files changed, 6 insertions(+)kernel/bpf/helpers.c +1
-0@@ -2470,6 +2470,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)*/ kasan_unpoison_task_stack_below ((void *)ctx.sp); ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); +WARN (1 ,"A call to BPF exception callback should never return\n" ); }
Quoted 17 lines
kernel/bpf/verifier.c int bpf_check_attach_target(struct bpf_verifier_log *log, · line 19628
Quoted 13 lines
kernel/bpf/helpers.c +0
-0__diag_pop ();kernel/bpf/verifier.c +4
-0@@ -19622,6 +19622,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, "Extension programs should be JITed\n");return -EINVAL ; } +if (aux->func && aux->func[subprog]->aux->exception_cb) { +bpf_log (log, +"Extension programs cannot replace exception callback\n" ); +return -EINVAL ;Should we disallow fentry/fexit to exception cb as well? Probably things will go wrong for similar reasons as freplace.
Yes, great catch. I think you are right. I will disable both of them as well. Trampoline does not expect the stack frame to be reset as it pushes data to it and will need to restore it after the call to exception cb.
Quoted 2 lines
And also disallow fentry/fexit for main prog that is
exception_boundary? since bpf trampoline doesn't know that it needs to save r12.
Hmm, I think I should probably enable that instead of blocking it. I think it's a common enough use case. Compared to exception cb it also seems a valid one. We can enable pushing of r12 for such generate trampolines, IIUC it's not a lot of complexity and we will know if we are attaching to exception boundary prog.
In any case, I will add more selftests to ensure these cases are handled/rejected properly in v3, thanks!
Kumar Kartikeya Dwivedi
maintainerPATCH v2 11/14 bpf: Fix kfunc callback register type handling
In reply to Kumar Kartikeya DwivediThe kfunc code to handle KF_ARG_PTR_TO_CALLBACK does not check the reg type before using reg->subprogno. This can accidently permit invalid pointers from being passed into callback helpers (e.g. silently from different paths). Likewise, reg->subprogno from the per-register type union may not be meaningful either. We need to reject any other type except PTR_TO_FUNC.
--- kernel/bpf/verifier.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/bpf/verifier.c +4 -0
@@ -11334,6 +11334,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ; }case KF_ARG_PTR_TO_CALLBACK : +if (reg->type !=PTR_TO_FUNC ) { +verbose (env,"arg%d expected pointer to func\n" , i); +return -EINVAL ; + } meta->subprogno = reg->subprogno;break ;case KF_ARG_PTR_TO_REFCOUNTED_KPTR :
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 11/14] bpf: Fix kfunc callback register type handling
Quoted 11 lines · Kumar Kartikeya Dwivedi
The kfunc code to handle
KF_ARG_PTR_TO_CALLBACKdoes not check the reg type before usingreg->subprogno. This can accidently permit invalid pointers from being passed into callback helpers (e.g. silently from different paths). Likewise,reg->subprognofrom the per-register type union may not be meaningful either. We need to reject any other type exceptPTR_TO_FUNC.Cc Dave Marchevsky <[email protected]>Fixes 5d92ddc3de1b ("bpf: Add callback validation to kfunc verifier logic")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Kumar Kartikeya Dwivedi
maintainerPATCH v2 12/14 libbpf: Add support for custom exception callbacks
In reply to Kumar Kartikeya DwivediAdd support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
The process is done in two steps. First, for each main program, the bpf_object__sanitize_and_load_btf function finds and marks its corresponding exception callback as defined by the declaration tag on it. Second, bpf_object__reloc_code is modified to append the indicated exception callback at the end of the instruction iteration (since exception callback will never be appended in that loop, as it is not directly referenced).
--- tools/lib/bpf/libbpf.c | 166 +++++++++++++++++++++++++++++++++++------ 1 file changed, 142 insertions(+), 24 deletions(-)
tools/lib/bpf/libbpf.c +142 -24
@@ -432,9 +432,11 @@ struct bpf_program {int fd;bool autoload;bool autoattach; +bool sym_global;bool mark_btf_static;enum bpf_prog_type type;enum bpf_attach_type expected_attach_type; +int exception_cb_idx;int prog_ifindex;__u32 attach_btf_obj_fd;
@@ -760,6 +762,7 @@ bpf_object__init_prog(struct bpf_object *obj, struct bpf_program *prog, prog->type =BPF_PROG_TYPE_UNSPEC ; prog->fd =-1 ; + prog->exception_cb_idx =-1 ;/* libbpf's convention for SEC("?abc...") is that it's just like * SEC("abc...") but the corresponding bpf_program starts out with
@@ -866,20 +869,28 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data,if (err)return err; +if (ELF64_ST_BIND (sym->st_info) !=STB_LOCAL ) + prog->sym_global = true; +/* if function is a global/weak symbol, but has restricted * (STV_HIDDEN or STV_INTERNAL) visibility, mark its BTF FUNC * as static to enable more permissive BPF verification mode * with more outside context available to BPF verifier */ -if (ELF64_ST_BIND (sym->st_info) !=STB_LOCAL - && (ELF64_ST_VISIBILITY (sym->st_other) ==STV_HIDDEN - ||ELF64_ST_VISIBILITY (sym->st_other) ==STV_INTERNAL )) +if (prog->sym_global && (ELF64_ST_VISIBILITY (sym->st_other) ==STV_HIDDEN + ||ELF64_ST_VISIBILITY (sym->st_other) ==STV_INTERNAL )) prog->mark_btf_static = true; nr_progs++; obj->nr_programs = nr_progs; } +/* After adding all programs, now pair them with their exception +* callbacks if specified. +*/ +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto out; +out:return 0 ; }
@@ -3137,6 +3148,80 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } } +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto skip_exception_cb; +for (i =0 ; i < obj->nr_programs; i++) { +struct bpf_program *prog = &obj->programs[i]; +int j, k, n; + +if (prog_is_subprog (obj, prog)) +continue ; + n =btf__type_cnt (obj->btf); +for (j =1 ; j < n; j++) { +const char *str ="exception_callback:" , *name; +size_t len =strlen (str); +struct btf_type *t; + + t =btf_type_by_id (obj->btf, j); +if (!btf_is_decl_tag (t) ||btf_decl_tag (t)->component_idx !=-1 ) +continue ; + + name =btf__str_by_offset (obj->btf, t->name_off); +if (strncmp (name, str, len)) +continue ; + + t =btf_type_by_id (obj->btf, t->type); +if (!btf_is_func (t) ||btf_func_linkage (t) !=BTF_FUNC_GLOBAL ) { +pr_warn ("prog '%s': exception_callback:<value> decl tag not applied to the main program\n" , + prog->name); +return -EINVAL ; + } +if (strcmp (prog->name,btf__str_by_offset (obj->btf, t->name_off))) +continue ; +/* Multiple callbacks are specified for the same prog, +* the verifier will eventually return an error for this +* case, hence simply skip appending a subprog. +*/ +if (prog->exception_cb_idx >=0 ) { + prog->exception_cb_idx =-1 ; +break ; + } + + name += len; +if (str_is_empty (name)) { +pr_warn ("prog '%s': exception_callback:<value> decl tag contains empty value\n" , + prog->name); +return -EINVAL ; + } + +for (k =0 ; k < obj->nr_programs; k++) { +struct bpf_program *subprog = &obj->programs[k]; + +if (!prog_is_subprog (obj, subprog)) +continue ; +if (strcmp (name, subprog->name)) +continue ; +/* Enforce non-hidden, as from verifier point of +* view it expects global functions, whereas the +* mark_btf_static fixes up linkage as static. +*/ +if (!subprog->sym_global || subprog->mark_btf_static) { +pr_warn ("prog '%s': exception callback %s must be a global non-hidden function\n" , + prog->name, subprog->name); +return -EINVAL ; + } + prog->exception_cb_idx = k; +break ; + } + +if (prog->exception_cb_idx >=0 ) +continue ; +pr_warn ("prog '%s': cannot find exception callback '%s'\n" , prog->name, name); +return -ENOENT ; + } + } +skip_exception_cb: + sanitize =btf_needs_sanitization (obj);if (sanitize) {const void *raw_data;
@@ -6184,14 +6269,46 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_prograreturn 0 ; } +static int +bpf_object__append_subprog_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *subprog) +{ +struct bpf_insn *insns; +size_t new_cnt; +int err; + + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); +if (!insns) { +pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); +return -ENOMEM ; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + +memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt *sizeof (*insns)); + +pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , + main_prog->name, subprog->insns_cnt, subprog->name); + +/* The subprog insns are now appended. Append its relos too. */ + err =append_subprog_relos (main_prog, subprog); +if (err) +return err; +return 0 ; +} +static int bpf_object__reloc_code (struct bpf_object *obj,struct bpf_program *main_prog,struct bpf_program *prog) { -size_t sub_insn_idx, insn_idx, new_cnt; +size_t sub_insn_idx, insn_idx;struct bpf_program *subprog; -struct bpf_insn *insns, *insn;struct reloc_desc *relo; +struct bpf_insn *insn;int err; err =reloc_prog_func_and_line_info (obj, main_prog, prog);
@@ -6266,25 +6383,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,* and relocate. */ if (subprog->sub_insn_off ==0 ) { - subprog->sub_insn_off = main_prog->insns_cnt; - - new_cnt = main_prog->insns_cnt + subprog->insns_cnt; - insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); -if (!insns) { -pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); -return -ENOMEM ; - } - main_prog->insns = insns; - main_prog->insns_cnt = new_cnt; - -memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, - subprog->insns_cnt *sizeof (*insns)); - -pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , - main_prog->name, subprog->insns_cnt, subprog->name); - -/* The subprog insns are now appended. Append its relos too. */ - err =append_subprog_relos (main_prog, subprog); + err =bpf_object__append_subprog_code (obj, main_prog, subprog);if (err)return err; err =bpf_object__reloc_code (obj, main_prog, subprog);
@@ -6308,6 +6407,25 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, prog->name, insn_idx, insn->imm, subprog->name, subprog->sub_insn_off); } +/* Now, also append exception callback if it has not been done already. */ +if (main_prog == prog && main_prog->exception_cb_idx >=0 ) { + subprog = &obj->programs[main_prog->exception_cb_idx]; + +/* Calling exception callback directly is disallowed, which the +* verifier will reject later. In case it was processed already, +* we can skip this step, otherwise for all other valid cases we +* have to append exception callback now. +*/ +if (subprog->sub_insn_off ==0 ) { + err =bpf_object__append_subprog_code (obj, main_prog, subprog); +if (err) +return err; + err =bpf_object__reloc_code (obj, main_prog, subprog); +if (err) +return err; + } + } +return 0 ; }
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
Quoted 5 lines · Kumar Kartikeya Dwivedi
Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
...
tools/lib/bpf/libbpf.c bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, · line 893
Quoted 6 lines
tools/lib/bpf/libbpf.c +6
-0+/* After adding all programs, now pair them with their exception +* callbacks if specified. +*/ +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto out; +out:
The above looks odd. Accidental leftover?
Quoted 8 lines
return 0;}
tools/lib/bpf/libbpf.c +1
-0@@ -3137,6 +3148,80 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } } +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG ))
tools/lib/bpf/libbpf.c static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · line 3161
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ goto skip_exception_cb; +for (i =0 ; i < obj->nr_programs; i++) { +struct bpf_program *prog = &obj->programs[i]; +int j, k, n; + +if (prog_is_subprog (obj, prog)) +continue ; + n =btf__type_cnt (obj->btf); +for (j =1 ; j < n; j++) { +const char *str ="exception_callback:" , *name;
On the first read of this patch and corresponding kernel support I started doubting my earlier suggestion to use decl_tag and reconsidered going back to fake bpf_register_except_cb() call, but after sleeping on it I think it is a useful extension for both kernel and libbpf to support such tagging.
We might specify ctors and dtors with decl_tag in the future and other various callbacks that are never explicitly referenced in bpf_call, ld_imm64 or other bpf insns.
So having libbpf and kernel support such tagging will help in the long run. It's not going to be limited to exceptions.
Despite the extra complexity this is a good step forward.
tools/lib/bpf/libbpf.c static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra · line 6275
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+static int +bpf_object__append_subprog_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *subprog) +{
Please split this refactoring into a separate patch for ease of review.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
Quoted 10 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
...
Quoted 9 lines
tools/lib/bpf/libbpf.c bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, · line 893
Quoted 6 lines
tools/lib/bpf/libbpf.c +6
-0+/* After adding all programs, now pair them with their exception +* callbacks if specified. +*/ +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto out; +out:The above looks odd. Accidental leftover?
Oops, yes. I have dropped it now.
Quoted 7 lines
Quoted 7 lines
return 0;}
(quoted diff) +0
-0@@ -3137,6 +3148,80 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } }
Quoted 15 lines
tools/lib/bpf/libbpf.c static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · line 3161
Quoted 11 lines
tools/lib/bpf/libbpf.c +11
-0+if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto skip_exception_cb; +for (i =0 ; i < obj->nr_programs; i++) { +struct bpf_program *prog = &obj->programs[i]; +int j, k, n; + +if (prog_is_subprog (obj, prog)) +continue ; + n =btf__type_cnt (obj->btf); +for (j =1 ; j < n; j++) { +const char *str ="exception_callback:" , *name;On the first read of this patch and corresponding kernel support I started doubting my earlier suggestion to use
decl_tagand reconsidered going back to fakebpf_register_except_cb()call,
This is exactly what I thought when I realised it's not that simple when implementing it :).
Quoted 9 lines
but after sleeping on it I think it is a useful extension for both kernel and libbpf to support such tagging.
We might specify ctors and dtors withdecl_tagin the future and other various callbacks that are never explicitly referenced inbpf_call,ld_imm64or other bpf insns.
So having libbpf and kernel support such tagging will help in the long run. It's not going to be limited to exceptions.
Despite the extra complexity this is a good step forward.
I agree. This same code can also be reused to establish an edge from one BTF type to some other BTF type (by name).
function -> exception_cb. struct -> ctor, struct -> dtor etc.
I did have some questions though.
First of all this is explicitly an unstable feature. How do we feel about putting related support for it in libbpf and making breaking changes later?
Will the expectation be to pair the libbpf with its corresponding kernel release to use such features? Or do we have to make the changes in a backwards compatible fashion?
Secondly, due to proliferation of BTF tag usage, do you think it's time we reserve a namespace for all tags that would be recognized by the verifier? E.g. require all of them to be prefixed with "bpf." similar to "llvm." etc.? Since they are simply attributes for a specific BTF type (or component of a type).
It may be too late for some BTF tags, but we could do better going forward.
It may also allow us to indicate that a tag is experimental until its effect on the program becomes stabilized. E.g. bpf.experimental.exception_callback instead of exception_callback? Or do you feel it's unnecessary?
Quoted 6 lines
tools/lib/bpf/libbpf.c static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra · line 6275
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+static int +bpf_object__append_subprog_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *subprog) +{Please split this refactoring into a separate patch for ease of review.
Ack, will do.
Re: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
Quoted 65 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Alexei Starovoitov
Quoted 5 lines · Kumar Kartikeya Dwivedi
Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
...
Quoted 9 lines
tools/lib/bpf/libbpf.c bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, · line 893
Quoted 6 lines
tools/lib/bpf/libbpf.c +6
-0+/* After adding all programs, now pair them with their exception +* callbacks if specified. +*/ +if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto out; +out:The above looks odd. Accidental leftover?
Oops, yes. I have dropped it now.
Quoted 7 lines
Quoted 7 lines
return 0;}
(quoted diff) +0
-0@@ -3137,6 +3148,80 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } }Quoted 15 lines
tools/lib/bpf/libbpf.c static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) · line 3161
Quoted 11 lines
tools/lib/bpf/libbpf.c +11
-0+if (!kernel_supports (obj,FEAT_BTF_DECL_TAG )) + goto skip_exception_cb; +for (i =0 ; i < obj->nr_programs; i++) { +struct bpf_program *prog = &obj->programs[i]; +int j, k, n; + +if (prog_is_subprog (obj, prog)) +continue ; + n =btf__type_cnt (obj->btf); +for (j =1 ; j < n; j++) { +const char *str ="exception_callback:" , *name;On the first read of this patch and corresponding kernel support I started doubting my earlier suggestion to use
decl_tagand reconsidered going back to fakebpf_register_except_cb()call,This is exactly what I thought when I realised it's not that simple when implementing it :).
Quoted 9 lines
but after sleeping on it I think it is a useful extension for both kernel and libbpf to support such tagging.
We might specify ctors and dtors withdecl_tagin the future and other various callbacks that are never explicitly referenced inbpf_call,ld_imm64or other bpf insns.
So having libbpf and kernel support such tagging will help in the long run. It's not going to be limited to exceptions.
Despite the extra complexity this is a good step forward.I agree. This same code can also be reused to establish an edge from one BTF type to some other BTF type (by name).
function ->exception_cb. struct -> ctor, struct -> dtor etc.
Quoted 7 lines
I did have some questions though.
First of all this is explicitly an unstable feature. How do we feel about putting related support for it in libbpf and making breaking changes later?
Will the expectation be to pair the libbpf with its corresponding kernel release to use such features? Or do we have to make the changes in a backwards compatible fashion?
We should always do backwards compatible changes in both kernel and libbpf, but sooner or later we might hit an issue where we would have to break things. At that time the special prefix won't save us, so...
Quoted 8 lines
Secondly, due to proliferation of BTF tag usage, do you think it's time we reserve a namespace for all tags that would be recognized by the verifier? E.g. require all of them to be prefixed with "bpf." similar to "llvm." etc.? Since they are simply attributes for a specific BTF type (or component of a type).
It may be too late for some BTF tags, but we could do better going forward.
Quoted 4 lines
It may also allow us to indicate that a tag is experimental until its effect on the program becomes stabilized. E.g. bpf.experimental.exception_callback instead of
exception_callback? Or do you feel it's unnecessary?
... I don't think any of that is necessary.
Whether btf tag is prefixed with "exception_callback:" or "bpf.experimental.debug.exception_callback:" we will be doing the same thing. We'll keep backward compat if trade-offs allow.
Re: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
In reply to Kumar Kartikeya DwivediQuoted 14 lines · Kumar Kartikeya Dwivedi
Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
The process is done in two steps. First, for each main program, the
bpf_object__sanitize_and_load_btffunction finds and marks its corresponding exception callback as defined by the declaration tag on it. Second,bpf_object__reloc_codeis modified to append the indicated exception callback at the end of the instruction iteration (since exception callback will never be appended in that loop, as it is not directly referenced).
Quoted 2 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Just two point before you send next version:
a) it seems like this appending of exception callback logically fits bpf_object__relocate() step, where other subprogs are appended. Any reason we can't do it there?
b) all the callbacks are static functions, right? Which means in the case of static linking, we can have multiple subprogs with the same name. So this whole look up by name thing doesn't guarantee unique match. At the very least libbpf should check that the match is unique and error out otherwise.
Ideally though, would be great if something like this would be supported instead (but I know it's way more complex, Alexei already mentioned that in person and on the list):
try (my_callback) {
... code that throws ...
}
try (my_other_callback) {
... some other code that throws ...
}
This try() macro can be implemented in a form similar to bpf_for() by using fancy for() loop. It would look and feel way more like try/catch.
Quoted 19 lines
tools/lib/bpf/libbpf.c | 166 +++++++++++++++++++++++++++++++++++------ 1 file changed, 142 insertions(+), 24 deletions(-)tools/lib/bpf/libbpf.c +2
-0@@ -432,9 +432,11 @@ struct bpf_program {int fd;bool autoload;bool autoattach; +bool sym_global;bool mark_btf_static;enum bpf_prog_type type;enum bpf_attach_type expected_attach_type; +int exception_cb_idx;int prog_ifindex;__u32 attach_btf_obj_fd;
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
Quoted 16 lines · Andrii Nakryiko
Quoted 14 lines · Kumar Kartikeya Dwivedi
Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:<value>' and finding the callback in the value of the tag.
The process is done in two steps. First, for each main program, the
bpf_object__sanitize_and_load_btffunction finds and marks its corresponding exception callback as defined by the declaration tag on it. Second,bpf_object__reloc_codeis modified to append the indicated exception callback at the end of the instruction iteration (since exception callback will never be appended in that loop, as it is not directly referenced).
Quoted 9 lines
Quoted 2 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Just two point before you send next version:
a) it seems like this appending of exception callback logically fits
bpf_object__relocate()step, where other subprogs are appended. Any reason we can't do it there?
We should be able to do it there as well. But I felt it is better to do it in bpf_object__reloc_code as the logic is similar to the handling of bpf_pseudo_func/bpf_pseudo_call insns. And then we need to recurse using bpf_object__reloc_code for exception cb again.
Quoted 5 lines
b) all the callbacks are static functions, right? Which means in the case of static linking, we can have multiple subprogs with the same name. So this whole look up by name thing doesn't guarantee unique match. At the very least libbpf should check that the match is unique and error out otherwise.
Ack, will fix this in v3.
Quoted 9 lines
Ideally though, would be great if something like this would be supported instead (but I know it's way more complex, Alexei already mentioned that in person and on the list):
try (
my_callback) {
... code that throws ...
}
Quoted 9 lines
try (
my_other_callback) {
... some other code that throws ...
}This
try()macro can be implemented in a form similar tobpf_for()by using fancyfor()loop. It would look and feel way more like try/catch.
These try blocks are easier than having a try/catch block which the execution jumps to when the exception is thrown. I think the latter will involve some form of compiler support, because otherwise there is no control flow that is seen by the compiler into the catch block, which will mess up things, and I plan to atleast explore that approach (already looking at LLVM) once I am done with the second part of this feature.
Having just these try (callback) {} blocks is easier as we can record which subprog corresponds to [begin_ip, end_ip] (per frame) and stop unwinding when we find a suitable handler for the ip of a parent frame. The callback will be invoked and will return to the parent frame (or kernel if it's the main frame). So if this seems like a more useful thing, I can make this work and send it out as a follow up to this set.
Quoted 1 line
[...]
Re: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks
Quoted 28 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines
Ideally though, would be great if something like this would be supported instead (but I know it's way more complex, Alexei already mentioned that in person and on the list):
try (
my_callback) {
... code that throws ...
}Quoted 9 lines
try (
my_other_callback) {
... some other code that throws ...
}This
try()macro can be implemented in a form similar tobpf_for()by using fancyfor()loop. It would look and feel way more like try/catch.These try blocks are easier than having a try/catch block which the execution jumps to when the exception is thrown. I think the latter will involve some form of compiler support, because otherwise there is no control flow that is seen by the compiler into the catch block, which will mess up things, and I plan to atleast explore that approach (already looking at LLVM) once I am done with the second part of this feature.
Quoted 7 lines
Having just these try (callback) {} blocks is easier as we can record which subprog corresponds to [
begin_ip,end_ip] (per frame) and stop unwinding when we find a suitable handler for the ip of a parent frame. The callback will be invoked and will return to the parent frame (or kernel if it's the main frame). So if this seems like a more useful thing, I can make this work and send it out as a follow up to this set.
I suspect even "just try {}" will not work without compiler support. {} is a semantic structure from compiler pov. It doesn't have any delineation in LLVM IR. They don't exist to optimization passes.
Kumar Kartikeya Dwivedi
maintainerPATCH v2 13/14 selftests/bpf: Add BPF assertion macros
In reply to Kumar Kartikeya DwivediAdd macros implementing an 'assert' statement primitive using macros, built on top of the BPF exceptions support introduced in previous patches.
The bpf_assert_*_with variants allow supplying a value which can the be inspected within the exception handler to signify the assert statement that led to the program being terminated abruptly, or be returned by the default exception handler.
Note that only 64-bit scalar values are supported with these assertion macros, as during testing I found other cases quite unreliable in presence of compiler shifts/manipulations extracting the value of the right width from registers scrubbing the verifier's bounds information and knowledge about the value in the register.
Thus, it is easier to reliably support this feature with only the full register width, and support both signed and unsigned variants.
The bpf_assert_range is interesting in particular, which clamps the value in the [begin, end] (both inclusive) range within verifier state, and emits a check for the same at runtime.
--- .../testing/selftests/bpf/bpf_experimental.h | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+)
tools/testing/selftests/bpf/bpf_experimental.h +242 -0
@@ -176,4 +176,246 @@ extern void bpf_throw(u64 cookie) __ksym;*/ #define __exception_cb (name) __attribute__((btf_decl_tag("exception_callback:" #name))) +#define __bpf_assert_signed (x) _Generic((x), \ + unsigned long: 0, \ + unsigned long long: 0, \ + signed long: 1, \ + signed long long: 1 \ +) + +#define __bpf_assert_check (LHS , op,RHS ) \ + _Static_assert(sizeof(&(LHS)), "1st argument must be an lvalue expression"); \ + _Static_assert(sizeof(LHS) == 8, "Only 8-byte integers are supported\n"); \ + _Static_assert(__builtin_constant_p(__bpf_assert_signed(LHS)), "internal static assert"); \ + _Static_assert(__builtin_constant_p((RHS)), "2nd argument must be a constant expression") + +#define __bpf_assert (LHS , op, cons,RHS ,VAL ) \ + ({ \ + asm volatile ("if %[lhs] " op " %[rhs] goto +2; r1 = %[value]; call bpf_throw" \ + : : [lhs] "r"(LHS), [rhs] cons(RHS), [value] "ri"(VAL) : ); \ + }) + +#define __bpf_assert_op_sign (LHS , op, cons,RHS ,VAL , supp_sign) \ + ({ \ + __bpf_assert_check(LHS, op, RHS); \ + if (__bpf_assert_signed(LHS) && !(supp_sign)) \ + __bpf_assert(LHS, "s" #op, cons, RHS, VAL); \ + else \ + __bpf_assert(LHS, #op, cons, RHS, VAL); \ + }) + +#define __bpf_assert_op (LHS , op,RHS ,VAL , supp_sign) \ + ({ \ + if (sizeof(typeof(RHS)) == 8) { \ + const typeof(RHS) rhs_var = (RHS); \ + __bpf_assert_op_sign(LHS, op, "r", rhs_var, VAL, supp_sign); \ + } else { \ + __bpf_assert_op_sign(LHS, op, "i", RHS, VAL, supp_sign); \ + } \ + }) + +/* Description +* Assert that a conditional expression is true. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert (cond) if (!(cond)) bpf_throw(0); + +/* Description +* Assert that a conditional expression is true. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_with (cond, value) if (!(cond)) bpf_throw(value); + +/* Description +* Assert that LHS is equal to RHS. This statement updates the known value +* of LHS during verification. Note that RHS must be a constant value, and +* must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_eq (LHS ,RHS ) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, 0, true); \ + }) + +/* Description +* Assert that LHS is equal to RHS. This statement updates the known value +* of LHS during verification. Note that RHS must be a constant value, and +* must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_eq_with (LHS ,RHS , value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, value, true); \ + }) + +/* Description +* Assert that LHS is less than RHS. This statement updates the known +* bounds of LHS during verification. Note that RHS must be a constant +* value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_lt (LHS ,RHS ) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, 0, false); \ + }) + +/* Description +* Assert that LHS is less than RHS. This statement updates the known +* bounds of LHS during verification. Note that RHS must be a constant +* value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_lt_with (LHS ,RHS , value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, value, false); \ + }) + +/* Description +* Assert that LHS is greater than RHS. This statement updates the known +* bounds of LHS during verification. Note that RHS must be a constant +* value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_gt (LHS ,RHS ) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, 0, false); \ + }) + +/* Description +* Assert that LHS is greater than RHS. This statement updates the known +* bounds of LHS during verification. Note that RHS must be a constant +* value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_gt_with (LHS ,RHS , value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, value, false); \ + }) + +/* Description +* Assert that LHS is less than or equal to RHS. This statement updates the +* known bounds of LHS during verification. Note that RHS must be a +* constant value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_le (LHS ,RHS ) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, 0, false); \ + }) + +/* Description +* Assert that LHS is less than or equal to RHS. This statement updates the +* known bounds of LHS during verification. Note that RHS must be a +* constant value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_le_with (LHS ,RHS , value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, value, false); \ + }) + +/* Description +* Assert that LHS is greater than or equal to RHS. This statement updates +* the known bounds of LHS during verification. Note that RHS must be a +* constant value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_ge (LHS ,RHS ) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, 0, false); \ + }) + +/* Description +* Assert that LHS is greater than or equal to RHS. This statement updates +* the known bounds of LHS during verification. Note that RHS must be a +* constant value, and must fit within the data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_ge_with (LHS ,RHS , value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, value, false); \ + }) + +/* Description +* Assert that LHS is in the range [BEG, END] (inclusive of both). This +* statement updates the known bounds of LHS during verification. Note +* that both BEG and END must be constant values, and must fit within the +* data type of LHS. +* Returns +* Void. +* Throws +* An exception with the value zero when the assertion fails. +*/ +#define bpf_assert_range (LHS ,BEG ,END ) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, 0, false); \ + __bpf_assert_op(LHS, <=, END, 0, false); \ + }) + +/* Description +* Assert that LHS is in the range [BEG, END] (inclusive of both). This +* statement updates the known bounds of LHS during verification. Note +* that both BEG and END must be constant values, and must fit within the +* data type of LHS. +* Returns +* Void. +* Throws +* An exception with the specified value when the assertion fails. +*/ +#define bpf_assert_range_with (LHS ,BEG ,END , value) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, value, false); \ + __bpf_assert_op(LHS, <=, END, value, false); \ + }) +#endif
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerPATCH v2 14/14 selftests/bpf: Add tests for BPF exceptions
In reply to Kumar Kartikeya DwivediAdd selftests to cover success and failure cases of API usage, runtime behavior and invariants that need to be maintained for implementation correctness.
--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/exceptions.c | 324 +++++++++++++++ .../testing/selftests/bpf/progs/exceptions.c | 368 ++++++++++++++++++ .../selftests/bpf/progs/exceptions_assert.c | 135 +++++++ .../selftests/bpf/progs/exceptions_ext.c | 59 +++ .../selftests/bpf/progs/exceptions_fail.c | 347 +++++++++++++++++ 7 files changed, 1235 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_assert.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ext.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_fail.c
tools/testing/selftests/bpf/DENYLIST.aarch64 +1 -0
@@ -1,5 +1,6 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 +exceptions # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 kprobe_multi_test/attach_api_addrs # bpf_program__attach_kprobe_multi_opts unexpected error: -95
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -6,6 +6,7 @@ bpf_loop # attaches to __x64_sys_nanosleep cgrp_local_storage # prog_attach unexpected error: -524 (trampoline) dynptr/test_dynptr_skb_data dynptr/test_skb_readonly +exceptions # JIT does not support calling kfunc bpf_throw (exceptions) fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) iters/testmod_seq* # s390x doesn't support kfuncs in modules yet
tools/testing/selftests/bpf/prog_tests/exceptions.c new file +324 -0
324 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions.c new file +368 -0
368 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_assert.c new file +135 -0
135 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_ext.c new file +59 -0
59 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_fail.c new file +347 -0
347 lines not on this page · raw message
Signature · 1 line
2.41.0
Kumar Kartikeya Dwivedi
maintainerQuoted 10 lines · Kumar Kartikeya Dwivedi
[...]
Known issues
------------* Just asm volatile ("call
bpf_throw" :::) does not emit DATASEC .ksyms forbpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.
Hi Yonghong, I'd like to ask you about this issue to figure out whether this is something worth fixing in clang or not. It pops up in programs which only use bpf_assert macros (which emit the call to bpf_throw using inline assembly) and not bpf_throw kfunc directly.
I believe in case we emit a call bpf_throw instruction, the BPF backend code will not see any DWARF debug info for the respective symbol, so it will also not be able to convert it and emit anything to .BTF section in case no direct call without asm volatile is present. Therefore my guess is that this isn't something that can be fixed in clang/LLVM.
There are also options like the one below to work around it. if ((volatile int){0}) bpf_throw();
asm volatile ("call bpf_throw");
Quoted 19 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines
[...]
Known issues
------------* Just asm volatile ("call
bpf_throw" :::) does not emit DATASEC .ksyms forbpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.Hi Yonghong, I'd like to ask you about this issue to figure out whether this is something worth fixing in clang or not. It pops up in programs which only use
bpf_assertmacros (which emit the call tobpf_throwusing inline assembly) and notbpf_throwkfunc directly.
Quoted 6 lines
I believe in case we emit a call
bpf_throwinstruction, the BPF backend code will not see any DWARF debug info for the respective symbol, so it will also not be able to convert it and emit anything to .BTF section in case no direct call without asm volatile is present. Therefore my guess is that this isn't something that can be fixed in clang/LLVM.
Besides, please keep in mind that GCC doens't have an integrated assembler, and therefore relying on clang's understanding on the instructions in inline assembly is something to avoid.
Quoted 3 lines
There are also options like the one below to work around it. if ((volatile int){0})
bpf_throw();
asm volatile ("callbpf_throw");
I can confirm the above results in a BTF entry for bpf_throw with bpf-unknown-none-gcc -gbtf.
Yonghong Song
reviewerQuoted 30 lines · Jose E. Marchesi
Quoted 19 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines
[...]
Known issues
------------* Just asm volatile ("call
bpf_throw" :::) does not emit DATASEC .ksyms forbpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.Hi Yonghong, I'd like to ask you about this issue to figure out whether this is something worth fixing in clang or not. It pops up in programs which only use
bpf_assertmacros (which emit the call tobpf_throwusing inline assembly) and notbpf_throwkfunc directly.Quoted 6 lines
I believe in case we emit a call
bpf_throwinstruction, the BPF backend code will not see any DWARF debug info for the respective symbol, so it will also not be able to convert it and emit anything to .BTF section in case no direct call without asm volatile is present. Therefore my guess is that this isn't something that can be fixed in clang/LLVM.Besides, please keep in mind that GCC doens't have an integrated assembler, and therefore relying on clang's understanding on the instructions in inline assembly is something to avoid.
Quoted 6 lines
Quoted 3 lines
There are also options like the one below to work around it. if ((volatile int){0})
bpf_throw();
asm volatile ("callbpf_throw");I can confirm the above results in a BTF entry for
bpf_throwwith bpf-unknown-none-gcc -gbtf.
Kumar, you are correct.
For clang, symbols inside 'asm volatile' statement or generally inside any asm code (e.g., kernel .s files) won't generate an entry in dwarf. The
if ((volatile int){0}) bpf_throw();
will force a dwarf, hence btf, entry.
The unfortunately thing is the above code will generate redundant code like
0000000000000000 <foo>:
0: b7 01 00 00 00 00 00 00 r1 = 0x0
1: 63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 0x4) = r1
2: 61 a1 fc ff 00 00 00 00 r1 = *(u32 *)(r10 - 0x4)
3: 15 01 01 00 00 00 00 00 if r1 == 0x0 goto +0x1 <LBB0_2>
4: 85 10 00 00 ff ff ff ff call -0x10000000000000028 <LBB0_2>:
5: 85 10 00 00 ff ff ff ff call -0x1
6: b7 00 00 00 00 00 00 00 r0 = 0x0
7: 95 00 00 00 00 00 00 00 exitI am curious why in bpf_assert macro bpf_throw() kfunc cannot be used?
Kumar Kartikeya Dwivedi
maintainerQuoted 57 lines · Yonghong Song
Quoted 30 lines · Jose E. Marchesi
Quoted 19 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines
[...]
Known issues
------------* Just asm volatile ("call
bpf_throw" :::) does not emit DATASEC .ksyms forbpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.Hi Yonghong, I'd like to ask you about this issue to figure out whether this is something worth fixing in clang or not. It pops up in programs which only use
bpf_assertmacros (which emit the call tobpf_throwusing inline assembly) and notbpf_throwkfunc directly.Quoted 6 lines
I believe in case we emit a call
bpf_throwinstruction, the BPF backend code will not see any DWARF debug info for the respective symbol, so it will also not be able to convert it and emit anything to .BTF section in case no direct call without asm volatile is present. Therefore my guess is that this isn't something that can be fixed in clang/LLVM.Besides, please keep in mind that GCC doens't have an integrated assembler, and therefore relying on clang's understanding on the instructions in inline assembly is something to avoid.
Quoted 6 lines
Quoted 3 lines
There are also options like the one below to work around it. if ((volatile int){0})
bpf_throw();
asm volatile ("callbpf_throw");I can confirm the above results in a BTF entry for
bpf_throwwith bpf-unknown-none-gcc -gbtf.Kumar, you are correct.
For clang, symbols inside 'asm volatile' statement or generally inside any asm code (e.g., kernel .s files) won't generate an entry in dwarf. The
if ((volatile int){0})bpf_throw();
will force a dwarf, hence btf, entry.The unfortunately thing is the above code will generate redundant code like
0000000000000000 <foo>:0: b7 01 00 00 00 00 00 00 r1 = 0x0 1: 63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 0x4) = r1 2: 61 a1 fc ff 00 00 00 00 r1 = *(u32 *)(r10 - 0x4) 3: 15 01 01 00 00 00 00 00 if r1 == 0x0 goto +0x1 <LBB0_2> 4: 85 10 00 00 ff ff ff ff call -0x1
Quoted 5 lines
0000000000000028 <
LBB0_2>:5: 85 10 00 00 ff ff ff ff call -0x1 6: b7 00 00 00 00 00 00 00 r0 = 0x0 7: 95 00 00 00 00 00 00 00 exit
Yes, I am relying on the verifier to eliminate dead code later, but it is obviously a hack.
Quoted 2 lines
I am curious why in
bpf_assertmacrobpf_throw()kfunc cannot be used?
The reason was to force the compiler to emit a specific branch for the assertion check without being influenced by compiler optimizations, and tying comparison to the register holding a value being tested in assertion. Secondly we also enforce that the first argument is in a register and the second a constant, so as to apply the verifier bounds gained after comparison op to the original register.
I am aware (though correct me if this is wrong) that the compiler can select a different register for the input operand of the asm constraint, but find_equal_id_scalars should still do correct propagation. This is partly why I disabled usage of this macro for variable widths < 64-bit, because then the compiler sometimes performs shifts etc. for integer promotion implicitly, sometimes destroying whatever information we gained through an assertion check. Depending on the signedness of the variable, we emit the signed/unsigned comparison.
I suppose we could switch to the ' if (!(LHS <op> RHS)) bpf_throw(); ' sequence in C, force volatile load for LHS and __builtin_constant_p for RHS to get the same behavior. Emitting these redundant checks is definitely a bit weird just to emit BTF.
Quoted 6 lines · Kumar Kartikeya Dwivedi
I suppose we could switch to the ' if (!(LHS <op> RHS))
bpf_throw(); ' sequence in C, force volatile load for LHS and__builtin_constant_pfor RHS to get the same behavior. Emitting these redundant checks is definitely a bit weird just to emit BTF.
I guess we can try
#define bpf_assert(LHS, OP, RHS) if (!(LHS OP RHS)) bpf_throw(); with barrier_var(LHS) and __builtin_constant_p(RHS) and
keep things completely in C,
but there is no guarantee that the compiler will not convert == to !=, swap lhs and rhs, etc.
Maybe we can have both asm and C style macros, then recommend C to start and switch to asm if things are dodgy.
Feels like dangerous ambiguity.
Quoted 10 lines · Alexei Starovoitov
Quoted 6 lines · Kumar Kartikeya Dwivedi
I suppose we could switch to the ' if (!(LHS <op> RHS))
bpf_throw(); ' sequence in C, force volatile load for LHS and__builtin_constant_pfor RHS to get the same behavior. Emitting these redundant checks is definitely a bit weird just to emit BTF.
Quoted 9 lines
I guess we can try
#definebpf_assert(LHS, OP, RHS)if (!(LHS OP RHS))bpf_throw(); withbarrier_var(LHS)and__builtin_constant_p(RHS)and
keep things completely in C,
but there is no guarantee that the compiler will not convert == to !=, swap lhs and rhs, etc.
Maybe we can have both asm and C style macros, then recommend C to start and switch to asm if things are dodgy.
Feels like dangerous ambiguity.
This seems similar to the issue I had with __attribute__((cleanup(some_kfunc)))) not emitting BTF info for that some_kfunc? See bpf_for_each(), seems like just adding (void)bpf_iter_##type##_destroy makes Clang emit BTF info.
It would be nice to have this fixed for cleanup() attribute and asm, of course. But this is a simple work around.
Kumar Kartikeya Dwivedi
maintainerQuoted 23 lines · Andrii Nakryiko
Quoted 10 lines · Alexei Starovoitov
Quoted 6 lines · Kumar Kartikeya Dwivedi
I suppose we could switch to the ' if (!(LHS <op> RHS))
bpf_throw(); ' sequence in C, force volatile load for LHS and__builtin_constant_pfor RHS to get the same behavior. Emitting these redundant checks is definitely a bit weird just to emit BTF.Quoted 9 lines
I guess we can try
#definebpf_assert(LHS, OP, RHS)if (!(LHS OP RHS))bpf_throw(); withbarrier_var(LHS)and__builtin_constant_p(RHS)and
keep things completely in C,
but there is no guarantee that the compiler will not convert == to !=, swap lhs and rhs, etc.
Maybe we can have both asm and C style macros, then recommend C to start and switch to asm if things are dodgy.
Feels like dangerous ambiguity.
Quoted 7 lines
This seems similar to the issue I had with __attribute__((cleanup(some_kfunc)))) not emitting BTF info for that
some_kfunc? Seebpf_for_each(), seems like just adding(void)bpf_iter_##type##_destroymakes Clang emit BTF info.It would be nice to have this fixed for
cleanup()attribute and asm, of course. But this is a simple work around.
Good to know, this is cleaner than my solution. But I am planning to switch to the direct C approach, so it should not be needed anymore.
Kumar Kartikeya Dwivedi
maintainerQuoted 27 lines · Jose E. Marchesi
Quoted 20 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines
[...]
Known issues
------------* Just asm volatile ("call
bpf_throw" :::) does not emit DATASEC .ksyms forbpf_throw, there needs to be explicit call in C for clang to emit the DATASEC info in BTF, leading to errors during compilation.Hi Yonghong, I'd like to ask you about this issue to figure out whether this is something worth fixing in clang or not. It pops up in programs which only use
bpf_assertmacros (which emit the call tobpf_throwusing inline assembly) and notbpf_throwkfunc directly.Quoted 6 lines
I believe in case we emit a call
bpf_throwinstruction, the BPF backend code will not see any DWARF debug info for the respective symbol, so it will also not be able to convert it and emit anything to .BTF section in case no direct call without asm volatile is present. Therefore my guess is that this isn't something that can be fixed in clang/LLVM.
Quoted 4 lines
Besides, please keep in mind that GCC doens't have an integrated assembler, and therefore relying on clang's understanding on the instructions in inline assembly is something to avoid.
Thank you for reminding me that. I will be more careful about this. We certainly cannot rely on clang-specific behavior for this.
Quoted 6 lines
Quoted 3 lines
There are also options like the one below to work around it. if ((volatile int){0})
bpf_throw();
asm volatile ("callbpf_throw");I can confirm the above results in a BTF entry for
bpf_throwwith bpf-unknown-none-gcc -gbtf.
Thanks for confirming.