BPF indirect jumps
Subsystems (9)
v6 cover BPF indirect jumps
This patchset implements a new type of map, instruction set, and uses it to build support for indirect branches in BPF (on x86). (The same map will be later used to provide support for indirect calls and static keys.) See [1], [2] for more context.
Short table of contents:
* Patches 1-7 implement the new map of type BPF_MAP_TYPE_INSN_SET and corresponding selftests. This map can be used to track the "original -> xlated -> jitted mapping" for a given program. Patches 6,7 add support for "blinded" variant.
* Patches 8-12 implement the support for indirect jumps
* Patches 13-17 add support for LLVM-compiled programs containing indirect jumps and selftests.
Since the v3 the jump table support was merged to LLVM and now can be enabled with -mcpu=v4. See [3] for the PR which added the support. The selftests, however, don't use LLVM for now: the verifier_gotox is implemented in asm, and bpf_gotox is actually turned off, as CI only runs with LLVM 20, and the indirect jumps are supported starting from 22.
See individual patches for more details on the implementation details.
v5 -> v6 (this series):
* instruction arrays:
* better document struct bpf_insn_array_value (Eduard)
* remove a condition in bpf_insn_array_adjust_after_remove (Eduard)
* make userspace see original, xlated, and jitted indexes (+original) (Eduard)
* indirect jumps, kernel:
* reject writes to the map
* reject unaligned ops
* add a check what w is not outside the program in check_config for gotox (Eduard)
* do not introduce unneeded bpf_find_containing_subprog_idx
* simplify error processing for bpf_find_containing_subprog (Eduard)
* add insn_state |= DISCOVERED when it's discovered (Eduard)
* support SUB operations on PTR_TO_INSN (Eduard)
* make gotox_tmp_buf a bpf_iarray and use helper to relocate it (Eduard)
* rename fields of bpf_iarray to more generic (Eduard)
* re-implement visit_gotox_insn in a loop (Eduard)
* some minor cleanups (Eduard)
* libbpf:
* struct reloc_desc: add a comment about union (Eduard)
* rename parameters of (and one other place in code) {create,add}_jt_map to sym_off (Eduard)
* create_jt_map: check that size/off are 8-byte aligned (Eduard)
* Selftests:
* instruction array selftests:
* only run tests on x86_64
* write a more generic function to test things to reduce code (Eduard)
* errno wasn't used in checks, so don't reset it (Eduard)
* print i, xlated_off and map_out[i] here (Eduard)
* added verifier_gotox selftests which do not depend on LLVM:
* disabled bpf_gotox tests by default
* other changes:
* remove an extra function in bpf disasm (Eduard)
* some minor cleanups in the insn_successors patch (Eduard)
* update documentation in Documentation/bpf/linux-notes.html about jumps, now it is supported :)
v3 -> v4 -> v5 (https://lore.kernel.org/bpf/[email protected]/):
* [v4 -> v5] rebased on top of the last bpf-next/master
* instruction arrays:
* add copyright (Alexei)
* remove mutexes, add frozen back (Alexei)
* setup 1:1 prog-map correspondence using atomic_xchg
* do not copy/paste array_map_get_next_key, add a common helper (Alexei)
* misc minor code cleanups (Alexei)
* indirect jumps, kernel side:
* remove jt_allocated, just check if insn is gotox (Eduard)
* use copy_register_state instead of individual copies (Eduard)
* in push_stack is_speculative should be inherited (Eduard)
* a few cleanups for insn_successors, including omitting error path (Eduard)
* check if reserved fields are used when considering gotox instruction (Eduard)
* read size and alignment of read from insn_array should be 8 (Eduard)
* put buffer for sorting in subfun info and realloc to grow as needed (Eduard)
* properly do jump_point / prune_point from push_gotox_edge (Eduard)
* use range_within to check states (Eduard)
* some minor cleanups and fix commit message (Eduard)
* indirect jumps, libbpf side:
* close map_fd in some error paths in create_jt_map (Andrii)
* maps for jump tables are actually not closed at all, fix this (Andrii)
* rename map from jt to .jumptables (Andrii)
* use errstr in an error message (Andrii)
* rephrase error message to look more standard (Andrii)
* misc other minor renames and cleanups (Andrii)
* selftests:
* add the frozen selftest back
* add a selftest for two jumps loading same table
* some other changes:
* rebase and split insn_successor changes into separate patch
* use PTR_ERR_OR_ZERO in the push stack patch (Eduard)
* indirect jumps on x86: properly re-read *pprog (Eduard)
v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/):
* fix build failure when CONFIG_BPF_SYSCALL is not set (kbuild-bot)
* reformat bpftool help messages (Quentin)
v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/):
* push_stack changes:
* sanitize_speculative_path should just return int (Eduard)
* return code from sanitize_speculative_path, not EFAULT (Eduard)
* when BPF_COMPLEXITY_LIMIT_JMP_SEQ is reached, return E2BIG (Eduard)
* indirect jumps:
* omit support for .imm=fd in gotox, as we're not using it for now (Eduard)
* struct jt -> struct bpf_iarray (Eduard)
* insn_successors: rewrite the interface to just return a pointer (Eduard)
* remove min_index/max_index, use umin_value/umax_value instead (Alexei, Eduard)
* move emit_indirect_jump args change to the previous patch (Eduard)
* add a comment to map_mem_size() (Eduard)
* use verifier_bug for some error cases in check_indirect_jump (Eduard)
* clear_insn_aux_data: use start,len instead of start,end (Eduard)
* make regs[insn->dst_reg].type = PTR_TO_INSN part of check_mem_access (Eduard)
* constant blinding changes:
* make subprog_start adjustment better readable (Eduard)
* do not set subprog len, it is already set (Eduard)
* libbpf:
* remove check that relocations from .rodata are ok (Anton)
* do not freeze the map, it is not necessary anymore (Anton)
* rename the goto_x -> gotox everywhere (Anton)
* use u64 when parsing LLVM jump tables (Eduard)
* split patch in two due to spaces->tabs change (Eduard)
* split bpftool changes to bpftool patch (Andrii)
* make sym_size it a union with ext_idx (Andrii)
* properly copy/free the jumptables_data section from elf (Andrii)
* a few cosmetic changes around create_jt_map (Andrii)
* fix some comments + rewrite patch description (Andrii)
* inline bpf_prog__append_subprog_offsets (Andrii)
* subprog_sec_offst -> subprog_sec_off (Andrii)
* !strcmp -> strcmp() == 0 (Andrii)
* make some function names more readable (Andrii)
* allocate table of subfunc offsets via libbpf_reallocarray (Andrii)
* selftests:
* squash insn_array* tests together (Anton)
* fixed build warnings (kernel test robot)
RFC -> v1 (https://lore.kernel.org/bpf/[email protected]/):
* I've tried to address all the comments provided by Alexei and Eduard in RFC. Will try to list the most important of them below.
* One big change: move from older LLVM version [5] to newer [4]. Now LLVM generates jump tables as symbols in the new special section ".jumptables". Another part of this change is that libbpf now doesn't try to link map load and goto *rX, as
1) this is absolutely not reliable 2) for some use cases this is impossible (namely, when more than one jump table can be used in the same gotox instruction).
* Added insn_successors() support (Alexei, Eduard). This includes getting rid of the ugly bpf_insn_set_iter_xlated_offset() interface (Eduard).
* Removed hack for the unreachable instruction, as new LLVM thank to Eduard doesn't generate it.
* Set mem_size for direct map access properly instead of hacking. Remove off>0 check. (Alexei)
* Do not allocate new memory for min_index/max_index (Alexei, Eduard)
* Information required during check_cfg is now cached to be reused later (Alexei + general logic for supporting multiple JT per jump)
* Properly compare registers in regsafe (Alexei, Eduard)
* Remove support for JMP32 (Eduard)
* Better checks in adjust_ptr_min_max_vals (Eduard)
* More selftests were added (but still there's room for more) which directly use gotox (Alexei)
* More checks and verbose messages added
* "unique pointers" are no more in the map
Links:
1. https://lpc.events/event/18/contributions/1941/
2. https://lwn.net/Articles/1017439/
3. https://github.com/llvm/llvm-project/pull/149715
4. https://github.com/llvm/llvm-project/pull/149715#issuecomment-3274833753
6. rfc: https://lore.kernel.org/bpf/[email protected]/
Anton Protopopov (17):
bpf: fix the return value of push_stack
bpf: save the start of functions in bpf_prog_aux
bpf: generalize and export map_get_next_key for arrays bpf, x86: add new map type: instructions array selftests/bpf: add selftests for new insn_array map
bpf: support instructions arrays with constants blinding selftests/bpf: test instructions arrays with blinding bpf, x86: allow indirect jumps to r8...r15
bpf: make bpf_insn_successors to return a pointer bpf, x86: add support for indirect jumps
bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X bpf, docs: do not state that indirect jumps are not supported
libbpf: fix formatting of bpf_object__append_subprog_code
libbpf: support llvm-generated indirect jumps
bpftool: Recognize insn_array map type selftests/bpf: add new verifier_gotox test selftests/bpf: add C-level selftests for indirect jumps
Documentation/bpf/linux-notes.rst | 8 - arch/x86/net/bpf_jit_comp.c | 39 +- include/linux/bpf.h | 44 ++ include/linux/bpf_types.h | 1 + include/linux/bpf_verifier.h | 23 +- include/uapi/linux/bpf.h | 21 + kernel/bpf/Makefile | 2 +- kernel/bpf/arraymap.c | 19 +- kernel/bpf/bpf_insn_array.c | 303 ++++++++++ kernel/bpf/core.c | 21 + kernel/bpf/disasm.c | 3 + kernel/bpf/liveness.c | 39 +- kernel/bpf/log.c | 1 + kernel/bpf/syscall.c | 22 + kernel/bpf/verifier.c | 556 +++++++++++++++--- .../bpf/bpftool/Documentation/bpftool-map.rst | 3 +- tools/bpf/bpftool/map.c | 3 +- tools/include/uapi/linux/bpf.h | 21 + tools/lib/bpf/libbpf.c | 282 ++++++++- tools/lib/bpf/libbpf_probes.c | 4 + tools/lib/bpf/linker.c | 10 +- tools/testing/selftests/bpf/Makefile | 4 +- .../selftests/bpf/prog_tests/bpf_gotox.c | 185 ++++++ .../selftests/bpf/prog_tests/bpf_insn_array.c | 492 ++++++++++++++++ .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/bpf_gotox.c | 414 +++++++++++++ .../selftests/bpf/progs/verifier_gotox.c | 277 +++++++++ 27 files changed, 2667 insertions(+), 132 deletions(-) create mode 100644 kernel/bpf/bpf_insn_array.c create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c
Signature · 1 line
2.34.1
PATCH v6 1/17 bpf: fix the return value of push_stack
In [1] Eduard mentioned that on push_stack failure verifier code should return -ENOMEM instead of -EFAULT. After checking with the other call sites I've found that code randomly returns either -ENOMEM or -EFAULT. This patch unifies the return values for the push_stack (and similar push_async_cb) functions such that error codes are always assigned properly.
[1] https://lore.kernel.org/bpf/[email protected]
--- kernel/bpf/verifier.c | 80 +++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 40 deletions(-)
kernel/bpf/verifier.c +40 -40
@@ -2109,7 +2109,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, elem =kzalloc (sizeof (struct bpf_verifier_stack_elem ),GFP_KERNEL_ACCOUNT );if (!elem) -return NULL ; +return ERR_PTR (-ENOMEM ); elem->insn_idx = insn_idx; elem->prev_insn_idx = prev_insn_idx;
@@ -2119,12 +2119,12 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, env->stack_size++; err =copy_verifier_state (&elem->st, cur);if (err) -return NULL ; +return ERR_PTR (-ENOMEM ); elem->st.speculative |= speculative;if (env->stack_size >BPF_COMPLEXITY_LIMIT_JMP_SEQ ) {verbose (env,"The sequence of %d jumps is too complex.\n" , env->stack_size); -return NULL ; +return ERR_PTR (-E2BIG ); }if (elem->st.parent) { ++elem->st.parent->branches;
@@ -2919,7 +2919,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, elem =kzalloc (sizeof (struct bpf_verifier_stack_elem ),GFP_KERNEL_ACCOUNT );if (!elem) -return NULL ; +return ERR_PTR (-ENOMEM ); elem->insn_idx = insn_idx; elem->prev_insn_idx = prev_insn_idx;
@@ -2931,7 +2931,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,verbose (env,"The sequence of %d jumps is too complex for async cb.\n" , env->stack_size); -return NULL ; +return ERR_PTR (-E2BIG ); }/* Unlike push_stack() do not copy_verifier_state(). * The caller state doesn't matter.
@@ -2942,7 +2942,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, elem->st.in_sleepable = is_sleepable; frame =kzalloc (sizeof (*frame),GFP_KERNEL_ACCOUNT );if (!frame) -return NULL ; +return ERR_PTR (-ENOMEM ); init_func_state(env, frame, BPF_MAIN_FUNC /* callsite */, 0 /* frameno within this callchain */,
@@ -9045,8 +9045,8 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx, prev_st =find_prev_entry (env, cur_st->parent, insn_idx);/* branch out active iter state */ queued_st =push_stack (env, insn_idx +1 , insn_idx, false); -if (!queued_st) -return -ENOMEM ; +if (IS_ERR (queued_st)) +return PTR_ERR (queued_st); queued_iter =get_iter_from_state (queued_st, meta); queued_iter->iter.state =BPF_ITER_STATE_ACTIVE ;
@@ -10616,8 +10616,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins async_cb =push_async_cb (env, env->subprog_info[subprog].start, insn_idx, subprog,is_async_cb_sleepable (env, insn)); -if (!async_cb) -return -EFAULT ; +if (IS_ERR (async_cb)) +return PTR_ERR (async_cb); callee = async_cb->frame[0 ]; callee->async_entry_cnt = caller->async_entry_cnt +1 ;
@@ -10633,8 +10633,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins* proceed with next instruction within current frame. */ callback_state =push_stack (env, env->subprog_info[subprog].start, insn_idx, false); -if (!callback_state) -return -ENOMEM ; +if (IS_ERR (callback_state)) +return PTR_ERR (callback_state); err =setup_func_entry (env, subprog, insn_idx, set_callee_state_cb, callback_state);
@@ -13859,9 +13859,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,struct bpf_reg_state *regs; branch =push_stack (env, env->insn_idx +1 , env->insn_idx, false); -if (!branch) { +if (IS_ERR (branch)) {verbose (env,"failed to push state for failed lock acquisition\n" ); -return -ENOMEM ; +return PTR_ERR (branch); } regs = branch->frame[branch->curframe]->regs;
@@ -14316,16 +14316,15 @@ struct bpf_sanitize_info {bool mask_to_left; }; -static struct bpf_verifier_state * -sanitize_speculative_path (struct bpf_verifier_env *env, -const struct bpf_insn *insn, -u32 next_idx,u32 curr_idx) +static int sanitize_speculative_path (struct bpf_verifier_env *env, +const struct bpf_insn *insn, +u32 next_idx,u32 curr_idx) {struct bpf_verifier_state *branch;struct bpf_reg_state *regs; branch =push_stack (env, next_idx, curr_idx, true); -if (branch && insn) { +if (!IS_ERR (branch) && insn) { regs = branch->frame[branch->curframe]->regs;if (BPF_SRC (insn->code) ==BPF_K ) {mark_reg_unknown (env, regs, insn->dst_reg);
@@ -14334,7 +14333,7 @@ sanitize_speculative_path(struct bpf_verifier_env *env,mark_reg_unknown (env, regs, insn->src_reg); } } -return branch; +return PTR_ERR_OR_ZERO (branch); } static int sanitize_ptr_alu(struct bpf_verifier_env *env,
@@ -14353,7 +14352,6 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,u8 opcode =BPF_OP (insn->code);u32 alu_state, alu_limit;struct bpf_reg_state tmp; -bool ret;int err;if (can_skip_alu_sanitation (env, insn))
@@ -14426,11 +14424,12 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env, tmp = *dst_reg;copy_register_state (dst_reg, ptr_reg); } - ret =sanitize_speculative_path (env,NULL , env->insn_idx +1 , - env->insn_idx); -if (!ptr_is_dst_reg && ret) + err =sanitize_speculative_path (env,NULL , env->insn_idx +1 , env->insn_idx); +if (err <0 ) +return REASON_STACK ; +if (!ptr_is_dst_reg) *dst_reg = tmp; -return !ret ?REASON_STACK :0 ; +return 0 ; }static void sanitize_mark_insn_seen (struct bpf_verifier_env *env)
@@ -16750,8 +16749,8 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,/* branch out 'fallthrough' insn as a new state to explore */ queued_st =push_stack (env, idx +1 , idx, false); -if (!queued_st) -return -ENOMEM ; +if (IS_ERR (queued_st)) +return PTR_ERR (queued_st); queued_st->may_goto_depth++;if (prev_st)
@@ -16829,10 +16828,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,* the fall-through branch for simulation under speculative * execution. */ -if (!env->bypass_spec_v1 && - !sanitize_speculative_path (env, insn, *insn_idx +1 , - *insn_idx)) -return -EFAULT ; +if (!env->bypass_spec_v1) { + err =sanitize_speculative_path (env, insn, *insn_idx +1 , *insn_idx); +if (err <0 ) +return err; + }if (env->log.level &BPF_LOG_LEVEL )print_insn_state (env, this_branch, this_branch->curframe); *insn_idx += insn->off;
@@ -16842,11 +16842,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,* program will go. If needed, push the goto branch for * simulation under speculative execution. */ -if (!env->bypass_spec_v1 && - !sanitize_speculative_path (env, insn, - *insn_idx + insn->off +1 , - *insn_idx)) -return -EFAULT ; +if (!env->bypass_spec_v1) { + err =sanitize_speculative_path (env, insn, *insn_idx + insn->off +1 , + *insn_idx); +if (err <0 ) +return err; + }if (env->log.level &BPF_LOG_LEVEL )print_insn_state (env, this_branch, this_branch->curframe);return 0 ;
@@ -16867,10 +16868,9 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,return err; } - other_branch =push_stack (env, *insn_idx + insn->off +1 , *insn_idx, - false); -if (!other_branch) -return -EFAULT ; + other_branch =push_stack (env, *insn_idx + insn->off +1 , *insn_idx, false); +if (IS_ERR (other_branch)) +return PTR_ERR (other_branch); other_branch_regs = other_branch->frame[other_branch->curframe]->regs;if (BPF_SRC (insn->code) ==BPF_X ) {
Signature · 1 line
2.34.1
PATCH v6 2/17 bpf: save the start of functions in bpf_prog_aux
In reply to Anton ProtopopovIntroduce a new subprog_start field in bpf_prog_aux. This field may be used by JIT compilers wanting to know the real absolute xlated offset of the function being jitted. The func_info[func_id] may have served this purpose, but func_info may be NULL, so JIT compilers can't rely on it.
--- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 1 + 2 files changed, 2 insertions(+)
include/linux/bpf.h +1 -0
@@ -1623,6 +1623,7 @@ struct bpf_prog_aux {u32 ctx_arg_info_size;u32 max_rdonly_access;u32 max_rdwr_access; +u32 subprog_start;struct btf *attach_btf;struct bpf_ctx_arg_aux *ctx_arg_info;void __percpu *priv_stack_ptr;
kernel/bpf/verifier.c +1 -0
@@ -21607,6 +21607,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->func_idx = i;/* Below members will be freed only at prog->aux */ func[i]->aux->btf = prog->aux->btf; + func[i]->aux->subprog_start = subprog_start; func[i]->aux->func_info = prog->aux->func_info; func[i]->aux->func_info_cnt = prog->aux->func_info_cnt; func[i]->aux->poke_tab = prog->aux->poke_tab;
Signature · 1 line
2.34.1
PATCH v6 3/17 bpf: generalize and export map_get_next_key for arrays
In reply to Anton ProtopopovThe kernel/bpf/array.c file defines the array_map_get_next_key() function which finds the next key for array maps. It actually doesn't use any map fields besides the generic max_entries field. Generalize it, and export as bpf_array_get_next_key() such that it can be re-used by other array-like maps.
--- include/linux/bpf.h | 6 ++++++ kernel/bpf/arraymap.c | 19 +++++++++---------- 2 files changed, 15 insertions(+), 10 deletions(-)
include/linux/bpf.h +6 -0
@@ -2107,6 +2107,12 @@ struct bpf_array { }; }; +/* +* The bpf_array_get_next_key() function may be used for all array-like +* maps, i.e., maps with u32 keys with range [0 ,..., max_entries) +*/ +int bpf_array_get_next_key (struct bpf_map *map,void *key,void *next_key); +#define BPF_COMPLEXITY_LIMIT_INSNS 1000000/* yes. 1M insns */ #define MAX_TAIL_CALL_CNT 33
kernel/bpf/arraymap.c +9 -10
@@ -335,18 +335,17 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) }/* Called from syscall */ -static int array_map_get_next_key (struct bpf_map *map,void *key,void *next_key) +int bpf_array_get_next_key (struct bpf_map *map,void *key,void *next_key) { - struct bpf_array *array = container_of(map, struct bpf_array, map);u32 index = key ? *(u32 *)key :U32_MAX ;u32 *next = (u32 *)next_key; -if (index >= array->map.max_entries) { +if (index >= map->max_entries) { *next =0 ;return 0 ; } -if (index == array->map.max_entries -1 ) +if (index == map->max_entries -1 )return -ENOENT ; *next = index +1 ;
@@ -789,7 +788,7 @@ const struct bpf_map_ops array_map_ops = { .map_alloc_check = array_map_alloc_check, .map_alloc = array_map_alloc, .map_free = array_map_free, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_release_uref = array_map_free_internal_structs, .map_lookup_elem = array_map_lookup_elem, .map_update_elem = array_map_update_elem,
@@ -815,7 +814,7 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_alloc_check = array_map_alloc_check, .map_alloc = array_map_alloc, .map_free = array_map_free, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, .map_update_elem = array_map_update_elem,
@@ -1204,7 +1203,7 @@ const struct bpf_map_ops prog_array_map_ops = { .map_poke_track = prog_array_map_poke_track, .map_poke_untrack = prog_array_map_poke_untrack, .map_poke_run = prog_array_map_poke_run, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = fd_array_map_lookup_elem, .map_delete_elem = fd_array_map_delete_elem, .map_fd_get_ptr = prog_fd_array_get_ptr,
@@ -1308,7 +1307,7 @@ const struct bpf_map_ops perf_event_array_map_ops = { .map_alloc_check = fd_array_map_alloc_check, .map_alloc = array_map_alloc, .map_free = perf_event_fd_array_map_free, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = fd_array_map_lookup_elem, .map_delete_elem = fd_array_map_delete_elem, .map_fd_get_ptr = perf_event_fd_array_get_ptr,
@@ -1344,7 +1343,7 @@ const struct bpf_map_ops cgroup_array_map_ops = { .map_alloc_check = fd_array_map_alloc_check, .map_alloc = array_map_alloc, .map_free = cgroup_fd_array_free, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = fd_array_map_lookup_elem, .map_delete_elem = fd_array_map_delete_elem, .map_fd_get_ptr = cgroup_fd_array_get_ptr,
@@ -1429,7 +1428,7 @@ const struct bpf_map_ops array_of_maps_map_ops = { .map_alloc_check = fd_array_map_alloc_check, .map_alloc = array_of_map_alloc, .map_free = array_of_map_free, - .map_get_next_key = array_map_get_next_key, + .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = array_of_map_lookup_elem, .map_delete_elem = fd_array_map_delete_elem, .map_fd_get_ptr = bpf_map_fd_get_ptr,
Signature · 1 line
2.34.1
PATCH v6 4/17 bpf, x86: add new map type: instructions array
In reply to Anton ProtopopovOn bpf(BPF_PROG_LOAD) syscall user-supplied BPF programs are translated by the verifier into "xlated" BPF programs. During this process the original instructions offsets might be adjusted and/or individual instructions might be replaced by new sets of instructions, or deleted.
Add a new BPF map type which is aimed to keep track of how, for a given program, the original instructions were relocated during the verification. Also, besides keeping track of the original -> xlated mapping, make x86 JIT to build the xlated -> jitted mapping for every instruction listed in an instruction array. This is required for every future application of instruction arrays: static keys, indirect jumps and indirect calls.
A map of the BPF_MAP_TYPE_INSN_ARRAY type must be created with a u32 keys and value of size 8. The values have different semantics for userspace and for BPF space. For userspace a value consists of two u32 values – xlated and jitted offsets. For BPF side the value is a real pointer to a jitted instruction.
On map creation/initialization, before loading the program, each element of the map should be initialized to point to an instruction offset within the program. Before the program load such maps should be made frozen. After the program verification xlated and jitted offsets can be read via the bpf(2) syscall.
If a tracked instruction is removed by the verifier, then the xlated offset is set to (u32)-1 which is considered to be too big for a valid BPF program offset.
One such a map can, obviously, be used to track one and only one BPF program. If the verification process was unsuccessful, then the same map can be re-used to verify the program with a different log level. However, if the program was loaded fine, then such a map, being frozen in any case, can't be reused by other programs even after the program release.
Example. Consider the following original and xlated programs:
Original prog: Xlated prog:
0: r1 = 0x0 0: r1 = 0
1: *(u32 *)(r10 - 0x4) = r1 1: *(u32 *)(r10 -4) = r1
2: r2 = r10 2: r2 = r10
3: r2 += -0x4 3: r2 += -4
4: r1 = 0x0 ll 4: r1 = map[id:88]
6: call 0x1 6: r1 += 272
7: r0 = *(u32 *)(r2 +0)
8: if r0 >= 0x1 goto pc+3
9: r0 <<= 3
10: r0 += r1
11: goto pc+1
12: r0 = 0
7: r6 = r0 13: r6 = r0
8: if r6 == 0x0 goto +0x2 14: if r6 == 0x0 goto pc+4
9: call 0x76 15: r0 = 0xffffffff8d2079c0
17: r0 = *(u64 *)(r0 +0)
10: *(u64 *)(r6 + 0x0) = r0 18: *(u64 *)(r6 +0) = r0
11: r0 = 0x0 19: r0 = 0x0
12: exit 20: exitAn instruction array map, containing, e.g., instructions [0,4,7,12] will be translated by the verifier to [0,4,13,20]. A map with index 5 (the middle of 16-byte instruction) or indexes greater than 12 (outside the program boundaries) would be rejected.
The functionality provided by this patch will be extended in consequent patches to implement BPF Static Keys, indirect jumps, and indirect calls.
--- arch/x86/net/bpf_jit_comp.c | 8 + include/linux/bpf.h | 36 +++++ include/linux/bpf_types.h | 1 + include/linux/bpf_verifier.h | 2 + include/uapi/linux/bpf.h | 21 +++ kernel/bpf/Makefile | 2 +- kernel/bpf/bpf_insn_array.c | 288 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 22 +++ kernel/bpf/verifier.c | 43 +++++ tools/include/uapi/linux/bpf.h | 21 +++ 10 files changed, 443 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/bpf_insn_array.c
arch/x86/net/bpf_jit_comp.c +8 -0
@@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +u32 abs_xlated_off = bpf_prog->aux->subprog_start + i -1 ;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;
@@ -2751,6 +2752,13 @@ st: if (is_imm8(insn->off))return -EFAULT ; }memcpy (rw_image + proglen, temp, ilen); + +/* +* Instruction arrays need to know how xlated code +* maps to jitted code +*/ +bpf_prog_update_insn_ptr (bpf_prog, abs_xlated_off, proglen, + image + proglen); } proglen += ilen; addrs[i] = proglen;
include/linux/bpf.h +36 -0
@@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * const char **linep, int *nump);struct bpf_prog *bpf_prog_find_from_stack (void ); +int bpf_insn_array_init (struct bpf_map *map,const struct bpf_prog *prog); +int bpf_insn_array_ready (struct bpf_map *map); +void bpf_insn_array_release (struct bpf_map *map); +void bpf_insn_array_adjust (struct bpf_map *map,u32 off,u32 len); +void bpf_insn_array_adjust_after_remove (struct bpf_map *map,u32 off,u32 len); + +/* +* The struct bpf_insn_ptr structure describes a pointer to a +* particular instruction in a loaded BPF program. Initially +* it is initialised from userspace via user_value.xlated_off. +* During the program verification all other fields are populated +* accordingly: +* +* jitted_ip: address of the instruction in the jitted image +* user_value: user-visible original, xlated, and jitted offsets +*/ +struct bpf_insn_ptr { +void *jitted_ip; +struct bpf_insn_array_value user_value; +}; + +#ifdef CONFIG_BPF_SYSCALL +void bpf_prog_update_insn_ptr (struct bpf_prog *prog, +u32 xlated_off, +u32 jitted_off, +void *jitted_ip); +#else +static inline void +bpf_prog_update_insn_ptr (struct bpf_prog *prog, +u32 xlated_off, +u32 jitted_off, +void *jitted_ip) +{ +} +#endif +#endif /* _LINUX_BPF_H */
include/linux/bpf_types.h +1 -0
@@ -133,6 +133,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)BPF_LINK_TYPE (BPF_LINK_TYPE_TRACING , tracing)
include/linux/bpf_verifier.h +2 -0
@@ -745,8 +745,10 @@ struct bpf_verifier_env {struct list_head free_list;/* list of struct bpf_verifier_state_list */ struct bpf_map *used_maps[MAX_USED_MAPS ];/* array of map's used by eBPF program */ struct btf_mod_pair used_btfs[MAX_USED_BTFS ];/* array of BTF's used by BPF program */ +struct bpf_map *insn_array_maps[MAX_USED_MAPS ];/* array of INSN_ARRAY map's to be relocated */ u32 used_map_cnt;/* number of used maps */ u32 used_btf_cnt;/* number of used BTF objects */ +u32 insn_array_map_cnt;/* number of used maps of type BPF_MAP_TYPE_INSN_ARRAY */ u32 id_gen;/* used to generate unique reg IDs */ u32 hidden_subprog_cnt;/* number of hidden subprogs */ int exception_callback_subprog;
include/uapi/linux/bpf.h +21 -0
@@ -1026,6 +1026,7 @@ enum bpf_map_type {BPF_MAP_TYPE_USER_RINGBUF ,BPF_MAP_TYPE_CGRP_STORAGE ,BPF_MAP_TYPE_ARENA , +BPF_MAP_TYPE_INSN_ARRAY , __MAX_BPF_MAP_TYPE };
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags { BPF_F_PAD_ZEROS = (1ULL << 0), }; +/* +* Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type. +* +* Before the map is used the orig_off field should point to an +* instruction inside the program being loaded. The other fields +* must be set to 0. +* +* After the program is loaded, the xlated_off will be adjusted +* by the verifier to point to the index of the original instruction +* in the xlated program. If the instruction is deleted, it will +* be set to (u32)-1. The jitted_off will be set to the corresponding +* offset in the jitted image of the program. +*/ +struct bpf_insn_array_value { +__u32 orig_off; +__u32 xlated_off; +__u32 jitted_off; + __u32 :32; +}; +#endif /* _UAPI__LINUX_BPF_H__ */
kernel/bpf/Makefile +1 -1
@@ -9,7 +9,7 @@ CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o -obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o +obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
kernel/bpf/bpf_insn_array.c new file +288 -0
288 lines not on this page · raw message
kernel/bpf/syscall.c +22 -0
@@ -1493,6 +1493,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)case BPF_MAP_TYPE_STRUCT_OPS :case BPF_MAP_TYPE_CPUMAP :case BPF_MAP_TYPE_ARENA : +case BPF_MAP_TYPE_INSN_ARRAY :if (!bpf_token_capable (token,CAP_BPF )) goto put_token;break ;
@@ -2853,6 +2854,23 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attrreturn err; } +static int bpf_prog_mark_insn_arrays_ready (struct bpf_prog *prog) +{ +int err; +int i; + +for (i =0 ; i < prog->aux->used_map_cnt; i++) { +if (prog->aux->used_maps[i]->map_type !=BPF_MAP_TYPE_INSN_ARRAY ) +continue ; + + err =bpf_insn_array_ready (prog->aux->used_maps[i]); +if (err) +return err; + } + +return 0 ; +} +/* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD keyring_id
@@ -3082,6 +3100,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)if (err <0 ) goto free_used_maps; + err =bpf_prog_mark_insn_arrays_ready (prog); +if (err <0 ) + goto free_used_maps; + err =bpf_prog_alloc_id (prog);if (err) goto free_used_maps;
kernel/bpf/verifier.c +43 -0
79 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +21 -0
@@ -1026,6 +1026,7 @@ enum bpf_map_type {BPF_MAP_TYPE_USER_RINGBUF ,BPF_MAP_TYPE_CGRP_STORAGE ,BPF_MAP_TYPE_ARENA , +BPF_MAP_TYPE_INSN_ARRAY , __MAX_BPF_MAP_TYPE };
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags { BPF_F_PAD_ZEROS = (1ULL << 0), }; +/* +* Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type. +* +* Before the map is used the orig_off field should point to an +* instruction inside the program being loaded. The other fields +* must be set to 0. +* +* After the program is loaded, the xlated_off will be adjusted +* by the verifier to point to the index of the original instruction +* in the xlated program. If the instruction is deleted, it will +* be set to (u32)-1. The jitted_off will be set to the corresponding +* offset in the jitted image of the program. +*/ +struct bpf_insn_array_value { +__u32 orig_off; +__u32 xlated_off; +__u32 jitted_off; + __u32 :32; +}; +#endif /* _UAPI__LINUX_BPF_H__ */
Signature · 1 line
2.34.1
Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array
Quoted 23 lines · Anton Protopopov
arch/x86/net/bpf_jit_comp.c +8
-0@@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +u32 abs_xlated_off = bpf_prog->aux->subprog_start + i -1 ;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;@@ -2751,6 +2752,13 @@ st: if (is_imm8(insn->off))return -EFAULT ; }memcpy (rw_image + proglen, temp, ilen); + +/* +* Instruction arrays need to know how xlated code +* maps to jitted code +*/ +bpf_prog_update_insn_ptr (bpf_prog, abs_xlated_off, proglen, + image + proglen);
...
Quoted 14 lines
arch/x86/net/bpf_jit_comp.c +14
-0+void bpf_prog_update_insn_ptr (struct bpf_prog *prog, +u32 xlated_off, +u32 jitted_off, +void *jitted_ip) +{ +struct bpf_insn_array *insn_array; +struct bpf_map *map; +int i, j; + +for (i =0 ; i < prog->aux->used_map_cnt; i++) { + map = prog->aux->used_maps[i]; +if (!is_insn_array (map)) +continue ; +
arch/x86/net/bpf_jit_comp.c
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ insn_array =cast_insn_array (map); +for (j =0 ; j < map->max_entries; j++) { +if (insn_array->ptrs[j].user_value.xlated_off == xlated_off) { + insn_array->ips[j] = (long )jitted_ip; + insn_array->ptrs[j].jitted_ip = jitted_ip; + insn_array->ptrs[j].user_value.jitted_off = jitted_off; + } + } + } +}
This algorithm doesn't scale.
You're calling bpf_prog_update_insn_ptr() for every insn
and doing it as many times as they're JIT passes.
There could be up to 20 passes and hundreds of thousands of insns. x86 JIT already keeps the mapping from insn to IP in jit_datat->addrs[]. Use it. Roughly like:insn_array = cast_insn_array(map);
for (j = 0; j < map->max_entries; j++) {
ip = addrs[insn_array->ptrs[j].user_value.xlated_off -subprog_start] + image;
And this is done once per insn_array after JIT is done.
Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array
Quoted 60 lines · Alexei Starovoitov
Quoted 23 lines · Anton Protopopov
arch/x86/net/bpf_jit_comp.c +8
-0@@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +u32 abs_xlated_off = bpf_prog->aux->subprog_start + i -1 ;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;@@ -2751,6 +2752,13 @@ st: if (is_imm8(insn->off))return -EFAULT ; }memcpy (rw_image + proglen, temp, ilen); + +/* +* Instruction arrays need to know how xlated code +* maps to jitted code +*/ +bpf_prog_update_insn_ptr (bpf_prog, abs_xlated_off, proglen, + image + proglen);...
Quoted 14 lines
arch/x86/net/bpf_jit_comp.c +14
-0+void bpf_prog_update_insn_ptr (struct bpf_prog *prog, +u32 xlated_off, +u32 jitted_off, +void *jitted_ip) +{ +struct bpf_insn_array *insn_array; +struct bpf_map *map; +int i, j; + +for (i =0 ; i < prog->aux->used_map_cnt; i++) { + map = prog->aux->used_maps[i]; +if (!is_insn_array (map)) +continue ; +arch/x86/net/bpf_jit_comp.c
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ insn_array =cast_insn_array (map); +for (j =0 ; j < map->max_entries; j++) { +if (insn_array->ptrs[j].user_value.xlated_off == xlated_off) { + insn_array->ips[j] = (long )jitted_ip; + insn_array->ptrs[j].jitted_ip = jitted_ip; + insn_array->ptrs[j].user_value.jitted_off = jitted_off; + } + } + } +}This algorithm doesn't scale.
You're callingbpf_prog_update_insn_ptr()for every insn
and doing it as many times as they're JIT passes.
There could be up to 20 passes and hundreds of thousands of insns. x86 JIT already keeps the mapping from insn to IP in jit_datat->addrs[]. Use it. Roughly like:insn_array=cast_insn_array(map);
Quoted 4 lines
for (j = 0; j < map->max_entries; j++) { ip = addrs[insn_array->ptrs[j].user_value.xlated_off -
subprog_start] + image;
And this is done once perinsn_arrayafter JIT is done.
Thanks! Will do.
Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array
In reply to Anton ProtopopovOn Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 5 lines
The functionality provided by this patch will be extended in consequent patches to implement BPF Static Keys, indirect jumps, and indirect calls.
Signed-off-by Anton Protopopov <[email protected]>---
Aside from what Alexei pointed out, I only have a nit regarding jitted_ip.
[...]
Quoted 6 lines
include/linux/bpf.h +0
-0@@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * const char **linep, int *nump);
[...]
include/linux/bpf.h int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * · line 3809
Quoted 12 lines
include/linux/bpf.h +12
-0+/* +* The struct bpf_insn_ptr structure describes a pointer to a +* particular instruction in a loaded BPF program. Initially +* it is initialised from userspace via user_value.xlated_off. +* During the program verification all other fields are populated +* accordingly: +* +* jitted_ip: address of the instruction in the jitted image +* user_value: user-visible original, xlated, and jitted offsets +*/ +struct bpf_insn_ptr { +void *jitted_ip;
I think this one is no longer used anywhere.
I see it set in bpf_prog_update_insn_ptr() but it is not read anywhere.
Quoted 3 lines
include/linux/bpf.h +3
-0+struct bpf_insn_array_value user_value; +}; +
[...]
Quoted 2 lines
include/uapi/linux/bpf.h +0
-0
[...]
Quoted 12 lines
include/uapi/linux/bpf.h +8
-0@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags { BPF_F_PAD_ZEROS = (1ULL << 0), }; +/* +* Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type. +* +* Before the map is used the orig_off field should point to an +* instruction inside the program being loaded. The other fields +* must be set to 0. +* +* After the program is loaded, the xlated_off will be adjusted
include/uapi/linux/bpf.h enum bpf_kfunc_flags { · line 7666
Quoted 10 lines
include/uapi/linux/bpf.h +10
-0+* by the verifier to point to the index of the original instruction +* in the xlated program. If the instruction is deleted, it will +* be set to (u32)-1. The jitted_off will be set to the corresponding +* offset in the jitted image of the program. +*/ +struct bpf_insn_array_value { +__u32 orig_off; +__u32 xlated_off; +__u32 jitted_off; + __u32 :32;
This :u32, is it for alignment or for future extensibility?
Quoted 2 lines
include/uapi/linux/bpf.h +2
-0+}; +
[...]
Quoted 2 lines
kernel/bpf/bpf_insn_array.c new file +0
-0
[...]
Quoted 14 lines
kernel/bpf/bpf_insn_array.c new file +13
-0@@ -0,0 +1,288 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2025 Isovalent */ + +#include <linux/bpf.h> + +#define MAX_INSN_ARRAY_ENTRIES 256 + +struct bpf_insn_array { +struct bpf_map map; +atomic_t used; +long *ips; + DECLARE_FLEX_ARRAY(struct bpf_insn_ptr, ptrs); +};
[...]
kernel/bpf/bpf_insn_array.c line 26
Quoted 7 lines
kernel/bpf/bpf_insn_array.c new file +7
-0+static inline u32 insn_array_alloc_size (u32 max_entries) +{ +const u32 base_size =sizeof (struct bpf_insn_array ); +const u32 entry_size =sizeof (struct bpf_insn_ptr ); + +return base_size + entry_size * max_entries; +}
Since this is doing a flexible array thing anyway, maybe also include size for ips here? And in insn_array_alloc() point it at the tail area.
[...]
Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array
Quoted 25 lines · Eduard Zingerman
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 5 lines
The functionality provided by this patch will be extended in consequent patches to implement BPF Static Keys, indirect jumps, and indirect calls.
Signed-off-by Anton Protopopov <[email protected]>---
Aside from what Alexei pointed out, I only have a nit regarding
jitted_ip.Reviewed-by Eduard Zingerman <[email protected]>[...]
Quoted 6 lines
include/linux/bpf.h +0
-0@@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * const char **linep, int *nump);[...]
Quoted 15 lines
include/linux/bpf.h int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * · line 3809
Quoted 12 lines
include/linux/bpf.h +12
-0+/* +* The struct bpf_insn_ptr structure describes a pointer to a +* particular instruction in a loaded BPF program. Initially +* it is initialised from userspace via user_value.xlated_off. +* During the program verification all other fields are populated +* accordingly: +* +* jitted_ip: address of the instruction in the jitted image +* user_value: user-visible original, xlated, and jitted offsets +*/ +struct bpf_insn_ptr { +void *jitted_ip;I think this one is no longer used anywhere.
I see it set inbpf_prog_update_insn_ptr()but it is not read anywhere.
Nice, thanks :)
Quoted 27 lines
Quoted 3 lines
include/linux/bpf.h +3
-0+struct bpf_insn_array_value user_value; +}; +[...]
Quoted 2 lines
include/uapi/linux/bpf.h +0
-0[...]
Quoted 16 lines
include/uapi/linux/bpf.h +12
-0@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags { BPF_F_PAD_ZEROS = (1ULL << 0), }; +/* +* Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type. +* +* Before the map is used the orig_off field should point to an +* instruction inside the program being loaded. The other fields +* must be set to 0. +* +* After the program is loaded, the xlated_off will be adjusted +* by the verifier to point to the index of the original instruction +* in the xlated program. If the instruction is deleted, it will +* be set to (u32)-1. The jitted_off will be set to the corresponding +* offset in the jitted image of the program.
Quoted 8 lines
include/uapi/linux/bpf.h enum bpf_kfunc_flags { · line 7666
Quoted 6 lines
include/uapi/linux/bpf.h +6
-0+*/ +struct bpf_insn_array_value { +__u32 orig_off; +__u32 xlated_off; +__u32 jitted_off; + __u32 :32;This :u32, is it for alignment or for future extensibility?
Both. We maybe might have some flags added in future for different use cases...
Quoted 26 lines
Quoted 1 line
include/uapi/linux/bpf.h +1
-0+[...]
Quoted 2 lines
kernel/bpf/bpf_insn_array.c new file +0
-0[...]
Quoted 14 lines
kernel/bpf/bpf_insn_array.c new file +13
-0@@ -0,0 +1,288 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2025 Isovalent */ + +#include <linux/bpf.h> + +#define MAX_INSN_ARRAY_ENTRIES 256 + +struct bpf_insn_array { +struct bpf_map map; +atomic_t used; +long *ips; + DECLARE_FLEX_ARRAY(struct bpf_insn_ptr, ptrs); +};[...]
Quoted 11 lines
kernel/bpf/bpf_insn_array.c line 26
Quoted 7 lines
kernel/bpf/bpf_insn_array.c new file +7
-0+static inline u32 insn_array_alloc_size (u32 max_entries) +{ +const u32 base_size =sizeof (struct bpf_insn_array ); +const u32 entry_size =sizeof (struct bpf_insn_ptr ); + +return base_size + entry_size * max_entries; +}Since this is doing a flexible array thing anyway, maybe also include size for
ipshere? And ininsn_array_alloc()point it at the tail area.
Yes, great idea.
Quoted 1 line
[...]
PATCH v6 5/17 selftests/bpf: add selftests for new insn_array map
In reply to Anton ProtopopovAdd the following selftests for new insn_array map:
* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are present
--- .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +404 -0
@@ -0,0 +1,404 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <bpf/bpf.h> +#include <test_progs.h> + +#ifdef __x86_64__ +static int map_create (__u32 map_type,__u32 max_entries) +{ +const char *map_name ="insn_array" ; +__u32 key_size =4 ; +__u32 value_size =sizeof (struct bpf_insn_array_value ); + +return bpf_map_create (map_type, map_name, key_size, value_size, max_entries,NULL ); +} + +static int prog_load (struct bpf_insn *insns,__u32 insn_cnt,int *fd_array,__u32 fd_array_cnt) +{ +LIBBPF_OPTS (bpf_prog_load_opts, opts); + + opts.fd_array = fd_array; + opts.fd_array_cnt = fd_array_cnt; + +return bpf_prog_load (BPF_PROG_TYPE_XDP ,NULL ,"GPL" , insns, insn_cnt, &opts); +} + +static void __check_success (struct bpf_insn *insns,__u32 insn_cnt,__u32 *map_in,__u32 *map_out) +{ +struct bpf_insn_array_value val = {}; +int prog_fd =-1 , map_fd, i; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY , insn_cnt); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + +for (i =0 ; i < insn_cnt; i++) { + val.orig_off = map_in[i]; +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &i, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + } + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns, insn_cnt, &map_fd,1 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +for (i =0 ; i < insn_cnt; i++) { +char buf[64 ]; + +if (!ASSERT_EQ (bpf_map_lookup_elem (map_fd, &i, &val),0 ,"bpf_map_lookup_elem" )) + goto cleanup; + +snprintf (buf,sizeof (buf),"val.xlated_off should be equal map_out[%d]" , i); +ASSERT_EQ (val.xlated_off, map_out[i], buf); + } + +cleanup: +close (prog_fd); +close (map_fd); +} + +/* +* Load a program, which will not be anyhow mangled by the verifier. Add an +* insn_array map pointing to every instruction. Check that it hasn't changed +* after the program load. +*/ +static void check_one_to_one_mapping (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +__u32 map_in[] = {0 ,1 ,2 ,3 ,4 ,5 }; +__u32 map_out[] = {0 ,1 ,2 ,3 ,4 ,5 }; + +__check_success (insns,ARRAY_SIZE (insns), map_in, map_out); +} + +/* +* Load a program with two patches (get jiffies, for simplicity). Add an +* insn_array map pointing to every instruction. Check how it was changed +* after the program load. +*/ +static void check_simple (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_jiffies64), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_jiffies64), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +__u32 map_in[] = {0 ,1 ,2 ,3 ,4 ,5 }; +__u32 map_out[] = {0 ,1 ,4 ,5 ,8 ,9 }; + +__check_success (insns,ARRAY_SIZE (insns), map_in, map_out); +} + +/* +* Verifier can delete code in two cases: nops & dead code. From insn +* array's point of view, the two cases are the same, so test using +* the simplest method: by loading some nops +*/ +static void check_deletions (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +__u32 map_in[] = {0 ,1 ,2 ,3 ,4 ,5 }; +__u32 map_out[] = {0 ,-1 ,1 ,-1 ,2 ,3 }; + +__check_success (insns,ARRAY_SIZE (insns), map_in, map_out); +} + +/* +* Same test as check_deletions, but also add code which adds instructions +*/ +static void check_deletions_with_functions (void ) +{ +struct bpf_insn insns[] = { +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_jiffies64), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_jiffies64), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ),/* nop */ +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_EXIT_INSN (), + }; +__u32 map_in[] = {0 ,1 ,2 ,3 ,4 ,5 ,/* func */ 6 ,7 ,8 ,9 ,10 }; +__u32 map_out[] = {-1 ,0 ,-1 ,3 ,4 ,5 ,/* func */ -1 ,6 ,-1 ,9 ,10 }; + +__check_success (insns,ARRAY_SIZE (insns), map_in, map_out); +} + +/* +* Try to load a program with a map which points to outside of the program +*/ +static void check_out_of_bounds_index (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +int prog_fd, map_fd; +struct bpf_insn_array_value val = {}; +int key; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + key =0 ; + val.orig_off =ARRAY_SIZE (insns);/* too big */ +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &key, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) { +close (prog_fd); + goto cleanup; + } + +cleanup: +close (map_fd); +} + +/* +* Try to load a program with a map which points to the middle of 16-bit insn +*/ +static void check_mid_insn_index (void ) +{ +struct bpf_insn insns[] = { +BPF_LD_IMM64 (BPF_REG_0 ,0 ),/* 2 x 8 */ +BPF_EXIT_INSN (), + }; +int prog_fd, map_fd; +struct bpf_insn_array_value val = {}; +int key; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + key =0 ; + val.orig_off =1 ;/* middle of 16-byte instruction */ +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &key, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) { +close (prog_fd); + goto cleanup; + } + +cleanup: +close (map_fd); +} + +static void check_incorrect_index (void ) +{ +check_out_of_bounds_index (); +check_mid_insn_index (); +} + +/* Once map was initialized, it should be frozen */ +static void check_load_unfrozen_map (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; +struct bpf_insn_array_value val = {}; +int i; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,ARRAY_SIZE (insns)); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { + val.orig_off = i; +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &i, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + } + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: now freeze the map, the program should load fine */ + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { +if (!ASSERT_EQ (bpf_map_lookup_elem (map_fd, &i, &val),0 ,"bpf_map_lookup_elem" )) + goto cleanup; + +ASSERT_EQ (val.xlated_off, i,"val should be equal i" ); + } + +cleanup: +close (prog_fd); +close (map_fd); +} + +/* Map can be used only by one BPF program */ +static void check_no_map_reuse (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd, extra_fd =-1 ; +struct bpf_insn_array_value val = {}; +int i; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,ARRAY_SIZE (insns)); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { + val.orig_off = i; +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &i, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + } + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { +if (!ASSERT_EQ (bpf_map_lookup_elem (map_fd, &i, &val),0 ,"bpf_map_lookup_elem" )) + goto cleanup; + +ASSERT_EQ (val.xlated_off, i,"val should be equal i" ); + } + + extra_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_EQ (extra_fd, -EBUSY ,"program should have been rejected (extra_fd != -EBUSY)" )) + goto cleanup; + +/* correctness: check that prog is still loadable without fd_array */ + extra_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD): expected no error" )) + goto cleanup; + +cleanup: +close (extra_fd); +close (prog_fd); +close (map_fd); +} + +static void check_bpf_no_lookup (void ) +{ +struct bpf_insn insns[] = { +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + insns[0 ].imm = map_fd; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: check that prog is still loadable with normal map */ +close (map_fd); + map_fd =map_create (BPF_MAP_TYPE_ARRAY ,1 ); + insns[0 ].imm = map_fd; + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +cleanup: +close (prog_fd); +close (map_fd); +} + +static void check_bpf_side (void ) +{ +check_bpf_no_lookup (); +} + +void test_bpf_insn_array (void ) +{ +/* Test if offsets are adjusted properly */ + +if (test__start_subtest ("one2one" )) +check_one_to_one_mapping (); + +if (test__start_subtest ("simple" )) +check_simple (); + +if (test__start_subtest ("deletions" )) +check_deletions (); + +if (test__start_subtest ("deletions-with-functions" )) +check_deletions_with_functions (); + +/* Check all kinds of operations and related restrictions */ + +if (test__start_subtest ("incorrect-index" )) +check_incorrect_index (); + +if (test__start_subtest ("load-unfrozen-map" )) +check_load_unfrozen_map (); + +if (test__start_subtest ("no-map-reuse" )) +check_no_map_reuse (); + +if (test__start_subtest ("bpf-side-ops" )) +check_bpf_side (); +} +#else +void test_bpf_insn_array (void ) +{ + +} +#endif
Signature · 1 line
2.34.1
Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
Quoted 10 lines · Anton Protopopov
Add the following selftests for new
insn_arraymap:* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are present
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Quoted 9 lines
.../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.ctools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0
[...]
Quoted 1 line
+static void
check_bpf_no_lookup(void)
This one can be moved to prog_tests/bpf_insn_array.c, I think.
Quoted 36 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +36
-0+{ +struct bpf_insn insns[] = { +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + insns[0 ].imm = map_fd; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: check that prog is still loadable with normal map */ +close (map_fd); + map_fd =map_create (BPF_MAP_TYPE_ARRAY ,1 ); + insns[0 ].imm = map_fd; + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +cleanup: +close (prog_fd); +close (map_fd); +}
[...]
Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
Quoted 28 lines · Eduard Zingerman
Quoted 10 lines · Anton Protopopov
Add the following selftests for new
insn_arraymap:* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are presentQuoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>Quoted 9 lines
.../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.ctools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 3 lines
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.
A typo? (This is a patch for the prog_tests/bpf_insn_array.c)
Quoted 38 lines
Quoted 36 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +36
-0+{ +struct bpf_insn insns[] = { +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + insns[0 ].imm = map_fd; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: check that prog is still loadable with normal map */ +close (map_fd); + map_fd =map_create (BPF_MAP_TYPE_ARRAY ,1 ); + insns[0 ].imm = map_fd; + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +cleanup: +close (prog_fd); +close (map_fd); +}[...]
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
Quoted 29 lines · Anton Protopopov
Quoted 28 lines · Eduard Zingerman
Quoted 10 lines · Anton Protopopov
Add the following selftests for new
insn_arraymap:* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are presentQuoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>Quoted 9 lines
.../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.ctools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.A typo? (This is a patch for the
prog_tests/bpf_insn_array.c)
Yes, I mean progs/verifier_gotox.c, the one with inline assembly.
Quoted 39 lines
Quoted 39 lines
Quoted 37 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +36
-0+{ +struct bpf_insn insns[] = { +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + insns[0 ].imm = map_fd; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: check that prog is still loadable with normal map */ +close (map_fd); + map_fd =map_create (BPF_MAP_TYPE_ARRAY ,1 ); + insns[0 ].imm = map_fd; + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +cleanup: +close (prog_fd); +close (map_fd); +}[...]
Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
Quoted 30 lines · Eduard Zingerman
Quoted 29 lines · Anton Protopopov
Quoted 28 lines · Eduard Zingerman
Quoted 10 lines · Anton Protopopov
Add the following selftests for new
insn_arraymap:* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are presentQuoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>Quoted 9 lines
.../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++ 1 file changed, 404 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.ctools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.A typo? (This is a patch for the
prog_tests/bpf_insn_array.c)Yes, I mean
progs/verifier_gotox.c, the one with inline assembly.
I think it should stay here. There will be other usages of the instruction array, and neither should allow operations on it from a BPF prog (indirect calls, static keys).
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 37 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +36
-0+{ +struct bpf_insn insns[] = { +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_map_lookup_elem), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,1 ); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + + insns[0 ].imm = map_fd; + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_EQ (prog_fd, -EINVAL ,"program should have been rejected (prog_fd != -EINVAL)" )) + goto cleanup; + +/* correctness: check that prog is still loadable with normal map */ +close (map_fd); + map_fd =map_create (BPF_MAP_TYPE_ARRAY ,1 ); + insns[0 ].imm = map_fd; + prog_fd =prog_load (insns,ARRAY_SIZE (insns),NULL ,0 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +cleanup: +close (prog_fd); +close (map_fd); +}[...]
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:
[...]
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
A typo? (This is a patch for the
prog_tests/bpf_insn_array.c)Yes, I mean
progs/verifier_gotox.c, the one with inline assembly.I think it should stay here. There will be other usages of the instruction array, and neither should allow operations on it from a BPF prog (indirect calls, static keys).
It will be functionally identical and like 3x-4x time shorter in that file.
Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
Quoted 18 lines · Eduard Zingerman
On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:
[...]
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 11 lines
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.A typo? (This is a patch for the
prog_tests/bpf_insn_array.c)
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, I mean
progs/verifier_gotox.c, the one with inline assembly.I think it should stay here. There will be other usages of the instruction array, and neither should allow operations on it from a BPF prog (indirect calls, static keys).
It will be functionally identical and like 3x-4x time shorter in that file.
Ok, I see. I will either move it there, or add another verifier_insn_array (if I see how to convert/create more "insn array level" tests like this).
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map
In reply to Eduard ZingermanQuoted 18 lines · Eduard Zingerman
On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:
[...]
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 11 lines
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c new file +0
-0[...]
Quoted 1 line
+static void
check_bpf_no_lookup(void)This one can be moved to
prog_tests/bpf_insn_array.c, I think.A typo? (This is a patch for the
prog_tests/bpf_insn_array.c)
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, I mean
progs/verifier_gotox.c, the one with inline assembly.I think it should stay here. There will be other usages of the instruction array, and neither should allow operations on it from a BPF prog (indirect calls, static keys).
It will be functionally identical and like 3x-4x time shorter in that file.
Wait. I'm being stupid again. There is no simple way to get an FD for the iarray map when writing a bpf C. So your current implementation makes most sense. Please disregard this nitpick, the patch is good as it is.
PATCH v6 6/17 bpf: support instructions arrays with constants blinding
In reply to Anton ProtopopovWhen bpf_jit_harden is enabled, all constants in the BPF code are blinded to prevent JIT spraying attacks. This happens during JIT phase. Adjust all the related instruction arrays accordingly.
--- kernel/bpf/core.c | 20 ++++++++++++++++++++ kernel/bpf/verifier.c | 11 ++++++++++- 2 files changed, 30 insertions(+), 1 deletion(-)
kernel/bpf/core.c +20 -0
@@ -1450,6 +1450,23 @@ void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other)bpf_prog_clone_free (fp_other); } +static void adjust_insn_arrays (struct bpf_prog *prog,u32 off,u32 len) +{ +#ifdef CONFIG_BPF_SYSCALL +struct bpf_map *map; +int i; + +if (len <=1 ) +return ; + +for (i =0 ; i < prog->aux->used_map_cnt; i++) { + map = prog->aux->used_maps[i]; +if (map->map_type ==BPF_MAP_TYPE_INSN_ARRAY ) +bpf_insn_array_adjust (map, off, len); + } +#endif +} +struct bpf_prog *bpf_jit_blind_constants (struct bpf_prog *prog) {struct bpf_insn insn_buff[16 ], aux[2 ];
@@ -1505,6 +1522,9 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) clone = tmp; insn_delta = rewritten -1 ; +/* Instructions arrays must be updated using absolute xlated offsets */ +adjust_insn_arrays (clone, prog->aux->subprog_start + i, rewritten); +/* Walk new program and skip insns we just inserted. */ insn = clone->insnsi + i + insn_delta; insn_cnt += insn_delta;
kernel/bpf/verifier.c +10 -1
@@ -21574,6 +21574,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)struct bpf_insn *insn;void *old_bpf_func;int err, num_exentries; +int old_len, subprog_start_adjustment =0 ;if (env->subprog_cnt <=1 )return 0 ;
@@ -21648,7 +21649,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->func_idx = i;/* Below members will be freed only at prog->aux */ func[i]->aux->btf = prog->aux->btf; - func[i]->aux->subprog_start = subprog_start; + func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment; func[i]->aux->func_info = prog->aux->func_info; func[i]->aux->func_info_cnt = prog->aux->func_info_cnt; func[i]->aux->poke_tab = prog->aux->poke_tab;
@@ -21702,7 +21703,15 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;if (!i) func[i]->aux->exception_boundary = env->seen_exception; + +/* +* To properly pass the absolute subprog start to jit +* all instruction adjustments should be accumulated +*/ + old_len = func[i]->len; func[i] =bpf_int_jit_compile (func[i]); + subprog_start_adjustment += func[i]->len - old_len; +if (!func[i]->jited) { err = -ENOTSUPP ; goto out_free;
Signature · 1 line
2.34.1
PATCH v6 7/17 selftests/bpf: test instructions arrays with blinding
In reply to Anton ProtopopovAdd a specific test for instructions arrays with blinding enabled.
--- .../selftests/bpf/prog_tests/bpf_insn_array.c | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+)
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c +95 -0
@@ -227,6 +227,98 @@ static void check_incorrect_index(void)check_mid_insn_index (); } +static int set_bpf_jit_harden (char *level) +{ +char old_level; +int err =-1 ; +int fd =-1 ; + + fd =open ("/proc/sys/net/core/bpf_jit_harden" ,O_RDWR |O_NONBLOCK ); +if (fd <0 ) { +ASSERT_FAIL ("open .../bpf_jit_harden returned %d (errno=%d)" , fd, errno); +return -1 ; + } + + err =read (fd, &old_level,1 ); +if (err !=1 ) { +ASSERT_FAIL ("read from .../bpf_jit_harden returned %d (errno=%d)" , err, errno); + err =-1 ; + goto end; + } + +lseek (fd,0 ,SEEK_SET ); + + err =write (fd, level,1 ); +if (err !=1 ) { +ASSERT_FAIL ("write to .../bpf_jit_harden returned %d (errno=%d)" , err, errno); + err =-1 ; + goto end; + } + + err =0 ; + *level = old_level; +end: +if (fd >=0 ) +close (fd); +return err; +} + +static void check_blindness (void ) +{ +struct bpf_insn insns[] = { +BPF_MOV64_IMM (BPF_REG_0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), + }; +int prog_fd =-1 , map_fd; +struct bpf_insn_array_value val = {}; +char bpf_jit_harden ='@' ;/* non-exizsting value */ +int i; + + map_fd =map_create (BPF_MAP_TYPE_INSN_ARRAY ,ARRAY_SIZE (insns)); +if (!ASSERT_GE (map_fd,0 ,"map_create" )) +return ; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { + val.orig_off = i; +if (!ASSERT_EQ (bpf_map_update_elem (map_fd, &i, &val,0 ),0 ,"bpf_map_update_elem" )) + goto cleanup; + } + +if (!ASSERT_EQ (bpf_map_freeze (map_fd),0 ,"bpf_map_freeze" )) + goto cleanup; + + bpf_jit_harden ='2' ; +if (set_bpf_jit_harden (&bpf_jit_harden)) { + bpf_jit_harden ='@' ;/* open, read or write failed => no write was done */ + goto cleanup; + } + + prog_fd =prog_load (insns,ARRAY_SIZE (insns), &map_fd,1 ); +if (!ASSERT_GE (prog_fd,0 ,"bpf(BPF_PROG_LOAD)" )) + goto cleanup; + +for (i =0 ; i <ARRAY_SIZE (insns); i++) { +char fmt[32 ]; + +if (!ASSERT_EQ (bpf_map_lookup_elem (map_fd, &i, &val),0 ,"bpf_map_lookup_elem" )) + goto cleanup; + +snprintf (fmt,sizeof (fmt),"val should be equal 3*%d" , i); +ASSERT_EQ (val.xlated_off, i *3 , fmt); + } + +cleanup: +/* restore the old one */ +if (bpf_jit_harden !='@' ) +set_bpf_jit_harden (&bpf_jit_harden); + +close (prog_fd); +close (map_fd); +} +/* Once map was initialized, it should be frozen */ static void check_load_unfrozen_map (void ) {
@@ -382,6 +474,9 @@ void test_bpf_insn_array(void)if (test__start_subtest ("deletions-with-functions" ))check_deletions_with_functions (); +if (test__start_subtest ("blindness" )) +check_blindness (); +/* Check all kinds of operations and related restrictions */ if (test__start_subtest ("incorrect-index" ))
Signature · 1 line
2.34.1
PATCH v6 8/17 bpf, x86: allow indirect jumps to r8...r15
In reply to Anton ProtopopovCurrently the emit_indirect_jump() function only accepts one of the RAX, RCX, ..., RBP registers as the destination. Make it to accept R8, R9, ..., R15 as well, and make callers to pass BPF registers, not native registers. This is required to enable indirect jumps support in eBPF.
--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +21 -7
@@ -660,24 +660,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,#define EMIT_LFENCE () EMIT3(0x0F, 0xAE, 0xE8) -static void emit_indirect_jump (u8 **pprog,int reg,u8 *ip) +static void __emit_indirect_jump (u8 **pprog,int reg,bool ereg) {u8 *prog = *pprog; +if (ereg) +EMIT1 (0x41 ); + +EMIT2 (0xFF ,0xE0 + reg); + + *pprog = prog; +} + +static void emit_indirect_jump (u8 **pprog,int bpf_reg,u8 *ip) +{ +u8 *prog = *pprog; +int reg = reg2hex[bpf_reg]; +bool ereg =is_ereg (bpf_reg); +if (cpu_feature_enabled (X86_FEATURE_INDIRECT_THUNK_ITS )) {OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog,its_static_thunk (reg), ip); }else if (cpu_feature_enabled (X86_FEATURE_RETPOLINE_LFENCE )) {EMIT_LFENCE (); -EMIT2 (0xFF ,0xE0 + reg); +__emit_indirect_jump (&prog, reg, ereg); }else if (cpu_feature_enabled (X86_FEATURE_RETPOLINE )) {OPTIMIZER_HIDE_VAR (reg);if (cpu_feature_enabled (X86_FEATURE_CALL_DEPTH )) -emit_jump (&prog, &__x86_indirect_jump_thunk_array[reg], ip); +emit_jump (&prog, &__x86_indirect_jump_thunk_array[reg +8 *ereg], ip);else -emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); +emit_jump (&prog, &__x86_indirect_thunk_array[reg +8 *ereg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +__emit_indirect_jump (&prog, reg, ereg);if (IS_ENABLED (CONFIG_MITIGATION_RETPOLINE ) ||IS_ENABLED (CONFIG_MITIGATION_SLS ))EMIT1 (0xCC );/* int3 */ }
@@ -797,7 +811,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,* rdi == ctx (1st arg) * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET */ -emit_indirect_jump (&prog,1 /* rcx */ , ip + (prog - start)); +emit_indirect_jump (&prog,BPF_REG_4 /* R4 -> rcx */ , ip + (prog - start));/* out: */ ctx->tail_call_indirect_label = prog - start;
@@ -3551,7 +3565,7 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,if (err)return err; -emit_indirect_jump (&prog,2 /* rdx */ , image + (prog - buf)); +emit_indirect_jump (&prog,BPF_REG_3 /* R3 -> rdx */ , image + (prog - buf)); *pprog = prog;return 0 ;
Signature · 1 line
2.34.1
Re: [PATCH v6 bpf-next 08/17] bpf, x86: allow indirect jumps to r8...r15
Quoted 32 lines · Anton Protopopov
Currently the
emit_indirect_jump()function only accepts one of the RAX, RCX, ..., RBP registers as the destination. Make it to accept R8, R9, ..., R15 as well, and make callers to pass BPF registers, not native registers. This is required to enable indirect jumps support in eBPF.Signed-off-by Anton Protopopov <[email protected]>Acked-by Eduard Zingerman <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)arch/x86/net/bpf_jit_comp.c +8
-1@@ -660,24 +660,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,#define EMIT_LFENCE () EMIT3(0x0F, 0xAE, 0xE8) -static void emit_indirect_jump (u8 **pprog,int reg,u8 *ip) +static void __emit_indirect_jump (u8 **pprog,int reg,bool ereg) {u8 *prog = *pprog; +if (ereg) +EMIT1 (0x41 ); + +EMIT2 (0xFF ,0xE0 + reg); + + *pprog = prog; +}
arch/x86/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, · line 683
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +7
-0+ +static void emit_indirect_jump (u8 **pprog,int bpf_reg,u8 *ip) +{ +u8 *prog = *pprog; +int reg = reg2hex[bpf_reg]; +bool ereg =is_ereg (bpf_reg); +if (cpu_feature_enabled (X86_FEATURE_INDIRECT_THUNK_ITS )) {OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog,its_static_thunk (reg), ip);
AI found bug here: its_static_thunk(reg) should use reg+8*ereg. (The code here was not changed, however, before this patch this code only was called for eax and ecx.) Will fix in the next version.
Also added verifier_gotox tests which validate that gotox works with r0,...,r9.
Quoted 1 line
[...]
PATCH v6 9/17 bpf: make bpf_insn_successors to return a pointer
In reply to Anton ProtopopovThe bpf_insn_successors() function is used to return successors to a BPF instruction. So far, an instruction could have 0, 1 or 2 successors. Prepare the verifier code to introduction of instructions with more than 2 successors (namely, indirect jumps).
To do this, introduce a new struct, struct bpf_iarray, containing an array of bpf instruction indexes and make bpf_insn_successors to return a pointer of that type. The storage for all instructions is allocated in the env->succ, which holds an array of size 2, to be used for all instructions.
--- include/linux/bpf_verifier.h | 12 +++++++- kernel/bpf/liveness.c | 36 ++++++++++++++-------- kernel/bpf/verifier.c | 60 ++++++++++++++++++++++++------------ 3 files changed, 75 insertions(+), 33 deletions(-)
include/linux/bpf_verifier.h +11 -1
@@ -509,6 +509,15 @@ struct bpf_map_ptr_state {#define BPF_ALU_SANITIZE (BPF_ALU_SANITIZE_SRC | \ BPF_ALU_SANITIZE_DST) +/* +* An array of BPF instructions. +* Primary usage: return value of bpf_insn_successors. +*/ +struct bpf_iarray { +int cnt; +u32 items[]; +}; +struct bpf_insn_aux_data {union {enum bpf_reg_type ptr_type;/* pointer type for load/store insns */
@@ -830,6 +839,7 @@ struct bpf_verifier_env {/* array of pointers to bpf_scc_info indexed by SCC id */ struct bpf_scc_info **scc_info;u32 scc_cnt; +struct bpf_iarray *succ; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
@@ -1052,7 +1062,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_ststruct bpf_subprog_info *bpf_find_containing_subprog (struct bpf_verifier_env *env,int off);int bpf_jmp_offset (struct bpf_insn *insn); -int bpf_insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]); +struct bpf_iarray *bpf_insn_successors (struct bpf_verifier_env *env,u32 idx);void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask);bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx);
kernel/bpf/liveness.c +24 -12
@@ -34,7 +34,7 @@* - read and write marks propagation. * - The propagation phase is a textbook live variable data flow analysis: * -* state[cc, i].live_after = U [state[cc, s].live_before for s in insn_successors(i)] +* state[cc, i].live_after = U [state[cc, s].live_before for s in bpf_insn_successors(i)] * state[cc, i].live_before = * (state[cc, i].live_after / state[cc, i].must_write) U state[i].may_read *
@@ -54,7 +54,7 @@* The equation for "must_write_acc" propagation looks as follows: * * state[cc, i].must_write_acc = -* ∩ [state[cc, s].must_write_acc for s in insn_successors(i)] +* ∩ [state[cc, s].must_write_acc for s in bpf_insn_successors(i)] * U state[cc, i].must_write * * (An intersection of all "must_write_acc" for instruction successors
@@ -447,7 +447,12 @@ int bpf_jmp_offset(struct bpf_insn *insn)__diag_push ();__diag_ignore_all ("-Woverride-init" ,"Allow field initialization overrides for opcode_info_tbl" ); -inline int bpf_insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]) +/* +* Returns an array of instructions succ, with succ->items[0], ..., +* succ->items[n-1] with successor instructions, where n=succ->cnt +*/ +inline struct bpf_iarray * +bpf_insn_successors (struct bpf_verifier_env *env,u32 idx) { static const struct opcode_info { bool can_jump;
@@ -474,19 +479,25 @@ inline int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]) _J(BPF_JSET, {.can_jump = true, .can_fallthrough = true}),#undef _J }; +struct bpf_prog *prog = env->prog;struct bpf_insn *insn = &prog->insnsi[idx];const struct opcode_info *opcode_info; -int i =0 , insn_sz; +struct bpf_iarray *succ; +int insn_sz; + +/* pre-allocated array of size up to 2; reset cnt, as it may have been used already */ + succ = env->succ; + succ->cnt =0 ; opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough) - succ[i++] = idx + insn_sz; + succ->items[succ->cnt++] = idx + insn_sz;if (opcode_info->can_jump) - succ[i++] = idx +bpf_jmp_offset (insn) +1 ; + succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; -return i; +return succ; }__diag_pop ();
@@ -546,11 +557,12 @@ static inline bool update_insn(struct bpf_verifier_env *env,struct bpf_insn_aux_data *aux = env->insn_aux_data;u64 new_before, new_after, must_write_acc;struct per_frame_masks *insn, *succ_insn; -u32 succ_num, s, succ[2 ]; +struct bpf_iarray *succ; +u32 s;bool changed; - succ_num =bpf_insn_successors (env->prog, insn_idx, succ); -if (unlikely (succ_num ==0 )) + succ =bpf_insn_successors (env, insn_idx); +if (succ->cnt ==0 )return false; changed = false;
@@ -562,8 +574,8 @@ static inline bool update_insn(struct bpf_verifier_env *env,* of successors plus all "must_write" slots of instruction itself. */ must_write_acc =U64_MAX ; -for (s =0 ; s < succ_num; ++s) { - succ_insn =get_frame_masks (instance, frame, succ[s]); +for (s =0 ; s < succ->cnt; ++s) { + succ_insn =get_frame_masks (instance, frame, succ->items[s]); new_after |= succ_insn->live_before; must_write_acc &= succ_insn->must_write_acc; }
kernel/bpf/verifier.c +40 -20
122 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v6 10/17 bpf, x86: add support for indirect jumps
In reply to Anton ProtopopovAdd support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
which does an indirect jump to a location stored in Rx. The register Rx should have type PTR_TO_INSN. This new type assures that the Rx register contains a value (or a range of values) loaded from a correct jump table – map of type instruction array.
For example, for a C switch LLVM will generate the following code:
0: r3 = r1 # "switch (r3)"
1: if r3 > 0x13 goto +0x666 # check r3 boundaries
2: r3 <<= 0x3 # adjust to an index in array of addresses
3: r1 = 0xbeef ll # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M
5: r1 += r3 # r1 inherits boundaries from r3
6: r1 = *(u64 *)(r1 + 0x0) # r1 now has type INSN_TO_PTR
7: gotox r1 # jit will generate proper codeHere the gotox instruction corresponds to one particular map. This is possible however to have a gotox instruction which can be loaded from different maps, e.g.
0: r1 &= 0x1
1: r2 <<= 0x3
2: r3 = 0x0 ll # load from map M_1
4: r3 += r2
5: if r1 == 0x0 goto +0x4
6: r1 <<= 0x3
7: r3 = 0x0 ll # load from map M_2
9: r3 += r1
A: r1 = *(u64 *)(r3 + 0x0)
B: gotox r1 # jump to target loaded from M_1 or M_2During check_cfg stage the verifier will collect all the maps which point to inside the subprog being verified. When building the config, the high 16 bytes of the insn_state are used, so this patch (theoretically) supports jump tables of up to 2^16 slots.
During the later stage, in check_indirect_jump, it is checked that the register Rx was loaded from a particular instruction array.
--- arch/x86/net/bpf_jit_comp.c | 3 + include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 9 + kernel/bpf/bpf_insn_array.c | 15 ++ kernel/bpf/core.c | 1 + kernel/bpf/liveness.c | 3 + kernel/bpf/log.c | 1 + kernel/bpf/verifier.c | 363 ++++++++++++++++++++++++++++++++++- 8 files changed, 390 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +3 -0
@@ -2629,6 +2629,9 @@ st: if (is_imm8(insn->off))break ; +case BPF_JMP |BPF_JA |BPF_X : +emit_indirect_jump (&prog, insn->dst_reg, image + addrs[i -1 ]); +break ;case BPF_JMP |BPF_JA :case BPF_JMP32 |BPF_JA :if (BPF_CLASS (insn->code) ==BPF_JMP ) {
include/linux/bpf.h +1 -0
@@ -995,6 +995,7 @@ enum bpf_reg_type {PTR_TO_ARENA ,PTR_TO_BUF ,/* reg points to a read/write buffer */ PTR_TO_FUNC ,/* reg points to a bpf program function */ +PTR_TO_INSN ,/* reg points to a bpf program instruction */ CONST_PTR_TO_DYNPTR ,/* reg points to a const struct bpf_dynptr */ __BPF_REG_TYPE_MAX,
include/linux/bpf_verifier.h +9 -0
@@ -527,6 +527,7 @@ struct bpf_insn_aux_data {struct {u32 map_index;/* index into used_maps[] */ u32 map_off;/* offset from value base address */ +struct bpf_iarray *jt;/* jump table for gotox instruction */ };struct {enum bpf_reg_type reg_type;/* type of pseudo_btf_id */
@@ -840,6 +841,7 @@ struct bpf_verifier_env {struct bpf_scc_info **scc_info;u32 scc_cnt;struct bpf_iarray *succ; +struct bpf_iarray *gotox_tmp_buf; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
@@ -1050,6 +1052,13 @@ static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_return !(off %BPF_REG_SIZE ); } +static inline bool insn_is_gotox (struct bpf_insn *insn) +{ +return BPF_CLASS (insn->code) ==BPF_JMP && +BPF_OP (insn->code) ==BPF_JA && +BPF_SRC (insn->code) ==BPF_X ; +} +const char *reg_type_str (struct bpf_verifier_env *env,enum bpf_reg_type type);const char *dynptr_type_str (enum bpf_dynptr_type type);const char *iter_type_str (const struct btf *btf,u32 btf_id);
kernel/bpf/bpf_insn_array.c +15 -0
@@ -127,6 +127,20 @@ static u64 insn_array_mem_usage(const struct bpf_map *map)return insn_array_alloc_size (map->max_entries) + extra_size; } +static int insn_array_map_direct_value_addr (const struct bpf_map *map,u64 *imm,u32 off) +{ +struct bpf_insn_array *insn_array =cast_insn_array (map); + +if ((off %sizeof (long )) !=0 || + (off /sizeof (long )) >= map->max_entries) +return -EINVAL ; + +/* from BPF's point of view, this map is a jump table */ + *imm = (unsigned long )insn_array->ips + off; + +return 0 ; +} + BTF_ID_LIST_SINGLE(insn_array_btf_ids, struct, bpf_insn_array)const struct bpf_map_ops insn_array_map_ops = {
@@ -139,6 +153,7 @@ const struct bpf_map_ops insn_array_map_ops = { .map_delete_elem = insn_array_delete_elem, .map_check_btf = insn_array_check_btf, .map_mem_usage = insn_array_mem_usage, + .map_direct_value_addr = insn_array_map_direct_value_addr, .map_btf_id = &insn_array_btf_ids[0], };
kernel/bpf/core.c +1 -0
@@ -1708,6 +1708,7 @@ bool bpf_opcode_in_insntable(u8 code) [BPF_LD | BPF_IND | BPF_B] = true, [BPF_LD | BPF_IND | BPF_H] = true, [BPF_LD | BPF_IND | BPF_W] = true, + [BPF_JMP | BPF_JA | BPF_X] = true, [BPF_JMP | BPF_JCOND] = true, };#undef BPF_INSN_3_TBL
kernel/bpf/liveness.c +3 -0
@@ -485,6 +485,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)struct bpf_iarray *succ;int insn_sz; +if (unlikely (insn_is_gotox (insn))) +return env->insn_aux_data[idx].jt; +/* pre-allocated array of size up to 2; reset cnt, as it may have been used already */ succ = env->succ; succ->cnt =0 ;
kernel/bpf/log.c +1 -0
@@ -461,6 +461,7 @@ const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type) [PTR_TO_ARENA] = "arena", [PTR_TO_BUF] = "buf", [PTR_TO_FUNC] = "func", + [PTR_TO_INSN] = "insn", [PTR_TO_MAP_KEY] = "map_key", [CONST_PTR_TO_DYNPTR] = "dynptr_ptr", };
kernel/bpf/verifier.c +357 -6
474 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps
Quoted 17 lines · Anton Protopopov
Add support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0[...]
kernel/bpf/verifier.c +11
-0+static struct bpf_iarray * +create_jt (int t,struct bpf_verifier_env *env,int fd) +{ +static struct bpf_subprog_info *subprog; +int subprog_start, subprog_end; +struct bpf_iarray *jt; +int i; + + subprog =bpf_find_containing_subprog (env, t); + subprog_start = subprog->start; + subprog_end = (subprog +1 )->start;
kernel/bpf/verifier.c static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem) · line 17994
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ jt =jt_from_subprog (env, subprog_start, subprog_end); +if (IS_ERR (jt)) +return jt; + +/* Check that the every element of the jump table fits within the given subprogram */ +for (i =0 ; i < jt->cnt; i++) { +if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) { +verbose (env,"jump table for insn %d points outside of the subprog [%u,%u]" , + t, subprog_start, subprog_end); +return ERR_PTR (-EINVAL );
AI found a bug here: jt should have been freed in this error path. Will fix in the next version.
Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps
In reply to Anton ProtopopovQuoted 39 lines · Anton Protopopov
Add support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0which does an indirect jump to a location stored in Rx. The register Rx should have type
PTR_TO_INSN. This new type assures that the Rx register contains a value (or a range of values) loaded from a correct jump table – map of type instruction array.For example, for a C switch LLVM will generate the following code:
0: r3 = r1 # "switch (r3)" 1: if r3 > 0x13 goto +0x666 # check r3 boundaries 2: r3 <<= 0x3 # adjust to an index in array of addresses 3: r1 = 0xbeef ll # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M 5: r1 += r3 # r1 inherits boundaries from r3 6: r1 = *(u64 *)(r1 + 0x0) # r1 now has type INSN_TO_PTR 7: gotox r1 # jit will generate proper codeHere the gotox instruction corresponds to one particular map. This is possible however to have a gotox instruction which can be loaded from different maps, e.g.
0: r1 &= 0x1 1: r2 <<= 0x3 2: r3 = 0x0 ll # load from map M_1 4: r3 += r2 5: if r1 == 0x0 goto +0x4 6: r1 <<= 0x3 7: r3 = 0x0 ll # load from map M_2 9: r3 += r1 A: r1 = *(u64 *)(r3 + 0x0) B: gotox r1 # jump to target loaded from M_1 or M_2During
check_cfgstage the verifier will collect all the maps which point to inside the subprog being verified. When building the config, the high 16 bytes of theinsn_stateare used, so this patch (theoretically) supports jump tables of up to 2^16 slots.
Quoted 5 lines
During the later stage, in
check_indirect_jump, it is checked that the register Rx was loaded from a particular instruction array.Signed-off-by Anton Protopopov <[email protected]>---
LGTM, please, address a few remaining points.
[...]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[...]
kernel/bpf/verifier.c static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem) · line 17975
Quoted 2 lines
kernel/bpf/verifier.c +2
-0+static struct bpf_iarray * +create_jt (int t,struct bpf_verifier_env *env,int fd)
^^^^^^
This parameter is unused
Quoted 24 lines
kernel/bpf/verifier.c +24
-0+{ +static struct bpf_subprog_info *subprog; +int subprog_start, subprog_end; +struct bpf_iarray *jt; +int i; + + subprog =bpf_find_containing_subprog (env, t); + subprog_start = subprog->start; + subprog_end = (subprog +1 )->start; + jt =jt_from_subprog (env, subprog_start, subprog_end); +if (IS_ERR (jt)) +return jt; + +/* Check that the every element of the jump table fits within the given subprogram */ +for (i =0 ; i < jt->cnt; i++) { +if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) { +verbose (env,"jump table for insn %d points outside of the subprog [%u,%u]" , + t, subprog_start, subprog_end); +return ERR_PTR (-EINVAL ); + } + } + +return jt; +}
[...]
Quoted 15 lines
kernel/bpf/verifier.c +15
-0+/* gotox *dst_reg */ +static int check_indirect_jump (struct bpf_verifier_env *env,struct bpf_insn *insn) +{ +struct bpf_verifier_state *other_branch; +struct bpf_reg_state *dst_reg; +struct bpf_map *map; +u32 min_index, max_index; +int err =0 ; +int n; +int i; + + dst_reg =reg_state (env, insn->dst_reg); +if (dst_reg->type !=PTR_TO_INSN ) { +verbose (env,"R%d has type %d, expected PTR_TO_INSN\n" , + insn->dst_reg, dst_reg->type);
kernel/bpf/verifier.c static int process_bpf_exit_full(struct bpf_verifier_env *env, · line 20138
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return -EINVAL ; + } + + map = dst_reg->map_ptr; +if (verifier_bug_if (!map, env,"R%d has an empty map pointer" , insn->dst_reg)) +return -EFAULT ; + +if (verifier_bug_if (map->map_type !=BPF_MAP_TYPE_INSN_ARRAY , env, +"R%d has incorrect map type %d" , insn->dst_reg, map->map_type)) +return -EFAULT ;
Nit: we discussed this in v5, let's drop the verifier_bug_if() and
return -EINVAL?
> The program can be written in a way, such that e.g. hash map > pointer is passed as a parameter for gotox, that would be an > incorrect program, not a verifier bug.
Also, use reg_type_str() instead of "type %d"?
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+ + err =indirect_jump_min_max_index (env, insn->dst_reg, map, &min_index, &max_index); +if (err) +return err;
kernel/bpf/verifier.c static int process_bpf_exit_full(struct bpf_verifier_env *env, · line 20152
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ +/* Ensure that the buffer is large enough */ +if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index +1 ) { + env->gotox_tmp_buf =iarray_realloc (env->gotox_tmp_buf, + max_index - min_index +1 ); +if (!env->gotox_tmp_buf) +return -ENOMEM ; + } + + n =copy_insn_array_uniq (map, min_index, max_index, env->gotox_tmp_buf->items);
Nit: let's not forget about a follow-up to remove this allocation.
Quoted 18 lines
kernel/bpf/verifier.c +18
-0+if (n <0 ) +return n; +if (n ==0 ) { +verbose (env,"register R%d doesn't point to any offset in map id=%d\n" , + insn->dst_reg, map->id); +return -EINVAL ; + } + +for (i =0 ; i < n -1 ; i++) { + other_branch =push_stack (env, env->gotox_tmp_buf->items[i], + env->insn_idx, env->cur_state->speculative); +if (IS_ERR (other_branch)) +return PTR_ERR (other_branch); + } + env->insn_idx = env->gotox_tmp_buf->items[n-1 ]; +return 0 ; +} +
[...]
Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps
Quoted 59 lines · Eduard Zingerman
Quoted 39 lines · Anton Protopopov
Add support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0which does an indirect jump to a location stored in Rx. The register Rx should have type
PTR_TO_INSN. This new type assures that the Rx register contains a value (or a range of values) loaded from a correct jump table – map of type instruction array.For example, for a C switch LLVM will generate the following code:
0: r3 = r1 # "switch (r3)" 1: if r3 > 0x13 goto +0x666 # check r3 boundaries 2: r3 <<= 0x3 # adjust to an index in array of addresses 3: r1 = 0xbeef ll # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M 5: r1 += r3 # r1 inherits boundaries from r3 6: r1 = *(u64 *)(r1 + 0x0) # r1 now has type INSN_TO_PTR 7: gotox r1 # jit will generate proper codeHere the gotox instruction corresponds to one particular map. This is possible however to have a gotox instruction which can be loaded from different maps, e.g.
0: r1 &= 0x1 1: r2 <<= 0x3 2: r3 = 0x0 ll # load from map M_1 4: r3 += r2 5: if r1 == 0x0 goto +0x4 6: r1 <<= 0x3 7: r3 = 0x0 ll # load from map M_2 9: r3 += r1 A: r1 = *(u64 *)(r3 + 0x0) B: gotox r1 # jump to target loaded from M_1 or M_2During
check_cfgstage the verifier will collect all the maps which point to inside the subprog being verified. When building the config, the high 16 bytes of theinsn_stateare used, so this patch (theoretically) supports jump tables of up to 2^16 slots.Quoted 5 lines
During the later stage, in
check_indirect_jump, it is checked that the register Rx was loaded from a particular instruction array.Signed-off-by Anton Protopopov <[email protected]>---
LGTM, please, address a few remaining points.
Acked-by Eduard Zingerman <[email protected]>[...]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[...]
Quoted 4 lines
kernel/bpf/verifier.c static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem) · line 17975
Quoted 2 lines
kernel/bpf/verifier.c +2
-0+static struct bpf_iarray * +create_jt (int t,struct bpf_verifier_env *env,int fd)^^^^^^
This parameter is unused
Ah, yes, no more ->imm. Fixed.
Quoted 47 lines
Quoted 24 lines
kernel/bpf/verifier.c +24
-0+{ +static struct bpf_subprog_info *subprog; +int subprog_start, subprog_end; +struct bpf_iarray *jt; +int i; + + subprog =bpf_find_containing_subprog (env, t); + subprog_start = subprog->start; + subprog_end = (subprog +1 )->start; + jt =jt_from_subprog (env, subprog_start, subprog_end); +if (IS_ERR (jt)) +return jt; + +/* Check that the every element of the jump table fits within the given subprogram */ +for (i =0 ; i < jt->cnt; i++) { +if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) { +verbose (env,"jump table for insn %d points outside of the subprog [%u,%u]" , + t, subprog_start, subprog_end); +return ERR_PTR (-EINVAL ); + } + } + +return jt; +}[...]
Quoted 20 lines
kernel/bpf/verifier.c +20
-0+/* gotox *dst_reg */ +static int check_indirect_jump (struct bpf_verifier_env *env,struct bpf_insn *insn) +{ +struct bpf_verifier_state *other_branch; +struct bpf_reg_state *dst_reg; +struct bpf_map *map; +u32 min_index, max_index; +int err =0 ; +int n; +int i; + + dst_reg =reg_state (env, insn->dst_reg); +if (dst_reg->type !=PTR_TO_INSN ) { +verbose (env,"R%d has type %d, expected PTR_TO_INSN\n" , + insn->dst_reg, dst_reg->type); +return -EINVAL ; + } + + map = dst_reg->map_ptr; +if (verifier_bug_if (!map, env,"R%d has an empty map pointer" , insn->dst_reg))
Quoted 8 lines
kernel/bpf/verifier.c static int process_bpf_exit_full(struct bpf_verifier_env *env, · line 20138
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+return -EFAULT ; + +if (verifier_bug_if (map->map_type !=BPF_MAP_TYPE_INSN_ARRAY , env, +"R%d has incorrect map type %d" , insn->dst_reg, map->map_type)) +return -EFAULT ;Nit: we discussed this in v5, let's drop the
verifier_bug_if()and
return-EINVAL?
So, I think this is a verifier bug. We've checked above that the register is PTR_TO_INSN, so it must have map and map type should be BPF_MAP_TYPE_INSN_ARRAY. Now this is always true, for future I added these warnings, just in case. Wdyt?
Quoted 5 lines
> The program can be written in a way, such that e.g. hash map > pointer is passed as a parameter for gotox, that would be an > incorrect program, not a verifier bug.
Also, use
reg_type_str()instead of "type %d"?
This is map type, not register? Ah, you maybe meant the first message, I will fix it, thanks.
Quoted 8 lines
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+ + err =indirect_jump_min_max_index (env, insn->dst_reg, map, &min_index, &max_index); +if (err) +return err; + +/* Ensure that the buffer is large enough */ +if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index +1 ) { + env->gotox_tmp_buf = iarray_realloc(env->gotox_tmp_buf,
Quoted 8 lines
kernel/bpf/verifier.c static int process_bpf_exit_full(struct bpf_verifier_env *env, · line 20152
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+ max_index - min_index + 1); +if (!env->gotox_tmp_buf) +return -ENOMEM ; + } + + n =copy_insn_array_uniq (map, min_index, max_index, env->gotox_tmp_buf->items);Nit: let's not forget about a follow-up to remove this allocation.
Thanks, in the list of followups.
Quoted 20 lines
Quoted 18 lines
kernel/bpf/verifier.c +18
-0+if (n <0 ) +return n; +if (n ==0 ) { +verbose (env,"register R%d doesn't point to any offset in map id=%d\n" , + insn->dst_reg, map->id); +return -EINVAL ; + } + +for (i =0 ; i < n -1 ; i++) { + other_branch =push_stack (env, env->gotox_tmp_buf->items[i], + env->insn_idx, env->cur_state->speculative); +if (IS_ERR (other_branch)) +return PTR_ERR (other_branch); + } + env->insn_idx = env->gotox_tmp_buf->items[n-1 ]; +return 0 ; +} +[...]
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps
On Wed, 2025-10-22 at 06:51 +0000, Anton Protopopov wrote:
[...]
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
kernel/bpf/verifier.c +20
-0+/* gotox *dst_reg */ +static int check_indirect_jump (struct bpf_verifier_env *env,struct bpf_insn *insn) +{ +struct bpf_verifier_state *other_branch; +struct bpf_reg_state *dst_reg; +struct bpf_map *map; +u32 min_index, max_index; +int err =0 ; +int n; +int i; + + dst_reg =reg_state (env, insn->dst_reg); +if (dst_reg->type !=PTR_TO_INSN ) { +verbose (env,"R%d has type %d, expected PTR_TO_INSN\n" , + insn->dst_reg, dst_reg->type); +return -EINVAL ; + } + + map = dst_reg->map_ptr; +if (verifier_bug_if (!map, env,"R%d has an empty map pointer" , insn->dst_reg))Quoted 8 lines
kernel/bpf/verifier.c static int process_bpf_exit_full(struct bpf_verifier_env *env, · line 20138
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+return -EFAULT ; + +if (verifier_bug_if (map->map_type !=BPF_MAP_TYPE_INSN_ARRAY , env, +"R%d has incorrect map type %d" , insn->dst_reg, map->map_type)) +return -EFAULT ;Nit: we discussed this in v5, let's drop the
verifier_bug_if()and
return-EINVAL?
Quoted 4 lines
So, I think this is a verifier bug. We've checked above that the register is
PTR_TO_INSN, so it must have map and map type should beBPF_MAP_TYPE_INSN_ARRAY. Now this is always true, for future I added these warnings, just in case. Wdyt?
Wellp, yes this is a verifier bug, thank you for explaining. Sorry for the noise.
Quoted 8 lines
Quoted 5 lines
> The program can be written in a way, such that e.g. hash map > pointer is passed as a parameter for gotox, that would be an > incorrect program, not a verifier bug.
Also, use
reg_type_str()instead of "type %d"?This is map type, not register? Ah, you maybe meant the first message, I will fix it, thanks.
Hm, right, doesn't apply here, but helps with the first message. It looks like we don't have a similar function for maps. But that's a "bug", not verifier message, so probably fine.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+ + err =indirect_jump_min_max_index (env, insn->dst_reg, map, &min_index, &max_index); +if (err) +return err; +
[...]
PATCH v6 11/17 bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
In reply to Anton ProtopopovAdd support for indirect jump instruction.
Example output from bpftool:
0: (79) r3 = *(u64 *)(r1 +0)
1: (25) if r3 > 0x4 goto pc+666
2: (67) r3 <<= 3
3: (18) r1 = 0xffffbeefspameggs
5: (0f) r1 += r3
6: (79) r1 = *(u64 *)(r1 +0)
7: (0d) gotox r1
--- kernel/bpf/disasm.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/bpf/disasm.c +3 -0
@@ -358,6 +358,9 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, }else if (insn->code == (BPF_JMP |BPF_JA )) {verbose (cbs->private_data,"(%02x) goto pc%+d\n" , insn->code, insn->off); + }else if (insn->code == (BPF_JMP |BPF_JA |BPF_X )) { +verbose (cbs->private_data,"(%02x) gotox r%d\n" , + insn->code, insn->dst_reg); }else if (insn->code == (BPF_JMP |BPF_JCOND ) && insn->src_reg ==BPF_MAY_GOTO ) { verbose(cbs->private_data, "(%02x) may_goto pc%+d\n",
Signature · 1 line
2.34.1
Re: [PATCH v6 bpf-next 11/17] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
Quoted 12 lines · Anton Protopopov
Add support for indirect jump instruction.
Example output from bpftool:
0: (79) r3 = *(u64 *)(r1 +0) 1: (25) if r3 > 0x4 goto pc+666 2: (67) r3 <<= 3 3: (18) r1 = 0xffffbeefspameggs 5: (0f) r1 += r3 6: (79) r1 = *(u64 *)(r1 +0) 7: (0d) gotox r1
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
[...]
PATCH v6 12/17 bpf, docs: do not state that indirect jumps are not supported
In reply to Anton ProtopopovThe linux-notes.rst states that indirect jump instruction "is not currently supported by the verifier". Remove this part as outdated.
--- Documentation/bpf/linux-notes.rst | 8 -------- 1 file changed, 8 deletions(-)
Documentation/bpf/linux-notes.rst +0 -8
@@ -12,14 +12,6 @@ Byte swap instructions ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively. -Jump instructions -================= - -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function -integer would be read from a specified register, is not currently supported -by the verifier. Any programs with this instruction will fail to load -until such support is added. - Maps ====
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported
Quoted 17 lines · Anton Protopopov
The
linux-notes.rststates that indirect jump instruction "is not currently supported by the verifier". Remove this part as outdated.Signed-off-by Anton Protopopov <[email protected]>--- Documentation/bpf/linux-notes.rst | 8 -------- 1 file changed, 8 deletions(-)Documentation/bpf/linux-notes.rst +0
-1@@ -12,14 +12,6 @@ Byte swap instructions ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively. -Jump instructions
Documentation/bpf/linux-notes.rst Byte swap instructions · line 17
Quoted 10 lines
Documentation/bpf/linux-notes.rst +0
-7-================= - -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function -integer would be read from a specified register, is not currently supported -by the verifier. Any programs with this instruction will fail to load -until such support is added. - Maps ====
Nit: bpf/standardization/instruction-set.rst needs an update,
we don't have anything about JA|X|JMP in the "Jump instructions" section there.
Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported
Quoted 17 lines · Eduard Zingerman
Quoted 16 lines · Anton Protopopov
The
linux-notes.rststates that indirect jump instruction "is not currently supported by the verifier". Remove this part as outdated.Signed-off-by Anton Protopopov <[email protected]>--- Documentation/bpf/linux-notes.rst | 8 -------- 1 file changed, 8 deletions(-)Documentation/bpf/linux-notes.rst +0
-0@@ -12,14 +12,6 @@ Byte swap instructions ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
Quoted 15 lines
Documentation/bpf/linux-notes.rst Byte swap instructions · line 17
Quoted 11 lines
Documentation/bpf/linux-notes.rst +0
-8-Jump instructions -================= - -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function -integer would be read from a specified register, is not currently supported -by the verifier. Any programs with this instruction will fail to load -until such support is added. - Maps ====Nit:
bpf/standardization/instruction-set.rstneeds an update,
we don't have anything aboutJA|X|JMPin the "Jump instructions" section there.
Ah yes, thanks.
Also, there is a limitation listed in the llvm doc that -O0 can't be used due to absence of indirect jumps. I wonder if there should be more limitations introduced since the doc was written. (I've tried, briefly, to compile selftests with -O0, but this fails for other reasons, and I didn't have time to dig into this.)
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported
Quoted 34 lines · Anton Protopopov
Quoted 17 lines · Eduard Zingerman
Quoted 16 lines · Anton Protopopov
The
linux-notes.rststates that indirect jump instruction "is not currently supported by the verifier". Remove this part as outdated.Signed-off-by Anton Protopopov <[email protected]>--- Documentation/bpf/linux-notes.rst | 8 -------- 1 file changed, 8 deletions(-)Documentation/bpf/linux-notes.rst +0
-0@@ -12,14 +12,6 @@ Byte swap instructions ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.Quoted 15 lines
Documentation/bpf/linux-notes.rst Byte swap instructions · line 17
Quoted 11 lines
Documentation/bpf/linux-notes.rst +0
-8-Jump instructions -================= - -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function -integer would be read from a specified register, is not currently supported -by the verifier. Any programs with this instruction will fail to load -until such support is added. - Maps ====Nit:
bpf/standardization/instruction-set.rstneeds an update,
we don't have anything aboutJA|X|JMPin the "Jump instructions" section there.
Quoted 8 lines
Ah yes, thanks.
Also, there is a limitation listed in the llvm doc that -O0 can't be used due to absence of indirect jumps. I wonder if there should be more limitations introduced since the doc was written. (I've tried, briefly, to compile selftests with -O0, but this fails for other reasons, and I didn't have time to dig into this.)
Lets fill this section as we go.
From the top of my head, I can't say what will or will not happen to verifier if O0 is used. Things that don't happen at O0 include:
- SROA (variables are always on stack);
- constant propagation;
- inlining;
- loop unrolling.
In theory, none of that should confuse verifier in its current state. But I'm sure there are special cases.
Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported
Quoted 52 lines · Eduard Zingerman
Quoted 34 lines · Anton Protopopov
Quoted 17 lines · Eduard Zingerman
Quoted 16 lines · Anton Protopopov
The
linux-notes.rststates that indirect jump instruction "is not currently supported by the verifier". Remove this part as outdated.Signed-off-by Anton Protopopov <[email protected]>--- Documentation/bpf/linux-notes.rst | 8 -------- 1 file changed, 8 deletions(-)Documentation/bpf/linux-notes.rst +0
-0@@ -12,14 +12,6 @@ Byte swap instructions ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.Quoted 15 lines
Documentation/bpf/linux-notes.rst Byte swap instructions · line 17
Quoted 11 lines
Documentation/bpf/linux-notes.rst +0
-8-Jump instructions -================= - -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function -integer would be read from a specified register, is not currently supported -by the verifier. Any programs with this instruction will fail to load -until such support is added. - Maps ====Nit:
bpf/standardization/instruction-set.rstneeds an update,
we don't have anything aboutJA|X|JMPin the "Jump instructions" section there.Quoted 8 lines
Ah yes, thanks.
Also, there is a limitation listed in the llvm doc that -O0 can't be used due to absence of indirect jumps. I wonder if there should be more limitations introduced since the doc was written. (I've tried, briefly, to compile selftests with -O0, but this fails for other reasons, and I didn't have time to dig into this.)
Lets fill this section as we go.
From the top of my head, I can't say what will or will not happen to verifier if O0 is used. Things that don't happen at O0 include:
- SROA (variables are always on stack);
- constant propagation;
- inlining;
- loop unrolling.
Quoted 2 lines
In theory, none of that should confuse verifier in its current state. But I'm sure there are special cases.
Stack was the first thing I've bumped into:
progs/bpf_arena_spin_lock.h:164:12: error: Looks like the BPF stack limit is exceeded. Please move large on stack variables into BPF per-cpu array map. For non-kernel uses, the stack can be
increased using -mllvm -bpf-stack-size.
164 | } while (!atomic_try_cmpxchg_relaxed(&lock->val, &old, new));
| ^
But then also some things, say
tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:169:15: error: invalid operand for inline asm constraint 'i'
169 | asm volatile("r1 = %[ctx]\n\t"
| ^
"r2 = %[map]\n\t"
"r3 = %[slot]\n\t"
"call 12"
:: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot)
: "r0", "r1", "r2", "r3", "r4", "r5");Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported
On Tue, 2025-10-21 at 19:50 +0000, Anton Protopopov wrote:
[...]
Quoted 7 lines
Stack was the first thing I've bumped into:
progs/bpf_arena_spin_lock.h:164:12: error: Looks like the BPF stack limit is exceeded. Please move large on stack variables into BPF per-cpu array map. For non-kernel uses, the stack can be
increased using -mllvm -bpf-stack-size.164 | } while (!atomic_try_cmpxchg_relaxed(&lock->val, &old, new));
| ^
I mean, this makes sense ¯\_(ツ)_/¯.
Quoted 10 lines
But then also some things, say
tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:169:15: error: invalid operand for inline asm constraint 'i' 169 | asm volatile("r1 = %[ctx]\n\t" | ^ "r2 = %[map]\n\t" "r3 = %[slot]\n\t" "call 12" :: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot) : "r0", "r1", "r2", "r3", "r4", "r5");
That's probably because of the missing constant propagation.
PATCH v6 13/17 libbpf: fix formatting of bpf_object__append_subprog_code
In reply to Anton ProtopopovThe commit 6c918709bd30 ("libbpf: Refactor bpf_object__reloc_code") added the bpf_object__append_subprog_code() with incorrect indentations. Use tabs instead. (This also makes a consequent commit better readable.)
--- tools/lib/bpf/libbpf.c | 52 +++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
tools/lib/bpf/libbpf.c +26 -26
@@ -6436,32 +6436,32 @@ static int bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog, struct bpf_program *subprog) { - struct bpf_insn *insns; -size_t new_cnt; -int err; - - subprog->sub_insn_off = main_prog->insns_cnt; - - new_cnt = main_prog->insns_cnt + subprog->insns_cnt; - insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); -if (!insns) { -pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); -return -ENOMEM ; - } - main_prog->insns = insns; - main_prog->insns_cnt = new_cnt; - -memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, - subprog->insns_cnt *sizeof (*insns)); - -pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , - main_prog->name, subprog->insns_cnt, subprog->name); - -/* The subprog insns are now appended. Append its relos too. */ - err =append_subprog_relos (main_prog, subprog); -if (err) -return err; -return 0 ; + struct bpf_insn *insns; +size_t new_cnt; +int err; + + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); +if (!insns) { +pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); +return -ENOMEM ; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + +memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt *sizeof (*insns)); + +pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , + main_prog->name, subprog->insns_cnt, subprog->name); + +/* The subprog insns are now appended. Append its relos too. */ + err =append_subprog_relos (main_prog, subprog); +if (err) +return err; +return 0 ; } static int
Signature · 1 line
2.34.1
PATCH v6 14/17 libbpf: support llvm-generated indirect jumps
In reply to Anton ProtopopovFor v4 instruction set LLVM is allowed to generate indirect jumps for switch statements and for 'goto *rX' assembly. Every such a jump will be accompanied by necessary metadata, e.g. (llvm-objdump -Sr ...):
0: r2 = 0x0 ll
0000000000000030: R_BPF_64_64 BPF.JT.0.0Here BPF.JT.1.0 is a symbol residing in the .jumptables section:
Symbol table:
4: 0000000000000000 240 OBJECT GLOBAL DEFAULT 4 BPF.JT.0.0The -bpf-min-jump-table-entries llvm option may be used to control the minimal size of a switch which will be converted to an indirect jumps.
--- tools/lib/bpf/libbpf.c | 240 +++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf_probes.c | 4 + tools/lib/bpf/linker.c | 10 +- 3 files changed, 251 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +239 -1
@@ -190,6 +190,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", [BPF_MAP_TYPE_ARENA] = "arena", + [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array", };static const char *const prog_type_name[] = {
@@ -369,6 +370,7 @@ enum reloc_type {RELO_EXTERN_CALL ,RELO_SUBPROG_ADDR ,RELO_CORE , + RELO_INSN_ARRAY, }; struct reloc_desc {
@@ -379,7 +381,16 @@ struct reloc_desc {struct {int map_idx;int sym_off; -int ext_idx; +/* +* The following two fields can be unionized, as the +* ext_idx field is used for extern symbols, and the +* sym_size is used for jump tables, which are never +* extern +*/ +union { +int ext_idx; +int sym_size; + }; }; }; };
@@ -421,6 +432,11 @@ struct bpf_sec_def {libbpf_prog_attach_fn_t prog_attach_fn; }; +struct bpf_light_subprog { +__u32 sec_insn_off; +__u32 sub_insn_off; +}; +/* * bpf_prog should be a better name but it has been used in * linux/filter.h.
@@ -494,6 +510,9 @@ struct bpf_program {__u32 line_info_cnt;__u32 prog_flags;__u8 hash[SHA256_DIGEST_LENGTH ]; + +struct bpf_light_subprog *subprogs; +__u32 subprog_cnt; };struct bpf_struct_ops {
@@ -523,6 +542,7 @@ struct bpf_struct_ops {#define STRUCT_OPS_SEC ".struct_ops"#define STRUCT_OPS_LINK_SEC ".struct_ops.link"#define ARENA_SEC ".addr_space.1" +#define JUMPTABLES_SEC ".jumptables"enum libbpf_map_type {LIBBPF_MAP_UNSPEC ,
@@ -667,6 +687,7 @@ struct elf_state {int symbols_shndx;bool has_st_ops;int arena_data_shndx; +int jumptables_data_shndx; };struct usdt_manager ;
@@ -738,6 +759,16 @@ struct bpf_object {void *arena_data;size_t arena_data_sz; +void *jumptables_data; +size_t jumptables_data_sz; + +struct { +struct bpf_program *prog; +int sym_off; +int fd; + } *jumptable_maps; +size_t jumptable_map_cnt; +struct kern_feature_cache *feat_cache;char *token_path;int token_fd;
@@ -764,6 +795,7 @@ void bpf_program__unload(struct bpf_program *prog)zfree (&prog->func_info);zfree (&prog->line_info); +zfree (&prog->subprogs); }static void bpf_program__exit (struct bpf_program *prog)
@@ -3942,6 +3974,13 @@ static int bpf_object__elf_collect(struct bpf_object *obj) }else if (strcmp (name,ARENA_SEC ) ==0 ) { obj->efile.arena_data = data; obj->efile.arena_data_shndx = idx; + }else if (strcmp (name,JUMPTABLES_SEC ) ==0 ) { + obj->jumptables_data =malloc (data->d_size); +if (!obj->jumptables_data) +return -ENOMEM ; +memcpy (obj->jumptables_data, data->d_buf, data->d_size); + obj->jumptables_data_sz = data->d_size; + obj->efile.jumptables_data_shndx = idx; }else {pr_info ("elf: skipping unrecognized data section(%d) %s\n" , idx, name);
@@ -4634,6 +4673,16 @@ static int bpf_program__record_reloc(struct bpf_program *prog,return 0 ; } +/* jump table data relocation */ +if (shdr_idx == obj->efile.jumptables_data_shndx) { + reloc_desc->type =RELO_INSN_ARRAY ; + reloc_desc->insn_idx = insn_idx; + reloc_desc->map_idx =-1 ; + reloc_desc->sym_off = sym->st_value; + reloc_desc->sym_size = sym->st_size; +return 0 ; + } +/* generic map reference relocation */ if (type ==LIBBPF_MAP_UNSPEC ) {if (!bpf_object__shndx_is_maps (obj, shdr_idx)) {
@@ -6144,6 +6193,144 @@ static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, insn->imm =POISON_CALL_KFUNC_BASE + ext_idx; } +static int find_jt_map (struct bpf_object *obj,struct bpf_program *prog,int sym_off) +{ +size_t i; + +for (i =0 ; i < obj->jumptable_map_cnt; i++) { +/* +* This might happen that same offset is used for two different +* programs (as jump tables can be the same). However, for +* different programs different maps should be created. +*/ +if (obj->jumptable_maps[i].sym_off == sym_off && + obj->jumptable_maps[i].prog == prog) +return obj->jumptable_maps[i].fd; + } + +return -ENOENT ; +} + +static int add_jt_map (struct bpf_object *obj,struct bpf_program *prog,int sym_off,int map_fd) +{ +size_t new_cnt = obj->jumptable_map_cnt +1 ; +size_t size =sizeof (obj->jumptable_maps[0 ]); +void *tmp; + + tmp =libbpf_reallocarray (obj->jumptable_maps, new_cnt, size); +if (!tmp) +return -ENOMEM ; + + obj->jumptable_maps = tmp; + obj->jumptable_maps[new_cnt -1 ].prog = prog; + obj->jumptable_maps[new_cnt -1 ].sym_off = sym_off; + obj->jumptable_maps[new_cnt -1 ].fd = map_fd; + obj->jumptable_map_cnt = new_cnt; + +return 0 ; +} + +static int create_jt_map (struct bpf_object *obj,struct bpf_program *prog, +int sym_off,int jt_size,int adjust_off) +{ +const __u32 jt_entry_size =8 ; +const __u32 max_entries = jt_size / jt_entry_size; +const __u32 value_size =sizeof (struct bpf_insn_array_value ); +struct bpf_insn_array_value val = {}; +int map_fd, err; +__u64 insn_off; +__u64 *jt; +__u32 i; + + map_fd =find_jt_map (obj, prog, sym_off); +if (map_fd >=0 ) +return map_fd; + +if (sym_off % jt_entry_size) { +pr_warn ("jumptable start %d should be multiple of %u\n" , + sym_off, jt_entry_size); +return -EINVAL ; + } + +if (jt_size % jt_entry_size) { +pr_warn ("jumptable size %d should be multiple of %u\n" , + jt_size, jt_entry_size); +return -EINVAL ; + } + + map_fd =bpf_map_create (BPF_MAP_TYPE_INSN_ARRAY ,".jumptables" , +4 , value_size, max_entries,NULL ); +if (map_fd <0 ) +return map_fd; + +if (!obj->jumptables_data) { +pr_warn ("map '.jumptables': ELF file is missing jump table data\n" ); + err = -EINVAL ; + goto err_close; + } +if (sym_off + jt_size > obj->jumptables_data_sz) { +pr_warn ("jumptables_data size is %zd, trying to access %d\n" , + obj->jumptables_data_sz, sym_off + jt_size); + err = -EINVAL ; + goto err_close; + } + + jt = (__u64 *)(obj->jumptables_data + sym_off); +for (i =0 ; i < max_entries; i++) { +/* +* LLVM-generated jump tables contain u64 records, however +* should contain values that fit in u32. +* The adjust_off provided by the caller adjusts the offset to +* be relative to the beginning of the main function +*/ + insn_off = jt[i]/sizeof (struct bpf_insn ) + adjust_off; +if (insn_off >UINT32_MAX ) { +pr_warn ("invalid jump table value %llx at offset %d (adjust_off %d)\n" , + jt[i], sym_off + i, adjust_off); + err = -EINVAL ; + goto err_close; + } + + val.orig_off = insn_off; + err =bpf_map_update_elem (map_fd, &i, &val,0 ); +if (err) + goto err_close; + } + + err =bpf_map_freeze (map_fd); +if (err) + goto err_close; + + err =add_jt_map (obj, prog, sym_off, map_fd); +if (err) + goto err_close; + +return map_fd; + +err_close: +close (map_fd); +return err; +} + +/* +* In LLVM the .jumptables section contains jump tables entries relative to the +* section start. The BPF kernel-side code expects jump table offsets relative +* to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper +* computes a delta to be added when creating a map. +*/ +static int jt_adjust_off (struct bpf_program *prog,int insn_idx) +{ +int i; + +for (i = prog->subprog_cnt -1 ; i >=0 ; i--) { +if (insn_idx >= prog->subprogs[i].sub_insn_off) +return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off; + } + +return -prog->sec_insn_off; +} + +/* Relocate data references within program code: * - map references; * - global variable references;
@@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)case RELO_CORE :/* will be handled by bpf_program_record_relos() */ break ; +case RELO_INSN_ARRAY : { +int map_fd; + + map_fd =create_jt_map (obj, prog, relo->sym_off, relo->sym_size, +jt_adjust_off (prog, relo->insn_idx)); +if (map_fd <0 ) { +pr_warn ("prog '%s': relo #%d: can't create jump table: sym_off %u\n" , + prog->name, i, relo->sym_off); +return map_fd; + } + insn[0 ].src_reg =BPF_PSEUDO_MAP_VALUE ; + insn->imm = map_fd; + insn->off =0 ; + } +break ;default :pr_warn ("prog '%s': relo #%d: bad relo type %d\n" , prog->name, i, relo->type);
@@ -6432,6 +6634,24 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_prograreturn 0 ; } +static int save_subprog_offsets (struct bpf_program *main_prog,struct bpf_program *subprog) +{ +size_t size =sizeof (main_prog->subprogs[0 ]); +int new_cnt = main_prog->subprog_cnt +1 ; +void *tmp; + + tmp =libbpf_reallocarray (main_prog->subprogs, new_cnt, size); +if (!tmp) +return -ENOMEM ; + + main_prog->subprogs = tmp; + main_prog->subprogs[new_cnt -1 ].sec_insn_off = subprog->sec_insn_off; + main_prog->subprogs[new_cnt -1 ].sub_insn_off = subprog->sub_insn_off; + main_prog->subprog_cnt = new_cnt; + +return 0 ; +} +static int bpf_object__append_subprog_code (struct bpf_object *obj,struct bpf_program *main_prog,struct bpf_program *subprog)
@@ -6461,6 +6681,15 @@ bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main err =append_subprog_relos (main_prog, subprog);if (err)return err; + +/* Save subprogram offsets */ + err =save_subprog_offsets (main_prog, subprog); +if (err) { +pr_warn ("prog '%s': failed to add subprog offsets: %s\n" , + main_prog->name,errstr (err)); +return err; + } +return 0 ; }
@@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)zfree (&obj->arena_data); +zfree (&obj->jumptables_data); + obj->jumptables_data_sz =0 ; + +if (obj->jumptable_maps && obj->jumptable_map_cnt) { +for (i =0 ; i < obj->jumptable_map_cnt; i++) +close (obj->jumptable_maps[i].fd); + } +zfree (&obj->jumptable_maps); +free (obj); }
tools/lib/bpf/libbpf_probes.c +4 -0
@@ -364,6 +364,10 @@ static int probe_map_create(enum bpf_map_type map_type)case BPF_MAP_TYPE_SOCKHASH :case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY :break ; +case BPF_MAP_TYPE_INSN_ARRAY : + key_size =sizeof (__u32); + value_size =sizeof (struct bpf_insn_array_value ); +break ;case BPF_MAP_TYPE_UNSPEC :default :return -EOPNOTSUPP ;
tools/lib/bpf/linker.c +8 -2
@@ -27,6 +27,8 @@#include "strset.h" #define BTF_EXTERN_SEC ".extern" +#define JUMPTABLES_SEC ".jumptables" +#define JUMPTABLES_REL_SEC ".rel.jumptables"struct src_sec {const char *sec_name;
@@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj, obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;return 0 ; } + +if (strcmp (src_sec->sec_name,JUMPTABLES_SEC ) ==0 ) + goto add_sym; }if (sym_bind ==STB_LOCAL )
@@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; - }else { -pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" ); + }else if (strcmp (src_sec->sec_name,JUMPTABLES_REL_SEC ) !=0 ) { +pr_warn ("relocation against STT_SECTION in section %s is not supported!\n" , + src_sec->sec_name);return -EINVAL ; } }
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 10 lines
--- tools/lib/bpf/libbpf.c | 240 +++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf_probes.c | 4 + tools/lib/bpf/linker.c | 10 +- 3 files changed, 251 insertions(+), 3 deletions(-)tools/lib/bpf/libbpf.c +0
-0
[...]
tools/lib/bpf/libbpf.c static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, · line 6326
Quoted 12 lines
tools/lib/bpf/libbpf.c +12
-0+/* +* In LLVM the .jumptables section contains jump tables entries relative to the +* section start. The BPF kernel-side code expects jump table offsets relative +* to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper +* computes a delta to be added when creating a map. +*/ +static int jt_adjust_off (struct bpf_program *prog,int insn_idx) +{ +int i; + +for (i = prog->subprog_cnt -1 ; i >=0 ; i--) { +if (insn_idx >= prog->subprogs[i].sub_insn_off)
Sorry, I'm still confused about what happens here.
The insn_idx is comes from relocation, meaning that it is a value recorded relative to section start, right? On the other hand, .sub_insn_off is an offset of a subprogram within a concatenated program, about to be loaded. These values should not be compared directly.
I think, that my suggestion from v5 [1] should be easier to understand:
Quoted 8 lines
Or rename this thing to
find_subprog_idx(), pass relo object intocreate_jt_map(), callfind_subprog_idx()there, and do the following:xlated_off = jt[i] / sizeof(struct bpf_insn); /* make xlated_off relative to subprogram start */ xlated_off -= prog->subprogs[subprog_idx].sec_insn_off; /* make xlated_off relative to main subprogram start */ xlated_off += prog->subprogs[subprog_idx].sub_insn_off;
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 32 lines
tools/lib/bpf/libbpf.c +22
-0+return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off; + } + +return -prog->sec_insn_off; +} + +/* Relocate data references within program code: * - map references; * - global variable references; @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)case RELO_CORE :/* will be handled by bpf_program_record_relos() */ break ; +case RELO_INSN_ARRAY : { +int map_fd; + + map_fd =create_jt_map (obj, prog, relo->sym_off, relo->sym_size, +jt_adjust_off (prog, relo->insn_idx)); +if (map_fd <0 ) { +pr_warn ("prog '%s': relo #%d: can't create jump table: sym_off %u\n" , + prog->name, i, relo->sym_off); +return map_fd; + } + insn[0 ].src_reg =BPF_PSEUDO_MAP_VALUE ; + insn->imm = map_fd; + insn->off =0 ; + } +break ;default :pr_warn ("prog '%s': relo #%d: bad relo type %d\n" , prog->name, i, relo->type);
[...]
tools/lib/bpf/libbpf.c void bpf_object__close(struct bpf_object *obj) · line 9463
Quoted 8 lines
tools/lib/bpf/libbpf.c +4
-0@@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)zfree (&obj->arena_data); +zfree (&obj->jumptables_data); + obj->jumptables_data_sz =0 ; + +if (obj->jumptable_maps && obj->jumptable_map_cnt) {
Nit: outer 'if' seems unnecessary.
Quoted 7 lines
tools/lib/bpf/libbpf.c +5
-0+for (i =0 ; i < obj->jumptable_map_cnt; i++) +close (obj->jumptable_maps[i].fd); + } +zfree (&obj->jumptable_maps); +free (obj); }
[...]
tools/lib/bpf/linker.c line 31
Quoted 10 lines
tools/lib/bpf/linker.c +2
-0@@ -27,6 +27,8 @@#include "strset.h" #define BTF_EXTERN_SEC ".extern" +#define JUMPTABLES_SEC ".jumptables" +#define JUMPTABLES_REL_SEC ".rel.jumptables"
Nit: maybe avoid duplicating JUMPTABLES_SEC by moving all *_SEC macro
to libbpf_internal.h?
Quoted 14 lines
tools/lib/bpf/linker.c +3
-0struct src_sec {const char *sec_name;@@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj, obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;return 0 ; } + +if (strcmp (src_sec->sec_name,JUMPTABLES_SEC ) ==0 ) + goto add_sym; }if (sym_bind ==STB_LOCAL )
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2281
Quoted 8 lines
tools/lib/bpf/linker.c +3
-2@@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; - }else { -pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" ); + }else if (strcmp (src_sec->sec_name,JUMPTABLES_REL_SEC ) !=0 ) { +pr_warn ("relocation against STT_SECTION in section %s is not supported!\n" , + src_sec->sec_name);
Sorry, I missed this on a previous iteration.
LLVM generates section relative offsets for jump table contents, so it seems that relocations inside jump table section should not occur. Is this a leftover, or am I confused?
Quoted 3 lines
tools/lib/bpf/linker.c +0
-0return -EINVAL ; } }
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps
On Tue, 2025-10-21 at 15:18 -0700, Eduard Zingerman wrote:
[...]
Quoted 11 lines
Quoted 11 lines
tools/lib/bpf/libbpf.c +11
-0+/* +* In LLVM the .jumptables section contains jump tables entries relative to the +* section start. The BPF kernel-side code expects jump table offsets relative +* to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper +* computes a delta to be added when creating a map. +*/ +static int jt_adjust_off (struct bpf_program *prog,int insn_idx) +{ +int i; + +for (i = prog->subprog_cnt -1 ; i >=0 ; i--) {
Quoted 8 lines
tools/lib/bpf/libbpf.c static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, · line 6326
Quoted 1 line
tools/lib/bpf/libbpf.c +1
-0+if (insn_idx >= prog->subprogs[i].sub_insn_off)Sorry, I'm still confused about what happens here.
Theinsn_idxis comes from relocation, meaning that it is a value recorded relative to section start, right? On the other hand,.sub_insn_offis an offset of a subprogram within a concatenated program, about to be loaded. These values should not be compared directly.
I'm wrong on this account, append_subprog_relos() adjusts relo->insn_idx. Still, please consider refactoring as below.
Quoted 12 lines
I think, that my suggestion from v5 [1] should be easier to understand:
Quoted 8 lines
Or rename this thing to
find_subprog_idx(), pass relo object intocreate_jt_map(), callfind_subprog_idx()there, and do the following:xlated_off = jt[i] / sizeof(struct bpf_insn); /* make xlated_off relative to subprogram start */ xlated_off -= prog->subprogs[subprog_idx].sec_insn_off; /* make xlated_off relative to main subprogram start */ xlated_off += prog->subprogs[subprog_idx].sub_insn_off;
[...]
Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps
In reply to Eduard ZingermanQuoted 37 lines · Eduard Zingerman
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 10 lines
--- tools/lib/bpf/libbpf.c | 240 +++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf_probes.c | 4 + tools/lib/bpf/linker.c | 10 +- 3 files changed, 251 insertions(+), 3 deletions(-)tools/lib/bpf/libbpf.c +0
-0[...]
tools/lib/bpf/libbpf.c static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, · line 6326
Quoted 12 lines
tools/lib/bpf/libbpf.c +12
-0+/* +* In LLVM the .jumptables section contains jump tables entries relative to the +* section start. The BPF kernel-side code expects jump table offsets relative +* to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper +* computes a delta to be added when creating a map. +*/ +static int jt_adjust_off (struct bpf_program *prog,int insn_idx) +{ +int i; + +for (i = prog->subprog_cnt -1 ; i >=0 ; i--) { +if (insn_idx >= prog->subprogs[i].sub_insn_off)Sorry, I'm still confused about what happens here.
Theinsn_idxis comes from relocation, meaning that it is a value recorded relative to section start, right? On the other hand,.sub_insn_offis an offset of a subprogram within a concatenated program, about to be loaded. These values should not be compared directly.
Quoted 1 line
I think, that my suggestion from v5 [1] should be easier to understand:
Well, if you insist :) (I saw the next e-mail as well, thanks.)
Quoted 48 lines
Quoted 8 lines
Or rename this thing to
find_subprog_idx(), pass relo object intocreate_jt_map(), callfind_subprog_idx()there, and do the following:xlated_off = jt[i] / sizeof(struct bpf_insn); /* make xlated_off relative to subprogram start */ xlated_off -= prog->subprogs[subprog_idx].sec_insn_off; /* make xlated_off relative to main subprogram start */ xlated_off += prog->subprogs[subprog_idx].sub_insn_off;[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 32 lines
tools/lib/bpf/libbpf.c +22
-0+return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off; + } + +return -prog->sec_insn_off; +} + +/* Relocate data references within program code: * - map references; * - global variable references; @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)case RELO_CORE :/* will be handled by bpf_program_record_relos() */ break ; +case RELO_INSN_ARRAY : { +int map_fd; + + map_fd =create_jt_map (obj, prog, relo->sym_off, relo->sym_size, +jt_adjust_off (prog, relo->insn_idx)); +if (map_fd <0 ) { +pr_warn ("prog '%s': relo #%d: can't create jump table: sym_off %u\n" , + prog->name, i, relo->sym_off); +return map_fd; + } + insn[0 ].src_reg =BPF_PSEUDO_MAP_VALUE ; + insn->imm = map_fd; + insn->off =0 ; + } +break ;default :pr_warn ("prog '%s': relo #%d: bad relo type %d\n" , prog->name, i, relo->type);[...]
Quoted 2 lines
tools/lib/bpf/libbpf.c +0
-0@@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)
Quoted 8 lines
tools/lib/bpf/libbpf.c void bpf_object__close(struct bpf_object *obj) · line 9463
Quoted 6 lines
tools/lib/bpf/libbpf.c +4
-0zfree (&obj->arena_data); +zfree (&obj->jumptables_data); + obj->jumptables_data_sz =0 ; + +if (obj->jumptable_maps && obj->jumptable_map_cnt) {Nit: outer 'if' seems unnecessary.
I suspect this was a check for if obj->jumptable_maps is null or not. I think this should never happen that jumptable_map_cnt && !jumptable_map, so I will remove the if.
Quoted 17 lines
Quoted 7 lines
tools/lib/bpf/libbpf.c +5
-0+for (i =0 ; i < obj->jumptable_map_cnt; i++) +close (obj->jumptable_maps[i].fd); + } +zfree (&obj->jumptable_maps); +free (obj); }[...]
Quoted 7 lines
tools/lib/bpf/linker.c +0
-0@@ -27,6 +27,8 @@#include "strset.h"
Quoted 6 lines
tools/lib/bpf/linker.c line 31
Quoted 3 lines
tools/lib/bpf/linker.c +2
-0#define BTF_EXTERN_SEC ".extern" +#define JUMPTABLES_SEC ".jumptables" +#define JUMPTABLES_REL_SEC ".rel.jumptables"Nit: maybe avoid duplicating
JUMPTABLES_SECby moving all*_SECmacro
tolibbpf_internal.h?
Yes, ok.
Quoted 12 lines
Quoted 12 lines
tools/lib/bpf/linker.c +3
-0struct src_sec {const char *sec_name;@@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj, obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;return 0 ; } + +if (strcmp (src_sec->sec_name,JUMPTABLES_SEC ) ==0 ) + goto add_sym; }
Quoted 15 lines
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2281
Quoted 10 lines
tools/lib/bpf/linker.c +3
-2if (sym_bind ==STB_LOCAL )@@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; - }else { -pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" ); + }else if (strcmp (src_sec->sec_name,JUMPTABLES_REL_SEC ) !=0 ) { +pr_warn ("relocation against STT_SECTION in section %s is not supported!\n" , + src_sec->sec_name);Sorry, I missed this on a previous iteration.
LLVM generates section relative offsets for jump table contents, so it seems that relocations inside jump table section should not occur. Is this a leftover, or am I confused?
I think this is a leftover in LLVM, so I have to keep it here. I will check again with the latest LLVM.
Quoted 3 lines
Quoted 3 lines
tools/lib/bpf/linker.c +0
-0return -EINVAL ; } }
Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps
Quoted 157 lines · Anton Protopopov
Quoted 37 lines · Eduard Zingerman
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 10 lines
--- tools/lib/bpf/libbpf.c | 240 +++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf_probes.c | 4 + tools/lib/bpf/linker.c | 10 +- 3 files changed, 251 insertions(+), 3 deletions(-)tools/lib/bpf/libbpf.c +0
-0[...]
tools/lib/bpf/libbpf.c static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, · line 6326
Quoted 12 lines
tools/lib/bpf/libbpf.c +12
-0+/* +* In LLVM the .jumptables section contains jump tables entries relative to the +* section start. The BPF kernel-side code expects jump table offsets relative +* to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper +* computes a delta to be added when creating a map. +*/ +static int jt_adjust_off (struct bpf_program *prog,int insn_idx) +{ +int i; + +for (i = prog->subprog_cnt -1 ; i >=0 ; i--) { +if (insn_idx >= prog->subprogs[i].sub_insn_off)Sorry, I'm still confused about what happens here.
Theinsn_idxis comes from relocation, meaning that it is a value recorded relative to section start, right? On the other hand,.sub_insn_offis an offset of a subprogram within a concatenated program, about to be loaded. These values should not be compared directly.Quoted 1 line
I think, that my suggestion from v5 [1] should be easier to understand:
Well, if you insist :) (I saw the next e-mail as well, thanks.)
Quoted 48 lines
Quoted 8 lines
Or rename this thing to
find_subprog_idx(), pass relo object intocreate_jt_map(), callfind_subprog_idx()there, and do the following:xlated_off = jt[i] / sizeof(struct bpf_insn); /* make xlated_off relative to subprogram start */ xlated_off -= prog->subprogs[subprog_idx].sec_insn_off; /* make xlated_off relative to main subprogram start */ xlated_off += prog->subprogs[subprog_idx].sub_insn_off;[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 32 lines
tools/lib/bpf/libbpf.c +22
-0+return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off; + } + +return -prog->sec_insn_off; +} + +/* Relocate data references within program code: * - map references; * - global variable references; @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)case RELO_CORE :/* will be handled by bpf_program_record_relos() */ break ; +case RELO_INSN_ARRAY : { +int map_fd; + + map_fd =create_jt_map (obj, prog, relo->sym_off, relo->sym_size, +jt_adjust_off (prog, relo->insn_idx)); +if (map_fd <0 ) { +pr_warn ("prog '%s': relo #%d: can't create jump table: sym_off %u\n" , + prog->name, i, relo->sym_off); +return map_fd; + } + insn[0 ].src_reg =BPF_PSEUDO_MAP_VALUE ; + insn->imm = map_fd; + insn->off =0 ; + } +break ;default :pr_warn ("prog '%s': relo #%d: bad relo type %d\n" , prog->name, i, relo->type);[...]
Quoted 2 lines
tools/lib/bpf/libbpf.c +0
-0@@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)Quoted 8 lines
tools/lib/bpf/libbpf.c void bpf_object__close(struct bpf_object *obj) · line 9463
Quoted 6 lines
tools/lib/bpf/libbpf.c +4
-0zfree (&obj->arena_data); +zfree (&obj->jumptables_data); + obj->jumptables_data_sz =0 ; + +if (obj->jumptable_maps && obj->jumptable_map_cnt) {Nit: outer 'if' seems unnecessary.
I suspect this was a check for if
obj->jumptable_mapsis null or not. I think this should never happen thatjumptable_map_cnt&& !jumptable_map, so I will remove the if.Quoted 17 lines
Quoted 7 lines
tools/lib/bpf/libbpf.c +5
-0+for (i =0 ; i < obj->jumptable_map_cnt; i++) +close (obj->jumptable_maps[i].fd); + } +zfree (&obj->jumptable_maps); +free (obj); }[...]
Quoted 7 lines
tools/lib/bpf/linker.c +0
-0@@ -27,6 +27,8 @@#include "strset.h" Quoted 6 lines
tools/lib/bpf/linker.c line 31
Quoted 3 lines
tools/lib/bpf/linker.c +2
-0#define BTF_EXTERN_SEC ".extern" +#define JUMPTABLES_SEC ".jumptables" +#define JUMPTABLES_REL_SEC ".rel.jumptables"Nit: maybe avoid duplicating
JUMPTABLES_SECby moving all*_SECmacro
tolibbpf_internal.h?Yes, ok.
Quoted 12 lines
Quoted 12 lines
tools/lib/bpf/linker.c +3
-0struct src_sec {const char *sec_name;@@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj, obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;return 0 ; } + +if (strcmp (src_sec->sec_name,JUMPTABLES_SEC ) ==0 ) + goto add_sym; }Quoted 15 lines
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2281
Quoted 10 lines
tools/lib/bpf/linker.c +3
-2if (sym_bind ==STB_LOCAL )@@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; - }else { -pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" ); + }else if (strcmp (src_sec->sec_name,JUMPTABLES_REL_SEC ) !=0 ) { +pr_warn ("relocation against STT_SECTION in section %s is not supported!\n" , + src_sec->sec_name);Sorry, I missed this on a previous iteration.
LLVM generates section relative offsets for jump table contents, so it seems that relocations inside jump table section should not occur. Is this a leftover, or am I confused?
Quoted 2 lines
I think this is a leftover in LLVM, so I have to keep it here. I will check again with the latest LLVM.
Yes, the latest LLVM I've built (llvmorg-22-init-12400-g09eea2256e53) does generate it:
$ llvm-readelf -S bpf_gotox.bpf.o | grep jumptables
[ 8] .jumptables PROGBITS ...
[ 9] .rel.jumptables REL ...Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
tools/lib/bpf/linker.c +0
-0return -EINVAL ; } }
PATCH v6 15/17 bpftool: Recognize insn_array map type
In reply to Anton ProtopopovTeach bpftool to recognize instruction array map type.
--- tools/bpf/bpftool/Documentation/bpftool-map.rst | 3 ++- tools/bpf/bpftool/map.c | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-map.rst +2 -1
@@ -55,7 +55,8 @@ MAP COMMANDS | | **devmap** | **devmap_hash** | **sockmap** | **cpumap** | **xskmap** | **sockhash** | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** -| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** } +| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** +| | **insn_array** } DESCRIPTION ===========
tools/bpf/bpftool/map.c +2 -1
@@ -1477,7 +1477,8 @@ static int do_help(int argc, char **argv)" devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n" " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" -" task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena }\n" +" task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n" +" insn_array }\n" " " HELP_SPEC_OPTIONS " |\n" " {-f|--bpffs} | {-n|--nomount} }\n" "",
Signature · 1 line
2.34.1
PATCH v6 16/17 selftests/bpf: add new verifier_gotox test
In reply to Anton ProtopopovAdd a set of tests to validate core gotox functionality without need to rely on compilers.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_gotox.c | 277 ++++++++++++++++++ 2 files changed, 279 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -35,6 +35,7 @@#include "verifier_global_subprogs.skel.h" #include "verifier_global_ptr_args.skel.h" #include "verifier_gotol.skel.h" +#include "verifier_gotox.skel.h" #include "verifier_helper_access_var_len.skel.h" #include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h"
@@ -173,6 +174,7 @@ void test_verifier_div_overflow(void) { RUN(verifier_div_overflow); }void test_verifier_global_subprogs (void ) {RUN (verifier_global_subprogs); }void test_verifier_global_ptr_args (void ) {RUN (verifier_global_ptr_args); }void test_verifier_gotol (void ) {RUN (verifier_gotol); } +void test_verifier_gotox (void ) {RUN (verifier_gotox); }void test_verifier_helper_access_var_len (void ) {RUN (verifier_helper_access_var_len); }void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +277 -0
277 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test
Quoted 5 lines · Anton Protopopov
Add a set of tests to validate core gotox functionality without need to rely on compilers.
Signed-off-by Anton Protopopov <[email protected]>---
Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
- "the sum of R%u umin_value %llu and off %u is too big\n"
- "register R%d doesn't point to any offset in map id=%d\n" ?
Might be the case that some of these can't be triggered because of the check_mem_access() call.
[...]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +0
-0
[...]
tools/testing/selftests/bpf/progs/verifier_gotox.c line 55
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+/* +* Gotox is forbidden when there is no jump table loaded +* which points to the sub-function where the gotox is used +*/ +SEC("socket") +__failure __msg("no jump tables found for subprog starting at 0")
^^^^
Nit: one day we need to figure out a way to
report subprogram names, when reporting check_cfg() errors.
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+__naked void jump_table_no_jump_table (void ) +{ + asm volatile (" \ + .8byte %[gotox_r0]; \ + r0 = 1; \ + exit; \
tools/testing/selftests/bpf/progs/verifier_gotox.c line 71
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +10
-0+" : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) + : __clobber_all); +} + +/* +* Incorrect type of the target register, only PTR_TO_INSN allowed +*/ +SEC("socket") +__failure __msg("R1 has type 1, expected PTR_TO_INSN")
^^^^^^
log.c:reg_type_str() should help here.
Quoted 18 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +18
-0+__naked void jump_table_incorrect_dst_reg_type (void ) +{ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ +jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(u64 *)(r0 + 0); \ + r1 = 42; \ + .8byte %[gotox_r1]; \ + ret0_%=: \ + r0 = 0; \
tools/testing/selftests/bpf/progs/verifier_gotox.c line 99
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +10
-0+exit ; \ + ret1_%=: \ + r0 =1 ; \ + exit; \ +" : \ + : __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0)) + : __clobber_all); +} + +#define DEFINE_INVALID_SIZE_PROG (READ_SIZE ,OUTCOME ) \
Nit: this can be merged with DEFINE_SIMPLE_JUMP_TABLE_PROG.
Quoted 32 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +32
-0+ \ + SEC("socket") \ + OUTCOME \ + __naked void jump_table_invalid_read_size_ ## READ_SIZE(void) \ + { \ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ + jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(" #READ_SIZE " *)(r0 + 0); \ + .8byte %[gotox_r0]; \ + ret0_%=: \ + r0 = 0; \ + exit; \ + ret1_%=: \ + r0 = 1; \ + exit; \ + " : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \ + : __clobber_all); \ + } + +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u8, __failure __msg("Invalid read of 1 bytes from insn_array"))
[...]
Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test
Quoted 4 lines · Eduard Zingerman
Quoted 3 lines · Anton Protopopov
Add a set of tests to validate core gotox functionality without need to rely on compilers.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
- "the sum of R%uumin_value%llu and off %u is too big\n"
- "register R%d doesn't point to any offset in map id=%d\n"
Yeah, sorry, these actually were on my list, but I've postponed them for the next version. Will add. (I also need to add a few selftests on the offset when loading from map.)
Quoted 14 lines
Might be the case that some of these can't be triggered because of the
check_mem_access()call.[...]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +0
-0[...]
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 55
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+/* +* Gotox is forbidden when there is no jump table loaded +* which points to the sub-function where the gotox is used +*/ +SEC("socket") +__failure __msg("no jump tables found for subprog starting at 0")^^^^
Nit: one day we need to figure out a way to
report subprogram names, when reportingcheck_cfg()errors.
But those are not always present, right?
Quoted 11 lines
Quoted 11 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +10
-0+__naked void jump_table_no_jump_table (void ) +{ + asmvolatile (" \ +.8byte %[gotox_r0]; \ +r0 = 1; \ +exit; \ +" : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) + : __clobber_all); +}
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 71
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+ +/* +* Incorrect type of the target register, only PTR_TO_INSN allowed +*/ +SEC("socket") +__failure __msg("R1 has type 1, expected PTR_TO_INSN")^^^^^^
log.c:reg_type_str() should help here.
Yes, thanks, this was changed to address your comment in the other patch.
Quoted 22 lines
Quoted 22 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +22
-0+__naked void jump_table_incorrect_dst_reg_type (void ) +{ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ +jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(u64 *)(r0 + 0); \ + r1 = 42; \ + .8byte %[gotox_r1]; \ + ret0_%=: \ + r0 = 0; \ + exit; \ + ret1_%=: \ + r0 = 1; \ + exit; \
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 99
Quoted 6 lines
+" : \
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +5
-0+ : __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0)) + : __clobber_all); +} + +#define DEFINE_INVALID_SIZE_PROG (READ_SIZE ,OUTCOME ) \Nit: this can be merged with
DEFINE_SIMPLE_JUMP_TABLE_PROG.
Didn't want to overload the macro too much so the prog stays readable. (Here are two different regs are used.) I will check how it looks like if I merge them, and merge, if it looks ok-ish.
Quoted 34 lines
Quoted 32 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +32
-0+ \ + SEC("socket") \ + OUTCOME \ + __naked void jump_table_invalid_read_size_ ## READ_SIZE(void) \ + { \ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ + jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(" #READ_SIZE " *)(r0 + 0); \ + .8byte %[gotox_r0]; \ + ret0_%=: \ + r0 = 0; \ + exit; \ + ret1_%=: \ + r0 = 1; \ + exit; \ + " : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \ + : __clobber_all); \ + } + +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u8, __failure __msg("Invalid read of 1 bytes from insn_array"))[...]
Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test
Quoted 14 lines · Anton Protopopov
Quoted 4 lines · Eduard Zingerman
Quoted 3 lines · Anton Protopopov
Add a set of tests to validate core gotox functionality without need to rely on compilers.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
- "the sum of R%uumin_value%llu and off %u is too big\n"
- "register R%d doesn't point to any offset in map id=%d\n"
Quoted 3 lines
Yeah, sorry, these actually were on my list, but I've postponed them for the next version. Will add. (I also need to add a few selftests on the offset when loading from map.)
So, tbh, I can't actually find a way to trigger any of them, looks like these conditions are always caught earlier...
Quoted 119 lines
Quoted 14 lines
Might be the case that some of these can't be triggered because of the
check_mem_access()call.[...]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +0
-0[...]
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 55
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+/* +* Gotox is forbidden when there is no jump table loaded +* which points to the sub-function where the gotox is used +*/ +SEC("socket") +__failure __msg("no jump tables found for subprog starting at 0")^^^^
Nit: one day we need to figure out a way to
report subprogram names, when reportingcheck_cfg()errors.But those are not always present, right?
Quoted 11 lines
Quoted 11 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +10
-0+__naked void jump_table_no_jump_table (void ) +{ + asmvolatile (" \ +.8byte %[gotox_r0]; \ +r0 = 1; \ +exit; \ +" : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) + : __clobber_all); +}Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 71
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +6
-0+ +/* +* Incorrect type of the target register, only PTR_TO_INSN allowed +*/ +SEC("socket") +__failure __msg("R1 has type 1, expected PTR_TO_INSN")^^^^^^
log.c:reg_type_str() should help here.Yes, thanks, this was changed to address your comment in the other patch.
Quoted 22 lines
Quoted 22 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +22
-0+__naked void jump_table_incorrect_dst_reg_type (void ) +{ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ +jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(u64 *)(r0 + 0); \ + r1 = 42; \ + .8byte %[gotox_r1]; \ + ret0_%=: \ + r0 = 0; \ + exit; \ + ret1_%=: \ + r0 = 1; \ + exit; \Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c line 99
Quoted 6 lines
+" : \
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +5
-0+ : __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0)) + : __clobber_all); +} + +#define DEFINE_INVALID_SIZE_PROG (READ_SIZE ,OUTCOME ) \Nit: this can be merged with
DEFINE_SIMPLE_JUMP_TABLE_PROG.Didn't want to overload the macro too much so the prog stays readable. (Here are two different regs are used.) I will check how it looks like if I merge them, and merge, if it looks ok-ish.
Quoted 34 lines
Quoted 32 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c new file +32
-0+ \ + SEC("socket") \ + OUTCOME \ + __naked void jump_table_invalid_read_size_ ## READ_SIZE(void) \ + { \ + asm volatile (" \ + .pushsection .jumptables,\"\",@progbits; \ + jt0_%=: \ + .quad ret0_%=; \ + .quad ret1_%=; \ + .size jt0_%=, 16; \ + .global jt0_%=; \ + .popsection; \ + \ + r0 = jt0_%= ll; \ + r0 += 8; \ + r0 = *(" #READ_SIZE " *)(r0 + 0); \ + .8byte %[gotox_r0]; \ + ret0_%=: \ + r0 = 0; \ + exit; \ + ret1_%=: \ + r0 = 1; \ + exit; \ + " : \ + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \ + : __clobber_all); \ + } + +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array")) +DEFINE_INVALID_SIZE_PROG(u8, __failure __msg("Invalid read of 1 bytes from insn_array"))[...]
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test
Quoted 6 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Eduard Zingerman
Quoted 3 lines · Anton Protopopov
Add a set of tests to validate core gotox functionality without need to rely on compilers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
I'm surprised this one can't be triggered.
In case if there are multiple subprograms defined,
and map has two entries pointing to different subprograms, what other checks can catch this?
I see only the check in the bpf_insn_array_init() that will make sure that offset does not point outside of the program.
Quoted 1 line
Quoted 1 line
Quoted 1 line
- "the sum of R%u
umin_value%llu and off %u is too big\n"
Ok, this one is handled by map value read.
Quoted 1 line
Quoted 1 line
Quoted 1 line
- "register R%d doesn't point to any offset in map id=%d\n"
And this one as well.
Quoted 6 lines
Quoted 3 lines
Yeah, sorry, these actually were on my list, but I've postponed them for the next version. Will add. (I also need to add a few selftests on the offset when loading from map.)
So, tbh, I can't actually find a way to trigger any of them, looks like these conditions are always caught earlier...
Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test
Quoted 14 lines · Eduard Zingerman
Quoted 6 lines · Anton Protopopov
Quoted 5 lines · Anton Protopopov
Quoted 4 lines · Eduard Zingerman
Quoted 3 lines · Anton Protopopov
Add a set of tests to validate core gotox functionality without need to rely on compilers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Signed-off-by Anton Protopopov <[email protected]>---
Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
Quoted 6 lines
I'm surprised this one can't be triggered.
In case if there are multiple subprograms defined,
and map has two entries pointing to different subprograms, what other checks can catch this?
I see only the check in thebpf_insn_array_init()that will make sure that offset does not point outside of the program.
Yes, you were correct. I will add a test.
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
- "the sum of R%u
umin_value%llu and off %u is too big\n"Ok, this one is handled by map value read.
Quoted 1 line
Quoted 1 line
Quoted 1 line
- "register R%d doesn't point to any offset in map id=%d\n"
And this one as well.
Quoted 6 lines
Quoted 3 lines
Yeah, sorry, these actually were on my list, but I've postponed them for the next version. Will add. (I also need to add a few selftests on the offset when loading from map.)
So, tbh, I can't actually find a way to trigger any of them, looks like these conditions are always caught earlier...
PATCH v6 17/17 selftests/bpf: add C-level selftests for indirect jumps
In reply to Anton ProtopopovAdd C-level selftests for indirect jumps to validate LLVM and libbpf functionality. The tests are intentionally disabled, to be run locally by developers, but will not make the CI red.
--- tools/testing/selftests/bpf/Makefile | 4 +- .../selftests/bpf/prog_tests/bpf_gotox.c | 185 ++++++++ tools/testing/selftests/bpf/progs/bpf_gotox.c | 414 ++++++++++++++++++ 3 files changed, 602 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c
tools/testing/selftests/bpf/Makefile +3 -1
@@ -454,7 +454,9 @@ BPF_CFLAGS = -g -Wall -Werror -D__TARGET_ARCH_$(SRCARCH) $(MENDIAN) \ -I$(abspath $(OUTPUT)/../usr/include) \ -std=gnu11 \ -fno-strict-aliasing \ - -Wno-compare-distinct-pointer-types + -Wno-compare-distinct-pointer-types \ + -Wno-initializer-overrides \ + # # TODO: enable me -Wsign-compare CLANG_CFLAGS = $(CLANG_SYS_INCLUDES)
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +185 -0
185 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bpf_gotox.c new file +414 -0
414 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +9
-0@@ -0,0 +1,185 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <test_progs.h> + +#include <linux/if_ether.h> +#include <linux/in.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <linux/in6.h>
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +10
-0+#include <linux/udp.h> +#include <linux/tcp.h> + +#include <sys/syscall.h> +#include <bpf/bpf.h> + +#include "bpf_gotox.skel.h" + +/* Disable tests for now, as CI runs with LLVM-20 */ +#if 0
I removed all the #if 0 and tried to compile this test using LLVM at commit 1a9aba29b09e ("[RISCV] Remove unreachable break statements. NFC (#164481)").
LLVM refuses to compile showing errors like below:
CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
<U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:8
<U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8
<U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8
<U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8
<U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18
<U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16
<U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5In function: big_jump_table
CC'ing Yonghong.
[...]
Re: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps
Quoted 44 lines · Eduard Zingerman
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +9
-0@@ -0,0 +1,185 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <test_progs.h> + +#include <linux/if_ether.h> +#include <linux/in.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <linux/in6.h> tools/testing/selftests/bpf/prog_tests/bpf_gotox.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +10
-0+#include <linux/udp.h> +#include <linux/tcp.h> + +#include <sys/syscall.h> +#include <bpf/bpf.h> + +#include "bpf_gotox.skel.h" + +/* Disable tests for now, as CI runs with LLVM-20 */ +#if 0 I removed all the
#if 0and tried to compile this test using LLVM at commit1a9aba29b09e("[RISCV] Remove unreachable break statements. NFC (#164481)").LLVM refuses to compile showing errors like below:
CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
<U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64,progs/bpf_gotox.c:202:8<U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8 <U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8 <U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8 <U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18 <U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16 <U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5
Quoted 3 lines
In function:
big_jump_tableCC'ing Yonghong.
Looks like no -mcpu=v4. I had to hack Makefile (set v3 -> v4) to build it.
Even when one tries to build test_progs-cpuv4, the v3 version is also built. In the bpf prog this is easy to ifdef it: x86_64 + __clang + one can define a variable like BPF_CPU=4 in Makefile. But the prog_tests can't easily see was the skeleton compiled with v3 or v4, so I just ended up with putting #if 0 there for now...
Quoted 1 line
[...]
Re: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps
Quoted 49 lines · Anton Protopopov
Quoted 44 lines · Eduard Zingerman
On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
[...]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +9
-0@@ -0,0 +1,185 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <test_progs.h> + +#include <linux/if_ether.h> +#include <linux/in.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <linux/in6.h> tools/testing/selftests/bpf/prog_tests/bpf_gotox.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/bpf_gotox.c new file +10
-0+#include <linux/udp.h> +#include <linux/tcp.h> + +#include <sys/syscall.h> +#include <bpf/bpf.h> + +#include "bpf_gotox.skel.h" + +/* Disable tests for now, as CI runs with LLVM-20 */ +#if 0 I removed all the
#if 0and tried to compile this test using LLVM at commit1a9aba29b09e("[RISCV] Remove unreachable break statements. NFC (#164481)").LLVM refuses to compile showing errors like below:
CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
<U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64,progs/bpf_gotox.c:202:8<U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8 <U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8 <U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8 <U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18 <U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16 <U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5Quoted 3 lines
In function:
big_jump_tableCC'ing Yonghong.
Quoted 7 lines
Looks like no -mcpu=v4. I had to hack
Makefile(set v3 -> v4) to build it.Even when one tries to build test_progs-cpuv4, the v3 version is also built. In the bpf prog this is easy to ifdef it:
x86_64+__clang+ one can define a variable like BPF_CPU=4 inMakefile. But theprog_testscan't easily see was the skeleton compiled with v3 or v4, so I just ended up with putting #if 0 there for now...
Yes, checked with the latest LLVM that this is "v3 vs. v4". If you pass -mcpu=v4 the code will compile fine.
Quoted 1 line
Quoted 1 line
[...]
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 8 lines · On Sun, 19 Oct 2025 20:21:28 +0000 you wrote:
This patchset implements a new type of map, instruction set, and uses it to build support for indirect branches in BPF (on x86). (The same map will be later used to provide support for indirect calls and static keys.) See [1], [2] for more context.
Short table of contents:
[...]
Here is the summary with links:
- [v6,bpf-next,01/17] bpf: fix the return value of push_stack
https://git.kernel.org/bpf/bpf-next/c/6ea5fc92a0fc - [v6,bpf-next,02/17] bpf: save the start of functions in bpf_prog_aux https://git.kernel.org/bpf/bpf-next/c/f7d72d0b3f43 - [v6,bpf-next,03/17] bpf: generalize and export map_get_next_key for arrays https://git.kernel.org/bpf/bpf-next/c/44481e492532 - [v6,bpf-next,04/17] bpf, x86: add new map type: instructions array (no matching commit) - [v6,bpf-next,05/17] selftests/bpf: add selftests for new insn_array map (no matching commit) - [v6,bpf-next,06/17] bpf: support instructions arrays with constants blinding (no matching commit) - [v6,bpf-next,07/17] selftests/bpf: test instructions arrays with blinding (no matching commit) - [v6,bpf-next,08/17] bpf, x86: allow indirect jumps to r8...r15 (no matching commit) - [v6,bpf-next,09/17] bpf: make bpf_insn_successors to return a pointer https://git.kernel.org/bpf/bpf-next/c/2f69c5685427 - [v6,bpf-next,10/17] bpf, x86: add support for indirect jumps (no matching commit) - [v6,bpf-next,11/17] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X (no matching commit) - [v6,bpf-next,12/17] bpf, docs: do not state that indirect jumps are not supported (no matching commit) - [v6,bpf-next,13/17] libbpf: fix formatting of bpf_object__append_subprog_code https://git.kernel.org/bpf/bpf-next/c/e7586577b75f
- [v6,bpf-next,14/17] libbpf: support llvm-generated indirect jumps (no matching commit)
- [v6,bpf-next,15/17] bpftool: Recognize insn_array map type
(no matching commit)
- [v6,bpf-next,16/17] selftests/bpf: add new verifier_gotox test (no matching commit)
- [v6,bpf-next,17/17] selftests/bpf: add C-level selftests for indirect jumps (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html