bpf: introduce function calls
Subsystems (23)
Alexei Starovoitov
maintainercover bpf: introduce function calls
First of all huge thank you to Daniel, John, Jakub, Edward and others who reviewed multiple iterations of this patch set over the last many months and to Dave and others who gave critical feedback during netconf/netdev.
The patch is solid enough and we thought through numerous corner cases, but it's not the end. More followups with code reorg and features to follow.
TLDR: Allow arbitrary function calls from bpf function to another bpf function.
Since the beginning of bpf all bpf programs were represented as a single function and program authors were forced to use always_inline for all functions in their C code. That was causing llvm to unnecessary inflate the code size and forcing developers to move code to header files with little code reuse.
With a bit of additional complexity teach verifier to recognize arbitrary function calls from one bpf function to another as long as all of functions are presented to the verifier as a single bpf program. Extended program layout:
..
r1 = .. // arg1 r2 = .. // arg2 call pc+1 // function call pc-relative
exit
.. = r1 // access arg1
.. = r2 // access arg2
..
call pc+20 // second level of function call
...
It allows for better optimized code and finally allows to introduce the core bpf libraries that can be reused in different projects, since programs are no longer limited by single elf file. With function calls bpf can be compiled into multiple .o files.
This patch is the first step. It detects programs that contain multiple functions and checks that calls between them are valid. It splits the sequence of bpf instructions (one program) into a set of bpf functions that call each other. Calls to only known functions are allowed. Since all functions are presented to the verifier at once conceptually it is 'static linking'.
Future plans:
- introduce BPF_PROG_TYPE_LIBRARY and allow a set of bpf functions to be loaded into the kernel that can be later linked to other programs with concrete program types. Aka 'dynamic linking'.
- introduce function pointer type and indirect calls to allow bpf functions call other dynamically loaded bpf functions while the caller bpf function is already executing. Aka 'runtime linking'. This will be more generic and more flexible alternative to bpf_tail_calls.
FAQ:
Q: Interpreter and JIT changes mean that new instruction is introduced ?
A: No. The call instruction technically stays the same. Now it can call
both kernel helpers and other bpf functions.
Calling convention stays the same as well.
From uapi point of view the call insn got new 'relocation' BPF_PSEUDO_CALL similar to BPF_PSEUDO_MAP_FD 'relocation' of bpf_ldimm64 insn.
Q: What had to change on LLVM side?
A: Trivial LLVM patch to allow calls was applied to upcoming 6.0 release:
https://reviews.llvm.org/rL318614
with few bugfixes as well.
Make sure to build the latest llvm to have bpf_call support.
More details in the patches.
Alexei Starovoitov (12):
bpf: introduce function calls (function boundaries)
bpf: introduce function calls (verification) selftests/bpf: add verifier tests for bpf_call
bpf: teach verifier to recognize zero initialized stack selftests/bpf: add tests for stack_zero tracking
libbpf: add support for bpf_call selftests/bpf: add bpf_call test selftests/bpf: add xdp noinline test
bpf: add support for bpf_call to interpreter
bpf: fix net.core.bpf_jit_enable race
bpf: x64: add JIT support for multi-function programs
bpf: arm64: add JIT support for multi-function programs
Daniel Borkmann (1):
selftests/bpf: additional bpf_call tests
arch/arm/net/bpf_jit_32.c | 2 +- arch/arm64/net/bpf_jit_comp.c | 70 +- arch/mips/net/ebpf_jit.c | 2 +- arch/powerpc/net/bpf_jit_comp64.c | 2 +- arch/s390/net/bpf_jit_comp.c | 2 +- arch/sparc/net/bpf_jit_comp_64.c | 2 +- arch/x86/net/bpf_jit_comp.c | 49 +- include/linux/bpf.h | 4 + include/linux/bpf_verifier.h | 45 +- include/linux/filter.h | 13 +- include/uapi/linux/bpf.h | 6 + kernel/bpf/core.c | 104 +- kernel/bpf/disasm.c | 8 +- kernel/bpf/syscall.c | 3 +- kernel/bpf/verifier.c | 1120 ++++++++++++--- tools/include/uapi/linux/bpf.h | 6 + tools/lib/bpf/bpf.h | 2 +- tools/lib/bpf/libbpf.c | 170 ++- tools/testing/selftests/bpf/Makefile | 12 +- tools/testing/selftests/bpf/test_l4lb_noinline.c | 473 +++++++ tools/testing/selftests/bpf/test_progs.c | 95 +- tools/testing/selftests/bpf/test_verifier.c | 1624 +++++++++++++++++++++- tools/testing/selftests/bpf/test_xdp_noinline.c | 833 +++++++++++ 23 files changed, 4378 insertions(+), 269 deletions(-) create mode 100644 tools/testing/selftests/bpf/test_l4lb_noinline.c create mode 100644 tools/testing/selftests/bpf/test_xdp_noinline.c
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 1/13 bpf: introduce function calls (function boundaries)
From: Alexei Starovoitov <[email protected]>
Allow arbitrary function calls from bpf function to another bpf function.
Since the beginning of bpf all bpf programs were represented as a single function and program authors were forced to use always_inline for all functions in their C code. That was causing llvm to unnecessary inflate the code size and forcing developers to move code to header files with little code reuse.
With a bit of additional complexity teach verifier to recognize arbitrary function calls from one bpf function to another as long as all of functions are presented to the verifier as a single bpf program. New program layout:
r6 = r1 // some code
..
r1 = .. // arg1 r2 = .. // arg2 call pc+1 // function call pc-relative
exit
.. = r1 // access arg1
.. = r2 // access arg2
..
call pc+20 // second level of function call
...
It allows for better optimized code and finally allows to introduce the core bpf libraries that can be reused in different projects, since programs are no longer limited by single elf file. With function calls bpf can be compiled into multiple .o files.
This patch is the first step. It detects programs that contain multiple functions and checks that calls between them are valid. It splits the sequence of bpf instructions (one program) into a set of bpf functions that call each other. Calls to only known functions are allowed. In the future the verifier may allow calls to unresolved functions and will do dynamic linking. This logic supports statically linked bpf functions only.
Such function boundary detection could have been done as part of control flow graph building in check_cfg(), but it's cleaner to separate function boundary detection vs control flow checks within a subprogram (function) into logically indepedent steps. Follow up patches may split check_cfg() further, but not check_subprogs().
Only allow bpf-to-bpf calls for root only and for non-hw-offloaded programs. These restrictions can be relaxed in the future.
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- include/linux/bpf_verifier.h | 5 +- include/uapi/linux/bpf.h | 6 ++ kernel/bpf/disasm.c | 8 ++- kernel/bpf/verifier.c | 141 ++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 155 insertions(+), 5 deletions(-)
include/linux/bpf_verifier.h +4 -1
@@ -141,6 +141,8 @@ struct bpf_ext_analyzer_ops { int insn_idx, int prev_insn_idx); }; +#define BPF_MAX_SUBPROGS 256 +/* single container for all structs * one verifier_env per bpf_check() call */
@@ -159,8 +161,9 @@ struct bpf_verifier_env {bool allow_ptr_leaks;bool seen_direct_write;struct bpf_insn_aux_data *insn_aux_data;/* array of per-insn state */ -struct bpf_verifer_log log; +u32 subprog_starts[BPF_MAX_SUBPROGS ]; +u32 subprog_cnt; };static inline struct bpf_reg_state *cur_regs (struct bpf_verifier_env *env)
include/uapi/linux/bpf.h +6 -0
@@ -197,8 +197,14 @@ enum bpf_attach_type {*/ #define BPF_F_STRICT_ALIGNMENT (1U << 0) +/* when bpf_ldimm64->src_reg == BPF_PSEUDO_MAP_FD, bpf_ldimm64->imm == fd */ #define BPF_PSEUDO_MAP_FD 1 +/* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative +* offset to another bpf function +*/ +#define BPF_PSEUDO_CALL 1 +/* flags for BPF_MAP_UPDATE_ELEM command */ #define BPF_ANY 0/* create new element or update existing */ #define BPF_NOEXIST 1/* create new element if it didn't exist */
kernel/bpf/disasm.c +6 -2
@@ -189,8 +189,12 @@ void print_bpf_insn(bpf_insn_print_cb verbose, struct bpf_verifier_env *env,u8 opcode =BPF_OP (insn->code);if (opcode ==BPF_CALL ) { -verbose (env,"(%02x) call %s#%d\n" , insn->code, -func_id_name (insn->imm), insn->imm); +if (insn->src_reg ==BPF_PSEUDO_CALL ) +verbose (env,"(%02x) call pc%+d\n" , insn->code, + insn->imm); +else +verbose (env,"(%02x) call %s#%d\n" , insn->code, +func_id_name (insn->imm), insn->imm); }else if (insn->code == (BPF_JMP |BPF_JA )) {verbose (env,"(%02x) goto pc%+d\n" , insn->code, insn->off);
kernel/bpf/verifier.c +139 -2
193 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 2/13 bpf: introduce function calls (verification)
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
Allow arbitrary function calls from bpf function to another bpf function.
To recognize such set of bpf functions the verifier does:
1. runs control flow analysis to detect function boundaries
2. proceeds with verification of all functions starting from main(root) function It recognizes that the stack of the caller can be accessed by the callee (if the caller passed a pointer to its stack to the callee) and the callee can store map_value and other pointers into the stack of the caller.
3. keeps track of the stack_depth of each function to make sure that total stack depth is still less than 512 bytes
4. disallows pointers to the callee stack to be stored into the caller stack, since they will be invalid as soon as the callee returns
5. to reuse all of the existing state_pruning logic each function call is considered to be independent call from the verifier point of view. The verifier pretends to inline all function calls it sees are being called. It stores the callsite instruction index as part of the state to make sure that two calls to the same callee from two different places in the caller will be different from state pruning point of view
6. more safety checks are added to liveness analysis
Implementation details:
. struct bpf_verifier_state is now consists of all stack frames that
led to this function
. struct bpf_func_state represent one stack frame. It consists of
registers in the given frame and its stack
. propagate_liveness() logic had a premature optimization where
mark_reg_read() and mark_stack_slot_read() were manually inlined with loop iterating over parents for each register or stack slot. Undo this optimization to reuse more complex mark_*_read() logic
. skip_callee() logic is not necessary from safety point of view,
but without it mark_*_read() markings become too conservative, since after returning from the funciton call a read of r6-r9 will incorrectly propagate the read marks into callee causing inefficient pruning later
. mark_*_read() logic is now aware of control flow which makes it
more complex. In the future the plan is to rewrite liveness to be hierarchical. So that liveness can be done within basic block only and control flow will be responsible for propagation of liveness information along cfg and between calls.
. tail_calls and ld_abs insns are not allowed in the programs with
bpf-to-bpf calls
. returning stack pointers to the caller or storing them into stack
frame of the caller is not allowed
Testing:
. no difference in cilium processed_insn numbers
. large number of tests follows in next patches
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by John Fastabend <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- include/linux/bpf_verifier.h | 36 ++- kernel/bpf/verifier.c | 702 +++++++++++++++++++++++++++++++++---------- 2 files changed, 585 insertions(+), 153 deletions(-)
include/linux/bpf_verifier.h +34 -2
@@ -76,6 +76,14 @@ struct bpf_reg_state {s64 smax_value;/* maximum possible (s64)value */ u64 umin_value;/* minimum possible (u64)value */ u64 umax_value;/* maximum possible (u64)value */ +/* Inside the callee two registers can be both PTR_TO_STACK like +* R1=fp-8 and R2=fp-8, but one of them points to this function stack +* while another to the caller's stack. To differentiate them 'frameno' +* is used which is an index in bpf_verifier_state->frame[] array +* pointing to bpf_func_state. +* This field must be second to last, for states_equal() reasons. +*/ +u32 frameno;/* This field must be last, for states_equal() reasons. */ enum bpf_reg_liveness live; };
@@ -96,13 +104,34 @@ struct bpf_stack_state {/* state of the program: * type of all registers and stack info */ -struct bpf_verifier_state { +struct bpf_func_state {struct bpf_reg_state regs[MAX_BPF_REG ];struct bpf_verifier_state *parent; +/* index of call instruction that called into this func */ +int callsite; +/* stack frame number of this function state from pov of +* enclosing bpf_verifier_state. +* 0 = main function, 1 = first callee. +*/ +u32 frameno; +/* subprog number == index within subprog_stack_depth +* zero == main subprog +*/ +u32 subprogno; + +/* should be second to last. See copy_func_state() */ int allocated_stack;struct bpf_stack_state *stack; }; +#define MAX_CALL_FRAMES 8 +struct bpf_verifier_state { +/* call stack tracking */ +struct bpf_func_state *frame[MAX_CALL_FRAMES ]; +struct bpf_verifier_state *parent; +u32 curframe; +}; +/* linked list of verifier states used to prune search */ struct bpf_verifier_state_list {struct bpf_verifier_state state;
@@ -163,12 +192,15 @@ struct bpf_verifier_env {struct bpf_insn_aux_data *insn_aux_data;/* array of per-insn state */ struct bpf_verifer_log log;u32 subprog_starts[BPF_MAX_SUBPROGS ]; +u16 subprog_stack_depth[BPF_MAX_SUBPROGS +1 ];u32 subprog_cnt; };static inline struct bpf_reg_state *cur_regs (struct bpf_verifier_env *env) { -return env->cur_state->regs; +struct bpf_verifier_state *cur = env->cur_state; + +return cur->frame[cur->curframe]->regs; }#if defined(CONFIG_NET ) && defined(CONFIG_BPF_SYSCALL )
kernel/bpf/verifier.c +551 -151
1,161 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 3/13 selftests/bpf: add verifier tests for bpf_call
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
Add extensive set of tests for bpf_call verification logic:
calls: basic sanity
calls: using r0 returned by callee
calls: callee is using r1
calls: callee using args1
calls: callee using wrong args2
calls: callee using two args
calls: callee changing pkt pointers
calls: two calls with args
calls: two calls with bad jump
calls: recursive call. test1
calls: recursive call. test2
calls: unreachable code
calls: invalid call
calls: jumping across function bodies. test1
calls: jumping across function bodies. test2
calls: call without exit
calls: call into middle of ld_imm64
calls: call into middle of other call
calls: two calls with bad fallthrough
calls: two calls with stack read
calls: two calls with stack write
calls: spill into caller stack frame
calls: two calls with stack write and void return
calls: ambiguous return value
calls: two calls that return map_value
calls: two calls that return map_value with bool condition
calls: two calls that return map_value with incorrect bool check
calls: two calls that receive map_value via arg=ptr_stack_of_caller. test1
calls: two calls that receive map_value via arg=ptr_stack_of_caller. test2
calls: two jumps that receive map_value via arg=ptr_stack_of_jumper. test3
calls: two calls that receive map_value_ptr_or_null via arg. test1
calls: two calls that receive map_value_ptr_or_null via arg. test2
calls: pkt_ptr spill into caller stack
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- tools/testing/selftests/bpf/test_verifier.c | 961 +++++++++++++++++++++++++++- 1 file changed, 960 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_verifier.c +960 -1
@@ -2,6 +2,7 @@* Testsuite for eBPF verifier * * Copyright (c) 2014 PLUMgrid, http://plumgrid.com +* Copyright (c) 2017 Facebook * * This program is free software; you can redistribute it and/or * modify it under the terms of version 2 of the GNU General Public
@@ -277,7 +278,7 @@ static struct bpf_test tests[] = { .insns = { BPF_ALU64_REG(BPF_MOV, BPF_REG_0, BPF_REG_2), }, - .errstr = "jump out of range", + .errstr = "not an exit", .result = REJECT, }, {
@@ -8097,6 +8098,964 @@ static struct bpf_test tests[] = { .result = REJECT, .prog_type = BPF_PROG_TYPE_CGROUP_SOCK, }, + { + "calls: basic sanity", + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .result = ACCEPT, + }, + { +"calls: using r0 returned by callee" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .result = ACCEPT, + }, + { +"calls: callee is using r1" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , len)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_ACT , + .result = ACCEPT, + }, + { +"calls: callee using args1" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ), + BPF_EXIT_INSN(), + }, + .errstr_unpriv ="allowed for root only" , + .result_unpriv = REJECT, + .result = ACCEPT, + }, + { +"calls: callee using wrong args2" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "R2 !read_ok", + .result = REJECT, + }, + { +"calls: callee using two args" , + .insns = { +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_6 , + offsetof(struct __sk_buff , len)), +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_6 , + offsetof(struct __sk_buff , len)), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_0 ,BPF_REG_2 ), + BPF_EXIT_INSN(), + }, + .errstr_unpriv ="allowed for root only" , + .result_unpriv = REJECT, + .result = ACCEPT, + }, + { +"calls: callee changing pkt pointers" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_6 ,BPF_REG_1 , + offsetof(struct xdp_md , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_7 ,BPF_REG_1 , + offsetof(struct xdp_md , data_end)), +BPF_MOV64_REG (BPF_REG_8 ,BPF_REG_6 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_8 ,8 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_8 ,BPF_REG_7 ,2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +/* clear_all_pkt_pointers() has to walk all frames +* to make sure that pkt pointers in the caller +* are cleared when callee is calling a helper that +* adjusts packet size +*/ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_6 ,0 ), +BPF_MOV32_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_xdp_adjust_head), + BPF_EXIT_INSN(), + }, + .result =REJECT , + .errstr = "R6 invalid mem access 'inv'", + .prog_type = BPF_PROG_TYPE_XDP, + }, + { +"calls: two calls with args" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,6 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_7 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , len)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: two calls with bad jump" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,6 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_7 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , len)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,-3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "jump out of range from insn 11 to 9", + .result = REJECT, + }, + { +"calls: recursive call. test1" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-1 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge", + .result = REJECT, + }, + { +"calls: recursive call. test2" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge", + .result = REJECT, + }, + { +"calls: unreachable code" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "unreachable insn 6", + .result = REJECT, + }, + { +"calls: invalid call" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-4 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "invalid destination", + .result = REJECT, + }, + { +"calls: jumping across function bodies. test1" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,-3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "jump out of range", + .result = REJECT, + }, + { +"calls: jumping across function bodies. test2" , + .insns = { +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "jump out of range", + .result = REJECT, + }, + { +"calls: call without exit" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, -2), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "not an exit", + .result = REJECT, + }, + { +"calls: call into middle of ld_imm64" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), +BPF_LD_IMM64 (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "last insn", + .result = REJECT, + }, + { +"calls: call into middle of other call" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "last insn", + .result = REJECT, + }, + { +"calls: two calls with bad fallthrough" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,6 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_0 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , len)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "not an exit", + .result = REJECT, + }, + { +"calls: two calls with stack read" , + .insns = { +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,6 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_7 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_XDP , + .result = ACCEPT, + }, + { +"calls: two calls with stack write" , + .insns = { +/* main prog */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_10 ,-16 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,7 ), +BPF_MOV64_REG (BPF_REG_8 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_ALU64_REG (BPF_ADD ,BPF_REG_8 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_8 ), +/* write into stack frame of main prog */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* read from stack frame of main prog */ +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_XDP , + .result = ACCEPT, + }, + { +"calls: spill into caller stack frame" , + .insns = { +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_STX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_XDP , + .errstr = "cannot spill", + .result = REJECT, + }, + { +"calls: two calls with stack write and void return" , + .insns = { +/* main prog */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_10 ,-16 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_7 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* write into stack frame of main prog */ +BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ), +BPF_EXIT_INSN (),/* void return */ + }, + .prog_type = BPF_PROG_TYPE_XDP, + .result = ACCEPT, + }, + { +"calls: ambiguous return value" , + .insns = { +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,5 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EXIT_INSN (), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,1 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }, + .errstr_unpriv = "allowed for root only", + .result_unpriv = REJECT, + .errstr = "R0 !read_ok", + .result = REJECT, + }, + { +"calls: two calls that return map_value" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,8 ), + +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_10 ,-8 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +/* fetch secound map_value_ptr from the stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_10 ,-16 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +/* call 3rd function twice */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* first time with fp-8 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_7 ), +/* second time with fp-16 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +/* lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr into stack frame of main prog */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (),/* return 0 */ + }, + .prog_type = BPF_PROG_TYPE_XDP, + .fixup_map1 = { 23 }, + .result = ACCEPT, + }, + { +"calls: two calls that return map_value with bool condition" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +/* call 3rd function twice */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* first time with fp-8 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,9 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_6 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_7 ), +/* second time with fp-16 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,1 ,2 ), +/* fetch secound map_value_ptr from the stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_7 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +/* lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (),/* return 0 */ +/* write map_value_ptr into stack frame of main prog */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (),/* return 1 */ + }, + .prog_type = BPF_PROG_TYPE_XDP, + .fixup_map1 = { 23 }, + .result = ACCEPT, + }, + { +"calls: two calls that return map_value with incorrect bool check" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +/* call 3rd function twice */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* first time with fp-8 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,9 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_6 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_7 ), +/* second time with fp-16 */ +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +/* fetch secound map_value_ptr from the stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_7 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +/* lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (),/* return 0 */ +/* write map_value_ptr into stack frame of main prog */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (),/* return 1 */ + }, + .prog_type = BPF_PROG_TYPE_XDP, + .fixup_map1 = { 23 }, + .result = REJECT, + .errstr = "invalid read from stack off -16+0 size 8", + }, + { +"calls: two calls that receive map_value via arg=ptr_stack_of_caller. test1" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* 1st lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_8 ,1 ), + +/* 2nd lookup from map */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ),/* 20 */ +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 ,/* 24 */ + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_9 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-16 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_9 ,1 ), + +/* call 3rd func with fp-8, 0|1, fp-16, 0|1 */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ),/* 30 */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_8 ), +BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_9 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ),/* 34 */ +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* if arg2 == 1 do *arg1 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_2 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + +/* if arg4 == 1 do *arg3 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_4 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_3 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,2 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .fixup_map1 = { 12, 22 }, + .result = REJECT, + .errstr = "invalid access to map value, value_size=8 off=2 size=8", + }, + { +"calls: two calls that receive map_value via arg=ptr_stack_of_caller. test2" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* 1st lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_8 ,1 ), + +/* 2nd lookup from map */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ),/* 20 */ +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 ,/* 24 */ + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_9 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-16 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_9 ,1 ), + +/* call 3rd func with fp-8, 0|1, fp-16, 0|1 */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ),/* 30 */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_8 ), +BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_9 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ),/* 34 */ +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* if arg2 == 1 do *arg1 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_2 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + +/* if arg4 == 1 do *arg3 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_4 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_3 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .fixup_map1 = { 12, 22 }, + .result = ACCEPT, + }, + { +"calls: two jumps that receive map_value via arg=ptr_stack_of_jumper. test3" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* 1st lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-24 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-24 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_8 ,1 ), + +/* 2nd lookup from map */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-24 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_9 ,0 ),// 26 +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +/* write map_value_ptr into stack frame of main prog at fp-16 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_MOV64_IMM (BPF_REG_9 ,1 ), + +/* call 3rd func with fp-8, 0|1, fp-16, 0|1 */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ),// 30 +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_8 ), +BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_9 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_1 ,0 ,1 ),// 34 +BPF_JMP_IMM (BPF_JA ,0 ,0 ,-30 ), + +/* subprog 2 */ +/* if arg2 == 1 do *arg1 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_2 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + +/* if arg4 == 1 do *arg3 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_4 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_3 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,2 ,0 ), + BPF_JMP_IMM(BPF_JA, 0, 0, -8), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .fixup_map1 = { 12, 22 }, + .result = REJECT, + .errstr = "invalid access to map value, value_size=8 off=2 size=8", + }, + { +"calls: two calls that receive map_value_ptr_or_null via arg. test1" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* 1st lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr_or_null into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_MOV64_IMM (BPF_REG_8 ,1 ), + +/* 2nd lookup from map */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr_or_null into stack frame of main prog at fp-16 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_9 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_MOV64_IMM (BPF_REG_9 ,1 ), + +/* call 3rd func with fp-8, 0|1, fp-16, 0|1 */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_8 ), +BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_9 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* if arg2 == 1 do *arg1 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_2 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + +/* if arg4 == 1 do *arg3 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_4 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_3 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .fixup_map1 = { 12, 22 }, + .result = ACCEPT, + }, + { +"calls: two calls that receive map_value_ptr_or_null via arg. test2" , + .insns = { +/* main prog */ +/* pass fp-16, fp-8 into a function */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-16 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_2 ), +/* 1st lookup from map */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr_or_null into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_MOV64_IMM (BPF_REG_8 ,1 ), + +/* 2nd lookup from map */ +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr_or_null into stack frame of main prog at fp-16 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_7 ,BPF_REG_0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_9 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_MOV64_IMM (BPF_REG_9 ,1 ), + +/* call 3rd func with fp-8, 0|1, fp-16, 0|1 */ +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_8 ), +BPF_MOV64_REG (BPF_REG_3 ,BPF_REG_7 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_9 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + +/* subprog 2 */ +/* if arg2 == 1 do *arg1 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_2 ,1 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + +/* if arg4 == 0 do *arg3 = 0 */ +BPF_JMP_IMM (BPF_JNE ,BPF_REG_4 ,0 ,2 ), +/* fetch map_value_ptr from the stack of this function */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_3 ,0 ), +/* write into map value */ +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .fixup_map1 = { 12, 22 }, + .result = REJECT, + .errstr = "R0 invalid mem access 'inv'", + }, + { +"calls: pkt_ptr spill into caller stack" , + .insns = { +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +/* spill unchecked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,2 ), +/* now the pkt range is verified, read pkt_ptr from stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_4 ,0 ), +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), + BPF_EXIT_INSN(), + }, + .result =ACCEPT , + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + }, };static int probe_filter_length (const struct bpf_insn *fp)
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 4/13 bpf: teach verifier to recognize zero initialized stack
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
programs with function calls are often passing various pointers via stack. When all calls are inlined llvm flattens stack accesses and optimizes away extra branches. When functions are not inlined it becomes the job of the verifier to recognize zero initialized stack to avoid exploring paths that program will not take.
The following program would fail otherwise:
ptr = &buffer_on_stack;
*ptr = 0;
...func_call(.., ptr, ...) {
if (..)
*ptr = bpf_map_lookup(); }
...
if (*ptr != 0) {
// Access (*ptr)->field is valid.
// Without stack_zero tracking such (*ptr)->field access // will be rejected
}
since stack slots are no longer uniform invalid | spill | misc add liveness marking to all slots, but do it in 8 byte chunks. So if nothing was read or written in [fp-16, fp-9] range it will be marked as LIVE_NONE.
If any byte in that range was read, it will be marked LIVE_READ and stacksafe() check will perform byte-by-byte verification. If all bytes in the range were written the slot will be marked as LIVE_WRITTEN.
This significantly speeds up state equality comparison and reduces total number of states processed.
before after
bpf_lb-DLB_L3.o 2051 2003 bpf_lb-DLB_L4.o 3287 3164 bpf_lb-DUNKNOWN.o 1080 1080 bpf_lxc-DDROP_ALL.o 24980 12361 bpf_lxc-DUNKNOWN.o 34308 16605 bpf_netdev.o 15404 10962 bpf_overlay.o 7191 6679
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- include/linux/bpf_verifier.h | 3 +- kernel/bpf/verifier.c | 129 +++++++++++++++++++++++++++++++++---------- 2 files changed, 103 insertions(+), 29 deletions(-)
include/linux/bpf_verifier.h +2 -1
@@ -91,7 +91,8 @@ struct bpf_reg_state {enum bpf_stack_slot_type {STACK_INVALID ,/* nothing was stored in this stack slot */ STACK_SPILL ,/* register spilled into stack */ -STACK_MISC /* BPF program wrote some data into this slot */ +STACK_MISC ,/* BPF program wrote some data into this slot */ +STACK_ZERO ,/* BPF program wrote constant zero */ };#define BPF_REG_SIZE 8/* size of eBPF register in bytes */
kernel/bpf/verifier.c +101 -28
211 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 5/13 selftests/bpf: add tests for stack_zero tracking
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
adjust two tests, since verifier got smarter
and add new one to test stack_zero logic
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- tools/testing/selftests/bpf/test_verifier.c | 66 ++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/test_verifier.c +64 -2
@@ -5649,7 +5649,7 @@ static struct bpf_test tests[] = { "helper access to variable memory: size > 0 not allowed on NULL (ARG_PTR_TO_MEM_OR_NULL)", .insns = {BPF_MOV64_IMM (BPF_REG_1 ,0 ), -BPF_MOV64_IMM (BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_2 ,1 ),BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_2 ,-128 ),BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_10 ,-128 ), BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 64),
@@ -5884,7 +5884,7 @@ static struct bpf_test tests[] = {BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_0 ,-24 ),BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_0 ,-16 ),BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_0 ,-8 ), -BPF_MOV64_IMM (BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_2 ,1 ),BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_2 ,-128 ),BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_10 ,-128 ), BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 63),
@@ -9056,6 +9056,68 @@ static struct bpf_test tests[] = { .result = ACCEPT, .prog_type = BPF_PROG_TYPE_SCHED_CLS, }, + { + "calls: caller stack init to zero or map_value_or_null", + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_10 ,BPF_REG_0 ,-8 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +/* fetch map_value_or_null or const_zero from stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_10 ,-8 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ), +/* store into map_value */ +BPF_ST_MEM (BPF_W ,BPF_REG_0 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +/* if (ctx == 0) return; */ +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_1 ,0 ,8 ), +/* else bpf_map_lookup() and *(fp - 8) = r0 */ +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_2 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), +/* write map_value_ptr_or_null into stack frame of main prog at fp-8 */ +BPF_STX_MEM (BPF_DW ,BPF_REG_6 ,BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .fixup_map1 = { 13 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_XDP, + }, + { +"calls: stack init to zero and pruning" , + .insns = { +/* first make allocated_stack 16 byte */ +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-16 ,0 ), +/* now fork the execution such that the false branch +* of JGT insn will be verified second and it skisp zero +* init of fp-8 stack slot. If stack liveness marking +* is missing live_read marks from call map_lookup +* processing then pruning will incorrectly assume +* that fp-8 stack slot was unused in the fall-through +* branch and will accept the program incorrectly +*/ +BPF_JMP_IMM (BPF_JGT ,BPF_REG_1 ,2 ,2 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_10 ,-8 ,0 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-8 ), +BPF_LD_MAP_FD (BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_map_lookup_elem), + BPF_EXIT_INSN(), + }, + .fixup_map2 = { 6 }, + .errstr = "invalid indirect read from stack off -8+0 size 8", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_XDP, + }, };static int probe_filter_length (const struct bpf_insn *fp)
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 6/13 libbpf: add support for bpf_call
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
- recognize relocation emitted by llvm
- since all regular function will be kept in .text section and llvm takes care of pc-relative offsets in bpf_call instruction simply copy all of .text to relevant program section while adjusting bpf_call instructions in program section to point to newly copied body of instructions from .text
- do so for all programs in the elf file
- set all programs types to the one passed to bpf_prog_load()
Note for elf files with multiple programs that use different functions in .text section we need to do 'linker' style logic. This work is still TBD
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- tools/include/uapi/linux/bpf.h | 6 ++ tools/lib/bpf/bpf.h | 2 +- tools/lib/bpf/libbpf.c | 170 ++++++++++++++++++++++++++++++----------- 3 files changed, 134 insertions(+), 44 deletions(-)
tools/include/uapi/linux/bpf.h +6 -0
@@ -197,8 +197,14 @@ enum bpf_attach_type {*/ #define BPF_F_STRICT_ALIGNMENT (1U << 0) +/* when bpf_ldimm64->src_reg == BPF_PSEUDO_MAP_FD, bpf_ldimm64->imm == fd */ #define BPF_PSEUDO_MAP_FD 1 +/* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative +* offset to another bpf function +*/ +#define BPF_PSEUDO_CALL 1 +/* flags for BPF_MAP_UPDATE_ELEM command */ #define BPF_ANY 0/* create new element or update existing */ #define BPF_NOEXIST 1/* create new element if it didn't exist */
tools/lib/bpf/bpf.h +1 -1
@@ -40,7 +40,7 @@ int bpf_create_map_in_map(enum bpf_map_type map_type, const char *name, __u32 map_flags);/* Recommend log buffer size */ -#define BPF_LOG_BUF_SIZE 65536 +#define BPF_LOG_BUF_SIZE (256 * 1024) int bpf_load_program_name(enum bpf_prog_type type, const char *name, const struct bpf_insn *insns, size_t insns_cnt, const char *license,
tools/lib/bpf/libbpf.c +127 -43
287 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 7/13 selftests/bpf: add bpf_call test
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
strip always_inline from test_l4lb.c and compile it with -fno-inline to let verifier go through 11 function with various function arguments and return values
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- tools/testing/selftests/bpf/Makefile | 11 +- tools/testing/selftests/bpf/test_l4lb_noinline.c | 473 +++++++++++++++++++++++ tools/testing/selftests/bpf/test_progs.c | 14 +- 3 files changed, 492 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/bpf/test_l4lb_noinline.c
tools/testing/selftests/bpf/Makefile +8 -3
@@ -17,7 +17,8 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test_obj_id.o \ test_pkt_md_access.o test_xdp_redirect.o test_xdp_meta.o sockmap_parse_prog.o \ - sockmap_verdict_prog.o dev_cgroup.o sample_ret0.o test_tracepoint.o + sockmap_verdict_prog.o dev_cgroup.o sample_ret0.o test_tracepoint.o \ + test_l4lb_noinline.o TEST_PROGS := test_kmod.sh test_xdp_redirect.sh test_xdp_meta.sh \ test_offload.py
@@ -49,8 +50,12 @@ else CPU ?= generic endif +CLANG_FLAGS = -I. -I./include/uapi -I../../../include/uapi \ + -Wno-compare-distinct-pointer-types + +$(OUTPUT)/test_l4lb_noinline.o: CLANG_FLAGS += -fno-inline + %.o: %.c - $(CLANG) -I. -I./include/uapi -I../../../include/uapi \ - -Wno-compare-distinct-pointer-types \ + $(CLANG) $(CLANG_FLAGS) \ -O2 -target bpf -emit-llvm -c $< -o - | \ $(LLC) -march=bpf -mcpu=$(CPU) -filetype=obj -o $@
tools/testing/selftests/bpf/test_l4lb_noinline.c new file +473 -0
473 lines not on this page · raw message
tools/testing/selftests/bpf/test_progs.c +11 -3
@@ -169,10 +169,9 @@ static void test_xdp(void)#define NUM_ITER 100000#define VIP_NUM 5 -static void test_l4lb (void ) +static void test_l4lb (const char *file) {unsigned int nr_cpus =bpf_num_possible_cpus (); -const char *file ="./test_l4lb.o" ;struct vip key = {.protocol =6 };struct vip_meta {__u32 flags;
@@ -249,6 +248,15 @@ static void test_l4lb(void)bpf_object__close (obj); } +static void test_l4lb_all (void ) +{ +const char *file1 ="./test_l4lb.o" ; +const char *file2 ="./test_l4lb_noinline.o" ; + +test_l4lb (file1); +test_l4lb (file2); +} +static void test_tcp_estats (void ) {const char *file ="./test_tcp_estats.o" ;
@@ -757,7 +765,7 @@ int main(void)test_pkt_access ();test_xdp (); -test_l4lb (); +test_l4lb_all ();test_tcp_estats ();test_bpf_obj_id ();test_pkt_md_access ();
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 8/13 selftests/bpf: add xdp noinline test
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
add large semi-artificial XDP test with 18 functions to stress test bpf call verification logic
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- tools/testing/selftests/bpf/Makefile | 3 +- tools/testing/selftests/bpf/test_progs.c | 81 +++ tools/testing/selftests/bpf/test_xdp_noinline.c | 833 ++++++++++++++++++++++++ 3 files changed, 916 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/test_xdp_noinline.c
tools/testing/selftests/bpf/Makefile +2 -1
@@ -18,7 +18,7 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test_obj_id.o \ test_pkt_md_access.o test_xdp_redirect.o test_xdp_meta.o sockmap_parse_prog.o \ sockmap_verdict_prog.o dev_cgroup.o sample_ret0.o test_tracepoint.o \ - test_l4lb_noinline.o + test_l4lb_noinline.o test_xdp_noinline.o TEST_PROGS := test_kmod.sh test_xdp_redirect.sh test_xdp_meta.sh \ test_offload.py
@@ -54,6 +54,7 @@ CLANG_FLAGS = -I. -I./include/uapi -I../../../include/uapi \ -Wno-compare-distinct-pointer-types $(OUTPUT)/test_l4lb_noinline.o: CLANG_FLAGS += -fno-inline +$(OUTPUT)/test_xdp_noinline.o: CLANG_FLAGS += -fno-inline %.o: %.c $(CLANG) $(CLANG_FLAGS) \
tools/testing/selftests/bpf/test_progs.c +81 -0
93 lines not on this page · raw message
tools/testing/selftests/bpf/test_xdp_noinline.c new file +833 -0
833 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 9/13 bpf: add support for bpf_call to interpreter
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
though bpf_call is still the same call instruction and calling convention 'bpf to bpf' and 'bpf to helper' is the same the interpreter has to oparate on 'struct bpf_insn *'. To distinguish these two cases add a kernel internal opcode and mark call insns with it.
This opcode is seen by interpreter only. JITs will never see it. Also add tiny bit of debug code to aid interpreter debugging.
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- include/linux/bpf.h | 1 + include/linux/filter.h | 6 ++++ kernel/bpf/core.c | 90 ++++++++++++++++++++++++++++++++++++++++---------- kernel/bpf/verifier.c | 36 ++++++++++++++++++++ 4 files changed, 116 insertions(+), 17 deletions(-)
include/linux/bpf.h +1 -0
@@ -402,6 +402,7 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)/* verify correctness of eBPF program */ int bpf_check (struct bpf_prog **fp,union bpf_attr *attr); +void bpf_patch_call_args (struct bpf_insn *insn,u32 stack_depth);/* Map specifics */ struct net_device *__dev_map_lookup_elem (struct bpf_map *map,u32 key);
include/linux/filter.h +6 -0
@@ -58,6 +58,9 @@ struct bpf_prog_aux;/* unused opcode to mark special call to bpf_tail_call() helper */ #define BPF_TAIL_CALL 0xf0 +/* unused opcode to mark call to interpreter with arguments */ +#define BPF_CALL_ARGS 0xe0 +/* As per nm, we expose JITed images as text (code) section for * kallsyms. That way, tools like perf can find it to match * addresses.
@@ -710,6 +713,9 @@ bool sk_filter_charge(struct sock *sk, struct sk_filter *fp);void sk_filter_uncharge (struct sock *sk,struct sk_filter *fp);u64 __bpf_call_base (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5); +#define __bpf_call_base_args \ + ((u64 (*)(u64, u64, u64, u64, u64, const struct bpf_insn *)) \ + __bpf_call_base)struct bpf_prog *bpf_int_jit_compile (struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);
kernel/bpf/core.c +73 -17
141 lines not on this page · raw message
kernel/bpf/verifier.c +36 -0
@@ -1458,6 +1458,21 @@ static int update_stack_depth(struct bpf_verifier_env *env,return 0 ; } +static int get_callee_stack_depth (struct bpf_verifier_env *env, +const struct bpf_insn *insn,int idx) +{ +int start = idx + insn->imm +1 , subprog; + + subprog =find_subprog (env, start); +if (subprog <0 ) { +WARN_ONCE (1 ,"verifier bug. No program starts at insn %d\n" , + start); +return -EFAULT ; + } + subprog++; +return env->subprog_stack_depth[subprog]; +} +/* check whether memory at (regno + off) is accessible for t = (read | write) * if t==write, value_regno is a register which value is stored into memory * if t==read, value_regno is a register which will receive the value from memory
@@ -4997,6 +5012,24 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)return 0 ; } +static int fixup_call_args (struct bpf_verifier_env *env) +{ +struct bpf_prog *prog = env->prog; +struct bpf_insn *insn = prog->insnsi; +int i, depth; + +for (i =0 ; i < prog->len; i++, insn++) { +if (insn->code != (BPF_JMP |BPF_CALL ) || + insn->src_reg !=BPF_PSEUDO_CALL ) +continue ; + depth =get_callee_stack_depth (env, insn, i); +if (depth <0 ) +return depth; +bpf_patch_call_args (insn, depth); + } +return 0 ; +} +/* fixup insn->imm field of bpf_call instructions * and inline eligible helpers as explicit sequence of BPF instructions *
@@ -5225,6 +5258,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr)if (ret ==0 ) ret =fixup_bpf_calls (env); +if (ret ==0 ) + ret =fixup_call_args (env); +if (log->level &&bpf_verifier_log_full (log)) ret = -ENOSPC ;if (log->level && !log->ubuf) {
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 10/13 bpf: fix net.core.bpf_jit_enable race
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
global bpf_jit_enable variable is tested multiple times in JITs, blinding and verifier core. The malicious root can try to toggle it while loading the programs. This race condition was accounted for and there should be no issues, but it's safer to avoid this race condition.
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- arch/arm/net/bpf_jit_32.c | 2 +- arch/arm64/net/bpf_jit_comp.c | 2 +- arch/mips/net/ebpf_jit.c | 2 +- arch/powerpc/net/bpf_jit_comp64.c | 2 +- arch/s390/net/bpf_jit_comp.c | 2 +- arch/sparc/net/bpf_jit_comp_64.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- include/linux/filter.h | 5 +++-- kernel/bpf/core.c | 3 ++- kernel/bpf/verifier.c | 2 +- 10 files changed, 13 insertions(+), 11 deletions(-)
arch/arm/net/bpf_jit_32.c +1 -1
@@ -1824,7 +1824,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)/* If BPF JIT was not enabled then we must fall back to * the interpreter. */ -if (!bpf_jit_enable) +if (!prog->jit_requested)return orig_prog;/* If constant blinding was enabled and we failed during blinding
arch/arm64/net/bpf_jit_comp.c +1 -1
@@ -844,7 +844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)int image_size;u8 *image_ptr; -if (!bpf_jit_enable) +if (!prog->jit_requested)return orig_prog; tmp =bpf_jit_blind_constants (prog);
arch/mips/net/ebpf_jit.c +1 -1
@@ -1869,7 +1869,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)unsigned int image_size;u8 *image_ptr; -if (!bpf_jit_enable || !cpu_has_mips64r2) +if (!prog->jit_requested || !cpu_has_mips64r2)return prog; tmp =bpf_jit_blind_constants (prog);
arch/powerpc/net/bpf_jit_comp64.c +1 -1
@@ -993,7 +993,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)struct bpf_prog *tmp_fp;bool bpf_blinded = false; -if (!bpf_jit_enable) +if (!fp->jit_requested)return org_fp; tmp_fp =bpf_jit_blind_constants (org_fp);
arch/s390/net/bpf_jit_comp.c +1 -1
@@ -1300,7 +1300,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)struct bpf_jit jit;int pass; -if (!bpf_jit_enable) +if (!fp->jit_requested)return orig_fp; tmp =bpf_jit_blind_constants (fp);
arch/sparc/net/bpf_jit_comp_64.c +1 -1
@@ -1517,7 +1517,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)u8 *image_ptr;int pass; -if (!bpf_jit_enable) +if (!prog->jit_requested)return orig_prog; tmp =bpf_jit_blind_constants (prog);
arch/x86/net/bpf_jit_comp.c +1 -1
@@ -1121,7 +1121,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)int pass;int i; -if (!bpf_jit_enable) +if (!prog->jit_requested)return orig_prog; tmp =bpf_jit_blind_constants (prog);
include/linux/filter.h +3 -2
@@ -458,6 +458,7 @@ struct bpf_binary_header {struct bpf_prog {u16 pages;/* Number of allocated pages */ u16 jited:1 ,/* Is our filter JIT'ed? */ + jit_requested:1 ,/* archs need to JIT the prog */ locked:1 ,/* Program image locked? */ gpl_compatible:1 ,/* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */
@@ -804,7 +805,7 @@ static inline bool bpf_prog_ebpf_jited(const struct bpf_prog *fp)return fp->jited &&bpf_jit_is_ebpf (); } -static inline bool bpf_jit_blinding_enabled (void ) +static inline bool bpf_jit_blinding_enabled (struct bpf_prog *prog) {/* These are the prerequisites, should someone ever have the * idea to call blinding outside of them, we make sure to
@@ -812,7 +813,7 @@ static inline bool bpf_jit_blinding_enabled(void)*/ if (!bpf_jit_is_ebpf ())return false; -if (!bpf_jit_enable) +if (!prog->jit_requested)return false;if (!bpf_jit_harden)return false;
kernel/bpf/core.c +2 -1
@@ -94,6 +94,7 @@ struct bpf_prog *bpf_prog_alloc(unsigned int size, gfp_t gfp_extra_flags) fp->pages = size /PAGE_SIZE ; fp->aux = aux; fp->aux->prog = fp; + fp->jit_requested =ebpf_jit_enabled ();INIT_LIST_HEAD_RCU (&fp->aux->ksym_lnode);
@@ -721,7 +722,7 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)struct bpf_insn *insn;int i, rewritten; -if (!bpf_jit_blinding_enabled ()) +if (!bpf_jit_blinding_enabled (prog))return prog; clone =bpf_prog_clone_create (prog,GFP_USER );
kernel/bpf/verifier.c +1 -1
@@ -5080,7 +5080,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env)/* BPF_EMIT_CALL() assumptions in some of the map_gen_lookup * handlers are currently limited to 64 bit only. */ -if (ebpf_jit_enabled () &&BITS_PER_LONG ==64 && +if (prog->jit_requested &&BITS_PER_LONG ==64 && insn->imm == BPF_FUNC_map_lookup_elem) { map_ptr = env->insn_aux_data[i + delta].map_ptr; if (map_ptr == BPF_MAP_PTR_POISON ||
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 11/13 bpf: x64: add JIT support for multi-function programs
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
Typical JIT does several passes over bpf instructions to compute total size and relative offsets of jumps and calls. With multitple bpf functions calling each other all relative calls will have invalid offsets intially therefore we need to additional last pass over the program to emit calls with correct offsets. For example in case of three bpf functions:
main:
call foo
call bpf_map_lookup
exit
foo:
call bar
exit
bar:
exit
We will call bpf_int_jit_compile() indepedently for main(), foo() and bar() x64 JIT typically does 4-5 passes to converge.
After these initial passes the image for these 3 functions
will be good except call targets, since start addresses of foo() and bar() are unknown when we were JITing main()
(note that call bpf_map_lookup will be resolved properly
during initial passes).
Once start addresses of 3 functions are known we patch call_insn->imm to point to right functions and call bpf_int_jit_compile() again which needs only one pass. Additional safety checks are done to make sure this
last pass doesn't produce image that is larger or smaller
than previous pass.
When constant blinding is on it's applied to all functions at the first pass, since doing it once again at the last pass can change size of the JITed code.
Tested on x64 and arm64 hw with JIT on/off, blinding on/off. x64 jits bpf-to-bpf calls correctly while arm64 falls back to interpreter. All other JITs that support normal BPF_CALL will behave the same way since bpf-to-bpf call is equivalent to bpf-to-kernel call from JITs point of view.
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- arch/x86/net/bpf_jit_comp.c | 47 ++++++++++++++-- include/linux/bpf.h | 3 ++ include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 2 + kernel/bpf/core.c | 13 ++++- kernel/bpf/syscall.c | 3 +- kernel/bpf/verifier.c | 126 +++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 189 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +44 -3
@@ -1109,13 +1109,23 @@ xadd: if (is_imm8(insn->off))return proglen; } +struct x64_jit_data { +struct bpf_binary_header *header; +int *addrs; +u8 *image; +int proglen; +struct jit_context ctx; +}; +struct bpf_prog *bpf_int_jit_compile (struct bpf_prog *prog) {struct bpf_binary_header *header =NULL ;struct bpf_prog *tmp, *orig_prog = prog; +struct x64_jit_data *jit_data;int proglen, oldproglen =0 ;struct jit_context ctx = {};bool tmp_blinded = false; +bool extra_pass = false;u8 *image =NULL ;int *addrs;int pass;
@@ -1135,10 +1145,28 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = tmp; } + jit_data = prog->aux->jit_data; +if (!jit_data) { + jit_data =kzalloc (sizeof (*jit_data),GFP_KERNEL ); +if (!jit_data) { + prog = orig_prog; + goto out; + } + prog->aux->jit_data = jit_data; + } + addrs = jit_data->addrs; +if (addrs) { + ctx = jit_data->ctx; + oldproglen = jit_data->proglen; + image = jit_data->image; + header = jit_data->header; + extra_pass = true; + goto skip_init_addrs; + } addrs =kmalloc (prog->len *sizeof (*addrs),GFP_KERNEL );if (!addrs) { prog = orig_prog; - goto out; + goto out_addrs; }/* Before first pass, make a rough estimation of addrs[]
@@ -1149,6 +1177,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) addrs[i] = proglen; } ctx.cleanup_addr = proglen; +skip_init_addrs:/* JITed image shrinks with every pass and the loop iterates * until the image stops shrinking. Very large bpf programs
@@ -1189,7 +1218,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)if (image) {bpf_flush_icache (header, image + proglen); -bpf_jit_binary_lock_ro (header); +if (!prog->is_func || extra_pass) { +bpf_jit_binary_lock_ro (header); + }else { + jit_data->addrs = addrs; + jit_data->ctx = ctx; + jit_data->proglen = proglen; + jit_data->image = image; + jit_data->header = header; + } prog->bpf_func = (void *)image; prog->jited =1 ; prog->jited_len = proglen;
@@ -1197,8 +1234,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = orig_prog; } +if (!prog->is_func || extra_pass) { out_addrs: -kfree (addrs); +kfree (addrs); +kfree (jit_data); + prog->aux->jit_data =NULL ; + } out:if (tmp_blinded) bpf_jit_prog_release_other(prog, prog == orig_prog ?
include/linux/bpf.h +3 -0
@@ -200,6 +200,9 @@ struct bpf_prog_aux {u32 max_ctx_offset;u32 stack_depth;u32 id; +u32 func_cnt; +struct bpf_prog **func; +void *jit_data;/* JIT specific data. arch dependent */ struct latch_tree_node ksym_tnode;struct list_head ksym_lnode;const struct bpf_prog_ops *ops;
include/linux/bpf_verifier.h +1 -0
@@ -143,6 +143,7 @@ struct bpf_insn_aux_data {union {enum bpf_reg_type ptr_type;/* pointer type for load/store insns */ struct bpf_map *map_ptr;/* pointer for call insn into lookup_elem */ +s32 call_imm;/* saved imm field of call insn */ };int ctx_field_size;/* the ctx field size for load insn, maybe 0 */ bool seen;/* this insn was processed by the verifier */
include/linux/filter.h +2 -0
@@ -463,6 +463,8 @@ struct bpf_prog { gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ + blinded:1, /* Was blinded */ + is_func:1, /* program is a bpf function */ kprobe_override:1; /* Do we override a kprobe? */enum bpf_prog_type type;/* Type of BPF program */ u32 len;/* Number of filter blocks */
kernel/bpf/core.c +11 -2
@@ -722,7 +722,7 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)struct bpf_insn *insn;int i, rewritten; -if (!bpf_jit_blinding_enabled (prog)) +if (!bpf_jit_blinding_enabled (prog) || prog->blinded)return prog; clone =bpf_prog_clone_create (prog,GFP_USER );
@@ -764,6 +764,7 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) i += insn_delta; } + clone->blinded =1 ;return clone; }#endif /* CONFIG_BPF_JIT */
@@ -1629,11 +1630,19 @@ int bpf_prog_array_copy_info(struct bpf_prog_array __rcu *array,static void bpf_prog_free_deferred (struct work_struct *work) {struct bpf_prog_aux *aux; +int i; aux = container_of(work, struct bpf_prog_aux, work);if (bpf_prog_is_dev_bound (aux))bpf_prog_offload_destroy (aux->prog); -bpf_jit_free (aux->prog); +for (i =0 ; i < aux->func_cnt; i++) +bpf_jit_free (aux->func[i]); +if (aux->func_cnt) { +kfree (aux->func); +bpf_prog_unlock_free (aux->prog); + }else { +bpf_jit_free (aux->prog); + } }/* Free internal BPF program */
kernel/bpf/syscall.c +2 -1
@@ -1194,7 +1194,8 @@ static int bpf_prog_load(union bpf_attr *attr) goto free_used_maps;/* eBPF program is ready to be JITed */ - prog =bpf_prog_select_runtime (prog, &err); +if (!prog->bpf_func) + prog =bpf_prog_select_runtime (prog, &err);if (err <0 ) goto free_used_maps;
kernel/bpf/verifier.c +126 -0
138 lines not on this page · raw message
Signature · 1 line
2.9.5
Alexei Starovoitov
maintainerPATCH 12/13 bpf: arm64: add JIT support for multi-function programs
In reply to Alexei StarovoitovFrom: Alexei Starovoitov <[email protected]>
similar to x64 add support for bpf-to-bpf calls.
When program has calls to in-kernel helpers the target call offset is known at JIT time and arm64 architecture needs 2 passes. With bpf-to-bpf calls the dynamically allocated function start is unknown until all functions of the program are JITed. Therefore (just like x64) arm64 JIT needs one extra pass over the program to emit correct call offsets.
Implementation detail:
Avoid being too clever in 64-bit immediate moves and always use 4 instructions (instead of 3-4 depending on the address) to make sure only one extra pass is needed.
If some future optimization would make it worth while to optimize 'call 64-bit imm' further, the JIT would need to do 4 passes over the program instead of 3 as in this patch.
For typical bpf program address the mov needs 3 or 4 insns, so unconditional 4 insns to save extra pass is a worthy trade off at this state of JIT.
Signed-off-by Alexei Starovoitov <[email protected]>
Acked-by Daniel Borkmann <[email protected]>
--- arch/arm64/net/bpf_jit_comp.c | 68 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-)
arch/arm64/net/bpf_jit_comp.c +64 -4
@@ -99,6 +99,20 @@ static inline void emit_a64_mov_i64(const int reg, const u64 val, } } +static inline void emit_addr_mov_i64 (const int reg,const u64 val, +struct jit_ctx *ctx) +{ +u64 tmp = val; +int shift =0 ; + +emit (A64_MOVZ (1 , reg, tmp &0xffff , shift), ctx); +for (;shift <48 ;) { + tmp >>=16 ; + shift +=16 ; +emit (A64_MOVK (1 , reg, tmp &0xffff , shift), ctx); + } +} +static inline void emit_a64_mov_i (const int is64,const int reg,const s32 val,struct jit_ctx *ctx) {
@@ -603,7 +617,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)const u8 r0 = bpf2a64[BPF_REG_0 ];const u64 func = (u64 )__bpf_call_base + imm; -emit_a64_mov_i64 (tmp, func, ctx); +if (ctx->prog->is_func) +emit_addr_mov_i64 (tmp, func, ctx); +else +emit_a64_mov_i64 (tmp, func, ctx);emit (A64_BLR (tmp), ctx);emit (A64_MOV (1 , r0,A64_R (0 )), ctx);break ;
@@ -835,11 +852,19 @@ static inline void bpf_flush_icache(void *start, void *end)flush_icache_range ((unsigned long )start, (unsigned long )end); } +struct arm64_jit_data { +struct bpf_binary_header *header; +u8 *image; +struct jit_ctx ctx; +}; +struct bpf_prog *bpf_int_jit_compile (struct bpf_prog *prog) {struct bpf_prog *tmp, *orig_prog = prog;struct bpf_binary_header *header; +struct arm64_jit_data *jit_data;bool tmp_blinded = false; +bool extra_pass = false;struct jit_ctx ctx;int image_size;u8 *image_ptr;
@@ -858,13 +883,29 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = tmp; } + jit_data = prog->aux->jit_data; +if (!jit_data) { + jit_data =kzalloc (sizeof (*jit_data),GFP_KERNEL ); +if (!jit_data) { + prog = orig_prog; + goto out; + } + prog->aux->jit_data = jit_data; + } +if (jit_data->ctx.offset) { + ctx = jit_data->ctx; + image_ptr = jit_data->image; + header = jit_data->header; + extra_pass = true; + goto skip_init_ctx; + }memset (&ctx,0 ,sizeof (ctx)); ctx.prog = prog; ctx.offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (ctx.offset ==NULL ) { prog = orig_prog; - goto out; + goto out_off; }/* 1. Initial fake pass to compute ctx->idx. */
@@ -895,6 +936,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)/* 2. Now, the actual pass. */ ctx.image = (__le32 *)image_ptr; +skip_init_ctx: ctx.idx =0 ;build_prologue (&ctx);
@@ -920,13 +962,31 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)bpf_flush_icache (header, ctx.image + ctx.idx); -bpf_jit_binary_lock_ro (header); +if (!prog->is_func || extra_pass) { +if (extra_pass && ctx.idx != jit_data->ctx.idx) { +pr_err_once ("multi-func JIT bug %d != %d\n" , + ctx.idx, jit_data->ctx.idx); +bpf_jit_binary_free (header); + prog->bpf_func =NULL ; + prog->jited =0 ; + goto out_off; + } +bpf_jit_binary_lock_ro (header); + }else { + jit_data->ctx = ctx; + jit_data->image = image_ptr; + jit_data->header = header; + } prog->bpf_func = (void *)ctx.image; prog->jited =1 ; prog->jited_len = image_size; +if (!prog->is_func || extra_pass) { out_off: -kfree (ctx.offset); +kfree (ctx.offset); +kfree (jit_data); + prog->aux->jit_data =NULL ; + } out:if (tmp_blinded) bpf_jit_prog_release_other(prog, prog == orig_prog ?
Signature · 1 line
2.9.5
Re: [PATCH bpf-next 12/13] bpf: arm64: add JIT support for multi-function programs
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 903
Quoted 9 lines · Alexei Starovoitov
arch/arm64/net/bpf_jit_comp.c +7
-0+if (jit_data->ctx.offset) { + ctx = jit_data->ctx; + image_ptr = jit_data->image; + header = jit_data->header; + extra_pass = true; + goto skip_init_ctx; + }memset (&ctx,0 ,sizeof (ctx)); ctx.prog = prog;
The 'goto' jumps over the 'image_size' initialization
Quoted 3 lines
prog->bpf_func = (void *)ctx.image;
prog->jited = 1;
prog->jited_len = image_size;so we now get a warning here, starting with linux-next-20171218:
arch/arm64/net/bpf_jit_comp.c: In function 'bpf_int_jit_compile':arch/arm64/net/bpf_jit_comp.c:982:18: error: 'image_size' may be used uninitialized in this function [-Werror=maybe-uninitialized]
I could not figure out what the code should be doing instead, or if it is indeed safe and the warning is a false-positive.
Arnd
Daniel Borkmann
maintainerRe: [PATCH bpf-next 12/13] bpf: arm64: add JIT support for multi-function programs
Quoted 19 lines · Arnd Bergmann
Review
Quoted 9 lines · Alexei Starovoitov
(quoted diff) +7
-0+ if (jit_data->ctx.offset) { + ctx = jit_data->ctx; + image_ptr = jit_data->image; + header = jit_data->header; + extra_pass = true; + goto skip_init_ctx; + } memset(&ctx, 0, sizeof(ctx)); ctx.prog = prog;The 'goto' jumps over the '
image_size' initializationQuoted 3 lines
prog->bpf_func = (void *)ctx.image; prog->jited = 1; prog->jited_len = image_size;
Quoted 8 lines
so we now get a warning here, starting with linux-next-20171218:
arch/arm64/net/bpf_jit_comp.c: In function 'bpf_int_jit_compile':arch/arm64/net/bpf_jit_comp.c:982:18: error: 'image_size' may be used uninitialized in this function [-Werror=maybe-uninitialized]I could not figure out what the code should be doing instead, or if it is indeed safe and the warning is a false-positive.
Good catch, it's buggy indeed. Fix like below is needed; I can submit it properly a bit later today:
arch/arm64/net/bpf_jit_comp.c +3 -0
@@ -855,6 +855,7 @@ static inline void bpf_flush_icache(void *start, void *end)struct arm64_jit_data {struct bpf_binary_header *header;u8 *image; +int image_size;struct jit_ctx ctx; };
@@ -895,6 +896,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)if (jit_data->ctx.offset) { ctx = jit_data->ctx; image_ptr = jit_data->image; + image_size = jit_data->image_size; header = jit_data->header; extra_pass = true; goto skip_init_ctx;
@@ -975,6 +977,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) }else { jit_data->ctx = ctx; jit_data->image = image_ptr; + jit_data->image_size = image_size; jit_data->header = header; } prog->bpf_func = (void *)ctx.image;
Re: [PATCH bpf-next 12/13] bpf: arm64: add JIT support for multi-function programs
Quoted 26 lines · Daniel Borkmann
Quoted 25 lines · Arnd Bergmann
Review
Quoted 9 lines · Alexei Starovoitov
(quoted diff) +7
-0+ if (jit_data->ctx.offset) { + ctx = jit_data->ctx; + image_ptr = jit_data->image; + header = jit_data->header; + extra_pass = true; + goto skip_init_ctx; + } memset(&ctx, 0, sizeof(ctx)); ctx.prog = prog;The 'goto' jumps over the '
image_size' initializationQuoted 3 lines
prog->bpf_func = (void *)ctx.image; prog->jited = 1; prog->jited_len = image_size;so we now get a warning here, starting with linux-next-20171218:
arch/arm64/net/bpf_jit_comp.c: In function 'bpf_int_jit_compile':arch/arm64/net/bpf_jit_comp.c:982:18: error: 'image_size' may be used uninitialized in this function [-Werror=maybe-uninitialized]
Quoted 5 lines
Quoted 2 lines
I could not figure out what the code should be doing instead, or if it is indeed safe and the warning is a false-positive.
Good catch, it's buggy indeed. Fix like below is needed; I can submit it properly a bit later today:
good catch. My arm gcc 4.8.5 didn't warn about it.
the following would be better fix:
arch/arm64/net/bpf_jit_comp.c +1 -0
@@ -897,6 +897,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) image_ptr = jit_data->image; header = jit_data->header; extra_pass = true; + image_size =sizeof (u32) * ctx.idx; goto skip_init_ctx; }memset (&ctx,0 ,sizeof (ctx));
Quoted 29 lines
arch/arm64/net/bpf_jit_comp.c +3
-0@@ -855,6 +855,7 @@ static inline void bpf_flush_icache(void *start, void *end)struct arm64_jit_data {struct bpf_binary_header *header;u8 *image; +int image_size;struct jit_ctx ctx; };@@ -895,6 +896,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)if (jit_data->ctx.offset) { ctx = jit_data->ctx; image_ptr = jit_data->image; + image_size = jit_data->image_size; header = jit_data->header; extra_pass = true; goto skip_init_ctx;@@ -975,6 +977,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) }else { jit_data->ctx = ctx; jit_data->image = image_ptr; + jit_data->image_size = image_size; jit_data->header = header; } prog->bpf_func = (void *)ctx.image;
Alexei Starovoitov
maintainerPATCH 13/13 selftests/bpf: additional bpf_call tests
In reply to Alexei StarovoitovFrom: Daniel Borkmann <[email protected]>
Add some additional checks for few more corner cases.
Signed-off-by Daniel Borkmann <[email protected]>
Signed-off-by Alexei Starovoitov <[email protected]>
--- tools/testing/selftests/bpf/test_verifier.c | 597 ++++++++++++++++++++++++++++ 1 file changed, 597 insertions(+)
tools/testing/selftests/bpf/test_verifier.c +597 -0
@@ -8111,6 +8111,180 @@ static struct bpf_test tests[] = { .result = ACCEPT, }, { + "calls: not on unpriviledged", + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), + BPF_EXIT_INSN(), + }, + .errstr_unpriv ="function calls to other bpf functions are allowed for root only" , + .result_unpriv = REJECT, + .result = ACCEPT, + }, + { +"calls: overlapping caller/callee" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "last insn is not an exit or jmp", + .result = REJECT, + }, + { +"calls: wrong recursive calls" , + .insns = { +BPF_JMP_IMM (BPF_JA ,0 ,0 ,4 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,4 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "jump out of range", + .result = REJECT, + }, + { +"calls: wrong src reg" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,2 ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "BPF_CALL uses reserved fields", + .result = REJECT, + }, + { +"calls: wrong off value" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,-1 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "BPF_CALL uses reserved fields", + .result = REJECT, + }, + { +"calls: jump back loop" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-1 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge from insn 0 to 0", + .result = REJECT, + }, + { +"calls: conditional call" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "jump out of range", + .result = REJECT, + }, + { +"calls: conditional call 2" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,2 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .result = ACCEPT, + }, + { +"calls: conditional call 3" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,3 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,-6 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), + BPF_JMP_IMM(BPF_JA, 0, 0, -6), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge from insn", + .result = REJECT, + }, + { +"calls: conditional call 4" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,-5 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .result = ACCEPT, + }, + { +"calls: conditional call 5" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,3 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_0 ,1 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,-6 ), +BPF_MOV64_IMM (BPF_REG_0 ,3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge from insn", + .result = REJECT, + }, + { +"calls: conditional call 6" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,-2 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 , + offsetof(struct __sk_buff , mark)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "back-edge from insn", + .result = REJECT, + }, + {"calls: using r0 returned by callee" , .insns = { BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1),
@@ -8122,6 +8296,17 @@ static struct bpf_test tests[] = { .result = ACCEPT, }, { + "calls: using uninit r0 from callee", + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "!read_ok", + .result = REJECT, + }, + {"calls: callee is using r1" , .insns = { BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1),
@@ -8224,6 +8409,71 @@ static struct bpf_test tests[] = { .result = ACCEPT, }, { + "calls: calls with stack arith", + .insns = { +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-64 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-64 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-64 ), +BPF_MOV64_IMM (BPF_REG_0 ,42 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: calls with misaligned stack access" , + .insns = { +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-63 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-61 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_2 ,-63 ), +BPF_MOV64_IMM (BPF_REG_0 ,42 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .flags = F_LOAD_WITH_STRICT_ALIGNMENT, + .errstr = "misaligned stack access", + .result = REJECT, + }, + { +"calls: calls control flow, jump test" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,42 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,43 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,-3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: calls control flow, jump test 2" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,42 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,2 ), +BPF_MOV64_IMM (BPF_REG_0 ,43 ), +BPF_JMP_IMM (BPF_JA ,0 ,0 ,1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,-3 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "jump out of range from insn 1 to 4", + .result = REJECT, + }, + {"calls: two calls with bad jump" , .insns = { BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1),
@@ -8298,6 +8548,18 @@ static struct bpf_test tests[] = { .result = REJECT, }, { + "calls: invalid call 2", + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,0x7fffffff ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_TRACEPOINT , + .errstr = "invalid destination", + .result = REJECT, + }, + {"calls: jumping across function bodies. test1" , .insns = { BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2),
@@ -8367,6 +8629,30 @@ static struct bpf_test tests[] = { .result = REJECT, }, { + "calls: ld_abs with changing ctx data in callee", + .insns = { +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_LD_ABS (BPF_B ,0 ), +BPF_LD_ABS (BPF_H ,0 ), +BPF_LD_ABS (BPF_W ,0 ), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_6 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,5 ), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_7 ), +BPF_LD_ABS (BPF_B ,0 ), +BPF_LD_ABS (BPF_H ,0 ), +BPF_LD_ABS (BPF_W ,0 ), +BPF_EXIT_INSN (), +BPF_MOV64_IMM (BPF_REG_2 ,1 ), +BPF_MOV64_IMM (BPF_REG_3 ,2 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , + BPF_FUNC_skb_vlan_push), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "BPF_LD_[ABS|IND] instructions cannot be mixed", + .result = REJECT, + }, + {"calls: two calls with bad fallthrough" , .insns = { BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1),
@@ -8460,6 +8746,36 @@ static struct bpf_test tests[] = { .result = REJECT, }, { + "calls: write into caller stack frame", + .insns = { +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_1 ,-8 ), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_6 ,0 ), +BPF_EXIT_INSN (), +BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,42 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_XDP , + .result = ACCEPT, + }, + { +"calls: write into callee stack frame" , + .insns = { +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,2 ), +BPF_ST_MEM (BPF_DW ,BPF_REG_0 ,0 ,42 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,-8 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_XDP , + .errstr = "cannot return stack pointer", + .result = REJECT, + }, + {"calls: two calls with stack write and void return" , .insns = {/* main prog */
@@ -9057,6 +9373,287 @@ static struct bpf_test tests[] = { .prog_type = BPF_PROG_TYPE_SCHED_CLS, }, { + "calls: pkt_ptr spill into caller stack 2", + .insns = { +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +/* Marking is still kept, but not in all cases safe. */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_ST_MEM (BPF_W ,BPF_REG_4 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +/* spill unchecked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,2 ), +/* now the pkt range is verified, read pkt_ptr from stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_4 ,0 ), +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "invalid access to packet", + .result = REJECT, + }, + { +"calls: pkt_ptr spill into caller stack 3" , + .insns = { +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,2 ), +/* Marking is still kept and safe here. */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_ST_MEM (BPF_W ,BPF_REG_4 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +/* spill unchecked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,3 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* now the pkt range is verified, read pkt_ptr from stack */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_4 ,0 ), +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: pkt_ptr spill into caller stack 4" , + .insns = { +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,4 ), +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,2 ), +/* Check marking propagated. */ +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_ST_MEM (BPF_W ,BPF_REG_4 ,0 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +/* spill unchecked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,2 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: pkt_ptr spill into caller stack 5" , + .insns = { +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_1 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_4 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,3 ), +/* spill checked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "same insn cannot be used with different", + .result = REJECT, + }, + { +"calls: pkt_ptr spill into caller stack 6" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_4 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,3 ), +/* spill checked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "R4 invalid mem access", + .result = REJECT, + }, + { +"calls: pkt_ptr spill into caller stack 7" , + .insns = { +BPF_MOV64_IMM (BPF_REG_2 ,0 ), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_4 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,3 ), +/* spill checked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "R4 invalid mem access", + .result = REJECT, + }, + { +"calls: pkt_ptr spill into caller stack 8" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_JMP_REG (BPF_JLE ,BPF_REG_0 ,BPF_REG_3 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_4 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,3 ), +/* spill checked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, + }, + { +"calls: pkt_ptr spill into caller stack 9" , + .insns = { +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_JMP_REG (BPF_JLE ,BPF_REG_0 ,BPF_REG_3 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_4 ,BPF_REG_10 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_4 ,-8 ), +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,1 ,0 ,3 ), +BPF_LDX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_10 ,-8 ), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_4 ,0 ), +BPF_EXIT_INSN (), + +/* subprog 1 */ +BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 , + offsetof(struct __sk_buff , data)), +BPF_LDX_MEM (BPF_W ,BPF_REG_3 ,BPF_REG_1 , + offsetof(struct __sk_buff , data_end)), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +BPF_ALU64_IMM (BPF_ADD ,BPF_REG_0 ,8 ), +BPF_MOV64_IMM (BPF_REG_5 ,0 ), +/* spill unchecked pkt_ptr into stack of caller */ +BPF_STX_MEM (BPF_DW ,BPF_REG_4 ,BPF_REG_2 ,0 ), +BPF_JMP_REG (BPF_JGT ,BPF_REG_0 ,BPF_REG_3 ,2 ), +BPF_MOV64_IMM (BPF_REG_5 ,1 ), +/* don't read back pkt_ptr from stack here */ +/* write 4 bytes into packet */ +BPF_ST_MEM (BPF_W ,BPF_REG_2 ,0 ,0 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_5 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .errstr = "invalid access to packet", + .result = REJECT, + }, + {"calls: caller stack init to zero or map_value_or_null" , .insns = { BPF_MOV64_IMM(BPF_REG_0, 0),
Signature · 1 line
2.9.5
Daniel Borkmann
maintainerQuoted 60 lines · Alexei Starovoitov
First of all huge thank you to Daniel, John, Jakub, Edward and others who reviewed multiple iterations of this patch set over the last many months and to Dave and others who gave critical feedback during netconf/netdev.
The patch is solid enough and we thought through numerous corner cases, but it's not the end. More followups with code reorg and features to follow.
TLDR: Allow arbitrary function calls from bpf function to another bpf function.
Since the beginning of bpf all bpf programs were represented as a single function and program authors were forced to use
always_inlinefor all functions in their C code. That was causing llvm to unnecessary inflate the code size and forcing developers to move code to header files with little code reuse.With a bit of additional complexity teach verifier to recognize arbitrary function calls from one bpf function to another as long as all of functions are presented to the verifier as a single bpf program. Extended program layout:
..r1 = .. // arg1 r2 = .. // arg2 call pc+1 // function call pc-relativeexit
.. = r1 // access arg1
.. = r2 // access arg2
..
call pc+20 // second level of function call
...It allows for better optimized code and finally allows to introduce the core bpf libraries that can be reused in different projects, since programs are no longer limited by single elf file. With function calls bpf can be compiled into multiple .o files.
This patch is the first step. It detects programs that contain multiple functions and checks that calls between them are valid. It splits the sequence of bpf instructions (one program) into a set of bpf functions that call each other. Calls to only known functions are allowed. Since all functions are presented to the verifier at once conceptually it is 'static linking'.
Future plans:
- introduceBPF_PROG_TYPE_LIBRARYand allow a set of bpf functions to be loaded into the kernel that can be later linked to other programs with concrete program types. Aka 'dynamic linking'.- introduce function pointer type and indirect calls to allow bpf functions call other dynamically loaded bpf functions while the caller bpf function is already executing. Aka 'runtime linking'. This will be more generic and more flexible alternative to
bpf_tail_calls.FAQ:
Q: Interpreter and JIT changes mean that new instruction is introduced ?
A: No. The call instruction technically stays the same. Now it can call
both kernel helpers and other bpf functions.
Calling convention stays the same as well.
From uapi point of view the call insn got new 'relocation'BPF_PSEUDO_CALLsimilar toBPF_PSEUDO_MAP_FD'relocation' ofbpf_ldimm64insn.
Quoted 7 lines
Q: What had to change on LLVM side?
A: Trivial LLVM patch to allow calls was applied to upcoming 6.0 release:
https://reviews.llvm.org/rL318614
with few bugfixes as well.
Make sure to build the latest llvm to havebpf_callsupport.More details in the patches.
Series applied to bpf-next, thanks Alexei!