bpf: Support new insns from cpu v4
Subsystems (9)
Yonghong Song
reviewerv4 cover bpf: Support new insns from cpu v4
In previous discussion ([1]), it is agreed that we should introduce cpu version 4 (llvm flag -mcpu=v4) which contains some instructions which can simplify code, make code easier to understand, fix the existing problem, or simply for feature completeness. More specifically, the following new insns are proposed:
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offset
This patch set added kernel support for insns proposed in [1] except BPF_ST which already has full kernel support. Beside the above proposed insns, LLVM will generate BPF_ST insn as well under -mcpu=v4. The llvm patch ([2]) has been merged into llvm-project 'main' branch.
The patchset implements interpreter, jit and verifier support for these new insns.
For this patch set, I tested cpu v2/v3/v4 and the selftests are all passed. I also tested selftests introduced in this patch set with additional changes beside normal jit testing (bpf_jit_enable = 1 and bpf_jit_harden = 0)
- bpf_jit_enable = 0
- bpf_jit_enable = 1 and bpf_jit_harden = 1
and both testing passed.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://reviews.llvm.org/D144829
Changelogs:
v3 -> v4:
. some minor asm syntax adjustment based on llvm change. . add clang version and target arch guard for new tests
so they can still compile with old llvm compilers.
. some changes to the bpf doc.
v2 -> v3:
. add missed disasm change from v2.
. handle signed load of ctx fields properly.
. fix some interpreter sdiv/smod error when bpf_jit_enable = 0. . fix some verifier range bounding errors.
. add more C tests.
RFCv1 -> v2:
. add more verifier supports for signed extend load and mov insns. . rename some insn names to be more consistent with intel practice. . add cpuv4 test runner for test progs.
. add more unit and C tests.
. add documentation.
Yonghong Song (17):
bpf: Support new sign-extension load insns
bpf: Support new sign-extension mov insns
bpf: Handle sign-extenstin ctx member accesses
bpf: Support new unconditional bswap instruction
bpf: Support new signed div/mod instructions.
bpf: Fix jit blinding with new sdiv/smov insns
bpf: Support new 32bit offset jmp instruction
bpf: Add kernel/bpftool asm support for new instructions selftests/bpf: Fix a test_verifier failure selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing selftests/bpf: Add unit tests for new sign-extension load insns selftests/bpf: Add unit tests for new sign-extension mov insns selftests/bpf: Add unit tests for new bswap insns selftests/bpf: Add unit tests for new sdiv/smod insns selftests/bpf: Add unit tests for new gotol insn selftests/bpf: Test ldsx with more complex cases docs/bpf: Add documentation for new instructions
Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 115 ++- arch/x86/net/bpf_jit_comp.c | 141 +++- include/linux/filter.h | 17 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 196 ++++- kernel/bpf/disasm.c | 57 +- kernel/bpf/verifier.c | 348 ++++++-- tools/include/uapi/linux/bpf.h | 1 + tools/testing/selftests/bpf/.gitignore | 2 + tools/testing/selftests/bpf/Makefile | 28 +- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 139 ++++ .../selftests/bpf/prog_tests/verifier.c | 10 + .../selftests/bpf/progs/test_ldsx_insn.c | 118 +++ .../selftests/bpf/progs/verifier_bswap.c | 59 ++ .../selftests/bpf/progs/verifier_gotol.c | 44 + .../selftests/bpf/progs/verifier_ldsx.c | 131 +++ .../selftests/bpf/progs/verifier_movsx.c | 213 +++++ .../selftests/bpf/progs/verifier_sdiv.c | 781 ++++++++++++++++++ .../selftests/bpf/verifier/basic_instr.c | 6 +- 21 files changed, 2251 insertions(+), 170 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_bswap.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotol.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_movsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.c
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 1/17 bpf: Support new sign-extension load insns
Add interpreter/jit support for new sign-extension load insns which adds a new mode (BPF_MEMSX).
Also add verifier support to recognize these insns and to do proper verification with new insns. In verifier, besides to deduce proper bounds for the dst_reg, probed memory access is also properly handled.
--- arch/x86/net/bpf_jit_comp.c | 42 +++++++++- include/linux/filter.h | 3 + include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 21 +++++ kernel/bpf/verifier.c | 140 +++++++++++++++++++++++++++------ tools/include/uapi/linux/bpf.h | 1 + 6 files changed, 181 insertions(+), 27 deletions(-)
arch/x86/net/bpf_jit_comp.c +39 -3
@@ -779,6 +779,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* LDSX: dst_reg = *(s8*)(src_reg + off) */ +static void emit_ldsx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; + +switch (size) { +case BPF_B : +/* Emit 'movsx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBE ); +break ; +case BPF_H : +/* Emit 'movsx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBF ); +break ; +case BPF_W : +/* Emit 'movsx rax, dword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x63 ); +break ; + } +emit_insn_suffix (&prog, src_reg, dst_reg, off); + *pprog = prog; +} +/* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) {
@@ -1370,9 +1393,17 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : +/* LDXS: dst_reg = *(s8*)(src_reg + off) */ +case BPF_LDX |BPF_MEMSX |BPF_B : +case BPF_LDX |BPF_MEMSX |BPF_H : +case BPF_LDX |BPF_MEMSX |BPF_W : +case BPF_LDX |BPF_PROBE_MEMSX |BPF_B : +case BPF_LDX |BPF_PROBE_MEMSX |BPF_H : +case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; -if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) { +if (BPF_MODE (insn->code) ==BPF_PROBE_MEM || +BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {/* Conservatively check that src_reg + insn->off is a kernel address: * src_reg + insn->off >= TASK_SIZE_MAX + PAGE_SIZE * src_reg is used as scratch for src_reg += insn->off and restored
@@ -1415,8 +1446,13 @@ st: if (is_imm8(insn->off)) start_of_ldx = prog; end_of_jmp[-1 ] = start_of_ldx - end_of_jmp; } -emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); -if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) { +if (BPF_MODE (insn->code) ==BPF_PROBE_MEMSX || +BPF_MODE (insn->code) ==BPF_MEMSX ) +emit_ldsx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +else +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +if (BPF_MODE (insn->code) ==BPF_PROBE_MEM || +BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {struct exception_table_entry *ex;u8 *_insn = image + proglen + (start_of_ldx - temp);s64 delta;
include/linux/filter.h +3 -0
@@ -69,6 +69,9 @@ struct ctl_table_header;/* unused opcode to mark special load instruction. Same as BPF_ABS */ #define BPF_PROBE_MEM 0x20 +/* unused opcode to mark special ldsx instruction. Same as BPF_IND */ +#define BPF_PROBE_MEMSX 0x40 +/* unused opcode to mark call to interpreter with arguments */ #define BPF_CALL_ARGS 0xe0
include/uapi/linux/bpf.h +1 -0
@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
kernel/bpf/core.c +21 -0
@@ -1610,6 +1610,9 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (LDX ,MEM ,H ), \INSN_3 (LDX ,MEM ,W ), \INSN_3 (LDX ,MEM ,DW ), \ +INSN_3 (LDX ,MEMSX ,B ), \ +INSN_3 (LDX ,MEMSX ,H ), \ +INSN_3 (LDX ,MEMSX ,W ), \/* Immediate based. */ \INSN_3 (LD ,IMM ,DW )
@@ -1666,6 +1669,9 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) [BPF_LDX | BPF_PROBE_MEM | BPF_H] = &&LDX_PROBE_MEM_H, [BPF_LDX | BPF_PROBE_MEM | BPF_W] = &&LDX_PROBE_MEM_W, [BPF_LDX | BPF_PROBE_MEM | BPF_DW] = &&LDX_PROBE_MEM_DW, + [BPF_LDX | BPF_PROBE_MEMSX | BPF_B] = &&LDX_PROBE_MEMSX_B, + [BPF_LDX | BPF_PROBE_MEMSX | BPF_H] = &&LDX_PROBE_MEMSX_H, + [BPF_LDX | BPF_PROBE_MEMSX | BPF_W] = &&LDX_PROBE_MEMSX_W, };#undef BPF_INSN_3_LBL#undef BPF_INSN_2_LBL
@@ -1942,6 +1948,21 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDSX (SIZEOP ,SIZE ) \ + LDX_MEMSX_##SIZEOP: \ + DST = *(SIZE *)(unsigned long) (SRC + insn->off); \ + CONT; \ + LDX_PROBE_MEMSX_##SIZEOP: \ + bpf_probe_read_kernel(&DST, sizeof(SIZE), \ + (const void *)(long) (SRC + insn->off)); \ + DST = *((SIZE *)&DST); \ + CONT; + + LDSX(B, s8) + LDSX(H, s16) + LDSX(W, s32) +#undef LDSX +#define ATOMIC_ALU_OP (BOP ,KOP ) \ case BOP: \ if (BPF_SIZE(insn->code) == BPF_W) \
kernel/bpf/verifier.c +116 -24
266 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +1 -0
@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 2/17 bpf: Support new sign-extension mov insns
In reply to Yonghong SongAdd interpreter/jit support for new sign-extension mov insns. The original 'MOV' insn is extended to support reg-to-reg signed version for both ALU and ALU64 operations. For ALU mode, the insn->off value of 8 or 16 indicates sign-extension from 8- or 16-bit value to 32-bit value. For ALU64 mode, the insn->off value of 8/16/32 indicates sign-extension from 8-, 16- or 32-bit value to 64-bit value.
--- arch/x86/net/bpf_jit_comp.c | 43 +++++++++- kernel/bpf/core.c | 28 ++++++- kernel/bpf/verifier.c | 157 ++++++++++++++++++++++++++++++------ 3 files changed, 197 insertions(+), 31 deletions(-)
arch/x86/net/bpf_jit_comp.c +40 -3
@@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) *pprog = prog; } +static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); + }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); +EMIT3 (add_2mod (0x0f , src_reg, dst_reg),0xbf , +add_2reg (0xC0 , src_reg, dst_reg)); + } + } + + *pprog = prog; +} +/* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */ static void emit_insn_suffix (u8 **pprog,u32 ptr_reg,u32 val_reg,int off) {
@@ -1051,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ALU64 |BPF_MOV |BPF_X :case BPF_ALU |BPF_MOV |BPF_X : -emit_mov_reg (&prog, -BPF_CLASS (insn->code) ==BPF_ALU64 , - dst_reg, src_reg); +if (insn->off ==0 ) +emit_mov_reg (&prog, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg); +else +emit_movsx_reg (&prog, insn->off, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg);break ;/* neg dst */
kernel/bpf/core.c +26 -2
@@ -61,6 +61,7 @@#define AX regs[BPF_REG_AX]#define ARG1 regs[BPF_REG_ARG1]#define CTX regs[BPF_REG_CTX] +#define OFF insn->off#define IMM insn->immstruct bpf_mem_alloc bpf_global_ma;
@@ -1739,13 +1740,36 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)DST = -DST ;CONT ; ALU_MOV_X: -DST = (u32 )SRC ; +switch (OFF ) { +case 0 : +DST = (u32 )SRC ; +break ; +case 8 : +DST = (u32 )(s8 )SRC ; +break ; +case 16 : +DST = (u32 )(s16 )SRC ; +break ; + }CONT ; ALU_MOV_K:DST = (u32 )IMM ;CONT ; ALU64_MOV_X: -DST =SRC ; +switch (OFF ) { +case 0 : +DST =SRC ; +break ; +case 8 : +DST = (s8 )SRC ; +break ; +case 16 : +DST = (s16 )SRC ; +break ; +case 32 : +DST = (s32 )SRC ; +break ; + }CONT ; ALU64_MOV_K:DST =IMM ;
kernel/bpf/verifier.c +131 -26
192 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 3/17 bpf: Handle sign-extenstin ctx member accesses
In reply to Yonghong SongCurrently, if user accesses a ctx member with signed types, the compiler will generate an unsigned load followed by necessary left and right shifts.
With the introduction of sign-extension load, compiler may just emit a ldsx insn instead. Let us do a final movsx sign extension to the final unsigned ctx load result to satisfy original sign extension requirement.
--- kernel/bpf/verifier.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/bpf/verifier.c +6 -0
@@ -17716,6 +17716,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)for (i =0 ; i < insn_cnt; i++, insn++) {bpf_convert_ctx_access_t convert_ctx_access; +u8 mode; if (insn->code == (BPF_LDX | BPF_MEM | BPF_B) || insn->code == (BPF_LDX | BPF_MEM | BPF_H) ||
@@ -17797,6 +17798,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) ctx_field_size = env->insn_aux_data[i + delta].ctx_field_size; size =BPF_LDST_BYTES (insn); + mode =BPF_MODE (insn->code);/* If the read access is a narrower load of the field, * convert to a 4/8-byte load, to minimum program type specific
@@ -17856,6 +17858,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) (1ULL << size * 8) - 1); } } +if (mode ==BPF_MEMSX ) + insn_buf[cnt++] =BPF_RAW_INSN (BPF_ALU64 |BPF_MOV |BPF_X , + insn->dst_reg, insn->dst_reg, + size *8 ,0 ); new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt);if (!new_prog)
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 4/17 bpf: Support new unconditional bswap instruction
In reply to Yonghong SongThe existing 'be' and 'le' insns will do conditional bswap depends on host endianness. This patch implements unconditional bswap insns.
--- arch/x86/net/bpf_jit_comp.c | 1 + kernel/bpf/core.c | 14 ++++++++++++++ kernel/bpf/verifier.c | 7 +++++-- 3 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/net/bpf_jit_comp.c +1 -0
@@ -1322,6 +1322,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagebreak ;case BPF_ALU |BPF_END |BPF_FROM_BE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm32) {case 16 :/* Emit 'ror %ax, 8' to swap lower 2 bytes */
kernel/bpf/core.c +14 -0
@@ -1524,6 +1524,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (ALU64 ,DIV ,X ), \INSN_3 (ALU64 ,MOD ,X ), \INSN_2 (ALU64 ,NEG ), \ +INSN_3 (ALU64 ,END ,TO_LE ), \/* Immediate based. */ \INSN_3 (ALU64 ,ADD ,K ), \ INSN_3(ALU64, SUB, K), \
@@ -1848,6 +1849,19 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)break ; }CONT ; + ALU64_END_TO_LE: +switch (IMM ) { +case 16 : +DST = (__force u16 )__swab16 (DST ); +break ; +case 32 : +DST = (__force u32 )__swab32 (DST ); +break ; +case 64 : +DST = (__force u64 )__swab64 (DST ); +break ; + } +CONT ;/* CALL */ JMP_CALL:
kernel/bpf/verifier.c +5 -2
@@ -3012,8 +3012,10 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, } } +if (class ==BPF_ALU64 && op ==BPF_END && (insn->imm ==16 || insn->imm ==32 )) +return false; +if (class ==BPF_ALU64 || class ==BPF_JMP || -/* BPF_END always use BPF_ALU class. */ (class ==BPF_ALU && op ==BPF_END && insn->imm ==64 ))return true;
@@ -13076,7 +13078,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {if (insn->src_reg !=BPF_REG_0 || insn->off !=0 || (insn->imm !=16 && insn->imm !=32 && insn->imm !=64 ) || -BPF_CLASS (insn->code) ==BPF_ALU64 ) { + (BPF_CLASS (insn->code) ==BPF_ALU64 && +BPF_SRC (insn->code) !=BPF_TO_LE )) {verbose (env,"BPF_END uses reserved fields\n" );return -EINVAL ; }
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 5/17 bpf: Support new signed div/mod instructions.
In reply to Yonghong SongAdd interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus insn->off == 1. Also add basic verifier support to ensure new insns get accepted.
--- arch/x86/net/bpf_jit_comp.c | 27 ++++++--- kernel/bpf/core.c | 110 ++++++++++++++++++++++++++++++------ kernel/bpf/verifier.c | 6 +- 3 files changed, 117 insertions(+), 26 deletions(-)
arch/x86/net/bpf_jit_comp.c +19 -8
@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 ); -/* div src_reg */ -maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cdq */ +else +EMIT2 (0x48 ,0x99 );/* cqo */ + +/* idiv src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF8 , src_reg)); + }if (BPF_OP (insn->code) ==BPF_MOD && dst_reg !=BPF_REG_3 )
kernel/bpf/core.c +94 -16
130 lines not on this page · raw message
kernel/bpf/verifier.c +4 -2
@@ -13237,7 +13237,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {/* all other ALU ops: and, sub, xor, add, ... */ if (BPF_SRC (insn->code) ==BPF_X ) { -if (insn->imm !=0 || insn->off !=0 ) { +if (insn->imm !=0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }
@@ -13246,7 +13247,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)if (err)return err; }else { -if (insn->src_reg !=BPF_REG_0 || insn->off !=0 ) { +if (insn->src_reg !=BPF_REG_0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 6/17 bpf: Fix jit blinding with new sdiv/smov insns
In reply to Yonghong SongHandle new insns properly in bpf_jit_blind_insn() function.
--- include/linux/filter.h | 14 ++++++++++---- kernel/bpf/core.c | 4 ++-- 2 files changed, 12 insertions(+), 6 deletions(-)
include/linux/filter.h +10 -4
@@ -93,22 +93,28 @@ struct ctl_table_header;/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */ -#define BPF_ALU64_REG (OP ,DST ,SRC ) \ +#define BPF_ALU64_REG_OFF (OP ,DST ,SRC ,OFF ) \ ((struct bpf_insn) { \ .code = BPF_ALU64 | BPF_OP(OP) | BPF_X, \ .dst_reg = DST, \ .src_reg = SRC, \ - .off = 0, \ + .off = OFF, \ .imm = 0 }) -#define BPF_ALU32_REG (OP ,DST ,SRC ) \ +#define BPF_ALU64_REG (OP ,DST ,SRC ) \ + BPF_ALU64_REG_OFF(OP, DST, SRC, 0) + +#define BPF_ALU32_REG_OFF (OP ,DST ,SRC ,OFF ) \ ((struct bpf_insn) { \ .code = BPF_ALU | BPF_OP(OP) | BPF_X, \ .dst_reg = DST, \ .src_reg = SRC, \ - .off = 0, \ + .off = OFF, \ .imm = 0 }) +#define BPF_ALU32_REG (OP ,DST ,SRC ) \ + BPF_ALU32_REG_OFF(OP, DST, SRC, 0) +/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */ #define BPF_ALU64_IMM (OP ,DST ,IMM ) \
kernel/bpf/core.c +2 -2
@@ -1272,7 +1272,7 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,case BPF_ALU |BPF_MOD |BPF_K : *to++ =BPF_ALU32_IMM (BPF_MOV ,BPF_REG_AX , imm_rnd ^ from->imm); *to++ =BPF_ALU32_IMM (BPF_XOR ,BPF_REG_AX , imm_rnd); - *to++ =BPF_ALU32_REG (from->code, from->dst_reg,BPF_REG_AX ); + *to++ =BPF_ALU32_REG_OFF (from->code, from->dst_reg,BPF_REG_AX , from->off);break ;case BPF_ALU64 |BPF_ADD |BPF_K :
@@ -1286,7 +1286,7 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,case BPF_ALU64 |BPF_MOD |BPF_K : *to++ =BPF_ALU64_IMM (BPF_MOV ,BPF_REG_AX , imm_rnd ^ from->imm); *to++ =BPF_ALU64_IMM (BPF_XOR ,BPF_REG_AX , imm_rnd); - *to++ =BPF_ALU64_REG (from->code, from->dst_reg,BPF_REG_AX ); + *to++ =BPF_ALU64_REG_OFF (from->code, from->dst_reg,BPF_REG_AX , from->off);break ;case BPF_JMP |BPF_JEQ |BPF_K :
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 7/17 bpf: Support new 32bit offset jmp instruction
In reply to Yonghong SongAdd interpreter/jit/verifier support for 32bit offset jmp instruction. If a conditional jmp instruction needs more than 16bit offset, it can be simulated with a conditional jmp + a 32bit jmp insn.
--- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++---------- kernel/bpf/core.c | 19 ++++++++++++++++--- kernel/bpf/verifier.c | 32 ++++++++++++++++++++++---------- 3 files changed, 56 insertions(+), 23 deletions(-)
arch/x86/net/bpf_jit_comp.c +18 -10
@@ -1815,16 +1815,24 @@ st: if (is_imm8(insn->off))break ;case BPF_JMP |BPF_JA : -if (insn->off ==-1 ) -/* -1 jmp instructions will always jump -* backwards two bytes. Explicitly handling -* this case avoids wasting too many passes -* when there are long sequences of replaced -* dead code. -*/ - jmp_offset =-2 ; -else - jmp_offset = addrs[i + insn->off] - addrs[i]; +case BPF_JMP32 |BPF_JA : +if (BPF_CLASS (insn->code) ==BPF_JMP ) { +if (insn->off ==-1 ) +/* -1 jmp instructions will always jump +* backwards two bytes. Explicitly handling +* this case avoids wasting too many passes +* when there are long sequences of replaced +* dead code. +*/ + jmp_offset =-2 ; +else + jmp_offset = addrs[i + insn->off] - addrs[i]; + }else { +if (insn->imm ==-1 ) + jmp_offset =-2 ; +else + jmp_offset = addrs[i + insn->imm] - addrs[i]; + }if (!jmp_offset) {/*
kernel/bpf/core.c +16 -3
@@ -373,7 +373,12 @@ static int bpf_adj_delta_to_off(struct bpf_insn *insn, u32 pos, s32 end_old, {const s32 off_min =S16_MIN , off_max =S16_MAX ;s32 delta = end_new - end_old; -s32 off = insn->off; +s32 off; + +if (insn->code == (BPF_JMP32 |BPF_JA )) + off = insn->imm; +else + off = insn->off;if (curr < pos && curr + off +1 >= end_old) off += delta;
@@ -381,8 +386,12 @@ static int bpf_adj_delta_to_off(struct bpf_insn *insn, u32 pos, s32 end_old, off -= delta;if (off < off_min || off > off_max)return -ERANGE ; -if (!probe_pass) - insn->off = off; +if (!probe_pass) { +if (insn->code == (BPF_JMP32 |BPF_JA )) + insn->imm = off; +else + insn->off = off; + }return 0 ; }
@@ -1593,6 +1602,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (JMP ,JSLE ,K ), \INSN_3 (JMP ,JSET ,K ), \INSN_2 (JMP ,JA ), \ +INSN_2 (JMP32 ,JA ), \/* Store instructions. */ \/* Register based. */ \ INSN_3(STX, MEM, B), \
@@ -1989,6 +1999,9 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) JMP_JA: insn += insn->off;CONT ; + JMP32_JA: + insn += insn->imm; +CONT ; JMP_EXIT:return BPF_R0 ;/* JMP */
kernel/bpf/verifier.c +22 -10
@@ -2855,7 +2855,10 @@ static int check_subprogs(struct bpf_verifier_env *env) goto next;if (BPF_OP (code) ==BPF_EXIT ||BPF_OP (code) ==BPF_CALL ) goto next; - off = i + insn[i].off +1 ; +if (code == (BPF_JMP32 |BPF_JA )) + off = i + insn[i].imm +1 ; +else + off = i + insn[i].off +1 ;if (off < subprog_start || off >= subprog_end) {verbose (env,"jump out of range from insn %d to %d\n" , i, off);return -EINVAL ;
@@ -2867,6 +2870,7 @@ static int check_subprogs(struct bpf_verifier_env *env)* or unconditional jump back */ if (code != (BPF_JMP |BPF_EXIT ) && + code != (BPF_JMP32 |BPF_JA ) && code != (BPF_JMP |BPF_JA )) {verbose (env,"last insn is not an exit or jmp\n" );return -EINVAL ;
@@ -14792,7 +14796,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,static int visit_insn (int t,struct bpf_verifier_env *env) {struct bpf_insn *insns = env->prog->insnsi, *insn = &insns[t]; -int ret; +int ret, off;if (bpf_pseudo_func (insn))return visit_func_call_insn (t, insns, env, true);
@@ -14840,14 +14844,19 @@ static int visit_insn(int t, struct bpf_verifier_env *env)if (BPF_SRC (insn->code) !=BPF_K )return -EINVAL ; +if (BPF_CLASS (insn->code) ==BPF_JMP ) + off = insn->off; +else + off = insn->imm; +/* unconditional jump with single edge */ - ret =push_insn (t, t + insn->off +1 ,FALLTHROUGH , env, + ret =push_insn (t, t + off +1 ,FALLTHROUGH , env, true);if (ret)return ret; -mark_prune_point (env, t + insn->off +1 ); -mark_jmp_point (env, t + insn->off +1 ); +mark_prune_point (env, t + off +1 ); +mark_jmp_point (env, t + off +1 );return ret;
@@ -16643,15 +16652,18 @@ static int do_check(struct bpf_verifier_env *env)mark_reg_scratched (env,BPF_REG_0 ); }else if (opcode ==BPF_JA ) {if (BPF_SRC (insn->code) !=BPF_K || - insn->imm !=0 || insn->src_reg !=BPF_REG_0 || insn->dst_reg !=BPF_REG_0 || - class ==BPF_JMP32 ) { + (class ==BPF_JMP && insn->imm !=0 ) || + (class ==BPF_JMP32 && insn->off !=0 )) {verbose (env,"BPF_JA uses reserved fields\n" );return -EINVAL ; } - env->insn_idx += insn->off +1 ; +if (class ==BPF_JMP ) + env->insn_idx += insn->off +1 ; +else + env->insn_idx += insn->imm +1 ;continue ; }else if (opcode ==BPF_EXIT ) {
@@ -17498,13 +17510,13 @@ static bool insn_is_cond_jump(u8 code) {u8 op; + op =BPF_OP (code);if (BPF_CLASS (code) ==BPF_JMP32 ) -return true; +return op !=BPF_JA ;if (BPF_CLASS (code) !=BPF_JMP )return false; - op =BPF_OP (code);return op !=BPF_JA && op !=BPF_EXIT && op !=BPF_CALL ; }
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 9/17 selftests/bpf: Fix a test_verifier failure
In reply to Yonghong SongThe following test_verifier subtest failed due to
new encoding for BSWAP.
$ ./test_verifier ... #99/u invalid 64-bit BPF_END FAIL Unexpected success to load! verification time 215 usec stack depth 0 processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 #99/p invalid 64-bit BPF_END FAIL Unexpected success to load! verification time 198 usec stack depth 0 processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
Tighten the test so it still reports a failure.
--- tools/testing/selftests/bpf/verifier/basic_instr.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/verifier/basic_instr.c +3 -3
@@ -176,11 +176,11 @@ .retval = 1, }, { - "invalid 64-bit BPF_END", + "invalid 64-bit BPF_END with BPF_TO_BE", .insns = { BPF_MOV32_IMM(BPF_REG_0, 0), { - .code = BPF_ALU64 | BPF_END | BPF_TO_LE, + .code = BPF_ALU64 | BPF_END | BPF_TO_BE, .dst_reg = BPF_REG_0, .src_reg = 0, .off = 0,
@@ -188,7 +188,7 @@ }, BPF_EXIT_INSN(), }, - .errstr = "unknown opcode d7", + .errstr = "unknown opcode df", .result = REJECT, }, {
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 10/17 selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing
In reply to Yonghong SongSimilar to no-alu32 runner, if clang compiler supports -mcpu=v4, a cpuv4 runner is created to test bpf programs compiled with -mcpu=v4.
The following are some num-of-insn statistics for each newer instructions based on existing selftests, excluding subsequent cpuv4 insn specific tests.
insn pattern # of instructions reg = (s8)reg 4 reg = (s16)reg 4 reg = (s32)reg 144 reg = *(s8 *)(reg + off) 13 reg = *(s16 *)(reg + off) 14 reg = *(s32 *)(reg + off) 15215 reg = bswap16 reg 142 reg = bswap32 reg 38 reg = bswap64 reg 14 reg s/= reg 0 reg s%= reg 0 gotol <offset> 58
Note that in llvm -mcpu=v4 implementation, the compiler is a little bit conservative about generating 'gotol' insn (32-bit branch offset) as it didn't precise count the number of insns (e.g., some insns are debug insns, etc.). Compared to old 'goto' insn, newer 'gotol' insn should have comparable verification states to 'goto' insn.
With current patch set, all selftests passed with -mcpu=v4 when running test_progs-cpuv4 binary. The -mcpu=v3 and -mcpu=v2 run are also successful.
--- tools/testing/selftests/bpf/.gitignore | 2 ++ tools/testing/selftests/bpf/Makefile | 28 ++++++++++++++++++++++---- 2 files changed, 26 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/.gitignore +2 -0
@@ -13,6 +13,7 @@ test_dev_cgroup /test_progs /test_progs-no_alu32 /test_progs-bpf_gcc +/test_progs-cpuv4 test_verifier_log feature test_sock
@@ -36,6 +37,7 @@ test_cpp *.lskel.h /no_alu32 /bpf_gcc +/cpuv4 /host-tools /tools /runqslower
tools/testing/selftests/bpf/Makefile +24 -4
@@ -33,9 +33,13 @@ CFLAGS += -g -O0 -rdynamic -Wall -Werror $(GENFLAGS) $(SAN_CFLAGS) \ LDFLAGS += $(SAN_LDFLAGS) LDLIBS += -lelf -lz -lrt -lpthread -# Silence some warnings when compiled with clang ifneq ($(LLVM),) +# Silence some warnings when compiled with clang CFLAGS += -Wno-unused-command-line-argument +# Check whether cpu=v4 is supported or not by clang +ifneq ($(shell $(CLANG) --target=bpf -mcpu=help 2>&1 | grep 'v4'),) +CLANG_CPUV4 := 1 +endif endif # Order correspond to 'make run_tests' order
@@ -51,6 +55,10 @@ ifneq ($(BPF_GCC),) TEST_GEN_PROGS += test_progs-bpf_gcc endif +ifneq ($(CLANG_CPUV4),) +TEST_GEN_PROGS += test_progs-cpuv4 +endif + TEST_GEN_FILES = test_lwt_ip_encap.bpf.o test_tc_edt.bpf.o TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c)
@@ -383,6 +391,11 @@ define CLANG_NOALU32_BPF_BUILD_RULE $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) $(Q)$(CLANG) $3 -O2 --target=bpf -c $1 -mcpu=v2 -o $2 endef +# Similar to CLANG_BPF_BUILD_RULE, but with cpu-v4 +define CLANG_CPUV4_BPF_BUILD_RULE + $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) + $(Q)$(CLANG) $3 -O2 --target=bpf -c $1 -mcpu=v4 -o $2 +endef # Build BPF object using GCC define GCC_BPF_BUILD_RULE $(call msg,GCC-BPF,$(TRUNNER_BINARY),$2)
@@ -425,7 +438,7 @@ LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(ske # $eval()) and pass control to DEFINE_TEST_RUNNER_RULES. # Parameters: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER TRUNNER_OUTPUT := $(OUTPUT)$(if $2,/)$2
@@ -453,7 +466,7 @@ endef # Using TRUNNER_XXX variables, provided by callers of DEFINE_TEST_RUNNER and # set up by DEFINE_TEST_RUNNER itself, create test runner build rules with: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER_RULES ifeq ($($(TRUNNER_OUTPUT)-dir),)
@@ -584,6 +597,13 @@ TRUNNER_BPF_BUILD_RULE := CLANG_NOALU32_BPF_BUILD_RULE TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) $(eval $(call DEFINE_TEST_RUNNER,test_progs,no_alu32)) +# Define test_progs-cpuv4 test runner. +ifneq ($(CLANG_CPUV4),) +TRUNNER_BPF_BUILD_RULE := CLANG_CPUV4_BPF_BUILD_RULE +TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) +$(eval $(call DEFINE_TEST_RUNNER,test_progs,cpuv4)) +endif + # Define test_progs BPF-GCC-flavored test runner. ifneq ($(BPF_GCC),) TRUNNER_BPF_BUILD_RULE := GCC_BPF_BUILD_RULE
@@ -681,7 +701,7 @@ EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \ prog_tests/tests.h map_tests/tests.h verifier/tests.h \ feature bpftool \ $(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h *.subskel.h \ - no_alu32 bpf_gcc bpf_testmod.ko \ + no_alu32 cpuv4 bpf_gcc bpf_testmod.ko \ liburandom_read.so) .PHONY: docs docs-clean
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 11/17 selftests/bpf: Add unit tests for new sign-extension load insns
In reply to Yonghong SongAdd unit tests for new ldsx insns. The test includes sign-extension with a single value or with a value range.
If cpuv4 is not supported due to
(1) older compiler, e.g., less than clang version 18, or
(2) test runner test_progs and test_progs-no_alu32 which tests cpu v2 and v3, or
(3) non-x86_64 arch not supporting new insns in jit yet,
a dummy program is added with below output:
#318/1 verifier_ldsx/cpuv4 is not supported by compiler or jit, use a dummy test:OK
#318 verifier_ldsx:OK
to indicate the test passed with a dummy test instead of actually testing cpuv4. I am using a dummy prog to avoid changing the verifier testing infrastructure. Once clang 18 is widely available and other architectures support cpuv4, at least for CI run, the dummy program can be removed.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_ldsx.c | 131 ++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -31,6 +31,7 @@#include "verifier_int_ptr.skel.h" #include "verifier_jeq_infer_not_null.skel.h" #include "verifier_ld_ind.skel.h" +#include "verifier_ldsx.skel.h" #include "verifier_leak_ptr.skel.h" #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h"
@@ -133,6 +134,7 @@ void test_verifier_helper_value_access(void) { RUN(verifier_helper_value_accessvoid test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }void test_verifier_ld_ind (void ) {RUN (verifier_ld_ind); } +void test_verifier_ldsx (void ) {RUN (verifier_ldsx); }void test_verifier_leak_ptr (void ) {RUN (verifier_leak_ptr); }void test_verifier_loops1 (void ) {RUN (verifier_loops1); }void test_verifier_lwt (void ) {RUN (verifier_lwt); }
tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 12/17 selftests/bpf: Add unit tests for new sign-extension mov insns
In reply to Yonghong SongAdd unit tests for movsx insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_movsx.c | 213 ++++++++++++++++++ 2 files changed, 215 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_movsx.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -41,6 +41,7 @@#include "verifier_map_ret_val.skel.h" #include "verifier_masking.skel.h" #include "verifier_meta_access.skel.h" +#include "verifier_movsx.skel.h" #include "verifier_netfilter_ctx.skel.h" #include "verifier_netfilter_retcode.skel.h" #include "verifier_prevent_map_lookup.skel.h"
@@ -144,6 +145,7 @@ void test_verifier_map_ptr_mixing(void) { RUN(verifier_map_ptr_mixing); }void test_verifier_map_ret_val (void ) {RUN (verifier_map_ret_val); }void test_verifier_masking (void ) {RUN (verifier_masking); }void test_verifier_meta_access (void ) {RUN (verifier_meta_access); } +void test_verifier_movsx (void ) {RUN (verifier_movsx); }void test_verifier_netfilter_ctx (void ) {RUN (verifier_netfilter_ctx); }void test_verifier_netfilter_retcode (void ) {RUN (verifier_netfilter_retcode); }void test_verifier_prevent_map_lookup (void ) {RUN (verifier_prevent_map_lookup); }
tools/testing/selftests/bpf/progs/verifier_movsx.c new file +213 -0
213 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 13/17 selftests/bpf: Add unit tests for new bswap insns
In reply to Yonghong SongAdd unit tests for bswap insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_bswap.c | 59 +++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_bswap.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -11,6 +11,7 @@#include "verifier_bounds_deduction_non_const.skel.h" #include "verifier_bounds_mix_sign_unsign.skel.h" #include "verifier_bpf_get_stack.skel.h" +#include "verifier_bswap.skel.h" #include "verifier_btf_ctx_access.skel.h" #include "verifier_cfg.skel.h" #include "verifier_cgroup_inv_retcode.skel.h"
@@ -115,6 +116,7 @@ void test_verifier_bounds_deduction(void) { RUN(verifier_bounds_deduction);void test_verifier_bounds_deduction_non_const (void ) {RUN (verifier_bounds_deduction_non_const); }void test_verifier_bounds_mix_sign_unsign (void ) {RUN (verifier_bounds_mix_sign_unsign); }void test_verifier_bpf_get_stack (void ) {RUN (verifier_bpf_get_stack); } +void test_verifier_bswap (void ) {RUN (verifier_bswap); }void test_verifier_btf_ctx_access (void ) {RUN (verifier_btf_ctx_access); }void test_verifier_cfg (void ) {RUN (verifier_cfg); }void test_verifier_cgroup_inv_retcode (void ) {RUN (verifier_cgroup_inv_retcode); }
tools/testing/selftests/bpf/progs/verifier_bswap.c new file +59 -0
59 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 14/17 selftests/bpf: Add unit tests for new sdiv/smod insns
In reply to Yonghong SongAdd unit tests for sdiv/smod insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_sdiv.c | 781 ++++++++++++++++++ 2 files changed, 783 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -54,6 +54,7 @@#include "verifier_ringbuf.skel.h" #include "verifier_runtime_jit.skel.h" #include "verifier_scalar_ids.skel.h" +#include "verifier_sdiv.skel.h" #include "verifier_search_pruning.skel.h" #include "verifier_sock.skel.h" #include "verifier_spill_fill.skel.h"
@@ -159,6 +160,7 @@ void test_verifier_regalloc(void) { RUN(verifier_regalloc); }void test_verifier_ringbuf (void ) {RUN (verifier_ringbuf); }void test_verifier_runtime_jit (void ) {RUN (verifier_runtime_jit); }void test_verifier_scalar_ids (void ) {RUN (verifier_scalar_ids); } +void test_verifier_sdiv (void ) {RUN (verifier_sdiv); }void test_verifier_search_pruning (void ) {RUN (verifier_search_pruning); }void test_verifier_sock (void ) {RUN (verifier_sock); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }
tools/testing/selftests/bpf/progs/verifier_sdiv.c new file +781 -0
781 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 15/17 selftests/bpf: Add unit tests for new gotol insn
In reply to Yonghong SongAdd unit tests for gotol insn.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_gotol.c | 44 +++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotol.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -25,6 +25,7 @@#include "verifier_direct_stack_access_wraparound.skel.h" #include "verifier_div0.skel.h" #include "verifier_div_overflow.skel.h" +#include "verifier_gotol.skel.h" #include "verifier_helper_access_var_len.skel.h" #include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h"
@@ -131,6 +132,7 @@ void test_verifier_direct_packet_access(void) { RUN(verifier_direct_packet_accesvoid test_verifier_direct_stack_access_wraparound (void ) {RUN (verifier_direct_stack_access_wraparound); }void test_verifier_div0 (void ) {RUN (verifier_div0); }void test_verifier_div_overflow (void ) {RUN (verifier_div_overflow); } +void test_verifier_gotol (void ) {RUN (verifier_gotol); }void test_verifier_helper_access_var_len (void ) {RUN (verifier_helper_access_var_len); }void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }
tools/testing/selftests/bpf/progs/verifier_gotol.c new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 16/17 selftests/bpf: Test ldsx with more complex cases
In reply to Yonghong SongThe following ldsx cases are tested:
- signed readonly map value
- read/write map value
- probed memory
- not-narrowed ctx field access
- narrowed ctx field access.
Without previous proper verifier/git handling, the test will fail.
If cpuv4 is not supported either by compiler or by jit, the test will be skipped.
# ./test_progs -t ldsx_insn #113/1 ldsx_insn/map_val and probed_memory:SKIP #113/2 ldsx_insn/ctx_member_sign_ext:SKIP #113/3 ldsx_insn/ctx_member_narrow_sign_ext:SKIP #113 ldsx_insn:SKIP Summary: 1/0 PASSED, 3 SKIPPED, 0 FAILED
--- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 139 ++++++++++++++++++ .../selftests/bpf/progs/test_ldsx_insn.c | 118 +++++++++++++++ 3 files changed, 265 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/test_ldsx_insn.c
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +8 -1
@@ -98,6 +98,12 @@ bpf_testmod_test_struct_arg_8(u64 a, void *b, short c, int d, void *e,return bpf_testmod_test_struct_arg_result; } +noinline int +bpf_testmod_test_arg_ptr_to_struct (struct bpf_testmod_struct_arg_1 *a) { + bpf_testmod_test_struct_arg_result = a->a; +return bpf_testmod_test_struct_arg_result; +} + __bpf_kfunc voidbpf_testmod_test_mod_kfunc (int i) {
@@ -240,7 +246,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, .off = off, .len = len, }; -struct bpf_testmod_struct_arg_1 struct_arg1 = {10 }; +struct bpf_testmod_struct_arg_1 struct_arg1 = {10 }, struct_arg1_2 = {-1 };struct bpf_testmod_struct_arg_2 struct_arg2 = {2 ,3 };struct bpf_testmod_struct_arg_3 *struct_arg3;struct bpf_testmod_struct_arg_4 struct_arg4 = {21 ,22 };
@@ -259,6 +265,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, (void )bpf_testmod_test_struct_arg_8 (16 , (void *)17 ,18 ,19 , (void *)20 , struct_arg4,23 ); + (void )bpf_testmod_test_arg_ptr_to_struct (&struct_arg1_2); struct_arg3 =kmalloc ((sizeof (struct bpf_testmod_struct_arg_3 ) +sizeof (int )),GFP_KERNEL );
tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c new file +139 -0
139 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_ldsx_insn.c new file +118 -0
118 lines not on this page · raw message
Signature · 1 line
2.34.1
Yonghong Song
reviewerPATCH v4 17/17 docs/bpf: Add documentation for new instructions
In reply to Yonghong SongAdd documentation in instruction-set.rst for new instruction encoding and their corresponding operations. Also removed the question related to 'no BPF_SDIV' in bpf_design_QA.rst since we have BPF_SDIV insn now.
--- Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 115 ++++++++++++------ 2 files changed, 79 insertions(+), 41 deletions(-)
Documentation/bpf/bpf_design_QA.rst +0 -5
@@ -140,11 +140,6 @@ A: Because if we picked one-to-one relationship to x64 it would have made it more complicated to support on arm64 and other archs. Also it needs div-by-zero runtime check. -Q: Why there is no BPF_SDIV for signed divide operation? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A: Because it would be rarely used. llvm errors in such case and -prints a suggestion to use unsigned divide instead. - Q: Why BPF has implicit prologue and epilogue? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A: Because architectures like sparc have register windows and in general
Documentation/bpf/standardization/instruction-set.rst +79 -36
169 lines not on this page · raw message
Signature · 1 line
2.34.1
RE: [PATCH bpf-next v4 17/17] docs/bpf: Add documentation for new instructions
Quoted 252 lines
-----Original Message-----
From: Yonghong Song <[email protected]>
Sent: Wednesday, July 26, 2023 3:09 PM
To: Alexei Starovoitov <[email protected]>; Andrii Nakryiko <[email protected]>; [email protected]; Daniel Borkmann <[email protected]>; Martin KaFai Lau <[email protected]>
Cc: David Faust <[email protected]>; Fangrui Song <[email protected]>; Jose E . Marchesi <[email protected]>; [email protected]
Subject: [PATCH bpf-next v4 17/17] docs/bpf: Add documentation for new instructionsAdd documentation in
instruction-set.rstfor new instruction encoding and their corresponding operations. Also removed the question related to 'noBPF_SDIV' inbpf_design_QA.rstsince we haveBPF_SDIVinsn now.Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 115 ++++++++++++------ 2 files changed, 79 insertions(+), 41 deletions(-)Documentation/bpf/bpf_design_QA.rst +0
-0@@ -140,11 +140,6 @@ A: Because if we picked one-to-one relationship tox64 it would have made it more complicated to support on arm64 and other archs. Also it needs div-by-zero runtime check.
Documentation/bpf/bpf_design_QA.rst +0
-3-Q: Why there is no BPF_SDIV for signed divide operation? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A: Because it would be rarely used. llvm errors in such case and -prints asuggestion to use unsigned divide instead.
-
Q: Why BPF has implicit prologue and epilogue?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A: Because architectures like sparc have register windows and in general diff
--gita/Documentation/bpf/standardization/instruction-set.rstb/Documentation/bpf/standardization/instruction-set.rst
index751e657973f0..f36bee41c719100644Documentation/bpf/standardization/instruction-set.rst +0
-0@@ -154,24 +154,27 @@ otherwise identical operations. The 'code' field encodes the operation as below, where 'src' and 'dst' refer tothe values of the source and destination registers, respectively.
-======== =====
==========================================================
-code value description
-======== =====
==========================================================Documentation/bpf/standardization/instruction-set.rst +0
-14-BPF_ADD 0x00 dst += src -BPF_SUB 0x10 dst -= src -BPF_MUL 0x20 dst \*= src -BPF_DIV 0x30 dst = (src != 0) ? (dst / src) : 0 -BPF_OR 0x40 dst \|= src -BPF_AND 0x50 dst &= src -BPF_LSH 0x60 dst <<= (src & mask) -BPF_RSH 0x70 dst >>= (src & mask) -BPF_NEG 0x80 dst = -src -BPF_MOD 0x90 dst = (src != 0) ? (dst % src) : dst -BPF_XOR 0xa0 dst ^= src -BPF_MOV 0xb0 dst = src -BPF_ARSH 0xc0 sign extending dst >>= (src & mask) -BPF_END 0xd0 byte swap operations (see `Byte swap instructions`_below)
-======== =====
==========================================================
+======== ===== =======
==========================================================+code value offset description +======== ===== =================================================================
Documentation/bpf/standardization/instruction-set.rst +17
-0+BPF_ADD 0x00 0 dst += src +BPF_SUB 0x10 0 dst -= src +BPF_MUL 0x20 0 dst \*= src +BPF_DIV 0x30 0 dst = (src != 0) ? (dst / src) : 0 +BPF_SDIV 0x30 1 dst = (src != 0) ? (dst s/ src) : 0 +BPF_OR 0x40 0 dst \|= src +BPF_AND 0x50 0 dst &= src +BPF_LSH 0x60 0 dst <<= (src & mask) +BPF_RSH 0x70 0 dst >>= (src & mask) +BPF_NEG 0x80 0 dst = -src +BPF_MOD 0x90 0 dst = (src != 0) ? (dst % src) : dst +BPF_SMOD 0x90 1 dst = (src != 0) ? (dst s% src) : dst +BPF_XOR 0xa0 0 dst ^= src +BPF_MOV 0xb0 0 dst = src +BPF_MOVSX 0xb0 8/16/32 dst = (s8,s16,s32)src +BPF_ARSH 0xc0 0 sign extending dst >>= (src & mask) +BPF_END 0xd0 0 byte swap operations (see `Byte swap instructions`_below)
+======== ===== ============ +==========================================================Underflow and overflow are allowed during arithmetic operations, meaning
the 64-bit or 32-bit value will wrap. If eBPF program execution would @@ - 198,11 +201,20 @@ where '(u32)' indicates that the upper 32 bits are zeroed.dst = dst ^ imm32
-Also note that the division and modulo operations are unsigned. Thus, for - ``BPF_ALU``, 'imm' is first interpreted as an unsigned 32-bit value, whereas - for ``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result - interpreted as an unsigned 64-bit value. There are no instructions for -signed division or modulo.
Documentation/bpf/standardization/instruction-set.rst +4
-0+Note that most instructions have instruction offset of 0. But three +instructions (BPF_SDIV, BPF_SMOD, BPF_MOVSX) have non-zero offset. + +The devision and modulo operations support both unsigned and signedflavors.
Documentation/bpf/standardization/instruction-set.rst +6
-0+For unsigned operation (BPF_DIV and BPF_MOD), for ``BPF_ALU``, 'imm' is +first interpreted as an unsigned 32-bit value, whereas for +``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result +interpreted as an unsigned 64-bit value. For signed operation +(BPF_SDIV and BPF_SMOD), for ``BPF_ALU``, 'imm' is interpreted as a +signed value. For ``BPF_ALU64``, the 'imm' is sign extended from 32 to 64and interpreted as a signed 64-bit value.
Documentation/bpf/standardization/instruction-set.rst +3
-0+ +Instruction BPF_MOVSX does move operation with sign extension. +``BPF_ALU | MOVSX`` sign extendes 8-bit and 16-bit into 32-bit and upper32-bit are zeroed.
+``BPF_ALU64 | MOVSX`` sign extends 8-bit, 16-bit and 32-bit into 64-bit.Shift operations use a mask of
0x3F(63) for 64-bit operations and0x1F(31)
for 32-bit operations.Documentation/bpf/standardization/instruction-set.rst +2
-2@@ -210,21 +222,23 @@ for 32-bit operations. Byte swap instructions ~~~~~~~~~~~~~~~~~~~~~~ -The byte swap instructions use an instruction class of ``BPF_ALU`` and a 4-bit -'code' field of ``BPF_END``. +The byte swap instructions use instruction classes of ``BPF_ALU`` and +``BPF_ALU64`` and a 4-bit 'code' field of ``BPF_END``. The byte swap instructions operate on the destination register only and donot use a separate source register or immediate value.
-The 1-bit source operand field in the opcode is used to select what byte - order the operation convert from or to:
Documentation/bpf/standardization/instruction-set.rst +3
-0+For ``BPF_ALU``, the 1-bit source operand field in the opcode is used +to select what byte order the operation convert from or to. For +``BPF_ALU64``, the 1-bit source operand field in the opcode is not used andmust be 0.
-========= =====
=================================================
-source value description
-========= =====
=================================================Documentation/bpf/standardization/instruction-set.rst +0
-3-BPF_TO_LE 0x00 convert between host byte order and little endian -BPF_TO_BE 0x08 convert between host byte order and big endian -========= ======================================================
+========= ========= =====
=================================================+class source value description +========= ========= ======================================================
+BPF_ALUBPF_TO_LE0x00convert between host byte order and little
endian
+BPF_ALUBPF_TO_BE0x08convert between host byte order and big
endian+BPF_ALU64 BPF_TO_LE 0x00 do byte swap unconditionally +========= ========= =====+=================================================
The 'imm' field encodes the width of the swap operations. The following
widths are supported: 16, 32 and 64.Documentation/bpf/standardization/instruction-set.rst +6
-0@@ -239,6 +253,12 @@ Examples: dst = htobe64(dst) +``BPF_ALU64 | BPF_TO_LE | BPF_END`` with imm = 16/32/64 means:: + + dst = bswap16 dst + dst = bswap32 dst + dst = bswap64 dst + Jump instructions -----------------@@ -249,7 +269,8 @@ The 'code' field encodes the operation as below: ======== ===== === ====================================================================================
code value src description notes ======== ===== === ====================================================================================
Documentation/bpf/standardization/instruction-set.rst +13
-1-BPF_JA 0x0 0x0 PC += offset BPF_JMP only +BPF_JA 0x0 0x0 PC += offset BPF_JMP class +BPF_JA 0x0 0x0 PC += imm BPF_JMP32 class BPF_JEQ 0x1 any PC += offset if dst == src BPF_JGT 0x2 any PC += offset if dst > src unsigned BPF_JGE 0x3 any PC += offset if dst >= src unsigned@@ -278,6 +299,16 @@ Example: where 's>=' indicates a signed '>=' comparison. +``BPF_JA | BPF_K | BPF_JMP32`` (0x06) means:: + + gotol +imm + +where 'imm' means the branch offset comes from insn 'imm' field. + +Note there are two flavors of BPF_JA instrions. BPF_JMP class permits +16-bit jump offset while +BPF_JMP32 permits 32-bit jump offset. A >16bit conditional jmp can be +converted to a <16bit conditional jmp plus a 32-bit unconditional jump. + Helper functions ~~~~~~~~~~~~~~~~@@ -320,6 +351,7 @@ The mode modifier is one of: BPF_ABS 0x20 legacy BPF packet access (absolute) `Legacy BPFPacket access instructions`_
BPF_IND0x40legacy BPF packet access (indirect) `Legacy BPF Packet
access instructions`_
BPF_MEM0x60regular load and store operations `Regular load and
store operations`_
+BPF_MEMSX0x80sign-extension load operations `Sign-extension
load operations`_BPF_ATOMIC 0xc0 atomic operations `Atomic operations`_ ============= ===== =================================================
Documentation/bpf/standardization/instruction-set.rst +0
-0@@ -350,9 +382,20 @@ instructions that transfer data between a registerand memory.
``BPF_MEM | <size> | BPF_LDX`` means::
Documentation/bpf/standardization/instruction-set.rst +13
-3- dst = *(size *) (src + offset) + dst = *(unsigned size *) (src + offset) + +Where size is one of: ``BPF_B``, ``BPF_H``, ``BPF_W``, or ``BPF_DW`` +and 'unsigned size' is one of u8, u16, u32 and u64. + +The ``BPF_MEMSX`` mode modifier is used to encode sign-extension load +instructions that transfer data between a register and memory. + +``BPF_MEMSX | <size> | BPF_LDX`` means:: + + dst = *(signed size *) (src + offset) -Where size is one of: ``BPF_B``, ``BPF_H``, ``BPF_W``, or ``BPF_DW``. +Where size is one of: ``BPF_B``, ``BPF_H`` or ``BPF_W``, and 'signed +size' is one of s8, s16 and s32. Atomic operations ----------------- --
Quoted 2 lines
2.34.1
All changes under the "standardization" directory should also cc [email protected], so adding that DL to the cc line for this thread.
Dave